The reality is because of the lived experiences of people based on the color of their skin, people are different based on skin color. You’re right that it’s a stupid reason to think differently of people, but if people had been mistreated for many generations based on the color of their hair, and there was still a good chunk of people that something so arbitrary was somehow important, then you would want to approach a person with that hair color with understanding of that history and current struggle.

So why does it matter? Because 100 years ago, their great-great-grandparents had any wealth they managed to build up taken from them, 70 years ago their great-grandparents were kept boxed into separate, substandard areas, and 50 years ago their grandparents were kept from being able to buy homes outside low-income, substandard housing areas, and 30 years ago their parents were told it was their fault for growing up in crime-filled, poor areas with under-funded schools. And the whole time police have continuously treated them as that same substandard, poor, likely-criminal, so they have disproportionately been put in jail or grown up with one parent in jail. This obviously doesn’t apply to everyone, but it’s enough to lead people to treat them differently, either because they presume (until otherwise established) that they are poor, poorly educated, and likely criminal (by basically racist assholes) or with a certain amount of respect for their presumed struggles.

Taking it to an extreme, if a person comes across a very old person with a number tattooed in block letters on their forearm, they will respond one of two ways: with respect and concern for their presumed struggles and trauma, or with irrational hatred (by neo-nazis). Judging or “separating” a person for a barely noticeable tattoo that they didn’t even put on themselves may seem arbitrary, but only if you ignore the entire history that makes them different.