Alright I'll bite.
Let me make this clear. People who are "racists" don't give 2 shits about your skin colour, eye shape, skull size and any of that physical traits, excluding the fraction that finds those physical traits repulsive, which aren't racists but just people with different aesthetic tastes.
It has only been a couple centuries since people were able cross continents with ships. Naturally, people who lived nearby each other have the same skin colour, due to geographical location and the climate and whatnot. People who live near each other, surprise surprise, have similar culture with one another, and share similar ideologies, morality, and behaviour. It is both easy and reasonable to assume people with the same skin colour, the most notable physical characteristic, share the same ideology, morality, and behaviour.
When these people from different culture meet, naturally there will be conflict. Say what you want about democracy, majority will always win in terms of pure strength and accompanied by mob mentality, becomes "justice" and "morally right." Now, when the majority agrees to a certain set of ideology and morality, it becomes the set standard of that majority, and civilization is able to progress.
When people of different CULTURE, or idea of different "JUSTICE" attempt to convert the majority's civilization into their own set of ideology and morality, the majority identify these intruders by their skin colour, their most notable physical characteristics. In other words, "racists" have the RIGHT to assume you have a different idea of JUSTICE based on your SKIN COLOUR.
In the same manner a liberal would hate a guy wearing a swastika because the idiot thought it looked cool, but don't have a clue what the nazi party even is, "racists" ARE ALLOWED to assume you have degenerate morality and ideology based on your appearance.