It’s a failure of empathy. People hate and/or fear what they don’t understand.

They can’t empathize with someone not feeling right in their own skin like that, so all their rationalizations for things like “why would they want to use that bathroom” end up stemming from what they have left, which is unsavory intent, making them threatening.