If I agree with the moral logic behind it, then yes - it’ll upset me even if I’m not personally affected. If I hear someone shouting slurs at a black person, I’ll obviously take issue with it, despite not being black myself.

On the other hand, if I hear someone say, for example, “this thing is retarded,” then even if society broadly considers that offensive, I still wouldn’t personally have a problem with it - because I don’t agree with the reasoning behind that judgment.