One reason assholes got mad about you doing that was probably because they wanted your attention on them in that moment. Everyone zones out and stares at nothing sometimes but some people are faster at snapping out of it. If the default position for your social engagement switch is “Ignore,” you have a delay while you switch it to “Engage.” Meanwhile the asshole is already all focused on “Me! Me!”