This makes sense!

t which point I get annoyed at someone trying to force other people to stop doing something that matters to them, even if I’m not doing the thing.

That’s why I said that, in the hypothetical, you already agree with the ban! Otherwise, you’d be upset that someone was infringing on someone else’s autonomy for no good reason - in other words, you’d be upset that someone is being upset. Which, yeah I mean it’s really the same thing, in reverse, I guess.