Well, I did say (emphasis added):

I’d be willing to bet that most aren’t or even know about it. Defederating due to what some admins and where the majority of users don’t know

So are they Nazis if they don’t know they’re sitting at a table with a Nazi? Of course not. That saying only works when they knowingly accept it. It really is as simple as that.