Step one- the coworkers need to fuck off. Start with a polite, but firm “it’s none of your fucking husiness” and get less polite the more they push.

Step 2, if your brother is trying to weaponize your coworkers, tell him to fucking stop. If he’s just coming around, be somewhat more polite and still tell him to stop. If he switched to “I’m just visiting [the coworkers] your manager probably would like to know- ex employees still coming round is never a good idea even if they left amicably.

Step 2b (or maybe step 3) tell your brother that you don’t want to be drawn into whatever is going on with him and your mom, and that’s why you can’t go. If he’s continues trying to pull you in, you can tell him to fuck off too.

It sounds a lot like your brother has issues with boundaries. Set your boundaries and enforce them. Also don’t let your mom pull you into it on her side, either.