Shouldn’t you have “remove as mod” when you click on the 3 buttons icon on one of the inactive mods?

I used it in the past between my alts, it works without need of an admin.

I am not sure what you mean with mods being on top of you. AFAIK, mods on Lemmy are equal, there is no such hierarchy.