I’m not a native English speaker, but I always thought an expression like “oh boy” was more self-reflective. Like you say it more to yourself than to whoever you’re talking to.
In which case, the other’s gender shouldn’t really matter?
In any case, I personally don’t feel insulted or misgendered if someone uses it in a conversation with me.