You’ve said that about your manager mode twice now. It must be really important for you that you are a manager or maybe have a manager mode. Yet you seem to struggle with basic communication and letting people know what you want and why.

“This is a bad time for me, can you give me a number that I can call back later?” If they don’t respect that, hang up and let them try again later

I wouldn’t even consider that rude on either side of the call really - just business