I’m going to guess that it doesn’t occur to him that when someone demands your attention like that, you imagine the worst. It might help him to know that.

I’m like you in that regard. I got used to asking back “Is there a problem?” That seemed to help me feel less stress sooner.

Maybe the combination of these two things would help.

Good luck.