I understand this makes you unsure, but I really think this would be a perfectly normal question to him: “Hey, I really enjoy your company, but I have noticed in the past with other Japanese friends that they sometimes have mistaken a hug for me asking for more. How do you see that?”

Especially with him being an older guy (which generally means a bit less insecure and more mature) I would expect this to be a good start for an honest conversation.

I hope it does work out the way you want!