I’m torn because I want to tell him to leave me alone, that I don’t care about his life, but considering the ‘offense’ this seems too much and knowing me I’d immediately regret it and feel bad about it.

Imagine someone has a huge booger hanging out of their nose. It might be embarassing that you point it out to them, but it’s a lot more embarassing if they walk around with the booger hanging there. It’s the same with this. If you’re polite but direct, there’s nothing to feel bad about, you’re helping the guy learn where the line is.