People like him are dangerous.

For the love of God, do not directly say it to this person, speak to HR or your supervisor and tell them you are feeling harassed at work and can’t do your job.

If you say anything to the man all he’s going to conclude is that you are against him, and his paranoia will kick in, and he will make problems for you that you can’t anticipate.