“I’m not interested in discussing this matter with you; please drop the subject so I can focus on work.”

Politely insist he drop it multiple times. If he refuses, depending on how severely this is affecting you, you are well within your rights to seek out a supervisor to discuss the issue. But I’d be as plain and direct as possible to minimize any chance of him turning the tables on you and complaining to a superior first.

Or, make it clear you’re not interested and just go on ignoring him?