I think there are diplomatic ways for you to express your preference. You could say: “I don’t mean to be rude, but I prefer to really stay focused at work and socializing can drain my battery or distract me for a while afterward. Is it all right with you if I keep to myself for the most part?”

He’s obviously taking your behavior as a sign that he’s done something wrong, and not just your preference for how you like to be at work. He’s probably trying to fix it or improve relations with you, and this is making it worse. The standoffish behavior you’re showing him is feeding his sense that he needs to pay more attention to you, so take a different tack.