It does not seem uncommon for guys who are friendly with one another to throw insults at one another in a non-serious manner. That’s not really the relationship I have with my friends, but it is a thing. I’m not aware of that being much of a thing without some degree of established rapport though. He could have just been doing friendly ribbing as he said, but I think it’s more likely that he decided he was more annoyed at me than at you.