The difference between a joke and bullying is entirely down to the victim. They are allowed to be as sensitive as they want. “Its just a joke” tries to turn it back on them. The only reasonable response is along the lines of "I intended it as a joke, but obviously screwed up. I’m sorry. "

The only grey area are those who are happy to dish it out, but not receive. You should expect people to wind you up to the same level you wind them/others to.