It sounds completely natural to associate a language with the experiences you had in it.

Would it be weird to hate France if you went, had your luggage stolen, then got mugged, and then we’re diagnosed with cancer while you were there?

Humans brains are just association machines, and that language has associations for you. Maybe try to find friends in that community to make your own, positive associations?