I don’t think that’s correct. It seems like it originated from a magical context: something that puts a sexual “spell” on you

https://en.wikipedia.org/wiki/Sexual_fetishism#History