I think it’s more about attachment. We suffer because we get attached to feelings, desire, etc. When we should realize, those, along with most things, are ephemeral, or “not real”. I don’t think it is that Buddhist can’t have desire or are indifferent, but that they strive for lack of attachment. That’s probably a gross oversimplification and, like most religions, there are many different sects.