A big part of it, I think: the Steam Controller is different in ways that are unpleasant if you approach it like a standard controller. For example, it is not designed to be gripped around the handles like an Xbox controller, but to rest in your fingers. If you attempt to grip it like a traditional controller, it is uncomfortable and the trackpads are hard to use.

I have a friend who grew to like his Steam Controller after using the trackpads on his Steam Deck. For him, it was realizing the potential of the hardware combined with Steam Input.