But my question to that is, are primaries not decided by the voters to get the most delegates?

In the 2016 primaries, 15% of the delegates were superdelegates, who could vote however they wanted. So no, not necessarily. On that basis alone the 2016 primary could have gone roughly 58% / 42% in Bernie’s favor, as far as voters were concerned, and Hillary would have still won.