…that’s how it’s supposed to work, yes. You chose your party affiliation first, then you chose how this party is shaped.
Do I also think it’s stupid? Absolutely. But that’s the system you have, and you don’t get another one by just ignoring it really hard.