I don’t know if this is one of those situations where I should pick my battles and let it go. I feel they steal my free time.

Getting along in work environments is always a matter of which battles to pick and which compromises to make. You will always have to make some. To my mind, starting your shift five minutes early is a pretty minor concession to not irk your colleagues and get on everyone’s shit-list. But…you can always search for a new place to work. Just bear in mind you may be walking into an even less compatible environment for yourself.