I think there’re a few crypto based ones out there from when people were trying to make bitcoin in shops a thing, no idea if they’re even still maintained.

If you’re talking about a traditional ATM that hooks into a specific bank or interbank network, you’re not going to find a complete system, because the banks basically only want machines they (or partnered banks) own talking to their systems.

Depending what you’re interested in though, Linux based OSes seem to be becoming increasingly used, but there’s still probably a big majority running some form of windows (or funnily enough OS/2). And a quick Google has shown me the industry has seemingly actually worked out an SDK for talking to the ATM hardware that’s catching on, XFS4IoT. However the software written by the bank to implement that SDK and actually talk to the banking network is (and basically always will be) proprietary software.