Sounds like he’s just using you as his fucktoy.

Dump this bum. Have some more respect for yourself. You deserve love and respect. If someone is going to be in a poly relationship, everyone in that relationship needs to be OK with it. It sounds like his boyfriend isn’t. If his boyfriend being upset is a reason not to be with you, then you aren’t really in a poly relationship. He’s in a relationship with his boyfriend, and he’s just occasionally cheating on his boyfriend with you. A poly relationship requires that all members be fine with the other members getting together. That is the difference between being poly and being a cheating bastard.

Dump this bum.