If you were ever here, how did you solve this?

Simple. Just stay friends. I left a job 15 years ago because the CEO was an asshole. However, our team was great and I still visit some of my old colleagues regularly.