Spam protection is hard given SMTP was never designed with it in mind.
I also self-host my email, but I use an outbound SMTP relay to avoid having to deal with all that stuff. My server sends outbound emails to a company that’s got that all figured out. Maybe that’s not “true” self hosting, but it’s really no different to people that self-host but put Cloudflare in front of their server.