The list is immense, and I didn’t want to clutter my post with all the details. So just listing off things that spring to mind (because I don’t know what OP doesn’t know):
- Choosing an MTA - sendmail, postfix, exim, etc. and why you might choose one over the others
- Firewall settings
- Software/package management on your chosen distro
- Learning about DNS:
- Host it - yourself via BIND
- Or via a DNS service provider
- DNS record types
- Domains
- Subdomains
- A records/CNAMEs
- MX records
- Mail authority records - SPF’s
- Mail encryption records - DKIM
- Spam filtering, anti-virus
- Learning how to configure your MTA, which requires learning:
- the configuration file language your MTA uses
- what all the options mean and what they do
- what the bare minimum options are to get up and running
- how to make sure your configuration is secure and won’t be exploitable by bad actors
- how mail really gets delivered
- how to setup secure smtp
- how to set up SPFs
- troubleshooting why GMAIL or Microsoft won’t accept your mail
- troubleshooting why GMAIL or Microsoft have stopped accepting your mail
- dealing with blacklists/greylists when someone sends too many messages, or something that "looks too spammy"
- Mail hosting pitfalls
- Being an open relay
- Rate limiting
- Reputation management
- Vulnerabilities that let a hacker take over your server
- Resource management - disk, memory, processes, queues, etc.
- Downtime when you need to do updates
- Downtime if you change your DNS configuration
I’ve definitely missed some stuff, and each of those things requires knowing other stuff too, so you can see that it’s really a pretty deep subject. This is precisely why not many people self-host email themselves these days - the big guys have made it harder and harder to do so, in the name of eradicating spam, which they themselves are the biggest vectors for.