Which is why it is so vital for anyone with at least minimal tech know-how to run their own email infrastructure. The Internet is only a meaningful concept if it consists of peers running standard decentralized protocols.
If all we had was a few giant corporations with proprietary apps and proprietary protocols, that's not the Internet.
As I post every time on these email discussions, the difficulty of running your own email infrastructure is vastly overstated on HN. The more people do it, the better off the Internet world is for all of us.
I have run my own email infrastructure for 3 years, for personal use (I was the only user). I have since moved to using fastmail due to my mail failing to be delivered to any microsoft email. Microsoft would accept the mail, but it would never appear in the mailbox, and neither would it arrive in spam. The mail just disappeared somewhere in their system.
I have spent many days debugging the issue, to no avail. Every other operator accepts my email, and my domain was never used to send out spam (I set up DMARC to make sure I am aware of all outgoing email). I imagine the IP might have been the issue, but I cycled through a few hosts without success.
Everything was configured correctly AFAICT, I could send email to all major providers, except microsoft. SPF, DKIM, DMARC were all set up. I tried many different configuration testers, and they all returned green on my domain.
Since migrating to fastmail (keeping the same domain), I haven't had any problem.
The thing is, I agree with you, the world would be a better place if everyone could host their own email. But at the end of the day, I need the ability to contact users using the major email providers, and so do I suspect many HN users.
I've been running my own email infrastructure since 1993, starting off with a UUCP feed to an Amiga over a 9600 baud modem. After that, I think I went Smail -> Sendmail -> Qmail -> Postfix on a variety of platforms (Linux, Solaris, FreeBSD.)
I'm running an email server for myself since 1998 and in addition some low traffic email servers for a few small companies that I administrate and we have had no delivery problems so far, not even with Google, although I have heard from other people that run their own email servers, that Google occasionally puts delivered emails in the Spam folder.
I always keep an eye if some of our IPs may appear on any of the well known abuse lists, but so far that never has happened. I would guess if once there's coming spam from your mail server IP, it is negatively branded forever.
I've had my setup running for so many years with minimal changes so don't have recent links handy but I'll give it a shot. This also points to the fact that while the initial setup will take some reading and effort, the ongoing maintenance year to year is basically zero.
Start with a good IP address (static, not residential ISP, not on any block list). Search for sites that will help check this. https://mxtoolbox.com/ has many useful bits but there are others, try them all. Set up DNS correctly, forward and reverse.
Set up postfix. Tons of guides out there, here's just one list: https://www.linode.com/docs/guides/email/postfix/
Read the various guides but most importantly read the postfix documentation in detail. It is very good and has everything you need to know. Configure postfix to enforce all sanity checks that it supports during the delivery connection phase. This alone will cut off nearly all spam!
Install a local MUA (mutt would be my favorite) and you're up and running and should be able to send and receive, so test that thoroughly. Probably ending up in spam at this stage but should work. Mark it not spam in recipient. Note that if you had to register a new domain name, it'll likely be penalized as too new for a good while.
Set up a cert (Lets Encrypt) for postfix.
Configure SPF correctly. Read online guides but also read the RFC. Again, use online tools to validate your work. SPF is great.
DMARC, DKIM are more controversial and less clear-cut useful than SPF. Read about them. I do set these as well. Set the notification email so you get reports (which aren't that useful but still). Use the online tools to validate.
If all we had was a few giant corporations with proprietary apps and proprietary protocols, that's not the Internet.
As I post every time on these email discussions, the difficulty of running your own email infrastructure is vastly overstated on HN. The more people do it, the better off the Internet world is for all of us.