I used to do this. What finally killed it wasn't reputation, it was the fact that I needed 100% uptime or risk losing messages, getting my address blacklisted, etc. Email is supposed to be resilient to down time (retries, trying each MX record, etc.) but I found that large mail providers tend to just bounce and walk away.
Worse, GitHub (back in 2016 and 2018) would mark a recipient as "unavailable" after a single bounce, refusing to send any more notifications to that address. They since improved the situation and their support was actually very helpful and responsive here, but it's pretty clear that modern SMTP senders have an expectation that recipients will be "always online" that didn't exist when the protocol was invented.
(had to dug my comment from under a flagged parent)
I self-hosted for well over 20 years, I did not throw the towel and I do not plan to. Self-hosting is a sign of pride. Neither my government nor my Prime Minister nor even my Ministry of Interior or Foreign Ministry can host their own email.
Last time I checked, only State Security self-hosted.
I was probably lucky, but I rarely had delivery problems. The last one was a couple years ago with Microsoft swallowing my emails and it was due to the combination of a fairly old exim and a TLS certificate verification quirk at *.protection.outlook.com. I found a fix in the form of a configuration option somewhere on SO.
In all fairness, there is very little maintenance involved, and whenever I have to do maintenance work, I take the opportunity to learn something new. Like this year, I decided to finally replace my aging Debian jessie setup by Arch Linux, and I rewrote all cron jobs as systemd timers.
I must admit that when I send a really important email, I check the mail server log if it went off without errors, but this does not bother me as checking logs manually once in a while is a good thing anyway.
Lastly, a piece of advice: treat self-hosting like a hobby and learn to enjoy it.
Oh and the very last thing: the person who designed Exim configuration for Debian deserves a special place in hell for all the hours wasted. If you set up Exim on Debian, just figure out how to use the upstream exim config and adapt it to your needs.
Here is my advice to anyone wanting to test out self-hosting email. Start by using your self-hosted email to sign-up for accounts. You don't have to use the email address for your personal correspondence
Use Mail-in-a-box to get started [1]. You can literally set it up in a couple of hours by following the instructions and everything should just work.
After a few years, you can think about switching your personal correspondence to your new email.
I can recommend Stalwart [1] which is a complete mail service contained in a single binary, that doesn't really have any external dependencies, and is really easy to install and update.
I've looked (and tried) a few other projects in the past, but Stalwart was the easiest to setup, and I haven't had any issues with it so far.
It’s also what Thunderbird is using to build their paid email hosting. Seems like a very ambitious project mostly done by a single person – impressive!
I've been running MIAB for a few years now with generally good success as an outgoing sender using a rented cloud machine and a "clean" reputation IP. I've had to email the Microsoft postmaster on one occasion when my emails weren't reaching Outlook users, but they were surprisingly helpful and it's been working fine for years now. It's a good learning exercise in setting up stuff like DKIM/SPF/DMARC.
That said - receiving account sign-up emails is the absolute biggest pain in the backside with Mailinabox! The greylisting anti-spam feature relies on bouncing unknown senders and waiting for a retry. The trouble is, many legit sites just don't bother retrying. So email verification for new accounts and 2FA-type stuff often takes ages to come through, if at all. MIAB stubbornly has no easy, mail user-facing way to temporarily disable spam filtering and it's a real PITA at times.
Modern email providers, especially ones offered by ISPs often have the same problems that people criticize self-hosted providers for. Even Google has problems. For example, I regularly order via companies that use Shopify. Now, all of the shopify emails are going straight to spam in Gmail, despite constantly marking them as not spam. (These even pass dmarc/spf/dkim etc, so who knows what's going on here.)
Email delivery and receiving is not hard, but it's inevitably going to be imperfect, no matter the provider you use. There are so many bad actors out there, it's surprising that it works as well as it does.
> For example, I regularly order via companies that use Shopify. Now, all of the shopify emails are going straight to spam in Gmail, despite constantly marking them as not spam. (These even pass dmarc/spf/dkim etc, so who knows what's going on here.)
There's a pretty good chance this is because Shopify is sending a lot of email users mark as spam, or is using the same mail server as someone who does. Then you marking them as not spam gives them a better score but the sender's reputation is still so bad that it can't break the threshold to stay out of the spam folder.
> These even pass dmarc/spf/dkim etc, so who knows what's going on here.
Those have nothing to do with being spam, right? Spam is about content, those are about authenticity. Anybody can send authentic trash, or unauthenticated gold.
I have self hosted my email for about twenty years; fr about ten or fifteen I just forwarded everything to Gmail but had to revert to local ( started with local mail in emacs, but switched to imapd to solve the airplane ticket in the airport issue) because so much important stuff was marked as spam. Like in the middle of a conversation between me and on other person their reply to my email (which I always bcc:ed ack to myself) would disappear. Self hosted is much better. It took few iteration to get spf etc working.
That behaviour is the whole problem. If you use a self hosted or small time email provider you're much less likely to have email blocked or filtered by aggressive anti-not-gmail filters.
Hilarious Gmail addresses send tonnes of spam so filtering by provider doesn't do much there days anyway. But Google insists to continue
Bizarrely, I also find Gmail's spam algo is actually oversensitive to marketing emails from companies these days, which I never thought was something I would complain about. But like you said its super annoying when I actually want the emails.
Seems like we had the opposite problem 10ish years ago. But now the pendulum has swung a bit too far in the other direction.
Ultimately most of the spam I get these days is actually from individuals doing low volume cold outreach from personal email addresses...not companies sending bulk. The new gmail unsubscribe feature works great for marketing emails but is worthless against cold email spam -- which somehow rarely ever lands in spam.
Assuming this is not hosted on your home system, since ISPs may block the ports and also most of the dynamic ips allocated are blacklisted, the issue with postfix is that its difficult to have a single set and forget config if you intend to use it on multiple internal machines, like for getting your root email on each system to one mailbox. Ideally you want a single main.cf for all your internal machines and for the outgoing/incoming mailhost to be determined solely by your mx or internal dns alias, but this is next to impossible with a single postfix config without getting mail loops on the system that is the mailhost. Exim and sendmail at least separate out the submit config from the rest of the configuration.
Also you would be insane to try this without fail2ban or something similar. Postfix does a reasonable job of handling attackers but it does so quietly -- so you may not see the activity.
Say I want to test the waters for selfhosting email, and I already have my how domains setup with SaaS like Google workspace and equivalent. Is there a way to setup mx records so that both google and my own server gets email for a while? This would be useful to test the waters over a few months before fully migrating
You can set up a lower-priority MX to point to Google, so if your server fails, then email is delivered to Google. But if your server is misconfigured and returns permanent 5xx errors for legitimate emails, then it won't work, and the emails won't be delivered to Google.
No easy answer here. Individual MTAs or a cluster of them typically live under one unique domain. In your scenario, you'd have to point your existing records (or just MX) to your self-hosted instance, and have your self-hosted instance relay/autoforward to Gmail under a different domain. This might entail simply setting your Gmail back to @gmail.com.
Not really, SMTP relays will only send messages once, to one server.
But it’s not receiving that is the problem, that is generally fine, if ports are open at ISP / network level. It is the sending that is often tricky. Sending email on the other hand can be done from multiple servers (if SPF correctly configured) And nothing prevents you from sending email directly from your own relay. You could try that, and reception would not be affected.
Those wore the days :-) I remember playing on a University lab with half a dozen Unix workstations, sending an email with the path of server1!server2!server3 etc and hearing the email flowing from server to server by the noise of the disks!
Right. You better not self-host like it's 1984 because that would also mean you're an open relay. And vulnerable for pretty much anything you can think of.
Once hosting email for yourself, you may want to add new project-specific domains, or host email for friends and family. The database user accounts actually make it easier to add and remove users after the system is up and running.
This Purplehat guide provides a step by step procedure that's allowed many people and orgs to bring self-hosted email online...
Not sure why someone would go through the pain of cobbling up a self hosted solution based on Postfix when you have fully integrated solutions like https://stalw.art/, which are a breeze to setup.
What about mail servers generally rejecting email (or marking as spam) from residential IP ranges? Decades of malware sending spam has spoiled self hosting emails.
I needed some minimal mail delivery for user registration confirmation and password recovery, and I finally caved and just use some free service. It's okay since those emails are really, really, sparse in my case. But it sucks that email, this one old and open technology, is not realistically self-hostable.
Yeah, hosting on or at least tunneling through a commercial IP address is definitely required in order not to be flagged as spam. Personally, I chose the latter option of hosting my MTA at home but tunneling its traffic through a VPS in a datacenter. It's been working pretty well ever since, although I'm not sure it's worth the effort versus just using a cheap hosted provider.
Almost everything described in the article didn't exist in 1984. Postfix, OpenDKIM, TLS, SPF, DKIM, DMARC. Only very basic SMTP and DNS, but even MX records didn't exist.
I used to do this. What finally killed it wasn't reputation, it was the fact that I needed 100% uptime or risk losing messages, getting my address blacklisted, etc. Email is supposed to be resilient to down time (retries, trying each MX record, etc.) but I found that large mail providers tend to just bounce and walk away.
Worse, GitHub (back in 2016 and 2018) would mark a recipient as "unavailable" after a single bounce, refusing to send any more notifications to that address. They since improved the situation and their support was actually very helpful and responsive here, but it's pretty clear that modern SMTP senders have an expectation that recipients will be "always online" that didn't exist when the protocol was invented.
(had to dug my comment from under a flagged parent)
I self-hosted for well over 20 years, I did not throw the towel and I do not plan to. Self-hosting is a sign of pride. Neither my government nor my Prime Minister nor even my Ministry of Interior or Foreign Ministry can host their own email.
Last time I checked, only State Security self-hosted.
I was probably lucky, but I rarely had delivery problems. The last one was a couple years ago with Microsoft swallowing my emails and it was due to the combination of a fairly old exim and a TLS certificate verification quirk at *.protection.outlook.com. I found a fix in the form of a configuration option somewhere on SO.
In all fairness, there is very little maintenance involved, and whenever I have to do maintenance work, I take the opportunity to learn something new. Like this year, I decided to finally replace my aging Debian jessie setup by Arch Linux, and I rewrote all cron jobs as systemd timers.
I must admit that when I send a really important email, I check the mail server log if it went off without errors, but this does not bother me as checking logs manually once in a while is a good thing anyway.
Lastly, a piece of advice: treat self-hosting like a hobby and learn to enjoy it.
Oh and the very last thing: the person who designed Exim configuration for Debian deserves a special place in hell for all the hours wasted. If you set up Exim on Debian, just figure out how to use the upstream exim config and adapt it to your needs.
> I decided to finally replace my aging Debian jessie setup by Arch Linux, and I rewrote all cron jobs as systemd timers.
Man, I wish I had 1% of the motivation I had 20 years ago to do something like this, before all the full time job, wife and child.
Here is my advice to anyone wanting to test out self-hosting email. Start by using your self-hosted email to sign-up for accounts. You don't have to use the email address for your personal correspondence
Use Mail-in-a-box to get started [1]. You can literally set it up in a couple of hours by following the instructions and everything should just work.
After a few years, you can think about switching your personal correspondence to your new email.
[1] https://mailinabox.email./
I can recommend Stalwart [1] which is a complete mail service contained in a single binary, that doesn't really have any external dependencies, and is really easy to install and update.
I've looked (and tried) a few other projects in the past, but Stalwart was the easiest to setup, and I haven't had any issues with it so far.
[1] https://github.com/stalwartlabs/stalwart
It’s also what Thunderbird is using to build their paid email hosting. Seems like a very ambitious project mostly done by a single person – impressive!
I've been running MIAB for a few years now with generally good success as an outgoing sender using a rented cloud machine and a "clean" reputation IP. I've had to email the Microsoft postmaster on one occasion when my emails weren't reaching Outlook users, but they were surprisingly helpful and it's been working fine for years now. It's a good learning exercise in setting up stuff like DKIM/SPF/DMARC.
That said - receiving account sign-up emails is the absolute biggest pain in the backside with Mailinabox! The greylisting anti-spam feature relies on bouncing unknown senders and waiting for a retry. The trouble is, many legit sites just don't bother retrying. So email verification for new accounts and 2FA-type stuff often takes ages to come through, if at all. MIAB stubbornly has no easy, mail user-facing way to temporarily disable spam filtering and it's a real PITA at times.
Oh! That's what it is. I just thought some websites just took longer to send an email to my unknown domain.
I see that the only way to disable greylisting is to configure the underlying tool [1]. But it also means that SPAM will increase a lot.
[1] https://discourse.mailinabox.email/t/how-to-turn-off-edit-gr...
It's better to whitelist the domains you'll be getting mfa from.
Modern email providers, especially ones offered by ISPs often have the same problems that people criticize self-hosted providers for. Even Google has problems. For example, I regularly order via companies that use Shopify. Now, all of the shopify emails are going straight to spam in Gmail, despite constantly marking them as not spam. (These even pass dmarc/spf/dkim etc, so who knows what's going on here.)
Email delivery and receiving is not hard, but it's inevitably going to be imperfect, no matter the provider you use. There are so many bad actors out there, it's surprising that it works as well as it does.
> For example, I regularly order via companies that use Shopify. Now, all of the shopify emails are going straight to spam in Gmail, despite constantly marking them as not spam. (These even pass dmarc/spf/dkim etc, so who knows what's going on here.)
There's a pretty good chance this is because Shopify is sending a lot of email users mark as spam, or is using the same mail server as someone who does. Then you marking them as not spam gives them a better score but the sender's reputation is still so bad that it can't break the threshold to stay out of the spam folder.
> These even pass dmarc/spf/dkim etc, so who knows what's going on here.
Those have nothing to do with being spam, right? Spam is about content, those are about authenticity. Anybody can send authentic trash, or unauthenticated gold.
I have self hosted my email for about twenty years; fr about ten or fifteen I just forwarded everything to Gmail but had to revert to local ( started with local mail in emacs, but switched to imapd to solve the airplane ticket in the airport issue) because so much important stuff was marked as spam. Like in the middle of a conversation between me and on other person their reply to my email (which I always bcc:ed ack to myself) would disappear. Self hosted is much better. It took few iteration to get spf etc working.
That behaviour is the whole problem. If you use a self hosted or small time email provider you're much less likely to have email blocked or filtered by aggressive anti-not-gmail filters.
Hilarious Gmail addresses send tonnes of spam so filtering by provider doesn't do much there days anyway. But Google insists to continue
Bizarrely, I also find Gmail's spam algo is actually oversensitive to marketing emails from companies these days, which I never thought was something I would complain about. But like you said its super annoying when I actually want the emails.
Seems like we had the opposite problem 10ish years ago. But now the pendulum has swung a bit too far in the other direction.
Ultimately most of the spam I get these days is actually from individuals doing low volume cold outreach from personal email addresses...not companies sending bulk. The new gmail unsubscribe feature works great for marketing emails but is worthless against cold email spam -- which somehow rarely ever lands in spam.
What's the "like it's 1984" part?
Assuming this is not hosted on your home system, since ISPs may block the ports and also most of the dynamic ips allocated are blacklisted, the issue with postfix is that its difficult to have a single set and forget config if you intend to use it on multiple internal machines, like for getting your root email on each system to one mailbox. Ideally you want a single main.cf for all your internal machines and for the outgoing/incoming mailhost to be determined solely by your mx or internal dns alias, but this is next to impossible with a single postfix config without getting mail loops on the system that is the mailhost. Exim and sendmail at least separate out the submit config from the rest of the configuration. Also you would be insane to try this without fail2ban or something similar. Postfix does a reasonable job of handling attackers but it does so quietly -- so you may not see the activity.
More like 1994 thereabouts, in 1984 most of us would be very lucky to have a dial up connection to the local BBS, under local phone call price rates.
Say I want to test the waters for selfhosting email, and I already have my how domains setup with SaaS like Google workspace and equivalent. Is there a way to setup mx records so that both google and my own server gets email for a while? This would be useful to test the waters over a few months before fully migrating
You can set up a lower-priority MX to point to Google, so if your server fails, then email is delivered to Google. But if your server is misconfigured and returns permanent 5xx errors for legitimate emails, then it won't work, and the emails won't be delivered to Google.
No easy answer here. Individual MTAs or a cluster of them typically live under one unique domain. In your scenario, you'd have to point your existing records (or just MX) to your self-hosted instance, and have your self-hosted instance relay/autoforward to Gmail under a different domain. This might entail simply setting your Gmail back to @gmail.com.
Configure google to forward mails to your self hosted server.
When replying reply from your self hosted server.
That way you can gradually shift over.
I had been self hosting like this for years.
Not really, SMTP relays will only send messages once, to one server.
But it’s not receiving that is the problem, that is generally fine, if ports are open at ISP / network level. It is the sending that is often tricky. Sending email on the other hand can be done from multiple servers (if SPF correctly configured) And nothing prevents you from sending email directly from your own relay. You could try that, and reception would not be affected.
Where is UUCP? Why are addresses not bang paths? Where is sendmail.cf?
Those wore the days :-) I remember playing on a University lab with half a dozen Unix workstations, sending an email with the path of server1!server2!server3 etc and hearing the email flowing from server to server by the noise of the disks!
Right. You better not self-host like it's 1984 because that would also mean you're an open relay. And vulnerable for pretty much anything you can think of.
This config doesn’t make an open relay.
This config wasn't available in 1984.
A typical config from 1984 is an open relay and vulnerable to the Morris Worm.
Ditto. I was sorely disappointed to click through "1984" to find a subheading on "setting up postfix".
Actually, full strength virtual (multi-domain) email hosting is also quite doable.
This is a great guide that's been used and updated for many years:
https://www.purplehat.org/?page_id=1450
Once hosting email for yourself, you may want to add new project-specific domains, or host email for friends and family. The database user accounts actually make it easier to add and remove users after the system is up and running.
This Purplehat guide provides a step by step procedure that's allowed many people and orgs to bring self-hosted email online...
Not sure why someone would go through the pain of cobbling up a self hosted solution based on Postfix when you have fully integrated solutions like https://stalw.art/, which are a breeze to setup.
Because postfix is foss, will work with everything and for all time and if there's a problem with it you'll actually be able to fix it.
I thought Stalwart’s license, AGPL is foss.
What about mail servers generally rejecting email (or marking as spam) from residential IP ranges? Decades of malware sending spam has spoiled self hosting emails.
I needed some minimal mail delivery for user registration confirmation and password recovery, and I finally caved and just use some free service. It's okay since those emails are really, really, sparse in my case. But it sucks that email, this one old and open technology, is not realistically self-hostable.
Yeah, hosting on or at least tunneling through a commercial IP address is definitely required in order not to be flagged as spam. Personally, I chose the latter option of hosting my MTA at home but tunneling its traffic through a VPS in a datacenter. It's been working pretty well ever since, although I'm not sure it's worth the effort versus just using a cheap hosted provider.
I haven’t read the article and I am to afraid to open the link in case they are using sendmail.
How long are you going to keep the cat in the box?
Spoiler alert, it’s Postfix. So not really 1984 software. But then again, neither is Linux…
Almost everything described in the article didn't exist in 1984. Postfix, OpenDKIM, TLS, SPF, DKIM, DMARC. Only very basic SMTP and DNS, but even MX records didn't exist.
OpenDKIM seems dead:
https://github.com/trusteddomainproject/OpenDKIM/issues/236
But the experience of using mailx is close to that time, hence the title. Even though I’m too young to know for sure :)