So I recently managed to make my Self-Hosted mailserver an Open Relay. This is bad.
My mailserver (dockerized mailcow) currently runs on a little NUC under my stairs. It has worked well with only minor problems over the 3 or so years I’ve had it running; I got spamhaused once, etc.
The problem all started with me trying to patch a percieved security hole. See, docker doesn’t respect firewalls like UFW or firewalld (all based on iptables of course), instead opting to allow ports through iptables as you add -p flags to your containers in spite of any other rules you may have.
Now I thought this was rather terrible. I don’t want to have to look both at my firewall and at all my docker port bindings to check if something is open. So as many of us would do, I started trawling the internet for solutions and started to learn about why this behaviour existed.
According to some articles/stackoverflows/etc. the way to stop docker messing with iptables and creating its own rules is to disable the feature in the daemon.json. Seems simple enough. The only caveat that I found mentioned was that container networking would break (in terms of internet reachability) but that’s ok because I just had to add a firewalld rule to allow masquerading and that problem was solved.
Now the problem I failed to see was that of NAT changing. Prior to disabling the iptables flag, the mailserver would see connections’ IPs as their real public ones. However afterwards, every single IP was that of the internal docker network default route.
I didn’t think much of it at the time, merely that it would be more annoying to see who was connecting but that was fine because I had what I wanted. Firewalld was now the sole controller of my ports 🎉
Little did I know (or maybe I did and just forgot) that postfix has a trusted list of IPs and it will relay anything from them without question. These IPs include internal IPs such as that of the default route…
So essentially every SMTP request was being NATed to have a sender address of 172.22.1.1 and postfix started sending EVERYTHING 😵💫
It wasn’t long before a plethora of bots had saturated my poor NUC with HUNDERDS OF THOUSANDS of emails.
I got home this evening to lag spikes in Tarkov which prompted me to check the server where I found this mess.
After taking everything down, re-enabling the iptables and flushing all the postfix queues, I was able to spin back up and not have the whole thing start spiralling again.
Some tips for those hosting mailservers:
- Use a mail server checker like https://mxtoolbox.com/SuperTool.aspx
- Setup monitoring and alerts for server CPU usage/spikes in requests/etc
- Don’t fix what ain’t broke
I’m gonna go cry myself to sleep now and pray that the big mail hosts like Google and Microsoft take pity on me and my screw up. (We all know I’ll never be able to send another email to Microsoft again, who am I kidding)
You must log in or register to comment.
- Use a VPS.
- A write-up please. This is beyond my current understanding of Docker networking and more resources would really help
A VPS doesn’t protect you from configuration mistakes.
I’ve been trying to avoid one but the time has probably come. I’ve been thinking about writing more/making a YouTube channel so perhaps you’ll see something on it :)
I did something similar way back in 2002 and my ISP shut me down hard. I’m surprised any ISP in this day and age allows a residential connection send any emails out.
This is one of the many reasons I won’t ever host an email server again.
Good luck!
I’m praying mine hasn’t realised 😅 And thanks, I’ll need it
Plusnet don’t mind you running a mail server, I recall reading articles about it on their forum. May be worth a quick search for your own reassurance :)
They also allow you to control the firewall and port 25 at https://www.plus.net/member-centre/broadband/firewall
They can’t really stop you from sending emails out without potentially breaking legitimate mail. They can stop email from being submitted to you by blocking port 25/465/587 inbound though.
Most consumer ISPs these days block outgoing 25, and it’s been that way since the late 90s. Third party mail providers generally ask you to use 465 or 587 for that reason.
I just tested with four separate West coast ISPs and none of them are blocking outgoing port 25. Maybe it’s a regional thing
Spectrum in the US allows the customer to replace their provided router with their own equipment, so all they provided to me is the modem, after that I have my own router, so I can pretty much do whatever I want. Never had an issue hosting a mail server, sending/receiving works perfect.
This has got nothing to do with hosting your own equipment like routers (with recent regulatory changes, all ISPs in the US are now required to allow this - i.e. be able to run your own equipment without paying a monthly fee to the ISP).
But this is more about ISPs blocking sending on specific ports like port 25 for SMTP. Instead you need to use some mail relay to send your email, who in turn will send the mail over SMTP on port 25.
Some ISPs in the US do allow port 25 SMTP, but very few and none of the large ones like Comcast, AT&T, Verizon.
Spectrum is the second largest ISP in the US and I have zero issues sending or receiving on my mail server, no relay in place
How do you get good judgement? Experience.
How do you get experience?
Bad judgement.:-)
yep, I’ll bet op won’t make that mistake again! I did it back when I was running mdeamon on NT4, but that was a long, long time ago
This is the way
Yep, never again 🥹
Yeah you might as well write off the IP. At least with Google it will take months to recover, and if you don’t have a good volume of quality emails go through it the chances are even lower.
Yeah I was afraid of that. Good(?) news is I’m switching ISP soon so my IP will disappear soon. So long, static IPv4, hello CGNAT and IPv6. I’ll need a VPS somewhere to proxy any incoming IPv4 traffic at that point anyway.
You should write up an article on how to setup docker with firewalld
So I recently managed to make my Self-Hosted mailserver an Open Relay. This is bad.
Just because you can self-host smtp doesn’t mean you should.
I’ve run corporate email on prem for years, for tens of thousands of users, and I have no interest in self-hosting email for myself. I’ve also migrated over a hundred thousand mailboxes to EXO, and there was a good reason for that. I personally think smtp is riskier than https.
This is why ISPs typically block port 25. Also, I love containers as much as the next guy but for the reasons mentioned I reduce complexity in all areas of critical systems were it doesn’t belong such as a email server.
You are not the first to do this with docker hosted email servers and you won’t be the last. The Internet is full of people talking about this exact issue.
I did practically the same, while deploying a mail solution docker in an LXC container…
definitely on my list of why i dont run a mail server :-P
Don’t feel too bad. Back in the day, it seemed like every other edit to sendmail.mc would result in an open relay. I don’t miss Sendmail. :)
I would also encourage you to start monitoring your mailq. For a little home server, if you have more than 10 messages in there, you should probably have an alarm going off.
You might also do this for other things like web servers that might send out a contact form email or things like that.
For future purposes:
That or podman!
Honestly, it’s about time Postfix gets replaced with a modern email server. It’s a dinosaur designed around 1990s UNIX, with dozens of leftover footguns eagerly waiting to go off.
The fact that it even allows local mail submission or trusted subnets is already problematic, if you ask me. It is 2023, email should only be allowed after proper authentication & authorization!
This makes no sense. Postfix supports auth, since forever, with varius mechanisms. What is a modern mailserver in your opinion? Honestly curious.
Of course Postfix supports auth, I’m not disputing that.
The problem is that it also supports completely anonymous submission from localhost and from local networks, and there are half a dozen ways to accidentally turn your server into an open relay. This made sense in the 1990s when every machine was hosting its own mail server for the two dozen local users, but we don’t live in that world anymore and support for it should’ve been removed already. If you’re using it something is going seriously wrong in your setup, so why is it allowed at all?
I haven’t looked too closely into it, but something like Stalwart seems closer to my expectations: just a no-nonsense batteries-included secure-by-default mail server.
There are also dozens of “mail in a box” setups out there who try to do the same thing, but they all end up being Rube Goldberg machines built on top of legacy software.
I get your point and in the context of selfhosted it makes lots of sense. But I wouldn’t write of postfix, its included with all the linux distros, in a lot of them as a default mailserver. And would strongly argue that it still has its place, even though its configuration isn’t beginner friendly. Kinda like a crocodile, it hasn’t changed much in all those years, but maybe it didn’t have to.
I’ll add another one to the list of monitoring.
Add an alert of sent email threshold. I log to gray log, and if more than 5 are sent in 5 minutes it will send me an alert.
I am assuming that threshold would be logged in the postfix logs?
Hm. Lag spikes in Tarkov and you check your server? I mean Tarkov.
But yeah I can feel your misconception here. But I am also the other way around I uninstalled firewalld and do all on iptables level. I am just more used to iptables. And so the sole controlling instance is iptables. In the end it’s all netfilter in kernel space.
Excellent write up! I’ve saving this one for sure. How did you go about investigating this and troubleshooting? That deserves a write up by itself!
