I know everyone says it's not practical to run your own mail server, but I wonder if it's something I should have another look at. Even though this article says the cost of SES is 12:1 vs self managed infrastructure, and it's likely that SES is on the cheaper side of things, it's not really about the cost for me, it's about the loss of control.
I don't like having the potential for changes I can't control. It almost feels like I need two of everything just to be safe.
I wish some of the up-and-coming auth systems (ex: logto.io) would have some kind of tiered fallback for transactional email. For example, I have self hosted mail that's tier 1 (use first), AWS SES as tier 2 (use for failover), and Postmark as tier 3 (use as a last resort).
The reason I list Postmark last is because there is a point where price becomes a factor for me. If Postmark is 10x SES for 50k messages, and SES is 12x self managed infrastructure, that's 120x costs (12,000% markup), right? I feel like that's so expensive I might have gotten it wrong, so please correct me if I did.
I'm in a similar mindset... I've had on my personal todo to play with wildduck or similar for a few months, I had setup a Mailu server last year and it ran okay, though I didn't use it much. I switched to a dedicated hosted server account a few months ago, and didn't migrate my mailu setup as I wanted to do it a little differently. In the end, it feels like email is still way over priced for what it is, with a wierd contrast between too few complete options, and plenty of complex or commercial ones.
I'm not sure where the right balance is when you're one person with multiple domains and a handful of family/friends. I had relied more on Google Domains' free email forwarding, of course with them selling the domain business, who knows how that will land. I noticed I could do the same with Cloudflare and an email worker, even it a bit of a PITA, though kind of nice in that I can do an email distribution list with their email workers.
In the end, it's such a mixed bag of bad to worse options, I'm not sure how to feel, or how much effort is actually worth it in the end. And that doesn't cover spam or filtering.
I've implemented an email service using wildduck (https://get.bantumail.com). It works great, and the storage architecture is great, but I'm afraid it will cost a lot with scale, because I do not want to manage a large, business critical mongo cluster that stores attachments. I'd rather store these in object storage and have messages in postgres, so I'm considering implementing that for wildduck. But for your use case (family and friends), wd is a great choice.
Yeah... I had similar feelings about it... I'm planning on just running a single node stack via docker-compose for my use, with daily rsync for backup. Should be more than enough for the handful of users I'm likely to have.
I'd probably break attachments into object storage first, ahead of touching the rest of the MongoDB. If you do that, read replication for mdb should be relatively easy, or possibly just rely on their hosted/managed option. If you're in AWS, they have that as well. I get the feeling though, had to recover a borked cluster about a decade ago when Azure was going up/down for a few hours and the oplogs were hopelessly out of sync... Fortunately, they were just read replicas, not sharded, so was faster in the end (after trying other options) to just break and re-build the two replica nodes.
No idea how I would have handled things depending on the scenario if it was sharded. When I've used Mongo, I'll push the record update into a queue, that then gets stored in object storage. Worst case would be importing everything in to the collections from object storage. Mongo definitely isn't the first thing I reach for at all since then, and I'd rather use pg or cockroachdb myself.
Email infra is cheap; email reputation management is not.
If you're sending out a few hundred emails a month? Or even a couple thousand? Absolutely go your own way. But if you need to send out six or seven figures worth of email a month, the time cost of IP reputation management goes up very quickly.
> Email infra is cheap; email reputation management is not.
In that Reddit comment linked in that article the person says they were spending $325k on self-managed infrastructure and the same thing would cost them nearly $4 million on SES.
It's unlikely they're ignoring reputation management with their self-managed stuff, so that's included in the cost, right? What kind of value added reputation management would something like SES be providing that's worth +$3.65 million per year?
What's the real threshold where the costs makes the system unattractive to bad actors? It's definitely nowhere close to millions of dollars in my opinion. I would say the entrenched players like the status quo because it gives them astronomical margins, so they act in an anti-competitive manner that ensures we never get anything better.
Overpaying by thousands, hundreds of thousands, or even millions of dollars per year for email reputation management seems like a terrible deal to me. Maybe someone can break it down for me and explain what kind of value you're getting for the 10-100x increase in costs. Is it anything beyond a pay-to-play scheme that's the result of tacit collusion?
The value is that if your transactional mails stop working suddenly and you need a week to get it back on track you might not have a business by the time you solved the issue.
$325k in hardware also needs people with expertise to run it, those aren’t cheap.
Maintaining reputation also means keeping your IP of blacklists and continuously monitoring the major ones . Spending time debugging delivery issues all of which needs people who costs a lot of money.
Also 4 million is list price , at that scale most providers like Mailgun, sendgrid , postmark will give you pretty steep discounts and dedicated account managers and high priority support .
In any company every bit of effort spent should be on developing your product, not wasted in trying to do someone else job just as good as them then not selling that skill.
AWS exists because Amazon figured they were spending a time and effort managing large scale infra and should be monetized.
Even at Amazon.com scale it makes sense to externalize it to a shared service like AWS.
Maybe I've lucked out, but across the domains I own, as well as helping a few small startups clean up their outbound mail reputation, I don't see the hurdle in running your own email platforms. I've got about a dozen domains that send and receive email and have never had issues when they're on infra I'm running in cloud providers.
I guess it's the luck of the draw with the IP you get for hosting but the only real requirement is that the IP you're using is portable within the cloud/compute provider you're using (for long term viability between host upgrades). Beyond that I haven't found it hard to send mail reliably, even when bootstrapping a domain. Especially if you start from a place of decent email hygiene (e.g. DMARC/SPIF/DKIM/DNSSEC). I've even got a few lookalike domains that don't get caught up in filters because they've been maintained properly from the beginning.
> In that Reddit comment linked in that article the person says they were spending $325k on self-managed infrastructure and the same thing would cost them nearly $4 million on SES.
Okay, if you're spending $300k+/yr, then sure it's not surprising you've got an economy of scale there.
But how far can you get on a budget of $15k/yr? SES is competitive.
Self hosting the infrastructure is incredibly easy. There are a lot of products out there that make hosting your own email system a cinch. One I'd personally recommend would be Cloudron, but I've heard MailCow is very easy, and I'm sure there are many other tools out there that sort of just 'do it for you'.
Email deliverability on the other hand is... I'm going to say, it's not hard, but not easy either. Mainly you have to deal with blacklists, as well as certain providers being a bit weird. Staying off of blacklists can be both easy and hard. Most blacklist providers have several different layers of "Don't accept email from this server". The highest layer is easy to stay off of - just don't send spam. The layers below it might not be though. In particular, if your IP address is even in the same range as where other people might be sending spam, your going to end up on a blacklist because of it. So certain blacklists might have a 'Level 2' or 'Level 1' where if you use the same VPS provider as a spammer, your VPS that's never sent spam is going to get some of that 'bad reputation'.
Email servers deal with blacklists in different ways - some just accept most things, some are a lot more strict. Usually email server will count up indicators for spam, and score them, and then, if it's above a certain score, the email bounces. Usually having a little 'bad reputation' (like from Level 2 / Level 1 like I listed above) won't be enough 'score' to effect things, but having a lot of it certainly will. I had an email server hosted on a VPS with a Level 2 warning, and my Gmail still got my emails. But I don't know if my emails would go well to all email providers (didn't do enough testing). In addition, some email providers will silently fail your emails if they don't pass - Google is pretty notorious about doing this. So it can be a bit of a pain to debug problems.
The advantage of SES is that they deal with the reputation problem. They will jump on a Level 1 for a few days, then stay off of it for the rest of the month, then jump back on it again, and the cycle repeats. This is for the generic shared IP form of SES, so it's pretty good. It's certainly going to be more expensive, but you will need to send a good amount of email before it make sense to start managing reputation on your own probably.
If you want the best of both worlds (although this sounds like what you are already doing), I'd suggest hosting your own server, but then using Amazon SES as the outbound email relay. Amazon's outbound costs are very very cheap, and that's what you need the reputation for anyways.
It would be really nice if there was a system with fallback relays though, I agree. Let's hope that happens sometime :).
> If you want the best of both worlds (although this sounds like what you are already doing), I'd suggest hosting your own server, but then using Amazon SES as the outbound email relay. Amazon's outbound costs are very very cheap, and that's what you need the reputation for anyways.
The only thing I'm actually interested in is sending transactional email. I actually think paying a managed provider for it is the most pragmatic approach, but it seems like the existing providers are all pushing the limits of charging what the market will bear and, based purely on subjective info I've seen online, the margins aren't even close to anything resembling fair value (for me).
If there wasn't so much business and technical complexity related to bringing your own IPs, the $10k ish (?) it costs to buy a /24 starts to look like a reasonable expense when put alongside the pricing of a lot of email sending services.
If DNS doesn't work, nothing works. The hassle of having to keep it running is more work than just paying the few bucks a month for someone else to do it.
If you are an individual or smaller company, definitely. When AWS and GCP join the "don't want to do it myself" crowd though, you have to start to wonder who exactly can take on this responsibility. Email and DNS are some of the fundamental protocols of the internet. If we have reached a point where they are getting too complex to use, what is the path forward from here?
I run both of those things for a while now, and I think it's easier than many expect. And once it's set up correctly, both Email and DNS Servers do not require much maintenance and scale pretty well.
I wonder if Amazon wants out of the email reputation business. I was kind of shocked to find that GCP doesn't even offer its own SES equivalent, they direct customers to sendgrid.
This seems weird to me. It's easy to convince me to use an "always-free" tier service. It it gets big enough to cost money, I'm usually happy. If it costs $7/mo, suddenly it becomes a dumb thing that has to be thought about. Are they trying to avoid locking me in?
27 comments
[ 2.8 ms ] story [ 50.0 ms ] threadI don't like having the potential for changes I can't control. It almost feels like I need two of everything just to be safe.
I wish some of the up-and-coming auth systems (ex: logto.io) would have some kind of tiered fallback for transactional email. For example, I have self hosted mail that's tier 1 (use first), AWS SES as tier 2 (use for failover), and Postmark as tier 3 (use as a last resort).
The reason I list Postmark last is because there is a point where price becomes a factor for me. If Postmark is 10x SES for 50k messages, and SES is 12x self managed infrastructure, that's 120x costs (12,000% markup), right? I feel like that's so expensive I might have gotten it wrong, so please correct me if I did.
I'm not sure where the right balance is when you're one person with multiple domains and a handful of family/friends. I had relied more on Google Domains' free email forwarding, of course with them selling the domain business, who knows how that will land. I noticed I could do the same with Cloudflare and an email worker, even it a bit of a PITA, though kind of nice in that I can do an email distribution list with their email workers.
In the end, it's such a mixed bag of bad to worse options, I'm not sure how to feel, or how much effort is actually worth it in the end. And that doesn't cover spam or filtering.
I'd probably break attachments into object storage first, ahead of touching the rest of the MongoDB. If you do that, read replication for mdb should be relatively easy, or possibly just rely on their hosted/managed option. If you're in AWS, they have that as well. I get the feeling though, had to recover a borked cluster about a decade ago when Azure was going up/down for a few hours and the oplogs were hopelessly out of sync... Fortunately, they were just read replicas, not sharded, so was faster in the end (after trying other options) to just break and re-build the two replica nodes.
No idea how I would have handled things depending on the scenario if it was sharded. When I've used Mongo, I'll push the record update into a queue, that then gets stored in object storage. Worst case would be importing everything in to the collections from object storage. Mongo definitely isn't the first thing I reach for at all since then, and I'd rather use pg or cockroachdb myself.
If you're sending out a few hundred emails a month? Or even a couple thousand? Absolutely go your own way. But if you need to send out six or seven figures worth of email a month, the time cost of IP reputation management goes up very quickly.
In that Reddit comment linked in that article the person says they were spending $325k on self-managed infrastructure and the same thing would cost them nearly $4 million on SES.
It's unlikely they're ignoring reputation management with their self-managed stuff, so that's included in the cost, right? What kind of value added reputation management would something like SES be providing that's worth +$3.65 million per year?
What's the real threshold where the costs makes the system unattractive to bad actors? It's definitely nowhere close to millions of dollars in my opinion. I would say the entrenched players like the status quo because it gives them astronomical margins, so they act in an anti-competitive manner that ensures we never get anything better.
Overpaying by thousands, hundreds of thousands, or even millions of dollars per year for email reputation management seems like a terrible deal to me. Maybe someone can break it down for me and explain what kind of value you're getting for the 10-100x increase in costs. Is it anything beyond a pay-to-play scheme that's the result of tacit collusion?
$325k in hardware also needs people with expertise to run it, those aren’t cheap.
Maintaining reputation also means keeping your IP of blacklists and continuously monitoring the major ones . Spending time debugging delivery issues all of which needs people who costs a lot of money.
Also 4 million is list price , at that scale most providers like Mailgun, sendgrid , postmark will give you pretty steep discounts and dedicated account managers and high priority support .
In any company every bit of effort spent should be on developing your product, not wasted in trying to do someone else job just as good as them then not selling that skill.
AWS exists because Amazon figured they were spending a time and effort managing large scale infra and should be monetized.
Even at Amazon.com scale it makes sense to externalize it to a shared service like AWS.
I guess it's the luck of the draw with the IP you get for hosting but the only real requirement is that the IP you're using is portable within the cloud/compute provider you're using (for long term viability between host upgrades). Beyond that I haven't found it hard to send mail reliably, even when bootstrapping a domain. Especially if you start from a place of decent email hygiene (e.g. DMARC/SPIF/DKIM/DNSSEC). I've even got a few lookalike domains that don't get caught up in filters because they've been maintained properly from the beginning.
Okay, if you're spending $300k+/yr, then sure it's not surprising you've got an economy of scale there.
But how far can you get on a budget of $15k/yr? SES is competitive.
Email deliverability on the other hand is... I'm going to say, it's not hard, but not easy either. Mainly you have to deal with blacklists, as well as certain providers being a bit weird. Staying off of blacklists can be both easy and hard. Most blacklist providers have several different layers of "Don't accept email from this server". The highest layer is easy to stay off of - just don't send spam. The layers below it might not be though. In particular, if your IP address is even in the same range as where other people might be sending spam, your going to end up on a blacklist because of it. So certain blacklists might have a 'Level 2' or 'Level 1' where if you use the same VPS provider as a spammer, your VPS that's never sent spam is going to get some of that 'bad reputation'.
Email servers deal with blacklists in different ways - some just accept most things, some are a lot more strict. Usually email server will count up indicators for spam, and score them, and then, if it's above a certain score, the email bounces. Usually having a little 'bad reputation' (like from Level 2 / Level 1 like I listed above) won't be enough 'score' to effect things, but having a lot of it certainly will. I had an email server hosted on a VPS with a Level 2 warning, and my Gmail still got my emails. But I don't know if my emails would go well to all email providers (didn't do enough testing). In addition, some email providers will silently fail your emails if they don't pass - Google is pretty notorious about doing this. So it can be a bit of a pain to debug problems.
The advantage of SES is that they deal with the reputation problem. They will jump on a Level 1 for a few days, then stay off of it for the rest of the month, then jump back on it again, and the cycle repeats. This is for the generic shared IP form of SES, so it's pretty good. It's certainly going to be more expensive, but you will need to send a good amount of email before it make sense to start managing reputation on your own probably.
If you want the best of both worlds (although this sounds like what you are already doing), I'd suggest hosting your own server, but then using Amazon SES as the outbound email relay. Amazon's outbound costs are very very cheap, and that's what you need the reputation for anyways.
It would be really nice if there was a system with fallback relays though, I agree. Let's hope that happens sometime :).
The only thing I'm actually interested in is sending transactional email. I actually think paying a managed provider for it is the most pragmatic approach, but it seems like the existing providers are all pushing the limits of charging what the market will bear and, based purely on subjective info I've seen online, the margins aren't even close to anything resembling fair value (for me).
If there wasn't so much business and technical complexity related to bringing your own IPs, the $10k ish (?) it costs to buy a /24 starts to look like a reasonable expense when put alongside the pricing of a lot of email sending services.
Mainly DNS and email.
Challenging at the best of times.
You get better at it after the 20th try.