All that "security" just to fight spam. IIRC it was estimated that globally spammers make $300M per year from their spam. It doesn't seem like much. Somebody joked that it would be better if we just paid them that much to do nothing.
I think it is different. With Piracy people want something (product/service/media). The quote from Gabe is about the fact that they are willing to pay for it, if it is convenient and affordable.
Nobody wants spam or the things promoted in the spam. The companies want attention from the users. I think the better version is targeted ads on Facebook, etc. Maybe that’s the closest analogy to Gabe’s quote.
> Nobody wants spam or the things promoted in the spam.
Lots of people want Viagra without having to see a doctor. There definitely is a market for it, and now there are companies that do provide it, I wonder if this will cause spam to shift to other products.
That's true! I had considered those to be scams rather than "normal" spam. When I look at what my filter catches, it broadly falls into three categories: scams, viagra/dating/sex-related products, and every day products ("buy this great ladder for your garage"). Too bad you usually can't analyze how many clicks and sales they generate.
Steam and Netflix made it possible for people to play/watch media more easily than going to the pirate bay.
What are spammers trying to accomplish, that a better product would prevent them from using email (that isn't just them shitting up the other service the way they shit up email)?
The underlying problem is that a sufficiently motivated spammer can target tens or hundreds of millions of people with their spam without too much effort.
As a result, every possible scam and spam with even the slightest possibility of converting 0.00001% of recipients can now be a viable spam campaign.
The underlying problem is that it’s so easy to scale spam to a lot of targets.
Yup. Just requesting the sender to solve some riddle (and waste their energy in process) would turn the tables completely because the cost of sending would be non-zero. Unfortunately it would also mean that we would be sacrificing our planet again. But maybe the difficulty of the challenge could adapt according to some trust score?
I think the deeper problem is sending an email (or a billion emails) triggers someone else's servers to do most of the work. Email is beautifully cooperative computing, but that means it can easily be taken advantage of.
Spam is just the opposite, it is actually a pricing issue and not a service issue.
As long as the expected value per message of spam sent is positive, someone in the world will send as many messages as they are able to. You either fix this by raising their costs so that spam no longer has a positive expected value, remove their ability to send an unlimited number of messages, or both.
This is not exclusive to the e-mail system, robocalls are still a major annoyance, and the global telephone system is much more regulated. Mobile phones now automatically filter calls, even! It's wild.
> Somebody joked that it would be better if we just paid them that much to do nothing.
I know this is in jest, but in economics there's this concept called "induced demand" that comes to mind.
"Public extortion" would be an interesting challenge, as it would be difficult to solve the problem of "Hey why don't you also pay ME to do nothing too?"
This was a great read. Thanks for posting it here!
I found this line to be especially intriguing:
> Hellbanning everybody except for other big email providers is lazy and conveniently dishonest. It uses spam as a scapegoat to nerf deliverability and stifle competition.
The big tech firms criticized in this article are guilty of these sorts of transgressions in other arenas, as well. It's always been my contention that the "hellbanning" of user-generated content by big media and big tech alike comes from the same motivations. YouTube and CBS alike want to make niche content difficult to consume in order to stifle any competition that might get vaulted up as a result of that niche audience finding the new distribution endpoints. This comes with the added bonus of reducing cost of goods sold, by reducing the firehose of new content to process. Or, as the article puts it:
> Unfortunately, the computing power required to filter millions of emails per minute is huge. That's why the email industry has chosen a shortcut to reduce that cost. The shortcut is to avoid processing some email altogether. Selected email does not either get bounced nor go to spam. That would need processing, which costs money.
I would be very curious to learn if there are any proposed explanation as to why this phenomenon is so commonly spread throughout the big tech space. Do we get the same kind of behavior out of other enormous multi-national firms like oil producers, ocean freight companies, defense contractors, and chemical suppliers?
Is there actually a "big tech" email provider that accepts a message with a 2xx SMTP code and then deletes it? The only one I personally know of never does that. That one also does not use anything like an IP address blacklist. This article doesn't name names, it just waves its hands and throws around some innuendo. But as far as my own personal experience goes, this author has no idea what they are talking about.
> Is there actually a "big tech" email provider that accepts a message with a 2xx SMTP code and then deletes it?
Agreed, that is really broken behavior. Once you accept mail for delivery, it should be treated as a contract to deliver that mail. Rejects during SMTP conversation are fine as they notify the sender and do not generate backscatter.
I have heard complaints that Microsoft's hosted mail offerings accept then silently delete mail. It is somewhat believable, as MS Exchange server would respond 2xx for any message to any (including non-existent) destination address and later spam the possibly spoofed sender with bounce messages. Maybe MS has broken behavior like this in their hosted offerings too? And, rather than fix their software, they silently delete mail since they have finally learned that backscatter is a bad thing?
And, MS o365 allows 'delete' as an option for the centralized spam rules maintained by the admin. These mails are accepted 2xx then silently deleted.
MS does other questionable things on their 'free' hosted offerings to mitigate their abysmal spam filtering. I have a couple burner @outlook.com addresses, and they no longer receive any mail reliably from any sender. MS provides the user a place to whitelist senders and domains, but after wasting a bunch of time whitelisting domains, mail still is marked spam. "Junk" is effectively the inbox on those accounts.
Disclaimer: experience is dated, from a past job, running Postfix MTAs for a large organization and dealing with / mitigating MS issues, but never directly involved with any MS stuff.
You start with the premise that you can't host your own email, which is problematic. I don't know why the people who fail at self-hosting email are so adamant about telling others that nobody should self-host, but it seems more like a squeaky wheel problem than a real one.
In every scenario, deliverability problems can be solved by smarthosting through a reputable email provider. Period.
You get all the benefits of your own filtering, your own logs showing every delivery attempt, you get to store your data on your own systems, access it however you prefer, et cetera - all the reasons to self-host are there except for delivery logs, and those can be arranged through your smarthost provider.
Simple, huh? So why are so many people emphatically telling us to NOT self-host email?
Sure! I would never recommend, for instance, Dreamhost, but I set up a company's smarthosting with Dreamhost because the company was already using them. Dreamhost in general, though, is quite spammy.
Linode and Panix come to mind.
Really, though, many hosting companies won't necessarily list "smart hosting" / "smarthosting" as a service. If they offer a cheap VPS and offer outgoing SMTP, you can set up smarthosting through a VPS.
One can even smarthost through Outlook / Gmail, if you really want.
Sending via a VPS is not the same as sending through the VPS provider's email servers. Providers usually have their own outgoing SMTP servers for their clients with people who make sure that deliverability is always working, so sending through those is much more likely to work than sending directly from a VPS.
You're in control of all deliveries to you, so you can grep through your logs and see any and every delivery attempt, which you can't do with, say, Gmail.
You can store your data however you like, with as many (or as few) security considerations as you please. Want full disk and OS encryption with a Yubikey that has to be physically present when you boot the system? Sure! Want to encase it in a cube of concrete? Install your server at the top of a tree? Why not? Want to store all of your email encrypted in memory? Go for it!
You can back it up as you like, you can make sure nobody else sees or indexes it, you can access it via command line, webmail, IMAP, POP, whatever. You can more (less) your spool file directly. You can overwrite the disk where the mail was stored when you delete it. It's up to you.
Smarthosting does mean you have less control at delivery time, though. For instance, I can choose to refuse to deliver email to domains that don't negotiate TLS. Smarthosting doesn't easily allow this (or at least I don't know of any way to do this).
But every other part will still be 100% in your control.
I love the idea of self-hosting my email, but there's no way I'm going through all that work if getting my mail delivered will be a toss-up. Complete motivation-killer.
I am somewhere in the middle. Have been running my personal mail server for 15y, it mostly works but my emails do get often flagged as spam by the major providers (but not deleted). Though those very same providers are themselves a major source of the spam I receive. Do as I say, not as I do.
The nice thing if you control your domain is to be able to create unique email aliases, which is a way to cut spam to zero. A company starts spamming or leaks your email address, just delete the alias.
Part of the problem is silent deliverability issues. You can send to someone @gmail.com that you've never messaged before and it won't get rejected. It will probably end up in their Junk folder. You don't know it, they don't know it, and unless they check, which many people don't, it will be as if you never sent it.
I started hosting my own email in 2004 before finally giving up and migrating my email to Fastmail last year.
Besides the problems mentioned in this post the real problem I had was dealing with spam. The open source community around spam has really degraded over time, to the point where most solutions are extremely high maintenance and require regular tweaking. Methods that used to work, like greylisting, cause problems when dealing with GMail because google doesn't play nicely with it. The big spam blacklists have also gotten a bit less trustworthy over the years.
I’ve noticed since switching to Fastmail from gmail my spam filtering is very problematic. I always end up with legit emails in spam, and I just can change if I want even more legit in spam or more spam in inbox, which still gets through.
I think it’s both a tough problem, as well as something that only becomes easier when you have both talent and scale to have a wide perspective.
I started self hosting in 2017. (With a fresh domain. I'm the only user.) My sever closes the incoming connection if it doesn't has a rDNS or the HELO doesn't match the rDNS. Apart from that I have no anti spam measures! Friends and family have my firstname.lastname@domain address. Websites and companies get a <random>@domain address. So far I have only reserved 3 spam mails. Two times they got the address from a public mailing list I participated. One time it was from a webshop that sold or leaked my address.
Have around 100 users on my self-hosted mailserver. Works alright for the most part. Once or twice a year, there are connection issues to small companies with weird settings. I just route those over an external ESP.
Then there is also mxroute.com, which is an indie email provider. He seems to do fine too. Didn't use them yet.
So I think having at least some sending volume is key to running an indie server. You can't do it just for a few mailboxes/users.
I still wouldn't recommend to learn or start with email in 2022. There are better uses of your time.
The owner is from the lowendtalk.com hosting forums.
It was a side project for a few years and has been full time for last couple of years now.
He's very hands on. Has very strict rules about spam - both from sending and receiving perspective.
When he brings in new ip ranges, he babysits them by slowly warming them up until they have a good enough reputation to be accepted by major providers.
RBLs and other blacklists have been annoying for decades, even more so when they were maintained by annoyed individuals rather than corporations like today. You had to beg a single person to remove you off a list that an ISP somewhere used without thinking twice about it.
I relate to this. I also stopped hosting my own mail server for this exact reason.
However I do think it’s a case of damned if you do damned if you don’t. As a consumer of big tech email I become equally frustrated when spam makes it past the filter and I expect them to do more.
If it’s easy for the average person to setup a mail sever with high reputation then it’s easy for spammers to do the same. I can’t think of a great way to manage this at scale for the average person using a $5 a month Digital Ocean VPS sending < 10 emails a month.
One thing I have noticed is that there’s still a load of large organisations failing to implement basic deliverability best practices like SPF records. These organisations have themselves to blame.
The paradox here is the same one patio11 discussed in "The optimal amount of fraud is non-zero", on the front page yesterday [0]. The more non-tech people have an email address, the more we have to prevent fraudulent email, and the harder it becomes to run your own email address.
The original email users were much more savvy and needed less protecting against fraud. Now my grandma has an email, and if we're not careful she ends up on the phone with "Microsoft customer support" giving them full access to her computer. Spam filters aren't just a question of irritation anymore, people's life savings are at risk.
I actually would rephrase it as there weren’t enough users with money to make using email a worthwhile scam medium.
Email is low trust and almost any user is susceptible to being scammed, because there aren’t good trust markers in emails, and companies use it in ways that make it indistinguishable from spam.
I think both are true. The kinds of scams my grandparents (and even parents) fall for are trivially recognizable as scams to me. It takes more work to defraud someone who knows more about the way thing are supposed to work.
But yes, it's definitely still possible for anyone to fall for more sophisticated scams, and there being more money to be had is a huge part of it. Either way the effect is the same: more protection is necessary than was before.
Matrix protocol. Can be bridged with existing platforms and protocols, including Facebook Messenger, WhatsApp, and email. 60m+ publicly addressable users, not including EU military, gov, healthcare, emergency services et al.
The name evades me, but this was a thing a few years back. IIRC, it was the equivalent of what a stamp for an envelope would cost, and the recipient got a chunk, as did the service provider. Will try and find the name for you, but AFAIK, it's no longer doing business.
Alas not! Much more recent, had a very polished of-its-era SaaS style website. IIRC, you might have even been given a specific email address for it, which then forwarded to your chosen personal address.
The sweet spot for having control over your email while simultaneously minimizing unforseen headaches is to simply own your domain name and point the MX record to whatever hosting provider you want instead of self-hosting a server at home.
Same philosophy for exposing a your personal blog of html files or content like mp4 videos. The sweet spot is to focus on buying a domain name you control. Then let Amazon S3, or Cloudflare, Hezner etc, host your html or mp4 files.
I quit self-hosting email at home over 15 years ago. It's just not something I want to babysit anymore because I have other things to focus on. As long as I control the MX record on my own domain, that's really all that's necessary.
This is what I do. The downside is that a lot of email providers make it really difficult to set up 3rd party clients outside of their own clients. I've struggled to get mutt to work with a lot of email providers because they use their own auth mechanisms.
I agree with this assessment, and it's what I do. There is still the single point of failure of losing your domain due to a hostile registrar or mismanagement e.g. allowing registration to lapse.
Ideally there would be some a decentralized permanent domain registry keyed on certs (I know these exist but have not been adopted), or at least a fallback domain you could configure somehow in case you lost control of your main domain.
I have used Mailinabox on a Hetzner server for about an year. My email delivers to all the major providers. However, small providers will occasionally block my email. So I continue to use my Gmail address for now.
With small amounts of evidence, I think if my contacts on those providers email me first, and I reply to those emails, then my domain is not blocked.
there is one blocklist im aware of that simply categorically blocks all their IPs, but thankfully none of the "big tech" players use it, so its really of no consequence to me
For me, the issue wasn't that I had to fix it frequently, but that when I did it was urgent, stressful and disruptive. Eventually the VDS I was renting had a fatal HD crash and I gave up.
There is also a happy medium. Host your own MX servers but use someone else's SMTP servers. You have complete control over the incoming mail but dodge the filters by using the established business for sending mail.
> You can also even keep Gmail as your MX server! Just move messages off of it as soon as they arrive. It's just a mailbox, after all
Do you have more information about this?
I've been looking to do something similar so that I can have my mail sorted into folders without setting up the same rules on multiple clients. (Gmail's sorting doesn't seem to support some of the sorting I'm currently doing in Thunderbird)
I've been hosting and operating my own MTA MX with Postfix since 2005. I have Postfix set to use "MailDir" format storage (one file per email) and then use Procmail filters to direct emails into per-sender or per-topic specific folders when they arrive on the server - nothing needed to be done in the email client. Dovecot provides the IMAP4 client interface. Thunderbird connects via IMAP4.
Each domain has its own user home directory so for each there is a /home/${domain_name}/Maildir/ directory as the base for storing emails, and each IMAP4 folder has an associated directory. Snippet:
Adding to the happy medium is to teach your friends and family to use Thunderbird so they can easily GPG encrypt [1] their emails keeping the nosey email providers off the email body. Also teach them to use the IMAPS (TLS) endpoint for their mail provider, usually port 993. There are probably simpler how-to's with pictures, I just do not have any of them handy.
It could be we have different circles of acquaintances. I have managed to get non technical friends to GPG encrypt their emails. I also talked 2 lawyers into using this and the two lawyers are not only non technical but have nearly zero patience.
I think you’d be very surprised at how much lawyers don’t know or care about any of that. They store stuff in the cloud without ever having heard of the Third Party Doctrine (which allows the US government warrantless access to those documents).
> store stuff in the cloud without ever having heard of the Third Party Doctrine (which allows the US government warrantless access to those documents)
Or because they know the third-party doctrine doesn’t apply to attorney work product and privileged materials.
They do. Most of them use proprietary https web interfaces that usually have "secure email" or "secure messaging" in the description but they are just fancy web portals. I despise those systems. The content is not encrypted at rest and can be leaked. With OpenGPG the emails are only decrypted on the recipients end points and can be deleted by request.
Not all of them. I keep seeing lawyers sending photos of sensitive documents using their cellphone and Whatsapp (before encryption), and most of them don't even care about all those documents still residing on their cellphone, at the mercy of whoever steals it since all the security they have in place is that swipe thing that anyone with good eyes spots in 2 seconds straight.
- GPG Suite was released under an Open Source license in the past. Is that stil the case?
You GPG Suite including GPG Mail is still going to be released under an Open Source license.
- Am I still allowed to compile my own version of GPG Mail / GPG Suite removing any code regarding the trial or activation?
You absolutely are, the GPL makes sure of that. You will find the source code on this website next to the download link for GPG Suite.
We would kindly like to ask you not to use our names or icons if you plan to publish a binary for others to use. Also please do not release any complete installers as that may suggest that they are official releases.
To be pedantic, the version I linked to is a paid addin. However, you are correct - there is/are unofficial unpaid and unsupported versions available that I was unaware of, per the FAQ and the repository that you linked, which has nothing to do with GPGTools GmbH; from the repo, “This does not constitute any endorsement or promotion of our releases by the respective copyright or trademark holders.” Thanks for sharing this.
there is definitely software to help with pgp on ANY tablet, phone, or PC your friends and family have access to. The universal problem is no one cares to encrypt except a few computer nerds, spooks, and journalists.
It means you now have to worry about key management or else lose access to all your old mail, it possibly complicates logging in on new devices or via webmail, partially breaks server-side (spam) filtering, breaks server-side full-text search… so there are some definitive trade-offs there to be made.
I'm not going to remotely pretend that this would make PGP attractive or viable for the greater public. I do want to address your valid point that the general-purpose computer seems to be a dying breed, but that there are still options for mobile devices.
I've used K-9 Mail, though rarely do these days (it went poorly-maintained for a while).
There seem to be some options for iOS as well, such as iPG Mail, though I've no experience at all with it.
Perhaps you might care to share with us what types of devices they do have, seeing as my mental telepathy transgizmatron happens to be in the shop presently.
Note that I also listed an iOS option, and mentioned that there are others.
Again: PGP-enabled email apps are available on many platforms, though, again, this addresses only a small part of the larger challenge at encouraging adoption.
Not a single member of my friends or family is going to use GPG encryption, not even my mother who loves me dearly. I had a computer geek "penpal" for a while and we did monthly emails to each once a month to catch up via gpg, and even that died off lol
> Host your own MX servers but use someone else's SMTP servers.
Yes. My outgoing email goes out via Sonic's SMTP server, with the SPF records to allow it to have a source address of my own domain. Incoming email goes to my own domains and gets forwarded.
This seems to be trouble-free. The domain is on a cheap shared hosting account. I'm not running a server. I own the domain, and not though the hosting company, so I can switch to another provider if necessary. In 27 years, I've had to do that twice, because the hosting provider went out of business.
This is easy to do, and I don't have to deal with Google. I don't even get much spam. All the spammers seem to be targeting the big services now.
I just looked at my spam folder. I'm regularly being offered dental supplies, large hydraulic sheet metal bending presses, and ammonium sulfate fertilizer.
It seems that having your own domain now means you get mostly business-to-business spam. I subscribe to Machine Design, which gets me some heavy industrial marketing, but I have no idea why I get dental supply ads.
The fertilizer spams, from China, look like a scam - there's a fertilizer shortage, so that's a spam which might get replies.
>I just looked at my spam folder. I'm regularly being offered dental supplies, large hydraulic sheet metal bending presses, and ammonium sulfate fertilizer. It seems that having your own domain now means you get mostly business-to-business spam. I subscribe to Machine Design, which gets me some heavy industrial marketing, but I have no idea why I get dental supply ads. The fertilizer spams, from China, look like a scam - there's a fertilizer shortage, so that's a spam which might get replies.
I used to have a 'Shirley' email all the time from China about elevator parts. It was such a niche kind of spam that I eventually replied and requested pictures of a bunch of parts. They sent them!
> I used to have a 'Shirley' email all the time from China about elevator parts. It was such a niche kind of spam that I eventually replied and requested pictures of a bunch of parts. They sent them!
I remember in grad school, one of my classmates being upset about getting a porn spam email in Chinese and saying, “but don’t they know that I’m a girl?” I pointed out to her that I got the same emails (but had no idea what they were saying) and explaining to her that they send them to everybody.
I do this too with a different provider whose whole business is relaying for self hosters. I have had exactly 0 problems since I started a couple of years ago. Its kind of perfect.
It can very much depend on your domain. In the early 2000s I ran a domain for a firm and the domain had 'UT' in it. Well spammers thought it was related to university of Texas at some point and we went from around 10 messages a day to over quarter of a million. The immediate issue I has was poor back scatter protection so I had to configure that in a few days. That stopped 99.99% of the spam, but still caused massive issues with the mailboxes that did exist getting hundreds of messages per day even though we were blocking tens of thousands of mails per valid email addresses.
Spammers are miserable, I don't fault anyone for giving up.
SPF and DKIM no longer have any real value. Just look at SA and the values it assigns by default to mail with those headers. Couple that will the fact that spam now has valid SPF and DKIM - just makes them pointless.
DKIM and SPF, along with DMARC are more about authentication - they let you know that the message has come from where it says it does. This makes spoofing harder, not necessarily spam. Greylisting deals with most of the "spray and prey" spammers, though it does have it's own issues.
No...it isn't the only sane choice. If it works for you, great. Forunately for Internet freedom, there many of us small timers left who have no issues hosting our own email.
Third party email providers, the so-called "established businesses", get a free pass as sending SMTP servers that are accepted by almost all receiving STMP servers. Everyone just assumes everything coming from those SMTPs is legit.
Establishing this "legitimacy" and getting the "free pass" is difficult and some have suggested, the third parties may employ anticompetive tactics.
However, IMHO the receiving SMTPs is a different issue. Why do we let these third parties receive and store our email. (Why do our homes have their own mailboxes. Why not use a "P.O. Boxes" instead.) Eventually we could move away from letting third parties control the receipt of our mail. Neither "POP3" nor "webmail" was part of the original concept of email.
Today, it is easier than ever to set up overlay networks where we can assign our own IP addresses and run our own SMTP servers that can communicate directly with other SMTP servers on the overlay network. These networks are not open to the world, they may only be open to people we know. Much of our mail is between people who know each other, e.g., friends, family, colleagues. Or businesses that we contact first. We can separate different social and business networks on different overlay networks.
Anyway, the sending and receiving of mail can be separated. We do not need to let a third party control both.
Yeah, I've been doing exactly this for over 20 years. The only problem I can recall is related to the fact that the hosting provider uses a single SSL cert for the machine that hosts my domain (and many others, presumably), so of course the cert doesn't match my domain name. It's pretty easy to work around, and I only have to deal with it every few years when they do a hardware upgrade, which sometimes means moving my domain to a different machine.
I believe you're correct. I should clarify that the SSL cert problems I have are only related to my client connecting to the server to send or receive messages.
Certs for MX servers are supposed to have the MX as subject or SAN, not your email domain. It's important when the sender enforces encryption with a valid cert (e.g. MTA-STS, or config in the mail server, or many hosted solutions like Google Workspace also support enforcing this for selected or all domains).
Example:
example.com. MX aspmx.l.google.com.
Cert should have aspmx.l.google.com as subject or SAN.
"The sweet spot for having control over your email while simultaneously minimizing unforseen headaches is to simply own your domain name"
You would think. The issue no one seems to think about is that you need to make sure to pay for the domain for the duration of your life(at least). Otherwise, as soon as you lose your domain you lose ownership of your email. Any one that has control of the domain has control of your e-mail.
This dawned on me after a place I worked at reactivated the email I used for work when I worked there. It has my name but I have 0 control over it. Lucky for me I never used the email for things other than work so it's not a big deal. It is still bothersome that they can do that and I have no say so on its use.
> The issue no one seems to think about is that you need to make sure to pay for the domain for the duration of your life
But that's true for any delivery endpoint in any medium, including physical mail and phone numbers.
It's pretty easy to set up auto-pay for domain names so that all you really need to do is keep the billing info up to date. After that it all runs on autopilot.
The maximum bound for most gTLD registries is ten years (have a dig around the ICANN website if you'd like to check), not fifteen. Sure, there might be a ccTLD that allows you to register a domain for longer than that, but I'm not aware of any, and ccTLDs usually have lower maximum renewal/registration bounds than the gTLDs.
If a registrar is letting you renew a gTLD for 15 years, they're basically sitting on a third of your money until the domain's expiration date comes up.
> point the MX record to whatever hosting provider you want
You can even have a hybrid solution where incoming mail goes directly to your self-hosted server and (some) outgoing mail is relayed through a third party.
There are many benefits to running your own server. The three biggies for me are:
1. Control. A third party can change anything about the service any time they want, and if you don't like the change they made you're screwed.
2. Expectation of privacy. Because I am not contracting with a third party, the government cannot argue that I have waived my right to privacy. (As a practical matter of course this matters not at all. If the government -- or anyone with the right technical skill and access -- wants to read your email they will. But if push ever comes to shove in a court of law it could matter.)
3. Spam filtering. I think the whole industry is doing it wrong. The Right Way to filter spam is to use your outgoing mail as ground truth for what is not spam. I have a custom spam filter that I wrote based on this idea and it works like a charm. No Bayesian analysis needed. I don't even look at content at all. Just the headers are enough to achieve >99% accuracy.
Anything that comes in from an address I have never seen before is handled specially. But it's not hard to filter out the obvious spam. Just a handful of heuristics on the from and subject lines (e.g. if the sender's name contains common English words it's probably spam) takes care of >90% of the cold calls. The rest I just look through manually once a day or so.
I was planning to institute a system where my contact page included a special keyword to include in the subject line to get past the spam filter, but that has turned out not to be necessary so I haven't implemented that yet.
The only remaining case is things like confirmation emails for new accounts, but those just get lumped in with the other cold calls. They are super-easy to spot because I'm almost always expecting them, so they are always at the top of the list.
> simply own your domain name and point the MX record to whatever hosting provider you want
That's not necessarily a sure cure, depending on the hosting provider. RoadRunner (Spectrum / Charter) in the US and Shaw in Canada won't deliver emails from my domain hosted at Runbox.com (or sent directly from the runbox.com domain.) Spectrum's bounce message references an error code that translates to "Spectrum limits the number of concurrent connections from a sender, as well as the total number of connections allowed. Limits vary based on the reputation of the IP address. Reduce your number of connections and try again later."
I am ashamed to admit that I have no idea how email works. Is there a dumb down explanation of what are the moving parts and how you can achieve that sweet spot?
You send your email through a client. That client then sends (transfers / SMTPs) that email through an MTA either bundled with it or provided by your mail server.
The MTA parses the message, figures out who it needs to go to (To, CC, BCC headers), figures out what servers receives mail for those recipients, and then transfers (SMTPs) it to the server.
What OP is referring to is that the MTA essentially does a DNS lookup for the recipients domain for a record of type MX (Mail eXchanger).
If you own the domain you have complete control over where that mail goes: you own the MX record.
There is no such thing as a BCC header. That's the entire point of the BCC.
What you've described might be correct in some cases but is not universal. The mainstream way that messages are sent is your "email client" or MUA speaks the ESTMP protocol to a mail transfer or submission agent (MTA or MSA). The client directly specifies the envelope recipients, which are not, generally speaking, parsed out of the formatted message. That is why it is possible for me to send a message to myself but the message is subsequently delivered to hundreds of unnamed recipients.
That may happen, but not for the purpose you stated. MTA just uses addresses you told it to use in the SMTP dialog. It doesn't use addresses from the message. In fact the addresses in the message may be totally different.
Typically, if you sign up for an email account, you get an email address like skywal@gmail.com or skywal@yahoo.com. Alternatively, if you own/host skywal.com, you can have an email address like skywal@skywal.com served from a computer in your home.
The "sweet spot" is combining the two, where you own skywal.com, and have your email send/receive through Google or whoever. Then, if Google decides to ban you, you just register skywal.com with another company who provides that same service, and you keep your same email address.
Let's imagine I'm sending from user@zahllos.example to user@skywall.example. I'm doing it from say Thunderbird or Outlook, and you're using the same.
I need to send, and to do this I typically use an SMTP server. This is something configured, probably on zahllos.example, maybe with the domain smtp.zahllos.example. My mail client contacts this and 'logs in' with my details, then transmits the email message I want to send to this server. The server says 'right fine' and closes the connection.
At this point, the message is in a mail queue ready to go. The SMTP server then does a DNS request for the MX record of skywall.example. Let's say this is 'mail.skywall.example'. My server, smtp.zahllos.example, then connects to `mail.skywall.example', also speaking SMTP, and says "hey, I have this message to deliver".
It is at this point that mail.skywall.example can decide to do some things. It might check SPF, so it will query the SPF record of zahllos.example to find the list of servers that may send email for that domain. In this example, let's assume smtp.zahllos.example is in the list. Great.
It may then also check the mail headers for a signature, called a dkim signature. My server signed the message before sending; mail.skywall.example can query <uid>._domainkey.zahllos.example and find a public key (or not) and check that this signature matches (or not). Again, let's assume it matches.
It might also check something like TXT _dmarc.zahllos.example to see what my DMARC policies are. If I have something like v=DMARC1;p=reject;sp=reject;pct=100;ri=86400;fo=1;aspf=s;adkim=s;rua=mailto:postmaster@zahllos.example;ruf=mailto:postmaster@zahllos.example;" this tells you I'd like you to outright reject anything not matching policy, that I expect everything to match SPF and have DNS signatures, and you can send reports if you support that to postmaster@zahllos.example. Your server can then enforce these checks as it likes.
One of the first things that will happen is that my server will announce itself via an EHLO statement. An obvious check to do is to check that the sending IP actually matches smtp.zahllos.example. by querying 'reverse ptr' records.
Your receiving server will also likely hand the message over to various spam-checking tools for analysis, such as against DNS blocklists and so on. Larger providers likely have much more sophisticated infrastructure here. Ultimately, you're going to do one of a few things: 1) deliver to inbox or apply user-specified rules and deliver to a folder; 2) deliver to junk (which is typically just another folder, but treated specially by clients), 3) reject, and tell smtp.zahllos.example you don't want the email.
Once the email passes through the smtp dameon, assuming either 1 or 2, it then gets stored somehow and in some way. I'm being a little bit vague here, because 'it depends', but in the simplest scenario, the smtp daemon will write the message to an mbox or maildir-style format. More complex setups definitely exist, indeed, there can be multiple layers of servers doing analysis on separate machines, but for simplicity, mail.skywall.example is one VM that makes its decisions and the result ends up in /var/mail/user@domain/ or some such.
A key aspect of this step that makes email very nice is that if smtp.zahllos.example cannot, for some reason, reach your server now it will queue the message and try again at set intervals. You can reasonably safely turn off mail.skywall.example for a couple of hours.
Another aspect is that you can have multiple MX records where you are prepared to accept email, with priorities. So if you can't accept at one address because the server is down for maintenance, another will accept.
So, now you've technically got an email, but you don't know it. So you open thunderoutlook, and you connect to an IMAP server imap.skywall.exampl...
So in the example of the parent, he's got a domain name registered somewhere (mydomain.com) and he set, in its MX records, the gmail server. But how does gmail make the connection between that address and your gmail address then?
So normally in the cases like this, you also have to tell Google about this, and you typically do this by using one of their paid-for products like workspaces or apps for business or whatever they call it. So let's say that you decide to host skywall@skywall.example with Google. You pay them for workspaces and you likely tell them "I would like to use this domain I already have, with email". They then tell you "OK, add our servers as your MX record in your DNS, or transfer the whole domain to us and we'll do it for you". In this case we're doing the 'changing the MX records' part.
Now when a sending server asks "where should I deliver skywall.example email" by querying MX skywall.example it gets google's servers and starts an smtp conversation with them, saying "I'd like to deliver a message for skywall@skywall.example". At this point, Google knows it can accept that, so they say yes, and then continue doing whatever they do to check for spam beyond that, including queries for spf and friends.
The reason the parent suggests this is that if at any point you decide to move off Google, you can pay someone else, e.g. fastmail, for their services, and modify your MX record. 24-48 hours later, DNS around the world catches up and everyone will get fastmail as a response when they ask for your MX record. Any new email goes there instead of Google, and thus you aren't 'tied' to the provider: you just have to move all your old email over. Whereas Google cannot let you move a user@gmail.com address, because they can only change the MX records for the whole of gmail.
DNS is the source of truth here. Whatever your MX records are is where other servers will try to contact to send email. The MX record is typically just another DNS address that will be queried for AAAA/A (i.e. what is the IP), and that doesn't need to be on the same domain at all.
Here's an example of what it looks like:
delv MX ycombinator.com @9.9.9.9
; unsigned answer
ycombinator.com. 295 IN MX 20 alt2.aspmx.l.google.com.
ycombinator.com. 295 IN MX 10 aspmx.l.google.com.
ycombinator.com. 295 IN MX 20 alt1.aspmx.l.google.com.
ycombinator.com. 295 IN MX 30 aspmx4.googlemail.com.
This is me using DELV to ask "where should I send email for ycombinator.com?" and I have four responses. Column 5 tells me the priority. Lower numbers are higher priority. Unsurprisingly, this is Google. But let's see where they host their DNS, shall we?
delv NS ycombinator.com @9.9.9.9
ycombinator.com. 159148 IN NS ns-225.awsdns-28.com.
ycombinator.com. 159148 IN NS ns-1914.awsdns-47.co.uk.
ycombinator.com. 159148 IN NS ns-1411.awsdns-48.org.
ycombinator.com. 159148 IN NS ns-556.awsdns-05.net.
So AWS. So they have separate DNS to Email, and could change those MX records to host their email anywhere else, without needing to change or move ycombinator.com's DNS from AWS.
I'll cover off the SMTP outgoing as well while I'm at it. You _can_ also not run your own outgoing smtp server but use someone else's. The key here is that if you use SPF and DKIM, you should put their IPs into SPF and their keys into DKIM, as that is what the receiving server will use. So smtp.zahllos.example could be replaced by sendgrid, provided in my DNS I say so. This may work better, as sendgrid may have a better reputation than the server I chose.
delv is the successor to dig, and both are tools that come as part of BIND the DNS server for making queries and debugging DNS. Delv happens to understand DNSSEC a bit better, so if you ever need to debug that, it is handy to have.
In this case dig or delv would be fine. You can also use https://mxtoolbox.com/SuperTool.aspx and pick "MX lookup" to find the MX records. It will also resolve the IPs, both IPv4 and IPv6 (A and AAAA) for you for the returned records.
Their supertool also breaks down SPF records, for example, into their meaning and tells you if they're valid or not. I think other links were posted that do similar things but I haven't had a chance to try them yet.
I've mentioned reverse pointers in various cases, delv can do those lookups too. Here is how you do it: first you find the IP of say mail-ed1-x529.google.com, which happens to be a mailserver of Google that was the last step before my mail server when sending myself a message from gmail. For a bit of a change, this is an IPv6 only server, and it has IP 2a00:1450:4864:20::529. So let's look it up with -x:
There are some special domains of the format <reverse-ip-address>.in-addr.arpa and <reverse-ipv6-address-digits>.ip6.arpa that point (PTR) to the DNS name. dig/delv know how to reverse and query appropriately. You can use this for precisely this reason: to look up the intended host. This makes for a nice sanity check during email sending, because in SMTP I can announce myself as any host I like, but the recipient can do a reverse lookup on my IP and see if I'm telling the truth.
One of, but my no means the only, reasons you can't send email from a home connection is because you announce yourself as smtp.zahllos.example but your reverse dns resolves to something like <somegeneratedhostname>.dynamic.residential.isp.com. You can find out yours: type "what is my ip" into google, copy-paste that and do a delv -x on it and see what your "hostname" is according to the internet.
Email is complicated but the jist is there are relays and mailbox/mail-exchanger servers (and email clients). And a million different anti-spam measures that make making sure people actually get you email difficult.
When you send an email you mail client contacts you mail-exchanger (MX) and drops the mail in your outbox. Then the MX will look up the MX record for the domains in the TO field and attempt to send the email to it using SMTP.
The first thing receiving server ussally does and look up the IP of the sending server and see if it's on a spam black list. If it is it will probably just drop the connection. Then it will look up the SPF record for the domain in the FROM address and see if the sending server is allowed to send that mail.
Larger email services will have an internal 'reputation' scoring system that will use data from reported spam to figure out what IP's and domains are sending spam emails and filter them out. They'll also look at if it's a residential IP, in an IP pool for a major cloud provider etc. Each provider has a different system and it can be really difficult to get a provider to trust your IP or whitelist your IP so you can make sure that your mail actually gets to who you're trying to send it to.
Relays are pretty simple they'll take email from one place and send it to the recipient. The mail exchanger usually has a built-in relay. A lot of people will use a third party relay service so they don't have to worry about managing the reputation of the IP of their sending mail server. They'll just add an SPF record for the relays service to their domain. And then configure their mail exchanger to send all outbound mail to the relay and then the relay will do the MX lookup.
A lot of mail services will also have their MX records pointed at relays. These inbound relays will often have a lot of those anti-spam services bolted onto them and they can also be used for load balancing to make sure that the service is always able to accept mail even if it doesn't make it in the mailbox immediately.
You are totally missing the simple fact that the number of blessed email providers to choose from is slowly going down. I've seen ISPs with thousands of clients to give up and move the mailboxes to large players simply because their clients' email was ending up in the spam so often that running the support has gotten too expensive.
Indeed. I host my domains with dreamhost and it turns out that outgoing mail from their servers will get marked as spam by Google. The exact same mail sent through a gmail address (whether under gmail.com or a custom domain) will be delivered no problem (although after the sudden closing of legacy free email, I found that emails that I had been sending via gmail with my own domain that had been getting blocked by spam filters were being delivered when they came from a gmail.com address).
The problem is your outbound mail server does not have a good reputation. You also probably do not have a large enough swath of IP space to prevent bad neighbors from becoming your problem. Lots of tricks to getting your email sent reliably to most mail servers. What got me out of hosting mail was trying to send an email to a potential lead I met at a local meetup. His email was hosted at some very small and relatively unknown university, but my email was flat out rejected. Something about that moment just clicked that it's not worth my time to chase down the admins and resolve the problem. I'll just start shunting my email through O365. I still selfhost everything else I can, but I offload mail management to a provider
It's more than that. I'm on a mailing list for a social organization that I've belonged to for 20 years, and has been hosted at the same place for all of that time. I have marked hundreds of the messages not spam, and still Google sends about 80% of them to spam.
Google does not care even if you have regular correspondence with an address -- if the server isn't big enough, it's going to spam. The user's wishes or ideas about what is or isn't spam are irrelevant.
As a counterpoint, I host a couple of domains with Dreamhost and have had no such issues. They've been serving the email for my main domain for 15 years and AFAIK I've never had any of those emails go directly to spam within Gmail or elsewhere. I do periodic smoke testing by sending test emails to different place, plus I have a reasonable number of external users making use of the domains for email as well. FWIW I moved all my DNS records to Cloudflare, but that was only in last couple of years.
True. But they are not calling for a completely open system. I like this proposal from the author.
> Change blacklisting protocols so they are not permanent and use an exponential cooldown penalty. After spam is detected from an IP, it should be banned for, say, ten minutes. Then, a day. A week. A month, and so on. This discourages spammers from reusing IPs after the ban is lifted and will allow the IP pool to be cleaned over time by legitimate owners.
> There should be a recourse for legitimate servers. I'm not asking for a blank check. I don't mind doing some paperwork or paying a fee to prove I'm legit. Spammers will not do that, and if they do, they will get blacklisted anyways after sending more spam.
But Big Tech will not do that because they will gain more from eliminating the competition.
Spam has been been fought for decades, you can rest assured any obvious solution has been tried and either doesn’t have the desired effect or is impossible to implement.
similarly, any "hello pls add me to your allow-list" emails could be made auto-disappear to the "will be deleted in 30 days" folder in ~10-15 minutes, so even if you get a 100 spam messages per day you only see the last of those, you can easily pick what you are looking for, and don't worry about the rest, they'll just disappear.
(and you still have 30 days to look for messages that might be interesting/important/etc.)
...
the real missing piece is the feedback mechanism. DMARC is meh. of course large senders have implemented FBL, but they are not available for mere mortals.
signup confirm emails are not what i'm describing, because you need to establish and filter the initial offer that they send you via email itself, which is still prone to phishing.
What I'm describing is a situation where users themselves have to proactively subscribe to a connection using some sort of out-of-band mechanism. For example, if a website wanted to send you emails, they could produce some sort of "connection ticket" that you can give to your email client in order to subscribe to them.
This makes sense for service emails and is similar to how push notification services like Pushbullet work, but can't work for humans. You need to be able to give your email to someone IRL so they can send you a message. Mutual approval would be possible in the "we just met and want to exchange emails" situation, but that too breaks when you legitimately want to give anyone the chance to message you.
> but that too breaks when you legitimately want to give anyone the chance to message you.
Yup. I do want people to be able to contact me regarding my homepage. It's only a niche page on a niche subject, so only a handful of people has written in, but it was nice hearing from them and some of them did make quite a few valuable contributions.
Some minimal obfuscation seems to be enough to keep mail harvesters away, and beyond that those mails go through the same spam filter as all my other mail traffic. Putting up a contact form would definitively be more of a hassle than just a simple mailto:-link, and then I would additionally have to start worrying about how to keep the bots away from that contact form.
I know, but anything out of band won't really work, because that can be phished even more, plus as described above, there's no real need for it either (IMHO).
This is useless because users are stupid, people sending the mail are stupid, people getting the mail are stupid, UI people creating interfaces are so stupid society could be improved by putting them in a box and mailing them all to some wasteland and hoping they form their own society there or starve.
This would result in half the planet being frustrated all the time and the other half never getting their mail.
If your goal is to secretly destroy email this is the way.
That's essentially how www.hey.com works! I thought it would be tedious at first, but I don't mind it, and it's done a great job of making it so I only see what I want to see in my inbox.
From a quick read, hey.com still allows arbitrary people to message you and just initially puts them in "The Screener", which doesn't seem like quite the same thing as an absolutely "completely whitelist (opt-in to receive messages) basis".
That's fine by me because
a) I do want to give out some sort of contact info on my homepage and people being able to message me in relation to that (and putting up a contact form leads to its own spam problems), and
b) if you happen to swap contact details offline, you then have to remember that you still need to additionally whitelist that person inside of that message service, which also seems somewhat of a hassle.
Somebody who gets inundated in unwanted messages might have a different opinion on that subject, though, and might indeed prefer a strict opt-in mode, with no exceptions…
You ignore the fact that there are perverse incentives among the participants. It's possible to implement, and I'm doing it myself. If I had more time to spend on it, we could end spam. Instead I am fine as is: most of the spammers have given up.
Is there any actually money to be made in hosting email for people? I genuinely don’t know but my suspicion is that GMail, Yahoo, Outlook, et al are loss leaders for their owner companies. I suspect people at those companies would be quite happy if the protocol got unfucked enough that it small players could participate without negatively impacting the network.
> I don't mind doing some paperwork or paying a fee to prove I'm legit.
Then how about this: The big email companies all declare one day that any newly registered domain (with an MX record) needs to post a bond for good behaviour in escrow somewhere. If any of them find the domain being used to send spam, they can slash the bond (sending it to some charity or something).
This has the advantage that it doesn't affect any existing senders (so there's no one to complain about it), and it makes transparent the cartel-like power that these companies have over email. Perhaps, to democratise the process a bit, the ITU could organise a ballot (one vote per country) to elect 5 companies/non-profits who would have this bond-slashing power.
Unfortunately to implement something like this, they'd also probably have to demand that DKIM signing become mandatory (so there are cryptographic proofs of any evidence of spamming), and this sort of global consensus / money processing scheme would probably end up being built using a blockchain, whether that was a good idea or not.
I can just imagine the headline. “Ask HN: Google sent my mail bond to charity for no reason and has torpedoed my small business, and I can’t get in touch with anyone to make it right”
I can imagine headlines like that too, but the idea of electing 5 (or some other odd number of) entities is that they would be able to share among themselves the cryptographically signed evidence of the spam they detected, and then the bond slashing would require a majority vote.
So instead, the headline should be something like "Ask HN: Google, Amazon, and the Shanghai Cooperation Organisation forced me to send $100 of Ether to UNICEF and I couldn't send any new emails until I sent another $100 payment to my domain registrar. How do I take them to the World Court to force them to reimburse me?". That's not a great situation, but it's slightly better than the status quo.
You're describing one of the solutions made by Ironport, "Bonded Sender". Their solutions were sold to Return Path and Cisco later bought them out, presumably with the bonded sended solution still belonging to Return Path? [1,2]
I've never seen discussion of this in the mainstream though... so I'm not sure if it's actually being used or just shelved.
At this point, I think any proprietary they've created is game for usage. But it's very hard to get multiple large organizations to adopt this.
Spam has been a thing forever. Literally forever. Yet people have been dealing with spam far better than the big boys, and doing so for decades.
Want to talk about anti-competitive? Gmail will accept mails, provide a 250 SMTP response, then drop the email internally.
That's not right. At all. You can reject the email easily during SMTP exchange, and people have been doing that literally for 20+ years.
No valid excuse here. None. Zero.
And if your 250 OK accept then drop the message, and provide no way to notify, or discuss, or find out why, you make it impossible for a remote admin to fix the problem.
This is 100% on purpose. Yahoo, outlook/hotmail, gmail, collude to resolve issues like this, while blocking all others to resolve issues their own purposefully broken policies cause.
If you see Alphabet with a policy, or action, you can be 100% sure it is aligned to increase market dominance.
I'd imagine they do this to not tip off spammers when a message goes through. It's the same idea behind returning a 404 when trying to access a resource you don't have permission to, or not telling users if an account actually exists under an email when doing a password reset.
It's simplistic for a spammer to tell if mail is getting delivered to end point. EG, just have a gmail account, and spam that account. Simple. Done.
Breaking reliable mail delivery for everyone, is inline with "there's no excuse, ever". It's inline with "making it worse", not better.
If you 250 accept, you deliver the email. Worst case, it ends up in a spam folder. You do not drop it on the floor. Ever. No excuse, no reason is valid here.
And I certainly won't accept "But it's so hard!", considering how easy it is to handle email, including SPAM, for everyone... until Google purposefully breaks it.
Or maybe it's an overly competitive protocol? Like playing Monopoly. Even a neophyte can end sweeping the game despite not understanding any of the underlying mechanics that drive the game's outcome. Those remaining mail providers are also fighting back the insanity. They 'just' won.
A normal company could engage in these practices, but it is settled law that a monopolist may not. A monopolist may not use their monopoly in one are to supress activities in another area.
You can't put these big tech monopolists into the same bucket as normal companies. They have way too much power, and even when they do not intend to smash things, they end up smashing things.
Substitute “HTTP” with “SMTP” and you already have that today… SMTP in plain text isn’t generally used anymore, it’s always transported over TLS. Of course you can run whitelist-only, but how is that ever going to work?
SMTP is text based and we’ll defined, so I would also argue that the transport being “simpler” is nonsense.
Whitelist is easier. If I added an mail of a friend in this system, we can communicate. Everything else goes blackhole.
Nothing is easy about email! Having worked for years just to get reliable in and outboxes is definitely not trivial. Also SMTP is a system out of your control if you want anything verified and actually delivered.
Of course whitelist is easier, but how valuable would your email be if you only allowed your friends to email you? Goodbye order confirmations, subscription reminders, recruiter emails, notifications…
Sounds like a dream, really. And you could always have it be user-driven action, like "Copy this line and paste it into your email provider's whitelist", framing it as a positive win for the user since they have total control over who is allowed to communicate with them.
I do want arbitrary people to reach out to me regarding my home page, so any just-allow-whitelisted-messages proposals are useless in that regard.
And in practice some minimal address obfuscation has been enough to block any address harvesters, and setting up a contact form or something comparable (and I guess especially also subsequently keeping it spam-free!) would have been much more of a hassle.
Sure there is.. unless you mean no recourse in this environment. The natural recourse would be to force Google to divest GMail from it's core business.
No, the recourse is to force net neutrality. You must process all emails the same way and they must go through the same filters. In fact if spam processing is part of modern email it should be part of the standard and then all mail servers must be forced to conform to the standard.
I just randomly looked at 8 different emails in my inbox. All of them were from different email providers (except google which was there twice). There's hundreds or thousands of email providers you can chose from.
iphmx 1
google 2
kornet 1
linkedin 1
secureserver 1
amazonses 1
self hosted university email 1
Most emails are still going through servers owned by Microsoft or Google, so what does self-hosting email accomplish in reality? Some government entity likely has warrant-less access to most of your emails regardless. I like email as a way to communicate, but speaking pragmatically, it’s just not a secure means of communication in 2022.
what does self-hosting email accomplish in reality
Well, for one thing it prevents Google from building a profile of my online shopping behavior based on my email receipts. Also avoids them knowing every time I go on a trip based on booking confirmation emails.
Like most people, I'm not trying to hide from the government.
Using a 3rd party email host that isn't Google has the same effect with a lot less effort. I switched to FastMail for this reason, even if Google claims not to scan paid Gmail accounts for marketing purposes.
My server uses encrypted connections and my own certificate. If I send an email to a family member on my server, no one on the Internet can read its contents. If I add a recipient and FAANG or Microsoft then yes it wouldn't matter. If that is truly important, we can learn to encrypt emails locally first.
What does self-hosting accomplish? I'd say it's important in the way that having HAM radio operators is important.
I like the idea of having a node I can just plug into my network. I run my Urbit on a Mac mini with tailscale (which works great).
The core of what he writes about is correct though, email failed to be truly peer to peer (as imo all non-urbit-like federated systems will) because of the incentives that lead to centralization (spam, difficulty of running nodes, etc.)
We’re suffering the consequences of the local max we’re trapped in currently because of this. The promise of the 90s internet was a bunch of people using decentralized services they controlled - instead we're primarily thin clients connecting to a small handful of powerful ad companies. We're mostly serfs [0] allowed access if we give up our data for ad targeting, follow EULAs nobody reads, and don't say anything the company earls disagrees with.
> The sweet spot for having control over your email while simultaneously minimizing unforseen headaches is to simply own your domain name and point the MX record to whatever hosting provider you want instead of self-hosting a server at home.
So, mandatory third-party doctrine/warrantless surveillance.
Controlling your domain/mx is the most valuable thing.
Email reception has not been a problem for me so I enjoy having my mx pointing to my own mail server. It gives me more control and it requires very little maintenance.
If I was going to outsource anything the first choice would be outbound. My email system does everything right for reputation protection. All senders are authenticated on secure connections and the senders are people I trust. Nothing bad gets sent and my static IP is on a reputable server host and is not on any public black lists. I maintain SPF, DKIM etc. If someone decides to block my IP for no reason by accident or on purpose there is very little I can do or care to do anymore.
I have an alternate path for emails setup via a server I host elsewhere ready to go. If I run into a widespread delivery issue due to massive indiscriminate ip blacklisting of my provider I can enable it. That has happened once in 20 years. If delivery gets too hard I will change that policy to send all outgoing emails through a commercial smtp delivery service and let them deal with the problems.
It’s a misnomer to say you can “own” a domain name. You cannot. The better word would be “rent”. Because of that you should think about what could happen if your domain name registration lapses and someone else registers it.
I think everyone knows this who owns a domain. Probably people say this because (like I do too) they have all the control over the name. If I rent an apartment, I do have to consult the owner to make major changes.
I think people who get their email on their own domain generally consider it as their permanent home and don’t think about what will happen when the domain no longer is registered to them. Using the apartment rental analogy works up to a point. With an apartment you can get the postal service to forward your mail to a new address. With a domain name there is no way to get this to happen. Whoever holds the registration controls the MX record and can do whatever they want.
In case of death it's indeed an issue if someone isn't planning ahead.
With grace periods of the domain expiry process it's quite safe. People have plenty of time to recover it. It's also hard not to notice if your mail stops arriving.
This isn't actually that hard to fix, it's just that for whatever reason, we seem to frequently have this blindspot that we don't seem to have in other industries.
Namely that "do it yourself at home" and "massive oligopolist" aren't the only two options. It's like saying "You can only have hamburgers two ways, cook them yourself or McDonalds."
I do the third and it's been great. I let my paid webhost handle it. (hostdime if you're interested, but I'm sure others do it well also)
> like saying "You can only have hamburgers two ways, cook them yourself or McDonalds."
and your comment seems to be saying it’s OK if we lose the ability to cook hamburgers at home, because there are other (more ethical) restaurants that aren’t McDonalds. am i misunderstanding?
The metaphor is kinda good. It's okay for people to refuse your home-cooked burgers because they don't trust them to be safe. There are actual small food vendors that comply with sanitation rules and you can have them provide catering services, professionally. And that is okay.
I'll take a shot here. I'm not sure why you think Gmail or the other biggies can actually guarantee delivery any better or worse than anyone else. In a technical sense, they definitely can't.
In an anecdotal sense, I mean, I've dealt with my hostdime stuff, gmail and outlook. My stuff and gmail deliver just fine. Outlook does fail.
In a practical sense, see issue above. I think "non-delivery" via the big-boys getting things wrong on spam (and worse future outcomes, pretty easy to imagine gmail trying to "amp" or something similar email such that it cuts out little guys) is a far far greater danger or problem than an individual or smaller outfit tinkering with their own servers and occasionally getting things wrong.
Another way to look at it: In what world are you more likely to be able to cook your own burger: One where it's literally just McDonalds who can now control how people talk about burgers, or one where there's also other restaurants as well.
we mandate public health measures because it's kind of easy to check for "safe to eat" quality.
also arguably it's too ineffective considering how hard it is to open a restaurant. (because instead of harassing random taco trucks we should focus on other preventative measures, better visibility of food safety, better reporting and tracking)
sure maybe it's time for a "Let's Email" (after letsencrypt) service that handles reputation for email senders.
we already have certificate transparency logs, OCSP, CAA records, and all the fancy stuff, what's needed is something similar.
> sure maybe it's time for a "Let's Email" (after letsencrypt) service that handles reputation for email senders.
> what's needed is something similar.
There are more than a few out there. No offense but if you would have had dealt with spam, you would have known that already.
No. To beat up the metaphor, I'm saying if we support the restaurants that aren't McDonalds and that are more "mom and pop" (and get others on board) then we can beat McDonalds or at least live in harmony with them.
To colinsane's point, I don't want to be a restaurant, I want to cook my burger at home. A lot of us do. You're OK with the Hostdime restaurant. I'm OK cooking my own burger. A lot of us are and we don't want to lose that right.
Said it above; but I 100% agree with you. My point has "second-order" effects.
I'm not necessarily saying your email has to go away and you HAVE to use hostdime. I'm saying the goal of McDonalds is to dominate and obsucre everything and anything that cuts into that is useful, especially third-parties like Hostdime. For someone like me that finds it too hard, but also finds gmail unacceptable this is a good (and should be obvious) solution.
How I explain this in real life:
"You'd be stupid to run a business off of gmail. When your phone or internet or electricity goes down, you can call a human and yell at them because you pay them. What happens for google? This is why I pay someone."
For my personal mail server I ended up using the free tier of SMTP2GO as outbound smtp relay. You can just register your domain as a whole, and the setup was pretty painless. They let you send up to 1000 mails/month for free, which is plenty for my own personal use, but if you need any more than that, these relay services can get quite expensive. You usually end up paying the same price as the fully hosted solution that most of those providers offer alongside the relay service.
Sending emails with dkim + sfp and I never had big problems with reachability and a postfix server on Digital Ocean.
I haven't done it since last year though.
Has something gone terribly wrong?
I remember debugging issues with email sent via aws ses to Hotmail addresses at $dailyJob but I can't think of a single Microsoft product that works well (windows, teams, azure, now even GitHub is starting to work every other day) so it doesn't surprise me.
Proper, artisanal self-hosting of email can still be viable depending on your expectations and tolerance levels for random issues.
These days, I operate with the medium-temperature bowl of porridge: AWS WorkMail with custom domains & users. My use case is basically "Replace gmail for personal email". I don't have a lot of patience for running an actual email server, so this is about as custom as I can get.
Running a custom email domain can have other practical implications, such as having to carefully re-iterate spelling when mentioning your email address over the phone to a customer support agent. With a gmail or hotmail account, virtually everyone can type that hostname in without thinking about it. This concern is moderated by being able to select a username with fewer than 5 characters, rather than your full legal name appended with your date of birth.
A person at a company mistakenly created an email list segment (or lack thereof) resulting in an email to the entire email list of hundred of thousands of emails. This combined with inexistent (we were a naive startup without an email specialist role) list hygiene practices meant we were blacklisted by Gmail after some time.
Took a year to get a hold of someone on Gmail's spam team. We found out were on 4+ Gmail blocklists, some of which were ML-based. We couldn't do anything to remove ourselves after we fixed the issues. A $1-2 million revenue channel dried up because we couldn't get out of the Gmail blackhole (short of rebranding completely, rewriting content, and using a different ESP). Fun times.
> ... [They use] spam as a scapegoat to nerf deliverability and stifle competition.
Disagree. It was a way too open protocol to begin with. From a time of innocence best suited for places with inherit trust like inside a business. And it's not just spam. Phishing is also a huge issue.
As much as I want to sympathise, Email for the big WWW is unsalvageable IMO. Too many bad actors are out there.
> [Solution:...] * There should be a recourse for legitimate servers
This is the same Big tech story. They want to cut cost, you want a human touch. You can see similar stories here in HN every week. Which is why I think it will never happen.
> This is the same Big tech story. They want to cut cost, you want a human touch. You can see a similar story here in HN every week.
Dang, sounds like our monopolized tech dystopia is probably inevitable. If only there was some state level mechanism to help balance the interests of industries maximizing profits and those of users who need a robust system which doesn't harm them.
There is a really great bit from Robert Reich (labor admin from the Clinton years) about how if you're business would collapse if you need to pay people more than minimum wage, you shouldn't be in business.
If your business suddenly fails as soon as you need to... support your customers, then you probably shouldn't be in business and only exist as a fluke in the current economic model.
How do you know? SMTP wasn't designed for reliable delivery. You don't know if your emails are being received, and even i they are, you don't know if they'll be received tomorrow.
Statistically. I communicate with a lot of people via email, and if I don't get an expected response to something of consequence I follow up - it's far more common the recipient simply missed it/didn't get around to it than it is to get "I never got it/I found it in my spam folder".
> Blacklists should not include whole IP blocks. I am not responsible for what my IP neighbor is doing with their server.
This is obviously laughably naive and creates infinite sources of spam.
Before doing a proposal on a core Internet technology you should be required to be on the other side for a while. Do anti-spam at a large retail e-mail service provider for a year and then you can understand the problem space.
You might not be responsible for what your neighbor is doing with their server, but the ESP is responsible for filtering it. The idea that they need to treat each and every comcast IP with equal weight is nuts. IP reputation is the single most valuable tool in the industry; the largest statistical predictor of whether or not an email is abusive.
754 comments
[ 2.8 ms ] story [ 350 ms ] threadGabe Newell once said that "Piracy is not a pricing issue. It's a service issue". I believe this has been proven by netflix/spotify as well.
Is spam just a symptom of a much deeper problem? If so what is it? Or is it naive to think of spam this way?
Nobody wants spam or the things promoted in the spam. The companies want attention from the users. I think the better version is targeted ads on Facebook, etc. Maybe that’s the closest analogy to Gabe’s quote.
Lots of people want Viagra without having to see a doctor. There definitely is a market for it, and now there are companies that do provide it, I wonder if this will cause spam to shift to other products.
Still applies to games/music and probably audio books.
What are spammers trying to accomplish, that a better product would prevent them from using email (that isn't just them shitting up the other service the way they shit up email)?
The underlying problem is that a sufficiently motivated spammer can target tens or hundreds of millions of people with their spam without too much effort.
As a result, every possible scam and spam with even the slightest possibility of converting 0.00001% of recipients can now be a viable spam campaign.
The underlying problem is that it’s so easy to scale spam to a lot of targets.
As long as the expected value per message of spam sent is positive, someone in the world will send as many messages as they are able to. You either fix this by raising their costs so that spam no longer has a positive expected value, remove their ability to send an unlimited number of messages, or both.
This is not exclusive to the e-mail system, robocalls are still a major annoyance, and the global telephone system is much more regulated. Mobile phones now automatically filter calls, even! It's wild.
I know this is in jest, but in economics there's this concept called "induced demand" that comes to mind.
"Public extortion" would be an interesting challenge, as it would be difficult to solve the problem of "Hey why don't you also pay ME to do nothing too?"
I found this line to be especially intriguing:
> Hellbanning everybody except for other big email providers is lazy and conveniently dishonest. It uses spam as a scapegoat to nerf deliverability and stifle competition.
The big tech firms criticized in this article are guilty of these sorts of transgressions in other arenas, as well. It's always been my contention that the "hellbanning" of user-generated content by big media and big tech alike comes from the same motivations. YouTube and CBS alike want to make niche content difficult to consume in order to stifle any competition that might get vaulted up as a result of that niche audience finding the new distribution endpoints. This comes with the added bonus of reducing cost of goods sold, by reducing the firehose of new content to process. Or, as the article puts it:
> Unfortunately, the computing power required to filter millions of emails per minute is huge. That's why the email industry has chosen a shortcut to reduce that cost. The shortcut is to avoid processing some email altogether. Selected email does not either get bounced nor go to spam. That would need processing, which costs money.
I would be very curious to learn if there are any proposed explanation as to why this phenomenon is so commonly spread throughout the big tech space. Do we get the same kind of behavior out of other enormous multi-national firms like oil producers, ocean freight companies, defense contractors, and chemical suppliers?
> But as far as my own personal experience goes, this author has no idea what they are talking about.
He's not very informed about being on the receiving end.
Agreed, that is really broken behavior. Once you accept mail for delivery, it should be treated as a contract to deliver that mail. Rejects during SMTP conversation are fine as they notify the sender and do not generate backscatter.
I have heard complaints that Microsoft's hosted mail offerings accept then silently delete mail. It is somewhat believable, as MS Exchange server would respond 2xx for any message to any (including non-existent) destination address and later spam the possibly spoofed sender with bounce messages. Maybe MS has broken behavior like this in their hosted offerings too? And, rather than fix their software, they silently delete mail since they have finally learned that backscatter is a bad thing?
And, MS o365 allows 'delete' as an option for the centralized spam rules maintained by the admin. These mails are accepted 2xx then silently deleted.
MS does other questionable things on their 'free' hosted offerings to mitigate their abysmal spam filtering. I have a couple burner @outlook.com addresses, and they no longer receive any mail reliably from any sender. MS provides the user a place to whitelist senders and domains, but after wasting a bunch of time whitelisting domains, mail still is marked spam. "Junk" is effectively the inbox on those accounts.
Disclaimer: experience is dated, from a past job, running Postfix MTAs for a large organization and dealing with / mitigating MS issues, but never directly involved with any MS stuff.
In every scenario, deliverability problems can be solved by smarthosting through a reputable email provider. Period.
You get all the benefits of your own filtering, your own logs showing every delivery attempt, you get to store your data on your own systems, access it however you prefer, et cetera - all the reasons to self-host are there except for delivery logs, and those can be arranged through your smarthost provider.
Simple, huh? So why are so many people emphatically telling us to NOT self-host email?
Could you provide an example of such a smarthost email provider that has the benefits you mentioned?
Not challenging you, just genuinely curious.
Linode and Panix come to mind.
Really, though, many hosting companies won't necessarily list "smart hosting" / "smarthosting" as a service. If they offer a cheap VPS and offer outgoing SMTP, you can set up smarthosting through a VPS.
One can even smarthost through Outlook / Gmail, if you really want.
You're in control of all deliveries to you, so you can grep through your logs and see any and every delivery attempt, which you can't do with, say, Gmail.
You can store your data however you like, with as many (or as few) security considerations as you please. Want full disk and OS encryption with a Yubikey that has to be physically present when you boot the system? Sure! Want to encase it in a cube of concrete? Install your server at the top of a tree? Why not? Want to store all of your email encrypted in memory? Go for it!
You can back it up as you like, you can make sure nobody else sees or indexes it, you can access it via command line, webmail, IMAP, POP, whatever. You can more (less) your spool file directly. You can overwrite the disk where the mail was stored when you delete it. It's up to you.
Smarthosting does mean you have less control at delivery time, though. For instance, I can choose to refuse to deliver email to domains that don't negotiate TLS. Smarthosting doesn't easily allow this (or at least I don't know of any way to do this).
But every other part will still be 100% in your control.
Have not had any spam or blacklisting issues and it was super easy to setup.
The nice thing if you control your domain is to be able to create unique email aliases, which is a way to cut spam to zero. A company starts spamming or leaks your email address, just delete the alias.
That happens even if you send from @gmail to @gmail. Well, it happened to me not long ago.
But I've mostly been using lesser known email providers and I haven't noticed any problems.
Besides the problems mentioned in this post the real problem I had was dealing with spam. The open source community around spam has really degraded over time, to the point where most solutions are extremely high maintenance and require regular tweaking. Methods that used to work, like greylisting, cause problems when dealing with GMail because google doesn't play nicely with it. The big spam blacklists have also gotten a bit less trustworthy over the years.
I think it’s both a tough problem, as well as something that only becomes easier when you have both talent and scale to have a wide perspective.
Then there is also mxroute.com, which is an indie email provider. He seems to do fine too. Didn't use them yet.
So I think having at least some sending volume is key to running an indie server. You can't do it just for a few mailboxes/users.
I still wouldn't recommend to learn or start with email in 2022. There are better uses of your time.
It was a side project for a few years and has been full time for last couple of years now.
He's very hands on. Has very strict rules about spam - both from sending and receiving perspective.
When he brings in new ip ranges, he babysits them by slowly warming them up until they have a good enough reputation to be accepted by major providers.
However I do think it’s a case of damned if you do damned if you don’t. As a consumer of big tech email I become equally frustrated when spam makes it past the filter and I expect them to do more.
If it’s easy for the average person to setup a mail sever with high reputation then it’s easy for spammers to do the same. I can’t think of a great way to manage this at scale for the average person using a $5 a month Digital Ocean VPS sending < 10 emails a month.
One thing I have noticed is that there’s still a load of large organisations failing to implement basic deliverability best practices like SPF records. These organisations have themselves to blame.
The original email users were much more savvy and needed less protecting against fraud. Now my grandma has an email, and if we're not careful she ends up on the phone with "Microsoft customer support" giving them full access to her computer. Spam filters aren't just a question of irritation anymore, people's life savings are at risk.
[0] https://news.ycombinator.com/item?id=32701913
Email is low trust and almost any user is susceptible to being scammed, because there aren’t good trust markers in emails, and companies use it in ways that make it indistinguishable from spam.
But yes, it's definitely still possible for anyone to fall for more sophisticated scams, and there being more money to be had is a huge part of it. Either way the effect is the same: more protection is necessary than was before.
http://www.hashcash.org/
> At some point your IP range is bound to be banned, either by one asshole IP neighbor sending spam, *one of your users being pwned*,
feels like a hint that
> My current email server IP has been managed by me and used exclusively for personal email with zero spam, zero, for the last ten years.
Might not be entirely accurate.
Same philosophy for exposing a your personal blog of html files or content like mp4 videos. The sweet spot is to focus on buying a domain name you control. Then let Amazon S3, or Cloudflare, Hezner etc, host your html or mp4 files.
I quit self-hosting email at home over 15 years ago. It's just not something I want to babysit anymore because I have other things to focus on. As long as I control the MX record on my own domain, that's really all that's necessary.
Ideally there would be some a decentralized permanent domain registry keyed on certs (I know these exist but have not been adopted), or at least a fallback domain you could configure somehow in case you lost control of your main domain.
Dont know about you, but I have setup my mailserver years ago, and outside of regular OS updates, havent had to touch it.
With small amounts of evidence, I think if my contacts on those providers email me first, and I reply to those emails, then my domain is not blocked.
there is one blocklist im aware of that simply categorically blocks all their IPs, but thankfully none of the "big tech" players use it, so its really of no consequence to me
You can also even keep Gmail as your MX server! Just move messages off of it as soon as they arrive. It's just a mailbox, after all
Do you have more information about this?
I've been looking to do something similar so that I can have my mail sorted into folders without setting up the same rules on multiple clients. (Gmail's sorting doesn't seem to support some of the sorting I'm currently doing in Thunderbird)
Each domain has its own user home directory so for each there is a /home/${domain_name}/Maildir/ directory as the base for storing emails, and each IMAP4 folder has an associated directory. Snippet:
Here's an extended snippet example from $HOME/.procmailrc that directs deliveries into the correct directory (IMAP4 folder):[1] - https://support.mozilla.org/en-US/kb/openpgp-thunderbird-how...
Or because they know the third-party doctrine doesn’t apply to attorney work product and privileged materials.
https://moxie.org/2015/02/24/gpg-and-me.html
https://blog.cryptographyengineering.com/2014/08/13/whats-ma...
* https://articles.59.ca/doku.php?id=pgpfan:wtmwp
I use https://github.com/Free-GPGMail/Free-GPGMail which is a plugin for GNUPG, without the "support" plan.
GPG Mail is a paid for add-on, but it works and it works well.
Here is the same thing, without the paid support. https://github.com/Free-GPGMail/Free-GPGMail
- GPG Suite was released under an Open Source license in the past. Is that stil the case?
You GPG Suite including GPG Mail is still going to be released under an Open Source license.
- Am I still allowed to compile my own version of GPG Mail / GPG Suite removing any code regarding the trial or activation?
You absolutely are, the GPL makes sure of that. You will find the source code on this website next to the download link for GPG Suite.
We would kindly like to ask you not to use our names or icons if you plan to publish a binary for others to use. Also please do not release any complete installers as that may suggest that they are official releases.
https://gpgtools.org/faq
<https://k9mail.app/>
I'm not going to remotely pretend that this would make PGP attractive or viable for the greater public. I do want to address your valid point that the general-purpose computer seems to be a dying breed, but that there are still options for mobile devices.
I've used K-9 Mail, though rarely do these days (it went poorly-maintained for a while).
There seem to be some options for iOS as well, such as iPG Mail, though I've no experience at all with it.
<https://www.openpgp.org/software/ipgmail/>
Note that I also listed an iOS option, and mentioned that there are others.
Again: PGP-enabled email apps are available on many platforms, though, again, this addresses only a small part of the larger challenge at encouraging adoption.
https://support.apple.com/en-gb/guide/mail/mlhlp1180/mac
Yes. My outgoing email goes out via Sonic's SMTP server, with the SPF records to allow it to have a source address of my own domain. Incoming email goes to my own domains and gets forwarded.
This seems to be trouble-free. The domain is on a cheap shared hosting account. I'm not running a server. I own the domain, and not though the hosting company, so I can switch to another provider if necessary. In 27 years, I've had to do that twice, because the hosting provider went out of business.
This is easy to do, and I don't have to deal with Google. I don't even get much spam. All the spammers seem to be targeting the big services now.
I just looked at my spam folder. I'm regularly being offered dental supplies, large hydraulic sheet metal bending presses, and ammonium sulfate fertilizer. It seems that having your own domain now means you get mostly business-to-business spam. I subscribe to Machine Design, which gets me some heavy industrial marketing, but I have no idea why I get dental supply ads. The fertilizer spams, from China, look like a scam - there's a fertilizer shortage, so that's a spam which might get replies.
I used to have a 'Shirley' email all the time from China about elevator parts. It was such a niche kind of spam that I eventually replied and requested pictures of a bunch of parts. They sent them!
https://web.archive.org/web/20030412191437/http://www.penny-...
Spammers are miserable, I don't fault anyone for giving up.
Using an SMTP relay was a suitable trade-off, for me, but I respect those who are able to endure the pains of sending non-flagged emails without it.
Third party email providers, the so-called "established businesses", get a free pass as sending SMTP servers that are accepted by almost all receiving STMP servers. Everyone just assumes everything coming from those SMTPs is legit. Establishing this "legitimacy" and getting the "free pass" is difficult and some have suggested, the third parties may employ anticompetive tactics.
However, IMHO the receiving SMTPs is a different issue. Why do we let these third parties receive and store our email. (Why do our homes have their own mailboxes. Why not use a "P.O. Boxes" instead.) Eventually we could move away from letting third parties control the receipt of our mail. Neither "POP3" nor "webmail" was part of the original concept of email.
Today, it is easier than ever to set up overlay networks where we can assign our own IP addresses and run our own SMTP servers that can communicate directly with other SMTP servers on the overlay network. These networks are not open to the world, they may only be open to people we know. Much of our mail is between people who know each other, e.g., friends, family, colleagues. Or businesses that we contact first. We can separate different social and business networks on different overlay networks.
Anyway, the sending and receiving of mail can be separated. We do not need to let a third party control both.
Example:
example.com. MX aspmx.l.google.com.
Cert should have aspmx.l.google.com as subject or SAN.
You would think. The issue no one seems to think about is that you need to make sure to pay for the domain for the duration of your life(at least). Otherwise, as soon as you lose your domain you lose ownership of your email. Any one that has control of the domain has control of your e-mail.
This dawned on me after a place I worked at reactivated the email I used for work when I worked there. It has my name but I have 0 control over it. Lucky for me I never used the email for things other than work so it's not a big deal. It is still bothersome that they can do that and I have no say so on its use.
But that's true for any delivery endpoint in any medium, including physical mail and phone numbers.
It's pretty easy to set up auto-pay for domain names so that all you really need to do is keep the billing info up to date. After that it all runs on autopilot.
Is there a registrar that will allow you to do this? Most of the ones I’ve looked at have an upper limit of around 15 years or so
If a registrar is letting you renew a gTLD for 15 years, they're basically sitting on a third of your money until the domain's expiration date comes up.
You can even have a hybrid solution where incoming mail goes directly to your self-hosted server and (some) outgoing mail is relayed through a third party.
1. Control. A third party can change anything about the service any time they want, and if you don't like the change they made you're screwed.
2. Expectation of privacy. Because I am not contracting with a third party, the government cannot argue that I have waived my right to privacy. (As a practical matter of course this matters not at all. If the government -- or anyone with the right technical skill and access -- wants to read your email they will. But if push ever comes to shove in a court of law it could matter.)
3. Spam filtering. I think the whole industry is doing it wrong. The Right Way to filter spam is to use your outgoing mail as ground truth for what is not spam. I have a custom spam filter that I wrote based on this idea and it works like a charm. No Bayesian analysis needed. I don't even look at content at all. Just the headers are enough to achieve >99% accuracy.
Could you elaborate? Does this mean that email from people/domains you haven't corresponded with before is spam?
I was planning to institute a system where my contact page included a special keyword to include in the subject line to get past the spam filter, but that has turned out not to be necessary so I haven't implemented that yet.
The only remaining case is things like confirmation emails for new accounts, but those just get lumped in with the other cold calls. They are super-easy to spot because I'm almost always expecting them, so they are always at the top of the list.
That's not necessarily a sure cure, depending on the hosting provider. RoadRunner (Spectrum / Charter) in the US and Shaw in Canada won't deliver emails from my domain hosted at Runbox.com (or sent directly from the runbox.com domain.) Spectrum's bounce message references an error code that translates to "Spectrum limits the number of concurrent connections from a sender, as well as the total number of connections allowed. Limits vary based on the reputation of the IP address. Reduce your number of connections and try again later."
The MTA parses the message, figures out who it needs to go to (To, CC, BCC headers), figures out what servers receives mail for those recipients, and then transfers (SMTPs) it to the server.
What OP is referring to is that the MTA essentially does a DNS lookup for the recipients domain for a record of type MX (Mail eXchanger).
If you own the domain you have complete control over where that mail goes: you own the MX record.
What you've described might be correct in some cases but is not universal. The mainstream way that messages are sent is your "email client" or MUA speaks the ESTMP protocol to a mail transfer or submission agent (MTA or MSA). The client directly specifies the envelope recipients, which are not, generally speaking, parsed out of the formatted message. That is why it is possible for me to send a message to myself but the message is subsequently delivered to hundreds of unnamed recipients.
That may happen, but not for the purpose you stated. MTA just uses addresses you told it to use in the SMTP dialog. It doesn't use addresses from the message. In fact the addresses in the message may be totally different.
The "sweet spot" is combining the two, where you own skywal.com, and have your email send/receive through Google or whoever. Then, if Google decides to ban you, you just register skywal.com with another company who provides that same service, and you keep your same email address.
That's the broad strokes anyway.
Let's imagine I'm sending from user@zahllos.example to user@skywall.example. I'm doing it from say Thunderbird or Outlook, and you're using the same.
I need to send, and to do this I typically use an SMTP server. This is something configured, probably on zahllos.example, maybe with the domain smtp.zahllos.example. My mail client contacts this and 'logs in' with my details, then transmits the email message I want to send to this server. The server says 'right fine' and closes the connection.
At this point, the message is in a mail queue ready to go. The SMTP server then does a DNS request for the MX record of skywall.example. Let's say this is 'mail.skywall.example'. My server, smtp.zahllos.example, then connects to `mail.skywall.example', also speaking SMTP, and says "hey, I have this message to deliver".
It is at this point that mail.skywall.example can decide to do some things. It might check SPF, so it will query the SPF record of zahllos.example to find the list of servers that may send email for that domain. In this example, let's assume smtp.zahllos.example is in the list. Great.
It may then also check the mail headers for a signature, called a dkim signature. My server signed the message before sending; mail.skywall.example can query <uid>._domainkey.zahllos.example and find a public key (or not) and check that this signature matches (or not). Again, let's assume it matches.
It might also check something like TXT _dmarc.zahllos.example to see what my DMARC policies are. If I have something like v=DMARC1;p=reject;sp=reject;pct=100;ri=86400;fo=1;aspf=s;adkim=s;rua=mailto:postmaster@zahllos.example;ruf=mailto:postmaster@zahllos.example;" this tells you I'd like you to outright reject anything not matching policy, that I expect everything to match SPF and have DNS signatures, and you can send reports if you support that to postmaster@zahllos.example. Your server can then enforce these checks as it likes.
One of the first things that will happen is that my server will announce itself via an EHLO statement. An obvious check to do is to check that the sending IP actually matches smtp.zahllos.example. by querying 'reverse ptr' records.
Your receiving server will also likely hand the message over to various spam-checking tools for analysis, such as against DNS blocklists and so on. Larger providers likely have much more sophisticated infrastructure here. Ultimately, you're going to do one of a few things: 1) deliver to inbox or apply user-specified rules and deliver to a folder; 2) deliver to junk (which is typically just another folder, but treated specially by clients), 3) reject, and tell smtp.zahllos.example you don't want the email.
Once the email passes through the smtp dameon, assuming either 1 or 2, it then gets stored somehow and in some way. I'm being a little bit vague here, because 'it depends', but in the simplest scenario, the smtp daemon will write the message to an mbox or maildir-style format. More complex setups definitely exist, indeed, there can be multiple layers of servers doing analysis on separate machines, but for simplicity, mail.skywall.example is one VM that makes its decisions and the result ends up in /var/mail/user@domain/ or some such.
A key aspect of this step that makes email very nice is that if smtp.zahllos.example cannot, for some reason, reach your server now it will queue the message and try again at set intervals. You can reasonably safely turn off mail.skywall.example for a couple of hours.
Another aspect is that you can have multiple MX records where you are prepared to accept email, with priorities. So if you can't accept at one address because the server is down for maintenance, another will accept.
So, now you've technically got an email, but you don't know it. So you open thunderoutlook, and you connect to an IMAP server imap.skywall.exampl...
Now when a sending server asks "where should I deliver skywall.example email" by querying MX skywall.example it gets google's servers and starts an smtp conversation with them, saying "I'd like to deliver a message for skywall@skywall.example". At this point, Google knows it can accept that, so they say yes, and then continue doing whatever they do to check for spam beyond that, including queries for spf and friends.
The reason the parent suggests this is that if at any point you decide to move off Google, you can pay someone else, e.g. fastmail, for their services, and modify your MX record. 24-48 hours later, DNS around the world catches up and everyone will get fastmail as a response when they ask for your MX record. Any new email goes there instead of Google, and thus you aren't 'tied' to the provider: you just have to move all your old email over. Whereas Google cannot let you move a user@gmail.com address, because they can only change the MX records for the whole of gmail.
DNS is the source of truth here. Whatever your MX records are is where other servers will try to contact to send email. The MX record is typically just another DNS address that will be queried for AAAA/A (i.e. what is the IP), and that doesn't need to be on the same domain at all.
Here's an example of what it looks like:
This is me using DELV to ask "where should I send email for ycombinator.com?" and I have four responses. Column 5 tells me the priority. Lower numbers are higher priority. Unsurprisingly, this is Google. But let's see where they host their DNS, shall we? So AWS. So they have separate DNS to Email, and could change those MX records to host their email anywhere else, without needing to change or move ycombinator.com's DNS from AWS.I'll cover off the SMTP outgoing as well while I'm at it. You _can_ also not run your own outgoing smtp server but use someone else's. The key here is that if you use SPF and DKIM, you should put their IPs into SPF and their keys into DKIM, as that is what the receiving server will use. So smtp.zahllos.example could be replaced by sendgrid, provided in my DNS I say so. This may work better, as sendgrid may have a better reputation than the server I chose.
Turns out that I have to treat Apple-included binaries of OSS utilities with suspicion and instead use the version maintained by the Homebrew guys.
1: If anyone else is interested, I wrote a short article about it https://ayewo.com/how-to-get-delv-working-on-macos/
In this case dig or delv would be fine. You can also use https://mxtoolbox.com/SuperTool.aspx and pick "MX lookup" to find the MX records. It will also resolve the IPs, both IPv4 and IPv6 (A and AAAA) for you for the returned records.
Their supertool also breaks down SPF records, for example, into their meaning and tells you if they're valid or not. I think other links were posted that do similar things but I haven't had a chance to try them yet.
I've mentioned reverse pointers in various cases, delv can do those lookups too. Here is how you do it: first you find the IP of say mail-ed1-x529.google.com, which happens to be a mailserver of Google that was the last step before my mail server when sending myself a message from gmail. For a bit of a change, this is an IPv6 only server, and it has IP 2a00:1450:4864:20::529. So let's look it up with -x:
There are some special domains of the format <reverse-ip-address>.in-addr.arpa and <reverse-ipv6-address-digits>.ip6.arpa that point (PTR) to the DNS name. dig/delv know how to reverse and query appropriately. You can use this for precisely this reason: to look up the intended host. This makes for a nice sanity check during email sending, because in SMTP I can announce myself as any host I like, but the recipient can do a reverse lookup on my IP and see if I'm telling the truth.One of, but my no means the only, reasons you can't send email from a home connection is because you announce yourself as smtp.zahllos.example but your reverse dns resolves to something like <somegeneratedhostname>.dynamic.residential.isp.com. You can find out yours: type "what is my ip" into google, copy-paste that and do a delv -x on it and see what your "hostname" is according to the internet.
When you send an email you mail client contacts you mail-exchanger (MX) and drops the mail in your outbox. Then the MX will look up the MX record for the domains in the TO field and attempt to send the email to it using SMTP.
The first thing receiving server ussally does and look up the IP of the sending server and see if it's on a spam black list. If it is it will probably just drop the connection. Then it will look up the SPF record for the domain in the FROM address and see if the sending server is allowed to send that mail.
Larger email services will have an internal 'reputation' scoring system that will use data from reported spam to figure out what IP's and domains are sending spam emails and filter them out. They'll also look at if it's a residential IP, in an IP pool for a major cloud provider etc. Each provider has a different system and it can be really difficult to get a provider to trust your IP or whitelist your IP so you can make sure that your mail actually gets to who you're trying to send it to.
Relays are pretty simple they'll take email from one place and send it to the recipient. The mail exchanger usually has a built-in relay. A lot of people will use a third party relay service so they don't have to worry about managing the reputation of the IP of their sending mail server. They'll just add an SPF record for the relays service to their domain. And then configure their mail exchanger to send all outbound mail to the relay and then the relay will do the MX lookup.
A lot of mail services will also have their MX records pointed at relays. These inbound relays will often have a lot of those anti-spam services bolted onto them and they can also be used for load balancing to make sure that the service is always able to accept mail even if it doesn't make it in the mailbox immediately.
It's definitely an anticompetitive practice.
Google does not care even if you have regular correspondence with an address -- if the server isn't big enough, it's going to spam. The user's wishes or ideas about what is or isn't spam are irrelevant.
> Change blacklisting protocols so they are not permanent and use an exponential cooldown penalty. After spam is detected from an IP, it should be banned for, say, ten minutes. Then, a day. A week. A month, and so on. This discourages spammers from reusing IPs after the ban is lifted and will allow the IP pool to be cleaned over time by legitimate owners.
> There should be a recourse for legitimate servers. I'm not asking for a blank check. I don't mind doing some paperwork or paying a fee to prove I'm legit. Spammers will not do that, and if they do, they will get blacklisted anyways after sending more spam.
But Big Tech will not do that because they will gain more from eliminating the competition.
similarly, any "hello pls add me to your allow-list" emails could be made auto-disappear to the "will be deleted in 30 days" folder in ~10-15 minutes, so even if you get a 100 spam messages per day you only see the last of those, you can easily pick what you are looking for, and don't worry about the rest, they'll just disappear.
(and you still have 30 days to look for messages that might be interesting/important/etc.)
...
the real missing piece is the feedback mechanism. DMARC is meh. of course large senders have implemented FBL, but they are not available for mere mortals.
https://en.wikipedia.org/wiki/Feedback_loop_(email)
What I'm describing is a situation where users themselves have to proactively subscribe to a connection using some sort of out-of-band mechanism. For example, if a website wanted to send you emails, they could produce some sort of "connection ticket" that you can give to your email client in order to subscribe to them.
Yup. I do want people to be able to contact me regarding my homepage. It's only a niche page on a niche subject, so only a handful of people has written in, but it was nice hearing from them and some of them did make quite a few valuable contributions.
Some minimal obfuscation seems to be enough to keep mail harvesters away, and beyond that those mails go through the same spam filter as all my other mail traffic. Putting up a contact form would definitively be more of a hassle than just a simple mailto:-link, and then I would additionally have to start worrying about how to keep the bots away from that contact form.
This would result in half the planet being frustrated all the time and the other half never getting their mail.
If your goal is to secretly destroy email this is the way.
That's fine by me because
a) I do want to give out some sort of contact info on my homepage and people being able to message me in relation to that (and putting up a contact form leads to its own spam problems), and
b) if you happen to swap contact details offline, you then have to remember that you still need to additionally whitelist that person inside of that message service, which also seems somewhat of a hassle.
Somebody who gets inundated in unwanted messages might have a different opinion on that subject, though, and might indeed prefer a strict opt-in mode, with no exceptions…
If I'd actually use all of it a lot, sure but I don't.
It also doesn't work properly on Firefox on FreeBSD.
Even though on other OSes it works ok, it's so limited I can't imagine anyone using the web version of office 365 for any serious activity.
Then how about this: The big email companies all declare one day that any newly registered domain (with an MX record) needs to post a bond for good behaviour in escrow somewhere. If any of them find the domain being used to send spam, they can slash the bond (sending it to some charity or something).
This has the advantage that it doesn't affect any existing senders (so there's no one to complain about it), and it makes transparent the cartel-like power that these companies have over email. Perhaps, to democratise the process a bit, the ITU could organise a ballot (one vote per country) to elect 5 companies/non-profits who would have this bond-slashing power.
Unfortunately to implement something like this, they'd also probably have to demand that DKIM signing become mandatory (so there are cryptographic proofs of any evidence of spamming), and this sort of global consensus / money processing scheme would probably end up being built using a blockchain, whether that was a good idea or not.
So instead, the headline should be something like "Ask HN: Google, Amazon, and the Shanghai Cooperation Organisation forced me to send $100 of Ether to UNICEF and I couldn't send any new emails until I sent another $100 payment to my domain registrar. How do I take them to the World Court to force them to reimburse me?". That's not a great situation, but it's slightly better than the status quo.
I've never seen discussion of this in the mainstream though... so I'm not sure if it's actually being used or just shelved.
At this point, I think any proprietary they've created is game for usage. But it's very hard to get multiple large organizations to adopt this.
I definitely think it's a solution.
[1] https://archive.ph/CH98s [2] https://www.computerworld.com/article/2548788/cisco-to-acqui...
Want to talk about anti-competitive? Gmail will accept mails, provide a 250 SMTP response, then drop the email internally.
That's not right. At all. You can reject the email easily during SMTP exchange, and people have been doing that literally for 20+ years.
No valid excuse here. None. Zero.
And if your 250 OK accept then drop the message, and provide no way to notify, or discuss, or find out why, you make it impossible for a remote admin to fix the problem.
This is 100% on purpose. Yahoo, outlook/hotmail, gmail, collude to resolve issues like this, while blocking all others to resolve issues their own purposefully broken policies cause.
If you see Alphabet with a policy, or action, you can be 100% sure it is aligned to increase market dominance.
At this rate, eventually we'll have a handful of mail senders (Mailchimp, Sendgrid), and a handful of mail receivers (Google, MS).
Breaking reliable mail delivery for everyone, is inline with "there's no excuse, ever". It's inline with "making it worse", not better.
If you 250 accept, you deliver the email. Worst case, it ends up in a spam folder. You do not drop it on the floor. Ever. No excuse, no reason is valid here.
And I certainly won't accept "But it's so hard!", considering how easy it is to handle email, including SPAM, for everyone... until Google purposefully breaks it.
Or maybe it's an overly competitive protocol? Like playing Monopoly. Even a neophyte can end sweeping the game despite not understanding any of the underlying mechanics that drive the game's outcome. Those remaining mail providers are also fighting back the insanity. They 'just' won.
Don't hate the player, hate the game.
You can't put these big tech monopolists into the same bucket as normal companies. They have way too much power, and even when they do not intend to smash things, they end up smashing things.
SMTP is text based and we’ll defined, so I would also argue that the transport being “simpler” is nonsense.
Nothing is easy about email! Having worked for years just to get reliable in and outboxes is definitely not trivial. Also SMTP is a system out of your control if you want anything verified and actually delivered.
Say you enter a form at a company,and in the message box you just leave that code so they can mail you.
And in practice some minimal address obfuscation has been enough to block any address harvesters, and setting up a contact form or something comparable (and I guess especially also subsequently keeping it spam-free!) would have been much more of a hassle.
yeah, but in this climate what are you gonna do? It's not like there's any kinda recourse for monopolistic behavior that has any teeth to it.
iphmx 1 google 2 kornet 1 linkedin 1 secureserver 1 amazonses 1 self hosted university email 1
Please make your substantive points without swipes. Your comment would be fine without that bit.
https://news.ycombinator.com/newsguidelines.html
no forth amendment protection for your email because its stored by a third party.
Well, for one thing it prevents Google from building a profile of my online shopping behavior based on my email receipts. Also avoids them knowing every time I go on a trip based on booking confirmation emails.
Like most people, I'm not trying to hide from the government.
What does self-hosting accomplish? I'd say it's important in the way that having HAM radio operators is important.
I like the idea of having a node I can just plug into my network. I run my Urbit on a Mac mini with tailscale (which works great).
The core of what he writes about is correct though, email failed to be truly peer to peer (as imo all non-urbit-like federated systems will) because of the incentives that lead to centralization (spam, difficulty of running nodes, etc.)
We’re suffering the consequences of the local max we’re trapped in currently because of this. The promise of the 90s internet was a bunch of people using decentralized services they controlled - instead we're primarily thin clients connecting to a small handful of powerful ad companies. We're mostly serfs [0] allowed access if we give up our data for ad targeting, follow EULAs nobody reads, and don't say anything the company earls disagrees with.
[0]: https://zalberico.com/essay/2020/07/14/the-serfs-of-facebook...
So, mandatory third-party doctrine/warrantless surveillance.
Email reception has not been a problem for me so I enjoy having my mx pointing to my own mail server. It gives me more control and it requires very little maintenance.
If I was going to outsource anything the first choice would be outbound. My email system does everything right for reputation protection. All senders are authenticated on secure connections and the senders are people I trust. Nothing bad gets sent and my static IP is on a reputable server host and is not on any public black lists. I maintain SPF, DKIM etc. If someone decides to block my IP for no reason by accident or on purpose there is very little I can do or care to do anymore.
I have an alternate path for emails setup via a server I host elsewhere ready to go. If I run into a widespread delivery issue due to massive indiscriminate ip blacklisting of my provider I can enable it. That has happened once in 20 years. If delivery gets too hard I will change that policy to send all outgoing emails through a commercial smtp delivery service and let them deal with the problems.
With grace periods of the domain expiry process it's quite safe. People have plenty of time to recover it. It's also hard not to notice if your mail stops arriving.
Namely that "do it yourself at home" and "massive oligopolist" aren't the only two options. It's like saying "You can only have hamburgers two ways, cook them yourself or McDonalds."
I do the third and it's been great. I let my paid webhost handle it. (hostdime if you're interested, but I'm sure others do it well also)
and your comment seems to be saying it’s OK if we lose the ability to cook hamburgers at home, because there are other (more ethical) restaurants that aren’t McDonalds. am i misunderstanding?
In an anecdotal sense, I mean, I've dealt with my hostdime stuff, gmail and outlook. My stuff and gmail deliver just fine. Outlook does fail.
In a practical sense, see issue above. I think "non-delivery" via the big-boys getting things wrong on spam (and worse future outcomes, pretty easy to imagine gmail trying to "amp" or something similar email such that it cuts out little guys) is a far far greater danger or problem than an individual or smaller outfit tinkering with their own servers and occasionally getting things wrong.
Another way to look at it: In what world are you more likely to be able to cook your own burger: One where it's literally just McDonalds who can now control how people talk about burgers, or one where there's also other restaurants as well.
also arguably it's too ineffective considering how hard it is to open a restaurant. (because instead of harassing random taco trucks we should focus on other preventative measures, better visibility of food safety, better reporting and tracking)
sure maybe it's time for a "Let's Email" (after letsencrypt) service that handles reputation for email senders.
we already have certificate transparency logs, OCSP, CAA records, and all the fancy stuff, what's needed is something similar.
There are more than a few out there. No offense but if you would have had dealt with spam, you would have known that already.
the wonder of letsencrypt is that it does have the backing of Google et al.
I'm not necessarily saying your email has to go away and you HAVE to use hostdime. I'm saying the goal of McDonalds is to dominate and obsucre everything and anything that cuts into that is useful, especially third-parties like Hostdime. For someone like me that finds it too hard, but also finds gmail unacceptable this is a good (and should be obvious) solution.
How I explain this in real life:
"You'd be stupid to run a business off of gmail. When your phone or internet or electricity goes down, you can call a human and yell at them because you pay them. What happens for google? This is why I pay someone."
Not ideal, but it works.
I haven't done it since last year though. Has something gone terribly wrong?
I remember debugging issues with email sent via aws ses to Hotmail addresses at $dailyJob but I can't think of a single Microsoft product that works well (windows, teams, azure, now even GitHub is starting to work every other day) so it doesn't surprise me.
These days, I operate with the medium-temperature bowl of porridge: AWS WorkMail with custom domains & users. My use case is basically "Replace gmail for personal email". I don't have a lot of patience for running an actual email server, so this is about as custom as I can get.
Running a custom email domain can have other practical implications, such as having to carefully re-iterate spelling when mentioning your email address over the phone to a customer support agent. With a gmail or hotmail account, virtually everyone can type that hostname in without thinking about it. This concern is moderated by being able to select a username with fewer than 5 characters, rather than your full legal name appended with your date of birth.
A person at a company mistakenly created an email list segment (or lack thereof) resulting in an email to the entire email list of hundred of thousands of emails. This combined with inexistent (we were a naive startup without an email specialist role) list hygiene practices meant we were blacklisted by Gmail after some time.
Took a year to get a hold of someone on Gmail's spam team. We found out were on 4+ Gmail blocklists, some of which were ML-based. We couldn't do anything to remove ourselves after we fixed the issues. A $1-2 million revenue channel dried up because we couldn't get out of the Gmail blackhole (short of rebranding completely, rewriting content, and using a different ESP). Fun times.
[0] https://github.com/cfenollosa/os-tutorial
Disagree. It was a way too open protocol to begin with. From a time of innocence best suited for places with inherit trust like inside a business. And it's not just spam. Phishing is also a huge issue.
As much as I want to sympathise, Email for the big WWW is unsalvageable IMO. Too many bad actors are out there.
> [Solution:...] * There should be a recourse for legitimate servers
This is the same Big tech story. They want to cut cost, you want a human touch. You can see similar stories here in HN every week. Which is why I think it will never happen.
Dang, sounds like our monopolized tech dystopia is probably inevitable. If only there was some state level mechanism to help balance the interests of industries maximizing profits and those of users who need a robust system which doesn't harm them.
If your business suddenly fails as soon as you need to... support your customers, then you probably shouldn't be in business and only exist as a fluke in the current economic model.
Hosting an email server on a cheap (reputable) cloud server and doing the basics (PTR records, SPF etc) still works well.
Doesn't give full certainty, but can be well enough. (If all my recipients respond I don't care about the ones I don't send to anyways)
If you're really that worried about it, smarthost.
This is obviously laughably naive and creates infinite sources of spam.
Before doing a proposal on a core Internet technology you should be required to be on the other side for a while. Do anti-spam at a large retail e-mail service provider for a year and then you can understand the problem space.
You might not be responsible for what your neighbor is doing with their server, but the ESP is responsible for filtering it. The idea that they need to treat each and every comcast IP with equal weight is nuts. IP reputation is the single most valuable tool in the industry; the largest statistical predictor of whether or not an email is abusive.