At an even more basic level than what you described here, I recently improved my delivery rate by adding a name to the transactional emails that we send (from: 'My Name <admin@foo.com>') where previously we didn’t have a name. As well as cleaning up subject lines - where previously we included some abbreviations. Providers like mac.com were previously soft bouncing some emails, and now they seem to accepting them.
Hotmail/yahoo/aol all still seem to shove our transactional emails into spam pretty often (judging purely on the amount of those users who fail to confirm their accounts).
I'm surprised anyone's been getting through at all without perfectly configured SPD, DKIM, and DMARC. I've had a well configured self-hosted personal email server for years and still struggle to get through sometimes, though it does seem to be getting better.
Because it’s not just about configuration, it’s also about reputation and a low sending volume you are in danger of getting dropped merely out of not being a well known sender
I am surprised people think deliverability consists of configuring SPF (SPD is Seattle police), DKIM and DMARC. Spammers can do exactly same given low entry barrier.
It's important but have very little connection to deliverability in real world.
I run an email forwarding service[0] and it's damn hard to get into inbox of any major provider if SPF/DKIM aren't config properly. DMARC or ARC might be optionally but an email without SPF/DKIM, good luck having it hit any inbox.
Office365 is the toughest, email just randomly land on spam no matter what I do. Icloud, actually it's ProofPoint is tough sh*t too.
So I'm so surprise these guide just pop-up now like it's a new thing.
I’ve been running a personal mail server since 2016, and I’ve been struggling mostly with Microsoft off-and-on. I haven’t make any changes since setting up DMARC, SPK, and DKIM in 2016, but I’ll still sometimes randomly get blacklisted for 2~30 weeks for seemingly no reason, and then get unblocked again for seemingly no reason.
It’s recently started happening with iCloud too. For 5.5 weeks any email I sent would either get bounced or land straight in Junk, until yesterday when the powers that be decided my mail was worth delivering again.
I’ve somehow never had an issue with Gmail or Yahoo, and of course never with any other non-big-three mail servers.
Could be your IP block gets the odd spammer or you don't send enough emails and their servers resets you IP address reputation, meaning you essentially become a new email server.
Sometimes it's not about competence but about priority. If email is something a business does as a side thing and not a core thing, and if email keeps working without changing anything, there's no need or priority on setting up newer things. From a priority perspective at least.
But yes if it's a recent setup, or email is a core part of the product, any competent sysadmin should have been doing this.
Isn't there any open source project that solves the e-mail delivery problem? If not, why not? This sounds like something that can be fixed by software.
There's some online tools out there, and an OSS project that'll scan DKIM reports for you - but this was all I could find. Beyond this, there's commercial tooling / services. I may implement something that keeps a tab on things if I come across any issues.
for a long time it hasn't been a software problem. It's been a reputation problem, the only recent change is we're more interested in the domain then the IP now.
Please elaborate on this, as I'm not exactly sure what you mean. The email deliverability problem is a side effect of false positives in spam filtering. Unless you have a proposal to completely eradicate false positives?
What do you mean by the email delivery problem? Do you mean setting up DKIM, SPF, and DMARC? It's not that hard, compared to setting up an email server in the first place.
> Isn't there any open source project that solves the e-mail delivery problem?
Aplogies if you do, but it sounds like you misunderstand how email works.
The problems are stricly in policy, not in software. So there is nothing an open source project can do. The problem is that "too big to fail" megacorporations like microsoft just randomly decide to block incoming email from most of the Internet.
If email was fully decentralized (everyone runs their own server) this centralized power could not exist and there wouldn't be any problem.
(That all said, I run my own email infrastructure since long ago and it works fine. But I know some people struggle, which is contrary to the intent of the Internet.)
Yes, it is better. The reason is that when every email operator is small, the overall email community can pressure them to go fix things. You start by sending notes to postmaster@... with problem reports. If things go ignored you start blocking them and then they wake up fast.
In today's world when microsoft or gmail or yahoo start misbehaving in email handling, what are you going to do? It is impossible to reach them and while you could block them, they will never notice and you're only hurting yourself.
It is the classic problem of centralized power which hurts everyone except the (near-)monopolists.
> If email was fully decentralized (everyone runs their own server) this centralized power could not exist and there wouldn't be any problem.
You would have different problems though. Spam would still be hard, or arguably even harder, because you would have lots of small mail servers with no idea if they are trustworthy or not. Is this uncle Bob, who I havent spoken to for 30 years, sending me a heartfelt message about the family? Or is it a scammer trying to cream some cash out of me?
> Spam would still be hard, or arguably even harder
Doesn't make any difference. The major sources of spam today are places like gmail and outlook servers. You have to run some message classification regardless of where it came from.
> The major sources of spam today are places like gmail and outlook servers.
By volume perhaps, but is that true as a proportion of all mail that was sent via these services? I'd be surprised, though I'd be very interested if there where any numbers publicly available.
The spam situation could also be better if we had designed into that space. Want to send a single email to a sender that you don't have an existing relationship with? Hashcash. Have a web service that sends lots of emails? Get user consent beforehand with e.g. an oauth token to bypass or lower hashcash difficulty.
The worst is when they accept the mail but silently tag it spam and put it some place the intended receipient will never see it. Google's gmail is the worst about this. Corporate email isn't email anymore. It's a walled garden / silo like Facebook.
> Corporate email isn't email anymore. It's a walled garden
Thank god.
Otherwise I'd drown in cold email spam.
edit: But on a serious note — I'm using Gmail for all companies because I gave up trying to run and administer our own server. It's a travesty that this has become so hard. I feel if you're not on a well-configured Gmail Workspace there's no chance your email gets through, even if legit.
I haven't found this to be true at all. I've probably marked thousands of linkedin messages as spam, but they still land in my inbox occasionally. I also get random e-commerce related spam from sites I've never heard of. The false negative rate is massive on corporate spam that should be trivial to classify I'd think. e.g. some of them literally have some variation of "this is an advertisement" along with unsubscribe links.
For a while now I suspect that GMail has some bug with its own group email addresses:
When somebody sends an email to our GMail group email address team@example.com, it shows up in GMail as "Sombody via team@example.com". Of course, teamexample.com receives both spam and non-spam.
I suspect that when we mark spam that comes "via team@example.com", GMail learns that as "things 'via team@example.com' are often Spam", even though info@example.com is a Google Groups email.
And by now, everything that comes 'via team@example.com' is marked as Spam.
So it seems that when we mark an email that arrived at team@example.com as Spam, Google punishes its own Group email address, instead of the sender.
I think you can set it to be the other way around and have it be "never mark as spam when via team@example.com" - of course, depending on how much spam it gets that might be worse.
indeed very annoying. my understanding is they don't want to give spammers any signal whether their messages was recognized as ham or spam.
i would love some insights in how smart spammers are in actually leveraging such information. most spam seems to be hammering attempts that don't take failure feedback into account.
in my mail server, messages classified as junk keep getting temporarily rejected with a generic error message. at least the (legitimate misclassified) sender gets a delayed dsn, and finally feedback that a message wasn't received.
it seems many mail servers/services think it's more important to not give a signal to spammers than it is to give a useful signal to legitimate but misclassified users. perhaps they think their classification is really great and doesn't misclassify...
Unfortunately a lot of legal protections for unsolicited spam only apply to consumer usage. For b2b, every marketer with the ability to know your email address is entitled to send you as many messages as they want.
If it wasnt walled, it would be completely unusable.
I run my own mailserver. I do a catch-all so I receive all email sent to my domain. Even with this setup it is not "completely unusable" in fact it is very very simple to manage spam (entirely in thunderbird, no mailserver side spam filtering) and I have no issues with it. I've had the domain for 20 years now so it is not like it is not out there for spammers to send to.
Sure. Now try it again as an IT manager at a company that buys expensive technology products, with an email address that has been slumming it in lead gen databases for years. Try it again as a loan officer whose job is to trade high-value PDFs with externals whose email address and title is on the bank's 'about us' page.
The ability to know an address is the only license you need to send to it. The only reason you haven't had a bad time with this is because you haven't been a target.
This change was necessary and long overdue. Requiring domain owners who send significant volumes of email to properly sign their messages allows receivers to more clearly delineate good from bad based on domain reputation rather than IP address reputation.
As more domains send email through shared IP space on transactional and marketing services, having the ability to attach reputation reliably to the sender domain is incredibly helpful in reducing abuse.
The condition about "significant volumes" is not true.
Google states that the new requirements are mandatory only when you send at least 5000 messages per day.
This is a lie. I send at most a few messages per day and usually less than one per day was towards a gmail account and I had implemented a part of the requirements, but not all of them.
Nevertheless, Google has started to reject my messages, so I was forced to waste time with the implementation of all requirements, even if they are somewhat redundant.
That would not have been possible, with the already existing checks.
While the individual messages were not yet signed, Gmail should have already rejected any message claiming to be from my domain that was not sent by my own server.
My domain also sends no more than 10 messages a day. The domain is correctly-configured with SPF, DKIM and DMARC.
At the beginning of March, I started getting temporary rejections from gmail. Not all of my outgoing messages, maybe 1 in 10. Most of these messages were to one individual, to whom I've been sending for years. My domain has existed continously since 2002, and has never sent spam. The rejection message was startling: words to the effect of "Your message has been rejected because of the awful reputation of the sending domain". The reputation of my domain is spotless, according to various testing tools.
There have been no new rejections in the last two weeks.
According to TFA, Google started rejecting a proportion of mail from bulk senders in February. I wonder if I got caught up in some half-baked roll-out of this new (old) policy.
Same here with 20+ years old mail service on the same domain that has never sent spam with correctly configured DNS SPF DKIM DMARC, getting gmail rejections. I noticed a significant improvement after linking the domain to my google account https://support.google.com/mail/answer/9981691
No, I did not. I barely ever use Google services of any kind.
My "domain reputation" should mean something like "the consensus of multiple reputation services", where those reputation services are reasonably open about how they score. It shouldn't mean "the completely opaque opinion of a single, hyper-dominant, secretive provider with all kinds of conflicts of interest".
I can confirm this, I have a mail server with extremely low, non-bulk mail traffic and without SPF/DKIM/DMARC nothing works. Even with, Gmail started rejecting mail (probably noisy neighbor), so the only way around that was to use the little-advertized relay service in the paid Google Workspace when sending towards gmail or workspace domains.
It also says that you only need to hit this limit once and you’re forever on that list. Also domain is considered as the canonical domain, the part right before the TLD. Subdomains are counted together with the apex domain.
The 5,000 message limit adds _extra_ requirements for senders; ALL senders, regardless of volume, must adhere to these standards:
Starting February 1, 2024, all email senders who send email to Gmail accounts must meet the requirements in this section.
- Set up SPF or DKIM email authentication for your sending domains.
- Ensure that sending domains or IPs have valid forward and reverse DNS records, also referred to as PTR records. Learn more
- Use a TLS connection for transmitting email. For steps to set up TLS in Google Workspace, visit Require a secure connection for email.
- Keep spam rates reported in Postmaster Tools below 0.3%. Learn more about spam rates
- Format messages according to the Internet Message Format standard, RFC 5322.
- Don’t impersonate Gmail From: headers. Gmail will begin using a DMARC quarantine enforcement policy, and impersonating Gmail From: headers might impact your email delivery.
- If you manage a forwarding service, including mailing lists or inbound gateways, add ARC headers to outgoing email. ARC headers indicate the message was forwarded and identify you as the forwarder. Mailing list senders should also add a List-id: header, which specifies the mailing list, to outgoing messages.
As a sys admin, what do you do when you see 5% of your email hitting spam because the recipient’s Office365 mail server is misconfigured?
Agreed it’s a net positive, but it kills me when the reason emails land in spam is misconfiguration at the recipient’s end. (Like forwarding emails which breaks SPF)
My employer has had massive issues with this recently. Lots of office chatter from HR about needing to call candidates b/c the response rate is down this year due to spam folders eating the offers.
We also slowly but surely are moving towards IPv6 which will make reputation based on IP somewhat useless when I can have as many new IP addresses as I want. They would have to make all newly seen IPv6 addresses not trusted by default when bad actor could send each email from different IPv6 address.
That sucks! And fixing reputation is at best a nightmare. I've seen suggestions about purchasing / sending email from other domains - to protect your primary domain. Not something I really care to do.
Google themselves recommends sending different types of emails from different subdomains of the primary domain to help Google differentiate between transactional, marketing, newsletters, outbound, etc
Any half decent marketer will 100% use a different domain for outbound sales (or any use case where spam rate might be abnormally high).
I'm not nearly as sure. The mechanics of one aren't that hard to imagine. Encourage spammers and make it cheap. Don't ever fight them at their source. Invoke mechanisms that intentionally destroy public utility in public protocols. Force everyone to rely on a small handful of "reputable" senders.
Working backwards from "who decides reputation anyways?" might make it easier to see.
Sending requires quite a large volume for the big players to allow you to play. The only viable option for small servers is to use a SMTP relay service. Amazon's was a pain to get out of the sandbox mode but has been reliable and most importantly free.
As I care more about the recieving side than sending emails this works well enough for me.
yes, this is a pain. the blocklist operators either seem to be not so good at vetting abuse reports, or cause collateral damage to get network operators to take action (while i don't like getting on a blocklist, perhaps it is having a net positive effect for the wider internet?).
i do think mail servers/services are using ip-based blocklists wrong. yes, you can use it as one of many signals. give it some more weight for first-time senders. but if you've been mailing (with spf/dkim/dmarc-authenticated messages/transactions), from an ip that suddenly gets on a blocklist, the previous positive reputation should be stronger than that blocklisting, and you should be able to keep communicating with your known correspondents (until they mark your message as spam after which future deliveries can be rejected/junked). it seems those mail servers/services cheap out and apply ip blocklists early in the smtp process. good for their system load, bad for their analysis performance.
in general, it seems even bigfreemail services are bad at using existing reputation in their ham/spam decisions. i recently switched an online webservice i made (that is about sending certain notification by email) to signups via email (like how you can signup to mailing lists: by sending an email to an address). the idea: if you send my service an email, i'm in your list of known correspondents. so the confirmation reply from my mail server (spf/dkim/dmarc-aligned) should certainly be accepted by yours (you much more opt-in do you want?). i tested with some bigfreemails, and yahoo put my reply with confirmation (that even references the original message) in the junk folder. to people who think you can't compete with bigmail: the bar isn't as high as you may think.
The problem is that your VM provider is (unwittingly) offering the same address space to spammers. Maybe the spammer was using random addresses in the range? Or someone was starting up misconfigured smtp servers permitting relaying.
Running an outbound smtp server on a customer ip-range is going to be problematic anyway. All such ranges can be considered suspicious since the spammers who use them don't care about their standing.
The thing that kills me about DMARC is how often is fails with Microsoft specifically. And also with any use case involving the recipient forwarding mail (which breaks SPF alignment)
I want to follow best practices it recently changed p=quarantine to p=none after fear that legitimate emails aren’t passing DMARC despite properly configured DKIM and SPF.
Hell, I would love p=reject but not until recipients fix their incoming mail servers to handle edge cases like email forwarding breaking DMARC
This is the most important part. Exchange (due to its history as an X.400 server, not as an SMTP server) does sometimes mangle the message to the point that DKIM simply breaks. This both breaks origin-incoming and forwarded messages.
BTW, Apple also sometimes mangle messages that it fails DKIM, although I do not know why is this the case (as I doubt they use Microsoft Exchange for their mail service).
this is a long standing problem with mailing lists. they are often configured to add a "[...]" prefix to the subject or add a footer, breaking the dkim signature. this leads some more recently updated mailing lists to always rewrite to their own "message from" header, so they control dmarc alignment for their messages.
for incoming email on mailing lists i'm subscribed to, i don't enforce the dmarc policy. i think this is what the parent post hints at. i'm not sure how easy this is to configure with the various mail server software out there. i'm also not aware how you would configure this with sieve scripts (i looked, didn't find it, but it seems like a basic case).
if you're running a mailing list, hoping for all subscribers to not enforce dmarc policy enforcement doesn't seem like a great strategy.
the forwarding case should be easier to keep working.
This looks like a good document, but the author made it political by referencing "Hilary" Clinton and her emails and linking to some Trump stuff. I can't take tech stuff seriously that's dropping in political crap. Go away!
Yes, this put me off enough that I was no longer interested in reading the article. The silly reference is one thing, but was a link to that YouTube video really necessary?
1. GMail will block your email if you don’t allow one-click unsubscribe. But this is very insecure since anyone can unsubscribe you if you forward your email
Easy Unsubscribe: Implement easy unsubscribe options (One-click Unsubscribe). Gmail users have tools to report spam, unsubscribe from unwanted emails and control their inbox experience. If it is too difficult to unsubscribe from your emails, customers will be more likely to flag your email as spam. Additional links provided in the ‘References’ section at the end of this article.
2. At the same time, Apple’s ITP will start removing all the information from the URL and only leave the domain, if it classifies your site as a “bounce tracker”. This means you won’t even know who to unsubscribe on one click! So all your emails will be blocked.
Gmail does not require one-click unsubscribe, what they actually require is that you include the “List-Unsubscribe” header in bulk emails, with a functioning mailto or http target.
If I forward your newsletter, that’s not a bulk email and it won’t include that header.
This is an important distinction that seems to get glossed over in a lot of the coverage and guides about the recent Gmail and Yahoo changes.
The default forward action in Gmail sends it as content. Forwarding as an attachment is a well-hidden option in the web UI and I just looked in the Gmail mobile app and could not find a way to do it there.
>2. At the same time, Apple’s ITP will start removing all the information from the URL and only leave the domain, if it classifies your site as a “bounce tracker”. This means you won’t even know who to unsubscribe on one click!
Your source doesn't actually say that "ITP will start removing all the information from the URL", only that it will "limit it the same way as third-party cookies" and will be "purging website data in such instances".
Given how much weight “Gmail”, “Outlook”, and “Yahoo” email providers pull, I have always wondered about a different type of attack on business entities: “targeted failed deliverability”
Basically in this attack, a victim (particularly a business or mailing list or NGO) is sending out bulk emails to which the attacker owns. Even sourcing this out to shady off shore click farms would work too.
Attacker then marks the victim’s emails as spam in Gmail/Yahoo/Outlook. The “AI spam filters” pick up on this new “spam activity” and will then mark future emails as spam or even delete them before reaching real customers.
After a year, company bleeds money on a quarterly basis. Ad departments wonder why there is decreased engagement through email. Technical departments are bamboozled.
Maybe a big company will be able to weather the storm or just ditch email altogether. But small companies would definitely take a hit. Even smaller NGO or political mailing lists would lose donations (assuming email was a significant source of new donations).
Probably a very low vector of attack tbh, but something that has lingered in my mind.
A webforum I know has a rule against marking email notifications they send at spam (You can opt out of receiving them through the site, they just don't want you doing it on the email client end) to avoid this happening to them. For a small org, it's kind of a real risk?
They pay 3rd parties to handle relay duties. A large chunk of reliable delivery is based on those received headers, aka what systems your mail is relayed through.
But is the Internet really as centralized as Email has become? I was living under the impression that Internet backbone providers where more diversified than that.
With BGP, you are usually only peering with a couple entities unless you are yourself in the "big boys club". It's a relationship business, but you only need a relationship with the handful of engineers who are on the other side of your own links.
I don't send marketing, I only send transactional emails. Notifications are opt-in, and transactional. Logins / signups are done via sending a TOTP to the email, these too are transactional.
If someone marks notifications of new posts in a subscribed thread as spam... fine, but this is self-solving, as this person will trigger no more email ever reaching them for that email address, meaning they also cannot now sign in to the website, and therefore cannot subscribe to more email updates, or visit things to refresh "last viewed", and so would never be notified again.
Emailing a TOTP as a login has increased deliverability by self-selecting the removal of those who hit the spam button. Deliverability of email from the webforums is over 99%.
Reporting email as spam, effectively bans oneself from the website. I didn't even need to do anything.
When you report something as spam to Google they only claim they "might" cause later emails from the same sender to go to your spam folder; regardless, those emails going to one's spam folder doesn't mean they don't get them at all, it just means they have to look at their spam folder to log in.
When you use services like Sendgrid, having a spam report will automatically prevent future emails being deliverable, third parties and intermediaries do this precisely to protect their email reputation.
Marking spam may not be immediately be a death knell within Gmail, Hotmail, etc, but because of the potential impact from being marked as spam, virtually everything treats it like a death knell.
This is fine for me, my service is entirely opt-in, and if someone hits the spam button it risks impacting other users who _want_ the email, so I am not bothered that this person effectively unsubbed themselves and killed their account.
I can only share my own numbers: 90K sent, 97.12% delivered, 103.11% clicked (presumably some clicked more than once? or some older emails still being clicked?)
I do not track "opened" / "viewed" as that is not important to me, these are transactional emails, logins or "someone has replied to your conversation".
I run a small org (freelancer) and I have a super advanced bullet proof set of solutions against this. Any 1 of those should do the trick (I do all of these):
1. Don't trick people into signing up for mailinglists.
Nobody is spoofing anything, DMARC won't help. This is marking legitimate emails as spam at scale to trigger actual spam controls against the legitimate organization.
Yea I've thought about this but not from the "attack on entities" angle but moreso a consumer-rights / boycott angle. I've had a negative enough experience with a large "maximizing shareholder value" company that I went back through my email history and marked every single one of their comms as spam.
Might be a drop in the bucket, but it doesn't take many votes to make a difference in the spam world.
I'm sure this will evolve soon enough and email delivery might increasingly become pay-to-play with all sort of backroom agreements, if it isn't already.
I bought my dad a sweater for our local MLB team. I made the mistake of using my real email. Ever since I’ve gotten a steady drumbeat of marketing emails and other low value content from them.
Spammers want us to think there’s a significant difference between their newsletter or marketing notes we may have technically signed up for (certainly not willingly) and I don’t feel bad about reporting both of them. If this forces spammers to consider whether recipients will want their messages, good.
It sounds like you absolutely did sign up for the emails, though.
I'm not sure how you could "unwillingly technically sign up" for something like that, especially at the scope of an MLB team which is going to have a team of lawyers, marketing policies, etc. They're not just going to spam people the risk is way too high.
> It sounds like you absolutely did sign up for the emails, though.
Did he? The anecdote here is probably observed by everyone on this forum. How odd that you find it's unlikely to receive spam from a business transaction.
> I'm not sure how you could "unwillingly technically sign up" for something like that
Have you tried using the internet?
> the scope of an MLB team which is going to have a team of lawyers, marketing policies, etc. They're not just going to spam people the risk is way too high.
I would love to live in your world where there's little likelihood of getting spammed just for purchasing something once. Unfortunately, spamming people has effectively zero risk and all reward. If there were any real risk then we would see actual real and frequent consequences every day. We don't see that, but we do see lots of spam in our inboxes.
I do work in digital marketing and there easily could be an automatic sign-up to email without ever opting in. There are plenty of services that will "restart" your email or that will identify you and send emails without consent [1]. If someone is saying they didn't subscribe I would believe them.
There are many dark patterns that encourage people to accept their address going on the general spam list:
* wording on check-boxes being sufficiently ambiguous that yo have to think to get the right meaning
* check-boxes on forms in different stages of the ordering process that have inverted meaning from each other
* resetting check-boxes to the “please send me junk” state if the user goes back to correct some other issue or otherwise update the order
* wording the checkbox text to imply that opting out also opts out of getting order update emails (or in some cases opting out does opt out of order updates messages too, and you have to check via some other method what the state of your order is)
And so on.
It is easy to avoid all these accidental opt-ins individually, but it is also very easy to miss one in amongst the sea of dark patterns and one is all that it takes. You need to be vigilant all the time, they scummy marketers only have to get lucky once.
Furthermore, I'm sure some just ignore your opt-out and send crap anyway. When taken to task they'll say you must have accidentally [un]checked the wrong box (how do you prove otherwise?) and tell you that you can opt out from future emails by following the link in the messages.
This is Google's business model, they throw completely legitimate emails your business sends into spam/marketing, so you're forced to pay them for gmail ads.
Yes, that is what I'm saying so I already agree with you.
My point is Google takes content from Column A and puts in in Column B and collects money in their little toll booth. Google pretends to care about spam only when they can't make any money from it, otherwise they don't give a shit unless its a PR issue. Google ads have long been a malware vector - but they don't care because they make tons of cash from it.
I'm not taking the side of either the business or the user here, but this is a forum for start-ups who often have to send out legitimate business emails - which are not necessarily spam.
I don't think this would work in practice. My employer sends daily deal emails without using a third-party service like SendGrid or SES and what we do is pay a company like Validity who interface with the big email providers for us. They have honeypot emails that get and validate our emails, they get feedback from the providers on how much they like/dislike us, and we get reports on this.
So an attack like this would be very obvious very quickly, even leaving aside that we'd notice a huge spike in email sign-ups and probably kill their accounts (especially since they're not going to be buying anything from those sock puppet accounts!).
If email is business critical, you do it. I work for a pretty small company and we do stuff like this. We have a sender tool that gives us a static sender IP, reports deliverability, click rates and at least estimates open rates. We also have a tool to estimate the quality of a newsletter sign up email address and not collect any disposable emails.
That’s why most e-commerce sites send their marketing stuff from a different domain. If it gets flagged they can still send transactional e-mails from their main domain. Assuming that both mail servers are on different ips.
This seems kinda deluded given that spam prevention teams have been identifying domain clusters for sender reputation management since at least 1998 to my earliest direct knowledge and probably earlier. Maybe it works for a little while, but don't bet your company on it.
I'd argue transactional email should also be sent on a subdomain, to allow for a migration if that subdomain gets "burnt". I'd also argue that really nothing should send email from your root domain at all for this exact reason, but legacy environments can often make this a non-starter.
The problem is then the reputation to humans reviewing links. A legitimate company sending from legitimate-company-transactional-emails.com looks exactly like a phishing scam sending from legitimate-company-scam-emails.com. This can be mitigated by not putting anything important in the email ("You have a new message on MyChart, figure out on your own how to view it."), but people will be confused and it will waste support time.
It's honestly really bad no matter which route you choose, which is why everyone is so intent on getting you into their mobile app that can spam you without affecting their reputation.
Also, even if marketing emails are sent from a separate domain, sales people can be sending so much "spam" that your normal humans-at-the-company emails start getting rejected. I think this might have happened at my last startup; email reputation got bad, interview candidates didn't get their interview conformations anymore. But I might be misremembering.
Subdomains are basically restarting your domain reputation from scratch. I would use the apex domain and a validator in most cases, as long as you're doing the right thing and maintaining list health. Obviously mega-corps might be different and have needs for subdomains.
Isn't that what DMARC policy would prevent? If the emails being sent by the attacker are failing SPF/DKIM, then we can configure the DMARC policy so that Gmail never delivers those fake emails in the first place.
So that attack would not be happening.
The attacker isn't sending mail spoofed from the domain--they're intentionally signing up for legitimate newsletters from that domain using a ton of email accounts they control, then reporting those newsletters as spam in each email account.
This could be a pretty interesting use case for a botnet as well, compromise computers, compromise email accounts, for the sole purpose of selling the ability to mass-spam your competitors. Obviously immoral but simply from a technical standpoint I'm wondering if it would work and what scale you'd need for it to be effective.
If political candidates can’t get small donations then that shifts more power to large donations 1 wealthy companies, people, and unions (maybe not the last in the US)
Mostly that Unions went hard on endorsing Biden, but their members (at least in the industrial Midwest) dutifully lined up to pull the lever for Trump in 2020.
You know what drives me not to be involved? Knowing that buying a mug from a candidate I like virtually guarantees a torrent of emails from every downticket race in markets I’ve never lived in. And because of the way they set up the lists, unsubscribing means unsubscribing only from “Kelleher for Coroner” in a county halfway across the country. The worst part is the incredible entitlement you experience if you mention this to someone involved in a campaign. I continue to vote, but I haven’t donated a penny to a campaign in more than a decade, specifically to avoid the harassment.
I know of victims who had their legit email template lifted by actual slammers. The spammers would embed the legit template invisibly in their emails and then only have a few short lines visible with the actual scam. The idea being that filters would see the majority of the email is legit looking and let it in. Eventually users would flag enough of these as spam and the template itself would trigger the blocks. Then the spammers would move on to the next victim who's email template still gets through filters.
Meanwhile the first victim is left to pick up the mess where none of their email gets through anywhere.
Do political mailing lists get through in the first place? I don't live in the US, but someone must've sold one of my throwaway gmail addresses thinking I did.
Every couple of months I end up checking its spam folder and it's just a daily barrage of spam from both DNC and RNC, 1-2 emails per day like clockwork. None of them ever got through to the inbox though.
As usual in any email thread about email deliverability, the amount of FUD in these comments is absolutely mind-boggling to me. I’m not unusually smart or intelligent or capable. I wouldn’t consider myself a deliverability expert. It’s only a small small part of my job. I’ve never worked for any organisation that sells email delivery services to third parties. Why the hell can I understand this stuff, and get it to work, while there are so many people here that very clearly indicate (via what they’re saying in their comments) that they DON’T get it yet have a serious axe to grind?
I’m left feeling like homegrown email delivery is some sort of lightning rod for stuck-in-the-past faux-sysadmin types that can’t get past the fact that it’s not 2003 anymore and lazily / maliciously comply with SPF / DKIM.
Without knowing everyone’s domains and IPs and history, it’s hard to judge competency in a thread like this. SPF, DKIM, and DMARC are important to set up correctly, but doing so is NOT sufficient for good deliverability. In fact it is only a small component of success.
Well-established organizations that have a long history of sending steady volumes of high quality content with low complaints have a huge leg up. So if you work at such a place, or you contract with such a vendor, you’re going to feel like it’s obvious that the DNS entries work well.
>As usual in any email thread about email deliverability, the amount of FUD in these comments is absolutely mind-boggling to me. [...]
What type of "FUD" are you talking about? The objections in the thread seem pretty well founded (eg. being shadowbanned despite complying with SPD/DKIM, or this requirement breaking email forwarding), and there aren't really any that are against implementing SPF / DKIM.
A problem is that you can do everything technically right, and sitll land in spam, because some big players don't play by the usual rules.
For example, Microsoft apparently has an allowlist for IPv4 -- or equivalenty, blocks all IPv4 by default, until you manually de-list them at sender.office.com. At least I haven't found an IP yet for which I didn't have to do that (self-hosting email for 15 years).
(Imagine every provider did it like MS; you'd be sitting there and filling out web forms with 1000s of providers.)
So you have a technically perfect setup and MS stil rejects you.
--
That said, using some provider to send emails for you doesn't solve deliverability either. There, many customers share the same sender IP. If one of them sends marketing/spam, the entire IP gets bad reputation. In such cases, providers recommend to upgrade to "bring your own IP", which then needs to "gather reputation" [0]. Great, might as well have self-hosted in the first place, as repuation is the only thing I bought the service for.
Congratulations for being lucky enough to have big actors accept your e-mails, this is not the case for many of us who do understand and apply SPF, DKIM and DMARC. Guess we'll just try being luckier...?
I have the same gripe and in response published the notes to set up my self hosted email on my blog a few months ago [1]. It's really not that hard but yet we constantly see this topic on HN. I understand there are people who've set it up properly and still have their mail end up in spam. Maybe we're just lucky. But there's no need to write long pieces about this going into detail what this tech giant and that tech giant do. All you can do is set up DMARC, DKIM and SPF - that's it.
EDIT: Admittedly this post is also about bulk sending where other metrics like unsubscribe links and spaminess matter. But for the self hosted crew it really just comes down to DMARC, DKIM and SPF.
What I have gleaned from the HN discourse over the years is that the people who are mad about this topic actually are spammers. They want to spam you and they are super pissed off that their little scammy idea isn't working.
I'm surprised how many big companies fail the one-click unsubscribe test. Whether it's Cloudflare or Akamai blocking the connection, pages that take 5+ seconds to load, pages that require you to sign in or input your email address again... don't be surprised when customers reach for the Report Spam button instead.
I'm using NextDNS with AdBlock list, which is effectively a Pi-hole on the cloud.
The most annoying this is when email senders use click tracking on domains that are blocked by those AdBlock lists. I keep a separate browser instance to copy-paste those links into, but then I have to login again.
I prefer sending unsubscribe emails instead of clicking links. Gmail can automate it.
Same. If I subscribed to it (rare), I'll unsubscribe. If I get unsolicited mail of any kind, I'm not "unsubscribing", I'm nuking from orbit by any means available, including reporting to hosting providers. (This has, on occasion, resulted in the termination of a spammer's account, but the success rate is low.)
I'm mostly talking about those emails with helpful hints and upsells that you get when signing up for a new service. People who make a living from spamming^H^H^Hbulk sending may consider those legitimate.
I’ll unsubscribe, but now call BS on the “this may take 14 days to take effect…” nonsense in 2024. If I’m getting more emails in a couple of days, they’re getting marked as spam. (Looking at you, TripAdvisor. If you can figure out how to build AI-generated itineraries, you can figure out how to not email them.)
One thing the April changes break is forwarding between e-mail services. If you currently forward from say an old university address at foo@school.edu to a personal GMail account at bar@gmail.com that will no longer work. This must be relatively uncommon if the major providers are charging ahead with these changes but it's pretty annoying for the people affected.
That's my understanding as well, but I also understand that the email will pass DMARC if either SPF or DKIM passes, and DKIM will still pass on forwarded email.
This isn't a change. SPF has always broken forwarders that don't touch the envelope from address, and that is right and proper. You can still forward mail, but your forwarder must rewrite the return path.
It's is a change in mail service behavior, and arguably either a broken policy or broken spec (not sure and don't really care since the effect is the same).
"In April 2024, Google will start rejecting a percentage of non-compliant email traffic and gradually increase the rejection rate. For example, if 75% of a sender’s traffic meets our requirements, Google will start rejecting a percentage of the remaining 25% of traffic that isn’t compliant."
It's really not a change or any of those. Mail services have behaved this way in some form or way for a while, these specs have worked like this from the start and the policy is only a logical conclusion of it all.
Untrusted forwarders (also sometimes known as open relays) have been heavily frowned upon for a really long time. SPF has never worked for forwarders that don't do rewriting. Those letters have always been negatively looked upon, unauthenticated mail is a remnant of times long gone.
Google's policies about authentication are the bare minimum you should be doing anyways if you care even a little bit about your customers or recipients in general.
at least gmail.com has a dmarc policy p=none, so a failing dmarc check is not a reason to reject email. i don't think it's that common for small-scale installs to enforce dmarc policies. there are other signals to use though.
plenty of bigfreemail spam is actually sent from their network, they're an interesting target for spammers, and at least some of them put a lot of effort in preventing abuse.
You might be surprised, if you're young enough, to hear that this is exactly how it worked for a very long time.
At my first corporate job, and my second one as well, all email was sent and received directly from/to each persons individual workstation. There was no concept of centralized email server.
(Let alone outsourcing it to some other company, that would have been completely unthinkable.)
It’s not even stolen accounts. There is a huge amount of cold sales email (spam) sent specifically through Google Workspace accounts. Google do absolutely nothing about it.
Then there's the other side - receivability. IDrive is supposed to send me an email each day reporting backup status as seen by the backup servers. Those messages have been flaky since mid-February. Logs indicate the backups run; it's just the completion emails that fail.
Their support people blame me, although they admit others have the same problem.
They're not using a mail delivery service - the emails come directly from an IDrive server.
They're sending to my web site, which forwards to my personal address. There's no filtering at the first stage, and a division into Accept/Greymail/Junk at the next stage. Neither Google nor Yahoo is involved at any point.
I ran my own mail server for more than a decade. Same IP the entire time, never sent spam (for personal use only.) Finally threw in the hat last year and moved to a paid service - it was a pain to tell every person I sent mail to to check their spam box and mark me as not spam or add me to contacts. Beyond that, gmail smtp servers kept getting onto spam blocklists, so I wasn't receiving mail from gmail at times.
Gmail and Microsoft require a certain volume of email hitting their servers otherwise they forget you exist. I ended up switching to using Amazon's SES service to keep my selfhosted server running.
I use seperate emails for each service much like a seperate password so I'm heavily invested in keeping my own server at least for recieving.
The article is branded as a "deep dive". If it doesn't affect deliverability in a measurable way, then at least mention it in passing to say that it doesn't affect deliverability in a measurable way...
I have received exactly 2 such messages in the 25+ years I've had a mobile phone. I imagine this is a regional thing, and that where you are eithet mobile networks are insecure, or operators are in cahoots with spammers?
I totally agree, and also unsolicited calls. Right now the only options that I'm aware of are:
1) Blocking numbers (which is pointless because they rotate them and spoof residential numbers).
2) Whitelisting numbers and blocking everyone else (this would cause me to miss legitimate calls).
3) Blocking entire ranges (this doesn't work because of the spoofing).
4) Using one of those spam screening services (currently looking into this). Still a bit concerned about missing valid calls and the privacy issue with this.
Unsolicited calls are annoying but not that much of a problem, because unknown number real people that have a good reason to join me can always leave voicemail, while spammers never do (not worth it for them I guess) ?
I’ve gotten some emails from Gmail about delaying my emails to Gmail users because I apparently send too many emails. I use git-send-email(1) which might send a cover letter plus X patches right after each other. These Gmail users are then in the CC. So I’m not a mailing list. The email list is the To recipient.
I’ve been wondering if this was the cause. I don’t send out 5000 emails (I’m not 10X). But there’s this part:
> While these guidelines primarily affect bulk senders, senders with less volume per day can also be affected if they are not adhering to these guidelines.
I haven’t looked into it yet but I guess I should.
I use my own domain and I’m hosted by a not-Gmail provider.
I wonder how this ends up impacting government agencies and especially courts and law firms. My experience has been all three struggle with these things.
I think you mixed envelope (RFC5321) and headers (RFC5322) in your text.
The domain name in the From: field in the email envelop header is inspected and aligned with other domains authenticated by either SPF or DKIM:
The envelope does not have any header, the headers are in the content/body of the email. Also your screenshot of the "Here’s an example email envelop from an organization that passes all of the email security guidelines:" are the mail headers and not the envelope information.
230 comments
[ 2.5 ms ] story [ 152 ms ] threadHotmail/yahoo/aol all still seem to shove our transactional emails into spam pretty often (judging purely on the amount of those users who fail to confirm their accounts).
And Apple sends no dmarc reports nor do they implement any actionable feedback loop.
Just having them perfectly configured doesn't mean that the receiving servers will also see it that way.
Microsoft servers are particularly prone to randomly failing perfectly fine dkim setups for no reason whatsoever.
It's important but have very little connection to deliverability in real world.
Office365 is the toughest, email just randomly land on spam no matter what I do. Icloud, actually it's ProofPoint is tough sh*t too.
So I'm so surprise these guide just pop-up now like it's a new thing.
---
[0]: https://mailwip.com
And that's a decent thing.
Someone that doesn't do that has no business sending email.
The problem is that even doing that and even sending only a couple of emails a day is still usually considered SPAM.
It’s recently started happening with iCloud too. For 5.5 weeks any email I sent would either get bounced or land straight in Junk, until yesterday when the powers that be decided my mail was worth delivering again.
I’ve somehow never had an issue with Gmail or Yahoo, and of course never with any other non-big-three mail servers.
But yes if it's a recent setup, or email is a core part of the product, any competent sysadmin should have been doing this.
Aplogies if you do, but it sounds like you misunderstand how email works.
The problems are stricly in policy, not in software. So there is nothing an open source project can do. The problem is that "too big to fail" megacorporations like microsoft just randomly decide to block incoming email from most of the Internet.
If email was fully decentralized (everyone runs their own server) this centralized power could not exist and there wouldn't be any problem.
(That all said, I run my own email infrastructure since long ago and it works fine. But I know some people struggle, which is contrary to the intent of the Internet.)
In today's world when microsoft or gmail or yahoo start misbehaving in email handling, what are you going to do? It is impossible to reach them and while you could block them, they will never notice and you're only hurting yourself.
It is the classic problem of centralized power which hurts everyone except the (near-)monopolists.
You would have different problems though. Spam would still be hard, or arguably even harder, because you would have lots of small mail servers with no idea if they are trustworthy or not. Is this uncle Bob, who I havent spoken to for 30 years, sending me a heartfelt message about the family? Or is it a scammer trying to cream some cash out of me?
Doesn't make any difference. The major sources of spam today are places like gmail and outlook servers. You have to run some message classification regardless of where it came from.
By volume perhaps, but is that true as a proportion of all mail that was sent via these services? I'd be surprised, though I'd be very interested if there where any numbers publicly available.
Would you care? I can say that I don't care (as someone running my own email infrastructure).
I run filtering on all incoming email, I don't assign any weight to which email server it came from because it's not a useful factor.
Thank god.
Otherwise I'd drown in cold email spam.
edit: But on a serious note — I'm using Gmail for all companies because I gave up trying to run and administer our own server. It's a travesty that this has become so hard. I feel if you're not on a well-configured Gmail Workspace there's no chance your email gets through, even if legit.
Gmail's spam filter (and promotions filter) works with >99% reliability as a user, with really trivial numbers of false positives.
at this point maybe you should just create a filter rule
Customer requests for quotes, Paypal/Stripe security messages, lots of other important emails go to spam.
See e.g. https://github.com/nh2/gmail-spamfilters-paypal-security-mes...
For a while now I suspect that GMail has some bug with its own group email addresses:
When somebody sends an email to our GMail group email address team@example.com, it shows up in GMail as "Sombody via team@example.com". Of course, teamexample.com receives both spam and non-spam.
I suspect that when we mark spam that comes "via team@example.com", GMail learns that as "things 'via team@example.com' are often Spam", even though info@example.com is a Google Groups email.
And by now, everything that comes 'via team@example.com' is marked as Spam.
So it seems that when we mark an email that arrived at team@example.com as Spam, Google punishes its own Group email address, instead of the sender.
It's Google's own group of my own org. Does it make sense that the spam button would punish my group own email?
Also, if I mark it spam in the Groups interface, it's still polluting the inboxes of all my team members where it has already arrived, right?
I'd have to tell my team to never use the spam button on obviously spam emails. That doesn't make sense for a business email product!
i would love some insights in how smart spammers are in actually leveraging such information. most spam seems to be hammering attempts that don't take failure feedback into account.
in my mail server, messages classified as junk keep getting temporarily rejected with a generic error message. at least the (legitimate misclassified) sender gets a delayed dsn, and finally feedback that a message wasn't received.
it seems many mail servers/services think it's more important to not give a signal to spammers than it is to give a useful signal to legitimate but misclassified users. perhaps they think their classification is really great and doesn't misclassify...
If it wasnt walled, it would be completely unusable.
It sounds like stiffer penalties are necessary for sending spam, both to consumers and to businesses.
The ability to know an address is the only license you need to send to it. The only reason you haven't had a bad time with this is because you haven't been a target.
As more domains send email through shared IP space on transactional and marketing services, having the ability to attach reputation reliably to the sender domain is incredibly helpful in reducing abuse.
Google states that the new requirements are mandatory only when you send at least 5000 messages per day.
This is a lie. I send at most a few messages per day and usually less than one per day was towards a gmail account and I had implemented a part of the requirements, but not all of them.
Nevertheless, Google has started to reject my messages, so I was forced to waste time with the implementation of all requirements, even if they are somewhat redundant.
While the individual messages were not yet signed, Gmail should have already rejected any message claiming to be from my domain that was not sent by my own server.
At the beginning of March, I started getting temporary rejections from gmail. Not all of my outgoing messages, maybe 1 in 10. Most of these messages were to one individual, to whom I've been sending for years. My domain has existed continously since 2002, and has never sent spam. The rejection message was startling: words to the effect of "Your message has been rejected because of the awful reputation of the sending domain". The reputation of my domain is spotless, according to various testing tools.
There have been no new rejections in the last two weeks.
According to TFA, Google started rejecting a proportion of mail from bulk senders in February. I wonder if I got caught up in some half-baked roll-out of this new (old) policy.
My "domain reputation" should mean something like "the consensus of multiple reputation services", where those reputation services are reasonably open about how they score. It shouldn't mean "the completely opaque opinion of a single, hyper-dominant, secretive provider with all kinds of conflicts of interest".
Starting February 1, 2024, all email senders who send email to Gmail accounts must meet the requirements in this section.
- Set up SPF or DKIM email authentication for your sending domains.
- Ensure that sending domains or IPs have valid forward and reverse DNS records, also referred to as PTR records. Learn more
- Use a TLS connection for transmitting email. For steps to set up TLS in Google Workspace, visit Require a secure connection for email.
- Keep spam rates reported in Postmaster Tools below 0.3%. Learn more about spam rates
- Format messages according to the Internet Message Format standard, RFC 5322.
- Don’t impersonate Gmail From: headers. Gmail will begin using a DMARC quarantine enforcement policy, and impersonating Gmail From: headers might impact your email delivery.
- If you manage a forwarding service, including mailing lists or inbound gateways, add ARC headers to outgoing email. ARC headers indicate the message was forwarded and identify you as the forwarder. Mailing list senders should also add a List-id: header, which specifies the mailing list, to outgoing messages.
https://apps.google.com/supportwidget/articlehome?hl=en&arti...
Agreed it’s a net positive, but it kills me when the reason emails land in spam is misconfiguration at the recipient’s end. (Like forwarding emails which breaks SPF)
Someone (allegedly) sent SPAM and now my machine that sends maybe 3 emails a week is blacklisted
Any half decent marketer will 100% use a different domain for outbound sales (or any use case where spam rate might be abnormally high).
Email on a personal machine and domain has been dead for over 10 years.
You just can't own your data.
You can receive it, no problem. But you can't send it.
So, they have a "de facto" monopoly, or at best, are working with other providers to create a cartel.
> Email on a personal machine and domain has been dead for over 10 years.
Due to the actions of?
> You can receive it, no problem. But you can't send it.
You can send it. They're actively deciding to just block you. Then provide you no recourse.
> You can send it
Not even that. Nowadays it's not uncommon for some servers to even refuse the connection.
Mind you, I'm not talking about aggressive spammers.
And yes, there's pretty much no recourse.
I'm not nearly as sure. The mechanics of one aren't that hard to imagine. Encourage spammers and make it cheap. Don't ever fight them at their source. Invoke mechanisms that intentionally destroy public utility in public protocols. Force everyone to rely on a small handful of "reputable" senders.
Working backwards from "who decides reputation anyways?" might make it easier to see.
As I care more about the recieving side than sending emails this works well enough for me.
i do think mail servers/services are using ip-based blocklists wrong. yes, you can use it as one of many signals. give it some more weight for first-time senders. but if you've been mailing (with spf/dkim/dmarc-authenticated messages/transactions), from an ip that suddenly gets on a blocklist, the previous positive reputation should be stronger than that blocklisting, and you should be able to keep communicating with your known correspondents (until they mark your message as spam after which future deliveries can be rejected/junked). it seems those mail servers/services cheap out and apply ip blocklists early in the smtp process. good for their system load, bad for their analysis performance.
in general, it seems even bigfreemail services are bad at using existing reputation in their ham/spam decisions. i recently switched an online webservice i made (that is about sending certain notification by email) to signups via email (like how you can signup to mailing lists: by sending an email to an address). the idea: if you send my service an email, i'm in your list of known correspondents. so the confirmation reply from my mail server (spf/dkim/dmarc-aligned) should certainly be accepted by yours (you much more opt-in do you want?). i tested with some bigfreemails, and yahoo put my reply with confirmation (that even references the original message) in the junk folder. to people who think you can't compete with bigmail: the bar isn't as high as you may think.
Running an outbound smtp server on a customer ip-range is going to be problematic anyway. All such ranges can be considered suspicious since the spammers who use them don't care about their standing.
But I've had my IP for years and it was originally clean as well.
I want to follow best practices it recently changed p=quarantine to p=none after fear that legitimate emails aren’t passing DMARC despite properly configured DKIM and SPF.
Hell, I would love p=reject but not until recipients fix their incoming mail servers to handle edge cases like email forwarding breaking DMARC
This is the most important part. Exchange (due to its history as an X.400 server, not as an SMTP server) does sometimes mangle the message to the point that DKIM simply breaks. This both breaks origin-incoming and forwarded messages.
BTW, Apple also sometimes mangle messages that it fails DKIM, although I do not know why is this the case (as I doubt they use Microsoft Exchange for their mail service).
for incoming email on mailing lists i'm subscribed to, i don't enforce the dmarc policy. i think this is what the parent post hints at. i'm not sure how easy this is to configure with the various mail server software out there. i'm also not aware how you would configure this with sieve scripts (i looked, didn't find it, but it seems like a basic case).
if you're running a mailing list, hoping for all subscribers to not enforce dmarc policy enforcement doesn't seem like a great strategy.
the forwarding case should be easier to keep working.
that's a rhetorical question - you already answered it.
Easy Unsubscribe: Implement easy unsubscribe options (One-click Unsubscribe). Gmail users have tools to report spam, unsubscribe from unwanted emails and control their inbox experience. If it is too difficult to unsubscribe from your emails, customers will be more likely to flag your email as spam. Additional links provided in the ‘References’ section at the end of this article.
2. At the same time, Apple’s ITP will start removing all the information from the URL and only leave the domain, if it classifies your site as a “bounce tracker”. This means you won’t even know who to unsubscribe on one click! So all your emails will be blocked.
https://getcake.com/apples-intelligent-tracking-prevention-2...
If I forward your newsletter, that’s not a bulk email and it won’t include that header.
This is an important distinction that seems to get glossed over in a lot of the coverage and guides about the recent Gmail and Yahoo changes.
>https://getcake.com/apples-intelligent-tracking-prevention-2...
Your source doesn't actually say that "ITP will start removing all the information from the URL", only that it will "limit it the same way as third-party cookies" and will be "purging website data in such instances".
Basically in this attack, a victim (particularly a business or mailing list or NGO) is sending out bulk emails to which the attacker owns. Even sourcing this out to shady off shore click farms would work too.
Attacker then marks the victim’s emails as spam in Gmail/Yahoo/Outlook. The “AI spam filters” pick up on this new “spam activity” and will then mark future emails as spam or even delete them before reaching real customers.
After a year, company bleeds money on a quarterly basis. Ad departments wonder why there is decreased engagement through email. Technical departments are bamboozled.
Maybe a big company will be able to weather the storm or just ditch email altogether. But small companies would definitely take a hit. Even smaller NGO or political mailing lists would lose donations (assuming email was a significant source of new donations).
Probably a very low vector of attack tbh, but something that has lingered in my mind.
I'm not sure how larger orgs mitigate it.
Postmasters at large ESPs and inbox providers can and will text each other to resolve issues.
This is pretty much how the internet works as well (BGP, etc). It's opaque, but open.
I don't send marketing, I only send transactional emails. Notifications are opt-in, and transactional. Logins / signups are done via sending a TOTP to the email, these too are transactional.
If someone marks notifications of new posts in a subscribed thread as spam... fine, but this is self-solving, as this person will trigger no more email ever reaching them for that email address, meaning they also cannot now sign in to the website, and therefore cannot subscribe to more email updates, or visit things to refresh "last viewed", and so would never be notified again.
Emailing a TOTP as a login has increased deliverability by self-selecting the removal of those who hit the spam button. Deliverability of email from the webforums is over 99%.
Reporting email as spam, effectively bans oneself from the website. I didn't even need to do anything.
Marking spam may not be immediately be a death knell within Gmail, Hotmail, etc, but because of the potential impact from being marked as spam, virtually everything treats it like a death knell.
This is fine for me, my service is entirely opt-in, and if someone hits the spam button it risks impacting other users who _want_ the email, so I am not bothered that this person effectively unsubbed themselves and killed their account.
- Gmail might be able to detect what's important and not mark it as spam
- If it gets marked as spam anyway, the user can just open the spam folder to use it
- Even if you have high deliverability, that might not translate to it actually getting into the inbox
I do not track "opened" / "viewed" as that is not important to me, these are transactional emails, logins or "someone has replied to your conversation".
1. Don't trick people into signing up for mailinglists.
2. Don't spam.
3. Don't use mailinglists.
My small business is fine.
A clickfarm marking password reset emails as spam could create real harm.
Even a malicious individual doing this to 2-3 fair transactional emails could cause damage for a small business with low volume.
However of course, most small companies have no idea how to do SPF/DKIM/DMARC.
If I attempted to create dozens of accounts on a website, trigger password resets emails, and report them as spam, where would DMARC prevent me?
Might be a drop in the bucket, but it doesn't take many votes to make a difference in the spam world.
I'm sure this will evolve soon enough and email delivery might increasingly become pay-to-play with all sort of backroom agreements, if it isn't already.
Spammers want us to think there’s a significant difference between their newsletter or marketing notes we may have technically signed up for (certainly not willingly) and I don’t feel bad about reporting both of them. If this forces spammers to consider whether recipients will want their messages, good.
I'm not sure how you could "unwillingly technically sign up" for something like that, especially at the scope of an MLB team which is going to have a team of lawyers, marketing policies, etc. They're not just going to spam people the risk is way too high.
Did he? The anecdote here is probably observed by everyone on this forum. How odd that you find it's unlikely to receive spam from a business transaction.
> I'm not sure how you could "unwillingly technically sign up" for something like that
Have you tried using the internet?
> the scope of an MLB team which is going to have a team of lawyers, marketing policies, etc. They're not just going to spam people the risk is way too high.
I would love to live in your world where there's little likelihood of getting spammed just for purchasing something once. Unfortunately, spamming people has effectively zero risk and all reward. If there were any real risk then we would see actual real and frequent consequences every day. We don't see that, but we do see lots of spam in our inboxes.
[1] https://www.wunderkind.co/how-it-works/performance-marketing...
* wording on check-boxes being sufficiently ambiguous that yo have to think to get the right meaning
* check-boxes on forms in different stages of the ordering process that have inverted meaning from each other
* resetting check-boxes to the “please send me junk” state if the user goes back to correct some other issue or otherwise update the order
* wording the checkbox text to imply that opting out also opts out of getting order update emails (or in some cases opting out does opt out of order updates messages too, and you have to check via some other method what the state of your order is)
And so on.
It is easy to avoid all these accidental opt-ins individually, but it is also very easy to miss one in amongst the sea of dark patterns and one is all that it takes. You need to be vigilant all the time, they scummy marketers only have to get lucky once.
Furthermore, I'm sure some just ignore your opt-out and send crap anyway. When taken to task they'll say you must have accidentally [un]checked the wrong box (how do you prove otherwise?) and tell you that you can opt out from future emails by following the link in the messages.
My point is Google takes content from Column A and puts in in Column B and collects money in their little toll booth. Google pretends to care about spam only when they can't make any money from it, otherwise they don't give a shit unless its a PR issue. Google ads have long been a malware vector - but they don't care because they make tons of cash from it.
I'm not taking the side of either the business or the user here, but this is a forum for start-ups who often have to send out legitimate business emails - which are not necessarily spam.
If those "legitimate business emails" can be put as Gmail ads, then they're not legit business emails, they're spam.
So an attack like this would be very obvious very quickly, even leaving aside that we'd notice a huge spike in email sign-ups and probably kill their accounts (especially since they're not going to be buying anything from those sock puppet accounts!).
Those services will do a lot of checking for you and investigating this kind of weirdness/attack is exactly why you pay them money!
If they're sending directly, then they'll definitely be as vigilant as us, or they'd never get emails into people's inboxes!
It's honestly really bad no matter which route you choose, which is why everyone is so intent on getting you into their mobile app that can spam you without affecting their reputation.
Also, even if marketing emails are sent from a separate domain, sales people can be sending so much "spam" that your normal humans-at-the-company emails start getting rejected. I think this might have happened at my last startup; email reputation got bad, interview candidates didn't get their interview conformations anymore. But I might be misremembering.
Now you have me rooting for the bad guys.
Edit: USian speaking
Meanwhile the first victim is left to pick up the mess where none of their email gets through anywhere.
Every couple of months I end up checking its spam folder and it's just a daily barrage of spam from both DNC and RNC, 1-2 emails per day like clockwork. None of them ever got through to the inbox though.
I’m left feeling like homegrown email delivery is some sort of lightning rod for stuck-in-the-past faux-sysadmin types that can’t get past the fact that it’s not 2003 anymore and lazily / maliciously comply with SPF / DKIM.
IT’S NOT THAT HARD.
Well-established organizations that have a long history of sending steady volumes of high quality content with low complaints have a huge leg up. So if you work at such a place, or you contract with such a vendor, you’re going to feel like it’s obvious that the DNS entries work well.
What type of "FUD" are you talking about? The objections in the thread seem pretty well founded (eg. being shadowbanned despite complying with SPD/DKIM, or this requirement breaking email forwarding), and there aren't really any that are against implementing SPF / DKIM.
A problem is that you can do everything technically right, and sitll land in spam, because some big players don't play by the usual rules.
For example, Microsoft apparently has an allowlist for IPv4 -- or equivalenty, blocks all IPv4 by default, until you manually de-list them at sender.office.com. At least I haven't found an IP yet for which I didn't have to do that (self-hosting email for 15 years).
(Imagine every provider did it like MS; you'd be sitting there and filling out web forms with 1000s of providers.)
So you have a technically perfect setup and MS stil rejects you.
--
That said, using some provider to send emails for you doesn't solve deliverability either. There, many customers share the same sender IP. If one of them sends marketing/spam, the entire IP gets bad reputation. In such cases, providers recommend to upgrade to "bring your own IP", which then needs to "gather reputation" [0]. Great, might as well have self-hosted in the first place, as repuation is the only thing I bought the service for.
[0]: Example: https://www.mailgun.com/blog/deliverability/dedicated-shared... -- Especially entertaining is "Use a shared IP if: Your shared IP partners have built a good reputation." As if you had any control over that.
That applies to really all email providers. Part of fighting spam is (somewhat unfortunately) not telling the spammer what you're detecting.
EDIT: Admittedly this post is also about bulk sending where other metrics like unsubscribe links and spaminess matter. But for the self hosted crew it really just comes down to DMARC, DKIM and SPF.
[1] https://tschumacher.net/self-host-email/
The most annoying this is when email senders use click tracking on domains that are blocked by those AdBlock lists. I keep a separate browser instance to copy-paste those links into, but then I have to login again.
I prefer sending unsubscribe emails instead of clicking links. Gmail can automate it.
When you forward an email, unless the email forwarder modifies the message content, it should still match the DKIM signature, so it still passes.
"In April 2024, Google will start rejecting a percentage of non-compliant email traffic and gradually increase the rejection rate. For example, if 75% of a sender’s traffic meets our requirements, Google will start rejecting a percentage of the remaining 25% of traffic that isn’t compliant."
https://dmarcian.com/yahoo-and-google-dmarc-required/
Untrusted forwarders (also sometimes known as open relays) have been heavily frowned upon for a really long time. SPF has never worked for forwarders that don't do rewriting. Those letters have always been negatively looked upon, unauthenticated mail is a remnant of times long gone.
Google's policies about authentication are the bare minimum you should be doing anyways if you care even a little bit about your customers or recipients in general.
Because as you say, you break SPF if you don't touch the envelope from. There's SRS to fix this. But if you do that, you break DMARC alignment.
So... something is always broken? It appears to me there's no way to make mail forwarding that works with full SPF/DKIM/DMARC setups.
What exactly are they doing about that ?
plenty of bigfreemail spam is actually sent from their network, they're an interesting target for spammers, and at least some of them put a lot of effort in preventing abuse.
Nothing at all, because they are too big to care.
Aggressive decentralization is the only way to save the Internet. Host your own email, get everyone you know to host their own email.
At my first corporate job, and my second one as well, all email was sent and received directly from/to each persons individual workstation. There was no concept of centralized email server.
(Let alone outsourcing it to some other company, that would have been completely unthinkable.)
The result is that outgoing hotmail/outlook smtp servers are added to blacklists until they start content filtering their own users.
Their support people blame me, although they admit others have the same problem. They're not using a mail delivery service - the emails come directly from an IDrive server.
They're sending to my web site, which forwards to my personal address. There's no filtering at the first stage, and a division into Accept/Greymail/Junk at the next stage. Neither Google nor Yahoo is involved at any point.
I use seperate emails for each service much like a seperate password so I'm heavily invested in keeping my own server at least for recieving.
1) Blocking numbers (which is pointless because they rotate them and spoof residential numbers).
2) Whitelisting numbers and blocking everyone else (this would cause me to miss legitimate calls).
3) Blocking entire ranges (this doesn't work because of the spoofing).
4) Using one of those spam screening services (currently looking into this). Still a bit concerned about missing valid calls and the privacy issue with this.
I’ve been wondering if this was the cause. I don’t send out 5000 emails (I’m not 10X). But there’s this part:
> While these guidelines primarily affect bulk senders, senders with less volume per day can also be affected if they are not adhering to these guidelines.
I haven’t looked into it yet but I guess I should.
I use my own domain and I’m hosted by a not-Gmail provider.
https://github.com/trusteddomainproject/OpenDKIM/issues/186
OpenARC/OpenDKIM don't parse email headers to spec. Help wanted.
The domain name in the From: field in the email envelop header is inspected and aligned with other domains authenticated by either SPF or DKIM:
The envelope does not have any header, the headers are in the content/body of the email. Also your screenshot of the "Here’s an example email envelop from an organization that passes all of the email security guidelines:" are the mail headers and not the envelope information.
Great presentation on this topic from dmarc.org
https://dmarc.org/presentations/Email-Authentication-Basics-...