The whole point of the article is that Google's sender identity scoring system is more strict when the sending IP is an IPv6 one. That's a pretty clear cut link to IPv6, no?
Google is really strict in general about these kinds of things. I had to go a few rounds with my VPSes before emails from them would consistently not end up in spam, but looking at the headers I'm mostly using IPv6 so I don't draw the conclusion "don't use IPv6" just "if you have IPv6, which is more likely than not now, be careful and read the docs"
Let me rephrase: the whole article is equally valid for IPv4. Being more or less strict is a claim the article makes without proof. And as far as my experience goes, there is no difference.
So standing by it: the article has nothing to do with IPv6 per se.
Huh? I'm not even sure how you came to that conclusion.
Gmail is probably tougher on mail servers using IP6 addresses because they're plentiful and I suspect spammers were having a field day setting up temporary mail relays forcing google to play whack-a-mole.
I used to run my own email server years ago but spam and spam protection measures have made it time consuming and annoying. I'll leave it to the professionals.
Gmail has stricter requirements for v6 because they assume if you are using v6 then they can leave aside all the bozotic baggage they allow for legacy v4. In particular, your sending IP must have a PTR record and the name in that PTR record must have a AAAA record containing that address.
This is mentioned in the 550 message, but I guess people don't read the logs.
> Gmail has stricter requirements for v6 because they assume if you are using v6 then they can leave aside all the bozotic baggage they allow for legacy v4. In particular, your sending IP must have a PTR record and the name in that PTR record must have a AAAA record containing that address.
This is, in fact, quite difficult to convince your ISP they should do.
Round-trip DNS consistency is pretty basic and my ISP provides it by default. Note that I didn't say your ISP has to delegate to you, or that your PTR record needs to match your EHLO or anything of that nature.
Are IPv6 numbers not available from your local registry at low or no cost? Or in these late days must you rent them? Or is this just a problem of people not wanting or knowing how to run BGP so they can control their own fate?
Run BGP... with whom ? Let's say I have my own AS and a prefix given to me, which is indeed quite open and not too costly. If my ISP coming to my facility won't allow me to set a PTR record, they for sure won't allow me to run BGP session with them !
There are VPS providers out there (like Vultr) that offer BGP. You can either use the IPs on their cloud instances, or tunnel IPs back to your home network, etc.
You're supposed to get your IPv6 prefix from your ISP's allocation, not directly from the regional registry. Simplifying the routing tables with hierarchical address assignment was one of the major selling points for the larger address size in IPv6. If everyone gets their own prefix independent from their ISP then the core Internet routing tables will continue to grow ever more complicated.
> This is, in fact, quite difficult to convince your ISP they should do.
For server hosting, I have never seen a provider that doesn't allow me to either set the PTR record or at the very least keep it set to something that resolves back to the IP address in question.
For residential ISPs however the story is different - but who would want to send emails to googlemail from a residential IPv6 address without authenticating themselves? Only spammers would.
"Authenticating themselves" to who? This, to me, is just another way of saying "residential users must use an approved sender and can't send mail themselves" since residential users will statistically never be able to get PTRs changed to their domain and instead will have to deal with them pointing to the ISP.
It's another way to force people to use the handful of approved providers to send mail and it's really shitty.
I'm sure you'll disagree with me but if you're running a computer with an ephemeral address it's very very likely that you're not intending to send mail but instead it's a malicious program unknowingly installed flogging mail, and if you're ISP doesn't provide a static address with PTR chances are that they're also negligent to the point that they're blocked by even the smaller providers simply due to too much spam.
> It's another way to force people to use the handful of approved providers to send mail and it's really shitty.
You can always go and rent a server somewhere in a random datacenter, the lowest of the low VPS providers are at ~5€ a month, and send and receive mail from there. Hardly a "handful of providers".
There simply is no alternative to banning sending mail from residential IPs.
Accept everything by default and allow recipients to choose filtering strategies. Be explicit on reasons and behaviours for blocking.
Yes, 90 percent of users might pick some easy default that blocks you, but that moves the control back a notch from a web of opaque and deliberately hostile systems.
I was never particularly annoyed by spam. But I was deeply annoyed to find that important, time-sensitive work-related mail is merrily flagged by work-provided GSuite, and the best I can do is to try to Rube Goldberg some rules to force it back into the inbox. Who knows if there's other important stuff that rejected before even reaching "route to spam"?
> Accept everything by default and allow recipients to choose filtering strategies.
Yeah, I mean, that's never going to happen. You can only suggest this because you haven't seen email from the inside. That stuff in your Spam folder is the cream of the crop, the 1% of spam directed at your account that the classifier wasn't really 100% sure was spam. The rest of it was blocked at SMTP time without you ever seeing it. Letting all of that stuff get delivered would 1) cost a fortune, and 2) overwhelm the user who is in no way prepared to choose how to classify the resulting deluge.
GSuite users who find their own internal traffic in Spam folder are often using external systems to route and process mail, and those external systems often have jacked up DNS records or poor IP reputation, and they screw up the message by reformatting it or adding dumb footers and signatures. Consequently the mangled messages look sorta spammy.
GSuite also allows corporate IT jerks to blacklist words and phrases and reroute "objectionable content" to the spam folder. This is a very common problem and the solution is to fire the IT guy and delete the content policy.
If either of these are happening to you, talk to your GSuite admin.
Well, it wasn't internal traffic, it was communication with a third-party service provider we integrate with, which seems to be maintained by bargain-basement outsourced workers. I'd be unsurprised if their mail server had deliverability problems. Hell, their API test environment has been broken for weeks at a time.
But fundamentally, "deliverability problems" feels like we lost something about the promise of email. I don't have to talk to my postman and tell him "BTW, I'm expecting a box from Mouser next week, make sure you don't yeet it straight into the dumpster."
I suspect filtering is a S-curve thing: you set up a handful of fairly simple rules and get up to 80 or 90% rejections, and then you can spend the rest of your life taming the rest.
But part of the point is to restore some transparency. If you know what filters you have, you can refine them in a better way to get the outcomes you want. Maybe that default block for "Canadian Pharmacy" isn't so useful if you're a medical student looking into job opportunities in Quebec. Being able to manually isolate, understand, and manage the rules is much better than doing rituals to appease the Almighty Algorithm.
The 550 messages are quite clear, but Gmail has the nasty tendency to accept email and then flag it as spam, stuffing it away in the spam folder to be automatically deleted in 30 days.
Requiring SPF/DMARC/DKIM/PTR shouldn't really be a problem, but there are extra layers of spam filtering on top of the problems Gmail will give you feedback about.
> Requiring SPF/DMARC/DKIM/PTR shouldn't really be a problem
Oh I wish that were the case. One recent "lovely" example I stubled upon is Deutche Telekom (t-online.de) not willing to use SPF because it's not perfect enough for them.
This isn't specific to Google, but IP reputation was used heavily for IPv4... but there either isn't enough reputation for IPv6 or the systems haven't yet scaled to make sense of it (what works for the IPv4 address space doesn't necessarily work for the IPv6 address space, i.e. storing and looking up reputation and the slow changes in IPv4 ownership and the rapid burn-through of IPv6 ownership).
What people on IPv6 are getting is one of two things: 1) Harsher defaults or 2) No reputation system applied.
In the case of Google for email, it appears that it's harsher defaults.
My guess is that the defaults are designed to block a similar amount of spam if possible?
Most major mail providers have access to one or more ipv4 addresses. This allows for reputation + it's a bit more wasteful to burn through ipv4 blocks on spam campaigns.
ipv6 address are plentiful, so you can burn through them on a spam campaign.
IPv6 addresses are more plentiful. However, you don't assign reputation to an individual IPv6 address but rather to an IPv6 network prefix (a /64, or even /60). That gives roughly the same granularity as a single public IPv4 address: one ISP subscriber.
Naturally there are far more IPv6 /60s than IPv4 /32s. As a spammer, though, you don't get to pick just any /60 prefix you want—it has to be allocated to you.
So what if we just change email to work like this: Mail server will not let incoming messages through unless you've first A) sent a message to that domain AND B) received a reply that doesn't look like an error or bounce.
Yes, you'll have to send an email somewhere with little to no content to initiate reception of things like invoices, password resets, and monitoring emails. Is that such a bad thing?
Yes, spammers will desperately try to get you to send an email, any email, to their domain, or cybersquat on common typos. These problems might have easier and better solutions than letting tech giants takeover a formerly open Internet protocol that anyone should have the ability to use.
That sounds a lot like the idea from Peter Huth’s Mail-Terror-Blocker PRO, a theater drama studied guy that happened to become an _internet expert_ by accident here in Germany back in the 2000s.
Unfortunately, his program failed in the most hilarious ways and quickly became a joke in the usenet groups back in the day.
Nice framing. However, I was rather thinking about the _drama_ he created back then with his software due to a logical error. To me it had some resemblance of a nice play at the time.
That's sort of how Gmail does work in my experience. Once there is evidence of an account replying to email from a "less reputable" domain / IP, they tend to let all follow-up correspondence through.
The problem is that people managing their own server wouldn't be able to initiate conversations with people who give them business cards, for example.
The annoying thing about the major email services (Gmail, Outlook, et al) using reputation as a key input in their spam prevention algorithms is that they don't handle reputation in a way sympathetic to small and hobbyist mail server operators at all.
Google and Microsoft have the computation and data storage resources to forever record how much spam they've received from the mail servers operated by any given small company or hobbyist. If a domain's authoritative mail servers have a history of never sending spam, when that company or hobbyist (for any reason) has to switch hosting providers and gets a new IP address, Google/Microsoft should recognize that the domain, and its referenced mail servers, has never been malicious.
But instead, the majors treat a new IP address for a long-established domain as entirely new to the internet and assign it zero, or even negative, reputation. And they don't care to address this because, well, why care about hobbyists and the rare small company insolent enough to try to self-host email?
Indeed, although I think there is a little more rationale than simply not caring. The hobbyist/small mail server is historically more likely to go unpatched and pwnage is more likely to go unnoticed, so it’s unfortunately a good defensive move to penalize them reputationally by default.
PS yahoo is the worst for assuming poor reputation from a sender. In my experience they just introduce massive delivery delays at the drop of a hat even when the sender has a stellar reputation.
> The hobbyist/small mail server is historically more likely to go unpatched and pwnage is more likely to go unnoticed, so it’s unfortunately a good defensive move to penalize them reputationally by default.
But to reiterate my earlier point: Google and Microsoft have the computational and storage capacity to have a detailed history of all domains' authoritative mail servers behavior. They will know whether a domain has a history of patch negligence.
No, I think the far simpler explanation is that they just don't bother tracking reputation by domain. Or if they do, it's largely overshadowed by the weight given to IP-based reputation.
> I think there is a little more rationale than simply not caring.
You're right in one more way: both Google and Microsoft offer email hosting solutions for small businesses. Their main (and, if you've got more than 8-10 accounts or so, the only) selling point is that it makes managing email hassle-free. Making it as painful as possible for small businesses to host their email server helps these services tremendously. If they had real interoperability (either of their own accord or because it were forced upon them through regulatory measures), the biggest cash cows of these services -- companies that are well into "enough accounts that your own server would be much cheaper" territory but not large enough to afford or risk large infrastructure changes -- would evaporate pretty quickly.
This seems like a major legal liability for Google. It could be shown that Google and other major email hosting providers act like a cartel by unfairly discriminating against companies who don't use a major email provider.
I hope so! I am tired of explaining people that it's actually Google's fault they didn't receive my mail. I'll be happy when they pay dearly for the disservice they do to e-mail.
Last time I looked into it (I run a mailserver and mailman list for one of my hobby groups), S/MIME wouldn't change your "spamminess" reputation score.
DKIM, DMARC, and SPF do, though, and basically are table stakes if you want your mail (especially mailinglist messages) to go through to people at major providers.
No, would that help? I am however using SPF and one of DKIM / DMARC (I forget which). But anyway, I can live with a mail missing here and there. It is just annoying and isn't right.
It's not really Google's fault to be honest. The need to warm up new IP's has existed for a while and a lot of providers do it. Any postmaster with experience knows how and why it's done.
The root comment literally explains that that's BS.
Every post master knows that it is done. But it doesn't have to be that way, although it certainly can feel like it when a company like google decides to not budge on the matter.
DKIM in no way helps to get past the cartels' "reputation" filters. I send maybe one email every few months to microsoft accounts & it's always received as spam. My server setup & ip have been solid for a decade. It's only ever the globalist providers that block me. Google is 50/50 I get through. Everyone else (eg Protonmail) is no problem.
It absolutely does. Also that was an "if, then potentially" sentence about reputation tracking in the future.
In your case, it's likely that your volume and sending patterns aren't consistent and trustworthy enough to keep track of your domain and IP reputation.
You have to understand that they get millions of letters from new domains each day, sent from compromised Wordpress blogs and the alike. If you want to be deliverable, you have to be consistent and not suspicious.
Or, more likely, there's some other mistake in your configuration somewhere.
The need to warm up new IP's has existed for a while and a lot of providers do it. Any postmaster with experience knows how and why it's done.
I think the point people are trying to make, and I'm sympathetic to, is that if an ultra-low volume email poster, with a full-set of SPF DKIM and DMARC credentials configured and zero history of sending spam - that the majors (Yahoo/Google/Microsoft) could start off by not sending email from that domain immediately to spam, just because it isn't a well established and trusted IP address.
Alternatively - come up with something akin to D&B registration system so people can attest that they won't engage in spammy behavior.
> I think the point people are trying to make [...] not sending email from that domain immediately to spam, just because it isn't a well established and trusted IP address.
Yes, and I'm saying what's the prequisite for that to happen. As long as it's okay (which it currently is) to send unsigned mail, IP addresses have larger weight. DKIM needs more deployment for that to change.
There's absolutely no way that IP-based reputation schemes will be deprecated before alternatives are viable. Sure it would be nice for a few people here, but no, won't happen before the ecosystem improves.
> Alternatively - come up with something akin to D&B registration system so people can attest that they won't engage in spammy behavior.
It's cheaper and easier to munch through lots of throwaway domains than to keep moving IP neighbourhoods, isn't it? I don't know - is free domain tasting still a thing?
If you filter by IP block (or address!), it might be a block that has changed hands and is no longer spammy. Or it might be a block from the Zen Policy Blocklist, which blocks ranges that the responsible ISP has submitted as domestic or retail blocks that are supposed to send outbound mail through the provider's smarthost.
If you filter by domain, that could be the envelope sender, the From:, the Reply-to:, or the domain of the SMTP client. Only the last is reliable; and you also have the IP address for the client. In my experience, the IP address is more useful, for longer, than the domain name. But any good blocklist should age quickly (i.e. old stuff should drop off the list).
Even if you have DKIM, SPF and DMARC all set up, at least Microsoft still seems to give a decent weight to IP reputation and assign a negative reputation to unknown/low use IPs.
Absolutely, my second sentence says how that could change.
I would have thought that it's fairly self-explanatory that anti-spam measures utilize the strongest signals. If sender domain becomes that, it will get more weight.
So if in the future email providers could reject both SPF-less domains and DKIM-unsigned letters, IP's would definitely become less relevant. So, get people to deploy those things.
Sorry, but this is bullshit. If I employ relevant techniques to protect my domain and the protection works, and I am not sending spam (which I am not), then Google (no idea what MS does) should not care about the IP I am sending my e-mail through. I have proved that the e-mail is tied to my domain and they know that my domain is not spamming - what more do they want?
Even better - Google could help small mail server admins by actually providing the information that landed their e-mail in the spam folder. If the protection is tied to the domain, no spammer will be helped by this knowledge.
And I understand that maybe a new domain might be suspicious at first, but after a few years of unchanged ownership (backed by whois data) there is simply no reason to put any mail messages from these domain to spam. Whatever the IP is.
Stop making up excuses for them. They are negligent at best, malicious at worst. Can't wait till they get hit by a lawsuit over this.
> If I employ relevant techniques to protect my domain and the protection works, and I am not sending spam (which I am not), then Google (no idea what MS does) should not care about the IP I am sending my e-mail through.
They definitely take the IP less into account if other things are more trustworthy. Totally ignoring it would be short-sighted from them. There are many cases where the domain is fine but looking at the IP and its usage patterns helps prevent abuse. Be it misconfigured (and then abused) SPF, stolen DKIM keys, public website that's email-capable getting compromised, these things happen a lot.
> Even better - Google could help small mail server admins by actually providing the information that landed their e-mail in the spam folder. If the protection is tied to the domain, no spammer will be helped by this knowledge.
They have a significant amount of content-based filtering, knowing that helps spammer reword their crap and bypass those.
> I have proved that the e-mail is tied to my domain and they know that my domain is not spamming - what more do they want?
That's also part of the thing, you can't prove and enforce this for both envelope and header from at the same time. Not to mention how minuscule the amount of perfect and strict SPF+DKIM+DMARC is out in the wild. At this point in time IP's are a very strong signal.
> Stop making up excuses for them. They are negligent at best, malicious at worst.
I haven't made a single excuse, I'm explaining why things work the way they do. You calling it bullshit won't make it so.
Google has more than adequate capabilities to handle spam, as several comments point out above. Also, Hashcash[1], which is a virtual silver bullet for email spam, especially when combined with existing trust mechanisms (i.e. the less reputation you have, the harder the Hashcash challenge).
The problem is not that the level of strictness that Google is applying is necessary to reduce spam; the problem is that Google doesn't care and is in fact incentivized to not accept email from non-Google domains.
> Google has more than adequate capabilities to handle spam, as several comments point out above.
Yes, and sender reputation is part of it :D that's why they can handle it.
> Also, Hashcash[1], which is a virtual silver bullet for email spam, especially when combined with existing trust mechanisms
Who do you think has to and can spend more compute or money, well-trusted Google or a small player? Far from a silver bullet.
> i.e. the less reputation you have, the harder the Hashcash challenge
Who do you think will have better reputation? I won't even start at how different people define spam. Though, I can promise you that on average SpamChimp would get to send a lot of spam for much less "HashCash" than a small player.
> is in fact incentivized to not accept email from non-Google domains.
Either botnet devices can send tens if not hundreds of letters, or it's going to be slow and/or expensive for all the legitimate small senders. You really can't have both.
Let me try to explain. If I'm a legitimate small sender of email, then Gmail requiring that my email server solves a cryptographic challenge (Hashcash or a more modern version of it) that takes a minute to compute isn't a big deal if I only send 5 emails per day. All of my outgoing emails are going to be delayed by a minute which won't affect me at all, even though it's computationally slow.
On the other hand, spammers are likely sending hundreds of emails per minute per server, so limiting them to 1 email per minute would have significant consequences for them.
I hope this demonstrates the disproportionate effect Hashcash (or modern equivalent) would have on spammers when compared to legitimate senders?
We use exactly the same idea for secure password storage. Password verification is slow, really slow, on purpose. This makes it "slow for legitimate password verifier" (the website you're logging into), but it also makes password cracking on a large scale completely infeasible.
> I hope this demonstrates the disproportionate effect Hashcash (or modern equivalent) would have on spammers when compared to legitimate senders?
Yes, in a scenario where you're a legitimate sender that sends very few letters compared to a bulk spammer. Apples to oranges.
You unfortunately ignored both the low-rate spammers and the high-rate legitimate senders. The former is barely hindered and the latter is PoW-walled into inconvinience.
If you now bring in reputation schemes, then those can be played by adversaries, you'll end up at step zero but you have made a large climate impact.
> but it also makes password cracking on a large scale completely infeasible.
But this is not the current scenario. Intentionally complex password hashing is not really PoW we're talking about.
> You unfortunately ignored both the low-rate spammers and the high-rate legitimate senders.
Both of these seem like imaginary edge cases that don't really exist. I'm going to gloss over the oxymoron of low-rate spammers.
> If you now bring in reputation schemes, then those can be played by adversaries, you'll end up at step zero but you have made a large climate impact.
How do they do that? If we focus on the current state of things, high rate legitimate senders already have an established reputation so they don't need to change how they handle outgoing emails.
The only thing that would change is how large email providers such as Gmail handle incoming emails from small/unknown senders, by accepting a sufficiently complex PoW as an indication that the email is not spam. Over time this should increase the sender's reputation to a point where PoW is no longer required.
TLDR: Nothing would change for the big players, small senders are given an opportunity to prove they are legitimate, and spammers remain locked out.
> Both of these seem like imaginary edge cases that don't really exist. I'm going to gloss over the oxymoron of low-rate spammers.
I'm sorry? Both of those cases are very real and happen often, you don't get to just ignore them because of your lack of experience.
> How do they do that?
Many ways. Sending legitimate mail for example. You're acting like it's hard to show good behaviour for a while. It's one of the telltale ways I catch those fucks, they're doing it already.
But they also use their own inboxes to build up volume, hijack accounts, abuse SPF policies and DKIM misconfigurations. There probably are more ways to get a "trusted" domain I can't recall this instant.
Things regular senders can do, spammers can do, but they'll spend more effort to do it at scale!
You have demonstrated repeatedly how you have little knowledge of spammers' and others' behaviour, you handwave away all "edge cases" and say it will work. No, it won't.
All you'd add is wasted compute resource and time. Compute and time that especially spammers get for free by abusing others.
> Both of those cases are very real and happen often
"Often" but as a tiny fraction of all sender cases, something that anyone who's actually worked with email spam would know, so you either dishonestly withheld that fact or you don't know what you're talking about.
Also, low-volume spammers are quite clearly an oxymoron - the definition of "spam" includes high volume. It's literally high-volume by definition: "unsolicited usually commercial messages (such as emails, text messages, or Internet postings) sent to a large number of recipients or posted in a large number of places"[1]
And, the vast majority of individuals do not send high volumes of email. This is a fact. It is an edge case.
> because of your lack of experience
You seem to be making a lot of arguments from authority, yet you don't even have the credentials to back them up.
> You're acting like it's hard to show good behaviour for a while.
They're definitely not. It's crystal clear that the usefulness of Hashcash is not dependent on it being hard to show good behavior for a while, and in fact its most effective in the long term.
> But they also use their own inboxes to build up volume
Something that Hashcash solves, because they'll have to spend a lot of cycles solving challenges to send non-spam email to build up reputation repeatedly, because for each "clean" IP/server they'll have to repeat the "reputation ramp-up" that every legitimate sender only has to do once.
> hijack accounts
Literally no anti-spam situation protects against this, so it's irrelevant, and I don't know why you're bringing it up.
> abuse SPF policies and DKIM misconfigurations
Then these need to be fixed. Google indiscriminately blocking smaller senders isn't a fix.
> Things regular senders can do, spammers can do, but they'll spend more effort to do it at scale!
Exactly - that's what Hashcash does, is force spammers to spend more effort on the things that regular people do, because spammers do it at scale far more often than regular people do, and so it introduces a disproportionate cost on them.
> All you'd add is wasted compute resource and time. Compute and time that especially spammers get for free by abusing others.
This conclusively illustrates that you have no idea what you're talking about. Spammers do not get resources for free - they have to either spend time to acquire botnets and hijacked domains themselves, or pay money for someone to do it for them. This is pretty clear to even people who aren't in the industry.
Spamming only happens if its profitable, and the reason why its still profitable is because the cost of sending a spam email is currently low enough. Hashcash directly increases the cost of spending spam email by making it compute-bound on top of needing to acquire domains, IP addresses, and machines, and spending time establishing a good sender record, while (if combined with a reputation system) barely penalizing normal users in the long run.
Let me state it again: every new sender has to go through a reputation ramp-up process, but spammers are uniquely penalized by Hashcash because they have to repeatedly because after they start sending spam mail, they lose their reputation and have to start that process again. Normal users do not have to do this - they start a new server, solve a bunch of challenges, and then they don't have to do that again because they don't regularly burn their acquired reputation.
The fact that you can't actually counter these arguments about Hashcash sounds almost like a spammer who's worried about the deployment of an effective system based on Hashcash.
> "Often" but as a tiny fraction of all sender cases
Not really, no. I mentioned and you're dismissing high-rate legitimate senders and low-rate spammers. The former happens for example when companies send bills each month, they hit very high rates compared to their usual baseline. The latter is not the majority volume-wise, but is the most annoying - fairly logical that some spammers choose to fly under the radar and drop a letter or two in a minute. Do that during the night and you've landed a lot of letters in people's inboxes before you get reported.
> And, the vast majority of individuals do not send high volumes of email. This is a fact. It is an edge case.
It may be an edge case for you, but it must be accounted for. You can't wave it away because then people won't receive their mail. Just one minor example, some less tech-savy users emulate mailing lists with hundreds of people in CC. You can't say there that "oh this is my anti-spam solution but it doesn't account for you because you're an edge case"
> Also, low-volume spammers are quite clearly an oxymoron - the definition of "spam" includes high volume.
There's a difference between total volume and rate. You confusing the two is your problem. Not to mention that a dictionary definition is not the ground truth neither does it define "high volume". I'd also say unwanted unsolicited advertisement sent to 30 people is still spam, but you do you.
> You seem to be making a lot of arguments from authority, yet you don't even have the credentials to back them up.
From experience backed up with explanations, and you're dismissing them based on yours with no explanation.
> Literally no anti-spam situation protects against this, so it's irrelevant, and I don't know why you're bringing it up.
Incorrect. Looking at IP addresses suddenly behaving unusual is a great signal compared to the aggregate of a domain's total.
> Then these need to be fixed. Google indiscriminately blocking smaller senders isn't a fix.
They absolutely should be, but it's difficult and in the end you're not alone on the internet. Until people will, IP's continue to be used for spam classification.
And no, it's not indiscriminate.
> Exactly - that's what Hashcash does [...]
I think you misunderstood again and let me rephrase. Spammers spend more effort to lower effort spent per unit of spam. So in the end they spend less effort per letter than legitimate small senders.
> This conclusively illustrates that you have no idea what you're talking about. Spammers do not get resources for free
Your sentence conclusively illustrates you're clutching at straws. Declaring that something stolen hasn't been gotten for free is a shaky foundation to build your argument upon and pedantry at worst. The cost to acquire those resources is there anyways, some extra CPU they have to steal is not that significant.
> Hashcash directly increases the cost of spending spam email by making it compute-bound on top of needing to acquire domains, IP addresses, and machines.
> Normal users don't have to do that again because they don't regularly burn their acquired reputation.
You're going in a loop again. We're comparing relative effort/monetary cost per email here. If one side has lowered the cost of effort by parallelizing things it's unfair to the side that hasn't. Consider the scenario where a new online store owner starts sending email or someone's script is compromising yet another poorly secured blog, IoT device or SMTP account.
Now a legitimate sender has built up all the reputation, after a few flags by recipients it was spent in an instant. Everything will become slow and expensive for the actual owner, once again. For spammers it was the tenth one that day.
I read this entire tree of comments and I really don't understand why you continue to engage in this condescending manner. If you're some kind of subject matter expert on the topic then by all means, please elaborate on the intricate details and challenges of dealing with spam, from start to finish, as well as any proposed solutions to make self-hosting viable when dealing with providers such as Gmail.
These patronizing "you obviously know less than I do" type comments that occasionally drop a few bits information aren't adding anything to the conversation.
> I read this entire tree of comments and I really don't understand why you continue to engage in this condescending manner.
If you haven't noticed, I heavily copy phrases from the people I reply to. If you don't like your tone being matched, unfortunate.
> Please elaborate on the intricate details and challenges of dealing with spam, from start to finish, as well as any proposed solutions to make self-hosting viable when dealing with providers such as Gmail.
I have, and you included have ignored it as "edge cases" and alike. What can I more explain when someone has taken the stance of being dismissive?
Hashcash is an interesting idea and very creative. But I do think you're right. It may impose reasonable limits on a single computer. But if an attacker has a botnet they can still send out a crap load. Botnets and spam practically go hand-in-hand as-is, so I don't see how hashcash helps there.
> Who do you think has to and can spend more compute or money, well-trusted Google or a small player? Far from a silver bullet.
i'm not even a big fan of hashcash, but i don't think you understand it, or how it could fit in as a component of a broader spam mitigation system. (for better or worse)
google wouldn't need to compute hashcash for their outgoing emails, because they have DKIM and a solid reputation. nor is its computational power somehow at odds with that of smaller players.
> i don't think you understand it, or how it could fit in as a component of a broader spam mitigation system.
It's not that I don't understand, I know how it won't.
> google wouldn't need to compute hashcash for their outgoing emails, because they have DKIM and a solid reputation.
Great, so what's the point? Small players would have to pay Google to deliver mail :D
> computational power somehow at odds with that of smaller players.
As I said in my other comment, either botnet devices can send tens if not hundreds of spam letters, or it's going to be slow and/or expensive for all the legitimate small senders. You really can't have both.
> It's not that I don't understand, I know how it won't.
You really don't understand it, because you're unable to provide counterarguments to specific explanations of how it would fit in with a larger anti-spam system, and because you say things that are clearly false to someone who does understand it.
> Great, so what's the point? Small players would have to pay Google to deliver mail :D
...such as this, which is false. Nobody's paying anybody anything - Hashcash is a computational proof-of-work challenge with zero money transferred.
If you're talking about a metaphorical "payment" of the smaller players having to do some kind of work - that's a feature, not a bug, because it allows the smaller players to invest effort into convincing Google that they're legitimate, and if coupled with a well-designed spam system that then takes that reputation and associates it with DKIM signatures/IP addresses, then the smaller players burn n cycles and then stop having to do so because they now have reputation with Google, while spammers burn n cycles continually for every message that they send because they keep getting flagged as spam.
> As I said in my other comment, either botnet devices can send tens if not hundreds of spam letters, or it's going to be slow and/or expensive for all the legitimate small senders. You really can't have both.
Yes, you really can, because you don't understand how anti-spam systems work. A system with Hashcash would start out assigning most IP addresses+hostnames/DKIM signatures with a reputation of 0, and in order to accept a message, would require the sender solve a Hashcash challenge inversely proportional to their reputation. If the message is marked as spam, the reputation takes a hit - if it's not, the reputation increases slightly (along with all of the other factors that an anti-spam system uses).
The legitimate servers quickly build up reputation and stop having to solve challenges, while the spammers pay the computational tax until the end of time, making it unprofitable for them to send most spam to most targets. The end.
> because you're unable to provide counterarguments to specific explanations of how it would fit in
I have provided specific cases where the previously proposed use-cases wouldn't work.
> they now have reputation with Google, while spammers burn n cycles continually for every message that they send because they keep getting flagged as spam.
> The legitimate servers quickly build up reputation and stop having to solve challenges
So the spammers send a few warmup mails. It's very naive of you to think spammers can't imitate legit sender behavior.
It's also clear you haven't seen or analyzed any significant amount of spam. The end.
> So the spammers send a few warmup mails. It's very naive of you to think spammers can't imitate legit sender behavior.
It's clear to anyone who understands the basics of spam and Hashcash, and can use basic logic, that "So the spammers send a few warmup mails" literally doesn't change the effectiveness of Hashcash.
Let me state it again: every new sender has to go through a reputation ramp-up process, but spammers are uniquely penalized by Hashcash because they have to repeatedly because after they start sending spam mail, they lose their reputation and have to start that process again. Normal users do not have to do this - they start a new server, solve a bunch of challenges, and then they don't have to do that again because they don't regularly burn their acquired reputation.
You say
> It's also clear you haven't seen or analyzed any significant amount of spam. The end.
...but you're unable to use basic logic on publicly-known facts (the vast majority of spammers send large amounts of spam; the vast majority of users send small amounts of email; reputation is hard to gain and easy to lose) to infer the above or generate a logically cohesive counterargument. Every time you can't give an answer, you fallback to an argument to authority.
> Yes, and sender reputation is part of it :D that's why they can handle it.
They aren't doing a very good job of using sender reputation. If you have a domain with a several year record of no spamming whatsoever, and with every outgoing message using DKIM, they will still start blocking you if othersenders who just happen to have an IP address close to yours start spamming.
IP block reputation should only be used to set the default when dealing with new senders. If they have seen enough DKIM signed messages from a sender to know that the sender has a good reputation, IP block reputation should have weight 0 when receiving mail from that sender.
Google could do this without any increase whatsoever in the amount of spam that shows up in their customers' inboxes. All that would change is that their false positive rate would go down.
> A lot of spammers have DKIM, it's not a good reason to allow mail from a suspicious subnet.
You keep missing the point. Nobody is suggesting that mail be allowed merely because a domain is using DKIM.
What is being suggested is that if a domain has a proven history of not sending spam then blocking the domain's mail just because there are spammers in the same subnet results in blocking non-spam. Not blocking the domain does not increase the amount of spam that gets through because the domain isnotsendingspam.
Where DKIM comes in is that it allows receiving sites to distinguish between mail that is really from domain X and mail that is forged to appear to come from X. The latter can safely be automatically classified as spam. The former can be used to build an accurate history for the domain to determine if it sends spam or not.
> What is being suggested is that if a domain has a proven history of not sending spam
That is merely because of using DKIM! Everyone can create a 'history of not sending spam', including spammers.
With your solution someone can rent a subnet, warm up a bunch of domains, but then blast spam from the entire subnet without immidiately affecting the entire set of spam domains.
What you're proposing wouldn't allow trusting even signed mail from an already suspicious subnet. Be the domain with or without reputation, so are the spammers'.
> Not blocking the domain does not increase the amount of spam that gets through because the domain is not sending spam.
They don't know that before they deliver your mail to inboxes and someone does or doesn't mark it as spam.
Sender reputation where you block smaller email providers indiscriminately without weighting DKIM signatures or identity (or just blocking based on IP) is not a way of "handling" it. Your comment is akin to someone saying "the government can handle poverty" and you saying "yes, by killing all of the poor people" - it's very clearly a non-solution.
> Who do you think has to and can spend more compute or money, well-trusted Google or a small player?
I don't think that you understand how Hashcash is supposed to work, or what it's supposed to solve. The computational cost is imposed on non-Google players - both smaller email servers without a reputation (until they build one up, obviously), and spammers. I mean, sure, if everyone implemented the protocol, then non-Gmail servers could technically request that Gmail do Hashcash challenges in order to receive email, but nobody's going to do that, because nobody believes that Google is going to send spam mail.
> Though, I can promise you that on average SpamChimp would get to send a lot of spam for much less "HashCash" than a small player.
No, you really can't promise me this, unless you're literally in control of Gmail servers and intentionally adjust the difficulty settings to fulfill this condition.
> > is in fact incentivized to not accept email from non-Google domains.
> Everyone is.
I don't see what your point is here. The fact that Google is incentivized to not accept email from non-Google domains is not a good thing - it's a bad thing. And, really, the above is false - everyone is incentivized to accept legitimate, non-spam email from all of their customers, which is not the same as what you said.
> Sender reputation where you block smaller email providers indiscriminately without weighting DKIM signatures or identity
That's not true.
> because nobody believes that Google is going to send spam mail.
This sentence indicates you don't know much about who sends spam or not.
> The computational cost is imposed on non-Google players
You're going in circles. Do you not see how that especially is unfair to the smaller players? Spammers have just as much compute, it will just impose an additional burden on the legitimate senders.
So in your scenario a small startup has to wait tens of minutes (bunch of letters in the queue already) to send email confirmation letters. The spammer on the other hand does the same, it'll infect a few more devices, they've exeeded the startup's sending rate.
Alternatively, both the spammer and the hypothetical startup build reputation first. They'll warm up the domain, both get to send more in the same timeframe. So... we're at the beginning again, everyone warms up their domains and IP's like they do right now. PoW hasn't improved the sitation.
> I don't see what your point is here.
Because you haven't seen enough spam. On average everyone is incentivised not to accept mail from new or unknown domains.
> everyone is incentivized to accept legitimate, non-spam email from all of their customers
No shit, but also not the topic. Received mail is not perfect, that's what's the actual topic.
They hurt more than competitors. If anyone from gmail is reading and gives a crap about the collateral damage from policies like these, they have set me back at least a few months in my attempts to adopt a child. The agency my wife and I had been getting licensed through dropped us, citing lack of responsiveness in completing paperwork. But I've been e-mailing them for months with questions clarifying how to fill out unclear forms and they've been ignoring me. I presented them with a history of all the sent e-mails they never answered, and even gave my e-mail address in person last time I was there, asking them to add me to an allow-list, but it didn't make any difference.
It's particularly disappointing because even just getting the process started was delayed when no one answered our initial inquiries of interest for nearly a month. That was when I first figured out it was because my e-mails were getting routed to the agencies' spam folders. At this point, I'm probably just going to have to give in and get a gmail address for the sole purpose of completing an adoption, and then stop using it and go back to my real address.
Who'd have thought this would be the thing preventing me from completely de-googling?
My loss isn't as big as this but I've gotten some Paypal disputes because those users didn't get any of the welcome emails (for a rather costly purchase). My issue has been more with Apple's mail for some reason though.
Thankfully all my customers have been super understanding and were kind to take the disputes back once they were able to locate the emails in the Spam folder. Also I finally decided to just use a "reputed" smtp server to fix my problems (feels like extortion but what can you do).
I've written this before, but it's so sad that something so important and vital like email is so broken and we are the mercy of corps like Google/Apple/Microsoft and everyone has their own secret policy.
Hiding messages in a spam folder is essentially ghosting. Email servers should reject messages that they consider spam.
I've run into countless problems because of my messages being marked as spam. The worse is that now it as become socially acceptable to ghost people, I never know who ghosted me, the mail server or the person! I've run into "I ignore your email, get the hint" a few times when thinking I was marked as spam and tried to contact people through other means.
Right, and this is exactly why they need to reject. At least the sender would know and could try to contact the receiver to let them know Gmail/outlook is blocking them.
Right now, when you send an email, you don't know if it was received or not and if the lack of answer is a conscious decision or because of filtering.
Left unchecked I think it might have destroyed the e-mail ecosystem entirely.
Regarding the block vs reject, that unfortunately gives a big tip to the people whose working hours are spent trying to defeat three blocks and send you more spam.
Yes I remember, but this was a different world, it was a world without reputation networks, without ip black lists, with SPF/DKIM.
There is not reason at all to suspect that a message from a small server with proper SPF and DKIM that has never sent spam (for real or spoofed, see SPF), is spam. The other half is from small servers that sprung over night and have no reputation.
Half of the spam I receive is from google/outlook/amazon. They don't block each other, do they?
It’s only dropped if the relay/server sending it is not behaving nicely, otherwise it’s there in quarantine (spam) awaiting user signals to train it.
I think it’s such a typical mindset that people with very limited domain knowledge think that people with a lot more domain knowledge are just being stupid.
I don't care for the idea that any private network should be legally required to accept connections (let alone accepting data) from any other network if they see fit not to. No sender should have the power the force another server on another network to accept whatever they spew out to them.
If google is too aggressive about dropping mail and gmail users can't reliably get the email messages they want, those users should move to a platform that doesn't block those messages. Problem solved.
I've seen several articles in recent months about Google (and other providers) blocking too much, but they are nearly always about the senders and the burdens blocking and sender reputation places on them.
As much as I'd agree that senders have to do a lot to get mail accepted maybe this is working as intended more often than not and the fact that people aren't dropping their gmail accounts in droves seems to suggest that the people whose inboxes are directly impacted by Google's filtering are largely happy with how things are.
Most email is spam, and for the few senders who aren't just bitter that they can't force spam on the most popular email providers with impunity, I freely admit that this is an extremely frustrating experience for them and that it does help degrade the utility of email and contributes to the consolidation of useful email providers. The current situation isn't perfect.
The idea that we could send mail from any routable IP address, or send bulk email easily to anywhere, or even set up our own personal SMTP servers on a whim and still send email to any other network in the world with no deliverability issues is beautiful, but not practical.
Given the decades of scammers and advertisers abusing email, I'd much rather leave it up to the recipients to decide when a service has gone too far in blocking what ends up in their inboxes.
Back when domains were easy to spoof (I could setup a server and send mail as ycombinator.com easily enough) it made sense to track the IPs, but now that you have DKIM and SPF links to cross-check, you should be able to use the domain reliability as a strong indicator. Sure you would have to catch people buying a "good" domain that expired, but that shouldn't be an insane hurdle.
Exactly. Why should the major players care about those too small to matter?
> Sure you would have to catch people buying a "good" domain that expired
Adding to my earlier point, Google and Microsoft are well-enough connected to the domain registrars to know when that scenario has happened as well. If they put any effort into it, they could reliably determine whether a new IP address for an established domain is legitimate or a fraud. But as we've said, why put any effort into it when the only people complaining are not important?
And it's even worse - if we theorize a email competitor appearing out of nowhere to rival Gmail, and people complaining, all that would happen is Google and Microsoft would special-case that provider, and the underlying issue wouldn't be solved.
as far as how to solve this problem technically, I think a reputation system based not on domains or ips but on email certificates is the real answer here.
DKIM is certificates, so I'm guessing you're talking about sender certificates?
How would that help? Spammers can get certificates too. Maybe it cuts down on some of the misconfigured http email senders, maybe, but not enough to matter. Scam sites run https these days.
You can't use like age of activity of the cert to help because a) things get compromised, b) you need to rotate your certs frequently anyway.
Shouldn't the authority (or key) used to sign those certs be long lived? The certificates themselves should be rotated frequently, bit not the key used to sign them.
If the email is DKIM signed, it's expected that the sender was authorized to send the message.
Any wide spread certificate program will just have the email address as the identity, and it will be authorized by establishing control of the email (just like the majority of certificates used for https are domain control only, no organizational verification, not that organizational verification means much anyway). Anyway, identity is hard; there are many people with my name, including a Pulitzer winning author.
Using DKIm you don’t actually have the problem that people can buy so-called good domains, because the system works with private and public keys, so not only do you need a good domain, you also need the private key for that specific DKIM in signature
At the point of receipt, when verifying via DKIM that foobar.com did indeed send this email, then update your spam statistics for foobar.com and you're good.
Isn't the problem more about how to treat brand new domains the first time you encounter them? In order to be friendly to small/new email servers, you would presumably need to initially grant new domains a sufficient reputation for them to send mail reliably. But since domains are essentially unlimited, a bad actor can trivially circumvent your reputation system by spinning up endless domains. This seems like a fairly textbook example of a Sybil attack.
I could bet that most of mails from new IPs are spam because there are much fewer people willing to set up a mailserver than there are spammer operated mailservers. It is cheap to buy a new IP(comes free with many $2 hosting plans). The problem of identifying if a new IP is used for spam is not easy to solve.
> The problem of identifying if a new IP is used for spam is not easy to solve.
How is it not easy to check a reputation database for domains when evaluating a mail server?
Using DKIM records, Google could cross-reference the reputation for the sending domain and, where applicable, recognize that the domain in question has never been malicious or negligent. And in that case, extend a probationary reputation sufficient to allow the IP to establish its own new reputation.
> How is it not easy to check a reputation database for domains when evaluating a mail server?
The point I'm making in this thread is that they have the ability to maintain a database of domains that have been proven to be trustworthy. I am not talking about a database of IP addresses, and I am not sure why that is being raised here.
> The problem of identifying if a new IP is used for spam is not easy to solve.
Huh? With dataset of ~100k classified hams/spams I get like >99% precisison in identifying spam/ham with just bogofilter and 0 heuristics whatsoever (my mail server accepts everything, and I just use bogofilter client side).
I guratanee you MS has a dataset with > 100k emails, lol.
Right. While at the same time as they clumsily block any IPs with unknown reputation, they seemingly cannot reliably flag a poorly formatted, obvious and well-known fake invoice for "Norton 360 renewal" as spam (and really, it's worse than spam, it's a scam).
Why can I identify spam at a glance, with greater than 90% accuracy, from the subject line alone, and to Google it's an unsolved problem?
Google reads and indexes every email that they deliver to a gmail inbox. They probably have the worlds largest and highest paid staff of ML experts, and they resort to auto-flagging based on IP address?
> I don't think you understand how terrible even 99.998% accuracy is at their scale.
What does scale have to do with it, am I missing something? Wouldn’t a spam detection rate of 99.999% mean that only every 100000th spam mail would get through on an account basis, how’s that terrible?
> What does scale have to do with it, am I missing something?
The person I replied to said they can identify spam with 90% accuracy with a glance, my point was that it's abysmal as accuracy. Even orders of magnitude better it'd still be bad.
> Wouldn’t a spam detection rate of 99.999% mean that only every 100000th spam mail would get through on an account basis, how’s that terrible?
I didn't specify if it's per-account or a false positive rate, it's also actually not relevant to the point I'm trying to make.
When you receive billions of letters a month small errors start to matter and they're doing a better job than the person I replied to thinks they do.
You can identify the spam because it's your inbox; you know what your goodmail is supposed to look like. For that reason you can use locally-trained naive Bayes on the client, and get good recognition (you don't need ML).
Honestly, the best filter in my experience is Spamhaus Zen; it spots most spam with very few FPs. It's an IP blocklist.
If you use something like rspamd, then doing ML is not terribly hard. Then it becomes a problem of providing a proper teaching set. Combining RBL's (be it URI, IP or domain WL's or BL's) will help with that.
In the end it's still a hard problem, so takes a bunch of work.
Yes, spam classification is a solved problem. Perhaps even at the level of "this is new IP address with no reputation, is it sending 80% spam or 80% ham?" kind of questions with some defined precision at scale, but ceratinly at personal mailbox level, if you own and train your own clasification engine with thousands of samples of ham and spam you actually receive. That's very effective, and much better than my past experience with gmail.
From the perspective of an email receiver, I WANT things to be difficult for this group. Truly free email available to all would be unusable owing to spam. Oligopoly, I believe, results in the best experience for everyone.
> If a domain's authoritative mail servers have a history of never sending spam, when that company or hobbyist (for any reason) has to switch hosting providers and gets a new IP address, Google/Microsoft should recognize that the domain, and its referenced mail servers, has never been malicious.
That algorithm doesn't work. The internet is filled with parked domains that have "never been malicious". This just creates a new market for clean domains that you can use to evade protections. It makes spam a little more expensive, but it still gets through.
None of these tricks work. There are no tricks. All rules can be gamed. The only thing that can't be easily faked is reality: if Microsoft knows you're a big org with a well-managed IT group running your output email setup, then they know they can (probably) trust you not to spam their customers. If you are too small to prove that to MS in a scalable way, no amount of heuristic trickery is going to help you.
Google--a technology giant with algorithms and heuristics running most operations--is not up to the task of improving algorithms for email server reputation? No, they are definitely capable of improvement. It's simply not of interest to them.
I can hardly blame them because the cost-benefit analysis clearly says, "why bother?"
Of course anything can be gamed, but magnitude matters. I've had the same domain for 24 years, with many hundreds or thousands of email conversations between users of my server and those of the major email players over the years. I had to switch my mail server's IP address two years ago. Immediate zero reputation from many big players.
> is not up to the task of improving algorithms for email server reputation? No, they are definitely capable of improvement. It's simply not of interest to them.
Incorrect, if they stopped spending all that money and effort to keep up, their users would get flooded.
> Of course anything can be gamed
Sending spam/marketing e-mails is a multi-million industry. Both on illegal and legal markets. It's a constant race.
Tough love: neither do our vanity domains (and yes, I have one too, and feel the same pain trying to get gmail to take mail). No one is going to pay someone to dig through our decades of mailing list activity trying to figure out whether to let a half dozen replies through.
This isn't the 1990's anymore, recipient domains aren't just homes to a few hundred undergraduates or a dozen programmers. Email hosts manage communication for hundreds of millions of customers.
FWIW I run a service that sends a decent amount of email to GMail users. While it was a bit slow to get started (messages being marked as spam) once I had sent for a month or two and had a few users that marked the messages as not-spam I don't appear to have any problems. I say this sending from a Digital Ocean IP address that occasionally changes. It appears that Google highly values domain reputation and that once I have got onto their good list I am doing OK.
Disclaimers: I make sure to do everything else right. I have SPF and DKIM and my DMARC policy is to reject 100%. I also don't use IPv6 (DO Kube doesn't really support IPv6 well).
I have found other major providers to be much worse. Microsoft seems to rely almost entirely on IP reputation and marks everything as spam, even accounts that have marked messages as "not spam" many, many times. Apple outright blocks the IP range.
My experience is that all of SPF, DKIM and DMARC are almost completely ignored by Gmail -- one day I simply stopped DKIM and DMARC and Google keep happily accepting emails (and to this day I still send emails without). In fact they will happily accept emails even when the SPF check didn't pass and the policy clearly says strict reject aka -all .
While on the other hand I fully agree with TFA: I have _never_ been able to send an email to Gmail from a IPv6 address and have it not end up as spam, not even to accounts where I already whitelisted previous attempts.
I don't think it's a reputation issue, since my IPv4 addresses likely have much worse reputation. It's as if they just handicap all IPv6 addresses.
GMail definitely respects my DMARC reject policy. But IDK how missing one of SPF or DKIM affect its spam decision. But with DMARC reject and both missing or invalid it will bounce the message every time I have seen.
Automatic mail forwarding without altering the From address will cause DMARC alignment for SPF to break. This is a common enough legitimate setup that most providers seem to effectively downgrade the DMARC policy applied when they see this (usually reject becomes quarantine, quarantine becomes ignore).
My experience with digital ocean was, that their IP subnets have a horrible reputation for email. Google is easy, but try to get unblocked from Microsoft (office 365 or hotmail). In the end I switched to another provider.
DigitalOcean definitely does have a terrible reputation (for good reason). IIUC they used to let anyone send mail and I don't think their current process of requesting unblocking is too strong of a prevention against spamr either. However ideally this would only hurt "first contact" for a domain assuming it is authenticated. In practice some providers will barely consider domain reputation even after a long history.
There was a study somewhere once that concluded that digital ocean is the biggest single source of spam emails. And that the are responsible for around >20% of spam emails worldwide.
I noticed that some providers completely block digital ocean IPs on their firewalls. So no way to get unblocked.
they are basing their filters on the reputation of the netblock that your individual ipv4 /32 (or ipv6 equivalent) is contained in. And very often when you are hosting with a low cost VM or VPS provider the historical record of other IPs operated by the same hosting company in the same /24 or something is very poor.
while I concur with all of the points you make, there is a logical and statistically accurate reason for some of these spam filters.
even if your no-open-relay, rdns, spf, dkim, dmarc, SSL/TLS and other configuration is absolutely impeccable on your smtpd, your only recourse in a situation like this is to change to a new ISP that does not have a poor IP space reputation in all the adjacent IPs.
The netblock thing has happened to me. My Linode that I’ve had sending mail for 6 years gets blacklisted now purely for being on the netblock that it’s on.
Yep. I've encountered the exact same issues on multiple Linode servers in multiple data centers. I've also experienced it with an AWS server. It's often almost impossible to get it unblocked because the support teams at the email providers (if you can even contact them) don't realize that's what's happening. I didn't understand it myself until I got a call with an engineer at Microsoft who realized it in the middle of our call. He unlisted us on the call and we were fine for a while, but were eventually blocked again and haven't been able to get unblocked. It's really frustrating and I'm slowly ending all of my email services (both professional and hobby) as a result.
It’s frustrating. I had outbound smtp throttled right down and a script running that shut smtp down completely if the queue got too big. That triggers an alert and everything gets sorted. But no, we don’t like where you’re from so jog on. Monopolistic.
It’s one thing to advocate not using Gmail for your own mail, and another entirely to advocate not sending any mail to gmail accounts from your own mail server, which is what TFA is really covering.
SPF dkim and dmarc aren't exclusive to ipv6. I think what the author is bemoaning is just how exhaustive the requirements are to send to one of googles cloistered mailservers in the first place.
check senderbase and the rbls, you might be there too if you're on a less gilded cloud provider like oracle or ibm. in general new mailservers take some TLC to repair the IP space you're given before providers will trust it.
One in N of their IPv6 relays is also way more fussy than the others. What’s even better for causing frustrating debugging sessions than nebulous spam fighting rules? Intermittently implemented rules.
Google’s borglike embrace of email is a sad thing for The Internet that slipped quietly by before it was too late.
It’s totally understandable why a lot of mail providers require correct DNS, rDNS, SPF and DKIM. This is standard in 2022. And DMARC is also not hard to implement, it’s just a DNS record.
DNS is not the hard part of DMARC. It’s identifying ALL of the legitimate sources of mail from your domain, and keeping up as they change. Easy enough for a hobbyist, but often challenging for even small business.
I remember that was the first thing I used to do on my mail servers, ever before DKIM etc. Why? I have no idea and I will never know. Is sending email via IPv6 important to me? No. Is mail delivery important to me? Yes. Why is Google such a bad player in this field? Because in spite of having the biggest email network they don't want to allocate relevant resources to handle spammers properly. In other words, they are OK with false negatives, that is us. For this reason, I'm fine with this ugly workaround.
We were lucky enough to have a product idea that sold out immediately after some coverage on a few tech sites (A tiny hardware product for nerds).
Trying to update our customers, we sent out an email to about 500 people via Shopify, the spot the little guys use to do eCommerce. No links in the email at all, just a text based update that we’re here and working hard to keep up with demand.
Ever since then, every email we send to a gmail user (including friends and family) get’s bounced. It’s not even in their spam box!
We have since added google dns txt records and via something they call postmaster tools and we still can’t get emails out.
Just text, no links, no phone number, just a product update from a small company.
There is not a single provider out there that won't punish you for sending bulk email out of the blue. It's unfortunate you found out like that, but you should try a new domain and let the old one cool down.
Gmail does not ever "bounce" mail for spam reasons. It either temp-fails the SMTP conversation with a 4xx error code, or it delivers it, to the Inbox or to Spam as indicated.
If Shopify is transforming a temp-fail into a non-delivery receipt, that's their problem, and yours, but nothing to do with Google.
Have a small mail server, pretty low-volume (max bursts are 40-50 messages to Gmail, global volume to it is around 100 messages per week), delivering to gmail over IPv6. (Note: DMARC, DomainKeys and SPF all implemented) Mails are delivered reliably and have zero problems with it. Had man more issues with Yahoo, Hotmail and Apple, all of which only accept mail over IPv4.
outlook.com is even worse. If you bring up a mail server, you have to apply to them for permission to send them email. Doesn't matter that you have DKIM, SPF, DMARC, or that the domain is 25 years old. They don't care - you must apply through a process that ironically sends email that isn't DKIM certified in their response.
.... Unless you're a kooky spammer re-sending a captured YouTube terms-of-service-change e-mail to the inbox of some Gmail user who bears no relation to the original recipient of that e-mail.
Are there any VPS providers who try and help with mail delivery? Perhaps by actively watching for and shutting down spammers, or investigating and taking action when one of their IP addresses show up on a blocklist? Or something else?
There should be an amendment to the email protocol that routes emails to inboxes if they were paid for by some user-configurable amount. Cryptocurrency would be ideal for this unironically because its basically an open protocol (yeah I know the hacker news crowd hates cryptocurrency but whatever.)
183 comments
[ 0.21 ms ] story [ 265 ms ] threadThere is a comment saying it's still true in March 2022.
So standing by it: the article has nothing to do with IPv6 per se.
Gmail is probably tougher on mail servers using IP6 addresses because they're plentiful and I suspect spammers were having a field day setting up temporary mail relays forcing google to play whack-a-mole.
I used to run my own email server years ago but spam and spam protection measures have made it time consuming and annoying. I'll leave it to the professionals.
This is mentioned in the 550 message, but I guess people don't read the logs.
This is, in fact, quite difficult to convince your ISP they should do.
For server hosting, I have never seen a provider that doesn't allow me to either set the PTR record or at the very least keep it set to something that resolves back to the IP address in question.
For residential ISPs however the story is different - but who would want to send emails to googlemail from a residential IPv6 address without authenticating themselves? Only spammers would.
It's another way to force people to use the handful of approved providers to send mail and it's really shitty.
There are so many and cheap ways to have a matching PTR, so really not really.
The majority of mail from residential ranges is spam and has been for a long time. It's unlikely to change at this point.
You can always go and rent a server somewhere in a random datacenter, the lowest of the low VPS providers are at ~5€ a month, and send and receive mail from there. Hardly a "handful of providers".
There simply is no alternative to banning sending mail from residential IPs.
Accept everything by default and allow recipients to choose filtering strategies. Be explicit on reasons and behaviours for blocking.
Yes, 90 percent of users might pick some easy default that blocks you, but that moves the control back a notch from a web of opaque and deliberately hostile systems.
I was never particularly annoyed by spam. But I was deeply annoyed to find that important, time-sensitive work-related mail is merrily flagged by work-provided GSuite, and the best I can do is to try to Rube Goldberg some rules to force it back into the inbox. Who knows if there's other important stuff that rejected before even reaching "route to spam"?
Yeah, I mean, that's never going to happen. You can only suggest this because you haven't seen email from the inside. That stuff in your Spam folder is the cream of the crop, the 1% of spam directed at your account that the classifier wasn't really 100% sure was spam. The rest of it was blocked at SMTP time without you ever seeing it. Letting all of that stuff get delivered would 1) cost a fortune, and 2) overwhelm the user who is in no way prepared to choose how to classify the resulting deluge.
GSuite users who find their own internal traffic in Spam folder are often using external systems to route and process mail, and those external systems often have jacked up DNS records or poor IP reputation, and they screw up the message by reformatting it or adding dumb footers and signatures. Consequently the mangled messages look sorta spammy.
GSuite also allows corporate IT jerks to blacklist words and phrases and reroute "objectionable content" to the spam folder. This is a very common problem and the solution is to fire the IT guy and delete the content policy.
If either of these are happening to you, talk to your GSuite admin.
But fundamentally, "deliverability problems" feels like we lost something about the promise of email. I don't have to talk to my postman and tell him "BTW, I'm expecting a box from Mouser next week, make sure you don't yeet it straight into the dumpster."
I suspect filtering is a S-curve thing: you set up a handful of fairly simple rules and get up to 80 or 90% rejections, and then you can spend the rest of your life taming the rest.
But part of the point is to restore some transparency. If you know what filters you have, you can refine them in a better way to get the outcomes you want. Maybe that default block for "Canadian Pharmacy" isn't so useful if you're a medical student looking into job opportunities in Quebec. Being able to manually isolate, understand, and manage the rules is much better than doing rituals to appease the Almighty Algorithm.
Requiring SPF/DMARC/DKIM/PTR shouldn't really be a problem, but there are extra layers of spam filtering on top of the problems Gmail will give you feedback about.
Oh I wish that were the case. One recent "lovely" example I stubled upon is Deutche Telekom (t-online.de) not willing to use SPF because it's not perfect enough for them.
It's only the tip of the iceberg unfortunately.
What people on IPv6 are getting is one of two things: 1) Harsher defaults or 2) No reputation system applied.
In the case of Google for email, it appears that it's harsher defaults.
Most major mail providers have access to one or more ipv4 addresses. This allows for reputation + it's a bit more wasteful to burn through ipv4 blocks on spam campaigns.
ipv6 address are plentiful, so you can burn through them on a spam campaign.
Total prefixes are 18,446,744,073,709,551,615
Yes, you'll have to send an email somewhere with little to no content to initiate reception of things like invoices, password resets, and monitoring emails. Is that such a bad thing?
Yes, spammers will desperately try to get you to send an email, any email, to their domain, or cybersquat on common typos. These problems might have easier and better solutions than letting tech giants takeover a formerly open Internet protocol that anyone should have the ability to use.
Like somebody can't be an expert at something without having gone through formal education for it?
The problem is that people managing their own server wouldn't be able to initiate conversations with people who give them business cards, for example.
Google and Microsoft have the computation and data storage resources to forever record how much spam they've received from the mail servers operated by any given small company or hobbyist. If a domain's authoritative mail servers have a history of never sending spam, when that company or hobbyist (for any reason) has to switch hosting providers and gets a new IP address, Google/Microsoft should recognize that the domain, and its referenced mail servers, has never been malicious.
But instead, the majors treat a new IP address for a long-established domain as entirely new to the internet and assign it zero, or even negative, reputation. And they don't care to address this because, well, why care about hobbyists and the rare small company insolent enough to try to self-host email?
PS yahoo is the worst for assuming poor reputation from a sender. In my experience they just introduce massive delivery delays at the drop of a hat even when the sender has a stellar reputation.
But to reiterate my earlier point: Google and Microsoft have the computational and storage capacity to have a detailed history of all domains' authoritative mail servers behavior. They will know whether a domain has a history of patch negligence.
No, I think the far simpler explanation is that they just don't bother tracking reputation by domain. Or if they do, it's largely overshadowed by the weight given to IP-based reputation.
You're right in one more way: both Google and Microsoft offer email hosting solutions for small businesses. Their main (and, if you've got more than 8-10 accounts or so, the only) selling point is that it makes managing email hassle-free. Making it as painful as possible for small businesses to host their email server helps these services tremendously. If they had real interoperability (either of their own accord or because it were forced upon them through regulatory measures), the biggest cash cows of these services -- companies that are well into "enough accounts that your own server would be much cheaper" territory but not large enough to afford or risk large infrastructure changes -- would evaporate pretty quickly.
DKIM, DMARC, and SPF do, though, and basically are table stakes if you want your mail (especially mailinglist messages) to go through to people at major providers.
Every post master knows that it is done. But it doesn't have to be that way, although it certainly can feel like it when a company like google decides to not budge on the matter.
Sure, if you want spam. Don't like it, get people to deploy DKIM, then the domains will be used for reputation purposes.
What the root comment says is BS, the industry uses these methods for a very real and practical reasons.
In your case, it's likely that your volume and sending patterns aren't consistent and trustworthy enough to keep track of your domain and IP reputation.
You have to understand that they get millions of letters from new domains each day, sent from compromised Wordpress blogs and the alike. If you want to be deliverable, you have to be consistent and not suspicious.
Or, more likely, there's some other mistake in your configuration somewhere.
Alternatively - come up with something akin to D&B registration system so people can attest that they won't engage in spammy behavior.
Yes, and I'm saying what's the prequisite for that to happen. As long as it's okay (which it currently is) to send unsigned mail, IP addresses have larger weight. DKIM needs more deployment for that to change.
There's absolutely no way that IP-based reputation schemes will be deprecated before alternatives are viable. Sure it would be nice for a few people here, but no, won't happen before the ecosystem improves.
> Alternatively - come up with something akin to D&B registration system so people can attest that they won't engage in spammy behavior.
Already exists. That too gets abused.
If you filter by IP block (or address!), it might be a block that has changed hands and is no longer spammy. Or it might be a block from the Zen Policy Blocklist, which blocks ranges that the responsible ISP has submitted as domestic or retail blocks that are supposed to send outbound mail through the provider's smarthost.
If you filter by domain, that could be the envelope sender, the From:, the Reply-to:, or the domain of the SMTP client. Only the last is reliable; and you also have the IP address for the client. In my experience, the IP address is more useful, for longer, than the domain name. But any good blocklist should age quickly (i.e. old stuff should drop off the list).
Depends on your approach. If you hack IoT devices then you have a lot of IP's. If you hack Joomla sites, you have a bunch of domains.
> I don't know - is free domain tasting still a thing?
Yes. There are also discounts and stuff like that.
I would have thought that it's fairly self-explanatory that anti-spam measures utilize the strongest signals. If sender domain becomes that, it will get more weight.
So if in the future email providers could reject both SPF-less domains and DKIM-unsigned letters, IP's would definitely become less relevant. So, get people to deploy those things.
Even better - Google could help small mail server admins by actually providing the information that landed their e-mail in the spam folder. If the protection is tied to the domain, no spammer will be helped by this knowledge.
And I understand that maybe a new domain might be suspicious at first, but after a few years of unchanged ownership (backed by whois data) there is simply no reason to put any mail messages from these domain to spam. Whatever the IP is.
Stop making up excuses for them. They are negligent at best, malicious at worst. Can't wait till they get hit by a lawsuit over this.
They definitely take the IP less into account if other things are more trustworthy. Totally ignoring it would be short-sighted from them. There are many cases where the domain is fine but looking at the IP and its usage patterns helps prevent abuse. Be it misconfigured (and then abused) SPF, stolen DKIM keys, public website that's email-capable getting compromised, these things happen a lot.
> Even better - Google could help small mail server admins by actually providing the information that landed their e-mail in the spam folder. If the protection is tied to the domain, no spammer will be helped by this knowledge.
They have a significant amount of content-based filtering, knowing that helps spammer reword their crap and bypass those.
> I have proved that the e-mail is tied to my domain and they know that my domain is not spamming - what more do they want?
That's also part of the thing, you can't prove and enforce this for both envelope and header from at the same time. Not to mention how minuscule the amount of perfect and strict SPF+DKIM+DMARC is out in the wild. At this point in time IP's are a very strong signal.
> Stop making up excuses for them. They are negligent at best, malicious at worst.
I haven't made a single excuse, I'm explaining why things work the way they do. You calling it bullshit won't make it so.
The problem is not that the level of strictness that Google is applying is necessary to reduce spam; the problem is that Google doesn't care and is in fact incentivized to not accept email from non-Google domains.
[1] https://en.wikipedia.org/wiki/Hashcash
Yes, and sender reputation is part of it :D that's why they can handle it.
> Also, Hashcash[1], which is a virtual silver bullet for email spam, especially when combined with existing trust mechanisms
Who do you think has to and can spend more compute or money, well-trusted Google or a small player? Far from a silver bullet.
> i.e. the less reputation you have, the harder the Hashcash challenge
Who do you think will have better reputation? I won't even start at how different people define spam. Though, I can promise you that on average SpamChimp would get to send a lot of spam for much less "HashCash" than a small player.
> is in fact incentivized to not accept email from non-Google domains.
Everyone is.
Either botnet devices can send tens if not hundreds of letters, or it's going to be slow and/or expensive for all the legitimate small senders. You really can't have both.
On the other hand, spammers are likely sending hundreds of emails per minute per server, so limiting them to 1 email per minute would have significant consequences for them.
I hope this demonstrates the disproportionate effect Hashcash (or modern equivalent) would have on spammers when compared to legitimate senders?
We use exactly the same idea for secure password storage. Password verification is slow, really slow, on purpose. This makes it "slow for legitimate password verifier" (the website you're logging into), but it also makes password cracking on a large scale completely infeasible.
Yes, in a scenario where you're a legitimate sender that sends very few letters compared to a bulk spammer. Apples to oranges.
You unfortunately ignored both the low-rate spammers and the high-rate legitimate senders. The former is barely hindered and the latter is PoW-walled into inconvinience.
If you now bring in reputation schemes, then those can be played by adversaries, you'll end up at step zero but you have made a large climate impact.
> but it also makes password cracking on a large scale completely infeasible.
But this is not the current scenario. Intentionally complex password hashing is not really PoW we're talking about.
Both of these seem like imaginary edge cases that don't really exist. I'm going to gloss over the oxymoron of low-rate spammers.
> If you now bring in reputation schemes, then those can be played by adversaries, you'll end up at step zero but you have made a large climate impact.
How do they do that? If we focus on the current state of things, high rate legitimate senders already have an established reputation so they don't need to change how they handle outgoing emails.
The only thing that would change is how large email providers such as Gmail handle incoming emails from small/unknown senders, by accepting a sufficiently complex PoW as an indication that the email is not spam. Over time this should increase the sender's reputation to a point where PoW is no longer required.
TLDR: Nothing would change for the big players, small senders are given an opportunity to prove they are legitimate, and spammers remain locked out.
I'm sorry? Both of those cases are very real and happen often, you don't get to just ignore them because of your lack of experience.
> How do they do that?
Many ways. Sending legitimate mail for example. You're acting like it's hard to show good behaviour for a while. It's one of the telltale ways I catch those fucks, they're doing it already.
But they also use their own inboxes to build up volume, hijack accounts, abuse SPF policies and DKIM misconfigurations. There probably are more ways to get a "trusted" domain I can't recall this instant.
Things regular senders can do, spammers can do, but they'll spend more effort to do it at scale!
You have demonstrated repeatedly how you have little knowledge of spammers' and others' behaviour, you handwave away all "edge cases" and say it will work. No, it won't.
All you'd add is wasted compute resource and time. Compute and time that especially spammers get for free by abusing others.
"Often" but as a tiny fraction of all sender cases, something that anyone who's actually worked with email spam would know, so you either dishonestly withheld that fact or you don't know what you're talking about.
Also, low-volume spammers are quite clearly an oxymoron - the definition of "spam" includes high volume. It's literally high-volume by definition: "unsolicited usually commercial messages (such as emails, text messages, or Internet postings) sent to a large number of recipients or posted in a large number of places"[1]
And, the vast majority of individuals do not send high volumes of email. This is a fact. It is an edge case.
> because of your lack of experience
You seem to be making a lot of arguments from authority, yet you don't even have the credentials to back them up.
> You're acting like it's hard to show good behaviour for a while.
They're definitely not. It's crystal clear that the usefulness of Hashcash is not dependent on it being hard to show good behavior for a while, and in fact its most effective in the long term.
> But they also use their own inboxes to build up volume
Something that Hashcash solves, because they'll have to spend a lot of cycles solving challenges to send non-spam email to build up reputation repeatedly, because for each "clean" IP/server they'll have to repeat the "reputation ramp-up" that every legitimate sender only has to do once.
> hijack accounts
Literally no anti-spam situation protects against this, so it's irrelevant, and I don't know why you're bringing it up.
> abuse SPF policies and DKIM misconfigurations
Then these need to be fixed. Google indiscriminately blocking smaller senders isn't a fix.
> Things regular senders can do, spammers can do, but they'll spend more effort to do it at scale!
Exactly - that's what Hashcash does, is force spammers to spend more effort on the things that regular people do, because spammers do it at scale far more often than regular people do, and so it introduces a disproportionate cost on them.
> All you'd add is wasted compute resource and time. Compute and time that especially spammers get for free by abusing others.
This conclusively illustrates that you have no idea what you're talking about. Spammers do not get resources for free - they have to either spend time to acquire botnets and hijacked domains themselves, or pay money for someone to do it for them. This is pretty clear to even people who aren't in the industry.
Spamming only happens if its profitable, and the reason why its still profitable is because the cost of sending a spam email is currently low enough. Hashcash directly increases the cost of spending spam email by making it compute-bound on top of needing to acquire domains, IP addresses, and machines, and spending time establishing a good sender record, while (if combined with a reputation system) barely penalizing normal users in the long run.
Let me state it again: every new sender has to go through a reputation ramp-up process, but spammers are uniquely penalized by Hashcash because they have to repeatedly because after they start sending spam mail, they lose their reputation and have to start that process again. Normal users do not have to do this - they start a new server, solve a bunch of challenges, and then they don't have to do that again because they don't regularly burn their acquired reputation.
The fact that you can't actually counter these arguments about Hashcash sounds almost like a spammer who's worried about the deployment of an effective system based on Hashcash.
[1] Avamander ↗ > "Often" but as a tiny fraction of all sender cases
Not really, no. I mentioned and you're dismissing high-rate legitimate senders and low-rate spammers. The former happens for example when companies send bills each month, they hit very high rates compared to their usual baseline. The latter is not the majority volume-wise, but is the most annoying - fairly logical that some spammers choose to fly under the radar and drop a letter or two in a minute. Do that during the night and you've landed a lot of letters in people's inboxes before you get reported.
> And, the vast majority of individuals do not send high volumes of email. This is a fact. It is an edge case.
It may be an edge case for you, but it must be accounted for. You can't wave it away because then people won't receive their mail. Just one minor example, some less tech-savy users emulate mailing lists with hundreds of people in CC. You can't say there that "oh this is my anti-spam solution but it doesn't account for you because you're an edge case"
> Also, low-volume spammers are quite clearly an oxymoron - the definition of "spam" includes high volume.
There's a difference between total volume and rate. You confusing the two is your problem. Not to mention that a dictionary definition is not the ground truth neither does it define "high volume". I'd also say unwanted unsolicited advertisement sent to 30 people is still spam, but you do you.
> You seem to be making a lot of arguments from authority, yet you don't even have the credentials to back them up.
From experience backed up with explanations, and you're dismissing them based on yours with no explanation.
> Literally no anti-spam situation protects against this, so it's irrelevant, and I don't know why you're bringing it up.
Incorrect. Looking at IP addresses suddenly behaving unusual is a great signal compared to the aggregate of a domain's total.
> Then these need to be fixed. Google indiscriminately blocking smaller senders isn't a fix.
They absolutely should be, but it's difficult and in the end you're not alone on the internet. Until people will, IP's continue to be used for spam classification.
And no, it's not indiscriminate.
> Exactly - that's what Hashcash does [...]
I think you misunderstood again and let me rephrase. Spammers spend more effort to lower effort spent per unit of spam. So in the end they spend less effort per letter than legitimate small senders.
> This conclusively illustrates that you have no idea what you're talking about. Spammers do not get resources for free
Your sentence conclusively illustrates you're clutching at straws. Declaring that something stolen hasn't been gotten for free is a shaky foundation to build your argument upon and pedantry at worst. The cost to acquire those resources is there anyways, some extra CPU they have to steal is not that significant.
> Hashcash directly increases the cost of spending spam email by making it compute-bound on top of needing to acquire domains, IP addresses, and machines. > Normal users don't have to do that again because they don't regularly burn their acquired reputation.
You're going in a loop again. We're comparing relative effort/monetary cost per email here. If one side has lowered the cost of effort by parallelizing things it's unfair to the side that hasn't. Consider the scenario where a new online store owner starts sending email or someone's script is compromising yet another poorly secured blog, IoT device or SMTP account.
Now a legitimate sender has built up all the reputation, after a few flags by recipients it was spent in an instant. Everything will become slow and expensive for the actual owner, once again. For spammers it was the tenth one that day.
Such a massive extra cost for the spammers. /s
That was just one example. Again it a...
These patronizing "you obviously know less than I do" type comments that occasionally drop a few bits information aren't adding anything to the conversation.
If you haven't noticed, I heavily copy phrases from the people I reply to. If you don't like your tone being matched, unfortunate.
> Please elaborate on the intricate details and challenges of dealing with spam, from start to finish, as well as any proposed solutions to make self-hosting viable when dealing with providers such as Gmail.
I have, and you included have ignored it as "edge cases" and alike. What can I more explain when someone has taken the stance of being dismissive?
Still... it is a creative idea.
i'm not even a big fan of hashcash, but i don't think you understand it, or how it could fit in as a component of a broader spam mitigation system. (for better or worse)
google wouldn't need to compute hashcash for their outgoing emails, because they have DKIM and a solid reputation. nor is its computational power somehow at odds with that of smaller players.
It's not that I don't understand, I know how it won't.
> google wouldn't need to compute hashcash for their outgoing emails, because they have DKIM and a solid reputation.
Great, so what's the point? Small players would have to pay Google to deliver mail :D
> computational power somehow at odds with that of smaller players.
As I said in my other comment, either botnet devices can send tens if not hundreds of spam letters, or it's going to be slow and/or expensive for all the legitimate small senders. You really can't have both.
You really don't understand it, because you're unable to provide counterarguments to specific explanations of how it would fit in with a larger anti-spam system, and because you say things that are clearly false to someone who does understand it.
> Great, so what's the point? Small players would have to pay Google to deliver mail :D
...such as this, which is false. Nobody's paying anybody anything - Hashcash is a computational proof-of-work challenge with zero money transferred.
If you're talking about a metaphorical "payment" of the smaller players having to do some kind of work - that's a feature, not a bug, because it allows the smaller players to invest effort into convincing Google that they're legitimate, and if coupled with a well-designed spam system that then takes that reputation and associates it with DKIM signatures/IP addresses, then the smaller players burn n cycles and then stop having to do so because they now have reputation with Google, while spammers burn n cycles continually for every message that they send because they keep getting flagged as spam.
> As I said in my other comment, either botnet devices can send tens if not hundreds of spam letters, or it's going to be slow and/or expensive for all the legitimate small senders. You really can't have both.
Yes, you really can, because you don't understand how anti-spam systems work. A system with Hashcash would start out assigning most IP addresses+hostnames/DKIM signatures with a reputation of 0, and in order to accept a message, would require the sender solve a Hashcash challenge inversely proportional to their reputation. If the message is marked as spam, the reputation takes a hit - if it's not, the reputation increases slightly (along with all of the other factors that an anti-spam system uses).
The legitimate servers quickly build up reputation and stop having to solve challenges, while the spammers pay the computational tax until the end of time, making it unprofitable for them to send most spam to most targets. The end.
I have provided specific cases where the previously proposed use-cases wouldn't work.
> they now have reputation with Google, while spammers burn n cycles continually for every message that they send because they keep getting flagged as spam.
> The legitimate servers quickly build up reputation and stop having to solve challenges
So the spammers send a few warmup mails. It's very naive of you to think spammers can't imitate legit sender behavior.
It's also clear you haven't seen or analyzed any significant amount of spam. The end.
It's clear to anyone who understands the basics of spam and Hashcash, and can use basic logic, that "So the spammers send a few warmup mails" literally doesn't change the effectiveness of Hashcash.
Let me state it again: every new sender has to go through a reputation ramp-up process, but spammers are uniquely penalized by Hashcash because they have to repeatedly because after they start sending spam mail, they lose their reputation and have to start that process again. Normal users do not have to do this - they start a new server, solve a bunch of challenges, and then they don't have to do that again because they don't regularly burn their acquired reputation.
You say
> It's also clear you haven't seen or analyzed any significant amount of spam. The end.
...but you're unable to use basic logic on publicly-known facts (the vast majority of spammers send large amounts of spam; the vast majority of users send small amounts of email; reputation is hard to gain and easy to lose) to infer the above or generate a logically cohesive counterargument. Every time you can't give an answer, you fallback to an argument to authority.
They aren't doing a very good job of using sender reputation. If you have a domain with a several year record of no spamming whatsoever, and with every outgoing message using DKIM, they will still start blocking you if other senders who just happen to have an IP address close to yours start spamming.
IP block reputation should only be used to set the default when dealing with new senders. If they have seen enough DKIM signed messages from a sender to know that the sender has a good reputation, IP block reputation should have weight 0 when receiving mail from that sender.
Google could do this without any increase whatsoever in the amount of spam that shows up in their customers' inboxes. All that would change is that their false positive rate would go down.
That's your limited perspecive, sorry.
> they will still start blocking you if other senders who just happen to have an IP address close to yours start spamming.
Absolutely, how would they know how that provider assigns those IP's? A lot of spammers use entire /24's.
> If they have seen enough DKIM signed messages from a sender to know that the sender has a good reputation
A lot of spammers have DKIM, it's not a good reason to allow mail from a suspicious subnet.
Pick a provider that deals with their spam complaints. That's the harsh truth.
You keep missing the point. Nobody is suggesting that mail be allowed merely because a domain is using DKIM.
What is being suggested is that if a domain has a proven history of not sending spam then blocking the domain's mail just because there are spammers in the same subnet results in blocking non-spam. Not blocking the domain does not increase the amount of spam that gets through because the domain is not sending spam.
Where DKIM comes in is that it allows receiving sites to distinguish between mail that is really from domain X and mail that is forged to appear to come from X. The latter can safely be automatically classified as spam. The former can be used to build an accurate history for the domain to determine if it sends spam or not.
> What is being suggested is that if a domain has a proven history of not sending spam
That is merely because of using DKIM! Everyone can create a 'history of not sending spam', including spammers.
With your solution someone can rent a subnet, warm up a bunch of domains, but then blast spam from the entire subnet without immidiately affecting the entire set of spam domains.
What you're proposing wouldn't allow trusting even signed mail from an already suspicious subnet. Be the domain with or without reputation, so are the spammers'.
> Not blocking the domain does not increase the amount of spam that gets through because the domain is not sending spam.
They don't know that before they deliver your mail to inboxes and someone does or doesn't mark it as spam.
Sender reputation where you block smaller email providers indiscriminately without weighting DKIM signatures or identity (or just blocking based on IP) is not a way of "handling" it. Your comment is akin to someone saying "the government can handle poverty" and you saying "yes, by killing all of the poor people" - it's very clearly a non-solution.
> Who do you think has to and can spend more compute or money, well-trusted Google or a small player?
I don't think that you understand how Hashcash is supposed to work, or what it's supposed to solve. The computational cost is imposed on non-Google players - both smaller email servers without a reputation (until they build one up, obviously), and spammers. I mean, sure, if everyone implemented the protocol, then non-Gmail servers could technically request that Gmail do Hashcash challenges in order to receive email, but nobody's going to do that, because nobody believes that Google is going to send spam mail.
> Though, I can promise you that on average SpamChimp would get to send a lot of spam for much less "HashCash" than a small player.
No, you really can't promise me this, unless you're literally in control of Gmail servers and intentionally adjust the difficulty settings to fulfill this condition.
> > is in fact incentivized to not accept email from non-Google domains.
> Everyone is.
I don't see what your point is here. The fact that Google is incentivized to not accept email from non-Google domains is not a good thing - it's a bad thing. And, really, the above is false - everyone is incentivized to accept legitimate, non-spam email from all of their customers, which is not the same as what you said.
That's not true.
> because nobody believes that Google is going to send spam mail.
This sentence indicates you don't know much about who sends spam or not.
> The computational cost is imposed on non-Google players
You're going in circles. Do you not see how that especially is unfair to the smaller players? Spammers have just as much compute, it will just impose an additional burden on the legitimate senders.
So in your scenario a small startup has to wait tens of minutes (bunch of letters in the queue already) to send email confirmation letters. The spammer on the other hand does the same, it'll infect a few more devices, they've exeeded the startup's sending rate.
Alternatively, both the spammer and the hypothetical startup build reputation first. They'll warm up the domain, both get to send more in the same timeframe. So... we're at the beginning again, everyone warms up their domains and IP's like they do right now. PoW hasn't improved the sitation.
> I don't see what your point is here.
Because you haven't seen enough spam. On average everyone is incentivised not to accept mail from new or unknown domains.
> everyone is incentivized to accept legitimate, non-spam email from all of their customers
No shit, but also not the topic. Received mail is not perfect, that's what's the actual topic.
It's particularly disappointing because even just getting the process started was delayed when no one answered our initial inquiries of interest for nearly a month. That was when I first figured out it was because my e-mails were getting routed to the agencies' spam folders. At this point, I'm probably just going to have to give in and get a gmail address for the sole purpose of completing an adoption, and then stop using it and go back to my real address.
Who'd have thought this would be the thing preventing me from completely de-googling?
Thankfully all my customers have been super understanding and were kind to take the disputes back once they were able to locate the emails in the Spam folder. Also I finally decided to just use a "reputed" smtp server to fix my problems (feels like extortion but what can you do).
I've written this before, but it's so sad that something so important and vital like email is so broken and we are the mercy of corps like Google/Apple/Microsoft and everyone has their own secret policy.
I've run into countless problems because of my messages being marked as spam. The worse is that now it as become socially acceptable to ghost people, I never know who ghosted me, the mail server or the person! I've run into "I ignore your email, get the hint" a few times when thinking I was marked as spam and tried to contact people through other means.
Right now, when you send an email, you don't know if it was received or not and if the lack of answer is a conscious decision or because of filtering.
Do you remember the bad old days of spam?
Left unchecked I think it might have destroyed the e-mail ecosystem entirely.
Regarding the block vs reject, that unfortunately gives a big tip to the people whose working hours are spent trying to defeat three blocks and send you more spam.
There is not reason at all to suspect that a message from a small server with proper SPF and DKIM that has never sent spam (for real or spoofed, see SPF), is spam. The other half is from small servers that sprung over night and have no reputation.
Half of the spam I receive is from google/outlook/amazon. They don't block each other, do they?
It’s really really trivial to automate buying a domain, get some IPs and setup DKIM, SPF and DMARC. How can a provider determine intent?
Gmails approach is to mostly use user signals, which in my opinion is the best way.
At a company I’ve worked for, they sent some mail using sendgrid to a poorly made mailing list and we’re in gmail spam for ages.
My personal ran email mostly stays in inbox.
I think it’s such a typical mindset that people with very limited domain knowledge think that people with a lot more domain knowledge are just being stupid.
If google is too aggressive about dropping mail and gmail users can't reliably get the email messages they want, those users should move to a platform that doesn't block those messages. Problem solved.
I've seen several articles in recent months about Google (and other providers) blocking too much, but they are nearly always about the senders and the burdens blocking and sender reputation places on them.
As much as I'd agree that senders have to do a lot to get mail accepted maybe this is working as intended more often than not and the fact that people aren't dropping their gmail accounts in droves seems to suggest that the people whose inboxes are directly impacted by Google's filtering are largely happy with how things are.
Most email is spam, and for the few senders who aren't just bitter that they can't force spam on the most popular email providers with impunity, I freely admit that this is an extremely frustrating experience for them and that it does help degrade the utility of email and contributes to the consolidation of useful email providers. The current situation isn't perfect.
The idea that we could send mail from any routable IP address, or send bulk email easily to anywhere, or even set up our own personal SMTP servers on a whim and still send email to any other network in the world with no deliverability issues is beautiful, but not practical.
Given the decades of scammers and advertisers abusing email, I'd much rather leave it up to the recipients to decide when a service has gone too far in blocking what ends up in their inboxes.
The real story is nobody cares.
Exactly. Why should the major players care about those too small to matter?
> Sure you would have to catch people buying a "good" domain that expired
Adding to my earlier point, Google and Microsoft are well-enough connected to the domain registrars to know when that scenario has happened as well. If they put any effort into it, they could reliably determine whether a new IP address for an established domain is legitimate or a fraud. But as we've said, why put any effort into it when the only people complaining are not important?
as far as how to solve this problem technically, I think a reputation system based not on domains or ips but on email certificates is the real answer here.
How would that help? Spammers can get certificates too. Maybe it cuts down on some of the misconfigured http email senders, maybe, but not enough to matter. Scam sites run https these days.
You can't use like age of activity of the cert to help because a) things get compromised, b) you need to rotate your certs frequently anyway.
DKIM does not and DKIM keys should be rotated once in a while, but few do.
It wouldn't help to fully trust, nothing would, it's a human problem, it would help to trust more.
Any wide spread certificate program will just have the email address as the identity, and it will be authorized by establishing control of the email (just like the majority of certificates used for https are domain control only, no organizational verification, not that organizational verification means much anyway). Anyway, identity is hard; there are many people with my name, including a Pulitzer winning author.
At the point of receipt, when verifying via DKIM that foobar.com did indeed send this email, then update your spam statistics for foobar.com and you're good.
Not to mention all the websites that get hacked or the uber-cheap registrars.
How is it not easy to check a reputation database for domains when evaluating a mail server?
Using DKIM records, Google could cross-reference the reputation for the sending domain and, where applicable, recognize that the domain in question has never been malicious or negligent. And in that case, extend a probationary reputation sufficient to allow the IP to establish its own new reputation.
> How is it not easy to check a reputation database for domains when evaluating a mail server?
The point I'm making in this thread is that they have the ability to maintain a database of domains that have been proven to be trustworthy. I am not talking about a database of IP addresses, and I am not sure why that is being raised here.
Huh? With dataset of ~100k classified hams/spams I get like >99% precisison in identifying spam/ham with just bogofilter and 0 heuristics whatsoever (my mail server accepts everything, and I just use bogofilter client side).
I guratanee you MS has a dataset with > 100k emails, lol.
It's not like this is unsolved problem.
My god, are you joking or what? Spam, a solved problem? Far from it.
Google reads and indexes every email that they deliver to a gmail inbox. They probably have the worlds largest and highest paid staff of ML experts, and they resort to auto-flagging based on IP address?
Fix it.
It's an unsolved problem for anyone.
> Why can I identify spam at a glance, with greater than 90% accuracy
I don't think you understand how terrible even 99.998% accuracy is at their scale.
> They probably have the worlds largest and highest paid staff of ML experts, and they resort to auto-flagging based on IP address?
You think they aren't using them for this? IP addresses very likely go into their model as well, when determining something is spam.
> Fix it.
They are and adversaries are breaking them again.
What does scale have to do with it, am I missing something? Wouldn’t a spam detection rate of 99.999% mean that only every 100000th spam mail would get through on an account basis, how’s that terrible?
The person I replied to said they can identify spam with 90% accuracy with a glance, my point was that it's abysmal as accuracy. Even orders of magnitude better it'd still be bad.
> Wouldn’t a spam detection rate of 99.999% mean that only every 100000th spam mail would get through on an account basis, how’s that terrible?
I didn't specify if it's per-account or a false positive rate, it's also actually not relevant to the point I'm trying to make.
When you receive billions of letters a month small errors start to matter and they're doing a better job than the person I replied to thinks they do.
Honestly, the best filter in my experience is Spamhaus Zen; it spots most spam with very few FPs. It's an IP blocklist.
In the end it's still a hard problem, so takes a bunch of work.
From the perspective of an email receiver, I WANT things to be difficult for this group. Truly free email available to all would be unusable owing to spam. Oligopoly, I believe, results in the best experience for everyone.
That algorithm doesn't work. The internet is filled with parked domains that have "never been malicious". This just creates a new market for clean domains that you can use to evade protections. It makes spam a little more expensive, but it still gets through.
None of these tricks work. There are no tricks. All rules can be gamed. The only thing that can't be easily faked is reality: if Microsoft knows you're a big org with a well-managed IT group running your output email setup, then they know they can (probably) trust you not to spam their customers. If you are too small to prove that to MS in a scalable way, no amount of heuristic trickery is going to help you.
Google--a technology giant with algorithms and heuristics running most operations--is not up to the task of improving algorithms for email server reputation? No, they are definitely capable of improvement. It's simply not of interest to them.
I can hardly blame them because the cost-benefit analysis clearly says, "why bother?"
Of course anything can be gamed, but magnitude matters. I've had the same domain for 24 years, with many hundreds or thousands of email conversations between users of my server and those of the major email players over the years. I had to switch my mail server's IP address two years ago. Immediate zero reputation from many big players.
Incorrect, if they stopped spending all that money and effort to keep up, their users would get flooded.
> Of course anything can be gamed
Sending spam/marketing e-mails is a multi-million industry. Both on illegal and legal markets. It's a constant race.
This isn't the 1990's anymore, recipient domains aren't just homes to a few hundred undergraduates or a dozen programmers. Email hosts manage communication for hundreds of millions of customers.
Disclaimers: I make sure to do everything else right. I have SPF and DKIM and my DMARC policy is to reject 100%. I also don't use IPv6 (DO Kube doesn't really support IPv6 well).
I have found other major providers to be much worse. Microsoft seems to rely almost entirely on IP reputation and marks everything as spam, even accounts that have marked messages as "not spam" many, many times. Apple outright blocks the IP range.
While on the other hand I fully agree with TFA: I have _never_ been able to send an email to Gmail from a IPv6 address and have it not end up as spam, not even to accounts where I already whitelisted previous attempts.
I don't think it's a reputation issue, since my IPv4 addresses likely have much worse reputation. It's as if they just handicap all IPv6 addresses.
That's why your experience does not apply to the GP.
I noticed that some providers completely block digital ocean IPs on their firewalls. So no way to get unblocked.
while I concur with all of the points you make, there is a logical and statistically accurate reason for some of these spam filters.
even if your no-open-relay, rdns, spf, dkim, dmarc, SSL/TLS and other configuration is absolutely impeccable on your smtpd, your only recourse in a situation like this is to change to a new ISP that does not have a poor IP space reputation in all the adjacent IPs.
https://proton.me/
https://tutanota.com/
check senderbase and the rbls, you might be there too if you're on a less gilded cloud provider like oracle or ibm. in general new mailservers take some TLC to repair the IP space you're given before providers will trust it.
Google’s borglike embrace of email is a sad thing for The Internet that slipped quietly by before it was too late.
This way one doesn't accidentally bang up the other
The e-mails from your domain should only come from IT-run systems/services.
If other departments want to sent e-mail setup a sub-domain or a get a secondary/marketing/communications domain(s).
We were lucky enough to have a product idea that sold out immediately after some coverage on a few tech sites (A tiny hardware product for nerds).
Trying to update our customers, we sent out an email to about 500 people via Shopify, the spot the little guys use to do eCommerce. No links in the email at all, just a text based update that we’re here and working hard to keep up with demand.
Ever since then, every email we send to a gmail user (including friends and family) get’s bounced. It’s not even in their spam box! We have since added google dns txt records and via something they call postmaster tools and we still can’t get emails out.
Just text, no links, no phone number, just a product update from a small company.
Gmail is not Email anymore.
If Shopify is transforming a temp-fail into a non-delivery receipt, that's their problem, and yours, but nothing to do with Google.
.... Unless you're a kooky spammer re-sending a captured YouTube terms-of-service-change e-mail to the inbox of some Gmail user who bears no relation to the original recipient of that e-mail.
Then, hey, you have no issues with delivery.
https://news.ycombinator.com/item?id=31577087