I'm actually having a similar problem recently, although my messages are declined at the SMTP stage, "due to the very low reputation of the sending domain", which I attribute to the output of certain cron jobs being sent towards my own Gmail account — it seems like my whole primary domain name has subsequently been blacklisted a few weeks ago:
It also seems that more and more of the mailing list posts end up in the Spam folder, too. For example, I'm subscribed to the nginx list, and recently found that all the mails from one of the core developers are now in the Spam folder in Gmail.
Gmail is becoming less and less useful as days go by. If it cannot be used for the mailing lists anymore, and inbound forwarding into the account is so unreliable as well, leaves fewer and fewer reasons to continue using it, especially as it's no longer free as they've stopped their infinite storage growth, so, I now need to pay 1,99 USD/mo because I bought into their infinite storage claims back in the day, and subscribed to way too many mailing lists to fit in 15GB of space.
I sometimes wonder if certain companies can get white listed.
There's a particular perfume company that sends spam to my gmail account every month. I've been marking it spam for at least five years, but somehow it always gets through.
Remarkably, Fastmail has gotten to the point where its spam filter works better than Google's for the kind of messages I get.
The interesting thing about whitelisting is that you have to be sending a lot of emails so the company determine if you are sending spam or not. So if you have a private domain, your own mail server, and are sending only a small number of emails then your not going to get whitelisted. Sometimes you can get on a feedback loop but all that will tell you is if someone clicked on "this is spam".
If the mailing list is using a bulk email service, then they should use DomainKeys Identified Mail (DKIM) and Sender Policy Framework (SPF) for identifying ownership & not getting marked as spam i.e. If they aren't really a spam.
Sorry, but your advice have been detached from reality for a long time.
SPF is not a "should" anymore, it is a must. Many major e-mail providers will outright reject your e-mail at SMTP level, unless you have correct SPF record. Rest will move SPF-less e-mail to spam on automatic basis. In this sense SPF does help you to get delivered, and avoid getting into spam bin, but your chances can only go from "100% spam" to "90% likely spam", not "0% likehood of being spam".
DKIM is not a major consideration for Gmail (or anyone, as far as I know) and have never been one. You will not magically avoid spam bin by using DKIM. Instead DKIM provides a great opportunity to botch your reputation with mail servers: as soon as you forward a e-mail (between providers or within the same provider) it will lose it's DKIM validity, and thus may get classified as spam (!!) on the basis of invalid DKIM signature. Boom! you are a spammer now.
Valid SSL cert and valid SPF are the only things, that have noticeable impact on Gmail's treatment of your email. DKIM and DMARK and largely useless and can only get you in extra trouble.
As for bulk email, it is going to be classified as spam until your domain and IP gain sufficient reputation. The only way to gain that reputation is to send slowly increasing amount of email for long time without being marked as spam. Being a rich business, Google's client, hanging on Google's forums, using Gmail as your mail server and having at least dozen contacts among Google's employees also helps.
I have a hotmail throwaway account (it has since been migrated to outlook with the microsoft buyout) and its spam filter is completely backward. There is currently 373 e-mails in my inbox (100% spam, expected since it is a throwaway...) and 4 e-mails in my spam folder--100% of them the account registration/confirmation e-mails that I was wanted.
I have migrated to fastmail for all of my personal e-mail, which I have been pretty happy with. So far I haven't caught any legit email which has been categorized as spam and negligible spam that has made it through into my inbox (mailing lists notwithstanding--I sort them using automation rules). My one complaint is that there is no way to turn off their "thou shall not receive executable attachments" filter so I still need people to play the "set windows to show the file extension, change it to .txt, and then attach it" game when people want to send compiled code via e-mail.
In any case, it is certainly a much better situation than the gmail spam filter I used while in school, which had about a 10% chance of marking outside messages as spam--in fact it even started marking e-mails sent by my professor (via the university gmail) as spam...
Honestly not sure, but certainly sometime in the 20th century. It was my main e-mail back then, I migrated to Yahoo! shortly after the Microsoft buyout and kept the HoTMaiL as a throwaway. When Google announced Gmail and the 1GB (with projection to infinite) storage I switched to them as soon as I got an invite, but when they changed their slogan from "Don't be evil" to "Do the right thing" with the Alphabet restructure I ditched it and moved to my own domain with Fastmail. If Fastmail starts to show signs of evil it will be as easy as updating my domain records to point toward OurMailIsActuallyGoodIPromiseThisTime.com and click through the certificate warnings on my mail clients.
That Hotmail branding survived long after the purchase; in the last few years it's the same interface as their 365 business offering now, and under outlook.com branding.
When I last had to wrangle with MS (c.5 years ago, who were dropping all our email _replies_ to customers, they had a third party that you could pay to ensure your emails got through ... apparently the problem was our domain was on an IP hosted by a company who'd previously hosted (but didn't now) a company on a different IP and that other company has sent some spam.
In testing different mail servers I find that Gmail will treat them differently based upon there hello line. I can't remember which one was accepted most but there is a difference. Also you need to look into if others on your netblock are sending spam. If you are on the same netblock as a spammer then gmail will treat you as if you are the spammer.
The biggest problem to me with Gmails spam filter is that it is so totally hit and miss, and there seems to be no reliable way of ensuring that messages stop going to spam, even from other Gmail accounts, from people on my contact list, that I've persistently marked not spam.
At this point the only reason I'm still using Gmail is sheer inertia, and it's a question of when I get time to start untangling myself from it.
Yes. I've also used Google Apps and had it file e-mails from our own users as spam despite settings that's supposed to bypass it.
Meanwhile some Gmail addresses I have rarely have this problem, so it's not even consistent across accounts or domains.
The most annoying bit (and probably the first untangling from Gmail I'll be doing) is that I have my own domain hosted on Google Apps, but since my main Gmail mailbox is a different one, it's set to forward everything. It's meant to bypass all filtering and just forward. Or so the settings and support claims. It doesn't. I have to regularly log in for the sole purpose of marking things not spam.
Not sure about OP but to me rules in GMail don't work.
Gmail filters don't seem to work anymore. I had filters and labels and it was great but I noticed they seems to be failing.
If you go down the rabbit hole of Google Gmail support you'll find filters don't apply to email ("conversations") already in your inbox. But they also don't seem to apply to new email.
Asking on various groups people seem to be giving instructions for options in GMail that no longer exist.
I've tried that without success. Delete all rules/filters, labels and re-create it all again. Nothing will work on old or new messages. And "apply to zero conversations" is the only thing I see that seems like it would be it but the zero part means no chance of working.
Gmail even replied to me on Twitter saying the filters no longer work.
It doesn't seem to work. In fact, IIRC, they simply add a huge button dialogue to any such emails that pass due to a filter to mark it as spam, without any buttons to dismiss their improper suggestion — e.g., you can't even confirm that the mail isn't spam anymore if you use such filters.
This same thing started a few weeks ago for me as well. I have a cron job that parses my log file and sends me usage statistics every morning. I had been receiving this email for 100+ days when suddenly it stopped. I investigated thinking it was a problem in my script only to find that all worked well and all emails were in my spam.
I use SES from AWS and now my new user sign ups emails for email verification also end up in spam. It’s very frustrating because they changed some rule and now my low volume email sending is considered spam
In my experience struggling to deliver email from my self-hosted server, Gmail's by far the worst. Not that I haven't had the occasional issue with Yahoo, Outlook and AOL. But here's the key difference: Yahoo, Outlook and AOL actually offer tools to resolve the issue! For small senders like me who fall below Gmail's postmaster tools limits, we have no recourse other than trying to find some sympathetic Google employee and DMing them. The experience really blows.
I desperately wish Gmail would offer some sort of registration program that would let me attest to my identity and thereby give my server the benefit of the doubt. I'd happily cough up my private medical records if that meant my family's emails reached Gmail's inboxes most of the time.
Thanks for giving it a look! Enforcing DMARC and strict SPF tend to screw up mailing lists and forwarding. I don't see how I can enable them without causing a lot of other problems for me and my users.
No arguments about my provider. Any tips on how to select a better one? I'm thinking someone who adds some friction to unblocking outbound port 25, e.g. Vultr, might be better.
DMARC shouldn't be an issue for mailing lists, as long as one of DKIM/SPF passes. Are you seeing reports of both DKIM and SPF failing for mailing lists you use?
Properly configured mailing lists shouldn't modify contents of emails causing DKIM to fail.
I personally use Linode and would recommend them. But basically any tier 1/2 provider, i.e. too big/popular to block, will do. However, I would avoid DigitalOcean per my experience due to them completely blocking SMTP on IPv6 in addition to badly implemented IPv6 support and seemingly subpar support.
The problem is that Gmail and G Suite essentially have monopoly on email nowadays. Even some pretty big companies are amongst the users. All startups are using G Suite. As are many billion-dollar companies like Twitter, Arista Networks, Netflix, WeWork, a16z, YCombinator. What do you do if you can't email any of the people at all these corps?
On fastmail I haven't seen a single legit e-mail marked as spam, whereas it happened nearly weekly under gmail. In the last year I have not seen a single (non-mailing list) spam e-mail show up in my inbox, and I have received ~10k spam e-mail this year.
I have the problem of spammers sending emails faking they are from my domain but through hacked computers. So I get thousands of rejects. DMARC shows you where it is happening, spf can cause some of it to get blocked, but you are stuck with anyone can make up email addresses and send them out through other mail servers and in-turn your domain is certainly going to start going to spam due to this.
it sucks that even though i haven't actively used gmail in at least 5 years, google is still able to hoover up my personal, and what should be private, info. and this kind of anti-competitiveness, which seems to come to light occasionally and only dimly at that, stinks just as bad.
i can't wait until everyone else realizes how insidious a company google is. it's starting to happen. i'm hearing the low rumbles among (non-technical) folks i know. thank goodness!
The attitude of the replies he gets lower down in the thread is shocking. I hope whatever mailing list that is doesn't have any kind of good reputation, because it seems like the consensus opinion there is shockingly anti-cooperative, anti-internet, "me before the network", even going as far to say "if Google thinks you're a spammer then you probably are".
That's a meeting of technical and moral view. I'd love for corps to have no impact on people, for distributed knowledge and resources, for everyone being in control of their data, and general socialist direction.
But the technical answer to "why is my email landing in spam" is still "follow best practices, build up reputation, and you'll be fine... unless Google decides otherwise, which they can". Posting about how unfair that is is not going to change anything.
> Posting about how unfair that is is not going to change anything.
It won't change anything, unless enough people complain. And even just recognizing that things are not right is the first step required to make a change. Of course, I won't hold my breath waiting for Google to change their ways, but I think it is in fact important to make a clear note their is not the right way. As the guy writes:
"Maybe it is just so, that big companies ignore small external senders who are sending mail to them, simply because they can, and we can't do anything about this (however, it's always worth trying to check whether we really can't - and that's what I'm doing by posting my issue on this list). But even if we can't do anything about this, we should not pretend that everything is OK, they are correct, and it's the sender's fault. No, on the contrary: we should state it explicitly and clearly that this behaviour is not OK. Maybe we have to live with it; but it doesn't make it less bad."
It’s posted elsewhere in this comment thread but that’s after the point where it becomes clear that he wrapped a moral argument in a technical envelope and people spent a lot of time trying to help him out with the technical argument.
Put another way, play stupid games, win stupid prizes.
1. Our ML models see that your emails spam? You're spam.
2. You're using Tor or a VPN? Here's 10 reCaptcha's.
3. Don't use Google Webmaster or use AMPs? Good luck finding your page on the 1st page.
4. Youtube video includes education hacking tutorials? Demonetized.
Obviously some alternatives are moving to Fastmail and using Duck Duck Go, but we need Google to stop this "my way or you don't exist" attitude before they become a huge conglomerate which controls major aspects of many people's lives (which they arguably already do) and leaves everyone (like small companies that don't use Google or just regular people who don't want a Google account) separated from the world they're forming.
> Few bucks a month add up and can be considered expensive by many.
I pay around 3$/month for Fastmail, I don't think it's expensive, an email address is the primary communication method everyone uses nowadays and it's useful every day.
I have all my mailboxes with gandi.net. Each domain you buy from them comes with 2 fully managed emails, and I think 3GB storage each, for just over $1/month. I basically never get spam, nor blocked emails. They have 2 webmail interfaces too, but I don't use them. As a bonus, gandi financially supports a ton of open source projects, which never hurts.
I use https://mailbox.org/en/, it's from a small German company and I can use my own domain with the smallest plan for just 1 € per month. Their spam filter also seems pretty good.
"Moving to Fastmail" is such a popular alternative that if they ever get to any kind of reasonable size we'll be in the same position all over again.
That should not be the suggestion. Many smaller providers encourages the competition necessary for an open network, not a small number of huge providers.
I use Fastmail scared that my emails will go to spam. But ideally I want to live in a world where my emails hosted on my email server will send like they did 10-20 years ago.
I've been using Fastmail for about 5 or 6 years now. My only problem is the 1GB mailbox size limit I have. Maybe I should upgrade. Otherwise I love it. Their webmail client is phenomenal and works with mobile and desktop. I don't like their Android app but I haven't tried it in a year. Their web client is so good I don't need an app for them.
I do see mail going to spam though. It's never bugged me that much.
You can also just use the native imap client provided by your operating system. The beauty of using a provider and service that adheres to open email standards.
I have mostly moved to self hosting email. I still have a lot of legacy registrations with various companies that go to gmail, otherwise everything else is self host on a hetzner vps (2.50 eur per month) which doubles as a wireguard vpn.
First time setting up an email server was quite a chore as I have no background in email, getting all of the moving parts to play nice required some patience.
My second email server used Mail-in-a-box scripts which I highly recommend. Zero to functioning email server in a few minutes. The slowest part is DNS propagation. Anyone comfortable setting up a domain name and the basic DNS settings that are invloved will have no problem self hosting email. The setup includes web-based client. I use k-9 mail client on my phone.
I find the catch-all email feature very helpful in creating ad-hoc addresses for spammy companies (eg car insurance). I can sign-up with spammy.companyname@mydomain.com and it will direct to spammy@mydomain.com.
I'm not completely de-googled, but I am consciously and gradually reducing my exposure to google, and other large networks.
We're still a fair way away from that! We're growing steadily, but not at the same rate that "give it away free and work out how to monetize it later" services do. Worth it for having a sound business model and no venture capital breathing down our necks to turn evil though.
Thanks! I was looking for an export button for them earlier today so when I saw you comment I couldn't help myself.
A zip file with all notes in it would be fantastic. I just want to feel like my data is safe from Apple having a bug that wipes it. Read a lot of Catalina horror stories.
IMAP looks promising and might be all I need. Neither Spark or Apple Mail seem to let me export mail as txt but presumably there's a mail client out there that does in which case problem solved. Thank you for point that feature out, should throw it up on:
It's not the ideal solution and certainly not as easy as clicking a button but, for a quick 'n dirty solution, you might look into using "offlineimap" and configuring it to sync only your notes folder instead of all of them.
I want to use Fastmail, but their minimum price for using your own domain is $50/year. At that price I can get Exchange Online with double the storage. And given that they have an office in the US, I would expect them to follow the same laws that Microsoft and Google have to regarding requests from government agencies.
This looks to be a good alternative except it does not have HIPAA support.
Even though I have shut down my practice for the near future, I will still need to have information protected for several years to the Health Department's requirements for psychological records.
Email providers who lack this support will have difficulty gaining traction in the health world.
> we need Google to stop this "my way or you don't exist"
What about stop being so entitled? Google is offering free awesome services for very little and you’re saying these things. It’s so demoralizing. Get off your high horse.
It’s been a while since google made anything awesome hasn’t it? Maps is still great, but things like AMPs are so terrible that they were a bigger part in driving me to DDG than privacy.
Maybe I’m weird, but I’ve never been in an AMP where I wouldn’t rather go to the actual side and use reader mode.
What about stop being so entitled? Google is offering free awesome services for very little and you’re saying these things. It’s so demoralizing. Get off your high horse.
I don't use gmail these days (except as a honeypot out of laziness), but I get plenty of email from google hosted domains. The issue isn't (just) that people want to use these services and want Google to have more reasonable policies, there is also the issue of having to tolerate Google's policies because other people use their services.
Emails from Google servers are notable to me because there is no way to report spam (and the signal to noise ratio from Google is pretty low for my case). Google's opaque approach to email means that deliverability issues become even more tedious to deal with than they should be.
What about stop being so entitled? Google is offering free awesome services for very little and you’re saying these things.
It seems you're not getting the point. The problem is not for users of those "awesome" Google services the problem manifests itself for people sending legitimate emails from legitimate email accounts to a Google account.
Google tags those as spam and refuses to deliver them, while always shifting the goal posts and providing sub par support (an number of essentially useless links) to the sender of the email to a Google account.
This happens, btw, also when the sender account never spammed and sent legitimate mails to the Gmail user for years.
So your snarky comment is completely out of line here.
It's not about people mooching free "awesome" Google services for free it's that Google is a rotten net citizen and is behaving like shit to the entire world.
> the problem manifests itself for people sending legitimate emails from legitimate email accounts to a Google account
It's funny to me that people in this thread are complaining about Google being large enough to serve as a de-facto authority, when the only way one could solve a "legitimate emails are being blocked" problem is to have a definition of 'legitimate' that would come from... an authority.
Google is dropping the ball but I don't know what the solution would look like that isn't just "Someone else be Google now." ;)
>2. You're using Tor or a VPN? Here's 10 reCaptcha's.
I get recapthca ALL the time for just not being logged on and deleting all filthy cookies on exit; and not "honoring" 3rd party cookies at all... and I don't care. Best part is that if you live in a country you don't speak the language you get google services in one of the country official languages -- so
what you should do and mark is often unclear. The next best part is a road trip through Europe (edit adding "?hl=en" to the url tends to fix the issue in most of google services)
>4. Youtube video includes education hacking tutorials? Demonetized.
History is a far greater offender.
>1. Our ML models see that your emails spam? You're spam.
Years ago, I recall my company hosted gmail started detecting CEO mails as spam... and he was pissed for not getting responses.
I was getting really long and annoying recaptcha all the time, presumably because I use Safari which has all that anti-tracking stuff. But lately (past 2-3 months?) I only get a single challenge, I wonder if they've found a way around the protection or if they've just changed the model
OMG, me too, and all I did was to tighten up cookies (disable 3rd party trackers, erase all cookies on close, etc).
Now I've got a PhD on solving "traffic light" and "crosswalk" Captchas as a result.
I started to resent Google for this, except now I think twice before going to a web site that will give me these hurdles, so it ended up helping me to focus on my work. Now I appreciate the extra work.
>Your browser sends language preferences, doesn't it?
Since http1.0[0]... I have wanted to add another header -> "RespectAlHeaders:All-Custom-CrRafted", so the wimpy implementation of ip->country->language is duly removed.
Have you actually had a lot of problems sending e-mail to Gmail users from your mail server? For me Gmail has not been great about accepting e-mail and not marking it spam (occasionally have issues), but others like icloud.com have been worse.
> but we need Google to stop this "my way or you don't exist" attitude
I think that's exactly opposite. We need more competition, not forcing a private company to behave in some way that "we" (and who is "we"?) find acceptable.
The problem is that Google is not just Google the search engine, or Google the email provider, or Google the video service, or Google the website analytics engine, or Google the online office suite, or Google the ads syndicate.
Because Google is all of those things, it has accumulated too much market power, to the extent that they are market distorting. Bing and DDG (which is not an independent engine, but collates results) are irrelevant to a company's Internet presence. There is no such thing as SEO for Bing. Whereas Google can make-or-break you.
When Google the ad syndicate and Google the search engine collude (as it were) to place your competitor's ads ahead of your organic search result, a top result even, and to distinguish such ad results as ads only technically, by the narrowest of definitions, this compels you to advertise with Google the ad syndicate. Just as an example.
This behavior is harmful to the market (not to mention to users, but that's a different subject) and it is a given that Google the monopoly^Wsingle entity will continue to make such moves, whack-a-mole style so it's futile to squash any individual behavior even if you could. Ergo the reasonable solution is to break them up.
Could you sue Google for slander in a case like this? They are explicitly labeling your messages as being manipulative or dishonest without any just cause. Why does it matter it’s an algorithm instead of a careless person in power?
>Obviously some alternatives are moving to Fastmail and using Duck Duck Go, but we need Google to stop this "my way or you don't exist" attitude before they become a huge conglomerate which controls major aspects of many people's lives (which they arguably already do) and leaves everyone (like small companies that don't use Google or just regular people who don't want a Google account) separated from the world they're forming.
I mean, I get captcha's all the time while working from Google's network. You're exhausting from a shared quota. That's life on the modern internet.
Bitch about Google all you want, it's not like they decided this is how the internet should work. Prior to Google, my company integrated with reCaptcha because no other choice existed to reduce automated abuse.
Did you try changing your name to Netflix, Etrade or American Express? I swear, no matter how many times you mark their emails as spam, they eventually wind up back in your inbox ( admittedly in the Promotions section rather than the Primary inbox ).
Google will of course argue it's for it's users sake. What are the odds that if legislation passes, Google simply lets unnecessary spam through and pretends they can't do better?
I ran into this issue many years ago, and I eventually gave up trying to increase the reputation of my mail server. For all my projects that require mailing, I've now succumbed to spoofing headers using Gmail's SMTP, which surprisingly doesn't go to spam, or using a third party mailing service like AWS.
> A quick look at Talos Intelligence reports a 1600% increase in email from your domain over the last day; email reputation os 'neutral' at this time. Not good, not bad, but neutral.
> Also you're on OVH, about which a quick look through the list's archives will
possibly prove instructive. It's reasonably likely (as likely as not) that
you're running on an IP in a neighbourhood with some poor neighbours.
> As Mathieu pointed out in his second email, building a good reputation takes time. Losing a good reputation is a matter of rather less time and can be influenced by factors outside of your control.
The rest of that reply chain is the usual debate:
OP. My domain's emails are marked as spam by Gmail
No, it's not their network. It's our common network. If anybody imposes own rules on their part of the network, we are losing interoperability. Internet loses any sense.
He apparently thinks that he (or maybe more accurately "we") should dictate how private companies treat traffic. Despite it's huge footprint, Google does not have a monopoly on email, if they are treating users unfairly, they can switch -- but I use Gmail because their spam filtering is so good. And I'm ok with dropping a few legit emails if it means not seeing spam in my mailbox.
From my old law courses (IANAL but someone decided programmers should have a basic knowledge of law):
* An illegal monopoly should not be absolute. Having >20% of the market has been enough te be considered a monopoliy for antitrust. The main question is if your actions dominate the market.
* But having a monopoly is not enough. You can have 100% of the market and be legal. Abusing the monopoly is the problem. Abuse is typically crushing smaller competitors, or using your monopoly in 1 market to push yourself to another market.
* Is google abusing its potential email monopoly? My non-lawyer guess is no. Being non-responsive and arrogant is not illegal.
* But the google browser and search engine might very well be other stories. Here google seems dominant enough to influence web standards and change other corporations behaviour to raise in their search index.
* I don't think US or EU governement are willing to step in, though.
It's a bit surprising that he got anything through without SPF and an MX record. That seems to be the minimum to get mail into the inbox for a new domain at least, so his domain probably had some positive reputation at one point.
A while back I had someone setup a new domain with office 365 email and a completely broken spf record that only allowed his web server. Gmail accepted everything without a problem.
I agree an SPF record is an expected minimum but "use a big provider" is more relevant ime.
I would hazard a guess that this mostly happens when many people mark your messages as spam from your domain. Or if you send enough email to gmail users with certain keywords/phrases.
I also have my own mail server, and I have the same problem.
I'm guessing this is machine learning algorithms confusing correlation with causation once again. Being a small private mail server is strongly correlated with being a spammer. So even if you are not relying any spam, you certainly look like someone that would. And even if you have not done it yet, you will probably in the near future, so they might as well treat you as a spammer right away to avoid potential problems. (analogy with racism intended).
I actually check Spam daily now, after finding support emails whose logs got snagged in the dragnet & personal stuff — even email from my Dad got shunted to Spam.
I still tip the hat, though, to Google, whose algorithms are wise enough to assiduously stow both the “jump on a call” and “donate to my campaign” emails to spam.
If you're really curious setup DMARC as well and check out the reports. I used to work for a company that did DMARC analytics, and after setting it up for my personal domain it's kind of interesting to see who's trying to impersonate you (or who's getting spam purportedly from you).
None of the big providers send DMARC messages. Google doesn't. Microsoft doesn't. The only DMARC messages I've ever gotten were from Yahoo and mail.ru.
Google/Microsoft don't send DMARC reports because it would make it easier for spammers to know they had gotten past those filters. It's easier to just drop everything they don't like silently. It's why e-mail is terribly unreliable today (much more reliable if you DO NOT use Microsoft or Google).
Google/Microsoft don't send DMARC reports because it would make it easier for spammers to know they had gotten past those filters.
Is that a recent thing? I was getting DMARC reports from both Google and Yahoo as late as April this year. Around then I pared down my Sonic account and they ate a bunch of stuff that I was too lazy to setup again including DMARC. While Google and Yahoo make up most of the reports I got this year, I also see some from Microsoft and AOL from 2017. 163 (China) used to send out reports too.
Aggregate or forensic? I’m not sure I’ve ever seen a forensic report.
Both. In looking at my mailbox I see some forensic reports from 163 and aggregate reports from everyone I mentioned plus Comcast, Emailsrvr, XS4ALL, and QQ.
I don't remember if I eventually turned off forensic reporting, but I do remember that LinkedIn and Facebook were particularly strong proponents of DMARC (and both were really good about accidentally load testing our infrastructure). Oh and one very large provider that actually gave us unredacted messages.
I regularily receive dmarc reports from Google (every day) and yahoo for instance. Not sure I've ever received any dmarc report from hotmail though.
And email is pretty reliable with Google. It is a disaster with hotmail: you have to send a big number of emails to have a chance that hotmail treats your email as non spam. They put your self hosted emails in the spam folder of your regular contacts, and the fact that they respond to your emails and systematically mark them as non spam is an irrelevant detail for them.
The vast majority of messages in my spam folder are not spam on Gmail at the moment. Most of them are from providers I've semi-regularly tried to get it to recognize as not spam for months or years. Of course Gmail also outright rejects a lot of stuff at the SMTP stage, so their filters are better than they look, but the amount of misclassification is extremely frustrating.
I have this issue with OSSEC messages (automated messages from an Open Source IDS). Same thing, too, have been marking them as “not spam” for years. I think it’s because I don’t always review them...:/ But just because you don’t always open a message class doesn’t mean it’s spam, Google!
At one place we even had them start bulk-rejecting messages for our Google Apps domain from a whitelisted source.
The entire point of the whitelisting was to allow our internal mail server to forward transactional messages to addresses on our own domain, so it was not even that it was ever reported as spam by anyone, or ever hitting external addresses where I'd understand if Google was more trigger happy.
It was ridiculous realising that they basically had no way for even paying customers to tell them that "that e-mail server we told you is ours and we always want to receive all e-mail from no matter how spammy you think it is, using the mechanism you say it there to do exactly that, well, we really mean it". Instead it'd randomly just stop working. [this is a few years ago; who knows, maybe they've fixed it]
This guy has an odd (seemingly nonsensical) quote in his email signature.
To a machine learning algorithm, it might look kind of like the random nonsense some spammers put in spam to corrupt Bayesian filters. He might want to try temporarily taking out that signature if he’s using that as part of his emails.
good catch. In my limited experience, complicated signatures (e.g. hotlinked images), or emails with few words and several links are most likely by gmail to be filtered as spam.
I get like 50 emails a day entirely made up of hotlinked images from companies trying to sell me shit but god forbid someone have one in their signature. Thanks google.
For some reason, Google also marks "Google Alerts" notifications as spam. They contain spammy search results, but the email comes from Google.
At least they are consequent in not white-listing their own services :)
Gmail also marks messages from gmail or other google services as spam. Gmail also marks all legitimate mails from Paypal as spam (phishing). I use several email addresses and google accounts, but in the end I forward most mails into the same gmail box. Their spam filters cannot handle the concept of forwarded mail correctly.
I remember when Gmail was new. You got an e-mail that said "hey, we think you are sending spam, please do this". I replied and said that I always send from this domain, so if they think I'm spamming, just block the domain, then they white-listed it instead.
I always see DMARC as a solution in these kind of threads, but the problem is that it's trivial for spammers to also setup DMARC. In fact almost 100% of the spam I receive follow the e-mail guidelines better then most legitimate e-mails.
We need a new protocol! Where e-mails are opt-in, like with messaging apps, you first need to be introduced.
No, not at all. There are reports that even if you use G Suite, and whitelist your own domain to send yourself logs through crontab, your may still have problems with the email deliverability.
My own domain was blocked for nothing more than sending myself a list of a few dozen domain names through cron, which Google deems as containing malicious links, even though there are no links present.
Email is not the same it has been for years/decades. These days you need to use all the security tools available, most notably TLS and DKIM, set a strict DMARC policy, and don't do anything unexpected such as sending lots of emails out of the blue.
Email is all about reputation: it can take a lot of effort to build trust, but just one bad move and it's all gone.
From the comments here I get the impression that this means, “don't do anything Google's machine learning environment cannot interpret”, which has the added drawback of not ever knowing if you are doing something it considers unexpected.
> Email is all about reputation: it can take a lot of effort to build trust, but just one bad move and it's all gone.
Is it though? That may well be true for small individuals, but the Spam I get these days comes via the large mail providers, Amazon SES, sendgrid etc, and nobody in their right mind is going to block them. "Too big to block" is a thing now, "too big to have to care about abuse reports" is too.
Yes, the big mail providers are an issue. But it's also an issue the other way around. We use Postmark at my place of work and unfortunately, that doesn't prevent all of our mail from being marked as spam. Some mail gets blocked because it comes from the same IP address as some spam mail. I know that Postmark puts effort in preventing spammers to use their service but Amazon probably doesn't care a lot about people misusing SES.
Wouldn't that imply that they have a lot of spammers on their network? As far as I understand, the value proposition is both "you don't need to worry about email" and "we have so many customers (sending legitimate emails that are very similar in content) and so little spam, we have great reputation and you're inheriting that reputation".
If that doesn't work out, is their reputation not good enough? Is the receiving server too strict? Are those blocks occurring on major providers or for individual servers where an annoyed person might just blacklist an IP if they get spam from them more than once?
Note that "a lot of spammers" isn't really a lot in laymen terms. It varies by provider, but last I worked on this if anywhere from 2-5% of all emails you send are clicked as spam the provider will start blocking you.
We got blocked once because we got a single spam click: the ISP said they have a zero tolerance policy. We had years of sending hundreds of email per week without a spam click.
> Email is all about reputation: it can take a lot of effort to build trust, but just one bad move and it's all gone.
In my experience, the email blacklists and filters are somewhat forgiving. I accidentally configured my email server as an open relay (didn't know what that was at the time), and for a brief period was sending a not insignificant percentage of spam worldwide.
Once I discovered my mistake and corrected the issue, I was able to submit domain removal requests to the various blacklists, or just wait for my domain to fall off the radar. I expected it to be worse, and was pleasantly surprised to find that this kind of "forgiveness" is really an industry standard.
But then, that was just one bad move. I'm not so sure about two bad moves, and I'd rather not find out.
This is the consequence of the spam filters doing a good job. I too remember when consumers would clamour for better spam detections and different companies would hawk products that purported to do so.
But now that spam detection mostly works well for most people, it’s a forgotten issue. Now we have people on this thread questioning Google’s authority to filter spam because from their PoV, Google isn’t adding much value to begin with.
The problem is that Google errs too much on the it-is-spam side. Having to delete a few spam mails isn't a big issue if the alternative is losing legitimate mails because of overzealous filters.
Imagine your grandma getting phishing emails from banks. If only 1% goes through, because Google is a bit more lenient on the detection, I can see the phishing success rate going through the roof.
I thought this had been getting better. At least more e-mail from my personal server has been getting through to its destinations. I've written about this too a few years back:
..and I reference the Hostile E-mail Landscape blog post by Ribton (really good btw).. which is gone now and redirects to some bullshit software site (there are archives of it).
I've got DKIM, SPF, DMARC, reverse DNS for IPv4 and v6 and I still don't have the volume to show up in the postmaster tools Google has.
My own e-mail server occasionally has spam come through. I really don't care. I sometimes mark it to be learned, but honestly I'd rather have some SPAM get through than get HAM marked and placed in the big fucking SPAM bucket I never check.
But for big providers and dangers like ransomware, this isn't an option. So we have shitty unreliable e-mail (even on gmail back in 2012; I had delivery issues to other people on gmail).
Where does the blame go? The 1% of people who give scammers money, get ransomware infections and click on everything? Over aggressive spam filters? The lack of real e-mail server diversity? Who the fuck knows anymore.
> but honestly I'd rather have some SPAM get through than get HAM marked and placed in the big fucking SPAM bucket I never check.
This is why I like that my email provider (https://mailbox.org/en/) simply rejects all messages detected as spam: False positives don't get lost in a spam folder I never check, instead the sender gets a mail delivery failure notice by their own mail server and at least get a notification that they couldn't reach me.
so i got a notification that you didn't get the email. fine. i still have no way to talk to you unless i sign up to some other email service and try it again.
depending on the reason to email you, i may not care enough. (maybe i wanted to buy your service)
if a customer emails me and i don't respond because i didn't get the email, i have a problem, not the customer.
the better solution would be both. respond with a delivery notice, AND deliver the email to my spam folder.
I have no idea if it actually is, but it might be a legal issue too in some jurisdictions. German law for example requires commercial sites to have an option for quick and electronic communication - Email is generally a valid choice. If your mail server rejects incoming mail (be it because it's overly sensitive, offline, or you never bothered setting it up etc), that could very well be considered as non-compliant.
Yes, but that has it's own challenges with spam protection, privacy laws, users not spelling their email correctly etc, and you still need an email address to receive those messages (or check for new form entries regularly), so an email address is often used as a shortcut.
I sent a text format mail containing a couple of innocent words and a youtube link of a completely innocent clip to my wife. Gmail in their infinite wisdom classified it as spam. Obviously users need to be protected from watching youtube clips.
I used the smtp server of my ISP. I'm know that it is tightly monitored and not the source of any significant amount of real spam.
Trust me it’s not, I’ve read theM countless times, google just decided I’m spam, and don’t tell me why. I run a small business and have a business name registered in the same name which cost $500 AUD, I don’t appear to have an option apart from paying AWS or Microsoft to host me.
Sandstorm should have really caught on. We need to make self hosting drop dead simple so that even non-technical people can do it. The power of these mega-tech companies can only be whittled down if we take collective, well thought out action. And it is not just about tackling power imbalances. Owning your own data is a fundamental aspect of our digital lives which ought to have become simple and natural, given Moore's Law and the phenomenal device capabilities in recent years, yet somehow we are going the other way. Even as our devices become ever more capable (hardware wise), the software is moving to ever more to the Cloud and we're being locked out of the full capacities of our devices except in the narrow realm of media consumption.
406 comments
[ 2.2 ms ] story [ 318 ms ] thread* https://news.ycombinator.com/item?id=21340460
* https://mailman.nanog.org/pipermail/nanog/2019-October/10381...
It also seems that more and more of the mailing list posts end up in the Spam folder, too. For example, I'm subscribed to the nginx list, and recently found that all the mails from one of the core developers are now in the Spam folder in Gmail.
Gmail is becoming less and less useful as days go by. If it cannot be used for the mailing lists anymore, and inbound forwarding into the account is so unreliable as well, leaves fewer and fewer reasons to continue using it, especially as it's no longer free as they've stopped their infinite storage growth, so, I now need to pay 1,99 USD/mo because I bought into their infinite storage claims back in the day, and subscribed to way too many mailing lists to fit in 15GB of space.
There's a particular perfume company that sends spam to my gmail account every month. I've been marking it spam for at least five years, but somehow it always gets through.
Remarkably, Fastmail has gotten to the point where its spam filter works better than Google's for the kind of messages I get.
SPF is not a "should" anymore, it is a must. Many major e-mail providers will outright reject your e-mail at SMTP level, unless you have correct SPF record. Rest will move SPF-less e-mail to spam on automatic basis. In this sense SPF does help you to get delivered, and avoid getting into spam bin, but your chances can only go from "100% spam" to "90% likely spam", not "0% likehood of being spam".
DKIM is not a major consideration for Gmail (or anyone, as far as I know) and have never been one. You will not magically avoid spam bin by using DKIM. Instead DKIM provides a great opportunity to botch your reputation with mail servers: as soon as you forward a e-mail (between providers or within the same provider) it will lose it's DKIM validity, and thus may get classified as spam (!!) on the basis of invalid DKIM signature. Boom! you are a spammer now.
Valid SSL cert and valid SPF are the only things, that have noticeable impact on Gmail's treatment of your email. DKIM and DMARK and largely useless and can only get you in extra trouble.
As for bulk email, it is going to be classified as spam until your domain and IP gain sufficient reputation. The only way to gain that reputation is to send slowly increasing amount of email for long time without being marked as spam. Being a rich business, Google's client, hanging on Google's forums, using Gmail as your mail server and having at least dozen contacts among Google's employees also helps.
I have migrated to fastmail for all of my personal e-mail, which I have been pretty happy with. So far I haven't caught any legit email which has been categorized as spam and negligible spam that has made it through into my inbox (mailing lists notwithstanding--I sort them using automation rules). My one complaint is that there is no way to turn off their "thou shall not receive executable attachments" filter so I still need people to play the "set windows to show the file extension, change it to .txt, and then attach it" game when people want to send compiled code via e-mail.
In any case, it is certainly a much better situation than the gmail spam filter I used while in school, which had about a 10% chance of marking outside messages as spam--in fact it even started marking e-mails sent by my professor (via the university gmail) as spam...
Aside: Microsoft bought Hotmail in 1997! How old is that account?!
This site is very useful if you have your own mail server https://www.senderscore.org/
At this point the only reason I'm still using Gmail is sheer inertia, and it's a question of when I get time to start untangling myself from it.
Meanwhile some Gmail addresses I have rarely have this problem, so it's not even consistent across accounts or domains.
The most annoying bit (and probably the first untangling from Gmail I'll be doing) is that I have my own domain hosted on Google Apps, but since my main Gmail mailbox is a different one, it's set to forward everything. It's meant to bypass all filtering and just forward. Or so the settings and support claims. It doesn't. I have to regularly log in for the sole purpose of marking things not spam.
Gmail filters don't seem to work anymore. I had filters and labels and it was great but I noticed they seems to be failing.
If you go down the rabbit hole of Google Gmail support you'll find filters don't apply to email ("conversations") already in your inbox. But they also don't seem to apply to new email.
Asking on various groups people seem to be giving instructions for options in GMail that no longer exist.
TBH, it's just easier to re-do the filters. There's also an option to make it apply to old conversations on a one-time basis as well.
Gmail even replied to me on Twitter saying the filters no longer work.
Screenshot of Gmail reply on twitter https://i.imgur.com/URhBeXn.png
I like using filters I don't see why such an important email tool is so awkward to use on Gmail.
I use SES from AWS and now my new user sign ups emails for email verification also end up in spam. It’s very frustrating because they changed some rule and now my low volume email sending is considered spam
Do these providers have a better mechanism for getting yourself un-blocked?
I desperately wish Gmail would offer some sort of registration program that would let me attest to my identity and thereby give my server the benefit of the doubt. I'd happily cough up my private medical records if that meant my family's emails reached Gmail's inboxes most of the time.
* Enforce a DMARC policy instead of explicitly disabling it.
* Enforce a strict SPF policy instead of ~all.
* Send over IPv6. Usually stricter rules are applied and not-up-to-snuff emails are rejected at the SMTP level.
* Your provider hosts lots of spammers for their size. Consider switching.
* Enable MTA-STS.
The first two I'd consider obvious glaring mistakes. The next two are less important, but would help. MTA-STS is a nice-to-have.
I can't analyse your DKIM setup, since I don't know your selector. Feel free to send me a sample email if interested.
No arguments about my provider. Any tips on how to select a better one? I'm thinking someone who adds some friction to unblocking outbound port 25, e.g. Vultr, might be better.
Properly configured mailing lists shouldn't modify contents of emails causing DKIM to fail.
I personally use Linode and would recommend them. But basically any tier 1/2 provider, i.e. too big/popular to block, will do. However, I would avoid DigitalOcean per my experience due to them completely blocking SMTP on IPv6 in addition to badly implemented IPv6 support and seemingly subpar support.
i can't wait until everyone else realizes how insidious a company google is. it's starting to happen. i'm hearing the low rumbles among (non-technical) folks i know. thank goodness!
How things have changed.
But the technical answer to "why is my email landing in spam" is still "follow best practices, build up reputation, and you'll be fine... unless Google decides otherwise, which they can". Posting about how unfair that is is not going to change anything.
It won't change anything, unless enough people complain. And even just recognizing that things are not right is the first step required to make a change. Of course, I won't hold my breath waiting for Google to change their ways, but I think it is in fact important to make a clear note their is not the right way. As the guy writes:
"Maybe it is just so, that big companies ignore small external senders who are sending mail to them, simply because they can, and we can't do anything about this (however, it's always worth trying to check whether we really can't - and that's what I'm doing by posting my issue on this list). But even if we can't do anything about this, we should not pretend that everything is OK, they are correct, and it's the sender's fault. No, on the contrary: we should state it explicitly and clearly that this behaviour is not OK. Maybe we have to live with it; but it doesn't make it less bad."
Put another way, play stupid games, win stupid prizes.
1. Our ML models see that your emails spam? You're spam.
2. You're using Tor or a VPN? Here's 10 reCaptcha's.
3. Don't use Google Webmaster or use AMPs? Good luck finding your page on the 1st page.
4. Youtube video includes education hacking tutorials? Demonetized.
Obviously some alternatives are moving to Fastmail and using Duck Duck Go, but we need Google to stop this "my way or you don't exist" attitude before they become a huge conglomerate which controls major aspects of many people's lives (which they arguably already do) and leaves everyone (like small companies that don't use Google or just regular people who don't want a Google account) separated from the world they're forming.
In my opinion free email services either belong to other big corporations or provide lower quality service than gmail.
Few bucks a month add up and can be considered expensive by many.
I don’t like google and I pay for a different email account. I just wanted to remind that gmail may be hard to quit and that costs are subjective.
https://protonmail.com
I pay around 3$/month for Fastmail, I don't think it's expensive, an email address is the primary communication method everyone uses nowadays and it's useful every day.
We could probably start to divert all gmail and outlook originating mail to the spam folders on our mail servers in retaliation.
That should not be the suggestion. Many smaller providers encourages the competition necessary for an open network, not a small number of huge providers.
I use Fastmail scared that my emails will go to spam. But ideally I want to live in a world where my emails hosted on my email server will send like they did 10-20 years ago.
It's sad that it's no longer viable to have your own email server
I do see mail going to spam though. It's never bugged me that much.
First time setting up an email server was quite a chore as I have no background in email, getting all of the moving parts to play nice required some patience.
My second email server used Mail-in-a-box scripts which I highly recommend. Zero to functioning email server in a few minutes. The slowest part is DNS propagation. Anyone comfortable setting up a domain name and the basic DNS settings that are invloved will have no problem self hosting email. The setup includes web-based client. I use k-9 mail client on my phone.
I find the catch-all email feature very helpful in creating ad-hoc addresses for spammy companies (eg car insurance). I can sign-up with spammy.companyname@mydomain.com and it will direct to spammy@mydomain.com.
I'm not completely de-googled, but I am consciously and gradually reducing my exposure to google, and other large networks.
At the very least it is worth a try.
https://mailinabox.email
edit:spelling, added url.
Please allow me to export notes to txt.
As for a more general download - what would you be looking for? A zip file containing all the notes?
A zip file with all notes in it would be fantastic. I just want to feel like my data is safe from Apple having a bug that wipes it. Read a lot of Catalina horror stories.
IMAP looks promising and might be all I need. Neither Spark or Apple Mail seem to let me export mail as txt but presumably there's a mail client out there that does in which case problem solved. Thank you for point that feature out, should throw it up on:
https://www.fastmail.com/help/account/downloaddata.html#tran...
Even though I have shut down my practice for the near future, I will still need to have information protected for several years to the Health Department's requirements for psychological records.
Email providers who lack this support will have difficulty gaining traction in the health world.
What about stop being so entitled? Google is offering free awesome services for very little and you’re saying these things. It’s so demoralizing. Get off your high horse.
Maybe I’m weird, but I’ve never been in an AMP where I wouldn’t rather go to the actual side and use reader mode.
I don't use gmail these days (except as a honeypot out of laziness), but I get plenty of email from google hosted domains. The issue isn't (just) that people want to use these services and want Google to have more reasonable policies, there is also the issue of having to tolerate Google's policies because other people use their services.
Emails from Google servers are notable to me because there is no way to report spam (and the signal to noise ratio from Google is pretty low for my case). Google's opaque approach to email means that deliverability issues become even more tedious to deal with than they should be.
It seems you're not getting the point. The problem is not for users of those "awesome" Google services the problem manifests itself for people sending legitimate emails from legitimate email accounts to a Google account.
Google tags those as spam and refuses to deliver them, while always shifting the goal posts and providing sub par support (an number of essentially useless links) to the sender of the email to a Google account.
This happens, btw, also when the sender account never spammed and sent legitimate mails to the Gmail user for years.
So your snarky comment is completely out of line here.
It's not about people mooching free "awesome" Google services for free it's that Google is a rotten net citizen and is behaving like shit to the entire world.
It's funny to me that people in this thread are complaining about Google being large enough to serve as a de-facto authority, when the only way one could solve a "legitimate emails are being blocked" problem is to have a definition of 'legitimate' that would come from... an authority.
Google is dropping the ball but I don't know what the solution would look like that isn't just "Someone else be Google now." ;)
That's already the case.
I get recapthca ALL the time for just not being logged on and deleting all filthy cookies on exit; and not "honoring" 3rd party cookies at all... and I don't care. Best part is that if you live in a country you don't speak the language you get google services in one of the country official languages -- so what you should do and mark is often unclear. The next best part is a road trip through Europe (edit adding "?hl=en" to the url tends to fix the issue in most of google services)
>4. Youtube video includes education hacking tutorials? Demonetized.
History is a far greater offender.
>1. Our ML models see that your emails spam? You're spam.
Years ago, I recall my company hosted gmail started detecting CEO mails as spam... and he was pissed for not getting responses.
Default Safari option for 3rd party cookies is: "from visited". Check if you have cookies set, if you do - they know you have been a good boy.
Now I've got a PhD on solving "traffic light" and "crosswalk" Captchas as a result.
I started to resent Google for this, except now I think twice before going to a web site that will give me these hurdles, so it ended up helping me to focus on my work. Now I appreciate the extra work.
Seriously, you shouldn't have to. Your browser sends language preferences, doesn't it? Nope, IP trumps user preferences.
Had the same problem when using my job's VPN with an exit point in Germany (working from France). Very annoying.
Since http1.0[0]... I have wanted to add another header -> "RespectAlHeaders:All-Custom-CrRafted", so the wimpy implementation of ip->country->language is duly removed.
[0]: https://www.ietf.org/rfc/rfc1945.txt
Even if your language is English and your calendar is Gregorian, it will still display news dated in Hijri. No setting or option to change.
At start it has mark that search results based on results from yandex engine and crawled results
Yeah well, they offer a free service and you use it. They own their network and they can decide what they allow.
I think that's exactly opposite. We need more competition, not forcing a private company to behave in some way that "we" (and who is "we"?) find acceptable.
It's time to break them up.
Google can exist, and people can choose to use Bing, DDG, Fastmail, etc., right?
The problem is that Google is not just Google the search engine, or Google the email provider, or Google the video service, or Google the website analytics engine, or Google the online office suite, or Google the ads syndicate.
Because Google is all of those things, it has accumulated too much market power, to the extent that they are market distorting. Bing and DDG (which is not an independent engine, but collates results) are irrelevant to a company's Internet presence. There is no such thing as SEO for Bing. Whereas Google can make-or-break you.
When Google the ad syndicate and Google the search engine collude (as it were) to place your competitor's ads ahead of your organic search result, a top result even, and to distinguish such ad results as ads only technically, by the narrowest of definitions, this compels you to advertise with Google the ad syndicate. Just as an example.
This behavior is harmful to the market (not to mention to users, but that's a different subject) and it is a given that Google the monopoly^Wsingle entity will continue to make such moves, whack-a-mole style so it's futile to squash any individual behavior even if you could. Ergo the reasonable solution is to break them up.
I was surprised/disappointed to see this happen on HN.
too late?
This ship sailed years ago, though.
Bitch about Google all you want, it's not like they decided this is how the internet should work. Prior to Google, my company integrated with reCaptcha because no other choice existed to reduce automated abuse.
> A quick look at Talos Intelligence reports a 1600% increase in email from your domain over the last day; email reputation os 'neutral' at this time. Not good, not bad, but neutral.
> Also you're on OVH, about which a quick look through the list's archives will possibly prove instructive. It's reasonably likely (as likely as not) that you're running on an IP in a neighbourhood with some poor neighbours.
> As Mathieu pointed out in his second email, building a good reputation takes time. Losing a good reputation is a matter of rather less time and can be influenced by factors outside of your control.
The rest of that reply chain is the usual debate:
OP. My domain's emails are marked as spam by Gmail
REPLY. Their system, their rules
OP. I don't accept that
Repeat ad nauseam.
A "1600%" increase in email could be explained by 1 email yesterday, and 16 emails today.
Which is in fact what he claims lower down in the thread.
https://www.mail-archive.com/mailop@mailop.org/msg09020.html
This is where HN discussion should pick up, not at the beginning where we need to rehash everything.
> "Their network; their rules".
No, it's not their network. It's our common network. If anybody imposes own rules on their part of the network, we are losing interoperability. Internet loses any sense.
He apparently thinks that he (or maybe more accurately "we") should dictate how private companies treat traffic. Despite it's huge footprint, Google does not have a monopoly on email, if they are treating users unfairly, they can switch -- but I use Gmail because their spam filtering is so good. And I'm ok with dropping a few legit emails if it means not seeing spam in my mailbox.
53% of email accounts in the US are with Gmail.
What's the definition of a monopoly?
Definition of monopoly:
1 : exclusive ownership through legal privilege, command of supply, or concerted action
2 : exclusive possession or control
3 : a commodity controlled by one party
4 : one that has a monopoly
https://www.merriam-webster.com/dictionary/monopoly
* An illegal monopoly should not be absolute. Having >20% of the market has been enough te be considered a monopoliy for antitrust. The main question is if your actions dominate the market.
* But having a monopoly is not enough. You can have 100% of the market and be legal. Abusing the monopoly is the problem. Abuse is typically crushing smaller competitors, or using your monopoly in 1 market to push yourself to another market.
* Is google abusing its potential email monopoly? My non-lawyer guess is no. Being non-responsive and arrogant is not illegal.
* But the google browser and search engine might very well be other stories. Here google seems dominant enough to influence web standards and change other corporations behaviour to raise in their search index.
* I don't think US or EU governement are willing to step in, though.
It's simply one of those things one has to deal with if one operates an email server.
I agree an SPF record is an expected minimum but "use a big provider" is more relevant ime.
I'm guessing this is machine learning algorithms confusing correlation with causation once again. Being a small private mail server is strongly correlated with being a spammer. So even if you are not relying any spam, you certainly look like someone that would. And even if you have not done it yet, you will probably in the near future, so they might as well treat you as a spammer right away to avoid potential problems. (analogy with racism intended).
I actually check Spam daily now, after finding support emails whose logs got snagged in the dragnet & personal stuff — even email from my Dad got shunted to Spam.
I still tip the hat, though, to Google, whose algorithms are wise enough to assiduously stow both the “jump on a call” and “donate to my campaign” emails to spam.
Google/Microsoft don't send DMARC reports because it would make it easier for spammers to know they had gotten past those filters. It's easier to just drop everything they don't like silently. It's why e-mail is terribly unreliable today (much more reliable if you DO NOT use Microsoft or Google).
Is that a recent thing? I was getting DMARC reports from both Google and Yahoo as late as April this year. Around then I pared down my Sonic account and they ate a bunch of stuff that I was too lazy to setup again including DMARC. While Google and Yahoo make up most of the reports I got this year, I also see some from Microsoft and AOL from 2017. 163 (China) used to send out reports too.
Both. In looking at my mailbox I see some forensic reports from 163 and aggregate reports from everyone I mentioned plus Comcast, Emailsrvr, XS4ALL, and QQ.
I don't remember if I eventually turned off forensic reporting, but I do remember that LinkedIn and Facebook were particularly strong proponents of DMARC (and both were really good about accidentally load testing our infrastructure). Oh and one very large provider that actually gave us unredacted messages.
Boy am I glad GDPR is a thing these days.
And email is pretty reliable with Google. It is a disaster with hotmail: you have to send a big number of emails to have a chance that hotmail treats your email as non spam. They put your self hosted emails in the spam folder of your regular contacts, and the fact that they respond to your emails and systematically mark them as non spam is an irrelevant detail for them.
The entire point of the whitelisting was to allow our internal mail server to forward transactional messages to addresses on our own domain, so it was not even that it was ever reported as spam by anyone, or ever hitting external addresses where I'd understand if Google was more trigger happy.
It was ridiculous realising that they basically had no way for even paying customers to tell them that "that e-mail server we told you is ours and we always want to receive all e-mail from no matter how spammy you think it is, using the mechanism you say it there to do exactly that, well, we really mean it". Instead it'd randomly just stop working. [this is a few years ago; who knows, maybe they've fixed it]
To a machine learning algorithm, it might look kind of like the random nonsense some spammers put in spam to corrupt Bayesian filters. He might want to try temporarily taking out that signature if he’s using that as part of his emails.
Looks like a pretty interesting movie, but a bit strange for an email signature
https://www.mail-archive.com/mailop@mailop.org/msg09020.html
This is where HN discussion should pick up, not at the beginning where we need to rehash everything.
We need a new protocol! Where e-mails are opt-in, like with messaging apps, you first need to be introduced.
My own domain was blocked for nothing more than sending myself a list of a few dozen domain names through cron, which Google deems as containing malicious links, even though there are no links present.
Email is all about reputation: it can take a lot of effort to build trust, but just one bad move and it's all gone.
From the comments here I get the impression that this means, “don't do anything Google's machine learning environment cannot interpret”, which has the added drawback of not ever knowing if you are doing something it considers unexpected.
Is it though? That may well be true for small individuals, but the Spam I get these days comes via the large mail providers, Amazon SES, sendgrid etc, and nobody in their right mind is going to block them. "Too big to block" is a thing now, "too big to have to care about abuse reports" is too.
If that doesn't work out, is their reputation not good enough? Is the receiving server too strict? Are those blocks occurring on major providers or for individual servers where an annoyed person might just blacklist an IP if they get spam from them more than once?
We got blocked once because we got a single spam click: the ISP said they have a zero tolerance policy. We had years of sending hundreds of email per week without a spam click.
In my experience, the email blacklists and filters are somewhat forgiving. I accidentally configured my email server as an open relay (didn't know what that was at the time), and for a brief period was sending a not insignificant percentage of spam worldwide.
Once I discovered my mistake and corrected the issue, I was able to submit domain removal requests to the various blacklists, or just wait for my domain to fall off the radar. I expected it to be worse, and was pleasantly surprised to find that this kind of "forgiveness" is really an industry standard.
But then, that was just one bad move. I'm not so sure about two bad moves, and I'd rather not find out.
I honestly don't think they are nefarious, just overly insistent.
I remember email before Gmail's spam filters. Everyday was a task of deleting spam first then responding to emails.
But now that spam detection mostly works well for most people, it’s a forgotten issue. Now we have people on this thread questioning Google’s authority to filter spam because from their PoV, Google isn’t adding much value to begin with.
Imagine your grandma getting phishing emails from banks. If only 1% goes through, because Google is a bit more lenient on the detection, I can see the phishing success rate going through the roof.
It's a very hard problem to solve.
https://battlepenguin.com/tech/how-google-and-microsoft-made...
..and I reference the Hostile E-mail Landscape blog post by Ribton (really good btw).. which is gone now and redirects to some bullshit software site (there are archives of it).
I've got DKIM, SPF, DMARC, reverse DNS for IPv4 and v6 and I still don't have the volume to show up in the postmaster tools Google has.
My own e-mail server occasionally has spam come through. I really don't care. I sometimes mark it to be learned, but honestly I'd rather have some SPAM get through than get HAM marked and placed in the big fucking SPAM bucket I never check.
But for big providers and dangers like ransomware, this isn't an option. So we have shitty unreliable e-mail (even on gmail back in 2012; I had delivery issues to other people on gmail).
Where does the blame go? The 1% of people who give scammers money, get ransomware infections and click on everything? Over aggressive spam filters? The lack of real e-mail server diversity? Who the fuck knows anymore.
This is why I like that my email provider (https://mailbox.org/en/) simply rejects all messages detected as spam: False positives don't get lost in a spam folder I never check, instead the sender gets a mail delivery failure notice by their own mail server and at least get a notification that they couldn't reach me.
so i got a notification that you didn't get the email. fine. i still have no way to talk to you unless i sign up to some other email service and try it again.
depending on the reason to email you, i may not care enough. (maybe i wanted to buy your service)
if a customer emails me and i don't respond because i didn't get the email, i have a problem, not the customer.
the better solution would be both. respond with a delivery notice, AND deliver the email to my spam folder.
I have no idea if it actually is, but it might be a legal issue too in some jurisdictions. German law for example requires commercial sites to have an option for quick and electronic communication - Email is generally a valid choice. If your mail server rejects incoming mail (be it because it's overly sensitive, offline, or you never bothered setting it up etc), that could very well be considered as non-compliant.
I used the smtp server of my ISP. I'm know that it is tightly monitored and not the source of any significant amount of real spam.
With new technology, the laws of course lag. You see a wild west, then monopolies etc before it settles.