Two of the people that I've had this issue with are here on HN, I'll ping them so they can chime in on their end of the issue, as for examples I'm not quite sure what you're looking for. Random messages get marked or pass through without any apparent reason for this.
On one occasion the problem was so sever that we switched to using (the irony) a google docs document to continue the conversation.
One of the reasons I initially switched to Gmail (back in 'invite only' days) was because it seemed to have better spam filtering than anything else I'd tried (admittedly not much at that time).
I do occasionally check my spam folder and there's always a few false positives - but usually nothing critical (although admittedly I might have missed some that were and never known it)
So - day to day - I currently regard Gmail as 'good enough' but some of that is down to blind trust with little verification.
Can anyone else share their experiences - positive as well as negative?
I've had a few hams classified as spam - mostly mailgroup messages and github notifications, generally weird e-mails originating from addresses that send lots of mail. Generally, nothing critical, or even important for that matter. I vaguely recall finding one or two somewhat important e-mails in spam years ago, but it happened so rarely that I don't even remember the details. I still have the habit of checking my spam folder every other month, but generally I'm very satisfied with GMail's filtering.
I have the same thing, a few issues of the Go Newsletter GitHub notifications have been marked as spam, nothing extremely important. There are usually ~5 "real" spam messages in there - to me a scientist, it's amazing that the system also recognizes scientific spam. I just have " In order to speed up the development of its journals, Science Publishing Group currently gives authors the Special Offers for paper publication: " in my spam folder, which has nothing to do with the actual Science journal and is just a predatory scam journal.
Could be that other GMail users don't know how to unsubscribe from GitHub/other notifications and mark these as spam to get rid of them which introduces false positives into the spam notification system.
In some cases unsubscribe links can be used to determine if a targeted email address is actively monitored or not. As a result it's not a good idea to click them for email that you don't remember soliciting.
I use Gmail for some private mail, and use it at work. I always check my spam folder and keep it empty. It works for 99%, but some mails from advertising networks (which I subscribe to professionally) get marked as spam even though I have marked them "not spam" many times. For personal mail I haven't had much problems.
Why do you use gmail for your corporate communication? Why is your corporate email not worth a few dollars a month?
Just curious about the main motivation; not so much to comment on your particular choice for using google free services for your corporate communications.
Been using Gmail since the beta and never had a problem with the spam filter either. The only false positives I've ever had were some confirmation emails from Amazon.
> Can anyone else share their experiences - positive as well as negative?
I'm increasingly finding the following scenarios to occur when using Gmail:
- Letting email I've previously marked as SPAM (same subject, same sender) through the SPAM-filter as valid email. Again. This week.
- Marking non-SPAM email from known contacts I've emailed to and received emails from in the past marked as SPAM for completely non-obvious reasons.
So you DO get spam, and you DO get false positives, meaning you also need to check your SPAM folder for important email.
Basically the one feature which once made Gmail great is now utterly broken, and since for most parts of the internet people now equate Gmail with Email, Google has now officially broken email which was supposed to be decentralized and thus not vulnerable to stuff like this.
Great stuff there, big G. Great stuff. From the rest of the internet: Thanks a million.
Various spammers have been faking mail from addresses that look like they might be your contacts for a decade or more, and sometimes attacking freemail accounts in order to spam actual contacts. It's enough of a problem that gmail has a help page for it: https://support.google.com/mail/answer/50200
I've started checking my Spam folder a few times a week. I'll often get valuable, but automated, mails end up there. Things like billing invoices or emails about private messages from various forums.
Could you offer some more information on this?
What was the reason why your emails were marked as spam?
From what I've seen while debugging such issues the problems were always from misconfigured outgoing servers and their dns, so I would be very interested in knowing why this has happened.
ps. You might consider enabling comments on your blog so there is a place for such discussions.
Google does not specify any reason, the mail just silently gets dumped in the spam folder. If these were first time contacts it would be one thing but they're definitely not.
I guess this is Google's way of telling me that I should switch to Gmail if I expect to see my mail being delivered.
This almost exacerbated a misunderstanding between some of my family members. I recently worked to get my sister back in touch with her grandfather, took some coaxing on both sides, and then grandpa decided to email my sister, who never responded, so he tried one more time and then concluded that she didn't really want to talk to him.
She uses gmail and his messages were dropping into her junk folder. She almost exclusively uses her phone for her gmail access, so she never saw the messages.
Since gmail doesn't give anything like SpamAssassin headers for messages, there's nothing I can do to troubleshoot this aside from trial-and-error.
This is precisely the reason for my post. I'll get this fixed, one way or another, but likely there are numerous people that have this issue without realizing it.
He mentions he can't always tell who is and who is not using gmail, so the people he's mailing might be forwarding to gmail.
Gmail's support for this is not very good - for example, if someone with a SPF record e-mails an address that forwards to gmail it gets marked as an SPF failure because the forwarding server isn't an approved sender for the sender's domain. Unless your forwarding server supports SRS, which is very rare.
Yes, I'm sure it's google's fault. I've been maintaining an ongoing conversation with these people for many years, to suddenly see my mail marked as spam without any change on my end is clearly not the expected result.
The root cause is googles lame spam detection algorithm, I'm not in a position to debug it for them but clearly they're not doing their work properly if mail that is both clearly not spam and between parties that have previously had contact already gets marked as spam. It's super annoying because obviously I can't even monitor the other side, only by being persistent about follow up and/or lucky coincidences did we realise that gmail is the culprit.
80% of the mail sent to the same addresses... it's pretty much random as far as I can see as an outsider, I don't doubt there are triggers on google's end that decide one message is spam and another isn't but it's not deterministic enough to figure out what the real reason for the rejects is.
Not having SPF is a clear indication you have little idea of what you're doing with an email server.
A properly configured email server would have both SPF and DKIM signing.
Google's spam filter is a complicated black box. You're not going to get far understanding it by prodding it. But you're certainly not going to pass it by ignoring SPF.
Seems like basic debugging of complex systems to me. Yes, you may never by 100% sure that the lack of a SPF record is the root cause. But you do know that you don't have it, that some of your messages are getting spamfiltered, and it's well-known that Gmail uses it for spam detection. So just add it already. The worst that could happen is that you wasted a few minutes on something that doesn't fix the problem. How much time have you spent messing with this already?
I don't know what your history is, but there's piles of bugs that I've had to fix by just trying random things that seem vaguely related to see what happens. If I refused to try anything until I was 100% sure of the root cause, then lots of bugs I've been assigned would never have been fixed, and many more would have taken 10x longer.
Much spam filtering uses Bayes' theorem and it has been this way for a very long time. Your lack of SPF records must massively increase the chance that your emails are spam.
If you are still getting some emails through, they must have other characteristics that make them look less like spam than the others that are getting filtered.
Yes, that will definitely happen. But since my mail server is not administered directly by me I had to arrive at this point as a user of email and not as the administrator of the server.
This issue surely does not affect me alone hence the warning, the sender may have no idea their messages get dropped at all.
This was what i first checked too. At some day i started to receive more and more Mail Delivery Failure messages in my Inbox from addresses that seemed to be random or of some Chinese names. I just saw them because a catch all address was configured. Some MTA showed the messages and it was obviously some Chinese spam. After configuring SPF records this stopped immediately. So this might can be the reason why google thinks you are sending spam. Even if not, you will better your rating with them configured.
I wish Gmail would blacklist addresses when I clicked that button. Lately my Gmail address has been signed up for all kinds of "official" corporate mailing lists/newsletters (e.g. Home Depot). Even though I follow the Home Depot unsubscribe process AND I report the emails as SPAM, Gmail keeps letting them through.
Someone I regularly receive mail from has an address of the form forenamesurname@aol.com or ForenameSurname@aol.com, depending (I think) on which device they use. Gmail will treat one as spam, but not the other.
To stop the trapped emails ending up in the spam folder, I had to create a custom filter: simply marking mails as 'not spam' in the spam folder does not do the job.
And the filtered mails now have a yellow banner plastered across them saying "This message was not sent to Spam because of a filter you created."
Good for you, I wished I could say the same. Unfortunately it's just 'anecdotal evidence' as would be any report like this and there will be a ton of people for whom it just works. And that's where the problem lies, you don't expect this to happen at all.
Furthermore, Gmail is not to be trusted to forward email properly anymore. I can't figure it out exactly, but emails to hello@venn.lc randomly get non-forwarded to my personal email. This has caused us to miss out on a few six digit deals, which is completely insane, because it seems like the type of rule to never fail.
I also am seeing similar issues. Most (not all) retweets/mentions etc. that I get to my gmail go to spam. Others who are also getting these same emails sometimes are not seeing the same behavior.
TL;DR - Make sure your DMARC config isn't making your messages look like SPAM.
One one of the sysadmin professional organization mailing lists, email from one of the people speaking for the organization started to be marked as spam because the person had DMARC instructions on their domain, but messages remailed from the mailing list server weren't authorized. In this case, Google's mail servers were acting as instructed. I sent the person copies of the headers showing why Google's mail servers were taking this action, as, apparently, did a number of other people. The person became irritated and decided that people who use Google's mail servers needed to fix their own problems.
If your DMARC configuration is causing your messages to be marked as spam, you really should look into that.
If you appear to speak for a sysadmin professional organization, and your DMARC config causes your email to be marked as spam, you really need to fix that, and consider that blaming the recipient mail servers does not do good things for your credibility.
Of course, there are plenty of things that can go wrong other than DMARC, and some of those might well be Google's fault. You can typically track down the reasoning for a Google mail server spam decision from the headers of the delivered message.
I just checked the most recent 200 items in my gmail spam folder, and every one was an actual spam message. The article has no information to help explain what kinds of problems he was experiencing. I conclude that this is possibly false and certainly useless.
Your conclusion is nonsense, two respected HN members were on the other side of this issue.
Whether it is useful or not depends on the contents of your spam folder and already one person in this thread reports that for them it certainly was useful.
I'm not sitting here making stuff up and I seriously resent your suggestion that this is false.
There's zero evidence to show that, "Gmail not to be trusted with important email.", as the title suggests. None of it is factual. All of it is presented as-is with only what appears to be a very poor understanding of MTA configuration issues.
The answer has been here all along early in the thread. DKIM & SPF yet the complainant has simply shrugged it off with, "Well that would cause consistent issues." (which is also false.)
If he can not do basic troubleshooting or understand the importance of the SPF/DKIM settings in a modern MTA system, and as a basic starting point, he can not assert that Google can not be trusted with important emails.
I can confirm that Gmail has gotten a major pain for me with too eager spam filtering.
1. some Gmail users have told me they don't receive email from my non-gmail server/address, even though I set up SPF, reverse DNS lookup and everything else that was recommended except for DKIM. (The domain was bottoms.christianjaeger.ch.)
2. I've gotten a lot of false positives from mailing lists in my Gmail spam folder for a long time. Now the part that really pisses me off is that Gmail's learning mechanism, if present at all, does not seem to work at all. Why otherwise would mails from the same mailing lists, the same authors, with the same kind of content (e.g. "[PATCH]" in the subject, patch in the mail body) still make it to the spam folder with the noted reason "looks similar to other mails treated as spam"?
3. On top of that, every about 10 mails I mark as spam the Gmail UI asks me whether I'd like to submit it to the Gmail spam team to help improve the service. I never got any notice from them, nor does this seem to have helped.
4. I think I might be spending more time now going over my spam folder and unspam false positives there than I was going over my inbox in the past and tag the false negatives when I was hosting my own mail and used spamsassassin and grey listing.
5. It's getting worse and worse. Now all messages from my servers (crontab and other system mail) are going to spam, too[1]. So basically I now have to move to something else, and I'm ready to use my own hosting again, but I don't know what I should do to solve the "Gmail users won't receive my mail anymore" problem.
[1] e.g. From address "chris@leafpair.com" with "Return-Path: <chrisirc@a.christianjaeger.ch>", or From "bottomsmail@a.christianjaeger.ch" with that same address as Return-Path. (Also a header from Google says "Received-SPF: pass (google.com: domain of chrisirc@a.christianjaeger.ch designates 78.47.51.206 as permitted sender)" or "Received-SPF: pass (google.com: domain of bottomsmail@a.christianjaeger.ch designates 78.47.51.206 as permitted sender)" for the latter.)
I was using Hushmail for email up until 3 months ago. I had to switch to Gmail because my emails were randomly going into spam folders of Gmail users. Including users that had previously emailed me (so they should have already been in the very unlikely spam list).
I believe it is just a well meaning extension of Gmail's spam filter but it has real anti-competitive results. I had to take the pragmatic course of action but I now view email as inherently unreliable.
If any Google employees are here, consider adding the rule "if email address in (list of emails addresses sent to) then NOT SPAM". It would make a lot of people's lives easier.
It's not as simple as that. If someone I've corresponded with has had their account compromised and is sending me messages saying that they've been kidnapped, or have vi@gr@ to sell, I definitely want that filtered.
Really? If my friend had an account compromised I'd like to get these emails so I could notify them something was wrong. I could always filter the emails if nothing could be done.
Additionally I'd like to still receive emails from friends even if they mention vi@gr@ too much.
Not me. Those are so irregular, it lets me immediately get in touch with that person and let them know that their email is hacked. If my mom starts sending me vi@gr@ emails, you can bet I want to get on top of the situation and not have the email be redirected to dev/null.
I've never used gmail. I can't believe they don't have this rule. If you are correct, then yeah- duh: Please don't dump people I'm regularly corresponding with into a spam folder. I work with non-profits and such in my volunteer time and the servers may not look quite right. But if they are trying to reach me and it's a known address then please don't dump it.
In fairness some of the blame should be attached to the senders
Google is pretty efficient at eliminating phishing scams. But sure enough I found an email from my actual bank, offering me a credit card, tagged as phishing. Then again, when I looked at the sender domain I was half convinced this was as accurate as its categorisation of all the other phishing attempts
(Dear Santander. If I have to Google the yoursantander.co.uk domain because I'm not confident it's actually yours, you probably shouldn't be using it to try to get me to sign up for credit cards. If the top result is a three year old open letter written by someone else complaining about the same negligent approach to internet security...)
Similarly, there's an Amazon deal email in there, but there's also an explanation on the top that it's "failed amazon.co.uk's required tests for authentication", which might well not be Google's screwup
I was less impressed when Gmail filtered an innocuous email from my dad...
Yes, in my experience google spam filtering is useless. There are enough false positives that I don't trust it any more. I rarely have a false negative, so that's a bad sign that is tuned far too aggressively.
Spam filtering is an GAI-complete problem. Jacques, I'm willing to guess you have a misconfigured server and a little dose of bad luck. Given the amount of spam that goes over the wire, it's extraordinary GMail has so little false positives.
Welcome to the real world where email is unreliable. Gmail could be replaced with any service or corporation running their own email servers. Stop the spammers and email can be reliable again.
Enable SPF and DKIM before jumping to uneducated conclusions please. Even if it's randomly accepting for the same address during repeat testing does not indicate that it's not caring about SPF or DKIM, it indicates a random check!
To put it into perspective, don't complain about the car, the road, weather, or the cop when you get ticketed for hitting a parked car in the rain while speeding.
Email is highly complex, and GMail are one of the few that are doing it right! (and per RFC's instead of making their own standards like some other unnamed companies do :)
My email server has not given me any problems in a long long time (years) and to see this pop up is annoying. I'm sure google does their best but I really don't see why this particular set of emails suddenly would trigger their spam flag and previous (quite similar) emails from the same originator and from the same mail server would not.
Random checks are random and do not belong in a system like this.
Yes, email is highly complex, I'll grant you that but if you start adding a random element to it then it gets more complex without any apparent reason.
I'd be perfectly ok with all this if I had made changes on my end but I fail to see how not having made changes on my end this suddenly started happening and I'm supposed to be the one at fault.
Random checks do belong in a large scale mail system such as GMail. Their system only checks 1 in x emails for DKIM/SPF.
Could you imagine running DKIM/SPF checks on EVERY single email coming in? That would take considerable compute power, not to mention increasing memory requirements in a system built to minimize such things so that it can be scaled properly.
Your problem is simple. Make your own MTA compliant to any applicable RFC's and GMail will just work. Skipping important items like SPF and DKIM will ALWAYS cause intermittent issues with various other MTA's.
>> I'd be perfectly ok with all this if I had made changes on my end but I fail to see how not having made changes on my end this suddenly started happening and I'm supposed to be the one at fault.
You're making the assumption that every delivery is going to the same server, running the same code, which tests the same checks... every time. - It doesn't, and won't. Outcomes can change in a system designed to flex and deal with current problems.
> Random checks do belong in a large scale mail system such as GMail. Their system only checks 1 in x emails for DKIM/SPF.
I'd imagine google has by now figured out how to make a cache work.
> Could you imagine running DKIM/SPF checks on EVERY single email coming in?
No, but I can imagine keeping a set of cached data per origin address.
> You're making the assumption that every delivery is going to the same server, running the same code, which tests the same checks... every time. - It doesn't, and won't.
I should also note that no email provider should be trusted for 'important email' as mentioned in the article title.
Two reasons.
1) Email is not built with security in mind. Don't suggest SSL as it only protects (somewhat) while in transit and not at rest. You can also bet that not every recipient you send to is using an SSL-enabled POP3/IMAP connection to check mail, so you can't control that end of the conversation.
2) It has no delivery assurance and wasn't built with that in mind. You can never be sure if it arrived, or will be read (by an actual person).
That said, if you need to transmit important documents, use FAX (if the destination has their own (not shared) FAX. Or simply drop it off.
If you need delivery assurance, pick up the phone and ask if they got it, or send if via courier with signature request on delivery.
My belief is that this is mostly this is Google implementing a variety of anti-spam measures rather ineffectively. That and a general consensus that "running your own email server" is a somehow seditious act.
Like Jacques I've run my own mail server for decades, from sendmail, to qmail, to postfix there hasn't been all that much to worry about other than dealing with folks who would try to break into your system through bugs, or exploit your server to send spam.
But Google is such a tempting target for spammers, probably carrying a notable fraction of all the worlds email. Once you get "inside" you can deliver your spam to millions, if not billions if you're careful. So I understand their problem.
That said, it would seem possible to set up a system for "whitelisting" yourself with Google, something like Clear for travellers, an API key perhaps. But that would actually increase the cost of running gmail and that would cause it to lose even more money and well that is something Google can ill afford these days. And I realize that Google is unable to post strict rules "if you do this we will accept your email" because, well spammers would do all those things.
We are but collateral damage of the battle amongst those who would exploit the network for gain.
The reverse is also true -- I've seen a noticeable uptick in gmail-originating spam hitting my servers. Currently my software is configured to keep anything originating from google.com whitelisted, but it's getting a little obnoxious and I might have to reconsider.
One of the more annoying nuisances hitting me personally is from a YC startup, Zenefits. I really dislike them and their moronic hey-buddy spam and that I can't reasonably blackhole the originating network for a while.
72 comments
[ 5.0 ms ] story [ 133 ms ] threadOn one occasion the problem was so sever that we switched to using (the irony) a google docs document to continue the conversation.
I do occasionally check my spam folder and there's always a few false positives - but usually nothing critical (although admittedly I might have missed some that were and never known it)
So - day to day - I currently regard Gmail as 'good enough' but some of that is down to blind trust with little verification.
Can anyone else share their experiences - positive as well as negative?
Could be that other GMail users don't know how to unsubscribe from GitHub/other notifications and mark these as spam to get rid of them which introduces false positives into the spam notification system.
Just curious about the main motivation; not so much to comment on your particular choice for using google free services for your corporate communications.
I'm increasingly finding the following scenarios to occur when using Gmail:
- Letting email I've previously marked as SPAM (same subject, same sender) through the SPAM-filter as valid email. Again. This week.
- Marking non-SPAM email from known contacts I've emailed to and received emails from in the past marked as SPAM for completely non-obvious reasons.
So you DO get spam, and you DO get false positives, meaning you also need to check your SPAM folder for important email.
Basically the one feature which once made Gmail great is now utterly broken, and since for most parts of the internet people now equate Gmail with Email, Google has now officially broken email which was supposed to be decentralized and thus not vulnerable to stuff like this.
Great stuff there, big G. Great stuff. From the rest of the internet: Thanks a million.
Slightly histrionic, don't you think?
This was the the status quo a few years back and still is for many mail providers.
Though certainly 'I've previously had an email conversation with this person' should override any spam detecting heuristics.
From what I've seen while debugging such issues the problems were always from misconfigured outgoing servers and their dns, so I would be very interested in knowing why this has happened.
ps. You might consider enabling comments on your blog so there is a place for such discussions.
I guess this is Google's way of telling me that I should switch to Gmail if I expect to see my mail being delivered.
She uses gmail and his messages were dropping into her junk folder. She almost exclusively uses her phone for her gmail access, so she never saw the messages.
Since gmail doesn't give anything like SpamAssassin headers for messages, there's nothing I can do to troubleshoot this aside from trial-and-error.
Gmail's support for this is not very good - for example, if someone with a SPF record e-mails an address that forwards to gmail it gets marked as an SPF failure because the forwarding server isn't an approved sender for the sender's domain. Unless your forwarding server supports SRS, which is very rare.
This blog post is extremely worthless. The 'root cause' he mentions at the end isn't discussed at all.
SPF records are REQUIRED to get past Gmail's spam filtering.
Add the SPF record, damn.
A properly configured email server would have both SPF and DKIM signing.
Google's spam filter is a complicated black box. You're not going to get far understanding it by prodding it. But you're certainly not going to pass it by ignoring SPF.
I don't know what your history is, but there's piles of bugs that I've had to fix by just trying random things that seem vaguely related to see what happens. If I refused to try anything until I was 100% sure of the root cause, then lots of bugs I've been assigned would never have been fixed, and many more would have taken 10x longer.
If you are still getting some emails through, they must have other characteristics that make them look less like spam than the others that are getting filtered.
Just add the SPF.
This issue surely does not affect me alone hence the warning, the sender may have no idea their messages get dropped at all.
SPF is a DNS record on that domain name. You control that record, not your email provider.
Set that and watch this problem disappear.
To stop the trapped emails ending up in the spam folder, I had to create a custom filter: simply marking mails as 'not spam' in the spam folder does not do the job.
And the filtered mails now have a yellow banner plastered across them saying "This message was not sent to Spam because of a filter you created."
I'm going back to Mutt and my ISP's mail server.
One one of the sysadmin professional organization mailing lists, email from one of the people speaking for the organization started to be marked as spam because the person had DMARC instructions on their domain, but messages remailed from the mailing list server weren't authorized. In this case, Google's mail servers were acting as instructed. I sent the person copies of the headers showing why Google's mail servers were taking this action, as, apparently, did a number of other people. The person became irritated and decided that people who use Google's mail servers needed to fix their own problems.
If your DMARC configuration is causing your messages to be marked as spam, you really should look into that.
If you appear to speak for a sysadmin professional organization, and your DMARC config causes your email to be marked as spam, you really need to fix that, and consider that blaming the recipient mail servers does not do good things for your credibility.
Of course, there are plenty of things that can go wrong other than DMARC, and some of those might well be Google's fault. You can typically track down the reasoning for a Google mail server spam decision from the headers of the delivered message.
Like others, I'm interested in what root cause you've determined, if you can share.
Whether it is useful or not depends on the contents of your spam folder and already one person in this thread reports that for them it certainly was useful.
I'm not sitting here making stuff up and I seriously resent your suggestion that this is false.
There's zero evidence to show that, "Gmail not to be trusted with important email.", as the title suggests. None of it is factual. All of it is presented as-is with only what appears to be a very poor understanding of MTA configuration issues.
The answer has been here all along early in the thread. DKIM & SPF yet the complainant has simply shrugged it off with, "Well that would cause consistent issues." (which is also false.)
If he can not do basic troubleshooting or understand the importance of the SPF/DKIM settings in a modern MTA system, and as a basic starting point, he can not assert that Google can not be trusted with important emails.
So. False, and useless.
1. some Gmail users have told me they don't receive email from my non-gmail server/address, even though I set up SPF, reverse DNS lookup and everything else that was recommended except for DKIM. (The domain was bottoms.christianjaeger.ch.)
2. I've gotten a lot of false positives from mailing lists in my Gmail spam folder for a long time. Now the part that really pisses me off is that Gmail's learning mechanism, if present at all, does not seem to work at all. Why otherwise would mails from the same mailing lists, the same authors, with the same kind of content (e.g. "[PATCH]" in the subject, patch in the mail body) still make it to the spam folder with the noted reason "looks similar to other mails treated as spam"?
3. On top of that, every about 10 mails I mark as spam the Gmail UI asks me whether I'd like to submit it to the Gmail spam team to help improve the service. I never got any notice from them, nor does this seem to have helped.
4. I think I might be spending more time now going over my spam folder and unspam false positives there than I was going over my inbox in the past and tag the false negatives when I was hosting my own mail and used spamsassassin and grey listing.
5. It's getting worse and worse. Now all messages from my servers (crontab and other system mail) are going to spam, too[1]. So basically I now have to move to something else, and I'm ready to use my own hosting again, but I don't know what I should do to solve the "Gmail users won't receive my mail anymore" problem.
[1] e.g. From address "chris@leafpair.com" with "Return-Path: <chrisirc@a.christianjaeger.ch>", or From "bottomsmail@a.christianjaeger.ch" with that same address as Return-Path. (Also a header from Google says "Received-SPF: pass (google.com: domain of chrisirc@a.christianjaeger.ch designates 78.47.51.206 as permitted sender)" or "Received-SPF: pass (google.com: domain of bottomsmail@a.christianjaeger.ch designates 78.47.51.206 as permitted sender)" for the latter.)
Edit: add examples
I believe it is just a well meaning extension of Gmail's spam filter but it has real anti-competitive results. I had to take the pragmatic course of action but I now view email as inherently unreliable.
If any Google employees are here, consider adding the rule "if email address in (list of emails addresses sent to) then NOT SPAM". It would make a lot of people's lives easier.
Additionally I'd like to still receive emails from friends even if they mention vi@gr@ too much.
Google is pretty efficient at eliminating phishing scams. But sure enough I found an email from my actual bank, offering me a credit card, tagged as phishing. Then again, when I looked at the sender domain I was half convinced this was as accurate as its categorisation of all the other phishing attempts
(Dear Santander. If I have to Google the yoursantander.co.uk domain because I'm not confident it's actually yours, you probably shouldn't be using it to try to get me to sign up for credit cards. If the top result is a three year old open letter written by someone else complaining about the same negligent approach to internet security...)
Similarly, there's an Amazon deal email in there, but there's also an explanation on the top that it's "failed amazon.co.uk's required tests for authentication", which might well not be Google's screwup
I was less impressed when Gmail filtered an innocuous email from my dad...
To put it into perspective, don't complain about the car, the road, weather, or the cop when you get ticketed for hitting a parked car in the rain while speeding.
Email is highly complex, and GMail are one of the few that are doing it right! (and per RFC's instead of making their own standards like some other unnamed companies do :)
Random checks are random and do not belong in a system like this.
Yes, email is highly complex, I'll grant you that but if you start adding a random element to it then it gets more complex without any apparent reason.
I'd be perfectly ok with all this if I had made changes on my end but I fail to see how not having made changes on my end this suddenly started happening and I'm supposed to be the one at fault.
Random checks do belong in a large scale mail system such as GMail. Their system only checks 1 in x emails for DKIM/SPF.
Could you imagine running DKIM/SPF checks on EVERY single email coming in? That would take considerable compute power, not to mention increasing memory requirements in a system built to minimize such things so that it can be scaled properly.
Your problem is simple. Make your own MTA compliant to any applicable RFC's and GMail will just work. Skipping important items like SPF and DKIM will ALWAYS cause intermittent issues with various other MTA's.
>> I'd be perfectly ok with all this if I had made changes on my end but I fail to see how not having made changes on my end this suddenly started happening and I'm supposed to be the one at fault.
You're making the assumption that every delivery is going to the same server, running the same code, which tests the same checks... every time. - It doesn't, and won't. Outcomes can change in a system designed to flex and deal with current problems.
I'd imagine google has by now figured out how to make a cache work.
> Could you imagine running DKIM/SPF checks on EVERY single email coming in?
No, but I can imagine keeping a set of cached data per origin address.
> You're making the assumption that every delivery is going to the same server, running the same code, which tests the same checks... every time. - It doesn't, and won't.
Indeed. Unreliable by design.
Two reasons.
1) Email is not built with security in mind. Don't suggest SSL as it only protects (somewhat) while in transit and not at rest. You can also bet that not every recipient you send to is using an SSL-enabled POP3/IMAP connection to check mail, so you can't control that end of the conversation.
2) It has no delivery assurance and wasn't built with that in mind. You can never be sure if it arrived, or will be read (by an actual person).
That said, if you need to transmit important documents, use FAX (if the destination has their own (not shared) FAX. Or simply drop it off.
If you need delivery assurance, pick up the phone and ask if they got it, or send if via courier with signature request on delivery.
Like Jacques I've run my own mail server for decades, from sendmail, to qmail, to postfix there hasn't been all that much to worry about other than dealing with folks who would try to break into your system through bugs, or exploit your server to send spam.
But Google is such a tempting target for spammers, probably carrying a notable fraction of all the worlds email. Once you get "inside" you can deliver your spam to millions, if not billions if you're careful. So I understand their problem.
That said, it would seem possible to set up a system for "whitelisting" yourself with Google, something like Clear for travellers, an API key perhaps. But that would actually increase the cost of running gmail and that would cause it to lose even more money and well that is something Google can ill afford these days. And I realize that Google is unable to post strict rules "if you do this we will accept your email" because, well spammers would do all those things.
We are but collateral damage of the battle amongst those who would exploit the network for gain.
One of the more annoying nuisances hitting me personally is from a YC startup, Zenefits. I really dislike them and their moronic hey-buddy spam and that I can't reasonably blackhole the originating network for a while.