Unlike virtually all other email providers, Microsoft does not cooperate if you follow standard email practices. You can follow all their instructions, sign up for their monitoring tools, submit issue reports, try to build "reputation" and if you are self-hosting they will still have a high likelihood of silently dropping your emails after their server tells you it got them, or putting them in the recipients spam even after being marked as an approved sender.
Just ignore that it’s Microsoft breaking a legacy standard to advantage their other products that compete against email. I think they should start fining companies in stock. Dilute the shareholder value by 5% and see how quickly things change.
Moving back to a full MS stack after 15ish years away from it is so disappointing. I’d assumed things had got better.
Their previously good products are now some sort of split personality monster that sort of work together but mostly just exist near each other and have a common logo. Teams is a mushy world between the distant parts.
Go on, try change a footer on a “Word” document or move an image. Try and copy a document and change its contents and re-upload. It might work, and it might work most the time. But when it goes wrong it’s devastating.
Try to email your dad first. If you're the one initiate the conversation, then your dad will be able to reply to that same thread. After that, add him to contact list once he responded. The point is to make an attempt of your self initiate the connect to that server.
Even IPs that aren't sent any email in last 3 months.
Then one day the problem just disappears and I was able to send email just fine. Till this day, I still don't know what happen.
Another similar service is apple icloud, which I think even much more worse than Hotmail. At least, hotmail give you a portal that you can see some data.
With the announcement of apple icloud email service, I imagine people who are self-hosted email gonna have a lot issue with them.
Had the exact same experience when I worked at a non-profit: we had a self-hosted Postfix server, and our subscriber emails would get silently dropped. This only ever happened for Outlook.com users, and we had set up SPF, DKIM, and DMARC long ago. We got the same run-around with Microsoft support, and eventually we had to give up and move to using Google's SMTP Relay service.
It’s tacit collusion between the big providers to force all businesses onto their overpriced cloud services. One Office 365 user costs the same as 50 users at a shared host.
>Biggest problem is, that Microsoft always blacklists the whole server! If only one of my customers does something not in line with the secret rules of SmartScreen, emails of all other customers are blocked too. IMHO this is just unprofessional. Why not just block the domain under suspicion and send the adminstrator a message (maybe as part of the JMRP)?
The reason for blocking IP originates from a culture set before DKIM. Any IP could shoot out emails for any domain. You don't want to blacklist a domain for an email it did not legitimately author.
However, in a post DKIM world, where adoption is above 80% this could be a feasible solution at least for emails carrying a DKIM signature. The domain owner has provided proofs that the IP address is being used by them to send out emails. Flagging emails from a domain and IP could be blocked while allowing other domains from the same IP to operate.
Couldn't a matrix client with email and asynchronous features be a drop-in replacement for email ? Wouldn't that be a killer product ?
SPF, DKIM, and DMARC are not to core to email protocols if I understand correctly.
There shouldn't be a way for a matrix a client to delete a message sent to another client to keep with the email usage and expectation. So matrix messages wouldn't necessarily be used as email text messages but attached documents to messages would carry email that a client would store and archive.
Email is a delay/disruption tolerant network, and each hop in the network has a positive acknowledgment that the message has been committed to stable storage (a high bar). If the final destination rejects the message a delivery failure message is sent to the originator if one was specified (it's not mandatory), otherwise the postmaster is notified.
In general, I think whatever you are suggesting will not have all the same guarantees of delivery as email.
> and each hop in the network has a positive acknowledgment that the message has been committed to stable storage
If that were true there wouldn't be silent drops in the last mile.
> In general, I think whatever you are suggesting will not have all the same guarantees of delivery as email.
ACK is not to guarantee that messages are delivered though. It's a mechanism that ultimately would let the sender know the message was most likely not delivered.
Let me know when that new client is default installed on the phones and laptops of, say, 70% of all people. I’ll start taking a look at it and evaluate it against standard smtp/imap/pop3 then.
As the article mentions, Microsoft’s server was telling the sender that it had accepted it. An ACK had taken place. But there’s no point in having an ACK if the recipient lies about it.
Tbh queued for delivery doesn't mean the email has been successfully delivered, BUT, the mail server should inform the user if a delivery cannot be performed.
Most probably this whole problem realted to silently dropping emails is intended behavior, a type of shadowbanning applied to emails to prevent spammers from knowing their email was blocked.
Too bad that this in turn causes huge issues for legitimate users
If there is a delivery failure after acceptance of a message, the
receiver-SMTP MUST formulate and mail a notification message. This
notification MUST be sent using a null ("<>") reverse-path in the
envelope. The recipient of this notification MUST be the address
from the envelope return path (or the Return-Path: line).
And:
As discussed in Section 7.8 and Section 7.9 below, dropping mail
without notification of the sender is permitted in practice.
However, it is extremely dangerous and violates a long tradition and
community expectations that mail is either delivered or returned. If
silent message-dropping is misused, it could easily undermine
confidence in the reliability of the Internet's mail systems. So
silent dropping of messages should be considered only in those cases
where there is very high confidence that the messages are seriously
fraudulent or otherwise inappropriate.
The RFC makes clear the long-standing principle that a mail-server operator is entitled to make his own rules as to what mail he is and isn't willing to accept. But he's NOT allowed to accept it for delivery, and then silently discard it.
Hotmail was always awful for this. There are comments up-thread to the effect that Microsoft might be trying to squeeze out self-hosted mailservers. I suspect that was the case in the Hotmail days. Either it worked, and nobody self-hosts any more (FALSE) or it didn't work, and they should have given up. I suspect the same policies are still in place due to corporate inertia.
You can't ever replace email because Microsoft, Google and all the other big players will try to push their own proprietary solution instead of agreeing on a new federated protocol.
I have had the exact same experience. Even put the senders on your whitelist and it makes no difference at all. It means that Office 365 cannot be trusted for handling mail. When it comes time to renew, after two years of putting up with this, I am moving off it.
As someone who worked on reputation at Microsoft, it sounds like a bad case of the right hand not talking to the left. Outlook SmartScreen judgment should be available to tier 3 support along with the IP block list check as a primary investigation step. The author should not have needed to go through tier 3 on two tickets before escalating to someone who had visibility into SmartScreen judgments. Hopefully this blog gets some publicity and Microsoft support amends their investigation process, as I’m sure the author is not the only person running into this issue.
All opinions are my own and not that of Microsoft.
I've worked extensively with Microsoft support on deliverability issues, and I don't think it's a case of the right hand not talking to the left, it's a case of support wanting to close out issues as quickly as possible due to misaligned incentives. Microsoft has zero incentive to help small-time senders like the OP, since recipients are much more likely to blame the OP for delivery issues than Microsoft. Plus, Microsoft has no way of knowing for certain whether the OP is legitimate, so if they choose to help them, they might accidentally help a spammer. Therefore, Microsoft support is incentivized to simply ignore complaints like the OP or give them the run-around, which has been my experience.
The only part of the article that surprised me is the fact they actually got the issue escalated. I've never succeeded in doing that; talking to Microsoft support is like talking to a brick wall.
> The only part of the article that surprised me is the fact they actually got the issue escalated. I've never succeeded in doing that; talking to Microsoft support is like talking to a brick wall.
I’ve found that the best trick with Microsoft is: money talks. If you pay them for support suddenly you should have a much easier time getting things done.
Obviously paying MS to fix a problem they caused is abhorrent but sometimes you gotta hold your nose and do it.
> I’ve found that the best trick with Microsoft is: money talks.
It depends on the quantity. MS365 Business Basic/Premium customers are basically testers for the enterprise offerings. Look at release cadences for the different tiers and it becomes obvious.
> talking to Microsoft support is like talking to a brick wall
I once had DKIM keys get out of sync with DNS in Office 365. It’s supposed to be handled automatically and that should never happen. Support was as useful as a baby trying to do calculus. It took me 3 days to get the keys rotated and after I understood how it works I realized they should have been able to help me mitigate it right away (dkim keys can be rotated while disabled).
Does Smart Screen filter email originating from within the organisation that sends it?
My organisation used to self host everything -- wonderful times; everything worked, but with a HTML 1.0 crappy web interface. About four years ago they shifted to Exchange (urgh -- suddenly IMAP errors and some of my message-read state information became lost. I also noted the massive decrease in my Mac's battery life as Mail seemed to have a lot more work to do -- but I digress) and about two years ago self-hosted exchange became O365.
They recently started pushing harder into moving everything into Microsoft's cloud, and my quality of life has massively decreased -- the first time I was driven insane by it was when my colleague's human-written emails to me ended up in my spam folder; and vice versa when I emailed six undergraduate students to set a date and time for a tutorial. I turned up, they didn't, as I'd been shadowbanned from sending a five-line message to recipients within the same organisation -- messaging them all together got it dropped, sending it individually without the CC was fine. I complained and got given the run-around for a few weeks with no answer. The response of my local "IT people" was "not our fault mate".
The other thing I particularly hate about Microsoft 365 is that all the user-facing error messages are "Sorry, something went wrong". I'm not an idiot, please don't treat me like one.
The problem, of course, is that Tier 1 support are outsourced and a huge part of their evaluations are how many tickets they close. Escalating hurts their numbers, so it's hard to get cases escalated, especially if you're not a paying customer.
I had this problem repeatedly with both Google and Microsoft, though never at the same time and never, so far as I could tell, for the same reason.
Everything was set up properly. The domains I hosted had IPv4 and IPv6 addresses with forward and reverse DNS matching exactly. SPF was set perfectly, according to multiple test sites. DKIM was always used and always validated, again according to multiple test sites. I even signed up for a DMARC endpoint account at report-uri.com.
Didn't matter. Messages would be spontaneously rejected, sometimes in mid-conversation (this happened to my wife quite a lot). Other times, the remote end would claim to accept the message and then it would disappear into the great bit bucket in the sky. It would be like this for a day or two then spontaneously go back to working.
I finally gave up and switched entirely over to Fastmail awhile back. No more delivery problems, but no more self-hosting my own e-mail, something I'd done for twenty years prior (my domain is older than Mozilla and Google).
> I finally gave up and switched entirely over to Fastmail awhile back. No more delivery problems, but no more self-hosting my own e-mail, something I'd done for twenty years prior (my domain is older than Mozilla and Google).
Can anyone speak to what a provider like Fastmail does behind the scenes that guarantees their delivery? Is it the threat to sue/kick up dirt/raise issues with the appropriate regulatory body if perfectly good emails don't go through? Merely having a seat at some sort of regulatory body or committee?
From what I can tell, there are two main reasons companies like Fastmail don't have these problems. The first is that good email providers spend a lot of time maintaining the reputation of the IP addresses they use and shut down bad actors as quickly as possible. The second is that the more people that use the service, the higher percentage of good quality emails being sent, reducing the effect of the occasional bad actor.
I'm self-hosting my mails. Microsoft expects you to send enough mails to consider your mails not as spam. One thing Fastmail does that people who self-host can't is: sending enough mails.
I don't know what other measure they may take but this alone probably goes a long way.
And yes, this is fucking frustrating. I'm not going to send thousands of mails per months just for the sake of it and don't know how I should do that without spamming. This is so fucked up.
This is definitely something I've never heard of, thanks for mentioning it -- I had no idea there was such a consideration (or even that it would be a good way to filter out scammers)
Hasn't it been common knowledge for over a decade now that self-hosting email isn't "allowed" in practical terms? Sure, you can put up the server, but most people are on one of the big providers, and chances are they've got your IP range blocked, because nearly every IP range you might be using has probably been owned by spammers at some point.
Email was actually one of the first aspects of the "classic" Internet to fall to the gentrification of the tech monopolies, long before the web in general. In theory, it's an open ecosystem anyone can jump into. In reality, unless you have a lot of energy/time/money to get around the big providers banning you for "spam," it's not going to happen.
To be fair, I was also surprised to read that anyone was still using Hotmail, so maybe I just don't have my finger on the pulse of the hip new generation.
FWIW, not for me either. I run mail servers for a small(ish) organisation.
In fact, I'd say it's getting easier, not harder. There was a time when fairly arbitrary rules were applied (like "thou shall not send email from a dialup IP Address). They seem to have mostly gone now, and when they are still in place you can just self report via a web form you are not being naughty and it's all good.
But you do have to keep your nose clean. Do not send SPAM, under any circumstances. Sounds easy, and indeed it is easy if you are just running a mail server for an organisation where all mail you are sending is generated by people within that organisation. But if you are doing something that involves relaying emails, eg something like running a mailing list, it becomes harder.
To give an example - I run a mail server for a club of maybe 20 people. All email is strictly policed, eg you have to sign up, all sign ups are reviewed by a human, including any email asking to sign up. All good right? Nope, because the club has to have at least one email address (say for the secretary) that does not involve signing up. Those emails are all moderated of course - but you are notified of emails waiting for moderation via email. Those notifications contain snippets of the original. If one of the moderators has a GMail account, those snippets triggering SPAM filters can get your server banned by google. The solution is to run a imap server for secretary@ and get them to fetch it manually. Bypass Google for all relayed email in other words.
So you have to follow the rules and it's not always obvious what you have to do to follow the rules, but it ain't impossible, and in fact once you know what the rules are it isn't even hard. Any capable sysadmin could do it.
It's still doable. I self host and have managed to switch IP's providers and so forth without getting into trouble. It is, however, critical that you make sure you don't get an IP that was recently used by a spammer, and it's a lot of work to get everything up and running. Warming up an IP is difficult when talking volumes that small.
Do you have any tips on warming up your IP? I’ve been going down the self-hosting rabbithole again after years and just set up a server. I’m about to set up a mail server in the coming weeks and checked that my IP is not on any blocklists, but my normal email traffic is very low. Should I automate sending an email to my accounts on major platforms every other day or something along those lines?
Send authenticated mails. Send an email or two to your own Hotmail/Gmail address each day (or at least regularly), and make sure to interact with the email (read/reply/forward/put in folder/etc.). Personally, I use Hotmail for reading my shopping lists.
If you end up in your own spam folder, "Not junk" is your friend. If you don't get through at all, do the escalation dance with Microsoft until the situation improves.
Once you reliably hit your own Hotmail inbox, start sending mails to A FEW others as well. Verify that they receive the mail, and ask them to add you to their trusted sender lists, press not junk, etc, and keep bugging Microsoft if they just discard your emails silently. Once you have a few people doing that, reputation should improve quickly.
Your are still allowed. I self-host personal mail in a VM and still can send mail to gmail.
The problem is that big players (Gmail/MS) care about an anti-spam false positives only when they affects a relatively large fraction of all messages. Small mail servers account for a small fraction of all messages, so big players don't care about them at all.
I will not be surprised if they have SLI/KPI which allows them reject by a mistake millions of non-spam messages daily because it is a small percent of all messages they process.
> Hasn't it been common knowledge for over a decade now that self-hosting email isn't "allowed" in practical terms?
Well I've been self hosting my emails for 10 years now, and my emails still go through. I'm sure there must be cases where you face issues, but not my in my personal experience.
Only thing I've seen once or twice are websites with "validation" rules like /.@(gmail|hotmail|yahoo)./
I have also been self-hosting my email for almost 20 years.
Until a few years ago I never had any problems, but during the last 3-4 years I had sporadic problems with messages that either were not delivered or they were delivered with a very large delay, e.g. a day, or they were delivered only if I resent them 3 or 4 times, at some of the large providers, like Gmail.
It still works fine most of the time, but when problems happen they are annoying because there is no way to know that something happened, unless I have an alternative way to ask the recipient, e.g. by phone.
In any case, now I always check that at least the final destination e-mail server has accepted the message, which should have been enough to guarantee the delivery, according to standards, but nonetheless this is not true any more.
Yep. For important emails, I tail the exim mainlog. But in six sigma cases, it just works or I get to know that my mails were not delivered days later.
I'm pretty sure 0% of MS's customers asked them to silently drop emails without either recteting the mail at sumission time or dropping it in the spam foler.
I well remember the times when I set up sendmail and saw emails immediately sent in Netscape Mail instead waiting for minutes. This was using a 32kB dialup connection.
Was like 1997. Needless to say it was then sendmail struggling to get the mails over dialup. At least my single-threaded Netscape was responsive.
Worked until 1999 or so. Then spam became a thing and more and more mails were rejected and I folded my setup.
>> Hasn't it been common knowledge for over a decade now that self-hosting email isn't "allowed" in practical terms?
For some time seemingly legit mail from Azure Cloud Services also got flagged by Gmail. If that was the case -- I thought -- what changes do I have for my own hosted or non-major email provider? That is when I decided I couldnt win as a small startup and just went the managed GSuite route for email hosting.
> That is when I decided I couldnt win as a small startup and just went the managed GSuite route for email hosting.
Do you realize people like you giving up is precisely why these evil corporations can get away with what they do? Together, we small providers are strong. But if some resistance fighters drop the struggle and join Dark Vader 's side, we are bound to loose.
Maybe we should start a campaign. Give a one-year notice to those big email provider, in partnership with smaller email providers, saying they have one year to comply with email standards or we'll start boycotting and promoting alternatives as a federation of free/open email providers?
We could start doing like the comment on OP does. Send a rejection notice with a link to some list of all the various blogs documenting how big providers refuse to follow the standards.
> My other thought was to have my Postfix server reject all mail from Microsoft servers, with a message requesting the sender to use a different mail provider and pointing to this article.
I've often been tempted to do that. However, we would actively be hurting our own users who would be incapable to receive mails from such servers. We'd in fact be hurting the very principles of interoperability we're defending in the first place.
Adding an auto-reply or an email footer mentioning that their providers actively hurts the email ecosystem and linking to alternatives would be a good compromise in my view.
Along with a libre email provider federation, that could go a long way...
Some companies reject to communicate to you straight away if you use GMail and don't run your own company EMail. On the other hand some do the opposite and demand you use GMail.
After testing - looks like new sign-ups get a choice of @outlook.com or @hotmail.com (although users with previously available/legacy domains like @hotmail.co.uk retain them).
Apparently the service as a whole has been based on o365 since around 2015 according to wikipedia.
> Email was actually one of the first aspects of the "classic" Internet to fall to the gentrification of the tech monopolies, long before the web in general.
But isn't the federated nature of email the only thing which prevents the said monopolies from taking over email completely? If it can be proven that X giant is blocking mails from Y small email company then they're in lot of trouble at least in EU.
That's the reason chat and everything else is a lost cause; Can't send a message from Messenger to Telegram unless you use their bot API to build a specialized application[1] which is a downgrade from XMPP like interoperable protocol.
[1] A product from my previous startup which allowed communication between Messenger <-> Telegam <-> Viber <-> LINE for dating - https://hitstartup.com/about/#FindDate
Lots of people self-host their email, it works fine. The worst problem I've encountered is graylisting delays. Sometimes Gmail will throw a fit, but that's true whether you're originating from @icloud.com or from @yourtinydomain.com.
A couple years ago I did some superficial analysis on the domains of the registration emails our customers were giving us. I don't remember the details, but IIRC only about 80% of them were connected to one of the big providers (google, apple, qq, microsoft, yahoo(!), etc).
I knew about about but only found out about two years ago, and I'm not one who is bad with tech either. So how common of knowledge it is is debatable I would say.
Meanwhile, I get so much spam on my hotmail account (from obvious spam addresses too) that I’ve had to switch to gmail. It seems both the false positive and false negative rate is sky-high.
> SmartScreen® Filter technology is always adapting and learning more about what is and isn’t unwanted mail, it is not possible for us to offer specific advice about improving your mail content.
Translation: "The filtering is based on machine learning. If there are false positives, too bad, even we don't know what's causing them. Try to look less like spam or something?"
They should be liable for their shitty ML algorithms. It’s extra detrimental to small businesses who get their email dropped for no good reason. Silently dropping email is malicious IMO.
This sounds similar to my experience with Microsoft based email (Outlook, Hotmail, Live, custom 365 domains) I eventually decided to route Microsoft hosted emails over Amazon SES instead of a VPS hosted Postfix setup. This did help a bit. The fact that all other email providers work perfectly fine and Microsoft is the problematic one I think signals an issue with their spam filtering.
Oh hey, Microsoft is breaking another open decentralized standard again as they are pushing their own proprietary alternative. But let's keep pretending they changed.
Speaking as someone who has been running MTAs for more than 15 years - Microsoft has been breaking e-mail for years. They are an order of magnitude more awful than anyone else, and no one should have their e-mail with them for any reason.
Imagine if this was turned around, and simultaneously a huge chunk of the world instantly black-holed Office 365 outbound email.
Then in response to the roaring noise from Redmond, simply send back their canned response of "Not qualified for mitigation. Our investigation has determined that the above IP(s) do not qualify for mitigation."
Do that for 24 hours, then 48 hours, and so on until they smarten up.
I guarantee you they'll have their CEO breathing down the necks of every moron in the spam filter team until they fix it.
Until then? Nothing will happen. Nothing at all. We'll complain on YC News, we'll complain on /r/sysadmin, and a bunch of small businesses will go out of business. Microsoft 365 will continue to grow to consume the world. Email will be rebranded "Teams Mail", and as far as everyone is concerned that will make perfect sense.
Either we act, or we live with the consequences of inaction.
LOL.. watch what happens when you try and suddenly the government labels you as a domestic terrorist group because they are so heavily invested in Microsoft, and it actually helps the government to collect intelligence to only have a few approved conglomerate email providers.
Collectivist action is great, when it's great. But with it comes all the usual ossification problems. (For a collective to have power others have to give it to them. Either the collective manages to secure a monopoly, like the various professional licensing/accreditation bodies, legal bar association, medical residency programs with fixed number of slots. Or stakeholders - companies, professionals, regulators - have to join the collective and give up at least some of their independence during the process.)
Yes, email is kind of important. But it's also pretty shoddy. It always has been. (Confidentiality, integrity, authentication, signed receipt of delivery. All are important for regular business correspondence and by default all were [and still are] hit and miss with email.)
This practice of MS of not accepting email might be abuse of market power. But they can easily say their clients signed up for Outlook to have exactly this protection. What would the courts think? Who knows.
Governments don't like self-hosting, because they need to ask for data every time they need it. For big hosting providers like Google, Microsoft, Fastmail and others they have mails already. So there are incentives to make it harder and harder to self-host...
I started getting this on our transactional mail relay some years ago, hotmail responded with "Not qualified for mitigation", therefore instead of spending pointless effort
dealing with these miserable jollems i used o365 to counter it (my organization was already using o365, which also never did this S3150 thing to me) by:
Adding a relay user to the o365 then configuring all the domains i needed to send from on the o365 tenant (without changing the MX records naturally).
(Using a normal o365 user account to relay is possible for me as our typical customer base is enterprise and education, not random end users)
Adding distribution lists as the from-addresses that i needed to send from and gave send-as rights to the relay user,
compiled a list of hotmail/outlook domains (which i've had to add to over the years but it is relatively stable for me by now),
then configured the outgoing relay server that does all the dkim-stamping to route all outgoing mail to outlook/hotmail domains through o365.
When i see a S3150 in the logs i just add another domain to the list, i should probably make this pre-emptive by looking at the target mx records before sending,
as its not trivial to make exim resend something that ended in a 5xx.
All other mail providers have removed blocks by simply asking, not hotmail. Also all our newsletter-like mail goes through separate services,
these are all transactional mails (meeting invites, user account creation, password resets and such).
"As far as I know, emails from my business custmers are branded, formatted following best practices, have unsubscribe links and a valid contact footer."
In other words, his customers are sending out the type of emails that actually require unsubscribe links. You know, _spam_. So maybe it isn't Microsoft who is the bad guy here?
Microsoft is telling this guy they received the email for delivery and then silently dropped it. If you're not going to deliver an email you either inform the sender or shutdown your mail servers and remove your MX.
> Despite all the outrage
Hah! Please, there's hardly any. Certainly nowhere close the outrage this deserves.
Oh please. It was the commercial spammers that destroyed email as a useful communication medium. Not sending all sorts of useless notifications is just another defence mechanism against those a*holes. Once in a while I still get an undeliverable notification for Chinese spam; they just configured my personal email as their return address, and some stupid mail server decided to tell me it couldn't accept the email. Great job guys, now you're spamming innocent bystanders!
So yes, telling people that their spam-load was accepted, and then just not delivering it, seems perfectly fine to me. It's a tough world out there, and it was made that way by precisely the kind of people that need "unsubscribe" links in their emails to begin with. I feel no sympathy for him.
These days communication uses web forms, Teams, Skype, Facebook, Twitter, or any other electronic mechanism that isn't email. They are all regulated by their respective owners, so they have a slightly better chance of surviving the onslaught of the advertisers. Email works fine for contact with friends and family, but not for spam, and that's how it should be.
Backscatter spam only happens with a misconfigured email service. If you are not going to deliver the mail, reply with an error during submission, don't send a automatic notification email to the return address.
Yes where I live you have to specifically opt in for email communication.
I use Outlook and the few companies who still pull this shit off end up in the spam box courtesy of machine learning.
If MS is making life worse for businesses they have my support.
These details about the proprietary "SmartScreen" were really useful.
We have been sending login code emails via AWS SES and have had a lot of issues with Hotmail/Outlook.com not delivering them.
My takeaways:
1) There's a risk that a shared IP system like AWS SES will get blocked by SmartScreen based on emails someone else is sending.
2) We really should look at the contents of our (trivial) transactional emails, too.
Even the user side of Microsoft's email is confusing and prone to loosing emails. O365 defaults or at least the recommended settings will just quarantine emails without notifying anyone. I guess Microsoft doesn't trust users to not open emails in the spam folder?
I think I have it configured now to alert the recipient and me on quarantined emails but there are at least two relevant dashboard (Exchange and umm Security Center? who knows what the current names are today) that probably have like 35 submenus and 1000+ settings. Running my own email server would be easier.
Another thing, a few months ago I noticed a couple incoming emails were not quarantined/blocked but also not delivered. I don't think the issue has reoccurred since but it's just crazy that emails are just lost sometimes.
It's common knowledge that Microsoft considers everyone to be spammers until proven innocent. Getting to a point where your mails end up in the recipients' inboxes is a long path -- Microsoft doesn't care if you set everything up perfectly, as spammers can do the same -- and one has to provide them with enough data to learn to trust one's IP's/ranges/domains.
Contacting Microsoft means you have to go through a difficult dance through multiple canned replies, multiple escalation requests, and being lucky enough to get someone to actually escalate.
Silently dropping emails isn't Microsoft-specific, though. It's quite common to have a Postfix/Amavis setup set to discard spam over a certain spam score threshold.
When these long-winded issues arise, you need to call people up. Email clearly isn't working.
You'd be surprised (or not) by the difference verbal communication makes, not only do your verbal expressions translate better, but the tone of voice may (or not) instill trust in the person on the other end.
On the opposite side, I found Microsoft Azure to be one of the biggest sources of spam on the internet. I have blocked them completely on my private mail server.
Yeah, these problems are consistent with Microsoft email. I've been using Sendgrid to send emails for our SaaS and some customers who use Microsoft email don't receive transactional messages at all. There's nothing you can do about it.
Also, seen this happen with other apps too. Transactional messages don't arrive on Microsoft email addresses.
Does setting up MTA-STS have any effect on deliverability? I can see in my webserver logs that both Microsoft and Gmail/Google are checking the MTA-STS settings at regular intervals.
108 comments
[ 3.2 ms ] story [ 179 ms ] threadUnlike virtually all other email providers, Microsoft does not cooperate if you follow standard email practices. You can follow all their instructions, sign up for their monitoring tools, submit issue reports, try to build "reputation" and if you are self-hosting they will still have a high likelihood of silently dropping your emails after their server tells you it got them, or putting them in the recipients spam even after being marked as an approved sender.
Just ignore that it’s Microsoft breaking a legacy standard to advantage their other products that compete against email. I think they should start fining companies in stock. Dilute the shareholder value by 5% and see how quickly things change.
Moving back to a full MS stack after 15ish years away from it is so disappointing. I’d assumed things had got better.
Their previously good products are now some sort of split personality monster that sort of work together but mostly just exist near each other and have a common logo. Teams is a mushy world between the distant parts.
Go on, try change a footer on a “Word” document or move an image. Try and copy a document and change its contents and re-upload. It might work, and it might work most the time. But when it goes wrong it’s devastating.
They had a dedicated page at https://sendersupport.olc.protection.outlook.com/snds/data.a... that supposed to be used for these delivery issue. I tried to submit multiple requests from there.
Even IPs that aren't sent any email in last 3 months.
Then one day the problem just disappears and I was able to send email just fine. Till this day, I still don't know what happen.
Another similar service is apple icloud, which I think even much more worse than Hotmail. At least, hotmail give you a portal that you can see some data.
With the announcement of apple icloud email service, I imagine people who are self-hosted email gonna have a lot issue with them.
What announcement?
iCloud has had a E-Mail service for over 10 years now. iCloud was doing email back when it was called MobileMe.
So that means one more awkward email providers we have to deal with :)
Apologies, I misunderstood what you were trying to say.
Thanks for clarifying :)
The reason for blocking IP originates from a culture set before DKIM. Any IP could shoot out emails for any domain. You don't want to blacklist a domain for an email it did not legitimately author.
However, in a post DKIM world, where adoption is above 80% this could be a feasible solution at least for emails carrying a DKIM signature. The domain owner has provided proofs that the IP address is being used by them to send out emails. Flagging emails from a domain and IP could be blocked while allowing other domains from the same IP to operate.
Misconfigured email servers, with spam filters and SPF/DKIM setups being the most common issues, drop emails far more often people tend to realize.
Couldn't a matrix client with email and asynchronous features be a drop-in replacement for email ? Wouldn't that be a killer product ?
SPF, DKIM, and DMARC are not to core to email protocols if I understand correctly.
There shouldn't be a way for a matrix a client to delete a message sent to another client to keep with the email usage and expectation. So matrix messages wouldn't necessarily be used as email text messages but attached documents to messages would carry email that a client would store and archive.
In general, I think whatever you are suggesting will not have all the same guarantees of delivery as email.
If that were true there wouldn't be silent drops in the last mile.
> In general, I think whatever you are suggesting will not have all the same guarantees of delivery as email.
ACK is not to guarantee that messages are delivered though. It's a mechanism that ultimately would let the sender know the message was most likely not delivered.
Let me know when that new client is default installed on the phones and laptops of, say, 70% of all people. I’ll start taking a look at it and evaluate it against standard smtp/imap/pop3 then.
But in these days and age of webmail I am not sure 70% of people have a mail client.
I don't really understand the argument of pitching mass adoption against something that doesn't exist yet.
As the article mentions, Microsoft’s server was telling the sender that it had accepted it. An ACK had taken place. But there’s no point in having an ACK if the recipient lies about it.
Most probably this whole problem realted to silently dropping emails is intended behavior, a type of shadowbanning applied to emails to prevent spammers from knowing their email was blocked.
Too bad that this in turn causes huge issues for legitimate users
The RFC makes clear the long-standing principle that a mail-server operator is entitled to make his own rules as to what mail he is and isn't willing to accept. But he's NOT allowed to accept it for delivery, and then silently discard it.
Hotmail was always awful for this. There are comments up-thread to the effect that Microsoft might be trying to squeeze out self-hosted mailservers. I suspect that was the case in the Hotmail days. Either it worked, and nobody self-hosts any more (FALSE) or it didn't work, and they should have given up. I suspect the same policies are still in place due to corporate inertia.
All opinions are my own and not that of Microsoft.
The only part of the article that surprised me is the fact they actually got the issue escalated. I've never succeeded in doing that; talking to Microsoft support is like talking to a brick wall.
I’ve found that the best trick with Microsoft is: money talks. If you pay them for support suddenly you should have a much easier time getting things done.
Obviously paying MS to fix a problem they caused is abhorrent but sometimes you gotta hold your nose and do it.
It depends on the quantity. MS365 Business Basic/Premium customers are basically testers for the enterprise offerings. Look at release cadences for the different tiers and it becomes obvious.
I once had DKIM keys get out of sync with DNS in Office 365. It’s supposed to be handled automatically and that should never happen. Support was as useful as a baby trying to do calculus. It took me 3 days to get the keys rotated and after I understood how it works I realized they should have been able to help me mitigate it right away (dkim keys can be rotated while disabled).
Not sure how things are organised in microsoft, but it's one of worst support ever happened to me
My organisation used to self host everything -- wonderful times; everything worked, but with a HTML 1.0 crappy web interface. About four years ago they shifted to Exchange (urgh -- suddenly IMAP errors and some of my message-read state information became lost. I also noted the massive decrease in my Mac's battery life as Mail seemed to have a lot more work to do -- but I digress) and about two years ago self-hosted exchange became O365.
They recently started pushing harder into moving everything into Microsoft's cloud, and my quality of life has massively decreased -- the first time I was driven insane by it was when my colleague's human-written emails to me ended up in my spam folder; and vice versa when I emailed six undergraduate students to set a date and time for a tutorial. I turned up, they didn't, as I'd been shadowbanned from sending a five-line message to recipients within the same organisation -- messaging them all together got it dropped, sending it individually without the CC was fine. I complained and got given the run-around for a few weeks with no answer. The response of my local "IT people" was "not our fault mate".
The other thing I particularly hate about Microsoft 365 is that all the user-facing error messages are "Sorry, something went wrong". I'm not an idiot, please don't treat me like one.
Having to write this is so ridiculous. Better just use a throwaway and be yourself. Or is this PR?
PR will mention a minor technical problem that has been solved, nothing to worry about.
Everything was set up properly. The domains I hosted had IPv4 and IPv6 addresses with forward and reverse DNS matching exactly. SPF was set perfectly, according to multiple test sites. DKIM was always used and always validated, again according to multiple test sites. I even signed up for a DMARC endpoint account at report-uri.com.
Didn't matter. Messages would be spontaneously rejected, sometimes in mid-conversation (this happened to my wife quite a lot). Other times, the remote end would claim to accept the message and then it would disappear into the great bit bucket in the sky. It would be like this for a day or two then spontaneously go back to working.
I finally gave up and switched entirely over to Fastmail awhile back. No more delivery problems, but no more self-hosting my own e-mail, something I'd done for twenty years prior (my domain is older than Mozilla and Google).
Can anyone speak to what a provider like Fastmail does behind the scenes that guarantees their delivery? Is it the threat to sue/kick up dirt/raise issues with the appropriate regulatory body if perfectly good emails don't go through? Merely having a seat at some sort of regulatory body or committee?
I don't know what other measure they may take but this alone probably goes a long way.
And yes, this is fucking frustrating. I'm not going to send thousands of mails per months just for the sake of it and don't know how I should do that without spamming. This is so fucked up.
Email was actually one of the first aspects of the "classic" Internet to fall to the gentrification of the tech monopolies, long before the web in general. In theory, it's an open ecosystem anyone can jump into. In reality, unless you have a lot of energy/time/money to get around the big providers banning you for "spam," it's not going to happen.
To be fair, I was also surprised to read that anyone was still using Hotmail, so maybe I just don't have my finger on the pulse of the hip new generation.
Not for me and loads of my tiny and not so tiny customers. Just keep your nose clean and follow the rules. This is in the UK, elsewhere ymmv.
In fact, I'd say it's getting easier, not harder. There was a time when fairly arbitrary rules were applied (like "thou shall not send email from a dialup IP Address). They seem to have mostly gone now, and when they are still in place you can just self report via a web form you are not being naughty and it's all good.
But you do have to keep your nose clean. Do not send SPAM, under any circumstances. Sounds easy, and indeed it is easy if you are just running a mail server for an organisation where all mail you are sending is generated by people within that organisation. But if you are doing something that involves relaying emails, eg something like running a mailing list, it becomes harder.
To give an example - I run a mail server for a club of maybe 20 people. All email is strictly policed, eg you have to sign up, all sign ups are reviewed by a human, including any email asking to sign up. All good right? Nope, because the club has to have at least one email address (say for the secretary) that does not involve signing up. Those emails are all moderated of course - but you are notified of emails waiting for moderation via email. Those notifications contain snippets of the original. If one of the moderators has a GMail account, those snippets triggering SPAM filters can get your server banned by google. The solution is to run a imap server for secretary@ and get them to fetch it manually. Bypass Google for all relayed email in other words.
So you have to follow the rules and it's not always obvious what you have to do to follow the rules, but it ain't impossible, and in fact once you know what the rules are it isn't even hard. Any capable sysadmin could do it.
If you end up in your own spam folder, "Not junk" is your friend. If you don't get through at all, do the escalation dance with Microsoft until the situation improves.
Once you reliably hit your own Hotmail inbox, start sending mails to A FEW others as well. Verify that they receive the mail, and ask them to add you to their trusted sender lists, press not junk, etc, and keep bugging Microsoft if they just discard your emails silently. Once you have a few people doing that, reputation should improve quickly.
The problem is that big players (Gmail/MS) care about an anti-spam false positives only when they affects a relatively large fraction of all messages. Small mail servers account for a small fraction of all messages, so big players don't care about them at all.
I will not be surprised if they have SLI/KPI which allows them reject by a mistake millions of non-spam messages daily because it is a small percent of all messages they process.
Well I've been self hosting my emails for 10 years now, and my emails still go through. I'm sure there must be cases where you face issues, but not my in my personal experience.
Only thing I've seen once or twice are websites with "validation" rules like /.@(gmail|hotmail|yahoo)./
Until a few years ago I never had any problems, but during the last 3-4 years I had sporadic problems with messages that either were not delivered or they were delivered with a very large delay, e.g. a day, or they were delivered only if I resent them 3 or 4 times, at some of the large providers, like Gmail.
It still works fine most of the time, but when problems happen they are annoying because there is no way to know that something happened, unless I have an alternative way to ask the recipient, e.g. by phone.
In any case, now I always check that at least the final destination e-mail server has accepted the message, which should have been enough to guarantee the delivery, according to standards, but nonetheless this is not true any more.
Iirc at its height 90% of all email was spam.
Was like 1997. Needless to say it was then sendmail struggling to get the mails over dialup. At least my single-threaded Netscape was responsive.
Worked until 1999 or so. Then spam became a thing and more and more mails were rejected and I folded my setup.
For some time seemingly legit mail from Azure Cloud Services also got flagged by Gmail. If that was the case -- I thought -- what changes do I have for my own hosted or non-major email provider? That is when I decided I couldnt win as a small startup and just went the managed GSuite route for email hosting.
Do you realize people like you giving up is precisely why these evil corporations can get away with what they do? Together, we small providers are strong. But if some resistance fighters drop the struggle and join Dark Vader 's side, we are bound to loose.
Maybe we should start a campaign. Give a one-year notice to those big email provider, in partnership with smaller email providers, saying they have one year to comply with email standards or we'll start boycotting and promoting alternatives as a federation of free/open email providers?
> My other thought was to have my Postfix server reject all mail from Microsoft servers, with a message requesting the sender to use a different mail provider and pointing to this article.
Adding an auto-reply or an email footer mentioning that their providers actively hurts the email ecosystem and linking to alternatives would be a good compromise in my view.
Along with a libre email provider federation, that could go a long way...
Apparently the service as a whole has been based on o365 since around 2015 according to wikipedia.
But isn't the federated nature of email the only thing which prevents the said monopolies from taking over email completely? If it can be proven that X giant is blocking mails from Y small email company then they're in lot of trouble at least in EU.
That's the reason chat and everything else is a lost cause; Can't send a message from Messenger to Telegram unless you use their bot API to build a specialized application[1] which is a downgrade from XMPP like interoperable protocol.
[1] A product from my previous startup which allowed communication between Messenger <-> Telegam <-> Viber <-> LINE for dating - https://hitstartup.com/about/#FindDate
A couple years ago I did some superficial analysis on the domains of the registration emails our customers were giving us. I don't remember the details, but IIRC only about 80% of them were connected to one of the big providers (google, apple, qq, microsoft, yahoo(!), etc).
Translation: "The filtering is based on machine learning. If there are false positives, too bad, even we don't know what's causing them. Try to look less like spam or something?"
See also (recent): https://answers.microsoft.com/en-us/outlook_com/forum/all/ho...
Imagine if this was turned around, and simultaneously a huge chunk of the world instantly black-holed Office 365 outbound email.
Then in response to the roaring noise from Redmond, simply send back their canned response of "Not qualified for mitigation. Our investigation has determined that the above IP(s) do not qualify for mitigation."
Do that for 24 hours, then 48 hours, and so on until they smarten up.
I guarantee you they'll have their CEO breathing down the necks of every moron in the spam filter team until they fix it.
Until then? Nothing will happen. Nothing at all. We'll complain on YC News, we'll complain on /r/sysadmin, and a bunch of small businesses will go out of business. Microsoft 365 will continue to grow to consume the world. Email will be rebranded "Teams Mail", and as far as everyone is concerned that will make perfect sense.
Either we act, or we live with the consequences of inaction.
Yes, email is kind of important. But it's also pretty shoddy. It always has been. (Confidentiality, integrity, authentication, signed receipt of delivery. All are important for regular business correspondence and by default all were [and still are] hit and miss with email.)
This practice of MS of not accepting email might be abuse of market power. But they can easily say their clients signed up for Outlook to have exactly this protection. What would the courts think? Who knows.
Adding a relay user to the o365 then configuring all the domains i needed to send from on the o365 tenant (without changing the MX records naturally). (Using a normal o365 user account to relay is possible for me as our typical customer base is enterprise and education, not random end users) Adding distribution lists as the from-addresses that i needed to send from and gave send-as rights to the relay user, compiled a list of hotmail/outlook domains (which i've had to add to over the years but it is relatively stable for me by now), then configured the outgoing relay server that does all the dkim-stamping to route all outgoing mail to outlook/hotmail domains through o365.
When i see a S3150 in the logs i just add another domain to the list, i should probably make this pre-emptive by looking at the target mx records before sending, as its not trivial to make exim resend something that ended in a 5xx.
All other mail providers have removed blocks by simply asking, not hotmail. Also all our newsletter-like mail goes through separate services, these are all transactional mails (meeting invites, user account creation, password resets and such).
"As far as I know, emails from my business custmers are branded, formatted following best practices, have unsubscribe links and a valid contact footer."
In other words, his customers are sending out the type of emails that actually require unsubscribe links. You know, _spam_. So maybe it isn't Microsoft who is the bad guy here?
> Despite all the outrage
Hah! Please, there's hardly any. Certainly nowhere close the outrage this deserves.
So yes, telling people that their spam-load was accepted, and then just not delivering it, seems perfectly fine to me. It's a tough world out there, and it was made that way by precisely the kind of people that need "unsubscribe" links in their emails to begin with. I feel no sympathy for him.
These days communication uses web forms, Teams, Skype, Facebook, Twitter, or any other electronic mechanism that isn't email. They are all regulated by their respective owners, so they have a slightly better chance of surviving the onslaught of the advertisers. Email works fine for contact with friends and family, but not for spam, and that's how it should be.
I use Outlook and the few companies who still pull this shit off end up in the spam box courtesy of machine learning. If MS is making life worse for businesses they have my support.
We have been sending login code emails via AWS SES and have had a lot of issues with Hotmail/Outlook.com not delivering them.
My takeaways: 1) There's a risk that a shared IP system like AWS SES will get blocked by SmartScreen based on emails someone else is sending. 2) We really should look at the contents of our (trivial) transactional emails, too.
I think I have it configured now to alert the recipient and me on quarantined emails but there are at least two relevant dashboard (Exchange and umm Security Center? who knows what the current names are today) that probably have like 35 submenus and 1000+ settings. Running my own email server would be easier.
Another thing, a few months ago I noticed a couple incoming emails were not quarantined/blocked but also not delivered. I don't think the issue has reoccurred since but it's just crazy that emails are just lost sometimes.
Contacting Microsoft means you have to go through a difficult dance through multiple canned replies, multiple escalation requests, and being lucky enough to get someone to actually escalate.
Silently dropping emails isn't Microsoft-specific, though. It's quite common to have a Postfix/Amavis setup set to discard spam over a certain spam score threshold.
You'd be surprised (or not) by the difference verbal communication makes, not only do your verbal expressions translate better, but the tone of voice may (or not) instill trust in the person on the other end.
It's working just fine for most people. Email sometimes isn't working for Microsoft customers - but that is entirely Microsoft's doing.
Also, seen this happen with other apps too. Transactional messages don't arrive on Microsoft email addresses.