HN Explorer

Ask HN: Yahoo disregarding RFC 5321 retries

24 points by beezle ↗ HN
Had a firewall issue that temporarily affected port 25 but was not identified for just over 24 hours. In the process of resolving, a test message was sent from a yahoo account to the domain in question.

Yahoo sent an 'unable to deliver message after multiple retries, giving up' message after exactly three hours.

Three hours is exceedingly short and my understanding has always been that mail is re-tried for a number of days with a backoff algorithm controlling how often.

Is this now common? I am left to wonder how much mail was lost for good during this outage because others have also taken the Yahoo approach.

RFC 5321 suggests:

Retries continue until the message is transmitted or the sender gives up; the give-up time generally needs to be at least 4-5 days. It MAY be appropriate to set a shorter maximum number of retries for non-delivery notifications and equivalent error messages than for standard messages. The parameters to the retry algorithm MUST be configurable.

36 comments

[ 5.0 ms ] story [ 77.5 ms ] thread
paxys
There's a big difference between desktop mail clients that have an outbox and web-based experiences of services like Yahoo and Gmail. I doubt any of the latter kind will retry sending an email over several days.
tedunangst
SMTP relays should generally, and have in the past, retried for several days.
foxtrottbravo
There may be a plethora of differences but the retry delivery has nothing to do with the client you're sending your mail from.

That is definitely in the realm of the server being tasked with delivering your mail.

In traditional desktop Mail clients the Outbox is a local folder and stores Mail that hasn't been delivered to the Server tasked with sending out the mail.

The retry method mentioned here is about a Mail that has been delivered to the outgoing mailserver which in turn may have trouble reaching the destination server

dsr_
If they wanted to be good email neighbors, they would.

The other evidence, however, suggests that "good email neighbor" is not a goal for either Yahoo or Google.

rblatz
The expectation among the general population is that email is delivered in minutes. Any deviation from that timeline is a negative user experience. Failing fast and telling the sender is the best approach. Silently failing to deliver an email until 3 days later would cause more issues.
rescbr
At least some years ago, Gmail used to retry for 2 or 3 days before giving up.
londons_explore
It still does in some cases.

For example of you send a message from Gmail to another Gmail user with a full mailbox, it keeps retrying delivery for many days.

joshu
you are confused about the difference between MTA and MUA
tedunangst
1. They probably figure "the network is reliable" and therefore any failure that isn't short lived is permfail. I would disagree, but I imagine their own user base complains that they don't get bounces until days later. "That was a very important serious business email. How dare you not tell me it wasn't delivered promptly!" (And I imagine the old standard of a warning email that message hasn't been delivered but retries are ongoing isn't handled well by many people.)

2. I'd double check you didn't bring the server back online but misconfigured such that it rejected the message.

_moof
> And I imagine the old standard of a warning email that message hasn't been delivered but retries are ongoing isn't handled well by many people.

Can you expand on this? It's been a while since the previous century but I don't recall those messages ever being confusing.

tedunangst
I assume there's a lot of overlap between yahoo users and people incapable of reading words on screens. The message can clearly say "Email has not yet been delivered, but I will continue retrying." And users will frantically email yahoo support what do I do, what does this mean, how do I fix it, have I been hacked?

People refuse to read messages from computers because those could only be words that only computer nerds understand.

londons_explore
The fact the messages didn't have any consistent UI and frequently had lots of technical mumbo jumbo (ie. All the message headers) made them really user unfriendly.

They also didn't localize to the user's language when sent by remote servers, and were frequently sent at 3am (did you want your phone going ding new email at 3am?)

jedberg
Back in the day when the RFC was written it was common for people to run their own mail servers and have them be offline for a day or two, so it was a better experience for the sender to have retries for days.

Nowadays it's assumed that the mail server is set up for high availability on good networks so it's a better experience for the sender to know right away that it wasn't deliverable so that they can use another communication channel for semi-urgent messages.

The ideal situation would be for the sender to get a retry warning while still retrying (which is they way it used to be done), but I can see why Yahoo might not want to tie up their resources that way.

inopinatus
Agreed. Fail fast is now the more desirable outcome, particularly for consumer messaging.

Moreover it is just inside the boundary of the RFC. The standard does not explicitly reject a zero number of retries, nor does it insist on the failure window; the language is noncommittal, and for an RFC that is significant and can only be interpreted as intentional.

beezle
I have to disagree here.

The standard in the past has been "we tried to deliver the message but were unable, we will continue trying until XXX" where XXX was measured in a few days.

Getting the above message is sufficient to allow the use of other messages for something that is time critical.

Instead, Yahoo is bailing out completely after just three hours.

jedberg
Not sure where the disagreement is I said exactly the same.
beezle
Sorry if I misunderstood, thought you were saying it was good that they were simply dropping all future attempts after three hours.
denton-scratch
Not much sympathy, really.

If you rely on a free mail server, you can expect to get everything you paid for. Pending outbound mail has to hang around in a queue, which costs money.

In the days when the original specs were written, mail was often multi-hop, and delivery was unreliable. Most email nowadays gets transferred in seconds, so queueing something for 4 days seems unnecessary.

But people do still run their own mailservers, on home equipment - which might fail to accept delivery, because it blew up, or suffered a power-cut, or needed to be rebooted, or whatever.

It's not reasonable to expect a freemail provider to maintain a sensible outbound queue. And it's not reasonable for a freemail provider to scrub the queue after 3 hours. Solution: get a real mail provider.

soraminazuki
The OP is the one self-hosting mail servers, not the one relying on a free mail server.
nisegami
The Yahoo user is the one GP is referring to. The "free mail server" in question here is Yahoo.
soraminazuki
Pretty sure that "you" in the second sentence refers to the OP, which is the most natural assumption considering that the first sentence started off with "not much sympathy, really" without specifying who.

Hence my reply.

saurik
I read that as a generic "you", similar to "one".
denton-scratch
To be clear:

I wasn't referring to anyone in particular by the word "you"; I could have said "one", but to me that sounds poncey. I was addressing people here (in a rather preachy way; sorry).

I was explicitly critical of Yahoo's limitations as a mailserver. When I ran mailservers, Yahoo was the biggest enemy - they rejected mail, and they sent huge amounts of spam.

[Edit] Wrong, Hotmail was much worse than Yahoo. Sorry, Yahoo; I guess you were one of the good guys.

beezle
I am both the hosting server and the one who used one of my yahoo accounts to send a test message to the domain I host while watching port 25.

I could of course have used a gmail or other account but the yahoo mail page was open at that time.

dangus
(Take this comment as as least rude and combative as possible, I'm not trying to attack you. Also, I'm no expert on email)

It sounds like your mail server's uptime is below modern industry standards.

A 24 hour outage is somewhat long in the email world, isn't it?

That's a genuine question, I'm not sure, but it sounds like a long time.

Google's SLA is three nines, which is just under 9 hours per year.

If I were to guess, Yahoo doesn't really expect any mail server to be 100% down for anything close to 4 hours at a time.

usr1106
Several big services have had outages longer than 4 hours in a row during the last year. A big hosting data center burned down completely.

Such outages just happen. Of course Yahoo offering their service for "free" (i.e. some moderate price paid in customer data and advertising annoyances) won't necessarily make any efforts to tolerate them.

dangus
Very interesting.

Perhaps then Yahoo is just like, welp, resend your email.

beezle
Thanks for the insight, but this is the first time a situation like this has happened, the server has been operational since 2014. It is a private, low volume e-mail server as well, not something with external public users.

The reason for the delay in recognizing an issue existed is that same domain email was working as were the other internet facing services. The firewall had choked on state tracking of port 25 but was otherwise functional.

ipaddr
At certain times a text can take longer than 3 hours.

I still expect 72 hours.

throwawayboise
> I am left to wonder how much mail was lost for good

You'll never know this, because SMTP does not have delivery guarantees. It's totally possible for any message to just disappear. Not likely these days, but possible.

throwaway47292
4-5 days in queue is infeasible with modern amount of spam, especially generative spam (e.g. someone gets verisign's .com zone and sends one email to every john@domain)

My gut feeling is that almost nobody waits more than few hours, probably even less.

But at least whoever tried to send you email got a 'well we tried to send but failed' so they know to reach you in some other way.

nickdothutton
Email delivery behaviour is governed by money these days, not the standards.
beezle
I suspect you have hit it on the head.

Perhaps my shock at the 3 hour drop window is because I have 20+ years of email experience (including bang addressed uucp mail) and the prior norm was an undeliverable but will retry until XXX message, not just a we give up message.

beezle
Thanks for all the replies. As a few have mentioned, directly or indirectly, this may well be an industry shift away from the long standing method of handling temporarily undeliverable email, at least for the largest providers.

This used to be the typical way of handling things:

This is the mail system at host abc.xyz #################################################################### # THIS IS A WARNING ONLY. YOU DO NOT NEED TO RESEND YOUR MESSAGE. # #################################################################### Your message could not be delivered for more than 4 hour(s). It will be retried until it is 5 day(s) old. For further assistance, please send mail to postmaster. If you do so, please include this problem report. You can delete your own text from the attached returned message.

Bottom line seems to be, if you host your own email expect inbound mail to begin permanently failing after three hours should your server (or routing to it) be down longer.

toast0
In theory, you might want to set up a secondary MX; although it might end up being too much work, as you need your secondary to do the same spam policies, and you want a copy of the valid destinations so you can refuse messages instead of bouncing them later.