I saw an email storm on a mailing list for lawyers. Due to a misconfiguration out of office notices weren't suppressed and forwarded to the mailing list. Soon several lawyers who were taking off were contributing to the storm. I immediately saw the self-reinforcing avalanche and told the operator to shut down the mailing list server.
Its actually fairly easy to setup a decent mailing list without having to have approved senders, but Exchange and its built in functionality is probably the root cause of a lot of these.
The ones I have setup are pretty easy to stop since we do some basic checking of the e-mails sent anyway (e.g. attachment too large).
O365 (commonly used at large enterprises) makes this dead simple too. You can even have it be self-service, where individuals can empower themselves to have access to a list-serv (its not just "CEOs").
Yep, its also a pain when you are waiting for a legitimate needed text and cannot leave the group. I guess you can mute a group but I think its named something funny like "Hide Alerts". It is a tad bit beyond the average user because of a poor UI.
A friend of mine was in a business meeting on the same day UND announced its new logo and his phone went crazy. The person he was meeting with asked if he should answer it and he said "no, its a lot of bitter, bitter people and UND has no taste.". He did get the contract though.
We had a few of these in college. My favorite was when someone replied "you know, you can just press M in gmail to mute the conversation." What followed were hundreds of messages saying just, "M". College kids really are the greatest trolls.
When I was with the government if you were on workgroups or had to attend meetings, you could count on 30-40 reply all emails when just trying to schedule meetings or coordinate calendars. For me I received easily 175-225 emails on any given week that should not have included me because of people replying to everyone rather than the one person who needed their info. It drove me crazy and was such a time waster.
Saw these storms couple times at Nokia years ago. By far the funniest replies were those that explained correctly what's going on and then concluded that their message should be the end of the thread.
What they didn't realize was that the email server was overloaded and that their message would be delivered several hours later together with dozen similar messages from the other people.
The next round of people then complained why these people are not stopping.
I met the developer who was responsible when I was employed by Amazon. He was trying to reply to an email, but I believe it was a "reply all" (if I recall the details of the story correctly) and the list was massive. Meanwhile, people on that same massive list were also doing a "reply all" to submit their feedback or reply that the email didn't apply to them. Massive email storm resulted. He didn't give all the details but he was ear-to-ear grinning as he told us the story.
"The resulting storm of 'unsubscribe', 'me-too' requests, sarcastic facepalm images and recipes for broccoli casserole resulted in (by the time the list was closed) over 4 million emails and generating over 375GB of network traffic."
Why couldn't the sysadmins just kill it with a Transport Rule (silently delete all messages containing the original subject) or temporarily reject messages to the DL? Takes a couple mins.
That's most likely what happens after the sysadmins get notified of the issue. Take the NHS incident as an example. Within 1h and 15min the generated emails were around 500 million. A one hour response time is still relatively quick.
The more important question is, why email systems don't have default rules to prevent such an email storm?
When emails go to tens of thousands of people, a system should probably only allow it with special credentials or at least ensure that it's a read-only mailing list.
Other measurements would be to detect growing email chains and block emails when the chain has reached some threshold.
True enough, but it is fairly easy to restrict use of distribution groups in Exchange, which I would bet most of these orgs use.
I'd argue 1hr response is pretty slow for the NHS, since they require 24/7 support availability of even their vendors. I imagine they responded pretty quickly but nobody had a plan in place for situations like this and a large public sector organisation isn't a place conducive to one person taking charge and fixing the problem immediately.
Wouldn't a simple mitigation be to limit the number of recipients allowed in the 'to' and 'cc' fields? 'Bcc' can be used for the mega mailouts. That way, a user hitting reply-all sends to a much more limited set of addresses.
26 comments
[ 5.1 ms ] story [ 66.2 ms ] threadThe ones I have setup are pretty easy to stop since we do some basic checking of the e-mails sent anyway (e.g. attachment too large).
[0] http://osxdaily.com/2014/09/23/leave-group-message-chat-ios/
A friend of mine was in a business meeting on the same day UND announced its new logo and his phone went crazy. The person he was meeting with asked if he should answer it and he said "no, its a lot of bitter, bitter people and UND has no taste.". He did get the contract though.
What they didn't realize was that the email server was overloaded and that their message would be delivered several hours later together with dozen similar messages from the other people.
The next round of people then complained why these people are not stopping.
The more important question is, why email systems don't have default rules to prevent such an email storm?
When emails go to tens of thousands of people, a system should probably only allow it with special credentials or at least ensure that it's a read-only mailing list.
Other measurements would be to detect growing email chains and block emails when the chain has reached some threshold.
I'd argue 1hr response is pretty slow for the NHS, since they require 24/7 support availability of even their vendors. I imagine they responded pretty quickly but nobody had a plan in place for situations like this and a large public sector organisation isn't a place conducive to one person taking charge and fixing the problem immediately.