There was a thundering herd e-mail at Amazon about 10 years ago that I’ve heard stories about. It went on for days. There’s a funny internal talk with lots of data about it, maybe it’s on YouTube by now…
Something similar happened at Cisco something like maybe 5 years ago? Someone sent the Bay Area employees or whatever list (probably at least some tens of thousands of employees) an ask for a cook/chef they could hire to make meals for their family, IIRC. I think the reply-all's happened all week.
And Apple in 1991 [1]. It's a very fun story actually, good read if you're in the business of writing "reliable" or "recoverable" software (aka pretty much everyone). A bad design choice in sendmail caused a cascading explosion of emails. I also highly recommend reading the rest of the book (?), the Unix Hater's Handbook is a wonderful bit of history and discussion of the design issues of the NIXs we all know and love/hate today.
Compaq in the mid 90s on banyan vines a friend leaving the company sent a company wide email listing his house. It wasn't supposed to be possible to email *
When i worked at the NIH, a couple times an entire institute was accidentally cc'd instead of bcc'd on an email, and for weeks after the chain would continue, consisting only of people writing "please stop responding to this chain". I don't know what you do for that...
Would that happen to be the "wallet" incident? It was slightly before my time, but I also heard legends of it. During my time there, any email thread that looked headed for another reply-all storm had people replying-all to it with simply the word "wallet", apparently in an attempt to deliberately cause chaos.
Looking around, everyone has a story like this, so I don't want to just pile on with my similar experience at Verizon. But what stood out at me there was the low quality responses from low-level managers in far parts of the world, demanding, by the power vested in them, to be taken off the email chain... you know... IMMEDIATELY!
Yeah that is one thing that I remember from one or two that I was copied into in roughly 2000s with more than 200,000 CCed. Mini-kings and also people with enough technical skills to know better keeping the threads alive and flooding the Exchange servers woldwide for days.
This happened at all my previous employers at one point or another. Most famously a thread about unsubscribing from a mailing list that nobody really knew about but had everyone in the company on it. For weeks there'd be some random field sales guy or a marketing person in random parts of the world replying all.
friendly reminder: 400k is an apertif for a competently configured production postfix server. its about 14 seconds of mail, and about 8 seconds optimized at hw and filesystem level.
the real issue here is shitty projects from shitty companies.
Yes but that’s on top of what GitHub is already sending, plus it must be multiplied by the number of comments left on top of that. It feels like 35 straight minutes of 100% usage isn’t great on any system. It presumably sent 61 million emails.
> Yes but that’s on top of what GitHub is already sending
When you send at the volume someone like GitHub sends, you will always see peaks and valleys in your sending patterns that are much larger than 400k. It might cause an issue if they were already under peak load, but even then it would just take a bit longer.
Yeah I’m sure they’ll institute some kind of control to make this sort of thing harder, but honestly I’ve never understood why people get so worked up about this sort of thing. It makes me chuckle. The person looks like a dummy or a shit-stirrer and a lot of people have to delete HUNDREDS (oh my) of messages and have LITERAL MINUTES of their time wasted. The megacorp I work for wastes more of my time with silly self-congratulatory org-wide emails about business deals and fake benefits like seminars on retirement planning for dummies.
I love me a good bedlam drama. One of the commenters on the PR had the best take: “I just wasted 2 minutes of my life I'll never get back.” The ones with the scorched earth PUNISH HIM attitude need to chill the hell out.
Yeah. Also their real names and them being immature and getting angry also went out to 400k people and is now effectively part of history. That's probably even worse than being the kid who did this, what if a future employer comes across this?
> notifications are going to be scrutinized more in the future, from this point on.
This already happened with github & epic & unreal when it first did this organisation setup.
So, given no solution appeared after exact same incident, I wouldn't hold my breath
Ah yes, the non working unsubscribes of SaaS and social media companies. I prefer the straight up mailchimp type marketing campaigns with one click to unsubscribe
It does but each comment adds 400k emails to the queue before you, so all of the comments that happened before you unsubscribed are already queued. I unsubscribed and it did eventually stop.
I was sleeping and the constant new mails notification woke me. By then it was too late to unsubscribe as the topics was already locked, but I was only on like 60-70th mails.
It doesn’t really fix the “vulnerability” though? Some other jerk can open a new PR and tag everyone again. Or just tag @EpicGames/developers elsewhere, no PR needed.
The thing that gets me, as discussed in this thread, is that the solution is often just super easy. Literally just pop up a modal when the recipient list is above 1,000 users that says “You’re about to send this to X,000 users, Are You Sure?”
Adding that check to every incoming comment in a large scale system and adding the previously nonexistent interactive UI for it is far from “super easy”. This is not a Todo MVC. There must be a thousand higher impact stories that are better targets of engineering time (unless this form of trolling become a new pastime).
In addition, there are lots of actually malicious skids you can’t stop by asking nicely.
Teams can be secret. This might be a secret team to begin with? Anyway, I highly doubt they deleted the team since it’s used as an authorization mechanism.
Anyone could have done this by mistake, all it'd take is being a bit tired and @ing the wrong group.
The question is - why doesn't the platform warn you that you're going to send notifications to a large number of people in the same way that many email clients do?
Or in the way that `rn` used to do when posting to Usenet:
> This program posts news to thousands of machines throughout the entire civilized world. Your message will cost the net hundreds if not thousands of dollars to send everywhere. Please be sure you know what you are doing.
Regardless of the mass notification or a bad quality PR - you don't just remove someone from a major internet platform like this, it's an inhumane response to someone making a mistake.
Not to mention it's just a notification, who really cares unless it happens all the time which is just more of an argument to fix the platform behaviour. Some people are so high and mighty.
If it's possible for somebody to unintentionally piss off hundreds of thousands of people, that's not the person's fault. It's the system's. The internet allows proliferation of information at scale and speeds that can be disasterous if left unchecked
I was 'removed from an internet platform' when I was a kid inadvertently breaking the rules by posting off-topic threads in the wrong Sci-Fi community subforum. So apparently we do just do it.
Some people just want to be outraged for the sake of being outraged. Looking at the persons who wants the kid to be banned GH profile - I would not expect any other reaction.
The people who call for his ban in that thread should at least be consistent and call for banning themselves as well, because with their reply they just did the same thing.
Totally excessive. In my opinion, people should treat this incident with the "blameless post mortem" mentality in mind.
Don't blame the individual for an innocent mistake (we don't know if he knew that it would trigger 400k notifications). He is young and might be inexperienced, so we should be forgiving.
Think about why the system is set up in a way that an untrusted contributor can trigger so many notifications with a PR that is of little value.
That is a much harder problem to solve, so that is why some people go to the easy solution ("ban him, he is an evil bad person, gross social misconduct").
> The question is - why doesn't the platform warn you that you're going to send notifications to a large number of people in the same way that many email clients do?
The real question is why is it allowed to send at all? Depending on human judgment to stop spamming is a poor decision because bad actors don't care. I discovered this and GitHub s hilariously terrible setup a week ago when another large repository became a spam source and GitHub offered no easy way to unsubscribe.
My freshman class (~4000) at university was sent an email by administrators via the freshman class mailing list. Someone replied, which lead to their email being sent to the entire mailing list... another email asking to be removed from the mailing list, etc, suddenly everyone was hit by a deluge of hundreds of emails asking to be removed.
Getting real flashbacks to that from the snarky / annoyed comments on this :)
I had this happen at my job maybe six months ago? It was surreal because I'm young enough that I knew the trope from TV shows and stuff from the late 90s/early 2000s of people not knowing when to use "reply" instead of "reply all" but never actually had to deal with it by the time I grew up and starting working.
The only thing more amusing than all of the people replying asking to be taken off the list is all the people who reply to tell everyone else to stop replying, as if that would solve the problem instead of making it worse.
> The only thing more amusing than all of the people replying asking to be taken off the list is all the people who reply to tell everyone else to stop replying, as if that would solve the problem instead of making it worse.
There's a very good chance that those replies are reducing the flood, assuming they're nowhere near a majority.
When we migrated from on-premises Microsoft Exchange to Office 365, the default "reply" button in OWA was switched from reply-to-one to "reply all".
After the CEO sent the first all-staff email on the new platform, we had at least three highly
regrettable replies copied to the whole organisation, including one disclosing very confidential information intended only for the CEO.
Hehe when I was an intern working on a big chemical industry site someone also accidentally reply-alled a plant-wide safety mailing list.
This quickly started a similar deluge of mails until someone high up the chain of command all-caps yelled "NO MORE REPLYING ON THIS MAIL, OR IT WILL BE AN HR VISIT FOR YOU"
> Lock it. Lock it now. This is the friendliest message I'm going to send, while I look for ways to get OP banned from Github for gross social misconduct. I imagine that "owner of bots universe" might be enough to get that account tagged as a bot, who knows how many communities across however many repositories that person just bothered across all of Github.
Wasn't the message send as in innocent mistake? Why is this man adding fuel to the fire (sharpening his pitchfork?). Did I miss something about this?
Every large org I've ever been at has had someone reply-all to a company-wide mailing list and then promptly spawn a flood of more company-wide reply-alls. Slack's @channel/@here has made this even worse. On a good day, people have a laugh, tell them not to do it again, and we all move on. The only difference here is that it's in public on github.
I suspect there's some psychological phenomenon that convinces people that norms expected of them have somehow been broken because of the chaos.
> Perfect for gorgeous looks, can push asap @EpicGames/artv2-admin @EpicGames/developers @EpicTeamAdmin
The entire diff of the PR is:
diff --git a/README.md b/README.md
index b0b4f5d..61b95bb 100644
--- a/README.md
+++ b/README.md
@@ -1,12 +1,18 @@
# Epic Games
+<div align="center">
+ <img src="https://cdn2.unrealengine.com/Epic+Games+Node%2Fxlarge_whitetext_blackback_epiclogo_504x512_1529964470588-503x512-ac795e81c54b27aaa2e196456dd307bfe4ca3ca4.jpg" width="20%" min-width="100px" />
+</div>
+
Unreal Engine is now [free](https://www.unrealengine.com/blog/ue4-is-free)!
-To access our repositories, sign up for a free account at [UnrealEngine.com](https://www.unrealengine.com) and register your GitHub ID using [these instructions](https://www.unrealengine.com/ue4-on-github).
+To gain access for our repositories, sign up a free account on [UnrealEngine.com](https://www.unrealengine.com) and register your GitHub ID using [these instructions](https://www.unrealengine.com/ue4-on-github).
-After that, you can find our repositories here:
+After that, you may able to find our repositories here:
* [Unreal Engine](https://github.com/EpicGames/UnrealEngine)
* [Unreal Tournament](https://github.com/EpicGames/UnrealTournament)
(Note that you must be signed into GitHub for these links to work.)
+
+Have Fun !!
The original PR did nothing but add grammar errors to the README, with a mismatching PR description and title, then demanded that it actually be merged. I read this as a way to waste people's time, from there it's just a question of how many people. This user's bad intentions colour their entire interaction, both the deliberate and accidental parts, because this accident would not have happened if they weren't screwing around in the first place. If the PR were real, it should be treated as a teaching moment instead. If it were a PR made for learning purposes that was accidentally posted publicly, they wouldn't have demanded a merge.
Ok! I'm sorry that I didn't look at the PR itself. I get a lot of these on my projects, mostly from kids/students who I think just want to look like they want to build "reputation" and say they've contributed to n open source projects.
However I still suspect the notification was unintended.
The PR is still from a kid, who is likely figuring things out and looking at his commit history, I don't see any similar commits in other high profile projects. I agree it is a useless PR but the best thing here is to tell him not to do it and move on. Calling for a ban is excessive.
Is, according to their github page, a nary 18 years old.
One cannot say in good faith that they were a beacon of perfect judgement at 18. Anyone who says so is either looking to buy or sell a bridge in New York.
The delta between good and bad faith/intentions on the internet may be much larger than the delta between me at 18 and me at 44. There's also very bright 18 year olds contributing usefully to GitHub repositories. So, for me personally, the recurring assertion that they are "only" 18yo doesn't do much for me.
Being bright doesn't make you infallible. I was contributing to lots of things at 18, including digging into kernel debugging in ubuntu. That did not stop me from making occasionally poor choices with consequences much larger than I realized.
Being 18 means being given the benefit of the doubt that your actions are at least an attempt at good faith.
> Is, according to their github page, a nary 18 years old.
Stop posting this like an average 18 year old is supposed to be an idiot like OP. This is the age people get drafted, can start drinking alcohol, marry and start being adults.
Nit: in the US you can't drink alcohol at 18. I know in other countries that's not the case. Also another nit: the user's birth date does not check out to be 18, so his age is not entirely clear.
> Is, according to their github page, a nary 18 years old.
a bit further down he actually proclaims being born 11/11/2004, making him just 17. No idea why he has 18 y/o in his user description, not that it'd make a difference tho.
I agree. Even if it's possible there was mischievous intent here I feel like the configuration issue or something being used in a way it shouldn't be is the real story here.
Fun fact: at some point I wrote @param in a PR comment, talking about a parameter's doc. I only realized after posting that there was an actual user named @param.
I am @3x and I get ~10 notifications a day of people mentioning me in PRs and issues. Whenever anyone writes the name of a file containing `@3x`, which is commonly used for hi-res assets, I get a notification. I quickly learned to turn off email notifications!
I had to turn off my paypal.me because I naively picked a number for a username and people kept putting that number in the send-to-username field and sent me free money, which they always tried to refund later, getting me stuck dealing with PayPal's customer service through no fault of mine. I would explain to them that I didn't ask for the money, I do not want to keep the money, and I am not accusing the sender of scamming. Sometimes I was able to refund instantly but depending on the sender and payment method, PayPal had to get involved for weeks of back and forth.
Looking at the content of the OP and the PRs you linked, my first thought is "Is it Hacktoberfest[1] already?". I mean, seriously, what's up with the such low quality PRs? Is it common to have people spamming repos with trivial changes absent some sort of incentive?
The problem here is that they’re not even trying. How can they not know that such PR would never be merged? For this reason I don’t think public contributions are why they send the PRs
Sometimes fixing a typo would be a new coder’s first open source commit. It’s pretty stupid to assume every new PR would be a major new feature or bug fix. It makes OSS unnecessarily hostile to get into.
Although this commit in question isn’t even that sadly.
The main repo I work on for my job is open-source, and yes, it happens fairly regularly that someone opens a nonsensical PR that might randomly perturb some Markdown or YAML file, or attempt to merge commits from some other developer's in-progress branch.
I'd say we only get them about once every week or so, but then the repo is not anywhere near as high-profile as the Unreal Engine, nor as likely to be on the radar of children.
It's just very common GitHub spam/abuse. Suspect these are automated accounts trying to look legitimate by doing legitimate-ish things (e.g. cryptocurrency mining via PRs that require one-time approval).
I would suppose some video aimed at schoolchildren teaching them how to use GitHub has gone viral and they’re taking baby steps. At least that what I think is most likely.
It may be intended behavior, but for a trivial change like this to easily, without warning, mass ping ~400k users that is not desirable behavior. The submitter thought he was pinging developers of the project, but instead pinged mostly people who do not handle accepting upstream changes. Discord for examples handles mass pings by showing you a warning dialog saying you are about to ping X many people and if you are sure you want to do that.
Sending email notifications to 400,000 users for unimportant things is antisocial. (If 0.1% of those users then reply on the thread, it'd be another 400 comments sent to all 400,000 recipients).
I don't think anyone sits down and thinks the intentional behaviour should be "we should make it easy for someone to send a notification email to 400,000 people". That would clearly be annoying.
I think the behaviour is accidental, arising from "notifying all people in a team is useful".
> Sending email notifications to 400,000 users for unimportant things is antisocial.
What about requiring people to join a specific team to access the source for no apparent reason? In other words, how are the conditions which led to the creation of this inflated team not antisocial as well?
Creating a team with 400,000 users is antisocial. GitHub teams, and all their inter-related features, are built around a small group of people collaborating together on code. GitHub teams are not intended solely to grant access to source code after signing a EULA. Making a team with 400K users is just a bad idea and this problem was waiting to happen. Unreal needs to find a different solution for what they want.
What other solution? What tools does Github provide to give access to a private repo without creating a namespace which can be pinged? It seems like exactly the use teams were created for just on a scale not seen before. The answer would be Github adding a feature to mark a team as semi public and remove the ability to tag it.
They will probably get a special treatment from Github and Github's code will get an unnecessary `if` clause for the special cases (right now probably only the epic one).
467 comments
[ 1.5 ms ] story [ 290 ms ] threadI got the email. And about 20+ responses to that email.
Every person who replies to that issue triggers another 400k emails. Personally, my email client is crashing.
EpicGames, as a GitHub org is an outlier, it's basically an SSO for Unreal Engine. I forgot I was even a member of it.
[1] https://web.mit.edu/~simsong/www/ugh.pdf page 85
To: soandso@domain.tld Subject: blah Action: block
the real issue here is shitty projects from shitty companies.
When you send at the volume someone like GitHub sends, you will always see peaks and valleys in your sending patterns that are much larger than 400k. It might cause an issue if they were already under peak load, but even then it would just take a bit longer.
I love me a good bedlam drama. One of the commenters on the PR had the best take: “I just wasted 2 minutes of my life I'll never get back.” The ones with the scorched earth PUNISH HIM attitude need to chill the hell out.
Doesn't that just add to the comedy?
Yeah, being drama queens.
> Every person who replies to that issue triggers another 400k emails. Personally, my email client is crashing.
Your email client only received 20 messages; why is it crashing? The very long To: header?
That would be a massive privacy breach, and people here would be making a lot more noise about such a large email database being leaked.
This already happened with github & epic & unreal when it first did this organisation setup. So, given no solution appeared after exact same incident, I wouldn't hold my breath
Throw that thing in the bin. A human printing emails and placing them on your desk could handle that workload.
Best thing I've read today only after the email story.
[1] https://files.littlebird.com.au/Shared-Image-2022-06-05-12-1...
Would have totally avoided this situation.
In addition, there are lots of actually malicious skids you can’t stop by asking nicely.
The question is - why doesn't the platform warn you that you're going to send notifications to a large number of people in the same way that many email clients do?
> This program posts news to thousands of machines throughout the entire civilized world. Your message will cost the net hundreds if not thousands of dollars to send everywhere. Please be sure you know what you are doing.
Also the poster is essentially a kid. I would hold my judgement before flaming him fwiw.
Not to mention it's just a notification, who really cares unless it happens all the time which is just more of an argument to fix the platform behaviour. Some people are so high and mighty.
Don't blame the individual for an innocent mistake (we don't know if he knew that it would trigger 400k notifications). He is young and might be inexperienced, so we should be forgiving.
Think about why the system is set up in a way that an untrusted contributor can trigger so many notifications with a PR that is of little value.
That is a much harder problem to solve, so that is why some people go to the easy solution ("ban him, he is an evil bad person, gross social misconduct").
The real question is why is it allowed to send at all? Depending on human judgment to stop spamming is a poor decision because bad actors don't care. I discovered this and GitHub s hilariously terrible setup a week ago when another large repository became a spam source and GitHub offered no easy way to unsubscribe.
Getting real flashbacks to that from the snarky / annoyed comments on this :)
The only thing more amusing than all of the people replying asking to be taken off the list is all the people who reply to tell everyone else to stop replying, as if that would solve the problem instead of making it worse.
There's a very good chance that those replies are reducing the flood, assuming they're nowhere near a majority.
After the CEO sent the first all-staff email on the new platform, we had at least three highly regrettable replies copied to the whole organisation, including one disclosing very confidential information intended only for the CEO.
This quickly started a similar deluge of mails until someone high up the chain of command all-caps yelled "NO MORE REPLYING ON THIS MAIL, OR IT WILL BE AN HR VISIT FOR YOU"
Good times
https://www.bbc.co.uk/news/technology-37979456
- GitHub user sends notification to 400000 users
- How to stop USPS junk mail
> Lock it. Lock it now. This is the friendliest message I'm going to send, while I look for ways to get OP banned from Github for gross social misconduct. I imagine that "owner of bots universe" might be enough to get that account tagged as a bot, who knows how many communities across however many repositories that person just bothered across all of Github.
Wasn't the message send as in innocent mistake? Why is this man adding fuel to the fire (sharpening his pitchfork?). Did I miss something about this?
I suspect there's some psychological phenomenon that convinces people that norms expected of them have somehow been broken because of the chaos.
> Perfect for gorgeous looks, can push asap @EpicGames/artv2-admin @EpicGames/developers @EpicTeamAdmin
The entire diff of the PR is:
However I still suspect the notification was unintended.
The situation doesn't need any more escalation beyond fixing the underlying vector for spam.
Is, according to their github page, a nary 18 years old.
One cannot say in good faith that they were a beacon of perfect judgement at 18. Anyone who says so is either looking to buy or sell a bridge in New York.
Being 18 means being given the benefit of the doubt that your actions are at least an attempt at good faith.
Stop posting this like an average 18 year old is supposed to be an idiot like OP. This is the age people get drafted, can start drinking alcohol, marry and start being adults.
To this day, people @everyone in 10,000 person discord servers because they forget the scale of what that means.
a bit further down he actually proclaims being born 11/11/2004, making him just 17. No idea why he has 18 y/o in his user description, not that it'd make a difference tho.
https://github.com/EpicGames/Signup/pulls
https://github.com/EpicGames/Signup/pull/10/files
https://github.com/EpicGames/Signup/pull/18/files
What is going on?
[1] https://www.theregister.com/2020/10/01/digitalocean_hacktobe...
“Contributed quality improvements to Epic Games Unreal Engine”
luckily, GitHub makes it terribly easy to verify how much and what you actually contributed
Although this commit in question isn’t even that sadly.
I'd say we only get them about once every week or so, but then the repo is not anywhere near as high-profile as the Unreal Engine, nor as likely to be on the radar of children.
https://blog.domenic.me/hacktoberfest/
> The maximum value of long is 9,223,372,036,854,775,807
I don't think anyone sits down and thinks the intentional behaviour should be "we should make it easy for someone to send a notification email to 400,000 people". That would clearly be annoying.
I think the behaviour is accidental, arising from "notifying all people in a team is useful".
What about requiring people to join a specific team to access the source for no apparent reason? In other words, how are the conditions which led to the creation of this inflated team not antisocial as well?
it isn't for no apparent reason - it's to accept the terms of the agreement for access to the unreal source code.
Alright. So if Epic Games are abusing Github Teams, why should the poster not be able to also?