543 comments

[ 3.7 ms ] story [ 106 ms ] thread
the internet is just falling apart these days
The internet is held together with pixie dust and prayers (sometimes duct tape and occasionally spit and rust).
I work for a small shop so we have to use a lot of bubble gum too!
nah, the internet is fine.

Only the self-entitled "gatekeepers" are failing, and I'm glad they do.

Quite honestly, I wonder how much of this can be traced back to the fact that there are way too many layers of abstractions between the browser requests coming in and a cpu actually executing something to serve the request.
It's weird seeing the go-to downtime tracker go down. I'm so wired to check Twitter that I kept refreshing for a good 10 seconds.
Time to make HN the go-to downtime tracker. Did anyone measure HN's uptime over the past years?
I don’t know about empirical data, but HN occasionally goes into a mode where page loads that don’t hit the cache (logged in users) take 10+ seconds. I haven’t been on when it’s gone down completely since i signed up (not too long ago).
1. Wow, Twitter is down.

2. I bet it’s trending. Let me check.

3. …

4. Oh.

google, apple, microsoft, facebook.. and now twitter? I keep asking the same, when is amazon's outage day?
July 15 and 16, coming up!
Free shipping on outages for all prime members!
You missed cloudflare, stripe, slack
oh, right! the slack one sucked
The Google, Cloudflare, and Stripe outages all affected Shopify. So Shopify has had several meaningful outages over the past few weeks.
Isn‘t Shopify a Fastly customer? Do they still use Cloudflare?
I was also under that impression until the Cloudflare event happened. I do not actually know what their dependency was, but all shops were taken offline.
Reddit too :(
Reddit goes down constantly though, not a great benchmark.
(comment deleted)
Reddit goes down almost every day. I see "Ow!" thing a few times a week. Full confession: I'm constantly on reddit. :(
Hm. If you were going to plan a worldwide internet outage (or the appearance of one) you could test your ability to take down individual services first and then take them all down at once.
I wonder how many conspiracies this single outage will trigger.
Theory: The internet is really just a series of green tubes underground and the last earthquake knocked out the primary plumber named Mario.
My theory: The internet in fact IS a big truck, and it got a flat tire.
Given that today was the White House's "Social Media Summit", no doubt there will be a few conspiracies floated. I'm betting "Twitter wanted to block out all the criticism coming form the summit!" will be a popular one.
I've got a couple conspiracy theories at the moment:

1. It's a deployment of some infrastructure change the government got the big tech companies to sign onto.

2. It's a "shot across the bow" from some external party to demonstrate their control over major infrastructure.

2.a. Also could have been a mix of 1 and 2. The government orchestrated the outages in order to add fuel to the hysteria over Chinese "spy chips". However, given the story every time seems to be "Someone goofed a configuration", this theory doesn't seem to have much life left in it.

(comment deleted)
My conspiracy theory: The dehumanization regex was found to match too many non-conservatives, prompting a rollback that failed.
Nah that's low level conspiracies. The new tier are in the vein "Twitter/Facebook/Google is currently being raided and forced to copy over DBs for an ongoing investigation into surveillance/censorship/pizza/Molochworship".
The Tron man's biker shorts ripped right in the crotch, and they're rushing to repair the split in his pants.
Yeah, I'm sure it's all a big coincidence. If someone was targeting them all I'm sure the effects would be entirely different.
I'm counting three in these comments so far :)
Stay down. The world will be a better place.
I wish they would bring back the fail whale!
Agreed! Their little amputated robot looks too much like the reddit Snoo.
I'm sure it's purely coincidence, but interestingly Trump's also holding his Social Media Summit right now.
I had the exact same thought!
I'm wondering if this has anything to do with the announced new "Look and Feel" for Twitter. I got a banner yesterday talking about it and now Twitter is down. Maybe they messed up something in preparation for the rollout?
Really curious which part of their infrastructure was the root cause.
eh, it's a boring story

source: work there

It's always configuration change during deployment. Bad configuration, someone messed up a yaml or json config file.
Wow I hope the stock markets won't crash with Trump having no ability to talk it up constantly.
What’s the fallback for twitter if twitter is down, or worse: stops
Just threatening to rape random women in the street from the safety of a moving car?
part of me wishes it was the fediverse

part of me hates the idea

We'll have to do our own jokes.
Literally 0 negative consequence.
Looks at folder marked "projects" in dread
Any decentralised social media alternative that is not Twitter, like Mastodon.

Anything else is centralised and inherently evil. Even the orange website.

I first noticed the outage about 10 mins ago in Safari (Mac). It repeatedly gave me errors, even though Brave (Mac) was working fine. My iPhone app also worked fine (and appears to still be working).

Why would one browser work but another not work, on the same computer at the same IP? The only difference is the account I'm logged in through (personal/work).

You probably just hit different load balancers with your different browser sessions.
It has been a long time since I've seen the equivalent of the fail whale on Twitter. It was a weekly occurrence back in the days.
On the status posts in particular I really miss the ability to sort comments by new on this site.
Just last week I found a setting deep in my profile config that let me disable ‘recommended tweets first’ or similar. When it’s back up I can check the exact setting
I mean on Hacker News, not Twitter.
A lot of high-profile outages recently. Can't actually remember the last time Twitter went fully down. Have to confess I immediately assumed an issue with my own connection, even though every other site is working.

Unrelated, but for some reason the phrase "I have no mouth and I must scream" just popped into my head

Move fast and break things
I think they are past that stage! If a 28 billion company gets to say that I am not going to feel bad about my site going down.
Mercury is in Retrograde through July 31.
I never notice problems when I don't know about that, so I wish I hadn't read that. :)
> Unrelated, but for some reason the phrase "I have no mouth and I must scream" just popped into my head

That phrase was coined by Harlan Ellison in his classic scifi short story to represent a situation of complete despair and powerlessness.

I don't think a lack of Twitter, Whatsapp, Instagram, Facebook or Gmail -- however inconvenient -- would fill me with that kind of existential dread :)

honestly i feel unmoored without tweetdeck on my second screen ticking by. i realise this is probably a bad thing, but getting my daily news and info, i'd say 90% comes from twitter.
I got rid of all of those except Gmail (need it for work, school, etc) and WhatsApp (only use for one group chat) and it is very freeing. I realized social media (Instagram especially) breeds unhappiness with your life, while all of the models and "influencers" on it don't live realistic lives. In fact I've experienced first had that their posts are often doctored or don't acctually portray the situation accurately.
Not those things specifically, but take a step back and look at the Rube Goldberg inspired complexity of modern technology and despair.
Yesterday, Stripe went down for a half hour, and the later yesterday, Google's Android Payment Validation API went down for more than 2 hours.

Stripe at least acknowledged their downtime. Google was oblivious, made no update to any of their status pages. Really horrendous awareness and support from Google per usual.

Giving Amazon a run for their money I see.
(comment deleted)
Odd coincidence, seems like it might have been some upstream banks having a bad SCA rollout
Twitter is especially weird for this since it's often a platform where people talk about downtimes. I don't see this downtime mentioned on Reddit and I don't know of other sites where it might be discussed, so if Hacker News happened to go down at the same time, where would I go to talk about it with online strangers and find out if it's just me? Nowhere, I guess, I'd just wait it out with no extra insights on what's going on. A small reminder of what the world used to be like haha.
reddit feels like it has about a 4 hour lag time on most "breaking" events.
It's on purpose ever since the revolt against Ellen Pao. Was it sparked by firing Valerie, the AMA admin?
This is the time it takes for news in major subreddits to gather enough votes on the "new" tab to make it to the main subreddit front page, then the actual front page.

Smaller subreddits seem to be less affected by this, which is why /r/toosoon (a subreddit dedicated to dark humor related to current events) is often surfaces news hours before other subs for people who have it in their subreddit list.

it got brought up as quick as it did here

at least on r/twitter

It makes me wonder if a powerful malicious agent can devise a complex operation (planting people in several key places) and wipe out most of the databases of one these places. It would be interesting to see what would it happen if suddenly all of twitter of facebook is deleted , puff!, gone in a second.
Actually deleting stuff at these scales takes many, many days if you don't want it to be trivially recoverable.
You can't delete that much data in a matter of seconds digitally. You need to physically harm the hardware.
Wipe the encryption key of your SSD, small amount of data to wipe, and the whole SSD is unrecoverable. FWIW at least OCZ SSDs have an encryption key flashed into it even if you didn't turn encryption on. Putting a new firmware on it can wipe this key and make your old data inaccessible. Source: got a buggy OCZ firmware that failed to make the device appear on the SATA bus, only way OCZ could fix it was to install new firmware which wiped the key and hence my data was unrecoverable.
Interesting point. I stand corrected. Not sure why I didn't think that way.
I recommend you watching Mr.Robot :)
I watched the first season but got tired of the constant morose attitude of the main character. Does it get better?
the mood of the character doesn't change. The plot is quite good though.
(comment deleted)
yup, What's worse than releasing a product on twitter and found it down a few minutes later...
Quick un-release it and rerelease after it's up!
(comment deleted)
Call me a conspiracy theorist but seems odd the timing of this at the same time Trump is having his "Twitter Summit".

Then again things have been going down over the past two weeks so it's probably just coincidence.

Stuff like this makes me wonder if the Internet really is super vulnerable, and the only reason there isn't a mass disruption of communication all the time is because some script kiddies's Pizza Rolls were perfect today so he held off on attacking a backbone.
This is actually true of most of the modern world.

It is mostly still together because the venn diagram of those who want to see the world burn, and those who are clever enough to make it so has a very very small intersection, since the latter group is quite invested in the world not being on fire.

Yes, I think people tend to underestimate the chaos some malicious actors could do by even "just" coordinated litteral burning of stuff. Think a dozen people with cans of gasoline and matches spread over three different suburbias in a city, wooden churches or other wooden buildings of interest. Or simple firebombs on underground subway plattforms. Would probably not be that lethal but I guess very frighting.

(Not even mentioning explosives etc, but this could probably be prepared in like an hour by just purchasing supplies at local gas stations in any country)

as someone else mentioned above this is a website/service that's down, not the internet. When you can't open a socket to a server in another network then the Internet is down.
Read up on BGP hijacking. The internet really is super vulnerable.
And here I am thinking I got suspended again because right-wingers report every comment they don't like.
And here I am thinking I got suspended again because ~right-wingers~ lame people report every comment they don't like.

FTFY. In my experience both sides of politics have weak minded people that can't experience critical thinking.

ETA: Awesome. The downvotes and "flagged" status are great to support my statement. Thanks for the fish!

Pray for the team that has to handle this ticket.
That’ll be fine, a post mortem will show that ops weren’t the cause and their comp package will help them get over this little package of stress
Guess for some media outlets - it's going to be a slow news day.
Where is twitter when I really need it? :D
Down on a plain Thursday with no major news or anything , so definitely doesn't look like a infrastructure spike or anything.

Could be bad deployment or someone decided to pull few plug(s)

plug pulled by "youknowwho"'s team