It sure feels like there have been quite a few big outages this summer (Google in particular). I wonder if they are getting sloppy or this is just bad luck?
My gut reaction is that many companies make the majority of their income/deals during the fall/winter/spring (think back to school shopping, Christmas, re-signing contracts for the next year, tax season, etc.) Thus, many companies try to make major infrastructure changes during the summer, when there will be a slightly smaller negative impact on the business's bottom line in case something does go wrong.
Another way to think about that is that PMs spend three quarters of the year distracted by meetings with external clients, and only have summer to actually ride their teams on implementing the things they've negotiated.
My take is that many people are out on vacation during the summer months and sometimes things break (or break harder than usual) when certain knowledgeable people aren't available.
I have a good handle on how the customer base for cloud computing services perceives cloud vendors and aws is way ahead of everyone else by a mile.
This is because they have a far better and complete platform, and it appears to be better engineered too with fewer of the downtime incidents we’ve been seeing at the other vendors.
Why do you suspect this is? Let me tell you. Amazon had better people. Full stop.
Login from incognito works for me so I assumed one of my extensions was causing an issue somehow and started disabling them. I'm sorry I doubted you, extensions.
Doubtful. I put that in since absolute statements are always dangerous. I'd say at any given time, there's an excess of 20-50%. There's virtually no chance they were lacking capacity.
Within Google, there is plenty of wasted capacity, but getting capacity for a service to scale up is a fiendishly difficult task. They even employ hundreds of people whose main role is to try to allocate all the various types of resources to teams. "Oh - you want 1700 GB of bigtable in Reliability zone 3 in Atlanta? I'm afraid your department doesn't own any there - you can trade with the ads department who aren't using theirs, if you give them 2Tbits of network bandwidth between Peru and Brazil? That trade will only be till the end of the quarter though, because then they need it back."
We run a app making millions of API calls to Google every day, so they show up in our monitoring, even if they don't make the status page. It's a pattern I've noticed from getting paged at 3am (NZ time) this time of year going back the last 4 years, I don't have hard data at hand for it though. It's not the same thing every year either - this time OAuth got hit hard, previously we've mostly seen slowdowns and higher error rates on the Drive APIs.
I like the majority of GCP products I work with, but judging by the amount of issues in the past year, GCP feels amateurish compared to AWS, which we continue to user in order to host most mission critical operations.
I didn't think I would laugh that hard coming in here and reading these comments, but here we are. I can virtually guarantee that the following things are not the root of the problem:
* Lack of compute/networking/storage
* Incompetence of employees
* Back to school traffic spikes
* Just about anything else here
My $0.02. these are almost always due to bad roll outs, usually configuration changes. But I've been wrong countless times before!
Given that "87.623% of all outages are caused by changes" is generally accepted wisdom of running large scale services, I would tend to disagree with those people. Incompetence: no. Opportunity for improvement: yes.
Googler not on Cloud team, but using Google Cloud Platform for an internal project. I've encountered my share of bugs and other flaws while using this platform. I think all these platforms are just too damn complex and brittle. It's easy for even a smart SWE or SRE to overlook one little thing that will bring down a bigger part of system.
56 comments
[ 2.4 ms ] story [ 108 ms ] threadhttps://status.cloud.google.com/incident/developers-console/...
hey everybody, I spotted the problem! I'll keep an eye on my mailbox for the giant consulting fee that I assume is enroute.
Can you share the data?
This is because they have a far better and complete platform, and it appears to be better engineered too with fewer of the downtime incidents we’ve been seeing at the other vendors.
Why do you suspect this is? Let me tell you. Amazon had better people. Full stop.
btw never saw 500 from G before, usually just timeouts during outages
It's a temporary spike of traffic far outside of average use.
In order to handle it, they'd need a lot of resources that just aren't justifiable for the rest of the year.
You seem similar things when places sell highly desired items in online stores.
Temporarily shut down other people's services to favor their own?
I cant access any of my Airflow clusters atm. :/
I didn't think I would laugh that hard coming in here and reading these comments, but here we are. I can virtually guarantee that the following things are not the root of the problem:
* Lack of compute/networking/storage
* Incompetence of employees
* Back to school traffic spikes
* Just about anything else here
My $0.02. these are almost always due to bad roll outs, usually configuration changes. But I've been wrong countless times before!
Edit: made-up number has large margin of error