I would argue that two direct Google Cloud outages within a month is pretty concerning for GCP customers, and that it's possible that the calendar outage could also be related in someway since it is likely hosted on GCP, although that is speculation
AFAIK Amazon are running a lot of actual production loads on AWS. Dogfooding can be extremely valuable, especially if a massive portion of your staff have the same profession as your target market.
I've been using Google Cloud in a new role I started recently. There's definitely some parts of GCP I like, but whenever I use the Web Console I get the distinct impression nobody at Google actually uses it. If they did, I'm fairly sure all the annoying little warts I encounter would not exist.
EC2 was released in 2006.
Amazon.com last non ec2 server was 2012.
But a lot of features of amazon.com still don't run on the main AWS offerings.
GCP has not been out for that long. Also, quite easier to run an e-commerce site than to run the web's largest search engine as well as the largest email provider, as well as the largest maps provider. Each of these has an order of magnitude more traffic than amazon.com
I'm sure they'll get there though, just not the same scale. Not even close.
Large parts of AWS don't run on AWS either, due to issues with circular dependencies and similar problems. Similarly, if all of AWS onboards to use your AWS service, suddenly that's the business. Your 'real' customers and their traffic are dwarfed by the rest of AWS, making it hard to keep those real customers at the forefront. There's also an issue with those deps of cascading failures; having two separate fabrics/strata for internal and external offerings is similar to having a multi-regional offering in that it's more robust to random failures and such.
I agree, it's a huge problem. It also leads to divergence between internal variants of systems and external where in many cases the internal variants are leagues better than even what competitors have. But unfortunately since they're not on GCP they don't drive cloud adoption for Google.
* filtering traces by services has been broken in App Engine flex environments for more than a year.
* copy/pasting identifiers between places is a nightmare
* their IAM design is somehow worse than AWS. It’s so impressively bad I can’t even be mad. My favourite part of their IAM approach is how they have consolidated a majority of the IAM controls in the IAM page, but then random services like GCS have it defined elsewhere.
* not able to do basic time zooming of metric grafs on App Engine dashboard.
* multi-account paper cuts. Almost everyone on my team has their personal and work google accounts logged in. Whenever I send them a link to a dashboard or whatever, they end up getting a permission denied, without fail.
These are all just off the top of my head. Many of them seem silly and minor (and they are!) but there’s enough of them that I kinda dread doing anything in the Cloud console now. I need to take more time to get productive in the gcloud CLI I guess...
I always considered the Google Cloud approach of a "single account, multiple projects" a lot cleaner than the AWS "hundreds of accounts" approach. Do you not find this the case?
Google multi account support within a single browser is a pain. It kinda works until it doesn't. I'm sidestepping this issue by using distinct chrome profiles for work and personal.
In the other hand, I've not found Amazon multi-account situation to be cozy either. IIRC you literally have to logout and login again or use assume role and the switch applies to all the open tabs.
> I get the distinct impression nobody at Google actually uses it. If they did, I'm fairly sure all the annoying little warts I encounter would not exist.
For what it's worth, the internal-only systems also have warts ;)
So you are making a case for smaller companies run by different people in different ways? So that we don't have huge outages with common systems shared across entire platforms misbehave?
When you really care about high availability and security you really don't want all your systems run with the same software, hardware, and coded by the same teams.
What does google (or amazon/msft) do to ensure a software echo chambers are not made within their infrastructure that potentially could cause mass scale outages by way of the same bug or bugs propagating through their systems?
GCP, AWS, and Azure is the grate decentralization of the internet.
Afaik, these are all as homogenous as they can make them, but there are limits to that. It's hard to move big, old things forward which creates some diversity, but that's probably worse than consistently running the latest stuff everywhere.
If you want heterogeneous environments you have to cobble it together yourself by using multiple services.
There are only 3 things I can say about this situation.
1) These issues are currently unrelated.
2) We learn a lot from these situations.
3) A lot of these types of issues can be mitigated by running in more then 1 region.
I really cant promise that today's situations will never happen again. There are a lot of moving pieces in our system and sometimes there are things outside of Google's control.
But data centers are typically designed with network and power failures in mind, not? Isn’t this why these kind of ring based network topologies exist, so that whenever a single network connection fails, it can still easily be routed around?
Almost always, yes, but the problem is that everyone has to start routing around the problem and it creates congestion. Those redundant pipes don't sit idle. They are sharing the traffic.
As mentioned in another thread, in this case, Google has rerouted google.com traffic out of the region to try to mitigate the congestion.
On a smaller scale, to link up a few datacenters that are a few miles apart? Sure. On a grand scale though, no. Nobody's running an extra undersea cable from Japan to Singapore so that they can have a ring topology. Or trenching a second PBps of cables across the Appalachian Mountains. When something like that gets busted you go and reroute your least important traffic and send out the repair crew.
> The disruptions with Google Cloud Networking and Load Balancing have been root caused to physical damage to multiple concurrent fiber bundles serving network paths in us-east1, and we expect a full resolution within the next 24 hours.
From the dashboard. Looks like this can be blamed on an Act of Backhoe.
No, not unless you are someone like Netflix. Usually you can configure multi-region failover and such and that will keep your things running. It is more expensive but for most use cases I think the cost is still less than the dev time/complexity of setting up multi-provider workflows and the inevitable duplication of resources (which is part of the cost of multi-region anyway)
You can move 1.6TB between providers in a month for the same price as a single beefy DB server (m4.16xlarge here). That's a whole lot of logical replication..
No. And there's been a lot of talk recently about multi-provider being the right strategy to mitigate downtime, which IMHO is a farce peddled by expensive consultants. The parent comment is correct - this is why availability zones and regions have been established by each provider.
For the large majority of businesses investing in infrastructure-as-code far outweighs any crazy HA, redundant, multi-provider, whizzbang whatever setup you may have.
I have an awesome demo I give running a complex stateful workload across cloud providers to show off the system that I work on. What I have learned from giving that presentation many times is that while it is nice to say you can run cross cloud, for most workloads you should just pick one cloud, and be able to move to another provider if you ever need to.
Well, sure, if you hate your devops team and you want to make sure they can’t use any of the proprietary functionality of either provider. At which point, if you want to be managing a fleet of vanilla Linux boxes yourself, why use a cloud provider at all?
This seems overly negative. There are lots of ways to do hybrid clouds, especially if you’re doing it for only the more critical parts of your application.
For the same reason you want "to lock in" (meaning use) any solution. You do not want to build or operate it yourself. Why don't you take this further? Why to use a water utility if you can just drill your own wells? Most businesses are better of on cloud because their core business is not to build and operate datacenters but provide services to their customers (on the top of datacenters running their apps).
It's not really that I "want to lock into a cloud provider". Sometimes I simply don't have the human bandwidth available to handle devops and sysadmin work while building the actual product.
"Outsourcing" those functions to cloud services can be big win for a small team. Like all engineering, it's a trade off.
* You should not be locking yourself into proprietary functionality of a cloud provider unless you are deeply interested in what happened to Oracle customers getting raked over the coals happening to you.
* DevOps teams can be multi-cloud relatively easy when using infrastructure as code tooling (Terraform, Packer, etc) and traditional DevOps practices
* Why manage a fleet of vanilla boxes when you can use vanilla boxes with Kubernetes and not get gouged by cloud providers in the first place?
You don't need to jump off the hype train if you never got on in the first place.
Proprietary managed services can save a lot of dev/setup/SRE time though. Many businesses have more pressing things to work on than spending dev time to prevent vendor lock-in.
Not yet, but it seems obvious to me that the GP was referring to a situation where the price changes and then you are getting gouged. That's exactly what the negative connotations of lock-in refer to.
Each provider will seek to make you take their one true path, or you need to do your own engineering.
Using the providers path isn’t necessarily gouging, but it isn’t cost optimized either. The answer depends on you.
That said, cloud is like any tenant/landlord relationship. Your rights are linked to time and are whatever your contract provides. If you didn’t like Office 2007, you didn’t buy it. If you don’t like Office 365, 2021 edition, too bad.
It's not quite that black and white. You can use common/open APIs and cross-provider tooling whenever available and provider-flavored ones where necessary. It's more effort, but still less than hand-rolling everything.
Of course that only works as long as you're swapping out largely replaceable parts. If you built everything around some proprietary service then yeah, you've tied yourself to that anchor.
Cost+speed of scalability, and managed services. If you rarely need to scale, your workloads are all predictable, and you don't need managed services/support, you should just buy some VPSes or dedicated boxes.
It's quite common in cloud solution design to design for failure. One of the common assumptions that we hold to is that one region may go down. Other examples: Assume an instance of an app can go down. Assume a VM can go down. Assume a DC can go down.
However much we technical people might salivate at the prospect of designing a multi-cloud solution, for the vast majority of businesses it simply isn't worth the cost / complexity. I'd wager 90-something percent of applications could suffer multi-hour outages without impacting business function to any measurable degree.
Plus the fact that without serious investment, you're probably more liable to decrease availability by going multi-cloud thanks to the increased system complexity.
The real trick here, which many people don’t want to look at, is to avoid overly centralizing your workflow.
I can get a lot of work done while Outlook is down. Hell, probably more work done.
If our build server is down I can work for a couple hours (unless we’ve done something very bad). Same for git or our bug database or wiki or or or. When I get stuck on one thing I can swap to something else every couple of hours. And there is always documentation (writing or consuming).
But if some idiot, hypothetically speaking of course, puts most of these services into the same SAN, then we are truly and utterly screwed if there is a hardware failure.
Similarly if you make one giant app that handles your whole business, if that app goes down and there are no manual backups you might as well send everybody home.
I went to get a drink the other day and the place looked funny. They’d tripped a circuit breaker and the whole kitchen lost power. But the registers and the beverage machines were on a separate circuit. And since they sold drinks and food in that order, they stayed open and just apologized a lot. Whoever wired that place knew what they were doing.
Probably lost 1 of 3 phases. You're quite right in that the decision of what phase a circuit is on has a lot to do with business, and hopefully no major repurposing of the space without rewiring the space has occurred. For lighting, you'd want 1/3 of fixtures per room to go out, not 1/3 of your rooms in their entirety. For appliances and receptacles, you'd rather lose a whole function (the kitchen) than be able to cook but not do dishes, with every function trying to figure out oddball workarounds.
I don’t think it’s happened (yet) although some of the earlier outages when AWS was younger were pretty far reaching. I think all of S3 has gone down a time or two.
Some APIs were impacted, because they are global by nature (e.g create-bucket). But S3 was working fine in all other regions, for existing buckets.
However, many websites were affected, because they didn't use any of the existing S3 features that allow for regional redundancy, simply because S3 had been so reliable they didn't know/think they needed to have critical assets in a bucket in a 2nd region that they could fail over to.
Admittedly, even the AWS status page was impacted, because it also relied on S3 in us-east-1.
S3 has done a lot of work to improve matters since, and mechanisms have been put in place to ensure that all AWS services don't have inter-region dependencies for "static" operation.
However, it is still incorrect to claim that it was all of S3. Many customers who use S3 only in other regions were totally unaffected.
The chance that AWS goes down is much smaller than anything else going down. There are many SPOFs in a typical smaller company setup, most of those are not even obvious to the operators.
AWS had a multi-hour total S3 outage in us-east-1 in February 2017 that knocked out a huge number of things mostly because it turns out that a huge share of their customers run in only 1 region and it's us-east-1. Things mostly continued to work in other regions.
I recall Azure had some sort of multi-region database failover disaster that took several regions offline, and GCP has had several global elevated latency/error rate events, but I don't think that any cloud provider has been "down" in the sense that the word is usually used.
This is one of the reasons things like Federated Kubernetes is being worked on. Stick a CDN in front and your compute can be migrated from cloud to cloud. You still need to do a lot of thinking about data though.
To somewhat echo BurritoElPastor's comment, running a system/app that can be run in multiple clouds is orders of magnitude more difficult than just running a system/app that can be run in multiple regions.
And, not to be snarky, but many of the other responses that are along the lines of "It's not really that difficult to run in multiple clouds" - let's just say I have trouble believing these commenters have real world experience actually doing this. I'm not saying it's impossible, but it is extremely difficult for any system of reasonable complexity with a dev team of, say, 10 or more people.
And, if you can stomach the cost, you do give up the ability to really use any of the proprietary (and often times awesome) functionality of a particular provider, which can put your dev velocity at a big disadvantage.
It's not trivial but it's also not an order of magnitude more difficult anymore, as you describe it. There is a reason why Kubernetes gets a lot of backing from corporate customers - precisely because it hides and abstracts most of the underlying infrastructure and provides platform-agnostic primitives that make sense at the application level.
Once you have deployed your stack on Kubernetes, you can pretty much run it on any cloud or infrastructure with minor tweaks at most.
More than one region is pretty easy, more than one provider is harder (especially if your workload is designed from the ground up for it.) But, yes, just as multi-region protects you from things mere multi-AZ doesn't, multi-provider protects you from even more.
If you're running in multiple clouds for HA/DR reasons, you are limited to the lowest common denominator of features/services between them. Or maintaining multiple codebases/architectures, and the massive pile of issues that entails. I am not a fan of multi-cloud for this reason.
Multiple regions, as long as your provider offers all of the services, you can have a carbon copy. Much easier.
It depends on your needs, your architecture, your risk tolerance, etc. I think for most people "Use multiple regions" is the answer that strikes the correct balance. It probably isn't the correct answer for everyone.
Certain terms and conditions may apply :) Carbon copy of a static website or one whose data is only a one-way flow from some off-cloud source of truth? Sure! Multi-master or primary-secondary with failover? Stray too far from the narrow path of specialized managed solutions and things get very complex, very quickly. That being said - it's mostly just the nature of the beast. If you're not able to tolerate a regional outage, multi-region is a pill you're going to have to swallow, no buts about it.
For some customer it is the right thing for other customers it may not be the right thing.
Every provider will have failures. So the question mostly boils down to does paying for more then 1 region cost more or less then paying for the the lost productivity or revenue of an outage like this.
From some places the most costly things they spend money on is employees. If your whole company comes to a stop for even 1 hour. It may cost more then the engineering effort for multi zone, multi region or multi cloud for your critical environments.
Not sure why they closed that one at 9:12 just to open a new one at 10:25. We didn't see any traffic coming to us-east1 during that time period so I would assume the original issue is still the root cause.
Yeah, that happens sometimes based on which team notices, thinks it might be different and then opens an outage.
Sorry for the confusion, and yes, the fiber link issue is the root cause. Draining the Google.com traffic presumably resolved the issue for you, though you may still be seeing elevated latency as the updates suggest.
Since we use GCP Global LBs I presume that "draining the Google.com traffic" also meant that you're diverting all global LB traffic, which is what we see. The second incident (the OP's link) indicates that but at first it was very confusing to a customer when the first issue was marked as resolved but we still saw no traffic being sent to us-east1 via our global LBs. If that makes sense.
This part was somewhat nuanced, so I wasn’t sure to post it: yes, if you are using GCLB, and have more than 1 healthy Region, we will also rebalance to avoid us-east- for now (though not so statically as that sounds, mumble mumble).
Edit: added this to the top level comment so more folks see it.
There were reports of 404 from Google Cloud Run earlier today (I can confirm that I got both a 404 and a successful load after retrying that website): https://news.ycombinator.com/item?id=20336102
Was it related, it is a bit odd to get a 404 instead of a 50x?
There is no service with 100% availability. You put multiple AZs in one region but nobody was ever pretending that regional failures were impossible, just that single-AZ failures are more common than regional failures. You want high availability, you want multi-regional. Above that you want multi-provider.
The same decisions that make regions fail also makes infra-region traffic cheaper. This is true for all large cloud providers. If you are okay paying more for internal network traffic you can get multiregional. But multi-AZ is still better than single-AZ. Up to you to decide if it’s worth it. For that you need good SLAs and (IMO) support contracts.
> What’s the point of AZs if you lose whole DCs at a time.
The point is that AZs are higher level than DCs, so that they provide pretty decent independence guarantees (though you can further derisk with multi-region.)
Well, in AWS. Google's zones have weaker independence assurances (actually, as I read it, no assurances), stating only that a zone “usually has power, cooling, networking, and control planes that are isolated from other zones” [0] as opposed to AWS’s “Availability Zones are physically separated within a typical metropolitan region” and “In addition to discrete uninterruptable power supply (UPS) and onsite backup generation facilities, they are each fed via different grids from independent utilities to further reduce single points of failure. Availability Zones are all redundantly connected to multiple tier-1 transit providers.” [1]
In a moment that's likely to be very, very frustrating for a large number of you that have businesses and customers that depend on G cloud, let's try to remember that somewhere there's an engineer or an SRE having a really hard day just trying to fix things.
Please, be kind and decent to each other, especially when things are hard.
I don't follow comments like these, should people refrain from criticising giant companies because there are people working at them? I don't understand the purpose of this comment
Complaining about the communication and response time of a company is different from yelling in the direction of some stressed engineer that they are useless and incompetent at everything they do. Sadly you get too much of the latter around the Internet.
Who is yelling in the direction of the "stressed engineer"? Does anybody have direct channel to those guys or you think they rigorously monitor the comment section of HN in the middle of an outage for yelling?
If there’s a different deployment that could have worked, then the one that did go wasn’t the “best”. Critique should be around the 5 whys the actual best wasn’t selected for.
Turns out this is the conclusion in CloudFlare’s update:
> “Our testing processes were insufficient in this case and we are reviewing and making changes to our testing and deployment process to avoid incidents like this in the future.”
They’re going after the definition of good for their deployments.
My hope is that we can be kind and decent to other people even in moments of stress. Take two very different example comments:
"This is a frustrating outage for us, a huge part of the attraction in Google Cloud has been the premise that we get the underlying reliability of Google's infrastructure. If we'd known what the reliability of Google in practice this year would look like, we might have stayed with AWS."
and
"Why are the stupid SRE's at Google even paid such absurd numbers if they can't even go a whole month without multiple hours of downtime."
Criticizing companies is find, just please remember there are real people there.
"Kind and Decent" doesn't seem like a high bar.
If "please be kind and decent" is too much of an ask, I pray we never work together.
> This is a frustrating outage for us, a huge part of the attraction in Google Cloud has been the premise that we get the underlying reliability of Google's infrastructure. If we'd known what the reliability of Google in practice this year would look like, we might have stayed with AWS.
if this statement you quoted is something you're not comfortable with, i have a hard time believing you have ever encountered criticism in your life.
One is calling for tolerance while the other is cheap snark. Also, one is about the situation at hand while the other is a comment about a comment. Also, one was written as an ordinary sentence while the other signals low effort as a snark booster. Perhaps most importantly, the accounts have very different histories. When we do moderation replies we're usually reacting to the account's overall pattern as much as to the specific post.
As someone in the infrastructure side of the house, people rarely understand all the things that go on behind the scenes to keep things running. The only time people notice you are when things go down.
I wish these guys and gals luck on getting things working.
From another comment, original issue [1] was closed at 9:12, so looks like they got it back up for a bit over an hour before it went down again. Post-mortem will be interesting.
There were (and continue to be) connectivity issues due to a subset of the fiber links having trouble. But that’s different from being “down”, it’s “just” an outage. We won’t declare the outage over until the impact is minimal.
I don't think there is any mismatch, it was always three nines. I guess the only mismatch is in claims that three nines is enough to not be noticeable or annoying to people.
AWS EC2 promises 4 9's (4.3 minutes of downtime/month) before their SLA kicks in, but they only give a 10% discount until availability dips below 99% (7.5 hours of downtime/month) when they give a 30% discount. If availability is below 95% (36 hours) in a month, they give a full refund.
For an individual instance, they only promise 90% availability.
Availability of running instances, I don't think they make any guarantees for availability unreserved on-demand instances (I don't see how they could).
Availability of network access to existing instances.
What you're talking about with provisioning capacity is a totally different matter. Provisioning availability is not guaranteed (unless you purchase reserved instances) and there are frequently periods where certain instance types are not available in certain AZs, though they do try to resolve that as fast as practicality allows them to. It really stinks sometimes though - especially if you get into a situation where something fails in your autoscaling group and there is no capacity available for a replacement instance. Usually you can get around that though by making sure your ASG is set up for multiple AZs, or worst case changing instance types (though that can be problematic in it's own way).
Yeah, you can now provision multiple instance types in an ASG which mitigates this somewhat.
I think people sometimes forget that the cloud isn't magic, and a sudden burst of requests for new instances needs somebody to actually rack up some servers.
I don't think that thinking about cloud computing in terms of nines is the correct way to frame the issue. cloud storage, maybe, but cloud gives you the cheap bricks you need to build the nines for your customers using replication, independent zones and clever routing.
now I wouldn't go as far as to say outages are the normal state of thing, but cloud trades high nines for cheap redundancy. a raid for compute, if you will, and as such a single zone deployment is going to have outages.
(and then there's soft layer which has multiple unplanned sev1 per week)
Looks like an external issue. "The Cloud Networking service (Standard Tier) has lost multiple independent fiber links within us-east1 zone. Vendor has been notified and are currently investigating the issue."
It's not that hard. In India because of so much construction related digging cuts OFCs, we do the path planning quite well and our redundancies get tested quite regularly whether you want to or not.
It can be hard. Getting redundant separated paths under/over railroad tracks, for example, might require political power that not everyone has. Google, of course, has plenty.
> Getting redundant separated paths under/over railroad tracks, for example, might require political power that not everyone has. Google, of course, has plenty.
But Google's vendors might have less. One would hope that Google is auditing claims of independence from vendors at least somewhat, but at some level they have to rely on vendor representation and SLAs if they aren't going to do it all themselves.
The companies who operate the cross-country backbone fibers have independently verified fibre maps and you can also audit them with their cooperation. And those who operate last-mile metro networks are usually highly reputed ISPs (at least in India where there is decent competition in this space) who have a lot to lose if their reputation is damaged. Also, the community of their customers is small and they all talk to each other. So it is hard to make fake claims and get away with it. Usually, when cable cuts happen, it is more a question of whose traffic is rerouted on the available paths and whose traffic is dropped. If you are high-paying customer with strong SLAs then your traffic is usually safe and will displace a lower SLA customer's traffic. You will notice latency spikes due to rerouting and maybe temporary glitches w.r.t link stabilization. Since you see this so often, your BGP timers etc are all tuned to be patient and avoid cascading failures.
During that AWS outage I was training people on [enterprise software] as part of the certification portion of [enterprise software company annual conference].
Nobody really wanted to be [enterprise software]-certified, but it was a way to get their employers to pay for them to go to the conference with cool talks and perks and such.
We delayed the training most of the day, and couldn't say it was AWS' fault because they were sitting in the audience, waiting to get certified.
People were about to riot, that was not a fun day.
Can you explain what's better about the AWS one? They both do, approximately, the same thing: provide a few paragraphs of background, approximately one paragraph describing the actual issue, and a few paragraphs describing concrete followups. The AWS one has more timestamps.
You aren't confusing this[0] with the postmortem, are you?
i don't quite follow your logic. something about glass houses and bad luck?
the whole point when something like this happens is for you to ensure that a region going down will not impact you - not to laugh at people that use another cloud or to assume that X is better than Y. That being said, there have been several Google related failures lately that don't help building confidence in the GCP offering - if you're just starting in the cloud space this may actually impact the choices you make when you pick your cloud provider.
My point was that, there was a comment from someone saying 'laughing from aws' and I was trying to point out that each service is most likely (or should be considered to be) as fragile (in the relative sense) as each other. So just because google have gone down several times, doesn't mean that AWS won't have a line of outages next.
Really, their services are much of a black hole to us.. we can't see _how_ they deploy their changes, what kind of reviewing they do etc. etc. Even down to how cleverly they have _actually_ architected their DCs.
So my point was to _not_ to laugh at those at google (or those using their services), because AWS might be next.
The whole 'I share the same glass house', was a sort of karma thing.. if someone who uses AWS is laughing at Google. If karma came round and took out AWS, not only would it affect the guy laughing at google, but I'd be the one affected as well as a multitude of other people... and the tables could be easily turned
The title says "almost 4 hours" (was posted at around 3 PM EST), but the incident was created at 10:25 AM PST, which is 1:25 PM EST. Has it been more like 2 hours or is there more to this incident?
Cloudflare took us down this morning, but also shielded us from the impact of this fiber cut, due to direct peering with google (I’m assuming over different fiber paths.)
It's good we've built this massive decentralized network to withstand even major nuclear attacks only to have massive parts of it fail because we've put so much in a few centralized and failable hands.
Kind of related to this, but these types of outages are why I moved from Google Play to Spotify for streaming music. Their infrastructure seems so large that things that should be a standalone service, like streaming music, are bound to be collateral damage when they mess something up on another service. Having everything provided by one company is convenient until it all goes down at the same time and you can't access your email, videos, or music because they all run on the same infrastructure.
I have the opposite conclusion as OP. Google doesn't use Google Cloud for anything critical, so I wouldn't use Google Cloud for anything critical or services that run on Google Cloud for that matter.
App Engine and Cloud functions were apparently returning error rates of > 30 percent overall between 11 a.m. and 3 p.m., with some projects experiencing a 100 percent error rate. GCS was also experiencing issues for the first half, which was attributed to the networking issues. Google said the networking issues were resolved initially but then stated they were investigating the GAE issues. Those issues were resolved, and the networking issue has been reopened as of 2:35 eastern: https://status.cloud.google.com/incident/cloud-networking/19....
ObPedant: notice in google's status page "...as of Tuesday, 2019-07-02 09:11 US/Pacific." This notation is useful because it's stable year round. I don't recommend 'PDT', instead colloquially 'out here on the left coast' or specifically US/Pacific.
When choosing a big cloud provider people forget that it's many orders of magnitude more complicated to run something at Google scale then to maintain one single server. For example the whole Stack overflow website runs on one or two servers. World of Warcraft also used to run on one single (blade) server. Chances are one server will be good enough for most use cases. And if you don't want to have it in your closet there are plenty of dedicated hosting and colocations.
With a cloud it also means that when there is an outage there are potentially many sites/services affected all at once, and there is potentially nothing customers can do to fix it other than wait (or plan in advance, and use/pay for multi-AZ/multi-region/multi-provider redundancy).
Such outages are also possible with traditional hosting providers, and when an outage does happen I'm not convinced whether a large public cloud would recover more quickly (due to better resourcing/expertise available to fix the problem), or a small hosting provider (which may have a smaller team, but the problems they deal with are at a smaller scale and more easily fixable).
Either way you probably want some kind of CDN independent of your cloud/hosting provider that can help survive some of these glitches.
>World of Warcraft also used to run on one single (blade) server.
This is... kind of true, but not really. For a single realm, general game interactions in the open world might have been hosted on a single blade, but there are a lot of support systems that do as much work, or more, that were not. The databases with all of the character information, login servers, instance servers, etc. etc. etc.
But even if you look at just the game server portion, there was a blade for every realm - you can't say World of Warcraft as a monolithic entity ran on a single blade server.
(I'm also not sure if the general game servers for a realm were only on one blade - my understanding is that each "continent" was it's own blade - Kalimdor, Eastern Kingdoms, Northrend, Outland, etc.)
Disclosure: I work on Google Cloud (but I'm not in SRE, oncall, etc.).
As the updates to [1] say, we're working to resolve a networking issue. The Region isn't (and wasn't) "down", but obviously network latency spiking up for external connectivity is bad.
We are currently experiencing an issue with a subset of the fiber paths that supply the region. We're working on getting that restored. In the meantime, we've removed almost all Google.com traffic out of the Region to prefer GCP customers. That's why the latency increase is subsiding, as we're freeing up the fiber paths by shedding our traffic.
Edit: (since it came up) that also means that if you’re using GCLB and have other healthy Regions, it will rebalance to avoid this congestion/slowdown automatically. That seemed the better trade off given the reduced network capacity during this outage.
Tangential question: does Google allow employees, not directly tasked with it, to represent the company online as they wish? Most companies I know of have a strict ‘do not speak for the company’ policy.
It's probably less "as they wish" and more "here's an approved statement" or "your role involves engaging with external parties, here are some guidelines"
It's a fine line. We are not allowed to represent Google in any kind of public discussion. But we can talk about some things we do, as long as we state it's our own opinion and we don't represent Google's views.
And don't disclose material nonpublic information (since that would run afoul of insider trading laws).
It's probably okay to say that we know the problem and here are the steps we're taking to mitigate it. It would not be okay to say something with large scale stock price implications for Google it another publicly traded corporation. For instance a Google employee shouldn't say something like "faulty solar panels fried Google's 10 largest data centers and twelve others have been lost to rebel drone strikes", even if false, since it could have a drastic impact on the earnings and future value of Google, Google's customers, and Google's competitors.
I'm not a lawyer, but if the information is false I don't think you could get dinged for insider trading. The legal approach that's used to prosecute insider trading is basically "theft of secrets".
As kyrra says below, you're in the clear if you state that this is just your opinion. Naturally, prefacing something terrible as "just your opinion" doesn't make it fine.
In my case, Cloud PR knows me, but I also knowingly risk my job (I clearly believe I have good enough judgment in what I post). If Urs and Ben think I should be fired, I'm okay with that, as it would represent a significant enough difference in opinion, that I wouldn't want to continue working here anyway.
Finally, for what it's worth, I have been reported before for "leaking internal secrets" here on HN! It turned out to be a totally hilarious discussion with the person tasked with questioning me. Still not fired, gotta try harder :).
To add my own story. I have made comments about other teams services on hacker news before. I've been contacted by the SRE responsible for the service I commented on asking me to correct what I said. Luckily no reports for leaking info. :)
Whenever I talk about the inner workings of Google I try to reference to external talks, books, or white papers to go along with my comments. Luckily a lot has already been set externally about how Google works.
That’s...that’s some petty fucking shit. I didn’t go through your comments but considering your email is in your profile, someone really had to have a hard-on to report you for leaks.
I would love to understand the though process of someone going out of their way to remove someone’s livelihood from them because of a comment on HN (when applied in a normal circumstance of adding additional information or correcting a misconception — I’m clearly not saying that bonehead comments shouldn’t have consequences.)
> I would love to understand the though process of someone going out of their way to remove someone’s livelihood from them because of a comment on HN
This is not so difficult though. You just need to adjust your starting point to someone who doesn't like boulos' first. That's not so difficult IMO, it's a large org and boulos' seems to be a fairly prolific commenter here.
If you are their co-worker and believe he shared some info that shouldn’t be public, wouldn’t it be a simple curtesy to email them and get some clarity? That seems like a reasonable thing to do.
On the other hand, to anonymously submit a complaint feels, to me, like a personal attack. Someone who simply doesn’t like them in for whatever reason. To me, that action seem petty.
You're assuming that the person making the report said "boulos needs to be fired!".
Maybe the person making the report said "Hey, I found some internal details on this external site. I'm not sure if this is allowed. Maybe someone who knows more should take a look at it, here's the link to the page."
Their email is in their profile. I would think it is sensible to reach out to them directly or speak with your manager to get a second opinion.
Submitting a complaint to an internal review because “you’re not sure it’s allowed” is really petty.
In my opinion, and experience, folks who have good intentions usually pull you to the side to get a feel for a situation before filing a formal complaint.
woah that's weird. This (Hacceity) is a social media alias of mine. For a moment I thought I wrote this. Did you come across the word in the Mars trilogy too?
The character Sax is asked to describe his belief system, and he says, essentially, that it is haecceity, the this-ness of things, that is his belief system, and I thought that was awesome.
Google employees are commenting publicly and on Hacker News all the time. If there is a policy of not speaking publicly about the company, this has been the most blatantly ignored policy ever.
Another FANG = Amazon? If so, Amazon is pretty restrictive in how it wants employees to communicate about internal activities. Most people err on the side of caution and don't comment publicly.
Apple employees comment on Hacker News all the time: they just don’t identify themselves as speaking for the company and make sure to only talk about publicly available information.
I work at Google on an open source project and comment on it frequently.
One of the things I really like about working at Google is that they place a lot of trust in the judgement of the individual employees. I generally make it clear when I'm stating my personal opinion versus the "official" (for whatever that means given how informal the project is) one, but I don't have to carefully go through an approved list of talking points, run my HN by the legal department, etc.
Obviously, in certain situations, things get more official and formal. For example, when I went to Google IO to give a talk, we did have some documentation and coaching beforehand about how to handle various questions we might get about non-public stuff, other projects related to ours, etc. We are also expected to run any slides by legal before being publicly shown in a venue with a wide audience like IO. But, even then, the legal folks I've worked with have been a pleasure to talk to.
The company's culture is basically "We hired you because you're smart. We trust you to use your brain." It would be squandering resources to not let their employees use their own intelligence and judgement.
You seem to have 3 status messages on the dashboard at 14:31, 14:44 and 14:48 with exactly the same contents. Were those messages really posted 3 times, or did something go wrong and they got duplicated?
Your boss picked a ridiculous time to nitpick over wording, to shout and add stress to an already difficult situation, and giving up accuracy and precise understanding at a time those are most important.
I don’t think he’s the one nitpicking. From a business perspective the site was down. Nitpicking is telling him: No it is in fact up, the customer just can’t use it.
"Customers are complaining they can't access a thing"
"OK, we have databases up, load balancers responding, DNS records check out, last change/deployment was at this time, all these services are up, and the latest test suite is running all green, this narrows down the places where a failure might be with some useful differential diagnosis, now we can move attention to.."
"I DON'T CARE THAT YOU THINK THINGS ARE WORKING, IF THE CUSTOMER CANNOT GET TO IT, IT'S DOWN"
"Thanks for that helpful input, let's divert troubleshooting attention from this P1 incident, and have a discussion about what "DOWN" means. You want me to treat the working databases as down because the customer can't get to them? Even though they're working?
It's like the hatred for "works on my machine". "WELL I'M NOT RUNNING ON YOUR MACHINE". No you aren't, but this demonstrates the current build works, the commands you're using are coherent and sensible, excludes many possible causes of failure, and adds useful information to the situation.
Internal communications differ from external customer-facing ones.
For troubleshooting and internal use of course you want to describe the outage in precise terms (while being very sure you are not downplaying the impact).
For talking to customers, a sufficiently slow response is the same as no response, and nothing is more irritating than being told 'it's not really down' when they can't use the service.
As someone who lost critical business functionality yesterday when my appengine instances returned only 502s for 5 hours, I find the idea it was "a ridiculous time to nitpick" hilarious.
My customers don't care that the network is down, the servers are down, or aliens have landed. The severity is the same and our infrastructure, regardless of the cause, was down.
During the impacted time period, we did a full DR failover to appengine instances we spun up in west2. This was not a minor hiccup.
My customers don't care that the network is down, the servers are down, or aliens have landed. The severity is the same and our infrastructure, regardless of the cause, was down.
But the people who have to fix it, desperately care about which specific part is down. That's just about the highest priority information they need. Honing in on where the problem is, is one of the few ways to get to fixing the problem. Having a boss shout that "everything is down, it's all broken" is the opposite of identifying the problem.
find the idea it was "a ridiculous time to nitpick" hilarious.
What? You lost critical business functionality for 5 hours, and you'd rather the boss was shouting at the workers because the wording used doesn't accurately reflect the boss's understanding, instead of the workers working on solving the problem?
Hacker News: The real status page and help desk for the internet.
Do companies realize how absurd this is?
ETA: It seems someone at Google had a change of heart, and most of what boulos posted in this thread has been added as updates to the official google status page. Better late than never, I guess, especially if this is the start of a trend in outage reporting.
seriously, they've got a text field on the official status page, why not put the text boulos posted here in that instead of the meaningless text they've got there?
Can you expand on why you find it “meaningless”? As my other comment says, I’m not in SRE and the real people fixing it are trying their best to remediate the problem. I agree that the text I posted (with blessing from SRE!) gives you some more detail, but you can’t do anything differently with it, right? What about the new text do you prefer? (We’re happy to improve!)
Your, even brief, description is interpretable by your clients and some customers - and is actually really informative. It helps estimate the magnitude of the issue, and the types of downstream problems to expect or avoid.
Knowing an astroid took out the entire continent tells you something about the repairability, resources required to fix the problem, and generally provides context for later updates, as opposed to other contexts like a cut fiber line, a burning datacenter or a bad power supply.
I think the difference between your comment here and the info on the status page is that after reading your comment i feel like i know what's happening.
you're right, there's no additional actionable information there, the status page contains everything i actually need to know. but a bit more information makes me feel better. I guess the difference is your comment reassures me that you actually know what's going on. the status page text (prior to the 14:31 update) could equally mean "we've got this under control" or "shit's broken and we don't know why"
First case gives you all the information needed (denial), however in the second case I understand the situation much better. I wouldn't call the text on the status page meaningless though - it's pretty nice and concise already (which is what you want in a "crisis"). Just some brief description of the problem would be good, even though technically unnecessary.
I work for AWS. There is typically a balance that has to be struck when sharing information with customers. I would imagine this goes for most companies, which is why it isn't until a post-mortem that the messaging is fully refined.
True, but I'd argue that the "Customer Obsession" priciple would drive you to attempt some sort of good-faith effort towards real-time communication.
Back when I worked there, the AWS status board was (and probably still is) terrible b/c Service teams owned that communication channel, not AWS Support. That really ought to have been changed. Service teams don't have the time or incentive to give real-time updates. Why not just let the people who know the customers best deal with parsing the TT and giving updates?
> Back when I worked there, the AWS status board was (and probably still is) terrible b/c Service teams owned that communication channel, not AWS Support. That really ought to have been changed.
It has.
> Service teams don't have the time or incentive to give real-time updates. Why not just let the people who know the customers best deal with parsing the TT and giving updates?
The escalation team inside PS now drafts customer messaging within ~5 minutes of the impact being identified (usually about 5 minutes into an event) and if the impact is significant enough to post to the public dashboard, than may take another 5 minutes. Depending on the type of impact, affected customers will be notified via the personal health dashboard.
PS owns the tooling that does this, and is responsible for driving the process, but the service org's (e.g. EC2, S3 etc.) representative often makes the call on whether to post to the public status page or not (depending on the scale of the impact, e.g. 20% API failure rate for 5% customers probably won't make the status page, but affected customers
will get notices).
TT is almost out ... but the PS tooling supports it and its replacement, and provides easy access and summaries for internal teams (so you don't need to refresh TT or subscribe to the ticket just to see what the status is).
I'm late to this party but I just wanna add, boulos could be wrong or inaccurate and it wouldn't be a big deal. Those status updates are communications to customers, and thus tend to be more conservative. Inaccuracy is a much bigger deal there than the informal status here.
Sadly, the closer you are to the action of a thing like this (for example, I'm on NetInfra SRE and we were part of the group that put in place the current mitigations you're seeing work now), the less you can say without fear of subtle inaccuracy or releasing non-public information.
The outage information is fairly reasonable. Not everyone cares (nor should they!) about the why only what the situation is, and that people are on it. This is extra detail.
I mostly responded because there was confusion downthread (and in the title) about being “down”. During an outage is a tricky time for comms, so short corrections are best until a full postmortem can be done.
This reminds me of an incident in Sweden a couple of years ago.
We test our disaster alarms on a known schedule. And just a couple of years ago, during the peak vacation time in the summer, the alarm went off, off schedule.
This made the entire country panic. Were we being attacked? The agency that is supposed to let people know through the public channels like tv, radio etc were silent. They were themselves on vacation probably. The websites and apps they've setup were ridiculously underpowered and were basically DDOS'ed by the spike in traffic they were getting.
News outlets were also struggling, but did way better.
The only thing that withstood the sudden burst in traffic without a hitch was facebook and twitter.
The official statement i think was that the alarm was triggered by accident (never happened before, i think). But goes to show how badly our emergency response is setup.
It goes to show how badly it is set up for a false alarm. In a real emergency all the primary functions would go up (taking over radio broadcasts for example) so there wouldn't be the same problem. It is still bad of course because of the "cry wolf" factor.
340 comments
[ 3.6 ms ] story [ 251 ms ] threadYou make a good point though. Downtime seems to be awfully overloaded.
[0]: https://news.ycombinator.com/item?id=20213092
[1]: https://news.ycombinator.com/item?id=20077421
https://news.ycombinator.com/item?id=9428043
https://news.ycombinator.com/item?id=17576720
AFAIK Amazon are running a lot of actual production loads on AWS. Dogfooding can be extremely valuable, especially if a massive portion of your staff have the same profession as your target market.
I've been using Google Cloud in a new role I started recently. There's definitely some parts of GCP I like, but whenever I use the Web Console I get the distinct impression nobody at Google actually uses it. If they did, I'm fairly sure all the annoying little warts I encounter would not exist.
EC2 was released in 2006. Amazon.com last non ec2 server was 2012. But a lot of features of amazon.com still don't run on the main AWS offerings.
GCP has not been out for that long. Also, quite easier to run an e-commerce site than to run the web's largest search engine as well as the largest email provider, as well as the largest maps provider. Each of these has an order of magnitude more traffic than amazon.com
I'm sure they'll get there though, just not the same scale. Not even close.
The UI was maddeningly obtuse. This is from the second time I tried.. They did fix it eventually.
Very complex system for distributing new keys taking payments.
https://issuetracker.google.com/issues/124188941
* filtering traces by services has been broken in App Engine flex environments for more than a year. * copy/pasting identifiers between places is a nightmare * their IAM design is somehow worse than AWS. It’s so impressively bad I can’t even be mad. My favourite part of their IAM approach is how they have consolidated a majority of the IAM controls in the IAM page, but then random services like GCS have it defined elsewhere. * not able to do basic time zooming of metric grafs on App Engine dashboard. * multi-account paper cuts. Almost everyone on my team has their personal and work google accounts logged in. Whenever I send them a link to a dashboard or whatever, they end up getting a permission denied, without fail.
These are all just off the top of my head. Many of them seem silly and minor (and they are!) but there’s enough of them that I kinda dread doing anything in the Cloud console now. I need to take more time to get productive in the gcloud CLI I guess...
I always considered the Google Cloud approach of a "single account, multiple projects" a lot cleaner than the AWS "hundreds of accounts" approach. Do you not find this the case?
Google multi account support within a single browser is a pain. It kinda works until it doesn't. I'm sidestepping this issue by using distinct chrome profiles for work and personal.
In the other hand, I've not found Amazon multi-account situation to be cozy either. IIRC you literally have to logout and login again or use assume role and the switch applies to all the open tabs.
For what it's worth, the internal-only systems also have warts ;)
When you really care about high availability and security you really don't want all your systems run with the same software, hardware, and coded by the same teams.
What does google (or amazon/msft) do to ensure a software echo chambers are not made within their infrastructure that potentially could cause mass scale outages by way of the same bug or bugs propagating through their systems?
GCP, AWS, and Azure is the grate decentralization of the internet.
If you want heterogeneous environments you have to cobble it together yourself by using multiple services.
There are only 3 things I can say about this situation. 1) These issues are currently unrelated. 2) We learn a lot from these situations. 3) A lot of these types of issues can be mitigated by running in more then 1 region.
I really cant promise that today's situations will never happen again. There are a lot of moving pieces in our system and sometimes there are things outside of Google's control.
Are you implying that the cause of this outage is not Google's fault? If so, can you go into more details about that?
Datacenters also sometimes have other single points of failure such as DNS, but those are within the company's control.
https://www.networkworld.com/article/3373646/network-problem...
https://www.datacenterknowledge.com/uptime/equinix-power-out...
As mentioned in another thread, in this case, Google has rerouted google.com traffic out of the region to try to mitigate the congestion.
From the dashboard. Looks like this can be blamed on an Act of Backhoe.
For the large majority of businesses investing in infrastructure-as-code far outweighs any crazy HA, redundant, multi-provider, whizzbang whatever setup you may have.
But the degree of independence provided by AZs is not constant across providers, despite similar terminology.
You are really limiting your tech stack by using standardized things like Jenkins, Docker, K8, mqtt, kafka.
"Outsourcing" those functions to cloud services can be big win for a small team. Like all engineering, it's a trade off.
* DevOps teams can be multi-cloud relatively easy when using infrastructure as code tooling (Terraform, Packer, etc) and traditional DevOps practices
* Why manage a fleet of vanilla boxes when you can use vanilla boxes with Kubernetes and not get gouged by cloud providers in the first place?
You don't need to jump off the hype train if you never got on in the first place.
Using the providers path isn’t necessarily gouging, but it isn’t cost optimized either. The answer depends on you.
That said, cloud is like any tenant/landlord relationship. Your rights are linked to time and are whatever your contract provides. If you didn’t like Office 2007, you didn’t buy it. If you don’t like Office 365, 2021 edition, too bad.
Of course that only works as long as you're swapping out largely replaceable parts. If you built everything around some proprietary service then yeah, you've tied yourself to that anchor.
Cost+speed of scalability, and managed services. If you rarely need to scale, your workloads are all predictable, and you don't need managed services/support, you should just buy some VPSes or dedicated boxes.
This is not to excuse the downtime in any way.
Plus the fact that without serious investment, you're probably more liable to decrease availability by going multi-cloud thanks to the increased system complexity.
I can get a lot of work done while Outlook is down. Hell, probably more work done.
If our build server is down I can work for a couple hours (unless we’ve done something very bad). Same for git or our bug database or wiki or or or. When I get stuck on one thing I can swap to something else every couple of hours. And there is always documentation (writing or consuming).
But if some idiot, hypothetically speaking of course, puts most of these services into the same SAN, then we are truly and utterly screwed if there is a hardware failure.
Similarly if you make one giant app that handles your whole business, if that app goes down and there are no manual backups you might as well send everybody home.
I went to get a drink the other day and the place looked funny. They’d tripped a circuit breaker and the whole kitchen lost power. But the registers and the beverage machines were on a separate circuit. And since they sold drinks and food in that order, they stayed open and just apologized a lot. Whoever wired that place knew what they were doing.
The only S3 event here was limited to us-east-1: https://aws.amazon.com/premiumsupport/technology/pes/
Some APIs were impacted, because they are global by nature (e.g create-bucket). But S3 was working fine in all other regions, for existing buckets.
However, many websites were affected, because they didn't use any of the existing S3 features that allow for regional redundancy, simply because S3 had been so reliable they didn't know/think they needed to have critical assets in a bucket in a 2nd region that they could fail over to.
Admittedly, even the AWS status page was impacted, because it also relied on S3 in us-east-1.
S3 has done a lot of work to improve matters since, and mechanisms have been put in place to ensure that all AWS services don't have inter-region dependencies for "static" operation.
However, it is still incorrect to claim that it was all of S3. Many customers who use S3 only in other regions were totally unaffected.
IMO, you’re better off with a private data center or colo and separate integrations with cloud.
It’s happened more than once with Azure and GCP. I think it happened once with AWS, but not positive there.
I recall Azure had some sort of multi-region database failover disaster that took several regions offline, and GCP has had several global elevated latency/error rate events, but I don't think that any cloud provider has been "down" in the sense that the word is usually used.
https://www.theregister.co.uk/2013/08/17/google_outage/
Here’s one that’s on Azure. Not a 100% total outage like above, but bad enough most I know in the industry would call it being down:
https://www.zdnet.com/article/windows-azure-suffers-worldwid...
If I get a free moment, I’ll dig up other examples, but those were ones that were easy to find.
Redundant Array of independent Data Clouds.
I guess for RAID 5 would I need a min of 3 regions or 3 separate cloud providers.
And, not to be snarky, but many of the other responses that are along the lines of "It's not really that difficult to run in multiple clouds" - let's just say I have trouble believing these commenters have real world experience actually doing this. I'm not saying it's impossible, but it is extremely difficult for any system of reasonable complexity with a dev team of, say, 10 or more people.
And, if you can stomach the cost, you do give up the ability to really use any of the proprietary (and often times awesome) functionality of a particular provider, which can put your dev velocity at a big disadvantage.
Once you have deployed your stack on Kubernetes, you can pretty much run it on any cloud or infrastructure with minor tweaks at most.
Multiple regions, as long as your provider offers all of the services, you can have a carbon copy. Much easier.
It depends on your needs, your architecture, your risk tolerance, etc. I think for most people "Use multiple regions" is the answer that strikes the correct balance. It probably isn't the correct answer for everyone.
Certain terms and conditions may apply :) Carbon copy of a static website or one whose data is only a one-way flow from some off-cloud source of truth? Sure! Multi-master or primary-secondary with failover? Stray too far from the narrow path of specialized managed solutions and things get very complex, very quickly. That being said - it's mostly just the nature of the beast. If you're not able to tolerate a regional outage, multi-region is a pill you're going to have to swallow, no buts about it.
For some customer it is the right thing for other customers it may not be the right thing.
Every provider will have failures. So the question mostly boils down to does paying for more then 1 region cost more or less then paying for the the lost productivity or revenue of an outage like this.
From some places the most costly things they spend money on is employees. If your whole company comes to a stop for even 1 hour. It may cost more then the engineering effort for multi zone, multi region or multi cloud for your critical environments.
I recently left Google to start a startup and now everything is falling apart.
Not sure why they closed that one at 9:12 just to open a new one at 10:25. We didn't see any traffic coming to us-east1 during that time period so I would assume the original issue is still the root cause.
Sorry for the confusion, and yes, the fiber link issue is the root cause. Draining the Google.com traffic presumably resolved the issue for you, though you may still be seeing elevated latency as the updates suggest.
Edit: added this to the top level comment so more folks see it.
The same decisions that make regions fail also makes infra-region traffic cheaper. This is true for all large cloud providers. If you are okay paying more for internal network traffic you can get multiregional. But multi-AZ is still better than single-AZ. Up to you to decide if it’s worth it. For that you need good SLAs and (IMO) support contracts.
The point is that AZs are higher level than DCs, so that they provide pretty decent independence guarantees (though you can further derisk with multi-region.)
Well, in AWS. Google's zones have weaker independence assurances (actually, as I read it, no assurances), stating only that a zone “usually has power, cooling, networking, and control planes that are isolated from other zones” [0] as opposed to AWS’s “Availability Zones are physically separated within a typical metropolitan region” and “In addition to discrete uninterruptable power supply (UPS) and onsite backup generation facilities, they are each fed via different grids from independent utilities to further reduce single points of failure. Availability Zones are all redundantly connected to multiple tier-1 transit providers.” [1]
[0] https://cloud.google.com/compute/docs/regions-zones/
[1] https://docs.aws.amazon.com/whitepapers/latest/aws-overview/...
Please, be kind and decent to each other, especially when things are hard.
[0] https://www.law.cornell.edu/uscode/text/26/7701
Criticizing Google is fine, but sometimes, the best deployments to production can go wrong.
> “Our testing processes were insufficient in this case and we are reviewing and making changes to our testing and deployment process to avoid incidents like this in the future.”
They’re going after the definition of good for their deployments.
If you're a paying customer, you should be free to criticize as you damn well please.
Downvoters pls link here the yelling you have seen.
"This is a frustrating outage for us, a huge part of the attraction in Google Cloud has been the premise that we get the underlying reliability of Google's infrastructure. If we'd known what the reliability of Google in practice this year would look like, we might have stayed with AWS."
and
"Why are the stupid SRE's at Google even paid such absurd numbers if they can't even go a whole month without multiple hours of downtime."
Criticizing companies is find, just please remember there are real people there.
"Kind and Decent" doesn't seem like a high bar. If "please be kind and decent" is too much of an ask, I pray we never work together.
if this statement you quoted is something you're not comfortable with, i have a hard time believing you have ever encountered criticism in your life.
I wish these guys and gals luck on getting things working.
[1] https://status.cloud.google.com/incident/cloud-networking/19...
There were (and continue to be) connectivity issues due to a subset of the fiber links having trouble. But that’s different from being “down”, it’s “just” an outage. We won’t declare the outage over until the impact is minimal.
I'm going to guess load balancer cascading failures.
My impression from their PR seems to mismatch the number of outages and issues lately.
For an individual instance, they only promise 90% availability.
What you're talking about with provisioning capacity is a totally different matter. Provisioning availability is not guaranteed (unless you purchase reserved instances) and there are frequently periods where certain instance types are not available in certain AZs, though they do try to resolve that as fast as practicality allows them to. It really stinks sometimes though - especially if you get into a situation where something fails in your autoscaling group and there is no capacity available for a replacement instance. Usually you can get around that though by making sure your ASG is set up for multiple AZs, or worst case changing instance types (though that can be problematic in it's own way).
source: I used to work for AWS Support.
I think people sometimes forget that the cloud isn't magic, and a sudden burst of requests for new instances needs somebody to actually rack up some servers.
now I wouldn't go as far as to say outages are the normal state of thing, but cloud trades high nines for cheap redundancy. a raid for compute, if you will, and as such a single zone deployment is going to have outages.
(and then there's soft layer which has multiple unplanned sev1 per week)
Due to network failures etc. number visible to customers is unlikely to be going to be much higher than that in cloud services.
But Google's vendors might have less. One would hope that Google is auditing claims of independence from vendors at least somewhat, but at some level they have to rely on vendor representation and SLAs if they aren't going to do it all themselves.
Accidents happen. Regularly. :D
please don't...
glass house and all that... but I also share the same glass house as you.. I don't want bad luck
... and it's only a fluke that this happened to google in eu-east1 and not AWS in X region and then you (and I) would be having a time of hell! :/
Nobody really wanted to be [enterprise software]-certified, but it was a way to get their employers to pay for them to go to the conference with cool talks and perks and such.
We delayed the training most of the day, and couldn't say it was AWS' fault because they were sitting in the audience, waiting to get certified.
People were about to riot, that was not a fun day.
Their last one was laughable in it's lack of self-awareness.
Can you explain what's better about the AWS one? They both do, approximately, the same thing: provide a few paragraphs of background, approximately one paragraph describing the actual issue, and a few paragraphs describing concrete followups. The AWS one has more timestamps.
You aren't confusing this[0] with the postmortem, are you?
[0]: https://cloud.google.com/blog/topics/inside-google-cloud/an-...
the whole point when something like this happens is for you to ensure that a region going down will not impact you - not to laugh at people that use another cloud or to assume that X is better than Y. That being said, there have been several Google related failures lately that don't help building confidence in the GCP offering - if you're just starting in the cloud space this may actually impact the choices you make when you pick your cloud provider.
So my point was to _not_ to laugh at those at google (or those using their services), because AWS might be next.
The whole 'I share the same glass house', was a sort of karma thing.. if someone who uses AWS is laughing at Google. If karma came round and took out AWS, not only would it affect the guy laughing at google, but I'd be the one affected as well as a multitude of other people... and the tables could be easily turned
https://9to5google.com/2018/05/23/google-play-youtube-music-...
https://www.digitaltrends.com/music/what-happens-to-google-p...
GAE and all other services still show green here, of course: https://status.cloud.google.com/
https://nickcraver.com/blog/2016/03/29/stack-overflow-the-ha...
This is... kind of true, but not really. For a single realm, general game interactions in the open world might have been hosted on a single blade, but there are a lot of support systems that do as much work, or more, that were not. The databases with all of the character information, login servers, instance servers, etc. etc. etc.
But even if you look at just the game server portion, there was a blade for every realm - you can't say World of Warcraft as a monolithic entity ran on a single blade server.
(I'm also not sure if the general game servers for a realm were only on one blade - my understanding is that each "continent" was it's own blade - Kalimdor, Eastern Kingdoms, Northrend, Outland, etc.)
As the updates to [1] say, we're working to resolve a networking issue. The Region isn't (and wasn't) "down", but obviously network latency spiking up for external connectivity is bad.
We are currently experiencing an issue with a subset of the fiber paths that supply the region. We're working on getting that restored. In the meantime, we've removed almost all Google.com traffic out of the Region to prefer GCP customers. That's why the latency increase is subsiding, as we're freeing up the fiber paths by shedding our traffic.
Edit: (since it came up) that also means that if you’re using GCLB and have other healthy Regions, it will rebalance to avoid this congestion/slowdown automatically. That seemed the better trade off given the reduced network capacity during this outage.
[1] https://status.cloud.google.com/incident/cloud-networking/19...
It's probably okay to say that we know the problem and here are the steps we're taking to mitigate it. It would not be okay to say something with large scale stock price implications for Google it another publicly traded corporation. For instance a Google employee shouldn't say something like "faulty solar panels fried Google's 10 largest data centers and twelve others have been lost to rebel drone strikes", even if false, since it could have a drastic impact on the earnings and future value of Google, Google's customers, and Google's competitors.
Even less obvious things like Google's plans for adding privacy features to the Chromium open source project can have a serious impact (see https://www.barrons.com/articles/google-chrome-privacy-quest...).
In my case, Cloud PR knows me, but I also knowingly risk my job (I clearly believe I have good enough judgment in what I post). If Urs and Ben think I should be fired, I'm okay with that, as it would represent a significant enough difference in opinion, that I wouldn't want to continue working here anyway.
Finally, for what it's worth, I have been reported before for "leaking internal secrets" here on HN! It turned out to be a totally hilarious discussion with the person tasked with questioning me. Still not fired, gotta try harder :).
Whenever I talk about the inner workings of Google I try to reference to external talks, books, or white papers to go along with my comments. Luckily a lot has already been set externally about how Google works.
I would love to understand the though process of someone going out of their way to remove someone’s livelihood from them because of a comment on HN (when applied in a normal circumstance of adding additional information or correcting a misconception — I’m clearly not saying that bonehead comments shouldn’t have consequences.)
This is not so difficult though. You just need to adjust your starting point to someone who doesn't like boulos' first. That's not so difficult IMO, it's a large org and boulos' seems to be a fairly prolific commenter here.
He certainly shares stuff I wouldn't be comfortable sharing, but then again he's a lot better connected and in the know than I am.
On the other hand, to anonymously submit a complaint feels, to me, like a personal attack. Someone who simply doesn’t like them in for whatever reason. To me, that action seem petty.
Maybe the person making the report said "Hey, I found some internal details on this external site. I'm not sure if this is allowed. Maybe someone who knows more should take a look at it, here's the link to the page."
Submitting a complaint to an internal review because “you’re not sure it’s allowed” is really petty.
In my opinion, and experience, folks who have good intentions usually pull you to the side to get a feel for a situation before filing a formal complaint.
If you haven't read them, you have to!
I work at another FANG with a roughly equal engineering community and I don’t see my kind commenting as much at all!
One of the things I really like about working at Google is that they place a lot of trust in the judgement of the individual employees. I generally make it clear when I'm stating my personal opinion versus the "official" (for whatever that means given how informal the project is) one, but I don't have to carefully go through an approved list of talking points, run my HN by the legal department, etc.
Obviously, in certain situations, things get more official and formal. For example, when I went to Google IO to give a talk, we did have some documentation and coaching beforehand about how to handle various questions we might get about non-public stuff, other projects related to ours, etc. We are also expected to run any slides by legal before being publicly shown in a venue with a wide audience like IO. But, even then, the legal folks I've worked with have been a pleasure to talk to.
The company's culture is basically "We hired you because you're smart. We trust you to use your brain." It would be squandering resources to not let their employees use their own intelligence and judgement.
As one of my old bosses said: I don't care that the site/service is technically running, if the customers can't reach it, then IT'S DOWN.
"OK, we have databases up, load balancers responding, DNS records check out, last change/deployment was at this time, all these services are up, and the latest test suite is running all green, this narrows down the places where a failure might be with some useful differential diagnosis, now we can move attention to.."
"I DON'T CARE THAT YOU THINK THINGS ARE WORKING, IF THE CUSTOMER CANNOT GET TO IT, IT'S DOWN"
"Thanks for that helpful input, let's divert troubleshooting attention from this P1 incident, and have a discussion about what "DOWN" means. You want me to treat the working databases as down because the customer can't get to them? Even though they're working?
It's like the hatred for "works on my machine". "WELL I'M NOT RUNNING ON YOUR MACHINE". No you aren't, but this demonstrates the current build works, the commands you're using are coherent and sensible, excludes many possible causes of failure, and adds useful information to the situation.
For troubleshooting and internal use of course you want to describe the outage in precise terms (while being very sure you are not downplaying the impact).
For talking to customers, a sufficiently slow response is the same as no response, and nothing is more irritating than being told 'it's not really down' when they can't use the service.
My customers don't care that the network is down, the servers are down, or aliens have landed. The severity is the same and our infrastructure, regardless of the cause, was down.
During the impacted time period, we did a full DR failover to appengine instances we spun up in west2. This was not a minor hiccup.
But the people who have to fix it, desperately care about which specific part is down. That's just about the highest priority information they need. Honing in on where the problem is, is one of the few ways to get to fixing the problem. Having a boss shout that "everything is down, it's all broken" is the opposite of identifying the problem.
find the idea it was "a ridiculous time to nitpick" hilarious.
What? You lost critical business functionality for 5 hours, and you'd rather the boss was shouting at the workers because the wording used doesn't accurately reflect the boss's understanding, instead of the workers working on solving the problem?
Do companies realize how absurd this is?
ETA: It seems someone at Google had a change of heart, and most of what boulos posted in this thread has been added as updates to the official google status page. Better late than never, I guess, especially if this is the start of a trend in outage reporting.
Knowing an astroid took out the entire continent tells you something about the repairability, resources required to fix the problem, and generally provides context for later updates, as opposed to other contexts like a cut fiber line, a burning datacenter or a bad power supply.
you're right, there's no additional actionable information there, the status page contains everything i actually need to know. but a bit more information makes me feel better. I guess the difference is your comment reassures me that you actually know what's going on. the status page text (prior to the 14:31 update) could equally mean "we've got this under control" or "shit's broken and we don't know why"
"No" vs "No, I already have plans with X"
First case gives you all the information needed (denial), however in the second case I understand the situation much better. I wouldn't call the text on the status page meaningless though - it's pretty nice and concise already (which is what you want in a "crisis"). Just some brief description of the problem would be good, even though technically unnecessary.
Back when I worked there, the AWS status board was (and probably still is) terrible b/c Service teams owned that communication channel, not AWS Support. That really ought to have been changed. Service teams don't have the time or incentive to give real-time updates. Why not just let the people who know the customers best deal with parsing the TT and giving updates?
It has.
> Service teams don't have the time or incentive to give real-time updates. Why not just let the people who know the customers best deal with parsing the TT and giving updates?
The escalation team inside PS now drafts customer messaging within ~5 minutes of the impact being identified (usually about 5 minutes into an event) and if the impact is significant enough to post to the public dashboard, than may take another 5 minutes. Depending on the type of impact, affected customers will be notified via the personal health dashboard.
PS owns the tooling that does this, and is responsible for driving the process, but the service org's (e.g. EC2, S3 etc.) representative often makes the call on whether to post to the public status page or not (depending on the scale of the impact, e.g. 20% API failure rate for 5% customers probably won't make the status page, but affected customers will get notices). TT is almost out ... but the PS tooling supports it and its replacement, and provides easy access and summaries for internal teams (so you don't need to refresh TT or subscribe to the ticket just to see what the status is).
Sadly, the closer you are to the action of a thing like this (for example, I'm on NetInfra SRE and we were part of the group that put in place the current mitigations you're seeing work now), the less you can say without fear of subtle inaccuracy or releasing non-public information.
I mostly responded because there was confusion downthread (and in the title) about being “down”. During an outage is a tricky time for comms, so short corrections are best until a full postmortem can be done.
We test our disaster alarms on a known schedule. And just a couple of years ago, during the peak vacation time in the summer, the alarm went off, off schedule.
This made the entire country panic. Were we being attacked? The agency that is supposed to let people know through the public channels like tv, radio etc were silent. They were themselves on vacation probably. The websites and apps they've setup were ridiculously underpowered and were basically DDOS'ed by the spike in traffic they were getting.
News outlets were also struggling, but did way better.
The only thing that withstood the sudden burst in traffic without a hitch was facebook and twitter.
The official statement i think was that the alarm was triggered by accident (never happened before, i think). But goes to show how badly our emergency response is setup.