Can not create a new Clusters or Node Pool and can not resize exiting Node Pools, as far as users are reporting it's happening in all regions too.
Error message when creating a new Cluster:
Deploy error: Not all instances running in IGM after 35m7.509000994s. Expect 1. Current errors: [ZONE_RESOURCE_POOL_EXHAUSTED]: Instance 'gke-cluster-3-pool-1-41b0abf8-73d7' creation failed: The zone 'projects/url-shortner-218503/zones/us-west2-b' does not have enough resources available to fulfill the request. Try a different zone, or try again later. - ; .
I also find it weird that on HN where normally people are very skeptical of any argument without data backing it, when it comes to this outage, people are assuming everything written here affects everyone.
Perhaps some of the issues are localized? Perhaps it's even user error (it happens, you know?). But because a small amount of HN users say "it's everywhere!" then suddenly people reach for their pitchforks.
Yeah almost all regions and zones for any compute instance have been exhausted since about 1pm PST on Friday. I finally got one up last night on us-east1, but my older cluster is basically SOL until it's fixed on us-west1. It went down for an upgrade and never came back up because of the same resource issue.
GKE does the creation of the VMs and setup of them, joining them to the cluster and applying labels for example.
The specific issue appears to be about creating new "node pools". Creating standard VMs in GCP works fine however, so this is specific to GKE and their internal tooling that integrates with the rest of GCP.
GKE doesn't (at least to my knowledge) allow you to create VMs separately and join them to the cluster in any kind of easy fashion.
It's actually not just GKE, there have been issues creating normal VMs since late Friday night. It seems anything that required creating VMs gave back resource exhaustion errors. I finally got a cluster for us-east1 setup last night so it looks like the resource issues are clearing up though.
Been trying to spin up vm instances all day, had to try every single zone just to get one up. Not only is this incredibly harmful to a technology business dependent on this infra, it wasnt obvious to me what the issue was until I tried creating instances. Nothing says, hey resources are constrained here, try this one. Just about ready to bite the bullet and move to aws.
Same question... why would you do that?
AWS is super stable most of the time. I have been running k8s over EC2 (not eks) for a year and works like a charm.
I've even run experiments using spot instances and it's pretty good (no guarantee there of course).
We did. Our customer had a bucket shared to a role they created, we spent a week back and forth trying to mount such bucket in our instance using fuse. Mounting with gcsfuse took 5 minutes (although no role used, so perhaps an unfair comparison). In general, I found Gcloud a lot easier to work with.
I use preemptible machines in autodialing and for first time did not have any machines available for multiple hours yesterday. I am wondering whether this falls under the normal preemptible behaviour or this service degradation.
I can assure you that's not the case! Also, while people like to repeat this meme, Google Cloud does have a formal deprecation policy (https://cloud.google.com/terms/), whose intent is to give you some assurances.
(I work at Google, on GKE, though I am not a lawyer and thus don't work on the deprecation policy)
Nothing, sort of. Subject to "Section 1.7 Modifications" of Terms:
b. To the Agreement:
Google may make changes to this Agreement, including pricing (and any linked documents) from time to time.
....
Google will provide at least 90 days’ advance notice for materially adverse changes to any SLAs by either: (i) sending an email to Customer’s primary point of contact; (ii) posting a notice in the Admin Console; or (iii) posting a notice to the applicable SLA webpage. If Customer does not agree to the revised Agreement, please stop using the Services. Google will post any modification to this Agreement to the Terms URL.
Which is the deprecation policy. (I mean I share your frustration with Google's what-appears-to-be-at-least haphazard policy of shutting down services instead of trying to gain traction. But, let's not misrepresent what they say).
I thought that 'subject to 7.2' meant that they can use the escape-hatch there 'substantial economic or material technical burden'? They can list anything under that.
I don't think it's wrong - they can deprecate any service they want to do whenever they want, unless people have paid for and signed a contract that says otherwise which I guess people aren't doing.
But the policy doesn't really guarantee anything at all does it, due to the reference escape-hatches? It might as well not exist?
> substantial economic or material technical burden
Is one engineer working on an old service to keep it alive commercially reasonable or a substantial burden? I don't know. Do you?
In practice this policy lets them shut off anything they want any time they want. Again it's their playground they can do what they want unless they signed a contract saying they'd do something else for you so I don't have a problem with it.
I think you're ascribing an unreasonable amount of bad faith here, and, to rephrase what I had here before, you're approaching this from an engineering perspective, not a legal one. And that's not how those things work.
To be clear, that policy is a contract. And those things would be decided by a jury. And if my understanding is correct, the reasonable person standard applies. So you can answer this yourself, do you think a reasonable person would believe that your interpretation is valid?
Because it makes more people feel comfortable enough to use your services and pay you, without actually binding you towards any sort of behavior that would cost you money.
There's a direct financial incentive here to use legalese to give the semblance of reliability without having to deliver on it
But it's Google, in the end the same CEO has to sign off on these changes (I'm sure the Maps price hike was approved by C-level and not just the Maps department).
> 7.1 Discontinuance of Services. Subject to Section 7.2, Google may discontinue
> any Services or any portion or feature for any reason at any time without
> liability to Customer.
Let's take a look at Section 7.2:
> 7.2 Deprecation Policy. Google will announce if it intends to discontinue or
> make backwards incompatible changes to the Services specified at the URL in
> the next sentence. Google will use commercially reasonable efforts to continue
> to operate those Services versions and features identified at
> https://cloud.google.com/terms/deprecation without these changes for at least
> one year after that announcement, unless (as Google determines in its
> reasonable good faith judgment):
>
> (i) required by law or third party relationship (including if there is a change
> in applicable law or relationship), or
>
> (ii) doing so could create a security risk or substantial economic or material
> technical burden.
>
> The above policy is the "Deprecation Policy."
To me that looks like a reasonable deprecation policy.
It's ok I guess but still lets them turn it off if, in their judgement, its an economic burden i.e. costing them money.
If they ever do deprecate something people have built on though they're gonna get absolutely crucified. That's probably better protection than any terms of service.
With this terms of service, none. Which is why people don't trust them.
If I pay you for a service that would take time to migrate off of, and you are making money off me now, I am going to be ripshit if you decide to just turn it off because it's suddenly not making money for you in the short term. Google's done this a lot, and the fact that don't provide concrete time lines in their contract gives even less reason to trust them
It's not about the contract. AWS doesn't even have a deprecation policy in the contract - seriously, GCP provides more legally binding guarantees than AWS. It's about trust.
People look at AWS's track record, and trust that. People look at Google's track record, overlook what to an inside-the-company Googler perspective are dramatically significant organizational boundaries or product lifecycle definitions that are very poorly communicated outside the company, mentally apply reputational damage from one part of Google (or from a preview-stage GCP product) to a different part of the company (or to a generally available GCP product), and don't trust that.
Google has always been worse at externally facing PR than at the internal reality, even when I worked there (2011-2015). Major company weakness.
But the internal reality inside GCP, perceptions aside, is pretty good even now.
Customers won't pay money in the first place to use a service if it may vanish out from under them? I expect a cloud service provider not to offer a service unless they think it is going to be profitable, and I expect them to continue to offer it even if it turns out not to be profitable, because otherwise I will take my business to a cloud service provider that will give me that guarantee.
This is the subtle, but important, difference between SaaS and PaaS/IaaS. Services are to use. Platforms are built upon. Flickr is a service. If they shutdown I'll get another one. If they shutdown I'll just move to another. GCP is a platform, if they shutdown I have to re-architect the entire thing from scratch.
If it's costing them money they haven't figured out a model, yet, that works in their favour.
> If they ever do deprecate something people have built on though they're gonna get absolutely crucified.
They do this all the time, and they get crucified every time. I built a Google Hangout App and a Chrome App, both of which were platforms eventually shut down.
This is where the meme came from, and it's why I personally stopped building on top of Google products. A 1-year deprecation policy is no assurance to me if I plan for my app to live longer than that.
Their approach to things like GCP is very different than their approach to those other areas of Google. But they don't separate their branding or unify their deprecation attitudes enough to avoid cross-org-chart reputation damage like this.
So basically they screw you over on either medium or hard, and according to you the real problem is them not telling us clearly whether or not we receive the medium or hard screw over option.
By the way that GCP is so full of loopholes where Google can get out of its obligations its laughable. So it's not even that clear cut that the GCP is really a better alternative.
And even when it turns out to be legally sound, when stuff like this happens, who's going to sue google over it? Nobody, and they know it.
Oh, courts routinely give binding weight to words like Google's deprecation policy uses, and any large megacorp who is sufficiently badly impacted by a legalese violation (though SLA issues and deprecation issues are two separate things) wouldn't be scared away from a lawsuit by Google being big. I can imagine EU regulatory action or a class action lawsuit as other possible mechanisms.
But as I say in another comment, the contract is less important than both trust and reality. Keep in mind nobody focuses on how AWS doesn't even have a public deprecation policy.
I'm right there with many people in this thread in agreeing that Google has a trust problem, due mostly to real perception issues stemming from Google's habits outside GCP, which can and do impact people's perceptions of what they'll do with GCP.
The reality of what Google has done and will do with GCP, though, is pretty good. Sure they do sometimes deprecate things in ways Amazon never would. But not nearly as often or as abruptly as they do on the consumer side - that would be commercial suicide - and they do other things better than Amazon. Tradeoffs.
> The reality of what Google has done and will do with GCP, though, is pretty good.
No. It's just words. Actions speak louder than words. Googles' actions in the last couple of days spoke pretty loud. No amount of words will change that.
I haven't worked for Google since 2015, and I never worked for their PR department. I was just a rank-and-file engineer (and a rank-and-file tech lead for one small team near the end of my time there). If I worked for Google PR, my comments throughout this thread would have far less criticism of the company's messaging and branding than they do. :)
I'm still a fan of GCP as a suite of products and services, as much as I recognize many of Google's organizational failings and disagree with plenty of their product decisions in other areas of Google.
Google (including GCP) has been bad at external communication as long as I've paid attention, and that includes external communications around incidents. What actions are you referring to, beyond poor and confusing communication (i.e. words) around what is or isn't broken or fixed at what points during the incident? That's most of the problem I'm aware of from this incident.
With that said, part of the reason people notice GCP's outages more than AWS's is that GCP publicly notes their outages way more than AWS does. In other words, among the outages that either cloud has, Google much more often creates an incident on their public status page and Amazon much more often fails to.
My "reality of [...] GCP" comment was about the bigger picture of the cloud platform offering, not any one specific incident.
> To me that looks like a reasonable deprecation policy.
It might be, until they jack up the prices 15X with limited notice (looking at you, Google maps [1]). No deprecation needed, just force users off the platform unless they're willing to pay a massive premium.
Google Maps has never been subjected to that policy, unlike GCP services. These org chart divisions are real but only clear to Googlers, Xooglers (I'm in this category), and people who pay extremely close attention.
The fact that they're all Google makes reputation damage bleed across meaningfully different parts of what's in truth now a conglomerate under the umbrella name Google.
Except all the Google maps setup and API keys are generated from the gcp UI and the billing happens on the cloud platform as well. While maps didn't start as a gcp product, they seem to have rolled it in to gcp fully.
Not fully. Really what happened is they did a re-org that gave them Google Cloud as an umbrella brand including GCP, Google Maps Platform (this new version of Google Maps as a commercial service), Chrome, Android, G Suite...
The bit of Maps Platform integration for management of the billing and API layer was called out in the announcement blog as an integration with the console specifically, and the docs and other branding around Maps Platform remain distinct from GCP still in excessively subtle ways that Googlers pay more attention to than everyone else, like hosting the docs on developers.google.com instead of cloud.google.com and having Platform in its name separately from Cloud Platform.
This stuff makes sense to Googlers not only because of the org chart but also because Google has a pretty unified API layer technology and because Google put in a lot of work to unify billing tech & management. Reusing that is efficient but not always clear.
But you're right to be confused. Their branding is a mess and always has been. This is the same company that thought Google Play Books makes sense as a product name.
Google's product / PR / comms / exec people are very bad at understanding how external people who don't know Google's org chart and internal tech will perceive these things, or at least bad at prioritizing those concerns.
They live and breathe their corporate internals too much to realize this. Some Google engineers and tech writers realize the confusion but pick other battles to fight instead (like making good quality products).
As for what products are actually part of GCP, it's the parts of this page that aren't an external partner's brand name, aren't called out separately like G Suite or Cloud Identity or Cloud Search, and aren't purely open source projects like Knative and Istio (as opposed to the productized versions within GCP), with the caveat that the level so far of integration into GCP of Google acquisitions like Apigee, Firebase, and Stackdriver varies depending on per-company specifics: https://cloud.google.com/products/
G Suite and Cloud Identity accounts can be used with GCP, just like any other Google accounts. They are part of Google Cloud but not Google Cloud Platform.
Hope I waded through the mess correctly for you. :)
Let's not forget that Google can change the terms as they please with a 90 Days notice as per Section 7.1 Modifications of the terms. So any promise that is longer than 90 days, even without a escape hatch like Section 7.2, would be legally weak and subject to change at any time without much recourse.
I think it’s telling of Google’s culture that the corporate arm felt the need to formalize this in law. I won’t pretend to know what it’s telling. Just suggest that you listen for yourself. Look at rule of law versus the ideas of liberty if you’d like a stronger nudge.
AWS' Customer Agreement[1] essentially has the same language. I wouldn't be surprised to see similar language from other cloud providers as well. Seems rather prudent on their part.
I would suggest the inference I was alluding to would also apply amazon.
Note how I never stated the inference. This is because I wanted to share a way of thinking without feeling the responsibility to reply to people attempting to force me to prove some prescriptive, arbitrary inference rule by exhaustion. I do not participate in such practices casually. I also consider it rude to subject people to such practices without consent. I also believe it is a practice that kills online discussion platforms. See this community’s thought provoking guidelines :)
> Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.
Fair point but still seems odd that the people providing updates took the weekend off during a large scale customer impacting issue. I'm sure all the people spending the weekend trying to mitigate the impact of this on their infrastructure would love to have timely updates.
> Let's hope you don't have a life threatening medical emergency that can't wait near an affected healthcare facility while that silly software is down.
If your ability to operate an ER is dependent on a remote data center, you have no business being a public health provider.
This is a problem in the web ui creating additional node pools, there is a very simple workaround to use gcloud. What major impact are you referring to?
It's likely the fix is checked in and will start roll out on Monday.
Disclaimer: I work on Google Cloud and while I believe we could use more words here, this doesn't seem like a huge problem. It's embarrassing the the issue with the ui was shipped, and I'm sure this will be addressed in the post mortem as well as whether it could have been mitigated quicker than a roll forward.
>This is a problem in the web ui creating additional node pools, there is a very simple workaround to use gcloud. What major impact are you referring to?
Based on comments in this thread even gcloud is failing and so are other non-kubernetes services. Which may be inaccurate but there's a lot of people saying the same thing so maybe it is.
You're right however that the linked issue is only about the UI. So Google isn't even tracking the service distribution issue in it's issue tracker much less updating people on. I personally think that's even worse...
It's not just the UI though. Resources have been exhausted, or that's the error I'm getting, in a lot of regions for both my work and personal accounts (GKE and GCE). Someone on the GCP slack also said they were getting similar issues from Dataflow so it seems to be widespread across products.
Understandable but in my experience the incident manager assigned is still supposed to keep track of progress during weekends when you have a major incident.
And this is likely a major incident with significant customer impact.
The way google is handling all this gives a pretty poor impression. Seems like this kubernetes is just a PoC.
Yeah why not? A billion dollar cloud provider can't have one person communicating to customers that are facing multi day outage. That not updating is an option is absurd in that time range.
At a smaller, less corporate company the engineer/public comms person dividing line would not be so ossified the divide couldn't be bridged when the situation called for it.
I think the broader point here is that they (whomever they is) don't think the situation calls for it. That's why they said the next update would be on Monday on Friday.
You may feel that's a bad decision, but I doubt that people are in a panic because they can't push out an update that would not be noticably different from the last one.
>Just to clarify, what should this update contain?
"We're working on it", possibly an ETA for a fix or some details. Technically it's fluff but people are not machines and the update is for people. We like the feeling that people are working on a fix, that people care and that the end is in sight. It makes the situation less stressful and, as for why Google should care, less stressed engineers won't bad mouth Google as much after the fact.
I think this comes down to the two outages being not at all similar. An s3 outage affects practically every Amazon customer. This affects a relatively small number of go customers.
And I'm confused about what was good about that response. That article is about how the s3 outage caused so many issues that Amazon couldn't update their status dashboard to inform users at all.
My working assumption is that 18006 should have closed out 18005. But now it sounds like there's a different issue, which we're working to get to the bottom of.
A generic question: Our company is completely dependent on AWS. Sure we have taken all of the standard precautions for redundancy, but what happened here could just as easily happen with AWS - a needed resource is down globally.
What would a small business do as a contingency plan?
I don't think OP was intending "minimal" to mean it would be easy to get to the stage where it's possible, just that once you've got all your infrastructure-as-code stuff set up correctly, you ought to be able to just be pressing buttons / running scripts and have your infrastructure up and running in another cloud provider.
Even when working in small companies with small infrastructure, I've kept recreation of infrastructure as one of my high priorities (one reason it really bugged me in one job to have to depend on Oracle Databases that I couldn't automate to the same degree.)
In my mind, it's not different from the importance of having, and testing restoration of, backups. If your infrastructure gets compromised somehow, or you find yourself up the creek with your provider, you've got to be able to rebuild everything from scratch.
The infrastructure has always been the easy part, as long as the company is willing to pay for multiple datacenters.
Then you realize a lot of software and databases can only run from a single instance, zero support for multi regions, and you're not gonna to rewrite everything and resiliency just can't happen.
Totally impractical for any small business and of questionable usefulness for large ones. You'd be giving up the largest benefit of platforms like AWS--ready to use services for common tasks--to avoid the infinitesimally small chance of AWS having some doomsday global outage.
To answer the original question: It looks like this issue was just a UI bug that affected the console, the service itself wasn't impacted. Events that do impact the service will be contained to a region, meaning you can mitigate it with proper redundancy across regions, no zany multi-cloud solution required.
That’s not how Terraform works. Each provisioner has separate syntax depending on the cloud provider. The template for AWS wouldn’t work on GCP or Azure.
Assume I’m not fully informed here. What does “written properly” mean? Sure I can move over route53 to cloud DNS easily but Firehose to PubSub? Lambda to Cloud Functions? DynamoDB to BigTable and moving the data?
The syntax for provisioning these doesn’t work that well for some find and replace to work. Are you using a templater to generate cloud-specific HCL from a template or something? Sounds like a pretty big problem to solve to me and not just something where you can win via discipline.
You’re always “locked in” to your infrastructure. People have this brilliant strategy of thinking they have this brilliant strategy of avoiding “lock in” and again they get the worse of both worlds. They end up spending more to avoid lock-in and they aren’t taking advantage of what thier provider has to offer.
At the end of the day, changing your underlying infrastructure is so risky and usually not worth the cost benefit analysis, it’s rarely done.
If all you are doing with your cloud provider are a bunch of VMs, you’re wasting money. There are much cheaper and simpler ways just to host a bunch of VMs than to use one of the major cloud providers.
Are you not using any of thier managed services and are you maintaining your own on VMs? If so, you have the worse of both worlds. You’re spending more on hosting and you’re not saving money on letting someone else do the “undifferentiated heavy lifting”.
Going to the cloud thinking you are going to save money is silly. It is more expensive to run things in the cloud, what you are paying for is the ability to scale up and down without a logistics problem.
It’s only silly if you’re not changing your processes. If you’re keeping the same people doing everything manually instead of automating, or honestly if you’re keeping the same headcount and if you’re not developing and depending on services provided by your provider and instead you’re self hosting services on VMs, of course you’re going to spend more.
But I blame most of the cost overruns when using cloud providers on “consultants” who think they are “moving to the cloud” when all they really know is how to setup a little networking infrastructure and know nothing about how to use the developer, Devops, or other hosted solutions.
Most ”consultants” I’ve run across only know how to do a lift and shift and do a one to one mapping of the on prem VMs and networking infrastructure to the cloud. They know nothing about automation, transforming Devops practices, or transforming development practices and architecture.
Assuming you have architected multi-region I'm not sure how realistic that scenario is. AWS regions are mostly standalone, I have seen services go down in a region but never globally.
Roll the dice? What are the consequences for you, especially if you can shift the blame?
What are the odds of you having better uptime rolling your own tooling?
Can you afford the complexity of multi cloud?
Is the added complication worth it?
Basically, you have three options. You can go full milti cloud, with all the expenses and overhead that entails. You can have everything run in one place but have a plan to switch to a backup system. Or you can look at the overhead and cost associated with those, and decide that it's not worth it. If the business can handle the costs and risks, then any of them can be a valid option.
Very little, unless that small business has a big enough budget to design something that spans multiple clouds.
Ultimately it’s a risk/return decision.
“Is going exclusively with AWS/azure/GCP etc a better decision in reliability, financial and mantainability terms than complicating the design to improve resiliency? And will this more complex solution actually improve reliability?”
If a multi region, multi service meltdown happens on AWS, it will feel like most of the internet has gone down to a lot of users. Being such a catastrophic failure, I bet the service will be restored pretty fast, not in 3 days.
You could go multi cloud though. But when half of the internet struggles to work correctly, I’d not feel too bad about my small business’ downtime.
I'm not sure where you get this idea, but AWS has definitely had global outages. 1.5 years ago there were massive global issues with S3, causing even their own status dashboard to malfunction.
edit: I stand corrected. Apparently the S3 outage wasn't global, though its effects were.
Meanwhile, this outage has only really been noticeable to ops teams, since it doesn't affect existing nodes or anything outside GKE. It's definitely concerning and the fix is taking far too long, but as far as global outages go the impact is relatively minor.
On February 28th, 2017, S3 had issues in us-east-1, not globally. It just happens that most customers create their buckets in us-east-1. (It’s the default bucket creation region.)
Buckets creation/deletion is a global operation, namespaces are all global for it. When us-east-1 goes down, so does bucket creation. All other operations proceed merrily, but yeah as we see over and over again, whenever us-east-1 has an outage everything seems to shit the bed because they've built everything out in us-east-1. Hopefully us-east-2 is starting to eat in to that.
> If a multi region, multi service meltdown happens on AWS, it will feel like most of the internet has gone down to a lot of users. Being such a catastrophic failure, I bet the service will be restored pretty fast, not in 3 days.
Additionally, from a "nobody ever got fired for buying IBM" perspective, you're unlikely to catch much blame from your users for going down when everyone else was down too.
Multi cloud is almost always more pain than gain. You’d spend time and effort abstracting away the value that a cloud provider brings in canned services.
Hell, multi region is often more than many workloads need.
It could work then. We had the same setup. Also, we store everything on S3 but there is a lambda function that pushes new objects to Google Cloud storage too. We figured, storage is cheap so we can duplicate things and it will make moving to GCP a lot easier.
This was a while back though. Now we depend on a lot more AWS stuff.
Using Kubernetes is a good start. It should be easy to migrate your server between EKS and GKE. However data is trickier to move around. So you won’t be immune from all global outages.
First, determine your tolerance. How much does downtime per min cost you? How long can you be down? A lot of the time it may be cheaper to apologize to your customers than build a truly reliable system.
Then start looking at points of failure and sort them based on severity and probability. Is your own software deployment going to generate more downtime per year than a regional aws outage?
There are formal academic ways to determine what your overall availability is, but don't have those on hand. Suffice to say, it takes significant research, planning, execution, and testing to ensure a target availability. (See Netflix https://medium.com/netflix-techblog/the-netflix-simian-army-... ) if someone says they have 99.9% or better up time, they had better have proof in my mind (or a fat SLA violation payout)
People outsource to cloud providers not because they are cheap, but because managing infra in house is hard. Also move fast and break things.
Read AWS docs about availability, there are availability zones in a region, spread across those to minimize impact. Then test when something goes down. Fix/repeat.
People outsource to cloud providers because building / hiring / maintaining a team of decent engineers that provide a baseline industry bar of SLAs, SLOs is much more expensive than the eye watering costs of most cloud providers at even a IaaS level. Opex is tough.
Most companies I’ve been at don’t offer multi region support for their services because it’s too expensive for the service provided even in so-called “price insensitive” enterprises (you can’t just make up a price that’s huge, they do have budgets still) and most of their customers are unwilling / unable to pay more for the extra availability. If your software is designed from the start better, multi region failovers should be fairly inexpensive though. But all the bolted on “multi region” software I’ve seen has been hideously expensive and oftentimes less reliable due to the design being soundly not able to tolerate failures well.
Any stats / surveys on this? Most smaller (< 50 employees) b2c enterprise-style SaaS companies I'm anecdotally familiar with barely have enough funds to hire ops engineers let alone outsource to MSPs (even if the MSP practices labor rate arbitrage). A lot of the criteria I'd wager may be conditions around pre-revenue status and funding conditions moreso than headcount. I'm trying to understand just how biased my own experiences and indirect experiences are with the wider US market and appreciate data.
Small companies put every developer on call and here comes the support coverage.
Of course, that doesn't make them knowledgeable to run stable infrastructure and they will move on as soon as they realize they are being abused to work overnight and week end.
I am developer and I have the knowledge to run stable infrastructure at least on a small company SAAS scale. But, I wouldn’t go near a company that expected developers to be on call for netops work.
A company that doesn’t want the overhead of an MSP which in my experience is less than the cost of a full time Dev is not a company I’m going to work for. It would tell me a lot about thier mentality.
I don’t know about pre revenue companies, but, if you have to be secure and compliant with regulations from day one, you’re not going to take a chance with having a bunch of devs setting up your infrastructure. My bias probably comes from working mostly with companies in highly regulated fields.
Besides, I’m assuming that the cost savings a small company can get from being billed under a much larger organization account would make up for it. That and having cheap shared netops support.
That does bring up an interesting point. In hindsight, we already have duplicate infrastructure - a dev account and a production account. Why in the world was it decided to put both accounts in the same region?
The separate account was setup partially on my insistence but it was set up in the same region.
There are some significant differences region to region in AWS. Different numbers of availability zones, different latencies based on datacenter location, different types and availabilities of EC2 instances, etc. I think it makes sense to develop in the same region that your production service runs in just so you don't shoot yourself in the foot by deploying something that runs fine in your development region but doesn't run as well in your production region.
In most of the cases I've seen so far with multiple regions / POP (4 different companies of different sizes and verticals) the software never being deployed in another reason is the fundamental reason. Reasons ranged from "we hard-coded us-east-1 everywhere and don't know how to test region independence it turns out" to "we have 3 weeks to try to make our software distributed between different regions but have nowhere near the resources to test it well let alone develop anything new for it." Automation is completely remedial and non-repeatable in most of these companies that rarely provision clean environments from scratch. By keeping regions the same between different environments, you reduce the number of possible differences.
Some services in AWS are also not available in others (I'm quite familiar with AWS Data Pipeline not being available outside the "core" regions like us-east-1, eu-west-1) and having services in one region make usage of resources in another region is a huge change when most developers outside ones with technology literate customers are under the gun to push features out fast over sound design. The matrix of services and configurations necessary to mix and match regions and availability zones is non-trivial if you make extensive usage of AWS services above the IAAS layer.
Also, cross-region VPC peering has a TON of limitations that rather annoying depending upon how well your network has been architected (by default in most companies outside enterprises with a deep bench of network engineers, this would be rated at "complete crap barely better than a typical home wifi network"). Heck, even though I'm non-dumb at networks I have to keep reminding myself of various cross-region VPC limitations when working with refactoring cross-region VPCs like where you can reference security groups, how to propagate Route 53 records, etc.
> What would a small business do as a contingency plan?
Depends on your definition of small. If it's small enough not to have a dedicated infrastructure team designing multicloud solution, then the contingency plan may be: switch DNS to a static site saying "we're down until AWS fixes the issue, check back later".
Otherwise it depends on your specific scenario, your support contracts, and lots of other things. You need to decide what matters, how much the mitigation costs vs downtime, and go from there.
We invested early in being multi-region on GCP as well as multi-cloud with AWS as a fully redundant option if it ever became necessary to fail over to them.
It sure did. It took us about a minute to fully fail over to AWS, and we are running there now for the rest of the day. We will promote GCP to be the primary cloud again tonight.
Without our multi-cloud set up we would have been down for over an hour. In our business this is not an option.
You need to have a secondary location for backups, not AWS. Stores copie of customer data, orders, accounts, balances and anything that is critical to the business.
If AWS ever screws up, you will be able to continue running the business even if it might take weeks to start over.
For live redundancy, you should have a secondary datacenter on another provider, but realistically it's hard to do and most business never achieve that. Instead, just stick with AWS and if there is a problem the strategy is to sip coffee while waiting for them to resolve it. Much better this way than you having to fix it yourself.
Unpopular opinion: very little perhaps. You have to make sure all your your 3rd party dependencies have the same contingency plan as you do, but I guess it is going to be difficult to even figure that out...
Seems to be some weird underlying issue going on at GCP at the moment. Had cloud build webhooks returning a 500 error. Noticed we were at 255 images and deleting some fixed the issue. Created a P2 ticket about the issue before we managed to solve it and haven't had a response in 40+ hours.
The timeline of this disruption matches when we started experiencing cloud build errors.
Oh man must be a tough time to be an SRE at google cloud. But... they’re Google. They have been doing internal cloud for years and years. Borg — which K8s is a reimplementation if — has been the heart of Google for so long now you’d think they’d be able to architect their systems to have no outages whatsoever. I mean nobody is perfect but this looks bad.
Hi - I work at Google on GKE - sorry about the problems you're experiencing. There's a lot of people inside Google looking into this right now!
It looks like the UI issue was actually fixed, and that we just didn't update the status dashboard correctly. But we're double checking that and looking into some of the additional things you all have reported here.
I appreciate all the effort you're putting in and I understand such situations can be stressful but user's having to depend on someone responding on hacker news for status updates seems really amateur for an organization the size of google.
As much as I love bashing big corps I see HN as a supplementary communication channel for products like GCP - its a luxury we get to access alongside normal customer support channels in the GCP console, twitter, etc.
Critical service is failing, minimal information about why, but we should be so happy someone says a few sentences on here? For all of the engineering elitism coming out of google, Amazon is way more on their game across a number of products.
Let me put it this way. HackerNews, or in fact, any news outlets are not official. Customers should be getting emails from Google and be informed on its official webpage to explain what's going on. You don't want your neighbor to tell you you owe taxes. You want the government to send a notice to you.
>really amateur for an organization the size of google.
There is a reason while Google have been having hard time making inroads in the enterprise cloud. Kind of impedance mismatch between enterprise and the Google style. That 2 stories like high "We heart API" sign on the Google Enterprise building facing 237 just screams about it :)
Strangely and sadly with gmail account blocking and other such issues HN and Twitter is often better way to get Google's support than to contact support.
The status dashboard is inaccurate and/or a lie. It only tells about the GKE incident, while in fact the problem also impacts Google Compute Engine users. I was unable to create any google compute instance today, not even a basic 1vcpu, on NA and Europe-west.
As another comment pointed out, what's the point of having so many zones and redundancy around the globe if such global failure can still happen? I thought the "cloud" was supposed to make this kind of failure impossible
Thanks! I'm running Skylake 96 core instances but I haven't given up to try the 64 core instances for comparison yet. If I get another restart, I'll do a 96 vs 64 to try to narrow down the cause. Most likely, of course, this is a software issue on my end, not Google's.
This is unfortunately the norm. Like when AWS S3 went down (but couldn't update its own status images because they're in S3 and we all laughed) and along with it went Alexa, lambda, and every other service dependent on S3.
S3 is really the one of the few services on aws that can do that unfortunately. It has no concept of zone/region, it's truly global. To me it seems like a serious design flaw, as everything else in aws is striped by region, but not sure why exactly it was built like that.
edit:
nvm s3 has regions, it's the bucket names that are global.
Buckets are globally addressable because they planned for each S3 bucket + object key to have an associated URL (actually several), and URLs are a global namespace.
Given the historical context (S3 launched 12 years ago, 5 months before EC2 launched with the us-east-1 region), it's reasonable that S3 buckets were global because regions didn't really exist yet as a concept.
If you look at the docs now[1], new buckets are regionalized and the region is in the URL for non-us-east-1 regions.
Got it, makes sense. So the only way you'd have a global bucket is if you created it before a certain date (whenever they were regional-ized, I assume). Thanks!
I’ll suggest considering whether entities enamored with centralizing ideals are more likely to fail to properly realize the robustness of a distributed system.
> I thought the "cloud" was supposed to make this kind of failure impossible
You have to remember that you're trying to have access to backend platforms and infrastructure at all times, which almost no public utility does (assuming "the cloud" is "public utility computing"). Power plants go into partial shutdown, water treatment plants stop processing, etc. Utilities are only designed to provide constant reliability for the last mile.
If there's a problem with your power company, they can redirect power from another part of the grid to service customers. But some part of your power company is just... down. Luckily you have no need to operate on all parts of the grid at all times, so you don't notice it's down. But failure will still happen.
Your main concern should be the reliability of the last mile. Getting away from managing infrastructure yourself is the first step in that equation. AppEngine and FaaS should be the only computing resources you use, and only object storage and databases for managing data. This will get you closer to public utility-like computing.
But there's no way to get truly reliable computing today. We would all need to use edge computing, and that means leaning heavily on ISPs and content provider networks. Every cloud computing provider is looking into this right now, but considering who actually owns the last mile, I don't think we're going to see edge computing "take over" for at least a decade.
Thanks for jumping in here on your own time. The following question is not meant to be hostile, it is merely curiosity. Isn’t this supposed to be the kind of thing that monitoring and diagnostics software should find automatically? Serious question, not meant to embarrass you.
So, given that i filed this months ago via official support and it’s still not fixed, can you look into misleading container memory reporting ui bug. It reports memory_total but should be working_set
As of this morning, I am still unable to reliably start my docker+machine autoscaling instances. In all cases the error is "Error: The zone <my project> does not have enough resources available to fulfill the request"
An instance in us-central1-a has refused to start since last Thursday or Friday.
I created a new instance in us-west2-c, which worked briefly but began to fail midday Friday, and kept failing through the weekend.
On Saturday I created yet another clone in northamerica-northeast1-b. That worked Saturday and Sunday, but this morning, it is failing to start. Fortunately my us-west2-c instance has begun to work again, but I'm having doubts about continuing to use GCE as we scale up.
And yet, the status page says all services are available.
Why do you guys suffer global outages? This is your 2nd major global outage in less than 5 years. I’m sorry to say this, but it is the equivalent of going bankrupt from a trust perspective. I need to see some blog posts about how you guys are rethinking whatever design can lead to this - twice - or you are never getting a cent of money under my control. You have the most feature rich cloud (particularly your networking products), but down time like this is unacceptable.
I agree, in general, outages are almost inevitable, but global outages shouldn't occur. It suggests at least a couple of things:
1) Bad software deployments, without proper validation. A message elsewhere in this post on HN suggest that problems have been occurring for at least 5 days, which makes me think this is the most likely situation. If this is the case, presumably given this is multiple days in to the issue, rolling back isn't an option. That doesn't say good things about their testing or deployment stories, and possibly their monitoring of the product? Even if the deployment validation processes failed to catch it, you'd really hope alarming would have caught it.
or:
2) Regions aren't isolated from each other. Cross-region dependencies are bad, for all sorts of obvious reasons.
That shouldn't, but they do. S3 goes down [1]. The AWS global console goes down, right after Prime Day outages [2]. Lots of Google Cloud services go down [3, current thread]. Tens of Azure services go down hard [4].
Are software development and release processes improving to mitigate these outages? We don't know. You have to trust the marketing. Will regions ever be fully isolated? We don't know. Will AWS IAM and console ever not be global services? We don't know.
Blah blah blah "We'll do better in the future". Right. Sure. Some service credits will get handed out and everyone will forget until the next outage.
Disclaimer: Not a software engineer, but have worked in ops most of my career. You will have downtime, I assure you. It is unavoidable, even at global scale. You will never abstract and silo everything per region.
When I was at google, the big outages were almost always bad routing to the service - it was never that the service couldn't handle the load, and bad service instances were kind of hidden. my service did have some problems on new releases, but because we had multiple instances we could just redirect traffic to the instances we hadn't updated, so they stayed up.
The major issue is that outages are global instead of regional, effectively making it impossible to design around using the typical region/zone redundancy.
The pitch from cloud vendors always includes the idea that the cloud is more reliable than any in-house shop can achieve. So the expectation is set by the vendors.
Because they sell themselves as being far more reliable than internal IT. If they weren't selling on uptime, people probably wouldn't be quite so critical of downtime.
Let me know the next time you hear about the CIO of a fortune 500 asking his technology practitioners to validate what he read in Gartner and heard from Diane Greene.
My advice would be to find opportunities to get paid to tell people the right answer, not to implement the wrong answer against your better judgement. Hot job market right now, more jobs than talent, all that jazz.
If you're stuck implementing a suboptimal solution, that's not your fault, and not the intent of my above comment.
2 outage in 5 years sounds pretty low, to be honest.
Disclaimer: google employee in ads, who worked on many many fires throughout the years, but talking from my personal perspective and not from my employer. I am sure we are striving to have 0, but realistically, i have seen many that says things happen. Learn, and improve.
The issue people have with it is that it's global, not regional, indicating that there are dependencies in the entire architecture that people does not expect to be there.
It is a natural effect of building massive yet flat homogeneous systems, failures tend to be greatly amplified.
Most of what you can read of Google's approach will teach you their ideal computing environment is a single planetary resource, pushing any natural segmentation and partitioning out of view.
You're right in terms of breadth officially covered. But if you look at the features where they both officially have support, there are many examples where the GCP version is more reliable and usable than the AWS version. Even GKE is an example of this, despite the outage in node pool creation that we're discussing here. Way better than EKS.
(Disclosure: I worked for Google, including GCP, for a few years ending in 2015. I don't work or speak for them now and have no inside info on this outage.)
I think you're going to have to back up a claim like this with some facts.
GKE being the exception, since it was launched a couple years before EKS. AWS clearly has way more services, and the features are way deeper than GCP.
Just compare virtual machines and managed databases, AWS has about 2-3x more types of VMs (VMs with more than 4TB of RAM, FPGAs, AMD Epyc, etc.), and in databases, more than just MySQL and PostgreSQL. When you start looking at features you get features that you just can't get in GCP, like 16 read-replicas, point in time recovery, backtrack, etc.
Disclaimer: I work for AWS but my opinions are my own.
Each platform has features the other platform doesn't, even though AWS has more.
Some of GCP's unique compelling features include live VM migration that makes it less relevant when a host has to reboot, the new life that has recently been put into Google App Engine (both flexible environment and the second generation standard environment runtimes), the global load balancer with a single IP and no pre-warming, and Cloud Spanner.
In terms of feature coverage breadth I started my previous comment by agreeing that AWS was ahead, and I still reaffirm that.
But if you randomly select a feature that they both have to a level which purports to meet a given customer requirement, the GCP offering will frequently have advantages over the AWS equivalent.
Examples besides GKE: BigQuery is better regarded than Amazon Redshift, with less maintenance hassle. And EC2 instance, disk, and network performance is way more variable than GCE which generally delivers what it promises.
One bit of praise for AWS: when Amazon does document something, the doc is easier to find and understand, and one is less likely to find something out of date in a way that doesn't work. But GCP is more likely to have documented the thing in the first place, especially in the case of system-imposed limits.
To be clear, I want there to be three or four competitive and widely used cloud options. I just think GCP is now often the best of the major players in the cases where its scope meets customer needs.
Redshift is not a direct competitor with BigQuery. It's a relational data warehouse. BigQuery more directly competes with Athena, which is a managed version of Apache Presto, and my personal opinion is that Athena is way better than BigQuery because I can query data that is in S3 (object storage) without having to import it into BigQuery first.
Disk and network performance is extremely consistent with AWS so long as you use newer instance types and storage types. You can't reasonably compare the old EBS magnetic storage to the newer general purpose SSD and provisioned IOPS volume types, and likewise, newer instances get consistent non-blocking 25gbps network performance.
I'm not so sure I would praise our documentation; it is one of the areas that I wish we were better at. Some of the less used services and features don't have excellent documentation, and in some cases you really have to figure it out on your own.
GCP is a pretty nice system overall, but most of the time when I see comparisons, when GCP looks better its because the person making the comparison is comparing the AWS they remember from 5-6 years ago with the GCP of today, which would be like comparing GAE from 2012 with today.
The comments I made about Redshift vs BigQuery and about disk/network/etc reflect current opinions of colleagues who use AWS currently (or recently in some cases) and extensively, not 5-6 year old opinions. Even my own last use of AWS was maybe 2-3 years ago, when Redshift was AWS's closest competitor to BigQuery and when I saw disk/network issues directly.
You're right that Athena seems like the current competitor to BigQuery. This is one of those things that are easy to overlook when people made the comparison as recently as a couple of years ago (before Athena was introduced) and Redshift vs BigQuery is still often the comparison people make. This is where Amazon's branding is confusing to the customer: so many similar but slightly different product niches, filled at different times by entirely different products with entirely unrelated names.
When adding features, GCP would usually fill adjacent niches like "serverless Redshift" by adding a serverless mode to Redshift, or something like that, and behavior would be mostly similar. Harder to overlook and less risky to try.
Meanwhile, when Athena was introduced, people who had compared Redshift and BigQuery and ruled out the former as too much hassle said "ah, GCP made Amazon introduce a serverless Redshift. But it's built on totally different technology. I wonder if it will be one of the good AWS products instead of the bad ones." (Yes, bad ones exist. Amazon WorkMail is under the AWS umbrella but basically ignored, to give one example.)
And then they go back to the rest of their day, since moving products (whether from Redshift or BigQuery) to Athena would not be worth the transition cost, and forget about Athena entirely.
On the disk/network question, no I didn't see performance problems with provisioned IOPS volume types, but that doesn't matter: for GCE's equivalent of EBS magnetic storage, they do indeed give what they promise, at way less cost than their premium disk types. There's no reason it isn't a fair comparison.
And for the "instance" part of my EC2 performance comment, I was referring to a noisy neighbor problem where sometimes a newly created instance would have much worse CPU performance than promised and so sometimes delete and recreate was the solution. GCE does a much better job at ensuring the promised CPUs.
I'm glad AWS and GCP have lots of features, improve all the time, and copy each other when warranted. But I don't think the general thrust of my comparison has gone invalid, even if my recent data is more skewed toward GCP and my AWS data is skewed toward 2-3 years old. Only the specifics have changed (and the feature gap narrowed with respect to important features).
Yeah. Perhaps feature rich was an overstatement. I meant that when GCP does do a product it works like I’d expect it to work and has the features I need. Not always the case with a AWS, particularly around ELBs and VPCs.
Not to minimize here (well, yes, a little), but this was a UI-only outage, from what I can tell. You could still create the pools from the command-line. It doesn't seem unreasonable to have a single, global UI server, as long as the API gateway is distributed and not subject to global outages.
Global here refers to the geographical spread of the service, GKE in this case, measured in regions, not the number of services.
Edit: I saw your point a bit late. It was limited to GKE, which makes my initial comment about "service" incorrect, and it was global, which keeps my comment about "region" correct. On a related note, an SRE from GKE posted on Slack that GCE was out of resources and so GKE faced resource exhaustion as well [1][2] - so it _might_ have been a multi-service outage.
Google has a global SDN (software-defined network) that gives them some unique and beneficial capabilities, like being able to onboard traffic in the closest CDN POP and letting it ride over the Google backbone to the region your systems are running in.
The problem is that running a global SDN like this means if you do something wrong, you can have outages that impact multiple regions simultaneously.
This is why AWS has strict regional isolation and will never create cross-region dependencies (outside of some truly global services like IAM and Route 53 that have sufficient redundancy that they should (hopefully) never go down).
Disclaimer: I work for AWS, but my opinions are my own.
I have a question. At what point does k8s make sense?
I have a feeling that a microservice architecture is overkill for 99% of businesses. You can serve a lot of customers on a single node with the hardware available today. Often times, sharding on customers is rather trivial as well.
There's a huge range between monolith and microservice approach, and even a monolith will have dependent services. A simple web stack these days might include nginx, a database, a caching layer, some sort of task broker and then the 'monolith' web app itself. All of that can be sanely managed in k8s.
Right... IMO monolith is better understood as a reference to the data model than deployment topology. If you only have a single source of truth, then your application is naturally going to trend towards doing most of its business logic in one place. This still doesn't displace the need for other services like caching, async tasks, etc., that you identify.
I definitely wouldn’t be managing my own database or caching layer without a very good reason. I would use a managed service if I were using a cloud provider.
K8s is nice even without microservices. Yeah you don't get nearly the benefits you would in a microservice architecture, but I consider it a control plane for the infrastructure, with an active ecosystem and focus on ergonomics. If you have a really simple infrastructure, you will still need to script spinning up the VMs, setting up the load balancing, etc. but K8s gives you a homogenous layer upon which to put your containers. It's not too much of an overkill, especially with a hosted K8s from e.g. Google, AWS, and soon Digital Ocean and Scaleway.
Things like throwing another node into the cluster, or rolling updates are free, which you would otherwise need to develop yourself. All of that is totally doable, of course, but I like being able to lean on tooling that is not custom, when possible.
When your infrastructure does need to become more complicated, you're already ready for it. Even if I were only serving a single language, starting with a K8s stack makes a lot of sense, to me, from a tooling perspective. Yeah normal VMs might be simpler, conceptually, but I don't consider K8s terribly complicated from a user perspective, when you're staying around the lanes they intend you to stay in. Part of this may also be my having worked with pretty poor ops teams in the past, but I think K8s gives you a really good baseline that gives pretty good defaults about a lot of your infrastructure, without a lot of investment on your part.
That said, if you're managing it on a bare metal server, then VMs may be much easier for you. K8s The Hard Way and similar guides go into how that would work, but managing high availability etcd servers and the like is a bit outside my comfort zone. YMMV.
As someone whose daily work happens on k8s, I'd say you better be paining a lot before you move to k8s. I take great care to avoid this, but if you aren't careful, you can end up "feeling" productive on k8s without actually being productive. K8s gives a lot of room for one to tweak workflows, discuss deployment strategies, security, "best practices", etc. And you can get things done reasonably fast. But that's like a developer working all day on fine tuning their editor and comparing and writing plugins and claiming that they are getting productive.
The key issue here is that k8s was written with very large goals in mind. That a small business can easily spin it up quickly and run a few microservices or even a monolith + some workers is just coincidental. It is NOT the design goal. And the result of that is that a lot of the tooling and writing around k8s reflects that. A lot of the advice around practices like observability and service meshes comes from people who've worked in the top 1% (or less) of companies in terms of computing complexity. What I'm personally seeing is that this advice is starting to trickle down into the mainstream as gospel. Which strangely makes sense. No one else has the ability to preach with such assurance because not many people in small companies have actually been in the scenarios of the big guns. The only problem is that it's gospel without considering context.
So at what point does k8s make sense? Only when you have answers to the following:
* Getting started is easy, maintaining and keeping up with the going ons is a full time job - Do you have at least 1 engineer at least that you can spare to work on maintaining k8s as their primary job? It doesn't mean full time. But if they have to drop everything else to go work on k8s and investigate strange I/O performance issues, are you ready to allow that?
* The k8s eco system is like the JS framework ecosystem right now - There are no set ways of doing anything. You want to do CICD? Should you use helm charts? Helm charts inherited from a chart folder? Or are you fine using the PATCH API/kubectl patch commands to upgrade deployments. Who's going to maintain the pipeline? Who's going to write the custom code for your github deployments or your brigade scripts or your custom in house tool? Who's going to think about securing this stuff and the UX around it. That's just CICD mind you. We aren't anywhere close to the weeds about deciding if you want to use ingresses vs Load balancers and how you are going to run into service provider limits on certain resources. Are you ready to have at minimum 1 developer working on this stuff and taking time to talk to the team about it?
* Speaking about the team, k8s and Docker in general is a shift in thinking - This might sound surprising but the fact that Jessie Frazelle (y'all should all follow her btw) is occasionally seen reiterating the point that containers are NOT VM's is a decent indicator that people don't understand k8s or Docker at a conceptual level. When you adopt k8s, you are going to pass that complexity to your developers at some point. Either that or your dev ops team takes on that full complexity and that's a fair amount to abstract away from the developers which will likely increase the work load of devops and/or their team size. Are you prepared for either path?
* Oh also, what do your development environments start to look like? This is partly related to microservices but are you dockerizing your applications to work on the local dev environment? Who's responsible for that transition? As much as one tries to resist it, once you are on k8s you'll want to take advantage of it. Someone will build a small thing as a microservice or a worker that the monolith or other services depend on. How are you going to set that up locally? And again, who's going to help the devs accumulate that knowledge while they are busy trying to build the product. (Please ...
Very nicely written. While not a direct response to OP, you articulated some great points on k8s. k8s will naturally succeed as the future of data center orchestration as VM's give way to containers. But it is questionable if everyone needs it.
I fully agree with you, and personally have taken the path of using Docker Swarm as a step-up to k8s, as it was so much easier to get along with.
I would certainly recommend this to smaller businesses.
>The k8s eco system is like the JS framework ecosystem right now - There are no set ways of doing anything.
Given how both the JS and devops worlds seems to be progressing, is there any reason to believe that this will change before the next thing comes and K8S becomes a ghost town?
I hate the word Microservice, so I'm just going to use the word Service.
Most monoliths software companies build aren't actually monoliths, conceptually. Let's say you integrate with the Facebook API to pull some user data. Facebook is, within the conceptual model of your application, a service. Hell, you even have to worry "a little bit" about maintaining it; provisioning and rotating API keys, possibly paying for it, keeping up to date on deprecations, writing code to wire it up, worrying about network faults and uptime... That sounds like a service to me; we're three steps short of a true in-house service, as you don't have to worry about writing its code and actually running it, but conceptually its strikingly similar.
Facebook is a bad example here. Let's talk Authentication. Its a natural "first demonolithized service" that many companies will reach to build. Auth0, Okta, etc will sell you a SaaS product, or you can build your own with many freely available libraries. Conceptually they fill the same role in your application.
Let's say you use Postgres. That's pretty much a service in your application. A-ha; that's a cool monolith you've got there, already communicating over a network ain't it. Got a redis cache? Elasticsearch? Nginx proxy? Load balancer? Central logging and monitoring? Uh oh, this isn't really looking like a monolith anymore is it? You wanted it to be a monolith, but you've already got a few networked services. Whoops.
"Service-oriented" isn't first-and-foremost a way of building your application. It's a way of thinking about your architecture. It means things like decoupling, gracefully handling network failures, scaling out instead of up, etc. All of these concepts apply whether you're building a dozen services or you're buying a dozen services.
Monolithic architectures are old news because of this recognition; no one builds monoliths anymore. It's arguable if anyone ever did, truly. We all depend on networked services, many that other people provide. The sooner you think in terms of networked services, the sooner your application will be more reliable and offer a superior experience to customers.
And then, it's a natural step to building some in-house. I am staunchly in the camp of "'monolith' first, with the intention of going into services" because it forces you to start thinking about these big networking problems early. You can't avoid it.
We had an issue a few weeks ago where the google front-end servers were mangling responses from Pub/Sub and returning 502 responses, making the service completely unusable and knocking over a number of things we have running in production. Despite paying for enterprise support and having in a P1 ticket, we had to spend Friday to Sunday gathering evidence to prove to the support staff that there was indeed a problem, because their monitoring wasn't detecting it. Right now I'm doing something similar (and since Friday!) but for TLS issues they're having. Again, because their support reps don't believe there's a problem. There are so many more problems than they ever show on their status page...
I was so mad to read that until you said /sarcasm :p
That being said I really do think there is a difference between who is working at google today and the google we all fell in love with pre-2008.
I am sure there are a amazing people still working at google, but nowhere near like it was.
The way I like to think about google is that some amazing people mad ea awesome train that builds tracks in front of it -- you can call them gods maybe -- but those people are gone -- or a least the critical mass required to build such a train has dwindled to just dust. What we have left is a awesome train full of people pulling the many levers left behind.
To make things even worse my last interview as a SRE left me wondering if even the people who are there know this as well, and they are actually working hard to keep out those who might expose light on to this. I don't say that because I did not get the job -- I am actually happy I did not get extended a offer.
I say this with one exception, the old-timer who was my last interview. I could tell he was dripping in knowledge and eager to share it with any that would listen. I came out of his 45 min session learning many things -- I wold actually pay to work with a guy like that.
I would also like to point out that the work ethic was not what I expected. I was told that when on call, my duty was to figure out the root cause was in the segment I was responsible for. I don't know about you, but if my phone rings at night I am going to see through to a resolution and understand the problem in full -- even if it is not on the segment that I was assigned.
The work ethic is in tact. It is not fair to load people with stress and ask them to drop everything. You're conflating poor resource allocation with "work ethic". Burning the midnight oil when it can be avoided is not work ethic. The correct way is to load balance outage resolution.
I really don't know how to reply to you as you have set up a bunch of windmills you assumed from my previous post. Who said anything about poor resource allocation? Who said we need to load people with stress?
That being said -- when you are on call -- dropping everything is exactly what is expected.
But limiting responsibilities for on call employees is a way of limiting the workload they have during that time - ultimately benefiting the employees. They are on call, not working. I don't see the windmills here.
Many many AWS people have left citing on call as the worst part of their job. Also, it is really a far fetched allegation that interviewers try to fail interviewees to hide their own incompetence. I get that you may not think much of Google SREs but to allege that is just in bad spirit. I hope you do get to see that the people inside Google are one of the best perks of working here. They are smart and motivated and willing to help each other.
Even though you have troubleshooting skills and other skills that can help Google, Google assumes that these skills are derivative of what they are looking for in interview. So, they are looking for those primary skills.
You can't find and hire geniuses for every component. Systems should scale with the average engineer in mind (so should code). We have all smelt the coffee and it smells even better when your team is efficient and well rested.
I don't disagree. Building around 10x'ers in this way creates god complexes and unhappy "senior" engineers (since they don't do any of the cool work). Having one very early still juices your productivity
I totally heard that when trying to get engineer attention to YouTube Premium’s frequent “download errors” due to their transcoding-on-the-fly (or something). I was telling a support rep I had evidence that if I switched the setting from standard to high def (or vice versa) that the error would go away, but I could reproduce it with the same video, and thought it was a CDN/transcode issue. They kept marking the ticket as “unable to reproduce” and I had to wonder — as a paying customer, don’t they have analytics on my phone that would show exactly the request I was making which was failing in their logs? And if they saw it succeed, why not tell me the problem was my ISP? I’d have been happy to follow up... but nothing’s ever wrong in Google-land. :/
Many of the tier 1 GCP support reps work for external vendors nowadays, which is probably part of the problem.
During my time on the GCE team (note I don't work at Google now) I knew multiple full-time Google employee support reps, including some still at the company. They have the good attitude and deep knowledge you'd hope for.
The problem is simply about how Google scales their GCP support org. To be completely clear, AWS support is by and large not great either.
If you're a big or strategically important customer, of course, you can get a good response from either company.
In general GCP has quota limits so its expected of customers to catch 5xx and do exponential back off. But this info is not explicitly stated.
From my personal experience - i think all big cloud providers first two level support staff is no good if it isn't an obvious dumb one on your part. I always prefer to forgo support and try to go through every bit of their documentation to figure it out on our own. This helps to save huge amount of time. But if you have developer support - it can help to expedite things little faster though.
There is no magic public clouds have incredibly complex control planes and marketing fluff aside you would very likely experience much better uptime at singe top tier DC than @ a cloud provider.
I am currently evaluating GCP for two separate projects. I want to see if I understand this correctly:
1) For three whole days, it was questionable whether or not a user would be able to launch a node pool (according to the official blog statement). It was also questionable whether a user would be able to launch a simple compute instance (according to statements here on HN).
2) This issue was global in scope, affecting all of Google's regions. Therefore, in consideration of item 1 above, it was questionable/unpredictable whether or not a user could launch a node pool or even a simple node anywhere in GCP at all.
3) The sum total of information about this incident can be found as a few one or two sentence blurbs on Google's blog. No explanation nor outline of scope for affected regions and services has been provided.
4) Some users here are reporting that other GCP services not mentioned by Google's blog are experiencing problems.
5) Some users here are reporting that they have received no response from GCP support, even over a time span of 40+ hours since the support request was submitted.
6) Google says they'll provide some information when the next business day rolls around, roughly 4 days after the start of the problem.
I really do want to make sure I'm understanding this situation. Please do correct me if I got something wrong in this summary.
Right now we don't know. It's one of two possibilities from what I can tell:
a) Google had a global service disruption that impacted Kubernetes node pool creation and possible other services since Friday. They had a largely separate issue for a web UI disruption (what this thread links to) which they forgot to close on Friday. They still have not provided any issue tracker for the service distribution and it's possibly they only learned about it from this hacker news thread.
b) People are having various unrelated issues with services that they're mis-attributing to a global service disruption.
This is why GCP has no hope of ever taking significant market share from AWS. Google thinks they can treat their cloud customers like they treat users of their free services. Customer support and communication are essential.
Yes. When the AWS status page failed to accurately inform their customers for several hours, AWS used Twitter to ensure that there was communication with their customers.
“2) This issue was global in scope, affecting all of Google's regions. Therefore, in consideration of item 1 above, it was questionable/unpredictable whether or not a user could launch a node pool or even a simple node anywhere in GCP at all.”
Ok. So on aws we were* paying for putting systems across regions, but, honestly I don’t get the point. When an entire region is down what I have noticed is that all things are fucked globally on aws. Feel free to pay double - but it seems* if you are paying that much just pay for an additional cloud provider. Looks like it’s the same deal on GCP.
Just grabbed first article. Example: In this case capitalone went down. I don’t work at capitalone - but I imagine they had their data copied across every region 30 times.
On 17 October, there was a multi-AZ network failure at us-east-1. It only lasted 3m35s, but it was enough that our customers were calling about our site being down.
I can't comment regarding GKE as we don't use that particular service, however we are very heavy users of many other GCP services, including Compute, Datastore, BigQuery, Pub/Sub, Storage, Functions, Speech, and others. Zero issues this weekend, everything is running 100% as any normal day.
I've been failing all weekend to create nodes in a GKE cluster through either the UI console or gcloud. Even right now I can't get any nodes to spin up.
Edit: I finally got my cluster up and running by removing all nodes, letting it process for a few minutes, then adding new nodes.
You are right about the Google blog content itself not indicating three days of outage. Turns out they just forgot to mark that particular issue as resolved on Friday, as you point out. This is my mistake. I would update my comment to reflect this, but it doesn't seem to allow an edit at this point.
The items I put down in my comment are based largely on user reports, though (there isn't much else to go on). And I mean these items as questions (i.e. "is this accurate?"). Folks here on HN have definitely been reporting ongoing problems and seem to be suggesting that they are not resolved and are actually larger in scope than the Google blog post addressed.
Someone from Google commented here a few hours ago indicating Google was looking into it. And other folks here are reporting that they don't have the same problems. So it's kind of an open question what's going on.
I'm in the evaluation phase too. And I've found a lot to like about GCP. I'm hoping the problems are understandable.
As of this morning, I am still unable to reliably start my docker+machine autoscaling instances. In all cases the error is "Error: The zone <my project> does not have enough resources available to fulfill the request"
An instance in us-central1-a has refused to start since last Thursday or Friday.
I created a new instance in us-west2-c, which worked briefly but began to fail midday Friday, and kept failing through the weekend.
On Saturday I created yet another clone in northamerica-northeast1-b. That worked Saturday and Sunday, but this morning, it is failing to start. Fortunately my us-west2-c instance has begun to work again, but I'm having doubts about continuing to use GCE as we scale up.
And yet, the status page says all services are available.
We are GCP customers for the last couple of years. We use other cloud platforms(AWS, IBM, Oracle, OrionVM) too. We don't use GKE but use rancher/kubernetes combo on their standard platform.
So far GCP is the best, hands down in terms of stability. We never had a single outage or maintenance downtime notification till now. We are power users but our monitoring didn't pick any anomaly so i don't think this issue had rampant impact on other services.
But i find it concerning that they provided very little update on what went wrong. I also think its better to expect nil support out of any big cloud provider if you don't have paid support. Funny how all these big cloud providers think you are not eligible for support de-facto. Sigh.
I don't understand why someone would choose to deploy anything mission critical without having an support contract with the ISP, the manufacturer of the the software etc.
Simple, the cost of an outage is less than the cost of a support contract. Very few things are really mission critical as in they can never go down. Rather they simply have a cost to going down and you can choose to pay that one way or another.
I transitioned from collocation to self managed remote server farm and then onto self managed remote vms. All these providers provided de-facto support whether we opted for one or not. You can go to their portal and raise a ticket.
I am not saying with vast numbers its feasible but big cloud providers don't even give you the opportunity to raise a ticket if its their fault. There is a price you pay extra when you opt for any one of them but many don't realize. Having said that - almost all the time, our skilled expertise is better than their initial two level of support staff. We realized it early so we handle it better by going over the documentation and making our code resilient since all cloud platforms have some limit or another since overselling in a region is something they can't avoid. Going multiple regions across when you handle these exceptions is the only way through.
I use AWS free tier and get customer support through email, but thats not the case with GCP. Do they provide free email support?
If you are an early stage startup can you afford their 200/Month support, when your entire GCP bill is under $1. However, that doesn't mean you don't have to support them.
No, I understand that most hobby projects are just hobbies and Google/Amazon/etc are under no obligation to provide support for hobbies that are literally a net cost to them.
I'm glad AWS's free tier is working for you, but complaining that Google doesn't want to give you free capacity for your business and then also provide you free support for that business is pretty absurd.
You know there's 1-man "start ups"/companies out there that serve more users than 99.5% of VC startups ever will, which earn their owner a very livable wage, but who still can't afford a 500 bucks business support plan on all of the 20 services they use.
If you've got VC money to blow so you can pretend your SaaS toy can feed 500 people while having money left to throw at things, that's cool. Just remember that other people might be running sustainable businesses.
There are literally millions of small 1-2 people business that make less than 10k in profits.
I mean sure, they could go and probably afford to waste $200 extra on something random that will be useless to them most of the time, but that money is going straight out of their paycheck.
You don't remain profitable though by repeatedly making bad decisions like that. Which was my point.
Running a (small) profitable business is about making the right decisions consistently, and if you're likely to waste money on one thing, you're also likely to waste it on the 19 other similar things.
Maybe speak to literally anyone you know who is running a small businesses if you want to know more. Yes that includes your local small stores on your street.
At the end of the day you probably pissed off quite a few people on here when you called their livelihood a hobby project.
The scenario being discussed was building a company on top of GCP and being unable to afford $200/month for support costs. Tell me what “mom and pop” shops are pulling in $10k/month from GCP-based software but are unwilling/unable to pay $200/month for the very thing that underpins their livelihoods?
This is akin to saying that a mom and pop laundromat can’t afford insurance, or shouldn’t because they won’t frequently need it.
You’re trying to equate small businesses with hobbies. You’ve now resorted to straw men, slippery slopes, and false equivalency. Maybe consider that if you have to distort the situation this much to make your point, you might just be wrong.
> At the end of the day you probably pissed off quite a few people on here when you called their livelihood a hobby project.
I didn’t say anything about anyone’s livelihood. You’re the one pretending that small businesses bringing home $120k/year can’t afford a $200 monthly support bill.
I bet the guy who started this thread about GCP’s support cost has made a sum total of <$1000 from his “startup”. Likely <$10. Hobby.
I don’t care if “quite a few people” got pissed about my comment. People with egos that delicate shouldn’t use social media.
> The scenario being discussed was building a company on top of GCP and being unable to afford $200/month for support costs. Tell me what “mom and pop” shops are pulling in $10k/month from GCP-based software but are unwilling/unable to pay $200/month for the very thing that underpins their livelihoods?
I was trying to tell you that most small businesses can't go around spending hundreds of bucks of things that provide little value, whether that's a business support plans on services they use or something else. It's true regardless of whether you're a brick and mortar store or some online service.
> This is akin to saying that a mom and pop laundromat can’t afford insurance, or shouldn’t because they won’t frequently need it.
Speaking about about false equivalencies...
> You’re the one pretending that small businesses bringing home $120k/year can’t afford a $200 monthly support bill.
First off, I spoke of businesses making generally less than that.
Also (I already said this, good job ignoring that!) paying $200 bucks on a single useless thing is survivable for even a small business - but you know what's better than only making one bad business decision? Making no bad ones at all. Making too many will quickly break the camel's back.
Which was my whole argument and it's also what people generally refer to when they say they can't afford something.
For instance you may say "I can't afford to go to this restaurant", even though you'd have enough money to do it without going immediately bankrupt. But it'd be a bad decision, too many of which quickly add up.
> I was trying to tell you that most small businesses can't go around spending hundreds of bucks of things that provide little value
And I'm telling you that if you built your business on top of GCP, a support contract is probably not "low value". You'd happily pay $200 for support on your critical infrastructure, just as you'd happily pay $200 for a repairman to fix your washing machine if you owned a laundromat.
If you don't need support, then sure, don't pay for the plan. If you do need support, $200 seems pretty reasonable.
> Speaking about about false equivalencies...
Signing up for a monthly recurring support plan in case you need it is literally insuring your business.
> For instance you may say "I can't afford to go to this restaurant", even though you'd have enough money to do it without going immediately bankrupt. But it'd be a bad decision, too many of which quickly add up.
A support plan for your critical infrastructure probably isn't "useless". Which is the point. If your need for support is that low, then either you've built your own redundant systems to protect you or more likely you aren't running a real business.
We had an issue a few weeks back where all nodes in west1-a could not pull docker images. Google support was pinballing P1 issue around the globe and across multiple teams for a few days untill I root caused it for them - turned out to be gce service account issues affecting entire zone. 2 days to rollback (no status page update). I know nobody gives a fuck but can’t help but feel vindicated as an ex google sre.
I think a lot of people give a fuck here; I do, at least. Thanks for outlining it, these things are fascinating (to me anyway, who has never worked in IT/ops).
> For three whole days, it was questionable whether or not a user would be able to launch a node pool (according to the official blog statement)
What blog statement are you referring to? I don't see any such statement. Can you provide a link?
The OP incident status issue says "We are investigating an issue with Google Kubernetes Engine node pool creation through Cloud Console UI". It also says "Affected customers can use gcloud command in order to create new Node Pools."
So it sounds like a web interface problem, not a severely limiting, backend systems problem with global scope.
Also, the report says "The issue with Google Kubernetes Engine Node Pool creation through the Cloud Console UI had been resolved as of Friday, 2018-11-09 14:30 US/Pacific". So the whole issue lasted about 10 hours, not three whole days.
> Some users here are reporting that other GCP services not mentioned by Google's blog are experiencing problems
When everything works, GCP is the best. Stable, fast, simple, reliable.
When things stop working, GCP is the worst. Slow communications and they require way too much work before escalating issues or attempting to find a solution.
They already have the tools and access so most issues should take minutes for them to gather diagnostics, but instead they keep sending tickets back for "more info", inevitably followed by a hand-off to another team in a different time zone. We have spent days trying to convince them there was an issue before, which just seems unacceptable.
I can understand support costs but there should be a test (with all vendors) where I can officially certify that I know what I'm talking about and don't need to go through the "prove its actually a problem" phase every time.
As someone who works for Government and Enterprise - all I care about sometimes is how a company behaves when everything goes wrong.
The issue with outages for the Government organizations I have dealt with is rarely the outage itself - but strong communication about what is occurring and realistic approximate ETAs, or options around mitigation.
Being able to tell the Directors/Senior managers that issues have been "escalated" and providing regular updates are critical.
If all I could say was a "support ticket" was logged, and we are waiting on a reply (hours later) - I guarantee the conversation after the outage is going to be about moving to another solution provider with strong SLAs.
Very similar thing at our office. Considering the scale of which we run things, any outage could be a potential loss of millions _every minute_.
Sure, we use support tickets with vendors for small things. Console button bugging out, etc. But for large incidents, every vendor has a representative within an hour driving distance and will be called into a room with our engineers to fix the problem. This kind of outage, with zero communication, means the dropping of a contract.
Communication is critical for trust, especially if we're running a business off it.
Going single cloud on that scale is simply irresponsible though.
You need failovers to different providers and hopefully also have your hardware for general workloads
And suddenly the CEO doesn't care anymore if one of your potential failovers is behaving flaky in specific circumstances
Not saying it's good as it is.. communication as a saas provider is - as you said- one is the most important things... But this specific issue was not as bad as some people insinuate in this thread
You are incorrect about aws. If your pay for business support, and something is happening to your production environment, they are on a call with you in less than an hour.
With Dell you can certify with them so you can get replacement parts and such without the BS back and forth with some guy in india. Saves everyone time and money.
We actually got to a point where we had a couple of spare parts onsite (sticks of RAM, HD, etc) and so we repair immediately and then request the replacement. This was on a large HPC cluster so we had almost daily failures of some kind (most commonly we'd get a stick of RAM that would fail ECC checks repeatedly).
"Support costs" calculation often doesn't include the costs of not having support.
When I worked at GoDaddy, there were around 2/3 of the company was customer support.
At the current company I'm at, a cryptocurrency exchange, our support agents frequently hear they prefer our service over others because of our fast support response times (crypto exchanges are notorious for really poor support).
All of my interactions with Amazon support have been resolved to my satisfaction within 10 minutes or less.
Companies really ought to do the math on the value that comes from providing fast, timely, and easy (don't have to fight with them) customer support.
They have though; they've just drawn the conclusion that they'd rather put massive amounts of effort in to building services that users can use without needing support. This approach works well once the problems have been ironed out, but it's horrible until that's the case. Google's mature products like Ads, Docs, GMail, etc are amazing. Their new products ... aren't.
I've seen Google projects where the project lead explained (actually responded) that they don't want to provide support, end of story. Google puts folks in charge but does not give them enough in the way accountability objectives.
> instead they keep sending tickets back for "more info"
Isn't that the case with basically every support request, no matter the company or severity? The first couple of emails from 1st & even 2nd level support are mostly about answering the same questions about the environment over and over again. We've had this ping-pong situation with production outages (which we eventually analysed and worked around by ourselves) and fairly small issues like requesting more information of an undocumented behavior which didn't even effect us much. No matter how important or urgent the initial issue was, eventually most requests end up being closed unresolved.
I've definitely had interactions with smaller companies where you can effectively bypass first and second line by demonstrating you know what you're doing, mostly just saying the right things for them to accept that you've done basic troubleshooting steps already and really do need to talk to someone beyond that point.
Yes, same experience here, support at smaller companies can be more dedicated when talking to "knowledgeable" customers. It's generally easier to get to their 3rd level, sometimes just because there is no 1st or 2nd level at all. But at "big" enterprises - not so much.
> I've definitely had interactions with smaller companies where you can effectively bypass first and second line by demonstrating you know what you're doing, mostly just saying the right things for them to accept that you've done basic troubleshooting steps already and really do need to talk to someone beyond that point.
Smaller companies or personalized support structures (like named engineers) is very different. You can build up a relationship and usually bypass many questions to get to main issue, and even get it resolved before you can even open a case at larger organizations.
GCP does have role-based support models with a flat-rate plan, which is really great, but the overall quality of the responses leaves much to be desired.
I recently removed my hosting from GCP. The pricing is confusing and unbelievable. Their customer service is a joke. I don't trust Google for longterm consistency due to the way they shut their own apps but I let that slide as I doubt they will do that on their cloud services. I have experience with AWS (rock solid, world class support but also costly), digital ocean (improving fast), heroku (good for beginners but also expensive and not as full featured as AWS) and finally Hetzner (too early to judge).
Cloud providers have all of the potential in the world to make each region truly isolated. I shouldn't have to architect my application to be multi-cloud, at least for stability reasons.
Yet, somehow every major cloud provider experiences global outages.
That old AWS S3 outage in us-east-1 was an interesting one; when it went down, many services which rely on S3 also went down, in other regions beside us-east-1 because they were using us-east-1 buckets. I have a feeling this is more common than you'd think; globally-redundant services which rely on some single point of geographical failure for some small part.
AWS regions are very much isolated from each other.
We know because we are still waiting here in ap-southeast-2 for services such as EKS to be made available. Pretty sure that any reliance within their backend services on us-east-1 was just a temporary bug and nothing systemic.
418 comments
[ 4.3 ms ] story [ 283 ms ] threadError message when creating a new Cluster:
Deploy error: Not all instances running in IGM after 35m7.509000994s. Expect 1. Current errors: [ZONE_RESOURCE_POOL_EXHAUSTED]: Instance 'gke-cluster-3-pool-1-41b0abf8-73d7' creation failed: The zone 'projects/url-shortner-218503/zones/us-west2-b' does not have enough resources available to fulfill the request. Try a different zone, or try again later. - ; .
Perhaps some of the issues are localized? Perhaps it's even user error (it happens, you know?). But because a small amount of HN users say "it's everywhere!" then suddenly people reach for their pitchforks.
Sometimes we just don't have all the information.
I did this in the australia-southeast1-a zone.
The specific issue appears to be about creating new "node pools". Creating standard VMs in GCP works fine however, so this is specific to GKE and their internal tooling that integrates with the rest of GCP.
GKE doesn't (at least to my knowledge) allow you to create VMs separately and join them to the cluster in any kind of easy fashion.
Right when I convinced our project to get migrated from AWS...
- Someone at Google right now, probably.
(I work at Google, on GKE, though I am not a lawyer and thus don't work on the deprecation policy)
b. To the Agreement:
Google may make changes to this Agreement, including pricing (and any linked documents) from time to time. .... Google will provide at least 90 days’ advance notice for materially adverse changes to any SLAs by either: (i) sending an email to Customer’s primary point of contact; (ii) posting a notice in the Admin Console; or (iii) posting a notice to the applicable SLA webpage. If Customer does not agree to the revised Agreement, please stop using the Services. Google will post any modification to this Agreement to the Terms URL.
for any reason
at any time
Caveat emptor, folks.
Which is the deprecation policy. (I mean I share your frustration with Google's what-appears-to-be-at-least haphazard policy of shutting down services instead of trying to gain traction. But, let's not misrepresent what they say).
I don't think it's wrong - they can deprecate any service they want to do whenever they want, unless people have paid for and signed a contract that says otherwise which I guess people aren't doing.
But the policy doesn't really guarantee anything at all does it, due to the reference escape-hatches? It might as well not exist?
"Subject to the deprecation policy [which says that Google will give at least 1 year notice before cancelling services], Google may discontinue..."
In other words, at any time, google can give you a years notice.
(I work at Google, but am not a lawyer and this isn't official in any capacity).
Please don't selectively quote things out of context to give a misleading impression.
> commercially reasonable
> substantial economic or material technical burden
Is one engineer working on an old service to keep it alive commercially reasonable or a substantial burden? I don't know. Do you?
In practice this policy lets them shut off anything they want any time they want. Again it's their playground they can do what they want unless they signed a contract saying they'd do something else for you so I don't have a problem with it.
To be clear, that policy is a contract. And those things would be decided by a jury. And if my understanding is correct, the reasonable person standard applies. So you can answer this yourself, do you think a reasonable person would believe that your interpretation is valid?
If not, why mention it?
Because it makes more people feel comfortable enough to use your services and pay you, without actually binding you towards any sort of behavior that would cost you money. There's a direct financial incentive here to use legalese to give the semblance of reliability without having to deliver on it
> 7.1 Discontinuance of Services. Subject to Section 7.2, Google may discontinue > any Services or any portion or feature for any reason at any time without > liability to Customer.
Let's take a look at Section 7.2:
> 7.2 Deprecation Policy. Google will announce if it intends to discontinue or > make backwards incompatible changes to the Services specified at the URL in > the next sentence. Google will use commercially reasonable efforts to continue > to operate those Services versions and features identified at > https://cloud.google.com/terms/deprecation without these changes for at least > one year after that announcement, unless (as Google determines in its > reasonable good faith judgment): > > (i) required by law or third party relationship (including if there is a change > in applicable law or relationship), or > > (ii) doing so could create a security risk or substantial economic or material > technical burden. > > The above policy is the "Deprecation Policy."
To me that looks like a reasonable deprecation policy.
If they ever do deprecate something people have built on though they're gonna get absolutely crucified. That's probably better protection than any terms of service.
If a service Google runs is losing money, what reason would they have to not shut it down?
If I pay you for a service that would take time to migrate off of, and you are making money off me now, I am going to be ripshit if you decide to just turn it off because it's suddenly not making money for you in the short term. Google's done this a lot, and the fact that don't provide concrete time lines in their contract gives even less reason to trust them
People look at AWS's track record, and trust that. People look at Google's track record, overlook what to an inside-the-company Googler perspective are dramatically significant organizational boundaries or product lifecycle definitions that are very poorly communicated outside the company, mentally apply reputational damage from one part of Google (or from a preview-stage GCP product) to a different part of the company (or to a generally available GCP product), and don't trust that.
Google has always been worse at externally facing PR than at the internal reality, even when I worked there (2011-2015). Major company weakness.
But the internal reality inside GCP, perceptions aside, is pretty good even now.
If it's costing them money they haven't figured out a model, yet, that works in their favour.
They do this all the time, and they get crucified every time. I built a Google Hangout App and a Chrome App, both of which were platforms eventually shut down.
This is where the meme came from, and it's why I personally stopped building on top of Google products. A 1-year deprecation policy is no assurance to me if I plan for my app to live longer than that.
By the way that GCP is so full of loopholes where Google can get out of its obligations its laughable. So it's not even that clear cut that the GCP is really a better alternative.
And even when it turns out to be legally sound, when stuff like this happens, who's going to sue google over it? Nobody, and they know it.
But as I say in another comment, the contract is less important than both trust and reality. Keep in mind nobody focuses on how AWS doesn't even have a public deprecation policy.
I'm right there with many people in this thread in agreeing that Google has a trust problem, due mostly to real perception issues stemming from Google's habits outside GCP, which can and do impact people's perceptions of what they'll do with GCP.
The reality of what Google has done and will do with GCP, though, is pretty good. Sure they do sometimes deprecate things in ways Amazon never would. But not nearly as often or as abruptly as they do on the consumer side - that would be commercial suicide - and they do other things better than Amazon. Tradeoffs.
No. It's just words. Actions speak louder than words. Googles' actions in the last couple of days spoke pretty loud. No amount of words will change that.
Are you working for Google PR or something?
I'm still a fan of GCP as a suite of products and services, as much as I recognize many of Google's organizational failings and disagree with plenty of their product decisions in other areas of Google.
Google (including GCP) has been bad at external communication as long as I've paid attention, and that includes external communications around incidents. What actions are you referring to, beyond poor and confusing communication (i.e. words) around what is or isn't broken or fixed at what points during the incident? That's most of the problem I'm aware of from this incident.
With that said, part of the reason people notice GCP's outages more than AWS's is that GCP publicly notes their outages way more than AWS does. In other words, among the outages that either cloud has, Google much more often creates an incident on their public status page and Amazon much more often fails to.
My "reality of [...] GCP" comment was about the bigger picture of the cloud platform offering, not any one specific incident.
It might be, until they jack up the prices 15X with limited notice (looking at you, Google maps [1]). No deprecation needed, just force users off the platform unless they're willing to pay a massive premium.
[1] https://www.google.com/search?q=google+maps+price+increase
The fact that they're all Google makes reputation damage bleed across meaningfully different parts of what's in truth now a conglomerate under the umbrella name Google.
The bit of Maps Platform integration for management of the billing and API layer was called out in the announcement blog as an integration with the console specifically, and the docs and other branding around Maps Platform remain distinct from GCP still in excessively subtle ways that Googlers pay more attention to than everyone else, like hosting the docs on developers.google.com instead of cloud.google.com and having Platform in its name separately from Cloud Platform.
This stuff makes sense to Googlers not only because of the org chart but also because Google has a pretty unified API layer technology and because Google put in a lot of work to unify billing tech & management. Reusing that is efficient but not always clear.
But you're right to be confused. Their branding is a mess and always has been. This is the same company that thought Google Play Books makes sense as a product name.
Google's product / PR / comms / exec people are very bad at understanding how external people who don't know Google's org chart and internal tech will perceive these things, or at least bad at prioritizing those concerns.
They live and breathe their corporate internals too much to realize this. Some Google engineers and tech writers realize the confusion but pick other battles to fight instead (like making good quality products).
They do at least document which services are subjected to the GCP Deprecation Policy (Maps is not there): https://cloud.google.com/terms/deprecation
As for what products are actually part of GCP, it's the parts of this page that aren't an external partner's brand name, aren't called out separately like G Suite or Cloud Identity or Cloud Search, and aren't purely open source projects like Knative and Istio (as opposed to the productized versions within GCP), with the caveat that the level so far of integration into GCP of Google acquisitions like Apigee, Firebase, and Stackdriver varies depending on per-company specifics: https://cloud.google.com/products/
G Suite and Cloud Identity accounts can be used with GCP, just like any other Google accounts. They are part of Google Cloud but not Google Cloud Platform.
Hope I waded through the mess correctly for you. :)
[1] https://aws.amazon.com/agreement/
Note how I never stated the inference. This is because I wanted to share a way of thinking without feeling the responsibility to reply to people attempting to force me to prove some prescriptive, arbitrary inference rule by exhaustion. I do not participate in such practices casually. I also consider it rude to subject people to such practices without consent. I also believe it is a practice that kills online discussion platforms. See this community’s thought provoking guidelines :)
> Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.
>We will provide more information by Monday, 2018-11-12 11:00 US/Pacific.
Wait, did the people tasked with fixing this just take the weekend off?
Am I the only one that finds this slightly humorous?
Let's hope you don't have a life threatening medical emergency that can't wait near an affected healthcare facility while that silly software is down.
https://idatalabs.com/tech/products/kubernetes
If your ability to operate an ER is dependent on a remote data center, you have no business being a public health provider.
It's likely the fix is checked in and will start roll out on Monday.
Disclaimer: I work on Google Cloud and while I believe we could use more words here, this doesn't seem like a huge problem. It's embarrassing the the issue with the ui was shipped, and I'm sure this will be addressed in the post mortem as well as whether it could have been mitigated quicker than a roll forward.
Based on comments in this thread even gcloud is failing and so are other non-kubernetes services. Which may be inaccurate but there's a lot of people saying the same thing so maybe it is.
You're right however that the linked issue is only about the UI. So Google isn't even tracking the service distribution issue in it's issue tracker much less updating people on. I personally think that's even worse...
https://landing.google.com/sre/sre-book/chapters/managing-in...
Looks like this time Mary took the whole week off without telling Josephine :)
And this is likely a major incident with significant customer impact.
The way google is handling all this gives a pretty poor impression. Seems like this kubernetes is just a PoC.
The person updating that status dashboard may or may not be an engineer, the IM certainly is.
That's yes, it's still being investigated?
You may feel that's a bad decision, but I doubt that people are in a panic because they can't push out an update that would not be noticably different from the last one.
Just to clarify, what should this update contain?
"We're working on it", possibly an ETA for a fix or some details. Technically it's fluff but people are not machines and the update is for people. We like the feeling that people are working on a fix, that people care and that the end is in sight. It makes the situation less stressful and, as for why Google should care, less stressed engineers won't bad mouth Google as much after the fact.
What's the better communication plan: detailed, hourly updates or terse, one-line blog posts scattered across several days?
And I'm confused about what was good about that response. That article is about how the s3 outage caused so many issues that Amazon couldn't update their status dashboard to inform users at all.
My working assumption is that 18006 should have closed out 18005. But now it sounds like there's a different issue, which we're working to get to the bottom of.
What would a small business do as a contingency plan?
Terraform using AMIs plus chef recipes that work in the cloud and bare metal. Dont use AWS specific services.
This would allow you to spin over to another cloud provider , vsphere or bare metal with minimal work
Even when working in small companies with small infrastructure, I've kept recreation of infrastructure as one of my high priorities (one reason it really bugged me in one job to have to depend on Oracle Databases that I couldn't automate to the same degree.)
In my mind, it's not different from the importance of having, and testing restoration of, backups. If your infrastructure gets compromised somehow, or you find yourself up the creek with your provider, you've got to be able to rebuild everything from scratch.
Then you realize a lot of software and databases can only run from a single instance, zero support for multi regions, and you're not gonna to rewrite everything and resiliency just can't happen.
To answer the original question: It looks like this issue was just a UI bug that affected the console, the service itself wasn't impacted. Events that do impact the service will be contained to a region, meaning you can mitigate it with proper redundancy across regions, no zany multi-cloud solution required.
Disaster recovery by switching to another provider is simple when minimal centos/rhel images are used.
The syntax for provisioning these doesn’t work that well for some find and replace to work. Are you using a templater to generate cloud-specific HCL from a template or something? Sounds like a pretty big problem to solve to me and not just something where you can win via discipline.
At the end of the day, changing your underlying infrastructure is so risky and usually not worth the cost benefit analysis, it’s rarely done.
That's a pretty manpower intensive way of operating. I think the fact that you get cloud agnostic this way is probably not worth it.
Are you not using any of thier managed services and are you maintaining your own on VMs? If so, you have the worse of both worlds. You’re spending more on hosting and you’re not saving money on letting someone else do the “undifferentiated heavy lifting”.
But I blame most of the cost overruns when using cloud providers on “consultants” who think they are “moving to the cloud” when all they really know is how to setup a little networking infrastructure and know nothing about how to use the developer, Devops, or other hosted solutions.
Most ”consultants” I’ve run across only know how to do a lift and shift and do a one to one mapping of the on prem VMs and networking infrastructure to the cloud. They know nothing about automation, transforming Devops practices, or transforming development practices and architecture.
A lift and shift should only be the first phase.
Thinking that is the reason for consultants is a problem
Note : I worked directly for companies as an employee when building these stacks
https://blog.thousandeyes.com/amazon-route-53-dns-and-bgp-hi...
Well not a lot. On the plus side its pretty reliable.
Ultimately it’s a risk/return decision.
“Is going exclusively with AWS/azure/GCP etc a better decision in reliability, financial and mantainability terms than complicating the design to improve resiliency? And will this more complex solution actually improve reliability?”
Going multi region on AWS should be safe enough.
If a multi region, multi service meltdown happens on AWS, it will feel like most of the internet has gone down to a lot of users. Being such a catastrophic failure, I bet the service will be restored pretty fast, not in 3 days.
You could go multi cloud though. But when half of the internet struggles to work correctly, I’d not feel too bad about my small business’ downtime.
edit: I stand corrected. Apparently the S3 outage wasn't global, though its effects were.
Meanwhile, this outage has only really been noticeable to ops teams, since it doesn't affect existing nodes or anything outside GKE. It's definitely concerning and the fix is taking far too long, but as far as global outages go the impact is relatively minor.
Additionally, from a "nobody ever got fired for buying IBM" perspective, you're unlikely to catch much blame from your users for going down when everyone else was down too.
Multi cloud is almost always more pain than gain. You’d spend time and effort abstracting away the value that a cloud provider brings in canned services.
Hell, multi region is often more than many workloads need.
Nope.
This was a while back though. Now we depend on a lot more AWS stuff.
Cluster administration and identity management are unique to each provider and fairly challenging to get right.
Then start looking at points of failure and sort them based on severity and probability. Is your own software deployment going to generate more downtime per year than a regional aws outage?
There are formal academic ways to determine what your overall availability is, but don't have those on hand. Suffice to say, it takes significant research, planning, execution, and testing to ensure a target availability. (See Netflix https://medium.com/netflix-techblog/the-netflix-simian-army-... ) if someone says they have 99.9% or better up time, they had better have proof in my mind (or a fat SLA violation payout)
People outsource to cloud providers not because they are cheap, but because managing infra in house is hard. Also move fast and break things.
Read AWS docs about availability, there are availability zones in a region, spread across those to minimize impact. Then test when something goes down. Fix/repeat.
Any trade comes to me if it's urgent, and I appear more professional as I've got a functioning system.
I might be an chancer running my entire system off an shoestring but being up when everyone else has taken a dive looks good.
Most companies I’ve been at don’t offer multi region support for their services because it’s too expensive for the service provided even in so-called “price insensitive” enterprises (you can’t just make up a price that’s huge, they do have budgets still) and most of their customers are unwilling / unable to pay more for the extra availability. If your software is designed from the start better, multi region failovers should be fairly inexpensive though. But all the bolted on “multi region” software I’ve seen has been hideously expensive and oftentimes less reliable due to the design being soundly not able to tolerate failures well.
Considering that even tech companies hardly manage to have a pair of DevOps or Sysadmin, running one own infrastructure is completely out of question.
Of course, that doesn't make them knowledgeable to run stable infrastructure and they will move on as soon as they realize they are being abused to work overnight and week end.
A company that doesn’t want the overhead of an MSP which in my experience is less than the cost of a full time Dev is not a company I’m going to work for. It would tell me a lot about thier mentality.
Besides, I’m assuming that the cost savings a small company can get from being billed under a much larger organization account would make up for it. That and having cheap shared netops support.
2 at UTC 0-8
1 at UTC 4-12
2 at UTC 8-16
1 at UTC 12-20
2 at UTC 16-24
1 at UTC 20-04
(Repeat)
The separate account was setup partially on my insistence but it was set up in the same region.
If needed, we could have done VPC peerings across regions. (https://aws.amazon.com/about-aws/whats-new/2017/11/announcin...)
Some services in AWS are also not available in others (I'm quite familiar with AWS Data Pipeline not being available outside the "core" regions like us-east-1, eu-west-1) and having services in one region make usage of resources in another region is a huge change when most developers outside ones with technology literate customers are under the gun to push features out fast over sound design. The matrix of services and configurations necessary to mix and match regions and availability zones is non-trivial if you make extensive usage of AWS services above the IAAS layer.
Also, cross-region VPC peering has a TON of limitations that rather annoying depending upon how well your network has been architected (by default in most companies outside enterprises with a deep bench of network engineers, this would be rated at "complete crap barely better than a typical home wifi network"). Heck, even though I'm non-dumb at networks I have to keep reminding myself of various cross-region VPC limitations when working with refactoring cross-region VPCs like where you can reference security groups, how to propagate Route 53 records, etc.
Depends on your definition of small. If it's small enough not to have a dedicated infrastructure team designing multicloud solution, then the contingency plan may be: switch DNS to a static site saying "we're down until AWS fixes the issue, check back later".
Otherwise it depends on your specific scenario, your support contracts, and lots of other things. You need to decide what matters, how much the mitigation costs vs downtime, and go from there.
Without our multi-cloud set up we would have been down for over an hour. In our business this is not an option.
If AWS ever screws up, you will be able to continue running the business even if it might take weeks to start over.
For live redundancy, you should have a secondary datacenter on another provider, but realistically it's hard to do and most business never achieve that. Instead, just stick with AWS and if there is a problem the strategy is to sip coffee while waiting for them to resolve it. Much better this way than you having to fix it yourself.
I wish I was only being tongue-in-cheek.
The timeline of this disruption matches when we started experiencing cloud build errors.
It looks like the UI issue was actually fixed, and that we just didn't update the status dashboard correctly. But we're double checking that and looking into some of the additional things you all have reported here.
People who respond here could be employees of Google, caring about it and respond here because they know it.
What he can mention ( a lot of people are working on it) is what you can suspect when something is going down. All other cloud providers do the same.
edit: The default is also only about the UI issue and there's no issue tracker for the broader non-UI disruptions going on since Friday.
There is a reason while Google have been having hard time making inroads in the enterprise cloud. Kind of impedance mismatch between enterprise and the Google style. That 2 stories like high "We heart API" sign on the Google Enterprise building facing 237 just screams about it :)
As another comment pointed out, what's the point of having so many zones and redundancy around the globe if such global failure can still happen? I thought the "cloud" was supposed to make this kind of failure impossible
I've been creating GCP instances in us-central1-a and us-central1-c today without issue. Which zone were you using in NA?
I have been noticing unusual restarts, but I haven't been able to pin down the cause yet (may be my software and not GCP itself).
If set up properly to be utilized correctly, yeah. But, it's not a perfect world though.
edit:
nvm s3 has regions, it's the bucket names that are global.
http(s)://<bucket>.s3.amazonaws.com/<object> http(s)://s3.amazonaws.com/<bucket>/<object>
If you look at the docs now[1], new buckets are regionalized and the region is in the URL for non-us-east-1 regions.
[1] https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_...
You have to remember that you're trying to have access to backend platforms and infrastructure at all times, which almost no public utility does (assuming "the cloud" is "public utility computing"). Power plants go into partial shutdown, water treatment plants stop processing, etc. Utilities are only designed to provide constant reliability for the last mile.
If there's a problem with your power company, they can redirect power from another part of the grid to service customers. But some part of your power company is just... down. Luckily you have no need to operate on all parts of the grid at all times, so you don't notice it's down. But failure will still happen.
Your main concern should be the reliability of the last mile. Getting away from managing infrastructure yourself is the first step in that equation. AppEngine and FaaS should be the only computing resources you use, and only object storage and databases for managing data. This will get you closer to public utility-like computing.
But there's no way to get truly reliable computing today. We would all need to use edge computing, and that means leaning heavily on ISPs and content provider networks. Every cloud computing provider is looking into this right now, but considering who actually owns the last mile, I don't think we're going to see edge computing "take over" for at least a decade.
I created a new instance in us-west2-c, which worked briefly but began to fail midday Friday, and kept failing through the weekend.
On Saturday I created yet another clone in northamerica-northeast1-b. That worked Saturday and Sunday, but this morning, it is failing to start. Fortunately my us-west2-c instance has begun to work again, but I'm having doubts about continuing to use GCE as we scale up.
And yet, the status page says all services are available.
Why do you guys suffer global outages? This is your 2nd major global outage in less than 5 years. I’m sorry to say this, but it is the equivalent of going bankrupt from a trust perspective. I need to see some blog posts about how you guys are rethinking whatever design can lead to this - twice - or you are never getting a cent of money under my control. You have the most feature rich cloud (particularly your networking products), but down time like this is unacceptable.
It's the opposite really: the expectation that service providers have no unexpected downtime is unrealistic, and it's strange this idea persists.
I agree, in general, outages are almost inevitable, but global outages shouldn't occur. It suggests at least a couple of things:
1) Bad software deployments, without proper validation. A message elsewhere in this post on HN suggest that problems have been occurring for at least 5 days, which makes me think this is the most likely situation. If this is the case, presumably given this is multiple days in to the issue, rolling back isn't an option. That doesn't say good things about their testing or deployment stories, and possibly their monitoring of the product? Even if the deployment validation processes failed to catch it, you'd really hope alarming would have caught it.
or:
2) Regions aren't isolated from each other. Cross-region dependencies are bad, for all sorts of obvious reasons.
Are software development and release processes improving to mitigate these outages? We don't know. You have to trust the marketing. Will regions ever be fully isolated? We don't know. Will AWS IAM and console ever not be global services? We don't know.
Blah blah blah "We'll do better in the future". Right. Sure. Some service credits will get handed out and everyone will forget until the next outage.
Disclaimer: Not a software engineer, but have worked in ops most of my career. You will have downtime, I assure you. It is unavoidable, even at global scale. You will never abstract and silo everything per region.
[1] https://www.theregister.co.uk/2017/03/01/aws_s3_outage/
[2] https://www.cnbc.com/2018/07/16/aws-hits-snag-after-amazon-p...
[3] https://www.cnet.com/news/google-cloud-issues-causes-outages...
[4] https://www.datacenterknowledge.com/uptime/microsoft-blames-...
http://highscalability.com/blog/2012/5/9/cell-architectures....
> Facebook Platform Appears to be down
> A check of https://developers.facebook.com/status/dashboard/ returns an error and I'm unable to login with facebook to some of my mobile apps.
https://news.ycombinator.com/item?id=18434262
If you're stuck implementing a suboptimal solution, that's not your fault, and not the intent of my above comment.
Disclaimer: google employee in ads, who worked on many many fires throughout the years, but talking from my personal perspective and not from my employer. I am sure we are striving to have 0, but realistically, i have seen many that says things happen. Learn, and improve.
Then there was the global load balancer outage in July.
Looking though the incident history, there were essentially monthly multi-region or global service disruptions of various services.
Most of what you can read of Google's approach will teach you their ideal computing environment is a single planetary resource, pushing any natural segmentation and partitioning out of view.
(Disclosure: I worked for Google, including GCP, for a few years ending in 2015. I don't work or speak for them now and have no inside info on this outage.)
GKE being the exception, since it was launched a couple years before EKS. AWS clearly has way more services, and the features are way deeper than GCP.
Just compare virtual machines and managed databases, AWS has about 2-3x more types of VMs (VMs with more than 4TB of RAM, FPGAs, AMD Epyc, etc.), and in databases, more than just MySQL and PostgreSQL. When you start looking at features you get features that you just can't get in GCP, like 16 read-replicas, point in time recovery, backtrack, etc.
Disclaimer: I work for AWS but my opinions are my own.
Some of GCP's unique compelling features include live VM migration that makes it less relevant when a host has to reboot, the new life that has recently been put into Google App Engine (both flexible environment and the second generation standard environment runtimes), the global load balancer with a single IP and no pre-warming, and Cloud Spanner.
In terms of feature coverage breadth I started my previous comment by agreeing that AWS was ahead, and I still reaffirm that. But if you randomly select a feature that they both have to a level which purports to meet a given customer requirement, the GCP offering will frequently have advantages over the AWS equivalent.
Examples besides GKE: BigQuery is better regarded than Amazon Redshift, with less maintenance hassle. And EC2 instance, disk, and network performance is way more variable than GCE which generally delivers what it promises.
One bit of praise for AWS: when Amazon does document something, the doc is easier to find and understand, and one is less likely to find something out of date in a way that doesn't work. But GCP is more likely to have documented the thing in the first place, especially in the case of system-imposed limits.
To be clear, I want there to be three or four competitive and widely used cloud options. I just think GCP is now often the best of the major players in the cases where its scope meets customer needs.
Disk and network performance is extremely consistent with AWS so long as you use newer instance types and storage types. You can't reasonably compare the old EBS magnetic storage to the newer general purpose SSD and provisioned IOPS volume types, and likewise, newer instances get consistent non-blocking 25gbps network performance.
I'm not so sure I would praise our documentation; it is one of the areas that I wish we were better at. Some of the less used services and features don't have excellent documentation, and in some cases you really have to figure it out on your own.
GCP is a pretty nice system overall, but most of the time when I see comparisons, when GCP looks better its because the person making the comparison is comparing the AWS they remember from 5-6 years ago with the GCP of today, which would be like comparing GAE from 2012 with today.
You're right that Athena seems like the current competitor to BigQuery. This is one of those things that are easy to overlook when people made the comparison as recently as a couple of years ago (before Athena was introduced) and Redshift vs BigQuery is still often the comparison people make. This is where Amazon's branding is confusing to the customer: so many similar but slightly different product niches, filled at different times by entirely different products with entirely unrelated names.
When adding features, GCP would usually fill adjacent niches like "serverless Redshift" by adding a serverless mode to Redshift, or something like that, and behavior would be mostly similar. Harder to overlook and less risky to try.
Meanwhile, when Athena was introduced, people who had compared Redshift and BigQuery and ruled out the former as too much hassle said "ah, GCP made Amazon introduce a serverless Redshift. But it's built on totally different technology. I wonder if it will be one of the good AWS products instead of the bad ones." (Yes, bad ones exist. Amazon WorkMail is under the AWS umbrella but basically ignored, to give one example.)
And then they go back to the rest of their day, since moving products (whether from Redshift or BigQuery) to Athena would not be worth the transition cost, and forget about Athena entirely.
On the disk/network question, no I didn't see performance problems with provisioned IOPS volume types, but that doesn't matter: for GCE's equivalent of EBS magnetic storage, they do indeed give what they promise, at way less cost than their premium disk types. There's no reason it isn't a fair comparison.
And for the "instance" part of my EC2 performance comment, I was referring to a noisy neighbor problem where sometimes a newly created instance would have much worse CPU performance than promised and so sometimes delete and recreate was the solution. GCE does a much better job at ensuring the promised CPUs.
I'm glad AWS and GCP have lots of features, improve all the time, and copy each other when warranted. But I don't think the general thrust of my comparison has gone invalid, even if my recent data is more skewed toward GCP and my AWS data is skewed toward 2-3 years old. Only the specifics have changed (and the feature gap narrowed with respect to important features).
Affected customers can use gcloud command [1] in order to create new Node Pools. [1] https://cloud.google.com/sdk/gcloud/reference/container/node...
That led me to believe that only the web UI was affected.
5. Years.
Nothing to see here, move along.
Edit: I saw your point a bit late. It was limited to GKE, which makes my initial comment about "service" incorrect, and it was global, which keeps my comment about "region" correct. On a related note, an SRE from GKE posted on Slack that GCE was out of resources and so GKE faced resource exhaustion as well [1][2] - so it _might_ have been a multi-service outage.
1.https://googlecloud-community.slack.com/messages/C0B9GKTKJ/c...
2. https://googlecloud-community.slack.com/archives/C0B9GKTKJ/p...
The problem is that running a global SDN like this means if you do something wrong, you can have outages that impact multiple regions simultaneously.
This is why AWS has strict regional isolation and will never create cross-region dependencies (outside of some truly global services like IAM and Route 53 that have sufficient redundancy that they should (hopefully) never go down).
Disclaimer: I work for AWS, but my opinions are my own.
Is there another status page Google? Coz the last update I'm looking at...is dated on the 9th..
_If_ that's the case, something else is causing the error messages other people are seeing
I have a feeling that a microservice architecture is overkill for 99% of businesses. You can serve a lot of customers on a single node with the hardware available today. Often times, sharding on customers is rather trivial as well.
Monolith for the win! Opinions?
Even if you deploy k8s privately, or over at Amazon, I think there's enough horror stories to make you think twice about the technology.
Then, if it isn't going to be k8s for microservices, what's a more reliable alternative?
Also, migrating to microservices for existing services might not be worth it, especially if you don't operate at a massive scale.
Keep it simple stupid is still a solid design decision, despite all the microservice/container hype.
Most bussinesses only need a couple of servers that provide the service, spread redundantly with a HA capability.
Things like throwing another node into the cluster, or rolling updates are free, which you would otherwise need to develop yourself. All of that is totally doable, of course, but I like being able to lean on tooling that is not custom, when possible.
When your infrastructure does need to become more complicated, you're already ready for it. Even if I were only serving a single language, starting with a K8s stack makes a lot of sense, to me, from a tooling perspective. Yeah normal VMs might be simpler, conceptually, but I don't consider K8s terribly complicated from a user perspective, when you're staying around the lanes they intend you to stay in. Part of this may also be my having worked with pretty poor ops teams in the past, but I think K8s gives you a really good baseline that gives pretty good defaults about a lot of your infrastructure, without a lot of investment on your part.
That said, if you're managing it on a bare metal server, then VMs may be much easier for you. K8s The Hard Way and similar guides go into how that would work, but managing high availability etcd servers and the like is a bit outside my comfort zone. YMMV.
The key issue here is that k8s was written with very large goals in mind. That a small business can easily spin it up quickly and run a few microservices or even a monolith + some workers is just coincidental. It is NOT the design goal. And the result of that is that a lot of the tooling and writing around k8s reflects that. A lot of the advice around practices like observability and service meshes comes from people who've worked in the top 1% (or less) of companies in terms of computing complexity. What I'm personally seeing is that this advice is starting to trickle down into the mainstream as gospel. Which strangely makes sense. No one else has the ability to preach with such assurance because not many people in small companies have actually been in the scenarios of the big guns. The only problem is that it's gospel without considering context.
So at what point does k8s make sense? Only when you have answers to the following:
* Getting started is easy, maintaining and keeping up with the going ons is a full time job - Do you have at least 1 engineer at least that you can spare to work on maintaining k8s as their primary job? It doesn't mean full time. But if they have to drop everything else to go work on k8s and investigate strange I/O performance issues, are you ready to allow that?
* The k8s eco system is like the JS framework ecosystem right now - There are no set ways of doing anything. You want to do CICD? Should you use helm charts? Helm charts inherited from a chart folder? Or are you fine using the PATCH API/kubectl patch commands to upgrade deployments. Who's going to maintain the pipeline? Who's going to write the custom code for your github deployments or your brigade scripts or your custom in house tool? Who's going to think about securing this stuff and the UX around it. That's just CICD mind you. We aren't anywhere close to the weeds about deciding if you want to use ingresses vs Load balancers and how you are going to run into service provider limits on certain resources. Are you ready to have at minimum 1 developer working on this stuff and taking time to talk to the team about it?
* Speaking about the team, k8s and Docker in general is a shift in thinking - This might sound surprising but the fact that Jessie Frazelle (y'all should all follow her btw) is occasionally seen reiterating the point that containers are NOT VM's is a decent indicator that people don't understand k8s or Docker at a conceptual level. When you adopt k8s, you are going to pass that complexity to your developers at some point. Either that or your dev ops team takes on that full complexity and that's a fair amount to abstract away from the developers which will likely increase the work load of devops and/or their team size. Are you prepared for either path?
* Oh also, what do your development environments start to look like? This is partly related to microservices but are you dockerizing your applications to work on the local dev environment? Who's responsible for that transition? As much as one tries to resist it, once you are on k8s you'll want to take advantage of it. Someone will build a small thing as a microservice or a worker that the monolith or other services depend on. How are you going to set that up locally? And again, who's going to help the devs accumulate that knowledge while they are busy trying to build the product. (Please ...
Given how both the JS and devops worlds seems to be progressing, is there any reason to believe that this will change before the next thing comes and K8S becomes a ghost town?
Most monoliths software companies build aren't actually monoliths, conceptually. Let's say you integrate with the Facebook API to pull some user data. Facebook is, within the conceptual model of your application, a service. Hell, you even have to worry "a little bit" about maintaining it; provisioning and rotating API keys, possibly paying for it, keeping up to date on deprecations, writing code to wire it up, worrying about network faults and uptime... That sounds like a service to me; we're three steps short of a true in-house service, as you don't have to worry about writing its code and actually running it, but conceptually its strikingly similar.
Facebook is a bad example here. Let's talk Authentication. Its a natural "first demonolithized service" that many companies will reach to build. Auth0, Okta, etc will sell you a SaaS product, or you can build your own with many freely available libraries. Conceptually they fill the same role in your application.
Let's say you use Postgres. That's pretty much a service in your application. A-ha; that's a cool monolith you've got there, already communicating over a network ain't it. Got a redis cache? Elasticsearch? Nginx proxy? Load balancer? Central logging and monitoring? Uh oh, this isn't really looking like a monolith anymore is it? You wanted it to be a monolith, but you've already got a few networked services. Whoops.
"Service-oriented" isn't first-and-foremost a way of building your application. It's a way of thinking about your architecture. It means things like decoupling, gracefully handling network failures, scaling out instead of up, etc. All of these concepts apply whether you're building a dozen services or you're buying a dozen services.
Monolithic architectures are old news because of this recognition; no one builds monoliths anymore. It's arguable if anyone ever did, truly. We all depend on networked services, many that other people provide. The sooner you think in terms of networked services, the sooner your application will be more reliable and offer a superior experience to customers.
And then, it's a natural step to building some in-house. I am staunchly in the camp of "'monolith' first, with the intention of going into services" because it forces you to start thinking about these big networking problems early. You can't avoid it.
One thing I do care about though, is root cause analysis. I love reading a good RCA, it restores my faith in the company and makes me trust them more.
(I'm not affect by the GKE outage so opinions may differ right now!)
That being said I really do think there is a difference between who is working at google today and the google we all fell in love with pre-2008.
I am sure there are a amazing people still working at google, but nowhere near like it was.
The way I like to think about google is that some amazing people mad ea awesome train that builds tracks in front of it -- you can call them gods maybe -- but those people are gone -- or a least the critical mass required to build such a train has dwindled to just dust. What we have left is a awesome train full of people pulling the many levers left behind.
To make things even worse my last interview as a SRE left me wondering if even the people who are there know this as well, and they are actually working hard to keep out those who might expose light on to this. I don't say that because I did not get the job -- I am actually happy I did not get extended a offer.
I say this with one exception, the old-timer who was my last interview. I could tell he was dripping in knowledge and eager to share it with any that would listen. I came out of his 45 min session learning many things -- I wold actually pay to work with a guy like that.
I would also like to point out that the work ethic was not what I expected. I was told that when on call, my duty was to figure out the root cause was in the segment I was responsible for. I don't know about you, but if my phone rings at night I am going to see through to a resolution and understand the problem in full -- even if it is not on the segment that I was assigned.
/end rant
That being said -- when you are on call -- dropping everything is exactly what is expected.
At the same time you ignored the massive complexity and size of Google compared to what they were at the beginning.
This is voodoo organisational analysis.
During my time on the GCE team (note I don't work at Google now) I knew multiple full-time Google employee support reps, including some still at the company. They have the good attitude and deep knowledge you'd hope for.
The problem is simply about how Google scales their GCP support org. To be completely clear, AWS support is by and large not great either.
If you're a big or strategically important customer, of course, you can get a good response from either company.
20% of my support experiences are amazing.
Fortunately, I don't require decent support to keep my service running. My sales rep tells me that he's aware of the problem.
I speculate it's simply the result of GCP trying to grow the org very quickly.
From my personal experience - i think all big cloud providers first two level support staff is no good if it isn't an obvious dumb one on your part. I always prefer to forgo support and try to go through every bit of their documentation to figure it out on our own. This helps to save huge amount of time. But if you have developer support - it can help to expedite things little faster though.
That's my favorite.
Perhaps if you explained it on a whiteboard...
1) For three whole days, it was questionable whether or not a user would be able to launch a node pool (according to the official blog statement). It was also questionable whether a user would be able to launch a simple compute instance (according to statements here on HN).
2) This issue was global in scope, affecting all of Google's regions. Therefore, in consideration of item 1 above, it was questionable/unpredictable whether or not a user could launch a node pool or even a simple node anywhere in GCP at all.
3) The sum total of information about this incident can be found as a few one or two sentence blurbs on Google's blog. No explanation nor outline of scope for affected regions and services has been provided.
4) Some users here are reporting that other GCP services not mentioned by Google's blog are experiencing problems.
5) Some users here are reporting that they have received no response from GCP support, even over a time span of 40+ hours since the support request was submitted.
6) Google says they'll provide some information when the next business day rolls around, roughly 4 days after the start of the problem.
I really do want to make sure I'm understanding this situation. Please do correct me if I got something wrong in this summary.
a) Google had a global service disruption that impacted Kubernetes node pool creation and possible other services since Friday. They had a largely separate issue for a web UI disruption (what this thread links to) which they forgot to close on Friday. They still have not provided any issue tracker for the service distribution and it's possibly they only learned about it from this hacker news thread.
b) People are having various unrelated issues with services that they're mis-attributing to a global service disruption.
...and I'm a happy GCP customer.
What exactly is your point?
Ok. So on aws we were* paying for putting systems across regions, but, honestly I don’t get the point. When an entire region is down what I have noticed is that all things are fucked globally on aws. Feel free to pay double - but it seems* if you are paying that much just pay for an additional cloud provider. Looks like it’s the same deal on GCP.
Do you have an example on this?
https://www.geekwire.com/2018/widespread-outage-amazon-web-s...
"We are investigating an issue with Google Kubernetes Engine node pool creation through Cloud Console UI."
So, it's a UI console issue, it appears you can still manage
"Affected customers can use gcloud command [1] in order to create new Node Pools. [1]"
Similarly, it actually was resolved in Friday, but they forgot to mark it as so.
"The issue with Google Kubernetes Engine Node Pool creation through the Cloud Console UI had been resolved as of Friday, 2018-11-09 14:30 US/Pacific."
Edit: I finally got my cluster up and running by removing all nodes, letting it process for a few minutes, then adding new nodes.
The items I put down in my comment are based largely on user reports, though (there isn't much else to go on). And I mean these items as questions (i.e. "is this accurate?"). Folks here on HN have definitely been reporting ongoing problems and seem to be suggesting that they are not resolved and are actually larger in scope than the Google blog post addressed.
Someone from Google commented here a few hours ago indicating Google was looking into it. And other folks here are reporting that they don't have the same problems. So it's kind of an open question what's going on.
I'm in the evaluation phase too. And I've found a lot to like about GCP. I'm hoping the problems are understandable.
I created a new instance in us-west2-c, which worked briefly but began to fail midday Friday, and kept failing through the weekend.
On Saturday I created yet another clone in northamerica-northeast1-b. That worked Saturday and Sunday, but this morning, it is failing to start. Fortunately my us-west2-c instance has begun to work again, but I'm having doubts about continuing to use GCE as we scale up.
And yet, the status page says all services are available.
So far GCP is the best, hands down in terms of stability. We never had a single outage or maintenance downtime notification till now. We are power users but our monitoring didn't pick any anomaly so i don't think this issue had rampant impact on other services.
But i find it concerning that they provided very little update on what went wrong. I also think its better to expect nil support out of any big cloud provider if you don't have paid support. Funny how all these big cloud providers think you are not eligible for support de-facto. Sigh.
I am not saying with vast numbers its feasible but big cloud providers don't even give you the opportunity to raise a ticket if its their fault. There is a price you pay extra when you opt for any one of them but many don't realize. Having said that - almost all the time, our skilled expertise is better than their initial two level of support staff. We realized it early so we handle it better by going over the documentation and making our code resilient since all cloud platforms have some limit or another since overselling in a region is something they can't avoid. Going multiple regions across when you handle these exceptions is the only way through.
If you are an early stage startup can you afford their 200/Month support, when your entire GCP bill is under $1. However, that doesn't mean you don't have to support them.
I'm glad AWS's free tier is working for you, but complaining that Google doesn't want to give you free capacity for your business and then also provide you free support for that business is pretty absurd.
If you've got VC money to blow so you can pretend your SaaS toy can feed 500 people while having money left to throw at things, that's cool. Just remember that other people might be running sustainable businesses.
And just like that you turned a $200/month bill into a $10k/month strawman.
> Just remember that other people might be running sustainable businesses.
Why are you pretending that a startup that can't afford $200/month is a "sustainable businesses"?
I mean sure, they could go and probably afford to waste $200 extra on something random that will be useless to them most of the time, but that money is going straight out of their paycheck.
You don't remain profitable though by repeatedly making bad decisions like that. Which was my point.
Running a (small) profitable business is about making the right decisions consistently, and if you're likely to waste money on one thing, you're also likely to waste it on the 19 other similar things.
Maybe speak to literally anyone you know who is running a small businesses if you want to know more. Yes that includes your local small stores on your street.
At the end of the day you probably pissed off quite a few people on here when you called their livelihood a hobby project.
This is akin to saying that a mom and pop laundromat can’t afford insurance, or shouldn’t because they won’t frequently need it.
You’re trying to equate small businesses with hobbies. You’ve now resorted to straw men, slippery slopes, and false equivalency. Maybe consider that if you have to distort the situation this much to make your point, you might just be wrong.
> At the end of the day you probably pissed off quite a few people on here when you called their livelihood a hobby project.
I didn’t say anything about anyone’s livelihood. You’re the one pretending that small businesses bringing home $120k/year can’t afford a $200 monthly support bill.
I bet the guy who started this thread about GCP’s support cost has made a sum total of <$1000 from his “startup”. Likely <$10. Hobby.
I don’t care if “quite a few people” got pissed about my comment. People with egos that delicate shouldn’t use social media.
I was trying to tell you that most small businesses can't go around spending hundreds of bucks of things that provide little value, whether that's a business support plans on services they use or something else. It's true regardless of whether you're a brick and mortar store or some online service.
> This is akin to saying that a mom and pop laundromat can’t afford insurance, or shouldn’t because they won’t frequently need it.
Speaking about about false equivalencies...
> You’re the one pretending that small businesses bringing home $120k/year can’t afford a $200 monthly support bill.
First off, I spoke of businesses making generally less than that.
Also (I already said this, good job ignoring that!) paying $200 bucks on a single useless thing is survivable for even a small business - but you know what's better than only making one bad business decision? Making no bad ones at all. Making too many will quickly break the camel's back.
Which was my whole argument and it's also what people generally refer to when they say they can't afford something.
For instance you may say "I can't afford to go to this restaurant", even though you'd have enough money to do it without going immediately bankrupt. But it'd be a bad decision, too many of which quickly add up.
And I'm telling you that if you built your business on top of GCP, a support contract is probably not "low value". You'd happily pay $200 for support on your critical infrastructure, just as you'd happily pay $200 for a repairman to fix your washing machine if you owned a laundromat.
If you don't need support, then sure, don't pay for the plan. If you do need support, $200 seems pretty reasonable.
> Speaking about about false equivalencies...
Signing up for a monthly recurring support plan in case you need it is literally insuring your business.
> For instance you may say "I can't afford to go to this restaurant", even though you'd have enough money to do it without going immediately bankrupt. But it'd be a bad decision, too many of which quickly add up.
A support plan for your critical infrastructure probably isn't "useless". Which is the point. If your need for support is that low, then either you've built your own redundant systems to protect you or more likely you aren't running a real business.
What blog statement are you referring to? I don't see any such statement. Can you provide a link?
The OP incident status issue says "We are investigating an issue with Google Kubernetes Engine node pool creation through Cloud Console UI". It also says "Affected customers can use gcloud command in order to create new Node Pools."
So it sounds like a web interface problem, not a severely limiting, backend systems problem with global scope.
Also, the report says "The issue with Google Kubernetes Engine Node Pool creation through the Cloud Console UI had been resolved as of Friday, 2018-11-09 14:30 US/Pacific". So the whole issue lasted about 10 hours, not three whole days.
> Some users here are reporting that other GCP services not mentioned by Google's blog are experiencing problems
I don't see much of that.
https://status.cloud.google.com/incident/container-engine/18...
"We are investigating an issue with Google Kubernetes Engine node pool creation through Cloud Console UI."
> So it sounds like a web interface problem, not a severely limiting
Depends who you as to whether this is "severely" limiting, but yes there is a workaround by using an alternate interface.
When things stop working, GCP is the worst. Slow communications and they require way too much work before escalating issues or attempting to find a solution.
They already have the tools and access so most issues should take minutes for them to gather diagnostics, but instead they keep sending tickets back for "more info", inevitably followed by a hand-off to another team in a different time zone. We have spent days trying to convince them there was an issue before, which just seems unacceptable.
I can understand support costs but there should be a test (with all vendors) where I can officially certify that I know what I'm talking about and don't need to go through the "prove its actually a problem" phase every time.
The issue with outages for the Government organizations I have dealt with is rarely the outage itself - but strong communication about what is occurring and realistic approximate ETAs, or options around mitigation.
Being able to tell the Directors/Senior managers that issues have been "escalated" and providing regular updates are critical.
If all I could say was a "support ticket" was logged, and we are waiting on a reply (hours later) - I guarantee the conversation after the outage is going to be about moving to another solution provider with strong SLAs.
Google thinks all their customers are idiots and that their AI can handle everything.
Sure, we use support tickets with vendors for small things. Console button bugging out, etc. But for large incidents, every vendor has a representative within an hour driving distance and will be called into a room with our engineers to fix the problem. This kind of outage, with zero communication, means the dropping of a contract.
Communication is critical for trust, especially if we're running a business off it.
You need failovers to different providers and hopefully also have your hardware for general workloads
And suddenly the CEO doesn't care anymore if one of your potential failovers is behaving flaky in specific circumstances
Not saying it's good as it is.. communication as a saas provider is - as you said- one is the most important things... But this specific issue was not as bad as some people insinuate in this thread
Don't get it wrong. AWS is the exact same thing as Google. All you will is log a ticket and receive an automated ack by the next day.
We actually got to a point where we had a couple of spare parts onsite (sticks of RAM, HD, etc) and so we repair immediately and then request the replacement. This was on a large HPC cluster so we had almost daily failures of some kind (most commonly we'd get a stick of RAM that would fail ECC checks repeatedly).
When I worked at GoDaddy, there were around 2/3 of the company was customer support.
At the current company I'm at, a cryptocurrency exchange, our support agents frequently hear they prefer our service over others because of our fast support response times (crypto exchanges are notorious for really poor support).
All of my interactions with Amazon support have been resolved to my satisfaction within 10 minutes or less.
Companies really ought to do the math on the value that comes from providing fast, timely, and easy (don't have to fight with them) customer support.
Google hasn't learned this lesson.
They have though; they've just drawn the conclusion that they'd rather put massive amounts of effort in to building services that users can use without needing support. This approach works well once the problems have been ironed out, but it's horrible until that's the case. Google's mature products like Ads, Docs, GMail, etc are amazing. Their new products ... aren't.
Google Ads and such also have a terrible support reputation, even with clients spending 8 figures.
Until something goes wrong and the only recourse is to post an angry Hacker News thread or call up people you personally know at Google to get it fixed. For example https://techcrunch.com/2017/12/22/that-time-i-got-locked-out....
Isn't that the case with basically every support request, no matter the company or severity? The first couple of emails from 1st & even 2nd level support are mostly about answering the same questions about the environment over and over again. We've had this ping-pong situation with production outages (which we eventually analysed and worked around by ourselves) and fairly small issues like requesting more information of an undocumented behavior which didn't even effect us much. No matter how important or urgent the initial issue was, eventually most requests end up being closed unresolved.
"Shibboleet" https://www.xkcd.com/806/
GCP does have role-based support models with a flat-rate plan, which is really great, but the overall quality of the responses leaves much to be desired.
https://www.hanselman.com/blog/FizzBinTheTechnicalSupportSec...
Yet, somehow every major cloud provider experiences global outages.
That old AWS S3 outage in us-east-1 was an interesting one; when it went down, many services which rely on S3 also went down, in other regions beside us-east-1 because they were using us-east-1 buckets. I have a feeling this is more common than you'd think; globally-redundant services which rely on some single point of geographical failure for some small part.
We know because we are still waiting here in ap-southeast-2 for services such as EKS to be made available. Pretty sure that any reliance within their backend services on us-east-1 was just a temporary bug and nothing systemic.