Google Kubernetes Engine is introducing a cluster management fee on June 6 (cloud.google.com)
Just received an email from Google Cloud -
"On June 6, 2020, your Google Kubernetes Engine (GKE) clusters will start accruing a management fee, with an exemption for all Anthos GKE clusters and one free zonal cluster.
Starting June 6, 2020, your GKE clusters will accrue a management fee of $0.10 per cluster per hour, irrespective of cluster size and topology.
We’re making some changes to the way we offer Google Kubernetes Engine (GKE). Starting June 6, 2020, your GKE clusters will accrue a management fee of $0.10 per cluster per hour, irrespective of cluster size and topology. We’re also introducing a Service Level Agreement (SLA) that’s financially backed with a guaranteed availability of 99.95% for regional clusters and 99.5% for zonal clusters running a version of GKE available through the Stable release channel. Below, you’ll find additional details about the new SLA and information to help you reduce your costs."
633 comments
[ 4.6 ms ] story [ 225 ms ] threadTo your point about running your own K8S cluster - two things:
1. That's something you have always been (and still are) entitled to do.
2. Having personally run large-scale K8S clusters, the challenge isn't provisioning, it's maintenance, security patches, upgrades, etc.
You guys roll out a free service. You tell your sales people to hype it up as a benefit over other providers. You somehow don't anticipate that some users will "abuse" the free service, so you hike up rates for everyone?
Sorry, I don't think you're likely to find much empathy on this one.
Strange argument. It is basically what whole world do. Give some free or heavily discounted product or service in hope of gaining market and later on increase price / start charging for that thing.
It's not what the whole world does. Many companies gasp start charging for something right away! They're entire sales pitch was that it was free. People made decisions based on that which are not so easy to turn around.
It's a legal form of bait and switch and it's hardly accepted as an ok thing to do by consumers.
I believe the community is keen to engage with you on this based on the comments in this thread. If your team would like to talk to a disappointment (very small but hoping to grow) customer I'd be happy to jump on a call. I hope others here would be happy to do the same.
Edit: Additional comments: You could limit the number of nodes in the always-free-tier clusters. Above n nodes the free tier clusters aren't free.
With the new pricing, I can't choose to use GKE instead of app engine/cloud run and get the same availability without having to pay for both the nodes and the new control plane cost. Those managed products run over multiple zones in a region. It's disappointing that even just one multi-zone cluster is charged.
With the new pricing, I can't choose to use GKE instead of app engine/cloudrun and get the same availability (by this I mean multiple zones) without having to pay for both the nodes and the new control plane cost. Those managed products run over multiple zones in a region. It's disappointing that even just one multi-zone cluster is charged. I'd be very happy to see you include at least a single multi-zone cluster control plane in the free tier.
To be clear, it's not my team. I'm relaying feedback, but I can't make any guarantees or promises.
All this feedback is super valid and important, and it's being synthesized to the product team.
Thanks for being the recipient of everyone's (justifiable) frustrations. They probably don't pay you enough.
I think, what is especially frustrating about this, is that we do already pay for resources that are provisioned by our K8S clusters. We pay for the network traffic, the storage, the compute. I saw you mention StackDriver... we pay for that as well.
I can appreciate that actually setting up and managing GKE backplanes is a non-trivial expense, but I generally assumed that that cost was amortized out, just like I don't pay for the backplane that runs GCE and the rest of GCP's service suite.
I also appreciate that you mention some customers are perhaps taking advantage of this "free" resource. But, isn't that quotas are for?
Frankly, more concerning than the fact that now I have a new $73/mo. fee attached to my account (which, is not the end of the world) is that this really comes out of left field, and in the context of concerns about the the nature of GCP's new leadership, and reports of Google leadership debating GCP as a going concern. I realize a lot of that isn't well founded, but it's surprises like this one that keep that narrative alive. AWS ain't no saint, but they are pretty consistently who they are: not full of bad surprises.
This just leaves a bad taste in the mouth, and makes me wonder if I can expect other surprising cost increases, or perhaps, if these don't "work", worse surprises like deprecation notices. Is this the precursor to you all discontinuing GKE because, as the DevRel class likes to tweet, nobody should be using Kubernetes if they can use (more expensive) services like Cloud Run?
Are we about to get Oracled?
Can confirm :)
> ...we do already pay for resources that are provisioned by our K8S clusters
Customers are charged for worker nodes, but until this point, the control plane ("master") nodes have been free. In addition to the raw compute costs for those nodes, there's the SRE overhead for managing, upgrading, and securing them.
> ...but I generally assumed that that cost was amortized out
<googlehat>I'm not really sure.</googlehat> <civilian>My guess would be that, initially, this was the case. However, over time, people have created many zero-node clusters. Now the amortization isn't. Again, pure speculation.</civilian>
> But, isn't that quotas are for?
See my comment above about zero-node clusters.
> I have a new $73/mo. fee attached to my account (which, is not the end of the world) is that this really comes out of left field...
Acknowledge, but I do want to highlight that changes take place a few months from now (June 2020), not immediately. Furthermore, each billing account gets one zonal cluster with no management fee.
> Is this the precursor to you all discontinuing GKE because, as the DevRel class likes to tweet, nobody should be using Kubernetes if they can use (more expensive) services like Cloud Run?
100% no. Also, Cloud Run is almost always cheaper than running a Kubernetes cluster.
> Are we about to get Oracled?
I'm not sure what you mean by that verb.
> In addition to the raw compute costs for those nodes, there's the SRE overhead for managing, upgrading, and securing them.
By that logic, can we expect to see charges for GCP Projects and the GCP Console? Cloud IAM?
> people have created many zero-node clusters
I'd be really curious what is driving folks to do that. Are they using the backplane for CRDs and custom controllers and no compute?
This feels like it could be addressed similar to alpha clusters, or with a quota, e.g.: clusters with 0 nodes for > 24 hours will be terminated?
Separately, It seems like handing everyone 3 months to figure out what to do about a new $73 * X fee isn't the best plan. Including some kind of estimate in the emails that were sent out would have been helpful. There was a change in pricing for StackDriver a while back that did this. It was very helpful to understand how we would be impacted.
> Furthermore, each billing account gets one zonal cluster with no management fee.
My feedback is that you would probably get getting way less blowback if that free-tier didn't come across as inadequate. I can appreciate that there are use-cases where it makes sense for you all to be charging. But one zonal cluster... It makes the whole thing feel punitive.
> I'm not sure what you mean by that verb.
I have a feeling we're all about to go on a journey of discovery together.
I was one of those people. I got an email from Google this morning and thought "that's weird. I didn't even know I was running a Kubernetes cluster." I think I created it years ago to work through a Kubernetes tutorial and, since it was free, never bothered to delete it.
So, I can imagine this being a problem. Though it seems like having a minimum hourly charge per cluster would have been a better way to handle this (i.e. if your cluster is using less than $0.10/hr in resources, you get charged the difference).
I may be missing something here, but my guess is that a lot of people turn to GKE to learn how to use K8s, and then are like "wait, I'm in the middle of this project/tutorial/etc., but I don't want to be billed overnight when it's literally just going to be doing nothing, what do I do?" and find Stack Overflow or something recommending you just scale it to zero. See questions like this: https://serverfault.com/questions/877619/turn-off-a-cluster-...
This seems like a really important detail. For hobby projects, one cluster in one zone should be enough. Per your statement, those people will not be impacted. With this knowledge, I'm experiencing much less FUD.
> I'm not sure what you mean by that verb.
The CEO of Google Cloud is the former President of Product Development at Oracle Corporation. Oracle Corporation has a reputation for being incredibly hostile to their customers, which includes things like "finding creative new ways to charge our customers more money. I mean, what are they going to do, switch to Postgres? lol"
I think the fundamental problem is that Google Cloud's reputation is irreparably harmed by Google's overall reputation among developers. Treating this like a tactical or technical problem ("our solution is the best and cheapest!") is missing the forest for the trees.
This is what I'm worried about, because it's the second time this year that GCP started charging for something that was previously included. Back in January they started charging $2.92 per month for in-use IP addresses.
The IP charge might be justifiable because of IPv4 scarcity (although their major competitors still include IPs), but with today's announcement coming just a couple months later, I'm worried they're going to start nickel-and-diming us. I'll be skeptical of any new GCP services that are advertised as having no extra cost.
I'd like to share some feedback that echoes that of other commentators, from a different perspective.
I run a local cloud developer community with regional pull for attendees, as well as working directly with local early-stage startups looking to become cloud-native.
GCP has always been my go-to for recommendation for our attendees (mix of developers and technical founders, and some enterprise technology folks) given the affordability factor, pathways to additional credits to flesh out ideas, learn new technologies, or stretch the limited runway of their new organization, and ultimately my belief that GCP is one of, if not the best, clouds for developers given the investment in documentation and engagement DevRel channels.
With the rollback of open-enrollment into a smaller plan of Google Cloud for Startups, and price changes like this, I'm fearing I've chosen the wrong hill to die on when talking with these new customers.
I appreciate the inclusion of a free regional cluster per account, which will still afford myself the opportunity to demonstrate k8s at meetups and to end users without taking more of an out-of-pocket hit, and for folks to learn on their own or maintain hobbyist projects on the same budgets they are accustomed to.
My fear with this announcement is that the negative repercussions of this will not be felt on the bottom line or figures that it seems more and more is the priority of the Google Cloud leaders. Rather, it will be felt hardest by the smaller customers; the hobbyist developer or technical co-founder looking to learn new technologies, to scale up their operations, and who at least in my experience, are driving growth in the mindspace around GCP in their communities.
Put another way, moves like this will further tarnish the reputation of Google for those who the sales engineers have for the last two years, promoted heavily no cluster management fees like "the other guys," and in the eyes of many starting out in these areas (of which I recognize most will never become the big customers that satisfy the requirements of executives).
I hope that when the dust settles, this does not lead to a retraction of what makes Google Cloud great in my mind, which is specifically to developer experience and outreach.
With that in said, I would suggest really driving home this change through dismissable in-console communication at the point of cluster creation, on the dashboard, and in email communication to folks who this will impact, with a clear picture of the impact on them. No one wants another large disruption of thousands of small organizations and users, as was the case with the Google Maps pricing change.
Personally, I'd love to see it increased to one free regional or zonal cluster per account for the remainder of 2020, and then making only one zonal free per account effective 2021. Given the uncertainty around engineer capacity and scheduling given the ongoing human malware crisis affecting companies large and small, I think this could be a good middle ground to satisfy most customers affected by these changes, while still achieving the objective of moving this away from being a loss leader of sorts.
Seems weird, given that GKE is basically the main reason people seem to use Google Cloud. These kinds of users aren't big fish, but I suspect a lot of them are going to run.
I am part of a small company which has separated our deployment into a number of sub projects, some of which are: dev, staging, production, ci, etc.
The difference for us will be several hundred dollars per month, and that will make an actual (negative) difference for us. We didn't need a "financially backed SLA" before and we don't need it now.
You asked for a question and here it is: Why isn't a financially backed SLA a part of a billing negotiation? I mean, there are some really cool features in "Anthos" but I am not picking up a phone to find out how much that is going to cost.
If a really useful feature like "Cloud Run for GKE" is awkwardly placed in the "Anthos" box, then why isn't the SLA part of "Anthos" too?
Free clusters was a huge part of why we selected GCP. If this SLA nonsense isn't made optional, our next project is not landing on GCP.
I know you are just the messenger here, and I send my sincere sympathies that you have to work with a product manager there that can't compute strategic impact of this change :)
There are reasons to have multiple clusters, but I think the current trend takes that too far.
TO BE SURE - there's more work to do in k8s and in GKE.
The real issue comes into play when you try to make developer environments.
To give our developers any semblance of a "real production-like" workload, they need to work with an entire kubernetes cluster - maybe even a couple - to simulate what's happening in production.
This means at any given time, we have hundreds of GKE clusters because each developer needs a place to try things. Yes, these are ephemeral and can be tossed aside, and yes they cost a tiny bit in VM prices, but adding a per-cluster management fee is going to skyrocket this expense and push us towards trying to figure out ways to share these clusters between developers, which defeats the entire purpose of the project.
We'll have to seriously consider abandoning GKE for this use-case now and that sucks, because it's by far the fastest managed k8s solution we've found so far.
I hope GCP re-thinks this.
> Google's inability to actually commit to long term support...
This is _exactly_ what Google is doing in this case. We are providing an SLA - a legal agreement of availability and support. These changes introduce a guaranteed availability of the management control plane.
And having it opt-in will save face with those users of GKE where an additional $73/m is significant.
P.S.: german sites have the pricing wrong.
On a hopefully more constructive note, if this is the way it's going to be from now on, I would at least expect to see an exemption on such a management fee/SLA on preemptible nodes - having an SLA and management fee on the cluster whereby nodes can be killed in a 30 second window without prior warning seems to be a little more than pointless.
As someone else noted, it breaks a lot of recommended architectures where you would have auto provisioning and a lot of clusters to separate concerns and keep costs down.
Finally, the pricing changes are starting to look like a pattern, every time Google deems the usage of a product is good enough, they will increase the price.
They are the Ryanair of the cloud.
Edit 1: moreover, it will increase the cost of composer, and on top of that, the recommended pattern where composer is paired with a kubernetes cluster for executing the workloads
Isn't Ryanair literally the Ryanair of the cloud(s)?
Two and a half 9’s is a whole different story. We achieved about 20 hours of downtime last year even without HA on k8s bare metal in Alibaba cloud. But I’m uncertain whether that’s a feat we can repeat this year.
To be fair this is hardly new and by no means limited to Google. Any number of SaaS startups that have survived to at least moderate success have done similar things.
Look at UserVoice as an example: started out with a free tier plus some reasonable paid tiers with transparent pricing, then a year or two back killed the free tier and moved to a non-transparent "enterprise" pricing model with absolutely exhorbitant fees.
Plenty of other companies offer free to build their userbase and reach, then either water down the free tier, or remove it entirely. It's practically the SV modus operandi for the last decade.
Google is not a startup, it is one of the largest companies on the planet.
Do I still have to pay the bill first, fill out forms, get account managers involved, at some point receive a partial credit, and repeat this until the delta between what I was expected as the SLA credit and what I got as the SLA credit is less than the cost of the time to fight for another cycle?
250 * 0.1 * 1000 = $25,000/month.
This is quite the price hike for something that was a) free until now. b) Not a service that warrants such a fee given that it uses existing (GCE) resources and can frankly be done manually by one of the DevOps engineers for a few hours/month and some scripting. It's just a charge for convenience it seems.
If you guys find it hard to maintain/upgrade clusters, that's your business. All I am saying is that as a company giving you business, with this change, you are now no longer the cheapest, most reliable or most convenient. As a result, we will be moving to provision instances with other providers from now on.
I mostly try not to be too Google-focused here, but I have to say...
I'm pretty proud of GKE, and I think it offers a lot of value other than just being cheap. Managing clusters is not always easy. GKE handles all of that for you - including integrations, qualifications, upgrades, and patching clusters transparently BEFORE public security disclosures happen.
We have a large team of people who deal with making GKE the industry-leading Kubernetes experience that it is. They are on-call and active in every stage of the GKE product lifecycle, adding value that you maybe can't see every day, but I promise you is there. When things go sideways, there isn't a better team on the planet to field the tickets.
I don't understand the anger here - you're literally saying you'd rather pay more for a service of lower quality because... why? Because they will continue to charge you more? Does not compute.
For those people who use a large numbers of small clusters, I understand this may make you reconsider how you operate. As a Kubernetes maintainer, I WANT to say that a smaller number of larger clusters is generally a better answer. I know it's not always true, but I want to help make it true. GKE goes beyond pure k8s here, too. Things like NodePools and sandboxes give you even more robust control
GKE is the best managed Kubernetes you can get. And we're always making it better. Those clusters actually DO have overhead for Google, and as we make GKE better, that overhead tends to go up. As someone NOT involved in this decision, it seems reasonable to me that things which are genuinely valuable have a price.
Also, keep in mind that a single (zonal) cluster is free, which covers a notable fraction of people using GKE.
It's not the fee itself, it's the worry that GKE will do what Google Maps did and massively increase fees with very little notice, causing people to scramble to migrate.
Google has a really bad reputation right now when it comes to cancelling projects that people have built their businesses upon, or jacking up fees quickly. The $73 is irrelevant on its own - the issue is (a lack of) customer trust.
I encourage everyone to always stay nimble and keep your eyes on portability. I also encourage you to try to assess the REAL costs of doing things yourself. It's rarely as cheap as you think it is.
As a Kubernetes maintainer, I am fanatical about portability.
As part of the GKE team, I think we provide tremendous value that people tend to under-estimate.
NOTE: I was NOT involved in this decision, but I understand it, and I want to help other people understand it.
Thanks for the props. It means a lot to me personally.
Aside from that, Google has a reputation for pulling this shit, and now Google Cloud does too.
The value you provide is irrelevant to how you make people feel with decisions like this.
Keep in mind that you aren't selling to me, you are selling to middle management who hears "it's different we promise" and then goes home to have nightmares about their rival manager piling on: "GCP canceled a service from under you? Who could possibly have seen that coming? Oh, the salesman told you it wouldn't happen, my mistake. (Everyone laughs at manager's stupidity.)"
GCP needs to give these guys ammunition. AWS burns goodwill like they've got a city to light: surprise bills, abandoned (but technically not canceled) services, poor performance, sticky abstractions, shameless grand announcements of services that upon further investigation only exist in the sense that you can ask support and wait 5 days (rDNS), etc. Broken software and subtle (or overt!) killer caveats abound. Go with AWS, we scale to the moon! Oh, "the moon" is >5GB of data through our time series ML? Lol no. Hard limit. Oh, we let your buddies exceed that limit and we're advertising that you can exceed the limit? Lol -- not our problem. Yet nobody holds them accountable. Nobody gets fired for choosing AWS, because AWS over-promising and under-delivering is not a meme. It's reality, especially by Google standards, so GCP marketing could make it a meme if they had any sense about them, but as far as I can tell they aren't interested.
If GCP stays the course, the results are 100% predictable and frustrating as hell, because AWS is a real steaming pile and I hate to see them continue to win The Game because Google, of all companies, can't figure out how to advertise and manage their reputation.
I'm not trying to nitpick here, but that justification is awful. It goes against reliability engineering on a deeply fundamental level, pretty much guarantees to make already not that reliable things even less reliable. Generally the more isolated entities you have and the smaller they are the less they affect each other and the environment when something bad happens, the faster they can be recovered, the fewer end users they affect, etc. If I remember correctly, this is even how some Kubernetes people justified ideas behind Kubernetes itself that you want to drop now.
Everything is a tradeoff. If you want total isolation, you pay for it. If you don't want to pay for it, you make more value-based tradeoffs.
Concretely, Google runs "a handful" of "pretty reliable" services on a relatively small number of clusters.
If Google Cloud would have charged 73$ from the start (or after beta), i think there wouldn't be so much anger.
The anger comes from, a product was free and now it is not. A lot of people made architectural choices that depended on the price of 0. (You mentioned these cases in your post).
However, i believe the bigger issue is, that Google Cloud broke essentially a promise.
As a customer I need to be able to trust my cloud provider, because I am literally helpless without it.
Can I trust an entity that breaks promises ? No, I can't. I need to worry. Especially, if I cannot follow the reasoning behind it.
If it is true, that Google's overhead went up, because of improvements, then it would have better to have two kinds of clusters (better and paid, old-school and free). You would have not broken the promise. People can choose on their own pace to upgrade if they need to.
Also keep in mind, that you also carry the Google brand. Hence, if other teams of Google break promises (like f.e. Stadia) this will also reflect on the Google Cloud team. Unless you keep a crystal clear track record, i need to assume it can get worse than what you have done right now.
My conclusion is that, I will design the cloud architecture I am responsible for, such that it has minimal dependencies on Google Cloud specifics.
Which promise?
This response, right here, is everything you need to understand about why Google Cloud is failing to sell to the enterprise market.
The enterprise market only really cares about one thing: rock solid stability. It doesn't care about features, and it doesn't (really) care about price. It wants a product that it can forget is there.
What's really sad is, technically, GKE is that product. It just works. It is solid. You do get to forget that it's there. Until you get a random email telling you that you get to explain to your boss that your bill is going up next month and your project might end up running over budget as a result.
If you can understand why a large segment of the market prefers to pay a higher but stable charge over a lower but undependable charge, then you can understand why Google Cloud is failing at selling to enterprise.
I'm the CTO at a very small company. All our stuff is running on GKE. Our monthly bill tends is a lot less than $10,000/mo. We're currently in the process of splitting our stack into separate projects and clusters, because co-locating projects in a single cluster has gotten messy. We'll probably end up with 4-5. That will increase our bill by $292/mo, worst case, assuming the first cluster is free. For a company our size, it's not a huge expense. But these things add up.
Since moving from DigitalOcean, our Google Cloud setup has more than doubled our monthly bill. We're paying for more compute, but certainly not twice the amount, as we've only gone from 14-15 nodes to around 20; it's just more expensive across the board, both node cost and ingress/egress. We're even cost-cutting by using self-hosted services instead of Google's; for example, we use Postgres instead of Cloud SQL. I ran the numbers earlier today; the equivalent on Cloud SQL would be 3.4 times more expensive.
In short, Google Cloud is expensive, and it's not like the bill is getting smaller over time.
Developments like these factor in my choice of cloud provider for future projects.
We migrated our stuff from DigitalOcean around 2018. At the time, we briefly toyed with the notion of self-hosting Kubernetes on DO, but it's complex to manage, and we don't have any dedicated ops staff. GKE is significantly easier to manage.
At that time we migrated, the things you mentioned weren't available/mature, I think. Even today, I'd choose Kubernetes over a complex mishmash of different systems. I like the unified, extensible ops model. In fact, I'd go so far as to say that I wish all of GCP could be managed as Kubernetes objects.
I had been using it since May 2018 but it didn't come out of early access till December.
Sounds like you've identified an area of risk you should address
That also excludes stateful apps like Elasticsearch that would not be able to run on Cloud Run. Not sure what Google product is appropriate there.
For those that are not aware, App Engine Flex runs services based on Docker containers with autoscaling similarly to Kubernetes. It has way less features than GKE, but if you are just running a standard app that only needs to connect to a database that is more than enough.
Bonus points: you can have multiple services, like back-end and front-end and make them available in the same subdomain - to avoid CORS problems. You can also host your front-end for almost nothing with App Engine Standard. It has an awesome CDN built-in if you know how to use it.
You might not be at the scale where this is feasible yet since that's probably multiple full-time engineers, but eventually the cost functions intersect.
The saying: “the market’s perception is your reality”, is especially apt here. Google’s decision makers tends to forget that in the end they are dealing with human customers not machines. Contrary to the concept of economic rationality, humans are notorious for exhibiting behavior that, to the untrained eye, appear irrational.
A commenter helpfully explained their perception of the new pricing change:
“The anger comes from, a product was free and now it is not. A lot of people made architectural choices that depended on the price of 0. (You mentioned these cases in your post). However, i believe the bigger issue is, that Google Cloud broke essentially a promise.”
IOW, from their perspective, the pricing change was framed as a loss [1] which opened up a host of negative emotions (anger, mistrust, etc) that come with mitigating a loss that is imminent.
Google as an engineering company may look down on fields like psychology or behavioral econs, but if they genuinely want a fighting chance against AWS and Azure, they will need to court sales leaders with a strong humanities tinge, to avoid these kinds of decision-making that achieve the opposite intended effect — eroding people’s trust in GCP.
[1] https://en.wikipedia.org/wiki/Loss_aversion
Let me put it another way. Lunch perks at Google are probably worth way more than $73/mo. Googlers would probably not feel too great about having to start paying that amount... Even though it's also worth it, people working hard to prepare food, etc.
It happens time and time again that GCP paints itself in a corner with an unsustainable offering and finds itself in a situation where it has to dent customer trust by hiking prices or retiring products on short notice.
Why?
GCP is in no position to pull this off, competitively. Now doesn't seem like the time to pull unit margins up. Google could do so on ads or with Maps API because it has the best product by far in those areas. But not in Cloud, a market where stability is a feature, and GCP is a late, small player against larger competitors who are giving customers a much more stable product.
It doesn't matter if GKE is the best thing since sliced bread, and keeps getting better — as the CTO of a cloud-based business, you have to decide whether you want to build on it over the long run and take the risk of it becoming super expensive over time; there's not a lot of reassurance that this is even a concern at Google at this point.
Tbf, you don't have the data to say that. Maybe GCP hit the growth targets they had and are not interested in growing more; maybe they've grown so much that service quality is going down; maybe they want to stall and build up other parts of the offer... there are a lot of reasons why they might want to cash in. Whether they could have foreseen it happening at this specific point in time, and how they managed the communication, is another matter. As others said, Google is saddled by default with a perception of commercial unreliability among the tech-savvy, so any new service they launch should factor that issue in terms of long-term optics.
No, they are on a death march for market share. https://www.crn.com/news/cloud/google-reportedly-set-ambitio...
I'm with you here, and the price feels about right (looking at GCE prices)--it doesn't feel like it's a ripoff or someone's trying to squeeze out more revenue.
> I don't understand the anger here
This is Google being Google. The fee probably makes sense, but as flawed humans, we're subject to the endowment effect and feel like Google's taking something from us. Amazon's all about this customer. This move is all about accurately pricing services, and it's Google putting correctness before emotions.
73/month is not a lot, but it’s still turning back on the original value proposition. If it’s such a small amount, absorb the cost as cost of doing business, so you can retain customers better and grow new ones.
https://www.theguardian.com/technology/2008/sep/29/cloud.com...
On the content, RMS chose a colorful (insulting) language, yes, do hold that against him, because that is wrong, but in my opinion, at the core, his statements are quite legit.
As for the voting, join the club. Like all communities online HNN is an often hypocritical cliquefest where the mob will have its way. But at least it's not r ;)
>A single-zone cluster has a single control plane (master) running in one zone. This control plane manages workloads on nodes running in the same zone.
https://cloud.google.com/kubernetes-engine/docs/concepts/typ...
For hobby projects nothing will change.
> As of November 28, 2017, Google Kubernetes Engine no longer charges a flat fee per hour per cluster for cluster management, regardless of cluster size, as provided at https://cloud.google.com/kubernetes-engine/pricing. Accordingly, Google no longer offers a financially-backed service level agreement for the Google Kubernetes Engine service. The service availability of nodes in a Google Kubernetes Engine-managed cluster is covered by the Google Compute Engine SLA at https://cloud.google.com/compute/sla.
> Uptime for Google Kubernetes Engine is nevertheless highly important to Google, and Google has an internal goal to keep the monthly uptime percentage at 99.5% for the Kubernetes API server for zonal clusters and 99.95% for regional clusters regardless of the applicability of a financially-backed service level agreement.
https://cloud.google.com/kubernetes-engine/sla
Interesting that they're now walking back from this...
There's no way I'm going to pay an extra $73/mo -- I already pay for the computing resources, this should be free.
Looks like I'll be moving away from GKE. It's a shame, I _was_ a big advocate.
You could have grandfathered in current deployments but - nope. In the tech world this is up there with killing Google Reader.
G Cloud is already unreasonably expensive and nearly impossible to price manage. It's cool to see them double-down on that.
https://news.ycombinator.com/item?id=20064169 https://news.ycombinator.com/item?id=17225665
It’s a race to the bottom on price so this doesn’t surprise me. They chose this life.
If they delete the droplet the second a single CC payment fails, that's one thing, but I don't believe that's how their system works.
In both of these instances I am referring to a balance of less than $20.
So for less than $20 (a few weeks late) DO says, welp fuck this customer we are going to terminate all of their resources immediately.
This is what DO and others need to do: Put it in your terms that you will keep racking up charges and then send it to collections. Charge interest, charge fees, do whatever you want. Turn $20 into $40. Why? Because businesses do not give a shit... if it is between losing everything or a slap on the wrist (monetary fee) they will chose the latter every time.
One of my clients had to painstakingly trudge through archive.net to recreate their missing blog posts. How fucking miserable is that? Over a few hundred megabytes of disk that DO could have kept around...
Also, actally make an effort to reach out before doing anything serious. Call phone numbers, email other members on the team to alert them to the issue, etc...
Too many times I have seen some script kiddie throw together a client's WP site and toss it on DO because it is 'so cheap and cool' and yet they forget about everything else: backups, security, managing the box, etc... and inevitably shit will hit the fan.
I was really rootin' for DO in the beginning. I even applied to work there when they were first starting out but did not want to relo to NY. Now I am moving three clients OFF of DO because they are all very unhappy with the level (or lack) of service they've received.
In my experience, AWS is a very expensive cloud with clunky UI and big brand name. During consulting gigs, I have seen many customers want to go with AWS only because of brand. And later they cry when bills start to hit roof with vendor lock in.
====================== First email from AWS (several days after my request): ======================
Thank you for submitting your Limi Increase request.
I'm contacting your to inform you that we've received your Workspaces Application Manager - Total Products limit increase request, for a max of 5 in the Oregon region. I will be more than happy to submit this request on your behalf.
Please note that for a limit increase of this type, I will need to collaborate with our Service team to get approval. This process can take some time as the Service team must review your request first in order to proceed with the approval. This is to ensure that we can meet your needs while keeping existing infrastructure safe.
You may rest assured I will push towards expediting your request to be addressed as soon as possible. As soon as the Service team contacts me I will definitely let you know by email.
In the meantime, please feel free to let me know if you have any additional questions or concerns and I'll be happy to help!
I appreciate your patience while we evaluate your request.
====================== Second email: ======================
Thank you for your kind patience whiIe we continue to evaluate your Workspaces Application Manager - Total Products limit increase request.
I apologize for the time is taking to provide you with a resolution as we've always aimed to provide our customers with a rewarding experience that meets and goes beyond expectations. Unfortunately, from time to time there are cases where the final outcome is handled by another department and the time they take is completely out of our hands.
We certainly understand the sense of urgency that you have for this particular request and therefore, we have spent time communicating with the service team to let them know about it. Rest assured that your case is active, being looked into and the sense of priority has been transferred. As soon as we have an update from their end we'll be touching base with you immediately.
I am committed ensuring that you will get the help that you need as fast as possible, so we can ensure everything is being handled to your satisfaction, please feel free to let us know if you have any further questions or concerns through this case, so we can address them as soon as possible.
============= My response: =============
You can go ahead and cancel my request -- I've decided to not go forward with my project.
============= Their reply: =============
Greetings from Amazon Web Services.
We're sorry. You've written to an address that cannot accept incoming e-mail.
If you need to contact us, please visit http://www.aws.amazon.com/contact-us .
Thank you for your business.
Workspaces are already a specialized product built on the AWS ecosystem; WAM is a niche management tool inside that niche.
I run a Kubernetes cluster on Vultr and I haven't had any problems.
>As a free service, AKS does not offer a financially-backed service level agreement. We will strive to attain at least 99.5% availability for the Kubernetes API server. The availability of the agent nodes in your cluster is covered by the Virtual Machines SLA. Please see the Virtual Machines SLA for more details.
google didn't have one either, so they added it + a price for said SLA.
Replies on another thread by alleged Google employees appear to indicate they are learning on the fly that doing support matters and costs money.
There are a lot of comments from people wanting free things. You may get those things for a while but it won't last. Free only works for so long in anything no matter how hard that is for some people to figure out across all aspects of life.
Don't marry yourself to a provider, stay portable, it's just good risk management. Pricing changes? Spin up a cluster elsewhere, migrate data, migrate traffic, profit. The terms of the agreement can change at any time.
AWS, for example, begins with high prices and then lowers them over time. It costs money but you know the maximum.
Google seems to be grabbing you with cheap prices and then jacking them up when you're committed (google maps is another example offhand). Maybe no on purpose but bad initial pricing and ill-intent have the same impact externally.
Past performance is no guarantee of future benevolence or reasonable behavior.
For a startup, the risk of AWS doing something is tiny compared to all other risks so not worth spending effort to mitigate.
For a moderately sized company, true, being on multiple clouds may have an advantage.
For a large company, you get long term contracts with AWS that mitigate the risk.
Sure, but pushing prices down and eating competitors margin to the benefit of the customer has been Amazon's MO since inception.
Google seems to make a lot of missteps wrt pricing and the cloud. Remember the geo pricing change that put a lot of projects out of business??
That's only true for official pricing though.
I know of multiple cases where AWS had initially given significant discounts, only to stop doing so once they believed the customer to be firmly tied to the platform.
Adopt Anthos or pay, basically.
Oh, and per their docs, three-control-planes decision is not reversible - I cannot in fact shut two of those down without shutting down my production cluster and starting a new one. https://cloud.google.com/kubernetes-engine/docs/concepts/typ...
Awful. Just so awful.
Edit: To answer some questions below - we have a single-tenant model where we run an instance of our async discussion tool (https://aether.app) per customer for better isolation, that’s why we had bought into Kubernetes / GCP. Since we have our own hypervisor inside the clusters, it makes me wonder whether we can just deploy multiple hypervisors into the same cluster, or remove the Kubernetes dependency and run this on a Docker runtime in a more classical environment.
That said they do make it damn hard. Our k8s cluster is as basic as it comes, no databases, simple deployments, but we do still have a dependency on Google Cloud Loadbalancer (which we hate).
If pricing goes up too much from this we'll move, but the GCL dependency will be a PITA :/
https://github.com/jetstack/cert-manager
Notably, we may choose to make breaking changes to our API specification (i.e. the Issuer, ClusterIssuer and Certificate resources) in new minor releases.
It's not dark magic, it might make building off of it in the form of integrations prohibitive, but they have done a great job making sure users can upgrade one release to the next.
It is a little bit of a treadmill, but it certainly beats manually renewing certificates!
I'm guessing 99% of workloads won't notice either of these issues, but it is an actual issue.
It's no different than running your own load balancer like HAProxy pointed at the nodes which forward to a node-port service.
There's also MetalLB if you're running your own hardware: https://metallb.universe.tf/
Pretty much anyone who works in ops longer understood from the go that its impossible to be totally provider-agnostic. K8S is just a nice api on top of provider api that still requires provider specific configuration.
https://github.com/kubernetes/website/issues/19139
Google will ignore it like all of the tickets I file. The fact is that Google is in the business of making money and they are focused on enterprise users. Enterprise users are not sensitive to integration difficulty since they can just throw people at any problems. So eventually everything Google makes will become extremely time-consuming to learn and difficult to use. They're becoming another Oracle.
So either way you end up in a situation where you require dedicated devops team pr dedicated teammembers to keep up with changing requirements.
If you're going to run on bare-metal or in your own VMs, OpenShift is very much worth a look. There are hundreds, maybe thousands of ways to shoot yourself in the foot, and OpenShift puts up guard rails for you (which you can bypass if you want to). OpenShift 4 runs on top of RHCOS which makes node management much simpler, and allows you to scale nodes quickly and easily. Works on bare metal or in the cloud (or both, but make sure you have super low latency between data centers if you are going to do that). It's also pretty valuable to be able to call Red Hat support if something goes wrong. (I still shake my head over the number of days I spent debugging arcane networking issues on EKS before moving to OpenShift, which would have paid for a year or more of support just by itself).
It's not a rolling upgrade, but it allows you to move the apps, PVs, policies in as granular a manner as you wish.
Because of that major change, the cluster upgrade path is a little more involved than usual. It's a complete reinstall of the OS and rebuild of the cluster. There are tools to help tho, and as the path is tread things will get easier and better supported. If you go through the same wave as me, you'll be annoyed at first but then once it is done and you have a 4.x cluster you'll be really happy with it (especially when you can manage everything as an operator).
Luckily from an app perspective very little will change since Kubernetes is the API. A sibling comment linked to some helpful documentation. I can't be specific right now but I can tell you that your need is known, and some very smart people are working on it. If you want to email me (check my HN bio page) or jump in the Keybase group called "openshift_okd," I'm happy to chat more about it (not in an official Red Hat support capacity, just as friend to friend :-) ). I haven't done a migration myself yet but I know people who have, and I plan to get into it personally soon as well.
Don't get me wrong, I appreciate RHAT's code contributions very much, they have done a lot for k8s! But running OKD on one's own is a bad idea, while paying for IBM support makes you as much a hostage as anything Google will do to you. Better to just stick with a distribution with better community support and wait for RHAT's useful innovations to be merged upstream (while avoiding the pitfalls of their failed experiments...)
* Yes it's open source, but the community version (okd) isn't really supported, nor is it widely used, so if you're serious about running this you're doing so for the Enterprise support and you're going to be writing those checks
Your concern is valid, and I agree with you that OKD is not supported enough. I have my own theories as to why, but I will keep my criticism "in the family" (but do know there are people that want to see OKD be a first-class citizen, and know we are falling short right now). We had some challenges supporting OKD 4.x because the masters in 4.x now require Red Hat CoreOS (and nodes it is highly recommended), but RHCOS was not yet freely available. This is obvoiusly a big problem. Now that Fedora CoreOS is out, there is a freely distributable host OS on which to build OKD, so it will be better supported and usable. FWIW I have a personal goal to have a production-ish OKD cluster running for myself by end of the year.
I'll admit I am a little offended at being called a "proprietary IBM K8s distribution," but I don't think you meant to be offensive. IBM has nothing to do with OpenShift, beyond the fact that they are becoming customers of it. Every bit of our OpenShift code is open source and available. You are right that it's not in a production-usable state (although there are people using it) but it's a lot better than you'll get from other vendors. We are at least trying to get it to a usable state, unlike many of them. We are strapped for resources like everyone else, and K8s runs a mile a minute and requires significant effort to stay ahead). This space is still really young and really hot, and I am confident we'll get the open source in a good, usable, state, much like Fedora and CentOS are. I also don't think OpenShift is really all that expensive considering the enormous value it provides. The value really does shine at scale, and may not be there for smaller shops.
I don't blame you for waiting, I probably would too. Our current offering is made for enterprise scale, so isn't tailored to everyone. I've heard OpenShift online has gotten better, but haven't tried it myself. Eventually I plan to run all my personal apps on OKD (I have maybe a dozen, mostly with the only users being me and my family), but until then I've been using podman pods with systemd, which will be trivial to port to OKD once it's in a good state.
It's not that openshift is bad per se, I just don't imagine it solves many problems an org that is fretting about lock in or gcp pricing will have. Such an org is probably cost sensitive and looking for flexibility, but openshift is expensive, and if you adopt its differentiating features you are de facto locking yourself in. And if you do not leverage those features out of a desire to avoid lock in, you are effectively paying a whole lot just for k8s support...
And I really should say, for certain orgs (especially bigcos) this may well be worth it, I just don't think it is a good option for anybody worried primarily about avoiding vendor lock in and keeping costs in check.
Again, I think the idea of OS is great, but you've lost us, and likely other big customers because of that restriction. Having old kernels is just not an option for some people.
If you are like me and are old school and think "huh, yeah that makes me nervous" I completely understand that, but we've seen some serious success with it. I'm a skeptical person, and telling me I can't SSH to my node freaks me out a bit, but I'm becoming a convert.
I would also note if you buy OpenShift you get the infrastructure nodes (masters, and some workers for running openshift operator pods) for free (typically, but I'm not a salesperson so don't hold me to that if I've misspoke :-P), so you aren't paying for the super locked in OS. I suppose you do have to pay for RHEL8 or RHCOS on the worker nodes running your pods, and we don't support other distros (because we expect a very specific selinux config, CRI-O config (container runtime), among other things), so I guess there's some dependence there, although I recommend RHCOS for all your nodes and then just use the Machine API if you need it.
Yes RH did a lot for k8s, but killing of a working distribution without a direct migration path that is like "start again". will make your customers angry.
also I think the OpenShift terminology is way too much and OpenShift should be a way more thinner layer on top of k8s.
Sure it's opinionated, but at this point, which flavor of k8s isn't? Even k3s (which I play around with on https://github.com/rcarmo/azure-k3s-cluster) has its quirks.
Everyone who's targeting the enterprise market has a "value-added" brew of k8s at this point, so kudos to RH for the engineering effort :)
If you want something production-grade (i.e. doesn't say "beta" on the tin) then I think Calico should solve most of the same problems too (it does BGP peering to your ToR switch):
https://docs.projectcalico.org/networking/determine-best-net...
Does MetalLB do something extra I'm missing?
https://pages.awscloud.com/Gartner-Magic-Quadrant-for-Infras...
It has more features, yes. How well those features work is another matter entirely.
I loosely follow AWS, GCP and Azure but I always get mixed opinions on them, especially the last two
To pick a random example that I'm familiar with: Azure DNS Zones.
When I used AWS Route 53, the main issue I had with it was that I thought the cutesy marketing name was stupid. That's about it. By reading through the docs, I learned a little bit about DNS I didn't know, and I got to learn about the clever engineering that AWS did to work around issues with the DNS protocol itself. In the end, it had more features than I needed, and the basic stuff Just Worked.
When I tried to use Azure DNS, their import tool shredded my data. I then wrote a custom PowerShell import tool, but it took hours to import a mere few thousand records. The next day my account was locked out for "too many API calls" because I simply had the console web gui open. Not used. Open. The GUI showed entries different to the console tools. The GUI string limits don't match the console tool. The perf monitor graph was broken, and is still broken. Basic features were missing, broken, or "coming soon".
You would think DNS would be one of those services that "just works", but nope. Bug city.
Now mind you, most of those issues are fixed now, and they're adding more features and fixing the issues those new features are introducing.
But ask yourself: Why are buggy features being rolled out in production? Did nobody test this shit? Did they ever do a load test? Did they even try basic things like "have the console open with more than 10 records"? Why am I discovering this? Do they not have thousands of customers who have battle-tested this stuff?
Clearly they're just throwing things over the fence and letting support tickets be their QA feedback.
PS: It's even how they use DNS themselves that's just wrong. E.g.: If you use Azure CDN you end up with like 6 CNAME redirects in a row. The DNS standard says CNAMEs shouldn't point to other CNAMEs! At a minimum this is slower than it needs to be, but it's also less reliable because there's more points of failure...
https://www.canalys.com/static/press_release/2020/Canalys---...
AliCloud and Rackspace are very close to GCP as well.
That being said, if you're planning on running Kubernetes, I'd choose GCP over any other offering - the tooling and support just seems better, in my entirely subjective opinion.
My experience is that once AWS offers a new service that gets attention, a few month later also Azure offers it - and vice versa.
Also lack of SLA / shady SLA does not help.
Ps. Talking as someone with hands on experience.
Ps2. Azure support is terrible and their response times are constantly breaking SLA..
Honestly, I've been pleasantly surprised.
The best support I got a mispleasure to work with didn’t even know how the service Im having an issue is even working... And it was a “technical” ticket.
Seems like after the last round of pouching of best support engineers by AWS, Azure is left with only outsourced mob.
Another - constant disconnections of PV..
Another - 1/3 times the new provisioned node in vmss has a broken kubelet and doesn’t successfully register in control-plane. I was literally shocked when it happened twice in a single day. Response from support was that we are supposed to monitor that ourselves and drain unsuccessfully provisioned node - (we already were and it was mentioned in opening ticket) makes scaling horizontally REALLY PAINFUL..
CNI default reservation of IPs (30) - cannot be less - so if you have a service node running and you want only few pods to run on it for HA - well sucks to be you.
Kubenet not working until up recently with anything - for example AG, tho AG is a disaster of a service by itself.
Various API failures related to networking - sometimes control-plane lost connection to AKS subnet for some time (fixed by itself by still...)
The magic quadrant isn't a good housekeeping seal of approval. It's a screener for an architect in Fortune 500 or .Gov to show social proof that their product selection isn't insane.
The "cautions" for GCP are about the nascent state of their field sales and solution architects, enterprise relationship management, and limited partner community.
The "cautions" for Azure are poor reliability, poor support, and sticker shock.
My takeaway was very different from yours. When you read the analysis, it was reflective of a mature, dominant player (AWS) and two highly capable challengers with different issues.
Google is a newer business that is missing some services (ie. anything user facing) and is transitioning from a weird sales model to a more conventional enterprise one. Microsoft has an established business process and best in class sales org, but they tend to use sticks of various types to force adoption and are organizationally poorly equipped to support customers.
I also think of them as the second, inferior cloud, but they're almost certainly the better k8s hoster. If you're serious about running k8s on AWS, there's a good chance you're doing something like CloudPosse's Terraform-based setup, not EKS.
There's also one completely free zonal cluster for hobbyist projects.
The main issue is that not charging for the control plane and charging for the control plane leads to two very different Kubernetes architectures, and as per your docs, those decisions made at the start are very much set in stone. You cannot change your cluster from a regional cluster to a single zone cluster for example. So you have customers who built their stacks taking into account your free control plane, and you’re turning the screws in by adding a cost for it — but they cannot change the type of their cluster to optimise their spend, since, per your docs, those decisions are set in stone. That’s entrapment.
You should keep existing clusters in the pricing model they’ve been built in, and apply this change for clusters created after today.
That said, many of us made a bet on GCP. For us in particular, we made a bet to the point that our SQL servers are on AWS, but we still switched to GCP for ‘better’ Kubernetes and for not nickel and diming, since AWS had a charge that looked like it was designed to convey that they’d much rather have you use their own stuff than Kubernetes. It is a relatively trivial amount, but it makes a world of difference in how it feels and you guys know more than anyone how much of these GCP vs AWS decisions are made based not on data sheets but for the ‘general feel’ for the lack of a better word.
AWS’ message is that they’re the staid, sort of old fashioned, but reliable business partner. GCP’s message, as of this morning, is stop using GCP.
https://aws.amazon.com/eks/sla/
That about sums up most things Google does for developers.
However this means we are paying for egress on both sides. This was something we chose to eat due to GCP Kubernetes, but considering today’s changes, it probably no longer makes sense.
These changes won't take effect until June - customers won't start getting billed immediately. I'm sorry that you feel trapped, that's not our intention.
> You should keep existing clusters in the pricing model they’ve been built in, and apply this change for clusters created after today.
This is great feedback, but clusters should be treated like cattle, not pets. I'd love to learn more about why your clusters must be so static.
What’s inside our clusters are indeed cattle, but the clusters themselves do carry a lot of config that is set via GCP UI for trivial things like firewall rules. Of course we could script it and automate, but your CLI tool also changes fast enough that it becomes an ongoing maintenance burden shifted from DevOps to engineers to track. In other words, it will likely incur downtime due to unforeseen small issues.
It’s also in you guys’ interest that we don’t do this and clusters are as static as possible right now, since if we are risking downtime and moving clusters, we’re definitely moving that cluster back to AWS.
More importantly, this dialogue speaks volumes to Google’s stubbornness. Seth’s/Google’s position is: do it the Google way, sorry-not-sorry to all those that don’t fit into our model.
Like we haven’t heard of infrastructure as code? That can’t paper over basics like being unable to change your K8s cluster control plane. This is precisely the attitude that lands GCP as a distant #3 behind AWS and Azure.
AWS has the complete opposite model.
You provide a web interface, so it’s reasonable to assume people will use it.
(Or, really, I'd rather just run my workloads directly on some scale-free multitenant k8s cluster that resembled Borg itself—giving me something resembling a PaaS, but with my own custom resource controllers running in it. Y'know, the k8s equivalent to BigTable.)
Multiple clusters lets us easily firewall off communication to compute instances running in our account based on the allocated IP ranges for our various clusters (all our traffic is default-deny and has to be whitelisted). Multiple clusters lets us have a separate cluster for untrusted workloads that have no secrets/privileges/service accounts with access to gcloud.
Starting in June our monthly bill is going to go up by thousands. All regional clusters.
GKE has the ability to run pods with gVisor, which prevents the pod from communicating with the host kernel, even maliciously. (I think they call these sandboxes.)
The only reason to use multiple clusters is if you want CPU isolation without the drawbacks of cgroups limits (i.e., awful 99%-ile latency when an app is being throttled), or you suspect bugs in the Linux kernel, gVisor, or the CNI. (Remember that you're in the cloud, and someone can easily have a hypervisor 0-day, and then you have no isolation from untrusted workloads.)
Cluster-scoped (non-namespaced) resources are also a problem, though not too prevalent.
Overall, the biggest problem I see with using multiple clusters is that you end up wasting a lot of resources because you can't pack pods as efficiently.
https://cloud.google.com/kubernetes-engine/docs/best-practic...
Thank you.
Ultimately, all isolation has its limits. Traditional VMs suffer from hypervisor exploits. Dedicated machines suffer from network-level exploits (network card firmware bugs, ARP floods, malicious BGP "misconfigurations"), etc. You can spend an infinite amount of money while still not bringing the risk to zero, so you have to deploy your resources wisely.
Engineering is about balancing cost and benefit. It's not worth paying a team of CPU engineers to develop a new CPU for you because you're worried about Apache interfering with MySQL; the benefit is near-zero and the cost is astronomical. Similarly, it doesn't make sense to run the two applications in two separate Kubernetes clusters. It's going to cost you thousands of dollars a month in wasted CPUs sitting around, control plane costs, and management, while only protecting you against the very rare case of someone compromising Apache because they found a bug in MySQL that lets them escape the sandbox.
Meanwhile, people are sitting around writing IP whitelists for separate virtual machines because they haven't bothered to read the documentation for Istio or Linkerd which they get for free and actually adds security, observability, and protection against misconfiguration.
Everyone on Hacker News is that 1% with an uncommon workload and an unlimited budget, but 99% of people are going to have a more enjoyable experience by just sharing a pool of machines and enforcing policy at the Kubernetes level.
I agree with most things you’ve said around gVisor providing sufficient security, but it's not just about security, noisy neighbors are a big issue in large clusters.
We're okay with the waste as long as our software & deployment practices can treat any hosted Kubernetes service as essentially the same.
I'm sure there are tools out there to help with cluster migrations, but it is far from trivial.
Meanwhile most of my large government customers have a couple of tiny VMs for every website. That's it. That's already massive overkill because they see 10% max load, so they're wasting money on the rest of the resources that are there only for redundancy. Taking things to the next level would be almost absurd, but turning things off unnecessarily is still an outage.
This is why I don't believe any of the Cloud providers are ready for enterprise customers. None of you get it.
They rejected it because it's "too complex to do internal chargebacks" in a shared cluster model.
This is what I mean: The cloud is for orgs with one main application, like Netflix. It's not ready for enterprises where the biggest concern is internal bureaucracy.
Get your data centers all running VMWare, then VMDK import to AWS AMIs, then wrap them all in autoscaling groups, figure out where the SPOFs have moved to, and only then start moving to containers.
In the mean time, all new development happens on serverless.
Don't let anything new connect to a legacy database directly, only via API at worst, or preferably via events.
I know already too many people which are stuck at a certain k8s version. Do not allow that to happen!
Bull-fucking-shit.
Heh... how many teams actually treat their clusters like cattle, though? Every time I advocate automation around cluster management, people start complaining that "you don't have to do that anymore, we have Kubernetes!"
Some people get it, yes, but even of that group, few have the political will/strength to make sure that automation is set up on the cluster level—especially to a point where you could migrate running production workloads between clusters without a potentially large outage / maintenance window.
Sugarkube is designed to do exactly that.
[1] https://docs.sugarkube.io
I don't know GCP though. In the past I've seen kube cluster archs which are very very fragile as they spin up. If that's the case with GCP I can see why you wouldn't do the above and rather hand hold their creation.
Please don't do this. You can apologize for your actions work to improve in the future , but you cannot apologize for how someone feels as a result of your actions.
Also, intent doesn't matter unless you plan to change your behavior to undo or mitigate the unintended result.
Ha. They should, but they are absolutely not. Customers typically ask "why should we spend time on automating cluster deployment when we are going to do it just once?" and when I explain that it's for when the cluster goes away, if it goes away, they say it's an acceptable risk.
The truth of the matter is, even in some huge international companies, they don't have the resources to keep up with development of tools to have completely phoenix servers. They just want to write automation and have it work for the next 10 years, and that's definitely not the case.
Off-topic, but is this really how people do k8s these days? Years ago when I was at Google, each physical datacenter had at most several "clusters", which would have fifty thousand cores and run every job from every team. A single k8s cluster is already a task management system (with a lot of complexity), so what do people gain by having many clusters, other than more complexity?
People had exactly the same experiences with Mesos and OpenStack, but k8s has decent tooling for turning up many clusters, so there is an easy workaround
I mean, if people aren't smart enough to run a large shared infrastructure, how can I trust them to run a large number of shared clusters, even if each cluster is small. The final scale is still the same.
Clusters are probably still pets to most orgs, but the lessons about how to manage complexity still apply. Each of my terraform state files is a pet and I treat it like such... but I also use change-control to assure that even though I don't regularly recreate it from scratch, I understand all that was there.
* Fully reproducible cluster builds and deployments.
* The type of cluster (can be) an implementation detail, making it easy to move between e.g Minikube, Kops, EKS, etc. After all, K8s is just a runtime.
* Developers can create temporary dev environments or replicas of other clusters
* Promote code through multiple environments from local Minikube clusters to cloud environments
* Version your applications and dependent infrastructure code together
* Simplify upgrades by launching a brand new cluster, migrating traffic and tearing the old one down (blue/green)
* Test in-place upgrades by launching a replica of an existing cluster to test the upgrade before repeating it in production
* Increase agility by making it easier to rearchitect your systems - if you have a pet, modifying the overall architecture can be painful
* Frequently test your disaster recovery processes as a by-product for no extra effort (sans data)
* Reduced blast radius
[1] https://docs.sugarkube.io/#benefits-of-sugarkube
This is a pretty good in depth explanation, but at a high level if a your server dies and you are extremely upset about it (similar to if your pet died) you are putting too many eggs in that single basket, with no secondary plan. Conversely if you build your infra in such a way that your server dying is something you see no worse than how a farmer sees one of his cattle dying (which are raised to be killed) - you are much better prepared for the inevitable downtime from your server and can very easily recover
http://cloudscaling.com/blog/cloud-computing/the-history-of-...
Clusters often are not "cattle". If your operation is big enough, then yes, they might be. Usually they aren't, they are named systems and represent mostly static entity, even if the components of said entity change every hour.
Personally, I'm running in production a cluster that by now had witnessed upgrades from 1.3 to 1.15, in GKE, with some deployments running nearly unchanged since then.
Treating it as cattle makes no sense, especially since on API level, the clusters aren't volatile elements.
For us, Clusters are a promise to our developers. We can’t just spin up a new cluster because we feel like it. I must be missing something or maybe our culture is just different.
For every 1 competent person who can develop a solution to fully automate everything, there are 99 others who can automate most of that, maybe minus a cluster or DB or two, and another 500 whom cannot do either, but can run a CentOS box at a reasonable service level.
You experience using great tools and your vast knowledge of k8s each day to do all that, and you have the support of your org, but those other folks may not have the tools, support, knowledge, or sometimes even the capability to attain the knowledge to do that. That doesn't mean they're useless to anyone, to be cast off at will!
The type of thinking that leads to, "get new engineers/developers/designers/architects if yours aren't perfect" needs to die, and needs to be replaced with, "let's do what we can to train and support our current employees to do a great job" because, frankly, there aren't enough "superstar" people who have your skills to do that at every org.
We need to work on accepting people for who they are-- helping them to strive to be a bit better each day of course-- and utilizing those skills in the right place, rather than trying to make everyone the same person with the same skills doing the same things.
Some applications don't need clusters which can be rebuilt and destroyed at will, so let's not make that the bar for every project.
P.S. I consider "pet vs cattle" to be a horrible metaphor that should be taken behind the barn and short like the diseased plague bearer it is.
So my understanding that the official k8s way to upgrade your cluster is also to throw it away and start a new one (with some cloud provider proprietary alternatives).
Let's say there is something actually important, stateful, single-source-of-truth in my k8s cluster, like a relational DB that must not lose data. I don't want downtime for readers or writers, and I want at least one synchronous slave at all times (since the data is important). I also don't want to eat non-trivial latency overheads from setting up layers of indirection.
What's the recommended way of doing this?
I am not expert, but K8s handles task replication, and either spawn or route a request to another task instance somewhere else. However, the application logic itself must handle the fault-tolerance (by handling its states through transactions or something else) should an instance fail. K8s doesn't do that for you.
You need to instantiate a secondary DB replica somewhere else and start the DB migration. Since there will be "two instances" of the same DB running, you will also need to set up a (temporary) proxy for routing and handling the DB requests w/ something like this:
1) if the data being requested is already migrated, request is handled by the (new) secondary replica. 2) Primary instance handles the request, otherwise 2.1) Requested data should be migrated to secondary replica (asynchronously, but note that a repeated request may invalidate a migration).
Turn the proxy router off once the whole state of your primary DB instance is fully migrated, making the secondary replica the primary one. That's really just a napkin recipe for completing a live migration, though.
* We are now getting into the distributed transaction world because you can never be 100% sure that writing to 2 databases can succeed or fail at the same time. There is this talk from this guy who deals with similar problem you have: http://www.aviransplace.com/2015/12/15/safe-database-migrati...
This makes it seem like Google Cloud for Startups is aimed at startups that aren't really on a shoestring budget.
[1]: https://cloud.google.com/developers/startups/
Or, better yet, don't use k8s. You don't need it, especially as a startup on a shoestring budget. You can migrate later if you decide you really need to, but just a plain LAMP gets you 99% of the way.
Around the time "Google Cloud Platform" became a thing, Google changed GAE from an encapsulated bubble into a basic frontend management system that interacts with normal services through public APIs (either inside or outside GCP). It's more expensive than GCE, but it's fully managed and lets you skip the devops team.
My problem is that this fee doesn't look very "cloud" friendly. Sure the folks with big clusters won't even notice it, but others will sweat it.
The appeal of cloud is that costs increase as you go, and flat rates are typically there to add predictability (see BigQuery flat rate). This fee does the opposite.
This cost increases rapidly for those scenarios.
So I guess the big question in my mind is how do you run containerized apps in the major clouds besides K8s if it's a bulldozer and you just need a cargo bike? Is there something simpler?
(As a security person: ugh.)
More likely though, AWS or OpenShift running on bare metal on a beefy ATX tower in the office. We want to have production and staging as close to each other as possible, so this is an additional reason and a p0 flag on reducing the dependency on Google-specific bits of Kubernetes as much as possible, hopefully also useful for our exit strategy as well.
I'll use helm to install metallb for the load balancer, which you can then tie into whatever egress controller you like to use.
For persistent storage a simple NFS server is the bees knees. Works very well and a NFS provisioned is a helm install. Very nice, especially, over 10GbE. Do NOT dismiss NFSv4. It's actually very nice for this sort of thing. I just use a small separate Linux box with software raid on it for that.
If you want to have the cluster self-host storage or need high availability then GlusterFS works great, but it's more overhead to manage.
Then you just use normal helm install routines to install and setup logging, dashboards, and all that.
Openshift is going to be a lot better for people who want to do multi-tenant stuff in a corporate enterprise environment. Like you have different teams of people, each with their own realm of responsibility. Openshift's UI and general approach is pretty good about allowing groups to self-manage without impacting one another. The additional security is a double edged. Fantastic if you need it, but annoying barrier to entry for users if you don't.
As far as AWS goes... EKS recently lowered their cost from 20 cents per hour to 10 cents. So costs for the cluster is on par with what Google is charging.
Azure doesn't charge for cluster management (yet), IIRC.
However, if you can tolerate client side failure then go for it.
For example you could use NFS for 90% of the storage needs for logging and sharing files between pods. Then use local storage, FCOE, or iSCSI-backed PVs for databases.
If you are doing bare hardware and your requirements for latency are not too stringent then not hosting databases in the cluster is also a good approach. Just used dedicated systems.
If you can get state out of the cluster then that makes things easier.
All of this depends on a huge number of other factors, of course.
I think NFS is heavily underrated. It's a good match for things like hosting VM images on a cluster and for Kubernetes.
In the past I really wanted to use things iSCSI for hosting VM images and such things, but I've found that NFS is actually a lot faster for a lot of things. There are complications to NFS, of course, but they haven't caused me problems.
I would be happy to use it in production, and have recommended it, but it's not unconditional. It depends on a number of different factors.
The only problem with NFS is how do you manage the actual NFS infrastructure? How much experience does your org have with NFS? Do you already have a existing file storage solution in production you can expand and use that with Kubernetes?
Like if your organization already has a lot of servers running ZFS, then that is a nice thing to leverage for NFS persistent storage. Since you already have expertise in-house it would be a mistake not to take advantage of it. I wouldn't recommend this approach for people not already doing it, though.
If you can afford some sort of enterprise-grade storage appliance that takes care of dedupe, checksums, failovers, and all that happy stuff, then that's great. Use that and it'll solve your problems. Especially if there is some sort of NFS provisoner that Kubernetes supports.
The only place were I would say it's a 'Hard No' is if you have some sort of high scalability requirements. Like if you wanted to start some web hosting company or needed to have hundreds of nodes in a cluster. In that case then distributed file systems is what you need... Self-hosted storage aka "Hyper Converged Infrastructure". The cost and overhead of managing these things is then relative small to the size of the cluster and what you are trying to do.
It's scary to me to have a cluster self-host storage because storage can use a huge amount of ram and cpu at the worst times. You can go from a happy low-resource cluster, then a node fails or other component takes a shit, and then while everything is recovering and checksum'ng (and lord knows what) the resource usage goes through the roof right during a critical time. The 'perfect storm' scenarios.
And if that is your central infrastructure, shouldn't it be worth the money?
I do get the issue with having cheap and beefy hardware somewhere else, i do that as well, but only for private. My hourly salary spending or wasting time on stuff like that costs the company more than just paying for an additional cluster with the same settings but perhaps with much less Nodes.
If more than one person is using it, the multiplication effects for suddenly unproductive people, is much higher. Also that decreases the per head cost.
https://cloud.google.com/blog/products/containers-kubernetes...
https://cloud.google.com/kubernetes-engine/docs/best-practic...
Do you want to mention anything more about what you're hoping to get out of hierarchy? Is it just a management tool, is it for access control, metering/observability, etc...?
Thanks, A
Since there are no character limits to worry about here unlike Twitter, better to put up the full URL so the community can decide for themselves if the domain linked to is worth clicking through or not.
Friendly docs: https://docs.google.com/document/d/1R4rwTweYBWYDTC9UC-qThaMk...
Code: https://github.com/kubernetes-sigs/multi-tenancy/tree/master...
What kind of system have you seen where this isn't true?
(My apologies if we've chatted about this before in another venue, I'm losing track of whom I've already talked to)
https://cloud.google.com/kubernetes-engine/docs/best-practic...
It's a living product which comes with Terraform modules. We introduced various features to enable doing Multi-Tenancy as well (and more on their way!)
We do recommend robust configurations for production setup (e.g. dev, staging and production) however you can certainly squash and skip it if not necessary.
Thanks for the feedback though. We'll consider adding such notes explicitly.
I am trying to do that. Where would you suggest? Throughout the comments on this post, when people suggest namespace, pod or node level separation you ask them to PTAL and read the link which suggests the single-tenant cluster-per-project approach (that is under the Multi-tenant cluster, confusingly.) The link you sent talks about cluster-per-project, which is not multi-tenancy as I understand it. Perhaps a different name would be less confusing (robust federated cluster administration?)
This "multiple" multi-tenant clusters part isn't coming through. Please do jump into "Securing the cluster" section to cut corners and learn what to do in a single cluster. We're fixing the sections to avoid the confusions. Thanks for the feedback!
https://cloud.google.com/kubernetes-engine/docs/best-practic...
[1] https://github.com/kubernetes/minikube/issues/3207, https://github.com/docker/for-mac/issues/3065, https://github.com/docker/for-mac/issues/3539, etc; there must be dozens and dozens of these
Swarm, in comparison, is much friendlier (and you can use it for dev/test across multiple machines just fine)
Nice.
What about clusters that are used for lumpy work loads? Like data science pipelines? For example, our org has a few dozen clusters being used like that.
Each pipeline gets its own cluster instance as a way to enforce rough and ready isolation. Most of the times the clusters sit unused. To keep them alive we keep a small, cheap, preemptive node alive on the idle cluster. When a new batch of data comes in, we fire up kube jobs which then triggers GKE autoscaling that processes the workload.
This pricing change means we're looking at thousands of dollar more in billing per month. Without any tangible improvement in service. (The keepalive node hack only costs $5 a month per cluster.) We could consolidate the segmented cluster instances into a single cluster with separate namespaces, but that would also cost thousands in valuable developer time.
I don't know how common our use pattern is, but I think we would be a lot better served by a discounted management fee when the cluster is just being kept alive and not actually using any resources. At $0.01, maybe even $0.02, per hour we could justify it. But paying $0.10 to keep empty clusters alive is just egregious.
It will be interesting in any case to see if DigitalOcean and Azure are going to follow suit! I'd be very surprised if they do, (but I've also been wrong before, recently too.)
https://cloud.google.com/kubernetes-engine/docs/best-practic...
We don't recommend using node pools for isolation.
https://github.com/GoogleCloudPlatform/gke-enterprise-mt
Please give us feedback there in case you hit any issue!
PTAL at https://www.youtube.com/watch?v=6rMGRvcjvKc
I guess they realized they couldn't make cluster management MT.
https://github.com/rcarmo/azure-k3s-cluster (this is an Azure template that I use precisely for testing that kind of workloads - spinning up one of these, master included, takes a couple of minutes at most).
(full disclosure: I work at Microsoft - Azure Kubernetes Service works fine, but I built the above because I wanted full control over scaling and a very very simple shared filesystem)
I'm not sure what your usecase is that you would choose gke and you are worried about 300$ per month infra costs.
For Corp we use gke. For private I use selfhostet k3samd for our startup asuper cheap digital ocean cluster.
That feels like the wrong attitude.
If your kubernetes cluster is part of your core infrastructure, then 300$ more or less should not be an issue at all (not to say that i think 300$ is nothing).
That should not mean that you should waste money but often enough, if you buy cheap and your hardware breaks and your time&material costs much more then what a better hardware would have cost, then you wasted money by buying cheap.
Unfortunate with IT products, there are certain things which are not directly visible: Like how secure is your product. GCP offers 2FA, Digital Ocean does not. How much money is it worth to you to have your whole infrastructure protected by 2FA? For me in a business context, non 2FA would be a no go.
Digital Ocean definitely supports 2FA[1]
[1]: https://www.digitalocean.com/docs/accounts/security/2fa/
https://aws.amazon.com/blogs/containers/cost-optimization-fo...
>"The EKS control plane is the easiest to understand with a fixed cost of $0.20 per hour."
https://aws.amazon.com/eks/pricing/ is the up to date page. https://aws.amazon.com/blogs/aws/eks-price-reduction/ is the announcement of the price cut.
After seeing what they did to Google Maps and Api.AI / Dialogueflow jumped from free to 5k$ overnight - just can't trust them.
There is a billion dollar company on the horizon for whoever can best commoditize bare metal with an apple-esque usability model.
How is Google different from other cloud poviders in terms of vendor lock-in?
For AWS or Azure I developed way more trust over time - could be subjective - but also could be that there is a reason Google is distant 3rd in the game.
AWS _still_ has SimpleDB kicking around. That has been retired as a product for more than 8 years now.
I haven't seen Azure do similar, either, but I haven't paid as much attention to them.
I used to recommend GKE as the best way to get started with K8S. With this price change, that advantage is gone. This price change kills off any advantage Google had for getting started in the cloud. Their enterprise-unfriendly habits have already killed off any advantages Google had for established, larger customers of cloud computing.
With this fixed fee model, the change will barely make a difference (== Google revenue) for the large customers who can spare the money, but will create a significant entry barrier to that side project / super-early stage that considers getting hooked on GCP, specifically GKE.
Then again, not my decision to make.
There is no billing protection (which could make you very poor very fast) and every service has a certain cost and quality which is just not feasable in the beginning.
Even GKE with its free kubernetes master does block a lot of resources on the nodes: https://cloud.google.com/kubernetes-engine/docs/concepts/clu...
Also a ton of great features on gke you will probably never use if you are too small. It is so much cheaper to just get cheap hardware somewhere and put your own k8s onto it if you have more time then money.
Even on Digital Ocean you have the load balancer problem: you need to use the provided and also 'costly' LoadBalancer service. There is only one hacky way to prevent it by exposing your ingress on the host and mapping that one ip but then you loose all the self healing stuff and loadbalancing capability.
I mean, yes? You can either build your own for “free”, or pay for the value-add features DO provides that you have described. I don’t see where the problem is.
You can set billing alerts that will project your monthly budget every hour, and send you an alert when it's projected to be exceeded.
IMHO your claim (that there is an entry cost barrier) is the opposite of reality. AWS and Google have brought incredible power and choice to developers starting at zero initial cost.
The fundamental issue with setting a limit is it's technically infeasible to decide what to do when it's exceeded. They have no way of knowing what assets to terminate. The way to avoid what you describe is to shut off access to APIs that you don't want to use, and keep your credentials safe.
Then again, I know enough about AWS and how to control my cost.
The amount of management they make you do is amazing. Need to upgrade the kernel on your nodes? Fill out a five page CloudFormation workflow, and if you copy-pasted anything wrong, your cluster will just randomly work strangely for no reason. This is what they consider "fully managed"! (There are also lots of neat bugs, like if you create a cluster in the web interface while logged in with SSO, all credentials necessary to access the cluster will disappear in an hour. By design, apparently. So even though they have a web interface, you have to create a role account and use the command-line.)
It's really a wondrous product and shows how complacent you can be when you're in first place.
Everyone's having EKS cost problems while I'm just sitting over here paying nothing for ECS control planes.
https://aws.amazon.com/eks/pricing/
Turned out to be irrelevant. Following their instructions I couldn't get EC2 runtime hosts to attach after a couple hours of fucking around (which is my standard for, 'is this mature enough to use'), while with ECS I hit one button and was up an running. Wasn't a hard choice when we started dockerizing (especially since I could simplify most jobs even further having dev use Fargate, albeit at a premium).
EKS struck me as a feature parity product, not something you'd actually use.
1). CEO with delusions of grandeur who thought that Amazon was a direct competitor to their business and should not be given money.
2). Projects that used Kubernetes.
Two is the only type of project which doesn't result in tens of millions of dollars wasted.
I mean, look at Netflix.
In fact, Netflix helped build AWS. There were daily long-term scheduled meetings with AWS developers to review new features and versions.
In addition, the policy from mgmt. was that they were fine being mono-cloud, since if they changed their mind, they had the engineering resources to migrate whenever it made sense.
Source: worked at Netflix.
If I'm running a small cluster, I can use a small VM for the master node(s) and get my control plane costs down much lower than $73/month.
For cluster per customer architecture, would you be able to look into https://cloud.google.com/kubernetes-engine/docs/best-practic... to see if there is anything useful? We understand changing the architecture isn't easy at all and we'd love to know how we can help.
https://news.ycombinator.com/item?id=22487726 https://news.ycombinator.com/item?id=22487110
What I can do is to lower the bar for aggregating clusters with the investments we've done so far. Hope that makes sense.
They are 3rd for me, I will take AWS or Azure long before I would take GCP. Hell for some projects I would even take Digital Ocean or Linode over GCP.
That's because they are aware of their position as a distant fourth place. They just (well last quarter) allocated an additional $2 billion to try and get a foothold in the enterprise, shook up the management team, and is on a spending spree as well.
Because google search is so shitty I couldn't find the $2B announcement (though it was on HN at the time) but here's a recent article on the general subject https://www.fool.com/investing/2020/02/27/google-buying-way-...
That's a concerning take on business decisions on two levels.
I'm sorry, but first choice is AWS, second is Azure (mostly because retailers don't like AWS), and third is actually GCP.
Aren't they the third choice, after AWS and Azure?
For a business I prefer a company that starts with higher prices and then only lowers them to one that may increase them at any time.
AWS lowering compute costs is fairly largely shared, but I am curious if anyone has compiled a list of the cloud providers (AWS, Azure, and GCC) increasing the costs of services.
Makes me rethink whether I want to do any business with Google anymore.
AWS really strongly focuses on gaining customer trust, and they will only lower price, and never increase price. They won't turn off things until the last customer stops using it (they might stop new customers from onboarding)
I did not enjoy working for AWS so I left pretty fast, but some of the customer obsession there really impressed me.
with no IPv6 option (of course)