The incident impact (nodepool upgrade issue) seems to be matching the speed of mitigation rollout. One does not want the cure to be worse than the disease; roll forwards should be slow unless the impact is high (and even then, it should be a rollback/freeze rather than fast roll forward).
Wow, I literally did a full cluster version upgrade last night without knowing about this. I would have delayed the upgrade if I had known GKE was failing for "a small number of customers"
I wish cloud providers would just communicate outages to services I use like this to me!
Don't put this in a special channel just for vendor incidents. Hopefully you have a channel for each vendor tool where you have a vendor representative present. You put that bot in there. It's much more likely to be noticed and much less likely to be a channel that everyone ignores because it's SNR is to low.
Yeah, we did do this when we got large enough for a shared slack channel with aws. But for most orgs, just having this piped to your alerts channel is good enough.
I think they mean that companies rarely update their status pages to reflect reality (for instance, AWS outages are rarely shown on their own status pages). This is often by design, company policy, or a desire to save face.
And it's so incredibly dumb. Companies need to get it through their thick heads that this is so incredibly short-sighted.
Not once has a status page that's devoid of information or slow to update ever saved face. I am far more annoyed to have to continue to verify "no, it is indeed that your service is down, not mine" and then file a support ticket. I am triply annoyed if the response from support is "ah yes that's a known problem and we're working on it" — known, and you just didn't bother to communicate.
I miss the days when Github had graphs. Even if they simply hadn't had the time to put a message on the page, you could tell from the graphs that it was Github. But even with "more information" that some PM might not like being put out publicly, Github felt more reliable & stable in those days.
At the end of the day, no amount of political gamesmanship will save you from having to actually run a reliable service, and gamesmanship just makes it more likely I'll ascribe false positives to your service, further lowering my perception of its reliability.
It's so watered down that the "AWS" emoji in our Slack instance is literally a meme of the status page.
not sure how this could work, as some qa/sre would have to be paid even more in order to effectively work against this type of falsification. there would have to be a very strong incentive for the company to do that.
See all those green there? Once it started becoming "monitored" by VPs instead of the software engineers on call, they started to become political. I bet there are several sev2s happening for several of those services even as we speak but it still shows green to the outside observer. If one has access to the actual metrics for those services, i bet we would see a different story than what is shown on the "status" page.
There was something probably here a few days ago to the effect of 'Their 9s are not your 9s'. Like yes, their status is showing an error rate of less than .00001%. However, all of those errors are being generated by your 5 instances that are completely down.
In general, they seem bad at communicating relevant information. Just looking over at emails from Google, every single one of the last 10 emails they sent me was not relevant to me specifically.
> "[Important notice] Tax changes in Nepal" (yet I have never made a sale in Nepal)
> "Secure your Google Admin account with these best practices" (yet I already do all those things)
> "Preparing for the upcoming Google Ads change on October 31, 2023" (it's all about mediation waterfalls, and I'm 90% sure I don't use them, and 100% sure I don't know what they are)
I will say CircleCI’s dashboard makes it impossible to NOT know there is an outage going on by putting it in the sidebar. Unless it’s collapsed you’ll be aware of everything breaking (that they report) to the point where it feels like everything is breaking all the time
If you've ever worked at a cloud provider, then you know everything is breaking all the time. The good ones are just able to hide it most of the time/for most customers.
Honest status page are a really really hard problem.
If your connector between your status monitor and the service breaks you'll have some subset of users panicking and causing problems (or asking for refunds for outages) when the service was up the entire time.
3rd party services are the only ones that you'll get a "more honest" but not always correct view of what the actual status is.
When an incident is declared, have someone tasked with determining customer impact. If the impact radius is greater than a handful of customers, declare a public incident. If customer communication is made a priority, then you can actually have a helpful status page.
Where I work, just about any non-false-alarm incident ends up on the status page in a timely manner. There's nothing stopping the likes of AWS from doing the same except for culture.
Exactly. It's the kind of thing where one person managing it and deciding whether to post updates is probably gonna work better than anything automated.
Well, maybe not at AWS scales though, if they have thousands of everything =/
I'm not talking about scraping, I'm talking about manual usage.
> You can easily check it manually.
You cannot. Twitter's site is plagued by redirect loops. If you work around those, these days /search just redirs to the login page. You can view single tweets, but there won't be any replies. (I have no idea if the site formerly known as Twitter is still rate-limiting views, or if they canned that.)
It is unusable if you're not actively logged in, and some of us have no desire to give away a phone number just to see AWS's true status.
I haven't had that comms issue with Google but I have to say, even though I prefer GCP to AWS in terms of user friendliness, it is far too often the case that you find exactly the solution you need only to learn it's deprecated in favour of a less useful alternative.
There is a 'Google Cloud Service Health Updates' Slack app that has notifications about this incident. Here's what it looks like on our channel:
8:18 PM
APP UPDATE: Global: Google Kubernetes Engine Nodepool Upgrade Failures
Incident began at 2023-10-02 11:29 (all times are US/Pacific).Summary: Global: Google Kubernetes Engine Nodepool Upgrade Failures
Description: A mitigation has been rolling out and we are assessing its effectiveness. We will provide an update byTuesday 2023-10-10 12:00 US/Pacific with current details.
Diagnosis: A small number of customers are experiencing failed nodepool upgrades. Customers experiencing this, may see "Internal error" in Google Cloud Console. Retrying is suggested but may...
There are quite a lot of alerts about various issues.
It does and most people that used Nomad were very positive about it. But with the turn that Hashicorp have taken I'm not sure that it now stands much of a chance.
Seconded. Hashicorp Nomad has been a breath of fresh air for doing HA deployments for my workloads. Getting a small cluster setup to self host Nomad is so easier than Kubernetes and defining workloads is much easier to understand too IMO.
The only negatives about Nomad is the Hashicorp license drama that has happened recently and persistent storage can be a pain in the ass.
They pulled the rug out on a lot of people and people tend to hold grudges. Community edition is peanuts compared to using an open source tool you _could have_ fixed your own bugs with by creating PRs or adding features with PRs. I used to be a huge hashicorp fanboy... _used to_.
Just to clarify, you can still access the code, submit PRs to add features, etc. The only difference is that during a fixed time period (4 years) you cannot use Nomad to compete with HashiCorp. If you want to do that, you can ask for a license.
I love Nomad but having used it in two different roles and now invested time to understand k8s properly, I would absolutely not recommend it.
Nomad is simple on the surface and could have been a great tool - but it is basically unusable on its own without tighter integration with Hashi’s own tools (eg Vault). Configuring and maintaining all those things is nontrivial and ends up being more annoying (for a lesser end result) than just using a fully managed kubernetes cluster like GKE
You'd imagine that any scheduler that's well integrated with Vault would have a huge advantage over other ones. Surprising that it's not like that for another product from the same company.
Maybe https://kamal-deploy.org could be worth a try there, as 37Signals almost moved all their services from Kubernetes I'd say it can be called production grade.
HashiCorp Nomad is a perfectly easy and flexible orchestrator (not only containers, also VMs, random binaries, Firecracker, WASM, etc.) that is production grade. It has a number of advantages over Kubernetes (being drastically easier to deploy, use and maintain; having integrated native templating with logic; being much nimbler but also being able to scale much more, etc. etc.) and it's a very good choice in many scenarios. Of course there's also disadvantages, most notably ecosystem size, but often that doesn't matter. Kubernetes is great and extremely powerful, but also very complex and straight up overkill for many organisations.
We're using Nomad, and appreciate its comparative simplicity (we launch plain binaries created with Nix), but there are still basic features missing that can create struggles. The ones I hit are:
* It can't work down batch jobs in priority order [1] -- a feature even the ultra-old university cluster schedulers had
* Sometimes job starts can be in arbitrary order, in the API and the UI, making it difficult to find the most recent ones [2]
* Nomad seems to store pretty much everything in its Raft state, making it extremely slow to startup and operate (including GUI) if some rogue process created 100k allocations. Cleaning that up seems to require nuking the Raft state, thus also losing all info about all jobs.
I don't know how well Kubernetes does in these areas.
Someone I know once worked for the company maintaining etcd at the time and... they were appalled at how it was underpinning millions of dollars of VC dreams on it, implicitly via kubernetes, with the flaws and downsides, and operational complexities it has. And now to hear you say this of Nomad and Raft.
I guess, my take away at this second is to really be skeptical of hype that I don't have years of operational knowledge and experience with. Which, to be fair, regarding my current tech du jour (well, 9+ years), I have a list of complaints that could span a short novela.
I've run both. First k8s is free and nothing beats free but that's not about it.
Nomad's mental model is hard to grasp. Or let's say is not as elegant. It's is nowhere near k8s. And k8s at least from a usage standpoint can be explained in under ten minutes. Pod, Deployment, Service, Ingress and you're pretty much done.
I use AWS ECS (sometimes with ECS Anywhere to use my own servers + ECS's control plane) for orchestration where I can.
It's a little surprising how many folks are unaware of the non-Kubernetes orchestration options. Going full Kubernetes is rarely what organizations need. Getting CI/CD -> containers on servers and being able to configure the resources containers need, having the placement handled, etc. gets most of these folks 90% of what they want with much much less complexity.
We moved from DigitalOcean to ECS 2-3 months ago and it has been rock solid. Highly recommended. There is some initial setup required with container registries and wrangling AWS's vocab but once set up, push to master, GitHub Actions builds and deploys.
Does ECS support mounting configuration files (without needing the configuration file to be on the host)?
Being able to mount secrets and configmaps into the container file system (without having to modify the container image to provide an entrypoint) definitely seemed to be one major advantage of kubernetes over ECS a few years back.
You can setup something to mount configuration files stored in say S3 -> into an EFS volume that you attach to all ECS tasks, for example.
The problem with ECS is that everything, including service discovery, is an integration with another AWS service. Making it even more difficult to ever migrate, but it does support much of what folks use Kubernetes for.
There is, you create a load balancer, an autoscaling group, put an ubuntu image in it, set the userdata to install docker and docker-compose up your desired containers.
* Instance refresh to deploy new images.
* Every so often update your AMI to patch.
* Create an RDS database and hook the instances up to that if you need such things.
It's really hard to get more boring but works than this. It can be set up in a few hundred lines of terraform. I promise the code for your entire AWS account will be less than just the k8s YAML.
If your workload is extremely variable, something like k8s with auto-scaling makes sense.
But a lot of applications (maybe the majority?) have very predictable loads and you don't always need new services.
When you're at the size of large corps with hundreds/thousands of services, then k8s is amazing. If you're not even close to that number, it's very likely premature optimization.
There is AWS Autoscaling, ECS, EKS, Hashicorp Nomad, Fly.io and the few more.
However I would ask the question (same question I have been asking since the first appearance of k8s) what do you need so much that it justifies this complexity?
Implementing a simple autoscaling system or using the provided solutions like AWS ASG was and is results in more reliable infra.
If you want to use cutting edge there Lamda function that most cloud vendors offer. That is also less complex option.
To be honest, I'm pretty disappointed that this thread doesn't have some attempt at a root cause now that two major managed providers are having extended outtages linked to upgrades.
To think, I once was so in love with it. This kind of crap, and imagining the live-site chaos makes me so glad to have noped-out when I did. With my modest lifestyle, I probably passed on life changing money but I also have 1/10th the stress and anxiety I used to.
We've been stuck in this state for all 9 days. We've filed tickets, etc, but no resolution has come about yet. Just re-tried yesterday, still not able to update nodepool.
The incident dashboard does mention creating a new node pool with the new version instead of upgrading - as a workaround. Curious to know why this was not an option for you.
Even if there are stateful workloads, one has to be prepared to migrate at some point. Nodes can crash, become unreachable, etc. - workloads have to be able to survive such events.
Edit: But I see your point - if their stateful workloads are not "migration/K8S-incident" ready, it's hard to suddenly build that overnight.
In Google terminology is a "mitigation" the same as a solution? I read it as "Yeah, we still have no idea how to fix this correctly, but we have applied a temporary work-around".
I can’t speak for google, but in common parlance, mitigation simply means the problem isn’t affecting you any more. Doubtless they will perform a full post-mortem, identify the contributing factors, and fix them over the coming days, weeks, quarters.
Edit: “we are assessing its effectiveness” lol, yeah that sounds more like they they are throwing something at the wall to see if it sticks.
Disclaimer: Google SRE working in Cloud but not related to the GKE product.
Mitigation can vary: an additional firewall rule to stop certain traffic, a rollback to known good version, temporarily redirect traffic away from impacted data center. A mitigation is part of an incident response mainly focused on stopping the pain to whoever are impacted, it may not always be a long term solution.
It more or less means actions have been taken to reduce or eliminate the problem, but it is not the same as the long term fix. Mitigation might be 'give more CPU' to the system, despite the performance regression still being in the code etc.
May be this a hated take but just wondering - The place that pretty much invented cluster orchestration and reinvented it as k8s is having problems upgrading it.
What chance a bunch of poor sys admins stand running bunch of k8s clusters for a mid size company I wonder.
Every time I think of deploying (self managed, have done full stack) it for something mission critical, this upgrade scenario simply makes me rethink it altogether.
And even managed k8s has no guarantees and if managed is to be the option, nothing beats ECS in simplicity and smooth operation at certain scales.
PS: Full stack K8s means ingress controllers, DNS auto registration, GitOps , logging, monitoring, CI/CD and all the bells and wistles including a management UI behind oauth etc.
I am not sure what "old and busted way" you are referring to, but k8s does have value, particularly when we are not upgrading k8s itself
It would be good to think of the occasional downtime during a significant upgrade (once in 6 months?) separate from the ability to automatically replace failed nodes, scale up and down, having all the k8s magic.
I would imagine using a load balancer in front of multiple servers across several datacenters coupled with a stateless design that allows traffic to flow freely across either of them.
there are other variations of this, but this seems the most obvious. blue/green deploys and the like.
I think it's definitely overlooked. You should always have the ability to spin off a new cluster and run your workloads on that. At a minimum as a disaster recovery scenario.
Check out cluster-api. Complete zero downtime cluster upgrades is as simple as bumping the version and machine image name in the cluster manifest - or ClusterClass if you run classy clusters
While I see your point and agree, there's also the view that the people who invented the tech are simply different than the kind of people who Google hires nowadays - pure algo focused LeetCoders with zero-to-little operational experience and little interest in even learning how to run and maintain software.
The design of K8s is ridiculous. The fact that you're forced to upgrade every 9 months is even more absurd. It's "opinionated", but the "opinions" are those of one of the largest companies in the world, who hire the most engineers in the world, and everyone else is supposed to operate like that now.
K8s is a great example of a "tech jobs program": technology that is absurdly complicated to the point that you have to pay someone lots of money to make it keep working, but you can't afford not to pay for this, because everyone has decided to be stuck on it and now there's no alternative (and probably never will be)
No, it is not. K8s is an example of how it's impossible to be a jack of all trades and retain simplicity. If a tool seeks to be all things to everyone, then there's no sweeping the inherent complexity under the carpet.
Full-coverage solutions to complex domains are themselves complex by necessity - see SAP, Salesforce, Oracle, Windows backwards compatibility, or try writing a parser/stemmer for a spoken language that evolved naturally :)
The complexity of an implementation is a different thing than the complexity of a design, and each determines things like how frequently it needs maintenance or how difficult it is to use. As usual, the devil's in the details.
You can make something that has complexity without making it complicated or expensive to maintain. Take cars for example. There are many cars today that have a barrage of complex features, yet don't have costly maintenance bills and aren't difficult to use. That's because they have been engineered purposefully to reduce that complexity, and because the people that make them are masters of their craft, trying to create a premium product. Materials engineers, structural engineers, sound engineers, mechanical engineers, and more, all work toward the best design possible, to implement the most functionality with the minimum number of defects. They can tell you how likely it is for the paint to begin chipping on a new tailgate due to potential contamination in an alloy. And they'll use that knowledge to make it less likely that you'll have to take your car in for repair.
The software world today - especially where customers are an afterthought (such as backend software) - does not make attempts to improve its design or implementation to reduce the need for repairs. Instead they just "manage" it for you, doing constant maintenance on the constantly breaking vehicle they drive for you, while charging you a subscription fee. (the auto industry noticed how profitable this is, and is now doing the same wherever it can)
K8s is a great example of terrible design and half-assed implementation. It was not designed to be easy to install, configure, operate, upgrade, or maintain. It was designed the way a company designs software for its own internal use: start with some "opinions" that are more about inflated intellectual ego than practicality, throw some shit together that "basically works", and keep introducing breaking changes so it can never be operated continuously without requiring constant reworking and changing how it's run.
If a car were produced like this, nobody would buy it. But it's handed out "for free", so we all just accept it and end up driving this janky-ass school bus that constantly needs its components changed while the bus is running. But you can't ask a regular person to change the parts on a bus mid-cross-country-tour, so you have to hire a mechanic to sit there on the bus. It doesn't have to be that way, it just is because the people who made that bus don't give a shit about hiring a mechanic, they have 10,000 mechanics already.
IF K8s were a vehicle, it wouldn't be a car; it would be a multipurpose monster that could be used by some for getting groceries, or configurable into an 18-wheeler for interstate trucking, or as a off-road missile launch platform and somehow also amphibious/submersible. That.... thing will be complex because it has to be able to achieve a variety of tasks.
Also, cars are still complex beasts that have achieved economies of scale (I've had "repairs" where they just replaced an entire assembly instead of fixing a smaller component that was broken). I'm sure there are repair forums were mechanics complain about cars more than HN complains about K8s.
> K8s is a great example of terrible design and half-assed implementation
So where are the superior competitors by the master craftsmen with good taste, then? People willingly choose Kubernetes because the alternatives (third-party, homegrown or manual processes) are worse. Ops, configuration and orchestration management has been steadily getting better over the last few decades, but there is no silver bullet. If you know of one, I'd be happy to try it out and save a lot of money.
I agree that the monolithic nature of K8s is a bad idea and will always lead to more problems.
The competitors don't exist because incumbency prevents progress. Nobody wants to write a competitor to K8s, because it would be impossible to get people to switch away from K8s, because K8s is a billion-dollar ecosystem. Thousands of companies now exist solely to service it and build parts for it. None of them want to support two such systems, and none of them want to completely reinvent themselves for some other platform. And from the user's perspective, it's not like getting into a different car, it's like learning to drive an entirely different vehicle, so they have no incentive to switch.
This isn't just a tech thing, this is a universal thing. Incumbents suppress innovation and competition without even trying to.
Aren't you describing the perfect scenario of innovator's dilemma? Entrenched incumbents with a legacy and expensive technology, just waiting until a more nimble player comes along and disrupts the entire ecosystem from the bottom up. See the microcomputer, the PC, digital cameras, iPhone, Android, and so on.
Except, we haven't seen any of these nimble players gaining any traction. Maybe they're out there, or maybe they're yet to be created. Or maybe the complexity is an integral part of the problem they're supposed to be solving.
> The competitors don't exist because incumbency prevents progress. Nobody wants to write a competitor to K8s, because it would be impossible to get people to switch away from K8s, because K8s is a billion-dollar ecosystem.
Angular was an incumbent and still got toppled - albeit by a project from another tech giant (React). The question is - why hasn't Microsoft or Amazon incubated superior, simpler alternatives to K8s that works best on their respective clouds? The financial incentives are clear to me, but there's a big hole where that product ought to be
The biggest foot gun and most complicated part IMHO is networking.
It is a blackbox with many competing implementations and each implementation even differing in implementation details within it's own versions.
In theory, it seems reasonable to say that K8s networking is an implemention detail which you shouldn't bother about but in practice, it's going to bite you because of some misconfigured component somewhere someday and you would have no idea what to make of thousands of iptable rules or BGP routes or even least inspectable eBPF mini programs.
Most application developers don’t give a rats ass about how 2 nodes are connected as long as the pods can speak to each other. Throw these guys in a devops team and give them the task to setup a 2 node k8s cluster and guess how much consideration they will pay to networking. Correct, zero. Except you can’t, because you have to choose a CNI plugin or choose a distribution that has made an opinionated choice for you.
To lower the barrier of entry. A default installation must become a lot more opinionated with much less options and more sane and safe default.
It used to be straightforward when Google Cloud just offered everyone the same version of GKE essentially everywhere. It was one of not-too-many services that are vanilla and identical in pretty much every region. But, newer offerings have fractured things a bit (sovereign cloud, Google Distributed Cloud, FedRAMP, Tencent, etc) and I can see why it would become problematic to adequately test and keep everything in sync while also not breaking interfaces and/or downstream services. Also, Google used to allow customers essentially 0 say in when upgrades to services were applied. Now, as part of many vendor security audit processes, it's a hard stop requirement that customers are able to test new revs before applying to prod ... and GKE, like GCE, is one of those critical services.
What bugs me most about GKE is that incidents seem to span a whole of datacenters simultaneously (source: see zones affected by previous incidents), and you are often left with no option except wait for a resolution. Often there are issues like networking problems affecting almost the entire global GKE control plane. Whatever happened to resilience by isolation?
What's really unfortunate is that it feels like not choosing k8s is career suicide for SREs/DevOps/platform engineering.
I chose Nomad for my current role and it works fantastic.
I used ECS in my last role and it was also fantastic.
I'm currently gently looking for a job and the market is brutal. I'm not even getting responses back from applications and I have ten years of experience. Every job posting I see requires deep k8s knowledge and hands-on experience with some ancillary k8s project like ArgoCD or Flux. I'm actually thinking about downplaying non-k8s experience on my resume and just floating the k8s that I do have straight to the top because, at the very least, it'll get my resume looked at.
If I was going to go back in time, I'd pick k8s instead of Nomad or ECS even if they weren't the best contextual choices because, hey, I gotta' put food on the table.
Quick tip: You can still put your target keywords where you have zero to minimal experience in your resume/CV, just put it in an honest statement describing how much experience you have. In your case, you could write, for example, "X years of experience in HashiCorp Nomad and AWS ECS (Kubernetes alternatives for container orchestration)". That should put your resume/CV through the keyword filter without lying about it.
Noob here with some meta-questions about developer and operations complexity.
From an outsider’s perspective, it looks like in a 2x2 matrix of developer simplicity/complexity and operational simplicity/complexity, the current patterns all seem to be heavily biased for developer simplicity/operational complexity.
1. Is this assumption correct?
2. Does optimizing for another quadrant: developer complexity / operational simplicity make sense?
My intuition is that complexity in code can be managed far better than complexity in operations. Developers have abstractions, reusable libraries, unit tests/integration tests, etc. There may also be weird efficiencies that arise from having developers deal with some of these problems right from the design stage.
It seems kubernetes takes a problem and pushes it to fully to operations.
Is there a solution that takes this problem and turns it into a developer problem?
Infrastructure is, or at least should be, code as well. And as it is, you can write tests for it all the same!
However, writing those tests is incredibly hard. It doesn’t matter if you approach it from a dev or ops angle. The system under test doesn’t only have side effects, it is side effects. You also cannot mock most things (in my opinion…), as that is either also very hard to instrument or straight up removes the test usefulness altogether. Imagine mocking the AWS management API for your integration tests. Not possible.
So what Dev calls integration or e2e tests, ops calls the dev environment. Works, but differently to how devs would do it. I don’t see an alternative.
Next, as much as knowledge siloes are being heralded as evil, they exist. Undoing siloes altogether isn’t possible. You’ll end up reerecting them elsewhere. Devs have their skill sets, and ops isn’t part of that. The opposite is also true. The intersection can be substantial, but never enough to have dev to it all alone. I don’t think that’s a bad thing either.
The difficult part about Kubernetes for newcomers is that it tackles every problem at once. Large scale best-practices you would previously sweep under the rug are now things you can’t avoid addressing. Centralized logging, secret management, certificate rotation, rbac, liveness probes, rolling deployments, etc. A lot of people complaining about the complexity are doing all those things by hand instead - “just ssh into the server with a shared password, remove old logs and restart it lol”.
It’s great if you actually need all of this and consider those practices essential complexity. If you don’t - it feels like someone is shoving accidental complexity down your throat.
Turning it into a developer problem will not hide all these things or make them more manageable. It means you will reinvent k8s yourself, poorly.
I'm also incredibly annoyed at them displaying time in PDT. I genuinely don't understand why they decided on that instead of doing something normal like UTC or detecting my timezone. Especially annoying every six months because Europe and the US don't do Daylight Savings Time changes at the same time, so for a week or two there's an additional hour I have to account for.
Creating a new node group works which is super easy on GKE so it's pretty much a non issue. Definitely frustrating but not as bad as it sounds at first brush
124 comments
[ 2.9 ms ] story [ 158 ms ] threadI wish cloud providers would just communicate outages to services I use like this to me!
Slack let’s you subscribe directly to an RSS feed using /feed.
Sounds like a good approach nonetheless
Not once has a status page that's devoid of information or slow to update ever saved face. I am far more annoyed to have to continue to verify "no, it is indeed that your service is down, not mine" and then file a support ticket. I am triply annoyed if the response from support is "ah yes that's a known problem and we're working on it" — known, and you just didn't bother to communicate.
I miss the days when Github had graphs. Even if they simply hadn't had the time to put a message on the page, you could tell from the graphs that it was Github. But even with "more information" that some PM might not like being put out publicly, Github felt more reliable & stable in those days.
At the end of the day, no amount of political gamesmanship will save you from having to actually run a reliable service, and gamesmanship just makes it more likely I'll ascribe false positives to your service, further lowering my perception of its reliability.
It's so watered down that the "AWS" emoji in our Slack instance is literally a meme of the status page.
I am sure that counts. Probably tied into someone's bonus as well.
very likely not just one.
not sure how this could work, as some qa/sre would have to be paid even more in order to effectively work against this type of falsification. there would have to be a very strong incentive for the company to do that.
See all those green there? Once it started becoming "monitored" by VPs instead of the software engineers on call, they started to become political. I bet there are several sev2s happening for several of those services even as we speak but it still shows green to the outside observer. If one has access to the actual metrics for those services, i bet we would see a different story than what is shown on the "status" page.
In general, they seem bad at communicating relevant information. Just looking over at emails from Google, every single one of the last 10 emails they sent me was not relevant to me specifically.
> "[Important notice] Tax changes in Nepal" (yet I have never made a sale in Nepal)
> "Secure your Google Admin account with these best practices" (yet I already do all those things)
> "Preparing for the upcoming Google Ads change on October 31, 2023" (it's all about mediation waterfalls, and I'm 90% sure I don't use them, and 100% sure I don't know what they are)
https://cloud.google.com/service-health/docs/overview#how-pe...
When you suspect things are broken, check X (FKA Twitter).
Other devs will be talking about it before there's an official status page.
If we as an industry can't think of something better (cough honest status pages cough) … can we at least transition these tweets to Mastodon.
If your connector between your status monitor and the service breaks you'll have some subset of users panicking and causing problems (or asking for refunds for outages) when the service was up the entire time.
3rd party services are the only ones that you'll get a "more honest" but not always correct view of what the actual status is.
Where I work, just about any non-false-alarm incident ends up on the status page in a timely manner. There's nothing stopping the likes of AWS from doing the same except for culture.
Well, maybe not at AWS scales though, if they have thousands of everything =/
You can easily check it manually. Search for "EBS" and see a bunch of people talking about EBS timeouts or whatever. That was what I was getting at.
But yeah scraping is harder now.
> You can easily check it manually.
You cannot. Twitter's site is plagued by redirect loops. If you work around those, these days /search just redirs to the login page. You can view single tweets, but there won't be any replies. (I have no idea if the site formerly known as Twitter is still rate-limiting views, or if they canned that.)
It is unusable if you're not actively logged in, and some of us have no desire to give away a phone number just to see AWS's true status.
Maybe it’s time to yolo with a regular container that just restarts on failures, ha…
The only negatives about Nomad is the Hashicorp license drama that has happened recently and persistent storage can be a pain in the ass.
They pulled the rug out on a lot of people and people tend to hold grudges. Community edition is peanuts compared to using an open source tool you _could have_ fixed your own bugs with by creating PRs or adding features with PRs. I used to be a huge hashicorp fanboy... _used to_.
Nomad is simple on the surface and could have been a great tool - but it is basically unusable on its own without tighter integration with Hashi’s own tools (eg Vault). Configuring and maintaining all those things is nontrivial and ends up being more annoying (for a lesser end result) than just using a fully managed kubernetes cluster like GKE
It’s just a bug in something most people don’t use.
HashiCorp Nomad is a perfectly easy and flexible orchestrator (not only containers, also VMs, random binaries, Firecracker, WASM, etc.) that is production grade. It has a number of advantages over Kubernetes (being drastically easier to deploy, use and maintain; having integrated native templating with logic; being much nimbler but also being able to scale much more, etc. etc.) and it's a very good choice in many scenarios. Of course there's also disadvantages, most notably ecosystem size, but often that doesn't matter. Kubernetes is great and extremely powerful, but also very complex and straight up overkill for many organisations.
There is an unsupported community edition
* It can't work down batch jobs in priority order [1] -- a feature even the ultra-old university cluster schedulers had
* Sometimes job starts can be in arbitrary order, in the API and the UI, making it difficult to find the most recent ones [2]
* Nomad seems to store pretty much everything in its Raft state, making it extremely slow to startup and operate (including GUI) if some rogue process created 100k allocations. Cleaning that up seems to require nuking the Raft state, thus also losing all info about all jobs.
I don't know how well Kubernetes does in these areas.
[1]: https://github.com/hashicorp/nomad/issues/12792 [2]: https://github.com/hashicorp/nomad/issues/17742
I guess, my take away at this second is to really be skeptical of hype that I don't have years of operational knowledge and experience with. Which, to be fair, regarding my current tech du jour (well, 9+ years), I have a list of complaints that could span a short novela.
Nomad's mental model is hard to grasp. Or let's say is not as elegant. It's is nowhere near k8s. And k8s at least from a usage standpoint can be explained in under ten minutes. Pod, Deployment, Service, Ingress and you're pretty much done.
It's a little surprising how many folks are unaware of the non-Kubernetes orchestration options. Going full Kubernetes is rarely what organizations need. Getting CI/CD -> containers on servers and being able to configure the resources containers need, having the placement handled, etc. gets most of these folks 90% of what they want with much much less complexity.
- AWS ECS: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/...
- ECS Anywhere: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/...
Being able to mount secrets and configmaps into the container file system (without having to modify the container image to provide an entrypoint) definitely seemed to be one major advantage of kubernetes over ECS a few years back.
See a CDK example:
https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_...
You can setup something to mount configuration files stored in say S3 -> into an EFS volume that you attach to all ECS tasks, for example.
The problem with ECS is that everything, including service discovery, is an integration with another AWS service. Making it even more difficult to ever migrate, but it does support much of what folks use Kubernetes for.
It's a wonderful, observable, manageable orchestration layer that's easily and safely customizable if you're wise enough to treat it as such.
* Instance refresh to deploy new images.
* Every so often update your AMI to patch.
* Create an RDS database and hook the instances up to that if you need such things.
It's really hard to get more boring but works than this. It can be set up in a few hundred lines of terraform. I promise the code for your entire AWS account will be less than just the k8s YAML.
We do this but create custom images that have all their requirements installed via ansible.
Unfortunately there's now a push to move to GKE instead of GCE, for no real reason.
If your workload is extremely variable, something like k8s with auto-scaling makes sense. But a lot of applications (maybe the majority?) have very predictable loads and you don't always need new services.
When you're at the size of large corps with hundreds/thousands of services, then k8s is amazing. If you're not even close to that number, it's very likely premature optimization.
However I would ask the question (same question I have been asking since the first appearance of k8s) what do you need so much that it justifies this complexity?
Implementing a simple autoscaling system or using the provided solutions like AWS ASG was and is results in more reliable infra.
If you want to use cutting edge there Lamda function that most cloud vendors offer. That is also less complex option.
To think, I once was so in love with it. This kind of crap, and imagining the live-site chaos makes me so glad to have noped-out when I did. With my modest lifestyle, I probably passed on life changing money but I also have 1/10th the stress and anxiety I used to.
Edit: But I see your point - if their stateful workloads are not "migration/K8S-incident" ready, it's hard to suddenly build that overnight.
Edit: “we are assessing its effectiveness” lol, yeah that sounds more like they they are throwing something at the wall to see if it sticks.
Mitigation can vary: an additional firewall rule to stop certain traffic, a rollback to known good version, temporarily redirect traffic away from impacted data center. A mitigation is part of an incident response mainly focused on stopping the pain to whoever are impacted, it may not always be a long term solution.
What chance a bunch of poor sys admins stand running bunch of k8s clusters for a mid size company I wonder.
Every time I think of deploying (self managed, have done full stack) it for something mission critical, this upgrade scenario simply makes me rethink it altogether.
And even managed k8s has no guarantees and if managed is to be the option, nothing beats ECS in simplicity and smooth operation at certain scales.
PS: Full stack K8s means ingress controllers, DNS auto registration, GitOps , logging, monitoring, CI/CD and all the bells and wistles including a management UI behind oauth etc.
Sometimes a clean install of fresh cluster and switch over is so much simpler if we can just take a few minutes downtime to cut over.
Complex requirements (sometimes) end up with complex solutions.
It would be good to think of the occasional downtime during a significant upgrade (once in 6 months?) separate from the ability to automatically replace failed nodes, scale up and down, having all the k8s magic.
there are other variations of this, but this seems the most obvious. blue/green deploys and the like.
Booting up a cluster is fine but the real struggle starts from that point onwards.
K8s is a great example of a "tech jobs program": technology that is absurdly complicated to the point that you have to pay someone lots of money to make it keep working, but you can't afford not to pay for this, because everyone has decided to be stuck on it and now there's no alternative (and probably never will be)
No, it is not. K8s is an example of how it's impossible to be a jack of all trades and retain simplicity. If a tool seeks to be all things to everyone, then there's no sweeping the inherent complexity under the carpet.
Full-coverage solutions to complex domains are themselves complex by necessity - see SAP, Salesforce, Oracle, Windows backwards compatibility, or try writing a parser/stemmer for a spoken language that evolved naturally :)
You can make something that has complexity without making it complicated or expensive to maintain. Take cars for example. There are many cars today that have a barrage of complex features, yet don't have costly maintenance bills and aren't difficult to use. That's because they have been engineered purposefully to reduce that complexity, and because the people that make them are masters of their craft, trying to create a premium product. Materials engineers, structural engineers, sound engineers, mechanical engineers, and more, all work toward the best design possible, to implement the most functionality with the minimum number of defects. They can tell you how likely it is for the paint to begin chipping on a new tailgate due to potential contamination in an alloy. And they'll use that knowledge to make it less likely that you'll have to take your car in for repair.
The software world today - especially where customers are an afterthought (such as backend software) - does not make attempts to improve its design or implementation to reduce the need for repairs. Instead they just "manage" it for you, doing constant maintenance on the constantly breaking vehicle they drive for you, while charging you a subscription fee. (the auto industry noticed how profitable this is, and is now doing the same wherever it can)
K8s is a great example of terrible design and half-assed implementation. It was not designed to be easy to install, configure, operate, upgrade, or maintain. It was designed the way a company designs software for its own internal use: start with some "opinions" that are more about inflated intellectual ego than practicality, throw some shit together that "basically works", and keep introducing breaking changes so it can never be operated continuously without requiring constant reworking and changing how it's run.
If a car were produced like this, nobody would buy it. But it's handed out "for free", so we all just accept it and end up driving this janky-ass school bus that constantly needs its components changed while the bus is running. But you can't ask a regular person to change the parts on a bus mid-cross-country-tour, so you have to hire a mechanic to sit there on the bus. It doesn't have to be that way, it just is because the people who made that bus don't give a shit about hiring a mechanic, they have 10,000 mechanics already.
IF K8s were a vehicle, it wouldn't be a car; it would be a multipurpose monster that could be used by some for getting groceries, or configurable into an 18-wheeler for interstate trucking, or as a off-road missile launch platform and somehow also amphibious/submersible. That.... thing will be complex because it has to be able to achieve a variety of tasks.
Also, cars are still complex beasts that have achieved economies of scale (I've had "repairs" where they just replaced an entire assembly instead of fixing a smaller component that was broken). I'm sure there are repair forums were mechanics complain about cars more than HN complains about K8s.
> K8s is a great example of terrible design and half-assed implementation
So where are the superior competitors by the master craftsmen with good taste, then? People willingly choose Kubernetes because the alternatives (third-party, homegrown or manual processes) are worse. Ops, configuration and orchestration management has been steadily getting better over the last few decades, but there is no silver bullet. If you know of one, I'd be happy to try it out and save a lot of money.
The competitors don't exist because incumbency prevents progress. Nobody wants to write a competitor to K8s, because it would be impossible to get people to switch away from K8s, because K8s is a billion-dollar ecosystem. Thousands of companies now exist solely to service it and build parts for it. None of them want to support two such systems, and none of them want to completely reinvent themselves for some other platform. And from the user's perspective, it's not like getting into a different car, it's like learning to drive an entirely different vehicle, so they have no incentive to switch.
This isn't just a tech thing, this is a universal thing. Incumbents suppress innovation and competition without even trying to.
Except, we haven't seen any of these nimble players gaining any traction. Maybe they're out there, or maybe they're yet to be created. Or maybe the complexity is an integral part of the problem they're supposed to be solving.
I guess we'll know in a few years.
Angular was an incumbent and still got toppled - albeit by a project from another tech giant (React). The question is - why hasn't Microsoft or Amazon incubated superior, simpler alternatives to K8s that works best on their respective clouds? The financial incentives are clear to me, but there's a big hole where that product ought to be
It is a blackbox with many competing implementations and each implementation even differing in implementation details within it's own versions.
In theory, it seems reasonable to say that K8s networking is an implemention detail which you shouldn't bother about but in practice, it's going to bite you because of some misconfigured component somewhere someday and you would have no idea what to make of thousands of iptable rules or BGP routes or even least inspectable eBPF mini programs.
Paid support, consulting.
Most application developers don’t give a rats ass about how 2 nodes are connected as long as the pods can speak to each other. Throw these guys in a devops team and give them the task to setup a 2 node k8s cluster and guess how much consideration they will pay to networking. Correct, zero. Except you can’t, because you have to choose a CNI plugin or choose a distribution that has made an opinionated choice for you.
To lower the barrier of entry. A default installation must become a lot more opinionated with much less options and more sane and safe default.
I chose Nomad for my current role and it works fantastic.
I used ECS in my last role and it was also fantastic.
I'm currently gently looking for a job and the market is brutal. I'm not even getting responses back from applications and I have ten years of experience. Every job posting I see requires deep k8s knowledge and hands-on experience with some ancillary k8s project like ArgoCD or Flux. I'm actually thinking about downplaying non-k8s experience on my resume and just floating the k8s that I do have straight to the top because, at the very least, it'll get my resume looked at.
If I was going to go back in time, I'd pick k8s instead of Nomad or ECS even if they weren't the best contextual choices because, hey, I gotta' put food on the table.
From an outsider’s perspective, it looks like in a 2x2 matrix of developer simplicity/complexity and operational simplicity/complexity, the current patterns all seem to be heavily biased for developer simplicity/operational complexity.
1. Is this assumption correct?
2. Does optimizing for another quadrant: developer complexity / operational simplicity make sense?
My intuition is that complexity in code can be managed far better than complexity in operations. Developers have abstractions, reusable libraries, unit tests/integration tests, etc. There may also be weird efficiencies that arise from having developers deal with some of these problems right from the design stage.
It seems kubernetes takes a problem and pushes it to fully to operations.
Is there a solution that takes this problem and turns it into a developer problem?
However, writing those tests is incredibly hard. It doesn’t matter if you approach it from a dev or ops angle. The system under test doesn’t only have side effects, it is side effects. You also cannot mock most things (in my opinion…), as that is either also very hard to instrument or straight up removes the test usefulness altogether. Imagine mocking the AWS management API for your integration tests. Not possible.
So what Dev calls integration or e2e tests, ops calls the dev environment. Works, but differently to how devs would do it. I don’t see an alternative.
Next, as much as knowledge siloes are being heralded as evil, they exist. Undoing siloes altogether isn’t possible. You’ll end up reerecting them elsewhere. Devs have their skill sets, and ops isn’t part of that. The opposite is also true. The intersection can be substantial, but never enough to have dev to it all alone. I don’t think that’s a bad thing either.
It’s great if you actually need all of this and consider those practices essential complexity. If you don’t - it feels like someone is shoving accidental complexity down your throat.
Turning it into a developer problem will not hide all these things or make them more manageable. It means you will reinvent k8s yourself, poorly.
I'm also incredibly annoyed at them displaying time in PDT. I genuinely don't understand why they decided on that instead of doing something normal like UTC or detecting my timezone. Especially annoying every six months because Europe and the US don't do Daylight Savings Time changes at the same time, so for a week or two there's an additional hour I have to account for.