Using Docker on EC2 but not Kubernetes is like using a car but deciding you will never use the gas or break pedals. At that point might as well walk (simpler) or fly (use Kubernetes).
They have essentially semi mocked Kubernetes without any of the benefits.
I’m not sure I follow. Kubernetes is “just” an orchestrator for handling containers. This company use Docker Engine directly for handling their containers. They still get the benefit of containers even if they don’t happen to use Kubernetes. What’s wrong with that?
That's not really true. Running a service and orchestrating a fleet of services are two entirely separate problems. The orchestration requirements for many workloads are different than what k8s provides.
If your orchestration needs aren't "use every last % of server capacity" you might not need k8s.
As someone working in a place where we did the same (and it was mainly my idea), I beg to differ. You don't have to learn/cope with all the new low level k8s plumbing, especially pre-EKS. Just plain old EC2, in an immutable fashion, just using docker as your package manager instead of baking a RPM or DEB.
ANd even with the advent of EKS in AWS there are still a lot of overlapping functionalities and new failure modes (HPA and its metrics/limited rules WRT EC2 ASG rules, coreDNS etc).
I might use a different analogy -- it might be more like using a bike versus a motorcycle. You can run a delivery business on a bike, but you'd be able to do it with much less tinkering, if you were using a motorcycle.
They are using cloudformation which predates Kubernetes and it's meant to solve the same problem. CloudFormation still uses docker, originally it used AMIs. I've worked for a lot of Fortune 500 companies that don't use Kubernetes and use Docker.
I'll take a walk most of the time (virtual instance provisioned with shell script). Flying yes, but only with pilots and the crew - no way in hell I'll start tinkering with it on my own as a sidetask.
> We have even had interesting candidates walk away from job offers citing the fact that we don’t use Kubernetes as the reason!
This is not too surprising. Candidates want to join companies that are perceived to be hip and with it technology-wise in order to further their own resume.
You want your skills to be transferrable (unsolicited career advice). Someone isn't going to work on your bespoke infra management system for below or at market rate when those skills won't transfer to another org (and they'll be behind the curve compared to others who have been "in the trenches" with k8s during the same window of time). Google, Facebook, and similar get a pass because they will shower you in money for their bespoke infra work (and it looks good on your CV to other orgs, typically).
Personally, you don't have to be hip, but I want to be able to have another job in hand based on the tech I was working with and the work I was working on in a matter of days when I decide to bounce (or the decision is made for me). This is just good risk and career management.
There is also the quality of work factor. What is easier to use a bespoke infra or kubernetes? For the same pay I'd rather work with k8s than deal with whatever gremlins are buried in 'custom' infra.
> Google, Facebook, and similar get a pass because they will shower you in money for their bespoke infra work (and it looks good on your CV to other orgs, typically).
You left out "Their bespoke infra work tends to be the genesis of the commodity infrastructure other people are using."
It's not hard for a Googler familiar with Borg to pick up Kubernetes. It's not hard for a Facebooker with some UI work under their belt to figure out React.
Counter-point, these candidates foresee constant frustration and struggles because the company has a culture of NIH. Not saying that's actually the case, but since their website is down, it's kinda hard to tell.
Yeah I can't read the article, but as an applicant this would immediately trigger a yellow flag. If the CTO/director/whoever decided can justify it as a unique differentiator then maybe it's fine and it's a cost of doing business. If they just don't want to deal with k8s because they like their home grown thing, it's going to be a drag on the company long-term to maintain.
Yeah I could totally see the director of engineering at my old company gleefully writing a smug blog post about how they don’t use technology X because they don’t go for all the hip new trends (translation: we actively avoid anything trendy even if it’s good, and we have an extreme NIH attitude). As a result, everything from search indexing to queuing to RPC to deployment is 100% homegrown and you need a training course in how to use everything, everything has specific experts that you must consult, and everything is in need of debugging sometimes when it should just be standard tools used everywhere.
OR....bear with me for a minute...they could have strong opinions about how something should be done, and if a company is not doing it that way it might be a warning sign.
If a company says they don't use any modern JS framework and stick to JQuery, I'd be out before they even explain why. Not because I want to be hip and further my own resume but because I'd hate my job if I worked there.
That doesn't seem like the only possible reason, another one could be that it appeared the company was anti-k8s for the sake of being anti-k8s. I would walk away from a company that I felt was proud of not using the right tool for a job.
p.s. Not claiming it was the right tool for the job in this case, it would all depend on context.
I mean sure, that can be a case, but I for instance am bored of companies who waste enormous resources to reinvent public cloud offerings with their own version written in Bash/Python/Terraform etc. that constantly has to be maintained/debugged/documented and worked around.
So maybe if people hear, hey our infrastructure is running on some 10 year old Frankenstein cobbled together from Docker and Virtual Machines and AWS and on premise servers they just pass.
I’m tired of having to learn an new infrastructure management/deployment tool every time I shift jobs. Tired of having to work with company custom platform. If you’re deployed on kubernetes it would take me a day to figure out how/where everything is deployed just by poking around with kubectl. Probably around that much time to deploy new software too.
It will take you 18 months to get kubectl access on just the staging environment though, and then another 6 to track down only 90% of the obscure people and processes that generate and mutate the manifests before deploying. Finally you'll realize that the whole time it was one guy in a corner cubicle in a remote office typing custom magic commands acting as the sole link between the engineering org and the cluster.
K8s is just another giant footgun. Good and bad orgs will still be good or bad, with or without it, but it definitely amplifies the bad.
If we’re looking at bad orgs, they’re probably gonna fuck up their custom orchestration tools to the same extent.
If we’re looking at the median, most orgs will more or less work with k8s manifests, maybe with some abstraction layer on top to make it easier for developers.
I wouldn't walk away from a company just because they were not using a given technology.
I would walk away if they were using a technology not fit for the job, or were actively building an in-house version based on out-dated knowledge (the use of the word "minion" in Kubernetes was deprecated in 2014 IIRC).
Think about this in another realm, like plumbing. If you were a plumber, would you accept a plumbing job if the company told you that they didn't buy materials and tools, and were instead designing and building their own in-house version?
> based on out-dated knowledge (the use of the word "minion" in Kubernetes was deprecated in 2014 IIRC).
If they decided in 2014 not to use k8s, how much effort do you expect them to spend on keeping up-to-date with the latest and greatest of a technology that they're not using?
Sure, but if you don't want to keep up to date with something, don't write an article about it. If you choose to write an article in 2021 about it, I would expect some level of updated research first.
I'm actually curious as to how big of an issue this actually is. HN is a relatively niche site. Do we actually have the power to take down a blog of a tech company?
DB queries in blogs (especially with this kind of traffic spike) are super correlated, you probably don't even need a cache if you batch identical reads together when the requests arrive within say 10 or 100ms of each other. Not a common technique though.
> use a performant web server i.e. not PHP or Ruby.
PHP in an ordinary deployment behind Apache2 or nginx is very fast. If you're writing a Web service and your top two criteria for the language are "fast" and "scripting language" and PHP doesn't at least make your short-list, you've likely screwed up. It's the one to beat—you pretty much have to go with something compiled to have a good chance of beating it at serving dynamic responses to ordinary Web requests, and even then it's easy to screw up in minor ways and end up slower. Persistent connections, sure, maybe you want something else, but basic request-response stuff? Performance is not a reason to avoid it, unless your other candidate languages are C++ or very well-tuned and carefully written Java or Go, something along those lines. Speed-wise, it is certainly in a whole other category from Ruby or Python (in fact, even "speedy" Nodejs struggles to match it).
Now, WordPress is slow because a top design goal is that it's infinitely at-runtime extensible. And lots of PHP sites are slow for the same reason that lots of Ruby, or Node, or Python, or Java, or .Net, or whatever, sites are slow: the developer was very bad at querying databases.
I know you're being facetious and I chuckled, but it's pretty amazing to see sites fall over under even a moderate load in 2021. A simple Go web server running on a pretty low-end machine can handle an HN-sized torrent with ease, even before sticking a CDN in front of it.
It's usually a WordPress/Joomla/Magento website that can do 1-5 requests/second at most.
EDIT: I'm not saying these softwares can't do more, it's just that usually on most default configurations, people don't bother to use a CDN, optimize the database, use a caching plugin, etc. They just install it, then install another 39 plugins and then ask why everything is slow. It's very common to see WordPress websites failing under load (and I've helped many to optimize their installs so I know it's possible, it's just not what the average WordPress install looks like).
For 4 years I had Static site generated in Hugo git/CI deployed to CloudFront/S3. Then the team wanted to move and moved to hubspot, it was simply harder to find and train marketing talent who is ready to learn git or even HTML. Far easier to find hubspot/WordPress designer/developers/content writers . Also it becomes easy to blame on the unfamiliar thing your company uses when things don't work for them the way they want it to.
I have known teams move to back to very simple default wordpress hosting from more advanced stacks.
My point is it harder to get your marketing dept to upgrade tech even if we could do it.
I have learnt over the years that solutions that have to be for the people who will use everyday, even if it's poorly managed WordPress.
In your example You or your SRE/devops will do all the basic configuration tunning pretty much out of the box. Your IT dept may not be able to at all or unless someone tells them to
So one thing I've seen very little of, but have written (to great success), is basically a wordpress/etc scraper. I was lucky in that the site was fully crawlable, but it meant that the non-technical website editors could edit things easily using a CMS, and then when they were ready, someone could deploy to prod. A CI pipeline would then crawl the site, saving all the generated static files, diff them, and upload only the parts that had changed.
It meant that we -never- had naked requests hitting the underlying CMS, so our security footprint there was miniscule (to access it you'd need to already be in the network), there was no chance of failure (i.e., even with a caching layer, a bunch of updates and enough traffic could lead to a spike hitting the CMS, and in fact, if there was a bad edit in the CMS that lead to part of the site being malformed, the crawler would die, alerting us to the bad edits before going live), and we could host using any static site/CDN we wanted to, with the only downside being deploys took a while. Even that downside we could have worked around, if we'd wanted to get into the CMS internals enough to track modifications; we just really didn't want to.
It is easier today, Netlify like apps offer the best of both parts ,integrate SSG to graphical UI. You can add CI pipelines for things like say W3C html validation, translation or WCAG compliance or image/asset optimization (designers sometimes forget), AMP generation, S3/CDN like hosting etc.
I was tempted to building something like like you did, but came to the same conclusion many do : it is not our core business and marketing will still have issues as the offering won't be as good as a professional one.
Even a solution to bridge the two setups needs dev time to maintain. End of the day managed solutions like Hubspot comes in cheaper in total cost of ownership even though all of this kind of architecture is technically superior.
Netlify is a good solution, glad that it is getting some love.
The move for us was before netlify became popular so never could consider it.
That's why I was tempted to build, then I realized bulk of the work will be to build the graphical UI editor and templating tools - the parts I hate to begin with.
So we were dealing with a Django thing that was a complete cluster of unmaintainable, and it was about 6 years ago, so more limited options than today. It took...maybe 40 man hours to write a script, get it into the build pipeline, and have everything work? A few days for a coupla guys, basically. It would have taken us longer to switch CMSes, just in user training, I suspect.
I dislike Wordpress as much as the next guy but a modest WP site with a cache or a free tier CDN in front could eat that kind of traffic for breakfast…
I've had software solutions delivered by consultants in past jobs that were billed as being 'complete', that were missing that. 10 rps = Django fell over, site is down. That was the first thing we fixed.
I'm not a WP expert either, but I did implement a CMS for my company. Caching interferes with people editing the site and getting to see the updates immediately, and the easy way out of that is to disable caching. My guess is that default WP takes the easy way out, because of their largely untrained userbase.
Plenty of organizations in which the marketing or PR department maintains the company blog, not their core IT staff. This is likely their "good reason".
Unpatched WP instances full of vulns are also rampant for this reason.
Most of the times the server is not the problem, the database is. And those scale a lot harder if you don't have a proper design for reads, writes and caching. Business wants to see the typo fix to their blogpost online the very second they make it.
Cloudflare is in front of the blog, but they seem to have purposefully disabled caching both in the browser via Cache-Control headers[1], and apparently in Cloudflare too[2].
[2] The cf-cache-status header says "DYNAMIC", which means "This resource is not cached by default and there are no explicit settings configured to cache it."
IIRC CloudFlare doesn't cache HTML by default, so if your HTML is dynamically generated (and not, at the very least, cached on your own server/application/WordPress) and you don't enable HTML caching on CF, you can still easily run into trouble.
I'm pretty sure cf-cache-status is added by CF—that's not the site saying not to cache it, it's CF reporting that it's not cached (which, again, I think is the default, not something the site owner deliberately turned off).
It's a page rule, available at every plan level (I just checked the docs). Might be a problem if you've already used up your quota of page rules at a given level, but otherwise, no big deal.
Enterprise is largely about very high levels of bandwidth use, serving non-Webpage traffic, and all kinds of advanced networking features. It's also a hell of a jump in price from the $200/m top-tier "self-serve" plan (which is also the only self-serve plan with any SLA whatsoever, so, the only one any business beyond the "just testing the waters" phase should be on)
Right, the page rule is the setting, and AFAIK that's the same even on Enterprise. You're right that there's no check-box/toggle-slider to make it happen.
It's covered in what I'd consider to be one of a handful of docs pages that're must-read for most any level of usage of the service:
They appear to be not caching it on purpose, because of some non-obvious stuff like the dynamic "Site Status" thing in the footer. If it were me, though, I'd figure out how to cache stuff like the blog.
Ought to move any small, self-contained dynamic sections to something loaded by JS, with failure allowed without (user-facing) complaint. Or kick it old-school and use an iframe.
Then again, most orgs have a ton of stuff they ought to do and haven't gotten around to. You just hope it doesn't bite you quite so publicly and ironically as this.
Got curious and started looking around and found a few blog posts about being on HN front page, and it appears like you might get a traffic spike of 15-50k hits over the course of the day. That is a ridiculously low standard...basic WordPress with a caching plugin on a free tier AWS instance or raspberry pi can handle that sort of volume.
If you're going to make a big deal about being contrarian with your technological choices, you either have to be really good at what you do, or be really honest about what tradeoffs you're making. I'd be extremely embarrassed if I pushed this blog post out.
I’ve had a few pieces on my personal site hit the front page here, and there was no effect on site behavior at all. Just Apache on a very modest VPS, with some non-essential dynamic content handled by Django. I don’t notice the traffic spikes until I look at my server logs the next morning.
My thumb rule of how many hits a site on HN gets is roughly 100x upvotes. So 100 upvotes = 10k clicks over 8-10 hours. This is from my experience with 3 links that hit the front page with between 150-420 upvotes.
I was serving static HTML off a $5 VPN and later off Cloudflare Pages. The response times were stable in both cases for all users.
It's ironic they're preaching technical excellence from their high horse, yet their website is returning errors and the status page incorrectly shows everything is working as expected.
No, this is from one of the intermittent successful responses. You can see it on https://ably.com/support also, which seems to be still up.
And cloudflare caching of some previous successful page fetch seems to be turned off as well. A cf-cache-status header of "DYNAMIC" means "This resource is not cached by default and there are no explicit settings configured to cache it.".
WPengine is configured like that, and you can’t change it, even on their highest tiers. Their support doesn’t understand that it defeats the whole purpose of WPengine’s reseller relationship with cloudflare.
Put your blog on a separate domain blog.company.tld and throw it behind Cloudflare free tier or something. It's pretty amazing how many companies fail to do that, and a moderately viral blog post would bring down their whole damn site when it's the critical moment to convert customers.
Having this happen to a post bragging about infrastructure is just adding insult to injury.
Edit: As another comment mentioned, this site is actually behind Cloudflare, with cf-cache-status: DYNAMIC (i.e. caching is disabled). I don't know what to say...
CF doesn't cache html by default. You have to add a page rule to do it. If your html being dynamically generated is most of your load (it probably is, even if it's not most of your bandwidth—serving plain files with Apache or Nginx or whatever is incredibly cheap, processor-wise) this means CF doesn't help much unless you tell it to.
Or make simpler pages. It’s possible to achieve a modern responsive design for 250kb per page and <25 elements even for someone like me who is far from this part of the stack in interest and ability. If they sold a 256MB LightSail I’d buy it.
It is possible. If they didn't include well over 1MB of scripts that don't seem to do much but are an order of magnitude fatter than the PNG images on the site.
I don't even know what those scripts are supposed to do. This is a static blog post.
Which raises the question why would I listen to anything else they have to say when they've just demonstrated their inability to scale an even modest amount?
> This has been asked by current and potential customers, by developers interested in our platform, and by candidates interviewing for roles at Ably. We have even had interesting candidates walk away from job offers citing the fact that we don’t use Kubernetes as the reason!
I celebrate a diversity in opinion on infrastructure but… if I was a CTO/VP of engineering and I read that line, that would be enough to convince me to use kubernetes.
Someone walking away from an offer because the company doesn't use unnecessary tool X is potentially a sign of a flaky engineer. I'd hesitate before working at any company with fewer than 100 engineers that does use k8s, because it's way more complexity and make-work than is appropriate for small services and teams.
I'm all for not using Kubernetes (or any other tech) just because, but seeing their website giving 500.. I can't help but feel all the k8s' laughter. :(
I think the laughter is mostly just fun. I don't think anyone laughing believes K8s would have auto prevented the issue.
In fact there is probably a growing problem that the ways this could have been solved in Kubernetes is slowly approaching the number of ways that it could have been solved on good ol' baremetal/ec2's. Kubernetes != non-bespoke system.
I like the fact, however, that there do at least exist standard primitives in Kubernetes that could approach horizontal scaling (if that was the issue as is insinuated by the aforementioned laughing) since the best thing Kubernetes does for me is helping me keep my cloud vendor (aws,gcp,azure,etc) knowledge shallow for typical web hosting solutions.
The piss races like we use x or we don't use y in engineering sound hilarious for some reason. such discussions focus on the process, but few talk about the results ¯\_(ツ)_/¯
> To move to Kubernetes, an organization needs a full engineering team just to keep the Kubernetes clusters running
Perhaps my team runs a simpler cluster, but we have been running a Kubernetes cluster for 2+ years as a team of 2 and it has been nothing less than worth it
The way the author describes the costs of moving to Kubernetes makes me think that they don't have the experience with Kubernetes to actually realize the major benefits over the initial costs
I've been on a team of 6-8 running relatively large-scale clusters (2 big ones with lower hundreds of workloads through tens of namespaces/tenants, plus a couple smaller ones). To "keep the Kubernetes clusters running" is an afterthought at most. I mostly agree with your comment.
Not OP, but similar workloads, team of 5
We use kops to deplot and manage the clusters. The hardest part is updating, due to the workloads running on it. Other than that, little to no problems with kube itself.
This comes up a lot around here; many people look at “Kubernetes the hard way” and think they need to run their own cluster.
Just use GKE or whatever the equivalent managed offering is for your cloud provider. A couple of clicks in the ui and you have a cluster. Its really easy to run, I rarely have to look at the cluster at all.
Yes! Was going to say the same. Kubernetes is far easier to learn than some random bespoke setup. After reading the article, it just sounds like they reinvented the wheel but made it AWS specific for some reason.
Was brought on as a consultant years ago and found their bespoke setup of random ec2 instances and lambda scripts to be far more difficult to understand than just spinning up a managed Kubernetes cluster and having a generic interface to deploy the application, as well as monitoring, logging, metrics, etc.
> Kubernetes is far easier to learn than some random bespoke setup
This, to me, is the biggest advantage of kubernetes. Yes, you can do all the things yourself, with your own custom solution that does everything you need to do just as well as kubernetes.
What kubernetes gives you is a shared way of doing things. By using a common method, you can easily integrate different services together, as well as onboard new hires easily.
Using something ubiquitous like kubernetes helps both your code onboarding and your people onboarding.
Also not having to debug EC2 lifecycle issues is really nice. And you don't have to deal with all of the garbage of setting up ec2 instances (configuring SSH, process management, configuration management, log exfiltration, monitoring, infra as code, nginx, albs, etc).
In addition to ease, why would I, as a s/w engr, want to invest in learning your weird stack instead of learning or using an equivalent tech stack that is actually marketable? Learning a technology can be a huge investment. Why would I want to invest in a technology with not much future and not much ability to translate into another position should this gig go south.
So, I should be OK to use my most precious asset -- time -- on an investment in a dead-end technology, because that investment could possibly, maybe translate (in some reduced way) to me being able to use a different technology later on that is not a dead-end technology? How about I just invest directly in technologies that have a payback and not try to cram the entirety of all software engineering systems into my brain. I already have an unending amount of things and systems to learn in s/w, why shouldn't I prioritize the learning process towards investments that have the best paybacks.
Kubernetes is literally more complicated than any other similar system in existence. It is so by definition, because it builds in more complexity than any other system has. But it also lacks the features you often need, so you have to learn even more to be able to get it to do what you want. You could write several books on running K8s in production.
My experiences with k8s have led me to never propose k8s in a new project. The k8s instances I have seen were bureaucratic and technical nightmares of unnecessary complexity. It doesn't provide a common ground for deployments because everyone will use and configure it differently.
>It doesn't provide a common ground for deployments because everyone will use and configure it differently.
Helm charts are used by 99% of the open source projects I've seen that run on top of Kubernetes. They are all written in a similar style so transferring settings between them is fairly easy. Helm will even create a barebones chart for you automatically in the common style.
A helm chart is not a complete deployment system, it's just a wrapper around kubectl. Neither provides everything you need to completely manage deploys.
I was doing research to move set up some new system into scalable cloud infrastructure. My first option was K8s (EKS) and second was plain ECS+Fargate. Ramping up in K8s was so convoluted and painful that I decided to follow up with ECS. That has been quite straightforward.
But you can't write books on running Linux in production, or Apache, or Windows, or Oracle, or... since the book shelves are (not literally but metaphorically) too crowded for yet another one on the subject
Exactly, plus all major cloud provider will happily host a kubernetes cluster for you if you ask them with your money. In a previous project, we managed more than 20 clusters on Azure (AKS) and AWS (EC2) with 4 people. The only tool that made this possible was kubernetes.
I've been running my own kubernetes cluster on a Raspberry pi. Does my cat count as an engineering team?
I feel like I must be stupid because I've tried several times to set up k8s on in-house hardware and it has never worked. I've tried several different "recipes" and the networking never works right. IDK if it's because things are changing so fast that the popular "recipes" are already outdated, or if they are being published without actually testing them, but it's left a bad taste in my mouth.
I'm sure the big cloud providers make it easy for end users to use, but that doesn't help me.
There's definitely a lot of Jenga going on here, but on the other hand: Kubernetes, when set up, has some very simple constraints that work. You don't often need to touch things at that layer; They become invisible, and when you're dealing with a cloud provider's hosted K8s, you don't get to touch them directly anyway.
K8s was a lot more simple earlier on. It's actually dying from adoption: There's a ton of dumb ideas bolted on top, that have become "standard" and "supported" because of demands from bad customers. The core is very clean, though, and you rarely need to interact with that.
That's one of those great questions about AWS. We actually have had to contact AWS on multiple occasions about EC2 layer issues, and each time I was thankful that a VM construct is very simple, comparatively, to reason about.
The networking part is always the most challenging. Everything between your router and your kubernetes cluster should still be routed and firewalled manually. However, if you can live with your home router firewall and a simple port mapping to your machines/cluster, then routing the traffic and setting up the cluster should be relatively painless.
This^^ I started out with kubespray because I was familiar with Ansible (I even contributed a very small bug fix), BUT k3s is just so awesome and out of your way. It’s not only easy to install but easy to remove.
Same. We've been running EKS at my last two companies using Terraform without issues and very small teams. There's a learning curve for sure but the ongoing maintenance is pretty minimal. I suspect going with GKE would have a smaller learning curve.
Genuine question because I honestly have no idea: how much additional learning curve is added by the average cloud provider to create a Kubernetes hosting context/setup?
Knowing nothing about K, I’m constantly wondering how it could be simpler than dumping a binary into a PaaS (web) hosting context and letting it do its thing. I’m interested to learn.
We used to run a very well run Kubernetes clusters with a single devops, then two. If you have senior devops engineers who know what they are doing and have the opportunity to greenfield it's really not that bad. We now have 4 devops engineers.
At the end of the day it's not necessarily Kubernetes that's difficult/time consuming, it's all the context around it.
How do you monitor/observability? Security? What's your CI/CD to deploy everything? How do you fully automate everything in reproducible ways within the k8s clusters, and outside?
I've been doing infrastructure management / devops for well over 23 years. Built & managed datacenters running tens of thousands of servers.
Kubernetes and AWS have made my life easier. What's really important is having the right senior people fully aware of the latest practices to get started. Greenfield now is easier than ever.
Althought their entire website is returning 500 errors, including the blog.
It is very hard not to point the irony of the situation.
In general I would not be so critic, but this is a company claiming to run highly available, mission critical distributed computing systems. Yet, they publish a popular blog article and it brings down their enire web presence?
I would have agreed with you if 'their blog AND their main service' also went down; Not just the website. But their main service is still up.
95% of the comments here parade about a hiccup of this service having a blog going down when that is not even their main service but continues to use an unreliable service like GitHub even when that goes down every month and they also don't use K8s either.
GitHub (and GitHub Actions) was proven to be very unreliable and the whole service went down multiple times and somehow that gets a pass? Even GitHub's status page returned 500s due to panicking developers DOSing the site. Same with GitHub pages.
But a single blog getting 500s but the whole service didn't go down? Nope.
It’s to be expected. Criticism of Kubernetes is taken personally by people who have dedicated themselves to it since it’s in vogue. It is very much showing the cracks in its perceived perfection and this company is far from the only one bucking the trend. We are seeing the long tail of people who don’t realize (or acknowledge) that Kubernetes is already on the other side of the hype cycle. That long tail is currently driving this conversation because commenting on a 500 is the lowest effort imaginable, but they do not speak for the industry. (Source: 24 years in FAANG-scale ops. I remember this cycle for many, many other fetish tools.)
Seriously, a Wordpress blog goes down (like that’s never happened under HN load) and all of HN is saying “lol what dogshit if only they used k8s for basically static content, the morons” which tells you a lot about the psychology going on here. A LOT. Otherwise smart engineers just can’t help themselves and redefine “irony” even though it contributes absolutely nothing to the discussion and doesn’t even address the fundamental criticism.
We are in a thread that essentially started with “the blog is down, how can I trust the product?” If you can figure out that logic please let me know. That’s basically saying “I don’t understand operations at all,” but here we are, listening to it and its Strong Opinions on a resource allocator and orchestrator.
This thread basically confirmed for me that Kubernetes is losing air. I already knew that, but the signal is getting a little bit stronger every month.
You assume A LOT, personally I’m tired of the “don’t use kubernetes” edgy blog posts that have very little substance to offer and mostly seem like people who haven’t studied distributed systems
Kubernetes isn’t perfect but it’s the best we have. Losing air to what?
This has been asked by current and potential customers, by developers interested in our platform, and by candidates interviewing for roles at Ably. We have even had interesting candidates walk away from job offers citing the fact that we don’t use Kubernetes as the reason!"
Why do you think this is an "edgy blog post"? This is a piece of Marketing, both recruitment and regular. The implied subtext of the post is: "we don't use Kubernetes but its not because we don't know what we are doing".
Separately, what does having studied distributed systems or not have to do with Kubernetes?
It’s edgy because k8s is clearly the best system we have and has a lot of notoriety. The article is clickbait at best.
The distributed state algorithms that underlying kubernetes are what makes it what it is. I see very little added to the discussion. I see a hacked together AWS system that will be subject to the laws of entropy.
What's hacked together about using NLBs and EC2 instances with statically launched Docker instances on each EC2 instance driven by Terraform and Cloudformation? That's all pretty standard AWS stuff. I'd guess from having run that sort of infrastructure in the past they have less custom tooling than the average K8s shop. It's certainly easier to understand the failure modes.
Plus this doesn't look like a heavy microservice environment. With a monolith or small number of services that aren't changing often with dedicated LBs and autoscale groups I'm guessing their environment is fairly easy to monitor and manage.
They are selling a highly-available product. You don't pay for pub/sub, anybody can run that, you pay for reliability and scalability. Their approach to other parts of their infrastructure definitely can and should inform your decision to buy into their product.
In addition to insight in their infrastructure practices, this gives you a unique opportunity to look into how they deal with outages, whether they update their status page honestly (and automatically), and how fast they can solve technical issues.
It also highlights a growing pet peeve of mine: the uselessness of status dashboards, if not done very well.
They're much harder than they look for complex systems. Most companies want to put a human in the loop for verification/sanity check purposes, and then you get this result. Automate updates in too-simplistic a way, and you end up with customers with robots reacting to phantoms.
Status pages are designed to be green so that Sales can tell people you're never down. Everyone does it, so you have to do it too.
My experience with monitoring 100% of outgoing API calls is that many services have well below a 100% success rate. Sometimes there are just unexplained periods of less than 100% success, with no update to the status page, and sometimes there's even a total outage. (I had a sales call with one of these providers, and they tried to sell me a $40,000 a year plan. I just wanted their engineers to have my dashboards so they can see how broken their shit is.)
The one shining light in the software as a service community is Let's Encrypt. At the first sign of problems, their status page has useful information about the outage.
I'm not surprised when a company's status page only reports on (highly available) services, not the website or blog—which are likely run by marketing, not engineering.
Still, it's simple and free to set up status pages so there isn't much excuse.
> Althought their entire website is returning 500 errors, including the blog
I've been here in a similar situation, and my guess is they've:
1. Reverse-proxied /blog to a crappy Wordpress instance run without caching.
2. HN traffic killed the blog.
3. Cloudflare, in their infinite wisdom, if they see enough 5xx errors from /blog, will start returning 5xx errors for every uri on the site, with some opaque retry interval.
More likely the entire www. server is a CMS that fell over, and the application is on a separate subdomain, and they are only monitoring the application.
As a ton of engineers have transitioned to kube already. Like years ago, you have just made ramping up new engineers to your custom setup a pain point for scaling. But you probably already knew that. =)
If a marketing website/blog is not the thing you are selling, it very often does not make sense to host it yourself. Your time and energy is better spent focussing on other things, and paying a separate hosting/marketing company to manage that.
I can't say for certain that this is what happened here, and the irony is definitely there, but overall it's not valid to judge a company's reliability on a piece that is not what they are selling, and are likely to not even be managing themselves.
Is your assertion that if they were on Kubernetes, this wouldn't have happened? What's ironic about a blog going down from a company selling pub/sub software?
Besides, though the situation's undeniably funny, there are much more appropriate, simple, and proportionate ways to ensure your WordPress marketing site or blog don't fall over than putting them on k8s, even if you are a k8s shop.
> In general I would not be so critic, but this is a company claiming to run highly available, mission critical distributed computing systems. Yet, they publish a popular blog article and it brings down their enire web presence?
You may expect that company blog have a bit lower uptime than services it offers. As someone on purchasing side, I don't give a f** if a company website or blog is down for weeks, if their services are operational.
As a disclaimer - never heard of Ably, but wholeheartedly support non-kubernetes environment. Being CTO in large e-commerce company we do not use or even plan to use kubernetes.
is it really NIH if they made no attempts to build anything resembling a k8s clone themselves? They decided container orchestration is just not a major problem for them. Its a completely different model, they're embracing immutable infrastructure principles instead of the k8s model (container versions in their stack are bound to the lifetime of the ec2 instance they're on).
This reads as simply a choice to have less complexity in their stack, which I think is admirable.
> Packing servers has the minor advantage of using spare resources on existing machines instead of additional machines for small-footprint services. It also has the major disadvantage of running heterogeneous services on the same machine, competing for resources. ...
Have a look at your CPU/MEM resource distributions, specifically the tails. That 'spare' resource is often 25-50% of resource used for the last 5% of usage. Cost optimization on the cloud is a matter of raising utilization. Have a look at your pods' use covariance and you can find populations to stochastically 'take turns' on that extra CPU/RAM.
> One possible approach is to attempt to preserve the “one VM, one service” model while using Kubernetes. The Kubernetes minions don’t have to be identical, they can be virtual machines of different sizes, and Kubernetes scheduling constraints can be used to run exactly one logical service on each minion. This raises the question, though: if you are running fixed sets of containers on specific groups of EC2 instances, why do you have a Kubernetes layer in there instead of just doing that?
The real reason is your AWS bill. Remember that splitting up a large .metal into smaller VMs means that you're paying the CPU/RAM bill for a kernel + basic services multiple times for the same motherboard. Static allocation is inefficient when exposed to load variance. Allocating small VMs to reduce the sizes of your static allocations costs a lot more overhead than tuning your pod requests and scheduling prefs.
Think of it like trucks for transporting packages. Yes you can pay AWS to rent you just the right truck, in the right number for each package you want to carry. Or you can just rent big-rigs and carry many, many packages. You'll have to figure out how to pack them in the trailer, and to make sure they survive the vibration of the trip, but you will almost certainly save money.
it's absolutely insane, they claim several thousands (!) of VMs and each autoscaling group having their own load balancer. Just the cost worth saving when they have only 10-20% wasted resource and an ingress controller instead of multiple hundred ELBs pays for several years of development (+ headcount to maintain).
Great point. By packing heterogeneous workloads on the same underlying VM, you can amortize the spare capacity against non-correlated workloads versus the one workload per VM they are pushing.
We use AWS ElasticBeanstalk with Docker images for our app infrastructure and it has served us well. I think it ends up being similar in that it pushes docker images to EC2 containers. It may not be cutting edge but affords us all the conveniences of docker images for deps while not needing the deep knowledge (and team resources) Kubernetes often requires.
Totally agree, it can scale to a lot of requests until you need to level up to container orchestration, but to be fair it is not competing on the same space as K8s.
Interestingly enough I normally recommend people avoid Kubernetes, if they don't have a real need, which most don't.
This is one of the first cases where I think that maybe Kubernetes would be the right solution, and it's an article about not using it. While there's a lot of information in the article, there might be some underlaying reason why this isn't a good fit for Kubernetes.
One thing that is highlighted very well is that fact that Kubernetes is pretty much just viewed as orchestration now. It's no longer amount utilizing your hardware better (in fact it uses more hardware in a many cases).
> Interestingly enough I normally recommend people avoid Kubernetes, if they don't have a real need, which most don't.
I would have agreed with this statement 2 years ago but now I think K8s has been commoditized to the point where it makes sense for many organisations. The problem I have seen in the past is that every team rolls their own orchestration using something like Ansible, at least K8s brings some consistency across teams
> K8s has been commoditized to the point where it makes sense for many organisations
Absolutely, but we still need to lose the control plane for it to make sense for small scale systems. Most of my customers run on something like two VMs, and that's only for redundancy. Asking them to pay for an additional three VMs for a control plane isn't feasible.
I think we need something like kubectl, but which works directly on a VM.
A lot of this new cloud stuff doesn't really scale down, it reminds me of Oracle, back in the day.
I had something like a desktop PC with 1GB of RAM, maybe 20 years ago, don't remember exactly how long ago. It was average, not too much but not too low either. Once I installed Oracle DB, it was idling at 512MB. Absolutely no DBs and no data set up, just the DB engine, idling at half the RAM of an average desktop PC.
Single node K8s is a thing (but slightly non-standard; you just have to untaint your “control plane” node. I’m not 100%, but I think that works with multiple control nodes too).
It takes quite a bit of work to optimize your deployments and resource requests to get good bin packing/utilization of the hardware. If you do it right there can be pretty good savings for larger deployments.
If your scale isn't high enough though I think you're right, it's often not worth the complexity and Kubernetes' overhead might actually be detrimental.
Agreed. I worked on a team that built out a solution like Ably described and we would run into lots of weird issues with ec2 lifecycle (often init script / configuration management issues) and deploys would take a long time (had to drain ec2 instances and bring new instances up) and there's just a lot more to manage (autoscaling groups, instance profiles, AMIs, configuration management, process management, log exfiltration, SSH keys, infra as code, etc). If you really don't want to use Kubernetes, you can get a lot of the benefits by using ECS/Fargate, but I really don't know why you wouldn't just go all-in on EKS at that point.
> A small custom boot service on each instance that is part of our boot image looks at the instance configuration, pulls the right container images, and starts the containers.
> There are lightweight monitoring services on each instance that will respawn a required container if it dies, and self-terminate the instance if it is running a version of any software that is no longer the preferred version for that cluster.
They've built a poor-mans Kubernetes that they won't be able to hire talent for, scales slower and costs more.
I didn't think they were using containerd; this phrase made me think that:
> Functionally, we still do the same, as Docker images are just a bunch of tarballs bundled with a metadata JSON blob, but curl and tar have been replaced by docker pull.
Yeah the wording is confusing. But they are definitely using “docker run” rather than say “docker pull” then somehow extracting the image and executing the contents. That would be totally bonkers.
I would be tempted to skip the Docker part entirely and use raw EC2 instances plus Packer or whatever tool of choice to.deploy the code. You still get autoscaling that way, and additional isolation of processes too. In addition, Amazon handle all the packing of VMs into hosts. With the Docker container route, you're still doing some of that work yourself to minimise waste.
ably cto: Remember when you said the website was "Totally under the control of the CMO" and "I should mind my own business"? Well I don't even have a Wordpress login. I literally can't help.
I find it awfully sad and depressing how people on the internet pile onto a company that tells an opposing view without reading or discussing further into their choices.
Instead it is easier to to critique the low hanging fruit point rather than discussing their actual reason for not using this 'kubernetes' software.
So is their blog the main product? If not, then the 'their blog gone down lol' quips are irrelevant.
I found their post rather interesting and didn't suffer any issues the rest of the 'commenters' are facing.
The risk with using Docker in production is that you can end up building your own bad version of Kubernetes over time. K8s is fairly complex but it solves a lot of useful problems such as zero downtime upgrades, service discovery, declarative config etc
I was a big advocate for Docker-as-deployment-packaging for a long time, and cautious about Kubernetes. I imagined building something like Ably describes would be the most practical.
I was wrong. What I didn't understand is how easy kubernetes is to use for the application developers. It's the most natural, seamless way to do Docker-as-deployment-packaging.
If you're going to have infrastructure guys at all, might as well have them maintain k8s.
You don't build your own Kubernetes, though. You run EC2 instances that are sized for the container they run and let autoscaling take care of it. Load balancers distribute the load. Kubernetes makes use of a lot of the same stuff, but with even more complexity.
Orchestration, in general, isn't needed. The major cloud providers are already doing it with virtual machines, and have been for a long time. It's better isolation than Kubernetes can provide.
> Orchestration, in general, isn't needed. The major cloud providers are already doing it with virtual machines, and have been for a long time. It's better isolation than Kubernetes can provide.
Sure it's not needed! But cloud providers aren't needed in much the same way.
Whatever convenience you think Kubernetes provides over EC2/autoscaling (which Kubernetes uses, by the way) is several orders of magnitude less than the convenience of using a cloud provider.
That you would draw on-prem versus cloud as an equivalency to Kube vs other deployment methods reeks of inexperience, to me.
edit: Oh no, I seem to triggered a drive by Kube fanboy or two. Yes, stealth downvote because you disagree without defending your position. You will do so much to show how right you are.
Right, my point is, you can always run your infra much the same AWS runs their infra. Cloud providers give you certain advantages that on prem doesn't have, and vice versa. Equally, running container orchestration gives you advantages AWS cannot give you (as well as k8s does anyway), and vice versa.
I'm a definitely fan of K8s, but I'm not defending it here, however, to saying orchestration isn't needed is silly. In a way what AWS provides is orchestration, it's just for VMs instead of containers.
As a devops engineer I've worked with a lot of individuals and with a lot of tooling, and so far I can only say my opinion for container orchestation has only grown stronger. I recall having to explain to certain developers how they have to first figure out more than 5 different services for AWS, then use packer to build an AMI, which is provisioned using Chef, then they have to terraform the relevant services, then use cloud-init to pull configuration values. All in all I had to do most of the work, and the code was scattered in several places. Compare that with a Dockerfile, a pipeline, and some manifest in the same repo. I've seen teams deploy a completely new application from first commit to production in less than a week, with next to zero knowledge of K8s when they started. The same teams, who weren't pros but had a bit of experience with AWS, took several weeks to deploy services of similar complexity on EC2. Saving two weeks of Developer's time and headaches is a lot, considering they're one of the most expensive expenses a company has.
> I recall having to explain to certain developers how they have to first figure out more than 5 different services for AWS, then use packer to build an AMI, which is provisioned using Chef, then they have to terraform the relevant services, then use cloud-init to pull configuration values. All in all I had to do most of the work, and the code was scattered in several places.
That sounds horrible. You can actually do all that from a single repository, using just ansible alone, with one cli invocation:
- use packer to build an AMI => ansible
- provisioned using Chef => ansible
- terraform the relevant services => ansible
- use cloud-init to pull configuration values => baked into the image
You are still orchestrating though, you still need to provision nodes and related resources, ensure containers are running on the node and are healthy, mount volumes, clean up old docker resources. Plus you are now completely locked in to EC2, which may not be a concern for you but it is if your product or service needs to run on different cloud platforms like ours, or the CTO decides to change cloud providers one day because they got a sweet deal. I’m glad it works for you but after working with k8s for 2 years I never want to write another Ansible / Terraform script to provision IaaS resources
People think that Kubernetes is just for service orchestration and it's true, it's very good for that. But what really sets it apart for me is the ability to extend kubernetes through operators and hooks that enables platform teams to really start to abstract away most of the underlying platform.
A good example is the custom resources we've created that heavily abstract away what's happening underneath. The apps gives us a docker image and describe how they want to deploy and it's all done.
2) Apparent heavy reliance on 3rd party projects to use it well and sanely which can (in my experience, in other ecosystems) seriously hamper one's ability to resist the constant-upgrade train to mitigate problem #1.
Basically, I'm afraid it's the Nodejs (or React Native—folks who've been there will know what I mean) of cluster management, not the... I dunno, Debian, reliable and slowly changing and sits-there-and-does-its-job. Does it seem that way to you and is just so good it's worth it anyway, or am I misunderstanding the situation?
The K8s API is very conservative with backwards compatability and graceful deprecation. In general, cluster upgrades should have minimal impact on your workloads. The horror stories are usually from a cluster admin point of view and devs are usually shielded from that
Came here to say this. It sounds like Ably is slowly moving towards creating their own orchestration tooling. Before you know it they'll need a bunch of additional features such as scaling up an individual service that runs on the same EC2 as 5 other services. Or permissions. Or whatever else. Kubernetes has it all solved. With the current abstractions given by major cloud providers such as EKS and GKE, it makes running it a lot simpler than it used to be. All of this effort they've spent on doing this would have likely taken a lot less effort than their current setup.
I see this argument made pretty frequently, and it might be correct for a team that's just starting out.
But service discovery and zero downtime upgrades are not that hard to implement and maintain. Our company's implementations of those things from 12 years ago have required ~zero maintenance and work completely fine. Sure, it uses Zookeeper which is out of vogue, but that also has been "just working" with very little attention.
A thought experiment where we had instead been running a Kubernetes cluster for 12 years comes out pretty lopsided in favor of the path we took for which one minimizes the effort spent and complexity of the overall system.
Using Docker Swarm as an alternative to K8s in prod has been incredibly helpful for us as a growing startup. You get 90% of the advertised K8s goodies for almost zero extra complexity.
Using deployment scripts + docker alone would be insane, even at our small scale.
264 comments
[ 4.2 ms ] story [ 275 ms ] threadThey have essentially semi mocked Kubernetes without any of the benefits.
If your orchestration needs aren't "use every last % of server capacity" you might not need k8s.
This is not too surprising. Candidates want to join companies that are perceived to be hip and with it technology-wise in order to further their own resume.
Personally, you don't have to be hip, but I want to be able to have another job in hand based on the tech I was working with and the work I was working on in a matter of days when I decide to bounce (or the decision is made for me). This is just good risk and career management.
(disclosure: infra roles previously)
You left out "Their bespoke infra work tends to be the genesis of the commodity infrastructure other people are using."
It's not hard for a Googler familiar with Borg to pick up Kubernetes. It's not hard for a Facebooker with some UI work under their belt to figure out React.
This is another way of saying that they don't want to read too much documentation.
If a company says they don't use any modern JS framework and stick to JQuery, I'd be out before they even explain why. Not because I want to be hip and further my own resume but because I'd hate my job if I worked there.
p.s. Not claiming it was the right tool for the job in this case, it would all depend on context.
So maybe if people hear, hey our infrastructure is running on some 10 year old Frankenstein cobbled together from Docker and Virtual Machines and AWS and on premise servers they just pass.
I’m tired of having to learn an new infrastructure management/deployment tool every time I shift jobs. Tired of having to work with company custom platform. If you’re deployed on kubernetes it would take me a day to figure out how/where everything is deployed just by poking around with kubectl. Probably around that much time to deploy new software too.
K8s is just another giant footgun. Good and bad orgs will still be good or bad, with or without it, but it definitely amplifies the bad.
If we’re looking at the median, most orgs will more or less work with k8s manifests, maybe with some abstraction layer on top to make it easier for developers.
I would walk away if they were using a technology not fit for the job, or were actively building an in-house version based on out-dated knowledge (the use of the word "minion" in Kubernetes was deprecated in 2014 IIRC).
Think about this in another realm, like plumbing. If you were a plumber, would you accept a plumbing job if the company told you that they didn't buy materials and tools, and were instead designing and building their own in-house version?
If they decided in 2014 not to use k8s, how much effort do you expect them to spend on keeping up-to-date with the latest and greatest of a technology that they're not using?
Or even those with caching.
A proof of concept demo: https://github.com/MatthewSteel/carpool
PHP in an ordinary deployment behind Apache2 or nginx is very fast. If you're writing a Web service and your top two criteria for the language are "fast" and "scripting language" and PHP doesn't at least make your short-list, you've likely screwed up. It's the one to beat—you pretty much have to go with something compiled to have a good chance of beating it at serving dynamic responses to ordinary Web requests, and even then it's easy to screw up in minor ways and end up slower. Persistent connections, sure, maybe you want something else, but basic request-response stuff? Performance is not a reason to avoid it, unless your other candidate languages are C++ or very well-tuned and carefully written Java or Go, something along those lines. Speed-wise, it is certainly in a whole other category from Ruby or Python (in fact, even "speedy" Nodejs struggles to match it).
Now, WordPress is slow because a top design goal is that it's infinitely at-runtime extensible. And lots of PHP sites are slow for the same reason that lots of Ruby, or Node, or Python, or Java, or .Net, or whatever, sites are slow: the developer was very bad at querying databases.
EDIT: I'm not saying these softwares can't do more, it's just that usually on most default configurations, people don't bother to use a CDN, optimize the database, use a caching plugin, etc. They just install it, then install another 39 plugins and then ask why everything is slow. It's very common to see WordPress websites failing under load (and I've helped many to optimize their installs so I know it's possible, it's just not what the average WordPress install looks like).
I used to have a 5mil pageviews per month site running on 5$ instance from DigitalOcean
I have known teams move to back to very simple default wordpress hosting from more advanced stacks.
My point is it harder to get your marketing dept to upgrade tech even if we could do it.
I have learnt over the years that solutions that have to be for the people who will use everyday, even if it's poorly managed WordPress.
In your example You or your SRE/devops will do all the basic configuration tunning pretty much out of the box. Your IT dept may not be able to at all or unless someone tells them to
It meant that we -never- had naked requests hitting the underlying CMS, so our security footprint there was miniscule (to access it you'd need to already be in the network), there was no chance of failure (i.e., even with a caching layer, a bunch of updates and enough traffic could lead to a spike hitting the CMS, and in fact, if there was a bad edit in the CMS that lead to part of the site being malformed, the crawler would die, alerting us to the bad edits before going live), and we could host using any static site/CDN we wanted to, with the only downside being deploys took a while. Even that downside we could have worked around, if we'd wanted to get into the CMS internals enough to track modifications; we just really didn't want to.
I was tempted to building something like like you did, but came to the same conclusion many do : it is not our core business and marketing will still have issues as the offering won't be as good as a professional one.
Even a solution to bridge the two setups needs dev time to maintain. End of the day managed solutions like Hubspot comes in cheaper in total cost of ownership even though all of this kind of architecture is technically superior.
The move for us was before netlify became popular so never could consider it.
That's why I was tempted to build, then I realized bulk of the work will be to build the graphical UI editor and templating tools - the parts I hate to begin with.
Awfully presumptive there aren't you. :P
I've had software solutions delivered by consultants in past jobs that were billed as being 'complete', that were missing that. 10 rps = Django fell over, site is down. That was the first thing we fixed.
I’m no WP expert but maybe they have a good reason to leave caching out of the core.
Unpatched WP instances full of vulns are also rampant for this reason.
[1] See, for example, https://ably.com/support , which is currently still up.
[2] The cf-cache-status header says "DYNAMIC", which means "This resource is not cached by default and there are no explicit settings configured to cache it."
I'm pretty sure cf-cache-status is added by CF—that's not the site saying not to cache it, it's CF reporting that it's not cached (which, again, I think is the default, not something the site owner deliberately turned off).
Need to use a "Page Rule" telling CF to cache everything, and provide the required Cache-Control header values from app side.
It's not an option in the caching settings (or need a enterprise plan?)
Enterprise is largely about very high levels of bandwidth use, serving non-Webpage traffic, and all kinds of advanced networking features. It's also a hell of a jump in price from the $200/m top-tier "self-serve" plan (which is also the only self-serve plan with any SLA whatsoever, so, the only one any business beyond the "just testing the waters" phase should be on)
But the Cache Settings page, it doesn't have the "Cache Everything" option, or it's reserved to Enterprise plan.
It's covered in what I'd consider to be one of a handful of docs pages that're must-read for most any level of usage of the service:
https://support.cloudflare.com/hc/en-us/articles/200172516-U...
(it also links to a page that describes exactly how to create a "cache everything" page rule)
Then again, most orgs have a ton of stuff they ought to do and haven't gotten around to. You just hope it doesn't bite you quite so publicly and ironically as this.
If you're going to make a big deal about being contrarian with your technological choices, you either have to be really good at what you do, or be really honest about what tradeoffs you're making. I'd be extremely embarrassed if I pushed this blog post out.
I was serving static HTML off a $5 VPN and later off Cloudflare Pages. The response times were stable in both cases for all users.
Now, application servers and database tiers, that's a different story. Epic novel, really.
You gotta admit, it’s a pretty bad look to come here with a boldly titled systems architecture blog post and have your website crash…
A junior web developer nowadays can easily build a system to serve static content that avoids all but the worst hugs of death.
If this wasn’t a hug of death then you have service availability issues, regardless…
And cloudflare caching of some previous successful page fetch seems to be turned off as well. A cf-cache-status header of "DYNAMIC" means "This resource is not cached by default and there are no explicit settings configured to cache it.".
Having this happen to a post bragging about infrastructure is just adding insult to injury.
Edit: As another comment mentioned, this site is actually behind Cloudflare, with cf-cache-status: DYNAMIC (i.e. caching is disabled). I don't know what to say...
I don't even know what those scripts are supposed to do. This is a static blog post.
You don't need Kubernetes for a blog. If you do then something is wrong with your whole backend stack.
Perhaps they are the smart ones who want to save money and not burn millions of VC money.
The one that REALLY needs to use Kubernetes is actually GitHub. [0]
[0] https://news.ycombinator.com/item?id=27366397
Also Employers: Some of our applicants turned down our job offer, when we revealed that we don't use specific technologies!
I celebrate a diversity in opinion on infrastructure but… if I was a CTO/VP of engineering and I read that line, that would be enough to convince me to use kubernetes.
Google Cache doesn't work either https://webcache.googleusercontent.com/search?q=cache:YECd_I...
Luckily there's an Internet Archive https://web.archive.org/web/20210720134229/https://ably.com/...
In fact there is probably a growing problem that the ways this could have been solved in Kubernetes is slowly approaching the number of ways that it could have been solved on good ol' baremetal/ec2's. Kubernetes != non-bespoke system.
I like the fact, however, that there do at least exist standard primitives in Kubernetes that could approach horizontal scaling (if that was the issue as is insinuated by the aforementioned laughing) since the best thing Kubernetes does for me is helping me keep my cloud vendor (aws,gcp,azure,etc) knowledge shallow for typical web hosting solutions.
Perhaps my team runs a simpler cluster, but we have been running a Kubernetes cluster for 2+ years as a team of 2 and it has been nothing less than worth it
The way the author describes the costs of moving to Kubernetes makes me think that they don't have the experience with Kubernetes to actually realize the major benefits over the initial costs
It doesn't bear much weight though, and I've had experience with other toolings largely to the same effect.
I agree with the sibling comment about upgrades - this is IMO where GKE really shines as a cluster upgrade is mostly a button-pressing exercise.
Just use GKE or whatever the equivalent managed offering is for your cloud provider. A couple of clicks in the ui and you have a cluster. Its really easy to run, I rarely have to look at the cluster at all.
Was brought on as a consultant years ago and found their bespoke setup of random ec2 instances and lambda scripts to be far more difficult to understand than just spinning up a managed Kubernetes cluster and having a generic interface to deploy the application, as well as monitoring, logging, metrics, etc.
This, to me, is the biggest advantage of kubernetes. Yes, you can do all the things yourself, with your own custom solution that does everything you need to do just as well as kubernetes.
What kubernetes gives you is a shared way of doing things. By using a common method, you can easily integrate different services together, as well as onboard new hires easily.
Using something ubiquitous like kubernetes helps both your code onboarding and your people onboarding.
Helm charts are used by 99% of the open source projects I've seen that run on top of Kubernetes. They are all written in a similar style so transferring settings between them is fairly easy. Helm will even create a barebones chart for you automatically in the common style.
You used to work with Ably?
I've been running my own kubernetes cluster on a Raspberry pi. Does my cat count as an engineering team?
I'm sure the big cloud providers make it easy for end users to use, but that doesn't help me.
Or is this just the Jenga Model of Modern Software Development in action?
K8s was a lot more simple earlier on. It's actually dying from adoption: There's a ton of dumb ideas bolted on top, that have become "standard" and "supported" because of demands from bad customers. The core is very clean, though, and you rarely need to interact with that.
The networking part is always the most challenging. Everything between your router and your kubernetes cluster should still be routed and firewalled manually. However, if you can live with your home router firewall and a simple port mapping to your machines/cluster, then routing the traffic and setting up the cluster should be relatively painless.
Knowing nothing about K, I’m constantly wondering how it could be simpler than dumping a binary into a PaaS (web) hosting context and letting it do its thing. I’m interested to learn.
At the end of the day it's not necessarily Kubernetes that's difficult/time consuming, it's all the context around it.
How do you monitor/observability? Security? What's your CI/CD to deploy everything? How do you fully automate everything in reproducible ways within the k8s clusters, and outside?
I've been doing infrastructure management / devops for well over 23 years. Built & managed datacenters running tens of thousands of servers.
Kubernetes and AWS have made my life easier. What's really important is having the right senior people fully aware of the latest practices to get started. Greenfield now is easier than ever.
By myself.
> Ably is a Pub/Sub messaging platform that companies can use to develop realtime features in their products.
At this time, https://status.ably.com/ is reporting all green.
Althought their entire website is returning 500 errors, including the blog.
It is very hard not to point the irony of the situation.
In general I would not be so critic, but this is a company claiming to run highly available, mission critical distributed computing systems. Yet, they publish a popular blog article and it brings down their enire web presence?
This reaction is a storm in a tea cup left inside of a sunny cottage.
Downvoters: I don't see anyone screaming at this stock trading app that also stopped using Kubernetes [0].
[0] https://freetrade.io/blog/killing-kubernetes
I run on k8s and it works really well
I suspect the sort of individual visiting this website and reading the blog will know that their services and blog do not run on the same machines.
95% of the comments here parade about a hiccup of this service having a blog going down when that is not even their main service but continues to use an unreliable service like GitHub even when that goes down every month and they also don't use K8s either.
GitHub (and GitHub Actions) was proven to be very unreliable and the whole service went down multiple times and somehow that gets a pass? Even GitHub's status page returned 500s due to panicking developers DOSing the site. Same with GitHub pages.
But a single blog getting 500s but the whole service didn't go down? Nope.
Seriously, a Wordpress blog goes down (like that’s never happened under HN load) and all of HN is saying “lol what dogshit if only they used k8s for basically static content, the morons” which tells you a lot about the psychology going on here. A LOT. Otherwise smart engineers just can’t help themselves and redefine “irony” even though it contributes absolutely nothing to the discussion and doesn’t even address the fundamental criticism.
We are in a thread that essentially started with “the blog is down, how can I trust the product?” If you can figure out that logic please let me know. That’s basically saying “I don’t understand operations at all,” but here we are, listening to it and its Strong Opinions on a resource allocator and orchestrator.
This thread basically confirmed for me that Kubernetes is losing air. I already knew that, but the signal is getting a little bit stronger every month.
Kubernetes isn’t perfect but it’s the best we have. Losing air to what?
"the following question comes up a lot:
“So… do you use Kubernetes?”
This has been asked by current and potential customers, by developers interested in our platform, and by candidates interviewing for roles at Ably. We have even had interesting candidates walk away from job offers citing the fact that we don’t use Kubernetes as the reason!"
Why do you think this is an "edgy blog post"? This is a piece of Marketing, both recruitment and regular. The implied subtext of the post is: "we don't use Kubernetes but its not because we don't know what we are doing".
Separately, what does having studied distributed systems or not have to do with Kubernetes?
The distributed state algorithms that underlying kubernetes are what makes it what it is. I see very little added to the discussion. I see a hacked together AWS system that will be subject to the laws of entropy.
Plus this doesn't look like a heavy microservice environment. With a monolith or small number of services that aren't changing often with dedicated LBs and autoscale groups I'm guessing their environment is fairly easy to monitor and manage.
Best system for what?
https://xkcd.com/932/
In addition to insight in their infrastructure practices, this gives you a unique opportunity to look into how they deal with outages, whether they update their status page honestly (and automatically), and how fast they can solve technical issues.
It also highlights a growing pet peeve of mine: the uselessness of status dashboards, if not done very well.
They're much harder than they look for complex systems. Most companies want to put a human in the loop for verification/sanity check purposes, and then you get this result. Automate updates in too-simplistic a way, and you end up with customers with robots reacting to phantoms.
My experience with monitoring 100% of outgoing API calls is that many services have well below a 100% success rate. Sometimes there are just unexplained periods of less than 100% success, with no update to the status page, and sometimes there's even a total outage. (I had a sales call with one of these providers, and they tried to sell me a $40,000 a year plan. I just wanted their engineers to have my dashboards so they can see how broken their shit is.)
The one shining light in the software as a service community is Let's Encrypt. At the first sign of problems, their status page has useful information about the outage.
I'm not surprised when a company's status page only reports on (highly available) services, not the website or blog—which are likely run by marketing, not engineering.
Still, it's simple and free to set up status pages so there isn't much excuse.
I've been here in a similar situation, and my guess is they've:
1. Reverse-proxied /blog to a crappy Wordpress instance run without caching.
2. HN traffic killed the blog.
3. Cloudflare, in their infinite wisdom, if they see enough 5xx errors from /blog, will start returning 5xx errors for every uri on the site, with some opaque retry interval.
4. Voila, the entire site appears to be dead.
1) The realtime service and website are different things. The blog post is talking about the service, which has been continuously available.
2) Oops, the website fell over. We'll fix that. Thanks for all the advice :)
I can't say for certain that this is what happened here, and the irony is definitely there, but overall it's not valid to judge a company's reliability on a piece that is not what they are selling, and are likely to not even be managing themselves.
> Our blog and website are experiencing load-related issues, leading to slow loading and 5xx errors.
All backend services (realtime and REST apis) are on entirely separate infrastructure and are unaffected.
You may expect that company blog have a bit lower uptime than services it offers. As someone on purchasing side, I don't give a f** if a company website or blog is down for weeks, if their services are operational.
As a disclaimer - never heard of Ably, but wholeheartedly support non-kubernetes environment. Being CTO in large e-commerce company we do not use or even plan to use kubernetes.
seems like a me too, i know more than you type of project. looks geared towards cryptos though.
This reads as simply a choice to have less complexity in their stack, which I think is admirable.
* Well four, but one of the mentions of "custom x" is talking about k8s way of doing things.
> Packing servers has the minor advantage of using spare resources on existing machines instead of additional machines for small-footprint services. It also has the major disadvantage of running heterogeneous services on the same machine, competing for resources. ...
Have a look at your CPU/MEM resource distributions, specifically the tails. That 'spare' resource is often 25-50% of resource used for the last 5% of usage. Cost optimization on the cloud is a matter of raising utilization. Have a look at your pods' use covariance and you can find populations to stochastically 'take turns' on that extra CPU/RAM.
> One possible approach is to attempt to preserve the “one VM, one service” model while using Kubernetes. The Kubernetes minions don’t have to be identical, they can be virtual machines of different sizes, and Kubernetes scheduling constraints can be used to run exactly one logical service on each minion. This raises the question, though: if you are running fixed sets of containers on specific groups of EC2 instances, why do you have a Kubernetes layer in there instead of just doing that?
The real reason is your AWS bill. Remember that splitting up a large .metal into smaller VMs means that you're paying the CPU/RAM bill for a kernel + basic services multiple times for the same motherboard. Static allocation is inefficient when exposed to load variance. Allocating small VMs to reduce the sizes of your static allocations costs a lot more overhead than tuning your pod requests and scheduling prefs.
Think of it like trucks for transporting packages. Yes you can pay AWS to rent you just the right truck, in the right number for each package you want to carry. Or you can just rent big-rigs and carry many, many packages. You'll have to figure out how to pack them in the trailer, and to make sure they survive the vibration of the trip, but you will almost certainly save money.
EDIT: Formatting
https://hispanicbusinesstv.com/no-we-dont-use-kubernetes/
This is one of the first cases where I think that maybe Kubernetes would be the right solution, and it's an article about not using it. While there's a lot of information in the article, there might be some underlaying reason why this isn't a good fit for Kubernetes.
One thing that is highlighted very well is that fact that Kubernetes is pretty much just viewed as orchestration now. It's no longer amount utilizing your hardware better (in fact it uses more hardware in a many cases).
I would have agreed with this statement 2 years ago but now I think K8s has been commoditized to the point where it makes sense for many organisations. The problem I have seen in the past is that every team rolls their own orchestration using something like Ansible, at least K8s brings some consistency across teams
Absolutely, but we still need to lose the control plane for it to make sense for small scale systems. Most of my customers run on something like two VMs, and that's only for redundancy. Asking them to pay for an additional three VMs for a control plane isn't feasible.
I think we need something like kubectl, but which works directly on a VM.
I had something like a desktop PC with 1GB of RAM, maybe 20 years ago, don't remember exactly how long ago. It was average, not too much but not too low either. Once I installed Oracle DB, it was idling at 512MB. Absolutely no DBs and no data set up, just the DB engine, idling at half the RAM of an average desktop PC.
If your scale isn't high enough though I think you're right, it's often not worth the complexity and Kubernetes' overhead might actually be detrimental.
Jokes aside, it sounds like they should just use ECS instead.
> A small custom boot service on each instance that is part of our boot image looks at the instance configuration, pulls the right container images, and starts the containers.
> There are lightweight monitoring services on each instance that will respawn a required container if it dies, and self-terminate the instance if it is running a version of any software that is no longer the preferred version for that cluster.
They've built a poor-mans Kubernetes that they won't be able to hire talent for, scales slower and costs more.
> Functionally, we still do the same, as Docker images are just a bunch of tarballs bundled with a metadata JSON blob, but curl and tar have been replaced by docker pull.
but, yes, I agree; it's a hand-made Kubernetes
ably cto: Go talk to the CMO, I can't help.
ably ceo: What!!
ably cto: Remember when you said the website was "Totally under the control of the CMO" and "I should mind my own business"? Well I don't even have a Wordpress login. I literally can't help.
Instead it is easier to to critique the low hanging fruit point rather than discussing their actual reason for not using this 'kubernetes' software.
So is their blog the main product? If not, then the 'their blog gone down lol' quips are irrelevant.
I found their post rather interesting and didn't suffer any issues the rest of the 'commenters' are facing.
Ably using ghost blogging platform (https://ghost.ably.com) seeing requests in network console.
I was wrong. What I didn't understand is how easy kubernetes is to use for the application developers. It's the most natural, seamless way to do Docker-as-deployment-packaging.
If you're going to have infrastructure guys at all, might as well have them maintain k8s.
Orchestration, in general, isn't needed. The major cloud providers are already doing it with virtual machines, and have been for a long time. It's better isolation than Kubernetes can provide.
Sure it's not needed! But cloud providers aren't needed in much the same way.
Whatever convenience you think Kubernetes provides over EC2/autoscaling (which Kubernetes uses, by the way) is several orders of magnitude less than the convenience of using a cloud provider.
That you would draw on-prem versus cloud as an equivalency to Kube vs other deployment methods reeks of inexperience, to me.
edit: Oh no, I seem to triggered a drive by Kube fanboy or two. Yes, stealth downvote because you disagree without defending your position. You will do so much to show how right you are.
I'm a definitely fan of K8s, but I'm not defending it here, however, to saying orchestration isn't needed is silly. In a way what AWS provides is orchestration, it's just for VMs instead of containers.
As a devops engineer I've worked with a lot of individuals and with a lot of tooling, and so far I can only say my opinion for container orchestation has only grown stronger. I recall having to explain to certain developers how they have to first figure out more than 5 different services for AWS, then use packer to build an AMI, which is provisioned using Chef, then they have to terraform the relevant services, then use cloud-init to pull configuration values. All in all I had to do most of the work, and the code was scattered in several places. Compare that with a Dockerfile, a pipeline, and some manifest in the same repo. I've seen teams deploy a completely new application from first commit to production in less than a week, with next to zero knowledge of K8s when they started. The same teams, who weren't pros but had a bit of experience with AWS, took several weeks to deploy services of similar complexity on EC2. Saving two weeks of Developer's time and headaches is a lot, considering they're one of the most expensive expenses a company has.
That sounds horrible. You can actually do all that from a single repository, using just ansible alone, with one cli invocation:
- use packer to build an AMI => ansible
- provisioned using Chef => ansible
- terraform the relevant services => ansible
- use cloud-init to pull configuration values => baked into the image
People think that Kubernetes is just for service orchestration and it's true, it's very good for that. But what really sets it apart for me is the ability to extend kubernetes through operators and hooks that enables platform teams to really start to abstract away most of the underlying platform.
A good example is the custom resources we've created that heavily abstract away what's happening underneath. The apps gives us a docker image and describe how they want to deploy and it's all done.
1) Churn + upgrade horror stories,
2) Apparent heavy reliance on 3rd party projects to use it well and sanely which can (in my experience, in other ecosystems) seriously hamper one's ability to resist the constant-upgrade train to mitigate problem #1.
Basically, I'm afraid it's the Nodejs (or React Native—folks who've been there will know what I mean) of cluster management, not the... I dunno, Debian, reliable and slowly changing and sits-there-and-does-its-job. Does it seem that way to you and is just so good it's worth it anyway, or am I misunderstanding the situation?
But service discovery and zero downtime upgrades are not that hard to implement and maintain. Our company's implementations of those things from 12 years ago have required ~zero maintenance and work completely fine. Sure, it uses Zookeeper which is out of vogue, but that also has been "just working" with very little attention.
A thought experiment where we had instead been running a Kubernetes cluster for 12 years comes out pretty lopsided in favor of the path we took for which one minimizes the effort spent and complexity of the overall system.
Using deployment scripts + docker alone would be insane, even at our small scale.