[OP here] It feels bizarre saying this, having spent so much of my life advocating for and selling a distribution of Kubernetes and consulting services to help folks get the most of out it, but here goes! YOU probably shouldn't use Kubernetes and a bunch of other "cool" things for your product.
Most folks building software at startups and scale-ups should avoid Kubernetes and other premature optimisations. If your company uses Kubernetes, you are likely expending energy on something that doesn't take you towards your mission. You have probably fallen into the trap of premature optimisation.
Please don't take this post to be only aimed against Kubernetes. It is not. I am directing this post at every possible bit of premature optimisation engineers make in the course of building software.
Running a self-hosted Kubernetes is indeed ... questionable, but a managed Kubernetes? That's a pretty sane thing to do IMO. The alternatives are running your Docker containers either manually on some EC2 or other bare-metal server which is a nightmare to do deployments, or using something like Elastic Beanstalk which is even worse.
For me at least, Kubernetes has become something like an universal standard: if you're already running Docker or other containerization, "using Kubernetes" is nothing more than a clear, "single source of truth" documentation on what infrastructure your application expects. Certainly beats that Confluence document from 2018, last updated 2019, on how the production server is set up that doesn't even match up with what was reality in 2020 much less today.
Of course, managed means you don't take care of say, creating your own X.509 CA and issuing certificates, tending to their expiry, installing Tiller, setting up pod networking etc. etc.
All of these much more annoying and harder than `helm update --install`ing some charts to your own cluster.
To be fair installing tiller hasn't been a thing for years. And pretty much everyone using cert-manager and lets encrypt which makes the whole X.509 story pretty much a no brainer.
Even if they know it, unless they absolutely need to implement it, why would you waste such valuable resources on of all things infra? (Unless your product IS infra). No engineer I know who’s smart enough to effortlessly deploy k8s on their own would want to do that as the job. There’s a million other interesting things (hopefully?) that they can be doing.
Often to get access to things that integrate well with kubernetes.
Personally I don't see too much difference between kube yamls and systemd units and cloudformation or whatever and the "cluster maintenance" itself is not the burden it used to be if you stay on the "paved road" provided by your cloud.
Not at all. If you know Kubernetes well and your needs are fairly simple it takes no time at all.
On a project a couple of years ago my cofounder and I opted to use Kubernetes after running into constant frustration after following the advice to keep it as simple as possible and just use VMs.
Kubernetes makes many complicated things very easy and lets you stop worrying about your infra (provided you’re using a managed provider).
On our next project we used a PaaS offering because we had a bunch of credits for it and it was painful in comparison to what we had with Kubernetes. It was way more expensive (if we didn’t have credits), the deployments were slow, and we had less flexibility.
Kubernetes isn’t perfect, far from it. But for those who know it well it is usually a good option.
You can get away without an orchestrator right up until about when your ARR hits $10mm
Hiring people with k8s experience in 2022 is not a difficult task, it's not an obscure technology anymore, and when your initial crop of devops decides to leave, the new guys coming in can scan your deployments and namespaces and pretty much hit the ground running within 48-72 hours. That's a big, important part of running a business. Being able to hire people with the right skill set, and being able to keep the lights on.
Bespoke systems built on top of EC2 require time, extra documentation and a steep learning curve, not to mention the fact that the bespoke system probably isn't getting security audits or being built to modern standards or best practices, tooling isn't being kept up to date. I can build a mechanical wristwatch in my garage, but if I had full access to the Rolex factory floor for free, I'd probably take that option.
I just came out of a project where Kubernetes performance issues involved wild guess and blind tuning until the so called experts actually found out why the cloud cluster was behaving strangely, including support from Cloud vendor.
And good luck making sense of all the YAML spaghetti available for bootstrapping the whole cluster from scratch.
If your devops guys are struggling, you are hiring the wrong devops folks, or at the wrong end of the pay band. Most devops guys I know are paid in the same band as a senior developer.
Sure, it is like bug free C code. It is only a matter of having the top of the cream. Pity there aren't enough of them in the world, including on cloud vendor support team.
What about standardisation that comes with using a framework like kubernetes , while not using k8s you end up with adhoc deployment methods, clunky work arounds of handling networking, policies, secrets etc, with Kubernetes or even ECS it signals that the team or the developer is looking to use fixed set of rules for infrastructure, also k8s scales well even for smaller apps
Seriously, I use k8s for the same reason I use docker: it's a standard language for deployment. Yeah I could do the same stuff manually for a small project.. but why?
In the limit, there are some startups that could run production on a single Linux host - I recently helped one get off Heroku and their spend went from ~$1k/mo to ~$50/mo and it made debugging and figuring out performance issues so much easier than what they were doing previously...
I'm very much with you on this, but I do understand that it's one of those things that is just not feasible when your team has no sysadmin/devops experience.
You were able to do it, but what happens to them when you're not around? Does their team have the required experience to handle it? That's the difference in cost. It's like DYI - yes, if I have all the skills and experience I can do everything myself incredibly cheaply, but... I don't. So I gotta pay.
But if your team does have the skills and experience, it's definitely worth looking into.
I do think people deeply underestimate what can be achieved with a single (or two, if you need a standby) dedicated Linux server these days. A single server can easily go up to 2TB of RAM and 128+ cores. Long before you ever get to a scale where that's a limitation, you'll have more than enough resources to figure things out.
> I'm very much with you on this, but I do understand that it's one of those things that is just not feasible when your team has no sysadmin/devops experience.
Exactly - $950/mo is nowhere near enough to pay for those skills if you don't have them. It's good value for money.
Funnily enough, small amount of servers that you want to utilize as much as possible is pretty much one of the original use cases for kubernetes.
My first deployment involved virtual machines, but we were essentially packing as much as possible into lowest amount of VMs, then doubled it up so that we had failover. This way we had clear visibility of how many resources we were using and could allocate them according to how much we had.
You were able to do it, but what happens to them when you're not around?
This is why you pay someone. Saving $950/month means it's well worth spending $500 for a day of someone's time occasionally. You don't have to do everything internally when you run a startup. Buying in services that you only need occasionally is well worth the money.
Are there contractors out there that will take on call shifts? Because it seems unlikely, and if your proposal is "put into a production a system that you will have to spend $500/day every time it goes down and wait 2-4 business days for a resolution" then you're a braver person than I am.
Obviously not. You don't pay someone to just set it up. You pay to help do what you'd do if you had a dedicated devOps teams. You pay someone to set up the system with your team so they understand it, train your team to use it, write some documentation about it, script a rollback procedure, maybe help on developing playbooks, etc.
Besides, there are people out there who offer on-call services for a small retainer.
This is optimistic to say the least. I've worked as an SRE for 5 years and apart from the others in the team the devs don't have nearly as much knowledge. There's no way I'd rely on them to fix an outage.
And even on a small retainer you'd better hope they retained the knowledge of how all that stuff works if you're only calling on them every now and again.
The idea is devops as _culture_ - I come in as an expert and set it up, then show them how I did it, then run through various disaster recovery scenarios so they learn it and can handle the vast majority of problems.
And you'd be surprised how little problems you might have - I've had many VMs with literally years of uptime running without issue.
Most people focus on "devops" as a job - and they never bother to teach the rest of the team anything about how stuff works. Worse than that, modern clouds encourage you to build opaque and complicated systems which probably only someone working on them full-time has a hope to understand...
If it was so easy for devs to pick up SRE companies wouldn't be struggling to find good people.
Culture doesn't mean they'll know how to fix <some weird edge case> at 3am in the morning. The SRE with constant ops exposure is likely to have a much better chance, if only because they (ought to) really know how to debug it.
> I've had many VMs with literally years of uptime running without issue.
I hope they all have fully patched base libraries and kernels, security auditing is getting to a much more common requirement these days even among very small companies. For example, anybody using Facebook Login.
And even on a small retainer you'd better hope they retained the knowledge of how all that stuff works if you're only calling on them every now and again.
This is why we document things at the company I work for. If you're serious about a project and want it to exist for a long time there will be things that only come up every few years, and you won't remember them. You can either write things down or you have to relearn how things work every time. Wroting things down is much easier.
> I'm very much with you on this, but I do understand that it's one of those things that is just not feasible when your team has no sysadmin/devops experience.
But this applies to everything. You also need Heroku or Kubernetes or whatever experience to maintain those systems, right?
The question is how much you need to know. You need much less knowledge to run something in Heroku than in k8s; dramatically less in the "onboard a CRUD app" case. I'd argue that running k8s effectively is no less knowledge-intensive than running VMs.
> I'd argue that running k8s effectively is no less knowledge-intensive than running VMs.
LOL - are you f'ing kidding?
Running VMs is much closer to running your app locally than containers and k8s - it's not that hard to do badly, and only marginally harder to do well.
I say it's over-rated. You don't need a huge amount of "sysadmin/devops", that's only now becoming a thing since we started calling it that. Before it used to be that backend devs just had intimate knowledge of how their service was running on a system, and most likely had to login and debug a multitude of issues with it. 99% of backend devs used to (maybe not so anymore now with "devops" et al) be more than capable of administering a system and keeping it chugging along as well as setting it up. It might not be 100% bulletproof or consistent or whatever, but more than enough for a company or service starting up.
We've lost that, and now everyone thinks we need "sysadmin/devops" for simple and moderately complicated deploys. Heck, most of the guides are out there, follow them. Also, ready-made images and docker containers are amazing these days with config built in. If you look-for or hire devops, you get K8s and all the "formal" items they'll bring with them. You don't need CI/CD for the first 6 months, just PoC the damn thing, get an MVP out and start getting users/customers, the rest will follow.
I've worked at many places where everything was run off single instance (usually windows) VMs.
These systems rarely were constrained in performance by scaling issues that would have be solved by scaling horizontally, and in many cases the added latency of doing so would have caused more problems than it would have solved.
And as you say, having everything on one or two VMs is not just orders of magnitude cheaper hosting, but also comes with benefits of much more easy debugging and performance monitoring.
These weren't tiny start-ups either, these were long running services contracted out to clients including the government and other major companies.
It doesn't scale infinitely, but I'd wager that the traffic just isn't there to justify these kinds of setups in 99% of cases, and that some of the scaling needed is because all the added latency and service discovery etc is adding overhead that wouldn't be needed without it.
I've often found it odd that people who strive for YAGNI at the code level don't apply the same to the system architecture level.
The amount of traffic you can serve on a bunch of Linode boxes is pretty high. I know this sounds like a boomer yelling at the clouds --see what I did here?
Kubernetes is the right solution to a difficult problem you may or may not have in your future.
> In the limit, there are some startups that could run production on a single Linux host
I guess redundancy is not really a thing then?
With serverless offerings you can get rather good deals, I don't think you need your own K8S cluster if you can get away with a single Linux host, but a single Linux host is pretty pricey maintenance wise compared to google Cloud Spanner and cloud run.
Most of the time it really doesn't need to be. In the end what you care about is uptime and cost. A redundant solution doesn't have a perfect uptime just because it's redundant, in fact sometimes it might have even less uptime because of failures in the redundancy mechanism. Of course if you need to be always up it might be worth it. But for a lot of situations some downtime is acceptable and most of the time it's better to be on a simpler setup that's easier to debug and less costly to maintain, than to be on a complex one that requires more maintenance and still doesn't have perfect uptime.
Also, I wouldn't underestimate the cost of maintenance in managed solutions compared to self-hosted. It wouldn't be the first time that the managed solutions screw something up or apply some changes and you don't know whether it's your fault or theirs. You also have to add the cost of adapting your solution to their infrastructure, which is not trivial either.
> A redundant solution doesn't have a perfect uptime just because it's redundant, in fact sometimes it might have even less uptime because of failures in the redundancy mechanism
I'm fairly sure that google does a better job keeping cloud spanner and cloud run working, and their redundancy mechanisms working, than whoever runs your single linux box will do.
> most of the time it's better to be on a simpler setup that's easier to debug and less costly to maintain
Keeping a whole linux box running is more complicated and requires more maintenance than Cloud Run and Cloud Spanner.
> Also, I wouldn't underestimate the cost of maintenance in managed solutions compared to self-hosted.
It is not 0, it is just that if you manage cloud run and cloud spanner you don't manage all the other things you have to manage when you self host, and managing cloud run and spanner is really not a lot of effort, it is a lot less effort than managing a standalone database at least.
> You also have to add the cost of adapting your solution to their infrastructure, which is not trivial either.
Cloud run can run stock standard docker containers, you will have a bad time if your processes are not stateless though, and you will have the best time if you have a 12-factor app, but I would not count that as adapting to infrastructure.
> I'm fairly sure that google does a better job keeping cloud spanner and cloud run working, and their redundancy mechanisms working, than whoever runs your single linux box will do.
Most of the uptime loss won't come from your provider but from your applications and configuration. If you use Cloud Run and mess up the configuration for the redundancy, you'll still have downtime. If your application doesn't work well with multiple instances, you'll still have downtime.
> Keeping a whole linux box running is more complicated and requires more maintenance than Cloud Run and Cloud Spanner.
Is it? I keep quite some linux boxes running and they don't really require too much maintenance. Not to mention that when things do not work, I have complete visibility on everything. I doubt Cloud Run provides you with full visibility.
> It is not 0, it is just that if you manage cloud run and cloud spanner you don't manage all the other things you have to manage when you self host, and managing cloud run and spanner is really not a lot of effort, it is a lot less effort than managing a standalone database at least.
Managing a simple standalone database is not a lot of effort. For most cases, specially the ones that can get away with running production on a single box, you'll be ok with "sudo apt install postgresql". That's how much database management you'll have to do.
> Cloud run can run stock standard docker containers, you will have a bad time if your processes are not stateless though, and you will have the best time if you have a 12-factor app, but I would not count that as adapting to infrastructure.
That definitely counts as adapting to infrastructure. For example, if I want to use Cloud Run my container should start fairly quickly, if it doesn't I need an instance running all the time which increases costs.
I'm not saying Cloud Run/Spanner are bad. They'll have their use cases. But for simple deployments it's more complexity and more things to manage, and also more expensive. If doing "apt install postgres; git pull; systemctl start my-service" works and serves its purpose, why would I overcomplicate it with going to redundant systems, managed environments and complex distributed platforms? What do I stand to gain and at what cost?
> > A redundant solution doesn't have a perfect uptime just because it's redundant, in fact sometimes it might have even less uptime because of failures in the redundancy mechanism
> I'm fairly sure that google does a better job keeping cloud spanner and cloud run working, and their redundancy mechanisms working, than whoever runs your single linux box will do.
You'd be surprised - the major cloud providers have outages all the time.
Google in particular will have some random backing service firing 502's seemingly randomly while their dashboards say "all good".
I've worked in DevOps for a while and if I could pay $950 to not run and maintain a server then I'd consider it money well spent.
There's always 1-2 comments in these threads that advise using a Linode VM or Hetzner dedicated server in order to save money; but they are really skipping over the headaches that come with building and maintaining your own servers.
- Are the server provisioning scripts source controlled?
- Is there a pipeline if the server needs to be recreated or upgraded?
- How is the server secured?
- Are the server logs shipped somewhere? Is there monitoring in place to see if that is working?
- Does the server support zero-downtime deployments?
- Are backups configured? Are they tested?
I imagine the answer to a lot of these questions is no and to be fair not all PaaS systems provide all of these features (at least not for free).
The server becomes a pet, issues inevitably arise and that $950 starts to look like a false-economy.
I think this all ignores the opaqueness of most PaaS providers - when everything is on a single box, you have infinite observability, standard Linux perf analysis tools, etc.
If you do it correctly, it is loads easier to reason about and understand than a complex k8s deployment.
Having worked with Kubernetes, it's great - I think even smaller setups can benefit from it and its approach (especially when using a managed provider).
But for startups and/or simpler setups, ECS/Lambda is so much less work, while usually being powerful enough.
I've looked at it a few times but haven't been able to get my head around the concept. Reading the intro documentation I haven't been able to map the concept of "nodes" to a server, database, and GPU server.
The nodes are just the individual machines in your K8s cluster.
Which one(s) run your API/server, your database, etc is basically arbitrary. You’re free to let the K8s scheduler do as it see fits, or you can get increasing degrees of control by adjusting things like node and pod affinity/anti-affinity, etc.
* You describe what type of state you need (Deployment = one or more Pods running your container, Service = expose the deployment)
* Kubernetes runs a control loop that reconciles the current cluster's state with your desired state
Nodes are the virtual machines on which everything runs, you don't need to interact with them unless you have special requirements (e.g. GPU instances). In that case, you annotate GPU instances with some type of label (like `has-gpu=true`), then in your deployment you add a node affinity saying it needs to run on nodes that have that label. Kubernetes will schedule it for you if there's any node matching.
Our startup has a webapp that acts as a UI for a machine learning application. We have two different types of heavy workloads (ML and something else). The workers for these run on Kubernetes, which makes them easy to scale (which we do a lot, automatically). The app itself doesn't run on Kubernetes yet (simple VM). It would be better if it did, though! We keep building a lot of functionality ourselves (e.g. deployment without downtime) that would be easier with K8s. I think what you really want to avoid is Microservices, not K8s. K8s gives you many advantages, such as the entire config in Git, containerized workloads, deploy management, etc., which sooner or later, you'll find yourself wanting. Depending on how you get started, the learning curve can be steep, and some of the tools are not mature but supposedly used by everyone (ArgoCD for deployment?).
What is it giving you over an ec2 auto scaling group or ECS/Fargate?
Both can scale as much as you like, your config can live as cdk/cloudformation/terraform code?
I'm not an expert on auto scaling groups, but we have used the Google Cloud equivalent for a time. The biggest issue for us was that deployment is not as easy. Can you update the software on the ec2 instance without turning it on, for example? With K8s, we can leave the deployment scaled to 0 and just patch the image of the deployment to perform a release while the workers are all shut down. Similarly, we don't have to write code to wait for the workers to finish their current job before we shut them down in order to be replaced by a newer version; this is all managed by K8s, and the configuration for it lives in Git.
As others have already pointed out, it is also important for us to remain independent from Google Cloud / AWS.
Apache Spark can use Kubernetes as a scheduler out the box. I don’t know if op is using Spark.
A lot of data tools are starting to target Kubernetes directly as a runtime so using them with GKE/EKS is a bit simpler as it’s officially supported, allows to run locally and on the cloud with no vendor lock in.
ECS in a scaling group works well if your app is stateless but as soon as you scale workers dynamically, do service discovery, orchestration, you end up building some of the features Kubernetes provides.
k8s can be easy to set up, and if you know how to use it then you should. You shouldn't go to the trouble of learning it, but if you do know it already, then there is no reason not to go with services like gke autopilot.
I don't use it much right now but I've found kubernetes (at least the managed service variety) fairly staright forward compared to some of the altearntives
it beats the pants off the the random rubricks of ansible and puppet projects I've run into the past anyway
I feel like this is probably good advice for startups and AWS-using companies. But there remains a lot of us who have, for many reasons, some good, some bad, a lot of investment in physical infrastructure already. For those, the Hashicorp or Kubernetes stack makes a lot of sense, if only to help standardise the insanity of what you’re running in-house.
Sort of agree. There are caveats of course but I tend to think if you can just start with a bunch of manually setup instances and roll from there. If you find yourself spending hours per day starting up and shutting down instances manually then yes, automate that part.
Nothing I've ever done has taken off at all so automating DevOps would have been a total waste of time. Then again the projects themselves have been a complete waste of time.
I think it depends on whether you automate it so that it goes out of your way so you can spend more time on things crucial to your work - or you're just pontificating because you hate your job anyway.
Why should using different languages for front-end and back-end be a problem? I rather think that it is better to use languages that are appropriate for the given problem. It is not premature optimization to have parts of a back-end implemented in C/C++/Go/whatever else if high performance is needed. It would rather be a waste of resources, money and energy not to use an high-performance language for high-performance applications. Of course using the same language for the front-end might make no sense at all.
I think the argument was/is not "its a problem we cannot have javascript" but "if we can have javascript everywhere, we only need to hire javascript devs and only need to care about javascript tooling", which is a fair point.
That does ignore the fact that an experienced frontend JS dev is not necessarily also a good productive backend JS dev, but at least they know the language basics, can use the same IDE, etc.
If thats worth it is something that depends on what you try to achive, I guess. I personally would not pick JS (nor TS) for the backend.
Which is ironic as the DOM interface was designed as an abstract interface (the IDL used in the spec is more interested in compatibility with Java than JS).
In practice though the main reason is that to have decent DOM bindings you need to stabilize many other specs first (unless you do a ultra-specific DOM-only extension, but nobody wants that)
I have been very successful in replacing a JS browser application with Rust. It has been great, because Rust is much easier to change, and since the application is complex, I need to change it a lot.
But I wouldn't recommend it in general, because JS (and for slightly more complex things, TS) is much easier to create because of all the old Rust features that surface every time you mention it. And most GUI code is simple enough that you only have to write once, maybe fix a few details, and forget about it.
Wasm would be much more compelling if it was target by higher level languages.
I think this has been a huge failing of our industry of late.
The rise of the "fullstack developer" has mostly reduced quality across the board. When you hire a "fullstack developer with 5 years experience" you aren't getting someone who is as good as a frontend developer with 5 years AND a backend developer with 5 years but someone that adds up to 5 years split between those 2 endeavors but probably with less depth as a result of switching.
(as a side note I also think it's contributed to developer title inflation)
Learning a new language and its tooling is comparatively easy compared to learning the required domain knowledge to be effective in a new area. i.e transition from frontend -> backend or visa versa has very little to do with the language or tooling.
Your average frontend dev doesn't know squat about RDBMS schema design or query optimisation, probably aren't familar with backend observability patterns like logging, metrics and tracing, most likely have very little experience thinking about consistency and concurrency in distributed systems, etc, etc.
Just like the backend dev is going to struggle with the constraints of working in a frontend context, understanding UI interactions, optimizing for browser paint performance, dealing with layout and responsiveness, etc.
Meanwhile if you know say Java and some scripting language, say Python and you end up a new job doing backend in JS it's not going to take long for you to pick up JS and hit the ground running because you are going to encounter exactly the same stuff just different syntax and runtime.
Backend being substantially divorced from frontend isn't a bad thing, it's generally a good thing that results in nice clean separation of concerns and healthy push-pull in design decisions about where certain logic and responsibilities should lie etc.
Most 'fullstack' positions don't have nearly the complexity where worrying about concurrency etc. is actually that relevant. The idea that most frontend devs have any knowledge about optimising for browser paint performance beyond using the correct CSS or framework is funny ;-)
I know, we have seen insane developer inflation the last 10 years.
The bar is generally just a lot lower now across the board. We have people with the title "Senior Engineer" that still regularly need mentorship and peer-programming sessions.
The title I have now (Principal Engineer) I feel like I don't deserve, it was reserved for folks I looked up to when I was starting and I don't think I have reached their level. Yet at the same time it's essentially necessary to distinguish myself from what is now considered "Senior".
I have a separate rant for the lack of proper juniors and lack of structured mentoring etc but that is for another day.
> The rise of the "fullstack developer" has mostly reduced quality across the board.
Having a solo developer that can do it all well enough also allows useful products to reach users faster (and for me it's really about solving user problems, not just making money), without getting derailed by communication overhead, bureaucracy, fighting within the team, etc. Just make sure you don't pick a developer who cares too much about things that mostly matter to other nerds, because then they might get derailed with premature optimization. Yeah, I've been there.
The chunking unit is programming languages which means as long as you can call the task the specific programming language, people will believe it's the same thing.
In reality you have a hammer expert and are in the business of making everything look like a nail.
So now we have complicated bloated applications of JavaScript in places where it's absolutely inappropriate and you need far more skill to navigate those waters then someone who uses more appropriate tools.
It's a perversion in the name of simplicity because we're forcing too coarse of a model on too fine of a problem and as a result everything explodes in complexity, takes too long, is too expensive and comes out working like trash.
We could do better but first we have to eat crow and admit how foundationally wrong we were and frankly things aren't bad enough to make that mandatory so the circus continues just like it did when we tried to make everything OOP and it made the same smell.
They're useful tools, but that's their perimeter; they aren't zeitgeists... well they shouldn't be.
The impression I've gotten from some of my co-workers is that in bootcamps and college they only learned one language, remember the time and effort that went into that, and assume learning another language will take the same time and effort. Because they haven't really put effort into a second one, they don't yet realize just how much conceptually transfers between the languages.
While concepts transfer over between languages the language is only 10% of it. The rest of it is the standard library, ecosystem, buildsystem and all kinds of intricacies you have to know about and what not which is specific to the language (ecosystem).
There is also an issue with conceptual leakage, most noticeably I've found with devs well versed in one language bending another language into behaving like the former.
Agreed. You see this a lot with people with a C# and/or Java background using TypeScript with annotation based decorators and libraries/frameworks that implement dependency injection, etc. They don't really embrace the nature of the new language and ecosystem. And I can sympathise, it is simpler to do things as you have been doing them before.
You also see it when people who've mostly done imperative programming tries their hands with a lisp or some ml based language. I've been there myself. You still find yourself using the equivalent of variable bindings, trying to write in an imperative rather than declarative style.
I guess when trying to learn a new language in a different paradigm you also need to unlearn a lot of the concepts from the former language.
Well, that is the reason why outside HN bubble, Angular still wins the hearts of most enterprise consulting shops, as its quite close to JEE/Spring and ASP.NET concepts.
This is an underestimation of how hard it is to learn to program.
Someone learning a first language isn't just learning a new language: they're learning how to program. It's a new profession, a new hobby, a new superpower.
The rest of the stuff (standard library, ecosystem, buildsystem and all kinds of intricacies) is just a mix of trivia and bureaucratic garbage you gotta fill your brain with but will all be replaced in 10 years anyway. Sure it takes time but it's nowhere near as important as actually knowing how to program.
Even changing paradigms (imperative to functional) isn't as hard as learning the first language.
I think a lot of people here have been doing this programming thing for so long we've forgotten we once had trouble understanding things like:
x = 1
for i in [1, 2, 3] {
x = i
}
What is the value of "x" at the end? Assuming block scope, it will be 1, or assuming it doesn't have block scope (or that it uses the already defined "x" in this pseudo-example) it will be 3.
A lot of beginning programmers struggle with this kind of stuff, as did I. Reading this fluently and keeping track of what variables are set to takes quite a bit of practice.
I've been hired to work on languages I had no prior experience on, and while there was of course some ramp-up time and such, overall I managed pretty well because variables are variables, ifs are ifs, loops are loops, etc.
The concepts still roughly translate and help, though. You have projections/mappings in FP. Or if you want to go deeper, recursion. Understanding loops before those will definitely make them easier since they are equivalent.
The argument that learning a second language is as difficult as learning the first doesn’t really hold water in practice as well, lots of people have done so.
I'm not saying there are zero differences or that $other_language never has any new concepts to learn, but in functional languages variables are still variables, functions are still functions, conditionals are still conditionals, etc. Important aspects differ, but the basics are still quite similar as is a large chunk of the required reasoning and thinking.
> ...the standard library, ecosystem, buildsystem and all kinds of intricacies you have to know about and what not which is specific to the language (ecosystem)
IME a lot of the conceptual stuff transfers pretty well there too.
That's sad, JavaScript was already not great for front-end and we now get it in backend and even the edge.
Most job offers are for a mythical full-stack developper that'll master web design, CSS/HTML, front-end interactions and code, networking, back-end architectures, security,... You end-up with people who don't have time to get enough expertise and write and build clean stuff. Hacking poor JavaScript code everywhere.
With the same language, you may think you can somehow reuse stuff between front and back. A bad idea in most project, will typically create more problem than it'll solve.
I'm far from a full stack developer, but really how much code would actually be common across the front and and back end? I would have thought maybe some validation code, not sure how much else?
Sounds about right. I have an API service written in Go for which we also offer a client library in Go. We moved the common ground between the two into a separate module to avoid code duplication. That module ended up having the type declarations that appear on the API in JSON payloads, plus some serialization/validation helper methods, and that's it.
True, that can be a valid use-case. Another solution would be to leverage language agnostic API discovery solutions, generating the serialisation and part of validation code (swagger/open api/...). This way you provide a nice documentation and don't have to care about language differences while keeping good productivity.
If you do server-side rendering with something like nextjs, then its quite a lot of code.
With trpc you can share types without going through an intermediary schema language (https://trpc.io/) although I think it would've still benefitted from separating the schema from the implementation (hope you don't forget the type specifier and import your backend into your frontend by accident)
For business logic as always the anwwer is "it depends". Does your app benefit from zero-latency business logic updates? Do you need rich "work offline" functionality combined with a rich "run automation while I'm away" for the same processes? etc.
Even when using Nextjs, it's still pretty common to have a separate API backend that doesn't also have to be in Javascript/Node. There are parts of backend code (HTML generation) that very strongly benefit from being unified with the frontend code, and there are parts of backend code (like database persistence code) that much less strongly if at all in some cases benefit from unification with the frontend code. Many people split these parts of backend code up between a separate Nextjs backend service and API backend service, and Nextjs lends itself well to this.
I'm a big Nextjs fan; I just think it's useful to emphasize that using it doesn't necessarily have to mean "only javascript/typescript on the backend".
> While Remix can serve as your fullstack application,
Isn't that where most teams should start, and where they should stay unless they have a really good reason to get more complicated? This is what I was thinking of while reading the GP comment about Next.js.
Serialization/deserialization and templates are a huge pain to keep identical. The rest, not so much.
So, if you can keep all of your templates on a single place, and don't have a lot of data diversity, you won't have a problem. But if that's not your case, you do have a problem.
Personally, I think the frontend code should be more generic on its data and organized around operations that are different from the ones the backend sees. But I never could turn that ideal into a concrete useful style either.
In my mind it is not code reuse between frontend and backend, but expertise and standard library reuse that is the winner.
Better to have a full stack developer that can concentrate on becoming an expert and fluent in just one language rather than being kinda-ok in 3 or 4 IMHO.
It's not a question of "if" a developer can learn a new language, but of "how long" it takes. And it's not a question of how long it takes to become moderately productive, but how long it takes to reach an expert level of proficiency.
The struggle is not learning a new language nor becoming fluent.
The real expertise is being a front-end expert, authoring efficient architecture around user interactions, browsing the MDN without effort, mastering the DOM... Then, being a back-end expert, knowledgeable on scaling, could architectures, security issues, being able to author good API design...
If you can be considered an expert in all of this, and also edge computing, I don't think switching language would be an issue for you. Language is a tiny fraction of required expertise and you might be more productive by switching to an appropriate one.
It needs to be all Javascript, because Javascript in itself is already at least 5 languages, with ES6, browser runtimes, Node, ESM and CJS, and TypeScript, some CoffeeScript remenants, and the list doesn't end. There is no end to complexity.
Most developers don't seem to want to learn more than "one" thing. Once they do one tutorial, it seems they're done for life with learning.
And they don't really have time to learn new stuff, as they spend too much of their time on Hacker News complaining there's too many new frontend frameworks or something like that.
The sooner they accept that there were no such thing as one language to rule them all, the better developer they become. I have never seen the "isomorphic" claim to be seriously analyzed. One aspect example is how much logic behind the wall is overlap with the optimistic ui logic? Some logic may seem reusable but it may not. It's insane when I saw a popular js framework author on twitter said javascript is a language of the web, other backend languages are not (not exact words). Like WTH.
It's about hiring talent that can drive customer value without concern for anything else and the business can eventually hire more experienced individuals to fix the mud mountain if there's market fit and continued need.
It is extremely easier to find more affordable talent that has come out of a boot camps knowing javascript but more specifically "react/nodejs" which lets them work both frontend and backend. They rarely know best practices, how to properly debug & troubleshoot a problem that isn't a Google search away, etc, but they will be hungry and work their butts off to ship their CRUD style features.
You can't really tell for a Web App whether it'll be faster in JS or Python, but you can definitely expect a Computer Vision application with lots of heavy number crunching to be a lot faster in C++ than in Python. We have actually also made comparisons, and even if you use things like numpy and Python bindings for OpenCV, you won't reach the speed that a C++ application achieves easily without optimization.
That depends a lot on what that CV application does and what hardware it runs on. Naive C++ (that runs on CPU) is usually much slower than using Python as glue for libraries that run on GPU.
For a lot of applications there are pretty straightforward calculations to meet a desired frame rate.
Also I've seen UX research that established some guidelines for how long something can take while still feeling responsive allowing a user to remain focused.
Knowing you can achieve the required/desired performance in any given language is mostly a matter of experience solving similar problems.
> It is not premature optimization to have parts of a back-end implemented in C/C++/Go/whatever else if high performance is needed.
But the overwhelming majority of the time you don't need it, at least not yet. I would say that unless you have actual evidence that your other language would not be adequate - i.e. an implementation of your system or some representative subset of it, in your main language, that you spent a reasonable amount of time profiling and optimizing, and that still proved to have inadequate performance - then it is indeed premature.
That assumes it’s harder to build it in the other language though. Maybe if that language is C then that will the case, but building in say Go may be just as easy as building in JavaScript (or close enough that it doesn’t really matter), whereas rewriting it later would be a massive undertaking.
This I very different to say starting a with a micro service architecture which imposes relatively high overheads, with little benefit as splitting up a well designed monolith is easy.
> That assumes it’s harder to build it in the other language though. Maybe if that language is C then that will the case, but building in say Go may be just as easy as building in JavaScript (or close enough that it doesn’t really matter)
The cases where Go is significantly faster than JavaScript are vanishingly small.
> whereas rewriting it later would be a massive undertaking.
This is vastly overstated IME. Porting existing code as-is between languages is actually pretty easy.
> This I very different to say starting a with a micro service architecture which imposes relatively high overheads,
Disagree. You're imposing a huge overhead on hiring (if you want people with both languages) or on people's ability to work on the whole system (if you're happy hiring people who only cover one side or the other). Debugging also gets a lot harder. There's essentially twice as many tools to learn.
> This is vastly overstated IME. Porting existing code as-is between languages is actually pretty easy.
Yep. We always implemented (parts of) embedded software in Java first and later in JS and then port it. If no additional functionality is added, this is trivial and saves a lot of work and errors as you already tested, debugged and fixed the logic parts.
The experts in how to profile/optimize/etc a system aren't going to be JS devs though. They're going to be people who are used to dealing with systems that need to be written in languages from the machine-code-compiled lineage.
Which is to say ... while JS developers do know how to profile code, people who are routinely exposed to this problem are not going to be JS developers. The people who are good at identifying when a system needs to be re-written in Go for performance reasons are probably already Go developers and people who have lots of experience writing performant code.
Plus writing in that sort of language from the start means there is a chance to segue into performant code without needing to rewrite things in a new language.
> The experts in how to profile/optimize/etc a system aren't going to be JS devs though. They're going to be people who are used to dealing with systems that need to be written in languages from the machine-code-compiled lineage.
Sure they are. The skills aren't really language-dependent, and nowadays machine code is so far away from the actual hardware behaviour that it doesn't actually help a lot. Besides, the biggest speedups still come from finding errors or inappropriate algorithms or datastructures, and that's if anything easier to spot in a higher-level language where there's less ceremony to get in your way.
> finding errors or inappropriate algorithms or datastructures
when I interview developers, folks coding in Java can usually tell me what an appropriate datastructure would be, while half the JS developers can't explain how an array is different from a linked list. Because most of the time JS developers. dont have to think about it much.
That makes sense assuming you have already written your backend in JS. In that case, yeah the bar for rewriting in another language should be high (as it should be for any ground-up rewrite). But it's not a "premature optimization" when you are deciding on the tech stack to begin with.
Using a second language for performance (which is what the post I replied to was suggesting) is a premature optimization - you're paying all the costs of having your code in two different languages, for a benefit that will only start paying off when you're much bigger if at all.
We try to limit the amount of languages we use, but we have high performance Computer Vision code that is written in C++, we're interfacing that with Python for simplicity, and a web app in JS. Right tool for the job!
Sounds like "consultant talk", too much fluff and big words but no nuance
"Using different languages" is just Tuesday in most places. Sure, don't use languages needlessly, but it's not a big hurdle unless you're just a "nodejs bro"
> It would rather be a waste of resources, money and energy not to use an high-performance language for high-performance applications.
Given that probably most developers support the push to tackle the climate change, they seem to be making no effort to ensure their apps execute in as short time as possible using as little resources as possible.
You would expect that people would actually embrace doing things in C or Go to save energy.
Maybe cloud providers should think of showing the carbon footprint as a first class performance metric.
Just like with code optimization we should first make sure making code run slightly more efficient really has an impact on the climate. Because I very much doubt it does.
When you want to save the climate there are many, many low-hanging fruits. The choice of programming language is likely not one of them, as much as I like efficient code.
Honestly these are not optimizations at all, but rather architectural decisions. Large architectural decisions are generally best made at the outset, based on problem domain analysis. They are costly to change later.
Like so many others, the author appears to be latching on to the phrase "premature optimization" as a popular buzzword (buzz...phrase?). This is so far from what Knuth actually wrote in his book that it hurts.
If you are building a SaaS company and build your site in RoR, but also have experience in say Go, and decide 'hmm instead of using RoR ill use Go for this backend thing so its faster'
That's fine.
Premature optimization would be saying:
I don't know Go/C++/C... but I know its fast, so instead of using what I know to get up and running quickly, i'll waste time building it in something I don't know which probably wont be as fast as doing it in something I know well.
The thing is, when you're building it, no one is using it! If it's slower in RoR than Go, who cares, get it up and running, and fix it later when people are actually using it.
It depends how experienced you are with it. If your team has no experience with it, don't do it. I'm pretty experienced, having set up a bunch of infras, so I'd run a blog on it now
Indeed - having used it for few years, I'd start the initial infra (for testing/playground/whatever) as small k8s on a local VM or similar solution so that we could prototype against the target abstractions from day one.
Much, much less work to then move the deployments elsewhere, including building and teardown of integration testing environments.
When I get a contract to move a startups infra to k8s the majority of the work is actually untangling the services and making them accept configuration and communication in a sane way, the actual yamls are easy. Starting with k8s actually enforces some structure and standards and makes for a much more friendly environment to work in, usually you can run the whole infra on your laptop
K8s signals the team values infra and sees it as an important part of delivering their product.
It's an investment like any other and for some teams it makes sense to set infra on a solid path sooner rather than later.
You can move fast on things like serverless but the limitations of those systems can force architecture decisions that are sub-optimal whilst control of your own infrastructure can allow broader interpretation of problems and enablement of more wholistic solutions.
i.e it opens up building things "the right way" sooner in many cases and that can save a ton of time and frustration.
Also in the case where it's obvious you will outgrow serverless it also saves on a sometimes painful migration effort.
> K8s signals the team values infra and sees it as an important part of delivering their product.
But isn't "infra[structure]" what you run your product on, and not part of the product itself? If your "product" is so big it needs stuff like that built into it, then it feels to me that by defintion that stuff isn't "infra".
All in all, your sentence feels like "The Excel-Dell Humongo-XPS bundle signals that Microsoft values infra and sees it as an important part of delivering their product" to me. No, that only signals that A) Dell and Microsoft are using the dominant marketing position of Excel to milk money from naive customers, and B) If Excel can't run on more modest hardware, it's an unnecessarily bloated and inefficient product; look around for another spreadsheet.
First of all k8s isn't bloated, it's actually surprisingly compact if you think about what you are getting.
Second it doesn't imply the product -needs- it. Anything you can run on k8s can be run on bare boxes.
Just doing so probably means you either lack the scale to need k8s (i.e my spreadsheet doesn't have many users) or you lack the competence to understand you have reached the scale that it's required (i.e your spreadsheet is run by n00bs).
So it's much more of a positive than negative signal if you are seeking a vendor.
I think the product relationship thing is a poor analogy from that side though, it -mostly- matters for people that want to work there on/in/around the product itself.
As a primarily infra guy I have a preference for k8s because it allows me to have some reasonable set of services I can rely on being there if a place is "using k8s". If not I just have to hope either the existing infra team is competent enough to have things like service discovery, secret management, process monitoring, etc under control or that they are amenable to efforts to get it under control.
If I am approaching it from a backend developer perspective where most of that isn't my problem then it gives me some confidence the infra team has some idea what they are doing and that I won't need to deal with some sort of bespoke deployment system that has endless quirks and company-specific bugs that I shouldn't need to concern myself with.
Generally k8s just eliminates uncertainties by increasing standardisation so I don't need to watch people solve the same problem poorly over and over again for no reason at all.
> Imagine spending a lot of time and money picking out the best possible gear for a hobby before actually starting the hobby.
Haha, this is exactly what “hobby” means to a lot of people. Less judgmental: thinking and dreaming about the right tools in disproportion to the need is something people do a lot, presumably because it is a source of joy.
That bit was pretty interesting to me as well, because that's exactly what many people do. I don't agree that this is what "hobby" means, it's more that we tend to try to buy our way into a life style.
I feel that Kubernetes is exactly that for a large number of developers, operations people less so, they want to be part of a professional, world class, trendy environment/community. So they try leverage Kubernetes to elevate them to this Silicon Valley type tech company.
There's absolutely value to be had from Kubernetes, but I agree with the article, it's not for the majority. Except I think there is one valid reasoning: If you purely see Kubernetes as tooling for deployment, it's hard to present any good alternatives for deploying to VMs or physical hardware.
You know these types of people who like to wear army clothes to look bad-ass even though they are wimps?
I see it the same type of folks as ones using FAANG tooling :) just so they can feel better about their job or more important when they work on business line CRUD.
I am working on business line CRUD and I like it with boring servers :)
The flipside of this philosophy is that you have to be able to revisit your decisions when your requirements change.
This is a superpower that you want to have individually as an engineer but you need organisationally as a culture too. If you have that power then you when you need to write some similar code you can copy and paste, because you can trust that when you have a third use you will actually refactor properly. It means that you can get the service up and running quickly on a box you've got, because you can trust that if the service actually gets used your org will allow you the time to put it on the cloud properly.
It's the most important part of agile - make the right decision for what you've got now, and then make a different one later.
I don’t get these complaints AT ALL. I don’t use kubernetes, simply because I am running apps in managed environments, and have been using docker-compose with vscode remote to emulate those environments. But being able to define your resources and how they are linked via a schema, makes sense even from a dev perspective. Isn’t that all that kubernetes is doing at it’s most basic? Sounds like that saves time to me over manually setting everything up for every project you work on.
From what I've seen with Kubernetes, the problem is that in order to define the resources and links with a schema, it needs several abstractions and extra tooling. So you not only need to define the schema (which isn't that trivial) but also understand what Kubernetes does, how to work with it, and how to solve problems. It's a tool that adds complexity and difficulty, if you're not using the advantages it provides (mainly multi-node management and scale capabilities) you're just handicapping yourself.
This is a common misconception. k8s isn't about scale, multi-node or even reliability/resiliency. We had solutions for all of that before it came along.
It's about having a standard API for deployment artifacts.
The k8s manifests are trivial (if verbose) the complexity comes from running the underlying layer which when you are small you simply outsource to AWS or GCP.
There is some k8s know-how that is table stakes for a good experience, namely knowing what extra stuff you need on top of a base k8s cluster. i.e certmanager, external-dns.
Overall it's a lot less required knowledge than it takes to manipulate lower level primitives like GCP/AWS directly or be capable of setting up standalone boxes.
Before anyone starts with "but serverless!" you need to consider the spagetti you end up with if you go down that route, API Gateway, 100x Lambdas and thousands upon thousands of lines of Terraform boilterplate does not a happy infra team make.
It would be great if otherwise smart people Also learn to know when what’s trivial to them is not trivial to the masses. K8s is not trivial, not to me at the least. I’m not some super duper engineer but I do alright? That’s all I can say at the least.
Have you made a good faith attempt to use k8s? Or are you just regurgitating how hard it is to use based on what you hear on the Internet?
My experience is even mediocre engineers are capable of understanding k8s concepts with minimal assistance assuming things are either sufficiently standard (i.e google-able) or well documented with any company specific practices.
I have made a good faith attempt at using it, yes. Is spending an entire week on it good faith?
I did get everything running but also saw so many settings and features I had to take on faith as being handled without understanding it. I just chose not to bother further because this seems like a minefield of problems in the future as we on boarded other engineers. The number of times I’ve seen production deployments go down due to k8s misconfiguration on other teams has only validated my concerns.
So what you are saying is everything worked? That sounds like k8s did its job.
You aren't meant to fully understand it in a week, anymore than you are expected to fully understand all of sysadmin in a week.
Just because you don't know what every single directive in nginx config does doesn't mean you can't use it effectively and learn what they mean when the time comes.
k8s isn't much different. You don't need to know what a liveliness probe is the first time you use it, you can learn about it as you go (most likely when you run into a badly behaved program that needs forced restarts when it hangs).
Ofcourse if you are running it yourself that is entirely different, you really do need to know how it works to do that but you should be using hosted k8s unless you have hardcore systems folk and actually need that.
You’re right that nginx is also complicated , but seems like we are talking about different alternatives.
The solution I went with, given my limited knowledge of some of these things, was to use elastic beanstalk. You write your flask application, upload the zip and that’s it pretty much. You get a UI to do all the configurations and for the most part nothing there is hard to decipher or google. The only hiccup might be when you’re trying to connect it to RDS and to the external internet but even that is straightforward as long as you follow the right instructions. We run apps that power entire SaaS organizations and this system seems to be more than sufficient. Why complicate further? We have other fish to fry anyway.
I am not a SDE but I did try one weekend to set-up a self managed k8s cluster in the previous company. I did have some previous knowledge of using k8 in GCP. Although when trying to set it up on a completely new cluster outside of GCP, I did run into some issues where I felt I was out of my depth. I think unknowingly I exposed whole of the cluster to the public internet.
On the other hand, with docker swarm, I built a cluster in less than a day. So yeah setting up k8 isn't as trivial as you are making it out to be.
> Overall it's a lot less required knowledge than it takes to manipulate lower level primitives like GCP/AWS directly or be capable of setting up standalone boxes.
I do not agree with this. I have tried to start Kubernetes and it's far more confusing to set up than just setting up something on a standalone box, mainly because it looks like the set of knowledge I need to set up anything on Kubernetes is a superset of what I need to set up the same thing in bare Linux.
It's really not unless you are doing a half-assed job of setting up bare boxes.
Lets see to get something even half-reasonable on a bare box you need the minimum:
process monitor: For this you can use systemd, supervisord, etc.
logging: rsyslog or similar.
http reverse proxy: nginx or haproxy
deployment mechansim: probably scp if you are going this ghetto but git pull + build and/or pull from s3 are all common at this level plus a bunch of bash to properly restart things.
backups: depends on persistence, litestream for sqllite these days, pgdump/whatever for RDBMS, wal shipping if you are running the DB on your own boxes.
access control: some mechanism to manage either multiple users + homes or shared user account with shared authorized_keys.
security: lock down sshd, reverse proxy, etc. stay on top of patching.
So are bare minimum there is a ton of concepts you need to know, you are essentially regressing back into the days of hardcore sysadmins... just without all the hardcore sysadmins to do the job correctly (because they are now all busy running k8s setups).
Meanwhile to deploy a similarly simply app to k8s (assuming you are buying managed k8s) you need to know the following things:
deployment: Describes how to run your docker image, what args, how much resources, does it need volumes, etc. You can go ghetto and not use service accounts/workload-identity and still be in better shape than bare boxes.
service: how your service exposes itself, ports etc.
ingress: how to route traffic from internet into your service, usually HTTP host, paths etc.
Generally this is equivalent but with way less moving parts.
There is no ssh server, patching is now mostly just a matter of pressing the upgrade button your provider gives you. You now only need to know docker, kubectl and some yaml instead of systemd control files, sshd_config, nginx.conf, bash, rsync/tarsnap/etc.
K8s reputation as being hard to run is deserved.
K8s reputation as being hard to use is grossly undeserved.
I mean, container+registry is possibly more complex to start and maintain than just creating a service file with SystemD. Deployment can be as easy of "systemctl restart service", and seems to me that configuring all of those resources in Kubernetes is far more difficult that just setting up a simple service on a bare box. Not to mention that you could use Docker too.
And by the looks of it, ingress doesn't seem trivial, not only you need to understand how NGINX works for reverse proxy but also understand how Kubernetes interacts with NGINX. You also ignored backups for Kubernetes, if you're just doing full disk backups you can do that with regular boxes, specially if they're just VMs.
> There is no ssh server, patching is now mostly just a matter of pressing the upgrade button your provider gives you.
No SSH server might be an advantage for you, but for me it means it's far more difficult to know what's happening when Kubernetes doesn't do what I want it to do.
> You now only need to know docker, kubectl and some yaml instead of systemd control files, sshd_config, nginx.conf, bash, rsync/tarsnap/etc.
But I still need to know how to set up all applications I use, which is the most important part. And understanding systemd service files, sshd_config, nginx.conf and rsync is far, far easier than understanding Kubernetes. Kubernetes manages those things so you actually need to understand both the underlying concepts and also the abstractions Kubernetes is making.
You are also comparing self-hosted with managed, and then only mentioning the disadvantages of self-hosted. Of course with a managed setup you don't need to worry about SSH, but you have to worry about actually dealing with the manager service. If you were to set up self-hosted Kubernetes you'd still need to learn about SSH, or about account management. That's not something about Kubernetes but about self-hosted or managed.
You don't run your own registry (except in very rare circumstances where that makes sense). You author Dockerfile (or better yet use a tool like jib that creates containers automatically without even a Docker daemon), then you push to hosted registry.
Ingress is trivial to use. Internally it's less trivial but you don't need to peek inside unless you manage to break it which generally speaking, you won't; even if you think whatever you are doing is very special it probably does it already. That is the benefit of literal thousands of teams using the exact same abstraction layer.
You don't have SSH (this is objectively good), instead you can get exec on a container if you need it (and that container contains a shell). You can now control on a fine grained and integrated fashion exactly who is allowed to exec into a container and you get k8s audit events for free. (and if you are using a hosted system like GKE then it automatically flows into said providers audit system).
The point is you can buy managed k8s, there isn't an equivalent for old school sysadmin that doesn't amount to outsourcing to a body shop.
Given that you haven't managed to identify any of k8s real downsides here they are:
It's a bunch of yaml. Yeah I don't like that either but there are tools like Tanka that make that suck less.
It's complex under the hood. When your process runs its in a network, fs and pid namespaces and you have cgroup managing resources. There is probably some sort of overlay network or some other means of granting each container an IP and making that routable (BGP, OSPF, etc).
Interaction between ingress, service and pods is through an indirection layer called endpoints that most people don't even realize exists.
Scheduling is complex, there are tunables for affinity, anti-affinity, it also needs to interact with cluster autoscaling. Autoscaling itself is a complex topic where things like hot capacity etc aren't exactly sorted out yet.
QoS, i.e pod priority isn't well understood by most. When will a pod be pre-empted? What is the difference between eviction/pre-emption etc? Most people won't be able to tell you.
This means when things go wrong there are a lot of things that could be wrong, thankfully if you are paying someone else for your cluster a) it's their problem b) it's probably happening to -all- of their customers so they have a lot of incentive to fix it for you.
Multi-cluster is a mess, no reasonable federation options in sight. Overlay networking makes managing network architecture when involving multiple clusters more difficult (or external services outside of said cluster).
Ecosystem has some poor quality solutions. Namely helm, kustomize, Pulumi, etc. Hopefully these will die out some day and make way for better solutions.
Yet for all of these downsides it's still clearly a ton better than managing your own boxes manually.
Especially because you are -exceedingly- unlikely to encounter any of these problems above at any scale where self-hosting would have been tennable.
I think the best way to think about k8s is it's the modern distributed kernel. Just like Linux you aren't expected to understand every layer of it, merely the interface (i.e resource API for k8s, syscalls/ioctl/dev/proc/sysfs for Linux). The fact everyone is using it is what grants it the stability necessary to obviate the need for that internal knowledge.
Just want to note that we have a mix of ECS and stuff still running on bare VMs using tools like systemd, bash script deploys, etc..., and I 100% agree with you. Once someone understands container orchestration platform concepts, deploying to something like k8s or ECS, is dead simple.
> You don't run your own registry (except in very rare circumstances where that makes sense). You author Dockerfile (or better yet use a tool like jib that creates containers automatically without even a Docker daemon), then you push to hosted registry.
Dockerfiles aren't trivial. I've helped migrate some services to docker and it's not "just put it in Docker and that's it". Specially because most applications aren't perfectly isolated.
> Ingress is trivial to use. Internally it's less trivial but you don't need to peek inside unless you manage to break it which generally speaking, you won't; even if you think whatever you are doing is very special it probably does it already. That is the benefit of literal thousands of teams using the exact same abstraction layer.
Literal thousands of teams also use Nginx and still doesn't mean that there aren't configuration errors, issues and other things that are to debug. Not to mention that a lot of applications can run without requiring a reverse proxy.
> You don't have SSH (this is objectively good), instead you can get exec on a container if you need it (and that container contains a shell). You can now control on a fine grained and integrated fashion exactly who is allowed to exec into a container and you get k8s audit events for free. (and if you are using a hosted system like GKE then it automatically flows into said providers audit system).
And that's cool if you need that, but "ssh user@machine" looks far easier.
> The point is you can buy managed k8s, there isn't an equivalent for old school sysadmin that doesn't amount to outsourcing to a body shop.
Look, the other day I had to set up a simple machine to monitor some hosts in a network. I've Ansible to automate it, but ultimately it boils down to "sudo apt install grafana prometheus; scp provisioning-dashboards; scp prometheus-host-config; scp unattended-upgrades-config" plus some hardening configs. I don't need redundancy, uptime is good enough with that. If Prometheus can't connect to the hosts, I can test from the machine, capture traffic or whatever and know that there isn't anything in the middle. If Grafana doesn't respond I don't need to worry whether the ingress controller is working well or if I missed some configuration.
Could I run that with Kubernetes? Well, managed k8s is already out of the window because the service is internal to an enterprise network. And self-hosted kubernetes? I still need to do the same base linux box configuration, plus setting up k8s, plus setting up the infra and networking with k8s plus the tools I actually want to use.
And that's the point. I am not trying to identify any k8s downsides, what I said that setting up k8s looks far more confusing than setting up a standard Linux box, and by the things you say I am even more convinced than before than it is indeed. I still need to understand and run the programs I want (which is most of the cost of deployment), but I also need to understand Kubernetes and, if I am using a managed platform, I need to understand how the managed platform works, configure it, link it with my system... In other words, not only do I need the general knowledge for application deployment but also the specific implementation in Kubernetes.
“So you want to run a bunch of stuff on one computer, why?”
In a quest to get closer to the metal, Kubernetes keeps you far away, which is the opposite of what any production service should want.
What is the purpose of adding layers when uni-kernels and eco-kernels give you better isolation and better performance?
Your cloud provider already runs your virtual machines OS on a hardware hypervisor. Then running Kubernetes on top of the OS and then a zillion containers on it is a recipe for poor performance.
What is the logic here that I am clearly missing?
Cloud providers native Kubernetes stacks don’t improve performance or pricing, compared to their cloud compute instances virtual machines and a dedicated virtual machine per would-be-container that thankfully doesn’t need to now share processing resources with others.
What gives? Why on earth would anyone run production processes in Kubernetes?
Merely making scriptable infrastructure doesn’t require kubernetes or containers… Ansible or it’s ilk will do just fine.
Deploy cloud instances like you currently deploy containers.
Save money, get better performance, have real networking.
*i have used kubernetes and understand how to use it as intended. I feel like I am missing the motivation for it’s current widespread usage in web apps
It deduplicates the kernel memory and system image base disk.
The minimum virtual machine size for a Windows server that is at all useful for anything is 4 GB of memory. Okay, okay, so you can technically boot it up on 2 GB and some roles will work fine, this will last only until some dingbat remotes to it with RDP with a 4K monitor and it starts swapping to disk.
Even if you use Server Core and block port 3389, it still needs a ton of memory just to start.
Running in a container it uses a few hundred megabytes.
Similarly, the minimum system disk size you can get away with is 32 GB if it is a discardable / ephemeral instance. You need 64 GB minimum if you ever intend to run Windows Update on it.
With containers, the unique parts of the image might be just a few hundred megabytes, even for complex apps.
My experience is with Windows, but from what I hear Linux VMs vs Linux containers have vaguely similar ratios.
So with containers, a single host can run dozens of applications, all sharing the same base disk, and all sharing the same OS kernel. The savings can be staggering.
At $dayjob, the admins are very much stuck in the dedicated VMs for every role mentality, and they're burning through enormous piles of taxpayer money to run them at literally 0.1% load.
Having said that, Kubernetes has its own problems. As you said, layering it on top of cloud VMs is a bit silly, and can easily result in the container running in a nested hypervisor at molasses speeds. Similarly, every single typical process changes dramatically: Deployment, updates, monitoring, auditing, etc...
Combine the above with the incompatible underlying cloud layer and things get really messy really quickly.
In my experience 90% of the world just isn't ready for the learning curve. Windows as an operating system isn't ready, certainly. Microsoft Azure isn't really ready either. Their AKS managed offering is still undergoing massive churn and seems to have more preview features than stable features. Even in the Linux world I hear more horror stories than success stories. It seems that everyone who says they love Kubernetes is using it on like... one machine. Come back and tell me how you feel after troubleshooting a failed upgrade on a cluster managing $100M of finance transactions.
What I would like to see is "native Kubernetes clouds" where the hosts are bare metal and there is no impedance mismatch between K8s and the cloud provider APIs because K8s is the API. Instead of the Azure portal or the AWS console you literally log into a Kubernetes console.
IMHO that would allow a true commoditisation of the public cloud and start to erode the near-duopoly of AWS and Azure.
I think exokernels and isokernels solve many of these issues where containers are currently used, check the Ocaml community for examples.
They run on hardware.
Ultimately, there needs to be a singular scheduling system running on hardware and a singular HAL-like driver layer, and exo or iso kernels deliver just that, vs lxe containers provided by os services.
The advantage of K8s isn't really in the virtualization technique used, but in the orchestration it can be made to perform for you. You can for sure configure K8s to use a host per container, if this is what you want.
Example of thing that is pretty straightforward in K8s and much less straightforward outside of it.
1. For compliance reasons, you need to make sure that your underlying OS is patched with security updates.
2. This means you need to reboot the OS every X time.
3. You want to do this without downtime.
4. You have a replicated architecture, so you know you have at least two copies of each service you run (and each can handle the required traffic).
In K8s, this operation can be as simple as:
1. Mark your old nodes as unschedulable.
2. Drain your old nodes (which will grab new nodes from your cloud provider, installing the updated machine image).
3. Delete your old nodes.
The exact steps will differ based on your use case, but that's basically it.
Steps you didn't need to think about here:
1. If I'm about to take a node down, do I have enough redundancy to handle its loss? K8s understands the health of your software, and you have a policy configured so it understands if taking down a container will cause an outage (and avoid this). Note: third party tech will be naturally compatible - if you use Elastic's cloud-on-k8s operator to run ES, it'll appropriately migrate from host to host too, without downtime. Likewise, the same script will run on AWS, Azure, GCP.
2. How fast can I run this? If building this logic yourself, you'll probably run the upgrade one node at a time so as to not have to think about the different services you run. But if it takes 15 minutes to run a full upgrade, you can now only upgrade 100 hosts each day. K8s will run whatever it can, as soon as it can without you having to think about it.
3. What happens if concurrent operations need to be run (e.g. scale-up, scale-down)? With K8s, this is a perfectly reasonable thing to do.
4. Does this need to be monitored? This is a fairly standard K8s workflow, with most components identical to standard scale-up/scale-down operations. Most components will be exercised all the time.
Generally I've been impressed by how straightforward it's been to remove the edge cases, to make complex tech fit well with other complex tech.
A while back we upgraded between two CentOS versions. In such a case it's recommended to reinstall the OS - there's not a clear upgrade path. In K8s, this would have been the same set of steps as the above. In many orgs, this would be a far more manual process.
I can understand developers and devops - overengineering allows them to play with technologies for company money. Tech managers should control level of overengineering, but in many cases they just miss..
Docker Swarm Rocks
" .... I would recommend for teams of less than 200 developers, or clusters of less than 1000 machines.
This includes small / medium size organizations (like when you are not Google or Amazon), startups, one-man projects, and "hobby" projects. " https://dockerswarm.rocks/
When I see k8s I automatically think o13g due witnessing it being tried a couple of times for the coolness factor, not because it was actually required.
I have read that this article isn't about k8s, but in my experience, companies that try to avoid k8s in favor of other specialized solutions are leaning more into premature optimisations. The company that used Nomad spent weeks in analisys of simple auth service, but every company that used k8s, went with whatever and moved on.
You can use elastic beanstalk, and RDS. You get scaling, reliability and backups. And it’s not complicated and you can set up easy CICD with just GitHub actions.
If you don’t need scaling, just use lightsail and litestream.
You just showcased my point! People would spend weeks in search for solutions that implement part of required functionality, instead of using whatever they're most familiar with and move on.
Guys Kubernetes is a container platform for multiple nodes. I know it seems hard to understand from the outside, but its really not. You would naturally come up with ALL the same componets if you were to take your container strategy onto multiple computers.
What if you dont need multiple servers? Well go the single node approach and have a flexible, true and tested way to spin up containers, which can and should be able to crash whenever they have to.
Using Containers is pre mature optimization too? maybe I should get my typewriter.
well I grew up provisioning and using dedicated servers and nowadays you get much tighter security from the container ecosystem that I would on a dedicated server.
I have just provisioned another bare metal k8s cluster, taking software of dedicated servers and.... well if I dont have to, im not running bare metal anymore in 2022.
If you want to just "get a computer in there" have a look at harvester, its the best of the two worlds
Doesn't look like the author knows what he is talking about. His point about early stage startup should not use K8S is fine. But the next advice about not using a different language for frontend and backend is wrong. I think the most appropriate advice is to choose a stack which the founding team is most familiar with. If that means RoR then RoR is fine. If it means PHP then PHP is fine too. Another option is to use a technology which is best suited for the product your are trying to build. For example, if you are building a managed cloud service, then building on top of K8S, or FireCracker or Nomad can be a good choice. But then it means you need to learn the tech being used inside out.
Also he talks about all of this and then gives example of WhatsApp at the end. WhatsApp chose Erlang for the backend and their front end was written in Java and Objective-C. They could have chosen Java for backend to keep frontend language same but they didn't. They used Erlang because they based their architecture on Ejabberd which was open source and was built with Erlang. Also WhatsApp managed all their servers by themselves and didn't even move to managed cloud services when they became available. They were self hosting till FB acquired them and moved them to FB data centres later on (Source: http://highscalability.com/blog/2014/2/26/the-whatsapp-archi...).
I also thought WhatsApp is a bad example. They not only hosted themselves, but they used solely FreeBSD (as far as I know) in their servers. (which don't get me wrong, I find great as a FreeBSD sysadmin myself).
Using WhatsApp as an example of a lean engineering org should almost be banned at this point. WhatsApp had a high performing engineering team that used basically the perfect set of tools to build their application (which also had a narrow feature scope; plaintext messaging). Even with hindsight there is very little you could do to improve on how they executed.
Just because WhatsApp scaled to almost half a billion users with a small engineering team doesn't mean that's the standard, or even achievable, for almost all teams.
I don't think their advice about not using it in a startup is correct either. You just need to somewhat know what you're doing.
I know of such a case, where a single engineer could leverage the helm chart open source community, and set up a scalable infrastructure, with prometheus, grafana, worker nodes that can scale independently of web service, a CI/CD pipeline that can spin up complete stacks with TLS automated through nginx and cert-manager, do full integration tests, etc.
I found that to be quite impressive, for one person, one year, and would probably be completely impossible if it wasn't for k8s.
I've seen places hire a dev that write all the OPS stuff and they scaled awesomely.. I mean if they had purchased 100servers full time on amazon, they would have spent a fraction of the cost to scale, but they could scale.
In 5 years I think they've never once had to reach even near the 100servers.
At the same time. I can scale heroku to 500 servers, and still be under the cost of one ops person. I can make that change and leave it there. I can do that all in under 30 seconds. Oh. And CICD is built in as a github hook. Even with blue-green deploys.
I think his point was most start-ups don't need to scale more than a site like heroku can offer. If you need more than 500 servers running full time then it's time to start looking to "scale"
The fact that they purchased 100 nodes has nothing to do with k8s but with their incompetence. You can run it on one machine. Also you can set up auto scaling easily based on whichever parameters.
> At the same time. I can scale heroku to 500 servers, and still be under the cost of one ops person. I can make that change and leave it there. I can do that all in under 30 seconds. Oh. And CICD is built in as a github hook. Even with blue-green deploys.
And then Heroku shuts down.
If you're building something that needs to scale up rapidly if it succeeds, k8s is worth thinking about. Either you don't succeed, in which case it doesn't matter what your stack was, or you do, in which case you'll be glad that you can scale up easily, you'll be glad you are using a common platform which is easy to hire competent people in, and, if you were smart about how you used k8s, you'll be glad that you can relatively easily move between clouds or move to bare metal.
I think the set of cases where "we need to scale up rapidly if it succeeds" and "Kubernetes solves all of our scaling needs and we aren't going to have problems with other components" is almost empty. On the other hand, there are quite a lot of startups that fail because they put too much focus on the infrastructure and Kubernetes and the future and too little on the actual product for the users. Which is the point of the article, I think. Ultimately what matters is whether you sell your product or not.
> I think the set of cases where "we need to scale up rapidly if it succeeds" and "Kubernetes solves all of our scaling needs and we aren't going to have problems with other components" is almost empty.
I agree, but so what? K8s isn't magic, it won't make all your problems go away, but if you have people who are genuinely skilled with it, it solves a lot of problems and generally makes scaling (especially if you need to move between clouds or move onto bare metal) much smoother. Of course you'll still have other problems to solve.
Given that most startups never need to scale up much, it's not surprising that k8s is mostly used where it's not needed. But people usually prefer not to plan for failure, so it's also not surprising that people keep using it.
Meanwhile, random_pop_non-tech_website exploding in traffic wasn't setup to scale despite years actively seeking said popularity through virtually any means and spending top dollar on hosting, and it slows down to crawl.
"Why no k8s?", you ask, only to be met with incredulity: "We don't have those skills", says the profiteering web agency. Sure, k8s is hard… Not. Nevermind that it's pretty much the only important part of your job as of 2022.
Obviously not, I was just pointing out that infra like k8s even under-the-hood for intermediaries (like web agencies) is still not always the norm given the real-world failures. There's this intermediary world between startups and giant corporations, you know. ;-)
>infra like k8s even under-the-hood for intermediaries (like web agencies) is still not always the norm
That's because 'the norm' for web agencies is a site that does basically zero traffic. If a company hires a 'web agency' that's by definition because the company's business model does not revolve around either a web property or app.
Whether that's a gas station company or a charity or whatever, the website is not key to their business success and won't be used by most customers apart from incidentally.
With that in mind most agencies know only how to implement a CMS and simple deployment perhaps using Cloudflare or a similar automated traffic handling system. They don't know anything about actual infrastructure that's capable of handling traffic, and why would they?
A lot of agencies are 100% nontechnical (i.e. purely designers) and use a MSP to configure their desktop environment and backups and a hosting agency to manage their deployed sites.
I very much agree with you. I must have been unnecessarily critical in my initial comment, I did not mean it as a rant, more like an observation about where-we're-at towards what seems an inevitable conclusion to me. Sorry that came out wrong, clearly I got carried away.
In asking if "Kubernetes is a red flag signalling premature optimisation", you correctly explain why we're yet on the "yes" side for the typical web agency category.
[Although FWIW I was hinting at a non-trivial category who should know better than not to setup a scale-ready infra for some potentially explosive clients; which is what we do in the entertainment industry for instance, by pooling resources (strong hint that k8s fits): we may not know which site will eventually be a massive hit, but we know x% of them will be, because we assess from the global demand side which is very predictable YoY. It's pretty much the same thing for all few-hits-but-big-hits industries (adjust for ad hoc cycles), and yes gov websites are typically part of those (you never know when a big head shares some domain that's going to get 1000x more hits over the next few days/weeks), it's unthinkable they're not designed to scale properly. Anyway, I'm ranting now ^^; ]
My unspoken contention was that eventually, we move to a world where k8s-like infra is the de facto norm for 99% of infrastructure out there, and on that road we move to the "no" side of the initial question for e.g. web agencies (meaning, we've moved one notch comparable to the move from old-school SysAdmin to DevOps maybe, you know those 10 years circa 2007-2018 or so).
[Sorry for a too-terse initial comment, I try not to be needlessly verbose on HN.]
>My unspoken contention was that eventually, we move to a world where k8s-like infra is the de facto norm for 99% of infrastructure out there, and on that road we move to the "no" side of the initial question for e.g. web agencies (meaning, we've moved one notch comparable to the move from old-school SysAdmin to DevOps maybe, you know those 10 years circa 2007-2018 or so).
This is very very hard to parse BTW. I don't want to reply to what you've written because I can't determine for sure what it is that you're saying.
Essentially I mean: scalable infra may be premature optimization today in a lot of cases, but eventually it becomes the norm for pretty much all systems.
You could similarly parse the early signs of a "devops" paradigm in the mid-2000's. I sure did see the inception of the paradigm we eventually reached by 2018 or so. Most of it would have been premature optimization back then, but ten-ish years later the landscape has changed such that a devops culture fits in many (most?) organizations. Devops being just one example of such historical shifts.
I anticipate the general k8s-like paradigm (generic abstractions on the dev side, a full 'DSL' so to speak, scalable on the ops side) will be a fit for many (most?) organizations by 2030 or so.
I mean, you still have to invest time on putting k8s to work, get people skilled with it, maintain and debug the problems... If Kubernetes didn't cost anything to deploy I'd agree that using it is the better idea, but it costs time and people, and those things might be better invested in features that matter to the users.
It depends. There are many things that carry a cost early but pay for themselves many times over later. Whether that will be the case for your startup depends whether you end up needing to scale quickly or not.
It's also worth considering that appropriate use of k8s can quite likely save you time and money early on as well. It standardises things, making it very easy for new ops people to onboard, and you might otherwise end up spending time reinventing half-baked solutions to orchestration problems anyway.
> It depends. There are many things that carry a cost early but pay for themselves many times over later. Whether that will be the case for your startup depends whether you end up needing to scale quickly or not.
Well, precisely what I said is that 99.9% of startups won't find themselves in a situation where they need to scale quickly and the only scale problems they find can be solved with Kubernetes.
> It's also worth considering that appropriate use of k8s can quite likely save you time and money early on as well. It standardises things, making it very easy for new ops people to onboard, and you might otherwise end up spending time reinventing half-baked solutions to orchestration problems anyway.
The point is that you might not even need orchestration from the start. Instead of thinking how to solve an imagined scenario where you don't even know the constraints, go simple and iterate from that when you need it with the actual requirements in hand. And also, "make it easier for new ops people to onboard" doesn't matter if you don't have a viable product to support new hires.
You seem to be describing very early stage companies, and if so I agree, host it on your laptop if you need to, it makes zero difference. But it's not binary with Netflix on one side and early stage on the other.
There are a lot of companies in the middle, and following dogma like "you don't need k8s" leads them to reinvent the wheel, usually badly, and consequently waste enormous amounts of time and money as they grow.
Knowing when is the right time to think about architecture is a skill; dogmatic "never do it" or "always do it" helps nobody.
What about CD of similar but not identical collections of services to metal? No scaling problem, other than the number of bare metal systems is growing, and potentially the variety of service collections. For instance, would you recommend k8s to tesla dor the CD of software to their cars?
> if you were smart about how you used k8s, you'll be glad that you can relatively easily move between clouds or move to bare metal.
I'd argue you should definitely consider multi-cloud strategy from the get-go indeed in 2022. Something like Terraform helps statically setting k8s clusters on most clouds. Especially for startups, it's better to default to vanilla stuff and only complicate on a need-to basis.
Yes, completely agreed. Multi-cloud is really not that difficult nowadays, and it puts you in a better negotiating position (when you end up spending enough to be able to negotiate), as well as giving you more location flexibility and the ability to pick and choose the best services from each cloud.
Oh yes, negotiation is a strong argument in that context. One that makes or breaks a CTO's mission, me thinks, if that company expects a lean path to ROI.
A multi-cloud paradigm is also a great way to teach you about your application and about those clouds themselves. A good reminder that "implementation is where it's at", and "the devil is in the details".
Success is rarely so rapid that you can't throw money at a problem temporarily and build something more robust.
No one is advocating for a single server running in your closet, but a large and well funded PaaS can handle any realistic amount of growth at least temporarily, and something like Heroku is big enough (and more importantly, owned by a company big enough) that shutting down without notice is not a possibility worth considering.
> Either you don't succeed, in which case it doesn't matter what your stack was, or you do, in which case you'll be glad that you can scale up easily
This take brushes right past the causes of success and failure. Early stage success depends on relentless focus on the right things. There will be 1000 things you could do for every 1 that you should do. Early on this is going to tend to be product-market fit stuff. If things are going very well then scalability could become a concern, but it would be a huge red flag for me as an investor if an early stage company was focusing on multi-cloud.
I certainly wouldn't recommend that anyone "focus on multi-cloud" in an early-stage company (unless of course multi-cloud is a crucial part of their product in some way).
Kubernetes is basically an industry standard at this point. It's easy to hire ops people competent in it, and if you do hire competent people, it will save you time and money even while you are small. As an investor "we use this product for orchestration rather than trying to roll our own solutions to the same problems, so that we can focus on $PRODUCT rather than reinventing half-baked solutions to mundane ops problems" should be music to your ears.
I agree with all of that. That said, I don't think competence is a binary proposition, and if you hire people who have only worked at scale they will be calibrated very differently to the question of what is table stakes. One of the critical components of competence for early stage tech leadership is a keen sense of overhead and what is good enough to ratchet up to the next milestone.
As many problems as containerization solves, it's not without significant overhead. Personally I'm not convinced the value is there unless you have multiple services which might not be the case for a long time. You can get huge mileage out of RDS + ELB/EC2 using a thinner tooling stack like Terraform + Ansible.
The overhead of containerisation is mostly in the learning curve for teams that are not already familiar with it (and the consequent risk of a poor implementation). A well designed build pipeline and deployment is at least as efficient to work with as your Terraform+Ansible.
If you have such a team, it can of course make sense to delay or avoid containerisation if you don't see obvious major technical benefits.
But those teams will get rarer as time goes on, and since we're talking about startups, honestly it would be questionable to build a new ops team from people with no containers knowledge in 2022.
Almost every k8s project I've looked at in the last few years is database bound. k8s is not really going to solve their scaling needs. They needed to plan more up front about what their application needed to look like in order to avoid that.
Yes, if your application looks like a web application that is cache friendly, k8s can really take you a long way.
In case it's not clear, nothing in my comment suggests that k8s will magically solve all your problems. It just provides abstractions that make growth (in size and complexity) of systems easier to manage, and helps to avoid lock-in to a single cloud vendor. The wider point is that thinking about architecture early will make scaling easier, and for most companies, k8s is likely to end up being a part of that.
The "web application" / cache-friendly part of your comment doesn't make much sense to me; k8s is pretty well agnostic to those kinds of details. You can orchestrate a database-bound system just as well as you can anything else, of course.
How long did it take him to do this setup, a year you say, and that is impressive? I am not trying to be cute here, my question comes from a genuine place of curiosity. I've love to learn to spin-up a system like that, but from the tech/sales talks I see I am made to believe this can be done in a day. Expectation management is important, if people say ops is just a solved problem then I expect this to take very little time and to be easy to learn. Maybe I am learning the wrong thing here, and should do learn Helm or something more high level.
It took a year, but that was somewhat on the side of also building an OpenAPI based Web service, the gRPC based workers. So, it wasn't just the infrastructure stuff. If I were to estimate how much time for just the infrastructure and devops tooling, then two months. It's been up and running with less than 15 minute downtime over the course of two years.
I do consider this impressive. And, to be clear, I wouldn't say this is because of a "super-developer". In fact, he had no prior k8s experience. But rather that there are thousand upon thousand of infrastructure hours devoted to the helm charts, often maintained by the people who develop the services themselves. It is almost mind boggling how much you get for almost free. Usually with very good and sensible default configurations.
In my precious work place, we had a team of 5 good engineers purely devoted to infrastructure, and I honestly believe that all five would be able to spend their time doing much more valuable things, if k8s had existed.
As for whether or not such devops solutions could be done in a day. Hm. I don't know. These things should be tailored to the problem. If you've done all of this a few times, then maybe you can adjust a bunch of charts that you are already familiar with and do what took a couple of months and impressed me, in a couple of weeks. It's a lot more than just "helm install. Done", that goes into architecting a scalable solution. Implementing monitoring, alerting and logging. Load testing stuff. Etc.
That's seems like a very negative take in my opinion. This 'simpler operational tech' would still need to be able to scale, correct? If you think that there is a good and easier way to deploying 10-15 services, all of which can scale, and all of it defined in rather neat code, to be anything but "simple operational tech", then I believe you are confusing "solving a complex problem", with "simplifying the requirements of a complex problem". The latter of which has been stripped of many important features. K8S isn't anything magic, but it certainly isn't a bad tool to use. At least not in my experience, though I've heard of horror stories.
That does remind me that when that employee started, the existing "simple operational tech" was in fact to SSH into a VM and kill the process, git pull the latest changes, and start the service.
The only way you can solve the actual problem (not a simplified one) would in my opinion either be k8s or terraform of some kind. The latter would mostly define the resources in the cloud provider system, most of which would map to k8s resources anyways. So, I honestly just consider k8s to better solve what terraform was made for.
I'm sure the "simpler operational tech" meets few requirements for short disaster recovery. Unless you have infrastructure as code, I don't think that is possible.
Any startup that knows what their product is and are done with PoCs, should be able to deal with the consequence of succeeding, without failing. Scaling is one of those things that should be in place before you need it. In our case, scaling was a main concern.
and ... you might be justified in that concern. However... after having been in the web space for 25+ years, it's surprising to me how many people have this as a primary concern ("we gotta scale!") while simultaneously never coming close to having this concern be justified.
I'm not saying it should be an either/or situation, but... I've lost count of how many "can it scale?" discussions I've had where "is it tested?" and "does it work?" almost never cross anyone's lips. One might say "it's assumed it's tested" or "that's a baseline requirement" but there's rarely verification of the tests, nor any effort put in to maintaining the tests as the system evolves.
EDIT: so... when I hear/read "scaling is a main concern" my spidey-sense tingles a bit. It may not be wrong, but it's often not the right questions to be focused on during many of the conversations I have.
> I'm not saying it should be an either/or situation, but... I've lost count of how many "can it scale?" discussions I've had where "is it tested?" and "does it work?" almost never cross anyone's lips.
Also, discussions about rewrites to scale up service capacity, but nobody has actually load tested the current solution to know what it can do.
Just keep it simple, and if you take off scale vertically while you then work on a scalable solution. Since most businesses fail, premature optimisation just means you're wasting time that could have gone on adding more features or performing more tests.
It's a trap many of us fall into - I've done it myself. But next time I'll chuck money at the problem, using whatever services I can buy to get to market as fast as possible to test the idea. Only when it's proven will I go back and rebuild a better product. I'll either run a monolith or 1-2 services on VPSs, or something like Google cloud run or the AWS equivalent.
for example lambda (not microservices, running mini monoliths per lambda function)
yes by simple I mean covering high availability requirements, continuous deployment, good DORA measures - not simple as in half-baked non-functional operations (such as manually sshing to a server to deploy)
Ah, I see. Well, lambdas are also a nice tool to have, but it certainly do not fit for all applications (same as with k8s). I'd also point out that lambdas replace a rather small capabilities of k8s, and the type of systems you can put together. You would end up needing to set up the rest either through a terrible AWS UI or terraform. Neither of which I find to simplify things all that much, but perhaps this is a matter of taste.
In our case, the workers were both quite heavy in size (around 1 GB), and heavy in number crunching. For this reason alone (and there are plenty more), lambdas would be a poor fit. If you start hacking them to keep them alive because of long cold starts, you would lose me at the simple part.
Yeah to be honest, I run a k8s cluster now for my saas. But about 4 times more expensive then my previous company I ran on a VPS.
And scaling is the same that VPS I could just scale the same way. Run a resize in my hosting company panel. (I dont use autorescal atm)
Only if I would hit about 100x times the nrs I would get the advantage of k8s, but even then I could just split up customers into different VPS.
CI / CD can be done good and bad with both.
And in practice K8S's a lot less stable. Maybe because I'm less experienced with K8S. But also because I think its more complex.
To be honest k8s is one of those dev tools that has to reinvent every concept again, so it has it's own jargon. And then there are these ever changing tools on top of it. It reminds me of JS a few years ago.
>That's seems like a very negative take in my opinion. This 'simpler operational tech' would still need to be able to scale, correct?
Premature optimization is a top problem in startup engineering. You have no idea what your startup will scale to.
If you have 1,000 users today and 5 year goal of 2,000,000 users, then spending a year building infrastructure that can scale to 100,000,000 is an atrociously terrible idea. A good principal can setup a working git hook, circleci integration, etc capable of automated integration testing and rather close to ci/cd in about a weekend. Like you can go from an empty repo to serving a web app as a startup in a matter of days. A whole year is just wasteful insanity for a startup.
The reality for start-ups running on investor money with very specific plans full of OKRs and sales targets is very different: you need to be building product as fast as possible and not giving any fuck about scale. Your business may pivot 5 times before you get to a million users. Your product may be completely different and green-fielded two times before you hit a million users.
I can't imagine any investor being ok with wasting a quarter of a million+ and a year+ on a principal engineer derping around with k8s while the product stagnated and sales had nothing to drive business -- about as useful as burning money in a pit.
You hire that person in the scale-up phase during like the third greenfield to take you from the poorly-performing 2,000,000 user 'grew-out-of-it' stack to that 100,000,000+ stack, and at that point, you are probably hiring a talented devops team and they do it MUCH faster than a year
If you have a website with 1000 users today and product is going to be re-designed 5 times, it's probably best just to use sqlite and host on a single smallish machine. Not all problems are like this however.
Sometimes it's faster to implement certain features in another languages and deploy it as microservice instead of fighting your primary language/framework to do it. Deploying microservices in k8s is as easy as writing a single yaml file.
I am not privy to the details of the case, but a rule-of-thumb I heard once is that if it's far enough from your core, a SaaS can be used (obviating the whole question), and if it's part of the core, start by developing it as a separate functionality before moving it to another service.
In a lot of cases it's pattern abuse. I'm dealing with this all the time. People like to split things that can work perfectly as one whole, just for the sake of splitting it.
I should perhaps have clarified, but the 10-15 are not self maintained services. You need nginx for routing and ingress, set up cert-manager and other ingress endpoints are automatically configured to have TLS, deploy prometheus, which comes with node-exporter and alert-manager, deploy grafana.
So far, we're up at 6 services, yet still at almost zero developer overhead cost. Then add the SaaS stack for each environment (api, worker, redis) and you're up at 15.
Use a managed k8s cluster (eks, aks or gke). Creating a production ready k8s on vms or baremetal can be time consuming. Yes, you can do lamdba, serverless, etc. but k8s gives you the same thing and is generally cheaper.
You can get a majority of those things "running" in few days. If you don't want it to fall over every other day, then you need to have a ton of ancillaries which will take at least several months to set up, not to mention taking care of securing it.
With 1YoE I did most of that in about 3 months. Had a deadline of 6 months to get something functional to demonstrate the proposed new direction of the company, and I did just that. If I were to do it today I could probably rush it to a week, but that would mean no progress on the backend development that I was doing in parallel. A day is probably doable with more on-rails/ batteries included approaches.
Not because I'm amazing, but there's a frankly ridiculous amount of information out there, and good chunks of it are high quality too. I think I started the job early January, and by April I had CI/CD, K8s for backend/frontend/DBs, Nginx (server and k8s cluster), auto-renewing certs, Sentry monitoring, Slack alerts for ops issues, K8s node rollback on failures, etc.
The best way to learn, is to do. Cliche, but that's what it really comes down to. There's a fair few new concepts to grasp, and you probably have picked some of these up almost by osmosis. It sounds more overwhelming than it is, truly.
The problem is never spinning things up, it's in maintenance and ops. K8s brings tons of complexity. I wouldn't use it without thinking very carefully for anything other than a very complex startup while you're finding product-market fit.
It's actually pretty easy to do these days, even on bare metal servers. My go to setup for a small bare metal k8s cluster:
- initial nodes setup: networking configuration (private and public network), sshd setup (disallow password login), setting up docker, prepping an NFS share accessible on every nodes via private network
- install RKE and deploy the cluster, deploy nginx ingress controller
- (optional) install rancher to get the rest of the goodies (graphana, istio, etc). These ate a lot of resources though, so I usually don't do this for small clusters
Having very recently done this (almost, another dev had half time on it) solo, It's not _too_ terrible if you go with a hosted offering. Took about a month/month and a half to really get set up and has been running without much of a blip for about 5 months now. Didn't include things like dynamic/elastic scaling, but did include CD, persistent volumes, and a whole slew of terraform to get the rest of AWS set up (VPCs, RDS, etc). I'd say that it was fairly easy because I tinkered with things in my spare time, so I had a good base to work off of when reading docs and setting things up, so YMMV. My super hot take, if you go hosted and you ignore a ton of the marketing speak on OSS geared towards k8s, you'll probably be a-ok. K8s IME is as complex as you make it. If you layer things in gradually but be very conservative with what you pull in, it'll be fairly straightforward.
My otherhot take is to not use helm but rather something like jsonnet or even cue to generate your yaml. My preference is jsonnet because you can very easily make a nice OO interface for the yaml schemas with it. Helm's approach to templating makes for a bit of a mess to try and read, and the values.yml files _really_ leak the details.
That isn't the point. If he had a whole year, was there a tangibly better use of his time to get a product to market faster? What might the business implications be for doing or not doing so?
It seems many are focused on the time estimate. That was in creating the overall solution. About two months was to set up the infrastructure mentioned.
These often get developed side by side. GitLab, unit tests, api-server, nginx, cert-manager, deployments, integration tests, prometheus, metrics in services, grafana, alert-manager, log consolidation, work services and scaling, etc.
Just spinning up a cluster, nodepool, nginx, cert-manager w/let's encrypt cluster issuer, prometheus, grafana, can easily be done in a day. So, time estimates kinda depend entirely on what you mean by it.
Spinning up promerheus and grafana with automatic service discovery: one day.
Making good metrics, visualizations, and alerts: everything from a week to a month or two. So, take the time estimates with a grain salt.
I agree with that, setting up k8s on bare-metal took me 2 days, and we needed it to deploy elastic and some other helm charts as quickly as possible without loosing our minds maintaining nodes with some clunky shell scripts.
Also we bought us immediately an easy approach to build gitlab ci/cd pipelines + different environments (dev, staging, production) on the same cluster. Took me a week to set everything up completely and saved our team developing rapidly features really a lot of time and headache since then. But the point is, I knew how to do it, focus on the essentials and deliver quick reasonable results with large leverage down the route.
> deploy elastic and some other helm charts as quickly as possible
Bad culture alert! No one needs Elastic "as quickly as possible" unless their business, or the business they work for, is being very poorly run.
I would also argue that you might have got it running quickly, but how are you patching it? Maintaining it? Securing it? Backing it up? Have you got a full D/R plan in place? Can you bring it back to life if I delete everything within 3-6 hours? Doubt it.
> maintaining nodes with some clunky shell scripts.
Puppet Bolt, Ansible, Chef, ...
There are so many tools that are easy to understand that solve this issue.
I believe elastic is available as service from AWS and elastic.co, if you need it fast, use that. If you need it long term it may be worthwhile to deploy your own for cost and flexibility purposes
That’s all solved for you, helm upgrade in ci/cd and bump of versions has been straight forward, if not snapshot rollback via Longhorn, also for DR. Accidentally deleted data => get the last snapshot, 5 minutes it’s back (except that there is of course CI/CD in place for new code + no write permissions for devs on the cluster and „sudden“ data deletion somewhat rare).
Elastic usecase is for crawling crazy amount of data and make it searchable and aggregatable and historically available, don’t know any other solution than elastic who has reasonable response times and easy-to-use access (plus we can add some application logging and APM).
> Puppet Bolt, Ansible, Chef, ...
Helm chart values.yaml and you’re all set, security + easy version bump included.
> I know of such a case, where a single engineer could leverage the helm chart open source community, and set up a scalable infrastructure, with prometheus, grafana, worker nodes that can scale independently of web service, a CI/CD pipeline that can spin up complete stacks with TLS automated through nginx and cert-manager, do full integration tests, etc. I found that to be quite impressive, for one person, one year, and would probably be completely impossible if it wasn't for k8s.
But that's the thing though: they didn't do it alone. You literally pointed out that this wasn't true almost immediately: "leverage the helm chart open source community". They used the work of others to get to the result.
Also, I highly doubt they could debug something if it went wrong.
I simply cannot believe anyone would advocate, or believe, that because a Helm chart makes it simple to create a complex piece of infrastructure it must also be true that maintaining it is also simple. It's really.
Once you understand the concepts it's not hard to debug. It's fair to acknowledge that kubernetes is complex, but also we should not ignore the real work that has been done in the past few years to make this stuff more accessible to the developers that want to learn it.
Also, saying it's not "alone" in this example I think is not fair. What would you count as "alone"? Setting up the kubernetes cluster from scratch and writing your own helm charts? Using that same logic, I can't say that because someone else designed and built the hardware it's running on. I think it's fair to say that if someone, independent of coaching, regardless of the underlying infrastructure produced some production grade infrastructure by themselves, they certainly did it alone.
Using the helm charts from the community is arguably still doing it alone. There isn't anything back and forth. It's just the right tool for the right job. But, this starts being about language and semantics. Like saying that following a best practices on how to configure nginx isn't doing it alone, because someone else wrote that. Helm charts just very often expose what needs to be configured, and otherwise follows those best practices.
As for debugging. You do have a point that it becomes more difficult. But, this also holds true for any of the alternatives discussed her (lambdas, terraform). I'd argue that when it all comes down to it, that because you can spin up the entire infrastructure locally on things like minikube, that it makes it many times more easy to debug than other cloud-provider-only solutions.
> with prometheus, grafana, worker nodes that can scale independently of web service, a CI/CD pipeline that can spin up complete stacks with TLS automated through nginx and cert-manager, do full integration tests, etc.
> I found that to be quite impressive, for one person, one year, and would probably be completely impossible if it wasn't for k8s.
I've always found this interesting about web based development. I have no idea what Prometheus, grana, etc do. Ive never used k8s.
And yet, as a solo dev, I've written auto-scaling architecture using, for example, the AWS ec2 apis that let you launch configure and shutdown instances. I don't know what else you need.
Really the only advantage I see to morass of services is you get a common language so other devs can have a slightly easier time of picking up where someone left off. As long as they know all the latest bs.
> And yet, as a solo dev, I've written auto-scaling architecture using, for example, the AWS ec2 apis that let you launch configure and shutdown instances. I don't know what else you need.
This is fine, if you’re on AWS and can use AWS APIs. If you’re not (especially if you’re on bare metal), something like K8s can be nice.
In short. Prometheus is a worker that knows about all your other services. Each service of interest can expose an endpoint that prometheus scrapes periodically. So the services just say what the current state is, and prometheus, since it keeps asking, knows and stores what happens over time. Grafana is a web service that uses prometheus as a data source and can visualize it very nicely.
Prometheus also comes with an Alert Manager, where you can set up rules for when to trigger an alert, that can end up as an email or slack integration.
They are all very useful, and gives a much needed insight into how things are going.
I dont know AWS ec2 apis and for sure I'm not capable of writing auto-scaling architecture. This is the reason why I default to K8s. I have used it easily and successfully by myself for the last 4 years. It just keeps on running and hasn't given me any problems.
The thing is, unless using those technologies was somehow core to what the single engineer was trying to, it might be technically impressive but might not have actually provided value for users.
Users don't really care if you have a really impressive stack with cool technologies if it doesn't offer anything more than a couple of web servers and a DB server.
His point is to use whatever simplifies workflow / reduces operational overhead. To some people, that indeed would be k8s. To you, that may be "managing a couple of webservers and a db". And that is great for you.
yeah I use k8s for basic webapps and it works wonderfully, and is way way easier than anything else, and yes I started developing in the 90s so I've seen it all. There is a bit of overhead in learning k8s, but once you know it, it's dead simple for every use case I've found and takes you way further than anything else.
Rocket science isn't hard if you know it. Should we all build spaceships to deliver groceries? Good luck finding a few local rocket scientists in a pinch.
You can find plenty of auto mechanics though. Cars are cheaper and ubiquitous. Maybe they can't drive to the moon, but they can get most things done.
Unless your business is flying to the moon, stick to cars and trucks over spaceships.
I largely agree with the previous poster. If a single brief yaml file is too complicated, you're asking for too much from your tooling. I've seen hideous configuration management systems that did 1/100th of what kubernetes can do. There is some additional complexity around containers which is not a k8s complexity issue, and there are advanced features that most devs will never need to touch. Regardless, I'll take the complexity saved by k8s over custom built tooling or oversimplified PaaS any day.
Right on. Previous devs at co I joined wanted to play DevOps cowboys. They used Ansible scripts to spin up various AWS services costing the company over 100K/yr.
New lead came in, got rid of that crap by using 3rd party services to spin up infrastructure. Got a load balancer, a few VMs + DB. Reduced the cost down by 85% and greatly simplified the entire stack.
I learned a really valuable lesson without having to make that mistake myself.
I understand why people get excited about tooling. It's cool to learn new things and automate stuff away. I'm prone to that myself and do this on my own server when I get that itch.
Having said that, it's wrong to foist this stuff onto an unsuspecting company where the owners don't know any better about tech, that's why they hire other people to do that for them and seeing that just left a bad taste in my mouth for overcomplicated setups.
I get that SV is different, that's why tools like K8 are made and I would jump on those tools on a heartbeat as needed.
But for other smaller businesses, the truth is they just need a boring monolothic load balanced app with a few VMs and a DB sprinkled with 3rd party services for logging or searching or other stuff not core to the business.
I know this utterly misses the larger point of your comment, but:
> They used Ansible scripts to spin up various AWS services
This seems less about using the "cool/new" tech... rather it's about using the "right" tech. Config management tools like Ansible/Chef/Puppet are very much previous-generation when it comes to cloud infrastructure.
They... can manage cloud infrastructure, but they were created prior to the ubiquity of cloud deployments, and the features are glued on. Not choosing a more modern IaC framework tells me they(those devs) were going to be making sub-optimal implementation decisions regardless.
Yeah, this project was several years old. Take this with a grain of salt, I'm not familiar with timelines in terms of k8s, but I would guess that it had not yet risen to popularity as it has in more recent years.
And yet, to me it sounds like NIH since it's a pretty standard stack; couldn't they just get something like google app engine and get all of that from day one? Because did any of those things mentioned result in a more successful company?
I'd argue that using helm charts is the exact opposite of NIH. The things that take time are not the stack themselves, but the software and solutions. K8s just makes the stack defined in code, and written and managed by dedicated people (helm maintainers) as opposed to a bit "all over the place" and otherwise in-house, directly using cloud provider lock-in resources.
I'm sure there are plenty of use cases where that makes sense, and is a better approach. But, I disagree that k8s suggests a NIH-mindset.
Most startups get basic security for networking and compute wrong, K8s just adds even more things to mess up. Odds are even if you use an out of the box solution, unless you have prior experience you will get it wrong.
I will always recommend using whatever container / function as a service e.g. ECS, GCF, Lambda any day over K8s for a startup. With these services its back to more similar models of security such as networking rules, dependency scanning, authorization and access...
So question then - is it possible to found a tech startup without paying rent to a FAANG? Before I get the answer that anything is possible, I should say is it feasible or advisable to start a company without paying rent to the big guys?
There is nothing stopping you from hosting your own OpenStack, managed k8s, and all that, on your own hardware. You would need a good reason to not let someone else deal with all of this though.
For a small enough company you could even just use use k3s + offsite backups. Once you grow large enough you can setup machines in 2-4 locations across the land mass where your users exist. If you have enough than a hardware fault in one isn't an emergency and you'd be able to fly out to fix things if needed.
Realistically, on all flash, you are very unlikely to need to maintain anything on a server for a few years after deployment.
That is probably a good idea for many startups. However, once you get into the world of audits and compliance certifications, things become a lot harder. But, but then again, at this point, I suppose it is easy enough to transition to some managed hardware.
If and only if your business model depends on it. A startup's job is mostly to find product market fit; if being decoupled from AWS isn't part of your market, you are spending money on a non-problem.
The reality is unless you’re some rich dude who can borrow dad’s datacenter (And that’s cool if so), you’re either going to be renting colo space, virtual servers, etc.
It’s always a challenge in business to avoid the trap of spending dollars to save pennies.
IMO, you’re better off working in AWS/GCP/Azure and engineering around the strengths of those platforms. That’s all about team and engineering discipline. I’m not in the startup world, but I’ve seen people lift and shift on-prem architecture and business process to cloud and set money on fire. Likewise, I’ve seen systems that reduced 5 year TCO by 80% by building to the platform strengths.
I'm aware that no man is an island in some sense, but I'm not comfortable with locking myself into one of 3 companies who need to increase their revenue by double digits year over year. And as you say, a lift and shift is basically setting money on fire. Currently I run sort of a hybrid approach with a small IaaS provider and a colo. It seems to work well for us both technically and financially though that seems to go contrary to what is considered conventional wisdom these days.
That’s awesome. The most important thing is to understand why you’re making the decisions that you do.
Where I work, we can deliver most services cheaper on-prem due to our relative scale and cloud margins. But… we’re finding that vendors in the hardware space struggle to meet their SLAs. (Look at HPE — they literally sold off their field services teams and only have 1-2 engineers covering huge geographic regions. So increasingly critical workloads make the most sense in the cloud.
If you're priorities are 'which companies do my values align with among generally very high integrity companies to begin with' - then you might want to reconsider.
Google is not evil. They're just big, and define some practices which we might think should change in the future.
Once you have the thing up and running, you can start to think about hosting your own.
Also, you don't need to use fancy services because most startups can run just fine on a single instance of whatever, meaning, there are a lot of cloud providers out there.
As a rule, for anything in an startup, adding that "scalable" adjective is a waste.
Of course, exceptions exist, but it's one of those problems that if you have them, you probably know it, and if you have them and don't know it, you won't succeed anyway. So any useful advice has the form of "don't".
In a typical startup you are going to be strapped for cash and time. You need to optimize and compromise. In general, if you don't know how do something, then figure out if you need to learn it, or whether you can get by with a "good enough" solution, because there will be a queue of other things you need to do that might be more business-critical.
So if you already know Kubernetes, then great, use it. Leverage your existing skills. If you don't, then just use Heroku or fly.io or whatever, or go with AWS if that's your competence. Maybe revisit in a year or two and maybe then you'll have funding to hire a devops person or time to spend a week or two learning to do it yourself. Right now you want to get your SAAS MVP in front of customers and focus on the inevitable product churn of building something people want to pay for. The same advice goes for anything else in your stack. Do you know it well enough? Do you need it right now? Or is there a "good enough" alternative you can use instead?
> I think the most appropriate advice is to choose a stack which the founding team is most familiar with. If that means RoR then RoR is fine. If it means PHP then PHP is fine too.
Taking human resource information into consideration sounds very wise. Although, learning a new language is generally not that a huge barrier, while changing your whole stack once the minimum viable product cap is passed can be very expensive. And if you need to scale the team, the available developer pool is not the same depending on which technology you have to stick with.
It doesn’t invalidate your point, but maybe it brings some relevant nuances.
The Whole advice of using same language is especially silly - iOS is stuck with Swift, and the web is stuck with JS, and maybe you need an applitation that scales using actors across mutiple machines with Golang or Java, or maybe you need to plug into Windows tightly and need C#.
Kubernetes is not 'harder' if all you need is to host a webapp. Where it falls on the hardness spectrum depends on what you are trying to do, and what is the alternative. I am very fluent with Kubernetes but have no skills in managing traditional virtual machines.
> The Whole advice of using same language is especially silly - iOS is stuck with Swift, and the web is stuck with JS, and maybe you need an applitation that scales using actors across mutiple machines with Golang or Java, or maybe you need to plug into Windows tightly and need C#
And you're also forgetting Android and macOS and Linux.
That's why cross-platform frameworks like Electron and React Native are so popular. The time wasted in going native for every single platform is just infeasible for most non-huge companies.
But you could also have 2 people working on React Native and have 1 person each for getting it to play nice with iOS/Android, and eliminate the need for an extra engineer.
tss.. some people dont like being reminded that their favourite tech performs worse on an Nvidia 3090 than Winforms did on 800 mhz cpu running windows 98
Well, if React native is anything like the many react websites, then this isn't too far off actually. "modern" websites can already send your CPU puffing, when you hover over some element with your mouse pointer and it triggers some JS emulated style for :hover.
Agree the author is wrong on that specific point, though thankfully the bulk of the article content deals with the headline, and is mostly fine wrt k8s.
Rather than the author "not knowing" what they're talking about, I suspect they're taking narrow experience and generalising it to the entire industry. Their background is selling k8s as a solution to small/medium enterprises: it strikes me that there may be a strong correlation between startups interested in that offering and those deploying failed overengineered multilang micro-architectures. Suspect the author has seen their fair share of bad multilang stacks and not a lot of counter examples.
Choosing more than one language as a startup can become really expensive quickly. As long as your tribes are small, chances are high that you one day run out of. e.g., python developers while you still have a lot of Java guys (or vice versa). This introduces unnecessary pain. (And obviously, you should have used Rust or Haskell from the get go for everything.)
The sole exception to this rule I would make is javascript which is more or less required for frontend stuff and should be avoided like the plague for any other development. As soon as you can get your frontend done in Rust, though, you should also switch.
Idk, I am someone, who has looked at many programming language, including all of those you mentioned. But a capable developer can be expected to learn a new language over the course of a few weeks if needed. I don't see how you could "run out of devs of language x", if you have capable devs on board. Especially, when those languages are all in the same programming language family/club.
Even the most capable developer that learns a new language in a few weeks will not be an expert in it. The difference in productivity and quality of the code will be huge. This is because in different languages things can be done very differently, it is not about the syntax as much as the best ways to do things.
sounds like it just boils down to: try to choose the technology your team is familiar with, not what other teams are successful with
Of course there's some balance needed. If your team is familiar with some niche language then long term that might not be a good strategy if you intend to bring more devs on board later.
One side of this which I don't think is discussed often is the fun of choosing new technology. How do you balance having fun and being realistic at the same time?
Fun meaning trying new technology, learning as you go, setting up systems that make you feel proud, etc. It can lead to failure, but I think having fun is important too.
"I think the most appropriate advice is to choose a stack which the founding team is most familiar with."
I'd think that's exactly what typically happens most of the time. But the degree of stack-lockin that occurs with startups still surprises me even when it's clear a better choice might have been made. Mostly due to management not being prepared to grant the necessary rewrite time.
> But the next advice about not using a different language for frontend and backend is wrong.
Being charitable, what I think they are getting at is maybe more about having fully separated frontend and backend applications (since the front-end examples he gives are not languages but frameworks / libraries). Otherwise it seems really backwards - I'm definitely an advocate of not always needing SPA-type libraries, but using literally zero Javascript unless your backend is also JS seems like it goes to a too-far extreme.
Re: single language, there's a grain of truth to it - see http://boringtechnology.club/ - but that one mainly says there is a cost to adding more and more languages. When it comes to back- and frontend though, I would argue there is a cost to forcing the use of a single language. e.g. NodeJS is suboptimal, and web based mobile apps are always kinda bleh.
I suspect a lot of the gripes and grousing about Kubernetes comes from SMEs trying to run it themselves. That will often result in pain and cost.
Kubernetes is a perfectly good platform for any size operation, but until you are a large org, just use a managed service from Google/Amazon/DigitalOcean/whoever. Kubernetes, the data plane, is really no more complex that eg Docker Compose, and with managed services, the control plane won't bother you.
K8s allows composability of apps/services/authentication/monitoring/logging/etc in a standardised way, much more so than any roll-your-own or 3rd-party alternative IMO, the OSS ecosystem around it is large and getting larger, and the "StackOverflowability" is strong too (ie you can look up the answers to most questions easily).
So, TLDR, just use a managed K8s until you properly need your own cluster.
Exactly! I chose k8s managed by Google years ago. As a solo developer you can have a cluster up and running with a web server, fronted by a Google LB with Google managed certificates in under an hour. Just follow one of the hundred tutorials. Just as quick and easy as setting up a single VM. But that really isn't the point is it. If that is all I needed, yes I'd use a $10 VPS. But for an "application", not a web site, you always need more.
My k8s "cluster" is a single node that doesn't cost me any more than a VM would. I don't need the scalability at the moment. But I do need different environments for dev, qa, and prod. And all three are running identically next to each other using Namespaces. Saved us a ton of maintenance and cost.
Any project that grows has its needs change. GKE gives you a ton of integrated tools right from the start including logging, alerting, metrics, easy access to hosted databases, pub/sub, object storage, easy and automatic network setup, easy firewall setup, dns management, and a lot more. k8s is no different than using any other hosted service. It provides a great set of features that you configure using fairly consistent yaml configuration files. And it is all accessible from the web based "Google Console" as well.
Learning the k8s yaml format and some basic kubectl commands is all you need to get going and it saves a TON of time that can go back into developing your application rather than dealing with configuring disparate pieces with their own configuration methods.
I was fairly early to k8s while they were still competing with other similar solutions and other tools like Puppet and Chef. I tested all of them and truthfully, k8s was the easiest to learn, implement, and maintain my app with. Using GKE of course. I would NEVER as a one man or even small team of developers take on managing an installation of k8s myself.
>Doesn't look like the author knows what he is talking about.
This was my first thought, and I was to comment so, but saw you already did. The only reason we see this comment is because HN has an irrational hate of K8s, for us that do run things in production at scale, k8s is the best option. The rest is either wrapped in licenses or lacks basic functionality.
The most starry of starry eyed startups will want a team that is already fit for scale, so they build for scale before they need it. Possibly convinced by VC behaviour and getting as much from each funding round as possible. If you have to scrap your 1st round team and re-employ for 2nd at scale its going to be a hard sell.
557 comments
[ 3.5 ms ] story [ 339 ms ] threadMost folks building software at startups and scale-ups should avoid Kubernetes and other premature optimisations. If your company uses Kubernetes, you are likely expending energy on something that doesn't take you towards your mission. You have probably fallen into the trap of premature optimisation.
Please don't take this post to be only aimed against Kubernetes. It is not. I am directing this post at every possible bit of premature optimisation engineers make in the course of building software.
For me at least, Kubernetes has become something like an universal standard: if you're already running Docker or other containerization, "using Kubernetes" is nothing more than a clear, "single source of truth" documentation on what infrastructure your application expects. Certainly beats that Confluence document from 2018, last updated 2019, on how the production server is set up that doesn't even match up with what was reality in 2020 much less today.
All of these much more annoying and harder than `helm update --install`ing some charts to your own cluster.
If you have a bunch of people who know it, you can deploy a cluster into gcloud or aws with a few clicks or lines of IaC.
Would I recommend a startup team learn kube while trying to ship a product? No.
Would I think it's a red flag if a team who already know it choose it as their preferred platform? Also no.
Personally I don't see too much difference between kube yamls and systemd units and cloudformation or whatever and the "cluster maintenance" itself is not the burden it used to be if you stay on the "paved road" provided by your cloud.
On a project a couple of years ago my cofounder and I opted to use Kubernetes after running into constant frustration after following the advice to keep it as simple as possible and just use VMs.
Kubernetes makes many complicated things very easy and lets you stop worrying about your infra (provided you’re using a managed provider).
On our next project we used a PaaS offering because we had a bunch of credits for it and it was painful in comparison to what we had with Kubernetes. It was way more expensive (if we didn’t have credits), the deployments were slow, and we had less flexibility.
Kubernetes isn’t perfect, far from it. But for those who know it well it is usually a good option.
Hiring people with k8s experience in 2022 is not a difficult task, it's not an obscure technology anymore, and when your initial crop of devops decides to leave, the new guys coming in can scan your deployments and namespaces and pretty much hit the ground running within 48-72 hours. That's a big, important part of running a business. Being able to hire people with the right skill set, and being able to keep the lights on.
Bespoke systems built on top of EC2 require time, extra documentation and a steep learning curve, not to mention the fact that the bespoke system probably isn't getting security audits or being built to modern standards or best practices, tooling isn't being kept up to date. I can build a mechanical wristwatch in my garage, but if I had full access to the Rolex factory floor for free, I'd probably take that option.
I just came out of a project where Kubernetes performance issues involved wild guess and blind tuning until the so called experts actually found out why the cloud cluster was behaving strangely, including support from Cloud vendor.
And good luck making sense of all the YAML spaghetti available for bootstrapping the whole cluster from scratch.
72 hours? They better be 10x DevOps team.
The experience pushed me to decide "never again" and to plonk k3s or at least something like podman from the start on the server.
You were able to do it, but what happens to them when you're not around? Does their team have the required experience to handle it? That's the difference in cost. It's like DYI - yes, if I have all the skills and experience I can do everything myself incredibly cheaply, but... I don't. So I gotta pay.
But if your team does have the skills and experience, it's definitely worth looking into.
I do think people deeply underestimate what can be achieved with a single (or two, if you need a standby) dedicated Linux server these days. A single server can easily go up to 2TB of RAM and 128+ cores. Long before you ever get to a scale where that's a limitation, you'll have more than enough resources to figure things out.
Exactly - $950/mo is nowhere near enough to pay for those skills if you don't have them. It's good value for money.
My first deployment involved virtual machines, but we were essentially packing as much as possible into lowest amount of VMs, then doubled it up so that we had failover. This way we had clear visibility of how many resources we were using and could allocate them according to how much we had.
So disaster recovery is spin a new box, pull the most recent db backup in, and update dns - all scripted should I be gone.
This is why you pay someone. Saving $950/month means it's well worth spending $500 for a day of someone's time occasionally. You don't have to do everything internally when you run a startup. Buying in services that you only need occasionally is well worth the money.
Besides, there are people out there who offer on-call services for a small retainer.
And even on a small retainer you'd better hope they retained the knowledge of how all that stuff works if you're only calling on them every now and again.
And you'd be surprised how little problems you might have - I've had many VMs with literally years of uptime running without issue.
Most people focus on "devops" as a job - and they never bother to teach the rest of the team anything about how stuff works. Worse than that, modern clouds encourage you to build opaque and complicated systems which probably only someone working on them full-time has a hope to understand...
Culture doesn't mean they'll know how to fix <some weird edge case> at 3am in the morning. The SRE with constant ops exposure is likely to have a much better chance, if only because they (ought to) really know how to debug it.
I hope they all have fully patched base libraries and kernels, security auditing is getting to a much more common requirement these days even among very small companies. For example, anybody using Facebook Login.
This is why we document things at the company I work for. If you're serious about a project and want it to exist for a long time there will be things that only come up every few years, and you won't remember them. You can either write things down or you have to relearn how things work every time. Wroting things down is much easier.
But this applies to everything. You also need Heroku or Kubernetes or whatever experience to maintain those systems, right?
LOL - are you f'ing kidding?
Running VMs is much closer to running your app locally than containers and k8s - it's not that hard to do badly, and only marginally harder to do well.
We've lost that, and now everyone thinks we need "sysadmin/devops" for simple and moderately complicated deploys. Heck, most of the guides are out there, follow them. Also, ready-made images and docker containers are amazing these days with config built in. If you look-for or hire devops, you get K8s and all the "formal" items they'll bring with them. You don't need CI/CD for the first 6 months, just PoC the damn thing, get an MVP out and start getting users/customers, the rest will follow.
These systems rarely were constrained in performance by scaling issues that would have be solved by scaling horizontally, and in many cases the added latency of doing so would have caused more problems than it would have solved.
And as you say, having everything on one or two VMs is not just orders of magnitude cheaper hosting, but also comes with benefits of much more easy debugging and performance monitoring.
These weren't tiny start-ups either, these were long running services contracted out to clients including the government and other major companies.
It doesn't scale infinitely, but I'd wager that the traffic just isn't there to justify these kinds of setups in 99% of cases, and that some of the scaling needed is because all the added latency and service discovery etc is adding overhead that wouldn't be needed without it.
I've often found it odd that people who strive for YAGNI at the code level don't apply the same to the system architecture level.
Kubernetes is the right solution to a difficult problem you may or may not have in your future.
I guess redundancy is not really a thing then?
With serverless offerings you can get rather good deals, I don't think you need your own K8S cluster if you can get away with a single Linux host, but a single Linux host is pretty pricey maintenance wise compared to google Cloud Spanner and cloud run.
Most of the time it really doesn't need to be. In the end what you care about is uptime and cost. A redundant solution doesn't have a perfect uptime just because it's redundant, in fact sometimes it might have even less uptime because of failures in the redundancy mechanism. Of course if you need to be always up it might be worth it. But for a lot of situations some downtime is acceptable and most of the time it's better to be on a simpler setup that's easier to debug and less costly to maintain, than to be on a complex one that requires more maintenance and still doesn't have perfect uptime.
Also, I wouldn't underestimate the cost of maintenance in managed solutions compared to self-hosted. It wouldn't be the first time that the managed solutions screw something up or apply some changes and you don't know whether it's your fault or theirs. You also have to add the cost of adapting your solution to their infrastructure, which is not trivial either.
I'm fairly sure that google does a better job keeping cloud spanner and cloud run working, and their redundancy mechanisms working, than whoever runs your single linux box will do.
> most of the time it's better to be on a simpler setup that's easier to debug and less costly to maintain
Keeping a whole linux box running is more complicated and requires more maintenance than Cloud Run and Cloud Spanner.
> Also, I wouldn't underestimate the cost of maintenance in managed solutions compared to self-hosted.
It is not 0, it is just that if you manage cloud run and cloud spanner you don't manage all the other things you have to manage when you self host, and managing cloud run and spanner is really not a lot of effort, it is a lot less effort than managing a standalone database at least.
> You also have to add the cost of adapting your solution to their infrastructure, which is not trivial either.
Cloud run can run stock standard docker containers, you will have a bad time if your processes are not stateless though, and you will have the best time if you have a 12-factor app, but I would not count that as adapting to infrastructure.
And you can use AlloyDB if you want Postgres.
Most of the uptime loss won't come from your provider but from your applications and configuration. If you use Cloud Run and mess up the configuration for the redundancy, you'll still have downtime. If your application doesn't work well with multiple instances, you'll still have downtime.
> Keeping a whole linux box running is more complicated and requires more maintenance than Cloud Run and Cloud Spanner.
Is it? I keep quite some linux boxes running and they don't really require too much maintenance. Not to mention that when things do not work, I have complete visibility on everything. I doubt Cloud Run provides you with full visibility.
> It is not 0, it is just that if you manage cloud run and cloud spanner you don't manage all the other things you have to manage when you self host, and managing cloud run and spanner is really not a lot of effort, it is a lot less effort than managing a standalone database at least.
Managing a simple standalone database is not a lot of effort. For most cases, specially the ones that can get away with running production on a single box, you'll be ok with "sudo apt install postgresql". That's how much database management you'll have to do.
> Cloud run can run stock standard docker containers, you will have a bad time if your processes are not stateless though, and you will have the best time if you have a 12-factor app, but I would not count that as adapting to infrastructure.
That definitely counts as adapting to infrastructure. For example, if I want to use Cloud Run my container should start fairly quickly, if it doesn't I need an instance running all the time which increases costs.
I'm not saying Cloud Run/Spanner are bad. They'll have their use cases. But for simple deployments it's more complexity and more things to manage, and also more expensive. If doing "apt install postgres; git pull; systemctl start my-service" works and serves its purpose, why would I overcomplicate it with going to redundant systems, managed environments and complex distributed platforms? What do I stand to gain and at what cost?
You'd be surprised - the major cloud providers have outages all the time.
Google in particular will have some random backing service firing 502's seemingly randomly while their dashboards say "all good".
There's always 1-2 comments in these threads that advise using a Linode VM or Hetzner dedicated server in order to save money; but they are really skipping over the headaches that come with building and maintaining your own servers.
- Are the server provisioning scripts source controlled?
- Is there a pipeline if the server needs to be recreated or upgraded?
- How is the server secured?
- Are the server logs shipped somewhere? Is there monitoring in place to see if that is working?
- Does the server support zero-downtime deployments?
- Are backups configured? Are they tested?
I imagine the answer to a lot of these questions is no and to be fair not all PaaS systems provide all of these features (at least not for free).
The server becomes a pet, issues inevitably arise and that $950 starts to look like a false-economy.
If you do it correctly, it is loads easier to reason about and understand than a complex k8s deployment.
But for startups and/or simpler setups, ECS/Lambda is so much less work, while usually being powerful enough.
Which one(s) run your API/server, your database, etc is basically arbitrary. You’re free to let the K8s scheduler do as it see fits, or you can get increasing degrees of control by adjusting things like node and pod affinity/anti-affinity, etc.
* You describe what type of state you need (Deployment = one or more Pods running your container, Service = expose the deployment) * Kubernetes runs a control loop that reconciles the current cluster's state with your desired state
Nodes are the virtual machines on which everything runs, you don't need to interact with them unless you have special requirements (e.g. GPU instances). In that case, you annotate GPU instances with some type of label (like `has-gpu=true`), then in your deployment you add a node affinity saying it needs to run on nodes that have that label. Kubernetes will schedule it for you if there's any node matching.
Also something like karpenter.sh is much nicer than ec2 autoscaling group.
Yes, you can build an amazon machine image for ec2, update your cluster with the new image, without scaling from zero.
> As others have already pointed out, it is also important for us to remain independent from Google Cloud / AWS.
This seems like a good reason to use k8s, although as the original article touches on, it seems like switch cloud vendor rarely ever happens.
A lot of data tools are starting to target Kubernetes directly as a runtime so using them with GKE/EKS is a bit simpler as it’s officially supported, allows to run locally and on the cloud with no vendor lock in.
ECS in a scaling group works well if your app is stateless but as soon as you scale workers dynamically, do service discovery, orchestration, you end up building some of the features Kubernetes provides.
it beats the pants off the the random rubricks of ansible and puppet projects I've run into the past anyway
Nothing I've ever done has taken off at all so automating DevOps would have been a total waste of time. Then again the projects themselves have been a complete waste of time.
We never had an issue with multiple languages across tier-n architectures.
Suddenly with the uptake of HTML 5, it became an issue not being able to use JavaScript everywhere.
That does ignore the fact that an experienced frontend JS dev is not necessarily also a good productive backend JS dev, but at least they know the language basics, can use the same IDE, etc.
If thats worth it is something that depends on what you try to achive, I guess. I personally would not pick JS (nor TS) for the backend.
In practice though the main reason is that to have decent DOM bindings you need to stabilize many other specs first (unless you do a ultra-specific DOM-only extension, but nobody wants that)
Rust, for example, has a huge package with interfaces for every DOM function.
But I wouldn't recommend it in general, because JS (and for slightly more complex things, TS) is much easier to create because of all the old Rust features that surface every time you mention it. And most GUI code is simple enough that you only have to write once, maybe fix a few details, and forget about it.
Wasm would be much more compelling if it was target by higher level languages.
The rise of the "fullstack developer" has mostly reduced quality across the board. When you hire a "fullstack developer with 5 years experience" you aren't getting someone who is as good as a frontend developer with 5 years AND a backend developer with 5 years but someone that adds up to 5 years split between those 2 endeavors but probably with less depth as a result of switching.
(as a side note I also think it's contributed to developer title inflation)
Learning a new language and its tooling is comparatively easy compared to learning the required domain knowledge to be effective in a new area. i.e transition from frontend -> backend or visa versa has very little to do with the language or tooling.
Your average frontend dev doesn't know squat about RDBMS schema design or query optimisation, probably aren't familar with backend observability patterns like logging, metrics and tracing, most likely have very little experience thinking about consistency and concurrency in distributed systems, etc, etc.
Just like the backend dev is going to struggle with the constraints of working in a frontend context, understanding UI interactions, optimizing for browser paint performance, dealing with layout and responsiveness, etc.
Meanwhile if you know say Java and some scripting language, say Python and you end up a new job doing backend in JS it's not going to take long for you to pick up JS and hit the ground running because you are going to encounter exactly the same stuff just different syntax and runtime.
Backend being substantially divorced from frontend isn't a bad thing, it's generally a good thing that results in nice clean separation of concerns and healthy push-pull in design decisions about where certain logic and responsibilities should lie etc.
The bar is generally just a lot lower now across the board. We have people with the title "Senior Engineer" that still regularly need mentorship and peer-programming sessions.
The title I have now (Principal Engineer) I feel like I don't deserve, it was reserved for folks I looked up to when I was starting and I don't think I have reached their level. Yet at the same time it's essentially necessary to distinguish myself from what is now considered "Senior".
I have a separate rant for the lack of proper juniors and lack of structured mentoring etc but that is for another day.
Having a solo developer that can do it all well enough also allows useful products to reach users faster (and for me it's really about solving user problems, not just making money), without getting derailed by communication overhead, bureaucracy, fighting within the team, etc. Just make sure you don't pick a developer who cares too much about things that mostly matter to other nerds, because then they might get derailed with premature optimization. Yeah, I've been there.
The chunking unit is programming languages which means as long as you can call the task the specific programming language, people will believe it's the same thing.
In reality you have a hammer expert and are in the business of making everything look like a nail.
So now we have complicated bloated applications of JavaScript in places where it's absolutely inappropriate and you need far more skill to navigate those waters then someone who uses more appropriate tools.
It's a perversion in the name of simplicity because we're forcing too coarse of a model on too fine of a problem and as a result everything explodes in complexity, takes too long, is too expensive and comes out working like trash.
We could do better but first we have to eat crow and admit how foundationally wrong we were and frankly things aren't bad enough to make that mandatory so the circus continues just like it did when we tried to make everything OOP and it made the same smell.
They're useful tools, but that's their perimeter; they aren't zeitgeists... well they shouldn't be.
It's a very effective way to make sure your team is only composed by junior developers.
I write elixir that smells like rails, but as the months move on, I change.
You also see it when people who've mostly done imperative programming tries their hands with a lisp or some ml based language. I've been there myself. You still find yourself using the equivalent of variable bindings, trying to write in an imperative rather than declarative style.
I guess when trying to learn a new language in a different paradigm you also need to unlearn a lot of the concepts from the former language.
Someone learning a first language isn't just learning a new language: they're learning how to program. It's a new profession, a new hobby, a new superpower.
The rest of the stuff (standard library, ecosystem, buildsystem and all kinds of intricacies) is just a mix of trivia and bureaucratic garbage you gotta fill your brain with but will all be replaced in 10 years anyway. Sure it takes time but it's nowhere near as important as actually knowing how to program.
Even changing paradigms (imperative to functional) isn't as hard as learning the first language.
A lot of beginning programmers struggle with this kind of stuff, as did I. Reading this fluently and keeping track of what variables are set to takes quite a bit of practice.
I've been hired to work on languages I had no prior experience on, and while there was of course some ramp-up time and such, overall I managed pretty well because variables are variables, ifs are ifs, loops are loops, etc.
The argument that learning a second language is as difficult as learning the first doesn’t really hold water in practice as well, lots of people have done so.
IME a lot of the conceptual stuff transfers pretty well there too.
Most job offers are for a mythical full-stack developper that'll master web design, CSS/HTML, front-end interactions and code, networking, back-end architectures, security,... You end-up with people who don't have time to get enough expertise and write and build clean stuff. Hacking poor JavaScript code everywhere.
With the same language, you may think you can somehow reuse stuff between front and back. A bad idea in most project, will typically create more problem than it'll solve.
If you do server-side rendering with something like nextjs, then its quite a lot of code.
With trpc you can share types without going through an intermediary schema language (https://trpc.io/) although I think it would've still benefitted from separating the schema from the implementation (hope you don't forget the type specifier and import your backend into your frontend by accident)
For business logic as always the anwwer is "it depends". Does your app benefit from zero-latency business logic updates? Do you need rich "work offline" functionality combined with a rich "run automation while I'm away" for the same processes? etc.
I'm a big Nextjs fan; I just think it's useful to emphasize that using it doesn't necessarily have to mean "only javascript/typescript on the backend".
> While Remix can serve as your fullstack application,
Isn't that where most teams should start, and where they should stay unless they have a really good reason to get more complicated? This is what I was thinking of while reading the GP comment about Next.js.
So, if you can keep all of your templates on a single place, and don't have a lot of data diversity, you won't have a problem. But if that's not your case, you do have a problem.
Personally, I think the frontend code should be more generic on its data and organized around operations that are different from the ones the backend sees. But I never could turn that ideal into a concrete useful style either.
Better to have a full stack developer that can concentrate on becoming an expert and fluent in just one language rather than being kinda-ok in 3 or 4 IMHO.
The real expertise is being a front-end expert, authoring efficient architecture around user interactions, browsing the MDN without effort, mastering the DOM... Then, being a back-end expert, knowledgeable on scaling, could architectures, security issues, being able to author good API design...
If you can be considered an expert in all of this, and also edge computing, I don't think switching language would be an issue for you. Language is a tiny fraction of required expertise and you might be more productive by switching to an appropriate one.
And they don't really have time to learn new stuff, as they spend too much of their time on Hacker News complaining there's too many new frontend frameworks or something like that.
It is extremely easier to find more affordable talent that has come out of a boot camps knowing javascript but more specifically "react/nodejs" which lets them work both frontend and backend. They rarely know best practices, how to properly debug & troubleshoot a problem that isn't a Google search away, etc, but they will be hungry and work their butts off to ship their CRUD style features.
Also I've seen UX research that established some guidelines for how long something can take while still feeling responsive allowing a user to remain focused.
Knowing you can achieve the required/desired performance in any given language is mostly a matter of experience solving similar problems.
But the overwhelming majority of the time you don't need it, at least not yet. I would say that unless you have actual evidence that your other language would not be adequate - i.e. an implementation of your system or some representative subset of it, in your main language, that you spent a reasonable amount of time profiling and optimizing, and that still proved to have inadequate performance - then it is indeed premature.
This I very different to say starting a with a micro service architecture which imposes relatively high overheads, with little benefit as splitting up a well designed monolith is easy.
The cases where Go is significantly faster than JavaScript are vanishingly small.
> whereas rewriting it later would be a massive undertaking.
This is vastly overstated IME. Porting existing code as-is between languages is actually pretty easy.
> This I very different to say starting a with a micro service architecture which imposes relatively high overheads,
Disagree. You're imposing a huge overhead on hiring (if you want people with both languages) or on people's ability to work on the whole system (if you're happy hiring people who only cover one side or the other). Debugging also gets a lot harder. There's essentially twice as many tools to learn.
Yep. We always implemented (parts of) embedded software in Java first and later in JS and then port it. If no additional functionality is added, this is trivial and saves a lot of work and errors as you already tested, debugged and fixed the logic parts.
It's not only about performance but also energy and resource efficiency.
Which is to say ... while JS developers do know how to profile code, people who are routinely exposed to this problem are not going to be JS developers. The people who are good at identifying when a system needs to be re-written in Go for performance reasons are probably already Go developers and people who have lots of experience writing performant code.
Plus writing in that sort of language from the start means there is a chance to segue into performant code without needing to rewrite things in a new language.
Sure they are. The skills aren't really language-dependent, and nowadays machine code is so far away from the actual hardware behaviour that it doesn't actually help a lot. Besides, the biggest speedups still come from finding errors or inappropriate algorithms or datastructures, and that's if anything easier to spot in a higher-level language where there's less ceremony to get in your way.
when I interview developers, folks coding in Java can usually tell me what an appropriate datastructure would be, while half the JS developers can't explain how an array is different from a linked list. Because most of the time JS developers. dont have to think about it much.
Well said: this should be the only rule!
"Using different languages" is just Tuesday in most places. Sure, don't use languages needlessly, but it's not a big hurdle unless you're just a "nodejs bro"
Given that probably most developers support the push to tackle the climate change, they seem to be making no effort to ensure their apps execute in as short time as possible using as little resources as possible. You would expect that people would actually embrace doing things in C or Go to save energy. Maybe cloud providers should think of showing the carbon footprint as a first class performance metric.
When you want to save the climate there are many, many low-hanging fruits. The choice of programming language is likely not one of them, as much as I like efficient code.
Like so many others, the author appears to be latching on to the phrase "premature optimization" as a popular buzzword (buzz...phrase?). This is so far from what Knuth actually wrote in his book that it hurts.
That's fine.
Premature optimization would be saying:
I don't know Go/C++/C... but I know its fast, so instead of using what I know to get up and running quickly, i'll waste time building it in something I don't know which probably wont be as fast as doing it in something I know well.
The thing is, when you're building it, no one is using it! If it's slower in RoR than Go, who cares, get it up and running, and fix it later when people are actually using it.
Much, much less work to then move the deployments elsewhere, including building and teardown of integration testing environments.
K8s signals the team values infra and sees it as an important part of delivering their product.
It's an investment like any other and for some teams it makes sense to set infra on a solid path sooner rather than later.
You can move fast on things like serverless but the limitations of those systems can force architecture decisions that are sub-optimal whilst control of your own infrastructure can allow broader interpretation of problems and enablement of more wholistic solutions.
i.e it opens up building things "the right way" sooner in many cases and that can save a ton of time and frustration.
Also in the case where it's obvious you will outgrow serverless it also saves on a sometimes painful migration effort.
But isn't "infra[structure]" what you run your product on, and not part of the product itself? If your "product" is so big it needs stuff like that built into it, then it feels to me that by defintion that stuff isn't "infra".
All in all, your sentence feels like "The Excel-Dell Humongo-XPS bundle signals that Microsoft values infra and sees it as an important part of delivering their product" to me. No, that only signals that A) Dell and Microsoft are using the dominant marketing position of Excel to milk money from naive customers, and B) If Excel can't run on more modest hardware, it's an unnecessarily bloated and inefficient product; look around for another spreadsheet.
First of all k8s isn't bloated, it's actually surprisingly compact if you think about what you are getting.
Second it doesn't imply the product -needs- it. Anything you can run on k8s can be run on bare boxes.
Just doing so probably means you either lack the scale to need k8s (i.e my spreadsheet doesn't have many users) or you lack the competence to understand you have reached the scale that it's required (i.e your spreadsheet is run by n00bs). So it's much more of a positive than negative signal if you are seeking a vendor. I think the product relationship thing is a poor analogy from that side though, it -mostly- matters for people that want to work there on/in/around the product itself.
As a primarily infra guy I have a preference for k8s because it allows me to have some reasonable set of services I can rely on being there if a place is "using k8s". If not I just have to hope either the existing infra team is competent enough to have things like service discovery, secret management, process monitoring, etc under control or that they are amenable to efforts to get it under control.
If I am approaching it from a backend developer perspective where most of that isn't my problem then it gives me some confidence the infra team has some idea what they are doing and that I won't need to deal with some sort of bespoke deployment system that has endless quirks and company-specific bugs that I shouldn't need to concern myself with.
Generally k8s just eliminates uncertainties by increasing standardisation so I don't need to watch people solve the same problem poorly over and over again for no reason at all.
Haha, this is exactly what “hobby” means to a lot of people. Less judgmental: thinking and dreaming about the right tools in disproportion to the need is something people do a lot, presumably because it is a source of joy.
I feel that Kubernetes is exactly that for a large number of developers, operations people less so, they want to be part of a professional, world class, trendy environment/community. So they try leverage Kubernetes to elevate them to this Silicon Valley type tech company.
There's absolutely value to be had from Kubernetes, but I agree with the article, it's not for the majority. Except I think there is one valid reasoning: If you purely see Kubernetes as tooling for deployment, it's hard to present any good alternatives for deploying to VMs or physical hardware.
I see it the same type of folks as ones using FAANG tooling :) just so they can feel better about their job or more important when they work on business line CRUD.
I am working on business line CRUD and I like it with boring servers :)
This is a superpower that you want to have individually as an engineer but you need organisationally as a culture too. If you have that power then you when you need to write some similar code you can copy and paste, because you can trust that when you have a third use you will actually refactor properly. It means that you can get the service up and running quickly on a box you've got, because you can trust that if the service actually gets used your org will allow you the time to put it on the cloud properly.
It's the most important part of agile - make the right decision for what you've got now, and then make a different one later.
It's about having a standard API for deployment artifacts.
The k8s manifests are trivial (if verbose) the complexity comes from running the underlying layer which when you are small you simply outsource to AWS or GCP.
There is some k8s know-how that is table stakes for a good experience, namely knowing what extra stuff you need on top of a base k8s cluster. i.e certmanager, external-dns.
Overall it's a lot less required knowledge than it takes to manipulate lower level primitives like GCP/AWS directly or be capable of setting up standalone boxes.
Before anyone starts with "but serverless!" you need to consider the spagetti you end up with if you go down that route, API Gateway, 100x Lambdas and thousands upon thousands of lines of Terraform boilterplate does not a happy infra team make.
My experience is even mediocre engineers are capable of understanding k8s concepts with minimal assistance assuming things are either sufficiently standard (i.e google-able) or well documented with any company specific practices.
K8s == hard to run, k8s != hard to use.
I did get everything running but also saw so many settings and features I had to take on faith as being handled without understanding it. I just chose not to bother further because this seems like a minefield of problems in the future as we on boarded other engineers. The number of times I’ve seen production deployments go down due to k8s misconfiguration on other teams has only validated my concerns.
You aren't meant to fully understand it in a week, anymore than you are expected to fully understand all of sysadmin in a week.
Just because you don't know what every single directive in nginx config does doesn't mean you can't use it effectively and learn what they mean when the time comes.
k8s isn't much different. You don't need to know what a liveliness probe is the first time you use it, you can learn about it as you go (most likely when you run into a badly behaved program that needs forced restarts when it hangs).
Ofcourse if you are running it yourself that is entirely different, you really do need to know how it works to do that but you should be using hosted k8s unless you have hardcore systems folk and actually need that.
The solution I went with, given my limited knowledge of some of these things, was to use elastic beanstalk. You write your flask application, upload the zip and that’s it pretty much. You get a UI to do all the configurations and for the most part nothing there is hard to decipher or google. The only hiccup might be when you’re trying to connect it to RDS and to the external internet but even that is straightforward as long as you follow the right instructions. We run apps that power entire SaaS organizations and this system seems to be more than sufficient. Why complicate further? We have other fish to fry anyway.
On the other hand, with docker swarm, I built a cluster in less than a day. So yeah setting up k8 isn't as trivial as you are making it out to be.
For comparison, you can just as easily expose/hide your VM as your managed k8s cluster.
I do not agree with this. I have tried to start Kubernetes and it's far more confusing to set up than just setting up something on a standalone box, mainly because it looks like the set of knowledge I need to set up anything on Kubernetes is a superset of what I need to set up the same thing in bare Linux.
Lets see to get something even half-reasonable on a bare box you need the minimum:
process monitor: For this you can use systemd, supervisord, etc.
logging: rsyslog or similar.
http reverse proxy: nginx or haproxy
deployment mechansim: probably scp if you are going this ghetto but git pull + build and/or pull from s3 are all common at this level plus a bunch of bash to properly restart things.
backups: depends on persistence, litestream for sqllite these days, pgdump/whatever for RDBMS, wal shipping if you are running the DB on your own boxes.
access control: some mechanism to manage either multiple users + homes or shared user account with shared authorized_keys.
security: lock down sshd, reverse proxy, etc. stay on top of patching.
So are bare minimum there is a ton of concepts you need to know, you are essentially regressing back into the days of hardcore sysadmins... just without all the hardcore sysadmins to do the job correctly (because they are now all busy running k8s setups).
Meanwhile to deploy a similarly simply app to k8s (assuming you are buying managed k8s) you need to know the following things:
container + registry: i.e Dockerfile + docker build + docker push gcr.io/mything.
deployment: Describes how to run your docker image, what args, how much resources, does it need volumes, etc. You can go ghetto and not use service accounts/workload-identity and still be in better shape than bare boxes.
service: how your service exposes itself, ports etc.
ingress: how to route traffic from internet into your service, usually HTTP host, paths etc.
Generally this is equivalent but with way less moving parts.
There is no ssh server, patching is now mostly just a matter of pressing the upgrade button your provider gives you. You now only need to know docker, kubectl and some yaml instead of systemd control files, sshd_config, nginx.conf, bash, rsync/tarsnap/etc.
K8s reputation as being hard to run is deserved.
K8s reputation as being hard to use is grossly undeserved.
And by the looks of it, ingress doesn't seem trivial, not only you need to understand how NGINX works for reverse proxy but also understand how Kubernetes interacts with NGINX. You also ignored backups for Kubernetes, if you're just doing full disk backups you can do that with regular boxes, specially if they're just VMs.
> There is no ssh server, patching is now mostly just a matter of pressing the upgrade button your provider gives you.
No SSH server might be an advantage for you, but for me it means it's far more difficult to know what's happening when Kubernetes doesn't do what I want it to do.
> You now only need to know docker, kubectl and some yaml instead of systemd control files, sshd_config, nginx.conf, bash, rsync/tarsnap/etc.
But I still need to know how to set up all applications I use, which is the most important part. And understanding systemd service files, sshd_config, nginx.conf and rsync is far, far easier than understanding Kubernetes. Kubernetes manages those things so you actually need to understand both the underlying concepts and also the abstractions Kubernetes is making.
You are also comparing self-hosted with managed, and then only mentioning the disadvantages of self-hosted. Of course with a managed setup you don't need to worry about SSH, but you have to worry about actually dealing with the manager service. If you were to set up self-hosted Kubernetes you'd still need to learn about SSH, or about account management. That's not something about Kubernetes but about self-hosted or managed.
Ingress is trivial to use. Internally it's less trivial but you don't need to peek inside unless you manage to break it which generally speaking, you won't; even if you think whatever you are doing is very special it probably does it already. That is the benefit of literal thousands of teams using the exact same abstraction layer.
You don't have SSH (this is objectively good), instead you can get exec on a container if you need it (and that container contains a shell). You can now control on a fine grained and integrated fashion exactly who is allowed to exec into a container and you get k8s audit events for free. (and if you are using a hosted system like GKE then it automatically flows into said providers audit system).
The point is you can buy managed k8s, there isn't an equivalent for old school sysadmin that doesn't amount to outsourcing to a body shop.
Given that you haven't managed to identify any of k8s real downsides here they are:
It's a bunch of yaml. Yeah I don't like that either but there are tools like Tanka that make that suck less.
It's complex under the hood. When your process runs its in a network, fs and pid namespaces and you have cgroup managing resources. There is probably some sort of overlay network or some other means of granting each container an IP and making that routable (BGP, OSPF, etc). Interaction between ingress, service and pods is through an indirection layer called endpoints that most people don't even realize exists. Scheduling is complex, there are tunables for affinity, anti-affinity, it also needs to interact with cluster autoscaling. Autoscaling itself is a complex topic where things like hot capacity etc aren't exactly sorted out yet. QoS, i.e pod priority isn't well understood by most. When will a pod be pre-empted? What is the difference between eviction/pre-emption etc? Most people won't be able to tell you. This means when things go wrong there are a lot of things that could be wrong, thankfully if you are paying someone else for your cluster a) it's their problem b) it's probably happening to -all- of their customers so they have a lot of incentive to fix it for you.
Multi-cluster is a mess, no reasonable federation options in sight. Overlay networking makes managing network architecture when involving multiple clusters more difficult (or external services outside of said cluster).
Ecosystem has some poor quality solutions. Namely helm, kustomize, Pulumi, etc. Hopefully these will die out some day and make way for better solutions.
Yet for all of these downsides it's still clearly a ton better than managing your own boxes manually. Especially because you are -exceedingly- unlikely to encounter any of these problems above at any scale where self-hosting would have been tennable.
I think the best way to think about k8s is it's the modern distributed kernel. Just like Linux you aren't expected to understand every layer of it, merely the interface (i.e resource API for k8s, syscalls/ioctl/dev/proc/sysfs for Linux). The fact everyone is using it is what grants it the stability necessary to obviate the need for that internal knowledge.
Dockerfiles aren't trivial. I've helped migrate some services to docker and it's not "just put it in Docker and that's it". Specially because most applications aren't perfectly isolated.
> Ingress is trivial to use. Internally it's less trivial but you don't need to peek inside unless you manage to break it which generally speaking, you won't; even if you think whatever you are doing is very special it probably does it already. That is the benefit of literal thousands of teams using the exact same abstraction layer.
Literal thousands of teams also use Nginx and still doesn't mean that there aren't configuration errors, issues and other things that are to debug. Not to mention that a lot of applications can run without requiring a reverse proxy.
> You don't have SSH (this is objectively good), instead you can get exec on a container if you need it (and that container contains a shell). You can now control on a fine grained and integrated fashion exactly who is allowed to exec into a container and you get k8s audit events for free. (and if you are using a hosted system like GKE then it automatically flows into said providers audit system).
And that's cool if you need that, but "ssh user@machine" looks far easier.
> The point is you can buy managed k8s, there isn't an equivalent for old school sysadmin that doesn't amount to outsourcing to a body shop.
Look, the other day I had to set up a simple machine to monitor some hosts in a network. I've Ansible to automate it, but ultimately it boils down to "sudo apt install grafana prometheus; scp provisioning-dashboards; scp prometheus-host-config; scp unattended-upgrades-config" plus some hardening configs. I don't need redundancy, uptime is good enough with that. If Prometheus can't connect to the hosts, I can test from the machine, capture traffic or whatever and know that there isn't anything in the middle. If Grafana doesn't respond I don't need to worry whether the ingress controller is working well or if I missed some configuration.
Could I run that with Kubernetes? Well, managed k8s is already out of the window because the service is internal to an enterprise network. And self-hosted kubernetes? I still need to do the same base linux box configuration, plus setting up k8s, plus setting up the infra and networking with k8s plus the tools I actually want to use.
And that's the point. I am not trying to identify any k8s downsides, what I said that setting up k8s looks far more confusing than setting up a standard Linux box, and by the things you say I am even more convinced than before than it is indeed. I still need to understand and run the programs I want (which is most of the cost of deployment), but I also need to understand Kubernetes and, if I am using a managed platform, I need to understand how the managed platform works, configure it, link it with my system... In other words, not only do I need the general knowledge for application deployment but also the specific implementation in Kubernetes.
More than one container then cloud hosted kubernetes makes sense. It's a transferable skill a kind of learn once apply to any cloud technology.
In a quest to get closer to the metal, Kubernetes keeps you far away, which is the opposite of what any production service should want.
What is the purpose of adding layers when uni-kernels and eco-kernels give you better isolation and better performance?
Your cloud provider already runs your virtual machines OS on a hardware hypervisor. Then running Kubernetes on top of the OS and then a zillion containers on it is a recipe for poor performance.
What is the logic here that I am clearly missing?
Cloud providers native Kubernetes stacks don’t improve performance or pricing, compared to their cloud compute instances virtual machines and a dedicated virtual machine per would-be-container that thankfully doesn’t need to now share processing resources with others.
What gives? Why on earth would anyone run production processes in Kubernetes?
*i have used kubernetes and understand how to use it as intended. I feel like I am missing the motivation for it’s current widespread usage in web apps
The minimum virtual machine size for a Windows server that is at all useful for anything is 4 GB of memory. Okay, okay, so you can technically boot it up on 2 GB and some roles will work fine, this will last only until some dingbat remotes to it with RDP with a 4K monitor and it starts swapping to disk.
Even if you use Server Core and block port 3389, it still needs a ton of memory just to start.
Running in a container it uses a few hundred megabytes.
Similarly, the minimum system disk size you can get away with is 32 GB if it is a discardable / ephemeral instance. You need 64 GB minimum if you ever intend to run Windows Update on it.
With containers, the unique parts of the image might be just a few hundred megabytes, even for complex apps.
My experience is with Windows, but from what I hear Linux VMs vs Linux containers have vaguely similar ratios.
So with containers, a single host can run dozens of applications, all sharing the same base disk, and all sharing the same OS kernel. The savings can be staggering.
At $dayjob, the admins are very much stuck in the dedicated VMs for every role mentality, and they're burning through enormous piles of taxpayer money to run them at literally 0.1% load.
Having said that, Kubernetes has its own problems. As you said, layering it on top of cloud VMs is a bit silly, and can easily result in the container running in a nested hypervisor at molasses speeds. Similarly, every single typical process changes dramatically: Deployment, updates, monitoring, auditing, etc...
Combine the above with the incompatible underlying cloud layer and things get really messy really quickly.
In my experience 90% of the world just isn't ready for the learning curve. Windows as an operating system isn't ready, certainly. Microsoft Azure isn't really ready either. Their AKS managed offering is still undergoing massive churn and seems to have more preview features than stable features. Even in the Linux world I hear more horror stories than success stories. It seems that everyone who says they love Kubernetes is using it on like... one machine. Come back and tell me how you feel after troubleshooting a failed upgrade on a cluster managing $100M of finance transactions.
What I would like to see is "native Kubernetes clouds" where the hosts are bare metal and there is no impedance mismatch between K8s and the cloud provider APIs because K8s is the API. Instead of the Azure portal or the AWS console you literally log into a Kubernetes console.
IMHO that would allow a true commoditisation of the public cloud and start to erode the near-duopoly of AWS and Azure.
They run on hardware.
Ultimately, there needs to be a singular scheduling system running on hardware and a singular HAL-like driver layer, and exo or iso kernels deliver just that, vs lxe containers provided by os services.
The sizes are also quite impressive.
Example of thing that is pretty straightforward in K8s and much less straightforward outside of it.
1. For compliance reasons, you need to make sure that your underlying OS is patched with security updates.
2. This means you need to reboot the OS every X time.
3. You want to do this without downtime.
4. You have a replicated architecture, so you know you have at least two copies of each service you run (and each can handle the required traffic).
In K8s, this operation can be as simple as:
1. Mark your old nodes as unschedulable.
2. Drain your old nodes (which will grab new nodes from your cloud provider, installing the updated machine image).
3. Delete your old nodes.
The exact steps will differ based on your use case, but that's basically it.
Steps you didn't need to think about here:
1. If I'm about to take a node down, do I have enough redundancy to handle its loss? K8s understands the health of your software, and you have a policy configured so it understands if taking down a container will cause an outage (and avoid this). Note: third party tech will be naturally compatible - if you use Elastic's cloud-on-k8s operator to run ES, it'll appropriately migrate from host to host too, without downtime. Likewise, the same script will run on AWS, Azure, GCP.
2. How fast can I run this? If building this logic yourself, you'll probably run the upgrade one node at a time so as to not have to think about the different services you run. But if it takes 15 minutes to run a full upgrade, you can now only upgrade 100 hosts each day. K8s will run whatever it can, as soon as it can without you having to think about it.
3. What happens if concurrent operations need to be run (e.g. scale-up, scale-down)? With K8s, this is a perfectly reasonable thing to do.
4. Does this need to be monitored? This is a fairly standard K8s workflow, with most components identical to standard scale-up/scale-down operations. Most components will be exercised all the time.
Generally I've been impressed by how straightforward it's been to remove the edge cases, to make complex tech fit well with other complex tech.
A while back we upgraded between two CentOS versions. In such a case it's recommended to reinstall the OS - there's not a clear upgrade path. In K8s, this would have been the same set of steps as the above. In many orgs, this would be a far more manual process.
This includes small / medium size organizations (like when you are not Google or Amazon), startups, one-man projects, and "hobby" projects. " https://dockerswarm.rocks/
You can use elastic beanstalk, and RDS. You get scaling, reliability and backups. And it’s not complicated and you can set up easy CICD with just GitHub actions.
If you don’t need scaling, just use lightsail and litestream.
There are many many options to do things simply.
What if you dont need multiple servers? Well go the single node approach and have a flexible, true and tested way to spin up containers, which can and should be able to crash whenever they have to.
Using Containers is pre mature optimization too? maybe I should get my typewriter.
More like grab a damn pen and scrawl out what you actually need. It's flexible and fast, which in many cases suits the situation best.
You seem to have missed the alternative "get a computer" in between there.
I have just provisioned another bare metal k8s cluster, taking software of dedicated servers and.... well if I dont have to, im not running bare metal anymore in 2022.
If you want to just "get a computer in there" have a look at harvester, its the best of the two worlds
Also he talks about all of this and then gives example of WhatsApp at the end. WhatsApp chose Erlang for the backend and their front end was written in Java and Objective-C. They could have chosen Java for backend to keep frontend language same but they didn't. They used Erlang because they based their architecture on Ejabberd which was open source and was built with Erlang. Also WhatsApp managed all their servers by themselves and didn't even move to managed cloud services when they became available. They were self hosting till FB acquired them and moved them to FB data centres later on (Source: http://highscalability.com/blog/2014/2/26/the-whatsapp-archi...).
Just because WhatsApp scaled to almost half a billion users with a small engineering team doesn't mean that's the standard, or even achievable, for almost all teams.
I know of such a case, where a single engineer could leverage the helm chart open source community, and set up a scalable infrastructure, with prometheus, grafana, worker nodes that can scale independently of web service, a CI/CD pipeline that can spin up complete stacks with TLS automated through nginx and cert-manager, do full integration tests, etc.
I found that to be quite impressive, for one person, one year, and would probably be completely impossible if it wasn't for k8s.
I've seen places hire a dev that write all the OPS stuff and they scaled awesomely.. I mean if they had purchased 100servers full time on amazon, they would have spent a fraction of the cost to scale, but they could scale. In 5 years I think they've never once had to reach even near the 100servers.
At the same time. I can scale heroku to 500 servers, and still be under the cost of one ops person. I can make that change and leave it there. I can do that all in under 30 seconds. Oh. And CICD is built in as a github hook. Even with blue-green deploys.
I think his point was most start-ups don't need to scale more than a site like heroku can offer. If you need more than 500 servers running full time then it's time to start looking to "scale"
And then Heroku shuts down.
If you're building something that needs to scale up rapidly if it succeeds, k8s is worth thinking about. Either you don't succeed, in which case it doesn't matter what your stack was, or you do, in which case you'll be glad that you can scale up easily, you'll be glad you are using a common platform which is easy to hire competent people in, and, if you were smart about how you used k8s, you'll be glad that you can relatively easily move between clouds or move to bare metal.
I agree, but so what? K8s isn't magic, it won't make all your problems go away, but if you have people who are genuinely skilled with it, it solves a lot of problems and generally makes scaling (especially if you need to move between clouds or move onto bare metal) much smoother. Of course you'll still have other problems to solve.
Given that most startups never need to scale up much, it's not surprising that k8s is mostly used where it's not needed. But people usually prefer not to plan for failure, so it's also not surprising that people keep using it.
"Why no k8s?", you ask, only to be met with incredulity: "We don't have those skills", says the profiteering web agency. Sure, k8s is hard… Not. Nevermind that it's pretty much the only important part of your job as of 2022.
That's because 'the norm' for web agencies is a site that does basically zero traffic. If a company hires a 'web agency' that's by definition because the company's business model does not revolve around either a web property or app.
Whether that's a gas station company or a charity or whatever, the website is not key to their business success and won't be used by most customers apart from incidentally.
With that in mind most agencies know only how to implement a CMS and simple deployment perhaps using Cloudflare or a similar automated traffic handling system. They don't know anything about actual infrastructure that's capable of handling traffic, and why would they?
A lot of agencies are 100% nontechnical (i.e. purely designers) and use a MSP to configure their desktop environment and backups and a hosting agency to manage their deployed sites.
In asking if "Kubernetes is a red flag signalling premature optimisation", you correctly explain why we're yet on the "yes" side for the typical web agency category.
[Although FWIW I was hinting at a non-trivial category who should know better than not to setup a scale-ready infra for some potentially explosive clients; which is what we do in the entertainment industry for instance, by pooling resources (strong hint that k8s fits): we may not know which site will eventually be a massive hit, but we know x% of them will be, because we assess from the global demand side which is very predictable YoY. It's pretty much the same thing for all few-hits-but-big-hits industries (adjust for ad hoc cycles), and yes gov websites are typically part of those (you never know when a big head shares some domain that's going to get 1000x more hits over the next few days/weeks), it's unthinkable they're not designed to scale properly. Anyway, I'm ranting now ^^; ]
My unspoken contention was that eventually, we move to a world where k8s-like infra is the de facto norm for 99% of infrastructure out there, and on that road we move to the "no" side of the initial question for e.g. web agencies (meaning, we've moved one notch comparable to the move from old-school SysAdmin to DevOps maybe, you know those 10 years circa 2007-2018 or so).
[Sorry for a too-terse initial comment, I try not to be needlessly verbose on HN.]
This is very very hard to parse BTW. I don't want to reply to what you've written because I can't determine for sure what it is that you're saying.
Essentially I mean: scalable infra may be premature optimization today in a lot of cases, but eventually it becomes the norm for pretty much all systems.
You could similarly parse the early signs of a "devops" paradigm in the mid-2000's. I sure did see the inception of the paradigm we eventually reached by 2018 or so. Most of it would have been premature optimization back then, but ten-ish years later the landscape has changed such that a devops culture fits in many (most?) organizations. Devops being just one example of such historical shifts.
I anticipate the general k8s-like paradigm (generic abstractions on the dev side, a full 'DSL' so to speak, scalable on the ops side) will be a fit for many (most?) organizations by 2030 or so.
I hope that makes sense.
It's also worth considering that appropriate use of k8s can quite likely save you time and money early on as well. It standardises things, making it very easy for new ops people to onboard, and you might otherwise end up spending time reinventing half-baked solutions to orchestration problems anyway.
Well, precisely what I said is that 99.9% of startups won't find themselves in a situation where they need to scale quickly and the only scale problems they find can be solved with Kubernetes.
> It's also worth considering that appropriate use of k8s can quite likely save you time and money early on as well. It standardises things, making it very easy for new ops people to onboard, and you might otherwise end up spending time reinventing half-baked solutions to orchestration problems anyway.
The point is that you might not even need orchestration from the start. Instead of thinking how to solve an imagined scenario where you don't even know the constraints, go simple and iterate from that when you need it with the actual requirements in hand. And also, "make it easier for new ops people to onboard" doesn't matter if you don't have a viable product to support new hires.
There are a lot of companies in the middle, and following dogma like "you don't need k8s" leads them to reinvent the wheel, usually badly, and consequently waste enormous amounts of time and money as they grow.
Knowing when is the right time to think about architecture is a skill; dogmatic "never do it" or "always do it" helps nobody.
I'd argue you should definitely consider multi-cloud strategy from the get-go indeed in 2022. Something like Terraform helps statically setting k8s clusters on most clouds. Especially for startups, it's better to default to vanilla stuff and only complicate on a need-to basis.
A multi-cloud paradigm is also a great way to teach you about your application and about those clouds themselves. A good reminder that "implementation is where it's at", and "the devil is in the details".
No one is advocating for a single server running in your closet, but a large and well funded PaaS can handle any realistic amount of growth at least temporarily, and something like Heroku is big enough (and more importantly, owned by a company big enough) that shutting down without notice is not a possibility worth considering.
This take brushes right past the causes of success and failure. Early stage success depends on relentless focus on the right things. There will be 1000 things you could do for every 1 that you should do. Early on this is going to tend to be product-market fit stuff. If things are going very well then scalability could become a concern, but it would be a huge red flag for me as an investor if an early stage company was focusing on multi-cloud.
Kubernetes is basically an industry standard at this point. It's easy to hire ops people competent in it, and if you do hire competent people, it will save you time and money even while you are small. As an investor "we use this product for orchestration rather than trying to roll our own solutions to the same problems, so that we can focus on $PRODUCT rather than reinventing half-baked solutions to mundane ops problems" should be music to your ears.
As many problems as containerization solves, it's not without significant overhead. Personally I'm not convinced the value is there unless you have multiple services which might not be the case for a long time. You can get huge mileage out of RDS + ELB/EC2 using a thinner tooling stack like Terraform + Ansible.
If you have such a team, it can of course make sense to delay or avoid containerisation if you don't see obvious major technical benefits.
But those teams will get rarer as time goes on, and since we're talking about startups, honestly it would be questionable to build a new ops team from people with no containers knowledge in 2022.
Yes, if your application looks like a web application that is cache friendly, k8s can really take you a long way.
The "web application" / cache-friendly part of your comment doesn't make much sense to me; k8s is pretty well agnostic to those kinds of details. You can orchestrate a database-bound system just as well as you can anything else, of course.
I do consider this impressive. And, to be clear, I wouldn't say this is because of a "super-developer". In fact, he had no prior k8s experience. But rather that there are thousand upon thousand of infrastructure hours devoted to the helm charts, often maintained by the people who develop the services themselves. It is almost mind boggling how much you get for almost free. Usually with very good and sensible default configurations.
In my precious work place, we had a team of 5 good engineers purely devoted to infrastructure, and I honestly believe that all five would be able to spend their time doing much more valuable things, if k8s had existed.
As for whether or not such devops solutions could be done in a day. Hm. I don't know. These things should be tailored to the problem. If you've done all of this a few times, then maybe you can adjust a bunch of charts that you are already familiar with and do what took a couple of months and impressed me, in a couple of weeks. It's a lot more than just "helm install. Done", that goes into architecting a scalable solution. Implementing monitoring, alerting and logging. Load testing stuff. Etc.
That does remind me that when that employee started, the existing "simple operational tech" was in fact to SSH into a VM and kill the process, git pull the latest changes, and start the service.
The only way you can solve the actual problem (not a simplified one) would in my opinion either be k8s or terraform of some kind. The latter would mostly define the resources in the cloud provider system, most of which would map to k8s resources anyways. So, I honestly just consider k8s to better solve what terraform was made for.
I'm sure the "simpler operational tech" meets few requirements for short disaster recovery. Unless you have infrastructure as code, I don't think that is possible.
Only if "scaling" is the problem that your startup is solving.
and ... you might be justified in that concern. However... after having been in the web space for 25+ years, it's surprising to me how many people have this as a primary concern ("we gotta scale!") while simultaneously never coming close to having this concern be justified.
I'm not saying it should be an either/or situation, but... I've lost count of how many "can it scale?" discussions I've had where "is it tested?" and "does it work?" almost never cross anyone's lips. One might say "it's assumed it's tested" or "that's a baseline requirement" but there's rarely verification of the tests, nor any effort put in to maintaining the tests as the system evolves.
EDIT: so... when I hear/read "scaling is a main concern" my spidey-sense tingles a bit. It may not be wrong, but it's often not the right questions to be focused on during many of the conversations I have.
Also, discussions about rewrites to scale up service capacity, but nobody has actually load tested the current solution to know what it can do.
It's a trap many of us fall into - I've done it myself. But next time I'll chuck money at the problem, using whatever services I can buy to get to market as fast as possible to test the idea. Only when it's proven will I go back and rebuild a better product. I'll either run a monolith or 1-2 services on VPSs, or something like Google cloud run or the AWS equivalent.
Scaling something no one wants is pointless.
yes by simple I mean covering high availability requirements, continuous deployment, good DORA measures - not simple as in half-baked non-functional operations (such as manually sshing to a server to deploy)
In our case, the workers were both quite heavy in size (around 1 GB), and heavy in number crunching. For this reason alone (and there are plenty more), lambdas would be a poor fit. If you start hacking them to keep them alive because of long cold starts, you would lose me at the simple part.
this is a few years out of date of platform capability, just fyi
And scaling is the same that VPS I could just scale the same way. Run a resize in my hosting company panel. (I dont use autorescal atm)
Only if I would hit about 100x times the nrs I would get the advantage of k8s, but even then I could just split up customers into different VPS.
CI / CD can be done good and bad with both.
And in practice K8S's a lot less stable. Maybe because I'm less experienced with K8S. But also because I think its more complex.
To be honest k8s is one of those dev tools that has to reinvent every concept again, so it has it's own jargon. And then there are these ever changing tools on top of it. It reminds me of JS a few years ago.
Premature optimization is a top problem in startup engineering. You have no idea what your startup will scale to.
If you have 1,000 users today and 5 year goal of 2,000,000 users, then spending a year building infrastructure that can scale to 100,000,000 is an atrociously terrible idea. A good principal can setup a working git hook, circleci integration, etc capable of automated integration testing and rather close to ci/cd in about a weekend. Like you can go from an empty repo to serving a web app as a startup in a matter of days. A whole year is just wasteful insanity for a startup.
The reality for start-ups running on investor money with very specific plans full of OKRs and sales targets is very different: you need to be building product as fast as possible and not giving any fuck about scale. Your business may pivot 5 times before you get to a million users. Your product may be completely different and green-fielded two times before you hit a million users.
I can't imagine any investor being ok with wasting a quarter of a million+ and a year+ on a principal engineer derping around with k8s while the product stagnated and sales had nothing to drive business -- about as useful as burning money in a pit.
You hire that person in the scale-up phase during like the third greenfield to take you from the poorly-performing 2,000,000 user 'grew-out-of-it' stack to that 100,000,000+ stack, and at that point, you are probably hiring a talented devops team and they do it MUCH faster than a year
Why are so many micro-services needed? Could the software be deployed in a more concise manner?
Not getting into the whole monolith-vs-services arguments. In both cases, complexity of deployment is part of the cost of each option.
I am not privy to the details of the case, but a rule-of-thumb I heard once is that if it's far enough from your core, a SaaS can be used (obviating the whole question), and if it's part of the core, start by developing it as a separate functionality before moving it to another service.
So far, we're up at 6 services, yet still at almost zero developer overhead cost. Then add the SaaS stack for each environment (api, worker, redis) and you're up at 15.
Not because I'm amazing, but there's a frankly ridiculous amount of information out there, and good chunks of it are high quality too. I think I started the job early January, and by April I had CI/CD, K8s for backend/frontend/DBs, Nginx (server and k8s cluster), auto-renewing certs, Sentry monitoring, Slack alerts for ops issues, K8s node rollback on failures, etc.
The best way to learn, is to do. Cliche, but that's what it really comes down to. There's a fair few new concepts to grasp, and you probably have picked some of these up almost by osmosis. It sounds more overwhelming than it is, truly.
- initial nodes setup: networking configuration (private and public network), sshd setup (disallow password login), setting up docker, prepping an NFS share accessible on every nodes via private network
- install RKE and deploy the cluster, deploy nginx ingress controller
- (optional) install rancher to get the rest of the goodies (graphana, istio, etc). These ate a lot of resources though, so I usually don't do this for small clusters
Done in a single afternoon.
My otherhot take is to not use helm but rather something like jsonnet or even cue to generate your yaml. My preference is jsonnet because you can very easily make a nice OO interface for the yaml schemas with it. Helm's approach to templating makes for a bit of a mess to try and read, and the values.yml files _really_ leak the details.
These often get developed side by side. GitLab, unit tests, api-server, nginx, cert-manager, deployments, integration tests, prometheus, metrics in services, grafana, alert-manager, log consolidation, work services and scaling, etc.
Just spinning up a cluster, nodepool, nginx, cert-manager w/let's encrypt cluster issuer, prometheus, grafana, can easily be done in a day. So, time estimates kinda depend entirely on what you mean by it.
Spinning up promerheus and grafana with automatic service discovery: one day. Making good metrics, visualizations, and alerts: everything from a week to a month or two. So, take the time estimates with a grain salt.
Also we bought us immediately an easy approach to build gitlab ci/cd pipelines + different environments (dev, staging, production) on the same cluster. Took me a week to set everything up completely and saved our team developing rapidly features really a lot of time and headache since then. But the point is, I knew how to do it, focus on the essentials and deliver quick reasonable results with large leverage down the route.
Bad culture alert! No one needs Elastic "as quickly as possible" unless their business, or the business they work for, is being very poorly run.
I would also argue that you might have got it running quickly, but how are you patching it? Maintaining it? Securing it? Backing it up? Have you got a full D/R plan in place? Can you bring it back to life if I delete everything within 3-6 hours? Doubt it.
> maintaining nodes with some clunky shell scripts.
Puppet Bolt, Ansible, Chef, ...
There are so many tools that are easy to understand that solve this issue.
Setting it up is easy, but not operations.
Ive been managing multiple clusters on AWS and on Azure.
I would take a managed EKS/OS in a heartbeat if I was a startup.
Elastic usecase is for crawling crazy amount of data and make it searchable and aggregatable and historically available, don’t know any other solution than elastic who has reasonable response times and easy-to-use access (plus we can add some application logging and APM).
> Puppet Bolt, Ansible, Chef, ...
Helm chart values.yaml and you’re all set, security + easy version bump included.
But that's the thing though: they didn't do it alone. You literally pointed out that this wasn't true almost immediately: "leverage the helm chart open source community". They used the work of others to get to the result.
Also, I highly doubt they could debug something if it went wrong.
I simply cannot believe anyone would advocate, or believe, that because a Helm chart makes it simple to create a complex piece of infrastructure it must also be true that maintaining it is also simple. It's really.
Once you understand the concepts it's not hard to debug. It's fair to acknowledge that kubernetes is complex, but also we should not ignore the real work that has been done in the past few years to make this stuff more accessible to the developers that want to learn it.
Also, saying it's not "alone" in this example I think is not fair. What would you count as "alone"? Setting up the kubernetes cluster from scratch and writing your own helm charts? Using that same logic, I can't say that because someone else designed and built the hardware it's running on. I think it's fair to say that if someone, independent of coaching, regardless of the underlying infrastructure produced some production grade infrastructure by themselves, they certainly did it alone.
As for debugging. You do have a point that it becomes more difficult. But, this also holds true for any of the alternatives discussed her (lambdas, terraform). I'd argue that when it all comes down to it, that because you can spin up the entire infrastructure locally on things like minikube, that it makes it many times more easy to debug than other cloud-provider-only solutions.
Should everyone go back to assembly on bare metal, then?
I think you mean silicon though.
> I found that to be quite impressive, for one person, one year, and would probably be completely impossible if it wasn't for k8s.
I've always found this interesting about web based development. I have no idea what Prometheus, grana, etc do. Ive never used k8s.
And yet, as a solo dev, I've written auto-scaling architecture using, for example, the AWS ec2 apis that let you launch configure and shutdown instances. I don't know what else you need.
Really the only advantage I see to morass of services is you get a common language so other devs can have a slightly easier time of picking up where someone left off. As long as they know all the latest bs.
This is fine, if you’re on AWS and can use AWS APIs. If you’re not (especially if you’re on bare metal), something like K8s can be nice.
In short. Prometheus is a worker that knows about all your other services. Each service of interest can expose an endpoint that prometheus scrapes periodically. So the services just say what the current state is, and prometheus, since it keeps asking, knows and stores what happens over time. Grafana is a web service that uses prometheus as a data source and can visualize it very nicely.
Prometheus also comes with an Alert Manager, where you can set up rules for when to trigger an alert, that can end up as an email or slack integration.
They are all very useful, and gives a much needed insight into how things are going.
Users don't really care if you have a really impressive stack with cool technologies if it doesn't offer anything more than a couple of web servers and a DB server.
Personally I would take managing "a couple of webservers and a db" any day over k8s.
You can find plenty of auto mechanics though. Cars are cheaper and ubiquitous. Maybe they can't drive to the moon, but they can get most things done.
Unless your business is flying to the moon, stick to cars and trucks over spaceships.
New lead came in, got rid of that crap by using 3rd party services to spin up infrastructure. Got a load balancer, a few VMs + DB. Reduced the cost down by 85% and greatly simplified the entire stack.
I learned a really valuable lesson without having to make that mistake myself.
I understand why people get excited about tooling. It's cool to learn new things and automate stuff away. I'm prone to that myself and do this on my own server when I get that itch.
Having said that, it's wrong to foist this stuff onto an unsuspecting company where the owners don't know any better about tech, that's why they hire other people to do that for them and seeing that just left a bad taste in my mouth for overcomplicated setups.
I get that SV is different, that's why tools like K8 are made and I would jump on those tools on a heartbeat as needed.
But for other smaller businesses, the truth is they just need a boring monolothic load balanced app with a few VMs and a DB sprinkled with 3rd party services for logging or searching or other stuff not core to the business.
> They used Ansible scripts to spin up various AWS services
This seems less about using the "cool/new" tech... rather it's about using the "right" tech. Config management tools like Ansible/Chef/Puppet are very much previous-generation when it comes to cloud infrastructure.
They... can manage cloud infrastructure, but they were created prior to the ubiquity of cloud deployments, and the features are glued on. Not choosing a more modern IaC framework tells me they(those devs) were going to be making sub-optimal implementation decisions regardless.
I'm sure there are plenty of use cases where that makes sense, and is a better approach. But, I disagree that k8s suggests a NIH-mindset.
I will always recommend using whatever container / function as a service e.g. ECS, GCF, Lambda any day over K8s for a startup. With these services its back to more similar models of security such as networking rules, dependency scanning, authorization and access...
Realistically, on all flash, you are very unlikely to need to maintain anything on a server for a few years after deployment.
If and only if your business model depends on it. A startup's job is mostly to find product market fit; if being decoupled from AWS isn't part of your market, you are spending money on a non-problem.
The reality is unless you’re some rich dude who can borrow dad’s datacenter (And that’s cool if so), you’re either going to be renting colo space, virtual servers, etc.
It’s always a challenge in business to avoid the trap of spending dollars to save pennies.
IMO, you’re better off working in AWS/GCP/Azure and engineering around the strengths of those platforms. That’s all about team and engineering discipline. I’m not in the startup world, but I’ve seen people lift and shift on-prem architecture and business process to cloud and set money on fire. Likewise, I’ve seen systems that reduced 5 year TCO by 80% by building to the platform strengths.
I'm aware that no man is an island in some sense, but I'm not comfortable with locking myself into one of 3 companies who need to increase their revenue by double digits year over year. And as you say, a lift and shift is basically setting money on fire. Currently I run sort of a hybrid approach with a small IaaS provider and a colo. It seems to work well for us both technically and financially though that seems to go contrary to what is considered conventional wisdom these days.
Where I work, we can deliver most services cheaper on-prem due to our relative scale and cloud margins. But… we’re finding that vendors in the hardware space struggle to meet their SLAs. (Look at HPE — they literally sold off their field services teams and only have 1-2 engineers covering huge geographic regions. So increasingly critical workloads make the most sense in the cloud.
Google is not evil. They're just big, and define some practices which we might think should change in the future.
Once you have the thing up and running, you can start to think about hosting your own.
Also, you don't need to use fancy services because most startups can run just fine on a single instance of whatever, meaning, there are a lot of cloud providers out there.
As a rule, for anything in an startup, adding that "scalable" adjective is a waste.
Of course, exceptions exist, but it's one of those problems that if you have them, you probably know it, and if you have them and don't know it, you won't succeed anyway. So any useful advice has the form of "don't".
So if you already know Kubernetes, then great, use it. Leverage your existing skills. If you don't, then just use Heroku or fly.io or whatever, or go with AWS if that's your competence. Maybe revisit in a year or two and maybe then you'll have funding to hire a devops person or time to spend a week or two learning to do it yourself. Right now you want to get your SAAS MVP in front of customers and focus on the inevitable product churn of building something people want to pay for. The same advice goes for anything else in your stack. Do you know it well enough? Do you need it right now? Or is there a "good enough" alternative you can use instead?
Taking human resource information into consideration sounds very wise. Although, learning a new language is generally not that a huge barrier, while changing your whole stack once the minimum viable product cap is passed can be very expensive. And if you need to scale the team, the available developer pool is not the same depending on which technology you have to stick with.
It doesn’t invalidate your point, but maybe it brings some relevant nuances.
Kubernetes is not 'harder' if all you need is to host a webapp. Where it falls on the hardness spectrum depends on what you are trying to do, and what is the alternative. I am very fluent with Kubernetes but have no skills in managing traditional virtual machines.
And you're also forgetting Android and macOS and Linux.
That's why cross-platform frameworks like Electron and React Native are so popular. The time wasted in going native for every single platform is just infeasible for most non-huge companies.
Here is an example of a team that is doing great work in mobile, frontend and backend:
3 people doing native iOS, 2 people doing native Android, 3 backend engineers, 1 frontend and 1 QA.
Two engineering managers/team leaders: one for mobile and one for web.
Of course this is one single product offering native mobile apps and some limited web app functionalities.
The apps are great, smooth, nice UX, works fast, offers native experience.
Is this a huge company? I don't think so.
Rather than the author "not knowing" what they're talking about, I suspect they're taking narrow experience and generalising it to the entire industry. Their background is selling k8s as a solution to small/medium enterprises: it strikes me that there may be a strong correlation between startups interested in that offering and those deploying failed overengineered multilang micro-architectures. Suspect the author has seen their fair share of bad multilang stacks and not a lot of counter examples.
The sole exception to this rule I would make is javascript which is more or less required for frontend stuff and should be avoided like the plague for any other development. As soon as you can get your frontend done in Rust, though, you should also switch.
Of course there's some balance needed. If your team is familiar with some niche language then long term that might not be a good strategy if you intend to bring more devs on board later.
One side of this which I don't think is discussed often is the fun of choosing new technology. How do you balance having fun and being realistic at the same time?
Fun meaning trying new technology, learning as you go, setting up systems that make you feel proud, etc. It can lead to failure, but I think having fun is important too.
Being charitable, what I think they are getting at is maybe more about having fully separated frontend and backend applications (since the front-end examples he gives are not languages but frameworks / libraries). Otherwise it seems really backwards - I'm definitely an advocate of not always needing SPA-type libraries, but using literally zero Javascript unless your backend is also JS seems like it goes to a too-far extreme.
Er.
I read this as him saying "one of the things I've seen as a bad reason to use Kubernetes is that there are multiple languages in use."
I've seen people do this. Frontend in one container, backend in another, Kube to manage it.
If that's what author meant, author is right, that's a profoundly stupid (and common) reason to involve Kube.
Kubernetes is a perfectly good platform for any size operation, but until you are a large org, just use a managed service from Google/Amazon/DigitalOcean/whoever. Kubernetes, the data plane, is really no more complex that eg Docker Compose, and with managed services, the control plane won't bother you.
K8s allows composability of apps/services/authentication/monitoring/logging/etc in a standardised way, much more so than any roll-your-own or 3rd-party alternative IMO, the OSS ecosystem around it is large and getting larger, and the "StackOverflowability" is strong too (ie you can look up the answers to most questions easily).
So, TLDR, just use a managed K8s until you properly need your own cluster.
My k8s "cluster" is a single node that doesn't cost me any more than a VM would. I don't need the scalability at the moment. But I do need different environments for dev, qa, and prod. And all three are running identically next to each other using Namespaces. Saved us a ton of maintenance and cost.
Any project that grows has its needs change. GKE gives you a ton of integrated tools right from the start including logging, alerting, metrics, easy access to hosted databases, pub/sub, object storage, easy and automatic network setup, easy firewall setup, dns management, and a lot more. k8s is no different than using any other hosted service. It provides a great set of features that you configure using fairly consistent yaml configuration files. And it is all accessible from the web based "Google Console" as well.
Learning the k8s yaml format and some basic kubectl commands is all you need to get going and it saves a TON of time that can go back into developing your application rather than dealing with configuring disparate pieces with their own configuration methods.
I was fairly early to k8s while they were still competing with other similar solutions and other tools like Puppet and Chef. I tested all of them and truthfully, k8s was the easiest to learn, implement, and maintain my app with. Using GKE of course. I would NEVER as a one man or even small team of developers take on managing an installation of k8s myself.
This was my first thought, and I was to comment so, but saw you already did. The only reason we see this comment is because HN has an irrational hate of K8s, for us that do run things in production at scale, k8s is the best option. The rest is either wrapped in licenses or lacks basic functionality.