>The CEO now seeks help from Phutar Afrayughum, a psychic and extrasensory perception specialist who allegedly helped Google increase their marketshare in the messaging app market, and was also involved in developing the Material Design framework.
Denpok? I swear it reads like a skit from the Silicon Valley satire TV show. I thought Mike Judge was exaggerating but now I see he's just imitating the SV reality.
I think execs and managers at Google are just high on their own farts at this point.
I was also thinking of the show.. it was a great one.. need to rewatch in a few months. Also I always remember the Hotdog classification app when some CEO says AI these days :)
Would be 11-months where they would be now "cloud native" or something
Of course running their DB was a bit challenging as they forgot to set proper K8S storage configs and saw their data disappear after their pod was suddenly migrated
"Should" is perhaps disguising the fact that these projects normally do take a lot longer than 11 weeks. I know of multiple tech companies that are years into an unfinished k8s migration. Everyone on the project will have a ton of good reasons why that project was uniquely challenged and therefore took so long.
Either the entire IT sector is inept in general or your bar is way too high. I've not seen a single K8s migration project in an actual, business running company going in under 11 weeks
> Pretty lame. If you already use docker, it shouldn't take nearly that long.
Sounds like someone has only worked with trivial, low-complexity systems and limited themselves to operating within their own capabilities.
Where's the 40 year old IBM mainframe the business depends on, and which limits customers to 8-character alphanumeric passwords? Where's the critical business system which won't run on anything higher than Java 7 and which nobody maintains? Where's the corporate policy insisting such obsolete software isn't allowed on the new system, but providing no budget or headcount to upgrade it? Where's the software that needs to connect to external services, but the team doesn't know all the hostnames or IP addresses? Where's the guy blocking DHCP and DNS because they weren't on the list of approved external services? Where's the crucial software with its license locked to a server with a certain MAC address and CPU ID?
Do you even have a sworn statement from the creators of Kubernetes that no slave labour was used? No, I thought not. Unlike you, some of us are opposed to slavery.
Google released GCP 15 years ago and they still haven't moved their internal services to use it. That proves they're web-scale and you're not.
One thing I like about AI is that at least we don't hear about Kubernetes and blockchain that much anymore. AI has now its NFT moment so hopefully will be over soon.
I hear paperclips are new great thing to follow. Have you considered creating paperclips? Just order some metal wire then Cut an 8-10″ piece. Use pliers to bend in desired shape.
For my experience, this article is one of the scenes in my personal version of This is Spinal Tap.
Three times I’ve been involved in a Kubernetes (or competitor) migration where the plot gets lost along the way. At my current company we have canceled the product roadmap to focus on year three of the migration and we likely have at least another year to go.
No one can tell me why our current ECS based deployments are a problem. No one bothered to train people on appropriate uses of Kafka before double abstracting it into a framework.
But hey, the board was promised a cost savings and globally available product from this effort so at least the executive that promised it all has two more years to figure out how to make good on it.
Best of all is we have a new CTO and he has openly questioned why we don’t just use Kubernetes itself instead of the couple of Hashicorp products we’ve settled on. Maybe I’ll get to do this a fourth time once we finish the current migration.
It sounds to me like you're an engineer who is out of the loop.
Because suggesting Hashicorp over Kubernetes makes a lot of sense so long as you are ignorant about budgets and costs. Because from experience they are one of the worst vendors around at squeezing money out of their customers.
I imagine if you were the CTO you would likely agree with them.
The problem is not about the vendor at all. It’s not even about Kubernetes. It’s about the problem being solved. I’m not an IC so I get to see all the behind the scenes.
We’re just cosplaying a large tech org when this company, like the others I’ve been at, is a sales company with an IT team. There’s nothing inherently wrong with that. It’s a good business to be in. We just don’t need a complex runtime with software defined networking to deploy a series of three server blue/green deployments for relatively low traffic use cases.
The problem itself has been the same for the past three companies I’ve worked at.
I think it's a bit like a rorschach test, maybe. I would attribute the 11-week time period to a dysfunctional organization, but someone who doesn't like K8S and prefer to run everything themselves may interpret it as implying K8S is overcomplicated.
In reality, though, a migration to K8S in most organizations won't take only 11 weeks because most organizations are completely dysfunctional.
I know it’s a joke but if you did a post-mortem the reason for failure would probably be:
> Many at the company thought it's a good chance to upgrade software dependencies and libraries as well. A large part of a PostgreSQL database running on a single machine could as well be transformed into a distributed KV storage, leveraging the tremendous power of flexibility of AWS.
Stick to the company’s scope too. If your business isn’t selling cloud infrastructure, use an off the shelf cloud provider. Especially don’t use k8s if you’re already paying for a cloud provider.
1) Be clear about your requirements, understand your target platform and take the time to estimate costs. Blindly picking an off the shelf cloud provider can ruin you if you suddenly get hit with unexpected costs e.g. ingress/egress.
2) Kubernetes is often the sensible way to go so long as you're using a managed platform e.g EKS. As it allows you to build out your platform in a scalable, highly-available and geo-distributed way without needlessly tying yourself to any one cloud.
Because aaaany day now, AWS is going to turn around and jack up prices on EC2 instances because they can. Humans are terrible at risk analysis. There are good reasons to arbitrage between clouds, but business continuity isn't one of them.
Yes, you read my post correctly - using an off the shelf cloud provider for their managed Kubernetes service counts as using an off the shelf cloud provider. But besides EKS, also consider ECS, EC2, etc. and whatever Microsoft has going.
Oh god the mid level infrastructure engineers are on HN now.
No, running your own Kubernetes installation is absolutely not often the sensible way to go.
Yes you should measure costs, well done, but be realistic and include at additional annual salaries for the people whose full time jobs are now managing Kubernetes and productivity loss due to failures of your unnecessary installation of massively complex software.
If you haven’t noticed from posts like this, Kubernetes is an industry meme for wasted engineering effort and has a reputation of being something added to companies by young engineers practicing resume-driven development.
Why not make your own monitors? Computer displays have enormous markups and monitor providers could increase prices at any moment.
Finally: EKS also falls under the definition of using an off the shelf cloud provider as the other commenter has already pointed out.
Well, that's kinda the point of the point of the joke. There are many people out there more focused on the technology they are using rather than being focused the product they are making. Being focused on the product implies that yes you know your scope, and don't over-engineer it too soon.
The joke concentrates on Kubernetes (if I wanted to be kind, it's probably because a joke is better with an actual examples, but I actually think it's because the author is in that engineering world, and doesn't know the other places where the same applies), but you could do the same with server-side rendering, with AI, with $modernFrontendLib, $modernLanguage
I have been fighting day and night for four months to move 500000 blobs to managed blob storage from self-hosted MinIO, and of that actual productive work that was not about politics or bureaucracy is less than 1 week, so an 11-week migration to K8S sounds like a great success.
also, their homepage does not work well. i can read all the articles but i can't dismiss their "gimme your email" homepage popup. So i guess i'm not adding this site to my list of blogs i read often.
It looks like you haven't read the text on the link:
"The Board of Directors plans to hire several Agile Analyticistis, Working Culture Ambassador Researchers, Import-Export-Integrator-Optimizer Masters and Kanban Disruptors in order to help them investigate the unexpected outcome."
At $dayjob we are in the middle of such a migration, started 2 years ago and still less than 30% complete.
Now, the guys that were most vocals about "we must go Kubernetes, kill the Monolith" etc are already playing with LLMs, and forgot about Kubernetes.
Some people just love proves of concept and the shiny next thing. I guess that role is useful, in its own way.
You could say most people can run if they master walking first.
The problem is that companies take a Java 5 monolith without any CI/CD and want to take rewriting AND re-platforming AND business changes all at once in a giant project. Then they pick the technology that is anyone might be blaming and run with that excuse so it's nobody's fault.
PS. What you describe seems to be not even about PoC (the rule about PoC is that it should work at all), but about following the herd and inventing a setup to appear busy. But that'd be also not very good environment to stay at, as an employee..
It's why many brilliant minds appear to be quite content working for unethical tech giants, advertising or surveillance companies.
The reason why the company exists doesn't matter to them.
What the company actually does outside of their computer doesn't matter to them.
The technology and their freedom to pursue the New does. Companies love the production and enthusiasm they provide and pay well for it.
(Usually these types of developers do have a conscience and it's often taken up, used and displayed in worthwhile and benevolent corporate friendly style social activism)
What you say is true, and I read it as a positive not a negative. The "unethical" part is subjective so the manipulation doesn't get to me.
I see this as positive because computer nerds (I'm one) get to be paid very well to figure out the new and shiny. It's what I'd be doing at home anyway.
Most of the time nothing of value is created (Except my amusement), but every once in a while something brilliant is, and due to scale, it can have big impact.
Apparently you're not supposed to make comparisons to Nazi Germany (Godwin's law if you want to be too-online about it); it's considered gauche I suppose. What's funny is that that strikes me as a really good rule to follow if you do in fact want to sleepwalk your society into an analogue of Nazi Germany.
I wouldn’t say this is scope creep. This is intentional resume driven development. Someone is ticking through checkboxes so they can say they’ve done X. I can say with experience that on small teams this can really grind productivity to halt really quickly. It’s often masked by a desire to solve all the problems. Well the problem with that is that no problems are solved, although certainly more are created.
In FAANG adjacent companies it is very difficult to get promos. And promos are the only way to get increase in pay due to leveling.
Promos require promo packs, and promo packs require large beefy projects. (Good luck jack of all trades).
Thus, you end up with these situations where large beefy projects are solutions looking for a problem as the core problem being solved is a promo, not a business need.
0. Learn technologies first, if they are complex. Try on small not significant services first.
1. Do one thing at a time.
2. Start simple.
I migrated our services to Kubernetes without any issues. But it took two years of learning and trying, migrating small services. I tried several approaches before coming to the most suitable and that’s not something you would find in the Internet. I’m using gitops without automation, I just kubectl apply -k what I need, because I decided back at a time, that flux was unnecessarily complex to start with. Now I have few dozens of services and good understanding of things, I’m thinking about introducing flux.
I could replace "Kubernetes" with GraphQL/React/Next in my world.
All, of course, to migrate a perfectly functioning app (mostly CRUD) and doesn't need any of the tradeoffs given by GraphQL or an interactive frontend.
The longer I am in this industry the more I see people in charge have no idea of what they're doing.
People are never rewarded for keeping things running. They are rewarded for change, as long as they can at least pretend it is (or will be) paying off.
Definitely agree. But it's not only about keeping things running though. People want to fix issues by rewriting everything because it's gonna look good on their resumé and it's "easier". You don't have to deal with the old stuff.
It saddens me because improving code (especially one that changes often) is absolutely crucial, but we seem to never bite the bullet and improve the existing code, instead opting to rewrite the entire world in never-ending projects.
After a company gets burned by one of these rewrites, they often become averse to any refactoring and the few legitimate rewrites that need to happen.
We spend all our influence capital on useless projects and then wonder why our industry is so often viewed as unprofessional and amateurish.
> It saddens me because improving code (especially one that changes often) is absolutely crucial
It has burned me and it has burned others. Just doing cosmetic refactorings alone me and others I have observed caused new bugs. Let alone doing architectural or other "deeper" refactorings.
At this point I pretty much refuse touching anything not related to my ticket, everything else is a risk to my reputation.
When I see junior developers take on unrelated refactorings, I let them fly too close to the sun, they will learn through the pain.
Last thing I did a few years ago was changing a weird boolean field that could be null. I thought it would be smarter to default it to false, client team told me it was fine. We then found out in production that very old clients actually had different logic based on true, false AND null. I got pinged in my free time, it ruined the date I was on and thus at least the whole day.
In a perfect world you have CDC and all the testing in the world, but in reality you just don‘t.
Yeah, I've been there too. But the problem is that avoiding those completely and then only resorting to rewrites, costs magnitudes more and it's just as buggy. Many times it introduces new issues.
Doing small incremental improvements is, most of the time, the best way to improve. But because in general there's no incentive to have even a base level of quality in software, the incentive is towards rewriting and dealing with all the issues after the fact.
Our devops/prod team is constantly migrating, upgrading and changing stuff.
At this point I am convinced devops is a job security scam, all the way up to the director in charge, with many of the engineers in between actually believing they are doing something useful and happy to be learning $in_demand_skill.
Just like "cloud" and "cloud native" altogether. Costs more, requires more engineers to be kept running and is just as incomprehensible, customized and brittle as good old sh scripts on bare metal.
You can airdrop a spring developer from 15 years ago into any modern Java shop using spring boot with instant startup thanks to aoc and all he has to do is to learn the basics of docker, some nosql crap and aws services like SNS/SQS, which should be easily possible within a week and some help from ChatGPT.
Backend stuff (especially java/enterprisy stuff) moves slower. Frontend stuff however... Drop a jquery dev from 15 years ago into modern react/typescript shop and watch their blood pressure steadily rise every day.
This reminds me of real job post, I saw once for a Startup. No details on the actual product or company purpose. They were looking for a Kubernetes administrator. The main requirement, besides experience with Linux, was to write a long technical report on why Kubernetes was the right solution.
And yet, engineers would rather deliver a pizza by organizing an expedition team, climbing Mount Everest, taking a picture of the pizza at the summit, then fly the pizza back home, rent a Lamborghini and drive the Mongolian rally race, then 18 months later deliver said pizza.
While they do that, just get on a cheap scooter and drive it down the street. Win.
I have. There are multiple types of engineer which insist on migrating "legacy" (stable) systems to "state of the art" (cooler, modern, en vogue) tech.
You'd be surprised how many companies that maintained <20 vms migrated to k8s. The motivations include ordering a chaotic status quo, premature optimisation, young blood wanting to prove themselves, boredom, "its new so it must be better", "i want to try this" [...], you get the idea.
I hear loud-mouthed people at work talking about introducing "durable execution" when there's so many other fish to fry and the company can't achieve those well already.
111 comments
[ 2.9 ms ] story [ 190 ms ] threadAlmost lost my coffee.
I think execs and managers at Google are just high on their own farts at this point.
This is nonsense! Such migration could not be even "seemingly" completed in 11 weeks, rather 11 years. /s
Of course running their DB was a bit challenging as they forgot to set proper K8S storage configs and saw their data disappear after their pod was suddenly migrated
If you already use docker, it shouldn't take nearly that long. If you don't, then kube isn't the problem for any migration similar
Hackernews however is a frounder, frontend scene first site. So I'd imagine that is what normal feels like to most of you.
Sounds like someone has only worked with trivial, low-complexity systems and limited themselves to operating within their own capabilities.
Where's the 40 year old IBM mainframe the business depends on, and which limits customers to 8-character alphanumeric passwords? Where's the critical business system which won't run on anything higher than Java 7 and which nobody maintains? Where's the corporate policy insisting such obsolete software isn't allowed on the new system, but providing no budget or headcount to upgrade it? Where's the software that needs to connect to external services, but the team doesn't know all the hostnames or IP addresses? Where's the guy blocking DHCP and DNS because they weren't on the list of approved external services? Where's the crucial software with its license locked to a server with a certain MAC address and CPU ID?
Do you even have a sworn statement from the creators of Kubernetes that no slave labour was used? No, I thought not. Unlike you, some of us are opposed to slavery.
Google released GCP 15 years ago and they still haven't moved their internal services to use it. That proves they're web-scale and you're not.
/s
Three times I’ve been involved in a Kubernetes (or competitor) migration where the plot gets lost along the way. At my current company we have canceled the product roadmap to focus on year three of the migration and we likely have at least another year to go.
No one can tell me why our current ECS based deployments are a problem. No one bothered to train people on appropriate uses of Kafka before double abstracting it into a framework.
But hey, the board was promised a cost savings and globally available product from this effort so at least the executive that promised it all has two more years to figure out how to make good on it.
Best of all is we have a new CTO and he has openly questioned why we don’t just use Kubernetes itself instead of the couple of Hashicorp products we’ve settled on. Maybe I’ll get to do this a fourth time once we finish the current migration.
This is the most surreal thing I've read on here. You went from a company that delivers some product, to a company that delivers a migration.
You sure your company is not the one in the OP? ;)
The problem is it's not Kubernetes.
I am in the exact same situation. New manager arrives and wants to move to Kubernetes, just because.
Because suggesting Hashicorp over Kubernetes makes a lot of sense so long as you are ignorant about budgets and costs. Because from experience they are one of the worst vendors around at squeezing money out of their customers.
I imagine if you were the CTO you would likely agree with them.
We’re just cosplaying a large tech org when this company, like the others I’ve been at, is a sales company with an IT team. There’s nothing inherently wrong with that. It’s a good business to be in. We just don’t need a complex runtime with software defined networking to deploy a series of three server blue/green deployments for relatively low traffic use cases.
The problem itself has been the same for the past three companies I’ve worked at.
Of course these companies exist and there is a core of truth in what the article says. Otherwise, it wouldn't be satire.
In reality, though, a migration to K8S in most organizations won't take only 11 weeks because most organizations are completely dysfunctional.
> Many at the company thought it's a good chance to upgrade software dependencies and libraries as well. A large part of a PostgreSQL database running on a single machine could as well be transformed into a distributed KV storage, leveraging the tremendous power of flexibility of AWS.
Stick to the scope.
1) Be clear about your requirements, understand your target platform and take the time to estimate costs. Blindly picking an off the shelf cloud provider can ruin you if you suddenly get hit with unexpected costs e.g. ingress/egress.
2) Kubernetes is often the sensible way to go so long as you're using a managed platform e.g EKS. As it allows you to build out your platform in a scalable, highly-available and geo-distributed way without needlessly tying yourself to any one cloud.
Because aaaany day now, AWS is going to turn around and jack up prices on EC2 instances because they can. Humans are terrible at risk analysis. There are good reasons to arbitrage between clouds, but business continuity isn't one of them.
EC2 instance prices are nothing compared to the many hidden costs e.g. ingress, VPC, NAT.
ROBERT ENGLUND - Freddy Krueger
Don't use (self-managed) k8s when you can use the managed k8s provided by your cloud provider.
Oh god the mid level infrastructure engineers are on HN now.
No, running your own Kubernetes installation is absolutely not often the sensible way to go.
Yes you should measure costs, well done, but be realistic and include at additional annual salaries for the people whose full time jobs are now managing Kubernetes and productivity loss due to failures of your unnecessary installation of massively complex software.
If you haven’t noticed from posts like this, Kubernetes is an industry meme for wasted engineering effort and has a reputation of being something added to companies by young engineers practicing resume-driven development.
Why not make your own monitors? Computer displays have enormous markups and monitor providers could increase prices at any moment.
Finally: EKS also falls under the definition of using an off the shelf cloud provider as the other commenter has already pointed out.
The joke concentrates on Kubernetes (if I wanted to be kind, it's probably because a joke is better with an actual examples, but I actually think it's because the author is in that engineering world, and doesn't know the other places where the same applies), but you could do the same with server-side rendering, with AI, with $modernFrontendLib, $modernLanguage
https://www.theolognion.com/p/dev-builds-perfect-note-taking...
Edit: oh, and how about this one:
https://www.theolognion.com/p/ai-solves-all-political-econom...
also, their homepage does not work well. i can read all the articles but i can't dismiss their "gimme your email" homepage popup. So i guess i'm not adding this site to my list of blogs i read often.
"Company accidentally increased dev productivity 3x by laying off 20% of middle management"
https://www.theolognion.com/p/company-accidentally-increased...
"The Board of Directors plans to hire several Agile Analyticistis, Working Culture Ambassador Researchers, Import-Export-Integrator-Optimizer Masters and Kanban Disruptors in order to help them investigate the unexpected outcome."
Some people just love proves of concept and the shiny next thing. I guess that role is useful, in its own way.
The problem is that companies take a Java 5 monolith without any CI/CD and want to take rewriting AND re-platforming AND business changes all at once in a giant project. Then they pick the technology that is anyone might be blaming and run with that excuse so it's nobody's fault.
PS. What you describe seems to be not even about PoC (the rule about PoC is that it should work at all), but about following the herd and inventing a setup to appear busy. But that'd be also not very good environment to stay at, as an employee..
It's why many brilliant minds appear to be quite content working for unethical tech giants, advertising or surveillance companies.
The reason why the company exists doesn't matter to them.
What the company actually does outside of their computer doesn't matter to them.
The technology and their freedom to pursue the New does. Companies love the production and enthusiasm they provide and pay well for it.
(Usually these types of developers do have a conscience and it's often taken up, used and displayed in worthwhile and benevolent corporate friendly style social activism)
I see this as positive because computer nerds (I'm one) get to be paid very well to figure out the new and shiny. It's what I'd be doing at home anyway.
Most of the time nothing of value is created (Except my amusement), but every once in a while something brilliant is, and due to scale, it can have big impact.
Good specialist engineers designed gas chambers in Nazi Germany
Promos require promo packs, and promo packs require large beefy projects. (Good luck jack of all trades).
Thus, you end up with these situations where large beefy projects are solutions looking for a problem as the core problem being solved is a promo, not a business need.
1. Do one thing at a time.
2. Start simple.
I migrated our services to Kubernetes without any issues. But it took two years of learning and trying, migrating small services. I tried several approaches before coming to the most suitable and that’s not something you would find in the Internet. I’m using gitops without automation, I just kubectl apply -k what I need, because I decided back at a time, that flux was unnecessarily complex to start with. Now I have few dozens of services and good understanding of things, I’m thinking about introducing flux.
All, of course, to migrate a perfectly functioning app (mostly CRUD) and doesn't need any of the tradeoffs given by GraphQL or an interactive frontend.
The longer I am in this industry the more I see people in charge have no idea of what they're doing.
It saddens me because improving code (especially one that changes often) is absolutely crucial, but we seem to never bite the bullet and improve the existing code, instead opting to rewrite the entire world in never-ending projects.
After a company gets burned by one of these rewrites, they often become averse to any refactoring and the few legitimate rewrites that need to happen.
We spend all our influence capital on useless projects and then wonder why our industry is so often viewed as unprofessional and amateurish.
It has burned me and it has burned others. Just doing cosmetic refactorings alone me and others I have observed caused new bugs. Let alone doing architectural or other "deeper" refactorings.
At this point I pretty much refuse touching anything not related to my ticket, everything else is a risk to my reputation.
When I see junior developers take on unrelated refactorings, I let them fly too close to the sun, they will learn through the pain.
Last thing I did a few years ago was changing a weird boolean field that could be null. I thought it would be smarter to default it to false, client team told me it was fine. We then found out in production that very old clients actually had different logic based on true, false AND null. I got pinged in my free time, it ruined the date I was on and thus at least the whole day.
In a perfect world you have CDC and all the testing in the world, but in reality you just don‘t.
Doing small incremental improvements is, most of the time, the best way to improve. But because in general there's no incentive to have even a base level of quality in software, the incentive is towards rewriting and dealing with all the issues after the fact.
It really makes me sad.
https://www.theolognion.com/p/dev-realizes-he-is-still-in-th...
At this point I am convinced devops is a job security scam, all the way up to the director in charge, with many of the engineers in between actually believing they are doing something useful and happy to be learning $in_demand_skill.
Just like "cloud" and "cloud native" altogether. Costs more, requires more engineers to be kept running and is just as incomprehensible, customized and brittle as good old sh scripts on bare metal.
I liked this one - Dev realizes he is still in the interview stage after 2 years of work [https://www.theolognion.com/p/dev-realizes-he-is-still-in-th...]
Especially the conclusion:
> Pawri's hope is that the offer will come before his retirement date.
And yet, engineers would rather deliver a pizza by organizing an expedition team, climbing Mount Everest, taking a picture of the pizza at the summit, then fly the pizza back home, rent a Lamborghini and drive the Mongolian rally race, then 18 months later deliver said pizza.
While they do that, just get on a cheap scooter and drive it down the street. Win.
You'd be surprised how many companies that maintained <20 vms migrated to k8s. The motivations include ordering a chaotic status quo, premature optimisation, young blood wanting to prove themselves, boredom, "its new so it must be better", "i want to try this" [...], you get the idea.