I remain unconvinced that most companies actually need systems 'at scale'. Just get a big relational database and know what parts of your workflow have concurrency issues. If you ever actually approach BIGCORP_NAME scale, you'll have more than enough money at that point to rebuild it into something else.
DevOps has never been about scaling.
It’s about shipping faster. How do we design systems and processes that allow us to go from idead to production very fast.
Everyone needs that.
However you want. I’m not saying no one need scale, I’m just saying that’s not what the DevOps movement is about. Just like agile also won’t help you managing your cluster.
I don't mean to generalize but every time I had to deal with a DevOps team shipping velocity was much slower compared to when I was able to handle infrastructure myself through a vendor like Heroku, Vercel or even plain AWS.
I find myself agreeing with the part about "scaling", meaning a (good) DevOps team makes it easier to integrate new projects into an existing infrastructure that's built in house. The point where relying on the vendors above becomes more expensive than having a full time DevOps team only happens once you react a certain scale.
Ops person(s) embedded within the Dev team dealing with/helping with/teaching about that one team's "Ops stuff" --> DevOps --> team gets faster because no Ops team hoops to jump through.
A separate DevOps team --> corporate rebrand of the separate Ops team --> no real change, or even slower because more hoops to jump through.
DevOps was always about meant to be about breaking down the separate / silo'd teams and combining them together. Ownership and responsibility is with that single team for their code from development through to production.
The corporate rebrand of Ops teams is what you're coming up against. It does my nut in and I'm sorry to that you have to deal with it.
And also get a hot spare for that big relational database. And take good snapshot backups and verify they work and verify you can access them when required. Maybe your internal IT /can/ do all of this, in production, without making you want to cry on a daily basis.
We didn't really need the scale. We just didn't want to deal with any of those problems. Often there's a little checkbox you can check that solves one or more of these problems, the cost of checking the box is dependent on the size of the data itself. This is convenient _and_ predictable. We like that.
> what matters at this point is Platform Engineering or Site Reliabiliy Engineering.
Show me the difference between these 3 pictures (the third being DevOps Engineering.) Go ahead, I'll wait.
They're all the same role, where people are all doing a mixed bag of things for companies that their developers decided they didn't want to do (or sometimes didn't have the experience to do.) I'm a Senior DevOps Engineer, and you could give me any one of the 3 titles, and they would all be correct characterizations of what I do.
It got weird when “Devops” became a job role and the tools were CI/CD pipelines built on config automation and cluster orchestration. That becomes the baseline for everything. No matter how overkill it is.
The vast majority of web software is comically over-engineered. The deployment process has so many features and components that you need a full-time staff of specialists to configure and monitor the build system.
And it could be something as straightforward as a marketplace website or medium complex like some kinds of SaaS. The web application could probably be compiled into a single binary or container and it’s database could run in RAM. And yet to ship a single change doesn’t take one developer but a whole team, eye watering enterprise contracts with several vendors, and it’s still slow as molasses.
There’s a time and a place for config as code and container orchestration. The problem is that it’s used for every single project regardless of requirements.
> The vast majority of web software is comically over-engineered.
Im not sure downloading libraries from NPM and adding services till it works is "engineering"
Im not even sure it is good or fast.
I am certain that it is expensive.
For the 2nd time today Im going to say it the only metric that matters is Cost per customer, cost per user, that includes the systems that includes the engineers to run it.
If you don't know these numbers what the hell kind of business are you running.
I’m not gate-keeping what is and isn’t engineering. Where I live it’s a protected term anyway. People get sued for calling themselves software engineers.
I’m using the Wikipedia definition of, “over engineering.” Solving a problem in an elaborate way when a sufficient and simpler solution can be demonstrated to exist. In many cases you could rsync a binary to your servers and let the monitor process reload your web server. Or you can build out a build farm, dependency cache, and… spend $2, as you say, where someone who is practicing engineering could do well (or better) with $1.
I'm genuinely not sure how other companies do this.... But how is your team handling SOX controls WITHOUT at least one DevOps person?
I work at a medium size publicly traded company and our SOX compliance controls would take literal months to generate and/or prove to auditors without our CI/CD pipelines. It's just an extract from GH Actions with a report of who modified, who approved, and who actually pushed to main. All of these actions must be siloed (if you can commit to repo, you cannot push to main)
Potentially this is a consequence of micro service infra, my team alone manages nearly 25 separate git repositories.
Yeah this stuff is hilarious and sad. I'm in the middle of setting up a kubernetes cluster for a relatively big company. K8s in their environment makes sense.
However, the first conversation (after finishing up the contract) with the sysadmins went like: "Yeah, ingresses are fine and such, but we'll have our loadbalancer in front of that which will inspect the url paths to know which ingress it has to go to, we don't want some 'developer' to open up some new endpoint without us approving it! Those guys never know what they're doing."
At this point I'm not sure if he's pulling my leg or something, b/c before the we wrote up the contract the whole idea was to give their dedicated platform team and developers more self service. In one fell swoop, bam, negated.
Devops imo is about clear responsibility, it just doesn't work if boundaries are not communicated explicitly. In the above scenario once everything is setup, developers will not gain any velocity, and the sysadmin are still manually adding url routing to their loadbalancers.
What people often miss when talking about how to organize work is that problems are often culturally grounded. The idea of solving a cultural issue with technical tooling is never going to work. It can be profitable selling such promises, but this approach fundamentally misses the mark.
Adopting devops, agile, scrum ritual, whatever, will never fix on a technical level something that is a cultural problem. It's a category error.
Well... For me (sysadmin) DevOps is a thing in theory, another in practice. The overall results of this dichotomy are tragic, they led to harmful things like doker, k*s and so on down to GNU/Linux desktops crap like snap/flatpack/appimage. Yes that's not sound much "DevOps" but that's the product "DevOps" in practice have created IRL. That's because the Ops part is essentially reduced as a kind of low level mechanical works "please guys just give up the most bare bone system, we will do all the rest" and the Dev part surge to "if it works on my craptop [but well docked on a large set of display] than it works in production, we need to be quick, just grab something from DockerHub, the Nexus, ... and in few hours we will made a new successful Silicon Valley Fever project". It's natural evolution, that could be labeled "cloud APIs", part of the Serverless "model", is even harmful.
The real issue though are not these model but the modern software stack and the story date back decades, when Unix was born their author think they can separate the "system" (bootloader, kernel, basic userland) and "the rest", allowing cheap end-user programming via scripts assembling system functions through IPCs. They quickly discover this approach is fast but fairly limited and they added GUIs violating their own principles since their GUIs can't be assembled with scripts and have only cut&paste/D&D as sole IPCs. A more broad and precise analysis can be found in the Unix Haters Handbook, but as Unix get ground quickly so "modern" GUIs do the same and we quickly evolve from "internet as a network of flexible, user programmable desktops" to "internet as a network of hosts serving some limited desktop to a "internet as a network of hosts who own anything and some modern dumb terminals named endpoints" and that's the present with desktops that are very complex browsers bootloaders, witch are themselves very complex virtual machines, and "the real intelligence" in code is now some third party services offering APIs modern Dev(s) use to assemble their very complex and unmaintainable script named webapps. DevOps was an intermediary step of this hyper succinct mess.
We need to came back to an-OS-is-a-single-application where anything is integrated so things like reading emails meaning a network mount of a remote share, provided by the system, and a stat/read of few files in there, the messages, sending it is just mounting the receiver mailserver share creating a new file there if allowed, similarly reading a website is just the same and so on, oh, it's the Plan 9 model. Oh, such systems are like Smalltalk workstations or LispM where anything is a function that can be called, modified, combined with any other with few lines of code. This model is hard to evolve system side because there are MANY interdependencies to be taken into account, BUT allow to makes modern hyper-big projects a simple and manageable codebases where there is no need of CI/CD because updating is just patching in a version control system, so is rollback and third party services offers just what they want without specific APIs to support.
I know described like that in my limited English is a bit confuse, but if you try reasoning "being outside" you'll see the big picture easier.
20 comments
[ 3.4 ms ] story [ 49.9 ms ] threadThough “scale” seems to mean 10 concurrent users supported by a Kubernetes cluster of 50 systems in order to host a simple web application…
How do you support tens of thousands of GPUs in a cluster?
Now, I am building different clusters for AI.
I find myself agreeing with the part about "scaling", meaning a (good) DevOps team makes it easier to integrate new projects into an existing infrastructure that's built in house. The point where relying on the vendors above becomes more expensive than having a full time DevOps team only happens once you react a certain scale.
A separate DevOps team --> corporate rebrand of the separate Ops team --> no real change, or even slower because more hoops to jump through.
DevOps was always about meant to be about breaking down the separate / silo'd teams and combining them together. Ownership and responsibility is with that single team for their code from development through to production.
The corporate rebrand of Ops teams is what you're coming up against. It does my nut in and I'm sorry to that you have to deal with it.
We didn't really need the scale. We just didn't want to deal with any of those problems. Often there's a little checkbox you can check that solves one or more of these problems, the cost of checking the box is dependent on the size of the data itself. This is convenient _and_ predictable. We like that.
Show me the difference between these 3 pictures (the third being DevOps Engineering.) Go ahead, I'll wait.
They're all the same role, where people are all doing a mixed bag of things for companies that their developers decided they didn't want to do (or sometimes didn't have the experience to do.) I'm a Senior DevOps Engineer, and you could give me any one of the 3 titles, and they would all be correct characterizations of what I do.
A Eulogy for DevOps - https://news.ycombinator.com/item?id=40826236 - June 2024 (94 comments)
Also discussed a bit at the time (of the OP):
DevOps: The Funeral - https://news.ycombinator.com/item?id=36482646 - June 2023 (4 comments)
The vast majority of web software is comically over-engineered. The deployment process has so many features and components that you need a full-time staff of specialists to configure and monitor the build system.
And it could be something as straightforward as a marketplace website or medium complex like some kinds of SaaS. The web application could probably be compiled into a single binary or container and it’s database could run in RAM. And yet to ship a single change doesn’t take one developer but a whole team, eye watering enterprise contracts with several vendors, and it’s still slow as molasses.
There’s a time and a place for config as code and container orchestration. The problem is that it’s used for every single project regardless of requirements.
Im not sure downloading libraries from NPM and adding services till it works is "engineering"
Im not even sure it is good or fast.
I am certain that it is expensive.
For the 2nd time today Im going to say it the only metric that matters is Cost per customer, cost per user, that includes the systems that includes the engineers to run it.
If you don't know these numbers what the hell kind of business are you running.
Follow the money.
I’m not gate-keeping what is and isn’t engineering. Where I live it’s a protected term anyway. People get sued for calling themselves software engineers.
I’m using the Wikipedia definition of, “over engineering.” Solving a problem in an elaborate way when a sufficient and simpler solution can be demonstrated to exist. In many cases you could rsync a binary to your servers and let the monitor process reload your web server. Or you can build out a build farm, dependency cache, and… spend $2, as you say, where someone who is practicing engineering could do well (or better) with $1.
The point stands that it it is done without thought for consequences... "overly complex" is probably a better turn of phrase (and more accurate)
I work at a medium size publicly traded company and our SOX compliance controls would take literal months to generate and/or prove to auditors without our CI/CD pipelines. It's just an extract from GH Actions with a report of who modified, who approved, and who actually pushed to main. All of these actions must be siloed (if you can commit to repo, you cannot push to main)
Potentially this is a consequence of micro service infra, my team alone manages nearly 25 separate git repositories.
However, the first conversation (after finishing up the contract) with the sysadmins went like: "Yeah, ingresses are fine and such, but we'll have our loadbalancer in front of that which will inspect the url paths to know which ingress it has to go to, we don't want some 'developer' to open up some new endpoint without us approving it! Those guys never know what they're doing."
At this point I'm not sure if he's pulling my leg or something, b/c before the we wrote up the contract the whole idea was to give their dedicated platform team and developers more self service. In one fell swoop, bam, negated.
Devops imo is about clear responsibility, it just doesn't work if boundaries are not communicated explicitly. In the above scenario once everything is setup, developers will not gain any velocity, and the sysadmin are still manually adding url routing to their loadbalancers.
Adopting devops, agile, scrum ritual, whatever, will never fix on a technical level something that is a cultural problem. It's a category error.
The real issue though are not these model but the modern software stack and the story date back decades, when Unix was born their author think they can separate the "system" (bootloader, kernel, basic userland) and "the rest", allowing cheap end-user programming via scripts assembling system functions through IPCs. They quickly discover this approach is fast but fairly limited and they added GUIs violating their own principles since their GUIs can't be assembled with scripts and have only cut&paste/D&D as sole IPCs. A more broad and precise analysis can be found in the Unix Haters Handbook, but as Unix get ground quickly so "modern" GUIs do the same and we quickly evolve from "internet as a network of flexible, user programmable desktops" to "internet as a network of hosts serving some limited desktop to a "internet as a network of hosts who own anything and some modern dumb terminals named endpoints" and that's the present with desktops that are very complex browsers bootloaders, witch are themselves very complex virtual machines, and "the real intelligence" in code is now some third party services offering APIs modern Dev(s) use to assemble their very complex and unmaintainable script named webapps. DevOps was an intermediary step of this hyper succinct mess.
We need to came back to an-OS-is-a-single-application where anything is integrated so things like reading emails meaning a network mount of a remote share, provided by the system, and a stat/read of few files in there, the messages, sending it is just mounting the receiver mailserver share creating a new file there if allowed, similarly reading a website is just the same and so on, oh, it's the Plan 9 model. Oh, such systems are like Smalltalk workstations or LispM where anything is a function that can be called, modified, combined with any other with few lines of code. This model is hard to evolve system side because there are MANY interdependencies to be taken into account, BUT allow to makes modern hyper-big projects a simple and manageable codebases where there is no need of CI/CD because updating is just patching in a version control system, so is rollback and third party services offers just what they want without specific APIs to support.
I know described like that in my limited English is a bit confuse, but if you try reasoning "being outside" you'll see the big picture easier.