I assume you are pointing out that split-teams is the insane part, in which case I agree. The philosophy was supposed to define the blurred line between developers and operations, using tooling to help facilitate the shared responsibilities where sensible and make hand-off a breeze.
As you pointed out, the division existed 20 years ago, and it still exists now at many places. That lack of integration doesn't offer much improvement in relation to the past.
Because it's just a hip way of saying you have a sysadmin team. How can something or someone be "devops" if all the developers are in one team doing dev tasks, and the ops in a separate team doing ops stuff?
If we keep reinventing the same paradigms; isn't it a perfectly fine paradigm?
It's unfortunately inherent that some people want predictable, scalable, observable, stable- and at the same time they want fast moving and many new features.
I think it's not so much that dev and admin work is terribly difficult, it's more that "a man cannot serve two masters" and someone will be focusing on stability and someone else will be focusing on quickly deploying.
Trying to do both yourself I think leads to burnout.
The term is meant to encompass the "infrastructure as code" workflow. The "dev" in "devops" refers to the skills necessary to read and write code that manages the state of the system infrastructure.
A devops team is in practice typically very different than a sysadmin team. Good sysadmins automate the hell out of everything, but it’s usually not something other people can use. Devops teams automate all of that stuff but then they also let teams build new infrastructure when they do deploys and automate that whole process, but it’s client facing. It’s development and operations, it’s a pretty good name, though I think platform or production engineering is typically a better name.
The quality of devops scripts, from the perspective of a person who likes to write code that has to work right the first time, is about two standard deviations better than sysadmin scripts. But still of a par that would keep someone from getting promoted past a Senior Dev title.
What’s worse is they can be condescending while shoveling out absolute garbage. If I’m honest that’s the part that bugs me the most.
I don't deny your experience, but my experience is that when it comes to domain specific scripts sysadmins fared much better than developers.
I still see this, developers use archaic or absurd incantations in their build-scripts because they kludge something that works in bash on their machine then push it into yaml and move on, forgetting to comment, document or try to simplify the line so that it's easy to break apart later when it has to be debugged.
Some of the best interpreted code I have ever seen came from a former sysadmin (it was powershell though, that comes with it's own warts of course).
The issue is that we collectively decided that sysadmins should just stop existing instead of allowing it to grow properly. We forget how bad things were for developers back when we decided we didn't like sysadmins. There were companies still passing around code on a USB stick because developers didn't want to depend on SVN being down. (I am not kidding).
Git was introduced 2005 and wasn't really everywhere until 2012, but 2014 sysadmins had all but been retired. This is the timeframe we talk about.
Developers of that time were as bad as the sysadmins were, the problem was that the developers had time and space to grow and we judge the developers of 2012 as if they had all the tools, knowledge and incremental improvements in workflow that they have today.
I think part of this issue is: what is a sysadmin? That’s a pretty hard question to answer today. Maybe it was more straight forward in the 80s or 90s. But if you go to /r/sysadmin you have desktop support, help desk, office 365 admins who only use a gui, solo it guy in a small business, and also programmers, devops, cloud architects and network engineers.
The sysadmins who were already automating large amounts of tasks for n Unix servers and k switches with p services, had a vastly different skill set and migrated to the more modern software defined devops SRe world no problem. But plenty of sysadmins basically could (and some still) only use a gui, and do things from trial and error. Those people might have retired or become managers, but they had a lot more issues with the transition. Which to be fair the transition happened fast and hard in some companies.
The thing i have come to dislike is that in both companies i work for it lead to strange working relationships between ops/eng. anything that breaks in the deploy pipeline is now “the devops team problem” and they normally act like it’s also your fault. In the past this has meant people messaging me saying they are blocked because their dockerfile fails in CI. Then you walk through it with them and it’s also broken locally and it comes down to them pinning to something like latest tag and the image “shocker” changed. But now you are fixing it and after you go remind everyone that you should pin your image to as strict as possible image tag. But no one cares because they can move fast and break things and when it does blame it on CI and now it’s not their problem.
Anyway this is just one tiny example but the same underlying thing starts happening in all parts of the development cycle. “My deploy doesn’t work” … well your program has an error and crashes on startup but looking at the logs would be too much work. Let’s just say we are blocked and log off.
As you can see this is a culture issue but it seems to be common when you have a devops team. And even more common when management (read senior management and up) think that developers “should just be developing”.
> Half the time the developers in question won't even read the error output before raising a ticket.
If they had no DevOps team to raise a ticket with, and it had to get done they would find some way to figure it out if they had the ability to modify and initiate their own pipelines. The reason I like "you build it you run it" is I feel like I have far more control over my own destiny. You're not having to negotiate and coordinate with a separate team through some ticketing system. Far less wasted back and forth and miscommunication. If that means you have to expand one of your teams with an additional DevOps person to foster understanding of the tooling on a team who hasn't done it before that's fine. But don't make them the DevOps person. Their goal is to enable the rest of the team to pick up these skills. Their goal should be to transition into a normal contributing member of the team working on product development because the team no longer has a need for a DevOps person.
So in other words, you made silos. More throwing things over the wall and having it be someone else's problem. That's fundamentally the antithesis of DevOps.
People do not fix the pain they don't see, and managers doubly so. By shifting operations off to a "devops" team, you just end up with developer teams that don't know about, and don't care to fix, the things that make it painful to deploy or run their code in production.
Those that have to deal with the pain rarely have any political leverage they need to even enact change. There's zero motivation for either the developers writing the software _or their management_ to fix it. Dev manager incentives are purely aligned with getting code and features built.
To give a nice real world example, building out new regions for AWS services used to be a pain in the arse (since I left I hear some major top down initiatives have fixed that).
The service team I worked for, almost doubly so painful for region build. Standing up an entire region was something like ~50 days of engineering time. Not because it was especially hard, per se, but because the ops team always had to wade through piles of refactored shit that introduced new circular dependencies, or wasn't documented, or broke other aspects of any automation we could produce. (I remember one DynamoDB schema overhaul, where we did a complicated migration. Next region build, service components wouldn't stand up because we'd used the automation to create the new table schemas that had replaced the old table schema entries, but the service code still expected to see the old schemas in the right format. It took several months to get the dev manager to agree to engineers spending time cleaning up old schema references from their code base.)
The dev manager incentives were not aligned with making it easy. It wasn't their engineering resources being consumed, and they had features they needed to ship.
The developers never changed the way they worked because even though they were told the issues (and even though they were, virtually to a person, great people to work with, considerate etc.), they never experienced the pain so they never thought about those consequences of what they were doing when they were doing it.
Ops manager never had any leverage, because every single time we managed to get a region launched in time as to not be an issue, and every time things were down to the wire it was clearly as a result of the ops team not building the right automation :eyeroll:, rather than that the dev teams were making some kind of brand new nightmare each time.
It wasn't until the ops team had lost enough members of staff to that BS, and region build shifted on to the developers themselves, that suddenly most of the problems went away, never to appear again. They were aware and conscious of the pain certain decisions would cause because they had felt it themselves, and they made sure not to introduce more pain.
DevOps is about making sure incentives and politics align to make things overall better. Shoving it in to a silo is literally the complete opposite.
I think there is room for a DevOps team if they are focused on creating and maintaining the overall DevOps platform. But if they are taking tickets and setting up pipelines for individual applications, they are doing it wrong.
But a team focused on making a streamlined platform with ready to adopt patterns that already account for compliance and security related concerns, they can spread DevOps adoption more effectively and consistently throughout the organization.
The only thing I like about the return of sysadmins and the end of the "you build, you run it" culture of Devops it is that I'm 100% off the hook if something dies in the middle of the night.
The so-called DevOps guys are stressed out of their marbles, but when the CTO calls me 4 AM I can't say anything but "Sorry dude, I'm not a clairvoyant. I have no permission to do shit in prod, call Ops or give me AWS access now". He stopped calling.
The fun part is Ops wants us engineers to help more, but they don't even trust (and some don't know how to run) my Terraform or CloudFormation scripts (I let them choose which), so they just read the tf/yaml and click AWS buttons manually. Often making lots of mistakes.
I always do a better job of writing logic when I sympathize with the victims. Some of those late night calls are devs setting ops up for failure. It was meant to close an open loop.
> The fun part is Ops wants us engineers to help more, but they don't even trust (and some don't know how to run) my Terraform or CloudFormation scripts (I let them choose which), so they just read the tf/yaml and click AWS buttons manually. Often making lots of mistakes.
Yeahnah, you're part of the problem.
If you've not documented it properly, and more importantly, if the team are weary of prodfucking TF/CF scripts, there is a culture problem. You both need to understand how that works, if they don't have enough time, then you're understaffed, that needs to be fixed.
I worked at place that was going through a really painful ops->devops->dev oncall transition. Ops still had a shit ton of power, but nobody included them, and just shouted at them when stuff went red. It took a huge culture shift, and devs sitting in ops and doing a shift or two, to sort it out.
Sorry, but how did you manage to make so many assumptions from that?
Everything I need from Ops is already enumerated and specified more than well in tickets (which is required, as is par for course with the sysadmin from hell stereotype), and documented in Markdown files within my app's source code.
I generate scripts for them since that's what they WANT and ASKED for several times.
They literally want me and other managers to help with DevOps (Heck, I'm even AWS certified), but don't trust me and others to give access. They ask for scripts and I comply to save their time. How am I on the hook if they want me to create scripts, but some of the new hires don't know how to use Terraform/CloudFormation?
> "there is a culture problem. You both need to understand how that works, if they don't have enough time, then you're understaffed, that needs to be fixed."
The culture problem is Ops wanting to do everything and not trusting. If they don't want to "prodfuck", just review every script. Or perhaps test them in the staging AWS account that I pay for with my own money. Or just stop asking me to make infrastructure as code. Or suggest a fourth option instead of wasting my time and being disrespectful.
I can't change other people's behavior. I can only change mine.
If they are understaffed I can help, or they can hire more. Wasting my time and making me do pointless work is not the answer.
> I can't change other people's behavior. I can only change mine.
Ah my friend, you totally can change other people's behaviour. go on. hug an ops, sit next to them and work with them. show them a bit of love and make them feel part of the team. Think about it from their view, why don't they trust your scripts?
Why dont they feel empowered to spin up a staging area? is it a skills gap? fear? or do they not feel confident enough to ask.
It sounds like solving their problem will also solve yours....
Skills gap from a chunk of the team, lack of time, lack of trust in developers, not accepting help when offered, not accepting that developers have the best interest of the company in mind, not accepting qualification of developers.
I can only offer help, do what they ask and try to make their life as easy as possible... I can't hug people via Jira tickets. Yeah, they're that isolated.
That was never supposed to happen, and DevOps enthusiasts still think that's a joke. Unfortunately, at some places, that was the _only_ way to get higher-up people thinking about the culture.
Kind of like the idea of a scrum master, which was never supposed to be a role
The creation of DevOps teams instead of SRE / Platform seems pretty common and I think originates from ignorant engineering leaders and poorly defined roles and responsibilities. You want to leverage the expertise of the folks who focus on cloud infrastructure, but you can't move the responsibility of operations away from the application teams without seriously compromising incentives that effect quality, throughput and reliability.
Can you expand on what you mean by this? Specifically: "you can't move the responsibility of operations away from the application teams without seriously compromising incentives that effect quality, throughput and reliability"
Your application architecture dictates how it should be deployed and the other way around, if you try to decouple the two you're bound to run into issues. Depending on what your application does, high throughput with some loss might be ideal, or you might need at least once delivery instead while sacrificing throughput, or any other dynamic. If your deployment is done by an entirely different team from the team that builds the product itself you have little hope of predictable positive outcomes without a lot of pain.
To be more blunt about it: if the person writing the code is not the one on the pager at 3am, they have no serious incentive to make it correct or resilient.
This is why "you write it, you run it" is so important - the most reliable software is the software never written, but if something must be written, it's important that all incentives are aligned for it to work right.
Of course, this is only at the operational end - lack of incentive alignment is rampant in "product" organisations too.
as a software developer, every place I've tried to work that's "you write it, you run it" turns into me just spending all my time running the code as it already existed, with actually trying to improve reliability as kind of a bonus 10% task that no one ever talks about or asks after.
It always depends on aligned incentives, for some very specific lines of work and very specific people, they will totally intentionally upkeep known bad software without improving it too much because it's somewhat a guarantee of very profitable overtime pay.
> To be more blunt about it: if the person writing the code is not the one on the pager at 3am, they have no serious incentive to make it correct or resilient.
The problem is in 99% not that the developers don't want to do correct code. The problem is that product/project management doesn't allocate the time and budget for clean development practices, proper QA, and refactoring because their metrics are revenue, releases/timeframe, or similarly useless/unsustainable.
Can 100% confirm. Last place that had that pager duty tried to throw this shit on me. Product Owner was constantly denying time to build monitoring.
Everything in this loop was misaligned and rigged against devs. The only power I had as was to quit.
Now that this is over and traditional Ops is back and called SRE, it is Product Owners, Support, Managers, Developers and CTOs screaming all at once at their team. Let's see where this will lead us.
Regarding enterprise culture I’m curious who will be in this “platform engineering” team and what expertise they’ll have.
In (enterprise) reality it will more than likely be a bunch of sysadmins and former “network/storage guys” who saw which way the wind was blowing and got a cloud cert. They really have no business forcing a particular infra pattern on application teams because they have no application level expertise and nothing about adding LUNs to ESXi, typing Cisco configs, or writing bash scripts to automate yum update gives you expertise in application design/architecture.
Then because they know nothing about application level concerns the PE team’s golden patterns will either be ignored or will be customized for every application to the point that they’re unmaintainable.
Note this is the enterprise view. It’s probably different at tech focused companies.
I think that once a company is big enough you can have a few options to get code to prod and make sure it stays running as expected. You could have an option where a team is focused on building most of the platform up to an agreed point (e.g. everything under an application’s container image) and provide some common ways of setting up application monitoring / logging etc…. Making sure that while there is a core internal platform as a product team that you’re focused on being a capability rather than a business unit - by that I mean you should have some if not most folks embedded (or rotating through) dev teams. You can also have an option that lets teams self organise how they get their code running - with some light touch guard rails and potentially caveats for applications with PII etc…
> In (enterprise) reality it will more than likely be a bunch of sysadmins and former “network/storage guys” who saw which way the wind was blowing and got a cloud cert.
This sounds like a tale of woe. but. The fault is the culture, not the people doing the grunt work. Like devs, if you're used to nails, then you'll want to use a hammer for everything.
> gives you expertise in application design/architecture.
a sysadmin should know much better how your application runs in real life, and more importantly, how it behaves when it goes to shit.
They _should_ know more about all the different types of services (ie kafka vs NATS etc) because they will have had to set way more of them up than a typical dev. If thats not the case, you've had bad or poorly mentored juniors running your stuff.
The reason they should know, is because they have the pager to make it work at 4am.
Look, if I had it my way, and I was at a medium sized company, I would buy in 3 mainframes, in three disparate locations, set the lock step replication to "very yes" and make sure each dev used the local DB and use the job description lang, to define scale.
Pay the $1million a year to outsource reliability of DB and state. Because frankly most people can't program for either scale, redundancy or recoverability. (yes that includes k8s) let the hardware do all of that and save a bunch on staffing costs.
I work at a HUGE scale place, and yes there is a need for special sauce there. but even then not that much, and not enough to justify the massive churn in systems. Most PE is just cargo culting busy work. Stick to ECS and PSQL and just get on with making actual buisness features.
Whats your SRE / DevOps ratio? In my experience, in the 10 teams I had or oversaw, I had about 1 FTE in DevOps for 15 SRE. Of course it depends, but what’s your personal experience?
Yes, unfortunately, the misconception that DevOps is a role or person, rather than a culture, is the root of most problems with DevOps, DevSecOps, and DevSecMlTriceratops.
I just started working on a more devops/platform kind of role. I believe we're basically doing what you feel like is stuck in an era. Can you shed some light as to how JavaScript is used/ is an improvement?
I have a tool that I'm about to release soon that you may be interested in. It's a way to configure servers (with bash or whatever you want) in a distributed, decentralized way. You build "patterns" which are signed build and run scripts that the servers can be pushed or pull the patterns from a storage provider (like a storage bucket). The tool exports various prometheus metrics that you'd use to determine the state of it and whether the patterns were successfully applied etc. Best of all, it's stateful by design, so when you add or remove things the pattern will clean up after itself. Message me if you want to be part of the beta.
DevOps just makes the problem worse which it is intended to solve. DevOps started from "deploying is too hard, and the developers don't know how to run complicated shell scripts to deploy our software to release, anyone should be able to deploy it and have it scale!". However, most companies started to cargo-cult Google and just built everything in the "most scalable" way, incurring tons of overhead, both in deployment friction and hardware costs, so we are back to DevOps being an almost completely separate field from software engineering, and the programmers still can't deploy software.
Devops absolutely fixed a lot of issues. Going from manually merged SVN branches on a quarterly release monolith to a "click and release application" is a huge saver in developer time and pain.
that's great for you. at my company devops are still doing monthly releases and unable to deploy single microservices because all the services are too interconnected.
We use simple github workflows to release microservice based services and frontends on a daily basis. They're pushed through a series of dev environments for testing before being "released" into production by managers. It's not hard for devs at all.
> DevOps just makes the problem worse which it is intended to solve. DevOps started from "deploying is too hard, and the developers don't know how to run complicated shell scripts to deploy our software to release, anyone should be able to deploy it and have it scale!". However, most companies started to cargo-cult Google and just built everything in the "most scalable" way, incurring tons of overhead, both in deployment friction and hardware costs, so we are back to DevOps being an almost completely separate field from software engineering, and the programmers still can't deploy software.
Absolutely this. Instead of collaborating to solve problems we are wasting time on pet projects.
DevOps as a culture vs DevOps as the neo modern unix admin has no right answer. It's just like software development philosophy and organization.
It's a fine approach, if you have a well aligned group of people working with the same understanding. The same is true for places that prefer to draw boundaries of responsibility in other ways.
All this IT stuff fails, needs to be replaced, repaired, or adapted to a new situation.
The best solutions, whatever the methodology and avenue of implementation, are made better when the builders "live below the dam."
The incentive to build the best dam is when the party responsible must live below the dam.
The best metric is customer retention. Directors of customer support need to be on the phone with escalations, not just watching metrics. There's a leak? Take the complaints. Embrace the hate. Fix the damn dam.
If the DevOps "culture shift" didn't solve problems for you, this will not either. You will still have similar problems, now abstracted away into a different team that operate at their own pace. They will also be overbooked, like before, with the difference that now they are serving multiple teams you didn't have to contend with before. They will not share your incentives or priorities for specific features. Their time will still not be taken into account during product planning.
Both the DevOps move, and its current issues, as well as platform engineering are organisational problems. Your organisation needs to be set up around this in order for it to succeed.
Notice that the article paints platform engineering with the same utopia like brush strokes that once painted "DevOps as a culture", only to be mangled in its handling and blamed for the problems that it brought about.
To me, DevOps culture just feels like a way for businesses to save money by not staffing a dedicated infrastructure team and pushing all those responsibilities onto application developers.
The amount of time I've spent fiddling with Terraform, Ansible, Kubernetes manifests, Helm charts, Jenkins configuration, GitHub Actions configuration, AWS IAM, and so on over the past few years is absurd, probably more than the time I've spent writing actual application code.
What now? If you’re spending that much time on devops I feel like you’re doing it wrong. I do eng (team of 7) and devops (only me) and I feel like I spend about 5% of my time on devops.
These tools have made deploying production quality infrastructure take a fraction of the time. These environments are basically already preaudited and verified. They’re also just much higher quality in general. It’s quite typical for different projects to share infrastructure in traditionally deployed systems but with devops we can have much better isolation with less deployment time.
> To me, DevOps culture just feels like a way for businesses to save money by not staffing a dedicated infrastructure team and pushing all those responsibilities onto application developers.
One of the things that bothers me the most with the cooperate software development rat race is how many problems are being solved over and over again. Every company is staffing their own devops teams to build their own abstractions over these technologies so app developers don't have to worry about it. I personally know multiple devs who basically move from company to company reimplementing the same devops tools at each one.
It all just feels like a collosal waste of energy and collective resources.
Who patches your VMs? Installs Antivirus.. installs vulnerability protection.. defines your resource groups.. virtual networks.. sets up your Azure subscriptions.. performs billing management.. etc?
I hope its not you. The whole idea was that there is a platform team responsible for providing a service or better yet the CSP does this for you and then you just have to architect as a platform team the things I listed above. It does not happen via magic. But the CSP platforms makes it certainly easier to provide a pre-audited version of the OS and tools.
You should be able to self-service yourself within say a developer like portal as part of an IDP as described here. Not figure out ALL of the infra yourself.
There are more people than you think that can slap together some YAML and a Helm chart and deploy something reasonably resilient inside a Kubernetes cluster.
I've been around for the cloud migrations in every phase, what we have today with Kubernetes is miles better than the maze of Puppet and other crap we had 20 years ago.
What happens when your Cillium Service Mesh starts misbehaving due to missed network packets ?
Or when you have to setup a resillient backup of a cluster that runs 30 000 pods on 300 nodes ? And upgrade.
Or when you have to provide a telemetry for highly distributed system using 5 languages on landscape spread over frontend and backend with three different db engines and two clouds ?
Kubernetes is easier to start but 10 times as hard to do at scale you didnt even have back then in OnPremise era.
Yes you are right there are a lot of ppl who can write and deploy „Hello World” apps, but thats about it.
Since the in house server days, technology solutions have only become more and more time consuming to implement, ID & PII obsessed, and bulky (data footprint-wise). I am thankful I remember much more simple times, when apps were tiny, run on regular computers in my office, and you could symply restore from a local backup to bring a service back online.
Now we have companies paying thousands of dollars a month to host a simple site or web app, because now it's all in the cloud, no matter the apps function, while hosting customers assume about 80-90 percent of responsibility for securing and maintaining their app anyway. Containerization is leveraged to add a "backdoor" for individual admins, but that only serves to protect infrastructure, it actually makes a customer's individual app open to vulnerability that same as running your own instance if you miss vital updates or if there is a breakdown in the supply chain. We need to stop listening to the companies that market tools and be honest with ourselves... Security needs a better model than just adding new tools and entry points to the app chain.
There's gotta be a point where you evaluate things and tell yourself that there has to be a better and more affordable way than tunneling through 5 VPNs just to push updates for 8 disparate JS and PHP libraries and 5 containers. App and library updates are also far too frequent in 2023 as well... Frameworks need to ship instances with less features, and modules should be reduced to only those used and deemed most essential, reducing footprint is a key aspect lost on technological advancement, it is also a firm indicator of increased efficiency for app and library devs if you ask me.
Now with zero trust, devs literally spend a lot of their daily workload logging back in and re-establishing VPN connections due to timeouts. There's got to be a point where we develop a far more simple IT solution to all of this mess. Security is still getting compromised regularly no matter what is done. The solution also won't likely involve Ai at this point in my opinion, as security and complexity are assuredly not resolved by leveraging current-state Ai tools.
It's also important to note this is why PHP is still going strong, it doesn't require compilation, and is relatively easier than most other langs on learning curve to implement.
I literally built my IT career on making things simpler, and in explaining complex IT issues in human language to non-technical people. There is a lot of room in my field for growth due to the rest of the industry's constant focus on jargon and over-complexity.
Simpler solutions, conveyed and implemented in human language, will be king in 2024.
> Simpler solutions, conveyed and implemented in human language, will be king in 2024.
I would love this to be the case, but it seems that complex and expensive systems are too lucrative to everyone involved: CTOs get to manage big budgets and bit teams, devops people get paid large salaries to justify the CTOs salary and large service budgets, etc.
Agreed, many of those huge and overly complex projects also fail and overrun implementation schedules though, and they only ultimately operate with a ton of workarounds and shortcuts. I find a lot of projects that barely match original documentation limping along, or just even after a major failure or breach in my line of work, with the customer desperate to find a patch to just keep them running after a bunch of critical data is under siege within apps.
A key for me is convincing a customer to simplify goals and implement an MVP on a reliable framework, to plan versions, and to thoroughly test... Makes me a code cop, but there are reasons why these steps are in place for mission critical services, and if a customer wants to skip them and just deploy in careless "Twitter Dev Mode" I have them physically sign paperwork that states their accountability for the bad decision... When individual accountability is mentioned, they usually opt for the more stable & reliable methods of dev. Hah.
I think the biggest issue with devops is that it originally meant "socialising" your sysadmins, by getting them to sit with your devs, so that shit didn't get lost because nobody thought to talk to the right team.
But then it morphed into "oh lets innovate with infrastructure" but the innovation turned into "lol lets just restart from scratch and ignore history" Anybody who used early k8s can attest to how un production ready it was for the longest time. It _felt_ like progress because it required a lot of (to dev eyes) hidden magic to make it work.
Now, we are back with sysadmins, but they are called SREs. They provide a platform, which devs talk to to deploy stuff.
Basically we're back where we were in 2015, just with more yaml.
The good things that have come out of devops has been the dashboarding and metrics tools. Its just a shame that everything else touched by it appears to be an infinite source of busy work (looking at you kubernetes)
So you mean, DevOps has made Sysadmins more service oriented and now instead of having their pets and being in overprotective silos, they actually provide best practices and proper platforms for the poeple developing the software?
The place it worked the best was where a team of sysadmins were split up and embedded with each dev team. each devop would then rotate onto another team every 6 months or so. This meant that _we_ had to document our shit for the next devop, but also helped eliminate key person dependencies.
We would then have a weekly offsite where we'd bitch and moan about our teams and conspire to make things better.
Im not sure how former sysadmins feel about it, but from a dev standpoint, DevOps worked great for us. Making a clean cut on a plattform. Was my app down, I got message/called. If the platform was down it was the Infrastructure team / PaaS provider that needed to fix it.
In the old days we had so few deploys and a hotfix was something that was extraordinary. I actually had a board of something in another country approving that we where allowed to deploy. They knew nothing about the application..
yes this server is special to me. No, I cannot move workloads somewhere else and shoot the server in the head. Because customers are running GPU workloads there, I cannot transparently migrate them
How does the analogy falling flat for you? GPUs are interchangable resources; an H100 is just like any other H100, even if you can't take a snapshot of an intermediate state from one and load it onto another and resume.
I thought about it and I don't hate the analogy. Although usually I'm very against comparing IT stuff with physical stuff because it tends to produce pointless conversations.
What I hate is the attitude of "treating your server is a pet is wrong, you should be able to kill it and replace any time". I feel like people who say it never run anything more complex than an HTTP API.
Ah yeah I totally agree. Having pets is fine until it isn't. A single-digit handful is fine. 1,000 bespoke pets, way above Dunbar's number, isn't. Where exactly you run out of mental headroom and should convert is up to you; the point is that there is one and converting over should be planned for in the North Star roadmap of future development.
My (Raspberry Pi) home server is very much a pet, but I'm never going to productionize that thing, so I'm totally fine with it being a pet, despite the practice at work of having no pets, all cattle.
More a general question, but I’ve gotten pings in the past to be an SRE at a FAANG despite not doing much cloud stuff. Is it the type of role people take for a tour of duty and then transfer out of?
I imagine there are SREs who enjoy it and stay there, but curious why they’d invite someone heavy on the dev side to interview.
Depends on the FAANG, Google famously throws devs at ops problems.
Ironically they realised this wasn't a good idea in exclusivity, so now they hire two disciplines: SRE-SWE and SRE-SYS.
SRE-SYS is as you would expect, pretty standard sysadmin knowledge from 2005 (IE; not helpdesk sysadmin, some scripting knowledge, heavy OS knowledge, heavy debugging knowledge).
SRE-SWE is much more programmer heavy with a light touch on OS fundamentals (based on what I consider important for OS knowledge FWIW).
the downsides of SRE is that you tend to be super hyper specialised in something thats only really of any sense to that department inside a company.
Moreover, you're dealing with real steel machines, and amounts of data that are prohibitorily expensive to just start again with. (the joy of AWS is that most of the times you can delete it all and spin it up again, often side by side)
also, some SREs are really specialised software devs, others like me are sysadmins who'd been swept by the current into SRE roles.
My company SREs are not like google's, where they are rock stars swanning about doing one line fixes that make everything .48% more reliable. We are computer nurses making sure that the wards of systems we look after aren't wiped out because a doctor forgot to wash their hands after taking a poo.
However, they both require the same skills: murder mystery and the ability to dive super deep into the stack, go from python->c->assembly. or service->VFS->cache->network->time weirdness
Also it could just be copy pasting service defs and making dashboards, then pinging devs saying "shits on fire yo"
I know a few old school sysadmins who got left behind because they couldn't upskill to DevOps and consequently lost out on $$, others who were fired after moving everything off prem to AWS because the company no longer needed them. You understand that your theory is a "conspiracy theory", so why even say it? how does that add to the discussion?
Maybe in FAANG it's "back like we are in 2015" but the tools that have been brought into the mainstream like Kubernetes, automation tools and such have had a great impact across many small to large sized organizations.
It's done a great job to enforce consistency and immutable infrastructure in a way that is more approachable for most when you get past the scale of a few resources.
I've seen start ups write their on bash tooling, watch it devolve into some opaque thing that no one knows how it works or what it does. I've seen platform teams devolve into managing VM updates by hand, babysitting operating systems of every application deployed because it's treated as a long living pet, and it gets worse when team leads move on and then someone else comes in and half bakes their own flavor of change. Even better when networking gets involved, load balancing with HA where people roll their own solution. On top of that is the certificate story and how that gets managed. Don't even start with RBAC from a platform perspective, it's a nightmare. The toil filled platform days should be behind us. Plenty of companies are slow to evolve and you can find a lot of pain.
Perhaps I've had bad luck in my history of being a network engineer, infrastructure engineer, and an SRE. The one big benefit to these tools that are coming out is it can give some sort of standardization and it's giving you a ton of power in abstraction so that platform teams aren't micromanaging things that are just sorted for you.
Something like Kubernetes gives you load balancing, high availability, RBAC, networking and routing, resource management, node draining and rotations, auto scaling. Sure it's managed in yaml manifests, but it's doing a lot of lifting. I won't say that the abstraction can't bite you, you still need to understand the core concepts of things. However people should be understanding those concepts regardless.
The amount of work that takes for a small team at a startup that needs to scale services is immense without it. Right now I'm working with a 3 man platform team and we are running over 200 services in multiple regions, multiple database technologies and analytics stacks, we auto scale aggressively and we serve over a billion requests and terabytes of traffic a month with over 99.99% of uptime. We also deal with operations, security and every other hat that someone wants to throw our way. The metrics solutions to come out of this are also such a huge win, I remember the days of Nagios, Zabbix, SolarWinds etc. They were mostly static graphs showing you snapshots of what happens at a very high level, now we have metrics for everything and you can feed that into anything to self-remediate or escalate. I absolutely love it and can't wait to see it evolve more!
The previous iteration of platform at this company was all self-managed VMs with the old school sysadmin mindset. They were managing hundreds of VMs by hand or using some light ansible that was retrofitted to work with pre-existing machines, we were using corporate F5 load balancers and VMware which are immediately out of date the moment you deploy them. Some older sysadmins were legitimately afraid of rebooting servers because they were afraid they wouldn't come back. Some machines had over 900 days of uptime because they just couldn't keep up with the toil. Also this creates a wall between the platform and developers because it turns into a hand-off process where developers throw code over and the platform team has to implement some server and load balance logic, let alone meeting SLO/SLI criteria. Sadly, that's not uncommon to any industry. This team had 15+ people and had trouble keeping up let alone deploy and innovate. Outages in this mode are unavoidable, and pulling away from that old school mindset and starting over is VERY hard.
I'm sure a lot of that can be contributed to bad management and bad hires, however from my perspective using tooling and s...
> but the tools that have been brought into the mainstream like Kubernetes
unless you are running real steel, kubernetes is at best a complication layer over the underlying cloud tools. In 2018 me and my team built a real time, ML-based processing API. It was hugely scalable, optimised for speed, price and uptime.
At no point did we use need to use kubernetes. Not only would that have eaten most of our innovation budget, all it would have done is add latency and driven up the money we spent on inter VPC transfers.
> I remember the days of Nagios, Zabbix, SolarWinds etc
So do I, they were recognised as shit even in 2015. $work allowed me to lead the migration away from them and by then we were half way through replacing most of splunk with graphite/grafana based metrics instead. The rest of the alerting was mostly handled by a custom inhouse /__status end point parser.
> The previous iteration of platform at this company was all self-managed VMs with the old school sysadmin mindset. They were managing hundreds of VMs by hand or using some light ansible that was retrofitted to work with pre-existing machines, we were using corporate F5 load balancers and VMware which are immediately out of date the moment you deploy them.
That, again, is just bad sysadmin. Unless you have complete control over your stack with ansible, then its not ansible, its just sparkling bash script.
I worked at a VFX company that deployed to all real machines for virtually everything. They used a GNU configuration system that was older than SSH (I can't remember what its called, it used to use rsh, that's how old it was.) All the machines were fungible, We could replace a file server at the drop of a hat (the only limit was how fast we could copy 160tbs over 2x10gig nics) Most of the place ran on perl, in one git repo for just configuration. OS updates were applied weekly, and we pushed system changes daily. This was in 2013.
salt made all of that easier, especially as we had way too many hosts to sensibly run ansible at the time(I think we worked out it would take >1day to run a basic role).
They had automated it to the point that you could plug in an unconfigured blade host, turn it on, and within 30 minutes, the bios was updated, networking configured and each blade was starting to be installed.
Automation is an attitude. You can either invent new tools to solve hypothetical problems badly (k8s supposedly is high scale, but cant practically get above 1k hosts, its also balls deep in ipv4 ), or actually use/create tools to solve the problems you have now.
DevOps was originally about moving ops into dev teams. At Amazon they just gave devs pagers. Devs hated that. The SRE role seems to have taken over the app-facing side of what system engineers did being an interface between an app and the hardware. A lot of what I see SREs doing is things like app metrics and SLAs which were not what system engineers really did. And the SRE role has been pushed around from being embedded in dev teams, to being their own teams that managed apps to being a centralized DevOps team that looks an awful lot like the old silo'd split between Dev and Ops. I don't know where the actual system engineers went, they probably run Kubernetes clusters at FAANGs and have SRE titles these days.
...and all of that is before looking at the tools in the CNCF diagram.
Maybe this is "old man yells at cloud" me but I miss 2015 DevOps.
If you knew Terraform, Ansible/Chef, a monitoring tool, a cloud (usually AWS) and a programming language (usually Python), you were all set and could still be idiomatically/culturally DevOps.
And I say that as someone that knows Kubernetes very well
Yuck, Portainer... when I read that I get nightmares.
The problem is: when you give developers access to Kubernetes, to ECS, to Portainer, to whatever they. will. not. care. about literally anything. You'll find a hotpot of cobbled together Dockerfiles with zero provenance, with base images pinned to stuff years old, and completely inefficient layer ordering. Pipelines won't use caching or parallelism (okay, sometimes because they use Maven which is a clusterfuck on its own).
Software developers are developers, 90%+ will never have heard of how Linux works below the hood, how Docker actually works, they'll just copy and paste together stuff from Stackoverflow without even attempting to understand how it works.
And I don't even blame them. I can't, I don't want and I won't.
Companies, it's time to stop loading stuff onto your developers that they don't have to (have) any clue. JFC.
The developers are literally paid to do the least amount of work that could possibly do towards delivering some abstract MVP "this sprint".
Actually making something that is simple and maintainable is the 7th or 8th priority of business, if it is a priority at all.
Artificial "obstacles" to delivering the artifacts in time in fact, wanted, a sort of corporation Deus ex machina, externalities to explain why they did not meet delivery deadlines last time.
Incentives are not aligned. Things will remain broken until incentives are aligned.
And some devs do learn how Linux works but don't want to waste their time memorizing how shitty $DEVOPS_PRODUCT that'll be replaced in two years works. :D
Indeed. Which is why I waited for three years to take on Docker, and two more for Kubernetes after I got burned with DC/OS, and why I entirely left the NodeJS/frontend ecosystem. I'm too old to act as the ripener for bananaware any more, and I'm pretty sick of companies exploiting other companies or, even worse, individual developers to act as their free QA department.
As someone that knows how several UNIX variants have worked since I used Xenix in 1993, those devs usually learn how a specific Linux distribution works, place them in another work, and they will be hunting for configuration files, init systems, daemons and what not.
Same applies to me, "knowing" GNU/Linux since Slackware 2.0.
For a given level of complexity in an organisation, talk to any 10 experts on the subject in the organisation (ie how does the foobar process work) and you will find blind men describing an elephant. All technically correct but all missing important parts
The paradoxes
At some point an organisation acquired or grew an antogonistic part. Both cannot really exist but both have not been removed. Think either Google search vs Google ad sales (if you return perfect search terms each time, you get less page views hence less ad impressions. Make search worse get more ads served). Or simply other non core businesses. Pretty much every non-tech mega corp is really dozens and dozens of smaller businesses with a shared treasury so they can ride out each others ups and downs. But one or two businesses are really carrying oat of the load.
Anyway, you can find paradoxes or conflicts of interest in all businesses and badly aligned incentives hurt everyone.
Someone should fix that. Not me I don't have the power (indeed usually only the orignal centralmfounder has power to)
Safe spaces
Yeah these don't exist. If you ain't upbeat publically you are seen as trying to pull everyone else under. So how are problems acknowledged let alone fixed? Dunno.
What would really help would be free speech, open conversations and discussions based on data that can inform policy and provide impetus to chnage. we (kinda) have this in western journalism / media (including social media). But absolutely no large corporation anywhere has a open discussion area where different factions can raise their concerns. there are of course political infighting amoungst the powerful elites of the corporation but that's not the same thing - they might even be accurate about the problems but no one else knows.
Democracy matters
Sorry this was supposed to be about devops - and it kinda is. Most large corps have some devops thing going on where "this time we will fix it all for everyone" - but has there been open conversations with data supporting it from all parts of the org? oh come on.
"DevOps" as in Developers being closer to Ops, or "DevOps" as in "I'm a shiny Golang outfit that is going to solve Real Problems TM with YAML and a bunch of templated-generated Golang?"
I realize this is a bit of an odd tangent, but I would love to hear cadey talk more about what "devops" means at a shop like Tailscale. Actually, as I type this, even more curious. I know some of those folks are cut-me-and-I-bleed-Go types, but I also know they're deep into NixOS for server management. It would be very interesting to hear more about how those intersect for their infra/server/dev culture.
Most commenters here seem to ignore the fact that saas deployments have become increasingly complex over last decade in part bc low hanging fruit has been picked. Also the massive scale of even relatively no-name companies out there.
I'd add that most of the 1-4 "problems" (boo-hoo, managing IAM is boring for devs) OP mentioned as well as advent of so called "platform engineering" stem from one thing only - poor engineering leadership.
What drives me nuts is the number of places that expect you to have deep experience with whatever CI/CD solution they're using before they'll even consider you for a position. How have we gotten to the point where deploying an application is so fucking arcane of a process that you need to have 5 years of experience with whatever acronym soup some CTO bodged together or you are considered worthless?
Replacing "DevOps" with "Platform Engineering" is not waving a magic wand and removing problems. The original article's main point is: "use DevOps marketplaces", more or less, but that comes with its own set of headaches (mainly auditing and making sure you trust them, which might easily take more time than rolling your own stuff).
I fully stand behind the "just have programmers that write internal tooling" recommendation though. Over the course of my long career this is what has worked best almost every time, and these are the programmers who get the least amount of respect even though they have to dance and juggle on a burning rope more often than not.
In my opinion, more than half the problem is the tooling, as indicated by the cartoon in the linked article.
DevOps pipeline languages are generally either weakly typed or stringly-typed. They generally expect users to do the control character escaping, in several different obscure formats. Did I say escaping? I meant three layers of nested escaping!
Variable/macro substitution as a rule uses syntax that overlaps with either PowerShell, Bash, or both because otherwise it would be too easy.
Another basic thing is the absence of a debugger — I have to run a complete pipeline with a change to dump out environment variables!
Instructions contain the display names of paths, scripts use the variable names, but the output (and errors!) use the values with no clear mapping backwards. What the eff is C:\s\1\x? I dunno, but it caused an error!
The worst sin is not having a local development experience where the entire pipeline can be run without having to commit code and push stuff almost all the way to production. Even if the pipeline agent is made available, it’s useless without the underlying VM image which is huge and not easily obtained. Not to mention KMS/KeyVault and other similar services on which pipelines can depend.
Developers complain about having to do DevOps, but in my opinion the problem isn’t who is doing it, but how. The tooling is just so bad, so very very bad, that anyone would hate it if they were the ones forced to do it. I’m an ex-dev-now-SRE and I hate it.
125 comments
[ 3.1 ms ] story [ 370 ms ] threadThe idea behind devops was to be a single team.
I assume you are pointing out that split-teams is the insane part, in which case I agree. The philosophy was supposed to define the blurred line between developers and operations, using tooling to help facilitate the shared responsibilities where sensible and make hand-off a breeze.
As you pointed out, the division existed 20 years ago, and it still exists now at many places. That lack of integration doesn't offer much improvement in relation to the past.
It's unfortunately inherent that some people want predictable, scalable, observable, stable- and at the same time they want fast moving and many new features.
I think it's not so much that dev and admin work is terribly difficult, it's more that "a man cannot serve two masters" and someone will be focusing on stability and someone else will be focusing on quickly deploying.
Trying to do both yourself I think leads to burnout.
What’s worse is they can be condescending while shoveling out absolute garbage. If I’m honest that’s the part that bugs me the most.
I still see this, developers use archaic or absurd incantations in their build-scripts because they kludge something that works in bash on their machine then push it into yaml and move on, forgetting to comment, document or try to simplify the line so that it's easy to break apart later when it has to be debugged.
Some of the best interpreted code I have ever seen came from a former sysadmin (it was powershell though, that comes with it's own warts of course).
The issue is that we collectively decided that sysadmins should just stop existing instead of allowing it to grow properly. We forget how bad things were for developers back when we decided we didn't like sysadmins. There were companies still passing around code on a USB stick because developers didn't want to depend on SVN being down. (I am not kidding).
Git was introduced 2005 and wasn't really everywhere until 2012, but 2014 sysadmins had all but been retired. This is the timeframe we talk about.
Developers of that time were as bad as the sysadmins were, the problem was that the developers had time and space to grow and we judge the developers of 2012 as if they had all the tools, knowledge and incremental improvements in workflow that they have today.
The sysadmins who were already automating large amounts of tasks for n Unix servers and k switches with p services, had a vastly different skill set and migrated to the more modern software defined devops SRe world no problem. But plenty of sysadmins basically could (and some still) only use a gui, and do things from trial and error. Those people might have retired or become managers, but they had a lot more issues with the transition. Which to be fair the transition happened fast and hard in some companies.
Anyway this is just one tiny example but the same underlying thing starts happening in all parts of the development cycle. “My deploy doesn’t work” … well your program has an error and crashes on startup but looking at the logs would be too much work. Let’s just say we are blocked and log off.
As you can see this is a culture issue but it seems to be common when you have a devops team. And even more common when management (read senior management and up) think that developers “should just be developing”.
Anyway thats my rant.
The only solution I see is to enforce commit checks before hitting shared environments, but that comes with its own set of complaints.
If they had no DevOps team to raise a ticket with, and it had to get done they would find some way to figure it out if they had the ability to modify and initiate their own pipelines. The reason I like "you build it you run it" is I feel like I have far more control over my own destiny. You're not having to negotiate and coordinate with a separate team through some ticketing system. Far less wasted back and forth and miscommunication. If that means you have to expand one of your teams with an additional DevOps person to foster understanding of the tooling on a team who hasn't done it before that's fine. But don't make them the DevOps person. Their goal is to enable the rest of the team to pick up these skills. Their goal should be to transition into a normal contributing member of the team working on product development because the team no longer has a need for a DevOps person.
People do not fix the pain they don't see, and managers doubly so. By shifting operations off to a "devops" team, you just end up with developer teams that don't know about, and don't care to fix, the things that make it painful to deploy or run their code in production. Those that have to deal with the pain rarely have any political leverage they need to even enact change. There's zero motivation for either the developers writing the software _or their management_ to fix it. Dev manager incentives are purely aligned with getting code and features built.
To give a nice real world example, building out new regions for AWS services used to be a pain in the arse (since I left I hear some major top down initiatives have fixed that). The service team I worked for, almost doubly so painful for region build. Standing up an entire region was something like ~50 days of engineering time. Not because it was especially hard, per se, but because the ops team always had to wade through piles of refactored shit that introduced new circular dependencies, or wasn't documented, or broke other aspects of any automation we could produce. (I remember one DynamoDB schema overhaul, where we did a complicated migration. Next region build, service components wouldn't stand up because we'd used the automation to create the new table schemas that had replaced the old table schema entries, but the service code still expected to see the old schemas in the right format. It took several months to get the dev manager to agree to engineers spending time cleaning up old schema references from their code base.)
The dev manager incentives were not aligned with making it easy. It wasn't their engineering resources being consumed, and they had features they needed to ship. The developers never changed the way they worked because even though they were told the issues (and even though they were, virtually to a person, great people to work with, considerate etc.), they never experienced the pain so they never thought about those consequences of what they were doing when they were doing it.
Ops manager never had any leverage, because every single time we managed to get a region launched in time as to not be an issue, and every time things were down to the wire it was clearly as a result of the ops team not building the right automation :eyeroll:, rather than that the dev teams were making some kind of brand new nightmare each time.
It wasn't until the ops team had lost enough members of staff to that BS, and region build shifted on to the developers themselves, that suddenly most of the problems went away, never to appear again. They were aware and conscious of the pain certain decisions would cause because they had felt it themselves, and they made sure not to introduce more pain.
DevOps is about making sure incentives and politics align to make things overall better. Shoving it in to a silo is literally the complete opposite.
But a team focused on making a streamlined platform with ready to adopt patterns that already account for compliance and security related concerns, they can spread DevOps adoption more effectively and consistently throughout the organization.
The so-called DevOps guys are stressed out of their marbles, but when the CTO calls me 4 AM I can't say anything but "Sorry dude, I'm not a clairvoyant. I have no permission to do shit in prod, call Ops or give me AWS access now". He stopped calling.
The fun part is Ops wants us engineers to help more, but they don't even trust (and some don't know how to run) my Terraform or CloudFormation scripts (I let them choose which), so they just read the tf/yaml and click AWS buttons manually. Often making lots of mistakes.
As it is, the Ops manager is probably going to have a stroke due to overwork, and it wasn't for lack of people wanting to help.
Yeahnah, you're part of the problem.
If you've not documented it properly, and more importantly, if the team are weary of prodfucking TF/CF scripts, there is a culture problem. You both need to understand how that works, if they don't have enough time, then you're understaffed, that needs to be fixed.
I worked at place that was going through a really painful ops->devops->dev oncall transition. Ops still had a shit ton of power, but nobody included them, and just shouted at them when stuff went red. It took a huge culture shift, and devs sitting in ops and doing a shift or two, to sort it out.
Everything I need from Ops is already enumerated and specified more than well in tickets (which is required, as is par for course with the sysadmin from hell stereotype), and documented in Markdown files within my app's source code.
I generate scripts for them since that's what they WANT and ASKED for several times.
They literally want me and other managers to help with DevOps (Heck, I'm even AWS certified), but don't trust me and others to give access. They ask for scripts and I comply to save their time. How am I on the hook if they want me to create scripts, but some of the new hires don't know how to use Terraform/CloudFormation?
> "there is a culture problem. You both need to understand how that works, if they don't have enough time, then you're understaffed, that needs to be fixed."
The culture problem is Ops wanting to do everything and not trusting. If they don't want to "prodfuck", just review every script. Or perhaps test them in the staging AWS account that I pay for with my own money. Or just stop asking me to make infrastructure as code. Or suggest a fourth option instead of wasting my time and being disrespectful.
I can't change other people's behavior. I can only change mine.
If they are understaffed I can help, or they can hire more. Wasting my time and making me do pointless work is not the answer.
Ah my friend, you totally can change other people's behaviour. go on. hug an ops, sit next to them and work with them. show them a bit of love and make them feel part of the team. Think about it from their view, why don't they trust your scripts?
Why dont they feel empowered to spin up a staging area? is it a skills gap? fear? or do they not feel confident enough to ask.
It sounds like solving their problem will also solve yours....
I can only offer help, do what they ask and try to make their life as easy as possible... I can't hug people via Jira tickets. Yeah, they're that isolated.
Kind of like the idea of a scrum master, which was never supposed to be a role
This is why "you write it, you run it" is so important - the most reliable software is the software never written, but if something must be written, it's important that all incentives are aligned for it to work right.
Of course, this is only at the operational end - lack of incentive alignment is rampant in "product" organisations too.
The problem is in 99% not that the developers don't want to do correct code. The problem is that product/project management doesn't allocate the time and budget for clean development practices, proper QA, and refactoring because their metrics are revenue, releases/timeframe, or similarly useless/unsustainable.
Everything in this loop was misaligned and rigged against devs. The only power I had as was to quit.
Now that this is over and traditional Ops is back and called SRE, it is Product Owners, Support, Managers, Developers and CTOs screaming all at once at their team. Let's see where this will lead us.
In (enterprise) reality it will more than likely be a bunch of sysadmins and former “network/storage guys” who saw which way the wind was blowing and got a cloud cert. They really have no business forcing a particular infra pattern on application teams because they have no application level expertise and nothing about adding LUNs to ESXi, typing Cisco configs, or writing bash scripts to automate yum update gives you expertise in application design/architecture.
Then because they know nothing about application level concerns the PE team’s golden patterns will either be ignored or will be customized for every application to the point that they’re unmaintainable.
Note this is the enterprise view. It’s probably different at tech focused companies.
This sounds like a tale of woe. but. The fault is the culture, not the people doing the grunt work. Like devs, if you're used to nails, then you'll want to use a hammer for everything.
> gives you expertise in application design/architecture.
a sysadmin should know much better how your application runs in real life, and more importantly, how it behaves when it goes to shit.
They _should_ know more about all the different types of services (ie kafka vs NATS etc) because they will have had to set way more of them up than a typical dev. If thats not the case, you've had bad or poorly mentored juniors running your stuff.
The reason they should know, is because they have the pager to make it work at 4am.
Look, if I had it my way, and I was at a medium sized company, I would buy in 3 mainframes, in three disparate locations, set the lock step replication to "very yes" and make sure each dev used the local DB and use the job description lang, to define scale.
Pay the $1million a year to outsource reliability of DB and state. Because frankly most people can't program for either scale, redundancy or recoverability. (yes that includes k8s) let the hardware do all of that and save a bunch on staffing costs.
I work at a HUGE scale place, and yes there is a need for special sauce there. but even then not that much, and not enough to justify the massive churn in systems. Most PE is just cargo culting busy work. Stick to ECS and PSQL and just get on with making actual buisness features.
is my new favourite consistency setting.
For TS, install '@digitak/esrun' and run:
esrun file.ts
(no tsconfig, package.json, etc. required)
That basically turns JS into the async equivalent of python and is great for boring sysadmin tasks.
The name is used and contorted into some bastardized form so now we just throw the whole thing out.
Absolutely this. Instead of collaborating to solve problems we are wasting time on pet projects.
It's a fine approach, if you have a well aligned group of people working with the same understanding. The same is true for places that prefer to draw boundaries of responsibility in other ways.
DevOps? Oh you mean the developer experience people?
DevOps? Oh, the build pipeline people?
DevOps? Oh, the people who handle the cloud?
DevOps? Ah, you mean those folks who handle our observability!
DevOps? That's Katya, she's a developer but i'm not sure what she does except act stressed out all the time and slow us down.
The best solutions, whatever the methodology and avenue of implementation, are made better when the builders "live below the dam."
The incentive to build the best dam is when the party responsible must live below the dam.
The best metric is customer retention. Directors of customer support need to be on the phone with escalations, not just watching metrics. There's a leak? Take the complaints. Embrace the hate. Fix the damn dam.
Both the DevOps move, and its current issues, as well as platform engineering are organisational problems. Your organisation needs to be set up around this in order for it to succeed.
Notice that the article paints platform engineering with the same utopia like brush strokes that once painted "DevOps as a culture", only to be mangled in its handling and blamed for the problems that it brought about.
The amount of time I've spent fiddling with Terraform, Ansible, Kubernetes manifests, Helm charts, Jenkins configuration, GitHub Actions configuration, AWS IAM, and so on over the past few years is absurd, probably more than the time I've spent writing actual application code.
These tools have made deploying production quality infrastructure take a fraction of the time. These environments are basically already preaudited and verified. They’re also just much higher quality in general. It’s quite typical for different projects to share infrastructure in traditionally deployed systems but with devops we can have much better isolation with less deployment time.
Yeah, because that's exactly what it is.
It all just feels like a collosal waste of energy and collective resources.
I hope its not you. The whole idea was that there is a platform team responsible for providing a service or better yet the CSP does this for you and then you just have to architect as a platform team the things I listed above. It does not happen via magic. But the CSP platforms makes it certainly easier to provide a pre-audited version of the OS and tools.
You should be able to self-service yourself within say a developer like portal as part of an IDP as described here. Not figure out ALL of the infra yourself.
First there are not enough ppl with that huge landscape of knowledge.
Second you would have to pay a salary of 5 ppl i.e. 500k/y to make ppl willing to spend that much time on work.
I've been around for the cloud migrations in every phase, what we have today with Kubernetes is miles better than the maze of Puppet and other crap we had 20 years ago.
What happens when your Cillium Service Mesh starts misbehaving due to missed network packets ?
Or when you have to setup a resillient backup of a cluster that runs 30 000 pods on 300 nodes ? And upgrade.
Or when you have to provide a telemetry for highly distributed system using 5 languages on landscape spread over frontend and backend with three different db engines and two clouds ?
Kubernetes is easier to start but 10 times as hard to do at scale you didnt even have back then in OnPremise era.
Yes you are right there are a lot of ppl who can write and deploy „Hello World” apps, but thats about it.
Now we have companies paying thousands of dollars a month to host a simple site or web app, because now it's all in the cloud, no matter the apps function, while hosting customers assume about 80-90 percent of responsibility for securing and maintaining their app anyway. Containerization is leveraged to add a "backdoor" for individual admins, but that only serves to protect infrastructure, it actually makes a customer's individual app open to vulnerability that same as running your own instance if you miss vital updates or if there is a breakdown in the supply chain. We need to stop listening to the companies that market tools and be honest with ourselves... Security needs a better model than just adding new tools and entry points to the app chain.
There's gotta be a point where you evaluate things and tell yourself that there has to be a better and more affordable way than tunneling through 5 VPNs just to push updates for 8 disparate JS and PHP libraries and 5 containers. App and library updates are also far too frequent in 2023 as well... Frameworks need to ship instances with less features, and modules should be reduced to only those used and deemed most essential, reducing footprint is a key aspect lost on technological advancement, it is also a firm indicator of increased efficiency for app and library devs if you ask me.
Now with zero trust, devs literally spend a lot of their daily workload logging back in and re-establishing VPN connections due to timeouts. There's got to be a point where we develop a far more simple IT solution to all of this mess. Security is still getting compromised regularly no matter what is done. The solution also won't likely involve Ai at this point in my opinion, as security and complexity are assuredly not resolved by leveraging current-state Ai tools.
It's also important to note this is why PHP is still going strong, it doesn't require compilation, and is relatively easier than most other langs on learning curve to implement.
I literally built my IT career on making things simpler, and in explaining complex IT issues in human language to non-technical people. There is a lot of room in my field for growth due to the rest of the industry's constant focus on jargon and over-complexity.
Simpler solutions, conveyed and implemented in human language, will be king in 2024.
I would love this to be the case, but it seems that complex and expensive systems are too lucrative to everyone involved: CTOs get to manage big budgets and bit teams, devops people get paid large salaries to justify the CTOs salary and large service budgets, etc.
A key for me is convincing a customer to simplify goals and implement an MVP on a reliable framework, to plan versions, and to thoroughly test... Makes me a code cop, but there are reasons why these steps are in place for mission critical services, and if a customer wants to skip them and just deploy in careless "Twitter Dev Mode" I have them physically sign paperwork that states their accountability for the bad decision... When individual accountability is mentioned, they usually opt for the more stable & reliable methods of dev. Hah.
But then it morphed into "oh lets innovate with infrastructure" but the innovation turned into "lol lets just restart from scratch and ignore history" Anybody who used early k8s can attest to how un production ready it was for the longest time. It _felt_ like progress because it required a lot of (to dev eyes) hidden magic to make it work.
Now, we are back with sysadmins, but they are called SREs. They provide a platform, which devs talk to to deploy stuff.
Basically we're back where we were in 2015, just with more yaml.
The good things that have come out of devops has been the dashboarding and metrics tools. Its just a shame that everything else touched by it appears to be an infinite source of busy work (looking at you kubernetes)
(before anyone asks, I'm an SRE at a FAANG.)
The place it worked the best was where a team of sysadmins were split up and embedded with each dev team. each devop would then rotate onto another team every 6 months or so. This meant that _we_ had to document our shit for the next devop, but also helped eliminate key person dependencies.
We would then have a weekly offsite where we'd bitch and moan about our teams and conspire to make things better.
In the old days we had so few deploys and a hotfix was something that was extraordinary. I actually had a board of something in another country approving that we where allowed to deploy. They knew nothing about the application..
yes this server is special to me. No, I cannot move workloads somewhere else and shoot the server in the head. Because customers are running GPU workloads there, I cannot transparently migrate them
What I hate is the attitude of "treating your server is a pet is wrong, you should be able to kill it and replace any time". I feel like people who say it never run anything more complex than an HTTP API.
My (Raspberry Pi) home server is very much a pet, but I'm never going to productionize that thing, so I'm totally fine with it being a pet, despite the practice at work of having no pets, all cattle.
I imagine there are SREs who enjoy it and stay there, but curious why they’d invite someone heavy on the dev side to interview.
Ironically they realised this wasn't a good idea in exclusivity, so now they hire two disciplines: SRE-SWE and SRE-SYS.
SRE-SYS is as you would expect, pretty standard sysadmin knowledge from 2005 (IE; not helpdesk sysadmin, some scripting knowledge, heavy OS knowledge, heavy debugging knowledge).
SRE-SWE is much more programmer heavy with a light touch on OS fundamentals (based on what I consider important for OS knowledge FWIW).
the downsides of SRE is that you tend to be super hyper specialised in something thats only really of any sense to that department inside a company.
Moreover, you're dealing with real steel machines, and amounts of data that are prohibitorily expensive to just start again with. (the joy of AWS is that most of the times you can delete it all and spin it up again, often side by side)
also, some SREs are really specialised software devs, others like me are sysadmins who'd been swept by the current into SRE roles.
My company SREs are not like google's, where they are rock stars swanning about doing one line fixes that make everything .48% more reliable. We are computer nurses making sure that the wards of systems we look after aren't wiped out because a doctor forgot to wash their hands after taking a poo.
However, they both require the same skills: murder mystery and the ability to dive super deep into the stack, go from python->c->assembly. or service->VFS->cache->network->time weirdness
Also it could just be copy pasting service defs and making dashboards, then pinging devs saying "shits on fire yo"
Admins wanted to increase their $$ and started calling themselves DevOps
And developed some automation to justify that
</conspiracy theory>
Although sadly in most cases the automation was harder than the original service to look after.
It's done a great job to enforce consistency and immutable infrastructure in a way that is more approachable for most when you get past the scale of a few resources.
I've seen start ups write their on bash tooling, watch it devolve into some opaque thing that no one knows how it works or what it does. I've seen platform teams devolve into managing VM updates by hand, babysitting operating systems of every application deployed because it's treated as a long living pet, and it gets worse when team leads move on and then someone else comes in and half bakes their own flavor of change. Even better when networking gets involved, load balancing with HA where people roll their own solution. On top of that is the certificate story and how that gets managed. Don't even start with RBAC from a platform perspective, it's a nightmare. The toil filled platform days should be behind us. Plenty of companies are slow to evolve and you can find a lot of pain.
Perhaps I've had bad luck in my history of being a network engineer, infrastructure engineer, and an SRE. The one big benefit to these tools that are coming out is it can give some sort of standardization and it's giving you a ton of power in abstraction so that platform teams aren't micromanaging things that are just sorted for you.
Something like Kubernetes gives you load balancing, high availability, RBAC, networking and routing, resource management, node draining and rotations, auto scaling. Sure it's managed in yaml manifests, but it's doing a lot of lifting. I won't say that the abstraction can't bite you, you still need to understand the core concepts of things. However people should be understanding those concepts regardless.
The amount of work that takes for a small team at a startup that needs to scale services is immense without it. Right now I'm working with a 3 man platform team and we are running over 200 services in multiple regions, multiple database technologies and analytics stacks, we auto scale aggressively and we serve over a billion requests and terabytes of traffic a month with over 99.99% of uptime. We also deal with operations, security and every other hat that someone wants to throw our way. The metrics solutions to come out of this are also such a huge win, I remember the days of Nagios, Zabbix, SolarWinds etc. They were mostly static graphs showing you snapshots of what happens at a very high level, now we have metrics for everything and you can feed that into anything to self-remediate or escalate. I absolutely love it and can't wait to see it evolve more!
The previous iteration of platform at this company was all self-managed VMs with the old school sysadmin mindset. They were managing hundreds of VMs by hand or using some light ansible that was retrofitted to work with pre-existing machines, we were using corporate F5 load balancers and VMware which are immediately out of date the moment you deploy them. Some older sysadmins were legitimately afraid of rebooting servers because they were afraid they wouldn't come back. Some machines had over 900 days of uptime because they just couldn't keep up with the toil. Also this creates a wall between the platform and developers because it turns into a hand-off process where developers throw code over and the platform team has to implement some server and load balance logic, let alone meeting SLO/SLI criteria. Sadly, that's not uncommon to any industry. This team had 15+ people and had trouble keeping up let alone deploy and innovate. Outages in this mode are unavoidable, and pulling away from that old school mindset and starting over is VERY hard.
I'm sure a lot of that can be contributed to bad management and bad hires, however from my perspective using tooling and s...
unless you are running real steel, kubernetes is at best a complication layer over the underlying cloud tools. In 2018 me and my team built a real time, ML-based processing API. It was hugely scalable, optimised for speed, price and uptime.
At no point did we use need to use kubernetes. Not only would that have eaten most of our innovation budget, all it would have done is add latency and driven up the money we spent on inter VPC transfers.
> I remember the days of Nagios, Zabbix, SolarWinds etc
So do I, they were recognised as shit even in 2015. $work allowed me to lead the migration away from them and by then we were half way through replacing most of splunk with graphite/grafana based metrics instead. The rest of the alerting was mostly handled by a custom inhouse /__status end point parser.
> The previous iteration of platform at this company was all self-managed VMs with the old school sysadmin mindset. They were managing hundreds of VMs by hand or using some light ansible that was retrofitted to work with pre-existing machines, we were using corporate F5 load balancers and VMware which are immediately out of date the moment you deploy them.
That, again, is just bad sysadmin. Unless you have complete control over your stack with ansible, then its not ansible, its just sparkling bash script.
I worked at a VFX company that deployed to all real machines for virtually everything. They used a GNU configuration system that was older than SSH (I can't remember what its called, it used to use rsh, that's how old it was.) All the machines were fungible, We could replace a file server at the drop of a hat (the only limit was how fast we could copy 160tbs over 2x10gig nics) Most of the place ran on perl, in one git repo for just configuration. OS updates were applied weekly, and we pushed system changes daily. This was in 2013.
salt made all of that easier, especially as we had way too many hosts to sensibly run ansible at the time(I think we worked out it would take >1day to run a basic role).
They had automated it to the point that you could plug in an unconfigured blade host, turn it on, and within 30 minutes, the bios was updated, networking configured and each blade was starting to be installed.
Automation is an attitude. You can either invent new tools to solve hypothetical problems badly (k8s supposedly is high scale, but cant practically get above 1k hosts, its also balls deep in ipv4 ), or actually use/create tools to solve the problems you have now.
Maybe this is "old man yells at cloud" me but I miss 2015 DevOps.
If you knew Terraform, Ansible/Chef, a monitoring tool, a cloud (usually AWS) and a programming language (usually Python), you were all set and could still be idiomatically/culturally DevOps.
And I say that as someone that knows Kubernetes very well
The problem is: when you give developers access to Kubernetes, to ECS, to Portainer, to whatever they. will. not. care. about literally anything. You'll find a hotpot of cobbled together Dockerfiles with zero provenance, with base images pinned to stuff years old, and completely inefficient layer ordering. Pipelines won't use caching or parallelism (okay, sometimes because they use Maven which is a clusterfuck on its own).
Software developers are developers, 90%+ will never have heard of how Linux works below the hood, how Docker actually works, they'll just copy and paste together stuff from Stackoverflow without even attempting to understand how it works.
And I don't even blame them. I can't, I don't want and I won't.
Companies, it's time to stop loading stuff onto your developers that they don't have to (have) any clue. JFC.
Actually making something that is simple and maintainable is the 7th or 8th priority of business, if it is a priority at all.
Artificial "obstacles" to delivering the artifacts in time in fact, wanted, a sort of corporation Deus ex machina, externalities to explain why they did not meet delivery deadlines last time.
Incentives are not aligned. Things will remain broken until incentives are aligned.
Same applies to me, "knowing" GNU/Linux since Slackware 2.0.
The difficult paradoxes have not been resolved
No-one can say this loudly because there is no safe space
For a given level of complexity in an organisation, talk to any 10 experts on the subject in the organisation (ie how does the foobar process work) and you will find blind men describing an elephant. All technically correct but all missing important parts
The paradoxes
At some point an organisation acquired or grew an antogonistic part. Both cannot really exist but both have not been removed. Think either Google search vs Google ad sales (if you return perfect search terms each time, you get less page views hence less ad impressions. Make search worse get more ads served). Or simply other non core businesses. Pretty much every non-tech mega corp is really dozens and dozens of smaller businesses with a shared treasury so they can ride out each others ups and downs. But one or two businesses are really carrying oat of the load.
Anyway, you can find paradoxes or conflicts of interest in all businesses and badly aligned incentives hurt everyone.
Someone should fix that. Not me I don't have the power (indeed usually only the orignal centralmfounder has power to)
Safe spaces
Yeah these don't exist. If you ain't upbeat publically you are seen as trying to pull everyone else under. So how are problems acknowledged let alone fixed? Dunno.
What would really help would be free speech, open conversations and discussions based on data that can inform policy and provide impetus to chnage. we (kinda) have this in western journalism / media (including social media). But absolutely no large corporation anywhere has a open discussion area where different factions can raise their concerns. there are of course political infighting amoungst the powerful elites of the corporation but that's not the same thing - they might even be accurate about the problems but no one else knows.
Democracy matters
Sorry this was supposed to be about devops - and it kinda is. Most large corps have some devops thing going on where "this time we will fix it all for everyone" - but has there been open conversations with data supporting it from all parts of the org? oh come on.
I realize this is a bit of an odd tangent, but I would love to hear cadey talk more about what "devops" means at a shop like Tailscale. Actually, as I type this, even more curious. I know some of those folks are cut-me-and-I-bleed-Go types, but I also know they're deep into NixOS for server management. It would be very interesting to hear more about how those intersect for their infra/server/dev culture.
I'd add that most of the 1-4 "problems" (boo-hoo, managing IAM is boring for devs) OP mentioned as well as advent of so called "platform engineering" stem from one thing only - poor engineering leadership.
I fully stand behind the "just have programmers that write internal tooling" recommendation though. Over the course of my long career this is what has worked best almost every time, and these are the programmers who get the least amount of respect even though they have to dance and juggle on a burning rope more often than not.
DevOps pipeline languages are generally either weakly typed or stringly-typed. They generally expect users to do the control character escaping, in several different obscure formats. Did I say escaping? I meant three layers of nested escaping!
Variable/macro substitution as a rule uses syntax that overlaps with either PowerShell, Bash, or both because otherwise it would be too easy.
Another basic thing is the absence of a debugger — I have to run a complete pipeline with a change to dump out environment variables!
Instructions contain the display names of paths, scripts use the variable names, but the output (and errors!) use the values with no clear mapping backwards. What the eff is C:\s\1\x? I dunno, but it caused an error!
The worst sin is not having a local development experience where the entire pipeline can be run without having to commit code and push stuff almost all the way to production. Even if the pipeline agent is made available, it’s useless without the underlying VM image which is huge and not easily obtained. Not to mention KMS/KeyVault and other similar services on which pipelines can depend.
Developers complain about having to do DevOps, but in my opinion the problem isn’t who is doing it, but how. The tooling is just so bad, so very very bad, that anyone would hate it if they were the ones forced to do it. I’m an ex-dev-now-SRE and I hate it.