Equally common I think are engineers who care where and how things run, but are either disincentivized from or even disallowed from working on those aspects of systems.
It is more nuanced. I care, but my skillset demands I spend more time on the Development side, so Operations must take the backseat in my mind because of this incentive structure. Asking one person to do two (actually three) job functions is a scam.
Most developers have to do Frontend, Backend, and Ops. These have wildly different mindsets and feedback loops and not enough time exists. Don't hate the player, hate the game. The orgs are fucked, not the workers.
Thank you, yes this is the perfect analogy. I think every developer should have some kind of baseline understanding of devops stuff. But it’s a whole field in and of itself, and is also innovating fairly rapidly.
There are only so many hours in the day, it’s hard enough to stay on top of my core skill set.
She doesn't impose multiple business-critical schedules on that, so you can just wash them, pick your nose for a while, take out the garbage and go play games. There is nothing wrong with doing related jobs solo when situation allows, but this example is also far from learning and using "modern" full-stack of stacks or doing surgeries.
Well I didn't have to get an education to do that and if I mess up washing the dishes I didn't cause a death in the case of a doctor or a multi million dollar outage in tech.
That is an extremely bad analogy. The purpose of chores is to spread the load onto an otherwise unproductive member of the household. (And to teach said household member how to do them.)
Unless you're arguing that engineers are unproductive members of a company, I suppose.
Race car driver vs bus driver. Race car driver knows how their car functions and works with engineers/mechanics on improvements. Bus driver don’t give a shit (no offense to bus driver). Decide which company you are - a race team or a bus line
I will extend this analogy to say I am a race car mechanic but I'm having to pick up how the driver drives the car in order to make the car perform better, with the eventual goal of being able to drive the car myself (ops -> devops -> dev) so I can do any of the above roles.
Software devs who close their mind to the lower parts of the stack give me the opportunity to learn and do their jobs as well, becoming a more valuable employee as a result.
I think the argument is that calling ops different is like saying error handling is another department. To be a complete dev you must understand the code in situ regardless of your specialty.
> Asking one person to do two (actually three) job functions is a scam.
The word "scam" is overused. It's a scam if it was a bait-and-switch. It's not a scam if it was advertised this way up front. In the latter case it's just a job whose description you don't like.
It is a scam. It should absolutely be assumed that you could perform these job functions - it makes you a way better engineer! But being expected to is a scam because the skill ceiling on each is so amazingly high that only a scam artist would expect someone to be able to perform all three to a specialist level. The same issue happened when QA departments got the boot. I deeply understand testing strategies (both manual and automated) but this is an important enough job that it deserves a dedicated person. Not having this person is exclusively because the business is trying to save money and make the worker easily replaceable.
It is not better for the department members or the software itself. It is only good for the bottom line. Yes, if you know it all you'll be valued higher - but the company is still reaping the majority of that benefit from you and not rewarding you with 3 salaries. You should not perform all the roles because it's economically idiotic to do so. You could contract out all 3 services and make a shit ton more money than you would performing them in a salaried position.
I perform all three roles for one company. I agreed to it upfront. I do it in less than 40 hours a week at a very respectable salary. I get maybe 1 after hours call a year for an Ops task. My boss knows that I won't do QA or Ops nearly as well as I do Dev, and he knows that QA and Ops time come out of Dev time (not personal time) and he's okay with that because that's the stage we're in.
If you think that is a scam, I don't know what to tell you.
I don't understand. I want to develop code. I don't want to become an AWS/S3/Github/Jenkins/Action/terraform/etc expert. I know enough of this to be dangerous but not at a level that passes as professional. Yet I am regularly tasked with maintaining the full deployment of code.
There's a reason to have a team of people doing this "DevOps" work. Just like we have a team of people who do SRE. It creates a standard and a single point which all work flows through. Then you don't wander onto a new project only to realize they use $BESPOKE_DEPLOYMENT_METHOD because "it's what we used 6 months ago". Or worse, you don't have a developer playing with a massive, nuclear powered, foot gun like Terraform and accidentally destroying infrastructure.
Making DevOps/DevSecOps/$BUZZWORD the responsibility of developers is a cost-cutting measure not a responsibility measure.
It just doesn't work for the most part. Maybe as an Ops person, I just want to do Ops, I don't want to have to understand your code, but Lambda has specific limitations on how long code can run for. I can't allocate CPU and Memory resources in Kubernetes without a deep understanding of the application. S3 has limitations on how files can be distributed and accessed. Integrating with CI gets complicated quickly and requires understanding the code being integrated. A lot of the time building things in terraform I spend three times as much time getting the information out of a developer as they would doing it themselves. I mean yes, there's 60% or so of my job that involves working on lower level infrastructure that doesn't touch a dev, and we do need ops engineers for that, but the other 40%? It works both ways, a good "devops engineer" needs to understand code, but if we don't have a shared language we both just end up banging our heads against the wall.
If you're building a crud-app in a common framework with low volume, sure you can toss that over the wall.
At companies I've worked for where they actually do this the DevOps people are generally assigned to a team. So you might have 10 developers on a project, and one guy managing just the devops exactly to solve this problem. In smaller companies/etc I can see where this is a problem. I agree, and there's nothing wrong with knowledge sharing. This could even be as simple as having PR descriptions including benchmarks/code/etc and making sure the devops people make their points clear during design/planning.
In theory, devops shouldn't be responsible for maintaining the performance of code. The specification should say what it should run on, the devops guys set up a pipeline and manage that thing, and the developers are the ones taking heat for not hitting that goal. If devops guys are taking the heat for that it sounds more like cost-cutting measures flowing the other direction.
Yeah and this is a great solution (aside from the bus factor), maybe the person I responded to was really concerned less about knowledge and more about expertise (the word they did use in their was expert) At a company I worked with we called this T-shaped engineers. Deep in one thing, but broadly knowledgeable. Devs have to have knowledge of ops, but not "expertise" that is ultimately what ops is for, we may just ultimately be fighting over where the knowledge line is sufficient and what constitutes "expertise" :-). I for instance think terraform is not that much of a footgun and provides good rails to be used by developers.
I think you make a strong case for ops who can dev, but a fairly weak one for devs who can ops so I think you and the parent are actually agreeing. And this mirrors my experience pretty well, I need to know the code to be able to ops effectively but it's much rarer that devs need to know how to ops to dev effectively.
And in some ways this is by design, I want to have some distance between dev and ops because it gives me the freedom to rearrange infrastructure transparently. I can move workloads between Lambda, ECS, and EC2 based on the observed performance characteristics without anyone being the wiser.
This is going to depend on a lot of things. Size of company, cloud native or not, org structure. I mean everyone would love to live in the google world where a team of SRE's run everything. But even in the world where a Devops engineer is embedded on a team there's bus factor to consider.
I think most modern AWS services are more the equivalent of an API or a microservice than they are a server, and you need to understand the limitations of the services you integrate with. If you're a cloud native company and don't have a mature platform engineering team, devs are going to have to know alot about AWS.
If the developer has all the information necessary to create an S3 bucket, lambda function, kinesis stream, etc does it make sense for them to offload 10 pieces of information to me, or for them to learn HCL and interact with it themselves, especially if it is something they do often. Especially if there's a central dev/ops team and they're the limiting factor.
Devops taking over every infrastructure change for a broad team of devs is inefficient and expensive. It's also probably frustrating for devs that are aware of the above factors. Lots of devs I know would prefer to do it themselves.
And all of this is to mention that infrastructure and integrations with infrastructure are not static. Should I be reviewing all PR's to a system that touch the client code for an AWS service, because the dev team doesn't want to learn how that service works? Maybe, but in the end I don't know if this makes anyone happy.
Platform engineering is certainly the goal. Where ops creates a platform and dev just consumes that platform. But I don't know if it is realistic. Every system on rails is great until you try to take the rails off, and most devs I know hate rails :-) Fundamentally "devops" is all meant to solve human problems, not tech problems so it will have to be dynamic.
Our devops runs the infrastructure, but details like "what resources do we allocate where" are done primarily by the software devs. I don't really see the conflict here. I don't know and I don't care about how to put together the infrastructure required so I can change the CPU allocation on a Kubernetes pod, but I also don't expect devops to know jackshit about our code.
This makes me wonder if ops allocating compute resources is really a good use of time if you're needing precise details of an app (which can and do evolve). This isn't a slam against ops, either, it's a knock against the tech itself that it forces all this incidental complexity on you.
Yeah, I mean, fundamentally it's so complex because you have to make tradeoffs and people hate tradeoffs
"I don't want to have to worry about what machine my app runs on" vs "kubernetes is to complex"
"Dependencies change to often" vs "I don't have time to maintain this thing I wrote myself"
"I just want the infrastructure to figure out what I need" vs "I want to be able to build whatever I want with a bespoke language/framework/database/architecture"
> it's a knock against the tech itself that it forces all this incidental complexity on you.
If I were to say that Kubernetes is the magic secret sauce that fixed all the incidental complexity I would get laughed out of the room. There is no magic secret sauce to the incidental complexity, the more we try to fix it the more we create (or the cheap-fast-good problem. This is probably easy if there are no cost limitations)
Alan Kay said that “Everyone who is serious about software should make their own hardware.” How can you be a good developer if you don’t understand the architectural limitations and choices?
When I design a backend system, I need to think about how the front end developers are going to interact with it. My data storage characteristics and scaling. I need to know am I designing anything that’s hard to deploy. How will logging work and be aggregated. I have to be able to think about the entire system.
It’s not just “cost cutting” at a certain point in your career you are expected to know more than just “how to code”. I’m not saying learn AWS. But I would expect any senior developer to know about what their code runs on top of .
I don’t think you’re disagreeing with me. Developers should know what their code runs on. They shouldn’t have to add managing that to an already full schedule of work. That’s the difference.
Back when I was in the real world [1] working for a startup, I would do your typical serviceless solution with Lambdas, S3,SQS, etc. I couldn’t just use ClickOps and create everything on the console and expect someone else to recreate everything with IAC. I had to know how to do it.
I think to push back is rightfully coming from the “ops” part. I consider “creating the CloudFormation/CDK/Terraform” code as part of “development” as part of coding.
If you use Docker, wouldn’t you consider creating the Dockerfile as part of development?
Yes I knew AWS pretty well by the time I left and I needed to know it to be a good developer in that context and designed most of the processes around it. But I refused to do “operations” - ie “infrastructure babysitting”
There is a huge distinction between “I don’t think I should have to know how everything works” and “don’t call me in the middle of the night when something goes down “.
[1] I’m the first to admit that I left the “real world” once I started working in the cloud consulting department at $BigTech
> I think to push back is rightfully coming from the “ops” part. I consider “creating the CloudFormation/CDK/Terraform” code as part of “development” as part of coding.
> If you use Docker, wouldn’t you consider creating the Dockerfile as part of development?
Sure you could argue a developer could, or even should create these things in theory. The problem is then when it goes down I have made two problems out of one. Now I have to manage the infrastructure of a system and what is running on it. Realistically, and even in my current job, it's actually several systems. Now when something breaks I have to pray I can fix it. Instead of allowing a team of infrastructure professionals to at least insure the hardware is working my 8 hour day turns into 14 or 16 very quickly the second one thing goes wrong.
So if I need to create a bunch of Lambdas, queues, sns topics a few dynamodb tables an S3 bucket, etc and tie it all together, are you proposing that the developer should just create everything in the console and then call over someone else to go behind me and write the infrastructure as code?
> Yet I am regularly tasked with maintaining the full deployment of code.
Same shoes, but I have different perspectives. I can figure out where it's not working and if it's my/our team's area of responsibility then we go fix it.
Since we handle infrastructure (as code) and deployments in team, along with all the development, most of the problems are handled by us. Unless it's clear that it isn't, e.g. some API that we consume that keeps throwing 500, can't fix that.
Our operations is helped by 100s of automated tests and 1000s of metrics. I always thought that this is DevOps, but it sounds different from what most of the people here are alluding to.
> I don't understand. I want to develop code. I don't want to become an AWS/S3/Github/Jenkins/Action/terraform/etc expert.
They’re tools for the job, like your compiler and the language you use to program.
That’s like saying “I like to use Python and couldn’t care less for Java”. It’s fine to disagree with the team’s choice of the tools, but one needs to eventually commit to the choice, even it’s not your preferred one!
There’s an old adage that “if you’re writing clever code then you may not be clever enough to debug it”. It’s true! Operations requires deep understanding of the code running in production, the business rules and the customer. The person who will operate your code will eventually be smart enough to develop it entirely too, eventually cutting out the “dev”. I’ve personally seen this happen over and over.
> Making DevOps/DevSecOps/$BUZZWORD the responsibility of developers is a cost-cutting measure not a responsibility measure.
My background is large multinationals so my view here is a bit bias but i don't think cost cutting is the driver.
Large orgs get large change management processes and procedures. Over time, these change management teams become overwhelming behemoths with minds of their own.
I think "DevOps" was designed as a way to "bypass" the bureaucracy?
"We just use this CI/CD pipeline and no need to sit on a 3 hour change management review call..."
Almost everywhere I've worked that I helped run software in production for would have a step in CD which filed an automatically-approvable change req with automation; just like the automated deploys.
It becomes just robots pushing around paper for compliance.
DevOps came about because of developers who "just wanted to write code". They would write something, then throw the dead cat over the wall and say "you figure out how to run it". That... doesn't really work. Somebody needs to explain to the Ops people how to run the code. Hence: DevOps... a way to get Dev and Ops to avoid throwing dead cats over walls.
If you don't want to think about AWS S3, GitHub Actions/Jenkins, Terraform, etc, .... then we need to work together. All those tools and services exist because all the software developers are sitting in their sandbox, and don't want to come out and play. The systems and tools that we run your code with... suck. A lot. We need programmers to make the systems better. We (in Ops) are a little busy with trying to just figure out how to run your apps without them falling down. We don't have a lot of time to reinvent the state of the art of computer systems.
For example, we need a distributed operating system. Not some fucked-up kludge of a monolith of microservices overseen by a company that has more engineers than brains... but an honest to god, stable-ABI, simple, composeable, stable, general operating system. We need Linux to come out of the box, ready to run distributed applications, in a way that doesn't require a PhD. Once we have that, then you - yes, you, the developer! - will be able to make applications that automatically scale so easily that we will never need to utter the phrase "container" ever again. You will rarely ever need us again, because the system will just be so simple, so general, that anybody who can use the terminal can build and deploy applications without ever learning anything outside of your programming framework.
But we need you to make that distributed operating system. Until you do, we will just have more stupid kludges, more bizarre unnecessary complexity, in the futile attempt to constrain all the crazy shit we want to do with technology, while trying to run your apps for you. Please, I'm begging you - put me out of a job.
Well what about Google Sheets? Lots of people are able to build things in Google Sheets without knowing where and how it runs.
Perhaps most of our organizations are using the wrong tools and that the way we divide responsibilities is the problem.
Imagine if:
1.) instead of general purpose languages being used by product developers we had domain-specific languages and tooling that were agnostic to things like deployment, memory-management, data storage
2.) computer engineers built the domain-specific languages and tooling and focused on things like deployment, memory-management and data storage.
DevOps is generally solving for general purpose development while developers are generally solving domain-specific problems with general purpose tools. This seems to be a large part of the disconnect!
I’m currently working on a small team of computer engineers who support a custom Turing-complete DSL for an “operations” team that both programs and directly interfaces with clients.
It doesn’t seem that different from what Apple or Microsoft do, which is write custom languages and tools for developing on their platforms, albeit specifically for general purpose programming.
I've done some, and it feels like reading an arcane text in another language. I've seen the thing I want before, so I know there exists a magical YAML incantation that brings it about but I have no idea how to even start looking for it. It devolves to an extremely slow feedback loop of "did that work? No."
I've never found anything that explained how I'm supposed to create a systematic approach to creating anything I need.
Sometimes the engineers _do_ care about the deployment and operations of their apps and management doesn't let them administer them in production under the guise of "separation of responsibilities" or "regulatory compliance". It makes the ops team unhappy that they're on the hook for stuff they didn't build and the dev team unhappy because they're steering their metaphorical ship by proxy.
I'm not sure why this happens but it does and I don't like it (as somebody who's very M-shaped).
It's because often your software engineers know software development and just enough administration and operations to be dangerous, and your administrators know operations and just enough software development to be dangerous.
Letting them wade into others responsibilities often introduces a lot of that danger.
System administration is an entire discipline with a history and way of thinking and approaching problems at a systems level. Software engineering is the same, but with an approach that often emphasizes handling problems in the application.
Letting solutions be put into applications that are best handled at the system level (where system isn't just an OS, but might be a complex network of them), or put into the system when it could easily be handled by the application, caused inefficiency and problems.
The solution is to either hire only very accomplished people that can do both and make the right decision of where to put solutions, or hire people that can do one or the other and put them together with a few (maybe one) person that has a good grasp of both and can make executive decisions when the right time and place is for specific solutions (an architect).
Yup. I basically came here to quote the same line answering with "And I don't want to work with them."
Though mind you, the article touches - but doesn't go into - the real reason things go shit. It's not ops, it's not devops, because your knowledge silo can too, and in my experience (that's longer than the writers, lol) _will be_ pressured.
If the estimates are consistently too low (because honest ones would render many - maybe most - projects non-viable), the requirements are often bullshit (because otherwise the lies in the estimate would be too obvious, and everyone needs to agree on the lie) and and rewards are not the ones the jobs claims to be (in every workplace, your job is to improve your resume. That's all. That's the only thing that, materially, matters for your career. I can care about software I write, but if anything I end up punished for it) - then no amount of renaming things and shifting org charts around or motivational reading for managers is going to fix that. Nor will a Cloud or latest orchestration software.
You'll spread yourself too thin as a developer if you need to do a lot of ops too. Specialisation is a good thing.
You don't want a doctor who brags that she can do heart surgery, perform talk therapy and treat your skin rash. You want someone how knows a lot about your specific problem because she's prioritised it over other things.
You mean people relaxing WFH in a unbelievably great job market dont like being on call 24/7 and supporting random stuff they didn't build. Sounds like we need a recession.
> You have people who build stuff without caring where and how it runs.
In the old days, developers would also configure the design of the system it ran on.
"Alright, we're going to use some fast CPU front end nodes with smaller, faster disks, and then have a load balancer, our database server is going to have the most RAM, now let's set these resources up on a CDN to serve faster, this memacache server is really going to lighten the load..."
I think the idea was to use tools that automatically did all this work for you, but it just ended up creating another tier of people that configure this side of it.
It's always been like this. I'm a backend engineer that also does cloud operations / systems administration / network engineering / "devops" work. The job titles have changed, but the truth remains: (most) developers want to develop, not deal with infrastructure.
Agreed. I get a rush from designing a system from the ground up and bringing it to life with all the infra, but only when that's necessary. If it's just a basic CRUD app without any major performance or resiliency requirements, I'm more than happy to do as little of that infra work as is necessary to get it into production and move on with my life.
I absolutely care where and how my code runs. And yet I don't give a damn about writing terraform, managing IAM, VPCs, keeping up with backups, upgrades, etc. I want specialists who know what they're doing to deal with that stuff.
I'm also not convinced that the average junior or mid level application dev needs to know where or how their stuff beyond some high level concepts (i.e. our app uses auto scaling).
Previously we had better division of labor IMHO. 1) UX people (Photoshop, CSS) who made the front-end look nice, 2) Developers (JavaScript, C#, Ruby, Java) who wrote application code and business logic, 3) DBAs who controlled the database, and 4) Sys Admins (Unix, Windows Server) who deployed the built code and ensured it ran correctly and securely in production. Now some of us seem to think this should all be the same discipline, and if you don't care to, you are a bad engineer.
But isn't this the norm in many other fields? The engineers who design automobiles, aircraft, kitchen blenders, etc. aren't expected to maintain and repair the cars, planes, blenders. etc. after they're put into use. They go on to design other products. Architects and construction/civil-engineering firms design and build large structures and then hand them off to others to maintain. Authors generally write a book and then go on to write the next book rather than dedicating themselves to carefully curating a list of errata for a single book. There are academic authors who do play a part in churning out new, slightly modified editions of major textbooks every few years but they're not on-call and being woken up in the middle of the night to fix a punctuation error on page 982 of the physics text they co-authored.
So a symbol i.e. "DevOps" gets corrupted. What do you do? Make a new symbol, like the author is suggesting? Or try to reclaim the term? This happens time and again and I find the question fascinating.
FWIW, "Platform Engineering" used to be something different than what the author is suggesting.
I’ve long held this opinion but I consistently get drowned out.
DevOps has different meaning depending on who you’re talking to, even some definitions that appear similar are different in nuanced but important ways.
All “devops” as a job title has done has muddy responsibilities and given many folks the wrong impression of what an operations discipline should be.
There is also a lot of rewriting of history that gets thrown in, similar to how when people talk about cloud then the only alternative is to start making CPUs by hand and begin building your own nuclear reactors. It’s the idea of what came before, not the reality, that people seem to be defensive of.
It’s honestly exhausting to discuss.
So instead I became CTO so I can solve this mess properly, I don’t hire devops, I hire infra engineers, build engineers, release engineers and: backend engineers.
Roles so simple that you already have a clue what they do, which is sort of the point of job titles.
Agreed, anytime I talk with someone about DevOps ... we end up having to hash out the entire process to really know what either of us are actually talking about. Otherwise you have these situations
"Yea the DevOps guy messed up the widget and nobody notic---"
"Wait, what is the DevOps guy doing even touching that widget.... what is even DevOps to you?"
I feel like it has its place in large organization. But being a large organization means the odds of spaghetti are approaching 100% no matter what you do.
What is your definition of Spaghetti Architecture? Netflix had a good SOA that enabled rapid development and had strong cut lines between services, with no way to access the data of a service without going through the service's API.
I think that's where most people go wrong. They put a bunch of services in front of a shared database, which means that they don't have to go through a service's API to get to it's data, and that's what breaks everything.
This is really important, and I've gotten a lot of quizzical looks when making this assertion over the years: data is owned by one and only one service. If two pieces of code assert ownership of it by mutating that data or looking past the public encapsulation of that data, then that code is the same service.
If you see a queue between two services, that is usually an indication that some ownership is being transferred (even briefly), and that is a critical point where you need introspection and observation.
Yeah, services should be formed around the needs of data, and you shouldn't run multiple services in the same image/container just to split the code up. It shouldn't ever be about the code.
Amazon's SOA architecture at least all started out about pulling bits of data spread over hundreds of servers into services that could take advantage of caching.
> An important rule for microservices architecture is that each microservice must own its domain data and logic . Just as a full application owns its logic and data, so must each microservice own its logic and data under an autonomous lifecycle, with independent deployment per microservice.
The big issue I've seen is that takes a lot of work, so people cut corners. The problem is of course when you cut corners with microservices and rely on a shared database for instance, suddenly you're dealing with 40% of the costs of a microservices and 0% of the benefits.
Shared database can be a reasonable microservice boundary, especially when using database-as-queue or database-as-mucroservice. In the former, services a, b and c can insert and query but only D can update, i.e. any service can creat a work-order but only D can mark it completed. I. The latter, nobody can read or write and all access to tables is done through stored procedures.
I don't recommend either, and it's still a code smell, but with clear definitions they can work.
The benefits of the microservice pattern are you can build separate teams responsible for different business logic, and they can have their own deployment schedule. You don't get either of those benefits when a b and c have to coordinate on their work-order creation business logic, and a, b, c and d all need to be deployed at the same time anytime the schema changes.
Btw some of the confusion might be what I mean by shared database. I didn't mean two services sharing a limited set of tables, using the database as a rabbitmq replacement. I meant sharing the backing database of the microservices. It sounds like we probably agree I just wasn't very clear about what I meant by sharing a database.
(Ironically I am literally using a database as a queue to share data between two services we are running in prod. I don't think of it as a microservice because it's not separate teams, it just because our monolith is hosted on a platform that doesn't support certain libraries so we had to role those libraries onto a separate platform that does support them.)
I think we're in violent agreement - A database can be a queue and that doesn't link things as the same, but it's easy to break those promises and you need to be clear in that promise to begin with. If two "microservices" have to move in strict lockstep, they're not microservices, they're components of a larger service.
That's completely untrue and glosses over the very real costs of transaction management in such an environment.
Using a shared database allows you to punt a lot of that complexity to a system that's been specifically designed for it, and working well for probably 20+ years.
Too many people think microservices don't have their own, severe, downsides. The likes of netflix, google, et al, can afford to pay people whose entire job is to manage the complexities of these approaches that flat don't exist in other scenarios.
But it's a hell of a lot simpler to use a single database if you can get away with it.
I have seen the "data service" design pattern that tries to work around this, by having a CRUD microservice to front a database, providing hundreds of APIs to read/write the data for 50 other microservices. If one of these CRUD microservices goes down, everything breaks.
Has there been an architecture movement that formally embraces the spaghetti, aiming for peaceful coexistence? Detractors will certainly point at SOA and shout "that one!", but I mean one that openly admits..
I’m not sure if this architecture has a name other than “Event Sourcing” but I think giant org wide shared message bus with all services freely pushing and pulling data from it is the closest to embracing spaghetti.
this devops is also a dumping ground for anything else not happening on a developers computer which they should control but wont because "That is devops job" mostly this happens alot with node projects for some reason..
"anything else not happening on a developers computer which they should control but wont"
Ok, I've seen this, but IME (24y in industry, the last 6 as a consultant) in the vast majority of cases, it's more like "things devs should control but CAN'T [bc CICD etc are silo'd and owned jealously by an overburdened ops team unable or unwilling to facilitate self-service]".
Some of the 'jealousy' may also come from bad experiences.
It only takes a few instances of people royally messing stuff up in production for their deployment rights to be stripped away – and other groups catch the fallout too, as new "procedures" get implemented.
"Safety rules are written in blood" applies to parts of the organisation too.
Sure, there are some overbearing procedure monkeys who really want a process on everything, but a lot of "protection" rules are there because something really bad (and expensive, financially or reputationally) has happened before.
Also, some developers do not seem to know what kind of impact a production issue can have.
Having something not work during the development cycle is annoying for one person or a team. Having something break in production usually means people get paged and (depending on what you do) the resulting issues can have major societal aswell as economic impact.
In the ops world, this is especially visible if you go further down the stack. (from applications down the the network).
Network architecture moves at a toad's pace compared to webdevelopment, which is a good thing considering breaking the network will usually break every other system inside a IT department/landscape.
There was an angry Ask HN post a while back about how terrible the "new guys" are that are out there.
It then complained that the new hire wrote a horrible authentication service.
I thought it was an intentionally absurd post about the expectations put on new some rando new guy to write something important / they shouldn't be working alone on ... but they were serious.
DevOps to me is all about bridging what used to be a huge gap between operations and development. Developers need to know top 10 OWASP. One cannot simply say "it works locally". Ops people need to understand how your tools and programs work at a high level at minimum.
You hire those engineers and then you have them implement and support Continuous Delivery for your business, yes? DevOps is a silly term, but one that companies seem to use for "infrastructure engineers who help us continuously deliver value to our customers, efficiently and pain-free". Give your people whatever title you feel comfortable with - but essentially they are "doing the DevOps". :P
>>the only alternative is to start making CPUs by hand
Agreed. For some applications the cloud difference is significant; for many (most?) others though, "Cloud" is just rebrand of "Hosted". And even for more cloudy offerings, while I'm in a very specific and different part of IBM, some of the old timers/architects/powers-that-be keep trying to explain "We had that in 1969!!!" :-D
Agreed also at rewriting of history when it comes to development/support/operations models. My dad has been IT director and he chuckles when I talk to him about "new and exciting paradigms" which he of course sees as turning a circle to what they had in 70's and 80's :)
The two of you have validated my very existence as a coder. I keep jokingly telling everyone that we’re often going around in circles. No one ever believes me (lol!)
> My dad has been IT director and he chuckles when I talk to him about "new and exciting paradigms" which he of course sees as turning a circle to what they had in 70's and 80's :)
As someone with 20+ years in IT, I agree - a lot of these "new and exciting paradigms" are not new at all.
My personal favourite is how many large multi-nationals are now building in-house clouds?
WTF is the difference between an "in-house cloud" and a shared-use datacenter from the 1990's?
> WTF is the difference between an "in-house cloud" and a shared-use datacenter from the 1990's?
An in-house cloud sounds like the mainframe installed in the raised floor computer room at the school district office I worked at in the late 90s; of course, the 12 foot long Unisys mainframe was replaced with a Unisys 4U pentium pro box pretending to be a mainframe, and then there was a lot of extra floor space.
If you're running your own 'cloud' in a (shared) colo, I dunno that that's really in-house. I guess it's still 'private cloud' though.
The only possible value over the last iteration is if they cleaned house and the new team manages to be more permissive than the old one. I'd much rather have on-prem, but the sad fact is that for cloud I just need a signed check. For on-prem I also need buy-in from other divisions before I can even start experimenting with a new service.
I still have the exact same problem with respect to never having exactly the ratio of CPU to memory that would make my app happy.
The interface to the shared-use datacenter, if you're lucky, is a spreadsheet that declares the static resources you own and a remote hands guy that can tackle things beyond the capabilities of your remote KVM. If you need more capacity you need to work with the datacenter folks to order physical machines that might show up in a few months.
The interface to the in-house cloud is an API. In most instances, developers are completely abstracted away from the physical infrastructure and don't need to take a lock on some human in the datacenter to get their work done.
I still think that's a bit "rewriting the history". E. G. Vmware enabled fast self-provisioned VMs and nobody called them cloud. Heck in 1999ish or so, while at university and way before I was an IBMer, I could sign up to some ibm development program as a student, get an account, and provision Linux VMs on mainframe.
Not to say your scenario isn't valid and real, but I live that scenario every day today with in house cloud and virtualization too - it takes months of approvals and solutioning and security assessment and network engineering and procurement and costing and whatnot... To deploy a windows vm.
> Vmware enabled fast self-provisioned VMs and nobody called them cloud
Because it’s missing all the glue. I need to self-service the VMs, the database, the load balancer, the dns records, certs, and hook them all up so I can receive production traffic all via an API I could theoretically do in Terraform.
An in-house cloud will just be a bunch of commodity servers running a hypervisor that gives you an API that allows you to automate the provisioning of infrastructure.
I am guessing that in the 80s you weren't writing Infrastructure as Code to define exactly what resources you needed for your software, having it all set up automatically, and so on.
I had a couple of telco customers that ran their own internal cloud, and I’d say the main difference is that it’s way more expensive than the public cloud, heaps less flexible, and when you need compute it has to go through an approval process.
I mean … sometimes I shake my head in wonder, and other times I just shake my head.
To grossly oversimplify, there are essentially two views of DevOps.
1.) What's probably the traditional view which is breaking down the walls between devs and ops, developers carrying pagers, etc. I.e. at least in an idealized world, there are no devs and ops--only DevOps.
2.) As you suggest (and which probably more closely matches how "DevOps" works especially in larger organizations), an internal (or external) operations team provides a platform that developers can use. Developers are still going to be exposed to some operational details, but a lot of them are abstracted away.
3) The way I first heard it described years ago, bringing developer practices into ops: version control, testing, code reviews, continuous integration, etc
> bringing developer practices into ops: version control, testing, code reviews, continuous integration, etc
In my opinion, this last subset just doesn't work well for a lot of branches of operations.
Sure, it might work if you run stuff in AWS and have an operations team managing that, But what if your team is responsible for things like Storage Arrays, networking equipment?
Doing continious integration and code review on some of these is hard if not impossible
> Your devs shouldn't be required to do all of the actual ops stuff. They need access to an ops platform.
This becomes a lot more apparent once you're dealing with B2B customers and their nonfunctional requirements.
We've had situations in-house when devs built some simple microservice to handle some connection to a customers BI/DWH system - it's just 2 days of spring boot chugging to wrangle APIs around without state, nothing bad. But then, the customer started blasting that team with SLAs, backup questions like RTO, RPO, retention, regulation adherence of retention, the whole stack of IT security down to the physical access control.. That poor PO was caught just like a deer in the headlights.
This is some operational responsibility we're taking over for our dev-teams. We're providing the persistence, and this includes a defined backup and recovery strategy, a security strategy and such. And this also includes experience in dealing with these insane questionaires, and correctly pricing absurd requirements for custom backup solutions. And in fact, "we will have to schedule a discussion with ops about this" has ended quite a few of these requirements with "oh.. it's not that important". Intimidation with long job titles and appeals to external authority do work.
I think everyone can agree that for most complicated things there are many sides to a story that makes up the present state of the system. I think taking a very opinionated stand in any complicated subject by itself negates the possibility of a mature and complete analysis of any subject. So the title itself of this article is unfortunate.
>> I became CTO so I can solve this mess properly, I don’t hire devops, I hire infra engineers, build engineers, release engineers and: backend engineers.
This makes a lot of sense, but you need to realize that this is only a tiny part of a successful organization. There are many setups like that which you describe across many industries that have failed reasons beyond the role definitions. It's not enough to establish the organization. You have to keep the behaviors in check over time when you hire a build engineer with different aspirations.
I agree - I think it it is much better to use titles like these and with even a short JD describing what technology is used (many of which could be "devops" tools for infra engineers) everything becomes much more clear.
> I don’t hire devops, I hire infra engineers, build engineers, release engineers and: backend engineers.
This is a great way to do it. There seems to be a correlation between unnecessary product complexity and unnecessary corporate complexity. Being direct about roles goes a long way towards simplifying corporate complexity.
> There seems to be a correlation between unnecessary product complexity and unnecessary corporate complexity.
That's Conway's law.
"Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure."
Conway's Law is sorely overlooked. It really does have broad applicability in the industry, and sometimes you can use it to evaluate vendors better (read: this product is a bit haphazard...the vendor probably is also. Warning.)
Most of the useful insights in the Devops movement can be derived from knowing Conway's law and thinking about organizational patterns holistically. As usual the success stories get cargo culted by others without understanding the principles and thought processes that led there, and usually fail because every organization is different and has different needs. Change is hard. This will happen with every shiny new movement like Devops or Agile.
I'm sure we're due for a new one soon which will follow the same path, since people seem to be admitting that Devops has problems on here more and more. Will that cause deeper reflection? Probably not for many orgs, because soon some consultants will dream up a new brand of silver bullet to slay the immortal monster of organizational dysfunction and that's much more exciting.
That depends on the complexity of the development lifecycle, doesn’t it? If a single person can maintain it, having three different roles would mean unnecessary corporate complexity.
Roles are muddy because life is muddy...and because people can do more than one thing at a time. Also because save for a few roles, businesses rarely need people 100% allocated to one role or task. Responsibilities are nice, and job titles mean people mostly do x or y.
My favorite example is that of a restaurant. You hire waitors, diswashers, bartenders, line cooks, prep cooks, hosts and hostesses, and managers. And your waiters might be 100% focused on taking orders during prime dining hours, but at the end of the night, they help clean up assuming other roles. Your bartender might be busy making drinks, but they take a table or two if they can. Your prep cook might finish the prep early, and come on the line to help out. Your manager focuses on expediting everything and keeping the ship running smooth, but also can fill holes at time.
Communication with the other roles you work with is key though in that case, otherwise you'll have chaos - e.g. because the waiters don't know that the bartender took the table, or because the cooks need things the next morning that aren't where they're supposed to be.
You wouldn’t expect a bartender to table every person, prepare all their drinks, cook all their food and clean up their table.
But that’s kind of what we keep trying to do, because we have new bartending tools that make it so that you need fewer and because we claim communication is harder than doing everything in one role.
Then when that person invariably gets overloaded; we hire more bartenders and complain they’re not good enough at cooking your specific dish.
>DevOps has different meaning depending on who you’re talking to
1000% this.
I think it's an instance of how the conversation in software always talks about solutions as if they're end-all-be-all answers, rather than explain a problem in specifics and why that made the solution the right answer at the time.
Then that "solution" becomes a buzzword version of itself, and popular buzzwords are tools leaders use to overcome institutional inertia. Which is a good thing, to get over that inertia. But then what comes misses the insight and understanding that eventually turned into a buzzword.
> I hire infra engineers, build engineers, release engineers and: backend engineers.
I always thought DevOps the “function” just meant this, and being a DevOps engineer at a small company meant you did these with decreasing emphasis, where by the time you’re in the backend it’s just helping enforce logging, tracing, other observable components.
Has worked for me in hiring and being hired and almost everyone I know understands this.
I'm convinced the use of the term "DevOps" continues only because of its incredible polymorphic vagueness...
Two intelligent, sincere, experienced tech people can discuss devops, in detail, for a good length of time, and still be talking right past each other without really noticing.
DevOps is part of my job description. I was fuzzy on what it meant before, after reading this article, I have no idea what it means. As a general term, it's far too vague, when in practice it seems to just mean that you'll be splitting your time between writing regular application code and writing infrastructure as code. Oh and you get to be on-call.
i'm working on an RFP response that has the word "DevSecOps" sprinkled around here and there. That's an even more subjective and ambiguous term than devops :/
glad someone said it. I mean all the CI/CD pipeline are just in the old day *nix/bsd days bash/perl/awk/sed/python/ruby (fill in the blank) scripts. One problem when IT/Software development become more main stream is that everything got another layer of obfuscation especially in corporate cultures and amongst sales and recruiters and managements. Development Operation or pipeline sounds sooo much better bash scripts.
Just bash script isn't enough if there's no automated process that run it every time developers make changes to your main/release repository trunk. If you still run it manually to verify your build, then it's not a CI. If you're still doing manual release deployment, then it's not a CD.
Yerp. As a former *nix admin, around the time that "DevOps" became a term, I read it as "system administrator, who also now has to fix developer code." Which, honestly, was already part of the job at a lot of places I've worked, owed to the age old problem of people who "test" something on their weird personal desktop environment, then hand it off and shrug off any questions with "well, it works on my desktop". That said, the part that I found offensive was that I was doing 2 jobs, but only getting paid for one.
Then came DevSecOps. Around that time, I switched to just Security, and while I miss the ability to make and push changes to hundreds of thousands of machines to get things done, I don't miss any of the pressure or blame that automatically got lumped onto the sysadmin shoulders every time anything went wrong, and the complete lack of appreciation of all the times nothing went wrong, that were entirely the result of a tireless, efficient systems administration team.
It's tough, until you stop caring. And obviously, hitting the "don't care" point is pretty sub-optimal, so far as being able to muster any enthusiasm necessary to keep doing the job.
> I switched to just Security, and while I miss the ability to make and push changes to hundreds of thousands of machines to get things done, I don't miss any of the pressure or blame that automatically got lumped onto the sysadmin shoulders every time anything went wrong, and the complete lack of appreciation of all the times nothing went wrong, that were entirely the result of a tireless, efficient systems administration team.
This reminds me of a joke my networking collegues used to tell to the new hires.
Welcome to the networking team, where we are responsible for the network, aswell as anything that runs on the network because developers never bothered to pay attention in their networking class.
Yeah, I've often seen "DevOps" described as "System Administration, but automating everything you can with scripting or whatever", to which my first thought was "WTF do you think Sysadmins do, if not exactly that?!?"
It's like they'd never heard a BOFH tell someone, "Go away now, or I will replace you with a small and uncomplicated shell script."
Yep, this has been the hardest fight in our "culture shift towards devops" - getting the idea and wording right.
After a lot of discussion, we mostly realized that we get the most value if we define more specific operational roles. We now have the idea of infra-ops, and product-ops. infra-ops is providing a deployment platform - a container runtime and persistences. product-ops on the other hand is responsible for deploying and running the different products from development on this deployment platform.
And this is giving people good ideas. Some products are very simple without harsh requirements. In these cases, one of the backend devs just takes over the product-ops role by setting up a deployment pipeline, a job and some migration handling. Other products are bigger - for example we are providing some of our systems as essentially managed systems to invididual tenants and customers. In such a case, parts of the product operations is with the product team - such as writing jobs, releasing containers and artifacts, and other parts of the product-ops role takes place in the managed services consultancy. These products are currently looking at wordings to split up the product-ops role according to their needs in their context. And that's totally fine - the infra-ops role is also divided depending on the system the engineer is working on. A postgres admin is an infra-operator, for example.
And all of this results in a devops oriented culture of cooperating across team boundaries through automated processes.
It's ok to have build-engineer doing release engineering if it's simple enough, but doing build engineering properly (Source control, artifact control, shared caches, dependency management of the compiler outside of the standard toolchain) is quite a large job.
Release engineering itself gets complicated when you take into consideration the different target platforms, most games get published on Playstation, Xbox and one of a handful of PC platforms (notably steam), plus the online systems and of course the development environments and internal release systems which are ubiquitous.
Build Engineering usually is the step between developers and QA, and release engineering is the bit after QA.
Both interact a lot with QA so could be rolled into the same role.
I don't know, every time I see a title split people start saying that's "X's job", you can't do that because it's "y's job", or I don't want to be involved in "Z".
> DevOps has different meaning depending on who you’re talking to, even some definitions that appear similar are different in nuanced but important ways.
This is always a dead giveaway that something is a buzzword
Same for rest. Sometimes when people use it, it just means JSON + http requests. Other times it's supposed to be some kind of architectural style
Yes, at least when there's disagreement about what OOP means. You and I can disagree about it, but we can more easily answer "Does it match what Alan Kay wrote about?" with a yes or no.
I don't understand how seemingly 90% of developers, can't do anything outside of their narrow scope of experience. Writing a shell script? Throw a temper tantrum, that's devops job! Have to work with legacy code? I can't believe this!
Especially seniors and beyond. They force leetcode interviews they somehow pass or are grandfathered through and gatekeep "trash devs" by slamming gotchas about how the whiteboarded code discussion doesn't technically compile and how "returnAverage()" isn't a method ("did you know you have to write methods before they work? What does returnAverage() supposedly even do? Return a random character in the alphabet?")
I get excited about working in different tech areas. I'm exceptional at fixing other's bugs and maintaining code. Absolutely nobody seems to be hiring for this though. It's all about college exam trivia and leetcode.
When I do get into these companies, I have to work with people throwing around casual racism with HR joining in and f-slurs like it's corporate 4chan. I ask if anyone can help look at a critical bug I discovered, and nobody speaks up. Even when it turns out it was their last code change that caused it and they are the sole master expert in this area, and they were just working on organizing their desktop instead.
"I can try to look at the SQL problem, since nobody else spoke up." Then I'm explaining the basics of SQL to some guy sitting around blank faced and it turns out he was hired because of his 10+ years of expertise in SQL. His whole job is to tackle problems like that.
Meanwhile I'm out here looking for jobs when panic cuts happen and it takes forever because I'm drained from all the gotchas and gatekeeping in these interviews.
There's no way I could live with myself if I put this experience onto others. I lose sleep when I fail to call out someone talking over a quiet person on my team. I've never seen anybody else stand up for anybody, however.
I like to build up my teams with people that act and think like you. Sometimes I need a sniper and when I do I'll get one. But my teams need to be excited about building and running the systems that we make.
I learned SQL because my coworker was stuck on a bug she couldn't figure out and 'knew' I'd be able to solve her problem.
I just kept asking questions until I found the one she missed.
I also ended up becoming a bespoke VxWorks admin because the guy who volunteered was never available, and someone asserted my code wasn't working well on VxWorks, so of course I had to know enough to do benchmarks. I fixed a few problems but the real issue was the hard drive wasn't doing DMA due to the kernel not recognizing the processor revision number.
Somewhere between those two events I realized that if my part of the project is great but the whole project is on fire, nobody cares about my stuff. I don't get points for being right and the team being wrong. I mean, I do for some people, but I still feel bad at the end of the project, and those are the 'points' I have to live with the most.
The way that played on on the latter project is that once we got our shit together, we started work stealing from our peer orgs. Volunteering to carve off little pieces of interface between us and 'take care' of this bit of data handling here and that one over there. I came to realize that the Org subconsciously knew this and any past success they had was due to self-organization and collective work-stealing.
Throw people at it, turn a blind eye to precise mission statements, and hope for the best.
Sounds like medium or bigger sized company office politics.
At smaller companies the title tends to be only a title and many people are basically full-stack and know a bit of everything. And you're trading hats to get jobs done.
You hit the nail on the head - at many places, the hiring process seems to be great at blocking people who are great at solving real world problems that aren't as good at whiteboard leet-coding.
If you truly enjoy this part of the job, you can go for a freelance career - build a network and reputation as the person you call to solve the hard stuff, and do mostly that, no leetcode interviews required...
To be fair, shell scripts are pretty horrible to write and maintain from a programmer's point of view. They're good in that they're portable, but they're not a good programming language. I don't object to writing them, but I'd rather there be an alternative.
Anyway, it sounds like you've worked in some really shitty companies / shitty people.
Do those different breeds of engineers work in one team or in separate organizational branches?
The problem the term 'devops' is addressing, the way I understand it, isn't that of insufficient individual jack-of-tradesness, but that of too much separation between (internal) organisations. It should be perfectly possible to build a "devops" team from deeply specialized experts and turning it into a job description seems quite a stretch to me.
But you might actually want some jack-of-trades types nonetheless, because those isolated organizations weren't completely without merit: they are good at solving that problem affectionately called the bus factor. A single ops guy in a devops team is a single point of failure and the mitigation preparation of keeping some of the not-so-ops peers sufficiently in the loop will be much more dependent on organic motivation than the counterpart in a specialist company branch. There, substitutability would be much easier to ensure with formal process.
> Do those different breeds of engineers work in one team or in separate organizational branches?
It doesn’t matter as long as communication is open.
Proper ops is all about reducing the bus factor. Procedures don’t depend on people, they depend on being good at communicating.
If your ops person got hit by a car, I would expect that everything would continue running, I would also expect to find some clear documentation, this is mandatory.
Some time ago, i was talking to an MBA people /project manager guy was ho had a completely different definition of what "devops" meant. His meaning was some kind of project management approach.
We have a way to prostitute words in this field... agile, QA, devops.
Yes, I am, the project is mostly in stealth mode but please feel free to drop me a CV:
jan [at] competition [dot] company
If you want more information on the project we're working on the site is https://rennsport.gg
We're building a hard-core racing simulator game with a backend which can persist car ownership in a way that feels authentic. (IE; not just tied to a game).
I am interviewing right now and my previous experience includes devops roles and SRE roles, so I get contacted for both by recruiters. After hearing about the responsibilities and examples for these roles I can only come to the conclusion that titles are a waste of time. Even the "level" of a title of Staff, Principal, Lead, etc are a waste of time. Just call the job what it is and if you can't decide on what it is at least to an 80% level, maybe you're asking too much for that job?
I get called an "Infrastructure Engineer" sometimes, I've also had the title of SRE and SWE. I don't really feel like any of these actually fit me.
I do work on Cloud Infrastructure at times, but honestly it's the smallest part of what I do. It's usually addressed in the architectural designs and I have to touch it incrementally. What I do much more often is writing tools, daemons, and services for distributed systems. I end up calling myself a Distributed Systems Software Engineer to reflect the idea that what I work on is systems and software, and most of them are non-monoliths (from a systems perspective).
Do you have any thoughts on Systems Engineers or the title that I prefer to call myself?
In the games industry we have a role called “tools programmer” which is a person who works on tooling to accelerate development, sometimes these can be extensions to the editor for a specialisation for a game (say, a tyre builder for a racing game) and sometimes they can be more broad (a distribution tool using libtorrent so that a new build of the game can be shipped worldwide very quickly).
There are people who might refer to you as a backend programmer (if the focus is services and daemons), or platform engineer (if the focus is developer velocity).
I agree with this but I’d add that it’s also important that the roles aren’t overly siloed. A front end engineer, for example, should be able to and even encouraged to look at backend code, and even submit PRs for it.
And more importantly, the backend engineers should be encouraged to be grateful for the PRs (even if it’s not acceptable for some reason). The more eyes on the code, the better.
I think developers naturally tend to specialise (at least for a time) but ultimately we all need to understand and contribute to code regardless if it’s front end, backend, build, test, …
I'm not sure if having two teams is always going to be better than having one DevOps team, but my experience in having two teams is that it's rare to have the incentives aligned. The author of the post pointed out that dev teams will cut corners and throw broken applications over the wall to ops to deal with.
When ops gets woken up at 2am because someone in dev cut corners, what happens? Does dev feel the pain? Almost never.
The same thing happens when outside contractors develop code. They often provide a buggy and undocumented mess, and then no longer work on the project. They never feel the pain, so they're not incentivized to provide good code.
Until we find ways to align incentives, we're going to keep getting crap whenever more than one team is involved.
Reminds me of that time when my team was called into a meeting where the CTO "advised" us that "code does not have to be perfect", when all we wanted to do was review the code for a PoC that we were ordered to "own" and deploy to production (even the creator said he cannot guarantee the code he copied from Stack Overflow for the PoC is production-ready).
In the same meeting, the CTO was ranting about the instability of a service (which was also a PoC that was pushed to production before we were _also_ made to "own" it, yet never given the budget to even get acquainted to the codebase), claiming the reason for that because we devs are lazy and unprofessional.
I highly doubt making people who _responds_ to the incentives to "feel the pain" will fix much. I suspect things are more likely to get fixed if the people who _creates_ the incentives are the one "feeling the pain".
As they say in Hunger Games, "remember who the real enemy is".
On the other side, the ops team sets up a system of slow and complicated deployments, but they don’t have to deliver features with it, they don’t feel the pain of working with it as much as the feature devs.
> If the “DevOps” team ships a Postgres RDS instance it will run fine forever, that is until an application starts using it. All of a sudden a cascade of N+1s hit, the CPU spikes, and queries grind to a halt. Who is woken up? And why does this always happen at 2 AM? In this scenario, there is nothing for operations personnel to do, yet here they are.
This is definitely a symptom of a broken model and not what I would call devops. IMO the most important tenant of devops is "if you build it, you run it," meaning the appdev team that decided to use Postgres RDS is the one getting woken up at 2am.
It's also, in my experience, one of the best ways to reduce masturbatory engineering decisions and get people to focus on picking boring technology that works. Coding up a serverless application in Rust that's using a CockroachDB backend at a Python/MySQL shop would get a lot of engineers excited, but those people would be less excited knowing they're going to be the ones paged at 2am when this new and exciting architecture falls over in an unfamiliar way (as opposed to Python/MySQL, where a wealth of operational knowledge at the org has already been built up).
Similarly, it naturally reduces architecturally complexity. Younger senior engineers love drawing boxes of queues, multiple microservices, event buses, etc to show off their skill in creating the ultimate engineering fantasy, but once you throw enough late night operational incidents at a senior engineer, suddenly the preferred architecture becomes "an executable running on a box that I can SSH into when things go wrong."
> This is definitely a symptom of a broken model and not what I would call devops.
I'm I the crazy one here? This kind of work is my bread and butter as a "devops" person. PagerDuty fires, I find the bad query, match it up with the most recent PRs to find where the n+1 got introduced and either patch it right there at 2AM with the on-call manager's approval or roll it back. Then we have a postmorterm in the morning with the team.
I'm the person positioned the best to do this work because I'm god in my little ops domain, have the most visibility and the biggest toolbox of potential fixes.
This sounds like what I mean with building it and running it being devops, yeah -- the fact that you have access to make pull requests or code patches, or even know where the code repository exists in the first place, shows that you're at least somewhat involved in building the application. Contrast this to what was traditionally ops and is now "SRE"; in those roles, applications are usually black boxes where an ops person doesn't know/care how they're developed, because they're responsible for the overall health of the system, which could be managing 100 applications made by 50 development teams.
It also seems like we cycle through ideas, so what was outdated a few years ago becomes interesting again due to a new perspective on modern technology as well as new developers entering unfamiliar spaces.
I feel like I need some definitions to understand this article and discussion. What is dev? What is ops? What is devops? What is system administration? What is sysadmin?
I'm 47 and what is described seems like the old problems that I thought devops solved.
I’m also confused by this article. Perhaps this is more applicable to companies that develop software that is very tightly coupled with the web and cloud services. I work in the embedded industry and from my understanding (as naive as it may be), DevOps has simply meant the automation of building, testing and releasing our products to a customer accessible endpoint and is something that is life changing. It massively cuts back on manual human interaction getting our software to our customers. Perhaps since our product is not nearly as monolithic as most products that really hook into DevOps processes that can match the actual product in terms of its scale and complexity, DevOps is something wonderful that I would never define as ‘bullshit’.
DevOps is bullshit in the same way that Agile is bullshit. So, it's actually not at all, but just has become so fashionable that the term is sufficiently abused to become almost useless.
Agree. Both devops and agile were about empowering developers, the ones who actually make the products people use. And then managers and management consultants got hold of the terms and mutated the ideas and they ossified into "best practices" that get in everyone's way.
Most developers have no fucking clue how Linux, networking, storage or whatever works under the hood. They know how to develop whatever stack you're at, but stuff like latency, packet loss, redundancy factors, backup policies, monitoring or other classic ops topics are completely beyond the comprehension of 99% of developers.
"DevOps" usually means some C-level execs say "fire the expensive neckbeards that have the time to properly understand a system" followed by them saying one of
- "oh fuck, someone managed to compromise a service and because no one knows what the fuck firewalls are / Kubernetes doesn't come with ones OOTB the hacker got complete control of everything"
- "oh fuck, production is down because someone fat-fingered in Elastic Beanstalk which recreated the environment, dropped the RDS database and there were no backups" (I've been personally bitten in the arse by their definition of "recreate" - all I wanted it to do was to replace the damn EC2 instance)
- "oh fuck, we're seeing insane AWS bills because someone DDoS'd us and no one created a cost limit or a sensible limit for autoscale or a simple dumb monitor that alerts someone"
- "oh fuck, we're seeing an insane AWS bill because someone got his AWS access credentials stolen and some shithead spun up a ton of g3.xlarge instances to mine shitcoins"
Other C-level execs see constant issues between "ops and dev teams" because their team leads are friends of the silo model and decide that instead of getting rid of the dumbass managers they're getting rid of the ops team because "cloud", with the exact same result.
I have bewilderingly tried to discern why software development continues to grow more and more complex.
It wasn’t always like this. There was a time when we talked about languages and OSes and libraries as if they made a difference on how much you could get done with as little people and cognitive load as possible (the claims were very much overrated, but the point was we acted like it mattered).
And then it started ballooning. It seems to me that much of that coincides with a lot of new money being dumped into the economy, and software moving past the necessary-evil-so-how-do-I-drive-down-my-costs and on to the gold rush of you must have a web app for your service that rivals a video game in complexity and visual finesse. It seems to me that as long as their is so much free money sloshing around venture capital funding, that it was inevitable that the process of making software would complexify to soak up the extra cash. After all, you can only add so many levels and varieties of managers to add value. After that comes the variegation in roles of software development.
My theory is a variation on that, which is that the money has placed so many more people in the space you inevitably end up with more people working in different directions, the JS ecosystem especially comes to mind here.
Because you have, in 90 (99?) percent of instances a smart person (probably IQ 120+) doing a stupid menial job that requires a level of competence that only someone smart can provide.
So to prevent themselves from ending it all, they invent things to do to keep themselves sane while they churn out CRUD apps all day long for decades. Sometimes you get truly amazing software, but most of the times it's just reinventing the wheel, but worse.
Even worse, some of them become infatuated with doing things "Like Google," and errant CTOs enable them, leading to a 6 person team supporting a piece of software that could be replaced with WordPress, to the betterment of the primary stakeholder.
I've been consulting for over a decade now. The number of times a potential client has told me they've spent into the six figures (or eight) for something that free software does out of the box but better is depressing. The worst part? I usually can't convince that person they've been absolutely swindled, and so I have to let them keep on their merry way.
> So to prevent themselves from ending it all, they invent things to do to keep themselves sane while they churn out CRUD apps all day long for decades
Or they spend the spare time bullshitting on web forums.
> I have bewilderingly tried to discern why software development continues to grow more and more complex.
It's because of growth, software development today is both simpler and way more complex than before and that's entirely due to growth in the sector. In many ways it's simpler than it ever was, I'm writing an API using Lambda/API Gateway in AWS and it blows me away how quickly I am able to get services stood up and configure my API. But in another way it's so much more complicated, for this same API I spent 2-3 weeks experimenting and researching IAM roles and how all my AWS resources would interact with one another.
I would say the floor of software development has become way simpler, deploying a site to Netlify is way easier than dealing with webservers of the past. But while the floor is lower than ever before, the ceiling is in the stratosphere, with extremely complex systems that you can string together in the "Public Cloud".
> writing an API using Lambda/API Gateway in AWS and it blows me away how quickly I am able to get services stood up and configure my API
> for this same API I spent 2-3 weeks experimenting and researching IAM roles and how all my AWS resources would interact with one another.
same experience here, and trying to understand/test un/poorly documented AWS behavior with public/private APIs and EC2 resources has been a huge time sink :(
SaaS model requires software to be always online (strict sla) and you need to be able to update it without downtime or very little downtime. That’s the reason. For example, telecom software that always had these requirements was always complex
Your guess is as good as anyones at this point. There’s just many over corrections that resemble strong levels of grandstanding. React is complex? Okay, let’s ditch JavaScript and just do basic server rendered pages.
It’s a laughable over reaction (pun unavoidable).
These are manic states the dev community enters, and all pragmatism is lost in our discourse on the solution.
Cooler heads are not prevailing. There is certainly a solution in the middle.
Devops is bullshit is also an over correction. It’s worth saying that making infra more accessible is a democratization of that entire sub-field. How do we preserve the good part of that?
My theory is we need to add "software history" to computer science education programs. How many developers whose careers began after 2000 have ever hand written or know the power of a Makefile? I've worked at research labs and major animation studios that back in the 90's the entire infrastructure was fully automated thru Makefiles, and it purred like a well fed cat. There are hundreds of thousands of forgotten perfectly fine software tools, ignored simply because our industry has absolutely no respect for it's own history or lessons painfully learned beyond pulling up disparaging stories for new marketing angles.
Very interesting comment. What older software should devs know about?
My problem with Make and similar tools is I don't want to learn more ad-hoc syntax to accomplish something nearly trivial. But I don't know if there's any alternative.
Original Make was butt simple: any lines that begin without tabs are a filename followed by list of other file dependencies, files whose modified date must be older than the filename at the start of the list. Than any lines of text immediately beneath that first line starting with a tab character are the shell commands to be executed in order that brings the first file up to date against it's dependencies. That was it, that is the entire original Makefile syntax. Then software vendors started adding extensions...
I guess as far as other, older software: I'd need to know your interests or field. How I write software has completely changed at least 5 times, and now I'm deep diving on my 6th (docker/k8s) to deliver the same things I was delivering back in the 80's with software development process one. Actually, add 3 more complete re-wiring of how I write code - I forgot working on game consoles: every new console is a complete start from zero how to write, debug, package and ship software, software that is not that much different than was delivered with the last platform and dev process.
The way to define options is interesting.
It's hard to see the benefits from the docs in the README tho, given that it's yet another syntax to learn.
I suggest to add a complete example of common use cases like running the unit tests, building targets while setting the correct options depending on the host (linux vs windows vs macos).
While make is very powerful it's certainly not a tool that is nice to use. Theres a reason people built up layers and layers of complexity on top of it.
I completely agree that 20+ years of cheap money have absolutely caused much of the complexity we see today. That's because people aren't working on problems that matter; instead, they're just trying to make VCs rich--effectively passing the buck on to the greater fool. I think this era is coming to an end, and we should see a drastic reduction in software development costs with the end of cheap debt.
I hope I'm not being pedantic here, so please forgive me. Didn't interest rates go to zero after 2008? So that's 14 years. I guess 8 years before that there was also Dot Com.
We may see a strange hang over from this period: over the last 20 years we've created tech billionaires and tech trillionaire corporations - both entities that are not used to being told "no" and will spend their own money to extend this party that is their reality. Case in point: FB/Meta and their doomed Metaverse.
Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure.[2][3]
— Melvin E. Conway
The problem isn't necessarily software itself, but how we organize people (more than 2 or 3), how we communicate, how we mirror operations, expectations, etc. in software.
Scaling sustainable software feels like an "unsolved" Problem, because the society hasn't figured out how to organize better.
Small teams can be super productive, but unless the people in it are there for the full run of a product, you'll end up with a problem; taking over the work for one developer will take more than one developer.
I'm sure - but haven't witnessed this myself yet, so take it with a grain of salt - that if one productive developer builds an application in a year, it needs a team of 5-10 to continue development at a similar level, and even then it may not make it.
Companies need to focus on keeping software as simple as possible, well documented, and transferable. Unfortunately this also means curbing people's enthousiasm.
The complexity of what we are building keeps expanding. Theres no large conspiracy. We have a lot of tooling to build things that were near impossible to build in the past.
This is like asking whether innovation is necessary. Capitalism pushes us towards it. New forms of efficiency and value are unlocked by moving further out on the software complexity curve. Not everything thats complex works well, but some of it does solve the problem
What's "capitalism" in this context, and what's its actual connection to software development practices? What are "forms of efficiency" and "forms of value", and how does complexity "unlock" them?
Capitalism is a company that is attempting to solve X problem or do Y thing etc. Forms of efficiency are "how fast can they solve that problem". And complexity doesn't unlock anything. It's just inherent in the problem space a company takes on.
>Capitalism is a company that is attempting to solve X problem or do Y thing etc
So basically any organized human activity is now "capitalism"?
>Forms of efficiency are "how fast can they solve that problem"
From which follows that "new forms of efficiency" means that previously it didn't matter how fast they can solve that problem, but now, due to "moving further out on the software complexity curve", it has begun to matter?
>And complexity doesn't unlock anything.
Then what is meant by "moving further out on the software complexity curve unlocks new forms of efficiency"?
I semi-agree but my take is that money can buy you people, it can buy you lines of code, more moving parts, more stuff. But it can’t buy elegance, efficiency and good design - those have to be nurtured. Most engineers nowadays haven’t ever seen a codebase that isn’t a hot mess (no offence folks). The demand for talented and experienced engineers eclipses the available supply. The industry is growing too fast for that. The engineers that really are good are needed in leadership, or become founders.
I tried out cloudflare for the first time yesterday using a personal website. Previously I was just using a domain registrar and setting DNS to digital ocean. cloudflare is a very different/complex beast, but I can imagine that the pretty interface alone brings in customers.
I don't like this hyper-expansion either, but when you have an army of monkeys on typewriters, well, a lot gets "produced".
I sometimes marvel at how few people worked on various pieces of commercial software in the 90s. You'd have 4x the headcount and still take twice as long, today. And it'd probably be webshit instead of native because "productivity matters more than performance".
It has blown me away too. There was a time where it seemed like the default "devops" strategy for startups was to use Heroku. Developers could spend all their time working on code. Worked at a startup a few years ago. We had 2 devs, and one devops guy building out Kubernetes, Docker, running on Aws. Had to constantly spend me time with him troubleshooting why this or that thing wasn't working rather than writing application code. We had like a 100 concurrent users.
Now I am working at a large org for an internal application. There is an ops team I push code to github and Jenkins runs the CI and deploys it to a dev environment. I push a button in Jenkins to deploy to production. In two years there has been one ops related issue where I had to bump up the memory limit from 2GB to 4GB. There is a single server and a database. Setup works fine for a few thousand concurrent users. My skills are in development and understanding business reqs, not mucking about in config files. There are people that are good at that and enjoy doing that, let them handle it.
> We had 2 devs, and one devops guy building out Kubernetes, Docker, running on Aws. Had to constantly spend me time with him troubleshooting why this or that thing wasn't working rather than writing application code. We had like a 100 concurrent users.
I'm glad you said this. 2-dev shop here and we've used Heroku the last five years. We spend all our time developing and supporting our application and almost never worry about infrastructure. I keep getting tempted to move to K8s; Heroku is expensive for what we get and seems to have suffered some serious brain-drain. I've dabbled with K8s in side-projects and I can geek out on all the terraform and yaml stuff for days. But its probably not a great idea to inflict it on a 2-dev shop.
I like the Uncle Bob explanation: the number of developers has a doubling rate of ~5 years due to the constant entry of new developers [1]. Over time, the number of inexperienced developers far outweighs the experienced ones. Inexperienced developers naturally gravitate toward complexity because they don't know any better.
Couple that with a social drift toward hyper-specialization. That sort of hierarchy naturally creates minds that don't think at the systems level. This is the general theme of Buckminster Fuller's Operating Manual For Spaceship Earth: the shift toward hyper-specialization (or narrow focus) has long-term disastrous consequences.
When you couple inexperience with narrow focus, you get messes.
Part of this is to blame on the industry-standard thinking that because someone works for Company X, they're a competent, logical engineer (and should be granted authority/responsibility over essential products/projects). This is the downfall of "code tests" and "whiteboard coding." They don't evaluate for systems level thinking and so a developer with poor creativity and logic skills slips through the cracks because they're great at eeking performance out of a function which impresses the lollipop guild.
There are thresholds of effort for certain things at largely constant levels. E.g.: designing a new language, build tool, web application framework, or database all have some minimum effort that needs to be invested. In the past, there just weren't enough developers in the entire world to "overdo" these things, so there was a relatively small pool of languages, tools, and frameworks to choose from.
Now, individual corporations have armies of junior developers spitting out frameworks and query languages like a machine gun. There's so many now that you or I haven't even heard of 99% of them!
As the number of available developers grows exponentially, so does their capacity to "reinvent wheels". Their ignorance of existing wheels to they could be reusing grows exponentially also. The result is an exponentially exploding set of ad-hoc, incompatible systems.
In my recent semi-DevOps, semi-Cloud-Engineer role I've come across an absolutely bewildering array of tools even when I've generally restricted things to one cloud and one language's ecosystem. Heaven help you if your project has multiple languages!
This just reads like a "Damn kids, get off my lawn!" type comment to my admittedly young mind. Same with a lot of similar comments I see floating around HN, especially ones that have anything to do with web development.
This is my theory too, and as someone learning to program (hopefully as a career), I'm very worried what a new paradigm of tightening will do to the field. I feel like LOTS of people who think their stack is secure are going to get dropped because at the end of the day, lots of the B2B SaaS doesn't actually deliver anything to the world of material needs.
There is this moronic obsession with "scaling" in this industry, and it just blows my mind. You can have a wildly successful and profitable company while serving just a few hundred customers that requires nothing more than a basic LAMP stack.
Meanwhile people are now spending millions of dollars and years of person-hours building MVPs in the cloud that won't ever go anywhere because the business model sucks.
Focus on delivering value to your customer first. How you get there is quite literally irrelevant to your customer.
> There is this moronic obsession with "scaling" in this industry, and it just blows my mind. You can have a wildly successful and profitable company while serving just a few hundred customers that requires nothing more than a basic LAMP stack.
This is us. We have a couple thousand customers in a B2B space with large revenue per customer. Growing rapidly and our node.js stack is running on a tiny EC2 instance and RDS server.
I spent a lot of time in the last couple years trying to fight back against attempts to overcomplicate this by using new services & new tooling. I just see it all as stuff that's going to slow down the single most important thing to us from a business perspective - which is writing new features to optimise our internal workflows - and give us more headaches from an operational perspective.
Even the stack we're using feels ridiculously over the top and complicated compared to a basic LAMP application. Everything in node.js feels like a huge pain in the ass. If there's not a node module you can install to immediately solve your problem (which of course just adds a different set of problems) even writing basic things feels exceptionally arcane and time consuming compared to a basic PHP implementation of the same thing. Maybe it's just a side effect of our dev team size (currently very small, only five full time devs) but the overhead of it all just feels like it's not worth it.
There must be some a rite of passage or initiation ritual that I haven't been good enough for yet where you need to write an opinion piece about the term DevOps, because otherwise I'm not sure why in 2022 we're still having to read these articles. Whoever wrote the first one has trolled an entire generation of engineers.
Because, like Agile, it was seized on as a label for "whatever reorg I want to introduce in my three year term at this company before I leave for the next one".
I had a very weird sensation while reading this where I was shaking my head disapprovingly at this article. Specifically when they were talking about the commoditization of DevOps.
Speaking personally, I used to work on a Platform Engineering team for a major multinational that rhymes with ay-do-bay. And their requirements for the workloads that we needed to be built and hosted where very unique because you have teams running completely different languages and toolchains, esp. from acquisitions. I can argue there that the template K8s setups wouldn't work.
In the author's case, the only reason why they feel the way they do is because we finally have some sense of standardization on what Infra looks like for most companies. It's no longer a question for most folks if they should adopt Docker, we have accepted images into our lives. Same for K8s (after a certain point of scale). So uh... yea. Catchy blog title to sell some thin layer on top of K8s in which it doesn't solve the root issue that they are talking about.
This echoes the philosophy of Google Site Reliability Engineering, which (this is key) is an engineering discipline.
The job of DevOps is not to close tickets. That'd be like driving a car by shouting directions at someone lying on the floorboards holding a wrench to the steering pinion.
The job of DevOps is to build a steering wheel (and ideally, teach SWEs how to drive... at least enough that they understand what a "road" is and why it's a pleasant experience for everyone if you stay on it. If the road doesn't go where they need to be, then it's time to file a ticket, but that ticket had better be "Build a new road," not "Offroad this one car to the cabin in the woods and call it a job well done").
The raw hardware of an enterprise deployment is so flexible it solves nobody's problem. DevOps is in the business of writing the operating system for a mega-computer physically represented by hundreds to possibly millions of heterogeneous computers. It's a process of continuous growth to make that work.
Well they explicitly use the term SRE and not DevOps and I think that's very intentional and related the gist of the article. DevOps was never meant to be a role played by a person or a team. It's meant to reflect aligned incentives of dev and ops. Whether you have an SRE team or a platform team or a bunch of kitchen sink teams.
I much prefer the SRE distinction, it gives more focus and especially combined with the book and other materials, a much more professional workspace.
You want us to run and manage your software? Sure, here's a checklist of what it has to conform to. Oh it's unstable? We will no longer run it for you, here's the pager back.
"devops" is just when developers do operations, and most developers generally don't want to because they are more interesting in writing software. If your job is just to manage infrastructure and you use terraform, you're still just a sysadmin. It's a "boring" job but someone has to do it.
362 comments
[ 3.1 ms ] story [ 243 ms ] threadWait until you meet DevSecOps...
ProAnalDevSecOps
[01] https://linkedrecords.com/the-big-devops-misunderstanding-84...
There's your problem. You have people who build stuff without caring where and how it runs. Recipe for disaster.
Spot on, but not only that, also decisions made based on number of hands in a room.
Most developers have to do Frontend, Backend, and Ops. These have wildly different mindsets and feedback loops and not enough time exists. Don't hate the player, hate the game. The orgs are fucked, not the workers.
There are only so many hours in the day, it’s hard enough to stay on top of my core skill set.
Unless you're arguing that engineers are unproductive members of a company, I suppose.
Software devs who close their mind to the lower parts of the stack give me the opportunity to learn and do their jobs as well, becoming a more valuable employee as a result.
That’s iyho. The management of a bus line company doesn’t see it that way ;)
The word "scam" is overused. It's a scam if it was a bait-and-switch. It's not a scam if it was advertised this way up front. In the latter case it's just a job whose description you don't like.
It is not better for the department members or the software itself. It is only good for the bottom line. Yes, if you know it all you'll be valued higher - but the company is still reaping the majority of that benefit from you and not rewarding you with 3 salaries. You should not perform all the roles because it's economically idiotic to do so. You could contract out all 3 services and make a shit ton more money than you would performing them in a salaried position.
That is a scam.
If you think that is a scam, I don't know what to tell you.
There's a reason to have a team of people doing this "DevOps" work. Just like we have a team of people who do SRE. It creates a standard and a single point which all work flows through. Then you don't wander onto a new project only to realize they use $BESPOKE_DEPLOYMENT_METHOD because "it's what we used 6 months ago". Or worse, you don't have a developer playing with a massive, nuclear powered, foot gun like Terraform and accidentally destroying infrastructure.
Making DevOps/DevSecOps/$BUZZWORD the responsibility of developers is a cost-cutting measure not a responsibility measure.
If you're building a crud-app in a common framework with low volume, sure you can toss that over the wall.
In theory, devops shouldn't be responsible for maintaining the performance of code. The specification should say what it should run on, the devops guys set up a pipeline and manage that thing, and the developers are the ones taking heat for not hitting that goal. If devops guys are taking the heat for that it sounds more like cost-cutting measures flowing the other direction.
Yeah and this is a great solution (aside from the bus factor), maybe the person I responded to was really concerned less about knowledge and more about expertise (the word they did use in their was expert) At a company I worked with we called this T-shaped engineers. Deep in one thing, but broadly knowledgeable. Devs have to have knowledge of ops, but not "expertise" that is ultimately what ops is for, we may just ultimately be fighting over where the knowledge line is sufficient and what constitutes "expertise" :-). I for instance think terraform is not that much of a footgun and provides good rails to be used by developers.
And in some ways this is by design, I want to have some distance between dev and ops because it gives me the freedom to rearrange infrastructure transparently. I can move workloads between Lambda, ECS, and EC2 based on the observed performance characteristics without anyone being the wiser.
I think most modern AWS services are more the equivalent of an API or a microservice than they are a server, and you need to understand the limitations of the services you integrate with. If you're a cloud native company and don't have a mature platform engineering team, devs are going to have to know alot about AWS.
If the developer has all the information necessary to create an S3 bucket, lambda function, kinesis stream, etc does it make sense for them to offload 10 pieces of information to me, or for them to learn HCL and interact with it themselves, especially if it is something they do often. Especially if there's a central dev/ops team and they're the limiting factor. Devops taking over every infrastructure change for a broad team of devs is inefficient and expensive. It's also probably frustrating for devs that are aware of the above factors. Lots of devs I know would prefer to do it themselves.
And all of this is to mention that infrastructure and integrations with infrastructure are not static. Should I be reviewing all PR's to a system that touch the client code for an AWS service, because the dev team doesn't want to learn how that service works? Maybe, but in the end I don't know if this makes anyone happy.
Platform engineering is certainly the goal. Where ops creates a platform and dev just consumes that platform. But I don't know if it is realistic. Every system on rails is great until you try to take the rails off, and most devs I know hate rails :-) Fundamentally "devops" is all meant to solve human problems, not tech problems so it will have to be dynamic.
"I don't want to have to worry about what machine my app runs on" vs "kubernetes is to complex"
"Dependencies change to often" vs "I don't have time to maintain this thing I wrote myself"
"I just want the infrastructure to figure out what I need" vs "I want to be able to build whatever I want with a bespoke language/framework/database/architecture"
> it's a knock against the tech itself that it forces all this incidental complexity on you.
If I were to say that Kubernetes is the magic secret sauce that fixed all the incidental complexity I would get laughed out of the room. There is no magic secret sauce to the incidental complexity, the more we try to fix it the more we create (or the cheap-fast-good problem. This is probably easy if there are no cost limitations)
When I design a backend system, I need to think about how the front end developers are going to interact with it. My data storage characteristics and scaling. I need to know am I designing anything that’s hard to deploy. How will logging work and be aggregated. I have to be able to think about the entire system.
It’s not just “cost cutting” at a certain point in your career you are expected to know more than just “how to code”. I’m not saying learn AWS. But I would expect any senior developer to know about what their code runs on top of .
I think to push back is rightfully coming from the “ops” part. I consider “creating the CloudFormation/CDK/Terraform” code as part of “development” as part of coding.
If you use Docker, wouldn’t you consider creating the Dockerfile as part of development?
Yes I knew AWS pretty well by the time I left and I needed to know it to be a good developer in that context and designed most of the processes around it. But I refused to do “operations” - ie “infrastructure babysitting”
There is a huge distinction between “I don’t think I should have to know how everything works” and “don’t call me in the middle of the night when something goes down “.
[1] I’m the first to admit that I left the “real world” once I started working in the cloud consulting department at $BigTech
> If you use Docker, wouldn’t you consider creating the Dockerfile as part of development?
Sure you could argue a developer could, or even should create these things in theory. The problem is then when it goes down I have made two problems out of one. Now I have to manage the infrastructure of a system and what is running on it. Realistically, and even in my current job, it's actually several systems. Now when something breaks I have to pray I can fix it. Instead of allowing a team of infrastructure professionals to at least insure the hardware is working my 8 hour day turns into 14 or 16 very quickly the second one thing goes wrong.
Same shoes, but I have different perspectives. I can figure out where it's not working and if it's my/our team's area of responsibility then we go fix it.
Since we handle infrastructure (as code) and deployments in team, along with all the development, most of the problems are handled by us. Unless it's clear that it isn't, e.g. some API that we consume that keeps throwing 500, can't fix that.
Our operations is helped by 100s of automated tests and 1000s of metrics. I always thought that this is DevOps, but it sounds different from what most of the people here are alluding to.
They’re tools for the job, like your compiler and the language you use to program.
That’s like saying “I like to use Python and couldn’t care less for Java”. It’s fine to disagree with the team’s choice of the tools, but one needs to eventually commit to the choice, even it’s not your preferred one!
There’s an old adage that “if you’re writing clever code then you may not be clever enough to debug it”. It’s true! Operations requires deep understanding of the code running in production, the business rules and the customer. The person who will operate your code will eventually be smart enough to develop it entirely too, eventually cutting out the “dev”. I’ve personally seen this happen over and over.
My background is large multinationals so my view here is a bit bias but i don't think cost cutting is the driver.
Large orgs get large change management processes and procedures. Over time, these change management teams become overwhelming behemoths with minds of their own.
I think "DevOps" was designed as a way to "bypass" the bureaucracy?
"We just use this CI/CD pipeline and no need to sit on a 3 hour change management review call..."
It becomes just robots pushing around paper for compliance.
If you don't want to think about AWS S3, GitHub Actions/Jenkins, Terraform, etc, .... then we need to work together. All those tools and services exist because all the software developers are sitting in their sandbox, and don't want to come out and play. The systems and tools that we run your code with... suck. A lot. We need programmers to make the systems better. We (in Ops) are a little busy with trying to just figure out how to run your apps without them falling down. We don't have a lot of time to reinvent the state of the art of computer systems.
For example, we need a distributed operating system. Not some fucked-up kludge of a monolith of microservices overseen by a company that has more engineers than brains... but an honest to god, stable-ABI, simple, composeable, stable, general operating system. We need Linux to come out of the box, ready to run distributed applications, in a way that doesn't require a PhD. Once we have that, then you - yes, you, the developer! - will be able to make applications that automatically scale so easily that we will never need to utter the phrase "container" ever again. You will rarely ever need us again, because the system will just be so simple, so general, that anybody who can use the terminal can build and deploy applications without ever learning anything outside of your programming framework.
But we need you to make that distributed operating system. Until you do, we will just have more stupid kludges, more bizarre unnecessary complexity, in the futile attempt to constrain all the crazy shit we want to do with technology, while trying to run your apps for you. Please, I'm begging you - put me out of a job.
Perhaps most of our organizations are using the wrong tools and that the way we divide responsibilities is the problem.
Imagine if:
1.) instead of general purpose languages being used by product developers we had domain-specific languages and tooling that were agnostic to things like deployment, memory-management, data storage
2.) computer engineers built the domain-specific languages and tooling and focused on things like deployment, memory-management and data storage.
DevOps is generally solving for general purpose development while developers are generally solving domain-specific problems with general purpose tools. This seems to be a large part of the disconnect!
I’m currently working on a small team of computer engineers who support a custom Turing-complete DSL for an “operations” team that both programs and directly interfaces with clients.
It doesn’t seem that different from what Apple or Microsoft do, which is write custom languages and tools for developing on their platforms, albeit specifically for general purpose programming.
I've done some, and it feels like reading an arcane text in another language. I've seen the thing I want before, so I know there exists a magical YAML incantation that brings it about but I have no idea how to even start looking for it. It devolves to an extremely slow feedback loop of "did that work? No."
I've never found anything that explained how I'm supposed to create a systematic approach to creating anything I need.
"That didn't work, and the traceback (if I read it) isn't very helpful. Guess I'll try something else I found on SO."
You learn ops by doing it, preferably in an environment where when you inevitably do it wrong, it doesn't break prod.
Starting in development requires a laptop. Starting with operations requires a datacenter worth of hardware and applications.
I'm not sure why this happens but it does and I don't like it (as somebody who's very M-shaped).
Letting them wade into others responsibilities often introduces a lot of that danger.
System administration is an entire discipline with a history and way of thinking and approaching problems at a systems level. Software engineering is the same, but with an approach that often emphasizes handling problems in the application.
Letting solutions be put into applications that are best handled at the system level (where system isn't just an OS, but might be a complex network of them), or put into the system when it could easily be handled by the application, caused inefficiency and problems.
The solution is to either hire only very accomplished people that can do both and make the right decision of where to put solutions, or hire people that can do one or the other and put them together with a few (maybe one) person that has a good grasp of both and can make executive decisions when the right time and place is for specific solutions (an architect).
The latter is easier to hire for.
Though mind you, the article touches - but doesn't go into - the real reason things go shit. It's not ops, it's not devops, because your knowledge silo can too, and in my experience (that's longer than the writers, lol) _will be_ pressured.
If the estimates are consistently too low (because honest ones would render many - maybe most - projects non-viable), the requirements are often bullshit (because otherwise the lies in the estimate would be too obvious, and everyone needs to agree on the lie) and and rewards are not the ones the jobs claims to be (in every workplace, your job is to improve your resume. That's all. That's the only thing that, materially, matters for your career. I can care about software I write, but if anything I end up punished for it) - then no amount of renaming things and shifting org charts around or motivational reading for managers is going to fix that. Nor will a Cloud or latest orchestration software.
You don't want a doctor who brags that she can do heart surgery, perform talk therapy and treat your skin rash. You want someone how knows a lot about your specific problem because she's prioritised it over other things.
Why would someone who wants to build stuff not care about where and how it runs? Operational requirements are the same as any other requirements.
In the old days, developers would also configure the design of the system it ran on.
"Alright, we're going to use some fast CPU front end nodes with smaller, faster disks, and then have a load balancer, our database server is going to have the most RAM, now let's set these resources up on a CDN to serve faster, this memacache server is really going to lighten the load..."
I think the idea was to use tools that automatically did all this work for you, but it just ended up creating another tier of people that configure this side of it.
I'm also not convinced that the average junior or mid level application dev needs to know where or how their stuff beyond some high level concepts (i.e. our app uses auto scaling).
FWIW, "Platform Engineering" used to be something different than what the author is suggesting.
DevOps has different meaning depending on who you’re talking to, even some definitions that appear similar are different in nuanced but important ways.
All “devops” as a job title has done has muddy responsibilities and given many folks the wrong impression of what an operations discipline should be.
There is also a lot of rewriting of history that gets thrown in, similar to how when people talk about cloud then the only alternative is to start making CPUs by hand and begin building your own nuclear reactors. It’s the idea of what came before, not the reality, that people seem to be defensive of.
It’s honestly exhausting to discuss.
So instead I became CTO so I can solve this mess properly, I don’t hire devops, I hire infra engineers, build engineers, release engineers and: backend engineers.
Roles so simple that you already have a clue what they do, which is sort of the point of job titles.
"Yea the DevOps guy messed up the widget and nobody notic---"
"Wait, what is the DevOps guy doing even touching that widget.... what is even DevOps to you?"
"Bro that widget IS DevOps."
-silence-
Same applies to the topic of "micro-services".
I think that's where most people go wrong. They put a bunch of services in front of a shared database, which means that they don't have to go through a service's API to get to it's data, and that's what breaks everything.
If you see a queue between two services, that is usually an indication that some ownership is being transferred (even briefly), and that is a critical point where you need introspection and observation.
Amazon's SOA architecture at least all started out about pulling bits of data spread over hundreds of servers into services that could take advantage of caching.
> An important rule for microservices architecture is that each microservice must own its domain data and logic . Just as a full application owns its logic and data, so must each microservice own its logic and data under an autonomous lifecycle, with independent deployment per microservice.
https://learn.microsoft.com/en-us/dotnet/architecture/micros...
I don't recommend either, and it's still a code smell, but with clear definitions they can work.
Btw some of the confusion might be what I mean by shared database. I didn't mean two services sharing a limited set of tables, using the database as a rabbitmq replacement. I meant sharing the backing database of the microservices. It sounds like we probably agree I just wasn't very clear about what I meant by sharing a database.
(Ironically I am literally using a database as a queue to share data between two services we are running in prod. I don't think of it as a microservice because it's not separate teams, it just because our monolith is hosted on a platform that doesn't support certain libraries so we had to role those libraries onto a separate platform that does support them.)
Using a shared database allows you to punt a lot of that complexity to a system that's been specifically designed for it, and working well for probably 20+ years.
Too many people think microservices don't have their own, severe, downsides. The likes of netflix, google, et al, can afford to pay people whose entire job is to manage the complexities of these approaches that flat don't exist in other scenarios.
But it's a hell of a lot simpler to use a single database if you can get away with it.
http://resources.1060research.com/docs/IntroductionToResourc...
Ok, I've seen this, but IME (24y in industry, the last 6 as a consultant) in the vast majority of cases, it's more like "things devs should control but CAN'T [bc CICD etc are silo'd and owned jealously by an overburdened ops team unable or unwilling to facilitate self-service]".
It only takes a few instances of people royally messing stuff up in production for their deployment rights to be stripped away – and other groups catch the fallout too, as new "procedures" get implemented.
Sure, there are some overbearing procedure monkeys who really want a process on everything, but a lot of "protection" rules are there because something really bad (and expensive, financially or reputationally) has happened before.
Having something not work during the development cycle is annoying for one person or a team. Having something break in production usually means people get paged and (depending on what you do) the resulting issues can have major societal aswell as economic impact.
In the ops world, this is especially visible if you go further down the stack. (from applications down the the network).
Network architecture moves at a toad's pace compared to webdevelopment, which is a good thing considering breaking the network will usually break every other system inside a IT department/landscape.
There was an angry Ask HN post a while back about how terrible the "new guys" are that are out there.
It then complained that the new hire wrote a horrible authentication service.
I thought it was an intentionally absurd post about the expectations put on new some rando new guy to write something important / they shouldn't be working alone on ... but they were serious.
I never miss an opportunity to share my favorite piece of comedy this decade:
https://youtu.be/y8OnoxKotPQ
There is a good talk that is also comedic as it comes from the perspective of someone who wants to fail at doing microservices.
https://www.youtube.com/watch?v=GWgRw5jiYy0
- https://www.youtube.com/watch?v=zwSNjVTF168
- https://www.youtube.com/watch?v=MnyvgFDh-kw
Some teams historically worked fine together, others didn't.
I think the tools we have these days are much better (for ops) to ship what works on developers machines.
But if your organisation had a culture of not working together then devops didn't really do much.
and anyway, you've leaned into my point about it meaning different things to different people.
Agreed. For some applications the cloud difference is significant; for many (most?) others though, "Cloud" is just rebrand of "Hosted". And even for more cloudy offerings, while I'm in a very specific and different part of IBM, some of the old timers/architects/powers-that-be keep trying to explain "We had that in 1969!!!" :-D
Agreed also at rewriting of history when it comes to development/support/operations models. My dad has been IT director and he chuckles when I talk to him about "new and exciting paradigms" which he of course sees as turning a circle to what they had in 70's and 80's :)
As someone with 20+ years in IT, I agree - a lot of these "new and exciting paradigms" are not new at all.
My personal favourite is how many large multi-nationals are now building in-house clouds?
WTF is the difference between an "in-house cloud" and a shared-use datacenter from the 1990's?
A couple million in salaries and bonuses.
An in-house cloud sounds like the mainframe installed in the raised floor computer room at the school district office I worked at in the late 90s; of course, the 12 foot long Unisys mainframe was replaced with a Unisys 4U pentium pro box pretending to be a mainframe, and then there was a lot of extra floor space.
If you're running your own 'cloud' in a (shared) colo, I dunno that that's really in-house. I guess it's still 'private cloud' though.
Can't call a mainframe a bunch of buzzwords like "Hyper-converged, high availability, on-premise software as a service cloud platform".
https://www.ibm.com/ca-en/products/z16
warning: buzz word alert!
I still have the exact same problem with respect to never having exactly the ratio of CPU to memory that would make my app happy.
The interface to the shared-use datacenter, if you're lucky, is a spreadsheet that declares the static resources you own and a remote hands guy that can tackle things beyond the capabilities of your remote KVM. If you need more capacity you need to work with the datacenter folks to order physical machines that might show up in a few months.
The interface to the in-house cloud is an API. In most instances, developers are completely abstracted away from the physical infrastructure and don't need to take a lock on some human in the datacenter to get their work done.
Not to say your scenario isn't valid and real, but I live that scenario every day today with in house cloud and virtualization too - it takes months of approvals and solutioning and security assessment and network engineering and procurement and costing and whatnot... To deploy a windows vm.
Because it’s missing all the glue. I need to self-service the VMs, the database, the load balancer, the dns records, certs, and hook them all up so I can receive production traffic all via an API I could theoretically do in Terraform.
I am guessing that in the 80s you weren't writing Infrastructure as Code to define exactly what resources you needed for your software, having it all set up automatically, and so on.
https://en.wikipedia.org/wiki/Job_Control_Language
True, they didn't call it "Infrastructure as Code" but it was used to define mainframe resources for jobs to use when they ran?
I mean … sometimes I shake my head in wonder, and other times I just shake my head.
1. Push code to an automated pipeline
2. I understand that the automated pipeline may need input from me in order to run successfully.
Beyond that, you need other people building the "platform" that they're deploying to (for the most part). Probably the ideal example here is Heroku.
Your devs shouldn't be required to do all of the actual ops stuff. They need access to an ops platform.
1.) What's probably the traditional view which is breaking down the walls between devs and ops, developers carrying pagers, etc. I.e. at least in an idealized world, there are no devs and ops--only DevOps.
2.) As you suggest (and which probably more closely matches how "DevOps" works especially in larger organizations), an internal (or external) operations team provides a platform that developers can use. Developers are still going to be exposed to some operational details, but a lot of them are abstracted away.
In my opinion, this last subset just doesn't work well for a lot of branches of operations.
Sure, it might work if you run stuff in AWS and have an operations team managing that, But what if your team is responsible for things like Storage Arrays, networking equipment?
Doing continious integration and code review on some of these is hard if not impossible
This becomes a lot more apparent once you're dealing with B2B customers and their nonfunctional requirements.
We've had situations in-house when devs built some simple microservice to handle some connection to a customers BI/DWH system - it's just 2 days of spring boot chugging to wrangle APIs around without state, nothing bad. But then, the customer started blasting that team with SLAs, backup questions like RTO, RPO, retention, regulation adherence of retention, the whole stack of IT security down to the physical access control.. That poor PO was caught just like a deer in the headlights.
This is some operational responsibility we're taking over for our dev-teams. We're providing the persistence, and this includes a defined backup and recovery strategy, a security strategy and such. And this also includes experience in dealing with these insane questionaires, and correctly pricing absurd requirements for custom backup solutions. And in fact, "we will have to schedule a discussion with ops about this" has ended quite a few of these requirements with "oh.. it's not that important". Intimidation with long job titles and appeals to external authority do work.
This makes a lot of sense, but you need to realize that this is only a tiny part of a successful organization. There are many setups like that which you describe across many industries that have failed reasons beyond the role definitions. It's not enough to establish the organization. You have to keep the behaviors in check over time when you hire a build engineer with different aspirations.
This is a great way to do it. There seems to be a correlation between unnecessary product complexity and unnecessary corporate complexity. Being direct about roles goes a long way towards simplifying corporate complexity.
That's Conway's law.
"Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure."
https://en.m.wikipedia.org/wiki/Conway%27s_law
I'm sure we're due for a new one soon which will follow the same path, since people seem to be admitting that Devops has problems on here more and more. Will that cause deeper reflection? Probably not for many orgs, because soon some consultants will dream up a new brand of silver bullet to slay the immortal monster of organizational dysfunction and that's much more exciting.
My favorite example is that of a restaurant. You hire waitors, diswashers, bartenders, line cooks, prep cooks, hosts and hostesses, and managers. And your waiters might be 100% focused on taking orders during prime dining hours, but at the end of the night, they help clean up assuming other roles. Your bartender might be busy making drinks, but they take a table or two if they can. Your prep cook might finish the prep early, and come on the line to help out. Your manager focuses on expediting everything and keeping the ship running smooth, but also can fill holes at time.
You wouldn’t expect a bartender to table every person, prepare all their drinks, cook all their food and clean up their table.
But that’s kind of what we keep trying to do, because we have new bartending tools that make it so that you need fewer and because we claim communication is harder than doing everything in one role.
Then when that person invariably gets overloaded; we hire more bartenders and complain they’re not good enough at cooking your specific dish.
1000% this.
I think it's an instance of how the conversation in software always talks about solutions as if they're end-all-be-all answers, rather than explain a problem in specifics and why that made the solution the right answer at the time.
Then that "solution" becomes a buzzword version of itself, and popular buzzwords are tools leaders use to overcome institutional inertia. Which is a good thing, to get over that inertia. But then what comes misses the insight and understanding that eventually turned into a buzzword.
I always thought DevOps the “function” just meant this, and being a DevOps engineer at a small company meant you did these with decreasing emphasis, where by the time you’re in the backend it’s just helping enforce logging, tracing, other observable components.
Has worked for me in hiring and being hired and almost everyone I know understands this.
Then they also tend to have to do their own security and spec out their own tickets and then do Project Management.
Most tech people should be called DevSecOpsBizAnalystProjectManager.
Two intelligent, sincere, experienced tech people can discuss devops, in detail, for a good length of time, and still be talking right past each other without really noticing.
Then came DevSecOps. Around that time, I switched to just Security, and while I miss the ability to make and push changes to hundreds of thousands of machines to get things done, I don't miss any of the pressure or blame that automatically got lumped onto the sysadmin shoulders every time anything went wrong, and the complete lack of appreciation of all the times nothing went wrong, that were entirely the result of a tireless, efficient systems administration team.
This reminds me of a joke my networking collegues used to tell to the new hires.
Welcome to the networking team, where we are responsible for the network, aswell as anything that runs on the network because developers never bothered to pay attention in their networking class.
It's like they'd never heard a BOFH tell someone, "Go away now, or I will replace you with a small and uncomplicated shell script."
After a lot of discussion, we mostly realized that we get the most value if we define more specific operational roles. We now have the idea of infra-ops, and product-ops. infra-ops is providing a deployment platform - a container runtime and persistences. product-ops on the other hand is responsible for deploying and running the different products from development on this deployment platform.
And this is giving people good ideas. Some products are very simple without harsh requirements. In these cases, one of the backend devs just takes over the product-ops role by setting up a deployment pipeline, a job and some migration handling. Other products are bigger - for example we are providing some of our systems as essentially managed systems to invididual tenants and customers. In such a case, parts of the product operations is with the product team - such as writing jobs, releasing containers and artifacts, and other parts of the product-ops role takes place in the managed services consultancy. These products are currently looking at wordings to split up the product-ops role according to their needs in their context. And that's totally fine - the infra-ops role is also divided depending on the system the engineer is working on. A postgres admin is an infra-operator, for example.
And all of this results in a devops oriented culture of cooperating across team boundaries through automated processes.
It's ok to have build-engineer doing release engineering if it's simple enough, but doing build engineering properly (Source control, artifact control, shared caches, dependency management of the compiler outside of the standard toolchain) is quite a large job.
Release engineering itself gets complicated when you take into consideration the different target platforms, most games get published on Playstation, Xbox and one of a handful of PC platforms (notably steam), plus the online systems and of course the development environments and internal release systems which are ubiquitous.
Build Engineering usually is the step between developers and QA, and release engineering is the bit after QA.
Both interact a lot with QA so could be rolled into the same role.
Fewer titles can avoid these discussions.
If they're not doing it or was overlooked for some reason, then that's an issue but not really an argument to not do it.
That won't be fixed with a new job title.
This is always a dead giveaway that something is a buzzword
Same for rest. Sometimes when people use it, it just means JSON + http requests. Other times it's supposed to be some kind of architectural style
Especially seniors and beyond. They force leetcode interviews they somehow pass or are grandfathered through and gatekeep "trash devs" by slamming gotchas about how the whiteboarded code discussion doesn't technically compile and how "returnAverage()" isn't a method ("did you know you have to write methods before they work? What does returnAverage() supposedly even do? Return a random character in the alphabet?")
I get excited about working in different tech areas. I'm exceptional at fixing other's bugs and maintaining code. Absolutely nobody seems to be hiring for this though. It's all about college exam trivia and leetcode.
When I do get into these companies, I have to work with people throwing around casual racism with HR joining in and f-slurs like it's corporate 4chan. I ask if anyone can help look at a critical bug I discovered, and nobody speaks up. Even when it turns out it was their last code change that caused it and they are the sole master expert in this area, and they were just working on organizing their desktop instead.
"I can try to look at the SQL problem, since nobody else spoke up." Then I'm explaining the basics of SQL to some guy sitting around blank faced and it turns out he was hired because of his 10+ years of expertise in SQL. His whole job is to tackle problems like that.
Meanwhile I'm out here looking for jobs when panic cuts happen and it takes forever because I'm drained from all the gotchas and gatekeeping in these interviews.
There's no way I could live with myself if I put this experience onto others. I lose sleep when I fail to call out someone talking over a quiet person on my team. I've never seen anybody else stand up for anybody, however.
I just kept asking questions until I found the one she missed.
I also ended up becoming a bespoke VxWorks admin because the guy who volunteered was never available, and someone asserted my code wasn't working well on VxWorks, so of course I had to know enough to do benchmarks. I fixed a few problems but the real issue was the hard drive wasn't doing DMA due to the kernel not recognizing the processor revision number.
Somewhere between those two events I realized that if my part of the project is great but the whole project is on fire, nobody cares about my stuff. I don't get points for being right and the team being wrong. I mean, I do for some people, but I still feel bad at the end of the project, and those are the 'points' I have to live with the most.
The way that played on on the latter project is that once we got our shit together, we started work stealing from our peer orgs. Volunteering to carve off little pieces of interface between us and 'take care' of this bit of data handling here and that one over there. I came to realize that the Org subconsciously knew this and any past success they had was due to self-organization and collective work-stealing.
Throw people at it, turn a blind eye to precise mission statements, and hope for the best.
At smaller companies the title tends to be only a title and many people are basically full-stack and know a bit of everything. And you're trading hats to get jobs done.
If you truly enjoy this part of the job, you can go for a freelance career - build a network and reputation as the person you call to solve the hard stuff, and do mostly that, no leetcode interviews required...
Anyway, it sounds like you've worked in some really shitty companies / shitty people.
The problem the term 'devops' is addressing, the way I understand it, isn't that of insufficient individual jack-of-tradesness, but that of too much separation between (internal) organisations. It should be perfectly possible to build a "devops" team from deeply specialized experts and turning it into a job description seems quite a stretch to me.
But you might actually want some jack-of-trades types nonetheless, because those isolated organizations weren't completely without merit: they are good at solving that problem affectionately called the bus factor. A single ops guy in a devops team is a single point of failure and the mitigation preparation of keeping some of the not-so-ops peers sufficiently in the loop will be much more dependent on organic motivation than the counterpart in a specialist company branch. There, substitutability would be much easier to ensure with formal process.
It doesn’t matter as long as communication is open.
Proper ops is all about reducing the bus factor. Procedures don’t depend on people, they depend on being good at communicating.
If your ops person got hit by a car, I would expect that everything would continue running, I would also expect to find some clear documentation, this is mandatory.
We have a way to prostitute words in this field... agile, QA, devops.
jan [at] competition [dot] company
If you want more information on the project we're working on the site is https://rennsport.gg
We're building a hard-core racing simulator game with a backend which can persist car ownership in a way that feels authentic. (IE; not just tied to a game).
I am interviewing right now and my previous experience includes devops roles and SRE roles, so I get contacted for both by recruiters. After hearing about the responsibilities and examples for these roles I can only come to the conclusion that titles are a waste of time. Even the "level" of a title of Staff, Principal, Lead, etc are a waste of time. Just call the job what it is and if you can't decide on what it is at least to an 80% level, maybe you're asking too much for that job?
I do work on Cloud Infrastructure at times, but honestly it's the smallest part of what I do. It's usually addressed in the architectural designs and I have to touch it incrementally. What I do much more often is writing tools, daemons, and services for distributed systems. I end up calling myself a Distributed Systems Software Engineer to reflect the idea that what I work on is systems and software, and most of them are non-monoliths (from a systems perspective).
Do you have any thoughts on Systems Engineers or the title that I prefer to call myself?
There are people who might refer to you as a backend programmer (if the focus is services and daemons), or platform engineer (if the focus is developer velocity).
Largely it depends what your primary focus is.
And more importantly, the backend engineers should be encouraged to be grateful for the PRs (even if it’s not acceptable for some reason). The more eyes on the code, the better.
I think developers naturally tend to specialise (at least for a time) but ultimately we all need to understand and contribute to code regardless if it’s front end, backend, build, test, …
Reader mode on Safari is a savior in these cases.
When ops gets woken up at 2am because someone in dev cut corners, what happens? Does dev feel the pain? Almost never.
The same thing happens when outside contractors develop code. They often provide a buggy and undocumented mess, and then no longer work on the project. They never feel the pain, so they're not incentivized to provide good code.
Until we find ways to align incentives, we're going to keep getting crap whenever more than one team is involved.
In the same meeting, the CTO was ranting about the instability of a service (which was also a PoC that was pushed to production before we were _also_ made to "own" it, yet never given the budget to even get acquainted to the codebase), claiming the reason for that because we devs are lazy and unprofessional.
I highly doubt making people who _responds_ to the incentives to "feel the pain" will fix much. I suspect things are more likely to get fixed if the people who _creates_ the incentives are the one "feeling the pain".
As they say in Hunger Games, "remember who the real enemy is".
This is definitely a symptom of a broken model and not what I would call devops. IMO the most important tenant of devops is "if you build it, you run it," meaning the appdev team that decided to use Postgres RDS is the one getting woken up at 2am.
It's also, in my experience, one of the best ways to reduce masturbatory engineering decisions and get people to focus on picking boring technology that works. Coding up a serverless application in Rust that's using a CockroachDB backend at a Python/MySQL shop would get a lot of engineers excited, but those people would be less excited knowing they're going to be the ones paged at 2am when this new and exciting architecture falls over in an unfamiliar way (as opposed to Python/MySQL, where a wealth of operational knowledge at the org has already been built up).
Similarly, it naturally reduces architecturally complexity. Younger senior engineers love drawing boxes of queues, multiple microservices, event buses, etc to show off their skill in creating the ultimate engineering fantasy, but once you throw enough late night operational incidents at a senior engineer, suddenly the preferred architecture becomes "an executable running on a box that I can SSH into when things go wrong."
I'm I the crazy one here? This kind of work is my bread and butter as a "devops" person. PagerDuty fires, I find the bad query, match it up with the most recent PRs to find where the n+1 got introduced and either patch it right there at 2AM with the on-call manager's approval or roll it back. Then we have a postmorterm in the morning with the team.
I'm the person positioned the best to do this work because I'm god in my little ops domain, have the most visibility and the biggest toolbox of potential fixes.
I'm 47 and what is described seems like the old problems that I thought devops solved.
Since you’re 47 (I’m 52), it’s the equivalent of hiring an “AutoConf engineer” in the late 90s.
* Infra as code
* Infra code sits in a repo
* Infra code is managed by the (senior) devs
* No separate ops team, or ops role, is needed: it's a responsibility of the devs now.
* Monitoring production: also by the devs -- if an issue arises "we" take turns in solving them as devs.
DevOps is no longer managing your own hardware, but using hardware-behind-an-API in the cloud.
Most developers have no fucking clue how Linux, networking, storage or whatever works under the hood. They know how to develop whatever stack you're at, but stuff like latency, packet loss, redundancy factors, backup policies, monitoring or other classic ops topics are completely beyond the comprehension of 99% of developers.
"DevOps" usually means some C-level execs say "fire the expensive neckbeards that have the time to properly understand a system" followed by them saying one of
- "oh fuck, someone managed to compromise a service and because no one knows what the fuck firewalls are / Kubernetes doesn't come with ones OOTB the hacker got complete control of everything"
- "oh fuck, production is down because someone fat-fingered in Elastic Beanstalk which recreated the environment, dropped the RDS database and there were no backups" (I've been personally bitten in the arse by their definition of "recreate" - all I wanted it to do was to replace the damn EC2 instance)
- "oh fuck, we're seeing insane AWS bills because someone DDoS'd us and no one created a cost limit or a sensible limit for autoscale or a simple dumb monitor that alerts someone"
- "oh fuck, we're seeing an insane AWS bill because someone got his AWS access credentials stolen and some shithead spun up a ton of g3.xlarge instances to mine shitcoins"
Other C-level execs see constant issues between "ops and dev teams" because their team leads are friends of the silo model and decide that instead of getting rid of the dumbass managers they're getting rid of the ops team because "cloud", with the exact same result.
It wasn’t always like this. There was a time when we talked about languages and OSes and libraries as if they made a difference on how much you could get done with as little people and cognitive load as possible (the claims were very much overrated, but the point was we acted like it mattered).
And then it started ballooning. It seems to me that much of that coincides with a lot of new money being dumped into the economy, and software moving past the necessary-evil-so-how-do-I-drive-down-my-costs and on to the gold rush of you must have a web app for your service that rivals a video game in complexity and visual finesse. It seems to me that as long as their is so much free money sloshing around venture capital funding, that it was inevitable that the process of making software would complexify to soak up the extra cash. After all, you can only add so many levels and varieties of managers to add value. After that comes the variegation in roles of software development.
That’s my working theory anyway.
So to prevent themselves from ending it all, they invent things to do to keep themselves sane while they churn out CRUD apps all day long for decades. Sometimes you get truly amazing software, but most of the times it's just reinventing the wheel, but worse.
Even worse, some of them become infatuated with doing things "Like Google," and errant CTOs enable them, leading to a 6 person team supporting a piece of software that could be replaced with WordPress, to the betterment of the primary stakeholder.
I've been consulting for over a decade now. The number of times a potential client has told me they've spent into the six figures (or eight) for something that free software does out of the box but better is depressing. The worst part? I usually can't convince that person they've been absolutely swindled, and so I have to let them keep on their merry way.
Or they spend the spare time bullshitting on web forums.
It's because of growth, software development today is both simpler and way more complex than before and that's entirely due to growth in the sector. In many ways it's simpler than it ever was, I'm writing an API using Lambda/API Gateway in AWS and it blows me away how quickly I am able to get services stood up and configure my API. But in another way it's so much more complicated, for this same API I spent 2-3 weeks experimenting and researching IAM roles and how all my AWS resources would interact with one another.
I would say the floor of software development has become way simpler, deploying a site to Netlify is way easier than dealing with webservers of the past. But while the floor is lower than ever before, the ceiling is in the stratosphere, with extremely complex systems that you can string together in the "Public Cloud".
> for this same API I spent 2-3 weeks experimenting and researching IAM roles and how all my AWS resources would interact with one another.
same experience here, and trying to understand/test un/poorly documented AWS behavior with public/private APIs and EC2 resources has been a huge time sink :(
It’s a laughable over reaction (pun unavoidable).
These are manic states the dev community enters, and all pragmatism is lost in our discourse on the solution.
Cooler heads are not prevailing. There is certainly a solution in the middle.
Devops is bullshit is also an over correction. It’s worth saying that making infra more accessible is a democratization of that entire sub-field. How do we preserve the good part of that?
My problem with Make and similar tools is I don't want to learn more ad-hoc syntax to accomplish something nearly trivial. But I don't know if there's any alternative.
Run: Task runner that helps you easily manage and invoke small scripts and wrappers
https://github.com/TekWizely/run
Defining commands feels like make, but comes with a bunch of extras targeted at the needs of a task-runner.
I hope you'll check it out!
I suggest to add a complete example of common use cases like running the unit tests, building targets while setting the correct options depending on the host (linux vs windows vs macos).
Are you talking about interest rates? Thanks.
Even if you're wrong, it's an interesting theory.
Is probably one of the better explainers.
Scaling sustainable software feels like an "unsolved" Problem, because the society hasn't figured out how to organize better.
I'm sure - but haven't witnessed this myself yet, so take it with a grain of salt - that if one productive developer builds an application in a year, it needs a team of 5-10 to continue development at a similar level, and even then it may not make it.
Companies need to focus on keeping software as simple as possible, well documented, and transferable. Unfortunately this also means curbing people's enthousiasm.
Yes, there is a network of small to medium sized ones.
> things that were near impossible to build in the past.
Were they necessary in the past?
So basically any organized human activity is now "capitalism"?
>Forms of efficiency are "how fast can they solve that problem"
From which follows that "new forms of efficiency" means that previously it didn't matter how fast they can solve that problem, but now, due to "moving further out on the software complexity curve", it has begun to matter?
>And complexity doesn't unlock anything.
Then what is meant by "moving further out on the software complexity curve unlocks new forms of efficiency"?
I don't like this hyper-expansion either, but when you have an army of monkeys on typewriters, well, a lot gets "produced".
Now I am working at a large org for an internal application. There is an ops team I push code to github and Jenkins runs the CI and deploys it to a dev environment. I push a button in Jenkins to deploy to production. In two years there has been one ops related issue where I had to bump up the memory limit from 2GB to 4GB. There is a single server and a database. Setup works fine for a few thousand concurrent users. My skills are in development and understanding business reqs, not mucking about in config files. There are people that are good at that and enjoy doing that, let them handle it.
I'm glad you said this. 2-dev shop here and we've used Heroku the last five years. We spend all our time developing and supporting our application and almost never worry about infrastructure. I keep getting tempted to move to K8s; Heroku is expensive for what we get and seems to have suffered some serious brain-drain. I've dabbled with K8s in side-projects and I can geek out on all the terraform and yaml stuff for days. But its probably not a great idea to inflict it on a 2-dev shop.
Couple that with a social drift toward hyper-specialization. That sort of hierarchy naturally creates minds that don't think at the systems level. This is the general theme of Buckminster Fuller's Operating Manual For Spaceship Earth: the shift toward hyper-specialization (or narrow focus) has long-term disastrous consequences.
When you couple inexperience with narrow focus, you get messes.
Part of this is to blame on the industry-standard thinking that because someone works for Company X, they're a competent, logical engineer (and should be granted authority/responsibility over essential products/projects). This is the downfall of "code tests" and "whiteboard coding." They don't evaluate for systems level thinking and so a developer with poor creativity and logic skills slips through the cracks because they're great at eeking performance out of a function which impresses the lollipop guild.
[1] https://blog.cleancoder.com/uncle-bob/2014/06/20/MyLawn.html
There are thresholds of effort for certain things at largely constant levels. E.g.: designing a new language, build tool, web application framework, or database all have some minimum effort that needs to be invested. In the past, there just weren't enough developers in the entire world to "overdo" these things, so there was a relatively small pool of languages, tools, and frameworks to choose from.
Now, individual corporations have armies of junior developers spitting out frameworks and query languages like a machine gun. There's so many now that you or I haven't even heard of 99% of them!
As the number of available developers grows exponentially, so does their capacity to "reinvent wheels". Their ignorance of existing wheels to they could be reusing grows exponentially also. The result is an exponentially exploding set of ad-hoc, incompatible systems.
In my recent semi-DevOps, semi-Cloud-Engineer role I've come across an absolutely bewildering array of tools even when I've generally restricted things to one cloud and one language's ecosystem. Heaven help you if your project has multiple languages!
This is my theory too, and as someone learning to program (hopefully as a career), I'm very worried what a new paradigm of tightening will do to the field. I feel like LOTS of people who think their stack is secure are going to get dropped because at the end of the day, lots of the B2B SaaS doesn't actually deliver anything to the world of material needs.
Meanwhile people are now spending millions of dollars and years of person-hours building MVPs in the cloud that won't ever go anywhere because the business model sucks.
Focus on delivering value to your customer first. How you get there is quite literally irrelevant to your customer.
This is us. We have a couple thousand customers in a B2B space with large revenue per customer. Growing rapidly and our node.js stack is running on a tiny EC2 instance and RDS server.
I spent a lot of time in the last couple years trying to fight back against attempts to overcomplicate this by using new services & new tooling. I just see it all as stuff that's going to slow down the single most important thing to us from a business perspective - which is writing new features to optimise our internal workflows - and give us more headaches from an operational perspective.
Even the stack we're using feels ridiculously over the top and complicated compared to a basic LAMP application. Everything in node.js feels like a huge pain in the ass. If there's not a node module you can install to immediately solve your problem (which of course just adds a different set of problems) even writing basic things feels exceptionally arcane and time consuming compared to a basic PHP implementation of the same thing. Maybe it's just a side effect of our dev team size (currently very small, only five full time devs) but the overhead of it all just feels like it's not worth it.
"The best minds of my generation are thinking about how to make people click ads."
https://quoteinvestigator.com/2017/06/12/click/
Speaking personally, I used to work on a Platform Engineering team for a major multinational that rhymes with ay-do-bay. And their requirements for the workloads that we needed to be built and hosted where very unique because you have teams running completely different languages and toolchains, esp. from acquisitions. I can argue there that the template K8s setups wouldn't work.
In the author's case, the only reason why they feel the way they do is because we finally have some sense of standardization on what Infra looks like for most companies. It's no longer a question for most folks if they should adopt Docker, we have accepted images into our lives. Same for K8s (after a certain point of scale). So uh... yea. Catchy blog title to sell some thin layer on top of K8s in which it doesn't solve the root issue that they are talking about.
The job of DevOps is not to close tickets. That'd be like driving a car by shouting directions at someone lying on the floorboards holding a wrench to the steering pinion.
The job of DevOps is to build a steering wheel (and ideally, teach SWEs how to drive... at least enough that they understand what a "road" is and why it's a pleasant experience for everyone if you stay on it. If the road doesn't go where they need to be, then it's time to file a ticket, but that ticket had better be "Build a new road," not "Offroad this one car to the cabin in the woods and call it a job well done").
The raw hardware of an enterprise deployment is so flexible it solves nobody's problem. DevOps is in the business of writing the operating system for a mega-computer physically represented by hundreds to possibly millions of heterogeneous computers. It's a process of continuous growth to make that work.
You want us to run and manage your software? Sure, here's a checklist of what it has to conform to. Oh it's unstable? We will no longer run it for you, here's the pager back.
Glad to finally see someone else saying this.