It's actually the perfect litmus test, and a good way to check someone's seniority or competence in in about 5 minutes flat:
Developers who disclaim responsibility, or show no interest in, how their software is scaled, securely locked down, deployed or monitored are probably not that great. Or, perhaps, they simply haven't been humbled yet by a bad incident, miscommunication, or a complete lack of understanding of how real life works.
I worked with a Microservices Architect (and developer) once, and when I asked how he intended for failures to cascade properly between services; how he would maintain cohesive state, especially if things went wrong; and how he would diagnose problems between them, he confidently told me: "That is for devops to work out."
Back in the day of desktop apps, the people who showed no interest in how their software was installed on desktop computers were more or less the same ones that are now disinterested in operations, deployment, etc.
Whenever speaking with pure developers, inevitably I will hear this sentence:
"I just want to...."
It's terribly rude, but I stop paying attention, because inevitably the rest of the sentence can be translated to:
"... ignore my responsibilities, such as security, observability, availability, privacy, security, recoverability, documentation, scalability, and long term supportability.
I would say the context of such a sentence is usually something along the lines of:
"Because X wasn't designed with <problem> security, observability, availability, privacy, security, recoverability, documentation, scalability, and long term supportability </problem> in mind, to just do Y it is now also my responsibility to do <problem>"
What answer did you want to hear? Mine would be "Goes to dead-letter queue to be reprocessed OR fails, logs to central logs, team gets an alert." Haven't seen "distributed logging" magic work in practice. I have co-workers who spent years on tracing request ids through each layer. It got scrapped after a couple of months because no one used it.
That sounds cute to you, but it's a huge waste of time for most applications. Pulling out the banking example is a cliche at this point.
If it's a move money issue - that's solved - transaction to credit/debit. Reject and log if failed. Maybe with a retry. Maybe from a queue.
If it's financial documents moving, they go through a serious of queues and get dead-lettered and handled appropriately.
The kicker? All of these have a customer / transaction ID and don't need some special request id "traceable throughout the system" - it's traceable as is, search for it by customer / transaction ID.
This even applies if you are using K8S for no reason. The only company I know that's justified in using K8S is an AWS/GCP/Azure competitor in the ML space needing to spin up VMs instantly at scale.
k8s saves a TON of time if you need to run on more than 1 machine and you use a managed solution (by AWS/GCP/Azure) and only use it as a glorified docker container orchestrator. Ingress solves how to route HTTP traffic coming into your domain. PersistentVolume solves how and where to store data. LivenessCheck solves how to restart your crapware when it freezes. Just don't go crazy with it unless you actually need the fancy stuff.
"k8s saves a TON of time if you need to run on more than 1 machine and you use a managed solution (by AWS/GCP/Azure) and only use it as a glorified docker container orchestrator."
How? Just use AppService / EC2 for managed machines. Scales just as well.
"Ingress solves how to route HTTP traffic coming into your domain."
This is actually helpful and "easier" out of the box than putting nginx in front of statically hosted sites, so I will reluctantly give you this one.
"PersistentVolume solves how and where to store data."
You need a DB most of the time. Managed DBs are easier.
"LivenessCheck solves how to restart your crapware when it freezes. Just don't go crazy with it unless you actually need the fancy stuff."
Our crapware doesn't freeze. Needs a custom Health Check endpoint. Can just as easily restart the instance when that check fails.
K8S is a ton of extra overhead for no gain. We have this debate at work all the time. Great resume builder though.
> How? Just use AppService / EC2 for managed machines. Scales just as well.
It's not about scaling. It's about being able to make a docker image, run that locally on your developer laptop (even if it's a mac) and running the exact same thing in production. Sure you can use docker on top of EC2 but why would you? Just use the ready to go solution you can just buy (managed k8s) and avoid the lock-in. Again, it's critical that you use managed k8s. Don't roll it yourself on top of EC2.
> You need a DB most of the time. Managed DBs are easier.
Sure. Using k8s for your app isn't stopping you from doing that if you want. We deploy 1 database server per testing environment, which kubernetes makes easy to do. For production we use a managed database, but they're too expensive for testing environments.
> Our crapware doesn't freeze. Needs a custom Health Check endpoint. Can just as easily restart the instance when that check fails.
Mine doesn't either. But I have to run plenty of software that isn't mine.
> K8S is a ton of extra overhead for no gain
If you used managed k8s you're just trading server management for kubernetes resource management. I've migrated our system from Ansible + Cloud VMs to kubernetes several years ago, after running it for years before, and things have become significantly easier to maintain and our costs are roughly the same as before. Less vendor lock-in, too. It would take me an about an afternoon to switch between any of the major managed kubernetes offerings, and some of our clients have expressed preferences for a specific cloud.
Hey, thanks for the reply. I am being slow because it looks like if comments are deeply nested, the "reply" cooldown gets longer, so I am just replying now.
We use .NET/React for the most part, so I haven't had an issue with Mac/PC - we have multiple teams and, for example, a team helping is all on Macs - they can run stuff fine from bash. Azure Data Studio let's them access Postgres (and would let them access SQL Server) if needed. React only has issues with file pathing sometimes, so most of the time it's fine. I find that devs also get lost in setting up Docket/K8S locally, especially when people roll scripts to set up the whole environment in one command.
Still, I am genuinely arguing to help me fill the gaps where I might be incorrect, so I really appreciate your replies.
On the flip side I've seen dev teams grind to a halt because of how much difficulty they had tracing a request end to end in their system.
Granted, they had entirely too many separate web services, and that was really the root of their problem, but being able to trace a request all the way through your system isn't useful until it is (meaning, when shit hits the fan).
Most enterprise developers aren’t allowed to know the answers to your questions as none of them fits into a feature rooted in a quarterly OKR. For most enterprises developers it takes a couple of weeks to create a git repo because it requires a service ticket to Devops.
> I worked with a Microservices Architect (and developer) once, and when I asked how he intended for failures to cascade properly between services; how he would maintain cohesive state, especially if things went wrong; and how he would diagnose problems between them, he confidently told me: "That is for devops to work out."
JFC, that's an incompetent.
I know a lot of people will consider that too strongly said, but I don't think so. Any jackass can succcessfully build for the happy path, it's precisely where the happy path isn't that real skill comes into play.
The reason I don't want to do devops where I work is because the current devops guys, by equal parts both low skill and insane business requirements, have made a flaming garbage pile.
The problem is that so many of the operational concerns has to do with the code. I’m on the OPS side and I’m trying to find the right balance but it’s so much more work for me to the work if I don’t have the context, not to mention the fact that lots of operational considerations require code changes. I do think this means it’s the same in both directions though, if you want to be on the “ops” side and don’t want to touch or at least read code, you’re going to have a bad time.
Most places I've worked, <50% of devs could do ops. They dont know systems.
We give them some exposure & some pick it up slowly over time. Unlike learning, say, React, most systems are pretty different. Everywhere I've gone has had very different CI/CD & different platforms for running our workload. There's a ton of fundamentals about computing & containers & building (compiling, bundling, packaging) software underneath, but usually the tools are working at higher level & those fundamentals are sparsely seen/encountered, so while a dev might start to skill up in the local ci/cd/platform, the experience only crudely translates l, they have to start way way back, if they go work somewhere else.
And then there's the question of architecting. Knowing all the varuous bits & pieces. Our apps use sql, redis, elasticsearch, an object store, an esb... good luck being able to effectively understand & operate all these machines, newly minted senior sev.
The underlying problem is that ops doesn't scale well unless they can use generic abstractions over things like reliability, scalability, security, etc. Those generic abstractions require applications to adhere to ops best practices, or they don't work (e.g. Kubernetes can't scale a stateful app).
Without those generic abstractions, you need a ton of ops people. Enough that there are at least a couple members of ops with in-depth knowledge about each app, and each ops member can only realistically know <5 apps that well. Really, you need several ops members to know each app that well, because that style of ops tends to burn people out rather quickly and they quit. Ops also has to be intimately involved in the design process, and have some kind of veto power to say "you can't build a stateful app, then throw it over the fence and demand we somehow make it scalable".
With the generic abstractions, you need fewer ops people, but they really only know the generic abstractions and can't help diagnose errors via log messages.
A lot of (bad) devs want a world where ops is just held captive by dev. Ops has no authority to say no to the design process, and no ability to say that some things aren't possible with the application architecture. It just doesn't work. Even if ops can somehow do things like remove state from stateful applications, everyone capable of doing that work is going to quit. It's not fun or interesting work to do, it's exhausting and frustrating to expend man-months of effort on a problem that could have been solved with man-hours or weeks of development effort.
> “Fundamentally, it’s what happens when you ask a software engineer to design an operations function,” Ben Treynor, vice president of engineering at Google and the godfather of SRE, is often quoted as saying.
A foolproof plan. Much like hardware, software is generally known for its reliability, interface standardization, and design stability.
23 comments
[ 3.7 ms ] story [ 42.5 ms ] threadDevelopers who disclaim responsibility, or show no interest in, how their software is scaled, securely locked down, deployed or monitored are probably not that great. Or, perhaps, they simply haven't been humbled yet by a bad incident, miscommunication, or a complete lack of understanding of how real life works.
I worked with a Microservices Architect (and developer) once, and when I asked how he intended for failures to cascade properly between services; how he would maintain cohesive state, especially if things went wrong; and how he would diagnose problems between them, he confidently told me: "That is for devops to work out."
Back in the day of desktop apps, the people who showed no interest in how their software was installed on desktop computers were more or less the same ones that are now disinterested in operations, deployment, etc.
"I just want to...."
It's terribly rude, but I stop paying attention, because inevitably the rest of the sentence can be translated to:
"... ignore my responsibilities, such as security, observability, availability, privacy, security, recoverability, documentation, scalability, and long term supportability.
I would say the context of such a sentence is usually something along the lines of:
"Because X wasn't designed with <problem> security, observability, availability, privacy, security, recoverability, documentation, scalability, and long term supportability </problem> in mind, to just do Y it is now also my responsibility to do <problem>"
Excuses are just that: excuses.
"Hey bob, it seems the money we sent through the system disappeared. Can we trace it?"
"Sorry no, we got rid of the reconciliation id we use across our systems."
If it's a move money issue - that's solved - transaction to credit/debit. Reject and log if failed. Maybe with a retry. Maybe from a queue.
If it's financial documents moving, they go through a serious of queues and get dead-lettered and handled appropriately.
The kicker? All of these have a customer / transaction ID and don't need some special request id "traceable throughout the system" - it's traceable as is, search for it by customer / transaction ID.
This even applies if you are using K8S for no reason. The only company I know that's justified in using K8S is an AWS/GCP/Azure competitor in the ML space needing to spin up VMs instantly at scale.
How? Just use AppService / EC2 for managed machines. Scales just as well.
"Ingress solves how to route HTTP traffic coming into your domain."
This is actually helpful and "easier" out of the box than putting nginx in front of statically hosted sites, so I will reluctantly give you this one.
"PersistentVolume solves how and where to store data."
You need a DB most of the time. Managed DBs are easier.
"LivenessCheck solves how to restart your crapware when it freezes. Just don't go crazy with it unless you actually need the fancy stuff."
Our crapware doesn't freeze. Needs a custom Health Check endpoint. Can just as easily restart the instance when that check fails.
K8S is a ton of extra overhead for no gain. We have this debate at work all the time. Great resume builder though.
It's not about scaling. It's about being able to make a docker image, run that locally on your developer laptop (even if it's a mac) and running the exact same thing in production. Sure you can use docker on top of EC2 but why would you? Just use the ready to go solution you can just buy (managed k8s) and avoid the lock-in. Again, it's critical that you use managed k8s. Don't roll it yourself on top of EC2.
> You need a DB most of the time. Managed DBs are easier.
Sure. Using k8s for your app isn't stopping you from doing that if you want. We deploy 1 database server per testing environment, which kubernetes makes easy to do. For production we use a managed database, but they're too expensive for testing environments.
> Our crapware doesn't freeze. Needs a custom Health Check endpoint. Can just as easily restart the instance when that check fails.
Mine doesn't either. But I have to run plenty of software that isn't mine.
> K8S is a ton of extra overhead for no gain
If you used managed k8s you're just trading server management for kubernetes resource management. I've migrated our system from Ansible + Cloud VMs to kubernetes several years ago, after running it for years before, and things have become significantly easier to maintain and our costs are roughly the same as before. Less vendor lock-in, too. It would take me an about an afternoon to switch between any of the major managed kubernetes offerings, and some of our clients have expressed preferences for a specific cloud.
We use .NET/React for the most part, so I haven't had an issue with Mac/PC - we have multiple teams and, for example, a team helping is all on Macs - they can run stuff fine from bash. Azure Data Studio let's them access Postgres (and would let them access SQL Server) if needed. React only has issues with file pathing sometimes, so most of the time it's fine. I find that devs also get lost in setting up Docket/K8S locally, especially when people roll scripts to set up the whole environment in one command.
Still, I am genuinely arguing to help me fill the gaps where I might be incorrect, so I really appreciate your replies.
Granted, they had entirely too many separate web services, and that was really the root of their problem, but being able to trace a request all the way through your system isn't useful until it is (meaning, when shit hits the fan).
JFC, that's an incompetent.
I know a lot of people will consider that too strongly said, but I don't think so. Any jackass can succcessfully build for the happy path, it's precisely where the happy path isn't that real skill comes into play.
We give them some exposure & some pick it up slowly over time. Unlike learning, say, React, most systems are pretty different. Everywhere I've gone has had very different CI/CD & different platforms for running our workload. There's a ton of fundamentals about computing & containers & building (compiling, bundling, packaging) software underneath, but usually the tools are working at higher level & those fundamentals are sparsely seen/encountered, so while a dev might start to skill up in the local ci/cd/platform, the experience only crudely translates l, they have to start way way back, if they go work somewhere else.
And then there's the question of architecting. Knowing all the varuous bits & pieces. Our apps use sql, redis, elasticsearch, an object store, an esb... good luck being able to effectively understand & operate all these machines, newly minted senior sev.
Coding is way more insular.
I don't want to do ops, but I will if they're holding back development.
Without those generic abstractions, you need a ton of ops people. Enough that there are at least a couple members of ops with in-depth knowledge about each app, and each ops member can only realistically know <5 apps that well. Really, you need several ops members to know each app that well, because that style of ops tends to burn people out rather quickly and they quit. Ops also has to be intimately involved in the design process, and have some kind of veto power to say "you can't build a stateful app, then throw it over the fence and demand we somehow make it scalable".
With the generic abstractions, you need fewer ops people, but they really only know the generic abstractions and can't help diagnose errors via log messages.
A lot of (bad) devs want a world where ops is just held captive by dev. Ops has no authority to say no to the design process, and no ability to say that some things aren't possible with the application architecture. It just doesn't work. Even if ops can somehow do things like remove state from stateful applications, everyone capable of doing that work is going to quit. It's not fun or interesting work to do, it's exhausting and frustrating to expend man-months of effort on a problem that could have been solved with man-hours or weeks of development effort.
> “Fundamentally, it’s what happens when you ask a software engineer to design an operations function,” Ben Treynor, vice president of engineering at Google and the godfather of SRE, is often quoted as saying.
A foolproof plan. Much like hardware, software is generally known for its reliability, interface standardization, and design stability.