Perhaps all this description is showing is that in many organizations there simply is a genuine need for a "Developer IT" support function with appropriate skills and resources, and because there isn't one, it's being done haphazardly by teams who aren't a good fit for it, as the author describes. If there's a few niche issues then that's solvable by e.g. dev training, but if the issues are systematic as the article asserts, then that's an organizational problem that needs an organizational solution. If your company can't ensure that devs are capable and/or motivated to troubleshoot issues that work on their laptop but don't in a real deployment, then your company needs some "internal consultations" mechanism to connect them to someone who does have this capability and can explain and/or fix the issues for them.
Responding to "Someone will always have to own that gap and nobody wants to, because there is no incentive to. Who wants to own the outage, the fuck up or the slow down?" with "Not me." is not sufficient, it's a very valid question for which any organization definitely needs an answer pointing at some specific people - if it's not going to be pure ops people, it's IMHO not going to be the feature-developing devs as well, that would likely need separate 'site reliability engineer' teams as some major companies do.
I agree that something needs to change at the organisation level in your case, but i think it's hiring and promotion. This "developer IT" stuff is part of a developer's job. Juniors won't join you knowing it all, but they can learn it from seniors on their team who do. If you are recruiting seniors who don't know this stuff, stop, and if you are promoting juniors to senior before they've learned this stuff, stop.
I had to escalate to basically a CIO of a fortune 500 company for someone to take a look at the network performance of our system, all teams were blaming applications despite the evidences. It ended up to be a bug in a VMWare driver that was impacting their whole infrastructure.
I would add reasonable retry logic also. I've seen quite a lot of outages that would not have been noticed if there were decent retry logic with backoff, etc.
At one point running wireshark and reviewing network traces with developers was a full time job. Guess what percentage of time it was actually a network problem?
Just today I saw another “works on my machine” issue. The dev didn’t complain for 3 weeks that his latest code isn’t deployed. QA found out today (on a Friday) about it and the dev has his day off. The issues were not hard to fix, but it’s not the DevOps job.
Especially when the dev wanted to migrate from Java 8 to Java 11 and didn’t even attempt to lookup our documentation on how to change JVM parameters.
>developers are not incentivized or even encouraged to gain broader knowledge of how their systems work
This is the crux of the problem. Coding in isolation. Replies of 'It's java, it should work anywhere' etc.
The other gear grinding commom theme is not even doing basic troubleshooting. To the point of not even googling the error message or the symptoms, and being 'blocked' because they are waiting on a ticket they opened with the 'other' team.
We’re a small company so we sometimes do many things, but it’s taught me a lot of networking fault finding.
There’s some very clever ppl that know all about how networks/vm stuff work, and I’ve learnt enough from them that I can fix most of my own infra related things - or at least give them a run down of what I’ve done first to save them some time.
It got me back into hardware and networky stuff, so now I’ve got a MikroTik at home, some proxmox machines, Tailscale network etc - more fun than just spin up a box on DO and be done with it.
A lot of ppl just aren’t interested though, they just want to code (and maybe learn a new language) but because a lot of stuff is now PaaS and it’s super easy, there is no need to learn it (in their eyes)
I think incentive for developer is to be relevant. If you don't do it, someone else will. And that becomes the new norm. Like how DevOps has become the new norm.
This is a great showcase of the silo mentality and split.
One should not build silos where experts sit.
One should participate in a team of many different experts.
If you still have to call "whatever department" for fixing your slow SQL query, your disk space, your repo access, production deployments etc. Then in most cases I feel you should get out and find a place where silos are not being exercised.
Is it? In highly regulated environments, silos are practically a requirement. Security access controls are intentionally put in place to limit access to systems and their respective pieces. If you need repo access, or access to the CI pipeline, or access to the database, you have to go to the appropriate channel.
The scale of security necessities in organizations runs from:
- no need for anything but minimal security, perhaps because the business is trying to surf the margin between their AWS bill and their Google ad revenue.
- there is only a need for security in the part of the business that deals with money
- some stuff that the users do, they would prefer to maintain integrity but they don't care a lot about confidentiality
- the users want reasonable confidentiality, too
- everything about the business is money or secrets
Where your business is on that scale determines how much you are regulated and how many internal gatekeepers are necessary.
I'm writing this as a DevOps/SysEng/whatever you prefer: I think there are two big type of developers groups in the industry, and two ways to think about these issues.
The first group advocates for an ultimate "NoOps" world where every team, composed only by SWE, is responsible for the whole code lifecycle, from inception to deployment to maintenance to decommission. This is aspirational in many, many companies and probably true in a bunch of really good companies. Anyway I still wonder if there aren't tensions there between product/business asking for developer power to bring new features/changes in and developers doing the Operations work beside writing business code.
The second group is formed by the developers described in the article, which just focus on their code and need a "developer helpdesk" for everything else related to actually deploying and operating the software written. IME this is how the vast majority of companies work, especially the "normal" ones. Some developer steps "up" and try to understand/do this extra work and they usually rise to the top because they are the "good" engineers.
Yes, absolutely, but it's not actually widely implemented. Orgs does "DevOps" but they are just automating/writing as code some things that previously were done manually by Ops/SysAdmins. Now we have DevOps roles doing that same work but with other tools (Terraform, Cloudformation etc), much more automation, less gruntwork and toll but STILL used as "developers IT" nonetheless.
This isn't true at all, though. Many organizations, especially smaller ones, seem to have taken it this way. Everyone is responsible for everything. But that wasn't the DevOps thesis. The idea was for eliminating silos in the sense that dev and ops would talk to each other continuously throughout the software lifecycle, with developers understanding operational challenges and creating software that didn't just meet functional requirements, but was easier and faster to deploy, change, rollback, and troubleshoot. Ops would understand the pressures of development and create automation systems catered to the specifics of their software stack.
It wasn't supposed to mean no more division of labor. Division of labor is a key innovation in human society that enables civilizations to exist. It was supposed to mean the teams in different categories of labor interact throughout and consider each other's needs, and not throw shit at each other over a wall and only ever interact through a ticketing system.
I believe that it was supposed to mean no more division of labour, and i believe that based on conversations with people who were early adopters of it ten years ago.
This waffly "breaking down the silos" stuff is a later redefinition, i believe reacting to the fact that the original meaning was extremely unpalatable to existing organisations, with existing employees and hierarchies who would be severely disrupted by it.
But I still challenge the fact that it can really work well for normal performers and not only top notch developers. Taking care of all the phases of a software lifecycle is not an easy task, it has a big enough cognitive load. I understand that this is supposed to be done in (small) teams, but within those teams you still need some degree of separation of labour, you cannot have everybody having wide and deep knowledge of everything software. Well, you can do it, if you are Google and the like and set the bar for hiring really, really high. But that doesn't apply to most organizations and engineers. Not all of us are wicked smart.
From Patrick Debois in the interview "Later, I saw a talk by Jean-Paul Sergent about developer (Dev) teams and operations (Ops) teams working together."
So no, breaking down the silos was baked in from the beginning.
The original thesis was better cooperation between developers and sysadmins (ops). It didn't focus on trying to make operations redundant or transfer every sysadmin into a SWE.
The team that writes the code should also deploy the code and get paged in if there are problems in production.
That creates a tight feedback loop that requires developers to learn and manage the whole stack, code defensively, and test enough to be confident to deploy to production.
Didn't test your code enough? You will be paged in the middle of the night to fix it. It creates a strong incentive to make good decisions because you will be living in the mess you create.
In those cases, though, it's usually a team with some Dev people who know or can learn some Ops and some Ops people who know or can learn some Dev. It's not just laying off all the sysadmins, network admins, stack architects, and all then letting the developers freefall until they find a way to right themselves.
And this only works if your Ops team provides good tooling for deploying, logging, monitoring and alerting.
I have heard of companies deciding to do 'devops' and it turns into a free for all of dev teams having to handle/build things end to end. Everyone loses in that scenario.
To put some color on the "NoOps" world: it is not that product engineers are directly touching cloud or metal. We have an infrastructure group. It ships a PaaS. It doesn't get involved with a specific tenant of that PaaS unless a product engineer has evidence that there's something wrong with it & escalates a page. Product engineers click the deploy and rollback buttons for themselves.
> every team, composed only by SWE, is responsible for the whole code lifecycle, from inception to deployment to maintenance to decommission
This is how it is at my startup. All of the engineers are involved in managing the infrastructure for everything we build. I find it gives me much better insight into my app, and the feedback loop is much tighter since I am in control of everything.
well, they have to. Does not mean they do all that. They just skip the hard parts and focus on what is important in order to survive.
large organisations usually have a much bigger responsibility and is held to higher standards, frequently audited and controlled to stay within rules and legal compliance
100%. Over the last two years, a majority of my "sysadmin" work has been devoted to audit and compliance tasks. Mostly validating and working with auditors, but also making significant changes to work processes.
If you've got 2 developers, they're both doing everything and on call 24/7 and all have read/write access to everything on demand.
If you've got 200 developers, you're going to start wanting a team of shift workers keeping an eye on the systems, and maybe you won't want every developer to have read/write access to production data.
If you've got 20,000 developers your working practices and infrastructure are almost completely cemented in place, and anyone who doesn't like them has already left because it's easier to change jobs than to get 20k people to change their behaviour.
Having worked in Operations in some form or another for the past ~20 years this articulates so well the feelings I have been increasingly having over the past few of those 20 years.
Now I manage a small operations team and we experience pretty much all of the issues highlighted in the article.
There needs to be a rethink of how infrastructure, development and deployment is handled.. maybe the solution is to slow things down and insert a little carefully thought out bureaucracy between the layers (can't believe I'm advocating for more bureaucracy!)
From the article: ”Often they have not even bothered to do basic troubleshooting, things like read the documentation on what the error message is attempting to tell you.”
This has been the bane of my work happiness for a while now. I keep having to tell junior devs to actually _read_ the fine error message, just in case it actually _contains information about the error_, you know. Not that it seems to help much, it’s like they can’t get the concept into their heads.
This is 100% a problem with younger, bootcamp-”educated” devs, in my experience. I know the common wisdom on social media is ”no one reads text anymore”, but if that includes aspiring developers, it might be tough to replace the current workforce when that day comes…
"lack of newness" is a characteristic many will expend untold hours to extinguish. to my perspective, the "rewrite it in Rust crowd" is the peak; all non-Rust code is soiled, and worthy of replacement.
(it is very possible that the "rewrite in Rust" movement is just a guerrilla marketing project)
I don't really care for Rust, but all non-memory safe code could benefit from being replaced. This does, for some people, mean rewriting it in Rust because C and C++ make it harder to achieve the goal of memory safety.
It was not about 27 years ago that the senior sysadmin at my first real job told me: "Most young people don't have any hacker spirit at all, they just give up the first time they see an error message. The world is not going to survive that."
I was doubtful that this was a universal truth then, and I think it's the same now: there are a lot of people who do mediocre work, and they are and were supported by a smaller group of people who do really good work. And the world keeps turning.
One of the joys of the Internet and of open source is the increased ease of sharing ideas and solutions.
> they are and were supported by a smaller group of people who do really good work
I think the only solution for this is when "support" includes education. In the simplest form, we can give support by helping that colleague to find the issue himself, rather than giving the solution directly.
In a more advanced form, you're making structural changes to your company. Like in how you share knowledge with the team.
Reality check: they run off and nag someone else...very discreetly...until someone caves and fixes it for them. If they get enough pushback they will just quit.
I think that education ought to be available to everyone, but spending effort trying to teach someone who doesn't want to learn is a waste of everyone's time.
We've all heard the stories where someone bright joins a new workplace, is assigned drudgework and after a bit automates it until they only work two hours a week? That's someone who is willing to learn, and everyone else in the office was willing to experience boredom in order to avoid learning.
For all I know, some of those people would have perked right up if the subject matter were milling, millinery or masonry moving millier-weights of stone. Not everyone is interested in the same things, and even when their job is all about it, sometimes people aren't invested in it.
If it works, everyone involved wins. Sometimes it doesn't work.
Keep at it: some crusty guy telling me to just read the sodding error 12 years ago opened my eyes.
I think there's something about certain kinds of tools that are cryptic, unpredictable, and frustrating that can teach you to be helpless - to just Google and hope. It's fixable though.
I do have one anecdote/counterpoint. At my company we do DevOps, but we have a central group that creates the templates we are supposed to use for various AWS resources, build plans, and deploy plans. It makes it extremely frustrating to troubleshoot issues specific to the templates or how you configured them because they usually aren't something you can google and the documentation for them is extremely basic. Sometimes you have to ask the people who created them since they have the deep understanding and experience.
I agree that it has to be done to have consistency. I'm just pointing out that when using IaC designed by another team where the internal workings are largely hidden, us primarily dev guys are going to need some help with issues that appear to be ops related. (And yeah, occasionally root cause will be that I'm just an idiot)
I've been contracting at a company for a few years and things progressed from manually setting up servers (before I was involved) to using Ansible to provision multiple servers and now it's transitioning to a Kubernetes cluster and infrastructure as code from day 1 along with tomes of documentation to go with it.
It really is worth it to go the extra mile and write comprehensive docs, even going as far as writing them in a conversational tone as if it's a blog post or a book. I'm really happy I found a company who treats documentation and workflows as first class resources.
For a small team where only 1 person is working on this it helps eliminate the bus factor and it also makes it easier to have non-hardcore ops folks do code reviews on your IaC. Having them be able to get the gist of it with a little bit of background knowledge is so much better than nothing. All of this results in higher reliability of the services your company offers.
I have experienced this same issue many times. But from my experience, I cannot simply pinpoint it to young developers or their education. I have seen this behavior with several older people, university educated, with many years of professional experience. So not just an age/generational thing, in my opinion.
Maybe they have seen so many red herrings that they don't even trust that the error message could contain something useful and relevant? Or maybe they just learned to skim through everything, and don't actually read stuff.
I also don't understand why, when they ask for help, they can never be bothered to say what they're trying to do, what error message they got, etc. It feels like they're doing me a favor when I try to help them fix something.
Right. When I was learning to code a more seasoned dev told me "The code doesn't do what you want it to do, only what you _tell_ it to do."
Really helped me gain the mindset that not only was it my mistake that resulted in code not running, but that it was fixable. Like a game of ping pong. You hit the ball, sometimes the compiler hits it back.
Then there’s the StackOverflow effect: where you ask a reasonable question X and get a bunch of upvoted condemnations along with directions to do Y instead.
I think it comes down to passion and curiosity. Those are two things can happen at any age and education. It is also something that ebs and flows based on energy levels.
I think the same. It's so typical everyone here tries to pin characteristics to exact professional groups. The amount of anecdotal evidence here is too damn high.
I think people don't read error messages if fixing the issue is beyond their immediate capabilities.
Computers are scary things that fail in counterintuitive ways. When the handle of your tea cup breaks, the issue is intuitive and most people will be able to understand why it is happening and how to work around it(handle it carefully from the top end end enjoy your tea?).
But when it comes to computers, often you need deep understanding of its inner workings to make sense of your observations of problems. Why Xcode would say that it failed to compile my project because usefulExtensions.swift already exists? What it is supposed to mean, I see only one file with that name? That information gives intuitive idea about the issue only if you know how the compiling process works.
Why would I know why the package couldn't be found? Unless of course I know how that package manager works. Then I can check if the package manager is configured to look at the correct places.
Most error messages are like that. Instantly makes intuitive sense if you know how everything is glued together and makes no sense and needs study if it's outside of you domain of expertise. No one reads error messages unless they can recognise the pattern instantly and there's a data(like the name of the variable) guiding you to the fix.
> Computers are scary things that fail in counterintuitive ways.
Not being scared of the tool and believing one's inherent supremacy over it must be the most basic criterion for practicing this craft, but these days this fear is nursed, at times encouraged, at times even exalted (corollary of the failure fetish) especially by those who publicly place themselves as ambassadors.
Any introduction to computers must start with the statement that they are all heaps of plastic and sand and the only things they are able to do are because some mortal sat down and spent time figuring it out.
People starting out now are at a disadvantage because their first encounters happen mostly through extremely polished looking apps and it is hard to see at the outset how one could go from weird incantations in a text editor to that.
I think it's harder now than it used to be. When I started programming, any error message you saw would almost certainly originate from the beige box on the desk in front of you, which had only a single CPU core and often didn't really do multi-tasking. Over time, our computers, operating systems, applications and networks have become vastly more complex, with the effect that you can't easily build an intuition about which component is responsible for a failure and why.
Personally, I love debugging things. I have a very good "theory of mind" for dealing with computer failures, and figuring out why the computer isn't doing what I might naively expect it to do is a lot of fun. However, it's only fun because I've been able to stay on top of the curve as the systems I work with have become more complex. Starting from zero today sounds a lot more daunting.
True, the stacks are more complex today, but the resources are greater. Back in the days when the error was in the beige box, if the books/CDs/DVDs you had on your shelf didn't address the exact problem, you had to roll up your sleeves or you were SOL.
Nowadays, research skills are more important, but I see a lot of devs who just don't have them. Can't find the answer on the first page of your first (poorly formed) search? Run get the senior dev. To me it reads like incuriousity and laziness, or lack of training.
I don't mind doing some coaching, but if you're a dev, and you can't even be bothered to read the error message, what does that say about your effectiveness?
> Any introduction to computers must start with the statement that they are all heaps of plastic
This scales to everything IMHO, everything is simple once you understand it. Levels of abstractions is what makes it scary and complex. I.e. electricity or fire is also not scary once you know how to handle it.
One should not have an unreasonable fear of fire, but "unreasonable" is a quite key word there. One should have a healthy respect for it and feel some fear if those around you don't.
> But when it comes to computers, often you need deep understanding of its inner workings to make sense of your observations of problems.
They're supposed to have that knowledge, or at least not be afraid to dive in and get that knowledge.
There's only one way to build an intuition of what kind of problem probably causes some error (most famously, if the error is completely incomprehensible, you missed a closing thingy on the previous line), and that's by doing the work a lot.
Very very true. When something causes a 1 point story to take three days; bring on the hacks and compromises and ignore anything that doesn't need dealt with to get it out the door.
I don't think so. We can do so many amazing things with the computers precisely because we don't have to know how things work. Computers are so many levels of abstractions over printed metal on melted sand.
People who know what they are doing will understand the errors of their own creations and will learn the workings of the tools they use to some degrees and will be able to understand the failing modes of these tools with experience over time. No one starts with complete knowledge before start building things.
> or at least not be afraid to dive in and get that knowledge.
Of course they should have the drive but people's first instinct would be to make the error go away so that they can do their actual work. People have limited time and energy, you can't expect a JS developer, for example, to study inner workings of a Linux box to understand all errors. It's cool when they do and gives them superpowers but it also makes them less productive as JS developers. Sometimes you simply need to implement that button to render on the server without studying the server.
> I think people don't read error messages if fixing the issue is beyond their immediate capabilities.
How would they obtain those abilities though if not while spending time on the issues brought up and learning how to learn.
I think sometimes people are just bored and can't be bothered to find the cause and solution to their issues, and over a long period of time that mentality sticks and becomes second nature resulting in phrases like "this software sucks, I need to read the docs to use it".
The problem is, learning is taxing and many times you encounter these errors when you have more important things to do.
When you want to develop your game and the IDE is complaining about something about locating some files, do you think that it is good idea to learn how that IDE organises dependencies?
Sometimes you suck it up and learn it and you know next time. However, your first instinct would be to look for ways to make the error go away so that you can immediately start working on the task that you are supposed to work on. That's why we have abstractions and when things work fine we don't know how things work.
It shouldn't be expected of you having complete knowledge of all computer systems, tools and frameworks before you can make a ball image bounce on the screen.
Taxing as hell. I have personal projects well underway that go unfinished because of tooling complexity or some other issue causing me to completely derail and spend days figuring out some type of in issue that has absolutely nothing to do with what I'm trying to accomplish. Granted, since I have gotten away from visual studio it is much better but I'm t still happens. If it isn't the IDE or package upgrades it's AWS or Azure issues.
First of all I agree with you. But I would like to note one aspect of this that has always been true. Many times the error message itself is terrible. They say that an error happened and then they give you absolutely no context for the error. My favorite example is when an application happily bubbles up the error from the operating system when a File read/write operation fails. The OS will tell what kind of operation failed. It won't tell you which file you were trying to read or write. It won't tell you what location you were trying to read or write. Basically none of the context you need to really understand what went wrong. You'll have to go read the code in order to mentally reconstruct what the context is. It's silly. If you are writing a file and don't at a minimum catch and then wrap the error with an error of your own that adds the necessary context then you are contributing to the problem here. And that's just one common example. There are many more.
I'm an SRE, so my programs are generally Python scripts < 1K LOC; maybe this isn't scalable, but I write verbose log statements (if it's launched with --verbose, of course). It's not that much effort to change `except OSError as e: log.error(e)` to `except OSError as e: log.error(f"Error accessing {file} - {e}")`
If I know typical causes of errors (forgot to connect to the VPN, etc.), I'll include them in the log message as well as things to check.
This is a large and ongoing part of becoming a developer. It also happens again every time you try to learn a sufficiently new or alien technology. You know you start to make progress when the error messages begin to make sense.
Often you need to know a lot of context before you're even able to determine what the error message is! One error message can lead to a cascade of other error messages, or it's something breaking down as a result of multiple layers of indirection, requiring the developer to careful track the trail of what went wrong and led to another thing failing, which broke down the next thing and ultimately, decided to stop the program and mention only the very last thing falling apart to the user. There might be a directly sensible connection with the original error, but often it's quite unrelated. An experienced developer often immediately recognizes: this is not the actual error message, that other thing is! But for a junior it's all equally incomprehensible.
It is detective work with many false leads, and being very new at something it can be so overwhelming you don't know where to begin and immediately assume you will not succeed finding out 'whodunnit', asking your senior co-worker for help.
My own perspective on this was that it felt like my brain would turn to goop when I got an error message, my eyes would cross and I would start frantically googling literally anything, skimming stack overflow, and getting nothing done. In order to progress I had to learn to slow down and start reading error messages and learning what they meant. Sometimes this meant I had to look up a bunch of words, one after another, to understand something incredibly dense.
So yeah, I think I largely agree with your assessment, and would only go on to state that the path forward is slowing down to learn vocabulary and think critically. You really speed up after that.
Big fat "it depends" on that. It might be superficially correct (dependong on the scope/skills of "support" staff). Even if there is a meaningful financial difference between the day rates, it doesn't necessarily follow when the process is viewed end to end:
1. If ops staff have limited expertise/authority, it's less likely they can resolve problems. They might acknowledge (so maintaining some aspect of client SLA), or have a limited set of pre-defined remedial actions (reset button). Anything beyond that, though, and it needs the dev team. So it's arguable whether the ops staff provide much value in the equation.
2. As a dev, there's nothing quite like the prospect of being paged at 2am on a Sunday to incentive more robust code.
End to end dev accountability isn't a panacea either - but the problem is more nuanced than just pay rates.
also, devs seem to really underestimate the pay support staff makes, especially those capable of troubleshooting deep, low-level and complex issues.
this might be less visible on the Dev side, but troubleshooting infrastructure is not an easy to find skill, especially if dozens of moving parts are involved.
More power to them. The rest of us 'Ops' guys are slowly transitioning to DevOps Engineers and SRE's and are gladly taking handfuls of cash because I both know how to read hex dumps, and are also smart enough to know your IP address isn't going to change because I replaced your ethernet cable.
One of the best arguments for shared responsibility is that it avoids “not my job” thinking. I’ve seen large organizations burn resources and downtime because developers and ops are in an adversarial relationship where nobody has a stronger sense of responsibility for an application working than they do for shifting the blame to their counterparts. If everyone is getting things escalated to them it tends to cut through that cycle.
Often the root cause is in dev code without proper logging and documentation = dev problem. The only way for Ops to troubleshoot might be to sniff the network and make educated guesses, if it's recurring. All that extra work costs much more and halts more urgent production work.
L1 support are users of the software, just like their customers. If the application doesn't provide them with enough information to do their job, then that's a failure of the application.
And by "application" I mean the combined software, release, documentation, runbooks, etc. Not just the latest git tag pushed.
Force them to write heavily templates C++ code, they either learn how to read pages of comprehensible error message to diagnose one little typo, or they can’t do their job and are forced to look for a new one.
It’s usually not hard to figure out what’s wrong from the messages, but man do they look scary and hard to understand when they appear. Yes I’ve been writing C++17 lately using some very template heavy libraries.
“Force them to write heavily templates C++ code, they either learn how to read pages of comprehensible error message to diagnose one little typo, or they can’t do their job and are forced to look for a new one.”
When I did C++ we sometimes made little competitions for the smallest change that can produce the craziest error messages. On the other hand I always found it extremely satisfying to make one little change that removed thousands of errors and warnings.
I had a nice one today where it complained about some thing not being invokable deep in some std code somewhere. Lots of crazy template instantiation errors. It turned out I forgot to pass the variant parameter to std::visit.
I've avoided C++ for most of my career, but one idiotic mistake I do remember making was accidentally leaving an open curlybrace at the end of one of my source files. The C++ compiler ran and reported 1000s of compilation errors all through every single other file -- in my code, throughout all the library code I included.
Easily diagnosed if you're working incrementally, one small change at a time, and making checkpoints with version control: `git diff`, carefully review the diff of what you changed since the last checkpoint where things were more or less working. I must have not been disciplined enough to work like that at the time.
Troubleshooting systems integration failures is also character building for getting better at diagnosis from errors. Sure, it's failing, but let's try to figure out the immediate layer of failure from the logs, error messages, symptoms: name resolution? tcp? tls? http proxy? authentication? authorisation? api spec misalignment? error in our application code or the system we're directly talking to? unexpected data? error in some other system that we depend upon transitively? each time you hit a new novel failure mode, or fail at one level deeper, you're making progress!
Some seniors remind me of street preachers shouting "Hell and damnation awaits you!", and they get much the same response from people. Not saying this is you, but I'd be interested to know what ways you've tried and what's worked or not.
> This is 100% a problem with younger, bootcamp-”educated” devs, in my experience.
In my experience it’s also common with older and college-educated ones; contractors trying to avoid extra hours; senior architects; and especially anyone who thinks ops is someone else’s job. It’s definitely not specific to age or training mode.
There are a few contributing factors I see: tunnel-vision focused on the particular detail they think they’re working on, causing them to ignore anything they “know” isn’t related; shoddy tools like much of the Java ecosystem where poor culture around logging trains every user that it’s normal to have huge amounts of log spew; etc. but the biggest problem I have seen is ego — either unwillingness to believe that the product of their staggering intellect could be less than perfect or that the mundane task of getting their grand vision to actually work is for the little people.
I’m thinking of a “senior architect” who was quite surprised to learn that networks are neither perfect nor instantaneous, and that his app might have some issues due to needing thousands of XHR calls to load the UI. It was so much easier to ignore the error messages and say the problem was Chrome. He had a CS degree – the problem was the wrong mindset and having been enabled to avoid good troubleshooting skills.
It could be argued that there is a technological Maya that has to be overcome before you mature as a developer. Most people grow up spoon fed the "it just works" ideal, even old school computer people never really had to doubt that their floppy drive would read a disk or that their PC BIOS would accurately boot their computer.
It's only after the technological illusion of Maya breaks that you realize floppies have read heads, hard drives have moving parts, CPUs have conductive traces and all of these are vulnerable to breakdown, entropy exists in the system and cannot be expelled, that the previous "ideal" state of your system was temporary, an illusion, that nothing always works the way it is supposed to and that your options boil down to "burn it to the ground and start over" or "leap into Hades both feet first to rescue the soul of what you love".
Most people go the first route. Buy a new one. Replace what is broken with something else. That way the illusions are never broken. The technology didn't fail, only its current & easily replaceable avatar.
As the Son of God once did, after its death it will rise again, immortally replaceable.
However, it is only after you have faced that 2nd trial by fire and returned with your elixir that you as a changed being can peer through the veil. The meme about "CPUs being rocks we filled with lightning & tricked into thinking" rings differently to you now.
You're touched the bones of the God and found that they crumble. There is no God here, only a beautiful shambling nightmare that has eaten the minds and souls of millions, built by mad scientists and engineers in a vain attempt to create the God whose physical absence they find themselves longing for the same way a neglected child longs for the embrace of their mother.
> I keep having to tell junior devs to actually _read_ the fine error message
I wonder if it's also to do with the environment in which they learn. When I was learning to program, like probably others here, I didn't have anyone around me who knew anything about computers so was generally on my own until my first job and had to dig through stack traces and read error messages and had to try and figure out what was wrong. Kind of a blessing and a curse as I imagine my rate would have been a quicker and I wouldn't have hit so many brick walls but I learned to debug independently.
I have the opposite experience, boomer devs that see red or yellow in their terminal and send me an email or jira ticket before trying to parse it. The younger generation at least can make judgement calls on warnings.
this feels like one of those things that needs to be tackled from both ends
teach people to read error messages and simultaneously improve the readability (and utility) or error messages
i don't know why we put up with such bad error messages anymore. i imagine it's a function of stockholm syndrome and the difficulty in getting messages changed
I sort of blame exceptions here. They make it really easy to just let the error bubble up to the top. But often times the place where the error gets thrown doesn't have all the necessary context to have a really good error message. If you want a good readable error message you have to trap the exception at the appropriate place and then wrap it with the appropriate amount of context in the message. But the easy path is to pretend there is no error and let the very top layer surface anything that went wrong to the user.
I don’t know most of the times error messages on computers are garbage. People build up some type of fear for the error message format.
I have an example:
I built a logistics and invoicing tool a few years back with error messages that where human readable with clear proper messages that told the user exactly what they did wrong and it even proposed how they might fix the problem.
I don’t know how many times I had to go to the users workstations read the text out loud for them like they where a 5 year old and ask them what they thought it meant.
They always knew what it meant but I had to read it for them it was embarrassing.
And these where university educated accountants that where using the software.
After a lifetime of garbage error messages like “error code 4513” people just zone out.
For real. How difficult would it be to have the computer tell us what the error code means instead of a single sentence.
I get codes back in the day when storage for a whole book was costly, but that isn't the case anymore. Just tell us the error, show us the pointers, and then tell us what typical fixes are instead of expecting us to go to the internet for a solution.
My take is that cryptic error messages have always been a cross between 'protecting company secrets' and 'never really admit to a mistake'. Sure, MS or Apple could just throw up a dialog that said "Sorry, we trashed the file you were working on, here are the last 1024 chars" but people would actually be more angry then, instead of just "Error 1234 occurred" (or at least they'd have to go look it up to be mad).
As developers, we also have to be used to a lot of completely unhelpful errors. Yeah, couldn't connect to the DB, sure... oh, but actually because my code ate all of memory, why didn't you say that in the first place?
I suspect it grew out of electrical engineering. Machines/parts could only return errors as integers, so you would expect people to open up the documents and read that error code 1021 meant that the unit had caught fire.
Software just kept the tradition of error codes, since that meant you could also sell that juicy documentation (localized into whatever language you wanted) to the user as well. I suspect it also a localization issue because OracleDB would never return the table/column in the error message, so as to be easier to translate.
From my mentioned example one reason was cause the user loaded a csv file with incompatible data.
Or cause the user had filled in part two of a task but not part one and then tried to continue with parts of the task that where dependent on filling in part one.
Or the user tried to synchronize orders from the erp system but the erp system would not return any orders.
> This is 100% a problem with younger, bootcamp-”educated” devs, in my experience.
I resent this. Not because I'm a bootcamp-"educated" dev. I'm not. But it suggests somehow that devs with CS degrees are somehow better in this aspect. If anything, they're arguably worse (obligatory, not everyone disclaimer).
I have never met these kinds of people and it boggles my mind that they exist. Actually reading error messages is the most basic thing I'd expect from a developer considering that you're going to encounter them regularly during the development process.
I always preach to people that they have to make it easy for others to help them. Don’t just say “it doesn’t work” and expect them to analyze the issue and take care of things. Instead provide information what you did, send log files, screenshots and whatever other information you may have.
I think the people most likely to fall into the “it doesn’t work” category are people who don’t have much experience troubleshooting difficult problems.
In the end it’s about compassion and understanding of each other. Unfortunately in a lot of companies the only direction people are getting is “get it done on time”. It’s rare that management asks people to have empathy for each other.
I don't think it's just because of bootcamp education, I think people are growing up in a world where error messages are either never displayed, or displayed in the from of "The program has did a sad :( try again later."
They're not used to reading error messages because they've been brought up seeing nothing but completely useless error messages.
We “process” maybe 100 times more text than we used - but to do that we scan for patterns and ignore without reading most of it. (Partly due to the prevalence of advertising as well, of which we prefer to read 0%)
> This is 100% a problem with younger, bootcamp-"educated" devs, in my experience.
You'll get a lot of pushback here but it's definitely true.
That doesn't mean it doesn't happen with CS grads as well, but it's quite rampant among bootcamp devs. I think the reason for that is that, since the bootcamps are so short, they "stay on rails" and mostly work on simple projects (that will give out something they can push to a github repo and use as a portfolio).
It's the same with git. Every bootcamp will use git and claim to teach it to their grads, but then watch them do anything on a repo with multiple users. A lot of them just rote memorized commands to pull and push to main and that's it. Branching? Rebase? Using the commit history? Never heard of.
For new hires from serious Engineering or CS Degree, they should have had at least a few classes dedicated to projects where they built something non-trivial. On top of theoretical classes teaching the fundamentals.
To my the line between dev and ops is where we put it. And I like to make it very explicit.
In shared hosting times. Ops maintained Puppet definitions, create new "deploy environments" (using Puppet), gave/revoked server access the employees, monitored servers. Devs maintained the source repos and deployed to the environments provided by ops.
Now we live in virtual machine times (docker). Ops does the cloud infra (terraform), monitors the services, gives/revokes access to cloud services. Devs maintaines the source repos and deploys to the cloud clusters provided by ops.
Years ago, everyone working in tech was an IT generalist. They did everything (DB design, systems, applications, algorithms, code, networking, etc.). Today, the field has matured and people are able to specialize.
Sometimes, when old IT generalists work with new IT specialists, these sort of misunderstandings occur.
I'm not sure. I take full-stack to mean the front-end, middleware and back-end of a webapp. I would still call it being a "generalist" when applying the concept to computer technology in general. For example, the CTO of an org should be a technology generalist, not a full-stack web dev.
Of course, this is just my personal opinion based on what I have experienced over the last 40 years.
Years before everyone was apparently a generalist, you would have a separate DBA, a separate architect, sometimes even separate teams for implementing algorithms, etc. The mythical man month has a very nice section on splitting up the work over the various teams and that's a book written in the early 70s.
I think the actual boundary lies more in big vs small companies: small companies do not have the resources to hire specialists for every little subproblem, while big companies typically have enough employees that specialisation becomes a possibility.
To the author of this article: Really great job on it, had great fun reading and also lots of truths in there. But this part:
”Often they have not even bothered to do basic troubleshooting, things like read the documentation on what the error message is attempting to tell you.”
This happens, but this just means that your Development Team needs some coaching or to improve their quality.
This tells more of a quality of the development team you have been working with. You have to pass along this feedback and ensure that Development team also works with professionalism as everybody else.
DevOps would tell be that "Dev & Ops" would look up issues together (Yes, he will be blocked as well WORKING with you), if you find that it was developer's fault. Tell them: "Hey, this is on your side. You saw how we troubleshooted together. Now each of us has new tricks to use in the future".
If you don't do that, you are the shortest path to get THEIR problem solved. And it is too easy to go that path.
This is a failure of management, not the "team". They don't need "coaching", this bullshit molly-coddling of people is pathetic.
Good management will ensure that if a problem like this occurs, they don't "coach" but they "counsel" the appropriate dev to do their job and not waste everyone else's time.
On mobile this webpage has a thin hovering black bar at the top that fills to the right as you scroll further into the article. Very nice feature that I have not seen before.
We used to have this browser-supplied thing which would tell you how far you were in the document, what percentage of the document you were currently looking at, and afforded you the ability to quickly change your position.
You and your dinosaur technology, next thing you will want clickable interactive text to be in a different font or colour to differentiate from regular text.
Everyone needs to be a bit of everything to mitigate the cases where one team doesn’t understand another team’s domain and Applications begins blaming IT or Operations admits to not understanding the applications they facilitate.
There is a reason SRE/DeVOps Eng jobs are taking off in number and comp, and entities GitHub is (slowly) figuring out how to automate dev work.
Running code at scale turned into a very challenging comp sci program, and uptime vs code slickness is getting prioritized by clients.
The career support and innovation in that corner of the world (ops eng jobs) reflects it. Sort of gets after what software architects do, but the requirements to know that come way earlier in the career for Ops. Ops Engs with cloud knowledge, Python, and IaC tend to go far.
Oh boy, it feels like someone is ringing bells in my head because of aligned thoughts.
Let me share an experience. In 2010, I worked on a project for a large business in the US(Fortune 100). The process was set so rigidly that it worked well, but I was among the group of people who were mad at it saying”why is this so rigid? Trust us and let us do things faster!!”. Context : There were change management rules in place. The software was to be released only on a regular cadence of about 6 months, only after thorough integration tests, and approval from the change mgmt board. Should anything go wrong in “move to prod” there will be representation from dev, QA, Ops, change mgmt, and Mgmt orgs to immediately decide on actions until the release to prod is successful. There will be thorough documentation of what to do (run books) on what changes occurred, what their impact could be and how to rollback if something unexpected occurs. It was always a party after a successful release :-)
Trust me there were a lot of bugs, but they were mostly found and fixed during the laborious QA and integration tests by people whose job it was.
Fast forward to now, I am a “Cloud Engineer” in a small team that does everything from app development to building CI pipelines to running services on AWS to being on-call to keep them running.
I must say, I wish for the old days back. Sure, it was slow and laborious, but it resulted in better outcomes and manageability. IMHO, it also resulted in better reliability of software due to the diligence done by several layers.
It is easy to say do the same just faster in your small team. But, in practicality it just doesn’t happen. I work on setting up Observability one week, then onto designing infra for a new service, then onto some development and so on. I feel like my scope would have been limited, and I would have had an easier time becoming an expert at something than becoming so broad skilled like I am today.
Sometimes, old, slow, and mature is not so bad. Not everyone needs to follow the FAANG SV companies to be successful.
Here here. If the bulk of your “products” are for internal consumers, you likely aren’t paying enough to attract talent who know how to operate in the the FAANG model.
I like to distinguish between “product developers” (i.e. building products for consumers with guaranteed scale, so do it right the first time) and “project developers” (get it done ASAP and cut the corners you need to do so).
In the “project developer” world, 50-75% of your requirements gathering happens before a line of code is written. There is usually a “right way” to implement a process of which technology is only one component and figuring that out as you go will actually slow down the project due to the maker / manager schedule conflict. True “agile” in this environment just leads to scope creep as there usually aren’t dedicated product owners to say no to every little request.
I’ve stopped pushing agile as hard because the corporates simply can’t afford the kind of engineers to make it work correctly, and they don’t have the roles required to gather and feed requirements to a dev team in an agile format. Sprints are a good way to time-box feature development, but most business projects work better with a more waterfall approach. Your customers and project plan operate under waterfall so there’s less downside to begin with.
excellent comments.
instead of straight waterfall, i would suggest a time boxed requirements phase, followed by incremental development with a reasonable cadence (dictated by the product; web might be 2 weeks, more serious domains might be 90 days).
you need iteration, but having a solid grounding on what you are going to build eliminates churn.
Great comment about “Project developers” and “Product developers”. It is almost an entirely different art to get the requirements right by iterating on a project, and bringing out a solution to life versus engineering a scalable, maintainable product that evolves after a good start.
I never had to think of such distinction.
Waterfall model has its downsides in extracting the requirements out properly whereas the Agile approach(the little I have seen of it) seems to lose the layered stability of a waterfall based approach.
> I must say, I wish for the old days back. Sure, it was slow and laborious, but it resulted in better outcomes and manageability. IMHO, it also resulted in better reliability of software due to the diligence done by several layers.
Those were also the days where it took many years to go from Java 6 to Java 8. Or perhaps to try out Kotlin.
They were the days where legacy code was the norm, and we kept supporting it because nobody dared to change anything for the better. In practice, that's just not something you can maintain in a competitive market, because your competitors _will_ use new technologies and faster/better development processes.
"it just works" might be good enough for maintaining your application, but will it be good enough to find people willing to work in that code base or that environment?
I work for a large business where both the old and new practices are in place (mostly the new ones, though). Focusing on "going fast" is definitely not a good idea, but I believe there's a sweet spot in between.
Sure, mature processes encourage tech stagnation, and discourage even beneficial changes as collateral damage. But, as you say there is line somewhere at which project should move on from "Go fast, ship often, change much and get feature-rich" to "focus on correctness, stable releases, actually maintain our existing features". Perhaps it is really a cycle of both and missing one for the other leads to problems.
That's oddly familiar... I frequently get: The server is slow. Well, no it's not really doing anything, but your applications is responding remarkably slowly.
Or: Can I get a bigger server... Yes, but you have 32 cores and 256GB of RAM, and your applications isn't that complex.
This blog perfectly articulates the strife that inspired and drove the DevOps trend.
I am always saddened when I hear "our organisation has a DevOps team" - immediately this demonstrates the fundamenetal lack of understanding the very premise of what DevOps set out to solve: Bringing Development and Operations together.
Even the very name "DevOps" was constructed such to symbolise the combining of the two domains into one. But no. Now we just have a new cool title to throw on people who will be ringfenced just as they were before.
I'm more and more convinced that "DevOps" (and "Agile" before it) are just buzzwords that can be leveraged to make whatever change the person implementing it wanted to do all along, with zero regard for what the buzzword actually means. If real devops was going to happen, it wouldn't take a brand to sell cross functional collaboration. It would have just been yet another one of the constant stream of incremental improvements we make by folding lessons learned in industry into our own orgs.
Yep, because developer might not know what ops needs in terms of traceability, logs and so on, to be able to run their code in production without having to wake them up at 2AM. Similarly Ops knows a lot about what can be done with existing infrastructure, or off the shelf components, which can save a huge amount of work, while providing a more stable system.
I do mostly operations now, and I'm lucky enough to work with really talents developers, who care to listen to input, before writing 5000 lines of code. I also work with customers, who have their own developers, with their own weird ideas about the world.
The biggest problem I see right now, except for occasions cowboy pretending to be a professional developer, is developer picking technologies without understanding it. We work with customer who picked technologies because they're interesting, not because it's what they need. When performance is terrible it becomes and operations issue and being told "Kafka is not actually a database and should be used as one" often isn't the answer they want. Or try telling a developer that the code he worked on for three months can be done by the existing load balancer in a few hours or that the ORM is actually writing terrible queries.
DevOps team, as in: "We use the shared knowledge of both parties" is fantastic, but operations is frequently an afterthought and not involved in the design fase.
If we're to take "DevOps" as developers doing operation, I'd prefer that we do the opposite and let operations do development. I think we'd get better results.
Another root cause in our environment is alluded to in the article. With the rise of test frameworks, devs seem to test to prove the API is correct, not to find problems.
Another symptom of this is that when the QA/Staging function went away, load testing became perfunctory. Many of the performance problems we see should have been caught in QA. Devs are anxious to ship and get on to the next sprint, leaving app support and operations on the hook.
>Devs are anxious to ship and get on to the next sprint
I think it goes a step farther back to product. PMs and analysts put constant pressure on developer teams to complete work quickly and that time pressure shows up on the next guy's plate, etc
335 comments
[ 2.4 ms ] story [ 283 ms ] threadResponding to "Someone will always have to own that gap and nobody wants to, because there is no incentive to. Who wants to own the outage, the fuck up or the slow down?" with "Not me." is not sufficient, it's a very valid question for which any organization definitely needs an answer pointing at some specific people - if it's not going to be pure ops people, it's IMHO not going to be the feature-developing devs as well, that would likely need separate 'site reliability engineer' teams as some major companies do.
Seems like OP works for a shitty company.
I had to chuckle - everyone (not just developers) seems to blame the network first! (including blame the firewall rules)
Especially when the dev wanted to migrate from Java 8 to Java 11 and didn’t even attempt to lookup our documentation on how to change JVM parameters.
This is the crux of the problem. Coding in isolation. Replies of 'It's java, it should work anywhere' etc.
The other gear grinding commom theme is not even doing basic troubleshooting. To the point of not even googling the error message or the symptoms, and being 'blocked' because they are waiting on a ticket they opened with the 'other' team.
There’s some very clever ppl that know all about how networks/vm stuff work, and I’ve learnt enough from them that I can fix most of my own infra related things - or at least give them a run down of what I’ve done first to save them some time.
It got me back into hardware and networky stuff, so now I’ve got a MikroTik at home, some proxmox machines, Tailscale network etc - more fun than just spin up a box on DO and be done with it.
A lot of ppl just aren’t interested though, they just want to code (and maybe learn a new language) but because a lot of stuff is now PaaS and it’s super easy, there is no need to learn it (in their eyes)
One should not build silos where experts sit.
One should participate in a team of many different experts.
If you still have to call "whatever department" for fixing your slow SQL query, your disk space, your repo access, production deployments etc. Then in most cases I feel you should get out and find a place where silos are not being exercised.
This is a thing of the past.
Is it? In highly regulated environments, silos are practically a requirement. Security access controls are intentionally put in place to limit access to systems and their respective pieces. If you need repo access, or access to the CI pipeline, or access to the database, you have to go to the appropriate channel.
- no need for anything but minimal security, perhaps because the business is trying to surf the margin between their AWS bill and their Google ad revenue.
- there is only a need for security in the part of the business that deals with money
- some stuff that the users do, they would prefer to maintain integrity but they don't care a lot about confidentiality
- the users want reasonable confidentiality, too
- everything about the business is money or secrets
Where your business is on that scale determines how much you are regulated and how many internal gatekeepers are necessary.
I think this is the original thesis for DevOps.
It wasn't supposed to mean no more division of labor. Division of labor is a key innovation in human society that enables civilizations to exist. It was supposed to mean the teams in different categories of labor interact throughout and consider each other's needs, and not throw shit at each other over a wall and only ever interact through a ticketing system.
This waffly "breaking down the silos" stuff is a later redefinition, i believe reacting to the fact that the original meaning was extremely unpalatable to existing organisations, with existing employees and hierarchies who would be severely disrupted by it.
From Patrick Debois in the interview "Later, I saw a talk by Jean-Paul Sergent about developer (Dev) teams and operations (Ops) teams working together."
So no, breaking down the silos was baked in from the beginning.
The team that writes the code should also deploy the code and get paged in if there are problems in production.
That creates a tight feedback loop that requires developers to learn and manage the whole stack, code defensively, and test enough to be confident to deploy to production.
Didn't test your code enough? You will be paged in the middle of the night to fix it. It creates a strong incentive to make good decisions because you will be living in the mess you create.
I have heard of companies deciding to do 'devops' and it turns into a free for all of dev teams having to handle/build things end to end. Everyone loses in that scenario.
This is how it is at my startup. All of the engineers are involved in managing the infrastructure for everything we build. I find it gives me much better insight into my app, and the feedback loop is much tighter since I am in control of everything.
I feel like devs at small companies are doing everything -
coding, testing, supporting customer, deployments, troubleshooting and of course straight over ssh+winscp cuz vps/bare metal are cheaper
large organisations usually have a much bigger responsibility and is held to higher standards, frequently audited and controlled to stay within rules and legal compliance
If you've got 2 developers, they're both doing everything and on call 24/7 and all have read/write access to everything on demand.
If you've got 200 developers, you're going to start wanting a team of shift workers keeping an eye on the systems, and maybe you won't want every developer to have read/write access to production data.
If you've got 20,000 developers your working practices and infrastructure are almost completely cemented in place, and anyone who doesn't like them has already left because it's easier to change jobs than to get 20k people to change their behaviour.
There needs to be a rethink of how infrastructure, development and deployment is handled.. maybe the solution is to slow things down and insert a little carefully thought out bureaucracy between the layers (can't believe I'm advocating for more bureaucracy!)
This has been the bane of my work happiness for a while now. I keep having to tell junior devs to actually _read_ the fine error message, just in case it actually _contains information about the error_, you know. Not that it seems to help much, it’s like they can’t get the concept into their heads.
This is 100% a problem with younger, bootcamp-”educated” devs, in my experience. I know the common wisdom on social media is ”no one reads text anymore”, but if that includes aspiring developers, it might be tough to replace the current workforce when that day comes…
well except for using Rust, and a bunch of dependencies from the web, version determined when downloaded at compile time.
"lack of newness" is a characteristic many will expend untold hours to extinguish. to my perspective, the "rewrite it in Rust crowd" is the peak; all non-Rust code is soiled, and worthy of replacement.
(it is very possible that the "rewrite in Rust" movement is just a guerrilla marketing project)
This lead to a lot of them learning common failure modes of the software they ran.
Ironically the people who were script kiddies in their teens have been some of the best troubleshooters I know.
I was doubtful that this was a universal truth then, and I think it's the same now: there are a lot of people who do mediocre work, and they are and were supported by a smaller group of people who do really good work. And the world keeps turning.
One of the joys of the Internet and of open source is the increased ease of sharing ideas and solutions.
I think the only solution for this is when "support" includes education. In the simplest form, we can give support by helping that colleague to find the issue himself, rather than giving the solution directly. In a more advanced form, you're making structural changes to your company. Like in how you share knowledge with the team.
We've all heard the stories where someone bright joins a new workplace, is assigned drudgework and after a bit automates it until they only work two hours a week? That's someone who is willing to learn, and everyone else in the office was willing to experience boredom in order to avoid learning.
For all I know, some of those people would have perked right up if the subject matter were milling, millinery or masonry moving millier-weights of stone. Not everyone is interested in the same things, and even when their job is all about it, sometimes people aren't invested in it.
If it works, everyone involved wins. Sometimes it doesn't work.
You're doing everyone a disfavour giving out full answers or not demanding some homework first, and often, it's really an ego-issue.
I think there's something about certain kinds of tools that are cryptic, unpredictable, and frustrating that can teach you to be helpless - to just Google and hope. It's fixable though.
It really is worth it to go the extra mile and write comprehensive docs, even going as far as writing them in a conversational tone as if it's a blog post or a book. I'm really happy I found a company who treats documentation and workflows as first class resources.
For a small team where only 1 person is working on this it helps eliminate the bus factor and it also makes it easier to have non-hardcore ops folks do code reviews on your IaC. Having them be able to get the gist of it with a little bit of background knowledge is so much better than nothing. All of this results in higher reliability of the services your company offers.
Maybe they have seen so many red herrings that they don't even trust that the error message could contain something useful and relevant? Or maybe they just learned to skim through everything, and don't actually read stuff.
I also don't understand why, when they ask for help, they can never be bothered to say what they're trying to do, what error message they got, etc. It feels like they're doing me a favor when I try to help them fix something.
Really helped me gain the mindset that not only was it my mistake that resulted in code not running, but that it was fixable. Like a game of ping pong. You hit the ball, sometimes the compiler hits it back.
Computers are scary things that fail in counterintuitive ways. When the handle of your tea cup breaks, the issue is intuitive and most people will be able to understand why it is happening and how to work around it(handle it carefully from the top end end enjoy your tea?).
But when it comes to computers, often you need deep understanding of its inner workings to make sense of your observations of problems. Why Xcode would say that it failed to compile my project because usefulExtensions.swift already exists? What it is supposed to mean, I see only one file with that name? That information gives intuitive idea about the issue only if you know how the compiling process works.
Why would I know why the package couldn't be found? Unless of course I know how that package manager works. Then I can check if the package manager is configured to look at the correct places.
Most error messages are like that. Instantly makes intuitive sense if you know how everything is glued together and makes no sense and needs study if it's outside of you domain of expertise. No one reads error messages unless they can recognise the pattern instantly and there's a data(like the name of the variable) guiding you to the fix.
Not being scared of the tool and believing one's inherent supremacy over it must be the most basic criterion for practicing this craft, but these days this fear is nursed, at times encouraged, at times even exalted (corollary of the failure fetish) especially by those who publicly place themselves as ambassadors.
Any introduction to computers must start with the statement that they are all heaps of plastic and sand and the only things they are able to do are because some mortal sat down and spent time figuring it out.
People starting out now are at a disadvantage because their first encounters happen mostly through extremely polished looking apps and it is hard to see at the outset how one could go from weird incantations in a text editor to that.
Personally, I love debugging things. I have a very good "theory of mind" for dealing with computer failures, and figuring out why the computer isn't doing what I might naively expect it to do is a lot of fun. However, it's only fun because I've been able to stay on top of the curve as the systems I work with have become more complex. Starting from zero today sounds a lot more daunting.
Nowadays, research skills are more important, but I see a lot of devs who just don't have them. Can't find the answer on the first page of your first (poorly formed) search? Run get the senior dev. To me it reads like incuriousity and laziness, or lack of training.
I don't mind doing some coaching, but if you're a dev, and you can't even be bothered to read the error message, what does that say about your effectiveness?
/rant
This scales to everything IMHO, everything is simple once you understand it. Levels of abstractions is what makes it scary and complex. I.e. electricity or fire is also not scary once you know how to handle it.
They're supposed to have that knowledge, or at least not be afraid to dive in and get that knowledge.
There's only one way to build an intuition of what kind of problem probably causes some error (most famously, if the error is completely incomprehensible, you missed a closing thingy on the previous line), and that's by doing the work a lot.
I don't think so. We can do so many amazing things with the computers precisely because we don't have to know how things work. Computers are so many levels of abstractions over printed metal on melted sand.
People who know what they are doing will understand the errors of their own creations and will learn the workings of the tools they use to some degrees and will be able to understand the failing modes of these tools with experience over time. No one starts with complete knowledge before start building things.
> or at least not be afraid to dive in and get that knowledge.
Of course they should have the drive but people's first instinct would be to make the error go away so that they can do their actual work. People have limited time and energy, you can't expect a JS developer, for example, to study inner workings of a Linux box to understand all errors. It's cool when they do and gives them superpowers but it also makes them less productive as JS developers. Sometimes you simply need to implement that button to render on the server without studying the server.
When even line numbers are missing, simple syntax errors can generate new errors and mind numbing troubleshooting.
How would they obtain those abilities though if not while spending time on the issues brought up and learning how to learn.
I think sometimes people are just bored and can't be bothered to find the cause and solution to their issues, and over a long period of time that mentality sticks and becomes second nature resulting in phrases like "this software sucks, I need to read the docs to use it".
The problem is, learning is taxing and many times you encounter these errors when you have more important things to do.
When you want to develop your game and the IDE is complaining about something about locating some files, do you think that it is good idea to learn how that IDE organises dependencies?
Sometimes you suck it up and learn it and you know next time. However, your first instinct would be to look for ways to make the error go away so that you can immediately start working on the task that you are supposed to work on. That's why we have abstractions and when things work fine we don't know how things work.
It shouldn't be expected of you having complete knowledge of all computer systems, tools and frameworks before you can make a ball image bounce on the screen.
If I know typical causes of errors (forgot to connect to the VPN, etc.), I'll include them in the log message as well as things to check.
Often you need to know a lot of context before you're even able to determine what the error message is! One error message can lead to a cascade of other error messages, or it's something breaking down as a result of multiple layers of indirection, requiring the developer to careful track the trail of what went wrong and led to another thing failing, which broke down the next thing and ultimately, decided to stop the program and mention only the very last thing falling apart to the user. There might be a directly sensible connection with the original error, but often it's quite unrelated. An experienced developer often immediately recognizes: this is not the actual error message, that other thing is! But for a junior it's all equally incomprehensible.
It is detective work with many false leads, and being very new at something it can be so overwhelming you don't know where to begin and immediately assume you will not succeed finding out 'whodunnit', asking your senior co-worker for help.
At least capture them so somebody who knows that area can make sense of them.
So yeah, I think I largely agree with your assessment, and would only go on to state that the path forward is slowing down to learn vocabulary and think critically. You really speed up after that.
1. If ops staff have limited expertise/authority, it's less likely they can resolve problems. They might acknowledge (so maintaining some aspect of client SLA), or have a limited set of pre-defined remedial actions (reset button). Anything beyond that, though, and it needs the dev team. So it's arguable whether the ops staff provide much value in the equation.
2. As a dev, there's nothing quite like the prospect of being paged at 2am on a Sunday to incentive more robust code.
End to end dev accountability isn't a panacea either - but the problem is more nuanced than just pay rates.
You broke it, you fix it.
And by "application" I mean the combined software, release, documentation, runbooks, etc. Not just the latest git tag pushed.
It’s usually not hard to figure out what’s wrong from the messages, but man do they look scary and hard to understand when they appear. Yes I’ve been writing C++17 lately using some very template heavy libraries.
When I did C++ we sometimes made little competitions for the smallest change that can produce the craziest error messages. On the other hand I always found it extremely satisfying to make one little change that removed thousands of errors and warnings.
Easily diagnosed if you're working incrementally, one small change at a time, and making checkpoints with version control: `git diff`, carefully review the diff of what you changed since the last checkpoint where things were more or less working. I must have not been disciplined enough to work like that at the time.
Troubleshooting systems integration failures is also character building for getting better at diagnosis from errors. Sure, it's failing, but let's try to figure out the immediate layer of failure from the logs, error messages, symptoms: name resolution? tcp? tls? http proxy? authentication? authorisation? api spec misalignment? error in our application code or the system we're directly talking to? unexpected data? error in some other system that we depend upon transitively? each time you hit a new novel failure mode, or fail at one level deeper, you're making progress!
In my experience it’s also common with older and college-educated ones; contractors trying to avoid extra hours; senior architects; and especially anyone who thinks ops is someone else’s job. It’s definitely not specific to age or training mode.
There are a few contributing factors I see: tunnel-vision focused on the particular detail they think they’re working on, causing them to ignore anything they “know” isn’t related; shoddy tools like much of the Java ecosystem where poor culture around logging trains every user that it’s normal to have huge amounts of log spew; etc. but the biggest problem I have seen is ego — either unwillingness to believe that the product of their staggering intellect could be less than perfect or that the mundane task of getting their grand vision to actually work is for the little people.
I’m thinking of a “senior architect” who was quite surprised to learn that networks are neither perfect nor instantaneous, and that his app might have some issues due to needing thousands of XHR calls to load the UI. It was so much easier to ignore the error messages and say the problem was Chrome. He had a CS degree – the problem was the wrong mindset and having been enabled to avoid good troubleshooting skills.
It's only after the technological illusion of Maya breaks that you realize floppies have read heads, hard drives have moving parts, CPUs have conductive traces and all of these are vulnerable to breakdown, entropy exists in the system and cannot be expelled, that the previous "ideal" state of your system was temporary, an illusion, that nothing always works the way it is supposed to and that your options boil down to "burn it to the ground and start over" or "leap into Hades both feet first to rescue the soul of what you love".
Most people go the first route. Buy a new one. Replace what is broken with something else. That way the illusions are never broken. The technology didn't fail, only its current & easily replaceable avatar.
As the Son of God once did, after its death it will rise again, immortally replaceable.
However, it is only after you have faced that 2nd trial by fire and returned with your elixir that you as a changed being can peer through the veil. The meme about "CPUs being rocks we filled with lightning & tricked into thinking" rings differently to you now.
You're touched the bones of the God and found that they crumble. There is no God here, only a beautiful shambling nightmare that has eaten the minds and souls of millions, built by mad scientists and engineers in a vain attempt to create the God whose physical absence they find themselves longing for the same way a neglected child longs for the embrace of their mother.
https://psychology.wikia.org/wiki/Maya_(illusion)
I wonder if it's also to do with the environment in which they learn. When I was learning to program, like probably others here, I didn't have anyone around me who knew anything about computers so was generally on my own until my first job and had to dig through stack traces and read error messages and had to try and figure out what was wrong. Kind of a blessing and a curse as I imagine my rate would have been a quicker and I wouldn't have hit so many brick walls but I learned to debug independently.
teach people to read error messages and simultaneously improve the readability (and utility) or error messages
i don't know why we put up with such bad error messages anymore. i imagine it's a function of stockholm syndrome and the difficulty in getting messages changed
I have an example:
I built a logistics and invoicing tool a few years back with error messages that where human readable with clear proper messages that told the user exactly what they did wrong and it even proposed how they might fix the problem.
I don’t know how many times I had to go to the users workstations read the text out loud for them like they where a 5 year old and ask them what they thought it meant. They always knew what it meant but I had to read it for them it was embarrassing.
And these where university educated accountants that where using the software.
After a lifetime of garbage error messages like “error code 4513” people just zone out.
I get codes back in the day when storage for a whole book was costly, but that isn't the case anymore. Just tell us the error, show us the pointers, and then tell us what typical fixes are instead of expecting us to go to the internet for a solution.
As developers, we also have to be used to a lot of completely unhelpful errors. Yeah, couldn't connect to the DB, sure... oh, but actually because my code ate all of memory, why didn't you say that in the first place?
Software just kept the tradition of error codes, since that meant you could also sell that juicy documentation (localized into whatever language you wanted) to the user as well. I suspect it also a localization issue because OracleDB would never return the table/column in the error message, so as to be easier to translate.
The modal dialog breaks UX spectacularly.
Even logs should have messages that actually are intuitive to follow.
Or cause the user had filled in part two of a task but not part one and then tried to continue with parts of the task that where dependent on filling in part one.
Or the user tried to synchronize orders from the erp system but the erp system would not return any orders.
I resent this. Not because I'm a bootcamp-"educated" dev. I'm not. But it suggests somehow that devs with CS degrees are somehow better in this aspect. If anything, they're arguably worse (obligatory, not everyone disclaimer).
I think the people most likely to fall into the “it doesn’t work” category are people who don’t have much experience troubleshooting difficult problems.
In the end it’s about compassion and understanding of each other. Unfortunately in a lot of companies the only direction people are getting is “get it done on time”. It’s rare that management asks people to have empathy for each other.
They're not used to reading error messages because they've been brought up seeing nothing but completely useless error messages.
You'll get a lot of pushback here but it's definitely true.
That doesn't mean it doesn't happen with CS grads as well, but it's quite rampant among bootcamp devs. I think the reason for that is that, since the bootcamps are so short, they "stay on rails" and mostly work on simple projects (that will give out something they can push to a github repo and use as a portfolio).
It's the same with git. Every bootcamp will use git and claim to teach it to their grads, but then watch them do anything on a repo with multiple users. A lot of them just rote memorized commands to pull and push to main and that's it. Branching? Rebase? Using the commit history? Never heard of.
For new hires from serious Engineering or CS Degree, they should have had at least a few classes dedicated to projects where they built something non-trivial. On top of theoretical classes teaching the fundamentals.
In shared hosting times. Ops maintained Puppet definitions, create new "deploy environments" (using Puppet), gave/revoked server access the employees, monitored servers. Devs maintained the source repos and deployed to the environments provided by ops.
Now we live in virtual machine times (docker). Ops does the cloud infra (terraform), monitors the services, gives/revokes access to cloud services. Devs maintaines the source repos and deploys to the cloud clusters provided by ops.
Sometimes, when old IT generalists work with new IT specialists, these sort of misunderstandings occur.
Of course, this is just my personal opinion based on what I have experienced over the last 40 years.
I think the actual boundary lies more in big vs small companies: small companies do not have the resources to hire specialists for every little subproblem, while big companies typically have enough employees that specialisation becomes a possibility.
But in the cruft of legacy systems it probably won't be for a long time, probably never.
I'm a freelancer. I "use" the project managers of my clients, I don't hire them myself.
Same goes for my applications. I use the cloud, managed services and such. The providers hire operations people, I don't.
”Often they have not even bothered to do basic troubleshooting, things like read the documentation on what the error message is attempting to tell you.”
This happens, but this just means that your Development Team needs some coaching or to improve their quality.
This tells more of a quality of the development team you have been working with. You have to pass along this feedback and ensure that Development team also works with professionalism as everybody else.
DevOps would tell be that "Dev & Ops" would look up issues together (Yes, he will be blocked as well WORKING with you), if you find that it was developer's fault. Tell them: "Hey, this is on your side. You saw how we troubleshooted together. Now each of us has new tricks to use in the future".
If you don't do that, you are the shortest path to get THEIR problem solved. And it is too easy to go that path.
Good management will ensure that if a problem like this occurs, they don't "coach" but they "counsel" the appropriate dev to do their job and not waste everyone else's time.
It was called a scrollbar.
Running code at scale turned into a very challenging comp sci program, and uptime vs code slickness is getting prioritized by clients.
The career support and innovation in that corner of the world (ops eng jobs) reflects it. Sort of gets after what software architects do, but the requirements to know that come way earlier in the career for Ops. Ops Engs with cloud knowledge, Python, and IaC tend to go far.
Similar in nature to "Running arbitrary containers" but without the human trust-to-do-no-evil policy in place.
Let me share an experience. In 2010, I worked on a project for a large business in the US(Fortune 100). The process was set so rigidly that it worked well, but I was among the group of people who were mad at it saying”why is this so rigid? Trust us and let us do things faster!!”. Context : There were change management rules in place. The software was to be released only on a regular cadence of about 6 months, only after thorough integration tests, and approval from the change mgmt board. Should anything go wrong in “move to prod” there will be representation from dev, QA, Ops, change mgmt, and Mgmt orgs to immediately decide on actions until the release to prod is successful. There will be thorough documentation of what to do (run books) on what changes occurred, what their impact could be and how to rollback if something unexpected occurs. It was always a party after a successful release :-)
Trust me there were a lot of bugs, but they were mostly found and fixed during the laborious QA and integration tests by people whose job it was.
Fast forward to now, I am a “Cloud Engineer” in a small team that does everything from app development to building CI pipelines to running services on AWS to being on-call to keep them running.
I must say, I wish for the old days back. Sure, it was slow and laborious, but it resulted in better outcomes and manageability. IMHO, it also resulted in better reliability of software due to the diligence done by several layers.
It is easy to say do the same just faster in your small team. But, in practicality it just doesn’t happen. I work on setting up Observability one week, then onto designing infra for a new service, then onto some development and so on. I feel like my scope would have been limited, and I would have had an easier time becoming an expert at something than becoming so broad skilled like I am today.
Sometimes, old, slow, and mature is not so bad. Not everyone needs to follow the FAANG SV companies to be successful.
I like to distinguish between “product developers” (i.e. building products for consumers with guaranteed scale, so do it right the first time) and “project developers” (get it done ASAP and cut the corners you need to do so).
In the “project developer” world, 50-75% of your requirements gathering happens before a line of code is written. There is usually a “right way” to implement a process of which technology is only one component and figuring that out as you go will actually slow down the project due to the maker / manager schedule conflict. True “agile” in this environment just leads to scope creep as there usually aren’t dedicated product owners to say no to every little request.
I’ve stopped pushing agile as hard because the corporates simply can’t afford the kind of engineers to make it work correctly, and they don’t have the roles required to gather and feed requirements to a dev team in an agile format. Sprints are a good way to time-box feature development, but most business projects work better with a more waterfall approach. Your customers and project plan operate under waterfall so there’s less downside to begin with.
Waterfall model has its downsides in extracting the requirements out properly whereas the Agile approach(the little I have seen of it) seems to lose the layered stability of a waterfall based approach.
Those were also the days where it took many years to go from Java 6 to Java 8. Or perhaps to try out Kotlin.
They were the days where legacy code was the norm, and we kept supporting it because nobody dared to change anything for the better. In practice, that's just not something you can maintain in a competitive market, because your competitors _will_ use new technologies and faster/better development processes.
"it just works" might be good enough for maintaining your application, but will it be good enough to find people willing to work in that code base or that environment?
I work for a large business where both the old and new practices are in place (mostly the new ones, though). Focusing on "going fast" is definitely not a good idea, but I believe there's a sweet spot in between.
I'm on a project now that has not released to prod. It has a lot of new legacy code.
It seems more and more places want it to be. DevOps is all the rage.
Developer: Host XYZ is very busy.
Sysadmin: Yes, Yes it is. The top 10 processes are your Java App.
Developer: Fix it.
Sysadmin: ???? You can request a larger virtual machine, you can try these options to the JVM, or you can fix your code.
Developer: Can you do it?
Or: Can I get a bigger server... Yes, but you have 32 cores and 256GB of RAM, and your applications isn't that complex.
This is a huge problem. Working on reliability and security is hard, shipping broken features is easy.
In that regard, those roles are slowing things down and costing money
I am always saddened when I hear "our organisation has a DevOps team" - immediately this demonstrates the fundamenetal lack of understanding the very premise of what DevOps set out to solve: Bringing Development and Operations together.
Even the very name "DevOps" was constructed such to symbolise the combining of the two domains into one. But no. Now we just have a new cool title to throw on people who will be ringfenced just as they were before.
"DevOps" today is just codified Shadow IT.
Yep, because developer might not know what ops needs in terms of traceability, logs and so on, to be able to run their code in production without having to wake them up at 2AM. Similarly Ops knows a lot about what can be done with existing infrastructure, or off the shelf components, which can save a huge amount of work, while providing a more stable system.
I do mostly operations now, and I'm lucky enough to work with really talents developers, who care to listen to input, before writing 5000 lines of code. I also work with customers, who have their own developers, with their own weird ideas about the world.
The biggest problem I see right now, except for occasions cowboy pretending to be a professional developer, is developer picking technologies without understanding it. We work with customer who picked technologies because they're interesting, not because it's what they need. When performance is terrible it becomes and operations issue and being told "Kafka is not actually a database and should be used as one" often isn't the answer they want. Or try telling a developer that the code he worked on for three months can be done by the existing load balancer in a few hours or that the ORM is actually writing terrible queries.
DevOps team, as in: "We use the shared knowledge of both parties" is fantastic, but operations is frequently an afterthought and not involved in the design fase.
If we're to take "DevOps" as developers doing operation, I'd prefer that we do the opposite and let operations do development. I think we'd get better results.
Another symptom of this is that when the QA/Staging function went away, load testing became perfunctory. Many of the performance problems we see should have been caught in QA. Devs are anxious to ship and get on to the next sprint, leaving app support and operations on the hook.
I think it goes a step farther back to product. PMs and analysts put constant pressure on developer teams to complete work quickly and that time pressure shows up on the next guy's plate, etc
"Trickle down software engineering"
In a sense, kubernetes is the new Linux / Bash of our time.
If it's painful, maybe it's just the abstraction not done right, but not the fault of abstraction itself.