I think this touches on an interesting problem. I don't see a reason to praise security experts though.
Here's a theory on why (some form of) Agile is problematic: Subdividing work into smaller parts is typically beneficial. However, in computer programming, this approach often fails because even the smaller tasks require some creativity, and unexpected challenges are likely to arise.
The person who did the subdividing gains a lot of insight while breaking down the larger problem. However, when transferring a portion of the work to a developer, much of this knowledge is inherently lost. The developer must then devise a creative solution, and lacking the necessary information, may either come up with a suboptimal solution or need further communication with the original architect.
There is no clear-cut solution. Some might argue for more experienced developers who have all the necessary knowledge readily available in their heads. Others might advocate for better design diagrams and documentation to capture all the relevant information. Ultimately, it requires careful consideration, or luck, to strike the right balance. But the dogmas of Agile certainly aren't helping much.
There are always interdependent units and I think the main issue is when we broke down those units into smaller tasks and then transfer the individual task to be implemented in isolation. There should always be someone that own that unit as a whole even if his job is only to answer architectural questions. A unit is assigned to one team (ad hoc or fixed) and a person (if the members count is greater than 3) is directing the implementation.
I still think that the team layout in The Mythical Man month is one of the best for software projects.
Right; "the surgeon" in Brooks' proposed team structure has the responsibility of maintaining the "theory" of the program as described by Peter Naur in 1985.
> lacking the necessary information, may either come up with a suboptimal solution or need further communication with the original architect.
> Others might advocate for better design diagrams and documentation to capture all the relevant information.
During the design phase group B made suggestions for the manner in which the extensions
should be accommodated and submitted them to group A for review. In several major cases
it turned out that the solutions suggested by group B were found by group A to make no
use of the facilities that were not only inherent in the structure of the existing
compiler but were discussed at length in its documentation, and to be based instead on
additions to that structure in the form of patches that effectively destroyed its power
and simplicity. [...] This is an example of how the full program text and additional
documentation is insufficient in conveying to even the highly motivated group B the
deeper insight into the design, that theory which is immediately present to the members
of group A. [...] A very important consequence of the Theory Building View is that
program revival, that is reestablishing the theory of a program merely from the
documentation, is strictly impossible. [0]
This is the simple and best way. It is also sustainable unlike a persistent state of Scrum. The engineer can take on features as the engineer sees fit. In fact it works this way for freelancers.
I don’t know about “own”. But you gotta let engineers lead projects that let them soak in a single code base enough to develop expertise. Letting every ticket go to a random dev, so it’s constant context switching is really silly.
My take on this is that the developers are costly. A working hour of single developer costs a lot of money. The organization will always try to get the maximum out of that one hour. Spending more money so that the developers are 'better' is not efficient. They just need to be good enough.
You’ve contradicted yourself. Which is it? Will companies try to get the maximum out of each hour of developer time, or do they just need to be good enough?
Perhaps what they meant is that companies will try to get the maximum in terms of development speed, while settling for good enough in terms of quality. And that's why we have so much crap software.
But they so consistently get terrible development speed and crap software. I’ve never been less productive than when I worked in a team that did big-A Agile.
That's the magic phrase that really does the heavy lifting here.
Companies and leadership have to define what "good enough" is for them, and in almost every case that comes down to profit and prestige (or power if you take it a step further).
As you said, developers are costly. When a company just wants profit it usually makes sense to hire someone just skilled enough to make the product shippable.
I think this is often the reasoning of management, however, what they don't see is that developer skill often follows a power law and the flow state is a multiplier on top of that. The cost of a midday check-in is not the cost of how much you pay that developer per hour because developers work output per time is nonlinear.
I believe the flow state is why it's so easy to believe the idea of the 10x engineer, we've all experienced the feeling of flying through code without barriers, and we have all felt the below average days where you may be tired and just can't quite load all that complexity into your head to get any real work done. So we believe that there are people out there who are so skilled that they exist in this flow state all the time.
Yes, developers should work on themselves to rely less on a state of flow to be high performers, but even so there will always a benefit to being "in the zone".
Had to do it at a company and thank god we fought against daily standups, but the constant retrospectives, the useless breaking down of tasks into subtasks and ranking how long it’ll take just for the benefit of the non technical manager and nobody else… I admit it was so bad I never wanted to work for a company after that.
Both creators of the Agile movement come from US Military experience. You did not realize the Agile daily stand-ups are basically military morning formations in disguise?....Just a way to keep Developers in line?
It’s like they took 'mission briefings' and turned them into 'tell me why your sticky note isn't moved yet.' Collaboration is ....Daily micromanagement...
> Both creators of the Agile movement come from US Military experience.
Last time I checked, the Agile Manifesto was drafted up by no less than 17 developers [1], the most prominent of them being Kent Beck, who's not directly famous for his US Military experience [2].
Last time I checked...Ken Schwaber of infamous fame, of the pyramid scheme called Scrum and their Certifications pyramid... and also Jeff Sutherland of infamous Scrum fame, are considered two of the core forces behind the Agile Manifest. And are two of the 17 Developers behind the manifest and both ex-US Military.
At least Kent Beck had the integrity of calling Agile certifications: “dishonest”, “a pyramid scheme”, & “cancer”.
And you know you can't spell Agile without reserving space for your Scrum Master... ;-)
Also selectively adding and removing characters of the Agile Manifest, to maximize or minimize their importance in the movement, as it is convenient for damage control purposes is not going to work out....
The dailies actually aren't bad in my experience (especially as a full remote), but the "pokerization" using "points" is one of dumbest shit I've seen. I hate working at a company too.
Software has gotten so bad over the past couple of decades. It's not as responsive as it was even in the 80-90s, yet we have orders of magnitude more powerful machines. Even simple website forms are often broken, due to everything from poor implementations to bit rot. It's perhaps just a coincidence that this time period has also been the rise of agile.
I can tell you for a fact that there was a period of 2 to 5 years in many parts of Europe—ranging from the Netherlands to Luxembourg, Belgium, Austria, France, and beyond—where you would not be hired as a contractor or consultant if you even dared to question some of the core tenets of Agile.
There are good software, but you have to look at the indy side or the FOSS side. It’s like how big companies conduct projects is not how it should be done.
Nah you're misremembering. I'd say it's stayed about the same. I think it took a step improvement when SSDs became available so maybe it's slightly better than it used to be.
Do you not remember when it used to take minutes to boot your PC? Starting MS Word could easily take 30s. Unless your computer is broken it takes nowhere near that long now.
> Do you not remember when it used to take minutes to boot your PC? Starting MS Word could easily take 30s. Unless your computer is broken it takes nowhere near that long now.
No, it never took that long if you took care of your PC, e.g. not have 50 things in your startup programs or a 99% full undefragged hard drive.
But yes, SSDs improved startup times massively, and were responsible for masking how bad software has gotten in the last 10 years.
Make software warrantable. If you sell software that purports to do X, you are at least implicitly warranting it is fit for purpose, etc. Allowing companies to disclaim all warranties means they have no incentive to make sure their software actually works well, because if it doesn’t, there ain’t shit a customer can do aside from spin the wheel and try the next one.
You know, our torts and contracts common law is a powerful mechanism, but there’s always movement to limit people’s ability to make use of it (ranging from tort reform to various doctrines on warranties and so on).
Agile, scrum, and OKRs have always bothered me (I know, how profound!)
My issue with them, though, is one that I don't usually see spelled out very often. All three promise, whether directly or implied, freedoms and responsibility being pushed further down the org chart. In reality all three are implemented in a way that further centralizes control towards the top of the org, leaving independent contributors at the bottom feeling almost entirely powerless.
I've always wanted to see a company flip OKRs on their head. Every employee should define key results for their little corner of the company. Managers should be responsible for taking those as fact and working to build a direction for the team that aligns as many of those results as possible. By the time you get to the top you have leadership setting the company's direction based on what the employees know to be most important rather than leadership seeming to energy on a hilltop to force key results down on everyone else.
Bottom up, not top down. Hire well, train well, and trust your people. It could fail spectacularly but at least your people would be respected and enjoy the ride, and we'd all learn something about the pros and cons of running a company without a dictatorship.
In my experience, most people I’ve worked with have a lot more capacity than they show at work. Who knows what your people are capable of if you empower your people to fix their own problems and treat them like adults? I’m sure they won’t all be able to step up to the plate, but it’s amazing how much of a difference the right corporate culture makes to the psychology of your employees.
For example, in my experience, people will only take initiative if they believe that they’ll get support from their team and manager for doing so. If not, they’ll bitch and moan about how bad things are but if you suggest fixing things, they’ll insist that change is impossible.
You are lucky you worked with those kind of people. A lot of the businesses I consult for have very mediocre teams (that’s why they need external help), and if they would decide what’s best for the business they wouldn’t get anything done that adds value to the business. These aren’t even necessarily large businesses, a lot of startups that are not doing high tech work (eg they “just” have an app or equivalent, and leadership is very strong commercially) have these kind of teams, and these developers would run the business into the ground if they would be dictating their leadership what to work on.
Again, it isn’t a bad idea, it’s just something that only works when you have a lot of senior engineers that have experience with sales/commercial aspects. Alternatively, it’s an already very much established business, with lots of processes and management layers, but in those cases these processes and management layers are designed as a safety net against poor business decisions from developers.
Companies usually exist to make money. I’ve seldomly seen software engineers that know the most effective way to make money. Therefore I see 2 problems with this bottom up approach:
- Software Engineers don’t understand the best way to make money
- Bottom up means that everyone will optimize locally for their own turf instead of converging on a common goal/set of goals for the company (ie reaching a global optimum)
>Software Engineers don’t understand the best way to make money
Why not? Isn't efficiency and profitability fundamentally an engineering problem? Design a product to be as profitable as possible, understanding which items to focus on, and which corners to cut?
Yes, typically the reason software engineers don't know how to make money (a premise I actually agree with) is because some product person is trying to translate the business problem into an engineering problem, rather than exposing the business problem to the engineer directly.
IMO too much is lost in this translation step and you end up often with engineers not understanding why the product they're building actually needs to be the way it does, and then a product person chasing them in circles and telling them absolutely everything except what the business problem actually is.
Efficiency is, profitability is not I think. Maybe I’ve worked with the wrong engineers, but many engineers I have worked with know how to solve technical problems, which does not necessarily mean customer problems.
The reasons disciplines like marketing and sales exist and also why processes like product discovery exist is that if we would leave product development to engineers, we’d get super efficient machines and software that are really elegant but don’t necessarily solve the problems customers have, are way too expensive to build and are not being sold.
It was my comment to a different comment on this thread. I’ll repeat it here:
Yes, and this somewhat recent notion that corporations only exist to benefit their shareholders. Corporations exist in order to do the business that they’re in the business of doing, and their structure is such that it allows for the distribution of risk and the possibility of profit for investors. Ideally, people would make investment decisions based on whether the business was sound. Like, yeah, this business has tons of satisfied customers, is doing relatively efficient business and makes money doing it: I’ll invest because that company is a winner. Everything now, though, is entirely judged on growth and bottom lines that are easily manipulable quarter to quarter. You’re a successful corporation if number goes up. The financialization of literally everything erodes the real purposes the corporation exists. And the mindless repetition of the line that corporations exist simply to make profits for the benefit of shareholders is bullshit. How many times have we seen companies chasing those profits in such a way as to destroy the longer-term viability/success of their business? Yeah, they made profit for the shareholder in the short term. Hurray. And destroyed the business in the longer-term. That seems to me to be pretty anti-shareholder.
In this model, and again I'm absolutely not claiming it would definitely work, leadership's role is to align feedback and goals defined by their direct reports. Everyone won't get what they want, but its every manager's role to both trust and respect their team and to play the role of aligning it to whatever business goals exist st their level. If done well, the manager would also be defining business goals based on team-provided key indicators.
I would also add that if a company scales to such a size that this model wouldn't work, in my opinion the company is too large. I would expect this to work very well on a small team, if managed well. I'd expect it to scale okay for a while but to fail miserably at the scale of big tech. That's a feature in my opinion, I expect we'd be better off collectively without such massive corporations accruing so much power.
I'd also note that i don't think your concern is unique to the bottom up approach. With the usual top down approach, leadership at the top is likely very skilled in defining business goals and how to make money. They rarely are skilled at the day to day work that actually keeps the company running. I've personally worked at companies where having leaders define direction while so disconnected from the front lines led to a very poorly functioning company.
Seems unlikely. A more plausible countertheory is that software is a very young industry and we're leaving the era where there were a lot of easy unknowns. The industry as a whole has figured out social networks, got a pretty good bead on the economics of software (especially open source). Databases are well developed, networking is well developed, new basic algorithms seem to have stopped appearing with regularity.
We might not have killed software engineering as much as we are exhausting the list of things that very smart programmers will just figure out on their own.
I agree with Moxie. But I’d go even further: I think the real problem stems from the modern corporate structure.
We have this idea in modern management theory that responsibility and decision making must travel up the corporate hierarchy. That execs are best positioned to make product level decisions. Or decisions of any kind. And peons at the bottom - the “individual contributors” who do all the programming must know the least about the product.
But that’s completely backwards. The people on the ground in any organisation know the most about what’s really going on. If we disempower them, and try to turn software engineering into an assembly line process (jira tickets go in, code comes out) then of course programmers stop innovating. They’re explicitly, and intentionally distanced from their capacity to improve the product in creative ways.
To be clear, I also don’t think the answer is to pretend we’re all equal and have an entirely flat management hierarchy. Different people have different skills, and a healthy organisation needs lots of talents to thrive. But there’s lots of ways to do that which don’t disempower the people on the ground.
The book Reinventing Organisations is a gem of a book. It totally blew my mind on this stuff. It talks about all sorts of different ways that innovative companies have upended their corporate structures, and along the way have been able to get the best from their people. If / when I start a company, I’m tempted to make it required reading.
> We have this idea in modern management theory that responsibility and decision making must travel up the corporate hierarchy. That execs are best positioned to make product level decisions. Or decisions of any kind. And peons at the bottom - the “individual contributors” who do all the programming must know the least about the product.
The problem is that the decision making is not about the product. The decision making is about making money. Unfortunately, for a lot of companies, making money is not equal to building good product. IC is completely unequipped to make decisions that would maximise the company top/ bottom line (this is a descriptive statement, and it is not about the individual either: modern corporation is famously broken)
> making money is not equal to building good product
I think in most cases, building a good product makes money in the long term. In the short term it might be possible to make money despite having a bad product, but as soon as the short term ends... look at Boeing and Intel.
This, short-term decisions win money in the short-term but businesses are always talking about retention and retention is most durable with a good product.
> IC is completely unequipped to make decisions that would maximise the company top/ bottom line
Right; companies want and expect ICs to keep their heads down and close tickets. Of course they’re ill equipped to make decisions outside the scope of their job role. If you expect nothing of someone, they will give you nothing in return.
I recently started work at a new company that tries to turn everyone in to a cog. No agency, every step requires other teams and people that may or may not respond. It's maddening.
It is almost like the modern software enterprise optimises towards minimising productivity, an emergent self-sabotaging. Everything is just avoiding responsibilities and blaming the proces, others or just the universe in general for things going to shit.
Anyway, I'm going to read that book now before i start my own company ;)
This is something I was actually shocked about coming from Government. People at the lower levels in a corporation just have no say, and might get lucky to hear about product direction and major decisions, often at far less detail than they actually need. The only way to really get in the know is to move up the chain, but those up the chain don't really seem to understand their own software products very well. Any discussion from an engineer is brushed off as being too "in the weeds" or "not high level enough," but it's not clear that those operating at a high level usually understand what's going on.
I really don't know what it is, but the corporate folks are just obsessed with hierarchy. It feels like it's in their blood. Even at the lower levels, they want to move up the chain so that they can have "real input," and they believe that culture must be set "from the top." Why? This is certainly one way to run things but it's not inherently the only way to run things. Individual teams _can_ set culture, and an individual team _could_ have better input on a strategic decision than the "decision makers." It just seems to be a forgone conclusion in the corporate world, and I can't see any reason for it other than status-seeking.
> corporate folks are just obsessed with hierarchy
You should read "The Dictator's Handbook." While it primarily applies to politicians in both democracies and authoritarian regimes, the thinking proposed in the book also applies to the corporate world. In other words, some people thrive on power, but these individuals are not always the best for the company due to a misalignment between their desires and the company's needs. Nevertheless, the system still works because... well, just read the book (or at least a summary of it)!
I'm somewhat pessimistic about corporate culture: companies have an instinctual drive to hunt for profit. If they didn't, they would go extinct by going bankrupt. This leads to a self-selection process that explains a lot. Combine this with the tendency of some people to strive for power—as you mentioned, "they want to move up the chain"—and we end up with a somewhat toxic mixture in today's economy.
Regarding the topic "Agile is killing software innovation," I would argue that it's not Agile itself that's stifling innovation, but rather the emergent structure and behavior of corporate entities. The combination of power-hungry individuals and profit-seeking companies stifles a lot of potential. It's highly inefficient and causes considerable misery, but it seems to function just well enough to keep the economy limping along.
One of the saving graces of corporate entities is their ability to buy up startups, for example.
That's an amazing book. Easily one of the most entertaining reads of that year for me. I remember it saying a lot about "inner circles" and the absolutely critical need for leadership to maintain a hold over them. So much so that it isn't that important for leadership to be good at the "job description" as long as they can identify the other centers of power and keep them happy.
I strongly agree, but winning this cultural/political war is hard.
And it's an extra hard war to fight in part because of what Moxie is saying, which I appreciate. The agile style promotes a cancerous software growth. Delivery deliver deliver, all the time, features forever.
It's good to be value minded, but making good long term decisions, thinking through your software and where it's headed, making long term decisions, and having artifacts and buy in for these long term plans is setting yourself up to resist the rot and decay that agile so easily lets in.
Agule and books like team topologies don't really have any commitment to the software as a whole. The team has a compact to ship ship ship ship, and so does every other team. But it imagines them all as islands, with only some interdependence. I struggle to think of examples to make this clear, but good software is holistic, good software tries to slam concerns & create smoothness across joints.
When agile lets software grow to be so very discombobulated, even if the corporation wants to be bottom up, the chaotic software architecture keeps engineers from seeing and springing on good ideas. Our potential as engineers is defined in sizable part by what we can understand and how well we understand the internals of our software, how well everything fits together, how legible the system is. Agile actively destroys this kernel for understanding, sacrifices the engineer's greatest leverage, for short virtue, for the reward of always be shipping.
If agile hasn't so overwhelmingly become a license for teams & product to do whatever, so long as the features keep shipping, the engineers would be in a better stance to have executive will & function, be able to go out there & steer & direct; we'd deserve to be heard better. But we are kept low by the thrashing chaos that comes in the absence of thinking ahead. The technical inadequacy erodes our ability to make political/organizational progress.
It seems that you are mystifying agile as this all-powerful machine.
Most corporations are mediocre, slow to change, and their software projects are just not ideal. And agile is a mindset to accept this, focus on the people, both devs and stakeholders (who fund the project, and ultimately the company that pays their salary). Probably by now it's a cliche to reference the first line of the manifesto, but... people above process, etc.
Agile sets the stage, helps grease the gears, by default it doesn't keep the project "accountable".
How the actual plans come out of it is up to the actual team, the leadership (where the classic fish and head olfactory law applies).
Islands and cliques form naturally anyway. And in some sense it's okay, because making too big plans tends to fail. (Again, agile at best nudges folks to think about commitments, what their work items mean for the team, for other teams ... but if course the further the others the less relatable is the importance of their needs.)
> we'd deserve to be heard better
yes, of course. and other "bottom feeders" too (excuse my phrasing), and middle managers too ... alas corporations tend to be broken as I mentioned.
And as a consequence, I found that outside our vocal minority most people develop the "if I don't care it doesn't hurt" mindset, they do want to close tickets and nothing more.
> Delivery deliver deliver, all the time, features forever.
This is the mindset, and a huge problem. But Agile does not cause it. (Or solve it, of course.) Money causes it. Features bring money, more than other things, most of the time. Money in the short term. And few managers seem to care about the long term.
If anything, Agile is liked by managers because it claims to allow building features non-stop.
Yes, and this somewhat recent notion that corporations only exist to benefit their shareholders. Corporations exist in order to do the business that they’re in the business of doing, and their structure is such that it allows for the distribution of risk and the possibility of profit for investors. Ideally, people would make investment decisions based on whether the business was sound. Like, yeah, this business has tons of satisfied customers, is doing relatively efficient business and makes money doing it: I’ll invest because that company is a winner. Everything now, though, is entirely judged on growth and bottom lines that are easily manipulable quarter to quarter. You’re a successful corporation if number goes up. The financialization of literally everything erodes the real purposes the corporation exists. And the mindless repetition of the line that corporations exist simply to make profits for the benefit of shareholders is bullshit. How many times have we seen companies chasing those profits in such a way as to destroy the longer-term viability/success of their business? Yeah, they made profit for the shareholder in the short term. Hurray. And destroyed the business in the longer-term. That seems to me to be pretty anti-shareholder.
I don't think the structure is the problem. I think it's fine that the higher you go the bigger decisions and responsibility there is.
The part that I think is not working is that managers neglect one of the basic behaviors they should be engaging in. Weekly one-on-ones done properly. That's the place for information to go from directs to managers.
The same management and teams that turn agile into the hellscape many folks experience, would make waterfall or whatever the same mess. I've seen pre and post agile, it's all due to attitude, skills, and experience.
Training, too. The people who load up on certifications then attempt to make reality conform to their understanding of the course content they learned rather than vice versa are the culprits here.
> the Signal founder stole the show with an opening chat laying out a case for reclaiming the "magic" of software development that's been lost after 20 years
How to tell you’ve been in an industry for more than 20 years.
Spend time around excited young programmers, and you’ll find the magic is still there. It’s just different than it was 20 years ago. I remember the people with 20 years of experience back then telling me how awful everything was, but I was having way too much fun making things to give them much attention.
The industry took the good ideas of Agile (don't overplan, work in small steps,
reassess regularly, oh and write tests and do code reviews) and absorbed them into normal software engineering. So now whenever someone describes themselves as an "agile" programmer it suggests a strict devotion to stand-up meetings, kanban boards, and various other cultic rituals.
But I'm not convinced that Agile is really to blame for the compartmentalization of knowledge and down-skilling of software engineering, as it has happened in every other industry too. It seems like a tendency of mass production. Car companies may be founded by small teams of master generalist mechanics, but they end up employing hundreds of thousands of ordinary technicians who know only what they need to know. Same for furniture factories and carpenters. Why not for software too?
And Moxie is complaining about something that goes against the early tenets of agile as listed in the manifesto.
He’s complaining that hierarchical silod black box teams are bad. Early agile was advocating that was bad!
The bigger issue is that Agile doesn’t mean anything anymore because when it came about it was just a reaction to the previous form of software management. It was at most a few completely different methodologies agreeing on some high level principles. You had to pick one of those methodologies to get any real practices out of it.
And many of those practices would today seem like no brainers. Of course we should be merging everyone’s software frequently (do quarterly merges exist anywhere anymore? They did). Developers writing automated tests is near universal now, it was rare then. And with the rise is SaaS we’ve taken frequent deployment to a level unthinkable when Agile was developed.
When people complain about “Agile” now it’s meaningless. It’s a catch all for everything unpleasant in software planning and coordination, just like waterfall was when Agile was ascendant.
Yes, interchangeable parts are a central component of mass production. And you have to remove the application of skilled labor to get to interchangeable parts, because that requires mechanization, not the intensively individualized input of skilled workers.
You can make sense of the framework-ization of front-end that way. Monkey pick pre-fabbed block from framework basket, monkey wire together blocks, monkey get banana. But of course the reality is far different. Monkey no troubleshoot or bug-hunt. You need humans for that. Skilled labor.
We had a backlog grooming meeting yesterday. The boss asked me to estimate a task that was a bug fix for some code I am completely unfamiliar with. If I had a dollar for every time that has happened I think I could buy a used jet ski in good condition. So I did the usual thing. I told her that I didn't know how long it would take. Then she said ok, then this is a task to estimate the size of the bug fix task, how long will that take. Rather than continue to struggle I said "I'm about to lie to you" and then pulled a (Fibonacci) number out of my butt. That was enough to make her happy and we moved on. I'll give it my best effort and there will never be any accountability for my stinky estimate.
This has been my general experience with agile as practiced at three different outfits.
I've been around a bit while contracting and have experienced about 15 organisations, a handful of which did not estimate work except the occasional natural tendency of one engineer to say to another "this seems a bit big". That group that doesn't insist on boilerplate agile ritual is way more effective. The trouble is that some manager or another higher up the chain will assume time and motion studies on tech can be done locally (or at all) demands a dashboard of what is essentially made up information about what kind of work the organisation is doing.
Agile per se isn't killing software innovation: training non-technical people who have no concept whatsoever of what's going on and then insisting they invent arbitrary funny money to express work size.
The ideal way to resist this appears to emerge naturally in most even vaguely experienced engineers: double your estimates.
Taylorism was bullshit in its original form too. I had a history professor in graduate school whose PhD/book was about this. I can try to find it if you’re interested.
Hm, the book is about Taylorism from 1945 on it seems, but it’s Stephen Waring, Taylorism Transformed: Scientific Management Theory since 1945. Looks like it was published by UNC Press 1991.
I do remember conversations with him to the effect that Taylorism ab initio was as much ideological in aim as seeking efficiency, but (1) that’s obviously his view (though a considered one after studying the sources) and (2) I don’t have any citations I could give you to support that beyond vaguely recalled conversations after class.
This is my pet peeve with Jira culture. Estimates are always exact. You were forced to estimate 15±0 instead of 1-50 which would be a true reflection of your uncertainty.
I have yet to see a project management tool that even lets you write down uncertainty.
I use my own Excel sheet that calculates project risk from each task using a formula based on:
Estimated hours, in whole hour increments (i.e. 1 hour is the minimum allowed estimate). Usually you’d then multiply the engineering estimate by 3 to get something more realistic.
Uncertainty of the estimate, with a scale of 1 to 3, where 1 is “I’ve done this before and have a very good idea of how long it takes” and 3 is “I have no idea and making my best guess”
Impact to the project if it doesn’t get done in time on a scale of 1 to 3, where 1 is a small impact and 3 is high
Then the Exposure which is simply Uncertainty * Impact.
This method highlights the most critical areas so at least people can get an idea of when we’re entering into uncharted waters and to expect more possible delays.
An issue still unaddressed is that when you estimate 40 hours for something, management thinks “ok, great, it will be done in one week”, and then proceed to bury you in emails and meetings completely unrelated to the project.
Let’s be honest. Your boss asked a reasonable question and your answer could have been something like “I need a day to familiarize myself with the code so I can give a rough but reasonable estimate. Also, I apologize I and the team fucked up by not sharing knowledge properly with each other and letting knowledge islands emerge”
You are very good at this. Without even knowing what the code is about you are able to discern that it would take me about a day to get to sufficient familiarity. Meanwhile I, who have been working on the project for twelve years and wrote most of it, still don't know. Are you available as a mentor?
So you worked for twelve years on that code base and you still don’t know how to estimate a bug fix? There seem to be other problems then.
And you are right, I don’t know anything about that code base, certainly not enough to know if estimates should be easy or not. But the fact that they aren’t is a sign of problems either way.
And I stand by my point that your boss’ question is reasonable, and the team should do some introspection to learn why such a fair question cannot be easily answered
I have seen bugs, in this project and several others, that range from actually "actually a feature" to both serious and unsolvable. Most of them are somewhere in the middle. But before knowing anything about the code or the bug, I can't tell where this task will land on that distribution. If you coerce an estimate out of me all I can do is guess the median value, which you can fill in without my help, while I go and work on the bug. Asking me for a number before I know anything about it is a waste of both of our time.
The people who originally created the agile principles devised it for consulting agencies, not product development. A lot of the ideas are relevant still, but there's a bit of chaff to sort through to get at the wheat.
Not to mention the now capital-A Agile practices, courses, certifications(!!) and processes (!!?!?) which end up being exactly against the spirit of the original manifesto.
My impression is larger or more synthetic orgs don't want devs to lead things anymore, they want a very top down development of vision -> product -> ux -> project management, and then the developers scurry around and do whatever it is they do using cloud something something, hopefully in as commodity a way as possible. The article clearly states the problems with this; devs can no longer understand the whole picture or make important top level suggestions, and the cloud dictates what can be done.
This meme that "Agile is evil", needs to die. No, people, you've just had really bad managers who don't know how to manage software engineering teams.
SCRUM isn't agile, it's SCRUM. Kanban isn't agile, it's Kanban. Doing something that has lots of process but that isn't waterfall isn't agile just because it isn't waterfall - it's a thing with lots of process.
Agile is about responding to the context of the team and the customer. You probably need a way to keep track of what needs to be done, and have regular updates about blockers and progress, but that's it.
The problem is if you ask people who have never been software engineers to manage a software engineering project, they're going to struggle to communicate, because they have no natural intuition of what the hell is going on. The project management industry is quite happy to serve these people by inventing big complex processes to keep them busy and sell them books, courses, certifications and conferences.
Most of big tech won't allow someone to manage an engineering team or project manage those projects without a good amount of hands-on engineering experience. The rest of industry needs to pick up on this.
People without that experience can learn. But they should adapt to software being what software is, not try and bring processes derived from those invented for other domains to software.
Agile done well is absolutely amazing, when its in the hands of teams who have all built software, understand their customer problem and the milestones they need to hit (yes, deadlines are unavoidable in any modern business environment, sorry), and each other.
Trust me, I remember when waterfall was the way. Waterfall is so bad, that even when you do 2-week long waterfall sprints (waves at SCRUM), it feels awful.
Go back to the Agile Manifesto [1]. It's all you need. Keep track of what was decided or discovered (bugs, features that are needed, and so on). If you have regular bottlenecks due to skill/knowledge imbalances, steal pairing from XP. Maybe explore Theory of Constraints but resist Kanban unless you understand exactly what it is doing and why. Have regular conversations as a group, and one-on-ones, not just between managers and reports, but between each other. Hold short but regular retros to understand systemic issues and then make sure the actions are actioned.
> This meme that "Agile is evil", needs to die. No, people, you've just had really bad managers
If the bad managers are outliers, then fine. If the bad managers are the norm, then we need a way of working that fits the average manager, even if you call them "bad". Most people in most companies are necessarily going to be of average quality.
Judging from years of talking to devs and reading books and articles and comments on the net, what you call "bad managers" are not exceptions, but rather the norm.
> Agile done well is absolutely amazing
If your team is made up of very competent people, any methodology is likely to work great. But most teams are not. We need something that works reasonably well even when the managers and devs are just average.
Agile can work great for average people, it needs less skill than any other method I know.
Can you explain to me your thinking - which I might be implying and you might not actually think - as to how heavy processes like SCRUM and Kanban turn average teams into productive ones?
Trying to reinvent the story of Agile is not going to fly. You have to look at who are the signers of the manifest and what they did with Agile, to understand what Agile really is.
I'm not interested in the history. I'm interested in the practice of building software in changing contexts where everyone is constantly learning and discovering. It doesn't matter who signed it and what they did next, it only matters how you and your colleagues get good work done.
> I'm interested in the practice of building software in changing contexts where everyone is constantly learning and discovering.
No. Not this please no. Where is that context where your team is constantly learning and discovering. ?
Please tell me what Software is that? A real time control system for a high speed train, airplane or space ship, where lives are at stake? A hospital or ER patient management system where a library bug can mean the patient get's a fatal dose of 250 ml of something instead of the life saving 2,5 ml?
Or a context where everyone is constantly "learning and discovering" the best color for the web page of your online shopping?
Sure, waterfall might work for very small fixed contexts. Control software for a spacecraft is a great example. I can sit down now and start designing software to do physical calculations for a system that won't ship for 7+ years, confident the laws of physics won't change in that time. In fact, NASA did this in the Apollo program.
Most software is used by customers with changing needs and competitive pulls. I've built software in the last 2 years that we would have taken a radically different approach to if LLMs were available, because they're great at summarising natural language. But they weren't. Imagine if we'd been building something for 2 years and never learned anything about changing technology or customer expectations - instead we shipped something fast, and then looked back after we learned about new things and made it better. That's how the World should work, and why software is not like building a bridge or a house or basically anything else in the history of engineering.
As to your specific exampleS: I have some detailed knowledge of hospital EHR systems, and they are so massively, hilariously broken in ways that modern software practices could easily help address. I suggest reading Accelerate [1] for data and examples of how agile applied to DevOps practice in particular means you can be more responsive and produce safer software, especially in that context.
I, like you, also prefer software for stuff that matters. I've been trained in formal methods, I've done embedded systems development, I've worked in medical/healthcare contexts. Your statement that what I've written above does not apply to those environments (including the use of formal verification methods), doesn't make sense to me.
Agile advocates love to contrast their approach with Waterfall, which contrary to Agile folklore, was neither rigid nor uniformly applied.
Let's review the Agile Manifest from the perspective of delivering Software That Matters, that I define as Software where failures can cause significant loss of Lives, Money, or Reputation:
1. Early and Continuous Delivery
Speed over quality in critical software contexts invites catastrophic failures.
Instead just implement phased delivery schedules that prioritize comprehensive testing and stability over speed, ensuring system reliability.
2. Welcoming Late Changes
This flexibility often leads to scope creep and consequential delays, undermining system integrity.
Instead have a change control board to rigorously evaluate the implications of proposed changes, safeguarding project stability.
3. Frequent Software Releases
Regular release demands degrade software quality and exhaust development teams, detracting from critical system safety.
Instead base your release schedules on thorough testing and feature completion, ensuring each release enhances system robustness.
4. Daily Business-Developer Interaction
Frequent meetings disrupt productivity and can force development compromises, jeopardizing precise project requirements.
Instead opt for structured weekly updates and essential communication to maintain clarity and focus without sacrificing project integrity.
5. Trusting Motivated Individuals
Sole reliance on individual motivation without structured oversight introduces risks of inconsistent quality in high-stakes projects.
Instead clearly define roles and accountability metrics, aligning individual contributions directly with critical project outcomes.
6. Face-to-Face Communication
This traditional communication mode fails to accommodate remote participants or ensure accurate documentation of critical decisions.
Instead leverage advanced digital communication tools that enhance documentation, suitable for modern, distributed teams.
7. Working Software as Progress Measure
Focusing solely on functional completions neglects essential aspects like security, scalability, and user experience.
Instead expand progress indicators to include comprehensive evaluations of non-functional requirements, providing a holistic view of system health.
8. Sustainable Development Pace
The ideal of an unvarying pace is impractical under the variable pressures of high-stakes software development.
Instead adopt development rhythms to real-world project dynamics, accommodating natural workflow variations to prevent burnout and ensure quality.
9. Continuous Technical Excellence
An unrelenting pursuit of technical perfection can obstruct pragmatic and timely deliverables in critical scenarios.
Instead aim for pragmatically viable solutions that meet current needs with provisions for iterative enhancements as future requirements evolve.
10. Maximizing Work Not Done
Striving for minimalism can neglect essential complexities, endangering the whole system.
Instead prioritize a thorough approach that covers all necessary bases, balancing simplicity with comprehensive risk management.
11. Self-Organizing Teams
Teams without structured guidance can diverge from critical safety and performance standards.
Instead implement a framework that offers direction and support, ensuring teams adhere to best practices essential for system reliability.
12. Regular Reflection and Adjustment
Constant reassessment cycles can disrupt ongoing projects and dilute focus on strategic goals.
Instead space out reflective practices to better assess their impact and refine processes without compromising ongoing project stability.
OK, so this is getting contentious, but it seems to me you're not up to speed with industry studies.
Every study of continuous delivery suggests it improves quality by finding bugs earlier (because it has continuous integration - automated and rigorous testing to help identify unintended regressions - as a dependency), and reducing MTTR when undetected bugs do occur, because releasing fixes is quick and standardised. I'll refer you to the book in my comment above which demonstrates this with data collected from thousands of companies.
If you don't understand the reasons behind this, you're not going to understand why any of the other things lead to safer software, released earlier, and producing more value in the short, medium and long terms. So we should stop here.
I'd strongly encourage you to start by reading Accelerate and then come back and take another swing at it if you want, but given your current perspective - which sounds highly authoritarian, cynical, and divorced from data collected over a wide range of industry contexts - there's no point in us discussing this further.
Control systems are a special case, where there is indeed no constant change. And there are more special cases like this, of course.
But for most software, the customer keeps changing his mind and requesting new or different things. This includes hospitals. What's the impact of a bug have to do with it? I don't see the logic in "bugs are very bad -> this proves requirements don't change". Non sequitur.
If you have not experienced this, you're very lucky. But look around, in this thread and elswhere, and see how many people are claiming the "adapt to constant change" claim of Agile is pointless. You'll find very few.
Not that Agile is great. But this particular problem exists, and needs a solution. Software where requirements don't change is rare. It just is.
109 comments
[ 0.23 ms ] story [ 103 ms ] threadHere's a theory on why (some form of) Agile is problematic: Subdividing work into smaller parts is typically beneficial. However, in computer programming, this approach often fails because even the smaller tasks require some creativity, and unexpected challenges are likely to arise.
The person who did the subdividing gains a lot of insight while breaking down the larger problem. However, when transferring a portion of the work to a developer, much of this knowledge is inherently lost. The developer must then devise a creative solution, and lacking the necessary information, may either come up with a suboptimal solution or need further communication with the original architect.
There is no clear-cut solution. Some might argue for more experienced developers who have all the necessary knowledge readily available in their heads. Others might advocate for better design diagrams and documentation to capture all the relevant information. Ultimately, it requires careful consideration, or luck, to strike the right balance. But the dogmas of Agile certainly aren't helping much.
I still think that the team layout in The Mythical Man month is one of the best for software projects.
> Others might advocate for better design diagrams and documentation to capture all the relevant information.
[0] https://pages.cs.wisc.edu/~remzi/Naur.pdfThat's the magic phrase that really does the heavy lifting here.
Companies and leadership have to define what "good enough" is for them, and in almost every case that comes down to profit and prestige (or power if you take it a step further).
As you said, developers are costly. When a company just wants profit it usually makes sense to hire someone just skilled enough to make the product shippable.
I believe the flow state is why it's so easy to believe the idea of the 10x engineer, we've all experienced the feeling of flying through code without barriers, and we have all felt the below average days where you may be tired and just can't quite load all that complexity into your head to get any real work done. So we believe that there are people out there who are so skilled that they exist in this flow state all the time.
Yes, developers should work on themselves to rely less on a state of flow to be high performers, but even so there will always a benefit to being "in the zone".
It’s like they took 'mission briefings' and turned them into 'tell me why your sticky note isn't moved yet.' Collaboration is ....Daily micromanagement...
Last time I checked, the Agile Manifesto was drafted up by no less than 17 developers [1], the most prominent of them being Kent Beck, who's not directly famous for his US Military experience [2].
[1] https://en.wikipedia.org/wiki/Agile_software_development
[2] https://en.wikipedia.org/wiki/Kent_Beck
At least Kent Beck had the integrity of calling Agile certifications: “dishonest”, “a pyramid scheme”, & “cancer”.
https://zhiminzhan.medium.com/agile-certifications-are-wrong...
And you know you can't spell Agile without reserving space for your Scrum Master... ;-)
Also selectively adding and removing characters of the Agile Manifest, to maximize or minimize their importance in the movement, as it is convenient for damage control purposes is not going to work out....
Do you not remember when it used to take minutes to boot your PC? Starting MS Word could easily take 30s. Unless your computer is broken it takes nowhere near that long now.
No, it never took that long if you took care of your PC, e.g. not have 50 things in your startup programs or a 99% full undefragged hard drive.
But yes, SSDs improved startup times massively, and were responsible for masking how bad software has gotten in the last 10 years.
You know, our torts and contracts common law is a powerful mechanism, but there’s always movement to limit people’s ability to make use of it (ranging from tort reform to various doctrines on warranties and so on).
My issue with them, though, is one that I don't usually see spelled out very often. All three promise, whether directly or implied, freedoms and responsibility being pushed further down the org chart. In reality all three are implemented in a way that further centralizes control towards the top of the org, leaving independent contributors at the bottom feeling almost entirely powerless.
I've always wanted to see a company flip OKRs on their head. Every employee should define key results for their little corner of the company. Managers should be responsible for taking those as fact and working to build a direction for the team that aligns as many of those results as possible. By the time you get to the top you have leadership setting the company's direction based on what the employees know to be most important rather than leadership seeming to energy on a hilltop to force key results down on everyone else.
Bottom up, not top down. Hire well, train well, and trust your people. It could fail spectacularly but at least your people would be respected and enjoy the ride, and we'd all learn something about the pros and cons of running a company without a dictatorship.
For example, in my experience, people will only take initiative if they believe that they’ll get support from their team and manager for doing so. If not, they’ll bitch and moan about how bad things are but if you suggest fixing things, they’ll insist that change is impossible.
Again, it isn’t a bad idea, it’s just something that only works when you have a lot of senior engineers that have experience with sales/commercial aspects. Alternatively, it’s an already very much established business, with lots of processes and management layers, but in those cases these processes and management layers are designed as a safety net against poor business decisions from developers.
- Software Engineers don’t understand the best way to make money - Bottom up means that everyone will optimize locally for their own turf instead of converging on a common goal/set of goals for the company (ie reaching a global optimum)
Why not? Isn't efficiency and profitability fundamentally an engineering problem? Design a product to be as profitable as possible, understanding which items to focus on, and which corners to cut?
IMO too much is lost in this translation step and you end up often with engineers not understanding why the product they're building actually needs to be the way it does, and then a product person chasing them in circles and telling them absolutely everything except what the business problem actually is.
The reasons disciplines like marketing and sales exist and also why processes like product discovery exist is that if we would leave product development to engineers, we’d get super efficient machines and software that are really elegant but don’t necessarily solve the problems customers have, are way too expensive to build and are not being sold.
Don’t take offense at the “mindless repetition” part though, please. I genuinely want feedback from somebody who says what you just said.
Yes, and this somewhat recent notion that corporations only exist to benefit their shareholders. Corporations exist in order to do the business that they’re in the business of doing, and their structure is such that it allows for the distribution of risk and the possibility of profit for investors. Ideally, people would make investment decisions based on whether the business was sound. Like, yeah, this business has tons of satisfied customers, is doing relatively efficient business and makes money doing it: I’ll invest because that company is a winner. Everything now, though, is entirely judged on growth and bottom lines that are easily manipulable quarter to quarter. You’re a successful corporation if number goes up. The financialization of literally everything erodes the real purposes the corporation exists. And the mindless repetition of the line that corporations exist simply to make profits for the benefit of shareholders is bullshit. How many times have we seen companies chasing those profits in such a way as to destroy the longer-term viability/success of their business? Yeah, they made profit for the shareholder in the short term. Hurray. And destroyed the business in the longer-term. That seems to me to be pretty anti-shareholder.
I would also add that if a company scales to such a size that this model wouldn't work, in my opinion the company is too large. I would expect this to work very well on a small team, if managed well. I'd expect it to scale okay for a while but to fail miserably at the scale of big tech. That's a feature in my opinion, I expect we'd be better off collectively without such massive corporations accruing so much power.
I'd also note that i don't think your concern is unique to the bottom up approach. With the usual top down approach, leadership at the top is likely very skilled in defining business goals and how to make money. They rarely are skilled at the day to day work that actually keeps the company running. I've personally worked at companies where having leaders define direction while so disconnected from the front lines led to a very poorly functioning company.
We might not have killed software engineering as much as we are exhausting the list of things that very smart programmers will just figure out on their own.
Closing and locking thousands of Signal issues automatically and locking all conversations for "spring cleaning"...
... which feels exactly like something somebody oversubscribed to agile methods in the wrong way would do.
Great article. If only we could make the people holding the bags of money understand...
We have this idea in modern management theory that responsibility and decision making must travel up the corporate hierarchy. That execs are best positioned to make product level decisions. Or decisions of any kind. And peons at the bottom - the “individual contributors” who do all the programming must know the least about the product.
But that’s completely backwards. The people on the ground in any organisation know the most about what’s really going on. If we disempower them, and try to turn software engineering into an assembly line process (jira tickets go in, code comes out) then of course programmers stop innovating. They’re explicitly, and intentionally distanced from their capacity to improve the product in creative ways.
To be clear, I also don’t think the answer is to pretend we’re all equal and have an entirely flat management hierarchy. Different people have different skills, and a healthy organisation needs lots of talents to thrive. But there’s lots of ways to do that which don’t disempower the people on the ground.
The book Reinventing Organisations is a gem of a book. It totally blew my mind on this stuff. It talks about all sorts of different ways that innovative companies have upended their corporate structures, and along the way have been able to get the best from their people. If / when I start a company, I’m tempted to make it required reading.
The problem is that the decision making is not about the product. The decision making is about making money. Unfortunately, for a lot of companies, making money is not equal to building good product. IC is completely unequipped to make decisions that would maximise the company top/ bottom line (this is a descriptive statement, and it is not about the individual either: modern corporation is famously broken)
I think in most cases, building a good product makes money in the long term. In the short term it might be possible to make money despite having a bad product, but as soon as the short term ends... look at Boeing and Intel.
Right; companies want and expect ICs to keep their heads down and close tickets. Of course they’re ill equipped to make decisions outside the scope of their job role. If you expect nothing of someone, they will give you nothing in return.
I recently started work at a new company that tries to turn everyone in to a cog. No agency, every step requires other teams and people that may or may not respond. It's maddening.
It is almost like the modern software enterprise optimises towards minimising productivity, an emergent self-sabotaging. Everything is just avoiding responsibilities and blaming the proces, others or just the universe in general for things going to shit.
Anyway, I'm going to read that book now before i start my own company ;)
I really don't know what it is, but the corporate folks are just obsessed with hierarchy. It feels like it's in their blood. Even at the lower levels, they want to move up the chain so that they can have "real input," and they believe that culture must be set "from the top." Why? This is certainly one way to run things but it's not inherently the only way to run things. Individual teams _can_ set culture, and an individual team _could_ have better input on a strategic decision than the "decision makers." It just seems to be a forgone conclusion in the corporate world, and I can't see any reason for it other than status-seeking.
You should read "The Dictator's Handbook." While it primarily applies to politicians in both democracies and authoritarian regimes, the thinking proposed in the book also applies to the corporate world. In other words, some people thrive on power, but these individuals are not always the best for the company due to a misalignment between their desires and the company's needs. Nevertheless, the system still works because... well, just read the book (or at least a summary of it)!
I'm somewhat pessimistic about corporate culture: companies have an instinctual drive to hunt for profit. If they didn't, they would go extinct by going bankrupt. This leads to a self-selection process that explains a lot. Combine this with the tendency of some people to strive for power—as you mentioned, "they want to move up the chain"—and we end up with a somewhat toxic mixture in today's economy.
Regarding the topic "Agile is killing software innovation," I would argue that it's not Agile itself that's stifling innovation, but rather the emergent structure and behavior of corporate entities. The combination of power-hungry individuals and profit-seeking companies stifles a lot of potential. It's highly inefficient and causes considerable misery, but it seems to function just well enough to keep the economy limping along.
One of the saving graces of corporate entities is their ability to buy up startups, for example.
And it's an extra hard war to fight in part because of what Moxie is saying, which I appreciate. The agile style promotes a cancerous software growth. Delivery deliver deliver, all the time, features forever.
It's good to be value minded, but making good long term decisions, thinking through your software and where it's headed, making long term decisions, and having artifacts and buy in for these long term plans is setting yourself up to resist the rot and decay that agile so easily lets in.
Agule and books like team topologies don't really have any commitment to the software as a whole. The team has a compact to ship ship ship ship, and so does every other team. But it imagines them all as islands, with only some interdependence. I struggle to think of examples to make this clear, but good software is holistic, good software tries to slam concerns & create smoothness across joints.
When agile lets software grow to be so very discombobulated, even if the corporation wants to be bottom up, the chaotic software architecture keeps engineers from seeing and springing on good ideas. Our potential as engineers is defined in sizable part by what we can understand and how well we understand the internals of our software, how well everything fits together, how legible the system is. Agile actively destroys this kernel for understanding, sacrifices the engineer's greatest leverage, for short virtue, for the reward of always be shipping.
If agile hasn't so overwhelmingly become a license for teams & product to do whatever, so long as the features keep shipping, the engineers would be in a better stance to have executive will & function, be able to go out there & steer & direct; we'd deserve to be heard better. But we are kept low by the thrashing chaos that comes in the absence of thinking ahead. The technical inadequacy erodes our ability to make political/organizational progress.
Most corporations are mediocre, slow to change, and their software projects are just not ideal. And agile is a mindset to accept this, focus on the people, both devs and stakeholders (who fund the project, and ultimately the company that pays their salary). Probably by now it's a cliche to reference the first line of the manifesto, but... people above process, etc.
Agile sets the stage, helps grease the gears, by default it doesn't keep the project "accountable".
How the actual plans come out of it is up to the actual team, the leadership (where the classic fish and head olfactory law applies).
Islands and cliques form naturally anyway. And in some sense it's okay, because making too big plans tends to fail. (Again, agile at best nudges folks to think about commitments, what their work items mean for the team, for other teams ... but if course the further the others the less relatable is the importance of their needs.)
> we'd deserve to be heard better
yes, of course. and other "bottom feeders" too (excuse my phrasing), and middle managers too ... alas corporations tend to be broken as I mentioned.
And as a consequence, I found that outside our vocal minority most people develop the "if I don't care it doesn't hurt" mindset, they do want to close tickets and nothing more.
This is the mindset, and a huge problem. But Agile does not cause it. (Or solve it, of course.) Money causes it. Features bring money, more than other things, most of the time. Money in the short term. And few managers seem to care about the long term.
If anything, Agile is liked by managers because it claims to allow building features non-stop.
The part that I think is not working is that managers neglect one of the basic behaviors they should be engaging in. Weekly one-on-ones done properly. That's the place for information to go from directs to managers.
How to tell you’ve been in an industry for more than 20 years.
Spend time around excited young programmers, and you’ll find the magic is still there. It’s just different than it was 20 years ago. I remember the people with 20 years of experience back then telling me how awful everything was, but I was having way too much fun making things to give them much attention.
But I'm not convinced that Agile is really to blame for the compartmentalization of knowledge and down-skilling of software engineering, as it has happened in every other industry too. It seems like a tendency of mass production. Car companies may be founded by small teams of master generalist mechanics, but they end up employing hundreds of thousands of ordinary technicians who know only what they need to know. Same for furniture factories and carpenters. Why not for software too?
He’s complaining that hierarchical silod black box teams are bad. Early agile was advocating that was bad!
The bigger issue is that Agile doesn’t mean anything anymore because when it came about it was just a reaction to the previous form of software management. It was at most a few completely different methodologies agreeing on some high level principles. You had to pick one of those methodologies to get any real practices out of it.
And many of those practices would today seem like no brainers. Of course we should be merging everyone’s software frequently (do quarterly merges exist anywhere anymore? They did). Developers writing automated tests is near universal now, it was rare then. And with the rise is SaaS we’ve taken frequent deployment to a level unthinkable when Agile was developed.
When people complain about “Agile” now it’s meaningless. It’s a catch all for everything unpleasant in software planning and coordination, just like waterfall was when Agile was ascendant.
You can make sense of the framework-ization of front-end that way. Monkey pick pre-fabbed block from framework basket, monkey wire together blocks, monkey get banana. But of course the reality is far different. Monkey no troubleshoot or bug-hunt. You need humans for that. Skilled labor.
This has been my general experience with agile as practiced at three different outfits.
Agile per se isn't killing software innovation: training non-technical people who have no concept whatsoever of what's going on and then insisting they invent arbitrary funny money to express work size.
The ideal way to resist this appears to emerge naturally in most even vaguely experienced engineers: double your estimates.
I do remember conversations with him to the effect that Taylorism ab initio was as much ideological in aim as seeking efficiency, but (1) that’s obviously his view (though a considered one after studying the sources) and (2) I don’t have any citations I could give you to support that beyond vaguely recalled conversations after class.
I have yet to see a project management tool that even lets you write down uncertainty.
Estimated hours, in whole hour increments (i.e. 1 hour is the minimum allowed estimate). Usually you’d then multiply the engineering estimate by 3 to get something more realistic.
Uncertainty of the estimate, with a scale of 1 to 3, where 1 is “I’ve done this before and have a very good idea of how long it takes” and 3 is “I have no idea and making my best guess”
Impact to the project if it doesn’t get done in time on a scale of 1 to 3, where 1 is a small impact and 3 is high
Then the Exposure which is simply Uncertainty * Impact.
This method highlights the most critical areas so at least people can get an idea of when we’re entering into uncharted waters and to expect more possible delays.
An issue still unaddressed is that when you estimate 40 hours for something, management thinks “ok, great, it will be done in one week”, and then proceed to bury you in emails and meetings completely unrelated to the project.
And you are right, I don’t know anything about that code base, certainly not enough to know if estimates should be easy or not. But the fact that they aren’t is a sign of problems either way.
And I stand by my point that your boss’ question is reasonable, and the team should do some introspection to learn why such a fair question cannot be easily answered
To me, it is not a reasonable question because there are so many assumptions smuggled in with it.
Not to mention the now capital-A Agile practices, courses, certifications(!!) and processes (!!?!?) which end up being exactly against the spirit of the original manifesto.
SCRUM isn't agile, it's SCRUM. Kanban isn't agile, it's Kanban. Doing something that has lots of process but that isn't waterfall isn't agile just because it isn't waterfall - it's a thing with lots of process.
Agile is about responding to the context of the team and the customer. You probably need a way to keep track of what needs to be done, and have regular updates about blockers and progress, but that's it.
The problem is if you ask people who have never been software engineers to manage a software engineering project, they're going to struggle to communicate, because they have no natural intuition of what the hell is going on. The project management industry is quite happy to serve these people by inventing big complex processes to keep them busy and sell them books, courses, certifications and conferences.
Most of big tech won't allow someone to manage an engineering team or project manage those projects without a good amount of hands-on engineering experience. The rest of industry needs to pick up on this.
People without that experience can learn. But they should adapt to software being what software is, not try and bring processes derived from those invented for other domains to software.
Agile done well is absolutely amazing, when its in the hands of teams who have all built software, understand their customer problem and the milestones they need to hit (yes, deadlines are unavoidable in any modern business environment, sorry), and each other.
Trust me, I remember when waterfall was the way. Waterfall is so bad, that even when you do 2-week long waterfall sprints (waves at SCRUM), it feels awful.
Go back to the Agile Manifesto [1]. It's all you need. Keep track of what was decided or discovered (bugs, features that are needed, and so on). If you have regular bottlenecks due to skill/knowledge imbalances, steal pairing from XP. Maybe explore Theory of Constraints but resist Kanban unless you understand exactly what it is doing and why. Have regular conversations as a group, and one-on-ones, not just between managers and reports, but between each other. Hold short but regular retros to understand systemic issues and then make sure the actions are actioned.
It's not that hard. It's just Agile.
[1] http://agilemanifesto.org
If the bad managers are outliers, then fine. If the bad managers are the norm, then we need a way of working that fits the average manager, even if you call them "bad". Most people in most companies are necessarily going to be of average quality.
Judging from years of talking to devs and reading books and articles and comments on the net, what you call "bad managers" are not exceptions, but rather the norm.
> Agile done well is absolutely amazing
If your team is made up of very competent people, any methodology is likely to work great. But most teams are not. We need something that works reasonably well even when the managers and devs are just average.
Can you explain to me your thinking - which I might be implying and you might not actually think - as to how heavy processes like SCRUM and Kanban turn average teams into productive ones?
No. Not this please no. Where is that context where your team is constantly learning and discovering. ?
Please tell me what Software is that? A real time control system for a high speed train, airplane or space ship, where lives are at stake? A hospital or ER patient management system where a library bug can mean the patient get's a fatal dose of 250 ml of something instead of the life saving 2,5 ml?
Or a context where everyone is constantly "learning and discovering" the best color for the web page of your online shopping?
I prefer Software for stuff that matters....
Most software is used by customers with changing needs and competitive pulls. I've built software in the last 2 years that we would have taken a radically different approach to if LLMs were available, because they're great at summarising natural language. But they weren't. Imagine if we'd been building something for 2 years and never learned anything about changing technology or customer expectations - instead we shipped something fast, and then looked back after we learned about new things and made it better. That's how the World should work, and why software is not like building a bridge or a house or basically anything else in the history of engineering.
As to your specific exampleS: I have some detailed knowledge of hospital EHR systems, and they are so massively, hilariously broken in ways that modern software practices could easily help address. I suggest reading Accelerate [1] for data and examples of how agile applied to DevOps practice in particular means you can be more responsive and produce safer software, especially in that context.
I, like you, also prefer software for stuff that matters. I've been trained in formal methods, I've done embedded systems development, I've worked in medical/healthcare contexts. Your statement that what I've written above does not apply to those environments (including the use of formal verification methods), doesn't make sense to me.
[1] https://www.amazon.com/Accelerate-Software-Performing-Techno...
Let's review the Agile Manifest from the perspective of delivering Software That Matters, that I define as Software where failures can cause significant loss of Lives, Money, or Reputation:
1. Early and Continuous Delivery
Speed over quality in critical software contexts invites catastrophic failures.
Instead just implement phased delivery schedules that prioritize comprehensive testing and stability over speed, ensuring system reliability.
2. Welcoming Late Changes
This flexibility often leads to scope creep and consequential delays, undermining system integrity.
Instead have a change control board to rigorously evaluate the implications of proposed changes, safeguarding project stability.
3. Frequent Software Releases
Regular release demands degrade software quality and exhaust development teams, detracting from critical system safety.
Instead base your release schedules on thorough testing and feature completion, ensuring each release enhances system robustness.
4. Daily Business-Developer Interaction
Frequent meetings disrupt productivity and can force development compromises, jeopardizing precise project requirements.
Instead opt for structured weekly updates and essential communication to maintain clarity and focus without sacrificing project integrity.
5. Trusting Motivated Individuals
Sole reliance on individual motivation without structured oversight introduces risks of inconsistent quality in high-stakes projects.
Instead clearly define roles and accountability metrics, aligning individual contributions directly with critical project outcomes.
6. Face-to-Face Communication
This traditional communication mode fails to accommodate remote participants or ensure accurate documentation of critical decisions.
Instead leverage advanced digital communication tools that enhance documentation, suitable for modern, distributed teams.
7. Working Software as Progress Measure
Focusing solely on functional completions neglects essential aspects like security, scalability, and user experience.
Instead expand progress indicators to include comprehensive evaluations of non-functional requirements, providing a holistic view of system health.
8. Sustainable Development Pace
The ideal of an unvarying pace is impractical under the variable pressures of high-stakes software development.
Instead adopt development rhythms to real-world project dynamics, accommodating natural workflow variations to prevent burnout and ensure quality.
9. Continuous Technical Excellence
An unrelenting pursuit of technical perfection can obstruct pragmatic and timely deliverables in critical scenarios.
Instead aim for pragmatically viable solutions that meet current needs with provisions for iterative enhancements as future requirements evolve.
10. Maximizing Work Not Done
Striving for minimalism can neglect essential complexities, endangering the whole system.
Instead prioritize a thorough approach that covers all necessary bases, balancing simplicity with comprehensive risk management.
11. Self-Organizing Teams
Teams without structured guidance can diverge from critical safety and performance standards.
Instead implement a framework that offers direction and support, ensuring teams adhere to best practices essential for system reliability.
12. Regular Reflection and Adjustment
Constant reassessment cycles can disrupt ongoing projects and dilute focus on strategic goals.
Instead space out reflective practices to better assess their impact and refine processes without compromising ongoing project stability.
Every study of continuous delivery suggests it improves quality by finding bugs earlier (because it has continuous integration - automated and rigorous testing to help identify unintended regressions - as a dependency), and reducing MTTR when undetected bugs do occur, because releasing fixes is quick and standardised. I'll refer you to the book in my comment above which demonstrates this with data collected from thousands of companies.
If you don't understand the reasons behind this, you're not going to understand why any of the other things lead to safer software, released earlier, and producing more value in the short, medium and long terms. So we should stop here.
I'd strongly encourage you to start by reading Accelerate and then come back and take another swing at it if you want, but given your current perspective - which sounds highly authoritarian, cynical, and divorced from data collected over a wide range of industry contexts - there's no point in us discussing this further.
The key distinction between waterfall and agile (or even Agile for that matter) is: do you change your product when your customer changes his mind?
If you agree to build x, then your customer wants y, what do you do at the margin?
Building stuff your client no longer wants is guaranteed not to matter.
But for most software, the customer keeps changing his mind and requesting new or different things. This includes hospitals. What's the impact of a bug have to do with it? I don't see the logic in "bugs are very bad -> this proves requirements don't change". Non sequitur.
If you have not experienced this, you're very lucky. But look around, in this thread and elswhere, and see how many people are claiming the "adapt to constant change" claim of Agile is pointless. You'll find very few.
Not that Agile is great. But this particular problem exists, and needs a solution. Software where requirements don't change is rare. It just is.
There may be some outliers where its not... but this is the cult like bullshit i'm seeing everywhere.
agile is the Ivermectin of business process. except it makes me WANT to inject bleach.
See https://en.wikipedia.org/wiki/No_true_Scotsman
The meme that any critique of agile is not "true agile" should be first.
Want to stay creative as a programmer, be a lone wolf... Do what you like and whenever you like, follow the creative spark...
Value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
Principles:
1. Customer satisfaction by early and continuous delivery of valuable software.
2. Welcome changing requirements, even in late development.
3. Deliver working software frequently (weeks rather than months).
4. Close, daily cooperation between business people and developers.
5. Projects are built around motivated individuals, who should be trusted.
6. Face-to-face conversation is the best form of communication (co-location).
7. Working software is the primary measure of progress.
8. Sustainable development, able to maintain a constant pace.
9. Continuous attention to technical excellence and good design.
10. Simplicity—the art of maximizing the amount of work not done—is essential.
11. Best architectures, requirements, and designs emerge from self-organizing teams.
12. Regularly, the team reflects on how to become more effective, and adjusts accordingly.
https://en.m.wikipedia.org/wiki/Agile_software_development
For most of the critics, I would say "become more Agile" as a solution.