The thing this (very good) post doesn't mention is that big companies select for blub languages because that's where the most low-cost labor is, in that you can hire multiple Java developers for the cost of one Haskell developer, even if Haskell might be objectively a better choice for the project.
One nitpicky detail is that the executives may be a rep for the customer/consumer, but are also very much reps for the shareholders and that’s a pretty big distinction
I liked this post. I may have some minor qualms (e.g. while I think execs should be proxies for the customer, they have many other competing motivations that can push customer needs way down), but I especially liked the closing section:
> Understanding before criticizing
> Large software companies have real problems. Some are structural. Some are cultural. Many are self-inflicted. But many of the behaviors people complain about are not pathologies – they are consequences.
> If you want to criticize how large organizations operate, it helps to first understand why they operate that way. Without that understanding, the criticism may feel sharp, but it will not be useful.
I see that kind of "criticizing before understanding" all the time on HN, and while that's probably just inherent to an open forum, commenters who do that should realize it makes them come across as "less than insightful", to put it generously. Like I see tons of comments often about how managers only get to their position through obsequious political plots. And sure, that may exist in some orgs. But you can always tell when folks have never even considered the competing forces that act on managers (i.e not just the folks they directly manage, but requirements coming from higher ups, and peer teams, and somehow being responsible for a ton when you actually have few direct levers to push) and solely view things through the lens of someone being managed.
Equating process to risk aversion is a mistake I hadn't seen before. It's true that moving slow makes it less likely to make something worse per unit time, but it also makes it less likely that anything will get better. It means that you don't try many things, and can't get important things done quickly.
You can ensure quality by making features opt-in, having a beta program available to risk tolerant users, adding QA resources, having representative users in captivity (employed at the company).
There is no law that says that you must move slow or do less in order to be low risk, you can also do a lot, move fast, and only let the best out.
I prefer Parkinson's take on the matter. Parkinson's law has a lot more to do with why companies grow so large. Why WhatsApp can can't write meaningful software with fewer than 10 people
Coordination is almost free in a ten-person startup. It is still relatively easy in a forty-person company.
I find coordination difficult even for two / three persons for any given topic where there the tree of dependencies (of sub tasks or others topics related to it) isn’t trivial and there are unknowns to research. Unless those persons are doing the same thing and are constantly communicating, which is very expensive
Someone who says there are too many meetings is probably actually saying they are having bad meetings. If they got value from those meetings, they wouldn't be complaining. So there is likely still a problem to address.
Also, as a somewhat trivial side note, an instinctive reaction to not getting the clarity you need from a meeting is to ask for another meeting. So even if the optimal level of meetings is annoyingly high, bad meetings will probably push the level of (bad) meetings even higher. So you'll still actually have "too many" meetings.
> “There are too many meetings” At very large software companies, programming ability, technical expertise, and raw resources are not the limiting factors. Coordination is.
In my opinion there exist much more efficient ways for coordination: for example, write down some really good documentation and explanations that are then read by the other stakeholders, so that these, at the end, also have a very deep knowledge about the topic.
Nearly all employees have studied at a university, so the people are very used to writing texts (papers, seminar papers, lecture notes, thesis, ...).
In my experience the reason for too many meetings is rather that many managers love meetings.
--
> “There is too much process and bureaucracy” [...] At a very large software company, the software matters. It may be relied on by millions of people. It may underpin businesses, infrastructure, or daily life. It may not be particularly glamorous software but it has to work. It has to keep working. Failure is not charming, and recovery is not always cheap. [...] Process exists to manage risk, correctness, and scale. Calling it “too much process” without acknowledging the stakes involved is like criticizing a bridge for having too many safety checks because you once built a treehouse with a hammer and some nails.
This is one reason. Other common reasons for so much process and bureaucracy are
- Many managers love processes, because they can "hide" their failures behind processes, and introducing new processes and bureaucracy lets the manager pretend that he is doing something to solve the problems that plague the department.
- Many processes and bureaucracy are simply demanded by the legislature when you work in some heavily regulated industry. These legal demands often don't make sense.
As a technology company scales up, making great software becomes one of a hundred things the company needs to do right in order to survive and grow. Doesn’t mean it isn’t absolutely essential, but so is having a strong GTM machine, finance competency, operational rigor, HR, and a long list of other essential functions.
It’s only the tech industry where the voice and ego of small companies hold outsized share of voice and love to claim the contrary.
Many of the author's rebuttals hinge on the assumption that everyone in an organisation is acting in its interest first - and not their own, often conflicting, self-interest. As such, they are not particularly convincing.
Large organisations absolutely do, as a function of their scale, produce pockets where slackers and incompetents can hide. They'll surround themselves with a web of process, pointless meetings, and substance-free buzzword-heavy documentation/presentations to disguise this fact. Others may become ensnared in this web, and will rightly express the criticisms that the author is attempting to debunk.
This important devil's advocate perspective reminds me of Chesterton's Fence [0].
I used to run a dev shop and had the opportunity to work with companies of all shapes and sizes. The startups often discovered Chesterton's Fence by declaring they didn't need this or that (meetings, accountability measures, etc), only to learn the hard way why they existed.
And meetings, beauracracy, et al are rightfully criticized for being inefficient and fostering mediocrity. But I think I'd agree with the author that it's glib to say meetings are dumb, no need for hierarchy without understanding their purpose
I've spent 20+ years in the industry seeking to understand first, and my conclusions have pretty consistently been that the systems are broken and inefficient as people already talk about. The caveat I'd add is that the critics themselves would also be just as much as a crapshoot.
Honestly it sounds to me like the author doesn't truly understand the inherent conflicts of interest at a large company. For example a really common anti-pattern is "Nobody knows X thing is a problem my team manages in a problem (e.g. our app eats battery usage), but if I draw attention to it they'll want to measure it forever. So do not make it a focus."
In short pretty much never does any employees/manager's/executives interest align with the company.
> At very large software companies, programming ability, technical expertise, and raw resources are not the limiting factors. Coordination is.
If coordination is your limiting factor I'd argue that it shouldn't be, and you're not investing enough in removing it as a factor. Companies can use various tools to do this, for example:
* Defining directly responsible individuals / single-threaded leaders so that every choice doesn't involve massive coordination
* Putting people who work together in the office sitting next to each other most days of the week
* Or, for remote work, having a strong culture of async communication that is visible to the broader group by default, for example with Slack
I don't get it. Article titled "Common misunderstanding". OK. “There are too many meetings”. How is this a misunderstanding? "...bunch of excuses confirming there are too many meetings...". Yyy... so it's not a misunderstanding?
The author talks about how software creation processes at large organizations are an artifact of how large organizations operate, but in all 3 of the 3 cases, I would ask “why does the large organization need to be that way?”
Most obviously, why do executives need to be the proxy to customers? Why can’t development teams simply talk to real customers? This isn’t just an abstract idea in agile, it grew out of actual Japanese product development practices practiced at large organizations: Toyota, Canon, and others, and documented in “The New, New Product Development Game” HBR review article that was so influential to early agile.
The point that in large organizations, most of the work is coordination, again demands the question, why? It’s been understood since at least World War I by some military planners (with organizations far larger than Google) that coordinating dependencies was far more complex than reducing or minimizing them. Goldratt wrote about it when designing a project management system for Theory of Contraints (indeed, you could argue this is a fundamental learning of ToC). And one of my favorite software conference talks of all time is Mary Poppendeck’s excellent “Tyranny of the Plan,” where she notes that as computer systems have been used in planning, we seem to have become more confident but no more competent in coordinating, rather than focusing on flow, in large-scale projects.
Finally, on the importance of software created at large organizations, I agree, something that will have millions of users on day one has a greater responsibility, but that doesn’t mean that loads of bureaucracy and checking are the pathway to quality software. First of all, does anyone believe that highly scrutinized and bureaucratic functions are general high quality services? The often provide access to even the most extreme edge cases, but they do so by reducing the quality of service to everyone else. Anyone who’s ever filled out their own tax forms in the United States knows that it covers every base of possibly income, but 80% of people really only need to be concerned with 2-3 common forms, and 99% could simply be asked about 10 forms or so. Instead we have to answer questions for “directors of foreign corporations who also happen to be Us citizens,” instead of just requiring those people to fill out an additional form. And, of course, to (probably badly mis-)quote Deming, “you can’t check quality into a product.”
I would turn it around in the author: yes, the software practices operate this way in large orgs because large orgs are structured differently—but why do large orgs need to be structured that way? Is it inherent when absentee owners with low domain context (shareholders) pass ownership over to a manager? Is it because hierarchies insulate good but not great managers from genuine value creation as long as they play politics well? Is it because these are first order ways to understand complexity, and again, low-context absentee owners aren’t going to do the work to understand the more complex dynamics at play?
> The point that in large organizations, most of the work is coordination, again demands the question, why?
You could say something similar about large-scale distributed software systems. A lot of it is just the nature of distributing and managing work across a collection of "nodes" or, as the case may be, people.
21 comments
[ 4.0 ms ] story [ 50.4 ms ] threadhttps://en.wikipedia.org/wiki/Apologia
The thing this (very good) post doesn't mention is that big companies select for blub languages because that's where the most low-cost labor is, in that you can hire multiple Java developers for the cost of one Haskell developer, even if Haskell might be objectively a better choice for the project.
> Understanding before criticizing
> Large software companies have real problems. Some are structural. Some are cultural. Many are self-inflicted. But many of the behaviors people complain about are not pathologies – they are consequences.
> If you want to criticize how large organizations operate, it helps to first understand why they operate that way. Without that understanding, the criticism may feel sharp, but it will not be useful.
I see that kind of "criticizing before understanding" all the time on HN, and while that's probably just inherent to an open forum, commenters who do that should realize it makes them come across as "less than insightful", to put it generously. Like I see tons of comments often about how managers only get to their position through obsequious political plots. And sure, that may exist in some orgs. But you can always tell when folks have never even considered the competing forces that act on managers (i.e not just the folks they directly manage, but requirements coming from higher ups, and peer teams, and somehow being responsible for a ton when you actually have few direct levers to push) and solely view things through the lens of someone being managed.
You can ensure quality by making features opt-in, having a beta program available to risk tolerant users, adding QA resources, having representative users in captivity (employed at the company).
There is no law that says that you must move slow or do less in order to be low risk, you can also do a lot, move fast, and only let the best out.
Also, as a somewhat trivial side note, an instinctive reaction to not getting the clarity you need from a meeting is to ask for another meeting. So even if the optimal level of meetings is annoyingly high, bad meetings will probably push the level of (bad) meetings even higher. So you'll still actually have "too many" meetings.
In my opinion there exist much more efficient ways for coordination: for example, write down some really good documentation and explanations that are then read by the other stakeholders, so that these, at the end, also have a very deep knowledge about the topic.
Nearly all employees have studied at a university, so the people are very used to writing texts (papers, seminar papers, lecture notes, thesis, ...).
In my experience the reason for too many meetings is rather that many managers love meetings.
--
> “There is too much process and bureaucracy” [...] At a very large software company, the software matters. It may be relied on by millions of people. It may underpin businesses, infrastructure, or daily life. It may not be particularly glamorous software but it has to work. It has to keep working. Failure is not charming, and recovery is not always cheap. [...] Process exists to manage risk, correctness, and scale. Calling it “too much process” without acknowledging the stakes involved is like criticizing a bridge for having too many safety checks because you once built a treehouse with a hammer and some nails.
This is one reason. Other common reasons for so much process and bureaucracy are
- Many managers love processes, because they can "hide" their failures behind processes, and introducing new processes and bureaucracy lets the manager pretend that he is doing something to solve the problems that plague the department.
- Many processes and bureaucracy are simply demanded by the legislature when you work in some heavily regulated industry. These legal demands often don't make sense.
It’s only the tech industry where the voice and ego of small companies hold outsized share of voice and love to claim the contrary.
Large organisations absolutely do, as a function of their scale, produce pockets where slackers and incompetents can hide. They'll surround themselves with a web of process, pointless meetings, and substance-free buzzword-heavy documentation/presentations to disguise this fact. Others may become ensnared in this web, and will rightly express the criticisms that the author is attempting to debunk.
I used to run a dev shop and had the opportunity to work with companies of all shapes and sizes. The startups often discovered Chesterton's Fence by declaring they didn't need this or that (meetings, accountability measures, etc), only to learn the hard way why they existed.
And meetings, beauracracy, et al are rightfully criticized for being inefficient and fostering mediocrity. But I think I'd agree with the author that it's glib to say meetings are dumb, no need for hierarchy without understanding their purpose
[0] https://en.wikipedia.org/wiki/G._K._Chesterton#Chesterton's_...
Honestly it sounds to me like the author doesn't truly understand the inherent conflicts of interest at a large company. For example a really common anti-pattern is "Nobody knows X thing is a problem my team manages in a problem (e.g. our app eats battery usage), but if I draw attention to it they'll want to measure it forever. So do not make it a focus."
In short pretty much never does any employees/manager's/executives interest align with the company.
If coordination is your limiting factor I'd argue that it shouldn't be, and you're not investing enough in removing it as a factor. Companies can use various tools to do this, for example:
* Defining directly responsible individuals / single-threaded leaders so that every choice doesn't involve massive coordination
* Putting people who work together in the office sitting next to each other most days of the week
* Or, for remote work, having a strong culture of async communication that is visible to the broader group by default, for example with Slack
Most obviously, why do executives need to be the proxy to customers? Why can’t development teams simply talk to real customers? This isn’t just an abstract idea in agile, it grew out of actual Japanese product development practices practiced at large organizations: Toyota, Canon, and others, and documented in “The New, New Product Development Game” HBR review article that was so influential to early agile.
The point that in large organizations, most of the work is coordination, again demands the question, why? It’s been understood since at least World War I by some military planners (with organizations far larger than Google) that coordinating dependencies was far more complex than reducing or minimizing them. Goldratt wrote about it when designing a project management system for Theory of Contraints (indeed, you could argue this is a fundamental learning of ToC). And one of my favorite software conference talks of all time is Mary Poppendeck’s excellent “Tyranny of the Plan,” where she notes that as computer systems have been used in planning, we seem to have become more confident but no more competent in coordinating, rather than focusing on flow, in large-scale projects.
Finally, on the importance of software created at large organizations, I agree, something that will have millions of users on day one has a greater responsibility, but that doesn’t mean that loads of bureaucracy and checking are the pathway to quality software. First of all, does anyone believe that highly scrutinized and bureaucratic functions are general high quality services? The often provide access to even the most extreme edge cases, but they do so by reducing the quality of service to everyone else. Anyone who’s ever filled out their own tax forms in the United States knows that it covers every base of possibly income, but 80% of people really only need to be concerned with 2-3 common forms, and 99% could simply be asked about 10 forms or so. Instead we have to answer questions for “directors of foreign corporations who also happen to be Us citizens,” instead of just requiring those people to fill out an additional form. And, of course, to (probably badly mis-)quote Deming, “you can’t check quality into a product.”
I would turn it around in the author: yes, the software practices operate this way in large orgs because large orgs are structured differently—but why do large orgs need to be structured that way? Is it inherent when absentee owners with low domain context (shareholders) pass ownership over to a manager? Is it because hierarchies insulate good but not great managers from genuine value creation as long as they play politics well? Is it because these are first order ways to understand complexity, and again, low-context absentee owners aren’t going to do the work to understand the more complex dynamics at play?
You could say something similar about large-scale distributed software systems. A lot of it is just the nature of distributing and managing work across a collection of "nodes" or, as the case may be, people.
When execs dominate decisions, it’s usually because the org failed to make customer truth unavoidable.
Most process exists to protect careers, not outcomes.
Understanding why something exists doesn’t make it justified or effective.