How do big companies stay alive for so long? I guess it's still possible to coast on a single success for so long that the company has time to completely rot from the inside before any of it reaches their customers. Or maybe the early money gets invested in a bajillion projects, on the very slim chance that the company isn't a one-trick pony.
What sort of regulations are preventing a google competitor from starting up? "GDPR/CCPA" isn't very convincing because it was only a recent thing, but google has been around for decades.
Regulatory capture covers more than just regulations. It also covers the lack of regulation; including anti-trust.
Do you want to do a search or mobile OS startup with Google’s market position? Do you think they’ll release a YouTube or Google Maps application for your phone OS?
Because they're not actually much more dysfunctional than smaller companies, you just hear more stories because they're bigger. Bigger means more employees for things to go wrong around, bigger means that people are more likely to be interested in hearing stories and thus other people are more likely to tell, remember, and repeat said stories.
A story about some idiot doing an idiotic thing at Google is vastly more likely to be passed around forever than a story at a random mom and pop company*. People love hearing about morons at big tech giants, and the storytellers oblige.
That, plus sufficient economies of scale or complementary tech products.
* unless that story is REALLY incredible, like that guy who went to an interview and accidentally killed the CEO's dog
The bigness itself causes dysfunction, apart from just having more employees and more chances to have a disaster. It's the whole Dunbar's number thing, we really haven't figured out a great way to work effectively in huge groups yet.
But... some things can only be done with huge groups, so even working ineffectively can be worth it. And in a competitive setting, your competition also has these problems, so you can win in the market despite them.
The problem with tiny startups is often that people are really inexperienced with people problems. Large corps somewhat fix that by having HR departments and all. But ultimately I'm starting to think today's business often relies on some sort of hype train.
HR is often awful, sure. But it helps get the basics right.
My brother works at a small (<100 people) startup in the material sciences, and the amount of times that his and his co-workers pay statements got messed up in creative ways is incredible. Meanwhile I'm working at a decades old big company and never had any problems like that.
There are plenty of businesses that don't rely on hype, you just won't find them on HN. The average supermarket chain, steel mill or insurance company just produce valuable goods or services that people and/or businesses want to buy, and they can be stable businesses for decades or even centuries. Hacker news is a forum hosted by a VC that predominantly invests in software startups however, and so you won't find a lot of stories here about companies that don't fit that mold.
In my experience bigger companies are more dysfunctional, but they appear less so the higher up in the company you are.
When you're just some low-level worker for a mega corp high-level decisions tend to get passed down with little to no context through several layers of management half of whom are disconnected from the issues you face on the ground, and the other half are disconnected from the decision making at the top.
This does cause some dysfunction, but I agree with you that it's not often not as bad as it seems when you have context of the why decisions are being made.
Not all companies do such a bad job at communication either and flatter organisation structures can help empower the people on the ground to make decisions which might otherwise need the approval of several layers of management. But still, in a large company there's inevitably always going to be some disconnect which will cause some level of dysfunction that you wouldn't get in a small company.
There are many companies that are so wealthy that they could just stop all revenue completely and coast no for decades while management continues to skim out their fat paychecks. Some theorize that this has already occurred.
Besides, -- if you think how some companies work is fugly, go take a look at biology or politics. Companies are evolved systems. Once they're old enough they're almost always messy even when they aren't especially dysfunctional.
The human body is full of almost vestigial parts and dead end metabolic pathways... huge chunks of our DNA are just self replicating repeating sequences.
In reality the cruft in companies is often not even non-functional, but sometimes it serves some more complex purpose. Lots of large companies, for example, have some useless figurehead security position that exists for the actual purpose of having someone to fire when there the inevitable serious security issue occurs, without putting anyone actually productive at risk.
Companies are built out of and owned by people. Like anything else they have to respect the the limitations of the building material and the needs of their owners. The resulting requirements are no less real even though they might not exist in a world populated by homoeconomicus instead of man kind. If man kind wants a blood sacrifice for security incidents then it's reasonable and good for a company to have one.
Sure, and for a period corporate raiders took advantage of companies like that-- but standard defenses against raiding have become well known and widely deployed. Like life learning how to make cellulose suddenly the landscapes were littered with the husks of things that nothing knew how to digest.
But the value to society at large isn't just a profit (much less an ever growing profit). It's good that we have businesses which simply satisfy the involved parties (their customers, their employees, and their owners)-- even if they don't perform well on any external metric of efficiency.
Investors certainly look at winddown scenarios for companies not performing. Sometimes broader considerations win out, sometimes a change in management is all that is needed, sometimes more aggressive things happen.
The employees are still getting paid! Quite often the product is genuinely useful. It may be a beloved brand. Companies don't vanish the microsecond they cease to be profitable and nor should they. It's only when they run out of cash.
I don't think it's anyone's job to be "euthanizing" low-profit businesses that might recover if there is a turn in the economy. The "zombie" concept is economically illiterate. Occasionally you will get a private equity firm doing this and the result is widely unpopular (Toys R Us, Maplin, etc)
The employees and customers are surely a part of society.
More offent than not, not making enough profit is overreaching the profit goals to justify unfair stock prices. And the executives and board make bad gambles since they lose the owners money if they lose and win the owners money of they win.
This is an incredible comment. The part that drives people to absolute frustration is that you might be in some weird vestigial organ, either with no purpose or with a purpose not at all aligned with what the organization is saying, and then waste years trying to "fix it". It's a bursting appendix, don't worry about it - or well, maybe this is a bit stretched now, remove it.
You might enjoy reading the external version [1] of View from Above [2] on apenwarr's blog. The internal version is much better if you can get a copy of that as it goes into his actual experience with trying to improve Google.
Although IIUC you both read High Output Management so it might just be nice to know that even within fang people think things are wrong.
Well, if you find yourself as an individual contributor in some weird vestigial organ, I feel like you have a few choices:
* Stay put to collect the money, with the morale cost of feeling like you don't have a purpose and the risk of getting axed if/when the bursting appendix gets removed
* Proactively seek a lateral transfer to a position doing something more meaningful
* Get out. Find a different employer.
The latter two are likely better, but have their own risks.
One overlooked reason is the government intervening to save these companies. Either by bailout, lowering rate, special loans, etc… This prevents these companies from going into bankruptcy and restructuring and keeps the status quo as is.
It’s kind of a disguised communism and the longer the government keeps proping the economy (and making it difficult to start new companies), the worse this will get.
A lot of big companies don’t. Additionally, a single success often implies multiple times smaller successes that are just not visible outside a corporation.
Sometimes being big is a structural advantage. Also there's a variant of the Peter Principle at work: any effective company grows until it is ineffective, therefore productive work is done only by companies that haven't finished growing yet.
At its core a company is a machine that makes money.
Once the machine is up and running, you just have to avoid not breaking it. An org at this stage can be completely dysfunctional as long as the machine keeps running. Also figuring out the "wins" while scaling are much more obvious than while trying to start.
That's true if the environment where the machine work doesn't change. However with business environment changing the machine need to change as well.
Big corporations have "moats" (as per Buffet's definition) and a big bank vendor locked into your solution will drag a badly working machine for years.
Author here. My organization continues to exist in its current massive form due to regulatory capture - I can't provide details and remain anonymous, unfortunately. We would probably survive at 1/10th the size or something akin to that without that protection, and we would be considerably more efficient.
EDIT: Oh, and the part that earns money through regulatory capture earns so much money that we can have all these departments doing nothing in the meantime. We might do something one day with amazing leadership, but the incentives and odds are stacked against us.
Big companies tend to have a handful of divisions that generate almost all of the revenue, and then everyone else is a cost centre. As long as the revenue generators outweigh the cost centres, no one really cares broadly about efficiency.
For Facebook the revenue generator is Ads. For Amazon it was initially physical shipping, and later cloud hosting. Outside of those divisions you could burn billions on long shot projects that went nowhere, and nobody really cared...
> There's a team out there where the main drama isn't one right faction and one wrong faction, there are two factions fighting to the death over how best to use a spreadsheet as a database for a production application.
This is the authors ideal of hell? oh you sweet summer child.
Wow, I made it to the compliments page! I feel like it's cheating a bit that I knew about the page before writing a comment.
I don't really have anything important to say, but I did find this interesting. I'm unused to this sort of indirect response. It feels slightly like a personal response, since he links to a page where my comment is one of a few added, and refers to "commenters". However, my criticism was close to the opposite of what he is replying to.
I criticized the author for writing a story where I felt they mostly acted like a cog in a dysfunctional system, but was sneering at everyone else doing the same. I suppose, though, that perhaps they believe that the best/only solution for being in a poorly-functioning company is to jump ship.
edit: To clarify, I do agree that most corporate systems are dysfunctional and I'm not an agile fan. I'm not defending these systems.
Hello! Sorry, that whole page is a bit tongue-in-cheek, but there are two types of comment on there. One batch is from people that... well, you can see that they're like.
The other half I put up there because I think they actually have me dead to rights. It's one of those weird things where, being fairly young, I already know that I'm going to look back on many of the things I write and realize they were hot-headed, and I wasn't nearly as good as I thought I was. I didn't think the writing came off as smug - I'm pretty aware that I clicked five buttons and skipped a computer science education - but I still appreciate the reality check that maybe I -am- being obnoxious. It's just fun to be a hothead sometimes, and I'm actually starting to get nervous with all the HN traffic. Yelling into the void is less threatening to my career.
The only quibble I have with the original comment is the assumption I didn't try very hard to let people know it was an issue - I promise I did, sometimes management just... doesn't care.
Anyway, what I'm trying to say is, sorry for any unintended offense I might have caused, because you were totally fair. It feels wrong to take it off the page anyway though!
>There is a high likelihood that many other people in the org were lightyears ahead of the author. It's even more likely that an engineer or manager in their group was "Banking" the inefficiencies to use during a cost-cutting period - and the author ruined that chance... which will inevitably cause massive suffering and possibly poor performance reviews when there is nothing to trim."
That's crazy, how it isn't pure sabotage? career oriented people, jezz christ.
That's instant fire by me if somebody considered such a thing
Seems like a normal day in a corporation? Bean counters who decide your salary use beyond-stupid metrics. For instance, fighting fires is often rewarded disproportionately, so it’s good to be “visible” during many, recurring fires. Of course people game that, intentionally or unintentionally. You don’t get the behavior you say you want, you get what you incentivize.
Even at pure tech companies with somewhat technically competent leadership the incentives are whack, and often the people at the very top are still replicating some MBA doctrine that will be debunked as horse manure within a decade.
Just because our computers get exponentially better doesn’t mean people do, particularly groups of people.
I think there’s a spectrum here, I doubt many people are deliberately hoarding huge, easy, and /certain/ cost or resource savings just to use later, but it is easy to end up with a long list of possible things that you haven’t had time to try, and won’t get prioritised until somebody else cares.
Also, be careful what you wish for regarding cost savings. You may think you’re doing well saving 500k, but in a medium sized company there are likely a lot of easy ways to save a chunk more than that which you really aren’t going to like.
I have a book about the management consulting industry that suggest there is at least one large successful firm of management consultants where the answer is always "fire 23% - 33% of your staff" regardless of what the question was.
Apparently this is the level that most organisations can stagger on long enough for senior management to bank the bonuses and move on and upwards and before long term effects kick in.
Yeah that’s always a bit depressing, but most companies only really seem to get the consultants in so they can be told the thing they were going to do anyway.
I’ve seen good products and companies brought low by making the wrong bets and doing what was safe and sensible and cost reducing but not really thinking long term enough. They can stagger on for a long time still turning a reasonable profit until they suddenly don’t.
I suspect senior management who hire that consultancy know exactly what will happen but they need the cover to be able to say that they had "independent" management consultants come in and look at the business and give their "honest" recommendations.
Yeah, but I’ve seen plenty of very smart engineers over the years who can’t see why a business values X which they think is worthless but not Y which they think is important.
I was thinking of all the other ways businesses can “save money”, like travel bans, or getting rid of nice things, or closing offices, or getting rid of staff… They may not really save money, but they’ll look just as good on somebody’s spreadsheet.
"Practical men who believe themselves to be quite exempt from any intellectual influence, are usually the slaves of some defunct economist."
― John Maynard Keynes
Sometimes companies are incentivised to grow, sometimes to cut costs - investors reward different things at different times. Knowing areas that could be trimmed and not trimming can be entirely rational then.
It's is not irrational to follow the desires of the owners. And those desires need not be irrational either, as capacity for change and execution is limited in any corporation - focus matters.
Do you fire engineers who consistently finish projects ahead of their estimates?
Is any degree of sandbagging is a firing offense? If not, wheres the threshold?
Is your organization completely and unconditionally free of mandates like each department must "Cut budget x%" or "improve output by y%" -- irrespective of the incremental cuts and improvements that a group may have had in recent memory? If not, isn't a group that diligently finds and implements efficiencies at a disadvantage compared to ones that just don't look until a mandate comes in?
Kudos if you really do have an organization where there is no structural incentive to sandbag or otherwise bank results.
People who are consistently conservative in their estimates do get noticed, and the estimates adjusted appropriately in the same way that people who always under estimate get their estimates adjusted. That’s fine, the worst that’s going to happen is a conversation about learning from previous estimates.
People who sandbag to give themselves an easy life, or to try and empire build by getting more resources, definitely get noticed, and not in a good way.
> try and empire build by getting more resources, definitely get noticed, and not in a good way
This is actively incentivised in a lot of tech companies - maybe more so for managers than ICs, but the bigger-your-fiefdom the easier it is to promote phenomenon tends to be pretty widespread.
"Adjusting estimates" is just another way to force engineers to lie in their Jira tickets about how much time they spend on each ticket.
At one point my agile master was complaining to me about estimations, so I just focused on reporting the correct amount of hours to make the burn down chart be straight. Decoupling reported time from actual time spent.
If your colleague always estimates that a particular type of task will take a couple of days, and it always takes a couple of weeks, you wouldn't discuss it next time they estimate it is only going to take a couple of days?
Yes, humans are bad at estimation, which is why it's good to reflect on how well you do and to correct if you consistently over or underestimate.
On that level? The original comment, which it is extremely funny to see getting roasted like this, was being hyper-cynical. It probably has happened somewhere but that level of dysfunction is not widespread.
On smaller levels .. it is well-known in organisations that employ "stack ranking" that there is no incentive to get rid of poor performers immediately, and indeed an incentive to acquire one or two, so they can be sacrificed at stack ranking time.
The cost saving comes from the fact that Snowflake virtual warehouses by default time out after 10 minutes and this can be lowered, realistically to 60 seconds.
They must have been using very overprovisioned servers to run up $1 million per year in Snowflake spend for occasional 10 minute bursts of activity prior to the change.
I think this points more to poor and ungoverned use of Snowflake and Cloud than it does organisational dysfunction.
> I think this points more to poor and ungoverned use of Snowflake and Cloud than it does organisational dysfunction.
Those are the same thing.
(one of the massive, massive advantages and disadvantages of cloud in general is that it circumvents "purchasing" departments. One the one hand: you don't have to spend $10,000 of time trying to buy $1000 of computer. On the other: you can waste millions on the cloud and maybe nobody will even notice.)
It can also circumvent in-house infrastructure teams - who I used to find were a far bigger challenge than purchasing. I used to work somewhere where the main mission of the infrastructure team seemed to be to prevent anyone actually running anything on all of the lovely hardware they managed - which reached peak madness when I was quoted £75,000 for hosting a single static HTML page for internal only access.
> I think this points more to poor and ungoverned use of Snowflake and Cloud than it does organisational dysfunction.
This is a "blame the user" argument that I hear a lot. It's true that in many or most cases users being super attentive can mitigate cost issues. But it's also true that this turns out to be very difficult in practice for a variety of reasons like scattered responsibility, complexity of cloud costs, and vendor cost models that encourage spending.
For my own money this problem looks like a business opportunity for vendors that offer customers ways to avoid it. The high margin era of cloud services with 10x markup on compute (e.g., Snowflake) won't last forever.
It is kinda interesting, that these rants about corporate life that are filled with little substance but with lots of swearing are so popular here. Is there one interesting takeaway here? Big organizations tend to grow in weird ways, opaque to the folks on the ground. They sometimes make big mistakes and spend money where they shouldn’t. But because of their size, that doesn’t mean the end for them.
This is why it's a major red flag when your org starts doing things "because big company X does it." And they never stop to think whether they do it because it works or that it's a failure and they are getting away with it because they are big.
To be fair, my other stuff is full of swearing. He's got me there.
Although I started writing at 6PM on a random day after a stupid meeting, and actually tend to agree that there's worthier stuff to be popular - but people click on what they want.
I have a whole bunch of more boring stuff around, with much less swearing, and I think I'm more comfortable with the low level of attention that gets.
Well, I don't mind the more flippant writing style anyway ;) I liked your writing and it reminded me of the phoenix project, not in writing but in content. I'll be sure to check out your more boring stuff!
> Big organizations tend to grow in weird ways, opaque to the folks on the ground. They sometimes make big mistakes and spend money where they shouldn’t.
The article is good and at the end of the day, it's a... rant.
The core issue that substance the main point of the author is that we do not have a scientific and empirically validated method for management in technology (IT). This is expected since tech is a quite new industry that surged after the late 90's.
On top of it, we do not have established literature related to the economics of IT to calculate the opportunity costs, average acceptable waste, association between cloud costs x productivity and delivery value, and so on.
Most leaders and managers are using methods that can or cannot work, but it's a discovery phase now in how to establish a management method that covers inefficiencies, opportunity costs, and value.
Not sure tech is fundamentally so different. There is typically a lot commonality between management practices of high performance companies across industries. There is also no reason to assume certain microeconomics don't apply. Some benchmarks might be different or perhaps still patchy, but I don't think that is really holding things back.
Assuming you can't leave the place (no other prospects around, family to feed, etc) you simply check out mentally. You do an okay job (no point in overachieving), stop being personally invested in the success of projects, roll your eyes a lot, and find meaning somewhere other than work. And if things get too overwhelming you can always vent to your co-workers who are also stuck in your same situation.
Don't stress too much about improving things there, spend time with the colleagues you like, and make incremental (and sometimes major) changes to your life, I think. Not super insightful, but you'll note it's a good idea in every domain of life.
Being aware of and adjusting your sphere of concern vs sphere of influence (see e.g. Covey & 7 Habits) can help you survive in a way that at least feels more ethical than, say, quiet quitting.
Ultimately though I got fed up and stopped working for large organizations. Small companies have their own problems, but they tend to be less maddening to me personally, so I'm happier overall. YMMV!
> What The Goddamn Hell Is Going On In The Tech Industry?
Management and agile happened. Instead of engineering, people chase tickets to achieve mediocre goals set by clueless non technical people. They thought something as creative as software development can easily be converted into assembly line manufacturing. Well, these are the results.
I was actually writing something on this, so I'm super interested that you brought this up. I have some strong theories, but I must admit I don't have the work experience to be that confident in them.
Pretty much all companies i’ve been with an did “agile” ended up debating agile more than doing it. The outcome expected by managers was unequivocally to be able to allocate work and measure output similar to factories. Obviously it has always been a struggle to add anything of technical importance because there was no user story for it (in their limited thinking everything needs to benefit users directly, clearing tech debt is only a nice to have). As systems degrade over time they’d hire more people to maintain or increase velocity (factory output basically). Goes without saying that that doesnt work and you end up with the horror stories described here.
Agile engineering practices on the other hand are beneficial. But those arent really invented by agile. All existed well before agile.
Also every single large corporation i served suffered from endless meetings. It was almost always managers demanding such meetings. Somehow their belief is that engineers shouldnt spend time thinking about a problem - they should instead discuss it right away. A dubious mindset and telling of how said managers have no clue how thinking is to be made generally speaking. There needs to be communication between team members, but a good technical person is one that understands that and does it when needed. However since management and a factory mindset leads to hiring assembly line workers that lack communication skills that can sometimes be an issue - then the vicious cycle dictates that there need to be more meetings to cover communication gaps made by managers that hired assembly line workers that dont communicate well with peers, and so on.
Companies run by founders usually fare better because founders are either technical, domain experts and or care about inefficiencies - thus they rarely hire non technical managers.
Such companies often get acquired by larger companies that then turn them into factories and so the cycle begins. They still generate revenue and profits because generating profits doesnt always require a stellar product or sometimes they abuse their dominant position (google’s regressing, amazon’s a swamp of counterfeits, and both still make money).
After a long time in tech i’ve reached the conclusion that tech has been turned into factory work (including threats of automation replacing “workers”). Growth will indeed come from ai companies because … such companies are not yet managed and factorised to death (still run by tech founders, and so on). Thats why google’s struggling to get ai working while openai is smashing it. But those too will become bastardised.
> I have some strong theories, but I must admit I don't have the work experience to be that confident in them.
This has not stopped you from posting the dozen or so blog posts you have already written so far. Go ahead and post what you are thinking. Do not let lack of experience stop you :)
To be fair, if he had the experience to know for sure, there would be no motivation to write about it anymore. People only talk about what they are unsure of.
Companies being inefficient has been around for WAY longer than agile and/or modern management practices. Franz Kafka wrote his novels in the 1910s, after all. In software specifically, Fred Brooks already wrote about the problems with organizing groups of people in the 1970s.
I didn't write any reply to the original, but enjoyed reading it, and this one too - but as someone who's been on both sides of the IC/Senior Management divide, mixed with and worked with plenty of C-level types and has a pretty good understanding of how companies "really" work - none of this is shocking. It's not even surprising. It's just really, really hard to organise a large group of people towards any common purpose, let alone profitably, let alone efficiently, especially in something so relatively new and little-understood as tech.
You think it's hard trying to find good programmers? Try finding good managers.
Tech companies need tech leads not managers, or managers with a solid technical background. Otherwise it ends up in a mediocre show of power and power games such as those described by the author. Sitting on your bottom and holding meetings all day long and discussing processes for the sake of processes has no place in tech. Tech companies need to understand that.
You can't be an effective tech lead in most (large) companies without at least passable ability to manage all of your dev team, your counterparts in product, and your management chain in general.
As a result, a lot of tech leads are engineer->management conversions who just haven't made the leap yet.
> As a result, a lot of tech leads are engineer->management conversions who just haven't made the leap yet.
And thats how it should be - folks with a technical background doing the management of technical people and work. How can you even manage that which you dont understand? Ticking boxes and measuring output is not management, that can be delegated to a script and graph charts. To manage the delivery of a technical project you need to understand the technology behind it - in an equal amount to how well you understand communicating with business stakeholders and balancing business and tech needs. And no, taking a udemy course in programming python doesnt mean you have tech experience, equally taking a udemy course in agile doesnt make you a manager.
> How can you even manage that which you dont understand?
Unless you’re an expert at literally everything, there is something in your life that you have to manage that you have no expertise in.
The typical way these things are managed is through measuring outcomes rather than focusing on how the work is performed. Tools include setting goals and milestones, auditing, cost bidding, reputation, etc.
The best managers are the ones who do develop expertise in everything they have to manage. For example, the US Navy requires anyone commanding a nuclear-powered ship to be trained as nuclear engineers.
Even on a nuclear submarine, there is a ton of stuff that isn’t “the reactor and propulsion system”, and there are a lot of activities that must be managed, starting with extremely basic things like “Do we have sufficient food and water?”
Yes, and that's expected to be something that's fully understood by any naval commander. My point is that, contrary to the MBA mentality of expecting managers to be intellectual mediocrities and never expecting them to gain expertise in anything hard or technical, the Navy actually does require hard technical expertise from commanding officers. Nuclear reactors are a lot more complicated than being able to count how much food and fresh water you have. This nonsense about "measuring outcomes rather than focusing on how the work is performed" is cope for incompetent leadership.
> This nonsense about "measuring outcomes rather than focusing on how the work is performed" is cope for incompetent leadership.
So your assertion is that every leader who is not expert in literally every function performed in their organization is “incompetent”.
And measuring outcomes is “cope”.
So, say, if I need to consult a physician I’m incompetent because I should be an expert at managing my health. If I feel better after visiting a physician that’s “cope”.
This seems unnecessarily limiting. There are so many things I’m not an expert in yet I’m responsible for. I can’t just resign from having my house repaired because I’m “incompetent”.
> So, say, if I need to consult a physician I’m incompetent because I should be an expert at managing my health.
I’m talking about leadership here, not living your own life. Obviously you aren’t going to have the same expertise as your doctor. That’s why you shouldn’t be running a medical practice.
In reality, much of the health care system is managed by incompetent bean counting administrators rather than actual experts, and that’s a huge problem.
The other side of the equation is that, at least in my experience, the best doctors are the ones who encourage and enable me to become an expert in my own health. If a doctor isn’t able to explain things to my satisfaction, I’m reluctant to trust their judgment.
> I can’t just resign from having my house repaired because I’m “incompetent”.
Nor do you want to be the guy who takes his car in for an oil change and gets billed an extra $100 for “replacing the headlight fluid.” When it comes to house stuff, the professionals are going to be better equipped and able to do the work faster than you are, but the basic principles behind how your house works shouldn’t be total mysteries to you, and if you pretend that you never have to bother learning about that stuff, you’re going to get taken for a ride.
What I stated at the outset is that the best managers “develop expertise in everything they have to manage”. You’re obviously not going to start off with all that expertise, but that doesn’t mean there’s a substitute for developing it. The idea that you can come into a field you know nothing about with zero expertise and be an effective leader by “measuring outcomes” is a fantasy. If we want to measure outcomes, let’s measure the outcomes of that philosophy. How many manufacturing or tech companies that used to be run by engineers continued to be successful after they were taken over by non-technical MBA’s? How many astronauts have died because NASA management lacked the expertise to understand and appropriately respond to the objections of the engineers who reported to them? Are American universities more cost-effective now that they’re run by administrators than they were when they were still run by academics? Is medicine better when treatment decisions are made by physicians or by HMO bureaucrats and government regulators?
Very fair. The funny thing is that I'm pretty jaded, and people tell me that I've got the wrong read. Then I see comments like this where people say "This is so obviously true that it's crazy it bears mentioning".
I suspect that it actually is surprising for some people, but based on the emails I get, it's mostly cathartic because so many engineers spend time being told by management that they're imagining all of this. Because, as you've said, it's super hard to find good managers.
I think that people are looking at waste in a wrong way. It's relatively easy to point out inefficiencies in a working system. What's harder is organizing a working system. Energy and attention are finite on many levels, and this is true on the organizational level too. Let's pick an example of a computer game. Games are usually riddled with performance problems, in fact, I think that it's hard to find a well-optimized game. Yet many of these inefficient games provide good fun for a lot of people. As it turns out, engine efficiency is not a top priority for a game, there's much more to it, like the game design, level design, and the context of the game, which is marketing, timing the release, setting the price, and so on.
Good management and leadership is also hard to quantify because it's a bidirectional relationship. The best manager in the world is going to struggle with a direct report who doesn't want to play ball, but will do a fantastic job with those who nurture the relationship or are trying to find their feet in their career. A poor manager isn't going to succeed with either of those people.
I find the author's dry sense of humour quite amusing, and I feel for them because I've been there. Most of us have been there I expect. I would say that feeding this mindset is likely to lead to burnout in the long term.
A lot of these issues are as inevitable as they are utterly confounding and at some point in your career I think you need to find that sweet spot where you can do your best work without having to push back too hard against the organisation. If you can manage to find that peace of mind then you're golden, and if you're lucky you can stay in a positive mindset or at least delay the onset of Jaded Greybeard Syndrome.
Every talented junior and mid hits this wall at some point. This is where a new mindset and new skills have to be acquired: moving with the whole company in the same direction and improving it from the inside. Pointing out inefficiencies is important, but more important is how you do it and to who. It's easy to complain, it's hard to act accordingly.
But why should any of us bother picking up new skills and adjusting mindsets if the author has to fight for a raise after saving the company $500k?
Secondarily, why should the author continue to fix things like this when they don't benefit, but others responsible are also not facing any consequences?
If you do it for yourself, that's great, but you should withhold the benefits of that from the company if they aren't going to reward you for being better.
I don't expect any reward, praising or eternal glory. I'm paid for doing the job. At some level, and to get to that level, doing a contract minimum is not enough.
Your skills and mindset are going to help boost your career when it comes time to find a new position. It's not really for your employer's benefit, but your own, and the job is just a convenient way for you to learn some new things for yourself rather than doing it at home.
Loyalty to a company isn't rewarded today the way it used to be, so you just have to take that experience to a company with a better proposition at some point.
> You think it's hard trying to find good programmers? Try finding good managers.
There's an ambiguity in the word "good": it can mean skilled, but it can also mean ethical. Personal ambition and greed can turn any organization dysfunctional, even if everyone is highly skilled. And you can't really blame individuals within a company for wanting to make as much money as they can, because that's more or less the point of the company itself.
As far as management is concerned, smart ethical people who are inexperienced will eventually become better at their jobs through experience, though this may involve a lot of trial and error. But in a certain important sense, smart unethical people will never become "better" at their jobs.
By the way, I personally don't think it's hard to find good (skilled) programmers. Companies make hiring a lot harder than it needs to be. Despite my years of documented excellence and my success as a self-employed indie developer, I'm essentially unhireable by most companies, because I'm not good at, or indeed willing to devote myself to, jumping through the arbitrary flaming hoops they set up to keep people out.
Perhaps this is an ethics problem too. Everyone seems very concerned about candidates who lie about their résumés. But for some strange reason, companies are exclusively focused on trying to evaluate the skills of candidates rather than evaluating their ethics. Here's a thought on this subject: companies practically demand that job candidates lie to them about the reasons they left their previous job. The truth is that a lot of people quit because their previous manager was an idiot or an asshole, but nobody is allowed to say that in a job interview, otherwise you'd be immediately rejected as someone who has a "bad attitude" and is not a "team player". In other words, playing along, having a "good" attitude, means lying.
It's not just intelligence and ethics but a strong-enough personality and verbal skills--yes, I'm projecting here.
I can't overstate how difficult it is to say "no" in the moment, and do so in a way that doesn't make you a problem and target in senior management's eyes.
> jumping through the arbitrary flaming hoops they set up to keep people out.
Amen. Having HR set up some bullshit criteria and resume robot to presort your candidates, then subject the ones who fall through that sieve to stupid gotcha programming questions is an amazing way to develop the attitude that it’s “hard to find good programmers”
The only part of this process that I don't have a problem with is the technical interview/test.
I hear people complain about this type of stuff all the time but based on the tests and challenges I've received they're reasonable. I've never been asked by a potential employer to do anything I didn't consider trivial or close to trivial. I've done some quizzes with easy questions (though the recruiters were very impressed that I aced them, apparently most don't come close), I've done a technical interview where I implemented a simple vehicle registry based on stubs and tests, just basic C# programming (I didn't even know C# when I did it, I knew Java) with basic data structures. A friend of mine had one where they were asked to implement a very simple frontend that gets some data from SpaceXs graphql api and display it in a table.
All of these are completely reasonable expectations for a competent developer. I could easily complete all of them as a fresh graduate with no work experience.
I understand not everyone is this reasonable, what I'm saying is that I think these kinds of simple exercises are a reasonable way to eliminate candidates who actually don't know basic shit you would expect them to know. There are a lot of them, even with degrees.
It's pretty much just bias. I'm biased towards these tests because to me they're trivial so it benefits me that I get an easy way to stand out. I also wouldn't want to hire anyone who can't do simple basic programming tasks, so I sympathize with the employers in that regard. On the other hand you have people who are mad because the tests reveal their incompetence.
The problem is when the tests let in "too many people" for big companies, they end up ramping them up to be unnecessarily hard to compensate. Then small companies feel that they need to copy what successful big cos do, so they end up with insane leetcode problems and then complain that they can't get any good candidates because they're all failing at the arbitrary barrier specifically designed to cut out 99% of the applicant pool.
When I hear "technical test", I don't think "reasonable implementation", I think of the interview I was in where I was asked a moderately difficult graph question, I reasoned my way to the solution from scratch, then got cut from the process with the explicit feedback that I should have recognized Djikstra's algorithm and had it memorized rather than needing to derive the correct solution myself (!!!)
(that still blows my mind, apparently being able to understand the problem and derive the correct solution is worse than memorizing it)
> I reasoned my way to the solution from scratch, then got cut from the process with the explicit feedback that I should have recognized Djikstra's algorithm and had it memorized rather than needing to derive the correct solution myself
Your mistake was that you thought they wanted someone with creative problem solving skills…when what they really wanted was a parrot.
Yea, I've "failed" technical interviews because my solution was not the one the interviewer was looking for. Sometimes I wonder if the interviewers lack the skill to evaluate a solution on its own merits, and are just pattern matching with the solution they want. It's not a class exam, I'm not required to demonstrate I learned the material and can regurgitate it, I'm (supposedly) being hired for my ability to solve problems.
I question your assertion that companies are looking to select ethical employees. In fact, I would argue they are looking to select unethical employees who know the game well enough to play along for the appearance of ethics (see: expectation of lying in a job interview)
Given the choice between someone who will have ethics and someone who will make money unethically, companies will choose the latter every time. And the people who are willing to be unethical to get ahead are the ones who get rewarded internally once hired.
see also: painfully byzantine and demeaning hiring processes. I'm thinking "behavioral assessments", clumsy applicant tracking software from 2006, long waits between communication. Its not inefficient, its very efficient, in fact, at selecting for subervent mules who respond well to the stick and don't need many carrots to stay in place.
> I question your assertion that companies are looking to select ethical employees.
I didn't assert that. To the contrary, I said, "But for some strange reason, companies are exclusively focused on trying to evaluate the skills of candidates rather than evaluating their ethics."
> I'm essentially unhireable by most companies, because I'm not good at, or indeed willing to devote myself to, jumping through the arbitrary flaming hoops they set up to keep people out.
Can relate. I was also a very good manager, but hated the job, and no one was interested in hiring me as one, anyway, (lost at Buzzword Bingo, yanno).
These days, I do my own thing, writing free software for folks that can't afford people like me, and I'm really happy.
You are allowed to say your previous manager was bad, but you are expected to be able to say it in certain ways ("development opportunities", "technical direction", "team vs individual responsibilities", ...).
At some point you get old enough that you stop trying, and then you literally stop understanding. You can no longer imagine how people can be fooled by the smoke and mirrors coming from some candidates. And yet still they are fooled. And you realize that their version of the right person is so completely upside-down and sideways from the reality of who is actually good, you'd need to send them through a semester of 400-level "How to Recognize Good Engineers/Managers/Product" in order to even have a clear understanding and common terminology to start a discussion of what they should be looking for. And upon that realization, you understand why big corporations are inefficient, because this peter-principle archetype is allowed to have a say in hiring decisions. And there is nothing you can do. It's who you look like, not who you are.
"It is really, really hard to organize a large group of people towards any common purpose"
Well, one very healthy rule to organize people is that people are responsible for their mistakes, which has a long-term effect of having less mistakes instead of having them covered under the carpet and still biting.
It all comes down to defining what is and isn't a mistake. A mistake at the team level may be an organization win and vice versa. And the instant you change the incentives to fix this, you are upsetting someone else's apple cart.
I'd talk about a common sense definition of a mistake, but common sense is so undervalued nowadays, that I don't even know how to start conversation about it.
By the way, can you give a concrete example of a team mistake being org-level win?
I rather think that incompetence is always a loose.
Is that necessary? I know 'Agile' isn't cool anymore, but at the heart of the Agile Manifesto is a suggestion that every contributor on the team should share in the management role rather than having a dedicated manager. Granted, it also suggests you need motivated people.
The amount and kind of work that managers do cannot be distributed across a scrum team. It's literally impossible and attempts to do so would be abusive and would tank productivity.
First, you can't expect everyone on the team to have all context on everything. That would be an incredible waste of everyone's time. Second, even if everyone did, in most cases the individuals would disagree on priority -- someone needs to be the final arbiter.
In practice, few people will make the "best decisions". They will often rationalize what they want. A leader needs to be there to "check" those decisions.
Every contributor can share the product or delivery management role, sure, but it doesn't really follow that 1:1s, career discussions and compensation reviews are also handled by the team, and those are important functions served by a manager.
Career and compensation discussions would be had with what the 12 Principles refers to as the "business people". In a traditional organization structure the manager would act as a liaison between the business people and the development team, but it is suggested that the developers should talk to the business people directly (and daily!).
Which is ideal, but the manager or someone in an equivalent position is required to shield the team from undue interference. Empower the team to reach out for answers but make sure that doesn't create backchannels that can derail the project.
> to organise a large group of people towards any common purpose, let alone profitably, let alone efficiently, especially in something so relatively new and little-understood as tech
...I don't think those are good examples. The very sympathetic recent bio of Musk details a lot of dysfunction in those organizations, and they largely achieve anyway by leaning on The Mission, working people harder, driving burnout, and then discarding them.
That story of mission driven burnout happens a whole lot in human history. During the race to the moon in the 60s I recall reading about the epidemic of broken marriages and families around the space program due to overwork and burnout.
A troubling question I often ask is: is it possible to do big things in a healthy balanced way? Or is greatness intrinsically a “Moloch” that demands human sacrifice? (That imagery was inspired by the 20s film Metropolis that touches on this question a little.)
If the latter is true then I don’t think this bodes well for the future of our species. It means we either die in the fire of our own striving or we die quietly in mediocrity, but either way we die. There’s never a lasting positive outcome.
BTW Elon Musk himself looks an awful lot like a victim of his own burnout culture. I suppose you can give him credit for taking his own medicine.
That’s certainly a source, but consider that this vicious capital market is largely driven by the fact that we all want to maximize returns on our savings.
We are simultaneously pulling the buggy, riding in it, and whipping ourselves to go faster.
We don't have the social technology to accomplish goals that require significant incomes and outlays over multiple lifetimes; as soon as Musk and Shotwell are gone, SpaceX is likely to undergo faster mission drift. Therefore, if you have something you want accomplished, it has to be done fast enough to see it through, which drives urgency and fast pacing.
Managing people is easy if you are personable and have the ability to adapt to culture and the people that comprise it. (managing 5 people is different than managing 20 people, which is different than managing 20 people in a different target industry)
Being a good manager also means shielding groups from other groups and making yourself the filter so you aren't wasting productive peoples times in endless meetings that are generally for other groups to show they are working to the wider company. You also have to provide a non-technical analysis of the internal state of the product/teams/tech to people outside your team. Those are the two hard things, everything else should be figuring out how you can make your team operate at it's highest levels without burning people out and hating their job.
Just an observation but its seems like your points run counter to each other. If I'm reading your idea correctly managing / leading 5, 10, 100, 1000, etc people appears to be (4) things:
1. Shielding the group
2. Provide non-technical analysis of the group/state of product
3. Remove road blocks
4. Provide resources
The industry that you are operating in seems to be adapted to the (4) principals you laid out.
I felt like there was so much on-sides violence against developers in the comments.
Where-as the I Saved submission was was such a great post about an org rotting out, about how jumbled stuff gets.
So much of this chaos & madness is organizational far more than it is individual, imo. Having a broader picture and context is hard. Having specific insight for squads facing specific issues is hard. I agree so much, it's really really hard to maintain a large org, on so many levels (including technical levels).
The problem is that there is no sort of ownership, both literal and metaphysical. For hundreds, thousands of years, people worked hard in small shops and farms (leaving the issue of slavery aside to keep the point short), delivered good services to their neighbors with whom they went to church and pub with, and they ended up with getting something in return to pass down to their children.
These days? The utter majority of workers are just another instantly-replaceable gear in a gearbox of madness, working for people on the other side of the globe no less - the exception are the few tradespeople and small shopkeepers that survived industrialization and Walmart/Amazon, and startups up until 100-ish people where the employee stock grant issues tend to get stopped.
My "Crack Theory That Everyone Is Going To Hate": the tech industry as we know it arose out of the Great Financial Crisis and was a way to ensure that people with the "right" educational and professional pedigree were getting the spoils of the money-printing that was preventing the utter collapse of our economy. (Dotcom-era salaries vis a vis productivity were high, but not anything like today.) No one has to do anything because "job creators" don't care; tech workers are just a convenient receptacle to prevent a wage-price spiral in the broader economy. Here's some money, kid, go to town, don't start any fires. The accountability that does exist is largely for social, rather than business, reasons; "Don't make me look bad," instead of, "Make me money."
Bonus, re: layoffs: Now, we are on the verge of another massive deflationary crisis for which we don't have any solution once the ball gets rolling, so such workers are no longer useful. The coddling ends, cuts to the bone begin (and justifications like TFA start appearing). Pigs get slaughtered.
To add to the concept, how a tech company today makes money is very different than in the past. Profit doesnt matter anymore. It is all speculation against the dream of future profits once a tech company dominates a market. Money people will pay the kids because in faint hope of being on the ground floor of the next facebook or microsoft. But with entrenched companies now dominating tech, nobody will be the next microsoft. Investors are starting to realize those days are gone. Tech companies will be started and many will succeed, but now must actually compete with each other, often in courtrooms, to justify why they should be allowed into existing markets.
Look at nokia. A few decades ago anyone with a billion dollars could start making cellphones. Good luck with that today. You would need a hundred billion, and a few decades of IP lawyering, before being allowed to compete with the likes of apple/samsung. If your device has a touch screen and connects to the internet, you will suffer innumerable patent issues.
Uh, you might have not been paying attention to it but profit does matter now. If you look out the window you can see all the startups doing mass layoffs or going out of business.
Profit has always mattered. The only change in attitude was because of historically low rates, it was easier to get more funding. Cheap rates = cheap money, investors are more willing to part ways.
Yes, cheap money literally means “profitable next year” becomes “profitable _someday_”. To people who have only been in the tech industry for the last decade this feels like “profits don’t matter” because it’s never been “someday” in their entire careers, but now with interest rates back up “someday” is coming very fast at many companies (and has maybe already arrived at some).
> Profit doesnt matter anymore. It is all speculation against the dream of future profits once a tech company <fill stuff here>.
Oh, this time is soooo different to the dotcom crash, where it was speculation on companies, where profits didn't matter because it was the dream of future profits once a tech company <fill other stuff here>.
Also, Google etc clearly are profitable. Insanely profitable. The question is... how? I think people are right to question whether the majority of the employees are engaged in work which ensures this profitability. But that's not the same as saying there's no profit.
It's all built on an edifice of advertising, which feels like a bit of a sandcastle to me. But I am also of the .com generation that saw the whole beach wash out into the ocean once already, and never understood how the advertising thing actually ended up yielding profits, and was initially quite pessimistic about the Google IPO (though was happy to take money from them by working there for a decade).
I've seen the CPMs and the CPCs from the inside. I don't get it. I worked on ad servers and in realtime ad bidding. But I still don't understand why ads make money.
> But I still don't understand why ads make money.
total guess here, but instead of thinking of an ad as something that actually converts someone into a paying customer, maybe it's more about buying "mental real estate"? If nobody knows your brand, you have almost no sales. If you're at least in the neighborhood, then by pigeonhole principle, at least some customers will stop at your property. So maybe ads are a sort of dilemma that (with very few exceptions), your business must burn money on advertising just to stay in business?
Absolutely, I get this impressions-as-marketing thing. But this also seems remarkably vulnerable to recession economics.
Back when I worked in ads, the line was always that digital ads budget were still a fraction of what was being spent in traditional media, and so there was huge room to grow. I'm kinda guessing this is no longer the case.
I worked on the other end of this as a buyer of Google Ads for some large travel companies in the UK. For all the craziness and disfunction that I saw at those companies[1], fundamentally the adverts did end up selling holidays to customers, and that was why they spent the money with Google.
> A few decades ago anyone with a billion dollars could start making cellphones
Did you mixed-up million and billion?
A billion dollar is a gigantic pill of money. This feel a bit like saying "anyone with a British crown on his head can reign over the UK".
Also, looking at Fairphone, they operate with a fairly reasonable budget: all revenue, including phone sells in 2022 are around $60M, and if we exclude production, ~$17M are needed to pay for product development, marketing, support and all other expenses for a year. At 2 years to create a phone, that's ~$30M.
Because you're going to have to explain how it works.
>way to ensure that people with the "right" educational and professional pedigree were getting the spoils of the money-printing that was preventing the utter collapse of our economy.
How is giving money to the "right" people supposed to do that?
>tech workers are just a convenient receptacle to prevent a wage-price spiral in the broader economy.
How is giving money to these people in particular different than giving money to, say, nurses?
It's because the tech economy is a new pathway for money to flow. The wage inflation spiral that terrifies economists is where current jobs start paying more which leads to inflation in prices for goods and services which leads to the requirement for higher wages... I think that's the theory of the wage spiral.
So OPs theory is that the government can print money, boost GDP, and pay out profits to the owners/shareholders but not trigger the inflation... until now
Correct. The people we're talking about - Ivy League alums, the children of professionals, immigrants, and people who want to be like them - are much likelier than the average American to drive their wealth into assets and savings. Housing prices, stock prices, retirement portfolios would balloon, not (for the most part) the prices of everyday goods.
Which is what happened; the wage-price spiral that we did see was largely localized to Silicon Valley/San Francisco (and, to a lesser extent, a few big cities). Until the Sept. '19 repo crisis and then the pandemic, of course.
> How is giving money to the "right" people supposed to do that?
According to various studies there is a limit to how much money per year changes your base happiness. The ones I have read say $80,000 but inflation probably makes that a dated number. Everybody meaningfully burning more than that is either a member of the upper class buying superyachts or stuck in a red queen race like a speculative real estate market or collectibles. The ones who make more than the happiness threshold who don't have some other sink for it (the majority) prudently lock it into index funds where it's made available to the people who didn't want to lose control in the first place. That is the "how".
> How is giving money to these people in particular different than giving money to, say, nurses?
Canonically, there are three competing types of people getting paid out with some overlap:
1. Small business professionals. The standard here is your family doctor or dentist. Restaraunt owners also qualify. These groups deal with fairly intense regulation / "insurance" policies to limit how successful they are.
2. High-end white collar workers. Prior to the tech bubble conscripting software engineers, the usual suspect was lawyers. Lawyers have a wide gulf in compensation between "Ivy League Grads at BigLaw" and everyone else. That trend mirrors the current trend of BigTech vs. everyone else as described in the "Crack Theory". Look up "bi-modal salary" for more details.
3. Strikers. The halls of power are more than willing to max out the happiness threshold of anyone from auto workers in the 50s to airline points in the 90s who can make a credible threat that the spice will not flow. Power is willing to do this because they have spare money to print and the ability to strip the compensation long term by shifting jobs around. Based on current events in NJ, you may or may not get to watch this pay out in real time with your nurses.
The only reason people might hate it is because it has no proof or defense of the argument. Its a narrative that would require so many steps that its hard to imagine being true.
How about a simpler reason. During the mortgage crisis/recession the fed rate dropped to historical lows. Cheap money is good for startups. The economy was doing well for a decade and towards the end of that decade rates started to creep up and we got hit by a black swan, a pandemic. Rates crashed again + stimulus money and incredibly strong consumer spending and we now have high inflation. The rates go back up and its less attractive for startups/tech companies.
You're right, it's a holistic rather than analytical argument. I would at least say that part of the "proof" is in the fact that it's trying to explain what's in the article it's responding to, which we've taken to be true on its face. That's the mechanical action, but I agree that it's difficult to show the intent I accuse tech boards ad C-suites of. All a bit circumstantial, innit?
>Cheap money is good for startups.
We're talking about MEGAMAN (E stands for "Everyone else that fits the definition in my head"). Big, relatively established corporations. I would also counter that real income didn't budge after the GFC, so saying that "the economy was doing well" is stretching things.
> stimulus money and incredibly strong consumer spending
Seems like you are ultimately operating on the same theory: That tech was used to keep the "Great Financial Crisis" money away from the broader economy in order to prevent an inflationary spiral. It is unlikely that it was fully pre-meditated in exacting detail, but it is clear that the money 'printed' was directed to interests that would not spend it on bread, so to speak.
The pandemic stimulus changed that. It saw 'printed' money flow to regular average Joes who would spend it on bread, and thus consumer inflation took hold. That is what earlier stimulus packages were trying to avoid.
I have heard many times: Google hires good engineers not to _do_ things, but to _not do things elsewhere_. (even if 80% of Google engineers are 'productively' working on products Google cares about, thats thousands of grade-A engineers spinning their wheels, and given the rate Google discontinues and replaces its own software, 80% is certainly high)
I have propagated this meme. It's only partially true.
SREs at Google are generally an insanely good use of Google's resources in keeping actually profitable services (like ads, search, cloud, etc.) running and running at a standard that I think few non-Googlers on this forum really have a concept of.
But the bulk of SWEs are working on developing things which are not part of those Product Areas. I actually don't think the discontinuation thing is the main problem. The problem is that outside of Search, Ads & Cloud, pretty much nothing else at Google is really a profitable revenue generator. But you don't need >100k engineers to make the profitable stuff run. There's only so many ways to sling ads and report on them and build the infra for them.
But Google's conundrum is that if they just focused on those areas, someone would eventually come along with something that would eat their lunch. So in the past Google would prefer to a) hire those people and shower money on them to stop them from doing that and b) farm out a bazillion "bets" and projects to try their hand at that stuff in the hopes of striking a vein of gold again, like they did with AdWords 20+ years ago.
Cancelling projects is the byproduct of this continual search for new gold veins.
Also a lot is changing now. SV execs seem to have made a gentleman's agreement among each other to tighten the labour market.
Anyways, I think Meta is in more trouble than Google, long run. They're in a much riskier, shakier position.
> Also a lot is changing now. SV execs seem to have made a gentleman's agreement among each other to tighten the labour market.
Nah. They tried that and there was an anti-trust case.
I think this time everyone is 1) recovering from COVID and Inflationary Money Printing, and 2) not hiring more because AI is coming and they'd rather hold their breath.
This is nonsense. How are multibillion dollar companies accountable to shareholders and other owners willingly giving away money for some nebulous principle such as having the "right" people make money. Conspiracies should at least be based on some tangible elements. Fringe roles are too few to be seriously considered.
The reality is that it's all a matter of supply and demand. Ofc, there is waste, but any industry having margins like google are bound to excesses eventually if leadership doesn't intervene.
>How are multibillion dollar companies accountable to shareholders and other owners willingly giving away money for some nebulous principle such as having the "right" people make money.
Because of the specter of the alternative, once QE et al. started. Proven by the fact that it straight-up started happening anyway when the (completely justified) COVID helicopter money started flying.
Of course, there were other options:
>Vacuuming up the excess money supply through taxation before it became a pillar of an entire sector's business model
>Accept lower profits, executive compensation, and asset returns by letting worker pay rise without raising prices
But those are anathema, I suppose. Which is why it didn't take a conspiracy for everyone to act in such a way that what's described came about.
Well, the jury's still out, but the important thing to consider is what credible threat is keeping Powell from going full-Volcker and slamming on the brakes.
I wouldn't say that we're on the verge of an inflationary crisis because we've been in one, which we do actually have tools necessary to try to thread the needle. However, China, Japan, and the US commercial real estate market loom ominously over proceedings (and also that one thing no one is thinking of).
Going full Volcker is not called for. Inflation hasn't been building for nearly two decades; it's been nearly nonexistent for nearly two decades. Inflation hasn't reached 14%, nor anything close. They are, as you say, trying to thread the needle. That's still an option, and a less damaging one than going full Volcker.
> and also that one thing no one is thinking of
Don't vaguepost. If you've got something to say, say it.
>Going full Volcker is not called for. Inflation hasn't been building for nearly two decades; it's been nearly nonexistent for nearly two decades. Inflation hasn't reached 14%, nor anything close.
It's actually about 20-30% since 2020, if you measure the way they used to. People at the bottom are hurting, but the goal is to prevent full institutional collapse, so they get sacrificed in a slow walk to stabilization that risks hyperinflation, versus just getting it over with and starting on a recovery.
>a less damaging one than going full Volcker.
Well, a lost generation or two begs to differ.
>Don't vaguepost. If you've got something to say, say it.
Unknown unknowns. I have no idea where a hypothetical sucker punch is coming from because it's a sucker punch.
In all honesty, Volcker wasn't even needed. Let the initial COVID crash wipe out asset prices, helicopter money to everyone as we did, and let the people who were responsible enough to not get wiped out buy into ownership of properties and businesses. Recovery starts that winter and we're already there by now. Same diff to the average American, at worst, and you're not puppeteering zombie corps that you have to keep dumping money into to stave off the defaults that should have come already. This circus where the same people and entities who got us into this mess get to jack up prices to offset wage increases, and then some, is avoided. I don't see the issue.
The tech industry and business in general has an enormous challenge in how management is hired and selected. Currently it is either.
1. Selected by MBA or some management certification
2. Engineers who perform well/are more visible.
Both are really poor choices for technology related management roles and inevitably lead to unethical/poor management practices regardless of the firm in many cases.
The best manager traits are usually soft skills which are nearly impossible to measure and more impossible for HR to find (HR departments are largely lost in tech hiring). I think if you want to see examples of excellent run large companies you need to look in the places where there is an internal culture of up-skilling employees whether its rotational leadership programs or learning credits. Often my experience is they are much better run and invested in the people and it is reflected in the managers/management.
It’s possible to develop those soft skills. Management concepts can be taught. It’s possible to teach a good engineer how to use existing management frameworks.
Trouble is, often the learning materials are written in a way that does not “land” for people with an analytical technical background.
Translating management tools and concepts into tech speak for engineers on the management has been my job for a good part of the past decade. It can be done.
Look into Cognitive Distortions, the fancy words for "how people play themselves". I've found that poor management and impostor syndrome are both manifestations of self deception. For managers, self deception takes the form of not listening or not taking seriously to developer concerns, trusting their management's every word rather than using critical analysis in all interactions with their management, and generally "playing the Cover-Your-Ass game" rather than actually being their upper management's objective communication channel to developers and the objective voice for developers back such that both are functionally optimally in respect to the other.
Things like personal development, organizational design, team maturity, marketing, and selling seem like strange and foreign soft skills and concepts that your either have and understands, or not. There are models, mental frameworks, playbooks, and tools bring structure and rigor to these seemingly soft concepts. Engineers tend to have more analytical minds, and once you connect these things with the analytical “engine,” it becomes easier to understand and apply these concepts to management. In a way, the soft skills become more like hard skills.
Management learning is best done through a combination of theoretical learning, deliberate practice, and hands-on coaching. The theory comes from books and other reading maerials (see below). Deliberate practice comes from using that theory in everyday situations. Coaching comes from an experienced manager via one-on-ones, or an external coach.
If you go with the "external" route, find someone who seems to "speak your language" and understands your business challenges. I offer such coaching for companies who want help developing their engineering managers and would be happy to connect (see my HN profile for contact info).
Here are some resources for new engineering managers that I routinely use:
- A first person account of what you may discover and learn as you grow as a manager.
- Lots of concrete tips intermingled with personal stories.
- Great for discovering what you don't know that you don't know about management
Book: High Output Management, by Andy Grove
For those that got the basic hang of things, and want to go to the source. Don't let the fact that it was written in the last century turn you off. The book is somewhat dry and academic. At the same time it is dense with information and advice that largely stood the test of time. This book is concrete, clear, and to the point.
HN also underestimates the level of pettiness in the non-people-persons who became managers in tech. I was recently down voted for commenting that the typical tech manager does not want their engineers to have communication skills such that they can push back on unreasonable demands. My experience has been that is absolutely true; sure, they will say they want quality communications, but only as long as those communications are in agreement with whatever that manager and management want. Try telling them the truth that the overall architecture is bad, or there are these fundamental negative issues that were never addressed and are now consuming larger and larger resources to continue to "ignore"...
Actually being a material operator in a company that makes a difference is exactly what many, the majority, of middle managers simply can't handle. It scares them. It is too large, too aggressive, and demands too much upper level communications they can't handle. Not rocking the boat is the only game most managers know how to play, as the management above them appears simply untouchable to them.
Yes, I think part of the problem is we view social skills, charisma, etc. as givens, or even as virtues.
We forget that these things actually take a lot of brain function, it's an ability. It's something we do effortlessly, but there's actually a lot going on.
For example, struggles of people with ADHD and ASD can be hard to understand because many things that we take for granted are harder and more nuanced than people realize.
One of the interesting things about the tech industry (to me) is the presence and massive success of open source movements. OS does some things well, and others not as well, but it does it with 1/100th of the familiar management institutions we’re used to in the workplace. I don’t know what the answer is, but it makes me wonder if we haven’t somehow mis-arranged the whole thing. I’m old enough that I recall a time when organizations had secretaries, sometimes many. Now days it feels that management is really just the above, getting paid “higher than the rest of you” salaries to do what more equal secretary/clerk functions used to do.
I suppose the difference is money. Open source needn’t be free of its involvement, but it often is. Add money to open source and you get either a functional org, but with the usual overhead, or dysfunction and corruption.
Often, nobody would be doing the closed source stuff they're doing without the cold incentive of money, unlike free software which is inherently decoupled from a profit motive. Maybe there's an externality to pay in herding and keeping the cats committed to the profit motive.
I want neither. But I recognize the need. Why can’t we have “secragers” or “manataries”, who fill this need in a more co-equal fashion (and no, that is demonstrably not what product managers usually end up being)
You need to view matters from the perspective the parasites do to understand the situation. Inefficiencies and waste are good for them because the wasted money is funneled into their own pockets. Controlling large budgets and teams gives them a sense of greater power.
But the fact is, they don't need to horde all this. With a laptop anyone can achieve anything they need with the touch of a button. Minimalism is just more effective in the long run.
They need to realise, the things you 'own', own you.
It's especially important to be efficient when you're acting as a custodian for other people's money and wellbeing.
I've mixed feelings after reading this. On one hand I've had the "opportunity" to briefly work for some dysfunctional companies on short term projects and I know exactly what the author is talking about. But after over 2 decades in this line of work I disagree that this is the majority of Fortune 300~500.
Every huge company I worked for had some level of apparent dysfunction that later, once I got to know the history why they did it this way turned out to be the best of the worst options available.
At the same time I had people tell me, "of what a horrible mess and a dumpster fire of crap, are we allowed to throw it all out and do it properly? " when showing them around corporate systems and codebases. Then after eventually we obtain permission to "throw out and do it properly" for some small part of the system it is discovered that "no, the people that did it before us were not idiots, we were just not aware of certain constraints that forced their choices and now are forcing ours". You would have to replace everything... Which never was and never will be possible in a large company that has the same system and apps working since they started developing it on punch cards in 1960s...
Well, there's levels to it, right? I got an invite for a senior role at a company that some people here would recognize, and they had problems that sucked but were reasonable. Most of the problems I see don't really have a good reason, and I think people's experience is heavily influenced by career path-dependence.
I started at a trash place that didn't even use version control, so the pathway of jobs through my contacts has been a very slow road to improvement. Similarly, people that start at better places just tend to stay at better places.
Or as a much more experienced friend said to me: "All places are dysfunctional, but the difference between the best and worst I've worked at is indescribable."
Software is too valuable, too high potential. It's production methods too creative, too subjective, and insufficiently legible.
So... We can't afford not to do it. Sometimes we can't do it, and we can't afford not to just pretend we're doing it.
Do we really need to quantify the number of firms who's main declared priorities involve producing technology (eg "AI") that they have no idea how to make and no chance producing? I mean those who communicate this clearly to the board, the employees and such. Many/most aren't even tech companies. Their little teams that coordinate salesforce integration & automate google sheets are tasked with tasks somewhere between "winning ycombinator" and developing AGI in 5 months.
The reason might be a penchant for "growth stories." It might be something else, but we know it is. Maybe someone said "why aren't we just using AI to cut our CS costs in half?" Maybe no one had a convincing retort.
"I am near absolutely convinced that the vast majority of our species' ability to produce things of value for the human race is just utterly squandered at large companies."
So... I studied economics circa 2004. A conservative school, during the peak of certainty in particular economic ideas. A very rational view of microeconomics... firms, prices, etc.
Anything outside that paradigm was weird "alternative theory" and we learned the famous polemics to address the obvious rhetorical questions.
I've gone back to most of those polemics over the years. I think they have some answers for us... Especially thought from the 1920s
Anyway... I no longer think the model predicting that our species (or industry/company/etc) ability to produce things should be a limiting factor is totally wrong.
Sometimes our actual ability is the limiting factor. Those problems get solved though. The rest of the time, other factors are limiting us and utilizing our productive potential is irrelevant.
Eg smartphones. They were a hit. In about a decade they went from yuppie toy to prevalence among subsistence farmers. They got good, and cheap, fast. That's technology being efficient.
Once everyone has a decent smartphone, that market has peaked. The "classical model" predicts that prices should now freefall. The market can't get bigger, but that shouldn't make price performance halt. Prices should take over as the driver of price performance now.
^FYI The classical model's definition of "technology" is pretty much "price performance" and it is treated as a black box.
Irl, hat's not how this works. A commodity business model is not the next step. There's no incentive to go this route at all.
The model T had a similar life journey. Rapidly reducing prices until market peak, then a different business model... one that wasn't really about efficiency anymore. Henry Ford hated it, but he did succumb eventually.
That means we are no longer pursuing efficiency. Wasting productive potential is default, when in this mode.
At least from the perspective of the individual firm... Efficiency is only the paradigm sometimes... Times when supply (the companies' output) is factor limiting profit, share price, reward...
The rest of the time, it's demand.
Take retail banking. I am sure, if the population of earth was growing 20% YoY... retail banking would keep pace. Accounts & credit cards for everyone would be no issue.
They wouldn't need to scale their workforce by 20% pa. Banks/banking would get more efficient instead... especially if competition is weak.
That doesn't mean that in a "normal" state, where banks have limited growth potential... Those efficiency gains are not going to happen. It doesn't matter what technology exists. Retail banks are not going to slowly shrink, as efficiency gains make it possible to operate with half the ...
Imagine you're in charge of Google Search in like 2005. You've got a killer product that could probably be run realistically by 50 extremely good engineers working efficiently. Take out server costs, what's your profit margin, like 99%? Well if you're the board, and you've got a 99% profit company, what do you do? On the one hand you can get rich and call it a day, but then how long before a competitor builds the same exact product and you lose half? Or lose everything if everyone switches?
So you take a huge chunk that money and reinvest it with the hope of cementing your position or finding another product.
At this stage there's tons of money flying around. Internal projects are started and stopped constantly. Extreme inefficiencies are tolerated, after all, as the business grows, plenty of people just want to get paid and do as little work as possible.
Let’s say you can make $100B with 50 engineers at 99% profit margin. You pay your employees $1B and they are the highest paid in the world and all ecstatically happy and the stockholders make $99B and they are pretty happy too.
The board decides they’d rather make $300B at 50% margins because that’s $150B to investors. So it’s lower margin, but they’ve already invested and they just want more money.
Of course, I know many programmers that have insanely high profit margins with 1-4 employees and will never go public and no one except for an IRS computer will know they exist.
I think corporations are usually pretty efficient. It’s just not always clear what they are efficient toward.
> I think corporations are usually pretty efficient. It’s just not always clear what they are efficient toward.
This. People often examine things on a small scale and lack the context to understand why certain aspects appear counterintuitive. Once we gain a broader perspective, things begin to make more sense.
Yup; any department with money in their budget left over at the end of the budget period will see their budget reduced, even if they need it.
And (I'm theorizing here, I know little of management layers) I'm confident that a manager's wage is a percentage of a department's budget. Therefore, it's in their interest to use up and ask for more budget constantly.
> And (I'm theorizing here, I know little of management layers) I'm confident that a manager's wage is a percentage of a department's budget
Unlikely in a conventional organisation with salaried staff split into multiple depts / divisions. Bonuses may depend on the performance of a specific dept but base salaries typically have company-wide bands.
> Yup; any department with money in their budget left over at the end of the budget period will see their budget reduced, even if they need it.
Anecdote here. I've been a senior manager at small, medium, and large (ie FAANG) tech companies over the past 20 years and I've never run into this. Teams make a case for the budgets they need and those are approved (or not) based on the finance team's overall guidance and the return on investment. I'd be curious if others have have actually seen this "use it or lose it" mentality for budgets in practice.
> And (I'm theorizing here, I know little of management layers) I'm confident that a manager's wage is a percentage of a department's budget. Therefore, it's in their interest to use up and ask for more budget constantly.
Again, not my experience. Like you say, this would create perverse incentives which would quickly become apparent to a company's overall finances.
I don't doubt these sorts of policy mistakes have happened at other companies in the past, but I'm doubtful that they're pervasive, or even common, in the tech industry.
50 engineers couldn't even keep Borg running, let alone keep all the machines maintained, let alone procure new ones to keep up with growth, let alone do any product development to get any growth, let alone keep up with the regulatory changes constantly coming in necessary to even operate.
50 SRE's sounds like they could keep borg running. There really isn't that much details on what borg is made up of outside of a 2015 research paper[1].
Just to address the parts you laid out.
1. Keep Borg running
What would entail running? There are plenty of small teams that manage hundreds of servers and virtual machines through monitoring, deployment, networking, decom, etc lifecycle. This seems counter to purpose of cloud computing, commodity hardware and abstracted compute resources that can be deployed on-demand. Not to mention redundant sites, availability zones, <insert your cloud providers name for High Availability(HA) feature here>. Is this the part that SRE's maintain?
2. Machine maintenance
From documentaries[2] and other comparable data center operations this would seem to be handled by on-site datacenter staff. Disk replacement, physical replacement, network cabling, etc. Without any other operational info I would doubt resources would just be dedicated to borg as its a bit of a overall technician site work. As for OS and software updates and again without any other info or insight would seem to fairly automated after passing testing or at least passed to another team that just handles updates.
3. Machine procurement
Again commodity hardware or if its all custom still that would seem to be more of a EE task for build out and accounting for purchasing. Otherwise at Google's scale you would just get pre-populated racks delivered and replace the entire rack when 51% of the machines have failed. This doesn't really seem SRE or developer specific. It would also be happening for other services if they are already aren't abstracted from the underlying hardware/network layer.
4. Regulatory changes
I'm not sure what this would be in relation to a job processing system? Is this checking where you are saving data? Seems like a feature that is built once.
I agree that if you have follow-the-sun and need to be up with 5 (9's) 50 people in one time zone would be hard to ensure that but once the foundation is setup its just (50) people in a different location or timezone there isn't really anything too different about what they are doing.
If you have additional information or insight that would interesting to hear about.
I think you oversimplify a lot and miss many real world things that a Google sized org has to deal with.
Machine maintenance - someone has to coordinate updates across multiple data centers run by different organizations. Imagine you want to update networking from 1Gbs to 10Gbs across 3 DS in 3 different countries and time zones. After a while you’ll wish to own data centers, but that means multiple legal entities, bank accounts, etc. in different countries.
Machine procurement - one DC can buy Intels XYZ while another cannot because there are no local vendors who can deliver. So now you have a dilemma - build software working on different hardware, complicate all future updates or go and negotiate with Intel and other vendors.
Regulatory changes is when Malaysian government decides that it wants to protect children from something and you have to take extra measures before you can show search results or publish ads. Or EU votes for something called GDPR. Or Russia suddenly decides that Ads must be VAT taxed.
And don’t forget about a Microsoft exec who wants to discuss how they can use Google Search in their browser.
Honestly I don’t see how it can be done with 50 engineers. You’ll need hundreds of people, mostly managers and clerks to keep this thing running.
I appreciate the examples but none of these items are difficult regardless of size of the organization. These are standard system and network administrative items or on-site technician work. Unless you have other data about the borg architecture to show, its entire premise from the paper[1] is that it just runs jobs and removes the developers needing to know about the underlying hardware or OS that its running on it.
From the paper I referenced its huge cluster and if its their own software I'm sure with the development experience at google they have accounted for loosing a cluster node. Losing any number of cluster nodes is the same scenario if you were performing an software or hardware upgrade in which case you would be taking a node offline. I would be very skeptical that there are people manually kicking off upgrade jobs for nodes in the cluster. Maybe an A/B deployment or burn-in for a week before fully pushing it out with a pipeline workflow. I'd more so say that their borg implementation probably runs at different minor versions frequently.
Google runs their own data centers, thats the video documentary I referenced[2].
For machine procurement I don't see that as a SRE task. If they are making custom boards and systems they are going to have an entire group supporting that. At the size of google I'm thinking they wouldn't be running into supplier issues that they could figure out easily or account for in their borg software. Again its supposed to be one big abstraction.
For the regulatory changes for services like SEO/advertising, GMAIL, search, etc they all have separate teams for those services and would seem separate from SRE's that are maintaining borg the service which is provided to those groups.
Was there any additional ideas or insights you wanted to expand upon? I'm interested in what operational view points you have which would require more than 50 SRE's.
If you could expand on which parts you think would take more than 50 people to work on for a search engine that would be interesting to hear about. With the current cloud offerings hardware, processing, storage, high availability, etc are not really barriers to entry anymore and are offered as services.
There look to be a number of search engine startups[1] some big, some small.
Is there an equivalent for software org spend? Obviously it couldn't be as principled as the above but if you look at engineering at StackOverflow and their compute / requests ratio it's orders of magnitude better than companies I've worked at. I just want numbers in spend on compute and headcount related to operational requirements.
Has someone compiled a set of simple heuristics that says "If you are serving thousands of requests per hour to hundreds of enterprise clients your AWS bills shouldn't be more than thousands per month?" Something which can be used to say very clearly "we are incompetent" to upper management?
You think the waste is high in tech saving a few mill here and there with db settings an ec2 optimizations? Imagine all the low sophistication marketing managers and CDOs, CMOS literally with 10 to 100+ million budgets much of which is completely wasted. Ten years in this space I have 2nd hand knowledge of specific cases that make your face melt.
> The places doing things right tend to be characterized by being small, not being obsessed with growth, and having calm, compassionate founders who still keep a hand on the wheel.
I don't know. Just because a company doesn't move fast doesn't mean that they are a good company to work for. Calm and compassionate can just mean out of touch and dispassionate in investing in the future. Having energy and passion to move forward as well as the ability to place your bets somewhere is important too. When nothing ever happens because everything is too much of a risk, work can get a bit tedious. But I suppose in todays climate people will look towards these companies as the holy grail just to get some stability.
You’re manufacturing a false dichotomy here. There’s no contradiction between founders being compassionate and calm (in the sense of level-headed) and also being passionate and innovative. In those small companies where I worked, the work was anything but tedious, and there was no lack of risk. But exponential growth wasn’t the goal, making good products in a sustainable fashion was the goal.
I will back up here a bit as my first claim came off as a bit confrontational. However, I find it a bit unrealistic to expect an environment where you have founders that are very compassionate and calm. The very nature of building a business is a maddening task. Which is something we seem to forget.
Most level-headed people would never do it. The very idea of starting something that has a very minimal chance of success isn’t rational. Therefore I do find the traits you describe at somewhat of a contradiction.
Many of the most known innovators and leaders are often described as intense, driven, and even obsessive. Their willingness to defy convention, take significant risks and challenge the status-quo can be seen as crazy.
I suppose I’ve seen the other side of this coin, in small companies where everything is ”too much of a risk” if it cannot generate a good cash flow within a few months. This is a very level-headed approach. I’ve also seen them miss some great opportunities because of it. Nevertheless, they are still operating and probably will be for a long time. However, I'm uncertain that they have enough energy and will power to "change" the world.
Now, I'm not saying that a small company with calm level-headed leaders are a bad thing. But I don’t believe that there aren’t trade-offs. Both have their pros and cons.
Since around 2008-2009, our customers have not been customers. Our customers have been investors. Money has been so cheap that the primary goal of most organizations has been to court investment, not to actually create value. It was easier to raise money than to build product or sell anything.
To do this the goal is to make the company look like a potential "unicorn." That means burying whatever is wrong, over-hiring and over-engineering to make the company look serious and big, complexifying everything to give all those over-hired engineers something to do, and trying to gain as many users as possible as fast as possible even if it's completely unsustainable.
In that world efficiency literally does not matter. In fact, it can be a liability since it means you're not spending investor cash fast enough and you don't have enough complexity to make your over-hired engineering teams feel like they are being productive. If you're not spending that cash you don't need to raise more. It's the startup fund raising version of the well known "you have to spend your entire budget by the end of the fiscal year or you will get less next year."
The callous regard for human life jumped at me. My uncle worked for a big Wall St firm in the 80s and 90s. A CE from a big company was doing some early morning maintenance and long story short, had a heart attack and died in the data center.
The ops people called 911, fire/ambulance came — and were turned away at the door by security because they took too long and they needed to avoid impact to trading.
Basically they put some sort of tarp on the guy and wheeled him in and out of the way throughout the day. My uncle was some sort of big shot, heard about it, came in and reamed out the security director guy. Pretty sure the family never knew.
> The ops people called 911, fire/ambulance came — and were turned away at the door by security
It’s astounding that impeding an emergency response wasn’t an immediately arrestable offense. If there’s a mortal threat and “security” is in the way, police should have every right to subdue the rent-a-cops.
I know the answer.... Free Money[1], when the Fed lowered the prime interest rate to zero, the free money distorted the market and allowed all this insanity to happen.
Now that the free money is drying up, all this shit has to evaporate. Just like the great simplification[2] that has to take place on the downside of the great carbon pulse that's feeding civilization right now.
What a depressing thought in the short/medium term, in the long run, it'll be good for Humanity, my grandkids, should they every be born
Here is a story not about wasting cloud resources, but about wasting careers:
A decade-long project at the ThreeLetterYouknowwho was turning into a separate business, so people were being convinced to abandon their dozen-years old employments at the mentioned ThreeLetterGiant (evil or not, but at least with perspectives and an established routine how to live in this corp), so they did the move, only to be fired 5 months later, as the newly born company was reduced to a half, to fit into parameters required by some stock market related operation.
The crux is that the management knew about all the planned moves when negotiating people's leave from HAL (oops, I mean, ThreeLetterGiant). Yet they needed the massive flux of initial employees to fullfill some other business parameters. Everything was done with a cold blood.
I wasn't reduced this way (I wasn't even at HAL originally, was part of the completely fresh hires) but I left these assholes a month after the layoffs. Even the mob is treating their own people better.
241 comments
[ 3.5 ms ] story [ 234 ms ] threadhttps://www.cnbc.com/2023/10/31/zombie-firm-bankruptcies-ami...
Do you want to do a search or mobile OS startup with Google’s market position? Do you think they’ll release a YouTube or Google Maps application for your phone OS?
- Unfair taxation advantages, and fines are laughable.
A story about some idiot doing an idiotic thing at Google is vastly more likely to be passed around forever than a story at a random mom and pop company*. People love hearing about morons at big tech giants, and the storytellers oblige.
That, plus sufficient economies of scale or complementary tech products.
* unless that story is REALLY incredible, like that guy who went to an interview and accidentally killed the CEO's dog
But... some things can only be done with huge groups, so even working ineffectively can be worth it. And in a competitive setting, your competition also has these problems, so you can win in the market despite them.
My brother works at a small (<100 people) startup in the material sciences, and the amount of times that his and his co-workers pay statements got messed up in creative ways is incredible. Meanwhile I'm working at a decades old big company and never had any problems like that.
It does, but the flip side is that the bigness also causes some greater efficiencies due to economies of scale or specialization.
When you're just some low-level worker for a mega corp high-level decisions tend to get passed down with little to no context through several layers of management half of whom are disconnected from the issues you face on the ground, and the other half are disconnected from the decision making at the top.
This does cause some dysfunction, but I agree with you that it's not often not as bad as it seems when you have context of the why decisions are being made.
Not all companies do such a bad job at communication either and flatter organisation structures can help empower the people on the ground to make decisions which might otherwise need the approval of several layers of management. But still, in a large company there's inevitably always going to be some disconnect which will cause some level of dysfunction that you wouldn't get in a small company.
Besides, -- if you think how some companies work is fugly, go take a look at biology or politics. Companies are evolved systems. Once they're old enough they're almost always messy even when they aren't especially dysfunctional.
The human body is full of almost vestigial parts and dead end metabolic pathways... huge chunks of our DNA are just self replicating repeating sequences.
In reality the cruft in companies is often not even non-functional, but sometimes it serves some more complex purpose. Lots of large companies, for example, have some useless figurehead security position that exists for the actual purpose of having someone to fire when there the inevitable serious security issue occurs, without putting anyone actually productive at risk.
Companies are built out of and owned by people. Like anything else they have to respect the the limitations of the building material and the needs of their owners. The resulting requirements are no less real even though they might not exist in a world populated by homoeconomicus instead of man kind. If man kind wants a blood sacrifice for security incidents then it's reasonable and good for a company to have one.
But the value to society at large isn't just a profit (much less an ever growing profit). It's good that we have businesses which simply satisfy the involved parties (their customers, their employees, and their owners)-- even if they don't perform well on any external metric of efficiency.
I don't think it's anyone's job to be "euthanizing" low-profit businesses that might recover if there is a turn in the economy. The "zombie" concept is economically illiterate. Occasionally you will get a private equity firm doing this and the result is widely unpopular (Toys R Us, Maplin, etc)
More offent than not, not making enough profit is overreaching the profit goals to justify unfair stock prices. And the executives and board make bad gambles since they lose the owners money if they lose and win the owners money of they win.
Although IIUC you both read High Output Management so it might just be nice to know that even within fang people think things are wrong.
[1]: https://news.ycombinator.com/item?id=34385570 [2]: https://apenwarr.ca/log/20190926
It’s kind of a disguised communism and the longer the government keeps proping the economy (and making it difficult to start new companies), the worse this will get.
Name some.
Many very dysfunctional companies are also very profitable and have no need of outside bailouts. How long has IBM been at it?
Once the machine is up and running, you just have to avoid not breaking it. An org at this stage can be completely dysfunctional as long as the machine keeps running. Also figuring out the "wins" while scaling are much more obvious than while trying to start.
Big corporations have "moats" (as per Buffet's definition) and a big bank vendor locked into your solution will drag a badly working machine for years.
EDIT: Oh, and the part that earns money through regulatory capture earns so much money that we can have all these departments doing nothing in the meantime. We might do something one day with amazing leadership, but the incentives and odds are stacked against us.
For Facebook the revenue generator is Ads. For Amazon it was initially physical shipping, and later cloud hosting. Outside of those divisions you could burn billions on long shot projects that went nowhere, and nobody really cared...
This is the authors ideal of hell? oh you sweet summer child.
I don't really have anything important to say, but I did find this interesting. I'm unused to this sort of indirect response. It feels slightly like a personal response, since he links to a page where my comment is one of a few added, and refers to "commenters". However, my criticism was close to the opposite of what he is replying to.
I criticized the author for writing a story where I felt they mostly acted like a cog in a dysfunctional system, but was sneering at everyone else doing the same. I suppose, though, that perhaps they believe that the best/only solution for being in a poorly-functioning company is to jump ship.
edit: To clarify, I do agree that most corporate systems are dysfunctional and I'm not an agile fan. I'm not defending these systems.
The other half I put up there because I think they actually have me dead to rights. It's one of those weird things where, being fairly young, I already know that I'm going to look back on many of the things I write and realize they were hot-headed, and I wasn't nearly as good as I thought I was. I didn't think the writing came off as smug - I'm pretty aware that I clicked five buttons and skipped a computer science education - but I still appreciate the reality check that maybe I -am- being obnoxious. It's just fun to be a hothead sometimes, and I'm actually starting to get nervous with all the HN traffic. Yelling into the void is less threatening to my career.
The only quibble I have with the original comment is the assumption I didn't try very hard to let people know it was an issue - I promise I did, sometimes management just... doesn't care.
Anyway, what I'm trying to say is, sorry for any unintended offense I might have caused, because you were totally fair. It feels wrong to take it off the page anyway though!
That's crazy, how it isn't pure sabotage? career oriented people, jezz christ.
That's instant fire by me if somebody considered such a thing
Even at pure tech companies with somewhat technically competent leadership the incentives are whack, and often the people at the very top are still replicating some MBA doctrine that will be debunked as horse manure within a decade.
Just because our computers get exponentially better doesn’t mean people do, particularly groups of people.
Also, be careful what you wish for regarding cost savings. You may think you’re doing well saving 500k, but in a medium sized company there are likely a lot of easy ways to save a chunk more than that which you really aren’t going to like.
Apparently this is the level that most organisations can stagger on long enough for senior management to bank the bonuses and move on and upwards and before long term effects kick in.
I’ve seen good products and companies brought low by making the wrong bets and doing what was safe and sensible and cost reducing but not really thinking long term enough. They can stagger on for a long time still turning a reasonable profit until they suddenly don’t.
I was thinking of all the other ways businesses can “save money”, like travel bans, or getting rid of nice things, or closing offices, or getting rid of staff… They may not really save money, but they’ll look just as good on somebody’s spreadsheet.
Always turn up second or third to the fire otherwise they might suspect you of being the arsonist - Firefighter Arson is a real thing https://en.wikipedia.org/wiki/Firefighter_arson
That is still irrational, and I see no reason to forgive or excuse it instead of calling it out as irrational.
It's understandable, but that doesn't mean tolerable or excusable, just that the mechanism by which the irrational process plays out is understood.
Is any degree of sandbagging is a firing offense? If not, wheres the threshold?
Is your organization completely and unconditionally free of mandates like each department must "Cut budget x%" or "improve output by y%" -- irrespective of the incremental cuts and improvements that a group may have had in recent memory? If not, isn't a group that diligently finds and implements efficiencies at a disadvantage compared to ones that just don't look until a mandate comes in?
Kudos if you really do have an organization where there is no structural incentive to sandbag or otherwise bank results.
>Is any degree of sandbagging is a firing offense? If not, wheres the threshold?
Intentionally just to slack off or just because they arent sure or want use that time to improve other small thing?
People who sandbag to give themselves an easy life, or to try and empire build by getting more resources, definitely get noticed, and not in a good way.
This is actively incentivised in a lot of tech companies - maybe more so for managers than ICs, but the bigger-your-fiefdom the easier it is to promote phenomenon tends to be pretty widespread.
At one point my agile master was complaining to me about estimations, so I just focused on reporting the correct amount of hours to make the burn down chart be straight. Decoupling reported time from actual time spent.
Good lord I switched jobs.
Your organization should acknowledge and handle the fallibility of humans, not try vainly to excise it.
Yes, humans are bad at estimation, which is why it's good to reflect on how well you do and to correct if you consistently over or underestimate.
On smaller levels .. it is well-known in organisations that employ "stack ranking" that there is no incentive to get rid of poor performers immediately, and indeed an incentive to acquire one or two, so they can be sacrificed at stack ranking time.
They must have been using very overprovisioned servers to run up $1 million per year in Snowflake spend for occasional 10 minute bursts of activity prior to the change.
I think this points more to poor and ungoverned use of Snowflake and Cloud than it does organisational dysfunction.
Those are the same thing.
(one of the massive, massive advantages and disadvantages of cloud in general is that it circumvents "purchasing" departments. One the one hand: you don't have to spend $10,000 of time trying to buy $1000 of computer. On the other: you can waste millions on the cloud and maybe nobody will even notice.)
This is a "blame the user" argument that I hear a lot. It's true that in many or most cases users being super attentive can mitigate cost issues. But it's also true that this turns out to be very difficult in practice for a variety of reasons like scattered responsibility, complexity of cloud costs, and vendor cost models that encourage spending.
For my own money this problem looks like a business opportunity for vendors that offer customers ways to avoid it. The high margin era of cloud services with 10x markup on compute (e.g., Snowflake) won't last forever.
Although I started writing at 6PM on a random day after a stupid meeting, and actually tend to agree that there's worthier stuff to be popular - but people click on what they want.
I have a whole bunch of more boring stuff around, with much less swearing, and I think I'm more comfortable with the low level of attention that gets.
The article is good and at the end of the day, it's a... rant.
The core issue that substance the main point of the author is that we do not have a scientific and empirically validated method for management in technology (IT). This is expected since tech is a quite new industry that surged after the late 90's.
On top of it, we do not have established literature related to the economics of IT to calculate the opportunity costs, average acceptable waste, association between cloud costs x productivity and delivery value, and so on.
Most leaders and managers are using methods that can or cannot work, but it's a discovery phase now in how to establish a management method that covers inefficiencies, opportunity costs, and value.
If the main reason to stay is a steady paycheck then you're only doing harm to yourself by being aware of and caring about the bigger picture.
Ultimately though I got fed up and stopped working for large organizations. Small companies have their own problems, but they tend to be less maddening to me personally, so I'm happier overall. YMMV!
Management and agile happened. Instead of engineering, people chase tickets to achieve mediocre goals set by clueless non technical people. They thought something as creative as software development can easily be converted into assembly line manufacturing. Well, these are the results.
Agile engineering practices on the other hand are beneficial. But those arent really invented by agile. All existed well before agile.
Also every single large corporation i served suffered from endless meetings. It was almost always managers demanding such meetings. Somehow their belief is that engineers shouldnt spend time thinking about a problem - they should instead discuss it right away. A dubious mindset and telling of how said managers have no clue how thinking is to be made generally speaking. There needs to be communication between team members, but a good technical person is one that understands that and does it when needed. However since management and a factory mindset leads to hiring assembly line workers that lack communication skills that can sometimes be an issue - then the vicious cycle dictates that there need to be more meetings to cover communication gaps made by managers that hired assembly line workers that dont communicate well with peers, and so on.
Companies run by founders usually fare better because founders are either technical, domain experts and or care about inefficiencies - thus they rarely hire non technical managers.
Such companies often get acquired by larger companies that then turn them into factories and so the cycle begins. They still generate revenue and profits because generating profits doesnt always require a stellar product or sometimes they abuse their dominant position (google’s regressing, amazon’s a swamp of counterfeits, and both still make money).
After a long time in tech i’ve reached the conclusion that tech has been turned into factory work (including threats of automation replacing “workers”). Growth will indeed come from ai companies because … such companies are not yet managed and factorised to death (still run by tech founders, and so on). Thats why google’s struggling to get ai working while openai is smashing it. But those too will become bastardised.
This has not stopped you from posting the dozen or so blog posts you have already written so far. Go ahead and post what you are thinking. Do not let lack of experience stop you :)
You think it's hard trying to find good programmers? Try finding good managers.
As a result, a lot of tech leads are engineer->management conversions who just haven't made the leap yet.
And thats how it should be - folks with a technical background doing the management of technical people and work. How can you even manage that which you dont understand? Ticking boxes and measuring output is not management, that can be delegated to a script and graph charts. To manage the delivery of a technical project you need to understand the technology behind it - in an equal amount to how well you understand communicating with business stakeholders and balancing business and tech needs. And no, taking a udemy course in programming python doesnt mean you have tech experience, equally taking a udemy course in agile doesnt make you a manager.
Unless you’re an expert at literally everything, there is something in your life that you have to manage that you have no expertise in.
The typical way these things are managed is through measuring outcomes rather than focusing on how the work is performed. Tools include setting goals and milestones, auditing, cost bidding, reputation, etc.
So your assertion is that every leader who is not expert in literally every function performed in their organization is “incompetent”.
And measuring outcomes is “cope”.
So, say, if I need to consult a physician I’m incompetent because I should be an expert at managing my health. If I feel better after visiting a physician that’s “cope”.
This seems unnecessarily limiting. There are so many things I’m not an expert in yet I’m responsible for. I can’t just resign from having my house repaired because I’m “incompetent”.
I’m talking about leadership here, not living your own life. Obviously you aren’t going to have the same expertise as your doctor. That’s why you shouldn’t be running a medical practice.
In reality, much of the health care system is managed by incompetent bean counting administrators rather than actual experts, and that’s a huge problem.
The other side of the equation is that, at least in my experience, the best doctors are the ones who encourage and enable me to become an expert in my own health. If a doctor isn’t able to explain things to my satisfaction, I’m reluctant to trust their judgment.
> I can’t just resign from having my house repaired because I’m “incompetent”.
Nor do you want to be the guy who takes his car in for an oil change and gets billed an extra $100 for “replacing the headlight fluid.” When it comes to house stuff, the professionals are going to be better equipped and able to do the work faster than you are, but the basic principles behind how your house works shouldn’t be total mysteries to you, and if you pretend that you never have to bother learning about that stuff, you’re going to get taken for a ride.
What I stated at the outset is that the best managers “develop expertise in everything they have to manage”. You’re obviously not going to start off with all that expertise, but that doesn’t mean there’s a substitute for developing it. The idea that you can come into a field you know nothing about with zero expertise and be an effective leader by “measuring outcomes” is a fantasy. If we want to measure outcomes, let’s measure the outcomes of that philosophy. How many manufacturing or tech companies that used to be run by engineers continued to be successful after they were taken over by non-technical MBA’s? How many astronauts have died because NASA management lacked the expertise to understand and appropriately respond to the objections of the engineers who reported to them? Are American universities more cost-effective now that they’re run by administrators than they were when they were still run by academics? Is medicine better when treatment decisions are made by physicians or by HMO bureaucrats and government regulators?
I suspect that it actually is surprising for some people, but based on the emails I get, it's mostly cathartic because so many engineers spend time being told by management that they're imagining all of this. Because, as you've said, it's super hard to find good managers.
I think that people are looking at waste in a wrong way. It's relatively easy to point out inefficiencies in a working system. What's harder is organizing a working system. Energy and attention are finite on many levels, and this is true on the organizational level too. Let's pick an example of a computer game. Games are usually riddled with performance problems, in fact, I think that it's hard to find a well-optimized game. Yet many of these inefficient games provide good fun for a lot of people. As it turns out, engine efficiency is not a top priority for a game, there's much more to it, like the game design, level design, and the context of the game, which is marketing, timing the release, setting the price, and so on.
I find the author's dry sense of humour quite amusing, and I feel for them because I've been there. Most of us have been there I expect. I would say that feeding this mindset is likely to lead to burnout in the long term.
A lot of these issues are as inevitable as they are utterly confounding and at some point in your career I think you need to find that sweet spot where you can do your best work without having to push back too hard against the organisation. If you can manage to find that peace of mind then you're golden, and if you're lucky you can stay in a positive mindset or at least delay the onset of Jaded Greybeard Syndrome.
Secondarily, why should the author continue to fix things like this when they don't benefit, but others responsible are also not facing any consequences?
Loyalty to a company isn't rewarded today the way it used to be, so you just have to take that experience to a company with a better proposition at some point.
There's an ambiguity in the word "good": it can mean skilled, but it can also mean ethical. Personal ambition and greed can turn any organization dysfunctional, even if everyone is highly skilled. And you can't really blame individuals within a company for wanting to make as much money as they can, because that's more or less the point of the company itself.
As far as management is concerned, smart ethical people who are inexperienced will eventually become better at their jobs through experience, though this may involve a lot of trial and error. But in a certain important sense, smart unethical people will never become "better" at their jobs.
By the way, I personally don't think it's hard to find good (skilled) programmers. Companies make hiring a lot harder than it needs to be. Despite my years of documented excellence and my success as a self-employed indie developer, I'm essentially unhireable by most companies, because I'm not good at, or indeed willing to devote myself to, jumping through the arbitrary flaming hoops they set up to keep people out.
Perhaps this is an ethics problem too. Everyone seems very concerned about candidates who lie about their résumés. But for some strange reason, companies are exclusively focused on trying to evaluate the skills of candidates rather than evaluating their ethics. Here's a thought on this subject: companies practically demand that job candidates lie to them about the reasons they left their previous job. The truth is that a lot of people quit because their previous manager was an idiot or an asshole, but nobody is allowed to say that in a job interview, otherwise you'd be immediately rejected as someone who has a "bad attitude" and is not a "team player". In other words, playing along, having a "good" attitude, means lying.
I can't overstate how difficult it is to say "no" in the moment, and do so in a way that doesn't make you a problem and target in senior management's eyes.
Amen. Having HR set up some bullshit criteria and resume robot to presort your candidates, then subject the ones who fall through that sieve to stupid gotcha programming questions is an amazing way to develop the attitude that it’s “hard to find good programmers”
I hear people complain about this type of stuff all the time but based on the tests and challenges I've received they're reasonable. I've never been asked by a potential employer to do anything I didn't consider trivial or close to trivial. I've done some quizzes with easy questions (though the recruiters were very impressed that I aced them, apparently most don't come close), I've done a technical interview where I implemented a simple vehicle registry based on stubs and tests, just basic C# programming (I didn't even know C# when I did it, I knew Java) with basic data structures. A friend of mine had one where they were asked to implement a very simple frontend that gets some data from SpaceXs graphql api and display it in a table.
All of these are completely reasonable expectations for a competent developer. I could easily complete all of them as a fresh graduate with no work experience.
I understand not everyone is this reasonable, what I'm saying is that I think these kinds of simple exercises are a reasonable way to eliminate candidates who actually don't know basic shit you would expect them to know. There are a lot of them, even with degrees.
It's pretty much just bias. I'm biased towards these tests because to me they're trivial so it benefits me that I get an easy way to stand out. I also wouldn't want to hire anyone who can't do simple basic programming tasks, so I sympathize with the employers in that regard. On the other hand you have people who are mad because the tests reveal their incompetence.
When I hear "technical test", I don't think "reasonable implementation", I think of the interview I was in where I was asked a moderately difficult graph question, I reasoned my way to the solution from scratch, then got cut from the process with the explicit feedback that I should have recognized Djikstra's algorithm and had it memorized rather than needing to derive the correct solution myself (!!!)
(that still blows my mind, apparently being able to understand the problem and derive the correct solution is worse than memorizing it)
Your mistake was that you thought they wanted someone with creative problem solving skills…when what they really wanted was a parrot.
Given the choice between someone who will have ethics and someone who will make money unethically, companies will choose the latter every time. And the people who are willing to be unethical to get ahead are the ones who get rewarded internally once hired.
I didn't assert that. To the contrary, I said, "But for some strange reason, companies are exclusively focused on trying to evaluate the skills of candidates rather than evaluating their ethics."
Can relate. I was also a very good manager, but hated the job, and no one was interested in hiring me as one, anyway, (lost at Buzzword Bingo, yanno).
These days, I do my own thing, writing free software for folks that can't afford people like me, and I'm really happy.
Well, one very healthy rule to organize people is that people are responsible for their mistakes, which has a long-term effect of having less mistakes instead of having them covered under the carpet and still biting.
By the way, can you give a concrete example of a team mistake being org-level win? I rather think that incompetence is always a loose.
Is that necessary? I know 'Agile' isn't cool anymore, but at the heart of the Agile Manifesto is a suggestion that every contributor on the team should share in the management role rather than having a dedicated manager. Granted, it also suggests you need motivated people.
Yes, it is.
First, you can't expect everyone on the team to have all context on everything. That would be an incredible waste of everyone's time. Second, even if everyone did, in most cases the individuals would disagree on priority -- someone needs to be the final arbiter.
In practice, few people will make the "best decisions". They will often rationalize what they want. A leader needs to be there to "check" those decisions.
> to organise a large group of people towards any common purpose, let alone profitably, let alone efficiently, especially in something so relatively new and little-understood as tech
...I don't think those are good examples. The very sympathetic recent bio of Musk details a lot of dysfunction in those organizations, and they largely achieve anyway by leaning on The Mission, working people harder, driving burnout, and then discarding them.
A troubling question I often ask is: is it possible to do big things in a healthy balanced way? Or is greatness intrinsically a “Moloch” that demands human sacrifice? (That imagery was inspired by the 20s film Metropolis that touches on this question a little.)
If the latter is true then I don’t think this bodes well for the future of our species. It means we either die in the fire of our own striving or we die quietly in mediocrity, but either way we die. There’s never a lasting positive outcome.
BTW Elon Musk himself looks an awful lot like a victim of his own burnout culture. I suppose you can give him credit for taking his own medicine.
Would you say Monet didn't do great things because he only produced on average about 3 paintings per year?
We are simultaneously pulling the buggy, riding in it, and whipping ourselves to go faster.
Being a good manager also means shielding groups from other groups and making yourself the filter so you aren't wasting productive peoples times in endless meetings that are generally for other groups to show they are working to the wider company. You also have to provide a non-technical analysis of the internal state of the product/teams/tech to people outside your team. Those are the two hard things, everything else should be figuring out how you can make your team operate at it's highest levels without burning people out and hating their job.
1. Shielding the group 2. Provide non-technical analysis of the group/state of product 3. Remove road blocks 4. Provide resources
The industry that you are operating in seems to be adapted to the (4) principals you laid out.
I felt like there was so much on-sides violence against developers in the comments.
Where-as the I Saved submission was was such a great post about an org rotting out, about how jumbled stuff gets.
So much of this chaos & madness is organizational far more than it is individual, imo. Having a broader picture and context is hard. Having specific insight for squads facing specific issues is hard. I agree so much, it's really really hard to maintain a large org, on so many levels (including technical levels).
These days? The utter majority of workers are just another instantly-replaceable gear in a gearbox of madness, working for people on the other side of the globe no less - the exception are the few tradespeople and small shopkeepers that survived industrialization and Walmart/Amazon, and startups up until 100-ish people where the employee stock grant issues tend to get stopped.
Bonus, re: layoffs: Now, we are on the verge of another massive deflationary crisis for which we don't have any solution once the ball gets rolling, so such workers are no longer useful. The coddling ends, cuts to the bone begin (and justifications like TFA start appearing). Pigs get slaughtered.
Look at nokia. A few decades ago anyone with a billion dollars could start making cellphones. Good luck with that today. You would need a hundred billion, and a few decades of IP lawyering, before being allowed to compete with the likes of apple/samsung. If your device has a touch screen and connects to the internet, you will suffer innumerable patent issues.
Uh, you might have not been paying attention to it but profit does matter now. If you look out the window you can see all the startups doing mass layoffs or going out of business.
Oh, this time is soooo different to the dotcom crash, where it was speculation on companies, where profits didn't matter because it was the dream of future profits once a tech company <fill other stuff here>.
And this doesn't even take into account the possibility of these companies getting forcibly shut down / competitors coming in from outside.
The only question is how long will this take ?
Until it does.
Also, Google etc clearly are profitable. Insanely profitable. The question is... how? I think people are right to question whether the majority of the employees are engaged in work which ensures this profitability. But that's not the same as saying there's no profit.
It's all built on an edifice of advertising, which feels like a bit of a sandcastle to me. But I am also of the .com generation that saw the whole beach wash out into the ocean once already, and never understood how the advertising thing actually ended up yielding profits, and was initially quite pessimistic about the Google IPO (though was happy to take money from them by working there for a decade).
I've seen the CPMs and the CPCs from the inside. I don't get it. I worked on ad servers and in realtime ad bidding. But I still don't understand why ads make money.
total guess here, but instead of thinking of an ad as something that actually converts someone into a paying customer, maybe it's more about buying "mental real estate"? If nobody knows your brand, you have almost no sales. If you're at least in the neighborhood, then by pigeonhole principle, at least some customers will stop at your property. So maybe ads are a sort of dilemma that (with very few exceptions), your business must burn money on advertising just to stay in business?
Back when I worked in ads, the line was always that digital ads budget were still a fraction of what was being spent in traditional media, and so there was huge room to grow. I'm kinda guessing this is no longer the case.
[1] https://news.ycombinator.com/item?id=21046731
Idk, wasn't this the sentiment 30-40 years ago towards IBM?
Did you mixed-up million and billion?
A billion dollar is a gigantic pill of money. This feel a bit like saying "anyone with a British crown on his head can reign over the UK".
Also, looking at Fairphone, they operate with a fairly reasonable budget: all revenue, including phone sells in 2022 are around $60M, and if we exclude production, ~$17M are needed to pay for product development, marketing, support and all other expenses for a year. At 2 years to create a phone, that's ~$30M.
Because you're going to have to explain how it works.
>way to ensure that people with the "right" educational and professional pedigree were getting the spoils of the money-printing that was preventing the utter collapse of our economy.
How is giving money to the "right" people supposed to do that?
>tech workers are just a convenient receptacle to prevent a wage-price spiral in the broader economy.
How is giving money to these people in particular different than giving money to, say, nurses?
So OPs theory is that the government can print money, boost GDP, and pay out profits to the owners/shareholders but not trigger the inflation... until now
Which is what happened; the wage-price spiral that we did see was largely localized to Silicon Valley/San Francisco (and, to a lesser extent, a few big cities). Until the Sept. '19 repo crisis and then the pandemic, of course.
According to various studies there is a limit to how much money per year changes your base happiness. The ones I have read say $80,000 but inflation probably makes that a dated number. Everybody meaningfully burning more than that is either a member of the upper class buying superyachts or stuck in a red queen race like a speculative real estate market or collectibles. The ones who make more than the happiness threshold who don't have some other sink for it (the majority) prudently lock it into index funds where it's made available to the people who didn't want to lose control in the first place. That is the "how".
> How is giving money to these people in particular different than giving money to, say, nurses?
Canonically, there are three competing types of people getting paid out with some overlap:
1. Small business professionals. The standard here is your family doctor or dentist. Restaraunt owners also qualify. These groups deal with fairly intense regulation / "insurance" policies to limit how successful they are.
2. High-end white collar workers. Prior to the tech bubble conscripting software engineers, the usual suspect was lawyers. Lawyers have a wide gulf in compensation between "Ivy League Grads at BigLaw" and everyone else. That trend mirrors the current trend of BigTech vs. everyone else as described in the "Crack Theory". Look up "bi-modal salary" for more details.
3. Strikers. The halls of power are more than willing to max out the happiness threshold of anyone from auto workers in the 50s to airline points in the 90s who can make a credible threat that the spice will not flow. Power is willing to do this because they have spare money to print and the ability to strip the compensation long term by shifting jobs around. Based on current events in NJ, you may or may not get to watch this pay out in real time with your nurses.
How about a simpler reason. During the mortgage crisis/recession the fed rate dropped to historical lows. Cheap money is good for startups. The economy was doing well for a decade and towards the end of that decade rates started to creep up and we got hit by a black swan, a pandemic. Rates crashed again + stimulus money and incredibly strong consumer spending and we now have high inflation. The rates go back up and its less attractive for startups/tech companies.
>Cheap money is good for startups.
We're talking about MEGAMAN (E stands for "Everyone else that fits the definition in my head"). Big, relatively established corporations. I would also counter that real income didn't budge after the GFC, so saying that "the economy was doing well" is stretching things.
Seems like you are ultimately operating on the same theory: That tech was used to keep the "Great Financial Crisis" money away from the broader economy in order to prevent an inflationary spiral. It is unlikely that it was fully pre-meditated in exacting detail, but it is clear that the money 'printed' was directed to interests that would not spend it on bread, so to speak.
The pandemic stimulus changed that. It saw 'printed' money flow to regular average Joes who would spend it on bread, and thus consumer inflation took hold. That is what earlier stimulus packages were trying to avoid.
SREs at Google are generally an insanely good use of Google's resources in keeping actually profitable services (like ads, search, cloud, etc.) running and running at a standard that I think few non-Googlers on this forum really have a concept of.
But the bulk of SWEs are working on developing things which are not part of those Product Areas. I actually don't think the discontinuation thing is the main problem. The problem is that outside of Search, Ads & Cloud, pretty much nothing else at Google is really a profitable revenue generator. But you don't need >100k engineers to make the profitable stuff run. There's only so many ways to sling ads and report on them and build the infra for them.
But Google's conundrum is that if they just focused on those areas, someone would eventually come along with something that would eat their lunch. So in the past Google would prefer to a) hire those people and shower money on them to stop them from doing that and b) farm out a bazillion "bets" and projects to try their hand at that stuff in the hopes of striking a vein of gold again, like they did with AdWords 20+ years ago.
Cancelling projects is the byproduct of this continual search for new gold veins.
Also a lot is changing now. SV execs seem to have made a gentleman's agreement among each other to tighten the labour market.
Anyways, I think Meta is in more trouble than Google, long run. They're in a much riskier, shakier position.
Nah. They tried that and there was an anti-trust case.
I think this time everyone is 1) recovering from COVID and Inflationary Money Printing, and 2) not hiring more because AI is coming and they'd rather hold their breath.
The reality is that it's all a matter of supply and demand. Ofc, there is waste, but any industry having margins like google are bound to excesses eventually if leadership doesn't intervene.
Because of the specter of the alternative, once QE et al. started. Proven by the fact that it straight-up started happening anyway when the (completely justified) COVID helicopter money started flying.
Of course, there were other options:
>Vacuuming up the excess money supply through taxation before it became a pillar of an entire sector's business model
>Accept lower profits, executive compensation, and asset returns by letting worker pay rise without raising prices
But those are anathema, I suppose. Which is why it didn't take a conspiracy for everyone to act in such a way that what's described came about.
> Now, we are on the verge of another massive deflationary crisis for which we don't have any solution once the ball gets rolling...
Um, what? Deflation? No, that is not the crisis that we're on the verge of.
I wouldn't say that we're on the verge of an inflationary crisis because we've been in one, which we do actually have tools necessary to try to thread the needle. However, China, Japan, and the US commercial real estate market loom ominously over proceedings (and also that one thing no one is thinking of).
> and also that one thing no one is thinking of
Don't vaguepost. If you've got something to say, say it.
It's actually about 20-30% since 2020, if you measure the way they used to. People at the bottom are hurting, but the goal is to prevent full institutional collapse, so they get sacrificed in a slow walk to stabilization that risks hyperinflation, versus just getting it over with and starting on a recovery.
>a less damaging one than going full Volcker.
Well, a lost generation or two begs to differ.
>Don't vaguepost. If you've got something to say, say it.
Unknown unknowns. I have no idea where a hypothetical sucker punch is coming from because it's a sucker punch.
Well you might want to know that shadow stats doesn’t actually recalculate inflation using old methodology, it adds some constant factors.
https://econbrowser.com/archives/2008/10/shadowstats_res
Even a conservative government estimate says 19%: https://lao.ca.gov/LAOEconTax/Article/Detail/766#:~:text=Pri....
In all honesty, Volcker wasn't even needed. Let the initial COVID crash wipe out asset prices, helicopter money to everyone as we did, and let the people who were responsible enough to not get wiped out buy into ownership of properties and businesses. Recovery starts that winter and we're already there by now. Same diff to the average American, at worst, and you're not puppeteering zombie corps that you have to keep dumping money into to stave off the defaults that should have come already. This circus where the same people and entities who got us into this mess get to jack up prices to offset wage increases, and then some, is avoided. I don't see the issue.
Remind me again are we using production_database_v6_new_final22_final.xlsm or production_database_v4_final_finaldraft23_final_v2.xls ?
The best manager traits are usually soft skills which are nearly impossible to measure and more impossible for HR to find (HR departments are largely lost in tech hiring). I think if you want to see examples of excellent run large companies you need to look in the places where there is an internal culture of up-skilling employees whether its rotational leadership programs or learning credits. Often my experience is they are much better run and invested in the people and it is reflected in the managers/management.
Trouble is, often the learning materials are written in a way that does not “land” for people with an analytical technical background.
Translating management tools and concepts into tech speak for engineers on the management has been my job for a good part of the past decade. It can be done.
Management learning is best done through a combination of theoretical learning, deliberate practice, and hands-on coaching. The theory comes from books and other reading maerials (see below). Deliberate practice comes from using that theory in everyday situations. Coaching comes from an experienced manager via one-on-ones, or an external coach.
If you go with the "external" route, find someone who seems to "speak your language" and understands your business challenges. I offer such coaching for companies who want help developing their engineering managers and would be happy to connect (see my HN profile for contact info).
Here are some resources for new engineering managers that I routinely use:
https://themanagershandbook.com/
Book: The Making of a Manager
- A first person account of what you may discover and learn as you grow as a manager.
- Lots of concrete tips intermingled with personal stories.
- Great for discovering what you don't know that you don't know about management
Book: High Output Management, by Andy Grove
For those that got the basic hang of things, and want to go to the source. Don't let the fact that it was written in the last century turn you off. The book is somewhat dry and academic. At the same time it is dense with information and advice that largely stood the test of time. This book is concrete, clear, and to the point.
I think HN underestimates the amount of petty bullshit that managers have to fix in order for a company to function reasonably well.
Actually being a material operator in a company that makes a difference is exactly what many, the majority, of middle managers simply can't handle. It scares them. It is too large, too aggressive, and demands too much upper level communications they can't handle. Not rocking the boat is the only game most managers know how to play, as the management above them appears simply untouchable to them.
We forget that these things actually take a lot of brain function, it's an ability. It's something we do effortlessly, but there's actually a lot going on.
For example, struggles of people with ADHD and ASD can be hard to understand because many things that we take for granted are harder and more nuanced than people realize.
Often, nobody would be doing the closed source stuff they're doing without the cold incentive of money, unlike free software which is inherently decoupled from a profit motive. Maybe there's an externality to pay in herding and keeping the cats committed to the profit motive.
But the fact is, they don't need to horde all this. With a laptop anyone can achieve anything they need with the touch of a button. Minimalism is just more effective in the long run.
They need to realise, the things you 'own', own you.
It's especially important to be efficient when you're acting as a custodian for other people's money and wellbeing.
Every huge company I worked for had some level of apparent dysfunction that later, once I got to know the history why they did it this way turned out to be the best of the worst options available.
At the same time I had people tell me, "of what a horrible mess and a dumpster fire of crap, are we allowed to throw it all out and do it properly? " when showing them around corporate systems and codebases. Then after eventually we obtain permission to "throw out and do it properly" for some small part of the system it is discovered that "no, the people that did it before us were not idiots, we were just not aware of certain constraints that forced their choices and now are forcing ours". You would have to replace everything... Which never was and never will be possible in a large company that has the same system and apps working since they started developing it on punch cards in 1960s...
I started at a trash place that didn't even use version control, so the pathway of jobs through my contacts has been a very slow road to improvement. Similarly, people that start at better places just tend to stay at better places.
Or as a much more experienced friend said to me: "All places are dysfunctional, but the difference between the best and worst I've worked at is indescribable."
So... We can't afford not to do it. Sometimes we can't do it, and we can't afford not to just pretend we're doing it.
Do we really need to quantify the number of firms who's main declared priorities involve producing technology (eg "AI") that they have no idea how to make and no chance producing? I mean those who communicate this clearly to the board, the employees and such. Many/most aren't even tech companies. Their little teams that coordinate salesforce integration & automate google sheets are tasked with tasks somewhere between "winning ycombinator" and developing AGI in 5 months.
The reason might be a penchant for "growth stories." It might be something else, but we know it is. Maybe someone said "why aren't we just using AI to cut our CS costs in half?" Maybe no one had a convincing retort.
"I am near absolutely convinced that the vast majority of our species' ability to produce things of value for the human race is just utterly squandered at large companies."
So... I studied economics circa 2004. A conservative school, during the peak of certainty in particular economic ideas. A very rational view of microeconomics... firms, prices, etc.
Anything outside that paradigm was weird "alternative theory" and we learned the famous polemics to address the obvious rhetorical questions.
I've gone back to most of those polemics over the years. I think they have some answers for us... Especially thought from the 1920s
Anyway... I no longer think the model predicting that our species (or industry/company/etc) ability to produce things should be a limiting factor is totally wrong.
Sometimes our actual ability is the limiting factor. Those problems get solved though. The rest of the time, other factors are limiting us and utilizing our productive potential is irrelevant.
Eg smartphones. They were a hit. In about a decade they went from yuppie toy to prevalence among subsistence farmers. They got good, and cheap, fast. That's technology being efficient.
Once everyone has a decent smartphone, that market has peaked. The "classical model" predicts that prices should now freefall. The market can't get bigger, but that shouldn't make price performance halt. Prices should take over as the driver of price performance now.
^FYI The classical model's definition of "technology" is pretty much "price performance" and it is treated as a black box.
Irl, hat's not how this works. A commodity business model is not the next step. There's no incentive to go this route at all.
The model T had a similar life journey. Rapidly reducing prices until market peak, then a different business model... one that wasn't really about efficiency anymore. Henry Ford hated it, but he did succumb eventually.
That means we are no longer pursuing efficiency. Wasting productive potential is default, when in this mode.
At least from the perspective of the individual firm... Efficiency is only the paradigm sometimes... Times when supply (the companies' output) is factor limiting profit, share price, reward...
The rest of the time, it's demand.
Take retail banking. I am sure, if the population of earth was growing 20% YoY... retail banking would keep pace. Accounts & credit cards for everyone would be no issue.
They wouldn't need to scale their workforce by 20% pa. Banks/banking would get more efficient instead... especially if competition is weak.
That doesn't mean that in a "normal" state, where banks have limited growth potential... Those efficiency gains are not going to happen. It doesn't matter what technology exists. Retail banks are not going to slowly shrink, as efficiency gains make it possible to operate with half the ...
Imagine you're in charge of Google Search in like 2005. You've got a killer product that could probably be run realistically by 50 extremely good engineers working efficiently. Take out server costs, what's your profit margin, like 99%? Well if you're the board, and you've got a 99% profit company, what do you do? On the one hand you can get rich and call it a day, but then how long before a competitor builds the same exact product and you lose half? Or lose everything if everyone switches?
So you take a huge chunk that money and reinvest it with the hope of cementing your position or finding another product.
At this stage there's tons of money flying around. Internal projects are started and stopped constantly. Extreme inefficiencies are tolerated, after all, as the business grows, plenty of people just want to get paid and do as little work as possible.
Let’s say you can make $100B with 50 engineers at 99% profit margin. You pay your employees $1B and they are the highest paid in the world and all ecstatically happy and the stockholders make $99B and they are pretty happy too.
The board decides they’d rather make $300B at 50% margins because that’s $150B to investors. So it’s lower margin, but they’ve already invested and they just want more money.
Of course, I know many programmers that have insanely high profit margins with 1-4 employees and will never go public and no one except for an IRS computer will know they exist.
I think corporations are usually pretty efficient. It’s just not always clear what they are efficient toward.
This. People often examine things on a small scale and lack the context to understand why certain aspects appear counterintuitive. Once we gain a broader perspective, things begin to make more sense.
And (I'm theorizing here, I know little of management layers) I'm confident that a manager's wage is a percentage of a department's budget. Therefore, it's in their interest to use up and ask for more budget constantly.
Unlikely in a conventional organisation with salaried staff split into multiple depts / divisions. Bonuses may depend on the performance of a specific dept but base salaries typically have company-wide bands.
Anecdote here. I've been a senior manager at small, medium, and large (ie FAANG) tech companies over the past 20 years and I've never run into this. Teams make a case for the budgets they need and those are approved (or not) based on the finance team's overall guidance and the return on investment. I'd be curious if others have have actually seen this "use it or lose it" mentality for budgets in practice.
> And (I'm theorizing here, I know little of management layers) I'm confident that a manager's wage is a percentage of a department's budget. Therefore, it's in their interest to use up and ask for more budget constantly.
Again, not my experience. Like you say, this would create perverse incentives which would quickly become apparent to a company's overall finances.
I don't doubt these sorts of policy mistakes have happened at other companies in the past, but I'm doubtful that they're pervasive, or even common, in the tech industry.
Just to address the parts you laid out.
1. Keep Borg running
What would entail running? There are plenty of small teams that manage hundreds of servers and virtual machines through monitoring, deployment, networking, decom, etc lifecycle. This seems counter to purpose of cloud computing, commodity hardware and abstracted compute resources that can be deployed on-demand. Not to mention redundant sites, availability zones, <insert your cloud providers name for High Availability(HA) feature here>. Is this the part that SRE's maintain?
2. Machine maintenance
From documentaries[2] and other comparable data center operations this would seem to be handled by on-site datacenter staff. Disk replacement, physical replacement, network cabling, etc. Without any other operational info I would doubt resources would just be dedicated to borg as its a bit of a overall technician site work. As for OS and software updates and again without any other info or insight would seem to fairly automated after passing testing or at least passed to another team that just handles updates.
3. Machine procurement
Again commodity hardware or if its all custom still that would seem to be more of a EE task for build out and accounting for purchasing. Otherwise at Google's scale you would just get pre-populated racks delivered and replace the entire rack when 51% of the machines have failed. This doesn't really seem SRE or developer specific. It would also be happening for other services if they are already aren't abstracted from the underlying hardware/network layer.
4. Regulatory changes
I'm not sure what this would be in relation to a job processing system? Is this checking where you are saving data? Seems like a feature that is built once.
I agree that if you have follow-the-sun and need to be up with 5 (9's) 50 people in one time zone would be hard to ensure that but once the foundation is setup its just (50) people in a different location or timezone there isn't really anything too different about what they are doing.
If you have additional information or insight that would interesting to hear about.
[1] https://research.google/pubs/pub43438/ [2] https://www.youtube.com/watch?v=XZmGGAbHqa0
Machine maintenance - someone has to coordinate updates across multiple data centers run by different organizations. Imagine you want to update networking from 1Gbs to 10Gbs across 3 DS in 3 different countries and time zones. After a while you’ll wish to own data centers, but that means multiple legal entities, bank accounts, etc. in different countries.
Machine procurement - one DC can buy Intels XYZ while another cannot because there are no local vendors who can deliver. So now you have a dilemma - build software working on different hardware, complicate all future updates or go and negotiate with Intel and other vendors.
Regulatory changes is when Malaysian government decides that it wants to protect children from something and you have to take extra measures before you can show search results or publish ads. Or EU votes for something called GDPR. Or Russia suddenly decides that Ads must be VAT taxed.
And don’t forget about a Microsoft exec who wants to discuss how they can use Google Search in their browser.
Honestly I don’t see how it can be done with 50 engineers. You’ll need hundreds of people, mostly managers and clerks to keep this thing running.
From the paper I referenced its huge cluster and if its their own software I'm sure with the development experience at google they have accounted for loosing a cluster node. Losing any number of cluster nodes is the same scenario if you were performing an software or hardware upgrade in which case you would be taking a node offline. I would be very skeptical that there are people manually kicking off upgrade jobs for nodes in the cluster. Maybe an A/B deployment or burn-in for a week before fully pushing it out with a pipeline workflow. I'd more so say that their borg implementation probably runs at different minor versions frequently.
Google runs their own data centers, thats the video documentary I referenced[2].
For machine procurement I don't see that as a SRE task. If they are making custom boards and systems they are going to have an entire group supporting that. At the size of google I'm thinking they wouldn't be running into supplier issues that they could figure out easily or account for in their borg software. Again its supposed to be one big abstraction.
For the regulatory changes for services like SEO/advertising, GMAIL, search, etc they all have separate teams for those services and would seem separate from SRE's that are maintaining borg the service which is provided to those groups.
[1] https://research.google/pubs/pub43438/ (research paper on borg)
[2] https://www.youtube.com/watch?v=XZmGGAbHqa0 (documentary at their data centers)
There look to be a number of search engine startups[1] some big, some small.
[1] https://startupsavant.com/startups-to-watch/search-engine
Server costs were actually bulk of the costs, not engineers salaries.
Is there an equivalent for software org spend? Obviously it couldn't be as principled as the above but if you look at engineering at StackOverflow and their compute / requests ratio it's orders of magnitude better than companies I've worked at. I just want numbers in spend on compute and headcount related to operational requirements.
Has someone compiled a set of simple heuristics that says "If you are serving thousands of requests per hour to hundreds of enterprise clients your AWS bills shouldn't be more than thousands per month?" Something which can be used to say very clearly "we are incompetent" to upper management?
This right here. Go look for those places.
Most level-headed people would never do it. The very idea of starting something that has a very minimal chance of success isn’t rational. Therefore I do find the traits you describe at somewhat of a contradiction.
Many of the most known innovators and leaders are often described as intense, driven, and even obsessive. Their willingness to defy convention, take significant risks and challenge the status-quo can be seen as crazy.
I suppose I’ve seen the other side of this coin, in small companies where everything is ”too much of a risk” if it cannot generate a good cash flow within a few months. This is a very level-headed approach. I’ve also seen them miss some great opportunities because of it. Nevertheless, they are still operating and probably will be for a long time. However, I'm uncertain that they have enough energy and will power to "change" the world.
Now, I'm not saying that a small company with calm level-headed leaders are a bad thing. But I don’t believe that there aren’t trade-offs. Both have their pros and cons.
Since around 2008-2009, our customers have not been customers. Our customers have been investors. Money has been so cheap that the primary goal of most organizations has been to court investment, not to actually create value. It was easier to raise money than to build product or sell anything.
To do this the goal is to make the company look like a potential "unicorn." That means burying whatever is wrong, over-hiring and over-engineering to make the company look serious and big, complexifying everything to give all those over-hired engineers something to do, and trying to gain as many users as possible as fast as possible even if it's completely unsustainable.
In that world efficiency literally does not matter. In fact, it can be a liability since it means you're not spending investor cash fast enough and you don't have enough complexity to make your over-hired engineering teams feel like they are being productive. If you're not spending that cash you don't need to raise more. It's the startup fund raising version of the well known "you have to spend your entire budget by the end of the fiscal year or you will get less next year."
The ops people called 911, fire/ambulance came — and were turned away at the door by security because they took too long and they needed to avoid impact to trading.
Basically they put some sort of tarp on the guy and wheeled him in and out of the way throughout the day. My uncle was some sort of big shot, heard about it, came in and reamed out the security director guy. Pretty sure the family never knew.
It’s astounding that impeding an emergency response wasn’t an immediately arrestable offense. If there’s a mortal threat and “security” is in the way, police should have every right to subdue the rent-a-cops.
Now that the free money is drying up, all this shit has to evaporate. Just like the great simplification[2] that has to take place on the downside of the great carbon pulse that's feeding civilization right now.
What a depressing thought in the short/medium term, in the long run, it'll be good for Humanity, my grandkids, should they every be born
[1] https://en.wikipedia.org/wiki/Zero_interest-rate_policy
[2] https://www.thegreatsimplification.com/about
A decade-long project at the ThreeLetterYouknowwho was turning into a separate business, so people were being convinced to abandon their dozen-years old employments at the mentioned ThreeLetterGiant (evil or not, but at least with perspectives and an established routine how to live in this corp), so they did the move, only to be fired 5 months later, as the newly born company was reduced to a half, to fit into parameters required by some stock market related operation.
The crux is that the management knew about all the planned moves when negotiating people's leave from HAL (oops, I mean, ThreeLetterGiant). Yet they needed the massive flux of initial employees to fullfill some other business parameters. Everything was done with a cold blood.
I wasn't reduced this way (I wasn't even at HAL originally, was part of the completely fresh hires) but I left these assholes a month after the layoffs. Even the mob is treating their own people better.
They actually treat them quite well. I know a number of ex-wiseguys.
When you manage folks that, by definition, don't like rules and structure, and can probably clip you in a moment, you learn to manage well.
Bad managers don't last (literally).