If a corporate lawyer ever ambushes you with this, the best thing to do is ask instead how you can make your current process auditable, rather than tearing the org apart. If that doesn't work, start looking for a more innovative place to work!
I'd say it's mostly regulation, risk-aversion, and company size.
When you're small, it's easy to avoid regulatory attention and whatever slap-on-the-wrist fine you may get isn't a lot of money.
The change and risk aversion really just comes from keeping the business running. When things work, try not to break them. Old large enterprises tend to have a lot of business critical things that "just work" and have been for a long time. Younger/smaller companies tend to have fewer battle scarred systems.
But once they get scarred, they typically add a process or policy to prevent it from happening again. After enough of them, congrats, you're enterprise speed.
All true, but I would add: there's a lot less downside to change, when nothing much is working yet. Much of enterprise slowness is irrational, but much is not. Big systems are harder to predict consequences for, and big old systems have a lot that already works.
Add in the tendency to consolidate and standardize technologies as the company grows for maintainability just means even little changes quickly ripple through the system.
It's like microservices, but with systems of people and businesses.
and SLAs. once you get big enough your client contracts usually have penalties for failures of any kind. that puts the pressure on getting it right on the first try or at least within contract terms, instead of working toward the goal incrementally.
partners that allow for some agile, iterative back and forth becomes rarer as they increase in size, especially since often their own system lack the flexibility for continuous integration or safely rolling back changes/data.
that's why you get requirements, specifications, architects and solution managers in the enterprises. people love to mock them for being dysfunctional and some truly are, but once contracts get into the millions and lawyer get involved mistakes are no longer "learning opportunities".
Eh. Article is a bit short on facts. Starting with the first bullet, the only place I've ever worked that rolled out OS updates continuously with a maximum 15-day latency was also an enormous global enterprise with tens of thousands of employees and plenty of regulatory red tape. Now I work at a much smaller company that thinks of itself as "scrappy" but is utterly paralyzed by OS updates in their production fleet because they don't have the tooling and processes around it. It is the tooling and the process that gives you the ability to move quickly!
A huge upside of Windows 10's six month rolling releases is that they initially broke a lot of badly written things, and software and processes have had to improve to handle them better.
The end result two years later is OS upgrades rarely break much, and regular updates never do. Enterprises can be unstuck but it can be painful.
I think enterprises are still shifting from the old approach where you make it work and then it's fine, to realizing that in an internet-connected world, you have to keep up at all times. A few big liability cases would do wonders to push this shift along.
I hope what you are saying about Win 10 is true. My organisation is in process of transitioning from Windows 7 to Windows 10 and things are breaking all over the place.
Almost everytime we change versions of software things break when we went moved to office 365 there was breakage (Stuff I would have thought for sure would have been well tested - PST files in outlook for example broke).
Same thing when we went from Internet Explorer 8 to 11 our internal intranet didn't work.
Nearly everything that works in Windows 7 works in Windows 10, if you poke enough holes in it. The biggest exception to that is really just drivers, because hardware manufacturers have no incentive to update them past a certain point. There was definitely software that had to be updated to the latest versions to work, but that's where we needed to get to anyways.
I work in an environment that includes custom-written apps from the Windows XP era, and usually the absolute worst we hit was "hey, this device doesn't work on Windows 10, we have to buy a newer one". Not a big deal when it's a $6 USB-to-serial adapter, but people will look at you like you're crazy when you say the network is incompatible with an HVAC system. (Solution in that case, just take it off the network, you're better off.)
But believe it or not, you can still sync Windows Mobile 2003 devices with Windows 10 PCs, Microsoft's backwards compatibility is still insanely solid in most cases.
In my experience, Win 10 updates are still very much a crapshoot as to whether or not any given one of them will break something important to your org.
I don't want a red queen scenario where if you go under 30 commits per day you explode, especially because it seems to hit both the long tail of low-resource off-beat projects and anything that interfaces with areas of society that benefit much more from being long-term stable. Besides the obvious infosecpocalypse of “update or get cracked”, what's causing this? Is there a way to turn this thing around, or some good ways to mitigate the effects?
I maintain code written by my company's founder from 1998.
It hadn't seen many updates to maintain it. It saw a LOT of enhancements.
I'm currently unwinding those changes into a full library of support features.
It's also not my primary focus.
Many new features to fulfill customer needs is a common reason. Many customers, with new requests, issues reported.
It's slow because if I'm not methodical I cause problems for people who don't want this next enhancements. It's slow, because the current emergency takes precedence, and taking to long on fixes means a meeting on why something is behind.
Business priority is not "deliver fast" it's "keep running, don't attract attention".
"A friend of mine got a job at a big company and was
shocked to see his colleagues worked just a few productive
hours a day.
They didn’t seem to care about their work or have relevant expertise.
My friend said: “Wow, this company is going under.” Then the company
released its quarterly reports and profits rose to an all-time high.
The momentum of the company’s brand and relationships was sufficient to
propel it forward."
It isn't just relationships and momentum. XKCD had an interesting what if -- what would happen if a bowling ball was blown up to the size of Earth?[1] Turns out at that scale metal behaves like a liquid.
Work at big companies is kind of like that. Once a company gets big enough, any given person doesn't really know anything. Seemingly no one is doing anything productive -- it's meetings all the way down. But the software still gets shipped. It doesn't happen the same way it happens at a startup where someone sits down and codes a feature. It happens in a totally different way, where one hundred engineers go to meetings all day and talk to QA and analytics and bizdev, and maybe commit one line of broken code a week. It takes way more people, and the production process is totally different, and it takes way longer, but stuff still happens (often quite successfully).
That was probably referencing the story of the origin of the IBM PC. This anecdote was given as the reason they got the green light to develop the PC in Boca Raton independent of the rest of IBM.
Then again, I've seen it repeated (though would appreciate someone backing it up with a citation) that on planetary scales, everything behaves like a high-viscosity liquid. That was in context of modeling e.g. moons hitting planets.
It seems to be, yes. [0] cites IAU definition, which mentions "hydrostatic equilibrium", suggesting at this level things are viewed through the lens of fluid mechanics.
This is my experience so far, I've been waiting for this place to implode for two years. Somehow we ship enough to tick boxes in the spreadsheets, and enough that customers aren't forced to leave (although they aren't happy). It's really weird, and a hellish place to work, but they pay well.
I was once required to log what I did during the day and it is depressing. Between the interuptions, and simply the fact that everything takes a lot longer that one estimates. I think it’s also why most timing estimates for software development are underestimated.
Big companies have something to lose. A brand, clients, revenue and profits. People don’t want to risk those.
A starup has nothing to lose. The site is janky for two hours? The only users are friends, and the outage might actually bring more visibilty if you make a good write-up.
Incidentally, this ties together with “why do young people found more startups than senior executives”
Great article. There’s also other good reasons why enterprises move slowly ( cultural and other organisational structures and priorities that conflicts with making the same kinds of decisions a startup might make).
The article is pretty spot on. One thing the author has left out is the fact that all of the problems are magnified 100x when the people involved are incompetent, which, unfortunately, happens a lot in enterprises (one can also argue that slow speed contributes to job security under normal circumstances).
Lots of smart, talented and energetic people are expending most of their power just figuring out who to talk to about what or what the hell that out-of-the-blue email from corporate finance means for them.
Just figuring out how to get a thing done can be exhausting.
Yep, can attest to this. One of the reasons why senior people are extremely productive & valuable is that they seem to know the landscape more than others, and therefore know who to talk to & where to look.
Half-jokingly, as soon as an "individual contributor" knows enough to be productive, they're promoted to management.
WRT the half-joke, this is one of those truisms that probably has a lot of basis in reality, but there needs to be a corollary about the choice of the individual as well.
I was heavily invested in our public facing technical side, working a lot with our on-boarding, teaching newbies to be really self-sufficient, working through the technical issues with our clients directly, and then I was asked to take on a Management position.
I liked the pay raise, but I made it clear exactly what I wanted from the position, and though we butt heads on it still, my boss respects the fact that I still am heavily invested with our technical team while doing the Managerial work. We had to revisit the job definition a few times to find a description that matched what I was (am) doing and also made HR happy, but we did it.
The point is that in some cases at least, a promotion to management doesn't have to be a death knell for "individual contributors". It depends on the company's willingness to accommodate the wants of the individual, and the individual's own desire to continue contributing. I know too many people who did get the Management golden ticket and relished it for just being an excuse to boss people around and make more dashboards in our CRM.
At the same time, I have read from people making the transition from IC to manager that it's a mistake to hold on to your technical work. Surely this will vary from person to person, company to company, and team to team, but I find it hard to believe that people with 10-15+ years of professional dev experience are best utilizing their skills sitting far away from the code; at the same time, they can't get too focused on it and ignore their management responsibilities.
I wonder how much of it has to do with team sizes. I feel like a 4 person team (inclusive) would be ideal for a manager who also wanted to act as a technical lead and contribute a moderate amount. Any bigger, and the scope of what your team does, as well as the amount of time needed to devote enough time to all your employees, starts to get too big for you to contribute at a technical (e.g. source code) level. Yet we continue to take great technical workers and put them in this weird PM-hiring-HR-architect mixture of a position where they barely continue to leverage their skills anyway
In the context of a startup, if you try something and fail, you'll generally be recognized for taking initiative even if the result does not pan out. At that level, learning what doesn't work is often as valuable as learning what does work. Startups often bias towards hiring people with just this mentality because that is what they actually need.
In the context of an established company, sticking your neck out is like career suicide unless it's a sure thing. Unlike a startup, you're risking an _established_ reputation on a new idea. So you end up working in an organization that has an inherently different view of risk and trying new things.
I'm not sure whether "nobody knows how to do anything" results from an institutional aversion to risk or from the natural result of hiring people that fit within that system, but the end effect is exactly as you describe. It's way too hard to do anything off of the beaten path, and trying to do anything along those lines _immediately_ results in negative feedback which makes employees a lot less likely to attempt such actions in the future.
Good points. In my comment, I was focused on the difficulties around getting the information you need and being aware of what is happening in other parts of the company, but there is also the huge factor around punishing risk-takers too.
For example, I can build a chatbot in a day that will take some load of a bunch of people in my department. Will it be deployed? No way. Will it be seen as a possible threat to people's patronage networks by cutting their teams and budgets? You bet. Will I possibly be chastised for doing something that is not in my JD? Possibly. Will I bother making it in this case? Nope.
Outside of delivering a new development or product that might need someone to stick their neck out, the larger the company mostly the worse they get at everything. Sales, support, raising an invoice, paying your bill, correcting a mistake, you name it.
To customers it really does seem as simple as passing Dunbar's number means support will probably suck or border on non-existant, pre-sales questions will go unanswered or get a process driven irrelevant answer, invoice payment gets ever more lax, and so on.
Or maybe it's that companies smaller than this still have enough people around that care.
What’s funny is that it is the attitude most people adopt in these organisations but not those who are the most successful at climbing the ladder, which you can’t really do without some boldness, bending or challenging the rules, etc, that’s how you get things done and getting things done is a quality that managenent likes.
But what is perverse is that the whole organisation is designed to prevent that. Distributing responsabilities among many people, none of who understand the whole process, all of who are just trying to apply some policies with no understanding of what these policies are meant for, and be ready to kill the company if the policy says so.
I work in one of these kafkaesque places on the business side, I remember a project manager from IT once telling me that they need to make a change mandated by the compliance department that makes absolutely no sense and would have prevented us from serving our clients. I challenge IT, no we don’t want to go to jail, it has to be done. I go to the compliance department, explain why it is a moronic decision (with a bit more tact), they agree, and reverse the policy.
And I have seen that done many many times. So many people in these orgs are paid to say no, to resist any change, or simply do not care or understand. One needs to plow through, which requires a huge amount of energy, and results in most project being half baked, over expensive or simply never getting anywhere.
That’s why large organisations, populated by otherwise smart, well educated people generate so much mediocrity.
I'm not sure what you're saying went wrong in that process. You were the one who wanted a compliance mandate reversed, so who should have talked to the compliance department about it if not you?
Rather that IT would have implemented a change even if they thought it didn't make any sense just out of fear of falling foul of some internal policy. What I say is that large organisation are pleagued by people doing things that are counterproductive either because they don't care, don't understand, or live under the fear of breaching some internal policy, instead of doing the right thing, challenging these policies when it makes sense.
You're not wrong, but I think you're looking at it backwards. Policies are just codified decisions - and large organizations make a lot of decisions. If everyone second-guessed every decision they didn't personally understand, there'd be no time left to get work done.
There should have been push-back on the change before it got to the implementation stage. It shouldn't be down to the implementor to query the decision (as they won't have full sight of all factors). That querying should have happened at the management level.
When a nonsensical requirement reaches implementation, it indicates that all the gatekeeping in place before this stage has failed.
It's important not to so thoroughly explain the problems that big companies have that you argue them right out of existence, because they clearly exist and as others point out, actually routinely pull off feats that no startup could dream of.
I was wondering if this is happening only in my company. I am frustrated and outraged that there is no easy way to find out who is responsible for X, nor what is person Y responsible for. I wrote and email to our CIO, explaining my struggle, but he does not seem to care.
Scanning over responses so far I think a relevant aspect is not underlined strongly enough -- the lack of expertise in some of the key positions in companies "to move things". The status quo prevails. Play it safe, don't shake the boat, bring in Gartner or some such consultant on strategy, and hum along. In the end, outsource a project with internal fanfare (bringing relief to people concerned). This happens all too often.
This is how life goes, and as some here pointed out already, even if large inefficiencies are unaddressed a company can be profitable for significant time because of an established market position, brand, extensive regulations in place to protect their interests, some little known high profit products, etc. However give it time, and we see them fade away when upstarts take the risks, have the vision, shake up their market one piece at a time. And the cycle repeats.
Risk-aversion, cost of relatively small changes can look large, and the tendency to "design by committee". You can send an email out to some colleagues throughout other units of the business and get inundated with reasons why it won't work, and edge-cases you need to handle, whereas if you'd just implemented it, they'd likely work-around or improve their processes and everyone would have been better off.
For me the #1 reason why they are slow is because everything requires consensus or maybe another way of putting it it's hard for people to make a decision on their own.
Let's say I want to change some part of the UI of some app. If I'm by myself I just change it. If I'm on a team first I have to convince whoever has the power to decide to change it to allow it to be changed, that might even be a whole committee. Then several people will be involved, likely a UX designer, possible a separate UI designer, all of which will have to discuss the feature and all of which also have other responsibilities.
The larger the company the worse this usually gets. Maybe the change has to be run by the security team to make sure the new UI is not a security issue. Maybe it has to run by the privacy team to make sure it's not going to leak data. Maybe it has to be run by an inclusivity team to make sure it's not offensive in some culture I'm not familiar with. Maybe it has to be run by a localization team to make sure it will work in every language.
All of those things add time. Maybe they are all required and good, especially at a large company with a global marketplace but they certainly add to the time taken to do something that a tiny team wouldn't do.
A lot of answers to these questions, and the questions themselves, come from further down the food chain.
The flip side of the coin is that no start-up is moving a billion tons of rice, or can park an army off a foreign shore in a week, or lay 10,000 tons of steel rail across a continent every day, or delivery a million packages a day, or make 10,000 processors a day, each with the complexity of the entire European system of roads.
So speed is relative to the size of the task at hand. Large enterprise can move big things so fast it literally takes the air out of the room.
> So speed is relative to the size of the task at hand.
I think of organisational decision making like organic decision making - that the speed of the organisation's reactions are dependent upon its frame of reference. The frame of reference here is the time between sensing and reacting.
The eyes and ears of small businesses are very close to the brains, and so decisions can be made rapidly by the small number of people who have capability to marshal resources into a response.
Enterprises have layers of processing between the eyes and brains, along with many more sensors than a small business, meaning that questions take longer to be settled and decisions made. However no small business can wield the same power as a large organisation when the army of deciders all focus on a single outcome.
You're missing one piece which degrades performance of all corporations over time - bureaucracy with politics. I don't mean the sane one, I mean processes for the sake of processes, people entrenched in their comfortable managerial positions, battling any change that doesn't benefit them directly. I mean brutal management pyramids, so often seen in older corporations or banks for example.
It comes naturally over time, every single time. Organization can fight it, but it needs to be a dedicated effort and priority. If let on its own, complexity arises.
I've seen medium-sized young companies trying hard to have flat hierarchy and least amount of processes, committees etc. It worked. It really did. But everybody understood this is the goal, it was clearly communicated. It attracted amazing talent of brilliant people fed up with the (common) opposite.
I can see now in oldish bank how a simple change might involve 20 people having their say, often managers with their own political agenda not always aligned with what's best for the employer. Stuff that gets actually developed in 2-3 MDs takes overall easily 10-20x more because of this.
That's gross, but the overall organization is still efficient (or it has some sort of monopoly or huge moat). Otherwise it would be completely out of business by now.
It's efficient at it's stated tasks. Usually this is tied to stock prices.
I'm working at a Fortune 500 currently that does a lot of chemical manufacturing. Outside of delivering boatloads of chems, everything in the company is a constant clusterfuck. Stock price stays alright so nothing changes.
I think this is some kind of law: Every company is good at only those things where external pressure is high, and usually those are sales and whatever they are selling. Everything else has little external pressure to work well, and because the natural state of things is suckage and stagnation, everything else sucks and is stagnant.
So much this. It gets so bad that at some of these places, there are folks whose real value seems to be in "knowing the right people" in the org to who get certain things done rather than any real technical/managerial skills.
Knowing the right people (and its corollary, knowing how to find the right people) are critical managerial skills. I've seen otherwise great managers struggle to get anything done, because they keep trying to push on people who can't enact the changes they want.
It goes far more beyond that. We have an IT team that makes decisions without going through the consensus route.
I strongly believe the agility of an organisation can be addressed with:
- number of projects/initiatives worked on
- accountability of said projects/initiatives
- technology to facilitate the automation of admin to track these projects/initiatives
- collaboration tools for knowledge sharing
- mindset shift
And this is not something for just the management team or the employees but a collective agreement on this is how we work. It goes back to building and commuting to maintaining a high performing culture. It needs buy-in from everyone on what the expectations are and what the rules of engagement should be. Any sign of politics, bureaucracy? Management needs to snuff it out. The more we accept certain behaviours, the more we allow it to fester within the organisation.
Too many projects being worked on just causes backlog and confusion. We did have an agile coach to come in but the organisation remains the same.
Even if you manage to go through all that, you still need it to be accepted by the product owner and prioritized into a sprint. Good luck fighting for your feature, when account managers, designers and other "competitors" are doing the same. You're just a developer, what do you know about how the application you work on daily can be improved... They have hired other people for that.
Sounds like you need to find a new job. In my organisation, and others I've worked with, input from developers is treated roughly equally with input from other stakeholders.
If anything, developers seem to get angsty when the organisation doesn't bend to do whatever they feel is most necessary immediately, failing to realise that there are also other priorities coming from other parts of the business.
Also the default state for most people for any given change is ignorance / no reason to change. Thus the default "no".
At a non coding job in the past, I've had to explain the same change to five different people or teams in dramatically different ways to get a job done sometimes. If you saw me talk to these different groups you might not know I was even talking about the same thing each time....but I was.
Others get a say in the enterprise because your decisions affect others. The larger the scope, the more people your decisions affect.
Ever seen the login system for a multimillion dollar app broken because someone was able to end-run around the slow boring stupid release process and just shove a little UI change in production? I have. And when the entire system of lead engineers is out trying to figure out what got broken and how it got broken, and managers are trying to get the information they need to explain to their managers why the company just got publicly embarrassed and lost thousands of dollars, their response is not going to be sympathy at your struggles with being slowed down by all this stupid consensus bureaucracy.
Speaking as someone big-co track record... that seems like a bit of a strawman though, doesn't it? Not every team, in fact, most teams are not going to be the pivotal shared component behind billion dollar pipelines. Enterprise internal policies re: things like accessibility and security review are _hardly_ reflecting of that, since it's impossible to be fine grained and accurate, so they typically apply "a big hammer" across most groups and "an even bigger hammer" across others.
This is close to heart as I've grown a few teams from "small green-field experimental 3 people groups" to full on 8-10 person launched feature teams, and watched the process slow to a crawl despite the additional manpower. There's definitely a _reason_ for some of the bigco paranoia, but being flexible and accomadative in this sense is certainly not one of our strong points, so you usually get a decent amount of institutional boilerplate/plumbing work too, _especially_ when some of these side groups (e.g. UI) have a vested interest in having their fingers in as many pies as possible.
A somewhat tongue and cheek but accurate picture to draw: I've seen a dropdown menu go from being "code it up before lunch" to "3 weeks of meetings, multiple reviews, and demos in front of stakeholders" task. Were some parts of that useful? Sure. All of it? Probably not.
(To be clear; I don't actually take issue with much of the conceptual process, and like that there's some amount of oversight, but the nature in which it gets placed on teams is often 1. often not fully aligned with their actual "threat model" and prioritization in whatever space this is and 2. often given with inopportune timings and without a good support system. I call out the negatives above pretty exclusively because I think a bigCo _could_ do the processi-fication with a lighter touch and serve the same ends.)
It's like Russian roulette. Sure, five chambers out of six are empty and won't hurt you at all. But you still might want to slow it down with a process to make sure there isn't a round in the chamber.
Someone, can't remember who (and I wish I did!) said "Process is the scar tissue of organizations". These processes don't come from nowhere. They come from specific negative experiences the organization had.
The problem is that it's easier to add process than remove it or clean it up so it isn't so stressful. So once process is really slowing things to a crawl, it's time to clean up, but it's a real chore.
This is my career, basically. I come in to big organizations with big legacy messes to try to make things easier. Reducing the headaches of awkward and often unnecessary process is close to my heart. But a big part of that is recognizing the necessity of process. There's a huge difference between "this is frustrating" and "this is worthless".
Yeah, Congruence is kind of dead in the water right now. I should fix that, but it's going to be a while before I have the time to tackle the project again.
My current work is as an ordinary devops engineer at a very, very large company.
I do want to go after the problem space again (mostly, helping find the problems caused by interdependencies of complex, slow-changing systems - "System A changed, so System B broke"). But it's a seriously nontrivial problem, which is why there aren't any good products out there for it.
I work in a muniplacity of 10.000+ employees, and network with other managers from 97 other muniplacities as well as the government, and I think most of those points are wrong. At least in enterprise that aren’t IT companies.
There aren’t really regulators the way it’s described, and you wouldn’t worry about them the way it’s described either. Unless you made changes without planning. What you would consider when upgrading your OS, for instance, is how you teach the new OS to the 5000 employees who can’t tell the difference between and iPhone and an android tablet.
Ownership is easy in enterprise, we have organized everything into departments and IT operations is owned by IT. It’s audited that way, and things are changed according to the guidelines.
You’d literally never have an audit come in and point any of the things suggested out, unless your enterprise isn’t run like an enterprise. The first step in the manual for upgrading, is to make sure you stay within the law and within company policy, and you’d never run a project all the way through without involving people who know regulations.
I don’t think enterprise is risk averse either, but I do think you need to be able to prove the benefits of a change, to be allowed the cost of it. You don’t in a startup, because startups aren’t managed very well because they can adopt, but I don’t think that’s necessarily an advantage. A lot of changes in IT are unnecessary, costly and done without any benefit realization.
I don’t think enterprise is slow either. We’re on the forefront on how to use stuff like ADFS and identity management, to the point where we’ve had to sent the countries leading consultants home because they where total newbies compared to our guys. I just think we’re only fast where it matters.
We still use stuff like LinqToSql in production for instance, which is a hopelessly old tech to a lot of people, and it’s also slow and inefficient. Except, when you look at it from an organizational point of view, it allows our developers to do their SQL integrations around 90% faster, and we rarely have more than 60.000 continuous users, so the efficiency never matters. The cost benefits of using LinqToSql is crazy high because developer time is expensive.
I don’t think a startup would use it, but I also don’t know of a single startup that doesn’t need to be explained how our entire public service does identity management because they are completely incompetent on matters they can’t look up on stackexchange.
Maybe it’s different in America, but I kind of doubt it, because the focus of enterprise will always be on value generation and benefit realization, where as the focus in startups can be almost anything in the world.
I have learned that these "slow" traits are all some form of safety/security system. They're in place to avoid corruption, single point of failure, bias, etc. and try to modularize/compartmentize the risk. Most of them are terribly designed and often working against its original purpose, but nonetheless they're there. Building a big enterprise that functions properly is just as hard as building a large software (if not harder), and upgrading a small organization to a big one without a problem is nearly impossible, again just like software.
Nah, it's because of the conjunction of two different factors:
First, changing anything large and complex with a lot of moving parts is difficult.
You can see the exact same effect in a code base as an organisation.
If your code base is small, making a change has limited consequences and you can usually predict them all up front.
If your code base is large, you lose the ability to predict what the consequences of a change are. So you need to do it to find out. If you don't have good test suites, with excellent regression tests, then the chances of missing a breaking change are good, and "doing it to find out" will result in a broken system.
Organisations don't have test suites, or regression tests. So there's a lot of effort done up front to try to minimise the impact of changes. A lot more work involved in even the smallest changes.
Secondly, small organisations are optimised for flexibility, large organisations are optimised for efficiency.
A two-person company needs to be flexible and adaptable. In startup terms, they probable haven't hit product-market fit, and are still developing their product, changing it to meet the market as they discover what the market needs. Change is constant, and so the organisation needs to be able to change fast. Doing things efficiently is premature, because the efficient process may change tomorrow and become unefficient.
Large organisations have found a business model and grown to fill it. They know exactly what they're making, for whom, for how much, and have a well-defined product and market. All the bottom-line growth will come from reducing costs rather than increasing sales. So they focus on reducing costs. But there's a trade-off between flexibility and efficiency. As the organisation becomes more efficient and more cost-effective, it also becomes less flexible. At the apex of this process, the organisation is so streamlined that any change (by definition) becomes less efficient and therefore less profitable, and therefore any change is difficult and unwanted.
I remember some "law" about communication in a "network" think its a power-law-distributions as to how bigger the network (organization) the longer it takes to communicate/relay a msg. And that's just in the best case scenarios. Then we start talking about ego and ppl and all odds are off. Unless you working on the production line... fewer ppl are "usually" better!
> In order to ensure that changes to systems can be attributed to responsible individuals, there is usually some kind of system that tracks and audits changes. One person will raise a ‘change record’, which will usually involve filling out an enormous form, and then this change must be ‘signed off’ by one or more other person to ensure that changes don’t happen without due oversight.
And in any other planet, we call that a 'Pull Request.' My first real tech job was a community college that really bought into the whole ITIL framework. Important changes typically had to go through a Change Management Board, which met weekly. Meanwhile, key authentication systems involved passing passwords from PHP to perl to bash to vbscript in cleartext, in such a way that dollar signs and other string interpolation sigils would be processed, and therefore were banned. The person who wrote this kludge is now in charge of IT security for the college. And there was no version control to speak of anywhere, definitely no puppet or chef or ansible. It worked, but there were pretty much monthly fuckups along the lines of 'and then the utility truck backed into our power distribution cabinet' or 'the SAN vendor's technician mentioned this is the third time this week he's been on site with a client to deploy this emergency stability patch,' or 'the new guy upgraded the antivirus running on our databases, and we can't roll back because nobody has the old installer anymore' or my personal favorite: 'this position requires oncall duties 24/7/365.'
My company uses ITIL. Definitely a four letter word. I feel like Peter Gibbons much of the time. For me to deploy a server, I need to create a Change Request, have it reviewed by my local Change Review Board. They know nothing, and are a rubber stamp. Next it goes the the Main Change Review Board. They are higher up, and know even less. I have to have my CR into the LCRB a week before they meet, and then it takes an additional week for the MCRB to approve it. So at least two weeks to deploy a new VM. Retirement can't come soon enough.
None of these things are insurmountable, your organization just has to be willing to try.
For example, Dev and Ops don't have to be separate, even in a regulated world. You just have to make the effort to show your auditors that there are sufficient controls in place to make changes safe. You also have to work to make sure as much change is out of scope as possible. Ok, maybe the payments system requires that one person writes code and one person deploys it, but if you make sure that almost all of the payment system is out of scope except for the actual money processing, it will be rare that you need to change that one piece of code, so effectively you've avoided the problem.
Same with change control -- you don't have to have a big onerous form. You can build a system that automatically logs all the relevant information as to who made a change, so that you have a log that you can show your auditors or whomever needs to check. Again though, your organization has to be willing to spend time on the tooling, but the tooling scales better than a new process.
And then for security, again, you don't necessarily need sign off on every change. If you build automated auditing systems that audit every change for security implications, and have sufficient logging, you can be reasonably sure that the changes are secure, and then only dig deeper on the ones that are suspect, or get people to agree to seek approval for things that are big changes or might cause a security problem.
If your organization is willing to spend the time building the software and working with the auditors to explain how everything works, you can avoid most of these problems, and then your enterprise will scale much better.
135 comments
[ 3.3 ms ] story [ 173 ms ] threadReally well written piece that clearly explains the reality of "Enterprise DevOps".
If a corporate lawyer ever ambushes you with this, the best thing to do is ask instead how you can make your current process auditable, rather than tearing the org apart. If that doesn't work, start looking for a more innovative place to work!
When you're small, it's easy to avoid regulatory attention and whatever slap-on-the-wrist fine you may get isn't a lot of money.
The change and risk aversion really just comes from keeping the business running. When things work, try not to break them. Old large enterprises tend to have a lot of business critical things that "just work" and have been for a long time. Younger/smaller companies tend to have fewer battle scarred systems.
But once they get scarred, they typically add a process or policy to prevent it from happening again. After enough of them, congrats, you're enterprise speed.
It's like microservices, but with systems of people and businesses.
partners that allow for some agile, iterative back and forth becomes rarer as they increase in size, especially since often their own system lack the flexibility for continuous integration or safely rolling back changes/data.
that's why you get requirements, specifications, architects and solution managers in the enterprises. people love to mock them for being dysfunctional and some truly are, but once contracts get into the millions and lawyer get involved mistakes are no longer "learning opportunities".
The end result two years later is OS upgrades rarely break much, and regular updates never do. Enterprises can be unstuck but it can be painful.
I think enterprises are still shifting from the old approach where you make it work and then it's fine, to realizing that in an internet-connected world, you have to keep up at all times. A few big liability cases would do wonders to push this shift along.
Almost everytime we change versions of software things break when we went moved to office 365 there was breakage (Stuff I would have thought for sure would have been well tested - PST files in outlook for example broke).
Same thing when we went from Internet Explorer 8 to 11 our internal intranet didn't work.
I work in an environment that includes custom-written apps from the Windows XP era, and usually the absolute worst we hit was "hey, this device doesn't work on Windows 10, we have to buy a newer one". Not a big deal when it's a $6 USB-to-serial adapter, but people will look at you like you're crazy when you say the network is incompatible with an HVAC system. (Solution in that case, just take it off the network, you're better off.)
But believe it or not, you can still sync Windows Mobile 2003 devices with Windows 10 PCs, Microsoft's backwards compatibility is still insanely solid in most cases.
Yeah, I don't think that happens much.
Your point may be that people should go to prison for presiding over bad things that weren’t actually against the law. That path sounds tricky.
Many new features to fulfill customer needs is a common reason. Many customers, with new requests, issues reported. It's slow because if I'm not methodical I cause problems for people who don't want this next enhancements. It's slow, because the current emergency takes precedence, and taking to long on fixes means a meeting on why something is behind.
Business priority is not "deliver fast" it's "keep running, don't attract attention".
Work at big companies is kind of like that. Once a company gets big enough, any given person doesn't really know anything. Seemingly no one is doing anything productive -- it's meetings all the way down. But the software still gets shipped. It doesn't happen the same way it happens at a startup where someone sits down and codes a feature. It happens in a totally different way, where one hundred engineers go to meetings all day and talk to QA and analytics and bizdev, and maybe commit one line of broken code a week. It takes way more people, and the production process is totally different, and it takes way longer, but stuff still happens (often quite successfully).
[1] https://what-if.xkcd.com/46/
From xkcd:
> On that scale, the polymers in the bowling ball would behave more like a liquid
So polymers, not metal.
--
[0] - https://en.wikipedia.org/wiki/Planet#2006_IAU_definition_of_...
lol please tell me you invented this
A starup has nothing to lose. The site is janky for two hours? The only users are friends, and the outage might actually bring more visibilty if you make a good write-up.
Incidentally, this ties together with “why do young people found more startups than senior executives”
A good book is The Innovators Dilemma: https://en.m.wikipedia.org/wiki/The_Innovator%27s_Dilemma
Lots of smart, talented and energetic people are expending most of their power just figuring out who to talk to about what or what the hell that out-of-the-blue email from corporate finance means for them.
Just figuring out how to get a thing done can be exhausting.
Half-jokingly, as soon as an "individual contributor" knows enough to be productive, they're promoted to management.
I was heavily invested in our public facing technical side, working a lot with our on-boarding, teaching newbies to be really self-sufficient, working through the technical issues with our clients directly, and then I was asked to take on a Management position.
I liked the pay raise, but I made it clear exactly what I wanted from the position, and though we butt heads on it still, my boss respects the fact that I still am heavily invested with our technical team while doing the Managerial work. We had to revisit the job definition a few times to find a description that matched what I was (am) doing and also made HR happy, but we did it.
The point is that in some cases at least, a promotion to management doesn't have to be a death knell for "individual contributors". It depends on the company's willingness to accommodate the wants of the individual, and the individual's own desire to continue contributing. I know too many people who did get the Management golden ticket and relished it for just being an excuse to boss people around and make more dashboards in our CRM.
I wonder how much of it has to do with team sizes. I feel like a 4 person team (inclusive) would be ideal for a manager who also wanted to act as a technical lead and contribute a moderate amount. Any bigger, and the scope of what your team does, as well as the amount of time needed to devote enough time to all your employees, starts to get too big for you to contribute at a technical (e.g. source code) level. Yet we continue to take great technical workers and put them in this weird PM-hiring-HR-architect mixture of a position where they barely continue to leverage their skills anyway
In the context of a startup, if you try something and fail, you'll generally be recognized for taking initiative even if the result does not pan out. At that level, learning what doesn't work is often as valuable as learning what does work. Startups often bias towards hiring people with just this mentality because that is what they actually need.
In the context of an established company, sticking your neck out is like career suicide unless it's a sure thing. Unlike a startup, you're risking an _established_ reputation on a new idea. So you end up working in an organization that has an inherently different view of risk and trying new things.
I'm not sure whether "nobody knows how to do anything" results from an institutional aversion to risk or from the natural result of hiring people that fit within that system, but the end effect is exactly as you describe. It's way too hard to do anything off of the beaten path, and trying to do anything along those lines _immediately_ results in negative feedback which makes employees a lot less likely to attempt such actions in the future.
For example, I can build a chatbot in a day that will take some load of a bunch of people in my department. Will it be deployed? No way. Will it be seen as a possible threat to people's patronage networks by cutting their teams and budgets? You bet. Will I possibly be chastised for doing something that is not in my JD? Possibly. Will I bother making it in this case? Nope.
To customers it really does seem as simple as passing Dunbar's number means support will probably suck or border on non-existant, pre-sales questions will go unanswered or get a process driven irrelevant answer, invoice payment gets ever more lax, and so on.
Or maybe it's that companies smaller than this still have enough people around that care.
But what is perverse is that the whole organisation is designed to prevent that. Distributing responsabilities among many people, none of who understand the whole process, all of who are just trying to apply some policies with no understanding of what these policies are meant for, and be ready to kill the company if the policy says so.
I work in one of these kafkaesque places on the business side, I remember a project manager from IT once telling me that they need to make a change mandated by the compliance department that makes absolutely no sense and would have prevented us from serving our clients. I challenge IT, no we don’t want to go to jail, it has to be done. I go to the compliance department, explain why it is a moronic decision (with a bit more tact), they agree, and reverse the policy.
And I have seen that done many many times. So many people in these orgs are paid to say no, to resist any change, or simply do not care or understand. One needs to plow through, which requires a huge amount of energy, and results in most project being half baked, over expensive or simply never getting anywhere.
That’s why large organisations, populated by otherwise smart, well educated people generate so much mediocrity.
When a nonsensical requirement reaches implementation, it indicates that all the gatekeeping in place before this stage has failed.
It's important not to so thoroughly explain the problems that big companies have that you argue them right out of existence, because they clearly exist and as others point out, actually routinely pull off feats that no startup could dream of.
This is how life goes, and as some here pointed out already, even if large inefficiencies are unaddressed a company can be profitable for significant time because of an established market position, brand, extensive regulations in place to protect their interests, some little known high profit products, etc. However give it time, and we see them fade away when upstarts take the risks, have the vision, shake up their market one piece at a time. And the cycle repeats.
Let's say I want to change some part of the UI of some app. If I'm by myself I just change it. If I'm on a team first I have to convince whoever has the power to decide to change it to allow it to be changed, that might even be a whole committee. Then several people will be involved, likely a UX designer, possible a separate UI designer, all of which will have to discuss the feature and all of which also have other responsibilities.
The larger the company the worse this usually gets. Maybe the change has to be run by the security team to make sure the new UI is not a security issue. Maybe it has to run by the privacy team to make sure it's not going to leak data. Maybe it has to be run by an inclusivity team to make sure it's not offensive in some culture I'm not familiar with. Maybe it has to be run by a localization team to make sure it will work in every language.
All of those things add time. Maybe they are all required and good, especially at a large company with a global marketplace but they certainly add to the time taken to do something that a tiny team wouldn't do.
The flip side of the coin is that no start-up is moving a billion tons of rice, or can park an army off a foreign shore in a week, or lay 10,000 tons of steel rail across a continent every day, or delivery a million packages a day, or make 10,000 processors a day, each with the complexity of the entire European system of roads.
So speed is relative to the size of the task at hand. Large enterprise can move big things so fast it literally takes the air out of the room.
I think of organisational decision making like organic decision making - that the speed of the organisation's reactions are dependent upon its frame of reference. The frame of reference here is the time between sensing and reacting.
The eyes and ears of small businesses are very close to the brains, and so decisions can be made rapidly by the small number of people who have capability to marshal resources into a response. Enterprises have layers of processing between the eyes and brains, along with many more sensors than a small business, meaning that questions take longer to be settled and decisions made. However no small business can wield the same power as a large organisation when the army of deciders all focus on a single outcome.
https://en.m.wikipedia.org/wiki/Newton%27s_laws_of_motion
It comes naturally over time, every single time. Organization can fight it, but it needs to be a dedicated effort and priority. If let on its own, complexity arises.
I've seen medium-sized young companies trying hard to have flat hierarchy and least amount of processes, committees etc. It worked. It really did. But everybody understood this is the goal, it was clearly communicated. It attracted amazing talent of brilliant people fed up with the (common) opposite.
I can see now in oldish bank how a simple change might involve 20 people having their say, often managers with their own political agenda not always aligned with what's best for the employer. Stuff that gets actually developed in 2-3 MDs takes overall easily 10-20x more because of this.
I'm working at a Fortune 500 currently that does a lot of chemical manufacturing. Outside of delivering boatloads of chems, everything in the company is a constant clusterfuck. Stock price stays alright so nothing changes.
So much this. It gets so bad that at some of these places, there are folks whose real value seems to be in "knowing the right people" in the org to who get certain things done rather than any real technical/managerial skills.
I strongly believe the agility of an organisation can be addressed with:
- number of projects/initiatives worked on - accountability of said projects/initiatives - technology to facilitate the automation of admin to track these projects/initiatives - collaboration tools for knowledge sharing - mindset shift
And this is not something for just the management team or the employees but a collective agreement on this is how we work. It goes back to building and commuting to maintaining a high performing culture. It needs buy-in from everyone on what the expectations are and what the rules of engagement should be. Any sign of politics, bureaucracy? Management needs to snuff it out. The more we accept certain behaviours, the more we allow it to fester within the organisation.
Too many projects being worked on just causes backlog and confusion. We did have an agile coach to come in but the organisation remains the same.
If anything, developers seem to get angsty when the organisation doesn't bend to do whatever they feel is most necessary immediately, failing to realise that there are also other priorities coming from other parts of the business.
And the fallacy of popularity bringing good solutions.
Saying no is safer than saying “Yes.”
When only a single “No” will halt a project, the chance of getting stuck is higher.
At a non coding job in the past, I've had to explain the same change to five different people or teams in dramatically different ways to get a job done sometimes. If you saw me talk to these different groups you might not know I was even talking about the same thing each time....but I was.
Ever seen the login system for a multimillion dollar app broken because someone was able to end-run around the slow boring stupid release process and just shove a little UI change in production? I have. And when the entire system of lead engineers is out trying to figure out what got broken and how it got broken, and managers are trying to get the information they need to explain to their managers why the company just got publicly embarrassed and lost thousands of dollars, their response is not going to be sympathy at your struggles with being slowed down by all this stupid consensus bureaucracy.
This is close to heart as I've grown a few teams from "small green-field experimental 3 people groups" to full on 8-10 person launched feature teams, and watched the process slow to a crawl despite the additional manpower. There's definitely a _reason_ for some of the bigco paranoia, but being flexible and accomadative in this sense is certainly not one of our strong points, so you usually get a decent amount of institutional boilerplate/plumbing work too, _especially_ when some of these side groups (e.g. UI) have a vested interest in having their fingers in as many pies as possible.
A somewhat tongue and cheek but accurate picture to draw: I've seen a dropdown menu go from being "code it up before lunch" to "3 weeks of meetings, multiple reviews, and demos in front of stakeholders" task. Were some parts of that useful? Sure. All of it? Probably not.
(To be clear; I don't actually take issue with much of the conceptual process, and like that there's some amount of oversight, but the nature in which it gets placed on teams is often 1. often not fully aligned with their actual "threat model" and prioritization in whatever space this is and 2. often given with inopportune timings and without a good support system. I call out the negatives above pretty exclusively because I think a bigCo _could_ do the processi-fication with a lighter touch and serve the same ends.)
Someone, can't remember who (and I wish I did!) said "Process is the scar tissue of organizations". These processes don't come from nowhere. They come from specific negative experiences the organization had.
The problem is that it's easier to add process than remove it or clean it up so it isn't so stressful. So once process is really slowing things to a crawl, it's time to clean up, but it's a real chore.
This is my career, basically. I come in to big organizations with big legacy messes to try to make things easier. Reducing the headaches of awkward and often unnecessary process is close to my heart. But a big part of that is recognizing the necessity of process. There's a huge difference between "this is frustrating" and "this is worthless".
Is your current business - http://congruence.io ?
It redirects to https://congruence.io with invalid SSL certificate...
My current work is as an ordinary devops engineer at a very, very large company.
I do want to go after the problem space again (mostly, helping find the problems caused by interdependencies of complex, slow-changing systems - "System A changed, so System B broke"). But it's a seriously nontrivial problem, which is why there aren't any good products out there for it.
There aren’t really regulators the way it’s described, and you wouldn’t worry about them the way it’s described either. Unless you made changes without planning. What you would consider when upgrading your OS, for instance, is how you teach the new OS to the 5000 employees who can’t tell the difference between and iPhone and an android tablet.
Ownership is easy in enterprise, we have organized everything into departments and IT operations is owned by IT. It’s audited that way, and things are changed according to the guidelines.
You’d literally never have an audit come in and point any of the things suggested out, unless your enterprise isn’t run like an enterprise. The first step in the manual for upgrading, is to make sure you stay within the law and within company policy, and you’d never run a project all the way through without involving people who know regulations.
I don’t think enterprise is risk averse either, but I do think you need to be able to prove the benefits of a change, to be allowed the cost of it. You don’t in a startup, because startups aren’t managed very well because they can adopt, but I don’t think that’s necessarily an advantage. A lot of changes in IT are unnecessary, costly and done without any benefit realization.
I don’t think enterprise is slow either. We’re on the forefront on how to use stuff like ADFS and identity management, to the point where we’ve had to sent the countries leading consultants home because they where total newbies compared to our guys. I just think we’re only fast where it matters.
We still use stuff like LinqToSql in production for instance, which is a hopelessly old tech to a lot of people, and it’s also slow and inefficient. Except, when you look at it from an organizational point of view, it allows our developers to do their SQL integrations around 90% faster, and we rarely have more than 60.000 continuous users, so the efficiency never matters. The cost benefits of using LinqToSql is crazy high because developer time is expensive.
I don’t think a startup would use it, but I also don’t know of a single startup that doesn’t need to be explained how our entire public service does identity management because they are completely incompetent on matters they can’t look up on stackexchange.
Maybe it’s different in America, but I kind of doubt it, because the focus of enterprise will always be on value generation and benefit realization, where as the focus in startups can be almost anything in the world.
There's a reason we call subroutines "procedures".
First, changing anything large and complex with a lot of moving parts is difficult.
You can see the exact same effect in a code base as an organisation.
If your code base is small, making a change has limited consequences and you can usually predict them all up front.
If your code base is large, you lose the ability to predict what the consequences of a change are. So you need to do it to find out. If you don't have good test suites, with excellent regression tests, then the chances of missing a breaking change are good, and "doing it to find out" will result in a broken system.
Organisations don't have test suites, or regression tests. So there's a lot of effort done up front to try to minimise the impact of changes. A lot more work involved in even the smallest changes.
Secondly, small organisations are optimised for flexibility, large organisations are optimised for efficiency.
A two-person company needs to be flexible and adaptable. In startup terms, they probable haven't hit product-market fit, and are still developing their product, changing it to meet the market as they discover what the market needs. Change is constant, and so the organisation needs to be able to change fast. Doing things efficiently is premature, because the efficient process may change tomorrow and become unefficient.
Large organisations have found a business model and grown to fill it. They know exactly what they're making, for whom, for how much, and have a well-defined product and market. All the bottom-line growth will come from reducing costs rather than increasing sales. So they focus on reducing costs. But there's a trade-off between flexibility and efficiency. As the organisation becomes more efficient and more cost-effective, it also becomes less flexible. At the apex of this process, the organisation is so streamlined that any change (by definition) becomes less efficient and therefore less profitable, and therefore any change is difficult and unwanted.
Big companies are slow for the same reason.
And in any other planet, we call that a 'Pull Request.' My first real tech job was a community college that really bought into the whole ITIL framework. Important changes typically had to go through a Change Management Board, which met weekly. Meanwhile, key authentication systems involved passing passwords from PHP to perl to bash to vbscript in cleartext, in such a way that dollar signs and other string interpolation sigils would be processed, and therefore were banned. The person who wrote this kludge is now in charge of IT security for the college. And there was no version control to speak of anywhere, definitely no puppet or chef or ansible. It worked, but there were pretty much monthly fuckups along the lines of 'and then the utility truck backed into our power distribution cabinet' or 'the SAN vendor's technician mentioned this is the third time this week he's been on site with a client to deploy this emergency stability patch,' or 'the new guy upgraded the antivirus running on our databases, and we can't roll back because nobody has the old installer anymore' or my personal favorite: 'this position requires oncall duties 24/7/365.'
For example, Dev and Ops don't have to be separate, even in a regulated world. You just have to make the effort to show your auditors that there are sufficient controls in place to make changes safe. You also have to work to make sure as much change is out of scope as possible. Ok, maybe the payments system requires that one person writes code and one person deploys it, but if you make sure that almost all of the payment system is out of scope except for the actual money processing, it will be rare that you need to change that one piece of code, so effectively you've avoided the problem.
Same with change control -- you don't have to have a big onerous form. You can build a system that automatically logs all the relevant information as to who made a change, so that you have a log that you can show your auditors or whomever needs to check. Again though, your organization has to be willing to spend time on the tooling, but the tooling scales better than a new process.
And then for security, again, you don't necessarily need sign off on every change. If you build automated auditing systems that audit every change for security implications, and have sufficient logging, you can be reasonably sure that the changes are secure, and then only dig deeper on the ones that are suspect, or get people to agree to seek approval for things that are big changes or might cause a security problem.
If your organization is willing to spend the time building the software and working with the auditors to explain how everything works, you can avoid most of these problems, and then your enterprise will scale much better.