> "So what’s the solution?
To reduce the software delivery gap, mainstream enterprises need to raise their game by transforming their own internal capability."
[...]
> "We founded Contino so that we could instead transform through delivery."
[...]
> "Benjamin Wootton is the Co-Founder and CTO, EMEA of Contino."
It's a corporate blog. Describing our view of the world and our approach and experience in solving it. Pick a hundred blog posts and they will follow the same format.
Of course it's partly marketing but it doesn't detract from the argument.
It's a sales pitch by a consulting shop looking for work with corporate clients. I find it odd that Benjamin Wootton posted his own marketing content here on HN. His message should be directed at senior corporate IT managers - decision makers with budget & spending authority who can take on a load of contino.io staff on juicy day rates. Those people who are unlikely to be reading HN.
Why are digital leaders (Facebook, Google, Amazon and friends) ten times better at rapidly delivering software than mainstream enterprise organisations?
That's the first sentence, hence the premise of the article. Don't you need to provide some supporting evidence for those figures?
Experience, again, suggests not. The digital leaders themselves are behemoths! Facebook has 1.19 billion active monthly users. Keeping them happy is quite a task, yet they manage to innovate alongside keeping the lights on. So how come enterprises are still so slow? Why do their technological efforts always seem to cost much more than they should? Why does the software development gap persist?
Does not follow. Facebook has lots of users but they have a relatively small workforce compared to traditional big companies. Over some threshold it's probably easier to "satisfy" lots of users via network effects than it is to please a small but critical community, anyway, but that's just conjecture.
The article is a continued series of unsupported assertions and "facts".
Anyone who has been around enterprise IT would probably say that this is at least the right order of magnitude. A Spotify, Netflix, Google, Facebook can fly compared to a traditional enterprise
I think it depends on the enterprise I recall when I worked at a big telco in the UK a stat that we delivered 80% better than the average - I though "Huh company propaganda" - but when I worked for other Large companies I was shocked a how crap they where.
Then again my division was the direct heir to Tommy Flowers group and at the time had more engineers than Google had employees.
Have you worked at any of these companies recently? If not, how do you know they fly?
My observation from when I worked at Google was that the organisation was rapidly slowing down over time, and projects that were essentially pointless or duplicative were multiplying. Very simple tasks were starting to become extremely hard work. Much like a traditional large enterprise, in fact.
Exactly, it is largely cultural and the talent thing may be just much about the kinds of employees. At a big enterprise company people want to keep their head down, collect their bonus and justify their existence. At these cooler companies, young talent is willing to work 60+ or 80+ because they are at FB and love how cool that makes them.
Look at how Microsoft stalled for a lot of years but after a change of culture (and focus!) they had a monster year and are poised for more. Their talent makeup wasn't that different during the bad or good times. Also, the argument fails when you look at all of the failed projects, spin outs and copy cat business of FB and Google and Microsoft. If they are 10x better in their very core, why all the failures? Because it is so easy to get wrong. Lots of companies have no choice but to do what their customers pay them to do or go out of business. Lots of companies don't have the cash cows of web search or Office to allow for huge mistakes.
However there needs to be an intent by the "enterprise" to achieve some end goal, which is aligned with their overall business strategy, to operate their IT more like the agile technical rockstar companies.
Such transformation is costly and disruptive, so all of this has to be properly vetted in the context of their overall strategy.
The 10X developer concept is a bit flawed. If only because computer scientists and students understand that a linear co-efficient is borderline meaningless in measuring algorithmic performance, it seems to pander to the less-technically-literate side of the engineering pool.
Another problem - the developer who accomplishes tenfold productivity of his peers in one environment might be absolutely uncompetitive in another. For example, what's the difference between tuning performance in a critical backend path and building a fully functioning UI? A developer who is ten-times better at one might be incapable of completing the other.
Instead, especially with the whole full-stack movement, wouldn't it be better to look at qualifications, ability to learn, motivation, interest and attitude? Even looking for a "ninja developer" might be better if it means someone with an attitude to hack things together at any request. And looking for a seasoned developer might be better for maintaining and rebuilding one of those hacked-together platforms.
What's the difference between a 10X and X^2/10 developer? It's time for hirers and SE bloggers to see X, log X, X log X, and X^2 developers, who are all out there.
But I think that a 10x developer is one who knows all the ins and outs of the tools in his or her belt.
Who will prefer a 10-year-old framework that served him well over an opportunity to procrastinate by means of reinventing the wheel. (I'm looking at you, JavaScript-land)
And who doesn't fall into the "tower of babel"-trap, which is the believe that there's a universal language and a perfect way of writing a program or structuring a system.
To be fair, there are few, if any, 10-years-old JS frameworks that one can still be productive with.
Well, there's jQuery, which is still fine for very small things (an animation here, an AJAX call there), but is totally inadequate if you need to build a rich UI of the kind commonly expected now. You'll spend less time learning something like React or Vue and implementing things in it than you'd spend fighting with the callback hell and manually crafted state machines.
I know and use React (and React-Native) daily, but, except the views which are in JSX/es6/7, I like to write everything in java or kotlin which I can run in the browser.
There are a gazillion very useful Java-libraries out there.
I just implemented a ListView with complex filtering, live text-matching and sorting for an Android/React-native-app which has no problem displaying 100.000 items or more using Java-libraries.
React-Native's implementation on the other hand seems to be riddled with problems, judging from the bug-tracker and the confused API.
Every few years I feel there must be a universal language; every few years I spend a few weekends and nights writing one. A Lisp always comes out. So next craving I will try more Clojure to span both web and native mobile.
Until then I use 10+ year old tech to get work done.
10X what though? KLOC never worked and we've not developed any good, transferable, objective metrics for developer productivity. We're left with highly subjective peer evaluation. Which is very vulnerable to bias.
Most people will agree that "some people are a lot more effective than other people". But if you take the members of a team aside individually and ask them to rank the effectiveness of their colleagues, do you get compatible answers? Has anyone ever tried getting team A to look at the anonymised commits of team B for evaluation?
This doesn't match my experience at all. What I have seen over and over, for 20 years, is more like this: The local guru is asked to help out with an area that's falling behind, even though everybody knows it's not their thing. They start out awkward, self-deprecating, asking a lot of questions. Yes, they suddenly seem like a newbie. A week or two later, they check in something brilliant that works perfectly and blows the "specialist's" work out of the water.
This concept was never meant to be precise. It just expresses that some people's productivity is hard for others to understand and hard to duplicate.
I think the problem lies in treating work units identically, because they very much are not. One developer can barely code a CRUD page. Another can write an OS. Another can write a compiler for a novel language. Another may not have any flashy skills but can work well enough in a team and can learn.
It doesn't matter how many "barely can write a CRUD page" people you task to write your distributed cloud filesystem, it won't work.
(Before someone complains, because I know you, Internet, these classifications aren't permanent. I was "barely capable of writing a CRUD page" at one point, too. But you have to work with the developers that you have today, not what they could be in 20 years.)
10x seems to apply to a specific developer in a specific environment. I've seen corporate farms where hackers would suffer but idiots would thrive, and the exact opposite in startups. I've abandoned ship on gigs where the CTO would oppose migrating from SVN to git - but I have also saved sinking ships where the team was clueless but willing to change & learn. Reality is not a simple 1x/10x.
enterprises also suffer from organisational and process legacy
Indeed. It's possible to waste huge amounts of time simply trying to decide what to do and communicating with all the 'stakeholders'. If you want to change the way your organisation builds software products you have to change the organisation.
We've been experiencing a lot of entropy where I'm at. Everyone could feel it, no one could understand it. I started looking for what might be happening and what might fix it. I came across Lex Sisney and Organizational Physics. It makes a lot of sense, at least to me.
This is an ad, more than anything, and I don't think it even remotely addresses the issues it brings up. I do have to say, it is refreshing to have yet another empty slogan, transform through delivery, to use at work.
I think that is unfair having just re-read it. (I'm the author.)
90% of the article is about our theory that large enterprise organisations have a huge gulf when compared to digital leaders, and analysing why they are in this place.
To break out of it, they need to do things in a completely different way, more focussed on building their own capability than outsourcing work.
Again, you don't just "break out" and act like "digital leaders" just for the sake of it.
Why does a fortune 50 company need to do that? What is the ROI on such effort? What are their risks if their IT operation is not as cutting edge as Amazon's? Those are the core questions, which your article does not even mention.
I mean no disrespect, Ben. I am merely pointing out the fact that you bring up the issue of technical debt, for instance, but you don't delve into the root cause of it. Instead, you talk about how enterprises have technical debt vs. digital leaders who don't. However, the root cause of technical debt is not enterprise vs. digital leaders. It exists in both camps, and the reasons for it are largely political. So, how do you address that? How do you convince the owner of a company that creating technical debt to save a week now will almost surly blossom into the loss of months down the road?
Legacy technology is another issue I don't feel you fully addressed. For example, there are many companies that start new projects using legacy technology, or the wrong technology, again for political reasons. I've seen reasoning like "We're a Microsoft shop," and "It's easy to hire Ruby developers," as the main drivers of technology choice, rather than a thoughtful consideration of the requirements of a project, and what technologies are best suited to solve the problem at hand.
I appreciate the time and effort you put in to communicating your ideas, and do not mean to diminish from that at all. I do think if you added a little more meat and citations to your arguments, and got rid of some of the sales-pitching at the end, you could strengthen your article quite a bit though.
It's not really a theory, though. These are all well known facts, which is why there's already buzzwords for them (i.e. the ones you were using).
A good tech blog post would show how you're unique, how you're innovating, and how you're making real change for your customers (with stats to back it up).
I suggest thinking about a blogging strategy with at least 1-2 posts / month in 2017. Take a few hours to plan out what you're trying to achieve and who your audience will be... Clients? Potential sales / tech hires? Being a thought-leader? Think about what you want to communicate to them that's more personal than a brochure. That's the fundamentals of a good tech biz blog at your stage of blogging.
That an empty slogan like "transform through delivery" should seem meaningful speaks volumes about the sad state of enterprise IT. Alas, in my experience, there is often little delivery of new systems and features. Too many egos, careers and vested interests are invested in the status quo. Change brings disruption and the prospect of unhappy users. So ass covering political IT managers pretend to "risk manage" the enterprise IT portfolio by attending endless meetings at which they resist any change then claim to have "delivered" a stable service offering to "the business".
In my experience most of these problems are cultural and there is nothing to be done once bad software dev culture takes root. The most common manifestation is management asking for something reasonable but with constraints that make it impossible. The developer who agrees is favored. The one who warns is labeled as "negative". A year later when the project is in disaster mode the manager never reviews what happened at the beginning because he doesn't understand it.
It's essentially a Darwinian selection process for bad developers.
The good software companies I've worked for had a culture where people are able and empowered to push back on terrible ideas. The best ones are where that behavior is not just possible but expected.
What wonderful utopian companies did you work for? In the last round of layoffs in my company, I saw good, experienced, developers get pushed out because they promoted good ideas and tried to stop bad ideas from happening. Now everyone keeps their head down and projects are always in panic mode.
Unfortunately the places you describe far outnumber the places I describe. My "utopian" companies both degenerated into into "head down panic mode" places and I bailed.
Management is very different from engineering because there is no objective right or wrong, it is all subjective and all about the feelings of the people around you. Management is essentially a babysitting exercise. If you keep your people from throwing tantrums, having meltdowns, and going against the grain, you succeeded, and if you didn't, you failed.
Peer criticism is thus inherently orthogonal to management because very few people can maturely handle even a mild quantity of criticism. Keeping colleagues and superiors in a good emotional state is even more important for career development than executing your job description in a non-disastrous way.
Directing a business is much different from management, even if some management is required, but even then it's difficult to pinpoint cause and effect because there are so many externalities.
You'd think this would mean that good people who can deftly juggle a ton of variables are attracted to such positions, but in practice it seems that people who are bad at things and depend on having many ambiguous potential causes to diffuse fault over take them instead.
That's definitely the case in management; people who are incapable fall into those positions because their performance can't be definitely measured.
That's not true. Managers are measured on delivery, customer satisfaction, and revenue or profitability. These measures are directly related to ability to manage inputs (engineering resource) to outputs (finished code that drives those measures).
You can see types of results at a high-level, sure, but you can't see cause and effect.
Because management is about managing employee feelings, whenever an objection is raised, Bob can say "Well, Alice and Dan were having a really serious issue, which I can't discuss with peers due to confidentiality concerns, and we really had to dedicate a lot of effort to defusing that and retaining them in order to keep our mindshare and minimize legal risk". That's a legitimate managerial effort. Unless you're also in a position to intimately know the issue with Alice and Dan (and if you are then why does this manager exist?), you can't really say "That was an incorrect way to solve the issue."
Human emotions create a drag on productivity, can't get around that. A manager's job is to minimize, defuse, and de-escalate that drag. You can sort of say "Bob, your team always has the most problems, you clearly can't keep them under control", but it really depends on a lot of variables, and unlike software engineering, you can't directly quantify the impact of most actions/tradeoffs. It's very non-scientific.
If you know of a management culture that facilitates this kind of peer feedback, please let me know. My personal experience is that successful management types are extremely emotionally aware and well-polished, and know that providing criticism to anyone only serves to generate hostility. They will try to manage their reports indirectly, attempting to curb behaviors and get results without having to directly criticize.
Yes, this is critical. 99% of companies are run by cogs. It really takes a committed visionary at the top to set the tone for the culture, and at some point, those visionaries have to go away due to retirement, death, etc. There is a 99% likelihood that they will be replaced with another cog when this happens.
We must not become overly committed to a company, because a company is only as good as its people, and people come in and out all the time.
I'd say bigger and more frequent. I've never had a problem that went unsolved because the developers didn't have a reasonable tech solution. I have had things fail because developers don't grok the spec/requirements/domain.
I'd rank technical challenges well behind culture and domain understanding as reasons software projects fail
Would you say a groupthink is a possible cause? In the past I've seen developers band behind a specific technical platform or framework without considering the possible downsides and constraints that were raised by a few individuals. Not even having a conversation around it but "this is how we do things here, we are an X shop".
It makes me uneasy when everybody's answer is exactly the same and they arrived it not through individual opinions but by adapting to whatever was set in place. This isn't bad if all options were thoroughly explored or possible pitfalls addressed (there are no absolute, only subjective and relative truths that come with pros & cons which each decision)....but when there won't even be a debate or conversation asking if what we are choosing to use is really the right thing because we thought it through independently or just because some big name at big company X declared it is the new paradigm.
After all, we are all fallible, development and tools we use is influenced more by marketing than proper cost/benefit analysis. Whatever seems hip, new, cool tends to triumph.
"What? PHP is sooo 2006, we need a React/Redux app for our blog so it will be easy to scale using node.js asynchronity"
What you're describing doesn't have any obvious "out" group in it. I think instead what you're describing is what you'd expect to see in an environment where judging the merits of tools or processes is very hard and requires a lot of experience, which may be lacking. In such an environment it's natural to look to very successful companies and just copy whatever they're doing, even if you don't fully understand it, in the hope that it works out. And sometimes it will.
This isn't helped by the massive expansion of the developer workforce over the past decades, the fact that experienced people tend to get promoted up into management where they lose touch with coding and become 'architects' working entirely in the abstract, the fact that it's almost impossible for firms to correctly match team sizes to the amount of work required resulting in frequent periods of either overload or boredom (that then leads to reinvention of the wheel simply to find something interesting to do).
And yes, finally, fashion plays way too large a role in things these days. If I had a dollar for every time I read a blog post along the lines of "we had entirely generic serving problem X and we had to pick between Rust, Node or Go", with the far older and more mature .NET/Java stacks not just discarded but entirely unconsidered, I'd be a rich guy. We end up locked in a vicious circle that works against maturity of tools and processes:
- Profitable but slow enterprises look at startups, envy their ability to get things done and wonder if it's because of the stodgy old tools/processes they use.
- Fast but money-bleeding startups look at enterprises and say "we aren't a stuffy enterprise shop so we'll do whatever cool thing I read about on hacker news the other day" and often end up boxing themselves into a corner with immature tools or business approaches. Eventually they (might) turn into big slow companies themselves, often entailing either a rewrite along the way to use more traditional tools or absurd investments into digging themselves out of technical debt (Facebook's PHP runtime efforts being a case in point)
In your opinion, where does the current state of Javascript and it's fractured development tools, processes and frameworks fare against mature stacks?
You are totally spot on in that mature stacks don't even make the list. Obviously, new developers are going to be exposed to the new and hip fashion trends while veterans prefer predictability and stability.
It keeps me awake at night thinking about who is right. It could be that we are on the eve of a new paradigm where JS dominates the enterprise or we could be seeing another blip on the radar.
I do however think that AWS is a definite paradigm shift in how developers work, especially as serverless is gaining momentum but this is more of a devops/serverops shift imo.
If you're putting together something relatively simple, Javascript is a perfectly fine choice. And the fact that front and back end shares a language is a real win.
However if, for example, you have a complex site with intermittent performance problems, good luck. This applies whether you are running complex async code in node.js and you're trying to track down the badly coded callback that blocks everything else. It also applies if you have multiple microservices and you're trying to track down which back end service is causing front end requests to be painfully slow 1% of the time.
And, of course, I'm waiting for Javascript to have something like the problems that Rails did in 2013 when it was discovered that automatic deserialization of requests allowed remote code execution..in multiple ways. In the case of Rails, you just had to get an updated package. In the case of npm, however, the dependency system means that you won't be able to eliminate the bug without updating the whole toolchain in a way that is very, very hard to do. The problem is that in Javascript, libraries A and B can and often do import different versions of library C. If C has a critical bug, then you can't get rid of it until you've upgraded all three. Now imagine this with hundreds of dependencies, each of which has pinned its dependencies and has not been tested with breaking changes in underlying libraries that they depend on...
Every widely used stack eventually has an "upgrade the world" fire drill. That fire drill is going to be very, very messy with Javascript.
AWS is just a fancier form of server outsourcing. It's helpful and progress but doesn't represent a paradigm shift. Great chance for companies to shoot themselves in the foot by wasting lots of money though:
Javascript is a mess. Node got traction because it enabled lots of web devs to port their skills to the server side without having to learn new languages. Heck, the web is a mess. We desperately need a better app platform.
I wouldn't. I also think it is a small subset of projects in which specific technology choices are critical. Also, it seems you are describing to opposing problems, one shop only wants to use the technology it knows (in my opinion this should be the default position) and another in which they mindlessly want to use the latest and coolest. It takes a very knowledgeable manager to tell the rare cases of "we need the latest tech" from "my engineers want to play with the latest tech"
Strong engineering orgs hire managers who could do and have done most of what their team does. Most TPMs at G, for example, have CS degrees.
Non-technical managers have the organizational leverage to undo any amount of design & planning that skilled people underneath them set up. It's the brain equivalent of having ADHD -- your executive control shuts down and that sabotages long-term goals.
If you've never lived through this, watch the beginning season 3 of silicon valley where their new CEO wants them to 'just build a box' and gets angry when the conversation gets any more specific than that. (If you have lived through this, don't watch it -- it's too real).
Another point WRT Silicon Valley episode that has lessons for the real world. Developer will do a much better job on a project they picked rather than one that was assigned to them. This might be a lot of the advantage tiny, new startups have over the giants.
There's an assumption here that the amount of design and planning is the correct amount of design and planning. Your statement suggests you want your cake and to eat it too. You want freedom from management, but then want exact specification when advised to "build a box." There is likely a happy medium and I would suggest that your manager probably understands that dynamic more completely than you.
Edit: Agree that technical managers are best suited to evaluate the level of design and planning.
I have seen this work the other way. Manager asks for something reasonable. Devs do not execute successfully on timeline. Devs complain it was unreasonable or unrealistic but refuse to review the process that got them there.
I talked about this on a lunch with a coworker few days ago. We work in a traditional software company which sells legacy software to (big enterprise) customers. We were frustrated that everything takes us so long to change.
I think the reason is that we have large customers who expect a certain level of service. So even a trivial fix needs to be properly tested, documented, packaged. Also, we don't have a really good insight into how our customers use our software. We don't know what things we can break or deprecate, so we have to be extra careful. Support calls from users cost additional money, and we want to avoid those.
Companies that do not have external customers, but write software only for their internal use (or have cloud platform), such as Facebook, Amazon or Google, do not face these problems. They know who their customers are (guys from the other cubicle), what features they use. The can take a risk of deploying an improperly tested feature.
So yeah, they have an advantage. Whether this advantage will translate to the only internal software development model surviving in the future remains to be seen.
> We don't know what things we can break or deprecate, so we have to be extra careful.
I'd argue that almost nobody really knows that. [1] It's just that many developers / companies somehow get away with not being extra careful. How many libraries introduce some "unimportant" backwards incompatible change without bumping their major version number?
[1] ... unless you have a web application and really good statistics. But then, you end up being a large company like GitHub or Amazon, I guess.
Thinking about it some more, conservative users are also a factor. If we change anything, we impose additional cost on our customers, which have to retest, redeploy and retrain. So it's not easy for this reason alone.
When you have a platform for internal use only (or a platform that is available free of charge without any warranties), you can decide to make a change by fiat.
We would like to "move fast and break things", but our customers unfortunately don't appreciate that sentiment.
Thinking even more about it, many changes are "cosmetic", such as renaming fields or restructuring essentially flat structures. These may make the system more consistent and easier to understand for newcomers, but break old code for "no reason".
So developers should develop some kind of taste to distinguish between "real" changes and "cosmetic" changes.
What? Facebook, Amazon and Google do not have external users?
"We need to be careful because customers use our software" is a pretty lame excuse. Every company needs to properly test, document, package their software. Some companies do it better than others.
Where they have external paying users, they still operate the software for them (i.e. provide it as a service), so they have access to all what these customers are doing.
It's not a lame excuse, unfortunately, there is a very real barrier of trust when you talk to a coworker (even from another group) and when you talk to a customer. At the very least, you can agree on a common schedule.
IMHO it's not a coincidence that the blog listed those companies as more "nimble".
He also hints that his customers might have more power to push back on changes. That's certainly the case for smaller software houses that sell to big enterprise customers.
Facebook, Google, etc, are somewhat beholden to end users, but not in a way that impedes them from rolling out what they want, when they want to.
Good point. The worst-performing companies I've worked for had one thing in common: they were small and unable or unwilling to push back on toxic customers.
Pushing back doesn't always work. A good example...support for IE6, way past the time when that was a good idea.
Your contact at "big mega corp" that pays for your software may have no ability to change their internal policy that they've stuck with IE6 for years after it was sunset. So, if you want the sale or renewal, you have to support it.
Knowing your customer lets you take shortcuts. For example, if you have a monorepo, you can do a code search to find all usages of a function and fix them, instead of going through a formal deprecation process.
There is a tension here between efficiency and privacy. Inside a corporation with an "open" culture, you can be a lot more transparent and efficient since you don't have the same privacy issues.
Open source projects can sometimes do similar things, for example with the Linux policy of having device drivers in-tree.
I think with customers he meant mainly customers who customize and extend the software. Especially if you publish an API and customers can run the software on their own servers you are kind of stuck because you don't know what is being used and how. With cloud based stuff it's a little easier because you can track what people are doing.
Also, Facebook has a history of breaking things for 3rd developers. You can't really do this with enterprise customers who pay a lot of money for your stuff.
Developers always have to answer to someone. That might be a client who's buying bespoke software, or a customer who bought something off the shelf, or a user who uses something that's made internally within a company. If there's someone who uses the application then there's someone who you have to justify things to. No developer who writes code for money can just push out changes without worrying about breaking things.
The view that people who build and sell products can go faster and have less stress than people who write code for other people is nonsense. The pressures basically are the same; they're just from different groups of people.
I made updates to some of my code security reasons that ended up speeding things up a little bit (smaller binary to load + less reliance on permanent storage). This caused other developer's code to crash because he inadvertently needed my code to run slow under certain conditions (it wasn't obvious that the issue was a speed thing).
How do issues like this happen? I'm fairly new to software development and I haven't really seen issues like this, and I'm having trouble thinking of what types of code would cause it.
I'm assuming it's usually related to things using multiple threads or at the boundaries of different programs communicating?
Yes of course. It's concurrency related. You write a program that starts something, does some work, then tries to access the something expecting that it's already started. On your computer, the "does some work" stage happens to take long enough that the thing has indeed already started. On some other machine, it doesn't and you get a crash or hang.
It happens quite often when developing concurrent software.
Yeah, I worked for a startup that grew large quickly through a drive to get innovations to market fast and fix bugs later, and what slowed us down was winning enough large contracts with enterprise companies and governments who wanted more and more red tape and documentation and guaranteed processes, and basically exported their risk-averse culture to us.
We worked around this by having separate release channels, one for our enterprisey customers who opposed all change, and one for our customers who just wanted the coolest things first and were willing to accept some rough edges and lack of documentation.
That doesn't work if _all_ your customers are risk-averse, of course.
You're trying to use Google, Amazon, Facebook, as an example of a company that has no external customers and takes no risk when deploying untested features???
It's actually quite apt. AWS is a different beast, but Google and Facebook can and routinely do ship broken code to production that drops queries on the floor, serves broken JavaScript, redesigns the UI in ways that confuse millions of people, etc. It doesn't matter because the products are free. The worst that can happen is you lose money by failing to serve ads for a while.
Selling into the enterprise space can be very different. There even a single dropped or bogus query can have a big impact on the business. They are paying, so they get to decide if and when they upgrade. You have a small number of very important customers, instead of a billion unimportant customers, so losing one because you didn't meet their bizarre requirement is a big deal, whereas in the consumer space if you decide your app is iOS only, OK, no problem, you can still be successful.
> Companies that do not have external customers, but write software only for their internal use [...] do not face these problems. They know who their customers are (guys from the other cubicle), what features they use. The can take a risk of deploying an improperly tested feature.
In a non-software company, this is not my experience at all.
While some end-users are local, all communication must route through distorting managerial layers. Huge swaths of features exist from fearful inertia.
I've spent 5 weeks working with our offshore sysadmins to get Python3 installed on a dev box. Its a 30 second job, does anyone in a startup have these problems? Big banks suck right now.
Here was my experience at GS, I didn't last too long:
- I was never able to get email search in outlook enabled on my computer. It was a "security risk".
- Every Monday, my Visual Studio install broke and I'd have to call support in Bangalore to reinstall it.
- Our "make" system's subtasks would segfault every 1/3 times so I always ran it 3 times, sometimes 4.
- The only way to commit code (after using an out of date CVS) was to use this second in-house utility manually on every commit revision on every file. The asshole who wrote that thing made MD.
- Our 32bit system would run out of memory and take hours to compile because Java talked to C# talked to C++ talked to Slang talked to javascript all in the same address space.
Its a 30 second job, does anyone in a startup have these problems?
I did some work for a startup that required me to integrate with an external API. The API had a bug that the service provider didn't fix despite many requests over a period of about 6 months. We had to switch to a different provider and reimplement a lot of code using a their API instead. So, yes.
I'm at a startup that works with big enterprisey customers, maybe 50% of the time I have a ton of freedom and "agility", and the other half of the time it's like what you describe. Just insanely frustrating and painful.
And because our customers know us as being agile, there's a huge double standard to all of it. We can request changes, or even write all the code for them and send it, but won't see it live for 3-12 months. Even for critical bugs. However, when they request anything for our services, features require like a 1 month deadline from conception to deployment, and critical bugs is something like 72 hours.
A little ranty because 2016 for me was more like 90% "dealing with enterprise", 10% freedom. Thankfully next month I'm being switched to a team dealing more with internal services, so there's a lot less BS (because my boss says he's noticed how stressed it has made me, so that's nice).
The problem, as defined in the article: "Why are digital leaders (Facebook, Google, Amazon and friends) ten times better at rapidly delivering software than mainstream enterprise organisations?"
Then, the conclusion is "skills and talent lie at the heart of this issue".
That's not been my experience. There's usually plenty of skill and talent at mainstream enterprise organisations. It's other barriers: cultural, organizational, funding, etc, that keep the talent hamstrung.
Is the premise even true? Do facebook, google or twitter ship more major features than traditional enterprise companies? That sounds off to me.
Even if it were true, it has less to do with talent than with organization. Legacy organizations typically have legacy customers on legacy contracts that dictate legacy development models. Just try to be agile or do continuous deployment with a typical government contract. It can't be done.
Probably irrelevant, but I'm tired of seeing angled pictures of blurry code that looks like minified javascript on every article about software development from a business-focused source.
That would require someone on the stock image company's payroll who knows something about code. That costs money. How much are you willing to pay for articles with good stock photos?
It accurately represents the target audience's view of software: complex, terse 'code' that wizards understand.
It reminds me of when I was reading The Time Traveller's Wife and got to the part where the main character's friend hosted an exhibit of her art, which consisted of paintings of "the HTML source code of computer viruses". I nearly threw the book across the room. But I'm not the target audience. :-)
Yawn. Paul Graham basically wrote this article twelve years ago. I'm no fanboy, but at least he wasn't trying to coin bland catchphrases by italicizing them.
this was an interesting read, but i have an similar, but slightly different observation.
google and facebook also have mountains of technical debt and they don't necessarily fix it. in particular some google projects i have had to build have had the absolute worst kinds of dependencies and requirements to build - from what i've heard from the inside they don't suffer so much from this because they build elaborate systems to deal with it... that is not paying back technical debt, it is creating the risk of more in the future to avoid having to pay it at all.
aside from google some facebook projects i've seen also contain the very worst of practices of the kind that invariably result in technical debt and difficult maintenance work... but i know nothing about what they do
maybe it has the same effect, and maybe that is part of the trick, that its more productive to live with technical debt if you do it right than it is to pay it back
I suspect there's an additional factor incidental to the "skills and talent" issue- the ability of employees to care. I would wager that there are few engineers who are really, truly passionate about Oracle or Salesforce, but for some reason people still get excited about Google, Facebook, or the next Uber-for-dogs startup. This means that the latter category get an entirely different kind of value from the skills of their engineers.
If you're a good programmer, you can optimize for various different things- getting something done quickly, writing pretty code, making the boss happy, following every last implied detail of spec, or, importantly, making the software as useful as possible. What you need is for an engineer to be able to think, "well, bossman won't necessarily like it and it will take another ten minutes, but I should write this in this specific way because I predict it will prevent some problems if this code gets reused in certain contexts." There are innumerable small things engineers can do solve problems that might happen, but no salary can incentivize them to do this (or even to look for the opportunities to do so) since there's effectively no way to tell if they're doing it- engineers who don't care will likely be optimizing for making their bosses happy, so engineers who do care will even often seem worse according to any effectiveness metric the company employs. And if your employees don't care, you quickly accumulate technical debt that weighs down the company, even if everyone is happy and productive.
I don't particularly think there's a solution to this. A sad fact about industrial economies is that a whole lot of boring things that no one cares about need to get done. Even most of the applications that are exciting now will be boring in the future (someday, internet search will be an old-hat, commoditized field everyone takes for granted, and Google will fall.) Maybe someday company lifecycles will just be considered a fact of nature, and everyone will simply expect the next Microsoft to roll over and expire when its flagship product becomes irrelevant; it might make things more sane.
I'm not sure it's about adding unneeded features, rather stuff like not ignoring returned error codes or failing to write a unit test for something that needs it.
Experience with the domain doesn't need to have much to do with it- most of what I mean by "problems that might happen" are just pure software maintenance issues. Perhaps your new function gets a value that you know can't possibly be null; you don't need to write a check for null, then, since it will still always work. But a good software developer might feel untidy leaving unchecked data into code that might be later reused by null-value-using colleagues, and will take the two seconds to write the check. Furthermore, if he doesn't write the check and this turns out to be wrong, "just throw it away" isn't an immediate option; finding and fixing the error caused could take orders of magnitude more time.
Caring (alternatively called engagement) is huge. Employees who care are curious enough to dig beyond their day job. And learn about their customers. And are willing to push back on authority when they know that they're right.
The systems at very large public companies drive this behavior out of employees. (They value standardization and defending the existing business too much)
Agreed. However the issue is quite a bit more subtle because the larger the company the harder it is for employees to have the big picture on a purely logistical level. So it's not necessarily true that employees pushing back are right.
That's not to say existing management structures are optimal. A big part of what the latest breed of tech giants has done is really optimize the culture to be able to get the best out of large teams of engineers, who themselves are force multipliers via code. The computers are essentially like perfectly obedient slave labor.
In the old days you always needed a lot of man power, so the biggest organizations were all about rigid command and control structures and policies that would enable a large organization to function. Sometimes these organizations were defacto distributed systems with multi-month latencies (eg. The British Empire). Under that environment it's just not viable for individuals to question procedures on a regular basis or the whole thing breaks down.
What companies like Facebook have done is come up with an organizational architecture that can use the perfect labor of computers to break through to a level of impact that could not be achieved by a human-powered organizations. There's probably still an upper limit to efficiency, but it's orders of magnitude higher given the right engineering leverage and culture.
Agreed. However the issue is quite a bit more subtle because the larger the company the harder it is for employees to have the big picture on a purely logistical level. So it's not necessarily true that employees pushing back are right.
Yes - I've seen this too. The bigger the company, the longer you have to be there to understand all the far flung parts.
'Sometimes these organizations were defacto distributed systems with multi-month latencies (eg. The British Empire).'
Sorry if this is tangential, but the way the British empire managed it was by devolving almost all decision making and power to the people running the colonies. Like tax, law, culture, religion, language etc. Foreign policy was the only area where the crown called the shots.
The result was a very loosely coupled, very flexible, and adaptable distributed system, where one part would look and act competently differently to another part (I.e. Canada vs India).
Never say that something can't be measured (at least if it can be observed in the real world...).
If engagement manifests as pushback, then measure pushback. If employees are only working according to spec, then product ownership is effectively centralized with management - instead, increase engineering ownership, back it up by budgeting time for engineering to work on their 20% ideas for the platform, and stackrank the results if needed (big if) by giving bonuses to the most engaged engineers and considering terminating the engineers who won't engage (considering being the operative word here, many perfectly OK reasons why an engineer won't engage which aren't grounds for pushing him out of the team), which would open up spots on the team for engineers who might engage.
It's really kind of sad reading HN comments on how "oh it must be the fault of all those crappy engineers." We've all met those people who couldn't FizzBuzz their way out of a paper bag, but among the rest, most suffer from poor management. The solution to poor management is not running after mythical rock stars, it's better management.
Maybe I'm in a bad mood, but this seems to be a poorly researched article that throws around lots of catchphrases and unsourced claims in a way that speaks to people's prejudices. Which means that people want to accept it, but shouldn't blindly.
Let's work our way through it.
It starts with a claim that there is a 10x gap between software leaders and the rest. Where does that figure come from? Is there any data backing it up? If you dig in you'll find that the figure dates back to some poorly researched studies in the 1970s and originally was about individual software developers. There isn't any backing to this as a figure for organizations as a whole.
Next we have a reference to the innovator's dilemma that completely gets Clayton Christensen's thesis wrong. His actual thesis is that companies are very, very good at adopting improvements that they see as necessary to improving their core competency. What companies are bad at is jumping on inferior (by the measures their current market uses) technologies that are about to destroy that market. He offers lots of examples and his follow up book, The Innovator's Solution, walks through the internal organizational reasons for WHY it is so hard.
It is well worth understanding this thesis. However it has absolutely nothing to do with building a better website.
The too big to innovate section had no flaws that I see.
The technological legacy and technical debt section comes closer to the mark - then misses it entirely. The truth is that any organization has a culture, core strengths, and core weaknesses that are part of the corporate DNA. These matter and are very, very hard to change. These will determine what organizations are good at. "Digital natives" don't simply win by not having to spend on maintenance - my experience is that their ratio of maintenance work to new development isn't different than anyone else's. They win by having cultures, processes and decision making that work better for software. Cultural elements like not trying to use schedule pressure to motivate developers, processes that try to surface potential problems as quickly as possible, and decision making that works hard to avoid a blame based culture.
Unfortunately you would learn none of that from this section. You would learn more about actual core organizational problems from http://funnyshit.com.au/the_plan.html and http://www.tamingdata.com/wp-content/uploads/2010/07/tree-sw.... (Both of these highlight very, very real problems. The Plan illustrates the consequences of information traveling through an organization where each hears what they got in line with the bias that they want, then communicates upwards the best sounding thing that fits their understanding. It takes serious discipline to counteract this. And the project management cartoon demonstrates exactly how each role tends to fail.)
Skills and talent would have been an excellent place to raise some of these points. Would have been. Opportunity missed. Instead we get trite comments about how talent perceives big companies, with no useful information on why they are that way or what would be required to change.
Next, outsourcing. As a US based developer I like calls to reduce outsourcing, but this one is sadly just wrong. All of the big "digital natives" have outsourcing as a component of their development, and make it work. The actual tradeoff is that outsourcing adds delays and creates cultural challenges, but makes for cheaper development. Proactively address the challenges, outsource things where development speed is not a priority, and outsourcing is a very viable piece of your development. It should be part of th...
>> Why are digital leaders (Facebook, Google, Amazon and friends) ten times better at rapidly delivering software than mainstream enterprise organizations?
Because they pay significantly more, and are able to set up extremely narrow hiring filters?
118 comments
[ 6.2 ms ] story [ 202 ms ] thread[...]
> "So what’s the solution? To reduce the software delivery gap, mainstream enterprises need to raise their game by transforming their own internal capability."
[...]
> "We founded Contino so that we could instead transform through delivery."
[...]
> "Benjamin Wootton is the Co-Founder and CTO, EMEA of Contino."
Of course it's partly marketing but it doesn't detract from the argument.
That's the first sentence, hence the premise of the article. Don't you need to provide some supporting evidence for those figures?
Experience, again, suggests not. The digital leaders themselves are behemoths! Facebook has 1.19 billion active monthly users. Keeping them happy is quite a task, yet they manage to innovate alongside keeping the lights on. So how come enterprises are still so slow? Why do their technological efforts always seem to cost much more than they should? Why does the software development gap persist?
Does not follow. Facebook has lots of users but they have a relatively small workforce compared to traditional big companies. Over some threshold it's probably easier to "satisfy" lots of users via network effects than it is to please a small but critical community, anyway, but that's just conjecture.
The article is a continued series of unsupported assertions and "facts".
Then again my division was the direct heir to Tommy Flowers group and at the time had more engineers than Google had employees.
My observation from when I worked at Google was that the organisation was rapidly slowing down over time, and projects that were essentially pointless or duplicative were multiplying. Very simple tasks were starting to become extremely hard work. Much like a traditional large enterprise, in fact.
Look at how Microsoft stalled for a lot of years but after a change of culture (and focus!) they had a monster year and are poised for more. Their talent makeup wasn't that different during the bad or good times. Also, the argument fails when you look at all of the failed projects, spin outs and copy cat business of FB and Google and Microsoft. If they are 10x better in their very core, why all the failures? Because it is so easy to get wrong. Lots of companies have no choice but to do what their customers pay them to do or go out of business. Lots of companies don't have the cash cows of web search or Office to allow for huge mistakes.
Your software sucks because you don't have enough technical talent. + sales pitch to develop technical talent at your company.
Hopefully it doesn't detract from the point that it is a massive gulf between a Netflix and a traditional enterprise, which runs deep into their DNA.
The big boys simply have to overcome this over the next few years.
However there needs to be an intent by the "enterprise" to achieve some end goal, which is aligned with their overall business strategy, to operate their IT more like the agile technical rockstar companies.
Such transformation is costly and disruptive, so all of this has to be properly vetted in the context of their overall strategy.
The 10X developer concept is a bit flawed. If only because computer scientists and students understand that a linear co-efficient is borderline meaningless in measuring algorithmic performance, it seems to pander to the less-technically-literate side of the engineering pool.
Another problem - the developer who accomplishes tenfold productivity of his peers in one environment might be absolutely uncompetitive in another. For example, what's the difference between tuning performance in a critical backend path and building a fully functioning UI? A developer who is ten-times better at one might be incapable of completing the other.
Instead, especially with the whole full-stack movement, wouldn't it be better to look at qualifications, ability to learn, motivation, interest and attitude? Even looking for a "ninja developer" might be better if it means someone with an attitude to hack things together at any request. And looking for a seasoned developer might be better for maintaining and rebuilding one of those hacked-together platforms.
What's the difference between a 10X and X^2/10 developer? It's time for hirers and SE bloggers to see X, log X, X log X, and X^2 developers, who are all out there.
But I think that a 10x developer is one who knows all the ins and outs of the tools in his or her belt.
Who will prefer a 10-year-old framework that served him well over an opportunity to procrastinate by means of reinventing the wheel. (I'm looking at you, JavaScript-land)
And who doesn't fall into the "tower of babel"-trap, which is the believe that there's a universal language and a perfect way of writing a program or structuring a system.
Well, there's jQuery, which is still fine for very small things (an animation here, an AJAX call there), but is totally inadequate if you need to build a rich UI of the kind commonly expected now. You'll spend less time learning something like React or Vue and implementing things in it than you'd spend fighting with the callback hell and manually crafted state machines.
There are a gazillion very useful Java-libraries out there.
I just implemented a ListView with complex filtering, live text-matching and sorting for an Android/React-native-app which has no problem displaying 100.000 items or more using Java-libraries.
React-Native's implementation on the other hand seems to be riddled with problems, judging from the bug-tracker and the confused API.
Until then I use 10+ year old tech to get work done.
Most people will agree that "some people are a lot more effective than other people". But if you take the members of a team aside individually and ask them to rank the effectiveness of their colleagues, do you get compatible answers? Has anyone ever tried getting team A to look at the anonymised commits of team B for evaluation?
Yes. As a general rule, everyone knows who's doing the work.
Usually, even the manager knows. (Whether he cares and tries to keep the productive guys is another topic)
This concept was never meant to be precise. It just expresses that some people's productivity is hard for others to understand and hard to duplicate.
It doesn't matter how many "barely can write a CRUD page" people you task to write your distributed cloud filesystem, it won't work.
(Before someone complains, because I know you, Internet, these classifications aren't permanent. I was "barely capable of writing a CRUD page" at one point, too. But you have to work with the developers that you have today, not what they could be in 20 years.)
Indeed. It's possible to waste huge amounts of time simply trying to decide what to do and communicating with all the 'stakeholders'. If you want to change the way your organisation builds software products you have to change the organisation.
We've been experiencing a lot of entropy where I'm at. Everyone could feel it, no one could understand it. I started looking for what might be happening and what might fix it. I came across Lex Sisney and Organizational Physics. It makes a lot of sense, at least to me.
P.S. I'm not affiliated with Lex. (Link to his Kindle book on Amazon: http://amzn.to/2iUh09B or find a lot of what's covered in his book on his website for free: http://organizationalphysics.com )
90% of the article is about our theory that large enterprise organisations have a huge gulf when compared to digital leaders, and analysing why they are in this place.
To break out of it, they need to do things in a completely different way, more focussed on building their own capability than outsourcing work.
Why does a fortune 50 company need to do that? What is the ROI on such effort? What are their risks if their IT operation is not as cutting edge as Amazon's? Those are the core questions, which your article does not even mention.
Legacy technology is another issue I don't feel you fully addressed. For example, there are many companies that start new projects using legacy technology, or the wrong technology, again for political reasons. I've seen reasoning like "We're a Microsoft shop," and "It's easy to hire Ruby developers," as the main drivers of technology choice, rather than a thoughtful consideration of the requirements of a project, and what technologies are best suited to solve the problem at hand.
I appreciate the time and effort you put in to communicating your ideas, and do not mean to diminish from that at all. I do think if you added a little more meat and citations to your arguments, and got rid of some of the sales-pitching at the end, you could strengthen your article quite a bit though.
A good tech blog post would show how you're unique, how you're innovating, and how you're making real change for your customers (with stats to back it up).
I suggest thinking about a blogging strategy with at least 1-2 posts / month in 2017. Take a few hours to plan out what you're trying to achieve and who your audience will be... Clients? Potential sales / tech hires? Being a thought-leader? Think about what you want to communicate to them that's more personal than a brochure. That's the fundamentals of a good tech biz blog at your stage of blogging.
It's essentially a Darwinian selection process for bad developers.
If you are at a place with sane management, though, don't be in a great hurry to leave...
What is the equivalent in management? Does it exist?
Management is very different from engineering because there is no objective right or wrong, it is all subjective and all about the feelings of the people around you. Management is essentially a babysitting exercise. If you keep your people from throwing tantrums, having meltdowns, and going against the grain, you succeeded, and if you didn't, you failed.
Peer criticism is thus inherently orthogonal to management because very few people can maturely handle even a mild quantity of criticism. Keeping colleagues and superiors in a good emotional state is even more important for career development than executing your job description in a non-disastrous way.
Directing a business is much different from management, even if some management is required, but even then it's difficult to pinpoint cause and effect because there are so many externalities.
You'd think this would mean that good people who can deftly juggle a ton of variables are attracted to such positions, but in practice it seems that people who are bad at things and depend on having many ambiguous potential causes to diffuse fault over take them instead.
That's definitely the case in management; people who are incapable fall into those positions because their performance can't be definitely measured.
Because management is about managing employee feelings, whenever an objection is raised, Bob can say "Well, Alice and Dan were having a really serious issue, which I can't discuss with peers due to confidentiality concerns, and we really had to dedicate a lot of effort to defusing that and retaining them in order to keep our mindshare and minimize legal risk". That's a legitimate managerial effort. Unless you're also in a position to intimately know the issue with Alice and Dan (and if you are then why does this manager exist?), you can't really say "That was an incorrect way to solve the issue."
Human emotions create a drag on productivity, can't get around that. A manager's job is to minimize, defuse, and de-escalate that drag. You can sort of say "Bob, your team always has the most problems, you clearly can't keep them under control", but it really depends on a lot of variables, and unlike software engineering, you can't directly quantify the impact of most actions/tradeoffs. It's very non-scientific.
If you know of a management culture that facilitates this kind of peer feedback, please let me know. My personal experience is that successful management types are extremely emotionally aware and well-polished, and know that providing criticism to anyone only serves to generate hostility. They will try to manage their reports indirectly, attempting to curb behaviors and get results without having to directly criticize.
We must not become overly committed to a company, because a company is only as good as its people, and people come in and out all the time.
I'd rank technical challenges well behind culture and domain understanding as reasons software projects fail
It makes me uneasy when everybody's answer is exactly the same and they arrived it not through individual opinions but by adapting to whatever was set in place. This isn't bad if all options were thoroughly explored or possible pitfalls addressed (there are no absolute, only subjective and relative truths that come with pros & cons which each decision)....but when there won't even be a debate or conversation asking if what we are choosing to use is really the right thing because we thought it through independently or just because some big name at big company X declared it is the new paradigm.
After all, we are all fallible, development and tools we use is influenced more by marketing than proper cost/benefit analysis. Whatever seems hip, new, cool tends to triumph.
"What? PHP is sooo 2006, we need a React/Redux app for our blog so it will be easy to scale using node.js asynchronity"
https://en.wikipedia.org/wiki/Groupthink
What you're describing doesn't have any obvious "out" group in it. I think instead what you're describing is what you'd expect to see in an environment where judging the merits of tools or processes is very hard and requires a lot of experience, which may be lacking. In such an environment it's natural to look to very successful companies and just copy whatever they're doing, even if you don't fully understand it, in the hope that it works out. And sometimes it will.
This isn't helped by the massive expansion of the developer workforce over the past decades, the fact that experienced people tend to get promoted up into management where they lose touch with coding and become 'architects' working entirely in the abstract, the fact that it's almost impossible for firms to correctly match team sizes to the amount of work required resulting in frequent periods of either overload or boredom (that then leads to reinvention of the wheel simply to find something interesting to do).
And yes, finally, fashion plays way too large a role in things these days. If I had a dollar for every time I read a blog post along the lines of "we had entirely generic serving problem X and we had to pick between Rust, Node or Go", with the far older and more mature .NET/Java stacks not just discarded but entirely unconsidered, I'd be a rich guy. We end up locked in a vicious circle that works against maturity of tools and processes:
- Profitable but slow enterprises look at startups, envy their ability to get things done and wonder if it's because of the stodgy old tools/processes they use.
- Fast but money-bleeding startups look at enterprises and say "we aren't a stuffy enterprise shop so we'll do whatever cool thing I read about on hacker news the other day" and often end up boxing themselves into a corner with immature tools or business approaches. Eventually they (might) turn into big slow companies themselves, often entailing either a rewrite along the way to use more traditional tools or absurd investments into digging themselves out of technical debt (Facebook's PHP runtime efforts being a case in point)
You are totally spot on in that mature stacks don't even make the list. Obviously, new developers are going to be exposed to the new and hip fashion trends while veterans prefer predictability and stability.
It keeps me awake at night thinking about who is right. It could be that we are on the eve of a new paradigm where JS dominates the enterprise or we could be seeing another blip on the radar.
I do however think that AWS is a definite paradigm shift in how developers work, especially as serverless is gaining momentum but this is more of a devops/serverops shift imo.
However if, for example, you have a complex site with intermittent performance problems, good luck. This applies whether you are running complex async code in node.js and you're trying to track down the badly coded callback that blocks everything else. It also applies if you have multiple microservices and you're trying to track down which back end service is causing front end requests to be painfully slow 1% of the time.
And, of course, I'm waiting for Javascript to have something like the problems that Rails did in 2013 when it was discovered that automatic deserialization of requests allowed remote code execution..in multiple ways. In the case of Rails, you just had to get an updated package. In the case of npm, however, the dependency system means that you won't be able to eliminate the bug without updating the whole toolchain in a way that is very, very hard to do. The problem is that in Javascript, libraries A and B can and often do import different versions of library C. If C has a critical bug, then you can't get rid of it until you've upgraded all three. Now imagine this with hundreds of dependencies, each of which has pinned its dependencies and has not been tested with breaking changes in underlying libraries that they depend on...
Every widely used stack eventually has an "upgrade the world" fire drill. That fire drill is going to be very, very messy with Javascript.
http://idlewords.com/talks/website_obesity.htm#heavyclouds
Javascript is a mess. Node got traction because it enabled lots of web devs to port their skills to the server side without having to learn new languages. Heck, the web is a mess. We desperately need a better app platform.
Strong engineering orgs hire managers who could do and have done most of what their team does. Most TPMs at G, for example, have CS degrees.
Non-technical managers have the organizational leverage to undo any amount of design & planning that skilled people underneath them set up. It's the brain equivalent of having ADHD -- your executive control shuts down and that sabotages long-term goals.
If you've never lived through this, watch the beginning season 3 of silicon valley where their new CEO wants them to 'just build a box' and gets angry when the conversation gets any more specific than that. (If you have lived through this, don't watch it -- it's too real).
Edit: Agree that technical managers are best suited to evaluate the level of design and planning.
I think the reason is that we have large customers who expect a certain level of service. So even a trivial fix needs to be properly tested, documented, packaged. Also, we don't have a really good insight into how our customers use our software. We don't know what things we can break or deprecate, so we have to be extra careful. Support calls from users cost additional money, and we want to avoid those.
Companies that do not have external customers, but write software only for their internal use (or have cloud platform), such as Facebook, Amazon or Google, do not face these problems. They know who their customers are (guys from the other cubicle), what features they use. The can take a risk of deploying an improperly tested feature.
So yeah, they have an advantage. Whether this advantage will translate to the only internal software development model surviving in the future remains to be seen.
Or eventually they do as internally everything starts to rely on everything else and they can't move fast enough when competition arrives.
I'd argue that almost nobody really knows that. [1] It's just that many developers / companies somehow get away with not being extra careful. How many libraries introduce some "unimportant" backwards incompatible change without bumping their major version number?
[1] ... unless you have a web application and really good statistics. But then, you end up being a large company like GitHub or Amazon, I guess.
When you have a platform for internal use only (or a platform that is available free of charge without any warranties), you can decide to make a change by fiat.
We would like to "move fast and break things", but our customers unfortunately don't appreciate that sentiment.
So developers should develop some kind of taste to distinguish between "real" changes and "cosmetic" changes.
"We need to be careful because customers use our software" is a pretty lame excuse. Every company needs to properly test, document, package their software. Some companies do it better than others.
It's not a lame excuse, unfortunately, there is a very real barrier of trust when you talk to a coworker (even from another group) and when you talk to a customer. At the very least, you can agree on a common schedule.
IMHO it's not a coincidence that the blog listed those companies as more "nimble".
Facebook, Google, etc, are somewhat beholden to end users, but not in a way that impedes them from rolling out what they want, when they want to.
Your contact at "big mega corp" that pays for your software may have no ability to change their internal policy that they've stuck with IE6 for years after it was sunset. So, if you want the sale or renewal, you have to support it.
There is a tension here between efficiency and privacy. Inside a corporation with an "open" culture, you can be a lot more transparent and efficient since you don't have the same privacy issues.
Open source projects can sometimes do similar things, for example with the Linux policy of having device drivers in-tree.
Also, Facebook has a history of breaking things for 3rd developers. You can't really do this with enterprise customers who pay a lot of money for your stuff.
The view that people who build and sell products can go faster and have less stress than people who write code for other people is nonsense. The pressures basically are the same; they're just from different groups of people.
I made updates to some of my code security reasons that ended up speeding things up a little bit (smaller binary to load + less reliance on permanent storage). This caused other developer's code to crash because he inadvertently needed my code to run slow under certain conditions (it wasn't obvious that the issue was a speed thing).
I'm assuming it's usually related to things using multiple threads or at the boundaries of different programs communicating?
It happens quite often when developing concurrent software.
We worked around this by having separate release channels, one for our enterprisey customers who opposed all change, and one for our customers who just wanted the coolest things first and were willing to accept some rough edges and lack of documentation.
That doesn't work if _all_ your customers are risk-averse, of course.
You must be kidding.
Selling into the enterprise space can be very different. There even a single dropped or bogus query can have a big impact on the business. They are paying, so they get to decide if and when they upgrade. You have a small number of very important customers, instead of a billion unimportant customers, so losing one because you didn't meet their bizarre requirement is a big deal, whereas in the consumer space if you decide your app is iOS only, OK, no problem, you can still be successful.
In a non-software company, this is not my experience at all.
While some end-users are local, all communication must route through distorting managerial layers. Huge swaths of features exist from fearful inertia.
But the really sad thing is: Startups in the financial sector suck even more. Mt. Gox is just one of the many screw-ups there.
- I was never able to get email search in outlook enabled on my computer. It was a "security risk".
- Every Monday, my Visual Studio install broke and I'd have to call support in Bangalore to reinstall it.
- Our "make" system's subtasks would segfault every 1/3 times so I always ran it 3 times, sometimes 4.
- The only way to commit code (after using an out of date CVS) was to use this second in-house utility manually on every commit revision on every file. The asshole who wrote that thing made MD.
- Our 32bit system would run out of memory and take hours to compile because Java talked to C# talked to C++ talked to Slang talked to javascript all in the same address space.
I did some work for a startup that required me to integrate with an external API. The API had a bug that the service provider didn't fix despite many requests over a period of about 6 months. We had to switch to a different provider and reimplement a lot of code using a their API instead. So, yes.
And because our customers know us as being agile, there's a huge double standard to all of it. We can request changes, or even write all the code for them and send it, but won't see it live for 3-12 months. Even for critical bugs. However, when they request anything for our services, features require like a 1 month deadline from conception to deployment, and critical bugs is something like 72 hours.
A little ranty because 2016 for me was more like 90% "dealing with enterprise", 10% freedom. Thankfully next month I'm being switched to a team dealing more with internal services, so there's a lot less BS (because my boss says he's noticed how stressed it has made me, so that's nice).
Then, the conclusion is "skills and talent lie at the heart of this issue".
That's not been my experience. There's usually plenty of skill and talent at mainstream enterprise organisations. It's other barriers: cultural, organizational, funding, etc, that keep the talent hamstrung.
Even if it were true, it has less to do with talent than with organization. Legacy organizations typically have legacy customers on legacy contracts that dictate legacy development models. Just try to be agile or do continuous deployment with a typical government contract. It can't be done.
Can we get some better stock images?
It reminds me of when I was reading The Time Traveller's Wife and got to the part where the main character's friend hosted an exhibit of her art, which consisted of paintings of "the HTML source code of computer viruses". I nearly threw the book across the room. But I'm not the target audience. :-)
google and facebook also have mountains of technical debt and they don't necessarily fix it. in particular some google projects i have had to build have had the absolute worst kinds of dependencies and requirements to build - from what i've heard from the inside they don't suffer so much from this because they build elaborate systems to deal with it... that is not paying back technical debt, it is creating the risk of more in the future to avoid having to pay it at all.
aside from google some facebook projects i've seen also contain the very worst of practices of the kind that invariably result in technical debt and difficult maintenance work... but i know nothing about what they do
maybe it has the same effect, and maybe that is part of the trick, that its more productive to live with technical debt if you do it right than it is to pay it back
If you're a good programmer, you can optimize for various different things- getting something done quickly, writing pretty code, making the boss happy, following every last implied detail of spec, or, importantly, making the software as useful as possible. What you need is for an engineer to be able to think, "well, bossman won't necessarily like it and it will take another ten minutes, but I should write this in this specific way because I predict it will prevent some problems if this code gets reused in certain contexts." There are innumerable small things engineers can do solve problems that might happen, but no salary can incentivize them to do this (or even to look for the opportunities to do so) since there's effectively no way to tell if they're doing it- engineers who don't care will likely be optimizing for making their bosses happy, so engineers who do care will even often seem worse according to any effectiveness metric the company employs. And if your employees don't care, you quickly accumulate technical debt that weighs down the company, even if everyone is happy and productive.
I don't particularly think there's a solution to this. A sad fact about industrial economies is that a whole lot of boring things that no one cares about need to get done. Even most of the applications that are exciting now will be boring in the future (someday, internet search will be an old-hat, commoditized field everyone takes for granted, and Google will fall.) Maybe someday company lifecycles will just be considered a fact of nature, and everyone will simply expect the next Microsoft to roll over and expire when its flagship product becomes irrelevant; it might make things more sane.
Do the simple thing and if you're wrong then you have less work to throw away.
The systems at very large public companies drive this behavior out of employees. (They value standardization and defending the existing business too much)
That's not to say existing management structures are optimal. A big part of what the latest breed of tech giants has done is really optimize the culture to be able to get the best out of large teams of engineers, who themselves are force multipliers via code. The computers are essentially like perfectly obedient slave labor.
In the old days you always needed a lot of man power, so the biggest organizations were all about rigid command and control structures and policies that would enable a large organization to function. Sometimes these organizations were defacto distributed systems with multi-month latencies (eg. The British Empire). Under that environment it's just not viable for individuals to question procedures on a regular basis or the whole thing breaks down.
What companies like Facebook have done is come up with an organizational architecture that can use the perfect labor of computers to break through to a level of impact that could not be achieved by a human-powered organizations. There's probably still an upper limit to efficiency, but it's orders of magnitude higher given the right engineering leverage and culture.
Yes - I've seen this too. The bigger the company, the longer you have to be there to understand all the far flung parts.
I think we give managers and executives too much credit. The employees on the ground often know much better what's really going on.
Said another way, if executives have it right and employees have it wrong, the organization structure is broken.
Sorry if this is tangential, but the way the British empire managed it was by devolving almost all decision making and power to the people running the colonies. Like tax, law, culture, religion, language etc. Foreign policy was the only area where the crown called the shots.
The result was a very loosely coupled, very flexible, and adaptable distributed system, where one part would look and act competently differently to another part (I.e. Canada vs India).
If engagement manifests as pushback, then measure pushback. If employees are only working according to spec, then product ownership is effectively centralized with management - instead, increase engineering ownership, back it up by budgeting time for engineering to work on their 20% ideas for the platform, and stackrank the results if needed (big if) by giving bonuses to the most engaged engineers and considering terminating the engineers who won't engage (considering being the operative word here, many perfectly OK reasons why an engineer won't engage which aren't grounds for pushing him out of the team), which would open up spots on the team for engineers who might engage.
It's really kind of sad reading HN comments on how "oh it must be the fault of all those crappy engineers." We've all met those people who couldn't FizzBuzz their way out of a paper bag, but among the rest, most suffer from poor management. The solution to poor management is not running after mythical rock stars, it's better management.
Let's work our way through it.
It starts with a claim that there is a 10x gap between software leaders and the rest. Where does that figure come from? Is there any data backing it up? If you dig in you'll find that the figure dates back to some poorly researched studies in the 1970s and originally was about individual software developers. There isn't any backing to this as a figure for organizations as a whole.
Next we have a reference to the innovator's dilemma that completely gets Clayton Christensen's thesis wrong. His actual thesis is that companies are very, very good at adopting improvements that they see as necessary to improving their core competency. What companies are bad at is jumping on inferior (by the measures their current market uses) technologies that are about to destroy that market. He offers lots of examples and his follow up book, The Innovator's Solution, walks through the internal organizational reasons for WHY it is so hard.
It is well worth understanding this thesis. However it has absolutely nothing to do with building a better website.
The too big to innovate section had no flaws that I see.
The technological legacy and technical debt section comes closer to the mark - then misses it entirely. The truth is that any organization has a culture, core strengths, and core weaknesses that are part of the corporate DNA. These matter and are very, very hard to change. These will determine what organizations are good at. "Digital natives" don't simply win by not having to spend on maintenance - my experience is that their ratio of maintenance work to new development isn't different than anyone else's. They win by having cultures, processes and decision making that work better for software. Cultural elements like not trying to use schedule pressure to motivate developers, processes that try to surface potential problems as quickly as possible, and decision making that works hard to avoid a blame based culture.
Unfortunately you would learn none of that from this section. You would learn more about actual core organizational problems from http://funnyshit.com.au/the_plan.html and http://www.tamingdata.com/wp-content/uploads/2010/07/tree-sw.... (Both of these highlight very, very real problems. The Plan illustrates the consequences of information traveling through an organization where each hears what they got in line with the bias that they want, then communicates upwards the best sounding thing that fits their understanding. It takes serious discipline to counteract this. And the project management cartoon demonstrates exactly how each role tends to fail.)
Skills and talent would have been an excellent place to raise some of these points. Would have been. Opportunity missed. Instead we get trite comments about how talent perceives big companies, with no useful information on why they are that way or what would be required to change.
Next, outsourcing. As a US based developer I like calls to reduce outsourcing, but this one is sadly just wrong. All of the big "digital natives" have outsourcing as a component of their development, and make it work. The actual tradeoff is that outsourcing adds delays and creates cultural challenges, but makes for cheaper development. Proactively address the challenges, outsource things where development speed is not a priority, and outsourcing is a very viable piece of your development. It should be part of th...