Well, as companies get larger they can afford less and less to make mistakes. If I'm running a start-up company, I can get away with changing things abruptly, because I have a smaller base of users and I have fewer people relying on my running at production-level. Also, chances are I'll be able to look at user feedback effectively.
Once you get large, and your company is providing sustenance for all your employees, people rely on your product, and you've got too many users for effective feedback-checking, you have to close up, take fewer risks. Because suddenly, people want you to move slowly. They don't want you constantly skyrocketing ahead with their playing backup. Look at any big company - even Google, which was once famous for moving quickly - and you'll see that part of what gives a big company a good reputation is its being "solid." They have to give things up for an advantage.
It's why newspapers are so relied-upon. Of course, now it's what is hurting newspapers the most. They're being beaten by the flexible Internet. But even there, we're seeing a trade-off. Look at the quality of stories by the top writers online and by the top NY Times writers, and the online writers are much more amateur. They're faster, occasionally they're more interesting, but the Internet is thus far not retaining a high level of professionalism among reporting. Similarly, start-ups are much less reliable on the whole than large companies - look at Twitter and its problems, for instance.
So I think that PG's article is right. You can't restrict people and expect them to do as well. However, too much freedom leads to less stability, so it becomes a trade-off. Everything in moderation.
This thinking has always kind of confused me. Why are customers/users at a big company more important than those at a startup? Just because there's more of them, now you can't make mistakes?
If you have the agility to make rapid production changes, you also have the ability to rapidly rollback. So the argument that larger companies require more checks and testing than startups isn't really valid, especially when you consider the costs.
I think this phenomenon is caused by the culture of fear among most employees. Screwing up badly will get you fired, the greatest fear of all. Most employees have heavy mortgages and children, and require a consistent and predictable income. (This state of things is a problem in itself, but that's another topic).
The big issue is that the level of fear increases as you move up the command structure.
Let's assume one of your startups has crashed and burned. Something really spectacular must have happened if this makes potential customers of your next startup spurn you. Taking risks may have consequences, but they are temporary. But if you are a high-level manager for a large company, spectacular failure will make you unemployable, or in the best case push you several steps down the ladder - steps which took years of boredom and politics to climb, and which you will never get back. Failure is punished disproportionately, success is rarely compensated. Conservatism is the correct choice for each individual actor.
We all know bureaucracy, culture and politics - existing companies won't change in this area.
No, it's that most people don't want to use a product that changes often, if at all. They get it for what it is and if it changes, it means disrupting their way of using it. When you're working with a smaller product, you can iterate more and work closely with the people using it. When it gets larger, any change you make will alienate some users, so you have to plan to minimize that alienation.
You can change and roll back features quickly, but that gives an impression of instability among users. If things are constantly changing, they'll seek out something more stable. It's "lowest common denominator" thinking. It results in something that nobody dislikes, and that's the goal of larger groups. Niche companies are able to work much better, but even then there's some slowdown.
I think this is particularly important given that often the biggest cost in using a new product\tool is learning to use it well... if it's changing all the time, with no release schedule or no attempt to stabilize the main release via betas\QA\etc, then it gets very frustrating to learn.
If you have the agility to make rapid production changes, you also have the ability to rapidly rollback.
This is just not true. Rollbacks are always more expensive than changes, because you can't rewind time to undo the consequences of having your software be broken for minutes, hours, or days. Worse, in the absence of "checks", the cost of making a production change tends to be roughly constant as the company grows -- it takes the Amazon sysadmin no more time to type "make deploy" than it does me -- but the cost of a rollback scales directly with the size of your company's customer base.
Within a few seconds after Amazon.com breaks S3, thousands of companies begin to lose money, and they lose money second by second until the rollback happens. Even if Amazon is only down for a minute, that's one minute of downtime multiplied by its number of customers. The larger the customer base, the larger the stakes.
And, unfortunately, the cost of downtime is nonlinear. If Amazon goes down for a mere two minutes, hundreds of peacefully sleeping system administrators will get emergency pages from their uptime-monitoring systems. They will get out of bed. They will check their logs and their failover mechanisms. They will lose a lot of sleep, and soak up a bunch of overtime pay, and a lot of their good will towards Amazon will dissipate like the morning dew. Once you lose your reputation for quality it takes a lot of work to get it back.
This is why larger companies have more controls. The controls are in place to try and pass the ever-increasing cost of a rollback back to the team that causes the rollbacks. The reason it seems so gosh-darned expensive to add a trivial feature to your flagship app is that it is expensive: If the average rollback costs $1m in revenue and every new feature is only 95% reliable, every new feature costs the company $50k to deploy.
The secret here is: If you want to deploy changes rapidly, don't work on a product that has a lot of uptime-sensitive customers! Start a different product line, or start a beta program, or found a smaller company.
S3 is a really bad example because they provide infrastructure. Their customers actually see their entire site go down. Those kinds of companies are the exception. I hope Heroku has rigorous testing and scrutinizes every change, even though they are a startup.
Let's say I own a video site and I want to add threaded comments. If I have 5 users and the site goes down for 5 minutes, those 5 users will get 5 minutes each of annoyance. If I have a million users, each of those users will get 5 minutes of annoyance each also. There is no difference to the user there. So, by adding more checks to make sure the site doesn't go down for 5 minutes when you have more users, you're saying the more users you have, more the important each user becomes. I think that's a strange way of thinking.
(The same is true here of an infrastructure service-- if S3 had 5 users and were more cavalier about their release schedule and broke something, those 5 users would exact the same net effects of downtime as if S3 had 5 million users.)
The awesome benefit of getting threaded comments developed, tested briefly, and pushed in one evening is worth the risk of 5 minutes of downtime compared to the 2 weeks of rigorous testing and approval-by-committee. No matter how many users you have.
I used an infrastructure site as an example because the value proposition is easy to understand when you use a site that has a clear and simple monetization strategy. Video sharing sites are arguably an even worse example than S3, because the value of uptime is so hard to perceive or compute. It's likely that even Twitter doesn't understand the true value of a customer-hour of Twitter uptime, because the site isn't monetized and so much of the value is concentrated in the brand. Measuring that is like voodoo, only less empirical. ;)
If I have 5 users and the site goes down for 5 minutes, those 5 users will get 5 minutes each of annoyance. If I have a million users, each of those users will get 5 minutes of annoyance each also. There is no difference to the user there.
No, but there is a big difference for you! If a user is worth a dollar per year, the five-user site is worth five bucks per year, but the million-user site is worth a million bucks. If each patch to your code causes 0.1% of users to abandon your product (a number which depends on the odds that a patch will cause a rollback, and on the odds that a rollback will annoy a user enough to make them leave), patching a 5-user site costs you half a cent per year on average (most likely it has no perceptable cost, since odds are no users will leave) but each patch to a million-user site costs you $1000 per year in revenue. And that's just the linear cost. There are nonlinear consequences: one or zero annoyed users is nothing to worry about -- unless that user is Michael Arrington -- but a clique of 1000 annoyed users is potentially a movement: a critical mass of people who will all start complaining about your company on Twitter on the same day, potentially costing you your next 10,000 or 100,000 or 1 million users while simultaneously empowering your competitors, who may begin building the site that will take you down by poaching those dissatisfied users.
This is just the flip side of scalability. As a programmer you enjoy mighty economies of scale: Running a site with a million users is more expensive than running a single-user site, but it is much less than a million times as expensive. But this leverage also applies to your mistakes: a mistake that costs you a dollar when your site is small might cost you $1,000,000 when your site is big. And it's the same mistake! Typos are just as easy to make on big sites as on small ones.
Obviously, this doesn't mean that you shouldn't ever change the site. Presumably each and every one of your patches is valuable, and will bring in revenue to pay for its own insurance premiums. Right? :) But you do need to think about that calculation, because you do occasionally make mistakes. As your userbase grows, you may wish to test each patch on a subset of users to be sure they will really like it, and that the additional revenue is really going to be there. You may wish to institute tests and internal audits that lower the risk of rollbacks, or failover mechanisms to lower the cost of rollbacks. And before long, lo, you will be that which you deplore: A company with a bunch of annoying internal controls! But at least you'll have revenue to console yourself with.
But I think what Dustin is saying (correct me if I'm wrong) is that the multiplier applies both ways. And that the total cost of making a 5 minute downtime mistake, even to a million users, could easily be outweighed by the benefits of releasing a product/feature/site 2 weeks early. In most cases, I think large companies are risk adverse instead of risk neutral to situations like this.
I agree with both of you that it varies considerably based on what the site does (infrastructure, videos, games, etc).
In most cases, I think large companies are risk adverse instead of risk neutral to situations like this.
I'm not going to argue with that. Just because a certain increase of caution is rational doesn't mean that caution isn't being overapplied in many cases, just as PG suggests in his original post.
If you have the agility to make rapid production changes, you also have the ability to rapidly rollback
It doesn't matter. If you make a mistake with other people's money (e.g. calculating a payment wrong, crediting/debiting the wrong person, etc), even if you put it right quickly, they'll start losing trust and looking at your competitors.
When do you institute the rule -- on day one, in which case it is functionally equivalent to "do not have any checks"? On day 13? Day 7386?
What is the quantum of a "check"? Is Sarbanes-Oxley one "check" or a collection of hundreds of "checks"?
Does the limit apply department-by-department or company-wide? Your sysadmins will naturally tend to employ a lot more formal checks (most of which are hopefully enforced by tiny Perl scripts) than your R&D prototyping team. Do you force the sysadmins to compete with the accounting department for a quota of checks? Do you incentivize sysadmins to evade checks by getting tasks reassigned to the R&D prototyping team, which (alas) can't afford to have any checks?
The last thing you want to do is encourage teams of lawyerly meta-checkers to run around enforcing rules about rules. That's costly, squared.
Well yes, because that assumes that there is some magical amount of checks that must never be exceeded. Some checks are extremely valuable and should be added based on their merits alone.
That's the point I was trying to make. Something like
"always commit and run the tests before going to production" makes a lot of sense, and should be mandated. Even something like "three signed copies of an authorization form" makes sense for some projects if, for example, the cost of damages (say, lost data, downtime or annoyed paying customers) is big enough.
Measuring this, though, is the hard part. It's like trying to do the math behind "no silver bullet" and estimate communication costs (too many unknowns and conditional dependencies).
How about just measuring how many bits it takes to describe the checks (perl code etc + informal text). Do not make any hard and fast rules - just monitor this metric.
This past summer I worked as an intern for a big company. On the first day of orientation the head of the legal compliance department said very flatly to us that any individual could do far more harm to the firm than good.
This mentality, reenforced by the company's bureaucratic change management system, really did not sit well with me.
Fortunately for my summer experience, my "buddy"/summer mentor and I found a loophole which we used to "hack" the change management system so that once we got our initial approval, we could propagate changes to prod without running through the whole process again for each change (which otherwise would have been required).
For large companies with a valuable reputation, that's almost guaranteed to be true. Most people won't generate a million dollars worth of value in a given year, but nearly everyone could do that amount of damage to their company's reputation in just a few minutes.
Sure, yes, if e.g. the customers' health or safety was at risk. But I don't think the average bug in a web app would damage a company's reputation significantly. GMail occasionally shows me a message saying "Oops, an error occurred." It doesn't make me think any less of Google.
Imagine if, say, an Amazon customer service rep actually yelled, screamed, and abused a customer calling about his order.
A single individual - one who is not even in a high position. But you bet your buttons that it'll be all over the internet within ours.
Sure, it probably wouldn't even cause a blip to the stock price, but it's a huge amount of harm for a single person to cause.
I once worked for a company where anybody can push changes to prod without much feedback whatsoever. Certain more senior guys watched the commit logs for any obvious shenanigans or idiocy, but generally changes were within customers' hands within a day or two (and this was a desktop app!). They suffered some horrifying stability problem for a while that necessitated some new policies. It was certainly liberating to work in that environment, but I can see why many places have safeties in place.
You aren't a typical non-early-adopter user. I've had people make loud, public complaints about such trivial matters as a new logo being too tall, or a text box being too wide, or an optional WYSIWYG editor feature ruining the "text-only flavor of the community". These on websites far smaller than GMail.
Look at some of the major Web2.0 kerfuffles in recent years. Off the top of my head, I can think of:
- The HD/DVD mutiny on Digg
- Public suicides on both JoelOnSoftware and Justin.TV
- Reddit storing passwords in cleartext and them getting stolen off a laptop
Yeah, you could argue that those services are all still around, so obviously they haven't been hurt too much. How much management time was wasted in dealing with them? How many non-users decided not to become users because of something they heard 3rd-hand about how it's a terrible company?
(And I don't think the solution is to never innovate. I can understand why a middle manager at a big company would think so, though - a PR disaster is by definition public and disastrous. In the short run, it always makes more sense to not mess with success, it's just that this doesn't lead to success in the long run.)
None of these were the kind of bug that comes from releasing code with insufficient testing. The only one that even involved code was the Reddit password problem, and that was more a design mistake than a bug.
A culture of testing is a lot more than just verifying functionality. Code reviews with experienced engineers would have caught that design flaw before it was released. Or made it a high priority to fix.
I've dealt with the security teams at large companies. They end up implementing sometimes draconian policies, but it seems that programmers refuse to learn to write secure code any other way. I have to admit that, if you consider the large company as being a legitimate entity, they're doing legitimate work.
Privacy breach could harm a company a lot. Japan Network Security Association reported that, in 2007, total of 30 million personal information records are breached and estimated 22 billion dollars for damage/compensation in Japan.
Most of these incidents caused by employees taking out the data, but there are some cases caused by sloppy web app.
The kind of bug big companies are concerned about is the kind that causes google.com to be down for 15 minutes. That's a significant amount of revenue. Or the sort of bug that causes a release of an infrastructure component to be rolled back, causing more work for every project touched by that release. It isn't the famous sort of bug, these don't get picked up by the news, it's the sort of bug that nobody really notices but costs the organization a lot of money or work.
Well, what if this bug happened to allow content which was not-safe-for-children onto the front page of the most heavily trafficked web page in the world, widely promoted to Middle America as being G-rated? I speak purely hypothetically of course.
If the error was a security vulnerability in Lotus notes that resulted in all of GM's confidential email being uploaded to thepiratebay I think IBM's reputation would take a pretty serious hit.
And this is an error that one unchecked programmer would be perfectly able to make if there were no checks in place.
The flip side is that if an employee does generate that much income in one year, a large company isn't set up to reward it.
Case in point: I work for a $3Bn company and a year ago I developed a feature that saved the business ~$2M that year. We know the amount because when the problem raised its head we called together a number of people to calculate what the cost would be and estimates were between $1.5M - $3.5M if we couldn't find a solution. Not loss of revenue, but we would have actually had to pay out that much in support costs.
The problem was highly visible across the organization and I was able to find a way to solve the it in a way that was completely invisible to the customer, implemented it and released to the field. My manager looked good, his manager looked good. I got an "average" performance review. I was pissed and to this day I haven't regained my motivation to do that caliber of work again. I wonder if I ever will without leaving!
As companies grow big, they like to emulate larger companies, and hence the people they like to hire are from larger companies since they understand scale -- this comes with a lot of baggage ofcourse. This in turn, most times, brings with it a culture of people who are worried more about not screwing up, covering their own ass, and thinking about getting promoted rather than doing what's right for the company or the business and screwing up as a byproduct of making quick decisions.
Re Joel Spolsky: The cost of the sale is not only the cost of the product, but what an organization would pay for a software/service, and what they would value it at. This is why most companies have a sales force, and don't advertise their prices on their website. As Joel mentions, for a lot of companies, charging less than $50k is a rounding error and not worth their time.
The flip side of people at big companies buying is based not entirely on the performance of the service or the product. As long as the service/product is reliable & ok, and it doesn't screw up in any major ways, they won't get fired for making that buying decision, rather than looking at ways of maximizing/optimizing the performance of the product -- yet another reason committees make buying decisions.
Re SOX: I like what founder's fund + facebook is doing -- letting early employees cash some of their equity out, thereby increasing the time early employees will stay with the company.
I also like Fred Wilson's thoughts on a secondary market for startup stock (similar to what goog does), this would in some sense, I guess let you be a private company, and not deal with the challenges associated with SOX compliance and at the same time not raise a highly dilutive Series D, in addition there is some sort of liquidation event for the investors.
PS: (std. Buchheit comments about Limited Life Experiences + Overgeneralization = Advice apply)
I think the costs of safety checks can be more pronounced in software than in other areas of business. It's not just that good hackers are particularly irritated by them -- they're irritated for a reason.
A two-week release-process lag is so much worse than it might sound. If you can ship instantly, you can get feedback and make improvements rapidly, over and over. Sure, without a careful release process you might occasionally break things, but when you do you can fix them in minutes rather than weeks. When you insert release-process lag into that cycle, improvements that might have been made in a rapid series of releases over a day or two can take months. Sometimes those improvements just never get made, because no one is motivated to keep working on them for so long.
In software, "stable" often means "unchanging", but only rarely means "high-quality".
The cost of the checks should balance the cost of the risks. Those are not always determinate or exactly estimable, but, statistically, that would be a simple basis of a rational approach. Wouldn't it?
This is what I call "playing defense" instead of "playing offense". By nature, the people at large companies are more concerned with risk aversion than innovation. This is also how managers like to "hold power" over the real innovators by controlling the change management process. What makes a manager more qualified to deploy to production than the programmer?
Some things that PG didn't make explicit: web applications can have an extraordinarily tight feedback loop between customer and coder. Multiple revisions per day are easy, especially if any single user's data is rather low-value. Many a startup of this kind has zoomed past its corporate competitors simply by iterating faster. If your product is embedded firmware for a home security device, this strategy is just not available.
But here's another twist: when you start working for a giant company that provides myriad services under the same login, suddenly everyone has to move as slowly as the slowest part of the business. It doesn't matter if you are just doing Yippee! Backgammon, because who knows -- maybe you could somehow accidentally expose the data at Yippee! Payment Solutions.
This is a completely rational consequence of being a giant company and delivering multiple web services under the same login. It is possible to have slightly different authentication policies for each service. But I wonder if OpenID providers have thought about this enough.
Programmers are unlike many types of workers in that the best ones actually prefer to work hard. This doesn't seem to be the case in most types of work. When I worked in fast food, we didn't prefer the busy times. And when I used to mow lawns, I definitely didn't prefer it when the grass was long after a week of rain.
Programmers, though, like it better when they write more code. Or more precisely, when they release more code. Programmers like to make a difference. Good ones, anyway.
Salesman like it best when they can sell more. When I used to sell used cars, I loved it best when I could keep pushing cars out of the lot.
Writers like it best when they can write more. They like it best when their imagination can produce thousands of stories.
The difference is that coders usually know what they will tackle and have a strategy so it is easier for them to get to the end point. Also they are almost sure they will hit the end point of a certain problem.
With other professions you cannot predict much. Your next customer is not guaranteed, your next story just does not want to show up in your brain etc....
That being said, Yes, a lot of programmers I know work hard.
I think most people want to do a good job and prefer to work hard. They want to see an impact from their labors and want their firm's customers to be delighted with their experience of interacting with the firm's products and service teams. It's our challenge as entrepreneurs to create the right environment for this.
I suspect that what is really going on here is that workers who love their job actually prefer to work hard AND good programmers love their job. So it is probably true that good programmers actually prefer to work hard, but that is not the whole truth...
Also, since most workers don't love their job, it is probably also true that good programmers are unlike many workers. To say that programmers are unlike many TYPES OF workers we'd have to believe that a higher proportion of programmers love their job than do workers in other types of work. From what I have seen on HN (albeit sample biased) I'd say that is true.
Yes. I've worked in fast food, and as a cashier at a convenience store, and in both cases, those who were better at the job preferred busier times ("It makes the day fly by, doncha think?").
Not true. Good salesmen love the power, convincing others to do their will. They are more like actors or politicians. You can't fake charisma just because you might get paid more.
You can't do anything well without liking to do it. The difference between hackers and salesmen is that hacking is what hackers like most. Salesmen who have sufficient money seem empirically to prefer playing golf, whereas hackers with money tend to keep hacking.
Even in the case of the fast food worker, PG gets it wrong. When it gets slow in a restaurant, you end up cleaning the grease traps or other unpleasant jobs. I'd take the adrenaline rush of juggling a dozen orders on the grill any time.
There's an excellent prototyping policy in effect at Maxis:
In terms of time, they have a policy of permission vs. forgiveness. You need to be prepared to fail early – but that’s okay! If a prototype takes less than two days, don’t worry: just go ahead and do it. It if takes more... you should probably have permission.
I would add that in addition to having to show the cost of any new checks that one should have to show how the check will actually prevent the problem it proposes to prevent and the likely-hood of that problem occurring or re-curring. A major problem I've seen with checks is that they just become CYA material rather than anything truly beneficial or preventative and people will work around them.
I think this can be generalized to say that start-ups have a different cost-benefit analysis - one where losses are capped at the (relatively small) value of the company. On the other hand, large companies have to be risk-averse because the worst-case is several magnitudes worse. When you're working at ConEd or AIG, the "tiny probability/worst-case-loss" factors start to matter, because "worst-case" can include investors losing life savings, federal investigations, and jail time for your boss's boss.
During their executive retreats they decided to be risk seeking instead.
I wonder if the problem at the recently troubled companies (AIG, Merrill, Lehman etc) was not enough checks in place or not enough power allotted to the employees who knew something was wrong.
From memory: the Excel bug that caused Moody's to give high ratings of mortgage securities (one of the probable causes of the whole mess) was known, but management decided not to fix it. So, greed can override checks and balances.
The issue was not a bug, it was a fundamental misunderstanding of how real estate markets work. Moody's had a model for pricing mortgages which assumed real estate prices never went down. It was basically the real life equivalent of the famous "Assume a spherical cow..." problem.
The major factor at work is simply a lack of personal responsibility towards the end product in a large company.
In a startup there's noone to blame but you, and thus no need for politics. As anyone who has worked in middle management or above in a large company can tell you half of the job is making sure you can always find someone else to blame, while taking credit for the successes of others.
"... Steve Jobs's famous maxim "artists ship" works both ways. Artists aren't merely capable of shipping. They insist on it. So if you don't let people ship, you won't have any artists ..."
Steve Yegge has had the unenviable luck of working on 3 products all of which have not shipped (cited as business reasons) yet he still works for Google ~ http://blog.stackoverflow.com/2008/10/podcast-25/ So it seems there is something else going on that keeps programmers on board. Is it money, having the freedom to talking about it the process? I don't know.
I spent 3 months at the end of recent deployment furiously developing an (arguably) game-changing app for the Army's secret intranet. I knew in advance it wouldn't actually get deployed, despite everyone up to the Task Force Commander loving it and touting its importance. The day came when it was "done" (of course I had become attached and there were a million things I wanted to add) and ready to be deployed... then the red tape tied it up until it was time for us to go. As is generally the case, our replacements weren't planning to do things our way (the country has gotten much worse since we left, imagine that) so the project was basically sunk.
I worked on it because I love programming and trying new things. Everything I was doing at the time was new to me so it was fun, engaging, and a learning experience. Plus, it beat the hell out of anything else I could have been doing in that wasteland ;)
"... I worked on it because I love programming and trying new things. Everything I was doing at the time was new to me so it was fun, engaging, and a learning experience. Plus, it beat the hell out of anything else I could have been doing in that wasteland ;) ..."
Good explanation.
I put it as a question as I'm wasn't conclusively sure why people would sign on to assignments that have a probability of failure. But I've since found that smart organisations allow pursuit of training and learning as a reason to keep staff and motivation. I figure the USF know a thing or 2 about motivation. Hope your putting the lessons you learned into some hack your working on :)
To "make something people want", you need to know your customer's value system: what's important to him and what's not. This can be hard for startups, especially ycombinator types which may be very low budget, because their own value system is so different. Startups start with a company worth about $1 and want to turn it into a company worth about $1,000,000. Therefore startups value the ability to change rapidly -- that's why they want good programmers.
Big companies on the other hand, start with a company worth $100,000,000, and want to turn it into a company worth $150,000,000. They value steady process improvement that doesn't risk their existing revenue -- that's why they want good managers.
It's important to know this if you're trying to sell to a big company as well as if you're trying to compete with a big company.
Having never had a "normal" job per se (okay, I bagged groceries for a year when I was younger), I still find it hard to believe that it can take two weeks just to get something deployed. Maybe one day I'll finally truly believe all the stories I've heard -- two-week deployments, meetings about future meetings, etc. For now, some part of me still believes it's simply implausible and everyone is engaging in hyperbole...
It's not abnormal. In rails I have deployed to a customer in 12 hours, starting from scratch. I was using my own framework though. In gwt, I have done this in ~24 hours.
These weren't simple apps either, not just scaffolding, but had complex user interfaces.
The delay is between when it's first claimed finished by the author(s) and when it's agreed to be ready by everyone involved. After that it can be deployed instantly...
I guess it would be possible to note in a bug tracking app "we would have deployed here" and then see how many, and what kind of bugs were found between then and actual deployment, which checks found them, and whether any of those were rollback-causers.
That would give an estimate of how much good the checks did in each case, and a way to choose which checks are ineffective for their cost. (Ish - some checks may only trigger once, ever, to be priceless).
Sure. It is often due to dysfunction, but it can even happen to good teams.
Imagine you have to fix a simple bug, like the description of an option in an HTML page.
Imagine that: the codebase is a bajillion lines of Java or C++ that no one person truly understands any more.
Imagine that: fucking up is considered bad. Maybe you're protecting important user data, or maybe it's just code for the online store, and even an hour of downtime can wipe out the day's profits.
The team is so big that your feature is scheduled for deployment along with four or five others this week.
As usual, management has skimped on QA so there are not enough of them to go around. Also, QA was added late, so there's no fast automated test suite. It takes them a full week of manual and semi-automatic testing to go through all regressions.
After that week of testing in development is over, then it goes to staging, where it's tested with real live data and staging subsystems.
At least somewhere in this, something breaks (something is always breaking) and a fix is produced. Then the new build is tested all over again, with the basic smoke tests as well as any test suites that are applicable to that unit of code which broke.
This is causing another trainwreck because we promised feature X to client Y by end of the month. What do we do? Emergency team meeting! Let's rebalance the work schedule.
Meanwhile, your change, which just a simple HTML fix, languishes as this build goes into its second round of testing.
Oh, and did we mention that you have to have submitted your code in time for translation into six languages? I'm afraid that the French translator did not produce a satisfactory translation in time, so we're going to have to hold your change back. Can you revert this and put it in next week's build? Thanks.
"...not only wouldn't these guys have broken anything, they'd have gotten a lot more done."
I don't think the claim that they would not have broken anything can be backed by facts.
If you look at Microsoft, for instance, their programmers used to write very buggy code, but with the introduction of better processes, like their Secure Development Lifecycle, they saw a substantial improvement in code quality (as measured through security and reliability metrics).
The damage done by not following a quality assurance process and writing buggy code was so big that Microsoft's image will be affected for a long time, even though they have now improved the code quality.
I worked without testing for 2 years and released a single bug into production. In that specific case I thought something might have been wrong but they needed it ASAP so I said who cares. My father once had a tester say "Ops I stopped testing your code over a year ago" when he released a fix to some buggy code someone else wrote which still had a few bugs. After adding a formal QA process I stopped double checking how buggy my code so I could get more done but I have ended up releasing more bugs.
QA is really useful, but there's a big downside to it.
When you have people who's job is to test code, developers get lazy and sloppy.
Also, bugs do appear in production if you can release code the same day you wrote it, but you can fix it as soon as you find out about it. In big companies that work with milestones (sprints in agile speak), only really critical bugs can be fixed between releases. Not to mention that most bugs are fixed after code-freeze, when all the deliverables of that milestone are ready, long after you find out about it.
So I guess it's best to have QA, but also the freedom to deploy code on production whenever you want.
Checks have a cost, for sure.
What about Eric Reis's split-testing axiom?
If in a startup the developer/coder is basically the customer of the enterprise because the cost of the check is paid by the rising frustration of the coder, then should split-testing be looked upon as a check?
Even if it is, the idea of an available metric to backup claims of efficacy is really compelling.
Yes, startups make their job better than big companies. But I think that it is a part of a more general rule: people work productively in OPEN systems, and the open systems themselves encourage working hard. Open source projects are developed more rapidly, because it is hard to imagine the system which is more open. Both users and developers control the system, and every user can become a developer. In startups we have a limited number of people who can control the process of development, but they still controll it. In other words, the system is open for founders and employees. Big companies work unefficiently because they are too closed, and a great number of checks is only a part of it.
The movie "Meet the Robinsons" makes a point to children that we make the most progress when we are free to fail. Thanks for making the same point for grownups.
131 comments
[ 2.7 ms ] story [ 221 ms ] threadOnce you get large, and your company is providing sustenance for all your employees, people rely on your product, and you've got too many users for effective feedback-checking, you have to close up, take fewer risks. Because suddenly, people want you to move slowly. They don't want you constantly skyrocketing ahead with their playing backup. Look at any big company - even Google, which was once famous for moving quickly - and you'll see that part of what gives a big company a good reputation is its being "solid." They have to give things up for an advantage.
It's why newspapers are so relied-upon. Of course, now it's what is hurting newspapers the most. They're being beaten by the flexible Internet. But even there, we're seeing a trade-off. Look at the quality of stories by the top writers online and by the top NY Times writers, and the online writers are much more amateur. They're faster, occasionally they're more interesting, but the Internet is thus far not retaining a high level of professionalism among reporting. Similarly, start-ups are much less reliable on the whole than large companies - look at Twitter and its problems, for instance.
So I think that PG's article is right. You can't restrict people and expect them to do as well. However, too much freedom leads to less stability, so it becomes a trade-off. Everything in moderation.
If you have the agility to make rapid production changes, you also have the ability to rapidly rollback. So the argument that larger companies require more checks and testing than startups isn't really valid, especially when you consider the costs.
The big issue is that the level of fear increases as you move up the command structure.
Let's assume one of your startups has crashed and burned. Something really spectacular must have happened if this makes potential customers of your next startup spurn you. Taking risks may have consequences, but they are temporary. But if you are a high-level manager for a large company, spectacular failure will make you unemployable, or in the best case push you several steps down the ladder - steps which took years of boredom and politics to climb, and which you will never get back. Failure is punished disproportionately, success is rarely compensated. Conservatism is the correct choice for each individual actor.
We all know bureaucracy, culture and politics - existing companies won't change in this area.
You can change and roll back features quickly, but that gives an impression of instability among users. If things are constantly changing, they'll seek out something more stable. It's "lowest common denominator" thinking. It results in something that nobody dislikes, and that's the goal of larger groups. Niche companies are able to work much better, but even then there's some slowdown.
This is just not true. Rollbacks are always more expensive than changes, because you can't rewind time to undo the consequences of having your software be broken for minutes, hours, or days. Worse, in the absence of "checks", the cost of making a production change tends to be roughly constant as the company grows -- it takes the Amazon sysadmin no more time to type "make deploy" than it does me -- but the cost of a rollback scales directly with the size of your company's customer base.
Within a few seconds after Amazon.com breaks S3, thousands of companies begin to lose money, and they lose money second by second until the rollback happens. Even if Amazon is only down for a minute, that's one minute of downtime multiplied by its number of customers. The larger the customer base, the larger the stakes.
And, unfortunately, the cost of downtime is nonlinear. If Amazon goes down for a mere two minutes, hundreds of peacefully sleeping system administrators will get emergency pages from their uptime-monitoring systems. They will get out of bed. They will check their logs and their failover mechanisms. They will lose a lot of sleep, and soak up a bunch of overtime pay, and a lot of their good will towards Amazon will dissipate like the morning dew. Once you lose your reputation for quality it takes a lot of work to get it back.
This is why larger companies have more controls. The controls are in place to try and pass the ever-increasing cost of a rollback back to the team that causes the rollbacks. The reason it seems so gosh-darned expensive to add a trivial feature to your flagship app is that it is expensive: If the average rollback costs $1m in revenue and every new feature is only 95% reliable, every new feature costs the company $50k to deploy.
The secret here is: If you want to deploy changes rapidly, don't work on a product that has a lot of uptime-sensitive customers! Start a different product line, or start a beta program, or found a smaller company.
Let's say I own a video site and I want to add threaded comments. If I have 5 users and the site goes down for 5 minutes, those 5 users will get 5 minutes each of annoyance. If I have a million users, each of those users will get 5 minutes of annoyance each also. There is no difference to the user there. So, by adding more checks to make sure the site doesn't go down for 5 minutes when you have more users, you're saying the more users you have, more the important each user becomes. I think that's a strange way of thinking.
(The same is true here of an infrastructure service-- if S3 had 5 users and were more cavalier about their release schedule and broke something, those 5 users would exact the same net effects of downtime as if S3 had 5 million users.)
The awesome benefit of getting threaded comments developed, tested briefly, and pushed in one evening is worth the risk of 5 minutes of downtime compared to the 2 weeks of rigorous testing and approval-by-committee. No matter how many users you have.
If I have 5 users and the site goes down for 5 minutes, those 5 users will get 5 minutes each of annoyance. If I have a million users, each of those users will get 5 minutes of annoyance each also. There is no difference to the user there.
No, but there is a big difference for you! If a user is worth a dollar per year, the five-user site is worth five bucks per year, but the million-user site is worth a million bucks. If each patch to your code causes 0.1% of users to abandon your product (a number which depends on the odds that a patch will cause a rollback, and on the odds that a rollback will annoy a user enough to make them leave), patching a 5-user site costs you half a cent per year on average (most likely it has no perceptable cost, since odds are no users will leave) but each patch to a million-user site costs you $1000 per year in revenue. And that's just the linear cost. There are nonlinear consequences: one or zero annoyed users is nothing to worry about -- unless that user is Michael Arrington -- but a clique of 1000 annoyed users is potentially a movement: a critical mass of people who will all start complaining about your company on Twitter on the same day, potentially costing you your next 10,000 or 100,000 or 1 million users while simultaneously empowering your competitors, who may begin building the site that will take you down by poaching those dissatisfied users.
This is just the flip side of scalability. As a programmer you enjoy mighty economies of scale: Running a site with a million users is more expensive than running a single-user site, but it is much less than a million times as expensive. But this leverage also applies to your mistakes: a mistake that costs you a dollar when your site is small might cost you $1,000,000 when your site is big. And it's the same mistake! Typos are just as easy to make on big sites as on small ones.
Obviously, this doesn't mean that you shouldn't ever change the site. Presumably each and every one of your patches is valuable, and will bring in revenue to pay for its own insurance premiums. Right? :) But you do need to think about that calculation, because you do occasionally make mistakes. As your userbase grows, you may wish to test each patch on a subset of users to be sure they will really like it, and that the additional revenue is really going to be there. You may wish to institute tests and internal audits that lower the risk of rollbacks, or failover mechanisms to lower the cost of rollbacks. And before long, lo, you will be that which you deplore: A company with a bunch of annoying internal controls! But at least you'll have revenue to console yourself with.
I agree with both of you that it varies considerably based on what the site does (infrastructure, videos, games, etc).
I'm not going to argue with that. Just because a certain increase of caution is rational doesn't mean that caution isn't being overapplied in many cases, just as PG suggests in his original post.
It doesn't matter. If you make a mistake with other people's money (e.g. calculating a payment wrong, crediting/debiting the wrong person, etc), even if you put it right quickly, they'll start losing trust and looking at your competitors.
I wonder if there's a way to limit the total cost without hard-and-fast silly rules.
e.g. you can only add a new step to a process if a step is removed from another.
Yes. ;)
When do you institute the rule -- on day one, in which case it is functionally equivalent to "do not have any checks"? On day 13? Day 7386?
What is the quantum of a "check"? Is Sarbanes-Oxley one "check" or a collection of hundreds of "checks"?
Does the limit apply department-by-department or company-wide? Your sysadmins will naturally tend to employ a lot more formal checks (most of which are hopefully enforced by tiny Perl scripts) than your R&D prototyping team. Do you force the sysadmins to compete with the accounting department for a quota of checks? Do you incentivize sysadmins to evade checks by getting tasks reassigned to the R&D prototyping team, which (alas) can't afford to have any checks?
The last thing you want to do is encourage teams of lawyerly meta-checkers to run around enforcing rules about rules. That's costly, squared.
When you realise you have a (potential) problem. It stops it getting worse.
"What is the quantum of a 'check'?"
More tricky, certainly. That's a potential big problem.
"Does the limit apply department-by-department or company-wide?"
Whatever makes sense.
The idea of the exercise would be basically to:
- build an appreciation throught the organisation that checks have a cost
- provide a (albeit imperfect) mechanism for exercising some control over it
Something doesn't have to be perfect to be of use.
"The last thing you want to do is encourage teams of lawyerly meta-checkers to run around enforcing rules about rules. That's costly, squared."
That's just ISO9001, isn't it? :-)
Measuring this, though, is the hard part. It's like trying to do the math behind "no silver bullet" and estimate communication costs (too many unknowns and conditional dependencies).
This mentality, reenforced by the company's bureaucratic change management system, really did not sit well with me.
Fortunately for my summer experience, my "buddy"/summer mentor and I found a loophole which we used to "hack" the change management system so that once we got our initial approval, we could propagate changes to prod without running through the whole process again for each change (which otherwise would have been required).
A single individual - one who is not even in a high position. But you bet your buttons that it'll be all over the internet within ours.
Sure, it probably wouldn't even cause a blip to the stock price, but it's a huge amount of harm for a single person to cause.
I once worked for a company where anybody can push changes to prod without much feedback whatsoever. Certain more senior guys watched the commit logs for any obvious shenanigans or idiocy, but generally changes were within customers' hands within a day or two (and this was a desktop app!). They suffered some horrifying stability problem for a while that necessitated some new policies. It was certainly liberating to work in that environment, but I can see why many places have safeties in place.
Look at some of the major Web2.0 kerfuffles in recent years. Off the top of my head, I can think of:
- The HD/DVD mutiny on Digg
- Public suicides on both JoelOnSoftware and Justin.TV
- Reddit storing passwords in cleartext and them getting stolen off a laptop
- Ariel Waldman and the Twitter harassment fiasco
- The Flickr censorship debate here: http://www.flickr.com/help/forum/40074/page3/
Yeah, you could argue that those services are all still around, so obviously they haven't been hurt too much. How much management time was wasted in dealing with them? How many non-users decided not to become users because of something they heard 3rd-hand about how it's a terrible company?
(And I don't think the solution is to never innovate. I can understand why a middle manager at a big company would think so, though - a PR disaster is by definition public and disastrous. In the short run, it always makes more sense to not mess with success, it's just that this doesn't lead to success in the long run.)
I've dealt with the security teams at large companies. They end up implementing sometimes draconian policies, but it seems that programmers refuse to learn to write secure code any other way. I have to admit that, if you consider the large company as being a legitimate entity, they're doing legitimate work.
Most of these incidents caused by employees taking out the data, but there are some cases caused by sloppy web app.
(I could find only Japanese version of the report: http://www.jnsa.org/result/2007/pol/incident/index.html )
And this is an error that one unchecked programmer would be perfectly able to make if there were no checks in place.
Case in point: I work for a $3Bn company and a year ago I developed a feature that saved the business ~$2M that year. We know the amount because when the problem raised its head we called together a number of people to calculate what the cost would be and estimates were between $1.5M - $3.5M if we couldn't find a solution. Not loss of revenue, but we would have actually had to pay out that much in support costs. The problem was highly visible across the organization and I was able to find a way to solve the it in a way that was completely invisible to the customer, implemented it and released to the field. My manager looked good, his manager looked good. I got an "average" performance review. I was pissed and to this day I haven't regained my motivation to do that caliber of work again. I wonder if I ever will without leaving!
As companies grow big, they like to emulate larger companies, and hence the people they like to hire are from larger companies since they understand scale -- this comes with a lot of baggage ofcourse. This in turn, most times, brings with it a culture of people who are worried more about not screwing up, covering their own ass, and thinking about getting promoted rather than doing what's right for the company or the business and screwing up as a byproduct of making quick decisions.
Re Joel Spolsky: The cost of the sale is not only the cost of the product, but what an organization would pay for a software/service, and what they would value it at. This is why most companies have a sales force, and don't advertise their prices on their website. As Joel mentions, for a lot of companies, charging less than $50k is a rounding error and not worth their time.
The flip side of people at big companies buying is based not entirely on the performance of the service or the product. As long as the service/product is reliable & ok, and it doesn't screw up in any major ways, they won't get fired for making that buying decision, rather than looking at ways of maximizing/optimizing the performance of the product -- yet another reason committees make buying decisions.
Re SOX: I like what founder's fund + facebook is doing -- letting early employees cash some of their equity out, thereby increasing the time early employees will stay with the company.
I also like Fred Wilson's thoughts on a secondary market for startup stock (similar to what goog does), this would in some sense, I guess let you be a private company, and not deal with the challenges associated with SOX compliance and at the same time not raise a highly dilutive Series D, in addition there is some sort of liquidation event for the investors.
PS: (std. Buchheit comments about Limited Life Experiences + Overgeneralization = Advice apply)
PPS: Say hi to the reddit guys for me ;-)
A two-week release-process lag is so much worse than it might sound. If you can ship instantly, you can get feedback and make improvements rapidly, over and over. Sure, without a careful release process you might occasionally break things, but when you do you can fix them in minutes rather than weeks. When you insert release-process lag into that cycle, improvements that might have been made in a rapid series of releases over a day or two can take months. Sometimes those improvements just never get made, because no one is motivated to keep working on them for so long.
In software, "stable" often means "unchanging", but only rarely means "high-quality".
But here's another twist: when you start working for a giant company that provides myriad services under the same login, suddenly everyone has to move as slowly as the slowest part of the business. It doesn't matter if you are just doing Yippee! Backgammon, because who knows -- maybe you could somehow accidentally expose the data at Yippee! Payment Solutions.
This is a completely rational consequence of being a giant company and delivering multiple web services under the same login. It is possible to have slightly different authentication policies for each service. But I wonder if OpenID providers have thought about this enough.
He has made that point in other essays.
Programmers, though, like it better when they write more code. Or more precisely, when they release more code. Programmers like to make a difference. Good ones, anyway.
Salesman like it best when they can sell more. When I used to sell used cars, I loved it best when I could keep pushing cars out of the lot.
Writers like it best when they can write more. They like it best when their imagination can produce thousands of stories.
The difference is that coders usually know what they will tackle and have a strategy so it is easier for them to get to the end point. Also they are almost sure they will hit the end point of a certain problem.
With other professions you cannot predict much. Your next customer is not guaranteed, your next story just does not want to show up in your brain etc....
That being said, Yes, a lot of programmers I know work hard.
Also, since most workers don't love their job, it is probably also true that good programmers are unlike many workers. To say that programmers are unlike many TYPES OF workers we'd have to believe that a higher proportion of programmers love their job than do workers in other types of work. From what I have seen on HN (albeit sample biased) I'd say that is true.
People like to make a difference. Good ones, anyway.
They like to get paid for it, they can even be greedy. They like to get recognition for it, that can go to far too.
But they also enjoy making the sale.
to me, hacking > sales (no, charity/non-profit/giving away money doesn't count -> there's moral incentive there)
In terms of time, they have a policy of permission vs. forgiveness. You need to be prepared to fail early – but that’s okay! If a prototype takes less than two days, don’t worry: just go ahead and do it. It if takes more... you should probably have permission.
http://www.gamasutra.com/php-bin/news_index.php?story=11628
I wonder if the problem at the recently troubled companies (AIG, Merrill, Lehman etc) was not enough checks in place or not enough power allotted to the employees who knew something was wrong.
In a startup there's noone to blame but you, and thus no need for politics. As anyone who has worked in middle management or above in a large company can tell you half of the job is making sure you can always find someone else to blame, while taking credit for the successes of others.
Steve Yegge has had the unenviable luck of working on 3 products all of which have not shipped (cited as business reasons) yet he still works for Google ~ http://blog.stackoverflow.com/2008/10/podcast-25/ So it seems there is something else going on that keeps programmers on board. Is it money, having the freedom to talking about it the process? I don't know.
I worked on it because I love programming and trying new things. Everything I was doing at the time was new to me so it was fun, engaging, and a learning experience. Plus, it beat the hell out of anything else I could have been doing in that wasteland ;)
Good explanation.
I put it as a question as I'm wasn't conclusively sure why people would sign on to assignments that have a probability of failure. But I've since found that smart organisations allow pursuit of training and learning as a reason to keep staff and motivation. I figure the USF know a thing or 2 about motivation. Hope your putting the lessons you learned into some hack your working on :)
Big companies on the other hand, start with a company worth $100,000,000, and want to turn it into a company worth $150,000,000. They value steady process improvement that doesn't risk their existing revenue -- that's why they want good managers.
It's important to know this if you're trying to sell to a big company as well as if you're trying to compete with a big company.
I guess it would be possible to note in a bug tracking app "we would have deployed here" and then see how many, and what kind of bugs were found between then and actual deployment, which checks found them, and whether any of those were rollback-causers.
That would give an estimate of how much good the checks did in each case, and a way to choose which checks are ineffective for their cost. (Ish - some checks may only trigger once, ever, to be priceless).
Imagine you have to fix a simple bug, like the description of an option in an HTML page.
Imagine that: the codebase is a bajillion lines of Java or C++ that no one person truly understands any more.
Imagine that: fucking up is considered bad. Maybe you're protecting important user data, or maybe it's just code for the online store, and even an hour of downtime can wipe out the day's profits.
The team is so big that your feature is scheduled for deployment along with four or five others this week.
As usual, management has skimped on QA so there are not enough of them to go around. Also, QA was added late, so there's no fast automated test suite. It takes them a full week of manual and semi-automatic testing to go through all regressions.
After that week of testing in development is over, then it goes to staging, where it's tested with real live data and staging subsystems.
At least somewhere in this, something breaks (something is always breaking) and a fix is produced. Then the new build is tested all over again, with the basic smoke tests as well as any test suites that are applicable to that unit of code which broke.
This is causing another trainwreck because we promised feature X to client Y by end of the month. What do we do? Emergency team meeting! Let's rebalance the work schedule.
Meanwhile, your change, which just a simple HTML fix, languishes as this build goes into its second round of testing.
Oh, and did we mention that you have to have submitted your code in time for translation into six languages? I'm afraid that the French translator did not produce a satisfactory translation in time, so we're going to have to hold your change back. Can you revert this and put it in next week's build? Thanks.
I don't think the claim that they would not have broken anything can be backed by facts.
If you look at Microsoft, for instance, their programmers used to write very buggy code, but with the introduction of better processes, like their Secure Development Lifecycle, they saw a substantial improvement in code quality (as measured through security and reliability metrics).
The damage done by not following a quality assurance process and writing buggy code was so big that Microsoft's image will be affected for a long time, even though they have now improved the code quality.
Even these particular guys certainly would have written buggy code. A QA process could correct it.
So I have little respect for formal QA.
Also, bugs do appear in production if you can release code the same day you wrote it, but you can fix it as soon as you find out about it. In big companies that work with milestones (sprints in agile speak), only really critical bugs can be fixed between releases. Not to mention that most bugs are fixed after code-freeze, when all the deliverables of that milestone are ready, long after you find out about it.
So I guess it's best to have QA, but also the freedom to deploy code on production whenever you want.
I think it is very cool to have the comments here instead of his website. It is nice to have them consolidated and not some comments here, some there.