Lately it seems that every article I read about web technology is about scaling techniques. This stuff is fun to hack on but I have to agree that it's very unlikely that it should be the focus of any small company. I spoke recently with an engineer from a small but profitable and highly visible specialized e-commerce site. He runs their site quite comfortably and inexpensively on two machines.
I once worked for a start-up that had a grand vision of lightweight real-time messaging. This was 1996, and we were going to broadcast in real time, to zillions of clients, the events of the Summer Olympics. We had "push" stuff that worked really well, we had permission from IBM to do periodic scrapes of their event database, and we came up with a scaling system for handling feeds to massive numbers of clients. This was going to be a fantastic demo for potential customers. The publicity was going to be great.
We got everything ready, rented some co-lo space, got a bunch of servers set up (I think each miserable little Pentium 90 server could handle tens of thousands of clients), and . . . our top was maybe a couple dozen concurrent users. We could have run the thing out of our offices on our worst sales person's crummiest laptop. While that sales guy played Quake.
[Later, our /best/ sales guy had a patter that went like: "Well, we did this real-time monitoring app for the Olympics, and it scaled to ten, twenty thousand clients on a machine. You know how many we got? THIRTY."
Start-up was eventually bought, and then transitively bought, and I wound up with some shares of Oracle (spit).]
The best advice I've ever heard regarding scaling is to only worry about one level of magnitude. If you have 20 users, make sure you can scale to 200 - any farther than that and you're likely wasting your time.
I've never understood the obsession with scale when there are so many challenging problems left in user interface technology are relatively ignored. The room for innovation is staggering, yet we obsess over nosql.
Why is scaling such a popular obsession? Some hypotheses:
It's survivor bias. When you go to found a venture, where do you look for inspiration? You find a very successful analogue and read all about them. For example, if you want to found a social network, you look to Facebook for inspiration. And what is something that many incredibly successful web properties are worried about? Scaling.
It's objective, or it feels objective. You load the web page, it takes 20ms. You make a tweak and load the page again, it takes 15ms. You can measure benchmarks, and then you can point to the benchmarks getting better and claim that you were objectively successful. The fact that there's a scoring system lures people into the game.
It's fun to think about. Nobody likes the guy who points out the danger that the product will flop. Everyone likes the guy who points out that the product might just be so successful that we'll be overwhelmed by success. You get to sound like an ambitious, expansive optimist when planning out all the fancy hardware you're going to need. You're focusing on the positive, just as all the self-help manuals tell you to do.
It's Parkinson's Law, in its original form: By default, the size of a business unit grows until it is limited by resources, because the path to power in an organization is to be big. Lots of subordinates, lots of infrastructure, lots of budget. You might think that an organization would lavishly reward the person who could singlehandedly run the entire web infrastructure off of a single PC, but often the opposite is the case. So it pays to puff up the scalability problem as big as you can, in order to get the resources to build a big team, so that you gain political power from being the head of a big team...
I really think it's as simple as that. When you read about a successful startup, they invariably mention the troubles they had scaling. So you immediately think "how can I avoid scaling troubles?", when you should be thinking "how can I build a site that will have scaling troubles?"
>you should be thinking "how can I build a site that will have scaling troubles?"
I work for a company that does consulting for startups, and occasionally one of our clients takes off like a rocket. Everyone has scaling problems, even the people who have thought about and worked on scaling. Load tests and projections are only so good: at the end of the day theory will meet practice and we all know who wins there.
We always tell them the same thing, "these are good problems to have". Everyone has a sleepless week or so until the bottlenecks are cleared, and then life is back to normal (except now people have heard about that thing you've been working on for the past couple years).
There are many startups that should just buy a dual-quad core Nehalem (or AMD Magny-Cours) system with redundant power supplies, 32GB or more of RAM, and fast disks (or even spend on a RAID5 of SSDs) and stop worrying about scaling for the meantime.
It's like a game of poker: always leave yourself outs. Have a few plans for how you can scale beyond "more RAM and faster disks" if the need arises, but don't move on them until you have to.
The problem for me is not hardware, got a couple of older machines here that could easily run my current project (nginx + node + mongo), but the problem is symmetrical bandwidth is fabulously expensive.
Certainly the most disappointing startup experience I ever had was with an early Internet firm that had spent too much of '95-6 building their hardware and most especially software (custom C++ database) to insanely scale, taking so long their angel timed out due to a sudden need to pump money into another venture.
When I joined they were closing with new investors who turned out to be devils only interested in owning 100% of nothing, they essentially ran it as a vanity company after chasing off almost all the competent staff, KPCB who got very interested in what we were doing and who could have made a fantastic difference due to their relationship with Netscape etc.
That plus the company didn't do any Customer Development, ran in total stealth mode and spin in circles a whole lot theorizing about how people might use our service; as Blank says, there are no facts in the office, only opinions.
And we bet too much on our initial debut splash, which turned out to be a dull thud due to the incompetence of our marketing guy and/or the tech media (every article focused on the least attractive way out of 3 to use our service which was downloading a plug-in for your browser).
Netword. The idea was that companies and people could buy Networds like "Intel annual report" and you could e.g. type into your browser address [whatever that widget it called]:
netword/intel annual report
And you'd get redirected to the right page on Intel's site (at the time both Netscape and Microsoft Did The Right Thing with the above example (e.g. add .com)).
Besides solving search and design problems that hadn't yet been well addressed (this was before Google and truly ubiquitous use of search engines and at the dawn of usable site design) it could have bridged the old media to new media gap. I.e. "Netword foo bar" works a whole lot better than http://www.mycompany.com/foo/bar or whatever, it's something you might be able to remember from a radio, TV, highway sign etc. ad, less obnoxious on print ads etc.
A lot of the upfront development effort was devoted to dealing with misspellings quickly (the custom C++ tree database) and database replication so that it could insanely scale. We were using Digital Alpha hardware so we could have cheated in all sorts of ways just to get launched and then do Customer Development, (ADDED:) but there was this overriding fear that we'd get crushed with demand immediately after launch.
1) Know your bottlenecks and write them down. This is where you will look first if you have problems.
2) Know how you will throw hardware at it or change configurations if you find yourself with sudden growth you didn't expect. You need to create a buffer so you have the time to solve real code issues. Write it down.
3) Whenever you make an architectural choice, make sure there isn't one that's equivalent in engineer time that will be easier to extend later.
Otherwise, just do whatever helps you learn about the product the quickest.
I think there is an exception to this idea for startups that require data-centric scaling. If you are planning to grab as much data as you can and as quickly as possible, this could require scaling even when the user count itself is small. I had a startup that ground to a halt because I hadn't anticipated the sheer amount of data that the app would collect even before I had more than a handful of users. Of course, that isn't the traditional kind of scaling that people talk about.
You could even write that more starkly: worry about supporting your customers because customer support is the only thing that you can't scale easily, everything else will eventually work out. Are there lists of start-ups that failed because of scaling issues? Are there comparable lists of start-ups that failed because of marketing, sales, product, vision, support and a 100 other pitfalls that a start-up can easily fall in to?
I could not name a single company that folded because they could not handle the luxury problem of scaling. Because really, when scaling is your problem, you have no problem, you've figured out a way to beat the odds on all that other stuff so scaling will be solved as well, one way or another.
As Friendster became more popular, its overwhelmed Web site became slower. Things would become so bad that a Friendster Web page took as long as 40 seconds to download. Yet, from where Mr. Lindstrom sat, technical difficulties proved too pedestrian for a board of this pedigree. The performance problems would come up, but the board devoted most of its time to talking about potential competitors and new features, such as the possibility of adding Internet phone services, or so-called voice over Internet protocol, or VoIP, to the site.
They didn't fold though, did they? They got bought by some asian consortium iirc. Also, the way you describe that it strikes me as a management problem more than a technical one. Once they had scaling issues they decided to not address them. They could have, but they didn't.
True, they didn't fold per se, but they likely could have held their own against MySpace and Facebook. They just got bought out a few months ago for $40 million, when they could have had a billion dollar valuation like Facebook or a half-billion dollar buyout like MySpace.
Maybe if they had thought about scaling from the very beginning (like so many startups supposedly waste time doing), they would have had a plan in place to address it without having to bring it up at board meetings.
They could have held their own against MySpace and Facebook if they'd been focused on solving actual technical problems and adding features people wanted to use, instead of solving theoretical technical problems.
When Friendster finally hired Joyce Park to lead a reimplementation of the site (in PHP instead of Java) there was so much conflict over this inside the company that they fired her immediately after she finished, ostensibly for blogging. Imagine what Friendster could have done if user-focused, practical people like Joyce were honored and listened to, instead of tossed out like trash.
(I know Joyce. I'd even say she's a friend of mine, and I hope she'd agree.)
They didn't fold, but they suffered a massive drop in usership. They rolled out a performance overhaul in June 2004 after the majority of users had already given up on the site.
Incidently, the blogger above was fired for posting about the release, so it would seem they had some other serious corporate issues beyond performance.
I worked at a startup where the relentless focus on being able to scale to support millions of users from day one hurt the business very severely and was one of two reasons that the company failed. Spending almost 50% of the venture funding raised on hardware that was never used when 10 servers would've sufficed was not a smart idea.
However, just for fun, I'll note here that we are/were an extreme case where we spent a lot of time thinking about scaling in the early months and it paid off when we finally got traction (100 visits per day to 2,000,000 visits per day in 30 days). Sometimes it does matter. Moral: don't follow advice blindly.
One of our goals is to create weather watchers in much the same way mint.com created budgeters (i.e. get people to enjoy an activity they previously considered mundane).
Its amusing that none of these comments even touch on the articles premise, which is that you need to be solving problems that someone (i.e. customers) else believes is important.
This is not really a startling revelation, businesses need revenue to succeed but where does revenue come from? Customers .. duh! They also need to attract customers to gain angel inventors, funding, and partnership deals.
I'm not sure that I agree with the author's opinion that time would be better spent innovating on User Interface / usability. Having run a business since 2000, I would say that delivering an innovative service to a very specific marketplace and at the same time developing a rapport with that segment should be the ultimate goal of any startup.
I think it'd be an oversight to not stay on top of the new technologies that scale better.
A document store fits a lot of problems much better than doing everything in Postgres, and offers easier maintenance and replication. And can offer a lot simpler architecture when all is said and done.
But yes, obsessing about scaling on an app that has no users can be a very big time sink. Much better to just hack out some software with whatever tools are the most productive, and get a few customers to drive the human side of business requirements.
Because lots of startup die for not having customers enough (or at all!), but I haven’t heard of any died because at first was not able to scale millions of users.
44 comments
[ 0.61 ms ] story [ 90.1 ms ] threadWe got everything ready, rented some co-lo space, got a bunch of servers set up (I think each miserable little Pentium 90 server could handle tens of thousands of clients), and . . . our top was maybe a couple dozen concurrent users. We could have run the thing out of our offices on our worst sales person's crummiest laptop. While that sales guy played Quake.
[Later, our /best/ sales guy had a patter that went like: "Well, we did this real-time monitoring app for the Olympics, and it scaled to ten, twenty thousand clients on a machine. You know how many we got? THIRTY."
Start-up was eventually bought, and then transitively bought, and I wound up with some shares of Oracle (spit).]
Which basically boils down to don't worry about performance in the +/- 50% range, worry about performance when you are going to see x^3 or +/- 5,000%.
It's survivor bias. When you go to found a venture, where do you look for inspiration? You find a very successful analogue and read all about them. For example, if you want to found a social network, you look to Facebook for inspiration. And what is something that many incredibly successful web properties are worried about? Scaling.
It's objective, or it feels objective. You load the web page, it takes 20ms. You make a tweak and load the page again, it takes 15ms. You can measure benchmarks, and then you can point to the benchmarks getting better and claim that you were objectively successful. The fact that there's a scoring system lures people into the game.
It's fun to think about. Nobody likes the guy who points out the danger that the product will flop. Everyone likes the guy who points out that the product might just be so successful that we'll be overwhelmed by success. You get to sound like an ambitious, expansive optimist when planning out all the fancy hardware you're going to need. You're focusing on the positive, just as all the self-help manuals tell you to do.
It's Parkinson's Law, in its original form: By default, the size of a business unit grows until it is limited by resources, because the path to power in an organization is to be big. Lots of subordinates, lots of infrastructure, lots of budget. You might think that an organization would lavishly reward the person who could singlehandedly run the entire web infrastructure off of a single PC, but often the opposite is the case. So it pays to puff up the scalability problem as big as you can, in order to get the resources to build a big team, so that you gain political power from being the head of a big team...
I really think it's as simple as that. When you read about a successful startup, they invariably mention the troubles they had scaling. So you immediately think "how can I avoid scaling troubles?", when you should be thinking "how can I build a site that will have scaling troubles?"
I work for a company that does consulting for startups, and occasionally one of our clients takes off like a rocket. Everyone has scaling problems, even the people who have thought about and worked on scaling. Load tests and projections are only so good: at the end of the day theory will meet practice and we all know who wins there.
We always tell them the same thing, "these are good problems to have". Everyone has a sleepless week or so until the bottlenecks are cleared, and then life is back to normal (except now people have heard about that thing you've been working on for the past couple years).
There are many startups that should just buy a dual-quad core Nehalem (or AMD Magny-Cours) system with redundant power supplies, 32GB or more of RAM, and fast disks (or even spend on a RAID5 of SSDs) and stop worrying about scaling for the meantime.
When I joined they were closing with new investors who turned out to be devils only interested in owning 100% of nothing, they essentially ran it as a vanity company after chasing off almost all the competent staff, KPCB who got very interested in what we were doing and who could have made a fantastic difference due to their relationship with Netscape etc.
That plus the company didn't do any Customer Development, ran in total stealth mode and spin in circles a whole lot theorizing about how people might use our service; as Blank says, there are no facts in the office, only opinions.
And we bet too much on our initial debut splash, which turned out to be a dull thud due to the incompetence of our marketing guy and/or the tech media (every article focused on the least attractive way out of 3 to use our service which was downloading a plug-in for your browser).
Besides solving search and design problems that hadn't yet been well addressed (this was before Google and truly ubiquitous use of search engines and at the dawn of usable site design) it could have bridged the old media to new media gap. I.e. "Netword foo bar" works a whole lot better than http://www.mycompany.com/foo/bar or whatever, it's something you might be able to remember from a radio, TV, highway sign etc. ad, less obnoxious on print ads etc.
A lot of the upfront development effort was devoted to dealing with misspellings quickly (the custom C++ tree database) and database replication so that it could insanely scale. We were using Digital Alpha hardware so we could have cheated in all sorts of ways just to get launched and then do Customer Development, (ADDED:) but there was this overriding fear that we'd get crushed with demand immediately after launch.
1) Know your bottlenecks and write them down. This is where you will look first if you have problems.
2) Know how you will throw hardware at it or change configurations if you find yourself with sudden growth you didn't expect. You need to create a buffer so you have the time to solve real code issues. Write it down.
3) Whenever you make an architectural choice, make sure there isn't one that's equivalent in engineer time that will be easier to extend later.
Otherwise, just do whatever helps you learn about the product the quickest.
I could not name a single company that folded because they could not handle the luxury problem of scaling. Because really, when scaling is your problem, you have no problem, you've figured out a way to beat the odds on all that other stuff so scaling will be solved as well, one way or another.
Friendster is an obvious example that comes to mind:
http://www.nytimes.com/2006/10/15/business/yourmoney/15frien...
As Friendster became more popular, its overwhelmed Web site became slower. Things would become so bad that a Friendster Web page took as long as 40 seconds to download. Yet, from where Mr. Lindstrom sat, technical difficulties proved too pedestrian for a board of this pedigree. The performance problems would come up, but the board devoted most of its time to talking about potential competitors and new features, such as the possibility of adding Internet phone services, or so-called voice over Internet protocol, or VoIP, to the site.
Of course that's stupid.
Maybe if they had thought about scaling from the very beginning (like so many startups supposedly waste time doing), they would have had a plan in place to address it without having to bring it up at board meetings.
When Friendster finally hired Joyce Park to lead a reimplementation of the site (in PHP instead of Java) there was so much conflict over this inside the company that they fired her immediately after she finished, ostensibly for blogging. Imagine what Friendster could have done if user-focused, practical people like Joyce were honored and listened to, instead of tossed out like trash.
(I know Joyce. I'd even say she's a friend of mine, and I hope she'd agree.)
http://troutgirl.wordpress.com/2004/06/29/friendster-goes-ph...
Incidently, the blogger above was fired for posting about the release, so it would seem they had some other serious corporate issues beyond performance.
However, just for fun, I'll note here that we are/were an extreme case where we spent a lot of time thinking about scaling in the early months and it paid off when we finally got traction (100 visits per day to 2,000,000 visits per day in 30 days). Sometimes it does matter. Moral: don't follow advice blindly.
One of our goals is to create weather watchers in much the same way mint.com created budgeters (i.e. get people to enjoy an activity they previously considered mundane).
I'm not sure that I agree with the author's opinion that time would be better spent innovating on User Interface / usability. Having run a business since 2000, I would say that delivering an innovative service to a very specific marketplace and at the same time developing a rapport with that segment should be the ultimate goal of any startup.
A document store fits a lot of problems much better than doing everything in Postgres, and offers easier maintenance and replication. And can offer a lot simpler architecture when all is said and done.
But yes, obsessing about scaling on an app that has no users can be a very big time sink. Much better to just hack out some software with whatever tools are the most productive, and get a few customers to drive the human side of business requirements.
...So you never heard of Friendster, then?
Dying from lack of customers looks completely different.