Ask HN: I built a site that just went ridiculously viral. What do I do now?
So I built http://threewords.me - literally an MVP that I posted on Facebook and my friends started using. Fast forward 3 days. The entire Twitter results page for "threewords.me" is of tweets that happened less than five minutes ago. The site grew 2x in pageviews over the past hour.
Two problems: 1) what do I do now? 2) how do I afford this?
Advertising? Hosting partner? ...daresay... investment?
Sorry to be brief. If you're curious, the stack is Rails + Ruby Enterprise Edition + Passenger + nginx, which isn't cheap like PHP to host. On the $40 Linode right now but maxing out CPU at 350%.
227K pageviews today. 50K uniques. 71% traffic referred. 8,285 users. 3,100 new users in the past 1 hour.
What happens now?
EDIT: Hello! This thread is not going unnoticed while I take my metaphorical fire extinguisher to the fires that are happening. Will reply soon.
EDIT: cranked the Linode up to 4096. $160 server, woo!
EDIT: David from Duostack (http://duostack.com) is helping with the load on his cloud Ruby platform. Many many thanks to him.
194 comments
[ 2.8 ms ] story [ 242 ms ] threadI need to up the copy on the site.
This way - you could target ads to FB accounts/users that are all "snowboarders"
Or you can at least search who among your friends have the tag [snowboarder].
Or if you give more categories to the tags, you can ask your friends to fill in their favorite 3 [X] -- so you can simply ask all your friends a question and then build an interesting DB of tags/info/interests of all FB users.
http://www.facebook.com/pages/Snowboarding/107496599279538 1,784,641 people like this 9 Friends Like This
I don't know if you read the "Sacked by a Google Algorithm" article from yesterday. http://www.duckworksmagazine.com/11/columns/guest/winter/ind... The comment you make on the results page sounds dangerously close to what got Adsense to boot him.
A few people have posted my URL to social news sites like Reddit and Digg, but even when those gain traction it's usually just a few thousand and quite short-lived.
Sites like this typically see eCPM between $0.10 and $0.30 for advertising. So you're looking at around $30/day revenue right now.
Consider switching to EC2. This will allow you to easily scale up to a more powerful server if growth continues or down to a less powerful one if traffic tapers off.
Depending on how you set things up, a common bottleneck for Facebook applications is API calls. Ensure you are not making any API calls from Rails, since this will cause Rails processes to hang until Facebook returns results. With high traffic this could easily cause your server to become overwhelmed. Move all API calls to Delayed Job or another background processing system if you haven't already.
It looks like you have relatively few remotely loaded assets, which is great. You can likely speed things up more by making sure that your database is well optimized for the server (Make sure query cache is enabled, and you have indexes on the right columns for example), and identify any actions that take a long time to return.
As for what to expect in the future, growth will not be infinite. There are a finite number of people on earth that are going to love your site and Facebook integration is a fantastic way to reach nearly all of those people in a very short amount of time. After a while most of those people will get bored and you will see traffic begin to fall. The best ways to address this are by consistently releasing new features to reengage users or by referring your users to other similar sites that you build.
In general the biggest trait is that the app is built around sharing. Apps where sharing is a secondary feature generally do much worse in comparison to those where the whole point of the app is to share something. Even successful games on the Facebook platform universally require users to share content with friends to progress.
Many of the most basic viral apps are simply based around sending gifts to friends. If 10% of users that receive some sort of virtual gift then go on to send it to an average of 11 friends, your app will be highly viral in no time.
Without knowing more about your setup I'd offload assets to S3 and switch from Passenger to Unicorn.
Good tip on Unicorn. Any benchmarks on the performance gain over Passenger+REE?
But something you could do immediately to help with the load is to run haproxy on your current linode and funnel some of the traffic to a new linode that you spin up.
I suggest migrating it to Heroku. Push your code, copy over the database, and crank your dynos up. They charge by the hour, so as an example if you have ~10 dynos running (roughly equivalent to 10 thin instances, but slightly better) for 5 days (or however long it takes for the initial traffic surge to subside), you're only going to pay about $50. Then, you can turn down your dyno count and pay a reasonable monthly fee.
Anyway Mark, I have a bunch of apps on Heroku that use all kinds of wacky stuff—including the aforementioned gems—so email me at jarin (at) robotmodehq.com if you decide to try out Heroku and get stuck. I can send over working code snippets that you should be able to just drop in.
Think about moving the code over to Heroku now, and keep scaling to meet demand.
If you don't have the cash, email me and I'll front you what you need to keep it up until you figure out how you want to roll with this. This is not a time to trip over pennies.
Congratulations.
A more traditional approach of 1 web and 1 db server (possibly even physical boxes) might end up providing better value and give him enough time to figure out how to scale this horizontally on EC2 (the web part should be easy...remove any in memory state data, the db might be a bit more work).
Even a small machine can trivially handle 150k pageviews per hour, even with ruby strapped around its neck.
Spiky traffic is an issue of its own, but it sounds like Mark's site is growing fairly steadily (albeit steeply).
I agree with latch that heroku becomes (ridiculously) overpriced the further you depart from the free plan. If a stopgap is urgently needed then it may of course still be a valid choice. But in the midterm, if you're pushing serious traffic, look elsewhere.
I have yet to use them at higher loads. We shall see.
Also, it's my understanding that they're still less expensive than something like EngineYard, and still generally cheaper than paying for a sysadmin. Dedicated hardware will of course outperform any of this stuff, but I also don't have to think about it at all. Even VPSes involve spending initial setup time and then periods of upkeep, you can't just install and say "done."
Then again, I'm coming from a mostly theoretical standpoint. I don't have real world experience at true scale.
With regard to heroku you have a point about the "not needing a sysadmin" part. That, however, only works for a fairly short period during the lifecycle of a business; heroku is ideal for bootstrapping.
Once your site grows to the scale that mark is seeing (if that persists, which I doubt) then the heroku value proposition rapidly shrinks.
At that point you start to need more customization than heroku can provide [for a reasonable price] and you also need at least one person with systems knowledge in your company to prevent expensive mistakes in the software architecture.
Gladly that person will then more or less pay for itself, simply by moving the app to a cheaper hosting platform.
Edit: Google Cache got it: http://webcache.googleusercontent.com/search?sclient=psy&...
EDIT: Cranked to 4096
Xen is pretty great about partitioning everything except for I/O. Network I/O probably won't be a problem for you (xen doesn't really start getting tripped up until you are well above 1G/sec. I /believe/ linode has a 10,000 packet per second limit on top of that, but that's a whole lot of packets. This is probably not your problem.) but disk I/O will be.
Cache everything. If you cache enough, and have enough ram, slow disk I/O may not be a problem at all. watch your linode to make sure you don't go over quota on your disk I/O allocation... maybe email support to make sure you are in the clear. My understanding is that linode has better disk I/O than I do because they have a process in place for limiting heavy disk users. (my understanding is that it's a good process, they warn you before limiting you and generally do it in a fair manner. I plan on emulating them in this regard. )
No two ways around it; I/O to spinning disk and virtualization go together poorly. What would have been a sequential access on a dedicated server becomes random access... and on a good day, that means 1/10th the performance.
If you are using a lot of disk I/O, a dedicated server is about your only option. Considering the prices nearly all American providers charge, if you plan on keeping it for more than 2 months, you are nearly always better off buying. (and that way you can buy better disk, too.)
Of course, if you aren't limited by I/O, or if you can solve the I/O problem using ram caches, then it might make sense to stay 'in the cloud' on linode or ec2 or the like until you are sure you are going to need the space long-term.
I was able to sign up and see just enough to get a feel before the site became unresponsive. I like that it's dead simple. I can also see how it's incredibly viral as it asks you to ask others to describe you -- very smart. At first glance it looks to me like something that could enjoy tons of traffic, but which would probably be short-lived. The challenge would be finding a way to have that traffic stick around...
I view this as a bit of a long shot, but I would do a few things. First, you've got to have the site handle the traffic. No advice on how, but find a way to get that done. Next, I would try to become known as the place to get a quick summary on anybody -- even celebrities. I imagine people will receive multiple adjectives from different people, but I'd have the site tally the three most popular and promote those as best describing of the user. :) I see a couple of possible pages for "viewer stickiness". The homepage could feature very popular users. Imagine showing the three most submitted words to describe /paulg or /marrington and other users with large responses... Next, on each user's page it could show their "friends" and the words that describe them. The site also provides the most recently submitted three words on the user's page. This might at least make each user curious about checking back at their own profile to see how people were labeling them. As for monetizing, that's probably with ads, as usual, but you have to be careful about when and where to put them. Like I said, probably a long shot, but who knows? Good luck!
You mean like About.me is attempting to do? This is an interesting twist though...
However - the quick summary I look for on anyone of interest (professionally) is linkedin.
I do not and shall not have a facebook account.
A gossipy feel might be better anyway. Imagine the "word status" on Paris Hilton going from "hot" to "classy", "generous" and "caring" or something like that... ;)
How about analytics? For each person who writes about me, I could pay to see where in the world they lived, how long it took them to type in the three words (or even watch a movie of them typing them in), whether they clicked the link that was on my Facebook or the one on my Twitter feed (based on HTTP referrer), etc.
If you were confident you could give each user the ability to see this information for 1 user for free, and then have them pay if they wanted to see it for the rest of the users.
1. Implement caching. 2. Get faster servers to handle your 'debutant' phase' ex: EC2 3. bask in limelight, enjoy!
long term: app engine (just port it to webapp framework)
App engine after that b/c it should be fairly easy to port and it's an extremely headache-free way to host an app. Now that they have always on instances, it's very fast and scales efficiently and effortlessly.
As a DB they say it's designed for SPEED and SCALE. It might be worth looking at. (who's got the time right?) :)
Understatement of the year.
Entirely necessary here, even a hint of hyperbole and the pedants come out of the woodwork.
threewords for threewords.me: hot new cool
thats how i survived a mini fb app thatwent from zero to 44 million users in 7 days... rackspace cloud + memcache + redis insert&update queue + round robin. after a month i even killed the rackspace cloud interface and api with over 80 nodes and growing. had to get the memory limit on my account raised twice. peak were around 40k req/sec. after the second month we moved it to four bulky servers (16 core, raid5, 32gb ram) each cost around 12k dollar.
the whole secret is caching. everything, everywhere and as much as possible.
1. Upgrade your linode to something around 2 Gig for now.
2. Start playing with your MySQL or Postgresql config (assuming you're not already using redis or something similar, in which case just throw more RAM at the box). Google performance settings for your DB of choice and look for relevant posts to the memory you have.
3. If there's anything cachable, install memcached and implement it.
4. Purchase a second linode and move your database to it.
5. Upgrade each VPS as needed.
6. At this point you get into fancy scaling and there are loads of options. You should have plenty of breathing room by this point though and can plan it out.
Maybe in put down your follow ideas in short little phrases and have people provide three words on that, if they wish, along with three words on their friends. It will give you great insight into what THIS audience thinks is a good or bad next step for you.
This is a site that will not maintain interest as it is beyond a short period unless you add more to it. So the key here is to move forward quickly, even if it is in tiny incremental steps. letting the users help you make some of those choices of what steps to take can only help.
Hi there, your home page is very simple, render it in plain HTML and post the signup to queue.php.
In queue.php put:
<?php $data = serialize($_POST); mysql_connect('localhost','dbuser','dbpass'); mysql_select_db('dbname'); mysql_query('INSERT INTO cache(cache_data) VALUES(\''.mysql_real_escape_string($data).'\')'); echo 'YOUR REAL PAGE SHOULD BE HERE WITH A MESSAGE ABOUT BEING IN THE QUEUE'; ?>
The cache table should just be a PK cache_id and a text field (or perhaps mediumtext). I've suggested using a database instead of a filesystem based cache because, if you're doing this quickly, doing it using MySQL means you're less likely to run into file permissions or security problems.
Then you're at least capturing everyone's information whilst you figure out how to scale.