Ask HN: I built a site that just went ridiculously viral. What do I do now?

370 points by markbao ↗ HN
Hey HN,

So I built http://threewords.me - literally an MVP that I posted on Facebook and my friends started using. Fast forward 3 days. The entire Twitter results page for "threewords.me" is of tweets that happened less than five minutes ago. The site grew 2x in pageviews over the past hour.

Two problems: 1) what do I do now? 2) how do I afford this?

Advertising? Hosting partner? ...daresay... investment?

Sorry to be brief. If you're curious, the stack is Rails + Ruby Enterprise Edition + Passenger + nginx, which isn't cheap like PHP to host. On the $40 Linode right now but maxing out CPU at 350%.

227K pageviews today. 50K uniques. 71% traffic referred. 8,285 users. 3,100 new users in the past 1 hour.

What happens now?

EDIT: Hello! This thread is not going unnoticed while I take my metaphorical fire extinguisher to the fires that are happening. Will reply soon.

EDIT: cranked the Linode up to 4096. $160 server, woo!

EDIT: David from Duostack (http://duostack.com) is helping with the load on his cloud Ruby platform. Many many thanks to him.

194 comments

[ 2.8 ms ] story [ 242 ms ] thread
Just curious: what does it do? (no offense, but it asks for too many things and permissions for me to try it right now)
You don't even need Facebook connect to try it out, just sign up. Send the site to your friends, they give you three words anonymously, you see them.

I need to up the copy on the site.

Then why do you need all those Facebook permissions?
Ah, you're right. This is literally an MVP I threw together in 4 hours.
You can go to facebook and turn off the posting and offline access without disconnecting the app.
Well, it's a nifty idea and I know I've seen that trend on Facebook before. But that's also the problem - it's a 1 time use fad type thing. I don't know how much I'd invest in the current incarnation - perhaps just enough to keep the site functioning. Aim for some kind of traffic-based business strategy like ads or something. Maybe it can help pay for the site itself. If you could perhaps take the current idea and evolve it into something with a little more sustaining power then perhaps you could turn it into a nice little side income thing.
let the words become tags linked to the FB account. Such that if users have a tag like "Snowboarder" associated with their FB account - he can build out a tagDB of FB accounts.

This way - you could target ads to FB accounts/users that are all "snowboarders"

Or you can at least search who among your friends have the tag [snowboarder].

Or if you give more categories to the tags, you can ask your friends to fill in their favorite 3 [X] -- so you can simply ask all your friends a question and then build an interesting DB of tags/info/interests of all FB users.

the site got twenty two hundred hits within two hours?
Two hundred twenty thousand. In two hours.
220.000 hits in 2 hours is about 220000/3600/2 = 30 hits per second. The server needs to produce pages in about 33 msec. And all that is average; there could be spikes. Hmmm, interesting. And certainly doable. On my site I use memcached and that produces pages in 3 to 4 msec.
In the movie, the figure quoted was 22.000 hits, so 3 pages per second. Which is reasonably serviceable but still high for a new site.
Adsense. With only about 10k hits a day, my bedtime calculator http://sleepyti.me brings in $10-$20/day. At the rate you're getting users, you could probably bring in a significant load of cash. Good luck!
Hey Shaw, it sounds interesting. Can u guide me about how u started this project and contacted Adsense. I wanna start one like this and make some earning.
Depends on the niche and the website. I have made a blogspot blog and brought it to 4% CTR and $20 CPM. That's +$200 a day.
those are good stats. Care to share the URL?
"Sleepyti.me is provided as a free service thanks to support from our ads."

I don't know if you read the "Sacked by a Google Algorithm" article from yesterday. http://www.duckworksmagazine.com/11/columns/guest/winter/ind... The comment you make on the results page sounds dangerously close to what got Adsense to boot him.

Thanks for the tip. I saw that article on HN the other day, but the wording is designed to encourage people to leave adblock disabled to see relevant, sleep-related ads. You might be right though, it's always better safe than sorry; I'll look into clarifying the wording there. Wouldn't want to offend my Google overlords!
Curious: What kind of traffic sources are you getting? I notice an FB like button, is that the sole source?
Facebook and Twitter help spread the message to new users, as well as a nice daily referral from StumbleUpon.

A few people have posted my URL to social news sites like Reddit and Digg, but even when those gain traction it's usually just a few thousand and quite short-lived.

First, do not put ads on your site as it will annoy the users. Rather, take feedback and see what it takes to get more traffic. I guess you can double up your existing infrastructure to handle any surge in traffic. You do not want your users to see your page down due to excessive traffic.
"502 Bad Gateway" - hosting troubles.. Can you explain a bit more about what threewords.me is?
It's a site where you can ask people to describe you, in three words.
I am not sure how long it is going to last. i mean you will run out of friends to write about, Isn't it?
then it starts suggesting people. apply it to more than people. then it starts suggesting more than people. you get the picture.
I recently built a Facebook application that approached the same kind of growth.

Sites like this typically see eCPM between $0.10 and $0.30 for advertising. So you're looking at around $30/day revenue right now.

Consider switching to EC2. This will allow you to easily scale up to a more powerful server if growth continues or down to a less powerful one if traffic tapers off.

Depending on how you set things up, a common bottleneck for Facebook applications is API calls. Ensure you are not making any API calls from Rails, since this will cause Rails processes to hang until Facebook returns results. With high traffic this could easily cause your server to become overwhelmed. Move all API calls to Delayed Job or another background processing system if you haven't already.

It looks like you have relatively few remotely loaded assets, which is great. You can likely speed things up more by making sure that your database is well optimized for the server (Make sure query cache is enabled, and you have indexes on the right columns for example), and identify any actions that take a long time to return.

As for what to expect in the future, growth will not be infinite. There are a finite number of people on earth that are going to love your site and Facebook integration is a fantastic way to reach nearly all of those people in a very short amount of time. After a while most of those people will get bored and you will see traffic begin to fall. The best ways to address this are by consistently releasing new features to reengage users or by referring your users to other similar sites that you build.

Would it be possible to see your Facebook app? I'm curious to see what traits these fast growing apps have in common in person.
(comment deleted)
2 words: self-interest.
Sorry. It got banned from platform for a few reasons including tagging photos that didn't have people in them. (Similar to the cartoons that people upload and tag their friends as characters).

In general the biggest trait is that the app is built around sharing. Apps where sharing is a secondary feature generally do much worse in comparison to those where the whole point of the app is to share something. Even successful games on the Facebook platform universally require users to share content with friends to progress.

Many of the most basic viral apps are simply based around sending gifts to friends. If 10% of users that receive some sort of virtual gift then go on to send it to an average of 11 friends, your app will be highly viral in no time.

That's really interesting. My recent viral app, Profile Banner, also utilizes tagging of photos without people. How long did it take before your app was banned?
New Feature Idea if it's not already in the works: threewords.me places (foursquare?). would leverage current users engagement to tag locations with a dead simple description. could keep users coming back, provide utility(value)to businesses, and make you some more money. email me (mw63214@gmail.com) if you want more ideas, although it seems you have your hands full as it is. Good Luck!
Also, I don't know how time-intensive it would be, but it could be helpful to have some sort of drop-down box that initiates when users start typing in the three word spaces. Sort of like a google-instant-type thing. 5 synonyms pop up ranging from strong to weak. So if I started typing "h-o-", it would infer I was typing "hot" and (on the left, or strong side) it would have "blazing" and on the right(or weak side) it would have "warm". Or something of that nature. You get the picture. Might be easier for mobile folks to get the best possible description and faster.
What's your main bottleneck? What's driving that high CPU?

Without knowing more about your setup I'd offload assets to S3 and switch from Passenger to Unicorn.

User-specific assets are on S3... will need to offset the javascript and CSS there too.

Good tip on Unicorn. Any benchmarks on the performance gain over Passenger+REE?

Not sure if there's any benchmarks out there. But I had your similar setup when my app went viral. Switching to Unicorn w/REE replaced an expensive engineyard setup. Learned about it from github (https://github.com/blog/517-unicorn)

But something you could do immediately to help with the load is to run haproxy on your current linode and funnel some of the traffic to a new linode that you spin up.

Heh, I was wondering what this was when I first saw it a few hours ago. I've seen 5-6 people I follow on Tumblr use this thing already.

I suggest migrating it to Heroku. Push your code, copy over the database, and crank your dynos up. They charge by the hour, so as an example if you have ~10 dynos running (roughly equivalent to 10 thin instances, but slightly better) for 5 days (or however long it takes for the initial traffic surge to subside), you're only going to pay about $50. Then, you can turn down your dyno count and pay a reasonable monthly fee.

There's a bit of magic involved with things like compass/sass and Paperclip on Heroku (you have to use Amazon S3 for storage and there is some other setting I can check on).

Anyway Mark, I have a bunch of apps on Heroku that use all kinds of wacky stuff—including the aforementioned gems—so email me at jarin (at) robotmodehq.com if you decide to try out Heroku and get stuck. I can send over working code snippets that you should be able to just drop in.

I got a few emails about the Compass/SASS on Heroku thing, so I posted it up on my blog in case anyone who didn't email me needs it. Compass and SASS are notoriously painful to get up and running on Heroku, and everyone says to use the Hassle gem but it never works. So here's the SUPER MAGIC WAY: http://jarinheit.posterous.com/the-ultimate-secret-to-gettin...
Your site seems to be down.

Think about moving the code over to Heroku now, and keep scaling to meet demand.

If you don't have the cash, email me and I'll front you what you need to keep it up until you figure out how you want to roll with this. This is not a time to trip over pennies.

Congratulations.

Might not be a time to trip over pennies, but there's also no reason to rush into things. Heroku has always seemed to be horribly overpriced as a scaling solution.

A more traditional approach of 1 web and 1 db server (possibly even physical boxes) might end up providing better value and give him enough time to figure out how to scale this horizontally on EC2 (the web part should be easy...remove any in memory state data, the db might be a bit more work).

The release of Hackety was fairly large, I got LifeHackered, TUAWd... 50k uniques, 100k pageviews over three days. Heroku's free account didn't blink.
Sorry but that's nowhere near "huge" in terms of server load.

Even a small machine can trivially handle 150k pageviews per hour, even with ruby strapped around its neck.

Spiky traffic is an issue of its own, but it sounds like Mark's site is growing fairly steadily (albeit steeply).

I agree with latch that heroku becomes (ridiculously) overpriced the further you depart from the free plan. If a stopgap is urgently needed then it may of course still be a valid choice. But in the midterm, if you're pushing serious traffic, look elsewhere.

I'm just saying that it was more than I expected out of a totally free account, that's all.

I have yet to use them at higher loads. We shall see.

Also, it's my understanding that they're still less expensive than something like EngineYard, and still generally cheaper than paying for a sysadmin. Dedicated hardware will of course outperform any of this stuff, but I also don't have to think about it at all. Even VPSes involve spending initial setup time and then periods of upkeep, you can't just install and say "done."

Then again, I'm coming from a mostly theoretical standpoint. I don't have real world experience at true scale.

Yea sorry if I came across rude, didn't mean to attack you or heroku, just wanted to set the figures straight.

With regard to heroku you have a point about the "not needing a sysadmin" part. That, however, only works for a fairly short period during the lifecycle of a business; heroku is ideal for bootstrapping.

Once your site grows to the scale that mark is seeing (if that persists, which I doubt) then the heroku value proposition rapidly shrinks.

At that point you start to need more customization than heroku can provide [for a reasonable price] and you also need at least one person with systems knowledge in your company to prevent expensive mistakes in the software architecture.

Gladly that person will then more or less pay for itself, simply by moving the app to a cheaper hosting platform.

No worries, absolutely no offense taken. :)
This has happened to me a few times and usually the Twitter/social media storm blows over fairly quickly. I suggest getting an extra Linode for now and wait a few days to see what happens. If the numbers do stay up the site will probably be able to pay for itself with ads. Your site is down at the moment so I can't really see what it's about.

Edit: Google Cache got it: http://webcache.googleusercontent.com/search?sclient=psy&...

There isn't a twitter/social media storm ie- hype cycle. It's genuine growth with no press or even optimization for sharing built in. It can certainly die down, but it's doing it in a highly organic fashion.
Rather than get another Linode, for the very short term, I'd just crank the current Linode as big as they go (I assume Linode has super-quick resizing) and tell passenger to run a bunch more processes.
Just cranked it to 2048MB. Cranked Passenger to 55 workers in the pool. Using 400% CPU still. Linode's probably not happy.

EDIT: Cranked to 4096

I doubt they're unhappy. Resources are shared in such a way that everyone is guaranteed their fair share. I think the 400% reflects that you have access to multiple cores. 100%=maxing out one core, I believe. (And you might not be truly maxing it out, just maxing out your share of it.)
If you want to see how much CPU is being denied to you because other XEN VMs on the box are using it, in top, look at "steal" - that's the percentage of the CPU that you want that you are not getting because it's been allocated to other people.

Xen is pretty great about partitioning everything except for I/O. Network I/O probably won't be a problem for you (xen doesn't really start getting tripped up until you are well above 1G/sec. I /believe/ linode has a 10,000 packet per second limit on top of that, but that's a whole lot of packets. This is probably not your problem.) but disk I/O will be.

Cache everything. If you cache enough, and have enough ram, slow disk I/O may not be a problem at all. watch your linode to make sure you don't go over quota on your disk I/O allocation... maybe email support to make sure you are in the clear. My understanding is that linode has better disk I/O than I do because they have a process in place for limiting heavy disk users. (my understanding is that it's a good process, they warn you before limiting you and generally do it in a fair manner. I plan on emulating them in this regard. )

No two ways around it; I/O to spinning disk and virtualization go together poorly. What would have been a sequential access on a dedicated server becomes random access... and on a good day, that means 1/10th the performance.

If you are using a lot of disk I/O, a dedicated server is about your only option. Considering the prices nearly all American providers charge, if you plan on keeping it for more than 2 months, you are nearly always better off buying. (and that way you can buy better disk, too.)

Of course, if you aren't limited by I/O, or if you can solve the I/O problem using ram caches, then it might make sense to stay 'in the cloud' on linode or ec2 or the like until you are sure you are going to need the space long-term.

Hey Mark, first of all congratulations. Do you have a caching layer? If not now is the time to implement one.
No offense, but putting it on HN in the middle of traffic spike pressures would not help things. ;)

I was able to sign up and see just enough to get a feel before the site became unresponsive. I like that it's dead simple. I can also see how it's incredibly viral as it asks you to ask others to describe you -- very smart. At first glance it looks to me like something that could enjoy tons of traffic, but which would probably be short-lived. The challenge would be finding a way to have that traffic stick around...

I view this as a bit of a long shot, but I would do a few things. First, you've got to have the site handle the traffic. No advice on how, but find a way to get that done. Next, I would try to become known as the place to get a quick summary on anybody -- even celebrities. I imagine people will receive multiple adjectives from different people, but I'd have the site tally the three most popular and promote those as best describing of the user. :) I see a couple of possible pages for "viewer stickiness". The homepage could feature very popular users. Imagine showing the three most submitted words to describe /paulg or /marrington and other users with large responses... Next, on each user's page it could show their "friends" and the words that describe them. The site also provides the most recently submitted three words on the user's page. This might at least make each user curious about checking back at their own profile to see how people were labeling them. As for monetizing, that's probably with ads, as usual, but you have to be careful about when and where to put them. Like I said, probably a long shot, but who knows? Good luck!

>>Next, I would try to become known as the place to get a quick summary on anybody -- even celebrities.

You mean like About.me is attempting to do? This is an interesting twist though...

However - the quick summary I look for on anyone of interest (professionally) is linkedin.

I do not and shall not have a facebook account.

Yes, but I'd view this one as more more social/playful, not really serious. :)

A gossipy feel might be better anyway. Imagine the "word status" on Paris Hilton going from "hot" to "classy", "generous" and "caring" or something like that... ;)

Good idea. Like a trends.google for celebs.
>As for monetizing, that's probably with ads, as usual, but you have to be careful about when and where to put them.

How about analytics? For each person who writes about me, I could pay to see where in the world they lived, how long it took them to type in the three words (or even watch a movie of them typing them in), whether they clicked the link that was on my Facebook or the one on my Twitter feed (based on HTTP referrer), etc.

If you were confident you could give each user the ability to see this information for 1 user for free, and then have them pay if they wanted to see it for the rest of the users.

Hmm, that's interesting. Very popular users might find some use for that kind of thing. For example, up and coming music artists trying to build a following and connect with fans maybe... I definitely think sites should consider charging when and where possible, but sometimes there is good opportunity for it, and sometimes not.
yo mark, congrats. So what are your next steps? Grab an advertising network quick and throw it on there for now.
First of all, congrats on your problems :) I've signed up, and things are responsive.

1. Implement caching. 2. Get faster servers to handle your 'debutant' phase' ex: EC2 3. bask in limelight, enjoy!

short term: heroku

long term: app engine (just port it to webapp framework)

Why?
Heroku in the short term b/c you wrote it in rails and you can easily scale it up or down based on virality.

App engine after that b/c it should be fairly easy to port and it's an extremely headache-free way to host an app. Now that they have always on instances, it's very fast and scales efficiently and effortlessly.

FYI: Heroku is fantastic for scaling apps. But if it's your database that's the bottleneck...you've got problems. Because your options with SQL are scaling up or sharding, each quite painful. If you want any advice, ping me.
Perhaps rewriting the database to use MongoDB? Their http://www.mongodb.org/ tutorial took about 10 minutes and is really easy to get to know.

As a DB they say it's designed for SPEED and SCALE. It might be worth looking at. (who's got the time right?) :)

Alternatively, I've gotten a lot of mileage out of using redis as a cache to selectively optimize data piece by piece.
hey Mark, if you need server capacity I can set you up with four machines on very short notice, let me know. j@ww.com (no charge, help you handle the spike and see if it has staying power without having to get in to long term contracts).
^ This is reason why I love this place and never believe the occasional "hacker news is going to shit" posts.
To be fair, Reddit has a noted capacity for charity and generosity, but I do agree.
>To be fair, Reddit has a noted capacity for charity and generosity

Understatement of the year.

>Understatement

Entirely necessary here, even a hint of hyperbole and the pedants come out of the woodwork.

Entirely necessary?
This starts the Pink Panther theme playing in my head.
Jacques, you are, and continue to be, one of my favorite people on Hacker News. Thank you. Shooting you an email. :)
Haha I recieved links to threewords.me before I even opened HN this morning. I don't know how to solve your fires but I've been asked to tell you to let the text color be customised because otherwise you can make unreadable on certain backgrounds. Maybe you should setup a getsatisfaction-ish feedback/support area?

threewords for threewords.me: hot new cool

minimize the server work; get some cheap nodes at rackspace cloud; use one node for sessions in memcache; use two nodes for mongodb (master/slave) read from both; try to minimize xss/html/js and gzip everything; cache as much as possible; maybe even queue inserts/updates with redis and a cron;

thats how i survived a mini fb app thatwent from zero to 44 million users in 7 days... rackspace cloud + memcache + redis insert&update queue + round robin. after a month i even killed the rackspace cloud interface and api with over 80 nodes and growing. had to get the memory limit on my account raised twice. peak were around 40k req/sec. after the second month we moved it to four bulky servers (16 core, raid5, 32gb ram) each cost around 12k dollar.

the whole secret is caching. everything, everywhere and as much as possible.

which was mini fb-app, if you don't mind me asking?
That's why I use google appengine. It would have handled what you describe without even going over the daily free limit. Of course, you wouldn't be cool as rails.
You could be cool as Clojure, though: https://github.com/gcv/appengine-magic
Whats the start-up time? i.e. when google spins up a new instance to handle traffic, how quickly can it come up?
In the free configuration, I've clocked it between 4 and 6 seconds. (Sometimes up to 10 seconds, but not recently.) With the 1.4.0 release of the App Engine SDK, the cold start time has become less important, though. First, you can pay $9/month to have three instances running all the time. Second, App Engine can send a "warmup" request to your application.
The following until you no longer have issues handling the traffic (which might be right after step 2).

1. Upgrade your linode to something around 2 Gig for now.

2. Start playing with your MySQL or Postgresql config (assuming you're not already using redis or something similar, in which case just throw more RAM at the box). Google performance settings for your DB of choice and look for relevant posts to the memory you have.

3. If there's anything cachable, install memcached and implement it.

4. Purchase a second linode and move your database to it.

5. Upgrade each VPS as needed.

6. At this point you get into fancy scaling and there are loads of options. You should have plenty of breathing room by this point though and can plan it out.

Upgrading to the 4 gig linode as we speak. this is getting kind of ridiculous.
The linode guys are great. You might want to flip them an email and let them know you're getting hit. Could speed up the resizing etc.
Hey Mark, it's been a while. Nicely done. You said it was just an MVP so I assume you have a product designed beyond this point. So, NOW is the time to do some validation on what those other ideas are. You have a large audience who already know who you are and are willingly giving you input.

Maybe in put down your follow ideas in short little phrases and have people provide three words on that, if they wish, along with three words on their friends. It will give you great insight into what THIS audience thinks is a good or bad next step for you.

This is a site that will not maintain interest as it is beyond a short period unless you add more to it. So the key here is to move forward quickly, even if it is in tiny incremental steps. letting the users help you make some of those choices of what steps to take can only help.

EDIT: HN doesn't render that code very nicely, here's a pastie: http://pastie.org/1416631

Hi there, your home page is very simple, render it in plain HTML and post the signup to queue.php.

In queue.php put:

<?php $data = serialize($_POST); mysql_connect('localhost','dbuser','dbpass'); mysql_select_db('dbname'); mysql_query('INSERT INTO cache(cache_data) VALUES(\''.mysql_real_escape_string($data).'\')'); echo 'YOUR REAL PAGE SHOULD BE HERE WITH A MESSAGE ABOUT BEING IN THE QUEUE'; ?>

The cache table should just be a PK cache_id and a text field (or perhaps mediumtext). I've suggested using a database instead of a filesystem based cache because, if you're doing this quickly, doing it using MySQL means you're less likely to run into file permissions or security problems.

Then you're at least capturing everyone's information whilst you figure out how to scale.

Cache the hell out of it! I'd spend all of my time looking into caching mechanisms for Rails right now.