HN Explorer

Ask HN: My web startup is growing fast what next?

8 points by digamber_kamat ↗ HN
Dont ask me whats the url of the startup. Let us assume the question to be hypothetical.

Our website is in beta phase and I am the only coder in the team. Looking at the idea some investors have given us a decent amount of money.

We launched this site through invitations to 100 people. Within a week we hit the 1000 registered users mark. Till then we had hosted our site on rackspace cloud with a server with 256MB (thats not a typo) machine. The tech stack was LAMP.

Sensing the growth we increased the RAM to 1 GB. The number of users at any given time was around 40. The RAM was more than sufficient for such a scenario but still the site started slowing down.

I immediately shifted the mysql server to a different machine. That really did not make much difference.

So I looked at the most frequently used operations (which was some queries with joins). I indexed all the tables accordingly and installed memcache to cache the queries for half an hour or so (wherever possible).

Now the number of online at any given time is increased to 120. Total registered users is 9000. Our investors are happy and co-founders think we are successful and now trying to buy an office for ourselves and build a proper team.

However I am clueless as to think how we can scale hence forth.

Can people give me some pointers as to the different paths I can follow? Please note that we cant host the server ourselves, we would love to continue with rack-space.

19 comments

[ 2.7 ms ] story [ 35.6 ms ] thread
bee
Maybe you should change the db structure, to optimize sql queries, use a caching system to delivery static content, or you can ask a more experienced developer to look over db structure and give you ideas. 9000 users don't seem that many to make the server run slow...
sagacity
> 9000 users don't seem that many to make the server run slow...

I second that, especially when you have moved mysql on a separate server. I'm sure you're missing something basic here (as happens to many from time to time :-)).

digamber_kamat
Our server is not slow at the moment. It is pretty responsive. But as 9000 reaches closer to 100000 I am sure existing safe-gaurds will not be enough, so I am asking what else can be done.
aonic
Also agree with this. I'd be happy to help you by taking a look and sharing my thoughts on where improvements can be made in the DB structure, configuration, scalability, as well as places where caching will give good results vs implementation costs and effort.

Feel free to email me, address in profile.

stevenklein
It's really to comment on things like an office, raising money, growing the team, doing more customer development, etc. without knowing much (if anything) about your situation. I do have just one quick comment though -- you mentioned the speed of your site being a factor pretty early on. There are many factors that go into a user's overall experience but perceived speed of your site is a HUGE one. Don't let that be an issue for you right now as it can severely impede your growth.
rjamestaylor
I would be happy to look at your configuration and suggest options for scaling further (does not necessarily mean more resources or $$!). Please send an email with your account number to twitter@rackspace.com. Myself or one of my teammates will continue our discussion from there!

Robert Taylor Sr Systems Engineer (Linux) Rackspace Hosting robot@rackspace.com

digamber_kamat
Thanks Rob. I will keep this in mind and will ask for your help as soon as possible.
badkins
When optimizing, don't do it blindly. Do some actual tests or profiling to find the true bottleneck. It might be a specific query, the DB as a whole, etc. But unless you fix the biggest bottleneck, other improvements won't make much diference. Once you fix the biggest bottleneck, go back and do it again for the next biggest.

This way, you will know for sure if you need another machine, more RAM, or just another table index.

digamber_kamat
Yes true. I realized this during my initial optimization. I realized that some queries involving joins were taking unusually long time, a slight tweaking tremendous gain in performance.
pilom
From a scale/speed perspective, step one is figure out your next bottleneck. You should do this before it actually becomes a bottleneck. Obvious things come to mind:

1. Memory - very easy to scale up to a machine with more, maybe throw a Nginx frontend on it if you are getting too many connections 2. Bandwidth - move resources to a CDN assuming you already gzip and minify text, combine javascript files, and have appropriately sized photos. 3. Disk I/O - change DB structure again, more/better caching, set up load balancers (there are tons of other optimizations to do before you get to this point) 4. CPU - profile your PHP to see what takes so long. Flush your output as soon as possible so users see something faster.

Lastly be sure you have a redundant setup with auto-failover. 9000 users get very unhappy very quickly when you start having unexpected downtime more than 0 seconds.

digamber_kamat
Let me look closely at your checklist

1. Memory: We have 2 GB and "free -m" shows that more than 200MB (least) is free at any given time. Do I still need to add more memory?

2. Our site scores perfect A on Yslow. So minification , gzip, far future headers etc is in place.

3. We have improved out DB structure a lot. Since we are adding more features continuously we are constantly changing the DB structure. With the help of some more experienced programmers I will get these things tweaked. But in MySQL okay if our user base will hit 1 Lakh in next three months ?

4. CPU profiling is something that we haven't done yet. Will do this asap. Can you suggest some resources starting points ?

You also said something about load balancers. Will you please give me some resources to start up with ( I know I can google but ...) ?

5. Fail-over is something that I did not think of initially. What are the best tools around ?for the same ?

The whole discussion might sound as if I want you guys to spoon feed me. But that is not the case. Let us look at the whole questions as an opportunity to share each others views on the same topic, even the experts might get to learn new things.

sagacity
> But in MySQL okay if our user base will hit 1 Lakh in next three months ?

I doubt if many here will understand what is 1 Lakh. Better make it 100,000. :-)

(BTW, where do you live?)

aonic
4. Instead of CPU profiling at a lower level, I'd suggest enabling profiling in xdebug for a debugging account and hit the pages and look for what pages took the longest to render (server side). Then examine the xdebug profiling logs for those pages to see what php calls took the longest, if youre calling MySQL through PHP functions this will also show you the slowest queries. You should also enable slow query logged in MySQL. I suggest anything taking more than 50ms

5. For a quick, easy, simple setup, assuming you're using Apache for your web server(s), setup another server running nginx and then use proxy_pass to load balance traffic between multiple web servers. Example: https://gist.github.com/2f4118028ed04c40862e

I suggest doing more reading on nginx load balancing and PHP xdebug profiling

sagacity
This is pretty basic, but in addition to mysql table structure and query optimization, have you done/considered any tweaking of the mysql config itself? (Usually at /etc/my.cnf)

We've repeatedly experienced huge performance gains even with basic/minor tweaks of my.cnf.

HTH

digamber_kamat
Cool. Something that I dint know. So far I have changed some table types to innoDB from MyISAM. Rewritten a lot of queries and in some cases maintained redundancy instead of normalization.
sagacity
Likewise, you might want to check/tweak your apache config too (usually at /etc/httpd/conf/httpd.conf).

As you probably know, the default configs of both - mysql and apache are usually pretty minimalistic so tuning these in accordance with your load/traffic scenarios can pay you off big-time in terms of performance boost.

On that note, I'm sure they have some highly competent people out there at RackSpace who should also be able to help you out with all this.