HN Explorer

Hire YC: Need high-traffic Django help ASAP

6 points by mattdennewitz ↗ HN
hey,

we just re-launched out site w/ Django, and are cant handle the peak traffic load. we have apache in the back (mod_python), nginx in the front, memcached. the database doesn't seem to be the bottleneck.

restarting apache yields fast response times up front, but gradually drifts to about 6-12 seconds per request.

we can't handle getting crushed with unacceptable load times much more without serious consequences, so if anyone is available to help, email me at `mattdennewitz at gmail` and lets talk.

8 comments

[ 2.2 ms ] story [ 23.7 ms ] thread
erlanger
mod_wsgi. If nginx is serving as a proxy or a file server too? If the former, I'd go with HAProxy. If the latter, Lighttpd works very well with Python and FastCGI.
mattdennewitz
just switched to modwsgi

response time is up to 30 sec with 1 thread and 50 processes. nginx is up front as a proxy and a file server.

erlanger
Have you benchmarked the same app running on Lighty and FCGI? You shouldn't have to change much, if anything.
mattdennewitz
no. is that a common solution for hi-traffic service?

what would lighty/fcgi replace?

a_toker
Lighttpd would serve both static files and the application. Reddit uses web.py with Lighty+FCGI, and I see no reason that you couldn't run your Django app on it. I use Apache + mod_wsgi for the Apache modules, but I'm taking a good look at lighttpd+FCGI to get more out of my server for my Django and Trac installations.
suhail
buy a server load balance with nginx and throw a duplicate copy of everything on that and use the same db since it's not your bottleneck.

Start looking for code you can optimize such as bad loops.

chiffonade
... why can't you just add more application servers?
maney
The first thing you want to do is measure the problem in enough detail to identify the cause. I can recommend Rob's Debug Toolbar (though with reservations about the possible overhead, though I'd have been happy to chance that when I needed to instrument an underperforming site):

http://github.com/robhudson/django-debug-toolbar/

though his tree doesn't have the injectable measurement points I've added to the timer panel yet. I hesitate to suggest this for a production setting only because it's not had much third party exposure (or if it has no one's let me know), but "it works for me":

http://github.com/mmaney/django-debug-toolbar/

You might want to avoid the most recent commit (so use da1429...) - Rob had asked about adding support for the cache API's "add" method, and I threw that together based on the spec; that one part is completely untested other than that it doesn't cause a compilation error in my test setting.