Ask HN: My host has me over a barrel. I've placed myself in a pickle. How to handle?
While we had 5,000+ uniques per day last week, on average we've been getting around 1200-1500 per day.
This doesn't seem like a lot of traffic to me. However for the last 4 days, apache has been failing every 3-4 hours and restarting about 10 minutes later, meaning 10 minutes of downtime for each crash. My host is saying the site is hitting its memory limit all the time. Needless to say, this is a problem.
For the last three days I've been filing tickets with my support guys (I bought a 5-minute response support package for $60/month thinking it would free me up to do more writing/editing) every time there's an outage. I've now filed 34 tickets. Their responses say the same thing every time "We are making apache configuration changes and will monitor your performance." Last night we had about 5 hours without a crash. I thought things were fixed.
They just went down again.
I've spoken with a manager at Midphase. He said that he doesn't see any obvious problems with how their server is configured or how my site is configured. It's just using up too much memory. His suggestion (of course) is to move to a dedicated server with Midphase at $200/month.
Now, I'm more than willing to concede that my site could be better optimized. I'm working on that and have some Drupal pro's who've offered to give it a look sometime in the near future. I'm also aware that I should be planning for future growth with our server setup.
I just don't want to get ripped off.
It seems like I've got three options.
1. Ask my hosting guys the right questions and find out what's causing the problem and get it fixed.
2. Upgrade to a dedicated Midphase server, potentially encounter the same problems.
3. Go elsewhere, potentially encounter the same problems.
Are there some specific questions I should ask my hosting people to help me get a better picture of the situation?
42 comments
[ 28.4 ms ] story [ 309 ms ] thread2) Go elsewhere. Slicehost (or linode) has better VPS deals, and Layered Tech used to have some pretty good deals on 'real' servers.
Ok. So 384 looks low. That's helpful, objective advice. Thanks.
I've always been partial to Slicehost, but my main need uses a lot of bandwidth and disk space and they absolutely can't compare to overseas hosts for bw/$. slicehost's most generous plan is 1600gb transfer at $280 and our 4000gb for 79 euros from leaseweb is vastly cheaper, although you have to mix and match some of the other specs.
Honestly, do not go with midphase dedicated servers. If I remember correctly they're just reselling softlayer(.com), and softlayer is a LOT better in every aspect compared to midphase. I've had some friends use midphase and have so much trouble with their customer support when something goes wrong...softlayer in my experience has been everything great about a company providing these kinds of services.
Oh, and its $38/month for 516 meg slice. Can't say enough kind words about 'em (if you're willing to fire up SSH yourself:).
You could maybe offer to pay for (some of) the help with a sponsored link on your site. That could be reasonably attractive to a Chicago-based contractor with a low pipeline.
ahem
Is anyone a Drupal/apache shark and willing to help me tune my site and migrate to Slicehost?
We were featured on the HuffingtonPost, the Freakonomics blog, Playboy.com and numerous other "name" sites last week. Would be happy to discuss price or a link.
The other option is to find a host that specializes in hosting drupal sites and is willing to take care of all that for you. The biggest problem with this is that you can only use the features they install for you. Of course, they will also charge more than generic hosting, but it's still cheaper than getting a SysAdmin.
i had similar problems with a default config of apache on a 256 slice. if you tweak things correctly, you'll be fine.
For some combination of Drupal's general heavy-weightness, lack of caching, slow server, your front page performance is sporadic and mostly very poor. I get times mostly in the 0.6 - 1.4 sec range just for loading the HTML, and that's with about 0.15 sec as the baseline for just loading a similar-sized static image from the site. I'm not a Drupal expert even a little, but if you could fully cache the front page (so that apache serves a static file directly) and update it, e.g., once every 10 min, you'd be flying, comparatively.
But about what's going on, I'd lay money it's all of the above: you have a low-resource server, drupal isn't fast/light by default and you haven't made it any better, you're not using any caching where it would really help. You should work on fixing all of those.
If you don't have the skill set be a good sysadmin and you "dont know" if its your app, you are in for a world of hurt. I highly suggest you find help: a good sysadmin and maybe an outsider to look at your code and web page structure. I have not heard enough to say your hosting company is screwing you. And its maybe wise that you are paying $60/month for better tech support but you still only get $60/month worth of tech support and that's not much ;(.
I'm also going to look more into improving my site's performance. I have enabled Drupal's page caching, but I guess it's not doing much to help. I'll pursue that further.
Do you have Optimize CSS files (CSS file aggregation) enabled?
Do you have Caching mode (Normal) enabled?
Cheers
I'm also using the blockcache module for most of my blocks and have js aggregation switched on (though I have to omit tinymce and a few other js files in order for them to work).
http://drupal.org/getting-started/6/admin/settings The manual says caching is under the Performance section and it's disabled by default.
From what you've stated, I'll take a stab at what's going on. Apache processes are big, from a memory usage standpoint. A single Apache + mod_php could use 25 MB or more. If you have a dozen or more requests that hit at once -- or processes that are feeding slow clients -- blam, you've run out of memory.
One solution is to switch to lighttpd, nginx, or another webserver with a smaller footprint. Here a single process (or small set) serves a pool of PHP processes. But that's a bunch of work for a guess at what's going wrong.
Instead, try running nginx as a proxy in front of Apache, and let it spoonfeed slow clients. It's likely you will not need to change Drupal at all, and make only minor changes to Apache to do this. A good description is here: http://blog.kovyrin.net/2006/05/18/nginx-as-reverse-proxy/
Turn Keep-Alive Off. (If you're site doesn't have tightly packed multiple requests). The default is 15 seconds, I saw a 50% speed increase when I disabled this. Set MaxRequestsPerChild to 500, in case your code is leaking.
Second, 384 MB just isn't much memory for a medium-traffic Drupal site. I would recommend at least a gig of memory, and perhaps more if you use plugins heavily. Drupal is neither particularly lean nor blazingly fast, so you're going to need a lot of Apache listeners working to handle that load, with each process probably consuming 30-50 MB of RAM.
Third, you're right to think that you need someone on-call who knows Apache, Drupal, and database tuning and configuration inside and out. A $60/mo. support package isn't going to do it, though. If you think about it, that's really only paying for an hour or two of tech time each month, so expecting to get 24/7 interactive performance tuning and optimization for free along with your support contract is pretty unreasonable. Put a performance guru on retainer, and establish an SLA with them if necessary to maintain your site uptime.
Finally, opening >30 tickets in less than a week is not the best way to get quality support. It's like the stereotypical impatient kid on a road trip, asking "are we there yet?" over and over again.
My suggestion would be for your to find an experienced LAMP admin who is willing to manage your hosting for you, and pay them a reasonable fee for doing so.
But I can sort of commiserate with the OP. Midphase support is not exactly the best...I've spent hours trying to get them to understand the most trivial things, like when a friend's website wouldn't work because the mysql server was awol and the support person didn't understand what was wrong with nothing showing on the website. Yeaaaahhhhh.
I realize the website looks shady as all hell. But we use them for EVERYTHING and we've never had a problem. Their support is also top notch. Usually answers+fixes <30 mins.
http://www.serverbeach.com/catalog/cust_ref_landing_new.php?...
1. buy a much more powerful server/VPS.
for seventy some odd dollars a month you can just leave your amazon ec2 image up all the time, and that's something like 1.7Gb ram. if you are willing to use a small provider (like me) you can do even better- I'll give you 2GB ram for $64/month. Having more ram makes a lot of problems just go away. (using a Xen based host also means that you can use swap, which means that your out of memory events would be slowdowns rather than crashes.)
2. get a skilled SysAdmin (possibly you) to optimize your system from a systems standpoint.
ultimately, you want both. But often just buying a better server is the cheapest option.
But yeah. no matter how much you like your host, I recommend that you always have a hot copy of your website/application at another server, ready to switch the dns over at the first sign of trouble. you should test this periodically.