HN, please help... My host just screwed me over
Without telling me, my host decided to perform a file system check (http://en.wikipedia.org/wiki/Fsck) and so now my server is offline for a few hours.
An hour or two ago I began telling dozens of high priority potential clients to check it out... each of which could bring in thousands of dollars per month. About 30 minutes ago I noticed the site was offline... and I'm pretty sure quite a few have tried to look at it since then. None have responded. And I'm afraid they've all gotten a bad first impression, and may never bother to check it out again.
I've sent them all messages informing them of the server maintenance and that it should be back up soon. I'm planning on sending them all another message when it's back up.
This is absolutely infuriating. It feels like 8 months of work just went down the drain because my dumbass host didn't telling me they were going to perform some maintenance.
What can I do to make sure I don't lose these clients?
14 comments
[ 3.3 ms ] story [ 38.1 ms ] threadYou're a big boy (or girl); take responsibility for the issue, because even though the most proximate cause of the outage wasn't within your control, the choice of host and choice of architecture are under your control.
If you tell the clients that there was a rare downtime event, that you are working with your technical team to understand and prevent a recurrence, they are less likely to think that's what they should expect if they host with you.
As a potential client, I'd rather hear that you take responsibility for ANY issue that happens to my site; passing blame to a third party ISP to make yourself look better today actually makes you look a little weasely IMO.
Then as soon as humanly possible, transfer your original domain to the new host and never use the host that screwed you ever again.
There's no need to be melodramatic about losing thousands of dollars per month; you haven't lost your work so nothing is "down the drain." No ISP runs fsck for no reason, in the middle of the day, without warning, nor as a matter of routine maintenance.
Then, the post game analysis. Server apocalypses happen. There is a deeper lesson here.
On the technical side, there are some things you can do to prevent this kind of failure. Running fsck for hours on a live server is really inexcusable both from a technical as well as a service point of view. Most small hosting providers simply rent some rack space, put in their standard Dell server unit and call it a day. This way lies madness, total data loss and possibly the early seeds of an anxiety disorder. I'd suggest you look into inherently redundant hosting offers (like MediaTemple's grid service) or some high-availability virtual hosting (like Amazon EC2).
But the core of the issue is unrelated to this technical stuff. Telling everyone at once was a mistake. There is a reason why most projects start out gradually, starting with a beta phase when everyone still has a lot of compassion for bugs and crashes. It's tempting to send invites to all possible contacts once you're finished, but it really has no upside because you're introducing a single point of failure. I'm not even talking about having only one server and no backup. In this case, the blitzkrieg launch date itself was that single point of failure. Even if the server hadn't crashed and burned, your app is very likely to still have bugs and kinks to work out. Bringing potential clients in one by one would have given you the opportunity to collect valuable feedback so you can show a more refined version to people who come in later.
When it comes to handling this issue from the CR side: don't send out apologies for the outage. Instead, check back with high-potential contacts in a few days and ask "did you have a chance to look at my new project?" and when they reply they couldn't access the server, invite them to try again and concede there were some outages but the problem is fixed now. My guess is, you'll be surprised how few people were actually affected and noticed that the server was gone.
In my opinion, it's unlikely that your bad luck actually ruined anything. Most people don't look at random invites in their mail right away. If they're really interested in the product and/or know you personally, they'll check back again later even if your site is offline at first. Interested people will still look at your stuff. Chances are, those candidates for your first mass invite will not even turn out to be your most valuable clients. Nothing is lost.
2. I'm still curious regardless of what the point is, who's the host?
The app looks great. It's got great utility. Hang in there, just a minor glitch today.