Joyent: 54+ hours of downtime
I am a customer of Joyent.
For the past two days, the server on which I host some customer websites and mail servers has been down. I did not receive any warning or explanation by email or telephone. When I filed a support ticket, I was given a link to their help site with some status updates.
Status updates (paraphrased):
2011-02-26 09:22:19 GMT: server offline
2011-02-26 14:38:38 GMT: no ETA
2011-02-26 17:09:42 GMT: estimated downtime 12+ hours
2011-02-27 22:23:08 GMT: estimated downtime 20+ hours
That final estimate of 20+ hours is in addition to what is now 37 hours of downtime. Joyent has offered no reparation or backup service (e.g. backup MX service to keep mail from bouncing).
For your consideration when choosing hosting providers.
129 comments
[ 4.1 ms ] story [ 74.6 ms ] threadIt looks like these message are from a problem with their Howe server. That is one of their old servers, which came over from the Textdrive merger years ago. I used to have some sites hosted on that same server, and was likewise not very happy with things. It appears to me that Joyent as a company doesn't care very much about that legacy equipment and line of business.
The way to make the pain stop, is to stop using Joyent's old TextDrive servers. You can do that by leaving Joyent entirely, or by moving to their newer stuff. Either way it's a fair amount of work, depending on the complexity of what you are hosting.
I took the path of moving to their newer stuff, and have been very happy with Joyent ever since.
The overal quality of their (Hurricane Electric, not Linode) operation is poor, historically they have been lax on shady people colo'ing rack space in Fremont (google hurricane electric mccolo), HE-1 is full and so little room for expansion and its on a fault line.
I've NEVER had any of these problems in Newark (which is Linode's geographically local DC as they are in NJ) or London.
However I chose Fremont because it's the closest datacentre they have to Australia. Most of my readers are here.
(see http://www.linode.com/speedtest/ for pingable hostnames of servers in each DC + speedtests)
If you do want to move, you can put in a ticket and they can move you between DC's with no downtime I believe or just replicate your instance and switch IP's.
However, I've learnt, and hopefully have others, that having everything in one basket isn't the best option, even with how excellent Linode are, they are at the mercy of HE etc.
So now we have a split between Fremont, Dallas & Newark.
It's worked pretty well, we are most happy.
Anyone running a large app or site would do well to run multiple systems with Linode across multiple sites where possible.
Also I think I have learned from this thread that 'lifetime' service packages suck because after the company closes that deal they have absolutely no incentive whatsoever to do anything more for you as you've basically just agreed to become a burden for them forever.
They should pull up the old contract, find out exactly what can be done to either terminate or transition people to the new services and get it over with. This seems like they're just letting the issue run off the rails and then when people complain that a server is down an extended period of time, a server that they paid a lifetime membership fee for, they start spouting off that these people are whiny freeloaders.
If the servers are old and rickety, migrate them to the lowest level of your new service for the same cost as what the person paid for initially, given that they've already paid in full, this would put the customer cost at $0. If you don't want to do that, then maintain the old servers and give them as much love as you would your other customers. That love includes not getting pissy in public about people whining over "free" servers, especially when they _weren't free_.
Are you even still offering regular (not “cloud”) web hosting? I can't find it on Joyent's site anymore; it seems as if the Shared Accelerator plans are discontinued, too.
[1]: http://wiki.joyent.com/shared:kb:start
Edit: Found it. This is the page that implies that TextDrive users have needed to wait for a “golden ticket” to be offered to them: http://replay.waybackmachine.org/20090305040858/http://wiki....
http://oldwiki.joyent.com/
And that wiki has up-to-date information on migrating from TextDrive to what are apparently now called “Shared SmartMachines” (formerly Shared Accelerators). And that page finally does resolve the question of whether we need to contact them or wait to be contacted: We do need to ask for a golden ticket, not just wait for it.
That's pretty well buried. Joyent really should send out a mass email to all TextDrive users asking us to upgrade, telling us exactly what effect it would have on our billing, and telling us exactly what we need to do.
It's still unclear whether, in the special case of Howe, we still need to ask for golden tickets, or all Howe users are being migrated to Shared SmartMachines without having to ask. The support post:
http://help.joyent.com/index.php?pg=forums.posts&id=949
implies that they are rebuilding Howe, which implies that we will not simply be migrated to a Shared SmartMachine as a result of the failure (we still would need to ask), but they have not directly said.
The one remaining worry I have is that there's nothing on the main Joyent site about Shared SmartMachines. It seems like they're inviting users of one deprecated service to migrate to another deprecated service.
No, the way to make the pain stop (and when I say pain, I mean hours and hours of downtime) is to work with a real managed hosting vendor that cares about your servers and your business, and to move your mission critical software off of commodity shared servers where it will get lost in a sea of customers, none of whom have an account executive, a sales representative, or assigned engineers who understands your stack from top to bottom.
Cloud services are great for some things like elastic computing power and rapid prototyping, but having my business' front door hosted on a service where I can't even call someone up in case of trouble seems like absolute craziness. I mean, how would you even implement a disaster recovery/business continuity solution when all the damn servers are in one building?
I would try to touch base with them again and see if they can help you out.
For clause 2:
So probably you can get this month for very little (Takes only ten hours for getting full refund if SLA is signed)But at the same time their exceptions list are pretty long, things like emergency maintenance and upgrades are considered as exceptions.
They consistently took days to answer simple questions via email. I generally take it that if a company can't even get its sales people responding quickly there's a snowballs chance in hell it's techs will.
You Joyent folks will never get a subset of your customers to migrate off textdrive hardware (so it's no use to keep yelling "just migrate!"). However, if you let those older servers fall apart you'll only get more negative PR.
Without a doubt, Jason has the best intentions here, but I'm getting a really bad vibe from reading the entire thread -- particularly the by the way, you contributed to a ticket and I can look it up so I'm going to remind you so that you'll look stupid for bitching about us bit.
The hostile you-are-wrong attitude, while maybe technically correct, is a disaster for outside perception.
Textdrive promised lifetime hosting. Where does it say they promised hosting on the same box for that entire lifetime? TxD/Joyent has been gracious to provide hosting on their new platform.
One could argue that the 'lifetime' was referring to the actual box, and if the downtime is increasing, the lifetime of the server is ending.
I say pull the plug and force the migration. Granted, it could be a small PR hit.
They aren't horrible, but I would never use or recommend them to anyone.
I am reading all the comments, and I'm not spamming the thread. I'm having a conversation on a site that I frequent everyday. This is called being responsive.
But because you (Jason) appear to be actively monitoring this thread, I'm going to make a suggestion:
E-mail everyone on the old BSD textdrive servers and tell them that they really should migrate.
Tell the month-to-month customers that you're end-of-lifing that product and on February 29th of 2012 (because nothing ever happens on Feb 29th) it will no longer be there. If you like, offer them whatever account is similar to the one they're on with a 50% discount 1 year discount if they take you up on it in the next month, 25% the three months following that and 10% up to the 29th.
For the VC customers just tell it how it is. You will continue to support the systems for however long you need to, but the newer stuff is just better. You appreciate their trust in your company etc. but they will probably be happier with the shinier stuff if they migrate.
Then follow up every couple of months with another e-mail. You could even make it interesting and say what percentage of VCs have migrated, what cool new stuff is available to those who have moved etc. (though if you do that, I'd like to receive those e-mails too).
If you want to incentivize the move for the stragglers, go ahead, their bitching pisses me off anyway and if you can get them to move and STFU by offering them something I didn't get, that's fine by me.
But most importantly send an e-mail. Don't post on the forum, or on the blog, or the wiki. I know that the forum, RSS feeds and blog are how Joyent likes to communicate but e-mail goes into my inbox. If I don't pay attention to it I'm the only one to blame.
If two or three years down the line people still haven't migrated. EOL the BSD servers. I'm pretty confident the "lifetime" I paid for was for lifetime of hosting with particular storage, server access, and bandwidth levels, not lifetime of hosting on a single machine with a particular OS. I could be wrong, but I think it's reasonable to assume that your hosting provider will change technologies as time progresses. I also am happy if they keep on giving me higher quality service with no reoccurring fees.
You are offering an unacceptable level of service to customers who have supported and recommended you from your early days. You were paid in advance for a lifetime of service. I took you on your word and offered lifetime hosting to friends with small websites for small businesses. If you can't make good on your agreements, I will eat the cost of moving these sites to Rackspace Cloud. I can't risk another 54 hours of downtime.
With proper backups and a fresh server it should not take 54 hours to get back online.
The packing system is the same, it's BSD userland. The paths are the same. Everything is the same. The kernel shouldn't matter. And if you had told support that you wanted us to just migrate you because nothing was out of the ordinary, what would they have done? They would have migrated you ....
And this has been quite a good deal for you. The logic of moving from something that costs you nothing to something that does costs you something is interesting.
Downtime costs him. So much so that he's willing to pay for uptime.
Also, it didn't cost him "nothing". He paid for service. It wasn't free.
Your reply sounds like you don't value him as a customer anymore because he's not giving you money anymore. Nobody forced textdrive into offering the "lifetime" package. You should treat him like a valued old customer, rather than calling his logic "interesting".
I think you mean: replies.
I completely understand that he's 'getting a great deal', and that he isn't currently paying you anything, and that he may well have lasted as a customer longer than you expected, but your company offered lifetime service for a fixed cost, and he bought it. As I understand it, this was to bootstrap your later efforts.
To constantly demean his position is as insulting to him as it is to your integrity. While you may feel like you're in the right here, I can guarantee that you've lost at least one potential customer, because I will not be using Joyent. I'm guessing that running potential customers away isn't your intent here so please, don't do yourself any further harm.
Wait, what? Who owns and operates the servers?
Paying for a service forever in advance is a devious mis-alignment of incentives. The customer will feel that they are owed everything. The business of it won't work out.
As I said, the lesson is, don't give shit for free - even for an initial payment. Ongoing services need to be paid for in an ongoing manner.
I currently pay ~$215/year for a linode account.
I will continue to pay for linode.
Maybe you can shut down jervis.textdrive.com if I migrate my account :)
Although relatively harmless, it seems like planting the seed to a disregard for privacy.
One more reason to shy away from Joyent.
Edit - Jed from Linode?! Hey! You sir are my favourite support person to reach, yet I rarely get you anymore.
A 6 year old server would have a max RAM capacity of say 32GB, you can get a newish 64GB RAM system for what, $3k or so?
In such a case, migration is basically a few shell scripts plus rsync from old hardware to new VM. You rsync a couple of times until the rsync takes very little time, then turn off services; do a final rsync; turn on services on the new VM, swap IP addresses and test.
You can even run FreeBSD/Xen on top of Solaris XVM, right?
I would be interested in getting my account into a virtual env but I can't really justify paying $45/month at this moment...what are my options?
tl;dr: how can I go about moving my stuff? To where? And why am I in this handbasket?
I have been with them for a couple of years and the only reason why I left was because the project I used to host there was put on hiatus and I don't need that kind of 'enterprise' hosting for my other stuff.
I agree there shouldn't be downtimes on any of their machines but just going by the headline, you'd think Joyent is unreliable when indeed their own server plans are rock-solid.
You make it sound like they are Dreamhost. That said, hope your problems will get solved soon.
Edit: I can no longer edit the original post, but I will post a comment.
If you have a small set of users who bought lifetime accounts to bet on a company you founded than even with new company I would bend over backwards to make sure their needs are met as a thank you.
Otherwise some gets the bright idea to put up a joynet sucks web page with proper Google approved SEO and makes this little PR bump a big EFFING MOUNTAIN!'
jasonhoffman, I imagine you have less than 12 hours before that happens..as TechCrunch may do it for 'free'..:)
Cutco will re-sharpen your knives for free, no matter how old they are, even if you bought them on eBay, because they made a 'lifetime resharpening' promise with the original purchase. And they're basically a multi-level marketing scam.
The lesson for service hosts, NEVER offer lifetime anything.
Sure, you paid $200. But if you've been a customer since 2007 then you've basically paid the same price as any cheap shared hosting provider. Contrast this experience with Layered Technologies that ran three price hikes and then when I still didn't leave the server I had with them essentially told me to fuck off once their harddrive failed.
At least jasonhoffman had the courtesy to tell people their best option is to switch servers to avoid trouble.
But, like they say, you get what you pay for. If lifetime support and stability is what you'll come to expect, don't pay bottom dollar prices up front.
Disclaimer: I am not a Joyent customer in any way.
I'm a lifetime Joyent (TextDrive) member. When Joyent started offering newer shared hosting accounts, they allowed me and all other lifetime members the opportunity to migrate. No new fees, no hassle. Just a nice, shiny new space. To me, it's a testament of their support that they didn't try and keep all the lifetime members stuck on the old FreeBSD system.
I also find it unrealistic to expect lifetime customers to never have to migrate from one server to another. Do you really expect the server to run for 20 years without ever having to do anything yourself? Seriously, take some responsibility for your site and move to the latest supported service.
It sounds like the offers to migrate have either not been sent to all or the messaging has been inconsistent as to costs/ease with which this can be performed.
I see you posting quite a bit here...you keep saying the same thing: "take the free upgrade that is being offered".
There also seems to be quite a few old textdrive lifetime members here.
Perhaps the best approach would be for you to outline the steps for those textdrive folks to migrate off of those servers and take the free upgrade you are offering. I personally have no idea how...so if you could explain it, give a link or some concrete information it would be helpful.
I for one would be happy to migrate if it means better servers, I just need to know how...I'm sure I'm not alone.
"If you have not sent in a request and would like to get a jump on the migration, please send an email to support [at] joyent [dot] com with the following information:
- Subject line: migration request + plan type if known (startup, plus, premier) - Full Name - User Name(s) / Primary Domain(s) (please note new info if you want to change it at this time) - Existing Server(s)
If you wish your username or domain name to be changed, please include this in the email and we will process when we send your golden ticket. Waiting until after your account is set up will incur a $50 fee and you will receive a brand new account (old one will be blasted).
Please do not send follow up requests as it only slows the process down. If you asked questions when you sent in your migration request, we will answer those when we send your golden ticket out.
Once you receive your 'golden ticket', you will have 60 days to migrate to the OpenSolaris environment. If you need the second account for more than 60 days, regular monthly charges will apply."
I find it a tad bit odd that I received NO emails on migration. Frankly, there is no conceivable way that announcing in a Joyent forum and expecting users to become aware of it is a sound strategy. Pull vs Push and all that.
Even so, I'm willing to migrate if I can find some additional information as to what is involved.
Again, thanks for posting...I'll go digging through Joyent's forums now....
Frankly, the tone of most of his comments have been borderline denigrating and it's doing nothing but adding nuisance value to the thread.
This isn't a glib statement. A business transaction is about more than coming to agreement about the price of a thing or service. It's also about creating and maintaining incentives for both sides.
I pay slicehost for a VPS. They get $50/month from me in return for disk space, CPU cycles, uptime and support. Our incentives are aligned.
With Joyent/Textdrive, every month you get hosting and every month they get nothing. The incentives are not aligned. In fact, from Joyent's perspective, the incentive to provide any meaningful service diminishes over time. No about of cajoling or complaining or threatening Joyent will change this imbalance. Sure, it's 54 hours of downtime today - but I can guarantee that a year from now the outages will be far longer and far more frequent.
"But we had a deal!" Yes, you did. And you were a fool to enter into it.
Here's the bottom line:
Your customers don't care what's causing the downtime. It's still your fault.
Website users don't care that the deal has soured. It's still your responsibility.
Joyent doesn't care that they're providing cr@ppy service to Textdrive users, despite the postings on HN by Jason Hoffman (hey Jason, instead of wasting time posting on some internet forum, why not spend that time fixing the actual problem?)
Seriously - Joyent doesn't give a sh?t about the lifetime accounts. If they did, then why are you experiencing 54 hours of downtime? If they do care, and are applying resources to solve the problem, then why is it taking so long? 54 HOURS TO FIX A SERVER. That, to me, is evidence of a shocking level of incompetence. In either case I would get the hell out of there.
Look, we were investors in the truest sense. TextDrive was this revolutionary new shared hosting platform with a lot of promise (flexibility, no overselling, etc). Problem was that shared hosting is a commodity with no profit margins. Since TextDrive was started by people who want to build cool shit, rather than people who knew anything about operations in a customer service industry, the result is a massive series of pivots that left us out in the cold.
I'm not making excuses for anyone here, personally I think Jason Hoffman spent way too much time in the forum sucking up the love, and not enough time figuring out how to run a support operation. So he burnt his bridge with me as a customer. I won't touch Joyent with a ten foot pole no matter how good it is.
But here's the thing: that was their bridge to burn.
It doesn't matter what 200 early customers think, because those 200 one-time payments were not going to build a successful company. The bottom line is that the market for a "premium" shared hosting service got squeezed by commodity hosting on one side, and VPSes on the other. Even if they had managed reasonable uptime (which they didn't because they totally underestimated the technical cost of giving people the freedom they did), it still would have been a doomed market, because VPSes got so cheap anyway. We can complain about what a bad deal we got, but it doesn't change the fact that the last salary we paid was in 2005, and the hosting world has moved on. They did what they had to.
http://techcrunch.com/2008/01/31/twitter-and-joyent-split-am...
Twitter moved to Verio and here's Biz Stone's reply to Techcrunch:
We’re still very much engaged in our efforts to bring solid reliability to Twitter. Achieving our goals is a sustained effort, not an overnight fix. Performance is our most important measure of success and we appreciate both the patience and frustration folks are sharing with us.
With regard to discussing technical specifics about last night’s efforts, we’ll be more keen to do that once we have a chance to come up for air and cover it with some perspective.