It's pretty unbelievable that a hosting company like that doesn't make backups for their customers by default. Even if the guy didn't follow the "instructions" to the letter you can still assume that the hosting company are the experts here. And why are these guys physically deleting files on cancellation of an account? Pretty amateur if you'd ask me.
In their press release[1] they say they do make backups for their customers by default except in cases where the size of the backup exceeds a certain threshold which happened in this case. Supposedly the customer was notified of this in writing and acknowledged.
Discontinuing backups and deleting data on account cancellation seem like reasonable acts to me. It is a sorry case but it sounds like misunderstanding by the customer, though the process seems a touch messy and could be improved.
Hopefully that will be VentraIP's take away from this, though they probably won't acknowledge this if legal action is pending.
As I see they are not taking care of their customers. If just have a retention policy of the data after the cancelation, let's say for 3 days, or a week, the problem could have being avoided. Win-win situation.
Here's why a hosting company doesn't make backups by default: it costs money. You can either be a premium hosting party where the service is all inclusive, or you can offer your services a la carte, where customers can choose what they want and pay for. Both are valid business models, either way, TANSTAFL.
It's up to the customer to choose. Why would a business give shit away for free after the client made their choice not to buy it?
And as far as I can tell they didn't "physically delete files", they just pulled the plug on a VPS as per the clients requests. You think being able to terminate an EC2 instance is amateurish from Amazon?
Or have something like the EFF but focused on young web businesses which could be funded by web business elders [those who already made money from the common web ecosystem] to protect the rights of web businesses especially those who are just starting-up and to advocate/design/lobby for legal systems that are conducive to the web ecosystem as a whole. For legal practitioners, this would arguably be a better place to work for(if one can earn a living from it) rather than let's say working for patent trolls.
It says right at the top that the owner "has limited tech knowledge". It's clear that he suffered the cost of trusting his hosting company to do this for him, but the real problem here seems to be trying to run a large website as a business without having any idea how it works.
Wow this is just silly. This event could've easily been avoided and all this VentraIP company has gained is a shitstorm. Surely this host could have a process where the migration from one account to another requires the data be transferred before any account cancellation and deletion. Stop playing the blame game and think of how you should treat your customers.
The posts from the VentraIP representatives on the Whirlpool forums are also childish. Companies like this make me sad to be in the Australia IT sector.
It definitely seems like VentraIP could improve their communications as it seems the migration process is separate from the old account cancellation processes (from what I have read across the forums and their press releases). So a misunderstanding on the clients behalf caused them to delete the site before he asked for the migration to go ahead.
I am a customer of ventraip but only for my budget sites. They have built a very good dashboard which allows you to do most of what you would need.
If you try to do something is that is not built into the dashboard it becomes extremly difficult and their customer service is often of no help. Having had to transfer one of my domains from there economy plan to a business plan. I had to cancel the economy plan first to disconnect the domain and then add it to the new server.
If i had no been technically proficient and been aware that i needed to take a cpanel backup before deleting the account so as to be able to restore it, i would have been in the same position as this gentleman.
Ventraip really causes shockwaves in the Australia hosting industry because the competition is overprices and inefficient. There system worked well while only technically capable people used them but once you have beginners using your system you better make it fool proof. I hope they learn a lot from this debacle.
Let's be clear - nobody cares about backups. Any seasoned SysAdmin (That is, one who has been axed, or been on a team where someone was axed for not having a proper recovery process in place) will tell you that restores (and the testing of said restores) is what is essential.
Weekly/monthly testing of restores is critical - preferably in an automated fashion.
People would be amazed to know how many large enterprises have invested 10s of millions of dollars in large tape array/robots, which they religiously back up petabytes of data to, without actually knowing whether they'll be able to restore properly should the day come that they need to.
Frequent automated restoration and testing of database backups would have avoided the downtime. Amazon would have noticed the Oracle bug soon after the extents change, and would have had a chance to identify and work around it before taking down the production database for routine maintenance.
Furthermore, by using their backup DEC storage array for the routine backup restoration tests, they would have identified their new DEC storage array as faulty before they put it into production use.
Okay, total sysadmin noob here: how do you usually test restoration? If I was to test HDD image restoration on my laptop, for example, and it didn't work, I would be doubly up a creek, because not only is my disk hosed, but I've just proven I can't restore it. On our home server it's not quite as bad, but there's only so much you can do with dry runs. I suppose if it's just the files you're interested in, it's not as hard, but much more of a pain if you mess up your OS.
A hard drive with multiple partitions or another boot drive would be a good start.
A good tool for the Mac by the way (if you use OSX) is "Super Duper" which will clone your hard drive. Then you can simply pick the cloned drive as your boot drive and if it boots you will know you have a good backup.
When I was recently traveling I took several pocket hard drives with clones of my laptop drive so if the drive failed (or the laptop failed) I could hook the hard drive up to another laptop (one rented or purchased) and boot from that.
Not using OSX, but I do have an external drive that I might be able to make bootable backups on if I can find the right tools. Sounds like a useful trick regardless.
Indeed, and this goes for home backups too. You never know something is there unless you test recovering it.
For example: Apple's Time Machine seems like a slick, safe way to backup your entire hard drive. And, you can use it to transfer the entire old drive to a new drive. But, if there is an I/O error reading any of the files on backup, it will simply skip that file, and the only indication of a problem is in the /var/log/system.log. It can skip the file for months... eventually reaching the point where any older backup where the file was successfully read is aged out. And even on recovery: the file was skipped, so there's no indication it's even missing. Only when an app needs it is there any indication that months of backups, and a supposed full-recovery, had silent failures.
I had a problem with time machine that caused me to stop using it.
When I switched (think it was under 10.5) from regular to filevault it stopped doing backups.
I happened to pay attention to the time that it took and noticed it seemed extremely quick.
I decided (as I mentioned in another reply here) to use Super Duper which makes a clone of your hard drive and is easy to verify (just boot from the cloned drive).
There are a couple 3rd-party apps to address: one's called 'Time Machine Buddy' and another is 'Log Viewer for Time Machine'. Neither is super-slick (or seems to do anything more than grepping /var/log/system.log for backupd info), but they cover the basic need... once you know it's an issue, which may only happen after being bitten.
"that restores (and the testing of said restores) is what is essential"
Not sure if it's true or urban legend but I remember reading a story about 15 years ago about tape backups being left on the seat of a car with heated seats. When they finally needed them they had been erased by the electric current in the seats. (That's the rough story that I remember at least).
It's probably not true. The magnetic field generated by heated seats is very very low. The coercivity of magnetic tape is much higher than that, plus the car field is constant, which doesn't work very well to erase thing.
The heat maybe, that could do it, magnetic media doesn't like heat. A car warmer would not be enough, but a hot car in the summer might be.
Having bought an tape eraser at radio shack some time ago I would tend to agree with you. To me though the story has value and sticks out in my mind simply to reinforce unknowns that should cause you to always verify backups.
Has much more impact than simply reading "important to test backups".
That said the tapes I believe were large reels and the seats were an old Saab or something like that. So who knows the force field that thing gave off.
also an example of why you shouldn't be a moron. the host instructed him to cancel his service before he was migrated over to a new server, and he did? what the hell was he thinking?
not saying the host is innocent here, but the 'victim' is really only a victim of his own stupidity.
Based on my experience with end users I wouldn't call his behavior moronic. I would say it's about average.
We regularly have people who go into their accounts and change dns without realizing what that means. They only know that either the web "guy" or the new host told them to do that. Or they read they should do that in the setup instructions.
They may not have purchased email service at the new host and they think that everything will just work.
Many times if they speak to 1st level tech support at the new host they don't know enough to even know what will or will not happen with a dns server change.
It's not surprising that this has now made it international.
If you read into the history, the guy clearly did not follow basic instructions. Backups are not part of the service he bought; yet the host backed up anyway. The host informed him that backups were disabled on this account because it was over quota and he did nothing to resolve the situation. He took no backups of his own.
He only lost his data because he failed to follow instructions or respond to emails and subsequently cancelled his shared hosting account before the data had been migrated over to a new VPS service.
I think the hosting company were in the right here.
Man, Godaddy pulled a similar trick on me. It was totally our fault that we screwed up our server, and things were in a limbo. Their advise was to provisionne a new server. I asked them about the data, they told me it was gone, and that's why I need to backup. I had to fight back and forth and ensure we didn't provisionne a new server. They were CONVINCED the data was gone. After a small fix from our end, the data was back and everything ok. Had I followed their advise, one year of daily work would have been poof gone. Incompetent.
31 comments
[ 4.8 ms ] story [ 80.5 ms ] threadDiscontinuing backups and deleting data on account cancellation seem like reasonable acts to me. It is a sorry case but it sounds like misunderstanding by the customer, though the process seems a touch messy and could be improved.
Hopefully that will be VentraIP's take away from this, though they probably won't acknowledge this if legal action is pending.
[1] http://www.ventraip.com.au/files/pr-tvc-130212.pdf
It's up to the customer to choose. Why would a business give shit away for free after the client made their choice not to buy it?
And as far as I can tell they didn't "physically delete files", they just pulled the plug on a VPS as per the clients requests. You think being able to terminate an EC2 instance is amateurish from Amazon?
The posts from the VentraIP representatives on the Whirlpool forums are also childish. Companies like this make me sad to be in the Australia IT sector.
If you try to do something is that is not built into the dashboard it becomes extremly difficult and their customer service is often of no help. Having had to transfer one of my domains from there economy plan to a business plan. I had to cancel the economy plan first to disconnect the domain and then add it to the new server.
If i had no been technically proficient and been aware that i needed to take a cpanel backup before deleting the account so as to be able to restore it, i would have been in the same position as this gentleman.
Ventraip really causes shockwaves in the Australia hosting industry because the competition is overprices and inefficient. There system worked well while only technically capable people used them but once you have beginners using your system you better make it fool proof. I hope they learn a lot from this debacle.
Weekly/monthly testing of restores is critical - preferably in an automated fashion.
People would be amazed to know how many large enterprises have invested 10s of millions of dollars in large tape array/robots, which they religiously back up petabytes of data to, without actually knowing whether they'll be able to restore properly should the day come that they need to.
Frequent automated restoration and testing of database backups would have avoided the downtime. Amazon would have noticed the Oracle bug soon after the extents change, and would have had a chance to identify and work around it before taking down the production database for routine maintenance.
Furthermore, by using their backup DEC storage array for the routine backup restoration tests, they would have identified their new DEC storage array as faulty before they put it into production use.
A good tool for the Mac by the way (if you use OSX) is "Super Duper" which will clone your hard drive. Then you can simply pick the cloned drive as your boot drive and if it boots you will know you have a good backup.
That is how I test that my backup works.
http://www.shirt-pocket.com/SuperDuper/SuperDuperDescription...
When I was recently traveling I took several pocket hard drives with clones of my laptop drive so if the drive failed (or the laptop failed) I could hook the hard drive up to another laptop (one rented or purchased) and boot from that.
For business purposes where maximum performance is not so critical, running necessary apps in a VM avoids the problem you describe.
You can do this with multiple partitions on the same physical drive assuming you have the space available.
For example: Apple's Time Machine seems like a slick, safe way to backup your entire hard drive. And, you can use it to transfer the entire old drive to a new drive. But, if there is an I/O error reading any of the files on backup, it will simply skip that file, and the only indication of a problem is in the /var/log/system.log. It can skip the file for months... eventually reaching the point where any older backup where the file was successfully read is aged out. And even on recovery: the file was skipped, so there's no indication it's even missing. Only when an app needs it is there any indication that months of backups, and a supposed full-recovery, had silent failures.
When I switched (think it was under 10.5) from regular to filevault it stopped doing backups.
I happened to pay attention to the time that it took and noticed it seemed extremely quick.
I decided (as I mentioned in another reply here) to use Super Duper which makes a clone of your hard drive and is easy to verify (just boot from the cloned drive).
I see an small app opportunity here.
There are a couple 3rd-party apps to address: one's called 'Time Machine Buddy' and another is 'Log Viewer for Time Machine'. Neither is super-slick (or seems to do anything more than grepping /var/log/system.log for backupd info), but they cover the basic need... once you know it's an issue, which may only happen after being bitten.
Not sure if it's true or urban legend but I remember reading a story about 15 years ago about tape backups being left on the seat of a car with heated seats. When they finally needed them they had been erased by the electric current in the seats. (That's the rough story that I remember at least).
The heat maybe, that could do it, magnetic media doesn't like heat. A car warmer would not be enough, but a hot car in the summer might be.
Has much more impact than simply reading "important to test backups".
That said the tapes I believe were large reels and the seats were an old Saab or something like that. So who knows the force field that thing gave off.
not saying the host is innocent here, but the 'victim' is really only a victim of his own stupidity.
We regularly have people who go into their accounts and change dns without realizing what that means. They only know that either the web "guy" or the new host told them to do that. Or they read they should do that in the setup instructions.
They may not have purchased email service at the new host and they think that everything will just work.
Many times if they speak to 1st level tech support at the new host they don't know enough to even know what will or will not happen with a dns server change.
If you read into the history, the guy clearly did not follow basic instructions. Backups are not part of the service he bought; yet the host backed up anyway. The host informed him that backups were disabled on this account because it was over quota and he did nothing to resolve the situation. He took no backups of his own.
He only lost his data because he failed to follow instructions or respond to emails and subsequently cancelled his shared hosting account before the data had been migrated over to a new VPS service.
I think the hosting company were in the right here.
Here's the forum thread for those that are interested: http://forums.ventraip.com.au/showthread.php?t=5572