"Customers hit the Whirlpool forums to complain that Distribute.IT had not adequately responded with information about the break-in and that the hack 'has probably killed my business'."
I'm sorry for people when their web sites are disrupted. I am astonished when it 'kills their business.' One would think that in this day and age one would 'restore from the last backup' and move on.
So is there a web opportunity to add a value "over" AWS/S3 which is the equivalent of managing a strip mall property in the real world?
> I'm sorry for people when their web sites are disrupted. I am astonished when it 'kills their business.' One would think that in this day and age one would 'restore from the last backup' and move on.
Having worked for a hosting company in Australia, I can say that most of the customers who suffer these problems are 1) small business owners and 2) not tech savvy.
Most of their sites are pre-packaged e-commerce solutions, and the idea of doing a weekly DB dump + /www tar is just beyond them. They would probably be better off with something like Shopify, but many of them also want local support & like the idea of "DIY". Unfortunately for them, doing it yourself means you're handling all the risks.
Cool, thanks for the direct perspective. So it does sound like there is perhaps some space for s service which a company could provide which looked to the shop owner as a 'DIY' service but was implemented as setting up various values in standard templates on the back end? Sort of like 'App Engine' for small business where you "design" it but the modules you use are just front ends for the underlying cloud services?
> Cool, thanks for the direct perspective. So it does sound like there is perhaps some space for s service which a company could provide which looked to the shop owner as a 'DIY' service but was implemented as setting up various values in standard templates on the back end? Sort of like 'App Engine' for small business where you "design" it but the modules you use are just front ends for the underlying cloud services?
I still think many people "DIY'ing" shouldn't. The sites often look horrible, have no way to track stats (conversions), etc.
What would be better is it if the hosted e-commerce players really pushed to market to these people who think DIY'ing is the better way.
Chuckle. Agreed 100% however folks want to be independent, even when they aren't as good at it as they might think they are. From the psychology perspective if you can make a system that "feels" DIY but really isn't so DIY that the final product looks DIY, that is the value proposition in this market. You sell people the notion they can "do it themselves, no need for techno-nerds telling you how you're doing it wrong!" and then you give them a design system on rails for which the 'exit' points are all decent web sites.
Apart from FBI raids, hosting the USA is a much better option.
I run a blog network (http://ozblogistan.com.au) on a pair of Linode VPSes out of their Fremont centre. I periodically reprice hosting in Australia, but basically bandwidth charges make it prohibitive to host locally for the amount of traffic I get.
I'm talking $100+/mth differences.
Still, independent backups are vital. I use tarsnap.
Its changing slowly. A lot of my clients are happy to host Overseas these days since the prices are a LOT more competitive (my preference being Atum in Canada).
It was certainly true 5-10 years ago, and may still be true for older people, but my experience has been that you get better support overseas now anyway. The biggest issue for me has been that scheduled downtimes are usually in the middle of my day, but you can work around that.
In my experience the lag time to the US is a huge factor in overseas hosting. Otherwise I'd just have everything setup overseas because Telstra has the entire industry by the nuts in terms of data costs (bring on the NBN!)
> In my experience the lag time to the US is a huge factor in overseas hosting. Otherwise I'd just have everything setup overseas because Telstra has the entire industry by the nuts in terms of data costs (bring on the NBN!)
It's not Telstra. There's plenty of non-Telstra international transit, and enough (but not plenty) of non-Telstra domestic transit.
The problem is that to run a hosting business in Australia, you don't get huge benefits from scale, and you're having to maintain a high staff member-to-customer ratio. Customers have a lot of protection (a good thing) through Fair Trading, and it seems that (from my experiences working in the industry) Australian users have a higher expectation of customer service.
Plus, there's also just the time-zone thing. Calling up mid-afternoon in Australia puts you at late-night in the US.
We have certain data on Aussie servers thats not allowed to be hosted outside of Australia. Think medical records and such. That said we have backups of backups of backups. The said part about this incident is everyone in the chain who really really should have seen to it that they had good backups had no backups at all.
It may just be that hosting overseas is inappropriate for a local audience - a connection to the USA is far slower than a connection in the same city.
For example, from my work on a fibre connection, pinging a local website takes about 2ms, and a US website about 160ms. Double that for residential DSL.
Also, national bandwidth is essentially free, and bandwidth to the USA is $$$.
I'm actually in NZ, it might be different in AU, but generally when you buy colocation here you only pay for international bandwidth; national bandwidth is free.
Unless you are running a gaming service, latency hardly matters for 'everyday' websites. With most people on broadband of some sort, 2ms or 160ms ping times is not relevant - they wouldn't even notice the split second delay before the page started loading.
Fair point Steve - I was a little too broad brush in my response. Yes I agree latency matters - but I was replying to your response in the context that you proposed that business was hesitant to host overseas due to latency issues.
My reply was in line with the parent post that the (implied) reason that Aussie business are reluctant to host overseas due to ignorance of options, perceived issues around support etc. From my dealings with Aussie business (of the sort that would host at lower tier hosting firms like the one in the article), they are ecstatic just to have a website. Issues such as latency don't even come into consideration.
What I find completely amazing is that business's don't have adequate backups of their websites and information and are not able to move to another hosting provider and get set back up.
I have multiple copies of my own personal websites in all kinds of different locations, stored both at home, as well as on remote servers, and backed up within the cloud.
Any eCommerce company needs to realise that their livelihood is at stake and that if they can't get back online within a day then maybe they are in the wrong business.
Unfortunately many if not most basic businesses who start operating online just hire 'some guy' who uses the bare minimum tools like Dreamweaver, templates and common CMSes, to throw together something like what the client wants/needs and then abandons them and moves on to their next victim.
He never tells them about correct backup handling or much else, really.
Yep, spot on. That, or they learn how to get a basic ZenCart or OSCommerce install running themselves.
I agree that if you don't know how to get your business back online within a day or so, that you don't deserve to be running a business and that you've taken that risk on by trying to cut costs/corners elsewhere. Still, my idealising doesn't stop people from doing it.
Just as a point of contrast between what I would consider minimum best practices, and what they were doing:
We not only back up all of our hosted websites every day, on behalf of our clients, and make those backups available to our clients going back 120 days so that they can restore any files they accidentally lose on their own, but we also back up all mail accounts every hour and can restore individual mail messages if necessary. All of the backups are stored in a private location separate from all of our servers, which are scattered across the country.
To have 4000 customers and not have even a whiff of that kind of architecture is, to me, completely and totally inexcusable.
I don't think it's accurate to say they didn't have a whiff of this architecture. The hackers accessed and deleted their backups. If your employees can do this then you're vulnerable.
The backup system can't be accessed from the servers, and individuals can't delete backups, for exactly this reason. The backup system uses a public key ssh login to the servers and has access only to a pre-configured rsync command through sudo. The backup file system is mounted noexec to prevent any other goofiness.
Having backups that are accessible if you get hacked doesn't really cut it. I'm guessing there wasn't a huge amount of data involved here, and if there was a set of tapes in a safe somewhere a lot of people would be much happier right now.
Maybe this isn't viable for budget hosting. But even the host's website is unavailable, obviously that is critical for their business and it appears it itself wasn't securely backed up. And this is hardly a totally unexpected scenario, it should have come up pretty quickly in a "what could go wrong" stage of their backup planning.
Yeah, I agree completely. What they had wasn't adequate and the proof is the outcome.
All I'm saying is that from the outside it's easy to take a fairly simple view and propose a technical solution. This looks like a very malicious attack designed to take down the business, the sort of thing that doesn't happen without a reason. If this is the case here then the prolem changes from one that's purely technical to something bigger. Something that can't have a purely technical solution. (Note: I'm not part of distribute it as a may be suspected by the fact that I'm new to HN.)
IMO, anybody that decides to venture into hosting should behave as though they've just walked into a warzone with a huge red bullseye on their back, and take precautions accordingly. Yes, there are certain things that just aren't immediately feasible before you launch, but the goal should be to get basic redundancies online quickly while you're operating.
By the time you have 4,000 customers, if you don't have backups for your backups, you're being negligent. And I say that without any malice whatsoever ... my experience with a lot of hosting companies, both big and small, is that Distribute wasn't doing anything out of the ordinary.
The thing is, targeting backups isn't new at all. It's been done before, and made the news before; at this point, it's not something that should surprise a sysadmin. i.e., the thought process immediately after setting up your backups should be, "OK, now what happens if a hacker tries to hit them too?"
So, yes, this is armchair quarterbacking, and yes, this is common behavior in the industry. But that still doesn't make it excusable in the least.
EDIT: Just to expound a little more on this, the reason I have such a hard-line stance on this is that, as a hosting provider, you are effectively taking responsibility for your customers' data and, in some cases, their livelihood. Yes, ideally, every customer would have their own backups and could move themselves to another host within an hour, but the reality is that it doesn't happen that way. Customers often have websites whose only copy is on your systems, email that's stored only on your systems (because they habitually use webmail, a service that you provide which makes that problem possible). Having "not our responsibility" in your TOS is very much not enough; you must be taking every reasonable precaution to safeguard your users' data, and in this case, Distribute -- along with many, many other hosting providers -- was not, because they did not have secured backups.
As an ex-developer of a company who had servers co-located with them, I'm feeling mighty smug about using our own backup system instead of their in-house one.
As mentioned earlier, bandwidth in Australia is prohibitively expensive so the choice wasn't as simple as "use both".
It's absolutely terrifying that there exists in the world the type of individual who would plan and carry out such vast and permanent destruction of unique information. I can't imagine a motive that would rationalize something like this.
The ignorance level of some of those comments is amazing (not the comments here, the ones on the article).
"I am a apple fan, but you can be assured that I will NOT be using iCloud for this very reason.
Always have a back up and NEVER rely on cyber space."
"Moral of the storyis GET YOUR HEAD AND EVERYTHING ELSE OUT OPF THE CLOUD(S)"
"Like others on here, I feel that the rush to the "latest and greatest" IT thing of the cloud has serious implications - when you are not in direct possession or responsibility of your data, and something happens to it, what do you do?"
"This is why cloud computing is destined to fail. 'Cloud Shocks' like this will force people to rethink the whole concept and write cloud computing off as just a 'fad'. This isn't the first cloud shock and will not be the last."
The company in the article wasn't even a cloud provider...and further more, if they were a cloud provider they would have more chance of getting their data back. Some people just don't know the difference between the internet and cloud computing..
This is an article on a mainstream newspaper's website. Despite what they might think of themselves, the reader base mainly consists of people who know rather little about how computers and the internet work.
The general level of reader commentary on the SMH website is pretty pathetic, no matter what the subject matter is.
This isn't helped by the current trend for their articles to be link bait or scraped off a real tech source, rather than anything approximating journalism.
43 comments
[ 4.8 ms ] story [ 107 ms ] threadI'm sorry for people when their web sites are disrupted. I am astonished when it 'kills their business.' One would think that in this day and age one would 'restore from the last backup' and move on.
So is there a web opportunity to add a value "over" AWS/S3 which is the equivalent of managing a strip mall property in the real world?
Having worked for a hosting company in Australia, I can say that most of the customers who suffer these problems are 1) small business owners and 2) not tech savvy.
Most of their sites are pre-packaged e-commerce solutions, and the idea of doing a weekly DB dump + /www tar is just beyond them. They would probably be better off with something like Shopify, but many of them also want local support & like the idea of "DIY". Unfortunately for them, doing it yourself means you're handling all the risks.
I still think many people "DIY'ing" shouldn't. The sites often look horrible, have no way to track stats (conversions), etc.
What would be better is it if the hosted e-commerce players really pushed to market to these people who think DIY'ing is the better way.
Chuckle. Agreed 100% however folks want to be independent, even when they aren't as good at it as they might think they are. From the psychology perspective if you can make a system that "feels" DIY but really isn't so DIY that the final product looks DIY, that is the value proposition in this market. You sell people the notion they can "do it themselves, no need for techno-nerds telling you how you're doing it wrong!" and then you give them a design system on rails for which the 'exit' points are all decent web sites.
I run a blog network (http://ozblogistan.com.au) on a pair of Linode VPSes out of their Fremont centre. I periodically reprice hosting in Australia, but basically bandwidth charges make it prohibitive to host locally for the amount of traffic I get.
I'm talking $100+/mth differences.
Still, independent backups are vital. I use tarsnap.
It was certainly true 5-10 years ago, and may still be true for older people, but my experience has been that you get better support overseas now anyway. The biggest issue for me has been that scheduled downtimes are usually in the middle of my day, but you can work around that.
It's not Telstra. There's plenty of non-Telstra international transit, and enough (but not plenty) of non-Telstra domestic transit.
The problem is that to run a hosting business in Australia, you don't get huge benefits from scale, and you're having to maintain a high staff member-to-customer ratio. Customers have a lot of protection (a good thing) through Fair Trading, and it seems that (from my experiences working in the industry) Australian users have a higher expectation of customer service.
Plus, there's also just the time-zone thing. Calling up mid-afternoon in Australia puts you at late-night in the US.
For example, from my work on a fibre connection, pinging a local website takes about 2ms, and a US website about 160ms. Double that for residential DSL.
Also, national bandwidth is essentially free, and bandwidth to the USA is $$$.
It's generally billed at the same rate for all traffic without distinction. By hosting in the USA you assign the cost for bandwidth to users.
The economics of our ISPs and hosts are driven entirely by the price of about 3 major pipes out of the country.
http://mike.bailey.net.au/2010/07/latency-is-a-killer/
My reply was in line with the parent post that the (implied) reason that Aussie business are reluctant to host overseas due to ignorance of options, perceived issues around support etc. From my dealings with Aussie business (of the sort that would host at lower tier hosting firms like the one in the article), they are ecstatic just to have a website. Issues such as latency don't even come into consideration.
http://radar.oreilly.com/2009/07/velocity-making-your-site-f...
So potentially, the money you save by hosting overseas is lost in lower conversion.
Bingo
I have multiple copies of my own personal websites in all kinds of different locations, stored both at home, as well as on remote servers, and backed up within the cloud.
Any eCommerce company needs to realise that their livelihood is at stake and that if they can't get back online within a day then maybe they are in the wrong business.
He never tells them about correct backup handling or much else, really.
I agree that if you don't know how to get your business back online within a day or so, that you don't deserve to be running a business and that you've taken that risk on by trying to cut costs/corners elsewhere. Still, my idealising doesn't stop people from doing it.
We not only back up all of our hosted websites every day, on behalf of our clients, and make those backups available to our clients going back 120 days so that they can restore any files they accidentally lose on their own, but we also back up all mail accounts every hour and can restore individual mail messages if necessary. All of the backups are stored in a private location separate from all of our servers, which are scattered across the country.
To have 4000 customers and not have even a whiff of that kind of architecture is, to me, completely and totally inexcusable.
Maybe this isn't viable for budget hosting. But even the host's website is unavailable, obviously that is critical for their business and it appears it itself wasn't securely backed up. And this is hardly a totally unexpected scenario, it should have come up pretty quickly in a "what could go wrong" stage of their backup planning.
All I'm saying is that from the outside it's easy to take a fairly simple view and propose a technical solution. This looks like a very malicious attack designed to take down the business, the sort of thing that doesn't happen without a reason. If this is the case here then the prolem changes from one that's purely technical to something bigger. Something that can't have a purely technical solution. (Note: I'm not part of distribute it as a may be suspected by the fact that I'm new to HN.)
IMO, anybody that decides to venture into hosting should behave as though they've just walked into a warzone with a huge red bullseye on their back, and take precautions accordingly. Yes, there are certain things that just aren't immediately feasible before you launch, but the goal should be to get basic redundancies online quickly while you're operating.
By the time you have 4,000 customers, if you don't have backups for your backups, you're being negligent. And I say that without any malice whatsoever ... my experience with a lot of hosting companies, both big and small, is that Distribute wasn't doing anything out of the ordinary.
The thing is, targeting backups isn't new at all. It's been done before, and made the news before; at this point, it's not something that should surprise a sysadmin. i.e., the thought process immediately after setting up your backups should be, "OK, now what happens if a hacker tries to hit them too?"
So, yes, this is armchair quarterbacking, and yes, this is common behavior in the industry. But that still doesn't make it excusable in the least.
EDIT: Just to expound a little more on this, the reason I have such a hard-line stance on this is that, as a hosting provider, you are effectively taking responsibility for your customers' data and, in some cases, their livelihood. Yes, ideally, every customer would have their own backups and could move themselves to another host within an hour, but the reality is that it doesn't happen that way. Customers often have websites whose only copy is on your systems, email that's stored only on your systems (because they habitually use webmail, a service that you provide which makes that problem possible). Having "not our responsibility" in your TOS is very much not enough; you must be taking every reasonable precaution to safeguard your users' data, and in this case, Distribute -- along with many, many other hosting providers -- was not, because they did not have secured backups.
As mentioned earlier, bandwidth in Australia is prohibitively expensive so the choice wasn't as simple as "use both".
At least hold the data for ransom or something.
Does anyone know of a security auditing service that customers could look for when evaluating hosting?
"I am a apple fan, but you can be assured that I will NOT be using iCloud for this very reason. Always have a back up and NEVER rely on cyber space."
"Moral of the storyis GET YOUR HEAD AND EVERYTHING ELSE OUT OPF THE CLOUD(S)"
"Like others on here, I feel that the rush to the "latest and greatest" IT thing of the cloud has serious implications - when you are not in direct possession or responsibility of your data, and something happens to it, what do you do?"
"This is why cloud computing is destined to fail. 'Cloud Shocks' like this will force people to rethink the whole concept and write cloud computing off as just a 'fad'. This isn't the first cloud shock and will not be the last."
The company in the article wasn't even a cloud provider...and further more, if they were a cloud provider they would have more chance of getting their data back. Some people just don't know the difference between the internet and cloud computing..
This isn't helped by the current trend for their articles to be link bait or scraped off a real tech source, rather than anything approximating journalism.
The general level of reader commentary on newspaper websites is pretty pathetic, no matter what the subject matter is.
As much as I despise The Herald Sun and News Ltd., they do the whole online community thing better and the site is better for it.