"It's interesting how many companies simply shut down service rather than say give a warning and wait for a response (or at least start a clock)."
I'm sure many people have started their companies firmly convinced that they'll give plenty of warnings and never automatically shut anything down.
The problem is, you rapidly discover that doesn't scale, not even on a human level. You send your notice. 48 hours later, you've gotten no response. If you act now, it isn't materially different from your point of view as if you simply acted right away.
Also, in a cloud environment, even Digital Ocean, as many people have learned the hard way with leaked credentials, you can rack up charges faster than the relevant humans can even conceivably be notified. As the hosting company, you can't just let abusive or accidental usage go. You can refund their money, but that's still resources of yours that went to something that failed to produce revenue rather than something that did; you can't absorb that indefinitely.
I'm pretty sure you'll inevitably discover that you have no choice but to put automation in.
This is exactly why AWS has relatively low default account limits, and you have to open a support ticket to raise them. It's largely to prevent run-away costs from surprising the customer.
I accidentally left a 24xlarge instance running for a month without realizing it and they looked at the activity and were totally cool about zeroing the bill for that instance for the month. Basically gave me us a $2000 credit.
It does probably help that I said I would be careful not to do that again and had already put in a CloudWatch Alarm to automatically power-off the instance after a set period of idleness before filing the ticket.
There have been so many stories of AWS accounts being “hacked” (actually they weren’t. someone posted their keys to a public github repo), the person panicking, then sending a ticket to AWS and then getting a refund. AWS support is excellent - especially on the business tier and above.
I will gladly pay the extra money for AWS than to even think about DO or even GCP for a money making project.
But more on topic: with Aurora/MySQL you can have an on-site hosted read replica from an AWS hosted database. That would be a cheap, easy real time backup solution if I were really worried about AWS screwing me over.
The good will generated by the stream of customer testimonials of this process we hear about is priceless.
The proposition seems to go something like this: it's a new thing, mistakes are statistically expected, you make an honest one and plead "oops!" and we refund you, no doubt pointing you to resources on best practices and account throttling. As long as the customer takes the lesson to heart, everyone wins.
This. It's fairly easy to setup from the provider side and easily solves this problem. Rate limits are fairly easy and can be automated based on criteria like account length/payment/abuse incidents. I think disabling the account is a little heavy handed unless it's brand new
For the customer, there is a big difference. Hitting an API to send an email and a text 48hrs before shutting down services is a common courtesy and easily automatable.
The host should throttle resources in the interim if its at risk of running up massive bills or adversely impacting other clients. None of this is breaking new ground, there isn't a good reason for large hosts to act like shit.
Your point is reasonable in many cases. But in this particular case, the charges would not have been substantially bigger than what this company paid before, and an automated system ought to take that into account.
If I’m a $10/month customer, kill my account early, and it’ll save me more often than not. If I’m a big spender, maybe wait a bit longer.
They locked my account, without refunding the ~$200 balance, with no reason given except "We reviewed the account and found it matches unusual patterns associated with violations of our Terms of Service and Acceptable Use Policy." When asked, they would not reveal which terms were violated.
No warning was given, and no way to retrieve any data. Fortunately nothing essential was lost.
I know this doesn't help you now, but you may want to consider distributing your site or setting up DR sites across multiple VPS providers. If your application supports it, you may even want to consider using a DNS provider that can do health checks and fail over the site for you.
I have happily used DO for small things, like hosting a Ghost blog, or running an Algo VPN, but I am a little surprised to see people doing bigger infrastructure with them - not that they deserve to lose everything for making that choice, but it seems like it would have been clearly riskier than AWS/GC/Azure?
I'm surprised that people put significantly more faith into GCP, AWS, Azure etc... similar things have happened in lots of scenarios. Not to mention when registrars have taken down sites.
This is a shame, and imho it sucks a lot. One of my biggest points of paranoia are with backups and scripting a return of a site on another provider should the worst happen.
Getting ready to launch something barely more than a hobby and was planning on DO because the hosted Postgres and a small K8s cluster is significantly less for what's there than the alternatives. Frankly, I don't want to go from ~$100/month to over $200 for another provider for something that likely won't lead anywhere.
that last response from them is pretty damning. it's like what happens when your customer success turns off their brain and just applies blanket rules to everything.
yikes, just when i thought their kubernetes and managed db's were looking attractive...
Strong anti-recommendation for this advice. The cost in dev time plus keeping things running is not worth it. This is a rare event. Just use a more well-established cloud service.
That's generally good advice, but I think it's important to keep backups and other disaster-recovery stuff somewhere else. Heck, just buy a NAS box and sync things nightly.
Not at all trying to blame the original posters and victims:
While they seem great for hobbyist and small business sites, there’s no way I’d trust Fortune 500 client business to something like DigitalOcean. I just don’t see the benefits over a more established operation like AWS, Azure or GCP. Saving $50 here and there isn’t worth it.
I can't speak on Digital Ocean and I guess it depends how you define "viable" but I've been running my upper six-figure revenue business on Heroku with tremendous success over the past 6 years. Customer support is super responsive and very helpful. The minimal downtime I've faced had has largely been the fault of AWS.
Couldn't agree more! We've been using Heroku at ReadMe for ~5 years and it is easily my favorite piece of technology we use. Off the top of my head I can't think of a single issue that entire time that has been their fault.
Heroku is expensive but fantastic. Unless you have complicated needs and the price point doesn’t shock you, it’s a fantastic way to use AWS while sparing yourself a lot of headaches.
Conversely, I don't see how a company like DO can afford to offload their customer support to automation for these "we'll shut down your business" kinds of tickets. Tweets like these generate a lot of press.
Your "established operations" are more expensive and GCP is newer than DO. Plus, getting the attention of someone to restore your account is probably easier in DO than in a faceless giant company like Amazon, Microsoft or Google.
My AWS support tickets usually get answered in two minutes or less and I have a dedicated rep who I can call whenever I want and we have regular check-ins anyway.
You get what you pay for. We're even upgrading from this support plan to an Enterprise account.
Startups can usually get enough in AWS credits that they probably could have their entire first year of service _for free_.
AWS support is a lot more responsive even if you are not a big source of revenue. One time when I had an issue, I directly reached out to our startup program point of contact and they made sure everything was resolved by constantly following up with the interval team responsible and keeping me in the loop
Yes, this is a bad look for DO. But the way they're able to beat AWS on price includes things like "worse support." And if AWS goes out of business, you'll know in advance. DO isn't the same story. You should be planning for redundancy if DO is truly business critical for you.
See, you totally can run big stuff DO (or Vultr or whatever). It just takes 20 minutes on the phone to make sure they know they can be paid when the time comes.
That was true a long time ago, but recently DO has also proven to be a professional cloud hosting provider in the same league. Frankly, I was expecting the same level of support/service as AWS from them.
BTW, I don't even see a downvote button around posts, there is only upvote button! Is it because I'm new here?
You can only downvote once your own comments have been upvoted a bit.
Keep in mind though that you should only downvote for abusive comments (which also deserve a flag), gross misinformation, or other things like that. Disagreeing with someone is not a good reason to downvote.
DigitalOcean is established. They're newer than GCP and about the same age as Azure, and IIRC at one point were the second largest VPS provider, second only to AWS.
And unfortunately this stuff happens to your "established" examples as well. Here [1] is a particular example of Google shutting down an entire GCP account with no explanation. Some comments report the same on AWS as well. Ironically, people in that HN thread are actually suggesting Vultr (kinda like DO, but even smaller) as a good alternative.
We rely almost exclusively on DigitalOcean and the stories about Google shutting down accounts really gives me pause. It's not an isolated incident either, and raising support is next to impossible, so I hear.
I'm lucky enough that my spend with DO is high enough to qualify for support, so if this ever happened to me at least I know I'd get a couple chances to make things right
Were this to happen on GCP I'm fairly certain they'd just black hole my account since I'm spare change to them.
To be blunt I've heard of GCP doing this to other people before. AWS and Azure though both understand that customer service is extremely important, and shutting down services destroys confidence. In the case of them suspecting malicious activity they'd have actual security people look at what's happening, and then maybe blackhole their traffic while they start calling people.
After even a single incident like this no sane company would relay on DigitalOcean. This is the kind of crap you expect from a shared web host overselling resources, not from a company that wants to provide infrastructure to tech companies.
I think it's the opposite. The cofounder intervened and the access was restored. This would never happen with Google or Amazon -- once they lock you out, your entire business is permabanned, and you won't be able to reach any human with authority to help you.
Maybe - AWS at least, you'll have an account rep to bang on whose job it is to remove obstacles to your spending more. I'm not sure how big exactly you have to be to get an account rep, but my small company with only 50 instances has one.
There's a pretty hard cap on the level of redundancy you can do with a two-man company, as I assume a two-man company does not bring in a lot of money.
The number of employees shouldn't be the deciding factor when you are a tech company that apparently has fortune 500 companies as customers.
I'm not talking 5 9's redundancy. I'm talking grab a backup once a week or something, anything, to help mitigate a scenario like this. According to the thread, they lost ~1 year of data. That should be unfeasible to a company serving customers, let alone Fortune 500 customers.
Disaster recovery planning is key for a technical company to succeed. It is clear they never considered a scenario where there DO account would be closed/compromised/down.
>It is clear they never considered a scenario where there DO account would be closed/compromised/down.
I don't think the chance of DigitalOcean automatically freezing your account to a point where only a co-founder can do something about it has been well publicised.
In all practicality, DO freezing your account has the same effect of DO being down (or closing, etc.), or your account being compromised and you being locked out of it.
A contingency plan should ideally have been in place for a scenario where, regardless of root cause, you have lost access to your DO account.
Sure, but them closing combined with the chance of them freezing your account (feasible, considering the topic here) and the chance of account compromise, and the chance they go down for extended maintenance... It is inexcusable not to have a disaster recovery plan for the scenario where you cannot access your DO account.
Imagine you are a customer of this company. Would you be rallying to their defense, "backups aren't needed because the scenarios are unlikely", or would you be angry that the company had zero contingency planning and lost all of your data (or the data you rely upon)?
If you can honestly say, as a (hypothetical) customer of the company in the thread, that you wouldn't care if a company you relied upon has no disaster recovery planning, more power to you. I, however, like to make sure that the companies I'm relying on have some sort of contingency that protects me as a customer.
Plenty of one man companies have Fortune 500 customers. Most are ran like the startup in question, as basic disaster mitigation is overhead. Don't expect the world from one guy keeping a company afloat, unless you enjoy disappointment.
From my experience you don't have fortune 500 companies as a customer as a >50person,>5mil revenue company.
You may have employees from such companies paying you via their business credit card, but will never get through real procurement without documented SLA procedures which are required to prevent a scenario exactly like this.
Yeah maybe, but not having an offsite backup is asking for trouble. They just failed to consider that "their site" == "digital ocean's cloud", and that since it's someone else's computers, they could easily be locked out at any time for any dumb reason including this one.
An expensive lesson to learn to take and check backups regularly.
Are there guides to this that take you through good configs for smaller environments like this? i.e. you have a postgres DB and two web servers. What's the simplest backup process and how do you replicate to another DB VM over on another VPS provider securely?
Stuff like this is relatively simple when you are trying to learn it, but keeping it operation is hard at a small level. Is the less to really use PaaS until it's viable for you to be running a small K8s cluster, or equivalent fleet? Seems really expensive compared to VPSes, but having better guides might help.
True, but it isn’t hard to add S3 as an additional backup destination. Ideally if they had their infrastructure setup as code, it would have made recovery possible.
There are certainly a lot of limits on what you can do with such limited resources, but a reasonable backup with a different provider is certainly doable at that small scale.
It won't be entirely up to date when the worst case happens, you'll be unavailable and you'll probably have lost a day of data or so, but you won't have lost everything.
Rclone to AWS or Google or whatever is easy to set up, add a daily dump of your database to the folder you back up. Unless you handle a lot of data, costs are probably not a big factor.
Rather unfortunate customer service experience that you can't get help from the actual support and you can't get help from the official Twitter account. You just need to pray that your Twitter thread gets noticed hard enough for the actual co-founder to notice it so they can make the call that saves your company.
Also, even the co-founder doesn't seem to know exactly why the service was suspended, even though he clearly managed to arrange things.
Right? I hate that justice in these cases relies on the person tweeting and then that tweet hooking on and getting popular on Hacker News. So depressing.
This reinforces my gut feeling that companies like DigitalOcean and Scaleway are fine for hobby usage, but for any serious business operations, I'd go with AWS.
Well if anything else this case just made that abundantly clear. Locking a customer resources without warning, and not allowing access to their data, is unacceptable, I'd argue even for hobby services. There's a lot of VPS providers cheaper than DigitalOcean, I don't believe they have room to act like this.
Not even giving the account a warning is honestly enough for me not to go with DigitalOcean for even small projects in the future. Their prices are not that good for fly by night VPSs if that's how they're going to act.
It's not just the volume of usage that can indicates fraud, but the pattern. In addition, relying on quotas creates a system that is easily games by perpetrators of fraud.
Caps or quotas are not sufficient to deal with this problem.
If a cloud service is supposed to be ready for production, then customers should be safe to assume that they will not simply be shut down, especially not without warning. Otherwise, the provider must make clear that the service is only for hobby use and not for commercial use.
What kind of fraud do you have in mind and do you know of a case in which, for example, Azure switched off an enterprise customer due to unusual usage patterns?
Yes! My college roommate was shutdown in Las Vegas after eating for 3 hours. That was thirty years ago and I still can't stop laughing after witnessing such a debacle.
DigitalOcean is operating worse than a fly by night host (like AlphaRacks, GreenValueHost, etc). The reasonable course of action would've been to email the customer and throttle their API access to prevent load spikes, but DO instead locked their entire account (not just the service that DO felt was being abused).
A fly by night will often only suspend the VM or database that is in question, not other services on the account (having been in that position before).
Hmm, I've done lots of stuff on OVH that other hosts consider abusive, with nary a word of complaint from OVH.
OVH seems to have this whole automation thing down cold, monitoring boxes for dying components and letting me run hog wild on their VMs without limit...
The one time I used their phone support, the guy I got was fairly helpful. They seem like a hands off company overall.
This doesn't seem unreasonable, OVH is not advertising DDOS protection (nor do they seem to be structured to offer it). Some hosts will can your VM and throw out its data, which is a much worse outcome.
Far from flawless. They did it to me and it was dropping half of DNS requests I was receiving so my websites were down. And you practically have to beg them to take you out of DDoS protection.
Not in my experience they don't. I use OVH and nfoservers and I've had an issue like this exactly once on both hosts.
On OVH one of my servers was hacked and running typical scripts that are run once that happens (port checking, common admin credentials, brute force attempts, etc)
They cut off all internet access to and from the server and sent me an alert stating what was happening and that I needed to VNC into the server, resolve the issue, and let them know how/why it happened and how I resolved the issue. Once that was done they just removed all the blocks on the server and we all went on our merry way.
Edit: To clarify the VNC console is on their site, not a remote connection.
I run the IPFS daemon on Hetzner, and it was trying to connect to local IPs because of some misconfiguration. They sent me an email saying my server was portscanning their LAN, and I should fix it and email them how I fixed it.
I didn't know what they were talking about so I replied saying that, they helped me shut down the local port connections and I never heard another complaint from them. There was no downtime or banning at any point.
A machine on a GCP project got compromised. GCP emailed us right away and we fixed the issues promptly. No outages, no arbitrary suspension, and no grief apart from the compromise.
I saw a great deal on LEB for a KVM VPS from Alpharacks and signed up for a 2 year plan (my first mistake).
When SSHing in to the VPS it didn't have the advertised specs, and when I raised the issue with their support, they eventually fixed it..
Then I realized the second problem, they gave me the same IP address as someone else. You could still use the web VNC console, and as soon as you made an outbound network connection, inbound connections would work... for a few seconds... then SSH would drop. Reconnecting by SSH says "host key changed" i.e. you hit someone else's server sharing the same IP address. Using the web VNC console, works again for a few seconds, drops again.
It took about 7 days of arguing with their support to explain these two problems to them... by which time, the 3-day refund window had expired...
Lesson learned about low-end boxes. I have had about 50/50 good and bad experiences, using half a dozen providers like this, not really making financial sense overall.
I recommend everyone stay far far away from AlphaRacks. If anything remains of them after this week.
My point was focused on how shithosts handle abuse, they'd suspend your container, but any other products or services you have would remain available, your account generally wouldn't be locked (short of disputing a payment).
Best Twitter comment (grammar errors and all): "And if your full business relies on one tech partner (no offsite backups) your not doing your tech job right."
You're correct. But there are also unicorns that don't follow this rule. It's just that their compute activity would never trigger a false-positive, so everyone (except their ops team) is blissfully unaware of their fragility.
Maybe. But don't forget this is a small company of just two people. Yes the backups should have been off site but relying on just one digital partner at that scale isn't the worst (they're unlikely to have the money or time to federate out to other services).
Yes, ideally they would have already tested their back-up solution, the back-ups would be offsite and, if something like this happened, they could stand-up on another provider. But that just ignores the reality of them being a super tiny business. Almost no one at that size is going to do that.
I disagree. Many 1 and 2 person shops implement proper off-site backups because they understand that loosing their data is a death sentence for their business. Proper off-site backups are neither expensive nor time consuming to establish these days. There is no good excuse for even the tiniest of companies to not implement them.
Actually, you can implement cheap external backups even in a 1-man company.
But I still support that DO here is a 100% liable toward their client. Now the liability between the said client and his own clients is an other matter.
Ideally partners should be trusted (and trustful), in practice, they aren't
Though trusting DO/AWS/GCP, etc is much more reliable, than, let's say, betting your whole business on somebody's proprietary API (like an FB game, an Linkedin API something, etc)
I agree. Putting all your eggs in one basket these days indicates poor decisions or risk assessment.
> Digital Ocean "Trust and Safety"
Does this phrase give anyone else the heebie-jeebies? They deliberately locked his account without looking at previous metrics over the previous months to conclude this was obviously not a concern, and why?
Things that have crossed my mind:
Is their automatic system was poorly tuned?
Was this deliberately initiated? If so, why?
I agree. Who are all these people that pull backups, that maybe GBs or TBs in size for offline storage? How does that even work in practical terms in disaster scenarios like this where resolution times are expected within hours and not days?
It doesn't. I would hazard a guess that there are zero medium to large companies out there right now that could swap to a new cloud provider in a few hours.
Indeed. I'd urge everyone here to pay attention how many sites/companies are dead in the water the next time there is a full blown AWS outage and see if they are as quick to heave the same criticisms at those fortune 500 companies that they are at this two man operation.
What happened w/ Digital Ocean is inexcusable, and has potentially dire consequences for two individuals livlihood. In the immediate aftermath of such an event, focusing on the devs percieved lack of disaster preparedness seems petty.
That was the stupidest comment to me. This guy is desperate for help and this fart-sniffer is finger-waving someone he doesn't know online with unnecessary platitudes.
This is conflating two different things. One point is valid, the other is not.
- No offsite backups? Agreed. Even for a two person team it is sloppy.
- "Relies on one tech partner?" Strongly disagree. Even large enterprises often have a hard dependency on AWS, Azure, Rackspace, or similar. To suggest that a two person team should have deployment plans for multiple independent cloud vendors is just fantastical thinking with no basis in reality.
Plus if they did over-engineer it by making it cloud agnostic, setting up accounts to sit dormant, cold instances elsewhere, etc, people would just criticize them for that inefficiency/wasting time.
Some may have an availability dependency on those services, but if they don't have a full BC and DR plan ready to go within a few hours of losing those service they're not going to be a big enterprise for long.
The topic is having a single tech partner. If the partner is large enough you can have a high level of redundancy by utilizing their cross-zone/geographically distributed services (e.g. US East, EU West, Asia Pacific, etc).
How you got from "single tech partner" to "have no disaster recovery plan" I don't know.
Are you talking about a small WordPress site or something?
Very, very few tech companies could simply move everything to a new cloud provider in a few hours. I would even hazard a guess that almost none can.
I have all my infrastructure as code and can break it all down and spin it back up in kubernetes clusters in minutes. But due to the quirks of each cloud provider, there are tons of little fixes that would inevitably need to be made.
Not to mention that many companies have way more data than could even be copied over in a few hours.
On the other hand: Get customers and traction before you build a multi-site, fault-tolerant, self-healing, webscale platform that Google would be proud to have.
I think we needlessly shame one-person operations for focusing on actual customer needs instead of ops busywork and yak shaving.
No one that I've seen so far is saying they should have a system that is "muilti-site, fault-tolerant, self-healing, webscale that Google would be proud to have".
They are saying run a simple backup and keep it literally anywhere else.
I think we needlessly hyperbolize "do a backup once a month and keep it somewhere else" into some sort of NSA operation.
Its backups. Its 2019. Its dead-easy and very affordable.
A few years ago I used to run a VPS on DO with a mail server, VPN and some code I was writing. Once I was done with everything, I used their snapshots feature to backup my VPS and shut it down.
Two years later I wanted to restore the VPS but turns out my snapshot become "outdated" and they stopped supporting the format for restoration... Support was completely useless, wouldn't even let me download the snapshot, said at most they could mount it into a new VPS and I could recover the data myself.
That seems reasonable. Launching security-bug-ridden software seems dangerous, so mounting to a VPS so you can copy the important files seems an entirely reasonable access method.
You know, there is usually an easy way to mitigate this. When you create an account, be sure to supply business information e.g EIN in the USA and see if you can do invoice billing. Once you get set-up as a business, most providers assume you know what you are doing.
These sorts of things tend to slip through the cracks. If you are running is a business capacity, make sure you treat everything like any other business would.
Even as a two-man company, you can't afford the cost and potential liability of not operating as a registered business. It also shows that this is a serious business and not a hobby.
Not even surprised by the way Digital Ocean have handled this. They pulled something similar on me back in 2014 at a previous company I used to work in. They essentially shut down my account and did not even let me get my backups out.
It seems like there should be a middle-ground between all-on and all-off. If I'm paying customer, I should be able to access my account in some capacity even if some abuse related issue closes off server access.
Dude has no disaster recovery plan, so he throws a public hissy fit like a spoiled child. Public shaming as a customer support proxy is shameful. It’s a form of manipulation.
Who is good about not locking accounts or taking similar actions? Apple and Google are both notorious for blocking things for no reason, is AWS or Azure any better?
They are all going to have some form of automated protection against malicious activity, but I suspect AWS and Google’s algorithms are better than the others. My experience with AWS and Google in general is that your treatment varies with your support plan. With business or enterprise level, you have dedicated resources within the company that are going to be aware of such issues or can escalate and sort it out quickly. I understand not wanting to shell out the base cost for enterprise if you are a small company on a budget, but paying at least for business support is a good idea if you are actually running a business. I have never actually heard of this happening to such a customer though, so perhaps they have extra processes in place?
I've only had one instance where Linode contacted me about suspicious activity. I responded promptly, it was obvious they understood my answer, and nothing came of it. Another comment thread on this story has people saying Linode's support is better.
That said, any company, especially one working with Fortune 500's, should have DB backups in at least two places. If they'd had the data, they could have spun up their service on a different hosting provider relatively easily.
Sure. Hopefully it results in a change of policy, or at least a public statement of some kind. Everyone can't depend on the cofounder to come in and save them from bad automation.
Agreed, but it's not like the original poster had a huge platform, he just posted about it on Twitter. I may despise Twitter for a bunch of different reasons, but I can't deny it's a great tool for raising issues to companies.
That's the kind of response you only send when you're convinced the customer is actually nefarious and you don't care about losing them. I wonder if there is any missing backstory here or if it really is just a case of mistaken analysis.
I think the major issue there is process and management related. The account should have been reviewed by someone with the authority to activate it, and it definitely shouldn't have been flagged a second time. But looks like DO thought the user was malicious, and issues raised by malicious users don't get much information. The response was horrible though.
Used to work at Linode, let's flip this on it's head:
-When the majority of abuse support dealt with was people angrily calling and asking about the fraudulent charges on their cards for dozens of Lie-nodes you consider putting caps in place to reduce support burden and reduce chargebacks.
At the time at Linode, if it was a known customer, we could easily and quickly raise that limit and life is good.
I've always wondered how Amazon dealt with fraud/abuse at their scale.
I don't think DO was wrong here to have a lock, but the post lock procedure seemed to be the problem.
DO has shown that their service is simply not suitable for some use cases: those that impose an "unreasonable" load on their infraestructure.
Even worse: they don't explicitly state what is considered "unreasonable". So, if your business is serious, you have to assume the worst-case scenario: DO can't be used for anything
Conclusion: Digital Ocean is just for testing, playing around, not suitable for production.
> Conclusion: Digital Ocean is just for testing, playing around, not suitable for production.
I think that's always been the standard position most people take. DO, Linode, etc are for personal side projects, hosting community websites, forums etc. They are not for running a real business on. Some people do, sure but if hosting cost is really that big a portion of your total budget you probably don't have a real business model yet anyway.
I am of the impression that people rent cloud services because they can expense the cost to someone else or because of an inability to plan long term or a need of low latency.
You can provide a helpful message with options for recourse without giving abuser's "clues." These are not somehow mutually exclusive. By your logic it makes sense to punish a marginal element at the expense of the majority.
More realistically they would have done backups inside DO and would still be locked out. Not many people actually do complete offsite backups to a completely different hosting provider, getting locked out of your account is usually just not a consideration. It’s unrealistic to expect this of a tiny startup.
Whether it's normally a consideration or not, there are no meaningful barriers in terms of cost or effort, so it's totally realistic to expect it of a tiny startup.
Every week there's another article on HN about a tiny business being squished in the gears of a giant, automated platform. In some cases like app stores this is unavoidable, but there are plenty of hosting providers to choose from. People need to learn that this is something that can happen to you in today's world, and take reasonable steps to prepare for it.
>getting locked out of your account is usually just not a consideration
How many horror stories need to reach the front page of HN before people stop believing this? Getting locked out of your cloud provider is a very common failure mode, with catastrophic effects if you haven't planned for it. To my mind, it should be the first scenario in your disaster recovery plan.
Dumping everything to B2 is trivially easy, trivially cheap and gives you substantial protection against total data loss. It also gives you a workable plan for scenarios that might cause a major outage like "we got cut off because of a billing snafu" or "the CTO lost his YubiKey".
> How many horror stories need to reach the front page of HN before people stop believing this
Sounds like the opposite of the survivor bias. I don't believe it's any sort of common (though it does happen), even less that "it should be the first scenario in your disaster recovery plan"
Even if the stories we hear of account lockouts isn't typical, the absolute number of them that we see -- especially those (like this one) that appear to be locked (and re-locked) by automated processes -- should be cause for concern when setting up a new business on someone else's infrastructure.
If you plan for the "all of our cloud infrastructure has failed simultaneously and irreparably" scenario, you get a whole bunch of other disaster scenarios bundled in for free.
I don't know, it seems simple enough to me. I have a server on DO hosting some toy-level projects, and IIRC it took me 15-30 min to set up a daily Cron job to dump the DB, tar it, and send it to S3, with a minimum-privilege account created for the purpose, so that any hacker that got in couldn't corrupt the backups. I'm not a CLI or Linux automation whiz, others could probably do it faster.
We don't know the structure of their DB and whether failover is important or not, so we don't know if the DB can be reliably pulled as a flat file backup and still have consistent data.
We also don't know how big the dataset is or how often it changes. Sometimes "backup over your home cable connection" just isn't practical.
Cron jobs can (and do) silently fail in all kinds of annoying and idiotic ways.
And as most of us are all too painfully aware, sometimes you make less-than-ideal decisions when faced with a long pipeline of customer bug reports and feature requests, vs. addressing the potential situation that could sink you but has like a 1 in 10,000 chance of happening any given day.
But yes, granted that as a quick stop-gap solution it's better than nothing.
> We also don't know how big the dataset is or how often it changes.
I'm going to take a stab at small and infrequently.
Every 2-3 months we had to execute a python script that takes 1s on all our data (500k rows), to make it faster we execute it in parallel on multiple droplets ~10 that we set up only for this pipeline and shut down once it’s done.
Yeah, probably. But we shouldn't be calling these guys out for not taking the "obvious and simple" solution when we aren't 100% certain that it would actually work. That happens too often on HN, and then sometimes the people involved pop in to explain why it's not so simple, and everyone goes "...oh." Seems like we should learn something from that. I've gone with "don't assume it's as simple as your ego would lead you to believe."
I suggested that solution because everyone is saying "they're only a two-man shop so they don't have the time and money to do things properly". Anyone has the time and money to do the above, and there's a 90% chance that it would save them in a situation like this.
Even if they lost some data, even if the backup silently failed and hadn't been running for two months, it's the difference between a large inconvenience and literally your whole business disappearing.
> "2-man teams generally don't prioritize backups" isn't an excuse for not prioritizing backups.
They had backups, but being arbitrarily cut-off from their hosting provider wasn't part of their threat model.
Isn't a big part of cloud marketing the idea that they're so good at redundancy, etc. that you don't need to attempt that stuff on your own? The idea that you have to spread your infrastructure across multiple cloud hosting providers, while smart, removes a lot of the appeal of using them at all. In any case, it's also probably too much infrastructure cost for a 2-man company.
> In any case, it's also probably too much infrastructure cost for a 2-man company.
keeping your production and your backups in the same cloud provider is the equivalent of keeping your backup tapes right next to the computer they're backing up. you're exposing them both to strongly correlated risks. you've just changed those risks from "fire, water, theft" to "provider, incompetence, security breach"
So what is the purpose of the massive level of redundancy that you are already paying for when you store a file on S3? I don’t think it’s terribly common for even medium sized companies to have a multi tier1 cloud backup strategy.
Storing a file on two tier1s would surely protect you from fire, water, theft no? Yet you will also be paying for all the extra copies Amazon and Google each make. I'm not disagreeing that this is the right strategy, just pointing out that the market offerings and trends don't support it.
Back in the day, we used to talk a lot about how RAID is not a backup strategy. The modern version of that is that S3 is not a backup strategy.
> So what is the purpose of the massive level of redundancy that you are already paying for when you store a file on S3?
You're paying to try and ensure you don't need to restore from backups. Our data lives in an RDS cluster (where we pay for read replicas to try and make sure we don't need to restore from backups) and in S3 (where we pay for durable storage to try and make sure we don't need to restore from backups), but none of that is a backup!
If you're not on the AWS cloud S3 is a decent place to store your backups of course, but storing your backups on S3 when you're already on AWS is, at best, negligent, while treating the durability of S3 as a form of backups is simply absurd.
> I don’t think it’s terribly common for even medium sized companies to have a multi tier1 cloud backup strategy.
The company I work for is on the AWS cloud, so we store our backups on B2 instead. It's no more work than storing them on S3, and it means we still have our data in the event that we, for whatever reason, lose access to the data we have in S3. Who the hell doesn't have offsite backups?
> Back in the day, we used to talk a lot about how RAID is not a backup strategy. The modern version of that is that S3 is not a backup strategy.
This is not remotely the same thing. A RAID offers no protection against logical corruption from an erroneous script or even something as simple as running a truncate on the wrong table. Having a backup of your database in a different storage medium on the same cloud provider protects from vastly more failure modes.
> Who the hell doesn't have offsite backups?
No one. But S3 is already storing your data in three different data centers even if you have a single bucket in one region, and you also have SQL log replication to another region. Multi-region is as easy as enabling replication but that is only available within a single cloud provider (I can't replicate RDS to Google Cloud SQL, only to another RDS region). I would guess that a lot of people use that rather than using a different cloud provider.
> This is not remotely the same thing. A RAID offers no protection against logical corruption from an erroneous script [...] But S3 is already storing your data in three different data centers
That sounds like...the same argument?
A RAID array stores your data on multiple physical drives in the machine, but offers no protection against logical corruption (where you store the same bad data on every drive), destruction of the machine, or loss of access to the machine.
S3 stores your data in multiple physical data centres in the region, but offers no protection against logical corruption, downtime of the entire region, or loss of access to the cloud.
You can't count replicas as providing durability against any threat that will apply equally to all the replicas.
Yeah I think this is what people are not getting. Redundant backups might mean "don't worry, in addition to backups on the instance, I have them going to a S3 bucket in region 1 and then also region 2 in case that region goes down," which of course doesn't protect from malicious activity from the provider. You certainly _should_ make sure you have backups locally available or in a secondary cloud provider but this is some hindsight.
> being arbitrarily cut-off from their hosting provider wasn't part of their threat model
Let's be fair: The threat model here is "lose access to our data".
This can happen in a number of ways, lost (or worse, leaked) password to the cloud provider, provider goes bankrupt, developer gets hacked, and a thousand other things.
Even if you trust your provider to have good uptime, there's really no excuse for not having any backups. Especially not if you're doing business with Fortune 500's.
I feel for these guys, but that's not "all the proper backup procedures". I'm part of a three-man shop and storing backups in another place is the second thing you do immediately after having backups in the first place. Never mind being locked out by the company - what happens if the data centre burns to the ground?
I would say that it doubles the cost of backups, but using this math, we start with one copy plus one backup, and add a second backup; that means only a 50% increase.
As a startup, generally your secondary backup could literally be an external hard drive from best buy, or an infrequent access S3 bucket (or hell, even Glacier). No excuse, especially when "dealing with Fortune 500 companies".
Literally just push a postgres dump to S3 (or any other storage provider) once a night as a "just in case something stupid happens with my primary cloud provider". It'd take a couple hours tops to set up and cost next to nothing.
Most of the costs aren't from storage space, but compute power. We aren't talking about duplicating the whole infrastructure, just backing up the data. Disk space is dirt cheap.
Also, by "two places" I meant the live DB and one backup that's somewhere completely different. My wording may have been confusing.
They did have backups. Thats why I asumed you meant double backups. If you do cold storage you should have 3 copies due to possible corruptions. Sure tape drives are cheap but someone also have to run and check the backups.
That's probably a reason to use containerization / other technologies so that you can spin up your services in a couple minutes on a different cloud provider.
You don't need to use containers for that.. all you have to do is set up a warm replica of the service with another provider. The fail over doesn't even have to be automatic, but that is the minimum amount of redundancy any production SaaS should have.
A "warm replica" is going to cost money though, while containerization allows you to not have anything spun up until the moment you need it, and then have it ready to go minutes / an hour later.
That is patently false, unless you plan on starting from a clean slate on the new environment. Any one who proposed such a solution as a business continuity practice to me would be immediately fired.
Containers solve the easy problem, which is how to make sure the dev environment matches the production environment. That is it.
Replicating TBs worth of data and making sure the replica is relatively up to date is the hard part. So is fail over and fail back. Basically everything but running the code/service/app, which is the part containers solve.
> Sure, a backup would have been a significant improvement, but still – a backup only protects against data loss and not against downtime.
Assuming you have data backup / recovery good to go, the downtime issue needs to be solved by getting your actual web application / logic up and running again. With something like docker-compose, you can do this on practically any provider with a couple of commands. Frontend, backend, load-balancer -- you name it, all in one command.
> Containers solve the easy problem, which is how to make sure the dev environment matches the production environment. That is it.
> Account should be re-activated - need to look deeper into the way this was handled. It shouldn't have taken this long to get the account back up, and also for it not be flagged a second time.
So... he doesn't address what is the scariest part to me, the message that just says "Nope, we've decided never to give your account back, it's gone, the end."
I think it's entirely reasonable for companies to have that option. "You are doing something malicious and against the rules, you have been permanently removed". In this case, that option was misused, but I don't think the existence of that possiblity is inheritly surprising.
Access to your data should never be denied. Ever. It was not DigitalOcean's data. If you are a hosting provider, you can't ever hold customer data hostage or deny them access to it in any way.
Again, I must disagree. If DO genuinely believed that you were doing something malicious and that data was harmful or evil for you to own (e.g. other people's SSN, etc) then they are in the "right" to deny access to it. DO should not be forced to aid bad actors.
And, regardless of what DO should or should not do, they can do whatever they want with their own hard drives. You should structure your business accordingly.
For some practical, if extreme, examples: if a customer were to host a phishing site, or a site hosting CP, it would be grossly irresponsible (and likely even illegal) for the hosting provider to retain the customer's data after account suspension and allow them to download it.
And do what in the mean time? The legal system acts slowly. In the age of social media outrage, would you allow the headline "Digital Ocean knew they were serving criminals, and they didn't stop them" if you were CEO?
It's easy to be outraged when these systems and procedures are used against the innocent. That does not mean we should stop using rational thought. If someone is using DO to cause harm, then DO should (be allowed to) stop the harmful actions.
> Your account has been temporarily locked pending the result of an ongoing investigation.
You lock down the image, and let law enforcement do their thing. If law enforcement clear them, you then give the customer access to their data, perhaps for a short time before you cut them off as they seem to be a risky customer to have.
You don't unilaterally make the decision, you offload your responsibility onto the legal process.
>would you allow the headline "Digital Ocean knew they were serving criminals, and they didn't stop them" if you were CEO?
Seems to work just fine for AWS, Google and Cloudflare. In fact, counter to your argument, Cloudflare got in massive shit when they did decide to play God.
> If DO genuinely believed that you were doing something malicious and that data was harmful or evil for you to own (e.g. other people's SSN, etc) then they are in the "right" to deny access to it.
The observant will note the particular corner you're backing into here -- that a business might be justified in denying access to code/data being used in literally criminal behavior -- is notably distinct from the general and likely much more common case.
> they can do whatever they want with their own hard drives.
Sure. But to the extent they take that approach, Digital Ocean or any other service is publicly declaring that however affordable they may be for prototyping, they're unsuitable for reliable applications.
Businesses that can be relied on generally instead offer terms of service and processes that don't really allow them to act arbitrarily.
> ... a business might be justified in denying access to code/data being used in literally criminal behavior...
I agree. Look at the absolutism of the comment I am replying to. My whole point is that there might be some nuance to the situation.
> ...Digital Ocean or any other service is publicly declaring that however affordable they may be for prototyping, they're unsuitable for reliable applications.
Again, I agree. Considering how cheap AWS, backblaze, and Google drive is, it is completely ridiculous to depend on any one single hosting service to hold all your data forever and never err.
At no point did DO ever believe this. This happened purely and simply because of usage patterns changing. It was done automatically and a bot locked them out. They should not be locking out data based on an automated script.
You seem to be accusing the aggrieved party of being a bad actor, when that is not the case.
If DO believed that there was criminal activity (notice I am not using the word "malicious"), they should have reported it to the police, and it that case they might be justified in securing a copy of the data. Blocking access would be justified only in the most extreme cases (such as if the data could be harmful to others, e.g. pictures of minors).
If there is no police report, then they are trying to act as police themselves, which I think is unacceptable. It is not their data.
Your argument that they can do whatever they want with their hard drives is indeed something I will take care to remember — I definitely would not want to host anything with DO.
Reasonable to have the shutdown part of the option, yes.
At the very least, they should also provide ALL, as in every last byte, of data, schemas, code, setup etc. to the defenestrated customers. As in: "sorry, we cannot restart your account, but you can download a full backup of your system as of it's last running configuration here: -location xyz-, and all previous backups are available here: -location pdq-".
Anything less is simply malicious destruction of a customer's property.
If you violate a lease and get evicted, they don't keep your furniture & equipment unless you abandon it.
>That said, any company, especially one working with Fortune 500's, should have DB backups in at least two places.
They should have, at the very least, one DR site on a different provider in a different region that is replicated in real-time and ready to go live after an outage is confirmed by the IT Operations team (or automatically depending on what services are being run).
I can understand why a cloud platform would proactively disable workloads that appear to be acting dangerously. At the same time it seems quite unreasonable that there's no mechanism for a customer to get their data back when their account is unilaterally shut down.
619 comments
[ 1.9 ms ] story [ 368 ms ] threadGranted that would require people to communicate and use some form of reason.
Even DMCA for all its warts fires up a warning and has a response mechanism (granted other issues there).
I'm sure many people have started their companies firmly convinced that they'll give plenty of warnings and never automatically shut anything down.
The problem is, you rapidly discover that doesn't scale, not even on a human level. You send your notice. 48 hours later, you've gotten no response. If you act now, it isn't materially different from your point of view as if you simply acted right away.
Also, in a cloud environment, even Digital Ocean, as many people have learned the hard way with leaked credentials, you can rack up charges faster than the relevant humans can even conceivably be notified. As the hosting company, you can't just let abusive or accidental usage go. You can refund their money, but that's still resources of yours that went to something that failed to produce revenue rather than something that did; you can't absorb that indefinitely.
I'm pretty sure you'll inevitably discover that you have no choice but to put automation in.
It does probably help that I said I would be careful not to do that again and had already put in a CloudWatch Alarm to automatically power-off the instance after a set period of idleness before filing the ticket.
I will gladly pay the extra money for AWS than to even think about DO or even GCP for a money making project.
But more on topic: with Aurora/MySQL you can have an on-site hosted read replica from an AWS hosted database. That would be a cheap, easy real time backup solution if I were really worried about AWS screwing me over.
The proposition seems to go something like this: it's a new thing, mistakes are statistically expected, you make an honest one and plead "oops!" and we refund you, no doubt pointing you to resources on best practices and account throttling. As long as the customer takes the lesson to heart, everyone wins.
The host should throttle resources in the interim if its at risk of running up massive bills or adversely impacting other clients. None of this is breaking new ground, there isn't a good reason for large hosts to act like shit.
If I’m a $10/month customer, kill my account early, and it’ll save me more often than not. If I’m a big spender, maybe wait a bit longer.
No warning was given, and no way to retrieve any data. Fortunately nothing essential was lost.
This is a shame, and imho it sucks a lot. One of my biggest points of paranoia are with backups and scripting a return of a site on another provider should the worst happen.
Getting ready to launch something barely more than a hobby and was planning on DO because the hosted Postgres and a small K8s cluster is significantly less for what's there than the alternatives. Frankly, I don't want to go from ~$100/month to over $200 for another provider for something that likely won't lead anywhere.
Goosfrabah... goosfrabah...
https://twitter.com/w3Nicolas/status/1134529379701334017
https://twitter.com/moiseyuretsky/status/1134547532149854208
yikes, just when i thought their kubernetes and managed db's were looking attractive...
Not at all trying to blame the original posters and victims:
While they seem great for hobbyist and small business sites, there’s no way I’d trust Fortune 500 client business to something like DigitalOcean. I just don’t see the benefits over a more established operation like AWS, Azure or GCP. Saving $50 here and there isn’t worth it.
You get what you pay for. We're even upgrading from this support plan to an Enterprise account.
Startups can usually get enough in AWS credits that they probably could have their entire first year of service _for free_.
Yes, this is a bad look for DO. But the way they're able to beat AWS on price includes things like "worse support." And if AWS goes out of business, you'll know in advance. DO isn't the same story. You should be planning for redundancy if DO is truly business critical for you.
(I'm not trying to blame author of that tweet, we all make mistakes and I think DO should give access to backups even when account is locked)
I think it is reasonable to expect your hosting provider won't randomly shut down your account...twice.
BTW, I don't even see a downvote button around posts, there is only upvote button! Is it because I'm new here?
Keep in mind though that you should only downvote for abusive comments (which also deserve a flag), gross misinformation, or other things like that. Disagreeing with someone is not a good reason to downvote.
And unfortunately this stuff happens to your "established" examples as well. Here [1] is a particular example of Google shutting down an entire GCP account with no explanation. Some comments report the same on AWS as well. Ironically, people in that HN thread are actually suggesting Vultr (kinda like DO, but even smaller) as a good alternative.
1: https://news.ycombinator.com/item?id=17431609
I'm lucky enough that my spend with DO is high enough to qualify for support, so if this ever happened to me at least I know I'd get a couple chances to make things right
Were this to happen on GCP I'm fairly certain they'd just black hole my account since I'm spare change to them.
After even a single incident like this no sane company would relay on DigitalOcean. This is the kind of crap you expect from a shared web host overselling resources, not from a company that wants to provide infrastructure to tech companies.
I don't think you realize how big is the price difference between AWS/GCP/Azure and old-school VPS providers like DO, Hetzner or OVH.
It's a shame to see things play out this way, but sometimes a lesson is taught in a brutal fashion.
I imagine the author will be more careful in the future regarding off-site backups, additional technical partners, contingency plans, etc.
I'm not talking 5 9's redundancy. I'm talking grab a backup once a week or something, anything, to help mitigate a scenario like this. According to the thread, they lost ~1 year of data. That should be unfeasible to a company serving customers, let alone Fortune 500 customers.
Disaster recovery planning is key for a technical company to succeed. It is clear they never considered a scenario where there DO account would be closed/compromised/down.
I don't think the chance of DigitalOcean automatically freezing your account to a point where only a co-founder can do something about it has been well publicised.
A contingency plan should ideally have been in place for a scenario where, regardless of root cause, you have lost access to your DO account.
Given their size, that is extremely unlikely to happen without warning.
Imagine you are a customer of this company. Would you be rallying to their defense, "backups aren't needed because the scenarios are unlikely", or would you be angry that the company had zero contingency planning and lost all of your data (or the data you rely upon)?
If you can honestly say, as a (hypothetical) customer of the company in the thread, that you wouldn't care if a company you relied upon has no disaster recovery planning, more power to you. I, however, like to make sure that the companies I'm relying on have some sort of contingency that protects me as a customer.
An expensive lesson to learn to take and check backups regularly.
Stuff like this is relatively simple when you are trying to learn it, but keeping it operation is hard at a small level. Is the less to really use PaaS until it's viable for you to be running a small K8s cluster, or equivalent fleet? Seems really expensive compared to VPSes, but having better guides might help.
It won't be entirely up to date when the worst case happens, you'll be unavailable and you'll probably have lost a day of data or so, but you won't have lost everything.
Rclone to AWS or Google or whatever is easy to set up, add a daily dump of your database to the folder you back up. Unless you handle a lot of data, costs are probably not a big factor.
Clearly they should be charging more.
Even the database replication is mostly setup and forget these days. Very reliable.
And they already did backups. It's just they did them to the wrong (same service) host.
Also, even the co-founder doesn't seem to know exactly why the service was suspended, even though he clearly managed to arrange things.
Caps or quotas are not sufficient to deal with this problem.
A fly by night will often only suspend the VM or database that is in question, not other services on the account (having been in that position before).
Hardly fly-by-night.
The one time I used their phone support, the guy I got was fairly helpful. They seem like a hands off company overall.
Good practice, actually, but it's something people usually get frustrated with and they're literally famous for doing it.
Edit: I stand corrected ¯\_(ツ)_/¯
And they also have a lot of experience with it because they host a lot of game servers which are very prone to DDoS attacks (see e.g. https://securityaffairs.co/wordpress/51640/cyber-crime/tbps-...), so they are definitely “structured to offer it”.
On OVH one of my servers was hacked and running typical scripts that are run once that happens (port checking, common admin credentials, brute force attempts, etc)
They cut off all internet access to and from the server and sent me an alert stating what was happening and that I needed to VNC into the server, resolve the issue, and let them know how/why it happened and how I resolved the issue. Once that was done they just removed all the blocks on the server and we all went on our merry way.
Edit: To clarify the VNC console is on their site, not a remote connection.
I didn't know what they were talking about so I replied saying that, they helped me shut down the local port connections and I never heard another complaint from them. There was no downtime or banning at any point.
I saw a great deal on LEB for a KVM VPS from Alpharacks and signed up for a 2 year plan (my first mistake).
When SSHing in to the VPS it didn't have the advertised specs, and when I raised the issue with their support, they eventually fixed it..
Then I realized the second problem, they gave me the same IP address as someone else. You could still use the web VNC console, and as soon as you made an outbound network connection, inbound connections would work... for a few seconds... then SSH would drop. Reconnecting by SSH says "host key changed" i.e. you hit someone else's server sharing the same IP address. Using the web VNC console, works again for a few seconds, drops again.
It took about 7 days of arguing with their support to explain these two problems to them... by which time, the 3-day refund window had expired...
I admit some schadenfreude watching their recent disaster (all servers down since the last 2 weeks - https://www.lowendtalk.com/discussion/157613/popcorn-time-du... ).
Lesson learned about low-end boxes. I have had about 50/50 good and bad experiences, using half a dozen providers like this, not really making financial sense overall.
I recommend everyone stay far far away from AlphaRacks. If anything remains of them after this week.
I don't ever back up to the same service that I use for production. Or is that just me being paranoid?
Clearly not.
Yes, ideally they would have already tested their back-up solution, the back-ups would be offsite and, if something like this happened, they could stand-up on another provider. But that just ignores the reality of them being a super tiny business. Almost no one at that size is going to do that.
But I still support that DO here is a 100% liable toward their client. Now the liability between the said client and his own clients is an other matter.
Ideally partners should be trusted (and trustful), in practice, they aren't
Though trusting DO/AWS/GCP, etc is much more reliable, than, let's say, betting your whole business on somebody's proprietary API (like an FB game, an Linkedin API something, etc)
> Digital Ocean "Trust and Safety"
Does this phrase give anyone else the heebie-jeebies? They deliberately locked his account without looking at previous metrics over the previous months to conclude this was obviously not a concern, and why?
Things that have crossed my mind:
Is their automatic system was poorly tuned? Was this deliberately initiated? If so, why?
What happened w/ Digital Ocean is inexcusable, and has potentially dire consequences for two individuals livlihood. In the immediate aftermath of such an event, focusing on the devs percieved lack of disaster preparedness seems petty.
- No offsite backups? Agreed. Even for a two person team it is sloppy.
- "Relies on one tech partner?" Strongly disagree. Even large enterprises often have a hard dependency on AWS, Azure, Rackspace, or similar. To suggest that a two person team should have deployment plans for multiple independent cloud vendors is just fantastical thinking with no basis in reality.
Plus if they did over-engineer it by making it cloud agnostic, setting up accounts to sit dormant, cold instances elsewhere, etc, people would just criticize them for that inefficiency/wasting time.
How you got from "single tech partner" to "have no disaster recovery plan" I don't know.
Very, very few tech companies could simply move everything to a new cloud provider in a few hours. I would even hazard a guess that almost none can.
I have all my infrastructure as code and can break it all down and spin it back up in kubernetes clusters in minutes. But due to the quirks of each cloud provider, there are tons of little fixes that would inevitably need to be made.
Not to mention that many companies have way more data than could even be copied over in a few hours.
I think we needlessly shame one-person operations for focusing on actual customer needs instead of ops busywork and yak shaving.
No one that I've seen so far is saying they should have a system that is "muilti-site, fault-tolerant, self-healing, webscale that Google would be proud to have".
They are saying run a simple backup and keep it literally anywhere else.
I think we needlessly hyperbolize "do a backup once a month and keep it somewhere else" into some sort of NSA operation.
Its backups. Its 2019. Its dead-easy and very affordable.
Two years later I wanted to restore the VPS but turns out my snapshot become "outdated" and they stopped supporting the format for restoration... Support was completely useless, wouldn't even let me download the snapshot, said at most they could mount it into a new VPS and I could recover the data myself.
Very unprofessional.
These sorts of things tend to slip through the cracks. If you are running is a business capacity, make sure you treat everything like any other business would.
Even as a two-man company, you can't afford the cost and potential liability of not operating as a registered business. It also shows that this is a serious business and not a hobby.
Is there any evidence for this? For all we know his DO account was set up as a business.
That said, any company, especially one working with Fortune 500's, should have DB backups in at least two places. If they'd had the data, they could have spun up their service on a different hosting provider relatively easily.
I find it concerning that they have such a low threshold for triggering a lock. 10 droplets is hardly cloud scale.
-When the majority of abuse support dealt with was people angrily calling and asking about the fraudulent charges on their cards for dozens of Lie-nodes you consider putting caps in place to reduce support burden and reduce chargebacks.
At the time at Linode, if it was a known customer, we could easily and quickly raise that limit and life is good.
I've always wondered how Amazon dealt with fraud/abuse at their scale.
I don't think DO was wrong here to have a lock, but the post lock procedure seemed to be the problem.
Even worse: they don't explicitly state what is considered "unreasonable". So, if your business is serious, you have to assume the worst-case scenario: DO can't be used for anything
Conclusion: Digital Ocean is just for testing, playing around, not suitable for production.
I think that's always been the standard position most people take. DO, Linode, etc are for personal side projects, hosting community websites, forums etc. They are not for running a real business on. Some people do, sure but if hosting cost is really that big a portion of your total budget you probably don't have a real business model yet anyway.
That text suggests larger organizational problems within the company.
Yes they should.
How many 2-man shops do you think follow all the proper backup and security procedures?
Every week there's another article on HN about a tiny business being squished in the gears of a giant, automated platform. In some cases like app stores this is unavoidable, but there are plenty of hosting providers to choose from. People need to learn that this is something that can happen to you in today's world, and take reasonable steps to prepare for it.
You can't dot every I, cross every t and also build a compelling product as a 2 person shop.
How many horror stories need to reach the front page of HN before people stop believing this? Getting locked out of your cloud provider is a very common failure mode, with catastrophic effects if you haven't planned for it. To my mind, it should be the first scenario in your disaster recovery plan.
Dumping everything to B2 is trivially easy, trivially cheap and gives you substantial protection against total data loss. It also gives you a workable plan for scenarios that might cause a major outage like "we got cut off because of a billing snafu" or "the CTO lost his YubiKey".
Sounds like the opposite of the survivor bias. I don't believe it's any sort of common (though it does happen), even less that "it should be the first scenario in your disaster recovery plan"
I could not disagree more. There's a right way and a wrong way to do this, it's trivial to do it right, and the risks of doing it wrong are enormous.
Then it's unrealistic to trust them with your business.
We don't know the structure of their DB and whether failover is important or not, so we don't know if the DB can be reliably pulled as a flat file backup and still have consistent data.
We also don't know how big the dataset is or how often it changes. Sometimes "backup over your home cable connection" just isn't practical.
Cron jobs can (and do) silently fail in all kinds of annoying and idiotic ways.
And as most of us are all too painfully aware, sometimes you make less-than-ideal decisions when faced with a long pipeline of customer bug reports and feature requests, vs. addressing the potential situation that could sink you but has like a 1 in 10,000 chance of happening any given day.
But yes, granted that as a quick stop-gap solution it's better than nothing.
I'm going to take a stab at small and infrequently.
Every 2-3 months we had to execute a python script that takes 1s on all our data (500k rows), to make it faster we execute it in parallel on multiple droplets ~10 that we set up only for this pipeline and shut down once it’s done.
Even if they lost some data, even if the backup silently failed and hadn't been running for two months, it's the difference between a large inconvenience and literally your whole business disappearing.
They had backups, but being arbitrarily cut-off from their hosting provider wasn't part of their threat model.
Isn't a big part of cloud marketing the idea that they're so good at redundancy, etc. that you don't need to attempt that stuff on your own? The idea that you have to spread your infrastructure across multiple cloud hosting providers, while smart, removes a lot of the appeal of using them at all. In any case, it's also probably too much infrastructure cost for a 2-man company.
keeping your production and your backups in the same cloud provider is the equivalent of keeping your backup tapes right next to the computer they're backing up. you're exposing them both to strongly correlated risks. you've just changed those risks from "fire, water, theft" to "provider, incompetence, security breach"
...
> > "fire, water, theft"
i'm sure you could add a few more things to the list.
> I don’t think it’s terribly common for even medium sized companies to have a multi tier1 cloud backup strategy.
not terribly common to understand risk.
> So what is the purpose of the massive level of redundancy that you are already paying for when you store a file on S3?
You're paying to try and ensure you don't need to restore from backups. Our data lives in an RDS cluster (where we pay for read replicas to try and make sure we don't need to restore from backups) and in S3 (where we pay for durable storage to try and make sure we don't need to restore from backups), but none of that is a backup!
If you're not on the AWS cloud S3 is a decent place to store your backups of course, but storing your backups on S3 when you're already on AWS is, at best, negligent, while treating the durability of S3 as a form of backups is simply absurd.
> I don’t think it’s terribly common for even medium sized companies to have a multi tier1 cloud backup strategy.
The company I work for is on the AWS cloud, so we store our backups on B2 instead. It's no more work than storing them on S3, and it means we still have our data in the event that we, for whatever reason, lose access to the data we have in S3. Who the hell doesn't have offsite backups?
This is not remotely the same thing. A RAID offers no protection against logical corruption from an erroneous script or even something as simple as running a truncate on the wrong table. Having a backup of your database in a different storage medium on the same cloud provider protects from vastly more failure modes.
> Who the hell doesn't have offsite backups?
No one. But S3 is already storing your data in three different data centers even if you have a single bucket in one region, and you also have SQL log replication to another region. Multi-region is as easy as enabling replication but that is only available within a single cloud provider (I can't replicate RDS to Google Cloud SQL, only to another RDS region). I would guess that a lot of people use that rather than using a different cloud provider.
That sounds like...the same argument?
A RAID array stores your data on multiple physical drives in the machine, but offers no protection against logical corruption (where you store the same bad data on every drive), destruction of the machine, or loss of access to the machine.
S3 stores your data in multiple physical data centres in the region, but offers no protection against logical corruption, downtime of the entire region, or loss of access to the cloud.
You can't count replicas as providing durability against any threat that will apply equally to all the replicas.
Let's be fair: The threat model here is "lose access to our data".
This can happen in a number of ways, lost (or worse, leaked) password to the cloud provider, provider goes bankrupt, developer gets hacked, and a thousand other things.
Even if you trust your provider to have good uptime, there's really no excuse for not having any backups. Especially not if you're doing business with Fortune 500's.
Literally just push a postgres dump to S3 (or any other storage provider) once a night as a "just in case something stupid happens with my primary cloud provider". It'd take a couple hours tops to set up and cost next to nothing.
Also, by "two places" I meant the live DB and one backup that's somewhere completely different. My wording may have been confusing.
Containers solve the easy problem, which is how to make sure the dev environment matches the production environment. That is it.
Replicating TBs worth of data and making sure the replica is relatively up to date is the hard part. So is fail over and fail back. Basically everything but running the code/service/app, which is the part containers solve.
> Sure, a backup would have been a significant improvement, but still – a backup only protects against data loss and not against downtime.
Assuming you have data backup / recovery good to go, the downtime issue needs to be solved by getting your actual web application / logic up and running again. With something like docker-compose, you can do this on practically any provider with a couple of commands. Frontend, backend, load-balancer -- you name it, all in one command.
> Containers solve the easy problem, which is how to make sure the dev environment matches the production environment. That is it.
Speaking of "patently false"...
> Account should be re-activated - need to look deeper into the way this was handled. It shouldn't have taken this long to get the account back up, and also for it not be flagged a second time.
So... he doesn't address what is the scariest part to me, the message that just says "Nope, we've decided never to give your account back, it's gone, the end."
I think it's entirely reasonable for companies to have that option. "You are doing something malicious and against the rules, you have been permanently removed". In this case, that option was misused, but I don't think the existence of that possiblity is inheritly surprising.
And, regardless of what DO should or should not do, they can do whatever they want with their own hard drives. You should structure your business accordingly.
And do what in the mean time? The legal system acts slowly. In the age of social media outrage, would you allow the headline "Digital Ocean knew they were serving criminals, and they didn't stop them" if you were CEO?
It's easy to be outraged when these systems and procedures are used against the innocent. That does not mean we should stop using rational thought. If someone is using DO to cause harm, then DO should (be allowed to) stop the harmful actions.
You lock down the image, and let law enforcement do their thing. If law enforcement clear them, you then give the customer access to their data, perhaps for a short time before you cut them off as they seem to be a risky customer to have.
You don't unilaterally make the decision, you offload your responsibility onto the legal process.
The fact that there are hundreds of comments on HN condemning them for this action proves my point.
Seems to work just fine for AWS, Google and Cloudflare. In fact, counter to your argument, Cloudflare got in massive shit when they did decide to play God.
The observant will note the particular corner you're backing into here -- that a business might be justified in denying access to code/data being used in literally criminal behavior -- is notably distinct from the general and likely much more common case.
> they can do whatever they want with their own hard drives.
Sure. But to the extent they take that approach, Digital Ocean or any other service is publicly declaring that however affordable they may be for prototyping, they're unsuitable for reliable applications.
Businesses that can be relied on generally instead offer terms of service and processes that don't really allow them to act arbitrarily.
I agree. Look at the absolutism of the comment I am replying to. My whole point is that there might be some nuance to the situation.
> ...Digital Ocean or any other service is publicly declaring that however affordable they may be for prototyping, they're unsuitable for reliable applications.
Again, I agree. Considering how cheap AWS, backblaze, and Google drive is, it is completely ridiculous to depend on any one single hosting service to hold all your data forever and never err.
You seem to be accusing the aggrieved party of being a bad actor, when that is not the case.
https://news.ycombinator.com/item?id=20066331
If there is no police report, then they are trying to act as police themselves, which I think is unacceptable. It is not their data.
Your argument that they can do whatever they want with their hard drives is indeed something I will take care to remember — I definitely would not want to host anything with DO.
At the very least, they should also provide ALL, as in every last byte, of data, schemas, code, setup etc. to the defenestrated customers. As in: "sorry, we cannot restart your account, but you can download a full backup of your system as of it's last running configuration here: -location xyz-, and all previous backups are available here: -location pdq-".
Anything less is simply malicious destruction of a customer's property.
If you violate a lease and get evicted, they don't keep your furniture & equipment unless you abandon it.
They should have, at the very least, one DR site on a different provider in a different region that is replicated in real-time and ready to go live after an outage is confirmed by the IT Operations team (or automatically depending on what services are being run).