Totally. I'm fairly new to DO and after seeing what happened was re-thinking my decision.
But this is a solid followup, "we made a mistake" post so I think I can rest easy.
One wonders how many others didn't get enough Twitter cred, before. That some low-level ticket stamper (even a high-level ticket-stamper) had authority to deep-six a customer on no more say-so than high CPU usage tells us more about the company than an incident report massaged by marketing communication specialists. Simply, the latter sounds good because it has been made to sound good by sounds-good experts, and could say anything; but the event itself is ground truth.
They will need a lot more time and good behavior to live this down.
I agree on the twitter cred point. The fact that this happened in the end, personally I think it is a good thing as it highlighted a weakness we must fix.
We trust our people high-level, low-level whatever to make important decisions everyday. thats why they are here.
The "marketing communications specialists" are getting slammed a lot here, so I will just point out that they spend most of their time rolling their eyes at my crappy grammar, spelling and ludicrous number of comma splices. I don't think our goal was to sound like anything. We just wanted to lay out our investigation and the follow on work we are undertaking.
Totally agree with your point that trust is earned and we lost many peoples in the last few days. That will take time and as you say good behavior to earn back, but that is what we are committed to doing.
I talk about mktg comms because I have worked at places where angry customers got earnest letters promising changes, but the manager expected to implement the changes said "No, we're not doing that!" Or "OK" but nothing happened. So I don't give much credit for promises, even when it was the right thing to promise.
Giving your ticket punchers authority is good when they are authorized to do what customers need to get or keep going. Giving them authority to eliminate customers, not so much.
I have to agree with the commenters who say it was an exemplary postmortem.
Hospitals have been doing formal postmortems for many years, but the number of them didn't start down until they instituted checklists.
Depending on the severity and length, it could still have a long-term impact on that business. Also a bit unsettling that seemingly basic safety controls failed. But it is good to see DO being open and thorough about this incident.
In what way is DigitalOcean a "huge company"? At ~300 employees, it's closer to the SMB's definition of a small business (<250 employees) than mid-size (<500 employees).
In all fairness, having worked at a few tech startups, it can be hard to scale customer service to keep up with demand—you don't control how many support tickets come in, and it takes a lot more time to hire and train new customer service agents than it does to spin up new servers, and if you over-hire, it's a lot more costly than shutting down some servers.
DO is claimed to be "third-largest hosting company in the world in terms of web-facing computers", so that should give you an idea of how many customers they have.
I feel like a better measure for the "size" of a tech company is number of customers, rather than number of employees. Considering software doesn't need a linearly increasing number of people to produce/support it as your share of the market gets bigger.
Hey Nkozyra - Zach here from DO Support. One of our remediation efforts, that is already underway, is that Support and Security Operations leadership will create new workflows to allow abuse-related events to leverage the 24/7 structure of Support.
On Support, we have additional Support Engineers joining our Developer Experience team in mid-June and early July. We will continually assess our ability to provide high-quality responses as fast as possible to all tickets. Our customer feedback will continue to be the measure of how well we're doing, but our goal is that no one will ever need to use social as an escalation path.
I've been using DO spaces for about a year now, and for the later half of that time, my experience has been pretty terrible.
- Spaces throwing up errors that magically fix itself a couple of days later.
- Asking about the credits we were promised for when Spaces lost our files results in the question being ignored. Still haven't received the promised credits for 6+ months. I can't even look back at the tickets now as the support system has deleted all the tickets older than a month.
It's gotten to the point where we have started work on migrating off DO, which is unfortunate because DO's offerings looked very attractive.
Hi Sladey - Zach here from DO. I'd like to help you out and investigate what might be going on. Can you send me an email (first name at) and I'll investigate right away?
I had wondered if they were a large company or not. I have used them in the past and there was a specific reason that we picked them over AWS/Azure/Goober, but those details escape me.
Based on the other comments, they don't sound like a huge company. Someone mentioned about 300 employees. I am not sure their revenue and chuckled at the reference that they're "third largest" given some specific criteria (of course, "largest" doesn't mean anything, either -- what we're really looking for is "third best" by some criteria that we've defined in our head)
That jives more with other things I've read and anecdotal experience.
I think the size of the company is less relevant than the last point that you make about the clear issues around training/support. At a company that size, CSR training might be less formal than it needs to be. When a one-off like a customer who is legitimate but in every other way appears to be fraud might involve a slack messages to people with wrong information rather than clear guidance followed up with formal training.
It's difficult for a smaller (average cash-flow for their size) company to succeed in highly competitive markets that are, effectively, commodities. A larger company can handle being a loss leader to knock smaller competitors out of the market, providing excellent customer service. They don't. But it'd be easier for them to do so. :)
Reading this response it seems that crypto-mining is not allowed on digital ocean as they have checks against it. The TOS doesn't say so explicitly but does note that:
>violation of any of these Terms of Service or any law, or if you misuse system resources, such as, by employing programs that consume excessive network capacity, CPU cycles, or disk IO
By my reading that seems to mean that you're not allowed to use your VMs to their full capacity due to them being over-provisioned. This is in contrast to AWS who are more explicit on which instances (T instances) are over-provisioned and exactly how they're throttled.
I don't think that it isn't allowed but that they have seen that fraudulently acquired resources are generally used for crypto-mining so they felt that this was a good signal to look for.
Right. Reading between the lines a bit, it's not the activity itself that DO is worried about, but a pattern of usage that suggests that the account may have been created fraudulently or compromised.
This is correct. This was the primary thing we were attempting to solve for in this case and the bug in the algorithm started the chain of events documented in the postmortem.
I recently logged into my long-dormant DO account to kick the tires on their now-GA managed Kubernetes service and to contribute some CPU cycles to a distributed computing project I'm interested in (not cryptocurrency, I swear).
I first requested and was approved for a droplet limit increase (to 25). I started 20 nodes, deployed my very CPU intensive workload, and 12 hours later my account was locked and my nodes all went NotReady.
I immediately replied to the abuse ticket explaining my usage. 4 days later, my account was unlocked and received the "allow high CPU" flag... but they billed me for the nodes as if they had been running that whole time. I asked for a credit (5 days ago) and they haven't replied yet. Probably a little busy over there right now.
So... I'm not too thrilled with DO. I get that cryptocurrency ruined everything, but this has been a frustrating experience and I'm glad I was only using it for a pet project.
Hey xnxn - Thanks for raising this issue. I'm Zach, Director of Support at DO. Please send me an email with your account detalis, first name at, and I'll take care of this.
That's my point. Their support got so bad that they had to be called out on in publicly before they did anything. That's hardly what you want to hear from a company like DO.
> That's my point. Their support got so bad that they had to be called out on in publicly before they did anything. That's hardly what you want to hear from a company like DO.
That's what you get with most providers these days. Ever tried reaching Google or Amazon?
I don't believe that Google or Amazon have ever done anything like what DO did before. However, if you notice their support guy Zach was responding to a completely different incident that is not related to the one being addressed with their post-mortem.
Actually I have complained to Amazon... and they have always responded within a couple days and in one case where I complained about Prime Video, they called me on a Sunday, which shocked me because everything is otherwise closed in Austria, except gas stations, restaurants and hospitals.
Google on the other hand, I make complaints or suggestions once every couple months and 99% of the time I don't even get a boilerplate response.
If you pay for enterprise support with Amazon you can open a case with a less than 15 minute SLA [1]. With business that goes down to an hour SLA. With both plans you can create chats or calls that typically get answered within a matter of minute with and assigned to an engineer with a background in the service.
Hey Sneak - I totally agree* :) We've already started efforts between Support and Security leadership to leverage the 24/7 structure of Support. Our goal is that no one will ever need to use social as an escalation path, and our new Support Engineers who are joining in mid-June and early July will be part of this a reality for our customers.
Fyi, I'm planning to migrate off of DO due to the crummy payment options. With every other recurring service I can have a PayPal subscription, DO doesn't support PayPal properly though.
Also, your billing system is incredibly spammy (1 to 2 emails a day at times). The bills I get are always for partial months despite the VMs on the account running for a full month.
Hey metildaa - Zach here from DO support. It is accurate that that do not currently support PayPal subscriptions, this certainly isn't the overall experience that you should be having. Can you shoot me an email (first name at) and I'll look into this for you?
If you want to do cryptocurrency mining on DO that is actually okay with us. Some of the other respondents are correct the behavior we were looking for was really around fraudulent accounts being created and performing cryptocurrency mining. This is why the trigger that flagged this account was using payment history as a key factor in the triggering.
Your post-mortem implies this is not allowed at all.
Not sure why you were downvoted, I had the same impression, after reading:
...an automated service that monitors for cryptocurrency mining activity (Droplet CPU loads and Droplet create behaviors). These signals, coupled with a number of account-level signals (including payment history and current run rate compared to total payments) are used to determine if automated action is warranted to minimize the impact of potential fraudulent high-cpu-loads on other customers
This sounds like they don't permit extended high CPU loads due to the impact it can have on other customers.
No, that quite clearly states that they treat high CPU loads as suspect on accounts without an established good payment history or if it significantly deviates from previous usage patterns.
What would "fraud" mean in this context? Are they talking about customers who don't pay their bill to DO? (If so, seems like the account should just be temporarily suspended until the bill is paid.) Or are they talking about fraud to other parties, like phishing sites? (If so, I don't see the connection to crypto mining.)
My understanding is that they're trying to prevent users from creating new accounts, running 100%CPU until it's time to pay the bill and then just not paying, moving on to another new account.
edit: from elsewhere ITT it seems they're doing this with stolen credit cards.
Cryptojacking is a well-known, major problem for cloud compute providers. Catching and squashing new exploits that allow people to create a fresh account, run up compute bills and then abandon the account without paying is very important.
My guess would be that this is such a well-known problem (within the field of cloud compute at least) that they just didn't think they had to state that normal crypto mining by paying customers is completely fine.
Depending on the cryptocurrency's proof-of-work algorithm and new-ness, it can be profitable to mine in the cloud. I've done it briefly in the past. But generally it's not profitable.
In every cryptocurrency (the popular and functional ones anyway), there's a set global rate of mining rewards. All miners compete for a slice of that reward, so as more people mine, each individual miner gets less reward. (This causes an equilibrium to be reached where more people mine until it's no longer profitable for more people to start mining. If mining becomes unprofitable, some miners will drop out, and the remaining miners will each make a little more.) If masses of people realize that cloud mining for a particular cryptocurrency is profitable, then what generally happens is that lots of people pounce on cloud providers to mine, it becomes barely profitable, and then people operating their own hardware that's cheaper than cloud providers come in and push the mining rewards down to where it's no longer profitable for people to cloud mine.
Because cloud mining is never profitable in the long run, most cloud mining that happens is fraudulent activity using stolen cloud accounts or payment info. (If you're not paying for it, then making any amount of money from it is profitable.)
It depends on what crypto is going to be mined an how many accounts can be stolen given the fact that there is already plethora of bots that look all over GitHub for accidentally committed credentials. Heck, just a year ago people did scans for outdated WordPress installations to inject, among other things, some JavaScript (!) Monero miners [0]…
No. Cryptomining represents an arbitrage opportunity such that the spot price of the instances should be adjusted. In the long run it should not be profitable.
The thing that has me scratching my head is how this chain of events unfolded.
I get that your fraud algorithm flagged it because of lack of established payment. how is this possible if what the tweet referred to as "locking us out of all of our backups and work"? surely an account history of any significance would have an established payment record. From their tweets they mention that they had 5 droplets and some storage of a not insignificant number of records (~500k) and that a script is required to be run every 2-3 months to process some data and that script spins up 10 droplets during that time. seems like it will take 13 hours to process the data based on row count and per record time.[0] I am struggling to see how they didn't have payment history. can you elaborate?
In addition another thing I'd think would help assuage fears of a complete lockout is some process where you can request and download the db or a snapshot of the virtual machine.
I believe the purpose of these checks is for when people's outdated wordpress install or whatever gets compromised by script kiddies. Generally, the scripts install crypto miners to mine for the hacker until your account gets shut down, running you up a huge bill in the process.
"Cryptocurrency mining mitigation detects suspicious behavior, including very high CPU utilization on an account with no payment history, which results in an account lock"
Lots might have been done wrong here, but it sounds like they had an account with trial or promotional credits - I can see how this could easily be abused.
Sounds like it's designed to counter stolen credit cards.
An attacker might load a stolen credit card number into an account and only use enough resources to generate a few dollars worth of billing. The owner of the credit card might not notice the small charge.
Then after a few months of low billing (to bypass a previous heuristic), they ramp up the utilization, mine a bunch of coins then the holder gets a massive bill.
The holder does a charge back and DO is left holding the bill.
It's also designed to keep everyone relatively happy in a shared-CPU environment. "Standard" droplets share CPU with others on the same node, so one droplet pegging the CPU 24/7 can be problematic.
AWS doesn't have this problem because either your instance is allowed to use all the CPU that's allicated to it, or else (t2 & t3) the platform will automatically limit your CPU usage. You don't have to care about how your usage affects other people. It's one more thing that AWS abstracts away. DO's abstraction, on the other hand, is rather leaky in this area. That's a problem in and of itself, in addition to the matter of credit card fraud that every company has to deal with.
Completely. At the same time, those promotional credits are going to be used by guys like me who will have to decide if their services are worth having to spend an extended period of time explaining why "we're not recommending AWS/Azure/Goober"
An account shutdown, or enough complaints from verifiable sources, and I'm not going to the trouble. Not to pull out an old trope "Nobody got fired for picking IBM". But that's the case: pulls a move like this and the entirety of the customer, who likely came in with AWS in mind (in some cases, was advised against it and insisted on it) is going to shrug their shoulders. Pick a provider that the customer hasn't heard of and I'm going to get a phone call that goes something like "You're the one that said we should use that basement-operation!" with raised voices. Heck, the last time there was an Azure outage, we didn't hear from most of our customers. It was so impacting that even customers well outside of software development/technology read news articles and connected the dots. I had one customer tell me he thought it was just their corporate internet connection; they assumed it was working[1].
[0] Plus, too lazy to put in the research; sorry.
[1] They were a customer who insisted on doing the app monitoring, themselves -- that guy was getting the alerts and similarly assumed it was the network since that happened regularly with another application they developed -- the monitoring server was on-prem.
Cryptocurrency mining is a favourite of fraudsters. Get a fake or stolen credit card number / identity, sign up for an account on a cloud provider, and spin up instances to do your mining. Depending on how quickly the provider reacts to the pattern of behaviour, or identifies the account as fraudulent etc, you can have earned a reasonable pay-off.
If the provider isn't paying attention and/or doesn't have good fraud detection in place, it may be a few months before your account and resources are terminated (assuming the payments eventually bounce, and the provider gives you a chance to fix it)
Good response from DO, but this line jumps out at me.
> Responses to account locks were not prioritized differently from a ticket management standpoint to be above less severe tickets.
That's arguably the biggest failure, IMO. The fact that an action which locks/terminates an account is not prioritized any different than a general ticket is pretty jaw-dropping, and I'm glad they're going to change that.
Yeah... that one was painful and we are fixing it. At least if the priority placed this at the top of the queue we could have acted faster. Probably the same outcome due to the other issues involved in this incident though.
I appreciate all your transparency and engagement on this issue. It probably would have had the same outcome, yes, but potentially resolved much more quickly. Regardless, the fact that you're fixing it is music to my ears.
What jumped out to you as weaselly? It read more like a blameless postmortem to me than weaseling around (unlike any of the recent big PII breaches), and they acknowledged that this needs to be something they respond more quickly to directly.
Thanks for the pointer. I'm going to blame my dad/up bringing for my over use of passive voice. He will be deeply amused by this. I will read the paper and attempt to improve.
As happy as I am to see this post itself, the mistakes made here are pretty appalling.
Killing customer accounts by automated action without any human check just seems like a recipe for disaster. Even if you can respond faster to crypto issues, the effects of a false positive are just unacceptable.
Though apparently the human checks at Digital Ocean don’t work either.
According to the post, that's not what happened? The customer account wasn't terminated by the automated system, but rather by the second Abuse Ops agent.
Upon a second review by a different Abuse Operations agent [...] the agent fully denied access back into the account. This action triggered the final “access denied” communication to the customer.
> The initial account lock and resource power down resulted from an automated service that monitors for cryptocurrency mining activity (Droplet CPU loads and Droplet create behaviors).
You're looking at the one false positive instead the potentially thousands of true positives. Those true positives are people using free DO credits to cryptomine on DO. And probably 99.99% of the time the system works as intended. Dealing with bad actors and abusers is not pleasant or easy.
This sort of 0.01% case is exactly what developers and sysadmins deal with on a regular basis -- some crazy scenario that lead to a bug you've never experienced before. The correct and only response is to fix the bug (whether it be software or process), offer whatever compensation to the injured parties, and move on.
Barry Cooks did a phenomenal job with this after-action. He not only publicly accepted fault on DO's behalf (+1), not only stated the incident timeline clearly and without bias (+2), but also showed mitigation steps and procedural changes to avoid this in the future and prioritize customer business interests (+3). Many medium and larger sized companies should take note of this handling style (looking at you, Google and Facebook). I love that there was no generic PR "we're very sorry". Succinct, accurate, and without spin (+4).
I agree, the incident report was well done. The combination of factors that led to the issue was described in clear detail, and I was glad to see a concrete plan to improve various aspects to avoid future cases like this. It certainly helped to regain trust.
The startup claimed they had all their backups on digitalocean, which contained data of Fortune 500 companies.
A startup who has fortune 500 clients must have history. I don't get then why digitalocean says they do not have payment history. Either the startup moved a few weeks ago - but then why don't they have offsite backups if they just moved. Or because they're french they did have payment history but did not have an American credit card to similar.. not sure what's up
Maybe one of the problems is DO's views of credits. Maybe things would work out better if they would treat credits like real money instead of phony money and tighten up how the credits are handed out.
AWS is a 8,000 pound gorilla. This means throwing massive amounts of promotional credit at new accounts is often the only viable way to get them into the funnel.
Now that we're all not picking it, however, I think they should remove the "People" section. They did a good job of adjusting process instead of blaming people. The people section, however, might lean toward blaming people. They didn't, in this case, but it could.
I disagree. People are often (perhaps usually) a significant contributor to incidents and I like seeing that called out explicitly.
Internally, we want to know exactly who did what, when, and why they thought that was the right approach. We're pretty ruthless about getting those facts down; in exchange, we're beyond lenient on anything that resembles punishment of people (barring multiply repeated cases of extraordinarily poor judgment).
We don't publish post-mortems publicly (though we do internally); still, we generally elide names from the published docs (replacing with role names such as "Operator1", "SquadLead1", etc.), but internal to the teams, we really value understanding exactly what happened and don't shy away from understanding the specific people involved. It's not in any way a black mark on someone's record to have downed prod or made a problem worse. It happens; better we understand and accept that.
Hey there. Thanks for this feedback. I think it is important to be open honest but not blame-oriented in our review of the situation. People make mistakes and that is okay, so long as they aren't willful or due to incompetence. Neither of which was the case here. The key thing is not to create a situation where a mistake is an individuals fault. My general view is if people are making mistakes then we have done something wrong as a company and need to understand and fix the tools/training/process that led to the mistake.
Many (all?) of the cloud providers offer credits to startups (credits as in free $ to spend on their services). So if they hadn’t burned through that, there would be no payment yet. (The startup I work at got $20k in credits and didn’t pay a dime for the first year)
Did anyone notice DO leaked customer financials in this post? If I were a startup running on credit, I definitely wouldn't want to advertise that. WTF?
I was very worried about that specific detail and we reached out to the customer before posting this postmortem to expressly get his permission to share those details. If he had said no, we would have worked around the detail but not be able to explain as clearly what went wrong. He gave us his permission to share the information.
I don't understand what you think the issue here is?
My interpretation of this is that the customer had pre-paid credit on their account. Meaning they had not been through the typical bill cycle yet (hitting an external payment method).
How are you interpreting that they are running on credit? As in their account is in debt and they haven't paid yet?
Perception is everything. Many companies, especially Fortune 500, will do deep research before doing business with anyone. I've been through them where have been disqualified specifically due to our infancy and lack of proof of long standing. If someone read/mis-read someone's post that gave them the idea that the company didn't have enough runway, they might move to the next potential suitor.
Digital Ocean: allow me to do some extended verification so you know exactly who I am and reduce your risk. In exchange, there is no automated locking, rather we are contacted and have 24 hours to mitigate the issue.
Requirements:
* 1 year continuous ontime payments at $250+/mo usage
* automatic billing is set up
* billing limits are set up and have been reviewed within the last year
Various businesses in the usa require licensure to operate. These are often done at state or even federal levels. Not every business has one, but even an EIN provides them more information about who's paying for the service.
Not every business has an EIN. I'm kind of surprised at the number of regulatory hoops you're willing to jump through to negotiate a minimal level of service.
Looking at it from the point of operations of the affected company, this doesn't sound like minimum level of service. Instead, DO was the entire business structure of the company. As a company, if you're entire business plan depends on the services of a 3rd party, it is not unusual to go through extra steps to ensure 3rd party can't end your business. Running an internet driven company from a consumer service from ATT/Spectrum etc is suicidal. If your network goes down, they'll fix it when the get to it. With a business level account, they have much more guarantees to keep your signal hot. Running a POC off of a free tier of AWS/DO/etc is fine, but these guys were well past POC.
The first few items on your list are actually a part of what we meant by "having billing history with us". There are a number of things we look at in that bucket. We use these items as a part of validating users before taking any action (yes, we clearly failed on this account due to the credits which is a clear bug). As far as offering things like a copy of your business license or other means of verification that isn't a bad idea. As an example people paying with POs today are excluded from the algorithm already.
Please make it official, so that people can have peace of mind knowing that they've got that "verified" badge. People hate having to wonder whether they're at risk of crossing an invisible, inscrutable, and constantly changing threshold. See: PayPal and AdSense account forfeitures. You could do so much better than that.
Not sure why you’re being downvoted so hard - while I think your list of requirements perhaps aren’t the right ones, I like the idea in general. Have a process where I can make assurances of some form that we’re good guys, and treat me accordingly.
Nothing in the statement from Digital Ocean indicates that they won't kill your account or shutdown your systems - that's not the sort of cloud host any company can afford to use.
Cloud providers that kill accounts - or SAY they kill accounts, must be dropped and not used.
The worst thing that should be possible is for your account to be suspended.
AWS, if there is a billing issue, prevents you making changes to your infrastructure via the console until the billing issue is sorted out - this is good and reasonable.
""Peer review of account terminations. For any account appealing a lock, two agents will be required to review the submission prior to issuing a final deny.""
- I can imagine how this plays out:
(service agent 1 turns to next service agent along) 'This looks like a bad account - I think I should shut it down, what do you think buddy?'
(service agent 2) - 'Yep I trust you, shut it down.'
The article discusses violating TOS, not a billing issue. I don't think it's unreasonable to disable an account in that circumstance but I agree that deleting images/resources without allowing customers to defend themselves and backup the systems would be unfair.
> The article discusses violating TOS, not a billing issue.
That's not the impression I got. It sounds like the issue was that a account with misinterpreted payment history was showing bitcoin-mining-like usage patterns. Mining is not against the terms of use, they were just erroneously convinced themselves that the customer was not going to pay for it.
> Nothing in the statement from Digital Ocean indicates that they won't kill your account or shutdown your systems - that's not the sort of cloud host any company can afford to use.
There is no cloud that will issue such a promise. If your criterion is that a cloud has to promise that you won't get shut down, you just can't use cloud hosting providers.
I had been expecting a short blip of an update denying anything of consequence (a twitter post promised a status update, but well, you know...) but this transparency significantly exceeds expectations. Nicely done DO.
You may want to explain service credits in some light detail though, for those that are unfamiliar with them.
Key takeaway for me: DigitalOcean might still kill your account at any time, revoking your access to your data.
Refusing service is fine, but holding my data hostage and refusing access to it is not, so I am making a note to not consider DO for any kind of hosting.
Because I reinstalled my droplet with a different filesystem manually, the snapshot restore doesnt work. Support tells me they cant do anything, so my 2~ of chat logs are sitting in a disk image that they cant restore bc they need to mount it for some reason...
Hey there! I would love to follow up on the issue you're describing here. It looks like you tried to bring a disk image over from a provider in a format that we don't support, and unfortunately there's nothing trivial we could do to get a working Droplet out of it (which is a requirement for us to expose the volume within the systems we have).
I can't promise a super fast resolution - but I'd be happy to work internally to see if there's any outside-the-ordinary workarounds we can supply here if you're willing to follow back up on the ticket.
I am curious which alternative service providers you use that offer guarantees to never kill your account, and to provide copies of data stored on their infrastructure.
(As noted by many observers during the initial event, anyone keeping their data exclusively within one organisation's walls is a profound mistake.)
I don't know if it's a promise, but I know from experience that AWS won't kill your account when they automatically detect fraud-like increased usage. They send warnings that the account will be shut down in several days.
Because shutting down running servers without warning is completely unacceptable for a B2B infrastructure provider.
As IT professionals we should do better than use words like 'kill' to describe system and account changes.
As I understand it, Digital Ocean suspended the account, and because the (perceived) problem was related to excessive / potentially fraudulent CPU usage, they suspended the machines. The data contained on them was not deleted. Does that match your reading?
As someone who suffered from extremely noisy neighbours in AWS (in the very early days), risking significant damage to the performance guarantees to our customers, I'm actually cautiously happy with automated protection systems. Naturally I'd rather noisy neighbours were throttled so that I never heard them, and I expect that's closer to what happens these days
I think 'kill' is appropriate here. They shut down access to his company's account along with all running infrastructure and after escalating multiple times, their final official response (after 30hrs of silence) was that his company was permanently denied access to the account (DO calls it 'account termination' in their postmortem). The first tweet of the thread that went viral was 'How Digital Ocean just killed our company' [https://twitter.com/w3Nicolas/status/1134529316904153089].
I don't want to spent too much time dumping on them since they clearly know how badly this situation was handled, but this is an example of terrible automated protection leading to a company that's not enterprise-ready. As you say, AWS probably doesn't publicly promise not to terminate your account, but this would never happen because they understand that availability and security matter more than anything else when running B2B infrastructure.
> Shortly thereafter, DigitalOcean investigated the issue and the Raisup account was unlocked and powered back on.
But it's not clear if any data was deleted by DigitalOcean.
The suggestion the account was unlocked rather than re-created suggests it was not, but OTOH there's no reference to erase, delete, restore, or indeed current state of customer data in that post mortem.
The fact that data didn't get deleted is irrelevant if the customer can't actually get at that data.
That the customer got unlocked is of course a good thing, but for at least 30 hours the customer couldn't access their data, that's highly problematic.
Nothing was deleted or removed. The droplets were powered off and the access to them locked. once (way too long later) the unlock happened the customer had full control and access again.
We dropped DO from our company usage after similar issues, though honestly DO probably wasn't the right place for our product at that stage of development. What was meant as a POC became technical debt and an outage forced is to come to terms with the fact that by the time the issue happened we had more than enough of our own capacity to run on our own metal.
Kudos to DO for the open incident management. As someone who does this myself, these are often really painful and hard to get right.
I have no relation to DO, but I'm surprised by the negative responses in this thread. I can't think of any other major company conducting a public postmortem for a customer service failure (as opposed to networking/ops failure). Not only are they changing their policies across the board, taking on more risk to improve customer experience, but they are hiring extra people so it does not happen again. Kudos for that!
And of course DO will still retain the ability to suspend your account for suspected fraud - that is the case with any cloud services company, and any online business in general (check your ToS). Again, I can't think of any business that will en masse promise to never react to any fraudulent users. It's how this process is performed that matters and that's what they are improving.
I haven't been around the block enough to know, but I'll take your word for it.
What I don't understand is why anyone would complain about a post-mortem as PR. If that's what DO was up to here, I'm sold—it shows transparency, thoughtful problem analysis, and swift execution.
More than what this means to me as a customer, I could surely stand to learn a lot in my own work from the way they approached this situation.
A PR move isn't worth anything unless they actually follow through. I suppose many people are skeptical that things will actually improve as PR is often all talk and no actual action. Not saying that's the case here, but it certainly is difficult to trust a company purely by what they say for PR these days.
> I can't think of any other major company conducting a public postmortem for a customer service failure (as opposed to networking/ops failure).
There are many companies that have done this in the past. They are not doing this out of the goodness of their hearts, this is lip service for the fact that their mishap blew up in their faces on twitter. Do you really think they would have gone at length to highlight to the public this incident had it not gone viral?
> Not only are they changing their policies across the board, taking on more risk to improve customer experience, but they are hiring extra people so it does not happen again. Kudos for that!
There is no telling that they are actually going to follow through with anything. Mere lip service.
The bottom line is, people host their businesses and livelihoods on cloud providers and they (the cloud providers) should take the necessary care and precautions when taking destructive actions. Maybe err on the side of the customer instead of shutting down someone's entire business because of some automated heuristic. Maybe have a better response time than 29 hours. Maybe teach basic communication and develop processes so that care agents can see and react appropriately to recent activity on the account. These are not revolutionary concepts, they are simple things that demonstrate customer care, something DigitalOcean is sorely lacking.
> No data was lost, it is not destructive in anyway.
Except in the way that the guy may[1] have lost customers or revenue due to the downtime. Being offline, even without data loss, is very destructive for many businesses.
There's no amount of money you can pay them to get decent support, that I'm aware; at least, if you could, you can't just pay them upfront. They screwed up their weird bespoke network configuration system (which they inexplicably use in place of DHCP) on FreeBSD, and their only support output is: lol blow down your instance and start a fresh one.
You tell me why a company like theirs, which should be mature, ditched their proprietary support and ticketing system (which actually worked) for an shoddy, misconfigured off-the-shelf product which has equally little excuse for being that bad.
I think it's customary here to be unfair to companies, even when they're doing "the right thing" as DO is right now; but DO has spent their customer patience budget elsewhere, so I'm not going to be surprised if people view the post mortem as an inadequate replacement for getting it right the first time, rather than a followup to an honest mistake they will actually try hard to avoid in the future.
> There's no amount of money you can pay them to get decent support, that I'm aware;
We at NodeBB have a high enough spend to have access to level 2 agents (their responses are around 1 hour turnaround, if not sooner).
We don't actually spend that much either, compared to what some of our clients pay AWS, etc...
Now, we certainly don't qualify for their highest tier with the dedicated support manager and slack access, but that's ok. DO's been amazing to us as a host.
Their pricing page reads: "world-class technical support to all of our customers—around the clock" (https://www.digitalocean.com/pricing/#Included_services ) b.t.w. which doesn't seem totally accurate to me, with the 12h and 29h response times in this account banned case.
Yeah, I'd just like to be able to pay for the support, without having so much volume. I can't really consider sending them enough business that we'd qualify for level 2 agents if I run into regular technical issues with the services before I even get that chance.
Is it bad for image if they just have a "send a couple hundred bucks because you need to speak with someone real bad" button?
The tweet thread suggested using for years. So either they are giving credits long time for startups they don't even know how and what they are doing or messed up a simple logic about history of the customer.
Similarly, as I have skin in the game and didn't want to get blocked by DO, I was expecting a muddy response and ready to make a throwaway account and complain about "keeping processes opaque to give carte blanche to take whatever arbitrary action they like" in the normal complaint about deplatforming and tech censorship, but then I read their document and was surprised and encouraged by how transparent DO were. Congratulations DO!
> I'm surprised by the negative responses in this thread.
That is actually a general pattern. Negative, dismissive responses come fast because they're reflexive. They don't require processing significant information, nor reflection, nor thoughtful writing—all of which take time. Therefore they are the first to appear.
Yep. I haven't had the best experience with DO previously (not terrible, just not great), and was vocal in the original thread about this, and I think this is one of the best reactions to a customer service failure that I've seen from a tech company.
This wasn't a failure that impacted thousands of customers [#]. DO could've just fixed things up for this customer and said not a word more, and changed nothing, and everyone would've forgotten about it by about ... now.
Instead they dedicated a nontrivial amount of resources to understanding what went wrong -- identifying not just a single cause, but several -- and publicly explained what happened, without a lot of weasel words, and what they're going to do to fix it.
It's an awesome response.
[#]: ...at that specific time. Yes, others have probably previously been impacted.
281 comments
[ 5.8 ms ] story [ 305 ms ] threadOne wonders how many others didn't get enough Twitter cred, before. That some low-level ticket stamper (even a high-level ticket-stamper) had authority to deep-six a customer on no more say-so than high CPU usage tells us more about the company than an incident report massaged by marketing communication specialists. Simply, the latter sounds good because it has been made to sound good by sounds-good experts, and could say anything; but the event itself is ground truth.
They will need a lot more time and good behavior to live this down.
We trust our people high-level, low-level whatever to make important decisions everyday. thats why they are here.
The "marketing communications specialists" are getting slammed a lot here, so I will just point out that they spend most of their time rolling their eyes at my crappy grammar, spelling and ludicrous number of comma splices. I don't think our goal was to sound like anything. We just wanted to lay out our investigation and the follow on work we are undertaking.
Totally agree with your point that trust is earned and we lost many peoples in the last few days. That will take time and as you say good behavior to earn back, but that is what we are committed to doing.
Giving your ticket punchers authority is good when they are authorized to do what customers need to get or keep going. Giving them authority to eliminate customers, not so much.
I have to agree with the commenters who say it was an exemplary postmortem.
Hospitals have been doing formal postmortems for many years, but the number of them didn't start down until they instituted checklists.
It's sad to me that your only chance in hell of getting huge companies to listen to you is by shamespamming across social media.
That, coupled with the clear issues following procedure from support, paint a clear picture: customer service is an area to skimp on for big tech.
In all fairness, having worked at a few tech startups, it can be hard to scale customer service to keep up with demand—you don't control how many support tickets come in, and it takes a lot more time to hire and train new customer service agents than it does to spin up new servers, and if you over-hire, it's a lot more costly than shutting down some servers.
On Support, we have additional Support Engineers joining our Developer Experience team in mid-June and early July. We will continually assess our ability to provide high-quality responses as fast as possible to all tickets. Our customer feedback will continue to be the measure of how well we're doing, but our goal is that no one will ever need to use social as an escalation path.
- Spaces throwing up errors that magically fix itself a couple of days later.
- Asking about the credits we were promised for when Spaces lost our files results in the question being ignored. Still haven't received the promised credits for 6+ months. I can't even look back at the tickets now as the support system has deleted all the tickets older than a month.
It's gotten to the point where we have started work on migrating off DO, which is unfortunate because DO's offerings looked very attractive.
Thank you, Zach
Based on the other comments, they don't sound like a huge company. Someone mentioned about 300 employees. I am not sure their revenue and chuckled at the reference that they're "third largest" given some specific criteria (of course, "largest" doesn't mean anything, either -- what we're really looking for is "third best" by some criteria that we've defined in our head)
I found this: https://www.canalys.com/newsroom/cloud-market-share-q4-2018-...
That jives more with other things I've read and anecdotal experience.
I think the size of the company is less relevant than the last point that you make about the clear issues around training/support. At a company that size, CSR training might be less formal than it needs to be. When a one-off like a customer who is legitimate but in every other way appears to be fraud might involve a slack messages to people with wrong information rather than clear guidance followed up with formal training.
It's difficult for a smaller (average cash-flow for their size) company to succeed in highly competitive markets that are, effectively, commodities. A larger company can handle being a loss leader to knock smaller competitors out of the market, providing excellent customer service. They don't. But it'd be easier for them to do so. :)
>violation of any of these Terms of Service or any law, or if you misuse system resources, such as, by employing programs that consume excessive network capacity, CPU cycles, or disk IO
By my reading that seems to mean that you're not allowed to use your VMs to their full capacity due to them being over-provisioned. This is in contrast to AWS who are more explicit on which instances (T instances) are over-provisioned and exactly how they're throttled.
I recently logged into my long-dormant DO account to kick the tires on their now-GA managed Kubernetes service and to contribute some CPU cycles to a distributed computing project I'm interested in (not cryptocurrency, I swear).
I first requested and was approved for a droplet limit increase (to 25). I started 20 nodes, deployed my very CPU intensive workload, and 12 hours later my account was locked and my nodes all went NotReady.
I immediately replied to the abuse ticket explaining my usage. 4 days later, my account was unlocked and received the "allow high CPU" flag... but they billed me for the nodes as if they had been running that whole time. I asked for a credit (5 days ago) and they haven't replied yet. Probably a little busy over there right now.
So... I'm not too thrilled with DO. I get that cryptocurrency ruined everything, but this has been a frustrating experience and I'm glad I was only using it for a pet project.
-Zach
That's what you get with most providers these days. Ever tried reaching Google or Amazon?
Google on the other hand, I make complaints or suggestions once every couple months and 99% of the time I don't even get a boilerplate response.
So don't lump everyone together in an attempt to paint DO in a better light when it's not true.
[1] https://aws.amazon.com/premiumsupport/plans/enterprise/
Two points establish a line.
Also, your billing system is incredibly spammy (1 to 2 emails a day at times). The bills I get are always for partial months despite the VMs on the account running for a full month.
Thank you! Zach
Not sure why you were downvoted, I had the same impression, after reading:
...an automated service that monitors for cryptocurrency mining activity (Droplet CPU loads and Droplet create behaviors). These signals, coupled with a number of account-level signals (including payment history and current run rate compared to total payments) are used to determine if automated action is warranted to minimize the impact of potential fraudulent high-cpu-loads on other customers
This sounds like they don't permit extended high CPU loads due to the impact it can have on other customers.
edit: from elsewhere ITT it seems they're doing this with stolen credit cards.
My guess would be that this is such a well-known problem (within the field of cloud compute at least) that they just didn't think they had to state that normal crypto mining by paying customers is completely fine.
In every cryptocurrency (the popular and functional ones anyway), there's a set global rate of mining rewards. All miners compete for a slice of that reward, so as more people mine, each individual miner gets less reward. (This causes an equilibrium to be reached where more people mine until it's no longer profitable for more people to start mining. If mining becomes unprofitable, some miners will drop out, and the remaining miners will each make a little more.) If masses of people realize that cloud mining for a particular cryptocurrency is profitable, then what generally happens is that lots of people pounce on cloud providers to mine, it becomes barely profitable, and then people operating their own hardware that's cheaper than cloud providers come in and push the mining rewards down to where it's no longer profitable for people to cloud mine.
Because cloud mining is never profitable in the long run, most cloud mining that happens is fraudulent activity using stolen cloud accounts or payment info. (If you're not paying for it, then making any amount of money from it is profitable.)
[0] https://arstechnica.com/information-technology/2018/01/more-...
Do you disclose this anywhere? Are there any special steps one could take to avoid issues while doing legitimate mining?
I get that your fraud algorithm flagged it because of lack of established payment. how is this possible if what the tweet referred to as "locking us out of all of our backups and work"? surely an account history of any significance would have an established payment record. From their tweets they mention that they had 5 droplets and some storage of a not insignificant number of records (~500k) and that a script is required to be run every 2-3 months to process some data and that script spins up 10 droplets during that time. seems like it will take 13 hours to process the data based on row count and per record time.[0] I am struggling to see how they didn't have payment history. can you elaborate?
In addition another thing I'd think would help assuage fears of a complete lockout is some process where you can request and download the db or a snapshot of the virtual machine.
[0] https://twitter.com/w3Nicolas/status/1134529322902007809
"Cryptocurrency mining mitigation detects suspicious behavior, including very high CPU utilization on an account with no payment history, which results in an account lock"
Lots might have been done wrong here, but it sounds like they had an account with trial or promotional credits - I can see how this could easily be abused.
An attacker might load a stolen credit card number into an account and only use enough resources to generate a few dollars worth of billing. The owner of the credit card might not notice the small charge.
Then after a few months of low billing (to bypass a previous heuristic), they ramp up the utilization, mine a bunch of coins then the holder gets a massive bill.
The holder does a charge back and DO is left holding the bill.
AWS doesn't have this problem because either your instance is allowed to use all the CPU that's allicated to it, or else (t2 & t3) the platform will automatically limit your CPU usage. You don't have to care about how your usage affects other people. It's one more thing that AWS abstracts away. DO's abstraction, on the other hand, is rather leaky in this area. That's a problem in and of itself, in addition to the matter of credit card fraud that every company has to deal with.
An account shutdown, or enough complaints from verifiable sources, and I'm not going to the trouble. Not to pull out an old trope "Nobody got fired for picking IBM". But that's the case: pulls a move like this and the entirety of the customer, who likely came in with AWS in mind (in some cases, was advised against it and insisted on it) is going to shrug their shoulders. Pick a provider that the customer hasn't heard of and I'm going to get a phone call that goes something like "You're the one that said we should use that basement-operation!" with raised voices. Heck, the last time there was an Azure outage, we didn't hear from most of our customers. It was so impacting that even customers well outside of software development/technology read news articles and connected the dots. I had one customer tell me he thought it was just their corporate internet connection; they assumed it was working[1].
[0] Plus, too lazy to put in the research; sorry.
[1] They were a customer who insisted on doing the app monitoring, themselves -- that guy was getting the alerts and similarly assumed it was the network since that happened regularly with another application they developed -- the monitoring server was on-prem.
If the provider isn't paying attention and/or doesn't have good fraud detection in place, it may be a few months before your account and resources are terminated (assuming the payments eventually bounce, and the provider gives you a chance to fix it)
> Responses to account locks were not prioritized differently from a ticket management standpoint to be above less severe tickets.
That's arguably the biggest failure, IMO. The fact that an action which locks/terminates an account is not prioritized any different than a general ticket is pretty jaw-dropping, and I'm glad they're going to change that.
The paper will be read and attempts to improve will be made.
Dad will be blamed.
Killing customer accounts by automated action without any human check just seems like a recipe for disaster. Even if you can respond faster to crypto issues, the effects of a false positive are just unacceptable.
Though apparently the human checks at Digital Ocean don’t work either.
Upon a second review by a different Abuse Operations agent [...] the agent fully denied access back into the account. This action triggered the final “access denied” communication to the customer.
> The initial account lock and resource power down resulted from an automated service that monitors for cryptocurrency mining activity (Droplet CPU loads and Droplet create behaviors).
This sort of 0.01% case is exactly what developers and sysadmins deal with on a regular basis -- some crazy scenario that lead to a bug you've never experienced before. The correct and only response is to fix the bug (whether it be software or process), offer whatever compensation to the injured parties, and move on.
I’m prepared to say the lost income potential is massive.
Who would ever use them for production if they can arbitrarily decide that your servers need to be powered down?
A startup who has fortune 500 clients must have history. I don't get then why digitalocean says they do not have payment history. Either the startup moved a few weeks ago - but then why don't they have offsite backups if they just moved. Or because they're french they did have payment history but did not have an American credit card to similar.. not sure what's up
Now that we're all not picking it, however, I think they should remove the "People" section. They did a good job of adjusting process instead of blaming people. The people section, however, might lean toward blaming people. They didn't, in this case, but it could.
Internally, we want to know exactly who did what, when, and why they thought that was the right approach. We're pretty ruthless about getting those facts down; in exchange, we're beyond lenient on anything that resembles punishment of people (barring multiply repeated cases of extraordinarily poor judgment).
We don't publish post-mortems publicly (though we do internally); still, we generally elide names from the published docs (replacing with role names such as "Operator1", "SquadLead1", etc.), but internal to the teams, we really value understanding exactly what happened and don't shy away from understanding the specific people involved. It's not in any way a black mark on someone's record to have downed prod or made a problem worse. It happens; better we understand and accept that.
Generally, a "People" section that mentions processes not being followed is an incomplete root cause analysis.
Why was it was possible for the process not to be followed?
There's obviously a limit to how far it makes sense to drill down with why why why, but stopping at "someone didn't follow guidance" is too early.
My interpretation of this is that the customer had pre-paid credit on their account. Meaning they had not been through the typical bill cycle yet (hitting an external payment method).
How are you interpreting that they are running on credit? As in their account is in debt and they haven't paid yet?
Requirements:
* 1 year continuous ontime payments at $250+/mo usage
* automatic billing is set up
* billing limits are set up and have been reviewed within the last year
* copy of our business insurance and license
* u2f on all accounts
Fair?
Cloud providers that kill accounts - or SAY they kill accounts, must be dropped and not used.
The worst thing that should be possible is for your account to be suspended.
AWS, if there is a billing issue, prevents you making changes to your infrastructure via the console until the billing issue is sorted out - this is good and reasonable.
""Peer review of account terminations. For any account appealing a lock, two agents will be required to review the submission prior to issuing a final deny.""
- I can imagine how this plays out:
(service agent 1 turns to next service agent along) 'This looks like a bad account - I think I should shut it down, what do you think buddy?'
(service agent 2) - 'Yep I trust you, shut it down.'
That's not the impression I got. It sounds like the issue was that a account with misinterpreted payment history was showing bitcoin-mining-like usage patterns. Mining is not against the terms of use, they were just erroneously convinced themselves that the customer was not going to pay for it.
There is no cloud that will issue such a promise. If your criterion is that a cloud has to promise that you won't get shut down, you just can't use cloud hosting providers.
You may want to explain service credits in some light detail though, for those that are unfamiliar with them.
Refusing service is fine, but holding my data hostage and refusing access to it is not, so I am making a note to not consider DO for any kind of hosting.
I can't promise a super fast resolution - but I'd be happy to work internally to see if there's any outside-the-ordinary workarounds we can supply here if you're willing to follow back up on the ticket.
(As noted by many observers during the initial event, anyone keeping their data exclusively within one organisation's walls is a profound mistake.)
Because shutting down running servers without warning is completely unacceptable for a B2B infrastructure provider.
I suspect that it almost definitely is not.
As IT professionals we should do better than use words like 'kill' to describe system and account changes.
As I understand it, Digital Ocean suspended the account, and because the (perceived) problem was related to excessive / potentially fraudulent CPU usage, they suspended the machines. The data contained on them was not deleted. Does that match your reading?
As someone who suffered from extremely noisy neighbours in AWS (in the very early days), risking significant damage to the performance guarantees to our customers, I'm actually cautiously happy with automated protection systems. Naturally I'd rather noisy neighbours were throttled so that I never heard them, and I expect that's closer to what happens these days
I don't want to spent too much time dumping on them since they clearly know how badly this situation was handled, but this is an example of terrible automated protection leading to a company that's not enterprise-ready. As you say, AWS probably doesn't publicly promise not to terminate your account, but this would never happen because they understand that availability and security matter more than anything else when running B2B infrastructure.
From TFA:
> Shortly thereafter, DigitalOcean investigated the issue and the Raisup account was unlocked and powered back on.
But it's not clear if any data was deleted by DigitalOcean.
The suggestion the account was unlocked rather than re-created suggests it was not, but OTOH there's no reference to erase, delete, restore, or indeed current state of customer data in that post mortem.
That the customer got unlocked is of course a good thing, but for at least 30 hours the customer couldn't access their data, that's highly problematic.
>I suspect that it almost definitely is not.
It's not a promise. But they don't shut it down without letting you know they're going to be shutting it down first.
Kudos to DO for the open incident management. As someone who does this myself, these are often really painful and hard to get right.
And of course DO will still retain the ability to suspend your account for suspected fraud - that is the case with any cloud services company, and any online business in general (check your ToS). Again, I can't think of any business that will en masse promise to never react to any fraudulent users. It's how this process is performed that matters and that's what they are improving.
What I don't understand is why anyone would complain about a post-mortem as PR. If that's what DO was up to here, I'm sold—it shows transparency, thoughtful problem analysis, and swift execution.
More than what this means to me as a customer, I could surely stand to learn a lot in my own work from the way they approached this situation.
There are many companies that have done this in the past. They are not doing this out of the goodness of their hearts, this is lip service for the fact that their mishap blew up in their faces on twitter. Do you really think they would have gone at length to highlight to the public this incident had it not gone viral?
> Not only are they changing their policies across the board, taking on more risk to improve customer experience, but they are hiring extra people so it does not happen again. Kudos for that!
There is no telling that they are actually going to follow through with anything. Mere lip service.
The bottom line is, people host their businesses and livelihoods on cloud providers and they (the cloud providers) should take the necessary care and precautions when taking destructive actions. Maybe err on the side of the customer instead of shutting down someone's entire business because of some automated heuristic. Maybe have a better response time than 29 hours. Maybe teach basic communication and develop processes so that care agents can see and react appropriately to recent activity on the account. These are not revolutionary concepts, they are simple things that demonstrate customer care, something DigitalOcean is sorely lacking.
What cloud business do you run that does better, according to your standards?
No data was lost, it is not destructive in anyway.
> because of some automated heuristic.
If the customer had "payment history" none of this would have happened. Probably it was being used under "startup credits"
> people host their businesses and livelihoods on cloud providers
people shouldn't run entire operation on credits and blame DO in twitter.
Only issue is that DO took 29 hours, apart from that i see no problem with DO.
Why not? Until now, I wouldn't have considered that using credits might make me a second-class customer. They should at least be upfront about that.
Except in the way that the guy may[1] have lost customers or revenue due to the downtime. Being offline, even without data loss, is very destructive for many businesses.
[1] I don't know anything about his business.
Tell that to the owner who was begging DO for their data back on Twitter. Again, had this not blown up on twitter nothing would have been done.
> If the customer had "payment history" none of this would have happened. Probably it was being used under "startup credits"
What's your point in saying this?? The fact is that the customer faced downtime because of a bug in DO's code.
> people shouldn't run entire operation on credits and blame DO in twitter.
Are you saying that customers on credits aren't subject to SLA's?
> Only issue is that DO took 29 hours, apart from that i see no problem with DO.
I think you seem very biased.
Haven't seen those, can you point me to them? I love companies doing this.
You tell me why a company like theirs, which should be mature, ditched their proprietary support and ticketing system (which actually worked) for an shoddy, misconfigured off-the-shelf product which has equally little excuse for being that bad.
I think it's customary here to be unfair to companies, even when they're doing "the right thing" as DO is right now; but DO has spent their customer patience budget elsewhere, so I'm not going to be surprised if people view the post mortem as an inadequate replacement for getting it right the first time, rather than a followup to an honest mistake they will actually try hard to avoid in the future.
We at NodeBB have a high enough spend to have access to level 2 agents (their responses are around 1 hour turnaround, if not sooner).
We don't actually spend that much either, compared to what some of our clients pay AWS, etc...
Now, we certainly don't qualify for their highest tier with the dedicated support manager and slack access, but that's ok. DO's been amazing to us as a host.
Their pricing page reads: "world-class technical support to all of our customers—around the clock" (https://www.digitalocean.com/pricing/#Included_services ) b.t.w. which doesn't seem totally accurate to me, with the 12h and 29h response times in this account banned case.
https://pages.news.digitalocean.com/n/b0000VI6mEn0DfeZ0305PT...
Is it bad for image if they just have a "send a couple hundred bucks because you need to speak with someone real bad" button?
https://www.digitalocean.com/hatch/
That is actually a general pattern. Negative, dismissive responses come fast because they're reflexive. They don't require processing significant information, nor reflection, nor thoughtful writing—all of which take time. Therefore they are the first to appear.
After that, a second wave of objections gets triggered because people read the first wave and are dismayed by how negative it was. That is the contrarian dynamic: https://hn.algolia.com/?query=%22contrarian%20dynamic%22&sor....
This wasn't a failure that impacted thousands of customers [#]. DO could've just fixed things up for this customer and said not a word more, and changed nothing, and everyone would've forgotten about it by about ... now.
Instead they dedicated a nontrivial amount of resources to understanding what went wrong -- identifying not just a single cause, but several -- and publicly explained what happened, without a lot of weasel words, and what they're going to do to fix it.
It's an awesome response.
[#]: ...at that specific time. Yes, others have probably previously been impacted.
DO never suitable for business use case.