Hi cyberferret. Two quick trys to provide some clarity for you here: 1. On your vendors don't say why they took action point. I think a lot of vendors, us included, worry about the bad actors knowing the _how_ of our…
DO has a startup program called Hatch: https://www.digitalocean.com/hatch/ That is the starting place for many folks.
Hi weaksauce... sorry I must have missed your earlier question. The account had been live for some time and in that sense had history but because of credits it didn't have payment history. As some others have commented…
This is correct. This was the primary thing we were attempting to solve for in this case and the bug in the algorithm started the chain of events documented in the postmortem.
Sorry to hear that you had a bad experience and left with a bad impression of that team. We have a number of data sciences efforts including in the core R&D group where we are growing and working to improve models in…
I agree on the twitter cred point. The fact that this happened in the end, personally I think it is a good thing as it highlighted a weakness we must fix. We trust our people high-level, low-level whatever to make…
Correct.
Nothing was deleted or removed. The droplets were powered off and the access to them locked. once (way too long later) the unlock happened the customer had full control and access again.
Thanks for the pointer. I'm going to blame my dad/up bringing for my over use of passive voice. He will be deeply amused by this. I will read the paper and attempt to improve.
Hey there. Thanks for this feedback. I think it is important to be open honest but not blame-oriented in our review of the situation. People make mistakes and that is okay, so long as they aren't willful or due to…
The first few items on your list are actually a part of what we meant by "having billing history with us". There are a number of things we look at in that bucket. We use these items as a part of validating users before…
Yeah... that one was painful and we are fixing it. At least if the priority placed this at the top of the queue we could have acted faster. Probably the same outcome due to the other issues involved in this incident…
I was very worried about that specific detail and we reached out to the customer before posting this postmortem to expressly get his permission to share those details. If he had said no, we would have worked around the…
If you want to do cryptocurrency mining on DO that is actually okay with us. Some of the other respondents are correct the behavior we were looking for was really around fraudulent accounts being created and performing…
Last week ended on a real low note for many of us at DO. We took a perfectly good customer and gave them an experience no one should have to go through (all while he was trying to leave on vacation no less). We can and…
Yup. LeonM, you are correct. In this case that was the cryptocurrency mining detector that was triggered. More details in the postmortem.
I wanted to provide you all with an update on the postmortem I promised on Friday. Our analysis has been completed. We will be sharing the full document soon and will publish a link in this thread for those wanting to…
Thanks for the replies. Let me try to address a few of the things I have seen here. We haven't completed our investigation yet which will include details on the timeline, decisions made by our systems, our people, and…
As DigitalOcean's CTO, I'm very sorry for this situation and how it was handled. The account is now fully restored and we are doing an investigation of the incident. We are planning to post a public postmortem to…
Hi cyberferret. Two quick trys to provide some clarity for you here: 1. On your vendors don't say why they took action point. I think a lot of vendors, us included, worry about the bad actors knowing the _how_ of our…
DO has a startup program called Hatch: https://www.digitalocean.com/hatch/ That is the starting place for many folks.
Hi weaksauce... sorry I must have missed your earlier question. The account had been live for some time and in that sense had history but because of credits it didn't have payment history. As some others have commented…
This is correct. This was the primary thing we were attempting to solve for in this case and the bug in the algorithm started the chain of events documented in the postmortem.
Sorry to hear that you had a bad experience and left with a bad impression of that team. We have a number of data sciences efforts including in the core R&D group where we are growing and working to improve models in…
I agree on the twitter cred point. The fact that this happened in the end, personally I think it is a good thing as it highlighted a weakness we must fix. We trust our people high-level, low-level whatever to make…
Correct.
Nothing was deleted or removed. The droplets were powered off and the access to them locked. once (way too long later) the unlock happened the customer had full control and access again.
Thanks for the pointer. I'm going to blame my dad/up bringing for my over use of passive voice. He will be deeply amused by this. I will read the paper and attempt to improve.
Hey there. Thanks for this feedback. I think it is important to be open honest but not blame-oriented in our review of the situation. People make mistakes and that is okay, so long as they aren't willful or due to…
The first few items on your list are actually a part of what we meant by "having billing history with us". There are a number of things we look at in that bucket. We use these items as a part of validating users before…
Yeah... that one was painful and we are fixing it. At least if the priority placed this at the top of the queue we could have acted faster. Probably the same outcome due to the other issues involved in this incident…
I was very worried about that specific detail and we reached out to the customer before posting this postmortem to expressly get his permission to share those details. If he had said no, we would have worked around the…
If you want to do cryptocurrency mining on DO that is actually okay with us. Some of the other respondents are correct the behavior we were looking for was really around fraudulent accounts being created and performing…
Last week ended on a real low note for many of us at DO. We took a perfectly good customer and gave them an experience no one should have to go through (all while he was trying to leave on vacation no less). We can and…
Yup. LeonM, you are correct. In this case that was the cryptocurrency mining detector that was triggered. More details in the postmortem.
I wanted to provide you all with an update on the postmortem I promised on Friday. Our analysis has been completed. We will be sharing the full document soon and will publish a link in this thread for those wanting to…
Thanks for the replies. Let me try to address a few of the things I have seen here. We haven't completed our investigation yet which will include details on the timeline, decisions made by our systems, our people, and…
As DigitalOcean's CTO, I'm very sorry for this situation and how it was handled. The account is now fully restored and we are doing an investigation of the incident. We are planning to post a public postmortem to…