Don't use Digital Ocean (DO) for production
Long story short, I've had an account with them for roughly 5+ years as I recall, good credit card on file, monthly bill around $20-50, US address. I use a few droplets to run dokku/docker containers for dev and 1 private Google Outline VPN endpoint, only for me. I only ssh into these from non-root and root should be turned off completely (but I'm only a hacker and not a F/T systems person).
Not sure if it matters but I don't do anything nefarious on these droplets-- no webscraping, no botnet creation, no crypto mining. I don't know what else would be considered verboten!?!
Try to log in with github, won't work and asks for email login. Try to login from there, and it asks for credit card info for "authorization" although the card is the one on file and being charged for so many years.
No recourse, and what a crappy way to treat clients even if they are small potatoes in the grand scheme of things. Thankfully I am building a new MVP all on AWS and have my company prod there too running for 5 years. So I don't have to put up with unprofessional dealings and major risks of service interruption.
Assume through no fault of your own that you could be shut out of everything. You have been warned.
79 comments
[ 2.9 ms ] story [ 158 ms ] threadBut somehow a droplet was part of a DDOS and network was booted off ( 1 network spike for 10 minutes). Let's just say it was stressful and very unwelcome at the time.
I also went to mention that i did receive feedback and there was definitely a human from their end involved.
But, I wouldn't want to imagine what would have happened if it were websites of clients. I took more measures by now, but i think the actual culprit was a contractor that did some work for me.
I'm wondering how other providers handle that ( a VPS for example)? Anyone has some insights on how it can be handled better?
Depending on what you mean by "booted off", any and every reputable provider will do the same. They all have AUPs that allow them to stop routing for your, at a minimum, if you're found to be sourcing traffic that's against their AUP.
That's all normal. You can't expect a company to keep you online when you're causing problems for them and the rest of the internet, and their contracts/AUPs are all worded so they can take action if your systems are found to be a problem.
What can you do to prevent this? Don't let your systems be part of a botnet or get hacked in general. More feasibly, stay up to date on all security issues and take multiple system backups so if you have to restore from a good prior backup, if that's possible.
> I took more measures by now, but i think the actual culprit was a contractor that did some work for me.
Could be, but my guess is not. Botnets survive on having massive numbers of servers in them, and are often automated exploits. More likely, you were running some exploitable software, and it was exploited. Updates need to be run, daemons often need to be restarted after updates because of older code still running in memory, etc. And if you were running some CMS like Wordpress or Drupal, well if you aren't doing it yourself and keeping on top of updates and new versions, you either better have someone on retainer to do it for you or give up and do something else, because you're a perfect target for a botnet at that point because those generally have problems on a regular basis (as well as the problems introduced by different modules they each might support).
The context is missing without my original sentence. "Network was booted off." Obviously not my account but "network access" and i could only connect through the console through their web dashboard.
> Could be, but my guess is not.
According to the files i found and the zipfiles he left behind. There would be a huge confidence of similar timing and folders involved. He even named the zip similar to the work he did.
I also know how botnets work, backups and security updates. I already took measures as mentioned before.
I never said and/or blamed DO for it. I'm not running sites on it, but some personal stuff as i mentioned ( again).
I just stated the situation as it was in my original comment and wanted some insights on how ( and mostly if ) it could be handled better...
Tbh. I handled your guesses in my original comment already at the time and you seem to be focusing on attacking my explanation of the situation instead of my only question and answering questions i didn't ask and already resolved. Which is really frustrating to respond too... As you're not talking to me, but a reader of the comments.
I didn't take it as written because it was too ambiguous to know. Often your actual IP or network range might be null routed from upstream providers. Or perhaps DO just turns off routing for the systems in question. Or it could be a poor description of something else.
I work at an ISP. We will null route colocated customer IP addresses if they are causing problems until they resolve them and contact us, or if they are the target of a DDOS. The point of null routing is that other providers won't even send you the traffic anymore, so it neatly solves the problem. Sometimes we have to do it for IPs for our own services if they are the target of a DDOS.
> I never said and/or blamed DO for it.
I'm just letting you know how network providers work. If you cause hard to the network, and all their other customers, whether on purpose or just because you're the person responsible for the resources in question, they will shut you down. That's expected. It's a good thing they do, otherwise none of these providers could even offer useful service because every day there's be an event destroying their network, and your experience every day prior to the problem would have been crappy service.
> I just stated the situation as it was in my original comment and wanted some insights on how ( and mostly if ) it could be handled better...
The correct way to handle your service being part of a botnet is to immediately remove your server from the network, whether they do it or you do. You server is causing active harm to other people on the internet while it's left online in that state. It sucks for you that it goes down, but really, that's how the world works. When something you control is causing harm to the public, it will be dealt with whether you like how it happens or not. The way to deal with that problem is to do everything in your power to prevent if from getting to that state.
Because of that, iit's impossible to be absolutely secure from a network provider shutting down your service. If it's that important, you need to have backup plans, le other services you can can bring up at a moment's notice and swing DNS to. People make good money designing and implementing disaster recovery and business continuity plans like this. That's likely not feasible for a personal site. But usually people understand and can deal with a few hours or even days of downtime for personal sites, while a business might not recover as well from that.
> I handled your guesses in my original comment already at the time and you seem to be focusing on attacking my explanation of the situation instead of my only question and answering questions i didn't ask and already resolved.
I think perhaps you didn't understand what I was trying to express. What you experienced is normal and the correct and expected way network providers will deal with a problem like that. I tried to express that.
As for whether your consultant installed a botnet on your system, it's possible I thought it unlikely because the amount they could gain from one extra hour of repeat business from you likely dwarfs the gain from a single extra botnet computer, of which botnet networks have thousands. There's just not a lot of incentive for a consultant to put a system they're working on on a botnet. I would rate it as more likely that something the consultant did left the system in a vulnerable state, but it doesn't really matter does it? I wasn't trying to attack you or indicate you screwed up, just indicate that if your system was on a botnet it probably got there like 99.999% of other systems on them, which is some automated work that infects it, adds a control mechanism, and then scans to add other hosts. Or some script kiddie ran some scanner/infector tool and added the control mechanism to join it to a botnet. Because the ti...
As mentioned. I thought that it was normal, i was just curious if there was a better way that's.. less stressful :p
DigitalOcean Killed Our Company: https://news.ycombinator.com/item?id=20064169
I’ve found support response time from both so far to be excellent (UpCloud in particular though their feature set is just slightly behind Vultr and DO currently).
I had been testing the water with DO with a view to hosting some production stuff with them, but I’ve seen a few too many similar stories like this one and decided to get out of DO and rule them out for production stuff for now. Many of their direct competitors seem to provide much better support at lower levels of spend too.
I would happily make a 20 or 50 EUR/USD payment in advance to avoid the risk. But AWS is not flexible enough to allow me to do so. I understand in some countries it's possible, but in EU there are no options.
> Your card payment ending in XXXX failed when we attempted to charge it for AWS services on account YYYYYYYYYYYY because it requires additional authentication. To complete the payment, please complete your credit card authentication.
Expect an e-mail with subject "Action required – Your AWS account is past due" if you haven't gone through the authentication process for your card.
Luckily they were pretty noisy about it. I received the first mail about it on Dec 2nd and then reminders on Dec 6th, 9th and 13th. At this point I noticed it and made the payment. They didn't shut down any service meanwhile.
Good to know that they warn at least 10 days. If that is constant and not dependent on some algorithm. I could not find any information how fast they stop your server and whether data would get locked first or deleted immediately.
[0]: https://news.ycombinator.com/item?id=29596138
The problem is that customer "service" is handled by algorithms with no chance to reach competent humans who have the power to make reasonable decisions.
For work, I run a cloud service like this, and we do indeed have this kind of fraud. Every stolen credit card number under the sun has signed up for an account. People create thousands of emails (and brand new domains) to get free trials. And, when our systems identify them and we shut them down, they whine in all possible support forums. I don't think we've ever made a mistake, but that is likely due to the small scale of our operation.
The only thing that keeps me sane is the extremely poor opsec of the fraudsters. They have pools of thousands of compromised IPs they're using for the command-and-control infrastructure that they install, but they often mistakenly use the same destination crypto wallet (they have a little LD_PRELOAD script they install to try to prevent us from seeing this, but it doesn't work if you don't load dynamic libraries to read the process table), or use a predictable naming scheme, or taint accounts by accessing a few of them from the same IP. (I have a graph that connects the accounts through things like IP address, so this is often the fatal screwup for thousands of accounts all at once.)
I imagine that Digital Ocean has this same problem, but at a million times the volume. So, there are going to be mistakes. It's intrinsically difficult to establish trust at these small scales. DO probably feels pretty uncomfortable giving someone a bunch of compute resources when they haven't even tried billing the credit card. You would feel pretty uncomfortable paying up front. So this is where we land, sometimes you look like fraud and get killed with fire. (As you get bigger, you'll have weekly meetings with your cloud provider, and they probably won't shut you down automatically. But at that small scale, it's tough for everyone!)
For cloud services to small customers there is luckily still a bit of choice. Unfortunately I don't see that any of them would really be better if things go wrong for some reason or another. Well, if you pay them less than a hourly salary for a whole month, customer service is not really an economic option.
This is insane, if I wouldn't have those backups and the same thing happened to me, I would be screwed. How come they don't contact you beforehand, or at least let you move out your data (keep the account open but cut the droplets from public internet if for whatever reason their algorithm detected nefarious activity on it, or a bogus DMCA, exploits happens, etc).
I wanted to try an alternative to AWS but look like I'll be back to it soon. Does AWS have a similar track record of closing account out of the blue? Is there any other alternative that have better support and would at the very least try to contact the owner before taking drastic action like this?
[0] https://aws.amazon.com/lightsail/pricing/
[1] https://www.digitalocean.com/pricing
[2] https://www.linode.com/pricing/
[0] Linode got DDOS'ed literally entirely off the Internet in December 2016. You can't take AWS down with a DDOS. You can take down an individual customer server but you can't take AWS offline with a DDOS.
[1] https://news.ycombinator.com/item?id=20064169
A while back I made a stupid mistake while tired, and I was panicking and couldn't find the feature I needed in the account.
I called Linode at around 11PM CDT, and got a helpful friendly guy on the line who walked me through what I needed. I was amazed that I could get a human on the line in less than a minute after hours, when only paying $10/month.
This is probably the most reasonable risk mitigation you can fairly easily do.
I don't think there exists a vendor that's going to be foolproof. Most users aren't going to any have issues with any of the major providers, including DO, Linode, Vultr, Oracle, AWS, Azure, etc...
What does everyone do to easily and automatically back up their side project droplets offsite?
It's supposed to be a last resort kind of backup as I also use DO backup solution (managed database) but seeing how they can close an account out of nowhere, it ain't worth nothing.
Every time I've messaged support for an account lock, 24 hours later I'm informed it was a false positive and the account is unlocked. Needless to say, this does not inspire confidence that my resources are going to be kept up and running past Kafkaesque fraud monitoring systems
Sounds like good customer service to me ;) I tried to open an account with Oracle cloud and it rejected my credit card. They never even answered to my messages.
Well, I was not overly convinced I want to do business with Oracle anyway, so now AWS charges the same card every month. Oracle seemed somewhat cheaper for the use case, but nothing that kills me.
It was blocked but DO unlocked it once I explained it in an email to them.
I've been happy with DigitalOcean for a while now, but reports like this concern me, and I've also discovered after investigating that the reason MS-hosted email accounts (@hotmail.com, @live.com, etc) reject all emails from my VPS is because the entire provider is on the UCEPROTECT blocklist, and apparently has no interest in changing this.
I am immensely relieved. They've done right by me.
Did they explain why it took over 48 hours to reply to you?
I'm looking for the next Digital Ocean. Something still bespoke, with higher cost but better customer support. So far oldschool VPS hosts are looking good.
I see value in supporting smaller companies (relative to large ones like Amazon, Microsoft or Google), but smaller companies usually tend to have better customer support when things go wrong, and are generally a lot more helpful. With DO, it has the kind of offerings by a smaller company with the kind of support by a large company.
I am immensely relieved. They've done right by me. Kudos for a delayed but excellent response.
Luckily for us we added redundancy on payments and alerts, so I hope we won't be in the same case again.