HN Explorer

Ask HN: DigitalOcean is shutting down my site. What to do?

10 points by suhair ↗ HN
I host my personal blog generated through Jekyll in DigitalOcean and they are closing the droplet for the second time. My network is disabled for the droplet so i dont have much to do. They are telling about ddos attack initiated from my droplet and previously they closed the droplet and said they can't bring it back so i moved to another droplet, now this thing is getting repeated. Does anyone else face such problem?.

------------------- Hi there,

We are sorry to report that we have detected what appears to be a DDoS attack being launched from one or more of your servers.

To shut down this attack, we have disabled the networking interface on the server or servers involved, so further connection would have to be accomplished via the console in our control panel. You need to use the direct link provided, as your control panel access will be limited.

This is the direct link to the console of the effected droplet https://cloud.digitalocean.com/droplets/1559956/console

Please log in at your earliest convenience in order to investigate and remove the program generating the traffic. Once this is done, please also determine how this software came to be installed on your droplet and prevent it from being installed again in the future.

If you need any guidance on how to find and resolve this issue, we recommend reviewing this: https://www.digitalocean.com/community/questions/my-droplet-is-locked-by-support-staff-because-because-of-an-outgoing-flood-or-ddos-what-do-i-do

Once you are done let us know and we will investigate re-enabling your networking.

Please understand that this is a very serious issue, and that should it re-occur we may suspend or even terminate your account to prevent further incidents. If have any questions or need any guidance on how to protect your servers please let us know.

Thank you, DigitalOcean Support ----------------

19 comments

[ 0.21 ms ] story [ 49.9 ms ] thread
justinsb
If it's static, then run it from S3. It'll be cheaper, and there are no servers to secure.
suhair
At the moment, i am considering moving back to Linode. I dont have experience with S3 but will give a try as i am not very sure about what will happen in Linode too
wanghq
Highly recommend s3 - it's fast/cheap/highly available. You don't need any server to host static site.
yashness
Digital Ocean is not at all safe to use. They disabled my account, having credit of 100$ of student pack. I used to pay 5$ for droplet since 4 months and they are now without any genuine reason locking my account. Neither do they listen to any queries and rather they respond back saying that We will not unlock your account. I am also frustrated by them. It's not safe and reliable to host application on digital ocean servers because they can become assholes anytime.
suhair
But the general picture for the one who frequents to HackerNews to get about DigitalOcean is that they are very good at resolving issues and my experience now is very bitter. I will not rely on them for hosting anything important. Considering to move on. They have not replied to my ticket and now my blog is down. previously it got down for two days and finally i moved to new droplet.
yashness
Yes, on their way, technically 99.99% uptime is bullshit. because they can make us lose customers and users due service unavailability and downtime. I used to suggest digital ocean to my friends but now, I deny people for the same due their such behaviour.
jjjdev
Did you actually read their email to you? You have console access so you should probably access your droplet that way, find how your droplet got compromised and remove what's causing the outbound DDOS. DO is an unmanaged service, so if you're incapable of managing and securing your server, perhaps try a managed provider.
stevekemp
Reading the mail seems clear enough - your server/droplet was sending traffic. If you think it was just hosting a (static) blog then it has presumably been compromised.

Did you follow the advice link? Did you look for sign of compromise?

On the face of it disabling a compromised server is precisely the right thing to do - to stop it attacking other users, even if that puts your site offline. Or do you disagree?

suhair
This is my second incidence in a span of one month. Just search and you can find a large number of such incidences in digitalocean. When this happens, the options to resolve are very minimal. I can see that some strange files are being created inside the /boot/directory of that droplet. DigitalOcean support says, only option is to just create another droplet and migrate. I thought in this situation i could install a malware scanner and remove the threat or is this the standard state of addressing security?
stevekemp
It sounds like you have some files appear, which proves a compromise has occurred, but you don't know the source of those files then?

If that is the case, if you're running software that has security vulnerabilities if you install that same software on Linode, or anohter host, you'll just get compromised again.

The solution has to be for you to:

* Learn how you were compromised.

* Actively take steps to avoid it.

Otherwise you'll find yourself posting in six months time "Linode disabled my server, help!" and "I'm moving to Hetzner".

suhair
My wild guess is that, the version of the ubuntu for that droplet is not supported now. Yes from a learning perspective this is fine. But you cannot recover the droplet from malware is not a pleasant news.
[dead] axeny
About that, I had the absolutely issue, but I was hosting WordPress.

The first thing I was doing after setting a droplet was changing the password to something that was easier for me to remember, something along the line of 'qweasdzxc' but a bit harder combination. This was huge mistake on my part.

Apparantly my password was being bruteforced and once they get root access the DDOS attacks were being performed. What I did was delete the first droplet, starting a new one and just adding a few numbers after the randomly generated password that is sent to you by email. Then I went ahead and installed fail2ban(https://www.digitalocean.com/community/tutorials/how-to-prot...) + some iptables configurations thats are shown in that link. It practicly makes bruteforcing your droplet close to impossible(at least I think so).

If you need any assistence you can contact me through my profile e-mail and I would gladly help you. Remember though you will need a clean droplet, because your system was already compromised and there are holes in it, that simply installing fail2ban will not be enough.

P.S I have no idea why this post is 'dead' and I can't see your e-mail in your profile to shoot you an email with this information.

spv
if you generate your blog using jekyll and are not using any custom plugins, why dont you host it on github.

Even if you use custom plugins, you can still generate the blog locally and push the resulting HTML to a Git repo.

look at https://github.com/jekyll/jekyll/issues/325

codegeek
Not sure if this matters but I realized that DigitalOcean recycles droplets i.e. they re-assign a same IP to a new client taking it from a previous client who might have left/deleted their account. It happened to me actually. Of course, that may not be a big deal but I noticed that I was getting a lot of traffic from a certain domain that still points to my IP (they probably had that droplet IP before). I don't like that but DO said there is nothing tehy can do because the domain owner has to change the nameservers.
nemexy
About that, I had the absolutely same issue, but I was hosting WordPress.

The first thing I was doing after setting a droplet was changing the password to something that was easier for me to remember, something along the line of 'qweasdzxc' but a bit harder combination. This was huge mistake on my part.

Apparantly my password was being bruteforced and once they get root access the DDOS attacks were being performed. What I did was delete the first droplet, starting a new one and just changing the default password by adding a few numbers after it. Then I went ahead and installed fail2ban(https://www.digitalocean.com/community/tutorials/how-to-prot...) + some iptables configurations thats are shown in that link. It practicly makes bruteforcing your droplet close to impossible(at least I think so). If you need any assistence you can contact me through my profile e-mail and I would gladly help you. Remember though you will need a clean droplet, because your system was already compromised and there are holes in it, then simply installing fail2ban will not be enough.

P.S I had to make a new account to post that comment, I guess my old account was punished or something.

[dead] haidrali
ISSUE: Actually the main issue is after creating droplet on digital ocean you have to manage it completely from administrative as well as security point of view. They only provides you network level security.

HOW TO RESOLVE: First secure your droplet by installing firewall, fail2ban, IPV4 routing etc security measures, there is lots of information about this on digital ocean blogs. Then go and talk to them that you have secured your droplet (via support ticket) they will check and restore your droplet

I also have experience same issue of DDOS on digital ocean. It took almost one month to negotiate and to make them realize that i have secured my droplet U hope it will help