Why isn't that the point? If the author really wanted to get people "on his side", he should have included Digital Ocean's side, specifically the part of their TOS they claimed was being violated.
Agreed. From the screenshots we are given it doesn't look like DO gave much information to the customer. However the customer did leave out some screenshots of messages where they supposedly replied with all of their personal information. I suppose to not reveal said sensitive information.
I'm wondering now if dropbox reported them for abuse and DO suspects that the account was compromised and is being used maliciously...?
Edit: Looking at their backup script, I don't see any error condition or exception handling when the file upload is posted via the dropbox client.
It might be that dropbox triggered some kind of abuse/rate limiting with a message to that extent. Without catching that scenario, the cron job would simply keep running... could be a situation where the dropbox user didn't know they were being abusive, were reported by dropbox to DO for abuse, and here we are.
I feel like there is more to this story than we are being led to believe. I'll definitely keep a eye on this as it develops. So far I've found DO support to be heavenly.
I think their service being so cheap is more reason for a plan B in case one of your accounts go down. There are so many options for making this happen. And again, all pretty cheap. With a good back-up / fail-over plan, this shouldn't even be much of a blip. Personally, I never contact support. If I have a problem with one provider, I'm a button push / command line command away from moving everything over to another provider.
> First of all, the support which was really helpful and kind at the beginning has became much worse and somewhat hard to understand.
How many times do you contact support? For me to contact support, there has to be a problem with the actual service itself. That is, it's an obvious problem that is outside of my control. I have been running VM's with quite a few providers, certainly all the big names. I almost never have to contact support (I would have to think pretty hard about a time that I have done this.) These services are usually quite solid and if there is a problem, usually there is some sort of status which lets you know there is a problem so that you don't have to ask.
For me, having to contact support even once would be a big problem. Having to contact support often enough that I have noticed that support was once good but then turned bad would have been enough for me to have long since moved on.
I'd totally agree with Google part. Their free services (Search, Maps, Mail) are amazing but once you're doing business with them, they are a PITA to work with.
Notice that the author doesn't address the issue of whether they actually were violating the TOS and/or AUP, or what the alleged violations actually were. My interpretation is that the author likely was legitimately violating the TOS, because otherwise he would have mentioned how dumb the allegations are.
If the author was breaking the AUP, I don't feel very sorry for him. If someone's doing something that legitimately violates acceptable use, they probably should be shut down without prior notification.
Hey thanks for the comment.
To let you know the account has 2 droplets which serves 2 mobile apps and does crawling 20-30 sites hourly. That's it actually and also the Dropbox script that i mention.
Surely, you would have figured out which part of the TOS were you violating (if you were). It would be nice if you could relay that in the post or in the comments.
I run 100 droplets on DO and fetch 7MM feeds a day and have never had an issue with DO, either limiting me or turning off my machines. In fact, I've had the opposite experience: fast support, fast spin-up times, and an overall great service.
I realize you sound angry because you were asked to verify your account, but this reads vindictive and makes me think that the same behavior that caused their abuse department to flag your account is what's responsible for this attack blog post.
You mention a script to backup the database. Do you think that's what caused the account to be flagged as violating the TOS? Or is it the case that you don't know why it has happened, and just want help, and to let people know?
This doesn't actually tell me to not use Digital Ocean. All it tells me is that the author is trying to make an emotional appeal to other people, without giving any information other than Digital Ocean locked his account, citing violation of their TOS.
While I can't say that they're TOS violation claim is bogus, I also can't say it's not bogus, because the author didn't publish (or, apparently, ask) for that information.
Great, you've got a large number of users, and you used Digital Ocean to host your backend. Why are they saying you're in violation of the TOS?
Ask the follow up, figure out why they're not pleased with you, and don't try to appeal to emotion when you have no substantiating information.
As a side note, I have no horse in this race. I don't use Digital Ocean, and I have no affiliation with them. I just don't see the author's claim as being 100% put together and honest.
Hey thanks for the comment.
To let you know the account has 2 droplets which serves 2 mobile apps and does crawling 20-30 sites hourly. That's it actually and also the Dropbox script that i mention.
Really? So if a customer was paying you $5 a month to run a VPS, you wouldn't shut it down if say... it was hosting child porn? Or if it was sending so many spam emails you were at risk of having all your other customers' servers impacted?
And if you did have some automated system and various criteria for detecting likely abuses, if someone got shut down under those terms, would you tell them exactly where the line was? Or would you keep your detection methods to yourself?
"Your site has been turned off due to hosting child porn, in contravention of local/international law XXXXXX, per complaint YYYYYY".
Then they can at least dialogue.
Worst case scenario: the VPS is shut off automatically, I get a semi-urgent notice, and I reply to the customer with further details as soon as possible.
And they would get a message letting them as much.
I still think it's important to reiterate as much as possible why a service was shut off, both for a customer understanding viewpoint, AND from a "cover your ass" viewpoint.
It can't hurt to say "As stated in the past email, your server was shut off due to YYYYY", rather than just "Game over, man. Stop emailing us." equivalents.
Using a smiling icon like the fish and then issuing a copypaste/boilerplate message is insulting. If you are going to deliver non-informing messages, do not SMILE in you icon.
This is nitpicking but in the end, you (DO) have a smiling icon because you expect you support group smiles. If your support group is a machine, well then... use this one.
My (a couple weeks old) Ubuntu 10.04 server locked up, no ssh or VPN access but could ping it. I contacted support and was told to destroy the droplet and create a new one. Still have no idea why it happened or how to avoid it on my production server.
I had the same problem. I was messing with my iptables and messed something up to the point where there was no network. I contacted support and they installed a Ubuntu Live image. (Something that has to be requested)
I used the Live image of my droplet to mount the disk, copy the databases, and websites. Once I was satisfied I got everything I destroyed the droplet.
Try a different browser. I've used Chrome Mac, Firefox Win, Firefox Mac, Safari. I just logged in using Safari (mac) 6.0.5 and found it working while 6.0.4 did not.
I've just skimmed over the TOS. It seems a bit atypical:
> DigitalOcean reserves the right to modify the Terms of Service without notice.
Well... that more or less invalidates the sanctity of the terms.
> DigitalOcean also reserves the right to terminate a customers account if they are targeted by malicious activity from other parties.
What is the justification for that? Simply being the target of malicious activity means that your account can be terminated? It's not as though one is responsible for malicious activity directed against them.
> DIGITALOCEAN DISCLAIMS ALL WARRANTIES, EXPRESS, IMPLIED, OR STATUTORY, REGARDING THE SERVICES PROVIDED HEREUNDER, INCLUDING ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE ...
I see this a lot with software, but doesn't this mean that DigitalOcean has no actual obligation to provide the cloud or VPS services they advertise? I didn't see any such clause in the EC2 TOS, but I did not look very hard.
Is this the usual sort of agreement for such a service?
Frankly, none of those terms seems atypical. Amazon does have SLAs[1], but they do not apply if your account has been suspended. Reasons for suspension can include your use of the service adversely affecting any other AWS customer. They may also terminate you if providing services creates a substantial technical burden. Both of those can be triggered by you being "targeted by malicious activity from other other parties".
>Well... that more or less invalidates the sanctity of the terms.
You'll find the plurality (if not majority) of ToS's out there include such a statement.
>What is the justification for that? Simply being the target of malicious activity means that your account can be terminated? It's not as though one is responsible for malicious activity directed against them.
They don't want to deal with the networking implications of a large scale DDoS. They're providing low priced VPS services, not high bandwidth, high load custom racks. Also, they "reserve the right" not the "always exercise the right". DO likely won't kick you for a small DoS/DDoS but sustained attacks take much more manpower to deal with than you're paying them for.
>I see this a lot with software, but doesn't this mean that DigitalOcean has no actual obligation to provide the cloud or VPS services they advertise?
No, it means you can't sue DO because your VPS went down and you lost 10K is sales because customers couldn't access your storefront. It's an indemnity clause.
>Is this the usual sort of agreement for such a service?
Yes, unless you're paying a good deal of money for something like a Colo with 1hour SLA or an AWS instance with a high SLA.
Being the target of e.g. a DDOS attack can consume considerable resources and affect many other DigitalOcean customers. It may not seem "fair" to terminate your account over that, but having your site unavailable because of an attack on a neighboring customer is no fun either.
IANAL, but I don't think that warranty disclaimer is relevant here. I don't think it means much at all actually. As a business you can say you're not liable for anything bad that happens, but that doesn't necessarily make it true.
I don't do a lot of comparison reading of the TOS on a lot of hosting providers, but this stuff doesn't seem as unusual to me as it does to you.
Most providers will shut you off if you're running a server that attracts a lot of DDOS attempts. Many providers explicitly disallow running irc servers up front to avoid this very scenario. If your site is constantly getting DDOSed for whatever reason that is going to have a big impact on the hoster's entire network and other servers they are hosting and while it may be through no fault of your own, almost all hosters are going to bounce you if it is a recurring problem.
This happened to a friend who runs a DO server on July 22nd. The server had been running for several weeks when it was suddenly taken down out of nowhere for over 4 hours to "verify" the account. There wasn't any warning and my friend immediately provided what they had requested.
Their response? They shut down the server because of "unusual traffic" coming from the server which wasn't even being used yet.. but there was "outbound traffic about 977.38 Mbps at its peak at around 2013-07-22 14:50:00UTC"
They need to fully verify accounts before letting anyone create a server. You shouldn't just take down a server out of nowhere, for several hours, after it's been running for weeks.
It's just not practical to fully verify accounts before letting them create a server. It is practical to tell customers why they are being black-holed and give them a chance to fix it when it happens.
I've been unfortunate enough to be DDOS-ed on both linode and digital ocean for some of my sites. suffice to say, linode handles the ddos much more gracefully than digital ocean. digital ocean promptly bans you for TOS violation and locks your account after 3 DDOS attempts. I know most people don't have sites that are DDOS-ed often, but if you do run that risk, just know that digital ocean will most likely have you moving pretty quickly.
That is troubling to know. I have my pretty important website running on DO, and though this is only for the short term, I might have to accelerate my plans to move off DO.
going to share another unpleasant experience here too. attended a google glasses hackathon and we ended up deciding to use digital ocean for quick prototyping of an AR app. i made the mistake of giving my teammate my paypal account, not knowing that the account had been blacklisted. a day after the successful demo, the exact same "conversation" unfolded and the account was promptly locked. i'm just glad it didn't happen an hour before the demo!
Our privacy policy prevents us from disclosing any account or personal information so we can not speak to this customers situation publicly.
If the customer would authorize this disclosure then we can discuss it publicly.
Otherwise, as much as it pains us to get negative feedback, whether or not it is deserved, it is beyond our control as the privacy of our customers is the most important issue at stake in a public discussion.
I can understand this position and can even understand why you can't just take this article as being authorization..... even though it's pretty clear (imho) that the linked article is practically begging for DO to explain publicly why the account was terminated.
That is entirely unclear (about the authorization) since people often want to present their side of the story in terms that they see as best, and having someone say "well you were doing <x>" where X is some evil thing.
As we (Blekko) get better at identifying and shutting down click fraud and other schemes to defraud advertisers a lot them end up being traced back to hosts in inexpensive hosting services. That is but one of many ways of having a machine on the Internet that generates cash for you. We ended up basically shutting out a couple of class Bs from an hosting service in the Ukraine for that reason, people can write us and get white-listed but in over 27,000 IP's not a single one was a legitimate user. Way too many people coming up with schemes to get some "free" extra cash.
The affected user posted in this thread a good number of minutes after this rose to the top (and subsequently deleted that post). This seems telling to me.
He is indeed. If you browse any of the DO support/tutorial articles he's been active all over.
I've been nothing but impressed with DO, right now there seems to be a misunderstanding and OP is not providing full details nor is he allowing DO to discuss what happened.
If he truly cared about others not using DO he'd allow it to be discussed, but it does seem like OP is hiding some fact...
So now I'm left with great service and product vs 1 guy who seems like he did something shady and got caught. Maybe it was an accident and maybe there is a way forward, but only if he brings to light the details of his actions.
Edit: Plus now you delete your post where you hint at your DDoS (by mistake or not) does not add up to your favor: http://i.imgur.com/1pxIxiN.png
This happens all the time in the industry. If you cut off a customer for abuse, they get /mad/ at you. I mean, they get way more angry than when I screw up a customer's domain through my own incompetence. When you shut someone down for abuse, they react as if it was a direct, personal attack.
One user sent me a bill for $10,000 after I cut 'em off for running a spam cannon, for lost business. I mean, I'd say 90% chance that they were just incompetent sysadmins trying to run wordpress and they got compromised, but ground truth was that they were sending out rather a lot of spam.
I sent a few notices before and after shutting them down, but their mail was hosted on the down VPS, so they never got it. Man, they were angry.
Dealing with stuff like this is the hardest part, I think, of running a low-cost (and thus low support expectation) VPS service. (one of the advantages, I think, of a high-touch service is that if you are paying me to manage your VPS, well, if it gets compromised, well, first it's my job to see to it that it doesn't get compromised, but if it does, it becomes my job to clean up the mess, bring up a new host, restore from pre-compromise backups, etc... - you get to be the hero rather than the bad guy.)
Key here, I think, is setting expectations; setting expectations is harder than you think, though. I've got a bunch of random copy on my front page... which is probably the wrong way to do it. But the customer needs to know ahead of time what is going to happen if they start spewing spam or participating in a DoS... and the customer needs to understand that they are going to get shut down even if it was an accident.
This problem is made oh so much worse by this massive influx of developers without sysadmin skills who should properly be on a more managed (or PAAS) solution who are moving to VPSs because they are so cheap.
>Isn't this why most competent hosts request/require a secondary contact e-mail, not hosted by them?
Enforcing that requirement would be... non-trivial, though. I mean, I could look at the MX records for the domain in question, but those quite often don't point at the final delivery servers.
Still, I should at least make that a rule, even if I don't enforce it.
edit: Also note, even when you contact folks successfully, sometimes they refuse to deal with it, as, well, properly dealing with a compromise involves formatting and re-installing, then restoring from a backup taken before the thing was compromised. Most people at this level are unwilling to go through the effort, meaning that they will remain compromised for as long as you leave them on your system.
Often I'll setup a new domain for them, with the old domain read-only, so they can pick through the data. More than half the time, within 24 hours, they are compromised again.
>Is that because they don't upgrade their software in response to being compromised?
Recovering from a competent compromise is... more complex than that. You pretty much have gotta read all the code you move from the old (compromised) system to the new system. As part of that, you want to minimize the code you move from the compromised to the new system; so you should re-install as much as you can from scratch, then move over your custom stuff, one file at a time, after reading it carefully.
If you have a good pre-compromise backup, and you have some idea of where the hole was and that hole was in one of the packages/support libraries, and that hole has been patched, then you can restore and upgrade.
As I'm running a no-support service, I'm not digging in deeply enough to tell you exactly what happened, but I believe that in most cases, users are just copying over their documentroot wholesale, and at best upgrading over the top of the (possibly compromised) copy, which quite often won't help you.
> You pretty much have gotta read
all the code you move from the old
(compromised) system to the new
system.
If they don't have a know-good local copy of the code to upgrade the stock components of and redeploy on a fresh VM that just sounds like asking for trouble.
What do you do with such customers? If you "fire" the repeat offenders would you say it's been worth the forum backlash from them (which this story seems like an attempt at so far)?
Yes, I fire them. The form backlash varies a lot depending on the customer and how I handle it. I'm rather smaller than Digital Ocean (I recently dropped below 2000 customers) so I have something of a more personal touch, which usually helps more than it ought to help.
I also strongly select for customers who have some sysadmin experience (e.g. I turned someone away this morning who wanted a VPS but didn't want to authenticate with an OpenSSH public key) which helps a lot (but also vastly decreases my customer base.)
Worst case is when someone knowledgeable setup someone else on my service, then broke off contact; I'm now expected to step in and essentially be their sysadmin for $12/month. - That's the problem with unsophisticated users in a unsupported environment; they don't know enough to know if the problem is a hardware problem (which really is my responsibility) or a configuration on their own VPS.
but yeah, occasionally I get someone really, really angry. It's no fun.
My least favorite part of the job is firing customers who are the /target/ of DDoS attacks. I mean, if the attack is smaller than my pipe, I can tolerate it, but I've had customers hit with 10 gigabit+ attacks. Few providers can deal with that, so you are forced to get rid of the customer; It's really sad and messed up, because sometimes it's not even the customer's fault. Someone on the internet who controls a botnet doesn't like them. It's not fair that you finish the job, but there often isn't a whole lot of choice in the matter.
I have a whole lot less sympathy for people who allow their domains to be compromised and used in those attacks against other people.
The OP has deleted two posts so far in this thread, both saying that DO found him to be engaging in a DDOS. This "authorization" could disappear soon as well.
Certainly, if you would like to draft up a quick letter absolving us from our privacy policy in regards to your account, scan it, and email it to contact @ do then we can provide the full details here.
A signed letter allowing us to discuss your private account details that would violate our 'Privacy Policy' and reveal the factors that went into making the decision.
People keep asking the OP about his apps, and their answer is very a very vague "the account has 2 droplets which serves 2 mobile apps and does crawling 20-30 sites hourly".
My question is what do the apps do? That could easily be where the violation of the TOS is happening. Child porn? Fraud?
The lack of clarification is making the OP seem shady. Whats to hide?
I run a site on DO for about 2 months - last week I got a similar mail to "verify" my account. Luckily they had not taken down my droplet. I am willing to give them the benefit of doubt here - but I am preparing my Plan B right now in case they decide to "verify" me again and take my server down.
We really need to hear from DO to hear the other side of the story though.
"Please Provide us with the following:
...
1. Your public Twitter handle
2. Your blog
3. Your company or personal website
4. Your public Facebook profile"
That's roughly the procedure I use to make sure people aren't slimeballs when I rent my house out to them on AirBNB. I imagine it would also be very useful for determining that someone's an actual human.
I guess I'm a spambot.
I have no twitter handle, no personal blog, my company website is a large corporate site with no mention of me, and my facebook profile is completely barren.
This type of requirement is a solid reason for me to avoid DO rather than the non-specific issue that the OP ran into. I don't use twitter or facebook, and the reason that I've considered DO was to host a blog and company website. So if they were to bring me down for unknown reasons, I wouldn't be able to pass their extraneous id requirements. As a result, they are off of my consideration list. From some other comments in this thread, it sounds like Linode is the better way to go.
It's irrelevant, I suppose, but I'm sorta curious how big this database was (or, more specifically, how much traffic he was generating to Dropbox).
There was also no mention in this article (except in one of the screenshots) about whatever -- "[a] UDP flood or so" "at the beginning of this month".
It would certainly suck to have your account locked out when you're serving production sites but it sounds like we haven't heard all of the story (we obviously haven't heard DO's, of course).
What about the crawling? How much traffic does that generate? I'm just wondering if you are hitting one or more of the sites very hard and they complained to DO about it.
10MB once no, but lets say hypothetically, if the dropbox api returned an error or wasn't available which denied your request but your script didn't account for it and it constantly kept making the request over and over and over. The 10mb would soon add up.
Also, are you sure it wasn't something else, not the dropbox db script which could have caused problems?
I seriously doubt it, there is almost certainly something else going on. Does the dropbox api even use udp at all? A udp flood has little to do with the size of a file transfer, and everything to do with the rate of udp packets per second. Generally to be identified as a "flood" your server would have to be doing something very poorly coded (best case) or downright malicious. Did you check where this udp traffic was being sent to (destination ip addresses/ports)? Did you compare the dates/times of the "flood" to when your scripts/jobs were running? Did you check your server for any suspicious activity or indications of compromise? A little bit of critical thinking can go a long way here.
Dumping your DB daily to dropbox doesnt sound like a production solution. It looks like you violated their TOC somehow but dont state what you did.
This honestly sounds like growing pains, now that we are "taking off" we need to make sure users arent abusing our service. Lets do a user verification process for certain instances that are flagged for whatever reason.
I have experienced similar verify account problems with Pay Pal/eBay. While its a burden on people who are doing legitimate business, if it keeps spammers/scammers off of a service then it is worth it.
First, a person with such poor grammar has no business criticizing another person's communication skills.
Second, having worked for a hosting company, I suspect the OP violated the TOS and wants them to articulate why he was suspended so that he can try to argue his way back in. He was probably successful the first time getting them to reactivate his account after promising not to use his "dropbox script" again. But that was probably a convenient lie to talk support into thinking the problem would not reoccur. When he violated the TOS again, the automatic safeguards kicked in again. DO has a good reason to believe it will happen every time.
Hosting companies do not suspend accounts for no reason. Something he was doing was negatively impacting other customers. I suspect he knows exactly what that is.
Hosting companies have zero responsibility to tell customers what their scripts are up to. It's the customer's responsibility to make sure that they aren't violating the rules and hurting other customers.
There are many illegal things you can do with a web server, and this puts the hosting company at risk
1- Host/distribute copyrighted material
2- Run a torrent client and distribute copyrighted material
3- DOS attacks (especially when you can create several droplets for a limited period of time)
The hosting provider is like a bank, and has the right to ask you where you got that information from; and for what you are using it (though a bank doesn't ask you what you use your money for)
I think it's acceptable that the hosting provider ask for your activity and you provide a response for that (which you didn't).
Situations like this will keep getting worse and worse with our current attitude towards personal data on cloud. My personal thought on situations like this has always been that locking somebody's data is both unethical and should be considered a crime by law. Service providers have full rights to refuse providing servicing to somebody but they don't have a right to seize somebody's personal data.
On a side note, I looked through that database dumper script and isn't it just a giant hassle to script non-interactive authentications with OAuth? Hope I never have to do that again.
Either way, this is NOT how you deal with a customer who supposedly violated terms and conditions. That email response was atrocious. Arrogant and bullying. No sensitivity to whatever plight he may have, or reprieve offered. No 'we'll call you directly'. Nothing! I am about to make a BIG decision for my company and DO was in the discussion. It is not, not! Companies like this irritate the heck out of me and have no place doing business on a large scale. Keep up the attitude boys! ;)
119 comments
[ 3.0 ms ] story [ 181 ms ] threadI had to go through their 'verification procedure,' as well. I thought it was odd but if it keeps prices low due to fraud, I really don't mind.
I said that IS the point.
> he should have included Digital Ocean's side
OP claims they didn't give any
I'm wondering now if dropbox reported them for abuse and DO suspects that the account was compromised and is being used maliciously...?
Edit: Looking at their backup script, I don't see any error condition or exception handling when the file upload is posted via the dropbox client.
It might be that dropbox triggered some kind of abuse/rate limiting with a message to that extent. Without catching that scenario, the cron job would simply keep running... could be a situation where the dropbox user didn't know they were being abusive, were reported by dropbox to DO for abuse, and here we are.
Did you find out what the breach of terms of service they spoke about was? Unless I missed it the article...
How many times do you contact support? For me to contact support, there has to be a problem with the actual service itself. That is, it's an obvious problem that is outside of my control. I have been running VM's with quite a few providers, certainly all the big names. I almost never have to contact support (I would have to think pretty hard about a time that I have done this.) These services are usually quite solid and if there is a problem, usually there is some sort of status which lets you know there is a problem so that you don't have to ask.
For me, having to contact support even once would be a big problem. Having to contact support often enough that I have noticed that support was once good but then turned bad would have been enough for me to have long since moved on.
If the author was breaking the AUP, I don't feel very sorry for him. If someone's doing something that legitimately violates acceptable use, they probably should be shut down without prior notification.
I realize you sound angry because you were asked to verify your account, but this reads vindictive and makes me think that the same behavior that caused their abuse department to flag your account is what's responsible for this attack blog post.
Unless you provide that information this post comes across as a rant.
While I can't say that they're TOS violation claim is bogus, I also can't say it's not bogus, because the author didn't publish (or, apparently, ask) for that information.
Great, you've got a large number of users, and you used Digital Ocean to host your backend. Why are they saying you're in violation of the TOS?
Ask the follow up, figure out why they're not pleased with you, and don't try to appeal to emotion when you have no substantiating information.
As a side note, I have no horse in this race. I don't use Digital Ocean, and I have no affiliation with them. I just don't see the author's claim as being 100% put together and honest.
What if a site was compromised? What if an employee typoed the IP of an abuser? What if, what if?
Their "fullstop" replies are not professional.
And if you did have some automated system and various criteria for detecting likely abuses, if someone got shut down under those terms, would you tell them exactly where the line was? Or would you keep your detection methods to yourself?
"Your site has been turned off due to hosting child porn, in contravention of local/international law XXXXXX, per complaint YYYYYY".
Then they can at least dialogue.
Worst case scenario: the VPS is shut off automatically, I get a semi-urgent notice, and I reply to the customer with further details as soon as possible. And they would get a message letting them as much.
http://i.imgur.com/1pxIxiN.png
Just because the blog post doesn't say why doesn't mean he doesn't know why. Only that he's selectively sharing the information.
I still think it's important to reiterate as much as possible why a service was shut off, both for a customer understanding viewpoint, AND from a "cover your ass" viewpoint.
It can't hurt to say "As stated in the past email, your server was shut off due to YYYYY", rather than just "Game over, man. Stop emailing us." equivalents.
This is nitpicking but in the end, you (DO) have a smiling icon because you expect you support group smiles. If your support group is a machine, well then... use this one.
I used the Live image of my droplet to mount the disk, copy the databases, and websites. Once I was satisfied I got everything I destroyed the droplet.
Before you destroy your droplet, request this!
[Lish]: (https://library.linode.com/using-lish-the-linode-shell)?
> DigitalOcean reserves the right to modify the Terms of Service without notice.
Well... that more or less invalidates the sanctity of the terms.
> DigitalOcean also reserves the right to terminate a customers account if they are targeted by malicious activity from other parties.
What is the justification for that? Simply being the target of malicious activity means that your account can be terminated? It's not as though one is responsible for malicious activity directed against them.
> DIGITALOCEAN DISCLAIMS ALL WARRANTIES, EXPRESS, IMPLIED, OR STATUTORY, REGARDING THE SERVICES PROVIDED HEREUNDER, INCLUDING ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE ...
I see this a lot with software, but doesn't this mean that DigitalOcean has no actual obligation to provide the cloud or VPS services they advertise? I didn't see any such clause in the EC2 TOS, but I did not look very hard.
Is this the usual sort of agreement for such a service?
See section 10 of http://aws.amazon.com/agreement/
Frankly, none of those terms seems atypical. Amazon does have SLAs[1], but they do not apply if your account has been suspended. Reasons for suspension can include your use of the service adversely affecting any other AWS customer. They may also terminate you if providing services creates a substantial technical burden. Both of those can be triggered by you being "targeted by malicious activity from other other parties".
[1] http://aws.amazon.com/ec2-sla/
You'll find the plurality (if not majority) of ToS's out there include such a statement.
>What is the justification for that? Simply being the target of malicious activity means that your account can be terminated? It's not as though one is responsible for malicious activity directed against them.
They don't want to deal with the networking implications of a large scale DDoS. They're providing low priced VPS services, not high bandwidth, high load custom racks. Also, they "reserve the right" not the "always exercise the right". DO likely won't kick you for a small DoS/DDoS but sustained attacks take much more manpower to deal with than you're paying them for.
>I see this a lot with software, but doesn't this mean that DigitalOcean has no actual obligation to provide the cloud or VPS services they advertise?
No, it means you can't sue DO because your VPS went down and you lost 10K is sales because customers couldn't access your storefront. It's an indemnity clause.
>Is this the usual sort of agreement for such a service?
Yes, unless you're paying a good deal of money for something like a Colo with 1hour SLA or an AWS instance with a high SLA.
IANAL, but I don't think that warranty disclaimer is relevant here. I don't think it means much at all actually. As a business you can say you're not liable for anything bad that happens, but that doesn't necessarily make it true.
Most providers will shut you off if you're running a server that attracts a lot of DDOS attempts. Many providers explicitly disallow running irc servers up front to avoid this very scenario. If your site is constantly getting DDOSed for whatever reason that is going to have a big impact on the hoster's entire network and other servers they are hosting and while it may be through no fault of your own, almost all hosters are going to bounce you if it is a recurring problem.
Their response? They shut down the server because of "unusual traffic" coming from the server which wasn't even being used yet.. but there was "outbound traffic about 977.38 Mbps at its peak at around 2013-07-22 14:50:00UTC"
They need to fully verify accounts before letting anyone create a server. You shouldn't just take down a server out of nowhere, for several hours, after it's been running for weeks.
Does this not sound suspicious to you? Maybe the server was compromised. "Unused" servers usually don't generate 970 Mbps traffic randomly.
If the customer would authorize this disclosure then we can discuss it publicly.
Otherwise, as much as it pains us to get negative feedback, whether or not it is deserved, it is beyond our control as the privacy of our customers is the most important issue at stake in a public discussion.
As we (Blekko) get better at identifying and shutting down click fraud and other schemes to defraud advertisers a lot them end up being traced back to hosts in inexpensive hosting services. That is but one of many ways of having a machine on the Internet that generates cash for you. We ended up basically shutting out a couple of class Bs from an hosting service in the Ukraine for that reason, people can write us and get white-listed but in over 27,000 IP's not a single one was a legitimate user. Way too many people coming up with schemes to get some "free" extra cash.
I've been nothing but impressed with DO, right now there seems to be a misunderstanding and OP is not providing full details nor is he allowing DO to discuss what happened.
If he truly cared about others not using DO he'd allow it to be discussed, but it does seem like OP is hiding some fact...
So now I'm left with great service and product vs 1 guy who seems like he did something shady and got caught. Maybe it was an accident and maybe there is a way forward, but only if he brings to light the details of his actions.
Edit: Plus now you delete your post where you hint at your DDoS (by mistake or not) does not add up to your favor: http://i.imgur.com/1pxIxiN.png
One user sent me a bill for $10,000 after I cut 'em off for running a spam cannon, for lost business. I mean, I'd say 90% chance that they were just incompetent sysadmins trying to run wordpress and they got compromised, but ground truth was that they were sending out rather a lot of spam.
I sent a few notices before and after shutting them down, but their mail was hosted on the down VPS, so they never got it. Man, they were angry.
Dealing with stuff like this is the hardest part, I think, of running a low-cost (and thus low support expectation) VPS service. (one of the advantages, I think, of a high-touch service is that if you are paying me to manage your VPS, well, if it gets compromised, well, first it's my job to see to it that it doesn't get compromised, but if it does, it becomes my job to clean up the mess, bring up a new host, restore from pre-compromise backups, etc... - you get to be the hero rather than the bad guy.)
Key here, I think, is setting expectations; setting expectations is harder than you think, though. I've got a bunch of random copy on my front page... which is probably the wrong way to do it. But the customer needs to know ahead of time what is going to happen if they start spewing spam or participating in a DoS... and the customer needs to understand that they are going to get shut down even if it was an accident.
This problem is made oh so much worse by this massive influx of developers without sysadmin skills who should properly be on a more managed (or PAAS) solution who are moving to VPSs because they are so cheap.
Isn't this why most competent hosts request/require a secondary contact e-mail, not hosted by them?
Enforcing that requirement would be... non-trivial, though. I mean, I could look at the MX records for the domain in question, but those quite often don't point at the final delivery servers.
Still, I should at least make that a rule, even if I don't enforce it.
edit: Also note, even when you contact folks successfully, sometimes they refuse to deal with it, as, well, properly dealing with a compromise involves formatting and re-installing, then restoring from a backup taken before the thing was compromised. Most people at this level are unwilling to go through the effort, meaning that they will remain compromised for as long as you leave them on your system.
Often I'll setup a new domain for them, with the old domain read-only, so they can pick through the data. More than half the time, within 24 hours, they are compromised again.
Is that because they don't upgrade their software in response to being compromised?
Recovering from a competent compromise is... more complex than that. You pretty much have gotta read all the code you move from the old (compromised) system to the new system. As part of that, you want to minimize the code you move from the compromised to the new system; so you should re-install as much as you can from scratch, then move over your custom stuff, one file at a time, after reading it carefully.
If you have a good pre-compromise backup, and you have some idea of where the hole was and that hole was in one of the packages/support libraries, and that hole has been patched, then you can restore and upgrade.
As I'm running a no-support service, I'm not digging in deeply enough to tell you exactly what happened, but I believe that in most cases, users are just copying over their documentroot wholesale, and at best upgrading over the top of the (possibly compromised) copy, which quite often won't help you.
If they don't have a know-good local copy of the code to upgrade the stock components of and redeploy on a fresh VM that just sounds like asking for trouble.
What do you do with such customers? If you "fire" the repeat offenders would you say it's been worth the forum backlash from them (which this story seems like an attempt at so far)?
I also strongly select for customers who have some sysadmin experience (e.g. I turned someone away this morning who wanted a VPS but didn't want to authenticate with an OpenSSH public key) which helps a lot (but also vastly decreases my customer base.)
Worst case is when someone knowledgeable setup someone else on my service, then broke off contact; I'm now expected to step in and essentially be their sysadmin for $12/month. - That's the problem with unsophisticated users in a unsupported environment; they don't know enough to know if the problem is a hardware problem (which really is my responsibility) or a configuration on their own VPS.
but yeah, occasionally I get someone really, really angry. It's no fun.
My least favorite part of the job is firing customers who are the /target/ of DDoS attacks. I mean, if the attack is smaller than my pipe, I can tolerate it, but I've had customers hit with 10 gigabit+ attacks. Few providers can deal with that, so you are forced to get rid of the customer; It's really sad and messed up, because sometimes it's not even the customer's fault. Someone on the internet who controls a botnet doesn't like them. It's not fair that you finish the job, but there often isn't a whole lot of choice in the matter.
I have a whole lot less sympathy for people who allow their domains to be compromised and used in those attacks against other people.
"You might not get notified when shit hits the fan" seems like some natural enforcement :-P
edit: stop deleting your posts, you douche.
DigitalOcean is asking you for permission to discuss your case further publicly. Will you allow it or not?
Now, can a lawyer come in and discuss whether or not this can serve as authorization such that DO can't be sued?
My question is what do the apps do? That could easily be where the violation of the TOS is happening. Child porn? Fraud?
The lack of clarification is making the OP seem shady. Whats to hide?
We really need to hear from DO to hear the other side of the story though.
What? Is this a common procedure now?
It's so weird to see things like 'Twitter account' and 'Facebook account' being 'assumed'.
They are just two private companies why should we (directly/indirectly) force everyone to have one?
Email I'm fine with, it's a open technology.
It's only a matter of time until we turn Twitter and Facebook into more open technologies like email. I sure hope to see that happen.
There was also no mention in this article (except in one of the screenshots) about whatever -- "[a] UDP flood or so" "at the beginning of this month".
It would certainly suck to have your account locked out when you're serving production sites but it sounds like we haven't heard all of the story (we obviously haven't heard DO's, of course).
Also, are you sure it wasn't something else, not the dropbox db script which could have caused problems?
This honestly sounds like growing pains, now that we are "taking off" we need to make sure users arent abusing our service. Lets do a user verification process for certain instances that are flagged for whatever reason.
I have experienced similar verify account problems with Pay Pal/eBay. While its a burden on people who are doing legitimate business, if it keeps spammers/scammers off of a service then it is worth it.
Second, having worked for a hosting company, I suspect the OP violated the TOS and wants them to articulate why he was suspended so that he can try to argue his way back in. He was probably successful the first time getting them to reactivate his account after promising not to use his "dropbox script" again. But that was probably a convenient lie to talk support into thinking the problem would not reoccur. When he violated the TOS again, the automatic safeguards kicked in again. DO has a good reason to believe it will happen every time.
Hosting companies do not suspend accounts for no reason. Something he was doing was negatively impacting other customers. I suspect he knows exactly what that is.
Hosting companies have zero responsibility to tell customers what their scripts are up to. It's the customer's responsibility to make sure that they aren't violating the rules and hurting other customers.
Where are the OP's logs?
edit: Post disappeared. Was https://gist.github.com/ceejayoz/75dd46bf5ae957f9c4fd
1- Host/distribute copyrighted material
2- Run a torrent client and distribute copyrighted material
3- DOS attacks (especially when you can create several droplets for a limited period of time)
The hosting provider is like a bank, and has the right to ask you where you got that information from; and for what you are using it (though a bank doesn't ask you what you use your money for)
I think it's acceptable that the hosting provider ask for your activity and you provide a response for that (which you didn't).