Ask HN: My VPS got hacked and now I'm facing a massive bill. What can I do?
I've got a VPS which I use for small programming projects and college assignments. Two weeks ago I received an e-mail from my provider, stating that "your VPS has been transmitting a lot of outgoing traffic which results in a very large traffic usage bill". In September on my 500 GB data-limit VPS, it had been transmitting 27 TB of data traffic. This resulted in a € 3300 extra charge on my € 15 VPS. I'm expecting a similar bill for this month.
Of course I immediately shut down my VPS after the notice two weeks ago, but by then it had been using these amounts of traffic for a month and a half.
What are my options here? I can't afford to pay > € 5000 unfortunately. Does anyone have similar experiences?
82 comments
[ 3.1 ms ] story [ 152 ms ] threadYou might also offer to suggest writing up a post mortem for them, that they can provide to their customers as a lesson/tutorial on how to protect a VPS.
Finally, you can suggest that they might want to implement (and perhaps help them implement it) some kind of warning system, i.e., if a VPS suddenly begins using exorbitant amounts of bandwidth, and far more bandwidth than it ever has before, they really should email/text the owner an alert within 24 hours — not let it go on for 6 weeks. I'm surprised that they don't cap/throttle the bandwidth once you go over your plan's limit, to go along with sending you alerts. It borders on negligence on their part that they don't already have such a system in place.
I would suppose your first and best resort is to consult your lawyer, advocate, solicitor, barrister, Anwalt. I wonder what your relevant legal jurisdiction is.
I wonder whether it would help if you can account for your own whereabouts and your own usage of endpoint data services. I wonder if your method of payment to your VPS provider is mediated by a financial service that can help you dispute the bill.
I am not a lawyer.
Unfortunately, I can't think of anything else. I wish it was realistic to tell you to go to the police.
Also, if you would give your email, I would definitely consider sending a donation through paypal... Hopefully other readers here will do the same.
(Examples: police reports make CC disputes and legal declarations much easier and more likely to be given weight as other than self-serving explanations of a deadbeat. It may also trigger insurance policies either for you or for the VPS company.)
but that has nothing to do with helping HIM...
1) In dealing with the company that is billing you, a police report sends a strong, costly signal that "This is not a routine commercial transaction! A crime has been committed!" which will throw an exception in the billing process that will get caught elsewhere in the company. The accounts receivable department is scored on collecting revenue. That is, to a first approximation, the only thing they care about. The legal department, anti-abuse department, etc etc, is not scored on revenue impact.
You're much more likely to have a productive conversation with them that involves waiving the $4k than you will with Accounts Receivable. AR's calculation goes something like "We'd prefer getting $4k over getting $0." The rest of the company thinks "We'd prefer writing off an uncollected invoice -- costs us $0 -- versus spending hundreds of dollars an hour dealing with the legal process for an unbounded amount of time."
2) Assuming that European credit card providers work on the same heuristics as US ones, "I filed a police report" means their response to your chargeback will virtually invariably be "Wait, an actual criminal case? Eff that, sorry merchant, we do not get paid to litigate on your behalf. Best of luck figuring this out after we give the customer their money back." (Conversely they use police reports as a bar for treating claims of fraud seriously, for example, in the case of "family fraud." If you ever say, for example, "My wife bought that but I didn't let her?" the very next question will be "Have you filed a police report against her for stealing from you?" and if you haven't the very next thing they'll say will be "Well, work it out with her then -- the charge stands. Thxmuchbye!")
3) "Theft" is such a wonderfully useful word for dealing with insurance companies. (HNers might say "No theft happened! It was actually..." No, guys, really, theft happened.) It may be covered under your homeowner's insurance -- depends on the particulars of your policy -- and would almost certainly be covered if you had business insurance. It also may be covered under the VPS provider's insurance, in which case they may decide that rather than trying to ring water from a stone (you) that filing a claim is the quicker way to get paid. (I rather doubt, as a businessman with an insurance policy, that they'd jeopardize their future insurability over a $4k claim, because that is tiddliwinks, but risk management is exactly why that policy exists.)
If they insist for you to pay: simply don't. State the truth: You can't afford it. Tell them the only way they will see this money is by taking legal action against you and even in that case you won't be able to comply - as you don't have the money.
Hope it helps :(
2. Report it to the VPS provider. Explain that you've reported it to the police. Ask for their cooperation in investigating the problem.
You do not have to pay. If they try to force you to pay, depending on your country, you'll probably end up in small claims court where you'll find judges are very reasonable people who usually side with the little guy. (IANAL)
If it's just a garden variety dispute with a company, then it's more likely to be a) a civil matter, and b) subject to interpretation.
In my example, it appeared that a roommate unknowingly permitted a friend of theirs that I never met then or since to run up big charges at billable services on the apartment telephone line that was in my name. (I understand what my mistake was and because it was so long ago it's not a sore issue.)
You're right about reporting a crime though even if the police don't take it seriously. A crime ref number goes a long way on its own.
Edit: we had to move our kit sharpish though as the company exercised their right to throw it on the street within 24 hours.
I used that intentionally to prevent the association of HN with the other form of the word. It is polite IMHO.
> Am I right in assuming that "pr0n" is some kind of 1337, 80s hackerish misspelling of "..."
... with the ellipsis substituted to accomodate your preference. ;-)
Including ALL customer support interaction , police logs, diary , journal, etc.
Judges will show favor if you have a paper trail.
The reason for reporting is that it is the first step in initiating most civil means of compensation (insurance, company cancelling the charge, etc.).
He wanted to know so maybe I'd give the password so "he wouldn't be ripped off, too". I'll give him credit for having the balls to ask. But (as I later found out) he knew he was buying an "activation locked" iPhone.
I said a few things like how do I know you're not the thief, etc. He pointed me to the eBay listing which, sure enough even had the IMEI with two digits transposed (plausible deniability, I'm sure).
He contacted the seller and said "Tell me why I shouldn't give this phone to its rightful owner and then file a fraud complaint with eBay and get a refund?" Unsurprisingly the seller offered to take the phone back. So he sent it (I didn't care, and while I knew the insurance company was about as unlikely to care as the police were, I didn't want to do anything that might trigger insurance fraud questions - "This phone was reported stolen, unlocked using your credentials. Explain.") and got me the seller's home address.
I contacted the insurance company. They didn't care, just told me to file a police report and send them the case number.
Looked at the seller's profile, quite possibly the sketchiest thing I've ever seen.
Bunches of phones, all "activation locked, no charger". Tablets, no charger. Laptops, no accessories or charger. At least 50 or so.
Gave that info to my local PD. Their response, "We won't investigate. He probably bought it from someone and is selling them. Could have gone through a few people first."
I didn't want the original thief caught but this guy was openly selling stolen gear. Hell, the message on my GFs phone said "I don't care about stealing the phone. Will trade cash for it.".
They weren't interested. Bear in mind, this isn't someone complaining about their car at the impound lot in LA, a la Big Lebowski, this is town of about 40,000 with a well-funded PD (I work for Fire in the same town).
The urge to drive to this jokers house in the middle of the night and pour sugar in his gas tank was one I avoided, but only just.
It was a nerve recking couple of days but I contacted AWS support and they were extremely good. They helped me secure my machine and then cancelled the 1.4K payment they were going to take from my account.
In all the whole process took 2.5 weeks and I only had to pay $15 for the I/O requests.
The best thing I can recommend is to talk to your host and tell them honestly you can't pay that much and you weren't the cause of the charges either.
I had a reserved instance for 12 months, forgot to renew it, and on the 13th month (when it was on-demand) the usage creeped over my cap and I started getting alarms allowing me to kill the instance, renew my reserved, and restart it. Saved me at least $10.
On a related topic, I wish VPS providers allowed you to pre-pay. With Microsoft's Azure I have an MSDN Ultimate account, which has $150 of pre-paid credit on Azure. When you go over the $150 they just shut your stuff down rather than charging you (in fact I don't have a CC on there at all). They don't even offer this kind of service to non-MSDN subscriptions which sucks, I'd love to just pre-pay $50/month to them and have everything shut off when I exceed it (so it becomes a "no risk" playground).
> Can I pay in advance?
> We do not currently accept pre-payment, but you can add additional credit to your account at anytime by sending in a PayPal payment.
https://www.digitalocean.com/help/pricing-and-billing/
Which means your case is probably covered by consumer protection rules when it comes to informing you about data usage, and I seriously doubt a VPS provider has covered their ass as well as mobile providers tend to do.
If you expect to get DDoSed, buy protection or go with OVH.
This is one of many, many, many reasons why we don't generally do cost-based pricing and, when we do do cost-based pricing, the markup is absolutely phenomenal. It has to include risk premiums. As long as it do include risk premiums, you don't have to sweat the small stuff like e.g. an uncollectable $4k invoice. (n.b. Small stuff! $4k hiccups are utterly routine events and largely dealt with by processes rather than by treating them as sudden emergencies, even if they feel like that to natural humans.)
But, if you have a huge spike that wasn't really your fault, it doesn't cost them any more to write that off than it does the bandwidth that is consumed by a DDoS attack that is mitigated by a firewall.
When I worked at Rackspace, we hosted a very popular flash cartoon for one month after yahoo kicked them off the $5 hosting plan for pushing god knows how much bandwidth. They basically saturated a gigabit network port from a single server, and we sent them a bill for like a gajillion dollars. They went to another hosting company, of course, and I think got the bill down to something they managed to pay off.
Someone obviously had porsche-eyes, I thought it was kind of a shitty thing to do to the guys, who came to the office to make the voices of their characters for us and stuff.
The same thing happened to me with amazon. Amazon pid for it. It's highly unreasonable in my opinion to ask the customer of a VPS to pay for damages caused by a malicious attacker. It's tantamount to a landlord expecting you to pay after an arsonist comes along and burns down your apartment, just because you happened to be renting it at the time.
- a landlord* expecting you to pay after a squatter came along and opened a faucet in the basement (where OP rarely goes) to fill its own super tanker
* or the water company ?
That brings me to my point. How did the hack occur? When you get a VPS you are fully responsible for what goes on in there. It is your responsibility to secure it and keep it updated. It's not the provider's fault you did not apply the latest security updates. It's not the provider's fault your Java application was using outdated and vulnerable libraries nor is it their fault you didn't set a CAPTCHA in front of your submission forms. Either hire a competent sysadmin if you can't take care of that yourself or find a provider that offeres managed hosting instead of a VPS, as that's what you'd most likely need.
There are some cases where it's the provider's fault such as the Linode BitCoin hack a few years back but mostly it's just poor server maintenance
It's hardly worth hiring a sysadmin for (I find that suggestion laughable, to be frank). Managed hosting doesn't allow you to do much else besides hosting a website in PHP, which is not enough for plenty of use-cases, including OPs.
Do tell, where do I get one of those? Cheapest I know of is $60 ($5 a month).
Other than that, I'm sure you can find something on webhostingtalk forums.
Malicious entities runs 24/7 scans towards indexed URL's attempting to exploit various vulnerabilities, and many of the vulnerabilities allows remote code execution, upload of php files etc. This can be used to upload malicious code, simple php-webshells, and then your VPS is suddenly a part of a DDoS/Scanning network.
Exploited Wordpress sites are a problem, Zeus/Zbot-Trojan is often seen downloading updates/configs from these, and they are also often used to redirect users to Exploit Kits.
Not installed either yet (LMD could really use some .deb packages) but could be a useful alternative to Tripwire