Hacker Accessed AWS for $50k+ – AWS Ignoring Me
My business has used AWS for around 3 years and our normal usage is $1k per month in EC2 and S3. In early March a hacker accessed our AWS account through my login via an IP address in Austria (I'm in Austin, TX). They spun up 3 large instances of EC2 which began charging us $1k-$2k per day.
In mid-April, while reviewing our books for the month of March, I saw a $26k charge from AWS. I thought it was a typo as $2.6k and asked the accountant. She stated that was the correct amount. I immediately got my dev team involved and we discovered the 3 instances to which we did not have any access to and stopped them immediately.
I opened a support case immediately which somehow got posted twice. Because the case was posted twice, the support team marked both cases as duplicates. I reopened one of the cases, it was resolved again as a duplicate. This has now happened several times.
I Googled around looking for a way to escalate this matter and found the following emails and cc'ed them on May 5th with an urgent plea via the original support case thread with another summary of the issue and links to my cases with my phone number to no avail. ams-csdm@amazon.com ams-opsmanager@amazon.com, ams-director@amazon.com, ams-vp@amazon.com
That email was ignored and I'm not sure where I can turn to next. I've tweeted about this and tagged AWS here - https://twitter.com/csakon/status/1391873413107617799?s=20
I'm not sure where to go next, can anyone give me any advice?
54 comments
[ 1094 ms ] story [ 1863 ms ] threadBusiness and enterprise tier are 24 hours, and will be dealt with by more experienced technicians within an hour.
Anyway, it's important to frame what happened correctly: the security of someone on your team was sloppy, and most likely a bot was able to get an access key or access to one of your accounts, spin up crypto miners on EC2s and now you're responsible for the bill. If it hadn't been that, it'd have been ransomware, you probably got lucky.
Now, to see if your situation can be improved: Put up some dollars and get business support. Make a clear and polite case, from the beginning. Ask for a refund but you don't have grounds to demand it; if they issue one, it's a gesture of good will. They probably will issue one if you haven't had to ask for that before, but it reflects badly on everybody that cryptominers weren't caught for two months.
And before you create that ticket, make some billing alerts so you can show AWS support that this won't happen again.
I was at fault for the double post of the support case, but that was a simple error on my part due to not thinking the first went through.
Once access was made, we were completely unaware of their existence until I saw the charges and asked our devs about it. They said they didn't have any knowledge or access to these new instances.
I appreciate the advice, will upgrade the support and try again.
It’s not on AWS to flag anything. They give you the tools to comprehensively monitor your account, but you choose not to use them. Cloudwatch is also “standard”. Datadog has a free tier, I’d suggest checking that out because you don’t seem to have any infrastructure monitoring?
I’d honestly hire some people that have experience in this area, because your “dev team” sounds clueless.
I myself have never created an instance, I set up the account then gave access to the devs. I only log into the account to provide new access to devs that need it and none of them are full time.
Ultimately the responsibility lies with me, but I would disagree that my dev team is clueless. Rather they're working on development, not watching what the servers are doing on a daily basis so I think that's a bit unfair.
I assume you revoke access later, but I doubt you audit anything that they may have done (like create keys that outlive their access) in the account or that any of it is version controlled or traceable.
And you’re surprised you’re in this situation?
And fair enough, you pay your contractors to do a specific job. None of them are going to point out that the way you’re managing your infrastructure is pretty slow and inefficient, or that perhaps there’s a better way to do any of what you’re doing on AWS that is cheaper, faster, more secure and that might give you a far quicker iteration time with the added advantage that you won’t fall apart with a surprise bill like this. They, after all, are working on “development”.
So yes, it was bad form for OP to not establish billing alerts, but you can't in any way say that AWS is transparent about most aspects of their service tiers.
Anyway I was talking more about cloudwatch monitoring, including tracing all API calls. The billing is secondary: someone has been in his account and might have exfiltrated everything.
AWS can certainly do a better job in telling me how to optimise my AWS hardware and expenses
In OPs case, if you search in Google, you will find this AWS hacking happen a lot with many users including me and AWS support was extremely kind to me to give additional credits to offset that expense.
It would be trivial for AWS to force you to setup spending limits when setting up an account, one for an alert, one for a hard lock.
AWS billing is terrible too, even now I'm getting charged a few dollars on my personal account for who knows what, I've cancelled everything I can find. It's like whack-a-mole everytime you spin something up.
In the end that's basically theft by AWS but you can't make too much noise or your account gets cancelled.
Why? Because they serve just about every use case and scenario. I've myself spun up $5k/day resources on accounts that did four magnitudes less / day before that. There are some limits by default which you can get raised, but they're not going to prevent a $50k bill - at best, they're here to prevent a $500k one.
Anyway I agree they could make it clearer that you have to do all this crap yourself but it makes for a poor sales pitch.
Be a little patient, but don't try to superescalate by going through emails and what not. Just file a business ticket. If this needs escalation, they will do so.
CloudWatch is standard AWS, or what do you mean?
Amazon gives you all the tools to monitor your usage and management of the cloud. You did not use those tools, had sloppy security and now it is Amazon's fault?
Take it as an expensive lesson learned. And get someone who knows what they are doing in AWS to administer it. Maybe your company was pennywise pound foolish by not having someone who knew AWS?
This sounds like a cautionary tale. I have spending alarms on my personal account for this very reason, I'll know within 5-10 minutes if my monthly spend is going to break $50 because I've set up my alarms.
Your other option is to start a Cloudtrail and alarm on foreign IPs that are logging in, new IAM users and keys being created and changes to any alarms you have in place to check for this stuff. It won't necessarily stop it, but you'll be able to react a lot faster.
I'm not that worried though.
But what would be best practices for billing alarms? I have used them in the past when I used a bit more of AWS, but I don't think I ever got one (which is good).
But it happens to me that I miss emails that I would have liked to read in time for weeks. That could happen with a billing alarm, too.
Maybe you could forward it to SNS with SMS delivery. But as a matter of fact SMS is one of the few services (if not the only one) with a spending limit. If that is reached you silently won't get SMSes anymore, I have experienced that.
My average ticket response with paid AWS Developer is probably 72+ hours, and that's just for the initial triage what's up query.
Unless you are either a seasoned Linux developer with many years of Linux internals experience, or you have an enterprise level account, all of the lower class AWS support rungs are largely meaningless.
Billing team is separate anyway. You get a better "treatment" if you're paying (as in, they will look at your case closer), but you have access to the same service regardless.
We had a few training and introduction sessions with their people and it usually ended with us tuning out/joking about their useless slides and presentations (with "inspiring" Jeff Bezos quotes and brags about the number of packages ordered on Amazon.com that have 0 relevance for anything our company would like from AWS) in a private channel.
As much as I'd like to agree with you, AWS makes controlling this WAY too stupidly difficult.
Out here in the real world, many of us are part of startups that have 4 people and a dog. We wear many hats, and "AWS Billing Expert" is not one we have time for.
I actually grind my teeth and recommend Azure to most small companies on this alone. GCP is nice, but I simply won't recommend them due to Google.
However, sometimes AWS has "that service" that you really need. And I just have to caution people that AWS will not protect you.
AWS could solve this. Simply allow people to opt into a hard stop on spending--some of us would rather be down than overspend. Let us make that choice.
The fact that AWS absolutely refuses to solve this speaks volumes.
Hard stop on spending would delete all your data and would not cover things which are billed e.g. monthly. There are AWS budget actions which address some of the issues (e.g. can put a hard deny on any actions or stop your EC2 instances), admittedly a relatively new service.
I don't know what "generally" means here or what you're basing the claim on, but I'm with an organization that pays a lot of money to AWS and they regularly ghost us after giving a wrong or incomplete "works for me"-style answer.
I have no idea why we're paying.
They don't suck once they actually start doing something, but they do seem to have KPIs which incentivise wasting my time.
Whether the original security breach was the user's fault or not, Amazon Support dropped the ball here.
In my case we’ve got a miner on our jenkin for a day. I just call my aws sales rep and he get me a free lunch and a few credit to pay the business support for 1 month, then open the ticket through that business support. At the end of the week aws gave us extra credits around 10% of our yearly usage.
I don’t think they will waived all yours 26k. Thats your dev team fault and also your finance team or whoever that don’t watch the billing. But they can give you a lot of aws credit for many reason (promising startup, loyal customer, big company, etc)
As a busy founder, I'll just go with the providers that don't chain me to their dashboard/email instead of providing meaningful caps.
It reminds me of SELinux where the permissions are difficult enough to deal with that you can write an audit log while performing an action and simply enable all the permissions that were logged.
The second biggest problem is that runaway billing is far worse for small users than for large users and big tech only cares about other big users because that's where the money is. Everything revolves around catering to huge users who don't care if they need to hire a consultant to tame their AWS billing, so the smaller users and startups are left with systems that are far too complex to meet their needs.
I prefer the way Digital Ocean works, but there are some things you just can't do with them. For example, Lambdas and SES don't have good alternatives at DO.
I also like Cloudflare Workers since I find it significantly easier to reason about price in the context of cost per execution instead of the complex formula used for Lambdas, etc.. I think Cloudflare is in a very good position to claw market share from the big clouds, but their Workers Unbound is pretty much a copy of Lambdas, Functions, etc. in terms of pricing structure, so it looks like they might be starting to go after those fat egress charges that everyone else makes their money from.
Overall the best defense is defense in depth - use MFA for all human accounts, use IAM roles wherever possible, don't put stuff in public subnets, use restrictive firewall rules, follow least privilege principle, use secrets manager or similar services for storing credentials. You could write a book about it. Many people pretty much have.
Of course, if OP didn't have MFA / let their keys leak this may not have helped anyway if the hacker was able to just remove the limits.
In your organization, add a service control policy which denies access to services you don't use. This will prevent all member accounts from executing actions you don't want, including root users. You can also deny any action on any resource in region other than whatever you expect to use (with some exceptions due to legacy stuff).
i had set up some billing alerts on my AWS and it was just all terrible and sh*tty, clicking around in a million places to get not what i actually wanted.
i thought about building my own service to just let me know a daily total of my expected bill at the end of the month
then i'd add stuff to tell me about fast-increasing charges, as quickly as was necessary depending on the steepness of the charge rise curve.
then i found Billgist, which i tried for a bit -- it worked great, looks great, etc., so i'm not building my own. no connection to them.
https://www.billgist.com/
i used it at first just to see if it worked at all, then to see if it sucked (in which case I would prob try to build my own), and then ultimately to try to help me get comfortable with the idea that i probably wasn't going to wake up one beautiful Saturday morning to a $50k AWS bill.
that never worked -- that is, i never got comfortable with the idea that I would _not_ wake up to a $50k AWS bill one beautiful Saturday morning -- it just seems completely plausible, even likely.
so i shut down most of my aws stuff (i was always particularly worried about my Lambda stuff), moved some things to Digital Ocean, and i'm guessing i'll revisit AWS at some point when i reach some critical mass of:
one thing i learned is that AWS charges you for _everything_ -- including your single daily API call to figure out how much you're going to owe at the end of the month -- the fee for that call is 3 cents per call, or at least for the first call.tho you can log in thru the console and check this estimate for free.
one of the things you get charged for is 'Configs' -- pretty much any config setting you've customized in any way -- permissions, roles, tags (?), etc.
i understand the logic, but damn -- i'm trying to use AWS so i can get things done, not so i can worry about the costs of every. single. not-completely-optimized. miniscule. design decision. down to the penny. nickle and diming would be a luxury.
i can imagine my hypothetical company's AWS Cost Saving Specialist coming to me and saying, "I'm glad you've set up this incredibly fast and secure and resilient system, but....we need to save a few bucks, so....yeah, i'm gonna need you to come in tomorrow...."
i may have a former co-worker that works there - if you run out of options i'll try to ping someone in my chain, see if i can contact them.
Have you contacted the FBI and Europol? A police report is the first thing you need before any company starts taking you seriously about crime being committed on your billing accounts.
https://www.fbi.gov/investigate/cyber
https://www.europol.europa.eu/report-a-crime/report-cybercri...
It seems earning money from users' mistakes is part of their business model.
Yes, resource limits are not a silver bullet. Users will complain when their important service goes down because it would have gone slightly over budget during "normal" use. Implementing it in a reasonable way is not perfectly simple. Probably you want separate limits for network, storage, and processing as well as different ways to enforce the limit. Deleting all S3 data might not be what many users would want. They might still be willing to pay for keeping the existing data.
And obviously changing the limit must not be possible with the same credentials that allow you to use resources. Another fundamental challenge.
But with the size of AWS's business there is no excuse not to implement anything. They just value profit over customers.
(Sorry, not a reply to the original poster's question. I don't have anything significantly different from what has been mentioned by others.)