HN Explorer

Ask HN: Our AWS account got compromised after their outage

395 points by kinj28 ↗ HN
Could there be any link between the two events?

Here is what happened:

Some 600 instances were spawned within 3 hours before AWS flagged it off and sent us a health event. There were numerous domains verified and we could see SES quota increase request was made.

We are still investigating the vulnerability at our end. our initial suspect list has 2 suspects. api key or console access where MFA wasn’t enabled.

33 comments

[ 2.7 ms ] story [ 62.6 ms ] thread
bdcravens
Any chance you did something crazy while troubleshooting downtime (before you knew it was an AWS issue)? I've had to deal with a similar situation, and in my case, I was lazy and pushed a key to a public repo. (Not saying you are, just saying in my case it was a leaked API key)
klysm
Sounds like a coincidence to me
yfiapo
Highly likely to be coincidence. Typically an exposed access key. Exposed password for non-MFA protected console access happens but is less common.
ThreatSystems
Cloudtrail events should be able to demonstrate WHAT created the EC2s. Off the top of my head I think it's the runinstance event.
sousastep
couple folks on reddit said while they were refreshing during the outage, they were briefly logged in as a whole different user
TZubiri
A security incident like this would dwarf in comparision to partial unavailability of services.
itsnowandnever
i cant imagine it's related. if it is related, hello Bloomberg News or whoever will be reading this thread because that would be a catastrophic breach of customer trust that would likely never fully return
timdev2
I would normally say that "That must be a coincidence", but I had a client account compromise as well. And it was very strange:

Client was a small org, and two very old IAM accounts had suddenly had recent (yesterday) console log ins and password changes.

I'm investigating the extent of the compromise, but so far it seems all they did was open a ticket to turn on SES production access and increase the daily email limit to 50k.

These were basically dormant IAM users from more than 5 years ago, and it's certainly odd timing that they'd suddenly pop on this particular day.

LeonardoTolstoy
Almost this exact thing happened to me about a year ago. Very old account login, SES access with request to raise the email limit. We were only quickly tipped off because they had to open a ticket to get the limit raised.

If you haven't check newly made Roles as well. We quashed the compromised users pretty quickly (including my own, the origin we figured out), but got a little lucky because I just started cruising the Roles and killing anything less than a month old or with admin access.

To play devil's advocate a bit. In our case we are pretty sure my key actually did get compromised although we aren't precisely sure how (probably a combination of me being dumb and my org being dumb and some guy putting two and two together). But we did trace the initial users being created to nearly a month prior to the actual SES request. It is entirely possible whomever did your thing had you compromised for a bit, and then once AWS went down they decided that was the perfect time to attack, when you might not notice just-another-AWS-thing happening.

timdev2
Thanks for sharing. After digging in, it appears that something very similar happened here, after all. It looks like an access key with admin role leaked some time ago. At first, they just ran a quiet GetCallerIdentity, then sat on it. Then, on outage day, they leveraged it. In our case, they just did the SES thing, and tried to persist access by setting up IAM Identity Center.
orblivion
I wonder if a few cases of compromise right after the outage can also be a coincidence. If we have a lot of reports of the same, then it gets interesting.

(The particulars of your case being strange is a separate question though.)

CaptainOfCoit
Is it possible that people who already managed to get access (that they confirmed) has been waiting for any hiccups in AWS infrastructure in order to hide among the chaos when it happens? So maybe the access token was exposed weeks/months ago, but instead of going ahead directly, idle until there is something big going on.

Certainly feels like an strategy I'd explore if I was on that side of the aisle.

shadowpho
Wouldn’t this be a terrible time because everyone is looking/logging into AWS?

If my company used AWS I would be hyper aware about anything that it’s doing right now

djeastm
Might be, but also could be the opposite. With peoples' heads swimming just to get back online they might de-prioritize something else that just looks odd where under normal times they'd have the time/energy to go investigate.
AtNightWeCode
Not uncommon that machines get exposed during trouble-shooting. Just look at the Crowdstrike incident just the other year. People enabled RDP on a lot machines to "implement the fix" and now many of these machines are more vulnerable than if if they never installed that garbage security software in the first place.
geor9e
If I was a burgler holding a stolen key to a house, waiting to pick a good day, a city-wide blackout would probably feel like a good day.
what
That’s likely a pretty bad day to burgle. People are probably going to be at home. You should wait for garbage day and see who hasn’t put their bins out.
brador
Lot of keys and passwords being panic entered on insecure laptops yesterday.

Do not discount the possibility of regular malware.

[dead] NedF
[dead]
kondro
us-east-1 is unimaginably large. The last public info I saw said it had 159 datacenters. I wouldn't be surprised if many millions of accounts are primarily located there.

While this could possibly be related to the downtime, I think this is probably an unfortunate case of coincidence.

didip
During time of panic, that’s when people are most vulnerable to phishing attacks.

Total password reset and tell your AWS representative. They usually let it slide on good faith.

[deleted]
(comment deleted)
mr_windfrog
Considering AWS’s position as the No.1 cloud provider worldwide, their operational standards are extremely high. If something like this happened right after an outage, coincidence is the most plausible explanation rather than incompetence.
jmward01
If I were an attacker I would choose when to attack and a major disruption happening leaving your logging is in chaos seems like it could be a good time. Is it possible you had been compromised for a while and they took that moment to take advantage of it? Or, similarly, they took that moment to use your resources for a different attack that was spurred by the outage?
defraudbah
weird, can you send me your API key so I can verify it's not in the list of compromised credentials?
Traubenfuchs
It makes me very uncomfortable to know I got my CC in GCP, AWS and oracle cloud and that I have access to 3 corporate AWS accounts with bills on the level of 10's of millions per month.

Why don't cloud providers offer IP restrictions?

I can only access GitHub from my corporate account if I am in the VPN and it should be like that for every of those services with the capability to destroy lives.

WesleyJohnson
Our Alexa had a random person "drop in" yesterday. We could hear a child talking on the other end, but no idea who it was. It may just be a coincidence, but it's never happened before so it's easy to imagine it might be related to the AWS issues.
whoknew1122
The AWS issue related to DNS entries. And IAM doesn't use Dynamo DB. It wasn't related, other than an outage gives a good way to obfuscate TTPs.