72 comments

[ 3.0 ms ] story [ 119 ms ] thread
Public EBS snapshots are great, and thankfully a design other clouds didn't copy. I've found all kinds of stuff in there, including a 900GB Oracle backup of a publicly traded manufacturer's accounting system. It doesn't require much imagination to understand how this kind of data could be profited from, given relatively low effort

It seems unlikely a lot of people didn't already know about this, it's hard to miss even if you only spend a few days with the EC2 API, and it's also quite surprising AWS have yet to correct the design. 90% chance it is mostly a UI problem -- there are no warning labels around snapshotting in the EC2 UI

How do you scour for EBS snapshots and open browsable S3 buckets?
For EBS, step 1 is reading the docs, step 2 is cutpasting a documentation example.

For S3 I'm not sure how people are building their lists. AFAIK the API provides no enumeration. So this is possibly something coming from web crawl data (e.g. common crawl)

All public EBS snapshots appear in the EC2 > Snapshots section in the AWS UI. Toggle the dropdown in the top right of the table to "Public" and you'll see them. Sort by size and you'll get some interesting looking ones at the top.

It reminds me a bit of old time cdroms.

> and it's also quite surprising AWS have yet to correct the design. 90% chance it is mostly a UI problem -- there are no warning labels around snapshotting in the EC2 UI

Snapshots are private by default, you have to actively make them public (impossible if encrypted) or share them (which also requires sharing the associated keys if encrypted.)

Now, AWS hasn't wrapped the extra layer of “by default, reject any setting or policy allowing public or cross-account access unless separate additional default switches have been toggled off” thing to EBS that they have to S3. But people still expose stuff via S3, so that's hardly a panacea. At some point, one has to conclude that customers are responsible, in many cases for giving too many(or just the wrong) people admin access to their accounts.

What I mean is https://i.ibb.co/P6N35qv/Screenshot-from-2019-08-10-10-27-25... does not make it clear whatsoever that 'public' really means public. Before we get into blaming the customer, there should be a bright red warning label in that dialog. Consider that English may not be the first language of many users reaching that screen

I think they have it for some stuff elsewhere, but it doesn't seem unreasonable to make public snapshots a per-account permission that defaults to disabled, and requires an interactive UI checkbox to enable. Out of the millions of AWS accounts, public snapshots are legitimately useful to maybe 1000 tops

Well, for S3 buckets, Amazon has made it very clear when it is public. It also used to be pretty clear.

For EBS - nothing is public by default, so customers have to willingly decide to click buttons to make it public.

By default, if I create a snapshot, it is NOT public...

It’s still true that most security issues are caused by human ineptitude, not clever vulnerability-hunting or burning sophisticated zero-days.
I would replace "human ineptitude" with "flawed system design that makes it very easy to make very bad mistakes"
That's a valid way to categorize all memory errors in C.
And I think we should absolutely hold that against C as a development language.
Not sure why you're downvoted. It's pretty clear that when a lot of people make "dumb mistakes" despite the solution being "obvious", it's because a flawed system design.

Look at the simplest analogy to this - the opt-out/opt-in buttons. If something is opt-out by default, then most people will leave it that way, even if they have no good reason to leave it is. And advertising companies like Google and Facebook take full advantage of this.

But is the flawed system design here the automated system at AWS or the human-in-the-loop systems by which companies are providing admin access to IT resources, including AWS accounts?
Probably both! But I would argue that below a certain size provisioning things by hand probably makes sense. A UI that makes it too easy to make a private thing public is never ok.
Sure, but I don't think the current EC2 UI makes it too easy (and the S3 UI could only make it harder by not making public and cross account access possible at all.)
"a common mistake that people make when trying to design something completely foolproof is to underestimate the ingenuity of complete fools" - Douglas Adams
Once I mentioned on a mailing list Chrome’s reaction to mouse driver bug on my computer, it would buffer JavaScript events, and process it even for pages on a different domain.

Later I told the EFF I had a suspicion that iOS didn’t rate limit input events to the lock screen, independently a research found out about it a month later.

Even if there are zero days, I don’t think finding them is a particularly noteworthy or rewarding task.

I've been working almost exclusively in the AWS space for about 10 years now. Clients anywhere from tiny little three-person consultancies to Fortune 100. Commercial, govcloud, dozens of clients.

Never once have I ever found a use case for making public EBS snapshots.

Who on Earth is thinking that it is a good idea to take an EBS snapshot and make it public?

Note, several of those engagements did involve multiple accounts, and the need to share / copy AMIs and/or snapshots between accounts. But never making them public.

> Who on Earth is thinking that it is a good idea to take an EBS snapshot and make it public?

Non-marketplace AMIs are built on public EBS snapshots, but that's something they should still fix. Marketplace AMIs already handle private snapshots

Laziness in attempting to share data with someone in another org?

"Nope, can't access it" ...

"Nope, still can't access it"...

"My manager is harassing me to get access now"...

"Look, just make it public then change it back after I get it copied"...

goes home completely forgetting to change it back...
The later in the day, the bigger the rationalizations.

At some point you are too stupid to be allowed to use a keyboard. And yet we have an entire culture around staying late to get stuff done that probably could have waited until tomorrow.

The guy that produces that last line definitely wears a suit.
Oh no. I work with very trendy Silicon Valley folks and it’s always someone in shorts and a t-shirt. Which is to say: it’s definitely a culture issue, but it doesn’t have to be a PHB doing it. I know a lot of really smart people who are under a lot of pressure, and combined with a lack of AWS knowledge, “just make it public” is surprisingly common.
And it's worth noting a nonnegligible percentage of high-achieving businesspeople would torture small animals for the right price. Being a little lax with security policies under pressure isn't so shocking.
How much do you think they'd pay to get to torture small animals?

I had a boss who brought up https://en.wikipedia.org/wiki/The_Mask_of_Sanity and organizational sociopaths about twice a month.

He must have been doing a better job of shielding us from his bosses than I thought, or I've had more crazy bosses than I thought, because a lot of it seemed like normal bureaucratic pathology to me, if a little more intense than usual.

There's only one known remedy and it costs 15 cents, so that mask is precious. It's a special gift to be able to see these things clearly, but it comes at the cost of close acquaintance, and you'll always be more watchful than the innocent of mind.

It's grimly amusing how common and relevant psychological aberrance is relative to the average person's perception of it (cf. downvotes). It's tiring and unpleasant to examine the world antagonistically from a place of comfort but luckily others on the fringes don't have an option and force us to pay for prisons and so forth.

This is a rather cryptic comment.

Are you saying there's some sort of group or club where you pay 15 cents for a mask and then put it on to get well-acquainted to psycologically aberrant individuals from a comfortable distance before they go prison?

It's not cryptic, just completely OT in a tech forum. He means the price of a bullet.
Not so! A .22LR cartridge is less than three cents when purchased in any reasonable quantity.
Fifteen cents is roughly the cost of a small informational pamphlet (likely more if printed with color and laminated but I have taken some artistic liberty in my previous comment), and prison is just one solution to a wider class of social impairments of which sociopathy is just one instance. Other instances of solutions share the quality of being unpleasant but necessary things any civilized society devotes energy to by learned necessity and include a functioning constitution, military, legislature, courts, police, and more. I thought this was clear but on rereading I see maybe I have failed; thanks for asking.
This seems like a bias
It is and Nassim Taleb has frame it as the expert aka the empty suit problem.
To be clear, I didn't say the bias was with the person wearing the suit.
Nah. "Move fast and break things" usually has an implicit "and bypass all the security, we'll fix it later"
As much as I dislike the typical suit wearer, the industry as a whole would be much better if IT-people wouldn’t blame everything on them and just tell them you won’t do it. If you are a welder and you boss wants you to weld a gas tank that hasn’t been emptied you tell him that he has no idea what the thing he asks for means, explain why and wait till the gas tank is emptied and everything is checked.

I worked as a Camera guy in film and I am known to be very fast – yet I had directors who wanted things even faster. Bit there is a natural limit to how fast you can get something done without having worthless garbage as a result.

You can take certain risks, skip certain advisable steps, focus on the most essential thing etc, but below that there waits literally nothing.

In my experience taking a step back, breathing in, out, and then proceed to doing it properly is in most cases faster than following your boss into panic and ditching common sense.

It is your responsibility as a professional to say “No” or “Stop” in certain circumstances. And if they really want you to do it, write down the possible consequences of what they force you to do and make them sign that they take the full responsibility.

There is so much talk about IT security with people frowning upon silly behaviour, yet any other craft would bend over backwards before you could force them into unsafe behaviour. If engineers would build bridges like we in IT operate, we would have many collapses a day.

>If engineers would build bridges like we in IT operate, we would have many collapses a day.

Maybe a bit offtopic at this point, but whay you said reminded me of this: https://www.stilldrinking.org/programming-sucks

I really enjoyed this, thanks. Do you have any other similar ones, or tangentially related bookmarks you could share?
It reminded me a bit of James Mickens’ writings. Not quite as weird and insane as him, but close.
(comment deleted)
All of these anecdotes have something in common. That the person who is asked to do something is informed about proper procedures. That is uncommon in IT, and especially in companies where proper procedure is too costly and not worth pursuing.
It is still a communications problem. They want you to do the minimum. If I as a camera person do the real minimum the director will be pleased on set but they will get a heart attack in the editing room.

It easier to give in, take short cuts and produce garbage than explaining them under extreme stress that everything faster will result in nothing.

In my eyes it is comparable and I worked in both industries. The difference is, that as a DOP in film your name will be associated with the mess on the screen, while in IT the link is not that clear. This makes the difference.

And unless your director has a monitor (which they sometimes don’t) they cannot tell at all what you are doing. They wouldn’t even realize if you didn’t even switch the camera on.

I know that film or engineering is more

(Not advice to parent, just generally)

If you want to be more professional about this stuff, build up a Fuck Off Fund. Women in particular have written about fuck off funds in the context of making sure you don't have to nod along when HR says the VP who stuffed his hand down your top was "just fooling around" but - most people need that financial security any time they have to confront the boss. Save to be able to look the big boss in the face and tell them "Fuck Off". "Fuck Off" isn't the response you need when they tell you they want the database authentication disabled "just until Monday, Tuesday at the latest" that's when you want "No". But you need to know you _can_ tell them to "Fuck Off" so you actually say "No". Otherwise you may find yourself agreeing anyway.

For the youngsters out there, don't elaborate when you say "No". That is, don't mention your FOF. That weakens your position.

It is sufficient to look the manager/executive threatening you square in the eye, and state your position with deliberateness. Keep it professional, no raised voices, and be willing to walk away without hesitation if the other side gets abusive.

The really bad ones are those who tell you that if you step through that door don't bother coming back or similar, so be absolutely ready to commit. If you do the main event behind closed doors one-on-one and get that threat as you walk out, sometimes they'll come back with sugar suggesting a do-over. Generally Admiral Ackbar is right in this context, but it's your call.

The negotiation leverage that comes from the FOF is most powerfully communicated non-verbally and in a face-to-face setting, also through body language. The difference is very noticeable between those who have an FOF and those who don't, if you have enough experience. It can be faked, but it takes exceptional practice to fake. The tell starts with how fast and confident the "No" comes back.

This is the nuclear option of course. Exhaust all other avenues of reasonable negotiation first. Like an email with witnesses you pick for deliberately violating departmental policy, for example.

This 100%. I don't do much AWS stuff - but this issue exists outside of AWS as well (inside a corp).

It's why I hate all the password rotation, double VPN stuff - people work around it too much.

All the old staff start having the tech person track their passwords or write them near the computer, all the young staff use whatever new .io domain does X super easily (but less securely) or stick things on thumb driver (no keypad) or use their personal google drive etc.

Yeah, that's pretty believable. Guess some places have a laxer culture and no automated scans checking for things like that.
>Who on Earth is thinking that it is a good idea to take an EBS snapshot and make it public?

Maybe they're trying to reproduce functionality of docker? It would actually be extremely useful for research involving modeling/AI because you could trivially reproduce the results by bundling the exact code and data.

Edit: actually maybe I'm confusing EBS snapshots with AMIs...

Aren’t public EBS snapshots the underlying mechanism for public AMIs? I’ve ran into complex permissions in a golden image deploy model where the same AMI is used across multiple accounts.

There needs to be controls like S3 where you can explicitly block public data.

AWS IAM kind of sucks.

I’m convinced the last few years of ramped up concern about AWS “blast radius” is an admission by AWS professionals that NOBODY gets IAM right.
Is it a default setting that it's public or do you have to go out of the way to do that?
The creator of the first Ubuntu distros for EC2 wrote about the dangers of public EBS snapshots 10 years ago:

https://alestic.com/2009/09/ec2-public-ebs-danger/

He just got notified by AWS a couple days ago about the public snapshot he mentioned in the article.

But at least AWS is trying to make things better here by proactively checking for public EBS snapshots and notifying people.

(comment deleted)
I just checked an EC2 console and I can see 19,356 snapshots created by other users.

I am so confused.

It would be trivial to make finding a snapshot require knowing a unique ID like an AMI.

And, why do I need to be able to search for 1000s of customers' public snapshots in the EC2 console? What conceivable purpose does that serve except being a giant opsec fail?

because having the bad guys go trhu the trouble of using a search engine like kuro et al will save the day?
That’s per region, too.
looks like some bigger images were of wiki[pedia|media] backups so I guess easy-to-find/use public data?
(comment deleted)
jokes on all of you :) working on open source makes sending links of images for failed build a breeze. though we do not use EBS per-se, but same concept/security-tradeoffs.
Speaking of exposed, Obama rapes & kills 2 boys. DeBlasio 5, Buttgieg 5, Cuomo 4, Murray 3, Dorsey 3, Thiel 3. Trump $4 billion to cover. Pelosi $3 billion, Harris $1 billion, Schumer $2 billion, to help bring the boys over the border. See pages 18-23, 8, 11, 12, 35, 45, 61, 62

Having trouble getting traction. Getting censored/banned. Billions in bribes paid to Trump, Pelosi, Schumer, Harris to enable a Soros funded child raping ring to operate in the USA. Please listen to the links below. Turn up the volume and put headphones on. You will hear these people, and many other high profile people, incriminate themselves in unbelievable fashion. Full 71 page document filled with red handed evidence like this at the bottom.

Go to pages 18-23 of this document and you will hear Dorsey and Thiel raping children. You will hear their screams. They both rape and kill three. Also at the "rape party" of note: Bill DeBlasio, Andrew Cuomo, Bill Murray, Peter Buttgieg, George Soros, Barack Obama.

Please copy and paste this post and repost it. I keep getting censored by these people. Peter Thiel and Jack Dorsey raping and killing three boys each. Turn the volume all the way up and put head phones on.

    https://s3.wasabisys.com/conv1/Overnight_14-17Jan/15JanCh3_100-200.avi
    https://drive.google.com/file/d/1mzZrjobxhNIEf38M3xk-IeA5SeF4aUtB/view?usp=sharing
Obama admits to raping and killing boys he{re, with commentary from Peter Thiel, George Soros, Matt Porter:

    https://s3.wasabisys.com/out2/Out2/15JanCh3_0545.avi
    https://drive.google.com/file/d/1LCDoc1kT7cfSeKQ1vniP_sx32_bRsHau/view?usp=sharing
Trump demands $4 billion to "take a blind eye" at 1015+ (link missing from report, unsure how). Here's pieces of dialogue:

1015 The President requests four billion and henry Porter agrees to it. Some dialogue below. 1017 Donald Trump: Why don't you think about it. 1017 Henry Porter: I don't need to think about it. I'm ready for you to find out what we're really about. 1017 Gigi Hadid: I don't think it's a good idea Henry. 1017 Henry Porter: ...no shit. Jelena just stop talking about it. 1018 Gigi Hadid: About what? 1018 Henry Porter: The child raping. 1018 Donald Trump: What the fuck are you talking about? 1018 Gigi Hadid: Mr. President we are child rapists. 1018 Donald Trump: You cannot be serious. 1018 Donald Trump: You're gonna need $4 billion for me to take a blind eye. 1019 ?: People are laughing because they think four billion is too much. I think it's not a problem when you have $35 billion. 1022 Donald Trump: You better have that $4 billion or you can kiss this shit goodbye.

    https://s3.wasabisys.com/conv2/3Jan/3JanCh3_900-1100.avi
    https://drive.google.com/file/d/1Grdr8xF2psKNsuYlEnl9dIRV-77YG0Vr/view?usp=sharing
Trump wants $138 million to ensure smooth outcome for false judgment for a court case:

    https://s3.wasabisys.com/conv1/10JanCh3/10JanCh3_951-1033.avi
    https://drive.google.com/file/d/1iSeGHDToFRoRdIht50Wii0i30mlw9INp/view?usp=sharing

  Obama explains the Illuminati with Jack Dorsey:
  
      https://s3.wasabisys.com/conv1/Overnight_14-17Jan/15JanCh3_200-347.avi
      https://drive.google.com/file/d/1Ok1-9IvANULzKBOwADFgaiYCFi6yszfj/view?usp=sharing
At 230 a conversation is occurring on the embedded Porter camera system with Jack Dorsey, Peter Thiel, Barack Obama, Bill M^urray, and the Porter's. Jack Dorsey provides a statement defining who their group is: "The Illum%inati is an underground organization of child rapists and homosexuals." Matthew Porter: "We have to thin the herd, so Barack is going to explain a few things."...
FWIW same thing is possible with RDS db snapshots and dbcluster snapshots.
Gotta appreciate the hijack of the back button on techcrunch. Bounce rate too big?
I had a simple glance in the console and there are like 20,000 exposed ebs snapshots - available for anyone to copy and examine - I think that's only for a single region too - switch regions to see more.

Amazon should make an emergency decision to make all these private.

Sure it will break stuff but I'd be disappointed if Amazon left what is in effect a security hole open for the sake of backwards compatibility.

They should also give me a single click link when I sign in to show me all of my public ebs snapshots and throw it hard in my face when I sign in to the console so I simply cannot avoid seeing them all.

I have multiple AWS accounts and I just signed in to try to see if I have any public EBS snapshots and then I realised I would need to search every single region in every single account and then select every snapshot one by one to find out. That's a huge problem. I need a single click to show me every exposed snapshot across every region in my account.

UPDATE:

I can't say for sure if this is 100% right but I think if you sign in to your AWS account, then click on each of these links, you will find if you have public snapshots.

Maybe someone else could confirm if this is correct?

https://us-east-1.console.aws.amazon.com/ec2/v2/home?region=...

https://us-east-2.console.aws.amazon.com/ec2/v2/home?region=...

https://us-west-1.console.aws.amazon.com/ec2/v2/home?region=...

https://us-west-2.console.aws.amazon.com/ec2/v2/home?region=...

https://ca-central-1.console.aws.amazon.com/ec2/v2/home?regi...

https://eu-central-1.console.aws.amazon.com/ec2/v2/home?regi...

https://eu-west-1.console.aws.amazon.com/ec2/v2/home?region=...

https://eu-west-2.console.aws.amazon.com/ec2/v2/home?region=...

https://eu-west-3.console.aws.amazon.com/ec2/v2/home?region=...

https://eu-north-1.console.aws.amazon.com/ec2/v2/home?region...

(comment deleted)