> Yeah, in reality its like complaining someone "broke" into your care when you left all of your doors open.
And you wouldn't get much sympathy, but this is a legally viable complaint. When my car was stolen, the thief was charged for theft, but his girlfriend who was joyriding with him was charged for criminal trespass. Just being in the car when you don't have permission is illegal.
In the physical world (think "breaking and entering"), any amount of force (e.g. Pushing an already partially open door) counts as "breaking".
It isn't necessary to actually "break" anything.
Edit: Of course, it's still not a great metaphor because it implies there was at least some kind of barrier to entry that had to be pushed aside - but from the sounds of it, there wasn't.
It's difficult. I like the idea of subresource integrity a lot, but a counter example: Twilio discovers a bug in their JS SDK. Without subresource integrity they can push up a fix and propagate it to all clients immediately. With subresource integrity they would need every client to update their web site to change the integrity attribute. If it was a serious bug then that could be a huge issue.
Even putting the security arguments aside you can bet a lot of customers would be really irritated having to constantly update this stuff on their site. If a competitor didn't have that restriction it would become a selling point. IMO the ideal is that Twilio provides both options: a "sdk-latest.js" that's a moving target, as well as "sdk-v1.3.js" that is frozen and can be locked with subresource integrity.
in which clients compute the integrity and asynchronously ping a lightweight payload of the integrity to the third party? You would then seed your report service with a list of known hashes and set up alerting for when new hashes were observed.
Script tags (added programmatically) should support such errors via "onerror" functions. Though, I don't know if integrity errors could truly be determined. If they can then you can do whatever you want with such an error.
>If it was a serious bug then that could be a huge issue.
Regardless of the severity of the bug, the only-case scenario is that all the sites you have pulling from that CDN break until you recompute the hash. How annoying this is scales directly to how frequently your libraries have to release vital security bugfixes.
> the only-case scenario is that all the sites you have pulling from that CDN break until you recompute the hash
Until you recompute the hash and communicate that new hash to them and they implement it on their site. It’s not nothing from an implementation point of view.
I'm not sure who "you" and "they" are in this scenario so I apologize if this isn't what you're getting at: the failure -> fallback -> threat evaluation-> remediation process can be entirely automated except the evaluation part. I'm not a web developer so I don't know the incidence rate of using unstable builds in production and having to patch stable builds, so maybe having to manually evaluate threats in this way /would/ be annoying, but the actual devops seems pretty trivial.
> Without subresource integrity they can push up a fix and propagate it to all clients immediately.
In reality, no you cannot.
You have multiple layers of cache between the S3 bucket and rendering, unless you disable caching entirely at a massive increased cost. Some of these caches are poorly behaving (e.g. intermediary caching).
The correct way of doing this ALREADY is to increment the version number in the URL (e.g. /3.1.0/My.Lib.js to /3.1.1/My.Lib.js) and to re-point the pages to use the updated library. This is reliably cache breaking, and will assure end users are getting the bug fixed version faster (or ever in some cases).
Once you're already doing it correctly, Subresource Integrity is a freebie. There's a reason why "sdk-latest.js" is largely dead concept from a bygone era: it is a huge anti-pattern and anti-feature.
Heck allowing upstream to blindly push you changes without being informed is nuts regardless. This argument is essentially: "I'm doing it wrong, and Subresource Integrity would stop that, so it is a non-starter." Without considering that what you're doing is bad practice before Subresource Integrity joined the party.
You do not have absolute certainty that every old cached version of a file you have published has been purged by every third party cache, no. But the vast, _vast_ majority do get purged very quickly. I've done this myself when in a bind many times and it's worked fine (and I've been able to verify that via error logging).
> There's a reason why "sdk-latest.js" is largely dead concept from a bygone era
Is it, though? From a quick check, it's what Google Maps does. It's what the Facebook SDK does. We already know it's what Twilio does.
> This argument is essentially: "I'm doing it wrong, and Subresource Integrity would stop that, so it is a non-starter."
Zero dispute with that characterisation from me. But it's how things already operate in the real world. I'll join you in shouting from the rooftops that people shouldn't be doing it, but that doesn't really get you any closer to actually stopping them. For a great many people the flexibility to quickly push up changes is a feature, not a bug.
For sure, and I believe Twilio did not intend to "have their door open" and that whoever modified their code knew that they were "trespassing" similarly.
I am not a lawyer, I'm definitely not your lawyer.
Trespassing and possibly theft, I think.
Theft includes moving stuff without the owners permission. so if I ,say, have a tow truck and move a car across the street without the owners consent, I'm a thief.
I only know this because a friend talked about a case where they rotated a car in-place, and there was a question about is that theft? its center of mass is still where they left it, so it wound up not being theft. Moving furniture in a house is a weird one. Probably leans on ill intent, moving a chair to block a door seems like it's pointed to giving the owner a hard time. moving a chair to perform cpr has much purer motives.
I thought in common law countries theft required you to have the intent to permanently deprive the victim of the property. this is why there are separate statutes to cover stealing cars because people just claimed they were taking it temporarily.
S3 security is a pain. S3 options are tricky to master, docs are beyond confusing.
A common S3 use case is: Use a Bucket for Read-only static content, (js, html, imgs). From a dev perspective ideally it will work like a protected folder + Web Server, whereas the webserver will have read-only privileges, but reality is far more complicated.
I can understand why was easy for Twillio to have this infosec issue, since we've done it ourselves many times when using S3.
Is it really a pain though? Maybe because I’ve been working with it since the beginning. Bucket ACL with NO permissions, then manage all permissions on an IAM role on the account.
If it’s cross account then allow assuming to other accounts, but no reason to bother with the bucket ACL.
Leave the bucket policy blank and you’ll never have to worry about an open bucket. Better yet make a deny rule to everything but a single role, and you don’t have to worry about rogue roles exposing the bucket.
Perhaps some of the issue is that permission can be managed on both IAM and a bucket policy?
I just started a toy project using S3, and at least to a novice user, it was a confusing, overwhelming array of options and linked pieces. It felt like being hit with a lot of advanced options all at once, with no obvious happy path for common cases.
I'm sure an expert finds it all very natural. Security is always hard and has to be done right the first time. That's a domain that's always going to deter novices, and perhaps the only way forward is to hire an expert to ensure you're doing the right thing.
Still... I've been programming for four decades and it felt a little odd to be quite so out of my depth. It's not the same as anything I've done before, but it felt like it shouldn't have been quite so bewildering.
This is the correct design pattern as of 2020. Roles + STS, and always be stingy when granting a role any permissions, both at the resource or API level.
But, as others have said, the documentation at times is poor at best, and it can be easy to make mistakes.
Some preventives every ops team can take are:
1. Never make manual permission changes in the console. Everything is done via SCM’d code.
2. All permission changes are peer reviewed.
3. Create logical separations of buckets so it’s more obvious what the permissions should be of a bucket. You can go so far as to have separate accounts that are for public uploads, separate accounts for public assets, etc. Avoid granting elevated permissions at a folder level, which is harder to understand as a developer without access to view the ACLs.
> Never make manual permission changes in the console. Everything is done via SCM’d code.
The first time I heard “infrastructure as code” I thought it was some dumb new buzz words, but no, it’s good advice. Your cloud formation, serverless, terraform whatever should be stored in GIT a stack replace is how changes should be made.
IMO this is code/console dichotomy is a design flaw in most cloud offerings that will haunt the industry for a long time.
The web console should be just another UI for editing the configuration in source control. As an example of why that's important, a UI can give you a warning and use affordances to guide beginners away from dangerous setups. In the case of S3, a slider for basic access level can use colors and show a list of objects that would become world readable or writable at a given level (for that matter why is world writable even an option?). A giant YAML file looks the same whether it's safe or dangerous.
If you really don't want to use a UI, then maybe abstracting a linting layer that feeds the exact same warnings to the UI as to your CI system and text editor would help.
I occasionally spin up things in the AWS console so I can describe them through the CLI and then delete them. IaC is great until you have no idea what the code should be.
Certainly not discouraging this, so thank you for pointing it out.
I do the same, and sometimes I need to do something manually so that I can grok what configurations I need to define in terraform / whatever other tool I'm using. POC'ing is different than a production or even staging service.
It is way too easy to screw up S3 bucket permissions. I caught it once when a coworker told me they setup a bucket for testing that allowed employees to upload, I checked the permissions and it allowed the world to upload.
It would be much harder make that mistake with Google Cloud Storage, at least in the web console. GCS also supports disabling bucket ACLs permanently at bucket creation time, and that is the option they recommend[1].
AWS does scan and alert for some public bucket issues. But with all the public S3 issues, I think it is clear there needs to be more prominent warnings in the AWS console, the command line, and with tools like Terraform.
>GCS also supports disabling bucket ACLs permanently at bucket creation time, and that is the option they recommend[1].
S3 does too. There's an entire page during the creation wizard that is dedicated to blocking any and all public access, even causing the bucket to ignore ACLs or other settings that would otherwise expose the bucket publicly. All public access is disabled by default, and enabling it actually requires the user to actively uncheck 5 different checkboxes, each of which explains that unchecking it will open the bucket up to public access, and then even requires an additional attestation, in a large orange warning box, that says you acknowledge that the options you chose will result in the bucket being public.
I'll be the first person to tell you that AWS is overcomplicated and hard to use, but this isn't that. IAM and bucket policies are a pain to work with, but if you screw up with those, the worse you will do is expose your bucket internally. But exposing a bucket publicly to the internet is an entirely different act, and there's not really any excuse for it other than just not reading the directions.
However, neither this option in S3 nor the option you linked in GCS would have served Twilio's use case. They wanted their bucket to be publicly accessible, just not publicly writable. That's an entirely different access management issue.
It is absolutely a pain. I'm using S3 for the first time on a project right now and literally every time I have to do anything in AWS I end up confused and scared that I'm leaving a wide open security vulnerability. The documentation is complete insanity to anyone just trying to accomplish what should be an extremely common use case in a reasonable amount of time.
Before using AWS I was in the 'what morons!' camp whenever some company got breached via leaving things wide open on AWS. Now I'm in the 'ok I understand why its so common' camp.
The problem with S3 is the exact same problem with everything else in AWS. In their quest to be compatible and attractive to every single possible niche use case of the world's largest companies, they completely forgot the lone developer who just wants to upload a couple files.
You'll even see this in their sales pitches. AWS will spend so much time talking about the most advanced use cases and how they are possible, but if you ask them a simple question about how to run a single bucket that doesn't also involve using all of their ML tools and global accelerator and cloudfront, they'll be blindsided. It's almost as if they consider the common use cases to be "too simple/obvious" and therefor they never bothered to create any documentation or put any thought into it.
Certainly not, and in their layers of defense against this stuff, they should have had processes to train people, audit bucket policies, conduct penetration tests etc, and there's no excuse for them not having caught it before.
But with that said, one of those layers of defense is also using tools that are easy to keep secure so that the vulnerability doesn't appear in the first place. While Twilio isn't "a lone developer", I wouldn't be surprised if the person who did create that bucket on behalf of Twilio was just a lone developer fumbling around in the console. And again, that's no excuse, but it still is an area for improvement.
I kind of miss the times when working with the cloud was easier or more straightforward.
Now, as has been stated before, reading any cloud vendor documentation is like reading the Oracle manuals of years ago, not a happy experience.
Granted, these clouds now do things much more powerful and target more difficult enterprise scenarios. One can miss Heroku, and AWS has Elastic Beanstalk for that, but EB gets much more difficult fast.
We have been solving this issue with the app I have built Commandeer. Here is the main S3 dashboard https://imgur.com/gallery/JutGUI3 And here is the view of an individual bucket https://imgur.com/gallery/0OlKY7Q . It let's you completely manage S3 and 25 other AWS and Cloud services from a desktop app. You can easily see permissions and also preview every type of file. We have also done this for DynamoDB, Lambda, CloudWatch Alarms/Logs/Rules. The tool was built out of our frustration that many have.
You're using S3 for the first time. Twilio has the capability to hire Ops staff that have used S3 for 10+ years. Throw a penny down the street in any major city and you'll hit a DevOps person with AWS experience. It's amazing to me how often I hear a company not set up s3 properly, or not enable budget alerts on AWS, etc. This is all day 1 AWS stuff. If that means taking away the ability for your developers to even create buckets then do so. A lot of you should just focus on development work and stay far away from operations/sec tasks, and that's fine. I'm going to assume most of these buckets aren't even made via code and are manual. That's bad enough.
I understand someone brand new to S3 being in the dark but someone in your PRs approval chain should have enough Ops skill to know to check these things if you're not going to hire an actual Ops person. And no, none of this is hard for anyone who has ever touched S3. Ops teams have known about these issues for over a decade. It's practically a meme at this point.
I understand what you're saying but you do realize that a huge number of software shops don't have massive teams or even dedicated OPs people right?
The number of heads on my project is ...4. We're it. There is no place or person to just push the work off to. There is no PR approval chain. We regularly meet to sync and discuss plans and thats it.
Until very recently all of our infrastructure was in house, right down the hall from me. I (or whoever was in that day) took care of making sure it was running smoothly.
Cloud is new to us, and there is no budget to hire a guy who manages AWS full time. That is the case at an insanely high percentage of software shops. Heck I would wager most people outside of 'new tech (sorry, I can't think of a better term)' shops are using point & click in the web UI to manage AWS. There is no code in source control that manages our infrastructure. I'm jealous of the people that get to do it all the 'right way' but for many there isn't enough manpower or time in the day to do it.
Now why we're in this position in the first place is another debate about why management thinks we need to go cloud at all. But its reality for many.
To me this is like buying a high end sports car, and neglecting the maintenance because you never factored in the expensive maintenance, and therefore the true cost of what you're buying.
Oh trust me I know totally what you mean, I'm usually the first ops person hired at a startup once they've gotten the budget for one. My goal when I work with devs is to get as much of the infrastructure frustration out of their way as possible (unless they want to help with it) and automate the processes that slow down their code from getting deployed. This usually comes with months of fixing cruft/security issues, etc that could've lead to a "hack" like this when just the devs were running the ship.
I wish MORE small dev teams brought in ops people and didn't wait until they had a staff of 20-30 or whatever it typically winds up being. But I know that's a pipe dream and you're basically losing a developer salary to hire a devops/ops person which isn't the direction they want to go.
I totally understand where this is coming from, but even if we factor AWS out. IAM _IS_ difficult to implement, there really isn't a way to make it "simple" while still make it extremely granular. You can't have without the other.
It's gotten better, if you can believe it. Regardless, I've managed to come up with one foolproof method for securing my data: Never put anything important on S3.
I have external partners who provide their own principal and I provision a subset of bucket access at the bucket policy level. Is there a better way to do this that's more manageable? The end users are using software or python code where they need to gain access via [access, secret, bucket, prefix/key].
actually aws changed the default lately and added more options to really see what you configured. in the past, there was no indicator if s3 is private only oder public
Wow. They were serving one of their SDKs out of this unsecured bucket and someone was able to upload a new version and it was distributed to customers.
Twilio says the only modification done to the script was to check if the modification could be done. If the hackers modified the script so it would call their own servers to check for that, they now have a list of IP addresses and timestamps.
There are big signs on world-accessible buckets when you go into the S3 console. It's a fairly new thing but they are (slowly) trying to help auditing and finding these things I guess.
However I'm not sure if there are any built-in tools to find weird nested policies attached that could cause a similar permission to be granted to all..
What does AWS have to do with this? Sure, the technical debt of S3 being one of the first services is clear to see. But it was a world writable bucket. What is AWS going to do about that? Eventually, you have to know and care about what the hell you are doing.
AWS's shared responsibility model is clear. AWS is responsible for security of the cloud. The customer is responsible for the security in the cloud (i.e. the customer's resources). By the way, enterprise support customers do get access to well-architected reviews by AWS.
If you ask for help from AWS, AWS will provide it. It may not be free, but it's available.
Even if AWS were to start proactively auditing customer setups, how in the world is AWS supposed to know what a customer's usecase is? Nevermind the fact it's a breach of the customer's privacy to just go rooting around in the customer's account without permission.
But let's assume AWS is going to take responsibility for customers' configuration decisions and violate customers' privacy by proactively auditing their accounts. Would AWS auditing Twilio's configuration here work?
The default is for S3 buckets to be private. The customer has to take specific, affirmative steps to give s3 buckets public access. You really have to jump through hoops to make a bucket public accessible.
Since the Twilio chose to make their bucket public, AWS auditing Twilio's setup wouldn't be helpful. AWS would just assume the Twilio knows what they're doing. How is AWS supposed to know there is a misconfiguration? Because Twilio clearly decided to make their S3 bucket publicly available.
Not only do you have to jump through hoops to make it public, if you are worried can't you use Amazon Macie and have a separate org level view that alerts you to any public buckets?
I don't bother because it seems pretty clear what buckets are public.
That said, quick tip to make your life easier.
DON'T use S3 ACL's
DON'T use S3 policies.
If I was AWS and didn't have so many customers I'd probably just create one mental model (IAM policies probably) as the place to manage things and block the rest.
> someone was able to get into Twilio's Amazon Web Services S3 bucket, which was left unprotected and world-writable
"Someone" was able? If the bucket was unprotected and world-writable, then _everyone_ was able.
> 'non-malicious'
> And judging from the URL involved, it appears to be an attempt to install a payment-card skimmer – RiskIQ has spotted the same URL in other S3 buckets targeted by miscreants.
In what world is that non-malicious? Non-immediately-damaging _maybe_, but it seems clear that there was a malicious chain of reasoning involved. Twilio got super lucky if this is all that happened.
One of the most shocking things I realized when I graduated college and started working as a software engineer is how insecure, or really just improperly-configured, large enterprise systems are.
This is anecdotal, of course, but it seems to me that any company that didn't start cloud-native is going to have flaws like this, and it's often not at the fault of developers but at the fault of top-down initiatives to "get to the cloud" ASAP instead of correctly.
Having an S3 Bucket with writing permissions to anyone is a major fuck-up. I know many people open buckets to the world at the read-level for whatever reason. I do that myself for static websites. But also writing permissions?
This requires a Bucket Policy that is written explicitly to allow this. I don't want to make assumptions because of course we don't know all the information and I personally know very little about infosec, but that sounds very strange.
Why would you do that? I assume the SDK gets uploaded from a secure environment that has the credentials to upload objects, so there's no legitimate reason to have writing permissions open to the world.
Yeah this is absolutely inexcusable for a public tech giant. I wonder how incompetently secured the rest of their backend is. This is terrifying, the amount of PII flowing through Twilio is insane, there really needs to be a way to bite companies that completely ignore all of these security compliance checks they say they pass...
> PSTN-based 2FA in 2020 is not a vote of confidence in the security team's competence.
A lot of companies go with email and/or PSTN based 2FA for customer convenience and to get them to use 2FA for their accounts.
I try to avoid using 2FA for accounts that only offer PSTN or email based 2FA because it's less secure than my randomly generated long password. It would be nice if more services offered 2FA based on client side TLS certificates.
i would not say it's less secure. Even if somebody overtakes one of my machine and gets to my lastpass vault or steals my clipboard having my smartphone with me and a SMS or PSTN based 2FA is better then not having it at all.
If implemented properly, it's totally possible that security is not reduced for that user. However there are at least three problems with it:
1) When not implemented properly, it is treated as an alternative authentication mechanism, not an additional required mechanism. My ultra strong password is useless if it's enough to steal my phone number and convince customer support that the attacker simply forgot the password.
2) It gives a false sense of security. I already have 2FA enabled! What do you mean I should be using TOTP/FIDO2/TLS client certificates?!
3) It reinforces an unfortunately widespread security anti-pattern, which reduces security for Web users in aggregate. So what if it's not a great idea? Everyone is doing it! Or put another way, MD5 password hashing is better than storing passwords in plain text. But please, please, please do not use MD5 password hashing! There are much better ways to accomplish that goal without introducing more flaws.
1. Drift detection is a relatively new feature.
2. CloudFormation was so basic for so long that most AWS professionals don't use it. They use Terraform or a similar, non AWS product.
CloudFormation didn't support importing existing resources until last November. For years you had to let it recreate everything you need, if you hadn't used it from day one. That's a big part of what kept us on Terraform.
You're absolutely right, and I think most of us have a hard time a. learning about all of the new features and b. justifying going back and fixing up stuff to adopt the new features, but IMO for such a critical piece of infrastructure it should have been done and the engineers/architects should have been adding that sort of stuff to the backlog or requesting training or asking for resources, etc. For me, that sort of stuff is the only way I can get the green light to go back and refactor.
AWS is getting really confusing, IAM is not simple, which is bad.
But S3 is a bread-and-butter service, and the policy is such that anyone using it should understand. I mean it says there in 'almost plain English' than 'anyone' can write content.
This is a pretty major blunder, because it hints that maybe they don't have any security reviews at all.
If I, a completely non security-knowledgable person were asked to do an audit of Twilio, I would find this issue. It's way on the lower treshold of 'obvious problem'.
Twilio is not 'Medium Blog' - they provided IT plumbing and therefore it's doubly problematic.
"Specifically, the modification added code to the end of the TaskRouter.js v1.20 SDK that made an HTTP GET request to hxxps://gold.platinumus.top/track/awswrite?q=dmn and followed the URL returned in the HTML by that request."
> followed the URL returned in the HTML by that request
Translated: we can't possibly know it's non-malicious.
TFA even has this to say:
> And judging from the URL involved, it appears to be an attempt to install a payment-card skimmer – RiskIQ has spotted the same URL in other S3 buckets targeted by miscreants.
Details in the linked blog post[0].
So, very much malicious.
Why the hell they included "non-malicious" in the title (granted, in quotes), I don't know. Readers could have easily dismissed this as actually non-malicious.
Magecart is a well known framework in the underground for stealing credit card numbers from websites (if they can get the JavaScript running on a checkout page).
If you hit it directly they will redirect to a random "Spin the wheel win a prize" type site in an attempt to hide the malicious nature of the payload.
I just don’t see how that diminishes the action of crime though.
A failed drug dealer (who got caught), by your definition, should get less time because they never managed to sell the drugs they were transporting over the border and got caught :)
Maybe only in America (but American law system is heavily biased towards people who have enough money to navigate it) - where deals and various legalese are a usual concept.
Non malicious? Does the person. That wrote that bit of bullshit even believe it.
They might as well have said "we are not smart enough to figure out the nature of this code. In unrelated news we are hiring senior JavaScript engineers, experience in opsec is a bonus"
Quick shout out to GuardScript which exists to help catch malicious 3rd party (or 1st party) javascript changes: https://www.guardscript.com/
It's a really simple service that fetches a list of script URLs that you provide, and notifies you with the before/after diff whenever the file changes.
SRI only works for static files whose contents never change. In this example, it would mean that Stripe could never push minor bug fixes to the SDK.
The workaround would be to use static versioned files instead of a generic script URL (or self-host the script), so that the value of the SRI checksum never changes.
Edit: In the case of Twilio, their docs say "By linking directly to our CDN in production, any patch-level (non-breaking) changes made to the SDK will automatically be applied [...]" meaning that if you link to Twilio's CDN, you can't expect the file to be static, and therefore can't use SRI.
I love AWS as any other developer, but this is their fault.
They've invested all their resources into building out the infrastructure, and have spent very little if any time on building the UX around it. You have to be fairly technical, never mind painstakingly detailed-oriented, in order to manage their services, a big chunk of which is hidden behind black screens and various control nobs.
Amazon's logic is probably to provide the piping, and leave the rest of the end user, but at this point it's not enough. In my opinion, they need to heavily invest into the customer-facing layer of their applications that shows all the status, gaps and holes, or these things keep happening.
Then again, they can continue as is, and the incidents will keep as is, and as long as nothing implodes and no big customer leaves, financially it makes sense to do nothing. Just thinking out loud.
I am still amazed some people want to do devops. Write yaml, run script, wait 15 minutes, oh no a typo lets retry. The developer experience is worse than nearly everything that came before it, and I would include ancient PHP and callback-hell nodejs in that. The AWS web UI isn’t much better.
Looking forward to finally try Pulumi in a project.
By doing what XML-with-schema did 15 years ago: give compile-time type checking, making sure at least the syntax and types are valid.
Ever had CloudFormation yell at you because some function like "Sub!" only accepts alphanumeric characters? Created complicated constructs because there are no variables, and very limited conditionals? Copy-pasted templates like an insane person because there is no way to abstract anything?
Not my idea of fun. All of this is solved by using a proper programming language.
Could you elaborate on what you think it does better than say Terraform. I looked at their comparison page https://www.pulumi.com/docs/intro/vs/terraform/ and while there is an argument to be made re: being able to write in a more familiar language, its kind of a moot point when config is mostly key value pairs.
The biggest paint point with something like Terraform is the fact that it can't actually validate cloud provider specific problems (for example declaring an Compute instance with a compute type (string) that doesn't actually exist). Does Pulumi try to address this in some way (granted its much easier to write your own checks like that in language as opposed to writing a go module for Terraform providers).
> (for example declaring an Compute instance with a compute type (string) that doesn't actually exist). Does Pulumi try to address this in some way
Yes, the AWS Pulumi SDK package includes a discriminated union of valid instance types, and the type of the property `aws.ec2.Instance::instanceType` is that union.
(Disclaimer: I am a former maintainer of Terraform and a contributor to Pulumi).
I used Azure for a year on a project and they did seem to have a pretty big focus on UI/UX. Without having previously done any cloud management work directly, I found it fairly straightforward.
I really have to disagree here, at least with your conclusions re complexity. No doubt AWS S3 configuration is very technical and has a lot of docs and corner cases. However, if you are a company like Twilio, and S3 is front-and-center the source of your business (and if it's serving your SDK, it probably is), your S3 bucket permissions are clearly worthy of periodic review. And you should have this done by somebody who knows it well, or who learns it well in the process.
I worked for a company where our CTO identified the same thing (unauthorized modification of JS SDK served via S3) as a potential vulnerability. It took us half a day to get the bucket locked down to where only the root AWS IAM user—secured by physical 2fac—was permitted to change the SDK, and then only after "unlocking" the configuration changes. Since the SDK was not updated frequently, having a little manual ceremony was fine. If it had been a high-touch path we would have had to do a little more engineering to keep it safe, but not that much more.
That said, I don't blame Twilio for not catching this; shit happens at a company with years of legacy built up. I do blame anybody who says "S3 is hard" and throws up their hands. You're a professional! Read the docs, they're dry but quite extensive! Play around with it in a test bucket! It's no harder than learning any other moderately complicated professional system. If you can learn Rust, you can learn S3's API. If you can learn Kubernetes, you can learn S3 ACLs and the IAM authz model. It may not be as interesting as the topics you want to read about, but it will probably be even more useful.
The flaw in your logic is that "S3 is front-and-center the source of your business" is never true with AWS.
To do anything in AWS there's at least 3 or 4 vaguely connected services, including IAM and RAM as a completely separate UIs. Usually with S3 you also have Cloudfront or some other CDN. You probably also have Cloudwatch logs and CLoudtrail event tracking. You might have VPCs involved. Some or all of these things, in an org like Twillio, could have entirely different teams managing them.
The problem isn't that S3 is hard. The problem is, as OP suggests, the UX of actually doing anything non-trivial across half a dozen services is somewhere between abysmal and war-crime.
Each of these services are highly different concerns and don’t belong together, and are honestly not _that_ hard to orchestrate together with the liberal application of Terraform.
If you’ve got one team managing your logging who don’t talk to whatever team who manages your s3/cloudfront setup, then that’s your problem.
> and are honestly not _that_ hard to orchestrate together with the liberal application of Terraform.
I love Terraform, but the fact that you require an entirely different company's orchestration tools to make AWS "not that hard to orchestrate" is evidence of how poor the UX of AWS is.
But terraform is using the aws sdk behind the covers.
Most of the stuff available with terraform is available with CloudFormation minus a thing or two. (takes them a minute to catch-up to other teams new services/features)
You can use Cloudformation. Like, what do you want here? Some super console that nobody asked for because anyone who knows what they are doing knows why you want infrastructure as code?
Might be a war-crime, but people at the enterprise level like twilio, don't do anything at the UI level, and if someone was to do something at the UI level, AWS blocks public access _by default_
AWS console was only complicated as a complete noob. Once you understand how services work you actually realize is decent.
These only seem “vaguely connected” if you haven’t studied them together. IAM and S3 are intricately intertwined and you should understand both at at least an intermediate level to work with them in sensitive environments. (Note that IAM has permissions specific to each Amazon service, all documented in detail.) None of the rest of the systems you mentioned gate write access, though certainly if they’re in your production flow you should study and review them as well.
If you have CloudFront in front of your bucket, the bucket shouldn't even be world-readable, let alone world-writable. Permission should be delegated to CF to read out of the bucket.
>That said, I don't blame Twilio for not catching this
I do.
Twilio is a multi-billion dollar company and there is no excuse for them not having proper security processes to catch stuff like this early. Even if we take the "S3 is hard" arguments at face value, this wasn't a 0-day or some complicated unpredictable exploit. This was an extremely basic misconfiguration on a mission-critical part of their architecture that would have been caught by even the simplest out-of-the-box penetration test or audit. For a company like Twilio to not be doing basic, fundamental stuff like that is a big deal and they certainly should be blamed for it.
your S3 bucket permissions are clearly worthy of periodic review.
better tooling makes this easier to do.
And you should have this done by somebody who knows it well, or who learns it well in the process.
No one knows S3 better than the AWS S3 team.
It took us half a day to get the bucket locked down to where only the root AWS IAM user—secured by physical 2fac—was permitted to change the SDK
Or a small feature team could spend a month or two improving the user experience so that literally every S3 customer doesn't have to have the technical acumen, knowledge of risks, and time to get this right. 0.5 days * a million users (to be conservative) is 500,000 days.
I agree AWS can do a lot better with the tooling UX, but adding write permissions like this is a conscious decision. In this case, someone purposefully added it (outside the creation UX flow). I'm not sure we can put the blame squarely on AWS for this one.
And they definitely haven't been doing nothing. They've drastically redesigned the permissions UX around S3 over the years. They introduced GuardDuty to allow automatic threat detection (such as S3 bucket having open permissions) which can be applied across your organization (not just an account). They definitely have to keep doing more, but implying they are doing nothing just isn't true.
> They've invested all their resources into building out the infrastructure, and have spent very little if any time on building the UX around it.
In AWS you need to explicitly set a bucket as publicly accessible, which requires you to go through a couple of dialogs where you're expected to say "yes, yes I want to allow others to access this S3 bucket."
Afterwards you still get a bucket that does not have write access by default. To allow write access you need to specifically set bucket access policies that explicitly allow others to write to your bucket.
In the past, S3 buckets were created with much less sane defaults and you didn't have the ability to unilaterally forbid any public access that you do today. Also, configuring Bucket Profiles is a confusing and hard to test situation. And if that weren't enough, S3 buckets also have ACLs (though they are deprecated) that represent yet another way to configure access to your buckets.
So, from the confusing UX to the multiple permissions settings, I can easily see how a company could get in this situation. In fact, the problem is so bad that AWS has revised the whole S3 thing several times, with each new iteration more secure than the last. Unfortunately, that doesn't help the company that configured their buckets 5 years ago and haven't touched them since.
I disagree. They started by just providing the plumbing, and Twillio is using that plumbing.
But today if you make an S3 bucket, they take you through a wizard that makes it secure by default. They also run audits for free and send you the results.
They also have Lightsail that just does everything for you securely.
I'm not seeing any UX problems here. They configured an S3 access policy incorrectly while troubleshooting, 5 years ago.
FTFA:
> During our incident review, we identified that this path was not initially configured with public write access when it was added in 2015. We implemented a change 5 months later while troubleshooting a problem with one of our build systems and the permissions on that path were not properly reset once the issue had been fixed.
This is exactly one of the things Terraform would catch and correct in a snap. I rather think Twilio was so confident before they had everything understood and smooth running that they didn't need other tools like Terraform. Infrastructure as code emerged as a thing EXACTLY because cloud's hundreds of consoles and checkboxes and whatnot, although each one simple and logical, spinned out of our mind's capacity to hold.
I agree to an extent. AWS is beginning to feel like a duct taped spaghetti mess. It is becoming increasingly difficult to do things “right” without getting deep into their confusing documentation.
However, I’m not necessarily sure that it is the wrong approach. You can use AWS at almost any level you like and if you go deep into the woods with any service you’re eventually going to find yourself buried in documentation.
Amazon probably can't change S3 for backwards compatibility reasons (there was an outcry about just changing some legacy URLs last year), but they definitely can and should make an "S4" which would be an S3 compatible storage layer with sane defaults.
Essentially, people only ever want one of two things:
- A private FTP bucket
- A public static asset website (with built-in CDN and HTTPS!)
Thing people never want:
- A public upload bucket
Seriously, no one wants that in any circumstance, ever. It's a huge liability. I literally cannot imagine anyone wanting anything even similar to it. But many, many companies have accidentally made one with S3.
While they're at it, S4 should also actually use IAM and all that other "modern" stuff that S3 doesn't have because it's so ancient.
>Thing people never want:
>
>- A public upload bucket
Thus is outright wrong. Public upload s3 buckets are AWS's go-to basic technique to handle file uploads. AWS's API Gateway has a max file limit, and AWS advises developers to leverage S3 and lambda triggers to implement that usecases.
My immediate thought was about their Authy Desktop app - looks and feels like a web app in a container. If someone gets access to that then boom, all your 2FA tokens are stolen.
Twilio's examples[0] for taskrouter.min.js, even now, don't include Subresource Integrity hashes. Subresource Integrity would have stopped the modified script from being loaded, and removed a single point of failure.
Subresource Integrity is supported by every evergreen browser and 80%+ of market share (no IE11/pre-Chromium Edge).
Allow admins to "down feature" the account. Ie, simple mode.
Basically, allow admins to toggle OFF S3 ACL's and even S3 Bucket Policies - zero those out. Then let users use IAM policies for access which may be a more familiar / more used mental model.
A better approach is simply not to grant normal users modify access to IAM, VPC, or anything else potentially destructive, and force them to make all changes via a ci/cd pipeline with ops/sre oversight and review.
Thanks for posting this. I'm really impressed with the transparency Twilio showed in actually admitting to having such a silly, silly bucket policy. Not impressed that it was there in the first place; but that should go without saying.
This incident report should really put to bed all of the "It's AWS's fault for making things so complex" complaints. (To be clear, it won't... but it should.)
Even a cursory look at that bucket policy should tell you something named "Allow Public Read" should NOT be associated with anything named 'Put'. This takes 0 AWS knowledge to figure out.
Really not impressed with the obligatory "really impressed with transparency" pat-on-the-back under every incident report for a big corp screw-up that provides any details at all.
And stating to the press the clearly malicious payload is "non-malicious" (assuming TFA didn't lie about Twilio's statement)? That's ridiculous.
When talking about screw-ups on AWS, public incident reports try to obfuscate and spin the Hell out of issues that boil down to "really, really stupid configuration issue".
They owned it. That is more than can be said about other large incident reports that I've seen regarding AWS.
This wouldn't have happened if they just used an old school nginx file server. It has sane defaults like never letting random people edit your files through an API. Oh and it also would have been cheaper than S3. Just sayin.
161 comments
[ 7.2 ms ] story [ 221 ms ] threadAnd you wouldn't get much sympathy, but this is a legally viable complaint. When my car was stolen, the thief was charged for theft, but his girlfriend who was joyriding with him was charged for criminal trespass. Just being in the car when you don't have permission is illegal.
It isn't necessary to actually "break" anything.
Edit: Of course, it's still not a great metaphor because it implies there was at least some kind of barrier to entry that had to be pushed aside - but from the sounds of it, there wasn't.
https://developer.mozilla.org/en-US/docs/Web/Security/Subres...
However, on Twilio's documentation site, they do not include the integrity attribute in their examples: https://www.twilio.com/docs/taskrouter/js-sdk/workspace/task...
Even putting the security arguments aside you can bet a lot of customers would be really irritated having to constantly update this stuff on their site. If a competitor didn't have that restriction it would become a selling point. IMO the ideal is that Twilio provides both options: a "sdk-latest.js" that's a moving target, as well as "sdk-v1.3.js" that is frozen and can be locked with subresource integrity.
Regardless of the severity of the bug, the only-case scenario is that all the sites you have pulling from that CDN break until you recompute the hash. How annoying this is scales directly to how frequently your libraries have to release vital security bugfixes.
Until you recompute the hash and communicate that new hash to them and they implement it on their site. It’s not nothing from an implementation point of view.
In reality, no you cannot.
You have multiple layers of cache between the S3 bucket and rendering, unless you disable caching entirely at a massive increased cost. Some of these caches are poorly behaving (e.g. intermediary caching).
The correct way of doing this ALREADY is to increment the version number in the URL (e.g. /3.1.0/My.Lib.js to /3.1.1/My.Lib.js) and to re-point the pages to use the updated library. This is reliably cache breaking, and will assure end users are getting the bug fixed version faster (or ever in some cases).
Once you're already doing it correctly, Subresource Integrity is a freebie. There's a reason why "sdk-latest.js" is largely dead concept from a bygone era: it is a huge anti-pattern and anti-feature.
Heck allowing upstream to blindly push you changes without being informed is nuts regardless. This argument is essentially: "I'm doing it wrong, and Subresource Integrity would stop that, so it is a non-starter." Without considering that what you're doing is bad practice before Subresource Integrity joined the party.
> There's a reason why "sdk-latest.js" is largely dead concept from a bygone era
Is it, though? From a quick check, it's what Google Maps does. It's what the Facebook SDK does. We already know it's what Twilio does.
> This argument is essentially: "I'm doing it wrong, and Subresource Integrity would stop that, so it is a non-starter."
Zero dispute with that characterisation from me. But it's how things already operate in the real world. I'll join you in shouting from the rooftops that people shouldn't be doing it, but that doesn't really get you any closer to actually stopping them. For a great many people the flexibility to quickly push up changes is a feature, not a bug.
Nope. Google Maps loads an uncachable JS file that itself points to version specific sub-files that are cache breaking.
> It's what the Facebook SDK does.
Nope, they have versions in the fbAsyncInit configuration.
> We already know it's what Twilio does.
Indeed, and if you want to copy a company that almost had a massive security problem then go right ahead.
I only know this because a friend talked about a case where they rotated a car in-place, and there was a question about is that theft? its center of mass is still where they left it, so it wound up not being theft. Moving furniture in a house is a weird one. Probably leans on ill intent, moving a chair to block a door seems like it's pointed to giving the owner a hard time. moving a chair to perform cpr has much purer motives.
So anyway, there's my ill-informed view.
A common S3 use case is: Use a Bucket for Read-only static content, (js, html, imgs). From a dev perspective ideally it will work like a protected folder + Web Server, whereas the webserver will have read-only privileges, but reality is far more complicated.
I can understand why was easy for Twillio to have this infosec issue, since we've done it ourselves many times when using S3.
If it’s cross account then allow assuming to other accounts, but no reason to bother with the bucket ACL.
Leave the bucket policy blank and you’ll never have to worry about an open bucket. Better yet make a deny rule to everything but a single role, and you don’t have to worry about rogue roles exposing the bucket.
Perhaps some of the issue is that permission can be managed on both IAM and a bucket policy?
You already have a role in AWS that you use. Go into the console and say “this role can do S3 to this bucket” and you’re done.
I'm sure an expert finds it all very natural. Security is always hard and has to be done right the first time. That's a domain that's always going to deter novices, and perhaps the only way forward is to hire an expert to ensure you're doing the right thing.
Still... I've been programming for four decades and it felt a little odd to be quite so out of my depth. It's not the same as anything I've done before, but it felt like it shouldn't have been quite so bewildering.
But, as others have said, the documentation at times is poor at best, and it can be easy to make mistakes.
Some preventives every ops team can take are:
1. Never make manual permission changes in the console. Everything is done via SCM’d code.
2. All permission changes are peer reviewed.
3. Create logical separations of buckets so it’s more obvious what the permissions should be of a bucket. You can go so far as to have separate accounts that are for public uploads, separate accounts for public assets, etc. Avoid granting elevated permissions at a folder level, which is harder to understand as a developer without access to view the ACLs.
Good luck everyone!
The first time I heard “infrastructure as code” I thought it was some dumb new buzz words, but no, it’s good advice. Your cloud formation, serverless, terraform whatever should be stored in GIT a stack replace is how changes should be made.
I consider the console as read-only.
The web console should be just another UI for editing the configuration in source control. As an example of why that's important, a UI can give you a warning and use affordances to guide beginners away from dangerous setups. In the case of S3, a slider for basic access level can use colors and show a list of objects that would become world readable or writable at a given level (for that matter why is world writable even an option?). A giant YAML file looks the same whether it's safe or dangerous.
If you really don't want to use a UI, then maybe abstracting a linting layer that feeds the exact same warnings to the UI as to your CI system and text editor would help.
Should. But isn’t.
So until it is, IaC for production. Console or CLI for development.
I do the same, and sometimes I need to do something manually so that I can grok what configurations I need to define in terraform / whatever other tool I'm using. POC'ing is different than a production or even staging service.
It would be much harder make that mistake with Google Cloud Storage, at least in the web console. GCS also supports disabling bucket ACLs permanently at bucket creation time, and that is the option they recommend[1].
AWS does scan and alert for some public bucket issues. But with all the public S3 issues, I think it is clear there needs to be more prominent warnings in the AWS console, the command line, and with tools like Terraform.
[1] https://cloud.google.com/storage/docs/access-control
S3 does too. There's an entire page during the creation wizard that is dedicated to blocking any and all public access, even causing the bucket to ignore ACLs or other settings that would otherwise expose the bucket publicly. All public access is disabled by default, and enabling it actually requires the user to actively uncheck 5 different checkboxes, each of which explains that unchecking it will open the bucket up to public access, and then even requires an additional attestation, in a large orange warning box, that says you acknowledge that the options you chose will result in the bucket being public.
I'll be the first person to tell you that AWS is overcomplicated and hard to use, but this isn't that. IAM and bucket policies are a pain to work with, but if you screw up with those, the worse you will do is expose your bucket internally. But exposing a bucket publicly to the internet is an entirely different act, and there's not really any excuse for it other than just not reading the directions.
However, neither this option in S3 nor the option you linked in GCS would have served Twilio's use case. They wanted their bucket to be publicly accessible, just not publicly writable. That's an entirely different access management issue.
Before using AWS I was in the 'what morons!' camp whenever some company got breached via leaving things wide open on AWS. Now I'm in the 'ok I understand why its so common' camp.
You'll even see this in their sales pitches. AWS will spend so much time talking about the most advanced use cases and how they are possible, but if you ask them a simple question about how to run a single bucket that doesn't also involve using all of their ML tools and global accelerator and cloudfront, they'll be blindsided. It's almost as if they consider the common use cases to be "too simple/obvious" and therefor they never bothered to create any documentation or put any thought into it.
But with that said, one of those layers of defense is also using tools that are easy to keep secure so that the vulnerability doesn't appear in the first place. While Twilio isn't "a lone developer", I wouldn't be surprised if the person who did create that bucket on behalf of Twilio was just a lone developer fumbling around in the console. And again, that's no excuse, but it still is an area for improvement.
Now, as has been stated before, reading any cloud vendor documentation is like reading the Oracle manuals of years ago, not a happy experience.
Granted, these clouds now do things much more powerful and target more difficult enterprise scenarios. One can miss Heroku, and AWS has Elastic Beanstalk for that, but EB gets much more difficult fast.
I understand someone brand new to S3 being in the dark but someone in your PRs approval chain should have enough Ops skill to know to check these things if you're not going to hire an actual Ops person. And no, none of this is hard for anyone who has ever touched S3. Ops teams have known about these issues for over a decade. It's practically a meme at this point.
The number of heads on my project is ...4. We're it. There is no place or person to just push the work off to. There is no PR approval chain. We regularly meet to sync and discuss plans and thats it.
Until very recently all of our infrastructure was in house, right down the hall from me. I (or whoever was in that day) took care of making sure it was running smoothly.
Cloud is new to us, and there is no budget to hire a guy who manages AWS full time. That is the case at an insanely high percentage of software shops. Heck I would wager most people outside of 'new tech (sorry, I can't think of a better term)' shops are using point & click in the web UI to manage AWS. There is no code in source control that manages our infrastructure. I'm jealous of the people that get to do it all the 'right way' but for many there isn't enough manpower or time in the day to do it.
Now why we're in this position in the first place is another debate about why management thinks we need to go cloud at all. But its reality for many.
I wish MORE small dev teams brought in ops people and didn't wait until they had a staff of 20-30 or whatever it typically winds up being. But I know that's a pipe dream and you're basically losing a developer salary to hire a devops/ops person which isn't the direction they want to go.
edit: typos
* the cost of auditing would be spread across all customers equally, punishing competent customers and benefitting incompetent ones..
However I'm not sure if there are any built-in tools to find weird nested policies attached that could cause a similar permission to be granted to all..
If you ask for help from AWS, AWS will provide it. It may not be free, but it's available.
Even if AWS were to start proactively auditing customer setups, how in the world is AWS supposed to know what a customer's usecase is? Nevermind the fact it's a breach of the customer's privacy to just go rooting around in the customer's account without permission.
But let's assume AWS is going to take responsibility for customers' configuration decisions and violate customers' privacy by proactively auditing their accounts. Would AWS auditing Twilio's configuration here work?
The default is for S3 buckets to be private. The customer has to take specific, affirmative steps to give s3 buckets public access. You really have to jump through hoops to make a bucket public accessible.
Since the Twilio chose to make their bucket public, AWS auditing Twilio's setup wouldn't be helpful. AWS would just assume the Twilio knows what they're doing. How is AWS supposed to know there is a misconfiguration? Because Twilio clearly decided to make their S3 bucket publicly available.
https://aws.amazon.com/macie
I don't bother because it seems pretty clear what buckets are public.
That said, quick tip to make your life easier.
DON'T use S3 ACL's DON'T use S3 policies.
If I was AWS and didn't have so many customers I'd probably just create one mental model (IAM policies probably) as the place to manage things and block the rest.
"Someone" was able? If the bucket was unprotected and world-writable, then _everyone_ was able.
> 'non-malicious'
> And judging from the URL involved, it appears to be an attempt to install a payment-card skimmer – RiskIQ has spotted the same URL in other S3 buckets targeted by miscreants.
In what world is that non-malicious? Non-immediately-damaging _maybe_, but it seems clear that there was a malicious chain of reasoning involved. Twilio got super lucky if this is all that happened.
This is anecdotal, of course, but it seems to me that any company that didn't start cloud-native is going to have flaws like this, and it's often not at the fault of developers but at the fault of top-down initiatives to "get to the cloud" ASAP instead of correctly.
This requires a Bucket Policy that is written explicitly to allow this. I don't want to make assumptions because of course we don't know all the information and I personally know very little about infosec, but that sounds very strange.
Why would you do that? I assume the SDK gets uploaded from a secure environment that has the credentials to upload objects, so there's no legitimate reason to have writing permissions open to the world.
> [...] a 2FA verification code sent to your phone (via a call, SMS message, or an authentication app like Authy).
PSTN-based 2FA in 2020 is not a vote of confidence in the security team's competence.
A lot of companies go with email and/or PSTN based 2FA for customer convenience and to get them to use 2FA for their accounts.
I try to avoid using 2FA for accounts that only offer PSTN or email based 2FA because it's less secure than my randomly generated long password. It would be nice if more services offered 2FA based on client side TLS certificates.
1) When not implemented properly, it is treated as an alternative authentication mechanism, not an additional required mechanism. My ultra strong password is useless if it's enough to steal my phone number and convince customer support that the attacker simply forgot the password.
2) It gives a false sense of security. I already have 2FA enabled! What do you mean I should be using TOTP/FIDO2/TLS client certificates?!
3) It reinforces an unfortunately widespread security anti-pattern, which reduces security for Web users in aggregate. So what if it's not a great idea? Everyone is doing it! Or put another way, MD5 password hashing is better than storing passwords in plain text. But please, please, please do not use MD5 password hashing! There are much better ways to accomplish that goal without introducing more flaws.
It seems this incident has reminded Twilio to audit all their access right, which is a net positive.
Actually, imagine you were an employee who wanted to get this done, but couldn't get management approval? This is one way to fix things ... :)
a. adopted the resource into a Cloudformation stack & a1. Enabled drift detection.
b. Use an AWS config rule to monitor (appropriate) s3 buckets for any public access.
That's not a AWS problem. That's a "our team failed to manage the service" problem.
This includes monitoring a publicly accessible bucket.
AWS is getting really confusing, IAM is not simple, which is bad.
But S3 is a bread-and-butter service, and the policy is such that anyone using it should understand. I mean it says there in 'almost plain English' than 'anyone' can write content.
This is a pretty major blunder, because it hints that maybe they don't have any security reviews at all.
If I, a completely non security-knowledgable person were asked to do an audit of Twilio, I would find this issue. It's way on the lower treshold of 'obvious problem'.
Twilio is not 'Medium Blog' - they provided IT plumbing and therefore it's doubly problematic.
"Specifically, the modification added code to the end of the TaskRouter.js v1.20 SDK that made an HTTP GET request to hxxps://gold.platinumus.top/track/awswrite?q=dmn and followed the URL returned in the HTML by that request."
Translated: we can't possibly know it's non-malicious.
TFA even has this to say:
> And judging from the URL involved, it appears to be an attempt to install a payment-card skimmer – RiskIQ has spotted the same URL in other S3 buckets targeted by miscreants.
Details in the linked blog post[0].
So, very much malicious.
Why the hell they included "non-malicious" in the title (granted, in quotes), I don't know. Readers could have easily dismissed this as actually non-malicious.
[0] https://www.riskiq.com/blog/labs/misconfigured-s3-buckets/
If you hit it directly they will redirect to a random "Spin the wheel win a prize" type site in an attempt to hide the malicious nature of the payload.
A failed drug dealer (who got caught), by your definition, should get less time because they never managed to sell the drugs they were transporting over the border and got caught :)
The impact of these actions matters far more to the victims than whatever consequence the perpetrator may face.
If your website was affected by this hack, surely you’d care about how your customers were impacted.
Lots of people here keep repeating the claim that this was magecart, that’s just not true.
They might as well have said "we are not smart enough to figure out the nature of this code. In unrelated news we are hiring senior JavaScript engineers, experience in opsec is a bonus"
Report: https://any.run/report/e8c581d52ca3527b61c7ddd9aba5dfeaf9440...
Original URL: h t t p s : / / gold.platinumus.top/track/awswrite?q=dmn
It's a really simple service that fetches a list of script URLs that you provide, and notifies you with the before/after diff whenever the file changes.
I found GuardScript through a Show HN a while ago (https://news.ycombinator.com/item?id=20265141) and have been very pleased with it.
https://developer.mozilla.org/en-US/docs/Web/Security/Subres...
For example, here is Stripe's script tag:
<script src="https://js.stripe.com/v3/"></script>
SRI only works for static files whose contents never change. In this example, it would mean that Stripe could never push minor bug fixes to the SDK.
The workaround would be to use static versioned files instead of a generic script URL (or self-host the script), so that the value of the SRI checksum never changes.
Edit: In the case of Twilio, their docs say "By linking directly to our CDN in production, any patch-level (non-breaking) changes made to the SDK will automatically be applied [...]" meaning that if you link to Twilio's CDN, you can't expect the file to be static, and therefore can't use SRI.
They've invested all their resources into building out the infrastructure, and have spent very little if any time on building the UX around it. You have to be fairly technical, never mind painstakingly detailed-oriented, in order to manage their services, a big chunk of which is hidden behind black screens and various control nobs.
Amazon's logic is probably to provide the piping, and leave the rest of the end user, but at this point it's not enough. In my opinion, they need to heavily invest into the customer-facing layer of their applications that shows all the status, gaps and holes, or these things keep happening.
Then again, they can continue as is, and the incidents will keep as is, and as long as nothing implodes and no big customer leaves, financially it makes sense to do nothing. Just thinking out loud.
Looking forward to finally try Pulumi in a project.
Ever had CloudFormation yell at you because some function like "Sub!" only accepts alphanumeric characters? Created complicated constructs because there are no variables, and very limited conditionals? Copy-pasted templates like an insane person because there is no way to abstract anything?
Not my idea of fun. All of this is solved by using a proper programming language.
The biggest paint point with something like Terraform is the fact that it can't actually validate cloud provider specific problems (for example declaring an Compute instance with a compute type (string) that doesn't actually exist). Does Pulumi try to address this in some way (granted its much easier to write your own checks like that in language as opposed to writing a go module for Terraform providers).
Yes, the AWS Pulumi SDK package includes a discriminated union of valid instance types, and the type of the property `aws.ec2.Instance::instanceType` is that union.
(Disclaimer: I am a former maintainer of Terraform and a contributor to Pulumi).
I worked for a company where our CTO identified the same thing (unauthorized modification of JS SDK served via S3) as a potential vulnerability. It took us half a day to get the bucket locked down to where only the root AWS IAM user—secured by physical 2fac—was permitted to change the SDK, and then only after "unlocking" the configuration changes. Since the SDK was not updated frequently, having a little manual ceremony was fine. If it had been a high-touch path we would have had to do a little more engineering to keep it safe, but not that much more.
That said, I don't blame Twilio for not catching this; shit happens at a company with years of legacy built up. I do blame anybody who says "S3 is hard" and throws up their hands. You're a professional! Read the docs, they're dry but quite extensive! Play around with it in a test bucket! It's no harder than learning any other moderately complicated professional system. If you can learn Rust, you can learn S3's API. If you can learn Kubernetes, you can learn S3 ACLs and the IAM authz model. It may not be as interesting as the topics you want to read about, but it will probably be even more useful.
To do anything in AWS there's at least 3 or 4 vaguely connected services, including IAM and RAM as a completely separate UIs. Usually with S3 you also have Cloudfront or some other CDN. You probably also have Cloudwatch logs and CLoudtrail event tracking. You might have VPCs involved. Some or all of these things, in an org like Twillio, could have entirely different teams managing them.
The problem isn't that S3 is hard. The problem is, as OP suggests, the UX of actually doing anything non-trivial across half a dozen services is somewhere between abysmal and war-crime.
If you’ve got one team managing your logging who don’t talk to whatever team who manages your s3/cloudfront setup, then that’s your problem.
I love Terraform, but the fact that you require an entirely different company's orchestration tools to make AWS "not that hard to orchestrate" is evidence of how poor the UX of AWS is.
Most of the stuff available with terraform is available with CloudFormation minus a thing or two. (takes them a minute to catch-up to other teams new services/features)
AWS console was only complicated as a complete noob. Once you understand how services work you actually realize is decent.
Thanks for the giggle.
I do.
Twilio is a multi-billion dollar company and there is no excuse for them not having proper security processes to catch stuff like this early. Even if we take the "S3 is hard" arguments at face value, this wasn't a 0-day or some complicated unpredictable exploit. This was an extremely basic misconfiguration on a mission-critical part of their architecture that would have been caught by even the simplest out-of-the-box penetration test or audit. For a company like Twilio to not be doing basic, fundamental stuff like that is a big deal and they certainly should be blamed for it.
better tooling makes this easier to do.
And you should have this done by somebody who knows it well, or who learns it well in the process.
No one knows S3 better than the AWS S3 team.
It took us half a day to get the bucket locked down to where only the root AWS IAM user—secured by physical 2fac—was permitted to change the SDK
Or a small feature team could spend a month or two improving the user experience so that literally every S3 customer doesn't have to have the technical acumen, knowledge of risks, and time to get this right. 0.5 days * a million users (to be conservative) is 500,000 days.
And they definitely haven't been doing nothing. They've drastically redesigned the permissions UX around S3 over the years. They introduced GuardDuty to allow automatic threat detection (such as S3 bucket having open permissions) which can be applied across your organization (not just an account). They definitely have to keep doing more, but implying they are doing nothing just isn't true.
In AWS you need to explicitly set a bucket as publicly accessible, which requires you to go through a couple of dialogs where you're expected to say "yes, yes I want to allow others to access this S3 bucket."
Afterwards you still get a bucket that does not have write access by default. To allow write access you need to specifically set bucket access policies that explicitly allow others to write to your bucket.
So, from the confusing UX to the multiple permissions settings, I can easily see how a company could get in this situation. In fact, the problem is so bad that AWS has revised the whole S3 thing several times, with each new iteration more secure than the last. Unfortunately, that doesn't help the company that configured their buckets 5 years ago and haven't touched them since.
But today if you make an S3 bucket, they take you through a wizard that makes it secure by default. They also run audits for free and send you the results.
They also have Lightsail that just does everything for you securely.
They have lots of user friendly options.
FTFA:
> During our incident review, we identified that this path was not initially configured with public write access when it was added in 2015. We implemented a change 5 months later while troubleshooting a problem with one of our build systems and the permissions on that path were not properly reset once the issue had been fixed.
> { "Sid": "AllowPublicRead", "Effect": "Allow", "Principal": { "AWS": "" }, "Action": [ "s3:GetObject", "s3:PutObject" ], "Resource": "arn:aws:s3:::media.twiliocdn.com/taskrouter/" }
Obviously you have to be using CloudFormation, so if they're using i.e. Terraform they're out of luck.
AWS Config is easy to setup and can be used to warn for that type of access-control issues. It's a good layer of additional defense.
The new S3 UI is also much better and won't let you do insecure things by default.
AWS also sends emails when they find an open bucket. I don't know exactly when those are triggered but I got them in the past.
However, I’m not necessarily sure that it is the wrong approach. You can use AWS at almost any level you like and if you go deep into the woods with any service you’re eventually going to find yourself buried in documentation.
Essentially, people only ever want one of two things:
- A private FTP bucket
- A public static asset website (with built-in CDN and HTTPS!)
Thing people never want:
- A public upload bucket
Seriously, no one wants that in any circumstance, ever. It's a huge liability. I literally cannot imagine anyone wanting anything even similar to it. But many, many companies have accidentally made one with S3.
While they're at it, S4 should also actually use IAM and all that other "modern" stuff that S3 doesn't have because it's so ancient.
Second Simple Storage Service.
Thus is outright wrong. Public upload s3 buckets are AWS's go-to basic technique to handle file uploads. AWS's API Gateway has a max file limit, and AWS advises developers to leverage S3 and lambda triggers to implement that usecases.
[1] https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-post-e...
[2] https://docs.aws.amazon.com/AmazonS3/latest/dev/PresignedUrl...
Time to find an alternative.
Subresource Integrity is supported by every evergreen browser and 80%+ of market share (no IE11/pre-Chromium Edge).
[0] https://www.twilio.com/docs/taskrouter/js-sdk/workspace/work...
Allow admins to "down feature" the account. Ie, simple mode.
Basically, allow admins to toggle OFF S3 ACL's and even S3 Bucket Policies - zero those out. Then let users use IAM policies for access which may be a more familiar / more used mental model.
(I work for Twilio)
This incident report should really put to bed all of the "It's AWS's fault for making things so complex" complaints. (To be clear, it won't... but it should.)
Even a cursory look at that bucket policy should tell you something named "Allow Public Read" should NOT be associated with anything named 'Put'. This takes 0 AWS knowledge to figure out.
And stating to the press the clearly malicious payload is "non-malicious" (assuming TFA didn't lie about Twilio's statement)? That's ridiculous.
They owned it. That is more than can be said about other large incident reports that I've seen regarding AWS.