Interesting. The conclusion you draw from this differs wildly from my conclusion.
Whenever I read stories like these, it seems clear to me that someone moved to the cloud in order to not have to care about security. The 'cloud does everything for you!'. Just like you imply in your answer that PaaS, the next level of abstraction, will solve all your security problems.
This move, however, will inevitably lead to a situation where people work with new and complex systems that they don't understand (remember: not having to understand them is the sole reason they use them). Unfortunately, working with complex systems you don't understand is the number one reason for vulnerabilities in the first place.
I am not convinced that a service exists that abstracts security away from you.
There's another side to this. The people deploying things with a security group open to the internet and no auth might have been saved by a grumpy network admin forcing them to use a VPN or even a firewall, but that's IMHO only hiding the real problem ( security not taken seriously) behind a thin fence. When that fence gets breached, everything would be up for grabs, so the main difference with "the cloud" is that such terrible security postures are easier and faster to spot ( from both sides).
It depends which kind of security and which provider.
I would trust Microsoft more on patches/configurations for an email system than something which is managed on premises. You need to have really good people to maintain a good level of security. Not only technically good, but also with a string cold management that will force updates even if it means the CEO will not get his maol for 15 minutes - and say that this is life and that the discussion i sover.
On top of that, MS would (I hope) install patches on their customer-facing systems in advance of an official patch release.
The above applies to the majority of large SaaS services.
Now when you have a "Platform", a hoster that requires you to bring in knowledge and not only data then it gets dangerous. You need to maintain the security of what you bring in. This can be an OS (your "Platform" provides VMs), or code (your "Platform" provides code runners). Unfortunately, when a company moves to the cloud, they sometimes forget to do this assessment and end up with monstrosities they installed themselves (which is not different, security wise, from having it on premises - augmente nu the 7B population that potentially has now access)
I think the word "managed" is the clear differentiator here. There is a huge difference between setting up ElasticSearch on some EC2 instances yourself, and paying ElasticCo for a managed cluster.
Managed services don't save you from that. You still have to configure security groups to reach them and a lot of devs just use the easiest way and expose the thing to the internet.
This same article was posted previously, so I will post the same response I posted on the other thread:
Elasticsearch until recently did not nudge you to set up a username or password by default. I noticed the last time I installed it on a fresh instance that on completion of the install it gives you a warning about this and tells you what to do to set a password. That is a small improvement.
Most people would not have the service bound to a public interface, but for those who do for whatever reason have a set up where they are accessing it remotely, at least now there is a tip off that it is completely open to the world by default. This is different from pretty much any other service you might install. MariaDB for instance by default does not allow remote root login even if you change the config to bind it to 0.0.0.0.
A lot of people are just totally unaware of this issue. When I read about database leakage, I generally assume 90% probability it was elastic, these days. Defaults are so important.
My solution to this has always been to auto-generate a password and dump it in a by file.
If it's a test instance then this is a nice obvious place to go get it from. If it's not then the service isn't open by default, and in both cases we're not asking people to enable remote access themselves by some scheme which may allow them to do it without setting a password.
Same thing with TLS IMO: let certs be specified, or default to generating some - any option is better then default no encryption.
There are lots of lessons in the internet's limited history that show poorly conceived defaults are the most common and dangerous mistake. Microsoft let Frontpage Extension credentials default to the domain name and a blank password through multiple major versions, without any prompt to change anything (you had to seek out the settings, and it wasn't even obvious where they were). Throughout the late 90s and early 00s, entire hosting companies were continuously compromised because of this, along with nearly every school district in the US and tons of major universities. Schools typically had the worst security and most available processing power and bandwidth, already making them the preferred targets of script kiddies everywhere.
That exploit was single-handedly responsible for hundreds of thousands of bots (and more likely millions in later years) which wreaked havoc on the internet as a whole -- these were the days when a handful of bots on good connections could take down the Amazons of the world for lengthy periods (days, not hours). That they let it go on for so long was comical at the time, but in hindsight seems criminally negligent.
I remember hearing about FrontPage Extensions back in the day. I never had an opportunity to make use of them since I used Dreamweaver at the time, but what exactly were they? I tried googling, but couldn’t find a plain language explanation, only opaque marketing speak.
Extensions were a way of connecting the FrontPage editor directly to an IIS server, which provided publishing functionality (i.e. uploads) and basic server management. Ultimately, they were just a bunch of server scripts that responded to commands from the client over an FTP-like connection.
Interesting that ES is still such a widely used component, this is a huge red flag about a software product. And of course there are lots of other regular complaints about it (eg uses a lot of memory and wants a 3-node cluster so costs 4 figures/mo to run on AWS).
Network segregation is your last line of defense. Having anything rely on it is a recipe for a bad security that's always just one step away from someone getting around it due to misconfiguration, request forgery, networks configuration changes over time, malware transiting over via VPNs etc. And of course from the SW vendor POV they don't know if the customer env employs this defense in depth layer, so it's really irresponsible to rely on it. Like is amply demonstrated here...
If a product upon unboxing promptly flops on its back with "come here internet" access controls, even if by good fortune it's saved by your network ACLs, it's time to put it back in the box and return it.
(not gp) I would prefer having a link to the other post (as someone above provided) so the comments there can be read. Duplicating the comments means I see the same thing twice, or it means the previous replies to that comment are now obscured to me. Which of the two it is depends on whether I find the other post. Not a desirable situation imho.
Secure defaults are super important, especially on complex systems as people will not want to mess with the out of the box configuration too much when setting things up.
One additional consideration is that the secure defaults should be usable, otherwise the tendency will be to just disable them entirely (think SELinux and how commonly that's turned off as the first action on server setup)
This has been going on for years with AWS. Unfortunately, there has always been unprotected data sitting on AWS for anyone to read.
This Google Search returned an extremely high number of mentions of various companies' ills, a high signal to noise ratio. It's very saddening to see that.
I used to think you have to be pretty good af your job to get trusted to deploy stuff to the cloud for even medium sized companies. As these articles keep reminding me, you only need to fake competence to management to get the green light.
Even if you forget that the cloud is the internet and that the entire internet can reach you over the internet, it doesn't take a genius to set up a password for a cloud service. I have no idea how many of these databases are honey pots, but finding open and vulnerable servers is depressingly simple.
I suppose it's kind of liberating to know that you can be dumb enough to fling patient data into an unprotected cloud server and still get a job in IT. The bar is really set that low.
> As these articles keep reminding me, you only need to fake competence to management to get the green light.
Well, managers aren't scientists looking for truth. They are people running businesses that need to stay afloat and get stuff done. There's a lot of wiggle room there, mostly due to time pressure (from a charitable perspective, less charitable: also because of company politics).
With so many 10x full-stack developers who're just looking to get sh done and clock out, it can't be otherwise. If managing a server becomes just another bullet point in a developer job, this is exactly what we get.
Sure I can deploy stuff, but you shouldn't trust me. Unfortunately many are not so honest or even aware of the complexity (unknown unknowns).
> If managing a server becomes just another bullet point in a developer job, this is exactly what we get.
Yup...the same management are the ones that laid off the DBAs, system admins, network engineers, etc because "the cloud" hand waves all this stuff away now.
I work for a large cloud provider and it’s even worse than you imagine because this same thing has happened to us. I don’t work on cloud, but we use cloud internally for our other products and devs on those teams are expected to manage all of this themselves with little guidance. We have no dedicated ops or infra person on my team. Everyone is just considered a generic software engineer. This wasn’t always the case until the recent push for devs to own all parts of the stack.
"I suppose it's kind of liberating to know that you can be dumb enough to fling patient data into an unprotected cloud server and still get a job in IT. The bar is really set that low."
On the other hand, I would argue that there are way too many footguns hidden everywhere, like the default binding of docker mentioned here. So yes, a highly skilled IT security professional knows them all, but they are rare. And I think it should be possible to set things up securely in a straight forward way, for people with their speciality in other areas, of which there are plenty.
I don't know how you idiot proof a Turing complete language, but I don't imagine it would be much of a pleasure to use for non-idiots.
If networking 101 is a foot gun to you, I don't think you know enough to be responsible for anything important. I don't think there is a software fix for that lack of knowledge.
Isn't the first point a goal of most "no-code" solutions? Sure, you can most likely still infinite loop but it is a far easier to not screw up compared to "normal" programming.
> I used to think you have to be pretty good at your job ...
I didnt need to get beyond there to see when you went wrong, and the bar for competence is not high across the board. Sadly I havent found a way to make money out of it, but you have to be aware of it to navigate the world safely.
Also, all databases and app servers should be private, accessed by a bastion server. The only thing the outside should be able to reach are the load balancers/reverse proxies.
Yikes. I can't believe this and the responses all agree.
There are two things at play here:
The reality of what happens is that devs with limited experience are asked to do things far outside their comfort zone because they aren't staffed enough. This isn't on the dev, it's on the company and a reality when it comes to growth. People make mistakes.
The second thing is that the databases themselves should be secure by default but are not. That's on the developers of that software. DBs should require setting strong passwords on creation.
Aws should also warn users about exposing these services directly to the outside world instead of just within the vpc.
It's as simple as that. And at scale, it adds up to a lot of mistakes.
And for better or worse, much of the tech industry does not require extensive certification and training with checking and rechecking of every step. (And even aerospace makes mistakes.)
And, yes, junior people make even more mistakes. But I'm not sure the fix to that is to have junior people's every action be check and rechecked by a more senior person. It can be done. That's generally how things work in medicine for example. Along with stringent certifications. But I'm not sure how many people in tech would want that. It's a reasonable discussion to have. Maybe training wheels are taken off too early in many cases.
"it doesn't take a genius to set up a password for a cloud service."
Where are you going to store the password?
I'm not saying it's rocket science but it does take consideration. Do you store it in a plain text file on the server? How do you deal with CI/CD? Do you run a separate service for one credential?
AWS itself has a Secrets Manager from which other authorized services can pull secrets. If it's an outside CI/CD platform then those usually also have a place to store credentials.
I agree that this would be a good solution. I'm merely bringing up that it takes consideration. Also someone will need to learn the Secrets Manager and also learn credentials management for CI/CD. It's probably at least a day of work if you're not familiar with it.
Don't get me wrong. I don't think anyone should skip this step. You need auth for your datastores.
> DevOps, Developers, and IT practitioners often misconfigure some of the following:
Binding the socket on the wrong network interfaces.
For example, listening to connections from 0.0.0.0/* — So it is visible to all network interfaces, instead of only the inner-network interface IP address (172.x.x.x)
Binding to 0.0.0.0 is unfortunately the default for Docker. I wish it would have been different.
I think so, this has been discussed in many HN threads and it always surprises a lot of people how big of a hole this is. I was certainly not aware of it the first time I read about it. Docker inserts its own iptables rules before your drop rule I think to handle its virtual interfaces.
The Docker container is using a separate network namespace with a separate virtual interface that packets are routed to. The "PREROUTING" chain runs before the "INPUT" chain so yes, the rules you put in "INPUT" won't apply.
Even better, docker-compose doesn't do IPV6 (well). But if you have an IPV6 address, it will NAT all the incoming IPV6 traffic on the open ports to IPV4 using the internal network gateway address.
So, if you're doing any filtering for internal network addresses == internal traffic, congratulations, now your inbound external IPV6 traffic is recognized as internal.
Strange, I've never seen this before. I've been running Docker for years on one of my pet project machines, but while Docker is exposing the ports on 0.0.0.0 my iptables firewall blocks all of them.
I'm not using --iptables=false but I'm not discounting the possibility that maybe I've configured this on a more global level somewhere.
That is still strange ; as you can read from others, docker refreshes it’s own iptable rules which then can mess with your own. Maybe you were lucky but seems unlikely to have that much luck. Be careful anyway; if you opened a port, it’ll be on 0.0.0.0 and docker does rearrange your iptables when the container(s) start. Especially default redis and mongo are good foot-shooters.
I don't quite understand it either. I just triple checked though, the ports are filtered from the outside. I do have a bunch of "docker network isolation" things going on in my iptables setup, so maybe that's why.
It's the kind of example that would make me uncomfortable to work with Docker / cloud things; I'm sure I could set up stuff within a day, but there's a lot of Things That You Should Happen To Know like this example that are easily missed - especially when you're in an environment where there's pressure on you to deliver, instead of a focus on doing things right.
I'm pretty sure that if you have no security group attached to an instance no traffic is allowed. And a new security group doesn't allow traffic from "everywhere" by default. You have to actively allow that.
> Binding to 0.0.0.0 is unfortunately the default for Docker.
That isn't just a docker thing, though a few notable accidents with docker have shone a light on the issue in recent times, and is one of the reasons I much prefer to keep my firewall away from where things are running, between them and the unwashed masses on the public network.
Even at home I have a separate small firewall box between the server that hosts things (and the rest of my network) instead of trusting that host to manage its own security in that way. If I run a service via docker or anything else on there, the outside world doesn't see it unless I open a way in elsewhere. For extra paranoia, the network leg that others can connect to (wireless AP & wired network ports in the guest room) is also separate and only sees parts of the server(s) I explicitly want it to (you can do this with vlans and such depending on your kit, I've actually just got an extra NIC on the router).
Never assume a process/service/container/etc is at all locked down, unless you have checked it out yourself or have it sufficiently wrapped in things you have locked off.
A lot of tutorials that get copy/pasted for things like Python that do the same thing. It isn't until someone actually reads the docs and realizes what this code does that they understand that it's probably not what they want to do.
Exactly my past experience. Carefully tweaking firewalld a couple of years ago, then docker punched a hole through them by injecting their own iptables rules. Rules which where opaque to the firewall-cmd of course.
So many have been sold on cloud using the idea that they don’t need to worry about this stuff, because the big cloud provider has better security teams, etc. It’s not surprising at all that you end up with issues like this — any Ops person could’ve seen it coming from miles away.
From the article its main focus vulnerability seems to be setups that don't use VPCs, which is default on AWS since long. It discusses human oversights like a server that needs to accept connections from other servers and where they then have opened a too large IP block. But this only makes sense if they were using internet IPs.. within a VPC this wouldn't matter.
With some even rudimentary knowledge of how IPV4 works this wouldn't be an issue on AWS, there are good defaults and tools to help avoiding this, no matter if you run Dockerized databases or bare metal dbs or use RDS.. you really have to explicitly open up your servers to end up open on the net..
I think in a lot of cases folks still make certain things public instead of using a VPC out of convenience.
For example if a developer needs to access a resource within a VPC then this sky rockets complexity. You can't just connect to something running in a VPC from the outside world. You'd have to install and configure a VPN and it becomes a whole ceremony for both the person setting all of this up and the person connecting.
Using a VPC/VPN and keeping your important things internal (like a database or staging environment, etc.) is a really good idea but unless you have someone who has a really solid understanding of this it's a whole lot easier to make something public, restrict a security group by IP and keep a whitelist of IPs up to date for developer access.
One thing I've been surprised with is how difficult it is to set up database connectivity. I thought you'd just connect to a private VPC but it makes connections and management difficult. Much easier to go with external IP and rely on firewall rules - though then you risk ending up open.
Many years ago it was BBS systems and then as Windows took off it became SMB shares, fun times for sure. It then transitioned to databases open and exposed to the WAN and once those admins became educated the targets were RDP clients that were poorly designed, radmin anyone? Everything to this point led to the Payment Card Industry (P.C.I.) founding as losses accrued and blame assignment for cost absorption was needed. As APIs took off, where we are now, the era of data vacuuming everything insecure continues inclusive of AWS. The future however will redefine what it means to build secure software from day zero as IoT devices that control life and property will continue to grow in popularity as well as attack. Industrial, commercial and residential targets grow by the day such as SCADA, lighting control, hot water tanks, faucets, toilets and all the other devices that will become smart in one's life but I would be remiss to not mention the most important items that will enable remote life manipulation, medical devices. What you cannot see matters most and as connectivity of everything grows exponentially one is likely to not even see it coming. Stay Healthy!
74 comments
[ 2.1 ms ] story [ 162 ms ] threadWhenever I read stories like these, it seems clear to me that someone moved to the cloud in order to not have to care about security. The 'cloud does everything for you!'. Just like you imply in your answer that PaaS, the next level of abstraction, will solve all your security problems. This move, however, will inevitably lead to a situation where people work with new and complex systems that they don't understand (remember: not having to understand them is the sole reason they use them). Unfortunately, working with complex systems you don't understand is the number one reason for vulnerabilities in the first place.
I am not convinced that a service exists that abstracts security away from you.
Zero trust everything.
I would trust Microsoft more on patches/configurations for an email system than something which is managed on premises. You need to have really good people to maintain a good level of security. Not only technically good, but also with a string cold management that will force updates even if it means the CEO will not get his maol for 15 minutes - and say that this is life and that the discussion i sover.
On top of that, MS would (I hope) install patches on their customer-facing systems in advance of an official patch release.
The above applies to the majority of large SaaS services.
Now when you have a "Platform", a hoster that requires you to bring in knowledge and not only data then it gets dangerous. You need to maintain the security of what you bring in. This can be an OS (your "Platform" provides VMs), or code (your "Platform" provides code runners). Unfortunately, when a company moves to the cloud, they sometimes forget to do this assessment and end up with monstrosities they installed themselves (which is not different, security wise, from having it on premises - augmente nu the 7B population that potentially has now access)
I would expect the latter to be sure by default.
Elasticsearch until recently did not nudge you to set up a username or password by default. I noticed the last time I installed it on a fresh instance that on completion of the install it gives you a warning about this and tells you what to do to set a password. That is a small improvement.
Most people would not have the service bound to a public interface, but for those who do for whatever reason have a set up where they are accessing it remotely, at least now there is a tip off that it is completely open to the world by default. This is different from pretty much any other service you might install. MariaDB for instance by default does not allow remote root login even if you change the config to bind it to 0.0.0.0.
A lot of people are just totally unaware of this issue. When I read about database leakage, I generally assume 90% probability it was elastic, these days. Defaults are so important.
If it's a test instance then this is a nice obvious place to go get it from. If it's not then the service isn't open by default, and in both cases we're not asking people to enable remote access themselves by some scheme which may allow them to do it without setting a password.
Same thing with TLS IMO: let certs be specified, or default to generating some - any option is better then default no encryption.
That exploit was single-handedly responsible for hundreds of thousands of bots (and more likely millions in later years) which wreaked havoc on the internet as a whole -- these were the days when a handful of bots on good connections could take down the Amazons of the world for lengthy periods (days, not hours). That they let it go on for so long was comical at the time, but in hindsight seems criminally negligent.
It cost that much because AWS is incredibly expensive.
If a product upon unboxing promptly flops on its back with "come here internet" access controls, even if by good fortune it's saved by your network ACLs, it's time to put it back in the box and return it.
Don't repaste your own comments just because there's been a dupe.
Also dupes are pretty common, it'd be weird to the repaste the same discussion into every dupe thread.
One additional consideration is that the secure defaults should be usable, otherwise the tendency will be to just disable them entirely (think SELinux and how commonly that's turned off as the first action on server setup)
This Google Search returned an extremely high number of mentions of various companies' ills, a high signal to noise ratio. It's very saddening to see that.
https://www.google.com/search?q=unprotected+databases+on+aws
Even if you forget that the cloud is the internet and that the entire internet can reach you over the internet, it doesn't take a genius to set up a password for a cloud service. I have no idea how many of these databases are honey pots, but finding open and vulnerable servers is depressingly simple.
I suppose it's kind of liberating to know that you can be dumb enough to fling patient data into an unprotected cloud server and still get a job in IT. The bar is really set that low.
Well, managers aren't scientists looking for truth. They are people running businesses that need to stay afloat and get stuff done. There's a lot of wiggle room there, mostly due to time pressure (from a charitable perspective, less charitable: also because of company politics).
With so many 10x full-stack developers who're just looking to get sh done and clock out, it can't be otherwise. If managing a server becomes just another bullet point in a developer job, this is exactly what we get.
Sure I can deploy stuff, but you shouldn't trust me. Unfortunately many are not so honest or even aware of the complexity (unknown unknowns).
Yup...the same management are the ones that laid off the DBAs, system admins, network engineers, etc because "the cloud" hand waves all this stuff away now.
On the other hand, I would argue that there are way too many footguns hidden everywhere, like the default binding of docker mentioned here. So yes, a highly skilled IT security professional knows them all, but they are rare. And I think it should be possible to set things up securely in a straight forward way, for people with their speciality in other areas, of which there are plenty.
If networking 101 is a foot gun to you, I don't think you know enough to be responsible for anything important. I don't think there is a software fix for that lack of knowledge.
I didnt need to get beyond there to see when you went wrong, and the bar for competence is not high across the board. Sadly I havent found a way to make money out of it, but you have to be aware of it to navigate the world safely.
There are two things at play here:
The reality of what happens is that devs with limited experience are asked to do things far outside their comfort zone because they aren't staffed enough. This isn't on the dev, it's on the company and a reality when it comes to growth. People make mistakes.
The second thing is that the databases themselves should be secure by default but are not. That's on the developers of that software. DBs should require setting strong passwords on creation.
Aws should also warn users about exposing these services directly to the outside world instead of just within the vpc.
It's as simple as that. And at scale, it adds up to a lot of mistakes.
And for better or worse, much of the tech industry does not require extensive certification and training with checking and rechecking of every step. (And even aerospace makes mistakes.)
And, yes, junior people make even more mistakes. But I'm not sure the fix to that is to have junior people's every action be check and rechecked by a more senior person. It can be done. That's generally how things work in medicine for example. Along with stringent certifications. But I'm not sure how many people in tech would want that. It's a reasonable discussion to have. Maybe training wheels are taken off too early in many cases.
Where are you going to store the password?
I'm not saying it's rocket science but it does take consideration. Do you store it in a plain text file on the server? How do you deal with CI/CD? Do you run a separate service for one credential?
AWS itself has a Secrets Manager from which other authorized services can pull secrets. If it's an outside CI/CD platform then those usually also have a place to store credentials.
Don't get me wrong. I don't think anyone should skip this step. You need auth for your datastores.
Binding to 0.0.0.0 is unfortunately the default for Docker. I wish it would have been different.
I accidentally opened up a Redis test-instance to the world like this.
Assuming I add a rule via
`iptables -I INPUT -s 123.123.123.123 -p tcp --dport 8080 -j DROP`
and on interface:port 0.0.0.0:8080 of the host is a container listening which got run as
`docker run -d -p 0.0.0.0:8080:8080 some/server:latest`
Will that container still be accessible to 123.123.123.132?
I was bit by this once. Be very careful with docker. Do not rely blindly on your firewall of it runs on the same machine.
So, if you're doing any filtering for internal network addresses == internal traffic, congratulations, now your inbound external IPV6 traffic is recognized as internal.
I'm not using --iptables=false but I'm not discounting the possibility that maybe I've configured this on a more global level somewhere.
I even follow this standard rule for pet projects
That isn't just a docker thing, though a few notable accidents with docker have shone a light on the issue in recent times, and is one of the reasons I much prefer to keep my firewall away from where things are running, between them and the unwashed masses on the public network.
Even at home I have a separate small firewall box between the server that hosts things (and the rest of my network) instead of trusting that host to manage its own security in that way. If I run a service via docker or anything else on there, the outside world doesn't see it unless I open a way in elsewhere. For extra paranoia, the network leg that others can connect to (wireless AP & wired network ports in the guest room) is also separate and only sees parts of the server(s) I explicitly want it to (you can do this with vlans and such depending on your kit, I've actually just got an extra NIC on the router).
Never assume a process/service/container/etc is at all locked down, unless you have checked it out yourself or have it sufficiently wrapped in things you have locked off.
Logical cloud protection layers - like security groups - are an additional security layer but not a replacement for proper route design.
With some even rudimentary knowledge of how IPV4 works this wouldn't be an issue on AWS, there are good defaults and tools to help avoiding this, no matter if you run Dockerized databases or bare metal dbs or use RDS.. you really have to explicitly open up your servers to end up open on the net..
For example if a developer needs to access a resource within a VPC then this sky rockets complexity. You can't just connect to something running in a VPC from the outside world. You'd have to install and configure a VPN and it becomes a whole ceremony for both the person setting all of this up and the person connecting.
Using a VPC/VPN and keeping your important things internal (like a database or staging environment, etc.) is a really good idea but unless you have someone who has a really solid understanding of this it's a whole lot easier to make something public, restrict a security group by IP and keep a whitelist of IPs up to date for developer access.