Having all of the context from operating system version to all the dependent software packages has terrific advantages in deployment and provisioning. One single command for the predictable and repeatable launch of the DB and all the services is super useful even without Kubernetes or other orchestration.
> The mistake here is a MongoDB that didn't require authentication, not that docker's clunky iptables setup exposed it to the internet.
If you're backing up diligently, have a problem, go to restore backup, and discover that the backups are unusable due to an issue caused by the backup software's default configuration, you could say "the mistake here is you relying on a single backup system, pardner, not the backup software's bad default configuration," and in the sense that it's good advice to have multiple backup strategies, sure, you're technically correct. But that doesn't excuse the backup software from having a dangerously bad default configuration. That would be relatively trivial to fix. That they've apparently known about for years.
The docker issue and original HN thread have countless instances of folks who have hit and been hacked by the exact same issue. There is clearly a major problem with the docker documentation, usage, etc. that is causing people to continually be taken by surprise with its iptables behavior.
The blame game doesn't help solve the real problem that people are unknowingly putting services directly on the public internet when they think they are secured by a firewall. With how many millions of people use docker every day IMHO the severity of this issue is far too great to ignore. A simple mitigation to fail and warn a user when it appears they might be hitting this issue (i.e. before docker run changes iptables config, give a quick look to see if it looks like it's already configured for the most common ufw or other setup) would stop this from happening for 99% of cases.
A simple runtime check by docker to detect the most common configuration issue (ufw enabled for a service port that's about to be run by a container) is just another layer of protection to add to the stack IMHO. How can it hurt?
> I don't know why this is being downvoted. Multiple overlapping layers of security would have given newsblur a backup in case of accidental "footguns".
Security specialist here. Startups are built by generalists. Good decisions were made here which made other defense-in-depth considerations not as critical when reconciled with go-to-market needs.
If every generalist focused on every security risk in their product, they'd expose themselves to the business risk of not moving quickly enough.
It's pretty clear NewsBlur did what they could and relied on the expertise of others to not fail them in exceptionally basic, entirely avoidable ways. They were betrayed by that reliance.
The core message ("using auth on MongoDB would have prevented this, it's always a good idea to add password auth just in case") is perfectly reasonable; we can all learn from this, and it's perfectly fine to point out such things.
But the way it was phrased was absolutely not okay. People make mistakes all the time and they are not "incompetent". This is the classic "I am very smart, I never make mistakes, if you made a mistake then you're an idiot. You probably eat poop. I am smart btw"-attitude that's just ... ugh...
People rely on firewalls to prevent mistakes from becoming disastrous. Defeating that silently is super surprising. People don't know everything about every piece of tech they use either; very few people do: it's just too much information.
And it's not like auth alone is perfect. Remember when a bug in MySQL allowed people to bypass the auth? Good thing I put a firewall in front of my Doc... oh, no, wait...
Having unauthenticated private services on directly-internet-connected hosts, regardless of the state of the host based firewall, is a mistake that a competent sysadmin does not make. (Then again, so is running MongoDB.)
Open MongoDB servers getting hacked is literally a meme at this point, there is no excuse for someone to configure a _production_ instance with zero authentication.
Agree that authentication should be table stakes, though I would argue that the actual mistake is that the MongoDB application/docker container was on a host/VM with a network interface on the public internet.
Should they have hardened in all the other ways for defense in depth, e.g requiring authentication from localhost? Sure. Should the attacker have not done the attack? Without a doubt. But should docker have not completely undone a security control?
Yes, and the fact that docker seems to have persisted with the current state is the topic of discussion.
This same one got me after years of using Docker, I only discovered it after using the combination of Ubuntu Server (and it's ufw) on a DMZed device.
I was running what I thought was an internal FTP instance for almost a week.
Luckily it was about as hardened as regular ftp can be, but I noticed the problem when my service wasn't able to log in as the (very low) connection limit was filled by someone attempting passwords.
I've been using https://github.com/shinebayar-g/ufw-docker-automated to make docker compliant with ufw, and defining firewall rules as labels for the containers.
It needs some work still, namely the service should be hosted in it's own container for easier updating, but it works reasonably well.
Requiring authentication from localhost does not seem relevant to me, given that the creds would be stored somewhere, either in memory either in a file anyway, but exposing a port is not "binding on localhost".
However testing your firewall after publishing a docker port seems common sense.
Indeed, that's how I found out Docker was using the DOCKER-USER iptables chain that you can customize:
Another thing, instead of using exposing ports like that, the easiest is to use Docker-Compose, so that your containers of a stack have their own private shared network, then you won't have to publish ports to make your services communicate. Otherwise, just create a private network yourself and containerize your stuff in it.
So for me that's two newbie mistakes which conducted to falling for this non targeted, script kiddie attack which has been going on since 2017.
But yeah, go ahead publish your ports instead of using docker networks how you should, "believe" in your firewall while you're at it, and then blame "docker footgun".
> Another thing, instead of using exposing ports like that, the easiest is to use Docker-Compose, so that your containers of a stack have their own private shared network, then you won't have to publish ports to make your services communicate.
That only works with local communication unless you use docker swarm. So byebye high availability.
If you're going to make suggestions, at least think about them from a production point of view.
Most productions are fine with 99.9% uptime, and a single server is fine for that, so HA is not necessary for most production out there.
For example, I'm running a governmental, nationwide service handling 300k req/day, with 1k admins working daily on the site, on a single server without anyone complaining, and without less than 99.9% uptime even though I reboot to upgrade the kernel twice a month, I did have to fine tune the system but that was actually pretty easy.
Quick calc: 99.9% of 365 means 8:45 downtime a year. That's not a lot, especially if hardware or power fails.
In my personal experience, given a vendor SLA of 4:00 hours, Dell manages to send you replacement HW on time, but HP has up to now failed to deliver in window every time. YMMV of course. RAID is the only thing keeping us online in these cases.
There is also the normal server maintenance. Rebooting a kernel twice a month, if the reboot takes 2 minutes, already swallows 48 minutes.
Now to be honest 99.9% business hours only is mostly doable if you serve only 1 continent and manage to update/reboot outside business hours. I have a non-critical debian public web server running that updates and reboots every day at 3AM. Mostly because nginx didn't consistently apply lets encrypt certs. So far no one has noticed or at least complained.
If the server burns I'll just deploy another one from scripts and restore the data, I won't need to wait 4 hours I can do it right away, I will surely be able to do that in less than 8 hours!!! However, we don't own the servers, we just rent them from OVH, so we can spawn one in less than an hour and provision it in another, so maybe the fourth time my server actually completely burns in a year I will feel some kind of fear to do 99.8 instead of 99.9 :'''( <--- these are many tears.
Funny to see my above comment get a final score of -4, people don't like us saying "well just get a cheap linux box and we'll get you 99.9% uptime" ahah and this is "hacker news"!
Not to mention the comment that was like "you can't do production without HA", I have been developing, deploying, maintaining a bunch of productions for the last 20 years and very very few of them needed any kind of HA.
Anyway, with that experience I can get you >99.9% uptime with a 3 buck/month linux box, or HA for anything above 99.9, cause that's what experience is about kids ;)
The machine with DB had a public interface. No matter firewalls, this is just bad. DB machine should be in private subnets, preferably with no inet access at all, even via NAT.
Proof that it had public interface. TFA: "Docker helpfully inserted an allow rule into iptables, opening up MongoDB to the world"
Even Google Cloud SQL just is a managed instance with a public IP (that you don't seem to be able to make private).
Firewalls always help, and thats what Cloud SQL does: whitelist your ip when using the CLI.
What I do in AWS is having a VPC with 3 subnet types, public, private and restricted. Public ones have default route to inet, private to NAT, restricted - none. DBs go in restricted, load balancers in public and web/app servers in private subnets. Plus whitelist firewalling using Security Groups.
when i evaluated using a NAT setup using AWS NAT gateways it appeared to be quite expensive. we settled for strict VPC rules using only SecurityGroups in order to restrict communication. the instances did have public ip but nothing but intended services could connect to it anyways... i.e an instance running a HTTP API could only be connected to if initiated from the load balancer and itself could only connect to relevant services such as a RDS cluster...
later on they also had the need for it to be honest, albeit only for a very specific external service that required us to have a fixed public IP address to connect from. The only solution deemed scaleable was using such an AWS provided NAT Gateway..
>The machine with DB had a public interface. No matter firewalls, this is just bad. DB machine should be in private subnets, preferably with no inet access at all, even via NAT.
This is the type of thinking that lead to the attack in the first place. No, you should not pretend that your database is secure because it is hidden deep behind dozens of layers of firewalls.
You should assume that your database is always accessible from the interne and make sure that internet access to your database is not relevant.
With mongodb you are supposed to use TLS encryption and authentication even for communication within the same host because a low privilege hack or mistake could forward a database port or make it public.
In this scenario you not only need to bypass all the network security theater, you also need the password and a valid client certificate signed by a private CA.
What he said about the DB being in a subnet wasn't incorrect, nor what you just said about setting a password and using certs. It's defense in layers and not "network security theater". You don't do just one of them.
> Yes, and the fact that docker seems to have persisted with the current state is the topic of discussion.
It's clearly written in the docs:
To expose a container’s internal port, an operator can start the container with the -P or -p flag. The exposed port is accessible on the host and the ports are available to any client that can reach the host.
It should be common knowledge by now to either create a virtual interface / network on the host to publish ports on for usage by services that aren't running in Docker or (if you are in an environment where all services are in Docker) use --link between containers.
I don't think that's clear at all. If I set "bind_ip = *" in some application then it's also "available to any client that can reach the host", but the firewall is in front of that. I certainly wouldn't expect an application to frob with my firewall.
And as I understand it, this is very much an unintentional side-effect of ufw and Docker interacting – it's not Docker's the intention at all to override any iptable rules, just an unfortunate side-effect.
"It should be common knowledge by now" is very hand-wavy. I never used Docker much, and I could have been bitten by this. Where do I get this "common knowledge" from? Not everyone is a full-time sysadmin; some people just want to run a small service, read the basic documentation on various things etc., and set things up. Not everyone is super-invested in Docker (or any other tool for that matter) to have all the "common knowledge" that all the experts have. This is why defaults matter.
Same here. It's even a little worse because of the vagaries of Docker networking. I have a server, it has a firewall that has all ports blocked. If I install Docker to run some containers and use the --network=host option, it all works fine. Containers listen on ports on the host, my existing firewall controls it, great. If I use what would seem like the more secure port-forward option, Docker adds a new rule to the top of my firewall ruleset opening the forwarded ports to all traffic, overriding my existing firewall for that port entirely.
I get that Docker is often run by devs on their laptops, but this is bad behavior. The simplest solution I've seen so-far is Docker should only automatically set these firewall rules on devices that only have RFC1918 IPs. That would still keep things simple for devs on laptops, but would prevent firewall override on servers.
What would you estimate as the total word count of the Docker documentation? The page you linked to alone is about 10k words, and there are a lot of pages. At a guess, we're looking at something the length of notoriously long novels like War and Peace. What percentage of the documentation had you read and internalized before you first put Docker into production?
It's very easy to come along after something blows up, find something a person could have done differently, and blame that person, treating them as stupid or negligent. It's easy, satisfying, and often status-enhancing. It's also going to make the world less safe, because it prevents us from solving the actual problems.
It comes out of the world of airplane accident investigations. But so much of it is applicable to software. The first chapter alone, the one contrasting old and new views, can be enough for a lot of people. It's available for free via "look inside this book" or the Kindle sample.
> What percentage of the documentation had you read and internalized before you first put Docker into production?
Admittedly, not as much (and since I've got a couple of years experience, back then the docs were bananaware), which is why I ran into the same issue - but one thing I always do is set up staging environments and check if at least the basic expectations (=can't reach an internal service from the Internet, but can reach public services) are working. Aka: a portscan.
A basic check would have prevented so many carelessly-configured services. Generally, it doesn't hurt anyone to run a nmap scan as part of the regular monitoring - if the line count doesn't match the previous one, raise an alarm.
My point is that this is the same behavior: looking for a reason why the person is wrong. That is a behavior that makes things worse because it takes the focus away from systemic improvements that make us safer. Please don't do it.
> That is a behavior that makes things worse because it takes the focus away from systemic improvements that make us safer.
Systemic improvement would be to educate people to build fail-resilient systems. There are a number of reasonable precautions that could have prevented all of these "I exposed my unsecured MongoDB docker to the Internet" instances:
1) Dedicated hardware firewalls (or in the cloud, firewall rules): If all your server is accessible on is 80/443 anyway from the Internet, it doesn't matter if you punch a hole in the server's firewall.
2) Regular outside port scanning as part of your monitoring. This helps you catch misconfigurations of your firewall.
3) When planning a new infrastructure, think about which system needs access to which other systems. Your MongoDB doesn't need to have an Internet connection
4) Have someone experienced look over your staging system, do a full scale security audit or attempt basic measures of pentesting on your own.
5) Don't copy shit from stackoverflow or years old random blog post without understanding every single thing it does.
6) Don't use MongoDB which has been so fucking often in the news for being an insecure pile of dog poo that it physically hurts to read that name!
Organizations that fail to do at least the first four points as part of standard operating procedures deserve to be hacked, period. There is no excuse for slacking off on either of these in 2021.
Everyone wants to be treated as a (software) engineer here, but the engineer's perspective in this situation would be the opposite: The victims are the customers, and the perpetrator, acting in negligence, was Newsblur. Risk management is a core part of the engineer's job. https://www.sebokwiki.org/wiki/Risk_Management
> Everyone wants to be treated as a (software) engineer here, but the engineer's perspective in this situation would be the opposite: The victims are the customers, and the perpetrator, acting in negligence, was Newsblur.
There can be multiple victims, multiple causes/threat actors, and overlap between two categories.
But what's the end user a victim of in this case? Other than a brief lack of availability?
Is there any other profession were the professional, doing something wrong, would be called the victim? Would an engineer, having made an error when doing structural engineering calculations, be called a victim if the software used was too complex? Or a physician who overlooked adverse affects written in fine print? Or the electrician making errors when fixing things in a house with old wiring?
Of course the behavior of Docker is really problematic in this case, and ditto for MongoDB not having auth active per default. But both applications were consciously chosen by Newsblur.
> Is there any other profession were the professional, doing something wrong, would be called the victim? Would an engineer, having made an error when doing structural engineering calculations, be called a victim if the software used was too complex? Or a physician who overlooked adverse affects written in fine print? Or the electrician making errors when fixing things in a house with old wiring?
I don't understand the relevance of this to my original question.
> Of course the behavior of Docker is really problematic in this case, and ditto for MongoDB not having auth active per default. But both applications were consciously chosen by Newsblur.
Or this, for that matter
---
So that you have it, my original ask was: But what's the end user a victim of in this case? Other than a brief lack of availability?
Of course this incident was inconsequential for Newsblur's customers. That in this case the impact for the customers was approx. zero doesn't change the relationship of Mongo+Docker <> Newsblur <> customers (i.e. who has responsibility and who feels the consequences, which defines who can be a victim), just the outcome for the customers.
> Of course this incident was inconsequential for Newsblur's customers. That in this case the impact for the customers was approx. zero doesn't change the relationship of Mongo+Docker <> Newsblur <> customers (i.e. who has responsibility and who feels the consequences, which defines who can be a victim), just the outcome for the customers.
See my previous note:
> There can be multiple victims, multiple causes/threat actors, and overlap between two categories.
I think we've both concluded now that newsblur was largely a victim as a consumer of docker.
If there is a third-party exploiting that error to cause damage, sure (not the case in your examples, which is why they miss the point). If you don't get security for your business right and it gets burned down by an arsonist, nobody is going to not consider the business owner a victim too.
Software culture is IMHO problematic there in multiple dimensions.
This is completely wrong. By this logic me killing a parent turns the parent into a perpetrator because the parent didn't learn proper self defense.
No, the parent is a victim and the children are dependents of the victim. Thus hurting the parent also hurts the children. It's the same thing with Newsblur. Hurting Newsblur hurts their customers.
* Do not perform the migration in prod replica setup with test data and monitor for, observe and fix any oddities
* Cause a 100% data breach by not following a sound change management procedure
Just because this individual is being blamed and not a Ltd. or Corp. doesn't mean that this is how a quality SaaS should be run. I know that more it's-just-a-side-gig SaaS wing it more than they do not but this is exactly what happens when you haven't got good ops. Rather than random error, this is a consequence from a systematic error.
No, they run database that listens to 0.0.0.0/0 and/or ::0/0 on a private isolated network interface (eth0 inside Docker namespace), then let Docker create a NAT to forward packets from a public network interface on the host to this database.
The latter is a mistake. It doesn't ever make sense to expose hosted container ports to host, with the exception of public-facing stuff (like a webserver). The whole backoffice should be on a separate network.
And this is how Docker works by default, since forever, unless one very explicitly requests that it exposes ("publishes") the ports.
Just don't expose stuff. If the host needs to talk to a database or something, it can always talk to it via that `docker0` (or however it's called, I don't really remember the details) interface. (I believe it doesn't always add a route, because on one of my machines I have to manually add a route so Traefik on a host is able to talk to the containers on a isolated virtual network.)
That depends in if you use docker networks. By default, -p maps a port, afaik to 0.0.0.0:port. So this is more an issue with the default docker settings than anything else.
Yes they should’ve configured, checked, etc. But defaults sound be secure by default
It did, but by interesting coincidence, only really got traction when Google Reader shut down, which happened to be the same month of the initial release of Docker.
Off-topic but I had never seen the Tor exit node's default text before. Pretty hilarious that it states Tor provides privacy to people need it most: "average computer users." Yep, that's definitely who is using Tor...
I can't help but be reminded that zero trust architecture for security has been a thing for at least a decade, and that the 2004 Jericho Forum concluded that perimeter security was illusory, more akin to a picket fence than a wall.
Everyone embracing immutable server configuration would help too. We give so many tools like docker complete trust to touch configuration and do the right thing. Sometimes it bites us though. In a perfect world you'd see docker try to change iptables and it fail, then investigate what's up and understand that a specific change has to be allowed and all the implications of that change.
Perimeter security would have been just fine here. The breach occurred because the host was exposed directly to the internet, rather than e.g. sharing a private network with a load balancer.
There was no perimeter security here. The attacker did not first enter a private network and then pivot to MongoDB; he dialed MongoDB right from the internet. Had Mongo been un-authenticated on a private network it still might have been owned, but the bar would have been a lot higher.
Side note: everything that’s ever existed is “flawed,” it’s a weird word to use in the context of something you want to discredit, because then your alternative had better be “flawless” and it obviously isn’t.
There were several pieces here that conspired to produce the unfortunate end result, the blackmailing attacker exploiting the broken perimeter security was just the last piece.
...yet MongoDB's (default) configuration assumes there is. That's the big problem with perimeter security: applications offloading their security responsibility to other, possibly imaginary, parts of the system.
There was perimeter security in this case. The user diligently configured a white-list only UFW firewall. That is their perimeter.
Docker diligently sidestepped that firewall, and in so doing exposed that this was a case of perimeter security. Because by bypassing that single external filter, the entire service was now vulnerable.
I guess this is hair-splitting semantics, but I think when most people say "perimeter security" in the context of a web production environment, they mean that things like DBs, message queues, and backend services share a private network with the servers that actually terminate user TCP connections.
Obviously with only perimeter security, those servers are soft targets to an attacker who compromises a frontend host. I am all for hardening the interior.
"Don't put stuff on the internet that doesn't need to be, even if you think it's secure, because it's probably complicated enough for you to be mistaken about that." This is a perimeter security philosophy, and also what OP needed. If anything the host-level firewall mishap seems closer to an application-level authz bypass than to a pivot across a "trustworthy" network.
Believing that a zero trust architecture is a replacement for perimeter security is just as illusory and dangerous. Defense in depth ensures that a vulnerability or temporary configuration mistake in one facet of security does not lead to total compromise.
Crap like this is why I dont run docker. I'm glad RedHat took a principled stance on it and dropped it. Podman doesn't punch holes you didn't ask for in your firewall. What a ludicrous anti-feature.
All my async job code lives in a codebase and redis merely contains the payload and some ids.
What system, framework or stack are you using that stores the code of the job in the database? I'm curious, because I can imagine it solves some issues, e.g. deploying new code while allowing old code to finish running and scheduled jobs.
I think what they mean is Redis' Lua scripting stuff with EVAL and such.
Also, if you have stuff like this in Redis, which will get picked up by a job runner:
{"runjob": "foojob", "params": ["one", "two"]}
Depending on how your job runner interprets this, it can be a big problem if people can write to this. At the very least it'll be a DoS in most cases, at worst (SQL) database, shell access, or leaking of passwords.
Depending on the implementation of your job framework you can either:
- run jobs that aren’t supposed to run (removing an account for example, is that’s scheduled after a 2 week grace period) maybe an export or import job. Can be anything of course
- if your job runner allows scripts, or arbitrary class methods, you can do whatever you want
- you can remove jobs if you feel like it
- if you can escape redis because of whatever exploit, you now have access to the internal network
In general I use sidekiq, but resque and inspired implementations generally work on simply calling perform(), so any class with such a method can be called, depending on the typesystem.
The biggest two issues are the ability to perform any defined task, and the larger attack surface of an exposed redis server
From 3.6 the default binding is localhost. That would be ook, except that if you mail the power, you listen to all interfaces. This will happen with Redis as well and is a problem with both docker and its dockerfile
If someone breaks into your server, database Auth won't protect you. And you have to reinstall everything including your database just to be sure he didn't leave holes in those.
And the hacker can still delete your database or steal your data. He will simply download the datafiles and then delete those data from the server. Drop database done without a database password. And no, databases aren't kept encrypted on the server.
Not necessarily -- there can be a number of reasons one can access localhost over the loopback interface that does _not_ imply root access: SSRF, misconfigured tunnels, or just a plain unpriviliged account where the attacker couldn't perform privilege escalation (either because the attacker's incompetence or the system being up-to-date and/or hardened)
MySQL/MariaDB have a completely open root account too... although default firewall rules should prevent public access too, unless Docker likes to punch that hole open too.
Yes, root account password and access permissions should be changed upon a fresh install, but the real issue here is Docker's "helpfulness" by opening ports without explicit permission. That's absurd, and has no reasonable excuse.
> Sure, it would be great to have a password on it
It has a password, which the user has to change, before they can do anything else. From that same page you linked:
"""
For data directory initialization performed manually using mysqld --initialize, mysqld generates an initial random password, marks it expired, and writes it to the server error log.
"""
That documentation is outdated. Debian does not prompt you for a password, it creates a root account but you can't connect to the database using "mysql -uroot" you can only use it if you are root on the server (I forgot how they do the trick)
I've ran a few webhosting companies back in the heyday of LAMP/LEMP.
This is a feature and a dangerous one.
Our provisioning had to carefully manage the firewall during installation, because MySQL would be open to the world *while provisioning*. Often this is mere seconds or less, but on a reasonable popular webhosting, this is enough time for automated tools to take over the server.
If, during that provisioning some other tool would start punching holes in the firewall you carefully manage from provisining, I would be enraged. I dislike Docker for a lot of reasons[0], and this kind of behaviour is one of them.
So, yes: what MySQL or MongoDB do is dangerous and poor security practice. But what Docker does is worse: it breaks the countermeasures you take and rely on. IF all your layers of protection are removed by poor security practices of a tool that says "yea, but you need more than just this one layer" you still have no protection.
[0] yet I do use it a lot on development-machines to run the services needed on localhost and not have to manage all those with apt/snap/brew or whatever they come with.
> MySQL/MariaDB have a completely open root account too...
That's flat out wrong. You can't start MySQL/MariaDB docker images without either explicitly specifying a root password, have it generate a random one on the first start of the container, or explicitly allowing an empty password.
Regarding native installs of MySQL/MariaDB, the situation is a bit more murky, but at least the Ubuntu/Debian packages will ask you for a password during setup (and the bind-ip is 127.0.0.1 by default which means you'll have to mess around with conf.d files if you want to setup an externally-reachable server).
You can totally start a new mongodb container and connect other containers to it without ever using auth. I'm doing it right now with docker-compose for local development.
I and the post I replied to was talking about MySQL. MongoDB is an insecure pile of cow poo that has been so often implicated in hacks and data thefts that I don't get why a) anyone is STILL using it and b) the containers STILL don't have a security-first default.
It's Docker's responsibility to not open firewall ports unless explicitly asked to do so.
Opening random ports without A) Telling the developer first, B) Requesting permission to do so, C) Explaining why Docker wants to do so and D) Providing the ability to "opt-out" of this insecure-by-default configuration - is very bad.
It doesn't matter how secure or insecure the software inside the container is... it shouldn't be allowed to communicate with the public unless the developer specifically requests it.
> It's Docker's responsibility to not open firewall ports unless explicitly asked to do so.
You are asking Docker to open firewall ports with the -p option, it explicitly even says in the manual that published ports will be reachable from everywhere that can access the IP you're publishing on (either default 0.0.0.0 or an IP of your choosing).
The OP didn't pass this option to Docker, and it seems it's pretty well established to be a significant footgun[1].
I don't have a horse in this race, as I don't use Docker for a plethora of reasons dating back to it's inception. There are plenty of far superior containers to use. Just adding this footgun to the list of reasons...
I don't believe this is true. `mysql_secure_installation` script exists to create a password on the root account, and disable root@'%'. If what you said about the Docker image is true, this article wouldn't exist either.
Regardless, a supposed "enterprise" tool shouldn't start changing your config in surprising ways, like punching firewall holes without explicitly being asked to do so. That's just crazy...
Modern Debian/Ubuntu MySQL packaging doesn't even do that. It defaults to Unix socket authentication, and so you can only log in if you are actually root on the system (eg. if you use "sudo mysql"). It does also only bind to 127.0.0.1 by default as you say.
That's how both Mongo and MySQL, back in the day, became so popular.
They had super lax security, making them easy to use for newbie devs, who are frequently scared/easily distracted by security settings.
I'm quite convinced the lack of security was by design. Growth hacking and all that. Get everyone onboard and once you have big business going on, you can focus on the minutiae of security, scaling, not losing data.
Node.js, PHP, Docker (heh!) most other popular techs did the same thing: super lax and frequently wrong defaults, just to get adoption.
As a startup it seems you either choose right/security or wrong/money :-)
I had the 'opposite' thought: if the whole system were in a VPC, Docker wouldn't have had the authority to expose a port to the outside world.
Looks like the author is ahead of both us. Later in the post he mentions both a planned transition to a VPC, and plans to beef up the security of MongoDB itself.
Nonsense. If you know what you are doing, the feature automates things you know you want to do.
As several other comments point out, to setup your DB server so that one open port can expose it to free access from the Internet is incompetent on multiple levels.
If I run a docker container on my macbook and expose a port over 0.0.0.0 (docker run -d -p 8080:8080 nginx) and it's available on my LAN network, how does it get exposed over the internet? Unless there's a rule in my router/gateway which does the port forwarding, it's not possible.
I wish the author goes into a bit more detail onto how his setup was vulnerable rather than bringing up an old Github Issue.
It's not a problem on docker desktop for mac or windows. It's only an issue on linux, and only for systems that use iptables as a firewall (typically ubuntu/debian & ufw). And even more specifically it really only affects servers that are in a hosting environment with no other firewall or protection in front--Digital Ocean's default droplet config is a prime example, on AWS by default you have a cloud-specific firewall provided by Amazon's networking.
On your system docker is running in a little virtual machine that has its own linux kernel, virtual peripherals, etc. This gives you an extra layer of security (by pure happenstance) such that the VM has to allow traffic in too.
And this is kind of why this issue is so nasty. You as a mac user might never know that this is a problem or potential footgun. You could happily develop entire production services on your mac, then move them to a shared linux host like Digital Ocean and... uh oh, now your services are open and you had no idea it was even possible.
If you're deploying code without knowing how networking works, you're always going to have problems with this like this.
Perhaps my example was a little too simplified as AWS is my day-to-day cloud host but I find it hard to believe opening up ports on your VM and exposing it to the world and then placing the blame on docker is fair.
Any old hosted server is going to public to the internet by default. There's no indication that this is hosted locally or in some private network (like AWS VPC).
It's fairly normal for a laptop to be on a network behind a router/gateway.
It's also fairly normal for a server to have public IPs associated with it directly, and for those IPs to be available to the public internet directly.
The latter is what probably happened here.
Newsblur appears to run on digital ocean, where I believe they still default to associating a public IPv4 address to each droplet, which again is pretty normal in the server world.
>In NewsBlur’s case, because NewsBlur is a home of free speech, allowing users in countries with censored news outlets to bypass restrictions and get access to the world at large, the continuing risk of supporting anonymous Internet traffic is worth the cost.
This, the backups, the write-up, all make it really hard for me to want to victim-blame the dev for not catching a very silly Docker default.
That having been said
- public ip?
- no fw?
- no password on the mongod instance?
Idk, couldn't be me, not even in dev, just take the 20 seconds to plumb the pw to both sides. Modding me down won't change these facts and won't keep you from being compromised if you take the same lazy steps
The problem with the Docker default is that they did have a firewall configured but Docker will reconfigure it.
That said, some external firewall would prevent this. My VPS provider allows me to configure which ports I want to expose to the outside world which would mitigate this kind of issue.
Buck stops at NewsBlur on this one. Let's call a spade a spade; unqualified. Newb. It happens. It happened. Port scan next time? Metasploit maybe? "Script Kiddie".. It's in the name.
What surprises me most is the machine running Docker was 100% connected directly to a public network interface. That’s the real root of the problem here.
Ambiguous addresses are generally a pretty big headache and have a lot of security downsides, network level firewalls are better and would have worked here too. It's literally the old NAT vs firewall case.
This (Docker opening a hole in my firewall) is why I moved my dev server from Linode to Digital Ocean. DO provides a “cloud firewall” that provides something akin to AWS security groups and therefore can’t be messed by Docker. Linode doesn’t have anything like that (last time I checked at least).
They recently added one. In fact I had to move many of my VM's to new hypervisors because the ones that didn't support the cloud firewall were deprecated. I don't even use their cloud firewall.
This is perhaps the best arguments I’ve seen for a separate firewall device even if it’s in the cloud (and just software) - something on your box running as root may bypass your rules just to help you.
Well, if they get root on your mongo vm they can still drop all your tables (or ransomware you) right? So would it make a difference in this particular case? Outside the VM tooling probably not being so insane as to bypass the firewall?
I recently started hosting something on Linode, thanks for calling this feature out. It looks like they started launching Cloud Firewalls back in November. Full rollout was maybe April?
Seems like poor network architecture to me. The fact that you can misconfigure a firewall on a VM or server which in turn allows public traffic is pretty dangerous.
Docker has worked like this for a long time. It's a really bad default, and they damn well ought to stop operating this way by default, but it's also like Docker Devops 101. When you install Docker on a Linux system, you should configure the DOCKER-USER chain to drop everything originating on your public network interface. You should also stop running services bound to localhost and instead run them on a private network. You can proxy any traffic that really ought to be allowed through from the outside, or if you can't do that for some reason, then make a single exception to the iptables and host binding rules for that one container.
When I moved my docket setup to a new fedora server a few months ago I was surprised I had to add firewall rules to allow traffic to the ports exposed in the containers. I’m not sure whether this is for any port or specially for things running under the docker user but RHEL should be safe by default.
Would you know of some blog post or guide in the vein of "essential sane(r) defaults to apply after installing Docker" kind of tutorial? Which included, for example, the chain config you mentioned
Does mongodb still ship in a no-auth-required-by-default mode? I could've sworn that was the case years ago at the early NoSQL days. But if they're still around, surely they must have fixed that by now?
The biggest issue with docker is the false sense of security it seems to give a lot of engineers into thinking they know infra when they really don't. I stay away from this because I don't understand it fundamentally, and now this proves it's better to not think these new technologies are your friend unless you actually know what you're doing (which apparently most don't).
Staying away from it is still not the best strategy - at least learn and play with it to understand its strengths and weaknesses.
> it seems to give a lot of engineers into thinking they know infra when they really don't
Maybe, but technology changes over time - I don’t see many new projects choosing VMware over docker/OCI for new infrastructure deployment since you usually don’t need a full VM for applocations that just need isolation and easy static deployments.
I suppose my premise is that engineers often delude themselves into thinking that they know it well enough to implement docker when they dont. Id much rather prefer that the regular developers focus on the coding and leave those decisions to actual devops folks who can focus on getting this stuff right. But the simplicity of dockerfile lulls many into thinking if they run something with it it's production ready.
Doesn't that run counter to what devops actually should be? I know that in practice, this split is what happens at companies, but we should work to close that rift, not widen it.
Whatever devops is supposed to be, it shouldn't involve relying on a tool whose security characteristics the team does not fully understand. If requiring such knowledge results in widening a pre-existing rift, so be it.
That's true, and something that often gets lost in the acceleration of "devops culture". But, in my opinion, the consequence then should be to try to not use Docker at all, and look for alternatives (like Podman) instead of just taking control from the dev team.
> Staying away from it is still not the best strategy
There's a whole generation of sysadmins that use docker so that they can stay away from foundational knowledge. We interview experienced devops who do not know/understand how to build basic packages from source (e.g. they don't understand the ./configure, make, make install chain) and who only have basic knowledge of the underlying operating system.
To be honest, I know how to type `./configure`, `make` and `make install`. And I know more about Gnu Make than I would admit in a job interview (for fear of someone expecting me to work with Make).
But so far, life has been too short for me to waste my time on Gnu Autoconf. And I don't feel guilty about this, or like someone who doesn't know fundamentals.
Autoconf is by all accounts a horrible system. Gnu Make ain't much better.
So I can't fault people for trying to avoid this mess.
Autoconf is one of those programs that solved real problems when it was created, in a way that was actually fairly reasonable at the time too. There's a reason it became so popular. But it's now 30 years later, and what was reasonable then is a liability now.
Sendmail is another great example of this phenomenon.
> They are just not something anyone should be using nowadays.
I guess it depends, take Vim for example: been around since the early 90s, the developers are familiar with autoconf, and it seems to work well for them. Why fix something if it ain't broke?
Switching build systems is not necessarily an easy task. Also, as a user I always found cmake quite hard to deal with. Actually, as a user, I actually prefer autotools in spite of its shortcomings. I think there's still quite a bit to win here.
Either way, I personally wouldn't really call autoconf "terrible".
It's like the saying "the shortest path between two points is the one you know" - the simplest build system is the one that exposes the least complexity to you, which is usually the one that is already deployed and working to your satisfication.
I wholeheartedly agree, that I spend to much time of my live with GNU autoconf and make. But docker is on completely other level. After spending years and years of working with and around docker and speaking with others that do the same, I came to the conclusion that I will recommend docker only for one use-case in the future: Making easy examples to startup a system in one line.
> So I can't fault people for trying to avoid this mess.
Ex-Amazon here. Being deeply familiar with OS fundamentals and internals, "low level" tools, and so on is crucial.
A lot of candidates show to interviews with CVs filled with names of popular frameworks and fancy devops tools and often don't understand what really happens behind the curtain.
People are becoming less familiar with the basics and tend to reinvent the wheel. In many teams this does not get you hired.
I'm not saying you are wrong but this sounds like it is in part a hiring problem.
> Being deeply familiar with OS fundamentals and internals, "low level" tools, and so on is crucial.
...needs to be front and center on the job description then. Far too often these kinds of things are hidden on the initial job descriptions and it just wastes everyone's time. I get that HR/recruiters want to throw a big net but vague job descriptions are often part of the problem.
> the false sense of security it seems to give a lot of engineers into thinking they know infra when they really don't.
I don’t say this lightly: this is a big problem in our industry right now. DevOps means (to some) that developers now handle operations. The reality is that it’s difficult to juggle an operations mindset with a feature driven one.
Even if you focus on infra problems full time there is so much ground left uncovered, it would be impossible for anyone to juggle all of dev and all of infra at once, and this is compounded by the fact that ops is reductionist and dev is additive, which is an incredibly difficult problem for a person to reconcile and give full attention to one side. That’s why devops was supposed to be a division of labour; not just a dude/dudette who can configure nginx and write code.
> In case of refusal to pay, we will contact the General Data Protection Regulation, GDPR and notify them that you store user data in an open form and is not safe. Under the rules of the law, you face a heavy fine or arrest and your base dump will be dropped from our server!
Does anybody know if this threat is at all credible?
The way it's phrased doesn't make much sense but they could contact whoever is responsible for GDPR enforcement in a particular country (if in the EU) and make a complaint against you. But it's unlikely to come to anything (especially if all the data was deleted) and you won't be arrested.
You could end up paying a fine if the regulatory authority investigates and finds that personal data was exposed (or deleted) due to lack of "appropriate technical and organisational measures to ensure a level of security appropriate to the risk"[1].
Relatedly, you can be fined for not telling the regulatory authority about breaches yourself: "In the case of a personal data breach, the controller shall without undue delay and, where feasible, not later than 72 hours after having become aware of it, notify the personal data breach to the supervisory authority competent in accordance with Article 55, unless the personal data breach is unlikely to result in a risk to the rights and freedoms of natural persons."[2]
British Airways[3] for example was fined for inadequate security measures. I also remember a case where some company (cannot quite remember which) itself told the regulator about some security snafu as they should, a snafu that most likely was not exploited. They were still issued a fine but a reduced one because they were very cooperative and most likely no actual damage happened.
There is an enforcement tracker website[4] where you can filter for e.g. "security" in the type column.
But nobody will go to prison for being the victim of a hack, that's just fearmongering by that attacker.
274 comments
[ 4.0 ms ] story [ 282 ms ] threadRelying solely on a host-based firewall for access control is, for reasons which must now be obvious, admin incompetence.
They are responsible for securing the containers. They didn't.
Your services should be using authentication even if they are only bound to localhost.
If you're backing up diligently, have a problem, go to restore backup, and discover that the backups are unusable due to an issue caused by the backup software's default configuration, you could say "the mistake here is you relying on a single backup system, pardner, not the backup software's bad default configuration," and in the sense that it's good advice to have multiple backup strategies, sure, you're technically correct. But that doesn't excuse the backup software from having a dangerously bad default configuration. That would be relatively trivial to fix. That they've apparently known about for years.
The blame game doesn't help solve the real problem that people are unknowingly putting services directly on the public internet when they think they are secured by a firewall. With how many millions of people use docker every day IMHO the severity of this issue is far too great to ignore. A simple mitigation to fail and warn a user when it appears they might be hitting this issue (i.e. before docker run changes iptables config, give a quick look to see if it looks like it's already configured for the most common ufw or other setup) would stop this from happening for 99% of cases.
Everyone knows docker runs as root, and therefore be careful using it, just as you would sudo. It seems this is a major fail in docs.
I kinda get why it's the default, but it needs to be made way better known that by default -p punches a hole through ufw.
You're confusing the proximate cause for the root cause.
Unauthenticated mongodb instances are a pretty common problem - it's why a "script kiddie" was so successful.
Security specialist here. Startups are built by generalists. Good decisions were made here which made other defense-in-depth considerations not as critical when reconciled with go-to-market needs.
If every generalist focused on every security risk in their product, they'd expose themselves to the business risk of not moving quickly enough.
It's pretty clear NewsBlur did what they could and relied on the expertise of others to not fail them in exceptionally basic, entirely avoidable ways. They were betrayed by that reliance.
The core message ("using auth on MongoDB would have prevented this, it's always a good idea to add password auth just in case") is perfectly reasonable; we can all learn from this, and it's perfectly fine to point out such things.
But the way it was phrased was absolutely not okay. People make mistakes all the time and they are not "incompetent". This is the classic "I am very smart, I never make mistakes, if you made a mistake then you're an idiot. You probably eat poop. I am smart btw"-attitude that's just ... ugh...
People rely on firewalls to prevent mistakes from becoming disastrous. Defeating that silently is super surprising. People don't know everything about every piece of tech they use either; very few people do: it's just too much information.
And it's not like auth alone is perfect. Remember when a bug in MySQL allowed people to bypass the auth? Good thing I put a firewall in front of my Doc... oh, no, wait...
So this is why I downvoted it.
It's not an insult to call someone incompetent.
Yes it is, especially when based on a single data point.
It can be both.
It’s a docker problem, which should listen to localhost by default
Should they have hardened in all the other ways for defense in depth, e.g requiring authentication from localhost? Sure. Should the attacker have not done the attack? Without a doubt. But should docker have not completely undone a security control?
Yes, and the fact that docker seems to have persisted with the current state is the topic of discussion.
Previously: https://news.ycombinator.com/item?id=27613217
I was running what I thought was an internal FTP instance for almost a week. Luckily it was about as hardened as regular ftp can be, but I noticed the problem when my service wasn't able to log in as the (very low) connection limit was filled by someone attempting passwords.
I've been using https://github.com/shinebayar-g/ufw-docker-automated to make docker compliant with ufw, and defining firewall rules as labels for the containers. It needs some work still, namely the service should be hosted in it's own container for easier updating, but it works reasonably well.
Totally unexpected outcome.
Luckily I discovered it in testing, so didn't make it to production. But annoying that those issues still remain for so long.
> This creates a firewall rule which maps a container port to a port on the Docker host to the outside world.
Source : https://docs.docker.com/config/containers/container-networki...
Requiring authentication from localhost does not seem relevant to me, given that the creds would be stored somewhere, either in memory either in a file anyway, but exposing a port is not "binding on localhost".
However testing your firewall after publishing a docker port seems common sense.
Indeed, that's how I found out Docker was using the DOCKER-USER iptables chain that you can customize:
https://docs.docker.com/network/iptables/
And that's how I made a simple firewall that works:
https://yourlabs.io/oss/yourlabs.docker/-/blob/master/tasks/...
Another thing, instead of using exposing ports like that, the easiest is to use Docker-Compose, so that your containers of a stack have their own private shared network, then you won't have to publish ports to make your services communicate. Otherwise, just create a private network yourself and containerize your stuff in it.
So for me that's two newbie mistakes which conducted to falling for this non targeted, script kiddie attack which has been going on since 2017.
But yeah, go ahead publish your ports instead of using docker networks how you should, "believe" in your firewall while you're at it, and then blame "docker footgun".
That only works with local communication unless you use docker swarm. So byebye high availability.
If you're going to make suggestions, at least think about them from a production point of view.
For example, I'm running a governmental, nationwide service handling 300k req/day, with 1k admins working daily on the site, on a single server without anyone complaining, and without less than 99.9% uptime even though I reboot to upgrade the kernel twice a month, I did have to fine tune the system but that was actually pretty easy.
In my personal experience, given a vendor SLA of 4:00 hours, Dell manages to send you replacement HW on time, but HP has up to now failed to deliver in window every time. YMMV of course. RAID is the only thing keeping us online in these cases.
There is also the normal server maintenance. Rebooting a kernel twice a month, if the reboot takes 2 minutes, already swallows 48 minutes.
Now to be honest 99.9% business hours only is mostly doable if you serve only 1 continent and manage to update/reboot outside business hours. I have a non-critical debian public web server running that updates and reboots every day at 3AM. Mostly because nginx didn't consistently apply lets encrypt certs. So far no one has noticed or at least complained.
Funny to see my above comment get a final score of -4, people don't like us saying "well just get a cheap linux box and we'll get you 99.9% uptime" ahah and this is "hacker news"!
Not to mention the comment that was like "you can't do production without HA", I have been developing, deploying, maintaining a bunch of productions for the last 20 years and very very few of them needed any kind of HA.
Anyway, with that experience I can get you >99.9% uptime with a 3 buck/month linux box, or HA for anything above 99.9, cause that's what experience is about kids ;)
Only on IPv4...
The machine with DB had a public interface. No matter firewalls, this is just bad. DB machine should be in private subnets, preferably with no inet access at all, even via NAT.
Proof that it had public interface. TFA: "Docker helpfully inserted an allow rule into iptables, opening up MongoDB to the world"
so yeah, it surely depends on the case given...
This is the type of thinking that lead to the attack in the first place. No, you should not pretend that your database is secure because it is hidden deep behind dozens of layers of firewalls.
You should assume that your database is always accessible from the interne and make sure that internet access to your database is not relevant.
With mongodb you are supposed to use TLS encryption and authentication even for communication within the same host because a low privilege hack or mistake could forward a database port or make it public.
In this scenario you not only need to bypass all the network security theater, you also need the password and a valid client certificate signed by a private CA.
It's clearly written in the docs:
(from https://docs.docker.com/engine/reference/run/#expose-incomin...)It should be common knowledge by now to either create a virtual interface / network on the host to publish ports on for usage by services that aren't running in Docker or (if you are in an environment where all services are in Docker) use --link between containers.
And as I understand it, this is very much an unintentional side-effect of ufw and Docker interacting – it's not Docker's the intention at all to override any iptable rules, just an unfortunate side-effect.
"It should be common knowledge by now" is very hand-wavy. I never used Docker much, and I could have been bitten by this. Where do I get this "common knowledge" from? Not everyone is a full-time sysadmin; some people just want to run a small service, read the basic documentation on various things etc., and set things up. Not everyone is super-invested in Docker (or any other tool for that matter) to have all the "common knowledge" that all the experts have. This is why defaults matter.
I get that Docker is often run by devs on their laptops, but this is bad behavior. The simplest solution I've seen so-far is Docker should only automatically set these firewall rules on devices that only have RFC1918 IPs. That would still keep things simple for devs on laptops, but would prevent firewall override on servers.
It is written explicitly in the manual that any port you choose to publish is reachable by all machines that can reach the host.
Just how more explicit does this warning have to be?!
And anyways: a simple "netstat -lnp" would have shown the docker-proxy process and world-wide reachability.
It's very easy to come along after something blows up, find something a person could have done differently, and blame that person, treating them as stupid or negligent. It's easy, satisfying, and often status-enhancing. It's also going to make the world less safe, because it prevents us from solving the actual problems.
Those serious about reducing failures should read Dekker's "Field Guide to Understanding 'Human Error'": https://www.amazon.com/Field-Guide-Understanding-Human-Error...
It comes out of the world of airplane accident investigations. But so much of it is applicable to software. The first chapter alone, the one contrasting old and new views, can be enough for a lot of people. It's available for free via "look inside this book" or the Kindle sample.
Admittedly, not as much (and since I've got a couple of years experience, back then the docs were bananaware), which is why I ran into the same issue - but one thing I always do is set up staging environments and check if at least the basic expectations (=can't reach an internal service from the Internet, but can reach public services) are working. Aka: a portscan.
A basic check would have prevented so many carelessly-configured services. Generally, it doesn't hurt anyone to run a nmap scan as part of the regular monitoring - if the line count doesn't match the previous one, raise an alarm.
My point is that this is the same behavior: looking for a reason why the person is wrong. That is a behavior that makes things worse because it takes the focus away from systemic improvements that make us safer. Please don't do it.
Systemic improvement would be to educate people to build fail-resilient systems. There are a number of reasonable precautions that could have prevented all of these "I exposed my unsecured MongoDB docker to the Internet" instances:
1) Dedicated hardware firewalls (or in the cloud, firewall rules): If all your server is accessible on is 80/443 anyway from the Internet, it doesn't matter if you punch a hole in the server's firewall.
2) Regular outside port scanning as part of your monitoring. This helps you catch misconfigurations of your firewall.
3) When planning a new infrastructure, think about which system needs access to which other systems. Your MongoDB doesn't need to have an Internet connection
4) Have someone experienced look over your staging system, do a full scale security audit or attempt basic measures of pentesting on your own.
5) Don't copy shit from stackoverflow or years old random blog post without understanding every single thing it does.
6) Don't use MongoDB which has been so fucking often in the news for being an insecure pile of dog poo that it physically hurts to read that name!
Organizations that fail to do at least the first four points as part of standard operating procedures deserve to be hacked, period. There is no excuse for slacking off on either of these in 2021.
Everyone wants to be treated as a (software) engineer here, but the engineer's perspective in this situation would be the opposite: The victims are the customers, and the perpetrator, acting in negligence, was Newsblur. Risk management is a core part of the engineer's job. https://www.sebokwiki.org/wiki/Risk_Management
There can be multiple victims, multiple causes/threat actors, and overlap between two categories.
But what's the end user a victim of in this case? Other than a brief lack of availability?
Of course the behavior of Docker is really problematic in this case, and ditto for MongoDB not having auth active per default. But both applications were consciously chosen by Newsblur.
I don't understand the relevance of this to my original question.
> Of course the behavior of Docker is really problematic in this case, and ditto for MongoDB not having auth active per default. But both applications were consciously chosen by Newsblur.
Or this, for that matter
---
So that you have it, my original ask was: But what's the end user a victim of in this case? Other than a brief lack of availability?
> It's not all that great blaming the victim.
Which is what I was replying to.
Of course this incident was inconsequential for Newsblur's customers. That in this case the impact for the customers was approx. zero doesn't change the relationship of Mongo+Docker <> Newsblur <> customers (i.e. who has responsibility and who feels the consequences, which defines who can be a victim), just the outcome for the customers.
See my previous note:
> There can be multiple victims, multiple causes/threat actors, and overlap between two categories.
I think we've both concluded now that newsblur was largely a victim as a consumer of docker.
Cheers, glad this helped.
Software culture is IMHO problematic there in multiple dimensions.
No, the parent is a victim and the children are dependents of the victim. Thus hurting the parent also hurts the children. It's the same thing with Newsblur. Hurting Newsblur hurts their customers.
* Run a SaaS that costs money
* Do not perform the migration in prod replica setup with test data and monitor for, observe and fix any oddities
* Cause a 100% data breach by not following a sound change management procedure
Just because this individual is being blamed and not a Ltd. or Corp. doesn't mean that this is how a quality SaaS should be run. I know that more it's-just-a-side-gig SaaS wing it more than they do not but this is exactly what happens when you haven't got good ops. Rather than random error, this is a consequence from a systematic error.
The latter is a mistake. It doesn't ever make sense to expose hosted container ports to host, with the exception of public-facing stuff (like a webserver). The whole backoffice should be on a separate network.
And this is how Docker works by default, since forever, unless one very explicitly requests that it exposes ("publishes") the ports.
Just don't expose stuff. If the host needs to talk to a database or something, it can always talk to it via that `docker0` (or however it's called, I don't really remember the details) interface. (I believe it doesn't always add a route, because on one of my machines I have to manually add a route so Traefik on a host is able to talk to the containers on a isolated virtual network.)
Yes they should’ve configured, checked, etc. But defaults sound be secure by default
Side note: everything that’s ever existed is “flawed,” it’s a weird word to use in the context of something you want to discredit, because then your alternative had better be “flawless” and it obviously isn’t.
...yet MongoDB's (default) configuration assumes there is. That's the big problem with perimeter security: applications offloading their security responsibility to other, possibly imaginary, parts of the system.
Docker diligently sidestepped that firewall, and in so doing exposed that this was a case of perimeter security. Because by bypassing that single external filter, the entire service was now vulnerable.
Obviously with only perimeter security, those servers are soft targets to an attacker who compromises a frontend host. I am all for hardening the interior.
"Don't put stuff on the internet that doesn't need to be, even if you think it's secure, because it's probably complicated enough for you to be mistaken about that." This is a perimeter security philosophy, and also what OP needed. If anything the host-level firewall mishap seems closer to an application-level authz bypass than to a pivot across a "trustworthy" network.
What system, framework or stack are you using that stores the code of the job in the database? I'm curious, because I can imagine it solves some issues, e.g. deploying new code while allowing old code to finish running and scheduled jobs.
Also, if you have stuff like this in Redis, which will get picked up by a job runner:
Depending on how your job runner interprets this, it can be a big problem if people can write to this. At the very least it'll be a DoS in most cases, at worst (SQL) database, shell access, or leaking of passwords.- run jobs that aren’t supposed to run (removing an account for example, is that’s scheduled after a 2 week grace period) maybe an export or import job. Can be anything of course
- if your job runner allows scripts, or arbitrary class methods, you can do whatever you want
- you can remove jobs if you feel like it
- if you can escape redis because of whatever exploit, you now have access to the internal network
In general I use sidekiq, but resque and inspired implementations generally work on simply calling perform(), so any class with such a method can be called, depending on the typesystem.
The biggest two issues are the ability to perform any defined task, and the larger attack surface of an exposed redis server
And the hacker can still delete your database or steal your data. He will simply download the datafiles and then delete those data from the server. Drop database done without a database password. And no, databases aren't kept encrypted on the server.
So let’s start blaming every piece of software that listens to localhost, but is exposed by docker.
Not a fan of mongo, but the defaults are way better than before.
Docker needs to fix their defaults
Yes, root account password and access permissions should be changed upon a fresh install, but the real issue here is Docker's "helpfulness" by opening ports without explicit permission. That's absurd, and has no reasonable excuse.
> Installation of MySQL creates only a 'root'@'localhost' superuser account that has all privileges and can do anything.
It's only a local account. Sure, it would be great to have a password on it, but it's not "a completely open root account".
It has a password, which the user has to change, before they can do anything else. From that same page you linked:
""" For data directory initialization performed manually using mysqld --initialize, mysqld generates an initial random password, marks it expired, and writes it to the server error log. """
This is a feature and a dangerous one.
Our provisioning had to carefully manage the firewall during installation, because MySQL would be open to the world *while provisioning*. Often this is mere seconds or less, but on a reasonable popular webhosting, this is enough time for automated tools to take over the server. If, during that provisioning some other tool would start punching holes in the firewall you carefully manage from provisining, I would be enraged. I dislike Docker for a lot of reasons[0], and this kind of behaviour is one of them.
So, yes: what MySQL or MongoDB do is dangerous and poor security practice. But what Docker does is worse: it breaks the countermeasures you take and rely on. IF all your layers of protection are removed by poor security practices of a tool that says "yea, but you need more than just this one layer" you still have no protection.
[0] yet I do use it a lot on development-machines to run the services needed on localhost and not have to manage all those with apt/snap/brew or whatever they come with.
That's flat out wrong. You can't start MySQL/MariaDB docker images without either explicitly specifying a root password, have it generate a random one on the first start of the container, or explicitly allowing an empty password.
Regarding native installs of MySQL/MariaDB, the situation is a bit more murky, but at least the Ubuntu/Debian packages will ask you for a password during setup (and the bind-ip is 127.0.0.1 by default which means you'll have to mess around with conf.d files if you want to setup an externally-reachable server).
It's Docker's responsibility to not open firewall ports unless explicitly asked to do so.
Opening random ports without A) Telling the developer first, B) Requesting permission to do so, C) Explaining why Docker wants to do so and D) Providing the ability to "opt-out" of this insecure-by-default configuration - is very bad.
It doesn't matter how secure or insecure the software inside the container is... it shouldn't be allowed to communicate with the public unless the developer specifically requests it.
You are asking Docker to open firewall ports with the -p option, it explicitly even says in the manual that published ports will be reachable from everywhere that can access the IP you're publishing on (either default 0.0.0.0 or an IP of your choosing).
I don't have a horse in this race, as I don't use Docker for a plethora of reasons dating back to it's inception. There are plenty of far superior containers to use. Just adding this footgun to the list of reasons...
[1] https://github.com/moby/moby/issues/4737
Regardless, a supposed "enterprise" tool shouldn't start changing your config in surprising ways, like punching firewall holes without explicitly being asked to do so. That's just crazy...
The packaging does the necessary setup for you, and modern MySQL has secure defaults anyway.
They had super lax security, making them easy to use for newbie devs, who are frequently scared/easily distracted by security settings.
I'm quite convinced the lack of security was by design. Growth hacking and all that. Get everyone onboard and once you have big business going on, you can focus on the minutiae of security, scaling, not losing data.
Node.js, PHP, Docker (heh!) most other popular techs did the same thing: super lax and frequently wrong defaults, just to get adoption.
As a startup it seems you either choose right/security or wrong/money :-)
The development strategy did work, though.
right/no users or wrong/yes users
Not much point being right if no one uses the product :)
My point was: why didn't the author of the article enable the use of authorisation? To rely on only a firewall is just stupid.
Looks like the author is ahead of both us. Later in the post he mentions both a planned transition to a VPC, and plans to beef up the security of MongoDB itself.
As several other comments point out, to setup your DB server so that one open port can expose it to free access from the Internet is incompetent on multiple levels.
If I run a docker container on my macbook and expose a port over 0.0.0.0 (docker run -d -p 8080:8080 nginx) and it's available on my LAN network, how does it get exposed over the internet? Unless there's a rule in my router/gateway which does the port forwarding, it's not possible.
I wish the author goes into a bit more detail onto how his setup was vulnerable rather than bringing up an old Github Issue.
On your system docker is running in a little virtual machine that has its own linux kernel, virtual peripherals, etc. This gives you an extra layer of security (by pure happenstance) such that the VM has to allow traffic in too.
And this is kind of why this issue is so nasty. You as a mac user might never know that this is a problem or potential footgun. You could happily develop entire production services on your mac, then move them to a shared linux host like Digital Ocean and... uh oh, now your services are open and you had no idea it was even possible.
If you're deploying code without knowing how networking works, you're always going to have problems with this like this.
Perhaps my example was a little too simplified as AWS is my day-to-day cloud host but I find it hard to believe opening up ports on your VM and exposing it to the world and then placing the blame on docker is fair.
It's also fairly normal for a server to have public IPs associated with it directly, and for those IPs to be available to the public internet directly.
The latter is what probably happened here.
Newsblur appears to run on digital ocean, where I believe they still default to associating a public IPv4 address to each droplet, which again is pretty normal in the server world.
So there was no firewall then? https://www.digitalocean.com/blog/cloud-firewalls-secure-dro...
So if you’re not careful you may be exposed on ipv6 even if incidentally protected on ipv4 by the NAT.
This, the backups, the write-up, all make it really hard for me to want to victim-blame the dev for not catching a very silly Docker default.
That having been said - public ip? - no fw? - no password on the mongod instance?
Idk, couldn't be me, not even in dev, just take the 20 seconds to plumb the pw to both sides. Modding me down won't change these facts and won't keep you from being compromised if you take the same lazy steps
That said, some external firewall would prevent this. My VPS provider allows me to configure which ports I want to expose to the outside world which would mitigate this kind of issue.
why the redaction? name and shame.
Having root in a VM doesn't typically give you any rights on the hypervisor (at least not on eg Xen).
On a hypervisor, it's much harder for VMs to influence each other.
Linux containers (and docker amongst them) started out as convenient and reasonably performant, and added security later. One patch at a time.
Historically, hypervisors typically started secure and added performance and convenience over time.
(Very simplified. But I used to work for XenSource back in the day.)
https://www.linode.com/blog/linode/cloud-firewall-beta-open/
Review mamonde lip tint with Discount offer https://topsaleproductsonline.blogspot.com/2021/06/review-ma...
The aerodynamics of Argentavis, the world's largest flying bird https://www.interestingnews.club/2021/06/the-aerodynamics-of...
https://techtips2021.blogspot.com/2021/06/best-way-to-get-qu...
That said, it also seems like the docker images are a different beast, so be careful what containers you ship and with what configuration…
> it seems to give a lot of engineers into thinking they know infra when they really don't
Maybe, but technology changes over time - I don’t see many new projects choosing VMware over docker/OCI for new infrastructure deployment since you usually don’t need a full VM for applocations that just need isolation and easy static deployments.
There's a whole generation of sysadmins that use docker so that they can stay away from foundational knowledge. We interview experienced devops who do not know/understand how to build basic packages from source (e.g. they don't understand the ./configure, make, make install chain) and who only have basic knowledge of the underlying operating system.
But so far, life has been too short for me to waste my time on Gnu Autoconf. And I don't feel guilty about this, or like someone who doesn't know fundamentals.
Autoconf is by all accounts a horrible system. Gnu Make ain't much better.
So I can't fault people for trying to avoid this mess.
Autoconf is just really terrible.
Sendmail is another great example of this phenomenon.
Just like PHP and COBOL solved real problems back in their heyday.
They are just not something anyone should be using nowadays.
(Btw, Facebook's Hack is a surprisingly pleasant language. Probably the best one they could have made, coming from PHP.)
I guess it depends, take Vim for example: been around since the early 90s, the developers are familiar with autoconf, and it seems to work well for them. Why fix something if it ain't broke?
Switching build systems is not necessarily an easy task. Also, as a user I always found cmake quite hard to deal with. Actually, as a user, I actually prefer autotools in spite of its shortcomings. I think there's still quite a bit to win here.
Either way, I personally wouldn't really call autoconf "terrible".
And, yes, switching an existing system is a different proposition from starting from scratch.
Though see what the Neovim people have been doing to remove some of the accumulated cruft from Vim, if you are interested.
Ex-Amazon here. Being deeply familiar with OS fundamentals and internals, "low level" tools, and so on is crucial.
A lot of candidates show to interviews with CVs filled with names of popular frameworks and fancy devops tools and often don't understand what really happens behind the curtain.
People are becoming less familiar with the basics and tend to reinvent the wheel. In many teams this does not get you hired.
> Being deeply familiar with OS fundamentals and internals, "low level" tools, and so on is crucial.
...needs to be front and center on the job description then. Far too often these kinds of things are hidden on the initial job descriptions and it just wastes everyone's time. I get that HR/recruiters want to throw a big net but vague job descriptions are often part of the problem.
docker breaches the localhost VS external port security model (and others)
I don’t say this lightly: this is a big problem in our industry right now. DevOps means (to some) that developers now handle operations. The reality is that it’s difficult to juggle an operations mindset with a feature driven one.
Even if you focus on infra problems full time there is so much ground left uncovered, it would be impossible for anyone to juggle all of dev and all of infra at once, and this is compounded by the fact that ops is reductionist and dev is additive, which is an incredibly difficult problem for a person to reconcile and give full attention to one side. That’s why devops was supposed to be a division of labour; not just a dude/dudette who can configure nginx and write code.
Reminds me of this: https://www.weforum.org/agenda/2021/04/brains-prefer-adding-...
Does anybody know if this threat is at all credible?
Relatedly, you can be fined for not telling the regulatory authority about breaches yourself: "In the case of a personal data breach, the controller shall without undue delay and, where feasible, not later than 72 hours after having become aware of it, notify the personal data breach to the supervisory authority competent in accordance with Article 55, unless the personal data breach is unlikely to result in a risk to the rights and freedoms of natural persons."[2]
British Airways[3] for example was fined for inadequate security measures. I also remember a case where some company (cannot quite remember which) itself told the regulator about some security snafu as they should, a snafu that most likely was not exploited. They were still issued a fine but a reduced one because they were very cooperative and most likely no actual damage happened.
There is an enforcement tracker website[4] where you can filter for e.g. "security" in the type column.
But nobody will go to prison for being the victim of a hack, that's just fearmongering by that attacker.
[1] https://gdpr-info.eu/art-32-gdpr/
[2] https://gdpr-info.eu/art-33-gdpr/
[3] https://ico.org.uk/about-the-ico/news-and-events/news-and-bl...
[4] https://www.enforcementtracker.com/