FWIW:
"The name Postgres is an accepted alias for the PostgreSQL project. However it is an alias or nickname and is not the official name of the project."
Shit happens. Sometimes it's an oversight, sometimes it's accidental, sometimes it's from ignorance to the dangers. Sometimes it's even a honeypot, like this time :)
I want to know why exposing databases on the Internet publicly is considered a problem. Is it because a vulnerability found in a database would allow to exploit the database directly?
I ask this because people expose web-apps on the Internet publicly too and an SQL injection vulnerability on the web-app would also be equally catastrophic.
I guess exposing web-apps on the Internet is a risk we need to accept because it has to be available to users of the Internet. But we should not expose anything else that we don't need to? Is that the reason why exposing databases publicly is considered best practice.
I would like to know what experienced professionals think about this.
A web app has a connection to a database that can be limited to its exact permission needs, input can be sanitized, database execution can be limited to exactly what the interface has available. Apps can also use connection pools to manage limits of the database.
If the database itself is publicly exposed, even if a read only connection with access to an empty table is provided an attacker could simply max out the connection pool to kill your application. If a vulnerability was published or a password with more access was available they can not only access all of your data but they can corrupt it and/or delete it.
SQL injection has been pretty trivial to stop for a couple of decades now.
I think you've hit it; never expose anything without good reason, and the database doesn't need to be exposed. This is compounded because, while apps can have injection vulnerabilities, I expect it to be easier to sanitize things at that level rather than exposing the database but expecting it to filter things enough. Or if you prefer: The failure mode of an app is that it passes raw (escapable) queries to the database. If you expose the DB, then raw queries are the starting line.
> But we should not expose anything else that we don't need to?
Yes. Why would you ever need public logins from the open internet to your production database? Even if you need to ability to log in from a random computer, there are better ways, like requiring the use of a vpn so logins are always coming from a white listed ip.
a web app presents a coarse-grained API, in that it exposes a very small, fixed number of API calls with very generic, non-system specific parameters and responses, each of which has been tailored for public exposure.
The database OTOH represents an entire programming shell, e.g. extremely fine-grained and in many cases linked directly to the underlying system as was shown in this article - additionally, the number of entrypoints is undefined. If you somehow were convinced that you've locked down every single function of your database from attack, the next day you upgrade the database to a new version and it can very well expose new functions that are again exposed.
It’s because of the stupid overhead that SSH has out of the box and the complexity of installing and managing db certs. A db admin will say “eh it’s fine; we’ll just use iptables to restrict it to prod server” and then somewhere along the way they’ll get a second prod server and someone junior will somehow fuck it up then the whole thing is open.
It’s really hard to move fast and stay secure. You have to nail so many things and you’re either running out of money or running out of time 99% of the time.
If we can subvert computers running databases (think healthcare, insurance companies, the IRS, etc) then isn’t it time to shut down the internet? People can’t have nice things. I think this is proof that we are in hell. There is a god and she is watching from the other side.
Well, I take it back. Keep the infrastructure so that the ISPs and broadcasters can keep taking our money but make it like tv used to be - one way only - peasants like us can only receive.
at this point, the attacker has already owned the database and found an exploit that allows arbitrary shell execution on the host. i feel like the title is a little bit click-baity, because the attacker could have just hosted their executable payload on any of a thousand shady file hosting sites without needing to hide it in an image.
even polyglot image/executables are usually _mostly_ two files mashed together. some cleverness is always required to construct a header that is valid for both formats, but then the trick is usually to make the executable jump past the image data (or, for interpreted languages, comment out the image data) and then store the executable payload at the end of the file.
Totally unrelated, but I'm always reminded of Spore's (the videogame) save files. Creatures, buildings and such would be saved as .png and you'd get a nice preview screenshot of the creation, but the same file also contained the information for the game to actually load the thing. It was pretty cool
I don't think so, the images were pretty low res iirc. I did some googling though, and the only thing I found was somebody that said "the game reads the model data out from the file's alpha channel". However, I'd expect the data to just be appended to the end of the file, since PNG works anyway with that.
PNG is extensible, so they could totally have defined a PNG "chunk" in the private namespace and shoved the data in that. They didn't though. They treated PNG as just a way to store image data and wrote their extra data into the image, encrypted in an amateurish way.
How does the image get executed? I went through the article and in the example the author extracts the executable manually using dd.
But how would an unsuspecting user run the executable? Perhaps I missed this, but is there some image viewer or browser that runs the trailing bytes of images?
It's executed from PostgreSQL - the article shows SQL queries containing shell commands. This is a way for an attacker who's already obtained the ability to run arbitrary queries against a PostgreSQL server to run an exploitation payload there; the real solution is not to let the attacker get into that situation.
The "exploit" is the part where the attacker gets root/remote code execution. Using the computer to mine cryptocurrency rather than participate in DDOS attacks or host child porn is just a detail.
Getting from "execute arbitrary SQL queries" to "execute specific exploit code" is not a completely trivial step, though of course it's not the most important part.
I was hoping for some sort of image or binary processing exploit too, but the attack just uses a Postgres vulnerability to execute arbitrary shell code. [0][1] The fact that the executed code was buried in an image seems to just be a camouflage step for the attacker.
- And the Postgres database should be vulnerable to various remote code execution
- Once they're able to execute code remotely, they then download an image which has binary data tacked onto it
- They then parse out the executable part of the image using dd
- Then they're able to execute and mine away
While an interesting read the shortest takeaway is:
1. Don't leave your Postgres open to the public internet and
2. Ensure to upgrade when security releases come out.
If you're unsure if the version your on has security patches available or other reasons to upgrade consider checking out https://why-upgrade.depesz.com/
Edit: Looks like the user that gained access had ability to execute pl/c. Which has to run as superuser. You won't find things like pl/c, pl/python generally supported on most Postgres services like Heroku, RDS, Citus because of just this reason. So in this case database access with pl/c was enabled, suspect could have been done equally via other vectors (once database access was achieved).
Actually, the takeaway is not about updates. Because the problem is actual intended functionality, that is "misused" for other purposes: indirect function calls.
So, the takeaway would be to control, which functionality is needed and which not, then to take action accordingly.
The real problem is that Postgres will let you load C functions to run in the server context. That's inherently a bad idea.
Your databases should have multiple users configured with limited roles. Nothing web-facing should have a user with the ability to create a function. A SELECT-only user is a good start.
> 1. Don't leave your Postgres open to the public internet and
you need to do better than that, since attacks within the firewall can occur as well, and this is probably a more common vector for corporate espionage these days. A compromised laptop can get a wide open remote exploit onto a companies' production environment if the database itself is open within the firewall.
Yup. Your database (as well as the majority of ports for other hosts) shouldn't be available from your corporate network either. 99% of corporate users don't need network access beyond a few ports to most hosts.
Also:
3. Ensure that you don't use or provide anybody superuser in Postgres. (Otherwise, there is much simpler way to download&run anything that explain in the article – just `copy tablename from program 'wget ... && chmod a+x ... && ...'`
Don't have superuser accounts at all if you can help it.
Run Postgres in a limited user. Something that can't access any file or execute any command (like wget) it doesn't need, can't do chmod +x. Can't run a shell. Don't know if postgres needs that.
No, postgres can't work without accessing files or having a working shell (latter kinda works in some limited configurations).
And I don't think it's a reasonable idea to not have a superuser at all. But you can have it password less and only accessible from the local machine and a specific account (eg root).
If the image is not lossy compressed (even if it is for the really clever) you can use a variety steganographic tricks to hide any data you want inside an image. Of course you have to have code to decode it.
Creating a C language function is not allowed to regular users by default because "language C" is an untrusted language only superusers can create functions using that. Additionally, regular users don't have the privileges to insert into pg_proc. So unless the attacked application uses a superuser for database access (which is a big security hole to begin with) or uses a superuser account with a weak password and allow superuser access from the outside, I don't see how this could be exploited.
Seems to be an intentional honeypot. That said, the storing the binary code on an innocent image host was a nice find that wouldn't have happened had the attack been blocked immediately. Pretty nice win for that image host to learn to strip images and they can stop hosting malware...
Look for the wide open MongoDB then most recently Memcached servers, it's possible for many people just don't secure their internet connected DB server. The Shodan result mentioned in the article proved.
> So unless the attacked application uses a superuser for database access (which is a big security hole to begin with) or uses a superuser account with a weak password and allow superuser access from the outside, I don't see how this could be exploited.
It can't be exploited if the security "best practices" are used. But I've come across situations where people were using "sa" as the account for production SQL Server connections because they just didn't know any better. Things are much better in the linux/bsd world where there is generally more competence and people tend to know what they are doing.
Having said that, things are getting better. IT/developers are much more mindful of security concerns today than 10 years ago.
Much like the porn industry drives progress in the video format wars, it seems that malware drives progress in the cryptocurrency space. I for one am very excited!
This reminds me of SQL Slammer. Remember that? After that mess, there were Slashdot threads just like we have HN threads right now, and the overwhelming consensus among the sysadmins there was, "Database servers should never be visible to the public Internet. They should always be behind a VPN or application server". And then, as is usual for nerd fora, someone would try to come up with a counterexample, "But what if...", and the sysadmins would just cut them off with, "No. Never ever."
That hasn't changed, folks. If someone on the Internet can talk to your Postgres database, you are Doing It Wrong.
Cryptocurrencies are run on, by, and for crime. It's immoral to participate in cryptocurrencies. You wouldn't be a member of a club that had people like this owning the club house and everyone on the board, but due to pure greed and wilful ignorance people keep "investing" in this organized crime.
Clickbait headline. Was expecting a weird Postgres buffer overflow or something, instead it's a honeypot and the picture is almost completely irrelevant (just a matter of where the attacker hosted their binary).
These attacks are not interesting. They require superuser functionality. Can't believe this wasn't mentioned. If someone has superuser access on your database, it's game over.
The real solution is not to go around making DBAs' lives harder by disabling all this stuff. The real solution is to not give attackers on the internet superuser access on your database!!! Why is the database exposed to the public internet to begin with?
98 comments
[ 3.3 ms ] story [ 139 ms ] threadhttps://wiki.postgresql.org/wiki/ProjectName
https://www.imperva.com/blog/2018/02/database-attacks-sql-ob...
https://www.imperva.com/blog/2018/02/deep-dive-database-atta...
[0]: http://www.zdnet.com/article/mongodb-ransacking-starts-again...
I ask this because people expose web-apps on the Internet publicly too and an SQL injection vulnerability on the web-app would also be equally catastrophic.
I guess exposing web-apps on the Internet is a risk we need to accept because it has to be available to users of the Internet. But we should not expose anything else that we don't need to? Is that the reason why exposing databases publicly is considered best practice.
I would like to know what experienced professionals think about this.
If the database itself is publicly exposed, even if a read only connection with access to an empty table is provided an attacker could simply max out the connection pool to kill your application. If a vulnerability was published or a password with more access was available they can not only access all of your data but they can corrupt it and/or delete it.
SQL injection has been pretty trivial to stop for a couple of decades now.
Yes. Why would you ever need public logins from the open internet to your production database? Even if you need to ability to log in from a random computer, there are better ways, like requiring the use of a vpn so logins are always coming from a white listed ip.
The database OTOH represents an entire programming shell, e.g. extremely fine-grained and in many cases linked directly to the underlying system as was shown in this article - additionally, the number of entrypoints is undefined. If you somehow were convinced that you've locked down every single function of your database from attack, the next day you upgrade the database to a new version and it can very well expose new functions that are again exposed.
It’s really hard to move fast and stay secure. You have to nail so many things and you’re either running out of money or running out of time 99% of the time.
Well, I take it back. Keep the infrastructure so that the ISPs and broadcasters can keep taking our money but make it like tv used to be - one way only - peasants like us can only receive.
Who has to run the Postgres database?
In what kind of way does it has to be accessed to get this happening?
Are we talking about web apps that use Postgres on the back end and run arbitrary queries?
Are we talking about people who somehow extract the Postgres database username and password and it has admin permissions?
I wasn't sure what's happening.
- make it easy to host it on a public, reputable, unblocked web site;
- have a format AV detect less often.
But how would an unsuspecting user run the executable? Perhaps I missed this, but is there some image viewer or browser that runs the trailing bytes of images?
Is there a name for these kind of contingent exploits? If step 1 is get root/execute code, it it reduces the risk of the exploit.
Still valuable to know about, but not that much if you can execute code on a server.
0. https://github.com/nixawk/pentest-wiki/blob/master/2.Vulnera... 1. https://www.rapid7.com/db/modules/exploit/linux/postgres/pos...
"Error establishing a database connection"
From / credit to: https://news.ycombinator.com/item?id=16618030
- Gain access to the database itself
- And the Postgres database should be vulnerable to various remote code execution
- Once they're able to execute code remotely, they then download an image which has binary data tacked onto it
- They then parse out the executable part of the image using dd
- Then they're able to execute and mine away
While an interesting read the shortest takeaway is:
1. Don't leave your Postgres open to the public internet and
2. Ensure to upgrade when security releases come out.
If you're unsure if the version your on has security patches available or other reasons to upgrade consider checking out https://why-upgrade.depesz.com/
Edit: Looks like the user that gained access had ability to execute pl/c. Which has to run as superuser. You won't find things like pl/c, pl/python generally supported on most Postgres services like Heroku, RDS, Citus because of just this reason. So in this case database access with pl/c was enabled, suspect could have been done equally via other vectors (once database access was achieved).
This means that real attackers are attempting right now.
So, the takeaway would be to control, which functionality is needed and which not, then to take action accordingly.
Secondly, things like lo_export() should be disabled by default.
It does require privileges.
> Secondly, things like lo_export() should be disabled by default.
It's superuser only.
Your databases should have multiple users configured with limited roles. Nothing web-facing should have a user with the ability to create a function. A SELECT-only user is a good start.
you need to do better than that, since attacks within the firewall can occur as well, and this is probably a more common vector for corporate espionage these days. A compromised laptop can get a wide open remote exploit onto a companies' production environment if the database itself is open within the firewall.
4. Don't let your DB host make outbound connections (iptables/ip6tables)
Which can also be prevented with SELinux
Would the payload still be there if the image was rebuilt with Imagemagick?
Don't allow superuser connections from outside of your network.
Run Postgres in a limited user. Something that can't access any file or execute any command (like wget) it doesn't need, can't do chmod +x. Can't run a shell. Don't know if postgres needs that.
And I don't think it's a reasonable idea to not have a superuser at all. But you can have it password less and only accessible from the local machine and a specific account (eg root).
For image hosting services it is required to process images anyway - for example to strip EXIF GPS data. It looks like imagemagick can't really do it for JPEGs: https://stackoverflow.com/questions/2654281/how-to-remove-ex...
It can't be exploited if the security "best practices" are used. But I've come across situations where people were using "sa" as the account for production SQL Server connections because they just didn't know any better. Things are much better in the linux/bsd world where there is generally more competence and people tend to know what they are doing.
Having said that, things are getting better. IT/developers are much more mindful of security concerns today than 10 years ago.
Wow. IMO buried lede. That is... very impressive.
your head
It takes someone with astute observation skills to see this.
That hasn't changed, folks. If someone on the Internet can talk to your Postgres database, you are Doing It Wrong.
* AWS RDS + Heroku and not knowing better
* Sharing a database with a customer
There are a lot of reasons. Some are ok, some are not
The only reason I can see is for honeypot.
Cryptocurrencies are run on, by, and for crime. It's immoral to participate in cryptocurrencies. You wouldn't be a member of a club that had people like this owning the club house and everyone on the board, but due to pure greed and wilful ignorance people keep "investing" in this organized crime.
Shame on you all.
This was one currency and bad behavior used to obtain it doesn't even reflect on that.
Investors ignoring or rationalising all the bad because of their greed is, in my opinion, disgusting.
I am fairly sure that this could have been prevented by SELinux.
The real solution is not to go around making DBAs' lives harder by disabling all this stuff. The real solution is to not give attackers on the internet superuser access on your database!!! Why is the database exposed to the public internet to begin with?