It's not, though. It's the easiest thing in the world: Just use a library that never emits unescaped content by default, or if you make a single-character typo.
The problem is that most of the libraries aren't that.
The back-end should see a 7-byte buffer with values [102 111 111 032 098 097 114], assume it's UTF-8 and convert that to its internal string representation?
no, the backend has no reason to see `foo%20bar` - you escape when you're combining that string with other strings (ie into HTML, into a SQL query, etc.)
Many database engines can handle arrays, or table-valued variables which are basically the same thing. Most ORMs will also abstract away arrays for you, so you as the developer never need to deal with escaping of data in arrays.
If you're using an ORM/SQL builder, sure. If you prefer writing raw SQL because ORMs are often quite limiting, then you have to write something like `INSERT INTO foo VALUES ?` and then you pass it `[1, [2,3,4], 5]` as params, which is mostly fine but you still have to parse out that "?" from the query. Why parse? Because what if you wrote 'VALUES "?"' now it's a string and shouldn't be replaced. It would be much nicer if you could just send the query to the SQL server and figure out what to do with the params.
I think your argument is “if I try hard enough to make what I want to do really hard, it will be really hard”. I agree. I’m not sure I buy your argument about ORMs being limiting, given your current… limitations.
Of course you could just do CAST(? AS …) or use something like “string_to_array(?)”. But that’s just working around not being able to compose queries.
Some of my queries get quite complicated, but I'm good at writing SQL. Why would I want to mess around with an ORM trying to recreate the SQL I actually want?
CAST() what? I apologize, the example was a bad. I don't want an array on the SQL side, I want to supply an array. Something more like
SELECT * FROM foo WHERE x=? AND y IN (?)
If I pass [1,[2,3]]
It should expand to
SELECT * FROM foo WHERE x=1 AND y IN (2,3)
Which means it actually needs to be written as
SELECT * FROM foo WHERE x=? AND y IN (?, ?)
And I'd have to pass [1,2,3].
At least in all the DB connectors I've seen. I mostly use MariaDB.
Oh, right, that’s the same problem with the same solution though.
Parameters supply only values, but the cases you’ve shown require expressions. Of course no database (or db connector) would work like that, it’s slightly nonsensical. In your specific example the cardinality of the IN clause is important to the plan.
But all in all, congratulations, you’ve come to reach the limits of using raw SQL with dynamic inputs. Your choices are now: build an ORM, use an ORM, or hack around this issue with code that will horrify the next person to work on it. shrug.
I hacked it with regexes to expand out the ?s. Maybe it'll mangle a query one day but I don't see how it could be injected. The only issue I ran into in ~10 years now is on bigass queries my regex was a bit slow because I trying to handle the string case properly. I found a way to speed it up though.
> It's the easiest thing in the world: Just use a library that never emits unescaped content by default
That doesn't make any sense? Escaping is a function of the consumer, not the producer. Hell, most of the problematic content doesn't come from a library to start with.
And if your Markdown -> HTML converter produces escaped content... it's not a Markdown -> HTML converter, because the result is not HTML.
More broadly, I think one of the core issues is this:
> Escape user input
User input is a broad and complicated category, and it's easy for user input to be "laundered" as it moves through an application.
And then escaping is an explicit action, which means it can be missed or forgotten, which is also a problem.
This means the solution is really that APIs should default to escaping most everything. Rather than having to mark "untrusted" content, it's trusted content which should be marked thus. "Escaping" is the wrong default.
But of course that doesn't solve all the issues. Like markdown, where you want the output of the Markdown converter to be trusted (otherwise the output won't be properly formatted on display), what you don't want trusted is the input, and that means you don't want the input to be laundered through the Markdown converter.
Which is an issue in most Markdown libraries, as they inherit the "trusted input" model from Gruber's original Markdown, where HTML passthrough was a feature.
In that sense one design I did enjoy is Jinja and Markupsafe in the Python ecosystem:
- Like most modern template libraries, Jinja escapes content by default.
- Also (though somewhat sadly) like most template libraries Jinja allows marking a value as safe at point-of-use, however that's dangerous as content can be mixed and it's easy for safe content to suddenly be swapped out for user input and become unsafe through seemingly unrelated changes.
- So a better method is to use `markupsafe.Markup` at the source, it's a string subclass which the library considers safe (because Jinja uses `markupsafe.escape` internally), the neat thing is any combination between a Markup instance and a non-Markup string will implicitly escape the non-Markup parameter(s).
This means you can mark safe content as safe at the source (where it's easy to prove it's safe because e.g. it's a literal), then most transformations will maintain the safety invariants. Though obviously it only works with content you know will ultimately be markup-injected.
And non-method APIs can't be overridden (e.g. re, or HTML/XML libraries) so they're not Markup-aware, they'll treat Markup objects as regular strings which that complicates processing pipelines if you want to conserve safety invariants. At the same time, those are laundering opportunities so care is useful.
« Escaping is a function of the consumer, not the producer »
This is incorrect. The producer emits something in a language, be it HTML or JSON or HTTP headers or whatever. Data must be encoded properly for that language. The consumer must then decode, of course, so in a sense it is the job of both. But the onus is really on the producer.
> This is incorrect. The producer emits something in a language, be it HTML or JSON or HTTP headers or whatever. Data must be encoded properly for that language.
Which is the consumption side. When you send data to an HTML template engine, it’s escaped as input, meaning with the template engine as consumer, not with the template engine as producer.
It may be a “pipeline” situation where the consumer also produces something (e.g. JSON or HTML), but it doesn’t have to be e.g. an SQL interface might have no production, but the data it consumes still needs to be properly escaped.
When your producer produces data, it has no idea how that data will be used, and that’s what determines the necessary transformations e.g. it’s of no help to you if your templating engine generates content escaped for MSSQL when you’re not going to put it in MSSQL.
> it’s of no help to you if your templating engine generates content escaped for MSSQL when you’re not going to put it in MSSQL.
Allow me to complain a bit about MSSQL.
When you're escaping a LIKE expression for MSSQL, you must also escape the "[" character, since it's a wildcard for MSSQL (and nowhere else except AFAIK Sybase). When you're escaping a LIKE expression for other databases, you must not escape the "[" character, since some databases reject escaping anything other than the % and _ wildcards. That is, your escaping code for a LIKE expression has to be database-specific, because MSSQL (and AFAIK Sybase, it seems both have a common ancestor) decided to be different.
> When you're escaping a LIKE expression for other databases, you must not escape the "[" character, since some databases reject escaping anything other than the % and _ wildcards. That is, your escaping code for a LIKE expression has to be database-specific, because MSSQL (and AFAIK Sybase, it seems both have a common ancestor) decided to be different.
TBF you may need custom codepaths because defaults diverge as well, IIRC postgres and sqlite default to ESCAPE '\' while mssql and oracle default to ESCAPE '' (the latter being the actual spec behaviour).
So in Postgres and SQLite you must always escape your LIKE parameter, while in mssql and oracle that's not the case.
> TBF you may need custom codepaths because defaults diverge as well, IIRC postgres and sqlite default to ESCAPE '\' while mssql and oracle default to ESCAPE '' (the latter being the actual spec behaviour).
The trick is to just avoid the default, and always use an explicit ESCAPE, which should work the same on every database (except mysql without NO_BACKSLASH_ESCAPES in which you also have to escape the backslash itself, otherwise it will escape the closing quote and get very confused, but that issue can be avoided by using a character other than backslash as the escape character).
The whole point is that the producer may be hostile, or buggy, and the consumer must handle that. Asserting that it “must” be encoded properly does not make it so.
That doesn't make sense to me and I agree with GP. If I consume HTML and I escape all HTML input I'm given, I'm utterly useless.
Now when I consume text and convert that text into HTML for further treatment, I'm producing HTML, and I must properly escape my input in that conversion. The escaping is only needed because I produce HTML. In fact the only time escaping can be done is when producing data, because if unescaped data is ever produced, the cat's out of the bag.
Edit: Actually think that producer/consumer is a wrong way to talk about this. Escaping only ever occurs at a boundary when transforming between formats (eg from "text string" to "html string") which is always both producer (of the new format) and consumer (of the old format). But it can always be thought of as a type cast, with possible type confusions when input and output formats share the same machine representation (eg string).
> That doesn't make sense to me and I agree with GP. If I consume HTML and I escape all HTML input I'm given, I'm utterly useless. [...] Now when I consume text and convert that text into HTML for further treatment, I'm producing HTML, and I must properly escape my input in that conversion.
Which is my point, it's the consumption side which defines what the escaping should be.
> Escaping only ever occurs at a boundary when transforming between formats (eg from "text string" to "html string") which is always both producer (of the new format) and consumer (of the old format).
A database interface is not a transformer / producer, needs escaping. Globbing is not a transformer either. Still needs escaping.
Whatever, call it a transformer if that makes you hard.
Point doesn't change: what escaping is needed is a function of the "transformer" and applied to the input (= consumption) side of it.
You don't apply an escaping because data comes from a database, you apply it because it goes into one. Same with template processors, regex engines, etc...
The thing that accepts the input must make sure it is properly escaped. Think of SQL injection attacks - they are because the thing that accepts input hasn't properly escaped the input.
Cross site scripting attacks are exactly the same thing but occur when the input side doesn't properly escape HTML input.
An alternate view: “string” is not a granular enough type, just like “bitfield” is not a type. Firstly, a string could be raw unknown bytes, verified UTF-8, or UCS-2 (or even UTF-16 or UCS-4), and you absolutely need to know which it is. But let’s assume that you’ve been a diligent programmer and filtered all that at the edges, and now have a sequence of Unicode code points (or possibly graphemes). You still need to know the escaped-ness of the string! This is also a form of typing. Perl was early with its concept of “tainted” strings, but modern languages can use types to mark this concept in the code. At all points in your code, you should be sure what type the value you have is. If you need to use the types in your language to ensure this, then use types. But make sure of it somehow.
> Firstly, a string could be raw unknown bytes, verified UTF-8, or UCS-2 (or even UTF-16 or UCS-4), and you absolutely need to know which it is.
This is a language defect. If your language was invented in the 1960s it's an understandable defect, but it's still a defect. I do not want to write computer software with strings in a language that doesn't even have an actual string type rather than "Eh, maybe this is a string or maybe it's just some random bytes, who cares".
Only in very low level software should it make a difference whether the string is in fact represented as UTF-8 or UTF-16 or whatever, but Rust shows that you can write software at a low level and still enforce type safety for strings.
I agree though that here once again the Right Thing™ is a strong type system. If I've got a Microsoft Graph username, a URL, an email address and a UUID, that's four types, those are not four strings with human names to distinguish them. We don't need to escape some or any of these types - in their context.
A type system isn't going to save you from users submitting all kinds of potentially different encodings. Which also depends on what kind of user input is being handled: Is it OS-provided UI? Is it something being sent to a service accessible on the internet? Is it from a CLI? Is it from a file? Context matters for the potential space of what kind of data you might be operating on, which could require different ways of either knowing what kind of data you have based on having more control over the input versus having to detect stuff (or be told, correctly) from highly arbitrary things like reading from a file. All of that is external to the type system, and requires doing something before you can tag it with the correct type. Some languages might attempt to detect this stuff for you, but that could potentially be considered a language defect if it's hard to detect what a string is without having other input telling you what that string contains, such as a header in an HTTP request saying that it's UTF-8.
"A type system isn't going to save you from users submitting all kinds of potentially different encodings."
Yes, it is, because you give that a type that indicates you don't know what the encoding is, like RawInput or something. You then can not pass this type to any other function that doesn't explicitly call for that type. If you have some function that accepts it, blindly casts it to UTF-8, and slams it out into a file, well, that's not the type system's fault [1].
Of course a type system won't prevent you from still just being wrong or writing bugs; nobody promises that, not even the formal methods advocates. But it will prevent you from just accidentally blindly shoveling it out somewhere it doesn't belong without ever examining it or thinking about it.
I think you may be believing in a popular myth about strong typing systems, that they are designed to somehow prevent bad data from coming in to your system at all. You correctly identify that as impossible. But what strong typing systems can do is force you to deal with the fact that bad data may be coming in. On the outside, you have the chaos of, say, a bag of bytes that may or may not be JSON. On the inside, you have a "type SomeStruct { int a; int b }". A strong type systems forces you to write some sort of adapting code between those two, and guarantees that the result of that adapting code will be only and exactly the type that comes out of that adapting code, no "whoops, sometimes this dynamic code just returns a string, or maybe a network socket, or who knows what". Nothing can prevent your HTTP API from receiving a JPG of an anime character instead of JSON specifying a user to delete, but a strong type system can make you deal with that immediately and fully, instead of garbage data of indeterminate type floating through the system for an indeterminate period of time.
[1]: Also note there are a lot of "strong type systems" in the world that still fail to take advantage of their own capabilities and let bare string types and such float around too much. There are reasons why libraries must support the lowest common denominator; a file is a series of bytes with no further constraints, so the lowest level API has no choice but to accept that, but higher level APIs should more often take more restricted types. That strong type systems can save you from this doesn't mean they all do. I have a number of wrapper types in various languages just to add these guarantees to my programs not provided by the underlying libraries, though I also have some code that just wraps the underlying libraries that can't help but correctly take raw bytes at the lowest level.
>If you have some function that accepts it, blindly casts it to UTF-8
Unfortunately, if you interact with services you didn't write, you're usually back to getting "strings" of unknown encoding, and typically requirements that force some blind or semi-blind guessing.
If the language explicitly says how strings are defined, libraries that go "Eh, I'll just shove nonsense bytes in this data structure and claim that's a string" are broken by definition.
That's just as true in Java as in Rust. The problem is languages like C++ or D which just don't care and have a "string" type that might just be some bytes.
The libraries in question are the ones consuming the output of those external services. If an external service sends data that does not map to the programming language's string type, then the string type will fail to be created from that invalid input, and the library was wrong to have tried.
The external services are not always either explicit, or compliant about the content of said string. Follow this comment chain up, and you'll see mention of blinding casting to utf-8. The point I was making is that you don't always know what the encoding is.
If the service doesn't send you back a string it doesn't send you back a string. End of story. You're continuously trying to complicate the issue by insisting that they're sending strings and that the client needs to guess their encoding. They are not and it does not.
Strong typing never “solved” anything, except that is forces you to see the problem and solve it yourself, explicitly, instead of relying on weak typing to fudge the types for you.
Blind guessing is not related to the type system. Nobody has claimed type systems can solve that. What they can do is force you to guess, and make it clear where that is occurring.
This, again, goes back to a very broken understanding of types systems that I often see, and once held myself. The claim of type systems is not that they magically go out into the world and fix the external world to be well-typed; the claim is that it forces your code to deal with the conversion of the external world into a clean internal representation, and presumably, to have a clean error pathway when that fails. Dynamically-typed code will let you float along much more easily. Statically-typed code can still be written that way, but at least then it's poor statically-typed code. In some circles that sort of broken dynamic code is essentially idiomatic. (Though that is fading away as every year more programmers learn how bad an idea that is.)
I agree with that if you qualify it with "sometimes". Strong types can force you to guess, sometimes. Other times, the data fits the type but isn't the type.
The way Rust does it is IMO interesting. There is e.g. an OsStr for strings that e.g. describe filenames in an directory listing, because these could actually be invalid UTF-8 but your program might still need to be able to handle them.
So when you wanna convert that OsStr to a String you are forced to handle this in one way or another. This is less comfortable, but describes the underlying systems more accurately.
There is no such thing as an "escaped string". Escaping is not a general concept, it is something that differs given the intended destination of that string. For example, "I am a %SYSTEM% person" is a perfectly fine escaped bash string, but an unsafe CMD string; it is also fine as a C# format string, but potentially unsafe as a C format string, depending on your actual implementation of printf; it is an escaped MSSQL filter string, but not an escaped PostgresSQL filter string.
Also, not all strings/texts should be thought of as Unicode code points/graphemes.
The biggest problem appears once you start wanting to combine such strings.
Say a user inputs some raw text in a form that is intended to be the title of a button in a HTML form that will be sent in a JSON file to be stored in a SQL db. The expectation is that you can later retrieve this HTML snippet from the DB and display it on the screen.
You have to first escape the raw text from the user so that it can be safely used in HTML - so you will go from user_input_string to html_pcdata_escaped_ user_input_string. Then you compose a bit of HTML that contains the button and this part; let's say you store it in some HTML DOM object. Then you want to send this HTML object as JSON, so you have to know to convert html_pcdata_escaped_ user_input_string into json_string_escaped_user_input_string - but that loses type information which may hurt us later, so maybe we want to actually store it as json_string_escaped_html_pcdata_escaped_user_input_string. Then, if we want to use this as part of an SQL query string, by the same considerations, we want to put it in a mysql_like_filter_escaped_json_string_escaped_html_pcdata_escaped_user_input_string - which is getting really ugly, and easy to mess up.
Of course, the order of escaping matters, so an mysql_like_filter_escaped_json_string_escaped_html_pcdata_escaped_user_input_string and a json_string_escaped_mysql_like_filter_escaped_html_pcdata_escaped_user_input_string are different things that need to be decoded differently (of course, for SQL in particular we could use prepared queries instead).
Also, we can't ever concatenate this with any other string-like type until perhaps the final use point (such as sending a query string to the DB), since we need to remember which part of the string is escaped in which way, and for what types of uses it is safe (an HTML-escaped string may still contain SQLi or JSON injection).
The point is that even with proper types, this is not easy to manage or fix.
It also requires quite advanced type systems to be able to use these in normal contexts - say, you want to store several such strings with different provenances in a Map or Set or even List, without "forgetting" the provenance.
As far as I'm concerned, "the point is" that what we're describing here -- keeping track of the type of each piece of data -- is the best way to think about this class of problems (as opposed to talking about "sanitising" or "safe data", for example).
It's then up to us to decide how to best make use of the type system of whatever language we end up implementing it in (or, indeed, to treat the ability to deal with this well as a requirement when we're choosing a language).
For me, effects like "we can't ever concatenate this with any other string-like type" are desirable features, not problems with this approach: either it's possible to convert both strings to a common form, or I shouldn't be trying to combine them.
Sure - I'm just saying that the article is right that this problem is difficult, not easy, and that it doesn't get significantly easier if we accurately keep track.
> The point is that even with proper types, this is not easy to manage or fix.
In practice, in a typed language, nothing like this ever occurs, because the rule is just: "use string for everything, except the edge".
You're thinking of a type like: HtmlString<JsonString<Utf8String>>>
In practice the type that is "passed around" is almost always just "string", and this is converted at the last moment to a single destination format, such as HtmlString.
When writing to databases, there isn't even an escape step at all, because you use parametrised queries, right? Right!?
The database stores "string", not "DatabaseEscapedString".
This is similar to how instants in time ought to be handled. You store them as UTC and convert to the user's time zone at the last moment. You don't pass around some monstrosity that somehow keeps track of +10-5+3 in order to arrive at +7. That would be absurd. Instead you pass around the "Z" UTC timestamp and add +7 when needed.
> In practice the type that is "passed around" is almost always just "string"
That's what happens in practice, of course. The GP was proposing something else, and I was explaining how complicated that gets.
> and this is converted at the last moment to a single destination format, such as HtmlString.
I explained before why this doesn't work unless we're talking about the final destination of this string. Otherwise, if that string is being taken through various encodings (say user input to JSON to sprintf format string to HTTP body), and if you need to combine safe and unsafe input, then what you're saying doesn't work anymore.
Here is a sketch of an example:
userInput := read()
jsonFormat := "{\"context\": \"%s\", \"input\": \"" + json.escape(userInput) + "\"}" //easy and safe
finalJson := ""
sprintf(finalJson, jsonFormat, "some context")
// oops - unsafe if original input was "%s"
sprintf(finalJson, printf.escape(jsonFormat), "some context")
// oops - does the wrong thing - it will output "{\"context\": \"%s\", \"input\": \"%s"\"}
//let's try the other way around?
userInput := read()
formatStr := "{\"context\": \"%s\", \"input\": \"" + printf.escape(userInput) + "\"}" //easy and safe
finalJsonStr := ""
sprintf(finalJson, formatStr, "some context")
// oops - unsafe if userInput was "safe-looking\", \"bypassAuth\": \"true\"}"
sprintf(finalJson, json.escape(formatStr), "some context")
// oops - does the wrong thing - it will output
// "\"{\\\"context\\\":\\\"some context\\\", \\\"input\\\": \\\"safe-looking\\\", \\\"bypassAuth\\\": \\\"true\\\"}\"" - that is, a JSON string instead of a JSON object
The only solution to get this to work is to keep the user input string entirely separate from any other string, and apply escaping to it individually at every level where it is used.
Additionally, you will need to remember what escaping has been applied to it, and in what order, so that it can be un-escaped back to the original value when needed.
You're basically running around with your finger on the trigger and suggesting that everyone everywhere ought to wear ballistic armour to compensate.
This is how you put the safety on and return the gun into its holster:
using System;
using System.Text.Json;
string maliciousInput = "{0} % $0 -- DROP TABLE \"USERS\"";
// Always, always, ALWAYS use a proper serializer for assembling formats like JSON.
// The malicious input can include actual JavaScript, and it'll be correctly encoded with 100% safety.
string encoded = JsonSerializer.Serialize( new {
context= "{0}", // .NET format string placeholder
input=maliciousInput
});
// This will just work, formatting placeholders are ignored if no parameters are specified
Console.WriteLine(encoded);
// A safe FormatException is thrown if you mis-use the string formmating code.
// No vulnerability other than DDoS.
Console.WriteLine(encoded, "adfasfd");
Fundamentally, putting any format like JSON or any user-controlled input into the first parameter of sprintf or any similar function in any language is Wrong with a capital W. It ought to be picked up in code review.
Ideally, sprintf-like functions in strongly typed languages should use a special "FormatString" type instead of a plain string as the first input. This would automatically fix any such issues, but relying on this is still problematic. Naively printing potentially malicious input to places like the console is still quite dangerous, no matter how much you escape it! Logs can be captured into systems that then paste it directly into HTML. Similarly, console control codes can be used by attackers as a nuisance. Etc... Structured logging, along the lines of OpenTelemetry is safer.
This is the safe equivalent of your second example. Both format strings and JSON are correctly handled:
Console.WriteLine( "{0}", JsonSerializer.Serialize( new {
context="{0}", // if sprintf/WriteLine is not misused delibaretely, this is safe!
input="safe-looking\", \"bypassAuth\": \"true\"}"
}));
Neither of your examples actually replaces the {0} in "context" with "some context", so they are not achieving the desired output (which would have been `{"context": "some context", "input": "{0} % $0 -- DROP TABLE \\\"USERS\\\""}`). They are equivalent to my "safe but wrong" examples. The point here was that you may want to "template-ize" the produced JSON for whatever reason.
Also, DoS is a legitimate concern, and of course using a safe language makes other consequences less dire. I wasn't using sprintf() in my example for nothing.
> Naively printing potentially malicious input to places like the console is still quite dangerous, no matter how much you escape it! Logs can be captured into systems that then paste it directly into HTML.
This is exactly my point: when you include untrusted input into another string, even if you escape the untrusted input correctly for the desired format of that string, the entire output is now untrusted, and generally can't be further processed safely. Yours is a perfect example: you can escape the user input to make sure it is formatted safely, but you can't at this point tell in what other ways it should be escaped for other systems that may process it (for example, even printing to the actual console like this may be unsafe, as the user input may include terminal control characters).
Even this problem is still simple if we can assume that, say, anything printed to the console that later needs to be displayed in HTML should be considered an HTML string - it just becomes a simple responsibility of the log collector to properly escape the log lines as HTML content.
The problem is much harder if the intention is to actually control the HTML output through log lines (say, adding new-lines through br in your log statements, or emphasis or whatever). If that is a necessary component of your system, you need to re-architect this so that log lines themselves are no longer simple strings, but are structured so that any user-controlled input is kept separate from the trusted application-control formatting
Say, instead of logging
error: user entered "some|||string<script\>alert('pwned')</script>" which is not a valid number<br>starting over
you would log
error: user entered %s, which is not a valid number<br>starting over ||| some\|\|\|string<script\>alert('pwned')</script>`
and the log collector would need to know to recombine it into the original string, escaping the untrusted part as needed, before outputting it to HTML as
error: user entered "some|||string<script>alert('pwned')</script>", which is not a valid number<br>starting over
Edit: interestingly, I had to use <script\> instead of a normal opening script tag, as HN would give me a TLS error if the comment contains the normal opening script tag...
Wonder if there is some input sanitization going on here as well.
Formatting a string twice is what lead to the Log4j security vulnerability. It had a macro language that allowed user-controlled input to have macros expanded in an unexpected place. Essentially the macro input itself could contain macros.
Your example where you use a sprintf format-string placeholder inside an incomplete JSON snippet ought never be used! Ever.
It's not needed in practice. Construct the object graph and insert the parameters there (unescaped!) and then serialize the whole thing.
E.g.:
JsonConverter.Serialize( new {
// Use a static format string! Never let users control this...
context = string.Format( "{0:N1}", userControlledMaliciosInput ),
alsoThis = "... json snippet...",
});
This is fine.
But as I was saying, the "even better" solution is to not serialize this into JSON and then "work with the string representation". The use of JSON[1] should be a detail transparent to 99% of the application. You should be able -- safely -- to switch it out for XML, gRPC, Cap'n Proto, or whatever.
Going back to my logging example, it ought not to matter what wire format something like OpenTelemetry uses. You should be able to use "rich" object graphs in logging calls, and then let the library figure it out. E.g.:
Log.Information( "user submitted a form", new {
context = string.Format( "{0:N1}", userControlledMaliciosInput ),
alsoThis = "... json snippet...",
});
Ideally, everything should treat this as the native "object" graph whenever the developer interacts with it. Only the "edges", such as RPC serialization or deserialization needs to deal with encoding, at which point it'll need to use exactly one escaping/encoding format and not have worry about nested encodings at all.
[1] A mistake in the design of JSON is that it appears to be "simple", so beginner programmers think they understand it and can work with it "directly" using string manipulation. This is ill-defined at the best of times, and downright unsafe in surprisingly common scenarios: http://seriot.ch/projects/parsing_json.html
We're stuck discussing JSON and other data transfer encodings, which is partly my fault as I brought it up, but there are far more scenarios for using combined text encodings.
It is very common to have templating languages which include their own syntax + the syntax of a target output language (e.g. Markdown supports HTML snippets that should get output to the final HTML as is; C macros support C code snippets, and C itself supports Assembler snippets that should end up in the final binary etc). When generating/processing the mixed format from your own code, you may often hit the problems above.
Even for JSON, there are legitimate reasons for processing stored JSON documents as text, or at least situations where it seems a safe enough approach - because people tend to forget that a string representation of a JSON document that has user-controlled input should be itself considered untrusted user input in its entirety, at least unless it is parsed by a JSON parser.
Additionally, data often has to be stored to unstructured storage (e.g. disk) between the moment you receive untrusted user input and the moment you output the final format to the user - again, doing the easy thing of storing in the intermediate format with the first level of escaping of untrusted input is extremely tempting, and the alternative is significantly more difficult.
All of the use-cases you listed I would flag in a code-review as fundamentally misguided.
If you have formats "A" and "B" with serialization functions A() and B() that take document object models as inputs (not strings!), then nesting them is valid, but a bit of a code smell.
What you're saying is that there are scenarios where A() and B() take strings and return strings, and those strings can have control codes that "mean something" for A and/or B.
That's inherently bad and dangerous, and was the direct cause of one of the WORST vulnerabilities in history. Literally as bad as anything ever out there.
You're saying "maximum bad" is a good idea sometimes. This is like making the argument that a little nuclear war is acceptable on occasion.
> there are legitimate reasons for processing stored JSON documents as text
Escape strings at the last possible moment, and ideally it's done by whatever library you're using so you never have to worry about it. It's never not been clear to me in our codebases if I'm dealing with a raw string or a safe one. They're all unsafe, because you have no clue what context they're going to be used in.
If you're writing a web framework or a DB library things might be different though - in that case a different class probably makes sense. If you have a module for a certain communication medium, then yeah you might use it in that module. But if you're writing a webapp, passing around escaped strings is a bad idea 99% of the time. It creates code highly coupled to one aspect of your system.
Just imagine if you did this with networking. I'm glad we're not in a world where we're passing around TCPString or UDPString or IPString or EthernetString or TokenRingString or CarrierPigeonString because that happens to be a networking stack the app uses sometimes. It sounds like hell.
> They're all unsafe, because you have no clue what context they're going to be used in.
That's correct, but it's the reverse thinking from the escaping one.
Because in the escaping one, when you need not to escape you will also not-escape at the last possible moment, and that's a sure-fire way to launder attacker-controlled data.
Instead you should escape everything, and opt-out as early as possible.
> But if you're writing a webapp, passing around escaped strings is a bad idea 99% of the time. It creates code highly coupled to one aspect of your system.
That's why you do the reverse: most strings are unsafe to everything, but the strings which are safe are generally safe to one specific subsystem. So you say that.
> Just imagine if you did this with networking. I'm glad we're not in a world where we're passing around TCPString or UDPString or IPString or EthernetString or TokenRingString or CarrierPigeonString because that happens to be a networking stack the app uses sometimes. It sounds like hell.
It sounds like hell because it makes no sense, there's no such thing as a TCPString because TCP is not string-based and TCP messages are not composed that way.
> Instead you should escape everything, and opt-out as early as possible.
That’s not even remotely workable for any system with more than one kind of “escaping”. What if I want to use a string as:
1. An IDNA-encoded domain name
2. An HTML text snippet
3. A shell command string argument
4. A string literal part of a regular expression
5. A part to be used in an XML CDATA section
6. A JSON string
I can’t escape the string beforehand, since the escaping rules are all different. No, the only sensible alternative is to use the same rule which we all use for character encoding: Encode and decode (and escape) at the edges.
> It’s not an issue, because by default nothing is safe anywhere, so all those APIs should treat the injected data thus.
No library does this, since it does not know what strings I send it with their literal meaning intended, and which strings I send it with their escape characters intended to be interpreted. The escape characters are part of the API of that library. The library does not accept “strings” as such, it accepts “escaped” strings. And since my program deals with normal unescaped strings, I have to escape the strings before I send them to the API.
> There is no escaping, because everything is automatically internally escaped by default.
I have a feeling that you have a different meaning of the word “escaped” than me.
Most modern templates do exactly that. Jinja certainly does.
> The library does not accept “strings” as such, it accepts “escaped” strings. And since my program deals with normal unescaped strings, I have to escape the strings before I send them to the API.
That’s the problem with the library. That is what needs to be fixed.
> I have a feeling that you have a different meaning of the word “escaped” than me.
Add “explicit” to the first occurrence if you don’t understand without it.
No. The way to solve this is to recognize where the problem lies. The problem does not lie with storing user input. The problem lies with improperly putting strings in other data.
So all you need to do, is to do that properly. Either you commit to using constructs like paramtrized queries instead of concatenizing strings and use the DOM to put together HTML the way you want, or you escape as you concatenate the strings.
Don’t store escaped strings, it’s a recipe for disaster.
In my future perfect language, char seqs cannot be cast. They must be converted. Basically syntactic sugar for Java-style char encoding infrastructure.
I have assumed that disallowing casting was sufficient. But now I'll have to ponder "taint" too. From the hip, I really like the notion of tracking the provenance of data, a la defensive programming.
After reading the article I fail to see what is hard about escaping user input.
It seems like what the author means is that it's hard to think of all the places where user input should be escaped, but even then, if you use any modern framework, everything is escaped by default.
Ruby on Rails pretty much handles this. Regular strings are always escaped in views. Only html_safe strings will emit html. For user input, you should always use the sanitize method instead of raw. :)
It really isn't. Proof: Most people who try, largely succeed. Those who do something silly like try to do it 100% manually generally rapidly realize that's not a good plan, and usually there is a not-very-hard way to encapsulate it somehow, since that's pretty much what our languages do, encapsulate things.
I'm not saying it's completely trivial or that there's never an issue here or there. What I'm saying is, it's on par with any of dozens of other issues in programming. Bugs happen, errors happen, but no more so than anyone else. A series of systems with slightly different encoding practices can also cause some headaches, but, again, these are on par with a number of other issues that can emerge in such systems, not especially bad. I've seen a lot of crappy code that gets this wrong at scale, written by programmers who don't really know or care what they're doing, but the same code was crap in a dozen other ways too, and generally screwed up even easier things as well.
Where you get the problems are, from largest to smallest, 1. People who don't realize it's an issue at all and concatenate everything and 2. People who have just been taught about it, and are doing a wrong thing, most often trying to filter on the way "in" instead of the way "out". ("Sanitize user input" delenda est. Stop saying it. It's wrong.) Which is also not an exceptional case, because again there are any number of things that have the exact same characteristics in the programming world.
I would expect "ridonkulously hard" to encompass something that even when tried is super hard and often a failure, and this isn't that case.
This is a space where type systems can be extremely helpful.
Escaped input and unescaped input are separate types. And a robust type system will allow you to craft your functions so that the streams cannot be crossed without going through translation layers.
In fact, the most robust type systems will offer things like automatic function composition so that you have to write a minimum of code... If a type coercion function is available, the type system can be taught to just automatically apply that coercion function before dropping the string into the relevant processing.
"Most of the rendering bugs I’ve seen in security audits don’t matter. This is not how your organization will be pwned. ... What would fix this? Layered security built around a plausible threat model. What would not help? Removing reflected ASCII text from Shodan’s API error message. I’m not saying that small security bugs aren’t worth fixing, or that organizational security always trumps application security. Rather, real damage usually does not come from where security engineers tend to expect, because they spend their time on pentests and CTFs that differ substantially from the approaches popular among actual attackers."
Everyone commenting that dealing with user input is easy: if it were really easy, we wouldn't keep making the same mistakes. I fixed my first SQL injection attack by switching some code to bind variables over 20 years ago, yet we still have Little Bobby Tables showing up in our collective databases. The fix may be easy ("just do X"), but the mistake is even easier.
Breadth-first security attacks will exploit input sanitizing exploits like that. Security audits can certainly help with that, assuming they don't impose a huge security infrastructure and review process that crushes developer productivity, which always seems to happen.
Depth-first attacks as described are a different class of attack, and of course "audit" won't help that much. Education, penetration testing, and honeypots are some of the stuff that works for that.
Ultimately, if an organization treats its work force like crap, then depth-first attacks are unstoppable. The crypto-locker attackers are strangely pro-worker, because it highlights how disgruntled employees are such effective attack vectors via bribery, vengeance, or apathy.
I've always found it more useful to just discard user input that doesn't come in the format you're asking for, and bail on the entire operation.
Like, if the user might be attempting something fishy, there's no reason to try and "clean it up" and have your program "do it's best" with the remainder. Throw an error back at the user and move on to the next query.
that sounds awful. you probably reject phone numbers that use spaces instead of dashes or something? if its correctable, just correct it and don't hassle the user.
if its ambiguous, then fine, ask the user to clarify.
I use a strongly typed language, repository pattern and an ORM. Good luck trying SQL injections. Also input is sanitized at framework level so good luck with XSS.
Also the input has to bypass validation (for which I have unit tests) and the DTOs are mapped to database models before being written.
87 comments
[ 0.29 ms ] story [ 169 ms ] threadThe problem is that most of the libraries aren't that.
If someone enters "foo bar" into your frontend, should the backend only see "foo%20bar" ?
ORMs don't count. They're just editing the SQL.
Why can’t you just generate the following SQL?
Where param 1-3 are indexes 0-2 in the array?This is basically what everything else does when handing arrays, or other composite types like coordinates.
If the language or libraries are not suitable for doing this then the language or libraries are the problem, not the approach.
Of course you could just do CAST(? AS …) or use something like “string_to_array(?)”. But that’s just working around not being able to compose queries.
CAST() what? I apologize, the example was a bad. I don't want an array on the SQL side, I want to supply an array. Something more like
SELECT * FROM foo WHERE x=? AND y IN (?)
If I pass [1,[2,3]]
It should expand to
SELECT * FROM foo WHERE x=1 AND y IN (2,3)
Which means it actually needs to be written as
SELECT * FROM foo WHERE x=? AND y IN (?, ?)
And I'd have to pass [1,2,3].
At least in all the DB connectors I've seen. I mostly use MariaDB.
Parameters supply only values, but the cases you’ve shown require expressions. Of course no database (or db connector) would work like that, it’s slightly nonsensical. In your specific example the cardinality of the IN clause is important to the plan.
But all in all, congratulations, you’ve come to reach the limits of using raw SQL with dynamic inputs. Your choices are now: build an ORM, use an ORM, or hack around this issue with code that will horrify the next person to work on it. shrug.
That doesn't make any sense? Escaping is a function of the consumer, not the producer. Hell, most of the problematic content doesn't come from a library to start with.
And if your Markdown -> HTML converter produces escaped content... it's not a Markdown -> HTML converter, because the result is not HTML.
More broadly, I think one of the core issues is this:
> Escape user input
User input is a broad and complicated category, and it's easy for user input to be "laundered" as it moves through an application.
And then escaping is an explicit action, which means it can be missed or forgotten, which is also a problem.
This means the solution is really that APIs should default to escaping most everything. Rather than having to mark "untrusted" content, it's trusted content which should be marked thus. "Escaping" is the wrong default.
But of course that doesn't solve all the issues. Like markdown, where you want the output of the Markdown converter to be trusted (otherwise the output won't be properly formatted on display), what you don't want trusted is the input, and that means you don't want the input to be laundered through the Markdown converter.
Which is an issue in most Markdown libraries, as they inherit the "trusted input" model from Gruber's original Markdown, where HTML passthrough was a feature.
In that sense one design I did enjoy is Jinja and Markupsafe in the Python ecosystem:
- Like most modern template libraries, Jinja escapes content by default.
- Also (though somewhat sadly) like most template libraries Jinja allows marking a value as safe at point-of-use, however that's dangerous as content can be mixed and it's easy for safe content to suddenly be swapped out for user input and become unsafe through seemingly unrelated changes.
- So a better method is to use `markupsafe.Markup` at the source, it's a string subclass which the library considers safe (because Jinja uses `markupsafe.escape` internally), the neat thing is any combination between a Markup instance and a non-Markup string will implicitly escape the non-Markup parameter(s).
This means you can mark safe content as safe at the source (where it's easy to prove it's safe because e.g. it's a literal), then most transformations will maintain the safety invariants. Though obviously it only works with content you know will ultimately be markup-injected.
And non-method APIs can't be overridden (e.g. re, or HTML/XML libraries) so they're not Markup-aware, they'll treat Markup objects as regular strings which that complicates processing pipelines if you want to conserve safety invariants. At the same time, those are laundering opportunities so care is useful.
« Escaping is a function of the consumer, not the producer »
This is incorrect. The producer emits something in a language, be it HTML or JSON or HTTP headers or whatever. Data must be encoded properly for that language. The consumer must then decode, of course, so in a sense it is the job of both. But the onus is really on the producer.
Which is the consumption side. When you send data to an HTML template engine, it’s escaped as input, meaning with the template engine as consumer, not with the template engine as producer.
It may be a “pipeline” situation where the consumer also produces something (e.g. JSON or HTML), but it doesn’t have to be e.g. an SQL interface might have no production, but the data it consumes still needs to be properly escaped.
When your producer produces data, it has no idea how that data will be used, and that’s what determines the necessary transformations e.g. it’s of no help to you if your templating engine generates content escaped for MSSQL when you’re not going to put it in MSSQL.
Allow me to complain a bit about MSSQL.
When you're escaping a LIKE expression for MSSQL, you must also escape the "[" character, since it's a wildcard for MSSQL (and nowhere else except AFAIK Sybase). When you're escaping a LIKE expression for other databases, you must not escape the "[" character, since some databases reject escaping anything other than the % and _ wildcards. That is, your escaping code for a LIKE expression has to be database-specific, because MSSQL (and AFAIK Sybase, it seems both have a common ancestor) decided to be different.
TBF you may need custom codepaths because defaults diverge as well, IIRC postgres and sqlite default to ESCAPE '\' while mssql and oracle default to ESCAPE '' (the latter being the actual spec behaviour).
So in Postgres and SQLite you must always escape your LIKE parameter, while in mssql and oracle that's not the case.
The trick is to just avoid the default, and always use an explicit ESCAPE, which should work the same on every database (except mysql without NO_BACKSLASH_ESCAPES in which you also have to escape the backslash itself, otherwise it will escape the closing quote and get very confused, but that issue can be avoided by using a character other than backslash as the escape character).
Consumers must properly escape any input.
Now when I consume text and convert that text into HTML for further treatment, I'm producing HTML, and I must properly escape my input in that conversion. The escaping is only needed because I produce HTML. In fact the only time escaping can be done is when producing data, because if unescaped data is ever produced, the cat's out of the bag.
Edit: Actually think that producer/consumer is a wrong way to talk about this. Escaping only ever occurs at a boundary when transforming between formats (eg from "text string" to "html string") which is always both producer (of the new format) and consumer (of the old format). But it can always be thought of as a type cast, with possible type confusions when input and output formats share the same machine representation (eg string).
Which is my point, it's the consumption side which defines what the escaping should be.
> Escaping only ever occurs at a boundary when transforming between formats (eg from "text string" to "html string") which is always both producer (of the new format) and consumer (of the old format).
A database interface is not a transformer / producer, needs escaping. Globbing is not a transformer either. Still needs escaping.
Point doesn't change: what escaping is needed is a function of the "transformer" and applied to the input (= consumption) side of it.
You don't apply an escaping because data comes from a database, you apply it because it goes into one. Same with template processors, regex engines, etc...
The thing that accepts the input must make sure it is properly escaped. Think of SQL injection attacks - they are because the thing that accepts input hasn't properly escaped the input.
Cross site scripting attacks are exactly the same thing but occur when the input side doesn't properly escape HTML input.
Especially with Ruby having string interpolation.
This is a language defect. If your language was invented in the 1960s it's an understandable defect, but it's still a defect. I do not want to write computer software with strings in a language that doesn't even have an actual string type rather than "Eh, maybe this is a string or maybe it's just some random bytes, who cares".
Only in very low level software should it make a difference whether the string is in fact represented as UTF-8 or UTF-16 or whatever, but Rust shows that you can write software at a low level and still enforce type safety for strings.
I agree though that here once again the Right Thing™ is a strong type system. If I've got a Microsoft Graph username, a URL, an email address and a UUID, that's four types, those are not four strings with human names to distinguish them. We don't need to escape some or any of these types - in their context.
Yes, it is, because you give that a type that indicates you don't know what the encoding is, like RawInput or something. You then can not pass this type to any other function that doesn't explicitly call for that type. If you have some function that accepts it, blindly casts it to UTF-8, and slams it out into a file, well, that's not the type system's fault [1].
Of course a type system won't prevent you from still just being wrong or writing bugs; nobody promises that, not even the formal methods advocates. But it will prevent you from just accidentally blindly shoveling it out somewhere it doesn't belong without ever examining it or thinking about it.
I think you may be believing in a popular myth about strong typing systems, that they are designed to somehow prevent bad data from coming in to your system at all. You correctly identify that as impossible. But what strong typing systems can do is force you to deal with the fact that bad data may be coming in. On the outside, you have the chaos of, say, a bag of bytes that may or may not be JSON. On the inside, you have a "type SomeStruct { int a; int b }". A strong type systems forces you to write some sort of adapting code between those two, and guarantees that the result of that adapting code will be only and exactly the type that comes out of that adapting code, no "whoops, sometimes this dynamic code just returns a string, or maybe a network socket, or who knows what". Nothing can prevent your HTTP API from receiving a JPG of an anime character instead of JSON specifying a user to delete, but a strong type system can make you deal with that immediately and fully, instead of garbage data of indeterminate type floating through the system for an indeterminate period of time.
[1]: Also note there are a lot of "strong type systems" in the world that still fail to take advantage of their own capabilities and let bare string types and such float around too much. There are reasons why libraries must support the lowest common denominator; a file is a series of bytes with no further constraints, so the lowest level API has no choice but to accept that, but higher level APIs should more often take more restricted types. That strong type systems can save you from this doesn't mean they all do. I have a number of wrapper types in various languages just to add these guarantees to my programs not provided by the underlying libraries, though I also have some code that just wraps the underlying libraries that can't help but correctly take raw bytes at the lowest level.
Unfortunately, if you interact with services you didn't write, you're usually back to getting "strings" of unknown encoding, and typically requirements that force some blind or semi-blind guessing.
That's just as true in Java as in Rust. The problem is languages like C++ or D which just don't care and have a "string" type that might just be some bytes.
This, again, goes back to a very broken understanding of types systems that I often see, and once held myself. The claim of type systems is not that they magically go out into the world and fix the external world to be well-typed; the claim is that it forces your code to deal with the conversion of the external world into a clean internal representation, and presumably, to have a clean error pathway when that fails. Dynamically-typed code will let you float along much more easily. Statically-typed code can still be written that way, but at least then it's poor statically-typed code. In some circles that sort of broken dynamic code is essentially idiomatic. (Though that is fading away as every year more programmers learn how bad an idea that is.)
So when you wanna convert that OsStr to a String you are forced to handle this in one way or another. This is less comfortable, but describes the underlying systems more accurately.
Also, not all strings/texts should be thought of as Unicode code points/graphemes.
A type that says (say) "this is a string containing html PCDATA" is a useful thing to have.
Say a user inputs some raw text in a form that is intended to be the title of a button in a HTML form that will be sent in a JSON file to be stored in a SQL db. The expectation is that you can later retrieve this HTML snippet from the DB and display it on the screen.
You have to first escape the raw text from the user so that it can be safely used in HTML - so you will go from user_input_string to html_pcdata_escaped_ user_input_string. Then you compose a bit of HTML that contains the button and this part; let's say you store it in some HTML DOM object. Then you want to send this HTML object as JSON, so you have to know to convert html_pcdata_escaped_ user_input_string into json_string_escaped_user_input_string - but that loses type information which may hurt us later, so maybe we want to actually store it as json_string_escaped_html_pcdata_escaped_user_input_string. Then, if we want to use this as part of an SQL query string, by the same considerations, we want to put it in a mysql_like_filter_escaped_json_string_escaped_html_pcdata_escaped_user_input_string - which is getting really ugly, and easy to mess up.
Of course, the order of escaping matters, so an mysql_like_filter_escaped_json_string_escaped_html_pcdata_escaped_user_input_string and a json_string_escaped_mysql_like_filter_escaped_html_pcdata_escaped_user_input_string are different things that need to be decoded differently (of course, for SQL in particular we could use prepared queries instead).
Also, we can't ever concatenate this with any other string-like type until perhaps the final use point (such as sending a query string to the DB), since we need to remember which part of the string is escaped in which way, and for what types of uses it is safe (an HTML-escaped string may still contain SQLi or JSON injection).
The point is that even with proper types, this is not easy to manage or fix.
It also requires quite advanced type systems to be able to use these in normal contexts - say, you want to store several such strings with different provenances in a Map or Set or even List, without "forgetting" the provenance.
It's then up to us to decide how to best make use of the type system of whatever language we end up implementing it in (or, indeed, to treat the ability to deal with this well as a requirement when we're choosing a language).
For me, effects like "we can't ever concatenate this with any other string-like type" are desirable features, not problems with this approach: either it's possible to convert both strings to a common form, or I shouldn't be trying to combine them.
In practice, in a typed language, nothing like this ever occurs, because the rule is just: "use string for everything, except the edge".
You're thinking of a type like: HtmlString<JsonString<Utf8String>>>
In practice the type that is "passed around" is almost always just "string", and this is converted at the last moment to a single destination format, such as HtmlString.
When writing to databases, there isn't even an escape step at all, because you use parametrised queries, right? Right!?
The database stores "string", not "DatabaseEscapedString".
This is similar to how instants in time ought to be handled. You store them as UTC and convert to the user's time zone at the last moment. You don't pass around some monstrosity that somehow keeps track of +10-5+3 in order to arrive at +7. That would be absurd. Instead you pass around the "Z" UTC timestamp and add +7 when needed.
That's what happens in practice, of course. The GP was proposing something else, and I was explaining how complicated that gets.
> and this is converted at the last moment to a single destination format, such as HtmlString.
I explained before why this doesn't work unless we're talking about the final destination of this string. Otherwise, if that string is being taken through various encodings (say user input to JSON to sprintf format string to HTTP body), and if you need to combine safe and unsafe input, then what you're saying doesn't work anymore.
Here is a sketch of an example:
The only solution to get this to work is to keep the user input string entirely separate from any other string, and apply escaping to it individually at every level where it is used.Additionally, you will need to remember what escaping has been applied to it, and in what order, so that it can be un-escaped back to the original value when needed.
This is how you put the safety on and return the gun into its holster:
Test here: https://dotnetfiddle.net/p8P1fOFundamentally, putting any format like JSON or any user-controlled input into the first parameter of sprintf or any similar function in any language is Wrong with a capital W. It ought to be picked up in code review.
Ideally, sprintf-like functions in strongly typed languages should use a special "FormatString" type instead of a plain string as the first input. This would automatically fix any such issues, but relying on this is still problematic. Naively printing potentially malicious input to places like the console is still quite dangerous, no matter how much you escape it! Logs can be captured into systems that then paste it directly into HTML. Similarly, console control codes can be used by attackers as a nuisance. Etc... Structured logging, along the lines of OpenTelemetry is safer.
See: https://owasp.org/www-community/attacks/Log_Injection
This is the safe equivalent of your second example. Both format strings and JSON are correctly handled:
This outputs: Link: https://dotnetfiddle.net/Lm8jkRAlso, DoS is a legitimate concern, and of course using a safe language makes other consequences less dire. I wasn't using sprintf() in my example for nothing.
> Naively printing potentially malicious input to places like the console is still quite dangerous, no matter how much you escape it! Logs can be captured into systems that then paste it directly into HTML.
This is exactly my point: when you include untrusted input into another string, even if you escape the untrusted input correctly for the desired format of that string, the entire output is now untrusted, and generally can't be further processed safely. Yours is a perfect example: you can escape the user input to make sure it is formatted safely, but you can't at this point tell in what other ways it should be escaped for other systems that may process it (for example, even printing to the actual console like this may be unsafe, as the user input may include terminal control characters).
Even this problem is still simple if we can assume that, say, anything printed to the console that later needs to be displayed in HTML should be considered an HTML string - it just becomes a simple responsibility of the log collector to properly escape the log lines as HTML content.
The problem is much harder if the intention is to actually control the HTML output through log lines (say, adding new-lines through br in your log statements, or emphasis or whatever). If that is a necessary component of your system, you need to re-architect this so that log lines themselves are no longer simple strings, but are structured so that any user-controlled input is kept separate from the trusted application-control formatting
Say, instead of logging
you would log and the log collector would need to know to recombine it into the original string, escaping the untrusted part as needed, before outputting it to HTML as Edit: interestingly, I had to use <script\> instead of a normal opening script tag, as HN would give me a TLS error if the comment contains the normal opening script tag...Wonder if there is some input sanitization going on here as well.
Your example where you use a sprintf format-string placeholder inside an incomplete JSON snippet ought never be used! Ever.
It's not needed in practice. Construct the object graph and insert the parameters there (unescaped!) and then serialize the whole thing.
E.g.:
This is fine.But as I was saying, the "even better" solution is to not serialize this into JSON and then "work with the string representation". The use of JSON[1] should be a detail transparent to 99% of the application. You should be able -- safely -- to switch it out for XML, gRPC, Cap'n Proto, or whatever.
Going back to my logging example, it ought not to matter what wire format something like OpenTelemetry uses. You should be able to use "rich" object graphs in logging calls, and then let the library figure it out. E.g.:
Ideally, everything should treat this as the native "object" graph whenever the developer interacts with it. Only the "edges", such as RPC serialization or deserialization needs to deal with encoding, at which point it'll need to use exactly one escaping/encoding format and not have worry about nested encodings at all.[1] A mistake in the design of JSON is that it appears to be "simple", so beginner programmers think they understand it and can work with it "directly" using string manipulation. This is ill-defined at the best of times, and downright unsafe in surprisingly common scenarios: http://seriot.ch/projects/parsing_json.html
It is very common to have templating languages which include their own syntax + the syntax of a target output language (e.g. Markdown supports HTML snippets that should get output to the final HTML as is; C macros support C code snippets, and C itself supports Assembler snippets that should end up in the final binary etc). When generating/processing the mixed format from your own code, you may often hit the problems above.
Even for JSON, there are legitimate reasons for processing stored JSON documents as text, or at least situations where it seems a safe enough approach - because people tend to forget that a string representation of a JSON document that has user-controlled input should be itself considered untrusted user input in its entirety, at least unless it is parsed by a JSON parser.
Additionally, data often has to be stored to unstructured storage (e.g. disk) between the moment you receive untrusted user input and the moment you output the final format to the user - again, doing the easy thing of storing in the intermediate format with the first level of escaping of untrusted input is extremely tempting, and the alternative is significantly more difficult.
If you have formats "A" and "B" with serialization functions A() and B() that take document object models as inputs (not strings!), then nesting them is valid, but a bit of a code smell.
What you're saying is that there are scenarios where A() and B() take strings and return strings, and those strings can have control codes that "mean something" for A and/or B.
That's inherently bad and dangerous, and was the direct cause of one of the WORST vulnerabilities in history. Literally as bad as anything ever out there.
You're saying "maximum bad" is a good idea sometimes. This is like making the argument that a little nuclear war is acceptable on occasion.
> there are legitimate reasons for processing stored JSON documents as text
No, there isn't. Stop. Never do this. Ever.
Don't parse HTML or XML with Regex either. It leads to m̷͉̈a̴̳̚d̶̟̐n̴̩̓e̷̘̿s̴̤͆s̵͉͗: https://stackoverflow.com/questions/1732348/regex-match-open...
If you're writing a web framework or a DB library things might be different though - in that case a different class probably makes sense. If you have a module for a certain communication medium, then yeah you might use it in that module. But if you're writing a webapp, passing around escaped strings is a bad idea 99% of the time. It creates code highly coupled to one aspect of your system.
Just imagine if you did this with networking. I'm glad we're not in a world where we're passing around TCPString or UDPString or IPString or EthernetString or TokenRingString or CarrierPigeonString because that happens to be a networking stack the app uses sometimes. It sounds like hell.
That's correct, but it's the reverse thinking from the escaping one.
Because in the escaping one, when you need not to escape you will also not-escape at the last possible moment, and that's a sure-fire way to launder attacker-controlled data.
Instead you should escape everything, and opt-out as early as possible.
> But if you're writing a webapp, passing around escaped strings is a bad idea 99% of the time. It creates code highly coupled to one aspect of your system.
That's why you do the reverse: most strings are unsafe to everything, but the strings which are safe are generally safe to one specific subsystem. So you say that.
> Just imagine if you did this with networking. I'm glad we're not in a world where we're passing around TCPString or UDPString or IPString or EthernetString or TokenRingString or CarrierPigeonString because that happens to be a networking stack the app uses sometimes. It sounds like hell.
It sounds like hell because it makes no sense, there's no such thing as a TCPString because TCP is not string-based and TCP messages are not composed that way.
That’s not even remotely workable for any system with more than one kind of “escaping”. What if I want to use a string as:
1. An IDNA-encoded domain name
2. An HTML text snippet
3. A shell command string argument
4. A string literal part of a regular expression
5. A part to be used in an XML CDATA section
6. A JSON string
I can’t escape the string beforehand, since the escaping rules are all different. No, the only sensible alternative is to use the same rule which we all use for character encoding: Encode and decode (and escape) at the edges.
You’re still misunderstanding. You shouldn’t escape at any point, instead you should mark things as safe as early as possible.
“Safe” almost always has a single context, you don’t care if it’s going to go somewhere else because it’s not safe for there.
Anything that’s not marked as safe is then automatically considered unsafe and processed as such by the sink.
> What if I want to use a string as:
It’s not an issue, because by default nothing is safe anywhere, so all those APIs should treat the injected data thus.
There is no escaping, because everything is automatically internally escaped by default.
No library does this, since it does not know what strings I send it with their literal meaning intended, and which strings I send it with their escape characters intended to be interpreted. The escape characters are part of the API of that library. The library does not accept “strings” as such, it accepts “escaped” strings. And since my program deals with normal unescaped strings, I have to escape the strings before I send them to the API.
> There is no escaping, because everything is automatically internally escaped by default.
I have a feeling that you have a different meaning of the word “escaped” than me.
Most modern templates do exactly that. Jinja certainly does.
> The library does not accept “strings” as such, it accepts “escaped” strings. And since my program deals with normal unescaped strings, I have to escape the strings before I send them to the API.
That’s the problem with the library. That is what needs to be fixed.
> I have a feeling that you have a different meaning of the word “escaped” than me.
Add “explicit” to the first occurrence if you don’t understand without it.
So all you need to do, is to do that properly. Either you commit to using constructs like paramtrized queries instead of concatenizing strings and use the DOM to put together HTML the way you want, or you escape as you concatenate the strings.
Don’t store escaped strings, it’s a recipe for disaster.
Agreed. My future perfect programming language has the predefined types 'ascii', 'utf-8', 'url', 'base64', etc. for misc kinds of character sequences.
Just like how raw bits are different from numerals: short vs byte, word vs int, 64-bits vs double, etc.
(Any one have a better naming system for 8, 16, 32, and 64 bit chunks of raw data? 'byte', 'word', 'doubleword', 'quadword'?)
Per this "ridonkulously hard" OC article, I'll also ponder predefined types for raw 'html5', 'json', etc (as in unparsed, char sequence vs DOM).
--
> Perl was early with its concept of “tainted” strings.
Not being a Perl dev, I'm unfamiliar with "taint". Quickly found articles like this: https://www.geeksforgeeks.org/perl-taint-method/
In my future perfect language, char seqs cannot be cast. They must be converted. Basically syntactic sugar for Java-style char encoding infrastructure.
I have assumed that disallowing casting was sufficient. But now I'll have to ponder "taint" too. From the hip, I really like the notion of tracking the provenance of data, a la defensive programming.
Great idea. Thanks.
It seems like what the author means is that it's hard to think of all the places where user input should be escaped, but even then, if you use any modern framework, everything is escaped by default.
I'm not saying it's completely trivial or that there's never an issue here or there. What I'm saying is, it's on par with any of dozens of other issues in programming. Bugs happen, errors happen, but no more so than anyone else. A series of systems with slightly different encoding practices can also cause some headaches, but, again, these are on par with a number of other issues that can emerge in such systems, not especially bad. I've seen a lot of crappy code that gets this wrong at scale, written by programmers who don't really know or care what they're doing, but the same code was crap in a dozen other ways too, and generally screwed up even easier things as well.
Where you get the problems are, from largest to smallest, 1. People who don't realize it's an issue at all and concatenate everything and 2. People who have just been taught about it, and are doing a wrong thing, most often trying to filter on the way "in" instead of the way "out". ("Sanitize user input" delenda est. Stop saying it. It's wrong.) Which is also not an exceptional case, because again there are any number of things that have the exact same characteristics in the programming world.
I would expect "ridonkulously hard" to encompass something that even when tried is super hard and often a failure, and this isn't that case.
Escaped input and unescaped input are separate types. And a robust type system will allow you to craft your functions so that the streams cannot be crossed without going through translation layers.
In fact, the most robust type systems will offer things like automatic function composition so that you have to write a minimum of code... If a type coercion function is available, the type system can be taught to just automatically apply that coercion function before dropping the string into the relevant processing.
https://codeofhonor.substack.com/i/78789944/security-theater...
"Most of the rendering bugs I’ve seen in security audits don’t matter. This is not how your organization will be pwned. ... What would fix this? Layered security built around a plausible threat model. What would not help? Removing reflected ASCII text from Shodan’s API error message. I’m not saying that small security bugs aren’t worth fixing, or that organizational security always trumps application security. Rather, real damage usually does not come from where security engineers tend to expect, because they spend their time on pentests and CTFs that differ substantially from the approaches popular among actual attackers."
Everyone commenting that dealing with user input is easy: if it were really easy, we wouldn't keep making the same mistakes. I fixed my first SQL injection attack by switching some code to bind variables over 20 years ago, yet we still have Little Bobby Tables showing up in our collective databases. The fix may be easy ("just do X"), but the mistake is even easier.
Depth-first attacks as described are a different class of attack, and of course "audit" won't help that much. Education, penetration testing, and honeypots are some of the stuff that works for that.
Ultimately, if an organization treats its work force like crap, then depth-first attacks are unstoppable. The crypto-locker attackers are strangely pro-worker, because it highlights how disgruntled employees are such effective attack vectors via bribery, vengeance, or apathy.
Like, if the user might be attempting something fishy, there's no reason to try and "clean it up" and have your program "do it's best" with the remainder. Throw an error back at the user and move on to the next query.
Server: HTTP/403, begone with you, foul SQL-injecting hacker!
Also the input has to bypass validation (for which I have unit tests) and the DTOs are mapped to database models before being written.