> Because in golang everything is UTF-8. After all, Rob Pike authored both golang and utf-8.
That's a terrible reason. HTTP header values are not necessarily UTF-8, and an implementation that blindly assumes they are is broken.
> Now, what are the security implications of this? (beside the chunk thingy)
Many variations of the "chunk thingy" where values considered safe in one context may be unsafe in another context.
Like so many parts of Golang, this is a good example of how "good enough" engineering often really isn't good enough. HTTP is pretty thoroughly specified, and the HTTP stack included in a language should be thoroughly tested against that spec. If you can't achieve that, just leave it out of the stdlib!
> Actually, they are explicitly specified as ASCII
They are not, only new headers added after RFC 7320 are ASCII, pre-existing ones may contain high-bit characters which implementations compliant with 7230 version of HTTP/1.1 SHOULD not parse. They stopped short from making it a MUST so that applications compliant with RFC 2616 (where they were to be treated as ISO-8859-1!) was allowed. Or, more simply, they stopped themselves from making RFC 7230 describe new, incompatible HTTP/1.2.
Yes, but this simply means that header values are not (Go) strings, they're byte arrays. Many languages represent them this way, with a (fallible) function to get the string value iff the byte array contains valid UTF-8.
The whole "it would be more work" as a justification for not doing things properly is one of the roots of the more general issues with Go I alluded to in my post above.
1) Go string type is specified as a sequence of bytes. Full stop. Any encoding, binary blob, string does not care. Particular stdlib string functions _do_ care, expect UTF-8 encoding and document the fact. Other stdlib string function work with the sequence as a sequence.
3) Go specification requires UTF-8 in only one place: the encoding of Go source files string literals.
2) Pure ASCII is a subset of UTF-8, ie. pure ASCII _is_ valid UTF-8. There is _no_ more work and there are _no_ security problems when working with pure ASCII as UTF-8, because that's what it is.
I think it's worth spelling out that 'chunk thingy' can lead to complete site takeover. For example, using this behaviour I was able to get persistent control of PayPal's login page.
That said, I tweeted this rather than posting a whitepaper because I haven't thoroughly explored the impact. I've shared my guess about what other issues you'll find here: https://news.ycombinator.com/item?id=25858015
> In a go a string is an immutable byte slice assumed to be valid utf8, with associated methods and users which all assume that slice of memory is valid utf8.
Which is in my opinion the worst choice you could make when designing a type. If I'm working in a typed language because of so-called "type safety", if I can't trust my types to actually enforce their guarantees, what can I trust ?
Exactly. It's a convention, not a law of physics. If a string's UTF-8 validity is important to the correctness of your function, check it. It's not expensive.
The real problem is that Go does not actually have a string type. It only has byte strings which can accidentally also be text strings encoded in utf-8.
This is a very, very sloppy way to handle text, and will inevitably lead to a huge number of mistakes and broken software.
Any well-designed modern language really should make a strong separation between raw byte buffers and text strings. They are very different things, and one can definitely not trivially be converted to the other.
But there's a bunch of hard-to-quantify side-risks based on corner-cases that could lead to stuff like request header injection, server-side parameter pollution, and host-header attacks. I'm excited to see what shows up.
Technically isn't incorrect but is a violation of SHOULD. I still think a SHOULD should be a requirement for a general purpose library. With reasons given for not following it.
Edit: And with HTTP having case-insensitive matching (which is most likely broken in lots of hand written implementations). This is rife with the possibility for errors
Taken from RFC 7230 3.2.4 Field parsing
Historically, HTTP has allowed field content with text in the
ISO-8859-1 charset [ISO-8859-1], supporting other charsets only
through use of [RFC2047] encoding. In practice, most HTTP header
field values use only a subset of the US-ASCII charset [USASCII].
Newly defined header fields SHOULD limit their field values to
US-ASCII octets. A recipient SHOULD treat other octets in field
content (obs-text) as opaque data.
> A recipient SHOULD treat other octets in field
content (obs-text) as opaque data.
This is not really a 'should', IMHO, because fields are defined as OCTETS, iirc. Based on that, a compliant and robust implementation must treat them as opaque data.
RFC 7230 makes it a point not to make it a MUST as that would make unknown number of existing applications non-compliant with HTTP/1.1-as-redefined. They are free to treat the incoming headers as ISO-8859-1 8bit instead of dropping to 7bit US-ASCII.
RFC 2616 defined header fields as OCTETs, and regarding this change RFC 7230 states:
> Non-US-ASCII content in header fields and the reason phrase has been obsoleted and made opaque (the TEXT rule was removed).
RFC 2616:
field-value = ( field-content | LWS )
field-content = <the OCTETs making up the field-valu and consisting of either TEXT or combinations of token, separators, and quoted-string>
Hence to me fields must be treated as opaque data for backward compatibility and robustness. If anything, existing applications that are compliant with RFC 2616 already do that, right? ;)
RFC 2616 OCTETs are defined as "<any 8-bit sequence of data>" quote unquote, nothing is said about their value beign opaque.
TEXT = <any OCTET except CTLs,
but including LWS>
IETF rewrote the productions not to use TEXT, but stopped short from banning the old behaviour.
So, for instance, where 2616 states:
Reason-Phrase = <TEXT, excluding CR, LF>
And 7230 has:
reason-phrase = ( HTAB / SP / VCHAR / obs-text )
It is making sure that any application that conforms to 2616 still conforms to 7230 by not making it illegal (MUST) to parse obs-text... Just something you SHOULD not not do. They are simply making it so any new header added is defined as SP / VCHAR only (quoted, possibly).
Let's not argue semantics here. An arbitrary sequence of bytes is an opaque data type, it has no structure, no meaning, no assumption can be made, and it must simply be passed on as is because it can be anything.
That's why they write that it should be treated as opaque data. My point (and the point of the comment I was replying to) is that 'should' is perhaps too weak a word in the context because previous history. In any case for robustness it is a must to treat it that way.
… which does not permit non-ASCII. As a generic header, maybe (but the decoding should be into ISO-8859-1, as the RFC notes…), but at the point at which you parse it into a Transfer-Encoding header, it is no longer valid.
Every time someone creates a new programming language, the world builds yet another pile of broken HTTP implementations--clients and servers--because everyone refuses to read the specifications, thinking "none of this is hard". I remember dealing with some early Ruby web servers, 15 years ago, and was just shocked at how broken they were despite being used by everyone in production! In fact, there is a lot of nuance to HTTP, and as a civilization we seemed doomed to reliving the same mistakes over and over again every few years when the entire stack gets rewritten from scratch :/.
So, I thereby was neither surprised to read that the Go HTTP server did this obviously-wrong thing... NOR was I particularly surprised to read the person complaining about it also didn't seem to understand the correct behavior, either (though I do appreciate this is a bit "unfair"--I mean, it's a tweet ;P--but I think still makes for an interesting point here) :/. I have dealt with so many broken HTTP clients and servers over the years written by people who probably also never read the manual that came with their microwave because "documentation is for idiots" or whatever, that I have just become demoralized about it :(.
A few months ago, I was complaining about this, with a short list of stuff everyone tended to get wrong, and this HTTP header field encoding issue was on my shortlist :(.
> FWIW, I have encountered tons of issues in peoples' "this is easy, right?" implementations of 100 Continue (including in some at-the-time major Ruby HTTP servers), non-ASCII encoding in HTTP headers (which has a ton of subtle rules, and just throwing around UTF-8 is not correct), and chunked Transfer-Encoding (though I can't remember what was wrong); further, people often don't even realize HTTP trailers exist at all, which is a bit infuriating :/. (Note, though, that I am still 100% on the side of the anti-HTTP/2 camp in this thread.) The core problem is that somehow people think they can implement a spec without even, you know, reading the spec, which is so far past hubris as to have gone through incompetence into obvious negligence.
AFAIK, HTTP header fields are (in their generality) specified to be read as binary octets... but then are restricted to using ISO-8859-1 with the premise you are supposed to run them through a MIME parser? Like, some fields have strict formats, so you can't do the full decoding until you know what you are working with--and so I would argue an HTTP library should give you the raw binary data (though decoding them through ISO-8859-1 first isn't supposed to be "wrong")--but for fields with user-displayable text (including HTTP status messages!) you can seriously embed "Q" encoded strings (of all things) if you want, a la RFC 2047. You can't assume a field is restricted to ASCII too early in the parsing process.
reading this i'm getting flashbacks from python 2 -> 3 migration and all those people complaining that they're now getting Unicode*Errors thrown at them. the fact is, none of this stuff is simple! you can pretend it is until a corner case bites you in the behind, suddenly the tower of abstraction falls apart and you're left picking up the parts in four different string encoding formats. (see the abysmal C which at least doesn't pretend anything at all, C++ which doesn't require you to know all this stuff and then it just blows up if you make a particularly wrong misstep and most recently Rust, which will call you names for using the wrong type in the wrong place, in turn greatly confusing people who say strings are easy).
In this case, Python and Rust's behaviours are correct. You only get type errors / Unicode* exceptions when you're making wrong assumptions; if you get them, and don't understand why, you're writing buggy code to do things you don't understand and should be thankful for the errors letting you know there's a problem.
So, my favorite thought process from the Python 2 to 3 transition comes from staring deeply into the hubris of the Python 3000 effort itself as wanting to insist "everything is Unicode"; on the one hand they were trying to shame developers using Python 2 for using it incorrectly (even though it was totally possible to handle Unicode correctly with that stack), yet on the other they were pumping out ridiculous unavoidable security SNAFUs in their API surface by assuming that filenames on disk are UTF-8 :/.
To tell another story of the importance of reading the--in this case, not even long!--specification: on Windows, filenames are stored as UCS-2 (as it all predates UTF-16), and is most likely most correctly modeled as an arbitrary sequence of non-0 16-bit values; meanwhile, on Unix systems, a filename is pretty much just an arbitrary sequence of non-0 8-bit values (with some weirdness around the slash character... the API surface doesn't support it really but sometimes the filesystem does). The premise that filenames have a display encoding mostly comes from the software rendering the strings and is then defined in terms of a bunch of hacks involving "locales", and isn't a property of the underlying API or storage semantics.
You can't assume in either case that they are canonical Unicode representations, and so storing them as "strings" in your program is super wrong and can lead to anything from data storage mistakes to security vulnerabilities. The Python 3 API filesystem API was thereby so bad that it would do things like just skip over files during directory traversal that didn't map to its attempt to decode stuff as Unicode. Like: if you wanted to hide a file from a Python 3 program, you could just make it invalid Unicode :/.
(I realized people reading this comment might find a reference on this fun, and so found this seemingly-comprehensive postmortem. I haven't read it, but it at least looks really good ;P.)
I thereby find the whole Python 3 debacle extra demoralizing as it felt like a bunch of people who didn't truly understand how hard the problem they were dealing with was--and how much of the world actually is binary, and how often (when dealing with things like filesystems and network protocols) parsing binary sequences as pseudo-strings is "correct"--both sometimes-correctly insisting everyone was wrong while themselves failing to build something even minimally useful until like, Python 3.4 or so; and, in the end, I feel like they just ended up creating a "new mess" by having filenames now encoded as "Unicode strings"... but with weird exceptions for allowing round-trip behaviors of strings that couldn't be decoded :(.
The reality is that there are a number of things that developers work with which are pseudo-strings, and languages should provide good tooling for working with them. We see the same issue come up with HTTP headers and filenames: things that are actually binary but have string-like structure. There is a reason why both C++ and Python 2 are often looked at fondly by low-level developers for providing low-level string representations, even while they end up encouraging broken behavior higher up the stack. I haven't looked enough at Rust's standard library to see if they truly nailed this yet, but maybe one day we will have a language that encourages the right behavior for these mechanisms that are neither strictly string not strictly binary.
I just did some digging, and the Python 3 built-in HTTP stuff seems to represent HTTP headers using a reasonably-comprehensive mechanism that is shared with MIME parsing and the various IMF semantics, and so I am pretty sure you end up getting back an object of type email.header.Header (but I might have misunderstood the documentation just now... I mostly gave up on Python when they moved to Python 3 so I haven't experienced this first hand).
This actually looks like a really solid implementation--like, I am super impressed by this (though watch it suck... everything sucks ;P)--as it doesn't assume any ability to flatten the result! That said, I am also deeply concerned that this might pre-decode something that should actually be in ASCII all the way down from MIME (which, notably, is complex enough of a format to include support for embedded comments). I kind of want to do some auditing here, to make sure there aren't some parser discrepancy vulnerabilities in common Python HTTP frameworks ;P.
But like, I think this object represents a sequence of strings each with its own encoding, allowing it to "Q" encode the result exactly as specified (and correctly round trip something it parsed, which still does feel a tad dangerous, and maybe the API lets you avoid it in those cases). It then has an __str__ implementation (to let you use it as a string) that explicitly claims to return an "approximation" of the header (which may or may not be a safe thing to use in different circumstances; but like, that in some sense isn't the API's fault?...).
Rust is a bit notorious for its string-related complexity, and that's for the reasons you talked about.
The most common form of string you'll see is `String`[0], or its view type `str`[1], which are UTF-8 encoded. These have most of the string functions you'd expect to have.
Of course, as you mentioned, operating systems don't all use UTF-8, which brings us to `OsString`[2], and its view type `OsStr`[3]. This is the data type that string data from the OS is represented as. Both are basically bundles of bytes, with no specific encoding. You'll note that these are in the FFI namespace.
You then have `PathBuf`[4] and its view type `Path`[5], which are wrappers around `OsString` and `OsStr` respectively, and provide path-specific functionality.
All the file system related APIs take anything that implements `AsRef<Path>`, which is runtime-cost-free for all of these types. There's not many functions that take an `OsStr`, but those that do take it in the form `AsRef<OsStr>`, which like `Path` is runtime-cost-free for all these types. None of the APIs take `PathBuf` or `OsString`.
Return values are typically `PathBuf` for file system APIs, and `OsString` for other APIs. The only exceptions I know of are the functions for getting environment variables and command line arguments, which also provide functions that return `String` because a common use case would be converting them anyway.
I quite like what languages with proper sum types -- Rust, OCaml, Haskell -- do in these situations.
We cannot tolerate half-written buggy clients for an eternity.
Let us have more errors. Computers aren't humans. You have to specify things precisely, not throw a blob at them and let them figure it out. They can't. The attempts at that have only led to countless bugs and costly security breaches.
is there only one rfc for http? How can I find the related one's? Tried that during studying how a GET/POST etc. works and what the diff to HTTP/2 was. But I always got a bunch of rfc's where I didn't know if they are related or already obsolete.
So, it's a mess to navigate, but in general at the beginning of the RFC you have a list of other documents that update or obsolete it, and also list of documents updated or obsoleted by it.
Great! I didn't saw the information on top left. For example for the header compression there isn't a two way connection? I see only a reference to the http rfc.
Or you just want to interact with a few servers, so you implement what the servers do
In my case I wanted to interact with a few servers and not write a HTTP client, so I used an existing HTTP client library. Turns out, the library did not handle those servers, so I had to write the header parsing myself, at least the cookie parsing.
I wonder if there are RFC 2047 cookies and you need to parse them, or just send them back as they came
HTTP headers are generally treated as ASCII, and Golang's decision to treat them as UTF-8 opens the gate to a range of issues, including the linked unicode normalization attack.
Indeed --- if HTTP was a binary protocol, with headers and their values being defined constant integers, this would definitely not happen. No one would confuse 1 with 2, and implementations would probably be quite a bit more efficient... but the "lamers" really want their pile of leaky abstractions because they can't be bothered using a hex editor. </s>
Nevermind the fact that some parts of HTTP are really just binary written as text, and now even most non-developers know what a 404 is...
I didn’t know this kind of attack before but wouldn't any sane backend system confirm the permissions of requesting client first before handing out admin pages?
I mean you need to have a client-id and their permissions for any outside request coming to server.
I’m using Golang in my webserver but it wouldn’t hand-out secret pages as requesting client wouldn’t have authority to do so.
I feel like Golang's UTF8 handling is the least of the worries for these kinds of attacks, it points out that entire backend system was constructed wrongly, number one rule of any backend systems: You cannot trust the client.
There are sequences in ISO-8859-1 that will cause UTF8 application to balk if they are read without conversion - sometimes in a part that is well away from the input itself.
A common example "órny", part of a number of Polish place and personal names. ISO 8859-1 (and -2, and others) form would be F3 72 6E 79. UTF-8 treats F3 as a start of a 4 byte sequence where each subsequent character is in 80 - BF range (as specified in RFC 3629):
This nicely causes an exception in both Java and Python, for instance, if they were to do what Go does here - and there is nothing anyone from adding an X- header to the request or response to try just that.
This is the cost of "extraneous hidden complexity", or as I like to think of it, "abstraction hides bugs". As others here have noted, it is all too tempting to use a language whose string type actually does something very fancy, and be unaware of it. Most of the time it "only" causes inefficiency.
I don't remember if the HTTP spec has it, but I clearly remember the HTML spec has some parts that explicitly say "case-insensitive in the ASCII range only" (i.e. 61-7a = 41-5a), presumably to prevent things like this from happening.
It's for two (well, at least two) reasons: one is to prevent things like this. The other is that in generally doing case-insensitive stuff for non-ASCII is language-dependent.
For example, in English "i" and "I" are case-insensitively equal but in Turkish they are not: "i" is case-insensitively equal to "İ" (U+0130) and "I" is case-insensitively equal to "ı" (U+0131).
So if you wanted to do full-unicode case-insensitive things you would need to decide what happens with <DİV> or <dıv> and whether it depends on the language of the page or what. Doing ASCII-only case-insensitivity sidesteps all that, while admittedly privileging English over other languages.
There's a reply in that Twitter Thread. Go will be switching to ASCII: "Thanks for the ping. Our policy is that anything that requires a proxy to forward an invalid Transfer-Encoding header is a vulnerability in the proxy, not in the server. We are planning to switch all net/http to ASCII strings functions anyway for defense in depth and correctness." - Filippo Valsorda, Go Security Lead
I do love the simplicity of Go's UTF-8[0] approach, though this did also burn me when I was first learning the language and did not really understand the implication of it.
I translated an x-face encoder from js to go for one of my introductory projects and initially couldn't get the correct output from `b64.StdEncoding.EncodeToString` due to the fact that x-face strings are single-byte ASCII ISO-8859-1 encoded[1].
> why not just have a utf-8 string type in addition to a byte string type?
I'm not a go expert, but my understanding is that go "strings" are really all just a series of bytes and it's the standard library functions which treat string operations as UTF-8 encoded by default. I think this reddit post[0] explains it nicely. I suppose the simplicity of "just work with the bytes and apply encoding when necessary" follows go's simplicity and composability/interface patterns well.
> Also, as a nit, I don't think iso-8859-1 is ASCII.
We're splitting hairs here, but ISO-8859-1 is the eight-bit extension to 7-bit ASCII, no? It includes the standard 128 ASCII characters + up to an additional 128 characters (191 in total for ISO-8859-1 Latin-1 Western European, iirc).
The real wtf here is a separate K for Kelvin, a separate N for Newton, etc. Why on gods' greenish earth do we need something that looks like a duck, quacks like a duck, but actually is not a duck but an eldritch horror?
One of the design criteria for Unicode was to guarantee lossless round-trip conversion through Unicode for a bunch of encodings that Unicode superceded. If one of those encodings contained both a K and a K-for-Kelvin, then Unicode also had to allocate two codepoints for it. I don't know if that's the case for these characters in particular, but it is one possible explanation.
Supporting round-trip conversion may sound pointless now that Unicode is ubiquitous, but it was probably necessary to make gradual adoption practical.
That could explain it, but it doesn't make it any less crazy. I'm tempted to use K-for-Kelvin in my code now, just to fuck with anyone who works with a worse IDE or tries finding things with grep.
74 comments
[ 4.4 ms ] story [ 147 ms ] threadNow, what are the security implications of this?
(beside the chunk thingy)
That's a terrible reason. HTTP header values are not necessarily UTF-8, and an implementation that blindly assumes they are is broken.
> Now, what are the security implications of this? (beside the chunk thingy)
Many variations of the "chunk thingy" where values considered safe in one context may be unsafe in another context.
Like so many parts of Golang, this is a good example of how "good enough" engineering often really isn't good enough. HTTP is pretty thoroughly specified, and the HTTP stack included in a language should be thoroughly tested against that spec. If you can't achieve that, just leave it out of the stdlib!
Actually, they are explicitly specified as ASCII ("follow the same generic format as that given in Section 3.1 of RFC 822"): https://tools.ietf.org/html/rfc822#section-3.1
They are not, only new headers added after RFC 7320 are ASCII, pre-existing ones may contain high-bit characters which implementations compliant with 7230 version of HTTP/1.1 SHOULD not parse. They stopped short from making it a MUST so that applications compliant with RFC 2616 (where they were to be treated as ISO-8859-1!) was allowed. Or, more simply, they stopped themselves from making RFC 7230 describe new, incompatible HTTP/1.2.
https://tools.ietf.org/html/rfc7230#section-3.2.4
Consider this: all strings are utf-8, so working with pure ASCII is much more work in golang.
It leads to all kind of issues, not only security problems.
The whole "it would be more work" as a justification for not doing things properly is one of the roots of the more general issues with Go I alluded to in my post above.
3) Go specification requires UTF-8 in only one place: the encoding of Go source files string literals.
2) Pure ASCII is a subset of UTF-8, ie. pure ASCII _is_ valid UTF-8. There is _no_ more work and there are _no_ security problems when working with pure ASCII as UTF-8, because that's what it is.
That said, I tweeted this rather than posting a whitepaper because I haven't thoroughly explored the impact. I've shared my guess about what other issues you'll find here: https://news.ycombinator.com/item?id=25858015
And I posted a full whitepaper/presentation on the technique here: https://portswigger.net/research/http-desync-attacks-request...
[0] https://daringfireball.net/linked/2020/05/02/psychic-paper
You would think an old http header is a string. It's not. It looks like a string though :)
Actually its not guaranteed to be UTF-8, only a sequence of bytes with some convenience methods: https://golang.org/ref/spec#String_types
However, the existence of the 'strings' package and the 'range' operator give the impression that all strings are UTF-8.
> In a go a string is an immutable byte slice assumed to be valid utf8, with associated methods and users which all assume that slice of memory is valid utf8.
Which is in my opinion the worst choice you could make when designing a type. If I'm working in a typed language because of so-called "type safety", if I can't trust my types to actually enforce their guarantees, what can I trust ?
This is not a defence of Go but more of a warning to Go programmers.
This is a very, very sloppy way to handle text, and will inevitably lead to a huge number of mistakes and broken software.
Any well-designed modern language really should make a strong separation between raw byte buffers and text strings. They are very different things, and one can definitely not trivially be converted to the other.
But there's a bunch of hard-to-quantify side-risks based on corner-cases that could lead to stuff like request header injection, server-side parameter pollution, and host-header attacks. I'm excited to see what shows up.
Edit: And with HTTP having case-insensitive matching (which is most likely broken in lots of hand written implementations). This is rife with the possibility for errors
Taken from RFC 7230 3.2.4 Field parsing
This is not really a 'should', IMHO, because fields are defined as OCTETS, iirc. Based on that, a compliant and robust implementation must treat them as opaque data.
> Non-US-ASCII content in header fields and the reason phrase has been obsoleted and made opaque (the TEXT rule was removed).
RFC 2616:
field-value = ( field-content | LWS )
field-content = <the OCTETs making up the field-valu and consisting of either TEXT or combinations of token, separators, and quoted-string>
Hence to me fields must be treated as opaque data for backward compatibility and robustness. If anything, existing applications that are compliant with RFC 2616 already do that, right? ;)
So, for instance, where 2616 states: Reason-Phrase = <TEXT, excluding CR, LF> And 7230 has: reason-phrase = ( HTAB / SP / VCHAR / obs-text )
It is making sure that any application that conforms to 2616 still conforms to 7230 by not making it illegal (MUST) to parse obs-text... Just something you SHOULD not not do. They are simply making it so any new header added is defined as SP / VCHAR only (quoted, possibly).
That's why they write that it should be treated as opaque data. My point (and the point of the comment I was replying to) is that 'should' is perhaps too weak a word in the context because previous history. In any case for robustness it is a must to treat it that way.
So, I thereby was neither surprised to read that the Go HTTP server did this obviously-wrong thing... NOR was I particularly surprised to read the person complaining about it also didn't seem to understand the correct behavior, either (though I do appreciate this is a bit "unfair"--I mean, it's a tweet ;P--but I think still makes for an interesting point here) :/. I have dealt with so many broken HTTP clients and servers over the years written by people who probably also never read the manual that came with their microwave because "documentation is for idiots" or whatever, that I have just become demoralized about it :(.
A few months ago, I was complaining about this, with a short list of stuff everyone tended to get wrong, and this HTTP header field encoding issue was on my shortlist :(.
https://news.ycombinator.com/item?id=24835617
> FWIW, I have encountered tons of issues in peoples' "this is easy, right?" implementations of 100 Continue (including in some at-the-time major Ruby HTTP servers), non-ASCII encoding in HTTP headers (which has a ton of subtle rules, and just throwing around UTF-8 is not correct), and chunked Transfer-Encoding (though I can't remember what was wrong); further, people often don't even realize HTTP trailers exist at all, which is a bit infuriating :/. (Note, though, that I am still 100% on the side of the anti-HTTP/2 camp in this thread.) The core problem is that somehow people think they can implement a spec without even, you know, reading the spec, which is so far past hubris as to have gone through incompetence into obvious negligence.
AFAIK, HTTP header fields are (in their generality) specified to be read as binary octets... but then are restricted to using ISO-8859-1 with the premise you are supposed to run them through a MIME parser? Like, some fields have strict formats, so you can't do the full decoding until you know what you are working with--and so I would argue an HTTP library should give you the raw binary data (though decoding them through ISO-8859-1 first isn't supposed to be "wrong")--but for fields with user-displayable text (including HTTP status messages!) you can seriously embed "Q" encoded strings (of all things) if you want, a la RFC 2047. You can't assume a field is restricted to ASCII too early in the parsing process.
https://tools.ietf.org/html/rfc2047
To tell another story of the importance of reading the--in this case, not even long!--specification: on Windows, filenames are stored as UCS-2 (as it all predates UTF-16), and is most likely most correctly modeled as an arbitrary sequence of non-0 16-bit values; meanwhile, on Unix systems, a filename is pretty much just an arbitrary sequence of non-0 8-bit values (with some weirdness around the slash character... the API surface doesn't support it really but sometimes the filesystem does). The premise that filenames have a display encoding mostly comes from the software rendering the strings and is then defined in terms of a bunch of hacks involving "locales", and isn't a property of the underlying API or storage semantics.
You can't assume in either case that they are canonical Unicode representations, and so storing them as "strings" in your program is super wrong and can lead to anything from data storage mistakes to security vulnerabilities. The Python 3 API filesystem API was thereby so bad that it would do things like just skip over files during directory traversal that didn't map to its attempt to decode stuff as Unicode. Like: if you wanted to hide a file from a Python 3 program, you could just make it invalid Unicode :/.
(I realized people reading this comment might find a reference on this fun, and so found this seemingly-comprehensive postmortem. I haven't read it, but it at least looks really good ;P.)
https://vstinner.github.io/python30-listdir-undecodable-file...
I thereby find the whole Python 3 debacle extra demoralizing as it felt like a bunch of people who didn't truly understand how hard the problem they were dealing with was--and how much of the world actually is binary, and how often (when dealing with things like filesystems and network protocols) parsing binary sequences as pseudo-strings is "correct"--both sometimes-correctly insisting everyone was wrong while themselves failing to build something even minimally useful until like, Python 3.4 or so; and, in the end, I feel like they just ended up creating a "new mess" by having filenames now encoded as "Unicode strings"... but with weird exceptions for allowing round-trip behaviors of strings that couldn't be decoded :(.
The reality is that there are a number of things that developers work with which are pseudo-strings, and languages should provide good tooling for working with them. We see the same issue come up with HTTP headers and filenames: things that are actually binary but have string-like structure. There is a reason why both C++ and Python 2 are often looked at fondly by low-level developers for providing low-level string representations, even while they end up encouraging broken behavior higher up the stack. I haven't looked enough at Rust's standard library to see if they truly nailed this yet, but maybe one day we will have a language that encourages the right behavior for these mechanisms that are neither strictly string not strictly binary.
How does Python 3 handle these HTTP headers and other things that are purely ASCII? Does it treat them as type `bytes` instead of (Unicode) `str`?
https://docs.python.org/3/library/email.header.html#email.he...
This actually looks like a really solid implementation--like, I am super impressed by this (though watch it suck... everything sucks ;P)--as it doesn't assume any ability to flatten the result! That said, I am also deeply concerned that this might pre-decode something that should actually be in ASCII all the way down from MIME (which, notably, is complex enough of a format to include support for embedded comments). I kind of want to do some auditing here, to make sure there aren't some parser discrepancy vulnerabilities in common Python HTTP frameworks ;P.
But like, I think this object represents a sequence of strings each with its own encoding, allowing it to "Q" encode the result exactly as specified (and correctly round trip something it parsed, which still does feel a tad dangerous, and maybe the API lets you avoid it in those cases). It then has an __str__ implementation (to let you use it as a string) that explicitly claims to return an "approximation" of the header (which may or may not be a safe thing to use in different circumstances; but like, that in some sense isn't the API's fault?...).
The most common form of string you'll see is `String`[0], or its view type `str`[1], which are UTF-8 encoded. These have most of the string functions you'd expect to have.
Of course, as you mentioned, operating systems don't all use UTF-8, which brings us to `OsString`[2], and its view type `OsStr`[3]. This is the data type that string data from the OS is represented as. Both are basically bundles of bytes, with no specific encoding. You'll note that these are in the FFI namespace.
You then have `PathBuf`[4] and its view type `Path`[5], which are wrappers around `OsString` and `OsStr` respectively, and provide path-specific functionality.
All the file system related APIs take anything that implements `AsRef<Path>`, which is runtime-cost-free for all of these types. There's not many functions that take an `OsStr`, but those that do take it in the form `AsRef<OsStr>`, which like `Path` is runtime-cost-free for all these types. None of the APIs take `PathBuf` or `OsString`.
Return values are typically `PathBuf` for file system APIs, and `OsString` for other APIs. The only exceptions I know of are the functions for getting environment variables and command line arguments, which also provide functions that return `String` because a common use case would be converting them anyway.
[0] https://doc.rust-lang.org/std/string/struct.String.html
[1] https://doc.rust-lang.org/std/primitive.str.html
[2] https://doc.rust-lang.org/std/ffi/struct.OsString.html
[3] https://doc.rust-lang.org/std/ffi/struct.OsStr.html
[4] https://doc.rust-lang.org/std/path/struct.PathBuf.html
[5] https://doc.rust-lang.org/std/path/struct.Path.html
We cannot tolerate half-written buggy clients for an eternity.
Let us have more errors. Computers aren't humans. You have to specify things precisely, not throw a blob at them and let them figure it out. They can't. The attempts at that have only led to countless bugs and costly security breaches.
As an example: https://tools.ietf.org/html/rfc6126. The document starts with "Updated by: 7298, 7557".
Regarding HTTP, if you pass by the wikipedia article, you have the list of RFC on the menu on the right: https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol.
RFC 1945 HTTP/1.0 (1996)
RFC 2616 HTTP/1.1 (1999)
RFC 7540 HTTP/2 (2015)
RFC 7541 Header Compression (2, 2015)
RFC 7230 Message Syntax and Routing (1.1, 2014)
RFC 7231 Semantics and Content (1.1, 2014)
RFC 7232 Conditional Requests (1.1, 2014)
RFC 7233 Range Requests (1.1, 2014)
RFC 7234 Caching (1.1, 2014)
RFC 7235 Authentication (1.1, 2014)
- RFC are inconsistent
- out of touch with the real world
- contain security issues
- not clear
- contradict themselves
So you can look at three majors HTTP server and sometimes they will implement / allow things differently becauset they have no other choice.
In my case I wanted to interact with a few servers and not write a HTTP client, so I used an existing HTTP client library. Turns out, the library did not handle those servers, so I had to write the header parsing myself, at least the cookie parsing.
I wonder if there are RFC 2047 cookies and you need to parse them, or just send them back as they came
This tells a lot about HTTP as well.
Nevermind the fact that some parts of HTTP are really just binary written as text, and now even most non-developers know what a 404 is...
I mean you need to have a client-id and their permissions for any outside request coming to server.
I’m using Golang in my webserver but it wouldn’t hand-out secret pages as requesting client wouldn’t have authority to do so.
I feel like Golang's UTF8 handling is the least of the worries for these kinds of attacks, it points out that entire backend system was constructed wrongly, number one rule of any backend systems: You cannot trust the client.
A common example "órny", part of a number of Polish place and personal names. ISO 8859-1 (and -2, and others) form would be F3 72 6E 79. UTF-8 treats F3 as a start of a 4 byte sequence where each subsequent character is in 80 - BF range (as specified in RFC 3629):
This nicely causes an exception in both Java and Python, for instance, if they were to do what Go does here - and there is nothing anyone from adding an X- header to the request or response to try just that.Finally just wrote code to loop through all possible encodings and ways to change encodings. Wait until a match against a known string was found.
Proper encodings found in a few minutes.
I don't remember if the HTTP spec has it, but I clearly remember the HTML spec has some parts that explicitly say "case-insensitive in the ASCII range only" (i.e. 61-7a = 41-5a), presumably to prevent things like this from happening.
For example, in English "i" and "I" are case-insensitively equal but in Turkish they are not: "i" is case-insensitively equal to "İ" (U+0130) and "I" is case-insensitively equal to "ı" (U+0131).
So if you wanted to do full-unicode case-insensitive things you would need to decide what happens with <DİV> or <dıv> and whether it depends on the language of the page or what. Doing ASCII-only case-insensitivity sidesteps all that, while admittedly privileging English over other languages.
It's great that they're fixing this anyway, but that's a bad policy.
I translated an x-face encoder from js to go for one of my introductory projects and initially couldn't get the correct output from `b64.StdEncoding.EncodeToString` due to the fact that x-face strings are single-byte ASCII ISO-8859-1 encoded[1].
[0]: https://blog.golang.org/strings
[1]: https://stackoverflow.com/questions/43461679/escaping-of-hex...
Also, as a nit, I don't think iso-8859-1 is ASCII.
I'm not a go expert, but my understanding is that go "strings" are really all just a series of bytes and it's the standard library functions which treat string operations as UTF-8 encoded by default. I think this reddit post[0] explains it nicely. I suppose the simplicity of "just work with the bytes and apply encoding when necessary" follows go's simplicity and composability/interface patterns well.
> Also, as a nit, I don't think iso-8859-1 is ASCII.
We're splitting hairs here, but ISO-8859-1 is the eight-bit extension to 7-bit ASCII, no? It includes the standard 128 ASCII characters + up to an additional 128 characters (191 in total for ISO-8859-1 Latin-1 Western European, iirc).
[0]: https://old.reddit.com/r/golang/comments/a5v3fn/are_strings_...
Supporting round-trip conversion may sound pointless now that Unicode is ubiquitous, but it was probably necessary to make gradual adoption practical.
Ref: https://en.m.wikipedia.org/wiki/Unicode_compatibility_charac...