This is really excellent, it’s the main thing that curb the ruby binding I work on never felt right. We used ruby URL for the parsing and as it turns out it’s slightly less forgiving than libcurls internal parser.
The blog post mentions WHATWG URL spec and RFC 3986 - what is libcurl's URL parser implementing, what is its goal?
By the way, parsing of URLs is a large-ish task; it'd be nice to have URL API in a separate library, which libcurl depends on, and which can be used without libcurl.
> what is libcurl's URL parser implementing, what is its goal?
I get the impression the goal would be "parsing whatever users expect to be able to use as a URL in practice". He's written before about the difficulty of pinning down "the syntax of URLs":
I've touched on a few URL parsing issues lately. Valid UTF-8 code points depending on TLD and determining the user level of the host e.g. 3rd level for co.uk seem to require gathering a bunch of disparate sources and making a best guess.
I agree it's a large task, and it'll be interesting to see the scope of libcurl's API for it.
It will be consistent with the logic used by the codebase that actually fetches the URL (curl), so it won't leave gaps between the expected behavior and actual behavior the way two disjoint implementations would.
> By offering applications access to libcurl's own URL parser, we hope to tighten a problematic vulnerable area for applications where the URL parser library would believe one thing and libcurl another. This could and has sometimes lead to security problems. (See for example Exploiting URL Parser in Trending Programming Languages! by Orange Tsai)
10 comments
[ 2.3 ms ] story [ 31.1 ms ] threadBy the way, parsing of URLs is a large-ish task; it'd be nice to have URL API in a separate library, which libcurl depends on, and which can be used without libcurl.
I get the impression the goal would be "parsing whatever users expect to be able to use as a URL in practice". He's written before about the difficulty of pinning down "the syntax of URLs":
https://daniel.haxx.se/blog/2016/05/11/my-url-isnt-your-url/
I agree it's a large task, and it'll be interesting to see the scope of libcurl's API for it.
> By offering applications access to libcurl's own URL parser, we hope to tighten a problematic vulnerable area for applications where the URL parser library would believe one thing and libcurl another. This could and has sometimes lead to security problems. (See for example Exploiting URL Parser in Trending Programming Languages! by Orange Tsai)