> LineReader splits input into lines, handles \n and \r\n, and trims the stray trailing \r that malformed input likes to leave behind
Is there a common source of extra \r in malformed inputs, beyond those existing as part of \r\n? Or is this just a dig at Windows-style line endings? If there's something weird going on I think I'd rather fail loudly.
> Bounding the inner scanner to a single line makes “run past the end of a malformed line” unrepresentable rather than merely unlikely.
I don't really see what makes it "unrepresentable", and this reads more like "if you used the right scanning logic, you can't have used the wrong scanning logic".
The hardest thing about writing a parser is cognitively accepting what is going to be considered valid input. You can make the best parser that is fast and well specified but invariably someone will (ab)use it in an unexpected way.
Famous examples: despite so many initial good intentions, html tags don’t need to be closed, JSON numbers are too often encoded as strings, YAML can look like what most people expect or it can look progressively more like JSON… and on and on.
And harder than that? Report the error, in a way that make some sense.
This is compounded by the fact that you need the semantics involved, the environment (ie: everything on scope), the source (that means you need to keep carrying big strings around).
And what is efficient means to be destructive, but you need instead the opposite for semantics, error messages, optimizations and the like.
This post doesn't touch on something that makes parsers complicated no matter how simple the grammar: good error messages. Parsing a well formed input is the easy part, but not just spitting out a byte index but actually telling the user why their input is not good and what they could do to make it conform is super hard.
The Rust compiler is a common example of a compiler that does a good job here, and I think it is one of only a few.
I'm going to collect this post after 24 hours, extract the methodologies from everyone's comments, and write them down in my notes. The reason I like HN is that people freely share their tips in the comments
If you created a format that is so difficult to parse that it cannot be parsed with simple readable C code then the problem is the format not the parser code.
FORTH parsers are ultra simple - get the next space-separated token, if it is a number, push it on the stack, otherwise it's a word - look it up in the dictionary and (if it exists there) execute it.
A parser/compiler could obviously be improved with an LLM (AI!) to suggest improvements to invalid input. That is actually a super good use case of LLM/AI.
Having clang/gcc, or any other parser, implement that is of course impossible, they are too conservative and would rather die than to implement modern helpful tools.
14 comments
[ 2.8 ms ] story [ 32.7 ms ] threadLooking at the linked URL parser, why doesn't it look like
It looks totally ad-hoc.Is there a common source of extra \r in malformed inputs, beyond those existing as part of \r\n? Or is this just a dig at Windows-style line endings? If there's something weird going on I think I'd rather fail loudly.
> Bounding the inner scanner to a single line makes “run past the end of a malformed line” unrepresentable rather than merely unlikely.
I don't really see what makes it "unrepresentable", and this reads more like "if you used the right scanning logic, you can't have used the wrong scanning logic".
One common way to test it is just to pass ipv6 url: http://[f021:d981:b487:e57d:193e:550e::]/
https://github.com/rust-bakery/nom
Famous examples: despite so many initial good intentions, html tags don’t need to be closed, JSON numbers are too often encoded as strings, YAML can look like what most people expect or it can look progressively more like JSON… and on and on.
This is compounded by the fact that you need the semantics involved, the environment (ie: everything on scope), the source (that means you need to keep carrying big strings around).
And what is efficient means to be destructive, but you need instead the opposite for semantics, error messages, optimizations and the like.
The Rust compiler is a common example of a compiler that does a good job here, and I think it is one of only a few.
Is this really ergonomic?
> trims the stray trailing \r that malformed input likes to leave behind
how does this distinguisg the non-stray variety?
Having clang/gcc, or any other parser, implement that is of course impossible, they are too conservative and would rather die than to implement modern helpful tools.