I thought it was pretty well known that everything related to strings in C stdlib (including all str... functions) is bad. You just need to bring in your own string library.
Here's a readability tip for working with ASCII numbers: Treat adding and subtracting the ASCIIness as you would multiplying and dividing by a unit in physics. You can add '0' to convert a numeral to ASCII and subtract '0' to convert it back, and you can do direct comparisons between ASCII numerals.
if(characters[i] <= '9' && characters[i] >= '0')
{
ret = ret * 10 + characters[i] - '0';
}
One of the first homework assignments when I learned C back in '83 was after a long lecture on how the string functions are fundamentally broken, and the class introduction to writing C was fixing all of them.
I remember an old project that ran into something like this. I think we just used atoi() or similar and the error check was a string comparison between the original input and a sprintf() of the converted value.
Ugly (and not performant if in a hot path) but it works.
use self::IntErrorKind::*;
use self::ParseIntError as PIE;
// guard: radix must be 2..=36
if 2 > radix || radix > 36 {
from_ascii_radix_panic(radix);
}
if src.is_empty() {
return Err(PIE { kind: Empty });
}
// Strip leading '+' or '-', detect sign
// (a bare '+' or '-' with nothing after it is an error)
// accumulate digits, checking for overflow
Ok(result)
}
Can't you regex that given string contains just numbers and then use any of the provided methods? Then check if the returning value is a number to cater for edge cases
Ok, having a method to do that for you would be nice, but the post reads like it's an issue that std library doesn't provide you with a method behaving as you exactly want
I wasn't in this class myself, but one prof at my alma mater started his "Programming 201" class with the simplest assignment: write a C program that accepts two integers from the user and prints their sum. It actually was the only assignment for the rest of the semester, since he has a test suite that would humiliate the students gently at first, but would ultimately pipe a billion nines into stdin as the first argument.
This would be kind of a fun challenge. If you are handling random numbers, well you are limited by disk or memory size. But if the numbers are compressible ala LZ77 or Gzip, then there are ways to use the value’s compression trees to sum the numbers from the least significant digits using the LZ77 style compressed value tree representation. If you go that route, and the numbers are compressible (not random) then the question is whether the compressed input and output trees fit in memory or disk.
There's no one correct way to parse integers. Do you want to support 0x prefixes? Is a leading zero an indicator or octal, a zero-padded decimal, or a syntax error? Are you willing to accept a leading "+"? Are leading whitespaces OK? Trailing ones? Is 0x0c a whitespace? What about all the weird Unicode ones? Do you allow exponential notation (1e1)? Etc, etc.
In every language, the standard library makes some assumptions about this. In JavaScript, an empty string parses to zero.
The standard C library, which dates back to the stone age, does the simplest thing you can do without range checking, because, well, that's kinda the C paradigm. If you want parsing that handles edge cases in a specific way, you do it yourself. It's just digits.
> It is not OK to stop at the first sign of trouble, and return whatever maybe is right. “123timmy” is not a number, nor is the empty string.
None of the C functions referenced (atol, strtol, sscanf) are number-parsing functions per se. Rather, they're numeric-lexeme scanning+extraction functions.
These functions are all designed to avoid making any assumptions about the syntax of the larger document the numeric lexeme might be embedded in. You might, after all, be using a syntax where numbers can come with units on the end. Or you might be reading numbers as comma-separated values.
And, as a key point the author might be missing: C, in being co-designed with UNIX, offers primitives tuned for the context of:
- writing UNIX CLI tools that work with unbounded streams of input (i.e. piped output from other UNIX CLI tools),
- where, crucially, the stream is just text, and so carries no TLV-esque framing protocol to tell you the definitive length of a thing;
- and nor (especially in early memory-constrained systems) are you able to perform allocations of heap memory in order to employ an unbounded growable buffer for retaining the current lexeme until you do reach the end of it (which, if you could, would let you use a scanner state-machine that doubles as a parser/validator, returning either a parsed value or an error)
- but instead, to deal with the 1. unbounded input, 2. of textual encoding, 3. in constant memory, you must eagerly scan the input stream (i.e. synchronously reduce over each received byte, or at most each fixed-length N-byte chunk using a static or stack-allocated fixed-length buffer, discarding the original string bytes once reduced-over) to produce lexically-decoded (but not parsed/validated) lexemes; and then do this again, on a higher level, feeding your stream of lexemes into a fixed-sized sum-typed ring-buffer (i.e. an array-of-union-typed-lexeme-struct-type-entries), where you can then invoke a function that attempts to scan over + consume them (but unlike the original stream-parsing function, doesn't consume the buffer unless successful, and so isn't functioning as a scanner per se, but rather as an LR parser.)
If you're not writing UNIX CLI tools, direct use of the C-stdlib numeric-lexeme scan functions is operating on the wrong abstraction layer. What you want, if you have pre-framed strings that are "either valid numbers or parse errors", is to implement an actual parsing function... that can then invoke these numeric-lexer functions to do the majority of its work.
And if you're writing C, and yet you're not in UNIX-pipeline unbounded-text-stream land, but rather are parsing well-defined bounded-length "documents" (like, say, C source files)... then you probably want to use a real lexer-generator (like flex) to feed a parser-generator (like yacc/bison). Where:
- you'd validate the token in context, in the parsing phaase;
- and your lexing rules would make certain classes of input invalid at lexing time. (E.g. you can write your lexeme matching rules such that multi-digit numbers with leading zeroes, or floating-point values with no digits before/after the decimal place, simply aren't "numbers" from your lexer's perspective.)
...which means that, once again, you can "get away with" invokeing the regular C numeric-lexeme scanner functions; i.e. `yylval = atoi(yytext);` in bison terms. (And you'd want to, since doing so saves memory vs. keeping the numbers around as strings.)
The problem is that float parsing is highly non-trivial if you want it to be correct for all edge cases.
For integers, you're faster (in both development time and runtime) to write your own parser than to try and assemble the pieces in this pile of shit into a half-working one.
C++17 from_chars excluded. Incidentally, 2022 seems about right for the year that ONE open source implementation finally actually implemented the float part of that. Or was it more like 2024?
How could an api for number parsing ever be designed to return 0 for invalid input, for a function where 0 is also a common (perhaps the most common) return value for a valid input?
This wouldn't even pass a cursory sanity check of the api from a beginner developer, how did it end up in a standard library at all? Was it a mistake and then it was just too late to remove it?
Any function that can either succeed or fail, which is basically every parsing function, must typically indicate success or failure. You can terminate the program or you can return an object that itself indicates failure (such as -1 when finding a positive index) but if ALL values of the return type CAN be valid then the success state must be a separate return value.
What's the purpose of the function atol() if it doesn't have that? Is it "It's still useful for trusted input we know is a string representation of a long" (E.g. for bounded number roundtrip)? That seems awfully limited. But perhaps such a scenario was perhaps more common in 1960?
22 comments
[ 3.0 ms ] story [ 53.5 ms ] threadYes, the standard library is bad. This is by far the worst part of the C legacy. But it is not that hard to write your own.
String functions like this are not difficult at all, and you can use better naming and semantics, write faster code etc.
C is not the C standard library, ffs.
Ugly (and not performant if in a hot path) but it works.
Edit: https://doc.rust-lang.org/src/core/num/mod.rs.html#1537
interesting! It boils down to this
pub const fn from_ascii_radix(src: &[u8], radix: u32) -> Result<u32, ParseIntError> {
Ok, having a method to do that for you would be nice, but the post reads like it's an issue that std library doesn't provide you with a method behaving as you exactly want
:)
That should be opt-in via a flag, if it needs to be supported at all. Unix file permissions are the only deliberate use of octal I've ever seen.
Would make an excellent “interview question from Hell”!
In every language, the standard library makes some assumptions about this. In JavaScript, an empty string parses to zero.
The standard C library, which dates back to the stone age, does the simplest thing you can do without range checking, because, well, that's kinda the C paradigm. If you want parsing that handles edge cases in a specific way, you do it yourself. It's just digits.
Perhaps the right title should be "No way to parse pathological edge cases in 'C'"
And then see how other languages do.
None of the C functions referenced (atol, strtol, sscanf) are number-parsing functions per se. Rather, they're numeric-lexeme scanning+extraction functions.
These functions are all designed to avoid making any assumptions about the syntax of the larger document the numeric lexeme might be embedded in. You might, after all, be using a syntax where numbers can come with units on the end. Or you might be reading numbers as comma-separated values.
And, as a key point the author might be missing: C, in being co-designed with UNIX, offers primitives tuned for the context of:
- writing UNIX CLI tools that work with unbounded streams of input (i.e. piped output from other UNIX CLI tools),
- where, crucially, the stream is just text, and so carries no TLV-esque framing protocol to tell you the definitive length of a thing;
- and nor (especially in early memory-constrained systems) are you able to perform allocations of heap memory in order to employ an unbounded growable buffer for retaining the current lexeme until you do reach the end of it (which, if you could, would let you use a scanner state-machine that doubles as a parser/validator, returning either a parsed value or an error)
- but instead, to deal with the 1. unbounded input, 2. of textual encoding, 3. in constant memory, you must eagerly scan the input stream (i.e. synchronously reduce over each received byte, or at most each fixed-length N-byte chunk using a static or stack-allocated fixed-length buffer, discarding the original string bytes once reduced-over) to produce lexically-decoded (but not parsed/validated) lexemes; and then do this again, on a higher level, feeding your stream of lexemes into a fixed-sized sum-typed ring-buffer (i.e. an array-of-union-typed-lexeme-struct-type-entries), where you can then invoke a function that attempts to scan over + consume them (but unlike the original stream-parsing function, doesn't consume the buffer unless successful, and so isn't functioning as a scanner per se, but rather as an LR parser.)
If you're not writing UNIX CLI tools, direct use of the C-stdlib numeric-lexeme scan functions is operating on the wrong abstraction layer. What you want, if you have pre-framed strings that are "either valid numbers or parse errors", is to implement an actual parsing function... that can then invoke these numeric-lexer functions to do the majority of its work.
And if you're writing C, and yet you're not in UNIX-pipeline unbounded-text-stream land, but rather are parsing well-defined bounded-length "documents" (like, say, C source files)... then you probably want to use a real lexer-generator (like flex) to feed a parser-generator (like yacc/bison). Where:
- you'd validate the token in context, in the parsing phaase;
- and your lexing rules would make certain classes of input invalid at lexing time. (E.g. you can write your lexeme matching rules such that multi-digit numbers with leading zeroes, or floating-point values with no digits before/after the decimal place, simply aren't "numbers" from your lexer's perspective.)
...which means that, once again, you can "get away with" invokeing the regular C numeric-lexeme scanner functions; i.e. `yylval = atoi(yytext);` in bison terms. (And you'd want to, since doing so saves memory vs. keeping the numbers around as strings.)
For integers, you're faster (in both development time and runtime) to write your own parser than to try and assemble the pieces in this pile of shit into a half-working one.
C++17 from_chars excluded. Incidentally, 2022 seems about right for the year that ONE open source implementation finally actually implemented the float part of that. Or was it more like 2024?
This wouldn't even pass a cursory sanity check of the api from a beginner developer, how did it end up in a standard library at all? Was it a mistake and then it was just too late to remove it?
Any function that can either succeed or fail, which is basically every parsing function, must typically indicate success or failure. You can terminate the program or you can return an object that itself indicates failure (such as -1 when finding a positive index) but if ALL values of the return type CAN be valid then the success state must be a separate return value.
What's the purpose of the function atol() if it doesn't have that? Is it "It's still useful for trusted input we know is a string representation of a long" (E.g. for bounded number roundtrip)? That seems awfully limited. But perhaps such a scenario was perhaps more common in 1960?