Tell HN: C Experts Panel – Ask us anything about C
We are members of the C Standard Committee and associated C experts, who have collaborated on a new book called Effective C, which was discussed recently here: https://news.ycombinator.com/item?id=22716068. After that thread, dang invited me to do an AMA and I invited my colleagues so we upgraded it to an AUA. Ask us about C programming, the C Standard or C standardization, undefined Behavior, and anything C-related!
The book is still forthcoming, but it's available for pre-order and early access from No Starch Press: https://nostarch.com/Effective_C.
Here's who we are:
rseacord - Robert C. Seacord is a Technical Director at NCC Group, and author of the new book by No Starch Press “Effective C: An Introduction to Professional C Programming” and C Standards Committee (WG14) Expert.
AaronBallman - Aaron Ballman is a compiler frontend engineer for GrammaTech, Inc. and works primarily on the static analysis tool, CodeSonar. He is also a frontend maintainer for Clang, a popular open source compiler for C, C++, and other languages. Aaron is an expert for the JTC1/SC22/WG14 C programming language and JTC1/SC22/WG21 C++ programming language standards committees and is a chapter author for Effective C.
msebor - Martin Sebor is Principal Engineer at Red Hat and expert for the JTC1/SC22/WG14 C programming language and JTC1/SC22/WG21 C++ programming language standards committees and the official Technical Reviewer for Effective C.
DougGwyn - Douglas Gwyn is Emeritus at US Army Research Laboratory and Member Emeritus for the JTC1/SC22/WG14 C programming language and a major contributor to Effective C.
pascal_cuoq - Pascal Cuoq is the Chief Scientist at TrustInSoft and co-inventor of the Frama-C technology. Pascal was a reviewer for Effective C and author of a foreword part.
NickDunn - Nick Dunn is a Principal Security Consultant at NCC Group, ethical hacker, software security tester, code reviewer, and major contributor to Effective C.
Fire away with your questions and comments about C!
987 comments
[ 2.8 ms ] story [ 432 ms ] threadYou're not the only person who misses the old newsgroups! The format that Hacker News uses is one that became sort of standard on the web in the early 2000s. It works differently than usenet did, but you get threaded comments in the sense that replies are nested under the posts they're replying to.
This might not be too deep a question on the C language in regards to this book, but I've been wondering, why did you decide to have an eldritch horror as the book's cover?
The C==Sea brings to mind the book Expert C Programming: Deep C Secrets.
The GoLang defer statement defers the execution of a function until the surrounding function returns. The deferred call's arguments are evaluated immediately, but the function call is not executed until the surrounding function returns. It looks like an interesting mechanism for cleaning up resources.
For me "defer" only makes sense in the context of exceptions, basically as an equivalent to "finally". This is a slippery slope though, since golang's exceptions are, for a reason, rudimentary.
Would the proposed defer statement apply to loops as well? How would one implement such defers without dynamic allocation?
There are pieces of software that should be given priority for a rewrite in Rust, but most of C software is never going to be rewritten, because there is simply too much of it.
Therefore, even if C did not have any advantage of its own over Rust, there would still be legacy software to maintain and to extend.
The advantages of C include that sometimes, an embedded processor with a proprietary instruction set is provided by the chipmaker with its own C compiler, which is the only compiler supporting the instruction set; that C is still currently used to write the runtimes of higher-level languages (I'm familiar with OCaml, but it isn't too much of a stretch to imagine that the runtimes of Python, Haskell,… are also written in C).
I sincerely hope this is not the general attitude of the standards committee. Some of us actually prefer C, and would like to see the language continue to flourish.
It goes deeper than that, in a couple of places Rust depends on the C standard: the fixed-layout `#[repr(C)]` structs (without that attribute, the compiler is free to reorder the struct fields; with that attribute, it's laid out the way C would do it), and the `extern "C"` function call ABI. The way to call any other language from Rust, or Rust from any other language, is to go through `extern "C"` functions passing `#[repr(C)]` structs. So even if the C language dies one day, parts of it will live in Rust forever (or as long as the Rust language lives).
My job at NCC Group involves a lot of code reviews, so frequently the files that are of interest to me are the ones that contain the most defects. I typically identify these by compiling with compiler warnings turned up and warning suppression turned down. I'll frequently also make use of static and dynamic analysis, including the GCC and Clang sanitizers.
One goal is to re-unify C with the concurrency object model used by C++ to make std::atomic<T> and _Atomic(T) be ABI compatible as intended in C11. Some small fixes in this area are the removal of ATOMIC_VAR_INIT, clarifying whether library functions can use thread_local storage for internal state, and things along those lines. However, we expect there to be more efforts in this area as we progress the standard.
In other words, will the C standard be effectively “done” at some time in the future?
Reviewing proposals to incorporate features supported by common implementations is another.
Aligning with other standards (e.g., floating point) and improving compatibility with others (C++) is yet another.
In general, when an ISO standard is done it essentially becomes dead. So for the C standard to continue to be active (on ISO's books) it needs to evolve.
I see the classic path of any programming language -- regardless of standardization -- is to continuously add features until it's too big and complex that nobody wants to deal with it any more. Then it's replaced by a newer, simpler language that takes the important bits and drops the unnecessary complexities. At that point, everybody sees that the older language was barking up the wrong tree, and they stop wasting time on it.
It's not the cessation of language change that causes language death -- that's merely a symptom. You can't keep a language alive simply by changing it every year. Some people sure have tried.
Alternatively, until it's evolved so much that there is so much diversity of implementation that simply knowing a library is written in "language X" doesn't tell me much about how it's written, or whether I can use it in my program which is also written in "language X".
Then again, C is the exception to every rule, so maybe we can keep piling on features indefinitely, and people will have to use it (even if they don't like it), for the same reason they started using it decades ago (even if we didn't like it).
http://www.open-std.org/jtc1/sc22/wg14/www/wg14_document_log...
These papers are usually quite interesting.
“Static analysis” may be the wrong name to classify the tools that work in that area, because “static analysis” is usually used for purely automatic tools, whereas the tools used to guarantee the absence of undefined behaviors are not entirely automatic except for the simplest of programs.
Results of a static analyzer are often characterized in terms of “false positives” and “false negatives”. It is a possible design choice to make an analyzer with no false negatives. It is absolutely not impossible! (Some people think it is fundamentally impossible because it sounds like a computer science theorem, but it isn't one. The theorem would apply if one intended to make an analyzer with no false positives and no false negatives—and if computers were Turing machines.)
Analyzers designed to have no false positives are called “sound”. In practice, this kind of analyzer may prove that a simple program is free of Undefined Behavior if the program is a simple example of 100 lines, but for a more realistic software component of at least a few thousand lines, the result will be obtained after a collaborative human-analyzer process (in which the analyzer catches reasoning errors made the human, so the result is still better than what you can get with code reviews alone).
Here is what the result of this collaborative human-analyzer process may look like for a library as cleanly designed and self-contained as Mbed TLS (formerly PolarSSL): https://trust-in-soft.com/polarSSL_demo.pdf?
EDIT: I work on embedded systems, where C is king, and it seems like a spend an inordinate amount of time working with code generators that build simple tables. All of which could go away with this feature.
I’ve always thought that idiomatic C for constexpr would be to write the code you want to be executed at compile-time in a separate file(s), build it, execute and then #include the result in your program before building the final executable, adding a build step but keeping overall complexity minimal.
This is different from C++ approach, where everything and the kitchen sink is added to the standard and then you have to issue errata for errata for the standard and hope that the compiler you have to use for your current platform is keeping up for the last changes.
2. Why it is harder to find lgpl licenced libraries to access windows directories over network like jcifs pysmb (and libraries overall) when needed to close most part of software source to sell small softwares to businesses?
3. If you needed to combo C with another language to do everything you need to do forever and never look back what other language would that be?
6.33 Sebor, Add strnlen to C2X [N 2351] Result: No consensus on putting N2351 into C2X.
6.34 Sebor, Add stpcpy, and stpncpy to C2X [N 2352] Result: No consensus to put N2352 into C2X.
6.35 Sebor, Add strdup and strndup to C2X [N 2353] Result: N2353 be put into C2X. The committee wants a proposal for the wide character versions of any POSIX functions voted in this meeting.
Err... I thought Annex K is deprecated and dead? Whereas strl* seem very much alive, some compilers even give a "strcpy/strncpy is unsafe, use strlcpy instead" warning.
> Microsoft Visual Studio implements an early version of the APIs. However, the implementation is incomplete and conforms neither to C11 nor to the original TR 24731-1.
> As a result of the numerous deviations from the specification the Microsoft implementation cannot be considered conforming or portable.
http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1967.htm
> The C Committee has taken two votes on this, and in each case, the committee has been equally divided. Without a consensus to change the standard, the status quo wins.
The fact that it has only survived on status quo is a pretty crass hint that things aren't well with Annex K.
IMO they're a lot more ergonomic than the Annex K functions, and do the thing most programmers think the strncat/strncpy functions do (admittedly, not part of ISO C).
Annex K should be forgotten as the mistake it is and we can move on with existing real-world interfaces instead of inventing features from whole-cloth. I thought that was generally the C standard operating practice.
- Languages like Rust will gain more mindshare over the next decade, and be used in more and more new projects, but there are billions of lines of existing code in C, and those aren't going away.
- Hardware architects, for better or worse, largely think about software in terms of [a somewhat dated and idealized mental model of] C. So if you want to be able to converse with architects (which anyone doing systems programming should want to do), you need to have some basic fluency with C.
You need to know enough C to interface with the OS, and enough C to talk about memory layout, memory management, dynamic libraries, ABI, etc.
Most higher language runtimes need C, even with a self hosting compiler. Not being able to work on the C parts is limiting.
You also need to know enough assembly to be able to understand what the compiler did with your own code, even if you never write assembly yourself. Not being able to compare the disassembly to the high level language to understand why it doesn't work (or is order of magnitude slower than expected) is limiting.
IIRC, this macro was added to C11 along with a batch of other "these are optional" macros for atomics, complex, threads, etc. However, I don't recall whether C99 adopted the features as optional features and missed the feature testing macro, or if they were required features in C99 that we made optional in C11.
Is a controverial feature, that can produce bugs, and are banned in a lot of project (one famouse, the Linux kernel).
[] Has there been any discussion of adding GCC's "typeof" to the standard?
Another thing very cumbersome is to do in C is object creation; creating instantiable objects is possible very cumbersome. Is there some feature in the thoughy process to deal with it. To make it clear, in C we can create a data structure like a Stack or a queue easily. But if the program needs 10 stacks then presently no simple way of achieving it.
To minimize the external identifiers, one could make just the name of a container structure the sole entry access handle, with structure members pointing to the functions. Then use it like:
On a slightly more personal note: What are some undefined behaviors that you would like to turn into defined behavior, but can't change for whatever reasons that be?
Personally, I would like to get rid of many of the trap representations (e.g., for integers) because there is no existing hardware in many cases that supports them and it gives implementers the idea that uninitialized reads are undefined behavior.
On the other hand, I just wrote a proposal to WG14 to make zero-byte reallocations undefined behavior that was unanimously accepted for C2x.
Unsigned types have their own issues, though: they overflow at "small" values like -1, which means that doing things like correctly looping "backwards" over an array with an unsigned index is non-trivial.
> On the other hand, I just wrote a proposal to WG14 to make zero-byte reallocations undefined behavior that was unanimously accepted for C2x.
You're saying that realloc(foo, 0) will no longer free the pointer?
It might seem like defining the semantics for signed overflow would be helpful but it turns out it's not, either from a security view or for efficiency. In general, defining the behavior in cases that commonly harbor bugs is not necessarily a good way to fix them.
I’m not trying to argue that signed overflow is the right tool for the job here for expressing ideas like “this loop will terminate”, but making signed overflow defined behavior will impact the performance of numerics libraries that are currently written in C.
From my personal experience, having numbers wrap around is not necessarily “better” than having the behavior undefined, and I’ve had to chase down all sorts of bugs with wraparound in the past. What I’d personally like is four different ways to use integers: wrap on overflow, undefined overflow, error on overflow, and saturating arithmetic. They all have their places and it’s unfortunate that it’s not really explicit which one you are using at a given site.
So it's not the best solution. If we want to make this behaviour for optimizations (that are to me not worthed, giving the risk of potentially critical bugs) we must make that behavior explicit, not implicit: thus is the programmer that has to say to the compiler, I guarantee you that this operation will never overflow, if it does it's my fault.
We can agree that having a number that wraps around is not a particularly good choice. But unless we convince Intel in some way that this is bad and make the CPU trap on an overflow, so we can catch that bug, this is the behaviour that we have because is the behaviour of the hardware.
The language is not a model of hardware, nor should it be. If you want to write to the hardware, the only option continues to be assembly.
This is exactly what every C programmer does, all the time.
Just to be sure I understand the fine details of this -- what would the impact be if the compiler assumed (correctly) that the loop might not terminate? What optimization would that prevent?
Loaded question—the compiler is absolutely correct here. There are two viewpoints where the compiler is correct. First, from the C standard perspective, the compiler implements the standard correctly. Second, if we have a real human look at this code and interpret the programmer’s “intent”, it is most reasonable to assume that overflow does not happen (or is not intentional).
The only case which fails is where N = INT_MAX. No other case invokes undefined behavior.
Here is an example you can compile for yourself to see the different optimizations which occur:
At -O2, GCC 9.2 (the compiler I happened to use for testing) will use pointer arithmetic, compiling it as something like the following: At -O3, GCC 9.2 will emit SSE instructions. You can see this yourself with Godbolt.Now, try replacing "int" with "unsigned". Neither of these optimizations happen any more. You get neither autovectorization nor pointer arithmetic. You get the original loop, compiled in the most dumb way possible.
I wouldn’t read into the exact example here too closely. It is true that you can often figure out a way to get the optimizations back and still use unsigned types. However, it is a bit easier if you work with signed types in the first place.
Speaking as someone who does some numerics work in C, there is something of a “black art” to getting good numerics performance. One easy trick is to switch to Fortran. No joke! Fortran is actually really good at this stuff. If you are going to stick with C, you want to figure out how to communicate to the compiler some facts about your program that are obvious to you, but not obvious to the compiler. This requires a combination of understanding the compiler builtins (like __builtin_assume_aligned, or __builtin_unreachable), knowledge of aliasing (like use of the "restrict" keyword), and knowledge of undefined behavior.
If you need good performance out of some tight inner loop, the easiest way to get there is to communicate to the compiler the “obvious” facts about the state of your program and check to see if the compiler did the right thing. If the compiler did the right thing, then you’re done, and you don’t need to use vector intrinsics, rewrite your code in a less readable way, or switch to assembly.
(Sometimes the compiler can’t do the right thing, so go ahead and use intrinsics or write assembly. But the compiler is pretty good and you can get it to do the right thing most of the time.)
You're correct about how it behaves with "int" and "unsigned", very interesting. But it occurs to me that on x64 we'd probably want to use 64-bit values. If I change your typedef to either "long" or "unsigned long" that seems to give me the SSE version of the code! (in x86-64 gcc 9.3) Why should longs behave so differently from ints?
I very much agree that getting good numerics performance out of the optimizer seems to be a black art. But does the design of C really help here, or are there ways it could help more? Does changing types from signed to unsigned, or int to long, really convey your intentions as clearly as possible?
I remain skeptical that undefined behaviour is a good "hook" for compilers to use to judge programmer intention, in order to balance the risks and rewards of optimizations. (Admittedly I'm not in HPC where this stuff is presumably of utmost importance!) It all seems dangerously fragile.
If you need good performance out of some tight inner loop, the easiest way to get there is to communicate to the compiler the “obvious” facts about the state of your program and check to see if the compiler did the right thing. If the compiler did the right thing, then you’re done, and you don’t need to use vector intrinsics, rewrite your code in a less readable way, or switch to assembly.
I strongly agree with the first part of this -- communicating your intent to the compiler is key.
It's the second part that seems really risky. Just because your compiler did the right thing this time doesn't mean it will continue to do so in future, or on a different architecture, and of course who knows what a different compiler might do? And if you end up with the "wrong thing", that may not just mean slow code, but incorrect code.
I’m sure there are ways that it could help more. But you have to find an improvement that is also feasible as an incremental change to the language. Given the colossal inertia of the C standard, and the zillions of lines of existing C code that must continue to run, what can you do?
What I don’t want to see are tiny, incremental changes that make one small corner of your code base slightly safer. Most people don’t want to see performance regressions across their code base. That doesn’t leave a lot of room for innovation.
> It all seems dangerously fragile.
If performance is critical you run benchmarks on CI to detect regression.
> It's the second part that seems really risky.
It is safer than the alternatives, unless you write it in a different language. The “fast” code here is idiomatic, simple C the way you would write it in CS101, with maybe a couple builtins added. The alternative is intrinsics, which poses additional difficulty. Intrinsics are less portable and less safe. Less safe because their semantics are often unusual or surprising, and also less safe because code written with intrinsics is hard to read and understand (so if it has errors, they are hard to find). If you are not using intrinsics or the autovectorizer, then sorry, you are not getting vector C code today.
This is also not, strictly speaking, just an HPC concern. Ordinary phones, laptops, and workstations have processors with SIMD for good reason—because they make an impact on the real-life usability of ordinary people doing ordinary tasks on their devices.
So if we can get SIMD code by writing simple, idiomatic, and “obviously correct” C code, then let’s take advantage of that.
Some additional value would be gained by allowing stores to automatic objects whose address isn't taken to maintain such extra range at their convenience, without any requirement to avoid having such extra range randomly appear and disappear. Provided that a program coerces values into range in when necessary, such semantics would often be sufficient to meet application requirements without having to prevent overflow.
What additional benefits are achieved by granting compilers unlimited freedom beyond that? I don't see any such benefits that would be worth anything near the extra cost imposed on programmers.
I guess that doesn’t necessarily help in the “+=2” case, where you probably want the optimizer to do a “result += x/2”.
In general, I’d greatly prefer to work with a compiler that detected the potential infinite loop and flagged it as an error.
https://stackoverflow.com/a/16436479/530160
Quoting http://blog.llvm.org/2011/05/what-every-c-programmer-should-...
> This behavior enables certain classes of optimizations that are important for some code. For example, knowing that INT_MAX+1 is undefined allows optimizing "X+1 > X" to "true". Knowing the multiplication "cannot" overflow (because doing so would be undefined) allows optimizing "X*2/2" to "X". While these may seem trivial, these sorts of things are commonly exposed by inlining and macro expansion. A more important optimization that this allows is for "<=" loops like this:
> for (i = 0; i <= N; ++i) { ... }
> In this loop, the compiler can assume that the loop will iterate exactly N+1 times if "i" is undefined on overflow, which allows a broad range of loop optimizations to kick in. On the other hand, if the variable is defined to wrap around on overflow, then the compiler must assume that the loop is possibly infinite (which happens if N is INT_MAX) - which then disables these important loop optimizations. This particularly affects 64-bit platforms since so much code uses "int" as induction variables.
for (int i = 0; i < 64; i++) result[i] = inputA[i] * inputB[i];
If inputA[i] * inputB[i] overflowed, why are my credit card details at risk? The question is: can we come up with an alternate behaviour that incorporates both advantages of the i<=N optimization, as well as leave my credit card details safe if the multiplication in the inner loop overflowed? Is there a middle road?
I'd rather have a special type with defined behavior. That's actually what a lot of shops do anyways, and there are some niche compilers that support types with defined overflow (ADI's fractional types on their Blackfin tool chain, for example). It's just annoying to do in C, this is one of those cases where operator overloading in C++ is really beneficial.
Right, but I think the problem is that UB means literally anything can happen and be conformant to the spec. If you do an integer overflow, and as a result the program formats your hard drive, then it is acting within the C spec.
Now compiler writers don't usually format your hard drive when you trigger UB, but they often do things like remove input sanitation or other sorts of safety checks. It's one thing if as a result of overflow, the number in your variable isn't what you thought it was going to be. It's completely different if suddenly safety checks get tossed out the window.
When you handle unsanitized input in C on a security boundary, you must literally treat the compiler as a "lawful evil" accomplice to the attackers: you must assume that the compiler will follow the spec to the letter, but will look for any excuse to open up a gaping security hole. It's incredibly stressful if you know that fact, and incredibly dangerous if you don't.
I'd say more chaotic evil, since the Standard has many goofy and unworkable corner cases, and no compiler tries to handle them all except, sometimes, by needlessly curtailing optimizations. Consider, for example:
The way the Standard defines "based upon", if a and b are both equal to x, then p would be based upon a (since replacing a with a pointer to a copy of x would change the value of p). Some compilers that ignore "restrict" might generate code that accommodates the possibility that a and b might both equal x, but I doubt there are any that would generally try to optimize based on the restrict qualifier, but would hold off in this case.Compiler writers tend to assume the processor is a dumb machine. But modern ones aren't, they do a lot of resource allocation and optimization on the fly. And they do it in hardware in real time.
A lot of C developers tend to assume the compiler is a dumb program ;) There are significant hoisting and vectorization optimizations that signed overflow can unlock, but they can't always be applied.
It's easier than it sounds. One of the major problems you usually run into when learning to juggle is that you throw the balls too far forward (their arc should be basically parallel to your shoulders, but it's easy to accidentally give them some forward momentum too), which pulls you forward to catch them. Being allowed to walk means that's OK.
(For the curious, there are three major problems you're likely to have when first learning to juggle:
1. I can throw the balls, but instead of catching them, I let them fall on the ground.
2. My balls keep colliding with one another in midair.
3. I keep throwing the balls too far forward.)
There's actually a niche hobby called "joggling" which, as the name implies, involves juggling while jogging.
A couple of easy optimizations, for example, that would be available to a smart compiler processing straightforwardly-written code to use automatic overflow checking, but not to one fed code that uses intrinsics:
1. If code computes x=yz, but then never uses the value of x, a compiler that notices that x is unused could infer that the computation could never be observed to produce an arithmetically-incorrect result, and thus there would be no need to check for overflow.
2. If code computes xy/z, and a compiler knows that y=z*2, the compiler could simplify the calculation to x+x, and would thus merely have to check for overflow in that addition. If code used intrinsics, the compiler would have to overflow check the multiplication, which on most platforms would be more expensive. If an implementation uses wrapping semantics, the cost would be even worse, since an implementation would have to perform an actual division to ensure "correct" behavior in the overflow case.
Having a language offer options for the aforementioned style of loose overflow checking would open up many avenues of optimization which would be unavailable in language that only over precise overflow checking or no overflow checking whatsoever.
The worst thing is that people take it as acceptable that this loop is going to operate differently upon overflow (e.g. assume N is TYPE_MAX) depending on whether i or N are signed vs. unsigned.
I've gone a lifetime programming, and this kind of stuff never, ever matters one iota.
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=30475 shows someone going hyperbolic over the issue. The technical arguments favor the GCC maintainers. However I prefer the position of the person going hyperbolic.
GCC has the behavior that overflowing a signed integer gives you a negative one. But an if tests that TESTS for that is optimized away!
The reason is that overflow is undefined behavior, and therefore they are within their rights to do anything that they want. So they actually overflow in the fastest way possible, and optimize code on the assumption that overflow can't happen.
The fact that almost no programmers have a mental model of the language that reconciles these two facts is an excellent reason to say that very few programmers should write in C. Because the compiler really is out to get you.
The fact is that if the compiler encounters undefined behaviour, it can do basically whatever it wants and it will still be standard-compliant.
Sometimes you're writing code where it really, really matters and you're more than willing to spend the extra cycles for every add/mul/etc. Having these new types as a portable idiom would help.
N2466 2020/02/09 Svoboda, Towards Integer Safety
http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2466.pdf
The committee asked the proposers for further work on this effort.
Integer types that saturate are an interesting idea. Because signed integer overflow is undefined behavior, implementations are not prohibited from implementing saturation or trapping on overflow.
The entire notion that "since this is undefined behavior it does not exist" is the biggest fallacy in modern compilers.
You're exactly right that this is why there is a distinction between conforming and strictly conforming code.
Implementations define undefined behavior all the time and users rely on it. For instance, POSIX defines that you can convert an object pointer into a function pointer (for dlsym to work), or implementations often rely on offsets from a null pointer for their 'offsetof' macro implementation.
If people used them while parsing binary inputs that would prevent a lot of security bugs.
The fact that this question exists and is full of wrong answers suggests a language solution is needed: https://stackoverflow.com/questions/1815367/catch-and-comput...
http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2466.pdf
(signal is a strong word... maybe indicate?)
I know there's recently been some movement towards standardizing something in this direction, but I don't know what the status of that work is. Probably one of the folks doing the AUA can update.
The generic heterogeneous operations also avoid the identifier blowup. The only real argument against them that I see is that they are not easily implementable in C itself, but that's nothing new for the standard library (and should be a non-goal, in my not-a-committee-member opinion).
Obviously, I'm not privy to the committee discussions around this, so there may be good reasons for the choice, but it worries me a lot to see that document.
Hi Stephen, thank you for bringing this to our attention. David Svoboda and I are now working to revise the proposal to add a supplemental proposal to support operations on heterogeneous types. We are leaning toward proposing a three-argument syntax, where the 3rd argument specifies the return type, like:
where a and b are integer values and T is an integer type, in addition to the two-argument form (Or maybe the two-argument and three-argument forms should have different names, to make it easier to implement.)>once you have the heterogeneous operations, is there any reason to keep the others around
The two-argument form is shorter, but perhaps that isn't a strong enough reason to keep it. Also, requiring a redundant 3rd argument can provide an opportunity for mistakes to happen if it gets out of sync with the type of first two arguments.
As for the non-generic functions (e.g., ckd_int_add, ckd_ulong_add, etc.), we are considering removing them in favor of having only the generic function-like macros.
(Note that while `-ftrapv` would seem equivalent, I've found it to be less reliable, particularly with compile-time checking.)
I presume you'd want signed overflow to have the usual 2's-complement wraparound behavior.
One problem with that is that a compiler (probably) couldn't warn about overflows that are actually errors.
For example:
With integer overflow having undefined behavior, if the compiler can determine that the value of n is INT_MAX it can warn about the overflow. If it were defined to yield INT_MIN, then the compiler would have to assume that the wraparound was what the programmer intended.A compiler could have an option to warn about detected overflow/wraparound even if it's well defined. But really, how often do you want wraparound for signed types? In the code above, is there any sense in which INT_MIN is the "right" answer for any typical problem domain?
There is no answer different that INT_MIN that would be right and make sense, i.e. the natural properties of the + operator (associativity, commutativity) are respected. Thus, by want of another possibility, INT_MIN is precisely the right answer to your code.
I read your code and it seems to me very clear that INT_MIN is exactly what the programmer intended.
Well, I'm the author and that's not what I intended.
I used INT_MAX as the initial value because it was a simple example. Imagine a case where the value happens to be equal to INT_MAX, and then you add 1 to it.
The fact that no result other than INT_MIN makes sense doesn't imply that INT_MIN does make sense. Saturation (having INT_MAX + 1 yield INT_MAX) or reporting an error seem equally sensible. We don't know which behavior is "correct" without knowing anything about the problem domain and what the program is supposed to do.
A likely scenario is that the programmer didn't intend the computation to overflow at all, but the program encountered input that the programmer hadn't anticipated.
INT_MAX + 1 commonly yields INT_MIN because typical hardware happens to work that way. It's not particularly meaningful in mathematical terms.
As for "natural properties", it violates "n + 1 > n". C integers are not, and cannot be, mathematical integers (unless you can restrict values to the range they support).
Your code assumes that negating a negative value is positive. Your division check forgot about INT_MIN / -1. Your signed integer average is wrong. You confused bitshift with division. etc. etc. etc.
Unsigned arithmetic is tractable and should be treated with caution. Signed arithmetic is terrifying and should be treated with the same PPE as raw pointers or `volatile`.
This applies if arithmetic maps to CPU instructions, but not to Python or Haskell or etc. If you have automatic bignums, signed arithmetic is of course better.
Joining the committee requires you to be a member of your country's national body group (in the US, that's INCITS) and attend at least some percentage of the official committee meetings, and that's about it. So membership is not difficult, but it can be expensive. Many committee members are sponsored by their employers for this reason, but there's no requirement that you represent a company.
I joined the committees because I have a personal desire to reduce the amount of time it takes developers to find the bugs in their code, and one great way to reduce that is to design features to make harder to write the bugs in the first place, or to turn unbounded undefined behavior into something more manageable. Others join because they have specific features they want to see adopted or want to lend their domain expertise in some area to the committee.
This helps bringing these languages to embedded targets with closed toolchains (with an existing C compiler).
Will there be developments to use a subset of C as a “portable assembly” in a standard way? Like there is WebAssembly for JavaScript.
Have you ever considered or will you consider deprecating char, int, long, (s)size_t, float, double and etc in favour of specific length types?
Will you ever add / have you considered adding [su]\d+ and f\d+ as synonyms for those mentioned stdint.h?
Since char is signed on most platforms, arm eabi being an exception and even there it's really just a matter of compile time flags, will you ever just drop char from being able to be either and just say it's signed, as int is also signed?
Will you ever define / have you considered defining signed overflow behaviour?
C does provide fixed width types like uint8_t, uint16_t, uint32_t, and uint64_t. These are optional types because they can't be implemented on implementations that don't have the appropriate word sizes. We also have required types such as
uint_least16_t uint_least32_t uint_least64_t uint_least8_t
If not deprecate, then at least make fixed width types as equivalent members to them, ie all char based apis should accept s8 (typedef signed char s8) and all int based apis should accept s32.
I would go beyond that, requiring all sizes that are a multiple of 8 bits from 8-bit through 512-bit. This better supports cryptographic keys and vector registers.
Why?
I was on an OS development team in the 1990s. We were using the SHARC DSP, which was naturally a word-addressed chip. Endianness didn't exist in hardware, since everything was whatever size (32, 40, or 48 bits) you had on the other end of the bus. Adding 1 to a hardware pointer would move by 1 bus width. The chip vendor thought that CHAR_BIT could be 32 and sizeof(long) could be 1.
We couldn't ship it that way. Customers wanted to run real-world source code and they wanted to employ normal software developers. We hacked up the compiler to rotate data addresses by 2 bits so that we could make CHAR_BIT equal to 8.
That was the 1990s, with an audience of embedded RTOS developers who were willing to put up with almost anything for performance. People are even less forgiving today. If strangely sized char couldn't be a viable product back in the 1990s, it has no chance today. It's dead. CHAR_BIT is 8 and will forever be so.
Unsigned arithmetic never overflows, and guarantees two's-complement behavior, because unsigned arithmetic is always carried out modulo 2^n:
> A computation involving unsigned operands can never overflow, because a result that cannot be represented by the resulting unsigned integer type is reduced modulo the number that is one greater than the largest value that can be represented by the resulting type. (6.2.5, Types)
Doing the computation in unsigned always does the "right thing"; the thing that one needs to be careful of with this approach is the conversion of the final result back to the desired signed type (which is very easy to get subtly wrong).
> Otherwise, if the new type is unsigned, the value is converted by repeatedly adding or subtracting one more than the maximum value that can be represented in the new type until the value is in the range of the new type (6.3.1.3 Signed and unsigned integers)
Unsigned to signed is the hard direction. If the result would be positive (i.e. in range for the signed type), then it just works, but if it would be negative, the result is implementation-defined (but note: not undefined). You can further work around this with various constructs that are ugly and verbose, but fully defined and compilers are able to optimize away. For example, `x <= INT_MAX ? (int)x : (int)(x + INT_MIN) + INT_MIN` works if int has a twos-complement representation (finally guaranteed in C2x, and already guaranteed well before then for the intN_t types), and is optimized away entirely by most compilers.
For unsigned operations the carry flag is used, and for signed operations, the overflow flag is used.
One is that unsigned arithmetic can overflow, and the behavior on overflow is defined to wrap around.
Another is to say that unsigned arithmetic cannot overflow because the result wraps around.
Both correctly describe the way it works; they just use the word "overflow" in different ways.
The C standard chooses the second way of describing it.
I think most people colloquially call going A + 1 = B where B < A an overflow. Interesting. I knew they're different things, but never really thought about my word choice.
Basically, we prefer seeing features that real users have used as opposed to an experimental branch of a compiler that doesn't have usage experience. Knowing it can be implemented is one thing, but knowing users want to use it is more compelling.
Have you considered adding access to structure members by index or by string name? Have you considered dynamic structures?
I'm not certain about the historical answer to this, but I do know that we're currently considering a proposal to introduce an exact bit-width integer type '_ExtInt(N)' to the language, and how to handle format specifiers for it is part of those discussions, so we are considering some changes in this area.
> Have you considered adding access to structure members by index or by string name? Have you considered dynamic structures?
I don't recall seeing any such proposals. I'm not familiar with the term "dynamic structures", what do you have in mind there?
Please, please, please pick short and descriptive format specifiers, like %[suf]\d+, ie
_ExtInt(N) and PRIx64 etc look absolutely horrid. u?int\d+_t are also really bad, it would be great to have just [suf]\d+ as types, where \d+ is 8, 16, 32, 64 for [us] and 32 and 64 for f.>what do you have in mind there?
Say like VLAs but structures with members that are dynamically defined and used.
That's my personal preference as well. Using the PRI macros always makes me feel sad.
> Say like VLAs but structures with members that are dynamically defined and used.
Ah, no, I don't recall any proposals along those lines. It's an interesting idea, and I'd be curious what the runtime performance characteristics would be vs what kind of new coding patterns would emerge that you couldn't do previously though!
Stucture member access by name is useful. It's slow, but it doesn't affect code that isn't using the feature. The worst runtime issue is that the runtime support requirement grows. For example, libgcc would gain a few functions.
We can do it today with awkward code, sometimes involving hacks that are outside the C language. Implementations vary by how much they hide what is going on. When I implemented libproc.so for Linux, I made two implementations. The high-performance one used a perfect hash table that was generated by gperf and then hand-edited. Name look-up would do the hash, index into an array of structs, compare the name for a match, and then use gcc's computed goto extension to jump to code that would handle the struct member. Had I not been also parsing the data in various distinct ways, I might have used an offsetof() macro to let generic code fill in the struct fields. The other implementation I made, with lower performance, used bsearch on a sorted array.
Dynamically defined struct members are also useful, but even slower and with even more overhead. Again though, I don't think that other code would be affected beyond the growth of the compiler's libgcc equivalent.
Seeing what I just wrote above, the computed goto extension is more important. It's great for any kind of table look-up that needs code to run. Emulators use it a lot, and would use it much more if it were in the C standard.
Mbed TLS, since I have it in mind from another thread, is also a pretty clean C library for the problem it tries to solve; it's a testament to its design that we (TrustInSoft, who had not participated to its development) were able to verify that some uses of the library were free of Undefined Behavior: https://tls.mbed.org
Opened a random part of musl out of sheer boredom. Here's what I see:
https://git.musl-libc.org/cgit/musl/tree/include/aio.h
A bunch of return codes #defined like so (see https://git.musl-libc.org/cgit/musl/tree/src/aio/aio.c):
#define AIO_CANCELED 0 #define AIO_NOTCANCELED 1 #define AIO_ALLDONE 2
#define LIO_READ 0 #define LIO_WRITE 1 #define LIO_NOP 2
#define LIO_WAIT 0 #define LIO_NOWAIT 1
Why weren't they using an enum instead? I wouldn't sign off on this code (and I don't think it lives up to best practices).
Additionally the used UNICODE_MAJOR and _MINOR are needed. They are always years behind, and you never know which tables versions are implemented.