64 comments

[ 3.7 ms ] story [ 203 ms ] thread
This makes C look like it's full of land mines. The minesweeper language.
The title is making it seem as the listed behavior is largely unknown to C programmers. It isn't. And it is in the manual.
Yeah, hopefully if you are programming in C you'll know using an undefined variable is bad.
Yep. :-)

The article doesn't make much sense at times: Dereferencing a NULL Pointer: >>>contrary to popular belief<<<, dereferencing a null pointer in C is undefined.

I guess C programmers are not counted in the "popular" group.

I don't think it's wrong at all. I suspect if you asked a large number of even moderately competent C programmers, you'd find quite a lot of them believe it's defined to trap. That's exactly what the rest of it is talking about. Never mind the issue raised of how it potentially affects optimization, which I think very few C programmers would actually be able to speak to.
Are you really sure that most C programmers know about this?

IMO, if you'd ask the question "what happens if you dereference a NULL pointer", most C programmers would answer "a segfault".

If you are in a company that codes largely in C, do a pool and report back. I'm honestly curious.
Unfortunately, I don't work in such a company.

A completely unrepresentative poll among two C++ programmers showed that 50% thought it was a segfault, whereas the other 50% suggested it could be undefined behavior, deducing that from the knowledge that the behavior is different in Linux userspace and kernel.

Although I wouldn't call those two typical C++ programmers. They do security CTFs for fun, so they might have a broader knowledge about exploitable behaviors in C/C++ code.

That, of course, would be a leading question. Most people would probably tend to assume that you're asking "in the environment in which we're programming, what happens if you dereference NULL?" which is not what you meant.

I didn't poll my coworkers either, but at least I think the answers would be different if I asked "what does C say should happen when a NULL pointer is dereferenced?".

Perhaps that's just wishful thinking on my behalf, though. :)

I don't think that's particularly wishful thinking, however, your alternative phrasing could be seen as equally, if not more, leading.

When quizzed on the behaviour of a C program I would expect a careful and experienced C programmer to consider the behaviour described by the standard (which standard?); unspecified and implementation defined behaviours; and deviations from the standard in both the compiler and the environment. Often I'd expect the answer to be "it depends".

I think, once you unanimously conclude that something should not be done, that people don't really care _why_ it shouldn't be done until someone tries to suggest you should do it.

For example, we know not to dereference null pointers. It is pretty much never correct to do so, and it's hard to imagine otherwise. What does it matter what C says should happen when you do it? You're never going to intentionally do it, so it becomes an incongruous hypothetical. Like "what would happen if you were never born?" The answer is useless because the question is inherently flawed.

Clang, for example, will optimize out some NULL dereferences. So even in real environments that people use, "it segfaults" is wrong.

Edit: some NULL dereferences.

In that case the machine code generated doesn't dereference NULL, so the answer "it segfaults" isn't wrong.
In the C code, a null pointer is dereferenced. A segfault is not happening in the executable. Arguing about intermediate steps is kinda moot IMO.
The very concept of dereferencing doesn't exist at that level.
I'll take the downvote as a request for elaboration. In C, you dereference a pointer. This typically translates to machine code that loads memory at an address. The translation is so direct that we tend to think of them as identical, i.e. dereferencing a pointer is loading memory at an address. But this is not so; first of all, nothing says pointers have to be implemented as memory addresses in the first place, and even if you do implement them that way (as virtually every C implementation does), not every pointer dereference has to generate a load instruction. That's not just the case for optimizing out NULL dereferences. Optimizations mean that in general, an expression p can have the memory load optimized away if the value at that location is already known from a previous computation (say, because you just assigned a value there, or you read the value a bit earlier and it's still available in a register).

In C, you deference a pointer. In assembly or machine code, you load memory at an address. The two don't have to happen at the same time, though, and just because you don't end up loading memory at an address doesn't mean you didn't dereference a pointer.

In short:

    *(char *)NULL
This always dereferences NULL, even if the compiler optimizes out the statement entirely.
Good point. Some embedded environments will have different behavior. I don't recall what happened under MS DOS or if MS compiler provided an debug option.
For a while you were actually able to mmap the memory at address NULL under linux, which is usable for exploiting (you can pass arbitrary data to kernel code that you can bring to dereference null pointers)
Wouldn't virtual address translation prevent this from happening?
Well I mean it kind of is. C's my favorite language but it's like a machete, simple enough, fairly easy to use and useful in a lot of places but if you don't know how to use it properly it's very easy to cut yourself with.
In a certain way, it is. But for a lot of things C and (C++) is still necessary, and the fact that C is not safe is a requirement. Can you imagine the Linux kernel not being written in C? Just imagine all the overhead if the Linux kernel has to perform overflow checking after every internal integer increment.

Maybe some day Rust will be a good candidate, but that day hasn't come yet.

> Can you imagine the Linux kernel not being written in C?

No, only because the Linux kernel started as an UNIX clone and C is the UNIX's systems programming language.

Other OS were written in another systems programming languages. Before and after C got invented.

C only became widespread because a few successful startups in the 80's adopted UNIX for their workstation OS.

In the OS-programming world C/C++ is staple.
I am old enough to remember when it wasn't even an option.
I'd say that in terms of seats sold, Borland and Microsoft had more to do with it. Maybe first one then the other. But 'C' was the semi-official language of the old DOS PC world from 1985 until Linux happened. It's somewhat tragic, because Modula-2 and Ada toolchains existed, but were harder to use than 'C' for performance and memory constraint reasons ( not to mention the toolchain costs )

Borland started out with Pascal, but went to 'C'.

>Borland started out with Pascal, but went to 'C'.

As someone that used all their products before they became Imprise or how it was called, I never saw this happening.

Even nowadays C++ Builder actually uses the Delphi framework.

EDIT: Just wanted to add that in my part of the planet, it was all about Assembly for systems programming, games and real applications. Turbo Pascal for general purpose applications without much performace issues and CLIPPER for the typical CRUD applications.

It's not the 80s anymore, there area many cases where there are much better alternatives to C.

But it's still a language you need for certain things, like responsiveness or real time applications. Like linus torvalds said it, C is like a more comprehensive assembly generator than anything else.

Having such a language allows you to change every little boring details to do exactly what you want. It has its use, and it's surely not intended to be mainstream as of today. But there are many cases where C comes back as a constant common denominator.

It's sure that it's kinda shitty that there's nothing really better than a language designed in the 70s, but I guess you could either blame language designers for not making anything better. There also are business/political issues and preferences for certain things that comes into play.

All in all, C just seems to be pretty good when you want to have code that works well with systems. Maybe there's just inertia as to where computer development is headed, which might be heavy related to C.

All I wish, is someday to be able to learn some more "virtuous" language like haskell, scheme, or smalltalk.

The 80's already had Mesa and Modula-2, but then those guys at Berkley had to create a few successful startups.
You may want to look up the age of those "virtuous" languages you listed. While it's fun to bash C, you could drive a damn yacht through the undefined behavior in Scheme. High-level languages are not immune here.
I recommend reading the articles linked elsewhere in this thread:

    https://news.ycombinator.com/item?id=8879948
A lot of this is C compiler writers using the spec to perform counterintuitive "optimizations" (with dubious performance value), and forgetting that they're writing software for C programmers to use, not trying to win a pedantry war with them.
Some of the "performance penalties" look like FUD to me. At least to the extent implied in the article.

For example "use of an unitialized variable". Given that the compiler already has to analyze the program flow to determine if the variable could be use unitialized to output that warning, the optimization step to not initialize those variables that are written to before used doesn't seem so far away. This happens at compile time.

But, there's also the fact that a modern OS gives you heap memory zeroed mostly for free. A call to calloc is highly optimized nowadays. It needs to be for security reasons, as a program must never receive memory with data left from a different process.

Likewise the signed integer overflow check. Modern CPUs are so superscalar internally that an overflow check (which doesn't get flagged most of the time anyway) doesn't add anything to the execution speed.

And finally we are able to use that without resorting to assembler. GCC 5 has added support for built-in functions for arithmetics with overflow checking.

> But, there's also the fact that a modern OS gives you heap memory zeroed mostly for free. A call to calloc is highly optimized nowadays. It needs to be for security reasons, as a program must never receive memory with data left from a different process.

Memory often is coming from previously-freed heap space within the process. There is no security need to zero such memory. Zeroing it wouldn't even help, because the process is free to read the memory before malloc returns it.

And calloc, no matter how optimized it is, will still be more expensive than malloc which doesn't have to zero the memory.

> Modern CPUs are so superscalar internally that an overflow check (which doesn't get flagged most of the time anyway) doesn't add anything to the execution speed.

"Doesn't add anything" is not correct. These checks are not free:

http://danluu.com/integer-overflow/

https://news.ycombinator.com/item?id=8765714#up_8765979

Zeroed memory should not be particularly expensive on modern machines, because we have a lot of parallelism available to zero it off working threads. There might be some constraint on memory bandwidth, but IMO memory latency is a bigger issue, usually preventing maximizing memory throughput, so there should be some spare.

The danluu link doesn't really support your argument. Yes, the checks aren't free, but they aren't very costly either in practice, and much of the cost is down to C compilers not being optimized for generating range-checking code.

I feel you're also possibly missing a longer term feedback loop issue. When software starts relying on things like overflow checks and zeroed memory, hardware manufacturers like Intel start building and optimizing hardware to make them faster.

The "prefer performance over correctness" bias that's existed in the C world has been responsible for a lot of damage. We need to get out of it.

Zeroing memory is certainly not free, and if you're doing some types of real-time applications, can have a significant cost.

http://randomascii.wordpress.com/2014/12/10/hidden-costs-of-...

If the compiler could prove you never read an array element before writing (like it is most of the time in Java, and should be all of the time in C), it could optimize the entire zeroing step away. Java does this very often. And initializing your variables close to the point of declaring them is a good practice in C anyways, which should make it easy for the compiler to analyze.
But if zeroing memory is necessary for the program to be correct, hang the cost. I don't know what sort of "real time applications" would require gambling on the initial state of memory.

If you really can't afford to use dynamic memory, don't. 'C' will cheerfully allow you to hog as much as you want outside the heap.

> But if zeroing memory is necessary for the program to be correct, hang the cost.

Sure. And your application should bear that cost and not necessarily every application written in C. Maybe there is a case for using a safer malloc implementation for your program?

> I don't know what sort of "real time applications" would require gambling on the initial state of memory.

This isn't really gambling. If you know you are going to initialize the memory with sane values immediately (and this is a common practice), it is a waste to do that initial zeroing.

Ah - in that case, I'd agree. I couldn't draw to that conclusion from the text provided - that's just as good as zeroing ( and, as you say ) renders the zeroing moot.
> Zeroed memory should not be particularly expensive on modern machines, because we have a lot of parallelism available to zero it off working threads.

It must be nice living a world where energy is free, you have unlimited cores and unbounded caches, and context switches, thread synchronisation and cache coherence have no cost or latency.

I'm not saying that reducing the number of cases with undefined behaviour isn't worth it in most cases, but claiming that you can just zero application memory at virtually no cost on a multicore is misguided.

> Zeroed memory should not be particularly expensive on modern machines, because we have a lot of parallelism available to zero it off working threads.

If your plan is to use other threads to zero the memory, then your malloc() implementation needs to move every freed region into a thread-safe queue to be zero'd and wait for the other thread to pick it up before the memory is eligible to be allocated again. This adds synchronization costs, creates opportunities for false sharing, and introduces starvation issues.

Also, there are applications that genuinely peg all CPUs already. In these cases, this approach makes even less sense because there is no spare CPU sitting around, so you pay the overhead for no actual parallelism.

> When software starts relying on things like overflow checks and zeroed memory, hardware manufacturers like Intel start building and optimizing hardware to make them faster.

I'm pretty sure Intel already has plenty of incentive to make memset(ptr, 0, n) fast. Also I'm not sure how overflow checks could be made much faster, except maybe to create an entirely separate set of integer instructions that raise hardware exceptions on overflow. Maybe that would be worthwhile, I don't know.

> The "prefer performance over correctness" bias that's existed in the C world has been responsible for a lot of damage. We need to get out of it.

The "correctness checks are free because we want them to be" bias that exists in other languages has been responsible for a lot of languages and runtimes that are "nice" but that no one could actually write a competitive web browser (for example) in.

Maybe no-one really tried hard enough? The correctness checks are not a problem for server-side software, so I guess they are not the primary reason for not having a web-browser written in those nice languages, but something else (like historical reasons).
>> Modern CPUs are so superscalar internally that an overflow check (which doesn't get flagged most of the time anyway) doesn't add anything to the execution speed.

> "Doesn't add anything" is not correct. These checks are not free:

This statement was in regard to what happens inside the CPU. What happens on the level of the individual compilers is still a problem as we only see about now that the compilers are starting to optimize those cases.

In that regard we don't know yet the true costs of those checks but time will tell.

> This statement was in regard to what happens inside the CPU.

In that regard, the statement is still incorrect. The link I posted above analyzes the cost at the CPU level given the optimal machine code. It estimates an upper bound of a 2x overhead per overflow-checked add. ie. not free.

>the optimization step to not initialize those variables that are written to before used doesn't seem so far away

Determining whether or not variables will be used uninitialized is equivalent to solving the halting problem.

In general case only. For most of typical code, it is quite obvious.

If your program makes initialization depend on many conditions which makes it hard to analyze for the compiler, you're probably writing very fragile code that will soon bite you with UB.

> For most of typical code, it is quite obvious.

[citation needed]

> If your program makes initialization depend on many conditions which makes it hard to analyze for the compiler, you're probably writing very fragile code that will soon bite you with UB.

Say you allocate a stack, and access it with the following pattern:

    bool push(stack *st) {
      if (st->top == st->limit) return false;
      st->top++;
      st->top->foo = FOO_INIT;
      st->top->bar = BAR_INIT;
      return true;
    }

    void pop(stack *st) {
      assert(st->top != st->stack);
      st->top--;
    }

    void stack_alloc(stack *st, int n) {
      st->stack = malloc(n * sizeof(*st->stack));
      st->top = st->stack;
      st->limit = st->stack + n - 1;
    }
If you only access the stack through st->top, it is easy to see (for a human) that the stack will never be accessed uninitialized. Good luck getting the compiler to build an inductive proof of that though.

This is just the first example that popped into my mind because I was doing this yesterday.

I said most, not every.

  int x;
  ....
  x = 5;
  printf("%d\n", x);
Anyway, in you code the cost of malloc is a lot higher than the cost of zeroing out. And I guess most of the time it is hard to elide variable initialization would be in places where you have non-local variable use and you do heap allocation.
> Anyway, in you code the cost of malloc is a lot higher than the cost of zeroing out.

You can't possibly know that. The cost of malloc() is (roughly) independent of the size of the memory, and the cost of zeroing it is linear in the size of the memory. So there's an inflection point where zeroing the memory becomes more expensive than the malloc() itself.

Point taken, however in practice this point is quite high to not matter that much. At least in these benchmarks zeroing out is typically responsible for less than 10% of total time overhead, and this including the fact the benchmarked VMs didn't do any zeroing-out elision:

http://www.cs.utexas.edu/users/mckinley/papers/zero-oopsla-2...

I was mostly talking about general variable initialization, not just huge arrays - and for any case "hard" like yours (but actually not impossible to elide; state-machines and inductive proofs are not a rocket surgery) - you've got several trivial cases like I posted above which can be solved by simple local data-flow analysis.

This is the same like with NP-hard problems. Some people get scared by NP-hard, but it only matters if you need an exact solution, and is not a problem in practice if you're ok with a result at 0.95 of the optimum.

It is also sometimes more time-effective to reuse huge arrays to work around obligatory variable initialization done by the runtime for rare cases where it would be a performance problem, than to fight UBs in 10s of other places where performance doesn't matter.

Not if you can initialize them at the time of declaration. IMO, that should be the default policy, and there should be ample comments apologizing for not doing it.

People don't build unterminated communications lines nor use PIO without pullups/pulldowns. Programmers should be the same way.

Perhaps I've led a charmed life, but I've never seen a case where proper initialization was a problem.

Anyone know any other websites that talk about surprising things about C? I saw a really good one about syntax and semantics of C, but I lost it.
He forgot one: modifying & using a variable between sequence points (e.g., i = i++ + i++;). EDIT: not forgot, but only mentioned briefly at the end.
Obligatory links: http://blog.metaobject.com/2014/04/cc-osmartass.html, http://robertoconcerto.blogspot.co.uk/2010/10/strict-aliasin... (I post these every time this subject crops up. TL;DR - stand up for yourself, and don't let your C compiler treat you like shit.)
See also the "maybe it's not the programmers who are being stupid, it's the language that is in need of change" proposal at http://blog.regehr.org/archives/1180 with discussion https://news.ycombinator.com/item?id=8233484
There are better alternatives to C if a friendly language is what you want. C has its niche and I don't see C changing, and I've yet to see a "dialect" be really successful. If you're willing to switch to "friendly C", why wouldn't you already be switching to Rust, or D, or even Haskell?
Because you might need the low-level, reasoning about memory/cpu/etc capabilities of C -- which Haskell doesn't offer -- but don't want all the undefined behavior and extra unneeded complexity.

As of Rust, it's not even 1.0-stable out yet. Even if it was, it would take 4-5 years to mature, get enough support, tooling etc to be something you'd use instead of C/C++ for everything.

> If you're willing to switch to "friendly C", why wouldn't you already be switching to Rust, or even Haskell?

Because you are already using it. You, the median C developer, are already writing code assuming that it will be compiled by a friendly C compiler. Friendly C is one of the most widely used programming languages on the planet as of 2015. The most widely used SSL/TLS stacks are written in friendly C.

The “friendly C” manifesto is not intended for C developers, an amazing proportion of whom already write friendly C. The manifesto is aimed at C compiler developers.

Friendly C just needs a compiler, or perhaps several of them.

Yes. To quote one of the links I posted: "The tl;dr version of this is that the C compiler writers have no idea who their users are, so they keep fucking up working code [...] All of this only hurts the systems-level programmer (and I claim is of dubious value to the high-level guys). Which is why I insist that the compiler writers don't know their own customers. C is a systems-programming language! Why do you want to screw the systems-programmer to help the high-level programmer, when the high-level guy has long since moved to C++ and is bumbling around with STL?"

That whole post is worth reading. Though something of a tossed-off ranty blog post, I think it very accurate.

The only kind of systems-programmer who should be writing C is the kind who is capable of manual memory management. The "optimized DSP kernel loops" usually only possible to write with unsigned arithmetic anyway. And if you're using unions at all it's because you're in a super-performance-critical place, in which case the strict-aliasing rules absolutely are valuable (and "other parts of your generated code are slower than MSVC" is a stupid argument). The problem isn't with C, it's with people using C where it's the wrong tool.
> Though something of a tossed-off ranty blog post

Touché.

Think of it not as a criticism, nor a suggestion that there is anything wrong with its author writing it that way. It's simply an admittance in advance that this thing I'm recommending might come across, on first reading, as more easily dismissed than I think it should be.

Perhaps I phrased it poorly, though. My first go sounded wishy-washy, so I rejigged it... I could well have over-tightened things.

I'm actually curious: is there anything in C that would prevent some sort of annotation being made part of the standard that essentially meant "Do what I say"?

For example:

    [[dwis]]
    {
    if ( signed_int + another_signed_int < signed_int ) {
        // do stuff if there is an overflow
    }
    }
Or something similar, which wouldn't be optimized away?