In my personal experience you haven't really completely tested your software until you've hit a couple compiler bugs. :) Though they've gotten better in recent years.
The hits in the implementation of draft-brezak-win2k-krb-rc4-hmac-04.txt seem ... interesting.
> As I rushed to recompile my computer system using GCC 8, I contemplated the vast consequences of such a bug could be, and pondered how it was possible that computers could function at all.
When something doesn't work as expected, I'll often check disassembly. That can massively cut troubleshooting time when something "smells" like a compiler bug.
This is why preferably everyone should learn to read assembler output. This is not limited to C/C++/Rust/etc. native code, the same output is typically also available for example for JVM and Javascript JIT.
Haven't found any miscompilations so far (unless you count braindead codegen), but quite a few hardware bugs. Including one CPU bug.
Single step through functions in a debugger to see how things work, and for example how a stack frame is set up. Pay attention how registers and flags are affected by the execution. See how conditional branches are affected by the flags.
When you're looking at the code, remember that there are often weird looking details, like unused portions in stack and in compiled function codegen (loops, entry points) for alignment purposes — modern CPUs hate unaligned things.
Note: Some x86 instructions have implicit register use that might not be directly obvious. Like PUSH, POP, IMUL, IDIV, LOOP, STOS[BWD], MOVS[BWD], etc. They can affect registers that are not mentioned in the instruction operands.
In general, if the things look weird, just google the instruction. Much less surprises in other mainstream architectures, like ARM. All architectures do have vector instructions that might confuse you at first, like x86 vectored double precision add, VADDPD. Again, just web search them. No one remembers all of the instructions by heart, there's just no point.
Web search for assembler tutorials and simulators. They're too many to list, just pick something suitable for your taste.
In short, play around. Don't get scared by something weird, just look it up.
Don't stress if you don't understand everything, you can always look it up or try it out in a debugger. Even a little bit can help quite a bit.
JITs usually have some way to display generated assembler. For example, to see the native x86/ARM/whatever code generated by JVM you'd say something like:
> Again, just web search them. No one remembers all of the instructions by heart, there's just no point.
Another good tip: download the official architecture manuals, they're freely available for the most relevant major architectures (and for x86, both Intel and AMD have their own version). They're monstrous doorstoppers (several volumes with many thousand pages), but include in excruciating detail the description of every single instruction, with the advantage that they'll work even if your Internet connection is offline.
> but include in excruciating detail the description of every single instruction
well, at least the official ones with the officially supported operands. Sandsifter has a lot to say about how many undocumented instructions exist on just the "exposed" x86 part of the CPU, nevermind the ARM and other sub ring-0 stuff.
Yes, but in the context of this thread: undocumented instructions can change from generation to generation of a processor architecture, while documented instructions are much more stable; and because of that, compilers will not generate undocumented instructions. When reading compiler output, the official documentation of the instructions (plus documentation of the calling conventions) should be enough.
I'm still a bit of an assembly/JIT newbie, but a few thoughts:
When you write assembly code, it's generally specific to several details, which in turn affects your choice of tutorials from which to work:
1) The target architecture (x86-64, AArch64, MIPS, etc.) This defines what instructions you have available, the memory model, etc.
2) The architecture ABI. E.g., what's the protocol for passing parameters to functions you call, and for receiving the return code. [0]
3) The particular assembler you plan to use. Gnu As (and I assume others) provide some directives that don't map directly to machine instructions, but do some book-keeping to make your life easier.
4) (Potentially) the file format for the resulting code [1]. I think this is one area where the assembler and linker utilities can shelter you from the ugly details.
FWIW regarding JIT: Lately I've started playing around with Xbyak [2]. It's probably a bit light on the documentation, but for the most part I've found it to be an easy way to get started with JIT.
Make it more accessible from your build system. That way, when your curiosity is piqued (or frustration has peaked...), its easy to pick it up and see for yourself.
With GCC, its as simple as adding `-save-temps=obj`. You get preprocessed source and assembly emitted alongside the object files.
I look at a fair amount of disassembly. One tip to understand something complex in the debugger is to skip to callsites if they are available in the interesting part of the code. The compiler usually has to put values in conventional locations, either as arguments or live-through variables, so your variables are easier to track.
Compiler bugs are indeed pretty frightening. A few years ago I bumped into one in some code that had potential to have a big impact. Unfortunately I am not at liberty to give details about the business setting except to say that we had processes in place that prevented any danger.
In the end I whittled it down to the following tiny C# program:
namespace UhOh
{
internal class Program
{
private static void Main()
{
System.Console.WriteLine(Test(0, 0));
}
private static bool Test(uint a, uint b)
{
var b_gte_a = b >= a;
var b_gt_a = b > a;
System.Console.WriteLine(b_gte_a);
return b_gte_a && b_gt_a;
}
}
}
Compiling and running this with Microsoft's .NET stack with versions 4.7.0 and below, the output was incorrectly: "True, True" instead of "True, False". (IIRC, it also had to be a 64-bit Release build.)
The intermediate language was correct; it was a bug in RyuJIT.
A couple of years back I ran into a JDK JIT bug during a project. The code ran fine until I ran it through benchmarks, which triggered JIT on a method causing it to return incorrect results.
Took a long time to find, because there were no errors, just wrong results (a specific if statement taking the wrong branch).
Trying to get assistance from others were mostly met with responses along the lines of "It's probably a race condition" (in single-threaded code) / "very unlikely to be a bug in the JIT". I did end up finding a way to disable JIT for the specific method, which solved the issue, and never got around to finding the root cause. I do believe it has been fixed in the meantime at least.
I haven't run into major compiler bugs since then, but often have to dive deep into libraries to find obscure bugs (database drivers and web servers most often).
A couple of decades ago i worked with GSM handsets, and because you certify the GSM stack along with the compiler, you're pretty much tied to a specific compiler version.
At the time we were working on a new "feature phone" with a 160x120 pixel display in 4 shades of grey, which was a huge upgrade compared to our previous models. Another feature was full screen images for the various applications, and we'd been implementing is into the software and testing it for weeks without problem. After the development cycle came to an end, our release team created a new software release and sent it to our test department, which almost instantly reported graphical errors back to us. We tested the software image on our own handsets, and half the screen was "garbage".
We spent weeks looking over the merged code, as well as debugging pointers code and found nothing. It wasn't until we were stepping through the paint code with a Lauterbach debugger that we noticed something was "off" with a pointer.
The platform was a 16 bit platform and memory was addressed using a page pointer pointing to a 64kb memory page. When we traversed the bitmaps fate would have it that this particular bitmap, in this particular build, was split between two pages, and when incrementing a pointer beyond the page limit, instead of incrementing the page pointer, it simply just overflowed and started from the beginning of the current page.
Another interesting bug we chased in the compiler was it's inability to add more than 2 variables in initial assignment.
i.e.
int a = 1+2; // a = 3
int b = 1+2+3; // b = 3
int c = 10 + a + b; // c = 13
Of note to me: certified compiler was actually non performing :)
Lauterbach, the savior of embedded wizardry. Still in use today with chips one would not consider "embedded".
Probably more to do with compile time static expression evaluation (optimization) than parsing. Seems like it's not actually recursing down, just evaluating the root.
That could be, but my first thought was it's only compiling it to a single assembly instruction (which can only have two operands). Or it generates ADD instructions using a succession of registers or stack locations, but the initialization code blindly assumes the result is in the first register or stack location.
Well, on one hand, writing correct albeit slow memcmp() is easy, on the other hand, it has some gotchas...
int my_memcmp(const void * ptr1, const void * ptr2, size_t num) {
const unsigned char * p1 = ptr1;
const unsigned char * p2 = ptr2;
for (size_t i = 0; i < num; i++) {
if p1[i] < p2[i] {
return -1;
}
if p1[i] > p2[i] {
return 1;
}
}
return 0;
}
For example, technically speaking, unsigned char can be as wide as an int, so "p1[i] - p2[i]" may actually evaluate to unsigned int which is not what you want.
I maintain embedded development C and C++ toolchains for a living. I have seen my share of compiler bugs. For example, some optimization pass in a popular open-source compiler that would lose track of dereferences of pointer variables if they were more than 12 bytes deep in the stack, meaning that a reference capture in a C++ lambda would get converted to a value capture if it was the third or later capture in order and changes to the referenced capture would be lost....
Anyway, my experience is that compiler bugs do exist, but maybe 99% or so of "compiler bugs" reported by my users turn out to be undefined behaviour in their code.
Maintaining an application that still has Symbian users (some people are really conservative and like their Nokia E52s, plus there isn't as much malware for this dead OS), bugs in old GCCE are rather annoying.
Sometimes, for no reason, GCCE just crashes compiling totally innocent code. Usually, a minor rewrite of the logic helps, or even weird edits such as adding a new (useless) parameter to a method.
The last GCCE toolchain for Symbian was released by CodeSourcery in March 2012. It contains GCC version 4.6.3. It is theoretically possible to adapt and compile a newer version, but the sources need so many edits that I gave up after a few days.
Symbian in Brazil was rapidily climbing in popularity when MS killed it.
It had 60% of phone market share and it was RISING despite the launch of iPhone and Android.
The reason is that:
1. It worked great.
2. Brazil had a vibrant dev community (people would even port PC games to Symbian O.o)
3. It was much cheaper than iPhone and clones.
4. Nokia phones were just solid and awesome.
After MS made that memo that killed Symbian, it died almost instantly, people got so disappointed with MS that they started to switch to android, even if it was some chinese-made "shit-phone" instead of "feature-phone", the amount of really, really crappy androids that flooded the market was mind boggling, many didn't even work right, for example wouldn't complete calls properly or wouldn't connect to some Wi-Fi channels.
Some years ago i worked on a part of a specialized steering system for a car. This was done with certified everything (Certified Compiler, Processor, a lot of paper work etc.)
This was a 16-Bit processor and the C-compiler had a "funny" bug. If you had a struct with 3 8Bit Values in a row and a 16Bit Value afterwards it would overlap the 8Bit Value with the 16Bit value:
struct {
int8 a;
int8 b;
int8 c;
int16 d;
}
In this case the variable c and d would have the same address. This was on a cpu where we didn't had a debuger (not enough memory left for it), we only had a serial port for debuging.
A serial port is all you need, if you have room for a wee GDB stub. Then you get the full power of GDB.
I do this routinely where the target has 256GB of RAM, and (not incidentally) specialized network hardware, but no dev infrastructure except gdb-server (which provides the stub) and sshd. I build in a docker image that matches the target, but with dev tools, with the output bin directory sshfs-mapped to a directory on the target. I run the binary on the target under gdb-server, opening a socket listener. Then I run gdb natively on my dev machine, and `target remote server:61231` to attach to that socket. If I didn't have easy access to listening ports on it, I could ssh-tunnel one in.
So, a serial port and small RAM doesn't have to mean you have no debugger.
In our case it would probably not have helped. We had a fixed old mcu board where the functionality grow over the years. We were fighting over bytes...
It sounds like the poster was targeting a microcontroller rather than a more generally capable "embedded" CPU. 16 bit CPU, probably on the order of 100KB RAM and code space total if you're lucky. No operating system in the common sense, although you might have some notion of task switching if you're fancy. "Wee" in that context would imply a footprint on the order of 1KB of code and maybe 100 bytes of RAM.
I assume, by saying there wasn't room for debugging functionality, the poster meant that the "jtag" or equivalent hardware port simply couldn't work for single stepping due to the particular architecture requiring compiled-in cooperation of the firmware, and they didn't have the kilobytes of memory to spare.
These days, it's becoming more reasonable to throw Linux based compute nodes at problems previously best served by microcontrollers. A more powerful CPU isn't a superset of a microcontroller, though. Microcontrollers are still necessary when you have "hard" timing requirements and you need to account for where your CPU cycles are going. Even seemingly "solved" problems like participating on a CAN bus is difficult for a Linux based node. For example, while you can easily purchase CAN interface boards for raspberry pi and send and receive messages, you are pretty much guaranteed to drop some percentage of incoming messages at realistic bitrates. All the boards use MCP2515 SPI CAN controllers, and the linux driver simply can't schedule SPI transfers in response to interrupts fast enough to avoid mailbox overruns inside the controller. Maybe it's somehow been cleverly fixed since I last looked at it though?
It is fairly common nowadays to run Linux as one task on an RTOS, and have other tasks manage the CAN controller and other devices that need a low latency response. Or, just to use a coprocessor for low-latency work, as is commonly done to manage wifi.
Routing gdb stub traffic through a hosted Linux to an RTOS task or coprocessor is not an elementary exercise, but is something an engineering student might be expected to implement, even as just part of the real project.
My immediate tangential thought is about Ken Thompson's paper, "Reflections on Trusting Trust".
The gist is that our security issues could come layers away from where we may expect them to, all the way up to the compiler. It's a great paper, but who would have expected anything less from Ken Thompson.
Note also that "it's never a compiler bug" applies more to things like GCC and so on.
If you're working with a new language or quickly changing, e.g. Nim, Crystal, etc, or even something as old as Rust, then it can much more easily just be a compiler bug...
Yes, the older and more popular a compiler is, the less likely it is to have bugs.
The buggiest compiler I ever used was a C compiler that ran on a PC and generated code for the 68000 processor. We seemed to trip over something about once a month.
I once fought a Java compiler which did not produce the proper bytecode when I used the "@Loggable" annotation. (https://aspects.jcabi.com/annotation-loggable.html) Worst off, the incorrect bytecode was on the exception path. I spent at least 2 days making sense out of it.
1. If anyone has significant experience or even just interest in this, they should work to collaborate on how compiler bugs could be taken advantage of for unintended purpose. Otherwise, there will continue to be a lot at risk.
2. You think compiler bugs are bad? How about hardware hacking, quantum effects, etc.
Similar to Schrödinger's cat, security is relative, never absolutely exists, but always exists. It's a Platonic ideal.
"Maturity" of understanding the nature of security is variable and multi-state, like a spiritual journey; greater understanding of security may lead onto to great loss of faith or you may go beyond into the light of true awakening. Most will only get a firewall, though.
Oh compilers are fun. Just recently I was reading through Rust's bug tracker, as one does, and learned that comparing function pointers is not deterministic. Compiling this code [0] in the Debug mode yields different results than in the Release mode. You can read the whole discussion about whether it's a LLVM bug, a Rustc bug, an undefined behavior, an intended behavior, a pretty serious bug, or nothing to worry about over at [1].
> comparing function pointers is not deterministic
The comparison is deterministic - the perhaps unexpected part is that two distinct but identical functions in the source code are folded into one in the binary.
That bug was actually a bit deeper; it merged the functions but would statically fold some comparisons to false while letting others persist to runtime (where they would evaluate to true) in the same build. You could do `x == y` twice and get different results, one of which was a lie.
God, that issue is a mess. Lots of people missing the forest (Rust and LLVM are breaking constant folding guarantees) for the trees (function pointer equality is weird).
It should be noted that function pointer comparison is defined to not be deterministic between different compiler option sets, it's at least for now possible but mostly useless.
Through there are bugs involved, too ;=)
(Due to const folding comparisons vs. doing them at runtime there was/is non-determinism in the same build between different call sites ... )
Ah kids these days. New compilers used to be written every year or so, and had the most horrific bugs. For instance, the one that reordered complex 'if' conditions to evaluate in chunks, ignoring precedence. Or the compiler that stored parameter context while compiling with a different lifetime than the actual one - resulting in references to deleted memory during compilation. And on and on.
Used to be, a compiler bug was right up there with a memory issue in your list of 'what might be wrong'.
The problem with compiler and standard lib bugs is it's the last thing you suspect. You're always going to look at your own code first, because 99/100 it's you and not them. You're never going to immediately think "compiler bug", your first port of call is gonna be "I must be using the API wrong".
I discovered a bug in the Swift standard lib once, and it took ages before I got to the point where I decided to strip out my own code, just to make sure it was me. And it wasn't, there was genuinely something wrong in the lib that other people on SO were also able to reproduce.
Good on him for finding a bug in secp256 too. When it comes to cryptography code, it can be very hard to know what the right answer is. I always find some examples on the internet and put them in a unit test to make sure I'm not misusing the API, because if you do your answer looks the same: bunch of numbers in a byte array. To know that your numbers are wrong, you need to be sure you are testing them correctly. Which you can't be if you don't know if you're using the API correctly.
That's a rookie mistake. After being wrong enough times, most people will catch on that the chances of a compiler bug are near enough to zero to not worry about. The last actual compiler bug I can remember was probably 30 years ago.
If we expand looking for a compiler bug to looking for a bug in dependencies, I still don't think that should be the first step of searching for a problem (unless your dependencies are that bad, which sometimes they are). However, it's not a bad thing to do. Sometimes, you will have bugs in your dependencies, so you should know how to debug them, so it's a skill you should practice. Also, any time spent debugging your dependencies is also time spent getting deep knowledge of your dependencies which is important to have. Anyway, what else are you going to do when you're stuck on a debugging task where it looks to you that the code is right, but the answer is wrong.
Back in the day Red Hat decided to ship GCC 2.96 which was unstable and not meant for general release. I've seen a handful of compiler bugs but all (or almost all) of them can be traced back to that decision, but because that happened right when I started writing software full time it took a bit before I started trusting the compiler.
Or it was undefined behavior in your code, like depending on the evaluation order of function parameters, or one of mine, testing an array element before testing the index was within bound.
Optimizer: "j was used to access values array, j outside array would be undefined behavior, therefore it must have been within the array boundaries and does not need to be tested."
Oh, and threading bugs where some variable that should have been mutex locked or made atomic gets moved in or out of a register depending on the phase of the moon, causing other threads to fail to notice changes to that variable...but add a debug line and it suddenly works.
A wonderful example of this was some Linux kernel code that ended up causing a security vulnerability. It was something approximating:
int foo(struct data *data) {
struct member *member = &data->member;
if (data == NULL) {
return -EINVAL;
}
// Do stuff
}
Assigning a value to member is just a matter of taking data and adding the value of member, so won't explode if data is NULL since it's not actually dereferencing that address. But this is still in undefined behaviour territory, so gcc assumed that we must know that data could never be NULL and optimised out the check.
This sort of thing is why I have definite feelings about the use of C in security sensitive contexts.
hahah good point. The bug I discovered was in the erlang compiler, which does not have UB (in the language being compiled), and threading bugs are either rarer or obvious.
Like every 6 months something impossible happens on my visual studio. For example the program enters an if it shouldnt. It evaulates to false but still enters.
This happens because somehow it uses old cached intermediate files that are no longer valid. No matter what i do it refuses to build it correctly. Even clean and rebuild does not work, it forces me the delete project files and recreate them
It probably an err on our side, especially aince clean dowsnt work. But it is quite annoying when it happens and gets hard to figure out what is going on
Hardly. This might have been the case years ago but these days it is pretty rare, unless you’re running the development version and using the newest features which is pretty understandable imo.
I'm speaking of personal experience, these days. Templates, macros, quote -- full of bugs. There is a very large number of open bugs, and I encounter them. I've been through the compiler code--a lot of low quality stuff, numerous bugs never reported.
Reminds me of an incident in 2015 where a colleague and I stayed late trying to figure out a strange case of our physics engine producing NaN values out of nowhere. Even when we found the culprit it took us so long to believe our findings. There was a bug in Visual Studio's intrinsics for certain geometric functions (I believe it was sin). There was even a bug report for it with a reply saying they were aware of the issue and had no immediate plan to fix it.
Back in 2015, GCC was already[0] over 14 million lines of code. It's surely much bigger now (although google didn't immediately provide a number). No one should be surprised when bugs crop up.
135 comments
[ 1596 ms ] story [ 2590 ms ] threadThe hits in the implementation of draft-brezak-win2k-krb-rc4-hmac-04.txt seem ... interesting.
This hits home.
This is why preferably everyone should learn to read assembler output. This is not limited to C/C++/Rust/etc. native code, the same output is typically also available for example for JVM and Javascript JIT.
Haven't found any miscompilations so far (unless you count braindead codegen), but quite a few hardware bugs. Including one CPU bug.
First of all, https://godbolt.org/. C/C++/Rust to assembler, very useful.
Learn the calling conventions, basic arithmetic, flags, conditional branching. Understand stack management.
Single step through functions in a debugger to see how things work, and for example how a stack frame is set up. Pay attention how registers and flags are affected by the execution. See how conditional branches are affected by the flags.
When you're looking at the code, remember that there are often weird looking details, like unused portions in stack and in compiled function codegen (loops, entry points) for alignment purposes — modern CPUs hate unaligned things.
Note: Some x86 instructions have implicit register use that might not be directly obvious. Like PUSH, POP, IMUL, IDIV, LOOP, STOS[BWD], MOVS[BWD], etc. They can affect registers that are not mentioned in the instruction operands.
In general, if the things look weird, just google the instruction. Much less surprises in other mainstream architectures, like ARM. All architectures do have vector instructions that might confuse you at first, like x86 vectored double precision add, VADDPD. Again, just web search them. No one remembers all of the instructions by heart, there's just no point.
Web search for assembler tutorials and simulators. They're too many to list, just pick something suitable for your taste.
In short, play around. Don't get scared by something weird, just look it up.
Don't stress if you don't understand everything, you can always look it up or try it out in a debugger. Even a little bit can help quite a bit.
JITs usually have some way to display generated assembler. For example, to see the native x86/ARM/whatever code generated by JVM you'd say something like:
Maybe throw in -Xcomp for maximum optimization. I don't remember the details, just look them up. :-)Other JITs have similar ways, once you know this kind of thing exists it should be easy to look it up.
Another good tip: download the official architecture manuals, they're freely available for the most relevant major architectures (and for x86, both Intel and AMD have their own version). They're monstrous doorstoppers (several volumes with many thousand pages), but include in excruciating detail the description of every single instruction, with the advantage that they'll work even if your Internet connection is offline.
well, at least the official ones with the officially supported operands. Sandsifter has a lot to say about how many undocumented instructions exist on just the "exposed" x86 part of the CPU, nevermind the ARM and other sub ring-0 stuff.
When you write assembly code, it's generally specific to several details, which in turn affects your choice of tutorials from which to work:
1) The target architecture (x86-64, AArch64, MIPS, etc.) This defines what instructions you have available, the memory model, etc.
2) The architecture ABI. E.g., what's the protocol for passing parameters to functions you call, and for receiving the return code. [0]
3) The particular assembler you plan to use. Gnu As (and I assume others) provide some directives that don't map directly to machine instructions, but do some book-keeping to make your life easier.
4) (Potentially) the file format for the resulting code [1]. I think this is one area where the assembler and linker utilities can shelter you from the ugly details.
FWIW regarding JIT: Lately I've started playing around with Xbyak [2]. It's probably a bit light on the documentation, but for the most part I've found it to be an easy way to get started with JIT.
[0] https://en.wikipedia.org/wiki/X86_calling_conventions#System...
[1] https://en.wikipedia.org/wiki/Comparison_of_executable_file_...
[2] https://github.com/herumi/xbyak
Maybe start with JF Bastien talk at CppCon 2020. He did a very nice historical overview of JITs.
"Just-in-Time Compilation"
https://www.youtube.com/watch?v=tWvaSkgVPpA
Then some quick stuff:
"Understanding HotSpot JVM Performance with JITWatch"
https://www.infoq.com/presentations/jitwatch/
In Visual Studio you can directly see CLR JIT Assembly in debug mode (F12)or via WinDbg and SOS plugin.
https://docs.microsoft.com/en-us/windows-hardware/drivers/de...
https://docs.microsoft.com/en-us/dotnet/framework/tools/sos-...
You can also play with it online, https://sharplab.io/
V8 Blog
https://v8.dev/
With GCC, its as simple as adding `-save-temps=obj`. You get preprocessed source and assembly emitted alongside the object files.
For example:
- With Clang, you can dump the C/C++ AST and/or the LLVM IR.
- With GCC, you can dump the Gnu Assembly (with source-level annotations).
These views can be helpful, especially for someone unfamiliar with the target machine's instruction architecture and ABI.
In the end I whittled it down to the following tiny C# program:
Compiling and running this with Microsoft's .NET stack with versions 4.7.0 and below, the output was incorrectly: "True, True" instead of "True, False". (IIRC, it also had to be a 64-bit Release build.)The intermediate language was correct; it was a bug in RyuJIT.
Took a long time to find, because there were no errors, just wrong results (a specific if statement taking the wrong branch).
Trying to get assistance from others were mostly met with responses along the lines of "It's probably a race condition" (in single-threaded code) / "very unlikely to be a bug in the JIT". I did end up finding a way to disable JIT for the specific method, which solved the issue, and never got around to finding the root cause. I do believe it has been fixed in the meantime at least.
I haven't run into major compiler bugs since then, but often have to dive deep into libraries to find obscure bugs (database drivers and web servers most often).
At the time we were working on a new "feature phone" with a 160x120 pixel display in 4 shades of grey, which was a huge upgrade compared to our previous models. Another feature was full screen images for the various applications, and we'd been implementing is into the software and testing it for weeks without problem. After the development cycle came to an end, our release team created a new software release and sent it to our test department, which almost instantly reported graphical errors back to us. We tested the software image on our own handsets, and half the screen was "garbage".
We spent weeks looking over the merged code, as well as debugging pointers code and found nothing. It wasn't until we were stepping through the paint code with a Lauterbach debugger that we noticed something was "off" with a pointer.
The platform was a 16 bit platform and memory was addressed using a page pointer pointing to a 64kb memory page. When we traversed the bitmaps fate would have it that this particular bitmap, in this particular build, was split between two pages, and when incrementing a pointer beyond the page limit, instead of incrementing the page pointer, it simply just overflowed and started from the beginning of the current page.
Another interesting bug we chased in the compiler was it's inability to add more than 2 variables in initial assignment.
i.e.
That took a while to figure out.Did they write a special expression parser just for declarations? I mean, this shouldn't be possible if there was just one expression parser?
One day I found out about DJGPP, and even if the download cost a fortune in phone time, life was so much better.
Now the C reference book included with MIX C turned out to be much better than the compiler itself, and served me well for a decade.
I begin to understand the people who write their own libc for security reasons.
> unsigned char can be as wide as an int, so "p1[i] - p2[i]" may actually evaluate to unsigned int which is not what you want
Would this matter?
No, it is not true for almost all non-embedded systems. And char cannot be as wide as int, it is the opposite true, int can be as wide as char.
Anyway, my experience is that compiler bugs do exist, but maybe 99% or so of "compiler bugs" reported by my users turn out to be undefined behaviour in their code.
Sometimes, for no reason, GCCE just crashes compiling totally innocent code. Usually, a minor rewrite of the logic helps, or even weird edits such as adding a new (useless) parameter to a method.
The last GCCE toolchain for Symbian was released by CodeSourcery in March 2012. It contains GCC version 4.6.3. It is theoretically possible to adapt and compile a newer version, but the sources need so many edits that I gave up after a few days.
It had 60% of phone market share and it was RISING despite the launch of iPhone and Android.
The reason is that:
After MS made that memo that killed Symbian, it died almost instantly, people got so disappointed with MS that they started to switch to android, even if it was some chinese-made "shit-phone" instead of "feature-phone", the amount of really, really crappy androids that flooded the market was mind boggling, many didn't even work right, for example wouldn't complete calls properly or wouldn't connect to some Wi-Fi channels.This was a 16-Bit processor and the C-compiler had a "funny" bug. If you had a struct with 3 8Bit Values in a row and a 16Bit Value afterwards it would overlap the 8Bit Value with the 16Bit value:
In this case the variable c and d would have the same address. This was on a cpu where we didn't had a debuger (not enough memory left for it), we only had a serial port for debuging.I do this routinely where the target has 256GB of RAM, and (not incidentally) specialized network hardware, but no dev infrastructure except gdb-server (which provides the stub) and sshd. I build in a docker image that matches the target, but with dev tools, with the output bin directory sshfs-mapped to a directory on the target. I run the binary on the target under gdb-server, opening a socket listener. Then I run gdb natively on my dev machine, and `target remote server:61231` to attach to that socket. If I didn't have easy access to listening ports on it, I could ssh-tunnel one in.
So, a serial port and small RAM doesn't have to mean you have no debugger.
In our case it would probably not have helped. We had a fixed old mcu board where the functionality grow over the years. We were fighting over bytes...
I assume, by saying there wasn't room for debugging functionality, the poster meant that the "jtag" or equivalent hardware port simply couldn't work for single stepping due to the particular architecture requiring compiled-in cooperation of the firmware, and they didn't have the kilobytes of memory to spare.
These days, it's becoming more reasonable to throw Linux based compute nodes at problems previously best served by microcontrollers. A more powerful CPU isn't a superset of a microcontroller, though. Microcontrollers are still necessary when you have "hard" timing requirements and you need to account for where your CPU cycles are going. Even seemingly "solved" problems like participating on a CAN bus is difficult for a Linux based node. For example, while you can easily purchase CAN interface boards for raspberry pi and send and receive messages, you are pretty much guaranteed to drop some percentage of incoming messages at realistic bitrates. All the boards use MCP2515 SPI CAN controllers, and the linux driver simply can't schedule SPI transfers in response to interrupts fast enough to avoid mailbox overruns inside the controller. Maybe it's somehow been cleverly fixed since I last looked at it though?
Routing gdb stub traffic through a hosted Linux to an RTOS task or coprocessor is not an elementary exercise, but is something an engineering student might be expected to implement, even as just part of the real project.
The gist is that our security issues could come layers away from where we may expect them to, all the way up to the compiler. It's a great paper, but who would have expected anything less from Ken Thompson.
https://dl.acm.org/doi/10.1145/358198.358210
If you're working with a new language or quickly changing, e.g. Nim, Crystal, etc, or even something as old as Rust, then it can much more easily just be a compiler bug...
The buggiest compiler I ever used was a C compiler that ran on a PC and generated code for the 68000 processor. We seemed to trip over something about once a month.
1. If anyone has significant experience or even just interest in this, they should work to collaborate on how compiler bugs could be taken advantage of for unintended purpose. Otherwise, there will continue to be a lot at risk.
2. You think compiler bugs are bad? How about hardware hacking, quantum effects, etc.
Similar to Schrödinger's cat, security is relative, never absolutely exists, but always exists. It's a Platonic ideal.
"Maturity" of understanding the nature of security is variable and multi-state, like a spiritual journey; greater understanding of security may lead onto to great loss of faith or you may go beyond into the light of true awakening. Most will only get a firewall, though.
[0] https://play.rust-lang.org/?version=stable&mode=release&edit...
[1] https://github.com/rust-lang/rust/issues/54685
The comparison is deterministic - the perhaps unexpected part is that two distinct but identical functions in the source code are folded into one in the binary.
Through there are bugs involved, too ;=)
(Due to const folding comparisons vs. doing them at runtime there was/is non-determinism in the same build between different call sites ... )
Used to be, a compiler bug was right up there with a memory issue in your list of 'what might be wrong'.
https://bugs.debian.org/971027#10
I discovered a bug in the Swift standard lib once, and it took ages before I got to the point where I decided to strip out my own code, just to make sure it was me. And it wasn't, there was genuinely something wrong in the lib that other people on SO were also able to reproduce.
Good on him for finding a bug in secp256 too. When it comes to cryptography code, it can be very hard to know what the right answer is. I always find some examples on the internet and put them in a unit test to make sure I'm not misusing the API, because if you do your answer looks the same: bunch of numbers in a byte array. To know that your numbers are wrong, you need to be sure you are testing them correctly. Which you can't be if you don't know if you're using the API correctly.
1) your test suite works over several revisions of the compiler
2) after you upgrade, suddenly one test fails (especially if it's out of hundreds)
2a) you can isolate the minimum condition and it really leaves your head scratching
3) rewriting the code in a slightly different way makes it pass
The time I found a compiler bug it was because of an optimization that missed a corner case that I just happened to be using.
Broken: table[table_i].values[j] && j<category_map_count && to_i<category_array_len; Fixed: j<category_map_count && table[table_i].values[j] && to_i<category_array_len;
Optimizer: "j was used to access values array, j outside array would be undefined behavior, therefore it must have been within the array boundaries and does not need to be tested."
Oh, and threading bugs where some variable that should have been mutex locked or made atomic gets moved in or out of a register depending on the phase of the moon, causing other threads to fail to notice changes to that variable...but add a debug line and it suddenly works.
This sort of thing is why I have definite feelings about the use of C in security sensitive contexts.
Well, until you start suspecting hardware bugs.
This happens because somehow it uses old cached intermediate files that are no longer valid. No matter what i do it refuses to build it correctly. Even clean and rebuild does not work, it forces me the delete project files and recreate them
It probably an err on our side, especially aince clean dowsnt work. But it is quite annoying when it happens and gets hard to figure out what is going on
Visual Studio's build system is horrible. The compiler and msbuild are actually quite ok if using the command line.
[0] https://www.phoronix.com/scan.php?page=news_item&px=MTg3OTQ