I'm honestly a little surprised that someone would stream this -- so much potential for egg on face for the debugger, and had it been me, much time would have been spent scratching my head, which I don't think would make for compelling viewing.
Bonus snark: Fractional line numbers sounds like something Larry Wall would add to Perl, deliberately, as a "feature".
Do students get much practice in debugging and root cause analysis on existing systems? That's something people do a lot of at work, but perhaps not students so much. I'd imagine students work more in a clean "green-field" environment.
Kind of like experience gained as a plumber working with dirty clogged real pipes in a basement, vs doing exercises in plumbing school with new pipes and theory about pipe sizes, gradients etc. In other words, school can only give part of the picture.
> Do students get much practice in debugging and root cause analysis on existing systems?
It's a waste of a student's time. Deep debugging isn't just one set of skills; each problem is different. It's usually very time-consuming, and you will need to learn new skills and tools.
That is: you have to encounter a problem that is a blocker, so that your motivation is that you have to solve the problem.
My experience was a long time ago: I had linked a library compiled with Borland C with code compiled with Microsoft C. It wouldn't work. I wss only using one function from the Borland library; that's where the error was occurring. It turned out that the Microsoft compiler required the callee to restore the flag register; the Borland compiler required the caller to do that. Therefore the carry bit wasn't being restored correctly, causing the bug. Took several days to figure out.
Short answer is start doing things. Somewhat longer answer - develop interest in understanding deeper level things because when things seem hopeless it's the interest/curiosity that will keep you going. Secondly find things to break and fix - this will give you opportunities to learn wider range of things.
Computers aren't magic, much as they seem to be. In general, if you can reproduce a bug reliably, it is always possible to keep digging until you find the cause. It may take more and more esoteric tools to do so, and you may need to start breaking apart the components you're working with, but at least ask yourself the question "what is preventing me from examining what's going on here" and try to find out how to look inside whatever opaque box you've run up against. Those boxes will start out being like "the function I just wrote", but as your understanding increases and you become familiar with how to look inside the least opaque boxes, the hurdles become more like "the JavaScript runtime", "the C compiler", "the syscalls made to the operating system" or even "the microcode emitted by the CPU" (though I've never needed to go to that level myself!).
Further nit: Modern x86 CPUs use PLAs to generate microops ("uops") that feed the execution engine. For complex instructions (like CPUID, RDMSR, VM-related, etc.), the microcode ("ucode", which can be updated) generates microops.
The execution engine works in discrete[a] operations, hence uops. The microcode is a sequencer that tears apart ISA instructions into those uops.
[a]: I'm considering fused operations (like CMP+Jcc) as single "operations" for simplicity.
If you're writing a language it supports, use godbolt - https://godbolt.org/. It's great for settling arguments (do these two versions compile to the same code? If not figure out why! Make sure you're compiling with optimizations)
Spend some time learning gdb and other classic cli tools.
By spending a lot of time on the problem and not panicking, essentially.
It you don't rush and focus on making progress, you will find it eventually. There is no wizardry involved, skills will help you find the solution faster, but that's your ability to focus and keep your calm that will get to it in the first place.
If you are at work, ignore your boss trying to pressure you (that's not necessary easy), your boss only has the "right" to tell you if you should work or the problem or not, if you are asked to work on the problem, do it as if you have your entire life to do it. With experience, you can make estimates that can help decide if the problem is worth trying to solve or not, but with the correct mindset, one doesn't need experience to solve the problem itself, instead one builds experience by solving the problem.
I learned more about the details of how computer worked in a class on programming an Intel 8080 (on a CP/M system with dual 8" floppy drives) than any other course. Computer science majors were either nascent or non-existent at that time so I don't know what is available today. And microprocessors are a lot more complex than the 8080.
Nevertheless, I recommend learning the details of computer architecture by learning some assembly programming. Higher level languages are an abstraction meant to shield the developer from the details of the H/W but it remains useful to understand the details, particularly if the abstraction is broken.
I personally got there by treating bugs as problems I could try and ask questions about until I had a smaller more precise bug. So either I work out something to ask and answer about the nature of some output, or I try to ask and answer something that helps me more precisely pin down the bug.
Sometimes that can be turning an intermittent problem into something I can reproduce consistently, and sometimes it can be constructing new test cases and seeing if their results match my hypothesis.
It helps to either have a good memory, or to keep notes so you know what you’ve already explored and tested, and it’s something you get better at with practice. I spent several years on third line product support working out what was going wrong for customers, producing hot fixes, and working with devs to turn those into proper fixes in the next version.
I happen to work on profiling tools where we rely on compilers to output metadata about the machine executable code correctly. It turns out this is surprisingly hard and goes wrong quite often, and sometimes it's even impossible to maintain correctness across multiple rounds of compiler optimizations. For us the way compilers solve this is basically the worst possible scenario, when the compiler can't correctly track it across optimizations, it just starts emitting less or sometimes even no information. These bugs are just awful to understand.
At the end of the day compilers are also just software and it's just a matter of how many features you end up being exposed to.
The sort of thing profilers and debuggers want doesn't really exist at high optimisation levels. In my experience that kind of thing isn't exactly a bug, it's just that the information you want can't even exist in theory.
Kind of the same thing as reporting memory usage for programs. You want to be able to say "A uses 3MB, B uses 2MB", but then there's shared memory, file caches, fragmentation, etc. So much complication that "how much memory is this process using?" isn't even a valid question anymore.
Debug info under high optimization is very tricky. A simple example:
for (i = 0; i < 4; i++) ...
If the compiler unrolls the loop four times the eliminates the induction variable altogether, then combines the four iteration loop into a single vector instruction, what debug info should it emit for the value of i in the loop body?
You can argue that it should emit what i would have had at a given point, but now you have four different values for i all at that one vector instruction. Reasoning about that is quite difficult.
True, but I would be happy if at least the debugger could point me at the value which is clearly being iterated over in a register, or tell me the value of a standard ABI function's arguments (and no, it hasn't been overwritten by something else).
> the information you want can't even exist in theory.
There are things that do exist, though, right? For example, the compiler knows which values are live at every point of the code, and in which registers and stack slots they are located. In most cases it even knows the SSA that produced them, and—again, in most cases—that SSA is expressible in the source language in terms of user-visible values. (How to display that ergonomically in the presence of inlining is a problem, though.)
None of this will fit in DWARF, but honestly DWARF is enough of a slog to work with that I’m not really sorry about that. (And PDB is not a contender until actual first-party documentation and library code exists.) The design work is thorny, don’t get me wrong, but I don’t think this is impossible.
> For example, the compiler knows which values are live at every point of the code,
No because "point of the code" is not well defined after optimisation. The compiler can reorder statements, split them, merge them, inline them, outline them, eliminate them entirely etc.
The compiler needs to know that information in order to do said optimisations, so it does have the information somewhere, it just fails to express it in the debug information.
A better workaround would be to call __HrLoadAllImportsForDll[1].
Delay loading lets you defer grabbing function pointers from a dll (normally until the first call of that function). But if you call __HrLoadAllImportsForDll from the addon's init code, you can attempt to load them all in one go. That way it doesn't happen during a call with a float argument, so it shouldn't cause problems.
i found my first compiler bug in my first job in my first few weeks... nobody was keen to believe me except the ms support guy i eventually got in touch with - who was kindly able to tell me that it was caught and fixed in a later version of the compiler that we couldn't use, but then at least gave me the clout to get the issue worked around instead of ignored as the fantasy of a junior programmer.
> $ make install PREFIX=/Projects/binutils-gdb/stragerusr
> After a minute, I realized that binutils was being installed in /ucrt64, not into my temporary directory. Oh no!
Tip: to install Autotools- or otherwise GNU-Coding-Standards-compliant projects (e.g. CMake-based ones using GNUInstallDirs) into places you haven’t configured them to go to, on the hope they will kinda sorta work, use DESTDIR[1]. The value of PREFIX (et al.) is set at configuration time, and overriding it via make does not work, while DESTDIR is there specifically to be overridden. It’s what Linux package managers use to create a parallel bin/, lib/, usr/, etc. hierarchy that’s then archived up into a package. (The value of DESTDIR is prepended to all destination paths, it does not override them.)
Also, support DESTDIR in your build systems. Distro maintainers will thank you.
My general philosophy is the compiler or runtime will be to blame about once or twice a career, the library will be to blame a couple times a year (obviously varies with library and language - Node developers find a lot more library bugs because JS) and it’s your fault the rest of the time.
I've only been developing a few years, and I've already found one compiler bug, in Typescript's type-checker [1]. It's a bit easier to find compiler bugs when new features are getting added pretty frequently.
To be specific, it was a bug in “dlltool”, a part of GNU Binutils used on Windows platforms. The bug was fixed in May 25, 2023, and released in Binutils 2.41 on July 30 2023.
Oh my god. I have been chasing a null value error in our profiling SDK for the past 3 days. Calling napi_create_double was for some reason always generating a null value for the first entry and I tried everything to reproduce it thinking I have a bug in my code. Thank you, you saved me!
Not exactly a compiler bug, but Borland Turbo C (which predates ANSI C) had bizarre semantics around character pointers such that char *s was almost a managed string type, and assignment would make a deep copy of the string.... much to the surprise of anyone expecting identical addresses.
The manual maybe (?) hints at this behavior:
> Using a Character Pointer
> The second method you can use to define strings is a character pointer. Edit your program to look like this:
> The asterisk (*) in front of msg tells the compiler that msg is a pointer to a character; in other words, msg can hold the address of some character. However, the compiler sets aside no space to store characters and does not initialize msg to any particular value.
> When the compiler finds the statement
msg = "Hello, world\n";
it does two things:
- As before, it creates the string "Hello, world\n", followed by a null character somewhere within the object code file.
- It assigns to the starting address of that string-the address of the character H-to msg.
> The command puts(msg) works just as it did before, printing characters until it encounters the null character.
48 comments
[ 2.9 ms ] story [ 102 ms ] threadI'm honestly a little surprised that someone would stream this -- so much potential for egg on face for the debugger, and had it been me, much time would have been spent scratching my head, which I don't think would make for compelling viewing.
Bonus snark: Fractional line numbers sounds like something Larry Wall would add to Perl, deliberately, as a "feature".
Kind of like experience gained as a plumber working with dirty clogged real pipes in a basement, vs doing exercises in plumbing school with new pipes and theory about pipe sizes, gradients etc. In other words, school can only give part of the picture.
And great parties as well.
It's a waste of a student's time. Deep debugging isn't just one set of skills; each problem is different. It's usually very time-consuming, and you will need to learn new skills and tools.
That is: you have to encounter a problem that is a blocker, so that your motivation is that you have to solve the problem.
My experience was a long time ago: I had linked a library compiled with Borland C with code compiled with Microsoft C. It wouldn't work. I wss only using one function from the Borland library; that's where the error was occurring. It turned out that the Microsoft compiler required the callee to restore the flag register; the Borland compiler required the caller to do that. Therefore the carry bit wasn't being restored correctly, causing the bug. Took several days to figure out.
That's how I got there at least...
The execution engine works in discrete[a] operations, hence uops. The microcode is a sequencer that tears apart ISA instructions into those uops.
[a]: I'm considering fused operations (like CMP+Jcc) as single "operations" for simplicity.
Spend some time learning gdb and other classic cli tools.
Don't be afraid of reading books!
It you don't rush and focus on making progress, you will find it eventually. There is no wizardry involved, skills will help you find the solution faster, but that's your ability to focus and keep your calm that will get to it in the first place.
If you are at work, ignore your boss trying to pressure you (that's not necessary easy), your boss only has the "right" to tell you if you should work or the problem or not, if you are asked to work on the problem, do it as if you have your entire life to do it. With experience, you can make estimates that can help decide if the problem is worth trying to solve or not, but with the correct mindset, one doesn't need experience to solve the problem itself, instead one builds experience by solving the problem.
Nevertheless, I recommend learning the details of computer architecture by learning some assembly programming. Higher level languages are an abstraction meant to shield the developer from the details of the H/W but it remains useful to understand the details, particularly if the abstraction is broken.
Sometimes that can be turning an intermittent problem into something I can reproduce consistently, and sometimes it can be constructing new test cases and seeing if their results match my hypothesis.
It helps to either have a good memory, or to keep notes so you know what you’ve already explored and tested, and it’s something you get better at with practice. I spent several years on third line product support working out what was going wrong for customers, producing hot fixes, and working with devs to turn those into proper fixes in the next version.
At the end of the day compilers are also just software and it's just a matter of how many features you end up being exposed to.
Kind of the same thing as reporting memory usage for programs. You want to be able to say "A uses 3MB, B uses 2MB", but then there's shared memory, file caches, fragmentation, etc. So much complication that "how much memory is this process using?" isn't even a valid question anymore.
for (i = 0; i < 4; i++) ...
If the compiler unrolls the loop four times the eliminates the induction variable altogether, then combines the four iteration loop into a single vector instruction, what debug info should it emit for the value of i in the loop body?
You can argue that it should emit what i would have had at a given point, but now you have four different values for i all at that one vector instruction. Reasoning about that is quite difficult.
There are things that do exist, though, right? For example, the compiler knows which values are live at every point of the code, and in which registers and stack slots they are located. In most cases it even knows the SSA that produced them, and—again, in most cases—that SSA is expressible in the source language in terms of user-visible values. (How to display that ergonomically in the presence of inlining is a problem, though.)
None of this will fit in DWARF, but honestly DWARF is enough of a slog to work with that I’m not really sorry about that. (And PDB is not a contender until actual first-party documentation and library code exists.) The design work is thorny, don’t get me wrong, but I don’t think this is impossible.
No because "point of the code" is not well defined after optimisation. The compiler can reorder statements, split them, merge them, inline them, outline them, eliminate them entirely etc.
1: https://learn.microsoft.com/en-us/cpp/build/reference/linker...
> After a minute, I realized that binutils was being installed in /ucrt64, not into my temporary directory. Oh no!
Tip: to install Autotools- or otherwise GNU-Coding-Standards-compliant projects (e.g. CMake-based ones using GNUInstallDirs) into places you haven’t configured them to go to, on the hope they will kinda sorta work, use DESTDIR[1]. The value of PREFIX (et al.) is set at configuration time, and overriding it via make does not work, while DESTDIR is there specifically to be overridden. It’s what Linux package managers use to create a parallel bin/, lib/, usr/, etc. hierarchy that’s then archived up into a package. (The value of DESTDIR is prepended to all destination paths, it does not override them.)
Also, support DESTDIR in your build systems. Distro maintainers will thank you.
[1] https://www.gnu.org/prep/standards/html_node/DESTDIR.html
Or that old saying….
Any science well done, is as good as magic.
— <http://freefall.purrsia.com/ff300/fv00255.htm>
[1] https://github.com/microsoft/TypeScript/issues/35970
The manual maybe (?) hints at this behavior:
> Using a Character Pointer
> The second method you can use to define strings is a character pointer. Edit your program to look like this:
> The asterisk (*) in front of msg tells the compiler that msg is a pointer to a character; in other words, msg can hold the address of some character. However, the compiler sets aside no space to store characters and does not initialize msg to any particular value.> When the compiler finds the statement
it does two things:- As before, it creates the string "Hello, world\n", followed by a null character somewhere within the object code file.
- It assigns to the starting address of that string-the address of the character H-to msg.
> The command puts(msg) works just as it did before, printing characters until it encounters the null character.
The workaround is of course to use s1 = &s2[0]