Author missed one of the best features: easy access to hardware breakpoints. Breaking on a memory read or write, either a raw address or via a symbol, is one of the most time saving debugging tools I know.
Most languages let you print the stack, so you can easily see the stack using print debugging.
Anecdotally, dynamic expressions are impossibly slow in the cases I’ve tried them.
As the author mentions, there are also a number of cases where debuggers don’t work. Personally, I’m going to reach for the tool that always works vs. sometimes works.
It may sound obvious to folks who already use a debugger, but in my experience a decent chunk of people don't use them because they just don't know about them.
I am surprised all the time in this industry how many software engineers still debug with printf. It's entirely baffling how senior / staff folks in FAANG can get there without this essential skill.
Every engineer should understand how to use a debugger and a time profiler (one that gives a call tree). Knowing how to do memory profiling is incredibly valuable too.
So many problems can be solved with these.
And then there's some more specialized tooling depending on what you're doing that can be a huge help.
For SQL, the query planner and index hit/miss / full table scan.
And things like valgrind or similar for cache hit/miss.
Proper observability (spans/ traces) for APIs...
Knowing that the tools exist and how to use them can be the difference between software and great software.
Though system design / architecture is very important as well.
It's not a silver bullet, but Visual Studio is leaps and bounds ahead of gdb et. al. for debugging C/C++ code. "Attach to process" and being able to just click a window is so easy when debugging a large Windows app.
It really depends. The VS debugger is excellent, and it is usually easier I agree there. But gdb is more easily scriptable and more powerful in certain situations, so I reach for gdb sometimes for the harder bugs even though I use VS more often. Of course, they don’t really compete with each other that much, since VS is mainly Windows and gdb is mainly Linux. Of course I know you can do either debugger on either OS, but in practice the debugger choice is primarily OS based.
IME console-based debuggers work great for single-threaded code without a lot of console output. They don't work that well otherwise. GUI-based debuggers can probably fix both of those issues. I just haven't really tried them as much.
Meh. None of these sway me. I'm a die hard printf() debugger and always will be. But I do use debuggers regularly, for circumstances where printf() isn't quite up to the task. And there really are only two such categories (neither of which appear in the linked article!):
1. Code where the granularity of state change is smaller than a function call. Sometimes you actually have to step through things one instruction at a time, and I'm lucky enough to have such problems to solve. You can't debug your assembly with printf(), basically[1a].
2. State changes that can't be easily isolated. Sometimes you want to log when something change but can't for the life of you figure out when it's changing. Debuggers have watchpoints.
But... that's really it. If I'm not hitting one of those I'm not reaching for the debugger. Logging is just faster, because you type it in right at the code you're already reading.
[1a] Though there's a caveat: sometimes you need to write assembly and don't even have anything like a printk. Bootstrap code for a new device is a blast. You just try stuff like writing one byte to a UART address or setting one GPIO pin as the first instructions and hope it works, then use that one bit of output to pull the rest up.
I had to avoid doing that inside other macros, or inside Struct or Class definitions, enums, etc. But it wasn't hard, and it was a pretty sizeable codebase.
The DEBUGVIKINGCODER macro, or whatever I called it, was a no-op in release. But in Debug or testing builds, would do something like:
So when I'd run the program, I'd get a directory full of files, one per thread.
Then I wrote another program that would read those all up, and would also read the code, and learn the File Name, Line Number of every GUID...
And, in Visual Studio, this tool program would print to the Output window, the File Name and Line Number, of every call and return.
And, in Visual Studio, you can step forward AND BACK in this Output window, and if you format it correctly, it'll open the file at that point, too.
So I could step forwards and backwards, through the code, to see who called where, etc. I could search in this Output window to jump to the function call I was looking for, and then walk backwards...
Then I added some code that would compare one run to another, and argued we could use that to figure out which of our automated tests formed a "basis set" to execute all of our code...
And to recommend which automated tests we should run, based on past analysis.
In addition to being able to time calls to functions, of course.
So then I added printing out some variables... And printing out lines in the middle of functions, when I wanted to time a section...
And if people respected the GUIDs, making a new one when they forked code, and leaving it alone if they moved code, we could have tracked how unit tests and other automation changed over time.
That got me really wishing that every new call scope really did have a GUID, in all the code we write... And I wished that it was essentially hidden from the developers, because who wants to see that? But, wow, it'd be nice if it was there.
I know there are debuggers that can go backwards and forwards in time... But I feel like being able to compare runs, over weeks and months, as the code is changing, is an under-appreciated objective.
While a debugger is of high value, having access to a REPL also covers the major use cases.
In particular, REPL tools will work on remote session, on pre-production servers etc. _if_ the code base is organized in a somewhat modular way, it can be more pleasant than a debugger at times.
Makes me wonder if the state of debugging improved in PHP land. It was mostly unusable for batch process debugging, or when the server memory wasn't infinite, which is kinda the case most of the time for us mere mortals.
Honestly, I feel like the print vs. debugger debate isn't about the tool, it's about the mindset. Print statements feel like you're just trying to patch a leak, while the debugger is about understanding the plumbing. I’m starting to think relying only on print is a symptom of not truly wanting to understand the system you're working in.
There are two kinds of bugs: the rare, tricky race conditions and the everyday “oh shucks” ones. The rare ones show up maybe 1% of the time—they demand a debugger, careful tracing, and detective work. The “oh shucks” kind where I am half sure what it is when I see the shape of the exception message from across the room - that is all the rest of the time. A simple print statement usually does the trick for this kind.
things I can do with print statements but not a debugger: trace the flow of several values across a program, seeing their values at several different times and execution points in a single screen.
Two of the benefits listed (call stack and catch exceptions at the source) are available in logging as well. A good logging framework lets you add the method name, source file and line number for the logging call-after a few debugging sessions you will construct the call stack quite easily. And C# at least lets you print the exception call stack from where it was thrown.
I agree that adhoc dynamic expression evaluation at run time is very useful and can only be done in a debugger.
This is refreshing. I get triggered by people writing "I don't use a debugger because I'm too smart to need one".
Some other things I'd add:
Some debuggers allow you to add actions. For example logging at the breakpoint is great if I can't modify the source, plus there's nothing to revert afterward. This just scratches the surface. Some debuggers allow you to see entire GPU workloads, view textures etc.
Debuggers are extremely useful for exploring and helping edit code. I can't be the only person that sprinkles breakpoints during development which helps me visualise code flow and quickly jump between source locations.
Maybe someone can give me idea, how can I debug this particular rust app, which is extremely annoying. It's a one of Rustdesk.
It won't run if I compile with debug info. I think it's due to a 3rd party proprietary library. So, to run the app I have to use release profile, with debug info stripped.
So, when I fire up gdb, I can't see any function information or anything, and it has so many system calls it's really difficult to follow through blindly.
Print debugging is, checking patient's life signs, eye color, blood pressure, skin inflammation and so on. However using debuggers are like putting the patient through an MRI machine. It can provide you very advanced diagnostic information, but it's expensive, time consuming, requires specialized hardware and education. Alike medicinal doctors it's easier and logical to use the basics until absolutely necessary.
I don't really get the hate that debuggers sometimes get from old hands. "Who needs screwdrivers if we always used knives?" - You can still use your knife, but screwdriver is a useful tool.
It seems to me that this is one of the many phenomena where people want to judge and belittle their peers over something completely trivial.
Personally, I get the appeal of printing out debugging information, especially if some bug is rare and happens in unpredictable times (such as when you are sleeping). But the amount of info you get this way is necessarily lower than what can be gleaned from a debugger.
45 comments
[ 3.5 ms ] story [ 57.8 ms ] threadStumble upon some corrupted memory? Just put a watch point on it, and run the program backwards. Two weeks just turned into two minutes.
Anecdotally, dynamic expressions are impossibly slow in the cases I’ve tried them.
As the author mentions, there are also a number of cases where debuggers don’t work. Personally, I’m going to reach for the tool that always works vs. sometimes works.
Spread the good word!
So many problems can be solved with these.
And then there's some more specialized tooling depending on what you're doing that can be a huge help.
For SQL, the query planner and index hit/miss / full table scan.
And things like valgrind or similar for cache hit/miss.
Proper observability (spans/ traces) for APIs...
Knowing that the tools exist and how to use them can be the difference between software and great software.
Though system design / architecture is very important as well.
pdb is great for python, though.
1. Code where the granularity of state change is smaller than a function call. Sometimes you actually have to step through things one instruction at a time, and I'm lucky enough to have such problems to solve. You can't debug your assembly with printf(), basically[1a].
2. State changes that can't be easily isolated. Sometimes you want to log when something change but can't for the life of you figure out when it's changing. Debuggers have watchpoints.
But... that's really it. If I'm not hitting one of those I'm not reaching for the debugger. Logging is just faster, because you type it in right at the code you're already reading.
[1a] Though there's a caveat: sometimes you need to write assembly and don't even have anything like a printk. Bootstrap code for a new device is a blast. You just try stuff like writing one byte to a UART address or setting one GPIO pin as the first instructions and hope it works, then use that one bit of output to pull the rest up.
I once wrote a program that opened up all of my code, and at every single code curly brace, it added a macro call, and a guid.
I had to avoid doing that inside other macros, or inside Struct or Class definitions, enums, etc. But it wasn't hard, and it was a pretty sizeable codebase.The DEBUGVIKINGCODER macro, or whatever I called it, was a no-op in release. But in Debug or testing builds, would do something like:
(Using the right macros to append __LINE__ to the variable, so there's no collisions.)The constructor for DebugVikingCoder used a thread-local variable to write to a file (named after the thread id). It would write, essentially,
The destructor, when that scope was exited, would write to the same file: So when I'd run the program, I'd get a directory full of files, one per thread.Then I wrote another program that would read those all up, and would also read the code, and learn the File Name, Line Number of every GUID...
And, in Visual Studio, this tool program would print to the Output window, the File Name and Line Number, of every call and return.
And, in Visual Studio, you can step forward AND BACK in this Output window, and if you format it correctly, it'll open the file at that point, too.
So I could step forwards and backwards, through the code, to see who called where, etc. I could search in this Output window to jump to the function call I was looking for, and then walk backwards...
Then I added some code that would compare one run to another, and argued we could use that to figure out which of our automated tests formed a "basis set" to execute all of our code...
And to recommend which automated tests we should run, based on past analysis.
In addition to being able to time calls to functions, of course.
So then I added printing out some variables... And printing out lines in the middle of functions, when I wanted to time a section...
And if people respected the GUIDs, making a new one when they forked code, and leaving it alone if they moved code, we could have tracked how unit tests and other automation changed over time.
That got me really wishing that every new call scope really did have a GUID, in all the code we write... And I wished that it was essentially hidden from the developers, because who wants to see that? But, wow, it'd be nice if it was there.
I know there are debuggers that can go backwards and forwards in time... But I feel like being able to compare runs, over weeks and months, as the code is changing, is an under-appreciated objective.
In particular, REPL tools will work on remote session, on pre-production servers etc. _if_ the code base is organized in a somewhat modular way, it can be more pleasant than a debugger at times.
Makes me wonder if the state of debugging improved in PHP land. It was mostly unusable for batch process debugging, or when the server memory wasn't infinite, which is kinda the case most of the time for us mere mortals.
This isn't always the case. Maybe it's really hard in a lot of cases, but it's always not impossible.
Leave us be. We know what we’re doing.
I agree that adhoc dynamic expression evaluation at run time is very useful and can only be done in a debugger.
Kind of a hybrid of logging and standard debugging. "everything" is logged and you can go spelunk.
For example:
https://rr-project.org/
Some other things I'd add:
Some debuggers allow you to add actions. For example logging at the breakpoint is great if I can't modify the source, plus there's nothing to revert afterward. This just scratches the surface. Some debuggers allow you to see entire GPU workloads, view textures etc.
Debuggers are extremely useful for exploring and helping edit code. I can't be the only person that sprinkles breakpoints during development which helps me visualise code flow and quickly jump between source locations.
They're not just for debugging.
It won't run if I compile with debug info. I think it's due to a 3rd party proprietary library. So, to run the app I have to use release profile, with debug info stripped.
So, when I fire up gdb, I can't see any function information or anything, and it has so many system calls it's really difficult to follow through blindly.
So, what is the best way to handle this?
For really hairy bugs in programs that can't be stopped (kernel/drivers/realtime, etc) logging works.
And when it doesn't, like when you can't do I/O or switching of any kind, log non-blocking to a buffer that is dumped elsewhere.
also, related. It is harder than it should be to debug the linux kernel. Just getting a symboled stack trace is ridiculously hard.
It seems to me that this is one of the many phenomena where people want to judge and belittle their peers over something completely trivial.
Personally, I get the appeal of printing out debugging information, especially if some bug is rare and happens in unpredictable times (such as when you are sleeping). But the amount of info you get this way is necessarily lower than what can be gleaned from a debugger.