The article neglects one of my most important use cases for a debugger: figuring out how an undocumented piece of legacy code works. For example, if I perform a particular action in the UI, does function foo() get called? I find that setting breakpoints and tracing execution and variable changes in a debugger is an effective way of doing this kind of reverse engineering.
Also, when you're working with code that takes a long time to compile and link, using a debugger to check program state can be a lot quicker than recompiling the code with added print statements.
Different developers work with different types of code in different environments, so just because Linus Torvalds or Rob Pike does something one way doesn't mean that this is the most effective way for everyone else to do it.
Exactly. I seldom use it for my own code, can't remember the last time I did it but stepping through some huge undocumented Java codebase with Aspects oriented programming in it like Spring AOP (where the actual code that runs in that random function invocation is in some .xml file defined but not directly in the code) saved me a lot of time.
Also debuggers display contents of variables very nicely while stepping through code - for me a good way to verify my assumptions after reading a certain piece of code, just set a breakpoint and see if it's actually works that way - I'm often surprised.
Maybe it's not really required for C / low-level code but if you are wrangling complex Java codebases it's a very nice tool to have in the toolbox.
Many years ago, we were installing an ERM system, when we discovered that it would not run payroll--or rather, it would run payroll for every employee except for the two who were in a certain jurisdiction. The very expensive consultants who were overseeing the implementation did not have any useful ideas on how to resolve this. (Though they did have a useless and time-consuming one.) I figured out how to use the COBOL debugger (called an "animator"), and in a couple of hours located the problem. Perhaps if I knew how to write COBOL printfs, I could have done without the animator, but I was all but illiterate in COBOL.
I have since inherited the care of a WinForms system, and without the Visual Studio debugger I'd be lost there.
The article talks about live debugging & stepping through code, which I agree, I almost never do. But post-mortem debugging of coredumps is a different thing entirely.
In my job on the kernel team at Netflix's Open Connect CDN, I do post-mortem debugging of kernel core dumps almost daily. On a fleet the size of our CDN, there will invariably be a kernel panic which you'd not otherwise be able to reproduce. In fact, I often use 2 debuggers: kgdb for most things, and occasionally a port of the OpenSolaris mdb for easily scripting exploration of the core.
My hat is off to all the people who made my debugging so much easier. Eg, the llvm/clang folks who emit DWARF, gdb folks who make the FreeBSD kgdb debugger possible, and the Solaris folks who wrote the weird and wonderful mdb.
Yes to using debuggers for coredumps. I never found the whole UX of GDB good enough to displace my current print statement workflow, but for post analysis, it's really helpful when I can hit 'bt' (backtrace) and almost instantly see what went wrong.
> Doing “something else” means (1) rethinking your code so that it is easier to maintain or less buggy (2) adding smarter tests so that, in the future, bugs are readily identified effortlessly. Investing your time in this manner makes your code better in a lasting manner… whereas debugging your code line-by-line fixes one tiny problem without improving your process or your future diagnostics.
I've been a professional now for about 15 years and very, very rarely do I get to work on "my code". Almost all of the code I have to work with was written by someone else originally and I have to just modify the system for new requirements. Tests do not exist or if they do they are largely incomplete.
So the only thing to do is to step through find the problem, fix the ticket and move on.
Sure If I get to design the system I normally write it very simply / well structured with appropirate levels of abstraction and with enough tests to expose the bugs in my code. But very rarely do I get paid to work on my code, because my code doesn't need a lot of maintenance. I normally am asked to make changes to bad systems.
> Brian W. Kernighan and Rob Pike wrote that stepping through a program less productive than thinking harder and adding output statements and self-checking code at critical places. Kernighan once wrote that the most effective debugging tool is still careful thought, coupled with judiciously placed print statements.
Thinking harder when I have a project with millions of lines of code (this is normal in large financial systems) won't help me. A debugger will.
I think a lot of these famous programmers have never had to work with something terrible and probably never will and that is why they make such blasé statements.
I find this kind of thinking to be borderline insanity. Why would I expend so much mental energy trying to understand what the past dozen developers were thinking when it's trivial to just inspect values and deal with the reality of the system as it is right now. If a function is expected to return a value of 25 and it's returning 17, I don't need to reason about their code to fix it. My tactic is always to write some very tight unit tests around the sections of code closest to the problem and run them through a debugger so I can see where it goes awry.
This is my experience as well. Debuggers are extremely helpful when trying to integrate third party code with incomplete documentation. It's one thing to run a print statement at a specific level, but having the ability to explore an object and it's neighbouring objects can immediately provide insight instead of trying to find the function in the source code and deciphering it.
This is how the system ends up as a maintainable mess in the first place, where the fix for the aforementioned bug was to +8 and move on, which causes (at a later date) some other section to break because instead of 25, it's returning 33 so the developer adds a -8 and moves on...
I assume you meant an unmaintainable mess, and not a maintainable mess, but nevertheless, debuggers are not the cause.
You get messy code at least as quickly without a debugger, possibly even faster since bug fixes in my experience tend to be even more shallow, in many teams.
The values of the management, the values of the team, and resource availability is what matters in regard to writing, and keeping code maintainable.
I've worked with people who have had a similar idea, that using debuggers are somehow "bad", I have also had to rescue them by using a debugger to understand issues with code they had struggled to figure out for days, or even weeks, meanwhile causing severe issues for clients.
When you claim tools as the cause for your mistakes, then it could certainly be the case that the tool is of bad quality, which clearly is not the claimed issue with debuggers in the article. More likely though is that you haven't really learned when, and how, and why to use the tool.
Debuggers are brilliant for validating if you mental model of the code is correct, or more commonly, to understand how it's wrong. This can have a side effect of solving bugs, especially bugs in your own thinking
I don't think what you are doing is vastly different than what is described. "Thinking hard" is about trying to isolate the problem. RMS famously said that you should try to only debug the code that is broken rather than trying to debug the code that isn't broken (this is especially true in big systems). Where is the error happening? How are these things connected? Etc, etc. When you write very tight unit tests around the sections of the code, you have to do exactly the same thing (and I would argue that writing those tests is a good tool for "getting into" the code).
Next: running a debugger. I use print statements, but seriously, what's the difference? You use watch points in the debugger. Same thing. Once I have tests I find it easier to run them and look at the output of my prints as opposed to stepping through the code. Stepping through the code requires you to remember what you have done before and what the output was (granted debuggers that allow you to go backwards are helpful). If you can do that, then it's all good, but I find that it's easier for me to essentially create a log and read through it. It's just a bit more structured, but in the end it's exactly the same thing.
I think the biggest mistake that less experienced programmers do is that they don't try to reason about the code before they start. It's that isolation that's key, not how you are displaying the state of the program. Single stepping is fine when you've got 100 lines of code, but when you have thousands or millions of lines of code, it's not going to work. You need to be able to work your way backward from the error, reasoning using the source code as a map. You then use debugging tools (or printfs) to narrow down your options.
> I use print statements, but seriously, what's the difference? You use watch points in the debugger. Same thing. Once I have tests I find it easier to run them and look at the output of my prints as opposed to stepping through the code.
Maybe I have been spoiled by Visual Studio but as you said there are plenty of options in the debugger for winding back execution, changing execution, inspecting Objects, inspecting the state of the stack at the time, I can debug other people's assemblies, I can debug machines remotely.
Writing out to the console / log is my last resort.
> Stepping through the code requires you to remember what you have done before and what the output was (granted debuggers that allow you to go backwards are helpful).
No it doesn't require you to remember what you have done before. Even relatively basic debuggers such as the JavaScript debuggers in most browsers have a stack trace with what has been called where.
>If you can do that, then it's all good, but I find that it's easier for me to essentially create a log and read through it. It's just a bit more structured, but in the end it's exactly the same thing.
I don't understand why you would claim an inferior tool is better when a far superior one is available. It would like saying that a Impact Wrench / Spanner and a Spanner are the same thing, technically they both undo bolts, however one makes it far easier than the other.
> "Thinking hard" is about trying to isolate the problem. RMS famously said that you should try to only debug the code that is broken rather than trying to debug the code that isn't broken (this is especially true in big systems). Where is the error happening? How are these things connected? Etc, etc. When you write very tight unit tests around the sections of the code, you have to do exactly the same thing (and I would argue that writing those tests is a good tool for "getting into" the code).
Except it doesn't help you find the code. With a debugger I can stick in some breakpoints where I think the code is going to get hit and then walk myself back from there on what is called and in what order. I work with some terrible systems where it isn't obvious how the code is even executed because "senior" developers have abused IoC / DI, Reflection and Delegates in such a way to obscure what it is actually doing.
As for writing Unit-tests around sections of code. In quite a few areas I have been barred from doing it (for political reasons) and other times you realistically can't because your test setup would be just too complicated.
> Single stepping is fine when you've got 100 lines of code, but when you have thousands or millions of lines of code, it's not going to work
Yes it does work. I did it on Friday. I stick a break point on the entry point (in this case an ASP.NET MVC controller) and F10 and F11 my way down through the stack. I learned more doing this for 30 minutes then I did the previous 2 hours of looking at how the project was structured.
This "think harder" is akin to telling a man to dig harder with a shovel when they have a JCB excavator sitting round the corner.
I think it's a fantasy that if it were only "my code" it would be perfect and simple. Most of the worst code I've worked on was my code, after it had grown and adapted to new requirements and after I'd forgotten why I did things the way I did them. Tests are necessary but far from sufficient for making a system comprehensible.
> I think a lot of these famous programmers have never had to work with something terrible and probably never will and that is why they make such blasé statements.
Do you think they never had to work with something terrible, or when faced with something terrible, they took the time to make it not terrible? The track record for the software these two have built speaks for itself.
He cite Linus, who says that not using debugger result in longer time to market. which is good for Linux in his mind. How is this convincing for me?
Anyway, imagine a detective trying to figure out a murder scene without going through it step by step. Of-curse with experience come speed. So, weird flex but OK. I will still prefer C# over any language just because the debugger in Visual Studio is amazing!
After re-reading the article, I see it as another "Just code better" which hopefully will make it easy to pinpoint the bug just from the problem itself. In complex system with a lot of feedback loops, It's nearly impossible without debugging or using logs (which for me are just serialized debugger).
I use a debugger all the time. Why guess about any of this stuff, and why trust your fallible intuition, when you could have the computer tell you exactly what's really happening? I think of it as analogous to using a profiler in this respect.
Debuggers a great tool to have in your toolbelt. They can arbitrarily modify the state of the program at any point, let you safely inject code (I’ve written “coroutines” in LLDB for applications I did not have the ability to insert print statements in), and can be extremely helpful when performing dynamic analysis. That being said, sometimes you don’t have access to a debugger, so you have to do your best with the tools you have available. Finding a bug is always a challenge of removing extraneous state you don’t care about (“where do I put this print statement so it doesn’t get called a million times”, “how can I visualize the value of this variable when it changes in a way that is important”) so not having a debugger doesn’t change this: it just makes it somewhat more annoying (and requiring more ingenuity) to perform these tasks because you now have more limitations.
I think print statement makes it quicker and obvious to expose the internal state I want to observe at the precise point I want, thus fulfilling the hypothesis cycle faster. Debugger, on the other hand, seems effort taking to set up and expose too much additional details, it becomes overwhelming very quickly.
Debuggers are print statements with a slightly different “effort profile”. Adding the equivalent print statement is a little bit more annoying, but you can do this at any point during runtime and also drop down into the full debugger when you need it.
When something goes wrong after X iterations, print statements can show the evolution better. That is one thing that debuggers don't do I think (track variables over time). Or do they and am I not finding this feature in IntelliJ? :O
Another way to think of this is that the debugger is a tool, just like other familiar tools such as tcpdump, ping, traceroute, nslookup, nc (netcat) etc. Choosing the right tool for the right observations to make progress in squashing said bug(s) is key here. Sometimes, all you really need is a methodical approach in putting print statements in the right lines of code, or even pinging a host that doesn't seem like it's up in the first place.
I use debuggers (GDB) exclusively in batch mode. It allows for a very methodical debugging approach:
1 Formulate a Hypothesis
2 Determine what data you need for verification
3 Instrument code using gdb break commands, to hook function calls, sys-calls, signals, state-changes, etc. and print debugging information (variables, stack traces, timings) to a log file.
4 Then run an automated or manual test, of the failure you are debugging.
5 Then STOP data collection. Kill the process. Shut down the debugger.
6 Perform forensics on the log file.
7 Validate/Discard the hypothesis.
With this allows you reason reliably about the computational process:
- Why did I see this log line before this one?
- Why was this variable NULL here, but not in here.
You don't need to do that while you are in debugging session. You can take your time. You can seek back and forth in the file.
It's like dissecting a dead animal, not chasing a fly.
In addition, you can share concise information with your colleagues:
- This is the instrumentation
- This is the action I performed
- This is the output
And ask concise questions, that people might be able to answer:
> I expected XYZ in long line 25 to be ABC, but it was FGH. Why is this?
I find this to be completely unbelievable. If I'm in IntelliJ or Visual Studio, then adding breakpoints is equivalent, but easier than adding print statements. It's trivial to click the gutter on the line I'm suspicious of and run the program in debug mode. I can see all the local vars (exactly what I'd do with a print statement) and if something is amiss, easily go back up the call stack to see what went wrong. I find it mind-blowing when developers don't do this because it's so easy and so valuable. It's just so much harder to do with functional languages or languages without a first-rate IDE which seem to be what people like these days. In fact, I find it mind-blowing when developers don't consider this a mandatory feature to need when choosing a language.
Furthermore, you can even edit the values in place, to confirm or reject your hypothesis about a bug. And a debugger can often reach into framework or library code as well, saving the need to go and check out and rebuild dependencies.
Visual Studio's debugger is especially magical with .NET. With the debugger paused on a breakpoint you can drag the "next statement" pointer backwards to rerun pieces of your code. You can also edit the code while paused and your changes will be reloaded on the fly when you continue execution.
A lot of this is appeal to authority. It seems to be a contribution to a growing sort hip- or leet-ness around not using debuggers. Implicit in this seems to be: I don't need a debugger, neither do these famous people, why do you? Aren't you tough enough or smart enough like me and these famous people? You're clearly doing this programming thing wrong.
Well, I use debuggers. I think they're great tools. My feeling is that I'm very happy to use any tool that helps me create, understand, and improve software. When people tell me that a tool that is useful to me in that endeavor is not actually useful, all I can think to do is roll my eyes.
Having said that, something I am very interested in is learning new approaches to interrogate software complexity and solve problems. So, "here are some approaches to understanding and debugging code that have worked for me" from someone who doesn't use debuggers would be interesting to me. But I actually don't see any of that here.
I agree. The post was full of appealing to authority that gives me disgust. And I do not like and use debugger.
I am not going to appeal to authority, so I'll describe how I debug. 30% of my debugging is with pencil and scratch paper. 30% of my debugging is with print. And 30% of my debugging is with refactoring. The last 10% of debugging is with profiler. I do use gdb, mostly in scripting mode. I also do breakpoints manually, that goes into less than 1% and get rounded off.
I program with a meta-layer, so that I can write `$print "string interpolated with variables" in a semi-universal syntax and works for all languages that I work with. The meta layer does the parsing and translation. More importantly, the meta-layer organizes the debugging part so it can easily turn-on-off and not to clutter the actual code. Without that, I, too, cannot imagine debug with print will work.
By refactoring, I mean moving the part of code out of the way, or temporary removing or rewrite part of code that I am not sure. This is again, only possible with a meta layer that allows moving parts of code and re-organizing the code without necessarily changing the code. I cannot imaging how one can truly understand complex code without actually probing the code parts by parts.
Then I have seen the new generation of people who grow up with electronic devices and are more used to mouse than pencil. Electronic device can never match the ease and disposable nature of paper and pencil, so I cannot imagine how one could debug by merely navigating the code with an editor.
So while I do not use debugger for debugging, I can perfectly understand how one cannot live without debugger.
>I agree. The post was full of appealing to authority that gives me disgust. And I do not like and use debugger.
Appeals to authority as opposed to what alternative? Untested arguments? Anecdotal personal stories ("I am not going to appeal to authority, so I'll describe how I debug")?
(See my answer elsewhere in the thread for why almost everything is "appeal to authority" or worse in IT anyway).
I guess we are implying different objectives here. To convince someone, appeal to authority might be quick effective. But I don't care about convince you or anyone. I care about reasoning. So I show you how I debug, and I say something about my reasoning behind it, and I also think I understand how someone would not work and think the same way as I do. What results I am seeking? If you find there is flaw in my reasoning, blind spots in my thinking, point/comment back to me, it is an opportunity for me to improve. If you find my reasoning makes sense, but the conclusion does not, feedback to me and I may also improve. If you fully agree with me, I may get a boost in my ego. The last one is in my instinct, but I am not sure I really want that.
I will, however, most likely argue back for the potential feedback. Only through arguing back and forth, I truly can understand the feedback and morph into a form that I can use to improve. I often do improve my reasoning even there is no conclusion for the argument yet. It is easy to gain when background and reasoning are provided. So I always provide them, hopefully who ever participates also gain.
I often use a debugger for code with complex control flow and data structures, where you almost inevitably start high level (e.g., Postgres optimizer). I rarely use a debugger when I can fit everything in my head immediately, because the problem is well-scoped, and involves data structures I already understand.
I can quite easily imagine a person that only ever gets to work on problems in the latter category being dismissive of debuggers.
Debugger use is based on the same. Few things in development are not based on idiosyncratic preferences, fads, snake oil salesmen, tradition, or appeals to authority (and those are the math parts of CS). Of the 5 above categories tradition is probably the best and most "scientific" (at least it has stood the test of time).
There is little actual research on such issues (best practices, programming ergonomics, syntax leading to less errors, bug counts per style, etc), and even what little research there is usually flawed, with small samples, and non-reproducible (not that many teams bothered to reproduce it in the first place).
No authority told me to use a debugger, I use them because they work for me. They save me time, help me make money, and make it easier and more fun to code.
If I ever saw a "research study" that said "print statement users have 12% fewer bugs than debugger users" - or vice versa! - I would dismiss it out of hand. What possible relevance could it have to my work? That's not real research, it's just playing with statistics and making up overly generalized stories about them.
That kind of research would just become the "authority" in a new appeal to authority, complete with an infographic!
And nothing has to be "settled", nor should it be. There are many different kinds of programming that require different tools. It's best to be aware of the variety of choices available, and choose your tools according to the particular situation you're in.
>No authority told me to use a debugger, I use them because they work for me. They save me time, help me make money, and make it easier and more fun to code.
Well, I already wrote: "Few things in development are not based on idiosyncratic preferences, fads, snake oil salesmen, tradition, or appeals to authority". So your case would fall under the first option: it's an unscientifically tested personal preference.
>If I ever saw a "research study" that said "print statement users have 12% fewer bugs than debugger users" - or vice versa! - I would dismiss it out of hand. What possible relevance could it have to my work?
It could have all the relevance in the world.
It's not like if you believe strong enough that your preferred methods are the best (or the "best for you"), that you can't still be using an objectively worse method and get worse results...
>And nothing has to be "settled", nor should it be.
Well, if software development wants to be like an engineering discipline, many things should be (studied and eventually settled).
"Works for me" and letting developers improvise and come with their own hodgepodge of practices, preferences, and cargo cults, is how we got into this mess.
I specifically meant that the article was largely an appeal to authority, not that the practice of not using debuggers is itself based on appeal to authority.
Having said that, I find your argument here pretty weird. I use debuggers because they are useful to me, not because I of quotes about them from famous programmers.
>I use debuggers because they are useful to me, not because I of quotes about them from famous programmers.
Which is an equally fallacious reason to use them (scientifically wise) as appeal to authority.
"Useful to me" doesn't mean anything by itself. Something else could have been more useful to you, but without measuring it appropriately (just some personal anecdotes with trying some thing and then another and seeing which you prefer or which seems to you to give you better results, with no real measuring system or rigor), you'd never know. And the industry would never know which direction to invest, and how young programmers should be taught best -- not just about debuggers but about almost everything.
You're making a broader point, which is fine and all. But your comments would be just as relevant if I said, "I pull out shared code into functions because that technique is useful to me". Your point is so broad that it's like a tautology. I really don't lose any sleep over the foundations of my approach to my work being unsupported by rigorous studies. You can feel free to be a science maximalist and run studies on whether pulling code out into functions is actually useful and all the rest of the approaches we've developed over the years. Meanwhile, I'll happily carry on doing stuff that I know is useful to me, like pulling out functions and using debuggers.
It's a simple complaint that debuggers do not scale.
There is a very outspoken group that claims that if you are not using a debugger, you don't know how to write code. Appeals to authority is exactly what is required to undo the damage those people create.
Sounds like a theologist who wouldn't use a microscope, and instead think over/interpret harder their Holy-book-of-choice. You can get into interesting insights along the way, but I wouldn't call it practical or efficient.
Debuggers are the most amazing tools to explore and learn the code in your hands. They do have limitations, as pointed out, but presuming that your inner insight/gut feeling leads to more scalable and lasting results is ridiculous.
Bart Locanthi wrote the debugger for the BLIT (Bell Labs Intelligent Terminal) and named it joff because most of the time you’re in a debugger you’re just ....
I write scientific simulations in Matlab.
Debugging, that is, stopping the programs and interacting with the data and functions, is essential to me.
To a point, this is due to me not planning through my programs. So I can see that for some areas a debugger may not be essential.
But in other cases, I am actually interested to trace what happens with data in my mechanisms. For this kind of work, a debugger is essential.
Most importantly, as someone who maybe does not use the absolute best practices of designing software, debugging allows me to write solid and successful programs without having completed a CS degree.
The end of the essay implies that "debugger" is a poor name because of course you still have to fix the bug yourself. If we call it instead "bug-location-explorer" I find them invaluable for certain problems. In particular when you have a very complex system that takes a while to get into its fragile state, when I/O is difficult (e.g. many embedded systems), or as others have noted here when you are spelunking others' code. Breakpoints are crucial since most bugs with modern languages are wrong output and not bus errors.
I started out as a user of "print" as a debugging technique, but early on Bill Gosper took pity on me and introduced me to ITS DDT and the Lispm debugger. They both had an important property that they were always running, so when your program fails you can immediately inspect the state of open files and network connections. No automatic core dumps except for daemons. The fact that you explicitly have to attach a debugger before starting the program is a regression in Unix IMHO.
It doesn't surprise me that Linus doesn't use one as kernel debugging is its own can of worms.
I don't know if it's always been the case, but it is certainly possible to attach and detach a debugger from an existing process, gdb/lldb takes -p for pid. (Although, detaching from threaded processes seems to have a risk of the process ending up stuck in my experience)
Briefly popping into the debugger can be a really quick way to find where an infinite loop is happening, especially if it's not in your code, or not where you expect. I found this especially helpful in diagnosing a FreeBSD kernel loop I ran into, once the location of the loop was clear, the fix was simple.
As you can see we have identified the crash in this process. Good have Steiner attach GBD to it to find out what is wrong. Mein Fuhrer... Steiner... Steiner accidentally killed the process and rebooted the server.
How about 'state inspector?' This is, of course, exactly what this article proposes doing with "print" statements, but print is a paltry state inspection tool compared to them, ahem, debuggers to which we are accustomed.
> Kernighan once wrote that the most effective debugging tool is still careful thought, coupled with judiciously placed print statements.
Of course the first tool is careful thought. But when forced to fall back to print or log statements it feels like a handicap.
If I have to use a print statement it means I'm not sure what's going on so I'm not sure what, exactly, to print. By breaking on that code I don't have to know exactly because I can execute arbitrary code in that scope. What takes multiple iterations with print is often just one with break.
I can compare directly, because on a production-only bug I'm forced to use log debug statements. And usually in that case I have the same code open locally in a debugger. The difference is night versus day.
Maybe it's like a chess master who really doesn't need the board. For the kid in Searching for Bobby Fischer, it may really be a distraction. But I notice that grandmasters use a chess board in serious competition. And as a programmer I'm no grandmaster, and I play this weird game better with the board in front of me.
As always with these kinds of dogamtic posts it ignores the real world: working on years old legacy software with millions of lines of code and off-the-charts cyclomatic-complexity.
I could show anyone of these authorities a situation where their bravado would fail a trying to figure out a particular bug by "staring at the code and thinking harder" would be impossible.
Besides - they all tend to concede that they will use print statements or whatever as a last resorted. That is just retarded - you have to add the statements, recompile the code, and eventually remove them again. Just use the goddamn debugger that's what it's there for.
What starts as a completely reasonable "hey maybe sometimes you should try to read the code and not let the debugger become a crutch" turns into a sensationalist black/white statement "I NEVER USE A DEBUGGER". Grow up.
I’ve never understood the “I don’t need a debugger - print statements are fine” argument. Even if you only use a debugger to view application state, (which, for most debuggers, is but a tiny fraction of their capability) it’s still an order of magnitude more convenient than print statements.
No need to modify code, and no risk of forgetting to remove a print statement (I have seen a forgotten dump on a rare execution path leak sensitive data to users first hand). Not to mention the cases where printing can change or break the program in some cases; for example printing before sending headers in something like PHP.
Smarter people have expressed more ill considered opinions.
"In the 1950s von Neumann was employed as a consultant to IBM to review proposed and ongoing advanced technology projects. One day a week, von Neumann "held court" at 590 Madison Avenue, New York. On one of these occasions in 1954 he was confronted with the Fortran concept; John Backus remembered von Neumann being unimpressed and that he asked, "Why would you want more than machine language?" Frank Beckman, who was also present, recalled that von Neumann dismissed the whole development as "but an application of the idea of Turing's 'short code."' Donald Gilles, one of von Neumann's students at Princeton, and later a faculty member at the University of Illinois, recalled that the graduate students were being "used" to hand-assemble programs into binary for their early machine (probably the IAS machine). He took time out to build an assembler, but when von Neumann found out about it he was very angry, saying (paraphrased), "It is a waste of a valuable scientific computing instrument to use it to do clerical work.
It's easy to see this as a wrong opinion when computers are extremely cheap and fast. Was von Neumann wrong about computers being more expensive compilation tools than grad students at the time?
The 1950s were a different time. Nowadays computing power is almost too cheap to meter. I do not think anyone at almost any software company knows how to answer the question "how much money are we spending on running our compilers". In most cases, I suspect the cost of running the compiler dominated by the salary of the programmer as he types "make" and waits for the compilation to finish.
We live in a world where every employee has a previously unimaginable amount of processing power dedicated for their personal use that spends almost all of its time idling. That results in a far different calculus than the world where processing power is a scarce resource.
Yet the most prevalent tests these days for SE are based around the old problems. Then again these tests are less about how good of an engineer you are and more of how submissive you are.
If an assembler or compiler could save you from having to run dozens of extra batch jobs to fix a bug in your hard-to-understand machine language code, it might have allowed more time on an expensive machine to be used for productive purposes.
I see mentions of gdb - an extremely cumbersome tool - no mention of Visual Studio and its marvelous debugger.
I too, when I'm on Linux, don't use a debugger, because there's no good debugger and adding a print statement is faster and easier, and figuring out what is going on with gdb is just plain horrible and slow.
That's why as soon as I find a bug on Linux I try to have it on Windows to leverage VS's debugger. I can't count the number of times where I could instantly spot and understand a bug thanks to VS's debugger.
"Think harder about the code", sure, and what if you didn't write the code?
I wrote 3Kloc of C++ for a dwarf fortress plugin while convalescing with very poor eyesight, and did not ever get around to figuring out how to start gdb, so crashes could involve alot of pondering or compiling in logging to track down simply where in the source it occurred.
A couple of months ago I finally had a go at using gdb to help fix a reported crash, and found that with a debug built executable, just type "gdb progname" and then at gdb's prompt "run" - hey presto if the program crashes it gives the source line. No problem that the entry executable wasnt built with symbols. It was hugely useful for a tiny learning step.
There is nothing marvelous about the VS debugger (it even had bugs on basic functionality up to very recently). It just presents an simpler interface because you are launching it from an IDE, get a C++ IDE on Linux and it will be as easy to launch.
But if you are looking for marvelous debuggers, I do recommend you look at the Python ecosystem.
I think we need to rethink what we mean by debugger. In Dark (https://darklang.com), we don't have a debugger. Instead we combine the editor with the language, and then when you're editing code you have available at all times a particular program trace (typically a real production value, which are automatically saved because dark also provides infra). As a result, you can immediately see the value of a variable in that trace.
Is this a debugger? Sure. It sorta lets you step through the program and inspect state at various points. But it's also a REPL, and it also strongly resembles inserting print statements everywhere. It's also like an exception tracker/crash reporter and a tracing framework.
IMO it's both simpler and more powerful than any of these. It's like if every statement had a println that you never have to add but is available whenever you want to inspect it. Or like a debugger where you never have to painfully step through the program to get to the state you want.
So overall, I think we need to think deeper about what a debugger is and how it can work. Most of the people quoted do not have a good debugger available to them, nor a good debugging workflow.
185 comments
[ 3.3 ms ] story [ 191 ms ] threadAlso, when you're working with code that takes a long time to compile and link, using a debugger to check program state can be a lot quicker than recompiling the code with added print statements.
Different developers work with different types of code in different environments, so just because Linus Torvalds or Rob Pike does something one way doesn't mean that this is the most effective way for everyone else to do it.
Also debuggers display contents of variables very nicely while stepping through code - for me a good way to verify my assumptions after reading a certain piece of code, just set a breakpoint and see if it's actually works that way - I'm often surprised.
Maybe it's not really required for C / low-level code but if you are wrangling complex Java codebases it's a very nice tool to have in the toolbox.
Many years ago, we were installing an ERM system, when we discovered that it would not run payroll--or rather, it would run payroll for every employee except for the two who were in a certain jurisdiction. The very expensive consultants who were overseeing the implementation did not have any useful ideas on how to resolve this. (Though they did have a useless and time-consuming one.) I figured out how to use the COBOL debugger (called an "animator"), and in a couple of hours located the problem. Perhaps if I knew how to write COBOL printfs, I could have done without the animator, but I was all but illiterate in COBOL.
I have since inherited the care of a WinForms system, and without the Visual Studio debugger I'd be lost there.
I
In my job on the kernel team at Netflix's Open Connect CDN, I do post-mortem debugging of kernel core dumps almost daily. On a fleet the size of our CDN, there will invariably be a kernel panic which you'd not otherwise be able to reproduce. In fact, I often use 2 debuggers: kgdb for most things, and occasionally a port of the OpenSolaris mdb for easily scripting exploration of the core.
My hat is off to all the people who made my debugging so much easier. Eg, the llvm/clang folks who emit DWARF, gdb folks who make the FreeBSD kgdb debugger possible, and the Solaris folks who wrote the weird and wonderful mdb.
I've been a professional now for about 15 years and very, very rarely do I get to work on "my code". Almost all of the code I have to work with was written by someone else originally and I have to just modify the system for new requirements. Tests do not exist or if they do they are largely incomplete.
So the only thing to do is to step through find the problem, fix the ticket and move on.
Sure If I get to design the system I normally write it very simply / well structured with appropirate levels of abstraction and with enough tests to expose the bugs in my code. But very rarely do I get paid to work on my code, because my code doesn't need a lot of maintenance. I normally am asked to make changes to bad systems.
> Brian W. Kernighan and Rob Pike wrote that stepping through a program less productive than thinking harder and adding output statements and self-checking code at critical places. Kernighan once wrote that the most effective debugging tool is still careful thought, coupled with judiciously placed print statements.
Thinking harder when I have a project with millions of lines of code (this is normal in large financial systems) won't help me. A debugger will.
I think a lot of these famous programmers have never had to work with something terrible and probably never will and that is why they make such blasé statements.
You get messy code at least as quickly without a debugger, possibly even faster since bug fixes in my experience tend to be even more shallow, in many teams.
The values of the management, the values of the team, and resource availability is what matters in regard to writing, and keeping code maintainable.
I've worked with people who have had a similar idea, that using debuggers are somehow "bad", I have also had to rescue them by using a debugger to understand issues with code they had struggled to figure out for days, or even weeks, meanwhile causing severe issues for clients.
When you claim tools as the cause for your mistakes, then it could certainly be the case that the tool is of bad quality, which clearly is not the claimed issue with debuggers in the article. More likely though is that you haven't really learned when, and how, and why to use the tool.
Debuggers are brilliant for validating if you mental model of the code is correct, or more commonly, to understand how it's wrong. This can have a side effect of solving bugs, especially bugs in your own thinking
Next: running a debugger. I use print statements, but seriously, what's the difference? You use watch points in the debugger. Same thing. Once I have tests I find it easier to run them and look at the output of my prints as opposed to stepping through the code. Stepping through the code requires you to remember what you have done before and what the output was (granted debuggers that allow you to go backwards are helpful). If you can do that, then it's all good, but I find that it's easier for me to essentially create a log and read through it. It's just a bit more structured, but in the end it's exactly the same thing.
I think the biggest mistake that less experienced programmers do is that they don't try to reason about the code before they start. It's that isolation that's key, not how you are displaying the state of the program. Single stepping is fine when you've got 100 lines of code, but when you have thousands or millions of lines of code, it's not going to work. You need to be able to work your way backward from the error, reasoning using the source code as a map. You then use debugging tools (or printfs) to narrow down your options.
Maybe I have been spoiled by Visual Studio but as you said there are plenty of options in the debugger for winding back execution, changing execution, inspecting Objects, inspecting the state of the stack at the time, I can debug other people's assemblies, I can debug machines remotely.
Writing out to the console / log is my last resort.
> Stepping through the code requires you to remember what you have done before and what the output was (granted debuggers that allow you to go backwards are helpful).
No it doesn't require you to remember what you have done before. Even relatively basic debuggers such as the JavaScript debuggers in most browsers have a stack trace with what has been called where.
>If you can do that, then it's all good, but I find that it's easier for me to essentially create a log and read through it. It's just a bit more structured, but in the end it's exactly the same thing.
I don't understand why you would claim an inferior tool is better when a far superior one is available. It would like saying that a Impact Wrench / Spanner and a Spanner are the same thing, technically they both undo bolts, however one makes it far easier than the other.
Except it doesn't help you find the code. With a debugger I can stick in some breakpoints where I think the code is going to get hit and then walk myself back from there on what is called and in what order. I work with some terrible systems where it isn't obvious how the code is even executed because "senior" developers have abused IoC / DI, Reflection and Delegates in such a way to obscure what it is actually doing.
As for writing Unit-tests around sections of code. In quite a few areas I have been barred from doing it (for political reasons) and other times you realistically can't because your test setup would be just too complicated.
> Single stepping is fine when you've got 100 lines of code, but when you have thousands or millions of lines of code, it's not going to work
Yes it does work. I did it on Friday. I stick a break point on the entry point (in this case an ASP.NET MVC controller) and F10 and F11 my way down through the stack. I learned more doing this for 30 minutes then I did the previous 2 hours of looking at how the project was structured.
This "think harder" is akin to telling a man to dig harder with a shovel when they have a JCB excavator sitting round the corner.
If I was writing code in c++ I would probably create something awful because I wouldn’t know what I would be doing.
Do you think they never had to work with something terrible, or when faced with something terrible, they took the time to make it not terrible? The track record for the software these two have built speaks for itself.
Anyway, imagine a detective trying to figure out a murder scene without going through it step by step. Of-curse with experience come speed. So, weird flex but OK. I will still prefer C# over any language just because the debugger in Visual Studio is amazing!
After re-reading the article, I see it as another "Just code better" which hopefully will make it easy to pinpoint the bug just from the problem itself. In complex system with a lot of feedback loops, It's nearly impossible without debugging or using logs (which for me are just serialized debugger).
1 Formulate a Hypothesis
2 Determine what data you need for verification
3 Instrument code using gdb break commands, to hook function calls, sys-calls, signals, state-changes, etc. and print debugging information (variables, stack traces, timings) to a log file.
4 Then run an automated or manual test, of the failure you are debugging.
5 Then STOP data collection. Kill the process. Shut down the debugger.
6 Perform forensics on the log file.
7 Validate/Discard the hypothesis.
With this allows you reason reliably about the computational process:
- Why did I see this log line before this one?
- Why was this variable NULL here, but not in here.
You don't need to do that while you are in debugging session. You can take your time. You can seek back and forth in the file. It's like dissecting a dead animal, not chasing a fly.
In addition, you can share concise information with your colleagues:
- This is the instrumentation
- This is the action I performed
- This is the output
And ask concise questions, that people might be able to answer:
> I expected XYZ in long line 25 to be ABC, but it was FGH. Why is this?
Works also in C++, though not very well when the debugger stops on an exception.
Well, I use debuggers. I think they're great tools. My feeling is that I'm very happy to use any tool that helps me create, understand, and improve software. When people tell me that a tool that is useful to me in that endeavor is not actually useful, all I can think to do is roll my eyes.
Having said that, something I am very interested in is learning new approaches to interrogate software complexity and solve problems. So, "here are some approaches to understanding and debugging code that have worked for me" from someone who doesn't use debuggers would be interesting to me. But I actually don't see any of that here.
I am not going to appeal to authority, so I'll describe how I debug. 30% of my debugging is with pencil and scratch paper. 30% of my debugging is with print. And 30% of my debugging is with refactoring. The last 10% of debugging is with profiler. I do use gdb, mostly in scripting mode. I also do breakpoints manually, that goes into less than 1% and get rounded off.
I program with a meta-layer, so that I can write `$print "string interpolated with variables" in a semi-universal syntax and works for all languages that I work with. The meta layer does the parsing and translation. More importantly, the meta-layer organizes the debugging part so it can easily turn-on-off and not to clutter the actual code. Without that, I, too, cannot imagine debug with print will work.
By refactoring, I mean moving the part of code out of the way, or temporary removing or rewrite part of code that I am not sure. This is again, only possible with a meta layer that allows moving parts of code and re-organizing the code without necessarily changing the code. I cannot imaging how one can truly understand complex code without actually probing the code parts by parts.
Then I have seen the new generation of people who grow up with electronic devices and are more used to mouse than pencil. Electronic device can never match the ease and disposable nature of paper and pencil, so I cannot imagine how one could debug by merely navigating the code with an editor.
So while I do not use debugger for debugging, I can perfectly understand how one cannot live without debugger.
Appeals to authority as opposed to what alternative? Untested arguments? Anecdotal personal stories ("I am not going to appeal to authority, so I'll describe how I debug")?
(See my answer elsewhere in the thread for why almost everything is "appeal to authority" or worse in IT anyway).
I guess we are implying different objectives here. To convince someone, appeal to authority might be quick effective. But I don't care about convince you or anyone. I care about reasoning. So I show you how I debug, and I say something about my reasoning behind it, and I also think I understand how someone would not work and think the same way as I do. What results I am seeking? If you find there is flaw in my reasoning, blind spots in my thinking, point/comment back to me, it is an opportunity for me to improve. If you find my reasoning makes sense, but the conclusion does not, feedback to me and I may also improve. If you fully agree with me, I may get a boost in my ego. The last one is in my instinct, but I am not sure I really want that.
I will, however, most likely argue back for the potential feedback. Only through arguing back and forth, I truly can understand the feedback and morph into a form that I can use to improve. I often do improve my reasoning even there is no conclusion for the argument yet. It is easy to gain when background and reasoning are provided. So I always provide them, hopefully who ever participates also gain.
I can quite easily imagine a person that only ever gets to work on problems in the latter category being dismissive of debuggers.
Debugger use is based on the same. Few things in development are not based on idiosyncratic preferences, fads, snake oil salesmen, tradition, or appeals to authority (and those are the math parts of CS). Of the 5 above categories tradition is probably the best and most "scientific" (at least it has stood the test of time).
There is little actual research on such issues (best practices, programming ergonomics, syntax leading to less errors, bug counts per style, etc), and even what little research there is usually flawed, with small samples, and non-reproducible (not that many teams bothered to reproduce it in the first place).
That's why almost nothing is ever settled.
If I ever saw a "research study" that said "print statement users have 12% fewer bugs than debugger users" - or vice versa! - I would dismiss it out of hand. What possible relevance could it have to my work? That's not real research, it's just playing with statistics and making up overly generalized stories about them.
That kind of research would just become the "authority" in a new appeal to authority, complete with an infographic!
And nothing has to be "settled", nor should it be. There are many different kinds of programming that require different tools. It's best to be aware of the variety of choices available, and choose your tools according to the particular situation you're in.
Well, I already wrote: "Few things in development are not based on idiosyncratic preferences, fads, snake oil salesmen, tradition, or appeals to authority". So your case would fall under the first option: it's an unscientifically tested personal preference.
>If I ever saw a "research study" that said "print statement users have 12% fewer bugs than debugger users" - or vice versa! - I would dismiss it out of hand. What possible relevance could it have to my work?
It could have all the relevance in the world.
It's not like if you believe strong enough that your preferred methods are the best (or the "best for you"), that you can't still be using an objectively worse method and get worse results...
>And nothing has to be "settled", nor should it be.
Well, if software development wants to be like an engineering discipline, many things should be (studied and eventually settled).
"Works for me" and letting developers improvise and come with their own hodgepodge of practices, preferences, and cargo cults, is how we got into this mess.
Having said that, I find your argument here pretty weird. I use debuggers because they are useful to me, not because I of quotes about them from famous programmers.
Which is an equally fallacious reason to use them (scientifically wise) as appeal to authority.
"Useful to me" doesn't mean anything by itself. Something else could have been more useful to you, but without measuring it appropriately (just some personal anecdotes with trying some thing and then another and seeing which you prefer or which seems to you to give you better results, with no real measuring system or rigor), you'd never know. And the industry would never know which direction to invest, and how young programmers should be taught best -- not just about debuggers but about almost everything.
There is a very outspoken group that claims that if you are not using a debugger, you don't know how to write code. Appeals to authority is exactly what is required to undo the damage those people create.
I have never seen that
Debuggers are the most amazing tools to explore and learn the code in your hands. They do have limitations, as pointed out, but presuming that your inner insight/gut feeling leads to more scalable and lasting results is ridiculous.
We "control" our code when tests prove it does what we think it should.
Debuggers are rarely-usable, inefficient tools for software development. I agree with OP, debuggers don't scale.
But when there is a bug in code I wrote, I can reason about it and place prints where I need them. For side projects I hardly ever use a debugger.
To a point, this is due to me not planning through my programs. So I can see that for some areas a debugger may not be essential.
But in other cases, I am actually interested to trace what happens with data in my mechanisms. For this kind of work, a debugger is essential.
Most importantly, as someone who maybe does not use the absolute best practices of designing software, debugging allows me to write solid and successful programs without having completed a CS degree.
I started out as a user of "print" as a debugging technique, but early on Bill Gosper took pity on me and introduced me to ITS DDT and the Lispm debugger. They both had an important property that they were always running, so when your program fails you can immediately inspect the state of open files and network connections. No automatic core dumps except for daemons. The fact that you explicitly have to attach a debugger before starting the program is a regression in Unix IMHO.
It doesn't surprise me that Linus doesn't use one as kernel debugging is its own can of worms.
Briefly popping into the debugger can be a really quick way to find where an infinite loop is happening, especially if it's not in your code, or not where you expect. I found this especially helpful in diagnosing a FreeBSD kernel loop I ran into, once the location of the loop was clear, the fix was simple.
Not sure if this is exactly true for Unix. For embedded targets you can often attach to a running uP with GDB.
Or even more generally, a "run-time code explorer", since debuggers are useful for understanding code even if you're not trying to fix a bug.
Of course the first tool is careful thought. But when forced to fall back to print or log statements it feels like a handicap.
If I have to use a print statement it means I'm not sure what's going on so I'm not sure what, exactly, to print. By breaking on that code I don't have to know exactly because I can execute arbitrary code in that scope. What takes multiple iterations with print is often just one with break.
I can compare directly, because on a production-only bug I'm forced to use log debug statements. And usually in that case I have the same code open locally in a debugger. The difference is night versus day.
Maybe it's like a chess master who really doesn't need the board. For the kid in Searching for Bobby Fischer, it may really be a distraction. But I notice that grandmasters use a chess board in serious competition. And as a programmer I'm no grandmaster, and I play this weird game better with the board in front of me.
I could show anyone of these authorities a situation where their bravado would fail a trying to figure out a particular bug by "staring at the code and thinking harder" would be impossible.
Besides - they all tend to concede that they will use print statements or whatever as a last resorted. That is just retarded - you have to add the statements, recompile the code, and eventually remove them again. Just use the goddamn debugger that's what it's there for.
What starts as a completely reasonable "hey maybe sometimes you should try to read the code and not let the debugger become a crutch" turns into a sensationalist black/white statement "I NEVER USE A DEBUGGER". Grow up.
No need to modify code, and no risk of forgetting to remove a print statement (I have seen a forgotten dump on a rare execution path leak sensitive data to users first hand). Not to mention the cases where printing can change or break the program in some cases; for example printing before sending headers in something like PHP.
> there are cases where a debugger is the right tool
"In the 1950s von Neumann was employed as a consultant to IBM to review proposed and ongoing advanced technology projects. One day a week, von Neumann "held court" at 590 Madison Avenue, New York. On one of these occasions in 1954 he was confronted with the Fortran concept; John Backus remembered von Neumann being unimpressed and that he asked, "Why would you want more than machine language?" Frank Beckman, who was also present, recalled that von Neumann dismissed the whole development as "but an application of the idea of Turing's 'short code."' Donald Gilles, one of von Neumann's students at Princeton, and later a faculty member at the University of Illinois, recalled that the graduate students were being "used" to hand-assemble programs into binary for their early machine (probably the IAS machine). He took time out to build an assembler, but when von Neumann found out about it he was very angry, saying (paraphrased), "It is a waste of a valuable scientific computing instrument to use it to do clerical work.
"https://history.computer.org/pioneers/von-neumann.html
We live in a world where every employee has a previously unimaginable amount of processing power dedicated for their personal use that spends almost all of its time idling. That results in a far different calculus than the world where processing power is a scarce resource.
See the friendship between cat and squirrel That's so cute see how much they are taking care of each others http://bit.ly/2VdHs4E
Friendship between cat and dog see in video how much they are teasing each others also they are loving http://bit.ly/2WoRU66
I too, when I'm on Linux, don't use a debugger, because there's no good debugger and adding a print statement is faster and easier, and figuring out what is going on with gdb is just plain horrible and slow.
That's why as soon as I find a bug on Linux I try to have it on Windows to leverage VS's debugger. I can't count the number of times where I could instantly spot and understand a bug thanks to VS's debugger.
"Think harder about the code", sure, and what if you didn't write the code?
A couple of months ago I finally had a go at using gdb to help fix a reported crash, and found that with a debug built executable, just type "gdb progname" and then at gdb's prompt "run" - hey presto if the program crashes it gives the source line. No problem that the entry executable wasnt built with symbols. It was hugely useful for a tiny learning step.
But if you are looking for marvelous debuggers, I do recommend you look at the Python ecosystem.
Is this a debugger? Sure. It sorta lets you step through the program and inspect state at various points. But it's also a REPL, and it also strongly resembles inserting print statements everywhere. It's also like an exception tracker/crash reporter and a tracing framework.
IMO it's both simpler and more powerful than any of these. It's like if every statement had a println that you never have to add but is available whenever you want to inspect it. Or like a debugger where you never have to painfully step through the program to get to the state you want.
So overall, I think we need to think deeper about what a debugger is and how it can work. Most of the people quoted do not have a good debugger available to them, nor a good debugging workflow.