The background of paper with lines is very misleading - because it implies that translation of the pyramid is forbidden, when this translation is the key.
It's a good insight for any transformation really, including change in code: look at what's in common between the desired outcome and the current situation to have an idea how how much work to do.
>Rob Pike of golang fame noted that while he reached for the debugger first thing, Ken Thompson would ignore what Rob was doing and just stand and think. Then after a while he'd say: "I know what's wrong" and very often be right.
I feel like this is not a helpful take on debugging for most of us. I'm certainly not as good at this as Ken Thompson, and very seldom is the problem space simple enough that I can hold a complete mental model of it in my head. Instead I reach for the debugger to collect more data on what's going on. And I can usually find the issue by looking a lot faster than I can correctly theorize about it.
In Debugging by Davig J. Agans, his third rule is "Quit thinking and look". And I tend to agree. In the majority of cases when I'm debugging a problem the solution only reveals itself when I see that missing piece of information.
EDIT:
As others have rightly pointed out, if I wrote the code I've already thought about it and have a mental model of what I wrote. And yes, I do think about what could be causing the problem. I guess it's those cases where I think it should be working in a certain way, but it isn't, where I find it most valuable to stop thinking and look. And usually in those cases I either have an incorrect or incomplete model, or I wrote something that didn't match what I was thinking.
Knowing that a particular bug can happen is usually enough to debug your mental model though. You just trace back to different assumptions you made and see which of them could cause this bug if wrong, and then fix it.
In my personal experience, most bugs come from changing a working code, especially at a refactoring. It mixes assumptions in an unpredictable way, because of chaotic code rearrangements and lots attention required to do that correctly. First-time code is much easier to challenge (if you ignore repl development phase which is not “code” yet).
If you have the luxury of 2 people on a bug, having one on strategy & idea generation and the other down in the weeds confirming hypotheses and gathering data is a common pattern.
For some reason, not being on the keyboard or operating tools seems to free up a lot of mental bandwidth.
Nothing gets done without someone being hands-on though so arguably yes, that role is more important. If there's just 1 person they usually have to stop and think really hard periodically and keep very good notes to avoid getting lost in the weeds.
In my experience, a debugger is most helpful when you are unsure of the behavior of the language or libraries or machine on which you are working. It shows you what happens at various levels and validate or invalidate your assumptions or what you were led to believe.
Sitting and thinking on the other hand, is best for dealing with errors in your own thinking. Ideally, we should have a well-thought-out argument for why something will work, so when it doesn't, our argument needs to be refined.
Exactly, perfect summary. Expanding on the first point, a debugger is invaluable for me if I can run my program up to a point and then use the "immediate" window to run a few API calls to see what would happen in that context.
I agree. Unfortunately over my career (going on 30 years) the ratio of my thinking to other people's thinking in the systems I have to develop is shrinking and shrinking. Modern software runs on systems layered onto systems layered on systems of systems. "Debuggers" are more sorely needed than ever but the traditional concept of a debugger is decades out of date and sorely needs to be reinvented for heterogenous, distributed, and parallel interactions.
I think this is very true. The most vexing problems I've run into lately have all been because something is going wrong in a step I don't have much control over. Like in one case the proxy was stripping out the Authorization header for PUT requests (seriously). Or in another the corporate-signed certs used by all the on-prem systems weren't validating on AWS because they don't have a valid root cert. I really wish I had a "debugger" that could trace the interconnections between all the systems my app depends on.
In both cases you would have been helped by proper error reporting and by tracing. (See OpenTelemetry)
For the first the remote server could say there is no Authorization header where tracing confirms it was there when sent by the client but not by the proxy.
For the second it would be really nice to see more detailed logging on the server side about why a particular peer certificate was rejected.
My biggest gripe with current OpenSSL and Java TLS implementations is this lack of detailed reporting.
In the first case the request worked when executed in Postman but failed when executed from a browser, which said it was a CORS error. Using the logs we traced it up to the authorization step in APIG, which wasn't being executed (because auth header was required to execute it) and so wasn't generating logs. And that's where I was stumped, because it was not a part of my mental model that request headers could disappear between client and APIG. We eventually realized the browser uses the proxy and postman doesn't, so we asked the networking group and they admitted they were removing the auth header in the proxy.
Regardless of the details this is one of those situations I wasn't able to solve by just thinking about it. If I had a 'debugger' that could step through these API requests it would have been much easier to figure this one out. I don't know enough about OpenTelemetry to say if it would have helped or not.
Exactly. If you live in a pure C world without external dependencies it makes sense to sit down and think it through. If you live in something like .NET or Java where you do a lot of interfacing with libraries and APIs a debugger is very useful to just discover how these systems are behaving.
I don't think this was intended to be an either/or choice, and "quit thinking and look" can be read as "if thinking isn't working, get more data" (and it could be read as such even if that wasn't the author's intended meaning!)
Even if thinking has not solved the problem, it may help you find where to look next.
One tip I have is that even if you wrote the code and think you know how it works, stepping through it in your mind might lead you to see where you are making a questionable assumption.
You may think that you can do that just as well in the debugger, and you may be right, but there are two ways of using the debugger: one is to hit 'step' and see what happens; the other is to figure out what you expect will happen, and then check if your expectation was correct. There have been many times when I have stepped up to the point where the bug is about to happen, and then realized what is wrong before taking that final step.
Personally, in the rare cases I really do not understand the behavior, I tend to just stare at the code. (It's not a recipe or so, it's just what I tend to do.) Having subtile or no syntax coloring helps.
It's a learnable skill although it is quite difficult, particularly if you didn't write the code (recently). And it's very useful because at some point the debugger (or logging[1]) will let you down.
[1] I myself don't use a debugger much; I'm frequently working on something distributed where a debugger is laughably useless.
So, I kind of agree with you. I had the same thought about the OP's emphasis on not going to the debugger quickly, resistance to that -- while mostly really identifying with the OP's framework.
BUT. Where I think the OP is right is what you do with the debugger. I agree with OP that you shouldn't use the debugger just for generalized "seeing what is going on" or jump too quickly into "finding the problem".
You should be using it to prove or disprove a hypothesis. Initially, hypotheses about your mental model, not about the problem itself.
But you can and often need to use the debugger in order to build the mental model.
The next part of quote from OP is:
> What Ken was doing was constructing a mental model of the program. When something broke it was an error in that model. He'd think of how that error might happen or where the code might not satisfy the model.
You absolutely can and I'd go so far as to say even should use the debugger as an aid to building the mental model. If you're not Ken Thompson, and depending on how familiar you are with the code, and the nature of the platform and codebase, it isn't necessarily useful to just be totally inside your head trying to "build the mental model" and think of what might be breaking it.
So it's not really about how quickly you pull out the debugger. But it is absolutely good advice not to immediately start debugging ("find the problem") with it, and not to just use the debugger aimlessly, but to start with a separate "mental model phase".
I see why the OP wants to tell you not to use the debugger in this phase, to help you understand it is in fact a separate phase, and because the debugger can lead you to miss the forest for the leaf. But you can and probably often should use the debugger there too, I agree. But you use it to build your mental model -- if you aren't sure you have a mental model, make some guesses (about how the code works, NOT about "what's wrong"), then use the debugger to confirm or deny them. To figure out "how does this code work, big picture" before trying to use it to figure out "why is it broken".
IF you are pretty familiar with the particular platform and codebase, AND pretty experienced with doing this kind of work, you might not find the debugger helpful for that first phase, like Ken Thomson. But i'd argue it's also fine to use the debugger there, it's a question of what you are using it to do.
I generally use the debugger to test my hypothesis. This is something that too many people overlook when debugging.
My debugging goes like this:
1) I need to have a model in my head of what I think is going on even if simple.
2) I make a change X with the idea that result Y should happen. VERIFYING THIS HYPOTHESIS IS VITALLY IMPORTANT. Your change should create cause and effect.
3) I test that result Y actually happened--generally this is where I'm using the debugger.
4a) If result Y happened and result Y means my program is fixed, congratulations, you are done.
4b) If result Y happened but my program is not yet fixed, go back to make another hypothesis and go back to step 2.
4c) If result Y did not happen, undo change X, update your mental model and go back to step 1. This is where the "think hard" part mostly comes in.
The toughest bugs are the ones that require you to update your mental model the most.
> Use the debugger sparingly if at all. See if you can use more exotic ways like printfs, dump out data via files, assertions etc just for the sake of getting a wider range of troubleshooting techniques should you find yourself without a debugger someday
Often it's the reverse: printfs, data file dumping, etc. are impossible, but the debugger is always there. If you find yourself chasing a bug into glibc or Rust stdlib, printf debugging won't be sufficient. Using a debugger is an underrated skill.
Talk to someone. Tell them about the problem. Show them your previous steps trying to figure it out. Talk about possible solution you've thought of. Share your frustration. Ask them if they have any ideas.
When you are done you would have come up with at least one new thing to try.
The other party you talk to need not to be a top-notch engineer. Good results were reported with the use of a rubber duck. With some training, you can talk to a more intelligent counterpart, yourself.
One more approach that worked several times for me is to start writing a Stackoverflow question, carefully formulating the problem for the reader who does not have your context. Often I saw the answer before you finish writing, and never posted the question.
I do almost the same. I don't usually use an actual rubber duck or another person and definitely not out on stack overflow but I start writing an explanation for why the bug I'm looking into happens as a reply to the bug ticket and/or ask questions to other people. It keeps evolving as I write down and check hypotheses and validate or invalidate them without ever actually posting the reply. In the end I either have a question left over that I can post or I have a trace of how/why the bug happened and post that, which comes in very handy when people inevitably ask "dumb questions" in the code review on why the fix is what the fix is.
It really helps to have these things written down somewhere as you can easily refer back to them as you revert back up the depth first search stack in your head. I guess some people can do this entirely in their head but I can't especially with all the interruptions one has.
One of our guys actually had an actual rubber duck sent to them by their employer. This was for a remote only position like 10 years ago by now.
> ...The diffuse mode shuts down when we're stressed. There's no time for dreamy big picture thinking during crunch time...
I'd add also that the focused mode often gets you stuck even more under stress. Both approaches are complimentary. This kind of problem solving needs thinking pace, just as if mind were a CPU with a program being weaved in real time, morphing constantly, seeking patterns, eliminating choices.
- Hey, Jim, are you sleeping at your desk? - No, Mr. Stone, my mind is thinking! Just be patient and please don't interrupt...
On a serious note, under the stress or not, it's important to recognize when one's own mind got stuck. Getting other people involved (as listeners or active participants) is often the only way to get through without exhausting oneself. I wish this was more encouraged in teams, instead of my task must do kind of approach.
35 comments
[ 10.5 ms ] story [ 1450 ms ] threadI had one fake AHA moment and but then it was gone again. Not sure what that says about me.
Edit and spoiler warning: Looked up the solution and I think the picture is misleading. It implies that pyramid staying on same height.
I feel like this is not a helpful take on debugging for most of us. I'm certainly not as good at this as Ken Thompson, and very seldom is the problem space simple enough that I can hold a complete mental model of it in my head. Instead I reach for the debugger to collect more data on what's going on. And I can usually find the issue by looking a lot faster than I can correctly theorize about it.
In Debugging by Davig J. Agans, his third rule is "Quit thinking and look". And I tend to agree. In the majority of cases when I'm debugging a problem the solution only reveals itself when I see that missing piece of information.
EDIT: As others have rightly pointed out, if I wrote the code I've already thought about it and have a mental model of what I wrote. And yes, I do think about what could be causing the problem. I guess it's those cases where I think it should be working in a certain way, but it isn't, where I find it most valuable to stop thinking and look. And usually in those cases I either have an incorrect or incomplete model, or I wrote something that didn't match what I was thinking.
This is why it is relatively easy to work on a project you were in on from the beginning.
and why it is always a struggle to come into a project with an existing codebase.
I've spent many years working on large existing codebases and it gets in the way of liking your job.
Yes, and unless there was a simple typo, the problem is the model in your head. Most bugs come from unchallenged assumptions.
For some reason, not being on the keyboard or operating tools seems to free up a lot of mental bandwidth.
Nothing gets done without someone being hands-on though so arguably yes, that role is more important. If there's just 1 person they usually have to stop and think really hard periodically and keep very good notes to avoid getting lost in the weeds.
Sitting and thinking on the other hand, is best for dealing with errors in your own thinking. Ideally, we should have a well-thought-out argument for why something will work, so when it doesn't, our argument needs to be refined.
For the first the remote server could say there is no Authorization header where tracing confirms it was there when sent by the client but not by the proxy.
For the second it would be really nice to see more detailed logging on the server side about why a particular peer certificate was rejected. My biggest gripe with current OpenSSL and Java TLS implementations is this lack of detailed reporting.
Regardless of the details this is one of those situations I wasn't able to solve by just thinking about it. If I had a 'debugger' that could step through these API requests it would have been much easier to figure this one out. I don't know enough about OpenTelemetry to say if it would have helped or not.
Even if thinking has not solved the problem, it may help you find where to look next.
One tip I have is that even if you wrote the code and think you know how it works, stepping through it in your mind might lead you to see where you are making a questionable assumption.
You may think that you can do that just as well in the debugger, and you may be right, but there are two ways of using the debugger: one is to hit 'step' and see what happens; the other is to figure out what you expect will happen, and then check if your expectation was correct. There have been many times when I have stepped up to the point where the bug is about to happen, and then realized what is wrong before taking that final step.
[1] I myself don't use a debugger much; I'm frequently working on something distributed where a debugger is laughably useless.
You already have a more narrow sense of the problem space, else you wouldn't know where to look with the debugger either...
BUT. Where I think the OP is right is what you do with the debugger. I agree with OP that you shouldn't use the debugger just for generalized "seeing what is going on" or jump too quickly into "finding the problem".
You should be using it to prove or disprove a hypothesis. Initially, hypotheses about your mental model, not about the problem itself.
But you can and often need to use the debugger in order to build the mental model.
The next part of quote from OP is:
> What Ken was doing was constructing a mental model of the program. When something broke it was an error in that model. He'd think of how that error might happen or where the code might not satisfy the model.
You absolutely can and I'd go so far as to say even should use the debugger as an aid to building the mental model. If you're not Ken Thompson, and depending on how familiar you are with the code, and the nature of the platform and codebase, it isn't necessarily useful to just be totally inside your head trying to "build the mental model" and think of what might be breaking it.
So it's not really about how quickly you pull out the debugger. But it is absolutely good advice not to immediately start debugging ("find the problem") with it, and not to just use the debugger aimlessly, but to start with a separate "mental model phase".
I see why the OP wants to tell you not to use the debugger in this phase, to help you understand it is in fact a separate phase, and because the debugger can lead you to miss the forest for the leaf. But you can and probably often should use the debugger there too, I agree. But you use it to build your mental model -- if you aren't sure you have a mental model, make some guesses (about how the code works, NOT about "what's wrong"), then use the debugger to confirm or deny them. To figure out "how does this code work, big picture" before trying to use it to figure out "why is it broken".
IF you are pretty familiar with the particular platform and codebase, AND pretty experienced with doing this kind of work, you might not find the debugger helpful for that first phase, like Ken Thomson. But i'd argue it's also fine to use the debugger there, it's a question of what you are using it to do.
My debugging goes like this:
1) I need to have a model in my head of what I think is going on even if simple.
2) I make a change X with the idea that result Y should happen. VERIFYING THIS HYPOTHESIS IS VITALLY IMPORTANT. Your change should create cause and effect.
3) I test that result Y actually happened--generally this is where I'm using the debugger.
4a) If result Y happened and result Y means my program is fixed, congratulations, you are done.
4b) If result Y happened but my program is not yet fixed, go back to make another hypothesis and go back to step 2.
4c) If result Y did not happen, undo change X, update your mental model and go back to step 1. This is where the "think hard" part mostly comes in.
The toughest bugs are the ones that require you to update your mental model the most.
Often it's the reverse: printfs, data file dumping, etc. are impossible, but the debugger is always there. If you find yourself chasing a bug into glibc or Rust stdlib, printf debugging won't be sufficient. Using a debugger is an underrated skill.
When you are done you would have come up with at least one new thing to try.
One more approach that worked several times for me is to start writing a Stackoverflow question, carefully formulating the problem for the reader who does not have your context. Often I saw the answer before you finish writing, and never posted the question.
It really helps to have these things written down somewhere as you can easily refer back to them as you revert back up the depth first search stack in your head. I guess some people can do this entirely in their head but I can't especially with all the interruptions one has.
One of our guys actually had an actual rubber duck sent to them by their employer. This was for a remote only position like 10 years ago by now.
I'd add also that the focused mode often gets you stuck even more under stress. Both approaches are complimentary. This kind of problem solving needs thinking pace, just as if mind were a CPU with a program being weaved in real time, morphing constantly, seeking patterns, eliminating choices.
- Hey, Jim, are you sleeping at your desk? - No, Mr. Stone, my mind is thinking! Just be patient and please don't interrupt...
On a serious note, under the stress or not, it's important to recognize when one's own mind got stuck. Getting other people involved (as listeners or active participants) is often the only way to get through without exhausting oneself. I wish this was more encouraged in teams, instead of my task must do kind of approach.