Ask HN: How do you get better at debugging/finding a solution?
Software dev with more than 10 years of experience. I always gotten praise by colleagues and companies I worked for. But, there are always moments where, without help from a colleague, I wouldn't have found a solution to a problem.
When joining a new company, 2 months in, and facing a problem where code wouldn't compile, I dig deep into the codebase but for the love of me, can't find a solution.
I wonder, is that normal? Or how can I say: "I don't care, if I am alone, I can figure this out and solve it."
Did anyone here evolve from this state of reliance of others and turned themselves into a "I can do it myself in a good amount of time"?
82 comments
[ 4.5 ms ] story [ 125 ms ] threadWhat also helps is just shutting the computer off and going for a walk, generally the moment you step away you'll figure it out!
Underrated. Diffused mode of brain.
1) Find an employer who doesn't pay by the hour (but e.g. per day or task/goal)
2) Find an employer who isn't strict on the hours.
3) Self employment.
4) Find or convince an employer who accepts this.
None are ideal, except for #4, but its the long breath.
And I am thoroughly convinced it works. Same with workout, mindfulness. Such behavior should be rewarded. My employer supports workout in the sense I get 1 hour per week paid leave for working out.
By the time you're done describing the problem and its symptoms, you'll have a solution.
I can also recommend going for a walk. I usually return from long motorcycle trips with the best ideas. Tea on the balcony is also effective.
Actually I think I enjoy problem solving more than the actual programming. Though frankly programming and problem solving are so closely related they are almost the same thing.
I can solve most problems I come up against day to day. I guess I can't solve them all because I do ask questions on Stack Overflow sometimes.
I'm not sure why, I guess it's a few things:
1: As I said, I enjoy problem solving - it feels like a game
2: I have more than 35 years hard computer problem solving, so there's alot of experience which helps.
3: I understand, and make a deliberate effort to understand, as much as I possibly can about every aspect of computer systems from hardware to operating system to database, to cloud to front end to back end.
4: I use the tried and tested problem solving method of divide and conquer - when you have a problem, break the software in half and keep doing so until you find where the problem is.
5: If you are still stuck, try to create a minimal test case the demonstrates the problem - this often solves it, and if it doesn't then you can post the minimal test case on StackOverflow.
6: Only ever work on one thing at a time.
7: Work hard to learn how to use problem solving tools.... the debugger in the browser, make sure you know a little strace for doing things like tracking what file the operating system is trying to open, ngrep so you can watch data passing over the network.
8: Be relentless. Sometimes I think this is a personality "issue", but I literally will work for 10 hours at a time trying to solve a problem until I crack it. I really don't like finishing work without having all the days problems solved. I just grind and grind and grind on problems until I work it out.
9: Have a really thorough knowledge of the technologies you are working with - don't be satisfied with learning as you go - actually take the time to read the language specification for the languages you program with, stuff like that.
10: Use a really good IDE and make sure you know how to use its ability to jump through the code, to jump to function definitions. You need to be good at following the logic of the program.
11: And finally, if no matter what you do the error remains the same, then you are probably not even working on the correct code.
In the case for example of why a program won't compile, well just start cutting code out until something compiles, then add it back in little by little until you work out where the line is that fails. Even better is to use appropriate debugging tools, but divide and conquer always works.
There's also no shame in scattering these everywhere you suspect the problem might be:
print('got here!')
As a more junior programmer, I would sometimes resist the perceived effort or inelegance of the divide-and-conquer approach and end up spending much more time than needed to get to a solution.
The very first thing should probably be to carefully check which commit introduced the error, and then carefully study the code changes in that one for clcues.
After that, and possibly based on that, what probably most people will do is start their debugging process by running a debugger through some suspicious parts of the code, based on varying degrees of well-informed suggestions.
For problems that escape those first tries though, what you often need to do is to start a systematic process of ruling out possible sources for the bug. In many ways this resembles scientific studies where you try to control as many variables as possible, and also include control samples with known states, trying to zoom in on only the particular variable you are studying without noise from other things.
That can mean feeding the system or code under study with carefully set up data for which you know what the effect should be, and then carefully trying to change each part of it and observing the outcomes. Things like that.
In my experience, this type of effort will eventually most often lead to the solution. The main challenge I think though, is to realize how deeply you might need to go with the systematization and automation of things, to really rule out possible sources and start zooming in on the general area around where the bug is. You might need to take some real drastic measures, and this is where I see most people who fail, don't go far enough. Here you might need to really get away from the screen to get your thoughts flowing more freely ... but not in an undirected way, but rather trying to answer the question "How can I do this in an even more systematic way ... to rule out even more possible sources, or identify unexpected behavior".
Not super easy to put into words, but this is in my experience the way to go.
Finally, one caveat is that there are certain things you should probably check before even going the systematic path. Things that can totally screw things up so that whatever you do, you never get any systematic pattern of behavior or behavior change. These things often are related to caches in various form. Make sure to turn off any and all kinds of caches in the system. They will almost guaranteed drive you insane otherwise.
Git bisect is a very useful tool for this assuming you have an easily reproducible case for the problem. (So generally it is decidely less useful when the bug is the result of a race condition)
Comment out or delete the code until the fundamentals of the system works. Then add stuff back until errors happen. Also add better logging to track down the problem. Tear down the system and build it back up. Delete nonstandard approaches you can’t figure out and use standard ones from QuickStart tutorials. It might be slow progress but progress will be made.
get good at finding and reaching out to contractable experts for rabbit-holes, whether for the spot-work or to pair on solving with their expertise
tl;dr, when debugging people naturally form a hypothesis about what is going wrong and then set about gathering evidence to support or refute that hypothesis. When you do so unconciously you are very prone to biases of all sorts, most importantly confirmation bias.
When you do so mindfully, and Write It All Down then you are much more likely to a) come up with an evidence gathering exercise which will usefully falsify your hypothesis, and b) respect the gathered evidence, reject the now false hypothesis, and move on to a new one.
https://www.youtube.com/watch?v=FihU5JxmnBg
When you're pulling on a thread and come across a function that looks completely broken and yet somehow is not.
Debugging has been one of those for me.
A long time ago I helped my mother with a murder mystery by researching how doctors diagnose things (I was a professional information broker then, with Dialog, Lexus/Nexus, GratefulMed, etc.), as much as I could without going to medical school. I learned a lot about differential diagnosis. That got me hooked on medical shows, where I learned a little more. At some point after that I wondered if I could apply differential diagnosis methods to debugging. Because there are often multiple possible causes for a bug, I found the differential diagnosis approach to work amazingly well for me.
I call this process D3 (Differential Diagnosis Debugging).
Below is roughly how I apply it. I am working on a book including this as a couple chapters, but that won't be out for at least another year. The material in this post is in the book, so I am told I must copyright anything smacking of an excerpt.
First, capture all the relevant details of the expected behavior. Create a unit test (or tests) to confirm the expected behavior.
Next, capture all the differences between the observed behavior and the expected behavior (the 'symptoms').
Then, examine those differences to come up with possible hypothesis about the causes.
After that, use a concept similar to Karnaugh Maps [1] to determine a sequence of small discrete unit tests whose truth (if true hypothesis could be true) determines a T or F for each hypothesis. If you wind up with more than one T then you need more tests (diagnostic testing).
Once you have a confirmed hypothesis, apply a fix an rerun all your tests. Rinse and repeat as needed, if needed (treatment), until all of your expected behavior tests pass.
Unpublished Work © Copyright 2022 William A. Barnhill, Jr. Some rights reserved. You may apply the D3 process as described herein; you may not incorporate the D3 process into a written work, a web site, or an email; you may discuss the D3 process if full attribution to the author is given.
Please don't hate me for the above folks. Been told I need to include that if I want to get published.
[1] https://en.wikipedia.org/wiki/Karnaugh_map
Trademark applies to special identifiers of products and services.
Patents apply to inventions. It may apply to processes, given tangible form.
In short, you are taking bad advice. Get better lawyers.
In the alternative that you think your licensing terms mean anything:
1. By reading these words you agree, on behalf of yourself and your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that you believe I have entered into with you or your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges.
2. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
If the problem is too obscure, I 'just' rebuild the system like I imagined it.
But I don't think everyone is as suited to this form of tackling problems. You might get more enjoyment out of creating new things?
Short term - while debugging a problem: 1. Gather whatever information you can. Look in the logs! 2. Debugging means tracking the flow of execution from beginning to end (and then back to the front in many cases). Where along that chain is the breakdown? 3. Often you can stop after 2, since when the point of failure is determined the issue becomes obvious. However, in some cases it is not obvious. In those cases, you have to look for 'something that does not make sense' or can't be true.
Case study: timeouts We started having errors on a service we owned. Calls would be made to our service (call it service A) and then calls to an upstream service (service B) would time out, and we didn't catch this exception so the request to A would end with a 500 response. However, when we looked at the logs for B we could see our requests from A and that they were not timing out at all, they were taking the usual amount of time. This is a huge contradiction! After a day of poking around we could reproduce the timeout with a script which called our service. Thinking it might be the load balancer, we started sending requests to the IP's of workers directly, which never timed out. On a hunch, we sent requests to the IP of the load balancer (this can't be done directly since the LB uses the hostname as part of how it routes requests to servers, but we could do this by adding the LB with the proper hostname to the /etc/hosts file.) Low and behold, the timeouts were still gone even though we were doing 'the exact same thing' as before! Removing the entry from the hosts file, the timeouts immediately came back for the same small percentage of requests.
Long term: Being good at debugging really consists of two things, knowing how things work and spending the time required to debug things. Interestingly, one great way to learn how things work is to debug things. Whenever something does not make sense, or you can't explain it, dig in and get to the bottom of it. Every time you do this you'll learn things that you never would have guessed, and your ability to debug some random future issue will go way up when that future issue just happens to intersect something that you had to dig into previously.
"You need a few more years of study before you can fully understand. But! You have a 600 page textbook. Do every exercise in that book. When you can do that without assistance, get another 600 page textbook and repeat."
He was trying to get us to build an intuition for certain class problem solving, while at the same time saying "Shut up and calculate".
I find that problem-solving in the programming space is the same. Just keep doing things. You'll develop an intuition for it eventually.
Knowing who knows what, and who can help, and asking for help the right way is key to getting better at problem solving.
Perhaps it is the school system that discourages problem solving through social means, and people feel guilty about taking help.
However, consider this perspective. What if the ones who could put together people into functional groups could be considered great problem solvers? Consider any of the great American enterprises, there are people at the top who figured out how to put people together to get things done.
Keep a debugging journal. Take notes on every step you take and its results. It's easy to go in circles because you forget what you tried or forgot some detail of its outcome. Seeing a summary of what you already know helps you rule out possibilities and inspires new ones.
I often forget to do this or feel like "I can handle this bug without a crutch". Yet every time I actually journal the process it's helpful.
I generally use Notepad++ and it's often just a set of notes about the value of important variables in certain files @ a particular line of code or at a particular time. I find that seeing that big picture at a glance can often give me immediate insight into what's happening. Something you can't always see when you're focused on a few lines of code in a method.
You can also refer back to your journal to remind yourself how you fixed that similar problem months ago.
Of course, a journal doesn't have to be limited to debugging. It's useful for development too.
* When did this start happening?
* What changed between when it was working when it stopped working?
* Have you asked person Y who worked on that change?
Have a hypothesis and try to prove otherwise.
Start cordoning off parts of the codebase where the problem ISN'T.
Leave printf breadcrumbs to trace execution.
I find my less capable colleagues make the mistake of false assumptions, like some subroutine is executing or what state an object is in without proving it to themselves. That keeps your investigation from proceeding without luck on the larger problems.
This advice would be very obvious to many, but I'm constantly surprised by how many of my coworkers don't know how to do this and test code by pushing logging statements into a production environment.
To quickly fix something a lot of the times it is easier to push debug statements to environment that setup whole shebang on your local.
Most of the times you don't know which part of the system is wrong so yeah you can unit test all you want but if data in database is incorrect you still have to reproduce incorrect data on your local - but you still have to somehow get to know that there is incorrect data somewhere.
I use PHP/Laravel mostly, and while XDEBUG will work with a remote server, it requires an extra extension be installed and activated with a restart of PHP-FPM. That might mean downtime if you only have one app server. After the extension is activated, it requires extra resources to profile each request. Sometimes an order of magnitude more.
Luckily, because it is just PHP, running it locally isn't too hard, but I wanted to outline a scenario where a bunch of people cannot really use a debugger in prod. I've definitely been on teams that have pushed log statements to production to track down issues that were only present in one environment and not reproducible locally. Actually, one just came up about a month ago. Our app server is behind a firewall and a VPN (internal company app) and one of those services isn't properly forwarding the `X-Forwarded-Host` header correctly which caused issues later down the line when the framework was attempting to generate URLs. This caused an issue in our frontend code where some JSON data included bad URLs to the UI. Wasn't reproducible at all locally. Took some logging to find out where in the process the URL was wrongly generated so we could develop a workaround.
I’m grunching this thread. But I have 20ish years of experience and consider myself a decent debugger and considered enough so by others that I’ve been asked this line of questioning while in a mentorship role on a number of occasions.
The number one thing is simply humility; asking the dumb question in the smartest way you can, as quickly as possible, to the person who can most likely give you the best answer.
As you’re learning and starting out, you may not have great resources for such, but within most orgs you should be able to find where the answers are, and first ask the dumb questions (caveated by what you do understand), then eventually ask for a brain dump.
It is through the accumulation of tons of disparate idiosyncratic knowledge that little clues and common patterns emerge that allow people to sniff out a root cause hypothesis before the facts are even in, and often be right.
But people who hide their ignorance, refuse to ask questions that may make them look silly, refuse to be the idiot in the room, they learn much less quickly because they don’t get unblocked as quickly.