agree entirely that both techniques are useful and should be learned by programmers
one thing I think the "just do print debugging" folks miss is what a good teaching tool a visual debugger is: you can learn about what a call stack really is, step through conditionals, iterations, closures, etc and get a feel for how they really work
at some level being a good programmer means you can emulate the code in your head, and a good visual developer can really help new programmers develop that skill, especially if they aren't naturals
i emphasize the debugger in all the classes i teach for this reason
> at some level being a good programmer means you can emulate the code in your head, and a good visual developer can really help new programmers develop that skill [emphasis added]
I think you meant "a good visual debugger". And I agree completely.
A visual debugger isn't just a tool for fixing bugs.
It's also a tool for understanding the code.
At my last job, the codebase was so complicated that you could spend hours scratching your head over what was going on in a function, especially how the code got to that function and what data the calling functions had that led to this point.
Of course you could add print or log statements, but then the question is what to print! And which calling functions needed more print statements.
With a visual debugger, I could just set a breakpoint in the confusing function, see all the data it had, and also move up the stack to see what data all the calling functions had.
There are cases where you need print debugging. I added one feature that worked perfectly locally and on a test server, but failed in a Jenkins job (ironically running the job on that same test server).
That was a case where I added print statements throughout the code, just to see how far it got when running locally vs. under Jenkins.
There are many ways to debug a problem. It is wise to be familiar with all of them and know what to use when.
Whatever works so I can fix it and be home on time. I will even print (in paper) the code and step through it with a pen. Again, whatever works.
Also, will we ever move forward from these sort of discussions? Back when I was a mechanic no one argued about basics troubleshooting strategies. We just aimed to learn them and apply them all (as necessary).
Based on the "lucky 10k" and the size of the internet, likely not. I'm sure if you find a mechanics enthusiast forum you will in fact find these relatively trivial arguments. It's just the nature of the beast.
I’d say programmers are taught to nitpick away at arguments by default. The nature of the code review is to verify that the code I wrote meets certain (opinions) standards. Hence nitpicking is built into the profession. Maybe one day it’ll improve. Till then, I’ll continue arguing about everything there is :-)
I wouldn’t say programmers are taught to nitpick, so much as look for non-obvious problems. The nitpicking comes as a side effect. Maybe it is cargo-culting.
oof, iterm's done a few bad releases recently, first with their AI integration and now this sloppiness. I think they're feeling the heat from competitors like Warp, especially now that it's removed the "create an account" barrier to entry.
> Whatever works so I can fix it and be home on time. I will even print (in paper) the code and step through it with a pen.
I’ve found before that sometimes I can’t see what’s wrong with the code on my screen but I can when I print it out. I think the printed page activates different regions of the brain compared to looking at a computer screen
You should try other things that force the brain to begin interpreting something again as new input; e.g., I have found success with changing the font, or the colour scheme, or looking at a diff rendition of a change, or looking at the file with less(1), or even reading the thing in reverse order, provoking a similarly new context where I see stuff I won't have seen staring at the editor for an hour previously. Another thing printing usually induces is that you simply stop looking at it for at least a few minutes before looking again, and a break often helps too!
Printing is obviously fine if you like doing that, but I've found there are lots of ways to shake yourself loose and more thoroughly review your own writing.
If I'm looking for something and can't find it despite all signs pointing to it being nearby, I'll try to continue looking while upside down. It's often helpful for I think the same reasons that you describe.
One time, I was looking for a set of keys which had been tossed over the aisle partition in a grocery story and promptly disappeared. We tore that place apart and still couldn't find them. So I was laying on my back on the aisle floor so I could see the search area with my head upside down, and there they were. They had bounced and were hanging from the bottom of one of the shelves.
Logging is great for long term issue resolution. There's tracepoints/logpoints which let you refine the debugging experience in runtime without accidentally committing prints to the repo.
There are specific types of projects that are very hard to debug (I'm working on one right now), that's a valid exception but it also indicates something that should be fixed in our stack. Print debugging is a hack, do we use it? Sure. Is it OK to hack? Maybe. Should a hack be normalized? Nope.
Print debugging is a crutch that covers up bad logging or problematic debugging infrastructure. If you reach for it too much it probably means you have a problem.
What is the difference between print debugging and logging? With most deployments setups now days you can pipe prints to some logging solution anyways.
Logging lets you refine the level of printing and is designed to make sense in the long term. There are many technical differences (structured logging, MDC etc.) that I won't get into but they are important.
To me it's mostly about the way you write the logs vs. the way you write a print. A log tries to solve the generic problem so you can deal with it in production if the need arises by dynamically enabling logs. It solves the core issue of a potential problem. A print is a local bandaid. E.g. when print debugging one would write stuff like "f*ck 1 this was reached"... In logs you would write something sensible like "Subsystem X initialized with the following arguments %s". That's a very different concept.
totally agree re: structured logging. my intro to that was w/GCP Logging and it changed my mind on when and what to log. proper structured logging and keying metrics/alerts from those metrics is extremely satisfying and legitimately useful.
If your code is connected to some external factory, it's impossible to pause on a breakpoint without breaking everything. And no, "just mock it" doesn't work if you have no idea on how the factory actually behaves. Printf is good because it doesn't messes up timing.
I suggest re-reading my comment and learning about tracepoints/logpoints.
Your comment highlights my exact problem with print debugging... You just aren't aware of the tools available to you and you reach to the rusty old broken hammer.
Time travel debugging (https://en.wikipedia.org/wiki/Time_travel_debugging) can be good for this situation too because it separates capturing a bug ("recording") and understanding it (the debugging phase, which happens while "replaying").
You don't need to pause anything whilst capturing the bug.
Now, if your code is literally connected to running critical systems in an actual factory then you've probably got additional realtime and safety-critical considerations that might push you towards debugging.
But (for more conventional use cases) time travel debuggers can handle multiple communicating systems without causing timeouts, capture bugs in software that interacts directly with hardware devices, etc. And you don't have to keep rebuilding / rerunning once you've reproduced the bug.
I've gotten an OS to run on a new platform with a debugging tool portfolio consisting of a handful of LEDs and a pushbutton. After getting to the point where I could printf() to the console felt like more than anyone could ever ask for.
Anecdote aside, it certainly doesn't hurt to be able to debug things without a debugger if it comes to that.
Since most of my work is in distributed systems, I find the advice to never printf downright laughable.
"Oh sure, lemme just set a breakpoint on this network service. Hm... Looks like my error is 'request timed out', how strange."
That having been said: there are some very clever solutions in cloud-land for "printf" debugging. (Edit: forgot this changed names) Snapshot Debugger (https://github.com/GoogleCloudPlatform/snapshot-debugger) can set up a system where some percentage of your instances are run in a breakpointed mode, and for some percentage of requests passing through the service, they can log relevant state. You can change what you're tracking in realtime by adding listeners in the source code view. Very slick.
TDD is ok, but LDD is better. If you can figure out what went wrong with your code from the logs, there's a non-negative chance that whoever gets the joy of maintaining said code might be able to as well
all the rest of the stuff is okay but seriously? 2 finger typing? as a programmer? you're gonna be there for AAAGES and your keyboard probably runs a sidebusiness with how long it takes you to type
I know I'm stirring up shit here but there really are benefits to touch typing (I mean just think about it, using 10 fingers instead of 2 is gonna be so much faster assuming you have 10 dingers)
Touch typing is objectively better, but 2 finger is enough to have a career at programming, it's not what is going to hold you back.
I'd be more concerned about RSI than speed. You _really_ don't need to type fast for programming. If you do, your tools should be helping you do boilerplate more.
Even that depends on what exactly you mean. I agree with the literal meaning of the words, but for too many people "touch typing" means "home-row touch typing". Which I find extremely awkward and difficult. But literally just typing by touch, I'm self-taught due to StarCraft multiplayer back in the 90s and early 2000s, and have a style that confuses home-row typists when they realize what I'm doing.
So it's more like, you need to type smoothly enough that you're not thinking about typing. As long as your hands are pretty much moving automatically to get your thoughts into the computer, you can use any number of fingers at (almost) any speed, and you're good.
The RSI thing is real. Until my late 20s I would type blindly with what I jokingly called 5 and a half fingers. But I did start to notice some strain, which is when I decided to finally teach myself "proper" touch typing. I don't think my typing got meaningfully faster, but it got smoother, and the strain went away.
Getting a split-layout "ergo" keyboard helped me learn to type better back in the day. I could already touch type fast but not with great technique.
When I got a split layout key it became quite apparent that my technique was "weird" - sometimes I'd notice one hand come wandering over to the other side of the keyboard to find a key it was used to pressing. That became easy to correct once it was so visible!
I'm a programmer, not a secretary, I don't need more than 2-4 fingers because I don't need to write code that fast. I don't understand why anyone who writes code that is not semi-trivial would need to write code so fast they needed to touch type.
There are ergonomic issues too. Hunt and peck gives you a lot more flexibility as to how you can use a keyboard.
Most programmers who don't touch type can type pretty fast. There is more than one way to do it, and I'm not convinced touch typing is the right one.
I've been programming for decades and I like... three finger type. Two index fingers and a thumb. I don't look at the keyboard though (I know where the keys are relative to each other inherently through years of typing this way) and I type at a very high rate of speed for someone using so few fingers.
A trace log of single-character print statements or equivalent is sometimes the simplest and most effective way to debug flow. This is particularly true for recursive functions, where anything more becomes too much, too fast.
I have a very specific technical (UX) reason for using `print()` to debug sometimes.
In VS Code, if you want to run debugger with arguments (especially for CLI programs), you have to put these arguments in launch.json and then run the debugger.
This is often tedious to do, because I usually have typed these arguments and tested in terminal before, and now I have to convert them into json format, which is annoying.
To make it worse, VS Code uses a separate window for debug console than your main terminal, so they don't share history/output.
So if I know what to look at already and don't really need a full debugger, I often just use print() temporally.
I have the same issue. The tool I’m working on has a CLI with multiple subcommands. So I just added a REPL subcommand to my CLI. That way, I can start the CLI REPL in VSCode debug and then paste in the subcommand/arguments I want to debug.
I'm pretty sure you could use something like https://code.visualstudio.com/docs/editor/variables-referenc... to allow you to copy and paste your args into a user prompt when you initiate the debugger so you don't have to convert them to json first for each command you want.
>The main arguments are “if you need to use a debugger you’re an idiot and you don’t understand the code you are writing” (that’s not an actual quote but there was a similar take along those lines). Then there is “If you can’t use a debugger you’re an idiot”. The hating on the ‘printf’ crew is omnipresent.
My holier-than-thou take on this topic is: Whenever possible, debug by adding assert statements and test cases.
25 years ago I worked on a port of PC -> Playstation 1 game. We did not had proper devkits, but the yaroze model ("amateur", allowing for "printf"-debugging of sorts)
Long story short, our game worked as long as the printfs we had were kept, we had macro to remove them (in "Release/Ship") but the game crashed.
The crash was due to side-effect of printf clearing some math errors.... So here you go!
As with so many categorical guidelines, there are circumstances to use it and circumstances to not.
The key insight is that printf() is a heavyweight operation ("What, you want to build a string? A human readable string? Okay, one second, lemme just pull in the locale library..."). If you're debugging something at the business-logic layer, it's probably fine.
If you're debugging a memory leak, calling a function that's going to make a deep-dive on the stack and move a lot of memory around is likely to obscure your bug.
> If you're debugging a memory leak, calling a function that's going to make a deep-dive on the stack and move a lot of memory around is likely to obscure your bug.
shudder memories of my early days in kernel-level programming where using a printf used to just "fix" some bugs.
That came down to an uninitialised variable (which calling printf was helpfully initialising by using the stack).
As a result of that early experience, when I'm in the headspace of very low-level bugs, I find it helps to think of printf both as "this will tell me some variable values" and as a source of other clues: if it makes a weird value disappear or change then you might have uninitialised data on your stack, if it makes a flaky behaviour become stable (either disappear or become repeatable) then it's probably a race condition, etc.
Isn't localization triggered for specific format strings only? There are many corner cases when typical printf implementations call malloc, but it's usually not too hard to avoid them.
If you are worried about including <stdio.h> and potential side effects from that, you can use __builtin_printf instead.
You're correct, but then you're adding the additional burden to your debugging process of memorizing which format strings trigger localization. I'd have to look it up, but I'm pretty sure numbers do, which means anytime you %d or %f you're inviting the devil in.
When I say move memory around in this context, I mean do a lot of stack operations. You can leak from the stack too (drop a pointer from the stack without freeing the underlying heap memory it referenced), and it's harder to catch that if printf has come along and completely rewritten your unused stack memory as consequence of reporting on the state of your program.
That having been said, the point is that context matters and what you're debugging matters for the question of what tool to use. If you're operating in an interpreted language, you can probably trust that The interpreter is making it difficult to leak memory like that. On the other hand, interpreters have bugs too, and using a language that is interpreted instead of compiled machine code makes bugs in the execution layer unlikely, but not impossible...
Admittedly, I forgot about %f/%g and the decimal separator. I think the rest is fairly obscure (such as %lc for wide character to multibyte character translation). For integers you need %Id (variant digits) or %'d (grouping).
The decimal separator is a bit of a problem for JSON generation, too. Some systems have snprintf_l, but it's not very widespread.
I also use something like printf() from the beginning, just with a shorter name, and its printing is conditional, so I usually don't remove it from the code. :)
Merging PREEMPT_RT (real-time) into the Linux kernel only occurred in Sept, 2024, after 20 years of work, in part due to a fundamental conflict with printk(), the traditional "printf" of Linux, widely used for kernel debugging. Linus is unwilling to forego printf debugging in the kernel, and has blocked significant evolutions of Linux on its behalf.
That counts for a lot to me. Anti-printf people have to go a long way to convince me that it's somehow not ok. I use debuggers every day. I use some form of printf debugging every day. I use whatever affordance I can get my hands on, and I ignore the peanut gallery.
The main advantage of printf is that it's usually just there, one line of code away. Setting up a useful debugger session can be much more involved (it varies widely depending on what you're working on), so my ADHD mind will absolutely try to get away with printf first if it's possible. Sometimes it ends up being counter-productive, and you just learn to recognize when to bother with experience.
That said, there are some contexts where this is reversed - printing something useful without affecting the debugged code may actually be more involved than, say, attaching a JTAG probe and stepping through with a debugger. Though sometimes both of those are a luxury you can't afford, so you better be able to manage without them anyway (and this may happen to you regardless of whether you're working on low-level hardware bring-up or some shiny new JavaScript framework).
It’s shocking how many people never use a debugger. Like sure printf debugging is good enough sometimes. But never? That’s wild.
Honestly I think attaching a debugger should be the first debugging tool you reach for. Sometimes printf debugging is required. Sometimes printf debugging is even better. But a debugger should always be the tool of first choice.
And if your setup makes it difficult to attach a debugger then you should re-evaluate your setup. I definitely jump through hoops to make sure that all the components can be run locally so I can debug. Sure you’ll have some “only in production” type bugs. But the more bugs you can trivially debug locally the better.
Of course I also primarily write C++ code. If you’re stuck with JavaScript you maybe the calculus is different. I wouldn’t know.
To be frank, I actually find debugger to rarely be useful. When it is useful, it is super useful - but those occasions aren't that numerous, so I can totally believe that many people may get away without learning how to use it. It feels like 90% of my time in gdb is reading backtraces after segfaults, which I wouldn't really consider to be a use of a debugger. I'm still glad I can use it for the other 10%, but it's just 10%.
GDB is a dog shit trash debugger. I keep forgetting that Linux and Mac have shit tools. I suppose if the only debugger available to me were GDB I would also prefer printf debugging!
The typical modern dev loves to shit on Windows. But Visual Studio (the adult version, not VSCode) is still a best-in-class debugger. Xcode is bloated as hell but did help me last week. Linux has… poor bastards.
I've spent plenty of time with various debuggers for various platforms and languages, both graphical and CLI-based, and it really doesn't matter to what I said. A debugger may be someone's preferred tool anyway, sure, but the cases where it's actually non-trivially improving workflow are relatively rare. Most of the time it just makes you a bit faster, if at all.
What is wrong with JetBrains' CLion? It is a cross-platform C & C++ IDE. I have used it on Linux, and the GUI-based debugger is so much easier to use compared to GDB from the command line.
Also: Do also say that LLDB is a "dog shit trash debugger"?
I can't imagine what it's like to be so passionate about something that works, and it's not some philosophical/moral issue, just -- real programmers use X?
I use vim, IDEs, debuggers, printf debugging, whatever works. A tool is a tool. I guess my holier-than-thou position is against the idea that there's one right way to do anything.
In my eyes Kibana / Elastic logging is even better, basic logging is useful for local-only dev work, but once its getting deployed, a more serious logging setup is infinitely more useful. You can log all relevant data down to a specific event or request and really dig into things as needed. If you use log levels correctly, you can get drastically more detailed by getting all the "debug" logs. This was the bread and butter at a former employer.
If using the debugger is less efficient than using printf then it's a symptom of a wider problem. To be clear though, "printf debugging" is not the same thing as adding structured debug logs to your service and enabling it on demand via some config. Most production services should never be logging unstructured output to stdout. Printf debugging is just throwing out some text to stdout, and it generally means running unit tests locally and iterating through an (add printf, run test, add more printf) loop until the bug is found. Unfortunately it's the default way to debug locally reproducible bugs for so many engineers. So while I don't see the point in not using printf for purely ideological reasons, I avoid building software where this is the simplest option and use the service's structured logger if not easily reproducible. I also generally think it's bad to default to printf debugging in the way I have defined it here and find that competent use of debugger is more often a faster way to debug.
I find that a lot of this discussion just melts away if you are aware of the options that are available and then take your pick. Why, sometimes I've switched between debuggers and printfs as many as six times before breakfast.
If you don't know what a debugger does though that's something you should really get on ASAP. Likewise if you can't figure out how to get log messages out of your thing. Really all there is to it, figure out what you want to do after than and spend your time actually doing something productive instead of getting in a stupid holy war on the internet about it.
If you have a reproducible test case that runs reasonably quickly, then I think printf debugging is usually just as good as a "real" debugger, and a lot easier. I typically have my test output log open in one editor frame. I make a change to the code in another frame, save it, my build system immediately re-runs the test, and the test log immediately refreshes. So when the test isn't working, I just keep adding printfs to get more info out of the log until I figure it out.
This sounds so dumb but it works out to be equivalent to some very powerful debugger features. You don't need a magical debugger that lets you modify code on-the-fly while continuing the same debug session... just change the code and re-run the test. You don't need a magical record-replay debugger that lets you go "back in time"... just add a printf earlier in the control flow and re-run the test. You don't need a magical debugger that can break when a property is modified... just temporarily modify the property to have a setter function and printf in the setter.
Most importantly, though, this sort of debugging is performed using the same language and user interface I use all day to write code in the first place, so I don't have to spend time trying to remember how to do stuff in a debugger... it's just code.
BUT... this is all contingent on having fast-running automated tests that can reproduce your bugs. But you should have that anyway.
> But for many bugs, getting to a reproduction is already more than half the battle.
Today I am working on a bug where a token expires after 2 hours and then we fail to request a new one instead we just keep on using the now expired one, which (of course) doesn’t work. I have a script to reproduce it but it takes 2 hours to run. It would be great if there was some configuration knob to turn down the expiry just for this test so we can reproduce it faster - but there isn’t because nobody thought of that.
The token isn’t being verified in my service it is being verified in a remote service. I can’t easily change the clock on the remote service. It runs in a K8S pod and (AFAIK) K8S doesn’t support time namespaces. And even if it does, the pod is deployed by a custom K8S operator so unsure if I could make the operator turn that on even if it is available. And the remote service is a complex monolith which uses lots of RAM so trying to run it on my own laptop will be painful
Reminds me of the classic adage that "everything can be solved by another layer of abstraction, except for the problem of too much abstraction layers".
> Most importantly, though, this sort of debugging is performed using the same language and user interface I use all day to write code in the first place, so I don't have to spend time trying to remember how to do stuff in a debugger... it's just code.
Absolutely! Running a commandline debugger adds additional context I have to keep in my head (syntax for all the commands, output format etc), that actively competes with context required to debug my code. Printfs work just fine without incurring that penalty, granted this argument applies less to IDE debuggers because their UX is usually intuitive.
The crashes/bugs I deal with are rarely straight down failures, they are often the 1 out of 100 runs kind, so printf debugging is the only way to go really. And I used to be big on using debuggers, but now I’m horribly out of practice.
> The crashes/bugs I deal with are rarely straight down failures, they are often the 1 out of 100 runs kind, so printf debugging is the only way to go really.
Another thing I’ve found helpful, is to write out a “system state dump” (say in JSON) to a file whenever certain errors happen. Like we had a production system that was randomly hanging and running out of DB connections. So now whenever the DB connection pool is exhausted, it dumps a JSON file to S3 listing the status of every DB connection, including the stack dump of the thread that owns it, the HTTP request that thread is serving, the user account, etc. Once we did that, it went from “we don’t understand why this application is randomly falling over” to “oh this is the thing that is consistently triggering it”
When it writes a dump, it then starts a “lockout period” in which it won’t write any further dumps even if the error reoccurs. Don’t want to make a meltdown worse by getting bogged down endlessly writing out diagnostics.
Record-and-replay debuggers are often better than anything else for debugging intermittent failures: run the program with recording many times until you eventually get the bug, then debug that recording at your leisure; you'll never have to waste time reproducing it again.
Can you share an example? In my whole career, I have only seen one or two of them, but most of my work is CRUD type of stuff, not really gaming or systems programming where such a thing might happen.
You reuse connections with a connection pool, but you accidentally reuse connections with different privileges and scopes. As a result, sometimes you get to read some data you shouldn't read and sometimes you don't.
Or, concurrency bugs.
You don't properly serialize transactions and sometimes two transactions overlap in time leading to conflicts.
In Android they are way too common. Like an animation that decides to fire after the UI element it was made for has been destroyed, or a race condition in destroying a resource, or VE logging.
I would say that like... saying "OK so `a.b` looks like this, what does `a.b.c` look like?" is a very nice flow with deep and messy objects (especially when working in languages with absolutely garbage default serialization logic like Javascript)
But even with a debugger, there's still loads of value of sitting up, moving from writing a bunch of single line statements all over, and to writing a real test harness ASAP to not have to rely on the debugger.
For any non-trivial problem, you'll often very quickly appreciate a properly formatted output stack, and that output will be shaped to the problem you are looking at. Very hard for an in-process debugger to have the answer for you there immediately.
Being serious about debugging can even get into things like writing actual new entrypoints (scripts with arguments and whatnot) and things like adding branches togglable through environment variables.
I think a lot of people's mindset in debugging is "if I walk the tightrope and put in _one more hack_ I'll find my answer" and it gets more and more precarious. Debugging is always a bit of an exercise of mental gymnastics, but if you practice being really good at printf debugging _and_ configuring program flow easily, you can be a lot less stressed out.
Like if you think your problem is going to take more than 20 minutes to debug, you probably should start writing a couple helper functions to get you on the right foot.
Adding print statements, rebuilding and rerunning the program and figuring out where in the logs the bug showed up this time can a lot more tedious than setting a (possibly conditional) breakpoint in a reverse-execution debugger and doing "reverse-continue".
We literally just saw a case[0] where an application (iterm2) mistakenly left verbose logging on in a component of a production build and logged sensitive console contents to connected hosts.
Are you diligent enough to remove your sensitive logging/printf statements EVERY time, for the rest of your career? Or should you make a habit of doing things properly from day one?
An interesting note: printf'ing a variable can sometimes alter how a variable is cached. This is a particularly useful fact in memory locations that can change as a result of hardware operations (like a change of a status bit by a hardware element).
printf'ing effectively enforces a similar condition to `volatile` on the underlying memory segment when it is read.
One can encounter tersely written code that works perfectly with printf statements, but status bits never get "updated" (CPU cache purged) without the printf and hangs the program.
151 comments
[ 3.5 ms ] story [ 207 ms ] threadone thing I think the "just do print debugging" folks miss is what a good teaching tool a visual debugger is: you can learn about what a call stack really is, step through conditionals, iterations, closures, etc and get a feel for how they really work
at some level being a good programmer means you can emulate the code in your head, and a good visual developer can really help new programmers develop that skill, especially if they aren't naturals
i emphasize the debugger in all the classes i teach for this reason
I think you meant "a good visual debugger". And I agree completely.
A visual debugger isn't just a tool for fixing bugs.
It's also a tool for understanding the code.
At my last job, the codebase was so complicated that you could spend hours scratching your head over what was going on in a function, especially how the code got to that function and what data the calling functions had that led to this point.
Of course you could add print or log statements, but then the question is what to print! And which calling functions needed more print statements.
With a visual debugger, I could just set a breakpoint in the confusing function, see all the data it had, and also move up the stack to see what data all the calling functions had.
There are cases where you need print debugging. I added one feature that worked perfectly locally and on a test server, but failed in a Jenkins job (ironically running the job on that same test server).
That was a case where I added print statements throughout the code, just to see how far it got when running locally vs. under Jenkins.
There are many ways to debug a problem. It is wise to be familiar with all of them and know what to use when.
Also, will we ever move forward from these sort of discussions? Back when I was a mechanic no one argued about basics troubleshooting strategies. We just aimed to learn them and apply them all (as necessary).
We need to have people continue the meme by missing the point of nitpicking nitpicked arguments. That’s a special thing here in HN.
Present them as options. "If you like X, you may also like Y." Leave it to the audience to discriminate when to apply the tool.
You're correct, the real issue comes down to
1. making sure those printf statements don't wind up in prod, spilling potentially sensitive data or corrupting a data stream
2. making sure that non-printf tooling is built so that only printf debugging isn't used
We tend to get caught up in false dichotomies.
https://news.ycombinator.com/item?id=42579472
I’ve found before that sometimes I can’t see what’s wrong with the code on my screen but I can when I print it out. I think the printed page activates different regions of the brain compared to looking at a computer screen
Printing is obviously fine if you like doing that, but I've found there are lots of ways to shake yourself loose and more thoroughly review your own writing.
One time, I was looking for a set of keys which had been tossed over the aisle partition in a grocery story and promptly disappeared. We tore that place apart and still couldn't find them. So I was laying on my back on the aisle floor so I could see the search area with my head upside down, and there they were. They had bounced and were hanging from the bottom of one of the shelves.
When in doubt, do something weird.
Logging is great for long term issue resolution. There's tracepoints/logpoints which let you refine the debugging experience in runtime without accidentally committing prints to the repo.
There are specific types of projects that are very hard to debug (I'm working on one right now), that's a valid exception but it also indicates something that should be fixed in our stack. Print debugging is a hack, do we use it? Sure. Is it OK to hack? Maybe. Should a hack be normalized? Nope.
Print debugging is a crutch that covers up bad logging or problematic debugging infrastructure. If you reach for it too much it probably means you have a problem.
Logging lets you refine the level of printing and is designed to make sense in the long term. There are many technical differences (structured logging, MDC etc.) that I won't get into but they are important.
To me it's mostly about the way you write the logs vs. the way you write a print. A log tries to solve the generic problem so you can deal with it in production if the need arises by dynamically enabling logs. It solves the core issue of a potential problem. A print is a local bandaid. E.g. when print debugging one would write stuff like "f*ck 1 this was reached"... In logs you would write something sensible like "Subsystem X initialized with the following arguments %s". That's a very different concept.
Your comment highlights my exact problem with print debugging... You just aren't aware of the tools available to you and you reach to the rusty old broken hammer.
You don't need to pause anything whilst capturing the bug.
Now, if your code is literally connected to running critical systems in an actual factory then you've probably got additional realtime and safety-critical considerations that might push you towards debugging.
But (for more conventional use cases) time travel debuggers can handle multiple communicating systems without causing timeouts, capture bugs in software that interacts directly with hardware devices, etc. And you don't have to keep rebuilding / rerunning once you've reproduced the bug.
Anecdote aside, it certainly doesn't hurt to be able to debug things without a debugger if it comes to that.
"Oh sure, lemme just set a breakpoint on this network service. Hm... Looks like my error is 'request timed out', how strange."
That having been said: there are some very clever solutions in cloud-land for "printf" debugging. (Edit: forgot this changed names) Snapshot Debugger (https://github.com/GoogleCloudPlatform/snapshot-debugger) can set up a system where some percentage of your instances are run in a breakpointed mode, and for some percentage of requests passing through the service, they can log relevant state. You can change what you're tracking in realtime by adding listeners in the source code view. Very slick.
Time travel debugging (https://en.wikipedia.org/wiki/Time_travel_debugging) can help with this because it separates "recording" (i.e. reproducing the bug) from "replaying" (i.e. debugging).
Breakpoints only need to be set in the replay phase, once you've captured a recording of the bug.
I know I'm stirring up shit here but there really are benefits to touch typing (I mean just think about it, using 10 fingers instead of 2 is gonna be so much faster assuming you have 10 dingers)
I'd be more concerned about RSI than speed. You _really_ don't need to type fast for programming. If you do, your tools should be helping you do boilerplate more.
Even that depends on what exactly you mean. I agree with the literal meaning of the words, but for too many people "touch typing" means "home-row touch typing". Which I find extremely awkward and difficult. But literally just typing by touch, I'm self-taught due to StarCraft multiplayer back in the 90s and early 2000s, and have a style that confuses home-row typists when they realize what I'm doing.
So it's more like, you need to type smoothly enough that you're not thinking about typing. As long as your hands are pretty much moving automatically to get your thoughts into the computer, you can use any number of fingers at (almost) any speed, and you're good.
When I got a split layout key it became quite apparent that my technique was "weird" - sometimes I'd notice one hand come wandering over to the other side of the keyboard to find a key it was used to pressing. That became easy to correct once it was so visible!
But now I'm curious, which thumb do most people use on a keyboard?
I'd suggest practicing your touch typing a bit more.
There are ergonomic issues too. Hunt and peck gives you a lot more flexibility as to how you can use a keyboard.
Most programmers who don't touch type can type pretty fast. There is more than one way to do it, and I'm not convinced touch typing is the right one.
In VS Code, if you want to run debugger with arguments (especially for CLI programs), you have to put these arguments in launch.json and then run the debugger.
This is often tedious to do, because I usually have typed these arguments and tested in terminal before, and now I have to convert them into json format, which is annoying.
To make it worse, VS Code uses a separate window for debug console than your main terminal, so they don't share history/output.
So if I know what to look at already and don't really need a full debugger, I often just use print() temporally.
My holier-than-thou take on this topic is: Whenever possible, debug by adding assert statements and test cases.
Long story short, our game worked as long as the printfs we had were kept, we had macro to remove them (in "Release/Ship") but the game crashed.
The crash was due to side-effect of printf clearing some math errors.... So here you go!
The key insight is that printf() is a heavyweight operation ("What, you want to build a string? A human readable string? Okay, one second, lemme just pull in the locale library..."). If you're debugging something at the business-logic layer, it's probably fine.
If you're debugging a memory leak, calling a function that's going to make a deep-dive on the stack and move a lot of memory around is likely to obscure your bug.
shudder memories of my early days in kernel-level programming where using a printf used to just "fix" some bugs.
That came down to an uninitialised variable (which calling printf was helpfully initialising by using the stack).
As a result of that early experience, when I'm in the headspace of very low-level bugs, I find it helps to think of printf both as "this will tell me some variable values" and as a source of other clues: if it makes a weird value disappear or change then you might have uninitialised data on your stack, if it makes a flaky behaviour become stable (either disappear or become repeatable) then it's probably a race condition, etc.
A reference to James Mickens' "The Night Watch" feels appropriate here: https://www.usenix.org/system/files/1311_05-08_mickens.pdf
If you are worried about including <stdio.h> and potential side effects from that, you can use __builtin_printf instead.
When I say move memory around in this context, I mean do a lot of stack operations. You can leak from the stack too (drop a pointer from the stack without freeing the underlying heap memory it referenced), and it's harder to catch that if printf has come along and completely rewritten your unused stack memory as consequence of reporting on the state of your program.
That having been said, the point is that context matters and what you're debugging matters for the question of what tool to use. If you're operating in an interpreted language, you can probably trust that The interpreter is making it difficult to leak memory like that. On the other hand, interpreters have bugs too, and using a language that is interpreted instead of compiled machine code makes bugs in the execution layer unlikely, but not impossible...
The decimal separator is a bit of a problem for JSON generation, too. Some systems have snprintf_l, but it's not very widespread.
Been waiting for "something else" ~30 years & counting.
That counts for a lot to me. Anti-printf people have to go a long way to convince me that it's somehow not ok. I use debuggers every day. I use some form of printf debugging every day. I use whatever affordance I can get my hands on, and I ignore the peanut gallery.
That said, there are some contexts where this is reversed - printing something useful without affecting the debugged code may actually be more involved than, say, attaching a JTAG probe and stepping through with a debugger. Though sometimes both of those are a luxury you can't afford, so you better be able to manage without them anyway (and this may happen to you regardless of whether you're working on low-level hardware bring-up or some shiny new JavaScript framework).
Honestly I think attaching a debugger should be the first debugging tool you reach for. Sometimes printf debugging is required. Sometimes printf debugging is even better. But a debugger should always be the tool of first choice.
And if your setup makes it difficult to attach a debugger then you should re-evaluate your setup. I definitely jump through hoops to make sure that all the components can be run locally so I can debug. Sure you’ll have some “only in production” type bugs. But the more bugs you can trivially debug locally the better.
Of course I also primarily write C++ code. If you’re stuck with JavaScript you maybe the calculus is different. I wouldn’t know.
The typical modern dev loves to shit on Windows. But Visual Studio (the adult version, not VSCode) is still a best-in-class debugger. Xcode is bloated as hell but did help me last week. Linux has… poor bastards.
Does Visual Studio do time traveling yet? https://www.replay.io/ is cross-platform.
WinDbg has a time traveling debugger.
replay.io does not support a single language or environment that I care about.
Also: Do also say that LLDB is a "dog shit trash debugger"?
I use vim, IDEs, debuggers, printf debugging, whatever works. A tool is a tool. I guess my holier-than-thou position is against the idea that there's one right way to do anything.
Me too!
If you don't know what a debugger does though that's something you should really get on ASAP. Likewise if you can't figure out how to get log messages out of your thing. Really all there is to it, figure out what you want to do after than and spend your time actually doing something productive instead of getting in a stupid holy war on the internet about it.
This sounds so dumb but it works out to be equivalent to some very powerful debugger features. You don't need a magical debugger that lets you modify code on-the-fly while continuing the same debug session... just change the code and re-run the test. You don't need a magical record-replay debugger that lets you go "back in time"... just add a printf earlier in the control flow and re-run the test. You don't need a magical debugger that can break when a property is modified... just temporarily modify the property to have a setter function and printf in the setter.
Most importantly, though, this sort of debugging is performed using the same language and user interface I use all day to write code in the first place, so I don't have to spend time trying to remember how to do stuff in a debugger... it's just code.
BUT... this is all contingent on having fast-running automated tests that can reproduce your bugs. But you should have that anyway.
Ideally, yes. But for many bugs, getting to a reproduction is already more than half the battle. And a debugger can help you with that.
Today I am working on a bug where a token expires after 2 hours and then we fail to request a new one instead we just keep on using the now expired one, which (of course) doesn’t work. I have a script to reproduce it but it takes 2 hours to run. It would be great if there was some configuration knob to turn down the expiry just for this test so we can reproduce it faster - but there isn’t because nobody thought of that.
Well, the hard part isn’t actually the code change (reasonably obvious) it is deploying my own copy of their service…
Or modify the expiry time locally / not use the one from the token at all.
Absolutely! Running a commandline debugger adds additional context I have to keep in my head (syntax for all the commands, output format etc), that actively competes with context required to debug my code. Printfs work just fine without incurring that penalty, granted this argument applies less to IDE debuggers because their UX is usually intuitive.
Another thing I’ve found helpful, is to write out a “system state dump” (say in JSON) to a file whenever certain errors happen. Like we had a production system that was randomly hanging and running out of DB connections. So now whenever the DB connection pool is exhausted, it dumps a JSON file to S3 listing the status of every DB connection, including the stack dump of the thread that owns it, the HTTP request that thread is serving, the user account, etc. Once we did that, it went from “we don’t understand why this application is randomly falling over” to “oh this is the thing that is consistently triggering it”
When it writes a dump, it then starts a “lockout period” in which it won’t write any further dumps even if the error reoccurs. Don’t want to make a meltdown worse by getting bogged down endlessly writing out diagnostics.
Not my original idea, I got it from some Oracle products which have a similar feature (most notably Oracle Database)
Where may I read about particular workflows involving debuggers, e.g. gdb?
(Mainly for programs written in C.)
Most of my issues are related to issues with concurrency though, deadlocks and whatnot.
You reuse connections with a connection pool, but you accidentally reuse connections with different privileges and scopes. As a result, sometimes you get to read some data you shouldn't read and sometimes you don't.
Or, concurrency bugs.
You don't properly serialize transactions and sometimes two transactions overlap in time leading to conflicts.
But even with a debugger, there's still loads of value of sitting up, moving from writing a bunch of single line statements all over, and to writing a real test harness ASAP to not have to rely on the debugger.
For any non-trivial problem, you'll often very quickly appreciate a properly formatted output stack, and that output will be shaped to the problem you are looking at. Very hard for an in-process debugger to have the answer for you there immediately.
Being serious about debugging can even get into things like writing actual new entrypoints (scripts with arguments and whatnot) and things like adding branches togglable through environment variables.
I think a lot of people's mindset in debugging is "if I walk the tightrope and put in _one more hack_ I'll find my answer" and it gets more and more precarious. Debugging is always a bit of an exercise of mental gymnastics, but if you practice being really good at printf debugging _and_ configuring program flow easily, you can be a lot less stressed out.
Like if you think your problem is going to take more than 20 minutes to debug, you probably should start writing a couple helper functions to get you on the right foot.
Are you diligent enough to remove your sensitive logging/printf statements EVERY time, for the rest of your career? Or should you make a habit of doing things properly from day one?
0 https://news.ycombinator.com/item?id=42579472
Yes, when a linter is set up that fails when printf debugging is found
printf'ing effectively enforces a similar condition to `volatile` on the underlying memory segment when it is read.
One can encounter tersely written code that works perfectly with printf statements, but status bits never get "updated" (CPU cache purged) without the printf and hangs the program.