I recently discovered https://github.com/weirdNox/emacs-gdb/ . The installation is a bit unconventional compared to the usual Emacs package installation process, but works pretty well.
Strange that VSCode + debugger frontend extensions (mentioned in the first part) caused such trouble, it's working flawlessly so far for me for mixed Zig/C projects, across Windows, macOS and Linux (Ubuntu 20)
The inspection capabilities in the VSCode debugger are very bare bones though, that's true. At least the variable view panel could need some work.
I was so goo at debugging the hardware I underestimated just how difficult will it be to switch to software even when you really good at that language. But set up is important esp for web technologies as they are buggy by default. Interestingly I never had issue with C
1. Debuggers are in some sense anti-unix. They only work well when they are thoroughly integrated with the rest of your environment. Trying to have some debugger program like gdb and then talking to it from emacs or a terminal or something will be a big pain. Emacs is also anti-unix but seems to have enough mass that it doesn’t matter so much (or maybe the debugger and emacs fighting for importance is problematics)
2. Getting a debugger to work well is hard. Apart from needing information to work out what bit of code is executing (and passing all that information through weird compilation steps or linking), the debugger needs to understand the memory layout of objects and interpret some subset of the language so that people can enter eg watch expressions. This is hard when the compiler and debugger are quite separate and just agree on the format for possible debug information. Lack of integration hurts again
3. Some people are so used to not having a debugger that they don’t know what they’re missing. Debuggers work well for what, windows and websites? I guess Java too? Lots of people end up debugging with prints and that just isn’t so effective. I think I’ve barely used a debugger for more than trivial websites in the last 10 years.
4. The problem of debugging has become harder: there are now more threads in programs and it is more common for languages to have some kind of async mechanism. If you naively debug that, you’ll see crap stacktraces (a few frames sat atop a scheduler) and execution may move randomly between threads in a pool. I feel like most debuggers can’t cope. (I guess website debuggers do understand async better and there are also new inventions like reversible debugging with rr, but the problem with integrating a debugger with a language or an editor remains.)
it's not a problem when you accept that the unix philosophy is extremely far from the holy grail of software design. "things that do one thing well" certainly has good sides for specific problems, mainly quick'n'dirty linear text processing dataflows, but is atrocious for other problems (anything that need integration and discoverability: art software, web browsers, or revolves more around user interaction than computing something)
> 3. Some people are so used to not having a debugger that they don’t know what they’re missing. Debuggers work well for what, windows and websites?
if debuggers took less time to start than a full "add print -> recompile -> relaunch app" cycle as is my experience with C++, they would certainly get more use.
Hell, in my main project I've been unable to use any version of gdb & lldb for something like two months as both segfault a couple seconds after reaching a breakpoint for some reason. It's a shitshow.
> Hell, in my main project I've been unable to use any version of gdb & lldb for something like two months as both segfault a couple seconds after reaching a breakpoint for some reason. It's a shitshow.
I'm really surprised. In my experience it's really rare for either of those debuggers to crash, let alone for both to crash on the same target.
A bit of a tangent, but would you mind sharing some details about your compiler / OS / debugger / target-hardware versions?
CLI version of gdb has worked extremely reliable for me across multiple projects and architectures (x86, arm, arm64, etc).
I have yet to encounter a bug can't be debug with gdb with some test automation + gdb script with conditional breakpoints / hw breakpoints. As long as I can reproduce the issue with test automation, I can debug it with gdb.
Most complex bugs are I have to debug was multiple threads (20+) memory leak shows up after days of testing.
I don't use GUI gdb. cli + gdb script is the best and most reliable debugging environment. I don't add printf to my code anymore, a new gdb breakpoint script + with proper debug build can replace the printf anywhere and anytime.
Interesting. I just took a look at the line of gdb code that's crashing on you. I'm only slightly familiar with gdb's internals, but I can imagine a bunch of different sources for this bug:
- gdb has a misconception about which c++ ABI was used when compiling the target program, or
- your program is stomping on an object's vtable pointer, leading gdb to follow invalid pointers when trying to parse the object's RTTI, or
- there's a bug in clang++, or
- there's a mix of abi versions in your program's code, perhaps because of unexpected linking issues (static or dynamic), or perhaps because of inconsistent compiler flags when building your code
I'm curious what happens if you build your program with g++ instead of clang++.
My program has no issues running with asan / ubsan so I don't think it would be a vtable / memory overwrite issue as those are pretty much 100% detected. Also I build my whole stack (except glibc and libx11 basically) statically with the same flags so I doubt there could be an abi problem (and LTO builds would complain).
> if debuggers took less time to start than a full "add print -> recompile -> relaunch app" cycle as is my experience with C++, they would certainly get more use.
This is a really good point - programmers seem to prefer making "dead"/batch-processing programs over "living", interactive, persistent systems (according to Steve Yegge's definition of those[1]).
This style, in addition to being very Unix-y, also lends itself really well to printf debugging - you just add printfs, recompile, and re-run.
I build "live" programs, and make heavy use of the debugger - although that might be because of tight integration of my debugger with the rest of my tools - although that might be because other users of this language tend to prefer the live style and build their tools to adapt.
I also build live programs, I just have a simple system that reloads its state from before a crash upon statup - that ends up working fine 99% of the time and I can get back to just before the faulty interaction without loosing time.
I agree that debuggers being anti-unix isn’t a problem if they embrace that and integrate deeply with their environment but such a phrase does not describe gdb or lldb.
> Trying to have some debugger program like gdb and then talking to it from emacs or a terminal or something will be a big pain.
If you're already familiar with Emacs commands, then the M-x gdb interface is as good as any debugger interface in an IDE.
If you're not already an Emacs user, then the TUI[1] (gdb -tui) is surprisingly good - much better than the "old school" approach I was taught in college of just interacting via the gdb prompt.
> Debuggers work well for what, windows and websites?
I spend most of my time on a large C++ project that does neither of these things, and I still use gdb a lot. In my experience, it's most useful for unit tests - you identify a bug, write a unit test that triggers the bug, and run the unit test under gdb to poke around and understand what's going on.
gdb and lldb don't work great when run on the whole program, mainly because it's hard to specify the actual behavior you want when debugging a system where multiple threads are executing concurrently. Almost all of our unit tests are single-threaded, so this isn't a problem when debugging unit tests.
If you're working on a C or C++ program and using a supported processor, I also highly recommend trying out rr[2], the record and replay debugger, which you mentioned in your comment. Unlike gdb, it's reasonably easy to use and understand even if your program has lots of concurrency, and it pretty much eliminates the need to do "printf" debugging, since you can print variables from any line in your program without recompiling and re-running. You can still use the Emacs or TUI gdb interfaces when using "rr replay".
>> Debuggers work well for what, windows and websites?
> I spend most of my time on a large C++ project that does neither of these things, and I still use gdb a lot.
I think that, when GP wrote "windows", they meant the operating system. And when they wrote "debuggers work well", they meant the quality of debuggers, not whether using a debugger is a good idea. Quality of Windows debuggers is generally on a whole other level than on Linux. Chromium DevTools also has a nice debugger. I don't know about Java.
I need to throw a million +1's at the `rr` suggestion. it's really great. It's a shame that it doesn't work on ARM processors, but there are good fundamental reasons why it doesn't (it's basically impossible iirc[0])
EDIT - just to mention, rr is mostly compatible with gdb, so frontends such as gdb-many-windows in emacs can work with it.
In terms of U.I's I found `gdb-many-windows` to be the nicest i found. While I hate almost everything about the platform, I have to concede that visual studio's debugging integration is actually really good. The only thing I found annoying was that I needed to use Visual Studio for it! (emacs on windows could be annoying enough that I got used to it eventually, but still preferred my linux environment. This is back when working on a cross platform C++ project if that much weren't already clear).
The last part of that page mentions that it has recently been made to work on aarch64 given the right CPU and all code including libraries built so they avoid using the offending instructions.
I spend my life in emacs (comments passim) but I put it to you that M-x gdb is mostly unpleasant, as is gdb --tui. Both pale in comparison to visual studio 10-15 years ago. The OP and the post which the OP is following up to both mention those debugger interfaces.
About composability since I've taken to fuzzing I've been improving my gdb chops and now I mostly write gdb scripts if the program is not too long to run, or I just use a simple bisect script with with core dump checkpointing and detecting changes of whatever I'm interested in. With a large tmpfs it's quite fast and modular.
If I'm doing real-time, Intel PT is great, especially if you can afford the signal-shit happening when dumping core (you get Intel PT trace in core dumps now and gdb supports them if compiled with...)
> 3. Some people are so used to not having a debugger that they don’t know what they’re missing. Debuggers work well for what, windows and websites? I guess Java too? Lots of people end up debugging with prints and that just isn’t so effective. I think I’ve barely used a debugger for more than trivial websites in the last 10 years.
They're extremely useful for dynamic languages (Python, Ruby, etc) since you can step through call chains and see the full objects being passed around. Wonderful way to learn a new codebase where you have a bunch of functions passing stuff around with no type information in the code.
The debugger is the main reason you're not gonna pull me away from IntelliJ for JVM or dynamic language stuff... it could be even better - like how CLion is discussed - but I'm not seeing a better debugger-less alternative.
Hooking into async callback stuff and losing useful stacktraces isn't great, but it's not like print statements are gonna do any better there either...
I continue to be shocked at the lack of productive development tools on linux. My current c/c++ toolset is: a makefile generator that I wrote after years of wasting time on autotools problems, Anjuta for debugging and rudimentary editing, and Visual Studio 2008 on Windows XP running in VirtualBox for most productive work (it's a cross platform project).
Qt Creator's debugger is almost perfect for my goals. It's, probably, the most user-friendly. However, I wish I could feed gdb commands directly to it.
> However, I wish I could feed gdb commands directly to it.
you can: while debugging, enable "Debug Log" (in the View menu), and this adds a pane where you can input gdb command. Output is not very pretty though.
> It doesn't actually seem to be trying to write "/tmp". It actually doesn't issue any syscalls at all when I press "Debug Main Project" and trigger the message.
The one thing I really miss about Windows is the Visual Studio debugger -- it was frickin' awesome even back w/VS 2005 and I have to believe it's gotten better.
On Linux I use ddd, and while it's definitely quirky (in that weird X way), it works just fine. I would guess that many of the poster's problems have to do with Linux's mediocre GUI support in general. I've recently switched to doing development on Ubuntu (vs. RH/CentOS) and all the GUI stuff works much better in that environment, incl. ddd.
I haven't tried Eclipse's C/C++ debugger, but its Perl debugger (part of EPIC) is quite nice and works well on both Mac and Linux.
It's worth mentioning that vim has added debugger integration support as a plugin, but that plugin is available with vim by default, you just need to enable it. It's not perfect but it's usable - you can set breakpoints visually and save/load them, you can step debug inside your code window (gdb is actually running in another vim window and can be controlled directly) and you can see the value of expressions in your code window.
Disclaimer (and a bit of context): I work for a company that builds a special type of debugger, aimed at debugging live applications in production.
As one would expect, we've spent an enormous amount of time thinking about debuggers, how they work from the inside and what they contribute to the general experience of developing software.
Debuggers are, for developers well-versed in working with them, priceless tools (see [1] for an example from this very thread). There's so, so much value in walking through a program not only in one's head (as well-put by pg at one time[2]), but by using a tool that enforces a more structured, "formal" process for program exploration. I use it when trying to understand how code I write actually looks like IRL, but also - and maybe more so - for understanding code OTHER PEOPLE wrote with better context than any piece of documentation or simply reading the code would ever give me.
But - as the article and some[3] of[4] the[5] comments[6] here point out - it has warts (quick plug - I recently wrote a blog post[7] that details my perspective on remote debuggers and how they can be better).
One thing that I didn't see mentioned in this thread or in the article, and is a real PITA, is performance. Debuggers are not built to deal with performance problems - or collection of temporal data, for that matter - at all.
Consider how one might go about debugging a real-life performance issue, especially in the context of modern applications - many threads, many hosts, many replicas of the same service, etc.
You probably have a tracer - something like Zipkin[8] or Jaeger[9] - that helps you to break apart latency issues in your endpoints. You also probably have a set of instrumented metrics (graphed inside your APM, perhaps) for quick observation when something is not reacting as fast as you expect it to. But sometimes they don't tell the full story - there's just too many possible edge cases.
Performance issues can be caused by data structures exploding in size, causing degraded performance and OOMs. They can be caused by one method, in a chain of dozens of different, inter-connected ones, that is especially slow. It can be related to non-optimized SQL queries (maybe hidden behind an ORM). It can be a weird code path - one the developer did not expect, and did not test for - that gets triggered by users with very specific characteristics.
43 comments
[ 4.5 ms ] story [ 101 ms ] threadThis recreates the Visual Studio style workflow described in https://twitter.com/pervognsen/status/1098453808651202560 and https://youtu.be/0woxSWjWsb8?t=254
The inspection capabilities in the VSCode debugger are very bare bones though, that's true. At least the variable view panel could need some work.
1. Debuggers are in some sense anti-unix. They only work well when they are thoroughly integrated with the rest of your environment. Trying to have some debugger program like gdb and then talking to it from emacs or a terminal or something will be a big pain. Emacs is also anti-unix but seems to have enough mass that it doesn’t matter so much (or maybe the debugger and emacs fighting for importance is problematics)
2. Getting a debugger to work well is hard. Apart from needing information to work out what bit of code is executing (and passing all that information through weird compilation steps or linking), the debugger needs to understand the memory layout of objects and interpret some subset of the language so that people can enter eg watch expressions. This is hard when the compiler and debugger are quite separate and just agree on the format for possible debug information. Lack of integration hurts again
3. Some people are so used to not having a debugger that they don’t know what they’re missing. Debuggers work well for what, windows and websites? I guess Java too? Lots of people end up debugging with prints and that just isn’t so effective. I think I’ve barely used a debugger for more than trivial websites in the last 10 years.
4. The problem of debugging has become harder: there are now more threads in programs and it is more common for languages to have some kind of async mechanism. If you naively debug that, you’ll see crap stacktraces (a few frames sat atop a scheduler) and execution may move randomly between threads in a pool. I feel like most debuggers can’t cope. (I guess website debuggers do understand async better and there are also new inventions like reversible debugging with rr, but the problem with integrating a debugger with a language or an editor remains.)
it's not a problem when you accept that the unix philosophy is extremely far from the holy grail of software design. "things that do one thing well" certainly has good sides for specific problems, mainly quick'n'dirty linear text processing dataflows, but is atrocious for other problems (anything that need integration and discoverability: art software, web browsers, or revolves more around user interaction than computing something)
> 3. Some people are so used to not having a debugger that they don’t know what they’re missing. Debuggers work well for what, windows and websites?
if debuggers took less time to start than a full "add print -> recompile -> relaunch app" cycle as is my experience with C++, they would certainly get more use.
Hell, in my main project I've been unable to use any version of gdb & lldb for something like two months as both segfault a couple seconds after reaching a breakpoint for some reason. It's a shitshow.
I'm really surprised. In my experience it's really rare for either of those debuggers to crash, let alone for both to crash on the same target.
A bit of a tangent, but would you mind sharing some details about your compiler / OS / debugger / target-hardware versions?
CLI version of gdb has worked extremely reliable for me across multiple projects and architectures (x86, arm, arm64, etc).
I have yet to encounter a bug can't be debug with gdb with some test automation + gdb script with conditional breakpoints / hw breakpoints. As long as I can reproduce the issue with test automation, I can debug it with gdb.
Most complex bugs are I have to debug was multiple threads (20+) memory leak shows up after days of testing.
I don't use GUI gdb. cli + gdb script is the best and most reliable debugging environment. I don't add printf to my code anymore, a new gdb breakpoint script + with proper debug build can replace the printf anywhere and anytime.
https://twitter.com/jcelerie/status/1348259387127824387/phot...
(lldb does not give me an error message, just segfault)
- gdb has a misconception about which c++ ABI was used when compiling the target program, or
- your program is stomping on an object's vtable pointer, leading gdb to follow invalid pointers when trying to parse the object's RTTI, or
- there's a bug in clang++, or
- there's a mix of abi versions in your program's code, perhaps because of unexpected linking issues (static or dynamic), or perhaps because of inconsistent compiler flags when building your code
I'm curious what happens if you build your program with g++ instead of clang++.
Best of luck. I hope you'll post an update if you ever get to the bottom of it.
This is a really good point - programmers seem to prefer making "dead"/batch-processing programs over "living", interactive, persistent systems (according to Steve Yegge's definition of those[1]).
This style, in addition to being very Unix-y, also lends itself really well to printf debugging - you just add printfs, recompile, and re-run.
I build "live" programs, and make heavy use of the debugger - although that might be because of tight integration of my debugger with the rest of my tools - although that might be because other users of this language tend to prefer the live style and build their tools to adapt.
[1] http://steve-yegge.blogspot.com/2007/01/pinocchio-problem.ht...
And yet somehow Microsoft made theirs work well since the early 90s
If you're already familiar with Emacs commands, then the M-x gdb interface is as good as any debugger interface in an IDE.
If you're not already an Emacs user, then the TUI[1] (gdb -tui) is surprisingly good - much better than the "old school" approach I was taught in college of just interacting via the gdb prompt.
> Debuggers work well for what, windows and websites?
I spend most of my time on a large C++ project that does neither of these things, and I still use gdb a lot. In my experience, it's most useful for unit tests - you identify a bug, write a unit test that triggers the bug, and run the unit test under gdb to poke around and understand what's going on.
gdb and lldb don't work great when run on the whole program, mainly because it's hard to specify the actual behavior you want when debugging a system where multiple threads are executing concurrently. Almost all of our unit tests are single-threaded, so this isn't a problem when debugging unit tests.
If you're working on a C or C++ program and using a supported processor, I also highly recommend trying out rr[2], the record and replay debugger, which you mentioned in your comment. Unlike gdb, it's reasonably easy to use and understand even if your program has lots of concurrency, and it pretty much eliminates the need to do "printf" debugging, since you can print variables from any line in your program without recompiling and re-running. You can still use the Emacs or TUI gdb interfaces when using "rr replay".
[1] https://sourceware.org/gdb/current/onlinedocs/gdb/TUI.html#T...
[2] https://rr-project.org/
> I spend most of my time on a large C++ project that does neither of these things, and I still use gdb a lot.
I think that, when GP wrote "windows", they meant the operating system. And when they wrote "debuggers work well", they meant the quality of debuggers, not whether using a debugger is a good idea. Quality of Windows debuggers is generally on a whole other level than on Linux. Chromium DevTools also has a nice debugger. I don't know about Java.
EDIT - just to mention, rr is mostly compatible with gdb, so frontends such as gdb-many-windows in emacs can work with it.
In terms of U.I's I found `gdb-many-windows` to be the nicest i found. While I hate almost everything about the platform, I have to concede that visual studio's debugging integration is actually really good. The only thing I found annoying was that I needed to use Visual Studio for it! (emacs on windows could be annoying enough that I got used to it eventually, but still preferred my linux environment. This is back when working on a cross platform C++ project if that much weren't already clear).
[0] https://github.com/rr-debugger/rr/issues/1373#issuecomment-1...
However I'd also like to say "reading that entire thread from the linked comment down is fascinating and I'd recommend it".
If I'm doing real-time, Intel PT is great, especially if you can afford the signal-shit happening when dumping core (you get Intel PT trace in core dumps now and gdb supports them if compiled with...)
They're extremely useful for dynamic languages (Python, Ruby, etc) since you can step through call chains and see the full objects being passed around. Wonderful way to learn a new codebase where you have a bunch of functions passing stuff around with no type information in the code.
The debugger is the main reason you're not gonna pull me away from IntelliJ for JVM or dynamic language stuff... it could be even better - like how CLion is discussed - but I'm not seeing a better debugger-less alternative.
Hooking into async callback stuff and losing useful stacktraces isn't great, but it's not like print statements are gonna do any better there either...
[0] https://remedybg.itch.io/remedybg
That's an indictment of the available debugger interfaces that they wouldn't.
[0] https://github.com/emoon/ProDBG
you can: while debugging, enable "Debug Log" (in the View menu), and this adds a pane where you can input gdb command. Output is not very pretty though.
Perhaps a subprocess is doing this?
On Linux I use ddd, and while it's definitely quirky (in that weird X way), it works just fine. I would guess that many of the poster's problems have to do with Linux's mediocre GUI support in general. I've recently switched to doing development on Ubuntu (vs. RH/CentOS) and all the GUI stuff works much better in that environment, incl. ddd.
I haven't tried Eclipse's C/C++ debugger, but its Perl debugger (part of EPIC) is quite nice and works well on both Mac and Linux.
This is a nice guide to it: https://www.dannyadam.com/blog/2019/05/debugging-in-vim/
As one would expect, we've spent an enormous amount of time thinking about debuggers, how they work from the inside and what they contribute to the general experience of developing software.
Debuggers are, for developers well-versed in working with them, priceless tools (see [1] for an example from this very thread). There's so, so much value in walking through a program not only in one's head (as well-put by pg at one time[2]), but by using a tool that enforces a more structured, "formal" process for program exploration. I use it when trying to understand how code I write actually looks like IRL, but also - and maybe more so - for understanding code OTHER PEOPLE wrote with better context than any piece of documentation or simply reading the code would ever give me.
But - as the article and some[3] of[4] the[5] comments[6] here point out - it has warts (quick plug - I recently wrote a blog post[7] that details my perspective on remote debuggers and how they can be better).
One thing that I didn't see mentioned in this thread or in the article, and is a real PITA, is performance. Debuggers are not built to deal with performance problems - or collection of temporal data, for that matter - at all.
Consider how one might go about debugging a real-life performance issue, especially in the context of modern applications - many threads, many hosts, many replicas of the same service, etc.
You probably have a tracer - something like Zipkin[8] or Jaeger[9] - that helps you to break apart latency issues in your endpoints. You also probably have a set of instrumented metrics (graphed inside your APM, perhaps) for quick observation when something is not reacting as fast as you expect it to. But sometimes they don't tell the full story - there's just too many possible edge cases.
Performance issues can be caused by data structures exploding in size, causing degraded performance and OOMs. They can be caused by one method, in a chain of dozens of different, inter-connected ones, that is especially slow. It can be related to non-optimized SQL queries (maybe hidden behind an ORM). It can be a weird code path - one the developer did not expect, and did not test for - that gets triggered by users with very specific characteristics.
Hard.
[1] https://news.ycombinator.com/item?id=25960923#unv_25960923:~......
[2] http://www.paulgraham.com/head.html
[3] https://news.ycombinator.com/item?id=25940064#unv_25957905:~...
[4] https://news.ycombinator.com/item?id=25940064#unv_25957481:~....
[5] https://news.ycombinator.com/item?id=25940064#unv_25957428:~....
[6] souprock ↗ There are debuggers that nicely handle real-time with "many threads, many hosts, many replicas of the same service".
If you don't mind paying, Concurrent has stuff for it: NightView, NightSim, NightTrace https://www.youtube.com/user/ConcurrentRealTime/videos
Other approaches include whole-system emulation, digital logic analyzers, and SystemTap.