I strongly encourage developments like that. It’s a shame that Linux and macOS didn’t adopt something like Microsoft’s symbol server. This project could become something like this.
If you have an installation like this you can also plug it into Sentry as symbol server and it picks it up.
Yep, in case of Linux the 'it's OSS. Download the tar.gz configure --enable-debug and deal with it' attitude hampered the development and use of something like a Symbol Server.
Microsoft has had it since at least 2005 - that's the first Visual Studio release I remember using it with - so it might have exited before that. It's things like these that separate MS from all others - Engineering culture of having to give 2 shits about other people - OEMs, developers, corporate IT, partners and having the chops to effectively address it at scale within their Engineering teams.
Yeah I know - that's a more recent thing however and the attitude in the early days was what I was referring to.
Oh and the debuginfo packages used to be and still are huge in size. Not sure if anything has changed most recently but the concept of lazy loading symbols from remote and storing only what you need locally is not something you can do with debuginfo packages.
Apple runs something similar internally, as it can correlate debug symbols to binaries by checking the UUID the linker sticks in it. Unfortunately, it’s not generally available…
There used to be a hack where you could write your own UUID-to-dSYM script, name it "dsymForUUID", and then lldb would assume it was Apple's internal magic thing and use it. Fortunately lldb seems to have improved since those days, and has less magical options available:
"Good enough" solutions were already in place for many years, that's why a symbol server was not a matter of urgency. Did you know about the following?
• A package manager can be configured to install the corresponding debuginfo/debugsource packages, this is literally one menu toggle. It's off by default to save disk space, as most users don't need them.
• A desktop environment crash handler (e.g. drkonqi) offers to install the missing debuginfo/debugsource packages and then produces a complete stacktrace.
• gdb shows the command for installing the missing debuginfo/debugsource packages it needs.
I'm not aware of those solutions. Very cool! Are they specific to one distro (for example, Fedora)? How do I get gdb to show me the command to installing missing packages?
One downside to those solutions, they only support debugging on the same machine running the executable. If I want to debug a crash dump from a machine different from my own I might not be able to install the packages.
In Opensuse (release, haven't used Tumbleweed) I found it easy to install debuginfo and debugsource packages.
In Ubuntu I found it slightly confusing to locate debug symbol packages. I understood (or misunderstood) that there are 2 options where to find debug symbols and it just depends on the package maintainer how it's done. I did not find how to install debugsource at all.
By doing debugsource I mean the source that has all the patches applied exactly as during the build and is in the same path as the symbols suggest.
With the Microsoft tooling, you debug a crash dump and your IDE automatically pulls in the symbol data from your symbol server and then automatically checks out the appropriate revisions from your config management system, because the references are embedded into the symbol files at build time.
It’s a much better mechanism than anything I’ve looked at in the Linux space (caveat: I have not been developing C++ for a while).
I was so impressed I started looking at what it would take to implement for the gcc/gdb toolchain but discovered Microsoft held a patent and gave up.
Any chance that Linux package managers will integrate this for their -dbg packages? I’d rather not fiddle with downloading those along with -src, untarring it, telling GDB where to find the code…
Very small. I asked before and the consensus was that nobody wants to maintain it. There was a project before (Fedora’s dark server) which died.
That said, debian started indexing build ids in the repos so at least for dwarf files there is a barebones index now. Sadly not for the companion executables.
This was originally built by RedHat and they are adding support for Fedora's package manager, which should make debugging in future versions of Fedora an amazing experience. The ability to debug any application on your machine with no effort.
Debuginfod does support debian packages as well (integrating with -dbg pacages), but it does not support src packages. The documentation mentions that this is tricky.
The debug server kinda builds upon the dbg/src packages. So all the maintainers of the distributions have to do is maintain the compatibility of the dbg/src package structure (which looks like happening for the RPM format) and to host a debug server in addition to the existing package servers.
The Nix ecosystem have dwarffs[0]. If you generate debug symbols during the build ( not enabled by default ) and save it to the "nix cache", you can use dwarffs to download it.
20 comments
[ 3.1 ms ] story [ 44.2 ms ] threadIf you have an installation like this you can also plug it into Sentry as symbol server and it picks it up.
Microsoft has had it since at least 2005 - that's the first Visual Studio release I remember using it with - so it might have exited before that. It's things like these that separate MS from all others - Engineering culture of having to give 2 shits about other people - OEMs, developers, corporate IT, partners and having the chops to effectively address it at scale within their Engineering teams.
Oh and the debuginfo packages used to be and still are huge in size. Not sure if anything has changed most recently but the concept of lazy loading symbols from remote and storing only what you need locally is not something you can do with debuginfo packages.
http://lldb.llvm.org/use/symbols.html
• A package manager can be configured to install the corresponding debuginfo/debugsource packages, this is literally one menu toggle. It's off by default to save disk space, as most users don't need them.
• A desktop environment crash handler (e.g. drkonqi) offers to install the missing debuginfo/debugsource packages and then produces a complete stacktrace.
• gdb shows the command for installing the missing debuginfo/debugsource packages it needs.
One downside to those solutions, they only support debugging on the same machine running the executable. If I want to debug a crash dump from a machine different from my own I might not be able to install the packages.
no
> How do I get gdb to show me the command
With the "run" command. You get a message "Missing separate debuginfos, use:" and then debuginfo-install, zypper etc. depending on the distro.
In Ubuntu I found it slightly confusing to locate debug symbol packages. I understood (or misunderstood) that there are 2 options where to find debug symbols and it just depends on the package maintainer how it's done. I did not find how to install debugsource at all.
By doing debugsource I mean the source that has all the patches applied exactly as during the build and is in the same path as the symbols suggest.
It’s a much better mechanism than anything I’ve looked at in the Linux space (caveat: I have not been developing C++ for a while).
I was so impressed I started looking at what it would take to implement for the gcc/gdb toolchain but discovered Microsoft held a patent and gave up.
That said, debian started indexing build ids in the repos so at least for dwarf files there is a barebones index now. Sadly not for the companion executables.
Debuginfod does support debian packages as well (integrating with -dbg pacages), but it does not support src packages. The documentation mentions that this is tricky.
[0] https://github.com/edolstra/dwarffs/blob/master/README.md