Maybe it’s just me but if I saw a stack trace like that I would think that I had some memory corruption somewhere ( I would think even in this case it’s more plausible than the mutex being allocated misaligned )
Thank you for writing kernel futex code. And your kernel articles are the best, thank you very much for writing those! I am sending love and positive vibes your way!
Do these crash reporting tools have mechanisms in place to not leak sensitive data in the coredumps? Would be awkward if a firefox coredump upload would leak my session cookies.
Unfortunately, removal of PIIs also means less data/cues for hard-to-diagnose bugs (it can even range to CPU bugs like [1]). For Firefox, potentially PII-containing memory dumps are still sent to the server, but they're only viewable by authorized personnel [2].
There are also rule-based PII removal methods as used by Sentry minidump processing and described at [3].
That's almost impossible, and one of the reasons automatic crash reporting is problematic.
It is an issue with e.g medical data: Doctors using mswindows or any browser based tool can at any moment export random medical data to god knows which country. In the EU, it is probably illegal to even look at it if you receive it, and you don't know which dreports even have this issue.
Afaik, the only reason things aren't crashing down is total non-enforcement by all authorities. Which raises interesting question about their will and capability to enforce e.g. the medical secret at all.
Automatic crash reporting doesn't necessarily contain sensitive info, it depends on how much detail is provided. When I get a crash on macOS, the reporter lets me the report before it's sent, and as far as I can tell it's just plaintext stacktraces of active threads, plus a handful of other details.
As I understand a 'core-dump' is not just a stack trace, but also contains some of the arguments passed in the functions in the stack-trace. Those can easily contain sensitive information. Especially if the heap is also included.
macOS (and iOS) do the stack unwinding on the device. The "backend" then only has to perform symbolication (mapping addresses to symbol names). This is how both the built-in macOS/iOS crash reporter work, as well as the third-party PLCrashReporter and KSCrash crash handlers.
On Linux and Android, you can similarly do unwinding using something like libunwind on the device. However, the traditional solution is that a "core" file is dumped and then is unwound later.
Years ago Microsoft invented a format called the "minidump" which is basically the stack for each thread, CPU registers, and other data. This format was adopted by Google Breakpad, and which ended up being used by both Firefox and Chrome for their crash reporting. The minidump format is often used when for whatever reason, unwinding can't be done on the device. It's a fairly flexible binary format.
Breakpad can perform sanitation before sending the minidump off the device, but this can be hit or miss.
Sentry (and probably other services) support minidumps among other formats, but minidumps are a PITA and I'll bet they hate supporting them. It's a large file despite its name, and it's expensive, complicated, and risky to unwind on the backend. The only open-source minidump unwinder used to be fairly esoteric C++ code that was part of Breakpad, but there's now a rust-based unwinder too.
Breakpad came up with something called a microdump, which is an even smaller version of a minidump that's text based instead of binary.
Both Apple and Google make mobile crash reporting way harder than it need be. Android and iOS have their own crash handlers that can perform on-device out-of-process unwinding, but the reports are not available to the app itself. You only get whatever reports Apple and Google feel like showing you after they get uploaded to their systems.
So apps that want more detail have to embed their own crash handlers. Because those handlers are operating in the same memory space as the app which just crashed, their operation is somewhat fraught with peril. At least on iOS, unwinding is straightforward due to mandatory frame pointes, and symbolication is easy because Apple makes iOS symbols available.
Android is a cluster fuck when it comes to native code crashes. You may or may not have frame pointers. So unwinding on the device is hit or miss. You also have to deal with both native code and ART (Android Run Time) code. Tracing back through the runtime is non-trivial to impossible. And you will almost definitely not have debugging symbols. So good luck with symbolication. The lack of debugging symbols also make off-device unwinding basically useless, so a minidump often isn't that helpful.
I designed and built the mobile crash reporting solution for my company about a decade ago, though we're currently migrating to a SAAS solution. On Android I evolved the solution from minidumps to microdumps to microdumps + on-device unwinding. It also turns out that there's a lot of helpful information sent to logcat and including that with native crash reports can often be more helpful than whatever you can capture from the stack during the crash.
As I understand it, the windows minidump contains the stack data for each thread. This is more than the stack trace: It also contains local variables. Today, most interesting things are probably pointers outside the minidump, but it is not uncommon in e.g. C to create a local array and dump some strings in them. Then the problem comes back.
Yes, when I wrote "which is basically the stack for each thread" I did mean the raw bytes of the stack. I see now that wasn't clear. I was writing on mobile and apologize for the confusion.
Unwinding then occurs later by walking through the stack memory, which is difficult to impossible unless either: a) the stack contains frame pointers; or b) you have unwind data. Without either of those the unwinder uses heuristics to guess at return addresses on the stack, but that's very unreliable and often leads the unwinder into a loop till it gives up.
x/w is a gdb command to dereference and print (x) 1 word-sized data (w) at the address given by the expression. So 0x08001ea1 was the unaligned address stored at $rsp + 8, passed as argument to __lll_lock_wait.
19 comments
[ 3.7 ms ] story [ 57.8 ms ] threadThere are also rule-based PII removal methods as used by Sentry minidump processing and described at [3].
[1]: https://news.ycombinator.com/item?id=37063459
[2]: https://crash-stats.mozilla.org/documentation/protected_data...
[3]: https://docs.sentry.io/product/data-management-settings/scru...
It is an issue with e.g medical data: Doctors using mswindows or any browser based tool can at any moment export random medical data to god knows which country. In the EU, it is probably illegal to even look at it if you receive it, and you don't know which dreports even have this issue.
Afaik, the only reason things aren't crashing down is total non-enforcement by all authorities. Which raises interesting question about their will and capability to enforce e.g. the medical secret at all.
On Linux and Android, you can similarly do unwinding using something like libunwind on the device. However, the traditional solution is that a "core" file is dumped and then is unwound later.
Years ago Microsoft invented a format called the "minidump" which is basically the stack for each thread, CPU registers, and other data. This format was adopted by Google Breakpad, and which ended up being used by both Firefox and Chrome for their crash reporting. The minidump format is often used when for whatever reason, unwinding can't be done on the device. It's a fairly flexible binary format.
Breakpad can perform sanitation before sending the minidump off the device, but this can be hit or miss.
Sentry (and probably other services) support minidumps among other formats, but minidumps are a PITA and I'll bet they hate supporting them. It's a large file despite its name, and it's expensive, complicated, and risky to unwind on the backend. The only open-source minidump unwinder used to be fairly esoteric C++ code that was part of Breakpad, but there's now a rust-based unwinder too.
Breakpad came up with something called a microdump, which is an even smaller version of a minidump that's text based instead of binary.
Both Apple and Google make mobile crash reporting way harder than it need be. Android and iOS have their own crash handlers that can perform on-device out-of-process unwinding, but the reports are not available to the app itself. You only get whatever reports Apple and Google feel like showing you after they get uploaded to their systems.
So apps that want more detail have to embed their own crash handlers. Because those handlers are operating in the same memory space as the app which just crashed, their operation is somewhat fraught with peril. At least on iOS, unwinding is straightforward due to mandatory frame pointes, and symbolication is easy because Apple makes iOS symbols available.
Android is a cluster fuck when it comes to native code crashes. You may or may not have frame pointers. So unwinding on the device is hit or miss. You also have to deal with both native code and ART (Android Run Time) code. Tracing back through the runtime is non-trivial to impossible. And you will almost definitely not have debugging symbols. So good luck with symbolication. The lack of debugging symbols also make off-device unwinding basically useless, so a minidump often isn't that helpful.
I designed and built the mobile crash reporting solution for my company about a decade ago, though we're currently migrating to a SAAS solution. On Android I evolved the solution from minidumps to microdumps to microdumps + on-device unwinding. It also turns out that there's a lot of helpful information sent to logcat and including that with native crash reports can often be more helpful than whatever you can capture from the stack during the crash.
Unwinding then occurs later by walking through the stack memory, which is difficult to impossible unless either: a) the stack contains frame pointers; or b) you have unwind data. Without either of those the unwinder uses heuristics to guess at return addresses on the stack, but that's very unreliable and often leads the unwinder into a loop till it gives up.
https://support.backtrace.io/hc/en-us/articles/360040517131-...
So there are at least some efforts to do it. IMO it can only ever be best effort, but it doesn't mean that it's not worth doing.
In my case I don't understand where x and w come from, and how the result of 0x08001ea1 is invalid (I can see it's not aligned to a 4 byte boundary).