Well every programmer probably made a mistake like that once, twice or more (or just, every programmers makes mistakes tout-court), but imo something like this should have been caught in the pull request/code review/general testing stage as it concerns a major OS? That being said I obviously have no idea what stages precede an actual release of an OS. Anyone does?
The bug is triggered by using a custom memory allocator. It's clearly an edge case. There will be bugs that slip over code review and/or testing no matter what. That's a fact in software development.
Sure, but the thing is: one developper (or a group of) said 'well let's enable this' and then nobody else reviewed that or thought 'hmm, surely it can't be good to just start logging pretty much everything'? Just wondering how that happens in large companies.
>In OS X 10.10, this logging was turned on by default for some applications that use a custom memory allocator, such as jemalloc, because of an initialization bug in the framework.
Nobody explicitly decided to enable the logging by default. It was caused by an uninitialized variable. Uninitialized variables result in undefined behavior, but can often go unnoticed in practice if the code around them results in the value in memory (or register) for that variable usually being a certain value. In this case, it happened to result in logging being disabled by default in most situations. But using jemalloc for the process rather than the standard malloc implementation was enough to cause the opposite result, apparently due to jemalloc poisoning malloc()ed regions (which overwrote the "usual" value).
And in case anyone thinks that sheer developer machismo (or 'rigor' or whatever) can keep you out of harm's way here, then I suggest reading John Carmack's excellent writeups on the horrors found by static code analysis.[1][2] Even the best programmers out there will get caught by any number of language traps.
[1] Sadly/happily, still available via archive.org. CSS seems to be busted here, so scroll to the bottom for Carmack's article:
Very very few OSX apps use an alternate allocator so this was probably not noticed. Having written a commercial memory allocator before (for MacOS prior to OSX) the one in OSX is actually quite good for most people. I don't know why Mozilla decided they needed jemalloc but maybe it tested better for their usage.
Still you would think tracking what's written to such a common directory would have been seen by someone at Apple. Firefox isn't exactly obscure.
It looks like that was driven by cross-platform concerns – I don't know how good the OS X allocator was in that era but it looks like most of the benchmarking focused on Windows and having a single malloc which performed well on all of their platforms. At that time, Firefox still supported at least as far back as Windows 2000 and the rise of consumer Mac usage hadn't reached current levels.
The WinHeap allocator before 7 was pretty awful, but apparently improved with 7. Chrome is actually using it (or at least thinking of it). It still amazes me that allocation is still a hard problem 15 years after I wrote mine.
The performance of the memory allocator is very much load-specific on darwin these days, and it would be easy to justify jemalloc. It's certainly not a poor choice of an allocator.
Agreed – absent some sort of benchmark showing major, hard-to-solve problems I'd assume that using one decent allocator everywhere would be a net win versus having to deal with platform-specific variations on at least 3 OSes.
OpenSSL came with its own memory allocator, which is basically an optimistic allocator and caching wrapper around libc malloc()/free() that tries to save calls to malloc() by not actually free()ing blocks immediately but holding onto them for some time. This allows them to skip the actual memory allocation as long as they can accommodate any incomming malloc() calls from their unused blocks buffer, and if it actually has to allocate fresh memory it doesn't just allocate the amount of memory needed but speculatively allocates more and stashes the excess in its unused blocks buffer.
One of the artifacts of this behavior is that OpenSSL data isn't evenly spread throughout a processes heap, which is what one would expect if the memory was allocated purely on demand, but tends to cluster in regions that can be fairly long-lived. Now, what actually happens with Heartbleed is that it can only leak data directly adjacent in virtual memory to the data segment containing the Heartbeat payload, but due to that clustering of memory in use by OpenSSL the chances are fairly high this isn't just some random block of memory from the application's heap but a block containing other OpenSSL data. Like decrypted HTTPS requests, with passwords and stuff.
24 comments
[ 2.8 ms ] story [ 66.6 ms ] thread>In OS X 10.10, this logging was turned on by default for some applications that use a custom memory allocator, such as jemalloc, because of an initialization bug in the framework.
So it seems to be a bug that it even got enabled.
And in case anyone thinks that sheer developer machismo (or 'rigor' or whatever) can keep you out of harm's way here, then I suggest reading John Carmack's excellent writeups on the horrors found by static code analysis.[1][2] Even the best programmers out there will get caught by any number of language traps.
[1] Sadly/happily, still available via archive.org. CSS seems to be busted here, so scroll to the bottom for Carmack's article:
[2] The above was discussed on HN, and that thread is pretty good as well: https://news.ycombinator.com/item?id=3388290Still you would think tracking what's written to such a common directory would have been seen by someone at Apple. Firefox isn't exactly obscure.
http://blog.pavlov.net/2007/12/04/vlad-and-analysis-of-dtrac... http://blog.pavlov.net/2008/03/11/firefox-3-memory-usage/
It looks like that was driven by cross-platform concerns – I don't know how good the OS X allocator was in that era but it looks like most of the benchmarking focused on Windows and having a single malloc which performed well on all of their platforms. At that time, Firefox still supported at least as far back as Windows 2000 and the rise of consumer Mac usage hadn't reached current levels.
It's fixed now and anything you had in there was already auto-deleted at some point. You're likely not missing anything.
One of the artifacts of this behavior is that OpenSSL data isn't evenly spread throughout a processes heap, which is what one would expect if the memory was allocated purely on demand, but tends to cluster in regions that can be fairly long-lived. Now, what actually happens with Heartbleed is that it can only leak data directly adjacent in virtual memory to the data segment containing the Heartbeat payload, but due to that clustering of memory in use by OpenSSL the chances are fairly high this isn't just some random block of memory from the application's heap but a block containing other OpenSSL data. Like decrypted HTTPS requests, with passwords and stuff.
"/tmp/CGLog_Google Earth_7790"
full of:
7629.0413465 (Google Earth): CGSGetNextEventRecordInternal: 7629.0410916 loc (90.4, 934.0) conn 0x34b5f MouseMoved win 0x4a0 (click 1) 7629.0581342 (Google Earth): CGSGetNextEventRecordInternal: 7629.0578892 loc (84.4, 929.0) conn 0x34b5f MouseMoved win 0x4a0 (click 1) 7629.0750728 (Google Earth): CGSGetNextEventRecordInternal: 7629.0749128 loc (70.4, 918.0) conn 0x34b5f MouseExited win 0x4a0 tArea 0x19494 7629.0750766 (Google Earth): CGSGetNextEventRecordInternal: 7629.0749338 loc (70.4, 918.0) conn 0x34b5f MouseExited win 0x4a0 tArea 0x19493