Vendor provided an outlook plugin (ew) that linked storage directly in outlook (double ew) and contained a built in pdf viewer (disgusting) for law firms to manage their cases.
One user, regardless of PC, user account or any other isolation factor, would reliably crash the program and outlook with it.
She could work for 40 minutes on another users logged in account on another PC and reproduce the issue.
Turns out it was a memory allocation issue. When you open a file saved in the addons storage, via the built in pdf viewer, it would allocate memory for it. However, when you close the pdf file, it would not deallocate that memory. After debugging her usage for some time, I noted that there was a memory deallocation, but it was performed at intervals.
If there were 20 or so pdf allocations and then she switched customer case file before a deallocation, regardless of available memory, the memory allocation system in the addon would shit the bed and crash.
This one user, an absolute powerhouse of a woman I must say, could type 300 wpm and would rapidly read -> close -> assign -> allocate -> write notes faster than anyone I have ever seen before. We legitimately got her to rate limit herself to 2 files per 10 minutes as an initial workaround while waiting for a patch from the vendor.
I had to write one hell of a bug report to the vendor before they would even look at it. Naturally they could not reproduce the error through their normal tests and tried closing the bug on me several times. The first update they rolled out upped it to something like 40 pdfs viewed every 15 minutes. But she still managed to touch the new ceiling on occasion (I imagine billing each of those customers 7 minutes a pop or whatever law firms do) and ultimately they had to rewrite the entire memory system.
> Vendor provided an outlook plugin (ew) that linked storage directly in outlook (double ew) and contained a built in pdf viewer (disgusting) for law firms to manage their cases.
I still don't understand how we've arrived at this state of affairs
Look I supported a few different legal platforms in that role and while I hated it, it was also the best.
Heres what a lawyer does:
1. They bill for time writing emails and on phone calls
2. They bill for time reviewing emails.
3. They bill for printing (and faxing if they are diehards)
4. They also bill for the time they are face to face with a human.
They also need to gather all the data, much of which flows in and out via email (or fax if they hate you) related to the case in a single space.
The sad state is that 80% of this can be achieved in outlook without much effort. Setting up an external application to capture all this shit is quite difficult, and generally requires mail to be run through it in some capacity. The question is, why reinvent the email client. (Sadly they reinvented the pdf reader) I have seen some lawfirms literally saving out every email as html, and uploading it with billing stats to a third party app. Its easier for me to support but the user experience can be awful.
The user already exists in Outlook, they already understand outlook. A few buttons in the ribbon (Mostly File this under X open case, time me, and bill this customer) make more sense from a user perspective.
From a support perspective its an absolute nightmare. Microsoft absolutely wont take a support case about an addon with shit memory management. And the addon provider will usually blame Microsoft.
This is close enough to the "can't log in to computer when standing up" bug... someone had swapped the keycaps for D/F (for example) so when 5-star general tried to log in when standing up he was typing "doobar" instead of "foobar" into the password field.
With the lady, if she'd dialed it back a bit on her pace of work "because people are watching", that could have been a crazy one to debug... "only happens when no one is watching (and I'm not beastly-WPM closing cases)"
I suppose the Google Doc team initially thought this would surely be a bug in their own code, not in Chrome or in V8, so it wouldn't help to bisect their own code. Nobody really begins to debug by blaming the compiler.
> It didn’t correspond to a Google Docs release. The stack trace added very little information. There wasn’t an associated spike in user complaints, so we weren’t even sure it was really happening — but if it was happening it would be really bad. It was Chrome-only starting at a specific release.
That sounds like a Chrome bug. Or, at least, a bug triggered by a change in Chrome. Bisecting your code when their change reveals a crash is folly, regardless of whose bug it is.
If your job is to solve the situation, your best hope is to figure out what change caused it; understand that change; and then do whatever needs to be done.
In a large complicated application where a change to the environment revealed a crash, finding out what changed in the environment and thinking about how that affects the application makes a lot more sense than going back through application changes to see if you can find it that way.
Once you figure out what the problem is, sure you can probably fix it in the application or the environment, and fixing the application is often easier if the environment is Chrome. But chrome changed and my app is broken means look at the changes in Chrome and work from there.
Interesting writeup, but 2 days to debug “the hardest bug ever”, while accurate, seems a bit overdone.
Though abs() returning negative numbers is hilarious.. “You had one job…”
To me, the hardest bugs are nearly irreproducible “Heisenbugs” that vanish when instrumentation is added.
I’m not just talking about concurrency issues either…
The kind of bug where a reproduction attempt takes a week, not parallelizable due to HW constraints, and logging instrumentation makes it go away or fail differently.
Author here! I debugged a fair number of those when I was a systems engineer in soft real time robotics systems, but none of them felt as bad in retrospect because you're just reading up on the system and mulling over it and eventually you get the answer in a shower thought. Maybe I just find the puzzle of them fun, I don't know why they don't feel quite so bad. This was just an exhausting 2-day brute-force grind where it turned out the damn compiler was broken.
I also came to the comments to weigh in on my perception of how rough this was, but instead will ask:
Regarding "exhausting 2-day brute-force grind": is/was this just how you like to get things done, or was there external pressure of the "don't work on anything else" sort? I've never worked at a large company, and lots of descriptions of the way things get done are pretty foreign to me :). I am also used to being able to say "this isn't getting figured out today; probably going to be best if I work on something else for a bit, and sleep on it, too".
The fatal error volume was so overwhelming that we didn't have any option but understanding the problem in perfect detail so that we could fix it if the problem was on our side, or avoid it if it was caused by something like our compiler or the browser.
Our team also had a very grindy culture, so "I'm going to put in extra hours focusing exclusively on our top crash" was a pretty normalized behavior. After I left that team (and Google), most of my future teams have been more forgiving on pace for non-outages.
The kind of bug where a reproduction attempt takes a week, not parallelizable due to HW constraints, and logging instrumentation makes it go away or fail differently.
The hardest one I've debugged took a few months to reproduce, and would only show up on hardware that only one person on the team had.
One of the interesting things about working on a very mature product is that bugs tend to be very rare, but those rare ones which do appear are also extremely difficult to debug. The 2-hour, 2-day, and 2-week bugs have long been debugged out already.
That reminded me of a former colleague at the desk next to me randomly exclaiming one day that he had just fixed a bug he had created 20 years ago.
The bug was actually quite funny in a way: it was in the code displaying the internal temperature of the electronics box of some industrial equipment. The string conversion was treating the temperature variable as an unsigned int when it was in fact signed. It took a brave field technician in Finland in winter, inspecting a unit in an unheated space to even discover this particular bug because the units' internal temperatures were usually about 20C above ambient.
This is a surprisingly common mistake with temperature readings. Especially when the system has a thermal safety power off that triggers if it's above some temperature, but then interprets -1 deg C as actually 255 deg C.
The rollout is still happening, but the new resident water meters for Victoria, Australia come with a temperature fix.
Prior to this year, they could only handle 0-127 degrees for the water temperature. Which used to be sensible, but there were some issues with pressurised water starting to be delivered to houses resulting in negative temperatures being reported, like -125C, which immediately has the water switch off to prevent icing problems.
The software side also switched from COBOL to Ada. So that's kewl.
My brother is a wifi expert at a hw manufacturer. He once had a case where the customer had issues setting the transmit power to like 100 times the legal limit. They happened to be an offshore drilling platform and had an exemption for the transmission power as their antenna was basically on a buoy on the ocean. He had to convince the developer to fix that very specific bug.
During the time I was working on a mature hardware product in maintenance, if I think about the number of customer bugs we had to close due to being not-reproducible or were only present for a brief amount of time in specific setup, it was really embarassing and we felt like a bunch of noobs.
This repro was a few times per day, but try fixing a Linux kernel panic when you don't even have C/C++ on your resume, and everyone who originally set stuff up has left...
Same here, we had an IE8 bug that prevented the initial voice over of the screen reader (JAWS). No dev could reproduce it because we all had DevTools open.
I can't remember the actual bug now, but one of my early career memories was hunting down an IE7 issue by using bookmarklets to alert() values. (Did IE7 even have dev tools?)
There was a downloadable developer toolbar for IE6 and IE7, and scripts could be debugged in the external Windows Script Debugger. The developer toolbar even told you which elements had the famous hasLayout attribute applied, which completely changed how it was rendered and interacted with other objects, which was invaluable.
"To me, the hardest bugs are nearly irreproducible “Heisenbugs” that vanish when instrumentation is added."
My favourite are bugs, that not only don't appear in the debugger - but also don't reproduce anymore on normal settings after I took a closer look in the debugger (Only to come back later at a random time).
Feels like chasing ghosts.
I don't think taking how long something took to debug in number of days is at all interesting. Trivial bugs can take weeks to debug for a noob. Insanely hard bugs takes hours to debug for genius devs, maybe even without any reproducer, just by thinking about it.
I always refer to them as “quantum bugs” because the act of observing the bug changes the bug. Absolutely infuriating. I like “heisenbug” better. Has a better ring to it.
Yes ! I've dealt with complex issues that turned out to be vendor-swapped-hardware-woopsie which we spent over a month trying to solve in software before finally figuring it out.
Part of it was difficulty of pinpointing the actual issue - fullness of drive vs throughput of writes.
A lot of it was unfortunately organizational politics such that the system spanned two teams with different reporting lines that didn't cooperate well / had poor testing practices.
Sometimes it isn't outright lying. I have had the issues with hardware, API and SDK documentation being subtly different from the product as shipped. With hardware with a mixture of revisions, some conforming to doco and other differing and even their engineers not being clear about which is which.
For stuff like this we used in-memory ring buffer logger that printed the logs on request. And it didn't save the strings, just necessary data bits and a pointer to formatting function. Writing to this logger didn't affect any timings.
In hardware, you regularly see behavior change when you probe the system. Your oscilloscope or LA probes affect the system just enough to make a marginal circuit work. It's absolutely maddening.
> To me, the hardest bugs are nearly irreproducible “Heisenbugs” that vanish when instrumentation is added.
A favourite of mine was a bug (specifically, a stack corruption) that I only managed to see under instrumentation. After a lot of debugging turns out that the bug was in the instrumentation software itself, which generated invalid assembly under certain conditions (calling one of its own functions with 5 parameters even though it takes only 4). Resolved by upgrading to their latest version.
The early-to-mid-90s "High C/C++" compiler had a bug in its floating point library for basic math functions. It ended up being a bit of a Heisenbug to track down, and I didn't initially believe it wasn't my code, but it actually ended up being in their supplied library.
It took me maybe three days to track down, from first clues to final resolution, on a 486/50 luggable with the orange on black monochrome built-in screen.
I’m not even close to being on par with other faang engineers but this is far from being a very difficult bug in my experience. The hardest bugs are the ones where the repro takes days to repro. But nonetheless the op’s tenacity is all that matters and I would trust them to solve any of the hard problems Ive faced in the past.
Hi, author here! At my job before Google I had to debug these kinds of bugs for our mobile robotics / computer vision stack, but I found them fun so they didn't feel "hard" per se. The most time-consuming one took a month on basically a camera-mounted computer vision system, where after an hour of use the system would start stuttering unusably. But the journey took us through heat throttling on 2009-era gaming laptops, esoteric windows APIs, hardware design, and ultimately distributed queuing. But fixing it was a blast! I learned a ton. I hated that project but fixing that bug was the highlight of it.
FWIW: this type of bug in Chrome is exploitable to create out-of-bounds array accesses in JIT-compiled JavaScript code.
The JIT compiler contains passes that will eliminate unnecessary bounds checks. For example, if you write “var x = Math.abs(y); if(x >= 0) arr[x] = 0xdeadbeef;”, the JIT compiler will probably delete the if statement and the internal nonnegative array index check inside the [] operator, as it can assume that x is nonnegative.
However, if Math.abs is then “optimized” such that it can produce a negative number, then the lack of bounds checks means that the code will immediately access a negative array index - which can be abused to rewrite the array’s length and enable further shenanigans.
> which can be abused to rewrite the array’s length and enable further shenanigans.
I followed all of this up until here. JavaScript lets you modify the length of an array by assigning to indexes that are negative? I'm familiar with the paradigm of negative indexing being used to access things from the end of the array (like -1 being the last element), but I don't understand what operation someone could do that would somehow modify the length of the array rather than modifying a specific element in-place. Does JIT-compiled JavaScript not follow the usual JavaScript semantics that would normally happen when using a negative index, or are you describing something that would be used in combination with some other compiler bug (which honestly sounds a lot more severe even in the absence of an usual Math.abs implementation).
>I followed all of this up until here. JavaScript lets you modify the length of an array by assigning to indexes that are negative?
This is my no doubt dumb understanding of what you can do, based on some funky stuff I did one time to mess with people's heads
do the following
const arr = [];
arr[-1] = "hi";
console.log(arr)
this gives you
"-1": "hi"
length: 0
which I figured is because really an array is just a special type of object. (my interpretation, probably wrong)
now we can see that the JavaScript Array length is 0, but since the value is findable in there I would expect there is some length representation in the lower level language that JavaScript is implemented in, in the browser, and I would then think that there could even be exploits available by somehow taking advantage of the difference between this lower level representation of length and the JS array length. (again all this is silly stuff I thought and have never investigated, and is probably laughably wrong in some ways)
I remember seeing some additions to array a few years back that made it so you could protect against the possibility of negative indexes storing data in arrays - but that memory may be faulty as I have not had any reason to worry about it.
You raise a good point that JavaScript arrays are "just" objects that let you assign to arbitrary properties through the same syntax as array indexing. I could totally imagine some sort of optimization where a compiler utilizes this to be able to map arrays directly to their underlying memory layout (presumably with a length prefix), and that would end up potentially providing access to it in the case of a mistaken assumption about omitting a bounds check.
yeah you know what you said made me think about these funny experiments that I haven't done in a long time and I remember now yeah, you can do
const arr = [];
arr[false] = "hi";
which console.log(arr); - in FF at least - gives
Array []
false: "hi"
length: 0
which means
console.log(arr[Boolean(arr.length)]); returns
hi
which is funny, I just feel there must be an exploit somewhere among this area of things, but maybe not because it would be well covered.
on edit: for example since the index could be achieved - for some reason - from numeric operation that output NaN, you would then have NaN: "hi", or since the arr[-1] gives you "-1": "hi" but arr[0 -1] returns that "hi" there are obviously type conversions going on in the indexing...which just always struck me as a place you don't expect the type conversions to be going on the way you do with a == b;
Maybe I am just easily freaked out by things as I get older.
Normally, there would be a bounds check to ensure that the index was actually non-negative; negative indices get treated as property accesses instead of array accesses (unlike e.g. Python where they would wrap around).
However, if the JIT compiler has "proven" that the index is never non-negative (because it came from Math.abs), it may omit such checks. In that case, the resulting access to e.g. arr[-1] may directly access the memory that sits one position before the array elements - which could, for example, be part of the array metadata, such as the length of the array.
You can read the comments on the sample CVE's proof-of-concept to see what the JS engine "thinks" is happening, vs. what actually happens when the code is executed: https://github.com/shxdow/exploits/blob/master/CVE-2020-9802.... This exploit is a bit more complicated than my description, but uses a similar core idea.
I understand the idea of the lack of a bounds check allowing access to early memory with a negative index, but I'm mostly struggling with wrapping my head around why the underlying memory layout is accessible in JavaScript in the first place. I hadn't considered the fact that the same syntax could be used for accessing arbitrary properties rather than just array indexes; that might be the nuance I was missing.
Because after bound checks have been taken care of, loading an element of a JS array probably compiles to a simple assembly-level load like mov. If you bypass the bounds checks, that mov can read or write any mapped address.
Yeah, I understand all of that. I think my surprise was that you can access arbitrary parts of this struct from within JavaScript at all; I guess I really just haven't delved deeply enough into what JIT compiling actually is doing at runtime, because I wouldn't have expected that to be possible.
When I was 12 I was just learning stuff and wrote something in C, which crashed at unpredictable intervals and I could not explain it. I took it to my 14 year old uncle who was better than me at coding for help. Now mind you this is ~ 40 years ago but I seem to remember that Borland Turbo C (I still love that IDE blue color) had debugging with breakpoints (mind blowing!) which eventually led to "duh you didn't dispose of your pointer and are reusing it and the memory there is now garbage" or something like that. I vaguely recall * or * being somewhere nearby. This was my first intro to RTFM and debugging and what a powerful intro.
> Then we called in our Tech Lead / Manager, who had a reputation of being a human JavaScript compiler. We explained how we got here, that Math.abs() is returning negative values, and whether she could find anything that we were doing wrong. After persuading her that we weren’t somehow horribly mistaken, she sat down and looked at the code. Her CPU spun up to 100%, and she was muttering in Russian about parse trees or something while staring at the code and typing into the debug console. Finally she leaned back and declared that Math.abs() was definitely returning negative values for negative inputs.
Complaining about "slow to reproduce" and talking _seconds_. Dear, oh dear those are rookie numbers!
Currently working a bug where we saw file system corruption after 3 weeks of automated testing, 10s of thousands of restarts. We might never see the problem again, even? Only happened once yet.
If it only happened once... it might be the final category of bugs where nothing you can do will fix it. Cosmic ray bit flipping bug. Which is something your software needs to be able to work around, or in this case, the file system itself... unless you're actually working on the file system itself, in which case, I wish you good luck.
Anything can fail, at any time. The best we can do is mitigate it and estimate bounds for how likely it is to mess up. Sometimes those bounds are acceptable.
So far my record is 3 weeks. It was a hiesenbug triggered when two different ebpf based systems raced with each other. Ebpf is a great tool in the right place but is it ever a pain in the ass to debug.
The fix ended up being one character -> change the priority of an ebpf tc filter from 0 to 1.
They have forums if you can find them, I think they call them 'communities', where you can complain.
Then a high-ranked non-employee 'product expert' will be along presently to tell you that's not really a problem and to stop bothering the almighty google with such trivialities, your views are not important and they have millions of users, really why should they listen to you?
It seems to me that V8 had very bad unit tests if this wasn't caught before release. Making sure all operators act the same way when optimized and not is a no-brainer.
Fair enough, it's busywork and easy to postpone. But code optimization is something that needs this kind of double-checking, so in the end you should have it for all opcodes, and then including the easy ones like abs isn't much extra work.
It sounds like their unit-tests cover abs(), but they weren't covering all of abs(), and were not reliably triggering the optimized codepath:
> When doing the refactoring, they needed to provide new implementations for every opcode. Someone accidentally turned Math.abs() into the identity function for the super-optimized level. But nobody noticed because it almost never ran — and was right half of the time when it did.
If it never was tested, plain and simple as that, then it couldn't matter that it 'almost never ran' or 'was right half the time'.
So the root problem here is that their test-suite neither exercised all optimized levels appropriately, nor flagged the omission as a fatal problem breaking 100% branch coverage (which for a simple primitive like abs you'd definitely want). This meant that they could break lots of other things too without noticing. OP doesn't discuss if the JS team dealt with it appropriately; one hopes they did.
And somewhere out there is a person reading this post and coming to the conclusion "How can Google be stupid enough to hire people stupid enough to have abs() return a negative value."
Love the story! There is so much complexity in the world around as that seemingly obviously wrong things happen through the most unlikely chains of dependency.
> And somewhere out there is a person reading this post and coming to the conclusion "How can Google be stupid enough to hire people stupid enough to have abs() return a negative value."
Weird things can happen anywhere but I was wondering why this issue wasn't caught by test cases before it escaped to production? I would think that a compiler team would have low-level tests for such common functions.
Honestly, of all the stupid ideas, having your engine switch to a completely untested mode when under heavy load, a mode that no one ever checks and it might take years to discover bugs in, is absolutely one of most insane things I can think of. That's at best really lazy, and at worst displays a corporate culture that prizes superficial performance over reliability and quality. Thankfully no one's deploying V8 in, like, avionics. I hope.
At least this is one of those bugs you can walk away from and say, it really truly was a low-level issue. And it takes serious time and energy to prove that.
I agree with your assessment of how stupid this is, but I'm not surprised.
To be clear, there are good reasons for this different mode. The fuck-up is not testing it properly.
These kinds of modes can be tested properly in various ways, e.g. by having an override switch that forces the chosen mode to be used all the time instead of using the default heuristics for switching between modes. And then you run your test suite in that configuration in addition to the default configuration.
The challenge is that you have now at least doubled the time it takes to run all your tests. And with this kind of project (like a compiler), there are usually multiple switches of this kind, so you very quickly get into combinatorial explosion where even a company like Google falls far short of the resources it would require to run all the tests. (Consider how many -f flags GCC has... there aren't enough physical resources to run any test suite against all combinations.)
The solution I'd love to see is stochastic testing. Instead of (or, more realistically, in addition to) a single fixed test suite that runs on every check-in and/or daily, you have an ongoing testing process that continuously tests your main branch against randomly sampled (test, config) pairs from the space of { test suite } x { configuration space }. Ideally combine it with an automatic bisector which, whenever a failure is found, goes back to an older version to see if the failure is a recent regression and identifies the regression point if so.
Isn't stochastic testing becoming more and more of a standard practice? Even if you have the hardware and time to run a full testsuite, you still want to add some randomness just to catch accidental dependencies between tests.
Maybe? I'd love to hear if there are some good tools for it that can be integrated into typical setups with Git repositories, Jenkins or GitHub Actions, etc.
I wish I could recall the details better but this was 20+ years ago now. In college I had an internship working at Bose, doing QA on firmware in a new multi CD changer addon to their flagship stereo. We were provided discs of music tracks with various characteristics. And had to listen to them over and over and over and over and over and over, running through test cases provided by QA management as we did. But also doing random ad-hoc testing once we finished the required tests on a given build.
At one point I found a bug where if you hit a sequence of buttons on the remote at a very specific time--I want to say it was "next track" twice right as a new track started--the whole device would crash and reboot. This was a show stopper; people would hit the roof if their $500 stereo crashed from hitting "next". Similar to the article, the engineering lead on the product cleared his schedule to reproduce, find, and fix the issue. He did explain what was going on at the time, but the specifics are lost to me.
Overall the work was incredibly boring. I heard the same few tracks so many times I literally started to hear them in my dreams. So it was cool to find a novel, highest severity bug by coloring outside the lines of the testcases. I felt great for finding the problem! I think the lead lost 20% of his hair in the course of fixing it, lol.
I haven't had QA as a job title in a long time but that job did teach me some important lessons about how to test outside the happy path, and how to write a reproducible and helpful bug report for the dev team. Shoutout to all the extremely underpaid and unappreciated QA folks out there. It sucks that the discipline doesn't get more respect.
A friend of mine has near PTSD from watching some movie over and over and over at a optician where she worked. Was on rotation so that their customers could gauge their eyesight.
That is great QAing. It also speaks to why QA should be a real role in more orgs, rather than a shrinking discipline. Engineers LOVE LOVE LOVE to test the happy path.
It's not even malice/laziness, it's their entire interpretation of the problem/requirements drives their implementation which then drives their testing. It's like asking restaurants to self-certify they are up to food safety codes.
If you do not follow the happy path something will break 100% of the time. That's why engineers always follow the happy path. Some engineers even think that anything outside the happy path is an exception and not even worth investigating. These engineers only thrives if the users are unable to switch to another product. Only competition will lead to better products.
My favorite happy path developer.. and he was by far 10x worse than any engineer I worked with at this, did the following:
Spec: allow the internal BI tool to send scheduled reports to the user
Implementation: the server required the desktop front end of said user to have been opened that day for the scheduled reports to work, even though the server side was sending the mails
Why this was hilariously bad - the only reason to have this feature is for when the user is out of office / away from desk for an extended period, precisely when they may not have opened their desktop UI for the day.
One of my favorite examples of how an engineer can get the entire premise of the problem wrong.
In the end he had taken so long and was so intransigent that desktop support team found it easier to schedule the desktop UIs to auto-open in windows scheduler every day such that the whole Rube Goldberg scheduled reports would work.
You just needed to find another one like him, and bam, +4×.
(It is actually conceivable that two bad engineers could mostly cancel each other out, if they can occupy each other enough, but it’s not the most likely outcome.)
> That is great QAing. It also speaks to why QA should be a real role in more orgs, rather than a shrinking discipline.
As a software engineer, I've always been very proud of my thoroughness and attention to detail in testing my code. However, good QA people always leave me wondering "how did they even think to do that?" when reviewing bug reports.
Pedantically pointing out the difference between doing some exploratory testing "testing outside the test cases" and QA which is setting up processes/procedures part of which should be "do exploratory testing as well as running the test cases" but the Testing is not QA distinction has been fought over for decades...
But, love the story and I collect tales like this all the time so thanks for sharing
I had one that took literally years to reproduce. It was in PLC code, on a touchscreen controller running a soft PLC with Busybox under the hood. These devices were used 24/7 and usually absolutely bullet proof. Every now and then I’d get a comment that sometimes they’d crash on startup but a power cycle usually fixed it. Finally managed to get it to happen in the workshop, and dropped everything to try and figure it out.
The ultimate cause was in the network initialisation using a network library that was a tissue-paper-thin wrapper around Linux sockets. When downloading a new software version to the device, it would halt the PLC but this didn’t cleanly shut down open sockets, which would stay open, preventing a network service from starting until the unit was restarted. So I did the obvious thing and wrote the socket handle to a file. On startup I’d check the file and if it existed, shut that socket handle. This worked great during development.
Of course this file was still there after a power cycle. 99% of the time nothing would happen, but very occasionally, closing this random socket handle on startup would segfault the soft PLC runtime. So dumb, but so hard to actually catch in the wild.
As far as I'm concerned if you can use a debugger it automatically shouldn't qualify as the most difficult ever.
As per the compute shader post from a few days ago, currently I'm "debugging" some pretty advanced code that's being ported to a shader, and the only way to do it is by creating an array of e.g. ints and inserting values into it in both the original and the shader code to see where they diverge. Its not the most difficult but its quite time consuming.
My hardest bug story, almost circling back to the origin of the word.
An intern gets a devboard with a new mcu to play with. A new generation, but mostly backwards compatible or something like that. Intern gets the board up and running with embedded equivalent of "hello world". They port basic product code - ${thing} does not work. After enough hair are pulled, I give them some guidance - ${thing} does not work. Okay, I instruct intern to take mcu vendor libraries/examples and get ${thing} running in isolation. Intern fails.
Okay, we are missing something huge that should be obvious. We start pair programming and strip the code down layer by layer. Eventually we are at a stage where we are accessing hand-coded memory addresses directly. ${thing} does not work. Okay, set up a peripheral and read state register back. Assertion fails. Okay, set up peripheral, nop some time for values to settle, read state register back. Assertion fails. Check generated assembly - nopsled is there.
We look at manual, the bit switching peripheral into the state we care about is not set. However we poke the mcu, whatever we write to control register, the bit is just not set and the peripheral never switches into the mode we need. We get a new devboard (or resolder mcu on the old one, don't remember) and it works first try.
"New device - must be new behavior" thinking with lack of easy access to the new hardware led us down a rabbit hole. Yes, nothing too fancy. However, I shudder thinking what if reading the state register gave back the value written?
what if reading the state register gave back the value written?
I've had that experience. Turned out some boards in the wild didn't have the bodge wire that connected the shift register output to the gate that changed the behavior.
It’s amusing how so many of the comments here are like “You think two days is hard? Well, I debugged a problem which was passed down to me by my father, and his father before him”. It reminds me of the Four Yorkshiremen sketch.
Yes, of course, I greatly enjoy the stories and it’s why I opened this thread. But that’s not what my comment is about, I was specifically referencing the parts of the comments which dismiss the difficulty and length of time the author spent tracking down this particular bug. I found that funny and my comment was essentially one big joke.
> What can I even do from here as the newsletter author? Normally I like finding a teachable lesson. But it was 2 days of grueling debugging and somehow there aren’t any teachable lessons there.
A lesson to learn seems obvious to me: the V8 team did not communicate upfront sufficiently on the "oops our Math.abs() may return negative numbers, we fixed that in version X, be warned".
Which the V8 should be able to do in a "advisory for Google developers that work on high-performance client-side view rendering stuff" sort of weekly newsletter.
192 comments
[ 3.0 ms ] story [ 216 ms ] threadVendor provided an outlook plugin (ew) that linked storage directly in outlook (double ew) and contained a built in pdf viewer (disgusting) for law firms to manage their cases.
One user, regardless of PC, user account or any other isolation factor, would reliably crash the program and outlook with it.
She could work for 40 minutes on another users logged in account on another PC and reproduce the issue.
Turns out it was a memory allocation issue. When you open a file saved in the addons storage, via the built in pdf viewer, it would allocate memory for it. However, when you close the pdf file, it would not deallocate that memory. After debugging her usage for some time, I noted that there was a memory deallocation, but it was performed at intervals.
If there were 20 or so pdf allocations and then she switched customer case file before a deallocation, regardless of available memory, the memory allocation system in the addon would shit the bed and crash.
This one user, an absolute powerhouse of a woman I must say, could type 300 wpm and would rapidly read -> close -> assign -> allocate -> write notes faster than anyone I have ever seen before. We legitimately got her to rate limit herself to 2 files per 10 minutes as an initial workaround while waiting for a patch from the vendor.
I had to write one hell of a bug report to the vendor before they would even look at it. Naturally they could not reproduce the error through their normal tests and tried closing the bug on me several times. The first update they rolled out upped it to something like 40 pdfs viewed every 15 minutes. But she still managed to touch the new ceiling on occasion (I imagine billing each of those customers 7 minutes a pop or whatever law firms do) and ultimately they had to rewrite the entire memory system.
I still don't understand how we've arrived at this state of affairs
Heres what a lawyer does:
1. They bill for time writing emails and on phone calls 2. They bill for time reviewing emails. 3. They bill for printing (and faxing if they are diehards) 4. They also bill for the time they are face to face with a human.
They also need to gather all the data, much of which flows in and out via email (or fax if they hate you) related to the case in a single space.
The sad state is that 80% of this can be achieved in outlook without much effort. Setting up an external application to capture all this shit is quite difficult, and generally requires mail to be run through it in some capacity. The question is, why reinvent the email client. (Sadly they reinvented the pdf reader) I have seen some lawfirms literally saving out every email as html, and uploading it with billing stats to a third party app. Its easier for me to support but the user experience can be awful.
The user already exists in Outlook, they already understand outlook. A few buttons in the ribbon (Mostly File this under X open case, time me, and bill this customer) make more sense from a user perspective.
From a support perspective its an absolute nightmare. Microsoft absolutely wont take a support case about an addon with shit memory management. And the addon provider will usually blame Microsoft.
With the lady, if she'd dialed it back a bit on her pace of work "because people are watching", that could have been a crazy one to debug... "only happens when no one is watching (and I'm not beastly-WPM closing cases)"
> It didn’t correspond to a Google Docs release. The stack trace added very little information. There wasn’t an associated spike in user complaints, so we weren’t even sure it was really happening — but if it was happening it would be really bad. It was Chrome-only starting at a specific release.
That sounds like a Chrome bug. Or, at least, a bug triggered by a change in Chrome. Bisecting your code when their change reveals a crash is folly, regardless of whose bug it is.
In a large complicated application where a change to the environment revealed a crash, finding out what changed in the environment and thinking about how that affects the application makes a lot more sense than going back through application changes to see if you can find it that way.
Once you figure out what the problem is, sure you can probably fix it in the application or the environment, and fixing the application is often easier if the environment is Chrome. But chrome changed and my app is broken means look at the changes in Chrome and work from there.
Though abs() returning negative numbers is hilarious.. “You had one job…”
To me, the hardest bugs are nearly irreproducible “Heisenbugs” that vanish when instrumentation is added.
I’m not just talking about concurrency issues either…
The kind of bug where a reproduction attempt takes a week, not parallelizable due to HW constraints, and logging instrumentation makes it go away or fail differently.
2 days is cute though.
Regarding "exhausting 2-day brute-force grind": is/was this just how you like to get things done, or was there external pressure of the "don't work on anything else" sort? I've never worked at a large company, and lots of descriptions of the way things get done are pretty foreign to me :). I am also used to being able to say "this isn't getting figured out today; probably going to be best if I work on something else for a bit, and sleep on it, too".
Our team also had a very grindy culture, so "I'm going to put in extra hours focusing exclusively on our top crash" was a pretty normalized behavior. After I left that team (and Google), most of my future teams have been more forgiving on pace for non-outages.
(And thanks for the war story!)
The hardest one I've debugged took a few months to reproduce, and would only show up on hardware that only one person on the team had.
One of the interesting things about working on a very mature product is that bugs tend to be very rare, but those rare ones which do appear are also extremely difficult to debug. The 2-hour, 2-day, and 2-week bugs have long been debugged out already.
The bug was actually quite funny in a way: it was in the code displaying the internal temperature of the electronics box of some industrial equipment. The string conversion was treating the temperature variable as an unsigned int when it was in fact signed. It took a brave field technician in Finland in winter, inspecting a unit in an unheated space to even discover this particular bug because the units' internal temperatures were usually about 20C above ambient.
Prior to this year, they could only handle 0-127 degrees for the water temperature. Which used to be sensible, but there were some issues with pressurised water starting to be delivered to houses resulting in negative temperatures being reported, like -125C, which immediately has the water switch off to prevent icing problems.
The software side also switched from COBOL to Ada. So that's kewl.
https://news.ycombinator.com/item?id=37859771
Point being that the difficulty of a fix can come from many possible places.
Turns out IE8 doesn't define console until the devtools are open. That caused me to pull a few hairs out.
Math.abs(Integer.MIN_VALUE) in Java very seriously returns -2147483648, as there is no int for 2147483648.
It throws an OverflowException: ("Negating the minimum value of a twos complement number is invalid.")
a = torch.tensor(-2*31, dtype=torch.int32) assert a == a.abs()
My favourite are bugs, that not only don't appear in the debugger - but also don't reproduce anymore on normal settings after I took a closer look in the debugger (Only to come back later at a random time). Feels like chasing ghosts.
Part of it was difficulty of pinpointing the actual issue - fullness of drive vs throughput of writes.
A lot of it was unfortunately organizational politics such that the system spanned two teams with different reporting lines that didn't cooperate well / had poor testing practices.
The hardest bugs in my experience are those where your only source of vital information is a third party who is straight-up lying to you.
The more abundant the undefined (mis)behavior, the more you're going to be tearing your hair out.
Almost the kind of frustration where you're supposed to have a logic-based system, and it rears it ugly head and defies logic anyway :\",
A favourite of mine was a bug (specifically, a stack corruption) that I only managed to see under instrumentation. After a lot of debugging turns out that the bug was in the instrumentation software itself, which generated invalid assembly under certain conditions (calling one of its own functions with 5 parameters even though it takes only 4). Resolved by upgrading to their latest version.
It took me maybe three days to track down, from first clues to final resolution, on a 486/50 luggable with the orange on black monochrome built-in screen.
The JIT compiler contains passes that will eliminate unnecessary bounds checks. For example, if you write “var x = Math.abs(y); if(x >= 0) arr[x] = 0xdeadbeef;”, the JIT compiler will probably delete the if statement and the internal nonnegative array index check inside the [] operator, as it can assume that x is nonnegative.
However, if Math.abs is then “optimized” such that it can produce a negative number, then the lack of bounds checks means that the code will immediately access a negative array index - which can be abused to rewrite the array’s length and enable further shenanigans.
Further reading about a Chrome CVE pretty much exactly in this mold: https://shxdow.me/cve-2020-9802/
I followed all of this up until here. JavaScript lets you modify the length of an array by assigning to indexes that are negative? I'm familiar with the paradigm of negative indexing being used to access things from the end of the array (like -1 being the last element), but I don't understand what operation someone could do that would somehow modify the length of the array rather than modifying a specific element in-place. Does JIT-compiled JavaScript not follow the usual JavaScript semantics that would normally happen when using a negative index, or are you describing something that would be used in combination with some other compiler bug (which honestly sounds a lot more severe even in the absence of an usual Math.abs implementation).
I.e. don't think fancy language shenanigans that do negative indexing. But negative offset from the beginning of the array memory access.
When there's some inlining, there will be no function call into some index operator function
This is my no doubt dumb understanding of what you can do, based on some funky stuff I did one time to mess with people's heads
do the following const arr = []; arr[-1] = "hi"; console.log(arr) this gives you "-1": "hi"
length: 0
which I figured is because really an array is just a special type of object. (my interpretation, probably wrong)
now we can see that the JavaScript Array length is 0, but since the value is findable in there I would expect there is some length representation in the lower level language that JavaScript is implemented in, in the browser, and I would then think that there could even be exploits available by somehow taking advantage of the difference between this lower level representation of length and the JS array length. (again all this is silly stuff I thought and have never investigated, and is probably laughably wrong in some ways)
I remember seeing some additions to array a few years back that made it so you could protect against the possibility of negative indexes storing data in arrays - but that memory may be faulty as I have not had any reason to worry about it.
You can do more and more in it and it's so fun, until it suddenly isn't anymore and dies.
const arr = []; arr[false] = "hi";
which console.log(arr); - in FF at least - gives
Array []
false: "hi"
length: 0
which means
console.log(arr[Boolean(arr.length)]); returns
hi
which is funny, I just feel there must be an exploit somewhere among this area of things, but maybe not because it would be well covered.
on edit: for example since the index could be achieved - for some reason - from numeric operation that output NaN, you would then have NaN: "hi", or since the arr[-1] gives you "-1": "hi" but arr[0 -1] returns that "hi" there are obviously type conversions going on in the indexing...which just always struck me as a place you don't expect the type conversions to be going on the way you do with a == b;
Maybe I am just easily freaked out by things as I get older.
However, if the JIT compiler has "proven" that the index is never non-negative (because it came from Math.abs), it may omit such checks. In that case, the resulting access to e.g. arr[-1] may directly access the memory that sits one position before the array elements - which could, for example, be part of the array metadata, such as the length of the array.
You can read the comments on the sample CVE's proof-of-concept to see what the JS engine "thinks" is happening, vs. what actually happens when the code is executed: https://github.com/shxdow/exploits/blob/master/CVE-2020-9802.... This exploit is a bit more complicated than my description, but uses a similar core idea.
the best line of the piece.
> Then we called in our Tech Lead / Manager, who had a reputation of being a human JavaScript compiler. We explained how we got here, that Math.abs() is returning negative values, and whether she could find anything that we were doing wrong. After persuading her that we weren’t somehow horribly mistaken, she sat down and looked at the code. Her CPU spun up to 100%, and she was muttering in Russian about parse trees or something while staring at the code and typing into the debug console. Finally she leaned back and declared that Math.abs() was definitely returning negative values for negative inputs.
Currently working a bug where we saw file system corruption after 3 weeks of automated testing, 10s of thousands of restarts. We might never see the problem again, even? Only happened once yet.
The fix ended up being one character -> change the priority of an ebpf tc filter from 0 to 1.
Where mere mortals can complain about Google product?
Then a high-ranked non-employee 'product expert' will be along presently to tell you that's not really a problem and to stop bothering the almighty google with such trivialities, your views are not important and they have millions of users, really why should they listen to you?
At least, that's been my experience.
That can't possibly be the hardest bug ever
> When doing the refactoring, they needed to provide new implementations for every opcode. Someone accidentally turned Math.abs() into the identity function for the super-optimized level. But nobody noticed because it almost never ran — and was right half of the time when it did.
If it never was tested, plain and simple as that, then it couldn't matter that it 'almost never ran' or 'was right half the time'.
So the root problem here is that their test-suite neither exercised all optimized levels appropriately, nor flagged the omission as a fatal problem breaking 100% branch coverage (which for a simple primitive like abs you'd definitely want). This meant that they could break lots of other things too without noticing. OP doesn't discuss if the JS team dealt with it appropriately; one hopes they did.
Love the story! There is so much complexity in the world around as that seemingly obviously wrong things happen through the most unlikely chains of dependency.
Weird things can happen anywhere but I was wondering why this issue wasn't caught by test cases before it escaped to production? I would think that a compiler team would have low-level tests for such common functions.
Oh my god.
Honestly, of all the stupid ideas, having your engine switch to a completely untested mode when under heavy load, a mode that no one ever checks and it might take years to discover bugs in, is absolutely one of most insane things I can think of. That's at best really lazy, and at worst displays a corporate culture that prizes superficial performance over reliability and quality. Thankfully no one's deploying V8 in, like, avionics. I hope.
At least this is one of those bugs you can walk away from and say, it really truly was a low-level issue. And it takes serious time and energy to prove that.
To be clear, there are good reasons for this different mode. The fuck-up is not testing it properly.
These kinds of modes can be tested properly in various ways, e.g. by having an override switch that forces the chosen mode to be used all the time instead of using the default heuristics for switching between modes. And then you run your test suite in that configuration in addition to the default configuration.
The challenge is that you have now at least doubled the time it takes to run all your tests. And with this kind of project (like a compiler), there are usually multiple switches of this kind, so you very quickly get into combinatorial explosion where even a company like Google falls far short of the resources it would require to run all the tests. (Consider how many -f flags GCC has... there aren't enough physical resources to run any test suite against all combinations.)
The solution I'd love to see is stochastic testing. Instead of (or, more realistically, in addition to) a single fixed test suite that runs on every check-in and/or daily, you have an ongoing testing process that continuously tests your main branch against randomly sampled (test, config) pairs from the space of { test suite } x { configuration space }. Ideally combine it with an automatic bisector which, whenever a failure is found, goes back to an older version to see if the failure is a recent regression and identifies the regression point if so.
At one point I found a bug where if you hit a sequence of buttons on the remote at a very specific time--I want to say it was "next track" twice right as a new track started--the whole device would crash and reboot. This was a show stopper; people would hit the roof if their $500 stereo crashed from hitting "next". Similar to the article, the engineering lead on the product cleared his schedule to reproduce, find, and fix the issue. He did explain what was going on at the time, but the specifics are lost to me.
Overall the work was incredibly boring. I heard the same few tracks so many times I literally started to hear them in my dreams. So it was cool to find a novel, highest severity bug by coloring outside the lines of the testcases. I felt great for finding the problem! I think the lead lost 20% of his hair in the course of fixing it, lol.
I haven't had QA as a job title in a long time but that job did teach me some important lessons about how to test outside the happy path, and how to write a reproducible and helpful bug report for the dev team. Shoutout to all the extremely underpaid and unappreciated QA folks out there. It sucks that the discipline doesn't get more respect.
It's not even malice/laziness, it's their entire interpretation of the problem/requirements drives their implementation which then drives their testing. It's like asking restaurants to self-certify they are up to food safety codes.
Spec: allow the internal BI tool to send scheduled reports to the user
Implementation: the server required the desktop front end of said user to have been opened that day for the scheduled reports to work, even though the server side was sending the mails
Why this was hilariously bad - the only reason to have this feature is for when the user is out of office / away from desk for an extended period, precisely when they may not have opened their desktop UI for the day.
One of my favorite examples of how an engineer can get the entire premise of the problem wrong.
In the end he had taken so long and was so intransigent that desktop support team found it easier to schedule the desktop UIs to auto-open in windows scheduler every day such that the whole Rube Goldberg scheduled reports would work.
edit: reminded me of the old joke
A programmer gets sent to the store by his wife. His wife says, “Get a gallon of milk, and if they have eggs, get a dozen.”
The programmer returns home with 12 gallons of milk and says, “They had eggs.”
> A programmer gets sent to the store by his wife. His wife says, “Get a gallon of milk. If they have eggs, get a dozen.”
You just needed to find another one like him, and bam, +4×.
(It is actually conceivable that two bad engineers could mostly cancel each other out, if they can occupy each other enough, but it’s not the most likely outcome.)
I mean, it's well known that there's very little engineering in most software "engineers", but you're describing a person I've never seen.
No, that means you're dealing with an early alpha, rigged demo, or some sort of vibe coding nonsense.
As a software engineer, I've always been very proud of my thoroughness and attention to detail in testing my code. However, good QA people always leave me wondering "how did they even think to do that?" when reviewing bug reports.
QA is both a skillset AND a mindset.
But, love the story and I collect tales like this all the time so thanks for sharing
It's like this https://geek-and-poke.com/geekandpoke/2017/8/13/just-happene... but actually true, which is really bad for mental health :D
The ultimate cause was in the network initialisation using a network library that was a tissue-paper-thin wrapper around Linux sockets. When downloading a new software version to the device, it would halt the PLC but this didn’t cleanly shut down open sockets, which would stay open, preventing a network service from starting until the unit was restarted. So I did the obvious thing and wrote the socket handle to a file. On startup I’d check the file and if it existed, shut that socket handle. This worked great during development.
Of course this file was still there after a power cycle. 99% of the time nothing would happen, but very occasionally, closing this random socket handle on startup would segfault the soft PLC runtime. So dumb, but so hard to actually catch in the wild.
As per the compute shader post from a few days ago, currently I'm "debugging" some pretty advanced code that's being ported to a shader, and the only way to do it is by creating an array of e.g. ints and inserting values into it in both the original and the shader code to see where they diverge. Its not the most difficult but its quite time consuming.
An intern gets a devboard with a new mcu to play with. A new generation, but mostly backwards compatible or something like that. Intern gets the board up and running with embedded equivalent of "hello world". They port basic product code - ${thing} does not work. After enough hair are pulled, I give them some guidance - ${thing} does not work. Okay, I instruct intern to take mcu vendor libraries/examples and get ${thing} running in isolation. Intern fails.
Okay, we are missing something huge that should be obvious. We start pair programming and strip the code down layer by layer. Eventually we are at a stage where we are accessing hand-coded memory addresses directly. ${thing} does not work. Okay, set up a peripheral and read state register back. Assertion fails. Okay, set up peripheral, nop some time for values to settle, read state register back. Assertion fails. Check generated assembly - nopsled is there.
We look at manual, the bit switching peripheral into the state we care about is not set. However we poke the mcu, whatever we write to control register, the bit is just not set and the peripheral never switches into the mode we need. We get a new devboard (or resolder mcu on the old one, don't remember) and it works first try.
"New device - must be new behavior" thinking with lack of easy access to the new hardware led us down a rabbit hole. Yes, nothing too fancy. However, I shudder thinking what if reading the state register gave back the value written?
I've had that experience. Turned out some boards in the wild didn't have the bodge wire that connected the shift register output to the gate that changed the behavior.
https://youtube.com/watch?v=sGTDhaV0bcw
The author’s “error”, of course, was calling it “the hardest bug I ever debugged”. It drives clicks, but comparisons too.
This is how humans work, and this is why I am reading the comments.
A lesson to learn seems obvious to me: the V8 team did not communicate upfront sufficiently on the "oops our Math.abs() may return negative numbers, we fixed that in version X, be warned".
Which the V8 should be able to do in a "advisory for Google developers that work on high-performance client-side view rendering stuff" sort of weekly newsletter.