Ask HN: Strange bug workarounds?
Software bugs are a fact of life and, sadly, many never see a (timely) fix. This can lead to some some unusual workarounds in order to continue using the software.
What are some unusual/quirky/bizarre workarounds to software bugs that have been encountered by the HN crowd?
A recent one I struck was with Google Earth desktop app on Linux. It has a tendency to crash on startup unless your mouse is contained within a small rectangle in the middle of the screen [1].
[1] http://askubuntu.com/questions/642027/google-earth-crashes-when-opened#comment1071599_677717
112 comments
[ 2.9 ms ] story [ 58.7 ms ] threadFinally we tracked down the issue: when the contents of a particular file were archived to tape, the tape drive crashed. I suspect it was a tape firmware issue, maybe to do with the native compression.
The workaround was to mark that particular file as "not to be archived" and we stopped having media and drive errors.
Fast forward a bit and a new backup system is being put in place. But it keeps breaking, but only on this box. While researching the issue, explorer keeps breaking when doing searches and third party search tools keep breaking.
Took me a little bit to remember what I'd done and fix it.
[0] - https://msdn.microsoft.com/en-us/library/windows/desktop/aa3...
Does anybody know the nature of the hidden directory hack I've referred to?
When we (another student and I, also a student) tipped the Teacher-cum-admin off, the folders masqueraded as a printer in the NT file explorer. They couldn't delete them.
We recommended that they wiped the machine and disabled public ftp. (weren't that big of an issue as it was mostly a print server).
So they had a wonderful core which was wrapped in baroque APIs, but the real problem was that core wasn't entirely wonderful, occasionally when you presented it a "Death TIFF" as we called the images for their file type, it would reliably crash. Software or firmware versions of the code (they had a hardware accelerated box with one or more Intel RISC chips), on the PC platform at least, e.g. Windows 3.x using a DOS box, this would entirely lock up the machine.
To get around this for a client that had 500,000 images to OCR on a tight deadline for a legal case (and this was the golden area of legal document imaging, back then lawyers would pay 50 cents per OCRed page, because a full text search could e.g. impeach a witness on the stand in real time), I created a system where the PC would always be printing out asterisks if it was OCRing pages. That allowed an operator to tour the machines and easily see when he had to manually reboot one stuck on a Death TIFF, after which my software would recognize what had happened and continue with the next image.
It turned out that Windows 98 shipped with an Imaging program (Licensed by MS, not written by them) which predated the standardization of the JPEG-in-TIFF subformat, but they'd basically guessed at how it would work and shipped that. The final spec (and the version of JPEG-in-TIFF nearly everyone else implemented) ended up being different. So basically nothing could read it.
We ended up calling them up every time a customer found one of these files and having them print out that image on one of their windows 98 machines, and scan the printout back in using one of the newer machines. Sure, we lost some quality, but at least the customers could access the data now.
For a time reference, these broken images were still showing up in newly scanned documents in 2011 (when we stopped working with them due to massive fraud), so they must have been using their Win98 scanner systems even then.
We were producing our own TIFF files using our own software that drove monster Kodak ImageLink scanners (software I in fact took over, redid the SCSI driver of, and eventually did a clean rewrite of the engine on Sun workstations), so the images and their compression came straight from Kodak, and going further, I don't recall those 600 pound beasts ever screwing up at that level.
And this was way before Windows 98, it was Windows 3.0 or by then 3.1, like in 1992, Windows was utterly naive about document image files. Which I can see was a blessing (although maybe it was losing quality, I'd long switched to NT by the time 98 came out).
I didn't mean it was definitely the same company, just a similarly annoying TIFF issue.
If i unplugged the networking card when i installed there were no issues.
It's kinda strange for an OS to be maintained for a long time with that style of backwards compatibility....
A lot of folks are used to the crazy ship-all-the-time-regardless-of-cost world of webdev, but there is a lot of business value in not breaking things randomly.
So it turns out there is a very small time slot where the sun can reach through a window into the hallway. That was enough to offset the light sensor that I attached to the power meter inside the closet. The threshold was set too tight.
Think about the possible sources that influence this 'bug': - the month - the time of day - the weather / state of the clouds - open/close state of the bathroom door - reflectivity of the hallway (objects, doors open/closed)
On-screen keyboards displayed Chinese after visiting a system menu. We freed the async operation when the system menu "canceled" the keyboard operation (it wasn't supposed to be even displaying), but apparently the system had a use-after-free bug. I worked around the problem by switching to a 4 entry LRU allocator, keeping the past 3 or 4 canceled operations around untouched (1 would've probably sufficed, but I'm paranoid.)
A WinRT API to check internet connectivity would exit(3) our app without error messages or related callstacks - but only if the Charm bar was open for more than 10 seconds, assuming you called it once per frame on the main thread. I had to bisect our history to figure that one out - and repro in a new test app to confirm it was the real cause.
EDIT: Third party injected DLLs crashed our app at least twice - once for some monitoring software on a coworker's computer (crashed when closing file handles as build tools tried to clean up and exit), once for an old Microsoft Word IME that predated the Win8 app sandbox who's restrictions it was violating. The monitoring software was uninstalled, the IME I couldn't think of a reasonable workaround for and left to Microsoft to fix.
My workaround for watching movies with vsync? Use Intel drivers in my main X session, modesetting in a secondary X session just for mpv.
There are ways, though, to make sure that your icons have priority over "Joe's poorly designed explorer plugin." :)
It was particularly painful because when you'd hit it (by, say, installing Sybase drivers or some other awful application that insisted on putting nearly every subdirectory it had in PATH), nothing would tell you that it was specifically the PATH being truncated that was at fault, you'd just get a large number of applications that would stop working and return obscure error messages.
http://winaero.com/blog/how-to-enable-ntfs-long-paths-in-win...
http://thedailywtf.com/articles/ITAPPMONROBOT
And pics of a build it inspired:
http://thedailywtf.com/articles/The-Son-of-ITAPPMONROBOT
Similarly - there were some minor issues with the cooling for my compute cluster at my previous job, where it weren't really designed to function in climates which had temperatures that varied too much. Notably, it'd turn off the compressors on hot summer days and cold winter days. While waiting for the tech, tiny rocks found on the roof were used in conjunction with some tape to force the mechanical relays on while waiting for the techs.
http://www.pvv.ntnu.no/~kjetijor/images/tape_rocks.jpg
The issue was similar. On nights where the temperature dropped too close to the dew point for too long, the units would freeze over. However, at the time, there wasn't any temperature monitoring. So someone figured out how to monitor the die temp on the FPGAs without changing the running code. Took them a few days. By the time they finished, someone realized they could tie streamers to the AC vent, which could be seen in the remote video stream.
Anyways, the fix was to connect to the network, switch the AC unit to fan only for a couple of hours, then switch them back on. If I remember correctly, it was like this for about 6-8 months before they finally had someone replace the AC system with a more commercial unit that could handle the condensation.
I used Fiddler to compare the requests from the two computers and eventually discovered that the request would fail if the `Accept` header was longer than some value (might have been 255 characters -- I don't remember).
Turns out when you install Microsoft Visio and Project, Internet Explorer's Accept header gets really long.
Workaround is ctrl-alt-f7 to switch to console then ctrl-alt-f1 to switch back to GUI, and the mouse cursor reappears.
https://bugs.launchpad.net/ubuntu/+bug/1573454
Another one is a sweet widget in OS X called iStatPro, which was no longer working ias of Mountain Lion. But, there is this workaround which for me still works on El Capitan: http://hints.binaryage.com/istat-pro-for-mountain-lion/
http://spectrum.ieee.org/aerospace/space-flight/titan-callin...
http://descanso.jpl.nasa.gov/seminars/abstracts/viewgraphs/H...
This was an extremely serious bug in NASA/ESA's Cassini-Huygens probe, in the S-band link between Huygens (landing on Saturn's moon Titan) and Cassini (acting as radio relay).
It was a timing bug. There'd be a very high relative velocity between Cassini and Huygens, creating a significant (~2e-5) Doppler shift in the link. This shifted the frequency of the 2 GHz carrier (by 38 kHz). Likewise, it shifted the symbol rate of the 16 kbps bit stream (by 0.3 bps). The second effect was overlooked. On the demodulating end (Cassini), the bit-synchronizer expected the nominal bit rate, not the Doppler-shifted bit rate. Since its bandwidth was narrower than the 0.3 bps Doppler shift, it was unable to recognize frame syncs; this was proven in experiments post-launch. The parameter that set the bitrate was stored in non-modifiable firmware.
As it was when launched, Huygens would be unable to return any instrument data. For some context, this was the only probe that's ever visited Titan, at a cost of about $400 million.
The workaround
[spoiler]
The workaround was a major change in the orbit trajectory of Cassini (a $3 billion probe). Details aside, it set up an orbit geometry with this feature: at the time Huygens was descending in Titan's atmosphere, Cassini would be flying at a ~90° angle to their separation. The relative velocity was still 20,000 kph, but tangential velocity doesn't contribute to Doppler shift.
The guy who wrote the crypto plugin had of course quit and nobody knew how it worked.
Fine-combing the C++, I found an off-by-one error that would cause the predicted failures: after rebooting SQL Server, the first entry would get encrypted with a zero key. (Hooray, we could now also fix all the corrupted data.)
For various reasons it would have been difficult to ship new DLLs to the affected customers. Only a handful used this particular crypto and it would be much easier to patch the existing binary DLLs on their servers.
Well... looking at the machine code, I found that the troublesome off-by-one operations were actually in the printable ASCII range... so I just taught my friend in tech support to do a particular obscure search and replace in Notepad++, something like changing ",}" into ",~" in the binary DLL... and then hot-reload it with an SQL Server command... worked perfectly.
As Variety can apply ImageMagick filters on the fly to the wallpaper being set, I set it up so that it just scaled down and cropped the image to my exact desktop resolution. This fixed the issue for me... at least, temporarily :)
To set up the filter, I edited the ~/.config/variety/variety.conf, and changed the line:
to Then I configured Variety to generate a single wallpaper file in a folder which is "watched" by the KDE Plasma desktop wallpaper changer, with the same interval. Voilà!We eventually traced the issue down to the Nginx server that was serving the files and one of it's cache buffer size config options, (I don't remember which one anymore). We noticed if the file being served was larger than a certain size it would occasionally truncate the file but not always. We tested increasing the buffer size by repeatedly doubling the default value, which was a power of two, up to a size of several GBs. But the files kept being truncated for some small percentage of the requests. At this point we knew it wasn't directly related to the size of the buffer since it was larger than any files being served. Finally someone suggested that we test a value that wasn't a power of two and the issue was gone.
We figured it was an internal bug in Nginx where it was growing an allocation buffer and used powers of two, but had an off by one error that didn't copy the second half of the buffer or something. We dug through the code but never found anything and so we left the cache setting at +1 from the default power of two value and never had an issue again.
I worked on an enterprise job scheduler that was initially outsourced to an Indian company but the project started failing and so we took back development. The software was required to be able to schedule tasks with a delay of up to a hundred or so hours but the GUI library only had a control for time of day up to 24 hours. The code we received had an interesting solution - they changed the format string to place the milliseconds part first and then some code in the data access layer that swapped hours and milliseconds back and forth on reads and writes. And there you have it, delays up to 999 hours.
So we were trying to recover it and I had a "It's a Unix System, I know This!"-moment and was able to manually type in the path to the previous binary during an emergency/rescue prompt (based on deductions from forums, the current failed loading message, and some obvious things like architecture) and got it up and going again.
Documented that, internally, to the best of my ability.
We had it all up and running - loading the content, waiting for the player to initialize, taking the snapshot, generated sizes - on a windows machine when, one day, the request came in to migrate that machine to a VM. After the migration, things were fine - until we disconnected RDP. Snapshots were coming back at the right size, but totally white.
The eventual "solution" was a laptop in the engineering area RDP'ed into this VM to keep the snapshots from going white. It got unplugged one holiday weekend, earning it a red hand-sharpied sign - "PRODUCTION LAPTOP: DO NOT UNPLUG". It was unplugged again one fateful weekend, this time prompting a healthcheck to be written that looked for all-white images in its output.
That rig ran that way, I believe, until someone had the insight to make a second VM, this one RDP'ed into the first.
Turtles, all the way down!
At "a large telecom" I used to work at, we had a specific process that handled billing that relied on a DOS application which was written targeting a specific modem's hardware. They'd tried to migrate it to something else for quite some time but the guy who wrote it lived in a different state and was let go from the company when we closed that site down and moved all of its equipment to Detroit. It ran on an old Compaq (not HP Compaq, Compaq) desktop PC and in 2014 or our VP received a frantic call that the drive had failed and the computer wouldn't boot (from a younger tech who was used to working on server class hardware). The code for this application had been lost forever and nobody had any idea how it actually worked but my understanding was that with it not functional, we were losing enough money to make it a "drop everything priority".
They brought the machine over to my building and the VP of my department called me to assist[0]. Sure enough, the system wouldn't even see the drive. It was at this point that I noticed three numbers with the letters "C", "H", "S" next to each. This had happened before, apparently, and someone discovered the BIOS battery had died. Thankfully, they were kind enough to put the drive parameters on a label for me. I popped into the BIOS, put 'em in and it booted. The computer remained powered on in the cubicle I repaired it in (just outside said VP's office) for a year until the dev team got around to modernizing the code.
[0] I was not a support person at this time but was in the past and it wasn't unusual for them to call me in on strange problems. I was also known for having recovered a hard drive with important data on it using the break-room fridge (though I'm not sure this VP was aware of that).
Trick with the freezer hard drive: if you ever order perishable items over the internet, they sometimes ship in boxes with large bags of "blue goo". Pop those in the fridge and the next time you need to keep a drive spinning long enough to get one last copy out of it, sandwich it between two of those. They don't get cold enough to pick up condensation and short the drive and the blue goo keeps cool for a long time if the bags are large enough.
1. spiped re-binds SIGINT if it is launched as pid 1, in order to work around a Docker bug: https://github.com/Tarsnap/spiped/blob/master/spiped/main.c#...
2. In my POSIX-violation-workarounds script, ironically enough, I work around a bug in bash which makes 'command -p sh' run with the incorrect path (this has since been fixed, but continues to be present in older installed versions of bash): https://github.com/Tarsnap/spiped/commit/e3968941c9c1b20c63d...
3. In my getopt code, I use a (non-C99-compliant) computed goto in order to work around a bug in LLVM's handling of sigsetjmp/siglongjmp: https://github.com/Tarsnap/libcperciva/commit/92e666e59503de...
4. Many years ago, I added a spurious 'volatile' into some Tarsnap code in order to prevent a buggy LLVM optimization step from running (it was making the Tarsnap build hang on on OS X 10.7): https://github.com/Tarsnap/tarsnap/blob/master/tar/multitape...
Starting out with a bunch of existing CSS, a developer added 2 new properties somewhere in the middle, but forgot a trailing semi between them. Reloading the page showed the first change, but not the second. He tried ten different variants of the second property name, spelling, values, etc. and nothing was showing up. He added another property before the broken one to help debug, and it started working. He then tried several variants on that to see if it was some arcane ordering bug, and eventually ruled that out by using two developers' names for property:value.
Because all of the intermediary versions included a semi, and because the first property allowed some kind of extended content that was ignored, it took half a dozen developers looking at the "weird bug" before someone noticed the missing semi on the first property.
Everyone had been talking about getting at that data for a year or so and one night I was just like fuck it, I'll give it my best shot. Honestly it wasn't that impressive, but I certainly do remember how cool it felt to tell "the man" to F off and this was our data :).