I inherited a giant hideous stock-management system. It did a certain amount of automated ordering without manual intervention.
Long story short: a nasty race condition meant that it was over-ordering duplicate products from the suppliers to the tune of tens of thousands of dollars per day.
On the general theme, my most frightening software experience was when I met a guy who was the star programmer for a company doing controllers for elevators. I got talking to him, and he showed me some code. It took me about 15 seconds to identify an edge case that would engage the motor while the elevator was at the top floor (thereby attempting to pull the elevator into the roof?). It took me a further hour to explain what the edge case was. The bug wasn't that scary, as I'm sure there are hardware failsafes, but the general dimness of the guy writing software to control lifts was scary.
One bug in Daylife's API tester's javascript. It just never worked in IE for some stupid reason. The solution was surprisingly simple.
After some analysis with firebug, I just figured out that the variable's value wasn't being preserved after a particular point. So I just had to take it's value and assign it back to itself.
[Full disclosure: I was not and I'm not a part of Daylife. I only participated in a contest they once conducted and was asked to try solving their problem by one of their staff.]
read the above 'C' statement.
Yeah, there are no brackets surrounding the '10*86400', INTERVAL was being used in some calculations.i wasted a lot of time debugging this crap.
A batch run of only a few thousand items was running all night long, rarely finishing and causing all kinds of problems when people logged in in the morning. The users had been complaining about this for years.
I was given the ticket and found a "SLEEP 10" (causing a 10 second pause for each item) in the 10 year old BASIC code, put in by the original programmer for debugging purposes, and never removed before it was promoted.
I removed the "SLEEP 10" and run time went from 12 hours to 23 seconds.
The users loved me, but my boss was not pleased. He said, "You should have changed it to a SLEEP 5 so we had something else to give them the next time they complained."
Reminds me of back in the old days writing integer assembly language (and later) C apps for microcomputers. As FPUs started to become available, users complained they spent all this money on an FPU and got no speed improvement.
I half-seriously suggested adding FPU detection and intentional delay loops if no FPU was present.
Every since time there is a memory corruption on a games console it takes hours and hours of tedious detective work to find.... shudder. Glad I don't do that any more.
A neat piece of software calculating the prices for customers Had a hierarchical pricing model for different contracts with large corporations. But the hierarchies were not separated. You started in one hierarchy and ended up in another one. While trying to read the tree from the bottom up you could end up reading the whole database for calculating a price with conditions from every customer.
Take your pick. Whatever works for you. For me the definition would probably be the hardest to diagnose, but I'm sure there are better ones that lead to more interesting stories and lessons to be learned.
Worst software bugs, in increasing order of severity:
An ASP.NET app that didn't consume it's record sets before closing it's database connections, thereby breaking IIS's connection pooling and causing hundreds SQL server login/logouts per second. Lots of interesting side effects on that bug.
An obscure page latching problem on a high volume SQL server (hundreds of transactions & thousands of queries per second). The SQL server would spend all of it's time waiting on page latches, response time would grow exponentially. An eleven hour phone call with MS support finally identified the problem.
Worst: c/c++ pointer memory errors, duh. Especially when there are thousands of pointers and don't know which one is overwriting memory that it shouldn't be overwriting.
Second worst: linking c programs when some of the symbols are duplicated in more than one library yet are defined differently, Nobody mentioning this to the developer made it all the more exciting.
Thanks to the Boost.org developers the C++ pointer issues are mostly a thing of the past. Using shared_ptr et. al. for all but the most time-critical code has made memory-smashing, memory leaks, etc. so easy to avoid!
(a) The CAN bus (which reports wheel velocity and position of the steering wheel among other things) micro-controller hardware stopped sending interrupts which caused the main computer to think we were stopped. As far as I can tell this is just a straight up hardware bug with the Philips ARM chip we were using. This caused the accelerator controller to floor it because it was only seeing the last CAN message we ever got which happened to be zero velocity. Same thing with the steering (hence the big swerve).
or
(b) I failed to consider the contingency of not getting any CAN interrupts (either because the of the [very intermittent] hardware bug or because the connector got disconnected--turns out the symptoms are the same) and didn't have any code written to deal with it. Guess what I wrote that night. :-) Luckily I had all day to think about it while a new tire was fetched and by the time I figured out what was going on it took about 10 minutes to code the fix: a watchdog timer that shuts the world down if there are no inputs from the sensors for any reason.
It seems obvious in retrospect but when things are working you sometimes forget about weird failure cases.
Any chance of getting 'second opinion' style sensors in there that can provide you with sanity checks ? Such as 'GPS reports movement, but wheel sensors do not, we have a problem ?'
That way you can avoid a paralysis of the control software until the vehicle has really come to a halt.
I read somewhere about realtime applications that do something like this but redundant sensors holding a consensus polling algorithm. Have three sensors reporting the same thing and if they are not all in agreement within some kind of delta then go into some kind of limp mode or have the two sensors in agreement be the ones that the system uses for it's algorithms. I cannot recall where I read it though.
Me too. The back of my brain is telling me it was for some sort of plane control software? Maybe? An interesting tidbit that I recall was that they used different manufacturers to hedge their bets against bugs.
That sounds about right. I seem to recall that it was for aerospace too. Maybe NASA? Something about zero defect software. I cannot for the life of me find the article right now though. I also vaguely remember that it was a HN submission too.
A multi threading system and a queue: it will be hard to explain all the algorithm here but sometimes you think that one thread can't affect others and it is not true I had to find an exception that kills no just the thread but doesn't allows the queue consuming to continue, I remember I went home something like 5am.
Back when text messaging capability was a rarity on mobile phones, which were themselves rare, I was testing an SMS-based weather forecast service that I had written on behalf of one of the mobile network operators.
The testing worked well on the emulator so I decided to test it over the public network to an actual handset. Only I forgot to advance a recordset through which I was looping, so the code never hit the end of recordset condition. It took me some time to notice there was a problem...
The fact that I crippled a national SMS network for a few hours was bad.
The fact that my company had to pay for each SMS, wiping out out profit for that month was worse.
The fact the handset was mine and on my first date with a girl later that evening (whom I later married) my handset kept beeping with incoming text messages (about 96,000 if I remember) was the ultimate.
The handset didn't have a silent-no vibrate function (either it beeped or it vibrated or it did both) and the SMS inbox filled up after 200 or so messages meant it took days for the inbox to fill up, me to clear it message by message, then fill up again ad nauseam.
One of my colleagues did this with our automated notification system. After his phone received about $60 worth of text messages, he panicked and shut down the server!
Had an obscure picking id wrap around because a table wasn't getting cleared for debugging purposes which resulted in excessive amounts of beer being delivered to unsuspecting customers at an automated gas station.
Non-strict comparator with the STL. Nearly impossible to identify when it's happening. It's happened once to me and once or twice to a coworker over the last couple years, and takes 3-5 days to debug every time.
Example:
struct FooSort {
bool operator()(Foo const& a, Foo const& b) const {
- return not a < b ;
+ return b < a;
}
};
This has happened to me several times. Especially in combination with the fact that C++ does not guarantee floating point operations happen the same every time... Argh. So it is unwise to do things like
bool operator()(Foo const& a, Foo const& b) const {
return a.some_float_func() < b.some_float_func();
}
if some_float_func does floating point operations rather than just returning a stored value.
There are situations where the order of evaluation is not guaranteed. The big one that I've seen is evaluation of expressions as arguments in a function call, but there are others. It is easy to create floating point expressions where because of rounding, the results will differ depending on the order of evaluation. I'm not aware of situations where the order of operation differs from one execution to the next. However, I could see a compiler generating different code for the same expression when it appears in more than one place in the source code because of optimization.
For example, a function might be inlined at some call sites and not inlined at others. Inlined versions might carry 80 bits through the result, but non-inlined version are truncated to 64. So in:
I'm not sure how the example could make the assertion fail; I think that the point is that if 3.3 is treated as a long double by the compiler, then the result of x*3.3 will be a long double, which is more precise than the double returned by foo; but, regardless of how bar1 and bar2 are compiled, they both make a call to foo() which returns a double. Is it the case then that the behavior depends on how foo is compiled? I don't see how else the 80-bit result could propagate to the test for equality.
returning a double can be optimized into returning an 80-bit result in a register. If this happens with bar1 but not bar2, then you get different results.
Try a worse version of this - comparing the results of a floating-point function call where the left operand gets moved into the 80bit fp unit during computation, and the right operand stays in register.
Obviously there is a precision difference between the two numbers, enough to make a < b and b < a return true in a surprising number of cases. The way I ended up fixing it was by putting the result of the function call in a member variable in every struct, pre-computing all of the results, and comparing based on that value.
I hit this, too. Regression tests were failing when I changed code that obviously shouldn't change the output of the program at all. This happened on a regular basis when we changed numeric code, because of the normal limitations of floating-point arithmetic; we just made sure the numerical results were accurate and updated the regression tests. (The regression tests were quite handy for finding logic errors; they weren't really used to test numerical accuracy.)
But in this case I was just adding some error checks, which weren't even being triggered. Clearly this shouldn't affect the results of our numerical calculations. Since my code shouldn't affect the calculations, I was convinced that our existing numerical code had a subtle memory or timing bug. (I knew that floating-point code was tricky, but clearly I was doing exactly the same operations on exactly the same values.) I spent days staring at code, and then my boss told me to stop working on it since the results were clearly correct in both versions, even if they weren't identical.
A few weeks later I read about how values change when they're copied out of the x87 stack into registers. And I thought, naw, we couldn't possibly be using x87 arithmetic. But we were. Which was horrifying, since floating-point calculations could be a bottleneck under some workloads. But we had been running that way since before I started working on it, so at least it wasn't my fault. I added a compiler option to request sse2 floating point instead of x87 floating point. Voila, predictable floating-point results, plus measurably faster performance on a few tests.
I'm glad I'm not the only one who has trouble when I hit this.
I really, really wish the STL had a mode you could compile it in which boiled down to, "Double check everything no matter how slow it makes things." Maybe it has it and I don't know it. But that was my top irritation with the little bit of C++ I've done.
I have no idea how a non-strict comparator can lead to memory corruption rather than just indeterminate sort order, but it is no fun tracking that down.
Some users of a (shipped, fairly heavily used) web app we had deployed were getting kicked back to the login screen at random. Sometimes, very frequently.
Looking in the logs we could see that these users were somehow losing their authentication cookie and the application was correctly bouncing them to login. So how were they losing their cookie? Assuming it was a bug in the code we searched and searched to no avail.
Finally I discovered that the hardware load-balancer our CTO/'IT' guy' had insisted on was the culprit. The load balancer would buffer fragmented requests and re-assemble them before sending them on to the server. Unfortunatly the load balancer had a huge bug in its firmare.
If a user was using firefox, on windows, and their request was fragmented such that the first packet contained data up to the end of a header line including the \r but not the \n, so the next packet would start with a \n and then a header name, the load balancer would insert a \n\r between the two packets, thus effectively truncating the HTTP headers, usually before the cookie lines.
When I found this bug I couldn't believe that this was actually happening, I thought I was taking crazy pills, but you could run a sniff on the front and back side of the load balancer and see the request go in well formed and come out busted. We ditched the hardware load balancer and all was well.
The bug itself wasn't very interesting, just a brainfart. But it wasn't doing what it should, so I added debugging output - one line per pixel. (I was generating a png from a custom image format.)
Because there was so much output, I did a `| head` to keep it manageable. Saw what I was doing wrong, fixed it, reran the command, it seemed fine - but the output image looked exactly the same as it had before.
It took me about an hour to realise that once head had exited, the pipeline sent a signal to my program, killing it. The image wasn't getting written until after all the pixels had been processed, so it never got to that stage.
Was converting an avionics subsystem from Ada to C. It was a client application that had to talk to an Ada server, sending and receiving rather huge chunks of data, large, deeply nested, intricate structure types. The C structure type had to match the Ada type exactly, or else it wouldn't work.
I got it working fine on our desktop simulation, but running on the actual hardware it was consistently off. After extensive testing, I realized that it was a bug in the compiler for the target hardware, such that a very particular type of structure (something like, {int, char, float}) was being packed incorrectly, resulting in a 2-byte pad that shouldn't be there. If I reordered the structure elements, it was fine, but that particular grouping and order refused to work correctly.
It was GCC, so we could fix the compiler ourselves, right? Not really, as, for avionics systems the compiler has to be thoroughly qualified for avionics use, and changes equal requalification. I "fixed" it by storing the float as an array of characters, converting it to and from a real float type as we needed to use the data value.
Trivial, perhaps, but I was very excited to resolve the problem, after spending days barking up wrong trees. One usually expects that the problem is not in the compiler... :-)
If I am understanding what you are saying, we really couldn't do that, as the server sent and expected to receive binary blobs of data; the only way to know what was what was to have a map of where the data elements were.
I like to use macros (in C) to spit out the structure size and offsets of each structure member over a serial port, in a format I can then cut-n-paste back into the source. This output consists of a bunch of (compile-time, when possible) assertions so that any changes to a structure break the build. These assertions go on both the embedded side and the PC server side, so any weird packing issues show up at compile time.
A while back, I developed a program to generate invoices for about a dozen busy warehouses. During testing, for convenience sake, I hard-coded in my local printer.
Unfortunately, I forgot to return the printer name to a variable when promoting into production. Hilarity ensued.
Guy I knew - awesomely good - hex edited a DOS boot sector on a in house machine to use FUCK.SYS instead of whatever.sys it normally is (I forget). He renamed that file to fuck.sys, rebooted and the machine ran. Cool!
We laughed and reinstalled DOS and two days later the boss come charging in yelling 'I have a client on the phone who says his new machine can't find FUCK.SYS!'
The worst bug I encountered was due to IBM MVS (or COBOL--I was never sure which was at fault) losing addressability of part of a variable length record. Now you see it, now you don't. The solution at our shop was to move the whole record to itself before attempting to look into the record. I was a newbie. If the old guard hadn't told me that workaround, I NEVER would have thought of it. This problem eventually went away, but 25 years later we still occasionally ask each other "did you try moving it to itself" when dealing with new problems. We chuckle, while today's newbies shake their heads at our Old Fart humor.
The worst one I ever caused was when Visa started carrying two amount fields in their credit card records. One was the amount in original currency, the other was converted to the receiving system's local currency. I used the original amount. Our hand-made test data used the same currency for both amounts, so no problem in test. Imagine my surprise when we went live and our system started posting original currency amounts to cardholder accounts, which at the time only supported US dollars. Luckily, we caught it early and senior management and cardholders were all good sports about it. I think those credit card statements with massive amounts became collector's items.
Since you mention COBOL records were variable-length I'd guess that they probably contained ODOs (Occurs-Depending-On, variable length arrays for those of you who aren't COBOL literate). In order for a group or record MOVE to work properly you had to move the subordinate ODO values first, otherwise the runtime system would miscalculate the target record length, possibly truncating the MOVE.
This also meant you couldn't use READ INTO for variable length records (which is equivalent to a READ followed by a MOVE) without taking some care.
I worked on a taxi booking and dispatch system, written in c, and running on dos with custom networking via RS232. This system was installed at around 300 locations around the UK, and on one fateful day, every installation crashed.
It came down to me to find and fix the problem, and it was subtle. The clue lay in the fact that all of the sites that crashed did so within about a minute of one another.
Turns out that some of the old, old sections of the software had been written by the MD, who, despite referring to himself as 'the emperor of c', was in fact an atrocious programmer.
The actual trigger was the comms system looking at a byte that determined as to whether a message had been received. This byte was set to the character 'A' if a message was received. It just so happened that the first byte of the current value of the number of seconds since 1970 evaluated to 'A', and had been written into that memory location a negative index into an array that hadn't been initialised.
This negative index into an array that shouldn't have been empty caused a section of memory to be overwritten that made the comms system think that it had received a packet. This snowballed quickly, and took down the system within about five seconds of boot.
Took the best part of two days to track down, and, of course, it was everyone else's fault but the emperors.
I once worked an issue with a crashing Exchange server.
It had to do with a connection being reset in the space of time between the server checking if the connection was good, and actually using it.
The client could reproduce it at will, and after a little bit of code at home, I could, too. Back at the office, no repro.
It took a lot of back and forth (to put it mildly), but the problem at work was that the network was too slow - the server got the reset with plenty of time to "notice" it, and the crash didn't occur.
Once we set up the repro on its own switch at work, everything failed in the expected manner.
The store locator function on a national pizza chain's web site would completely hang their web server whenever an international search was done. Many, many hours and days of testing and debugging led us to conclude, and build a proof that it was a reproducible bug within IBM's Domino platform, only on AIX boxes, only when:
1) A script using LotusScript, their proprietary language was kicked off, and...
2) A Java agent was then kicked off before the original script completed.
At the time, Java was a new feature within that platform, so there weren't many apps that mized both languages.
After getting to this point, IBM joined in the fix effort, and we had daily conference calls, on which we always had IBM execs lurking because their 6.0 release of the platform was imminent, and this bug had the potential to wreak major havoc if not fixed before launch.
So I cannot personally claim to have done the actual bugfix - the IBM programmer did that. But it was a great learning experience to work together with IBM to find and fix it.
The nastiest was an interaction between two libraries. If you simultaneously imported xml.dom and matplotlib (both python libraries), and then called functions in one of the libraries (I think matplotlib), the program would segfault.
My incomplete set of unit tests didn't catch it because they were imported by seperate submodules of mine.
I only managed to find it by writing a unit test, going back in version control, searching for the first revision where the problem occurred, and going line by line through the changeset (luckily I commit early and often).
127 comments
[ 27.7 ms ] story [ 2522 ms ] threadLong story short: a nasty race condition meant that it was over-ordering duplicate products from the suppliers to the tune of tens of thousands of dollars per day.
On the general theme, my most frightening software experience was when I met a guy who was the star programmer for a company doing controllers for elevators. I got talking to him, and he showed me some code. It took me about 15 seconds to identify an edge case that would engage the motor while the elevator was at the top floor (thereby attempting to pull the elevator into the roof?). It took me a further hour to explain what the edge case was. The bug wasn't that scary, as I'm sure there are hardware failsafes, but the general dimness of the guy writing software to control lifts was scary.
I started taking the stairs after that.
After some analysis with firebug, I just figured out that the variable's value wasn't being preserved after a particular point. So I just had to take it's value and assign it back to itself.
It's in this file: http://demos.daylife.com/api_tester/js/daylife.tester.js (lines prefixed with comments "IE Fix")
[Full disclosure: I was not and I'm not a part of Daylife. I only participated in a contest they once conducted and was asked to try solving their problem by one of their staff.]
http://karma.nucleuscms.org/item/101
" #define INTERVAL 10 * 86400 "
read the above 'C' statement. Yeah, there are no brackets surrounding the '10*86400', INTERVAL was being used in some calculations.i wasted a lot of time debugging this crap.
I was given the ticket and found a "SLEEP 10" (causing a 10 second pause for each item) in the 10 year old BASIC code, put in by the original programmer for debugging purposes, and never removed before it was promoted.
I removed the "SLEEP 10" and run time went from 12 hours to 23 seconds.
The users loved me, but my boss was not pleased. He said, "You should have changed it to a SLEEP 5 so we had something else to give them the next time they complained."
I half-seriously suggested adding FPU detection and intentional delay loops if no FPU was present.
I must not state the accuracy of that price...
Bugs are a great learning experience.
An ASP.NET app that didn't consume it's record sets before closing it's database connections, thereby breaking IIS's connection pooling and causing hundreds SQL server login/logouts per second. Lots of interesting side effects on that bug.
An obscure page latching problem on a high volume SQL server (hundreds of transactions & thousands of queries per second). The SQL server would spend all of it's time waiting on page latches, response time would grow exponentially. An eleven hour phone call with MS support finally identified the problem.
Oracle 10.2.0.[1234]
We need to have a word.
Second worst: linking c programs when some of the symbols are duplicated in more than one library yet are defined differently, Nobody mentioning this to the developer made it all the more exciting.
http://www.youtube.com/watch?v=b7i2KkYYulI
Damage: Blown tire, dented rim, looking like fools in front of our peers.
(a) The CAN bus (which reports wheel velocity and position of the steering wheel among other things) micro-controller hardware stopped sending interrupts which caused the main computer to think we were stopped. As far as I can tell this is just a straight up hardware bug with the Philips ARM chip we were using. This caused the accelerator controller to floor it because it was only seeing the last CAN message we ever got which happened to be zero velocity. Same thing with the steering (hence the big swerve).
or
(b) I failed to consider the contingency of not getting any CAN interrupts (either because the of the [very intermittent] hardware bug or because the connector got disconnected--turns out the symptoms are the same) and didn't have any code written to deal with it. Guess what I wrote that night. :-) Luckily I had all day to think about it while a new tire was fetched and by the time I figured out what was going on it took about 10 minutes to code the fix: a watchdog timer that shuts the world down if there are no inputs from the sensors for any reason.
It seems obvious in retrospect but when things are working you sometimes forget about weird failure cases.
That way you can avoid a paralysis of the control software until the vehicle has really come to a halt.
http://en.wikipedia.org/wiki/Kalman_filter
The testing worked well on the emulator so I decided to test it over the public network to an actual handset. Only I forgot to advance a recordset through which I was looping, so the code never hit the end of recordset condition. It took me some time to notice there was a problem...
The fact that I crippled a national SMS network for a few hours was bad.
The fact that my company had to pay for each SMS, wiping out out profit for that month was worse.
The fact the handset was mine and on my first date with a girl later that evening (whom I later married) my handset kept beeping with incoming text messages (about 96,000 if I remember) was the ultimate.
The handset didn't have a silent-no vibrate function (either it beeped or it vibrated or it did both) and the SMS inbox filled up after 200 or so messages meant it took days for the inbox to fill up, me to clear it message by message, then fill up again ad nauseam.
Still, I laugh about it now...
Then again $60 is only about 1200 messages.
You must not be in the US..
Here's a video of part of the result: http://www.youtube.com/watch?v=RUhLDtPnSuQ
Example:
Can you elaborate more on this? Are you implying that two identical calculations will return different results?
Obviously there is a precision difference between the two numbers, enough to make a < b and b < a return true in a surprising number of cases. The way I ended up fixing it was by putting the result of the function call in a member variable in every struct, pre-computing all of the results, and comparing based on that value.
But in this case I was just adding some error checks, which weren't even being triggered. Clearly this shouldn't affect the results of our numerical calculations. Since my code shouldn't affect the calculations, I was convinced that our existing numerical code had a subtle memory or timing bug. (I knew that floating-point code was tricky, but clearly I was doing exactly the same operations on exactly the same values.) I spent days staring at code, and then my boss told me to stop working on it since the results were clearly correct in both versions, even if they weren't identical.
A few weeks later I read about how values change when they're copied out of the x87 stack into registers. And I thought, naw, we couldn't possibly be using x87 arithmetic. But we were. Which was horrifying, since floating-point calculations could be a bottleneck under some workloads. But we had been running that way since before I started working on it, so at least it wasn't my fault. I added a compiler option to request sse2 floating point instead of x87 floating point. Voila, predictable floating-point results, plus measurably faster performance on a few tests.
I really, really wish the STL had a mode you could compile it in which boiled down to, "Double check everything no matter how slow it makes things." Maybe it has it and I don't know it. But that was my top irritation with the little bit of C++ I've done.
I have no idea how a non-strict comparator can lead to memory corruption rather than just indeterminate sort order, but it is no fun tracking that down.
Looking in the logs we could see that these users were somehow losing their authentication cookie and the application was correctly bouncing them to login. So how were they losing their cookie? Assuming it was a bug in the code we searched and searched to no avail.
Finally I discovered that the hardware load-balancer our CTO/'IT' guy' had insisted on was the culprit. The load balancer would buffer fragmented requests and re-assemble them before sending them on to the server. Unfortunatly the load balancer had a huge bug in its firmare.
If a user was using firefox, on windows, and their request was fragmented such that the first packet contained data up to the end of a header line including the \r but not the \n, so the next packet would start with a \n and then a header name, the load balancer would insert a \n\r between the two packets, thus effectively truncating the HTTP headers, usually before the cookie lines.
When I found this bug I couldn't believe that this was actually happening, I thought I was taking crazy pills, but you could run a sniff on the front and back side of the load balancer and see the request go in well formed and come out busted. We ditched the hardware load balancer and all was well.
Because there was so much output, I did a `| head` to keep it manageable. Saw what I was doing wrong, fixed it, reran the command, it seemed fine - but the output image looked exactly the same as it had before.
It took me about an hour to realise that once head had exited, the pipeline sent a signal to my program, killing it. The image wasn't getting written until after all the pixels had been processed, so it never got to that stage.
I got it working fine on our desktop simulation, but running on the actual hardware it was consistently off. After extensive testing, I realized that it was a bug in the compiler for the target hardware, such that a very particular type of structure (something like, {int, char, float}) was being packed incorrectly, resulting in a 2-byte pad that shouldn't be there. If I reordered the structure elements, it was fine, but that particular grouping and order refused to work correctly.
It was GCC, so we could fix the compiler ourselves, right? Not really, as, for avionics systems the compiler has to be thoroughly qualified for avionics use, and changes equal requalification. I "fixed" it by storing the float as an array of characters, converting it to and from a real float type as we needed to use the data value.
Trivial, perhaps, but I was very excited to resolve the problem, after spending days barking up wrong trees. One usually expects that the problem is not in the compiler... :-)
Anyway, the standard way (to handle protocols) is to parse the thing not making any assumption of the struct layout.
If I am understanding what you are saying, we really couldn't do that, as the server sent and expected to receive binary blobs of data; the only way to know what was what was to have a map of where the data elements were.
Unfortunately, I forgot to return the printer name to a variable when promoting into production. Hilarity ensued.
Guy I knew - awesomely good - hex edited a DOS boot sector on a in house machine to use FUCK.SYS instead of whatever.sys it normally is (I forget). He renamed that file to fuck.sys, rebooted and the machine ran. Cool!
We laughed and reinstalled DOS and two days later the boss come charging in yelling 'I have a client on the phone who says his new machine can't find FUCK.SYS!'
The awesome guy goes 'uh oh'. I laughed.
The worst one I ever caused was when Visa started carrying two amount fields in their credit card records. One was the amount in original currency, the other was converted to the receiving system's local currency. I used the original amount. Our hand-made test data used the same currency for both amounts, so no problem in test. Imagine my surprise when we went live and our system started posting original currency amounts to cardholder accounts, which at the time only supported US dollars. Luckily, we caught it early and senior management and cardholders were all good sports about it. I think those credit card statements with massive amounts became collector's items.
This also meant you couldn't use READ INTO for variable length records (which is equivalent to a READ followed by a MOVE) without taking some care.
As you say: "newbies shake their heads..."
It came down to me to find and fix the problem, and it was subtle. The clue lay in the fact that all of the sites that crashed did so within about a minute of one another.
Turns out that some of the old, old sections of the software had been written by the MD, who, despite referring to himself as 'the emperor of c', was in fact an atrocious programmer.
The actual trigger was the comms system looking at a byte that determined as to whether a message had been received. This byte was set to the character 'A' if a message was received. It just so happened that the first byte of the current value of the number of seconds since 1970 evaluated to 'A', and had been written into that memory location a negative index into an array that hadn't been initialised.
This negative index into an array that shouldn't have been empty caused a section of memory to be overwritten that made the comms system think that it had received a packet. This snowballed quickly, and took down the system within about five seconds of boot.
Took the best part of two days to track down, and, of course, it was everyone else's fault but the emperors.
It had to do with a connection being reset in the space of time between the server checking if the connection was good, and actually using it.
The client could reproduce it at will, and after a little bit of code at home, I could, too. Back at the office, no repro.
It took a lot of back and forth (to put it mildly), but the problem at work was that the network was too slow - the server got the reset with plenty of time to "notice" it, and the crash didn't occur.
Once we set up the repro on its own switch at work, everything failed in the expected manner.
At the time, Java was a new feature within that platform, so there weren't many apps that mized both languages.
After getting to this point, IBM joined in the fix effort, and we had daily conference calls, on which we always had IBM execs lurking because their 6.0 release of the platform was imminent, and this bug had the potential to wreak major havoc if not fixed before launch.
So I cannot personally claim to have done the actual bugfix - the IBM programmer did that. But it was a great learning experience to work together with IBM to find and fix it.
My incomplete set of unit tests didn't catch it because they were imported by seperate submodules of mine.
I only managed to find it by writing a unit test, going back in version control, searching for the first revision where the problem occurred, and going line by line through the changeset (luckily I commit early and often).