It is good to pore over these legendary tales. They come in handy when we need to break out of the moment and try something outrageous to solve the problem.
I can't scan documents when my daughter is asleep. When she is awake, all is fine, but the minute she goes to sleep, and I'd like to use my free time to scan documents and suchlikes, forget it. I could still print documents on the same device though. Here's what I found:
The printer-scanner was connected to wi-fi. The wi-fi router was in my daughter's room, as that is where the cable socket was, tucked just behind a bench in her room. It was also near that bench that her baby monitor camera was standing. It wasn't wi-fi connected, but for whatever reason it interfered with the wi-fi signal. Same with the receiver, if I put it near my laptop, the wi-fi connection would die.
The monitor was off most of the time, and on precisely when my daughter was asleep.
As for why I could still print, just not scan: presumably that's something to do with the bandwidth, I'm guessing it took more wi-fi bandwidth to send a scanned image than to print a document (I never printed pictures on this printer).
Yeah, baby monitors are THE worst behaving radio devices. They would be second after malfunctioning neon sign transformers, but transformers are not intended as radio devices.
As for why scanner and not printer losing connection - probably printer has small buffer and scanner doesn't pause when that send buffer is full due to trying retransmissions, but stops completely. Printer probably can wait for more data.
Still... my thinking is, I'd rather a crappy baby monitor than a badly-secured device connected to wifi, and by extension, the whole world.
We do actually have a spare camera we sometimes take away, that works on Wi-Fi. I accidentally left it once at my parent's place, then travelled back home overseas. I got a notification of my baby crying desolately as I got off the plane. Turns out it was my family arguing something in the room where my daughter was staying, and the camera somehow switched itself back on. It was freaky!
You really want to think about moving the Wi-fi out of the baby's room. Get a GQ-390 meter and you will see the torrent of dangerous radiation flooding her room, above recommended levels.
I was able to get my Internet provider to relocate the device to my basement.
My comment is an unresearched, not even cursorially google-searched, from-memory assertion, so take with 5 buckets of salt please, and if I'm wrong I'd like to correct my ignorance, so someone please correct me :)
My understanding is that Wi-Fi, cellular (no matter the 12345G) and radio signals are all harmless in 99.99999% circumstances as they are non-ionizing radiation. They are no more harmful to you than light photons.
In fact, Wi-Fi signals might be safer than light. Since you don't have eyes sensitive to the EM wavelength of Wi-Fi, you can't shine a 'wifi light' too brightly next to you.
Damaging radiation as we think of it is most often in the form of ionizing radiation such as too much sunlight, microwaves that excite water molecules, or high amounts of alpha / beta / gamma particles that can mess with DNA etc. and thus cause cancer as DNA damage accumulates.
> Since you don't have eyes sensitive to the EM wavelength of Wi-Fi, you can't shine a 'wifi light' too brightly next to you.
This is dangerously wrong. Vision damage is not prevented by the wavelength not being in the visible spectrum. Even if you can't see it, and even if it is non-ionizing, electromagnetic waves still impart energy.
Eyes cannot efficiently dissipate heat, so sufficiently strong EMF will damage the eye. This is a well-known risk when operating with strong microwave signals, for example.
"Effects of Microwave and Millimeter Wave Radiation on the Eye", J. A. D’AndreaS. Chalfin, NATO Science Series book series (ASHT, volume 82)
> Most of the early research was carried out in the lower portion of the microwave spectrum (at 2.45 GHz) and demonstrated a high dose response relationship between microwave exposure and cataract induction. For example, Carpenter and Van Ummersen irradiated anesthetized rabbits at 2.45 GHz and showed a decreasing threshold for cataractogenesis from 4 minute exposure at 400 mW/cm2 to 40 minutes at 80 mW/cm2• Guy et al. ... repeated some of the earlier research and found essentially the same threshold for cataract production in rabbits exposed with a near field applicator at 2.45 GHz. At minimum, they determined that 150 mW/cm2 was required for 100 min to produce a cataract.
Not saying your Wi-Fi router is going to cause cataracts, but don't think just because you can't see something it can't hurt your vision. In fact, it could be argued invisibly strong EMF is _more_ dangerous than visible light because it doesn't trigger the self-protective blink reflex.
So? EM induces other kind of cellular changes. Maybe not as immediately disruptive as ionizing radiation. Just look up the study linking neuroglioblastoma incidence correlates to the side of the head one uses their cellphone.
> Just look up the study linking neuroglioblastoma incidence correlates to the side of the head one uses their cellphone.
Yes, do. Then read the rest of it. Some choice comments from cancer.gov[0]:
> Most published analyses from this study have shown no statistically significant increases in brain or other central nervous system cancers related to higher amounts of cell phone use. One analysis showed a statistically significant, although modest, increase in the risk of glioma among the small proportion of study participants who spent the most total time on cell phone calls.
> However, the researchers considered this finding inconclusive because they felt that the amount of use reported by some respondents was unlikely and because the participants who reported lower levels of use appeared to have a slightly reduced risk of brain cancer compared with people who did not use cell phones regularly.
Oh look, actually cell phones prevent cancer. Totally. Because one-off results are reliable like that.
It's okay, most experts thought the earth was flat at some point. If you want to learn more about goverments' approach to many health issues I encourage you to read on the history of the concept of hormesis. While the first studies show ionizing radiation to be harmful at any level, of course they later founded researchers who came up with this theory so plenty of lawsuits could be avoided. I already live most aspects of my life as a top 1%, so if there is any risk that EMF can have harmful consequences and it is within my control, I'll mitigate it. 99% percent of the population don't care, don't believe, it's okay. Primary housing in a secluded area? Check. Wired Networking? Check. Non industrialized nutrition on point? Check.
Think there is no theory behind the impact of EMF on living organisms? Might want to read on how cells maintain ions in balance, through passive and active diffusion through the membrane. Easily altered with EMF, can our cells adapt? Sure. Could this adaptation bring other issues/degeneration? Sure
I used to work for an IPTV provider (who also happens to make a search engine). We received TV feeds from TV stations via satellite; we had an antenna farm, received their signals there, compressed/encoded the signal, and then sent them to customers over the network. Because we only had one antenna farm, we would have TV outages throughout the year -- sometimes the satellite happens to be directly in front of the sun, and the sun is a huge RF emitter that would overwhelm our receivers. (I asked if we can just move the satellites, but was told that we didn't have the delta-V budget. We eventually built another antenna farm. Another question I asked is why can't the TV stations just send us video files over the network. Apparently it simply isn't done; they have used satellites for decades, so why switch?)
The main reason 2.4GHz band is unlicensed (anyone can broadcast on it without e.g. a ham license) is because it's garbage. Baby monitors and microwave ovens are two of the most common offenders for dumping large amounts of noise into that band. If you can move the printer to a 5GHz band, that could help a lot.
And in the near future, wifi 6E will add a ton of spectrum, allowing devices to just avoid noisy channels. But to use that you'll have to upgrade the printer or add a wifi 6E bridge.
Had a Customer who complained that the Wi-Fi in one of their conference rooms was "always" unreliable. I checked it multiple times and didn't find any problems-- low SNR, strong signal, low airtime utilization.
Eventually I learned out that "always" meant a particular recurring lunchtime meeting, scheduled right when a steady stream of workers were going into the break room across the hall and heating food in a microwave oven.
I had a Panasonic 2.4Ghz jamming device. Could take out channels 1-11 all at the same time. It wasn't supposed to be a jamming device, it was supposed to be a cordless phone. Setup wifi for a customer, and it worked fine when I was on site. Later the customer complained it was completely unusable, in what they thought was a random fashion.
Turned out whenever they used the cordless it pretty much made so much noise on the 2.4G frequence wifi wouldn't work. Phone itself worked fine. Had them get new phones and the problem went away.
Reminds me of a story from my colleague, who used to work at a tech help desk at a telecom some years ago. He told me about a curious case of a house which, despite being located next to the network node, appeared as if it was kilometers away from the signal loss POV. It turned out that for unknown reasons, instead of cutting the necessary length of cable to install, someone just connected both ends through a fresh cable spool and buried that spool underground.
My favorite out of these is the 500 miles email limitation one. I work mostly on big bulky manufacturing equipment but my job is to abstract out the computing part. This story reminds me that every time I want to do something I am still limited by physics. I am reminded of this story whenever the hardware people ask me to insert an artificial delay in computation.
Yes the old limits of the time style bugs are great. SOme arbitrary limit or variable to hold a value deemed way more than enough at the time, for years/decades later to jump out and catch you out. Y2K was one of those well known ones, but been many of those types of bugs.
Here's the craziest one that actually happened to me.
The company I worked for had installed what's best described as a mini-supercomputer (though we avoided the term) at a site in Boulder. We started getting reports of failures on the internal communication links between the compute nodes ... only at high load, late in the day. Since I was responsible for the software that managed those links, I got sent out. Two days in a row, after trying everything we could to reproduce or debug the problem, I got paged minutes after I'd left (and couldn't get back in) to tell me that it had failed again.
Our original theory was that it had to do with cosmic rays causing bit-flips. This was a well known problem with installations in that area, having caused multi-month delays for some of the larger supercomputer installations in the area. But we'd already corrected for that. It wasn't the problem.
What it ultimately turned out to be was airflow and cooling. The air's thinner up there, so it carries less heat. But it wasn't the processors or links that were getting too hot. It was the power supply. When a power supply gets warmer it gets less efficient. Earlier in the day or with shorter runs as we tried different things this wasn't enough to cause a problem. With it being warmer later in the day, continuous load for longer periods was enough to cause slight brown-outs, and those were making our links flaky. And of course it would always restart just fine because it had cooled down a bit.
The fix ended up being one line in a fan-controller config.
> Our original theory was that it had to do with cosmic rays causing bit-flips. This was a well known problem with installations in that area, having caused multi-month delays for some of the larger supercomputer installations in the area. But we'd already corrected for that.
Wow, I sense a more interesting story in here. Care to reveal how it was first found out and how common it actually is?
In a nutshell, cosmic rays causing bit-flips really is a thing, and it's more of a thing at higher altitude because of less atmosphere. It's rarely a problem at sea level. At higher altitude you really need to use ECC memory, and do some sort of scrubbing (in Linux it's called Error Detection And Correction or EDAC) to correct single-bit errors before they accumulate and some word somewhere becomes uncorrectable.
The incident that brought this home to a lot of people was at either NCAR or UCAR, both near Boulder. Whichever it was, they were installing a new system - tens of thousands of nodes - and had not been careful about the EDAC settings. Therefore, EDAC wasn't running often enough, and wasn't catching those single-bit errors. Therefore^2, uncorrectable errors were bringing down nodes constantly. According to rumor, this caused a huge delay and almost torched the entire project. It's easy to say in retrospect that they should have checked the EDAC settings first, but as it happened they probably only got to that after multiple rounds of blaming the vendor for flaky hardware (which would generally be the more likely cause especially when you're on the bleeding edge).
> It's easy to say in retrospect that they should have checked the EDAC settings first, but as it happened they probably only got to that after multiple rounds of blaming the vendor for flaky hardware (which would generally be the more likely cause especially when you're on the bleeding edge).
Yeah, part of the nightmare of cosmic-ray bitflips (or any random bitflips, I suppose) is precisely that they don't look like anything. A server randomly locks up. A packet has a bad checksum (and is silently resent). A process gets into an unexpected state. That buggy batch job fails 1% more frequently than it used to. Nothing ever points to memory errors, except that there is no pattern.
Unrelated to a strange bug story or anything but you just reminded me of when I was also helping someone set up a, as you called it, mini-supercomputer. It was to do quantum simulations. We were setting it up and the researcher who was going to use it made the root user name skynet. Now I know that joke has probably been played out at campuses around the world but it just seems unnecessary to tempt the fates like that.
I had a loaner machine (RS-6000 minicomputer) that would have unrecoverable ECC errors when the cover was on. The tech would come and try to diagnose it, but with the cover off, everything would work fine. He'd swap the memory anyway and put the cover back on. within a few hours the memory bank would be failing again. Turned out the machine had been a loaner in a lab where it had acquired some alpha-emitting goo on the inside of the side panel. The lab had just run it with the side panel off to solve the problem, never noticing the goo, never mentioning it to IBM when they packed it up to ship.
It's a long story but the gist is after multiple board swaps, realizing we'd isolated the panel as the fault, I noticed the goo and on a hunch checked it with a scintillator, deducing it was alpha when cardboard blocked it. Turns out the ultra-precious-metal IBM heat sink on the board had an open path that effectively channeled the alpha particles into one of those multi-chip carrier thingies, which featured exposed chips.
As for why I had a scintillator lounging in my desk at a portfolio management company, don't ask. Let's just note the iconic IT anti-hero of that era was the Bastard Operator From Hell, and leave it at that.
One time I was writing some code in C. I found a bug, the solution seemed pretty obvious, so I fixed it, recompiled the code, and ran it again. The bug was still there.
I took a look at the rest of the code in case that I missed something. I couldn't find anything, so I added a few print statements and recompiled. I ran the code and nothing came up.
Interesting, apparently the code is not executing the branches it should. I verified the input data and code. It didn't make sense, there had to be some serious bug there that I didn't consider. I added a bunch more prints.
Recompile and execute. Still nothing. Wait a minute, THAT doesn't look good. I added a print statement right at the entry point of the program. Nothing.
At this point the root problem became apparent; my changes just weren't getting compiled. Phew, problem solved! I cleaned all the cached files and recompiled the source code. Those print statements still weren't coming up.
At the end I had to move my source code to another machine and compile it there to get it working. I suspect some global variables or path trickery to be involved, but up to this day I still haven't got a clue what was wrong, or have I seen it happen again.
Ahhahahaha. I have a similar story to that, but I eventually realized what happened.
I forget which command it was exactly, but I rsync'd or something to get a new code directory, and with the backup options in use the directory I was in got renamed.
But I still had command prompts open in that directory. And all of the files were there. So I didn't realize that one directory was not equal to the others even though it had the same name (it was a subdirectory) and appeared to have the same files.
One similar story: I was maintaining some C++ code that had a few #ifdefs in it. Someone reported a problem.
I put a breakpoint on the calling code and traced into my code. It went into the #ifdef code I expected, but the problem persisted.
Just to double-check, I let the program run until it hit the breakpoint again and traced in, but this time, it went into the #else code! That code should have been removed by the preprocessor, yet here I was, currenting stepping through it.
After questioning my understanding of the C preprocessor (and my own sanity), I luckily noticed that the module name in the debugger was very slightly different in the two cases mentioned above.
The world finally made sense again. My code was in a header file that was compiled (with different symbols defined) into two different modules and both of those modules were loaded into the same process. When I set the breakpoint in the debugger, it silently set breakpoints in both modules.
"Surprisingly, we have also seen this issue connected to gas lift office chairs. When people stand or sit on gas lift chairs, they can generate an EMI spike which is picked up on the video cables, causing a loss of sync. If you have users complaining about displays randomly flickering it could actually be connected to people sitting on gas lift chairs. Again swapping video cables, especially for ones with magnetic ferrite ring on the cable, can eliminate this problem. There is even a white paper about this issue."
Risks Digest was on my daily reading list for years. I've been in computing since the early 70s, the history of computing is fabulous background to inform e.g. debugging, or SRE.
I've seen this one on twitter https://twitter.com/royvanrijn/status/1214162400666103808?la... There's probably some civilizational complexity limit where the unexpected interactions between seemingly isolated pieces of tech become so bad that we cannot introduce anything new without introducing legion of weird bugs.
>There's probably some civilizational complexity limit where the unexpected interactions between seemingly isolated pieces of tech become so bad that we cannot introduce anything new without introducing legion of weird bugs.
Come to think of it, I believe that meme was wondering about on UseNet from before the early days. I think Vernor Vinge alluded to it in one of his novels, (paraphrasing here) some interplanetary civilization crashing because in-transit space traffic was so dense no new launches could occur in a useful time frame due to safety-lock outs, and they didn't want to accept the risk in changing the safety margins...
My most recent bizarre bug: a coworker came to me with a bug where no matter what he tried, he could not get an if some_var is null to be true. The debugger would show the value was null. The logger showed the value was null, but the if statement would not work. After a morning of trying to fix it, he asked if I would take a look. I told him to put the null in quotes in the if. It worked. Turned out a JavaScript library had a bug where it would use the string "null" instead of null.
There's a popular progressive API I've been forced to work with that uses "true" and "false" instead of their respective booleans. The most egregious that I've ever seen in this class of errors was an API (from Google!) that returned " false".
That describes a real machine, the LGP-30, which really had absolutely no business being as powerful for the price and era as it was. It used an oscilloscope for its (tiny) debug display: the operator’d read the voltage waveform directly as binary.
My favourite crazy bug was during a university course on autonomous robotics. One of the other groups was using a a metal castor at the back of the robot along with 2 driven wheels. After a little while their robot would completely crash and stop responding.
I'd previously encountered a similar issue which was due to the library code we'd been given which opened a new /dev/i2c file for each motor command, eventually exceeding the max file handles and killing the program. So I assumed it was something sensible like that.
Some time later they got all excited and called us over to explain the real reason it was crashing. Their robot would initially work fine for a reasonable period of time. Then when the robot drove over the metallic tape on the floor of the arena it would die. The robot must have been building up a static charge while moving around which would eventually be dissipated when the metal touched the tape.
I wouldn't have believed it had they not setup two tests, one outside the normal arena and one inside. Changing the metal castor for a bit of lego fixed the problem.
I used to work as a senior technical support for BEA Weblogic and had all sorts of crazy situations to debug remotely. Including one time when I had to get a person to edit a config file in Vim (which they never used before), on Unix (which they never used before), with me guiding them by phone (no visuals).
And if somebody really understands network and multicast, I would love to know whether I actually nailed the problem or just made it go away accidentally. I have no problems with being wrong, especially this much later :-)
I remember reading one years ago where someone had a problem installing new software on some embedded device - whatever they did it came up "checksum is bad".
After much testing they eventually realised that the checksum literally was the hex "bad".
Obvious in retrospect, but very surprising to my inexperienced past self:
I'd been working on some C code for an hour or two. It wasn't behaving how I expected it to (and at the time I knew nothing of debuggers), so I added a print statement and recompiled. I got a compilation error: something like "Syntax error on line 123: #incl5de <stdio.h>". Shocked, I scrolled to that line in my text editor to fix the typo, but it wasn't there. I compiled the same code again and there were no errors.
Turns out there wasn't a bug in my code! I immediately shut down my computer because my RAM was going bad. To this day, what surprises me most is that my computer was able to successfully boot and behave normally for an hour or two, even though random bits were apparently being flipped.
The RAM going bad in my PC was one of the most annoying issues I had to troubleshoot. It started with having my firefox pages randomly crash. first occasionally and then several times a day.
I then started getting errors when playing games with obscure error codes - which yielded nothing when searching them up.
I eventually found a comment in a thread about the crashing game that the RAM could be bad. I ran some diagnostic tests and with the number of errors that came up I was surprised my computer worked at all
> To this day, what surprises me most is that my computer was able to successfully boot and behave normally for an hour or two, even though random bits were apparently being flipped.
Yup. I once had a machine that would freeze up when I ran package updates. I thought (of course), that the package manager had a bug. Turned out, running the upgrade was the only thing memory-intensive enough to use the faulty memory that I'd installed. After all, on a light system you can totally boot and run in <1GB of RAM...
> To this day, what surprises me most is that my computer was able to successfully boot and behave normally for an hour or two, even though random bits were apparently being flipped.
With my current computer I overclocked the RAM to the best config I could get memtest to run without errors over the night. The RAM also has ECC and there were no problems reported during normal operation, even when re-compiling (most) packages. But when I got to compiling LLVM the system would crash shortly after logging ECC errors. Backing off the overclock a bit fixed that. So the rate of memory errors can definitely depend on what you are running.
"Fail on certain moon phases" reminds me of a C++ bug I encountered while trying to set up the demo for PSIP (Digital TV Guide) destined for NAB in Las Vegas. We had programming schedules resembling excel spreadsheets and my job was just to create a good one for the demo. I would spend all night making one and sent it to my boss and each morning would get in trouble for sending in blank schedules and had no idea why. On one occasion I happened to be editing at 3am and noticed all of my edits rolling back one by one. It was actually viewable on the screen as if someone took control of excel and was rolling back each field. My immediate thought was I really need to get some sleep but later we found the auto-save feature inverted itself after 3am exactly and would go through each delta one by one rolling itself back as it had been edited. The bug was found in the calculation of the vernal equinox which moves from 3am to 9pm to 3pm. Since it was triggering the leap year code 6 hours of time would get rolled back edits and all! This was of course 2008 year of the digital transition from analog cable which happened to also be a leap year.
We're actually working on a collection of such stories internal to our division. We've found that these tales are a great way of helping people understand the complexities and quirks of our nearly 3 decade old code base.
I think story telling is an underrated technique in our profession.
In all projects there are coding rules (like "make destructors noexcept"). A rule sticks much better if you also tell a story about some debugging caused by not following the rule.
I worked at a place one that had a style guide for the main front-end language used with links to terrible things that had happened as a result of breaking the style guides rules.
It was surprisingly effective, so I completely agree with you on the story-telling.
I love programmer stories like this. My favourite personal experience was on my first Ruby on Rails project after first moving to London. I was pretty green at the time, having had only a few years of PHP experience under my belt and little else.
We had to build a Rails app around a poker game. We didn't own the source to the poker game or its API, but we had to embed it nonetheless. We had this really strange issue where some people, under a certain circumstance, couldn't get into the game. It would just boot them out. Me and my team mate must have poured through the Ruby code dozens and dozens of times and found no evidence of this bug, no ability to reproduce it; bearing in mind I was still learning the ropes and jumping head first into an unfamiliar codebase is quite daunting.
Eventually I decide to get my hands dirty and I start poking into this game engine. We embedded it as s flash widget, but the server doing most of the work was written in a mix of C++ and Python. I didn't fully comprehend what I was looking at but, even though things looked suspicious, I couldn't put my finger on an actual problem until I looked at the API written in Flask and noticed that one line of code didn't look like any other.
some_value = params['some_key']
If the request didn't contain the parameter `some_key` then this would raise a KeyError.
After maybe three solid weeks of trying to debug this thing, I submitted a one line patch:
some_value = params.get('some_key')
It's not quite as weird or as fun as most examples but for me personally it was such a great lesson in debugging and being curious about unfamiliar stuff, rather than closed off or afraid.
My favorite was when I was working at SGI after it had taken over Cray Research. I was one of the lowly Cray guys in Wisconsin working with the wonderkind in California. I was to run the regression tests on the chip being design in California using some software that they had provided. I would run the tests, but some days they would crash in the middle of the night. Then the California guys would be angry that they got no tests results. I started debugging the code and got to a program called lswalk that would dole out jobs to the dozens of servers to be 'run'. The code was written by a hot shot young MIT graduate, but I was sure that the problem was with this code. I got the source code and started looking for problems and one thing I found was that if one of the servers resplonding with error the code would print out an error message. One problem though... The error string printed had an uninitialized string, so that when the printf routine would search for an end of string that was never there, probably overwriting buffers and crashing code all over the place. So one lesson is that even the best and brightest make mistakes.
Sometimes I wonder how we accomplished anything in those days with software that had so may trap doors beneath it.
I have two favourite bugs, one weird and one dumb.
Weirdest one was an IDE where the colorizer gave up on source lines longer than 998 chars. Instead it rendered the whole line as background, i.e. invisible. I once wasted two hours debugging a program with an invisible line of code!
The dumbest was a postage billing system for a bank using a third party Print-and-Mail company. Somehow the billing system went live adding the previous day's total postage costs to itself, then adding the new day's postage. These expontentially growing totals were then paid automatically by the accounting system each night... So it goes live, ... and a week later Finance gets an alert the account is overdrawn... They actually paid out nearly $1b in postage costs before hitting their internal credit limit with the bank's treasury.
A highly customised Visual Studio I think, with lots of in-house extensions. To be fair, this was long, long ago. Nowadays you'd write an extension and leverage the Language Server Protocol.
112 comments
[ 4.4 ms ] story [ 173 ms ] threadThe printer-scanner was connected to wi-fi. The wi-fi router was in my daughter's room, as that is where the cable socket was, tucked just behind a bench in her room. It was also near that bench that her baby monitor camera was standing. It wasn't wi-fi connected, but for whatever reason it interfered with the wi-fi signal. Same with the receiver, if I put it near my laptop, the wi-fi connection would die.
The monitor was off most of the time, and on precisely when my daughter was asleep.
As for why I could still print, just not scan: presumably that's something to do with the bandwidth, I'm guessing it took more wi-fi bandwidth to send a scanned image than to print a document (I never printed pictures on this printer).
As for why scanner and not printer losing connection - probably printer has small buffer and scanner doesn't pause when that send buffer is full due to trying retransmissions, but stops completely. Printer probably can wait for more data.
We do actually have a spare camera we sometimes take away, that works on Wi-Fi. I accidentally left it once at my parent's place, then travelled back home overseas. I got a notification of my baby crying desolately as I got off the plane. Turns out it was my family arguing something in the room where my daughter was staying, and the camera somehow switched itself back on. It was freaky!
I was able to get my Internet provider to relocate the device to my basement.
Could you point me to any articles etc. on this?
Thanks in advance!
My understanding is that Wi-Fi, cellular (no matter the 12345G) and radio signals are all harmless in 99.99999% circumstances as they are non-ionizing radiation. They are no more harmful to you than light photons.
In fact, Wi-Fi signals might be safer than light. Since you don't have eyes sensitive to the EM wavelength of Wi-Fi, you can't shine a 'wifi light' too brightly next to you.
Damaging radiation as we think of it is most often in the form of ionizing radiation such as too much sunlight, microwaves that excite water molecules, or high amounts of alpha / beta / gamma particles that can mess with DNA etc. and thus cause cancer as DNA damage accumulates.
This is dangerously wrong. Vision damage is not prevented by the wavelength not being in the visible spectrum. Even if you can't see it, and even if it is non-ionizing, electromagnetic waves still impart energy.
Eyes cannot efficiently dissipate heat, so sufficiently strong EMF will damage the eye. This is a well-known risk when operating with strong microwave signals, for example.
"Effects of Microwave and Millimeter Wave Radiation on the Eye", J. A. D’AndreaS. Chalfin, NATO Science Series book series (ASHT, volume 82)
> Most of the early research was carried out in the lower portion of the microwave spectrum (at 2.45 GHz) and demonstrated a high dose response relationship between microwave exposure and cataract induction. For example, Carpenter and Van Ummersen irradiated anesthetized rabbits at 2.45 GHz and showed a decreasing threshold for cataractogenesis from 4 minute exposure at 400 mW/cm2 to 40 minutes at 80 mW/cm2• Guy et al. ... repeated some of the earlier research and found essentially the same threshold for cataract production in rabbits exposed with a near field applicator at 2.45 GHz. At minimum, they determined that 150 mW/cm2 was required for 100 min to produce a cataract.
Not saying your Wi-Fi router is going to cause cataracts, but don't think just because you can't see something it can't hurt your vision. In fact, it could be argued invisibly strong EMF is _more_ dangerous than visible light because it doesn't trigger the self-protective blink reflex.
Yes, do. Then read the rest of it. Some choice comments from cancer.gov[0]:
> Most published analyses from this study have shown no statistically significant increases in brain or other central nervous system cancers related to higher amounts of cell phone use. One analysis showed a statistically significant, although modest, increase in the risk of glioma among the small proportion of study participants who spent the most total time on cell phone calls.
So, https://xkcd.com/882/ strikes again.
> However, the researchers considered this finding inconclusive because they felt that the amount of use reported by some respondents was unlikely and because the participants who reported lower levels of use appeared to have a slightly reduced risk of brain cancer compared with people who did not use cell phones regularly.
Oh look, actually cell phones prevent cancer. Totally. Because one-off results are reliable like that.
[0] https://www.cancer.gov/about-cancer/causes-prevention/risk/r...
Think there is no theory behind the impact of EMF on living organisms? Might want to read on how cells maintain ions in balance, through passive and active diffusion through the membrane. Easily altered with EMF, can our cells adapt? Sure. Could this adaptation bring other issues/degeneration? Sure
....errr....
Wait.
I have just been slipped a note that the first sentence I wrote means "I just went for a walk in sunshine."
I used to work for an IPTV provider (who also happens to make a search engine). We received TV feeds from TV stations via satellite; we had an antenna farm, received their signals there, compressed/encoded the signal, and then sent them to customers over the network. Because we only had one antenna farm, we would have TV outages throughout the year -- sometimes the satellite happens to be directly in front of the sun, and the sun is a huge RF emitter that would overwhelm our receivers. (I asked if we can just move the satellites, but was told that we didn't have the delta-V budget. We eventually built another antenna farm. Another question I asked is why can't the TV stations just send us video files over the network. Apparently it simply isn't done; they have used satellites for decades, so why switch?)
In fact, the other people in your home are bathing you in infrared radiation when you're near them.
And in the near future, wifi 6E will add a ton of spectrum, allowing devices to just avoid noisy channels. But to use that you'll have to upgrade the printer or add a wifi 6E bridge.
Eventually I learned out that "always" meant a particular recurring lunchtime meeting, scheduled right when a steady stream of workers were going into the break room across the hall and heating food in a microwave oven.
Turned out whenever they used the cordless it pretty much made so much noise on the 2.4G frequence wifi wouldn't work. Phone itself worked fine. Had them get new phones and the problem went away.
It turned out to be the subscribers (a little old lady) budgie swinging on its perch was causing the problem.
The company I worked for had installed what's best described as a mini-supercomputer (though we avoided the term) at a site in Boulder. We started getting reports of failures on the internal communication links between the compute nodes ... only at high load, late in the day. Since I was responsible for the software that managed those links, I got sent out. Two days in a row, after trying everything we could to reproduce or debug the problem, I got paged minutes after I'd left (and couldn't get back in) to tell me that it had failed again.
Our original theory was that it had to do with cosmic rays causing bit-flips. This was a well known problem with installations in that area, having caused multi-month delays for some of the larger supercomputer installations in the area. But we'd already corrected for that. It wasn't the problem.
What it ultimately turned out to be was airflow and cooling. The air's thinner up there, so it carries less heat. But it wasn't the processors or links that were getting too hot. It was the power supply. When a power supply gets warmer it gets less efficient. Earlier in the day or with shorter runs as we tried different things this wasn't enough to cause a problem. With it being warmer later in the day, continuous load for longer periods was enough to cause slight brown-outs, and those were making our links flaky. And of course it would always restart just fine because it had cooled down a bit.
The fix ended up being one line in a fan-controller config.
Wow, I sense a more interesting story in here. Care to reveal how it was first found out and how common it actually is?
The incident that brought this home to a lot of people was at either NCAR or UCAR, both near Boulder. Whichever it was, they were installing a new system - tens of thousands of nodes - and had not been careful about the EDAC settings. Therefore, EDAC wasn't running often enough, and wasn't catching those single-bit errors. Therefore^2, uncorrectable errors were bringing down nodes constantly. According to rumor, this caused a huge delay and almost torched the entire project. It's easy to say in retrospect that they should have checked the EDAC settings first, but as it happened they probably only got to that after multiple rounds of blaming the vendor for flaky hardware (which would generally be the more likely cause especially when you're on the bleeding edge).
Yeah, part of the nightmare of cosmic-ray bitflips (or any random bitflips, I suppose) is precisely that they don't look like anything. A server randomly locks up. A packet has a bad checksum (and is silently resent). A process gets into an unexpected state. That buggy batch job fails 1% more frequently than it used to. Nothing ever points to memory errors, except that there is no pattern.
It's a long story but the gist is after multiple board swaps, realizing we'd isolated the panel as the fault, I noticed the goo and on a hunch checked it with a scintillator, deducing it was alpha when cardboard blocked it. Turns out the ultra-precious-metal IBM heat sink on the board had an open path that effectively channeled the alpha particles into one of those multi-chip carrier thingies, which featured exposed chips.
As for why I had a scintillator lounging in my desk at a portfolio management company, don't ask. Let's just note the iconic IT anti-hero of that era was the Bastard Operator From Hell, and leave it at that.
One time I was writing some code in C. I found a bug, the solution seemed pretty obvious, so I fixed it, recompiled the code, and ran it again. The bug was still there.
I took a look at the rest of the code in case that I missed something. I couldn't find anything, so I added a few print statements and recompiled. I ran the code and nothing came up.
Interesting, apparently the code is not executing the branches it should. I verified the input data and code. It didn't make sense, there had to be some serious bug there that I didn't consider. I added a bunch more prints.
Recompile and execute. Still nothing. Wait a minute, THAT doesn't look good. I added a print statement right at the entry point of the program. Nothing.
At this point the root problem became apparent; my changes just weren't getting compiled. Phew, problem solved! I cleaned all the cached files and recompiled the source code. Those print statements still weren't coming up.
At the end I had to move my source code to another machine and compile it there to get it working. I suspect some global variables or path trickery to be involved, but up to this day I still haven't got a clue what was wrong, or have I seen it happen again.
I forget which command it was exactly, but I rsync'd or something to get a new code directory, and with the backup options in use the directory I was in got renamed.
But I still had command prompts open in that directory. And all of the files were there. So I didn't realize that one directory was not equal to the others even though it had the same name (it was a subdirectory) and appeared to have the same files.
I put a breakpoint on the calling code and traced into my code. It went into the #ifdef code I expected, but the problem persisted.
Just to double-check, I let the program run until it hit the breakpoint again and traced in, but this time, it went into the #else code! That code should have been removed by the preprocessor, yet here I was, currenting stepping through it.
After questioning my understanding of the C preprocessor (and my own sanity), I luckily noticed that the module name in the debugger was very slightly different in the two cases mentioned above.
The world finally made sense again. My code was in a header file that was compiled (with different symbols defined) into two different modules and both of those modules were loaded into the same process. When I set the breakpoint in the debugger, it silently set breakpoints in both modules.
https://support.displaylink.com/knowledgebase/articles/73861...
"Surprisingly, we have also seen this issue connected to gas lift office chairs. When people stand or sit on gas lift chairs, they can generate an EMI spike which is picked up on the video cables, causing a loss of sync. If you have users complaining about displays randomly flickering it could actually be connected to people sitting on gas lift chairs. Again swapping video cables, especially for ones with magnetic ferrite ring on the cable, can eliminate this problem. There is even a white paper about this issue."
Come to think of it, I believe that meme was wondering about on UseNet from before the early days. I think Vernor Vinge alluded to it in one of his novels, (paraphrasing here) some interplanetary civilization crashing because in-transit space traffic was so dense no new launches could occur in a useful time frame due to safety-lock outs, and they didn't want to accept the risk in changing the safety margins...
My most recent bizarre bug: a coworker came to me with a bug where no matter what he tried, he could not get an if some_var is null to be true. The debugger would show the value was null. The logger showed the value was null, but the if statement would not work. After a morning of trying to fix it, he asked if I would take a look. I told him to put the null in quotes in the if. It worked. Turned out a JavaScript library had a bug where it would use the string "null" instead of null.
https://en.wikipedia.org/wiki/LGP-30
I'd previously encountered a similar issue which was due to the library code we'd been given which opened a new /dev/i2c file for each motor command, eventually exceeding the max file handles and killing the program. So I assumed it was something sensible like that.
Some time later they got all excited and called us over to explain the real reason it was crashing. Their robot would initially work fine for a reasonable period of time. Then when the robot drove over the metallic tape on the floor of the arena it would die. The robot must have been building up a static charge while moving around which would eventually be dissipated when the metal touched the tape.
I wouldn't have believed it had they not setup two tests, one outside the normal arena and one inside. Changing the metal castor for a bit of lego fixed the problem.
This is the one I recorded that seems to fit into the current theme: https://www.outerthoughts.com/2004/10/perfect-multicast-stor... (tldr: multicasting on 237.0.0.1 is bad).
And if somebody really understands network and multicast, I would love to know whether I actually nailed the problem or just made it go away accidentally. I have no problems with being wrong, especially this much later :-)
I mean, if you simply want general computing legends of unconfirmed veracity, read “The Devouring Fungus” by Karla Jennings.
After much testing they eventually realised that the checksum literally was the hex "bad".
“COMPUTER-RELATED HORROR STORIES, FOLKLORE, AND ANECDOTES”
https://www.cs.earlham.edu/~skylar/humor/Unix/computer.folkl...
“Computer Stupidities”
http://www.rinkworks.com/stupid/
I'd been working on some C code for an hour or two. It wasn't behaving how I expected it to (and at the time I knew nothing of debuggers), so I added a print statement and recompiled. I got a compilation error: something like "Syntax error on line 123: #incl5de <stdio.h>". Shocked, I scrolled to that line in my text editor to fix the typo, but it wasn't there. I compiled the same code again and there were no errors.
Turns out there wasn't a bug in my code! I immediately shut down my computer because my RAM was going bad. To this day, what surprises me most is that my computer was able to successfully boot and behave normally for an hour or two, even though random bits were apparently being flipped.
I then started getting errors when playing games with obscure error codes - which yielded nothing when searching them up.
I eventually found a comment in a thread about the crashing game that the RAM could be bad. I ran some diagnostic tests and with the number of errors that came up I was surprised my computer worked at all
Ref: https://news.ycombinator.com/item?id=13859733
Yup. I once had a machine that would freeze up when I ran package updates. I thought (of course), that the package manager had a bug. Turned out, running the upgrade was the only thing memory-intensive enough to use the faulty memory that I'd installed. After all, on a light system you can totally boot and run in <1GB of RAM...
With my current computer I overclocked the RAM to the best config I could get memtest to run without errors over the night. The RAM also has ECC and there were no problems reported during normal operation, even when re-compiling (most) packages. But when I got to compiling LLVM the system would crash shortly after logging ECC errors. Backing off the overclock a bit fixed that. So the rate of memory errors can definitely depend on what you are running.
In all projects there are coding rules (like "make destructors noexcept"). A rule sticks much better if you also tell a story about some debugging caused by not following the rule.
It was surprisingly effective, so I completely agree with you on the story-telling.
We had to build a Rails app around a poker game. We didn't own the source to the poker game or its API, but we had to embed it nonetheless. We had this really strange issue where some people, under a certain circumstance, couldn't get into the game. It would just boot them out. Me and my team mate must have poured through the Ruby code dozens and dozens of times and found no evidence of this bug, no ability to reproduce it; bearing in mind I was still learning the ropes and jumping head first into an unfamiliar codebase is quite daunting.
Eventually I decide to get my hands dirty and I start poking into this game engine. We embedded it as s flash widget, but the server doing most of the work was written in a mix of C++ and Python. I didn't fully comprehend what I was looking at but, even though things looked suspicious, I couldn't put my finger on an actual problem until I looked at the API written in Flask and noticed that one line of code didn't look like any other.
If the request didn't contain the parameter `some_key` then this would raise a KeyError.After maybe three solid weeks of trying to debug this thing, I submitted a one line patch:
It's not quite as weird or as fun as most examples but for me personally it was such a great lesson in debugging and being curious about unfamiliar stuff, rather than closed off or afraid.Weirdest one was an IDE where the colorizer gave up on source lines longer than 998 chars. Instead it rendered the whole line as background, i.e. invisible. I once wasted two hours debugging a program with an invisible line of code!
The dumbest was a postage billing system for a bank using a third party Print-and-Mail company. Somehow the billing system went live adding the previous day's total postage costs to itself, then adding the new day's postage. These expontentially growing totals were then paid automatically by the accounting system each night... So it goes live, ... and a week later Finance gets an alert the account is overdrawn... They actually paid out nearly $1b in postage costs before hitting their internal credit limit with the bank's treasury.