Tell HN: The 10-bit timers are about to overflow on September 17th
Loss of historical data and event logs, logging and security problems, loss of process visualization - these are some of the surprises that can happen when, having received the wrong date, other devices also decide to "time travel."
What to do?
If your network is running one of the aforementioned devices, it's best to disconnect it in advance. Unfortunately, most of them are no longer supported by the manufacturer and no patches are expected. So it looks like they will become quite useless after September 17.
This leaves very little time, therefore, to replace them with new solutions. If you are not sure whether the problem also affects your device, you can unplug it on September seventeenth, and if it shows the correct date the next day, plug it in again. Of course, such a maneuver is possible only in those networks that can operate without time synchronization according to GPS for several/some hours.
83 comments
[ 3.6 ms ] story [ 184 ms ] threadBut instead it is a device-specific rollover. It’s relatively common for GPS receivers to handle the limited week counter by having a baked-in epoch that is more recent than the latest rollover, which gives them a nearly 20 year lifetime after that epoch instead of failing at the GPS rollover.
I think devices that need a longer life should keep track of the last seen timestamp, then they can detect a rollover. The device would keep a local rollover counter and just add 20 years per count. This should stay correct until that counter rolls over, or if the device is offline for >20 years.
If you’re going to use more space keep a rollover counter you may as well just add more bits to your timestamp field instead, since every bit doubles the amount of time that field can represent.
The GPS week counter is part of the signal GPS satellites send and is not easily changed by the manufacturer of receivers.
Of course, devices that accept CNAV now have a few extra bits to work with.
Better in some ways, worse in others.
Can still result in unexpected behavior if configuration memory is lost.
And if it ever accepts an invalid future time for whatever reason, it gets "stuck" in that failure state.
Truncate the current wall clock time to a 10 bit unsigned, then subtract the number reported by the device, and interpret the result as an unsigned 10 bit. That's the amount of time that has passed since the device emitted its timestamp. (Assuming at most one wraparound event.)
Sure it's still a relatively easy mistake to either wrongly assume that all of whatever you're creating will be in landfill by 2022 if you were making it long enough ago that all devices would be dead by now for other reasons (or to unethically do that intentionally, to cause customers to repurchase products this year - seems pretty unlikely but perhaps...), or to just forget to think about it, or think about it and assume it's fine for now and can be checked later before then forgetting.
But "We as a human race have been terrible at anticipating how fast numbers grow" - we're talking about a number that's growing exactly in sync with time, and with an entirely simple way of checking when the limit will be hit, so I think it's just sloppy/lazy or expecting less longevity of use for their product, more than a flaw of humanity.
Taking OP at face value, these devices' counter has a zero point on February 1, 2003 so if we don't know what happens when they roll over they must have been designed after that point.
In the world of the 2000s even 16 bit processing is old school, so there's no good reason this couldn't have been a 16 bit counter instead. If that had been the case the rollover wouldn't be until the year 3259.
IMO this is the correct way to handle situations where you think you don't need any more, round up from your reasonable limit to the next major "bit border". If you have a 10 bit counter, round up to 16. If you need 16, then make it 24 or 32. The limit shouldn't just be "above where we expect it to reach in its lifespan" but "if you hit this the device belongs in a museum".
Designing a device with a 10 bit week counter in the 2000s is bad design, prioritizing either laziness or cost over quality.
No, they just didn't anticipate that a GPS receiver would not know which decade it is (knowing the decade is enough to disambiguate a 10-bit week number) even though it knew the correct date a few moments ago. That is, they didn't anticipate counter rollover bugs caused by hard-coding of the starting point of the counter, instead of calculating it based on the last seen date.
Give some credit to the engineers who built these receivers and their firmware. They're extremely complex and nuanced. You can be absolutely sure that they carefully considered their options here and chose a tradeoff that made sense given the constrains they were designing for. I'm confident they anticipated them, but they just concluded that the fixes beyond using the mfgr date as a base point were worse than the disease.
It's easy to criticize from the future when we know what happened, e.g. e.g. that there was an almost 20 year delay in deployment of the modernized GPS signals and many people kept using these devices far longer than anticipated.
But in this case it was easy ... it's a timer!
The extra bits of storage are cheap. The problem is the addition circuitry. With a small counter, you can do addition in a single clock cycle, very easy to implement. With a large counter, addition of the highest bit has to wait for completion of the previous one, etc. so if this takes longer than one clock cycle you have to implement a different, pipelined addition algorithm. (Or run everything at lower clock frequency.)
This comes with all kinds of caveats and complications(PLL clock multiplication is "fuzzy" and subject to phase drift and variations in actual frequency while normal clock division is essentially as close to exact as you can really get). Because of this, like mentioned it's preferable to use clock division instead of clock multiplication where possible.
Yes, this is called a "clock domain"; there may be quite a lot of them, and they can often be powered off individually.
> I can imagine trying to hack this sort of thing by putting some sort of subprocessor structure that does addition for a particular set of registers at twice normal speed
It's the other way round: a particular arrangement of logic elements, at a particular size and on a particular wafer process, at a particular temperature, will have a worst-case timing. That timing determines how fast you can possibly clock that part of the circuit.
Adders are annoying because of carry: you can't determine the top bit until you've determined the effect of carry on all the other bits. So if it takes, say, 250ps to propagate through your 32-bit adder, you can clock that at 4GHz. If you widen it to 64 bits that takes 500ps, and now you can only clock that bit at 2GHz.
Carry look-ahead adders are a thing. The number of logic levels for computing the highest carry bit is logarithmic in the width of the numbers being added, not linear. Doubling the width of the numbers does not cut your clock rate in half, though you do have to pay for the faster cycle time in added area (more logic gates). There are all sorts of different trade-offs that are possible in the constant terms, but the standard adder designs have linear area in the number of bits, and logarithmic propagation time from inputs to outputs.
It's a linear number of gates, but slightly worse than linear area due to routing. (I've looked for a strictly-linear design before, so if you have such a design, I'd quite appreciate a citation.) It's still much better than N log N area, though, so your point stands.
> Do you have some reference
Nope; that's why I was asking for one.
Kogge Stone carry lookahead.
You can calculate the upper bits in parallel using Kogge-stone (which is the predecessor to what GPU programmers call "prefix-sum" parallelism)
I saw an MITx class on EdX called "Computation Structures" a few years ago and took it for fun. In the second part of that students design at the logic gate level a 32-bit RISC processor, except that we could assume black boxes for the register file and memory.
I considered trying to actually build mine using good old classic 74xx series logic. Mine would have needed 140 quad MUX2 chips, 75 quad AND2 chips, 81 dual MUX4 chips, and 55 quad XOR2 chips, 16 dual D flip flop chips, plus a handful of others.
It was around 370 chips in total.
My design included a shift unit that can do left or right shifts, arithmetic or logical, of 1 to 31 bits in 1 clock cycle.
If I replaced that with a shift unit that could only do 1 bit logical right shift, and changed the instruction set to have a new instruction for that, made the old shift instructions trap, and then emulated them in the trap handler, a whopping 88 of those 140 quad MUX2 chips would no longer be needed.
That would bring it down to around 280 chips. The fancy shifter was almost a quarter of the parts count!
If this is the GPS week number, there is very limited space for it in the GPS navigation message; every extra bit used for the counter means one less bit for everything else. From what I've read, the navigation message is transmitted at only 50 bits per second, and the time including the week number repeats every 6 seconds, so they had only 300 bits to play with. Given that a 10-bit week number already allows for nearly 20 years, a receiver only needs to know which decade it is today for that value not to be ambiguous, and that should be easy for a receiver with a few bits of local storage unless it's been powered off for more than a decade.
unless it gets a single spoofed or corrupted signal, then it's stuck in the future.
Of course you can implement countermeasures, like requiring months of observation to advance the flag, but that makes it much less easy (and harder to test).
> it's been powered off for more than a decade.
That's also a concern. You don't want a receiver to get hit by lightning then get swapped out with a spare on the shelf and suddenly get the wrong date. Now you need a provisioning step that requires remote access to a normally completely embedded component.
Again, not insurmountable-- but it's getting away from easy.
This seems like something that never should have existed.
I guess the ultimate cause is data framing in the transmission protocol. The timestamp contains the number of weeks since 00:00:00 1980-01-06, together with the number of seconds since the start of the current week. The number of seconds in a week won't fit into 16 bits, so my supposition is that the designers had to also use some of the bits that could otherwise have been used for a wider week counter.
I'd like to think that nowadays we'd use self-describing, upgradeable protocols. But GPS was designed in the 70s for the constrained technology of that era. And I'm pretty sure nobody anticipated how widely deployed it would be 45+ years later.
[1] https://en.m.wikipedia.org/wiki/GPS_week_number_rollover
That would open a whole new can of security worms though. Being able to modify a protocol in-band is something we're starting to move away from. Things are becoming more static as a precaution, like stored procedures on SQL so an attacker can't inject a change.
Or maybe they thought by now we would have replaced it with something better. Spacefaring was still riding high in the '70s, the brakes were only applied after '89 (sadly one of the many indirect effects of glasnost, RIP Gorbachev).
* GPS depends upon acurate timing and positions.
* GPS broadcasts a data packet.
The length of the packet is limited by the broadcast frequency and the time to update .. so the packet was kept tight.
The satellites drift (orbital decay, magnetic torque, solar wind, random particle pressure) and live in a different gravity (aka relativistic time) .. their 'self' data (position | time) is regularly updated from ground stations and their internal epochs only need to operate for some low N number of expected update cycles .. if they lack an accurate sense of 'self' they lack function as waypoints.
Adding clock data widths that count nanoseconds for the lifetime of the sun would be pointless: it makes the data packet longer and it doesn't change the hard requirement for regular time|position updates from the ground.
They were designed with short epoch counters with an understanding of functional constraints.
The engineers who designed and implemented the GPS system many decades ago were pioneering a fundamental technology, and worked within the constraints at the time to balance efficiency, cost, and complexity deliberately and as best they could.
Now we get to drive around in our luxurious cars, decked out with supercomputers listening to satellite communications, charting our course for us, all so we don't have to read our own maps.
Weeks worked nicely as a time unit due to the periodicity of the p-code signal.
If you have any doubts about the competency of the people who designed GPS, IMO it's worth checking out the Woodford/Nakamura report that considered a lot of alternative implementations. From an engineering perspective, they had a lot of tradeoffs to consider but typically took the direction that made the system more user friendly but harder to implement (like not requiring receivers to have atomic clocks or transmitters). It's also interesting to see how that sort of info was presented in the pre-powerpoint days
What matters is how rollover is dealt with.
For interest, we (mostly) all use GPS .. and the 'onboard' broadcast 10 bit GPS week counter has already rolled over twice:
* midnight (UTC) August 21 to 22, 1999
* midnight (UTC) April 6 to 7, 2019
The GPS "seconds since midnight last Sunday" timer resets to zero every week.
[1] https://en.wikipedia.org/wiki/GPS_week_number_rollover
[2] Satellite Geodesy Günter Seeber https://www.geokniga.org/bookfiles/geokniga-seeber-g-satelli...
(Not to mention the post corrections broadcast upwards from the ground to adjust for relativistic time drag between Earth surface nominal 1G and sat orbit height gravity)
Point being, many timing applications use lapsed time since X counters and are part of systems designed to handle rollover.
The mindset of just adding more bits to a counter or recording more decimal points isn't always appropriate or the 'best' fix.
That's just the General Relativity correction. There's also a Special Relativity correction to account for time dilation caused by the satellite's speed, which works in the opposite direction to the GR correction yet the two don't exactly cancel each other. Plus there are some others (Doppler shift, Sagnac effect, etc.)
https://www.aapt.org/doorway/TGRU/articles/Ashbyarticle.pdf
Are you surprised that people are still designing hardware where the time rolls over to 00:00 every 24 hours?
Correcting for it more elaborately than that is challenging, in particular if you depend on remembering the number of wraps a replacement part pulled off the shelf will return wrong dates when the original was fine, also if you remember then a false (or spoofed) signal can put the device in a bricked state.
Most GPS receivers (including most of the ones listed here) can be temporarily fixed by setting the date manually. But their interfaces may be inaccessible in the devices they're installed in and the fix will be lost after restart.
The L1C/L2C/L5 gps signals use a 13 bit week number. Hardware for the new signals was first deployed with the IIR-M satellites in 2005 with L2C support, which were delayed from something like 1997... but no broadcasting of L2C started until 2014. Right now very few receivers support the new signals.
If your hardware vendor did not include a “day” counter, you may need to implement in wetware.
The units could be defined by atomic vibrations or by the spinning of a large celestial object but these don’t agree. What is more the large celestial object (and others nearby) distorts space and time adding to the issue of where the time is (see my past submission on Barycentric Time)
Everyone knew of these issues and has known of these issues for ages. Why would they not address these well-known issues? The desire to sell new devices and/or support is most likely why.
(1) Technically the modernized L2C/L1C/L5 signals use a 13-bit week number, but because the deployment of these signals was massively delayed receivers using them are basically non-existent today. (I also have no idea if the current receivers supporting them successfully use the 13-bit week number, given that they're not guaranteed to be able to receive the modernized signals as they're still less available than L1)
I'm not aware of any devices that implement a rollover counter but there may be some, though a rollover counter needs to figure out how to deal with corrupted or spoofed signals and will have issues like spares off the shelf not working like the device they're replacing.
1024 weeks before Sept 17 is February 1, 2003. What is significant about that date?
References welcome.
That given the 10-bit limitation, some devices chose their own epoch date to get as much use of the 10-bits as possible. Meaning they chose some arbitrary date earlier than the device manufacture date.
Granted, it's a pretty accurate time system in the sense that it needs to know the time to an accuracy determined by the speed of light crossing the globe: that's how it knows where you are, that's its job.
Day, weeks, months, years... feh! In 7 minutes the Earth transits its own [edit] diameter; in 4 hours, the distance to the moon; in a year it's (roughly) back where it started.
Great hack, but maybe you ought to know what year it is if it matters to you. No way to figure that out? No clock available which is accurate enough?
At the massive scale of GPS, all observable behavior of the system are features that someone is relying on.
https://sync.empowerednetworks.com/keep-your-ntp-server-curr...
Another link stating "But other GPS receiver manufacturers set a delayed date for the rollover date to occur on – September 18th, 2022". "Other" in this case = Symmetricom / Microsemi.
https://www.orolia.com/will-your-network-time-servers-be-aff...