I can't comment on infrastructure in AWS, but I worked at an observatory for a hot minute when they used Precision Time Protocol (PTP). Their scheme was to record telemetry from all aspects of the observatory -- hardware state, environmental and atmospheric conditions, software and system state, commands, etc. The intent was to build a hyper-precise set of telemetry that could be used to retroactively reprocess old imaging from the observatory as more imaging and telemetry was generated and it would be easier to do that with more accurate timing.
With this level of resolution, multiple observatories can more effectively share and synthesize data; if you know the state of two observatories down to the tens or hundreds of nanoseconds you can do more advanced work with synthetic aperture imaging.
Beyond astronomy my other guess is that PTP or similar tech might be used for high frequency trading; it might also be useful for creating distributed systems; Google's Cloud Spanner uses TrueTime which IIRC is a proprietary equivalent of PTP.
PTP is nearly always proprietary. It's very configurable and has to be configured to work. So there are many sets of settings called "profiles", defined by various industries for their own needs. They're not completely interoperable, but all have in common a high precision.
So you could say PTP is a kit to build timesync protocols.
Since this comes from a physics context, it's most likely talking about the white rabbit implementation of ptp, which is open source and open hardware:
Garden variety NTP time / system time I wouldn’t trust as an oracle for distributed systems but this kind of high precision time is probably suitable for that use.
> In order to relieve your anxiety before you read further, no, correctness does not depend on timestamps, clock synchronization only helps to improve performance, as more accurate clocks result in more successful transactions and a serialization order that complies with real time.
High frequency trading (measuring latency, inferring event ordering, etc). Since most crypto derivatives trading takes place at ap-northeast-1, it feels like AWS is orienting this release towards financial markets customers.
…why? My napkin math says a pair of observers, one at sea level and one on the top of a 14,000ft mountain, will drift by about 15 microseconds per year.
In other words, the approximation that time is not relative breaks down at this level of accuracy - there’s literally no value in trying to synchronize things this closely.
My 60-cent 32678 Hz crystal at 12.5pf is specified at +/- 20ppm (aka: 0.002% accuracy). But with a caveat: I need to load the crystal with exactly 12.5pf capacitance.
So sure, I can go grab 2x 25pF capacitors and load the crystal to ground (series capacitors half the capacitance, so 2x 25pF in series == 12.5pf). Except... what?
Trace-capacitance and on-chip capacitance hasn't been factored in.
Hmmmmm... okay. Does anyone have a picoFarad accurate multimeter? Oh snap, those are expensive.
No, the easiest way to tune a crystal oscillator to its appropriate accuracy is through the use of a ground truth clock, and comparing that ground-truth clock vs the clock. So I make my guess with 2x 9pF capacitors (5pF internal capacitance on the chip x2 pins + 2pF estimated trace capacitance) and measure the result against the ground truth.
Well, to calibrate a 20ppm crystal oscillator means I need a time sync service that has 20ppm accuracy or better. That's ya know: +/- 20 microseconds per second accuracy.
-----------
That's my cheap 60-cent crystal by the way. If you go with a temperature-controlled oven crystal oscillator for better accuracy, you'll need more accuracy to tune that.
That's even more reason to sync to the microsecond level. Computer systems rarely need an an accurate representation of local time. Instead they benefit from being closely aligned with each other. So unavoidable drift at the microsecond level is a reason to sync to the microsecond level.
Most systems don’t care much about their clocks being physically super accurate, but they very much care their clocks being tightly in sync with the rest of the world. That’s what the sync service accomplishes.
If you have times that are synced to within a few microseconds, then any logs you have will have much less ambiguity to the sequence of events that happened in a system that crosses multiple computers. You also get a more unambiguous (but still imperfect) measure of "what event outside the system happened first."
Can someone explain how this works? My laptop synchronizes time by performing a network request to Microsoft's servers, but my laptop doesn't need microsecond accuracy.
In Amazon's case, a network request to synchronize time would take orders of magnitudes longer than microseconds. Is there specialized hardware involved? Or multiple requests to pin down the accuracy?
26 comments
[ 0.23 ms ] story [ 71.5 ms ] threadWith this level of resolution, multiple observatories can more effectively share and synthesize data; if you know the state of two observatories down to the tens or hundreds of nanoseconds you can do more advanced work with synthetic aperture imaging.
Beyond astronomy my other guess is that PTP or similar tech might be used for high frequency trading; it might also be useful for creating distributed systems; Google's Cloud Spanner uses TrueTime which IIRC is a proprietary equivalent of PTP.
https://white-rabbit.web.cern.ch/
Garden variety NTP time / system time I wouldn’t trust as an oracle for distributed systems but this kind of high precision time is probably suitable for that use.
Consensus example - check out Nezha (https://arxiv.org/abs/2206.03285), they came up with an interesting 'Deadline Ordered Multicast' (DOM) primitive
> DOM follows Liskov's suggestion of "depending on clock synchronization for performance but not for correctness"
Transactions example - look no further than DynamoDB, https://muratbuffalo.blogspot.com/2023/08/distributed-transa...
> In order to relieve your anxiety before you read further, no, correctness does not depend on timestamps, clock synchronization only helps to improve performance, as more accurate clocks result in more successful transactions and a serialization order that complies with real time.
In other words, the approximation that time is not relative breaks down at this level of accuracy - there’s literally no value in trying to synchronize things this closely.
So sure, I can go grab 2x 25pF capacitors and load the crystal to ground (series capacitors half the capacitance, so 2x 25pF in series == 12.5pf). Except... what?
Trace-capacitance and on-chip capacitance hasn't been factored in.
Hmmmmm... okay. Does anyone have a picoFarad accurate multimeter? Oh snap, those are expensive.
No, the easiest way to tune a crystal oscillator to its appropriate accuracy is through the use of a ground truth clock, and comparing that ground-truth clock vs the clock. So I make my guess with 2x 9pF capacitors (5pF internal capacitance on the chip x2 pins + 2pF estimated trace capacitance) and measure the result against the ground truth.
Well, to calibrate a 20ppm crystal oscillator means I need a time sync service that has 20ppm accuracy or better. That's ya know: +/- 20 microseconds per second accuracy.
-----------
That's my cheap 60-cent crystal by the way. If you go with a temperature-controlled oven crystal oscillator for better accuracy, you'll need more accuracy to tune that.
In Amazon's case, a network request to synchronize time would take orders of magnitudes longer than microseconds. Is there specialized hardware involved? Or multiple requests to pin down the accuracy?
> ... customers can now access local, GPS-disciplined reference clocks on supported EC2 Instances.
Specialized hardware.