Argh! I wish the NTP servers were usable from outside of AWS.
Outside of North America and Europe, there aren't as many public NTP servers as there should be. For example, according to http://www.pool.ntp.org/zone/south-america, South America has less than 50 NTP servers.
It would be awesome if the large, widely-distributed providers (not just AWS with CloudFront, but also Akami, Cloudflare, etc.) provided public, Stratum-1 NTP servers at their POPs.
Interesting point, we're looking at implementing this new ntp server. Wanted to know a bit more about the risks of mixing leap second algos. Is there any more info / links you can give on the subject?
Anyone can provide an NTP server, but that's much different from having an atomic clock powered NTP server.
The servers your datacenter provide are probably clients of other NTP servers, so the quality and availability of NTP servers near your datacenter is still really important.
I would assume a datacentre would put a GPS somewhere to provide time to an NTP server. It's quite cheap, easy to do, and a useful value-add for their customers; and it's something that can only can be done by someone with physical (and likely roof) access.
And I would add for the crowd that one can easily check what sort of NTP server is on offer. NTP tracks the "stratum" of a server, which is how far the server is from an authoritative time source:
$ ntpq
ntpq> peer
remote refid st t when poll reach delay offset jitter
==============================================================================
*y.ns.gin.ntt.ne 249.224.99.213 2 u 815 1024 377 1.230 -4.148 1.825
+hadb2.smatwebde 209.51.161.238 2 u 809 1024 377 49.672 -2.073 3.869
+six0.ntp3.mattn 128.227.205.3 2 u 409 1024 377 63.721 -4.328 2.669
-hadb1.smatwebde 45.79.187.10 3 u 808 1024 377 49.767 -6.600 2.927
These are 4 servers out of the NTP pool; 3 are stratum 2 and one is stratum 3. The * means it has selected the first, the + means it likes the next 2, and the - means it has discarded the last one.
Having an atomic clock behind your NTP server doesn't really gain you anything unless GPS is lost for an extended period of time. As long as a timeserver has a GPS lock, it's going to be disciplining the atomic clock the same way any other clock is disciplined, so you don't really gain anything. Even a cheapie timekeeping GPS chip can get down into the < 500ns accuracy range (and the Thunderbolt on my desk will get down to < 50ns).
(You can gain a little accuracy if your timeserver itself is running its main clock (the one that drives the CPU and all the busses) off of a precision frequency reference, be it GPSDO or atomic clock, but that's a separate discussion)
Oh, and if you're using NTP and not directly connected to a precision timesource (not over the network), you may as well give up on that level of precision anyhow. For that you need PTP.
The PPS is just something you get when the GPS chipset has a lock, though (and sometimes when it doesn't, depending on how you have it configured). Most non-timekeeping GPS chipsets give you a PPS output for free, and 100% of timekeeping GPS chipsets give you one.
Pretty sure about 50ns. Datasheet for the Trimble Thunderbolt E: http://trl.trimble.com/docushare/dsweb/Get/Document-383329/0... ...second page, "PPS accuracy: 15ns (one sigma)" (I picked a larger number than the spec sheet simply so that hopefully nobody would go "oh, that's the ONE SIGMA value, lets argue".)
My Thunderbolt, in a completely temperature-uncontrolled environment with ~20 degree daily swings, generally has a PPS accuracy of ±45ns, in what is pretty crap conditions (first rule of precision timekeeping is "temperature stability")
I should have been more specific. You need GPS, PPS and you need to correctly configure them. Its a bad idea if people think you can just plug a USB into a GPS and be good to go.
I have been hoping for this for a long time, with Google’s TrueTime system being so unique. Quite a lot more work would have to go into guaranteeing the results all the way into your application, but this is still a great start.
If you take it at face value, the TrueTime API is implementable on top of any clock synchronization protocol. It's just coincidentally true that if you're close to a stratum 0-equivalent clock, your skew will be low. In terms of implementation, TrueTime daemons essentially also use Marzullo's algorithm to estimate timestamps, just like NTP.
Amazon TimeSync is probably a lot more than NTP as well. It must be part of XEN/hw as it is not yet available on the Nitro based instances. That it provide's an NTP endpoint is just the top of the cake...
Details worth highlighting.. They recommend chrony over ntpd, which makes particular sense given that chrony is designed to work well on virtual machine. They are doing leap second smearing like Google does internally. They don't say anything about what sort of accuracy you can expect but if the clock source is truly local it could be very good.
It's kinda neat they're using a link local address.
Theoretically, yes, but it's never happened since leap seconds were established in 1972, and it seems very unlikely to happen for quite some time. Chances are that a lot of software doesn't actually support negative leap seconds at all.
Would be interesting to use smearing for daylights saving time too. Make the hour twice as long when changing to winter and only take half the time when changing to summer.
In some scenarios handling DST might not be necessary then!
It wouldn't work for a tv schedule though, since what is broadcasted still uses actual time. ;)
I’m amidst digging into getting some gps clock hardware to make sure an open source db we're developing at work can work in some interesting situations. (Hit me up if you wanna join the team ;) )
High accuracy time is a fascinating domain. It’s this weird mix of general relativity, hardware, and anthropology. Cesium atoms and satellites vs calendars vs astronomy: who wins? :)
Point being: more organizations making high accuracy time available to more folks is a good thing!
Yes! The system we're working on is one that benefits from the exact same sort of idea : having time info of better accuracy should allow better performance of the same system. (Logical time works too, but will often require much more coordination to resolve ordering or conflicts )
What trouble are you having with the GPSs? Getting them hooked up to ntpd is fairly easy after you get used to some of the common hiccups.
As far as who wins for a long time now it has been astronomers. The fact that Google and Amazon are doing leap smearing is a nice sign that the tide may be turning. It is very rare that you see organizations supporting leap seconds during big standards discussions that are not astronomers.
Getting a rig for some servers is easy. Figuring out a range of hardware to test and support for a range of different types of possible users with different budgets, from some tiny hobby user at home to the entire global economy... have very different needs and hardware budgets. :)
The entire global economy? Maybe you should narrow your market focus a little. NTPD and or GPSD support anything with NMEA and PPS. You should be able to satisfy a large range of users with any ublox 6, 7 or 8T.
I see no reason that it would ever go backward, so yes. Especially with leap second smearing, meaning it's not only monotonically increasing, but also never stops (never a slope of zero).
Double leap-seconds are allowed, though I can't find any evidence one has ever happened. In such an occurrence, POSIX time would actually go backwards afterwards.
System clock synchronization is a really hard problem but especially important for things like live video stream encoding where the frame timestamps need to correspond to the real wall-clock (within milliseconds) for things like sub-second play-by-play synchronization.
It varies by sport and how accurate you need your timers to be. NBA needs a timer within "seconds" -- Formula 1 and NASCAR need timers within milliseconds.
I'd be curious to know how this compares to Google's (internal ) TrueTime service[1], and if it allows for development of globally consistent but highly available storage systems like Spanner.
Interesting to know if they have an SLA for this and how it's formulated.
Time is important and hard. I doubt that customers who really need precise time would walk away from their ghetto solutions with no SLA because when something like this craps out, it's almost always a circus.
But hey, when time precision and accuracy is a bonus not a requirement for your AWS instance then 'free local NTP server!'
> uses a fleet of redundant satellite-connected and atomic clocks in each region
I am really curious to know what atomic frequency standards they are using. There is something a little suspicious about the wording. I have a feeling they mean "we use GPSDOs" and there are no atomic frequency standards physically located in each regional datacenter.
I'm also a little surprised to see that the remote clock is stratum 4. It would be interesting to see how they are distributing the time between their clocks.
If someone from amazon reads this it would be neat if you posted some more details about your setup.
The amount of dog fooding that happens at AWS is impressive - looks like they needed and built this for DynamoDB global tables (assumes clock sync across all regions) and made it available to all instances.
65 comments
[ 5.8 ms ] story [ 149 ms ] threadOutside of North America and Europe, there aren't as many public NTP servers as there should be. For example, according to http://www.pool.ntp.org/zone/south-america, South America has less than 50 NTP servers.
It would be awesome if the large, widely-distributed providers (not just AWS with CloudFront, but also Akami, Cloudflare, etc.) provided public, Stratum-1 NTP servers at their POPs.
You generally do not want to mix "smearing" and "non-smearing" time servers.
Apple's one is also available at time.asia.apple.com and time.euro.apple.com
The servers your datacenter provide are probably clients of other NTP servers, so the quality and availability of NTP servers near your datacenter is still really important.
https://en.wikipedia.org/wiki/Network_Time_Protocol#Clock_st...
If you're using ntpd, you can check it like this:
These are 4 servers out of the NTP pool; 3 are stratum 2 and one is stratum 3. The * means it has selected the first, the + means it likes the next 2, and the - means it has discarded the last one.Source: recently asked them this.
(You can gain a little accuracy if your timeserver itself is running its main clock (the one that drives the CPU and all the busses) off of a precision frequency reference, be it GPSDO or atomic clock, but that's a separate discussion)
Oh, and if you're using NTP and not directly connected to a precision timesource (not over the network), you may as well give up on that level of precision anyhow. For that you need PTP.
Not true, you need a GPS lock as well as PPS.
Are you sure about 50ns?
Pretty sure about 50ns. Datasheet for the Trimble Thunderbolt E: http://trl.trimble.com/docushare/dsweb/Get/Document-383329/0... ...second page, "PPS accuracy: 15ns (one sigma)" (I picked a larger number than the spec sheet simply so that hopefully nobody would go "oh, that's the ONE SIGMA value, lets argue".)
My Thunderbolt, in a completely temperature-uncontrolled environment with ~20 degree daily swings, generally has a PPS accuracy of ±45ns, in what is pretty crap conditions (first rule of precision timekeeping is "temperature stability")
If you take it at face value, the TrueTime API is implementable on top of any clock synchronization protocol. It's just coincidentally true that if you're close to a stratum 0-equivalent clock, your skew will be low. In terms of implementation, TrueTime daemons essentially also use Marzullo's algorithm to estimate timestamps, just like NTP.
What am I missing?
It's kinda neat they're using a link local address.
Yes.
They make a second last slightly long for some time period.
https://cloudplatform.googleblog.com/2016/11/making-every-le...
https://cloudplatform.googleblog.com/2015/05/Got-a-second-A-...
https://cloud.google.com/spanner/docs/true-time-external-con...
In some scenarios handling DST might not be necessary then!
It wouldn't work for a tv schedule though, since what is broadcasted still uses actual time. ;)
High accuracy time is a fascinating domain. It’s this weird mix of general relativity, hardware, and anthropology. Cesium atoms and satellites vs calendars vs astronomy: who wins? :)
Point being: more organizations making high accuracy time available to more folks is a good thing!
As far as who wins for a long time now it has been astronomers. The fact that Google and Amazon are doing leap smearing is a nice sign that the tide may be turning. It is very rare that you see organizations supporting leap seconds during big standards discussions that are not astronomers.
Software can be fiddly but I found this guide to be helpful: http://www.satsignal.eu/ntp/Raspberry-Pi-NTP.html
Hell, you can hook it up to a NodeMCU as well: https://blog.squix.org/2016/05/esp8266-peripherals-ublox-gps...
Bam, $10 wifi enabled GPS.
100ms? 10ms? 1ms? 100us?
Finra (Financial Industry Regulatory Authority) requires <50ms
MiFID II (EU centric) HFT requirement can be as low as 100 microseconds accuracy, or 0.1ms.
[1] https://research.google.com/pubs/pub45855.html
Time is important and hard. I doubt that customers who really need precise time would walk away from their ghetto solutions with no SLA because when something like this craps out, it's almost always a circus.
But hey, when time precision and accuracy is a bonus not a requirement for your AWS instance then 'free local NTP server!'
I am really curious to know what atomic frequency standards they are using. There is something a little suspicious about the wording. I have a feeling they mean "we use GPSDOs" and there are no atomic frequency standards physically located in each regional datacenter.
I'm also a little surprised to see that the remote clock is stratum 4. It would be interesting to see how they are distributing the time between their clocks.
If someone from amazon reads this it would be neat if you posted some more details about your setup.
Something like eg Google Spanner relies on atomic clocks - it's a combination of hardware and software that enable the use of wallclock time.