Using atomic clocks too, nice. From an older post:
> It uses a fleet of redundant satellite-connected and atomic clocks in each Region to deliver time derived from these highly accurate reference clocks.
I'm pretty sure if you use the GCP dataflow stuff, the dataflow worker VM's will be given access to an RPC server which has the necessary endpoints for TrueTime.
Obviously it's all a matter of reverse engineering rather than documented API's.
One of the key aspects of the TrueTime system is any device with serious clock error is simply murdered. It seems like offering that would significantly benefit users of this AWS API.
I'm curious about the title on this submission. I thought TrueTime had unusually strong guarantees about accuracy that don't seem to be called out in what I'm reading on the linked article.
Oh, I wasn't criticizing, just genuinely curious. If the AWS service has a similar design, that's interesting to me. AWS doesn't seem to say much about the details.
If you want to say what you think is important about an article, that's great, but please do it by adding a comment to the thread. Then your view will be on a level playing field with everyone else's: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&so...
"Please use the original title, unless it is misleading or linkbait; don't editorialize."
Not quite: Truetime has guarantees about ordering, rather than accuracy.
If you ask for the time and get A, and then ask for the time again and get B, then Truetime guarantees that A is less than B.
Obviously, in a distributed setting this is much easier to do if you have accurately sync'ed clocks, but that accuracy goes to reducing the uncertainty in the time (and hence making truetime faster) rather than providing accuracy.
Remember... your CPU can halt at any time for any number of milliseconds. That means simple things like:
upperBound, lowerBound = readTime()
if (upperBound<deadline)
do_stuff(x, y, z)
Are incorrect... There is no guarantee that the 'if' statement didn't take many milliseconds, and that the stuff didn't end up happening after the deadline.
It's also very easy to write code that works, but is theoretically wrong. You will leave a hidden bug that may only rear its head years down the line.
Yeah, I was thinking about this too. Without a real-time OS on separate hardware, and/or some kind of guaranteed "will not context-switch out of your process for more than X time" + strict coding habits, this seems like at best a practical (>?)99% improvement and at worst snake-oil. Certainly not reliable in any case.
edit: well, after reading Spivak's comment[1] a bit, I guess it does provide strict upper bounds. May be useful to reduce how often you need to use fallback behaviors / get more byzantine. Though I'm not yet sure how to turn that into something useful. No doubt there are some though.
I mean it’s not that hard even on non-rt preemptive schedulers. You can’t avoid false negatives because you could be preempted after func completes but before the time is fetched but if you get a result it will be valid.
def must_complete_before(func, deadline):
result = func()
lower, upper = time.now()
if upper < deadline:
return result
else:
throw MissedDeadlineException(“Can’t guarantee func completed before deadline”)
To be fair your example would be unsafe with and kind of time API.
The APIs provided by TrueTime and Time Sync are useful to compare two events, each with their own uncertainty intervals. Then you can be sure if any event "happened before" the other, or if they're concurrent.
Perhaps the biggest "fake-out" in 21st century computing: Google publicly released its MapReduce paper -- directing most of the rest of the industry toward loosely coupled, overly complex distributed data processing systems like Hadoop for the following decade -- but internally they just bought a bunch of atomic clocks and built a distributed RDBMS.
I know this is a somewhat simplified story, but it does make me chuckle.
Yes, which is why it's amusing in hindsight that for a decade everyone* outside Google was forcing all* their distributed data tasks into the MapReduce paradigm, without considering alternative approaches like the one used by Spanner.
I'm not sure how you think a distributed data processing technology would "fake-out" other companies when building/choosing database technology. They are totally different problem sets.
MapReduce does not have a set in stone data source/sink and can use multiple things like bigtable and spanner so they are complementary technologies.
I think the parent commenter might be referring to systems like Hive or HBase built on top of Hadoop and do have a lot of overlap with a large scale database system.
Exactly, thanks. "MapReduce paradigm" wasn't precise, and I can see why that makes everyone want to give a distributed systems 101 lecture in this comment thread :)
Yes it did. Google published a paper about it in 2012, and claimed that at that point it had been in development for 5 years and in production for more than 1.
Mmm, I worked at google in 2012 and F1 was in active development.
I’ve never heard of Spanner internally. Maybe it was in development, but it was not in use.
Edit: went and read more. Looks like Spanner existed but didn’t have sql, so it wasn’t what it is today. And looks like I don’t remember things any more.
at least by 2013 it had some basic SQL (although much more limited subset that one usable today); if you needed more you would be using f1; IIRC I used spanner (without F1) in production around 2013
The tech lead of the Google MapReduce team (which no longer exists) just received their award for turning down mapreduce. IIRC it was officially done 5 years ago. However I believe the code to delete MR was never checked in and I'm not sure if there are still users.
MapReduce was used at Google for highly inappropriate things. For example, the machine learning system I worked on, Sibyl https://www.datanami.com/2014/07/17/inside-sibyl-googles-mas... was implemented using mapreduce but there was no real technical justification for that- it's just that there was no other system that could scale to the volumes required or handle the constant failures endemic to GOogle's internal systems. It ended up requiring all sorts of heroic work to make MR scale, for example map-side combiners (which "reduced" items with common keys in the map output before it gets flushed to the shuffle files). All of this got replaced with TensorFlow and only the good bits of Sibyl were extracted to TFX.
i.e this is a crappy tool but it's best tool we have come up with for this problem thus far.
There are lots of such tools which are used begrudgingly by people that have an intuition for the fact it can be done better but not the concrete idea and/or time to implement it.
it wasn't a crappy tool (mapreduce was amazing) but it definitely was an impedence mismatch for this particular job. Later, we tried to get SIbyl to move to the underlying compute engine that Flume was built on top of it but it turned out to be more profitable to just let it die slowly.
MapReduce was deprecated because flume [0] the successor is better but it does practically the same thing and flume is used massively. I believe dataflow is the public google cloud version.
It's rare to see a comment on HN that misunderstands basic distributed systems concepts. MapReduce the paper has nothing to do with a database. You're likely conflating the fact that to achieve fault tolerant distributed computation, hadoop and hadoop like systems use a database like filesystem.
However, no one is looking at Map Reduce type jobs as a replacement for a database and vice versa. That's like saying "wow linkedin made kafka why do we need a webserver too". Those two technologies are only related in the loosest sense.
> You're likely conflating the fact that to achieve fault tolerant distributed computation, hadoop and hadoop like systems use a database like filesystem.
Yes, I specifically mentioned non-Google users adopting Hadoop, since it encompassed both a MapReduce implementation and supporting infrastructure.
Once on the bandwagon inspired by the MapReduce paper, many orgs didn't just use MapReduce itself for parallelized batch analytic purposes, but also HBase and Hive and other stuff with actual longer term state atop HDFS, YARN, etc.
> However, no one is looking at Map Reduce type jobs as a replacement for a database and vice versa.
The marketing and sales teams of HortonWorks, Cloudera, etc certainly sold Hadoop platforms, related Apache projects, and "MapReduce" (as a broad brand name for all this, not the specific technical concept) as replacements for databases, broadly speaking. It's that culture that was a bit shocked when Spanner was unveiled.
The reason spanner is a big deal is because it provides relational semantics including cross table transactions. That is why spanner is interesting.
No one, not even the most ambitious and clueless market folks at Cloudera, has ever sold Hadoop and MR type technologies as a replacement for RDBMs. Spanner can be used as a replacement. These two technologies don't solve the same problem.
You still need MR to do analytics at scale on spanner!
Ah, no, the misunderstanding was with something else. Apparently "Aesop ending" is a keyword/crossword definition. It means "the moral of the story" :-)
The combination of a time and guaranteed error bound can be used as a primitive in distributed systems. You can get timestamps from requests coming from different systems and relate them in a meaningful way. At least, in a false negative fashion. E.g., if A occurs at T + 100ms with 1ms uncertainty and B occurs at T + 105ms with 2ms uncertainty you can conclude that A happened before B. This does not allow you to always establish an order (e.g., if there is overlap) but this isn't always necessary. For instance if instead you are B and you want to have something occur after some time A with uncertainty A_uncertainty, then you can sleep until B with uncertainty B_uncertainty such that B - B_uncertainty > A + A_uncertainty.
None of this has to do with Amazon though. I am very uncomfortable with Amazon (or the others) pretending that they invented precision timekeeping and that it is a special feature only they have because it seems like a lot of people believe it and build systems that rely on an API that is proprietary to a cloud vendor rather than just installing ntp.
Two observations/questions, the first probably naive:
* It's not that hard to get your own "world class" time server, for under a thousand. A Rb standard slaved to a GPSDO is gonna be so accurate and stable, and use that to drive a SBC that supports IEEE1588, where you run your NTP and PTP server. Oh, but I guess that box, while inexpensive, isn't in Amazons DC, so doesn't help you.
* PTP's absence in the Amazon Time Sync Service article is quite conspicuous!
To be clear, what I meant was "the fact that they're advertising whiz bang time synchronization, but then not using the correct protocol, makes me question the quality of the whole enterprise".
Even on local networks, NTP can only get you so close. If you set up chrony just so, in ideal conditions, I've gotten hundreds of microseconds (more commonly ~500-1000us). But combined with PTP, you can get sub-microsecond accuracy.
That sounds different. Embrace and extend means implementing a protocol and then adding specific non-spec features. For example, postgres 'embraces' SQL and 'extends' it with specific features - in the future postgres could 'extinguish' SQL by breaking from the spec, forcing an ecosystem split.
You don’t even need a tcxo. If you don’t care about holdover, connecting the pps pin from a $5 GPS module to an $15 SBC will give you a time source that is significantly better than anything you can serve over NTP to your clients. NTP is always gonna be the weak link for a service like this.
"Amazon provides the Amazon Time Sync Service, which is accessible from all EC2 instances, and is also used by other AWS services. This service uses a fleet of satellite-connected and atomic reference clocks in each AWS Region to deliver accurate current time readings of the Coordinated Universal Time (UTC) global standard through Network Time Protocol (NTP). The Amazon Time Sync Service automatically smooths any leap seconds that are added to UTC.
"The Amazon Time Sync Service is available through NTP"
As I understand it, “amazon time sync service” is just a bunch of NTP servers run by Amazon. Their ClockBound library calculates error bounds the same way you would if you were using a different NTP server: |Local Offset| + Root Dispersion + (Root Delay / 2)
>how exactly would GPS work at scale for 10,000s of servers
I’m not sure what you meant by “clocks on the system board” but if you’re talking about the system clock on the NTP server or the NTP client, it really doesn’t matter. Any old crystal on the NTP server is accurate enough to stay locked to a PPS input. The clients are limited by NTP so their clocks don’t really matter either.
With that underneath you need just one gps clock reciever for the whole LAN, and PTP capable Ethernet adapter and switches throughout, of course. Which most are, these days.
Do you have experience with this? Playing around with PTP has been on my TODO list for quite a while, but it seems to be quite a hassle.
At home, I already have a GPS module with its PPS pin connected to an Pi running an NTP server. In my data center, I use clock.sjc.he.net and that's about as good as NTP gets. Certainly good enough for one-way latency measurements.
Are there any SBCs with hardware timestamping and a PPS input? I've heard mixed reviews of the BeagleBone Black as a PTP server. Then again, even if I get that working I have no way to run PTP in the data center since there's no GPS reception in the cabinets.
It _is_ a hassle, and you should have a need for that level of synchronicity. We did it in our team for an in-vehicle network, with the vehicle GPS reciever as the network PTP time master and we figured how to access the ethernet PTP register both from userspace and from the kernel. Kernel for the system clock, user space to avoid latencies from kernel Userland interaction in some high precision real time task. It was a major effort and it required low level hackery way above my personal head.
Back from my time at Amazon, one of my favourite technical videos on the internal educational/YouTube thing was from a service team explaining how hard accurate clock sync in distributed systems was. One of those problems
I’ve just taken for granted over the years. But just layers and layers of complexity where naive assumptions at any point get you the wrong result, but it’s not at all obvious you have the wrong result.
I really wish they made more of that stuff publicly available.
93 comments
[ 2.9 ms ] story [ 63.9 ms ] thread> It uses a fleet of redundant satellite-connected and atomic clocks in each Region to deliver time derived from these highly accurate reference clocks.
https://aws.amazon.com/blogs/mt/manage-amazon-ec2-instance-c...
I can't find any instance of the word "bound" in the GCP or Google NTP docs.
[1]: https://developers.google.com/time/guides#google_compute_eng...
[2]: https://cloud.google.com/compute/docs/instances/managing-ins...
Obviously it's all a matter of reverse engineering rather than documented API's.
"Please use the original title, unless it is misleading or linkbait; don't editorialize."
https://news.ycombinator.com/newsguidelines.html
If you ask for the time and get A, and then ask for the time again and get B, then Truetime guarantees that A is less than B.
Obviously, in a distributed setting this is much easier to do if you have accurately sync'ed clocks, but that accuracy goes to reducing the uncertainty in the time (and hence making truetime faster) rather than providing accuracy.
Remember... your CPU can halt at any time for any number of milliseconds. That means simple things like:
Are incorrect... There is no guarantee that the 'if' statement didn't take many milliseconds, and that the stuff didn't end up happening after the deadline.It's also very easy to write code that works, but is theoretically wrong. You will leave a hidden bug that may only rear its head years down the line.
edit: well, after reading Spivak's comment[1] a bit, I guess it does provide strict upper bounds. May be useful to reduce how often you need to use fallback behaviors / get more byzantine. Though I'm not yet sure how to turn that into something useful. No doubt there are some though.
[1] https://news.ycombinator.com/item?id=29103093
This API is to determine if something that happened at a given instant is definitely in the past. That’s it!
The APIs provided by TrueTime and Time Sync are useful to compare two events, each with their own uncertainty intervals. Then you can be sure if any event "happened before" the other, or if they're concurrent.
I know this is a somewhat simplified story, but it does make me chuckle.
True time helps with things like spanner transactions. It's just a totally different use case.
* slight exaggerations, I know
MapReduce does not have a set in stone data source/sink and can use multiple things like bigtable and spanner so they are complementary technologies.
https://cloud.google.com/files/storage_architecture_and_chal...
I’ve never heard of Spanner internally. Maybe it was in development, but it was not in use.
Edit: went and read more. Looks like Spanner existed but didn’t have sql, so it wasn’t what it is today. And looks like I don’t remember things any more.
MapReduce was used at Google for highly inappropriate things. For example, the machine learning system I worked on, Sibyl https://www.datanami.com/2014/07/17/inside-sibyl-googles-mas... was implemented using mapreduce but there was no real technical justification for that- it's just that there was no other system that could scale to the volumes required or handle the constant failures endemic to GOogle's internal systems. It ended up requiring all sorts of heroic work to make MR scale, for example map-side combiners (which "reduced" items with common keys in the map output before it gets flushed to the shuffle files). All of this got replaced with TensorFlow and only the good bits of Sibyl were extracted to TFX.
There are lots of such tools which are used begrudgingly by people that have an intuition for the fact it can be done better but not the concrete idea and/or time to implement it.
This sounds bad.
[0]: https://research.google/pubs/pub35650/
In terms of popular nosql vs google sql products, it's more Hadoop : BigQuery :: Mongo? : Spanner
You're pretty explicitly not supposed to run OLAP queries on spanner.
However, no one is looking at Map Reduce type jobs as a replacement for a database and vice versa. That's like saying "wow linkedin made kafka why do we need a webserver too". Those two technologies are only related in the loosest sense.
Yes, I specifically mentioned non-Google users adopting Hadoop, since it encompassed both a MapReduce implementation and supporting infrastructure.
Once on the bandwagon inspired by the MapReduce paper, many orgs didn't just use MapReduce itself for parallelized batch analytic purposes, but also HBase and Hive and other stuff with actual longer term state atop HDFS, YARN, etc.
> However, no one is looking at Map Reduce type jobs as a replacement for a database and vice versa.
The marketing and sales teams of HortonWorks, Cloudera, etc certainly sold Hadoop platforms, related Apache projects, and "MapReduce" (as a broad brand name for all this, not the specific technical concept) as replacements for databases, broadly speaking. It's that culture that was a bit shocked when Spanner was unveiled.
No one, not even the most ambitious and clueless market folks at Cloudera, has ever sold Hadoop and MR type technologies as a replacement for RDBMs. Spanner can be used as a replacement. These two technologies don't solve the same problem.
You still need MR to do analytics at scale on spanner!
MapReduce predates TrueTime by a decade or more. MR was critical to scaling internet systems at the time it was released.
However, Flume + Spanner was a much nicer system to work with than MR + GFS, I'll give you that.
Today they released a new OSS daemon and library for it.
[1] https://aws.amazon.com/about-aws/whats-new/2017/11/introduci...
By asking both super-microscopic stuff and stuff way out in space, you can now find out what time it is!
https://www.esa.int/Applications/Navigation/Galileo/Galileo_...
so, basically magic.
Also you can't tell a story just with the Aesop ending - you have to tell the fable and END with that line.
I missed that part :-)
* It's not that hard to get your own "world class" time server, for under a thousand. A Rb standard slaved to a GPSDO is gonna be so accurate and stable, and use that to drive a SBC that supports IEEE1588, where you run your NTP and PTP server. Oh, but I guess that box, while inexpensive, isn't in Amazons DC, so doesn't help you.
* PTP's absence in the Amazon Time Sync Service article is quite conspicuous!
Even on local networks, NTP can only get you so close. If you set up chrony just so, in ideal conditions, I've gotten hundreds of microseconds (more commonly ~500-1000us). But combined with PTP, you can get sub-microsecond accuracy.
https://docs.microsoft.com/en-us/azure/virtual-machines/linu...
Also, how exactly would GPS work at scale for 10,000s of servers within 1000s of racks within a DC?
"The Amazon Time Sync Service is available through NTP"
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time...
>how exactly would GPS work at scale for 10,000s of servers
GPS module (+ GPSDO + Rb clock optional) -> SBC running NTP server -> machines running NTP client
I’m not sure what you meant by “clocks on the system board” but if you’re talking about the system clock on the NTP server or the NTP client, it really doesn’t matter. Any old crystal on the NTP server is accurate enough to stay locked to a PPS input. The clients are limited by NTP so their clocks don’t really matter either.
https://en.m.wikipedia.org/wiki/Precision_Time_Protocol
https://en.m.wikipedia.org/wiki/Precision_Time_Protocol
With that underneath you need just one gps clock reciever for the whole LAN, and PTP capable Ethernet adapter and switches throughout, of course. Which most are, these days.
At home, I already have a GPS module with its PPS pin connected to an Pi running an NTP server. In my data center, I use clock.sjc.he.net and that's about as good as NTP gets. Certainly good enough for one-way latency measurements.
Are there any SBCs with hardware timestamping and a PPS input? I've heard mixed reviews of the BeagleBone Black as a PTP server. Then again, even if I get that working I have no way to run PTP in the data center since there's no GPS reception in the cabinets.
I really wish they made more of that stuff publicly available.