The article is split on 5 pages and due to the ads not easy to navigate through. The current situation of Harlan Stenn (59), who maintains NTP alone is on the fourth page:
"With the Linux Foundation's $7,000 in monthly cash flow, Stenn finances his movement between his home lab, in Talent, Ore., and the NTP servers located in San Jose, Calif. In Oregon, Stenn lives with his wife and does most of his patch inspection, code writing, and release building three weeks a month. The fourth week, he stays in San Jose, close to two co-location data center providers that host NTP computers. He rents a room there to work on server and network administration, maintain the email list, and check on server backups.
Much of the travel, room, replacement hardware such as disk drives, or needed commercial software such as the Coverity system for security checking on code, must come out of the $7,000 monthly stipend or be charged to his consulting business.
Most of his 17-20 servers came out of a one-time, $10,000 grant in 2010 from the Internet Society, a policy and technology infrastructure advisory body for the Internet founded in 1991. Those servers are running at ISC.org in Redwood City, Calif., which hosts BIND and several other open source pieces of Internet infrastructure. For 15 years, it has provided space, electricity and some management "smart hands" to host NTP operations, without charging, said Stenn. "They would love for us to pay them," he said, and he once totaled the monthly bill at $1,400. But ISC.org also knows the NTP project can't pay and continues to host it, Stenn added."
And on the first page:
"Its ongoing development and maintenance now rest mostly on the shoulders of Stenn, and that's why NTP faces a turning point. Stenn, who also works sporadically on his own consulting business, has given himself a deadline: Garner more financial support by April, "or look for regular work.”"
he doesn't have a clear goal set like GnuPG had. GnuPG doesn't state the goal because it reached it. But until then you were able to actually see how your support moves the bar. Maybe setting the goal would help here?
When I knock on someone's door, and then hand them a hot box of pizza, they smile and often hand me money: $5's, $10's, sometimes a $100. I'm just doing my job, just like this Harlan "Father Time" Stenn. I suppose we take for granted, that which we can't see.
He spends one week per month staying in San Jose ( where the servers are hosted for free ) to be... closer to the servers?
It's hard to sympathize when there are such frivolous expenses. There's what, a thousand bucks at least on travel and accommodation per month.
Going on-site is understandable when updates need to be performed that are too risky to do remotely. On-site on a fixed schedule, however, would quickly be over-ruled in any commercial operation.
"With the Linux Foundation's $7,000 in monthly cash flow, Stenn finances his movement between his home lab, in Talent, Ore., and the NTP servers located in San Jose, Calif. In Oregon, Stenn lives with his wife and does most of his patch inspection, code writing, and release building three weeks a month. The fourth week, he stays in San Jose, close to two co-location data center providers that host NTP computers. He rents a room there to work on server and network administration, maintain the email list, and check on server backups."
You couldn't get most people capable of doing the work to take on that responsibility for a flat out salary twice that amount, so how he spends the money is frankly relatively uninteresting - consider it a salary, and that even if you see it as a salary and ignore his costs it's way below market.
There does not exactly appear to be a queue of people wanting to take over the reigns.
Yes, it's far from trivial to even learn what's already implemented in the software. PHK got money to fix the bugs in it and instead he started to spend the money in "writing from the scratch" which certainly throws away a lot of what's already solved, tested and debugged through the decades.
I wouldn't call $84k/year "well below" market. Especially considering that the project isn't under heavy development. Also, I have to echo what other comments have said... he seems to be spending it oddly. He's choosing to live somewhere else, even though he seems to think monthly physical proximity to the servers is important. I currently live near work... it would be like me choosing to move 1000 miles away and then complaining that my salary won't cover my travel expenses.
There are at least 4 simple fixes I can think of:
1) Move him and his family closer to the data center (Perhaps difficult)
2) Move the servers to a data center that's closer to his home (Not as difficult)
3) He could start using VMs like the rest of the world... physical access really isn't needed except for physical security (making sure the rack is locked and access is controlled), which data center employees can manage (Fairly easy)
4) Fire him and find someone else who would like a $7k/month stipend to do bugfixes (Fairly easy)
You can't always treat humans as resources. Some fields are so complicated that you need superheroes, or else you need to build an organization to do the work. NTP is that complicated. What it's doing is relatively simple, but the surrounding infrastructure you need is ridiculous, and the security concerns enormous. If there's a vulnerability it affects the entire world. So you can't just find someone to replace him.
If you go with a superhero, then you have to deal with him. You can't just say, "Sorry bud, we need to uproot you and move you in-town." Even if he doesn't say no, which he very well could, leaving you and your plans fucked and the world unprotected, driving all the enjoyment out of the job will make him eventually say "fuck it, I'm retiring."
Okay, let's go with an organization. Who is going to build it? You still need the superhero. The critical nature of the infrastructure demands you use the guy whose been covering your ass all this time, because most anybody else you're going to find is going to fuck it up, and there can be no possibility of a fuck-up.
And you can't buy a superhero for any amount of money. There just aren't that many of them.
Some fields are so complicated that you need superheroes
If there's a vulnerability it affects the entire world
Who is going to build it? You still need the superhero.
Hyperbole much?
If it really was so dire then that would be all the more reason to not rely on a single "superhero" who may get run over by a bus tomorrow.
Gladly NTP is not the mythical voodoo rocket science that you make it out to be.
Most large corps run their own NTP servers, some of them public, e.g.:
The official NTPd impl doesn't even have a very good reputation (cf. the recent debate about that security vulnerability).
As I see it someone like Google should indeed just hire the guy and take pool.ntp.org under their wing. Throwing even more money at a single guy doesn't convince me as a good way to improve the situation here.
PS: And I don't mind him being paid well at all. I'm very much in favor of important OSS projects getting sponsored and rewarded. But if $7k/mo are not enough to maintain a package that others have re-implemented for free (openntpd etc.) then something seems seriously wrong.
> Most large corps run their own NTP servers, some of them public, e.g.:
Doesn't fix the problem. They have to sync to something, someone's gotta maintain the connection to actual atomic clocks. Something has to secure that connection. Someone has to maintain that something. You're severely underestimating the scope of the problem.
> As I see it someone like Google should indeed just hire the guy and take pool.ntp.org under their wing.
They should, but they probably won't, though they might as a result of this article. That would make protecting the world Google's responsibility and it doesn't make good corporate sense to do that. Google would have to find a real reason to throw $X0 million a year at this. The estimate given by Father Time was $4 million a year, that will only make sense if he builds it himself. If Google does it it will cost a lot more.
This is a resource allocation problem, one traditionally solved by governments. Nobody wants the government to do this, so we have to find some innovative way to fund critical technology projects.
The server you are showing is a stratum 1 time source intended for LAN use. It's not at all a given (and arguably not likely) that it would be suitable for exposing directly on the public internet to handle high traffic volumes. If not, then that means putting an ntpd in front of it, talking to it over the LAN, in which case you no longer have a stratum 1 time source available publicly.
I everyone had boxes like this, then yes, great, we'd not need reliable publicly accessible stratum 1 servers. But most people don't.
I everyone had boxes like this, then yes, great, we'd not need reliable publicly accessible stratum 1 servers. But most people don't.
Um. We have "reliable publicly accessible stratum 1 servers".
Regardless of what happens with the NTPd software, Google will probably continue to provide time1-4.google.com.
NIST will not shut down their timeservers.
And the 3654 servers in pool.ntp.org[1] (which seems to be maintained by a different guy) also won't just disappear overnight, though I'm not sure if these are Stratum 1 (probably not).
I still don't understand what exactly this guy is doing that should cost more than the $7k/mo that he's getting, much less the "$X0 million a year" that vinceguidry wants to allocate to the task.
Reading the NTP wiki page leads me to believe he's maintaining several reference clocks that can be used if you have no atomic clocks of your own. I would guess that he has to maintain servers with different implementations of the time algorithms so he can test new code against them. He needs to be able to simulate network configurations so he can reproduce issues.
> If it really was so dire then that would be all the more reason to not rely on a single "superhero" who may get run over by a bus tomorrow.
Which is why he for a couple of years have tried to get corporate sponsors for a foundation to hire more people to work on it, but even that has been met largely with apathy.
> But if $7k/mo are not enough to maintain a package that others have re-implemented for free (openntpd etc.) then something seems seriously wrong.
If you think that's all he does, then you don't understand what he's doing.
You also seem to have missed that $7k also covers all his costs, including hardware replacements and hosting for parts of the servers.
> Gladly NTP is not the mythical voodoo rocket science that you make it out to be.
Neither is DNS, but running the root DNS infrastructure (which serve hundreds of millions of users) is a wee bit more complicated than hosting forward lookup zones or even a public caching resolver on a handful of corporate nameservers. So too with the stratum-1 and stratum-2 NTP servers that serve as national or regional standards. They are not simple bits of kit that one throws together and shoves onto the public Internet, especially when they are ultimately used by just about every modern Internet-connected computer out there. That this guy's doing not only all of the software/release engineering but also operating some critical time servers for USD 84000 per year is too good of a deal to be true---and that he's running through his personal savings to finance the NTP project is just shameful on the IT community's (our) part.
You might not call it that, but for someone in devops with that length of experience, it's <50% of market value.
> Especially considering that the project isn't under heavy development.
According to the article, he spends ca. 100 hours a week on the project. It may not be under heavy development, but there's a constant stream of both development work and ops work.
> Also, I have to echo what other comments have said... he seems to be spending it oddly.
That is only relevant if you want to account for part of the $7000 as something other than salary, in which case he's even more woefully underpaid. Which he is anyway, given that he's also spending on some of the hosting and replacement parts out of his own pocket. Even so, his arrangement is not at all unusual at his level - I used to work for someone whose previous job involved flying from California to D.C. once a week for a couple of days. The cost of a travel arrangement like that is peanuts compared to his value. Especially given there's no established office or other admin costs.
> 1) Move him and his family closer to the data center (Perhaps difficult)
Why do you think he'd want to move for a job that's so underpaid, to one of the highest living cost regions in the world? Chances are good this would increase his monthly outgoings by more than he'd save on travel and board.
> 2) Move the servers to a data center that's closer to his home (Not as difficult)
Did you miss the part where isc.org that hosts the bulk of the server farm does not charge for it? If you can find someone with sufficiently low latency connection to major interchange points near him that are willing to host the bulk of the servers for free, I'm sure he'd be ecstatic to hear about it. If not, it'd almost certainly cost more than it'd save. It'd also represent a major risk to key internet infrastructure.
> 3) He could start using VMs like the rest of the world...
For stratum 1 servers? You clearly don't understand what it is he's doing. Stratum 1 servers needs a reliable physical connection to a reliable time source (such as a GPS receiver). He needs dedicated hardware. Try to get data center employees to properly handle non-standard hardware like that. I dare you.
> 4) Fire him and find someone else who would like a $7k/month stipend to do bugfixes (Fairly easy)
Good luck with that. Pretty much anyone prepared to take on that job for $7k/month including covering server costs and hardware repairs, is going to be woefully underqualified. In any case, if the situation doesn't improve, we'll find out soon given that he set an April deadline before he'll look for other things to do.
EDIT: Also note the part where it is unclear whether the $7k grant will even continue after April.
So he is apparently a family man, but he also spends 14 hours a day, 7 days a week on a project that hasn't largely changed in ages? Something doesn't add up.
> Did you miss the part where isc.org that hosts the bulk of the server farm does not charge for it?
It happened once. It could happen again. I refuse to believe there is no other organization willing to offer the same deal.
> For stratum 1 servers? You clearly don't understand what it is he's doing.
Yep, I realized my mistake after writing that, and the replies have made it clear that I was wrong. I still think there are options (remote access cards, for one) but I agree that VMs aren't the way to do it.
> it is unclear whether the $7k grant will even continue after April.
If that's the case, we'll see what happens in a month. If everything comes crashing down (it won't) I'll change my mind, but for now I believe one person being touted as the ONLY man who can do a particular job is incredibly wrong. Even if he were literally the only person on earth with the knowledge, it wouldn't be acceptable for everyone to rely on him... instead we should produce something as a community to replace this system. It's just not an acceptable option to have something this important in the sole hands of someone who could be in a plane crash tomorrow.
If he were working directly for Google or another bigco doing the same work, he'd be making more in the neighborhood of 200k/year, if not more. Someone skilled enough to do this work is worth considerably more than $7k/month.
> He could start using VMs like the rest of the world
High-accuracy/high-precision NTP servers are extraordinarily timing-sensitive, never mind the fact that they must be connected to some kind of external time standard. None of the currently available hypervisors support the kind of precision required by a stratum-1 NTP server. Even low-precision applications---such as Active Directory, which can handle clock skew of up to 5 minutes by default---recommend against running time servers in virtual machines.
To get an idea of the engineering tolerances involved, please refer to "Clock Quality" in the NTP FAQ:
The overall architectural goals are the same as every other FOSS project claims to follow: Simplicity, Quality, Security etc. etc. but I tend to think that we stick a little bit more closely to them.
This work is sponsored by Linux Foundation, partly in response to the HeartBleed fiasco, and after studying the 300,000+ lines of source-code in NTPD. I concluded that while it could be salvaged, it would be more economical, much faster and far more efficient to start from scratch.
The problem with all such initiatives ("we start clean") is that they typically never even reach the feature set of the previous software (implementing features takes orders of magnitude more time than some simple "proof of concept") and that they often make the same errors the older software already solved. JWZ wrote nicely somewhere about that effect (something like "it's not 'hard to read code,' those are the actual features implemented") but I can't find the link.
"Back to that two page function. Yes, I know, it's just a simple function to display a window, but it has grown little hairs and stuff on it and nobody knows why. Well, I'll tell you why: those are bug fixes. ...
Each of these bugs took weeks of real-world usage before they were found. The programmer might have spent a couple of days reproducing the bug in the lab and fixing it. If it's like a lot of bugs, the fix might be one line of code, or it might even be a couple of characters, but a lot of work and time went into those two characters.
When you throw away code and start from scratch, you are throwing away all that knowledge. All those collected bug fixes. Years of programming work."
"that's what happens when there is no incentive for people to do the parts of programming that aren't fun. Fixing bugs isn't fun; going through the bug list isn't fun; but rewriting everything from scratch is fun (because "this time it will be done right", ha ha) and so that's what happens, over and over again."
But the longer one, containing more or less the quote I first approximated, I just can't find. If I remeber he wrote about Netscape, the code for FTP and how long it took to get it right in all edge cases, and then it was thrown away.
I hardly think PHK is ADT at all. There are often good engineering reasons to rewrite software. I think Joel's point is that it is more often undertaken for wrong reasons.
There's the software that works, he gets the money to find the bugs and fix them, he decides to write from the scratch something that certainly isn't the replacement of the existing software except for some specific users. That is very ADT, exactly to JWZ's definition:
"This is, I think, the most common way for my bug reports to open source software projects to ever become closed. I report bugs; they go unread for a year, sometimes two; and then (surprise!) that module is rewritten from scratch -- and the new maintainer can't be bothered to check whether his new version has actually solved any of the known problems that existed in the previous version."
He got the money to look for the bugs and to fix them (hard). He instead goes to make the fully new undiscoverd bugs in fully new (his own) code (easy).
TDD can go horribly wrong, too -- but if your tests are on high-enough-level functionality, and you maintain them, then they can encompass the lessons you learn from fucking up every time you do it. Writing & Maintaining tests isn't fun, just as fixing bugs isn't fun -- so in spirit these kinds of arguments hold just as much sway: there are fun parts of programming, and there are parts of programming that are significantly less fun, but even if you fork and start from scratch, it is feasible to have a checklist of bugs/features developed in the order that they are developed initially.
Do most open source projects do this, even the more well-maintained? Of course not. However, if people are seriously worried about this phenomenon, that's probably one of the ways developed in the decade since this essay was published to approach it.
It doesn't convince me. There's no one answer that's correct for all situations.
Sometimes, had it been written well from the beginning, there wouldn't have been as many bugs in the first place, though.
I've seen codebases that really shouldn't be redone, just maintained and slightly patched over time - the risks are too high, the maintenance is low, patches are at least understood (mostly). I've also seen other codebases that should be scrapped and restarted. It really depends on the skills, commitment and expectations of the parties involved, and there's no one answer that fits all situations.
This time, PHK wasn't able to claim that he recognized any serious issues, but still started from scratch, leaving out even the idea to support all that NTPd already has fully implemented. And he received money to find and fix the bugs, not to make one more proof of concept.
Wasn't specifically defending the PHK decision, more just against the Spolsky article. I've seen it quoted as gospel over the past decade, and ... it just doesn't hold true in all situations.
It depends on the initial quality of the code. Spolsky happens to work with very talented people who tend not to write bad code, so if something looks weird and you don't know why it's there it's easy to give someone the benefit of the doubt.
But if you don't work with talented/skilled/experienced people and there's "weird stuff" in the codebase it might well be for no good reason at all.
You can only invoke the Spolsky "it's bug-fixes!" argument if you're not cleaning up someone's horrific mess.
Exactly, but that nuance is lost on many people. I've lost count of how many people have quoted "don't restart from scratch!" and cited that Spolsky article as some sort of irrefutable wisdom of the ages.
Is it the boy scouts that promote "leave the campground cleaner than you found it"? That's the attitude I try to bring to projects, but there are limits, both time and effort. I can leave you (client, employer) a much better system (by whatever metrics you want to establish) by rebuilding from scratch when what I'm starting with is a broken, unstable mess. not always, but the idea shouldn't be dismissed out of hand because of something Joel Spolsky wrote about Netscape in 2000.
I used to work at a company that cached international shipping rates. Those rates are BASICALLY by country because most of the fuel is burned to move the package the gross distance from one country to another.
But every once in a while we'd end up paying double for a shipment because the customer was way out in the sticks or whatever. Had we done more dynamic stuff like taking the whole address into account when quoting a price to a customer we'd never get bit by this kind of problem. But it was only a couple of times a month so I didn't worry too much about it.
My replacement found that this bothered him a lot and he figured he'd score points by fixing this problem. So he did exactly that, transitioned the entire quote system from local database lookups to remote UPS/FedEx/USPS/etc calls. 2-4 rates per shipper (Ground, Air, etc) for a total of about 10-15 every time a customer wanted a quote. And because we would repackage stuff (it was a logistics company) we often never knew the exact weight so we'd quote 3-4 prices so people could get a feel for which choice was their best bet for the best rate without delaying everything by an extra day or two in order to get a hard quote.
We cached these rates by country and weight (up to 1000lbs) so between all the service offerings and whatnot it was about 100,000 pieces of information in our actual, but occasionally incorrect database. So there were two choices:
1. Don't do any caching and just look them all up in realtime for customers. They're web APIs so there's latency associated.
2. Cache, but on a per-address basis. We had an address book for our customers so we knew the couple of addresses they would want to ship to and we could aggressively warm the cache so that all the rates would already be there. But there were about 10k unique addresses in the database * 100k total rates = 1 billion rates that needed to be cached.
When I presented this back-of-the-envelope calculation in a meeting do you know how he blew all of it off?
"Premature optimization is the root of all evil" -- Donald Knuth
I was so flabbergasted that someone could be aggressively ignorant and yet somehow twist Knuth's words to support their own position that I simply gave up. I was dealing with a powerful stupidity and it was stronger than me.
I later heard that during the transition it was touch-and-go for about a week and they had to issue a lot of credit to pissed off customers. The rate quoting page went from about 20ms to render (and maybe 300ms to load) now to about 4 seconds.
If my system requirements are that something run in 100ms, dismissing options that have a min 2s latency (however useful they are) is not "premature optimization". It's a system requirement, and options B, C and D don't make the cut. Simple as that.
That is exactly what PHK plans to do with ntimed: split it up into separate packages for clients, time distribution servers and authoritative time transmitters.
Stratum 1 support doesn't require a lot of different hardware, arguably it requires less: you need only a PPS device; which has become increasingly generic due to the kernel support needed for precise timestamping. (okay sure, you also need to initially name the second, but you can have another NTP server do that).
From [1] I get that PHK decided not to use privilege separation because it's not portable enough. AFAIK that's one of the main differences between OpenNTPD and Ntimed. Well, that and precision of course. OpenNTPDs accuracy is "only" around milliseconds [2] (which is probably good enough for anyone not using dedicated time hardware like a stratum server). The portable version missed the frequency adjustment code for years, but recently this has been added to the portable version as well.
I've also read a discussion with PHK somewhere that he didn't like OpenNTPD because it had no auth (sorry, can't find the source again). I guess this complaint has been mitigated recently now that TLS auth is supported in OpenNTPD [3].
/edit: not sure if it's clear but I was comparing OpenNTPD to Ntimed. But the named differences still exist when compared to ntp4.
ntimed is a weird project IMO, because it's not different enough from OpenNTPD: both are small NTP daemons written in C. He could've created a fork of OpenNTPD with more precision and without privilege separation instead.
If ntimed was written in Rust though... THAT would've been excellent.
> I've also read a discussion with PHK somewhere that he didn't like OpenNTPD
> because it had no auth (sorry, can't find the source again). I guess this
> complaint has been mitigated recently now that TLS auth is supported in
> OpenNTPD [3].
That's a completely new invention. It is not compatible with the NTP auth as specified in NTPv4 which uses a public key mechanism to ensure the NTP packets are not being spoofed.
Instead, this is just pointing your NTP server at an HTTPS server you trust so it can use the timestamps returned in the TLS packets as another highly trusted time source to make sure spoofed NTP packets can't skew your time. It's clever, that's for sure.
…commercial software such as the Coverity system for security checking on code, must come out of the $7,000 monthly stipend or be charged to his consulting business.
Coverity is free for Open Source projects, why is he paying for it at all?
Can somebody shed some light on how maintaining one network protocol implementation and possibly improving it every couple of years could consume three million Dollars every year? We all know how easy it is to underestimate the required effort and how quickly tiny things can blow up into major issues, but it is completely beyond my imagination in the case of NTP.
This sounds like a bad and unsustainable idea, with capital B and U. Shouldn't it be the responsibility of the vendor to provide a driver? Why would you put the burden to deal with all kinds of clock hardware onto someone maintaining a network protocol implementation? I could see reasons to have one or three or so generic drivers, but if the vendor comes up with something fancy not covered by one of the few generic drivers it should be their responsibility to provide a suitable driver.
Most time sources have other uses than serving NTP time. Those are the uses that people want to pay lots of money for. Serving NTP time isn't very lucrative.
Not wanting to be too blunt but how're the Linux Foundation ensuring that if this guy dies [or decides to quit] then the situation is readily recoverable - do they have a system where they keep credentials recorded and have access to the data-centres and such?
Wouldn't it be considerably cheaper to hire someone at the data-centre to do the reboots or whatever it is he's in situ to do? [What do they call that "remote hands" or something.]
"Apple Macintosh computers and servers running OSX use NTP, and Stenn said Apple developers have called him for help on several NTP issues. In the last such incident, he said he delayed a patch to give Apple more time to prepare OS X for it. When they were ready, he applied the patch and asked "whether Apple could send a donation to the Network Time Foundation," Stenn recalled. "They said they would do their best to see that Apple throws some money our way." But it hasn't happened yet."
Surely, he needs to say upfront that there is a consultancy fee. I'm sure most big companies can't make donations easily, what they can do is pay for services and products which they do all day, everyday.
It's becoming entirely clear to me that the vast bulk of nerds running open source projects do not have the requisite skills to operate their projects as a sustainable business adequate to pay their own bills.
I am, broadly, of the opinion that a non-profit either needs to take up (or form), an "infrastructure consultancy" firm with financial structure and incentives to ensure that projects like LibreSSL, GPG, NTP, etc are funded and maintained; some of that will involve consulting work for large firms for large piles of money.
Anyway, I don't have a lot of swing in that field, but... it's my conclusion. :-)
I've come to the same conclusion. It needs a set of junior consultants to do more basic stuff for a decent amount, leaving the main person/people as the highest cost per day - and only called in if necessary. Someone to setup funding or support structures with companies that need/want it.
Sorry, I tried to find an alternate article but this seems to be the original and the only other one I found was blogspam based on this. I'm a bit annoyed with the ads and the fact it's split over 5 pages, also (I even checked the "Printer Friendly" link to see if it would display all on one page, but it didn't). Still, I thought the content itself would be interesting here.
Well, the article does a terrible job at explaining why that rack is needed and why it can't be in a datacenter in the same city as the NTPd maintainer.
Honestly, I have a hard time believing there is new NTP hardware coming out every month that the maintainer then personally has to write drivers for.
And even if that was the case, what prevents that hardware from working when it's located, say, on a desk in the house of the maintainer (I think most timeservers are fanless and don't consume very much energy)?
Perhaps there's valid reasons and explanations for his monthly trips to that datacenter, but the article only left me with questions.
It would be nice if people questioning what he's paid would ask questions first, before making claims about what he should be doing...
> Well, the article does a terrible job at explaining why that rack is needed and why it can't be in a datacenter in the same city as the NTPd maintainer.
For starters, the majority of the servers are currently hosted for free at isc.org. You'd save a tiny amount on his monthly travel (I don't know what his costs are, but I used to stay in Palo Alto for weeks at a time in my last job, and it should be doable for him to travel to, and stay in, San Jose for a week for <$1k/month), in return for a as much or more in hosting costs unless you can find someone willing to give him free space.
That's assuming you can find a nearby suitable data centre.
> I wonder what prevents that hardware from working from, say, a desk in the house of the maintainer
Nothing. If you're willing to fund sufficient low latency, high capacity, redundant internet connections to his house. The price would almost certainly be far higher than flying him to San Jose once a month to maintain the servers there.
Apple is touting a 50 ms accuracy on their new Apple Watch. That sounds suspiciously like the accuracy of NTP. If Apple is relying on NTP for the watch, they would be crazy to not help out financially for its support.
No watch gets it's time using NTP, and I don't even know how that would work. Almost all radio timepieces take their time from WWVB (or international equivalents). There may be a few that use GPS like cell phones do.
Hmmm, you're right, the Apple site says it uses GPS and WiFi in the iphone, whatever that means. So I guess it's possible the watch syncs with the phone using NTP. Good catch.
Maybe I missed it, but I don't understand why one of the companies that creates NTP devices doesn't just hire him. They surely already have plenty of engineers on staff, what's another?
87 comments
[ 1.2 ms ] story [ 167 ms ] thread"With the Linux Foundation's $7,000 in monthly cash flow, Stenn finances his movement between his home lab, in Talent, Ore., and the NTP servers located in San Jose, Calif. In Oregon, Stenn lives with his wife and does most of his patch inspection, code writing, and release building three weeks a month. The fourth week, he stays in San Jose, close to two co-location data center providers that host NTP computers. He rents a room there to work on server and network administration, maintain the email list, and check on server backups.
Much of the travel, room, replacement hardware such as disk drives, or needed commercial software such as the Coverity system for security checking on code, must come out of the $7,000 monthly stipend or be charged to his consulting business.
Most of his 17-20 servers came out of a one-time, $10,000 grant in 2010 from the Internet Society, a policy and technology infrastructure advisory body for the Internet founded in 1991. Those servers are running at ISC.org in Redwood City, Calif., which hosts BIND and several other open source pieces of Internet infrastructure. For 15 years, it has provided space, electricity and some management "smart hands" to host NTP operations, without charging, said Stenn. "They would love for us to pay them," he said, and he once totaled the monthly bill at $1,400. But ISC.org also knows the NTP project can't pay and continues to host it, Stenn added."
And on the first page:
"Its ongoing development and maintenance now rest mostly on the shoulders of Stenn, and that's why NTP faces a turning point. Stenn, who also works sporadically on his own consulting business, has given himself a deadline: Garner more financial support by April, "or look for regular work.”"
Now although he has a donate link on http://www.ntp.org to:
http://networktimefoundation.org/donate/
he doesn't have a clear goal set like GnuPG had. GnuPG doesn't state the goal because it reached it. But until then you were able to actually see how your support moves the bar. Maybe setting the goal would help here?
It's hard to sympathize when there are such frivolous expenses. There's what, a thousand bucks at least on travel and accommodation per month.
Going on-site is understandable when updates need to be performed that are too risky to do remotely. On-site on a fixed schedule, however, would quickly be over-ruled in any commercial operation.
I agree.
If the article is a misrepresentation then they should correct that asap, it leaves a rather bad taste in the mouth.
It feels like this guy saw how well the sob-story fundraiser worked for GnuPG[1] and figured he'd give it a shot, too.
That's all good and well, but not while you already have a sponsor as generous as the Linux Foundation seems to be here.
[1] https://twitter.com/stripe/status/563449352635432960
There does not exactly appear to be a queue of people wanting to take over the reigns.
See the other comments here about that.
There are at least 4 simple fixes I can think of:
1) Move him and his family closer to the data center (Perhaps difficult)
2) Move the servers to a data center that's closer to his home (Not as difficult)
3) He could start using VMs like the rest of the world... physical access really isn't needed except for physical security (making sure the rack is locked and access is controlled), which data center employees can manage (Fairly easy)
4) Fire him and find someone else who would like a $7k/month stipend to do bugfixes (Fairly easy)
If you go with a superhero, then you have to deal with him. You can't just say, "Sorry bud, we need to uproot you and move you in-town." Even if he doesn't say no, which he very well could, leaving you and your plans fucked and the world unprotected, driving all the enjoyment out of the job will make him eventually say "fuck it, I'm retiring."
Okay, let's go with an organization. Who is going to build it? You still need the superhero. The critical nature of the infrastructure demands you use the guy whose been covering your ass all this time, because most anybody else you're going to find is going to fuck it up, and there can be no possibility of a fuck-up.
And you can't buy a superhero for any amount of money. There just aren't that many of them.
If there's a vulnerability it affects the entire world
Who is going to build it? You still need the superhero.
Hyperbole much?
If it really was so dire then that would be all the more reason to not rely on a single "superhero" who may get run over by a bus tomorrow.
Gladly NTP is not the mythical voodoo rocket science that you make it out to be. Most large corps run their own NTP servers, some of them public, e.g.:
You can also ask your friendly government for the time: http://tf.nist.gov/tf-cgi/servers.cgiThe official NTPd impl doesn't even have a very good reputation (cf. the recent debate about that security vulnerability).
As I see it someone like Google should indeed just hire the guy and take pool.ntp.org under their wing. Throwing even more money at a single guy doesn't convince me as a good way to improve the situation here.
PS: And I don't mind him being paid well at all. I'm very much in favor of important OSS projects getting sponsored and rewarded. But if $7k/mo are not enough to maintain a package that others have re-implemented for free (openntpd etc.) then something seems seriously wrong.
Doesn't fix the problem. They have to sync to something, someone's gotta maintain the connection to actual atomic clocks. Something has to secure that connection. Someone has to maintain that something. You're severely underestimating the scope of the problem.
> As I see it someone like Google should indeed just hire the guy and take pool.ntp.org under their wing.
They should, but they probably won't, though they might as a result of this article. That would make protecting the world Google's responsibility and it doesn't make good corporate sense to do that. Google would have to find a real reason to throw $X0 million a year at this. The estimate given by Father Time was $4 million a year, that will only make sense if he builds it himself. If Google does it it will cost a lot more.
This is a resource allocation problem, one traditionally solved by governments. Nobody wants the government to do this, so we have to find some innovative way to fund critical technology projects.
As I understand it, they usually sync to GPS?
You can buy those boxes on Amazon[1], starting at around $299.
Google would have to find a real reason to throw $X0 million a year at this
$X0 million?!
To construct and run their own atomic clock, or what would they spend so much money on?
I may indeed be completely missing what this guy does. He runs an actual atomic clock in that rack, like the one from NIST[2]?
[1] http://www.amazon.com/TM1000A-GPS-Network-Time-Server/dp/B00...
[2] http://en.wikipedia.org/wiki/NIST-F1
I everyone had boxes like this, then yes, great, we'd not need reliable publicly accessible stratum 1 servers. But most people don't.
Um. We have "reliable publicly accessible stratum 1 servers".
Regardless of what happens with the NTPd software, Google will probably continue to provide time1-4.google.com.
NIST will not shut down their timeservers.
And the 3654 servers in pool.ntp.org[1] (which seems to be maintained by a different guy) also won't just disappear overnight, though I'm not sure if these are Stratum 1 (probably not).
I still don't understand what exactly this guy is doing that should cost more than the $7k/mo that he's getting, much less the "$X0 million a year" that vinceguidry wants to allocate to the task.
[1] http://www.pool.ntp.org/zone
Which is why he for a couple of years have tried to get corporate sponsors for a foundation to hire more people to work on it, but even that has been met largely with apathy.
> But if $7k/mo are not enough to maintain a package that others have re-implemented for free (openntpd etc.) then something seems seriously wrong.
If you think that's all he does, then you don't understand what he's doing.
You also seem to have missed that $7k also covers all his costs, including hardware replacements and hosting for parts of the servers.
Neither is DNS, but running the root DNS infrastructure (which serve hundreds of millions of users) is a wee bit more complicated than hosting forward lookup zones or even a public caching resolver on a handful of corporate nameservers. So too with the stratum-1 and stratum-2 NTP servers that serve as national or regional standards. They are not simple bits of kit that one throws together and shoves onto the public Internet, especially when they are ultimately used by just about every modern Internet-connected computer out there. That this guy's doing not only all of the software/release engineering but also operating some critical time servers for USD 84000 per year is too good of a deal to be true---and that he's running through his personal savings to finance the NTP project is just shameful on the IT community's (our) part.
You might not call it that, but for someone in devops with that length of experience, it's <50% of market value.
> Especially considering that the project isn't under heavy development.
According to the article, he spends ca. 100 hours a week on the project. It may not be under heavy development, but there's a constant stream of both development work and ops work.
> Also, I have to echo what other comments have said... he seems to be spending it oddly.
That is only relevant if you want to account for part of the $7000 as something other than salary, in which case he's even more woefully underpaid. Which he is anyway, given that he's also spending on some of the hosting and replacement parts out of his own pocket. Even so, his arrangement is not at all unusual at his level - I used to work for someone whose previous job involved flying from California to D.C. once a week for a couple of days. The cost of a travel arrangement like that is peanuts compared to his value. Especially given there's no established office or other admin costs.
> 1) Move him and his family closer to the data center (Perhaps difficult)
Why do you think he'd want to move for a job that's so underpaid, to one of the highest living cost regions in the world? Chances are good this would increase his monthly outgoings by more than he'd save on travel and board.
> 2) Move the servers to a data center that's closer to his home (Not as difficult)
Did you miss the part where isc.org that hosts the bulk of the server farm does not charge for it? If you can find someone with sufficiently low latency connection to major interchange points near him that are willing to host the bulk of the servers for free, I'm sure he'd be ecstatic to hear about it. If not, it'd almost certainly cost more than it'd save. It'd also represent a major risk to key internet infrastructure.
> 3) He could start using VMs like the rest of the world...
For stratum 1 servers? You clearly don't understand what it is he's doing. Stratum 1 servers needs a reliable physical connection to a reliable time source (such as a GPS receiver). He needs dedicated hardware. Try to get data center employees to properly handle non-standard hardware like that. I dare you.
> 4) Fire him and find someone else who would like a $7k/month stipend to do bugfixes (Fairly easy)
Good luck with that. Pretty much anyone prepared to take on that job for $7k/month including covering server costs and hardware repairs, is going to be woefully underqualified. In any case, if the situation doesn't improve, we'll find out soon given that he set an April deadline before he'll look for other things to do.
EDIT: Also note the part where it is unclear whether the $7k grant will even continue after April.
So he is apparently a family man, but he also spends 14 hours a day, 7 days a week on a project that hasn't largely changed in ages? Something doesn't add up.
> Did you miss the part where isc.org that hosts the bulk of the server farm does not charge for it?
It happened once. It could happen again. I refuse to believe there is no other organization willing to offer the same deal.
> For stratum 1 servers? You clearly don't understand what it is he's doing.
Yep, I realized my mistake after writing that, and the replies have made it clear that I was wrong. I still think there are options (remote access cards, for one) but I agree that VMs aren't the way to do it.
> it is unclear whether the $7k grant will even continue after April.
If that's the case, we'll see what happens in a month. If everything comes crashing down (it won't) I'll change my mind, but for now I believe one person being touted as the ONLY man who can do a particular job is incredibly wrong. Even if he were literally the only person on earth with the knowledge, it wouldn't be acceptable for everyone to rely on him... instead we should produce something as a community to replace this system. It's just not an acceptable option to have something this important in the sole hands of someone who could be in a plane crash tomorrow.
High-accuracy/high-precision NTP servers are extraordinarily timing-sensitive, never mind the fact that they must be connected to some kind of external time standard. None of the currently available hypervisors support the kind of precision required by a stratum-1 NTP server. Even low-precision applications---such as Active Directory, which can handle clock skew of up to 5 minutes by default---recommend against running time servers in virtual machines.
To get an idea of the engineering tolerances involved, please refer to "Clock Quality" in the NTP FAQ:
http://www.ntp.org/ntpfaq/NTP-s-sw-clocks-quality.htm
3) We are talking about accurate time here. Using VMs is not so conducive to that.
4) You would need someone as knowledgeable as him, and preferably as passionate about accurate time. And he's also doing a bunch of other things.
The overall architectural goals are the same as every other FOSS project claims to follow: Simplicity, Quality, Security etc. etc. but I tend to think that we stick a little bit more closely to them.
This work is sponsored by Linux Foundation, partly in response to the HeartBleed fiasco, and after studying the 300,000+ lines of source-code in NTPD. I concluded that while it could be salvaged, it would be more economical, much faster and far more efficient to start from scratch.
Ntimed is the result.
http://phk.freebsd.dk/time/20140926.html
The problem with all such initiatives ("we start clean") is that they typically never even reach the feature set of the previous software (implementing features takes orders of magnitude more time than some simple "proof of concept") and that they often make the same errors the older software already solved. JWZ wrote nicely somewhere about that effect (something like "it's not 'hard to read code,' those are the actual features implemented") but I can't find the link.
"Back to that two page function. Yes, I know, it's just a simple function to display a window, but it has grown little hairs and stuff on it and nobody knows why. Well, I'll tell you why: those are bug fixes. ...
Each of these bugs took weeks of real-world usage before they were found. The programmer might have spent a couple of days reproducing the bug in the lab and fixing it. If it's like a lot of bugs, the fix might be one line of code, or it might even be a couple of characters, but a lot of work and time went into those two characters.
When you throw away code and start from scratch, you are throwing away all that knowledge. All those collected bug fixes. Years of programming work."
http://www.jwz.org/doc/cadt.html
"that's what happens when there is no incentive for people to do the parts of programming that aren't fun. Fixing bugs isn't fun; going through the bug list isn't fun; but rewriting everything from scratch is fun (because "this time it will be done right", ha ha) and so that's what happens, over and over again."
But the longer one, containing more or less the quote I first approximated, I just can't find. If I remeber he wrote about Netscape, the code for FTP and how long it took to get it right in all edge cases, and then it was thrown away.
"This is, I think, the most common way for my bug reports to open source software projects to ever become closed. I report bugs; they go unread for a year, sometimes two; and then (surprise!) that module is rewritten from scratch -- and the new maintainer can't be bothered to check whether his new version has actually solved any of the known problems that existed in the previous version."
http://www.jwz.org/doc/cadt.html
He got the money to look for the bugs and to fix them (hard). He instead goes to make the fully new undiscoverd bugs in fully new (his own) code (easy).
TDD can go horribly wrong, too -- but if your tests are on high-enough-level functionality, and you maintain them, then they can encompass the lessons you learn from fucking up every time you do it. Writing & Maintaining tests isn't fun, just as fixing bugs isn't fun -- so in spirit these kinds of arguments hold just as much sway: there are fun parts of programming, and there are parts of programming that are significantly less fun, but even if you fork and start from scratch, it is feasible to have a checklist of bugs/features developed in the order that they are developed initially.
Do most open source projects do this, even the more well-maintained? Of course not. However, if people are seriously worried about this phenomenon, that's probably one of the ways developed in the decade since this essay was published to approach it.
Sometimes, had it been written well from the beginning, there wouldn't have been as many bugs in the first place, though.
I've seen codebases that really shouldn't be redone, just maintained and slightly patched over time - the risks are too high, the maintenance is low, patches are at least understood (mostly). I've also seen other codebases that should be scrapped and restarted. It really depends on the skills, commitment and expectations of the parties involved, and there's no one answer that fits all situations.
But if you don't work with talented/skilled/experienced people and there's "weird stuff" in the codebase it might well be for no good reason at all.
You can only invoke the Spolsky "it's bug-fixes!" argument if you're not cleaning up someone's horrific mess.
Is it the boy scouts that promote "leave the campground cleaner than you found it"? That's the attitude I try to bring to projects, but there are limits, both time and effort. I can leave you (client, employer) a much better system (by whatever metrics you want to establish) by rebuilding from scratch when what I'm starting with is a broken, unstable mess. not always, but the idea shouldn't be dismissed out of hand because of something Joel Spolsky wrote about Netscape in 2000.
But every once in a while we'd end up paying double for a shipment because the customer was way out in the sticks or whatever. Had we done more dynamic stuff like taking the whole address into account when quoting a price to a customer we'd never get bit by this kind of problem. But it was only a couple of times a month so I didn't worry too much about it.
My replacement found that this bothered him a lot and he figured he'd score points by fixing this problem. So he did exactly that, transitioned the entire quote system from local database lookups to remote UPS/FedEx/USPS/etc calls. 2-4 rates per shipper (Ground, Air, etc) for a total of about 10-15 every time a customer wanted a quote. And because we would repackage stuff (it was a logistics company) we often never knew the exact weight so we'd quote 3-4 prices so people could get a feel for which choice was their best bet for the best rate without delaying everything by an extra day or two in order to get a hard quote.
We cached these rates by country and weight (up to 1000lbs) so between all the service offerings and whatnot it was about 100,000 pieces of information in our actual, but occasionally incorrect database. So there were two choices:
1. Don't do any caching and just look them all up in realtime for customers. They're web APIs so there's latency associated.
2. Cache, but on a per-address basis. We had an address book for our customers so we knew the couple of addresses they would want to ship to and we could aggressively warm the cache so that all the rates would already be there. But there were about 10k unique addresses in the database * 100k total rates = 1 billion rates that needed to be cached.
When I presented this back-of-the-envelope calculation in a meeting do you know how he blew all of it off?
"Premature optimization is the root of all evil" -- Donald Knuth
I was so flabbergasted that someone could be aggressively ignorant and yet somehow twist Knuth's words to support their own position that I simply gave up. I was dealing with a powerful stupidity and it was stronger than me.
I later heard that during the transition it was touch-and-go for about a week and they had to issue a lot of credit to pissed off customers. The rate quoting page went from about 20ms to render (and maybe 300ms to load) now to about 4 seconds.
If my system requirements are that something run in 100ms, dismissing options that have a min 2s latency (however useful they are) is not "premature optimization". It's a system requirement, and options B, C and D don't make the cut. Simple as that.
E.g. perhaps it makes sense to have an entirely separately client-only package with a separate codebase.
[1] http://chrony.tuxfamily.org/
NTP needs time sources too, not just clients.
I've also read a discussion with PHK somewhere that he didn't like OpenNTPD because it had no auth (sorry, can't find the source again). I guess this complaint has been mitigated recently now that TLS auth is supported in OpenNTPD [3].
/edit: not sure if it's clear but I was comparing OpenNTPD to Ntimed. But the named differences still exist when compared to ntp4.
[1] http://phk.freebsd.dk/_downloads/FOSDEM_2015.pdf page 13 and 14
[2] http://www.bsdnow.tv/episodes/2015_02_11-time_for_a_change at 30:30 and 34:00
[3] http://undeadly.org/cgi?action=article&sid=20150210103656
If ntimed was written in Rust though... THAT would've been excellent.
Instead, this is just pointing your NTP server at an HTTPS server you trust so it can use the timestamps returned in the TLS packets as another highly trusted time source to make sure spoofed NTP packets can't skew your time. It's clever, that's for sure.
Coverity is free for Open Source projects, why is he paying for it at all?
Wouldn't it be considerably cheaper to hire someone at the data-centre to do the reboots or whatever it is he's in situ to do? [What do they call that "remote hands" or something.]
"Apple Macintosh computers and servers running OSX use NTP, and Stenn said Apple developers have called him for help on several NTP issues. In the last such incident, he said he delayed a patch to give Apple more time to prepare OS X for it. When they were ready, he applied the patch and asked "whether Apple could send a donation to the Network Time Foundation," Stenn recalled. "They said they would do their best to see that Apple throws some money our way." But it hasn't happened yet."
Surely, he needs to say upfront that there is a consultancy fee. I'm sure most big companies can't make donations easily, what they can do is pay for services and products which they do all day, everyday.
Customer: I'd like new MacBook, I really need to get some work done.
Apple Employee: Here you go, this should solve your problem. (gives him a new MacBook)
Customer: Great
Apple Employee: Could donate to the Apple foundation?
Customer: Sure, I'll ask my partner what they think. (walks out the shop with the MacBook, without paying for anything).
I am, broadly, of the opinion that a non-profit either needs to take up (or form), an "infrastructure consultancy" firm with financial structure and incentives to ensure that projects like LibreSSL, GPG, NTP, etc are funded and maintained; some of that will involve consulting work for large firms for large piles of money.
Anyway, I don't have a lot of swing in that field, but... it's my conclusion. :-)
This goes to show that the person making the comment is thoroughly unqualified to make any kind of substantive comment on anything that this guy does.
Well, the article does a terrible job at explaining why that rack is needed and why it can't be in a datacenter in the same city as the NTPd maintainer.
Honestly, I have a hard time believing there is new NTP hardware coming out every month that the maintainer then personally has to write drivers for.
And even if that was the case, what prevents that hardware from working when it's located, say, on a desk in the house of the maintainer (I think most timeservers are fanless and don't consume very much energy)?
Perhaps there's valid reasons and explanations for his monthly trips to that datacenter, but the article only left me with questions.
> Well, the article does a terrible job at explaining why that rack is needed and why it can't be in a datacenter in the same city as the NTPd maintainer.
For starters, the majority of the servers are currently hosted for free at isc.org. You'd save a tiny amount on his monthly travel (I don't know what his costs are, but I used to stay in Palo Alto for weeks at a time in my last job, and it should be doable for him to travel to, and stay in, San Jose for a week for <$1k/month), in return for a as much or more in hosting costs unless you can find someone willing to give him free space.
That's assuming you can find a nearby suitable data centre.
> I wonder what prevents that hardware from working from, say, a desk in the house of the maintainer
Nothing. If you're willing to fund sufficient low latency, high capacity, redundant internet connections to his house. The price would almost certainly be far higher than flying him to San Jose once a month to maintain the servers there.
What for?
Does he run public timeservers himself?
EDIT: The apple watch uses GPS.
Is it that he doesn't want to work for a company?