This weekend the system that monitors the NTP servers in the NTP Pool got a major overhaul!

NTP servers are now monitored from a number of monitors across the world, usually closer to the server than the single monitor was before.

One of the most frustrating things about operating an NTP server in the NTP Pool was how random network issues far away from the server would impact the score. Sometimes cause an email to be sent to the operator about potential problems.

Given the far away network issues and that NTP Pool users generally are sent to NTP servers close to them, this wasn’t very helpful.

Our testing over the last year indicates that the new system will perform much better in this regard. It also removes the single monitoring system as a single point of failure (or sometimes just trouble).

There are some technical details about the new system.

During the first days of the new system being in production we found a couple of issues and bugs that caused some server operators to get extra emails about trouble that wasn’t (the opposite of what was intended with the upgrade!). This was resolved the evening of March 20th.

The production rollout and these issues were discussed in the community forum.

This year the NTP Pool Project has been serving time to the world for 20 years!

Trillions and trillions of DNS requests have been served to billions of clients, with the NTP requests handled by thousands of NTP servers across the world day and night.

Development on the project ebbs and flows, but maintenance and upgrades on the production infrastructure is consistent – and constant.

Our community is active; and the NTP server operators even more so with almost 2,000 operators managing the 3,000 IPv4 NTP IPs and 1,600 IPv6 IPs active in the system.

Since last spring the beta site has been running a new NTP server monitoring system that’s nearly ready for the production site.

How the vendor DNS zones are managed have also been vastly improved in this upgrade, paving the way for (finally!) expanding IPv6 support and decreasing the dependency on too few servers in some of the underserved regions. I also expect it for new operators in underserved regions to participate, because the updates will better balance traffic in those areas.

When I took over development and maintenance of the project in 2005 I couldn’t have imagined what crucial internet infrastructure this project would be in 2023.

The upcoming changes are bringing further dependability and longevity to the infrastructure and the project as a whole. I’m excited to see where the next decade takes us.

The main website (www.ntppool.org / www.pool.ntp.org) doesn’t set any browser cookies. Some access logs are generated strictly for diagnostics purposes. No long term storage or analytics on user behavior is attempted.

The manage website (manage.ntppool.org) sets a cookie on login to track authentication. The site also keeps a record of the account information you provide and NTP server IPs that are registered. We try hard to not keep any information that’s not essential for operating the system.

If you have questions, concerns or suggestions you can post in the development forum, on github or via email.

The sites use a few other services with their own policies or privacy statements:

• Google Fonts

• Statuspage.io don’t set a cookie but may track usage in their logs.

• DNS mapper

• Plausible

Packet is awesome.

When we started planning our recent unplanned server move, we investigated options for having not one, but two sites, for the “hub” systems for the NTP Pool. With 4000 NTP servers and hundreds of millions of clients using the system, it really should be a given!

Evaluating our options on a ridiculously short timeframe, Packet stood out as an interesting choice, though we were a little apprehensive at first if their setup would be too unusual compared to more familiar options.

After a quick chat with some of the friendly staff at Packet, we were off to the races to see if we could get everything migrated in less than a week of nights and weekends. If we could, we’d be able to move the physical servers the following Sunday without downtime to any critical services, and get us closer to having proper redundancy.

Working with the Packet system has been fascinating and extremely productive. Despite having done this sort of work for several decades, it was a surprise how mixing familiar capabilities, APIs and abstractions opened new ways for quickly building and managing powerful, reliable and scalable infrastructure.

Continue Reading →

The beta site has been updated with new features for managing the accounts. Until now each server had to be associated with just one user login. In the new system servers are associated with an account that can have multiple user logins.

If you can, please try it out and post bug reports or suggestions in the development forum or via email.

The NTP Pool consists of (as of this writing) more than 4000 NTP servers provided by the community, about 40 DNS servers and a good handful of “hub servers” running the website, databases, monitoring (for NTP, DNS, etc) and a bunch of other software to keep the system going.

This spring we learned that the facility the systems were in was being decommissioned and we needed to find other arrangements, quickly. A few weeks later we had some options lined up and a long list of work to make it a smooth migration.

We worked at a frantic pace for a week to prepare for moving the physical servers by the deadline without any service impact (more on that later) and a Sunday in early may we got everything un-installed and moved to a NetActuate in Los Angeles.

By the end of the day we had the network up and the equipment racked and relatively neatly cabled (we counted it’s about the 5th move in about 20 years of maintaining various Perl community infrastructure and about 14 years of the NTP Pool, so we only get sporadic practice!).

When we first got in touch, NetActuate very quickly understood what we needed and figured out how to make it work on time and budget. Their services align unusually closely with the problem domain of the NTP Pool, with many of their customers using them to host anycast services across 70+ worldwide facilities.

NetActuate has been extremely easy and quick to work with and we are looking forward to a long and productive relationship with them. They are a small, friendly and highly technical company. Our experience is that everyone we talked to were experts at what they did and always the “real people” doing the work.

The hectic move also taught us that for something that so many of you in the community and beyond depend on working, we should get a bit more redundancy, so more on that in the next post.

As you might have seen in the news or from the US CERT, there’s an internal counter in the GPS messages that will “rollover” this week.

Poorly implemented GPS receivers might lose track of time because of this.

Many servers in the NTP Pool are using GPS signals to set the time, either directly or indirectly from another server that listens for GPS signals.

The expected impact on users of the NTP Pool is very little or none at all. The NTP Pool monitoring system will detect any systems that are wildly off and have them removed from DNS responses within typically 10-20 minutes.

Most servers would also recognize that the “jump” in time from a potentially misbehaving GPS clock is too large and ignore it (the time would be set back to August 1999, most likely).

Most importantly, reasonably well written clients will poll 3 or 4 IPs and ignore the one that’s an outlier by decades, recognize that the time returned is many years older than the earliest possible manufacturing date of itself or that the jump in time is too large and again ignore it.

The time of the rollover is midnight UTC on April 7 (so late afternoon April 6th in California or the morning of April 7 in Asia). We will be checking the NTP Pool monitoring data for any signs of problems through the day.

See also the NTP Pool community discussion.

Daniel Ziegenberg wrote a tutorial for Digital Ocean on configuring NTP for the NTP Pool on Ubuntu.

Oliver Nadler has another tutorial covering non-Ubuntu, too.

There’s a new forum for discussion related to the NTP Pool at community.ntppool.org. Please come join us. There are a couple interesting threads about the recent leap second and lots of empty space for your questions or suggestions. :-)

Continue Reading →

Since last Tuesday some countries have seen an excessive number of queries to the NTP Pool.

After much detective work on nanog (conclusion) and the #NTP IRC channel it was determined to be a buggy Snapchat app update.

Continue Reading →