NTP Pool News

This week we had a period of weird behavior for the monitoring system for (mostly) German IPv6 servers.

After much back and forth on the mailing list and numerous debugging sessions we got this information from a network engineer at Hurricane Electric:

A bug was recently discovered in Force10 switches that cause unicast IPv6 NTP traffic to be erroneously broadcast to all ports. Due to this, there are currently access lists in place preventing some IPv6 NTP traffic from traversing the DECIX exchange, as it was causing a storm that generated nearly 1 terabit per second of traffic. This should be resolved in the near future.

The number of IPv6 servers active in the pool appears to be about back to normal.

Also this is the answer to "why don't we have IPv6 servers by default on all the pool zones" yet. As you might know only "2.pool.ntp.org" (and 2.debian.pool.ntp.org, etc) returns AAAA records currently.

The NTP Pool "backend systems" are moving racks at Phyber. To minimize the risk of things going wrong we're doing it the old-fashioned simple way of turning everything off, moving it and turning it on again. It will mean about an hour where servers are not monitored and we can't add new ones or access the www.pool.ntp.org site.

In the new rack there'll be more power available so when the move is done we'll have more capacity.

Over the last couple of months we had a couple of the "central servers" fail. It hasn't caused any service outage for the NTP clients, but some of you might have noticed that the manage NTP Pool site has been sluggish at times.

A few months ago I bought a few new servers and sent them down to our friends at Phyber Communications who wired them up in their hosting facility. Over the last weeks I've added puppet declarations) to configure them and since earlier this evening they're in production for the web sites and a few other services.

I have a long road map for the NTP Pool system and many of the items involve processing and storing more data to make our system better. The new servers are going to be helpful for that.

My other project for the months have been upgrades to the GeoDNS server to support EDNS-CLIENT-SUBNET. It has been live for users of Google DNS for a while. We're still working out some kinks with the OpenDNS folks to get it fully enabled there.

Over the last month the NTP Pool has gotten the biggest upgrade it has had in years. The changes has given us much more scalability and performance.

As you might know, the NTP Pool system is essentially a monitoring system and a smart DNS server. Server operators register their server in the system, the monitoring system checks and evaluates the submitted servers and the DNS server gives end-users a (hopefully) local selection of servers, weighted by preferences given by the server operator and other factors.

Last month there was a big change to the DNS server.

For years the geodns server has had a misconfiguration so users in Great Britain by default (accessing the non-country-code domain) would get a European server rather than a more local one.

The zone in the NTP Pool system has always been called ‘uk’, but the GeoIP library returns ‘gb’ for the relevant users. Oops! The system didn’t have a ‘gb’ zone configured, but knew it was in Europe so would fall back to that.

I fixed it about 3 weeks ago, so since then users in that region should be getting better service.

Related then servers registered in the ‘uk’ zone will have seen their traffic go up considerably. If you need to adjust how much traffic your server gets, you can adjust the netspeed on the manage site.

To safely upgrade some of the DNS configuration infrastructure updates to the DNS data will be suspended for 20-45 minutes. Some parts of the website might also return errors while everything is being updated.

For end-users of the pool there should be no interruption.

Update Maintenance was completed in 20 minutes. The changes were in part to get ready to deploy a new Go based DNS server to replace the current DNS server.

Meinberg have since long generously been supporting the NTP Pool and other open source projects. The monitoring system uses a Meinberg NTP server for "reference time" when checking the more than 3000 servers in the pool. I can't recommend their equipment or expertise enough.

This month they are giving away in a raffle seven DCF77 computer clocks and three GPS time receivers to current and soon-to-be participants in the NTP Pool.

The form and rules are short and simple, but the deadline is July 29th, so don't delay!

The client base for the NTP Pool continues to grow, so we also need to increase the number of servers. Being a "public utility" of sorts (you likely use it for some computer or device in your house, office or both even if you don't know it), we need help from, well, the public. At least the particular kind of public who is running a server or two with static IP addresses and know how to configure a new daemon on it.

There are several thousand and new ones are added regularly, however from natural attrition the total number of servers have been stagnating or even going down lately, even in Europe. Some countries still have very good coverage (Germany for example), but many others really could use more.

In Asia virtually all countries could use more servers, even or maybe in particular Japan, China and India. In South America there are virtually no servers outside Brazil.

Iceland recently joined the pool as a "full zone"; so far just with two servers.

More servers in any country are very welcome, but in particular in the countries with sparse coverage it'll be great to get more.

Today I am experimenting with hosting www.pool.ntp.org through Fastly. If you don’t know about them, they make an excellent CDN based on Varnish serving billions of requests a day.

The downside is that it is IPv4 only (currently), but then all the “static assets” (CSS files, images, etc) were already served by them, so using the site with only IPv6 was not a good experience.

Fastly is also hosting Perldoc.perl.org and have been doing so for a while.

Anyway, while the experiment is ongoing, accessing the pool site should be even faster than before, in particular for those of you who are in Europe or in the eastern US.

Some months ago our friends at Phyber setup a few more servers for the NTP Pool project. Over the last months I've been moving services to them to make the NTP Pool infrastructure run faster and with more redundancy.

The old RRD based graphs (deprecated a few months ago) are still being used a bit, but really don't play well with having multiple servers. Over the last week I changed the site to generate the offset and score graphs via the same D3.js based system and PhantomJS.

As you can see the score and the offset log are in one graph now, where the old graphs unhelpfully had separate charts for that. If you are using both on an internal dashboard, please update to just use the offset.png version.

Downloading the .rrd files don't work anymore, but instead you can fetch the recent monitoring data in JSON format (in addition to the old CSV format), for example http://www.pool.ntp.org/scores/207.171.7.152/json. If you really want the data in RRD format, you can easily take either of these to generate it yourself. Both log formats support a "since" parameter (taking a unix timestamp, for example since= 1339132534) to only show data points after that time.

I'll also add a javascript widget version that you can embed on your own pages.