As you might know, the NTP Pool system is essentially a monitoring system and a smart DNS server. Server operators register their server in the system, the monitoring system checks and evaluates the submitted servers and the DNS server gives end-users a (hopefully) local selection of servers, weighted by preferences given by the server operator and other factors.

Last month there was a big change to the DNS server.

The DNS server started in 2001 as a fork of lbnamed. A couple years later it was completely rewritten and now based on Net::DNS::Nameserver. It was used for the CPAN Search site and some other sites hosted by perl.org.

It's been serving pool.ntp.org DNS since 2007 after I added a 'JSON data' feature to make it easier to configure thousands of servers automatically and many automated tests. Over the last years it's been getting regular improvements.

The biggest (only, really) problem with the Net::DNS::Nameserver based server was the performance. Between the more than twenty name servers, the system could very comfortably handle the regular traffic flow of 4-5000 queries per second, but at certain times of the hour and day the request flow goes up significantly - 150,000 queries over 10 seconds or so; most of those within a couple of seconds. Woah! (This is also why you really really shouldn't use 'ntpdate' from cron at a certain time each day).

Net::DNS::Nameserver-based servers are single-threaded and blocking, so there's only so much that can (easily) be done to make it faster. I considered adapting the code to run as a PowerDNS "pipe-backend", but to make it fast would take some changes to PowerDNS and I didn't look forward to the increased deployment complexity.

New Golang-based server

Enter Go. I'd been tinkering a little with Go when it was announced first, but not written any real software with it. In August I spent a few weekend days researching the language eco-system (it's no CPAN, but it's growing quickly and generally has good code), took the Go tour and then worked on writing code to parse the pgeodns configuration files.

Within a couple weekends, I had a basic server running using the excellent go dns library written by Miek Gieben

I was surprised how quickly I made progress (and how fun it was to write in Go). The documentation and tools₁ are very good. It compiles fast and gives reasonably good error messages and has consistent code formatting that's sane enough.

Because I made the new server read the configuration files from the old system, we₂ could slowly replace the software a server at a time and know that it should give the same results (just faster).

Since mid-September, twenty of the name servers have been running the new software and except for a bit of trouble on i386 (32-bit) and low-memory systems, it's been running very smoothly.

Only one bug has been pointed out to me over the last 30 billion requests or so, but maybe bringing attention to it with this point will find a few more! :-)

The code is on github as well as the issue tracker. Bug reports and patches are welcome.

There's some discussion on reddit and hacker news.

1. I use Sublime Text with the GoSublime package to edit the code, which integrates my favorite feature: 'go fmt'. ↩

2. Guillaume Filion helps me manage the donated server resources that run the DNS servers. ↩

There are several thousand and new ones are added regularly, however from natural attrition the total number of servers have been stagnating or even going down lately, even in Europe. Some countries still have very good coverage (Germany for example), but many others really could use more.

In Asia virtually all countries could use more servers, even or maybe in particular Japan, China and India. In South America there are virtually no servers outside Brazil.

Iceland recently joined the pool as a "full zone"; so far just with two servers.

More servers in any country are very welcome, but in particular in the countries with sparse coverage it'll be great to get more.

The downside is that it is IPv4 only (currently), but then all the “static assets” (CSS files, images, etc) were already served by them, so using the site with only IPv6 was not a good experience.

Fastly is also hosting Perldoc.perl.org and have been doing so for a while.

Anyway, while the experiment is ongoing, accessing the pool site should be even faster than before, in particular for those of you who are in Europe or in the eastern US.

The old RRD based graphs (deprecated a few months ago) are still being used a bit, but really don't play well with having multiple servers. Over the last week I changed the site to generate the offset and score graphs via the same D3.js based system and PhantomJS.

As you can see the score and the offset log are in one graph now, where the old graphs unhelpfully had separate charts for that. If you are using both on an internal dashboard, please update to just use the offset.png version.

Downloading the .rrd files don't work anymore, but instead you can fetch the recent monitoring data in JSON format (in addition to the old CSV format), for example http://www.pool.ntp.org/scores/207.171.7.152/json. If you really want the data in RRD format, you can easily take either of these to generate it yourself. Both log formats support a "since" parameter (taking a unix timestamp, for example since= 1339132534) to only show data points after that time.

I'll also add a javascript widget version that you can embed on your own pages.

So far no problems have been reported, even the servers going through IPv6 tunnels seem to work fine.

We already have almost 200 IPv6 servers in the pool; though that's less than 10% of all the pool servers and they're mostly in a handful of countries versus the wide deployment we have for IPv4.

The pool system is tracking, but not yet showing separate stats for IPv6 servers by default. (You can however "hack" the graph URLs to show them by adding -v6 to the file name, for example the European counts.

You can see the growth in IPv6 servers available in the pool on the zone pages.

In addition all IPv6 servers are being monitored now and are slowly increasing their scores for inclusion in the "2." pool!

June 8th is World IPv6 Day where many sites small and big will enable the IPv6 protocol for the day to help test everyones IPv6 readiness.

Here at the NTP pool we are today enabling monitoring of IPv6 servers; and over the next 12 hours we will start in a limited fashion to serve AAAA (IPv6) DNS records to clients asking for them. Right now the pool site is enjoying a brief break while the new code and database updates are being deployed.

For now this will only be on the "2.pool.ntp.org" zone; but unlike Google, Facebook and most other sites we are not planning to turn this off again on June 9th.