NTP Pool News

As you might have seen a few days ago several potentially critical security vulnerabilities in all versions of ntpd were announced.

Most OS'es have released back-ported fixes. Depending on your specific ntp and network configuration you might not be exposed, but the easiest way to make sure your systems aren't vulnerable is to apply the software updates and make sure ntpd has restarted on the fixed version.

Alternatively you can read the announcement page linked above carefully and make configuration changes to mitigate the issues.

If you have built ntpd from source, the easiest fix is to update to 4.2.8. If you have trouble building that version, there's a "4.2.8p1-beta1" version available now from support.ntp.org as well with some fixes.

If you aren't already subscribed then you might be interested in subscribing to the NTP Pool discussion mailing list. For general discussion of NTP there's the comp.protocols.time.ntp newsgroup.

If you are using the standard ntpd daemon to serve time to the public internet, it’s important that you make sure it is configured to not reply to “monlist” queries. Many routers and other equipment are included in this.

The configuration recommendations include the appropriate “restrict” lines to disallow any management queries to ntpd. Most Linux distributions will have an updated version by now that just disables the “monlist” queries, that will also solve the primary problem.

The NTP Support wiki has more information.

If your server is part of the NTP Pool, the system also does a periodic check if your server responds to mode6/mode7 information queries and will warn you on the manage page if it does.

If you operate a network you can use the Open NTP Project to see if you have vulnerable devices on your network.

This week we had a period of weird behavior for the monitoring system for (mostly) German IPv6 servers.

After much back and forth on the mailing list and numerous debugging sessions we got this information from a network engineer at Hurricane Electric:

A bug was recently discovered in Force10 switches that cause unicast IPv6 NTP traffic to be erroneously broadcast to all ports. Due to this, there are currently access lists in place preventing some IPv6 NTP traffic from traversing the DECIX exchange, as it was causing a storm that generated nearly 1 terabit per second of traffic. This should be resolved in the near future.

The number of IPv6 servers active in the pool appears to be about back to normal.

Also this is the answer to "why don't we have IPv6 servers by default on all the pool zones" yet. As you might know only "2.pool.ntp.org" (and 2.debian.pool.ntp.org, etc) returns AAAA records currently.

The NTP Pool "backend systems" are moving racks at Phyber. To minimize the risk of things going wrong we're doing it the old-fashioned simple way of turning everything off, moving it and turning it on again. It will mean about an hour where servers are not monitored and we can't add new ones or access the www.pool.ntp.org site.

In the new rack there'll be more power available so when the move is done we'll have more capacity.

Over the last couple of months we had a couple of the "central servers" fail. It hasn't caused any service outage for the NTP clients, but some of you might have noticed that the manage NTP Pool site has been sluggish at times.

A few months ago I bought a few new servers and sent them down to our friends at Phyber Communications who wired them up in their hosting facility. Over the last weeks I've added puppet declarations) to configure them and since earlier this evening they're in production for the web sites and a few other services.

I have a long road map for the NTP Pool system and many of the items involve processing and storing more data to make our system better. The new servers are going to be helpful for that.

My other project for the months have been upgrades to the GeoDNS server to support EDNS-CLIENT-SUBNET. It has been live for users of Google DNS for a while. We're still working out some kinks with the OpenDNS folks to get it fully enabled there.

Over the last month the NTP Pool has gotten the biggest upgrade it has had in years. The changes has given us much more scalability and performance.

As you might know, the NTP Pool system is essentially a monitoring system and a smart DNS server. Server operators register their server in the system, the monitoring system checks and evaluates the submitted servers and the DNS server gives end-users a (hopefully) local selection of servers, weighted by preferences given by the server operator and other factors.

Last month there was a big change to the DNS server.

For years the geodns server has had a misconfiguration so users in Great Britain by default (accessing the non-country-code domain) would get a European server rather than a more local one.

The zone in the NTP Pool system has always been called ‘uk’, but the GeoIP library returns ‘gb’ for the relevant users. Oops! The system didn’t have a ‘gb’ zone configured, but knew it was in Europe so would fall back to that.

I fixed it about 3 weeks ago, so since then users in that region should be getting better service.

Related then servers registered in the ‘uk’ zone will have seen their traffic go up considerably. If you need to adjust how much traffic your server gets, you can adjust the netspeed on the manage site.

To safely upgrade some of the DNS configuration infrastructure updates to the DNS data will be suspended for 20-45 minutes. Some parts of the website might also return errors while everything is being updated.

For end-users of the pool there should be no interruption.

Update Maintenance was completed in 20 minutes. The changes were in part to get ready to deploy a new Go based DNS server to replace the current DNS server.

Meinberg have since long generously been supporting the NTP Pool and other open source projects. The monitoring system uses a Meinberg NTP server for "reference time" when checking the more than 3000 servers in the pool. I can't recommend their equipment or expertise enough.

This month they are giving away in a raffle seven DCF77 computer clocks and three GPS time receivers to current and soon-to-be participants in the NTP Pool.

The form and rules are short and simple, but the deadline is July 29th, so don't delay!

The client base for the NTP Pool continues to grow, so we also need to increase the number of servers. Being a "public utility" of sorts (you likely use it for some computer or device in your house, office or both even if you don't know it), we need help from, well, the public. At least the particular kind of public who is running a server or two with static IP addresses and know how to configure a new daemon on it.

There are several thousand and new ones are added regularly, however from natural attrition the total number of servers have been stagnating or even going down lately, even in Europe. Some countries still have very good coverage (Germany for example), but many others really could use more.

In Asia virtually all countries could use more servers, even or maybe in particular Japan, China and India. In South America there are virtually no servers outside Brazil.

Iceland recently joined the pool as a "full zone"; so far just with two servers.

More servers in any country are very welcome, but in particular in the countries with sparse coverage it'll be great to get more.