The NTP Pool "backend systems" are moving racks at Phyber. To minimize the risk of things going wrong we're doing it the old-fashioned simple way of turning everything off, moving it and turning it on again. It will mean about an hour where servers are not monitored and we can't add new ones or access the www.pool.ntp.org site.
In the new rack there'll be more power available so when the move is done we'll have more capacity.
Over the last couple of months we had a couple of the "central servers" fail. It hasn't caused any service outage for the NTP clients, but some of you might have noticed that the manage NTP Pool site has been sluggish at times.
A few months ago I bought a few new servers and sent them down to our friends at Phyber Communications who wired them up in their hosting facility. Over the last weeks I've added puppet declarations) to configure them and since earlier this evening they're in production for the web sites and a few other services.
I have a long road map for the NTP Pool system and many of the items involve processing and storing more data to make our system better. The new servers are going to be helpful for that.
My other project for the months have been upgrades to the GeoDNS server to support EDNS-CLIENT-SUBNET. It has been live for users of Google DNS for a while. We're still working out some kinks with the OpenDNS folks to get it fully enabled there.
Over the last month the NTP Pool has gotten the biggest upgrade it has had in years. The changes has given us much more scalability and performance.
As you might know, the NTP Pool system is essentially a monitoring system and a smart DNS server. Server operators register their server in the system, the monitoring system checks and evaluates the submitted servers and the DNS server gives end-users a (hopefully) local selection of servers, weighted by preferences given by the server operator and other factors.
Last month there was a big change to the DNS server.
For years the geodns server has had a misconfiguration so users in Great Britain by default (accessing the non-country-code domain) would get a European server rather than a more local one.
The zone in the NTP Pool system has always been called ‘uk’, but the GeoIP library returns ‘gb’ for the relevant users. Oops! The system didn’t have a ‘gb’ zone configured, but knew it was in Europe so would fall back to that.
I fixed it about 3 weeks ago, so since then users in that region should be getting better service.
Related then servers registered in the ‘uk’ zone will have seen their traffic go up considerably. If you need to adjust how much traffic your server gets, you can adjust the netspeed on the manage site.
To safely upgrade some of the DNS configuration infrastructure updates to the DNS data will be suspended for 20-45 minutes. Some parts of the website might also return errors while everything is being updated.
For end-users of the pool there should be no interruption.
Update Maintenance was completed in 20 minutes. The changes were in part to get ready to deploy a new Go based DNS server to replace the current DNS server.
Meinberg have since long generously been supporting the NTP Pool and other open source projects. The monitoring system uses a Meinberg NTP server for "reference time" when checking the more than 3000 servers in the pool. I can't recommend their equipment or expertise enough.
This month they are giving away in a raffle seven DCF77 computer clocks and three GPS time receivers to current and soon-to-be participants in the NTP Pool.
The form and rules are short and simple, but the deadline is July 29th, so don't delay!
The client base for the NTP Pool continues to grow, so we also need to increase the number of servers. Being a "public utility" of sorts (you likely use it for some computer or device in your house, office or both even if you don't know it), we need help from, well, the public. At least the particular kind of public who is running a server or two with static IP addresses and know how to configure a new daemon on it.
There are several thousand and new ones are added regularly, however from natural attrition the total number of servers have been stagnating or even going down lately, even in Europe. Some countries still have very good coverage (Germany for example), but many others really could use more.
In Asia virtually all countries could use more servers, even or maybe in particular Japan, China and India. In South America there are virtually no servers outside Brazil.
Iceland recently joined the pool as a "full zone"; so far just with two servers.
More servers in any country are very welcome, but in particular in the countries with sparse coverage it'll be great to get more.
Today I am experimenting with hosting www.pool.ntp.org through Fastly. If you don’t know about them, they make an excellent CDN based on Varnish serving billions of requests a day.
The downside is that it is IPv4 only (currently), but then all the “static assets” (CSS files, images, etc) were already served by them, so using the site with only IPv6 was not a good experience.
Fastly is also hosting Perldoc.perl.org and have been doing so for a while.
Anyway, while the experiment is ongoing, accessing the pool site should be even faster than before, in particular for those of you who are in Europe or in the eastern US.
Some months ago our friends at Phyber setup a few more servers for the NTP Pool project. Over the last months I've been moving services to them to make the NTP Pool infrastructure run faster and with more redundancy.
The old RRD based graphs (deprecated a few months ago) are still being used a bit, but really don't play well with having multiple servers. Over the last week I changed the site to generate the offset and score graphs via the same D3.js based system and PhantomJS.
As you can see the score and the offset log are in one graph now, where the old graphs unhelpfully had separate charts for that. If you are using both on an internal dashboard, please update to just use the offset.png version.
Downloading the .rrd files don't work anymore, but instead you can fetch the recent monitoring data in JSON format (in addition to the old CSV format), for example http://www.pool.ntp.org/scores/207.171.7.152/json. If you really want the data in RRD format, you can easily take either of these to generate it yourself. Both log formats support a "since" parameter (taking a unix timestamp, for example since= 1339132534) to only show data points after that time.
I'll also add a javascript widget version that you can embed on your own pages.
Today I updated the graphs from being generated by rrdtool to be generated with Javascript and SVG using the wonderful d3.js library. You can see an example of the new graphs or if you have a server in the pool you can use the new graphs.
There are a lot of things I wanted to do that using RRD made hard. Splitting the central parts of the system across more servers than the 2-3 it's running on now. Supporting multiple monitoring nodes. More interactive graphs. Showing historical data. Etcetera.
None of it impossible with RRDtool or for example it's cousin graphite, but those tools really just aren't a good match for our needs here.
The update today is just the beginning. Over on the "beta pool" we're already running with multiple monitors and on my development system I'll soon add some more features to the graphs so you can see the details of individual data points, go back in time a bit and so on. There are also a number of places on the site still where the old graphs are used even if your browser does support SVG.
I know many of you are using the old graphs on your own websites, ops charts, startpages and so on, and I intent do continue to maintain them at the current level for as long as possible. They're also supposed to show if you visit with a browser without Javascript and SVG support.
When the features and API have stabilized a bit, I'll also make a widget so you can put the D3 graphs on other sites and document the JSON API so you can use it to extract the data similar to the current CSV option.
Recent Comments