DNS server in Go - Big NTP Pool upgrade
Over the last month the NTP Pool has gotten the biggest upgrade it has had in years. The changes has given us much more scalability and performance.
As you might know, the NTP Pool system is essentially a monitoring system and a smart DNS server. Server operators register their server in the system, the monitoring system checks and evaluates the submitted servers and the DNS server gives end-users a (hopefully) local selection of servers, weighted by preferences given by the server operator and other factors.
Last month there was a big change to the DNS server.
Perl implementation of the DNS server
The DNS server started in 2001 as a fork of lbnamed. A couple years later it was completely rewritten and now based on Net::DNS::Nameserver. It was used for the CPAN Search site and some other sites hosted by perl.org.
It’s been serving pool.ntp.org DNS since 2007 after I added a ‘JSON data’ feature to make it easier to configure thousands of servers automatically and many automated tests. Over the last years it’s been getting regular improvements.
The biggest (only, really) problem with the Net::DNS::Nameserver based server was the performance. Between the more than twenty name servers, the system could very comfortably handle the regular traffic flow of 4-5000 queries per second, but at certain times of the hour and day the request flow goes up significantly - 150,000 queries over 10 seconds or so; most of those within a couple of seconds. Woah! (This is also why you really really shouldn’t use ’ntpdate’ from cron at a certain time each day).
Net::DNS::Nameserver-based servers are single-threaded and blocking, so there’s only so much that can (easily) be done to make it faster. I considered adapting the code to run as a PowerDNS “pipe-backend”, but to make it fast would take some changes to PowerDNS and I didn’t look forward to the increased deployment complexity.
New Golang-based server
Enter Go. I’d been tinkering a little with Go when it was announced first, but not written any real software with it. In August I spent a few weekend days researching the language eco-system (it’s no CPAN, but it’s growing quickly and generally has good code), took the Go tour and then worked on writing code to parse the pgeodns configuration files.
Within a couple weekends, I had a basic server running using the excellent go dns library written by Miek Gieben
I was surprised how quickly I made progress (and how fun it was to write in Go). The documentation and tools1 are very good. It compiles fast and gives reasonably good error messages and has consistent code formatting that’s sane enough.
Because I made the new server read the configuration files from the old system, we2 could slowly replace the software a server at a time and know that it should give the same results (just faster).
Since mid-September, twenty of the name servers have been running the new software and except for a bit of trouble on i386 (32-bit) and low-memory systems, it’s been running very smoothly.
Only one bug has been pointed out to me over the last 30 billion requests or so, but maybe bringing attention to it with this point will find a few more! :-)
The code is on github as well as the issue tracker. Bug reports and patches are welcome.
There’s some discussion on reddit and hacker news.
1. I use Sublime Text with the GoSublime package to edit the code, which integrates my favorite feature: ‘go fmt’. ↩
2. Guillaume Filion helps me manage the donated server resources that run the DNS servers. ↩