Monitoring system /
Technical details
This describes the monitoring system put in production March 2023.
Monitoring servers
Servers registered in the NTP Pool system are monitored by monitoring servers around the world.
The monitoring servers themselves can be in “testing” or “active” modes. “Testing” servers will only do a limited number of probes.
For each NTP server 5 “active” monitors are chosen as “active” for that server.
Selector
The selector re-evaluates every 20 minutes which monitors should be “active” for each server.
This is done via the GetMonitorPriority query (the link might have changed when you read this) and the code in the selector linked above. The criteria is primarily “does this monitor consider the server healthy?” to prioritize monitoring from monitors that have good network connectivity to the NTP server.
Among “healthy” monitor/server pairs monitors are chosen based on the recent test results and the network round trip time to the server.
When the selector changes the active monitors for a server it will wait an hour before making changes again. In most cases at most one “active” monitor is replaced per evaluation run.
Scorer
After each monitoring result a “step” is calculated and applied to the score for that monitor / server pair (the ‘1-score’).
The new scoring calculation is called recent median. In the UI it’s listed as “overall score”. It works simply by choosing the median score of the ‘1-scores’ from “active” monitors in the last 20 minutes. (GetScorerRecentScores). In testing this has shown to be very effective at avoiding impact from a few errant measurements, and still react quickly enough to server or network trouble (thanks to Miroslav Lichvar for this idea).
If there aren’t any scores from “active” monitors in the last 20 minutes (for example if the server is new) the scores are calculated based on the most recent scores from all monitors in the last 45 minutes.
The score is calculated after every monitoring probe, but only recorded every 15 minutes or when it has changed.