Monitoring system /
Technical details
This describes the monitoring system put in production March 2023.
Monitoring servers
Servers registered in the NTP Pool system are monitored by monitoring servers around the world.
The monitoring servers themselves can be in “testing” or “active” modes. “Testing” servers will only do a limited number of probes.
For each NTP server 5 “active” monitors are chosen as “active” for that server.
Selector
The selector re-evaluates every 20 minutes which monitors should be “active” for each server.
This is done via the GetMonitorPriority query (the link might have changed when you read this) and the code in the selector linked above. The criteria is primarily “does this monitor consider the server healthy?” to prioritize monitoring from monitors that have good network connectivity to the NTP server.
Among “healthy” monitor/server pairs monitors are chosen based on the recent test results and the network round trip time to the server.
When the selector changes the active monitors for a server it will wait an hour before making changes again. In most cases at most one “active” monitor is replaced per evaluation run.
Scorer
After each monitoring result a “step” is calculated and applied to the score for that monitor / server pair (the ‘1-score’).
The new scoring calculation is called recent median. It works simply by choosing the median score of the ‘1-scores’ from “active” monitors in the last 20 minutes. (GetScorerRecentScores). In testing this has shown to be very effective at avoiding impact from a few errant measurements, and still react quickly enough to server or network trouble (thanks to Miroslav Lichvar for this idea).
The scoring system is also calculating the “old” score (named ’legacy’ in the UI), basically a ‘1-score’ across all the monitors. This score isn’t used for server health, but just maintained as a comparison. The score based on this calculation typically increase much faster, but is also susceptible to more erratic swings based on single monitoring probes.