Sub-optimal monitoring performance

Early this morning (PST) we had a few hours of "sub-optimal" performance on the monitoring server. A hundred servers or so were marked "bad" and got unnecessary warning mails because of it. users of the pool should not have been impacted. Work is in progress to permanently improve on this.