Give hosts longer before ticketing them as gone
Lint / JS (eslint) (push) Successful in 16s
Lint / Python (flake8) (push) Successful in 1m23s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 1m49s
Lint / Python (flake8) (pull_request) Successful in 54s
Test / Python Tests (pytest) (push) Successful in 1m35s
Lint / JS (eslint) (pull_request) Successful in 12s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 1m42s
Test / Python Tests (pytest) (pull_request) Successful in 2m3s

A host's node_exporter stops as soon as a shutdown begins and a normal
reboot takes about 3-5 minutes, so the 2-check failure_threshold cut a
ticket for planned reboots. Add monitor.host_failure_threshold, default 5,
for hosts with no metrics and ping-only hosts. Interface link-down alerts
keep the shorter threshold, and the dashboard still shows the host as down
straight away.

Closes #14

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-06 19:35:50 -05:00
co-authored by Claude Opus 5.5
parent 7c917969d8
commit 178a03e898
3 changed files with 35 additions and 5 deletions
+8 -3
View File
@@ -844,6 +844,10 @@ class NetworkMonitor:
# Use a separate, more forgiving threshold so those momentary reboots
# don't cut tickets, while still catching genuine device failures.
self.unifi_fail_thresh = mon.get('unifi_failure_threshold', self.fail_thresh)
# A host's exporter stops as soon as a shutdown begins and a normal reboot
# takes about 3-5 minutes, so wait longer than that before ticketing a host
# that has disappeared. A NIC going down on a running host stays on fail_thresh.
self.host_fail_thresh = mon.get('host_failure_threshold', 5)
self.cluster_thresh = mon.get('cluster_threshold', 3)
self.cluster_name = mon.get('cluster_name', CLUSTER_NAME)
@@ -940,7 +944,7 @@ class NetworkMonitor:
host, instance,
f'No metrics from {host} ({instance}): host or node_exporter is down ({_now_utc()})',
)
if not sup and consec >= self.fail_thresh and not ticket_id:
if not sup and consec >= self.host_fail_thresh and not ticket_id:
self._ticket_metrics_missing(event_id, host, instance, consec)
db.set_baseline(new_baseline)
@@ -1107,7 +1111,7 @@ class NetworkMonitor:
name, ip,
f'Host {name} ({ip}) unreachable via ping ({_now_utc()})',
)
if not sup and consec >= self.fail_thresh and not ticket_id:
if not sup and consec >= self.host_fail_thresh and not ticket_id:
self._ticket_unreachable(event_id, name, ip, consec)
else:
db.resolve_event('host_unreachable', name, ip)
@@ -1474,7 +1478,8 @@ class NetworkMonitor:
def run(self) -> None:
logger.info(
f'Gandalf monitor started – poll_interval={self.poll_interval}s '
f'fail_thresh={self.fail_thresh} unifi_fail_thresh={self.unifi_fail_thresh}'
f'fail_thresh={self.fail_thresh} host_fail_thresh={self.host_fail_thresh} '
f'unifi_fail_thresh={self.unifi_fail_thresh}'
)
while True:
try: