Give hosts longer before ticketing them as gone
Lint / JS (eslint) (push) Successful in 16s
Lint / Python (flake8) (push) Successful in 1m23s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 1m49s
Lint / Python (flake8) (pull_request) Successful in 54s
Test / Python Tests (pytest) (push) Successful in 1m35s
Lint / JS (eslint) (pull_request) Successful in 12s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 1m42s
Test / Python Tests (pytest) (pull_request) Successful in 2m3s

A host's node_exporter stops as soon as a shutdown begins and a normal
reboot takes about 3-5 minutes, so the 2-check failure_threshold cut a
ticket for planned reboots. Add monitor.host_failure_threshold, default 5,
for hosts with no metrics and ping-only hosts. Interface link-down alerts
keep the shorter threshold, and the dashboard still shows the host as down
straight away.

Closes #14

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
jaredandClaude Opus 5.5 committed 2026-11-06 19:35:50 -05:00
1 parent 7c917969d8
commit 178a03e898
3 files changed
+35 -5

No files matched your search

+4 -2
View File
@@ -200,8 +200,8 @@ entry in `FACEPLATES`.
|---|---|
| UniFi device offline (≥`unifi_failure_threshold` consecutive checks) | P2 High |
| Proxmox host NIC link-down regression (≥2 consecutive checks) | P2 High |
| Host unreachable via ping (≥2 consecutive checks) | P2 High |
| Configured host has no metrics in Prometheus (host or `node_exporter` down, ≥2 consecutive checks) | P2 High |
| Host unreachable via ping (≥`host_failure_threshold` consecutive checks) | P2 High |
| Configured host has no metrics in Prometheus (host or `node_exporter` down, ≥`host_failure_threshold` consecutive checks) | P2 High |
| Prometheus unavailable, so host and interface alerting is paused (≥2 consecutive checks) | P2 High |
| ≥3 hosts simultaneously reporting interface or metrics failures | P1 Critical |
| WAN down: broadband or optical status not Up, or PON not in O5 (≥`modem.failure_threshold`) | P1 Critical |
@@ -293,6 +293,7 @@ Shared by both processes. Located in the working directory (`/var/www/html/prod/
"monitor": {
"poll_interval": 120,
"failure_threshold": 2,
"host_failure_threshold": 5,
"unifi_failure_threshold": 5,
"cluster_threshold": 3,
"ping_hosts": [
@@ -323,6 +324,7 @@ Shared by both processes. Located in the working directory (`/var/www/html/prod/
| `hosts` | Maps Prometheus instance labels → display hostnames |
| `monitor.poll_interval` | Seconds between full check cycles (default: 120) |
| `monitor.failure_threshold` | Consecutive failures before creating ticket (default: 2) |
| `monitor.host_failure_threshold` | Consecutive failures before ticketing a host that has disappeared, whether it has no metrics or doesn't answer ping (default: 5). A normal reboot takes about 3-5 minutes, so the default rides it out: the host still shows as down on the dashboard, but a ticket opens only after about 8 minutes at the default poll interval. For longer maintenance, suppress the host first. |
| `monitor.unifi_failure_threshold` | Consecutive failures before ticketing a UniFi device as offline (default: same as `failure_threshold`). Set higher than `failure_threshold` — UniFi switches/APs can take several minutes to rejoin after a firmware-update reboot, so a low threshold cuts tickets for momentary maintenance reboots instead of real failures. |
| `monitor.cluster_threshold` | Hosts with failures to trigger cluster-wide P1 (default: 3) |
| `monitor.ping_hosts` | Hosts checked only by ping (no node_exporter) |