Give hosts longer before ticketing them as gone
Lint / JS (eslint) (push) Successful in 16s
Lint / Python (flake8) (push) Successful in 1m23s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 1m49s
Lint / Python (flake8) (pull_request) Successful in 54s
Test / Python Tests (pytest) (push) Successful in 1m35s
Lint / JS (eslint) (pull_request) Successful in 12s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 1m42s
Test / Python Tests (pytest) (pull_request) Successful in 2m3s
Lint / JS (eslint) (push) Successful in 16s
Lint / Python (flake8) (push) Successful in 1m23s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 1m49s
Lint / Python (flake8) (pull_request) Successful in 54s
Test / Python Tests (pytest) (push) Successful in 1m35s
Lint / JS (eslint) (pull_request) Successful in 12s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 1m42s
Test / Python Tests (pytest) (pull_request) Successful in 2m3s
A host's node_exporter stops as soon as a shutdown begins and a normal reboot takes about 3-5 minutes, so the 2-check failure_threshold cut a ticket for planned reboots. Add monitor.host_failure_threshold, default 5, for hosts with no metrics and ping-only hosts. Interface link-down alerts keep the shorter threshold, and the dashboard still shows the host as down straight away. Closes #14 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
7c917969d8
commit
178a03e898
3 files changed
+35
-5
No files matched your search
@@ -200,8 +200,8 @@ entry in `FACEPLATES`.
|
||||
|---|---|
|
||||
| UniFi device offline (≥`unifi_failure_threshold` consecutive checks) | P2 High |
|
||||
| Proxmox host NIC link-down regression (≥2 consecutive checks) | P2 High |
|
||||
| Host unreachable via ping (≥2 consecutive checks) | P2 High |
|
||||
| Configured host has no metrics in Prometheus (host or `node_exporter` down, ≥2 consecutive checks) | P2 High |
|
||||
| Host unreachable via ping (≥`host_failure_threshold` consecutive checks) | P2 High |
|
||||
| Configured host has no metrics in Prometheus (host or `node_exporter` down, ≥`host_failure_threshold` consecutive checks) | P2 High |
|
||||
| Prometheus unavailable, so host and interface alerting is paused (≥2 consecutive checks) | P2 High |
|
||||
| ≥3 hosts simultaneously reporting interface or metrics failures | P1 Critical |
|
||||
| WAN down: broadband or optical status not Up, or PON not in O5 (≥`modem.failure_threshold`) | P1 Critical |
|
||||
@@ -293,6 +293,7 @@ Shared by both processes. Located in the working directory (`/var/www/html/prod/
|
||||
"monitor": {
|
||||
"poll_interval": 120,
|
||||
"failure_threshold": 2,
|
||||
"host_failure_threshold": 5,
|
||||
"unifi_failure_threshold": 5,
|
||||
"cluster_threshold": 3,
|
||||
"ping_hosts": [
|
||||
@@ -323,6 +324,7 @@ Shared by both processes. Located in the working directory (`/var/www/html/prod/
|
||||
| `hosts` | Maps Prometheus instance labels → display hostnames |
|
||||
| `monitor.poll_interval` | Seconds between full check cycles (default: 120) |
|
||||
| `monitor.failure_threshold` | Consecutive failures before creating ticket (default: 2) |
|
||||
| `monitor.host_failure_threshold` | Consecutive failures before ticketing a host that has disappeared, whether it has no metrics or doesn't answer ping (default: 5). A normal reboot takes about 3-5 minutes, so the default rides it out: the host still shows as down on the dashboard, but a ticket opens only after about 8 minutes at the default poll interval. For longer maintenance, suppress the host first. |
|
||||
| `monitor.unifi_failure_threshold` | Consecutive failures before ticketing a UniFi device as offline (default: same as `failure_threshold`). Set higher than `failure_threshold` — UniFi switches/APs can take several minutes to rejoin after a firmware-update reboot, so a low threshold cuts tickets for momentary maintenance reboots instead of real failures. |
|
||||
| `monitor.cluster_threshold` | Hosts with failures to trigger cluster-wide P1 (default: 3) |
|
||||
| `monitor.ping_hosts` | Hosts checked only by ping (no node_exporter) |
|
||||
|
||||
Reference in new issue
Block a user