Planned host reboots cut "No metrics from host" tickets #14

New Issue

2026-11-06T18:56:22-05:00

jared commented

2026-11-06 18:56:22 -05:00

What happened

A kernel update followed by a reboot of micro1 from the Proxmox UI cut a P2 ticket, 993441773 ("No metrics from host"). The reboot was intentional and the host came back healthy.

Times are UTC on 2026-11-06:

Time	Event
23:17:50	`systemd-logind` logs "The system will reboot now!"
~23:18	Prometheus `up` goes to 0, because `node_exporter` stops at the start of shutdown
23:21:11	New kernel boot time
23:21:57	Gandalf opens the ticket on its second failed check
~23:22	`up` returns to 1

Why

host_metrics_missing uses monitor.failure_threshold, which defaults to 2 checks. With a 120s poll, that can fire as soon as 2 minutes after the host disappears.

Gandalf has no way to know a reboot is planned unless a suppression is added first. The unifi_failure_threshold and modem.failure_threshold settings already give those devices extra time for maintenance reboots, but hosts have no equivalent.

Reboots measured from Prometheus up samples over the last 7 days:

Host	Time with `up` = 0
`compute-storage-01`	~3 min
`compute-storage-gpu-01`	~4 min
`micro1`	~4 min
`monitor-02`	~5 min

Ping-only hosts (host_unreachable) use the same threshold, so they have the same problem.

Proposed fix

Add monitor.host_failure_threshold, used when a whole host disappears (host_metrics_missing and host_unreachable). Default it to 5 checks, so a ticket opens no sooner than about 8 minutes after the first failed check.

Interface link-down alerts stay at failure_threshold. A NIC going down on a running host is still a fast, real signal.
The dashboard and event list still show the host as down straight away. Only the ticket waits.
The cluster-wide alert still counts a missing host from its first failed check.

Trade-offs

A real host outage is ticketed about 6 minutes later than it is today.
This is a time window, not intent detection. A crash that recovers within the window also won't cut a ticket, though the event is still recorded.
A reboot that takes longer than the window still cuts a ticket. Suppress the host first for long maintenance.

Possible follow-ups (not in this fix)

Intent detection: clean reboots do leave a signal, such as the systemd-logind message and the Proxmox stopall task in Loki. Gandalf could use it to tell a planned reboot from a crash, and to ticket crashes that recover quickly.
Closing tickets on recovery: the event resolves when a host returns, but its ticket stays open until someone closes it.
Cluster alert threshold: cluster_network_issue has no consecutive-check threshold, so one bad check across 3 hosts opens a P1 straight away. Suppressed hosts count toward it too, so a planned reboot of 3 suppressed hosts could still open one.

## What happened A kernel update followed by a reboot of `micro1` from the Proxmox UI cut a P2 ticket, [993441773](https://t.lotusguild.org/ticket/993441773) ("No metrics from host"). The reboot was intentional and the host came back healthy. Times are UTC on 2026-11-06: | Time | Event | |---|---| | 23:17:50 | `systemd-logind` logs "The system will reboot now!" | | ~23:18 | Prometheus `up` goes to 0, because `node_exporter` stops at the start of shutdown | | 23:21:11 | New kernel boot time | | 23:21:57 | Gandalf opens the ticket on its second failed check | | ~23:22 | `up` returns to 1 | ## Why `host_metrics_missing` uses `monitor.failure_threshold`, which defaults to 2 checks. With a 120s poll, that can fire as soon as 2 minutes after the host disappears. Gandalf has no way to know a reboot is planned unless a suppression is added first. The `unifi_failure_threshold` and `modem.failure_threshold` settings already give those devices extra time for maintenance reboots, but hosts have no equivalent. Reboots measured from Prometheus `up` samples over the last 7 days: | Host | Time with `up` = 0 | |---|---| | `compute-storage-01` | ~3 min | | `compute-storage-gpu-01` | ~4 min | | `micro1` | ~4 min | | `monitor-02` | ~5 min | Ping-only hosts (`host_unreachable`) use the same threshold, so they have the same problem. ## Proposed fix Add `monitor.host_failure_threshold`, used when a whole host disappears (`host_metrics_missing` and `host_unreachable`). Default it to 5 checks, so a ticket opens no sooner than about 8 minutes after the first failed check. - Interface link-down alerts stay at `failure_threshold`. A NIC going down on a running host is still a fast, real signal. - The dashboard and event list still show the host as down straight away. Only the ticket waits. - The cluster-wide alert still counts a missing host from its first failed check. ## Trade-offs - A real host outage is ticketed about 6 minutes later than it is today. - This is a time window, not intent detection. A crash that recovers within the window also won't cut a ticket, though the event is still recorded. - A reboot that takes longer than the window still cuts a ticket. Suppress the host first for long maintenance. ## Possible follow-ups (not in this fix) - **Intent detection:** clean reboots do leave a signal, such as the `systemd-logind` message and the Proxmox `stopall` task in Loki. Gandalf could use it to tell a planned reboot from a crash, and to ticket crashes that recover quickly. - **Closing tickets on recovery:** the event resolves when a host returns, but its ticket stays open until someone closes it. - **Cluster alert threshold:** `cluster_network_issue` has no consecutive-check threshold, so one bad check across 3 hosts opens a P1 straight away. Suppressed hosts count toward it too, so a planned reboot of 3 suppressed hosts could still open one.

jared referenced this issue from a commit

2026-11-06 19:35:54 -05:00

Give hosts longer before ticketing them as gone

jared referenced a pull request that will close this issue

2026-11-06 19:35:57 -05:00

Give hosts longer before ticketing them as gone #15

jared closed this issue

2026-11-06 19:41:46 -05:00

Sign in to join this conversation.