Give hosts longer before ticketing them as gone #15

Merged

jared merged 1 commits from host-reboot-threshold into main 2026-11-06 19:41:45 -05:00

Conversation 0 Commits 1 Files Changed 3

+35 -5

jared commented 2026-11-06 19:35:57 -05:00

Owner

Copy Link

Copy Source

Closes #14

A planned reboot of micro1 cut a false "No metrics from host" ticket. node_exporter stops as soon as a shutdown begins, a normal reboot takes about 3-5 minutes, and the 2-check failure_threshold can fire after about 2 minutes.

Changes

• Add monitor.host_failure_threshold, default 5 checks, so a ticket opens at least about 8 minutes after the first failed check at the default poll interval. No config change is needed.
• Use it for hosts with no metrics (host_metrics_missing) and ping-only hosts (host_unreachable).
• Document the setting in the README.

Unchanged

• Interface link-down alerts stay at failure_threshold.
• The dashboard and event list still show the host as down straight away.
• The cluster-wide alert still counts a missing host from its first failed check.

Trade-offs

• A real host outage is ticketed about 6 minutes later.
• This is a longer wait, not intent detection: a crash that recovers inside the window records an event but cuts no ticket.

Testing

pytest (214 passed) and flake8 are clean. New tests cover a reboot-length outage cutting no ticket, ping hosts waiting out a reboot, and link-down alerts keeping the shorter threshold. It hasn't been exercised against a real reboot yet.

🤖 Generated with Claude Code

Closes #14 A planned reboot of `micro1` cut a false "No metrics from host" ticket. `node_exporter` stops as soon as a shutdown begins, a normal reboot takes about 3-5 minutes, and the 2-check `failure_threshold` can fire after about 2 minutes. ## Changes - Add `monitor.host_failure_threshold`, default 5 checks, so a ticket opens at least about 8 minutes after the first failed check at the default poll interval. No config change is needed. - Use it for hosts with no metrics (`host_metrics_missing`) and ping-only hosts (`host_unreachable`). - Document the setting in the README. ## Unchanged - Interface link-down alerts stay at `failure_threshold`. - The dashboard and event list still show the host as down straight away. - The cluster-wide alert still counts a missing host from its first failed check. ## Trade-offs - A real host outage is ticketed about 6 minutes later. - This is a longer wait, not intent detection: a crash that recovers inside the window records an event but cuts no ticket. ## Testing `pytest` (214 passed) and `flake8` are clean. New tests cover a reboot-length outage cutting no ticket, ping hosts waiting out a reboot, and link-down alerts keeping the shorter threshold. It hasn't been exercised against a real reboot yet. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

jared added 1 commit 2026-11-06 19:35:58 -05:00

Give hosts longer before ticketing them as gone ... 178a03e898

A host's node_exporter stops as soon as a shutdown begins and a normal
reboot takes about 3-5 minutes, so the 2-check failure_threshold cut a
ticket for planned reboots. Add monitor.host_failure_threshold, default 5,
for hosts with no metrics and ping-only hosts. Interface link-down alerts
keep the shorter threshold, and the dashboard still shows the host as down
straight away.

Closes #14

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

jared merged commit 93828b38e9 into main 2026-11-06 19:41:45 -05:00

jared deleted branch host-reboot-threshold 2026-11-06 19:41:46 -05:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

cole (Cole Keller) jared (Jared Vititoe) nathan (Nathan Vititoe)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: LotusGuild/gandalf#15