Own all network tickets: check reachability from several sources (multiple Pulse workers) and require a sustained outage #2

New Issue

2026-09-25T18:37:25-04:00

jared commented

2026-09-25 18:37:25 -04:00

Background

hwmonDaemon has been creating "Management network is unreachable" tickets from one ping on one node (e.g. t.lotusguild.org/ticket/707736148, a transient loss). That's being removed from hwmonDaemon, and gandalf becomes the single owner of network tickets.

gandalf already has what's needed to judge network state properly: Prometheus interface state, the UniFi port view, ethtool, a baseline, failure_threshold / unifi_failure_threshold, and cluster_threshold. What's missing:

Only one Pulse worker (pulse.worker_id). If that worker, or its path to the target, has a problem, it looks like the target is down.
No reachability checks. gandalf watches link state, but not "can host X reach the gateway / ceph net" or "can others reach host X".

Proposal

Multiple vantage points: make pulse.worker_id a list (pulse.workers: [...], at least 2–3 on different hosts). Run reachability probes (ping -c 5 -W 2) from each worker to each host's mgmt and ceph IPs. Optionally ingest hwmonDaemon's local gateway loss/RTT as another source.
Combine the sources and require agreement. Only call a host/network unreachable when:
- a majority of the Pulse workers agree (e.g. at least 2 of 3), and
- it's backed up by at least one other source (Prometheus up == 0 / interface down, or the UniFi port down / no link), and
- this holds for failure_threshold consecutive cycles, or at least N minutes (configurable, e.g. reachability_min_duration_s: 300).
Classify the result:
- Every vantage point loses one host → host/NIC problem → one P2 ticket for that host.
- Only one worker loses everything → problem at that worker; log it or give it a low-priority "vantage point degraded" ticket, not per-host tickets.
- Most hosts lost at once → one cluster-wide ticket (existing cluster_threshold path).
- Brief drops / carrier flaps below the duration threshold → record the event, no ticket. Optionally a low-priority "link flapping" ticket if there are more than X flaps in 24h (use carrier_changes).
Use the existing upsert_event / dedup / suppression logic so each outage produces exactly one ticket and resolves automatically.

Acceptance

One dropped ping or a short flap never creates a ticket.
A real host/NIC outage seen by most workers plus a second source, sustained past the threshold, creates exactly one ticket.
A failure of one Pulse worker doesn't cause host tickets.
Thresholds and worker list live in config.json.

## Background hwmonDaemon has been creating "Management network is unreachable" tickets from one ping on one node (e.g. t.lotusguild.org/ticket/707736148, a transient loss). That's being removed from hwmonDaemon, and gandalf becomes the single owner of network tickets. gandalf already has what's needed to judge network state properly: Prometheus interface state, the UniFi port view, ethtool, a baseline, `failure_threshold` / `unifi_failure_threshold`, and `cluster_threshold`. What's missing: 1. **Only one Pulse worker** (`pulse.worker_id`). If that worker, or its path to the target, has a problem, it looks like the target is down. 2. **No reachability checks.** gandalf watches link state, but not "can host X reach the gateway / ceph net" or "can others reach host X". ## Proposal - **Multiple vantage points:** make `pulse.worker_id` a list (`pulse.workers: [...]`, at least 2–3 on different hosts). Run reachability probes (`ping -c 5 -W 2`) from each worker to each host's mgmt and ceph IPs. Optionally ingest hwmonDaemon's local gateway loss/RTT as another source. - **Combine the sources and require agreement.** Only call a host/network unreachable when: - a majority of the Pulse workers agree (e.g. at least 2 of 3), **and** - it's backed up by at least one other source (Prometheus `up == 0` / interface down, or the UniFi port down / no link), **and** - this holds for `failure_threshold` consecutive cycles, or at least N minutes (configurable, e.g. `reachability_min_duration_s: 300`). - **Classify the result:** - Every vantage point loses one host → host/NIC problem → one P2 ticket for that host. - Only one worker loses everything → problem at that worker; log it or give it a low-priority "vantage point degraded" ticket, not per-host tickets. - Most hosts lost at once → one cluster-wide ticket (existing `cluster_threshold` path). - Brief drops / carrier flaps below the duration threshold → record the event, no ticket. Optionally a low-priority "link flapping" ticket if there are more than X flaps in 24h (use `carrier_changes`). - Use the existing `upsert_event` / dedup / suppression logic so each outage produces exactly one ticket and resolves automatically. ## Acceptance - One dropped ping or a short flap never creates a ticket. - A real host/NIC outage seen by most workers plus a second source, sustained past the threshold, creates exactly one ticket. - A failure of one Pulse worker doesn't cause host tickets. - Thresholds and worker list live in `config.json`.

jared referenced this issue from LotusGuild/hwmonDaemon

2026-09-25 18:37:31 -04:00

Stop creating network tickets; hand network alerting to gandalf #30

jared commented

2026-09-25 18:37:31 -04:00

Counterpart: LotusGuild/hwmonDaemon#30 (hwmonDaemon stops creating network tickets and keeps reporting loss/RTT as data).

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: LotusGuild/gandalf#2