Stop creating network tickets; hand network alerting to gandalf #30

New Issue

2026-09-25T18:37:25-04:00

jared commented

2026-09-25 18:37:25 -04:00

Problem

_check_network_status() (hwmonDaemon.py ~L3368) sends one ping (PING_COUNT: 1, PING_TIMEOUT: 1) from the node to 10.10.10.1 / 10.10.90.1. _detect_issues() (~L2232) turns any failure into a P2 ticket straight away. Nothing is carried between hourly runs and there's only one vantage point, so one dropped ICMP packet looks the same as a real outage.

Example: https://t.lotusguild.org/ticket/707736148 (compute-storage-gpu-01, 2026-09-22 07:01:09, "Management network is unreachable"). The node's journal shows no link, r8169/mlx4 or corosync/knet events from 06:55 to 07:05, and the next hourly run was OK.

Change

Remove management_network / ceph_network from _detect_issues() so hwmonDaemon no longer opens network tickets.
Keep the check as data only: ping with more packets (-c 5 -W 2), record the loss % and RTT (fills the latency field, currently always None), and write both to the health report / exporter so gandalf can use it as one of its sources.
Network ticketing moves to gandalf (see the gandalf issue).

Acceptance

hwmonDaemon never creates a network or reachability ticket.
Network loss/RTT is still in the health report for other tools to read.

## Problem `_check_network_status()` (hwmonDaemon.py ~L3368) sends one ping (`PING_COUNT: 1`, `PING_TIMEOUT: 1`) from the node to 10.10.10.1 / 10.10.90.1. `_detect_issues()` (~L2232) turns any failure into a P2 ticket straight away. Nothing is carried between hourly runs and there's only one vantage point, so one dropped ICMP packet looks the same as a real outage. Example: https://t.lotusguild.org/ticket/707736148 (compute-storage-gpu-01, 2026-09-22 07:01:09, "Management network is unreachable"). The node's journal shows no link, r8169/mlx4 or corosync/knet events from 06:55 to 07:05, and the next hourly run was OK. ## Change - Remove `management_network` / `ceph_network` from `_detect_issues()` so hwmonDaemon no longer opens network tickets. - Keep the check as data only: ping with more packets (`-c 5 -W 2`), record the loss % and RTT (fills the `latency` field, currently always `None`), and write both to the health report / exporter so gandalf can use it as one of its sources. - Network ticketing moves to gandalf (see the gandalf issue). ## Acceptance - hwmonDaemon never creates a network or reachability ticket. - Network loss/RTT is still in the health report for other tools to read.

jared commented

2026-09-25 18:37:31 -04:00

Counterpart: LotusGuild/gandalf#2 (gandalf takes over network ticketing, using several sources).

jared referenced this issue from LotusGuild/gandalf

2026-09-25 18:37:31 -04:00

Own all network tickets: check reachability from several sources (multiple Pulse workers) and require a sustained outage #2

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: LotusGuild/hwmonDaemon#30