Aggregate alerts by root cause: one ticket per down host instead of one per symptom, plus auto-resolve #33

New Issue

2026-10-27T18:27:58-04:00

jared commented

2026-10-27 18:27:58 -04:00

Problem

When one server goes down, hwmonDaemon and the Tinker ticket API produce a flood of tickets, one per symptom. None of them close when the server comes back.

Evidence: compute-storage-01 outage, 2026-10-27

compute-storage-01 lost power at 11:53 EDT (PSU latched off; see Tinker #842416622) and was back at 15:20. That single event produced:

Ticket	Source	What it is
#910896002, #752244128, #788765526, #600144760, #337718870, #621550152, #618784458, #521122110, #498266194, #327710011	`ceph osd tree`	10 tickets, one per OSD (osd.0, 5, 7, 9, 16, 19, 25, 26, 29, 30), all on the same host
#906739396	HEALTH_WARN `OSD_HOST_DOWN`	"1 host (10 osds) down"
#170577775	HEALTH_WARN `OSD_DOWN`	"10 osds down". Reused an Open ticket from 2026-09-30 (an earlier large1 crash) that had never been closed
#842416622	HEALTH_WARN `MON_DOWN`	"1/3 mons down"
#237028956	HEALTH_WARN `MDS_INSUFFICIENT_STANDBY`	The standby MDS was on that host
#544996974	HEALTH_WARN `PG_DEGRADED`	Reused from 2026-09-30, also never closed

13 new tickets plus 2 stale ones updated, for one dead box. Every one of them was still Open hours after the host was back and Ceph had recovered. So were osd.12 and osd.14 (#569778912, #826445533) from the Sep 30 large1 crash, a month later.

A related symptom: #703260823 (closed SMART ticket) has 762 comments, all "Issue recurred, but the ticket could not be reopened automatically". One is added every hourly run, because there's no unattended Closed→Open workflow transition. (Recurrence handling is covered by #31; it's listed here because it's part of the same noise.)

Why it happens (current code)

hwmonDaemon

_check_ceph_health(): every down node in ceph osd tree becomes its own cluster_wide_issues entry: "Ceph OSD osd.N is DOWN".
The same function turns every HEALTH_WARN check into its own "Ceph HEALTH_WARN: <message>" issue. When a host is down, OSD_DOWN and OSD_HOST_DOWN repeat what the per-OSD entries already say, and MON_DOWN, MDS_INSUFFICIENT_STANDBY and PG_DEGRADED are consequences of the same event.
Nothing correlates symptoms to a host or root cause. The daemon never sends check_code (#29), so the API falls back to text matching.
No resolve path: the daemon only ever POSTs "this is happening", never "this stopped".

tinker_tickets create_ticket_api.php → generateTicketHash()

issue_subtype = 'osd_down_' . <id> for every OSD, so each OSD gets a distinct hash and its own ticket, by design.
Other HEALTH_WARNs hash on the warning text with counts stripped (warn_osds_down, warn_host_osds_down, warn_mons_down, ...). That's why today's "10 osds down" landed on the still-Open Sep 30 ticket.
No way to close a ticket from automation, and no parent/child grouping.

Proposal

1. Group symptoms under the down host (hwmonDaemon)

Before building tickets, work out whether a host is down, and fold its symptoms into one ticket for that host:

Host-down signals: ceph osd tree (all OSDs under a host bucket down), HEALTH_WARN OSD_HOST_DOWN, pvecm status / corosync membership (node missing), and optionally the new BMC data in Prometheus (bmc_host_power_on == 0, bmc_up == 0, from bmc-redfish-exporter in CT 118, see pve-infra).
One ticket per down host: [<host>][auto][hardware] Node down: <host> (N OSDs, mon, mds) [single-node][production][issue], priority P1 or P2. The description lists everything folded into it: OSD ids, mon/mgr/mds daemons on that host, HA guests recovered and where they went, Ceph degraded %, and the PG state.
Suppress the symptoms: while that host is down, don't create separate tickets for
- Ceph OSD osd.N is DOWN where osd.N is on that host
- OSD_DOWN / OSD_HOST_DOWN
- MON_DOWN naming that host's mon
- MDS_INSUFFICIENT_STANDBY
- PG_DEGRADED / PG_UNDERSIZED / PG_AVAILABILITY
Add them as a single status update on the host ticket instead, and only when they change, so there's no hourly comment.
Several OSDs down with the host up (e.g. a failing HBA): one ticket per host, N OSDs down on <host>: osd.a, osd.b, ..., not N tickets.
A single OSD down while its host is up keeps today's per-OSD ticket. That's a real disk problem and deserves its own ticket.
Raise severity, don't split: if PGs go inactive (PG_AVAILABILITY with inactive/down PGs), or a 2-copy pool has PGs at 0 copies, that's data unavailability. Raise the host ticket to P1 and say so in the title or description, rather than opening another ticket.

2. Auto-resolve (hwmonDaemon + tinker_tickets API)

hwmonDaemon keeps the set of conditions it reported (it can rebuild it from the API on start). When a condition has been absent for N consecutive runs (suggest 2, i.e. about 2 hours at the hourly timer; configurable), it reports it as resolved.
API: add resolve_ticket_api.php, or a "state": "resolved" field on the existing endpoint. It looks up the ticket by the same dedup hash, adds a comment ("Cleared at , detected by "), and moves it to Closed (or a "Resolved" status) through the Workflow Designer, using a transition that automation is allowed to perform unattended.
Don't auto-close manual tickets, or tickets a human has moved out of Open (In Progress, Pending, assigned). For those, add a "condition cleared" comment only.
Clean up the stale Sep 30 tickets the same way on the first run.

3. Fewer reporters (overlaps #29)

Cluster-wide Ceph checks should run on one node (CEPH_TICKET_NODE exists; make sure it's set and respected everywhere), and should send check_code so the API matches on Ceph's health-check codes rather than message text.

4. Parent/child links (tinker_tickets, optional)

The ticket MCP already has relates_to / blocks links. If the API accepted "parent_hash" or "group_key" on create or update, any symptom that still needs its own ticket (e.g. a SMART failure found on the node after it comes back) could be linked to the host ticket instead of standing alone.

Acceptance criteria

Re-run the 2026-10-27 scenario (power off one OSD host with 10 OSDs, a mon and the standby MDS) and check:

Exactly one new ticket is created: the host-down ticket. Its description lists all 10 OSD ids, the mon and the MDS.
No per-OSD, OSD_DOWN, OSD_HOST_DOWN, MON_DOWN, MDS_INSUFFICIENT_STANDBY or PG_DEGRADED tickets are created or updated.
Stale tickets from earlier outages are not reused. A new outage opens a new host ticket, even if an older matching ticket exists, unless that ticket is still Open and about the same host.
While the host stays down across several hourly runs, the host ticket gets no new comments unless something changed (e.g. PGs went inactive).
When the host returns and Ceph recovers, the ticket closes automatically within N runs, with a "cleared" comment.
One OSD failing on a healthy host still creates one per-OSD ticket.
The cluster-wide checks report from one node only.

#29: send Ceph check codes; don't report every warning from every node
#31: log recurrences instead of reopening closed tickets (fixes the 762-comment loop)
Tinker #842416622: compute-storage-01 outage 2026-10-27 (incident ticket)
pve-infra containers/118-prometheus: bmc-redfish-exporter (host power state per BMC)

Filed with Claude Code during the 2026-10-27 incident review.

## Problem When **one server** goes down, hwmonDaemon and the Tinker ticket API produce a flood of tickets, one per symptom. None of them close when the server comes back. ### Evidence: compute-storage-01 outage, 2026-10-27 compute-storage-01 lost power at 11:53 EDT (PSU latched off; see Tinker #842416622) and was back at 15:20. That **single event** produced: | Ticket | Source | What it is | |---|---|---| | #910896002, #752244128, #788765526, #600144760, #337718870, #621550152, #618784458, #521122110, #498266194, #327710011 | `ceph osd tree` | **10 tickets**, one per OSD (osd.0, 5, 7, 9, 16, 19, 25, 26, 29, 30), all on the same host | | #906739396 | HEALTH_WARN `OSD_HOST_DOWN` | "1 host (10 osds) down" | | #170577775 | HEALTH_WARN `OSD_DOWN` | "10 osds down". **Reused** an Open ticket from 2026-09-30 (an earlier large1 crash) that had never been closed | | #842416622 | HEALTH_WARN `MON_DOWN` | "1/3 mons down" | | #237028956 | HEALTH_WARN `MDS_INSUFFICIENT_STANDBY` | The standby MDS was on that host | | #544996974 | HEALTH_WARN `PG_DEGRADED` | **Reused** from 2026-09-30, also never closed | **13 new tickets plus 2 stale ones updated, for one dead box.** Every one of them was still Open hours after the host was back and Ceph had recovered. So were osd.12 and osd.14 (#569778912, #826445533) from the Sep 30 large1 crash, a month later. A related symptom: #703260823 (closed SMART ticket) has **762 comments**, all "Issue recurred, but the ticket could not be reopened automatically". One is added every hourly run, because there's no unattended Closed→Open workflow transition. (Recurrence handling is covered by #31; it's listed here because it's part of the same noise.) ## Why it happens (current code) **hwmonDaemon** - `_check_ceph_health()`: every `down` node in `ceph osd tree` becomes its own `cluster_wide_issues` entry: `"Ceph OSD osd.N is DOWN"`. - The same function turns **every** HEALTH_WARN check into its own `"Ceph HEALTH_WARN: <message>"` issue. When a host is down, `OSD_DOWN` and `OSD_HOST_DOWN` repeat what the per-OSD entries already say, and `MON_DOWN`, `MDS_INSUFFICIENT_STANDBY` and `PG_DEGRADED` are consequences of the same event. - Nothing correlates symptoms to a host or root cause. The daemon never sends `check_code` (#29), so the API falls back to text matching. - No resolve path: the daemon only ever POSTs "this is happening", never "this stopped". **tinker_tickets `create_ticket_api.php` → `generateTicketHash()`** - `issue_subtype = 'osd_down_' . <id>` for every OSD, so each OSD gets a distinct hash and its own ticket, by design. - Other HEALTH_WARNs hash on the warning text with counts stripped (`warn_osds_down`, `warn_host_osds_down`, `warn_mons_down`, ...). That's why today's "10 osds down" landed on the still-Open Sep 30 ticket. - No way to close a ticket from automation, and no parent/child grouping. ## Proposal ### 1. Group symptoms under the down host (hwmonDaemon) Before building tickets, work out whether a **host** is down, and fold its symptoms into one ticket for that host: - **Host-down signals:** `ceph osd tree` (all OSDs under a host bucket down), HEALTH_WARN `OSD_HOST_DOWN`, `pvecm status` / corosync membership (node missing), and optionally the new BMC data in Prometheus (`bmc_host_power_on == 0`, `bmc_up == 0`, from `bmc-redfish-exporter` in CT 118, see pve-infra). - **One ticket per down host:** `[<host>][auto][hardware] Node down: <host> (N OSDs, mon, mds) [single-node][production][issue]`, priority P1 or P2. The description lists everything folded into it: OSD ids, mon/mgr/mds daemons on that host, HA guests recovered and where they went, Ceph degraded %, and the PG state. - **Suppress the symptoms:** while that host is down, don't create separate tickets for - `Ceph OSD osd.N is DOWN` where osd.N is on that host - `OSD_DOWN` / `OSD_HOST_DOWN` - `MON_DOWN` naming that host's mon - `MDS_INSUFFICIENT_STANDBY` - `PG_DEGRADED` / `PG_UNDERSIZED` / `PG_AVAILABILITY` Add them as a single status update on the host ticket instead, and **only when they change**, so there's no hourly comment. - **Several OSDs down with the host up** (e.g. a failing HBA): one ticket per host, `N OSDs down on <host>: osd.a, osd.b, ...`, not N tickets. - **A single OSD down while its host is up** keeps today's per-OSD ticket. That's a real disk problem and deserves its own ticket. - **Raise severity, don't split:** if PGs go **inactive** (`PG_AVAILABILITY` with inactive/down PGs), or a 2-copy pool has PGs at 0 copies, that's data unavailability. Raise the host ticket to P1 and say so in the title or description, rather than opening another ticket. ### 2. Auto-resolve (hwmonDaemon + tinker_tickets API) - hwmonDaemon keeps the set of conditions it reported (it can rebuild it from the API on start). When a condition has been **absent for N consecutive runs** (suggest 2, i.e. about 2 hours at the hourly timer; configurable), it reports it as resolved. - **API:** add `resolve_ticket_api.php`, or a `"state": "resolved"` field on the existing endpoint. It looks up the ticket by the same dedup hash, adds a comment ("Cleared at <time>, detected by <host>"), and moves it to Closed (or a "Resolved" status) through the Workflow Designer, using a transition that automation is allowed to perform unattended. - **Don't auto-close manual tickets,** or tickets a human has moved out of Open (In Progress, Pending, assigned). For those, add a "condition cleared" comment only. - Clean up the stale Sep 30 tickets the same way on the first run. ### 3. Fewer reporters (overlaps #29) Cluster-wide Ceph checks should run on **one** node (`CEPH_TICKET_NODE` exists; make sure it's set and respected everywhere), and should send `check_code` so the API matches on Ceph's health-check codes rather than message text. ### 4. Parent/child links (tinker_tickets, optional) The ticket MCP already has `relates_to` / `blocks` links. If the API accepted `"parent_hash"` or `"group_key"` on create or update, any symptom that still needs its own ticket (e.g. a SMART failure found on the node after it comes back) could be linked to the host ticket instead of standing alone. ## Acceptance criteria Re-run the 2026-10-27 scenario (power off one OSD host with 10 OSDs, a mon and the standby MDS) and check: - [ ] **Exactly one** new ticket is created: the host-down ticket. Its description lists all 10 OSD ids, the mon and the MDS. - [ ] No per-OSD, `OSD_DOWN`, `OSD_HOST_DOWN`, `MON_DOWN`, `MDS_INSUFFICIENT_STANDBY` or `PG_DEGRADED` tickets are created or updated. - [ ] Stale tickets from earlier outages are not reused. A new outage opens a new host ticket, even if an older matching ticket exists, unless that ticket is still Open **and** about the same host. - [ ] While the host stays down across several hourly runs, the host ticket gets **no** new comments unless something changed (e.g. PGs went inactive). - [ ] When the host returns and Ceph recovers, the ticket closes automatically within N runs, with a "cleared" comment. - [ ] One OSD failing on a healthy host still creates one per-OSD ticket. - [ ] The cluster-wide checks report from one node only. ## Related - #29: send Ceph check codes; don't report every warning from every node - #31: log recurrences instead of reopening closed tickets (fixes the 762-comment loop) - Tinker #842416622: compute-storage-01 outage 2026-10-27 (incident ticket) - pve-infra `containers/118-prometheus`: `bmc-redfish-exporter` (host power state per BMC) --- Filed with Claude Code during the 2026-10-27 incident review.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: LotusGuild/hwmonDaemon#33