Ceph HEALTH_WARN tickets: send the health-check code and stop reporting every warning from every node #29

New Issue

2026-09-25T15:59:56-04:00

jared commented

2026-09-25 15:59:56 -04:00

Found while investigating LotusGuild/tinker_tickets#112 (a ticket page 500 caused by 52k audit rows on one hwmonDaemon ticket).

What happened

Every Ceph HEALTH_WARN without a special case in tinker_tickets' dedup hash (noout, nearfull, mon low on space, insecure key types, standby MDS, pgs not scrubbed, ...) landed on one ticket, 221040637. Each run, hwmonDaemon posts every active warning as its own ticket request. Each request overwrote that ticket's title, so it flip-flopped between warnings and logged ~47k title changes plus ~40k no-op "condition worsened" updates. The churn was about 288 audit rows/day on that ticket, from an hourly timer: several nodes report cluster-wide issues (CEPH_TICKET_NODE is unset) × several warnings × a title write each.

Fixed server-side in tinker_tickets

Description-only refreshes no longer write audit rows (ad20169).
An unknown HEALTH_WARN now gets its own ticket, keyed on the warning text before the first : with digits stripped (72feafb).
New-ticket creates no longer return a 500 after succeeding (ad20169). hwmonDaemon saw 8 of these as failures.
86,888 noise audit rows pruned (backup on 10.10.10.45 in /root/db-backups/).

Deploying the hash change makes each currently active warning open its own new ticket once. 221040637 stops receiving updates and can be closed by hand.

Asks for hwmonDaemon

Send Ceph's health-check code. check_ceph_health() already iterates health_data['checks'] by check_name (e.g. OSDMAP_FLAGS, POOL_NEARFULL, MON_DISK_LOW) but only forwards the summary message. Carry check_name through to the ticket payload as "check_code". The API already accepts it (^[A-Z0-9_]{1,64}$) and prefers it over text matching. That's more robust than message text, which embeds values like mon names that can change. Note: switching to check_code changes the hash once, so each active warning gets one more new ticket.
Report cluster-wide Ceph issues from one node. Set CEPH_TICKET_NODE (or elect a node deterministically, e.g. the lowest-sorted quorum mon) so N nodes don't post the same cluster-wide issue every run. Dedup keeps this correct but multiplies API traffic and write churn.
Only send a priority/title update when something changed. The API still rewrites the description on every duplicate post. Consider skipping the POST when the issue set is unchanged since the last run, or at least don't re-post identical cluster-wide issues from multiple nodes.
Closed recurrences: the prod log shows hourly hwmonDaemon recurrence for ticket 147937387 / 586721907 — no admin-free, comment-free transition from Closed is configured; leaving ticket Closed. Either those conditions are still active and need a Closed→Open workflow transition configured, or they are known/accepted and hwmonDaemon should suppress them.

Found while investigating LotusGuild/tinker_tickets#112 (a ticket page 500 caused by 52k audit rows on one hwmonDaemon ticket). ## What happened Every Ceph `HEALTH_WARN` without a special case in tinker_tickets' dedup hash (noout, nearfull, mon low on space, insecure key types, standby MDS, pgs not scrubbed, ...) landed on **one** ticket, 221040637. Each run, hwmonDaemon posts every active warning as its own ticket request. Each request overwrote that ticket's title, so it flip-flopped between warnings and logged **~47k title changes** plus ~40k no-op "condition worsened" updates. The churn was about 288 audit rows/day on that ticket, from an hourly timer: several nodes report cluster-wide issues (`CEPH_TICKET_NODE` is unset) × several warnings × a title write each. ## Fixed server-side in tinker_tickets - Description-only refreshes no longer write audit rows (ad20169). - An unknown `HEALTH_WARN` now gets its own ticket, keyed on the warning text before the first `:` with digits stripped (72feafb). - New-ticket creates no longer return a 500 after succeeding (ad20169). hwmonDaemon saw 8 of these as failures. - 86,888 noise audit rows pruned (backup on 10.10.10.45 in `/root/db-backups/`). Deploying the hash change makes each currently active warning open its **own** new ticket once. 221040637 stops receiving updates and can be closed by hand. ## Asks for hwmonDaemon 1. **Send Ceph's health-check code.** `check_ceph_health()` already iterates `health_data['checks']` by `check_name` (e.g. `OSDMAP_FLAGS`, `POOL_NEARFULL`, `MON_DISK_LOW`) but only forwards the summary message. Carry `check_name` through to the ticket payload as `"check_code"`. The API already accepts it (`^[A-Z0-9_]{1,64}$`) and prefers it over text matching. That's more robust than message text, which embeds values like mon names that can change. Note: switching to `check_code` changes the hash once, so each active warning gets one more new ticket. 2. **Report cluster-wide Ceph issues from one node.** Set `CEPH_TICKET_NODE` (or elect a node deterministically, e.g. the lowest-sorted quorum mon) so N nodes don't post the same cluster-wide issue every run. Dedup keeps this correct but multiplies API traffic and write churn. 3. **Only send a priority/title update when something changed.** The API still rewrites the description on every duplicate post. Consider skipping the POST when the issue set is unchanged since the last run, or at least don't re-post identical cluster-wide issues from multiple nodes. 4. **Closed recurrences:** the prod log shows hourly `hwmonDaemon recurrence for ticket 147937387 / 586721907 — no admin-free, comment-free transition from Closed is configured; leaving ticket Closed`. Either those conditions are still active and need a Closed→Open workflow transition configured, or they are known/accepted and hwmonDaemon should suppress them.

jared referenced this issue from LotusGuild/tinker_tickets

2026-09-25 16:42:27 -04:00

Ticket pages are giving a 500, maybe due to the mcp server deployment? #112

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: LotusGuild/hwmonDaemon#29