New ticket type for recurring/expected conditions: log recurrences instead of auto-reopening closed tickets #31

New Issue

2026-09-25T18:41:59-04:00

jared commented

2026-09-25 18:41:59 -04:00

Problem

Some conditions will come back no matter how often they're closed. For example, compute-storage-gpu-01's "CPU usage sustained above threshold of 95%" (https://t.lotusguild.org/ticket/488693049) is expected during Tdarr transcodes and heavy Ceph periods.

Today, any recurrence of a closed ticket reopens it. create_ticket_api.php looks up the dedup hash (open OR closed), and if the match is Closed it moves it to Open and posts "Issue recurred — ticket reopened automatically". So closing #488693049 doesn't help: it reopens during the next burst, which is at least monthly.

Evidence (hwmon journal on compute-storage-gpu-01, since PR #26 moved CPU alerts to the 15-min load):

The CPU issue passed the filter on most days from Jul 16 to Sep 25, often on every hourly run (24/24 on Aug 9–16, Aug 22, Aug 26–27). It was quiet on other days (1/day in early Sep).
It reopened on Jul 16, the day after it was closed as a false positive on Jul 15.
The ticket body shows Usage (now) 71.4% next to 15-min load 100.1% per core. Load average also counts tasks waiting on I/O (D-state), such as Ceph/RBD waits, so it can overstate CPU use.

Proposal: a new ticket type for recurring or expected conditions

Add a type (name TBD, e.g. Recurring or Advisory) for conditions that are known to be noisy or expected to happen periodically, which a human can acknowledge without being pinged every time.

hwmonDaemon side

Map the issues that recur on their own to this type in _determine_ticket_type() / TICKET_TYPES. Start with CPU sustained load. Candidates: Ceph HEALTH_WARN slow ops, pgs not deep-scrubbed in time.
Send occurrence info in the payload: first_seen, last_seen, and consecutive_runs (from a small state file, locked with fcntl like the SMART history, ~L1110).
Optional: move CPU alerting off load average to real CPU usage (a /proc/stat delta averaged over the interval, or node_exporter), so time spent waiting on I/O doesn't count as CPU pressure.

Ticket API side (tinker_tickets create_ticket_api.php, needs a matching change)
When a hash matches a Closed ticket of this type:

Don't reopen it. Record the recurrence instead: increment an occurrence counter and update last_seen. Add at most one "recurred N times since close" comment per day, or put it in a field so it doesn't spam comments.
Only reopen when it gets worse: e.g. consecutive_runs >= N (N ≈ 6, i.e. more than about 6h sustained), or more than X occurrences in 7 days, or the priority goes up. Put the thresholds in config.
While the ticket is Open, keep today's update/dedup behaviour.

Acceptance

Closing #488693049 (or any ticket of the new type) keeps it closed through normal short bursts.
A sustained or escalating condition past the threshold still reopens it, with a comment showing how many times and how long it has recurred.
Existing Issue/Problem types keep their current reopen-on-recurrence behaviour.

## Problem Some conditions will come back no matter how often they're closed. For example, compute-storage-gpu-01's "CPU usage sustained above threshold of 95%" (https://t.lotusguild.org/ticket/488693049) is expected during Tdarr transcodes and heavy Ceph periods. Today, any recurrence of a closed ticket reopens it. `create_ticket_api.php` looks up the dedup hash (open OR closed), and if the match is Closed it moves it to Open and posts "Issue recurred — ticket reopened automatically". So closing #488693049 doesn't help: it reopens during the next burst, which is at least monthly. Evidence (hwmon journal on compute-storage-gpu-01, since PR #26 moved CPU alerts to the 15-min load): - The CPU issue passed the filter on most days from Jul 16 to Sep 25, often on every hourly run (24/24 on Aug 9–16, Aug 22, Aug 26–27). It was quiet on other days (1/day in early Sep). - It reopened on Jul 16, the day after it was closed as a false positive on Jul 15. - The ticket body shows `Usage (now) 71.4%` next to `15-min load 100.1% per core`. Load average also counts tasks waiting on I/O (D-state), such as Ceph/RBD waits, so it can overstate CPU use. ## Proposal: a new ticket type for recurring or expected conditions Add a type (name TBD, e.g. **`Recurring`** or **`Advisory`**) for conditions that are known to be noisy or expected to happen periodically, which a human can acknowledge without being pinged every time. **hwmonDaemon side** - Map the issues that recur on their own to this type in `_determine_ticket_type()` / `TICKET_TYPES`. Start with CPU sustained load. Candidates: Ceph `HEALTH_WARN` slow ops, `pgs not deep-scrubbed in time`. - Send occurrence info in the payload: `first_seen`, `last_seen`, and `consecutive_runs` (from a small state file, locked with `fcntl` like the SMART history, ~L1110). - Optional: move CPU alerting off load average to real CPU usage (a `/proc/stat` delta averaged over the interval, or node_exporter), so time spent waiting on I/O doesn't count as CPU pressure. **Ticket API side (tinker_tickets `create_ticket_api.php`, needs a matching change)** When a hash matches a **Closed** ticket of this type: - **Don't reopen it.** Record the recurrence instead: increment an occurrence counter and update `last_seen`. Add at most one "recurred N times since close" comment per day, or put it in a field so it doesn't spam comments. - **Only reopen when it gets worse:** e.g. `consecutive_runs >= N` (N ≈ 6, i.e. more than about 6h sustained), or more than X occurrences in 7 days, or the priority goes up. Put the thresholds in config. - While the ticket is Open, keep today's update/dedup behaviour. ## Acceptance - Closing #488693049 (or any ticket of the new type) keeps it closed through normal short bursts. - A sustained or escalating condition past the threshold still reopens it, with a comment showing how many times and how long it has recurred. - Existing Issue/Problem types keep their current reopen-on-recurrence behaviour.

jared referenced this issue

2026-10-27 18:27:58 -04:00

Aggregate alerts by root cause: one ticket per down host instead of one per symptom, plus auto-resolve #33

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: LotusGuild/hwmonDaemon#31