Ticket pages are giving a 500, maybe due to the mcp server deployment? #112

New Issue

2026-09-25T01:05:36-04:00

jared commented

2026-09-25 01:05:36 -04:00

No description provided.

jared added the bug priority/high labels 2026-09-25 01:05:36 -04:00

jared referenced this issue from a commit

2026-09-25 15:34:46 -04:00

Fix ticket page 500 from unbounded audit timeline (#112)

jared referenced this issue from a commit

2026-09-25 15:45:09 -04:00

Merge development: fix ticket page 500 (#112)

jared commented

2026-09-25 15:54:02 -04:00

Not caused by the MCP server. Fixed in ad20169 (merged to main in da916bc, deployed).

Cause: ticket 221040637 (the Ceph noout ticket) had 51,970 audit rows, about 15 MB of JSON. The ticket page loaded its whole activity timeline with no limit, so PHP hit its 128 MB memory limit (Allowed memory size ... exhausted in models/AuditLogModel.php). The rows came from hwmonDaemon. Every run refreshes the ticket description, and that alone wrote an "auto-updated by hwmonDaemon (condition worsened)" audit row, so each open auto-ticket got a new row every few minutes. This has happened since April; the next biggest is 673679581 with 21k rows.

Fixes

getTicketTimeline() now takes a limit. The ticket page shows the newest 500 events and a note when older ones are left out. The JSON export is capped at 5000. Tested against prod data: 501 rows, 1.7 MB peak memory.
create_ticket_api.php writes an audit row only when the title or priority actually changes. Description-only refreshes no longer log anything.
Found while testing: after creating a brand-new ticket, the dedup retry loop ran a second time on a closed connection. That appended a 500 error body to the success response. The ticket was still created, but hwmonDaemon saw an error; prod logs show 8 of these. The endpoint now exits cleanly.

Verified locally against MariaDB over real HTTP: a new ticket returns a clean 200, two description-only refreshes add 0 audit rows, and a priority escalation still logs 1.

The old reason-only rows are still in audit_log. Pruning them is optional and left for a separate decision.

Not caused by the MCP server. Fixed in ad20169 (merged to main in da916bc, deployed). **Cause:** ticket 221040637 (the Ceph `noout` ticket) had **51,970 audit rows**, about 15 MB of JSON. The ticket page loaded its whole activity timeline with no limit, so PHP hit its 128 MB memory limit (`Allowed memory size ... exhausted in models/AuditLogModel.php`). The rows came from hwmonDaemon. Every run refreshes the ticket description, and that alone wrote an "auto-updated by hwmonDaemon (condition worsened)" audit row, so each open auto-ticket got a new row every few minutes. This has happened since April; the next biggest is 673679581 with 21k rows. **Fixes** - `getTicketTimeline()` now takes a limit. The ticket page shows the newest 500 events and a note when older ones are left out. The JSON export is capped at 5000. Tested against prod data: 501 rows, 1.7 MB peak memory. - `create_ticket_api.php` writes an audit row only when the title or priority actually changes. Description-only refreshes no longer log anything. - Found while testing: after creating a **brand-new** ticket, the dedup retry loop ran a second time on a closed connection. That appended a 500 error body to the success response. The ticket was still created, but hwmonDaemon saw an error; prod logs show 8 of these. The endpoint now exits cleanly. Verified locally against MariaDB over real HTTP: a new ticket returns a clean 200, two description-only refreshes add 0 audit rows, and a priority escalation still logs 1. The old reason-only rows are still in `audit_log`. Pruning them is optional and left for a separate decision.

jared closed this issue

2026-09-25 15:54:02 -04:00

jared referenced this issue from a commit

2026-09-25 15:59:39 -04:00

Give each Ceph HEALTH_WARN its own dedup ticket (#112)

jared referenced this issue from LotusGuild/hwmonDaemon

2026-09-25 15:59:56 -04:00

Ceph HEALTH_WARN tickets: send the health-check code and stop reporting every warning from every node #29

jared referenced this issue from a commit

2026-09-25 16:39:37 -04:00

Merge development: per-warning Ceph dedup (#112)

jared commented

2026-09-25 16:42:27 -04:00

Follow-up (72feafb, merged in 5b43e7d, deployed):

Pruned 86,888 noise audit rows: 40,319 reason-only rows across all tickets, plus 46,569 hwmonDaemon title flip-flops on 221040637. Backup is on 10.10.10.45 at /root/db-backups/audit-prune-112-20260925155822.sql (root-only). audit_log went from 88,148 to 1,260 rows.
Root cause of the flip-flops: every Ceph HEALTH_WARN without a special case shared one dedup hash, so all cluster warnings landed on 221040637. Each warning now gets its own ticket, keyed on the optional check_code field, or else on the warning text with counts stripped. Known subtypes hash as before.
Expect one new ticket for each Ceph warning currently active. After that, 221040637 won't be updated any more and can be closed.
hwmonDaemon-side follow-ups: LotusGuild/hwmonDaemon#29.

Follow-up (72feafb, merged in 5b43e7d, deployed): - Pruned 86,888 noise audit rows: 40,319 reason-only rows across all tickets, plus 46,569 hwmonDaemon title flip-flops on 221040637. Backup is on 10.10.10.45 at `/root/db-backups/audit-prune-112-20260925155822.sql` (root-only). `audit_log` went from 88,148 to 1,260 rows. - Root cause of the flip-flops: every Ceph `HEALTH_WARN` without a special case shared one dedup hash, so all cluster warnings landed on 221040637. Each warning now gets its own ticket, keyed on the optional `check_code` field, or else on the warning text with counts stripped. Known subtypes hash as before. - Expect one new ticket for each Ceph warning currently active. After that, 221040637 won't be updated any more and can be closed. - hwmonDaemon-side follow-ups: LotusGuild/hwmonDaemon#29.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: LotusGuild/tinker_tickets#112