Failed Matrix webhook posts are logged but never retried — confirm this best-effort design is intentional #78

New Issue

2026-09-01T00:07:13-04:00

jared commented

2026-09-01 00:07:13 -04:00

Severity: Low / informational

helpers/NotificationHelper::fire() (~lines 30-34) logs cURL errors / non-2xx responses via error_log() only, with no retry and no persistent record. If a webhook fails, the underlying DB write (audit_log entry, status change, etc.) has already committed — there's no record anywhere that the Matrix side of that event never went out. There is an existing code comment near notifyWatchers() (~lines 190-192) suggesting this best-effort behavior is intentional.

Flagging for confirmation rather than as a bug: is silent loss of failed notifications acceptable, or should failed webhook posts be queued for retry (e.g. a simple retry table processed by a cron job)? If the current behavior is fine, no action needed beyond this note.

**Severity:** Low / informational `helpers/NotificationHelper::fire()` (~lines 30-34) logs cURL errors / non-2xx responses via `error_log()` only, with no retry and no persistent record. If a webhook fails, the underlying DB write (audit_log entry, status change, etc.) has already committed — there's no record anywhere that the Matrix side of that event never went out. There is an existing code comment near `notifyWatchers()` (~lines 190-192) suggesting this best-effort behavior is intentional. Flagging for confirmation rather than as a bug: is silent loss of failed notifications acceptable, or should failed webhook posts be queued for retry (e.g. a simple retry table processed by a cron job)? If the current behavior is fine, no action needed beyond this note.

jared added the needs-decision notifications priority/low labels 2026-09-08 10:15:48 -04:00

jared referenced this issue from a commit

2026-09-12 01:18:19 -04:00

Retry failed Matrix webhook notifications with backoff (#78)

jared referenced this issue from a commit

2026-09-12 01:21:00 -04:00

Merge development into main: API key visibility, notification retry, watch audit logging (#70, #78, #93)

jared commented

2026-09-12 01:21:14 -04:00

Decision: add retry. Fixed in commit 35192aa (merged to main in d0a7630).

Failed Matrix webhook payloads are now queued to a new notification_retry_queue table (migration 007) instead of just being logged and lost. cron/retry_failed_notifications.php retries due rows with exponential backoff (2, 4, 8... capped at 60 min) up to 6 attempts, then leaves an exhausted row in place (not deleted) for manual investigation. Added to the README's Cron Jobs section (every 5 minutes).

Verified against real MariaDB and a local HTTP server standing in for the Matrix webhook, toggled between failing and succeeding: a real failure via fire() is correctly queued; the retry script reschedules a still-failing row with the expected backoff delay; flipping the fake webhook to succeed lets the same row's next retry delete it; an exhausted row is left in place and excluded from future runs; and a normal success queues nothing (no regression).

Decision: add retry. Fixed in commit 35192aa (merged to main in d0a7630). Failed Matrix webhook payloads are now queued to a new notification_retry_queue table (migration 007) instead of just being logged and lost. cron/retry_failed_notifications.php retries due rows with exponential backoff (2, 4, 8... capped at 60 min) up to 6 attempts, then leaves an exhausted row in place (not deleted) for manual investigation. Added to the README's Cron Jobs section (every 5 minutes). Verified against real MariaDB and a local HTTP server standing in for the Matrix webhook, toggled between failing and succeeding: a real failure via fire() is correctly queued; the retry script reschedules a still-failing row with the expected backoff delay; flipping the fake webhook to succeed lets the same row's next retry delete it; an exhausted row is left in place and excluded from future runs; and a normal success queues nothing (no regression).

jared closed this issue

2026-09-12 01:21:15 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: LotusGuild/tinker_tickets#78