Retry failed Matrix webhook notifications with backoff (#78)

NotificationHelper::fire() logged a failed webhook post via error_log()
only, with no retry and no persistent record — once a notification
failed, it was gone with no trace beyond the log line, even though the
underlying DB write (audit_log entry, status change, etc.) it was
reporting on had already committed.

Extracted the curl POST into attemptDelivery(), shared between fire()
(unchanged best-effort caller-facing behavior) and the new
cron/retry_failed_notifications.php. On failure, fire() now also queues
the payload to notification_retry_queue (migration 007) via
Database::getConnection() — most fire() call sites don't have a $conn
handy, and threading one through every caller would be a much larger,
more invasive change than reusing the existing connection singleton.

The cron script processes due rows with exponential backoff (2, 4, 8...
capped at 60 minutes) up to each row's max_attempts (default 6), then
leaves an exhausted row in place — not deleted — so it stays visible
for manual investigation instead of disappearing a second time.

Verified against real MariaDB and a local HTTP server standing in for
the Matrix webhook, toggled between failing and succeeding: confirmed
a real failure via fire() is correctly queued; the retry script
reschedules a still-failing row with the expected backoff delay;
flipping the fake webhook to succeed lets the same row's next retry
delete it; a row that exhausts all attempts is left in place and
correctly excluded from the next run's due-row query; and a success
via fire() queues nothing (no regression on the common case).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117oBw2jN4kALYeS8HPq4zV

migrations/007_notification_retry_queue.sql

@@ -0,0 +1,20 @@
+-- Queue for Matrix webhook notifications that failed to send (#78).
+--
+-- NotificationHelper::fire() previously logged a failed webhook post via
+-- error_log() only, with no retry and no persistent record — once a
+-- notification failed, it was gone. Failed payloads are now queued here and
+-- retried by cron/retry_failed_notifications.php with exponential backoff.
+--
+-- Safe to re-run.
+
+CREATE TABLE IF NOT EXISTS `notification_retry_queue` (
+  `retry_id` int(11) NOT NULL AUTO_INCREMENT,
+  `payload` longtext CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL CHECK (json_valid(`payload`)),
+  `attempts` int(11) NOT NULL DEFAULT 0,
+  `max_attempts` int(11) NOT NULL DEFAULT 6,
+  `next_attempt_at` timestamp NULL DEFAULT current_timestamp(),
+  `last_error` varchar(500) DEFAULT NULL,
+  `created_at` timestamp NULL DEFAULT current_timestamp(),
+  PRIMARY KEY (`retry_id`),
+  KEY `idx_next_attempt` (`next_attempt_at`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci;