Retry failed Matrix webhook notifications with backoff (#78)

NotificationHelper::fire() logged a failed webhook post via error_log()
only, with no retry and no persistent record — once a notification
failed, it was gone with no trace beyond the log line, even though the
underlying DB write (audit_log entry, status change, etc.) it was
reporting on had already committed.

Extracted the curl POST into attemptDelivery(), shared between fire()
(unchanged best-effort caller-facing behavior) and the new
cron/retry_failed_notifications.php. On failure, fire() now also queues
the payload to notification_retry_queue (migration 007) via
Database::getConnection() — most fire() call sites don't have a $conn
handy, and threading one through every caller would be a much larger,
more invasive change than reusing the existing connection singleton.

The cron script processes due rows with exponential backoff (2, 4, 8...
capped at 60 minutes) up to each row's max_attempts (default 6), then
leaves an exhausted row in place — not deleted — so it stays visible
for manual investigation instead of disappearing a second time.

Verified against real MariaDB and a local HTTP server standing in for
the Matrix webhook, toggled between failing and succeeding: confirmed
a real failure via fire() is correctly queued; the retry script
reschedules a still-failing row with the expected backoff delay;
flipping the fake webhook to succeed lets the same row's next retry
delete it; a row that exhausts all attempts is left in place and
correctly excluded from the next run's due-row query; and a success
via fire() queues nothing (no regression on the common case).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117oBw2jN4kALYeS8HPq4zV

cron/retry_failed_notifications.php

@@ -0,0 +1,133 @@
+#!/usr/bin/env php
+<?php
+
+/**
+ * Failed Matrix Notification Retry Cron Job
+ *
+ * NotificationHelper::fire() queues a failed webhook post to
+ * notification_retry_queue instead of just logging and losing it. This
+ * script retries due rows with exponential backoff, up to each row's
+ * max_attempts, then leaves an exhausted row in place (not deleted) so it
+ * remains visible for manual investigation.
+ *
+ * Run this via cron every 5-10 minutes (see README.md's Cron Jobs section
+ * for the exact crontab line — the "every 5 minutes" syntax isn't repeated
+ * here since it would terminate this comment block early).
+ */
+
+// Prevent web access
+if (php_sapi_name() !== 'cli') {
+    http_response_code(403);
+    exit('CLI access only');
+}
+
+chdir(dirname(__DIR__));
+
+require_once 'config/config.php';
+require_once 'helpers/Database.php';
+require_once 'helpers/NotificationHelper.php';
+
+function logMessage($message)
+{
+    echo '[' . date('Y-m-d H:i:s') . '] ' . $message . "\n";
+}
+
+/** Exponential backoff in minutes: 2, 4, 8, 16, 32, capped at 60. */
+function nextAttemptDelayMinutes(int $attemptNumber): int
+{
+    return min(60, 2 ** $attemptNumber);
+}
+
+$webhookUrl = $GLOBALS['config']['MATRIX_WEBHOOK_URL'] ?? null;
+if (empty($webhookUrl)) {
+    logMessage('MATRIX_WEBHOOK_URL not configured — nothing to retry, exiting.');
+    exit(0);
+}
+
+try {
+    $conn = Database::getConnection();
+} catch (Exception $e) {
+    logMessage('FATAL ERROR: could not connect to database: ' . $e->getMessage());
+    exit(1);
+}
+
+// Process a bounded batch per run so one cron tick can't run indefinitely if
+// the queue has backed up.
+$batchLimit = 50;
+
+$stmt = $conn->prepare(
+    "SELECT retry_id, payload, attempts, max_attempts
+       FROM notification_retry_queue
+      WHERE next_attempt_at <= NOW() AND attempts < max_attempts
+      ORDER BY retry_id ASC
+      LIMIT ?"
+);
+$stmt->bind_param('i', $batchLimit);
+$stmt->execute();
+$dueRows = $stmt->get_result()->fetch_all(MYSQLI_ASSOC);
+$stmt->close();
+
+if (empty($dueRows)) {
+    logMessage('No notifications due for retry.');
+    exit(0);
+}
+
+$succeeded = 0;
+$failed = 0;
+$exhausted = 0;
+
+foreach ($dueRows as $row) {
+    $payload = json_decode($row['payload'], true);
+    if (!is_array($payload)) {
+        // Corrupt row — can't retry something unparseable. Remove it rather
+        // than retrying forever against a row that will never succeed.
+        $del = $conn->prepare("DELETE FROM notification_retry_queue WHERE retry_id = ?");
+        $del->bind_param('i', $row['retry_id']);
+        $del->execute();
+        $del->close();
+        logMessage("Discarded retry #{$row['retry_id']}: payload is not valid JSON");
+        continue;
+    }
+
+    $result = NotificationHelper::attemptDelivery($webhookUrl, $payload);
+
+    if ($result['success']) {
+        $del = $conn->prepare("DELETE FROM notification_retry_queue WHERE retry_id = ?");
+        $del->bind_param('i', $row['retry_id']);
+        $del->execute();
+        $del->close();
+        $succeeded++;
+        logMessage("Retry #{$row['retry_id']} succeeded (attempt " . ((int)$row['attempts'] + 1) . ')');
+        continue;
+    }
+
+    $newAttempts = (int)$row['attempts'] + 1;
+    if ($newAttempts >= (int)$row['max_attempts']) {
+        // Exhausted: leave the row (attempts is now == max_attempts, so the
+        // WHERE clause above naturally excludes it from future runs) rather
+        // than deleting it, so it stays visible for manual investigation.
+        $upd = $conn->prepare(
+            "UPDATE notification_retry_queue SET attempts = ?, last_error = ? WHERE retry_id = ?"
+        );
+        $upd->bind_param('isi', $newAttempts, $result['error'], $row['retry_id']);
+        $upd->execute();
+        $upd->close();
+        $exhausted++;
+        logMessage("Retry #{$row['retry_id']} exhausted after {$newAttempts} attempts: {$result['error']}");
+        continue;
+    }
+
+    $delayMinutes = nextAttemptDelayMinutes($newAttempts);
+    $upd = $conn->prepare(
+        "UPDATE notification_retry_queue
+            SET attempts = ?, last_error = ?, next_attempt_at = DATE_ADD(NOW(), INTERVAL ? MINUTE)
+          WHERE retry_id = ?"
+    );
+    $upd->bind_param('isii', $newAttempts, $result['error'], $delayMinutes, $row['retry_id']);
+    $upd->execute();
+    $upd->close();
+    $failed++;
+    logMessage("Retry #{$row['retry_id']} failed (attempt {$newAttempts}), next attempt in {$delayMinutes}m: {$result['error']}");
+}
+
+logMessage("Done: {$succeeded} succeeded, {$failed} rescheduled, {$exhausted} exhausted (of " . count($dueRows) . ' processed)');