Merge pull request 'docs: stop publishing Hookshot webhook IDs; record Hookshot + call-stack upgrades' (#55) from docs/hookshot-ids-and-upgrades into main

.gitea/workflows/lint.yml

@@ -31,6 +31,20 @@ jobs:
       - name: Run ESLint
         run: npx eslint --ext .js hookshot/
 
+  readme-no-webhook-ids:
+    name: No Hookshot webhook IDs in README
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+
+      # A Hookshot webhook URL's UUID is its only secret, and this repo is public.
+      - name: Reject full UUIDs
+        run: |
+          if grep -nE '[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}' README.md; then
+            echo "::error::README.md contains a full UUID. Webhook IDs are secrets: list only the first 8 characters."
+            exit 1
+          fi
+
   hooks-no-secrets:
     name: No secrets in webhook configs
     runs-on: ubuntu-latest

README.md

@@ -191,9 +191,14 @@ Killing livekit-server while a call is active drops everyone. Instead:
 
 1. Deploy to LXC 151 copies the new `livekit-server.service` and sets a `/run/livekit-restart-pending` flag
 2. `livekit-graceful-restart.timer` runs every 5 minutes
-3. The timer script counts established TCP connections on port 7881 (`ss -tn state established`)
-4. If zero connections → restarts livekit-server and clears the flag
-5. If connections exist → logs and exits, retries in 5 minutes
+3. The timer script reads LiveKit's own `livekit_participant_total` (`:6789/metrics`) and counts open
+   signalling WebSockets on :7880. If the metrics can't be read it assumes a call is active.
+4. Zero participants and zero signalling connections → restarts livekit-server and clears the flag
+5. Otherwise → logs and exits, retries in 5 minutes
+
+Until 2026-11-03 it counted `ESTAB` lines on :7881, which was always 0 (an `ss` state filter drops the
+State column, and :7881 is only the ICE-TCP fallback), so it would have restarted mid-call (matrix #54).
+To stage a restart by hand, for example after replacing the binary: `touch /run/livekit-restart-pending`.
 
 ---
 
@@ -343,27 +348,33 @@ The token in `draupnir/production.yaml` in this repo is **intentionally redacted
 
 ---
 
-## Webhook Integrations (matrix-hookshot 7.3.2)
+## Webhook Integrations (matrix-hookshot 7.5.0)
 
 Generic webhooks bridged into **Spam and Stuff**.
 Each service gets its own virtual user (`@hookshot_<service>`) with a unique avatar.
 Webhook URL format: `https://matrix.lotusguild.org/webhook/<uuid>`
 
+> The UUID **is** the secret: anyone who has it can post into the room. This repo is public, so
+> the table lists only the first 8 characters (enough to match Hookshot's logs); the full URLs
+> live in each sender's settings. Full UUIDs were in this file from 2026-02-20 to 2026-11-03,
+> so the IDs below are exposed in git history until rotated (see *Webhook secrets — rotating*).
+> CI (`readme-no-webhook-ids`) rejects full UUIDs in this README.
+
 | Service | Webhook UUID | Notes |
 |---------|-------------|-------|
-| Grafana | `df4a1302-2d62-4a01-b858-fb56f4d3781a` | Unified alerting contact point |
-| Proxmox | `9b3eafe5-7689-4011-addd-c466e524661d` | Notification system (8.1+), Discord embed format |
-| Sonarr | `aeffc311-0686-42cb-9eeb-6757140c072e` | All event types |
-| Radarr | `34913454-c1ac-4cda-82ea-924d4a9e60eb` | All event types |
-| Readarr | `e57ab4f3-56e6-4dc4-8b30-2f4fd4bbeb0b` | All event types |
-| Lidarr | `66ac6fdd-69f6-4f47-bb00-b7f6d84d7c1c` | All event types |
-| Uptime Kuma | `1a02e890-bb25-42f1-99fe-bba6a19f1811` | Status change notifications |
-| Seerr | `555185af-90a1-42ff-aed5-c344e11955cf` | Request/approval events |
-| Owncast (Livestream) | `9993e911-c68b-4271-a178-c2d65ca88499` | STREAM_STARTED / STREAM_STOPPED |
-| Bazarr | `470fb267-3436-4dd3-a70c-e6e8db1721be` | Subtitle events (Apprise JSON notifier) |
-| Tinker-Tickets | `6e306faf-8eea-4ba5-83ef-bf8f421f929e` | Custom transformation code |
-| Ci-Alert | `5e7051e7-fd0c-4930-a711-9c67daf6f0b9` | Gitea Actions `notify-failure` jobs (org secret `MATRIX_WEBHOOK_URL`); transform `hookshot/ci-alert.js` |
-| Devdesk | `4a3ccfc9-e1fe-4124-b014-7842fe51b9dc` | DevDesk "Claude needs you" (CT 126 `devdesk-notify-matrix`, URL in `/etc/devdesk/matrix-webhook`); transform `hookshot/devdesk.js`; avatar set via the appservice token |
+| Grafana | `df4a1302…` | Unified alerting contact point |
+| Proxmox | `9b3eafe5…` | Notification system (8.1+), Discord embed format |
+| Sonarr | `aeffc311…` | All event types |
+| Radarr | `34913454…` | All event types |
+| Readarr | `e57ab4f3…` | All event types |
+| Lidarr | `66ac6fdd…` | All event types |
+| Uptime Kuma | `1a02e890…` | Status change notifications |
+| Seerr | `555185af…` | Request/approval events |
+| Owncast (Livestream) | `9993e911…` | STREAM_STARTED / STREAM_STOPPED |
+| Bazarr | `470fb267…` | Subtitle events (Apprise JSON notifier) |
+| Tinker-Tickets | `6e306faf…` | Custom transformation code |
+| Ci-Alert | `5e7051e7…` | Gitea Actions `notify-failure` jobs (org secret `MATRIX_WEBHOOK_URL`); transform `hookshot/ci-alert.js` |
+| Devdesk | `4a3ccfc9…` | DevDesk "Claude needs you" (CT 126 `devdesk-notify-matrix`, URL in `/etc/devdesk/matrix-webhook`); transform `hookshot/devdesk.js`; avatar set via the appservice token |
 
 **Hookshot notes:**
 - Spam and Stuff is intentionally **unencrypted** — hookshot bridges cannot join E2EE rooms
@@ -984,6 +995,47 @@ Leaving a space does **not** leave its child rooms; they are independent rooms w
 
 ---
 
+## Hookshot Upgrade Notes
+
+Hookshot is a **git checkout built from source** at `/opt/hookshot` (`ExecStart=node lib/App/BridgeApp.js`).
+Only `config.yml` and `passkey.pem` in it are not generated; storage is in-memory (connections are room state).
+
+- **7.4.1+ builds with pnpm**, and `scripts/build-app.sh` calls `pnpm` directly. Without a global pnpm,
+  put a wrapper first in `PATH`: `printf '#!/bin/sh\nexec corepack pnpm@<version> "$@"\n'` (version from
+  `devEngines.packageManager` in `package.json`). The Rust layer needs `~/.cargo/bin` in `PATH`.
+- **7.6.0 drops Node 22** (LXC 151 runs Node 22), so 7.5.0 is the newest that runs as-is.
+- Build the new tag in a separate directory (`git clone --depth 1 --branch <tag> … /opt/hookshot-<tag>`,
+  `pnpm install --frozen-lockfile && pnpm run build`), copy in `config.yml` + `passkey.pem`, then stop the
+  service, swap the directories (keep the old one for rollback) and start it. Check the log says
+  `Bridge is now ready. Found N connections`.
+- 2026-11-03: 7.3.2 → 7.5.0 (GHSA-5c49-6fvm-2xvc: `/metrics` exposed webhook URLs, plus two DoS fixes).
+  Old build kept at `/opt/hookshot-7.3.2`.
+
+## Call Stack Upgrade (lk-jwt-service 0.4.1 → 0.9.0, LiveKit 1.9.11 → 1.13.11) — planned
+
+Tested on the local stack on 2026-11-03 (calls through the guard, voice-bridge e2e audio both ways,
+encrypted and not). Required changes, all on LXC 151:
+
+1. **lk-jwt-service 0.9.0** (a Rust rewrite; fixes GHSA-rq24-fh66-rq7x). Existing env vars carry over.
+   Add `LIVEKIT_SANITY_CHECK_INTERVAL_SECONDS=30`.
+2. **LiveKit webhook → lk-jwt.** 0.9.0 serves `/delegate_delayed_leave`, so Element Call hands it the
+   job of removing a crashed client's call membership and sets its own fallback timer to **1 hour**.
+   lk-jwt only learns that someone left from LiveKit webhooks; without them crashed clients linger for an
+   hour (also on Discord, via the voice bridge). In `/etc/livekit/config.yaml`:
+   ```yaml
+   webhook:
+     api_key: <the key under keys:, same as LIVEKIT_KEY>
+     urls:
+       - http://127.0.0.1:8071/sfu_webhook
+   ```
+3. **`prometheus_port: 6789` → `prometheus: {port: 6789}`.** LiveKit 1.13.8 removed the old key (1.9.11
+   already accepts the new one). Otherwise metrics and the *LiveKit Down* alert stop working.
+4. Restart lk-jwt-service only when no calls are active: its leave jobs are in memory (or set
+   `LIVEKIT_REDIS_URL`). LiveKit: replace the binary and `touch /run/livekit-restart-pending`.
+
+Not affected: TURN (external coturn with `ttl` set, as 1.13.1 requires), `room.auto_create: false`,
+the voice bridge (uses the legacy `/sfu/get`, still served).
+
 ## Synapse Upgrade Runbook
 
 Synapse is installed from the **matrix.org apt repo** (`matrix-synapse-py3`) on **LXC 151** — it is
@@ -1176,10 +1228,10 @@ All commands use the `!` prefix. Run `!help` in any room for the full list.
 | Database | PostgreSQL | 17.9 |
 | TURN | coturn | latest |
 | Video/voice calls | LiveKit SFU | 1.9.11 |
-| LiveKit JWT | lk-jwt-service | latest |
+| LiveKit JWT | lk-jwt-service | 0.4.1 (0.9.0 planned, see *Call stack upgrade*) |
 | Moderation | Draupnir | 2.9.0 |
 | SSO | Authelia (OIDC) + LLDAP | — |
-| Webhook bridge | matrix-hookshot | 7.3.2 |
+| Webhook bridge | matrix-hookshot | 7.5.0 |
 | Discord bridge | mautrix-discord | 0.7.7 |
 | Reverse proxy | Nginx Proxy Manager | — |
 | Web client | Lotus Cinny (fork of `cinnyapp/cinny` main) | custom |