docs: stop publishing Hookshot webhook IDs; record the Hookshot and call-stack upgrades

- The README's webhook table listed every Hookshot webhook's full UUID. The UUID is the webhook's only secret and this repo is public, so anyone could post into Spam and Stuff. Shorten them to 8 characters and add CI (readme-no-webhook-ids) rejecting full UUIDs in README.md. They remain in git history until rotated. - Hookshot is now 7.5.0: how it's built (pnpm 7.4.1+, Node 22 limit of 7.6.0) and how to swap a new build in. - Planned lk-jwt-service 0.9.0 / LiveKit 1.13.11 upgrade: the LiveKit webhook delegated leave needs, the prometheus config key, and restart caveats, from local testing. - Graceful restart: describe the fixed check (#54). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5
2026-11-03 00:34:03 -05:00
parent 8a0e16cabd
commit 6e41477a0b
2 changed files with 85 additions and 19 deletions
@@ -191,9 +191,14 @@ Killing livekit-server while a call is active drops everyone. Instead:

 1. Deploy to LXC 151 copies the new `livekit-server.service` and sets a `/run/livekit-restart-pending` flag
 2. `livekit-graceful-restart.timer` runs every 5 minutes
-3. The timer script counts established TCP connections on port 7881 (`ss -tn state established`)
-4. If zero connections → restarts livekit-server and clears the flag
-5. If connections exist → logs and exits, retries in 5 minutes
+3. The timer script reads LiveKit's own `livekit_participant_total` (`:6789/metrics`) and counts open
+   signalling WebSockets on :7880. If the metrics can't be read it assumes a call is active.
+4. Zero participants and zero signalling connections → restarts livekit-server and clears the flag
+5. Otherwise → logs and exits, retries in 5 minutes
+
+Until 2026-11-03 it counted `ESTAB` lines on :7881, which was always 0 (an `ss` state filter drops the
+State column, and :7881 is only the ICE-TCP fallback), so it would have restarted mid-call (matrix #54).
+To stage a restart by hand, for example after replacing the binary: `touch /run/livekit-restart-pending`.

 ---

@@ -343,27 +348,33 @@ The token in `draupnir/production.yaml` in this repo is **intentionally redacted

 ---

-## Webhook Integrations (matrix-hookshot 7.3.2)
+## Webhook Integrations (matrix-hookshot 7.5.0)

 Generic webhooks bridged into **Spam and Stuff**.
 Each service gets its own virtual user (`@hookshot_<service>`) with a unique avatar.
 Webhook URL format: `https://matrix.lotusguild.org/webhook/<uuid>`

+> The UUID **is** the secret: anyone who has it can post into the room. This repo is public, so
+> the table lists only the first 8 characters (enough to match Hookshot's logs); the full URLs
+> live in each sender's settings. Full UUIDs were in this file from 2026-02-20 to 2026-11-03,
+> so the IDs below are exposed in git history until rotated (see *Webhook secrets — rotating*).
+> CI (`readme-no-webhook-ids`) rejects full UUIDs in this README.
+
 | Service | Webhook UUID | Notes |
 |---------|-------------|-------|
-| Grafana | `df4a1302-2d62-4a01-b858-fb56f4d3781a` | Unified alerting contact point |
-| Proxmox | `9b3eafe5-7689-4011-addd-c466e524661d` | Notification system (8.1+), Discord embed format |
-| Sonarr | `aeffc311-0686-42cb-9eeb-6757140c072e` | All event types |
-| Radarr | `34913454-c1ac-4cda-82ea-924d4a9e60eb` | All event types |
-| Readarr | `e57ab4f3-56e6-4dc4-8b30-2f4fd4bbeb0b` | All event types |
-| Lidarr | `66ac6fdd-69f6-4f47-bb00-b7f6d84d7c1c` | All event types |
-| Uptime Kuma | `1a02e890-bb25-42f1-99fe-bba6a19f1811` | Status change notifications |
-| Seerr | `555185af-90a1-42ff-aed5-c344e11955cf` | Request/approval events |
-| Owncast (Livestream) | `9993e911-c68b-4271-a178-c2d65ca88499` | STREAM_STARTED / STREAM_STOPPED |
-| Bazarr | `470fb267-3436-4dd3-a70c-e6e8db1721be` | Subtitle events (Apprise JSON notifier) |
-| Tinker-Tickets | `6e306faf-8eea-4ba5-83ef-bf8f421f929e` | Custom transformation code |
-| Ci-Alert | `5e7051e7-fd0c-4930-a711-9c67daf6f0b9` | Gitea Actions `notify-failure` jobs (org secret `MATRIX_WEBHOOK_URL`); transform `hookshot/ci-alert.js` |
-| Devdesk | `4a3ccfc9-e1fe-4124-b014-7842fe51b9dc` | DevDesk "Claude needs you" (CT 126 `devdesk-notify-matrix`, URL in `/etc/devdesk/matrix-webhook`); transform `hookshot/devdesk.js`; avatar set via the appservice token |
+| Grafana | `df4a1302…` | Unified alerting contact point |
+| Proxmox | `9b3eafe5…` | Notification system (8.1+), Discord embed format |
+| Sonarr | `aeffc311…` | All event types |
+| Radarr | `34913454…` | All event types |
+| Readarr | `e57ab4f3…` | All event types |
+| Lidarr | `66ac6fdd…` | All event types |
+| Uptime Kuma | `1a02e890…` | Status change notifications |
+| Seerr | `555185af…` | Request/approval events |
+| Owncast (Livestream) | `9993e911…` | STREAM_STARTED / STREAM_STOPPED |
+| Bazarr | `470fb267…` | Subtitle events (Apprise JSON notifier) |
+| Tinker-Tickets | `6e306faf…` | Custom transformation code |
+| Ci-Alert | `5e7051e7…` | Gitea Actions `notify-failure` jobs (org secret `MATRIX_WEBHOOK_URL`); transform `hookshot/ci-alert.js` |
+| Devdesk | `4a3ccfc9…` | DevDesk "Claude needs you" (CT 126 `devdesk-notify-matrix`, URL in `/etc/devdesk/matrix-webhook`); transform `hookshot/devdesk.js`; avatar set via the appservice token |

 **Hookshot notes:**
 - Spam and Stuff is intentionally **unencrypted** — hookshot bridges cannot join E2EE rooms
@@ -984,6 +995,47 @@ Leaving a space does **not** leave its child rooms; they are independent rooms w

 ---

+## Hookshot Upgrade Notes
+
+Hookshot is a **git checkout built from source** at `/opt/hookshot` (`ExecStart=node lib/App/BridgeApp.js`).
+Only `config.yml` and `passkey.pem` in it are not generated; storage is in-memory (connections are room state).
+
+- **7.4.1+ builds with pnpm**, and `scripts/build-app.sh` calls `pnpm` directly. Without a global pnpm,
+  put a wrapper first in `PATH`: `printf '#!/bin/sh\nexec corepack pnpm@<version> "$@"\n'` (version from
+  `devEngines.packageManager` in `package.json`). The Rust layer needs `~/.cargo/bin` in `PATH`.
+- **7.6.0 drops Node 22** (LXC 151 runs Node 22), so 7.5.0 is the newest that runs as-is.
+- Build the new tag in a separate directory (`git clone --depth 1 --branch <tag> … /opt/hookshot-<tag>`,
+  `pnpm install --frozen-lockfile && pnpm run build`), copy in `config.yml` + `passkey.pem`, then stop the
+  service, swap the directories (keep the old one for rollback) and start it. Check the log says
+  `Bridge is now ready. Found N connections`.
+- 2026-11-03: 7.3.2 → 7.5.0 (GHSA-5c49-6fvm-2xvc: `/metrics` exposed webhook URLs, plus two DoS fixes).
+  Old build kept at `/opt/hookshot-7.3.2`.
+
+## Call Stack Upgrade (lk-jwt-service 0.4.1 → 0.9.0, LiveKit 1.9.11 → 1.13.11) — planned
+
+Tested on the local stack on 2026-11-03 (calls through the guard, voice-bridge e2e audio both ways,
+encrypted and not). Required changes, all on LXC 151:
+
+1. **lk-jwt-service 0.9.0** (a Rust rewrite; fixes GHSA-rq24-fh66-rq7x). Existing env vars carry over.
+   Add `LIVEKIT_SANITY_CHECK_INTERVAL_SECONDS=30`.
+2. **LiveKit webhook → lk-jwt.** 0.9.0 serves `/delegate_delayed_leave`, so Element Call hands it the
+   job of removing a crashed client's call membership and sets its own fallback timer to **1 hour**.
+   lk-jwt only learns that someone left from LiveKit webhooks; without them crashed clients linger for an
+   hour (also on Discord, via the voice bridge). In `/etc/livekit/config.yaml`:
+   ```yaml
+   webhook:
+     api_key: <the key under keys:, same as LIVEKIT_KEY>
+     urls:
+       - http://127.0.0.1:8071/sfu_webhook
+   ```
+3. **`prometheus_port: 6789` → `prometheus: {port: 6789}`.** LiveKit 1.13.8 removed the old key (1.9.11
+   already accepts the new one). Otherwise metrics and the *LiveKit Down* alert stop working.
+4. Restart lk-jwt-service only when no calls are active: its leave jobs are in memory (or set
+   `LIVEKIT_REDIS_URL`). LiveKit: replace the binary and `touch /run/livekit-restart-pending`.
+
+Not affected: TURN (external coturn with `ttl` set, as 1.13.1 requires), `room.auto_create: false`,
+the voice bridge (uses the legacy `/sfu/get`, still served).
+
 ## Synapse Upgrade Runbook

 Synapse is installed from the **matrix.org apt repo** (`matrix-synapse-py3`) on **LXC 151** — it is
@@ -1176,10 +1228,10 @@ All commands use the `!` prefix. Run `!help` in any room for the full list.
 | Database | PostgreSQL | 17.9 |
 | TURN | coturn | latest |
 | Video/voice calls | LiveKit SFU | 1.9.11 |
-| LiveKit JWT | lk-jwt-service | latest |
+| LiveKit JWT | lk-jwt-service | 0.4.1 (0.9.0 planned, see *Call stack upgrade*) |
 | Moderation | Draupnir | 2.9.0 |
 | SSO | Authelia (OIDC) + LLDAP | — |
-| Webhook bridge | matrix-hookshot | 7.3.2 |
+| Webhook bridge | matrix-hookshot | 7.5.0 |
 | Discord bridge | mautrix-discord | 0.7.7 |
 | Reverse proxy | Nginx Proxy Manager | — |
 | Web client | Lotus Cinny (fork of `cinnyapp/cinny` main) | custom |