diff --git a/.gitea/workflows/lint.yml b/.gitea/workflows/lint.yml index bd05f48..3305d5e 100644 --- a/.gitea/workflows/lint.yml +++ b/.gitea/workflows/lint.yml @@ -31,6 +31,20 @@ jobs: - name: Run ESLint run: npx eslint --ext .js hookshot/ + readme-no-webhook-ids: + name: No Hookshot webhook IDs in README + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + + # A Hookshot webhook URL's UUID is its only secret, and this repo is public. + - name: Reject full UUIDs + run: | + if grep -nE '[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}' README.md; then + echo "::error::README.md contains a full UUID. Webhook IDs are secrets: list only the first 8 characters." + exit 1 + fi + hooks-no-secrets: name: No secrets in webhook configs runs-on: ubuntu-latest diff --git a/README.md b/README.md index 5d787aa..85be0e8 100644 --- a/README.md +++ b/README.md @@ -191,9 +191,14 @@ Killing livekit-server while a call is active drops everyone. Instead: 1. Deploy to LXC 151 copies the new `livekit-server.service` and sets a `/run/livekit-restart-pending` flag 2. `livekit-graceful-restart.timer` runs every 5 minutes -3. The timer script counts established TCP connections on port 7881 (`ss -tn state established`) -4. If zero connections → restarts livekit-server and clears the flag -5. If connections exist → logs and exits, retries in 5 minutes +3. The timer script reads LiveKit's own `livekit_participant_total` (`:6789/metrics`) and counts open + signalling WebSockets on :7880. If the metrics can't be read it assumes a call is active. +4. Zero participants and zero signalling connections → restarts livekit-server and clears the flag +5. Otherwise → logs and exits, retries in 5 minutes + +Until 2026-11-03 it counted `ESTAB` lines on :7881, which was always 0 (an `ss` state filter drops the +State column, and :7881 is only the ICE-TCP fallback), so it would have restarted mid-call (matrix #54). +To stage a restart by hand, for example after replacing the binary: `touch /run/livekit-restart-pending`. --- @@ -343,27 +348,33 @@ The token in `draupnir/production.yaml` in this repo is **intentionally redacted --- -## Webhook Integrations (matrix-hookshot 7.3.2) +## Webhook Integrations (matrix-hookshot 7.5.0) Generic webhooks bridged into **Spam and Stuff**. Each service gets its own virtual user (`@hookshot_`) with a unique avatar. Webhook URL format: `https://matrix.lotusguild.org/webhook/` +> The UUID **is** the secret: anyone who has it can post into the room. This repo is public, so +> the table lists only the first 8 characters (enough to match Hookshot's logs); the full URLs +> live in each sender's settings. Full UUIDs were in this file from 2026-02-20 to 2026-11-03, +> so the IDs below are exposed in git history until rotated (see *Webhook secrets — rotating*). +> CI (`readme-no-webhook-ids`) rejects full UUIDs in this README. + | Service | Webhook UUID | Notes | |---------|-------------|-------| -| Grafana | `df4a1302-2d62-4a01-b858-fb56f4d3781a` | Unified alerting contact point | -| Proxmox | `9b3eafe5-7689-4011-addd-c466e524661d` | Notification system (8.1+), Discord embed format | -| Sonarr | `aeffc311-0686-42cb-9eeb-6757140c072e` | All event types | -| Radarr | `34913454-c1ac-4cda-82ea-924d4a9e60eb` | All event types | -| Readarr | `e57ab4f3-56e6-4dc4-8b30-2f4fd4bbeb0b` | All event types | -| Lidarr | `66ac6fdd-69f6-4f47-bb00-b7f6d84d7c1c` | All event types | -| Uptime Kuma | `1a02e890-bb25-42f1-99fe-bba6a19f1811` | Status change notifications | -| Seerr | `555185af-90a1-42ff-aed5-c344e11955cf` | Request/approval events | -| Owncast (Livestream) | `9993e911-c68b-4271-a178-c2d65ca88499` | STREAM_STARTED / STREAM_STOPPED | -| Bazarr | `470fb267-3436-4dd3-a70c-e6e8db1721be` | Subtitle events (Apprise JSON notifier) | -| Tinker-Tickets | `6e306faf-8eea-4ba5-83ef-bf8f421f929e` | Custom transformation code | -| Ci-Alert | `5e7051e7-fd0c-4930-a711-9c67daf6f0b9` | Gitea Actions `notify-failure` jobs (org secret `MATRIX_WEBHOOK_URL`); transform `hookshot/ci-alert.js` | -| Devdesk | `4a3ccfc9-e1fe-4124-b014-7842fe51b9dc` | DevDesk "Claude needs you" (CT 126 `devdesk-notify-matrix`, URL in `/etc/devdesk/matrix-webhook`); transform `hookshot/devdesk.js`; avatar set via the appservice token | +| Grafana | `df4a1302…` | Unified alerting contact point | +| Proxmox | `9b3eafe5…` | Notification system (8.1+), Discord embed format | +| Sonarr | `aeffc311…` | All event types | +| Radarr | `34913454…` | All event types | +| Readarr | `e57ab4f3…` | All event types | +| Lidarr | `66ac6fdd…` | All event types | +| Uptime Kuma | `1a02e890…` | Status change notifications | +| Seerr | `555185af…` | Request/approval events | +| Owncast (Livestream) | `9993e911…` | STREAM_STARTED / STREAM_STOPPED | +| Bazarr | `470fb267…` | Subtitle events (Apprise JSON notifier) | +| Tinker-Tickets | `6e306faf…` | Custom transformation code | +| Ci-Alert | `5e7051e7…` | Gitea Actions `notify-failure` jobs (org secret `MATRIX_WEBHOOK_URL`); transform `hookshot/ci-alert.js` | +| Devdesk | `4a3ccfc9…` | DevDesk "Claude needs you" (CT 126 `devdesk-notify-matrix`, URL in `/etc/devdesk/matrix-webhook`); transform `hookshot/devdesk.js`; avatar set via the appservice token | **Hookshot notes:** - Spam and Stuff is intentionally **unencrypted** — hookshot bridges cannot join E2EE rooms @@ -984,6 +995,47 @@ Leaving a space does **not** leave its child rooms; they are independent rooms w --- +## Hookshot Upgrade Notes + +Hookshot is a **git checkout built from source** at `/opt/hookshot` (`ExecStart=node lib/App/BridgeApp.js`). +Only `config.yml` and `passkey.pem` in it are not generated; storage is in-memory (connections are room state). + +- **7.4.1+ builds with pnpm**, and `scripts/build-app.sh` calls `pnpm` directly. Without a global pnpm, + put a wrapper first in `PATH`: `printf '#!/bin/sh\nexec corepack pnpm@ "$@"\n'` (version from + `devEngines.packageManager` in `package.json`). The Rust layer needs `~/.cargo/bin` in `PATH`. +- **7.6.0 drops Node 22** (LXC 151 runs Node 22), so 7.5.0 is the newest that runs as-is. +- Build the new tag in a separate directory (`git clone --depth 1 --branch … /opt/hookshot-`, + `pnpm install --frozen-lockfile && pnpm run build`), copy in `config.yml` + `passkey.pem`, then stop the + service, swap the directories (keep the old one for rollback) and start it. Check the log says + `Bridge is now ready. Found N connections`. +- 2026-11-03: 7.3.2 → 7.5.0 (GHSA-5c49-6fvm-2xvc: `/metrics` exposed webhook URLs, plus two DoS fixes). + Old build kept at `/opt/hookshot-7.3.2`. + +## Call Stack Upgrade (lk-jwt-service 0.4.1 → 0.9.0, LiveKit 1.9.11 → 1.13.11) — planned + +Tested on the local stack on 2026-11-03 (calls through the guard, voice-bridge e2e audio both ways, +encrypted and not). Required changes, all on LXC 151: + +1. **lk-jwt-service 0.9.0** (a Rust rewrite; fixes GHSA-rq24-fh66-rq7x). Existing env vars carry over. + Add `LIVEKIT_SANITY_CHECK_INTERVAL_SECONDS=30`. +2. **LiveKit webhook → lk-jwt.** 0.9.0 serves `/delegate_delayed_leave`, so Element Call hands it the + job of removing a crashed client's call membership and sets its own fallback timer to **1 hour**. + lk-jwt only learns that someone left from LiveKit webhooks; without them crashed clients linger for an + hour (also on Discord, via the voice bridge). In `/etc/livekit/config.yaml`: + ```yaml + webhook: + api_key: + urls: + - http://127.0.0.1:8071/sfu_webhook + ``` +3. **`prometheus_port: 6789` → `prometheus: {port: 6789}`.** LiveKit 1.13.8 removed the old key (1.9.11 + already accepts the new one). Otherwise metrics and the *LiveKit Down* alert stop working. +4. Restart lk-jwt-service only when no calls are active: its leave jobs are in memory (or set + `LIVEKIT_REDIS_URL`). LiveKit: replace the binary and `touch /run/livekit-restart-pending`. + +Not affected: TURN (external coturn with `ttl` set, as 1.13.1 requires), `room.auto_create: false`, +the voice bridge (uses the legacy `/sfu/get`, still served). + ## Synapse Upgrade Runbook Synapse is installed from the **matrix.org apt repo** (`matrix-synapse-py3`) on **LXC 151** — it is @@ -1176,10 +1228,10 @@ All commands use the `!` prefix. Run `!help` in any room for the full list. | Database | PostgreSQL | 17.9 | | TURN | coturn | latest | | Video/voice calls | LiveKit SFU | 1.9.11 | -| LiveKit JWT | lk-jwt-service | latest | +| LiveKit JWT | lk-jwt-service | 0.4.1 (0.9.0 planned, see *Call stack upgrade*) | | Moderation | Draupnir | 2.9.0 | | SSO | Authelia (OIDC) + LLDAP | — | -| Webhook bridge | matrix-hookshot | 7.3.2 | +| Webhook bridge | matrix-hookshot | 7.5.0 | | Discord bridge | mautrix-discord | 0.7.7 | | Reverse proxy | Nginx Proxy Manager | — | | Web client | Lotus Cinny (fork of `cinnyapp/cinny` main) | custom |