security(deploy): webhook secrets out of git; all exposed ones rotated

deploy/hooks-lxc*.json in this public repo held the HMAC secrets for
every auto-deploy webhook (and 106's cinny-build token), readable
anonymously. Six values had been committed: the four matrix-deploy
secrets, 106's lotus-deploy secret (publicly reachable through
chat.lotusguild.org/hooks/lotus-deploy) and the cinny-build token.

- All six rotated on the servers and in Gitea (matrix hooks 6-9,
  cinny hook 69). The old values are rejected.
- Secrets now live in /etc/webhook/secrets.env (0600) on each LXC, loaded
  by a systemd drop-in that also enables webhook's -template mode; the
  hooks files use {{ getenv "NAME" | js }} placeholders.
- The repo hooks files are now exact copies of the live ones, so they
  also document the *-config-deploy hooks they were missing. Those
  three secrets were never committed and are unchanged.
- README: where the secrets live and how to rotate one.
- CI `hooks-no-secrets` fails if a literal secret lands in a hooks
  file again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPmy3tPq869XDW4njjVaKA

.gitea/workflows/lint.yml

@@ -31,6 +31,21 @@ jobs:
       - name: Run ESLint
         run: npx eslint --ext .js hookshot/
 
+  hooks-no-secrets:
+    name: No secrets in webhook configs
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+
+      # deploy/hooks-*.json are public; their secrets come from
+      # /etc/webhook/secrets.env on each LXC via `{{ getenv "..." }}`.
+      - name: Reject literal secrets
+        run: |
+          if grep -nE '[0-9a-fA-F]{32,}' deploy/hooks-*.json; then
+            echo "::error::A literal secret/token is committed in deploy/hooks-*.json. Use {{ getenv \"NAME\" | js }} and put the value in /etc/webhook/secrets.env on the LXC."
+            exit 1
+          fi
+
   python-lint:
     name: Python (ruff)
     runs-on: ubuntu-latest

README.md

@@ -159,7 +159,8 @@ Pushes to `main` on `LotusGuild/matrix` automatically deploy to the relevant LXC
 ### Installed Components (per LXC)
 
 - `webhook` binary (Debian package `webhook` v2.8.0) listening on respective port
-- `/etc/webhook/hooks.json` — unique HMAC-SHA256 secret per LXC
+- `/etc/webhook/hooks.json` — the `deploy/hooks-lxcNNN.json` file from this repo. It holds **no secrets**: each secret is a template placeholder (`{{ getenv "MATRIX_DEPLOY_SECRET" | js }}`), rendered by webhook's `-template` flag
+- `/etc/webhook/secrets.env` (root, `0600`, **not in git**) — `MATRIX_DEPLOY_SECRET` (plus `CONFIG_DEPLOY_SECRET` on 151/139/110, and `LOTUS_DEPLOY_SECRET` + `CINNY_BUILD_TOKEN` on 106). Loaded by the drop-in `/etc/systemd/system/webhook.service.d/secrets.conf` (`EnvironmentFile=` + `-template`); 106's `webhook-lotus.service` has the same drop-in
 - `/usr/local/bin/matrix-deploy.sh` — deploy script from this repo
 - `/etc/systemd/system/webhook.service` — enabled and running
 - `/opt/matrix-config/` — clone of this repo
@@ -461,7 +462,17 @@ EOF
 chmod 600 /etc/cinny-monitor.env
 ```
 
-**Cinny-build webhook token** (for LotusBot `!cinny-update`): stored in `deploy/hooks-lxc106.json` (`cinny-build` hook, header `X-Build-Token`). LotusBot must POST to `http://10.10.10.6:9000/hooks/cinny-build` with this header.
+**Cinny-build webhook token** (header `X-Build-Token` on the `cinny-build` hook): `CINNY_BUILD_TOKEN` in LXC 106's `/etc/webhook/secrets.env`, not in git. Reachable on `10.10.10.6:9001` (`webhook-lotus`); `:9000` is bound to `127.0.0.1`. Note: LotusBot doesn't implement `!cinny-update` at the moment, so nothing calls this hook.
+
+### Webhook secrets — rotating
+
+Secrets used to be committed in `deploy/hooks-lxc*.json` in this public repo; all exposed ones were rotated on 2026-09-23 and moved to `secrets.env`. To rotate one:
+
+1. Generate a value: `openssl rand -hex 32`.
+2. On the LXC: edit `/etc/webhook/secrets.env`, then `systemctl restart webhook` (and `webhook-lotus` on 106 — check no `lotus_deploy.sh` is running first, a restart kills it).
+3. In Gitea: the repo → Settings → Webhooks → that hook → set the same secret.
+
+CI (`hooks-no-secrets` in `.gitea/workflows/lint.yml`) fails if a 32-byte hex value is committed in a hooks file again.
 
 **Why 8GB RAM:** Vite's build process needs ~6GB Node heap (`--max_old_space_size=6144`) for the rendering-chunks phase. Previously at 4GB — OOM killed during render.
 

deploy/hooks-lxc106.json

@@ -7,7 +7,7 @@
     "trigger-rule": {
       "match": {
         "type": "payload-hash-sha256",
-        "secret": "76dd5febd1cc3458545ce37537f4bfe26f241a9635b57a2cba183ebc9221230b",
+        "secret": "{{ getenv "MATRIX_DEPLOY_SECRET" | js }}",
         "parameter": {
           "source": "header",
           "name": "X-Gitea-Signature"
@@ -23,7 +23,7 @@
     "trigger-rule": {
       "match": {
         "type": "value",
-        "value": "a82340fc2f07e6afda097494c34aa3a4877924932a0b063a76106fdab9816ec6",
+        "value": "{{ getenv "CINNY_BUILD_TOKEN" | js }}",
         "parameter": {
           "source": "header",
           "name": "X-Build-Token"
@@ -41,7 +41,7 @@
         {
           "match": {
             "type": "payload-hash-sha256",
-            "secret": "7b333a1bf4c02aacece65de00e6d8c3e86881f1254f4efa91aeddb82b67dd50f",
+            "secret": "{{ getenv "LOTUS_DEPLOY_SECRET" | js }}",
             "parameter": {
               "source": "header",
               "name": "X-Gitea-Signature"

deploy/hooks-lxc110.json

@@ -7,12 +7,42 @@
     "trigger-rule": {
       "match": {
         "type": "payload-hash-sha256",
-        "secret": "0d23fab8743e9ee6b52cbd05a889b04c927ffa2b2b21fe50244f1a534d1a22d0",
+        "secret": "{{ getenv "MATRIX_DEPLOY_SECRET" | js }}",
         "parameter": {
           "source": "header",
           "name": "X-Gitea-Signature"
         }
       }
     }
+  },
+  {
+    "id": "110-config-deploy",
+    "execute-command": "/usr/local/bin/draupnir_deploy.sh",
+    "command-working-directory": "/opt/pve-infra",
+    "response-message": "Deploying draupnir config...",
+    "trigger-rule": {
+      "and": [
+        {
+          "match": {
+            "type": "payload-hash-sha256",
+            "secret": "{{ getenv "CONFIG_DEPLOY_SECRET" | js }}",
+            "parameter": {
+              "source": "header",
+              "name": "X-Gitea-Signature"
+            }
+          }
+        },
+        {
+          "match": {
+            "type": "value",
+            "value": "refs/heads/main",
+            "parameter": {
+              "source": "payload",
+              "name": "ref"
+            }
+          }
+        }
+      ]
+    }
   }
 ]

deploy/hooks-lxc139.json

@@ -7,12 +7,42 @@
     "trigger-rule": {
       "match": {
         "type": "payload-hash-sha256",
-        "secret": "ddea576ef03bff35f0c9d138b626b273d9e9502434e0717899a87677cd5ac267",
+        "secret": "{{ getenv "MATRIX_DEPLOY_SECRET" | js }}",
         "parameter": {
           "source": "header",
           "name": "X-Gitea-Signature"
         }
       }
     }
+  },
+  {
+    "id": "139-config-deploy",
+    "execute-command": "/usr/local/bin/nginxproxymanager_deploy.sh",
+    "command-working-directory": "/opt/pve-infra",
+    "response-message": "Deploying nginxproxymanager config...",
+    "trigger-rule": {
+      "and": [
+        {
+          "match": {
+            "type": "payload-hash-sha256",
+            "secret": "{{ getenv "CONFIG_DEPLOY_SECRET" | js }}",
+            "parameter": {
+              "source": "header",
+              "name": "X-Gitea-Signature"
+            }
+          }
+        },
+        {
+          "match": {
+            "type": "value",
+            "value": "refs/heads/main",
+            "parameter": {
+              "source": "payload",
+              "name": "ref"
+            }
+          }
+        }
+      ]
+    }
   }
 ]

deploy/hooks-lxc151.json

@@ -7,12 +7,42 @@
     "trigger-rule": {
       "match": {
         "type": "payload-hash-sha256",
-        "secret": "38ba0e66763da2096c47645cbf636ce3c2c51232e006b964e57d6bb94a32dcaa",
+        "secret": "{{ getenv "MATRIX_DEPLOY_SECRET" | js }}",
         "parameter": {
           "source": "header",
           "name": "X-Gitea-Signature"
         }
       }
     }
+  },
+  {
+    "id": "151-config-deploy",
+    "execute-command": "/usr/local/bin/matrix_deploy.sh",
+    "command-working-directory": "/opt/pve-infra",
+    "response-message": "Deploying matrix config...",
+    "trigger-rule": {
+      "and": [
+        {
+          "match": {
+            "type": "payload-hash-sha256",
+            "secret": "{{ getenv "CONFIG_DEPLOY_SECRET" | js }}",
+            "parameter": {
+              "source": "header",
+              "name": "X-Gitea-Signature"
+            }
+          }
+        },
+        {
+          "match": {
+            "type": "value",
+            "value": "refs/heads/main",
+            "parameter": {
+              "source": "payload",
+              "name": "ref"
+            }
+          }
+        }
+      ]
+    }
   }
 ]