fix(cinny-web): per-client rate limits; own budget for decorations

Every request reaches LXC 106 from Nginx Proxy Manager (10.10.10.27), and
nginx had no realip config, so chat_limit (20 r/s, burst 60, keyed by
$binary_remote_addr) was ONE bucket for the whole community: a scanner
burst 503ed real users. Trust X-Real-IP from the proxy only, in both
server blocks.

Decorations (#22) now come from this host, and the profile picker can
fetch all 629 thumbnails: ~500 of them 503ed in testing. /decorations/
gets its own zone (100 r/s, burst 700) instead of chat_limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5

README.md

@@ -67,7 +67,7 @@ matrix/
 - coturn config: `/etc/turnserver.conf`
 - LiveKit config: `/etc/livekit/config.yaml`
 - LiveKit service: `livekit-server.service`
-- lk-jwt-service: `lk-jwt-service.service` (now binds `:8071` via drop-in `/etc/systemd/system/lk-jwt-service.service.d/override.conf`; serves JWT tokens for MatrixRTC at `/sfu/get` and legacy `/get_token`)
+- lk-jwt-service: `lk-jwt-service.service` (now binds `127.0.0.1:8071` via drop-in `/etc/systemd/system/lk-jwt-service.service.d/override.conf`; serves JWT tokens for MatrixRTC at `/sfu/get` and legacy `/get_token`)
 - voice-limit-guard: `voice-limit-guard.service` (binds `:8070`, fronts lk-jwt-service — enforces hard per-room voice participant limits **and publish permissions (screenshare/camera via JWT re-signing)** for ALL clients; script `/opt/voice-limit-guard/voice-limit-guard.py`) — see [Voice Channel Limits & Call Permissions](#voice-channel-limits--call-permissions)
 - Hookshot: `/opt/hookshot/`, service: `matrix-hookshot.service`
 - Hookshot config: `/opt/hookshot/config.yml`
@@ -238,7 +238,9 @@ install -D -m644 /opt/matrix-config/livekit/voice-limit-guard.py /opt/voice-limi
 install -m644 /opt/matrix-config/systemd/voice-limit-guard.service /etc/systemd/system/voice-limit-guard.service
 # one-time: rebind lk-jwt-service to :8071
 mkdir -p /etc/systemd/system/lk-jwt-service.service.d
-printf '[Service]\nEnvironment=LIVEKIT_JWT_BIND=:8071\n' > /etc/systemd/system/lk-jwt-service.service.d/override.conf
+# Loopback only (matrix #24): the guard runs on this LXC; binding all
+# interfaces let any LAN host mint unguarded tokens straight from :8071.
+printf '[Service]\nEnvironment=LIVEKIT_JWT_BIND=127.0.0.1:8071\n' > /etc/systemd/system/lk-jwt-service.service.d/override.conf
 systemctl daemon-reload && systemctl restart lk-jwt-service && systemctl enable --now voice-limit-guard
 ```
 
@@ -290,7 +292,7 @@ The token in `draupnir/production.yaml` in this repo is **intentionally redacted
 | 7880 | LiveKit HTTP | 0.0.0.0 |
 | 7881 | LiveKit RTC TCP | 0.0.0.0 |
 | 8070 | voice-limit-guard (fronts lk-jwt-service) | 0.0.0.0 |
-| 8071 | lk-jwt-service (behind guard) | 0.0.0.0 |
+| 8071 | lk-jwt-service (behind guard) | 127.0.0.1 (matrix #24; was 0.0.0.0) |
 | 8080 | synapse-admin (nginx) | 0.0.0.0 |
 | 3478 | coturn STUN/TURN | 0.0.0.0 |
 | 5349 | coturn TURNS/TLS | 0.0.0.0 |
@@ -470,6 +472,16 @@ chmod 600 /etc/cinny-monitor.env
 
 **Cinny-build webhook token** (header `X-Build-Token` on the `cinny-build` hook): `CINNY_BUILD_TOKEN` in LXC 106's `/etc/webhook/secrets.env`, not in git. Reachable on `10.10.10.6:9001` (`webhook-lotus`); `:9000` is bound to `127.0.0.1`. Note: LotusBot doesn't implement `!cinny-update` at the moment, so nothing calls this hook.
 
+### Avatar decorations (matrix #22)
+
+Served from **`https://chat.lotusguild.org/decorations/`**: `<slug>.png` and `thumbs/<slug>.webp`, `Cache-Control: public, max-age=604800`, no cookies. They used to come through Nextcloud's public DAV, which started a PHP session per image and sent no cache headers.
+
+- **Storage:** Gitea generic package `avatar-decorations` (`https://code.lotusguild.org/api/packages/LotusGuild/generic/avatar-decorations/<version>/`), in ≤150 MB tar parts (the proxy refuses larger uploads). Nothing goes in git except the pin.
+- **Pin:** `cinny/decorations.sha256`, the sha256 of each part, which names the version.
+- **Deploy:** `deploy/lxc106-cinny.sh` installs the pinned version when the pin changes (or nothing is installed yet). It downloads the parts, checks them against the pin and against the `MANIFEST.sha256` inside, unpacks to `/var/www/decorations-<version>`, and atomically repoints `/var/www/decorations`. The previous version is kept for rollback; a failed download or checksum leaves the live set unchanged. nginx: `location ^~ /decorations/` in `cinny/nginx.conf`.
+- **Updating the set:** put the files in a folder (`<slug>.png` + `thumbs/<slug>.webp`) and run `GITEA_TOKEN=… deploy/publish-decorations.sh <folder> <new-version>`. That rewrites the pin; commit it. Add or remove the catalog entries in cinny's `src/app/features/lotus/avatarDecorations.ts`. Use a new version for every change: an image replaced under the same name can be cached for up to a week.
+- **Rollback:** revert `cinny/decorations.sha256`, or on LXC 106 run `ln -sfn /var/www/decorations-<old>/decorations /var/www/decorations`.
+
 ### Webhook secrets — rotating
 
 Secrets used to be committed in `deploy/hooks-lxc*.json` in this public repo; all exposed ones were rotated on 2026-09-23 and moved to `secrets.env`. To rotate one:

cinny/config.json

@@ -8,9 +8,16 @@
   "allowCustomHomeservers": false,
   "featuredCommunities": {
     "openAsDefault": false,
-    "spaces": [],
-    "rooms": [],
-    "servers": []
+    "spaces": [
+      "!-1ZBnAH-JiCOV8MGSKN77zDGTuI3pgSdy8Unu_DrDyc",
+      "#homelab:codestorm.net"
+    ],
+    "rooms": [
+      "#jellyfin:matrix.org"
+    ],
+    "servers": [
+      "matrixrooms.info"
+    ]
   },
   "hashRouter": {
     "enabled": false,

cinny/decorations.sha256

@@ -0,0 +1,4 @@
+a7aacaa65cb06e4b1c990b9253362f2eb148aae72e2e11c396d34e899792d68d  avatar-decorations-1.0.0.part1.tar
+a6919d8bb8e1c2f838b0f9acdbc1131cbfd7bfe5fe9c373b25f406474bd41095  avatar-decorations-1.0.0.part2.tar
+fb7624f2470659d933463187d2afe73428d5554291f62484c8a28fac6196f427  avatar-decorations-1.0.0.part3.tar
+909e07db0134b29258514acf4d598bb467aff8b157f0767b6c3afeadc1eb048c  avatar-decorations-1.0.0.part4.tar

cinny/lotus_deploy.sh

@@ -160,7 +160,13 @@ echo "[$(date '+%Y-%m-%d %H:%M:%S')] Deploying to $WEBROOT..."
 # /var/www/html/config.json by lxc106-cinny.sh. The build ships a DEV default
 # (allowCustomHomeservers:true); rsyncing it would clobber the production config
 # on every deploy. Keep the app bundle and the runtime config separate.
-rsync -a --delete --exclude config.json dist/ "$WEBROOT/"
+# [matrix #23] Keep the previous builds' hashed assets for a while: a browser
+# still holding an older index.html (or service-worker precache) asks for its
+# assets/index-<old hash>.js, and deleting them on every deploy broke the app
+# for those users until their cached shell expired. Prune after 14 days.
+rsync -a --delete --exclude config.json --exclude assets/ dist/ "$WEBROOT/"
+rsync -a dist/assets/ "$WEBROOT/assets/"
+find "$WEBROOT/assets" -type f -mtime +14 -delete
 
 echo "[$(date '+%Y-%m-%d %H:%M:%S')] ===== Deploy complete ($VITE_APP_VERSION) ====="
 

cinny/nginx-security-headers.conf

@@ -12,4 +12,4 @@ add_header X-XSS-Protection "1; mode=block" always;
 add_header Referrer-Policy strict-origin-when-cross-origin always;
 add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload" always;
 add_header Permissions-Policy 'accelerometer=(), autoplay=(self "https://call.chat.lotusguild.org"), camera=(self "https://call.chat.lotusguild.org"), display-capture=(self "https://call.chat.lotusguild.org"), encrypted-media=(self), fullscreen=(self), geolocation=(self), gyroscope=(), magnetometer=(), microphone=(self "https://call.chat.lotusguild.org"), midi=(), payment=(), usb=()' always;
-add_header Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval' blob:; style-src 'self' 'unsafe-inline'; img-src 'self' data: blob: https://matrix.lotusguild.org https://matrix.org https://*.matrix.org https://mozilla.org https://mozilla.modular.im https://drive.lotusguild.org https://media.giphy.com https://media0.giphy.com https://media1.giphy.com https://media2.giphy.com https://media3.giphy.com https://media4.giphy.com https://www.openstreetmap.org https://tile.openstreetmap.org; font-src 'self' data:; connect-src 'self' https://matrix.lotusguild.org wss://matrix.lotusguild.org https://isitup.lotusguild.org https://matrix.org https://*.matrix.org https://mozilla.org https://mozilla.modular.im https://chat.mozilla.org https://vector.im https://api.giphy.com https://*.giphy.com wss:; media-src 'self' https: blob:; frame-src 'self' https:; worker-src 'self' blob:; object-src 'none'; frame-ancestors 'none'; base-uri 'self'; form-action 'self';" always;
+add_header Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval' blob:; style-src 'self' 'unsafe-inline'; img-src 'self' data: blob: https://matrix.lotusguild.org https://matrix.org https://*.matrix.org https://mozilla.org https://mozilla.modular.im https://drive.lotusguild.org https://media.giphy.com https://media0.giphy.com https://media1.giphy.com https://media2.giphy.com https://media3.giphy.com https://media4.giphy.com https://www.openstreetmap.org https://tile.openstreetmap.org; font-src 'self' data:; connect-src 'self' https://matrix.lotusguild.org wss://matrix.lotusguild.org https://isitup.lotusguild.org https://matrix.org https://*.matrix.org https://mozilla.org https://mozilla.modular.im https://chat.mozilla.org https://www.mozilla.org https://vector.im https://api.giphy.com https://*.giphy.com https://www.tiktok.com wss:; media-src 'self' https: blob:; frame-src 'self' https:; worker-src 'self' blob:; object-src 'none'; frame-ancestors 'none'; base-uri 'self'; form-action 'self';" always;

cinny/nginx.conf

@@ -1,3 +1,23 @@
+# [matrix #23] SPA routes (/, /home/…, /login/…) are rewritten to /index.html
+# inside `location /`, where no cache headers were set: browsers then reused a
+# heuristically fresh shell after a deploy and requested deleted assets. Only
+# the rewritten shell gets no-cache; other files there keep their defaults.
+map $uri $lotus_shell_cache_control {
+    /index.html "no-cache, no-store, must-revalidate";
+    default     "";
+}
+
+# All traffic arrives through Nginx Proxy Manager (10.10.10.27), so without
+# realip every visitor shared ONE chat_limit bucket (20 r/s, keyed by
+# $binary_remote_addr = the proxy): a scanner burst got real users 503s.
+# Each server block below trusts X-Real-IP from the proxy only (NPM sets it
+# to the client address, overwriting anything the client sent).
+#
+# [matrix #22] Decorations get their own, larger budget: opening the profile
+# decoration picker can fetch hundreds of thumbnails, which would otherwise use
+# up chat_limit and 503 the app's own files.
+limit_req_zone $binary_remote_addr zone=decorations_limit:10m rate=100r/s;
+
 server {
     listen 80;
     listen [::]:80;
@@ -14,6 +34,8 @@ server {
     server_tokens off;
     client_max_body_size 50m;
 
+    set_real_ip_from 10.10.10.27;
+    real_ip_header X-Real-IP;
     limit_req zone=chat_limit burst=60 nodelay;
     limit_conn chat_conn 25;
     index index.html;
@@ -98,7 +120,24 @@ server {
         proxy_connect_timeout 5;
     }
 
+    # [matrix #22] Avatar decorations, installed by deploy/lxc106-cinny.sh from
+    # the Gitea generic package (outside the web root, so app deploys'
+    # rsync --delete never touches them). ^~ beats the image regex below: these
+    # can be replaced in place under the same name, so a week, not immutable.
+    location ^~ /decorations/ {
+        alias /var/www/decorations/;
+        # Replaces the server-level chat_limit here (nginx doesn't inherit
+        # limit_req into a location that sets its own).
+        limit_req zone=decorations_limit burst=700 nodelay;
+        include snippets/cinny-security-headers.conf;
+        add_header Cache-Control "public, max-age=604800" always;
+        location ~ /MANIFEST\.sha256$ { return 404; }
+    }
+
     location / {
+        # add_header here drops the server-level headers, so include them again.
+        include snippets/cinny-security-headers.conf;
+        add_header Cache-Control $lotus_shell_cache_control always;
         rewrite ^/config\.json$          /config.json break;
         rewrite ^/manifest\.json$        /manifest.json break;
         rewrite ^/sw\.js$                /sw.js break;
@@ -128,6 +167,8 @@ server {
 
     root /var/www/html;
     server_tokens off;
+    set_real_ip_from 10.10.10.27;
+    real_ip_header X-Real-IP;
     limit_req zone=chat_limit burst=60 nodelay;
     limit_conn chat_conn 25;
 

deploy/lxc106-cinny.sh

@@ -16,7 +16,7 @@ echo "=== $(date) === LXC106 deploy triggered ==="
 
 if [ ! -d "$REPO_DIR/.git" ]; then
     git clone "$CLONE_URL" "$REPO_DIR"
-    CHANGED="cinny/config.json cinny/nginx.conf cinny/upstream-check.sh cinny/lotus-build.sh cinny/lotus_deploy.sh deploy/hooks-lxc106.json systemd/cinny-upstream-check.cron"
+    CHANGED="cinny/config.json cinny/decorations.sha256 cinny/nginx.conf cinny/upstream-check.sh cinny/lotus-build.sh cinny/lotus_deploy.sh deploy/hooks-lxc106.json systemd/cinny-upstream-check.cron"
 else
     cd "$REPO_DIR"
     git fetch --all
@@ -63,6 +63,55 @@ PY
     fi
 fi
 
+# [matrix #22] Avatar decorations: a versioned Gitea generic package, pinned
+# (version + sha256 of each part) in cinny/decorations.sha256, installed to
+# /var/www/decorations-<version> and served by nginx at /decorations/. Runs on a
+# pin change or when nothing is installed yet. A failed download or checksum
+# leaves the live set untouched; the previous version is kept for rollback.
+install_decorations() {
+    local pin="$REPO_DIR/cinny/decorations.sha256"
+    local ver base tmp dest
+    ver=$(grep -oE 'avatar-decorations-[0-9][^ ]*\.part1\.tar' "$pin" | sed -E 's/^avatar-decorations-(.*)\.part1\.tar$/\1/')
+    [ -n "$ver" ] || { echo "✗ decorations: no version in $pin"; return 1; }
+    dest="/var/www/decorations-$ver"
+    if [ -d "$dest/decorations" ] && [ "$(readlink -f /var/www/decorations)" = "$dest/decorations" ]; then
+        echo "decorations $ver already live"; return 0
+    fi
+    base="https://code.lotusguild.org/api/packages/LotusGuild/generic/avatar-decorations/$ver"
+    tmp=$(mktemp -d /var/www/.decorations-XXXXXX)
+    # Every step checks its own exit status: `set -e` does not apply inside a
+    # subshell on the left of `||`, so a failed checksum would be ignored.
+    if ! (
+        cd "$tmp" || exit 1
+        cp "$pin" SHA256SUMS || exit 1
+        while read -r _ f; do
+            curl -fsSL --retry 3 -o "$f" "$base/$f" || exit 1
+        done < SHA256SUMS
+        sha256sum -c --quiet SHA256SUMS || exit 1
+        while read -r _ f; do
+            tar -xf "$f" || exit 1
+            rm -f "$f"
+        done < SHA256SUMS
+        cd decorations || exit 1
+        sha256sum -c --quiet MANIFEST.sha256 || exit 1
+    ); then
+        echo "✗ decorations $ver: download/verify failed — live set unchanged"
+        rm -rf "$tmp"
+        return 1
+    fi
+    rm -rf "$dest"; mkdir -p "$dest"; mv "$tmp/decorations" "$dest/decorations"; rm -rf "$tmp"
+    chmod -R a+rX "$dest"
+    ln -sfn "$dest/decorations" /var/www/decorations.new && mv -T /var/www/decorations.new /var/www/decorations
+    # Keep the live version and the one before it; drop older ones.
+    find /var/www -maxdepth 1 -name 'decorations-*' -type d -printf '%T@ %p\n' | sort -rn \
+        | tail -n +3 | cut -d' ' -f2- | xargs -r rm -rf
+    echo "✓ decorations $ver live at /var/www/decorations ($(find "$dest/decorations" -maxdepth 1 -name '*.png' | wc -l) images)"
+}
+if echo "$CHANGED" | grep -q '^cinny/decorations.sha256' || [ ! -e /var/www/decorations ]; then
+    echo "Deploying avatar decorations..."
+    install_decorations || true
+fi
+
 if echo "$CHANGED" | grep -q '^cinny/nginx.conf'; then
     echo "Deploying cinny nginx site config..."
     # Back up the live config, swap in the repo copy, and validate before

deploy/publish-decorations.sh

@@ -0,0 +1,56 @@
+#!/bin/bash
+# Publish a new avatar-decorations version to the Gitea generic package
+# registry and write the pin that LXC 106's deploy installs (matrix #22).
+#
+#   GITEA_TOKEN=… deploy/publish-decorations.sh <dir> <version>
+#
+# <dir> holds <slug>.png files and thumbs/<slug>.webp (the same layout the app
+# requests). The set is split into ≤150 MB tar parts (the registry's proxy
+# refuses large uploads), each part is uploaded, and cinny/decorations.sha256
+# is rewritten. Commit that file to deploy; nothing changes until you do.
+set -euo pipefail
+SRC=${1:?usage: publish-decorations.sh <dir> <version>}
+VER=${2:?usage: publish-decorations.sh <dir> <version>}
+: "${GITEA_TOKEN:?set GITEA_TOKEN (package write scope)}"
+REPO_ROOT=$(cd "$(dirname "$0")/.." && pwd)
+BASE="https://code.lotusguild.org/api/packages/LotusGuild/generic/avatar-decorations/$VER"
+WORK=$(mktemp -d)
+trap 'rm -rf "$WORK"' EXIT
+
+mkdir -p "$WORK/decorations/thumbs"
+cp "$SRC"/*.png "$WORK/decorations/"
+cp "$SRC"/thumbs/*.webp "$WORK/decorations/thumbs/"
+cd "$WORK"
+for f in decorations/*.png; do
+    s=$(basename "$f" .png)
+    [ -f "decorations/thumbs/$s.webp" ] || { echo "missing thumbs/$s.webp" >&2; exit 1; }
+done
+(cd decorations && find . -type f \( -name '*.png' -o -name '*.webp' \) | sort | xargs sha256sum > MANIFEST.sha256)
+
+python3 - "$VER" <<'PY'
+import os, subprocess, sys
+ver, root, limit = sys.argv[1], 'decorations', 150 * 1024 * 1024
+files = sorted(os.path.join(root, f) for f in os.listdir(root) if f.endswith('.png'))
+first = [os.path.join(root, 'MANIFEST.sha256')] + sorted(os.path.join(root, 'thumbs', t) for t in os.listdir(os.path.join(root, 'thumbs')))
+parts, cur, size = [], list(first), sum(os.path.getsize(f) for f in first)
+for f in files:
+    if size + os.path.getsize(f) > limit and cur:
+        parts.append(cur); cur, size = [], 0
+    cur.append(f); size += os.path.getsize(f)
+parts.append(cur)
+for i, p in enumerate(parts, 1):
+    lst = f'part{i}.list'
+    open(lst, 'w').write('\n'.join(p) + '\n')
+    subprocess.run(['tar', '--sort=name', '--owner=0', '--group=0', '--mtime=@0', '-cf',
+                    f'avatar-decorations-{ver}.part{i}.tar', '-T', lst], check=True)
+PY
+
+sha256sum avatar-decorations-"$VER".part*.tar > SHA256SUMS
+for f in avatar-decorations-"$VER".part*.tar SHA256SUMS; do
+    code=$(curl -s -o /dev/null -w '%{http_code}' -X PUT -H "Authorization: token $GITEA_TOKEN" \
+        --upload-file "$f" "$BASE/$f")
+    [ "$code" = 201 ] || { echo "upload of $f failed: HTTP $code" >&2; exit 1; }
+    echo "uploaded $f"
+done
+cp SHA256SUMS "$REPO_ROOT/cinny/decorations.sha256"
+echo "Pinned $VER in cinny/decorations.sha256: commit it to deploy."