Lotus Matrix Infrastructure

Matrix server infrastructure for the Lotus Guild homeserver (matrix.lotusguild.org).

Repo: https://code.lotusguild.org/LotusGuild/matrix

---

Repo Structure

matrix/
├── hookshot/           # Hookshot JS transformation functions (one file per webhook)
│   ├── deploy.sh       # Deploys all .js files to Matrix room state via API
│   ├── proxmox.js
│   ├── grafana.js
│   ├── uptime-kuma.js
│   └── ...             # One .js per webhook service
├── cinny/
│   ├── config.json          # Cinny homeserver config (deployed to /var/www/html/config.json)
│   ├── upstream-check.sh    # Daily script: checks if cinnyapp/cinny main has new commits, pings Matrix
│   └── lotus-build.sh       # Merge + build script: fetches upstream/main, merges, builds, deploys
├── landing/
│   └── index.html      # matrix.lotusguild.org landing page
├── draupnir/
│   └── production.yaml # Draupnir config (access token is redacted — see rotation docs below)
├── deploy/             # Auto-deployment infrastructure
│   ├── lxc151-hookshot.sh          # Deploy script for LXC 151 (matrix/hookshot/livekit)
│   ├── lxc106-cinny.sh             # Deploy script for LXC 106 (cinny)
│   ├── lxc139-landing.sh           # Deploy script for LXC 139 (landing page)
│   ├── lxc110-draupnir.sh          # Deploy script for LXC 110 (draupnir)
│   ├── livekit-graceful-restart.sh # Waits for zero active calls before restarting livekit
│   ├── hooks-lxc151.json           # webhook binary config for LXC 151
│   ├── hooks-lxc106.json           # webhook binary config for LXC 106
│   ├── hooks-lxc139.json           # webhook binary config for LXC 139
│   └── hooks-lxc110.json           # webhook binary config for LXC 110
└── systemd/
    ├── livekit-server.service              # LiveKit systemd unit (with HA migration fix)
    ├── livekit-graceful-restart.service    # oneshot — checks pending restart flag
    ├── livekit-graceful-restart.timer      # Runs every 5 min
    ├── draupnir.service
    └── cinny-upstream-check.cron           # Installed to /etc/cron.d/ on LXC 106 — runs daily at noon

---

Infrastructure

Service	IP	LXC	RAM	Disk	Versions
Synapse	10.10.10.29	151	12GB	50GB	Synapse 1.157.2, LiveKit 1.9.11, hookshot 7.3.2, coturn latest
PostgreSQL 17	10.10.10.44	109	6GB	80GB	PostgreSQL 17.9
Cinny Web	10.10.10.6	106	2GB	8GB	Debian 12, nginx, Node 24, Lotus Cinny fork (custom, tracks cinnyapp/cinny main)
Draupnir	10.10.10.24	110	1GB	10GB	Draupnir v2.9.0, Node.js v22
Prometheus	10.10.10.48	118	—	—	Prometheus — scrapes all Matrix services
Grafana	10.10.10.49	107	—	—	Grafana 12.4.0 — dashboard.lotusguild.org
NPM	10.10.10.27	139	—	—	Nginx Proxy Manager + matrix landing page
Authelia	10.10.10.36	167	—	—	SSO/OIDC provider
LLDAP	10.10.10.39	147	—	—	LDAP user directory
Uptime Kuma	10.10.10.25	101	—	—	Uptime monitoring (micro1 node)

Key paths on Synapse LXC (151):

• Synapse config: /etc/matrix-synapse/homeserver.yaml
• Synapse conf.d: /etc/matrix-synapse/conf.d/ (metrics.yaml, report_stats.yaml, server_name.yaml)
• coturn config: /etc/turnserver.conf
• LiveKit config: /etc/livekit/config.yaml
• LiveKit service: livekit-server.service
• lk-jwt-service: lk-jwt-service.service (now binds :8071 via drop-in /etc/systemd/system/lk-jwt-service.service.d/override.conf; serves JWT tokens for MatrixRTC at /sfu/get and legacy /get_token)
• voice-limit-guard: voice-limit-guard.service (binds :8070, fronts lk-jwt-service — enforces hard per-room voice participant limits and publish permissions (screenshare/camera via JWT re-signing) for ALL clients; script /opt/voice-limit-guard/voice-limit-guard.py) — see Voice Channel Limits & Call Permissions
• Hookshot: /opt/hookshot/, service: matrix-hookshot.service
• Hookshot config: /opt/hookshot/config.yml
• Hookshot registration: /etc/matrix-synapse/hookshot-registration.yaml
• Bot: /opt/matrixbot/, service: matrixbot.service
• Repo clone (auto-deploy): /opt/matrix-config/
• Deploy env: /etc/matrix-deploy.env (MATRIX_TOKEN, MATRIX_SERVER, MATRIX_ROOM)
• Deploy log: /var/log/matrix-deploy.log

Key paths on Draupnir LXC (110):

• Install path: /opt/draupnir/
• Config: /opt/draupnir/config/production.yaml
• Data/SQLite DBs: /data/storage/
• Service: draupnir.service
• Management room: #management:matrix.lotusguild.org (!mEvR5fe3jMmzwd-FwNygD72OY_yu8H3UP_N-57oK7MI)
• Bot account: @draupnir:matrix.lotusguild.org (power level 100 in all protected rooms and the Lotus Guild space)
• Subscribed ban lists: #community-moderation-effort-bl:neko.dev, #matrix-org-coc-bl:matrix.org
• Rebuild: NODE_OPTIONS="--max_old_space_size=6144" npm run build
• Healthz endpoint: http://10.10.10.24:8081/healthz (200 = healthy, 418 = disconnected)
• Abuse reporting endpoint: POST http://10.10.10.24:8080/_matrix/draupnir/1/report/{roomId}/{eventId}
• Audit DBs: /data/storage/user-restriction-audit-log.db, /data/storage/room-audit-log.db

Key paths on PostgreSQL LXC (109):

• PostgreSQL config: /etc/postgresql/17/main/postgresql.conf
• Tuning conf.d: /etc/postgresql/17/main/conf.d/synapse_tuning.conf
• HBA config: /etc/postgresql/17/main/pg_hba.conf
• Data directory: /var/lib/postgresql/17/main

Key paths on Cinny LXC (106):

• Lotus fork source: /opt/lotus-cinny/ (fork of cinnyapp/cinny main, custom Lotus Guild branch)
• Upstream remote: https://github.com/cinnyapp/cinny.git (added as upstream)
• Built files: /var/www/html/
• Cinny config: /var/www/html/config.json
• Config backup (survives rebuilds): /opt/lotus-cinny/.cinny-config.json
• Monitor env: /etc/cinny-monitor.env (MATRIX_TOKEN, MATRIX_SERVER, MATRIX_ROOM, MATRIX_PING_USER — not in git)
• Upstream check script: /usr/local/bin/cinny-upstream-check.sh
• Build/deploy script: /usr/local/bin/cinny-build.sh (triggered by webhook or manual run)
• Cron: /etc/cron.d/cinny-upstream-check (runs at noon daily — checks only, does not auto-build)
• Monitor state: /var/lib/cinny-monitor/last-upstream-commit
• Monitor log: /var/log/cinny-monitor.log
• Build log: /var/log/cinny-build.log
• Nginx site config: /etc/nginx/sites-available/cinny

---

Auto-Deployment

Pushes to main on LotusGuild/matrix automatically deploy to the relevant LXC(s) via Gitea webhooks. All 4 LXCs are fully independent — each runs its own webhook listener and deploys only its own files. No cross-LXC SSH dependencies.

How It Works

1. Push to LotusGuild/matrix on Gitea
2. Gitea fires webhooks to all 4 LXCs simultaneously (HMAC-SHA256 validated)
3. Each LXC runs /usr/local/bin/matrix-deploy.sh via the webhook binary
4. Script does git fetch + reset --hard origin/main, checks which files changed, deploys only relevant ones
5. Logs to /var/log/matrix-deploy.log on each LXC

Per-LXC Webhook Endpoints

LXC	Service	IP	Port	Deploys When Changed
151	matrix/hookshot	10.10.10.29	9500	hookshot/*.js, systemd/livekit-server.service, livekit/voice-limit-guard.py, systemd/voice-limit-guard.service, matrixbot/*
106	cinny	10.10.10.6	9000	cinny/config.json, cinny/upstream-check.sh, cinny/lotus-build.sh, deploy/hooks-lxc106.json, systemd/cinny-upstream-check.cron
139	landing/NPM	10.10.10.27	9000	landing/index.html
110	draupnir	10.10.10.24	9000	draupnir/production.yaml

LXC 151 uses port 9500 because ports 9000–9004 are occupied by Synapse and Hookshot.

What Each Deploy Does

LXC 151 — hookshot/livekit:

• hookshot/*.js changed → runs hookshot/deploy.sh (pushes transform functions to Matrix room state via API, requires MATRIX_TOKEN in /etc/matrix-deploy.env)
• systemd/livekit-server.service changed → copies file, daemon-reload, sets /run/livekit-restart-pending flag (actual restart deferred — see Livekit Graceful Restart below)
• livekit/voice-limit-guard.py / systemd/voice-limit-guard.service changed → py_compile-validates, installs to /opt/voice-limit-guard/, daemon-reload (if unit changed), and restarts voice-limit-guard (restart only affects joins in a ~1s window; established calls talk directly to livekit-server, so no call is dropped)

LXC 106 — cinny:

• cinny/config.json → copies to /var/www/html/config.json
• cinny/upstream-check.sh → copies to /usr/local/bin/cinny-upstream-check.sh, chmod +x
• cinny/lotus-build.sh → copies to /usr/local/bin/cinny-build.sh, chmod +x
• deploy/hooks-lxc106.json → copies to /etc/webhook/hooks.json, restarts webhook service
• systemd/cinny-upstream-check.cron → copies to /etc/cron.d/cinny-upstream-check, chmod 644

LXC 139 — landing page:

• landing/index.html → copies to /var/www/matrix-landing/index.html, nginx -s reload

LXC 110 — draupnir:

• draupnir/production.yaml → extracts live accessToken from existing config, overwrites from repo, restores token via sed, restarts draupnir.service

Installed Components (per LXC)

• webhook binary (Debian package webhook v2.8.0) listening on respective port
• /etc/webhook/hooks.json — unique HMAC-SHA256 secret per LXC
• /usr/local/bin/matrix-deploy.sh — deploy script from this repo
• /etc/systemd/system/webhook.service — enabled and running
• /opt/matrix-config/ — clone of this repo
• /var/log/matrix-deploy.log — deploy log

LXC 151 additionally:

• /etc/matrix-deploy.env — MATRIX_TOKEN, MATRIX_SERVER, MATRIX_ROOM (not in git)
• /usr/local/bin/livekit-graceful-restart.sh
• /etc/systemd/system/livekit-graceful-restart.service + .timer

LXC 106 additionally:

• /etc/cinny-monitor.env — MATRIX_TOKEN, MATRIX_SERVER, MATRIX_ROOM, MATRIX_PING_USER (not in git)
• /var/lib/cinny-monitor/last-upstream-commit — state file (tracks last-seen upstream SHA)
• /opt/lotus-cinny/ — git clone of code.lotusguild.org/LotusGuild/cinny with upstream remote (cinnyapp/cinny)
• /root/.git-credentials — Gitea token lxc106-lotus-cinny (write:repository scope, revocable via Gitea UI)
• /var/lib/cinny-monitor/last-upstream-tag — last seen stable release tag (e.g. v4.11.1)

Livekit Graceful Restart

Killing livekit-server while a call is active drops everyone. Instead:

1. Deploy to LXC 151 copies the new livekit-server.service and sets a /run/livekit-restart-pending flag
2. livekit-graceful-restart.timer runs every 5 minutes
3. The timer script counts established TCP connections on port 7881 (ss -tn state established)
4. If zero connections → restarts livekit-server and clears the flag
5. If connections exist → logs and exits, retries in 5 minutes

---

Voice Channel Limits & Call Permissions

Per-room voice participant caps and publish permissions (screenshare / camera) are enforced server-side for every client (Element, FluffyChat, Lotus Chat, …), not just our own web client. Both are enforced by the same voice-limit-guard sidecar (livekit/voice-limit-guard.py), which fronts lk-jwt-service at token issue.

How it works

Every Matrix client must fetch a LiveKit JWT from lk-jwt-service before it can join a call. voice-limit-guard (a small fail-open Python sidecar) sits in front of that service:

• lk-jwt-service was moved off :8070 to :8071 (systemd drop-in). The guard now owns :8070, so NPM's existing /sfu/get + /get_token proxy targets are unchanged.
• On each token request the guard reads the room's Lotus policy from Synapse admin state (one /state fetch, cached 10 s): io.lotus.voice_limit → max_users, and io.lotus.room_quality → allow_screenshare / allow_camera. The room id is taken from the endpoint's own field (/get_token → room_id, /sfu/get → room) exactly as lk-jwt-service reads it, so a client sending both keys can't get a different room's policy applied than the token is minted for.
• Participant limit — it forwards to lk-jwt-service, and if a token is issued decodes the JWT to get the LiveKit alias (video.room) + requester (sub), then asks LiveKit ListParticipants how many distinct Matrix users are in the room. requester already present (rejoin) → allow · distinct users ≥ limit → 403 · otherwise → allow.
• Publish permissions (screenshare / camera) — LiveKit is a pure SFU and cannot cap a publisher's bitrate/framerate (no such field exists in the grant/config/API — that stays a Lotus-client-cooperative setting). But the JWT's video.canPublishSources is SFU-enforced for every client. Since the guard holds the LiveKit signing secret, when a room forbids a source it decodes the issued token, drops screen_share/screen_share_audio (and/or camera) from canPublishSources, and re-signs it (HS256, same key). Microphone is always kept. The SFU then rejects those tracks for all clients — nothing to opt into.
• Live (mid-call) enforcement — the JWT re-sign covers anyone joining after a policy change. For people already in the call, a background reconcile loop (every GUARD_RECONCILE_INTERVAL, default 3 s) calls LiveKit UpdateParticipant to narrow their canPublishSources, which unpublishes an in-progress screenshare/camera server-side for all clients and blocks re-publish (confirmed LiveKit 1.9.11 behavior: reducing can_publish_sources removes the offending live track). So flipping a room to audio-only kills existing cameras/screenshares within ~one interval. The loop learns each LiveKit room's Matrix id from tokens it issues, only ever removes forbidden sources (never grants), preserves every other permission flag (full-replace safety), and no-ops once compliant. Disable with GUARD_RECONCILE=0.
• Fail-open: any error (admin API down, bad/absent token, LiveKit unreachable, unparseable room id, unexpected JWT shape) returns the upstream response unchanged, so calls keep working even if enforcement is degraded. The limit check and the source-policy re-sign are independent (a LiveKit-admin outage during the limit count can't skip the source restriction, and vice-versa). Before re-signing, the guard verifies its own secret actually signed the token — on a LIVEKIT_SECRET mismatch it skips the restriction and passes the original token through (so a secret drift can never emit a token the SFU rejects). A room with no policy set takes a zero-overhead fast path (token untouched).

Security note: LIVEKIT_KEY/LIVEKIT_SECRET are currently hardcoded in systemd/voice-limit-guard.service (pre-existing). Since this secret now also signs re-issued join tokens, it should be moved into /etc/matrix-deploy.env (already an EnvironmentFile on LXC 151) and the exposed value rotated. Not changed automatically to avoid a deploy breaking before the env file carries it.

Pure logic (limit decision, source narrowing, JWT re-sign/verify roundtrip, tamper detection) is unit-tested in livekit/test_voice_limit_guard.py (python3 -m unittest livekit.test_voice_limit_guard).

Setting policy: room admins use Lotus Chat → Room Settings → General → Voice (Call Permissions switches + Quality Caps). Any tool that can send room state works too:

# max 5 participants; send {} to remove the limit
curl -X PUT -H "Authorization: Bearer <admin_token>" -H "Content-Type: application/json" \
  "https://matrix.lotusguild.org/_matrix/client/v3/rooms/<roomId>/state/io.lotus.voice_limit/" \
  -d '{"max_users": 5}'

# forbid screenshare + make it audio-only (hard, all clients); numeric caps are
# Lotus-client-cooperative hints in the same event
curl -X PUT -H "Authorization: Bearer <admin_token>" -H "Content-Type: application/json" \
  "https://matrix.lotusguild.org/_matrix/client/v3/rooms/<roomId>/state/io.lotus.room_quality/" \
  -d '{"allow_screenshare": false, "allow_camera": false, "audio_max_kbps": 32}'

Config: the guard reads MATRIX_TOKEN (server-admin) from /etc/matrix-deploy.env; LiveKit key/secret + ports are set in systemd/voice-limit-guard.service.

Deploy: auto-deploys on push (LXC 151 handler py_compile-validates then restarts the guard). Manual (re)deploy / first-time setup:

# On LXC 151
install -D -m644 /opt/matrix-config/livekit/voice-limit-guard.py /opt/voice-limit-guard/voice-limit-guard.py
install -m644 /opt/matrix-config/systemd/voice-limit-guard.service /etc/systemd/system/voice-limit-guard.service
# one-time: rebind lk-jwt-service to :8071
mkdir -p /etc/systemd/system/lk-jwt-service.service.d
printf '[Service]\nEnvironment=LIVEKIT_JWT_BIND=:8071\n' > /etc/systemd/system/lk-jwt-service.service.d/override.conf
systemctl daemon-reload && systemctl restart lk-jwt-service && systemctl enable --now voice-limit-guard

To fully revert (back to lk-jwt-service directly on :8070): systemctl disable --now voice-limit-guard, remove the drop-in, daemon-reload, systemctl restart lk-jwt-service.

---

Access Token Rotation

The MATRIX_TOKEN in /etc/matrix-deploy.env on LXC 151 is a Jared user token used to push hookshot transforms to Matrix room state (requires power level ≥ 50 in Spam and Stuff).

The token in draupnir/production.yaml in this repo is intentionally redacted (accessToken: REDACTED). The deploy script on LXC 110 extracts the live token from the running config before overwriting from the repo, then restores it.

To rotate the hookshot deploy token (LXC 151):

1. Generate a new token via Synapse admin API or Cinny → Settings → Security → Manage Sessions
2. SSH to LXC 151 (via ssh root@10.10.10.4 then pct enter 151): nano /etc/matrix-deploy.env
3. Replace MATRIX_TOKEN=<old> with new token
4. Test: MATRIX_TOKEN=<new> MATRIX_SERVER=https://matrix.lotusguild.org bash /opt/matrix-config/hookshot/deploy.sh

To rotate the Draupnir token:

1. Generate new token for @draupnir:matrix.lotusguild.org
2. On LXC 110: nano /opt/draupnir/config/production.yaml → update accessToken
3. systemctl restart draupnir
4. Do not commit the token to git — the repo version stays redacted

---

Port Maps

Router → 10.10.10.29 (forwarded):

• TCP+UDP 3478 — TURN/STUN
• TCP+UDP 5349 — TURNS/TLS
• TCP 7881 — LiveKit ICE TCP fallback
• TCP+UDP 49152-65535 — TURN relay range

Internal port map (LXC 151):

Port	Service	Bind
8008	Synapse HTTP	0.0.0.0
9000	Synapse metrics	127.0.0.1 + 10.10.10.29
9001	Hookshot widgets	0.0.0.0
9002	Hookshot bridge (appservice)	127.0.0.1
9003	Hookshot webhooks	0.0.0.0
9004	Hookshot metrics	0.0.0.0
9100	node_exporter	0.0.0.0
9101	matrix-admin exporter	0.0.0.0
9500	webhook (auto-deploy)	0.0.0.0
6789	LiveKit metrics	0.0.0.0
7880	LiveKit HTTP	0.0.0.0
7881	LiveKit RTC TCP	0.0.0.0
8070	voice-limit-guard (fronts lk-jwt-service)	0.0.0.0
8071	lk-jwt-service (behind guard)	0.0.0.0
8080	synapse-admin (nginx)	0.0.0.0
3478	coturn STUN/TURN	0.0.0.0
5349	coturn TURNS/TLS	0.0.0.0

Internal port map (LXC 110 — Draupnir):

Port	Service	Bind
8080	Draupnir web (abuse reporting)	0.0.0.0
8081	Draupnir healthz	0.0.0.0
9000	webhook (auto-deploy)	0.0.0.0
9100	node_exporter	0.0.0.0
9256	process_exporter	0.0.0.0

Internal port map (LXC 109 — PostgreSQL):

Port	Service	Bind
5432	PostgreSQL	0.0.0.0 (hba-restricted to 10.10.10.29)
9100	node_exporter	0.0.0.0
9187	postgres_exporter	0.0.0.0

---

Rooms (all v12)

Room	Room ID	Join Rule
The Lotus Guild (Space)	!-1ZBnAH-JiCOV8MGSKN77zDGTuI3pgSdy8Unu_DrDyc	public
General	!wfokQ1-pE896scu_AOcCBA2s3L4qFo-PTBAFTd0WMI0	public
Commands	!ou56mVZQ8ZB7AhDYPmBV5_BR28WMZ4x5zwZkPCqjq1s	restricted (Space members)
Memes	!GK6v5cLEEnowIooQJv5jECfISUjADjt8aKhWv9VbG5U	restricted (Space members)
Music	!ktQu0gavhjpCMkgxk8SYdb6mnJRY-u7mY7_KfksV0SU	restricted (Space members)
Voice Room	!ARbRFSPNp2U0MslWTBGoTT3gbmJJ25dPRL6enQntvPo	restricted (Space members)
Management	!mEvR5fe3jMmzwd-FwNygD72OY_yu8H3UP_N-57oK7MI	invite
Cool Kids	!R7DT3QZHG9P8QQvX6zsZYxjkKgmUucxDz_n31qNrC94	invite
Spam and Stuff	!GttT4QYd1wlGlkHU3qTmq_P3gbyYKKeSSN6R7TPcJHg	invite, no E2EE (hookshot)

Power level roles (Cinny tags):

• 100: Owner (jared, draupnir, lotusbot)
• 50: The Nerdy Council / Panel of Geeks (enhuynh, lonely)
• 0: Member

---

Webhook Integrations (matrix-hookshot 7.3.2)

Generic webhooks bridged into Spam and Stuff. Each service gets its own virtual user (@hookshot_<service>) with a unique avatar. Webhook URL format: https://matrix.lotusguild.org/webhook/<uuid>

Service	Webhook UUID	Notes
Grafana	df4a1302-2d62-4a01-b858-fb56f4d3781a	Unified alerting contact point
Proxmox	9b3eafe5-7689-4011-addd-c466e524661d	Notification system (8.1+), Discord embed format
Sonarr	aeffc311-0686-42cb-9eeb-6757140c072e	All event types
Radarr	34913454-c1ac-4cda-82ea-924d4a9e60eb	All event types
Readarr	e57ab4f3-56e6-4dc4-8b30-2f4fd4bbeb0b	All event types
Lidarr	66ac6fdd-69f6-4f47-bb00-b7f6d84d7c1c	All event types
Uptime Kuma	1a02e890-bb25-42f1-99fe-bba6a19f1811	Status change notifications
Seerr	555185af-90a1-42ff-aed5-c344e11955cf	Request/approval events
Owncast (Livestream)	9993e911-c68b-4271-a178-c2d65ca88499	STREAM_STARTED / STREAM_STOPPED
Bazarr	470fb267-3436-4dd3-a70c-e6e8db1721be	Subtitle events (Apprise JSON notifier)
Tinker-Tickets	6e306faf-8eea-4ba5-83ef-bf8f421f929e	Custom transformation code

Hookshot notes:

• Spam and Stuff is intentionally unencrypted — hookshot bridges cannot join E2EE rooms
• JS transformation functions use hookshot v2 API: result = { version: "v2", plain, html, msgtype }
• The result variable must be assigned without var/let/const (QuickJS IIFE sandbox)
• NPM proxies https://matrix.lotusguild.org/webhook/* → http://10.10.10.29:9003
• NPM proxies /sfu/get and /get_token → http://10.10.10.29:8070 (lk-jwt-service). Both paths are in /data/nginx/proxy_host/49.conf on LXC 139 — NPM will overwrite these if proxy host 49 is re-saved via the UI; re-add both location blocks after any NPM save
• Proxmox sends Discord embed format: data.embeds[0].{title,description,fields} — NOT flat fields
• Transform functions are stored as Matrix room state (uk.half-shot.matrix-hookshot.generic.hook) and deployed via hookshot/deploy.sh

Deploying hookshot transforms manually:

# On LXC 151 or from any machine with access
export MATRIX_TOKEN=<jared_token>
export MATRIX_SERVER=https://matrix.lotusguild.org
export MATRIX_ROOM='!GttT4QYd1wlGlkHU3qTmq_P3gbyYKKeSSN6R7TPcJHg'
bash /opt/matrix-config/hookshot/deploy.sh          # deploy all
bash /opt/matrix-config/hookshot/deploy.sh proxmox.js  # deploy one

---

Moderation (Draupnir v2.9.0)

Draupnir runs on LXC 110, manages moderation across all protected rooms (including the Lotus Guild space) via #management:matrix.lotusguild.org.

Subscribed ban lists:

• #community-moderation-effort-bl:neko.dev — 12,599 banned users, 245 servers, 59 rooms
• #matrix-org-coc-bl:matrix.org — 4,589 banned users, 220 servers, 2 rooms

Common commands (send in management room):

!draupnir status                          — current status + protected rooms
!draupnir ban @user:server * "reason"     — ban from all protected rooms
!draupnir redact @user:server             — redact their recent messages
!draupnir rooms add !roomid:server        — add a room to protection
!draupnir watch <alias> --no-confirm      — subscribe to a ban list

Abuse Reporting

When a Matrix client user clicks "Report" on a message, Synapse receives a POST /_matrix/client/v3/rooms/{roomId}/report/{eventId} request and stores the report internally. To forward these to the Draupnir management room, a Synapse Python module must be installed on LXC 151.

Draupnir web server is enabled (port 8080). The endpoint is:

POST http://10.10.10.24:8080/_matrix/draupnir/1/report/{roomId}/{eventId}

To complete Synapse integration (one-time, on LXC 151):

1. Install the module: pip install matrix-synapse-draupnir-abuse-reports (or equivalent — check Draupnir releases)
2. Add to /etc/matrix-synapse/homeserver.yaml:

   modules:
     - module: "draupnir.abuse_reports.AbuseReportEndpoint"
       config:
         draupnir_endpoint: "http://10.10.10.24:8080"
   

3. systemctl restart matrix-synapse

Until the Synapse module is installed, abuse reports are stored in Synapse's DB but do NOT appear in the management room. The Draupnir web server is running and ready to receive forwarded reports.

---

Lotus Cinny (chat.lotusguild.org)

chat.lotusguild.org serves a custom Lotus Guild fork of the official cinnyapp/cinny main branch. The fork lives at code.lotusguild.org/LotusGuild/cinny and tracks upstream via a git remote add upstream https://github.com/cinnyapp/cinny.git workflow.

Upstream monitoring (daily at noon):

• cinny-upstream-check.sh hits the GitHub API and compares the latest cinnyapp/cinny main commit against the stored SHA in /var/lib/cinny-monitor/last-upstream-commit
• If new commits exist, sends a Matrix message to Spam and Stuff with an @jared:matrix.lotusguild.org ping and a link to the commit
• Does not auto-build — you review the diff and decide when to merge

Merge + build workflow:

1. Receive upstream notification in Matrix
2. Review the diff: https://github.com/cinnyapp/cinny/compare/<old>...<new>
3. Send !cinny-update in any Matrix room — LotusBot POSTs to the cinny-build webhook on LXC 106
4. cinny-build.sh runs: git fetch upstream && git merge upstream/main, npm ci, npm run build, deploys to /var/www/html/
5. Build result (success or conflict) is posted back to Matrix

Manual build (SSH):

# On LXC 106
/usr/local/bin/cinny-build.sh

Merge conflict recovery:

# On LXC 106
cd /opt/lotus-cinny
git merge upstream/main    # resolve conflicts in editor
git add -A && git merge --continue
/usr/local/bin/cinny-build.sh

LXC 106 one-time setup (after forking cinnyapp/cinny to code.lotusguild.org/LotusGuild/cinny):

# On LXC 106
git clone https://code.lotusguild.org/LotusGuild/cinny.git /opt/lotus-cinny
cd /opt/lotus-cinny
git remote add upstream https://github.com/cinnyapp/cinny.git
git fetch upstream

# Create env file (fill in a valid Matrix token)
cat > /etc/cinny-monitor.env << 'EOF'
MATRIX_TOKEN=<jared_or_bot_token>
MATRIX_SERVER=https://matrix.lotusguild.org
MATRIX_ROOM=!GttT4QYd1wlGlkHU3qTmq_P3gbyYKKeSSN6R7TPcJHg
MATRIX_PING_USER=@jared:matrix.lotusguild.org
EOF
chmod 600 /etc/cinny-monitor.env

Cinny-build webhook token (for LotusBot !cinny-update): stored in deploy/hooks-lxc106.json (cinny-build hook, header X-Build-Token). LotusBot must POST to http://10.10.10.6:9000/hooks/cinny-build with this header.

Why 8GB RAM: Vite's build process needs ~6GB Node heap (--max_old_space_size=6144) for the rendering-chunks phase. Previously at 4GB — OOM killed during render.

🔱 Element Call fork — "Lotus Call" (true ownership) — LIVE

We self-build Element Call from a fork (LotusGuild/element-call) and publish it to our private Gitea npm registry as @lotusguild/element-call-embedded (0.20.1-lotus.1); cinny consumes that instead of the upstream @element-hq/element-call-embedded bundle. In-call behavior is now editable source, not just widget-API + DOM steering. This is AGPL (same license).

Shipped via the fork: in-source denoise (a LiveKit TrackProcessor that survives reconnects), in-call speaking/mute events, focus-a-participant during screenshare, avatar decorations on EC video tiles, native transparent background. Built but dormant (need cinny UI): call-audio injection (io.lotus.inject_audio, unblocks a real in-call soundboard) and quality controls (io.lotus.set_quality).

Infra notes for THIS repo:

• EC talks to our LiveKit SFU (livekit/, LXC 151) + lk-jwt-service; the fork's runtime config.json points at matrix.lotusguild.org + our LiveKit. The cinny EC config.json lives in cinny/config.json here.
• Build/deploy: the fork builds in the cinny pipeline (its dist/ is bundled into the cinny build that LXC 106 serves) — no separate EC LXC. A future quality controls feature (P5-31) would add a voice-limit-guard-style sidecar on LXC 151.

Full handoff & step-by-step plan: LotusGuild/cinny → HANDOFF_ELEMENT_CALL_FORK.md.

Custom Features

All custom code lives in src/app/ on the lotus branch of code.lotusguild.org/LotusGuild/cinny. Changes survive upstream merges as long as they don't conflict with the same files upstream touched.

Feature	Files	Notes
Element Call embed	src/app/plugins/call/, src/app/hooks/useCallEmbed.ts, src/app/components/CallEmbedProvider.tsx	🔱 [EC-FORK] LIVE — self-built fork @lotusguild/element-call-embedded@0.20.1-lotus.1 (source LotusGuild/element-call), bundled into the cinny build, served same-origin. Steered via matrix-widget-api + custom io.lotus.* actions (call_state, focus_participant, decorations, inject_audio, set_quality) — DOM-poking retained only as fallback. See LotusGuild/cinny → HANDOFF_ELEMENT_CALL_FORK.md
DM calls	src/app/features/room/Room.tsx, src/app/features/room/RoomViewHeader.tsx	Phone button in DM room header; useCallStart(true) passes intent: StartedByUser; Room.tsx switches to CallView layout when DM has active call
Picture-in-picture call	src/app/components/CallEmbedProvider.tsx	When navigating away from the call room, the embed shrinks to a 280×158px PiP in the bottom-right. Click navigates back. Implemented via useEffect imperatively overriding styles on callEmbedRef.current — cannot use a wrapper div because useCallEmbedPlacementSync writes top/left/width/height directly onto that element
Screenshare fullscreen	src/app/features/call/CallControls.tsx, src/app/features/call/Controls.tsx	When screensharing, a fullscreen button appears in call controls. Calls callEmbedRef.current?.requestFullscreen() on the Cinny call container. EC naturally spotlights the screenshare — the old 600ms grid-revert code was removed (it caused fullscreen to show avatars instead of the screen)
PiP screenshare focus	src/app/components/CallEmbedProvider.tsx, src/app/plugins/call/CallControl.ts	When the floating PiP window is active and screenshare is detected (no cameras present), auto-enables EC spotlight view so the screenshare fills the PiP rather than showing avatar tiles
Screenshare audio mute	src/app/features/call/Controls.tsx, src/app/features/call/CallControls.tsx, src/app/plugins/call/CallControl.ts	Dedicated button to independently mute/unmute audio from screenshares without muting microphone audio. Targets audio[data-lk-source="screen_share_audio"] LiveKit elements. Persists across deafen/undeafen cycles
In-call soundboard (P5-15)	src/app/features/call/CallSoundboard.tsx, src/app/hooks/useSoundboard.ts, src/app/utils/soundboardClips.ts, CallControl.ts#injectAudio	Call-bar popout of user-uploaded clips. Playing one sends the fork's io.lotus.inject_audio (armed via lotusAudioInject=1) so it publishes as a real LiveKit track heard by all, plus local playback. Clips are uploadable like emoji/sticker packs — stored in io.lotus.soundboard account data (synced across devices); host resolves mxc → authed download → blob: URL for the widget. Gated by soundboardEnabled setting
Call quality controls (P5-31)	src/app/utils/callQuality.ts, src/app/hooks/useCallQuality.ts, CallControl.ts#setQuality	Per-user mic/screenshare bitrate + screenshare framerate (Settings → Calls), applied via the fork's io.lotus.set_quality, clamped to any room cap (min(user, room)). Client-cooperative (numeric caps aren't SFU-enforceable). Unit-tested
Room call permissions (P5-31)	src/app/features/common-settings/general/RoomQuality.tsx, types/matrix/room.ts (LotusRoomQuality), CallControls.tsx	Admin switches in Room Settings → Voice write io.lotus.room_quality allow_screenshare/allow_camera; the call bar hides blocked buttons. Hard-enforced server-side for all clients by voice-limit-guard (this repo) — see Voice Channel Limits & Call Permissions
Custom status message	src/app/features/settings/account/Profile.tsx, src/app/features/room/MembersDrawer.tsx, src/app/components/user-profile/UserHero.tsx, src/app/components/user-profile/UserRoomProfile.tsx, src/app/hooks/useUserPresence.ts	Discord-style free-form status text. Set via Settings → Account → "Status Message" with an emoji picker (lazy-loaded EmojiBoard). Saved via mx.setPresence({ status_msg }). Displayed below the username in the members drawer and user profile popout. Syncs live via Matrix presence events
PTT (Push-to-Talk)	src/app/features/call/CallControls.tsx, src/app/state/settings.ts	Hold-to-talk key (default: Space, configurable). Mutes mic on join; holds mic open while key is held. Badge shows PTT — Hold SPACE / ● Live. Listens on both main window and EC iframe contentWindow for key events
PTT badge theming	src/app/features/call/CallControls.tsx	Plain folds Chip by default; neon terminal style (#00FF88/#FF6B00, JetBrains Mono) when lotusTerminal setting is on
GIF picker	src/app/components/GifPicker.tsx, src/app/features/room/RoomInput.tsx	Giphy JS/React SDK (@giphy/react-components, @giphy/js-fetch-api, styled-components). API key in config.json → gifApiKey. GIF button appears next to Send only when gifApiKey is set. Sends GIF as m.image (fetches blob → mx.uploadContent → mx.sendMessage). FocusTrap handles click-outside / Escape to close
GIF picker terminal theme	src/app/components/GifPicker.tsx	When lotusTerminal is on: dark navy background (#060c14), orange dim border, 4px radius, // GIF_SEARCH header, injected <style> overrides Giphy SDK SearchBar input (dark bg, orange border/focus ring, JetBrains Mono), custom orange scrollbar
Terminal Design System toggle	src/app/state/settings.ts, src/app/features/settings/	lotusTerminal boolean setting. When enabled: PTT badge, GIF picker, and voice message recorder use LotusGuild Terminal Design System aesthetics (green #00FF88 / orange #FF6B00, JetBrains Mono)
Presence status badges	src/app/features/room/MembersDrawer.tsx, src/app/features/common-settings/members/Members.tsx, src/app/hooks/useUserPresence.ts, src/app/components/presence/	Online/busy/away colored dot badges shown next to verification shields for every member in the room members drawer and settings members panel. Uses useUserPresence(userId) hook + PresenceBadge component. Members.tsx wraps the hook in a MemberPresenceBadge child component to satisfy React hook rules inside .map()
Discord-style presence tracking	src/app/hooks/usePresenceUpdater.ts, src/app/pages/client/ClientNonUIFeatures.tsx	Broadcasts online on startup, unavailable after 10 min idle or tab hidden, offline on page close (fetch+keepalive). Activity throttled to 1 event/sec. hidePresence setting broadcasts offline and disables all tracking
Per-member device sessions panel	src/app/components/user-profile/UserRoomProfile.tsx, src/app/hooks/useOtherUserDevices.ts	Collapsible "Sessions" card in user profile popout. Lists all devices with colored shield icons (green=verified, yellow=unverified, loading/error states). Per-device "Verify" button initiates cross-signing SAS emoji verification. Updates live via CryptoEvent.DevicesUpdated. Only shown when cross-signing is active
Privacy settings	src/app/features/settings/general/General.tsx, src/app/state/settings.ts	Dedicated Privacy section in General settings. hideActivity suppresses typing indicators and read receipts. hidePresence appears offline to everyone
Encrypted room search	src/app/features/message-search/useLocalMessageSearch.ts, src/app/features/message-search/MessageSearch.tsx	Searches locally cached decrypted events in E2EE rooms alongside server-side search. Per-room "Load more" buttons paginate 100 msgs at a time; shows oldest cached date and X/Y coverage counter. Sender-aware (respects from:@user filter)
Message search: sender filter	src/app/features/message-search/SearchInput.tsx, src/app/features/message-search/SearchFilters.tsx	Type from:@user in the search box for live autocomplete of known users (homeserver-biased ranking). Selected senders shown as removable chips. Works for both server search and local encrypted search
Message search: date range	src/app/features/message-search/SearchFilters.tsx, src/app/features/message-search/useMessageSearch.ts	From/To date pickers in the filter bar. Passed as from_ts/to_ts epoch ms to Matrix /search
Document title unread count	src/app/pages/client/ClientNonUIFeatures.tsx	Tab title updates to (N) Lotus Chat for mentions, · Lotus Chat for unreads, Lotus Chat when clear
Message draft persistence	src/app/features/room/RoomInput.tsx	Unsent messages survive page reload via localStorage (draft-msg-<roomId>). Jotai in-memory atom remains the primary store; localStorage used as fallback on reload. Cleared on send
PiP position persistence + snap	src/app/components/CallEmbedProvider.tsx	PiP position saved to localStorage on drag end; restored on next PiP enter (clamped to viewport). Double-click snaps to nearest corner with 180ms CSS transition
Threads (P3-8 + P4-1)	src/app/features/room/thread/, state/room/thread.ts, utils/threadNotifications.ts, hooks/useRoomsListener.ts	Full m.thread support: side panel (own composer, per-thread drafts), "N replies" unread chips, threaded receipts; SDK threadSupport on, markAsRead unthreaded; replies no longer render inline. Slack-style notifications: default = participating-only, per-thread All/Mentions/Mute in io.lotus.thread_notifications account data; muted threads subtracted from room badges client-side
KaTeX math + encrypted-search cache + session hardening + crypto diagnostics	utils/{mathParse,searchCache,cryptoDiagLog}.ts, state/sessions.ts, LOTUS_E2EE_INVESTIGATION.md	July 2026 batch: $…$/$$…$$ + data-mx-maths via lazy KaTeX; opt-in IndexedDB search index for E2EE rooms (wiped on logout); atomic cinny_session_v1 blob + cross-tab logout sync; KE-1→4 diagnostics capture card in Developer Tools
Inline media embeds	src/app/utils/videoEmbed.ts, src/app/components/url-preview/{UrlPreviewCard,UrlPreview.css}.tsx	Media links play/render in place behind a click-to-play facade (homeserver og:image thumbnail; third-party iframe mounts only on Play). 16 providers: video (YouTube/Shorts, Vimeo, Dailymotion, Streamable, Twitch, Loom, Kick), audio (Spotify, SoundCloud, Apple Music, Tidal), self-resizing posts (X/Twitter, Instagram, Reddit, Bluesky). TikTok short links resolve via CORS oEmbed; posts self-size via origin-scoped postMessage. Sandbox omits allow-top-navigation; previews capped at 6/msg. Setting inlineMediaEmbeds. CSP: every embed host is enumerated in the desktop Tauri frame-src (cinny-desktop/tauri.conf.json) and the deployed web nginx frame-src allowlist on LXC 106 (/etc/nginx/sites-available/cinny — hand-maintained; the wildcard frame-src 'self' https: in this repo's cinny/nginx.conf is the looser fallback)
On-device message translation	src/app/features/room/message/, src/app/hooks/useMessageTranslation.ts, src/app/state/settings.ts	"Translate" button on foreign-language messages renders the translation inline with a "Translated from <lang> · Show original" toggle + optional auto-translate mode. Runs 100% on-device via the browser's built-in Translator API (Chromium: Chrome/Edge + Lotus desktop app) — message text never leaves the device and never reaches any cloud translation service (Google/DeepL/Microsoft), so it works in E2EE rooms without weakening encryption. Target language in Settings → General (default English). Feature-detected — gracefully hidden where unsupported (Firefox/Safari/mobile)
Desktop app (Tauri)	cinny-desktop → src-tauri/src/native/*.rs, src-tauri/src/lib.rs; cinny src/app/hooks/useTauri*.ts, src/app/components/TauriDesktopFeatures.tsx	Tauri v2 native shell: rich WinRT toast notifications (click → open room, inline quick reply), Windows Focus Assist → DND sync, taskbar Jump List of recent rooms, taskbar thumbnail + volume-flyout call controls (mute/deafen/end), no-sleep during calls, network-change awareness (mx.retryImmediately), opt-in TDS window chrome, recursive folder drag-drop, auto-update toast. Windows-native pieces compile in CI (Gitea windows runner + GitHub windows-latest); detail in cinny LOTUS_FEATURES.md → Desktop App Features
LiveKit codec config	/etc/livekit/config.yaml (LXC 151)	enabled_codecs: VP8, H264, VP9, Opus, RED for better quality and redundancy

Key config values (/opt/lotus-cinny/config.json, root — vite copies this to dist):

{
  "defaultHomeserver": 0,
  "homeserverList": ["matrix.lotusguild.org"],
  "allowCustomHomeservers": false,
  "gifApiKey": ""
}

Note: gifApiKey is a runtime secret — it is not stored in git. The placeholder ships empty; lotus_deploy.sh injects the real key from GIPHY_API_KEY in /etc/lotus-deploy.env on LXC 106 after each deploy. (For that injection to work the deployed config.json must contain the "gifApiKey": "" placeholder line.)

Note: The root /opt/lotus-cinny/config.json is what matters — vite copies it to dist/. public/config.json is not used.

---

Known Issues

LiveKit Port Conflict After HA Migration

LXC 151 can migrate between Proxmox nodes via HA. After migration, the old livekit-server process on the source node can leave a stale entry holding port 7881 on the destination. Fixed in livekit-server.service via:

ExecStartPre=-/bin/bash -c 'pkill -x livekit-server; sleep 1'
KillMode=control-group

coturn TLS Reset Errors

Periodic TLS/TCP socket error: Connection reset by peer in coturn logs. Normal — clients probe TURN and drop once they establish a direct P2P path.

BBR Congestion Control

net.ipv4.tcp_congestion_control = bbr must be set on the Proxmox host, not inside an unprivileged LXC. All other sysctl tuning (TCP/UDP buffers, fin_timeout) is applied inside LXC 151.

Applied 2026-07-26 via /etc/sysctl.d/99-bbr.conf on compute-storage-01 and large1 (the two nodes the HA affinity rule allows LXC 151 to run on). Because the file lives on the host, a node that has never hosted 151 will not have it — check before allowing 151 to migrate somewhere new:

sysctl net.ipv4.tcp_congestion_control net.core.default_qdisc   # want: bbr / fq

⚠️ large1 has a failing DIMM — keep Synapse and PostgreSQL off it

large1 logs corrected ECC errors concentrated on one module (P0 CHANNEL B, DIMM 1 — EDAC mc#0csrow#2channel#1, 421 corrected errors observed, with the MCE Over/overflow flag set meaning more occurred than were logged). It has hard-crashed on its own twice — 2026-07-24 14:19 and 2026-07-26 00:20 — with no clean-shutdown markers and no OOM events, which is the classic signature of an uncorrectable ECC error.

Do not confuse these with the cluster-wide resets on 2026-07-22 01:56 and 2026-07-24 11:01-11:45, which took down all six nodes simultaneously and were power events, not host hardware.

Consequences for Matrix:

• On the 2026-07-26 crash, HA relocated LXC 109 (PostgreSQL) and LXC 151 (Synapse) to compute-storage-01, which has zero ECC errors across all boots. They should stay there.
• The HA rule ha-rule-43cb668b-becf in /etc/pve/ha/rules.cfg currently weights compute-storage-01:1,large1:1 — equal priority, non-strict — so a future failover can put Synapse straight back onto the bad-RAM node. Re-weighting to compute-storage-01:2,large1:1 would prefer the healthy node while keeping large1 as a fallback. Not changed automatically because that rule governs 10 resources and compute-storage-01 has only 46GB RAM vs large1's 128GB.
• Never run a large VACUUM FULL or state-compression job on a host with failing RAM — those rewrite the whole table through memory, and a bit-flip becomes silent, backup-propagated corruption.
• Ceph's noout flag is intentionally left set. With 10 OSDs on an intermittently-crashing node, noout prevents a full 10-OSD rebalance every time it bounces. Do not clear it until large1 is stable.

A replacement DIMM is not currently in budget. Free mitigations worth evaluating: physically removing the bad module (96GB in flex mode, or 64GB symmetric dual-channel), kernel page soft-offlining via rasdaemon (the errors cluster in a small set of physical pages), or a memmap= blacklist.

PostgreSQL state_groups_state bloat

Matrix state resolution is the dominant cost on this homeserver, not message volume.

2026-07-26 cleanup — database went 22GB → 2183MB (−90%):

Metric	Before	After
Database size	22 GB	2183 MB
state_groups_state	20 GB	540 MB
state_groups_state rows	89,543,584	2,325,198
LXC 109 disk used	25 G (89%)	8.5 G (12%)

Two independent wins: purging one useless space (17GB) and compressing five large federated rooms (a further ~1.3GB). Total downtime across the whole operation was ~3.5 minutes.

Runtime effect (Prometheus, 20-minute window vs. the same window 24h earlier):

Metric	Before	After
get_auth_chain_difference_chains DB time	0.0508 s/s	0.0 — never called
_get_state_groups_from_groups DB time	0.03 s/s	0.0004 s/s
Reactor tick (avg)	1.18 ms	0.52 ms
Peak CPU	0.42 cores	0.16 cores
stateGroupMembersCache	974,953/1M FULL, 8,007,530 evictions, 28% hit	104 entries, 0 evictions
*getEvent*	61,440/61,440 FULL, 546,724 evictions	286 entries, 0 evictions

Read the "after" cache hit-rates (30-60%) with care: they are low only because the caches are now nearly empty and cold-miss dominated. The number that matters is evictions: 8 million → 0. event_cache_size was left at 30K — raising it would be pointless when the cache sits at 286/61,440. Tune caches against eviction counts, not hit rates.

Caveat on the before/after comparison: the two windows are different times of day on a 26-user server, so reactor-tick and CPU deltas are directional rather than exact. The get_auth_chain_difference_chains result is structural, not load-dependent — that room is gone.

Worker split deliberately NOT done. Synapse remains monolithic (no Redis, no instance_map). It was on the table because peak CPU hit 0.96 cores — the single-process GIL ceiling. After this cleanup, peak is 0.16 cores, so the Redis + generic_workers + federation_sender + NPM routing machinery would be operational complexity for no gain. Revisit only if reactor tick returns to ~1ms.

The cost is driven almost entirely by large federated rooms, and it scales with the room's global member count, not with how much you actually use it. One local user joined to a huge public room makes the server pay that room's full state-resolution cost forever. Because Synapse's caches, GIL, DB pool, and PostgreSQL buffer cache are all global with no per-room fairness, a single such room degrades every other room on the server.

To find the offenders:

SELECT room_id, count(*) FROM state_groups_state GROUP BY room_id ORDER BY 2 DESC LIMIT 10;

Spaces (m.space) are the worst offenders per byte — they carry huge membership with zero messages. Leaving a space does not leave its child rooms; they are independent rooms with their own IDs.

---

Server Checklist

Server Checklist

Quality of Life

• Upgrade Synapse to v1.155.0 — Done 2026-06-18. LXC 151 was already on Debian 13 Trixie; no OS migration needed.

• Upgrade Synapse to v1.157.1 — Done 2026-07-23. Note: the host was found running 1.156.0, not the 1.155.0 recorded here — an intermediate upgrade had happened without being logged, so always verify with dpkg-query -W matrix-synapse-py3 rather than trusting this file. Not a security release (no CVEs between 1.155.0 and 1.157.1; the last advisories were fixed back in 1.152.1), but 1.157.0 fixes an application-service ephemeral-events regression that 1.156.0 shipped — which affects hookshot. No schema-version bump (94 → 94); the three new background updates finished immediately. experimental_features (MatrixRTC/Element Call MSCs) was preserved and is unaffected by the v1.157.0 removal of msc3861, which we do not use.

• Upgrade Synapse to v1.157.2 — Done 2026-07-28. Security release (11 advisories: 6 High / 3 Moderate / 2 Low; no CVE IDs assigned). Most High-severity issues require open federation (we federate) — /get_missing_events cross-room leak, relayed-401 crypto-state destruction, spoofed to-device, room-alias redirection; two call out untrusted local users (push-rule exhaustion, alias redirection). Pure apt bump: no # Upgrading to v1.157.2 notes, no schema migration, Python/PG minimums unchanged, config preserved via --force-confold. Restarted clean (NRestarts=0), federation healthy. NB: 1.157.2 adds an optional push_rules.limits config block (safe defaults: rule_count 10000 / rule_id_length 300 / rule_size 1024) for the push-rule-exhaustion advisory — not required, but worth tuning given untrusted local users. Verify with dpkg-query -W matrix-synapse-py3, not this file.

• Migrate from SQLite to PostgreSQL

• TURN/STUN server (coturn) for reliable voice/video

• URL previews

• Upload size limit 200MB

• Full-text message search (PostgreSQL backend)

• Media retention policy (remote: 1yr, local: 3yr)

• Sliding sync (native Synapse)

• LiveKit for Element Call video rooms

• Default room version v12, all rooms upgraded

• Landing page with client recommendations

• Synapse metrics endpoint (port 9000, Prometheus-compatible)

• Lotus Cinny fork — custom fork of cinnyapp/cinny main, daily upstream check + Matrix notification

• Auto-deployment via Gitea webhooks (all 4 LXCs)

• Push notifications gateway (Sygnal) — needs Apple/Google developer credentials

• Lotus Cinny custom branding — Lotus Guild theme (colours, title, favicon, PWA name)

Performance Tuning

• PostgreSQL shared_buffers → 1500MB, effective_cache_size, work_mem, checkpoint tuning

• PostgreSQL pg_stat_statements extension installed — was previously only in shared_preload_libraries; the extension itself had never been created in the synapse DB, so no query-level stats were ever collected. Fixed 2026-07-26 with CREATE EXTENSION pg_stat_statements. Verify with psql -d synapse -c '\dx pg_stat_statements', not just by grepping the config.

• PostgreSQL autovacuum tuned per-table (5 high-churn tables), autovacuum_max_workers → 5

• Synapse event_cache_size → 30K, per-cache factors tuned

• sysctl TCP/UDP buffer alignment on LXC 151 (/etc/sysctl.d/99-matrix-tuning.conf)

• LiveKit: empty_timeout: 300, departure_timeout: 20, max_participants: 50

• LiveKit ICE port range expanded to 50000-51000

• LiveKit TURN TTL reduced to 1h

• LiveKit VP9/AV1 codecs enabled

• TCP retransmit timeout lowered (tcp_retries2=5, tcp_syn_retries=4, tcp_keepalive_probes=3) — stalled outbound federation connections now fail in ~15-30s instead of ~15 min

• Unreachable routes added for servers with asymmetric connectivity (can reach us but we can't reach their federation port) — prevents 90s TCP hangs from being added to lag; defined in /etc/network/interfaces post-up hooks and survive reboots (bark.lgbt ×2, parodia.dev, chat.ohaa.xyz, matrix.k8ekat.dev)

• Stuck device_lists_remote_resync entries cleared for dead-server users (@dalite:bark.lgbt, @arndot:matrix.goch.social) — device list resync was firing every 30s

• BBR congestion control — applied 2026-07-26 on the Proxmox hosts that can run LXC 151 (compute-storage-01, large1) via /etc/sysctl.d/99-bbr.conf (net.ipv4.tcp_congestion_control=bbr, net.core.default_qdisc=fq). Persists across reboots. If 151 is ever allowed to run on another node, apply it there too.

• LXC 109 rootfs grown 30GB → 80GB (2026-07-26). It was at 89% full with only 3.4GB free, and pg_wal shares that volume — PostgreSQL stops outright at 100%. Also a hard prerequisite for any VACUUM FULL, which must rewrite the table alongside the original.

• Purged the #community:matrix.org space (!iMZEhwCvbfeAYUxAjZ:t2l.io) 2026-07-26 — 81,724,747 rows / 17GB for a directory wrapper with zero messages and one local member. Admin API DELETE /_synapse/admin/v2/rooms/ with purge:true, then VACUUM FULL. ~2.5 min downtime.

• Compressed room state with rust-synapse-compress-state (built at /opt/rust-synapse-compress-state/ on LXC 109). Every run verified New state map matches old one before commit:

  Room	Rows before	After	%
  Jellyfin General Chat	4,997,570	681,538	13.6%
  Element X Android	1,069,387	150,138	14.0%
  !fTjMjIzNKEsFlUIiru:neko.dev	811,608	461,347	56.8%
  Cinny community	258,191	142,668	55.3%
  Proxmox®	215,475	80,046	37.2%

  Do NOT compress Spam and Stuff. Its 107,763 state groups hold only 107,799 rows (~1 row each) — already minimal. The compressor's leveled structure would add rows (dry run: 110.16%) and it refuses to run. High state-group count is not the same as compressible bloat.

• PostgreSQL: jit=off, wal_compression=lz4, max_worker_processes 3→8, maintenance_work_mem 256MB→512MB, shared_buffers 1500MB→2GB in /etc/postgresql/17/main/conf.d/synapse_tuning.conf. Previous version kept as synapse_tuning.conf.bak-20260726. That file is not in this repo — the tuning is unversioned.

• synchronous_commit = off — deliberately NOT applied. Would be a real write-latency win, but this cluster hard-resets every few hours (HA self-fences on corosync quorum loss + the large1 DIMM), so it would drop up to ~600ms of committed messages per reset rather than once in a blue moon. Revisit once the cluster is stable.

Auth & SSO

• Token-based registration
• SSO/OIDC via Authelia
• allow_existing_users: true for linking accounts to SSO
• Password auth alongside SSO
• Terms of Service / consent enforcement — require_at_registration: false, block_events_error set; new users cannot send messages until they explicitly accept via /_matrix/consent; Synapse sends a Server Notice DM with the consent URL on first blocked send

Webhooks & Integrations

• matrix-hookshot 7.3.2 — 11 active webhook services
• Per-service JS transformation functions (stored in git, auto-deployed)
• Per-service virtual user avatars
• NPM reverse proxy for /webhook path

Room Structure

• The Lotus Guild space with all core rooms
• Correct power levels and join rules per room
• Custom room avatars
• Voice room visible to space members (suggested: true)

Hardening

• Rate limiting
• E2EE on all rooms (except Spam and Stuff — intentional for hookshot)
• coturn internal peer deny rules (blocks relay to RFC1918; allowed-peer-ip scoped to 10.10.10.29 only — LiveKit host)
• coturn TCP relay disabled (no-tcp-relay=true) — UDP only, reduces internal network SSRF risk
• coturn hardening: stale-nonce=600, user-quota=100, total-quota=1000, strong cipher list
• rc_joins and rc_invites rate limits explicitly set in homeserver.yaml
• pg_hba.conf locked down — remote access restricted to Synapse LXC only
• Federation open with key verification
• fail2ban on Synapse login endpoint (5 retries / 24h ban)
• Synapse metrics port 9000 restricted to 127.0.0.1 + 10.10.10.29
• coturn cert auto-renewal — daily sync cron on compute-storage-01
• /.well-known/matrix/client and /server live on lotusguild.org
• suppress_key_server_warning: true
• Automated database + media backups
• Federation bad-actor blocking via Draupnir ban lists (17,000+ entries)
• Webhook HMAC-SHA256 validation on all auto-deploy endpoints

Monitoring

• Grafana dashboard — dashboard.lotusguild.org/d/matrix-synapse-dashboard (140+ panels, Draupnir section added)
• Prometheus scraping all Matrix services (Synapse, Hookshot, LiveKit, node_exporter, postgres, Draupnir)
• 15 active alert rules across matrix-folder and infra-folder (includes Draupnir Down)
• Uptime Kuma monitors: Synapse, LiveKit, PostgreSQL, Cinny, coturn, lk-jwt-service, Hookshot
• Draupnir: node_exporter (9100), process_exporter (9256), healthz probe via blackbox (8081)

Admin

• Synapse admin API dashboard (synapse-admin at http://10.10.10.29:8080)
• Draupnir moderation bot — LXC 110, v2.9.0, all rooms + space, 2 ban lists
• Lotus Cinny custom branding — fork live at code.lotusguild.org/LotusGuild/cinny

---

Synapse Upgrade Runbook

Synapse is installed from the matrix.org apt repo (matrix-synapse-py3) on LXC 151 — it is not containerized, and none of the auto-deploy scripts in deploy/ touch it. Upgrades are manual. Access is via the Proxmox host: ssh compute-storage-01, then pct exec 151 -- <cmd>.

1. Verify the real current version — this file has been wrong before:

pct exec 151 -- dpkg-query -W -f='${Version}\n' matrix-synapse-py3
pct exec 151 -- bash -c 'apt-get update -qq && apt-cache policy matrix-synapse-py3'

2. Read upstream upgrade notes for every version in the gap: https://element-hq.github.io/synapse/latest/upgrade.html. Look for removed config keys, schema bumps, and minimum Python/PostgreSQL bumps. Then check ours against them:

pct exec 151 -- grep -nE 'experimental_features|msc[0-9]+|modules:' /etc/matrix-synapse/homeserver.yaml
pct exec 151 -- python3 -V                       # needs >= 3.10
pct exec 109 -- su postgres -c 'psql -tAc "select version()"'   # needs >= 14

3. Confirm a backup exists. Nightly vzdump snapshots all guests at 03:00 to mediafs (keep-last=2), plus a weekly run to pbs. Both LXC 151 (Synapse) and 109 (PostgreSQL) are covered — a rollback needs both, since they'd have to be restored to a consistent point:

pvesm list mediafs | grep -E '\-(151|109)\-'

Take an on-demand backup instead if the nightly one is stale: vzdump 151 109 --storage mediafs --mode snapshot. Note that Synapse does not support downgrading once new code has run against the database — treat every upgrade as one-way in practice, regardless of whether the schema version changed.

4. Upgrade. --force-confold is essential: homeserver.yaml is a dpkg conffile and we have modified it heavily, so without this apt may prompt or overwrite it.

pct exec 151 -- bash -c 'DEBIAN_FRONTEND=noninteractive apt-get install -y \
  -o Dpkg::Options::=--force-confold -o Dpkg::Options::=--force-confdef \
  matrix-synapse-py3=<VERSION>+trixie1'

The package restarts matrix-synapse itself.

5. Verify:

pct exec 151 -- systemctl is-active matrix-synapse
pct exec 151 -- ls /etc/matrix-synapse/*.dpkg-*        # must be empty — config was preserved
pct exec 151 -- curl -s http://localhost:8008/_matrix/federation/v1/version
pct exec 151 -- journalctl -u matrix-synapse --since '5 min ago' -p err --no-pager

Then confirm background updates drain (empty result = done) before any further upgrade:

pct exec 109 -- su postgres -c "psql -d synapse -tAc 'select update_name from background_updates'"

6. Record it — update the version in the infra table, the Tech Stack table, and the Server Checklist in this file.

---

Monitoring & Observability

Prometheus Scrape Jobs

Job	Target	Metrics
synapse	10.10.10.29:9000	Full Synapse internals
matrix-admin	10.10.10.29:9101	DAU, MAU, room/user/media totals
livekit	10.10.10.29:6789	Rooms, participants, packets, latency
hookshot	10.10.10.29:9004	Connections, API calls/failures, Node.js runtime
matrix-node	10.10.10.29:9100	CPU, RAM, network, load average, disk
postgres	10.10.10.44:9187	pg_stat_database, connections, WAL, block I/O
postgres-node	10.10.10.44:9100	CPU, RAM, network, load average, disk
draupnir-node	10.10.10.24:9100	CPU, RAM, network, load average, disk
draupnir-process	10.10.10.24:9256	Process CPU/memory/threads/uptime (process_exporter)
draupnir-healthz	10.10.10.24:8081/healthz → 127.0.0.1:9115	probe_success (1=healthy, 0=disconnected) via blackbox exporter

Disk I/O: All servers use Ceph-backed storage. Per-device disk I/O metrics are meaningless — use Network I/O panels to see actual storage traffic.

Alert Rules

Matrix folder:

Alert	Fires when	Severity
Synapse Down	up{job="synapse"} < 1 for 2m	critical
PostgreSQL Down	pg_up < 1 for 2m	critical
LiveKit Down	up{job="livekit"} < 1 for 2m	critical
Hookshot Down	up{job="hookshot"} < 1 for 2m	critical
Draupnir Down	up{job="draupnir-node"} < 0.5 for 2m	critical
PG Connection Saturation	connections > 80% of max for 5m	warning
Federation Queue Backing Up	pending PDUs > 100 for 10m	warning
Synapse High Memory	RSS > 2000MB for 10m	warning
Synapse High Response Time	p99 latency (excl. /sync) > 10s for 5m	warning
Synapse Event Processing Lag	any processor > 300s behind for 15m	warning
Synapse DB Query Latency High	p99 query time > 1s for 5m	warning

Infrastructure folder:

Alert	Fires when	Severity
Service Exporter Down	any up == 0 for 3m	critical
Node High CPU Usage	CPU > 90% for 10m	warning
Node High Memory Usage	RAM > 90% for 10m	warning
Node Disk Space Low	available < 15% (excl. tmpfs/overlay) for 10m	warning

/sync long-poll: The Matrix /sync endpoint is a long-poll (clients hold it open ≤30s). It is excluded from the High Response Time alert to prevent false positives.

Synapse Event Processing Lag alert fires when synapse_event_processing_lag > 300s for 15 consecutive minutes (threshold raised from 120s/5m to reduce noise from normal federation backoff cycling).

Root cause: several federated servers (bark.lgbt, parodia.dev, etc.) have asymmetric connectivity — they can reach us but we cannot reach their federation ports. Each inbound transaction they send resets our backoff to 0, triggering a new outbound connection attempt that hangs for ~90s (TCP User timeout). This causes the lag metric to spike. Mitigations in place:

1. tcp_retries2=5 in /etc/sysctl.d/99-matrix-tuning.conf — TCP hangs now fail in ~15-30s
2. ip route add unreachable <ip> in /etc/network/interfaces post-up — outbound connections to these servers fail in 0ms (ICMP unreachable)
3. Alert threshold raised to 300s/15m — only fires for genuine outages, not normal 10-min backoff cycles

To find new offending servers: grep "User timeout\|ConnectingCancell" /var/log/matrix-synapse/homeserver.log | grep -oP "\[([^\]]+)\]" | sort | uniq -c | sort -rn | head -20

---

LotusBot

LotusBot (@lotusbot:matrix.lotusguild.org) is a Matrix bot running on LXC 151 at /opt/matrixbot/.
All commands use the ! prefix. Run !help in any room for the full list.

AI / Fun

Command	Description
!ask <question>	Ask the AI anything
!fortune	Get a fortune cookie
!8ball <question>	Magic 8-ball (yes/no/maybe, funny style). --debug shows raw AI output
!roast @user	Roast someone
!story <prompt>	Generate a short story
!debate <topic>	AI argues both sides of a topic

Games

Command	Description
!wordle	Daily Wordle-style word game
!trivia [category]	Trivia question (gaming/tech/movies/music/science/anime/etc.)
!rps <rock|paper|scissors>	Rock Paper Scissors
!poll <question> | option1 | option2...	Create a reaction poll
!hangman [--hard] [--extended]	Hangman — --hard uses long words, --extended adds more body parts
!guess <letter or word>	Guess a letter or the full word in hangman
!scramble	Unscramble the word before time runs out
!wyr	Would You Rather — two AI-generated options, vote with reactions
!riddle	AI generates a riddle — try to solve it!
!numguess	Number Guess — bot picks 1–100
!ng <number>	Guess in an active number game (temperature hints included)
!wordchain	Word Chain — each word must start with the last letter of the previous
!wc <word>	Add a word to the chain
!endwc	End the word chain and see the final score
!acronym	AI picks an acronym — submit the funniest expansion with !ac then vote
!ac <expansion>	Submit an acronym expansion
!20q	20 Questions — AI thinks of something, you ask yes/no questions
!q <question>	Ask a yes/no question in 20Q
!answer <guess>	Guess the answer in 20Q
!nhie	Never Have I Ever — react 🙋 (have) or 🙅 (never)
!hottake	AI generates a hot take — react 🔥 (agree) or 💧 (disagree)
!ttt @user	Tic-Tac-Toe — challenge someone
!move <1-9>	Make a move in Tic-Tac-Toe
!blackjack	Play Blackjack against the dealer
!hit	Draw another card in Blackjack
!stand	Stand — dealer plays out
!triviaduel @user	Trivia Duel — first-to-3 battle
!da <A/B/C/D or answer>	Answer in a Trivia Duel

Random

Command	Description
!flip	Flip a coin
!roll [NdN]	Roll dice (e.g. !roll 2d6)
!random <min> <max>	Random number in range
!champion	Pick a random champion
!agent [role]	Pick a random Valorant agent

Server

Command	Description
!minecraft	Check Minecraft server status
!ping	Check bot latency
!health	Bot health + uptime stats

---

Tech Stack

Component	Technology	Version
Homeserver	Synapse	1.157.2
Database	PostgreSQL	17.9
TURN	coturn	latest
Video/voice calls	LiveKit SFU	1.9.11
LiveKit JWT	lk-jwt-service	latest
Moderation	Draupnir	2.9.0
SSO	Authelia (OIDC) + LLDAP	—
Webhook bridge	matrix-hookshot	7.3.2
Reverse proxy	Nginx Proxy Manager	—
Web client	Lotus Cinny (fork of cinnyapp/cinny main)	custom
Element Call embed	@lotusguild/element-call-embedded (self-built fork of element-hq/element-call)	0.20.1-lotus.1
GIF picker	Giphy JS/React SDK (@giphy/react-components)	—
Auto-deploy	adnanh/webhook	2.8.0
Bot language	Python 3	3.x
Bot library	matrix-nio (E2EE)	latest
Bot dependencies	matrix-nio[e2ee], aiohttp, python-dotenv, mcrcon	—