matrix/
├── hookshot/           # Hookshot JS transformation functions (one file per webhook)
│   ├── deploy.sh       # Deploys all .js files to Matrix room state via API
│   ├── proxmox.js
│   ├── grafana.js
│   ├── uptime-kuma.js
│   └── ...             # One .js per webhook service
├── cinny/
│   ├── config.json          # Cinny homeserver config (deployed to /var/www/html/config.json)
│   ├── upstream-check.sh    # Daily script: checks if cinnyapp/cinny main has new commits, pings Matrix
│   └── lotus-build.sh       # Merge + build script: fetches upstream/main, merges, builds, deploys
├── landing/
│   └── index.html      # matrix.lotusguild.org landing page
├── draupnir/
│   └── production.yaml # Draupnir config (access token is redacted — see rotation docs below)
├── deploy/             # Auto-deployment infrastructure
│   ├── lxc151-hookshot.sh          # Deploy script for LXC 151 (matrix/hookshot/livekit)
│   ├── lxc106-cinny.sh             # Deploy script for LXC 106 (cinny)
│   ├── lxc139-landing.sh           # Deploy script for LXC 139 (landing page)
│   ├── lxc110-draupnir.sh          # Deploy script for LXC 110 (draupnir)
│   ├── livekit-graceful-restart.sh # Waits for zero active calls before restarting livekit
│   ├── hooks-lxc151.json           # webhook binary config for LXC 151
│   ├── hooks-lxc106.json           # webhook binary config for LXC 106
│   ├── hooks-lxc139.json           # webhook binary config for LXC 139
│   └── hooks-lxc110.json           # webhook binary config for LXC 110
└── systemd/
    ├── livekit-server.service              # LiveKit systemd unit (with HA migration fix)
    ├── livekit-graceful-restart.service    # oneshot — checks pending restart flag
    ├── livekit-graceful-restart.timer      # Runs every 5 min
    ├── draupnir.service
    └── cinny-upstream-check.cron           # Installed to /etc/cron.d/ on LXC 106 — runs daily at noon

Infrastructure

Service	IP	LXC	RAM	Disk	Versions
Synapse	10.10.10.29	151	8GB	50GB	Synapse 1.149.0, LiveKit 1.9.11, hookshot 7.3.2, coturn latest
PostgreSQL 17	10.10.10.44	109	6GB	30GB	PostgreSQL 17.9
Cinny Web	10.10.10.6	106	2GB	8GB	Debian 12, nginx, Node 24, Lotus Cinny fork (custom, tracks `cinnyapp/cinny` main)
Draupnir	10.10.10.24	110	1GB	10GB	Draupnir v2.9.0, Node.js v22
Prometheus	10.10.10.48	118	—	—	Prometheus — scrapes all Matrix services
Grafana	10.10.10.49	107	—	—	Grafana 12.4.0 — dashboard.lotusguild.org
NPM	10.10.10.27	139	—	—	Nginx Proxy Manager + matrix landing page
Authelia	10.10.10.36	167	—	—	SSO/OIDC provider
LLDAP	10.10.10.39	147	—	—	LDAP user directory
Uptime Kuma	10.10.10.25	101	—	—	Uptime monitoring (micro1 node)

Key paths on Synapse LXC (151):

Synapse config: /etc/matrix-synapse/homeserver.yaml
Synapse conf.d: /etc/matrix-synapse/conf.d/ (metrics.yaml, report_stats.yaml, server_name.yaml)
coturn config: /etc/turnserver.conf
LiveKit config: /etc/livekit/config.yaml
LiveKit service: livekit-server.service
lk-jwt-service: lk-jwt-service.service (binds :8070, serves JWT tokens for MatrixRTC at /sfu/get and legacy /get_token)
Hookshot: /opt/hookshot/, service: matrix-hookshot.service
Hookshot config: /opt/hookshot/config.yml
Hookshot registration: /etc/matrix-synapse/hookshot-registration.yaml
Bot: /opt/matrixbot/, service: matrixbot.service
Repo clone (auto-deploy): /opt/matrix-config/
Deploy env: /etc/matrix-deploy.env (MATRIX_TOKEN, MATRIX_SERVER, MATRIX_ROOM)
Deploy log: /var/log/matrix-deploy.log

Key paths on Draupnir LXC (110):

Install path: /opt/draupnir/
Config: /opt/draupnir/config/production.yaml
Data/SQLite DBs: /data/storage/
Service: draupnir.service
Management room: #management:matrix.lotusguild.org (!mEvR5fe3jMmzwd-FwNygD72OY_yu8H3UP_N-57oK7MI)
Bot account: @draupnir:matrix.lotusguild.org (power level 100 in all protected rooms and the Lotus Guild space)
Subscribed ban lists: #community-moderation-effort-bl:neko.dev, #matrix-org-coc-bl:matrix.org
Rebuild: NODE_OPTIONS="--max_old_space_size=6144" npm run build
Healthz endpoint: http://10.10.10.24:8081/healthz (200 = healthy, 418 = disconnected)
Abuse reporting endpoint: POST http://10.10.10.24:8080/_matrix/draupnir/1/report/{roomId}/{eventId}
Audit DBs: /data/storage/user-restriction-audit-log.db, /data/storage/room-audit-log.db

Key paths on PostgreSQL LXC (109):

PostgreSQL config: /etc/postgresql/17/main/postgresql.conf
Tuning conf.d: /etc/postgresql/17/main/conf.d/synapse_tuning.conf
HBA config: /etc/postgresql/17/main/pg_hba.conf
Data directory: /var/lib/postgresql/17/main

Key paths on Cinny LXC (106):

Lotus fork source: /opt/lotus-cinny/ (fork of cinnyapp/cinny main, custom Lotus Guild branch)
Upstream remote: https://github.com/cinnyapp/cinny.git (added as upstream)
Built files: /var/www/html/
Cinny config: /var/www/html/config.json
Config backup (survives rebuilds): /opt/lotus-cinny/.cinny-config.json
Monitor env: /etc/cinny-monitor.env (MATRIX_TOKEN, MATRIX_SERVER, MATRIX_ROOM, MATRIX_PING_USER — not in git)
Upstream check script: /usr/local/bin/cinny-upstream-check.sh
Build/deploy script: /usr/local/bin/cinny-build.sh (triggered by webhook or manual run)
Cron: /etc/cron.d/cinny-upstream-check (runs at noon daily — checks only, does not auto-build)
Monitor state: /var/lib/cinny-monitor/last-upstream-commit
Monitor log: /var/log/cinny-monitor.log
Build log: /var/log/cinny-build.log
Nginx site config: /etc/nginx/sites-available/cinny

Auto-Deployment

Pushes to main on LotusGuild/matrix automatically deploy to the relevant LXC(s) via Gitea webhooks. All 4 LXCs are fully independent — each runs its own webhook listener and deploys only its own files. No cross-LXC SSH dependencies.

How It Works

Push to LotusGuild/matrix on Gitea
Gitea fires webhooks to all 4 LXCs simultaneously (HMAC-SHA256 validated)
Each LXC runs /usr/local/bin/matrix-deploy.sh via the webhook binary
Script does git fetch + reset --hard origin/main, checks which files changed, deploys only relevant ones
Logs to /var/log/matrix-deploy.log on each LXC

Per-LXC Webhook Endpoints

LXC	Service	IP	Port	Deploys When Changed
151	matrix/hookshot	10.10.10.29	9500	`hookshot/*.js`, `systemd/livekit-server.service`
106	cinny	10.10.10.6	9000	`cinny/config.json`, `cinny/upstream-check.sh`, `cinny/lotus-build.sh`, `deploy/hooks-lxc106.json`, `systemd/cinny-upstream-check.cron`
139	landing/NPM	10.10.10.27	9000	`landing/index.html`
110	draupnir	10.10.10.24	9000	`draupnir/production.yaml`

LXC 151 uses port 9500 because ports 9000–9004 are occupied by Synapse and Hookshot.

What Each Deploy Does

LXC 151 — hookshot/livekit:

hookshot/*.js changed → runs hookshot/deploy.sh (pushes transform functions to Matrix room state via API, requires MATRIX_TOKEN in /etc/matrix-deploy.env)
systemd/livekit-server.service changed → copies file, daemon-reload, sets /run/livekit-restart-pending flag (actual restart deferred — see Livekit Graceful Restart below)

LXC 106 — cinny:

cinny/config.json → copies to /var/www/html/config.json
cinny/upstream-check.sh → copies to /usr/local/bin/cinny-upstream-check.sh, chmod +x
cinny/lotus-build.sh → copies to /usr/local/bin/cinny-build.sh, chmod +x
deploy/hooks-lxc106.json → copies to /etc/webhook/hooks.json, restarts webhook service
systemd/cinny-upstream-check.cron → copies to /etc/cron.d/cinny-upstream-check, chmod 644

LXC 139 — landing page:

landing/index.html → copies to /var/www/matrix-landing/index.html, nginx -s reload

LXC 110 — draupnir:

draupnir/production.yaml → extracts live accessToken from existing config, overwrites from repo, restores token via sed, restarts draupnir.service

Installed Components (per LXC)

webhook binary (Debian package webhook v2.8.0) listening on respective port
/etc/webhook/hooks.json — unique HMAC-SHA256 secret per LXC
/usr/local/bin/matrix-deploy.sh — deploy script from this repo
/etc/systemd/system/webhook.service — enabled and running
/opt/matrix-config/ — clone of this repo
/var/log/matrix-deploy.log — deploy log

LXC 151 additionally:

/etc/matrix-deploy.env — MATRIX_TOKEN, MATRIX_SERVER, MATRIX_ROOM (not in git)
/usr/local/bin/livekit-graceful-restart.sh
/etc/systemd/system/livekit-graceful-restart.service + .timer

LXC 106 additionally:

/etc/cinny-monitor.env — MATRIX_TOKEN, MATRIX_SERVER, MATRIX_ROOM, MATRIX_PING_USER (not in git)
/var/lib/cinny-monitor/last-upstream-commit — state file (tracks last-seen upstream SHA)
/opt/lotus-cinny/ — git clone of code.lotusguild.org/LotusGuild/cinny with upstream remote (cinnyapp/cinny)
/root/.git-credentials — Gitea token lxc106-lotus-cinny (write:repository scope, revocable via Gitea UI)
/var/lib/cinny-monitor/last-upstream-tag — last seen stable release tag (e.g. v4.11.1)

Livekit Graceful Restart

Killing livekit-server while a call is active drops everyone. Instead:

Deploy to LXC 151 copies the new livekit-server.service and sets a /run/livekit-restart-pending flag
livekit-graceful-restart.timer runs every 5 minutes
The timer script counts established TCP connections on port 7881 (ss -tn state established)
If zero connections → restarts livekit-server and clears the flag
If connections exist → logs and exits, retries in 5 minutes

Access Token Rotation

The MATRIX_TOKEN in /etc/matrix-deploy.env on LXC 151 is a Jared user token used to push hookshot transforms to Matrix room state (requires power level ≥ 50 in Spam and Stuff).

The token in draupnir/production.yaml in this repo is intentionally redacted (accessToken: REDACTED). The deploy script on LXC 110 extracts the live token from the running config before overwriting from the repo, then restores it.

To rotate the hookshot deploy token (LXC 151):

Generate a new token via Synapse admin API or Cinny → Settings → Security → Manage Sessions
SSH to LXC 151 (via ssh root@10.10.10.4 then pct enter 151): nano /etc/matrix-deploy.env
Replace MATRIX_TOKEN=<old> with new token
Test: MATRIX_TOKEN=<new> MATRIX_SERVER=https://matrix.lotusguild.org bash /opt/matrix-config/hookshot/deploy.sh

To rotate the Draupnir token:

Generate new token for @draupnir:matrix.lotusguild.org
On LXC 110: nano /opt/draupnir/config/production.yaml → update accessToken
systemctl restart draupnir
Do not commit the token to git — the repo version stays redacted

Port Maps

Router → 10.10.10.29 (forwarded):

TCP+UDP 3478 — TURN/STUN
TCP+UDP 5349 — TURNS/TLS
TCP 7881 — LiveKit ICE TCP fallback
TCP+UDP 49152-65535 — TURN relay range

Internal port map (LXC 151):

Port	Service	Bind
8008	Synapse HTTP	0.0.0.0
9000	Synapse metrics	127.0.0.1 + 10.10.10.29
9001	Hookshot widgets	0.0.0.0
9002	Hookshot bridge (appservice)	127.0.0.1
9003	Hookshot webhooks	0.0.0.0
9004	Hookshot metrics	0.0.0.0
9100	node_exporter	0.0.0.0
9101	matrix-admin exporter	0.0.0.0
9500	webhook (auto-deploy)	0.0.0.0
6789	LiveKit metrics	0.0.0.0
7880	LiveKit HTTP	0.0.0.0
7881	LiveKit RTC TCP	0.0.0.0
8070	lk-jwt-service	0.0.0.0
8080	synapse-admin (nginx)	0.0.0.0
3478	coturn STUN/TURN	0.0.0.0
5349	coturn TURNS/TLS	0.0.0.0

Internal port map (LXC 110 — Draupnir):

Port	Service	Bind
8080	Draupnir web (abuse reporting)	0.0.0.0
8081	Draupnir healthz	0.0.0.0
9000	webhook (auto-deploy)	0.0.0.0
9100	node_exporter	0.0.0.0
9256	process_exporter	0.0.0.0

Internal port map (LXC 109 — PostgreSQL):

Port	Service	Bind
5432	PostgreSQL	0.0.0.0 (hba-restricted to 10.10.10.29)
9100	node_exporter	0.0.0.0
9187	postgres_exporter	0.0.0.0

Rooms (all v12)

Room	Room ID	Join Rule
The Lotus Guild (Space)	`!-1ZBnAH-JiCOV8MGSKN77zDGTuI3pgSdy8Unu_DrDyc`	public
General	`!wfokQ1-pE896scu_AOcCBA2s3L4qFo-PTBAFTd0WMI0`	public
Commands	`!ou56mVZQ8ZB7AhDYPmBV5_BR28WMZ4x5zwZkPCqjq1s`	restricted (Space members)
Memes	`!GK6v5cLEEnowIooQJv5jECfISUjADjt8aKhWv9VbG5U`	restricted (Space members)
Music	`!ktQu0gavhjpCMkgxk8SYdb6mnJRY-u7mY7_KfksV0SU`	restricted (Space members)
Voice Room	`!ARbRFSPNp2U0MslWTBGoTT3gbmJJ25dPRL6enQntvPo`	restricted (Space members)
Management	`!mEvR5fe3jMmzwd-FwNygD72OY_yu8H3UP_N-57oK7MI`	invite
Cool Kids	`!R7DT3QZHG9P8QQvX6zsZYxjkKgmUucxDz_n31qNrC94`	invite
Spam and Stuff	`!GttT4QYd1wlGlkHU3qTmq_P3gbyYKKeSSN6R7TPcJHg`	invite, no E2EE (hookshot)

Power level roles (Cinny tags):

100: Owner (jared, draupnir, lotusbot)
50: The Nerdy Council / Panel of Geeks (enhuynh, lonely)
0: Member

Webhook Integrations (matrix-hookshot 7.3.2)

Generic webhooks bridged into Spam and Stuff. Each service gets its own virtual user (@hookshot_<service>) with a unique avatar. Webhook URL format: https://matrix.lotusguild.org/webhook/<uuid>

Service	Webhook UUID	Notes
Grafana	`df4a1302-2d62-4a01-b858-fb56f4d3781a`	Unified alerting contact point
Proxmox	`9b3eafe5-7689-4011-addd-c466e524661d`	Notification system (8.1+), Discord embed format
Sonarr	`aeffc311-0686-42cb-9eeb-6757140c072e`	All event types
Radarr	`34913454-c1ac-4cda-82ea-924d4a9e60eb`	All event types
Readarr	`e57ab4f3-56e6-4dc4-8b30-2f4fd4bbeb0b`	All event types
Lidarr	`66ac6fdd-69f6-4f47-bb00-b7f6d84d7c1c`	All event types
Uptime Kuma	`1a02e890-bb25-42f1-99fe-bba6a19f1811`	Status change notifications
Seerr	`555185af-90a1-42ff-aed5-c344e11955cf`	Request/approval events
Owncast (Livestream)	`9993e911-c68b-4271-a178-c2d65ca88499`	STREAM_STARTED / STREAM_STOPPED
Bazarr	`470fb267-3436-4dd3-a70c-e6e8db1721be`	Subtitle events (Apprise JSON notifier)
Tinker-Tickets	`6e306faf-8eea-4ba5-83ef-bf8f421f929e`	Custom transformation code

Hookshot notes:

Spam and Stuff is intentionally unencrypted — hookshot bridges cannot join E2EE rooms
JS transformation functions use hookshot v2 API: result = { version: "v2", plain, html, msgtype }
The result variable must be assigned without var/let/const (QuickJS IIFE sandbox)
NPM proxies https://matrix.lotusguild.org/webhook/* → http://10.10.10.29:9003
NPM proxies /sfu/get and /get_token → http://10.10.10.29:8070 (lk-jwt-service). Both paths are in /data/nginx/proxy_host/49.conf on LXC 139 — NPM will overwrite these if proxy host 49 is re-saved via the UI; re-add both location blocks after any NPM save
Proxmox sends Discord embed format: data.embeds[0].{title,description,fields} — NOT flat fields
Transform functions are stored as Matrix room state (uk.half-shot.matrix-hookshot.generic.hook) and deployed via hookshot/deploy.sh

Deploying hookshot transforms manually:

# On LXC 151 or from any machine with access
export MATRIX_TOKEN=<jared_token>
export MATRIX_SERVER=https://matrix.lotusguild.org
export MATRIX_ROOM='!GttT4QYd1wlGlkHU3qTmq_P3gbyYKKeSSN6R7TPcJHg'
bash /opt/matrix-config/hookshot/deploy.sh          # deploy all
bash /opt/matrix-config/hookshot/deploy.sh proxmox.js  # deploy one

Moderation (Draupnir v2.9.0)

Draupnir runs on LXC 110, manages moderation across all protected rooms (including the Lotus Guild space) via #management:matrix.lotusguild.org.

Subscribed ban lists:

#community-moderation-effort-bl:neko.dev — 12,599 banned users, 245 servers, 59 rooms
#matrix-org-coc-bl:matrix.org — 4,589 banned users, 220 servers, 2 rooms

Common commands (send in management room):

!draupnir status                          — current status + protected rooms
!draupnir ban @user:server * "reason"     — ban from all protected rooms
!draupnir redact @user:server             — redact their recent messages
!draupnir rooms add !roomid:server        — add a room to protection
!draupnir watch <alias> --no-confirm      — subscribe to a ban list

Abuse Reporting

When a Matrix client user clicks "Report" on a message, Synapse receives a POST /_matrix/client/v3/rooms/{roomId}/report/{eventId} request and stores the report internally. To forward these to the Draupnir management room, a Synapse Python module must be installed on LXC 151.

Draupnir web server is enabled (port 8080). The endpoint is:

POST http://10.10.10.24:8080/_matrix/draupnir/1/report/{roomId}/{eventId}

To complete Synapse integration (one-time, on LXC 151):

Install the module: pip install matrix-synapse-draupnir-abuse-reports (or equivalent — check Draupnir releases)

Add to /etc/matrix-synapse/homeserver.yaml:

modules:
  - module: "draupnir.abuse_reports.AbuseReportEndpoint"
    config:
      draupnir_endpoint: "http://10.10.10.24:8080"

systemctl restart matrix-synapse

Until the Synapse module is installed, abuse reports are stored in Synapse's DB but do NOT appear in the management room. The Draupnir web server is running and ready to receive forwarded reports.

Lotus Cinny (chat.lotusguild.org)

chat.lotusguild.org serves a custom Lotus Guild fork of the official cinnyapp/cinny main branch. The fork lives at code.lotusguild.org/LotusGuild/cinny and tracks upstream via a git remote add upstream https://github.com/cinnyapp/cinny.git workflow.

Upstream monitoring (daily at noon):

cinny-upstream-check.sh hits the GitHub API and compares the latest cinnyapp/cinny main commit against the stored SHA in /var/lib/cinny-monitor/last-upstream-commit
If new commits exist, sends a Matrix message to Spam and Stuff with an @jared:matrix.lotusguild.org ping and a link to the commit
Does not auto-build — you review the diff and decide when to merge

Merge + build workflow:

Receive upstream notification in Matrix
Review the diff: https://github.com/cinnyapp/cinny/compare/<old>...<new>
Send !cinny-update in any Matrix room — LotusBot POSTs to the cinny-build webhook on LXC 106
cinny-build.sh runs: git fetch upstream && git merge upstream/main, npm ci, npm run build, deploys to /var/www/html/
Build result (success or conflict) is posted back to Matrix

Manual build (SSH):

# On LXC 106
/usr/local/bin/cinny-build.sh

Merge conflict recovery:

# On LXC 106
cd /opt/lotus-cinny
git merge upstream/main    # resolve conflicts in editor
git add -A && git merge --continue
/usr/local/bin/cinny-build.sh

LXC 106 one-time setup (after forking cinnyapp/cinny to code.lotusguild.org/LotusGuild/cinny):

# On LXC 106
git clone https://code.lotusguild.org/LotusGuild/cinny.git /opt/lotus-cinny
cd /opt/lotus-cinny
git remote add upstream https://github.com/cinnyapp/cinny.git
git fetch upstream

# Create env file (fill in a valid Matrix token)
cat > /etc/cinny-monitor.env << 'EOF'
MATRIX_TOKEN=<jared_or_bot_token>
MATRIX_SERVER=https://matrix.lotusguild.org
MATRIX_ROOM=!GttT4QYd1wlGlkHU3qTmq_P3gbyYKKeSSN6R7TPcJHg
MATRIX_PING_USER=@jared:matrix.lotusguild.org
EOF
chmod 600 /etc/cinny-monitor.env

Cinny-build webhook token (for LotusBot !cinny-update): stored in deploy/hooks-lxc106.json (cinny-build hook, header X-Build-Token). LotusBot must POST to http://10.10.10.6:9000/hooks/cinny-build with this header.

Why 8GB RAM: Vite's build process needs ~6GB Node heap (--max_old_space_size=6144) for the rendering-chunks phase. Previously at 4GB — OOM killed during render.

Custom Features

All custom code lives in src/app/ on the lotus branch of code.lotusguild.org/LotusGuild/cinny. Changes survive upstream merges as long as they don't conflict with the same files upstream touched.

Feature	Files	Notes
Element Call embed	`src/app/plugins/call/`, `src/app/hooks/useCallEmbed.ts`, `src/app/components/CallEmbedProvider.tsx`	EC 0.19.3 (`@element-hq/element-call-embedded`), dist copied to `public/element-call/` by vite
DM calls	`src/app/features/room/Room.tsx`, `src/app/features/room/RoomViewHeader.tsx`	Phone button in DM room header; `useCallStart(true)` passes `intent: StartedByUser`; Room.tsx switches to CallView layout when DM has active call
Picture-in-picture call	`src/app/components/CallEmbedProvider.tsx`	When navigating away from the call room, the embed shrinks to a 280×158px PiP in the bottom-right. Click navigates back. Implemented via `useEffect` imperatively overriding styles on `callEmbedRef.current` — cannot use a wrapper div because `useCallEmbedPlacementSync` writes `top/left/width/height` directly onto that element
Screenshare fullscreen	`src/app/features/call/CallControls.tsx`, `src/app/features/call/Controls.tsx`	When screensharing, a fullscreen button appears in call controls. Calls `callEmbedRef.current?.requestFullscreen()` on the Cinny call container. EC naturally spotlights the screenshare — the old 600ms grid-revert code was removed (it caused fullscreen to show avatars instead of the screen)
PiP screenshare focus	`src/app/components/CallEmbedProvider.tsx`, `src/app/plugins/call/CallControl.ts`	When the floating PiP window is active and screenshare is detected (no cameras present), auto-enables EC spotlight view so the screenshare fills the PiP rather than showing avatar tiles
Screenshare audio mute	`src/app/features/call/Controls.tsx`, `src/app/features/call/CallControls.tsx`, `src/app/plugins/call/CallControl.ts`	Dedicated button to independently mute/unmute audio from screenshares without muting microphone audio. Targets `audio[data-lk-source="screen_share_audio"]` LiveKit elements. Persists across deafen/undeafen cycles
Custom status message	`src/app/features/settings/account/Profile.tsx`, `src/app/features/room/MembersDrawer.tsx`, `src/app/components/user-profile/UserHero.tsx`, `src/app/components/user-profile/UserRoomProfile.tsx`, `src/app/hooks/useUserPresence.ts`	Discord-style free-form status text. Set via Settings → Account → "Status Message" with an emoji picker (lazy-loaded `EmojiBoard`). Saved via `mx.setPresence({ status_msg })`. Displayed below the username in the members drawer and user profile popout. Syncs live via Matrix presence events
PTT (Push-to-Talk)	`src/app/features/call/CallControls.tsx`, `src/app/state/settings.ts`	Hold-to-talk key (default: Space, configurable). Mutes mic on join; holds mic open while key is held. Badge shows `PTT — Hold SPACE` / `● Live`. Listens on both main window and EC iframe `contentWindow` for key events
PTT badge theming	`src/app/features/call/CallControls.tsx`	Plain folds `Chip` by default; neon terminal style (`#00FF88`/`#FF6B00`, JetBrains Mono) when `lotusTerminal` setting is on
GIF picker	`src/app/components/GifPicker.tsx`, `src/app/features/room/RoomInput.tsx`	Giphy JS/React SDK (`@giphy/react-components`, `@giphy/js-fetch-api`, `styled-components`). API key in `config.json` → `gifApiKey`. GIF button appears next to Send only when `gifApiKey` is set. Sends GIF as `m.image` (fetches blob → `mx.uploadContent` → `mx.sendMessage`). `FocusTrap` handles click-outside / Escape to close
GIF picker terminal theme	`src/app/components/GifPicker.tsx`	When `lotusTerminal` is on: dark navy background (`#060c14`), orange dim border, 4px radius, `// GIF_SEARCH` header, injected `<style>` overrides Giphy SDK SearchBar input (dark bg, orange border/focus ring, JetBrains Mono), custom orange scrollbar
Terminal Design System toggle	`src/app/state/settings.ts`, `src/app/features/settings/`	`lotusTerminal` boolean setting. When enabled: PTT badge and GIF picker use LotusGuild Terminal Design System aesthetics
Presence status badges	`src/app/features/room/MembersDrawer.tsx`, `src/app/features/common-settings/members/Members.tsx`, `src/app/hooks/useUserPresence.ts`, `src/app/components/presence/`	Online/busy/away colored dot badges shown next to verification shields for every member in the room members drawer and settings members panel. Uses `useUserPresence(userId)` hook + `PresenceBadge` component. Members.tsx wraps the hook in a `MemberPresenceBadge` child component to satisfy React hook rules inside `.map()`
Per-member device sessions panel	`src/app/components/user-profile/UserRoomProfile.tsx`, `src/app/hooks/useOtherUserDevices.ts`	Collapsible "Sessions" card in user profile popout. Lists all devices for any user with colored shield icons (green=verified, yellow=unverified). Shows per-device "Verify" button that initiates cross-signing SAS emoji verification via `crypto.requestDeviceVerification(userId, deviceId)`. Updates live via `CryptoEvent.DevicesUpdated`. Only shown when cross-signing is active
LiveKit codec config	`/etc/livekit/config.yaml` (LXC 151)	`enabled_codecs`: VP8, H264, VP9, Opus, RED for better quality and redundancy

Key config values (/opt/lotus-cinny/config.json, root — vite copies this to dist):

{
  "defaultHomeserver": 0,
  "homeserverList": ["matrix.lotusguild.org"],
  "allowCustomHomeservers": false,
  "gifApiKey": "AqqDuQwZNjYttz7Mn6ME4JH1bJIuZ5CO"
}

Note: The root /opt/lotus-cinny/config.json is what matters — vite copies it to dist/. public/config.json is not used.

Known Issues

LiveKit Port Conflict After HA Migration

LXC 151 can migrate between Proxmox nodes via HA. After migration, the old livekit-server process on the source node can leave a stale entry holding port 7881 on the destination. Fixed in livekit-server.service via:

ExecStartPre=-/bin/bash -c 'pkill -x livekit-server; sleep 1'
KillMode=control-group

coturn TLS Reset Errors

Periodic TLS/TCP socket error: Connection reset by peer in coturn logs. Normal — clients probe TURN and drop once they establish a direct P2P path.

BBR Congestion Control

net.ipv4.tcp_congestion_control = bbr must be set on the Proxmox host, not inside an unprivileged LXC. All other sysctl tuning (TCP/UDP buffers, fin_timeout) is applied inside LXC 151.

Server Checklist

Quality of Life

Migrate from SQLite to PostgreSQL
TURN/STUN server (coturn) for reliable voice/video
URL previews
Upload size limit 200MB
Full-text message search (PostgreSQL backend)
Media retention policy (remote: 1yr, local: 3yr)
Sliding sync (native Synapse)
LiveKit for Element Call video rooms
Default room version v12, all rooms upgraded
Landing page with client recommendations
Synapse metrics endpoint (port 9000, Prometheus-compatible)
Lotus Cinny fork — custom fork of cinnyapp/cinny main, daily upstream check + Matrix notification
Auto-deployment via Gitea webhooks (all 4 LXCs)
Push notifications gateway (Sygnal) — needs Apple/Google developer credentials
Lotus Cinny custom branding — Lotus Guild theme (colours, title, favicon, PWA name)

Performance Tuning

PostgreSQL shared_buffers → 1500MB, effective_cache_size, work_mem, checkpoint tuning
PostgreSQL pg_stat_statements extension installed
PostgreSQL autovacuum tuned per-table (5 high-churn tables), autovacuum_max_workers → 5
Synapse event_cache_size → 30K, per-cache factors tuned
sysctl TCP/UDP buffer alignment on LXC 151 (/etc/sysctl.d/99-matrix-tuning.conf)
LiveKit: empty_timeout: 300, departure_timeout: 20, max_participants: 50
LiveKit ICE port range expanded to 50000-51000
LiveKit TURN TTL reduced to 1h
LiveKit VP9/AV1 codecs enabled
TCP retransmit timeout lowered (tcp_retries2=5, tcp_syn_retries=4, tcp_keepalive_probes=3) — stalled outbound federation connections now fail in ~15-30s instead of ~15 min
Unreachable routes added for servers with asymmetric connectivity (can reach us but we can't reach their federation port) — prevents 90s TCP hangs from being added to lag; defined in /etc/network/interfaces post-up hooks and survive reboots (bark.lgbt ×2, parodia.dev, chat.ohaa.xyz, matrix.k8ekat.dev)
Stuck device_lists_remote_resync entries cleared for dead-server users (@dalite:bark.lgbt, @arndot:matrix.goch.social) — device list resync was firing every 30s
BBR congestion control — must be applied on Proxmox host

Auth & SSO

Token-based registration
SSO/OIDC via Authelia
allow_existing_users: true for linking accounts to SSO
Password auth alongside SSO
Terms of Service / consent enforcement — require_at_registration: false, block_events_error set; new users cannot send messages until they explicitly accept via /_matrix/consent; Synapse sends a Server Notice DM with the consent URL on first blocked send

Webhooks & Integrations

matrix-hookshot 7.3.2 — 11 active webhook services
Per-service JS transformation functions (stored in git, auto-deployed)
Per-service virtual user avatars
NPM reverse proxy for /webhook path

Room Structure

The Lotus Guild space with all core rooms
Correct power levels and join rules per room
Custom room avatars
Voice room visible to space members (suggested: true)

Hardening

Rate limiting
E2EE on all rooms (except Spam and Stuff — intentional for hookshot)
coturn internal peer deny rules (blocks relay to RFC1918; allowed-peer-ip scoped to 10.10.10.29 only — LiveKit host)
coturn TCP relay disabled (no-tcp-relay=true) — UDP only, reduces internal network SSRF risk
coturn hardening: stale-nonce=600, user-quota=100, total-quota=1000, strong cipher list
rc_joins and rc_invites rate limits explicitly set in homeserver.yaml
pg_hba.conf locked down — remote access restricted to Synapse LXC only
Federation open with key verification
fail2ban on Synapse login endpoint (5 retries / 24h ban)
Synapse metrics port 9000 restricted to 127.0.0.1 + 10.10.10.29
coturn cert auto-renewal — daily sync cron on compute-storage-01
/.well-known/matrix/client and /server live on lotusguild.org
suppress_key_server_warning: true
Automated database + media backups
Federation bad-actor blocking via Draupnir ban lists (17,000+ entries)
Webhook HMAC-SHA256 validation on all auto-deploy endpoints

Monitoring

Grafana dashboard — dashboard.lotusguild.org/d/matrix-synapse-dashboard (140+ panels, Draupnir section added)
Prometheus scraping all Matrix services (Synapse, Hookshot, LiveKit, node_exporter, postgres, Draupnir)
15 active alert rules across matrix-folder and infra-folder (includes Draupnir Down)
Uptime Kuma monitors: Synapse, LiveKit, PostgreSQL, Cinny, coturn, lk-jwt-service, Hookshot
Draupnir: node_exporter (9100), process_exporter (9256), healthz probe via blackbox (8081)

Admin

Synapse admin API dashboard (synapse-admin at http://10.10.10.29:8080)
Draupnir moderation bot — LXC 110, v2.9.0, all rooms + space, 2 ban lists
Lotus Cinny custom branding — fork live at code.lotusguild.org/LotusGuild/cinny

Monitoring & Observability

Prometheus Scrape Jobs

Job	Target	Metrics
`synapse`	`10.10.10.29:9000`	Full Synapse internals
`matrix-admin`	`10.10.10.29:9101`	DAU, MAU, room/user/media totals
`livekit`	`10.10.10.29:6789`	Rooms, participants, packets, latency
`hookshot`	`10.10.10.29:9004`	Connections, API calls/failures, Node.js runtime
`matrix-node`	`10.10.10.29:9100`	CPU, RAM, network, load average, disk
`postgres`	`10.10.10.44:9187`	pg_stat_database, connections, WAL, block I/O
`postgres-node`	`10.10.10.44:9100`	CPU, RAM, network, load average, disk
`draupnir-node`	`10.10.10.24:9100`	CPU, RAM, network, load average, disk
`draupnir-process`	`10.10.10.24:9256`	Process CPU/memory/threads/uptime (process_exporter)
`draupnir-healthz`	`10.10.10.24:8081/healthz` → `127.0.0.1:9115`	`probe_success` (1=healthy, 0=disconnected) via blackbox exporter

Disk I/O: All servers use Ceph-backed storage. Per-device disk I/O metrics are meaningless — use Network I/O panels to see actual storage traffic.

Alert Rules

Matrix folder:

Alert	Fires when	Severity
Synapse Down	`up{job="synapse"}` < 1 for 2m	critical
PostgreSQL Down	`pg_up` < 1 for 2m	critical
LiveKit Down	`up{job="livekit"}` < 1 for 2m	critical
Hookshot Down	`up{job="hookshot"}` < 1 for 2m	critical
Draupnir Down	`up{job="draupnir-node"}` < 0.5 for 2m	critical
PG Connection Saturation	connections > 80% of max for 5m	warning
Federation Queue Backing Up	pending PDUs > 100 for 10m	warning
Synapse High Memory	RSS > 2000MB for 10m	warning
Synapse High Response Time	p99 latency (excl. /sync) > 10s for 5m	warning
Synapse Event Processing Lag	any processor > 300s behind for 15m	warning
Synapse DB Query Latency High	p99 query time > 1s for 5m	warning

Infrastructure folder:

Alert	Fires when	Severity
Service Exporter Down	any `up == 0` for 3m	critical
Node High CPU Usage	CPU > 90% for 10m	warning
Node High Memory Usage	RAM > 90% for 10m	warning
Node Disk Space Low	available < 15% (excl. tmpfs/overlay) for 10m	warning

/sync long-poll: The Matrix /sync endpoint is a long-poll (clients hold it open ≤30s). It is excluded from the High Response Time alert to prevent false positives.

Synapse Event Processing Lag alert fires when synapse_event_processing_lag > 300s for 15 consecutive minutes (threshold raised from 120s/5m to reduce noise from normal federation backoff cycling).

Root cause: several federated servers (bark.lgbt, parodia.dev, etc.) have asymmetric connectivity — they can reach us but we cannot reach their federation ports. Each inbound transaction they send resets our backoff to 0, triggering a new outbound connection attempt that hangs for ~90s (TCP User timeout). This causes the lag metric to spike. Mitigations in place:

tcp_retries2=5 in /etc/sysctl.d/99-matrix-tuning.conf — TCP hangs now fail in ~15-30s

ip route add unreachable <ip> in /etc/network/interfaces post-up — outbound connections to these servers fail in 0ms (ICMP unreachable)

Alert threshold raised to 300s/15m — only fires for genuine outages, not normal 10-min backoff cycles

To find new offending servers: grep "User timeout\|ConnectingCancell" /var/log/matrix-synapse/homeserver.log | grep -oP "\[([^\]]+)\]" | sort | uniq -c | sort -rn | head -20

LotusBot

LotusBot (@lotusbot:matrix.lotusguild.org) is a Matrix bot running on LXC 151 at /opt/matrixbot/.
All commands use the ! prefix. Run !help in any room for the full list.

AI / Fun

Command	Description
`!ask <question>`	Ask the AI anything
`!fortune`	Get a fortune cookie
`!8ball <question>`	Magic 8-ball (yes/no/maybe, funny style). `--debug` shows raw AI output
`!roast @user`	Roast someone
`!story <prompt>`	Generate a short story
`!debate <topic>`	AI argues both sides of a topic

Games

Command	Description
`!wordle`	Daily Wordle-style word game
`!trivia [category]`	Trivia question (gaming/tech/movies/music/science/anime/etc.)
`!rps <rock\|paper\|scissors>`	Rock Paper Scissors
`!poll <question> \| option1 \| option2...`	Create a reaction poll
`!hangman [--hard] [--extended]`	Hangman — `--hard` uses long words, `--extended` adds more body parts
`!guess <letter or word>`	Guess a letter or the full word in hangman
`!scramble`	Unscramble the word before time runs out
`!wyr`	Would You Rather — two AI-generated options, vote with reactions
`!riddle`	AI generates a riddle — try to solve it!
`!numguess`	Number Guess — bot picks 1–100
`!ng <number>`	Guess in an active number game (temperature hints included)
`!wordchain`	Word Chain — each word must start with the last letter of the previous
`!wc <word>`	Add a word to the chain
`!endwc`	End the word chain and see the final score
`!acronym`	AI picks an acronym — submit the funniest expansion with `!ac` then vote
`!ac <expansion>`	Submit an acronym expansion
`!20q`	20 Questions — AI thinks of something, you ask yes/no questions
`!q <question>`	Ask a yes/no question in 20Q
`!answer <guess>`	Guess the answer in 20Q
`!nhie`	Never Have I Ever — react 🙋 (have) or 🙅 (never)
`!hottake`	AI generates a hot take — react 🔥 (agree) or 💧 (disagree)
`!ttt @user`	Tic-Tac-Toe — challenge someone
`!move <1-9>`	Make a move in Tic-Tac-Toe
`!blackjack`	Play Blackjack against the dealer
`!hit`	Draw another card in Blackjack
`!stand`	Stand — dealer plays out
`!triviaduel @user`	Trivia Duel — first-to-3 battle
`!da <A/B/C/D or answer>`	Answer in a Trivia Duel

Random

Command	Description
`!flip`	Flip a coin
`!roll [NdN]`	Roll dice (e.g. `!roll 2d6`)
`!random <min> <max>`	Random number in range
`!champion`	Pick a random champion
`!agent [role]`	Pick a random Valorant agent

Server

Command	Description
`!minecraft`	Check Minecraft server status
`!ping`	Check bot latency
`!health`	Bot health + uptime stats

Tech Stack

Component	Technology	Version
Homeserver	Synapse	1.149.0
Database	PostgreSQL	17.9
TURN	coturn	latest
Video/voice calls	LiveKit SFU	1.9.11
LiveKit JWT	lk-jwt-service	latest
Moderation	Draupnir	2.9.0
SSO	Authelia (OIDC) + LLDAP	—
Webhook bridge	matrix-hookshot	7.3.2
Reverse proxy	Nginx Proxy Manager	—
Web client	Lotus Cinny (fork of `cinnyapp/cinny` main)	custom
Element Call embed	`@element-hq/element-call-embedded`	0.19.3
GIF picker	Giphy JS/React SDK (`@giphy/react-components`)	—
Auto-deploy	adnanh/webhook	2.8.0
Bot language	Python 3	3.x
Bot library	matrix-nio (E2EE)	latest
Bot dependencies	matrix-nio[e2ee], aiohttp, python-dotenv, mcrcon	—

README.md Unescape Escape

Lotus Matrix Infrastructure

Repo Structure