- lk-jwt-service: `lk-jwt-service.service` (now binds `:8071` via drop-in `/etc/systemd/system/lk-jwt-service.service.d/override.conf`; serves JWT tokens for MatrixRTC at `/sfu/get` and legacy `/get_token`)
- voice-limit-guard: `voice-limit-guard.service` (binds `:8070`, fronts lk-jwt-service — enforces hard per-room voice participant limits for ALL clients; script `/opt/voice-limit-guard/voice-limit-guard.py`) — see [Voice Channel Limits](#voice-channel-limits)
Pushes to `main` on `LotusGuild/matrix` automatically deploy to the relevant LXC(s) via Gitea webhooks. All 4 LXCs are fully independent — each runs its own webhook listener and deploys only its own files. No cross-LXC SSH dependencies.
### How It Works
1. Push to `LotusGuild/matrix` on Gitea
2. Gitea fires webhooks to all 4 LXCs simultaneously (HMAC-SHA256 validated)
3. Each LXC runs `/usr/local/bin/matrix-deploy.sh` via the `webhook` binary
4. Script does `git fetch + reset --hard origin/main`, checks which files changed, deploys only relevant ones
5. Logs to `/var/log/matrix-deploy.log` on each LXC
### Per-LXC Webhook Endpoints
| LXC | Service | IP | Port | Deploys When Changed |
> LXC 151 uses port **9500** because ports 9000–9004 are occupied by Synapse and Hookshot.
### What Each Deploy Does
**LXC 151 — hookshot/livekit:**
-`hookshot/*.js` changed → runs `hookshot/deploy.sh` (pushes transform functions to Matrix room state via API, requires `MATRIX_TOKEN` in `/etc/matrix-deploy.env`)
-`systemd/livekit-server.service` changed → copies file, `daemon-reload`, sets `/run/livekit-restart-pending` flag (actual restart deferred — see Livekit Graceful Restart below)
**LXC 106 — cinny:**
-`cinny/config.json` → copies to `/var/www/html/config.json`
-`landing/index.html` → copies to `/var/www/matrix-landing/index.html`, `nginx -s reload`
**LXC 110 — draupnir:**
-`draupnir/production.yaml` → extracts live `accessToken` from existing config, overwrites from repo, restores token via `sed`, restarts `draupnir.service`
### Installed Components (per LXC)
-`webhook` binary (Debian package `webhook` v2.8.0) listening on respective port
-`/etc/webhook/hooks.json` — unique HMAC-SHA256 secret per LXC
-`/usr/local/bin/matrix-deploy.sh` — deploy script from this repo
-`/etc/systemd/system/webhook.service` — enabled and running
-`/opt/matrix-config/` — clone of this repo
-`/var/log/matrix-deploy.log` — deploy log
**LXC 151 additionally:**
-`/etc/matrix-deploy.env` — `MATRIX_TOKEN`, `MATRIX_SERVER`, `MATRIX_ROOM` (not in git)
Per-room voice participant caps are enforced **server-side for every client** (Element, FluffyChat, Lotus Chat, …), not just our own web client.
**How it works**
Every Matrix client must fetch a LiveKit JWT from lk-jwt-service before it can join a call. `voice-limit-guard` (a small fail-open Python sidecar, `livekit/voice-limit-guard.py` in this repo) sits in front of that service:
- lk-jwt-service was moved off `:8070` to `:8071` (systemd drop-in). The guard now owns `:8070`, so NPM's existing `/sfu/get` + `/get_token` proxy targets are unchanged.
- On each token request the guard reads `io.lotus.voice_limit` → `max_users` for the room (Synapse admin API, cached 10 s). `0` / absent = no limit.
- It forwards the request to lk-jwt-service, and if a token is issued it decodes the JWT to get the LiveKit alias (`video.room`) + requester identity (`sub`), then asks LiveKit `ListParticipants` how many **distinct Matrix users** are in the room.
- requester already present (rejoin / extra device) → allow
- distinct users ≥ limit → **403** (the client cannot get a token, so it cannot join)
- otherwise → allow
- **Fail-open:** any error (admin API down, bad token, LiveKit unreachable) returns the upstream response unchanged, so calls keep working even if enforcement is degraded.
**Setting a limit:** room admins set it from Lotus Chat → Room Settings → General → **Voice** (writes the `io.lotus.voice_limit` state event). Any tool that can send room state works too:
```bash
# max 5 participants in <roomId>; send {} to remove the limit
curl -X PUT -H "Authorization: Bearer <admin_token>" -H "Content-Type: application/json"\
**Config:** the guard reads `MATRIX_TOKEN` (server-admin) from `/etc/matrix-deploy.env`; LiveKit key/secret + ports are set in `systemd/voice-limit-guard.service`.
**Manual (re)deploy** (the file-specific auto-deploy pipeline does not cover this service):
The `MATRIX_TOKEN` in `/etc/matrix-deploy.env` on LXC 151 is a Jared user token used to push hookshot transforms to Matrix room state (requires power level ≥ 50 in Spam and Stuff).
The token in `draupnir/production.yaml` in this repo is **intentionally redacted** (`accessToken: REDACTED`). The deploy script on LXC 110 extracts the live token from the running config before overwriting from the repo, then restores it.
**To rotate the hookshot deploy token (LXC 151):**
1. Generate a new token via Synapse admin API or Cinny → Settings → Security → Manage Sessions
2. SSH to LXC 151 (via `ssh root@10.10.10.4` then `pct enter 151`): `nano /etc/matrix-deploy.env`
- NPM proxies `/sfu/get` and `/get_token` → `http://10.10.10.29:8070` (lk-jwt-service). Both paths are in `/data/nginx/proxy_host/49.conf` on LXC 139 — **NPM will overwrite these if proxy host 49 is re-saved via the UI; re-add both location blocks after any NPM save**
When a Matrix client user clicks "Report" on a message, Synapse receives a `POST /_matrix/client/v3/rooms/{roomId}/report/{eventId}` request and stores the report internally. To forward these to the Draupnir management room, a Synapse Python module must be installed on LXC 151.
**Draupnir web server** is enabled (port 8080). The endpoint is:
```
POST http://10.10.10.24:8080/_matrix/draupnir/1/report/{roomId}/{eventId}
```
**To complete Synapse integration (one-time, on LXC 151):**
> Until the Synapse module is installed, abuse reports are stored in Synapse's DB but do NOT appear in the management room. The Draupnir web server is running and ready to receive forwarded reports.
`chat.lotusguild.org` serves a custom Lotus Guild fork of the official `cinnyapp/cinny` main branch. The fork lives at `code.lotusguild.org/LotusGuild/cinny` and tracks upstream via a `git remote add upstream https://github.com/cinnyapp/cinny.git` workflow.
**Upstream monitoring (daily at noon):**
- `cinny-upstream-check.sh` hits the GitHub API and compares the latest `cinnyapp/cinny` main commit against the stored SHA in `/var/lib/cinny-monitor/last-upstream-commit`
- If new commits exist, sends a Matrix message to Spam and Stuff with an `@jared:matrix.lotusguild.org` ping and a link to the commit
- Does **not** auto-build — you review the diff and decide when to merge
**Cinny-build webhook token** (for LotusBot `!cinny-update`): stored in `deploy/hooks-lxc106.json` (`cinny-build` hook, header `X-Build-Token`). LotusBot must POST to `http://10.10.10.6:9000/hooks/cinny-build` with this header.
**Why 8GB RAM:** Vite's build process needs ~6GB Node heap (`--max_old_space_size=6144`) for the rendering-chunks phase. Previously at 4GB — OOM killed during render.
All custom code lives in `src/app/` on the `lotus` branch of `code.lotusguild.org/LotusGuild/cinny`. Changes survive upstream merges as long as they don't conflict with the same files upstream touched.
| **DM calls** | `src/app/features/room/Room.tsx`, `src/app/features/room/RoomViewHeader.tsx` | Phone button in DM room header; `useCallStart(true)` passes `intent: StartedByUser`; Room.tsx switches to CallView layout when DM has active call |
| **Picture-in-picture call** | `src/app/components/CallEmbedProvider.tsx` | When navigating away from the call room, the embed shrinks to a 280×158px PiP in the bottom-right. Click navigates back. Implemented via `useEffect` imperatively overriding styles on `callEmbedRef.current` — cannot use a wrapper div because `useCallEmbedPlacementSync` writes `top/left/width/height` directly onto that element |
| **Screenshare fullscreen** | `src/app/features/call/CallControls.tsx`, `src/app/features/call/Controls.tsx` | When screensharing, a fullscreen button appears in call controls. Calls `callEmbedRef.current?.requestFullscreen()` on the Cinny call container. EC naturally spotlights the screenshare — the old 600ms grid-revert code was removed (it caused fullscreen to show avatars instead of the screen) |
| **PiP screenshare focus** | `src/app/components/CallEmbedProvider.tsx`, `src/app/plugins/call/CallControl.ts` | When the floating PiP window is active and screenshare is detected (no cameras present), auto-enables EC spotlight view so the screenshare fills the PiP rather than showing avatar tiles |
| **Screenshare audio mute** | `src/app/features/call/Controls.tsx`, `src/app/features/call/CallControls.tsx`, `src/app/plugins/call/CallControl.ts` | Dedicated button to independently mute/unmute audio from screenshares without muting microphone audio. Targets `audio[data-lk-source="screen_share_audio"]` LiveKit elements. Persists across deafen/undeafen cycles |
| **Custom status message** | `src/app/features/settings/account/Profile.tsx`, `src/app/features/room/MembersDrawer.tsx`, `src/app/components/user-profile/UserHero.tsx`, `src/app/components/user-profile/UserRoomProfile.tsx`, `src/app/hooks/useUserPresence.ts` | Discord-style free-form status text. Set via Settings → Account → "Status Message" with an emoji picker (lazy-loaded `EmojiBoard`). Saved via `mx.setPresence({ status_msg })`. Displayed below the username in the members drawer and user profile popout. Syncs live via Matrix presence events |
| **PTT (Push-to-Talk)** | `src/app/features/call/CallControls.tsx`, `src/app/state/settings.ts` | Hold-to-talk key (default: Space, configurable). Mutes mic on join; holds mic open while key is held. Badge shows `PTT — Hold SPACE` / `● Live`. Listens on both main window and EC iframe `contentWindow` for key events |
| **PTT badge theming** | `src/app/features/call/CallControls.tsx` | Plain folds `Chip` by default; neon terminal style (`#00FF88`/`#FF6B00`, JetBrains Mono) when `lotusTerminal` setting is on |
| **GIF picker** | `src/app/components/GifPicker.tsx`, `src/app/features/room/RoomInput.tsx` | Giphy JS/React SDK (`@giphy/react-components`, `@giphy/js-fetch-api`, `styled-components`). API key in `config.json` → `gifApiKey`. GIF button appears next to Send only when `gifApiKey` is set. Sends GIF as `m.image` (fetches blob → `mx.uploadContent` → `mx.sendMessage`). `FocusTrap` handles click-outside / Escape to close |
| **GIF picker terminal theme** | `src/app/components/GifPicker.tsx` | When `lotusTerminal` is on: dark navy background (`#060c14`), orange dim border, 4px radius, `// GIF_SEARCH` header, injected `<style>` overrides Giphy SDK SearchBar input (dark bg, orange border/focus ring, JetBrains Mono), custom orange scrollbar |
| **Presence status badges** | `src/app/features/room/MembersDrawer.tsx`, `src/app/features/common-settings/members/Members.tsx`, `src/app/hooks/useUserPresence.ts`, `src/app/components/presence/` | Online/busy/away colored dot badges shown next to verification shields for every member in the room members drawer and settings members panel. Uses `useUserPresence(userId)` hook + `PresenceBadge` component. Members.tsx wraps the hook in a `MemberPresenceBadge` child component to satisfy React hook rules inside `.map()` |
| **Discord-style presence tracking** | `src/app/hooks/usePresenceUpdater.ts`, `src/app/pages/client/ClientNonUIFeatures.tsx` | Broadcasts `online` on startup, `unavailable` after 10 min idle or tab hidden, `offline` on page close (fetch+keepalive). Activity throttled to 1 event/sec. `hidePresence` setting broadcasts offline and disables all tracking |
| **Per-member device sessions panel** | `src/app/components/user-profile/UserRoomProfile.tsx`, `src/app/hooks/useOtherUserDevices.ts` | Collapsible "Sessions" card in user profile popout. Lists all devices with colored shield icons (green=verified, yellow=unverified, loading/error states). Per-device "Verify" button initiates cross-signing SAS emoji verification. Updates live via `CryptoEvent.DevicesUpdated`. Only shown when cross-signing is active |
| **Privacy settings** | `src/app/features/settings/general/General.tsx`, `src/app/state/settings.ts` | Dedicated Privacy section in General settings. `hideActivity` suppresses typing indicators and read receipts. `hidePresence` appears offline to everyone |
| **Encrypted room search** | `src/app/features/message-search/useLocalMessageSearch.ts`, `src/app/features/message-search/MessageSearch.tsx` | Searches locally cached decrypted events in E2EE rooms alongside server-side search. Per-room "Load more" buttons paginate 100 msgs at a time; shows oldest cached date and X/Y coverage counter. Sender-aware (respects `from:@user` filter) |
| **Message search: sender filter** | `src/app/features/message-search/SearchInput.tsx`, `src/app/features/message-search/SearchFilters.tsx` | Type `from:@user` in the search box for live autocomplete of known users (homeserver-biased ranking). Selected senders shown as removable chips. Works for both server search and local encrypted search |
| **Message search: date range** | `src/app/features/message-search/SearchFilters.tsx`, `src/app/features/message-search/useMessageSearch.ts` | From/To date pickers in the filter bar. Passed as `from_ts`/`to_ts` epoch ms to Matrix `/search` |
| **Document title unread count** | `src/app/pages/client/ClientNonUIFeatures.tsx` | Tab title updates to `(N) Lotus Chat` for mentions, `· Lotus Chat` for unreads, `Lotus Chat` when clear |
| **Message draft persistence** | `src/app/features/room/RoomInput.tsx` | Unsent messages survive page reload via `localStorage` (`draft-msg-<roomId>`). Jotai in-memory atom remains the primary store; localStorage used as fallback on reload. Cleared on send |
| **PiP position persistence + snap** | `src/app/components/CallEmbedProvider.tsx` | PiP position saved to `localStorage` on drag end; restored on next PiP enter (clamped to viewport). Double-click snaps to nearest corner with 180ms CSS transition |
LXC 151 can migrate between Proxmox nodes via HA. After migration, the old livekit-server process on the source node can leave a stale entry holding port 7881 on the destination. Fixed in `livekit-server.service` via:
`net.ipv4.tcp_congestion_control = bbr` must be set on the Proxmox host, not inside an unprivileged LXC. All other sysctl tuning (TCP/UDP buffers, fin_timeout) is applied inside LXC 151.
- [x] TCP retransmit timeout lowered (`tcp_retries2=5`, `tcp_syn_retries=4`, `tcp_keepalive_probes=3`) — stalled outbound federation connections now fail in ~15-30s instead of ~15 min
- [x] Unreachable routes added for servers with asymmetric connectivity (can reach us but we can't reach their federation port) — prevents 90s TCP hangs from being added to lag; defined in `/etc/network/interfaces` post-up hooks and survive reboots (bark.lgbt ×2, parodia.dev, chat.ohaa.xyz, matrix.k8ekat.dev)
- [x] Stuck `device_lists_remote_resync` entries cleared for dead-server users (@dalite:bark.lgbt, @arndot:matrix.goch.social) — device list resync was firing every 30s
- [x] Terms of Service / consent enforcement — `require_at_registration: false`, `block_events_error` set; new users cannot send messages until they explicitly accept via `/_matrix/consent`; Synapse sends a Server Notice DM with the consent URL on first blocked send
> **Disk I/O:** All servers use Ceph-backed storage. Per-device disk I/O metrics are meaningless — use Network I/O panels to see actual storage traffic.
> **`/sync` long-poll:** The Matrix `/sync` endpoint is a long-poll (clients hold it open ≤30s). It is excluded from the High Response Time alert to prevent false positives.
> **Synapse Event Processing Lag** alert fires when `synapse_event_processing_lag > 300s` for 15 consecutive minutes (threshold raised from 120s/5m to reduce noise from normal federation backoff cycling).
>
> Root cause: several federated servers (bark.lgbt, parodia.dev, etc.) have asymmetric connectivity — they can reach us but we cannot reach their federation ports. Each inbound transaction they send resets our backoff to 0, triggering a new outbound connection attempt that hangs for ~90s (TCP `User timeout`). This causes the lag metric to spike. Mitigations in place:
> 1. `tcp_retries2=5` in `/etc/sysctl.d/99-matrix-tuning.conf` — TCP hangs now fail in ~15-30s
> 2. `ip route add unreachable <ip>` in `/etc/network/interfaces` post-up — outbound connections to these servers fail in 0ms (ICMP unreachable)
> 3. Alert threshold raised to 300s/15m — only fires for genuine outages, not normal 10-min backoff cycles
>
> To find new offending servers: `grep "User timeout\|ConnectingCancell" /var/log/matrix-synapse/homeserver.log | grep -oP "\[([^\]]+)\]" | sort | uniq -c | sort -rn | head -20`
