feat(call): in-call soundboard, quality controls, room call-permissions
CI / Build & Quality Checks (push) Successful in 10m49s
CI / Trigger Desktop Build (push) Successful in 8s

Element Call is now consumed as our self-built fork
(@lotusguild/element-call-embedded); wire up its previously-dormant
capabilities and document the fork as live.

Soundboard (P5-15): a call-bar button plays user-uploaded audio clips into the
call as a real published track (io.lotus.inject_audio) plus local playback.
Clips are uploadable like emoji/sticker packs, stored in io.lotus.soundboard
account data (synced across devices). Gated by a Settings toggle + volume.

Quality controls (P5-31): per-user mic/screenshare bitrate + screenshare
framerate (Settings -> Calls), applied via io.lotus.set_quality clamped to any
room cap. Room admins set caps and hard call-permissions (allow_screenshare /
allow_camera) in Room Settings -> Voice; the call bar hides blocked buttons.

- New: CallSoundboard, useSoundboard, soundboardClips; RoomQuality,
  useCallQuality, callQuality (+ unit tests).
- Optimistic-write RoomQuality admin UI (no stale-state clobber).
- Docs: mark EC fork live across README/FEATURES/TODO/BUGS/TESTING; add D2
  manual-test steps.

Numeric quality caps are client-cooperative; screenshare/camera permissions are
hard-enforced server-side (see LotusGuild/matrix voice-limit-guard).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-30 22:34:17 -04:00
parent 02b2ce8109
commit 7c06b27c73
22 changed files with 1259 additions and 120 deletions
+22 -29
View File
@@ -39,38 +39,31 @@ Implemented and gate-green; confirm each per `LOTUS_TESTING.md`, then delete the
## 🧩 Element Call source-level items — now actionable via the fork ## 🧩 Element Call source-level items — now actionable via the fork
> 🔱 **[EC-FORK]** **UPDATE 2026-06-29: the fork is live.** We now own and > 🔱 **[EC-FORK]** **UPDATE 2026-06-30: Phase 2 IMPLEMENTED.** We own and
> self-build Element Call (`LotusGuild/element-call` → > self-build Element Call (`LotusGuild/element-call` →
> `@lotusguild/element-call-embedded`, Phase 1 done & cinny wired). A5/A6/A7 > `@lotusguild/element-call-embedded@0.20.1-lotus.1`, cinny wired). A5/A6/A7
> below are **no longer "won't fix"** — they are ordinary source changes. See > below are **fixed in the fork** — they are now ⚠️ awaiting **live
> [`HANDOFF_ELEMENT_CALL_FORK.md`](./HANDOFF_ELEMENT_CALL_FORK.md) §10 + the Phase > verification** (`LOTUS_TESTING.md` §D2), not open work. See
> 2 work list. (The iframe is **same-origin** / self-hosted; the old blocker was > [`HANDOFF_ELEMENT_CALL_FORK.md`](./HANDOFF_ELEMENT_CALL_FORK.md) §10. Delete each
> that we didn't own EC's compiled source — which we now do.) > row once verified live.
The in-call participant grid is rendered **inside EC's app**. Previously a The in-call participant grid is rendered **inside EC's app** — now editable source
pre-built npm bundle we could only style/place around; now editable source. (previously a prebuilt npm bundle we could only style around). Status of the items
Items from testing, with their fork-level fix path: from testing:
- **A5 — "Focus camera":** EC supports native tile-pinning. Our bottom-bar "Focus - **A5 — "Focus camera": ⚠️ FIXED in fork, awaiting verify (D2-3).** cinny now
camera" is a programmatic wrapper that **`.click()`s the tile** today sends an `io.lotus.focus_participant` widget action that pins a participant in
(`CallControl.ts` `focusCameraParticipant`), and during a screenshare EC EC's layout (coexisting with / overriding the screenshare spotlight); the old
spotlights the shared screen so a camera pin may not override it. **Fork fix:** `.click()`-the-tile DOM hack in `CallControl.ts` is deleted.
add an `io.lotus.focus_participant` widget action that pins a participant in - **A6 — avatar decorations in-call: ⚠️ FIXED in fork, awaiting verify (D2-4).**
EC's layout (coexisting with / overriding the screenshare spotlight); cinny cinny pushes `io.lotus.decorations` (per-user APNG URLs) and the fork renders
sends it via the widget API and the DOM-click hack is deleted. _Status: Open — them on EC's participant video-tile avatars — not just our pre-join lobby roster.
Actionable (Phase 2)._ - **A7 — mic dead after EC's "Reconnect": ⚠️ FIXED in fork, awaiting verify
- **A6 — avatar decorations in-call:** decorations render on **our** pre-join (D2-1).** Denoise moved into EC's mic-capture/publish pipeline as a first-class
lobby roster (`CallMemberCard`) but not on EC's in-call video tiles. **Fork LiveKit `TrackProcessor` (flag `lotusDenoiseSource=1`); EC re-runs it on every
fix:** render the decoration APNG inside EC's participant-tile component, fed (re)publish, so reconnects keep denoise alive natively. The build-time
decoration slugs via widget member data. _Status: Open — Actionable (Phase 2)._ `getUserMedia`/`index.html` injection (the root cause) is removed. **Highest
- **A7 — mic dead after EC's "Reconnect":** the mid-call "Connection lost / blast radius — everyone's mic; verify D2-1 carefully.**
Reconnect" screen is **EC's own** (our load watchdog only covers an initial
hung load). After EC reconnects, the mic isn't re-published through our denoise
`getUserMedia` shim until a clean End+rejoin. **Fork fix:** move denoise into
EC's mic-capture/publish pipeline as a first-class audio stage — EC re-runs it
on every (re)publish, so reconnects keep denoise alive natively, and the
build-time `index.html` injection is removed. _Status: Open — Actionable
(Phase 2); root cause is the `getUserMedia` monkeypatch, not EC itself._
--- ---
+111 -15
View File
@@ -322,14 +322,104 @@ Users can set a custom background color for `@mention` chips that highlight thei
## Voice / Video Call Improvements ## Voice / Video Call Improvements
> 🔱 **[EC-FORK]** Element Call is embedded as a **pre-built npm bundle** today. > 🔱 **[EC-FORK] LIVE (2026-06).** Element Call is now our **self-built fork**
> The plan to fork & self-build it from source for true ownership — and which of > (`@lotusguild/element-call-embedded@0.20.1-lotus.1`, source at
> the items below would move into our EC source — is in > `LotusGuild/element-call`), served same-origin — no longer the upstream
> pre-built npm bundle. Several in-call behaviors below are now first-class
> source changes rather than DOM/widget hacks. Background, plan, and the Phase-2
> work list are in
> [`HANDOFF_ELEMENT_CALL_FORK.md`](./HANDOFF_ELEMENT_CALL_FORK.md). > [`HANDOFF_ELEMENT_CALL_FORK.md`](./HANDOFF_ELEMENT_CALL_FORK.md).
### Element Call Upgrade ### Element Call — Self-Built Fork (`0.20.1-lotus.1`)
Upgraded embedded Element Call widget from **0.16.3** to **0.19.4**. The embedded widget was upgraded **0.16.3 → 0.19.4 → 0.20.1**, then **forked**.
We self-build `LotusGuild/element-call` and publish it to our private Gitea npm
registry as `@lotusguild/element-call-embedded`; cinny consumes that instead of
`@element-hq/element-call-embedded`. The iframe prints
`Element Call embedded-v0.20.1-lotus.1` in its console (vs. `embedded-v0.20.1`
upstream) — the quickest way to confirm a deploy landed the fork.
All custom behavior lives in the fork's `src/lotus/` modules and is **additive
and dormant by default**, gated by URL flags / widget actions the host opts into,
so a stock EC config is byte-for-byte upstream behavior.
**Active (cinny drives them today):**
| # | Feature | Mechanism | Replaces (old hack) |
| --- | --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| A7 | **Denoise in-source** | ML noise suppression runs inside EC as a LiveKit `TrackProcessor<Audio>` (flag `lotusDenoiseSource=1`); re-applied on every (re)publish | the build-time `getUserMedia` monkeypatch injected into `index.html`**removed**. Fixes mic-dead-after-reconnect. |
| #2 | **Speaking / mute events** | EC emits `io.lotus.call_state` (throttled); cinny reads speaker + mute state from it (flag `lotusCallState=1`) | scraping EC's DOM for `[data-lk-speaking]` (kept only as fallback) |
| A5 | **Focus participant** | host sends `io.lotus.focus_participant` to pin a tile, coexisting with / overriding the screenshare spotlight | the `.click()`-the-tile DOM hack in `CallControl.ts`**removed** |
| #6 | **In-call avatar decorations** | host pushes `io.lotus.decorations` (per-user APNG URLs); the fork renders them on EC's video-tile avatars | previously impossible — decorations only showed on our pre-join lobby roster |
| #5 | **Native transparent background** | flag `lotusTransparent=1` makes EC's surface transparent so the host wallpaper shows through | the injected `background:none !important` CSS |
**Now wired (cinny drives them — ⚠️ awaiting live verification):**
| # | Capability | Widget action | cinny surface |
| ----- | -------------------- | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------- |
| P5-15 | **Audio inject** | `io.lotus.inject_audio` — plays a clip into the call as a separately published track | In-Call Soundboard (uploadable clips) — see below |
| P5-31 | **Quality controls** | `io.lotus.set_quality` — sets audio/screenshare encoding bitrate/framerate | Call Quality Controls (user settings + room-admin caps) — see below |
> Both were dormant capabilities; cinny now drives them (armed via
> `lotusAudioInject=1`). The **only** EC item still open is the P5-31
> **server-side** quality guard (a `voice-limit-guard`-style sidecar reading
> `io.lotus.room_quality`) for hard enforcement across all Matrix clients — the
> client cap is best-effort.
### In-Call Soundboard (P5-15)
A soundboard button (🔔) in the call controls bar opens a popout of the user's
clips. Clicking one **injects it into the call as a real published LiveKit
track** (every participant hears it, via the fork's `io.lotus.inject_audio`) and
plays it locally for the presser (LiveKit doesn't loop your own track back).
- **User-uploadable, like custom emoji/sticker packs.** Clips are stored in the
`io.lotus.soundboard` account data event, so they **sync across all your
devices**. Upload short audio (≤ 1 MB, ≤ 40 clips) from the popout; delete
inline.
- Authenticated media can't be fetched from the widget's realm, so the host
resolves each mxc clip → an authenticated download → a same-session `blob:`
object URL and hands that to the widget.
- Gated by the **Soundboard** toggle (Settings → General → Calls) with a volume
slider. The button is hidden when disabled.
- Files: `utils/soundboardClips.ts`, `hooks/useSoundboard.ts`,
`features/call/CallSoundboard.tsx`, `plugins/call/CallControl.ts#injectAudio`.
### Call Quality Controls (P5-31)
Discord-style encoding controls applied to the local tracks via the fork's
`io.lotus.set_quality` (`RTCRtpSender.setParameters` across all simulcast
encodings, re-applied on every re-publish/reconnect).
- **User settings** (Settings → General → Calls): Microphone Bitrate,
Screenshare Bitrate, Screenshare Framerate (each defaults to **Auto**).
- **Room-admin caps**: admins set a ceiling in Room Settings → General → Voice
(`io.lotus.room_quality` state event); every Lotus client clamps its per-user
quality to `min(user setting, room cap)`.
- Applied by the `useCallQuality` hook on join and whenever settings/caps
change; `utils/callQuality.ts` builds the payload (unit-tested).
**Server-enforced call permissions (hard, ALL clients).** The same
`io.lotus.room_quality` event carries a **publish-source policy**
(`allow_screenshare`, `allow_camera`) enforced server-side by
`voice-limit-guard` (matrix repo, LXC 151): it re-signs the LiveKit JWT's
`canPublishSources`, so the SFU refuses screenshare/camera tracks for **every**
Matrix client (Element, FluffyChat, our fork) — not just Lotus. Admins toggle
these in Room Settings → Voice → **Call Permissions**; cinny also hides the
blocked buttons in the call bar. Enforcement is **live**: the JWT re-sign covers
new joins, and a background reconcile loop revokes an **in-progress**
screenshare/camera (via LiveKit `UpdateParticipant`) within ~3 s of an admin
flipping the policy — so it kills active shares mid-call, not just future ones.
- **Why numeric caps aren't server-enforced:** LiveKit is a pure SFU (forwards,
never transcodes) and has no publisher bitrate/fps field anywhere in the JWT
grant, room config, server `limit:`, or admin API; stock Element Call ignores
room metadata for publish quality. Numeric caps are therefore inherently
**cooperative** — our fork honors them, which is the design above. The
publish-source policy is the one genuine hard, cross-client lever, and it's
implemented.
- **Not yet**: screenshare resolution control (needs a `getDisplayMedia` hook in
the fork).
### Camera Default Off ### Camera Default Off
@@ -431,20 +521,26 @@ A comprehensive mic noise-suppression system in **Settings → General → Calls
- **Support Detection:** UI now detects `AudioWorklet` / `AudioContext` support and disables ML options in unsupported environments. - **Support Detection:** UI now detects `AudioWorklet` / `AudioContext` support and disables ML options in unsupported environments.
- **Status Reporting:** The ML shim notifies the host app via `postMessage`. If initialization fails, a system toast alerts the user of the fallback to the raw microphone. - **Status Reporting:** The ML shim notifies the host app via `postMessage`. If initialization fails, a system toast alerts the user of the fallback to the raw microphone.
**Open-Source Model Roadmap:** **Open-Source Models (all now in-source in the EC fork):**
| Model | Transients (Clicks) | Voice Quality | CPU Usage (WASM) | | Model | Transients (Clicks) | Voice Quality | CPU Usage (WASM) | Sample rate |
| :--- | :--- | :--- | :--- | | :--- | :--- | :--- | :--- | :--- |
| **RNNoise** | Poor | Moderate | < 5% | | **RNNoise** (default) | Poor | Moderate | < 5% | 48 kHz |
| **DTLN** | Good | High | 10-20% | | **Speex** | Poor | Low | < 5% | 48 kHz |
| **DeepFilterNet 3** | **Excellent** | **Very High** | 25-50%+ | | **DTLN** | Good | High | 10-20% | 16 kHz |
| **DeepFilterNet 3** | **Excellent** | **Very High** | 25-50%+ | 48 kHz |
> **Note:** DeepFilterNet 3 is planned for future inclusion in the desktop build where larger binaries and higher CPU overhead are more acceptable. > **Update (2026-06):** with the EC fork live, denoise runs **inside** Element
> Call as a LiveKit `TrackProcessor` and **all four models ship in-source**
> (DTLN at 16 kHz, the rest at 48 kHz; the processor degrades to the raw mic
> rather than ever going silent). The model picker selects between them. Real-call
> **audio-quality** comparison across models is still the open verification item
> (RNNoise output is known to be weak) — see `LOTUS_TESTING.md` §D2-1.
### Files ### Files
- `build/lotus-denoise.js` — multi-model getUserMedia shim - **EC fork** `src/lotus/lotusDenoise.ts` + `lotusDenoiseProcessor.ts` — in-source LiveKit `TrackProcessor` (RNNoise/Speex 48 kHz, DTLN 16 kHz, DeepFilterNet 48 kHz); activated by `lotusDenoiseSource=1`. (The old build-time `getUserMedia` shim `build/lotus-denoise.js` is **removed**.)
- `vite.config.js``lotusDenoise()` plugin (copies assets for RNNoise, Speex, and NoiseGate) - `vite.config.js``lotusDenoise()` plugin (now only **copies model assets** for the fork to load; no longer injects a shim)
- `src/app/plugins/call/CallEmbed.ts` — advanced tier → widget URL params - `src/app/plugins/call/CallEmbed.ts` — advanced tier → `lotusDenoiseSource` widget URL param
- `src/app/utils/lotusDenoiseUtils.ts` — support detection and model comparison metadata - `src/app/utils/lotusDenoiseUtils.ts` — support detection and model comparison metadata
- `src/app/features/settings/general/General.tsx` — advanced settings UI + mic meter - `src/app/features/settings/general/General.tsx` — advanced settings UI + mic meter
+51 -4
View File
@@ -267,12 +267,59 @@ Flag: `lotusTransparent=1` (native, replacing the injected `background:none !imp
- [ ] Call background looks right — host wallpaper/surface shows through; **no** black box, bad - [ ] Call background looks right — host wallpaper/surface shows through; **no** black box, bad
see-through, or layout breakage (also covered loosely by §D2 "looks right"). see-through, or layout breakage (also covered loosely by §D2 "looks right").
### D2-6. Dormant features — confirm they do NOTHING (no regression) ### D2-7. In-Call Soundboard (#3 / P5-15) — 👥 2 people — **NEW**
EC ships the capability but cinny has **no UI** to trigger them yet: Flag: `lotusAudioInject=1`. A 🔔 **Soundboard** button now sits in the call controls bar (left group,
next to the chat button). Clips are user-uploadable and sync across your devices like emoji packs.
_Prereq:_ Settings → General → Calls → **Soundboard** must be ON (default on).
- [ ] **Soundboard audio-inject (#3)** and **quality controls (#7)** — there should be no new UI and no - [ ] **Upload:** open the soundboard popout → **Upload** → pick a short audio file (mp3/ogg/wav, ≤ 1 MB).
effect. (Nothing to test; noted so a tester doesn't go hunting.) It appears as a clip tile. (Too-big / too-many shows an error, doesn't crash.)
- [ ] **Plays into the call:** with a second person in the call, click a clip. **They hear it**, and
**you hear it locally** too. ✅ good if both hear it; ❌ tell us if only one side does.
- [ ] **Sync:** the uploaded clip shows up on your **other device**/session (account-data sync).
- [ ] **Delete:** the ✕ on a tile removes it (everywhere, after sync).
- [ ] **Off switch:** turn Settings → Calls → **Soundboard** off → the call-bar button disappears.
- [ ] Injecting a clip does **not** mute/interrupt your mic or anyone else's audio.
### D2-8. Call Quality Controls (#7 / P5-31) — 👥 2 people — **NEW**
Action: `io.lotus.set_quality`. User settings in **Settings → General → Calls** (Microphone Bitrate,
Screenshare Bitrate, Screenshare Framerate; all default **Auto**). Admin caps in **Room Settings →
General → Voice → Call Quality Caps**.
- [ ] **No regression at Auto:** with everything on **Auto**, calls/screenshare work exactly as before.
- [ ] **User cap takes effect:** set Microphone Bitrate to **32 kbps**, rejoin/continue a call — audio
still flows (thinner is fine). Set Screenshare Framerate to **15 fps** and share your screen — it
still shares. ❌ tell us if any setting kills audio/screenshare.
- [ ] **Applies mid-call:** changing a setting **during** a call takes effect without End+rejoin.
- [ ] **Room-admin cap (admin needed):** as a room admin, set **Max Microphone Bitrate = 64 kbps** in
Room Settings → Voice. A member whose user setting is higher (e.g. 256) should be **clamped to 64**
(best-effort/UX — this is client-side; hard server enforcement is a separate follow-up).
- [ ] Resetting a setting back to **Auto** removes the cap for the rest of the call.
> Soundboard + quality are no longer "dormant" — if either does nothing, grab the **EC iframe console**
> and check for `io.lotus.inject_audio` / `io.lotus.set_quality` rejections.
### D2-9. Call Permissions — HARD server-side, cross-client (👥 2 people, admin) — **NEW**
This is enforced by the `voice-limit-guard` on the server (re-signs the LiveKit JWT), so it applies to
**every** client, not just Lotus Chat. Set in **Room Settings → General → Voice → Call Permissions**.
_(Requires the guard deployed on LXC 151 — auto-deploys on a `matrix` repo push.)_
- [ ] **Disable screenshare:** as admin, turn **Allow Screen Sharing** off. In a call, the
**screenshare button disappears** in Lotus Chat. ✅ good if no one can screenshare.
- [ ] **Cross-client (the important one):** have someone join the **same room from stock Element / Element
X** and try to screenshare → the server **refuses** the track (it won't publish). This proves it's
not just our client hiding a button.
- [ ] **Audio-only room:** turn **Allow Camera** off too → the camera button disappears and cameras are
server-blocked for all clients; **microphones still work**.
- [ ] **⭐ Live kill (mid-call):** while someone is **actively screensharing**, an admin turns **Allow
Screen Sharing** off. Within a few seconds their screenshare should **stop for everyone** on its own
(no rejoin needed) — this is the server reconcile loop revoking it live. Works even if the sharer is
on stock Element. ✅ good if the share drops within ~35 s; ❌ tell us if it keeps going.
- [ ] **Turning it back on** restores the ability to screenshare/camera (start a new share).
- [ ] **No policy = no change:** a room with Call Permissions left on defaults behaves exactly as before.
> If any D2 item fails, grab the **EC iframe console** (right-click the call → inspect the iframe) — a > If any D2 item fails, grab the **EC iframe console** (right-click the call → inspect the iframe) — a
> widget-action/payload mismatch shows up there as a `io.lotus.*` rejection or a `MissingKey`/transport log. > widget-action/payload mismatch shows up there as a `io.lotus.*` rejection or a `MissingKey`/transport log.
+65 -45
View File
@@ -48,6 +48,9 @@ Built and gate-green; verify per [LOTUS_TESTING.md](./LOTUS_TESTING.md), then th
| Desktop — proactive update notifications (Tauri) | J1 | | Desktop — proactive update notifications (Tauri) | J1 |
| Remind Me Later | K1 | | Remind Me Later | K1 |
| Mobile Bookmarks access | E5 | | Mobile Bookmarks access | E5 |
| In-Call Soundboard (P5-15, uploadable clips → real call inject) | D2-7 |
| Call Quality Controls (P5-31, user + room-admin caps) | D2-8 |
| Call Permissions (P5-31, hard server-side screenshare/camera policy) | D2-9 |
--- ---
@@ -72,32 +75,32 @@ Status: `[ ]` pending · `[~]` in progress · `[x]` completed
### Confirmed facts ### Confirmed facts
| Finding | Impact | | Finding | Impact |
| --------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | | -------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| **MSC flags ON:** `msc4140` · `msc3771` · `msc3440.stable` · `msc4133.stable` · `simplified_msc3575` · `msc4222` · `msc3266` · `msc3401_matrix_rtc` | All safe to use now | | **MSC flags ON:** `msc4140` · `msc3771` · `msc3440.stable` · `msc4133.stable` · `simplified_msc3575` · `msc4222` · `msc3266` · `msc3401_matrix_rtc` | All safe to use now |
| **MSC flags OFF:** `msc4306` (thread subscriptions) · `msc3882` · `msc3912` · `msc4155` | These features are BLOCKED | | **MSC flags OFF:** `msc4306` (thread subscriptions) · `msc3882` · `msc3912` · `msc4155` | These features are BLOCKED |
| **MSC3266** room summary: flag `msc3266_enabled: true` set but `GET /v1/rooms/{id}/summary` still returns 404 (M_UNRECOGNIZED) | Room Preview BLOCKED — endpoint not implemented in Synapse 1.155 | | **MSC3266** room summary: flag `msc3266_enabled: true` set but `GET /v1/rooms/{id}/summary` still returns 404 (M_UNRECOGNIZED) | Room Preview BLOCKED — endpoint not implemented in Synapse 1.155 |
| **MSC3892** relation redaction: not in flags | Reaction Redaction feature BLOCKED | | **MSC3892** relation redaction: not in flags | Reaction Redaction feature BLOCKED |
| **MSC4260** report user: `POST /_matrix/client/v3/users/{userId}/report` returns **200** ✅ | **Report User UNBLOCKED** — endpoint live since Synapse 1.133; ready to build | | **MSC4260** report user: `POST /_matrix/client/v3/users/{userId}/report` returns **200** | **Report User UNBLOCKED** — endpoint live since Synapse 1.133; ready to build |
| **MSC4151** report room: HTTP 405 on GET = endpoint exists (POST only) | Report Room live ✅ | | **MSC4151** report room: HTTP 405 on GET = endpoint exists (POST only) | Report Room live ✅ |
| `folds AvatarImage` does NOT accept children | Add frame/overlay inside `UserAvatar.tsx` itself — optional `frameName` prop | | `folds AvatarImage` does NOT accept children | Add frame/overlay inside `UserAvatar.tsx` itself — optional `frameName` prop |
| No in-app toast system exists (was) | Built `ToastProvider` + Jotai queue; at `App.tsx:65` | | No in-app toast system exists (was) | Built `ToastProvider` + Jotai queue; at `App.tsx:65` |
| `useUnverifiedDeviceCount()` hook exists | `src/app/hooks/useDeviceVerificationStatus.ts:65-106` | | `useUnverifiedDeviceCount()` hook exists | `src/app/hooks/useDeviceVerificationStatus.ts:65-106` |
| Voice player: `AudioContent.tsx:44-223` | Playback rate on hidden `<audio>` at line 217 | | Voice player: `AudioContent.tsx:44-223` | Playback rate on hidden `<audio>` at line 217 |
| `CallControl.setMicrophone(bool)` at `CallControl.ts:206-212` | For AFK auto-mute | | `CallControl.setMicrophone(bool)` at `CallControl.ts:206-212` | For AFK auto-mute |
| `CallControl.toggleSound()` at `CallControl.ts:230-251` | Push-to-deafen — just wire a hotkey to this | | `CallControl.toggleSound()` at `CallControl.ts:230-251` | Push-to-deafen — just wire a hotkey to this |
| matrix-js-sdk has NO arbitrary profile field methods | Use `mx.http.authedRequest()` for MSC4133 | | matrix-js-sdk has NO arbitrary profile field methods | Use `mx.http.authedRequest()` for MSC4133 |
| Sanitizer (`sanitize.ts`) allows table, div, span, a, code, hr | LFG HTML card is safe locally; test on Element/FluffyChat | | Sanitizer (`sanitize.ts`) allows table, div, span, a, code, hr | LFG HTML card is safe locally; test on Element/FluffyChat |
| Sanitizer STRIPS `<math>`/MathML tags | Math/LaTeX task must also modify sanitizer | | Sanitizer STRIPS `<math>`/MathML tags | Math/LaTeX task must also modify sanitizer |
| Service worker EXISTS at `src/sw.ts` | Quick-reply task: add `notificationclick` handler | | Service worker EXISTS at `src/sw.ts` | Quick-reply task: add `notificationclick` handler |
| `knockSupported()` utility exists at `matrix.ts:376-391` | Knock UX: only need "Request to Join" in `RoomIntro.tsx` | | `knockSupported()` utility exists at `matrix.ts:376-391` | Knock UX: only need "Request to Join" in `RoomIntro.tsx` |
| `KeywordMessages.tsx` already has custom keyword push rules | Full push rule editor: only non-keyword rule types need new UI | | `KeywordMessages.tsx` already has custom keyword push rules | Full push rule editor: only non-keyword rule types need new UI |
| `getMatrixToRoom()` in `matrix-to.ts` generates invite URLs | Invite link: just add QR code to room settings | | `getMatrixToRoom()` in `matrix-to.ts` generates invite URLs | Invite link: just add QR code to room settings |
| Cindy CANNOT inject audio into EC call stream | In-call soundboard must be redesigned as local-only | | ~~Cindy CANNOT inject audio into EC call stream~~ **UNBLOCKED by EC fork**`io.lotus.inject_audio` widget action publishes a clip as a real call track | In-call soundboard CAN now mix into the call (no longer local-only); needs cinny UI to drive the action |
| Folds uses vanilla-extract in non-TDS, NOT CSS custom properties | Custom accent color: must create new vanilla-extract theme variant dynamically | | Folds uses vanilla-extract in non-TDS, NOT CSS custom properties | Custom accent color: must create new vanilla-extract theme variant dynamically |
| Theme presets need ~50 CSS custom properties each | Significant design work before coding | | Theme presets need ~50 CSS custom properties each | Significant design work before coding |
| `useCallSpeakers.ts` CSS MutationObserver polling | Visual speaking indicator: TDS ring animation on top of existing data | | `useCallSpeakers.ts` CSS MutationObserver polling | Visual speaking indicator: TDS ring animation on top of existing data |
| MSC3489/3672 live location: BOTH false on server | Live Location BLOCKED | | MSC3489/3672 live location: BOTH false on server | Live Location BLOCKED |
--- ---
@@ -266,12 +269,17 @@ Features:
--- ---
### [ ] P5-15 · In-Call Soundboard ### [~] P5-15 · In-Call Soundboard — IMPLEMENTED (⚠️ awaiting live verification, D2-7)
**What:** Grid of short audio clips playable into the call audio stream via Web Audio API (AudioBufferSourceNode → MediaStreamDestinationNode → mixed with mic). Built-in clips + user-uploadable custom clips (stored as mxc://). Accessible from call controls bar. **What:** Soundboard button in the call controls bar → popout grid of the user's clips; clicking one plays it **into the call** as a real published track (peers hear it) and locally (presser hears it). Clips are **user-uploadable, just like custom emojis/stickers**.
**[AUDIT REQUIRED]** Verify the Element Call integration exposes the mic MediaStream for mixing. This is the highest-risk part of this feature. **🔱 [EC-FORK] Fork side + cinny side DONE.** The fork ships `io.lotus.inject_audio` (`LotusWidgetActions.InjectAudio`, allow-listed in `widget.ts`), armed via the `lotusAudioInject=1` flag; it publishes a clip as a separate LiveKit track — a **real** in-call soundboard mixed into the call, not local-only. cinny now drives it.
**🔱 [EC-FORK]** Owning the EC source (see [`HANDOFF_ELEMENT_CALL_FORK.md`](./HANDOFF_ELEMENT_CALL_FORK.md)) would unblock real audio-injection — a proper soundboard mixed into the call — which is impossible against the prebuilt bundle today. **Shipped (cinny):**
**Complexity:** High.
- Clips stored in `io.lotus.soundboard` account data → **synced across devices like emoji/sticker packs** (`useSoundboard` hook; `AccountDataEvent.LotusSoundboard`).
- Upload audio (≤1 MB, ≤40 clips) → `mx.uploadContent` → mxc; play resolves mxc → authed download → `blob:` object URL (the widget can't fetch authenticated media itself) → `control.injectAudio(url, volume)` + local playback.
- `CallSoundboard.tsx` popout in the call bar (upload / play / delete), gated on the `soundboardEnabled` setting (Settings → General → Calls, + volume slider).
**Remaining:** a dedicated Settings management page (optional — upload/delete already live in the popout); a small default clip set; live verification (D2-7). Files: `utils/soundboardClips.ts`, `hooks/useSoundboard.ts`, `features/call/CallSoundboard.tsx`, `plugins/call/CallControl.ts#injectAudio`.
**Complexity:** Medium — done.
--- ---
@@ -287,26 +295,38 @@ Features:
### [x] P5-30 · Advanced ML Noise Suppression (Krisp-style) ### [x] P5-30 · Advanced ML Noise Suppression (Krisp-style)
**What:** High-end background noise cancellation using a pre-trained ML model (RNNoise) running in the browser. Removes dogs, fans, and keyboard clicks from the mic stream. **What:** High-end background noise cancellation using a pre-trained ML model (RNNoise) running in the browser. Removes dogs, fans, and keyboard clicks from the mic stream.
**Shipped:** 3-tier setting (Off / Browser-native / ML) in Settings → General → Calls. ML tier injects a same-origin pre-init shim into the vendored Element Call `index.html` that monkeypatches `getUserMedia` and routes the captured mic through an RNNoise `AudioWorklet` before LiveKit publishes — no EC fork required. See LOTUS_FEATURES.md → "Noise Suppression (Advanced Multi-Tier)". **Shipped:** 3-tier setting (Off / Browser-native / ML) in Settings → General → Calls.
**🔱 [EC-FORK]** Once we own the EC source (see [`HANDOFF_ELEMENT_CALL_FORK.md`](./HANDOFF_ELEMENT_CALL_FORK.md)), denoise should become a first-class audio stage **inside** EC instead of an `index.html` getUserMedia monkeypatch — more robust, survives reconnects (fixes the A7 mic-after-reconnect bug), and removes the build-time injection hack. **🔱 [EC-FORK] DONE — moved in-source (2026-06).** ML denoise is now a first-class audio stage **inside** the forked Element Call: a LiveKit `TrackProcessor<Audio>` activated by `lotusDenoiseSource=1` (cinny sets it when ML is selected). The old build-time `getUserMedia`/`index.html` monkeypatch is **removed**. Because EC re-runs the processor on every (re)publish, denoise now **survives reconnects and mic-device switches** — this is the A7 fix (see `LOTUS_BUGS.md` A7, `LOTUS_TESTING.md` §D2-1). The processor degrades to the raw mic rather than going silent.
**Key decision:** LiveKit's Krisp filter is LiveKit-Cloud-only (we self-host the SFU); EC's own RNNoise PR #3892 is unmerged. The shim is the same post-capture pipeline #3892 uses, executed from the realm we control, so it survives EC version bumps. **Key decision:** LiveKit's Krisp filter is LiveKit-Cloud-only (we self-host the SFU); EC's own RNNoise PR #3892 is unmerged. Owning the fork let us implement the in-source stage directly.
**AEC note (resolved-as-accepted):** WebAudio capture routing can weaken browser AEC — same tradeoff as EC's upstream feature; mitigated by keeping `echoCancellation`/`autoGainControl` on the raw capture and labeling the tier "beta".
**Model Roadmap (priority order):** **Models — all in-source in the fork:**
- [ ] **Verify DTLN** (16 kHz narrowband fix) in a real call before investing further — wired but unverified. - [x] **RNNoise** (48 kHz, default) · **Speex** (48 kHz) · **DTLN** (16 kHz) · **DeepFilterNet 3** (48 kHz) — all four wired and selectable.
- [ ] **DeepFilterNet 3** — best self-hostable upgrade: Rust→WASM, CPU real-time, 48 kHz fullband. Effort: self-host `df_bg.wasm` + DFN3 ONNX model, wire a 48 kHz worklet. - [ ] **Open verification:** real-call **audio-quality** comparison across the four models (RNNoise output is known-weak). Track under the denoise quality project, `LOTUS_TESTING.md` §D2-1 / J2.
- [ ] **Desktop-only / HW-gated:** FRCRN or NVIDIA Maxine (RTX/Tensor only) — impossible in-browser; would run in Tauri Rust backend + bridge a virtual mic into the webview. Must detect capability and only offer on supported hardware; web falls back to RNNoise. - [ ] **Desktop-only / HW-gated (future):** FRCRN or NVIDIA Maxine (RTX/Tensor only) — impossible in-browser; would run in the Tauri Rust backend + bridge a virtual mic into the webview. Detect capability; web falls back to RNNoise.
- **Excluded:** Krisp (LiveKit Cloud only); FRCRN/Maxine on web (GPU/server-bound). - **Excluded:** Krisp (LiveKit Cloud only); FRCRN/Maxine on web (GPU/server-bound).
--- ---
### [ ] P5-31 · Granular Voice & Screenshare Quality Controls (Discord-style) ### [~] P5-31 · Granular Voice & Screenshare Quality Controls — IMPLEMENTED (⚠️ awaiting live verification, D2-8)
**What:** Let users (or room admins via room settings) adjust audio bitrates (e.g., 64kbps to 512kbps) and screenshare quality (resolution: 720p/1080p/Source, framerate: 15/30/60fps). **What:** Let users (and room admins) adjust audio bitrate and screenshare bitrate/framerate.
**Note:** Requires tight integration with the LiveKit SFU and custom state events for per-room quality caps. **🔱 [EC-FORK] Fork side + client side DONE.** The fork ships `io.lotus.set_quality` (`LotusWidgetActions.SetQuality`) that applies audio/screenshare encoding params (`RTCRtpSender.setParameters`, all simulcast encodings, re-applied on `TrackUnmuted`/republish) inside EC. cinny now drives it.
**[AUDIT REQUIRED]** Must verify if current `lk-jwt-service` can be extended with custom bitrate/resolution claims or if a new sidecar (similar to `voice-limit-guard`) is needed for server-side enforcement.
**Complexity:** Extreme. **Shipped (cinny):**
1. **User settings** (Settings → General → Calls): Microphone Bitrate, Screenshare Bitrate, Screenshare Framerate (`callAudioBitrate` / `screenshareBitrate` / `screenshareFramerate`).
2. **Room-admin caps**: `io.lotus.room_quality` state event (`StateEvent.LotusRoomQuality`) + `RoomQuality.tsx` in Room Settings → General → Voice (mirrors `RoomVoiceLimit`).
3. **Apply logic**: `useCallQuality` (wired in `CallEmbedProvider`'s `CallUtils`) builds `min(user setting, room cap)` and sends `io.lotus.set_quality` on join / when settings change (`utils/callQuality.ts`, unit-tested).
**Server-side enforcement (DONE — matrix repo):** extended `voice-limit-guard.py` (LXC 151) to also read `io.lotus.room_quality` and hard-enforce a **publish-source policy** for ALL clients.
- **Reality (researched, primary-source, LiveKit 1.9.11):** numeric bitrate/fps caps **cannot** be hard-enforced server-side — LiveKit is a pure SFU (forwards, never transcodes); there is NO bitrate/fps field in the JWT grant, `RoomConfiguration`, server `limit:` config, or any admin RPC, and stock Element Call ignores room metadata / custom claims for publish quality. So numeric caps stay **cooperative** (our fork honors them via `min()``set_quality`, already shipped).
- **What IS hard-enforced cross-client:** `VideoGrant.canPublishSources`. The guard holds the LiveKit secret, so when `io.lotus.room_quality` sets `allow_screenshare:false` / `allow_camera:false` it re-signs the issued JWT with a narrowed source list → the SFU refuses those tracks for **every** client (Element, FluffyChat, our fork). Mic always kept. Fail-open; unit-tested (`livekit/test_voice_limit_guard.py`). Admin UI: Room Settings → Voice → **Call Permissions** switches. cinny also hides the blocked buttons.
- **Live (mid-call) enforcement — DONE:** the JWT re-sign covers new joins; for participants **already in the call**, a background reconcile loop in the guard calls LiveKit `UpdateParticipant` every ~3 s to narrow `canPublishSources`, which unpublishes an in-progress screenshare/camera **server-side for all clients** and blocks re-publish (verified LiveKit 1.9.11 auto-unpublishes on permission narrowing). Only removes forbidden sources (never grants), preserves other permission flags, no-ops once compliant. So flipping a room audio-only kills live cameras/screenshares within ~one interval.
- **Not enforceable / deferred:** numeric server enforcement (impossible — see above); screenshare **resolution** control (`set_quality` covers bitrate + framerate; resolution needs a `getDisplayMedia` hook inside the fork).
**Complexity:** DONE — client (cooperative numeric caps) + server (hard publish-source policy). Only the physically-impossible numeric server enforcement is out of scope.
--- ---
@@ -540,7 +560,7 @@ Exhaustive, low-level implementation details for backlog items. Follow these pat
> ⚠️ **[Gemini_Found — CORRECTED]** Gemini originally suggested using LiveKit's `LocalAudioTrack.replaceTrack()` to mix audio into the call stream. This is **not possible** from Lotus Chat's realm: Element Call runs in a **cross-origin iframe** controlled via `matrix-widget-api` (postMessage). LiveKit's JS SDK and its `LocalAudioTrack` live inside EC's sandboxed context — inaccessible from our code. This directly contradicts the confirmed constraint already listed in the Server Capabilities table: _"Cindy CANNOT inject audio into EC call stream — In-call soundboard must be redesigned as local-only."_ The soundboard must be a local-playback-only feature (output through the user's speakers, not mixed into the call audio stream). > ⚠️ **[Gemini_Found — CORRECTED]** Gemini originally suggested using LiveKit's `LocalAudioTrack.replaceTrack()` to mix audio into the call stream. This is **not possible** from Lotus Chat's realm: Element Call runs in a **cross-origin iframe** controlled via `matrix-widget-api` (postMessage). LiveKit's JS SDK and its `LocalAudioTrack` live inside EC's sandboxed context — inaccessible from our code. This directly contradicts the confirmed constraint already listed in the Server Capabilities table: _"Cindy CANNOT inject audio into EC call stream — In-call soundboard must be redesigned as local-only."_ The soundboard must be a local-playback-only feature (output through the user's speakers, not mixed into the call audio stream).
> >
> 🔱 **[EC-FORK — partial correction]** The "cross-origin" claim above is **outdated**: EC is now **same-origin** / self-hosted (`iframe.sandbox` has `allow-same-origin`; we read `contentDocument`). The _practical_ blocker still holds — LiveKit's `LocalAudioTrack` lives in EC's **module scope** (not on `window`), so it's unreachable from cinny even same-origin. **Owning the EC source** (see [`HANDOFF_ELEMENT_CALL_FORK.md`](./HANDOFF_ELEMENT_CALL_FORK.md)) is the path to a real call-audio-inject API, which would unblock a true in-call soundboard. > 🔱 **[EC-FORK — RESOLVED]** Both the original claim and the earlier "practical blocker still holds" correction are now **outdated**. EC is same-origin **and** we own the source, so we no longer reach into EC's module scope from cinny — instead the fork **exposes the inject point itself**: the `io.lotus.inject_audio` widget action (`LotusWidgetActions.InjectAudio`) publishes a clip as a separate LiveKit track from inside EC. A **real** in-call soundboard (mixed into the call, not local-only) is therefore unblocked; only the cinny-side UI remains (see P5-15 above). The capability ships dormant today.
--- ---
+17 -14
View File
@@ -144,22 +144,25 @@ The source code lives in `/root/code/cinny`. All changes should be made on the `
See [LOTUS_FEATURES.md](LOTUS_FEATURES.md) for the full feature changelog and [LOTUS_TODO.md](LOTUS_TODO.md) for the work backlog. See [LOTUS_FEATURES.md](LOTUS_FEATURES.md) for the full feature changelog and [LOTUS_TODO.md](LOTUS_TODO.md) for the work backlog.
### 🔱 Planned: Element Call fork ("Lotus Call") ### 🔱 Element Call fork ("Lotus Call") — LIVE
Voice/video channels embed **Element Call**. Today it's a **pre-built npm bundle** Voice/video channels embed **Element Call**, which is now our **self-built fork**
(`@element-hq/element-call-embedded` 0.20.1) copied to `public/element-call/` and (`@lotusguild/element-call-embedded` `0.20.1-lotus.1`, source at
served same-origin; we steer it via the `matrix-widget-api` plus fragile DOM `LotusGuild/element-call`), published to our private Gitea npm registry and served
hacks. Because we don't own its compiled source, several in-call issues (avatar same-origin. We no longer depend on the upstream prebuilt bundle, so in-call
decorations on tiles, camera focus/fullscreen during screenshare, mic recovery behavior is editable source instead of fragile DOM/widget hacks.
after reconnect, native theming, real call-audio injection) are unfixable from
outside.
**The plan is to fork `element-hq/element-call` into a new `LotusGuild/element-call` **Shipped via the fork:** denoise as an in-source LiveKit audio stage (survives
repo, build it from source, and host our own build** for true ownership. The full reconnects), in-call speaking/mute events, focus-a-participant during screenshare,
self-contained plan and integration map — written for a fresh session with no avatar decorations on EC video tiles, and a native transparent background.
prior context — is in **[`HANDOFF_ELEMENT_CALL_FORK.md`](HANDOFF_ELEMENT_CALL_FORK.md)**. **Built but dormant (need cinny UI):** real call-audio injection
Infra/hosting notes also live in the `LotusGuild/matrix` repo README. Search the (`io.lotus.inject_audio` → in-call soundboard) and quality controls
docs for the **`[EC-FORK]`** tag to find every related note. (`io.lotus.set_quality`).
The full plan and integration map is in
**[`HANDOFF_ELEMENT_CALL_FORK.md`](HANDOFF_ELEMENT_CALL_FORK.md)**; infra/hosting +
build-pipeline notes live in the `LotusGuild/matrix` repo README. Search the docs
for the **`[EC-FORK]`** tag to find every related note.
### Build ### Build
+2
View File
@@ -45,6 +45,7 @@ import { useMatrixClient } from '../hooks/useMatrixClient';
import { previewRingtone, startRingtone } from '../utils/ringtones'; import { previewRingtone, startRingtone } from '../utils/ringtones';
import { useCallMembersChange, useCallSession } from '../hooks/useCall'; import { useCallMembersChange, useCallSession } from '../hooks/useCall';
import { useCallJoinLeaveSounds } from '../hooks/useCallJoinLeaveSounds'; import { useCallJoinLeaveSounds } from '../hooks/useCallJoinLeaveSounds';
import { useCallQuality } from '../hooks/useCallQuality';
import { useRemoteAllMuted } from '../hooks/useCallSpeakers'; import { useRemoteAllMuted } from '../hooks/useCallSpeakers';
import { useRoomAvatar, useRoomName } from '../hooks/useRoomMeta'; import { useRoomAvatar, useRoomName } from '../hooks/useRoomMeta';
import { mDirectAtom } from '../state/mDirectList'; import { mDirectAtom } from '../state/mDirectList';
@@ -584,6 +585,7 @@ function CallUtils({ embed }: { embed: CallEmbed }) {
useCallMemberSoundSync(embed); useCallMemberSoundSync(embed);
useCallJoinLeaveSounds(embed); useCallJoinLeaveSounds(embed);
useCallThemeSync(embed); useCallThemeSync(embed);
useCallQuality(embed);
useCallHangupEvent( useCallHangupEvent(
embed, embed,
useCallback(() => { useCallback(() => {
+37 -13
View File
@@ -37,6 +37,10 @@ import { stopPropagation } from '../../utils/keyboard';
import { AsyncStatus, useAsyncCallback } from '../../hooks/useAsyncCallback'; import { AsyncStatus, useAsyncCallback } from '../../hooks/useAsyncCallback';
import { useCallEmbedRef } from '../../hooks/useCallEmbed'; import { useCallEmbedRef } from '../../hooks/useCallEmbed';
import { useAfkAutoMute } from '../../hooks/useAfkAutoMute'; import { useAfkAutoMute } from '../../hooks/useAfkAutoMute';
import { CallSoundboard } from './CallSoundboard';
import { useStateEvent } from '../../hooks/useStateEvent';
import { StateEvent } from '../../../types/matrix/room';
import { RoomQualityContent } from '../../utils/callQuality';
type CallControlsProps = { type CallControlsProps = {
callEmbed: CallEmbed; callEmbed: CallEmbed;
@@ -88,6 +92,19 @@ export function CallControls({ callEmbed }: CallControlsProps) {
const [pttMode] = useSetting(settingsAtom, 'pttMode'); const [pttMode] = useSetting(settingsAtom, 'pttMode');
const [pttKey] = useSetting(settingsAtom, 'pttKey'); const [pttKey] = useSetting(settingsAtom, 'pttKey');
const [deafenKey] = useSetting(settingsAtom, 'deafenKey'); const [deafenKey] = useSetting(settingsAtom, 'deafenKey');
const [soundboardEnabled] = useSetting(settingsAtom, 'soundboardEnabled');
// [P5-31] Hard room publish policy — hide controls the server will refuse so
// users don't click dead buttons. Absent/true = allowed.
const roomQualityEvent = useStateEvent(callEmbed.room, StateEvent.LotusRoomQuality);
const roomQuality = roomQualityEvent?.getContent<RoomQualityContent>();
const cameraAllowed = roomQuality?.allow_camera !== false;
const screenshareAllowed = roomQuality?.allow_screenshare !== false;
// Keep a forbidden control visible while its track is still live (so the user
// can stop it); otherwise hide it entirely.
const showCamera = cameraAllowed || video;
const showScreenshare = screenshareAllowed || screenshare;
const showVideoGroup = showCamera || showScreenshare || !!document.fullscreenEnabled;
const [pttActive, setPttActive] = useState(false); const [pttActive, setPttActive] = useState(false);
// Track microphone via ref so the PTT effect doesn't need it as a dep (avoids listener churn) // Track microphone via ref so the PTT effect doesn't need it as a dep (avoids listener churn)
@@ -339,24 +356,31 @@ export function CallControls({ callEmbed }: CallControlsProps) {
onToggle={() => callEmbed.control.toggleScreenshareAudio()} onToggle={() => callEmbed.control.toggleScreenshareAudio()}
/> />
</Box> </Box>
{!compact && <ControlDivider />} {!compact && showVideoGroup && <ControlDivider />}
<Box shrink="No" alignItems="Inherit" justifyContent="Inherit" gap="200"> {showVideoGroup && (
<VideoButton enabled={video} onToggle={handleVideoToggle} /> <Box shrink="No" alignItems="Inherit" justifyContent="Inherit" gap="200">
<ScreenShareButton {/* Show a forbidden control while its track is still live so the
enabled={screenshare} user can stop it; once stopped it hides and can't be restarted. */}
onToggle={() => {showCamera && <VideoButton enabled={video} onToggle={handleVideoToggle} />}
screenshare ? callEmbed.control.toggleScreenshare() : setShareConfirm(true) {showScreenshare && (
} <ScreenShareButton
/> enabled={screenshare}
{!!document.fullscreenEnabled && ( onToggle={() =>
<FullscreenButton isFullscreen={isFullscreen} onToggle={handleFullscreen} /> screenshare ? callEmbed.control.toggleScreenshare() : setShareConfirm(true)
)} }
</Box> />
)}
{!!document.fullscreenEnabled && (
<FullscreenButton isFullscreen={isFullscreen} onToggle={handleFullscreen} />
)}
</Box>
)}
</Box> </Box>
{!compact && <ControlDivider />} {!compact && <ControlDivider />}
<Box alignItems="Center" gap="Inherit" grow="Yes" direction={compact ? 'Column' : 'Row'}> <Box alignItems="Center" gap="Inherit" grow="Yes" direction={compact ? 'Column' : 'Row'}>
<Box shrink="No" alignItems="Inherit" justifyContent="Inherit" gap="200"> <Box shrink="No" alignItems="Inherit" justifyContent="Inherit" gap="200">
<ChatButton /> <ChatButton />
{soundboardEnabled && <CallSoundboard callEmbed={callEmbed} />}
<PopOut <PopOut
anchor={cords} anchor={cords}
position="Top" position="Top"
+220
View File
@@ -0,0 +1,220 @@
import React, { MouseEventHandler, useCallback, useRef, useState } from 'react';
import {
Box,
Chip,
Icon,
IconButton,
Icons,
Menu,
PopOut,
RectCords,
Spinner,
Text,
Tooltip,
TooltipProvider,
color,
config,
} from 'folds';
import FocusTrap from 'focus-trap-react';
import { CallEmbed } from '../../plugins/call';
import { useMatrixClient } from '../../hooks/useMatrixClient';
import { useSoundboard } from '../../hooks/useSoundboard';
import { useSetting } from '../../state/hooks/settings';
import { settingsAtom } from '../../state/settings';
import { stopPropagation } from '../../utils/keyboard';
import {
SOUNDBOARD_ACCEPT,
SOUNDBOARD_MAX_CLIPS,
playClipLocally,
resolveClipObjectUrl,
} from '../../utils/soundboardClips';
type CallSoundboardProps = {
callEmbed: CallEmbed;
};
/**
* [P5-15] In-call soundboard: trigger user-uploaded clips into the call. Each
* clip is published to peers as a separate track by the EC fork
* (`io.lotus.inject_audio`) and also played locally for the presser's feedback.
* Clips are uploadable/managed here and synced across devices via the
* `io.lotus.soundboard` account data (like custom emoji/sticker packs).
*/
export function CallSoundboard({ callEmbed }: CallSoundboardProps) {
const mx = useMatrixClient();
const { clips, addClip, removeClip } = useSoundboard();
const [soundboardVolume] = useSetting(settingsAtom, 'soundboardVolume');
const [cords, setCords] = useState<RectCords>();
const [busyId, setBusyId] = useState<string>();
const [uploading, setUploading] = useState(false);
const [error, setError] = useState<string>();
const fileInputRef = useRef<HTMLInputElement>(null);
const volume = Math.max(0, Math.min(1, soundboardVolume / 100));
const handleOpen: MouseEventHandler<HTMLButtonElement> = (evt) => {
setError(undefined);
setCords(evt.currentTarget.getBoundingClientRect());
};
const handlePlay = useCallback(
async (id: string, mxc: string) => {
setBusyId(id);
setError(undefined);
try {
const objectUrl = await resolveClipObjectUrl(mx, mxc);
callEmbed.control.injectAudio(objectUrl, volume);
playClipLocally(objectUrl, volume);
} catch {
setError('Could not play that clip.');
} finally {
setBusyId(undefined);
}
},
[mx, callEmbed, volume],
);
const handleFile = useCallback(
async (file: File | undefined) => {
if (!file) return;
setUploading(true);
setError(undefined);
try {
await addClip(file);
} catch (e) {
setError(e instanceof Error ? e.message : 'Upload failed.');
} finally {
setUploading(false);
}
},
[addClip],
);
return (
<>
<input
ref={fileInputRef}
type="file"
accept={SOUNDBOARD_ACCEPT}
hidden
onChange={(e) => {
handleFile(e.target.files?.[0]);
e.target.value = '';
}}
/>
<PopOut
anchor={cords}
position="Top"
align="Center"
content={
<FocusTrap
focusTrapOptions={{
initialFocus: false,
onDeactivate: () => setCords(undefined),
clickOutsideDeactivates: true,
escapeDeactivates: stopPropagation,
}}
>
<Menu style={{ maxWidth: '320px' }}>
<Box direction="Column" gap="200" style={{ padding: config.space.S200 }}>
<Box alignItems="Center" justifyContent="SpaceBetween" gap="200">
<Text size="L400">Soundboard</Text>
<Chip
variant="Secondary"
radii="Pill"
disabled={uploading || clips.length >= SOUNDBOARD_MAX_CLIPS}
onClick={() => fileInputRef.current?.click()}
before={
uploading ? <Spinner size="100" /> : <Icon size="100" src={Icons.Plus} />
}
>
<Text size="B300">Upload</Text>
</Chip>
</Box>
{clips.length === 0 ? (
<Text size="T200" priority="300">
No clips yet. Upload a short audio clip (max 1 MB) to play it into the call.
Clips sync across your devices.
</Text>
) : (
<Box wrap="Wrap" gap="200">
{clips.map((clip) => (
<Box
key={clip.id}
direction="Column"
gap="100"
style={{ position: 'relative' }}
>
<Chip
variant="SurfaceVariant"
radii="300"
disabled={busyId === clip.id}
onClick={() => handlePlay(clip.id, clip.url)}
before={
busyId === clip.id ? (
<Spinner size="100" />
) : (
<Icon size="100" src={Icons.Play} />
)
}
after={
<Icon
size="50"
src={Icons.Cross}
style={{ cursor: 'pointer' }}
onClick={(e: React.MouseEvent) => {
e.stopPropagation();
removeClip(clip.id);
}}
/>
}
>
<Text size="B300" truncate style={{ maxWidth: '120px' }}>
{clip.name}
</Text>
</Chip>
</Box>
))}
</Box>
)}
{error && (
<Text size="T200" style={{ color: color.Critical.Main }}>
{error}
</Text>
)}
</Box>
</Menu>
</FocusTrap>
}
>
<TooltipProvider
tooltip={
<Tooltip>
<Text>Soundboard</Text>
</Tooltip>
}
>
{(triggerRef) => (
<IconButton
ref={triggerRef}
variant="Surface"
fill="Soft"
radii="400"
size="400"
onClick={handleOpen}
outlined
aria-label="Soundboard"
aria-expanded={!!cords}
aria-haspopup="menu"
>
<Icon size="400" src={Icons.BellRing} />
</IconButton>
)}
</TooltipProvider>
</PopOut>
</>
);
}
@@ -0,0 +1,198 @@
import React, { useCallback, useEffect, useMemo, useState } from 'react';
import { Box, Switch, Text } from 'folds';
import { SequenceCard } from '../../../components/sequence-card';
import { SequenceCardStyle } from '../../room-settings/styles.css';
import { SettingTile } from '../../../components/setting-tile';
import { SettingsSelect } from '../../../components/settings-select/SettingsSelect';
import { useMatrixClient } from '../../../hooks/useMatrixClient';
import { useRoom } from '../../../hooks/useRoom';
import { StateEvent } from '../../../../types/matrix/room';
import { useStateEvent } from '../../../hooks/useStateEvent';
import { RoomPermissionsAPI } from '../../../hooks/useRoomPermissions';
import { AsyncStatus, useAsyncCallback } from '../../../hooks/useAsyncCallback';
import {
AUDIO_BITRATE_OPTIONS,
RoomQualityContent,
SCREENSHARE_BITRATE_OPTIONS,
SCREENSHARE_FRAMERATE_OPTIONS,
} from '../../../utils/callQuality';
// Only the numeric cap keys are edited via `update`; the boolean policy keys
// are handled by `setAllow`.
type CapKey = 'audio_max_kbps' | 'screenshare_max_kbps' | 'screenshare_max_fps';
// String <-> numeric bridge for SettingsSelect (which needs string values).
const toValue = (n?: number): string => (typeof n === 'number' ? String(n) : 'auto');
const CAP_KEYS: (keyof RoomQualityContent)[] = [
'audio_max_kbps',
'screenshare_max_kbps',
'screenshare_max_fps',
'allow_screenshare',
'allow_camera',
];
const capsEqual = (a: RoomQualityContent, b: RoomQualityContent): boolean =>
CAP_KEYS.every((k) => a[k] === b[k]);
type RoomQualityProps = {
permissions: RoomPermissionsAPI;
};
/**
* [P5-31] Room-admin quality ceiling. Writes `io.lotus.room_quality`; every
* Lotus client clamps its per-user quality to these caps. Hard enforcement for
* ALL Matrix clients is a server-side follow-up (see LOTUS_TODO.md P5-31).
*/
export function RoomQuality({ permissions }: RoomQualityProps) {
const mx = useMatrixClient();
const room = useRoom();
const canEdit = permissions.stateEvent(StateEvent.LotusRoomQuality, mx.getSafeUserId());
const event = useStateEvent(room, StateEvent.LotusRoomQuality);
const caps = useMemo<RoomQualityContent>(() => event?.getContent() ?? {}, [event]);
const [submitState, submit] = useAsyncCallback(
useCallback(
async (next: RoomQualityContent) => {
// eslint-disable-next-line @typescript-eslint/no-explicit-any
await mx.sendStateEvent(room.roomId, StateEvent.LotusRoomQuality as any, next);
},
[mx, room.roomId],
),
);
const submitting = submitState.status === AsyncStatus.Loading;
// Optimistic mirror: `useStateEvent` only refreshes when the write echoes
// back via /sync (not when sendStateEvent resolves), so consecutive edits
// must build on the pending write — otherwise a second edit spreads a stale
// `caps` and silently drops the first. `effective` is what the UI shows and
// what each edit merges into; it's reconciled below once the echo lands.
const [pending, setPending] = useState<RoomQualityContent | null>(null);
const effective = pending ?? caps;
useEffect(() => {
if (!pending) return;
// Revert the optimistic view if the write failed…
if (submitState.status === AsyncStatus.Error) {
setPending(null);
return;
}
// …or drop it once the synced state actually reflects it.
if (capsEqual(caps, pending)) setPending(null);
}, [caps, pending, submitState.status]);
const commit = (next: RoomQualityContent) => {
setPending(next);
submit(next);
};
const update = (key: CapKey, value: string) => {
const next: RoomQualityContent = { ...effective };
if (value === 'auto') delete next[key];
else next[key] = parseInt(value, 10);
commit(next);
};
const setAllow = (key: 'allow_screenshare' | 'allow_camera', allowed: boolean) => {
const next: RoomQualityContent = { ...effective };
// Absent = allowed, so only persist the key when forbidding.
if (allowed) delete next[key];
else next[key] = false;
commit(next);
};
// Absent/true = allowed.
const screenshareAllowed = effective.allow_screenshare !== false;
const cameraAllowed = effective.allow_camera !== false;
return (
<SequenceCard
className={SequenceCardStyle}
variant="SurfaceVariant"
direction="Column"
gap="400"
>
<SettingTile
title="Call Permissions"
description={
<Text size="T200" priority="300">
Control what participants may share in this room. These are enforced on the server for
every Matrix client (Element, FluffyChat, Lotus Chat, ).
</Text>
}
/>
<Box direction="Column" gap="300">
<SettingTile
title="Allow Screen Sharing"
description="When off, no one can share their screen in this room."
after={
<Switch
variant="Primary"
value={screenshareAllowed}
onChange={(v) => setAllow('allow_screenshare', v)}
disabled={!canEdit || submitting}
/>
}
/>
<SettingTile
title="Allow Camera"
description="When off, this is an audio-only room — no one can turn on their camera. Microphones are always allowed."
after={
<Switch
variant="Primary"
value={cameraAllowed}
onChange={(v) => setAllow('allow_camera', v)}
disabled={!canEdit || submitting}
/>
}
/>
</Box>
<SettingTile
title="Call Quality Caps"
description={
<Text size="T200" priority="300">
Set a maximum microphone bitrate, screenshare bitrate, and screenshare framerate for
this room. Lotus Chat clamps each participant to these ceilings (best-effort applies
to Lotus Chat clients). Auto = no cap.
</Text>
}
/>
<Box direction="Column" gap="300">
<SettingTile
title="Max Microphone Bitrate"
after={
<SettingsSelect
value={toValue(effective.audio_max_kbps)}
onChange={(v) => update('audio_max_kbps', v)}
options={AUDIO_BITRATE_OPTIONS}
disabled={!canEdit || submitting}
/>
}
/>
<SettingTile
title="Max Screenshare Bitrate"
after={
<SettingsSelect
value={toValue(effective.screenshare_max_kbps)}
onChange={(v) => update('screenshare_max_kbps', v)}
options={SCREENSHARE_BITRATE_OPTIONS}
disabled={!canEdit || submitting}
/>
}
/>
<SettingTile
title="Max Screenshare Framerate"
after={
<SettingsSelect
value={toValue(effective.screenshare_max_fps)}
onChange={(v) => update('screenshare_max_fps', v)}
options={SCREENSHARE_FRAMERATE_OPTIONS}
disabled={!canEdit || submitting}
/>
}
/>
</Box>
</SequenceCard>
);
}
@@ -4,6 +4,7 @@ export * from './RoomHistoryVisibility';
export * from './RoomJoinRules'; export * from './RoomJoinRules';
export * from './RoomProfile'; export * from './RoomProfile';
export * from './RoomPublish'; export * from './RoomPublish';
export * from './RoomQuality';
export * from './RoomShareInvite'; export * from './RoomShareInvite';
export * from './RoomUpgrade'; export * from './RoomUpgrade';
export * from './RoomVoiceLimit'; export * from './RoomVoiceLimit';
@@ -11,6 +11,7 @@ import {
RoomLocalAddresses, RoomLocalAddresses,
RoomPublishedAddresses, RoomPublishedAddresses,
RoomPublish, RoomPublish,
RoomQuality,
RoomShareInvite, RoomShareInvite,
RoomUpgrade, RoomUpgrade,
RoomVoiceLimit, RoomVoiceLimit,
@@ -58,6 +59,7 @@ export function General({ requestClose }: GeneralProps) {
<Box direction="Column" gap="100"> <Box direction="Column" gap="100">
<Text size="L400">Voice</Text> <Text size="L400">Voice</Text>
<RoomVoiceLimit permissions={permissions} /> <RoomVoiceLimit permissions={permissions} />
<RoomQuality permissions={permissions} />
</Box> </Box>
<Box direction="Column" gap="100"> <Box direction="Column" gap="100">
<Text size="L400">Addresses</Text> <Text size="L400">Addresses</Text>
@@ -45,6 +45,7 @@ import {
} from '../../../utils/lotusDenoiseUtils'; } from '../../../utils/lotusDenoiseUtils';
import { useSetting } from '../../../state/hooks/settings'; import { useSetting } from '../../../state/hooks/settings';
import { import {
CallAudioBitrate,
ChatBackground, ChatBackground,
ComposerToolbarSettings, ComposerToolbarSettings,
DateFormat, DateFormat,
@@ -53,9 +54,16 @@ import {
MessageSpacing, MessageSpacing,
NoiseSuppressionMode, NoiseSuppressionMode,
RingtoneId, RingtoneId,
ScreenshareBitrate,
ScreenshareFramerate,
Settings, Settings,
settingsAtom, settingsAtom,
} from '../../../state/settings'; } from '../../../state/settings';
import {
AUDIO_BITRATE_OPTIONS,
SCREENSHARE_BITRATE_OPTIONS,
SCREENSHARE_FRAMERATE_OPTIONS,
} from '../../../utils/callQuality';
import { SeasonalPreview, SeasonTheme } from '../../../components/seasonal/SeasonalEffect'; import { SeasonalPreview, SeasonTheme } from '../../../components/seasonal/SeasonalEffect';
import { SEASON_DATE_RANGES } from '../../../components/seasonal/seasonSchedule'; import { SEASON_DATE_RANGES } from '../../../components/seasonal/seasonSchedule';
import { SettingTile } from '../../../components/setting-tile'; import { SettingTile } from '../../../components/setting-tile';
@@ -1221,6 +1229,18 @@ function Calls() {
const [ringtoneVolume, setRingtoneVolume] = useSetting(settingsAtom, 'ringtoneVolume'); const [ringtoneVolume, setRingtoneVolume] = useSetting(settingsAtom, 'ringtoneVolume');
const [ringtoneId, setRingtoneId] = useSetting(settingsAtom, 'ringtoneId'); const [ringtoneId, setRingtoneId] = useSetting(settingsAtom, 'ringtoneId');
const [callAudioBitrate, setCallAudioBitrate] = useSetting(settingsAtom, 'callAudioBitrate');
const [screenshareBitrate, setScreenshareBitrate] = useSetting(
settingsAtom,
'screenshareBitrate',
);
const [screenshareFramerate, setScreenshareFramerate] = useSetting(
settingsAtom,
'screenshareFramerate',
);
const [soundboardEnabled, setSoundboardEnabled] = useSetting(settingsAtom, 'soundboardEnabled');
const [soundboardVolume, setSoundboardVolume] = useSetting(settingsAtom, 'soundboardVolume');
const handleJoinLeaveSoundChange = (value: 'off' | 'chime' | 'soft' | 'retro') => { const handleJoinLeaveSoundChange = (value: 'off' | 'chime' | 'soft' | 'retro') => {
setCallJoinLeaveSound(value); setCallJoinLeaveSound(value);
if (value !== 'off') playCallJoinSound(value); if (value !== 'off') playCallJoinSound(value);
@@ -1616,6 +1636,80 @@ function Calls() {
} }
/> />
</SequenceCard> </SequenceCard>
<SequenceCard className={SequenceCardStyle} variant="SurfaceVariant" direction="Column">
<SettingTile
title="Microphone Bitrate"
description="Cap the audio bitrate your mic sends in calls. Lower saves bandwidth; higher is clearer. Auto lets Element Call decide."
after={
<SettingsSelect<CallAudioBitrate>
value={callAudioBitrate}
onChange={setCallAudioBitrate}
options={AUDIO_BITRATE_OPTIONS}
/>
}
/>
</SequenceCard>
<SequenceCard className={SequenceCardStyle} variant="SurfaceVariant" direction="Column">
<SettingTile
title="Screenshare Bitrate"
description="Cap the bitrate used when you share your screen. Lower is smoother on poor connections; higher is sharper."
after={
<SettingsSelect<ScreenshareBitrate>
value={screenshareBitrate}
onChange={setScreenshareBitrate}
options={SCREENSHARE_BITRATE_OPTIONS}
/>
}
/>
</SequenceCard>
<SequenceCard className={SequenceCardStyle} variant="SurfaceVariant" direction="Column">
<SettingTile
title="Screenshare Framerate"
description="Cap the frames-per-second of your screenshare. 60 fps suits motion/gaming; 15 fps suits slides and saves bandwidth."
after={
<SettingsSelect<ScreenshareFramerate>
value={screenshareFramerate}
onChange={setScreenshareFramerate}
options={SCREENSHARE_FRAMERATE_OPTIONS}
/>
}
/>
</SequenceCard>
<SequenceCard className={SequenceCardStyle} variant="SurfaceVariant" direction="Column">
<SettingTile
title="Soundboard"
description="Show a soundboard button in the call bar. Upload short audio clips (like custom emojis) to play them into the call. Clips sync across your devices."
after={
<Switch variant="Primary" value={soundboardEnabled} onChange={setSoundboardEnabled} />
}
/>
{soundboardEnabled && (
<SettingTile
title="Soundboard Volume"
after={
<Box alignItems="Center" gap="200" style={{ minWidth: toRem(180) }}>
<input
type="range"
min={0}
max={100}
step={5}
value={soundboardVolume}
onChange={(e) => setSoundboardVolume(parseInt(e.target.value, 10))}
style={{ flexGrow: 1 }}
aria-label="Soundboard volume"
/>
<Text size="T200" style={{ minWidth: toRem(36), textAlign: 'right' }}>
{soundboardVolume}%
</Text>
</Box>
}
/>
)}
</SequenceCard>
</Box> </Box>
); );
} }
+41
View File
@@ -0,0 +1,41 @@
import { useEffect } from 'react';
import { useAtomValue } from 'jotai';
import { CallEmbed } from '../plugins/call';
import { settingsAtom } from '../state/settings';
import { useStateEvent } from './useStateEvent';
import { StateEvent } from '../../types/matrix/room';
import { buildQualityPayload, RoomQualityContent } from '../utils/callQuality';
/**
* [P5-31] Apply the user's call quality settings (clamped by any room-level
* cap) to the Element Call fork via the `io.lotus.set_quality` widget action.
*
* The fork stores the settings and re-applies them on every (re)publish and
* reconnect, so we only need to (re)send when the payload changes or the widget
* becomes ready — no need to poll the track lifecycle here.
*/
export function useCallQuality(embed: CallEmbed): void {
const { callAudioBitrate, screenshareBitrate, screenshareFramerate } = useAtomValue(settingsAtom);
const roomQualityEvent = useStateEvent(embed.room, StateEvent.LotusRoomQuality);
const roomCaps = roomQualityEvent?.getContent<RoomQualityContent>();
// Depend on the primitive cap values (not the event object) so re-renders
// don't resend needlessly.
const audioCap = roomCaps?.audio_max_kbps;
const ssCap = roomCaps?.screenshare_max_kbps;
const fpsCap = roomCaps?.screenshare_max_fps;
useEffect(() => {
const payload = buildQualityPayload(
{ callAudioBitrate, screenshareBitrate, screenshareFramerate },
{ audio_max_kbps: audioCap, screenshare_max_kbps: ssCap, screenshare_max_fps: fpsCap },
);
const send = (): void => embed.control.setQuality(payload);
// Send now (settings are sticky fork-side even if tracks aren't up yet) and
// again once the widget signals ready, in case the transport wasn't up.
send();
const off = embed.onReady(send);
return off;
}, [embed, callAudioBitrate, screenshareBitrate, screenshareFramerate, audioCap, ssCap, fpsCap]);
}
+101
View File
@@ -0,0 +1,101 @@
import { useCallback, useEffect, useState } from 'react';
import { useMatrixClient } from './useMatrixClient';
import { useAccountDataCallback } from './useAccountDataCallback';
import { AccountDataEvent } from '../../types/matrix/accountData';
import {
SoundboardClip,
SoundboardContent,
SOUNDBOARD_MAX_CLIP_BYTES,
SOUNDBOARD_MAX_CLIPS,
SOUNDBOARD_NAME_MAX,
readSoundboardClips,
} from '../utils/soundboardClips';
const KEY = AccountDataEvent.LotusSoundboard;
/**
* [P5-15] Read/write the user's personal soundboard, stored in the
* `io.lotus.soundboard` account data event (synced across devices like custom
* emoji/sticker packs). Uploading writes the audio to the media repo and
* appends an mxc reference.
*/
export function useSoundboard(): {
clips: SoundboardClip[];
addClip: (file: File, name?: string) => Promise<void>;
removeClip: (id: string) => Promise<void>;
renameClip: (id: string, name: string) => Promise<void>;
} {
const mx = useMatrixClient();
const [clips, setClips] = useState<SoundboardClip[]>(() => readSoundboardClips(mx));
useAccountDataCallback(
mx,
useCallback((evt) => {
if (evt.getType() === KEY) {
const content = evt.getContent<SoundboardContent>();
setClips(Array.isArray(content?.clips) ? content.clips : []);
}
}, []),
);
useEffect(() => {
setClips(readSoundboardClips(mx));
}, [mx]);
const persist = useCallback(
async (next: SoundboardClip[]) => {
const content: SoundboardContent = { clips: next };
await (
mx as unknown as { setAccountData: (t: string, c: unknown) => Promise<void> }
).setAccountData(KEY, content);
},
[mx],
);
const addClip = useCallback(
async (file: File, name?: string) => {
const current = readSoundboardClips(mx);
if (current.length >= SOUNDBOARD_MAX_CLIPS) {
throw new Error(`Soundboard is full (max ${SOUNDBOARD_MAX_CLIPS} clips).`);
}
if (file.size > SOUNDBOARD_MAX_CLIP_BYTES) {
throw new Error('Clip is too large (max 1 MB).');
}
const res = await mx.uploadContent(file, { type: file.type || 'audio/mpeg' });
const mxc = res.content_uri;
if (!mxc) throw new Error('Upload failed.');
const label = (name ?? file.name.replace(/\.[^/.]+$/, ''))
.trim()
.slice(0, SOUNDBOARD_NAME_MAX);
const clip: SoundboardClip = {
id: `${Date.now()}-${Math.random().toString(36).slice(2, 8)}`,
name: label || 'Clip',
url: mxc,
mimetype: file.type || undefined,
size: file.size,
};
await persist([...current, clip]);
},
[mx, persist],
);
const removeClip = useCallback(
async (id: string) => {
const next = readSoundboardClips(mx).filter((c) => c.id !== id);
await persist(next);
},
[mx, persist],
);
const renameClip = useCallback(
async (id: string, name: string) => {
const trimmed = name.trim().slice(0, SOUNDBOARD_NAME_MAX);
if (!trimmed) return;
const next = readSoundboardClips(mx).map((c) => (c.id === id ? { ...c, name: trimmed } : c));
await persist(next);
},
[mx, persist],
);
return { clips, addClip, removeClip, renameClip };
}
+38
View File
@@ -7,6 +7,17 @@ export enum CallControlEvent {
StateUpdate = 'state_update', StateUpdate = 'state_update',
} }
/**
* [lotus #7 / P5-31] Payload for the fork's `io.lotus.set_quality` action.
* All fields optional; `null` clears that cap. Bits/sec for bitrates, fps for
* framerate.
*/
export type LotusQualityPayload = {
audioMaxBitrate?: number | null;
screenshareMaxBitrate?: number | null;
screenshareMaxFramerate?: number | null;
};
export class CallControl extends EventEmitter implements CallControlState { export class CallControl extends EventEmitter implements CallControlState {
private state: CallControlState; private state: CallControlState;
@@ -358,6 +369,33 @@ export class CallControl extends EventEmitter implements CallControlState {
this.call.transport.send('io.lotus.focus_participant', { userId: null }).catch(() => undefined); this.call.transport.send('io.lotus.focus_participant', { userId: null }).catch(() => undefined);
} }
/**
* [lotus #3 / P5-15] Inject a soundboard clip into the call so other
* participants hear it. The fork publishes it as a separate LiveKit audio
* track (`io.lotus.inject_audio`) rather than splicing the mic. `url` must be
* an https/blob URL the widget can fetch WITHOUT credentials — the host
* resolves an mxc clip to a `blob:` object URL first (authenticated media
* can't be fetched cross-realm by the widget). `volume` is 01.
*
* The local user does not hear their own published track, so callers should
* also play the clip locally for feedback.
*/
public injectAudio(url: string, volume = 1): void {
this.call.transport.send('io.lotus.inject_audio', { url, volume }).catch(() => undefined);
}
/**
* [lotus #7 / P5-31] Apply audio/screenshare encoding limits to the local
* published tracks (the fork's `io.lotus.set_quality` action, via
* `RTCRtpSender.setParameters` — no republish). Bitrates are bits/sec,
* framerate is fps. A field set to `null` clears that cap. Settings are
* sticky fork-side (re-applied on every re-publish / reconnect). Values are
* clamped fork-side, so out-of-range input can't brick the encoder.
*/
public setQuality(settings: LotusQualityPayload): void {
this.call.transport.send('io.lotus.set_quality', settings).catch(() => undefined);
}
public dispose() { public dispose() {
this.bodyMutationObserver.disconnect(); this.bodyMutationObserver.disconnect();
this.controlMutationObserver.disconnect(); this.controlMutationObserver.disconnect();
+5
View File
@@ -179,6 +179,11 @@ export class CallEmbed {
// - transparent background so the room wallpaper shows through natively // - transparent background so the room wallpaper shows through natively
lotusCallState: 'true', lotusCallState: 'true',
lotusTransparent: 'true', lotusTransparent: 'true',
// [lotus #3 / P5-15] Arm the fork's audio-inject handler so the in-call
// soundboard can publish clips into the call. Dormant until the host
// sends io.lotus.inject_audio (only on an explicit user click), so
// arming it for every call is safe.
lotusAudioInject: 'true',
}); });
if (denoiseMode === 'ml') { if (denoiseMode === 'ml') {
+21
View File
@@ -24,6 +24,13 @@ export type DenoiseModelId = 'rnnoise' | 'speex' | 'dtln' | 'deepfilternet';
// 'soft' / 'retro' are synthesized in-browser (see utils/ringtones.ts); // 'soft' / 'retro' are synthesized in-browser (see utils/ringtones.ts);
// 'none' is silent (visual-only incoming-call UI). // 'none' is silent (visual-only incoming-call UI).
export type RingtoneId = 'classic' | 'chime' | 'soft' | 'retro' | 'none'; export type RingtoneId = 'classic' | 'chime' | 'soft' | 'retro' | 'none';
// [P5-31] Granular call quality caps. 'auto' = don't cap (the EC fork keeps its
// default encoding). Numbers are kbps (audio/screenshare bitrate) or fps
// (screenshare framerate); converted to the fork's bits/sec + fps payload in
// utils/callQuality.ts and applied via the io.lotus.set_quality widget action.
export type CallAudioBitrate = 'auto' | '32' | '64' | '96' | '128' | '256';
export type ScreenshareBitrate = 'auto' | '500' | '1500' | '3000' | '8000';
export type ScreenshareFramerate = 'auto' | '15' | '30' | '60';
export type ChatBackground = export type ChatBackground =
| 'none' | 'none'
| 'blueprint' | 'blueprint'
@@ -156,6 +163,14 @@ export interface Settings {
ringtoneId: RingtoneId; ringtoneId: RingtoneId;
ringtoneVolume: number; // 0100 ringtoneVolume: number; // 0100
// [P5-31] Call quality controls
callAudioBitrate: CallAudioBitrate;
screenshareBitrate: ScreenshareBitrate;
screenshareFramerate: ScreenshareFramerate;
// [P5-15] In-call soundboard
soundboardEnabled: boolean;
soundboardVolume: number; // 0100
seasonalThemeOverride: seasonalThemeOverride:
| 'auto' | 'auto'
| 'off' | 'off'
@@ -253,6 +268,12 @@ const defaultSettings: Settings = {
ringtoneId: 'classic', ringtoneId: 'classic',
ringtoneVolume: 70, ringtoneVolume: 70,
callAudioBitrate: 'auto',
screenshareBitrate: 'auto',
screenshareFramerate: 'auto',
soundboardEnabled: true,
soundboardVolume: 80,
seasonalThemeOverride: 'auto', seasonalThemeOverride: 'auto',
}; };
+60
View File
@@ -0,0 +1,60 @@
import { describe, it } from 'node:test';
import assert from 'node:assert/strict';
import { buildQualityPayload } from './callQuality';
describe('buildQualityPayload', () => {
it("sends null for every 'auto' field so a prior cap is reset", () => {
const payload = buildQualityPayload({
callAudioBitrate: 'auto',
screenshareBitrate: 'auto',
screenshareFramerate: 'auto',
});
assert.deepEqual(payload, {
audioMaxBitrate: null,
screenshareMaxBitrate: null,
screenshareMaxFramerate: null,
});
});
it('converts kbps user settings to bits/sec and passes fps through', () => {
const payload = buildQualityPayload({
callAudioBitrate: '64',
screenshareBitrate: '1500',
screenshareFramerate: '30',
});
assert.equal(payload.audioMaxBitrate, 64_000);
assert.equal(payload.screenshareMaxBitrate, 1_500_000);
assert.equal(payload.screenshareMaxFramerate, 30);
});
it('clamps the user setting down to the room cap (lower wins)', () => {
const payload = buildQualityPayload(
{ callAudioBitrate: '256', screenshareBitrate: '8000', screenshareFramerate: '60' },
{ audio_max_kbps: 64, screenshare_max_kbps: 1500, screenshare_max_fps: 30 },
);
assert.equal(payload.audioMaxBitrate, 64_000);
assert.equal(payload.screenshareMaxBitrate, 1_500_000);
assert.equal(payload.screenshareMaxFramerate, 30);
});
it('does not raise a user setting that is already below the room cap', () => {
const payload = buildQualityPayload(
{ callAudioBitrate: '32', screenshareBitrate: 'auto', screenshareFramerate: '15' },
{ audio_max_kbps: 128, screenshare_max_kbps: 3000, screenshare_max_fps: 60 },
);
assert.equal(payload.audioMaxBitrate, 32_000);
// user 'auto' but room caps screenshare bitrate -> room cap applies
assert.equal(payload.screenshareMaxBitrate, 3_000_000);
assert.equal(payload.screenshareMaxFramerate, 15);
});
it('applies a room cap even when the user left the field on auto', () => {
const payload = buildQualityPayload(
{ callAudioBitrate: 'auto', screenshareBitrate: 'auto', screenshareFramerate: 'auto' },
{ audio_max_kbps: 96 },
);
assert.equal(payload.audioMaxBitrate, 96_000);
assert.equal(payload.screenshareMaxBitrate, null);
assert.equal(payload.screenshareMaxFramerate, null);
});
});
+96
View File
@@ -0,0 +1,96 @@
import { LotusQualityPayload } from '../plugins/call/CallControl';
import { CallAudioBitrate, ScreenshareBitrate, ScreenshareFramerate } from '../state/settings';
/**
* [P5-31] Room-level quality caps, stored in the `io.lotus.room_quality` state
* event. Admins set a ceiling every client must stay under. Values mirror the
* user-setting units (kbps / fps); `undefined`/absent = no cap.
*
* NOTE: the client applies these as a best-effort UX cap. Hard enforcement for
* ALL Matrix clients is a server-side follow-up (a `voice-limit-guard`-style
* sidecar on LXC 151 that reads this event) — see LOTUS_TODO.md P5-31.
*/
export type RoomQualityContent = {
// Numeric caps: client-cooperative only (our fork honors them; the SFU cannot
// enforce publisher bitrate/fps — LiveKit forwards, never transcodes).
audio_max_kbps?: number;
screenshare_max_kbps?: number;
screenshare_max_fps?: number;
// Publish-source policy: HARD-enforced server-side for ALL clients by the
// voice-limit-guard (it re-signs the LiveKit JWT's canPublishSources).
// Absent/true = allowed; only an explicit false forbids.
allow_screenshare?: boolean;
allow_camera?: boolean;
};
// Selectable options (kbps / fps), shared by the settings UI and the room-admin
// UI so they stay in sync. Values are strings so they satisfy SettingsSelect's
// `T extends string` constraint; parsed to numbers in buildQualityPayload.
export const AUDIO_BITRATE_OPTIONS: { value: CallAudioBitrate; label: string }[] = [
{ value: 'auto', label: 'Auto' },
{ value: '32', label: '32 kbps' },
{ value: '64', label: '64 kbps' },
{ value: '96', label: '96 kbps' },
{ value: '128', label: '128 kbps' },
{ value: '256', label: '256 kbps' },
];
export const SCREENSHARE_BITRATE_OPTIONS: { value: ScreenshareBitrate; label: string }[] = [
{ value: 'auto', label: 'Auto' },
{ value: '500', label: '0.5 Mbps' },
{ value: '1500', label: '1.5 Mbps' },
{ value: '3000', label: '3 Mbps' },
{ value: '8000', label: '8 Mbps' },
];
export const SCREENSHARE_FRAMERATE_OPTIONS: { value: ScreenshareFramerate; label: string }[] = [
{ value: 'auto', label: 'Auto' },
{ value: '15', label: '15 fps' },
{ value: '30', label: '30 fps' },
{ value: '60', label: '60 fps' },
];
/** Lower of two caps, treating `undefined` as "no cap on that side". */
const minCap = (a: number | undefined, b: number | undefined): number | undefined => {
if (a === undefined) return b;
if (b === undefined) return a;
return Math.min(a, b);
};
/** Parse a setting value ('auto' | numeric string) to a number or undefined. */
const num = (v: string): number | undefined => {
if (v === 'auto') return undefined;
const n = parseInt(v, 10);
return Number.isFinite(n) ? n : undefined;
};
type QualitySettings = {
callAudioBitrate: CallAudioBitrate;
screenshareBitrate: ScreenshareBitrate;
screenshareFramerate: ScreenshareFramerate;
};
/**
* Build the `io.lotus.set_quality` payload from the user's settings, clamped by
* any room-level cap. Every field is always present so clearing a setting back
* to 'auto' sends an explicit `null` that resets the fork-side cap (otherwise a
* previously-applied cap would stick for the rest of the call).
*/
export const buildQualityPayload = (
settings: QualitySettings,
roomCaps?: RoomQualityContent,
): LotusQualityPayload => {
const userAudio = num(settings.callAudioBitrate);
const userSsBitrate = num(settings.screenshareBitrate);
const userSsFps = num(settings.screenshareFramerate);
const audioKbps = minCap(userAudio, roomCaps?.audio_max_kbps);
const ssKbps = minCap(userSsBitrate, roomCaps?.screenshare_max_kbps);
const ssFps = minCap(userSsFps, roomCaps?.screenshare_max_fps);
return {
audioMaxBitrate: audioKbps === undefined ? null : audioKbps * 1000,
screenshareMaxBitrate: ssKbps === undefined ? null : ssKbps * 1000,
screenshareMaxFramerate: ssFps === undefined ? null : ssFps,
};
};
+72
View File
@@ -0,0 +1,72 @@
import { MatrixClient } from 'matrix-js-sdk';
import { downloadMedia, mxcUrlToHttp } from './matrix';
/**
* [P5-15] A user-uploaded soundboard clip. Stored (as a list) in the
* `io.lotus.soundboard` account data event, so clips sync across a user's
* devices exactly like custom emoji / sticker packs.
*/
export type SoundboardClip = {
/** Stable local id (not shared with peers). */
id: string;
/** Display name / shortcode shown on the tile. */
name: string;
/** mxc:// URI of the uploaded audio. */
url: string;
mimetype?: string;
size?: number;
};
export type SoundboardContent = {
clips?: SoundboardClip[];
};
export const SOUNDBOARD_NAME_MAX = 24;
/** Keep clips short: they publish to every peer and hold a track open. */
export const SOUNDBOARD_MAX_CLIP_BYTES = 1024 * 1024; // 1 MB
export const SOUNDBOARD_MAX_CLIPS = 40;
export const SOUNDBOARD_ACCEPT = 'audio/mpeg,audio/ogg,audio/wav,audio/webm,audio/mp4,audio/aac';
// Cache resolved object URLs per mxc so re-triggering a clip doesn't re-download
// it. Object URLs live for the page session; the set is tiny (<= MAX_CLIPS).
const objectUrlCache = new Map<string, string>();
/**
* Resolve an mxc clip to a `blob:` object URL the Element Call widget can fetch
* without credentials. Authenticated media (MSC3916) can't be fetched from the
* widget's realm, so the host downloads it (auth handled by the service worker)
* and hands the widget a same-session blob URL instead.
*/
export const resolveClipObjectUrl = async (mx: MatrixClient, mxcUrl: string): Promise<string> => {
const cached = objectUrlCache.get(mxcUrl);
if (cached) return cached;
const httpUrl = mxcUrlToHttp(mx, mxcUrl, true);
if (!httpUrl) throw new Error('invalid mxc url');
const blob = await downloadMedia(httpUrl);
const objectUrl = URL.createObjectURL(blob);
objectUrlCache.set(mxcUrl, objectUrl);
return objectUrl;
};
/**
* Play a resolved clip locally so the person who pressed it gets immediate
* feedback — LiveKit doesn't loop a participant's own published track back to
* them, so without this the presser would hear nothing. `volume` is 01.
*/
export const playClipLocally = (objectUrl: string, volume: number): void => {
try {
const audio = new Audio(objectUrl);
audio.volume = Math.max(0, Math.min(1, volume));
audio.play().catch(() => undefined);
} catch {
/* best effort */
}
};
export const readSoundboardClips = (mx: MatrixClient): SoundboardClip[] => {
const content = mx.getAccountData('io.lotus.soundboard' as never)?.getContent() as
| SoundboardContent
| undefined;
return Array.isArray(content?.clips) ? content.clips : [];
};
+4
View File
@@ -10,6 +10,10 @@ export enum AccountDataEvent {
PoniesUserEmotes = 'im.ponies.user_emotes', PoniesUserEmotes = 'im.ponies.user_emotes',
PoniesEmoteRooms = 'im.ponies.emote_rooms', PoniesEmoteRooms = 'im.ponies.emote_rooms',
// [P5-15] Personal, uploadable in-call soundboard clips (synced across
// devices like custom emoji/sticker packs).
LotusSoundboard = 'io.lotus.soundboard',
SecretStorageDefaultKey = 'm.secret_storage.default_key', SecretStorageDefaultKey = 'm.secret_storage.default_key',
CrossSigningMaster = 'm.cross_signing.master', CrossSigningMaster = 'm.cross_signing.master',
+1
View File
@@ -41,6 +41,7 @@ export enum StateEvent {
PoniesRoomEmotes = 'im.ponies.room_emotes', PoniesRoomEmotes = 'im.ponies.room_emotes',
PowerLevelTags = 'in.cinny.room.power_level_tags', PowerLevelTags = 'in.cinny.room.power_level_tags',
LotusVoiceLimit = 'io.lotus.voice_limit', LotusVoiceLimit = 'io.lotus.voice_limit',
LotusRoomQuality = 'io.lotus.room_quality',
} }
export enum MessageEvent { export enum MessageEvent {