From 7c06b27c73c8f7d6815f964f25fbfab94241dd25 Mon Sep 17 00:00:00 2001 From: Jared Vititoe Date: Tue, 30 Jun 2026 22:34:17 -0400 Subject: [PATCH] feat(call): in-call soundboard, quality controls, room call-permissions Element Call is now consumed as our self-built fork (@lotusguild/element-call-embedded); wire up its previously-dormant capabilities and document the fork as live. Soundboard (P5-15): a call-bar button plays user-uploaded audio clips into the call as a real published track (io.lotus.inject_audio) plus local playback. Clips are uploadable like emoji/sticker packs, stored in io.lotus.soundboard account data (synced across devices). Gated by a Settings toggle + volume. Quality controls (P5-31): per-user mic/screenshare bitrate + screenshare framerate (Settings -> Calls), applied via io.lotus.set_quality clamped to any room cap. Room admins set caps and hard call-permissions (allow_screenshare / allow_camera) in Room Settings -> Voice; the call bar hides blocked buttons. - New: CallSoundboard, useSoundboard, soundboardClips; RoomQuality, useCallQuality, callQuality (+ unit tests). - Optimistic-write RoomQuality admin UI (no stale-state clobber). - Docs: mark EC fork live across README/FEATURES/TODO/BUGS/TESTING; add D2 manual-test steps. Numeric quality caps are client-cooperative; screenshare/camera permissions are hard-enforced server-side (see LotusGuild/matrix voice-limit-guard). Co-Authored-By: Claude Opus 4.8 --- LOTUS_BUGS.md | 51 ++-- LOTUS_FEATURES.md | 126 ++++++++-- LOTUS_TESTING.md | 55 ++++- LOTUS_TODO.md | 110 +++++---- README.md | 31 +-- src/app/components/CallEmbedProvider.tsx | 2 + src/app/features/call/CallControls.tsx | 50 ++-- src/app/features/call/CallSoundboard.tsx | 220 ++++++++++++++++++ .../common-settings/general/RoomQuality.tsx | 198 ++++++++++++++++ .../features/common-settings/general/index.ts | 1 + .../room-settings/general/General.tsx | 2 + src/app/features/settings/general/General.tsx | 94 ++++++++ src/app/hooks/useCallQuality.ts | 41 ++++ src/app/hooks/useSoundboard.ts | 101 ++++++++ src/app/plugins/call/CallControl.ts | 38 +++ src/app/plugins/call/CallEmbed.ts | 5 + src/app/state/settings.ts | 21 ++ src/app/utils/callQuality.test.ts | 60 +++++ src/app/utils/callQuality.ts | 96 ++++++++ src/app/utils/soundboardClips.ts | 72 ++++++ src/types/matrix/accountData.ts | 4 + src/types/matrix/room.ts | 1 + 22 files changed, 1259 insertions(+), 120 deletions(-) create mode 100644 src/app/features/call/CallSoundboard.tsx create mode 100644 src/app/features/common-settings/general/RoomQuality.tsx create mode 100644 src/app/hooks/useCallQuality.ts create mode 100644 src/app/hooks/useSoundboard.ts create mode 100644 src/app/utils/callQuality.test.ts create mode 100644 src/app/utils/callQuality.ts create mode 100644 src/app/utils/soundboardClips.ts diff --git a/LOTUS_BUGS.md b/LOTUS_BUGS.md index 9338f5a3f..603f15c02 100644 --- a/LOTUS_BUGS.md +++ b/LOTUS_BUGS.md @@ -39,38 +39,31 @@ Implemented and gate-green; confirm each per `LOTUS_TESTING.md`, then delete the ## 🧩 Element Call source-level items β€” now actionable via the fork -> πŸ”± **[EC-FORK]** **UPDATE 2026-06-29: the fork is live.** We now own and +> πŸ”± **[EC-FORK]** **UPDATE 2026-06-30: Phase 2 IMPLEMENTED.** We own and > self-build Element Call (`LotusGuild/element-call` β†’ -> `@lotusguild/element-call-embedded`, Phase 1 done & cinny wired). A5/A6/A7 -> below are **no longer "won't fix"** β€” they are ordinary source changes. See -> [`HANDOFF_ELEMENT_CALL_FORK.md`](./HANDOFF_ELEMENT_CALL_FORK.md) Β§10 + the Phase -> 2 work list. (The iframe is **same-origin** / self-hosted; the old blocker was -> that we didn't own EC's compiled source β€” which we now do.) +> `@lotusguild/element-call-embedded@0.20.1-lotus.1`, cinny wired). A5/A6/A7 +> below are **fixed in the fork** β€” they are now ⚠️ awaiting **live +> verification** (`LOTUS_TESTING.md` Β§D2), not open work. See +> [`HANDOFF_ELEMENT_CALL_FORK.md`](./HANDOFF_ELEMENT_CALL_FORK.md) Β§10. Delete each +> row once verified live. -The in-call participant grid is rendered **inside EC's app**. Previously a -pre-built npm bundle we could only style/place around; now editable source. -Items from testing, with their fork-level fix path: +The in-call participant grid is rendered **inside EC's app** β€” now editable source +(previously a prebuilt npm bundle we could only style around). Status of the items +from testing: -- **A5 β€” "Focus camera":** EC supports native tile-pinning. Our bottom-bar "Focus - camera" is a programmatic wrapper that **`.click()`s the tile** today - (`CallControl.ts` `focusCameraParticipant`), and during a screenshare EC - spotlights the shared screen so a camera pin may not override it. **Fork fix:** - add an `io.lotus.focus_participant` widget action that pins a participant in - EC's layout (coexisting with / overriding the screenshare spotlight); cinny - sends it via the widget API and the DOM-click hack is deleted. _Status: Open β€” - Actionable (Phase 2)._ -- **A6 β€” avatar decorations in-call:** decorations render on **our** pre-join - lobby roster (`CallMemberCard`) but not on EC's in-call video tiles. **Fork - fix:** render the decoration APNG inside EC's participant-tile component, fed - decoration slugs via widget member data. _Status: Open β€” Actionable (Phase 2)._ -- **A7 β€” mic dead after EC's "Reconnect":** the mid-call "Connection lost / - Reconnect" screen is **EC's own** (our load watchdog only covers an initial - hung load). After EC reconnects, the mic isn't re-published through our denoise - `getUserMedia` shim until a clean End+rejoin. **Fork fix:** move denoise into - EC's mic-capture/publish pipeline as a first-class audio stage β€” EC re-runs it - on every (re)publish, so reconnects keep denoise alive natively, and the - build-time `index.html` injection is removed. _Status: Open β€” Actionable - (Phase 2); root cause is the `getUserMedia` monkeypatch, not EC itself._ +- **A5 β€” "Focus camera": ⚠️ FIXED in fork, awaiting verify (D2-3).** cinny now + sends an `io.lotus.focus_participant` widget action that pins a participant in + EC's layout (coexisting with / overriding the screenshare spotlight); the old + `.click()`-the-tile DOM hack in `CallControl.ts` is deleted. +- **A6 β€” avatar decorations in-call: ⚠️ FIXED in fork, awaiting verify (D2-4).** + cinny pushes `io.lotus.decorations` (per-user APNG URLs) and the fork renders + them on EC's participant video-tile avatars β€” not just our pre-join lobby roster. +- **A7 β€” mic dead after EC's "Reconnect": ⚠️ FIXED in fork, awaiting verify + (D2-1).** Denoise moved into EC's mic-capture/publish pipeline as a first-class + LiveKit `TrackProcessor` (flag `lotusDenoiseSource=1`); EC re-runs it on every + (re)publish, so reconnects keep denoise alive natively. The build-time + `getUserMedia`/`index.html` injection (the root cause) is removed. **Highest + blast radius β€” everyone's mic; verify D2-1 carefully.** --- diff --git a/LOTUS_FEATURES.md b/LOTUS_FEATURES.md index b03ff2a3f..25a699c95 100644 --- a/LOTUS_FEATURES.md +++ b/LOTUS_FEATURES.md @@ -322,14 +322,104 @@ Users can set a custom background color for `@mention` chips that highlight thei ## Voice / Video Call Improvements -> πŸ”± **[EC-FORK]** Element Call is embedded as a **pre-built npm bundle** today. -> The plan to fork & self-build it from source for true ownership β€” and which of -> the items below would move into our EC source β€” is in +> πŸ”± **[EC-FORK] LIVE (2026-06).** Element Call is now our **self-built fork** +> (`@lotusguild/element-call-embedded@0.20.1-lotus.1`, source at +> `LotusGuild/element-call`), served same-origin β€” no longer the upstream +> pre-built npm bundle. Several in-call behaviors below are now first-class +> source changes rather than DOM/widget hacks. Background, plan, and the Phase-2 +> work list are in > [`HANDOFF_ELEMENT_CALL_FORK.md`](./HANDOFF_ELEMENT_CALL_FORK.md). -### Element Call Upgrade +### Element Call β€” Self-Built Fork (`0.20.1-lotus.1`) -Upgraded embedded Element Call widget from **0.16.3** to **0.19.4**. +The embedded widget was upgraded **0.16.3 β†’ 0.19.4 β†’ 0.20.1**, then **forked**. +We self-build `LotusGuild/element-call` and publish it to our private Gitea npm +registry as `@lotusguild/element-call-embedded`; cinny consumes that instead of +`@element-hq/element-call-embedded`. The iframe prints +`Element Call embedded-v0.20.1-lotus.1` in its console (vs. `embedded-v0.20.1` +upstream) β€” the quickest way to confirm a deploy landed the fork. + +All custom behavior lives in the fork's `src/lotus/` modules and is **additive +and dormant by default**, gated by URL flags / widget actions the host opts into, +so a stock EC config is byte-for-byte upstream behavior. + +**Active (cinny drives them today):** + +| # | Feature | Mechanism | Replaces (old hack) | +| --- | --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | +| A7 | **Denoise in-source** | ML noise suppression runs inside EC as a LiveKit `TrackProcessor