feat(call): in-call soundboard, quality controls, room call-permissions
Element Call is now consumed as our self-built fork (@lotusguild/element-call-embedded); wire up its previously-dormant capabilities and document the fork as live. Soundboard (P5-15): a call-bar button plays user-uploaded audio clips into the call as a real published track (io.lotus.inject_audio) plus local playback. Clips are uploadable like emoji/sticker packs, stored in io.lotus.soundboard account data (synced across devices). Gated by a Settings toggle + volume. Quality controls (P5-31): per-user mic/screenshare bitrate + screenshare framerate (Settings -> Calls), applied via io.lotus.set_quality clamped to any room cap. Room admins set caps and hard call-permissions (allow_screenshare / allow_camera) in Room Settings -> Voice; the call bar hides blocked buttons. - New: CallSoundboard, useSoundboard, soundboardClips; RoomQuality, useCallQuality, callQuality (+ unit tests). - Optimistic-write RoomQuality admin UI (no stale-state clobber). - Docs: mark EC fork live across README/FEATURES/TODO/BUGS/TESTING; add D2 manual-test steps. Numeric quality caps are client-cooperative; screenshare/camera permissions are hard-enforced server-side (see LotusGuild/matrix voice-limit-guard). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
+22
-29
@@ -39,38 +39,31 @@ Implemented and gate-green; confirm each per `LOTUS_TESTING.md`, then delete the
|
||||
|
||||
## 🧩 Element Call source-level items — now actionable via the fork
|
||||
|
||||
> 🔱 **[EC-FORK]** **UPDATE 2026-06-29: the fork is live.** We now own and
|
||||
> 🔱 **[EC-FORK]** **UPDATE 2026-06-30: Phase 2 IMPLEMENTED.** We own and
|
||||
> self-build Element Call (`LotusGuild/element-call` →
|
||||
> `@lotusguild/element-call-embedded`, Phase 1 done & cinny wired). A5/A6/A7
|
||||
> below are **no longer "won't fix"** — they are ordinary source changes. See
|
||||
> [`HANDOFF_ELEMENT_CALL_FORK.md`](./HANDOFF_ELEMENT_CALL_FORK.md) §10 + the Phase
|
||||
> 2 work list. (The iframe is **same-origin** / self-hosted; the old blocker was
|
||||
> that we didn't own EC's compiled source — which we now do.)
|
||||
> `@lotusguild/element-call-embedded@0.20.1-lotus.1`, cinny wired). A5/A6/A7
|
||||
> below are **fixed in the fork** — they are now ⚠️ awaiting **live
|
||||
> verification** (`LOTUS_TESTING.md` §D2), not open work. See
|
||||
> [`HANDOFF_ELEMENT_CALL_FORK.md`](./HANDOFF_ELEMENT_CALL_FORK.md) §10. Delete each
|
||||
> row once verified live.
|
||||
|
||||
The in-call participant grid is rendered **inside EC's app**. Previously a
|
||||
pre-built npm bundle we could only style/place around; now editable source.
|
||||
Items from testing, with their fork-level fix path:
|
||||
The in-call participant grid is rendered **inside EC's app** — now editable source
|
||||
(previously a prebuilt npm bundle we could only style around). Status of the items
|
||||
from testing:
|
||||
|
||||
- **A5 — "Focus camera":** EC supports native tile-pinning. Our bottom-bar "Focus
|
||||
camera" is a programmatic wrapper that **`.click()`s the tile** today
|
||||
(`CallControl.ts` `focusCameraParticipant`), and during a screenshare EC
|
||||
spotlights the shared screen so a camera pin may not override it. **Fork fix:**
|
||||
add an `io.lotus.focus_participant` widget action that pins a participant in
|
||||
EC's layout (coexisting with / overriding the screenshare spotlight); cinny
|
||||
sends it via the widget API and the DOM-click hack is deleted. _Status: Open —
|
||||
Actionable (Phase 2)._
|
||||
- **A6 — avatar decorations in-call:** decorations render on **our** pre-join
|
||||
lobby roster (`CallMemberCard`) but not on EC's in-call video tiles. **Fork
|
||||
fix:** render the decoration APNG inside EC's participant-tile component, fed
|
||||
decoration slugs via widget member data. _Status: Open — Actionable (Phase 2)._
|
||||
- **A7 — mic dead after EC's "Reconnect":** the mid-call "Connection lost /
|
||||
Reconnect" screen is **EC's own** (our load watchdog only covers an initial
|
||||
hung load). After EC reconnects, the mic isn't re-published through our denoise
|
||||
`getUserMedia` shim until a clean End+rejoin. **Fork fix:** move denoise into
|
||||
EC's mic-capture/publish pipeline as a first-class audio stage — EC re-runs it
|
||||
on every (re)publish, so reconnects keep denoise alive natively, and the
|
||||
build-time `index.html` injection is removed. _Status: Open — Actionable
|
||||
(Phase 2); root cause is the `getUserMedia` monkeypatch, not EC itself._
|
||||
- **A5 — "Focus camera": ⚠️ FIXED in fork, awaiting verify (D2-3).** cinny now
|
||||
sends an `io.lotus.focus_participant` widget action that pins a participant in
|
||||
EC's layout (coexisting with / overriding the screenshare spotlight); the old
|
||||
`.click()`-the-tile DOM hack in `CallControl.ts` is deleted.
|
||||
- **A6 — avatar decorations in-call: ⚠️ FIXED in fork, awaiting verify (D2-4).**
|
||||
cinny pushes `io.lotus.decorations` (per-user APNG URLs) and the fork renders
|
||||
them on EC's participant video-tile avatars — not just our pre-join lobby roster.
|
||||
- **A7 — mic dead after EC's "Reconnect": ⚠️ FIXED in fork, awaiting verify
|
||||
(D2-1).** Denoise moved into EC's mic-capture/publish pipeline as a first-class
|
||||
LiveKit `TrackProcessor` (flag `lotusDenoiseSource=1`); EC re-runs it on every
|
||||
(re)publish, so reconnects keep denoise alive natively. The build-time
|
||||
`getUserMedia`/`index.html` injection (the root cause) is removed. **Highest
|
||||
blast radius — everyone's mic; verify D2-1 carefully.**
|
||||
|
||||
---
|
||||
|
||||
|
||||
+111
-15
@@ -322,14 +322,104 @@ Users can set a custom background color for `@mention` chips that highlight thei
|
||||
|
||||
## Voice / Video Call Improvements
|
||||
|
||||
> 🔱 **[EC-FORK]** Element Call is embedded as a **pre-built npm bundle** today.
|
||||
> The plan to fork & self-build it from source for true ownership — and which of
|
||||
> the items below would move into our EC source — is in
|
||||
> 🔱 **[EC-FORK] LIVE (2026-06).** Element Call is now our **self-built fork**
|
||||
> (`@lotusguild/element-call-embedded@0.20.1-lotus.1`, source at
|
||||
> `LotusGuild/element-call`), served same-origin — no longer the upstream
|
||||
> pre-built npm bundle. Several in-call behaviors below are now first-class
|
||||
> source changes rather than DOM/widget hacks. Background, plan, and the Phase-2
|
||||
> work list are in
|
||||
> [`HANDOFF_ELEMENT_CALL_FORK.md`](./HANDOFF_ELEMENT_CALL_FORK.md).
|
||||
|
||||
### Element Call Upgrade
|
||||
### Element Call — Self-Built Fork (`0.20.1-lotus.1`)
|
||||
|
||||
Upgraded embedded Element Call widget from **0.16.3** to **0.19.4**.
|
||||
The embedded widget was upgraded **0.16.3 → 0.19.4 → 0.20.1**, then **forked**.
|
||||
We self-build `LotusGuild/element-call` and publish it to our private Gitea npm
|
||||
registry as `@lotusguild/element-call-embedded`; cinny consumes that instead of
|
||||
`@element-hq/element-call-embedded`. The iframe prints
|
||||
`Element Call embedded-v0.20.1-lotus.1` in its console (vs. `embedded-v0.20.1`
|
||||
upstream) — the quickest way to confirm a deploy landed the fork.
|
||||
|
||||
All custom behavior lives in the fork's `src/lotus/` modules and is **additive
|
||||
and dormant by default**, gated by URL flags / widget actions the host opts into,
|
||||
so a stock EC config is byte-for-byte upstream behavior.
|
||||
|
||||
**Active (cinny drives them today):**
|
||||
|
||||
| # | Feature | Mechanism | Replaces (old hack) |
|
||||
| --- | --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
|
||||
| A7 | **Denoise in-source** | ML noise suppression runs inside EC as a LiveKit `TrackProcessor<Audio>` (flag `lotusDenoiseSource=1`); re-applied on every (re)publish | the build-time `getUserMedia` monkeypatch injected into `index.html` — **removed**. Fixes mic-dead-after-reconnect. |
|
||||
| #2 | **Speaking / mute events** | EC emits `io.lotus.call_state` (throttled); cinny reads speaker + mute state from it (flag `lotusCallState=1`) | scraping EC's DOM for `[data-lk-speaking]` (kept only as fallback) |
|
||||
| A5 | **Focus participant** | host sends `io.lotus.focus_participant` to pin a tile, coexisting with / overriding the screenshare spotlight | the `.click()`-the-tile DOM hack in `CallControl.ts` — **removed** |
|
||||
| #6 | **In-call avatar decorations** | host pushes `io.lotus.decorations` (per-user APNG URLs); the fork renders them on EC's video-tile avatars | previously impossible — decorations only showed on our pre-join lobby roster |
|
||||
| #5 | **Native transparent background** | flag `lotusTransparent=1` makes EC's surface transparent so the host wallpaper shows through | the injected `background:none !important` CSS |
|
||||
|
||||
**Now wired (cinny drives them — ⚠️ awaiting live verification):**
|
||||
|
||||
| # | Capability | Widget action | cinny surface |
|
||||
| ----- | -------------------- | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------- |
|
||||
| P5-15 | **Audio inject** | `io.lotus.inject_audio` — plays a clip into the call as a separately published track | In-Call Soundboard (uploadable clips) — see below |
|
||||
| P5-31 | **Quality controls** | `io.lotus.set_quality` — sets audio/screenshare encoding bitrate/framerate | Call Quality Controls (user settings + room-admin caps) — see below |
|
||||
|
||||
> Both were dormant capabilities; cinny now drives them (armed via
|
||||
> `lotusAudioInject=1`). The **only** EC item still open is the P5-31
|
||||
> **server-side** quality guard (a `voice-limit-guard`-style sidecar reading
|
||||
> `io.lotus.room_quality`) for hard enforcement across all Matrix clients — the
|
||||
> client cap is best-effort.
|
||||
|
||||
### In-Call Soundboard (P5-15)
|
||||
|
||||
A soundboard button (🔔) in the call controls bar opens a popout of the user's
|
||||
clips. Clicking one **injects it into the call as a real published LiveKit
|
||||
track** (every participant hears it, via the fork's `io.lotus.inject_audio`) and
|
||||
plays it locally for the presser (LiveKit doesn't loop your own track back).
|
||||
|
||||
- **User-uploadable, like custom emoji/sticker packs.** Clips are stored in the
|
||||
`io.lotus.soundboard` account data event, so they **sync across all your
|
||||
devices**. Upload short audio (≤ 1 MB, ≤ 40 clips) from the popout; delete
|
||||
inline.
|
||||
- Authenticated media can't be fetched from the widget's realm, so the host
|
||||
resolves each mxc clip → an authenticated download → a same-session `blob:`
|
||||
object URL and hands that to the widget.
|
||||
- Gated by the **Soundboard** toggle (Settings → General → Calls) with a volume
|
||||
slider. The button is hidden when disabled.
|
||||
- Files: `utils/soundboardClips.ts`, `hooks/useSoundboard.ts`,
|
||||
`features/call/CallSoundboard.tsx`, `plugins/call/CallControl.ts#injectAudio`.
|
||||
|
||||
### Call Quality Controls (P5-31)
|
||||
|
||||
Discord-style encoding controls applied to the local tracks via the fork's
|
||||
`io.lotus.set_quality` (`RTCRtpSender.setParameters` across all simulcast
|
||||
encodings, re-applied on every re-publish/reconnect).
|
||||
|
||||
- **User settings** (Settings → General → Calls): Microphone Bitrate,
|
||||
Screenshare Bitrate, Screenshare Framerate (each defaults to **Auto**).
|
||||
- **Room-admin caps**: admins set a ceiling in Room Settings → General → Voice
|
||||
(`io.lotus.room_quality` state event); every Lotus client clamps its per-user
|
||||
quality to `min(user setting, room cap)`.
|
||||
- Applied by the `useCallQuality` hook on join and whenever settings/caps
|
||||
change; `utils/callQuality.ts` builds the payload (unit-tested).
|
||||
|
||||
**Server-enforced call permissions (hard, ALL clients).** The same
|
||||
`io.lotus.room_quality` event carries a **publish-source policy**
|
||||
(`allow_screenshare`, `allow_camera`) enforced server-side by
|
||||
`voice-limit-guard` (matrix repo, LXC 151): it re-signs the LiveKit JWT's
|
||||
`canPublishSources`, so the SFU refuses screenshare/camera tracks for **every**
|
||||
Matrix client (Element, FluffyChat, our fork) — not just Lotus. Admins toggle
|
||||
these in Room Settings → Voice → **Call Permissions**; cinny also hides the
|
||||
blocked buttons in the call bar. Enforcement is **live**: the JWT re-sign covers
|
||||
new joins, and a background reconcile loop revokes an **in-progress**
|
||||
screenshare/camera (via LiveKit `UpdateParticipant`) within ~3 s of an admin
|
||||
flipping the policy — so it kills active shares mid-call, not just future ones.
|
||||
|
||||
- **Why numeric caps aren't server-enforced:** LiveKit is a pure SFU (forwards,
|
||||
never transcodes) and has no publisher bitrate/fps field anywhere in the JWT
|
||||
grant, room config, server `limit:`, or admin API; stock Element Call ignores
|
||||
room metadata for publish quality. Numeric caps are therefore inherently
|
||||
**cooperative** — our fork honors them, which is the design above. The
|
||||
publish-source policy is the one genuine hard, cross-client lever, and it's
|
||||
implemented.
|
||||
- **Not yet**: screenshare resolution control (needs a `getDisplayMedia` hook in
|
||||
the fork).
|
||||
|
||||
### Camera Default Off
|
||||
|
||||
@@ -431,20 +521,26 @@ A comprehensive mic noise-suppression system in **Settings → General → Calls
|
||||
- **Support Detection:** UI now detects `AudioWorklet` / `AudioContext` support and disables ML options in unsupported environments.
|
||||
- **Status Reporting:** The ML shim notifies the host app via `postMessage`. If initialization fails, a system toast alerts the user of the fallback to the raw microphone.
|
||||
|
||||
**Open-Source Model Roadmap:**
|
||||
| Model | Transients (Clicks) | Voice Quality | CPU Usage (WASM) |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| **RNNoise** | Poor | Moderate | < 5% |
|
||||
| **DTLN** | Good | High | 10-20% |
|
||||
| **DeepFilterNet 3** | **Excellent** | **Very High** | 25-50%+ |
|
||||
**Open-Source Models (all now in-source in the EC fork):**
|
||||
| Model | Transients (Clicks) | Voice Quality | CPU Usage (WASM) | Sample rate |
|
||||
| :--- | :--- | :--- | :--- | :--- |
|
||||
| **RNNoise** (default) | Poor | Moderate | < 5% | 48 kHz |
|
||||
| **Speex** | Poor | Low | < 5% | 48 kHz |
|
||||
| **DTLN** | Good | High | 10-20% | 16 kHz |
|
||||
| **DeepFilterNet 3** | **Excellent** | **Very High** | 25-50%+ | 48 kHz |
|
||||
|
||||
> **Note:** DeepFilterNet 3 is planned for future inclusion in the desktop build where larger binaries and higher CPU overhead are more acceptable.
|
||||
> **Update (2026-06):** with the EC fork live, denoise runs **inside** Element
|
||||
> Call as a LiveKit `TrackProcessor` and **all four models ship in-source**
|
||||
> (DTLN at 16 kHz, the rest at 48 kHz; the processor degrades to the raw mic
|
||||
> rather than ever going silent). The model picker selects between them. Real-call
|
||||
> **audio-quality** comparison across models is still the open verification item
|
||||
> (RNNoise output is known to be weak) — see `LOTUS_TESTING.md` §D2-1.
|
||||
|
||||
### Files
|
||||
|
||||
- `build/lotus-denoise.js` — multi-model getUserMedia shim
|
||||
- `vite.config.js` — `lotusDenoise()` plugin (copies assets for RNNoise, Speex, and NoiseGate)
|
||||
- `src/app/plugins/call/CallEmbed.ts` — advanced tier → widget URL params
|
||||
- **EC fork** `src/lotus/lotusDenoise.ts` + `lotusDenoiseProcessor.ts` — in-source LiveKit `TrackProcessor` (RNNoise/Speex 48 kHz, DTLN 16 kHz, DeepFilterNet 48 kHz); activated by `lotusDenoiseSource=1`. (The old build-time `getUserMedia` shim `build/lotus-denoise.js` is **removed**.)
|
||||
- `vite.config.js` — `lotusDenoise()` plugin (now only **copies model assets** for the fork to load; no longer injects a shim)
|
||||
- `src/app/plugins/call/CallEmbed.ts` — advanced tier → `lotusDenoiseSource` widget URL param
|
||||
- `src/app/utils/lotusDenoiseUtils.ts` — support detection and model comparison metadata
|
||||
- `src/app/features/settings/general/General.tsx` — advanced settings UI + mic meter
|
||||
|
||||
|
||||
+51
-4
@@ -267,12 +267,59 @@ Flag: `lotusTransparent=1` (native, replacing the injected `background:none !imp
|
||||
- [ ] Call background looks right — host wallpaper/surface shows through; **no** black box, bad
|
||||
see-through, or layout breakage (also covered loosely by §D2 "looks right").
|
||||
|
||||
### D2-6. Dormant features — confirm they do NOTHING (no regression)
|
||||
### D2-7. In-Call Soundboard (#3 / P5-15) — 👥 2 people — **NEW**
|
||||
|
||||
EC ships the capability but cinny has **no UI** to trigger them yet:
|
||||
Flag: `lotusAudioInject=1`. A 🔔 **Soundboard** button now sits in the call controls bar (left group,
|
||||
next to the chat button). Clips are user-uploadable and sync across your devices like emoji packs.
|
||||
_Prereq:_ Settings → General → Calls → **Soundboard** must be ON (default on).
|
||||
|
||||
- [ ] **Soundboard audio-inject (#3)** and **quality controls (#7)** — there should be no new UI and no
|
||||
effect. (Nothing to test; noted so a tester doesn't go hunting.)
|
||||
- [ ] **Upload:** open the soundboard popout → **Upload** → pick a short audio file (mp3/ogg/wav, ≤ 1 MB).
|
||||
It appears as a clip tile. (Too-big / too-many shows an error, doesn't crash.)
|
||||
- [ ] **Plays into the call:** with a second person in the call, click a clip. **They hear it**, and
|
||||
**you hear it locally** too. ✅ good if both hear it; ❌ tell us if only one side does.
|
||||
- [ ] **Sync:** the uploaded clip shows up on your **other device**/session (account-data sync).
|
||||
- [ ] **Delete:** the ✕ on a tile removes it (everywhere, after sync).
|
||||
- [ ] **Off switch:** turn Settings → Calls → **Soundboard** off → the call-bar button disappears.
|
||||
- [ ] Injecting a clip does **not** mute/interrupt your mic or anyone else's audio.
|
||||
|
||||
### D2-8. Call Quality Controls (#7 / P5-31) — 👥 2 people — **NEW**
|
||||
|
||||
Action: `io.lotus.set_quality`. User settings in **Settings → General → Calls** (Microphone Bitrate,
|
||||
Screenshare Bitrate, Screenshare Framerate; all default **Auto**). Admin caps in **Room Settings →
|
||||
General → Voice → Call Quality Caps**.
|
||||
|
||||
- [ ] **No regression at Auto:** with everything on **Auto**, calls/screenshare work exactly as before.
|
||||
- [ ] **User cap takes effect:** set Microphone Bitrate to **32 kbps**, rejoin/continue a call — audio
|
||||
still flows (thinner is fine). Set Screenshare Framerate to **15 fps** and share your screen — it
|
||||
still shares. ❌ tell us if any setting kills audio/screenshare.
|
||||
- [ ] **Applies mid-call:** changing a setting **during** a call takes effect without End+rejoin.
|
||||
- [ ] **Room-admin cap (admin needed):** as a room admin, set **Max Microphone Bitrate = 64 kbps** in
|
||||
Room Settings → Voice. A member whose user setting is higher (e.g. 256) should be **clamped to 64**
|
||||
(best-effort/UX — this is client-side; hard server enforcement is a separate follow-up).
|
||||
- [ ] Resetting a setting back to **Auto** removes the cap for the rest of the call.
|
||||
|
||||
> Soundboard + quality are no longer "dormant" — if either does nothing, grab the **EC iframe console**
|
||||
> and check for `io.lotus.inject_audio` / `io.lotus.set_quality` rejections.
|
||||
|
||||
### D2-9. Call Permissions — HARD server-side, cross-client (👥 2 people, admin) — **NEW**
|
||||
|
||||
This is enforced by the `voice-limit-guard` on the server (re-signs the LiveKit JWT), so it applies to
|
||||
**every** client, not just Lotus Chat. Set in **Room Settings → General → Voice → Call Permissions**.
|
||||
_(Requires the guard deployed on LXC 151 — auto-deploys on a `matrix` repo push.)_
|
||||
|
||||
- [ ] **Disable screenshare:** as admin, turn **Allow Screen Sharing** off. In a call, the
|
||||
**screenshare button disappears** in Lotus Chat. ✅ good if no one can screenshare.
|
||||
- [ ] **Cross-client (the important one):** have someone join the **same room from stock Element / Element
|
||||
X** and try to screenshare → the server **refuses** the track (it won't publish). This proves it's
|
||||
not just our client hiding a button.
|
||||
- [ ] **Audio-only room:** turn **Allow Camera** off too → the camera button disappears and cameras are
|
||||
server-blocked for all clients; **microphones still work**.
|
||||
- [ ] **⭐ Live kill (mid-call):** while someone is **actively screensharing**, an admin turns **Allow
|
||||
Screen Sharing** off. Within a few seconds their screenshare should **stop for everyone** on its own
|
||||
(no rejoin needed) — this is the server reconcile loop revoking it live. Works even if the sharer is
|
||||
on stock Element. ✅ good if the share drops within ~3–5 s; ❌ tell us if it keeps going.
|
||||
- [ ] **Turning it back on** restores the ability to screenshare/camera (start a new share).
|
||||
- [ ] **No policy = no change:** a room with Call Permissions left on defaults behaves exactly as before.
|
||||
|
||||
> If any D2 item fails, grab the **EC iframe console** (right-click the call → inspect the iframe) — a
|
||||
> widget-action/payload mismatch shows up there as a `io.lotus.*` rejection or a `MissingKey`/transport log.
|
||||
|
||||
+41
-21
@@ -48,6 +48,9 @@ Built and gate-green; verify per [LOTUS_TESTING.md](./LOTUS_TESTING.md), then th
|
||||
| Desktop — proactive update notifications (Tauri) | J1 |
|
||||
| Remind Me Later | K1 |
|
||||
| Mobile Bookmarks access | E5 |
|
||||
| In-Call Soundboard (P5-15, uploadable clips → real call inject) | D2-7 |
|
||||
| Call Quality Controls (P5-31, user + room-admin caps) | D2-8 |
|
||||
| Call Permissions (P5-31, hard server-side screenshare/camera policy) | D2-9 |
|
||||
|
||||
---
|
||||
|
||||
@@ -73,7 +76,7 @@ Status: `[ ]` pending · `[~]` in progress · `[x]` completed
|
||||
### Confirmed facts
|
||||
|
||||
| Finding | Impact |
|
||||
| --------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ |
|
||||
| -------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
|
||||
| **MSC flags ON:** `msc4140` · `msc3771` · `msc3440.stable` · `msc4133.stable` · `simplified_msc3575` · `msc4222` · `msc3266` · `msc3401_matrix_rtc` | All safe to use now |
|
||||
| **MSC flags OFF:** `msc4306` (thread subscriptions) · `msc3882` · `msc3912` · `msc4155` | These features are BLOCKED |
|
||||
| **MSC3266** room summary: flag `msc3266_enabled: true` set but `GET /v1/rooms/{id}/summary` still returns 404 (M_UNRECOGNIZED) | Room Preview BLOCKED — endpoint not implemented in Synapse 1.155 |
|
||||
@@ -93,7 +96,7 @@ Status: `[ ]` pending · `[~]` in progress · `[x]` completed
|
||||
| `knockSupported()` utility exists at `matrix.ts:376-391` | Knock UX: only need "Request to Join" in `RoomIntro.tsx` |
|
||||
| `KeywordMessages.tsx` already has custom keyword push rules | Full push rule editor: only non-keyword rule types need new UI |
|
||||
| `getMatrixToRoom()` in `matrix-to.ts` generates invite URLs | Invite link: just add QR code to room settings |
|
||||
| Cindy CANNOT inject audio into EC call stream | In-call soundboard must be redesigned as local-only |
|
||||
| ~~Cindy CANNOT inject audio into EC call stream~~ **UNBLOCKED by EC fork** — `io.lotus.inject_audio` widget action publishes a clip as a real call track | In-call soundboard CAN now mix into the call (no longer local-only); needs cinny UI to drive the action |
|
||||
| Folds uses vanilla-extract in non-TDS, NOT CSS custom properties | Custom accent color: must create new vanilla-extract theme variant dynamically |
|
||||
| Theme presets need ~50 CSS custom properties each | Significant design work before coding |
|
||||
| `useCallSpeakers.ts` CSS MutationObserver polling | Visual speaking indicator: TDS ring animation on top of existing data |
|
||||
@@ -266,12 +269,17 @@ Features:
|
||||
|
||||
---
|
||||
|
||||
### [ ] P5-15 · In-Call Soundboard
|
||||
### [~] P5-15 · In-Call Soundboard — IMPLEMENTED (⚠️ awaiting live verification, D2-7)
|
||||
|
||||
**What:** Grid of short audio clips playable into the call audio stream via Web Audio API (AudioBufferSourceNode → MediaStreamDestinationNode → mixed with mic). Built-in clips + user-uploadable custom clips (stored as mxc://). Accessible from call controls bar.
|
||||
**[AUDIT REQUIRED]** Verify the Element Call integration exposes the mic MediaStream for mixing. This is the highest-risk part of this feature.
|
||||
**🔱 [EC-FORK]** Owning the EC source (see [`HANDOFF_ELEMENT_CALL_FORK.md`](./HANDOFF_ELEMENT_CALL_FORK.md)) would unblock real audio-injection — a proper soundboard mixed into the call — which is impossible against the prebuilt bundle today.
|
||||
**Complexity:** High.
|
||||
**What:** Soundboard button in the call controls bar → popout grid of the user's clips; clicking one plays it **into the call** as a real published track (peers hear it) and locally (presser hears it). Clips are **user-uploadable, just like custom emojis/stickers**.
|
||||
**🔱 [EC-FORK] Fork side + cinny side DONE.** The fork ships `io.lotus.inject_audio` (`LotusWidgetActions.InjectAudio`, allow-listed in `widget.ts`), armed via the `lotusAudioInject=1` flag; it publishes a clip as a separate LiveKit track — a **real** in-call soundboard mixed into the call, not local-only. cinny now drives it.
|
||||
**Shipped (cinny):**
|
||||
|
||||
- Clips stored in `io.lotus.soundboard` account data → **synced across devices like emoji/sticker packs** (`useSoundboard` hook; `AccountDataEvent.LotusSoundboard`).
|
||||
- Upload audio (≤1 MB, ≤40 clips) → `mx.uploadContent` → mxc; play resolves mxc → authed download → `blob:` object URL (the widget can't fetch authenticated media itself) → `control.injectAudio(url, volume)` + local playback.
|
||||
- `CallSoundboard.tsx` popout in the call bar (upload / play / delete), gated on the `soundboardEnabled` setting (Settings → General → Calls, + volume slider).
|
||||
**Remaining:** a dedicated Settings management page (optional — upload/delete already live in the popout); a small default clip set; live verification (D2-7). Files: `utils/soundboardClips.ts`, `hooks/useSoundboard.ts`, `features/call/CallSoundboard.tsx`, `plugins/call/CallControl.ts#injectAudio`.
|
||||
**Complexity:** Medium — done.
|
||||
|
||||
---
|
||||
|
||||
@@ -287,26 +295,38 @@ Features:
|
||||
### [x] P5-30 · Advanced ML Noise Suppression (Krisp-style)
|
||||
|
||||
**What:** High-end background noise cancellation using a pre-trained ML model (RNNoise) running in the browser. Removes dogs, fans, and keyboard clicks from the mic stream.
|
||||
**Shipped:** 3-tier setting (Off / Browser-native / ML) in Settings → General → Calls. ML tier injects a same-origin pre-init shim into the vendored Element Call `index.html` that monkeypatches `getUserMedia` and routes the captured mic through an RNNoise `AudioWorklet` before LiveKit publishes — no EC fork required. See LOTUS_FEATURES.md → "Noise Suppression (Advanced Multi-Tier)".
|
||||
**🔱 [EC-FORK]** Once we own the EC source (see [`HANDOFF_ELEMENT_CALL_FORK.md`](./HANDOFF_ELEMENT_CALL_FORK.md)), denoise should become a first-class audio stage **inside** EC instead of an `index.html` getUserMedia monkeypatch — more robust, survives reconnects (fixes the A7 mic-after-reconnect bug), and removes the build-time injection hack.
|
||||
**Key decision:** LiveKit's Krisp filter is LiveKit-Cloud-only (we self-host the SFU); EC's own RNNoise PR #3892 is unmerged. The shim is the same post-capture pipeline #3892 uses, executed from the realm we control, so it survives EC version bumps.
|
||||
**AEC note (resolved-as-accepted):** WebAudio capture routing can weaken browser AEC — same tradeoff as EC's upstream feature; mitigated by keeping `echoCancellation`/`autoGainControl` on the raw capture and labeling the tier "beta".
|
||||
**Shipped:** 3-tier setting (Off / Browser-native / ML) in Settings → General → Calls.
|
||||
**🔱 [EC-FORK] DONE — moved in-source (2026-06).** ML denoise is now a first-class audio stage **inside** the forked Element Call: a LiveKit `TrackProcessor<Audio>` activated by `lotusDenoiseSource=1` (cinny sets it when ML is selected). The old build-time `getUserMedia`/`index.html` monkeypatch is **removed**. Because EC re-runs the processor on every (re)publish, denoise now **survives reconnects and mic-device switches** — this is the A7 fix (see `LOTUS_BUGS.md` A7, `LOTUS_TESTING.md` §D2-1). The processor degrades to the raw mic rather than going silent.
|
||||
**Key decision:** LiveKit's Krisp filter is LiveKit-Cloud-only (we self-host the SFU); EC's own RNNoise PR #3892 is unmerged. Owning the fork let us implement the in-source stage directly.
|
||||
|
||||
**Model Roadmap (priority order):**
|
||||
**Models — all in-source in the fork:**
|
||||
|
||||
- [ ] **Verify DTLN** (16 kHz narrowband fix) in a real call before investing further — wired but unverified.
|
||||
- [ ] **DeepFilterNet 3** — best self-hostable upgrade: Rust→WASM, CPU real-time, 48 kHz fullband. Effort: self-host `df_bg.wasm` + DFN3 ONNX model, wire a 48 kHz worklet.
|
||||
- [ ] **Desktop-only / HW-gated:** FRCRN or NVIDIA Maxine (RTX/Tensor only) — impossible in-browser; would run in Tauri Rust backend + bridge a virtual mic into the webview. Must detect capability and only offer on supported hardware; web falls back to RNNoise.
|
||||
- [x] **RNNoise** (48 kHz, default) · **Speex** (48 kHz) · **DTLN** (16 kHz) · **DeepFilterNet 3** (48 kHz) — all four wired and selectable.
|
||||
- [ ] **Open verification:** real-call **audio-quality** comparison across the four models (RNNoise output is known-weak). Track under the denoise quality project, `LOTUS_TESTING.md` §D2-1 / J2.
|
||||
- [ ] **Desktop-only / HW-gated (future):** FRCRN or NVIDIA Maxine (RTX/Tensor only) — impossible in-browser; would run in the Tauri Rust backend + bridge a virtual mic into the webview. Detect capability; web falls back to RNNoise.
|
||||
- **Excluded:** Krisp (LiveKit Cloud only); FRCRN/Maxine on web (GPU/server-bound).
|
||||
|
||||
---
|
||||
|
||||
### [ ] P5-31 · Granular Voice & Screenshare Quality Controls (Discord-style)
|
||||
### [~] P5-31 · Granular Voice & Screenshare Quality Controls — IMPLEMENTED (⚠️ awaiting live verification, D2-8)
|
||||
|
||||
**What:** Let users (or room admins via room settings) adjust audio bitrates (e.g., 64kbps to 512kbps) and screenshare quality (resolution: 720p/1080p/Source, framerate: 15/30/60fps).
|
||||
**Note:** Requires tight integration with the LiveKit SFU and custom state events for per-room quality caps.
|
||||
**[AUDIT REQUIRED]** Must verify if current `lk-jwt-service` can be extended with custom bitrate/resolution claims or if a new sidecar (similar to `voice-limit-guard`) is needed for server-side enforcement.
|
||||
**Complexity:** Extreme.
|
||||
**What:** Let users (and room admins) adjust audio bitrate and screenshare bitrate/framerate.
|
||||
**🔱 [EC-FORK] Fork side + client side DONE.** The fork ships `io.lotus.set_quality` (`LotusWidgetActions.SetQuality`) that applies audio/screenshare encoding params (`RTCRtpSender.setParameters`, all simulcast encodings, re-applied on `TrackUnmuted`/republish) inside EC. cinny now drives it.
|
||||
|
||||
**Shipped (cinny):**
|
||||
|
||||
1. **User settings** (Settings → General → Calls): Microphone Bitrate, Screenshare Bitrate, Screenshare Framerate (`callAudioBitrate` / `screenshareBitrate` / `screenshareFramerate`).
|
||||
2. **Room-admin caps**: `io.lotus.room_quality` state event (`StateEvent.LotusRoomQuality`) + `RoomQuality.tsx` in Room Settings → General → Voice (mirrors `RoomVoiceLimit`).
|
||||
3. **Apply logic**: `useCallQuality` (wired in `CallEmbedProvider`'s `CallUtils`) builds `min(user setting, room cap)` and sends `io.lotus.set_quality` on join / when settings change (`utils/callQuality.ts`, unit-tested).
|
||||
|
||||
**Server-side enforcement (DONE — matrix repo):** extended `voice-limit-guard.py` (LXC 151) to also read `io.lotus.room_quality` and hard-enforce a **publish-source policy** for ALL clients.
|
||||
|
||||
- **Reality (researched, primary-source, LiveKit 1.9.11):** numeric bitrate/fps caps **cannot** be hard-enforced server-side — LiveKit is a pure SFU (forwards, never transcodes); there is NO bitrate/fps field in the JWT grant, `RoomConfiguration`, server `limit:` config, or any admin RPC, and stock Element Call ignores room metadata / custom claims for publish quality. So numeric caps stay **cooperative** (our fork honors them via `min()` → `set_quality`, already shipped).
|
||||
- **What IS hard-enforced cross-client:** `VideoGrant.canPublishSources`. The guard holds the LiveKit secret, so when `io.lotus.room_quality` sets `allow_screenshare:false` / `allow_camera:false` it re-signs the issued JWT with a narrowed source list → the SFU refuses those tracks for **every** client (Element, FluffyChat, our fork). Mic always kept. Fail-open; unit-tested (`livekit/test_voice_limit_guard.py`). Admin UI: Room Settings → Voice → **Call Permissions** switches. cinny also hides the blocked buttons.
|
||||
- **Live (mid-call) enforcement — DONE:** the JWT re-sign covers new joins; for participants **already in the call**, a background reconcile loop in the guard calls LiveKit `UpdateParticipant` every ~3 s to narrow `canPublishSources`, which unpublishes an in-progress screenshare/camera **server-side for all clients** and blocks re-publish (verified LiveKit 1.9.11 auto-unpublishes on permission narrowing). Only removes forbidden sources (never grants), preserves other permission flags, no-ops once compliant. So flipping a room audio-only kills live cameras/screenshares within ~one interval.
|
||||
- **Not enforceable / deferred:** numeric server enforcement (impossible — see above); screenshare **resolution** control (`set_quality` covers bitrate + framerate; resolution needs a `getDisplayMedia` hook inside the fork).
|
||||
|
||||
**Complexity:** DONE — client (cooperative numeric caps) + server (hard publish-source policy). Only the physically-impossible numeric server enforcement is out of scope.
|
||||
|
||||
---
|
||||
|
||||
@@ -540,7 +560,7 @@ Exhaustive, low-level implementation details for backlog items. Follow these pat
|
||||
|
||||
> ⚠️ **[Gemini_Found — CORRECTED]** Gemini originally suggested using LiveKit's `LocalAudioTrack.replaceTrack()` to mix audio into the call stream. This is **not possible** from Lotus Chat's realm: Element Call runs in a **cross-origin iframe** controlled via `matrix-widget-api` (postMessage). LiveKit's JS SDK and its `LocalAudioTrack` live inside EC's sandboxed context — inaccessible from our code. This directly contradicts the confirmed constraint already listed in the Server Capabilities table: _"Cindy CANNOT inject audio into EC call stream — In-call soundboard must be redesigned as local-only."_ The soundboard must be a local-playback-only feature (output through the user's speakers, not mixed into the call audio stream).
|
||||
>
|
||||
> 🔱 **[EC-FORK — partial correction]** The "cross-origin" claim above is **outdated**: EC is now **same-origin** / self-hosted (`iframe.sandbox` has `allow-same-origin`; we read `contentDocument`). The _practical_ blocker still holds — LiveKit's `LocalAudioTrack` lives in EC's **module scope** (not on `window`), so it's unreachable from cinny even same-origin. **Owning the EC source** (see [`HANDOFF_ELEMENT_CALL_FORK.md`](./HANDOFF_ELEMENT_CALL_FORK.md)) is the path to a real call-audio-inject API, which would unblock a true in-call soundboard.
|
||||
> 🔱 **[EC-FORK — RESOLVED]** Both the original claim and the earlier "practical blocker still holds" correction are now **outdated**. EC is same-origin **and** we own the source, so we no longer reach into EC's module scope from cinny — instead the fork **exposes the inject point itself**: the `io.lotus.inject_audio` widget action (`LotusWidgetActions.InjectAudio`) publishes a clip as a separate LiveKit track from inside EC. A **real** in-call soundboard (mixed into the call, not local-only) is therefore unblocked; only the cinny-side UI remains (see P5-15 above). The capability ships dormant today.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -144,22 +144,25 @@ The source code lives in `/root/code/cinny`. All changes should be made on the `
|
||||
|
||||
See [LOTUS_FEATURES.md](LOTUS_FEATURES.md) for the full feature changelog and [LOTUS_TODO.md](LOTUS_TODO.md) for the work backlog.
|
||||
|
||||
### 🔱 Planned: Element Call fork ("Lotus Call")
|
||||
### 🔱 Element Call fork ("Lotus Call") — LIVE
|
||||
|
||||
Voice/video channels embed **Element Call**. Today it's a **pre-built npm bundle**
|
||||
(`@element-hq/element-call-embedded` 0.20.1) copied to `public/element-call/` and
|
||||
served same-origin; we steer it via the `matrix-widget-api` plus fragile DOM
|
||||
hacks. Because we don't own its compiled source, several in-call issues (avatar
|
||||
decorations on tiles, camera focus/fullscreen during screenshare, mic recovery
|
||||
after reconnect, native theming, real call-audio injection) are unfixable from
|
||||
outside.
|
||||
Voice/video channels embed **Element Call**, which is now our **self-built fork**
|
||||
(`@lotusguild/element-call-embedded` `0.20.1-lotus.1`, source at
|
||||
`LotusGuild/element-call`), published to our private Gitea npm registry and served
|
||||
same-origin. We no longer depend on the upstream prebuilt bundle, so in-call
|
||||
behavior is editable source instead of fragile DOM/widget hacks.
|
||||
|
||||
**The plan is to fork `element-hq/element-call` into a new `LotusGuild/element-call`
|
||||
repo, build it from source, and host our own build** for true ownership. The full
|
||||
self-contained plan and integration map — written for a fresh session with no
|
||||
prior context — is in **[`HANDOFF_ELEMENT_CALL_FORK.md`](HANDOFF_ELEMENT_CALL_FORK.md)**.
|
||||
Infra/hosting notes also live in the `LotusGuild/matrix` repo README. Search the
|
||||
docs for the **`[EC-FORK]`** tag to find every related note.
|
||||
**Shipped via the fork:** denoise as an in-source LiveKit audio stage (survives
|
||||
reconnects), in-call speaking/mute events, focus-a-participant during screenshare,
|
||||
avatar decorations on EC video tiles, and a native transparent background.
|
||||
**Built but dormant (need cinny UI):** real call-audio injection
|
||||
(`io.lotus.inject_audio` → in-call soundboard) and quality controls
|
||||
(`io.lotus.set_quality`).
|
||||
|
||||
The full plan and integration map is in
|
||||
**[`HANDOFF_ELEMENT_CALL_FORK.md`](HANDOFF_ELEMENT_CALL_FORK.md)**; infra/hosting +
|
||||
build-pipeline notes live in the `LotusGuild/matrix` repo README. Search the docs
|
||||
for the **`[EC-FORK]`** tag to find every related note.
|
||||
|
||||
### Build
|
||||
|
||||
|
||||
@@ -45,6 +45,7 @@ import { useMatrixClient } from '../hooks/useMatrixClient';
|
||||
import { previewRingtone, startRingtone } from '../utils/ringtones';
|
||||
import { useCallMembersChange, useCallSession } from '../hooks/useCall';
|
||||
import { useCallJoinLeaveSounds } from '../hooks/useCallJoinLeaveSounds';
|
||||
import { useCallQuality } from '../hooks/useCallQuality';
|
||||
import { useRemoteAllMuted } from '../hooks/useCallSpeakers';
|
||||
import { useRoomAvatar, useRoomName } from '../hooks/useRoomMeta';
|
||||
import { mDirectAtom } from '../state/mDirectList';
|
||||
@@ -584,6 +585,7 @@ function CallUtils({ embed }: { embed: CallEmbed }) {
|
||||
useCallMemberSoundSync(embed);
|
||||
useCallJoinLeaveSounds(embed);
|
||||
useCallThemeSync(embed);
|
||||
useCallQuality(embed);
|
||||
useCallHangupEvent(
|
||||
embed,
|
||||
useCallback(() => {
|
||||
|
||||
@@ -37,6 +37,10 @@ import { stopPropagation } from '../../utils/keyboard';
|
||||
import { AsyncStatus, useAsyncCallback } from '../../hooks/useAsyncCallback';
|
||||
import { useCallEmbedRef } from '../../hooks/useCallEmbed';
|
||||
import { useAfkAutoMute } from '../../hooks/useAfkAutoMute';
|
||||
import { CallSoundboard } from './CallSoundboard';
|
||||
import { useStateEvent } from '../../hooks/useStateEvent';
|
||||
import { StateEvent } from '../../../types/matrix/room';
|
||||
import { RoomQualityContent } from '../../utils/callQuality';
|
||||
|
||||
type CallControlsProps = {
|
||||
callEmbed: CallEmbed;
|
||||
@@ -88,6 +92,19 @@ export function CallControls({ callEmbed }: CallControlsProps) {
|
||||
const [pttMode] = useSetting(settingsAtom, 'pttMode');
|
||||
const [pttKey] = useSetting(settingsAtom, 'pttKey');
|
||||
const [deafenKey] = useSetting(settingsAtom, 'deafenKey');
|
||||
const [soundboardEnabled] = useSetting(settingsAtom, 'soundboardEnabled');
|
||||
|
||||
// [P5-31] Hard room publish policy — hide controls the server will refuse so
|
||||
// users don't click dead buttons. Absent/true = allowed.
|
||||
const roomQualityEvent = useStateEvent(callEmbed.room, StateEvent.LotusRoomQuality);
|
||||
const roomQuality = roomQualityEvent?.getContent<RoomQualityContent>();
|
||||
const cameraAllowed = roomQuality?.allow_camera !== false;
|
||||
const screenshareAllowed = roomQuality?.allow_screenshare !== false;
|
||||
// Keep a forbidden control visible while its track is still live (so the user
|
||||
// can stop it); otherwise hide it entirely.
|
||||
const showCamera = cameraAllowed || video;
|
||||
const showScreenshare = screenshareAllowed || screenshare;
|
||||
const showVideoGroup = showCamera || showScreenshare || !!document.fullscreenEnabled;
|
||||
const [pttActive, setPttActive] = useState(false);
|
||||
|
||||
// Track microphone via ref so the PTT effect doesn't need it as a dep (avoids listener churn)
|
||||
@@ -339,24 +356,31 @@ export function CallControls({ callEmbed }: CallControlsProps) {
|
||||
onToggle={() => callEmbed.control.toggleScreenshareAudio()}
|
||||
/>
|
||||
</Box>
|
||||
{!compact && <ControlDivider />}
|
||||
{!compact && showVideoGroup && <ControlDivider />}
|
||||
{showVideoGroup && (
|
||||
<Box shrink="No" alignItems="Inherit" justifyContent="Inherit" gap="200">
|
||||
<VideoButton enabled={video} onToggle={handleVideoToggle} />
|
||||
{/* Show a forbidden control while its track is still live so the
|
||||
user can stop it; once stopped it hides and can't be restarted. */}
|
||||
{showCamera && <VideoButton enabled={video} onToggle={handleVideoToggle} />}
|
||||
{showScreenshare && (
|
||||
<ScreenShareButton
|
||||
enabled={screenshare}
|
||||
onToggle={() =>
|
||||
screenshare ? callEmbed.control.toggleScreenshare() : setShareConfirm(true)
|
||||
}
|
||||
/>
|
||||
)}
|
||||
{!!document.fullscreenEnabled && (
|
||||
<FullscreenButton isFullscreen={isFullscreen} onToggle={handleFullscreen} />
|
||||
)}
|
||||
</Box>
|
||||
)}
|
||||
</Box>
|
||||
{!compact && <ControlDivider />}
|
||||
<Box alignItems="Center" gap="Inherit" grow="Yes" direction={compact ? 'Column' : 'Row'}>
|
||||
<Box shrink="No" alignItems="Inherit" justifyContent="Inherit" gap="200">
|
||||
<ChatButton />
|
||||
{soundboardEnabled && <CallSoundboard callEmbed={callEmbed} />}
|
||||
<PopOut
|
||||
anchor={cords}
|
||||
position="Top"
|
||||
|
||||
@@ -0,0 +1,220 @@
|
||||
import React, { MouseEventHandler, useCallback, useRef, useState } from 'react';
|
||||
import {
|
||||
Box,
|
||||
Chip,
|
||||
Icon,
|
||||
IconButton,
|
||||
Icons,
|
||||
Menu,
|
||||
PopOut,
|
||||
RectCords,
|
||||
Spinner,
|
||||
Text,
|
||||
Tooltip,
|
||||
TooltipProvider,
|
||||
color,
|
||||
config,
|
||||
} from 'folds';
|
||||
import FocusTrap from 'focus-trap-react';
|
||||
import { CallEmbed } from '../../plugins/call';
|
||||
import { useMatrixClient } from '../../hooks/useMatrixClient';
|
||||
import { useSoundboard } from '../../hooks/useSoundboard';
|
||||
import { useSetting } from '../../state/hooks/settings';
|
||||
import { settingsAtom } from '../../state/settings';
|
||||
import { stopPropagation } from '../../utils/keyboard';
|
||||
import {
|
||||
SOUNDBOARD_ACCEPT,
|
||||
SOUNDBOARD_MAX_CLIPS,
|
||||
playClipLocally,
|
||||
resolveClipObjectUrl,
|
||||
} from '../../utils/soundboardClips';
|
||||
|
||||
type CallSoundboardProps = {
|
||||
callEmbed: CallEmbed;
|
||||
};
|
||||
|
||||
/**
|
||||
* [P5-15] In-call soundboard: trigger user-uploaded clips into the call. Each
|
||||
* clip is published to peers as a separate track by the EC fork
|
||||
* (`io.lotus.inject_audio`) and also played locally for the presser's feedback.
|
||||
* Clips are uploadable/managed here and synced across devices via the
|
||||
* `io.lotus.soundboard` account data (like custom emoji/sticker packs).
|
||||
*/
|
||||
export function CallSoundboard({ callEmbed }: CallSoundboardProps) {
|
||||
const mx = useMatrixClient();
|
||||
const { clips, addClip, removeClip } = useSoundboard();
|
||||
const [soundboardVolume] = useSetting(settingsAtom, 'soundboardVolume');
|
||||
|
||||
const [cords, setCords] = useState<RectCords>();
|
||||
const [busyId, setBusyId] = useState<string>();
|
||||
const [uploading, setUploading] = useState(false);
|
||||
const [error, setError] = useState<string>();
|
||||
const fileInputRef = useRef<HTMLInputElement>(null);
|
||||
|
||||
const volume = Math.max(0, Math.min(1, soundboardVolume / 100));
|
||||
|
||||
const handleOpen: MouseEventHandler<HTMLButtonElement> = (evt) => {
|
||||
setError(undefined);
|
||||
setCords(evt.currentTarget.getBoundingClientRect());
|
||||
};
|
||||
|
||||
const handlePlay = useCallback(
|
||||
async (id: string, mxc: string) => {
|
||||
setBusyId(id);
|
||||
setError(undefined);
|
||||
try {
|
||||
const objectUrl = await resolveClipObjectUrl(mx, mxc);
|
||||
callEmbed.control.injectAudio(objectUrl, volume);
|
||||
playClipLocally(objectUrl, volume);
|
||||
} catch {
|
||||
setError('Could not play that clip.');
|
||||
} finally {
|
||||
setBusyId(undefined);
|
||||
}
|
||||
},
|
||||
[mx, callEmbed, volume],
|
||||
);
|
||||
|
||||
const handleFile = useCallback(
|
||||
async (file: File | undefined) => {
|
||||
if (!file) return;
|
||||
setUploading(true);
|
||||
setError(undefined);
|
||||
try {
|
||||
await addClip(file);
|
||||
} catch (e) {
|
||||
setError(e instanceof Error ? e.message : 'Upload failed.');
|
||||
} finally {
|
||||
setUploading(false);
|
||||
}
|
||||
},
|
||||
[addClip],
|
||||
);
|
||||
|
||||
return (
|
||||
<>
|
||||
<input
|
||||
ref={fileInputRef}
|
||||
type="file"
|
||||
accept={SOUNDBOARD_ACCEPT}
|
||||
hidden
|
||||
onChange={(e) => {
|
||||
handleFile(e.target.files?.[0]);
|
||||
e.target.value = '';
|
||||
}}
|
||||
/>
|
||||
<PopOut
|
||||
anchor={cords}
|
||||
position="Top"
|
||||
align="Center"
|
||||
content={
|
||||
<FocusTrap
|
||||
focusTrapOptions={{
|
||||
initialFocus: false,
|
||||
onDeactivate: () => setCords(undefined),
|
||||
clickOutsideDeactivates: true,
|
||||
escapeDeactivates: stopPropagation,
|
||||
}}
|
||||
>
|
||||
<Menu style={{ maxWidth: '320px' }}>
|
||||
<Box direction="Column" gap="200" style={{ padding: config.space.S200 }}>
|
||||
<Box alignItems="Center" justifyContent="SpaceBetween" gap="200">
|
||||
<Text size="L400">Soundboard</Text>
|
||||
<Chip
|
||||
variant="Secondary"
|
||||
radii="Pill"
|
||||
disabled={uploading || clips.length >= SOUNDBOARD_MAX_CLIPS}
|
||||
onClick={() => fileInputRef.current?.click()}
|
||||
before={
|
||||
uploading ? <Spinner size="100" /> : <Icon size="100" src={Icons.Plus} />
|
||||
}
|
||||
>
|
||||
<Text size="B300">Upload</Text>
|
||||
</Chip>
|
||||
</Box>
|
||||
|
||||
{clips.length === 0 ? (
|
||||
<Text size="T200" priority="300">
|
||||
No clips yet. Upload a short audio clip (max 1 MB) to play it into the call.
|
||||
Clips sync across your devices.
|
||||
</Text>
|
||||
) : (
|
||||
<Box wrap="Wrap" gap="200">
|
||||
{clips.map((clip) => (
|
||||
<Box
|
||||
key={clip.id}
|
||||
direction="Column"
|
||||
gap="100"
|
||||
style={{ position: 'relative' }}
|
||||
>
|
||||
<Chip
|
||||
variant="SurfaceVariant"
|
||||
radii="300"
|
||||
disabled={busyId === clip.id}
|
||||
onClick={() => handlePlay(clip.id, clip.url)}
|
||||
before={
|
||||
busyId === clip.id ? (
|
||||
<Spinner size="100" />
|
||||
) : (
|
||||
<Icon size="100" src={Icons.Play} />
|
||||
)
|
||||
}
|
||||
after={
|
||||
<Icon
|
||||
size="50"
|
||||
src={Icons.Cross}
|
||||
style={{ cursor: 'pointer' }}
|
||||
onClick={(e: React.MouseEvent) => {
|
||||
e.stopPropagation();
|
||||
removeClip(clip.id);
|
||||
}}
|
||||
/>
|
||||
}
|
||||
>
|
||||
<Text size="B300" truncate style={{ maxWidth: '120px' }}>
|
||||
{clip.name}
|
||||
</Text>
|
||||
</Chip>
|
||||
</Box>
|
||||
))}
|
||||
</Box>
|
||||
)}
|
||||
|
||||
{error && (
|
||||
<Text size="T200" style={{ color: color.Critical.Main }}>
|
||||
{error}
|
||||
</Text>
|
||||
)}
|
||||
</Box>
|
||||
</Menu>
|
||||
</FocusTrap>
|
||||
}
|
||||
>
|
||||
<TooltipProvider
|
||||
tooltip={
|
||||
<Tooltip>
|
||||
<Text>Soundboard</Text>
|
||||
</Tooltip>
|
||||
}
|
||||
>
|
||||
{(triggerRef) => (
|
||||
<IconButton
|
||||
ref={triggerRef}
|
||||
variant="Surface"
|
||||
fill="Soft"
|
||||
radii="400"
|
||||
size="400"
|
||||
onClick={handleOpen}
|
||||
outlined
|
||||
aria-label="Soundboard"
|
||||
aria-expanded={!!cords}
|
||||
aria-haspopup="menu"
|
||||
>
|
||||
<Icon size="400" src={Icons.BellRing} />
|
||||
</IconButton>
|
||||
)}
|
||||
</TooltipProvider>
|
||||
</PopOut>
|
||||
</>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,198 @@
|
||||
import React, { useCallback, useEffect, useMemo, useState } from 'react';
|
||||
import { Box, Switch, Text } from 'folds';
|
||||
import { SequenceCard } from '../../../components/sequence-card';
|
||||
import { SequenceCardStyle } from '../../room-settings/styles.css';
|
||||
import { SettingTile } from '../../../components/setting-tile';
|
||||
import { SettingsSelect } from '../../../components/settings-select/SettingsSelect';
|
||||
import { useMatrixClient } from '../../../hooks/useMatrixClient';
|
||||
import { useRoom } from '../../../hooks/useRoom';
|
||||
import { StateEvent } from '../../../../types/matrix/room';
|
||||
import { useStateEvent } from '../../../hooks/useStateEvent';
|
||||
import { RoomPermissionsAPI } from '../../../hooks/useRoomPermissions';
|
||||
import { AsyncStatus, useAsyncCallback } from '../../../hooks/useAsyncCallback';
|
||||
import {
|
||||
AUDIO_BITRATE_OPTIONS,
|
||||
RoomQualityContent,
|
||||
SCREENSHARE_BITRATE_OPTIONS,
|
||||
SCREENSHARE_FRAMERATE_OPTIONS,
|
||||
} from '../../../utils/callQuality';
|
||||
|
||||
// Only the numeric cap keys are edited via `update`; the boolean policy keys
|
||||
// are handled by `setAllow`.
|
||||
type CapKey = 'audio_max_kbps' | 'screenshare_max_kbps' | 'screenshare_max_fps';
|
||||
|
||||
// String <-> numeric bridge for SettingsSelect (which needs string values).
|
||||
const toValue = (n?: number): string => (typeof n === 'number' ? String(n) : 'auto');
|
||||
|
||||
const CAP_KEYS: (keyof RoomQualityContent)[] = [
|
||||
'audio_max_kbps',
|
||||
'screenshare_max_kbps',
|
||||
'screenshare_max_fps',
|
||||
'allow_screenshare',
|
||||
'allow_camera',
|
||||
];
|
||||
const capsEqual = (a: RoomQualityContent, b: RoomQualityContent): boolean =>
|
||||
CAP_KEYS.every((k) => a[k] === b[k]);
|
||||
|
||||
type RoomQualityProps = {
|
||||
permissions: RoomPermissionsAPI;
|
||||
};
|
||||
/**
|
||||
* [P5-31] Room-admin quality ceiling. Writes `io.lotus.room_quality`; every
|
||||
* Lotus client clamps its per-user quality to these caps. Hard enforcement for
|
||||
* ALL Matrix clients is a server-side follow-up (see LOTUS_TODO.md P5-31).
|
||||
*/
|
||||
export function RoomQuality({ permissions }: RoomQualityProps) {
|
||||
const mx = useMatrixClient();
|
||||
const room = useRoom();
|
||||
|
||||
const canEdit = permissions.stateEvent(StateEvent.LotusRoomQuality, mx.getSafeUserId());
|
||||
|
||||
const event = useStateEvent(room, StateEvent.LotusRoomQuality);
|
||||
const caps = useMemo<RoomQualityContent>(() => event?.getContent() ?? {}, [event]);
|
||||
|
||||
const [submitState, submit] = useAsyncCallback(
|
||||
useCallback(
|
||||
async (next: RoomQualityContent) => {
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
await mx.sendStateEvent(room.roomId, StateEvent.LotusRoomQuality as any, next);
|
||||
},
|
||||
[mx, room.roomId],
|
||||
),
|
||||
);
|
||||
const submitting = submitState.status === AsyncStatus.Loading;
|
||||
|
||||
// Optimistic mirror: `useStateEvent` only refreshes when the write echoes
|
||||
// back via /sync (not when sendStateEvent resolves), so consecutive edits
|
||||
// must build on the pending write — otherwise a second edit spreads a stale
|
||||
// `caps` and silently drops the first. `effective` is what the UI shows and
|
||||
// what each edit merges into; it's reconciled below once the echo lands.
|
||||
const [pending, setPending] = useState<RoomQualityContent | null>(null);
|
||||
const effective = pending ?? caps;
|
||||
|
||||
useEffect(() => {
|
||||
if (!pending) return;
|
||||
// Revert the optimistic view if the write failed…
|
||||
if (submitState.status === AsyncStatus.Error) {
|
||||
setPending(null);
|
||||
return;
|
||||
}
|
||||
// …or drop it once the synced state actually reflects it.
|
||||
if (capsEqual(caps, pending)) setPending(null);
|
||||
}, [caps, pending, submitState.status]);
|
||||
|
||||
const commit = (next: RoomQualityContent) => {
|
||||
setPending(next);
|
||||
submit(next);
|
||||
};
|
||||
|
||||
const update = (key: CapKey, value: string) => {
|
||||
const next: RoomQualityContent = { ...effective };
|
||||
if (value === 'auto') delete next[key];
|
||||
else next[key] = parseInt(value, 10);
|
||||
commit(next);
|
||||
};
|
||||
|
||||
const setAllow = (key: 'allow_screenshare' | 'allow_camera', allowed: boolean) => {
|
||||
const next: RoomQualityContent = { ...effective };
|
||||
// Absent = allowed, so only persist the key when forbidding.
|
||||
if (allowed) delete next[key];
|
||||
else next[key] = false;
|
||||
commit(next);
|
||||
};
|
||||
|
||||
// Absent/true = allowed.
|
||||
const screenshareAllowed = effective.allow_screenshare !== false;
|
||||
const cameraAllowed = effective.allow_camera !== false;
|
||||
|
||||
return (
|
||||
<SequenceCard
|
||||
className={SequenceCardStyle}
|
||||
variant="SurfaceVariant"
|
||||
direction="Column"
|
||||
gap="400"
|
||||
>
|
||||
<SettingTile
|
||||
title="Call Permissions"
|
||||
description={
|
||||
<Text size="T200" priority="300">
|
||||
Control what participants may share in this room. These are enforced on the server for
|
||||
every Matrix client (Element, FluffyChat, Lotus Chat, …).
|
||||
</Text>
|
||||
}
|
||||
/>
|
||||
<Box direction="Column" gap="300">
|
||||
<SettingTile
|
||||
title="Allow Screen Sharing"
|
||||
description="When off, no one can share their screen in this room."
|
||||
after={
|
||||
<Switch
|
||||
variant="Primary"
|
||||
value={screenshareAllowed}
|
||||
onChange={(v) => setAllow('allow_screenshare', v)}
|
||||
disabled={!canEdit || submitting}
|
||||
/>
|
||||
}
|
||||
/>
|
||||
<SettingTile
|
||||
title="Allow Camera"
|
||||
description="When off, this is an audio-only room — no one can turn on their camera. Microphones are always allowed."
|
||||
after={
|
||||
<Switch
|
||||
variant="Primary"
|
||||
value={cameraAllowed}
|
||||
onChange={(v) => setAllow('allow_camera', v)}
|
||||
disabled={!canEdit || submitting}
|
||||
/>
|
||||
}
|
||||
/>
|
||||
</Box>
|
||||
|
||||
<SettingTile
|
||||
title="Call Quality Caps"
|
||||
description={
|
||||
<Text size="T200" priority="300">
|
||||
Set a maximum microphone bitrate, screenshare bitrate, and screenshare framerate for
|
||||
this room. Lotus Chat clamps each participant to these ceilings (best-effort — applies
|
||||
to Lotus Chat clients). Auto = no cap.
|
||||
</Text>
|
||||
}
|
||||
/>
|
||||
<Box direction="Column" gap="300">
|
||||
<SettingTile
|
||||
title="Max Microphone Bitrate"
|
||||
after={
|
||||
<SettingsSelect
|
||||
value={toValue(effective.audio_max_kbps)}
|
||||
onChange={(v) => update('audio_max_kbps', v)}
|
||||
options={AUDIO_BITRATE_OPTIONS}
|
||||
disabled={!canEdit || submitting}
|
||||
/>
|
||||
}
|
||||
/>
|
||||
<SettingTile
|
||||
title="Max Screenshare Bitrate"
|
||||
after={
|
||||
<SettingsSelect
|
||||
value={toValue(effective.screenshare_max_kbps)}
|
||||
onChange={(v) => update('screenshare_max_kbps', v)}
|
||||
options={SCREENSHARE_BITRATE_OPTIONS}
|
||||
disabled={!canEdit || submitting}
|
||||
/>
|
||||
}
|
||||
/>
|
||||
<SettingTile
|
||||
title="Max Screenshare Framerate"
|
||||
after={
|
||||
<SettingsSelect
|
||||
value={toValue(effective.screenshare_max_fps)}
|
||||
onChange={(v) => update('screenshare_max_fps', v)}
|
||||
options={SCREENSHARE_FRAMERATE_OPTIONS}
|
||||
disabled={!canEdit || submitting}
|
||||
/>
|
||||
}
|
||||
/>
|
||||
</Box>
|
||||
</SequenceCard>
|
||||
);
|
||||
}
|
||||
@@ -4,6 +4,7 @@ export * from './RoomHistoryVisibility';
|
||||
export * from './RoomJoinRules';
|
||||
export * from './RoomProfile';
|
||||
export * from './RoomPublish';
|
||||
export * from './RoomQuality';
|
||||
export * from './RoomShareInvite';
|
||||
export * from './RoomUpgrade';
|
||||
export * from './RoomVoiceLimit';
|
||||
|
||||
@@ -11,6 +11,7 @@ import {
|
||||
RoomLocalAddresses,
|
||||
RoomPublishedAddresses,
|
||||
RoomPublish,
|
||||
RoomQuality,
|
||||
RoomShareInvite,
|
||||
RoomUpgrade,
|
||||
RoomVoiceLimit,
|
||||
@@ -58,6 +59,7 @@ export function General({ requestClose }: GeneralProps) {
|
||||
<Box direction="Column" gap="100">
|
||||
<Text size="L400">Voice</Text>
|
||||
<RoomVoiceLimit permissions={permissions} />
|
||||
<RoomQuality permissions={permissions} />
|
||||
</Box>
|
||||
<Box direction="Column" gap="100">
|
||||
<Text size="L400">Addresses</Text>
|
||||
|
||||
@@ -45,6 +45,7 @@ import {
|
||||
} from '../../../utils/lotusDenoiseUtils';
|
||||
import { useSetting } from '../../../state/hooks/settings';
|
||||
import {
|
||||
CallAudioBitrate,
|
||||
ChatBackground,
|
||||
ComposerToolbarSettings,
|
||||
DateFormat,
|
||||
@@ -53,9 +54,16 @@ import {
|
||||
MessageSpacing,
|
||||
NoiseSuppressionMode,
|
||||
RingtoneId,
|
||||
ScreenshareBitrate,
|
||||
ScreenshareFramerate,
|
||||
Settings,
|
||||
settingsAtom,
|
||||
} from '../../../state/settings';
|
||||
import {
|
||||
AUDIO_BITRATE_OPTIONS,
|
||||
SCREENSHARE_BITRATE_OPTIONS,
|
||||
SCREENSHARE_FRAMERATE_OPTIONS,
|
||||
} from '../../../utils/callQuality';
|
||||
import { SeasonalPreview, SeasonTheme } from '../../../components/seasonal/SeasonalEffect';
|
||||
import { SEASON_DATE_RANGES } from '../../../components/seasonal/seasonSchedule';
|
||||
import { SettingTile } from '../../../components/setting-tile';
|
||||
@@ -1221,6 +1229,18 @@ function Calls() {
|
||||
const [ringtoneVolume, setRingtoneVolume] = useSetting(settingsAtom, 'ringtoneVolume');
|
||||
const [ringtoneId, setRingtoneId] = useSetting(settingsAtom, 'ringtoneId');
|
||||
|
||||
const [callAudioBitrate, setCallAudioBitrate] = useSetting(settingsAtom, 'callAudioBitrate');
|
||||
const [screenshareBitrate, setScreenshareBitrate] = useSetting(
|
||||
settingsAtom,
|
||||
'screenshareBitrate',
|
||||
);
|
||||
const [screenshareFramerate, setScreenshareFramerate] = useSetting(
|
||||
settingsAtom,
|
||||
'screenshareFramerate',
|
||||
);
|
||||
const [soundboardEnabled, setSoundboardEnabled] = useSetting(settingsAtom, 'soundboardEnabled');
|
||||
const [soundboardVolume, setSoundboardVolume] = useSetting(settingsAtom, 'soundboardVolume');
|
||||
|
||||
const handleJoinLeaveSoundChange = (value: 'off' | 'chime' | 'soft' | 'retro') => {
|
||||
setCallJoinLeaveSound(value);
|
||||
if (value !== 'off') playCallJoinSound(value);
|
||||
@@ -1616,6 +1636,80 @@ function Calls() {
|
||||
}
|
||||
/>
|
||||
</SequenceCard>
|
||||
|
||||
<SequenceCard className={SequenceCardStyle} variant="SurfaceVariant" direction="Column">
|
||||
<SettingTile
|
||||
title="Microphone Bitrate"
|
||||
description="Cap the audio bitrate your mic sends in calls. Lower saves bandwidth; higher is clearer. Auto lets Element Call decide."
|
||||
after={
|
||||
<SettingsSelect<CallAudioBitrate>
|
||||
value={callAudioBitrate}
|
||||
onChange={setCallAudioBitrate}
|
||||
options={AUDIO_BITRATE_OPTIONS}
|
||||
/>
|
||||
}
|
||||
/>
|
||||
</SequenceCard>
|
||||
|
||||
<SequenceCard className={SequenceCardStyle} variant="SurfaceVariant" direction="Column">
|
||||
<SettingTile
|
||||
title="Screenshare Bitrate"
|
||||
description="Cap the bitrate used when you share your screen. Lower is smoother on poor connections; higher is sharper."
|
||||
after={
|
||||
<SettingsSelect<ScreenshareBitrate>
|
||||
value={screenshareBitrate}
|
||||
onChange={setScreenshareBitrate}
|
||||
options={SCREENSHARE_BITRATE_OPTIONS}
|
||||
/>
|
||||
}
|
||||
/>
|
||||
</SequenceCard>
|
||||
|
||||
<SequenceCard className={SequenceCardStyle} variant="SurfaceVariant" direction="Column">
|
||||
<SettingTile
|
||||
title="Screenshare Framerate"
|
||||
description="Cap the frames-per-second of your screenshare. 60 fps suits motion/gaming; 15 fps suits slides and saves bandwidth."
|
||||
after={
|
||||
<SettingsSelect<ScreenshareFramerate>
|
||||
value={screenshareFramerate}
|
||||
onChange={setScreenshareFramerate}
|
||||
options={SCREENSHARE_FRAMERATE_OPTIONS}
|
||||
/>
|
||||
}
|
||||
/>
|
||||
</SequenceCard>
|
||||
|
||||
<SequenceCard className={SequenceCardStyle} variant="SurfaceVariant" direction="Column">
|
||||
<SettingTile
|
||||
title="Soundboard"
|
||||
description="Show a soundboard button in the call bar. Upload short audio clips (like custom emojis) to play them into the call. Clips sync across your devices."
|
||||
after={
|
||||
<Switch variant="Primary" value={soundboardEnabled} onChange={setSoundboardEnabled} />
|
||||
}
|
||||
/>
|
||||
{soundboardEnabled && (
|
||||
<SettingTile
|
||||
title="Soundboard Volume"
|
||||
after={
|
||||
<Box alignItems="Center" gap="200" style={{ minWidth: toRem(180) }}>
|
||||
<input
|
||||
type="range"
|
||||
min={0}
|
||||
max={100}
|
||||
step={5}
|
||||
value={soundboardVolume}
|
||||
onChange={(e) => setSoundboardVolume(parseInt(e.target.value, 10))}
|
||||
style={{ flexGrow: 1 }}
|
||||
aria-label="Soundboard volume"
|
||||
/>
|
||||
<Text size="T200" style={{ minWidth: toRem(36), textAlign: 'right' }}>
|
||||
{soundboardVolume}%
|
||||
</Text>
|
||||
</Box>
|
||||
}
|
||||
/>
|
||||
)}
|
||||
</SequenceCard>
|
||||
</Box>
|
||||
);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,41 @@
|
||||
import { useEffect } from 'react';
|
||||
import { useAtomValue } from 'jotai';
|
||||
import { CallEmbed } from '../plugins/call';
|
||||
import { settingsAtom } from '../state/settings';
|
||||
import { useStateEvent } from './useStateEvent';
|
||||
import { StateEvent } from '../../types/matrix/room';
|
||||
import { buildQualityPayload, RoomQualityContent } from '../utils/callQuality';
|
||||
|
||||
/**
|
||||
* [P5-31] Apply the user's call quality settings (clamped by any room-level
|
||||
* cap) to the Element Call fork via the `io.lotus.set_quality` widget action.
|
||||
*
|
||||
* The fork stores the settings and re-applies them on every (re)publish and
|
||||
* reconnect, so we only need to (re)send when the payload changes or the widget
|
||||
* becomes ready — no need to poll the track lifecycle here.
|
||||
*/
|
||||
export function useCallQuality(embed: CallEmbed): void {
|
||||
const { callAudioBitrate, screenshareBitrate, screenshareFramerate } = useAtomValue(settingsAtom);
|
||||
|
||||
const roomQualityEvent = useStateEvent(embed.room, StateEvent.LotusRoomQuality);
|
||||
const roomCaps = roomQualityEvent?.getContent<RoomQualityContent>();
|
||||
|
||||
// Depend on the primitive cap values (not the event object) so re-renders
|
||||
// don't resend needlessly.
|
||||
const audioCap = roomCaps?.audio_max_kbps;
|
||||
const ssCap = roomCaps?.screenshare_max_kbps;
|
||||
const fpsCap = roomCaps?.screenshare_max_fps;
|
||||
|
||||
useEffect(() => {
|
||||
const payload = buildQualityPayload(
|
||||
{ callAudioBitrate, screenshareBitrate, screenshareFramerate },
|
||||
{ audio_max_kbps: audioCap, screenshare_max_kbps: ssCap, screenshare_max_fps: fpsCap },
|
||||
);
|
||||
const send = (): void => embed.control.setQuality(payload);
|
||||
// Send now (settings are sticky fork-side even if tracks aren't up yet) and
|
||||
// again once the widget signals ready, in case the transport wasn't up.
|
||||
send();
|
||||
const off = embed.onReady(send);
|
||||
return off;
|
||||
}, [embed, callAudioBitrate, screenshareBitrate, screenshareFramerate, audioCap, ssCap, fpsCap]);
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
import { useCallback, useEffect, useState } from 'react';
|
||||
import { useMatrixClient } from './useMatrixClient';
|
||||
import { useAccountDataCallback } from './useAccountDataCallback';
|
||||
import { AccountDataEvent } from '../../types/matrix/accountData';
|
||||
import {
|
||||
SoundboardClip,
|
||||
SoundboardContent,
|
||||
SOUNDBOARD_MAX_CLIP_BYTES,
|
||||
SOUNDBOARD_MAX_CLIPS,
|
||||
SOUNDBOARD_NAME_MAX,
|
||||
readSoundboardClips,
|
||||
} from '../utils/soundboardClips';
|
||||
|
||||
const KEY = AccountDataEvent.LotusSoundboard;
|
||||
|
||||
/**
|
||||
* [P5-15] Read/write the user's personal soundboard, stored in the
|
||||
* `io.lotus.soundboard` account data event (synced across devices like custom
|
||||
* emoji/sticker packs). Uploading writes the audio to the media repo and
|
||||
* appends an mxc reference.
|
||||
*/
|
||||
export function useSoundboard(): {
|
||||
clips: SoundboardClip[];
|
||||
addClip: (file: File, name?: string) => Promise<void>;
|
||||
removeClip: (id: string) => Promise<void>;
|
||||
renameClip: (id: string, name: string) => Promise<void>;
|
||||
} {
|
||||
const mx = useMatrixClient();
|
||||
const [clips, setClips] = useState<SoundboardClip[]>(() => readSoundboardClips(mx));
|
||||
|
||||
useAccountDataCallback(
|
||||
mx,
|
||||
useCallback((evt) => {
|
||||
if (evt.getType() === KEY) {
|
||||
const content = evt.getContent<SoundboardContent>();
|
||||
setClips(Array.isArray(content?.clips) ? content.clips : []);
|
||||
}
|
||||
}, []),
|
||||
);
|
||||
|
||||
useEffect(() => {
|
||||
setClips(readSoundboardClips(mx));
|
||||
}, [mx]);
|
||||
|
||||
const persist = useCallback(
|
||||
async (next: SoundboardClip[]) => {
|
||||
const content: SoundboardContent = { clips: next };
|
||||
await (
|
||||
mx as unknown as { setAccountData: (t: string, c: unknown) => Promise<void> }
|
||||
).setAccountData(KEY, content);
|
||||
},
|
||||
[mx],
|
||||
);
|
||||
|
||||
const addClip = useCallback(
|
||||
async (file: File, name?: string) => {
|
||||
const current = readSoundboardClips(mx);
|
||||
if (current.length >= SOUNDBOARD_MAX_CLIPS) {
|
||||
throw new Error(`Soundboard is full (max ${SOUNDBOARD_MAX_CLIPS} clips).`);
|
||||
}
|
||||
if (file.size > SOUNDBOARD_MAX_CLIP_BYTES) {
|
||||
throw new Error('Clip is too large (max 1 MB).');
|
||||
}
|
||||
const res = await mx.uploadContent(file, { type: file.type || 'audio/mpeg' });
|
||||
const mxc = res.content_uri;
|
||||
if (!mxc) throw new Error('Upload failed.');
|
||||
const label = (name ?? file.name.replace(/\.[^/.]+$/, ''))
|
||||
.trim()
|
||||
.slice(0, SOUNDBOARD_NAME_MAX);
|
||||
const clip: SoundboardClip = {
|
||||
id: `${Date.now()}-${Math.random().toString(36).slice(2, 8)}`,
|
||||
name: label || 'Clip',
|
||||
url: mxc,
|
||||
mimetype: file.type || undefined,
|
||||
size: file.size,
|
||||
};
|
||||
await persist([...current, clip]);
|
||||
},
|
||||
[mx, persist],
|
||||
);
|
||||
|
||||
const removeClip = useCallback(
|
||||
async (id: string) => {
|
||||
const next = readSoundboardClips(mx).filter((c) => c.id !== id);
|
||||
await persist(next);
|
||||
},
|
||||
[mx, persist],
|
||||
);
|
||||
|
||||
const renameClip = useCallback(
|
||||
async (id: string, name: string) => {
|
||||
const trimmed = name.trim().slice(0, SOUNDBOARD_NAME_MAX);
|
||||
if (!trimmed) return;
|
||||
const next = readSoundboardClips(mx).map((c) => (c.id === id ? { ...c, name: trimmed } : c));
|
||||
await persist(next);
|
||||
},
|
||||
[mx, persist],
|
||||
);
|
||||
|
||||
return { clips, addClip, removeClip, renameClip };
|
||||
}
|
||||
@@ -7,6 +7,17 @@ export enum CallControlEvent {
|
||||
StateUpdate = 'state_update',
|
||||
}
|
||||
|
||||
/**
|
||||
* [lotus #7 / P5-31] Payload for the fork's `io.lotus.set_quality` action.
|
||||
* All fields optional; `null` clears that cap. Bits/sec for bitrates, fps for
|
||||
* framerate.
|
||||
*/
|
||||
export type LotusQualityPayload = {
|
||||
audioMaxBitrate?: number | null;
|
||||
screenshareMaxBitrate?: number | null;
|
||||
screenshareMaxFramerate?: number | null;
|
||||
};
|
||||
|
||||
export class CallControl extends EventEmitter implements CallControlState {
|
||||
private state: CallControlState;
|
||||
|
||||
@@ -358,6 +369,33 @@ export class CallControl extends EventEmitter implements CallControlState {
|
||||
this.call.transport.send('io.lotus.focus_participant', { userId: null }).catch(() => undefined);
|
||||
}
|
||||
|
||||
/**
|
||||
* [lotus #3 / P5-15] Inject a soundboard clip into the call so other
|
||||
* participants hear it. The fork publishes it as a separate LiveKit audio
|
||||
* track (`io.lotus.inject_audio`) rather than splicing the mic. `url` must be
|
||||
* an https/blob URL the widget can fetch WITHOUT credentials — the host
|
||||
* resolves an mxc clip to a `blob:` object URL first (authenticated media
|
||||
* can't be fetched cross-realm by the widget). `volume` is 0–1.
|
||||
*
|
||||
* The local user does not hear their own published track, so callers should
|
||||
* also play the clip locally for feedback.
|
||||
*/
|
||||
public injectAudio(url: string, volume = 1): void {
|
||||
this.call.transport.send('io.lotus.inject_audio', { url, volume }).catch(() => undefined);
|
||||
}
|
||||
|
||||
/**
|
||||
* [lotus #7 / P5-31] Apply audio/screenshare encoding limits to the local
|
||||
* published tracks (the fork's `io.lotus.set_quality` action, via
|
||||
* `RTCRtpSender.setParameters` — no republish). Bitrates are bits/sec,
|
||||
* framerate is fps. A field set to `null` clears that cap. Settings are
|
||||
* sticky fork-side (re-applied on every re-publish / reconnect). Values are
|
||||
* clamped fork-side, so out-of-range input can't brick the encoder.
|
||||
*/
|
||||
public setQuality(settings: LotusQualityPayload): void {
|
||||
this.call.transport.send('io.lotus.set_quality', settings).catch(() => undefined);
|
||||
}
|
||||
|
||||
public dispose() {
|
||||
this.bodyMutationObserver.disconnect();
|
||||
this.controlMutationObserver.disconnect();
|
||||
|
||||
@@ -179,6 +179,11 @@ export class CallEmbed {
|
||||
// - transparent background so the room wallpaper shows through natively
|
||||
lotusCallState: 'true',
|
||||
lotusTransparent: 'true',
|
||||
// [lotus #3 / P5-15] Arm the fork's audio-inject handler so the in-call
|
||||
// soundboard can publish clips into the call. Dormant until the host
|
||||
// sends io.lotus.inject_audio (only on an explicit user click), so
|
||||
// arming it for every call is safe.
|
||||
lotusAudioInject: 'true',
|
||||
});
|
||||
|
||||
if (denoiseMode === 'ml') {
|
||||
|
||||
@@ -24,6 +24,13 @@ export type DenoiseModelId = 'rnnoise' | 'speex' | 'dtln' | 'deepfilternet';
|
||||
// 'soft' / 'retro' are synthesized in-browser (see utils/ringtones.ts);
|
||||
// 'none' is silent (visual-only incoming-call UI).
|
||||
export type RingtoneId = 'classic' | 'chime' | 'soft' | 'retro' | 'none';
|
||||
// [P5-31] Granular call quality caps. 'auto' = don't cap (the EC fork keeps its
|
||||
// default encoding). Numbers are kbps (audio/screenshare bitrate) or fps
|
||||
// (screenshare framerate); converted to the fork's bits/sec + fps payload in
|
||||
// utils/callQuality.ts and applied via the io.lotus.set_quality widget action.
|
||||
export type CallAudioBitrate = 'auto' | '32' | '64' | '96' | '128' | '256';
|
||||
export type ScreenshareBitrate = 'auto' | '500' | '1500' | '3000' | '8000';
|
||||
export type ScreenshareFramerate = 'auto' | '15' | '30' | '60';
|
||||
export type ChatBackground =
|
||||
| 'none'
|
||||
| 'blueprint'
|
||||
@@ -156,6 +163,14 @@ export interface Settings {
|
||||
ringtoneId: RingtoneId;
|
||||
ringtoneVolume: number; // 0–100
|
||||
|
||||
// [P5-31] Call quality controls
|
||||
callAudioBitrate: CallAudioBitrate;
|
||||
screenshareBitrate: ScreenshareBitrate;
|
||||
screenshareFramerate: ScreenshareFramerate;
|
||||
// [P5-15] In-call soundboard
|
||||
soundboardEnabled: boolean;
|
||||
soundboardVolume: number; // 0–100
|
||||
|
||||
seasonalThemeOverride:
|
||||
| 'auto'
|
||||
| 'off'
|
||||
@@ -253,6 +268,12 @@ const defaultSettings: Settings = {
|
||||
ringtoneId: 'classic',
|
||||
ringtoneVolume: 70,
|
||||
|
||||
callAudioBitrate: 'auto',
|
||||
screenshareBitrate: 'auto',
|
||||
screenshareFramerate: 'auto',
|
||||
soundboardEnabled: true,
|
||||
soundboardVolume: 80,
|
||||
|
||||
seasonalThemeOverride: 'auto',
|
||||
};
|
||||
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
import { describe, it } from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { buildQualityPayload } from './callQuality';
|
||||
|
||||
describe('buildQualityPayload', () => {
|
||||
it("sends null for every 'auto' field so a prior cap is reset", () => {
|
||||
const payload = buildQualityPayload({
|
||||
callAudioBitrate: 'auto',
|
||||
screenshareBitrate: 'auto',
|
||||
screenshareFramerate: 'auto',
|
||||
});
|
||||
assert.deepEqual(payload, {
|
||||
audioMaxBitrate: null,
|
||||
screenshareMaxBitrate: null,
|
||||
screenshareMaxFramerate: null,
|
||||
});
|
||||
});
|
||||
|
||||
it('converts kbps user settings to bits/sec and passes fps through', () => {
|
||||
const payload = buildQualityPayload({
|
||||
callAudioBitrate: '64',
|
||||
screenshareBitrate: '1500',
|
||||
screenshareFramerate: '30',
|
||||
});
|
||||
assert.equal(payload.audioMaxBitrate, 64_000);
|
||||
assert.equal(payload.screenshareMaxBitrate, 1_500_000);
|
||||
assert.equal(payload.screenshareMaxFramerate, 30);
|
||||
});
|
||||
|
||||
it('clamps the user setting down to the room cap (lower wins)', () => {
|
||||
const payload = buildQualityPayload(
|
||||
{ callAudioBitrate: '256', screenshareBitrate: '8000', screenshareFramerate: '60' },
|
||||
{ audio_max_kbps: 64, screenshare_max_kbps: 1500, screenshare_max_fps: 30 },
|
||||
);
|
||||
assert.equal(payload.audioMaxBitrate, 64_000);
|
||||
assert.equal(payload.screenshareMaxBitrate, 1_500_000);
|
||||
assert.equal(payload.screenshareMaxFramerate, 30);
|
||||
});
|
||||
|
||||
it('does not raise a user setting that is already below the room cap', () => {
|
||||
const payload = buildQualityPayload(
|
||||
{ callAudioBitrate: '32', screenshareBitrate: 'auto', screenshareFramerate: '15' },
|
||||
{ audio_max_kbps: 128, screenshare_max_kbps: 3000, screenshare_max_fps: 60 },
|
||||
);
|
||||
assert.equal(payload.audioMaxBitrate, 32_000);
|
||||
// user 'auto' but room caps screenshare bitrate -> room cap applies
|
||||
assert.equal(payload.screenshareMaxBitrate, 3_000_000);
|
||||
assert.equal(payload.screenshareMaxFramerate, 15);
|
||||
});
|
||||
|
||||
it('applies a room cap even when the user left the field on auto', () => {
|
||||
const payload = buildQualityPayload(
|
||||
{ callAudioBitrate: 'auto', screenshareBitrate: 'auto', screenshareFramerate: 'auto' },
|
||||
{ audio_max_kbps: 96 },
|
||||
);
|
||||
assert.equal(payload.audioMaxBitrate, 96_000);
|
||||
assert.equal(payload.screenshareMaxBitrate, null);
|
||||
assert.equal(payload.screenshareMaxFramerate, null);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,96 @@
|
||||
import { LotusQualityPayload } from '../plugins/call/CallControl';
|
||||
import { CallAudioBitrate, ScreenshareBitrate, ScreenshareFramerate } from '../state/settings';
|
||||
|
||||
/**
|
||||
* [P5-31] Room-level quality caps, stored in the `io.lotus.room_quality` state
|
||||
* event. Admins set a ceiling every client must stay under. Values mirror the
|
||||
* user-setting units (kbps / fps); `undefined`/absent = no cap.
|
||||
*
|
||||
* NOTE: the client applies these as a best-effort UX cap. Hard enforcement for
|
||||
* ALL Matrix clients is a server-side follow-up (a `voice-limit-guard`-style
|
||||
* sidecar on LXC 151 that reads this event) — see LOTUS_TODO.md P5-31.
|
||||
*/
|
||||
export type RoomQualityContent = {
|
||||
// Numeric caps: client-cooperative only (our fork honors them; the SFU cannot
|
||||
// enforce publisher bitrate/fps — LiveKit forwards, never transcodes).
|
||||
audio_max_kbps?: number;
|
||||
screenshare_max_kbps?: number;
|
||||
screenshare_max_fps?: number;
|
||||
// Publish-source policy: HARD-enforced server-side for ALL clients by the
|
||||
// voice-limit-guard (it re-signs the LiveKit JWT's canPublishSources).
|
||||
// Absent/true = allowed; only an explicit false forbids.
|
||||
allow_screenshare?: boolean;
|
||||
allow_camera?: boolean;
|
||||
};
|
||||
|
||||
// Selectable options (kbps / fps), shared by the settings UI and the room-admin
|
||||
// UI so they stay in sync. Values are strings so they satisfy SettingsSelect's
|
||||
// `T extends string` constraint; parsed to numbers in buildQualityPayload.
|
||||
export const AUDIO_BITRATE_OPTIONS: { value: CallAudioBitrate; label: string }[] = [
|
||||
{ value: 'auto', label: 'Auto' },
|
||||
{ value: '32', label: '32 kbps' },
|
||||
{ value: '64', label: '64 kbps' },
|
||||
{ value: '96', label: '96 kbps' },
|
||||
{ value: '128', label: '128 kbps' },
|
||||
{ value: '256', label: '256 kbps' },
|
||||
];
|
||||
|
||||
export const SCREENSHARE_BITRATE_OPTIONS: { value: ScreenshareBitrate; label: string }[] = [
|
||||
{ value: 'auto', label: 'Auto' },
|
||||
{ value: '500', label: '0.5 Mbps' },
|
||||
{ value: '1500', label: '1.5 Mbps' },
|
||||
{ value: '3000', label: '3 Mbps' },
|
||||
{ value: '8000', label: '8 Mbps' },
|
||||
];
|
||||
|
||||
export const SCREENSHARE_FRAMERATE_OPTIONS: { value: ScreenshareFramerate; label: string }[] = [
|
||||
{ value: 'auto', label: 'Auto' },
|
||||
{ value: '15', label: '15 fps' },
|
||||
{ value: '30', label: '30 fps' },
|
||||
{ value: '60', label: '60 fps' },
|
||||
];
|
||||
|
||||
/** Lower of two caps, treating `undefined` as "no cap on that side". */
|
||||
const minCap = (a: number | undefined, b: number | undefined): number | undefined => {
|
||||
if (a === undefined) return b;
|
||||
if (b === undefined) return a;
|
||||
return Math.min(a, b);
|
||||
};
|
||||
|
||||
/** Parse a setting value ('auto' | numeric string) to a number or undefined. */
|
||||
const num = (v: string): number | undefined => {
|
||||
if (v === 'auto') return undefined;
|
||||
const n = parseInt(v, 10);
|
||||
return Number.isFinite(n) ? n : undefined;
|
||||
};
|
||||
|
||||
type QualitySettings = {
|
||||
callAudioBitrate: CallAudioBitrate;
|
||||
screenshareBitrate: ScreenshareBitrate;
|
||||
screenshareFramerate: ScreenshareFramerate;
|
||||
};
|
||||
|
||||
/**
|
||||
* Build the `io.lotus.set_quality` payload from the user's settings, clamped by
|
||||
* any room-level cap. Every field is always present so clearing a setting back
|
||||
* to 'auto' sends an explicit `null` that resets the fork-side cap (otherwise a
|
||||
* previously-applied cap would stick for the rest of the call).
|
||||
*/
|
||||
export const buildQualityPayload = (
|
||||
settings: QualitySettings,
|
||||
roomCaps?: RoomQualityContent,
|
||||
): LotusQualityPayload => {
|
||||
const userAudio = num(settings.callAudioBitrate);
|
||||
const userSsBitrate = num(settings.screenshareBitrate);
|
||||
const userSsFps = num(settings.screenshareFramerate);
|
||||
|
||||
const audioKbps = minCap(userAudio, roomCaps?.audio_max_kbps);
|
||||
const ssKbps = minCap(userSsBitrate, roomCaps?.screenshare_max_kbps);
|
||||
const ssFps = minCap(userSsFps, roomCaps?.screenshare_max_fps);
|
||||
|
||||
return {
|
||||
audioMaxBitrate: audioKbps === undefined ? null : audioKbps * 1000,
|
||||
screenshareMaxBitrate: ssKbps === undefined ? null : ssKbps * 1000,
|
||||
screenshareMaxFramerate: ssFps === undefined ? null : ssFps,
|
||||
};
|
||||
};
|
||||
@@ -0,0 +1,72 @@
|
||||
import { MatrixClient } from 'matrix-js-sdk';
|
||||
import { downloadMedia, mxcUrlToHttp } from './matrix';
|
||||
|
||||
/**
|
||||
* [P5-15] A user-uploaded soundboard clip. Stored (as a list) in the
|
||||
* `io.lotus.soundboard` account data event, so clips sync across a user's
|
||||
* devices exactly like custom emoji / sticker packs.
|
||||
*/
|
||||
export type SoundboardClip = {
|
||||
/** Stable local id (not shared with peers). */
|
||||
id: string;
|
||||
/** Display name / shortcode shown on the tile. */
|
||||
name: string;
|
||||
/** mxc:// URI of the uploaded audio. */
|
||||
url: string;
|
||||
mimetype?: string;
|
||||
size?: number;
|
||||
};
|
||||
|
||||
export type SoundboardContent = {
|
||||
clips?: SoundboardClip[];
|
||||
};
|
||||
|
||||
export const SOUNDBOARD_NAME_MAX = 24;
|
||||
/** Keep clips short: they publish to every peer and hold a track open. */
|
||||
export const SOUNDBOARD_MAX_CLIP_BYTES = 1024 * 1024; // 1 MB
|
||||
export const SOUNDBOARD_MAX_CLIPS = 40;
|
||||
export const SOUNDBOARD_ACCEPT = 'audio/mpeg,audio/ogg,audio/wav,audio/webm,audio/mp4,audio/aac';
|
||||
|
||||
// Cache resolved object URLs per mxc so re-triggering a clip doesn't re-download
|
||||
// it. Object URLs live for the page session; the set is tiny (<= MAX_CLIPS).
|
||||
const objectUrlCache = new Map<string, string>();
|
||||
|
||||
/**
|
||||
* Resolve an mxc clip to a `blob:` object URL the Element Call widget can fetch
|
||||
* without credentials. Authenticated media (MSC3916) can't be fetched from the
|
||||
* widget's realm, so the host downloads it (auth handled by the service worker)
|
||||
* and hands the widget a same-session blob URL instead.
|
||||
*/
|
||||
export const resolveClipObjectUrl = async (mx: MatrixClient, mxcUrl: string): Promise<string> => {
|
||||
const cached = objectUrlCache.get(mxcUrl);
|
||||
if (cached) return cached;
|
||||
|
||||
const httpUrl = mxcUrlToHttp(mx, mxcUrl, true);
|
||||
if (!httpUrl) throw new Error('invalid mxc url');
|
||||
const blob = await downloadMedia(httpUrl);
|
||||
const objectUrl = URL.createObjectURL(blob);
|
||||
objectUrlCache.set(mxcUrl, objectUrl);
|
||||
return objectUrl;
|
||||
};
|
||||
|
||||
/**
|
||||
* Play a resolved clip locally so the person who pressed it gets immediate
|
||||
* feedback — LiveKit doesn't loop a participant's own published track back to
|
||||
* them, so without this the presser would hear nothing. `volume` is 0–1.
|
||||
*/
|
||||
export const playClipLocally = (objectUrl: string, volume: number): void => {
|
||||
try {
|
||||
const audio = new Audio(objectUrl);
|
||||
audio.volume = Math.max(0, Math.min(1, volume));
|
||||
audio.play().catch(() => undefined);
|
||||
} catch {
|
||||
/* best effort */
|
||||
}
|
||||
};
|
||||
|
||||
export const readSoundboardClips = (mx: MatrixClient): SoundboardClip[] => {
|
||||
const content = mx.getAccountData('io.lotus.soundboard' as never)?.getContent() as
|
||||
| SoundboardContent
|
||||
| undefined;
|
||||
return Array.isArray(content?.clips) ? content.clips : [];
|
||||
};
|
||||
@@ -10,6 +10,10 @@ export enum AccountDataEvent {
|
||||
PoniesUserEmotes = 'im.ponies.user_emotes',
|
||||
PoniesEmoteRooms = 'im.ponies.emote_rooms',
|
||||
|
||||
// [P5-15] Personal, uploadable in-call soundboard clips (synced across
|
||||
// devices like custom emoji/sticker packs).
|
||||
LotusSoundboard = 'io.lotus.soundboard',
|
||||
|
||||
SecretStorageDefaultKey = 'm.secret_storage.default_key',
|
||||
|
||||
CrossSigningMaster = 'm.cross_signing.master',
|
||||
|
||||
@@ -41,6 +41,7 @@ export enum StateEvent {
|
||||
PoniesRoomEmotes = 'im.ponies.room_emotes',
|
||||
PowerLevelTags = 'in.cinny.room.power_level_tags',
|
||||
LotusVoiceLimit = 'io.lotus.voice_limit',
|
||||
LotusRoomQuality = 'io.lotus.room_quality',
|
||||
}
|
||||
|
||||
export enum MessageEvent {
|
||||
|
||||
Reference in New Issue
Block a user