Compare commits

...
12 Commits
Author SHA1 Message Date
jaredandClaude Opus 5.5 f793523e03 feat(spotlight): flip between cameras as well as screenshares during a share (#67)
CI / Build embedded bundle (push) Successful in 3m19s
CI / Publish to Gitea npm registry (push) Successful in 1m23s
During a screenshare the spotlight carousel holds the shares first, then the host-pinned camera, then every other remote camera that is on (join order). The share stays the default view; the view falls back to the first share when the visible camera turns off or leaves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPmy3tPq869XDW4njjVaKA
2026-11-09 01:10:33 -05:00
jared 576c80a989 Merge pull request 'feat(denoise): switch noise suppression mid-call (+ fix re-captures ignoring noiseSuppression)' (#66) from feat/live-denoise-switch into lotus
CI / Publish to Gitea npm registry (push) Successful in 50s
CI / Build embedded bundle (push) Skipped
2026-10-30 21:15:44 -04:00
Lotus CIandClaude Opus 5.5 949103d649 feat(denoise): switch noise suppression mid-call (io.lotus.set_denoise)
CI / Build embedded bundle (push) Skipped
CI / Build embedded bundle (pull_request) Successful in 1m47s
CI / Publish to Gitea npm registry (pull_request) Skipped
The host can now change noise suppression on the live mic without a
rejoin: off / browser-native / ML, the ML model, the gate and stacking
browser NS in front of the model.

- Model swaps preload the new model while the old one keeps running,
  then swap the processor (no mic re-capture).
- Browser NS on/off re-captures the mic via restartTrack with the new
  constraint (passing the deviceId, without which LiveKit drops every
  constraint). The choice is kept in lotusCaptureOverride, which
  ConnectionFactory reads, so a later SFU reconnect captures the same way.
- Switches are serialized; a burst collapses into the last one. Each one
  reports io.lotus.denoise_state (now with `mode`).

Fixes found while testing in Chromium:
- The mic level meter (and muted-speech nudge) kept a clone of the
  ORIGINAL mic track and never noticed LiveKit's restartTrack. Chrome
  makes all open tracks of a device share audio processing, so every
  re-capture silently kept the old noiseSuppression (asked false, got
  true). The meter now drops its clone around our re-capture and
  re-taps on TrackEvent.Restarted, which also fixes it listening to the
  old device after a mic switch.
- LiveKit defaults voiceIsolation ON, a further voice filter in "off"
  and ML modes where the platform implements it. It now follows browser
  noise suppression.

Verified in a two-browser call: joined as browser and as ML(dtln), then
switched dtln -> deepfilternet -> rnnoise -> browser -> off; each step
applied within ~0.5 s and every re-capture got the constraints it asked
for.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5
2026-10-30 21:13:40 -04:00
jared 9e6fe75f14 Merge pull request 'fix(denoise): DeepFilterNet attenuation limit 60 → 25 dB (choppy word endings)' (#65) from fix/dfn-strength-25 into lotus
CI / Build embedded bundle (push) Successful in 5m59s
CI / Publish to Gitea npm registry (push) Successful in 4m12s
2026-10-30 18:58:29 -04:00
Lotus CIandClaude Opus 5.5 f180f76196 fix(denoise): DeepFilterNet attenuation limit 60 -> 25 dB
CI / Build embedded bundle (push) Skipped
CI / Build embedded bundle (pull_request) Successful in 8m5s
CI / Publish to Gitea npm registry (pull_request) Skipped
At 60 dB the background is muted outright (measured -61 to -64 dB), so
each word drops into dead silence and listeners hear the mic cutting out
at word ends. On room-echo speech (RT60 ~0.4 s) with fan and keyboard
noise at SNR 15 dB, 25 dB scored best:

  limit  PESQ clean/fan/keyboard   noise removed
  60     3.35 / 2.33 / 2.04        ~62 dB
  40     3.41 / 2.45 / 2.21        32-40 dB
  25     3.59 / 2.72 / 2.38        25 dB
  20     3.63 / 2.66 / 2.25        20 dB

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5
2026-10-30 18:50:06 -04:00
jared 4ca3b0695c Merge pull request 'fix(denoise): stop DeepFilterNet silently passing raw audio through' (#64) from fix/dfn-model-bytes into lotus
CI / Build embedded bundle (push) Successful in 8m41s
CI / Publish to Gitea npm registry (push) Skipped
2026-10-30 15:16:25 -04:00
Lotus CIandClaude Opus 5.5 283b231c7a style: oxfmt
CI / Build embedded bundle (push) Skipped
CI / Build embedded bundle (pull_request) Successful in 5m2s
CI / Publish to Gitea npm registry (pull_request) Skipped
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5
2026-10-30 15:11:19 -04:00
Lotus CIandClaude Opus 5.5 16f5f3d1a3 fix(denoise): stop DeepFilterNet silently passing raw audio through
CI / Build embedded bundle (push) Skipped
CI / Build embedded bundle (pull_request) Failing after 5m38s
CI / Publish to Gitea npm registry (pull_request) Skipped
If the DeepFilterNet model is served with `Content-Encoding: gzip` (vite
preview does this for .tar.gz; some CDNs and nginx gzip_static setups do
too), fetch() hands back the already-unpacked tar. The library's worklet
then fails inside df_create, swallows the error and passes the raw mic
through, while Lotus Call reports DeepFilterNet as active.

Check the bytes after initialize(): a gzip model is used as-is, an
unpacked tar is re-gzipped in the browser, and anything else throws so
the rnnoise fallback runs and the host is told the truth.

Measured in a two-browser call (fan noise, SNR 15 dB): before, 4.9 dB of
noise removed, i.e. none beyond gain differences; after, 30.9 dB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5
2026-10-30 14:18:01 -04:00
jared 117467b1fb Merge pull request 'chore(deps): security fixes for dev/build tooling (Storybook, Vitest, Vite, transitive)' (#63) from chore/dev-tool-security into lotus
CI / Build embedded bundle (push) Successful in 5m37s
CI / Publish to Gitea npm registry (push) Skipped
2026-10-25 23:59:03 -04:00
Lotus CIandClaude Opus 5.5 43cbd8409d chore(deps): security fixes for dev/build tooling (Storybook, Vitest, Vite, transitive)
CI / Build embedded bundle (push) Skipped
CI / Build embedded bundle (pull_request) Successful in 4m6s
CI / Publish to Gitea npm registry (pull_request) Skipped
The npm advisory database flagged 19 packages in pnpm-lock.yaml. None of them
ship in the embedded call bundle (checked against the built dist); they run
on developer machines and in CI:
- storybook / @storybook/* 10.6.0 -> 10.6.3: critical, dev server code
  injection from the local network.
- vitest 4.1.12 (+ @vitest/ui unpinned to ^4.1.12), tinypool ^2.1.2:
  mock registration code injection, prototype-pollution RCE.
- vite ^8.3.5: dev-server file exposure, launch-editor issues.
- Overrides for transitive tooling: brace-expansion, browserslist,
  smol-toml, source-map-js, postcss-selector-parser, baseline-browser-
  mapping, fflate, pbkdf2, zod 4; existing qs/js-yaml/undici/dompurify
  overrides raised to the fixed versions.

Left: elliptic (no fixed release, low), zod 3 (no 3.x fix; a dependency
pins it). Lotus unit tests 144/144, build:embedded OK.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5
2026-10-25 23:50:09 -04:00
jared 4c84c3a4f3 Merge pull request 'feat(lotus): expected-device check before joining (cinny #121)' (#62) from feat/expected-devices into lotus
CI / Build embedded bundle (push) Successful in 4m12s
CI / Publish to Gitea npm registry (push) Successful in 1m32s
2026-10-24 19:47:38 -04:00
Lotus CIandClaude Opus 5.5 a366243836 feat(lotus): expected-device check before joining (cinny #121)
CI / Build embedded bundle (push) Skipped
CI / Build embedded bundle (pull_request) Successful in 4m21s
CI / Publish to Gitea npm registry (pull_request) Skipped
EC remembers the picked mic / output by id and silently falls back to the
first device when it is missing, so a user who always joins with a headset
lands on the laptop mic. With lotusDeviceCheck=1:
- the last-picked device's NAME is remembered next to its id
  (lotus-audio-{input,output}-name), since ids can change on re-plug;
- before the automatic join, a usual device that is gone is reported to the
  host (io.lotus.device_missing) and EC waits for io.lotus.device_decision
  { join }; one that came back under a new id is just re-selected;
- in a call, losing it is reported (io.lotus.device_lost) for a toast.
Fails open (no names / no host / nothing remembered). Without the flag the
join stays synchronous, as upstream. The gate runs once per call: CallView's
effect re-runs while its inputs settle and must not restart it.

Embedded version 0.28.0-lotus.2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5
2026-10-24 19:42:23 -04:00
25 changed files with 1822 additions and 421 deletions
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "@lotusguild/element-call-embedded", "name": "@lotusguild/element-call-embedded",
"version": "0.28.0-lotus.1", "version": "0.28.0-lotus.2",
"files": [ "files": [
"README.md", "README.md",
"LICENSE-AGPL-3.0", "LICENSE-AGPL-3.0",
+8 -8
View File
@@ -64,9 +64,9 @@
"@rolldown/plugin-babel": "^0.2.3", "@rolldown/plugin-babel": "^0.2.3",
"@sentry/react": "^8.0.0", "@sentry/react": "^8.0.0",
"@sentry/vite-plugin": "^3.0.0", "@sentry/vite-plugin": "^3.0.0",
"@storybook/addon-docs": "^10.3.6", "@storybook/addon-docs": "^10.6.3",
"@storybook/addon-vitest": "^10.3.6", "@storybook/addon-vitest": "^10.6.3",
"@storybook/react-vite": "^10.3.6", "@storybook/react-vite": "^10.6.3",
"@testing-library/dom": "^10.1.0", "@testing-library/dom": "^10.1.0",
"@testing-library/jest-dom": "^6.6.3", "@testing-library/jest-dom": "^6.6.3",
"@testing-library/react": "^16.0.0", "@testing-library/react": "^16.0.0",
@@ -86,9 +86,9 @@
"@vector-im/compound-design-tokens": "^11.0.0", "@vector-im/compound-design-tokens": "^11.0.0",
"@vector-im/compound-web": "^10.0.0", "@vector-im/compound-web": "^10.0.0",
"@vitejs/plugin-react": "^6.0.2", "@vitejs/plugin-react": "^6.0.2",
"@vitest/browser-playwright": "^4.1.5", "@vitest/browser-playwright": "^4.1.12",
"@vitest/coverage-v8": "^4.0.18", "@vitest/coverage-v8": "^4.0.18",
"@vitest/ui": "4.1.11", "@vitest/ui": "^4.1.12",
"babel-plugin-react-compiler": "^1.0.0", "babel-plugin-react-compiler": "^1.0.0",
"classnames": "^2.3.1", "classnames": "^2.3.1",
"copy-to-clipboard": "^3.3.3", "copy-to-clipboard": "^3.3.3",
@@ -123,18 +123,18 @@
"react-use-measure": "^2.1.1", "react-use-measure": "^2.1.1",
"rxjs": "^7.8.1", "rxjs": "^7.8.1",
"sass": "^1.42.1", "sass": "^1.42.1",
"storybook": "^10.3.6", "storybook": "^10.6.3",
"typescript": "^5.8.3", "typescript": "^5.8.3",
"unique-names-generator": "^4.6.0", "unique-names-generator": "^4.6.0",
"uuid": "^14.0.0", "uuid": "^14.0.0",
"vaul": "^1.0.0", "vaul": "^1.0.0",
"vite": "^8.0.0", "vite": "^8.3.5",
"vite-plugin-html": "^3.2.2", "vite-plugin-html": "^3.2.2",
"vite-plugin-node-polyfills": "^0.28.0", "vite-plugin-node-polyfills": "^0.28.0",
"vite-plugin-node-stdlib-browser": "^0.2.1", "vite-plugin-node-stdlib-browser": "^0.2.1",
"vite-plugin-svgr": "^4.0.0", "vite-plugin-svgr": "^4.0.0",
"vite-plugin-wasm": "^3.6.0", "vite-plugin-wasm": "^3.6.0",
"vitest": "^4.1.5", "vitest": "^4.1.12",
"vitest-axe": "^1.0.0-pre.3" "vitest-axe": "^1.0.0-pre.3"
}, },
"packageManager": "pnpm@11.21.0+sha512.521705bce689924eac72f5a3587122f362689ef6571e55ba80076fd637c11132ecffada26fad4ea79c485bfddbfd3d5a2a5b05805a77e893de71ec8a6cca3bb1" "packageManager": "pnpm@11.21.0+sha512.521705bce689924eac72f5a3587122f362689ef6571e55ba80076fd637c11132ecffada26fad4ea79c485bfddbfd3d5a2a5b05805a77e893de71ec8a6cca3bb1"
+12 -6
View File
@@ -71,8 +71,9 @@ widgetTest("Sharing screen in group call", async ({ addUser, browserName }) => {
.locator('iframe[title="Element Call"]') .locator('iframe[title="Element Call"]')
.contentFrame(); .contentFrame();
// Expect 4 (3 + screen share) video tiles // Expect 6 video tiles: 3 cameras in the grid, plus the spotlight
await expect(frame.locator("video")).toHaveCount(4, { // carousel's screen share and the 2 remote cameras (lotus #67)
await expect(frame.locator("video")).toHaveCount(6, {
timeout: 5000, timeout: 5000,
}); });
@@ -112,8 +113,9 @@ widgetTest("Sharing screen in group call", async ({ addUser, browserName }) => {
.locator('iframe[title="Element Call"]') .locator('iframe[title="Element Call"]')
.contentFrame(); .contentFrame();
// Expect 5 (2 + screen share) video tiles // Expect 7 video tiles: 3 cameras in the grid, plus the spotlight
await expect(frame.locator("video")).toHaveCount(5, { // carousel's 2 screen shares and 2 remote cameras (lotus #67)
await expect(frame.locator("video")).toHaveCount(7, {
timeout: 5000, timeout: 5000,
}); });
@@ -121,8 +123,12 @@ widgetTest("Sharing screen in group call", async ({ addUser, browserName }) => {
frame.locator('video[data-lk-source="screen_share"]'), frame.locator('video[data-lk-source="screen_share"]'),
).toHaveCount(2); ).toHaveCount(2);
// Expect 2 indicators at the bottom // Expect 4 indicators at the bottom: the 2 shares first, then the 2
await expect(frame.getByTestId("screenshare-indicator")).toHaveCount(2); // remote cameras (lotus #67)
await expect(frame.getByTestId("screenshare-indicator")).toHaveCount(4);
await expect(
frame.locator('[data-testid="screenshare-indicator"][data-kind="user"]'),
).toHaveCount(2);
// Check the first indicator is visible // Check the first indicator is visible
await expect( await expect(
+418 -348
View File
File diff suppressed because it is too large Load Diff
+18 -4
View File
@@ -28,24 +28,38 @@ overrides:
# - https://github.com/advisories/GHSA-7r86-cg39-jmmj # - https://github.com/advisories/GHSA-7r86-cg39-jmmj
# - https://github.com/advisories/GHSA-23c5-xmqv-rm74 # - https://github.com/advisories/GHSA-23c5-xmqv-rm74
"minimatch": "^10.2.3" "minimatch": "^10.2.3"
# Security fixes for transitive build/test tooling (none of these ship in the
# embedded call bundle; checked against the npm advisory database 2026-10-25).
"tinypool": "^2.1.2"
"@vitest/mocker": "^4.1.12"
"brace-expansion@>=4": "^5.0.12"
"browserslist": "^4.28.7"
"smol-toml": "^1.8.1"
"source-map-js": "^1.2.2"
"postcss-selector-parser@>=7": "^7.1.6"
"baseline-browser-mapping": "^2.11.0"
"fflate@<0.5": "^0.4.9"
"fflate@>=0.8": "^0.8.3"
"pbkdf2": "^3.1.7"
"zod@>=4": "^4.6.6"
# Security fix: https://github.com/element-hq/element-call/security/dependabot/109 # Security fix: https://github.com/element-hq/element-call/security/dependabot/109
"glob": "^10.5.0" "glob": "^10.5.0"
# Security fixes: # Security fixes:
# - https://github.com/element-hq/element-call/security/dependabot/110 # - https://github.com/element-hq/element-call/security/dependabot/110
# - https://github.com/element-hq/element-call/security/dependabot/122 # - https://github.com/element-hq/element-call/security/dependabot/122
# - https://github.com/element-hq/element-call/security/dependabot/179 # - https://github.com/element-hq/element-call/security/dependabot/179
"qs": "^6.15.2" "qs": "^6.16.0"
# Security fixes: # Security fixes:
# - https://github.com/element-hq/element-call/security/dependabot/106 # - https://github.com/element-hq/element-call/security/dependabot/106
# - https://github.com/element-hq/element-call/security/dependabot/203 # - https://github.com/element-hq/element-call/security/dependabot/203
"js-yaml": "^4.2.0" "js-yaml": "^4.3.2"
# Storybook declares support for 0.27.0 only but empirically works fine with 0.28.0. # Storybook declares support for 0.27.0 only but empirically works fine with 0.28.0.
"esbuild": "^0.28.0" "esbuild": "^0.28.0"
# Security fixes: # Security fixes:
# - https://github.com/nodejs/undici/releases/tag/v6.24.0 # - https://github.com/nodejs/undici/releases/tag/v6.24.0
# - https://github.com/element-hq/element-call/security/dependabot/201 # - https://github.com/element-hq/element-call/security/dependabot/201
# - https://github.com/element-hq/element-call/security/dependabot/199 # - https://github.com/element-hq/element-call/security/dependabot/199
"undici": "^6.27.0" "undici": "^6.29.1"
# Security fix: https://github.com/advisories/GHSA-rf6f-7fwh-wjgh # Security fix: https://github.com/advisories/GHSA-rf6f-7fwh-wjgh
"flatted": "^3.4.2" "flatted": "^3.4.2"
# Security fix: https://github.com/element-hq/element-call/security/dependabot/184 # Security fix: https://github.com/element-hq/element-call/security/dependabot/184
@@ -59,4 +73,4 @@ overrides:
# - https://github.com/element-hq/element-call/security/dependabot/196 # - https://github.com/element-hq/element-call/security/dependabot/196
# - https://github.com/element-hq/element-call/security/dependabot/195 # - https://github.com/element-hq/element-call/security/dependabot/195
# - https://github.com/element-hq/element-call/security/dependabot/193 # - https://github.com/element-hq/element-call/security/dependabot/193
"dompurify": "^3.4.11" "dompurify": "^3.4.17"
+2
View File
@@ -35,6 +35,8 @@ describe("LotusWidgetActions", () => {
LotusWidgetActions.SetFrameScreenshare, LotusWidgetActions.SetFrameScreenshare,
LotusWidgetActions.PromptScreenshare, LotusWidgetActions.PromptScreenshare,
LotusWidgetActions.Reconnect, LotusWidgetActions.Reconnect,
LotusWidgetActions.DeviceDecision,
LotusWidgetActions.SetDenoise,
]; ];
expect(new Set(LOTUS_TO_WIDGET_ACTIONS)).toEqual(new Set(expectedToWidget)); expect(new Set(LOTUS_TO_WIDGET_ACTIONS)).toEqual(new Set(expectedToWidget));
+25 -1
View File
@@ -41,11 +41,19 @@ export enum LotusWidgetActions {
RequestState = "io.lotus.request_state", RequestState = "io.lotus.request_state",
/** /**
* fromWidget: real state of the in-source denoise engine — * fromWidget: real state of the in-source denoise engine —
* `{ active: boolean, model: string, error?: string }` — sent once the * `{ active: boolean, model: string, mode: "ml", error?: string }` (or
* `{ active: false, mode }` after a switch away from ML) — sent once the
* processor attaches (or after the rnnoise fallback also fails), so the host * processor attaches (or after the rnnoise fallback also fails), so the host
* toggle can reflect reality rather than the requested state. * toggle can reflect reality rather than the requested state.
*/ */
DenoiseState = "io.lotus.denoise_state", DenoiseState = "io.lotus.denoise_state",
/**
* toWidget: switch noise suppression mid-call —
* `{ mode: "off" | "browser" | "ml", model?, nativeNS?, gate?, gateThreshold? }`.
* `model`/`gate`/`gateThreshold` only matter for "ml"; `nativeNS` stacks the
* browser suppressor in front of the model. Omitted fields keep their value.
*/
SetDenoise = "io.lotus.set_denoise",
/** /**
* fromWidget: one-shot end-of-call readout for the local participant — * fromWidget: one-shot end-of-call readout for the local participant —
* `{ durationMs, reconnects, poorMs, verdict }` (#143) — sent on SFU * `{ durationMs, reconnects, poorMs, verdict }` (#143) — sent on SFU
@@ -130,6 +138,20 @@ export enum LotusWidgetActions {
CallError = "io.lotus.call_error", CallError = "io.lotus.call_error",
/** toWidget: press the error page's Reconnect / Try again (cinny #318). */ /** toWidget: press the error page's Reconnect / Try again (cinny #318). */
Reconnect = "io.lotus.reconnect", Reconnect = "io.lotus.reconnect",
/**
* fromWidget: before the automatic join (`lotusDeviceCheck=1`), the user's
* usual mic / output is missing — `{ missing: [{ kind: "audioinput" |
* "audiooutput", expected: name, fallback: name | null }] }`. EC waits for
* `DeviceDecision` (cinny #121).
*/
DeviceMissing = "io.lotus.device_missing",
/** toWidget: the user's answer, `{ join: boolean }` (cinny #121). */
DeviceDecision = "io.lotus.device_decision",
/**
* fromWidget: in a call, the usual device disappeared — `{ kind, expected,
* fallback }` — for a toast (cinny #121).
*/
DeviceLost = "io.lotus.device_lost",
} }
/** toWidget Lotus actions that must be allow-listed in `initializeWidget`. */ /** toWidget Lotus actions that must be allow-listed in `initializeWidget`. */
@@ -148,4 +170,6 @@ export const LOTUS_TO_WIDGET_ACTIONS: LotusWidgetActions[] = [
LotusWidgetActions.SetFrameScreenshare, LotusWidgetActions.SetFrameScreenshare,
LotusWidgetActions.PromptScreenshare, LotusWidgetActions.PromptScreenshare,
LotusWidgetActions.Reconnect, LotusWidgetActions.Reconnect,
LotusWidgetActions.DeviceDecision,
LotusWidgetActions.SetDenoise,
]; ];
+15
View File
@@ -25,6 +25,7 @@ import {
echoCancellationSetting, echoCancellationSetting,
noiseSuppressionSetting, noiseSuppressionSetting,
} from "../settings/settings.ts"; } from "../settings/settings.ts";
import { setLotusNoiseSuppression } from "./lotusCaptureOverride.ts";
// [lotus] Upstream v0.25.0 moved the audio-capture constraints to Settings. // [lotus] Upstream v0.25.0 moved the audio-capture constraints to Settings.
// The Lotus host still drives them per-call via URL params (it turns browser // The Lotus host still drives them per-call via URL params (it turns browser
@@ -83,6 +84,7 @@ beforeEach(() => {
}); });
afterEach(() => { afterEach(() => {
setLotusNoiseSuppression(null);
testScope.end(); testScope.end();
vi.mocked(LivekitRoom).mockClear(); vi.mocked(LivekitRoom).mockClear();
}); });
@@ -123,6 +125,7 @@ describe("[lotus] audio-capture URL param overrides", () => {
expect(capturedAudioDefaults()).toMatchObject({ expect(capturedAudioDefaults()).toMatchObject({
echoCancellation: true, echoCancellation: true,
noiseSuppression: false, noiseSuppression: false,
voiceIsolation: false,
autoGainControl: false, autoGainControl: false,
}); });
}); });
@@ -137,4 +140,16 @@ describe("[lotus] audio-capture URL param overrides", () => {
createRoom(); createRoom();
expect(capturedAudioDefaults()).toMatchObject({ echoCancellation: false }); expect(capturedAudioDefaults()).toMatchObject({ echoCancellation: false });
}); });
test("a mid-call set_denoise choice beats the noiseSuppression the call joined with", () => {
getUrlParams.mockReturnValue({ noiseSuppression: false });
setLotusNoiseSuppression(true);
createRoom();
expect(capturedAudioDefaults()).toMatchObject({ noiseSuppression: true });
getUrlParams.mockReturnValue({ noiseSuppression: true });
setLotusNoiseSuppression(false);
createRoom();
expect(capturedAudioDefaults()).toMatchObject({ noiseSuppression: false });
});
}); });
+26
View File
@@ -0,0 +1,26 @@
/*
Copyright 2026 Lotus Guild
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
/**
* Browser noise suppression chosen live by the host (`io.lotus.set_denoise`),
* overriding the `noiseSuppression` URL param the call was opened with. Read by
* ConnectionFactory, so a connection created after a mid-call switch (e.g. an
* SFU reconnect) captures the mic the way the user last picked, not the way
* they joined. Dependency-free so ConnectionFactory can import it cheaply.
*/
let noiseSuppressionOverride: boolean | null = null;
export function setLotusNoiseSuppression(on: boolean | null): void {
noiseSuppressionOverride = on;
}
/** The override if the host set one, else the URL param (default on). */
export function lotusNoiseSuppressionAllowed(
urlParam: boolean | undefined,
): boolean {
return noiseSuppressionOverride ?? urlParam !== false;
}
+181 -2
View File
@@ -9,7 +9,12 @@ import { afterEach, beforeEach, describe, expect, test, vi } from "vitest";
import { ParticipantEvent, Track } from "livekit-client"; import { ParticipantEvent, Track } from "livekit-client";
import { type CallViewModel } from "../state/CallViewModel/CallViewModel"; import { type CallViewModel } from "../state/CallViewModel/CallViewModel";
import { startLotusDenoise } from "./lotusDenoise"; import {
applySetDenoise,
browserNsFor,
type DenoiseTarget,
startLotusDenoise,
} from "./lotusDenoise";
import { LotusWidgetActions } from "./lotusActions"; import { LotusWidgetActions } from "./lotusActions";
// Track constructed instances so tests can assert exactly one processor was // Track constructed instances so tests can assert exactly one processor was
@@ -42,9 +47,25 @@ vi.mock("./lotusDenoiseProcessor", () => ({
releasePreparedDenoiseAssets: vi.fn().mockResolvedValue(undefined), releasePreparedDenoiseAssets: vi.fn().mockResolvedValue(undefined),
})); }));
const hostSend = vi.hoisted(() => vi.fn().mockResolvedValue({})); const hostSend = vi.hoisted(() => vi.fn().mockResolvedValue({}));
// Host → widget actions: tests call `sendAction()` to deliver one.
const actionHandlers = vi.hoisted(
() => new Map<string, (ev: CustomEvent) => void>(),
);
vi.mock("../widget", () => ({ vi.mock("../widget", () => ({
widget: { api: { transport: { send: hostSend } } }, widget: {
api: { transport: { send: hostSend, reply: vi.fn() } },
lazyActions: {
on: (name: string, cb: (ev: CustomEvent) => void): void => {
actionHandlers.set(name, cb);
},
off: (name: string): void => {
actionHandlers.delete(name);
},
},
},
})); }));
const sendAction = (name: string, data: unknown): void =>
actionHandlers.get(name)?.(new CustomEvent(name, { detail: { data } }));
/** A promise plus externally-callable resolve, for controlling ordering. */ /** A promise plus externally-callable resolve, for controlling ordering. */
function deferred<T>(): { function deferred<T>(): {
@@ -110,6 +131,7 @@ beforeEach(() => {
instances.length = 0; instances.length = 0;
calls.length = 0; calls.length = 0;
hostSend.mockClear(); hostSend.mockClear();
actionHandlers.clear();
// `lotusParam`/`lotusFlag` cache the URL params at first read; seed the hash // `lotusParam`/`lotusFlag` cache the URL params at first read; seed the hash
// before startLotusDenoise() so the dedicated `lotusDenoiseSource` flag reads on. // before startLotusDenoise() so the dedicated `lotusDenoiseSource` flag reads on.
window.location.hash = "#/room?lotusDenoiseSource=1"; window.location.hash = "#/room?lotusDenoiseSource=1";
@@ -188,6 +210,7 @@ describe("startLotusDenoise", () => {
expect(hostSend).toHaveBeenCalledWith(LotusWidgetActions.DenoiseState, { expect(hostSend).toHaveBeenCalledWith(LotusWidgetActions.DenoiseState, {
active: true, active: true,
model: "rnnoise", model: "rnnoise",
mode: "ml",
}); });
}); });
@@ -217,6 +240,7 @@ describe("startLotusDenoise", () => {
expect(hostSend).toHaveBeenCalledWith(LotusWidgetActions.DenoiseState, { expect(hostSend).toHaveBeenCalledWith(LotusWidgetActions.DenoiseState, {
active: true, active: true,
model: "rnnoise", model: "rnnoise",
mode: "ml",
}); });
}); });
@@ -239,6 +263,7 @@ describe("startLotusDenoise", () => {
active: false, active: false,
error: "no worklet for you", error: "no worklet for you",
model: "rnnoise", model: "rnnoise",
mode: "ml",
}); });
}); });
@@ -272,3 +297,157 @@ describe("startLotusDenoise", () => {
expect(instances[0].setMicMuted).not.toHaveBeenCalled(); expect(instances[0].setMicMuted).not.toHaveBeenCalled();
}); });
}); });
/** A mic whose processor and capture constraints behave like LiveKit's. */
function liveMic(): {
getProcessor: () => { config: { model: string } } | undefined;
setProcessor: ReturnType<typeof vi.fn>;
stopProcessor: ReturnType<typeof vi.fn>;
restartTrack: ReturnType<typeof vi.fn>;
constraints: MediaTrackConstraints;
mediaStreamTrack: { getSettings: () => MediaTrackSettings };
isMuted: boolean;
} {
let attached: { config: { model: string } } | undefined;
return {
isMuted: false,
getProcessor: () => attached,
setProcessor: vi.fn(async (p: { config: { model: string } }) => {
await Promise.resolve();
attached = p;
}),
stopProcessor: vi.fn(async () => {
await Promise.resolve();
attached = undefined;
}),
restartTrack: vi.fn().mockResolvedValue(undefined),
constraints: { echoCancellation: true, autoGainControl: true },
mediaStreamTrack: { getSettings: () => ({ deviceId: "mic-1" }) },
};
}
describe("io.lotus.set_denoise (switch mid-call)", () => {
test("swaps the ML model on the live mic without re-capturing it", async () => {
window.location.hash =
"#/room?lotusDenoiseSource=1&lotusModel=dtln&noiseSuppression=false";
const mic = liveMic();
const { vm } = makeRoomAndVm(mic);
startLotusDenoise(vm);
await flush();
expect(mic.getProcessor()?.config.model).toBe("dtln");
sendAction(LotusWidgetActions.SetDenoise, {
mode: "ml",
model: "deepfilternet",
});
await flush();
await flush();
expect(mic.stopProcessor).toHaveBeenCalledTimes(1);
expect(mic.getProcessor()?.config.model).toBe("deepfilternet");
expect(mic.restartTrack).not.toHaveBeenCalled();
expect(hostSend).toHaveBeenLastCalledWith(LotusWidgetActions.DenoiseState, {
active: true,
model: "deepfilternet",
mode: "ml",
});
});
test("ML → browser-native drops the model and re-captures with browser NS on", async () => {
window.location.hash =
"#/room?lotusDenoiseSource=1&lotusModel=rnnoise&noiseSuppression=false";
const mic = liveMic();
const { vm } = makeRoomAndVm(mic);
startLotusDenoise(vm);
await flush();
sendAction(LotusWidgetActions.SetDenoise, { mode: "browser" });
await flush();
await flush();
expect(mic.stopProcessor).toHaveBeenCalledTimes(1);
expect(mic.getProcessor()).toBeUndefined();
expect(mic.restartTrack).toHaveBeenCalledWith({
deviceId: "mic-1",
channelCount: 1,
echoCancellation: true,
autoGainControl: true,
noiseSuppression: true,
voiceIsolation: true,
});
expect(hostSend).toHaveBeenLastCalledWith(LotusWidgetActions.DenoiseState, {
active: false,
mode: "browser",
});
});
test("browser-native → ML works even when the call joined without ML", async () => {
window.location.hash = "#/room?noiseSuppression=true";
const mic = liveMic();
const { vm } = makeRoomAndVm(mic);
startLotusDenoise(vm);
await flush();
expect(mic.setProcessor).not.toHaveBeenCalled();
sendAction(LotusWidgetActions.SetDenoise, { mode: "ml", model: "speex" });
await flush();
await flush();
expect(mic.restartTrack).toHaveBeenCalledWith(
expect.objectContaining({ noiseSuppression: false }),
);
expect(mic.getProcessor()?.config.model).toBe("speex");
});
test("ignores a payload without a valid mode", async () => {
window.location.hash = "#/room?lotusDenoiseSource=1";
const mic = liveMic();
const { vm } = makeRoomAndVm(mic);
startLotusDenoise(vm);
await flush();
sendAction(LotusWidgetActions.SetDenoise, { mode: "loud" });
await flush();
expect(mic.stopProcessor).not.toHaveBeenCalled();
expect(mic.restartTrack).not.toHaveBeenCalled();
});
});
describe("applySetDenoise", () => {
const base: DenoiseTarget = {
mode: "ml",
nativeNS: false,
config: {
model: "rnnoise",
assetBase: "https://x/denoise/",
gate: false,
gateThreshold: -45,
floor: 0.15,
},
};
test("keeps fields the payload doesn't name or gets wrong", () => {
expect(
applySetDenoise(base, { mode: "ml", model: "nope", gate: "yes" }),
).toEqual(base);
});
test("moves only the named fields", () => {
const next = applySetDenoise(base, {
mode: "ml",
model: "dtln",
gateThreshold: -50,
});
expect(next?.config).toEqual({
...base.config,
model: "dtln",
gateThreshold: -50,
});
});
test("browser NS is on for browser mode and for ML with nativeNS", () => {
expect(browserNsFor({ ...base, mode: "browser" })).toBe(true);
expect(browserNsFor({ ...base, mode: "off" })).toBe(false);
expect(browserNsFor(base)).toBe(false);
expect(browserNsFor({ ...base, nativeNS: true })).toBe(true);
});
});
+184 -29
View File
@@ -13,8 +13,12 @@ import {
type TrackPublication, type TrackPublication,
} from "livekit-client"; } from "livekit-client";
import { logger } from "matrix-js-sdk/lib/logger"; import { logger } from "matrix-js-sdk/lib/logger";
import { type IWidgetApiRequest } from "matrix-widget-api";
import { type CallViewModel } from "../state/CallViewModel/CallViewModel"; import { type CallViewModel } from "../state/CallViewModel/CallViewModel";
import { widget } from "../widget";
import { setLotusNoiseSuppression } from "./lotusCaptureOverride";
import { withMicTapsReleased } from "./lotusMicLevel";
import { import {
LotusWidgetActions, LotusWidgetActions,
lotusFlag, lotusFlag,
@@ -64,49 +68,135 @@ function safeAssetBase(raw: string | null): string {
} }
} }
export type LotusNoiseMode = "off" | "browser" | "ml";
/** What the host wants: browser suppression and/or an ML model. */
export interface DenoiseTarget {
mode: LotusNoiseMode;
/** Stack the browser's suppressor in front of the model (ML only). */
nativeNS: boolean;
config: LotusDenoiseConfig;
}
const MODELS: ReadonlySet<string> = new Set<LotusDenoiseModel>([
"rnnoise",
"speex",
"dtln",
"deepfilternet",
]);
const isModel = (v: unknown): v is LotusDenoiseModel =>
typeof v === "string" && MODELS.has(v);
/** Whether the mic should be captured with the browser's noise suppression. */
export const browserNsFor = (t: DenoiseTarget): boolean =>
t.mode === "browser" || (t.mode === "ml" && t.nativeNS);
/**
* Apply an `io.lotus.set_denoise` payload to the current target. Unknown or
* malformed fields keep their current value, so a partial payload only moves
* what it names; a payload with no valid `mode` is ignored (returns null).
*/
export function applySetDenoise(
current: DenoiseTarget,
data: unknown,
): DenoiseTarget | null {
const d = (data ?? {}) as Record<string, unknown>;
if (d.mode !== "off" && d.mode !== "browser" && d.mode !== "ml") return null;
const model = isModel(d.model) ? d.model : current.config.model;
return {
mode: d.mode,
nativeNS: typeof d.nativeNS === "boolean" ? d.nativeNS : current.nativeNS,
config: {
...current.config,
model,
gate: typeof d.gate === "boolean" ? d.gate : current.config.gate,
gateThreshold:
typeof d.gateThreshold === "number" && Number.isFinite(d.gateThreshold)
? d.gateThreshold
: current.config.gateThreshold,
},
};
}
const sameMlConfig = (a: LotusDenoiseConfig, b: LotusDenoiseConfig): boolean =>
a.model === b.model &&
a.gate === b.gate &&
a.gateThreshold === b.gateThreshold &&
a.floor === b.floor;
/**
* Re-acquire the mic with browser noise suppression on or off, keeping the
* rest of its constraints. LiveKit re-attaches any processor to the new track;
* the level-meter clone is dropped first so the new constraint actually takes.
* The deviceId must be passed: without one LiveKit requests `audio: true` and
* silently drops every other constraint.
*/
async function restartCapture(
mic: LocalAudioTrack,
noiseSuppression: boolean,
): Promise<void> {
const c = mic.constraints;
await withMicTapsReleased(async () => {
await mic.restartTrack({
deviceId: c.deviceId ?? mic.mediaStreamTrack.getSettings().deviceId,
channelCount: c.channelCount ?? 1,
echoCancellation: c.echoCancellation,
autoGainControl: c.autoGainControl,
noiseSuppression,
voiceIsolation: noiseSuppression,
});
});
}
export function startLotusDenoise(vm: CallViewModel): () => void { export function startLotusDenoise(vm: CallViewModel): () => void {
// Gate on a DEDICATED flag — NOT the existing `lotusDenoise=ml` that the // Gate on a DEDICATED flag — NOT the existing `lotusDenoise=ml` that the
// host's build-time getUserMedia shim already uses. Otherwise simply shipping // host's build-time getUserMedia shim already uses. Otherwise simply shipping
// this fork (while the host still injects its shim and still sets // this fork (while the host still injects its shim and still sets
// lotusDenoise=ml) would denoise twice. The host opts into the in-source // lotusDenoise=ml) would denoise twice. The host opts into the in-source
// engine with `lotusDenoiseSource=1` AND stops injecting the shim at the same // engine with `lotusDenoiseSource=1` AND stops injecting the shim at the same
// time. Default off ⇒ the fork is inert and behaviour is unchanged. // time. Without the flag the engine stays idle unless the host switches to ML
if (!lotusFlag("lotusDenoiseSource")) return () => undefined; // mid-call (io.lotus.set_denoise), which needs the widget API.
const w = widget;
const mlAtJoin = lotusFlag("lotusDenoiseSource");
if (!mlAtJoin && !w) return () => undefined;
const requested = lotusParam("lotusModel"); const requested = lotusParam("lotusModel");
const model: LotusDenoiseConfig["model"] = const model: LotusDenoiseConfig["model"] = isModel(requested)
requested === "speex" || ? requested
requested === "dtln" || : "rnnoise";
requested === "deepfilternet"
? requested
: "rnnoise";
const rawThreshold = lotusParam("lotusGateThreshold"); const rawThreshold = lotusParam("lotusGateThreshold");
const rawFloor = lotusParam("lotusDenoiseFloor"); const rawFloor = lotusParam("lotusDenoiseFloor");
const config: LotusDenoiseConfig = { const browserNsAtJoin = lotusParam("noiseSuppression") !== "false";
model, // `let`: io.lotus.set_denoise replaces it mid-call.
assetBase: safeAssetBase(lotusParam("lotusDenoiseBase")), let target: DenoiseTarget = {
gate: lotusFlag("lotusGate"), mode: mlAtJoin ? "ml" : browserNsAtJoin ? "browser" : "off",
// Default -45 (matches the reference shim); accept an explicit 0 (don't nativeNS: mlAtJoin && browserNsAtJoin,
// coerce it away via `|| default`). config: {
gateThreshold: model,
rawThreshold !== null && Number.isFinite(Number(rawThreshold)) assetBase: safeAssetBase(lotusParam("lotusDenoiseBase")),
? Number(rawThreshold) gate: lotusFlag("lotusGate"),
: -45, // Default -45 (matches the reference shim); accept an explicit 0 (don't
// Dry/wet attenuation floor. Default 0.15 (~-16 dB) tames the // coerce it away via `|| default`).
// over-suppression "underwater"/pumping artifact; host can tune via gateThreshold:
// `lotusDenoiseFloor` (0 = full suppression, no floor). rawThreshold !== null && Number.isFinite(Number(rawThreshold))
floor: ? Number(rawThreshold)
rawFloor !== null && Number.isFinite(Number(rawFloor)) : -45,
? Math.min(0.5, Math.max(0, Number(rawFloor))) // Dry/wet attenuation floor. Default 0.15 (~-16 dB) tames the
: 0.15, // over-suppression "underwater"/pumping artifact; host can tune via
// `lotusDenoiseFloor` (0 = full suppression, no floor).
floor:
rawFloor !== null && Number.isFinite(Number(rawFloor))
? Math.min(0.5, Math.max(0, Number(rawFloor)))
: 0.15,
},
}; };
// [lotus #7] Start fetching wasm / creating the AudioContext / addModule-ing // [lotus #7] Start fetching wasm / creating the AudioContext / addModule-ing
// the worklet (and loading the DFN model) NOW, before any mic track exists. // the worklet (and loading the DFN model) NOW, before any mic track exists.
// `setProcessor()` holds LiveKit's trackChangeLock while awaiting `init()`, // `setProcessor()` holds LiveKit's trackChangeLock while awaiting `init()`,
// so anything still loading there freezes mute/unmute/device-switch. // so anything still loading there freezes mute/unmute/device-switch.
void prepareDenoiseAssets(config); if (target.mode === "ml") void prepareDenoiseAssets(target.config);
const micOf = (room: LivekitRoom): LocalAudioTrack | undefined => const micOf = (room: LivekitRoom): LocalAudioTrack | undefined =>
room.localParticipant.getTrackPublication(Track.Source.Microphone) room.localParticipant.getTrackPublication(Track.Source.Microphone)
@@ -127,7 +217,7 @@ export function startLotusDenoise(vm: CallViewModel): () => void {
mic: LocalAudioTrack, mic: LocalAudioTrack,
model: LotusDenoiseModel, model: LotusDenoiseModel,
): Promise<void> => { ): Promise<void> => {
const processor = new LotusDenoiseProcessor({ ...config, model }); const processor = new LotusDenoiseProcessor({ ...target.config, model });
pendingProcessors.set(room, processor); pendingProcessors.set(room, processor);
// [lotus #9] Seed the mute state so a processor attached while already // [lotus #9] Seed the mute state so a processor attached while already
// muted starts suspended rather than burning inference on silence. // muted starts suspended rather than burning inference on silence.
@@ -142,11 +232,15 @@ export function startLotusDenoise(vm: CallViewModel): () => void {
} }
}; };
// In-flight apply() per room, so a mid-call switch waits for it to settle.
const inflight = new Map<LivekitRoom, Promise<void>>();
const apply = (room: LivekitRoom): void => { const apply = (room: LivekitRoom): void => {
if (target.mode !== "ml") return;
const mic = micOf(room); const mic = micOf(room);
if (!mic || mic.getProcessor() || pendingProcessors.has(room)) return; if (!mic || mic.getProcessor() || pendingProcessors.has(room)) return;
void (async () => { const run = (async (): Promise<void> => {
let model = config.model; let model = target.config.model;
try { try {
try { try {
await attach(room, mic, model); await attach(room, mic, model);
@@ -164,6 +258,7 @@ export function startLotusDenoise(vm: CallViewModel): () => void {
lotusSendToHost(LotusWidgetActions.DenoiseState, { lotusSendToHost(LotusWidgetActions.DenoiseState, {
active: true, active: true,
model, model,
mode: "ml",
}); });
} catch (e) { } catch (e) {
logger.warn("[lotus] denoise unavailable; publishing raw mic", e); logger.warn("[lotus] denoise unavailable; publishing raw mic", e);
@@ -171,11 +266,69 @@ export function startLotusDenoise(vm: CallViewModel): () => void {
active: false, active: false,
error: e instanceof Error ? e.message : String(e), error: e instanceof Error ? e.message : String(e),
model, model,
mode: "ml",
}); });
} }
})(); })();
inflight.set(room, run);
void run.finally(() => {
if (inflight.get(room) === run) inflight.delete(room);
});
}; };
// ── Mid-call switching (io.lotus.set_denoise) ───────────────────────────
// `applied` is what the mic is actually set up for. Switches run one at a
// time; each works from `applied` to the latest `target`, so a burst of
// changes collapses into the last one.
let applied: DenoiseTarget = target;
let queue: Promise<void> = Promise.resolve();
const reconfigure = async (): Promise<void> => {
const prev = applied;
const next = target;
if (prev === next || stopped) return;
const nsChanged = browserNsFor(prev) !== browserNsFor(next);
const swapProcessor =
prev.mode === "ml" &&
(next.mode !== "ml" || !sameMlConfig(prev.config, next.config));
setLotusNoiseSuppression(browserNsFor(next));
// Load the new model while the old one keeps running, so the switch
// itself only re-wires the graph instead of sending raw audio meanwhile.
if (next.mode === "ml") await prepareDenoiseAssets(next.config);
for (const room of rooms) {
await inflight.get(room)?.catch(() => undefined);
const mic = micOf(room);
if (!mic) continue;
try {
if (swapProcessor && mic.getProcessor()) await mic.stopProcessor();
if (nsChanged) await restartCapture(mic, browserNsFor(next));
} catch (e) {
logger.warn("[lotus] set_denoise: re-capturing the mic failed", e);
}
apply(room);
}
applied = next;
if (next.mode !== "ml")
lotusSendToHost(LotusWidgetActions.DenoiseState, {
active: false,
mode: next.mode,
});
};
const onSetDenoise = (ev: CustomEvent<IWidgetApiRequest>): void => {
w?.api.transport.reply(ev.detail, {});
const next = applySetDenoise(target, ev.detail.data);
if (!next) return;
logger.info(
`[lotus] set_denoise: ${next.mode}${next.mode === "ml" ? ` (${next.config.model})` : ""}`,
);
target = next;
queue = queue.then(reconfigure).catch((e) => {
logger.warn("[lotus] set_denoise failed", e);
});
};
w?.lazyActions.on(LotusWidgetActions.SetDenoise, onSetDenoise);
// [lotus #9] Suspend/resume the processor's own AudioContext with the mic's // [lotus #9] Suspend/resume the processor's own AudioContext with the mic's
// mute state (LiveKit keeps the muted track flowing silent frames, so the // mute state (LiveKit keeps the muted track flowing silent frames, so the
// worklet would otherwise run inference the whole time the user is muted). // worklet would otherwise run inference the whole time the user is muted).
@@ -231,6 +384,8 @@ export function startLotusDenoise(vm: CallViewModel): () => void {
return () => { return () => {
stopped = true; stopped = true;
w?.lazyActions.off(LotusWidgetActions.SetDenoise, onSetDenoise);
setLotusNoiseSuppression(null);
sub.unsubscribe(); sub.unsubscribe();
for (const off of roomListeners.values()) off(); for (const off of roomListeners.values()) off();
roomListeners.clear(); roomListeners.clear();
+35
View File
@@ -11,6 +11,7 @@ import { type AudioProcessorOptions } from "livekit-client";
import { import {
assertModuleExport, assertModuleExport,
type LotusDenoiseConfig, type LotusDenoiseConfig,
normalizeDfnModelBytes,
LotusDenoiseProcessor, LotusDenoiseProcessor,
prepareDenoiseAssets, prepareDenoiseAssets,
releasePreparedDenoiseAssets, releasePreparedDenoiseAssets,
@@ -245,3 +246,37 @@ describe("assertModuleExport (#26)", () => {
); );
}); });
}); });
describe("normalizeDfnModelBytes", () => {
const tar = (): Uint8Array => {
const b = new Uint8Array(1024);
b.set(new TextEncoder().encode("ustar"), 257);
b.set(new TextEncoder().encode("model.onnx"), 0);
return b;
};
const gunzip = async (buf: ArrayBuffer): Promise<Uint8Array> =>
new Uint8Array(
await new Response(
new Response(buf).body!.pipeThrough(new DecompressionStream("gzip")),
).arrayBuffer(),
);
test("passes a gzipped model through untouched", async () => {
const gz = new Uint8Array([0x1f, 0x8b, 8, 0, 1, 2, 3]).buffer;
expect(await normalizeDfnModelBytes(gz)).toBe(gz);
});
test("re-gzips a tar that a Content-Encoding server already unpacked", async () => {
const out = await normalizeDfnModelBytes(tar().buffer as ArrayBuffer);
const u8 = new Uint8Array(out);
expect([u8[0], u8[1]]).toEqual([0x1f, 0x8b]);
expect(await gunzip(out)).toEqual(tar());
});
test("rejects anything else (e.g. an HTML error page)", async () => {
const html = new TextEncoder().encode("<!doctype html><title>404</title>");
await expect(
normalizeDfnModelBytes(html.buffer as ArrayBuffer),
).rejects.toThrow(/neither gzip nor tar/);
});
});
+39 -4
View File
@@ -144,6 +144,8 @@ interface DfnCore {
initialize: () => Promise<void>; initialize: () => Promise<void>;
createAudioWorkletNode: (ctx: AudioContext) => Promise<AudioNode>; createAudioWorkletNode: (ctx: AudioContext) => Promise<AudioNode>;
destroy: () => void; destroy: () => void;
/** Set by `initialize()`; the model bytes are handed to the worklet as-is. */
assets?: { wasmModule: WebAssembly.Module; modelBytes: ArrayBuffer } | null;
} }
interface DfnModule { interface DfnModule {
DeepFilterNet3Core: new (opts: { DeepFilterNet3Core: new (opts: {
@@ -167,6 +169,33 @@ export function assertModuleExport<T>(
return mod as T; return mod as T;
} }
/**
* Make sure the DeepFilterNet model is the gzipped tarball its wasm expects.
*
* A server or proxy that sends the `.tar.gz` with `Content-Encoding: gzip`
* (vite preview does; some CDNs and nginx `gzip_static` setups do too) makes
* `fetch()` hand back the inner tar. The library's worklet then fails inside
* `df_create`, swallows the error and passes the raw mic straight through,
* while we report DeepFilterNet as active. An already-decompressed tar is
* re-gzipped; anything else throws, so the rnnoise fallback runs and the host
* is told the truth.
*/
export async function normalizeDfnModelBytes(
bytes: ArrayBuffer,
): Promise<ArrayBuffer> {
const u8 = new Uint8Array(bytes);
if (u8.length > 2 && u8[0] === 0x1f && u8[1] === 0x8b) return bytes;
const isTar =
u8.length > 262 &&
String.fromCharCode(...u8.subarray(257, 262)) === "ustar";
if (!isTar)
throw new Error(
`denoise: DeepFilterNet model is neither gzip nor tar (${u8.length} bytes)`,
);
const gz = new Response(u8).body!.pipeThrough(new CompressionStream("gzip"));
return new Response(gz).arrayBuffer();
}
async function loadDfnCore(config: LotusDenoiseConfig): Promise<DfnCore> { async function loadDfnCore(config: LotusDenoiseConfig): Promise<DfnCore> {
const base = config.assetBase; const base = config.assetBase;
const url = `${base}deepfilternet/index.esm.js`; const url = `${base}deepfilternet/index.esm.js`;
@@ -178,13 +207,19 @@ async function loadDfnCore(config: LotusDenoiseConfig): Promise<DfnCore> {
); );
const core = new mod.DeepFilterNet3Core({ const core = new mod.DeepFilterNet3Core({
sampleRate: 48_000, sampleRate: 48_000,
// 60, not 80: full-strength suppression is the main source of the // Attenuation limit in dB. At 60 the background is muted outright
// "over-processed" character; a lower level keeps voice natural while // (measured -61 to -64 dB), so every word drops into dead silence and
// the dry/wet floor handles the noise tail. // listeners hear the mic "cutting out" at word ends. 25 keeps a quiet
noiseReductionLevel: 60, // background bed and scored best on room-echo speech with fan and
// keyboard noise (PESQ 2.72 / 2.38 vs 2.33 / 2.04 at 60).
noiseReductionLevel: 25,
assetConfig: { cdnUrl: dfnBase }, assetConfig: { cdnUrl: dfnBase },
}); });
await core.initialize(); await core.initialize();
if (core.assets)
core.assets.modelBytes = await normalizeDfnModelBytes(
core.assets.modelBytes,
);
return core; return core;
} }
+188
View File
@@ -0,0 +1,188 @@
/*
Copyright 2026 Lotus Guild
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { BehaviorSubject } from "rxjs";
import { afterEach, describe, expect, it, vi } from "vitest";
const sent: { action: string; data: unknown }[] = [];
const handlers = new Map<string, (ev: CustomEvent) => void>();
vi.mock("../widget", () => ({
widget: {
api: { transport: { reply: vi.fn() } },
lazyActions: {
on: (a: string, h: (ev: CustomEvent) => void): void =>
void handlers.set(a, h),
off: (a: string): void => void handlers.delete(a),
},
},
}));
vi.mock("./lotusWidget", () => ({
lotusFlag: (name: string): boolean => name === "lotusDeviceCheck",
lotusSendToHost: (action: string, data: unknown): boolean => {
sent.push({ action, data });
return true;
},
}));
const {
checkDevice,
lotusDeviceGate,
startLotusDevices,
audioInputName,
audioOutputName,
} = await import("./lotusDevices");
const { audioInput, audioOutput } = await import("../settings/settings");
const named = (
...names: [string, string][]
): Map<string, { type: string; name?: string }> =>
new Map(names.map(([id, name]) => [id, { type: "name", name }]));
type Labels = Map<string, { type: string; name?: string }>;
const source = (
labels: Labels,
): { available$: BehaviorSubject<Labels>; select: (id: string) => void } => ({
available$: new BehaviorSubject(labels),
select: vi.fn<(id: string) => void>(),
});
const devices = (
inputs: Labels,
outputs: Labels = named(["", "Default"]),
): {
audioInput: ReturnType<typeof source>;
audioOutput: ReturnType<typeof source>;
} => ({ audioInput: source(inputs), audioOutput: source(outputs) });
afterEach(() => {
sent.length = 0;
handlers.clear();
audioInput.setValue(undefined);
audioOutput.setValue(undefined);
audioInputName.setValue(undefined);
audioOutputName.setValue(undefined);
});
describe("[cinny #121] checkDevice", () => {
const avail = named(["laptop", "Laptop Mic"], ["arctis2", "Arctis 7"]);
it("is fine when the usual device is there, or nothing is remembered", () => {
expect(checkDevice("audioinput", "arctis2", "Arctis 7", avail).result).toBe(
"ok",
);
expect(checkDevice("audioinput", undefined, undefined, avail).result).toBe(
"ok",
);
expect(checkDevice("audioinput", "default", "x", avail).result).toBe("ok");
});
it("re-selects a device that came back under a new id", () => {
expect(checkDevice("audioinput", "arctis1", "Arctis 7", avail)).toEqual({
kind: "audioinput",
result: "reselect",
id: "arctis2",
});
});
it("reports a missing device and what EC would use instead", () => {
expect(
checkDevice(
"audioinput",
"arctis1",
"Arctis 7",
named(["laptop", "Laptop Mic"]),
),
).toEqual({
kind: "audioinput",
result: "missing",
expected: "Arctis 7",
fallback: "Laptop Mic",
});
});
it("fails open without device names (no permission yet)", () => {
const unnamed = new Map([["x", { type: "number" }]]);
expect(
checkDevice("audioinput", "arctis1", "Arctis 7", unnamed).result,
).toBe("ok");
});
});
describe("[cinny #121] lotusDeviceGate", () => {
it("joins straight away when everything is there", async () => {
audioInput.setValue("arctis");
audioInputName.setValue("Arctis 7");
await expect(
lotusDeviceGate(devices(named(["arctis", "Arctis 7"]))),
).resolves.toBe(true);
expect(sent).toEqual([]);
});
it("asks the host and waits for the answer", async () => {
audioInput.setValue("arctis");
audioInputName.setValue("Arctis 7");
const gate = lotusDeviceGate(devices(named(["laptop", "Laptop Mic"])));
await vi.waitFor(() => expect(sent).toHaveLength(1));
expect(sent[0]).toEqual({
action: "io.lotus.device_missing",
data: {
missing: [
{ kind: "audioinput", expected: "Arctis 7", fallback: "Laptop Mic" },
],
},
});
handlers.get("io.lotus.device_decision")!(
new CustomEvent("x", { detail: { data: { join: false } } }),
);
await expect(gate).resolves.toBe(false);
});
it("re-selects a re-plugged device without asking", async () => {
audioInput.setValue("old-id");
audioInputName.setValue("Arctis 7");
const d = devices(named(["new-id", "Arctis 7"]));
await expect(lotusDeviceGate(d)).resolves.toBe(true);
expect(d.audioInput.select).toHaveBeenCalledWith("new-id");
expect(sent).toEqual([]);
});
it("joins when device names never arrive", async () => {
audioInput.setValue("arctis");
audioInputName.setValue("Arctis 7");
const d = devices(
new Map([["x", { type: "number" }]]),
new Map([["y", { type: "number" }]]),
);
await expect(lotusDeviceGate(d, 50)).resolves.toBe(true);
});
});
describe("[cinny #121] startLotusDevices", () => {
it("remembers the picked device's name and reports when it goes", () => {
const d = devices(named(["laptop", "Laptop Mic"], ["arctis", "Arctis 7"]));
const stop = startLotusDevices(d);
audioInput.setValue("arctis");
expect(audioInputName.getValue()).toBe("Arctis 7");
d.audioInput.available$.next(named(["laptop", "Laptop Mic"]));
expect(sent).toEqual([
{
action: "io.lotus.device_lost",
data: {
kind: "audioinput",
expected: "Arctis 7",
fallback: "Laptop Mic",
},
},
]);
// Name kept, so the next join can still recognise it.
expect(audioInputName.getValue()).toBe("Arctis 7");
stop();
});
it("choosing the system default clears the remembered name", () => {
audioInputName.setValue("Arctis 7");
const stop = startLotusDevices(devices(named(["", "Default"])));
audioInput.setValue("");
expect(audioInputName.getValue()).toBeUndefined();
stop();
});
});
+236
View File
@@ -0,0 +1,236 @@
/*
Copyright 2026 Lotus Guild
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { logger } from "matrix-js-sdk/lib/logger";
import { type IWidgetApiRequest } from "matrix-widget-api";
import {
combineLatest,
filter,
firstValueFrom,
map,
type Observable,
timeout,
} from "rxjs";
import {
audioInput as audioInputSetting,
audioOutput as audioOutputSetting,
Setting,
} from "../settings/settings";
import { widget } from "../widget";
import { LotusWidgetActions } from "./lotusActions";
import { lotusFlag, lotusSendToHost } from "./lotusWidget";
/**
* [cinny #121] Expected-device check. EC remembers the chosen mic / output by
* id and, when that device is missing, silently uses the first one it finds,
* so a user who always joins with a headset ends up on the laptop mic.
*
* - "Usual" = the device the user last picked (EC's own audio-input /
* audio-output settings). Its NAME is remembered alongside (ids can change
* when a device is re-plugged on Windows).
* - Before the automatic join (`lotusDeviceCheck=1`), a missing usual device
* is reported to the host (`io.lotus.device_missing`), which asks the user
* and answers `io.lotus.device_decision { join }`. A device that came back
* under a new id but the same name is just re-selected.
* - In a call, losing the usual device is reported (`io.lotus.device_lost`)
* for a toast; no question mid-call.
* Fails open: no names (no permission yet), no host, or nothing remembered
* means no check.
*/
export type DeviceKind = "audioinput" | "audiooutput";
/** Any device label with an optional human name (EC's label unions). */
type AnyLabel = { type: string; name?: string | null };
export type DeviceCheck =
| { kind: DeviceKind; result: "ok" }
| { kind: DeviceKind; result: "reselect"; id: string }
| {
kind: DeviceKind;
result: "missing";
expected: string;
fallback: string | null;
};
const nameOf = (label: AnyLabel | undefined): string | null =>
label && typeof label.name === "string" && label.name ? label.name : null;
/** Whether the device list carries real names (permission granted). */
export const hasNames = (available: Map<string, AnyLabel>): boolean =>
[...available.values()].some((l) => l.type === "name");
/** The id EC is going to (or does) use as the "system default" choice. */
const isDefaultId = (id: string): boolean => id === "" || id === "default";
/**
* Pure: compare the remembered device with what is plugged in now.
* `fallback` names the device EC falls back to (the first in the list, as
* MediaDevices' selectDevice$ does).
*/
export function checkDevice(
kind: DeviceKind,
preferredId: string | undefined,
expectedName: string | undefined,
available: Map<string, AnyLabel>,
): DeviceCheck {
if (available.size === 0 || !hasNames(available))
return { kind, result: "ok" };
if (preferredId === undefined || isDefaultId(preferredId))
return { kind, result: "ok" };
if (available.has(preferredId)) return { kind, result: "ok" };
if (!expectedName) return { kind, result: "ok" };
for (const [id, label] of available) {
if (nameOf(label) === expectedName) return { kind, result: "reselect", id };
}
const first = available.values().next().value;
return {
kind,
result: "missing",
expected: expectedName,
fallback: nameOf(first),
};
}
/** The remembered NAME of the last-picked device of each kind. */
export const audioInputName = new Setting<string | undefined>(
"lotus-audio-input-name",
undefined,
);
export const audioOutputName = new Setting<string | undefined>(
"lotus-audio-output-name",
undefined,
);
interface DeviceSource {
available$: Observable<Map<string, AnyLabel>>;
select(id: string): void;
}
export interface LotusMediaDevices {
audioInput: DeviceSource;
audioOutput: DeviceSource;
}
const kinds = (
devices: LotusMediaDevices,
): {
kind: DeviceKind;
source: DeviceSource;
preferred: Setting<string | undefined>;
name: Setting<string | undefined>;
}[] => [
{
kind: "audioinput",
source: devices.audioInput,
preferred: audioInputSetting,
name: audioInputName,
},
{
kind: "audiooutput",
source: devices.audioOutput,
preferred: audioOutputSetting,
name: audioOutputName,
},
];
/** Whether the host asked for the check (a Lotus host with the flag). */
export const lotusDeviceCheckWanted = (): boolean =>
!!widget && lotusFlag("lotusDeviceCheck");
/**
* Before EC's automatic join: true = join now, false = the user cancelled.
* Waits up to `namesTimeoutMs` for named device lists, then checks both kinds.
*/
export async function lotusDeviceGate(
devices: LotusMediaDevices,
namesTimeoutMs = 3000,
): Promise<boolean> {
const w = widget;
if (!w || !lotusDeviceCheckWanted()) return true;
const entries = kinds(devices);
let lists: Map<string, AnyLabel>[];
try {
lists = await firstValueFrom(
combineLatest(entries.map((e) => e.source.available$)).pipe(
filter((all) => all.some(hasNames)),
timeout(namesTimeoutMs),
),
);
} catch {
return true; // no names (yet): nothing to compare by
}
const missing: Extract<DeviceCheck, { result: "missing" }>[] = [];
entries.forEach((e, i) => {
const check = checkDevice(
e.kind,
e.preferred.getValue(),
e.name.getValue(),
lists[i],
);
if (check.result === "reselect") {
logger.info(`[lotus] ${e.kind}: usual device is back under a new id`);
e.source.select(check.id);
} else if (check.result === "missing") missing.push(check);
});
if (missing.length === 0) return true;
return new Promise<boolean>((resolve) => {
const onDecision = (ev: CustomEvent<IWidgetApiRequest>): void => {
w.api.transport.reply(ev.detail, {});
w.lazyActions.off(LotusWidgetActions.DeviceDecision, onDecision);
resolve((ev.detail.data as { join?: unknown }).join !== false);
};
w.lazyActions.on(LotusWidgetActions.DeviceDecision, onDecision);
const sent = lotusSendToHost(LotusWidgetActions.DeviceMissing, {
missing: missing.map(({ kind, expected, fallback }) => ({
kind,
expected,
fallback,
})),
});
if (!sent) {
w.lazyActions.off(LotusWidgetActions.DeviceDecision, onDecision);
resolve(true);
}
});
}
/**
* In a call: remember the name of every device the user picks, and report
* when the picked device disappears. Returns a teardown function.
*/
export function startLotusDevices(devices: LotusMediaDevices): () => void {
const subs = kinds(devices).map((e) => {
let wasPresent: boolean | undefined;
return combineLatest([e.preferred.value$, e.source.available$])
.pipe(map(([id, available]) => ({ id, available })))
.subscribe(({ id, available }) => {
if (id === undefined || isDefaultId(id) || !hasNames(available)) {
if (id !== undefined && isDefaultId(id)) e.name.setValue(undefined);
wasPresent = undefined;
return;
}
const label = available.get(id);
if (label) {
const name = nameOf(label);
if (name && e.name.getValue() !== name) e.name.setValue(name);
wasPresent = true;
return;
}
if (wasPresent && e.name.getValue()) {
lotusSendToHost(LotusWidgetActions.DeviceLost, {
kind: e.kind,
expected: e.name.getValue(),
fallback: nameOf(available.values().next().value),
});
}
wasPresent = false;
});
});
return (): void => subs.forEach((s) => s.unsubscribe());
}
+48 -7
View File
@@ -6,10 +6,12 @@ Please see LICENSE in the repository root for full details.
*/ */
import { import {
type LocalTrack,
type LocalTrackPublication, type LocalTrackPublication,
type Room as LivekitRoom, type Room as LivekitRoom,
RoomEvent, RoomEvent,
Track, Track,
TrackEvent,
} from "livekit-client"; } from "livekit-client";
import { Observable, type Subscription, share, switchMap } from "rxjs"; import { Observable, type Subscription, share, switchMap } from "rxjs";
import { distinctUntilChanged, map } from "rxjs/operators"; import { distinctUntilChanged, map } from "rxjs/operators";
@@ -37,15 +39,35 @@ export interface LocalMicSample {
const micPublication = ( const micPublication = (
room: LivekitRoom, room: LivekitRoom,
): { track: MediaStreamTrack; muted: boolean } | null => { ): { track: MediaStreamTrack; local: LocalTrack; muted: boolean } | null => {
const pub: LocalTrackPublication | undefined = const pub: LocalTrackPublication | undefined =
room.localParticipant.getTrackPublication(Track.Source.Microphone); room.localParticipant.getTrackPublication(Track.Source.Microphone);
const track = pub?.track?.mediaStreamTrack; const local = pub?.track;
return track && track.readyState === "live" const track = local?.mediaStreamTrack;
? { track, muted: pub?.isMuted ?? false } return local && track && track.readyState === "live"
? { track, local, muted: pub?.isMuted ?? false }
: null; : null;
}; };
// Every live tap, so a mic re-capture can drop the clones first.
const liveTaps = new Set<{ release: () => void; reconcile: () => void }>();
/**
* Run a mic re-capture (`restartTrack`) with every level-meter clone stopped,
* then re-tap the new track. Chrome makes all open tracks of a device share
* one set of audio processing, so a clone still holding the old track makes
* the new capture ignore its `noiseSuppression` (or AGC/EC) constraint.
*/
export async function withMicTapsReleased<T>(fn: () => Promise<T>): Promise<T> {
const taps = [...liveTaps];
taps.forEach((t) => t.release());
try {
return await fn();
} finally {
taps.forEach((t) => t.reconcile());
}
}
/** Pure: the same LiveKit rooms, in the same order (by identity). */ /** Pure: the same LiveKit rooms, in the same order (by identity). */
export const sameRooms = <T>(a: readonly T[], b: readonly T[]): boolean => export const sameRooms = <T>(a: readonly T[], b: readonly T[]): boolean =>
a.length === b.length && a.every((r, i) => r === b[i]); a.length === b.length && a.every((r, i) => r === b[i]);
@@ -79,11 +101,17 @@ export function observeLocalMicSample$(
let clone: MediaStreamTrack | null = null; let clone: MediaStreamTrack | null = null;
let timer: ReturnType<typeof setInterval> | undefined; let timer: ReturnType<typeof setInterval> | undefined;
let tapped: MediaStreamTrack | null = null; let tapped: MediaStreamTrack | null = null;
// LiveKit swaps the MediaStreamTrack inside the same LocalTrack on
// restartTrack (device switch, re-capture) without any room event,
// so follow its Restarted event or the clone keeps the old one open.
let tappedLocal: LocalTrack | null = null;
let muted = false; let muted = false;
const stopTap = (): void => { const stopTap = (): void => {
if (timer !== undefined) clearInterval(timer); if (timer !== undefined) clearInterval(timer);
timer = undefined; timer = undefined;
tappedLocal?.off(TrackEvent.Restarted, reconcile);
tappedLocal = null;
clone?.stop(); clone?.stop();
clone = null; clone = null;
void ctx?.close().catch(() => undefined); void ctx?.close().catch(() => undefined);
@@ -92,7 +120,12 @@ export function observeLocalMicSample$(
subscriber.next(null); subscriber.next(null);
}; };
const startTap = (source: MediaStreamTrack): void => { const startTap = (
source: MediaStreamTrack,
local: LocalTrack,
): void => {
tappedLocal = local;
local.on(TrackEvent.Restarted, reconcile);
try { try {
clone = source.clone(); clone = source.clone();
// Muting disables the published track; the clone must still hear. // Muting disables the published track; the clone must still hear.
@@ -116,15 +149,22 @@ export function observeLocalMicSample$(
} }
}; };
const reconcile = (): void => { function reconcile(): void {
const pub = const pub =
rooms.map(micPublication).find((p) => p !== null) ?? null; rooms.map(micPublication).find((p) => p !== null) ?? null;
muted = pub?.muted ?? false; muted = pub?.muted ?? false;
const track = pub?.track ?? null; const track = pub?.track ?? null;
if (track === tapped) return; if (track === tapped) return;
if (tapped) stopTap(); if (tapped) stopTap();
if (track) startTap(track); if (track && pub) startTap(track, pub.local);
}
const control = {
release: (): void => {
if (tapped) stopTap();
},
reconcile,
}; };
liveTaps.add(control);
const events = [ const events = [
RoomEvent.TrackMuted, RoomEvent.TrackMuted,
@@ -143,6 +183,7 @@ export function observeLocalMicSample$(
rooms.forEach((room) => rooms.forEach((room) =>
events.forEach((ev) => room.off(ev, reconcile)), events.forEach((ev) => room.off(ev, reconcile)),
); );
liveTaps.delete(control);
if (tapped) stopTap(); if (tapped) stopTap();
}; };
}), }),
+79
View File
@@ -18,6 +18,7 @@ import {
overrideSpotlight$, overrideSpotlight$,
pinnedMedia$, pinnedMedia$,
resolveManualSpotlight, resolveManualSpotlight,
spotlightCameras,
} from "./lotusSpotlight"; } from "./lotusSpotlight";
import { parseFocusPayload } from "./lotusFocus"; import { parseFocusPayload } from "./lotusFocus";
@@ -27,6 +28,7 @@ interface FakeMedia {
type: "user"; type: "user";
local: boolean; local: boolean;
speaking$: BehaviorSubject<boolean>; speaking$: BehaviorSubject<boolean>;
videoEnabled$: BehaviorSubject<boolean>;
} }
function media( function media(
@@ -34,6 +36,7 @@ function media(
deviceId: string, deviceId: string,
speaking = false, speaking = false,
local = false, local = false,
video = false,
): FakeMedia { ): FakeMedia {
return { return {
id: `${userId}:${deviceId}`, id: `${userId}:${deviceId}`,
@@ -41,6 +44,7 @@ function media(
type: "user", type: "user",
local, local,
speaking$: new BehaviorSubject(speaking), speaking$: new BehaviorSubject(speaking),
videoEnabled$: new BehaviorSubject(video),
}; };
} }
const asUser = (m: FakeMedia): UserMediaViewModel => const asUser = (m: FakeMedia): UserMediaViewModel =>
@@ -186,6 +190,81 @@ describe("overrideSpotlight$", () => {
}); });
}); });
describe("spotlightCameras (#67)", () => {
const me = media("@me:x", "d", false, true, true);
const carol = media("@carol:x", "d", false, false, true);
const dave = media("@dave:x", "d", false, false, true);
const users = [me, carol, bob, dave];
test("shares first, then remote cameras that are on, in order", () => {
expect(
spotlightCameras([screen], users, new Set([me, carol, dave]), undefined),
).toEqual([screen, carol, dave]);
});
test("the pin sits right after the shares and appears once", () => {
expect(
spotlightCameras([screen], users, new Set([carol, dave]), dave),
).toEqual([screen, dave, carol]);
});
test("the pinned camera is kept even with its video off", () => {
expect(spotlightCameras([screen], users, new Set([carol]), bob)).toEqual([
screen,
bob,
carol,
]);
});
});
describe("overrideSpotlight$ cameras during a share (#67)", () => {
test("cameras join and leave the carousel live", () => {
const carol = media("@carol:x", "d", false, false, true);
const me = media("@me:x", "d", false, true, true);
// Own fixture: other tests leave subscriptions on the shared `bob`.
const erin = media("@erin:x", "d");
const users$ = new BehaviorSubject([
asUser(me),
asUser(carol),
asUser(erin),
]);
const seen: unknown[][] = [];
const sub = overrideSpotlight$(
of(asUser(carol)),
of(null),
of([screen as unknown as ScreenShareViewModel]),
of(undefined),
users$,
).subscribe((r) => seen.push(r.spotlight));
erin.videoEnabled$.next(true); // erin turns their camera on
carol.videoEnabled$.next(false); // carol turns hers off
users$.next([asUser(me), asUser(carol)]); // erin leaves
erin.videoEnabled$.next(false);
sub.unsubscribe();
expect(seen).toEqual([
[screen, carol],
[screen, carol, erin],
[screen, erin],
[screen],
]);
});
test("no share: cameras are not added (upstream speaker-follows)", () => {
const carol = media("@carol:x", "d", false, false, true);
let spotlight;
overrideSpotlight$(
of(asUser(bob)),
of(null),
of([]),
of(undefined),
of([asUser(carol), asUser(bob)]),
).subscribe((r) => (spotlight = r.spotlight));
expect(spotlight).toEqual([bob]);
});
});
describe("createManualSpotlight (#3 / #16)", () => { describe("createManualSpotlight (#3 / #16)", () => {
function setup(initialLayout: LayoutMode = "grid") { function setup(initialLayout: LayoutMode = "grid") {
const pin$ = new BehaviorSubject<ManualSpotlight | null>(null); const pin$ = new BehaviorSubject<ManualSpotlight | null>(null);
+69 -9
View File
@@ -26,6 +26,7 @@ import { type MediaViewModel } from "../state/media/MediaViewModel";
import { type LocalUserMediaViewModel } from "../state/media/LocalUserMediaViewModel"; import { type LocalUserMediaViewModel } from "../state/media/LocalUserMediaViewModel";
import { type LayoutMode } from "../state/LayoutSwitchViewModel"; import { type LayoutMode } from "../state/LayoutSwitchViewModel";
import { type Behavior } from "../state/Behavior"; import { type Behavior } from "../state/Behavior";
import { shallowEquals } from "../utils/array";
interface SpotlightAndPip { interface SpotlightAndPip {
spotlight: MediaViewModel[]; spotlight: MediaViewModel[];
@@ -102,6 +103,44 @@ export function pinnedMedia$(
); );
} }
/**
* [lotus #67] The spotlight carousel during a screenshare: every share first
* (upstream order), then the host-pinned camera if any (included even with
* its video off, as before #67), then every other REMOTE camera whose video
* is on, in `userMedia` order so the dots don't reshuffle. Each camera
* appears once. The local camera is left out: the PiP and grid show it.
*/
export function spotlightCameras<S, U extends { local: boolean }>(
screenShares: S[],
userMedia: U[],
videoOn: ReadonlySet<U>,
pinned: U | undefined,
): (S | U)[] {
const cameras = userMedia.filter(
(m) => m !== pinned && !m.local && videoOn.has(m),
);
return [...screenShares, ...(pinned ? [pinned] : []), ...cameras];
}
/**
* The remote user media whose camera is currently on, as a live set. Only
* subscribes to `videoEnabled$` of remote items (the local camera never goes
* in the carousel).
*/
function remoteVideoOn$(
userMedia$: Observable<UserMediaViewModel[]>,
): Observable<Set<UserMediaViewModel>> {
return userMedia$.pipe(
switchMap((items) => {
const remote = items.filter((m) => !m.local);
if (remote.length === 0) return of(new Set<UserMediaViewModel>());
return combineLatest(remote.map((m) => m.videoEnabled$)).pipe(
map((flags) => new Set(remote.filter((_, i) => flags[i]))),
);
}),
);
}
/** /**
* [lotus #4] Manual spotlight override. * [lotus #4] Manual spotlight override.
* *
@@ -121,6 +160,12 @@ export function pinnedMedia$(
* screenshare"); the PiP keeps showing the auto-selected speaker and is * screenshare"); the PiP keeps showing the auto-selected speaker and is
* only blanked when that speaker IS the pinned participant (#29). * only blanked when that speaker IS the pinned participant (#29).
* *
* [lotus #67] During a screenshare the spotlight also holds every remote
* camera that is on (after the shares and the pin, see
* {@link spotlightCameras}), so viewers can flip to a camera with the same
* arrows/dots they use between shares. The share stays the default view;
* SpotlightTile handles that via `useLotusSpotlightFallback`.
*
* @param autoSpotlightSpeaker$ upstream's speaker-follows auto selection * @param autoSpotlightSpeaker$ upstream's speaker-follows auto selection
* @param manualSpotlight$ host pin, or `null` for auto (default) * @param manualSpotlight$ host pin, or `null` for auto (default)
* @param screenShares$ current screen-share view models * @param screenShares$ current screen-share view models
@@ -146,23 +191,38 @@ export function overrideSpotlight$(
return screenShares$.pipe( return screenShares$.pipe(
switchMap((screenShares) => { switchMap((screenShares) => {
if (screenShares.length > 0) if (screenShares.length > 0)
// During a screenshare, if the host has explicitly pinned a // During a screenshare the spotlight holds the shares, then the
// participant, surface that camera in the spotlight alongside the // host-pinned camera (#4, "focus camera during screenshare"), then the
// shared screen (the whole point of "focus camera during screenshare"). // other remote cameras that are on (#67).
// With no manual pin this is unchanged: the screenshare alone is return combineLatest([
// spotlighted. pinned$,
return pinned$.pipe( userMedia$,
map((pinned) => remoteVideoOn$(userMedia$),
]).pipe(
map(([pinned, userMedia, videoOn]) => ({
pinned,
spotlight: spotlightCameras(
screenShares,
userMedia,
videoOn,
pinned,
),
})),
distinctUntilChanged(
(a, b) =>
a.pinned === b.pinned && shallowEquals(a.spotlight, b.spotlight),
),
map(({ pinned, spotlight }) =>
pinned pinned
? { ? {
spotlight: [...screenShares, pinned], spotlight,
// [lotus #29] Keep the active-speaker PiP; it is only // [lotus #29] Keep the active-speaker PiP; it is only
// redundant when the speaker is the pinned participant. // redundant when the speaker is the pinned participant.
pip$: autoSpotlightSpeaker$.pipe( pip$: autoSpotlightSpeaker$.pipe(
map((auto) => (auto === pinned ? undefined : auto)), map((auto) => (auto === pinned ? undefined : auto)),
), ),
} }
: { spotlight: screenShares, pip$: spotlightSpeaker$ }, : { spotlight, pip$: spotlightSpeaker$ },
), ),
); );
+42
View File
@@ -0,0 +1,42 @@
/*
Copyright 2026 Lotus Guild
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { describe, expect, test } from "vitest";
import { spotlightFallbackTarget } from "./lotusSpotlightView";
const share = { id: "s", type: "screen share" };
const share2 = { id: "s2", type: "screen share" };
const cam = { id: "c", type: "user" };
const cam2 = { id: "c2", type: "user" };
describe("spotlightFallbackTarget (#67)", () => {
test("nothing to do while the visible item is still there", () => {
expect(spotlightFallbackTarget([share, cam], "c", true)).toBeNull();
expect(spotlightFallbackTarget([share, cam], "s", true)).toBeNull();
});
test("visible camera gone: fall back to the first share", () => {
expect(spotlightFallbackTarget([share, cam2], "c", true)).toBe("s");
});
test("visible share ended: fall back to the remaining first item", () => {
expect(spotlightFallbackTarget([share2, cam], "s", true)).toBe("s2");
});
test("share started while a camera was spotlighted: jump to the share", () => {
expect(spotlightFallbackTarget([share, cam], "c", false)).toBe("s");
});
test("no share and no change: speaker-follows is left alone", () => {
expect(spotlightFallbackTarget([cam2], "c2", false)).toBeNull();
});
test("empty media: nothing to do", () => {
expect(spotlightFallbackTarget([], "c", true)).toBeNull();
});
});
+72
View File
@@ -0,0 +1,72 @@
/*
Copyright 2026 Lotus Guild
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { useEffect, useRef } from "react";
/**
* What the spotlight carousel should jump to after `media` changes, or `null`
* to leave it alone. Pure; see {@link useLotusSpotlightFallback}.
*
* - The visible item is gone (its camera turned off, its owner left, or a
* share ended): show the first item rather than whichever neighbour the
* scroll happens to land on.
* - A share has just started (the first item became a screen share): show
* it, even if the previously spotlighted camera is still in the list.
*/
export function spotlightFallbackTarget(
media: readonly { id: string; type: string }[],
visibleId: string | undefined,
firstWasShare: boolean,
): string | null {
const first = media[0];
if (first === undefined) return null;
const shareStarted = first.type === "screen share" && !firstWasShare;
const visibleGone =
visibleId !== undefined && !media.some((m) => m.id === visibleId);
if (!shareStarted && !visibleGone) return null;
return first.id === visibleId ? null : first.id;
}
/**
* [lotus #67] Keeps the spotlight carousel on a sensible item now that it can
* hold cameras as well as screenshares during a share (see
* `spotlightCameras` in lotusSpotlight.ts).
*
* Scrolls the item into view directly and also sets `visibleId`: when the
* target is already at the scroll position (e.g. the visible item was the
* last one), the tile's IntersectionObserver never fires, so the arrows and
* dots would otherwise stay pointed at an item that no longer exists.
*/
export function useLotusSpotlightFallback(
root: HTMLElement | null,
media: readonly { id: string; type: string }[],
visibleId: string | undefined,
setVisibleId: (id: string) => void,
): void {
const firstIsShare = media[0]?.type === "screen share";
const firstWasShare = useRef(firstIsShare);
useEffect(() => {
const target = spotlightFallbackTarget(
media,
visibleId,
firstWasShare.current,
);
firstWasShare.current = firstIsShare;
if (target === null) return;
const item = root?.querySelector<HTMLElement>(
`[data-id="${CSS.escape(target)}"]`,
);
const scroller = item?.parentElement;
if (item && scroller)
scroller.scrollLeft +=
item.getBoundingClientRect().left -
scroller.getBoundingClientRect().left;
setVisibleId(target);
}, [root, media, visibleId, firstIsShare, setVisibleId]);
}
+19 -1
View File
@@ -10,6 +10,7 @@ import {
type ReactNode, type ReactNode,
useCallback, useCallback,
useEffect, useEffect,
useRef,
useMemo, useMemo,
useState, useState,
} from "react"; } from "react";
@@ -39,6 +40,7 @@ import { findDeviceByName } from "../utils/media";
import { ActiveCall } from "./InCallView"; import { ActiveCall } from "./InCallView";
import { type MuteStates } from "../state/MuteStates"; import { type MuteStates } from "../state/MuteStates";
import { useMediaDevices } from "../MediaDevicesContext"; import { useMediaDevices } from "../MediaDevicesContext";
import { lotusDeviceCheckWanted, lotusDeviceGate } from "../lotus/lotusDevices";
import { useMatrixRTCSessionMemberships } from "../useMatrixRTCSessionMemberships"; import { useMatrixRTCSessionMemberships } from "../useMatrixRTCSessionMemberships";
import { import {
saveKeyForRoom, saveKeyForRoom,
@@ -277,6 +279,7 @@ const LoadedCallView: FC<LoadedProps> = ({
); );
const mediaDevices = useMediaDevices(); const mediaDevices = useMediaDevices();
const lotusGate = useRef<Promise<boolean> | null>(null);
const latestMuteStates = useLatest(muteStates); const latestMuteStates = useLatest(muteStates);
const enterRTCSessionOrError = useCallback( const enterRTCSessionOrError = useCallback(
@@ -357,7 +360,22 @@ const LoadedCallView: FC<LoadedProps> = ({
return (): void => subscription.unsubscribe(); return (): void => subscription.unsubscribe();
} else { } else {
// No lobby and no preload: we enter the rtc session right away // No lobby and no preload: we enter the rtc session right away
setJoined(true); // ([lotus #121] after checking the usual mic / output are present when
// the Lotus host asked for it; it may ask the user and cancel).
if (!lotusDeviceCheckWanted()) {
setJoined(true);
return;
}
// Started once per call: this effect re-runs as its inputs settle,
// and restarting the check each time could hold the join forever.
lotusGate.current ??= lotusDeviceGate(mediaDevices);
let cancelled = false;
void lotusGate.current.then((join) => {
if (!cancelled && join) setJoined(true);
});
return (): void => {
cancelled = true;
};
} }
} }
}, [ }, [
+3
View File
@@ -40,6 +40,7 @@ import { startLotusCallSummary } from "../lotus/lotusCallSummary";
import { startLotusAudioOutput } from "../lotus/lotusAudioOutput"; import { startLotusAudioOutput } from "../lotus/lotusAudioOutput";
import { startLotusScreenshareWatch } from "../lotus/lotusScreenshareWatch"; import { startLotusScreenshareWatch } from "../lotus/lotusScreenshareWatch";
import { startLotusDeafen } from "../lotus/lotusDeafen"; import { startLotusDeafen } from "../lotus/lotusDeafen";
import { startLotusDevices } from "../lotus/lotusDevices";
import { useHostBridge } from "../HostBridge.ts"; import { useHostBridge } from "../HostBridge.ts";
import { useRootElement } from "../RootElementContext"; import { useRootElement } from "../RootElementContext";
import { observeElementSize$ } from "../utils/elementSize"; import { observeElementSize$ } from "../utils/elementSize";
@@ -361,6 +362,8 @@ export const InCallView: FC<InCallViewProps> = ({
useEffect(() => startLotusDeafen(), []); useEffect(() => startLotusDeafen(), []);
// [lotus #119] Let the host pick the audio output device from its call bar. // [lotus #119] Let the host pick the audio output device from its call bar.
const lotusMediaDevices = useMediaDevices(); const lotusMediaDevices = useMediaDevices();
// [cinny #121] Remember the picked mic / output by name; report if it goes.
useEffect(() => startLotusDevices(lotusMediaDevices), [lotusMediaDevices]);
useEffect( useEffect(
() => startLotusAudioOutput(lotusMediaDevices), () => startLotusAudioOutput(lotusMediaDevices),
[lotusMediaDevices], [lotusMediaDevices],
@@ -47,6 +47,7 @@ import {
// default to true, so with no params this is byte-for-byte upstream behaviour; // default to true, so with no params this is byte-for-byte upstream behaviour;
// `?autoGainControl=false` forces AGC off regardless of the stored Setting. // `?autoGainControl=false` forces AGC off regardless of the stored Setting.
import { getUrlParams } from "../../../UrlParams.ts"; import { getUrlParams } from "../../../UrlParams.ts";
import { lotusNoiseSuppressionAllowed } from "../../../lotus/lotusCaptureOverride.ts";
// TODO evaluate if this should be done like the Publisher Factory // TODO evaluate if this should be done like the Publisher Factory
export interface ConnectionFactory { export interface ConnectionFactory {
@@ -193,9 +194,16 @@ function generateRoomOption({
echoCancellation: echoCancellation:
echoCancellationSetting.getValue() && echoCancellationSetting.getValue() &&
getUrlParams().echoCancellation !== false, getUrlParams().echoCancellation !== false,
// [lotus] The host can switch this mid-call (io.lotus.set_denoise).
noiseSuppression: noiseSuppression:
noiseSuppressionSetting.getValue() && noiseSuppressionSetting.getValue() &&
getUrlParams().noiseSuppression !== false, lotusNoiseSuppressionAllowed(getUrlParams().noiseSuppression),
// [lotus] LiveKit defaults voiceIsolation ON, which (where the platform
// implements it) is one more voice filter in front of the ML model or
// in "off" mode. Tie it to browser noise suppression.
voiceIsolation:
noiseSuppressionSetting.getValue() &&
lotusNoiseSuppressionAllowed(getUrlParams().noiseSuppression),
autoGainControl: autoGainControl:
autoGainControlSetting.getValue() && autoGainControlSetting.getValue() &&
getUrlParams().autoGainControl !== false, getUrlParams().autoGainControl !== false,
+87
View File
@@ -214,3 +214,90 @@ test("SpotlightTile displays ringing media", async () => {
act(() => pickupState$.next("timeout")); act(() => pickupState$.next("timeout"));
screen.getByText("Call ended"); screen.getByText("Call ended");
}); });
test("[lotus #67] falls back to the share when the visible camera goes away", async () => {
// A controllable IntersectionObserver so the tile's visibleId follows
// navigation the way it does in a browser.
let report: ((entries: IntersectionObserverEntry[]) => void) | undefined;
const RealIO = global.IntersectionObserver;
global.IntersectionObserver = class {
public constructor(
cb: (entries: IntersectionObserverEntry[]) => void,
options?: IntersectionObserverInit,
) {
// Only the carousel's own observer (rooted on the tile); LiveKit's
// video elements create observers of their own.
if (options?.root) report = cb;
}
public observe(): void {}
public unobserve(): void {}
public disconnect(): void {}
} as unknown as typeof IntersectionObserver;
try {
const share = mockRemoteScreenShare(
mockRtcMembership("@alice:example.org", "AAAA"),
{},
mockRemoteParticipant({}),
);
const camera = mockRemoteMedia(
mockRtcMembership("@carol:example.org", "CCCC"),
{ rawDisplayName: "Carol", getMxcAvatarUrl: () => "mxc://carol" },
mockRemoteParticipant({}),
);
// The mocks all use the id "remote"; the carousel keys on ids.
Object.assign(camera, { id: "remote-camera" });
const media$ = new BehaviorSubject([share, camera]);
const user = userEvent.setup();
const { container } = render(
<TooltipProvider>
<SpotlightTile
vm={
new SpotlightTileViewModel(
media$,
constant(false),
constant("solid"),
)
}
targetWidth={300}
targetHeight={200}
expanded={false}
onToggleExpanded={vi.fn()}
showIndicators
showNameTags
showRingingStatus
focusable
/>
</TooltipProvider>,
);
const dots = (): HTMLElement[] =>
screen.getAllByTestId("screenshare-indicator");
expect(dots().map((d) => d.dataset.kind)).toEqual(["screen share", "user"]);
expect(dots()[0].dataset.visible).toBe("true");
// Flip to Carol's camera; the browser would report it as intersecting.
await user.click(screen.getByRole("button", { name: "Next" }));
const carolItem = container.querySelector(`[data-id="${camera.id}"]`)!;
act(() =>
report!([
{
isIntersecting: true,
target: carolItem,
} as unknown as IntersectionObserverEntry,
]),
);
expect(dots()[1].dataset.visible).toBe("true");
expect(isInaccessible(screen.getByText("Carol"))).toBe(false);
// Carol turns her camera off: she drops out and the share is shown again.
act(() => media$.next([share]));
expect(dots()).toHaveLength(1);
expect(dots()[0].dataset.visible).toBe("true");
expect(
isInaccessible(container.querySelector(`[data-id="${share.id}"]`)!),
).toBe(false);
} finally {
global.IntersectionObserver = RealIO;
}
});
+6
View File
@@ -43,6 +43,7 @@ import { useLatest } from "../useLatest";
import { type SpotlightTileViewModel } from "../state/TileViewModel"; import { type SpotlightTileViewModel } from "../state/TileViewModel";
import { useBehavior } from "../useBehavior"; import { useBehavior } from "../useBehavior";
import { muteScreenshareAudio$ } from "../lotus/lotusScreenshareAudio"; import { muteScreenshareAudio$ } from "../lotus/lotusScreenshareAudio";
import { useLotusSpotlightFallback } from "../lotus/lotusSpotlightView";
import { type MemberMediaViewModel } from "../state/media/MemberMediaViewModel"; import { type MemberMediaViewModel } from "../state/media/MemberMediaViewModel";
import { type LocalUserMediaViewModel } from "../state/media/LocalUserMediaViewModel"; import { type LocalUserMediaViewModel } from "../state/media/LocalUserMediaViewModel";
import { type RemoteUserMediaViewModel } from "../state/media/RemoteUserMediaViewModel"; import { type RemoteUserMediaViewModel } from "../state/media/RemoteUserMediaViewModel";
@@ -487,6 +488,10 @@ export const SpotlightTile: FC<Props> = ({
[visibleId], [visibleId],
); );
// [lotus #67] The carousel can hold cameras during a share: keep the share
// as the default view and fall back to it when the visible camera goes away.
useLotusSpotlightFallback(useBehavior(root$), media, visibleId, setVisibleId);
const onBackClick = useCallback(() => { const onBackClick = useCallback(() => {
const media = latestMedia.current; const media = latestMedia.current;
const visibleIndex = media.findIndex( const visibleIndex = media.findIndex(
@@ -593,6 +598,7 @@ export const SpotlightTile: FC<Props> = ({
{media.map((vm) => ( {media.map((vm) => (
<div <div
data-testid="screenshare-indicator" data-testid="screenshare-indicator"
data-kind={vm.type}
key={vm.id} key={vm.id}
className={styles.item} className={styles.item}
data-visible={vm.id === visibleId} data-visible={vm.id === visibleId}