feat(calls): 3-tier mic noise suppression with on-device ML (P5-30)
Replace the boolean call noise-suppression setting with a 3-way control (Off / Browser-native / ML beta) in Settings -> General -> Calls. - Off: noiseSuppression=false to Element Call - Browser-native: EC's built-in WebRTC suppressor (prior default) - ML (beta): on-device RNNoise (@sapphi-red/web-noise-suppressor) Element Call captures the mic inside its iframe and publishes to LiveKit, so the host can't reach that track; LiveKit's Krisp filter is Cloud-only (we self-host the SFU) and EC's own RNNoise PR #3892 is unmerged. The ML tier instead injects a same-origin pre-init shim into the vendored EC index.html (build/lotus-denoise.js, wired by the lotusDenoise vite plugin) that patches getUserMedia and routes the captured mic through an RNNoise AudioWorklet before LiveKit sees it -- the same post-capture pipeline as #3892, with no EC fork/AGPL/rebase burden. Falls back to the raw mic if setup fails; keeps echoCancellation/AGC on the raw capture. - settings.ts: callNoiseSuppression -> 'off'|'browser'|'ml' + legacy boolean migration (true->browser, false->off) - CallEmbed/useCallEmbed: tier maps to noiseSuppression param and appends lotusDenoise=ml (native suppressor off in ML mode) - vite.config.js: copy RNNoise worklet/wasm + shim into the EC bundle and inject the shim <script> before EC's module entry - docs: LOTUS_FEATURES.md, LOTUS_TODO.md (P5-30 done) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
+25
-2
@@ -405,9 +405,32 @@ A local sound plays when another participant joins or leaves a call you're in.
|
|||||||
|
|
||||||
Files: `src/app/utils/callSounds.ts`, `src/app/hooks/useCallJoinLeaveSounds.ts`
|
Files: `src/app/utils/callSounds.ts`, `src/app/hooks/useCallJoinLeaveSounds.ts`
|
||||||
|
|
||||||
### Noise Suppression Toggle
|
### Noise Suppression (3-Tier, incl. on-device ML) (P5-30)
|
||||||
|
|
||||||
A `noiseSuppression` URL parameter is passed to the Element Call widget URL, allowing the noise suppression feature to be toggled from within Lotus settings.
|
A three-way mic noise-suppression control in **Settings → General → Calls**:
|
||||||
|
|
||||||
|
| Tier | What it does |
|
||||||
|
|---|---|
|
||||||
|
| **Off** | No suppression (`noiseSuppression=false` to Element Call). |
|
||||||
|
| **Browser-native** | Element Call's built-in WebRTC suppressor (`noiseSuppression=true`). Default. |
|
||||||
|
| **ML (beta)** | On-device RNNoise — Krisp-style removal of fans, keyboards, dogs, etc. |
|
||||||
|
|
||||||
|
**Why a shim, not a fork:** Element Call captures the mic *inside* its iframe and publishes to LiveKit; the host can't reach that track. LiveKit's Krisp filter is LiveKit-Cloud-only (we self-host the SFU), and EC's own RNNoise work (PR #3892) is unmerged. So the **ML tier** is delivered by injecting a same-origin pre-init script into the vendored EC `index.html` that monkeypatches `getUserMedia` and routes the captured mic through an RNNoise `AudioWorklet` (`@sapphi-red/web-noise-suppressor`) before LiveKit ever sees it — the same post-capture pipeline #3892 uses, executed from the realm we already control. Works on the self-hosted LiveKit SFU, survives EC version bumps, no EC fork/AGPL/rebase burden.
|
||||||
|
|
||||||
|
**How it's wired:**
|
||||||
|
|
||||||
|
- `callNoiseSuppression` setting is `'off' | 'browser' | 'ml'` (legacy boolean migrates: `true`→`browser`, `false`→`off`)
|
||||||
|
- `CallEmbed.getWidget()` maps the tier to the `noiseSuppression` URL param and appends `lotusDenoise=ml` for the ML tier (browser-native suppressor is disabled in ML mode so RNNoise owns suppression)
|
||||||
|
- The `lotusDenoise` vite plugin copies the RNNoise worklet + wasm into `public/element-call/denoise/`, copies the shim, and injects `<script src="./lotus-denoise.js">` before EC's module entry
|
||||||
|
- The shim keeps `echoCancellation`/`autoGainControl` on the raw capture and falls back to the raw mic if RNNoise setup fails, so calls never break
|
||||||
|
|
||||||
|
**Known beta caveat:** routing capture through WebAudio can weaken the browser's acoustic echo cancellation (AEC runs on the native capture track) — the same tradeoff EC's upstream feature makes; hence the "beta" label.
|
||||||
|
|
||||||
|
### Files
|
||||||
|
|
||||||
|
- `build/lotus-denoise.js` — injected RNNoise getUserMedia shim (classic script)
|
||||||
|
- `vite.config.js` — `lotusDenoise()` plugin (asset copy + index.html injection)
|
||||||
|
- `src/app/plugins/call/CallEmbed.ts` — tier → widget URL params
|
||||||
|
|
||||||
### Call Button Scoping
|
### Call Button Scoping
|
||||||
|
|
||||||
|
|||||||
+5
-5
@@ -243,12 +243,12 @@ Themes:
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### [ ] P5-30 · Advanced ML Noise Suppression (Krisp-style)
|
### [x] P5-30 · Advanced ML Noise Suppression (Krisp-style)
|
||||||
|
|
||||||
**What:** High-end background noise cancellation using a pre-trained ML model (e.g. RNNoise) running in the browser. Removes dogs, fans, and keyboard clicks from the mic stream.
|
**What:** High-end background noise cancellation using a pre-trained ML model (RNNoise) running in the browser. Removes dogs, fans, and keyboard clicks from the mic stream.
|
||||||
**Note:** This is a top-tier feature request and an EXTREME COMPLEXITY project.
|
**Shipped:** 3-tier setting (Off / Browser-native / ML beta) in Settings → General → Calls. ML tier injects a same-origin pre-init shim into the vendored Element Call `index.html` that monkeypatches `getUserMedia` and routes the captured mic through an RNNoise `AudioWorklet` (`@sapphi-red/web-noise-suppressor`) before LiveKit publishes — no EC fork required. See LOTUS_FEATURES.md → "Noise Suppression (3-Tier)".
|
||||||
**[AUDIT REQUIRED]** Must verify if mixing a processed stream into Element Call's WebRTC implementation causes latency or AEC (Echo Cancellation) issues.
|
**Key decision:** LiveKit's Krisp filter is LiveKit-Cloud-only (we self-host the SFU); EC's own RNNoise PR #3892 is unmerged. The shim is the same post-capture pipeline #3892 uses, executed from the realm we control, so it survives EC version bumps.
|
||||||
**Complexity:** Extreme.
|
**AEC note (resolved-as-accepted):** WebAudio capture routing can weaken browser AEC (AEC runs on the native track) — same tradeoff as EC's upstream feature; mitigated by keeping `echoCancellation`/`autoGainControl` on the raw capture and labeling the tier "beta". Validate echo quality on real multi-party calls after deploy.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,155 @@
|
|||||||
|
/*
|
||||||
|
* Lotus Chat — client-side ML noise suppression shim for Element Call.
|
||||||
|
*
|
||||||
|
* Element Call runs as a same-origin iframe widget that captures the mic
|
||||||
|
* internally (via livekit-client -> getUserMedia) and publishes it to LiveKit.
|
||||||
|
* We can't reach that track from the host. Instead this classic <script> is
|
||||||
|
* injected (by the vite `lotus-denoise` plugin) into EC's index.html BEFORE its
|
||||||
|
* deferred module entry, so it runs first and monkeypatches getUserMedia. When
|
||||||
|
* the "ml" tier is selected (lotusDenoise=ml in the widget URL) we route the
|
||||||
|
* captured mic through an RNNoise AudioWorklet (@sapphi-red/web-noise-suppressor)
|
||||||
|
* and hand the processed track back to EC/LiveKit.
|
||||||
|
*
|
||||||
|
* This mirrors Element Call's own (still-unmerged) PR #3892 pipeline, executed
|
||||||
|
* from the realm we already control instead of forking and rebuilding EC.
|
||||||
|
*
|
||||||
|
* Known beta caveat: routing capture through WebAudio can weaken the browser's
|
||||||
|
* acoustic echo cancellation (AEC operates on the native capture track). We keep
|
||||||
|
* echoCancellation/autoGainControl enabled on the raw capture to mitigate.
|
||||||
|
*/
|
||||||
|
(function () {
|
||||||
|
'use strict';
|
||||||
|
|
||||||
|
try {
|
||||||
|
var params = new URLSearchParams(window.location.search);
|
||||||
|
if (params.get('lotusDenoise') !== 'ml') return;
|
||||||
|
} catch (e) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
var md = navigator.mediaDevices;
|
||||||
|
if (!md || typeof md.getUserMedia !== 'function') return;
|
||||||
|
if (typeof AudioWorkletNode === 'undefined' || typeof AudioContext === 'undefined') return;
|
||||||
|
|
||||||
|
var PROCESSOR_NAME = '@sapphi-red/web-noise-suppressor/rnnoise';
|
||||||
|
var ASSET_BASE = './denoise/';
|
||||||
|
var SAMPLE_RATE = 48000; // RNNoise worklet assumes 48kHz
|
||||||
|
|
||||||
|
var origGetUserMedia = md.getUserMedia.bind(md);
|
||||||
|
var wasmPromise = null;
|
||||||
|
|
||||||
|
// SIMD feature detection (bytes from @sapphi-red/web-noise-suppressor / wasm-feature-detect)
|
||||||
|
function hasSimd() {
|
||||||
|
try {
|
||||||
|
return WebAssembly.validate(
|
||||||
|
new Uint8Array([
|
||||||
|
0, 97, 115, 109, 1, 0, 0, 0, 1, 5, 1, 96, 0, 1, 123, 3, 2, 1, 0, 10, 10, 1, 8, 0, 65, 0,
|
||||||
|
253, 15, 253, 98, 11,
|
||||||
|
])
|
||||||
|
);
|
||||||
|
} catch (e) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function loadWasm() {
|
||||||
|
if (!wasmPromise) {
|
||||||
|
var url = ASSET_BASE + (hasSimd() ? 'rnnoise_simd.wasm' : 'rnnoise.wasm');
|
||||||
|
wasmPromise = fetch(url).then(function (r) {
|
||||||
|
if (!r.ok) throw new Error('rnnoise wasm fetch failed: ' + r.status);
|
||||||
|
return r.arrayBuffer();
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return wasmPromise;
|
||||||
|
}
|
||||||
|
|
||||||
|
function processStream(stream) {
|
||||||
|
var audioTracks = stream.getAudioTracks();
|
||||||
|
if (audioTracks.length === 0) return Promise.resolve(stream);
|
||||||
|
|
||||||
|
return loadWasm()
|
||||||
|
.then(function (wasmBinary) {
|
||||||
|
var ctx = new AudioContext({ sampleRate: SAMPLE_RATE });
|
||||||
|
return ctx.audioWorklet
|
||||||
|
.addModule(ASSET_BASE + 'rnnoiseWorklet.js')
|
||||||
|
.then(function () {
|
||||||
|
if (ctx.state === 'suspended') return ctx.resume().then(function () { return ctx; });
|
||||||
|
return ctx;
|
||||||
|
})
|
||||||
|
.then(function () {
|
||||||
|
var node = new AudioWorkletNode(ctx, PROCESSOR_NAME, {
|
||||||
|
processorOptions: { maxChannels: 1, wasmBinary: wasmBinary },
|
||||||
|
});
|
||||||
|
var source = ctx.createMediaStreamSource(stream);
|
||||||
|
var dest = ctx.createMediaStreamDestination();
|
||||||
|
source.connect(node).connect(dest);
|
||||||
|
|
||||||
|
var origTrack = audioTracks[0];
|
||||||
|
var processedTrack = dest.stream.getAudioTracks()[0];
|
||||||
|
|
||||||
|
var torndown = false;
|
||||||
|
function cleanup() {
|
||||||
|
if (torndown) return;
|
||||||
|
torndown = true;
|
||||||
|
try {
|
||||||
|
node.port.postMessage('destroy');
|
||||||
|
} catch (e) {}
|
||||||
|
try {
|
||||||
|
source.disconnect();
|
||||||
|
node.disconnect();
|
||||||
|
} catch (e) {}
|
||||||
|
try {
|
||||||
|
origTrack.stop();
|
||||||
|
} catch (e) {}
|
||||||
|
try {
|
||||||
|
ctx.close();
|
||||||
|
} catch (e) {}
|
||||||
|
}
|
||||||
|
|
||||||
|
// When EC stops the track we handed it, release the raw capture + graph.
|
||||||
|
var rawStop = processedTrack.stop.bind(processedTrack);
|
||||||
|
processedTrack.stop = function () {
|
||||||
|
cleanup();
|
||||||
|
rawStop();
|
||||||
|
};
|
||||||
|
// Device unplugged / capture ended involuntarily.
|
||||||
|
origTrack.addEventListener('ended', function () {
|
||||||
|
try {
|
||||||
|
rawStop();
|
||||||
|
} catch (e) {}
|
||||||
|
cleanup();
|
||||||
|
});
|
||||||
|
|
||||||
|
// Return a stream with the processed audio plus any original video.
|
||||||
|
var out = new MediaStream();
|
||||||
|
out.addTrack(processedTrack);
|
||||||
|
stream.getVideoTracks().forEach(function (t) {
|
||||||
|
out.addTrack(t);
|
||||||
|
});
|
||||||
|
return out;
|
||||||
|
});
|
||||||
|
})
|
||||||
|
.catch(function (e) {
|
||||||
|
// Any failure -> fall back to the raw mic so calls never break.
|
||||||
|
// eslint-disable-next-line no-console
|
||||||
|
console.error('[lotus-denoise] RNNoise setup failed, using raw mic', e);
|
||||||
|
return stream;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
navigator.mediaDevices.getUserMedia = function (constraints) {
|
||||||
|
var wantsAudio = !!(constraints && constraints.audio);
|
||||||
|
var effective = constraints;
|
||||||
|
if (wantsAudio) {
|
||||||
|
// RNNoise owns noise suppression; keep AEC + AGC on the raw capture.
|
||||||
|
var audioC = typeof constraints.audio === 'object' ? Object.assign({}, constraints.audio) : {};
|
||||||
|
audioC.noiseSuppression = false;
|
||||||
|
if (audioC.echoCancellation === undefined) audioC.echoCancellation = true;
|
||||||
|
if (audioC.autoGainControl === undefined) audioC.autoGainControl = true;
|
||||||
|
effective = Object.assign({}, constraints, { audio: audioC });
|
||||||
|
}
|
||||||
|
return origGetUserMedia(effective).then(function (stream) {
|
||||||
|
return wantsAudio ? processStream(stream) : stream;
|
||||||
|
});
|
||||||
|
};
|
||||||
|
})();
|
||||||
Generated
+7
@@ -20,6 +20,7 @@
|
|||||||
"@giphy/js-types": "5.1.0",
|
"@giphy/js-types": "5.1.0",
|
||||||
"@giphy/js-util": "5.2.0",
|
"@giphy/js-util": "5.2.0",
|
||||||
"@giphy/react-components": "10.1.2",
|
"@giphy/react-components": "10.1.2",
|
||||||
|
"@sapphi-red/web-noise-suppressor": "0.3.5",
|
||||||
"@sentry/react": "10.53.1",
|
"@sentry/react": "10.53.1",
|
||||||
"@tanstack/react-query": "5.100.13",
|
"@tanstack/react-query": "5.100.13",
|
||||||
"@tanstack/react-query-devtools": "5.100.13",
|
"@tanstack/react-query-devtools": "5.100.13",
|
||||||
@@ -3774,6 +3775,12 @@
|
|||||||
"dev": true,
|
"dev": true,
|
||||||
"license": "MIT"
|
"license": "MIT"
|
||||||
},
|
},
|
||||||
|
"node_modules/@sapphi-red/web-noise-suppressor": {
|
||||||
|
"version": "0.3.5",
|
||||||
|
"resolved": "https://registry.npmjs.org/@sapphi-red/web-noise-suppressor/-/web-noise-suppressor-0.3.5.tgz",
|
||||||
|
"integrity": "sha512-jh3+V9yM+zxLriQexoGm0GatoPaJWjs6ypFIbFYwQp+AoUb55eUXrjKtKQyuC5zShzzeAQUl0M5JzqB7SSrsRA==",
|
||||||
|
"license": "MIT"
|
||||||
|
},
|
||||||
"node_modules/@sentry-internal/browser-utils": {
|
"node_modules/@sentry-internal/browser-utils": {
|
||||||
"version": "10.53.1",
|
"version": "10.53.1",
|
||||||
"resolved": "https://registry.npmjs.org/@sentry-internal/browser-utils/-/browser-utils-10.53.1.tgz",
|
"resolved": "https://registry.npmjs.org/@sentry-internal/browser-utils/-/browser-utils-10.53.1.tgz",
|
||||||
|
|||||||
@@ -44,6 +44,7 @@
|
|||||||
"@giphy/js-types": "5.1.0",
|
"@giphy/js-types": "5.1.0",
|
||||||
"@giphy/js-util": "5.2.0",
|
"@giphy/js-util": "5.2.0",
|
||||||
"@giphy/react-components": "10.1.2",
|
"@giphy/react-components": "10.1.2",
|
||||||
|
"@sapphi-red/web-noise-suppressor": "0.3.5",
|
||||||
"@sentry/react": "10.53.1",
|
"@sentry/react": "10.53.1",
|
||||||
"@tanstack/react-query": "5.100.13",
|
"@tanstack/react-query": "5.100.13",
|
||||||
"@tanstack/react-query-devtools": "5.100.13",
|
"@tanstack/react-query-devtools": "5.100.13",
|
||||||
|
|||||||
@@ -42,13 +42,11 @@ import {
|
|||||||
DateFormat,
|
DateFormat,
|
||||||
MessageLayout,
|
MessageLayout,
|
||||||
MessageSpacing,
|
MessageSpacing,
|
||||||
|
NoiseSuppressionMode,
|
||||||
Settings,
|
Settings,
|
||||||
settingsAtom,
|
settingsAtom,
|
||||||
} from '../../../state/settings';
|
} from '../../../state/settings';
|
||||||
import {
|
import { SeasonalPreview, SeasonTheme } from '../../../components/seasonal/SeasonalEffect';
|
||||||
SeasonalPreview,
|
|
||||||
SeasonTheme,
|
|
||||||
} from '../../../components/seasonal/SeasonalEffect';
|
|
||||||
import { SettingTile } from '../../../components/setting-tile';
|
import { SettingTile } from '../../../components/setting-tile';
|
||||||
import { KeySymbol } from '../../../utils/key-symbol';
|
import { KeySymbol } from '../../../utils/key-symbol';
|
||||||
import { isMacOS } from '../../../utils/user-agent';
|
import { isMacOS } from '../../../utils/user-agent';
|
||||||
@@ -1235,12 +1233,16 @@ function Calls() {
|
|||||||
<SequenceCard className={SequenceCardStyle} variant="SurfaceVariant" direction="Column">
|
<SequenceCard className={SequenceCardStyle} variant="SurfaceVariant" direction="Column">
|
||||||
<SettingTile
|
<SettingTile
|
||||||
title="Noise Suppression"
|
title="Noise Suppression"
|
||||||
description="Apply AI noise suppression to filter background noise during calls (powered by Element Call)."
|
description="Filter background noise from your mic during calls. Browser-native uses the built-in WebRTC suppressor; ML runs on-device RNNoise for stronger, Krisp-style removal (higher CPU)."
|
||||||
after={
|
after={
|
||||||
<Switch
|
<SettingsSelect<NoiseSuppressionMode>
|
||||||
variant="Primary"
|
|
||||||
value={callNoiseSuppression}
|
value={callNoiseSuppression}
|
||||||
onChange={setCallNoiseSuppression}
|
onChange={setCallNoiseSuppression}
|
||||||
|
options={[
|
||||||
|
{ value: 'off', label: 'Off' },
|
||||||
|
{ value: 'browser', label: 'Browser-native' },
|
||||||
|
{ value: 'ml', label: 'ML (beta)' },
|
||||||
|
]}
|
||||||
/>
|
/>
|
||||||
}
|
}
|
||||||
/>
|
/>
|
||||||
@@ -1346,22 +1348,25 @@ function Calls() {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
const SEASONAL_OPTIONS: { value: Settings['seasonalThemeOverride']; label: string; emoji: string }[] =
|
const SEASONAL_OPTIONS: {
|
||||||
[
|
value: Settings['seasonalThemeOverride'];
|
||||||
{ value: 'auto', label: 'Auto', emoji: '🗓' },
|
label: string;
|
||||||
{ value: 'off', label: 'Off', emoji: '×' },
|
emoji: string;
|
||||||
{ value: 'newyear', label: 'New Year', emoji: '🎆' },
|
}[] = [
|
||||||
{ value: 'lunar', label: 'Lunar New Year', emoji: '🏮' },
|
{ value: 'auto', label: 'Auto', emoji: '🗓' },
|
||||||
{ value: 'valentines', label: "Valentine's", emoji: '💖' },
|
{ value: 'off', label: 'Off', emoji: '×' },
|
||||||
{ value: 'stpatricks', label: "St. Patrick's", emoji: '🍀' },
|
{ value: 'newyear', label: 'New Year', emoji: '🎆' },
|
||||||
{ value: 'aprilfools', label: 'April Fools', emoji: '?' },
|
{ value: 'lunar', label: 'Lunar New Year', emoji: '🏮' },
|
||||||
{ value: 'earthday', label: 'Earth Day', emoji: '🌱' },
|
{ value: 'valentines', label: "Valentine's", emoji: '💖' },
|
||||||
{ value: 'autumn', label: 'Autumn', emoji: '🍂' },
|
{ value: 'stpatricks', label: "St. Patrick's", emoji: '🍀' },
|
||||||
{ value: 'halloween', label: 'Halloween', emoji: '🎃' },
|
{ value: 'aprilfools', label: 'April Fools', emoji: '?' },
|
||||||
{ value: 'christmas', label: 'Christmas', emoji: '❄️' },
|
{ value: 'earthday', label: 'Earth Day', emoji: '🌱' },
|
||||||
{ value: 'arcade', label: 'Arcade Day', emoji: '👾' },
|
{ value: 'autumn', label: 'Autumn', emoji: '🍂' },
|
||||||
{ value: 'deepspace', label: 'Deep Space', emoji: '🚀' },
|
{ value: 'halloween', label: 'Halloween', emoji: '🎃' },
|
||||||
];
|
{ value: 'christmas', label: 'Christmas', emoji: '❄️' },
|
||||||
|
{ value: 'arcade', label: 'Arcade Day', emoji: '👾' },
|
||||||
|
{ value: 'deepspace', label: 'Deep Space', emoji: '🚀' },
|
||||||
|
];
|
||||||
|
|
||||||
function SeasonalBgGrid({
|
function SeasonalBgGrid({
|
||||||
value,
|
value,
|
||||||
|
|||||||
@@ -15,7 +15,7 @@ import { CallControlState } from '../plugins/call/CallControlState';
|
|||||||
import { useCallMembersChange, useCallSession } from './useCall';
|
import { useCallMembersChange, useCallSession } from './useCall';
|
||||||
import { CallPreferences } from '../state/callPreferences';
|
import { CallPreferences } from '../state/callPreferences';
|
||||||
import { useSetting } from '../state/hooks/settings';
|
import { useSetting } from '../state/hooks/settings';
|
||||||
import { settingsAtom } from '../state/settings';
|
import { NoiseSuppressionMode, settingsAtom } from '../state/settings';
|
||||||
import { unlockCallSounds } from '../utils/callSounds';
|
import { unlockCallSounds } from '../utils/callSounds';
|
||||||
|
|
||||||
const CallEmbedContext = createContext<CallEmbed | undefined>(undefined);
|
const CallEmbedContext = createContext<CallEmbed | undefined>(undefined);
|
||||||
@@ -45,7 +45,7 @@ export const createCallEmbed = (
|
|||||||
themeKind: ElementCallThemeKind,
|
themeKind: ElementCallThemeKind,
|
||||||
container: HTMLElement,
|
container: HTMLElement,
|
||||||
pref?: CallPreferences,
|
pref?: CallPreferences,
|
||||||
noiseSuppression = true,
|
denoiseMode: NoiseSuppressionMode = 'browser',
|
||||||
forceAudioOff = false,
|
forceAudioOff = false,
|
||||||
): CallEmbed => {
|
): CallEmbed => {
|
||||||
const rtcSession = mx.matrixRTC.getRoomSession(room);
|
const rtcSession = mx.matrixRTC.getRoomSession(room);
|
||||||
@@ -59,7 +59,7 @@ export const createCallEmbed = (
|
|||||||
room,
|
room,
|
||||||
intent,
|
intent,
|
||||||
themeKind,
|
themeKind,
|
||||||
noiseSuppression,
|
denoiseMode,
|
||||||
initialAudio,
|
initialAudio,
|
||||||
initialVideo,
|
initialVideo,
|
||||||
);
|
);
|
||||||
@@ -96,7 +96,7 @@ export const useCallStart = (dm = false) => {
|
|||||||
theme.kind,
|
theme.kind,
|
||||||
container,
|
container,
|
||||||
pref,
|
pref,
|
||||||
callNoiseSuppression ?? true,
|
callNoiseSuppression ?? 'browser',
|
||||||
!!pttMode,
|
!!pttMode,
|
||||||
);
|
);
|
||||||
|
|
||||||
|
|||||||
@@ -18,6 +18,7 @@ import {
|
|||||||
} from 'matrix-widget-api';
|
} from 'matrix-widget-api';
|
||||||
import { CallWidgetDriver } from './CallWidgetDriver';
|
import { CallWidgetDriver } from './CallWidgetDriver';
|
||||||
import { trimTrailingSlash } from '../../utils/common';
|
import { trimTrailingSlash } from '../../utils/common';
|
||||||
|
import { NoiseSuppressionMode } from '../../state/settings';
|
||||||
import {
|
import {
|
||||||
ElementCallIntent,
|
ElementCallIntent,
|
||||||
ElementCallThemeKind,
|
ElementCallThemeKind,
|
||||||
@@ -100,7 +101,7 @@ export class CallEmbed {
|
|||||||
room: Room,
|
room: Room,
|
||||||
intent: ElementCallIntent,
|
intent: ElementCallIntent,
|
||||||
themeKind: ElementCallThemeKind,
|
themeKind: ElementCallThemeKind,
|
||||||
noiseSuppression = true,
|
denoiseMode: NoiseSuppressionMode = 'browser',
|
||||||
initialAudio = true,
|
initialAudio = true,
|
||||||
initialVideo = false,
|
initialVideo = false,
|
||||||
): Widget {
|
): Widget {
|
||||||
@@ -124,12 +125,21 @@ export class CallEmbed {
|
|||||||
perParticipantE2EE: room.hasEncryptionStateEvent().toString(),
|
perParticipantE2EE: room.hasEncryptionStateEvent().toString(),
|
||||||
lang: 'en-EN',
|
lang: 'en-EN',
|
||||||
theme: themeKind,
|
theme: themeKind,
|
||||||
noiseSuppression: noiseSuppression.toString(),
|
// EC's built-in WebRTC suppressor: on only for 'browser' tier. For 'ml' we
|
||||||
|
// disable it here so RNNoise (the Lotus denoise shim) owns suppression and
|
||||||
|
// the two don't fight each other.
|
||||||
|
noiseSuppression: (denoiseMode === 'browser').toString(),
|
||||||
audio: initialAudio.toString(),
|
audio: initialAudio.toString(),
|
||||||
video: initialVideo.toString(),
|
video: initialVideo.toString(),
|
||||||
header: 'none',
|
header: 'none',
|
||||||
});
|
});
|
||||||
|
|
||||||
|
if (denoiseMode === 'ml') {
|
||||||
|
// Signal the Lotus denoise shim (injected into the EC index.html) to route
|
||||||
|
// the mic through the RNNoise worklet before LiveKit publishes the track.
|
||||||
|
params.append('lotusDenoise', 'ml');
|
||||||
|
}
|
||||||
|
|
||||||
if (CallEmbed.startingCall(intent)) {
|
if (CallEmbed.startingCall(intent)) {
|
||||||
params.append('sendNotificationType', CallEmbed.dmCall(intent) ? 'ring' : 'notification');
|
params.append('sendNotificationType', CallEmbed.dmCall(intent) ? 'ring' : 'notification');
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -9,6 +9,11 @@ export type DateFormat =
|
|||||||
| 'YYYY-MM-DD'
|
| 'YYYY-MM-DD'
|
||||||
| '';
|
| '';
|
||||||
export type MessageSpacing = '0' | '100' | '200' | '300' | '400' | '500';
|
export type MessageSpacing = '0' | '100' | '200' | '300' | '400' | '500';
|
||||||
|
// Call mic noise suppression tier:
|
||||||
|
// - 'off' : no suppression
|
||||||
|
// - 'browser' : WebRTC built-in suppression (Element Call noiseSuppression param)
|
||||||
|
// - 'ml' : client-side RNNoise ML suppression (Lotus denoise shim)
|
||||||
|
export type NoiseSuppressionMode = 'off' | 'browser' | 'ml';
|
||||||
export type ChatBackground =
|
export type ChatBackground =
|
||||||
| 'none'
|
| 'none'
|
||||||
| 'blueprint'
|
| 'blueprint'
|
||||||
@@ -109,7 +114,7 @@ export interface Settings {
|
|||||||
perMessageProfiles: boolean;
|
perMessageProfiles: boolean;
|
||||||
|
|
||||||
cameraOnJoin: boolean;
|
cameraOnJoin: boolean;
|
||||||
callNoiseSuppression: boolean;
|
callNoiseSuppression: NoiseSuppressionMode;
|
||||||
pttMode: boolean;
|
pttMode: boolean;
|
||||||
pttKey: string;
|
pttKey: string;
|
||||||
|
|
||||||
@@ -199,7 +204,7 @@ const defaultSettings: Settings = {
|
|||||||
perMessageProfiles: false,
|
perMessageProfiles: false,
|
||||||
|
|
||||||
cameraOnJoin: false,
|
cameraOnJoin: false,
|
||||||
callNoiseSuppression: true,
|
callNoiseSuppression: 'browser',
|
||||||
pttMode: false,
|
pttMode: false,
|
||||||
pttKey: 'Space',
|
pttKey: 'Space',
|
||||||
|
|
||||||
@@ -235,6 +240,14 @@ export const getSettings = (): Settings => {
|
|||||||
return {
|
return {
|
||||||
...defaultSettings,
|
...defaultSettings,
|
||||||
...saved,
|
...saved,
|
||||||
|
// Migrate legacy boolean callNoiseSuppression -> 3-way mode:
|
||||||
|
// true => browser-native, false => off. New string values pass through.
|
||||||
|
callNoiseSuppression:
|
||||||
|
typeof saved.callNoiseSuppression === 'boolean'
|
||||||
|
? saved.callNoiseSuppression
|
||||||
|
? 'browser'
|
||||||
|
: 'off'
|
||||||
|
: (saved.callNoiseSuppression ?? defaultSettings.callNoiseSuppression),
|
||||||
composerToolbarButtons: {
|
composerToolbarButtons: {
|
||||||
...DEFAULT_COMPOSER_TOOLBAR,
|
...DEFAULT_COMPOSER_TOOLBAR,
|
||||||
...(saved.composerToolbarButtons ?? {}),
|
...(saved.composerToolbarButtons ?? {}),
|
||||||
|
|||||||
@@ -65,6 +65,52 @@ function copyPdfWorker() {
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Lotus ML noise suppression: ship the RNNoise worklet/wasm + our getUserMedia
|
||||||
|
// shim alongside the vendored Element Call bundle, and inject the shim <script>
|
||||||
|
// into EC's index.html so it runs before EC captures the mic. Runs after
|
||||||
|
// viteStaticCopy has populated dist/public/element-call (closeBundle order).
|
||||||
|
function lotusDenoise() {
|
||||||
|
return {
|
||||||
|
name: 'lotus-denoise',
|
||||||
|
closeBundle() {
|
||||||
|
const ecDir = path.resolve('dist/public/element-call');
|
||||||
|
if (!fs.existsSync(ecDir)) return;
|
||||||
|
|
||||||
|
const denoiseDir = path.join(ecDir, 'denoise');
|
||||||
|
fs.mkdirSync(denoiseDir, { recursive: true });
|
||||||
|
|
||||||
|
const sapphi = path.resolve('node_modules/@sapphi-red/web-noise-suppressor/dist');
|
||||||
|
const assets = [
|
||||||
|
[
|
||||||
|
path.join(sapphi, 'rnnoise/workletProcessor.js'),
|
||||||
|
path.join(denoiseDir, 'rnnoiseWorklet.js'),
|
||||||
|
],
|
||||||
|
[path.join(sapphi, 'rnnoise.wasm'), path.join(denoiseDir, 'rnnoise.wasm')],
|
||||||
|
[path.join(sapphi, 'rnnoise_simd.wasm'), path.join(denoiseDir, 'rnnoise_simd.wasm')],
|
||||||
|
];
|
||||||
|
assets.forEach(([s, d]) => {
|
||||||
|
if (fs.existsSync(s)) fs.copyFileSync(s, d);
|
||||||
|
});
|
||||||
|
|
||||||
|
const shimSrc = path.resolve('build/lotus-denoise.js');
|
||||||
|
if (fs.existsSync(shimSrc)) fs.copyFileSync(shimSrc, path.join(ecDir, 'lotus-denoise.js'));
|
||||||
|
|
||||||
|
const indexPath = path.join(ecDir, 'index.html');
|
||||||
|
if (fs.existsSync(indexPath)) {
|
||||||
|
let html = fs.readFileSync(indexPath, 'utf8');
|
||||||
|
if (!html.includes('lotus-denoise.js')) {
|
||||||
|
// Classic (non-deferred) script runs before EC's deferred module entry.
|
||||||
|
html = html.replace(
|
||||||
|
/<script type="module"/,
|
||||||
|
'<script src="./lotus-denoise.js"></script><script type="module"',
|
||||||
|
);
|
||||||
|
fs.writeFileSync(indexPath, html);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
function serverMatrixSdkCryptoWasm(wasmFilePath) {
|
function serverMatrixSdkCryptoWasm(wasmFilePath) {
|
||||||
return {
|
return {
|
||||||
name: 'vite-plugin-serve-matrix-sdk-crypto-wasm',
|
name: 'vite-plugin-serve-matrix-sdk-crypto-wasm',
|
||||||
@@ -133,6 +179,7 @@ export default defineConfig({
|
|||||||
wasm(),
|
wasm(),
|
||||||
react(),
|
react(),
|
||||||
copyPdfWorker(),
|
copyPdfWorker(),
|
||||||
|
lotusDenoise(),
|
||||||
...(process.env.SENTRY_AUTH_TOKEN
|
...(process.env.SENTRY_AUTH_TOKEN
|
||||||
? [
|
? [
|
||||||
sentryVitePlugin({
|
sentryVitePlugin({
|
||||||
|
|||||||
Reference in New Issue
Block a user