feat(calls): 3-tier mic noise suppression with on-device ML (P5-30)
CI / Build & Quality Checks (push) Successful in 10m33s
Trigger Desktop Build / trigger (push) Successful in 6s

Replace the boolean call noise-suppression setting with a 3-way control
(Off / Browser-native / ML beta) in Settings -> General -> Calls.

- Off: noiseSuppression=false to Element Call
- Browser-native: EC's built-in WebRTC suppressor (prior default)
- ML (beta): on-device RNNoise (@sapphi-red/web-noise-suppressor)

Element Call captures the mic inside its iframe and publishes to LiveKit,
so the host can't reach that track; LiveKit's Krisp filter is Cloud-only
(we self-host the SFU) and EC's own RNNoise PR #3892 is unmerged. The ML
tier instead injects a same-origin pre-init shim into the vendored EC
index.html (build/lotus-denoise.js, wired by the lotusDenoise vite plugin)
that patches getUserMedia and routes the captured mic through an RNNoise
AudioWorklet before LiveKit sees it -- the same post-capture pipeline as
#3892, with no EC fork/AGPL/rebase burden. Falls back to the raw mic if
setup fails; keeps echoCancellation/AGC on the raw capture.

- settings.ts: callNoiseSuppression -> 'off'|'browser'|'ml' + legacy
  boolean migration (true->browser, false->off)
- CallEmbed/useCallEmbed: tier maps to noiseSuppression param and appends
  lotusDenoise=ml (native suppressor off in ML mode)
- vite.config.js: copy RNNoise worklet/wasm + shim into the EC bundle and
  inject the shim <script> before EC's module entry
- docs: LOTUS_FEATURES.md, LOTUS_TODO.md (P5-30 done)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-15 20:29:59 -04:00
parent f9edd2023d
commit 5deed79b42
10 changed files with 299 additions and 38 deletions
+25 -2
View File
@@ -405,9 +405,32 @@ A local sound plays when another participant joins or leaves a call you're in.
Files: `src/app/utils/callSounds.ts`, `src/app/hooks/useCallJoinLeaveSounds.ts`
### Noise Suppression Toggle
### Noise Suppression (3-Tier, incl. on-device ML) (P5-30)
A `noiseSuppression` URL parameter is passed to the Element Call widget URL, allowing the noise suppression feature to be toggled from within Lotus settings.
A three-way mic noise-suppression control in **Settings → General → Calls**:
| Tier | What it does |
|---|---|
| **Off** | No suppression (`noiseSuppression=false` to Element Call). |
| **Browser-native** | Element Call's built-in WebRTC suppressor (`noiseSuppression=true`). Default. |
| **ML (beta)** | On-device RNNoise — Krisp-style removal of fans, keyboards, dogs, etc. |
**Why a shim, not a fork:** Element Call captures the mic *inside* its iframe and publishes to LiveKit; the host can't reach that track. LiveKit's Krisp filter is LiveKit-Cloud-only (we self-host the SFU), and EC's own RNNoise work (PR #3892) is unmerged. So the **ML tier** is delivered by injecting a same-origin pre-init script into the vendored EC `index.html` that monkeypatches `getUserMedia` and routes the captured mic through an RNNoise `AudioWorklet` (`@sapphi-red/web-noise-suppressor`) before LiveKit ever sees it — the same post-capture pipeline #3892 uses, executed from the realm we already control. Works on the self-hosted LiveKit SFU, survives EC version bumps, no EC fork/AGPL/rebase burden.
**How it's wired:**
- `callNoiseSuppression` setting is `'off' | 'browser' | 'ml'` (legacy boolean migrates: `true``browser`, `false``off`)
- `CallEmbed.getWidget()` maps the tier to the `noiseSuppression` URL param and appends `lotusDenoise=ml` for the ML tier (browser-native suppressor is disabled in ML mode so RNNoise owns suppression)
- The `lotusDenoise` vite plugin copies the RNNoise worklet + wasm into `public/element-call/denoise/`, copies the shim, and injects `<script src="./lotus-denoise.js">` before EC's module entry
- The shim keeps `echoCancellation`/`autoGainControl` on the raw capture and falls back to the raw mic if RNNoise setup fails, so calls never break
**Known beta caveat:** routing capture through WebAudio can weaken the browser's acoustic echo cancellation (AEC runs on the native capture track) — the same tradeoff EC's upstream feature makes; hence the "beta" label.
### Files
- `build/lotus-denoise.js` — injected RNNoise getUserMedia shim (classic script)
- `vite.config.js``lotusDenoise()` plugin (asset copy + index.html injection)
- `src/app/plugins/call/CallEmbed.ts` — tier → widget URL params
### Call Button Scoping
+5 -5
View File
@@ -243,12 +243,12 @@ Themes:
---
### [ ] P5-30 · Advanced ML Noise Suppression (Krisp-style)
### [x] P5-30 · Advanced ML Noise Suppression (Krisp-style)
**What:** High-end background noise cancellation using a pre-trained ML model (e.g. RNNoise) running in the browser. Removes dogs, fans, and keyboard clicks from the mic stream.
**Note:** This is a top-tier feature request and an EXTREME COMPLEXITY project.
**[AUDIT REQUIRED]** Must verify if mixing a processed stream into Element Call's WebRTC implementation causes latency or AEC (Echo Cancellation) issues.
**Complexity:** Extreme.
**What:** High-end background noise cancellation using a pre-trained ML model (RNNoise) running in the browser. Removes dogs, fans, and keyboard clicks from the mic stream.
**Shipped:** 3-tier setting (Off / Browser-native / ML beta) in Settings → General → Calls. ML tier injects a same-origin pre-init shim into the vendored Element Call `index.html` that monkeypatches `getUserMedia` and routes the captured mic through an RNNoise `AudioWorklet` (`@sapphi-red/web-noise-suppressor`) before LiveKit publishes — no EC fork required. See LOTUS_FEATURES.md → "Noise Suppression (3-Tier)".
**Key decision:** LiveKit's Krisp filter is LiveKit-Cloud-only (we self-host the SFU); EC's own RNNoise PR #3892 is unmerged. The shim is the same post-capture pipeline #3892 uses, executed from the realm we control, so it survives EC version bumps.
**AEC note (resolved-as-accepted):** WebAudio capture routing can weaken browser AEC (AEC runs on the native track) — same tradeoff as EC's upstream feature; mitigated by keeping `echoCancellation`/`autoGainControl` on the raw capture and labeling the tier "beta". Validate echo quality on real multi-party calls after deploy.
---
+155
View File
@@ -0,0 +1,155 @@
/*
* Lotus Chat — client-side ML noise suppression shim for Element Call.
*
* Element Call runs as a same-origin iframe widget that captures the mic
* internally (via livekit-client -> getUserMedia) and publishes it to LiveKit.
* We can't reach that track from the host. Instead this classic <script> is
* injected (by the vite `lotus-denoise` plugin) into EC's index.html BEFORE its
* deferred module entry, so it runs first and monkeypatches getUserMedia. When
* the "ml" tier is selected (lotusDenoise=ml in the widget URL) we route the
* captured mic through an RNNoise AudioWorklet (@sapphi-red/web-noise-suppressor)
* and hand the processed track back to EC/LiveKit.
*
* This mirrors Element Call's own (still-unmerged) PR #3892 pipeline, executed
* from the realm we already control instead of forking and rebuilding EC.
*
* Known beta caveat: routing capture through WebAudio can weaken the browser's
* acoustic echo cancellation (AEC operates on the native capture track). We keep
* echoCancellation/autoGainControl enabled on the raw capture to mitigate.
*/
(function () {
'use strict';
try {
var params = new URLSearchParams(window.location.search);
if (params.get('lotusDenoise') !== 'ml') return;
} catch (e) {
return;
}
var md = navigator.mediaDevices;
if (!md || typeof md.getUserMedia !== 'function') return;
if (typeof AudioWorkletNode === 'undefined' || typeof AudioContext === 'undefined') return;
var PROCESSOR_NAME = '@sapphi-red/web-noise-suppressor/rnnoise';
var ASSET_BASE = './denoise/';
var SAMPLE_RATE = 48000; // RNNoise worklet assumes 48kHz
var origGetUserMedia = md.getUserMedia.bind(md);
var wasmPromise = null;
// SIMD feature detection (bytes from @sapphi-red/web-noise-suppressor / wasm-feature-detect)
function hasSimd() {
try {
return WebAssembly.validate(
new Uint8Array([
0, 97, 115, 109, 1, 0, 0, 0, 1, 5, 1, 96, 0, 1, 123, 3, 2, 1, 0, 10, 10, 1, 8, 0, 65, 0,
253, 15, 253, 98, 11,
])
);
} catch (e) {
return false;
}
}
function loadWasm() {
if (!wasmPromise) {
var url = ASSET_BASE + (hasSimd() ? 'rnnoise_simd.wasm' : 'rnnoise.wasm');
wasmPromise = fetch(url).then(function (r) {
if (!r.ok) throw new Error('rnnoise wasm fetch failed: ' + r.status);
return r.arrayBuffer();
});
}
return wasmPromise;
}
function processStream(stream) {
var audioTracks = stream.getAudioTracks();
if (audioTracks.length === 0) return Promise.resolve(stream);
return loadWasm()
.then(function (wasmBinary) {
var ctx = new AudioContext({ sampleRate: SAMPLE_RATE });
return ctx.audioWorklet
.addModule(ASSET_BASE + 'rnnoiseWorklet.js')
.then(function () {
if (ctx.state === 'suspended') return ctx.resume().then(function () { return ctx; });
return ctx;
})
.then(function () {
var node = new AudioWorkletNode(ctx, PROCESSOR_NAME, {
processorOptions: { maxChannels: 1, wasmBinary: wasmBinary },
});
var source = ctx.createMediaStreamSource(stream);
var dest = ctx.createMediaStreamDestination();
source.connect(node).connect(dest);
var origTrack = audioTracks[0];
var processedTrack = dest.stream.getAudioTracks()[0];
var torndown = false;
function cleanup() {
if (torndown) return;
torndown = true;
try {
node.port.postMessage('destroy');
} catch (e) {}
try {
source.disconnect();
node.disconnect();
} catch (e) {}
try {
origTrack.stop();
} catch (e) {}
try {
ctx.close();
} catch (e) {}
}
// When EC stops the track we handed it, release the raw capture + graph.
var rawStop = processedTrack.stop.bind(processedTrack);
processedTrack.stop = function () {
cleanup();
rawStop();
};
// Device unplugged / capture ended involuntarily.
origTrack.addEventListener('ended', function () {
try {
rawStop();
} catch (e) {}
cleanup();
});
// Return a stream with the processed audio plus any original video.
var out = new MediaStream();
out.addTrack(processedTrack);
stream.getVideoTracks().forEach(function (t) {
out.addTrack(t);
});
return out;
});
})
.catch(function (e) {
// Any failure -> fall back to the raw mic so calls never break.
// eslint-disable-next-line no-console
console.error('[lotus-denoise] RNNoise setup failed, using raw mic', e);
return stream;
});
}
navigator.mediaDevices.getUserMedia = function (constraints) {
var wantsAudio = !!(constraints && constraints.audio);
var effective = constraints;
if (wantsAudio) {
// RNNoise owns noise suppression; keep AEC + AGC on the raw capture.
var audioC = typeof constraints.audio === 'object' ? Object.assign({}, constraints.audio) : {};
audioC.noiseSuppression = false;
if (audioC.echoCancellation === undefined) audioC.echoCancellation = true;
if (audioC.autoGainControl === undefined) audioC.autoGainControl = true;
effective = Object.assign({}, constraints, { audio: audioC });
}
return origGetUserMedia(effective).then(function (stream) {
return wantsAudio ? processStream(stream) : stream;
});
};
})();
+7
View File
@@ -20,6 +20,7 @@
"@giphy/js-types": "5.1.0",
"@giphy/js-util": "5.2.0",
"@giphy/react-components": "10.1.2",
"@sapphi-red/web-noise-suppressor": "0.3.5",
"@sentry/react": "10.53.1",
"@tanstack/react-query": "5.100.13",
"@tanstack/react-query-devtools": "5.100.13",
@@ -3774,6 +3775,12 @@
"dev": true,
"license": "MIT"
},
"node_modules/@sapphi-red/web-noise-suppressor": {
"version": "0.3.5",
"resolved": "https://registry.npmjs.org/@sapphi-red/web-noise-suppressor/-/web-noise-suppressor-0.3.5.tgz",
"integrity": "sha512-jh3+V9yM+zxLriQexoGm0GatoPaJWjs6ypFIbFYwQp+AoUb55eUXrjKtKQyuC5zShzzeAQUl0M5JzqB7SSrsRA==",
"license": "MIT"
},
"node_modules/@sentry-internal/browser-utils": {
"version": "10.53.1",
"resolved": "https://registry.npmjs.org/@sentry-internal/browser-utils/-/browser-utils-10.53.1.tgz",
+1
View File
@@ -44,6 +44,7 @@
"@giphy/js-types": "5.1.0",
"@giphy/js-util": "5.2.0",
"@giphy/react-components": "10.1.2",
"@sapphi-red/web-noise-suppressor": "0.3.5",
"@sentry/react": "10.53.1",
"@tanstack/react-query": "5.100.13",
"@tanstack/react-query-devtools": "5.100.13",
+28 -23
View File
@@ -42,13 +42,11 @@ import {
DateFormat,
MessageLayout,
MessageSpacing,
NoiseSuppressionMode,
Settings,
settingsAtom,
} from '../../../state/settings';
import {
SeasonalPreview,
SeasonTheme,
} from '../../../components/seasonal/SeasonalEffect';
import { SeasonalPreview, SeasonTheme } from '../../../components/seasonal/SeasonalEffect';
import { SettingTile } from '../../../components/setting-tile';
import { KeySymbol } from '../../../utils/key-symbol';
import { isMacOS } from '../../../utils/user-agent';
@@ -1235,12 +1233,16 @@ function Calls() {
<SequenceCard className={SequenceCardStyle} variant="SurfaceVariant" direction="Column">
<SettingTile
title="Noise Suppression"
description="Apply AI noise suppression to filter background noise during calls (powered by Element Call)."
description="Filter background noise from your mic during calls. Browser-native uses the built-in WebRTC suppressor; ML runs on-device RNNoise for stronger, Krisp-style removal (higher CPU)."
after={
<Switch
variant="Primary"
<SettingsSelect<NoiseSuppressionMode>
value={callNoiseSuppression}
onChange={setCallNoiseSuppression}
options={[
{ value: 'off', label: 'Off' },
{ value: 'browser', label: 'Browser-native' },
{ value: 'ml', label: 'ML (beta)' },
]}
/>
}
/>
@@ -1346,22 +1348,25 @@ function Calls() {
);
}
const SEASONAL_OPTIONS: { value: Settings['seasonalThemeOverride']; label: string; emoji: string }[] =
[
{ value: 'auto', label: 'Auto', emoji: '🗓' },
{ value: 'off', label: 'Off', emoji: '×' },
{ value: 'newyear', label: 'New Year', emoji: '🎆' },
{ value: 'lunar', label: 'Lunar New Year', emoji: '🏮' },
{ value: 'valentines', label: "Valentine's", emoji: '💖' },
{ value: 'stpatricks', label: "St. Patrick's", emoji: '🍀' },
{ value: 'aprilfools', label: 'April Fools', emoji: '?' },
{ value: 'earthday', label: 'Earth Day', emoji: '🌱' },
{ value: 'autumn', label: 'Autumn', emoji: '🍂' },
{ value: 'halloween', label: 'Halloween', emoji: '🎃' },
{ value: 'christmas', label: 'Christmas', emoji: '❄️' },
{ value: 'arcade', label: 'Arcade Day', emoji: '👾' },
{ value: 'deepspace', label: 'Deep Space', emoji: '🚀' },
];
const SEASONAL_OPTIONS: {
value: Settings['seasonalThemeOverride'];
label: string;
emoji: string;
}[] = [
{ value: 'auto', label: 'Auto', emoji: '🗓' },
{ value: 'off', label: 'Off', emoji: '×' },
{ value: 'newyear', label: 'New Year', emoji: '🎆' },
{ value: 'lunar', label: 'Lunar New Year', emoji: '🏮' },
{ value: 'valentines', label: "Valentine's", emoji: '💖' },
{ value: 'stpatricks', label: "St. Patrick's", emoji: '🍀' },
{ value: 'aprilfools', label: 'April Fools', emoji: '?' },
{ value: 'earthday', label: 'Earth Day', emoji: '🌱' },
{ value: 'autumn', label: 'Autumn', emoji: '🍂' },
{ value: 'halloween', label: 'Halloween', emoji: '🎃' },
{ value: 'christmas', label: 'Christmas', emoji: '❄️' },
{ value: 'arcade', label: 'Arcade Day', emoji: '👾' },
{ value: 'deepspace', label: 'Deep Space', emoji: '🚀' },
];
function SeasonalBgGrid({
value,
+4 -4
View File
@@ -15,7 +15,7 @@ import { CallControlState } from '../plugins/call/CallControlState';
import { useCallMembersChange, useCallSession } from './useCall';
import { CallPreferences } from '../state/callPreferences';
import { useSetting } from '../state/hooks/settings';
import { settingsAtom } from '../state/settings';
import { NoiseSuppressionMode, settingsAtom } from '../state/settings';
import { unlockCallSounds } from '../utils/callSounds';
const CallEmbedContext = createContext<CallEmbed | undefined>(undefined);
@@ -45,7 +45,7 @@ export const createCallEmbed = (
themeKind: ElementCallThemeKind,
container: HTMLElement,
pref?: CallPreferences,
noiseSuppression = true,
denoiseMode: NoiseSuppressionMode = 'browser',
forceAudioOff = false,
): CallEmbed => {
const rtcSession = mx.matrixRTC.getRoomSession(room);
@@ -59,7 +59,7 @@ export const createCallEmbed = (
room,
intent,
themeKind,
noiseSuppression,
denoiseMode,
initialAudio,
initialVideo,
);
@@ -96,7 +96,7 @@ export const useCallStart = (dm = false) => {
theme.kind,
container,
pref,
callNoiseSuppression ?? true,
callNoiseSuppression ?? 'browser',
!!pttMode,
);
+12 -2
View File
@@ -18,6 +18,7 @@ import {
} from 'matrix-widget-api';
import { CallWidgetDriver } from './CallWidgetDriver';
import { trimTrailingSlash } from '../../utils/common';
import { NoiseSuppressionMode } from '../../state/settings';
import {
ElementCallIntent,
ElementCallThemeKind,
@@ -100,7 +101,7 @@ export class CallEmbed {
room: Room,
intent: ElementCallIntent,
themeKind: ElementCallThemeKind,
noiseSuppression = true,
denoiseMode: NoiseSuppressionMode = 'browser',
initialAudio = true,
initialVideo = false,
): Widget {
@@ -124,12 +125,21 @@ export class CallEmbed {
perParticipantE2EE: room.hasEncryptionStateEvent().toString(),
lang: 'en-EN',
theme: themeKind,
noiseSuppression: noiseSuppression.toString(),
// EC's built-in WebRTC suppressor: on only for 'browser' tier. For 'ml' we
// disable it here so RNNoise (the Lotus denoise shim) owns suppression and
// the two don't fight each other.
noiseSuppression: (denoiseMode === 'browser').toString(),
audio: initialAudio.toString(),
video: initialVideo.toString(),
header: 'none',
});
if (denoiseMode === 'ml') {
// Signal the Lotus denoise shim (injected into the EC index.html) to route
// the mic through the RNNoise worklet before LiveKit publishes the track.
params.append('lotusDenoise', 'ml');
}
if (CallEmbed.startingCall(intent)) {
params.append('sendNotificationType', CallEmbed.dmCall(intent) ? 'ring' : 'notification');
}
+15 -2
View File
@@ -9,6 +9,11 @@ export type DateFormat =
| 'YYYY-MM-DD'
| '';
export type MessageSpacing = '0' | '100' | '200' | '300' | '400' | '500';
// Call mic noise suppression tier:
// - 'off' : no suppression
// - 'browser' : WebRTC built-in suppression (Element Call noiseSuppression param)
// - 'ml' : client-side RNNoise ML suppression (Lotus denoise shim)
export type NoiseSuppressionMode = 'off' | 'browser' | 'ml';
export type ChatBackground =
| 'none'
| 'blueprint'
@@ -109,7 +114,7 @@ export interface Settings {
perMessageProfiles: boolean;
cameraOnJoin: boolean;
callNoiseSuppression: boolean;
callNoiseSuppression: NoiseSuppressionMode;
pttMode: boolean;
pttKey: string;
@@ -199,7 +204,7 @@ const defaultSettings: Settings = {
perMessageProfiles: false,
cameraOnJoin: false,
callNoiseSuppression: true,
callNoiseSuppression: 'browser',
pttMode: false,
pttKey: 'Space',
@@ -235,6 +240,14 @@ export const getSettings = (): Settings => {
return {
...defaultSettings,
...saved,
// Migrate legacy boolean callNoiseSuppression -> 3-way mode:
// true => browser-native, false => off. New string values pass through.
callNoiseSuppression:
typeof saved.callNoiseSuppression === 'boolean'
? saved.callNoiseSuppression
? 'browser'
: 'off'
: (saved.callNoiseSuppression ?? defaultSettings.callNoiseSuppression),
composerToolbarButtons: {
...DEFAULT_COMPOSER_TOOLBAR,
...(saved.composerToolbarButtons ?? {}),
+47
View File
@@ -65,6 +65,52 @@ function copyPdfWorker() {
};
}
// Lotus ML noise suppression: ship the RNNoise worklet/wasm + our getUserMedia
// shim alongside the vendored Element Call bundle, and inject the shim <script>
// into EC's index.html so it runs before EC captures the mic. Runs after
// viteStaticCopy has populated dist/public/element-call (closeBundle order).
function lotusDenoise() {
return {
name: 'lotus-denoise',
closeBundle() {
const ecDir = path.resolve('dist/public/element-call');
if (!fs.existsSync(ecDir)) return;
const denoiseDir = path.join(ecDir, 'denoise');
fs.mkdirSync(denoiseDir, { recursive: true });
const sapphi = path.resolve('node_modules/@sapphi-red/web-noise-suppressor/dist');
const assets = [
[
path.join(sapphi, 'rnnoise/workletProcessor.js'),
path.join(denoiseDir, 'rnnoiseWorklet.js'),
],
[path.join(sapphi, 'rnnoise.wasm'), path.join(denoiseDir, 'rnnoise.wasm')],
[path.join(sapphi, 'rnnoise_simd.wasm'), path.join(denoiseDir, 'rnnoise_simd.wasm')],
];
assets.forEach(([s, d]) => {
if (fs.existsSync(s)) fs.copyFileSync(s, d);
});
const shimSrc = path.resolve('build/lotus-denoise.js');
if (fs.existsSync(shimSrc)) fs.copyFileSync(shimSrc, path.join(ecDir, 'lotus-denoise.js'));
const indexPath = path.join(ecDir, 'index.html');
if (fs.existsSync(indexPath)) {
let html = fs.readFileSync(indexPath, 'utf8');
if (!html.includes('lotus-denoise.js')) {
// Classic (non-deferred) script runs before EC's deferred module entry.
html = html.replace(
/<script type="module"/,
'<script src="./lotus-denoise.js"></script><script type="module"',
);
fs.writeFileSync(indexPath, html);
}
}
},
};
}
function serverMatrixSdkCryptoWasm(wasmFilePath) {
return {
name: 'vite-plugin-serve-matrix-sdk-crypto-wasm',
@@ -133,6 +179,7 @@ export default defineConfig({
wasm(),
react(),
copyPdfWorker(),
lotusDenoise(),
...(process.env.SENTRY_AUTH_TOKEN
? [
sentryVitePlugin({