feat(calls): implement advanced multi-model ML noise suppression system
CI / Build & Quality Checks (push) Failing after 4m49s
Trigger Desktop Build / trigger (push) Successful in 11s

Implement a flexible, multi-model noise suppression pipeline for Element Call/LiveKit integration:

- ML Engines: Added support for RNNoise, Speex, DTLN, and DeepFilterNet 3 models.
- Pipeline Architecture: Implemented modular audio processing in lotus-denoise.js, supporting 'Series Suppression' (running browser-native NSNet2 before ML) and a hardware-style Noise Gate.
- UI & UX Enhancements:
  - Settings UI: Added model comparison chart with CPU/Quality metadata.
  - Tuning: Added Live Microphone Meter for calibrating Noise Gate thresholds.
  - Reporting: Added LotusToast system to alert users when ML suppression fails or falls back to raw input.
- Robustness & Quality:
  - Capture Fidelity: Removed forced 48kHz capture constraints to allow native-rate capture (solving static issues with high-end audio interfaces).
  - Performance: Added WASM SIMD detection with transparent fallback.
  - Capability Detection: Added browser feature detection to disable unsupported ML modes.
- Build Integration: Updated Vite config to self-host all model WASM/tflite assets in /denoise/ directory.
This commit is contained in:
2026-06-16 00:50:12 -04:00
parent 938ead79f7
commit 5d5f5f4516
10 changed files with 606 additions and 105 deletions
+32 -1
View File
@@ -87,9 +87,40 @@ function lotusDenoise() {
],
[path.join(sapphi, 'rnnoise.wasm'), path.join(denoiseDir, 'rnnoise.wasm')],
[path.join(sapphi, 'rnnoise_simd.wasm'), path.join(denoiseDir, 'rnnoise_simd.wasm')],
[
path.join(sapphi, 'speex/workletProcessor.js'),
path.join(denoiseDir, 'speexWorklet.js'),
],
[path.join(sapphi, 'speex.wasm'), path.join(denoiseDir, 'speex.wasm')],
[
path.join(sapphi, 'noiseGate/workletProcessor.js'),
path.join(denoiseDir, 'noiseGateWorklet.js'),
],
// DTLN (WorkAdventure LiteRT implementation)
[
path.resolve('node_modules/@workadventure/noise-suppression/dist/noise-suppression-processor.js'),
path.join(denoiseDir, 'dtlnWorklet.js'),
],
[
path.resolve('node_modules/@workadventure/noise-suppression/dist/litert_wasm_internal.wasm'),
path.join(denoiseDir, 'litert_wasm_internal.wasm'),
],
[
path.resolve('node_modules/@workadventure/noise-suppression/dist/model_1.tflite'),
path.join(denoiseDir, 'model_1.tflite'),
],
[
path.resolve('node_modules/@workadventure/noise-suppression/dist/model_2.tflite'),
path.join(denoiseDir, 'model_2.tflite'),
],
];
assets.forEach(([s, d]) => {
if (fs.existsSync(s)) fs.copyFileSync(s, d);
if (fs.existsSync(s)) {
fs.copyFileSync(s, d);
} else {
// eslint-disable-next-line no-console
console.warn(`[lotus-denoise] Asset missing, will be populated by CI: ${s}`);
}
});
const shimSrc = path.resolve('build/lotus-denoise.js');