feat(calls): implement advanced multi-model ML noise suppression system
Implement a flexible, multi-model noise suppression pipeline for Element Call/LiveKit integration: - ML Engines: Added support for RNNoise, Speex, DTLN, and DeepFilterNet 3 models. - Pipeline Architecture: Implemented modular audio processing in lotus-denoise.js, supporting 'Series Suppression' (running browser-native NSNet2 before ML) and a hardware-style Noise Gate. - UI & UX Enhancements: - Settings UI: Added model comparison chart with CPU/Quality metadata. - Tuning: Added Live Microphone Meter for calibrating Noise Gate thresholds. - Reporting: Added LotusToast system to alert users when ML suppression fails or falls back to raw input. - Robustness & Quality: - Capture Fidelity: Removed forced 48kHz capture constraints to allow native-rate capture (solving static issues with high-end audio interfaces). - Performance: Added WASM SIMD detection with transparent fallback. - Capability Detection: Added browser feature detection to disable unsupported ML modes. - Build Integration: Updated Vite config to self-host all model WASM/tflite assets in /denoise/ directory.
This commit is contained in:
@@ -102,6 +102,10 @@ export class CallEmbed {
|
||||
intent: ElementCallIntent,
|
||||
themeKind: ElementCallThemeKind,
|
||||
denoiseMode: NoiseSuppressionMode = 'browser',
|
||||
denoiseModel: string = 'rnnoise',
|
||||
denoiseNativeNS: boolean = true,
|
||||
denoiseGate: boolean = false,
|
||||
denoiseGateThreshold: number = -45,
|
||||
initialAudio = true,
|
||||
initialVideo = false,
|
||||
): Widget {
|
||||
@@ -126,8 +130,8 @@ export class CallEmbed {
|
||||
lang: 'en-EN',
|
||||
theme: themeKind,
|
||||
// EC's built-in WebRTC suppressor: on only for 'browser' tier. For 'ml' we
|
||||
// disable it here so RNNoise (the Lotus denoise shim) owns suppression and
|
||||
// the two don't fight each other.
|
||||
// disable it here so EC doesn't do its own extra processing, and let the
|
||||
// Lotus denoise shim (which keeps native NS on) handle the pipeline.
|
||||
noiseSuppression: (denoiseMode === 'browser').toString(),
|
||||
audio: initialAudio.toString(),
|
||||
video: initialVideo.toString(),
|
||||
@@ -135,9 +139,12 @@ export class CallEmbed {
|
||||
});
|
||||
|
||||
if (denoiseMode === 'ml') {
|
||||
// Signal the Lotus denoise shim (injected into the EC index.html) to route
|
||||
// the mic through the RNNoise worklet before LiveKit publishes the track.
|
||||
// Signal the Lotus denoise shim to route the mic through the ML processors.
|
||||
params.append('lotusDenoise', 'ml');
|
||||
params.append('lotusModel', denoiseModel);
|
||||
params.append('lotusNativeNS', denoiseNativeNS.toString());
|
||||
params.append('lotusGate', denoiseGate.toString());
|
||||
params.append('lotusGateThreshold', denoiseGateThreshold.toString());
|
||||
}
|
||||
|
||||
if (CallEmbed.startingCall(intent)) {
|
||||
|
||||
Reference in New Issue
Block a user