feat(calls): implement advanced multi-model ML noise suppression system
Implement a flexible, multi-model noise suppression pipeline for Element Call/LiveKit integration: - ML Engines: Added support for RNNoise, Speex, DTLN, and DeepFilterNet 3 models. - Pipeline Architecture: Implemented modular audio processing in lotus-denoise.js, supporting 'Series Suppression' (running browser-native NSNet2 before ML) and a hardware-style Noise Gate. - UI & UX Enhancements: - Settings UI: Added model comparison chart with CPU/Quality metadata. - Tuning: Added Live Microphone Meter for calibrating Noise Gate thresholds. - Reporting: Added LotusToast system to alert users when ML suppression fails or falls back to raw input. - Robustness & Quality: - Capture Fidelity: Removed forced 48kHz capture constraints to allow native-rate capture (solving static issues with high-end audio interfaces). - Performance: Added WASM SIMD detection with transparent fallback. - Capability Detection: Added browser feature detection to disable unsupported ML modes. - Build Integration: Updated Vite config to self-host all model WASM/tflite assets in /denoise/ directory.
This commit is contained in:
+32
-1
@@ -87,9 +87,40 @@ function lotusDenoise() {
|
||||
],
|
||||
[path.join(sapphi, 'rnnoise.wasm'), path.join(denoiseDir, 'rnnoise.wasm')],
|
||||
[path.join(sapphi, 'rnnoise_simd.wasm'), path.join(denoiseDir, 'rnnoise_simd.wasm')],
|
||||
[
|
||||
path.join(sapphi, 'speex/workletProcessor.js'),
|
||||
path.join(denoiseDir, 'speexWorklet.js'),
|
||||
],
|
||||
[path.join(sapphi, 'speex.wasm'), path.join(denoiseDir, 'speex.wasm')],
|
||||
[
|
||||
path.join(sapphi, 'noiseGate/workletProcessor.js'),
|
||||
path.join(denoiseDir, 'noiseGateWorklet.js'),
|
||||
],
|
||||
// DTLN (WorkAdventure LiteRT implementation)
|
||||
[
|
||||
path.resolve('node_modules/@workadventure/noise-suppression/dist/noise-suppression-processor.js'),
|
||||
path.join(denoiseDir, 'dtlnWorklet.js'),
|
||||
],
|
||||
[
|
||||
path.resolve('node_modules/@workadventure/noise-suppression/dist/litert_wasm_internal.wasm'),
|
||||
path.join(denoiseDir, 'litert_wasm_internal.wasm'),
|
||||
],
|
||||
[
|
||||
path.resolve('node_modules/@workadventure/noise-suppression/dist/model_1.tflite'),
|
||||
path.join(denoiseDir, 'model_1.tflite'),
|
||||
],
|
||||
[
|
||||
path.resolve('node_modules/@workadventure/noise-suppression/dist/model_2.tflite'),
|
||||
path.join(denoiseDir, 'model_2.tflite'),
|
||||
],
|
||||
];
|
||||
assets.forEach(([s, d]) => {
|
||||
if (fs.existsSync(s)) fs.copyFileSync(s, d);
|
||||
if (fs.existsSync(s)) {
|
||||
fs.copyFileSync(s, d);
|
||||
} else {
|
||||
// eslint-disable-next-line no-console
|
||||
console.warn(`[lotus-denoise] Asset missing, will be populated by CI: ${s}`);
|
||||
}
|
||||
});
|
||||
|
||||
const shimSrc = path.resolve('build/lotus-denoise.js');
|
||||
|
||||
Reference in New Issue
Block a user