LotusGuild/cinny
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(calls): 3-tier mic noise suppression with on-device ML (P5-30)
CI / Build & Quality Checks (push) Successful in 10m33s
Details
Trigger Desktop Build / trigger (push) Successful in 6s
Details

Browse Source 
Replace the boolean call noise-suppression setting with a 3-way control
(Off / Browser-native / ML beta) in Settings -> General -> Calls.

- Off: noiseSuppression=false to Element Call
- Browser-native: EC's built-in WebRTC suppressor (prior default)
- ML (beta): on-device RNNoise (@sapphi-red/web-noise-suppressor)

Element Call captures the mic inside its iframe and publishes to LiveKit,
so the host can't reach that track; LiveKit's Krisp filter is Cloud-only
(we self-host the SFU) and EC's own RNNoise PR #3892 is unmerged. The ML
tier instead injects a same-origin pre-init shim into the vendored EC
index.html (build/lotus-denoise.js, wired by the lotusDenoise vite plugin)
that patches getUserMedia and routes the captured mic through an RNNoise
AudioWorklet before LiveKit sees it -- the same post-capture pipeline as
#3892, with no EC fork/AGPL/rebase burden. Falls back to the raw mic if
setup fails; keeps echoCancellation/AGC on the raw capture.

- settings.ts: callNoiseSuppression -> 'off'|'browser'|'ml' + legacy
  boolean migration (true->browser, false->off)
- CallEmbed/useCallEmbed: tier maps to noiseSuppression param and appends
  lotusDenoise=ml (native suppressor off in ML mode)
- vite.config.js: copy RNNoise worklet/wasm + shim into the EC bundle and
  inject the shim <script> before EC's module entry
- docs: LOTUS_FEATURES.md, LOTUS_TODO.md (P5-30 done)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Jared Vititoe
2026-06-15 20:29:59 -04:00
parent f9edd2023d
commit 5deed79b42
10 changed files with 299 additions and 38 deletions
Download Patch File Download Diff File Expand all files Collapse all files
LOTUS_TODO.md
+5 -5
View File
@@ -243,12 +243,12 @@ Themes:
---
### [ ] P5-30 · Advanced ML Noise Suppression (Krisp-style)
### [x] P5-30 · Advanced ML Noise Suppression (Krisp-style)
**What:** High-end background noise cancellation using a pre-trained ML model (e.g. RNNoise) running in the browser. Removes dogs, fans, and keyboard clicks from the mic stream.
**Note:** This is a top-tier feature request and an EXTREME COMPLEXITY project.
**[AUDIT REQUIRED]** Must verify if mixing a processed stream into Element Call's WebRTC implementation causes latency or AEC (Echo Cancellation) issues.
**Complexity:** Extreme.
**What:** High-end background noise cancellation using a pre-trained ML model (RNNoise) running in the browser. Removes dogs, fans, and keyboard clicks from the mic stream.
**Shipped:** 3-tier setting (Off / Browser-native / ML beta) in Settings → General → Calls. ML tier injects a same-origin pre-init shim into the vendored Element Call `index.html` that monkeypatches `getUserMedia` and routes the captured mic through an RNNoise `AudioWorklet` (`@sapphi-red/web-noise-suppressor`) before LiveKit publishes — no EC fork required. See LOTUS_FEATURES.md → "Noise Suppression (3-Tier)".
**Key decision:** LiveKit's Krisp filter is LiveKit-Cloud-only (we self-host the SFU); EC's own RNNoise PR #3892 is unmerged. The shim is the same post-capture pipeline #3892 uses, executed from the realm we control, so it survives EC version bumps.
**AEC note (resolved-as-accepted):** WebAudio capture routing can weaken browser AEC (AEC runs on the native track) — same tradeoff as EC's upstream feature; mitigated by keeping `echoCancellation`/`autoGainControl` on the raw capture and labeling the tier "beta". Validate echo quality on real multi-party calls after deploy.
---