feat(denoise): switch noise suppression mid-call (+ fix re-captures ignoring noiseSuppression) #66

2026-10-30T21:13:52-04:00

jared commented

2026-10-30 21:13:52 -04:00

Users can now change noise suppression during a call instead of rejoining: off, browser-native or ML, plus the ML model, the gate and "stack browser NS". Lotus (cinny) sends the new toWidget action io.lotus.set_denoise.

How it works

Model swap: the new model is preloaded (prepareDenoiseAssets) while the old one keeps running, then the processor is swapped. The mic is not re-captured.
Browser NS on/off: the mic is re-captured with restartTrack and the new constraint. The deviceId is passed explicitly, because without it LiveKit requests audio: true and drops every constraint. The choice is stored in lotusCaptureOverride, which ConnectionFactory reads, so an SFU reconnect captures the same way.
Ordering: switches are serialized, and a burst of changes collapses into the last one. Each switch reports io.lotus.denoise_state, which now includes mode.

Bugs found while testing in Chromium

Re-captures silently kept the old noiseSuppression. The re-capture asked for false and got true. Cause: the mic level meter (and muted-speech nudge) clones the original mic track and never noticed restartTrack, and Chrome makes all open tracks of a device share one audio-processing config. The meter now releases its clone around our re-capture and re-taps on TrackEvent.Restarted. That also fixes it listening to the old device after a mic switch (and keeping that device open).
voiceIsolation stayed on in "off" and ML modes. LiveKit defaults it on, which adds another voice filter in front of the model on platforms that implement it. It now follows browser noise suppression.

Verification

Unit tests: 155/155 lotus tests pass. New cases cover model swap, ML → browser, browser → ML, invalid payload, payload merging, and the reconnect constraint override. tsc and oxlint are clean.
Two-browser call, joining once as browser and once as ML (DTLN), then switching DTLN → DeepFilterNet → RNNoise → browser → off. Each step applied within about 0.5 s. A getUserMedia spy confirmed every re-capture got exactly what it asked for:
```
join ML     asked ns=false vi=false  got ns=false
→ browser   asked ns=true  vi=true   got ns=true
→ off       asked ns=false vi=false  got ns=false
```

🤖 Generated with Claude Code

https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5

Users can now change noise suppression during a call instead of rejoining: off, browser-native or ML, plus the ML model, the gate and "stack browser NS". Lotus (cinny) sends the new toWidget action `io.lotus.set_denoise`. ## How it works - **Model swap:** the new model is preloaded (`prepareDenoiseAssets`) while the old one keeps running, then the processor is swapped. The mic is not re-captured. - **Browser NS on/off:** the mic is re-captured with `restartTrack` and the new constraint. The `deviceId` is passed explicitly, because without it LiveKit requests `audio: true` and drops every constraint. The choice is stored in `lotusCaptureOverride`, which `ConnectionFactory` reads, so an SFU reconnect captures the same way. - **Ordering:** switches are serialized, and a burst of changes collapses into the last one. Each switch reports `io.lotus.denoise_state`, which now includes `mode`. ## Bugs found while testing in Chromium 1. **Re-captures silently kept the old `noiseSuppression`.** The re-capture asked for `false` and got `true`. Cause: the mic level meter (and muted-speech nudge) clones the original mic track and never noticed `restartTrack`, and Chrome makes all open tracks of a device share one audio-processing config. The meter now releases its clone around our re-capture and re-taps on `TrackEvent.Restarted`. That also fixes it **listening to the old device after a mic switch** (and keeping that device open). 2. **`voiceIsolation` stayed on in "off" and ML modes.** LiveKit defaults it on, which adds another voice filter in front of the model on platforms that implement it. It now follows browser noise suppression. ## Verification - Unit tests: 155/155 lotus tests pass. New cases cover model swap, ML → browser, browser → ML, invalid payload, payload merging, and the reconnect constraint override. `tsc` and oxlint are clean. - Two-browser call, joining once as browser and once as ML (DTLN), then switching DTLN → DeepFilterNet → RNNoise → browser → off. Each step applied within about 0.5 s. A `getUserMedia` spy confirmed every re-capture got exactly what it asked for: ``` join ML asked ns=false vi=false got ns=false → browser asked ns=true vi=true got ns=true → off asked ns=false vi=false got ns=false ``` 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5

jared added 1 commit 2026-10-30 21:13:53 -04:00

feat(denoise): switch noise suppression mid-call (io.lotus.set_denoise)

CI / Build embedded bundle (push) Skipped

CI / Build embedded bundle (pull_request) Successful in 1m47s

Details

CI / Publish to Gitea npm registry (pull_request) Skipped

Details

949103d649

The host can now change noise suppression on the live mic without a
rejoin: off / browser-native / ML, the ML model, the gate and stacking
browser NS in front of the model.

- Model swaps preload the new model while the old one keeps running,
  then swap the processor (no mic re-capture).
- Browser NS on/off re-captures the mic via restartTrack with the new
  constraint (passing the deviceId, without which LiveKit drops every
  constraint). The choice is kept in lotusCaptureOverride, which
  ConnectionFactory reads, so a later SFU reconnect captures the same way.
- Switches are serialized; a burst collapses into the last one. Each one
  reports io.lotus.denoise_state (now with `mode`).

Fixes found while testing in Chromium:
- The mic level meter (and muted-speech nudge) kept a clone of the
  ORIGINAL mic track and never noticed LiveKit's restartTrack. Chrome
  makes all open tracks of a device share audio processing, so every
  re-capture silently kept the old noiseSuppression (asked false, got
  true). The meter now drops its clone around our re-capture and
  re-taps on TrackEvent.Restarted, which also fixes it listening to the
  old device after a mic switch.
- LiveKit defaults voiceIsolation ON, a further voice filter in "off"
  and ML modes where the platform implements it. It now follows browser
  noise suppression.

Verified in a two-browser call: joined as browser and as ML(dtln), then
switched dtln -> deepfilternet -> rnnoise -> browser -> off; each step
applied within ~0.5 s and every re-capture got the constraints it asked
for.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5

jared scheduled this pull request to auto merge when all checks succeed 2026-10-30 21:14:02 -04:00

jared merged commit 576c80a989 into lotus

2026-10-30 21:15:44 -04:00

jared deleted branch feat/live-denoise-switch

2026-10-30 21:15:44 -04:00

jared referenced this pull request from LotusGuild/cinny

2026-10-30 21:20:12 -04:00

feat(calls): change noise suppression mid-call without rejoining #485

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: LotusGuild/element-call#66