LotusGuild/cinny
3
0
Fork 0
Code Issues 27 Pull Requests 5 Actions Packages Projects Releases Wiki Activity

ML noise-suppression failures are never shown: io.lotus.denoise_state is unhandled, and the old lotus-denoise-status listener is dead and accepts any window #260

New Issue
Closed
opened 2026-10-04 17:41:25 -04:00 by jared · 1 comment
jared commented 2026-10-04 17:41:25 -04:00
Owner
Copy Link
Copy Source

What happens
When the fork's in-source ML denoiser fails to attach, the user is never told, and the Settings choice keeps claiming ML is on. On failure the fork either retries with rnnoise or gives up and publishes the raw mic.

Why

  • The new message has no listener. The fork reports the real state as a widget action, io.lotus.denoise_state { active, model, error? } (element-call src/lotus/lotusDenoise.ts:164-175). cinny never registers a listener for it: CallEmbed.start() only handles request_state, call_summary, mic_level, controls_state, hotkey and call_state. So ClientWidgetApi answers "unsupported action" and the report is dropped.
  • The listener that does exist is dead. The only failure listener left is LotusDenoiseFeature (src/app/pages/client/ClientNonUIFeatures.tsx:1017-1041). It waits for a raw postMessage of {type: 'lotus-denoise-status'}, which only the removed build-time getUserMedia shim used to send. Nothing sends it now.
  • And it isn't authenticated. That listener doesn't check event.source or event.origin. Any frame (a room widget, an embed) can post {type:'lotus-denoise-status', active:false, error:'<any text>'} and the app shows it as a system toast: "Audio Quality: ML Noise Suppression failed: …".
  • The same gap for audio output. io.lotus.audio_output_state (element-call src/lotus/lotusAudioOutput.ts:35-39) is also unhandled. The host's audio-output picker only knows what it last asked for, not what EC actually selected (for example after a device is unplugged).

Fix ideas

  • Add listenAction('io.lotus.denoise_state', …) in CallEmbed, and show a toast when active is false or model differs from the requested one.
  • Delete LotusDenoiseFeature.
  • Handle io.lotus.audio_output_state in CallControl so AudioOutputButton shows the real device.

More context

Introduced by 5d5f5f45 (2026-06-16, the old getUserMedia shim and its lotus-denoise-status postMessage). The shim was later replaced by the in-source processor, which reports through io.lotus.denoise_state instead, but the host side was never switched over.

Live evidence (local Synapse + LiveKit, dev server where the worklets 404):

[lotus] denoise setProcessor failed AbortError: Failed to load worklet module script: …/denoise/rnnoiseWorklet.js
[lotus] denoise unavailable; publishing raw mic …
[lotus] host did not ack io.lotus.denoise_state … Unknown or unsupported from-widget action: io.lotus.denoise_state

Nothing appears in the app: the user believes ML suppression is on.

On a production build, all four models attach (denoise processor active (rnnoise|speex|dtln|deepfilternet)). So this only bites when something actually fails, for example an unsupported device, a blocked worklet, a CSP change, or a slow CDN. Those are exactly the cases where the user needs to know.

Who: anyone who picked "ML" noise suppression, on web or desktop.

Acceptance criteria

  • io.lotus.denoise_state {active:false} shows a one-time toast ("ML noise suppression couldn't start — using your raw mic"), and the Settings / call bar reflect the real model (including the rnnoise fallback).
  • The lotus-denoise-status postMessage listener is removed.
  • io.lotus.audio_output_state updates AudioOutputButton.
  • A unit test asserts each io.lotus.* fromWidget action the fork sends has a host listener (a list kept next to LotusWidgetActions).

See also element-call#43 (fork protocol docs), #293 (dev-mode denoise).

Found in the October 2026 client audit (Playwright against a local Synapse + LiveKit stack, plus code reading). Code refs are at cinny 30e2dbcd, cinny-desktop 8f67ec7, element-call 3b0050e8.

**What happens** When the fork's in-source ML denoiser fails to attach, the user is never told, and the Settings choice keeps claiming ML is on. On failure the fork either retries with `rnnoise` or gives up and publishes the raw mic. **Why** - **The new message has no listener.** The fork reports the real state as a widget action, `io.lotus.denoise_state { active, model, error? }` (element-call `src/lotus/lotusDenoise.ts:164-175`). cinny never registers a listener for it: `CallEmbed.start()` only handles `request_state`, `call_summary`, `mic_level`, `controls_state`, `hotkey` and `call_state`. So `ClientWidgetApi` answers "unsupported action" and the report is dropped. - **The listener that does exist is dead.** The only failure listener left is `LotusDenoiseFeature` (src/app/pages/client/ClientNonUIFeatures.tsx:1017-1041). It waits for a raw `postMessage` of `{type: 'lotus-denoise-status'}`, which only the removed build-time `getUserMedia` shim used to send. Nothing sends it now. - **And it isn't authenticated.** That listener doesn't check `event.source` or `event.origin`. Any frame (a room widget, an embed) can post `{type:'lotus-denoise-status', active:false, error:'<any text>'}` and the app shows it as a system toast: "Audio Quality: ML Noise Suppression failed: <any text>…". - **The same gap for audio output.** `io.lotus.audio_output_state` (element-call `src/lotus/lotusAudioOutput.ts:35-39`) is also unhandled. The host's audio-output picker only knows what it last asked for, not what EC actually selected (for example after a device is unplugged). **Fix ideas** - Add `listenAction('io.lotus.denoise_state', …)` in `CallEmbed`, and show a toast when `active` is false or `model` differs from the requested one. - Delete `LotusDenoiseFeature`. - Handle `io.lotus.audio_output_state` in `CallControl` so `AudioOutputButton` shows the real device. ### More context **Introduced by** 5d5f5f45 (2026-06-16, the old `getUserMedia` shim and its `lotus-denoise-status` postMessage). The shim was later replaced by the in-source processor, which reports through `io.lotus.denoise_state` instead, but the host side was never switched over. **Live evidence (local Synapse + LiveKit, dev server where the worklets 404):** ``` [lotus] denoise setProcessor failed AbortError: Failed to load worklet module script: …/denoise/rnnoiseWorklet.js [lotus] denoise unavailable; publishing raw mic … [lotus] host did not ack io.lotus.denoise_state … Unknown or unsupported from-widget action: io.lotus.denoise_state ``` Nothing appears in the app: the user believes ML suppression is on. On a **production build**, all four models attach (`denoise processor active (rnnoise|speex|dtln|deepfilternet)`). So this only bites when something actually fails, for example an unsupported device, a blocked worklet, a CSP change, or a slow CDN. Those are exactly the cases where the user needs to know. **Who:** anyone who picked "ML" noise suppression, on web or desktop. **Acceptance criteria** - `io.lotus.denoise_state {active:false}` shows a one-time toast ("ML noise suppression couldn't start — using your raw mic"), and the Settings / call bar reflect the real model (including the rnnoise fallback). - The `lotus-denoise-status` `postMessage` listener is removed. - `io.lotus.audio_output_state` updates `AudioOutputButton`. - A unit test asserts each `io.lotus.*` fromWidget action the fork sends has a host listener (a list kept next to `LotusWidgetActions`). See also element-call#43 (fork protocol docs), #293 (dev-mode denoise). --- _Found in the October 2026 client audit (Playwright against a local Synapse + LiveKit stack, plus code reading). Code refs are at cinny `30e2dbcd`, cinny-desktop `8f67ec7`, element-call `3b0050e8`._
jared added the bugpriority: mediumarea: calls labels 2026-10-04 17:41:25 -04:00
jared commented 2026-10-19 14:00:47 -04:00
Author
Owner
Copy Link
Copy Source

Fixed by #413, now live: a toast appears when ML noise suppression fails or falls back (io.lotus.denoise_state is handled), and the old spoofable lotus-denoise-status postMessage listener is removed. Closing.

Fixed by #413, now live: a toast appears when ML noise suppression fails or falls back (`io.lotus.denoise_state` is handled), and the old spoofable `lotus-denoise-status` postMessage listener is removed. Closing.
jared closed this issue 2026-10-19 14:00:47 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
a11y
Accessibility: keyboard, screen reader, contrast, motion
 area: appearance
Client area: appearance
 area: auth-session
Client area: auth-session
 area: build-ci
Client area: build-ci
 area: calls
Client area: calls
 area: desktop
Client area: desktop
 area: media
Client area: media
 area: messaging
Client area: messaging
 area: mobile
Client area: mobile
 area: moderation
Client area: moderation
 area: navigation
Client area: navigation
 area: notifications
Client area: notifications
 area: settings
Client area: settings
 area: threads
Client area: threads
 bug
Something is not working
 dependencies
Third-party package versions and advisories
 docs
README / LOTUS_* docs wrong or missing
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 needs-human-review
performance
Re-render storms, leaks, heavy work on hot paths
 planning
priority: critical
Data loss, security hole, or crash on a main path
 priority: high
Broken feature or serious usability problem
 priority: low
Minor issue or polish
 priority: medium
Wrong behaviour in an edge case or notable degradation
 qa
Manual QA: shipped, needs a human in a real environment
 question
More information is needed
 research
security
XSS, unsafe URLs, data leaks, auth/session
 tech-debt
Code health, dead code, fragile patterns
 ux
Usability or visual inconsistency
 wontfix
This won't be fixed
No labels area: calls bug priority: medium
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
cole (Cole Keller) jared (Jared Vititoe) nathan (Nathan Vititoe)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: LotusGuild/cinny#260