Review found native dynamic import() of the DTLN/DeepFilterNet ESM
resolves "./denoise/…" against the bundled JS chunk's URL (-> /assets/…)
not the document, so those two models 404'd and silently fell back to raw
mic in the default config. Resolve the asset base to an absolute
same-origin href against the document; addModule()/fetch() accept absolute
too, so all three load paths stay consistent. (rnnoise/speex were
unaffected since addModule resolves against the document.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Faithful port of cinny's proven pipeline into the TrackProcessor, closing
protocol gap F3 (host offers rnnoise/speex/dtln/deepfilternet; only the
first two existed in-source).
- Fix real bug: gate worklet registers as "noise-gate" (hyphenated), not
"noiseGate" — the gated path would have failed to construct the node.
- Per-model sample rate: DTLN runs at 16kHz, others 48kHz (worklets don't
resample); verify the context actually got the rate, else fall back.
- resume() a suspended context (host postMessage isn't a gesture).
- DTLN via dynamic-imported @workadventure helper (bypassUntilReady);
DeepFilterNet via dynamic-imported ESM + DeepFilterNet3Core pointed at
the self-hosted base. Same-origin base (kept from the C1 fix) makes these
dynamic imports safe.
- Prefer SIMD rnnoise.wasm with non-SIMD fallback; cache wasm per URL.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Denoise deep-review (CRITICAL): restart() read opts.audioContext, which
LiveKit does NOT pass on restart — so reconnect (the A7 scenario) and mic
device-switch threw after stopping the old track, leaving the mic SILENT
(A7 reintroduced). Fix:
- Processor owns a dedicated 48kHz AudioContext (sapphi worklets require
48kHz; H1), reused across restart, closed on destroy.
- restart() never throws and never leaves a stopped track on the sender:
builds the new graph first, then disposes the old; on failure degrades
to RAW mic audio rather than silence.
- Cache wasm per URL (no re-fetch each reconnect); gate threshold default
-45 and accept an explicit 0 (M2); document the cross-repo asset contract.
Protocol audit:
- Non-silent warning when an unsupported denoise model (dtln/deepfilternet)
is requested instead of silent rnnoise fallback (F3).
- Correct the call_state enum comment (immediate error-reply, not 10s) (F2).
Build/CI audit:
- Stamp VITE_APP_VERSION in CI; document the vX.Y.Z-lotus.N version scheme.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Holistic security audit findings:
- C1 (CRITICAL): force lotusDenoiseBase to same-origin before it reaches
audioWorklet.addModule()/fetch — a crafted call-link param could
otherwise load attacker JS/WASM as a worklet processing the live mic.
Non-same-origin/malformed values fall back to bundled ./denoise/.
- H1 (HIGH): gate audio-inject behind explicit lotusAudioInject=1 (still
acks the action so no transport hang) — it publishes under the local
user's identity, so it must not be silently armed for every call.
- M1 (MED): cap the decoration roster at 512 entries.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Implements RNNoise/Speex noise suppression as a LiveKit audio
TrackProcessor attached to the local mic track, replacing the host's
build-time getUserMedia monkeypatch. Because EC re-attaches the processor
on every (re)publish (LocalTrackPublished), denoise now survives EC's
mid-call reconnect — the root cause of A7 "mic dead after reconnect".
Reuses the worklet/wasm assets already shipped under ./denoise/ (no new EC
dependency); model/gate configured via lotusDenoise/lotusModel/lotusGate
URL params. Additive: no-op unless lotusDenoise=ml.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Review found in-call tiles use MediaView->Avatar, not TileAvatar, so the
decoration never rendered in-call (CRITICAL). Move the overlay into
MediaView, gated on the avatar's own visibility (!(video && videoEnabled))
so it never floats over live video; revert the TileAvatar changes.
Also ref-count the io.lotus.decorations registration (one shared handler,
no double-reply) and stop clearing the map on teardown so a transient
remount doesn't drop decorations (HIGH/MED).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds io.lotus.decorations (toWidget): the host pushes a userId->image-URL
map and EC overlays the profile decoration on each tile avatar
(TileAvatar), keyed by userId, with a useSyncExternalStore-backed store.
Makes A6 first-class in-call instead of absent. URLs are validated
https/blob. Additive: no-op unless the host sends decorations.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Apply the encoding patch to ALL simulcast layers, not just encodings[0]:
screenshare publishes simulcast (VP8), so the full-res layer (the real
bandwidth hog) was left uncapped (CRITICAL).
- Re-apply on TrackUnmuted/restart + a 500ms settle, since LiveKit's
refreshSenderEncodings() overwrites our caps on replaceTrack (device/
source switch, processor toggle) without firing LocalTrackPublished (HIGH).
- Clamp values to sane ranges so a typo can't brick the encoder (MED).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds body.lotus-transparent (lotusTransparent=1) so the host wallpaper
shows through the call natively, retiring cinny's injected
`html,body{background:none!important}` hack; and body.lotus-theme
(lotusTheme=1) as a source-level Compound-token override hook for the
Lotus/TDS palette (driven by the existing setTheme channel / URL flags).
Additive: no-op without the flags.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- resume() the AudioContext (host postMessage isn't a gesture) so the clip
isn't silent; warn if it stays suspended (HIGH).
- Close the AudioContext on decode failure (no context leak) (MED).
- Abort in-flight clips on teardown (unmount/vm-change/leave) so audio
doesn't keep blasting to peers (MED).
- Stop the cloned MediaStreamTrack when a room publish fails (MED).
- Validate url is https/blob and fetch with credentials:omit, mode:cors
(MED security).
- Guard against NaN clip duration; fix stale enum doc comment.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds io.lotus.set_quality (toWidget): caps mic audio bitrate and
screenshare bitrate/framerate via RTCRtpSender.setParameters (no
republish). Settings are sticky and re-applied on LocalTrackPublished so
they survive mute/unmute and reconnects. These encoding controls lived in
EC's module scope, unreachable from the host against the prebuilt bundle.
Additive: no-op unless the host sends the action.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Review found the manual pin only chose among cameras when no screenshare
was active; during a screenshare the screenshare won the spotlight and the
pinned camera was demoted to an ignored PiP. Apply the override at the
spotlightAndPip$ level: when a pin is explicitly set, surface that camera
in the spotlight alongside the shared screen. No manual pin = unchanged.
Note: a pin persists if the pinned user briefly leaves and rejoins; the
host clears it via focus_participant{userId:null} (by design).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds io.lotus.inject_audio (toWidget): mixes a soundboard clip into the
call so other participants hear it. Publishes the clip as a separate
Unknown-source LiveKit track (rendered by MatrixAudioRenderer) rather than
splicing into the mic track, so the denoise pipeline is untouched; the
track is unpublished when the clip ends (with a 30s safety cap). This is
the real call-audio injection that was impossible against the prebuilt EC
bundle. Additive: no-op unless the host sends the action.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Document that io.lotus.call_state is request/response and the host must
ack it (cinny listenAction replies {}) to avoid 10s-timeout churn (H1).
- Throttle 150ms -> 250ms to reduce widget traffic (M1).
- lotusParam: hash fragment wins over query, matching EC's ParamParser (L1).
- Fix the misleading "opaque" id comment; id is userId:deviceId (L2).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds io.lotus.focus_participant (toWidget): the host can pin a participant
to the spotlight by Matrix user id (or clear with userId:null), via a
manual override injected into CallViewModel.spotlightSpeaker$. Replaces
cinny's fragile DOM .click() tile-selector focus hack. Extracts the action
enum into lotusActions.ts (no circular import) and allow-lists Lotus
toWidget actions in initializeWidget. Additive: no-op unless the host
sends the action.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Opt-in (lotusCallState=1) bridge that emits io.lotus.call_state with each
participant's speaking/audio/video state, so the Lotus host can drive
speaking rings / mute badges / PiP from real events instead of scraping
EC's rendered DOM. Exposes vm.userMedia$ on the public CallViewModel.
Additive: no-op without the flag.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Lotus CIandClaude Opus 4.8
Valere FedronicandValere
RobinandGitHub
Timo K
TimoandGitHub
Johannes Marbach