The host can now change noise suppression on the live mic without a
rejoin: off / browser-native / ML, the ML model, the gate and stacking
browser NS in front of the model.
- Model swaps preload the new model while the old one keeps running,
then swap the processor (no mic re-capture).
- Browser NS on/off re-captures the mic via restartTrack with the new
constraint (passing the deviceId, without which LiveKit drops every
constraint). The choice is kept in lotusCaptureOverride, which
ConnectionFactory reads, so a later SFU reconnect captures the same way.
- Switches are serialized; a burst collapses into the last one. Each one
reports io.lotus.denoise_state (now with `mode`).
Fixes found while testing in Chromium:
- The mic level meter (and muted-speech nudge) kept a clone of the
ORIGINAL mic track and never noticed LiveKit's restartTrack. Chrome
makes all open tracks of a device share audio processing, so every
re-capture silently kept the old noiseSuppression (asked false, got
true). The meter now drops its clone around our re-capture and
re-taps on TrackEvent.Restarted, which also fixes it listening to the
old device after a mic switch.
- LiveKit defaults voiceIsolation ON, a further voice filter in "off"
and ML modes where the platform implements it. It now follows browser
noise suppression.
Verified in a two-browser call: joined as browser and as ML(dtln), then
switched dtln -> deepfilternet -> rnnoise -> browser -> off; each step
applied within ~0.5 s and every re-capture got the constraints it asked
for.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5
At 60 dB the background is muted outright (measured -61 to -64 dB), so
each word drops into dead silence and listeners hear the mic cutting out
at word ends. On room-echo speech (RT60 ~0.4 s) with fan and keyboard
noise at SNR 15 dB, 25 dB scored best:
limit PESQ clean/fan/keyboard noise removed
60 3.35 / 2.33 / 2.04 ~62 dB
40 3.41 / 2.45 / 2.21 32-40 dB
25 3.59 / 2.72 / 2.38 25 dB
20 3.63 / 2.66 / 2.25 20 dB
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5
If the DeepFilterNet model is served with `Content-Encoding: gzip` (vite
preview does this for .tar.gz; some CDNs and nginx gzip_static setups do
too), fetch() hands back the already-unpacked tar. The library's worklet
then fails inside df_create, swallows the error and passes the raw mic
through, while Lotus Call reports DeepFilterNet as active.
Check the bytes after initialize(): a gzip model is used as-is, an
unpacked tar is re-gzipped in the browser, and anything else throws so
the rnnoise fallback runs and the host is told the truth.
Measured in a two-browser call (fan noise, SNR 15 dB): before, 4.9 dB of
noise removed, i.e. none beyond gain differences; after, 30.9 dB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5
The npm advisory database flagged 19 packages in pnpm-lock.yaml. None of them
ship in the embedded call bundle (checked against the built dist); they run
on developer machines and in CI:
- storybook / @storybook/* 10.6.0 -> 10.6.3: critical, dev server code
injection from the local network.
- vitest 4.1.12 (+ @vitest/ui unpinned to ^4.1.12), tinypool ^2.1.2:
mock registration code injection, prototype-pollution RCE.
- vite ^8.3.5: dev-server file exposure, launch-editor issues.
- Overrides for transitive tooling: brace-expansion, browserslist,
smol-toml, source-map-js, postcss-selector-parser, baseline-browser-
mapping, fflate, pbkdf2, zod 4; existing qs/js-yaml/undici/dompurify
overrides raised to the fixed versions.
Left: elliptic (no fixed release, low), zod 3 (no 3.x fix; a dependency
pins it). Lotus unit tests 144/144, build:embedded OK.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5
EC remembers the picked mic / output by id and silently falls back to the
first device when it is missing, so a user who always joins with a headset
lands on the laptop mic. With lotusDeviceCheck=1:
- the last-picked device's NAME is remembered next to its id
(lotus-audio-{input,output}-name), since ids can change on re-plug;
- before the automatic join, a usual device that is gone is reported to the
host (io.lotus.device_missing) and EC waits for io.lotus.device_decision
{ join }; one that came back under a new id is just re-selected;
- in a call, losing it is reported (io.lotus.device_lost) for a toast.
Fails open (no names / no host / nothing remembered). Without the flag the
join stays synchronous, as upstream. The gate runs once per call: CallView's
effect re-runs while its inputs settle and must not restart it.
Embedded version 0.28.0-lotus.2.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5
Syncs the fork from upstream v0.25.0 to v0.28.0 (196 upstream commits;
v0.27 was skipped upstream), bringing matrix-js-sdk develop up to the
SDK 44 MatrixRTC changes (livekit_alias / oldest_membership removed),
delayed-leave delegation to the SFU, and the HostBridge host interface.
Lotus changes carried across the refactors:
- widget: upstream no longer exports a widget singleton (initializer hands
initializeWidget()'s result to React). The Lotus host integration
(src/lotus/*) reads `widget` at call time, so a Lotus-only exported
reference is set by initializeWidget.
- useTheme: theme classes now go on the root element (the body in the
widget build); the lotusTheme / lotusTransparent / lotusHostControls
classes follow it.
- LocalMember: [lotus #46] keep retrying network errors until the delayed
leave would fire (Math.max), on upstream's restructured timings.
- UrlConfiguration moved to its own file; autoGainControl moved with it.
- errors: upstream switched to i18n.t; SFUTokenRefusedError follows.
- keyboard shortcuts: Lotus host hotkeys still win over focusIsClaimed().
- GroupCallErrorBoundary: the reconnect handler uses the widget reference
(upstream dropped the widget prop).
Config: v0.26 renamed matrix_rtc_session.delayed_leave_event_* to
delayed_leave.{delay_ms,restart_ms,restart_timeout_ms} with no fallback,
and the embedded build no longer emits a config.json. The production
config (public/element-call/config.json on LXC 106, not in git) must move
to the new keys when this ships, or calls lose the 60 s grace (lotus #46).
Embedded version 0.28.0-lotus.1.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5
io.lotus.call_summary gains an optional `details`: the other Matrix users
seen and the peak in the call; how long your connection, each other
person's, and everyone at once (likely the server) was poor/lost; RTT
median/p95, outbound and inbound packet loss and jitter (sampled every 5 s
from LiveKit sender/receiver stats); and your talk, mute and screenshare
time. People already present when you connect are recorded on Connected
(LiveKit sends no ParticipantConnected for them), and joins are idempotent
per device. Older hosts ignore the field.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5
Clips publish as a separate Unknown-source track, which tokens limited to
listed sources (any room with io.lotus.room_quality) refuse, so #52 could
only answer not_permitted. Where the token allows the microphone, the
clip is now mixed into the published mic for its duration: the track on
the mic's RTCRtpSender (post-denoise) and the clip feed a
MediaStreamAudioDestinationNode, the sender switches to the mix with
replaceTrack, and the original goes back when the clip ends. Muting
mid-clip stops it; a track LiveKit swapped in meanwhile is left alone.
not_permitted remains for tokens without the microphone.
Fixes#47
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5
Upstream drops the local PiP in the one-on-one mobile layout whenever the
local camera is off. In Lotus voice rooms that is nearly always, so on a
phone you never saw yourself or your own speaking ring. Show it as the
desktop layout does; the tile falls back to your avatar. Tests updated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5
New fromWidget action io.lotus.screenshare_audio_state { present }: each
LivekitRoomAudioRenderer reports whether a remote participant's
ScreenShareAudio track is being received; the combined answer is sent to
the host on change, and cleared when a renderer unmounts. cinny uses it to
show "Mute Screenshare Audio" only when there is something to mute.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5
maximumNetworkErrorRetryCount was sized to min(sync grace period,
leave delay), i.e. 10 s, so the membership manager gave up 20-35 s into
a homeserver stall and showed "Connection lost", even when the server
would keep the membership for 60 s (prod config). Size it to the leave
delay instead: the embedded call has no /sync loop for the grace period
to describe, and the server's delayed leave is what actually ends the
membership.
Fixes#46
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5
Everything the host relies on was documented only in src/lotus comments.
Add a table of the 22 io.lotus.* actions (generated from lotusActions.ts,
with direction, payload and whether cinny handles each), the URL flags,
fork-only behaviour and the release/publish steps.
Fixes#43
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5
Clips publish as an Unknown-source track. In audio-only rooms the guard
signs a microphone-only token, LiveKit refuses the publish, and since
the fork had already replied {} the host never heard. Check the grants
up front and reply { played: false, reason: "not_permitted" }.
Refs #47
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5
observeLocalMicSample$ switchMapped on every allConnections$ emission,
which re-emits far more often than the set of rooms changes. Each one
stopped the clone, closed the AudioContext and emitted null, and the
null reset the muted-speech gate, so the "You're muted" nudge could
never stay true long enough to fire. Map to the room list and only
restart when it actually changes.
Fixes#45
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5
After a homeserver stall the frame shows "Connection lost · Reconnect"
while the host kept showing the call as joined, with controls that did
nothing. While the error page is up, report io.lotus.call_error
{ code, recoverable } (and { code: null } when it clears), and accept
io.lotus.reconnect to run the page's own Reconnect / Try again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5
Browsers hash deviceIds per frame/origin, so ids from the host's own
enumerateDevices() are unknown in here and select() fell back to the
default. io.lotus.audio_output_state now also carries
available: [{id,label}] (frame-valid ids), so the host's call-bar menu
can offer real choices.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5
The widget intent presets default sendNotificationType to "notification",
and the URL param could only switch it to ring/notification, so the host
had no way to say "don't notify". In persistent voice channels and space
rooms that notification carries an @room mention: every member got a
highlighted ping (and phone push) whenever a moderator joined first, and
other clients showed it as a missed call.
`sendNotificationType=none` now clears it after the presets are applied.
Unit test added.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5
jared
Lotus CIandClaude Opus 5.5
renovate[bot]
Johannes Marbach