Compare commits

..
Author SHA1 Message Date
Lotus CIandClaude Opus 5.5 35bc7a1e98 chore(lotus): 0.25.0-lotus.16
CI / Build embedded bundle (push) Successful in 2m4s
CI / Publish to Gitea npm registry (push) Successful in 56s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPmy3tPq869XDW4njjVaKA
2026-09-25 12:08:57 -04:00
Lotus CIandClaude Opus 5.5 cee6bab9a6 feat(lotus): report the local mic level to the host (cinny #146)
The host had only a boolean `speaking` for yourself, throttled to 500 ms, so
nothing on the call bar showed that the mic was actually picking you up.

- lotusMicLevel.ts: one shared local mic sampler (a clone of the published
  track, ~10 Hz RMS, while a mic is published, muted or not), shared per
  call view model so there is one AudioContext.
- MicLevelQuantizer: 0–3 bars (≈ −46/−36/−26 dBFS), rising at once and
  falling one bar per sample so it doesn't flicker between words.
- fromWidget io.lotus.mic_level { bars } only when the value changes; 0 while
  muted or with no mic. Opt-in with the host state stream (lotusCallState=1).
- "Talking while muted" (#37) now reads the same sampler instead of running
  its own; same gate and behaviour.

Verified in a local call with a fake-tone mic: the host receives 0–3 as the
tone rises/falls (about 4 messages/s while it changes), 0 on mute and levels
again on unmute; muted + tone still raises speakingWhileMuted and the host's
"You're muted" notice. Lotus/room suites pass on Node 22 (148 tests).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPmy3tPq869XDW4njjVaKA
2026-09-25 12:08:57 -04:00
Lotus CIandClaude Opus 5.5 44023a4c84 style(lotus): oxfmt the volume memory change (#36)
CI / Build embedded bundle (push) Successful in 1m51s
CI / Publish to Gitea npm registry (push) Successful in 1m5s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPmy3tPq869XDW4njjVaKA
2026-09-25 10:03:49 -04:00
Lotus CIandClaude Opus 5.5 3086edc64a chore(lotus): 0.25.0-lotus.15
CI / Build embedded bundle (push) Failing after 48s
CI / Publish to Gitea npm registry (push) Skipped
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPmy3tPq869XDW4njjVaKA
2026-09-25 09:57:55 -04:00
Lotus CIandClaude Opus 5.5 2b6bb20104 feat(lotus): remember each person's volume across calls (#36)
The per-participant volume slider lived only in the tile's view model, so it
reset to 100 % on the next call, a reload, or when that person reconnected.

- lotus/lotusVolumeMemory.ts: per-Matrix-user volume in localStorage
  (`lotus-per-user-volume`), keyed by user id so it follows them across
  devices; 1 (100 %) forgets the entry; at most 50 people, least recently set
  dropped first; junk in storage ignored. Local only, never synced.
- VolumeControls take an optional initialVolume and onCommit; remote user
  media seeds from and saves to the memory. Screenshare audio stays per-share.
- commitPlaybackVolume now takes the slider's committed value. With the
  keyboard the slider commits before its last change arrives, so the
  committed (and remembered) volume was one step behind, and returning to
  100 % with End never registered.

Verified in a two-person local call: Alice sets Bob to 70 % with the
keyboard → audio element 0.70, stored 0.7; leave + rejoin → slider and audio
0.70; End → entry removed. Unit tests: memory module (LRU, reset, junk),
remembered volume restore + commit-with-value; state/tile/room/lotus suites
all pass on Node 22 (410 tests).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPmy3tPq869XDW4njjVaKA
2026-09-25 09:57:55 -04:00
Lotus CIandClaude Opus 5.5 3ca252f633 chore(lotus): 0.25.0-lotus.14
CI / Build embedded bundle (push) Successful in 2m58s
CI / Publish to Gitea npm registry (push) Successful in 1m3s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPmy3tPq869XDW4njjVaKA
2026-09-24 22:16:39 -04:00
Lotus CIandClaude Opus 5.5 6beebd3ea7 feat(lotus): layout, settings and reactions over the widget API (cinny #43)
The Lotus host drove these by clicking EC's hidden controls in our DOM and
read screenshare/layout state from it, which only works while the frame is
same-origin and breaks whenever a data-testid changes. New actions:

- toWidget io.lotus.set_layout { layout: "grid" | "spotlight" } → the layout
  switch view model's setLayout.
- toWidget io.lotus.open_settings { open?: boolean } → setSettingsOpen
  (toggle when `open` is omitted).
- toWidget io.lotus.toggle_reactions → a window event the reactions button
  listens for.
- fromWidget io.lotus.controls_state { screensharing, layout } on change; its
  arrival tells the host the fork supports the actions above.

Screensharing stays host-driven for now: getDisplayMedia needs the user's
click to reach this frame (Capability Delegation), which a widget message
doesn't carry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPmy3tPq869XDW4njjVaKA
2026-09-24 22:16:24 -04:00
Lotus CIandClaude Opus 5.5 d8880daca5 chore(lotus): 0.25.0-lotus.13
CI / Build embedded bundle (push) Successful in 4m3s
CI / Publish to Gitea npm registry (push) Successful in 2m33s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPmy3tPq869XDW4njjVaKA
2026-09-23 20:51:43 -04:00
Lotus CIandClaude Opus 5.5 3e69a18a39 fix(lotus): explain a failed voice connection and offer Try again
"Failed to connect to Livekit server (Reason: ServerUnreachable)" left a
user with nothing to try; they retried for a long time and gave up.

- Title "Couldn't connect to voice". The message depends on the reason:
  unreachable/timeout-style errors explain that voice needs its own live
  connection, which VPNs, antivirus web shields and some networks block,
  even when chat works, and say what to try; NotAllowed says the call may
  be full or access changed.
- The raw reason, status and message move under Technical details.
- LivekitConnectionError and PeerConnectionTimeoutError now get a
  Try again button (the existing reconnect path), shown as primary: the
  Lotus theme renders `secondary` as dark-on-dark.

Verified in the Lotus client against a local LiveKit with the signal
socket and validate request blocked: the dialog appears after ~20 s;
unblocking and pressing Try again joins with the mic published.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPmy3tPq869XDW4njjVaKA
2026-09-23 20:51:43 -04:00
Lotus CIandClaude Opus 5 e9723a21b5 chore(lotus): 0.25.0-lotus.12
CI / Build embedded bundle (push) Successful in 1m32s
CI / Publish to Gitea npm registry (push) Successful in 54s
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPmy3tPq869XDW4njjVaKA
2026-09-20 15:51:43 -04:00
Lotus CIandClaude Opus 5 9f472fd710 fix(lotus): report an ended screenshare even when LiveKit unpublishes first (#39)
LiveKit's own ended handler runs before ours and unpublishes the share;
a listener removed during that dispatch never fires. Decide from the
track's readyState on LocalTrackUnpublished as well. Verified end to end:
closing the shared source now toasts in the host.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPmy3tPq869XDW4njjVaKA
2026-09-20 15:51:43 -04:00
Lotus CIandClaude Opus 5 e7c27dd8a3 feat(lotus): io.lotus.screenshare_notice — window closed, no frames, sharing alone (#39)
Watches the local screenshare publication and tells the host, once per
share each: the MediaStreamTrack ended (window closed), it has been muted
(no frames — minimised/occluded window) for 15 s, or 30 min of sharing
with nobody else in the call. Detection only; the host renders the
notices. LiveKit already unpublishes an ended share, so nothing lingers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPmy3tPq869XDW4njjVaKA
2026-09-20 15:44:22 -04:00
Lotus CIandClaude Opus 5 33b51e5dc6 feat(lotus): screenshare tile speaker glyph reflects real audio state (#38)
The spotlight screenshare tile already shows a speaker glyph only when the
share publishes audio. Two gaps: a publication the sharer had muted still
counted as audio, and the host's screenshare-audio mute (io.lotus.set_deafen
→ renderer muted) wasn't reflected — the glyph stayed "on" while nothing
played. audioEnabled$ now tracks publication presence AND !isMuted via
observeParticipantMedia; the glyph shows crossed when either the tile's
own mute or the Lotus mute is active.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPmy3tPq869XDW4njjVaKA
2026-09-20 15:42:48 -04:00
Lotus CIandClaude Opus 5 07def55469 chore(lotus): 0.25.0-lotus.11
CI / Build embedded bundle (push) Successful in 1m42s
CI / Publish to Gitea npm registry (push) Successful in 47s
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPmy3tPq869XDW4njjVaKA
2026-09-20 15:31:33 -04:00
Lotus CIandClaude Opus 5 4a48ec98d0 feat(lotus): io.lotus.set_audio_output — host-driven output device switch (#119)
The embed hides Element Call's footer, so its own output picker is out of
reach; the host's call bar now sends io.lotus.set_audio_output { deviceId }
and the fork applies it via mediaDevices.audioOutput.select(), answering
with io.lotus.audio_output_state { deviceId } on every change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPmy3tPq869XDW4njjVaKA
2026-09-20 15:31:33 -04:00
Lotus CIandClaude Opus 5 b9d0ac7cac chore(lotus): 0.25.0-lotus.10
CI / Build embedded bundle (push) Successful in 1m41s
CI / Publish to Gitea npm registry (push) Successful in 45s
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPmy3tPq869XDW4njjVaKA
2026-09-20 13:47:46 -04:00
Lotus CIandClaude Opus 5 1e34923f6a feat(lotus): io.lotus.call_summary — per-call quality readout at hangup (#143)
Tracks the local participant's LiveKit ConnectionQuality (time spent
poor/lost) and Reconnecting events for the life of the in-call view and
sends one io.lotus.call_summary { durationMs, reconnects, poorMs, verdict }
to the host on the first SFU disconnect or on teardown. Nothing is stored
or sent anywhere else. CallQualityTracker is pure and unit-tested.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPmy3tPq869XDW4njjVaKA
2026-09-20 13:39:18 -04:00
Lotus CIandClaude Opus 5 fcb7f8456d chore(lotus): 0.25.0-lotus.9
CI / Build embedded bundle (push) Successful in 1m34s
CI / Publish to Gitea npm registry (push) Successful in 1m7s
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPmy3tPq869XDW4njjVaKA
2026-09-19 22:45:23 -04:00
Lotus CIandClaude Opus 5 c2267800be feat(lotus): report speakingWhileMuted for the local participant in io.lotus.call_state (#37)
While the mic is muted LiveKit keeps the capture alive and merely disables the
published MediaStreamTrack, so VAD says nothing. lotusMutedSpeech taps a CLONE
of the published track (post-denoise when the in-source processor is active),
samples RMS at 10 Hz through an AnalyserNode and runs a hysteresis gate (300 ms
of voice on, 800 ms of quiet off, threshold ≈ −36 dBFS). The tap only exists
while a muted mic publication exists; the flag is set on the local entry of
io.lotus.call_state only and is never sent to other participants. Gate is
unit-tested; verified end-to-end: muted with a tone mic → true within 2.5 s,
silence → false, unmute → false.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPmy3tPq869XDW4njjVaKA
2026-09-19 22:45:23 -04:00
Lotus CIandClaude Opus 5 5421d545c4 chore(lotus): 0.25.0-lotus.8
CI / Build embedded bundle (push) Successful in 2m34s
CI / Publish to Gitea npm registry (push) Successful in 57s
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPmy3tPq869XDW4njjVaKA
2026-09-19 14:35:59 -04:00
Lotus CIandClaude Opus 5 2fafa3cf36 test(lotus): oxlint-clean mock in the LocalTransport refusal test
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPmy3tPq869XDW4njjVaKA
2026-09-19 14:35:59 -04:00
Lotus CIandClaude Opus 5 667230f6e3 chore(lotus): 0.25.0-lotus.7
CI / Build embedded bundle (push) Failing after 50s
CI / Publish to Gitea npm registry (push) Skipped
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPmy3tPq869XDW4njjVaKA
2026-09-19 14:35:03 -04:00
Lotus CIandClaude Opus 5 746917a4c6 test(lotus): satisfy oxlint require-await in the LocalTransport refusal test
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPmy3tPq869XDW4njjVaKA
2026-09-19 14:35:03 -04:00
Lotus CIandClaude Opus 5 d881833491 chore(lotus): 0.25.0-lotus.6
CI / Build embedded bundle (push) Failing after 56s
CI / Publish to Gitea npm registry (push) Skipped
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPmy3tPq869XDW4njjVaKA
2026-09-19 14:04:45 -04:00
Lotus CIandClaude Opus 5 021b1881e5 fix(lotus): LocalTransport must not re-wrap SFUTokenRefusedError
mapAuthErrorToUserFriendlyError turned every non-whitelisted error back into
FailToGetOpenIdToken, so lotus.5's refusal reason still surfaced as the
generic page (caught end-to-end with a routed 403 on /sfu/get). Pass it
through like the other user-facing auth errors. Unit-tested.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPmy3tPq869XDW4njjVaKA
2026-09-19 14:04:45 -04:00
Lotus CIandClaude Opus 5 1b609d997b chore(lotus): 0.25.0-lotus.5
CI / Build embedded bundle (push) Successful in 1m47s
CI / Publish to Gitea npm registry (push) Successful in 1m2s
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPmy3tPq869XDW4njjVaKA
2026-09-19 13:42:22 -04:00
Lotus CIandClaude Opus 5 d9ac9a0fa4 fix(lotus): show the SFU token service's refusal reason instead of "Something went wrong"
When the voice-limit guard (or any JWT service) answers 403 with a reason —
"This voice channel is full.", "You don't have permission to …" — EC wrapped it
in FailToGetOpenIdToken and the user saw the generic error page with
OPEN_ID_ERROR. A 403 MatrixError with a non-empty `error` now throws
SFUTokenRefusedError ("Can't join this call" + the server's sentence), so the
reason is the description. Other failures are unchanged. Unit-tested.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPmy3tPq869XDW4njjVaKA
2026-09-19 13:42:22 -04:00
Lotus CIandClaude Opus 5 66bfead7f1 chore(lotus): 0.25.0-lotus.4
CI / Build embedded bundle (push) Successful in 1m44s
CI / Publish to Gitea npm registry (push) Successful in 54s
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPmy3tPq869XDW4njjVaKA
2026-09-18 23:54:25 -04:00
Lotus CIandClaude Opus 5 ea579cb998 fix(lotus): screenshare-audio mute survives a re-share — mute via the renderer, not setVolume
"Mute Screenshare Audio" (io.lotus.set_deafen screenshareAudioMuted) used
RemoteParticipant.setVolume(0, ScreenShareAudio). EC's own createVolumeControls
writes volume 1 through the same setter the moment a new screenshare media item
resolves, so when the sharer stopped and re-shared (or a late joiner shared)
the audio came back at full volume while the host button still said
"Unmute Screenshare Audio". Reproduced on the local calls stack with two
headless clients: after a re-share the screen_share_audio element read vol=1.

Now the flag is a global behavior (muteScreenshareAudio$) that
LivekitRoomAudioRenderer turns into the `muted` prop of every
Track.Source.ScreenShareAudio element — the exact mechanism deafen already uses
(pub.setEnabled(false): the server stops sending). Verified via the
RemoteTrackPublication behind each <audio>: the re-published track (new sid)
mounts with enabled=false while muted and re-enables on unmute; deafen +
undeafen leaves it muted; teardown resets the flag so the next call starts
clean. Unit tests updated; renderer test asserts only ScreenShareAudio
elements get muted by the new prop.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPmy3tPq869XDW4njjVaKA
2026-09-18 23:54:25 -04:00
39 changed files with 1922 additions and 263 deletions
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@lotusguild/element-call-embedded",
"version": "0.25.0-lotus.3",
"version": "0.25.0-lotus.16",
"files": [
"README.md",
"LICENSE-AGPL-3.0",
+5 -2
View File
@@ -104,8 +104,9 @@
"generic_description": "Submitting debug logs will help us track down the problem.",
"insufficient_capacity": "Insufficient capacity",
"insufficient_capacity_description": "The server has reached its maximum capacity and you cannot join the call at this time. Try again later, or contact your server admin if the problem persists.",
"livekit_connection_error": "Failed to connect to Livekit server",
"livekit_connection_error_description": "An error occurred while connecting to the Livekit server (<1>Reason:</1> <2>{{ reason }}</2>).",
"livekit_connection_error": "Couldn’t connect to voice",
"livekit_not_allowed_description": "The voice server didn’t let you in. The call may be full, or your access to this room may have changed. Try again in a moment.",
"livekit_unreachable_description": "Your device couldn’t reach the voice server. Chat can still work when this happens: voice needs its own live connection, which VPNs, antivirus web protection and some work or school networks block. Try again. If it keeps failing, pause your VPN or antivirus web shield, or try another network such as a phone hotspot.",
"matrix_rtc_transport_missing": "The server is not configured to work with {{brand}}. Please contact your server admin (Domain: {{domain}}, Error Code: {{ errorCode }}).",
"membership_manager": "Membership Manager Error",
"membership_manager_description": "The Membership Manager had to shut down. This is caused by many consecutive failed network requests.",
@@ -116,8 +117,10 @@
"peer_connection_timeout_description": "Connection to the media server timed out. Try switching to a different network or disabling your VPN. If the problem persists, see our <0>troubleshooting guide</0> or contact your server administrator.",
"room_creation_restricted": "Failed to create call",
"room_creation_restricted_description": "Call creation might be restricted to authorized users only. Try again later, or contact your server admin if the problem persists.",
"sfu_token_refused": "Can't join this call",
"sticky_events_required": "Homeserver does not support Matrix 2.0 calls",
"sticky_events_required_description": "This deployment is configured to use Matrix 2.0 call mode, but the homeserver does not advertise support for sticky events (MSC4354). Ask your server admin to upgrade, or switch the deployment to a compatible mode.",
"try_again": "Try again",
"unexpected_ec_error": "An unexpected error occurred (<0>Error Code:</0> <1>{{ errorCode }}</1>). Please contact your server admin."
},
"group_call_loader": {
+10
View File
@@ -26,6 +26,7 @@ import { logger } from "matrix-js-sdk/lib/logger";
import classNames from "classnames";
import { useReactionsSender } from "../reactions/useReactionsSender";
import { LOTUS_TOGGLE_REACTIONS_EVENT } from "../lotus/lotusControls";
import styles from "./ReactionToggleButton.module.css";
import {
type RaisedHandInfo,
@@ -196,6 +197,15 @@ export function ReactionToggleButton({
setErrorText(undefined);
}, [showReactionsMenu]);
// [cinny #43] The Lotus host's call bar toggles this menu over the widget
// API (lotusControls.ts) instead of clicking this button in our DOM.
useEffect(() => {
const toggle = (): void => setShowReactionsMenu((open) => !open);
window.addEventListener(LOTUS_TOGGLE_REACTIONS_EVENT, toggle);
return (): void =>
window.removeEventListener(LOTUS_TOGGLE_REACTIONS_EVENT, toggle);
}, []);
const sendRelation = useCallback(
async (reaction: ReactionOption) => {
try {
+63 -2
View File
@@ -52,9 +52,16 @@ afterEach(() => {
vi.mock("@livekit/components-react", async (importOriginal) => {
return {
...(await importOriginal()),
AudioTrack: (props: { trackRef: TrackReference }): ReactNode => {
AudioTrack: (props: {
trackRef: TrackReference;
muted?: boolean;
}): ReactNode => {
return (
<audio data-testid={"audio"}>
<audio
data-testid={"audio"}
data-source={props.trackRef.publication.source}
data-muted={String(!!props.muted)}
>
{getTrackReferenceId(props.trackRef)}
</audio>
);
@@ -83,6 +90,7 @@ function renderTestComponent(
kind: Track.Kind;
source: Track.Source;
}[],
props: { muted?: boolean; screenshareAudioMuted?: boolean } = {},
): RenderResult {
const liveKitParticipants = livekitParticipantIdentities.map((identity) =>
mockRemoteParticipant({ identity }),
@@ -117,6 +125,7 @@ function renderTestComponent(
validIdentities={participants.map((p) => p.identity)}
livekitRoom={livekitRoom}
url={""}
{...props}
/>
</MediaDevicesProvider>,
);
@@ -286,3 +295,55 @@ it("should setup audioContext gain and pan", () => {
expect(testAudioContext.gain.gain.value).toEqual(0.1);
expect(testAudioContext.pan.pan.value).toEqual(1);
});
// [lotus] The host's "Mute Screenshare Audio" mutes ONLY screenshare-audio
// elements, via the same `muted` prop path as deafen (so it survives re-renders
// and later-published shares); deafen still mutes everything.
it("screenshareAudioMuted mutes only ScreenShareAudio tracks", () => {
const explicitTracks = [
{
participantId: "@alice:DEV0",
kind: Track.Kind.Audio,
source: Track.Source.Microphone,
},
{
participantId: "@alice:DEV0",
kind: Track.Kind.Audio,
source: Track.Source.ScreenShareAudio,
},
];
const mutedBySource = (r: RenderResult): Record<string, string> =>
Object.fromEntries(
r
.queryAllByTestId("audio")
.map((el) => [el.dataset.source, el.dataset.muted]),
);
expect(
mutedBySource(
renderTestComponent(
[{ userId: "@alice", deviceId: "DEV0" }],
["@alice:DEV0"],
explicitTracks,
{ screenshareAudioMuted: true },
),
),
).toEqual({
[Track.Source.Microphone]: "false",
[Track.Source.ScreenShareAudio]: "true",
});
expect(
mutedBySource(
renderTestComponent(
[{ userId: "@alice", deviceId: "DEV0" }],
["@alice:DEV0"],
explicitTracks,
{ muted: true },
),
),
).toEqual({
[Track.Source.Microphone]: "true",
[Track.Source.ScreenShareAudio]: "true",
});
});
+12 -1
View File
@@ -38,6 +38,12 @@ export interface MatrixAudioRendererProps {
* If set to `true`, the server will stop sending audio track data to the client.
*/
muted?: boolean;
/**
* [lotus] If set to `true`, mutes only the `Track.Source.ScreenShareAudio`
* tracks (the host's "Mute Screenshare Audio" control) — same mechanism as
* `muted`, so it holds across re-renders and later-published shares.
*/
screenshareAudioMuted?: boolean;
}
/**
@@ -58,6 +64,7 @@ export function LivekitRoomAudioRenderer({
livekitRoom,
validIdentities,
muted,
screenshareAudioMuted,
}: MatrixAudioRendererProps): ReactNode {
const logger = rootLogger.getChild("[MatrixAudioRenderer]");
const tracks = useTracks(
@@ -143,7 +150,11 @@ export function LivekitRoomAudioRenderer({
<AudioTrackWithAudioNodes
key={getTrackReferenceId(trackRef)}
trackRef={trackRef}
muted={muted}
muted={
muted ||
(screenshareAudioMuted &&
trackRef.publication.source === Track.Source.ScreenShareAudio)
}
audioContext={shouldUseAudioContext ? audioContext : undefined}
audioNodes={audioNodes}
/>
+26 -1
View File
@@ -20,7 +20,7 @@ import { MatrixError } from "matrix-js-sdk";
import { getSFUConfigWithOpenID, type OpenIDClientParts } from "./openIDSFU";
import { testJWTToken } from "../utils/test-fixtures";
import { ownMemberMock } from "../utils/test";
import { FailToGetOpenIdToken } from "../utils/errors";
import { FailToGetOpenIdToken, SFUTokenRefusedError } from "../utils/errors";
const sfuUrl = "https://sfu.example.org";
@@ -91,6 +91,31 @@ describe("getSFUConfigWithOpenID", () => {
expect.fail("Expected test to throw;");
});
it("[lotus] surfaces a 403 refusal's reason instead of the generic error", async () => {
fetchMock.post("https://sfu.example.org/sfu/get", () => {
return {
status: 403,
body: { errcode: "M_FORBIDDEN", error: "This voice channel is full." },
};
});
try {
await getSFUConfigWithOpenID(
matrixClient,
ownMemberMock,
"https://sfu.example.org",
"!example_room_id",
);
} catch (ex: unknown) {
expect(ex).toBeInstanceOf(SFUTokenRefusedError);
expect((ex as SFUTokenRefusedError).localisedMessage).toEqual(
"This voice channel is full.",
);
void (await fetchMock.flush());
return;
}
expect.fail("Expected test to throw;");
});
it("should retry without delay params if the JWT service legacy endpoint returns M_BAD_JSON 400", async () => {
let callCount = 0;
+10
View File
@@ -8,6 +8,7 @@ Please see LICENSE in the repository root for full details.
import {
type IOpenIDToken,
type MatrixClient,
MatrixError,
parseErrorResponse,
} from "matrix-js-sdk";
import { type CallMembershipIdentityParts } from "matrix-js-sdk/lib/matrixrtc/EncryptionManager";
@@ -16,6 +17,7 @@ import { type Logger } from "matrix-js-sdk/lib/logger";
import {
FailToGetOpenIdToken,
NoMatrix2AuthorizationService,
SFUTokenRefusedError,
} from "../utils/errors";
import { doNetworkOperationWithRetry } from "../utils/matrix";
import { Config } from "../config/Config";
@@ -165,6 +167,14 @@ export async function getSFUConfigWithOpenID(
logger?.info(`Got JWT from call's active focus URL.`);
return extractFullConfigFromToken(sfuConfig);
} catch (ex) {
// [lotus] A 403 from the token service carries the reason the user needs
// ("This voice channel is full.") — surface it instead of the generic error.
if (ex instanceof MatrixError && ex.httpStatus === 403) {
const reason = (ex.data as { error?: unknown } | undefined)?.error;
if (typeof reason === "string" && reason.trim()) {
throw new SFUTokenRefusedError(reason, ex);
}
}
throw new FailToGetOpenIdToken(
ex instanceof Error ? ex : new Error(`Unknown error ${ex}`),
);
+8
View File
@@ -26,6 +26,10 @@ describe("LotusWidgetActions", () => {
LotusWidgetActions.SetQuality,
LotusWidgetActions.Decorations,
LotusWidgetActions.SetDeafen,
LotusWidgetActions.SetAudioOutput,
LotusWidgetActions.SetLayout,
LotusWidgetActions.OpenSettings,
LotusWidgetActions.ToggleReactions,
];
expect(new Set(LOTUS_TO_WIDGET_ACTIONS)).toEqual(new Set(expectedToWidget));
@@ -43,5 +47,9 @@ describe("LotusWidgetActions", () => {
expect(LOTUS_TO_WIDGET_ACTIONS).not.toContain(
LotusWidgetActions.DenoiseState,
);
expect(LOTUS_TO_WIDGET_ACTIONS).not.toContain(
LotusWidgetActions.ControlsState,
);
expect(LOTUS_TO_WIDGET_ACTIONS).not.toContain(LotusWidgetActions.MicLevel);
});
});
+43
View File
@@ -46,6 +46,45 @@ export enum LotusWidgetActions {
* toggle can reflect reality rather than the requested state.
*/
DenoiseState = "io.lotus.denoise_state",
/**
* fromWidget: one-shot end-of-call readout for the local participant —
* `{ durationMs, reconnects, poorMs, verdict }` (#143) — sent on SFU
* disconnect or in-call teardown, whichever comes first.
*/
CallSummary = "io.lotus.call_summary",
/** toWidget: select the audio output device `{ deviceId }` (#119). */
SetAudioOutput = "io.lotus.set_audio_output",
/** fromWidget: the currently selected output `{ deviceId }` (#119). */
AudioOutputState = "io.lotus.audio_output_state",
/**
* fromWidget: local screenshare reminder `{ kind: "ended" | "no-frames" | "alone" }`
* — window closed, no frames for 15 s, or 30 min of sharing with nobody
* else in the call (#39). Each fires at most once per share.
*/
ScreenshareNotice = "io.lotus.screenshare_notice",
/**
* toWidget: switch the call layout `{ layout: "grid" | "spotlight" }`
* (cinny #43 — replaces the host clicking EC's hidden layout radio).
*/
SetLayout = "io.lotus.set_layout",
/**
* toWidget: open EC's settings modal, or close it with `{ open: false }`;
* omit `open` to toggle (cinny #43).
*/
OpenSettings = "io.lotus.open_settings",
/** toWidget: toggle the reactions / raise-hand menu (cinny #43). */
ToggleReactions = "io.lotus.toggle_reactions",
/**
* fromWidget: `{ screensharing: boolean, layout: "grid" | "spotlight" | null }`
* whenever either changes (cinny #43), so the host stops reading EC's DOM for
* them. Its arrival also tells the host this fork supports the actions above.
*/
ControlsState = "io.lotus.controls_state",
/**
* fromWidget: local mic level `{ bars: 0 | 1 | 2 | 3 }` (cinny #146), sent
* only when it changes (≤ 10 Hz); 0 while muted or with no mic.
*/
MicLevel = "io.lotus.mic_level",
}
/** toWidget Lotus actions that must be allow-listed in `initializeWidget`. */
@@ -55,4 +94,8 @@ export const LOTUS_TO_WIDGET_ACTIONS: LotusWidgetActions[] = [
LotusWidgetActions.SetQuality,
LotusWidgetActions.Decorations,
LotusWidgetActions.SetDeafen,
LotusWidgetActions.SetAudioOutput,
LotusWidgetActions.SetLayout,
LotusWidgetActions.OpenSettings,
LotusWidgetActions.ToggleReactions,
];
+46
View File
@@ -0,0 +1,46 @@
/*
Copyright 2026 Lotus Guild
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { type IWidgetApiRequest } from "matrix-widget-api";
import { logger } from "matrix-js-sdk/lib/logger";
import { type MediaDevices } from "../state/MediaDevices";
import { widget } from "../widget";
import { LotusWidgetActions } from "./lotusActions";
import { lotusSendToHost } from "./lotusWidget";
/**
* [Gitea #119] Let the host switch the audio output device (headset ↔
* speakers) from its own call bar: `io.lotus.set_audio_output { deviceId }`
* selects the output; the fork answers each change (and the initial state)
* with `io.lotus.audio_output_state { deviceId, available: [{id,label}] }`.
* No effect unless the host sends the action. Returns a teardown function.
*/
export function startLotusAudioOutput(mediaDevices: MediaDevices): () => void {
const w = widget;
if (!w) return () => undefined;
const handler = (ev: CustomEvent<IWidgetApiRequest>): void => {
w.api.transport.reply(ev.detail, {});
const data = ev.detail.data as { deviceId?: string } | undefined;
if (typeof data?.deviceId !== "string") return;
logger.debug(`[lotus] set_audio_output: ${data.deviceId}`);
mediaDevices.audioOutput.select(data.deviceId);
};
w.lazyActions.on(LotusWidgetActions.SetAudioOutput, handler);
const sub = mediaDevices.audioOutput.selected$.subscribe((selected) => {
lotusSendToHost(LotusWidgetActions.AudioOutputState, {
deviceId: selected?.id ?? null,
});
});
return () => {
sub.unsubscribe();
w.lazyActions.off(LotusWidgetActions.SetAudioOutput, handler);
};
}
+5 -1
View File
@@ -53,7 +53,11 @@ function mockMember(id: string, userId: string): Member {
}
function mockVm(members: Member[]): CallViewModel {
return { userMedia$: of(members) } as unknown as CallViewModel;
return {
userMedia$: of(members),
// no livekit connections → the muted-speech tap never starts
allConnections$: of({ getConnections: () => [] }),
} as unknown as CallViewModel;
}
function participantsOf(call: number): unknown[] {
+41 -28
View File
@@ -5,7 +5,7 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { combineLatest, of, type Subscription } from "rxjs";
import { combineLatest, of, startWith, type Subscription } from "rxjs";
import {
distinctUntilChanged,
map,
@@ -16,6 +16,7 @@ import {
import { type CallViewModel } from "../state/CallViewModel/CallViewModel";
import { widget } from "../widget";
import { LotusWidgetActions, lotusFlag, lotusSendToHost } from "./lotusWidget";
import { observeSpeakingWhileMuted$ } from "./lotusMutedSpeech";
interface ParticipantState {
/** EC media id (`${userId}:${deviceId}`), stable per participant device. */
@@ -25,6 +26,11 @@ interface ParticipantState {
speaking: boolean;
audioEnabled: boolean;
videoEnabled: boolean;
/**
* [lotus #37] LOCAL participant only: voice detected on the mic while it is
* muted (see lotusMutedSpeech.ts). Absent for remote participants.
*/
speakingWhileMuted?: boolean;
}
/**
@@ -45,7 +51,8 @@ function participantsEqual(
p.userId === b[i].userId &&
p.speaking === b[i].speaking &&
p.audioEnabled === b[i].audioEnabled &&
p.videoEnabled === b[i].videoEnabled,
p.videoEnabled === b[i].videoEnabled &&
p.speakingWhileMuted === b[i].speakingWhileMuted,
)
);
}
@@ -66,34 +73,40 @@ export function startLotusCallState(vm: CallViewModel): () => void {
// lotusDecorations.ts.
if (!widget) return () => undefined;
const sub: Subscription = vm.userMedia$
.pipe(
switchMap((members) =>
members.length === 0
? of([] as ParticipantState[])
: combineLatest(
members.map((m) =>
combineLatest([
m.speaking$,
m.audioEnabled$,
m.videoEnabled$,
]).pipe(
map(
([
speaking,
audioEnabled,
videoEnabled,
]): ParticipantState => ({
id: m.id,
userId: m.userId,
speaking,
audioEnabled,
videoEnabled,
}),
),
),
const participants$ = vm.userMedia$.pipe(
switchMap((members) =>
members.length === 0
? of([] as (ParticipantState & { local: boolean })[])
: combineLatest(
members.map((m) =>
combineLatest([
m.speaking$,
m.audioEnabled$,
m.videoEnabled$,
]).pipe(
map(([speaking, audioEnabled, videoEnabled]) => ({
id: m.id,
userId: m.userId,
speaking,
audioEnabled,
videoEnabled,
local: m.local === true,
})),
),
),
),
),
);
const sub: Subscription = combineLatest([
participants$,
observeSpeakingWhileMuted$(vm).pipe(startWith(false)),
])
.pipe(
map(([members, speakingWhileMuted]): ParticipantState[] =>
members.map(({ local, ...p }) =>
local ? { ...p, speakingWhileMuted } : p,
),
),
// `speaking` flips rapidly; drop no-op repeats BEFORE throttling so
// the throttle window isn't spent re-emitting an unchanged value.
+93
View File
@@ -0,0 +1,93 @@
/*
Copyright 2026 Lotus Guild
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { ConnectionQuality } from "livekit-client";
import { describe, expect, it } from "vitest";
import { CallQualityTracker } from "./lotusCallSummary";
const clock = (): { now: () => number; tick: (ms: number) => void } => {
let t = 1_000_000;
return {
now: () => t,
tick: (ms) => {
t += ms;
},
};
};
describe("CallQualityTracker", () => {
it("reports unknown when nothing was sampled", () => {
const c = clock();
const tr = new CallQualityTracker(c.now);
tr.start();
c.tick(60_000);
expect(tr.summary()).toEqual({
durationMs: 60_000,
reconnects: 0,
poorMs: 0,
verdict: "unknown",
});
});
it("is good when quality stayed fine", () => {
const c = clock();
const tr = new CallQualityTracker(c.now);
tr.start();
tr.setQuality(ConnectionQuality.Excellent);
c.tick(30 * 60_000);
expect(tr.summary().verdict).toBe("good");
});
it("accumulates poor time across episodes, including an open one", () => {
const c = clock();
const tr = new CallQualityTracker(c.now);
tr.start();
tr.setQuality(ConnectionQuality.Good);
c.tick(60_000);
tr.setQuality(ConnectionQuality.Poor);
c.tick(10_000);
tr.setQuality(ConnectionQuality.Good);
c.tick(60_000);
tr.setQuality(ConnectionQuality.Lost);
c.tick(5_000);
const s = tr.summary();
expect(s.poorMs).toBe(15_000);
expect(s.durationMs).toBe(135_000);
expect(s.verdict).toBe("fair");
});
it("is poor with many reconnects or mostly-poor quality", () => {
const c = clock();
const tr = new CallQualityTracker(c.now);
tr.start();
tr.setQuality(ConnectionQuality.Good);
for (let i = 0; i < 4; i += 1) tr.reconnect();
c.tick(60_000);
expect(tr.summary()).toMatchObject({ reconnects: 4, verdict: "poor" });
const tr2 = new CallQualityTracker(c.now);
tr2.start();
tr2.setQuality(ConnectionQuality.Poor);
c.tick(60_000);
expect(tr2.summary().verdict).toBe("poor");
});
it("ignores unknown samples and only starts once", () => {
const c = clock();
const tr = new CallQualityTracker(c.now);
tr.start();
c.tick(1_000);
tr.start();
tr.setQuality(ConnectionQuality.Unknown);
c.tick(1_000);
expect(tr.summary()).toMatchObject({
durationMs: 2_000,
verdict: "unknown",
});
});
});
+144
View File
@@ -0,0 +1,144 @@
/*
Copyright 2026 Lotus Guild
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import {
ConnectionQuality,
type Participant,
type Room as LivekitRoom,
RoomEvent,
} from "livekit-client";
import { type CallViewModel } from "../state/CallViewModel/CallViewModel";
import { widget } from "../widget";
import { LotusWidgetActions } from "./lotusActions";
import { lotusSendToHost } from "./lotusWidget";
export type CallQualityVerdict = "good" | "fair" | "poor" | "unknown";
export interface CallSummary {
/** Wall-clock time connected to the SFU, ms. */
durationMs: number;
/** LiveKit reconnect attempts during the call. */
reconnects: number;
/** Time the local connection quality was reported poor/lost, ms. */
poorMs: number;
verdict: CallQualityVerdict;
}
/**
* [Gitea #143] Per-call connection quality, kept in memory for the local
* participant only and summarised once at hangup. Nothing is stored or sent
* anywhere but to the host at the end.
*/
export class CallQualityTracker {
private startedAt: number | undefined;
private poorSince: number | undefined;
private poorMs = 0;
private reconnects = 0;
private sampled = false;
public constructor(private readonly now: () => number = () => Date.now()) {}
public start(): void {
if (this.startedAt === undefined) this.startedAt = this.now();
}
public setQuality(quality: ConnectionQuality): void {
if (quality === ConnectionQuality.Unknown) return;
this.sampled = true;
const bad =
quality === ConnectionQuality.Poor || quality === ConnectionQuality.Lost;
if (bad && this.poorSince === undefined) this.poorSince = this.now();
if (!bad && this.poorSince !== undefined) {
this.poorMs += this.now() - this.poorSince;
this.poorSince = undefined;
}
}
public reconnect(): void {
this.reconnects += 1;
}
public summary(): CallSummary {
const end = this.now();
const durationMs =
this.startedAt === undefined ? 0 : Math.max(0, end - this.startedAt);
const poorMs =
this.poorMs + (this.poorSince === undefined ? 0 : end - this.poorSince);
let verdict: CallQualityVerdict = "unknown";
if (this.sampled && durationMs > 0) {
const poorShare = poorMs / durationMs;
if (poorShare < 0.05 && this.reconnects <= 1) verdict = "good";
else if (poorShare < 0.25 && this.reconnects <= 3) verdict = "fair";
else verdict = "poor";
}
return { durationMs, reconnects: this.reconnects, poorMs, verdict };
}
}
/**
* Track the local participant's LiveKit connection quality and reconnects
* for the life of the in-call view, and send `io.lotus.call_summary` to the
* host once — on the first SFU disconnect or on teardown, whichever comes
* first — so the host can show "41 min · connection was good" at hangup.
*/
export function startLotusCallSummary(vm: CallViewModel): () => void {
if (!widget) return () => undefined;
const tracker = new CallQualityTracker();
let sent = false;
const send = (): void => {
if (sent) return;
sent = true;
lotusSendToHost(LotusWidgetActions.CallSummary, tracker.summary());
};
const listeners = new Map<LivekitRoom, () => void>();
const attach = (room: LivekitRoom): void => {
const onQuality = (
quality: ConnectionQuality,
participant: Participant,
): void => {
if (participant.isLocal) tracker.setQuality(quality);
};
const onConnected = (): void => tracker.start();
const onReconnecting = (): void => tracker.reconnect();
const onDisconnected = (): void => send();
room.on(RoomEvent.ConnectionQualityChanged, onQuality);
room.on(RoomEvent.Connected, onConnected);
room.on(RoomEvent.Reconnecting, onReconnecting);
room.on(RoomEvent.Disconnected, onDisconnected);
if (room.state === "connected") tracker.start();
listeners.set(room, () => {
room.off(RoomEvent.ConnectionQualityChanged, onQuality);
room.off(RoomEvent.Connected, onConnected);
room.off(RoomEvent.Reconnecting, onReconnecting);
room.off(RoomEvent.Disconnected, onDisconnected);
});
};
const sub = vm.allConnections$.subscribe((data) => {
const next = data.getConnections().map((c) => c.livekitRoom);
for (const [room, off] of listeners) {
if (!next.includes(room)) {
off();
listeners.delete(room);
}
}
for (const room of next) if (!listeners.has(room)) attach(room);
});
return () => {
sub.unsubscribe();
for (const off of listeners.values()) off();
listeners.clear();
send();
};
}
+35
View File
@@ -0,0 +1,35 @@
/*
Copyright 2026 Lotus Guild
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { describe, expect, it } from "vitest";
import { parseLayoutPayload, parseSettingsPayload } from "./lotusControls";
describe("parseLayoutPayload", () => {
it("accepts grid and spotlight", () => {
expect(parseLayoutPayload({ layout: "grid" })).toBe("grid");
expect(parseLayoutPayload({ layout: "spotlight" })).toBe("spotlight");
});
it("rejects anything else", () => {
expect(parseLayoutPayload({ layout: "pip" })).toBeUndefined();
expect(parseLayoutPayload({})).toBeUndefined();
expect(parseLayoutPayload(null)).toBeUndefined();
expect(parseLayoutPayload("grid")).toBeUndefined();
});
});
describe("parseSettingsPayload", () => {
it("uses an explicit open flag", () => {
expect(parseSettingsPayload({ open: true }, true)).toBe(true);
expect(parseSettingsPayload({ open: false }, false)).toBe(false);
});
it("toggles when open is missing or not a boolean", () => {
expect(parseSettingsPayload({}, false)).toBe(true);
expect(parseSettingsPayload(undefined, true)).toBe(false);
expect(parseSettingsPayload({ open: "yes" }, true)).toBe(false);
});
});
+104
View File
@@ -0,0 +1,104 @@
/*
Copyright 2026 Lotus Guild
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { type IWidgetApiRequest } from "matrix-widget-api";
import { combineLatest, of, type Subscription } from "rxjs";
import { distinctUntilChanged, map, switchMap } from "rxjs/operators";
import { type CallViewModel } from "../state/CallViewModel/CallViewModel";
import { type LayoutMode } from "../state/LayoutSwitchViewModel";
import { widget } from "../widget";
import { LotusWidgetActions, lotusSendToHost } from "./lotusWidget";
/** Window event the reactions button listens for (see ReactionToggleButton). */
export const LOTUS_TOGGLE_REACTIONS_EVENT = "lotus:toggle-reactions";
export interface LotusControlsState {
screensharing: boolean;
/** Null while EC offers no layout switch (e.g. PiP or a 1:1 layout). */
layout: LayoutMode | null;
}
/** `{ layout }` payload → a layout mode, or undefined if invalid. Exported for tests. */
export function parseLayoutPayload(data: unknown): LayoutMode | undefined {
if (typeof data !== "object" || data === null) return undefined;
const { layout } = data as { layout?: unknown };
return layout === "grid" || layout === "spotlight" ? layout : undefined;
}
/** `{ open? }` payload → the settings-open state to apply. Exported for tests. */
export function parseSettingsPayload(data: unknown, current: boolean): boolean {
if (typeof data === "object" && data !== null && "open" in data) {
const { open } = data as { open?: unknown };
if (typeof open === "boolean") return open;
}
return !current;
}
/**
* [cinny #43] Widget-API replacements for the host's DOM access to EC's
* controls: layout switch, settings modal and reactions menu, plus a
* `controls_state` report (screensharing + layout) so the host no longer reads
* EC's DOM for them. Screensharing itself stays host-DOM driven for now:
* `getDisplayMedia` needs the user's click to reach this frame (Capability
* Delegation), which a plain widget message doesn't carry.
*
* No effect unless the host sends the actions; registering is safe whenever
* we're a widget. Returns a teardown function.
*/
export function startLotusControls(vm: CallViewModel): () => void {
const w = widget;
if (!w) return (): void => undefined;
const onSetLayout = (ev: CustomEvent<IWidgetApiRequest>): void => {
w.api.transport.reply(ev.detail, {});
const layout = parseLayoutPayload(ev.detail.data);
if (layout) vm.layoutSwitchVm$.value?.setLayout(layout);
};
const onOpenSettings = (ev: CustomEvent<IWidgetApiRequest>): void => {
w.api.transport.reply(ev.detail, {});
vm.setSettingsOpen$.value(
parseSettingsPayload(ev.detail.data, vm.settingsOpen$.value),
);
};
const onToggleReactions = (ev: CustomEvent<IWidgetApiRequest>): void => {
w.api.transport.reply(ev.detail, {});
window.dispatchEvent(new Event(LOTUS_TOGGLE_REACTIONS_EVENT));
};
w.lazyActions.on(LotusWidgetActions.SetLayout, onSetLayout);
w.lazyActions.on(LotusWidgetActions.OpenSettings, onOpenSettings);
w.lazyActions.on(LotusWidgetActions.ToggleReactions, onToggleReactions);
const sub: Subscription = combineLatest([
vm.sharingScreen$,
vm.layoutSwitchVm$.pipe(
switchMap((l) => (l ? l.layout$ : of<LayoutMode | null>(null))),
),
])
.pipe(
map(
([screensharing, layout]): LotusControlsState => ({
screensharing,
layout,
}),
),
distinctUntilChanged(
(a, b) => a.screensharing === b.screensharing && a.layout === b.layout,
),
)
.subscribe((state) => {
lotusSendToHost(LotusWidgetActions.ControlsState, state);
});
return (): void => {
sub.unsubscribe();
w.lazyActions.off(LotusWidgetActions.SetLayout, onSetLayout);
w.lazyActions.off(LotusWidgetActions.OpenSettings, onOpenSettings);
w.lazyActions.off(LotusWidgetActions.ToggleReactions, onToggleReactions);
};
}
+25 -43
View File
@@ -7,13 +7,11 @@ Please see LICENSE in the repository root for full details.
import { EventEmitter } from "events";
import { afterEach, beforeEach, expect, test, vi } from "vitest";
import { of } from "rxjs";
import { Track } from "livekit-client";
import { type CallViewModel } from "../state/CallViewModel/CallViewModel";
import { setAudioEnabled$ } from "../controls";
import { startLotusDeafen } from "./lotusDeafen";
import { LotusWidgetActions } from "./lotusActions";
import { setScreenshareAudioMuted$ } from "./lotusScreenshareAudio";
const lazyActions = new EventEmitter();
@@ -27,18 +25,6 @@ vi.mock("../widget", () => ({
},
}));
/** Minimal CallViewModel stub: no connections, so no livekit rooms. */
function mockVm(participants: unknown[] = []): CallViewModel {
const livekitRoom = {
remoteParticipants: new Map(participants.map((p, i) => [String(i), p])),
on: vi.fn(),
off: vi.fn(),
};
return {
allConnections$: of({ getConnections: () => [{ livekitRoom }] }),
} as unknown as CallViewModel;
}
function send(data: unknown): void {
lazyActions.emit(LotusWidgetActions.SetDeafen, {
detail: { data },
@@ -60,7 +46,7 @@ afterEach(() => {
});
test("deafen mutes, and undeafen unmutes, EC's global audio output", () => {
const stop = startLotusDeafen(mockVm());
const stop = startLotusDeafen();
send({ deafened: true, screenshareAudioMuted: false });
expect(emissions).toEqual([false]);
@@ -73,7 +59,7 @@ test("deafen mutes, and undeafen unmutes, EC's global audio output", () => {
});
test("re-sending the same state is idempotent (host resendForkState)", () => {
const stop = startLotusDeafen(mockVm());
const stop = startLotusDeafen();
send({ deafened: true, screenshareAudioMuted: false });
send({ deafened: true, screenshareAudioMuted: false });
@@ -86,7 +72,7 @@ test("re-sending the same state is idempotent (host resendForkState)", () => {
});
test("undeafen does not re-enable audio the user had muted themselves", () => {
const stop = startLotusDeafen(mockVm());
const stop = startLotusDeafen();
// The user mutes all audio through EC's own control first.
setAudioEnabled$.next(false);
@@ -101,48 +87,44 @@ test("undeafen does not re-enable audio the user had muted themselves", () => {
expect(emissions).toEqual([false]);
});
test("deafen never touches per-participant volume", () => {
const participant = { setVolume: vi.fn() };
const stop = startLotusDeafen(mockVm([participant]));
test("deafen never touches the screenshare-audio flag", () => {
const stop = startLotusDeafen();
send({ deafened: true, screenshareAudioMuted: false });
send({ deafened: false, screenshareAudioMuted: false });
expect(participant.setVolume).not.toHaveBeenCalled();
expect(setScreenshareAudioMuted$.value).toBe(false);
stop();
});
test("screenshare-audio mute is applied per source and only undone for participants we muted", () => {
const participant = { setVolume: vi.fn() };
const stop = startLotusDeafen(mockVm([participant]));
test("screenshare-audio mute is a global flag the renderer reads, reset on teardown", () => {
const seen: boolean[] = [];
const flagSub = setScreenshareAudioMuted$.subscribe((v) => seen.push(v));
const stop = startLotusDeafen();
// Not muted yet: no volume writes at all.
// Not muted yet: nothing pushed beyond the BehaviorSubject's initial value.
send({ deafened: false, screenshareAudioMuted: false });
expect(participant.setVolume).not.toHaveBeenCalled();
expect(seen).toEqual([false]);
send({ deafened: false, screenshareAudioMuted: true });
expect(participant.setVolume).toHaveBeenCalledWith(
0,
Track.Source.ScreenShareAudio,
);
expect(setScreenshareAudioMuted$.value).toBe(true);
// Re-applying the same state (host resend after reconnect) is a no-op.
send({ deafened: false, screenshareAudioMuted: true });
expect(seen).toEqual([false, true]);
participant.setVolume.mockClear();
send({ deafened: false, screenshareAudioMuted: false });
expect(participant.setVolume).toHaveBeenCalledWith(
1,
Track.Source.ScreenShareAudio,
);
// Releasing again writes nothing: we no longer own that participant.
participant.setVolume.mockClear();
send({ deafened: false, screenshareAudioMuted: false });
expect(participant.setVolume).not.toHaveBeenCalled();
// Deafen + undeafen while screenshare audio is muted leaves it muted.
send({ deafened: true, screenshareAudioMuted: true });
send({ deafened: false, screenshareAudioMuted: true });
expect(setScreenshareAudioMuted$.value).toBe(true);
// Leaving the call clears it so the next call starts clean.
stop();
expect(setScreenshareAudioMuted$.value).toBe(false);
flagSub.unsubscribe();
});
test("a partial payload only moves the flag it names", () => {
const stop = startLotusDeafen(mockVm());
const stop = startLotusDeafen();
send({ deafened: true });
expect(emissions).toEqual([false]);
+17 -71
View File
@@ -5,19 +5,13 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import {
type RemoteParticipant,
type Room as LivekitRoom,
RoomEvent,
Track,
} from "livekit-client";
import { logger } from "matrix-js-sdk/lib/logger";
import { type IWidgetApiRequest } from "matrix-widget-api";
import { type CallViewModel } from "../state/CallViewModel/CallViewModel";
import { setAudioEnabled$ } from "../controls";
import { widget } from "../widget";
import { LotusWidgetActions } from "./lotusActions";
import { setScreenshareAudioMuted$ } from "./lotusScreenshareAudio";
/**
* Handle the host's `io.lotus.set_deafen` toWidget action, replacing cinny's
@@ -41,22 +35,23 @@ import { LotusWidgetActions } from "./lotusActions";
* volume/mute state.
*
* `screenshareAudioMuted` is a NARROWER, independent host control (drop shared
* tab/game audio while still hearing voices). EC has no global per-source mute,
* so that one still has to go through
* `RemoteParticipant.setVolume(volume, Track.Source.ScreenShareAudio)` — whose
* verified signature in livekit-client ^2.18.1 is
* `setVolume(volume, source?: Track.Source.Microphone | Track.Source.ScreenShareAudio)`.
* It is re-applied to late joiners via `RoomEvent.ParticipantConnected`, and we
* only ever restore participants we muted ourselves. Known limitation: EC's own
* screenshare volume slider writes the same `volumeMap`, so a user who moves
* that slider while screenshare audio is host-muted wins; the mic path (the
* actual deafen) is no longer affected by that race at all.
* tab/game audio while still hearing voices). It goes through the same
* mechanism as deafen — a global behavior (`muteScreenshareAudio$`) that
* `LivekitRoomAudioRenderer` turns into the `muted` prop of every
* `Track.Source.ScreenShareAudio` element — rather than
* `RemoteParticipant.setVolume(0, ScreenShareAudio)`: EC's own
* `createVolumeControls` writes volume 1 through that very setter whenever a
* new screenshare media item resolves, so a sharer who stopped and re-shared
* (or a late joiner's share) came back at full volume while the host's button
* still said "Unmute Screenshare Audio".
*
* Undeafen restores the user's OWN output-enabled state as it was before the
* deafen (and never touches the `mute-all-audio` setting), so a user who had
* already muted all audio themselves stays muted.
*
* State is closure-scoped (per invocation, matching the sibling lotus modules).
* State is closure-scoped (per invocation, matching the sibling lotus modules);
* the screenshare-audio flag additionally lives in `setScreenshareAudioMuted$`
* so the renderer can read it, and is reset on teardown.
* Applying the same state twice is a no-op, so the host's
* `CallControl.resendForkState()` after a reconnect is safe. The host re-sends
* the current state on every call join (CallControl.forceState), so a fresh
@@ -64,7 +59,7 @@ import { LotusWidgetActions } from "./lotusActions";
*
* No effect unless the host sends the action. Returns a teardown function.
*/
export function startLotusDeafen(vm: CallViewModel): () => void {
export function startLotusDeafen(): () => void {
const w = widget;
if (!w) return () => undefined;
@@ -107,53 +102,6 @@ export function startLotusDeafen(vm: CallViewModel): () => void {
}
});
// Participants whose screenshare audio WE turned down, so undoing the host's
// screenshare mute never writes a volume to anyone else.
const screenshareMuted = new Set<RemoteParticipant>();
const applyToParticipant = (p: RemoteParticipant): void => {
if (screenshareAudioMuted) {
p.setVolume(0, Track.Source.ScreenShareAudio);
screenshareMuted.add(p);
} else if (screenshareMuted.delete(p)) {
p.setVolume(1, Track.Source.ScreenShareAudio);
}
};
const applyToRoom = (room: LivekitRoom): void =>
room.remoteParticipants.forEach(applyToParticipant);
// Per-room ParticipantConnected listeners, so LATE JOINERS pick up the
// current screenshare-audio mute the moment they connect. Drive off the
// local participant's connection(s), not `livekitRoomItems$` — that stream is
// empty until a matrix-validated REMOTE member resolves, so listeners would
// be attached too late for the first joiner (the sibling lotus modules all
// use `allConnections$` for the same reason).
const roomListeners = new Map<LivekitRoom, () => void>();
let rooms: LivekitRoom[] = [];
const sub = vm.allConnections$.subscribe((data) => {
const next = data.getConnections().map((c) => c.livekitRoom);
rooms = next;
// Detach listeners for rooms that went away.
for (const [room, off] of roomListeners) {
if (!next.includes(room)) {
off();
roomListeners.delete(room);
}
}
// Attach to new rooms + apply the current state to their participants.
for (const room of next) {
if (!roomListeners.has(room)) {
room.on(RoomEvent.ParticipantConnected, applyToParticipant);
roomListeners.set(room, () =>
room.off(RoomEvent.ParticipantConnected, applyToParticipant),
);
applyToRoom(room);
}
}
});
const handler = (ev: CustomEvent<IWidgetApiRequest>): void => {
w.api.transport.reply(ev.detail, {});
const data = ev.detail.data as
@@ -168,20 +116,18 @@ export function startLotusDeafen(vm: CallViewModel): () => void {
`[lotus] set_deafen: deafened=${deafened} screenshareAudioMuted=${screenshareAudioMuted}`,
);
applyGlobalMute();
rooms.forEach(applyToRoom);
if (setScreenshareAudioMuted$.value !== screenshareAudioMuted)
setScreenshareAudioMuted$.next(screenshareAudioMuted);
};
w.lazyActions.on(LotusWidgetActions.SetDeafen, handler);
return () => {
sub.unsubscribe();
for (const off of roomListeners.values()) off();
roomListeners.clear();
// Leave the user's own output state as they had it before deafen.
if (deafened) {
deafened = false;
applyGlobalMute();
}
screenshareMuted.clear();
if (setScreenshareAudioMuted$.value) setScreenshareAudioMuted$.next(false);
audioSub.unsubscribe();
w.lazyActions.off(LotusWidgetActions.SetDeafen, handler);
};
+37
View File
@@ -0,0 +1,37 @@
/*
Copyright 2026 Lotus Guild
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { describe, expect, it } from "vitest";
import { MicLevelQuantizer } from "./lotusMicLevel";
describe("MicLevelQuantizer", () => {
it("maps RMS to 0–3 bars", () => {
expect(new MicLevelQuantizer().push(0.001)).toBe(0);
expect(new MicLevelQuantizer().push(0.008)).toBe(1);
expect(new MicLevelQuantizer().push(0.02)).toBe(2);
expect(new MicLevelQuantizer().push(0.2)).toBe(3);
});
it("rises at once and falls one bar per sample", () => {
const q = new MicLevelQuantizer();
expect(q.push(0.2)).toBe(3);
expect(q.push(0)).toBe(2);
expect(q.push(0)).toBe(1);
expect(q.push(0.02)).toBe(2);
expect(q.push(0)).toBe(1);
expect(q.push(0)).toBe(0);
expect(q.push(0)).toBe(0);
});
it("reset drops straight to 0", () => {
const q = new MicLevelQuantizer();
q.push(0.2);
q.reset();
expect(q.push(0)).toBe(0);
});
});
+192
View File
@@ -0,0 +1,192 @@
/*
Copyright 2026 Lotus Guild
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import {
type LocalTrackPublication,
type Room as LivekitRoom,
RoomEvent,
Track,
} from "livekit-client";
import { Observable, type Subscription, share, switchMap } from "rxjs";
import { distinctUntilChanged, map } from "rxjs/operators";
import { type CallViewModel } from "../state/CallViewModel/CallViewModel";
import { widget } from "../widget";
import { LotusWidgetActions, lotusFlag, lotusSendToHost } from "./lotusWidget";
/**
* [lotus #146] One local mic sampler shared by the host's mic level meter
* (`io.lotus.mic_level`) and "talking while muted" (#37, lotusMutedSpeech).
* It taps a CLONE of the published mic track (the post-processor track when
* the in-source denoiser is active, so what's measured is what's sent) and
* reads RMS at ~10 Hz while a mic track is published, muted or not.
* Local only: nothing here reaches other participants.
*/
const SAMPLE_MS = 100;
export interface LocalMicSample {
rms: number;
/** The mic is published but muted (the clone still hears it). */
muted: boolean;
}
const micPublication = (
room: LivekitRoom,
): { track: MediaStreamTrack; muted: boolean } | null => {
const pub: LocalTrackPublication | undefined =
room.localParticipant.getTrackPublication(Track.Source.Microphone);
const track = pub?.track?.mediaStreamTrack;
return track && track.readyState === "live"
? { track, muted: pub?.isMuted ?? false }
: null;
};
const samplers = new WeakMap<
CallViewModel,
Observable<LocalMicSample | null>
>();
/**
* RMS samples of the local mic, or `null` while no mic track is published.
* Shared per call view model, so the meter and the muted-speech detector use
* one AudioContext between them.
*/
export function observeLocalMicSample$(
vm: CallViewModel,
): Observable<LocalMicSample | null> {
const cached = samplers.get(vm);
if (cached) return cached;
const sampler$ = vm.allConnections$.pipe(
switchMap(
(data) =>
new Observable<LocalMicSample | null>((subscriber) => {
const rooms = data.getConnections().map((c) => c.livekitRoom);
let ctx: AudioContext | null = null;
let clone: MediaStreamTrack | null = null;
let timer: ReturnType<typeof setInterval> | undefined;
let tapped: MediaStreamTrack | null = null;
let muted = false;
const stopTap = (): void => {
if (timer !== undefined) clearInterval(timer);
timer = undefined;
clone?.stop();
clone = null;
void ctx?.close().catch(() => undefined);
ctx = null;
tapped = null;
subscriber.next(null);
};
const startTap = (source: MediaStreamTrack): void => {
try {
clone = source.clone();
// Muting disables the published track; the clone must still hear.
clone.enabled = true;
ctx = new AudioContext();
const analyser = ctx.createAnalyser();
analyser.fftSize = 1024;
ctx
.createMediaStreamSource(new MediaStream([clone]))
.connect(analyser);
const buf = new Float32Array(analyser.fftSize);
tapped = source;
timer = setInterval(() => {
analyser.getFloatTimeDomainData(buf);
let sum = 0;
for (let i = 0; i < buf.length; i += 1) sum += buf[i] * buf[i];
subscriber.next({ rms: Math.sqrt(sum / buf.length), muted });
}, SAMPLE_MS);
} catch {
stopTap();
}
};
const reconcile = (): void => {
const pub =
rooms.map(micPublication).find((p) => p !== null) ?? null;
muted = pub?.muted ?? false;
const track = pub?.track ?? null;
if (track === tapped) return;
if (tapped) stopTap();
if (track) startTap(track);
};
const events = [
RoomEvent.TrackMuted,
RoomEvent.TrackUnmuted,
RoomEvent.LocalTrackPublished,
RoomEvent.LocalTrackUnpublished,
RoomEvent.Reconnected,
RoomEvent.Disconnected,
] as const;
rooms.forEach((room) =>
events.forEach((ev) => room.on(ev, reconcile)),
);
subscriber.next(null);
reconcile();
return (): void => {
rooms.forEach((room) =>
events.forEach((ev) => room.off(ev, reconcile)),
);
if (tapped) stopTap();
};
}),
),
share(),
);
samplers.set(vm, sampler$);
return sampler$;
}
/** RMS at which each bar lights: ≈ −46, −36 and −26 dBFS. */
export const BAR_THRESHOLDS = [0.005, 0.015, 0.05] as const;
/**
* Quantise RMS to 0–3 bars with a little hysteresis: rises at once, falls one
* bar per sample, so the meter doesn't flicker between words. Unit-tested.
*/
export class MicLevelQuantizer {
private bars = 0;
public reset(): void {
this.bars = 0;
}
public push(rms: number): number {
const target = BAR_THRESHOLDS.filter((t) => rms >= t).length;
this.bars = target >= this.bars ? target : this.bars - 1;
return this.bars;
}
}
/**
* Send the host `io.lotus.mic_level { bars }` (0–3) whenever the quantised
* level changes; 0 while muted or with no mic. At most one message per sample
* (10 Hz) and none during steady silence. Opt-in with the rest of the host
* state stream (`lotusCallState=1`). Returns a teardown function.
*/
export function startLotusMicLevel(vm: CallViewModel): () => void {
if (!lotusFlag("lotusCallState") || !widget) return (): void => undefined;
const quantizer = new MicLevelQuantizer();
const sub: Subscription = observeLocalMicSample$(vm)
.pipe(
map((sample) => {
if (!sample || sample.muted) {
quantizer.reset();
return 0;
}
return quantizer.push(sample.rms);
}),
distinctUntilChanged(),
)
.subscribe((bars) => {
lotusSendToHost(LotusWidgetActions.MicLevel, { bars });
});
return (): void => sub.unsubscribe();
}
+27
View File
@@ -0,0 +1,27 @@
/*
Copyright 2026 Lotus Guild
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { expect, test } from "vitest";
import { MutedSpeechGate } from "./lotusMutedSpeech";
test("needs 300 ms of voice to switch on and 800 ms of quiet to switch off", () => {
const g = new MutedSpeechGate(0.015);
expect(g.push(0.1)).toBe(false);
expect(g.push(0.1)).toBe(false);
expect(g.push(0.1)).toBe(true);
// a short dip does not drop it
for (let i = 0; i < 7; i++) expect(g.push(0.0)).toBe(true);
expect(g.push(0.0)).toBe(false);
// one loud sample after quiet does not re-trigger
expect(g.push(0.2)).toBe(false);
});
test("keyboard-level noise below the threshold never triggers", () => {
const g = new MutedSpeechGate(0.015);
for (let i = 0; i < 50; i++) expect(g.push(0.01)).toBe(false);
});
+84
View File
@@ -0,0 +1,84 @@
/*
Copyright 2026 Lotus Guild
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import {
type Observable,
distinctUntilChanged,
map,
scan,
startWith,
} from "rxjs";
import { type CallViewModel } from "../state/CallViewModel/CallViewModel";
import { type LocalMicSample, observeLocalMicSample$ } from "./lotusMicLevel";
/**
* [lotus #37] "Talking while muted" detection for the LOCAL participant.
*
* While the mic is muted LiveKit keeps the capture alive and merely disables
* the published MediaStreamTrack, so VAD/`speaking` is false and nothing
* tells the user they are talking into a muted mic. This taps a CLONE of the
* published track (the post-processor track when the in-source denoiser is
* active, so keyboard noise doesn't count), samples RMS at ~10 Hz and emits a
* debounced boolean. Local-only —
* the flag rides `io.lotus.call_state` to the host and never reaches other
* participants.
*/
export const MUTED_SPEECH_RMS = 0.015; // ≈ −36 dBFS; normal speech into a headset is 0.05–0.3
const ON_SAMPLES = 3; // 300 ms of voice before we say "talking"
const OFF_SAMPLES = 8; // 800 ms of quiet before we drop it
/** Pure hysteresis gate over successive RMS samples (unit-tested). */
export class MutedSpeechGate {
private above = 0;
private below = 0;
private on = false;
public constructor(private readonly threshold = MUTED_SPEECH_RMS) {}
public push(rms: number): boolean {
if (rms >= this.threshold) {
this.above += 1;
this.below = 0;
if (!this.on && this.above >= ON_SAMPLES) this.on = true;
} else {
this.below += 1;
this.above = 0;
if (this.on && this.below >= OFF_SAMPLES) this.on = false;
}
return this.on;
}
public get value(): boolean {
return this.on;
}
}
/**
* Emits while the local mic is muted and voice is detected on it. Emits
* `false` whenever the mic is unmuted, unpublished or the connection changes.
* [lotus #146] Fed by the shared local mic sampler (lotusMicLevel.ts), which
* also drives the host's mic level meter while unmuted.
*/
export function observeSpeakingWhileMuted$(
vm: CallViewModel,
): Observable<boolean> {
return observeLocalMicSample$(vm).pipe(
scan((gate: MutedSpeechGate | null, sample: LocalMicSample | null) => {
if (!sample?.muted) return null;
const g = gate ?? new MutedSpeechGate();
g.push(sample.rms);
return g;
}, null),
map((gate) => gate?.value ?? false),
startWith(false),
distinctUntilChanged(),
);
}
+27
View File
@@ -0,0 +1,27 @@
/*
Copyright 2026 Lotus Guild
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { BehaviorSubject } from "rxjs";
import { globalScope } from "../state/ObservableScope";
/**
* Whether the host has muted remote screenshare AUDIO (`io.lotus.set_deafen`
* `screenshareAudioMuted`). Consumed by `LivekitRoomAudioRenderer` exactly the
* way `muteAllAudio$` is: it becomes the `muted` prop of every
* `Track.Source.ScreenShareAudio` element, so it survives re-renders, applies
* to tracks that are published LATER (a sharer stopping and re-sharing, a late
* joiner) and never fights EC's per-tile volume controls — which was the
* failure mode of the old `RemoteParticipant.setVolume(0, ScreenShareAudio)`
* approach: `createVolumeControls` writes its own volume (1) through the same
* sink the moment a new screenshare media item appears, un-muting it.
*/
export const setScreenshareAudioMuted$ = new BehaviorSubject(false);
export const muteScreenshareAudio$ = globalScope.behavior(
setScreenshareAudioMuted$,
);
+171
View File
@@ -0,0 +1,171 @@
/*
Copyright 2026 Lotus Guild
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { RoomEvent, Track } from "livekit-client";
import { BehaviorSubject } from "rxjs";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { type CallViewModel } from "../state/CallViewModel/CallViewModel";
import {
ALONE_MS,
NO_FRAMES_MS,
startLotusScreenshareWatch,
} from "./lotusScreenshareWatch";
const sent: unknown[] = [];
vi.mock("../widget", () => ({ widget: { api: {} } }));
vi.mock("./lotusWidget", () => ({
lotusSendToHost: (action: string, data: unknown): boolean => {
sent.push({ action, data });
return true;
},
}));
class FakeTrack extends EventTarget {}
const makeRoom = (): {
room: Record<string, unknown>;
emit: (event: string, ...args: unknown[]) => void;
remote: Map<string, unknown>;
} => {
const handlers = new Map<string, ((...args: unknown[]) => void)[]>();
const remote = new Map<string, unknown>();
const room = {
remoteParticipants: remote,
localParticipant: { trackPublications: new Map() },
on: (event: string, h: (...args: unknown[]) => void): void => {
handlers.set(event, [...(handlers.get(event) ?? []), h]);
},
off: (event: string, h: (...args: unknown[]) => void): void => {
handlers.set(
event,
(handlers.get(event) ?? []).filter((x) => x !== h),
);
},
};
return {
room,
remote,
emit: (event, ...args): void => {
(handlers.get(event) ?? []).forEach((h) => h(...args));
},
};
};
describe("startLotusScreenshareWatch", () => {
let clock = 0;
beforeEach(() => {
sent.length = 0;
clock = 1_000_000;
vi.useFakeTimers();
});
afterEach(() => vi.useRealTimers());
const start = (): ReturnType<typeof makeRoom> & { stop: () => void } => {
const fake = makeRoom();
const connections = new BehaviorSubject({
getConnections: () => [{ livekitRoom: fake.room }],
});
const stop = startLotusScreenshareWatch(
{ allConnections$: connections } as unknown as CallViewModel,
() => clock,
);
return { ...fake, stop };
};
it("reports a share whose track ended", () => {
const { emit, stop } = start();
const mst = new FakeTrack();
emit(RoomEvent.LocalTrackPublished, {
source: Track.Source.ScreenShare,
track: { mediaStreamTrack: mst },
});
mst.dispatchEvent(new Event("ended"));
expect(sent).toEqual([
{ action: "io.lotus.screenshare_notice", data: { kind: "ended" } },
]);
stop();
});
it("reports no frames after a sustained mute, once, and not after an unmute", () => {
const { emit, stop } = start();
const mst = new FakeTrack();
emit(RoomEvent.LocalTrackPublished, {
source: Track.Source.ScreenShare,
track: { mediaStreamTrack: mst },
});
mst.dispatchEvent(new Event("mute"));
vi.advanceTimersByTime(NO_FRAMES_MS / 2);
mst.dispatchEvent(new Event("unmute"));
vi.advanceTimersByTime(NO_FRAMES_MS);
expect(sent).toEqual([]);
mst.dispatchEvent(new Event("mute"));
vi.advanceTimersByTime(NO_FRAMES_MS);
mst.dispatchEvent(new Event("mute"));
vi.advanceTimersByTime(NO_FRAMES_MS);
expect(sent).toEqual([
{ action: "io.lotus.screenshare_notice", data: { kind: "no-frames" } },
]);
stop();
});
it("nudges after 30 min of sharing with nobody else, never while others are present", () => {
const { emit, remote, stop } = start();
emit(RoomEvent.LocalTrackPublished, {
source: Track.Source.ScreenShare,
track: { mediaStreamTrack: new FakeTrack() },
});
remote.set("bob", {});
clock += ALONE_MS + 60_000;
vi.advanceTimersByTime(60_000);
expect(sent).toEqual([]);
remote.clear();
vi.advanceTimersByTime(60_000);
vi.advanceTimersByTime(60_000);
expect(sent).toEqual([
{ action: "io.lotus.screenshare_notice", data: { kind: "alone" } },
]);
stop();
});
it("reports ended from the unpublish when LiveKit's handler ran first", () => {
const { emit, stop } = start();
const share = new FakeTrack() as FakeTrack & { readyState: string };
share.readyState = "ended";
const pub = {
source: Track.Source.ScreenShare,
track: { mediaStreamTrack: share },
};
emit(RoomEvent.LocalTrackPublished, pub);
emit(RoomEvent.LocalTrackUnpublished, pub);
share.dispatchEvent(new Event("ended"));
expect(sent).toEqual([
{ action: "io.lotus.screenshare_notice", data: { kind: "ended" } },
]);
stop();
});
it("ignores non-screenshare publications and stops watching on unpublish", () => {
const { emit, stop } = start();
const cam = new FakeTrack();
emit(RoomEvent.LocalTrackPublished, {
source: Track.Source.Camera,
track: { mediaStreamTrack: cam },
});
cam.dispatchEvent(new Event("ended"));
const share = new FakeTrack();
const pub = {
source: Track.Source.ScreenShare,
track: { mediaStreamTrack: share },
};
emit(RoomEvent.LocalTrackPublished, pub);
emit(RoomEvent.LocalTrackUnpublished, pub);
share.dispatchEvent(new Event("ended"));
expect(sent).toEqual([]);
stop();
});
});
+136
View File
@@ -0,0 +1,136 @@
/*
Copyright 2026 Lotus Guild
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import {
type LocalTrackPublication,
type Room as LivekitRoom,
RoomEvent,
Track,
} from "livekit-client";
import { type CallViewModel } from "../state/CallViewModel/CallViewModel";
import { widget } from "../widget";
import { LotusWidgetActions } from "./lotusActions";
import { lotusSendToHost } from "./lotusWidget";
export type ScreenshareNoticeKind = "ended" | "no-frames" | "alone";
/** A shared window minimised/occluded this long with no frames → tell the host. */
export const NO_FRAMES_MS = 15_000;
/** Sharing this long with nobody else in the call → one "still sharing?" nudge. */
export const ALONE_MS = 30 * 60_000;
const ALONE_CHECK_MS = 60_000;
/**
* [lotus #39] Watch the local screenshare and tell the host about the two
* cases people miss: the share went black/ended (window closed or minimised)
* and a long share with nobody else in the call. Detection only — the host
* renders the notices. Each notice fires at most once per share.
*/
export function startLotusScreenshareWatch(
vm: CallViewModel,
now: () => number = () => Date.now(),
): () => void {
if (!widget) return () => undefined;
const notify = (kind: ScreenshareNoticeKind): void => {
lotusSendToHost(LotusWidgetActions.ScreenshareNotice, { kind });
};
const perRoom = new Map<LivekitRoom, () => void>();
const attach = (room: LivekitRoom): void => {
let cleanupTrack: (() => void) | undefined;
let endedHook: (() => void) | undefined;
const watchPublication = (pub: LocalTrackPublication): void => {
if (pub.source !== Track.Source.ScreenShare) return;
cleanupTrack?.();
const mst = pub.track?.mediaStreamTrack;
const startedAt = now();
let noFramesTimer: ReturnType<typeof setTimeout> | undefined;
let sentNoFrames = false;
let sentAlone = false;
let sentEnded = false;
const onEnded = (): void => {
if (sentEnded) return;
sentEnded = true;
notify("ended");
};
const onMute = (): void => {
if (sentNoFrames) return;
noFramesTimer = setTimeout(() => {
sentNoFrames = true;
notify("no-frames");
}, NO_FRAMES_MS);
};
const onUnmute = (): void => {
if (noFramesTimer !== undefined) clearTimeout(noFramesTimer);
noFramesTimer = undefined;
};
endedHook = onEnded;
mst?.addEventListener("ended", onEnded);
mst?.addEventListener("mute", onMute);
mst?.addEventListener("unmute", onUnmute);
const aloneTimer = setInterval(() => {
if (sentAlone) return;
if (room.remoteParticipants.size > 0) return;
if (now() - startedAt < ALONE_MS) return;
sentAlone = true;
notify("alone");
}, ALONE_CHECK_MS);
cleanupTrack = (): void => {
mst?.removeEventListener("ended", onEnded);
mst?.removeEventListener("mute", onMute);
mst?.removeEventListener("unmute", onUnmute);
if (noFramesTimer !== undefined) clearTimeout(noFramesTimer);
clearInterval(aloneTimer);
cleanupTrack = undefined;
endedHook = undefined;
};
};
// LiveKit's own `ended` handler runs first and unpublishes the share; a
// listener removed during that dispatch never fires, so decide from the
// track's state here as well.
const onUnpublished = (pub: LocalTrackPublication): void => {
if (pub.source !== Track.Source.ScreenShare) return;
if (pub.track?.mediaStreamTrack?.readyState === "ended") endedHook?.();
cleanupTrack?.();
};
room.on(RoomEvent.LocalTrackPublished, watchPublication);
room.on(RoomEvent.LocalTrackUnpublished, onUnpublished);
room.localParticipant.trackPublications.forEach(watchPublication);
perRoom.set(room, () => {
cleanupTrack?.();
room.off(RoomEvent.LocalTrackPublished, watchPublication);
room.off(RoomEvent.LocalTrackUnpublished, onUnpublished);
});
};
const sub = vm.allConnections$.subscribe((data) => {
const rooms = data.getConnections().map((c) => c.livekitRoom);
for (const [room, off] of perRoom) {
if (!rooms.includes(room)) {
off();
perRoom.delete(room);
}
}
for (const room of rooms) if (!perRoom.has(room)) attach(room);
});
return () => {
sub.unsubscribe();
for (const off of perRoom.values()) off();
perRoom.clear();
};
}
+61
View File
@@ -0,0 +1,61 @@
/*
Copyright 2026 Lotus Guild
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { beforeEach, describe, expect, it } from "vitest";
import {
MAX_REMEMBERED,
getRememberedVolume,
rememberVolume,
} from "./lotusVolumeMemory";
describe("lotusVolumeMemory", () => {
beforeEach(() => localStorage.clear());
it("defaults to 1 and remembers a set volume", () => {
expect(getRememberedVolume("@bob:x")).toBe(1);
rememberVolume("@bob:x", 0.4);
expect(getRememberedVolume("@bob:x")).toBe(0.4);
});
it("forgets the entry when set back to 1", () => {
rememberVolume("@bob:x", 0.4);
rememberVolume("@bob:x", 1);
expect(getRememberedVolume("@bob:x")).toBe(1);
expect(localStorage.getItem("lotus-per-user-volume")).toBe("{}");
});
it("keeps only the most recently set entries", () => {
for (let i = 0; i < MAX_REMEMBERED + 5; i++)
rememberVolume(`@u${i}:x`, 0.5);
expect(getRememberedVolume("@u0:x")).toBe(1);
expect(getRememberedVolume("@u4:x")).toBe(1);
expect(getRememberedVolume("@u5:x")).toBe(0.5);
expect(getRememberedVolume(`@u${MAX_REMEMBERED + 4}:x`)).toBe(0.5);
});
it("re-setting an old entry makes it recent", () => {
for (let i = 0; i < MAX_REMEMBERED; i++) rememberVolume(`@u${i}:x`, 0.5);
rememberVolume("@u0:x", 0.7);
rememberVolume("@new:x", 0.3);
expect(getRememberedVolume("@u0:x")).toBe(0.7);
expect(getRememberedVolume("@u1:x")).toBe(1);
});
it("ignores junk in storage and invalid volumes", () => {
localStorage.setItem(
"lotus-per-user-volume",
JSON.stringify({ "@a:x": "loud", "@b:x": 0.2 }),
);
expect(getRememberedVolume("@a:x")).toBe(1);
expect(getRememberedVolume("@b:x")).toBe(0.2);
rememberVolume("@c:x", Number.NaN);
expect(getRememberedVolume("@c:x")).toBe(1);
localStorage.setItem("lotus-per-user-volume", "not json");
expect(getRememberedVolume("@b:x")).toBe(1);
});
});
+65
View File
@@ -0,0 +1,65 @@
/*
Copyright 2026 Lotus Guild
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
/**
* [element-call #36] Remember the per-participant volume slider across calls,
* reloads and reconnects. Keyed by Matrix user id (not device), so "Bob is
* loud" sticks when Bob switches devices. Local to this browser, never synced.
* Only voice is remembered; screenshare audio stays per-share.
*/
const STORAGE_KEY = "lotus-per-user-volume";
/** Most recently set entries kept; older ones are dropped. */
export const MAX_REMEMBERED = 50;
type VolumeMap = Record<string, number>;
const isVolume = (v: unknown): v is number =>
typeof v === "number" && Number.isFinite(v) && v >= 0 && v <= 4;
function load(): VolumeMap {
try {
const raw = localStorage.getItem(STORAGE_KEY);
if (!raw) return {};
const parsed: unknown = JSON.parse(raw);
if (typeof parsed !== "object" || parsed === null) return {};
const out: VolumeMap = {};
for (const [k, v] of Object.entries(parsed)) if (isVolume(v)) out[k] = v;
return out;
} catch {
return {};
}
}
function save(map: VolumeMap): void {
try {
localStorage.setItem(STORAGE_KEY, JSON.stringify(map));
} catch {
// Storage unavailable (private mode, quota): the slider just won't stick.
}
}
/** The remembered volume for `userId`, or 1 (100 %) if none. */
export function getRememberedVolume(userId: string): number {
return load()[userId] ?? 1;
}
/**
* Remember `volume` for `userId`. 1 (the default) forgets the entry. The entry
* moves to the end so the map is in least-recently-set order, trimmed to
* MAX_REMEMBERED.
*/
export function rememberVolume(userId: string, volume: number): void {
if (!isVolume(volume)) return;
const map = load();
delete map[userId];
if (volume !== 1) map[userId] = volume;
const keys = Object.keys(map);
for (const k of keys.slice(0, Math.max(0, keys.length - MAX_REMEMBERED)))
delete map[k];
save(map);
}
+24 -7
View File
@@ -365,17 +365,18 @@ describe("LiveKit ConnectionError variants", () => {
expectedReason: "InternalError",
},
])(
"should display LiveKit $name error correctly",
"should explain the LiveKit $name error and offer a retry",
async ({ error, expectedReason }) => {
const TestComponent = (): ReactNode => {
throw new LivekitConnectionError(error);
};
const recoveryActionHandler = vi.fn(async () => Promise.resolve());
const { asFragment } = render(
<BrowserRouter>
<GroupCallErrorBoundary
onError={vi.fn()}
recoveryActionHandler={vi.fn()}
recoveryActionHandler={recoveryActionHandler}
widget={null}
>
<TestComponent />
@@ -383,12 +384,28 @@ describe("LiveKit ConnectionError variants", () => {
</BrowserRouter>,
);
// Check title
await screen.findByText("Failed to connect to Livekit server");
await screen.findByText("Couldn’t connect to voice");
// Check that reason is displayed in the description
expect(screen.getByText(/Reason:/i)).toBeInTheDocument();
expect(screen.getByText(expectedReason)).toBeInTheDocument();
// [lotus] Plain-language guidance instead of "(Reason: X)".
expect(
screen.getByText(
expectedReason === "NotAllowed"
? /The voice server didn’t let you in/
: /Your device couldn’t reach the voice server/,
),
).toBeInTheDocument();
// The raw reason is still there, under Technical details.
expect(screen.getByText("Technical details")).toBeInTheDocument();
expect(
screen.getByText(new RegExp(`Reason: ${expectedReason}`), {
selector: "pre",
}),
).toBeInTheDocument();
// Try again re-enters the call.
await userEvent.click(screen.getByRole("button", { name: "Try again" }));
expect(recoveryActionHandler).toHaveBeenCalledWith("reconnect");
expect(asFragment()).toMatchSnapshot();
},
+19 -5
View File
@@ -30,6 +30,8 @@ import {
ElementCallError,
ErrorCategory,
ErrorCode,
LivekitConnectionError,
PeerConnectionTimeoutError,
UnknownCallError,
} from "../utils/errors.ts";
import { FullScreenView } from "../FullScreenView.tsx";
@@ -78,10 +80,22 @@ const ErrorPage: FC<ErrorPageProps> = ({
label: t("call_ended_view.reconnect_button"),
onClick: () => void recoveryActionHandler("reconnect"),
});
} else if (
// [lotus] A failed connect is often transient (busy server, flaky Wi-Fi);
// offer the same re-enter path instead of a dead end.
error instanceof LivekitConnectionError ||
error instanceof PeerConnectionTimeoutError
) {
actions.push({
label: t("error.try_again"),
onClick: () => void recoveryActionHandler("reconnect"),
});
}
const technicalError =
error.cause instanceof MatrixError ? error.cause : null;
error.cause instanceof MatrixError
? error.cause.message
: (error.technicalDetails ?? null);
return (
<FullScreenView>
@@ -124,15 +138,15 @@ const ErrorPage: FC<ErrorPageProps> = ({
<summary className={styles.technicalDetailsSummary}>
{t("technical_details")}
</summary>
<pre className={styles.technicalDetailsPre}>
{technicalError.message}
</pre>
<pre className={styles.technicalDetailsPre}>{technicalError}</pre>
</details>
) : null}
{actions &&
actions.map((action, index) => (
<Button
kind="secondary"
// [lotus] primary: the Lotus theme renders `secondary` as dark
// text on a dark fill, and this is the action we want taken.
kind="primary"
onClick={action.onClick}
key={`action${index}`}
>
+24 -1
View File
@@ -31,10 +31,15 @@ import { useCallViewKeyboardShortcuts } from "../useCallViewKeyboardShortcuts";
import { widget } from "../widget";
import { startLotusCallState } from "../lotus/lotusCallState";
import { startLotusFocus } from "../lotus/lotusFocus";
import { startLotusControls } from "../lotus/lotusControls";
import { startLotusMicLevel } from "../lotus/lotusMicLevel";
import { startLotusAudioInject } from "../lotus/lotusAudioInject";
import { startLotusQuality } from "../lotus/lotusQuality";
import { startLotusDecorations } from "../lotus/lotusDecorations";
import { startLotusDenoise } from "../lotus/lotusDenoise";
import { startLotusCallSummary } from "../lotus/lotusCallSummary";
import { startLotusAudioOutput } from "../lotus/lotusAudioOutput";
import { startLotusScreenshareWatch } from "../lotus/lotusScreenshareWatch";
import { startLotusDeafen } from "../lotus/lotusDeafen";
import styles from "./InCallView.module.css";
import { GridTile } from "../tile/GridTile";
@@ -73,6 +78,7 @@ import { matrixRTCMode as matrixRTCModeSetting } from "../settings/settings";
import { ReactionsReader } from "../reactions/ReactionsReader";
import { LivekitRoomAudioRenderer } from "../livekit/MatrixAudioRenderer.tsx";
import { muteAllAudio$ } from "../state/MuteAllAudioModel.ts";
import { muteScreenshareAudio$ } from "../lotus/lotusScreenshareAudio";
import { useMediaDevices } from "../MediaDevicesContext.ts";
import { EarpieceOverlay } from "./EarpieceOverlay.tsx";
import {
@@ -261,6 +267,8 @@ export const InCallView: FC<InCallViewProps> = ({
const { showControls, header: headerStyle } = useUrlParams();
const muteAllAudio = useBehavior(muteAllAudio$);
// [lotus] host-driven "Mute Screenshare Audio" (io.lotus.set_deafen).
const muteScreenshareAudio = useBehavior(muteScreenshareAudio$);
const toggleAudio = useBehavior(muteStates.audio.toggle$);
const toggleVideo = useBehavior(muteStates.video.toggle$);
const setAudioEnabled = useBehavior(muteStates.audio.setEnabled$);
@@ -294,6 +302,11 @@ export const InCallView: FC<InCallViewProps> = ({
// [lotus] Handle the host's io.lotus.focus_participant action to pin a
// participant to the spotlight (#4). No-op unless the host sends it.
useEffect(() => startLotusFocus(vm), [vm]);
// [cinny #43] layout / settings / reactions over the widget API, plus a
// screensharing + layout report, replacing the host's DOM access.
useEffect(() => startLotusControls(vm), [vm]);
// [cinny #146] Local mic level for the host's mute-button meter.
useEffect(() => startLotusMicLevel(vm), [vm]);
// [lotus] Handle the host's io.lotus.inject_audio action to mix a soundboard
// clip into the call as a separate track (#3). No-op unless the host sends it.
useEffect(() => startLotusAudioInject(vm), [vm]);
@@ -306,10 +319,19 @@ export const InCallView: FC<InCallViewProps> = ({
// [lotus] Apply ML denoise to the mic as a first-class audio processor that
// survives reconnects (#1 / A7). No-op unless lotusDenoiseSource=1.
useEffect(() => startLotusDenoise(vm), [vm]);
useEffect(() => startLotusCallSummary(vm), [vm]);
// [lotus #39] Screenshare reminders (window closed / black / sharing alone).
useEffect(() => startLotusScreenshareWatch(vm), [vm]);
// [lotus] Handle the host's io.lotus.set_deafen action to silence remote
// audio (and optionally screenshare audio) at the LiveKit source. No-op
// unless the host sends the action.
useEffect(() => startLotusDeafen(vm), [vm]);
useEffect(() => startLotusDeafen(), []);
// [lotus #119] Let the host pick the audio output device from its call bar.
const lotusMediaDevices = useMediaDevices();
useEffect(
() => startLotusAudioOutput(lotusMediaDevices),
[lotusMediaDevices],
);
const fatalCallError = useBehavior(vm.fatalError$);
// Stop the rendering and throw for the error boundary
@@ -657,6 +679,7 @@ export const InCallView: FC<InCallViewProps> = ({
livekitRoom={livekitRoom}
validIdentities={participants}
muted={muteAllAudio}
screenshareAudioMuted={muteScreenshareAudio}
/>
))}
{renderContent()}
@@ -135,7 +135,7 @@ exports[`ConnectionLostError: Action handling should reset error state 1`] = `
</p>
<button
class="_button_1nw83_8"
data-kind="secondary"
data-kind="primary"
data-size="lg"
role="button"
tabindex="0"
@@ -158,7 +158,7 @@ exports[`ConnectionLostError: Action handling should reset error state 1`] = `
</DocumentFragment>
`;
exports[`LiveKit ConnectionError variants > should display LiveKit 'internal' error correctly 1`] = `
exports[`LiveKit ConnectionError variants > should explain the LiveKit 'internal' error and offer a retry 1`] = `
<DocumentFragment>
<div
class="_page_4be5c0"
@@ -286,19 +286,35 @@ exports[`LiveKit ConnectionError variants > should display LiveKit 'internal' er
<h1
class="_typography_6v6n8_153 _font-heading-md-semibold_6v6n8_112"
>
Failed to connect to Livekit server
Couldn’t connect to voice
</h1>
<p>
An error occurred while connecting to the Livekit server (
<b>
Reason:
</b>
<code>
InternalError
</code>
).
Your device couldn’t reach the voice server. Chat can still work when this happens: voice needs its own live connection, which VPNs, antivirus web protection and some work or school networks block. Try again. If it keeps failing, pause your VPN or antivirus web shield, or try another network such as a phone hotspot.
</p>
<details
class="_technicalDetails_a69dc5"
>
<summary
class="_technicalDetailsSummary_a69dc5"
>
Technical details
</summary>
<pre
class="_technicalDetailsPre_a69dc5"
>
Reason: InternalError
Internal server error
</pre>
</details>
<button
class="_button_1nw83_8"
data-kind="primary"
data-size="lg"
role="button"
tabindex="0"
>
Try again
</button>
<button
class="_button_1nw83_8"
data-kind="tertiary"
@@ -315,7 +331,7 @@ exports[`LiveKit ConnectionError variants > should display LiveKit 'internal' er
</DocumentFragment>
`;
exports[`LiveKit ConnectionError variants > should display LiveKit 'notAllowed' error correctly 1`] = `
exports[`LiveKit ConnectionError variants > should explain the LiveKit 'notAllowed' error and offer a retry 1`] = `
<DocumentFragment>
<div
class="_page_4be5c0"
@@ -443,19 +459,36 @@ exports[`LiveKit ConnectionError variants > should display LiveKit 'notAllowed'
<h1
class="_typography_6v6n8_153 _font-heading-md-semibold_6v6n8_112"
>
Failed to connect to Livekit server
Couldn’t connect to voice
</h1>
<p>
An error occurred while connecting to the Livekit server (
<b>
Reason:
</b>
<code>
NotAllowed
</code>
).
The voice server didn’t let you in. The call may be full, or your access to this room may have changed. Try again in a moment.
</p>
<details
class="_technicalDetails_a69dc5"
>
<summary
class="_technicalDetailsSummary_a69dc5"
>
Technical details
</summary>
<pre
class="_technicalDetailsPre_a69dc5"
>
Reason: NotAllowed
Status: 403
Permission denied by server
</pre>
</details>
<button
class="_button_1nw83_8"
data-kind="primary"
data-size="lg"
role="button"
tabindex="0"
>
Try again
</button>
<button
class="_button_1nw83_8"
data-kind="tertiary"
@@ -472,7 +505,7 @@ exports[`LiveKit ConnectionError variants > should display LiveKit 'notAllowed'
</DocumentFragment>
`;
exports[`LiveKit ConnectionError variants > should display LiveKit 'serverUnreachable' error correctly 1`] = `
exports[`LiveKit ConnectionError variants > should explain the LiveKit 'serverUnreachable' error and offer a retry 1`] = `
<DocumentFragment>
<div
class="_page_4be5c0"
@@ -600,19 +633,36 @@ exports[`LiveKit ConnectionError variants > should display LiveKit 'serverUnreac
<h1
class="_typography_6v6n8_153 _font-heading-md-semibold_6v6n8_112"
>
Failed to connect to Livekit server
Couldn’t connect to voice
</h1>
<p>
An error occurred while connecting to the Livekit server (
<b>
Reason:
</b>
<code>
ServerUnreachable
</code>
).
Your device couldn’t reach the voice server. Chat can still work when this happens: voice needs its own live connection, which VPNs, antivirus web protection and some work or school networks block. Try again. If it keeps failing, pause your VPN or antivirus web shield, or try another network such as a phone hotspot.
</p>
<details
class="_technicalDetails_a69dc5"
>
<summary
class="_technicalDetailsSummary_a69dc5"
>
Technical details
</summary>
<pre
class="_technicalDetailsPre_a69dc5"
>
Reason: ServerUnreachable
Status: 503
Server is unreachable
</pre>
</details>
<button
class="_button_1nw83_8"
data-kind="primary"
data-size="lg"
role="button"
tabindex="0"
>
Try again
</button>
<button
class="_button_1nw83_8"
data-kind="tertiary"
@@ -629,7 +679,7 @@ exports[`LiveKit ConnectionError variants > should display LiveKit 'serverUnreac
</DocumentFragment>
`;
exports[`LiveKit ConnectionError variants > should display LiveKit 'serviceNotFound' error correctly 1`] = `
exports[`LiveKit ConnectionError variants > should explain the LiveKit 'serviceNotFound' error and offer a retry 1`] = `
<DocumentFragment>
<div
class="_page_4be5c0"
@@ -757,19 +807,35 @@ exports[`LiveKit ConnectionError variants > should display LiveKit 'serviceNotFo
<h1
class="_typography_6v6n8_153 _font-heading-md-semibold_6v6n8_112"
>
Failed to connect to Livekit server
Couldn’t connect to voice
</h1>
<p>
An error occurred while connecting to the Livekit server (
<b>
Reason:
</b>
<code>
ServiceNotFound
</code>
).
Your device couldn’t reach the voice server. Chat can still work when this happens: voice needs its own live connection, which VPNs, antivirus web protection and some work or school networks block. Try again. If it keeps failing, pause your VPN or antivirus web shield, or try another network such as a phone hotspot.
</p>
<details
class="_technicalDetails_a69dc5"
>
<summary
class="_technicalDetailsSummary_a69dc5"
>
Technical details
</summary>
<pre
class="_technicalDetailsPre_a69dc5"
>
Reason: ServiceNotFound
RTC service not found
</pre>
</details>
<button
class="_button_1nw83_8"
data-kind="primary"
data-size="lg"
role="button"
tabindex="0"
>
Try again
</button>
<button
class="_button_1nw83_8"
data-kind="tertiary"
@@ -786,7 +852,7 @@ exports[`LiveKit ConnectionError variants > should display LiveKit 'serviceNotFo
</DocumentFragment>
`;
exports[`LiveKit ConnectionError variants > should display LiveKit 'timeout' error correctly 1`] = `
exports[`LiveKit ConnectionError variants > should explain the LiveKit 'timeout' error and offer a retry 1`] = `
<DocumentFragment>
<div
class="_page_4be5c0"
@@ -914,19 +980,35 @@ exports[`LiveKit ConnectionError variants > should display LiveKit 'timeout' err
<h1
class="_typography_6v6n8_153 _font-heading-md-semibold_6v6n8_112"
>
Failed to connect to Livekit server
Couldn’t connect to voice
</h1>
<p>
An error occurred while connecting to the Livekit server (
<b>
Reason:
</b>
<code>
Timeout
</code>
).
Your device couldn’t reach the voice server. Chat can still work when this happens: voice needs its own live connection, which VPNs, antivirus web protection and some work or school networks block. Try again. If it keeps failing, pause your VPN or antivirus web shield, or try another network such as a phone hotspot.
</p>
<details
class="_technicalDetails_a69dc5"
>
<summary
class="_technicalDetailsSummary_a69dc5"
>
Technical details
</summary>
<pre
class="_technicalDetailsPre_a69dc5"
>
Reason: Timeout
Connection timed out
</pre>
</details>
<button
class="_button_1nw83_8"
data-kind="primary"
data-size="lg"
role="button"
tabindex="0"
>
Try again
</button>
<button
class="_button_1nw83_8"
data-kind="tertiary"
@@ -1084,6 +1166,15 @@ exports[`LiveKit ConnectionError variants > should link to troubleshoot guide wh
</a>
or contact your server administrator.
</p>
<button
class="_button_1nw83_8"
data-kind="primary"
data-size="lg"
role="button"
tabindex="0"
>
Try again
</button>
<button
class="_button_1nw83_8"
data-kind="tertiary"
@@ -1697,7 +1788,7 @@ exports[`should report correct error for 'Connection lost' 1`] = `
</p>
<button
class="_button_1nw83_8"
data-kind="secondary"
data-kind="primary"
data-size="lg"
role="button"
tabindex="0"
@@ -37,6 +37,7 @@ import { Epoch, ObservableScope } from "../../ObservableScope";
import {
MatrixRTCTransportMissingError,
FailToGetOpenIdToken,
SFUTokenRefusedError,
} from "../../../utils/errors";
import * as openIDSFU from "../../../livekit/openIDSFU";
import { customLivekitUrl } from "../../../settings/settings";
@@ -125,6 +126,46 @@ describe("LocalTransport", () => {
expect(() => active$.value).toThrow(expectedError);
});
it("[lotus] passes SFUTokenRefusedError through untouched", async () => {
const scope = new ObservableScope();
mockConfig({
livekit: { livekit_service_url: "https://lk.example.org" },
});
const refused = new SFUTokenRefusedError("This voice channel is full.");
vi.spyOn(openIDSFU, "getSFUConfigWithOpenID").mockImplementation(
async () => {
await Promise.resolve();
throw refused;
},
);
const errors: Error[] = [];
const { active$ } = createLocalTransport$({
scope,
roomId: "!example_room_id",
memberships$: constant(new Epoch<CallMembership[]>([])),
client: {
baseUrl: "https://example.org",
getDomain: () => "example.org",
// eslint-disable-next-line @typescript-eslint/naming-convention
_unstable_getRTCTransports: async () => Promise.resolve([]),
getOpenIdToken: vi.fn(),
getDeviceId: vi.fn(),
},
ownMembershipIdentity: ownMemberMock,
forceJwtEndpoint: JwtEndpointVersion.Legacy,
delayId$: constant("delay_id_mock"),
});
active$.subscribe(
() => undefined,
(e) => errors.push(e),
);
await flushPromises();
expect(errors).toStrictEqual([refused]);
expect((errors[0] as SFUTokenRefusedError).localisedMessage).toBe(
"This voice channel is full.",
);
});
it("emits preferred transport after OpenID resolves", async () => {
// Use config so transport discovery succeeds, but delay OpenID JWT fetch
mockConfig({
@@ -26,6 +26,7 @@ import { type Epoch, type ObservableScope } from "../../ObservableScope.ts";
import { Config } from "../../../config/Config.ts";
import {
FailToGetOpenIdToken,
SFUTokenRefusedError,
MatrixRTCTransportMissingError,
NoMatrix2AuthorizationService,
} from "../../../utils/errors.ts";
@@ -261,7 +262,9 @@ async function doOpenIdAndJWTFromUrl(
function mapAuthErrorToUserFriendlyError(e: unknown): Error {
if (
e instanceof FailToGetOpenIdToken ||
e instanceof NoMatrix2AuthorizationService
e instanceof NoMatrix2AuthorizationService ||
// [lotus] carries the token service's own refusal reason — keep it.
e instanceof SFUTokenRefusedError
) {
// rethrow as is
return e;
+71 -27
View File
@@ -5,7 +5,16 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { combineLatest, map, merge, of, Subject, switchMap } from "rxjs";
import {
combineLatest,
distinctUntilChanged,
map,
merge,
of,
skip,
Subject,
switchMap,
} from "rxjs";
import { type Behavior } from "./Behavior";
import { type ObservableScope } from "./ObservableScope";
@@ -25,7 +34,12 @@ export interface VolumeControls {
playbackMuted$: Behavior<boolean>;
togglePlaybackMuted: () => void;
adjustPlaybackVolume: (value: number) => void;
commitPlaybackVolume: () => void;
/**
* Commit the volume. [lotus #36] Pass the slider's committed value: with the
* keyboard the slider commits before its last change reaches us, which left
* the committed (and remembered) volume one step behind.
*/
commitPlaybackVolume: (value?: number) => void;
}
interface VolumeControlsInputs {
@@ -35,6 +49,10 @@ interface VolumeControlsInputs {
* requested volume.
*/
sink$: Behavior<(volume: number) => void>;
/** [lotus #36] Starting volume (a remembered one); defaults to 1. */
initialVolume?: number;
/** [lotus #36] Called with each newly committed volume. */
onCommit?: (volume: number) => void;
}
/**
@@ -43,38 +61,61 @@ interface VolumeControlsInputs {
*/
export function createVolumeControls(
scope: ObservableScope,
{ pretendToBeDisconnected$, sink$ }: VolumeControlsInputs,
{
pretendToBeDisconnected$,
sink$,
initialVolume = 1,
onCommit,
}: VolumeControlsInputs,
): VolumeControls {
const toggleMuted$ = new Subject<"toggle mute">();
const adjustVolume$ = new Subject<number>();
const commitVolume$ = new Subject<"commit">();
const playbackVolume$ = scope.behavior<number>(
const state$ = scope.behavior(
merge(toggleMuted$, adjustVolume$, commitVolume$).pipe(
accumulate({ volume: 1, committedVolume: 1 }, (state, event) => {
switch (event) {
case "toggle mute":
return {
...state,
volume: state.volume === 0 ? state.committedVolume : 0,
};
case "commit":
// Dragging the slider to zero should have the same effect as
// muting: keep the original committed volume, as if it were never
// dragged
return {
...state,
committedVolume:
state.volume === 0 ? state.committedVolume : state.volume,
};
default:
// Volume adjustment
return { ...state, volume: event };
}
}),
map(({ volume }) => volume),
accumulate(
{ volume: initialVolume, committedVolume: initialVolume },
(state, event) => {
switch (event) {
case "toggle mute":
return {
...state,
volume: state.volume === 0 ? state.committedVolume : 0,
};
case "commit":
// Dragging the slider to zero should have the same effect as
// muting: keep the original committed volume, as if it were never
// dragged
return {
...state,
committedVolume:
state.volume === 0 ? state.committedVolume : state.volume,
};
default:
// Volume adjustment
return { ...state, volume: event };
}
},
),
),
);
const playbackVolume$ = scope.behavior<number>(
state$.pipe(map(({ volume }) => volume)),
);
// [lotus #36] Report committed changes (not the starting value) so the
// caller can remember them.
if (onCommit) {
state$
.pipe(
map(({ committedVolume }) => committedVolume),
distinctUntilChanged(),
skip(1),
scope.bind(),
)
.subscribe(onCommit);
}
// Sync the requested volume with the audio playback module
combineLatest([
@@ -96,6 +137,9 @@ export function createVolumeControls(
),
togglePlaybackMuted: () => toggleMuted$.next("toggle mute"),
adjustPlaybackVolume: (value: number) => adjustVolume$.next(value),
commitPlaybackVolume: () => commitVolume$.next("commit"),
commitPlaybackVolume: (value?: number) => {
if (value !== undefined) adjustVolume$.next(value);
commitVolume$.next("commit");
},
};
}
+31 -1
View File
@@ -5,7 +5,7 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { expect, onTestFinished, test, vi } from "vitest";
import { afterEach, expect, onTestFinished, test, vi } from "vitest";
import {
type LocalTrackPublication,
LocalVideoTrack,
@@ -44,6 +44,36 @@ vi.mock("../../Platform", () => ({
const rtcMembership = mockRtcMembership("@alice:example.org", "AAAA");
// [lotus #36] Remote volumes are remembered in localStorage; keep tests apart.
afterEach(() => localStorage.clear());
test("a remembered volume is restored and a committed slider value is saved", () => {
localStorage.setItem(
"lotus-per-user-volume",
JSON.stringify({ "@alice:example.org": 0.5 }),
);
const setVolumeSpy = vi.fn();
const vm = mockRemoteMedia(
rtcMembership,
{},
mockRemoteParticipant({ setVolume: setVolumeSpy }),
);
withTestScheduler(({ expectObservable, schedule }) => {
schedule("-a|", {
a() {
// Keyboard order: the slider commits with its final value before that
// value's change event reaches the view model.
vm.commitPlaybackVolume(0.3);
expect(setVolumeSpy).toHaveBeenLastCalledWith(0.3);
expect(
JSON.parse(localStorage.getItem("lotus-per-user-volume") ?? "{}"),
).toEqual({ "@alice:example.org": 0.3 });
},
});
expectObservable(vm.playbackVolume$).toBe("ab", { a: 0.5, b: 0.3 });
});
});
test("control a participant's volume", () => {
const setVolumeSpy = vi.fn();
const vm = mockRemoteMedia(
+15 -5
View File
@@ -7,7 +7,7 @@ Please see LICENSE in the repository root for full details.
*/
import { Track, type RemoteParticipant } from "livekit-client";
import { map, of, switchMap } from "rxjs";
import { distinctUntilChanged, map, of, switchMap } from "rxjs";
import { type Behavior } from "../Behavior";
import {
@@ -17,7 +17,7 @@ import {
} from "./ScreenShareViewModel";
import { type ObservableScope } from "../ObservableScope";
import { createVolumeControls, type VolumeControls } from "../VolumeControls";
import { observeTrackReference$ } from "../observeTrackReference";
import { observeParticipantMedia } from "@livekit/components-core";
export interface RemoteScreenShareViewModel
extends BaseScreenShareViewModel, VolumeControls {
@@ -58,14 +58,24 @@ export function createRemoteScreenShare(
videoEnabled$: scope.behavior(
pretendToBeDisconnected$.pipe(map((disconnected) => !disconnected)),
),
// [lotus #38] "Has audio" means the sharer publishes screenshare audio AND
// hasn't muted it — a muted publication would show a speaker glyph for a
// share nobody can hear.
audioEnabled$: scope.behavior(
inputs.participant$.pipe(
switchMap((p) =>
p
? observeTrackReference$(p, Track.Source.ScreenShareAudio)
: of(null),
? observeParticipantMedia(p).pipe(
map(() => {
const pub = p.getTrackPublication(
Track.Source.ScreenShareAudio,
);
return !!pub && !pub.isMuted;
}),
)
: of(false),
),
map(Boolean),
distinctUntilChanged(),
),
),
};
@@ -11,6 +11,10 @@ import { combineLatest, map, of, switchMap } from "rxjs";
import { type Behavior } from "../Behavior";
import { createVolumeControls, type VolumeControls } from "../VolumeControls";
import {
getRememberedVolume,
rememberVolume,
} from "../../lotus/lotusVolumeMemory";
import {
type BaseUserMediaInputs,
type BaseUserMediaViewModel,
@@ -52,6 +56,9 @@ export function createRemoteUserMedia(
sink$: scope.behavior(
inputs.participant$.pipe(map((p) => (volume) => p?.setVolume(volume))),
),
// [lotus #36] The slider sticks per user across calls and reconnects.
initialVolume: getRememberedVolume(inputs.userId),
onCommit: (volume) => rememberVolume(inputs.userId, volume),
}),
local: false,
speaking$: scope.behavior(
+11 -5
View File
@@ -43,6 +43,7 @@ import { useReactiveState } from "../useReactiveState";
import { useLatest } from "../useLatest";
import { type SpotlightTileViewModel } from "../state/TileViewModel";
import { useBehavior } from "../useBehavior";
import { muteScreenshareAudio$ } from "../lotus/lotusScreenshareAudio";
import { type MemberMediaViewModel } from "../state/media/MemberMediaViewModel";
import { type LocalUserMediaViewModel } from "../state/media/LocalUserMediaViewModel";
import { type RemoteUserMediaViewModel } from "../state/media/RemoteUserMediaViewModel";
@@ -322,11 +323,13 @@ const ScreenShareVolumeButton: FC<ScreenShareVolumeButtonProps> = ({ vm }) => {
const audioEnabled = useBehavior(vm.audioEnabled$);
const playbackMuted = useBehavior(vm.playbackMuted$);
const playbackVolume = useBehavior(vm.playbackVolume$);
// [lotus #38] The host's screenshare-audio mute (io.lotus.set_deafen) mutes
// at the renderer, not through the volume controls; show it as muted too.
const lotusMuted = useBehavior(muteScreenshareAudio$);
const shownMuted = playbackMuted || lotusMuted;
const VolumeIcon = playbackMuted ? VolumeOffIcon : VolumeOnIcon;
const VolumeSolidIcon = playbackMuted
? VolumeOffSolidIcon
: VolumeOnSolidIcon;
const VolumeIcon = shownMuted ? VolumeOffIcon : VolumeOnIcon;
const VolumeSolidIcon = shownMuted ? VolumeOffSolidIcon : VolumeOnSolidIcon;
const [volumeMenuOpen, setVolumeMenuOpen] = useState(false);
const onMuteButtonClick = useCallback(() => vm.togglePlaybackMuted(), [vm]);
@@ -334,7 +337,10 @@ const ScreenShareVolumeButton: FC<ScreenShareVolumeButtonProps> = ({ vm }) => {
(v: number) => vm.adjustPlaybackVolume(v),
[vm],
);
const onVolumeCommit = useCallback(() => vm.commitPlaybackVolume(), [vm]);
const onVolumeCommit = useCallback(
(value: number) => vm.commitPlaybackVolume(value),
[vm],
);
return (
audioEnabled && (
+39 -4
View File
@@ -6,7 +6,7 @@ Please see LICENSE in the repository root for full details.
*/
import { t } from "i18next";
import { type ConnectionError } from "livekit-client";
import { type ConnectionError, ConnectionErrorReason } from "livekit-client";
import { i18nKey } from "./i18n";
@@ -23,6 +23,8 @@ export enum ErrorCode {
E2EE_NOT_SUPPORTED = "E2EE_NOT_SUPPORTED",
STICKY_EVENTS_NOT_SUPPORTED = "STICKY_EVENTS_NOT_SUPPORTED",
OPEN_ID_ERROR = "OPEN_ID_ERROR",
/** [lotus] The SFU token service refused us with a human-readable reason (e.g. the voice-limit guard: channel full / no permission). */
SFU_TOKEN_REFUSED = "SFU_TOKEN_REFUSED",
NO_MATRIX_2_AUTHORIZATION_SERVICE = "NO_MATRIX_2_0_AUTHORIZATION_SERVICE",
SFU_ERROR = "SFU_ERROR",
UNKNOWN_ERROR = "UNKNOWN_ERROR",
@@ -51,6 +53,9 @@ export class ElementCallError extends Error {
public localisedMessageKey?: string;
public localisedMessageValues?: Record<string, string>;
/** [lotus] Raw detail shown under a collapsed "Technical details", not in the message. */
public technicalDetails?: string;
protected constructor(
localisedTitle: string,
code: ErrorCode,
@@ -234,6 +239,24 @@ export class FailToStartLivekitConnection extends ElementCallError {
/**
* Error indicating that a LiveKit's server has hit its track limits.
*/
/**
* [lotus] The SFU token service answered 403 with a reason we can show verbatim
* — the voice-limit guard says things like "This voice channel is full." or
* "You don't have permission to share your screen here." Without this the user
* only ever saw "Something went wrong (OPEN_ID_ERROR)".
*/
export class SFUTokenRefusedError extends ElementCallError {
public constructor(reason: string, cause?: Error) {
super(
t("error.sfu_token_refused"),
ErrorCode.SFU_TOKEN_REFUSED,
ErrorCategory.CONFIGURATION_ISSUE,
reason,
cause,
);
}
}
export class InsufficientCapacityError extends ElementCallError {
public constructor() {
super(
@@ -287,9 +310,21 @@ export class LivekitConnectionError extends ElementCallError {
ErrorCode.SFU_ERROR,
ErrorCategory.NETWORK_CONNECTIVITY,
);
this.localisedMessageKey = i18nKey(
"error.livekit_connection_error_description",
);
// [lotus] "An error occurred while connecting to the Livekit server
// (Reason: ServerUnreachable)" left people with nothing to try (Gitea
// element-call: a friend retried for ages while Element worked). Say what
// it means and what to do; keep the raw reason under Technical details.
this.localisedMessageKey =
cause.reason === ConnectionErrorReason.NotAllowed
? i18nKey("error.livekit_not_allowed_description")
: i18nKey("error.livekit_unreachable_description");
this.localisedMessageValues = { reason: cause.reasonName };
this.technicalDetails = [
`Reason: ${cause.reasonName}`,
cause.status ? `Status: ${cause.status}` : null,
cause.message,
]
.filter(Boolean)
.join("\n");
}
}