Compare commits
10
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
  Lotus CIandClaude Opus 5.5
|
a951e5b1f2 | ||
|
Lotus CIandClaude Opus 5.5
|
438d09cde9 | ||
 jared
|
0534fa393c | ||
|
Lotus CIandClaude Opus 5.5
|
e12b3e5979 | ||
|
Lotus CIandClaude Opus 5.5
|
657271163f | ||
|
Lotus CIandClaude Opus 5.5
|
60398b0a7d | ||
|
jared
|
2c8d20ba82 | ||
|
Lotus CIandClaude Opus 5.5
|
fbf8e44276 | ||
|
jared
|
5693e81bb9 | ||
|
Lotus CIandClaude Opus 5.5
|
5f4844cad5 |
@@ -33,32 +33,33 @@ The host sets these on the widget URL (cinny: `CallEmbed.getWidget`).
|
||||
The table is generated from `src/lotus/lotusActions.ts`; the last column was
|
||||
checked against cinny's `lotus` branch.
|
||||
|
||||
| Action | Direction | Payload / meaning | Handled by cinny (`lotus`) |
|
||||
| -------------------------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------- |
|
||||
| `io.lotus.call_state` | call → host | in-call per-participant speaking / mute state. NOTE: matrix-widget-api `transport.send` is request/response — the host MUST register a handler that replies/acks each one (cinny's `listenAction` does, replying `{}`). If the host has no handler, ClientWidgetApi immediately error-replies "unsupported from-widget action", so the data is silently dropped and each throttled send rejects (caught) — functional miss + log churn, not a hang. | yes |
|
||||
| `io.lotus.focus_participant` | host → call | pin/spotlight (or clear, with userId=null) a participant. | yes |
|
||||
| `io.lotus.inject_audio` | host → call | play an audio clip into the call as a separate published track. | yes |
|
||||
| `io.lotus.set_quality` | host → call | set audio/screenshare encoding quality (bitrate/framerate). | yes |
|
||||
| `io.lotus.decorations` | host → call | per-user avatar-decoration image URLs for in-call tiles. | yes |
|
||||
| `io.lotus.set_deafen` | host → call | deafen remote audio (and optionally mute screenshare audio). | yes |
|
||||
| `io.lotus.request_state` | call → host | sent on (re)registration of a lotus toWidget handler (e.g. decorations) so the host can re-push state that would otherwise only be sent on change (decorations roster, the current focus pin) after an EC-side reconnect/remount. cinny's `resendForkState()` should respond to this the same way it responds to a fresh join. | yes |
|
||||
| `io.lotus.denoise_state` | call → host | real state of the in-source denoise engine — `{ active: boolean, model: string, error?: string }` — sent once the processor attaches (or after the rnnoise fallback also fails), so the host toggle can reflect reality rather than the requested state. | yes |
|
||||
| `io.lotus.call_summary` | call → host | one-shot end-of-call readout for the local participant — `{ durationMs, reconnects, poorMs, verdict }` (#143) — sent on SFU disconnect or in-call teardown, whichever comes first. | yes |
|
||||
| `io.lotus.set_audio_output` | host → call | select the audio output device `{ deviceId }` (#119). | yes |
|
||||
| `io.lotus.audio_output_state` | call → host | the currently selected output `{ deviceId }` (#119). | cinny #417 |
|
||||
| `io.lotus.screenshare_notice` | call → host | local screenshare reminder `{ kind: "ended" \| "no-frames" \| "alone" }` — window closed, no frames for 15 s, or 30 min of sharing with nobody else in the call (#39). Each fires at most once per share. | yes |
|
||||
| `io.lotus.set_layout` | host → call | switch the call layout `{ layout: "grid" \| "spotlight" }` (cinny #43 — replaces the host clicking EC's hidden layout radio). | yes |
|
||||
| `io.lotus.open_settings` | host → call | open EC's settings modal, or close it with `{ open: false }`; omit `open` to toggle (cinny #43). | yes |
|
||||
| `io.lotus.toggle_reactions` | host → call | toggle the reactions / raise-hand menu (cinny #43). | yes |
|
||||
| `io.lotus.set_screenshare` | host → call | start/stop sharing `{ on?: boolean }` (omit to toggle; cinny #43). Starting calls getDisplayMedia, which needs the user's click: the host sends this with Capability Delegation (`delegate: "display-capture"`), and without it the browser rejects the share. | yes |
|
||||
| `io.lotus.set_hotkeys` | host → call | `{ codes: string[] }` — KeyboardEvent.code values (PTT, deafen) to report while focus is inside this frame (cinny #43). `[]` stops. | yes |
|
||||
| `io.lotus.set_frame_screenshare` | host → call | `{ visible: boolean }` — whether the in-frame screenshare button (`lotusFrameScreenshare`) may show, e.g. false while the room's call policy forbids screensharing (cinny #43). Defaults to visible. | yes |
|
||||
| `io.lotus.prompt_screenshare` | host → call | `{}` — show the "Share your screen?" prompt inside this frame (cinny #43). For hosts that can't delegate the user's click: the prompt's Share button is clicked here, so getDisplayMedia gets its activation. | yes |
|
||||
| `io.lotus.hotkey` | call → host | a watched key went down/up here, or this window's focus changed; see `LotusHotkeyReport` (cinny #43). | yes |
|
||||
| `io.lotus.controls_state` | call → host | `{ screensharing: boolean, layout: "grid" \| "spotlight" \| null, screenshareAction: true, hotkeys: true, frameScreenshare: false, screensharePrompt: boolean, screensharePromptOpen: boolean }` whenever either changes (cinny #43), so the host stops reading EC's DOM for them. Its arrival also tells the host this fork supports the actions above. | yes |
|
||||
| `io.lotus.mic_level` | call → host | local mic level `{ bars: 0 \| 1 \| 2 \| 3 }` (cinny #146), sent only when it changes (≤ 10 Hz); 0 while muted or with no mic. | yes |
|
||||
| `io.lotus.call_error` | call → host | the call's error page is up (`{ code, recoverable }`) or cleared (`{ code: null }`) (cinny #318), so the host stops showing the call as joined after "Connection lost". | cinny #418 |
|
||||
| `io.lotus.reconnect` | host → call | press the error page's Reconnect / Try again (cinny #318). | cinny #418 |
|
||||
| Action | Direction | Payload / meaning | Handled by cinny (`lotus`) |
|
||||
| ---------------------------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------- |
|
||||
| `io.lotus.call_state` | call → host | in-call per-participant speaking / mute state. NOTE: matrix-widget-api `transport.send` is request/response — the host MUST register a handler that replies/acks each one (cinny's `listenAction` does, replying `{}`). If the host has no handler, ClientWidgetApi immediately error-replies "unsupported from-widget action", so the data is silently dropped and each throttled send rejects (caught) — functional miss + log churn, not a hang. | yes |
|
||||
| `io.lotus.focus_participant` | host → call | pin/spotlight (or clear, with userId=null) a participant. | yes |
|
||||
| `io.lotus.inject_audio` | host → call | play an audio clip into the call: as a separate published track, or mixed into the microphone where the token only allows listed sources (rooms with a call policy, #47). Replies `{ played: false, reason: "muted" \| "not_permitted" }` when it can't. | yes |
|
||||
| `io.lotus.set_quality` | host → call | set audio/screenshare encoding quality (bitrate/framerate). | yes |
|
||||
| `io.lotus.decorations` | host → call | per-user avatar-decoration image URLs for in-call tiles. | yes |
|
||||
| `io.lotus.set_deafen` | host → call | deafen remote audio (and optionally mute screenshare audio). | yes |
|
||||
| `io.lotus.request_state` | call → host | sent on (re)registration of a lotus toWidget handler (e.g. decorations) so the host can re-push state that would otherwise only be sent on change (decorations roster, the current focus pin) after an EC-side reconnect/remount. cinny's `resendForkState()` should respond to this the same way it responds to a fresh join. | yes |
|
||||
| `io.lotus.denoise_state` | call → host | real state of the in-source denoise engine — `{ active: boolean, model: string, error?: string }` — sent once the processor attaches (or after the rnnoise fallback also fails), so the host toggle can reflect reality rather than the requested state. | yes |
|
||||
| `io.lotus.call_summary` | call → host | one-shot end-of-call readout `{ durationMs, reconnects, poorMs, verdict, details? }` (#143); `details` (cinny #253): participants, peak, per-person poor time, everyone-poor time, RTT/loss/jitter, your talk/mute/screenshare time. | yes |
|
||||
| `io.lotus.set_audio_output` | host → call | select the audio output device `{ deviceId }` (#119). | yes |
|
||||
| `io.lotus.audio_output_state` | call → host | the currently selected output `{ deviceId }` (#119). | cinny #417 |
|
||||
| `io.lotus.screenshare_audio_state` | call → host | `{ present }`: whether any remote screenshare audio is being received, sent on change (cinny #277). | cinny #277 |
|
||||
| `io.lotus.screenshare_notice` | call → host | local screenshare reminder `{ kind: "ended" \| "no-frames" \| "alone" }` — window closed, no frames for 15 s, or 30 min of sharing with nobody else in the call (#39). Each fires at most once per share. | yes |
|
||||
| `io.lotus.set_layout` | host → call | switch the call layout `{ layout: "grid" \| "spotlight" }` (cinny #43 — replaces the host clicking EC's hidden layout radio). | yes |
|
||||
| `io.lotus.open_settings` | host → call | open EC's settings modal, or close it with `{ open: false }`; omit `open` to toggle (cinny #43). | yes |
|
||||
| `io.lotus.toggle_reactions` | host → call | toggle the reactions / raise-hand menu (cinny #43). | yes |
|
||||
| `io.lotus.set_screenshare` | host → call | start/stop sharing `{ on?: boolean }` (omit to toggle; cinny #43). Starting calls getDisplayMedia, which needs the user's click: the host sends this with Capability Delegation (`delegate: "display-capture"`), and without it the browser rejects the share. | yes |
|
||||
| `io.lotus.set_hotkeys` | host → call | `{ codes: string[] }` — KeyboardEvent.code values (PTT, deafen) to report while focus is inside this frame (cinny #43). `[]` stops. | yes |
|
||||
| `io.lotus.set_frame_screenshare` | host → call | `{ visible: boolean }` — whether the in-frame screenshare button (`lotusFrameScreenshare`) may show, e.g. false while the room's call policy forbids screensharing (cinny #43). Defaults to visible. | yes |
|
||||
| `io.lotus.prompt_screenshare` | host → call | `{}` — show the "Share your screen?" prompt inside this frame (cinny #43). For hosts that can't delegate the user's click: the prompt's Share button is clicked here, so getDisplayMedia gets its activation. | yes |
|
||||
| `io.lotus.hotkey` | call → host | a watched key went down/up here, or this window's focus changed; see `LotusHotkeyReport` (cinny #43). | yes |
|
||||
| `io.lotus.controls_state` | call → host | `{ screensharing: boolean, layout: "grid" \| "spotlight" \| null, screenshareAction: true, hotkeys: true, frameScreenshare: false, screensharePrompt: boolean, screensharePromptOpen: boolean }` whenever either changes (cinny #43), so the host stops reading EC's DOM for them. Its arrival also tells the host this fork supports the actions above. | yes |
|
||||
| `io.lotus.mic_level` | call → host | local mic level `{ bars: 0 \| 1 \| 2 \| 3 }` (cinny #146), sent only when it changes (≤ 10 Hz); 0 while muted or with no mic. | yes |
|
||||
| `io.lotus.call_error` | call → host | the call's error page is up (`{ code, recoverable }`) or cleared (`{ code: null }`) (cinny #318), so the host stops showing the call as joined after "Connection lost". | cinny #418 |
|
||||
| `io.lotus.reconnect` | host → call | press the error page's Reconnect / Try again (cinny #318). | cinny #418 |
|
||||
|
||||
Entries naming a cinny PR are handled once that PR merges.
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@lotusguild/element-call-embedded",
|
||||
"version": "0.25.0-lotus.26",
|
||||
"version": "0.25.0-lotus.28",
|
||||
"files": [
|
||||
"README.md",
|
||||
"LICENSE-AGPL-3.0",
|
||||
|
||||
@@ -19,6 +19,10 @@ import { logger as rootLogger } from "matrix-js-sdk/lib/logger";
|
||||
import { useEarpieceAudioConfig } from "../MediaDevicesContext";
|
||||
import { useReactiveState } from "../useReactiveState";
|
||||
import * as controls from "../controls";
|
||||
import {
|
||||
clearScreenshareAudio,
|
||||
reportScreenshareAudio,
|
||||
} from "../lotus/lotusScreenshareAudio";
|
||||
|
||||
export interface MatrixAudioRendererProps {
|
||||
/**
|
||||
@@ -97,6 +101,18 @@ export function LivekitRoomAudioRenderer({
|
||||
return true;
|
||||
});
|
||||
|
||||
// [lotus / cinny #277] Tell the host whether remote screenshare audio is
|
||||
// playing, so its "Mute Screenshare Audio" control only shows when useful.
|
||||
const hasScreenshareAudio = tracks.some(
|
||||
(ref) =>
|
||||
!ref.participant.isLocal &&
|
||||
ref.publication.source === Track.Source.ScreenShareAudio,
|
||||
);
|
||||
useEffect(() => {
|
||||
reportScreenshareAudio(url, hasScreenshareAudio);
|
||||
}, [url, hasScreenshareAudio]);
|
||||
useEffect(() => (): void => clearScreenshareAudio(url), [url]);
|
||||
|
||||
// This component is also (in addition to the "only play audio for connected members" logic above)
|
||||
// responsible for mimicking earpiece audio on iPhones.
|
||||
// The Safari audio devices enumeration does not expose an earpiece audio device.
|
||||
|
||||
@@ -56,6 +56,12 @@ export enum LotusWidgetActions {
|
||||
SetAudioOutput = "io.lotus.set_audio_output",
|
||||
/** fromWidget: the currently selected output `{ deviceId }` (#119). */
|
||||
AudioOutputState = "io.lotus.audio_output_state",
|
||||
/**
|
||||
* fromWidget: `{ present: boolean }`, whether any remote screenshare audio
|
||||
* is being received, so the host's "Mute Screenshare Audio" control only
|
||||
* shows when there is something to mute (cinny #277). Sent on change.
|
||||
*/
|
||||
ScreenshareAudioState = "io.lotus.screenshare_audio_state",
|
||||
/**
|
||||
* fromWidget: local screenshare reminder `{ kind: "ended" | "no-frames" | "alone" }`
|
||||
* — window closed, no frames for 15 s, or 30 min of sharing with nobody
|
||||
|
||||
@@ -13,6 +13,7 @@ import { type CallViewModel } from "../state/CallViewModel/CallViewModel";
|
||||
import {
|
||||
canPublishSoundboard,
|
||||
MAX_INJECT_BYTES,
|
||||
soundboardMode,
|
||||
parseInjectSource,
|
||||
startLotusAudioInject,
|
||||
} from "./lotusAudioInject";
|
||||
@@ -94,6 +95,10 @@ class FakeAudioContext {
|
||||
connect: vi.fn((n: unknown) => n),
|
||||
disconnect: vi.fn(),
|
||||
}));
|
||||
public createMediaStreamSource = vi.fn(() => ({
|
||||
connect: vi.fn((n: unknown) => n),
|
||||
disconnect: vi.fn(),
|
||||
}));
|
||||
public createBufferSource = vi.fn(() => ({
|
||||
buffer: undefined as AudioBuffer | undefined,
|
||||
connect: vi.fn((n: unknown) => n),
|
||||
@@ -116,6 +121,12 @@ beforeEach(() => {
|
||||
// leave a real timer pending.
|
||||
vi.useFakeTimers();
|
||||
vi.stubGlobal("AudioContext", FakeAudioContext);
|
||||
vi.stubGlobal(
|
||||
"MediaStream",
|
||||
class {
|
||||
public constructor(public tracks: unknown[]) {}
|
||||
},
|
||||
);
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn().mockResolvedValue({
|
||||
@@ -273,3 +284,127 @@ test("canPublishSoundboard: refused by a microphone-only grant (EC #47)", () =>
|
||||
).toBe(true);
|
||||
expect(canPublishSoundboard({ canPublish: false })).toBe(false);
|
||||
});
|
||||
|
||||
test("soundboardMode: own track when unrestricted, the mic under a call policy (EC #47)", () => {
|
||||
expect(soundboardMode(undefined)).toBe("track");
|
||||
expect(soundboardMode({ canPublish: true, canPublishSources: [] })).toBe(
|
||||
"track",
|
||||
);
|
||||
expect(soundboardMode({ canPublish: true, canPublishSources: [2, 0] })).toBe(
|
||||
"track",
|
||||
);
|
||||
expect(soundboardMode({ canPublish: true, canPublishSources: [2] })).toBe(
|
||||
"mic",
|
||||
);
|
||||
expect(
|
||||
soundboardMode({ canPublish: true, canPublishSources: [2, 3, 4] }),
|
||||
).toBe("mic");
|
||||
expect(
|
||||
soundboardMode({ canPublish: true, canPublishSources: [3] }),
|
||||
).toBeNull();
|
||||
expect(soundboardMode({ canPublish: false })).toBeNull();
|
||||
});
|
||||
|
||||
/** A room whose token only allows the microphone, with a published mic track. */
|
||||
function makePolicyRoom(sources: number[]): {
|
||||
room: unknown;
|
||||
sender: { track: MediaStreamTrack; replaceTrack: ReturnType<typeof vi.fn> };
|
||||
original: MediaStreamTrack;
|
||||
micTrack: EventEmitter;
|
||||
publishTrack: ReturnType<typeof vi.fn>;
|
||||
} {
|
||||
const original = makeTrack();
|
||||
const sender = {
|
||||
track: original,
|
||||
replaceTrack: vi.fn(async (t: MediaStreamTrack) => {
|
||||
await Promise.resolve();
|
||||
sender.track = t;
|
||||
}),
|
||||
};
|
||||
const micTrack = Object.assign(new EventEmitter(), { sender });
|
||||
const publishTrack = vi.fn();
|
||||
const room = {
|
||||
localParticipant: {
|
||||
isMicrophoneEnabled: true,
|
||||
permissions: { canPublish: true, canPublishSources: sources },
|
||||
publishTrack,
|
||||
unpublishTrack: vi.fn(),
|
||||
getTrackPublication: (): unknown => ({ track: micTrack }),
|
||||
},
|
||||
};
|
||||
return { room, sender, original, micTrack, publishTrack };
|
||||
}
|
||||
|
||||
test("[EC #47] under a mic-only grant the clip is mixed into the microphone, then the mic is restored", async () => {
|
||||
const { room, sender, original, publishTrack } = makePolicyRoom([2]);
|
||||
const stop = startLotusAudioInject(mockVm([room]));
|
||||
|
||||
send({ url: "https://example.com/clip.mp3" });
|
||||
await flush();
|
||||
|
||||
expect(reply).toHaveBeenCalledWith(expect.anything(), {});
|
||||
expect(publishTrack).not.toHaveBeenCalled();
|
||||
expect(sender.replaceTrack).toHaveBeenCalledTimes(1);
|
||||
const mixed = sender.track;
|
||||
expect(mixed).not.toBe(original);
|
||||
const source = contextInstances[0].createBufferSource.mock.results[0].value;
|
||||
expect(source.start).toHaveBeenCalled();
|
||||
|
||||
source.onended();
|
||||
await flush();
|
||||
expect(sender.replaceTrack).toHaveBeenLastCalledWith(original);
|
||||
expect(sender.track).toBe(original);
|
||||
expect(mixed.stop).toHaveBeenCalled();
|
||||
|
||||
stop();
|
||||
});
|
||||
|
||||
test("[EC #47] muting mid-clip stops the clip and restores the microphone", async () => {
|
||||
const { room, sender, original, micTrack } = makePolicyRoom([2]);
|
||||
const stop = startLotusAudioInject(mockVm([room]));
|
||||
|
||||
send({ url: "https://example.com/clip.mp3" });
|
||||
await flush();
|
||||
const source = contextInstances[0].createBufferSource.mock.results[0].value;
|
||||
|
||||
micTrack.emit("muted");
|
||||
await flush();
|
||||
expect(source.stop).toHaveBeenCalled();
|
||||
expect(sender.track).toBe(original);
|
||||
|
||||
stop();
|
||||
});
|
||||
|
||||
test("[EC #47] a track LiveKit swapped in meanwhile is not overwritten on restore", async () => {
|
||||
const { room, sender, original } = makePolicyRoom([2]);
|
||||
const stop = startLotusAudioInject(mockVm([room]));
|
||||
|
||||
send({ url: "https://example.com/clip.mp3" });
|
||||
await flush();
|
||||
const newer = makeTrack();
|
||||
sender.track = newer; // e.g. a device switch while the clip plays
|
||||
const source = contextInstances[0].createBufferSource.mock.results[0].value;
|
||||
source.onended();
|
||||
await flush();
|
||||
expect(sender.track).toBe(newer);
|
||||
expect(sender.replaceTrack).not.toHaveBeenCalledWith(original);
|
||||
|
||||
stop();
|
||||
});
|
||||
|
||||
test("[EC #47] no microphone grant at all is still refused up front", async () => {
|
||||
const { room, publishTrack, sender } = makePolicyRoom([3]);
|
||||
const stop = startLotusAudioInject(mockVm([room]));
|
||||
|
||||
send({ url: "https://example.com/clip.mp3" });
|
||||
await flush();
|
||||
|
||||
expect(reply).toHaveBeenCalledWith(expect.anything(), {
|
||||
played: false,
|
||||
reason: "not_permitted",
|
||||
});
|
||||
expect(publishTrack).not.toHaveBeenCalled();
|
||||
expect(sender.replaceTrack).not.toHaveBeenCalled();
|
||||
|
||||
stop();
|
||||
});
|
||||
|
||||
@@ -5,7 +5,12 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
|
||||
Please see LICENSE in the repository root for full details.
|
||||
*/
|
||||
|
||||
import { type Room as LivekitRoom, Track } from "livekit-client";
|
||||
import {
|
||||
type LocalTrack,
|
||||
type Room as LivekitRoom,
|
||||
Track,
|
||||
TrackEvent,
|
||||
} from "livekit-client";
|
||||
import { logger } from "matrix-js-sdk/lib/logger";
|
||||
import { type IWidgetApiRequest } from "matrix-widget-api";
|
||||
|
||||
@@ -52,6 +57,10 @@ function acquireSharedAudio(): {
|
||||
* that was impossible against the prebuilt EC bundle (LiveKit's
|
||||
* LocalParticipant lived in EC's module scope).
|
||||
*
|
||||
* [EC #47] Exception: rooms with a call policy issue tokens that only allow
|
||||
* listed sources, which refuse that separate track. There the clip is mixed
|
||||
* into the published microphone for its duration (`playIntoMicrophone`).
|
||||
*
|
||||
* Action data: `{ url?: string, audio?: ArrayBuffer, volume?: number }`.
|
||||
* `audio` is the clip's bytes (cinny #43: a host `blob:` URL can't be fetched
|
||||
* once EC is served from its own origin, so the host sends the bytes and they
|
||||
@@ -76,6 +85,25 @@ export function canPublishSoundboard(
|
||||
return sources.length === 0 || sources.includes(0);
|
||||
}
|
||||
|
||||
/**
|
||||
* Pure: how a clip can reach the call under these permissions (EC #47).
|
||||
* - `"track"`: as its own Unknown-source track (unrestricted rooms).
|
||||
* - `"mic"`: mixed into the published microphone track. Rooms with a call
|
||||
* policy grant only listed sources (microphone, maybe screenshare), which
|
||||
* refuse the separate track, but the mic is allowed.
|
||||
* - `null`: neither (no publishing at all).
|
||||
*/
|
||||
export function soundboardMode(
|
||||
perm:
|
||||
| { canPublish?: boolean; canPublishSources?: readonly number[] }
|
||||
| undefined,
|
||||
): "track" | "mic" | null {
|
||||
if (canPublishSoundboard(perm)) return "track";
|
||||
if (perm?.canPublish === false) return null;
|
||||
// TrackSource.MICROPHONE = 2
|
||||
return perm?.canPublishSources?.includes(2) ? "mic" : null;
|
||||
}
|
||||
|
||||
export function startLotusAudioInject(vm: CallViewModel): () => void {
|
||||
const w = widget;
|
||||
if (!w) return () => undefined;
|
||||
@@ -132,21 +160,24 @@ export function startLotusAudioInject(vm: CallViewModel): () => void {
|
||||
return;
|
||||
}
|
||||
|
||||
// [EC #47] Clips publish as an Unknown-source track. Audio-only rooms get
|
||||
// a token limited to the microphone source, where LiveKit refuses that
|
||||
// publish *after* we replied; say so up front instead of failing silently.
|
||||
if (
|
||||
rooms.some((r) => !canPublishSoundboard(r.localParticipant.permissions))
|
||||
) {
|
||||
// [EC #47] Clips publish as an Unknown-source track, which rooms with a
|
||||
// call policy refuse (their tokens list allowed sources). There the clip
|
||||
// is mixed into the published microphone instead; only when neither is
|
||||
// possible say so up front instead of failing silently.
|
||||
const modes = rooms.map((r) =>
|
||||
soundboardMode(r.localParticipant.permissions),
|
||||
);
|
||||
if (modes.some((m) => m === null)) {
|
||||
w.api.transport.reply(ev.detail, {
|
||||
played: false,
|
||||
reason: "not_permitted",
|
||||
});
|
||||
return;
|
||||
}
|
||||
const mode = modes.includes("mic") ? "mic" : "track";
|
||||
|
||||
w.api.transport.reply(ev.detail, {});
|
||||
void playInjectedClip(clip, volume, rooms, activeClips).catch((e) =>
|
||||
void playInjectedClip(clip, volume, rooms, activeClips, mode).catch((e) =>
|
||||
logger.warn("[lotus] inject_audio failed", e),
|
||||
);
|
||||
};
|
||||
@@ -211,6 +242,7 @@ async function playInjectedClip(
|
||||
volume: number,
|
||||
rooms: LivekitRoom[],
|
||||
activeClips: Set<() => void>,
|
||||
mode: "track" | "mic" = "track",
|
||||
): Promise<void> {
|
||||
if (rooms.length === 0) {
|
||||
logger.warn("[lotus] inject_audio: no connected rooms");
|
||||
@@ -295,8 +327,23 @@ async function playInjectedClip(
|
||||
gain.gain.value = volume;
|
||||
const source = ctx.createBufferSource();
|
||||
source.buffer = buffer;
|
||||
source.connect(gain).connect(dest);
|
||||
source.connect(gain);
|
||||
|
||||
if (mode === "mic") {
|
||||
playIntoMicrophone(
|
||||
ctx,
|
||||
source,
|
||||
gain,
|
||||
buffer,
|
||||
rooms,
|
||||
activeClips,
|
||||
placeholder,
|
||||
() => aborted,
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
gain.connect(dest);
|
||||
const mst = dest.stream.getAudioTracks()[0];
|
||||
if (!mst) {
|
||||
source.disconnect();
|
||||
@@ -375,3 +422,114 @@ async function playInjectedClip(
|
||||
|
||||
source.start();
|
||||
}
|
||||
|
||||
/**
|
||||
* [EC #47] Mix the clip into the published microphone for its duration. For
|
||||
* every room, the track currently going out on the mic's RTCRtpSender (the
|
||||
* denoised one, when noise suppression is on) and the clip are mixed in a
|
||||
* MediaStreamAudioDestinationNode, and the sender switches to the mix with
|
||||
* `replaceTrack`. The server still sees one ordinary microphone track, which
|
||||
* rooms with a call policy allow. When the clip ends the original track goes
|
||||
* back. Muting mid-clip stops the clip (#13: muted means silent), and if
|
||||
* LiveKit swapped its own track in meanwhile (device change, denoise toggle)
|
||||
* that one is left alone.
|
||||
*/
|
||||
function playIntoMicrophone(
|
||||
ctx: AudioContext,
|
||||
source: AudioBufferSourceNode,
|
||||
gain: GainNode,
|
||||
buffer: AudioBuffer,
|
||||
rooms: LivekitRoom[],
|
||||
activeClips: Set<() => void>,
|
||||
placeholder: () => void,
|
||||
isAborted: () => boolean,
|
||||
): void {
|
||||
type Mix = {
|
||||
sender: RTCRtpSender;
|
||||
original: MediaStreamTrack;
|
||||
mixed: MediaStreamTrack;
|
||||
mic: MediaStreamAudioSourceNode;
|
||||
dest: MediaStreamAudioDestinationNode;
|
||||
track: LocalTrack;
|
||||
};
|
||||
const mixes: Mix[] = [];
|
||||
let cleanedUp = false;
|
||||
const cleanup = (): void => {
|
||||
if (cleanedUp) return;
|
||||
cleanedUp = true;
|
||||
activeClips.delete(cleanup);
|
||||
activeClips.delete(placeholder);
|
||||
try {
|
||||
source.stop();
|
||||
} catch {
|
||||
/* not started or already stopped */
|
||||
}
|
||||
source.disconnect();
|
||||
gain.disconnect();
|
||||
for (const m of mixes) {
|
||||
m.track.off(TrackEvent.Muted, cleanup);
|
||||
m.mic.disconnect();
|
||||
// Only undo our own swap: LiveKit may have replaced the track since.
|
||||
if (m.sender.track === m.mixed)
|
||||
void m.sender.replaceTrack(m.original).catch(() => undefined);
|
||||
m.mixed.stop();
|
||||
}
|
||||
};
|
||||
|
||||
for (const room of rooms) {
|
||||
const track = room.localParticipant.getTrackPublication(
|
||||
Track.Source.Microphone,
|
||||
)?.track;
|
||||
const sender = track?.sender;
|
||||
const original = sender?.track;
|
||||
if (!track || !sender || !original) continue;
|
||||
const dest = ctx.createMediaStreamDestination();
|
||||
const mic = ctx.createMediaStreamSource(new MediaStream([original]));
|
||||
mic.connect(dest);
|
||||
gain.connect(dest);
|
||||
const mixed = dest.stream.getAudioTracks()[0];
|
||||
if (!mixed) {
|
||||
mic.disconnect();
|
||||
continue;
|
||||
}
|
||||
mixes.push({ sender, original, mixed, mic, dest, track });
|
||||
track.on(TrackEvent.Muted, cleanup);
|
||||
}
|
||||
|
||||
activeClips.delete(placeholder);
|
||||
activeClips.add(cleanup);
|
||||
if (mixes.length === 0) {
|
||||
logger.warn("[lotus] inject_audio: no published microphone to mix into");
|
||||
cleanup();
|
||||
return;
|
||||
}
|
||||
if (isAborted()) {
|
||||
cleanup();
|
||||
return;
|
||||
}
|
||||
|
||||
// Switch every sender to its mix, then start the clip once they're live so
|
||||
// its start isn't clipped.
|
||||
void Promise.all(
|
||||
mixes.map(async (m) =>
|
||||
m.sender.replaceTrack(m.mixed).catch((e) => {
|
||||
logger.warn("[lotus] inject_audio: replaceTrack failed", e);
|
||||
}),
|
||||
),
|
||||
).then(() => {
|
||||
if (cleanedUp || isAborted()) {
|
||||
cleanup();
|
||||
return;
|
||||
}
|
||||
source.onended = cleanup;
|
||||
const durationMs = Number.isFinite(buffer.duration)
|
||||
? buffer.duration * 1000 + 500
|
||||
: MAX_CLIP_MS;
|
||||
const guard = setTimeout(
|
||||
cleanup,
|
||||
Math.min(MAX_CLIP_MS, Math.max(0, durationMs)),
|
||||
);
|
||||
source.addEventListener("ended", () => clearTimeout(guard));
|
||||
source.start();
|
||||
});
|
||||
}
|
||||
|
||||
@@ -8,7 +8,11 @@ Please see LICENSE in the repository root for full details.
|
||||
import { ConnectionQuality } from "livekit-client";
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import { CallQualityTracker } from "./lotusCallSummary";
|
||||
import {
|
||||
CallDetailsTracker,
|
||||
CallQualityTracker,
|
||||
matrixUserOf,
|
||||
} from "./lotusCallSummary";
|
||||
|
||||
const clock = (): { now: () => number; tick: (ms: number) => void } => {
|
||||
let t = 1_000_000;
|
||||
@@ -91,3 +95,92 @@ describe("CallQualityTracker", () => {
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("[cinny #253] CallDetailsTracker", () => {
|
||||
it("participants, peak, and devices of one person counted once", () => {
|
||||
const c = clock();
|
||||
const d = new CallDetailsTracker(c.now);
|
||||
d.remoteJoined("@bob:x:DEV1");
|
||||
d.remoteJoined("@bob:x:DEV2");
|
||||
d.remoteJoined("@carol:x:D");
|
||||
d.remoteLeft("@carol:x:D");
|
||||
d.remoteJoined("@dave:x:D");
|
||||
const r = d.details();
|
||||
expect(r.participants).toEqual(["@bob:x", "@carol:x", "@dave:x"]);
|
||||
expect(r.peakParticipants).toBe(3); // you + bob + carol (dave came after carol left)
|
||||
});
|
||||
|
||||
it("joining is idempotent per device (seeding everyone present on connect)", () => {
|
||||
const d = new CallDetailsTracker();
|
||||
d.remoteJoined("@bob:x:DEV1");
|
||||
d.remoteJoined("@bob:x:DEV1"); // seen again by the on-connect seed
|
||||
d.remoteLeft("@bob:x:DEV1"); // one leave: bob is gone
|
||||
d.remoteJoined("@carol:x:D");
|
||||
d.remoteJoined("@dave:x:D"); // peak is you + carol + dave, not + a ghost bob
|
||||
d.setQuality("@bob:x:DEV1", ConnectionQuality.Poor);
|
||||
expect(d.details().participants).toEqual(["@bob:x", "@carol:x", "@dave:x"]);
|
||||
expect(d.details().peakParticipants).toBe(3);
|
||||
});
|
||||
|
||||
it("poor time per person, and everyone-at-once only while all are poor", () => {
|
||||
const c = clock();
|
||||
const d = new CallDetailsTracker(c.now);
|
||||
d.remoteJoined("@bob:x:D");
|
||||
d.setQuality(undefined, ConnectionQuality.Poor); // you poor
|
||||
c.tick(10_000);
|
||||
d.setQuality("@bob:x:D", ConnectionQuality.Lost); // now everyone
|
||||
c.tick(5_000);
|
||||
d.setQuality(undefined, ConnectionQuality.Good);
|
||||
c.tick(3_000);
|
||||
d.setQuality("@bob:x:D", ConnectionQuality.Excellent);
|
||||
const q = d.details().quality;
|
||||
expect(q.youPoorMs).toBe(15_000);
|
||||
expect(q.othersPoorMs).toEqual({ "@bob:x": 8_000 });
|
||||
expect(q.everyonePoorMs).toBe(5_000);
|
||||
});
|
||||
|
||||
it("network numbers: RTT median/p95, loss from the latest counters, jitter", () => {
|
||||
const d = new CallDetailsTracker();
|
||||
for (const rtt of [0.02, 0.03, 0.04, 0.05, 0.3]) d.addRtt(rtt);
|
||||
d.addRtt(Number.NaN);
|
||||
d.setSent("mic", 900, 5);
|
||||
d.setSent("mic", 995, 5); // cumulative: latest wins
|
||||
d.setReceived("bob", 490, 10);
|
||||
d.addJitter(0.004);
|
||||
d.addJitter(0.006);
|
||||
const n = d.details().network;
|
||||
expect(n.rttMedianMs).toBe(40);
|
||||
expect(n.rttP95Ms).toBe(300);
|
||||
expect(n.outboundLossPct).toBe(0.5);
|
||||
expect(n.inboundLossPct).toBe(2);
|
||||
expect(n.jitterMs).toBe(5);
|
||||
});
|
||||
|
||||
it("no samples: network numbers are absent, not zero", () => {
|
||||
expect(new CallDetailsTracker().details().network).toEqual({});
|
||||
});
|
||||
|
||||
it("talk, mute and screenshare time", () => {
|
||||
const c = clock();
|
||||
const d = new CallDetailsTracker(c.now);
|
||||
d.setMuted(true);
|
||||
c.tick(4_000);
|
||||
d.setMuted(false);
|
||||
d.setSpeaking(true);
|
||||
c.tick(2_000);
|
||||
d.setSpeaking(false);
|
||||
d.setScreensharing(true);
|
||||
c.tick(6_000); // still sharing when summarised
|
||||
expect(d.details().activity).toEqual({
|
||||
speakingMs: 2_000,
|
||||
mutedMs: 4_000,
|
||||
screenshareMs: 6_000,
|
||||
});
|
||||
});
|
||||
|
||||
it("matrixUserOf reduces a LiveKit identity to the Matrix user", () => {
|
||||
expect(matrixUserOf("@bob:example.org:ABCDEF")).toBe("@bob:example.org");
|
||||
expect(matrixUserOf("@bob:example.org")).toBe("@bob:example.org");
|
||||
expect(matrixUserOf("hashed-federated-id")).toBe("hashed-federated-id");
|
||||
});
|
||||
});
|
||||
|
||||
@@ -7,9 +7,16 @@ Please see LICENSE in the repository root for full details.
|
||||
|
||||
import {
|
||||
ConnectionQuality,
|
||||
type LocalAudioTrack,
|
||||
type LocalTrackPublication,
|
||||
type Participant,
|
||||
ParticipantEvent,
|
||||
type RemoteAudioTrack,
|
||||
type RemoteParticipant,
|
||||
type Room as LivekitRoom,
|
||||
RoomEvent,
|
||||
Track,
|
||||
type TrackPublication,
|
||||
} from "livekit-client";
|
||||
|
||||
import { type CallViewModel } from "../state/CallViewModel/CallViewModel";
|
||||
@@ -27,6 +34,241 @@ export interface CallSummary {
|
||||
/** Time the local connection quality was reported poor/lost, ms. */
|
||||
poorMs: number;
|
||||
verdict: CallQualityVerdict;
|
||||
/** [cinny #253] The breakdown behind the host's "Details" card. */
|
||||
details?: CallSummaryDetails;
|
||||
}
|
||||
|
||||
/** [cinny #253] Everything the host's post-call Details card shows. */
|
||||
export interface CallSummaryDetails {
|
||||
/** Other Matrix users seen in the call (`@user:server`), first-seen order. */
|
||||
participants: string[];
|
||||
/** Most people in the call at once, you included. */
|
||||
peakParticipants: number;
|
||||
quality: {
|
||||
/** Your connection reported poor/lost, ms. */
|
||||
youPoorMs: number;
|
||||
/** Each other person's connection reported poor/lost, ms (only > 0). */
|
||||
othersPoorMs: Record<string, number>;
|
||||
/** Everyone, you included, poor at the same time (points at the server). */
|
||||
everyonePoorMs: number;
|
||||
};
|
||||
network: {
|
||||
rttMedianMs?: number;
|
||||
rttP95Ms?: number;
|
||||
/** Your microphone's packets the SFU reported lost, %. */
|
||||
outboundLossPct?: number;
|
||||
/** Packets lost on the audio you received, %. */
|
||||
inboundLossPct?: number;
|
||||
jitterMs?: number;
|
||||
};
|
||||
activity: { speakingMs: number; mutedMs: number; screenshareMs: number };
|
||||
}
|
||||
|
||||
/** Time spent in an on/off state, e.g. muted. */
|
||||
class Stopwatch {
|
||||
private since: number | undefined;
|
||||
|
||||
private total = 0;
|
||||
|
||||
public set(on: boolean, now: number): void {
|
||||
if (on && this.since === undefined) this.since = now;
|
||||
if (!on && this.since !== undefined) {
|
||||
this.total += now - this.since;
|
||||
this.since = undefined;
|
||||
}
|
||||
}
|
||||
|
||||
public get on(): boolean {
|
||||
return this.since !== undefined;
|
||||
}
|
||||
|
||||
public ms(now: number): number {
|
||||
return this.total + (this.since === undefined ? 0 : now - this.since);
|
||||
}
|
||||
}
|
||||
|
||||
/** `@user:server:DEVICE` (a LiveKit identity) → `@user:server`. */
|
||||
export function matrixUserOf(identity: string): string {
|
||||
if (!identity.startsWith("@")) return identity;
|
||||
const first = identity.indexOf(":");
|
||||
const second = first === -1 ? -1 : identity.indexOf(":", first + 1);
|
||||
return second === -1 ? identity : identity.slice(0, second);
|
||||
}
|
||||
|
||||
const percentile = (sorted: number[], p: number): number =>
|
||||
sorted[Math.min(sorted.length - 1, Math.floor(p * sorted.length))];
|
||||
|
||||
/**
|
||||
* [cinny #253] Who was in the call, whose connection was poor and when,
|
||||
* network numbers, and your own activity. Fed by LiveKit events and periodic
|
||||
* stats samples; nothing leaves the frame except the final summary, sent to
|
||||
* the host only.
|
||||
*/
|
||||
export class CallDetailsTracker {
|
||||
// user -> identities (devices) currently in the call
|
||||
private readonly devices = new Map<string, Set<string>>();
|
||||
|
||||
private readonly seen: string[] = [];
|
||||
|
||||
private peak = 1;
|
||||
|
||||
private readonly poor = new Map<string, Stopwatch>();
|
||||
|
||||
private readonly youPoor = new Stopwatch();
|
||||
|
||||
private readonly everyonePoor = new Stopwatch();
|
||||
|
||||
private readonly rtts: number[] = [];
|
||||
|
||||
private readonly jitters: number[] = [];
|
||||
|
||||
private readonly sent = new Map<string, { packets: number; lost: number }>();
|
||||
|
||||
private readonly received = new Map<
|
||||
string,
|
||||
{ packets: number; lost: number }
|
||||
>();
|
||||
|
||||
private readonly speaking = new Stopwatch();
|
||||
|
||||
private readonly muted = new Stopwatch();
|
||||
|
||||
private readonly sharing = new Stopwatch();
|
||||
|
||||
public constructor(private readonly now: () => number = () => Date.now()) {}
|
||||
|
||||
/** Idempotent per device: safe to call for everyone already present too. */
|
||||
public remoteJoined(identity: string): void {
|
||||
const user = matrixUserOf(identity);
|
||||
let ids = this.devices.get(user);
|
||||
if (!ids) this.devices.set(user, (ids = new Set()));
|
||||
ids.add(identity);
|
||||
if (!this.seen.includes(user)) this.seen.push(user);
|
||||
this.peak = Math.max(this.peak, this.devices.size + 1);
|
||||
this.updateEveryone();
|
||||
}
|
||||
|
||||
public remoteLeft(identity: string): void {
|
||||
const user = matrixUserOf(identity);
|
||||
const ids = this.devices.get(user);
|
||||
ids?.delete(identity);
|
||||
if (!ids || ids.size === 0) {
|
||||
this.devices.delete(user);
|
||||
this.poor.get(user)?.set(false, this.now());
|
||||
}
|
||||
this.updateEveryone();
|
||||
}
|
||||
|
||||
/** `identity` undefined = the local participant. */
|
||||
public setQuality(
|
||||
identity: string | undefined,
|
||||
quality: ConnectionQuality,
|
||||
): void {
|
||||
if (quality === ConnectionQuality.Unknown) return;
|
||||
const bad =
|
||||
quality === ConnectionQuality.Poor || quality === ConnectionQuality.Lost;
|
||||
if (identity === undefined) this.youPoor.set(bad, this.now());
|
||||
else {
|
||||
const user = matrixUserOf(identity);
|
||||
let sw = this.poor.get(user);
|
||||
if (!sw) this.poor.set(user, (sw = new Stopwatch()));
|
||||
sw.set(bad, this.now());
|
||||
}
|
||||
this.updateEveryone();
|
||||
}
|
||||
|
||||
private updateEveryone(): void {
|
||||
const others = [...this.devices.keys()];
|
||||
const all =
|
||||
this.youPoor.on &&
|
||||
others.length > 0 &&
|
||||
others.every((u) => this.poor.get(u)?.on);
|
||||
this.everyonePoor.set(all, this.now());
|
||||
}
|
||||
|
||||
public addRtt(seconds: number | undefined): void {
|
||||
if (typeof seconds === "number" && Number.isFinite(seconds) && seconds >= 0)
|
||||
this.rtts.push(seconds * 1000);
|
||||
}
|
||||
|
||||
public addJitter(seconds: number | undefined): void {
|
||||
if (typeof seconds === "number" && Number.isFinite(seconds) && seconds >= 0)
|
||||
this.jitters.push(seconds * 1000);
|
||||
}
|
||||
|
||||
/** Cumulative counters for one sent/received track (latest value wins). */
|
||||
public setSent(key: string, packets?: number, lost?: number): void {
|
||||
if (typeof packets === "number")
|
||||
this.sent.set(key, { packets, lost: Math.max(0, lost ?? 0) });
|
||||
}
|
||||
|
||||
public setReceived(key: string, packets?: number, lost?: number): void {
|
||||
if (typeof packets === "number")
|
||||
this.received.set(key, { packets, lost: Math.max(0, lost ?? 0) });
|
||||
}
|
||||
|
||||
public setSpeaking(on: boolean): void {
|
||||
this.speaking.set(on, this.now());
|
||||
}
|
||||
|
||||
public setMuted(on: boolean): void {
|
||||
this.muted.set(on, this.now());
|
||||
}
|
||||
|
||||
public setScreensharing(on: boolean): void {
|
||||
this.sharing.set(on, this.now());
|
||||
}
|
||||
|
||||
public details(): CallSummaryDetails {
|
||||
const now = this.now();
|
||||
const othersPoorMs: Record<string, number> = {};
|
||||
for (const [user, sw] of this.poor) {
|
||||
const ms = sw.ms(now);
|
||||
if (ms > 0) othersPoorMs[user] = ms;
|
||||
}
|
||||
const lossPct = (
|
||||
m: Map<string, { packets: number; lost: number }>,
|
||||
): number | undefined => {
|
||||
let packets = 0;
|
||||
let lost = 0;
|
||||
for (const v of m.values()) {
|
||||
packets += v.packets;
|
||||
lost += v.lost;
|
||||
}
|
||||
return packets + lost > 0
|
||||
? Math.round((lost / (packets + lost)) * 1000) / 10
|
||||
: undefined;
|
||||
};
|
||||
const rtts = [...this.rtts].sort((a, b) => a - b);
|
||||
const network: CallSummaryDetails["network"] = {};
|
||||
if (rtts.length) {
|
||||
network.rttMedianMs = Math.round(percentile(rtts, 0.5));
|
||||
network.rttP95Ms = Math.round(percentile(rtts, 0.95));
|
||||
}
|
||||
const out = lossPct(this.sent);
|
||||
if (out !== undefined) network.outboundLossPct = out;
|
||||
const inn = lossPct(this.received);
|
||||
if (inn !== undefined) network.inboundLossPct = inn;
|
||||
if (this.jitters.length)
|
||||
network.jitterMs = Math.round(
|
||||
this.jitters.reduce((a, b) => a + b, 0) / this.jitters.length,
|
||||
);
|
||||
return {
|
||||
participants: [...this.seen],
|
||||
peakParticipants: this.peak,
|
||||
quality: {
|
||||
youPoorMs: this.youPoor.ms(now),
|
||||
othersPoorMs,
|
||||
everyonePoorMs: this.everyonePoor.ms(now),
|
||||
},
|
||||
network,
|
||||
activity: {
|
||||
speakingMs: this.speaking.ms(now),
|
||||
mutedMs: this.muted.ms(now),
|
||||
screenshareMs: this.sharing.ms(now),
|
||||
},
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -93,13 +335,48 @@ export class CallQualityTracker {
|
||||
export function startLotusCallSummary(vm: CallViewModel): () => void {
|
||||
if (!widget) return () => undefined;
|
||||
const tracker = new CallQualityTracker();
|
||||
const details = new CallDetailsTracker();
|
||||
let sent = false;
|
||||
const send = (): void => {
|
||||
if (sent) return;
|
||||
sent = true;
|
||||
lotusSendToHost(LotusWidgetActions.CallSummary, tracker.summary());
|
||||
clearInterval(sampler);
|
||||
lotusSendToHost(LotusWidgetActions.CallSummary, {
|
||||
...tracker.summary(),
|
||||
details: details.details(),
|
||||
});
|
||||
};
|
||||
|
||||
// [cinny #253] Network numbers: sample sender/receiver stats every 5 s.
|
||||
const sample = async (room: LivekitRoom): Promise<void> => {
|
||||
const mic = room.localParticipant.getTrackPublication(
|
||||
Track.Source.Microphone,
|
||||
)?.track as LocalAudioTrack | undefined;
|
||||
const s = await mic?.getSenderStats?.().catch(() => undefined);
|
||||
if (s) {
|
||||
details.addRtt(s.roundTripTime);
|
||||
details.setSent(`${room.name}:mic`, s.packetsSent, s.packetsLost);
|
||||
}
|
||||
for (const p of room.remoteParticipants.values()) {
|
||||
for (const pub of p.audioTrackPublications.values()) {
|
||||
const t = pub.track as RemoteAudioTrack | undefined;
|
||||
const r = await t?.getReceiverStats?.().catch(() => undefined);
|
||||
if (r) {
|
||||
details.addJitter(r.jitter);
|
||||
details.setReceived(
|
||||
`${room.name}:${p.identity}:${pub.trackSid}`,
|
||||
r.packetsReceived,
|
||||
r.packetsLost,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
};
|
||||
const sampler = setInterval(() => {
|
||||
for (const room of listeners.keys())
|
||||
void sample(room).catch(() => undefined);
|
||||
}, 5_000);
|
||||
|
||||
const listeners = new Map<LivekitRoom, () => void>();
|
||||
const attach = (room: LivekitRoom): void => {
|
||||
const onQuality = (
|
||||
@@ -107,20 +384,69 @@ export function startLotusCallSummary(vm: CallViewModel): () => void {
|
||||
participant: Participant,
|
||||
): void => {
|
||||
if (participant.isLocal) tracker.setQuality(quality);
|
||||
details.setQuality(
|
||||
participant.isLocal ? undefined : participant.identity,
|
||||
quality,
|
||||
);
|
||||
};
|
||||
// LiveKit doesn't send ParticipantConnected for people already in the
|
||||
// room when we connect, and the mic isn't published yet at attach time:
|
||||
// record both once the connection is up.
|
||||
const onConnected = (): void => {
|
||||
tracker.start();
|
||||
for (const p of room.remoteParticipants.values())
|
||||
details.remoteJoined(p.identity);
|
||||
details.setMuted(!room.localParticipant.isMicrophoneEnabled);
|
||||
};
|
||||
const onConnected = (): void => tracker.start();
|
||||
const onReconnecting = (): void => tracker.reconnect();
|
||||
const onDisconnected = (): void => send();
|
||||
const onJoined = (p: RemoteParticipant): void =>
|
||||
details.remoteJoined(p.identity);
|
||||
const onLeft = (p: RemoteParticipant): void =>
|
||||
details.remoteLeft(p.identity);
|
||||
const isLocalMic = (pub: TrackPublication, p: Participant): boolean =>
|
||||
p.isLocal && pub.source === Track.Source.Microphone;
|
||||
const onMuted = (pub: TrackPublication, p: Participant): void => {
|
||||
if (isLocalMic(pub, p)) details.setMuted(true);
|
||||
};
|
||||
const onUnmuted = (pub: TrackPublication, p: Participant): void => {
|
||||
if (isLocalMic(pub, p)) details.setMuted(false);
|
||||
};
|
||||
const onLocalPublished = (pub: LocalTrackPublication): void => {
|
||||
if (pub.source === Track.Source.ScreenShare)
|
||||
details.setScreensharing(true);
|
||||
if (pub.source === Track.Source.Microphone) details.setMuted(pub.isMuted);
|
||||
};
|
||||
const onLocalUnpublished = (pub: LocalTrackPublication): void => {
|
||||
if (pub.source === Track.Source.ScreenShare)
|
||||
details.setScreensharing(false);
|
||||
};
|
||||
const onSpeaking = (speaking: boolean): void =>
|
||||
details.setSpeaking(speaking);
|
||||
room.on(RoomEvent.ConnectionQualityChanged, onQuality);
|
||||
room.on(RoomEvent.Connected, onConnected);
|
||||
room.on(RoomEvent.Reconnecting, onReconnecting);
|
||||
room.on(RoomEvent.Disconnected, onDisconnected);
|
||||
if (room.state === "connected") tracker.start();
|
||||
room.on(RoomEvent.ParticipantConnected, onJoined);
|
||||
room.on(RoomEvent.ParticipantDisconnected, onLeft);
|
||||
room.on(RoomEvent.TrackMuted, onMuted);
|
||||
room.on(RoomEvent.TrackUnmuted, onUnmuted);
|
||||
room.on(RoomEvent.LocalTrackPublished, onLocalPublished);
|
||||
room.on(RoomEvent.LocalTrackUnpublished, onLocalUnpublished);
|
||||
room.localParticipant.on(ParticipantEvent.IsSpeakingChanged, onSpeaking);
|
||||
if (room.state === "connected") onConnected();
|
||||
listeners.set(room, () => {
|
||||
room.off(RoomEvent.ConnectionQualityChanged, onQuality);
|
||||
room.off(RoomEvent.Connected, onConnected);
|
||||
room.off(RoomEvent.Reconnecting, onReconnecting);
|
||||
room.off(RoomEvent.Disconnected, onDisconnected);
|
||||
room.off(RoomEvent.ParticipantConnected, onJoined);
|
||||
room.off(RoomEvent.ParticipantDisconnected, onLeft);
|
||||
room.off(RoomEvent.TrackMuted, onMuted);
|
||||
room.off(RoomEvent.TrackUnmuted, onUnmuted);
|
||||
room.off(RoomEvent.LocalTrackPublished, onLocalPublished);
|
||||
room.off(RoomEvent.LocalTrackUnpublished, onLocalUnpublished);
|
||||
room.localParticipant.off(ParticipantEvent.IsSpeakingChanged, onSpeaking);
|
||||
});
|
||||
};
|
||||
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
/*
|
||||
Copyright 2026 Lotus Guild
|
||||
|
||||
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
|
||||
Please see LICENSE in the repository root for full details.
|
||||
*/
|
||||
|
||||
import { beforeEach, expect, test, vi } from "vitest";
|
||||
|
||||
import { LotusWidgetActions } from "./lotusActions";
|
||||
import {
|
||||
clearScreenshareAudio,
|
||||
reportScreenshareAudio,
|
||||
resetScreenshareAudioPresence,
|
||||
} from "./lotusScreenshareAudio";
|
||||
|
||||
const send = vi.fn().mockResolvedValue({});
|
||||
|
||||
vi.mock("../widget", () => ({
|
||||
widget: {
|
||||
api: { transport: { send: (...a: unknown[]): unknown => send(...a) } },
|
||||
},
|
||||
}));
|
||||
|
||||
const sent = (): unknown[] =>
|
||||
send.mock.calls
|
||||
.filter(([action]) => action === LotusWidgetActions.ScreenshareAudioState)
|
||||
.map(([, data]) => data);
|
||||
|
||||
beforeEach(() => {
|
||||
resetScreenshareAudioPresence();
|
||||
send.mockClear();
|
||||
});
|
||||
|
||||
test("[cinny #277] reports presence only when the combined answer changes", () => {
|
||||
reportScreenshareAudio("sfu-a", false);
|
||||
reportScreenshareAudio("sfu-a", false);
|
||||
reportScreenshareAudio("sfu-a", true);
|
||||
reportScreenshareAudio("sfu-a", true);
|
||||
reportScreenshareAudio("sfu-a", false);
|
||||
expect(sent()).toEqual([
|
||||
{ present: false },
|
||||
{ present: true },
|
||||
{ present: false },
|
||||
]);
|
||||
});
|
||||
|
||||
test("[cinny #277] any room with screenshare audio counts as present", () => {
|
||||
reportScreenshareAudio("sfu-a", true);
|
||||
reportScreenshareAudio("sfu-b", false);
|
||||
expect(sent()).toEqual([{ present: true }]);
|
||||
// A share is still playing in sfu-b after sfu-a's stops.
|
||||
reportScreenshareAudio("sfu-b", true);
|
||||
reportScreenshareAudio("sfu-a", false);
|
||||
expect(sent()).toEqual([{ present: true }]);
|
||||
// The renderer for sfu-b unmounts (call left): nothing is playing.
|
||||
clearScreenshareAudio("sfu-b");
|
||||
expect(sent()).toEqual([{ present: true }, { present: false }]);
|
||||
});
|
||||
@@ -8,6 +8,8 @@ Please see LICENSE in the repository root for full details.
|
||||
import { BehaviorSubject } from "rxjs";
|
||||
|
||||
import { globalScope } from "../state/ObservableScope";
|
||||
import { LotusWidgetActions } from "./lotusActions";
|
||||
import { lotusSendToHost } from "./lotusWidget";
|
||||
|
||||
/**
|
||||
* Whether the host has muted remote screenshare AUDIO (`io.lotus.set_deafen`
|
||||
@@ -25,3 +27,40 @@ export const setScreenshareAudioMuted$ = new BehaviorSubject(false);
|
||||
export const muteScreenshareAudio$ = globalScope.behavior(
|
||||
setScreenshareAudioMuted$,
|
||||
);
|
||||
|
||||
/**
|
||||
* [cinny #277] Whether any remote participant's screenshare AUDIO is being
|
||||
* received, so the host shows its "Mute Screenshare Audio" control only when
|
||||
* there is something to mute. Each `LivekitRoomAudioRenderer` (one per SFU)
|
||||
* reports for its own room; the host gets `io.lotus.screenshare_audio_state
|
||||
* { present }` whenever the combined answer changes.
|
||||
*/
|
||||
const presentByRoom = new Map<string, boolean>();
|
||||
let lastSent: boolean | undefined;
|
||||
|
||||
function sendIfChanged(): void {
|
||||
const present = [...presentByRoom.values()].some(Boolean);
|
||||
if (present === lastSent) return;
|
||||
lastSent = present;
|
||||
lotusSendToHost(LotusWidgetActions.ScreenshareAudioState, { present });
|
||||
}
|
||||
|
||||
export function reportScreenshareAudio(
|
||||
roomKey: string,
|
||||
present: boolean,
|
||||
): void {
|
||||
presentByRoom.set(roomKey, present);
|
||||
sendIfChanged();
|
||||
}
|
||||
|
||||
/** The renderer for `roomKey` went away (call left, SFU switched). */
|
||||
export function clearScreenshareAudio(roomKey: string): void {
|
||||
presentByRoom.delete(roomKey);
|
||||
sendIfChanged();
|
||||
}
|
||||
|
||||
/** Test-only: forget all reports. */
|
||||
export function resetScreenshareAudioPresence(): void {
|
||||
presentByRoom.clear();
|
||||
lastSent = undefined;
|
||||
}
|
||||
|
||||
@@ -451,14 +451,16 @@ describe.each(modes)("CallViewModel (%s mode)", (mode) => {
|
||||
});
|
||||
});
|
||||
|
||||
test("one-on-one mobile layout shows local tile when video is enabled", () => {
|
||||
// [lotus / cinny #265] The local tile stays on mobile with the camera off
|
||||
// (upstream hid it), so you can see yourself in a voice call.
|
||||
test("one-on-one mobile layout shows the local tile whether or not video is enabled", () => {
|
||||
withTestScheduler(({ behavior, schedule, expectObservable }) => {
|
||||
// Local participant enables their video, then disables it
|
||||
const videoInputMarbles = " ny--n";
|
||||
// While tile is shown, tap the screen twice
|
||||
// Tap the screen twice
|
||||
const tapScreenInputMarbles = "--aa-";
|
||||
// Layout should show local tile, make it small, enlarge it again, then hide it
|
||||
const expectedLayoutMarbles = "abcba";
|
||||
// The local tile is shown throughout; taps make it small, then large again
|
||||
const expectedLayoutMarbles = "b-cb";
|
||||
|
||||
withCallViewModel(
|
||||
{
|
||||
@@ -477,12 +479,6 @@ describe.each(modes)("CallViewModel (%s mode)", (mode) => {
|
||||
expectObservable(summarizeLayout$(vm.layout$)).toBe(
|
||||
expectedLayoutMarbles,
|
||||
{
|
||||
a: {
|
||||
type: "one-on-one-mobile",
|
||||
spotlight: [`${aliceId}:0`],
|
||||
pip: undefined,
|
||||
pipSize: "lg",
|
||||
},
|
||||
b: {
|
||||
type: "one-on-one-mobile",
|
||||
spotlight: [`${aliceId}:0`],
|
||||
@@ -518,7 +514,7 @@ describe.each(modes)("CallViewModel (%s mode)", (mode) => {
|
||||
a: {
|
||||
type: "one-on-one-mobile",
|
||||
spotlight: [`${aliceId}:0`],
|
||||
pip: undefined,
|
||||
pip: `${localId}:0`,
|
||||
pipSize: "lg",
|
||||
},
|
||||
});
|
||||
@@ -566,7 +562,7 @@ describe.each(modes)("CallViewModel (%s mode)", (mode) => {
|
||||
a: {
|
||||
type: "one-on-one-mobile",
|
||||
spotlight: [`${aliceId}:0`],
|
||||
pip: undefined,
|
||||
pip: `${localId}:0`,
|
||||
pipSize: "sm",
|
||||
},
|
||||
b: {
|
||||
@@ -765,7 +761,7 @@ describe.each(modes)("CallViewModel (%s mode)", (mode) => {
|
||||
// This is the expected one-on-one layout for a narrow window
|
||||
type: "one-on-one-mobile",
|
||||
spotlight: [`${aliceId}:0`],
|
||||
pip: undefined,
|
||||
pip: `${localId}:0`,
|
||||
pipSize: "lg",
|
||||
},
|
||||
b: {
|
||||
|
||||
@@ -1220,19 +1220,22 @@ export function createCallViewModel$(
|
||||
}),
|
||||
);
|
||||
|
||||
// [lotus / cinny #265] Upstream drops the local PiP on mobile while your
|
||||
// camera is off; in Lotus voice rooms that is almost always, so on a phone
|
||||
// you never saw yourself (or your own speaking ring). Show it as desktop
|
||||
// does: the tile falls back to your avatar.
|
||||
const oneOnOneMobileLayoutMedia$: Observable<OneOnOneMobileLayoutMedia | null> =
|
||||
oneOnOneLayoutMedia$.pipe(
|
||||
switchMap((media) => {
|
||||
if (media === null) return of(null);
|
||||
return media.local.videoEnabled$.pipe(
|
||||
map((videoEnabled) => ({
|
||||
type: "one-on-one-mobile" as const,
|
||||
edgeToEdge: true as const,
|
||||
spotlight: media.remote,
|
||||
pip: videoEnabled ? media.local : undefined,
|
||||
})),
|
||||
);
|
||||
}),
|
||||
map((media) =>
|
||||
media === null
|
||||
? null
|
||||
: {
|
||||
type: "one-on-one-mobile" as const,
|
||||
edgeToEdge: true as const,
|
||||
spotlight: media.remote,
|
||||
pip: media.local,
|
||||
},
|
||||
),
|
||||
);
|
||||
|
||||
const pipLayoutMedia$: Observable<LayoutMedia> = spotlight$.pipe(
|
||||
|
||||
Reference in New Issue
Block a user