LotusGuild/element-call
3
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

merge: upstream v0.25.0 into lotus

Browse Source 
Merge upstream element-hq/element-call tag v0.25.0 into the Lotus fork
(previous base: v0.20.1; actual merge-base v0.20.1-rc.1). Every Lotus
feature and all six io.lotus.* widget actions are preserved. Version
bumped to 0.25.0-lotus.1.

Conflict files and how each Lotus hunk was re-expressed:

* src/state/CallViewModel/remoteMembers/ConnectionFactory.ts
  Upstream moved echoCancellation/noiseSuppression/autoGainControl from
  constructor params (fed by URL params) to persisted Settings
  (settings.ts) with a developer-settings UI. The cinny host still drives
  these per call via URL params (noiseSuppression=false /
  autoGainControl=false when the in-source ML denoiser is active, so the
  model gets a raw mic) - taking upstream verbatim would silently break
  the ML denoise tier. Re-wired as AND semantics in generateRoomOption():
  a constraint is enabled only if BOTH the Setting and the URL param
  allow it. Params default to true, so with no params this is
  byte-for-byte upstream behaviour. Upstream's own echoCancellation /
  noiseSuppression URL params (still parsed but dead in v0.25.0) work
  again as a side effect. Lotus autoGainControl URL param kept in
  UrlParams.ts (auto-merged, unchanged).

* src/state/CallViewModel/remoteMembers/ECConnectionFactory.test.ts
  Took upstream (tests now drive via Settings). The lost Lotus coverage
  is restored in a NEW colocated file src/lotus/lotusAudioConstraints.test.ts
  (3 tests) so the upstream test file stays pristine. Verified the new
  test fails against pure-upstream ConnectionFactory and passes with the
  re-wiring.

* src/state/CallViewModel/CallViewModel.ts
  Three small hunks: kept both the Lotus `userMedia$` interface member
  and upstream's new `keyRotationSuppressed$`; dropped the three Lotus
  audio constructor args (mechanism removed upstream, see above); kept
  both in the returned object. The [lotus #4] overrideSpotlight$ routing,
  manualSpotlightUserId$ and setManualSpotlight auto-merged; verified
  against upstream's changed ringingMedia$ (now single-or-null instead
  of array) - the merge correctly took upstream's outer branch and the
  inner screenShares$/spotlightSpeaker$ logic that lotusSpotlight.ts
  mirrors is unchanged upstream.

* src/index.css
  Kept both: Lotus lotus-transparent / lotus-theme blocks and upstream's
  new body[data-background="gradient"]::before full-viewport gradient.
  The naive merge swallowed the closing brace of body.lotus-theme -
  restored. Added a rule hiding the new gradient pseudo-element under
  body.lotus-transparent, since it would otherwise paint over the
  transparent body and hide the host wallpaper.

* src/components/CallFooterViewModel.tsx, src/components/CallFooter.stories.tsx
  No Lotus content - pure upstream-vs-upstream conflicts caused by the
  merge base being v0.20.1-rc.1. Took upstream (layoutMode ->
  layoutSwitchVm; setLayoutMode removed). No Lotus code uses
  setGridMode/layoutMode.

Non-conflicting but reviewed:

* src/widget.ts auto-merged cleanly. Upstream's removal of .well-known
  transport advertisement and the new RTC-transport capability request
  did not touch the action registration loop the LOTUS_TO_WIDGET_ACTIONS
  spread and widget.lazyActions ride on - nothing to re-wire.
* src/room/InCallView.tsx, src/useAudioContext.tsx, src/useTheme.ts,
  src/tile/MediaView.tsx(+.module.css), src/UrlParams.ts(+test),
  all *.module.css and .gitea/workflows/ci.yml auto-merged; each diff
  against v0.25.0 was checked to equal the original Lotus hunk.
* src/button/Button.module.css: the merge appended an exact duplicate
  of upstream's `.rotate`/`@keyframes spin` block (rc.1 merge-base
  artefact) - reset to upstream verbatim.
* src/grid/OneOnOnePortraitLayout.module.css was renamed upstream to
  OneOnOneMobileLayout.module.css; git followed the rename and the Lotus
  safe-area PiP inset fix applies there (the --content-inset-* vars it
  uses still exist upstream).

Tooling changes inherited from upstream that affect the fork:

* eslint + prettier were replaced by oxlint + oxfmt (`pnpm lint:oxlint`,
  `pnpm format:check`). oxlint flagged 10 issues, all in src/lotus/*:
  8x no-meaningless-void-operator (dropped the `void` before void-typed
  widget transport.reply / callbacks - no behaviour change), 1x
  consistent-type-imports (lotusWidget.ts: `import type`), and 2x
  unicorn/no-useless-spread in lotusAudioInject.ts which are FALSE
  POSITIVES - `[...activeClips]` is a required defensive copy because
  abort() deletes from the Set during iteration; suppressed with an
  explanatory eslint-disable-next-line. oxfmt reformatted 7 Lotus
  touched files (whitespace only).
* packageManager bumped by upstream to pnpm@11.21.0, which requires
  Node >= 22.13 (uses node:sqlite). Node 20 cannot run it; pnpm 10.33
  cannot read the new lockfile either (matrix-js-sdk is now a git
  dependency on develop, using a version-union pnpm 10 rejects). Fork CI
  already uses Node 24 (.node-version), so CI is unaffected.
* matrix-js-sdk is now github:matrix-org/matrix-js-sdk#develop (pinned
  by commit in pnpm-lock.yaml).

Lotus behaviour NOT preserved: none found.

Verification (Node 24.11.1, pnpm 11.21.0): pnpm install --frozen-lockfile
OK (lockfile taken from upstream unchanged, no regeneration needed);
tsc clean; oxlint clean; oxfmt --check clean; knip exit 0 (2 config
hints in upstream knip.ts only); vitest unit 84 files / 627 passed /
9 skipped; build:embedded OK, staged to embedded/web/dist (44M), all
six io.lotus.* action strings present in the bundle.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPmy3tPq869XDW4njjVaKA
This commit is contained in:
Lotus CI
2026-09-12 11:35:57 -04:00
co-authored by Claude Fable 5.1
parent e36aef8aa3 efde51b8c1
commit c6f9727870
243 changed files with 13311 additions and 10280 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/state/AndroidControlledAudioOutput.test.ts
+1 -1
View File
@@ -253,7 +253,7 @@ describe("Test mappings", () => {
});
describe("Test select a device", () => {
it(`Switch to correct device `, () => {
it(`Switch to correct device`, () => {
withTestScheduler(({ cold, schedule, expectObservable, flush }) => {
const controlledAudioOutput = new AndroidControlledAudioOutput(
cold("a", { a: FULL_DEVICE_LIST }),
src/state/CallViewModel/CallNotificationLifecycle.test.ts
+270 -306
View File
@@ -5,17 +5,19 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { describe, it } from "vitest";
import { test } from "vitest";
import {
EventType,
type IEvent,
type IRoomTimelineData,
MatrixEvent,
type Room,
} from "matrix-js-sdk";
import { type RTCCallIntent } from "matrix-js-sdk/lib/matrixrtc";
import { map, mergeMap, NEVER, type Observable, startWith } from "rxjs";
import { withTestScheduler } from "../../utils/test";
import {
alice,
aliceRtcMember,
local,
localRtcMember,
@@ -23,9 +25,10 @@ import {
import {
type CallNotificationWrapper,
createCallNotificationLifecycle$,
type Props as CallNotificationLifecycleProps,
type RingAttempt,
} from "./CallNotificationLifecycle";
import { trackEpoch } from "../ObservableScope";
import { Epoch, trackEpoch } from "../ObservableScope";
import { constant } from "../Behavior";
function mockRingEvent(
eventId: string,
@@ -40,311 +43,272 @@ function mockRingEvent(
} as unknown as CallNotificationWrapper;
}
describe("waitForCallPickup$", () => {
it("unknown -> ringing -> timeout when notified and nobody joins", () => {
withTestScheduler(({ scope, expectObservable, behavior, hot }) => {
// No one ever joins (only local user)
const props: CallNotificationLifecycleProps = {
scope,
memberships$: scope.behavior(
behavior("a", { a: [] }).pipe(trackEpoch()),
),
sentCallNotification$: hot("10ms a", {
a: mockRingEvent("$notif1", 30),
}),
receivedDecline$: hot(""),
options: {
waitForCallPickup: true,
autoLeaveWhenOthersLeft: false,
},
localUser: localRtcMember,
};
const defaultProps = {
memberships$: constant(new Epoch([])),
matrixRoomMembers$: constant(new Map([[alice.userId, alice]])),
receivedDecline$: NEVER,
options: {
waitForCallPickup: true,
autoLeaveWhenOthersLeft: false,
},
localUser: localRtcMember,
};
const lifecycle = createCallNotificationLifecycle$(props);
function summarizeRingAttempts$(
ringAttempts$: Observable<RingAttempt>,
): Observable<
| { intent: RTCCallIntent; recipient: string }
| { outcome: "accept" | "decline" | "timeout" }
> {
return ringAttempts$.pipe(
mergeMap(({ intent, recipient, outcome$ }) =>
outcome$.pipe(
map((outcome) => ({ outcome })),
startWith({ intent, recipient }),
),
),
);
}
expectObservable(lifecycle.callPickupState$).toBe("a 9ms b 29ms c", {
a: "unknown",
b: "ringing",
c: "timeout",
});
test("no ring attempt when waitForCallPickup=false", () => {
withTestScheduler(({ scope, expectObservable, hot }) => {
const { ringAttempts$ } = createCallNotificationLifecycle$({
scope,
...defaultProps,
sentCallNotification$: hot("-a", {
a: mockRingEvent("$notif1", 30),
}),
options: { ...defaultProps.options, waitForCallPickup: false },
});
});
it("ringing -> success if someone joins before timeout is reached", () => {
withTestScheduler(({ scope, hot, behavior, expectObservable }) => {
// Someone joins at 20ms (both LiveKit participant and MatrixRTC member)
const props: CallNotificationLifecycleProps = {
scope,
memberships$: scope.behavior(
behavior("a 19ms b", {
a: [localRtcMember],
b: [localRtcMember, aliceRtcMember],
}).pipe(trackEpoch()),
),
sentCallNotification$: hot("5ms a", {
a: mockRingEvent("$notif2", 100),
}),
receivedDecline$: hot(""),
options: {
waitForCallPickup: true,
autoLeaveWhenOthersLeft: false,
},
localUser: localRtcMember,
};
const lifecycle = createCallNotificationLifecycle$(props);
expectObservable(lifecycle.callPickupState$).toBe("a 4ms b 14ms c", {
a: "unknown",
b: "ringing",
c: "success",
});
});
});
it("success when someone joins before we notify", () => {
withTestScheduler(({ scope, hot, behavior, expectObservable }) => {
// Someone joins at 20ms (both LiveKit participant and MatrixRTC member)
const props: CallNotificationLifecycleProps = {
scope,
memberships$: scope.behavior(
behavior("a 9ms b", {
a: [localRtcMember],
b: [localRtcMember, aliceRtcMember],
}).pipe(trackEpoch()),
),
sentCallNotification$: hot("20ms a", {
a: mockRingEvent("$notif2", 50),
}),
receivedDecline$: hot(""),
options: {
waitForCallPickup: true,
autoLeaveWhenOthersLeft: false,
},
localUser: localRtcMember,
};
const lifecycle = createCallNotificationLifecycle$(props);
expectObservable(lifecycle.callPickupState$).toBe("a 9ms b", {
a: "unknown",
b: "success",
});
});
});
it("notify without lifetime -> immediate timeout", () => {
withTestScheduler(({ scope, hot, behavior, expectObservable }) => {
// Someone joins at 20ms (both LiveKit participant and MatrixRTC member)
const props: CallNotificationLifecycleProps = {
scope,
memberships$: scope.behavior(
behavior("a", {
a: [localRtcMember],
}).pipe(trackEpoch()),
),
sentCallNotification$: hot("10ms a", {
a: mockRingEvent("$notif2", undefined),
}),
receivedDecline$: hot(""),
options: {
waitForCallPickup: true,
autoLeaveWhenOthersLeft: false,
},
localUser: localRtcMember,
};
const lifecycle = createCallNotificationLifecycle$(props);
expectObservable(lifecycle.callPickupState$).toBe("a 9ms b", {
a: "unknown",
b: "timeout",
});
});
});
it("stays null when waitForCallPickup=false", () => {
withTestScheduler(({ scope, hot, behavior, expectObservable }) => {
// Someone joins at 20ms (both LiveKit participant and MatrixRTC member)
const validProps: CallNotificationLifecycleProps = {
scope,
memberships$: scope.behavior(
behavior("a--b", {
a: [localRtcMember],
b: [localRtcMember, aliceRtcMember],
}).pipe(trackEpoch()),
),
sentCallNotification$: hot("10ms a", {
a: mockRingEvent("$notif5", 30),
}),
receivedDecline$: hot(""),
options: {
waitForCallPickup: true,
autoLeaveWhenOthersLeft: false,
},
localUser: localRtcMember,
};
const propsDeactivated = {
...validProps,
options: {
...validProps.options,
waitForCallPickup: false,
},
};
const lifecycle = createCallNotificationLifecycle$(propsDeactivated);
expectObservable(lifecycle.callPickupState$).toBe("n", {
n: null,
});
const lifecycleReference = createCallNotificationLifecycle$(validProps);
expectObservable(lifecycleReference.callPickupState$).toBe("u--s", {
u: "unknown",
s: "success",
});
});
});
it("decline before timeout window ends -> decline", () => {
withTestScheduler(({ scope, hot, behavior, expectObservable }) => {
// Someone joins at 20ms (both LiveKit participant and MatrixRTC member)
const props: CallNotificationLifecycleProps = {
scope,
memberships$: scope.behavior(
behavior("a", {
a: [localRtcMember],
}).pipe(trackEpoch()),
),
sentCallNotification$: hot("10ms a", {
a: mockRingEvent("$decl1", 50),
}),
receivedDecline$: hot("40ms d", {
d: [
new MatrixEvent({
type: EventType.RTCDecline,
content: {
"m.relates_to": {
rel_type: "m.reference",
event_id: "$decl1",
},
},
}),
{} as Room,
undefined,
false,
{} as IRoomTimelineData,
],
}),
options: {
waitForCallPickup: true,
autoLeaveWhenOthersLeft: false,
},
localUser: localRtcMember,
};
const lifecycle = createCallNotificationLifecycle$(props);
expectObservable(lifecycle.callPickupState$).toBe("a 9ms b 29ms e", {
a: "unknown",
b: "ringing",
e: "decline",
});
});
});
it("decline after timeout window ends -> stays timeout", () => {
withTestScheduler(({ scope, hot, behavior, expectObservable }) => {
// Someone joins at 20ms (both LiveKit participant and MatrixRTC member)
const props: CallNotificationLifecycleProps = {
scope,
memberships$: scope.behavior(
behavior("a", {
a: [localRtcMember],
}).pipe(trackEpoch()),
),
sentCallNotification$: hot("10ms a", {
a: mockRingEvent("$decl", 20),
}),
receivedDecline$: hot("40ms d", {
d: [
new MatrixEvent({
type: EventType.RTCDecline,
content: {
"m.relates_to": {
rel_type: "m.reference",
event_id: "$decl",
},
},
}),
{} as Room,
undefined,
false,
{} as IRoomTimelineData,
],
}),
options: {
waitForCallPickup: true,
autoLeaveWhenOthersLeft: false,
},
localUser: localRtcMember,
};
const lifecycle = createCallNotificationLifecycle$(props);
expectObservable(lifecycle.callPickupState$, "50ms !").toBe(
"a 9ms b 19ms e",
{
a: "unknown",
b: "ringing",
e: "timeout",
},
);
});
});
//
function testStaysRinging(
declineEvent: Partial<IEvent>,
expectDecline: boolean,
): void {
withTestScheduler(({ scope, hot, behavior, expectObservable }) => {
// Someone joins at 20ms (both LiveKit participant and MatrixRTC member)
const props: CallNotificationLifecycleProps = {
scope,
memberships$: scope.behavior(
behavior("a", {
a: [localRtcMember],
}).pipe(trackEpoch()),
),
sentCallNotification$: hot("10ms a", {
a: mockRingEvent("$right", 50),
}),
receivedDecline$: hot("20ms d", {
d: [
new MatrixEvent(declineEvent),
{} as Room,
undefined,
false,
{} as IRoomTimelineData,
],
}),
options: {
waitForCallPickup: true,
autoLeaveWhenOthersLeft: false,
},
localUser: localRtcMember,
};
const lifecycle = createCallNotificationLifecycle$(props);
const marbles = expectDecline ? "a 9ms b 9ms d" : "a 9ms b";
expectObservable(lifecycle.callPickupState$, "21ms !").toBe(marbles, {
a: "unknown",
b: "ringing",
d: "decline",
});
});
}
const reference = (refId?: string, sender?: string): Partial<IEvent> => ({
event_id: "$decline",
type: EventType.RTCDecline,
sender: sender ?? "@other:example.org",
content: {
"m.relates_to": {
rel_type: "m.reference",
event_id: refId ?? "$right",
},
},
});
it("decline reference works", () => {
testStaysRinging(reference(), true);
});
it("decline with wrong id is ignored (stays ringing)", () => {
testStaysRinging(reference("$wrong"), false);
});
it("decline with wrong id is ignored (stays ringing)", () => {
testStaysRinging(reference(undefined, local.userId), false);
expectObservable(ringAttempts$).toBe("");
});
});
test("no ring attempt when notification type is not ring", () => {
withTestScheduler(({ scope, expectObservable, hot }) => {
const { ringAttempts$ } = createCallNotificationLifecycle$({
scope,
...defaultProps,
sentCallNotification$: hot("-a", {
a: {
...mockRingEvent("$notif1", 30),
notification_type: "notification",
},
}),
});
expectObservable(ringAttempts$).toBe("");
});
});
test("no ring attempt if lifetime is missing", () => {
withTestScheduler(({ scope, expectObservable, hot }) => {
const { ringAttempts$ } = createCallNotificationLifecycle$({
scope,
...defaultProps,
sentCallNotification$: hot("-a", {
a: mockRingEvent("$notif1", undefined),
}),
});
expectObservable(ringAttempts$).toBe("");
});
});
test("ring attempt times out after nobody joins", () => {
withTestScheduler(({ scope, expectObservable, hot }) => {
const { ringAttempts$ } = createCallNotificationLifecycle$({
scope,
...defaultProps,
// No one ever joins (only local user)
memberships$: constant(new Epoch([])),
sentCallNotification$: hot("-a", {
a: mockRingEvent("$notif1", 30),
}),
});
expectObservable(summarizeRingAttempts$(ringAttempts$)).toBe("-a 29ms A", {
a: { intent: "audio", recipient: alice.userId },
A: { outcome: "timeout" },
});
});
});
test("ring attempt is accepted once recipient joins", () => {
withTestScheduler(({ scope, expectObservable, hot, behavior }) => {
const { ringAttempts$ } = createCallNotificationLifecycle$({
scope,
...defaultProps,
memberships$: scope.behavior(
behavior("a-b", { a: [], b: [aliceRtcMember] }).pipe(trackEpoch()),
),
sentCallNotification$: hot("-a", {
a: mockRingEvent("$notif1", 30),
}),
});
expectObservable(summarizeRingAttempts$(ringAttempts$)).toBe("-aA", {
a: { intent: "audio", recipient: alice.userId },
A: { outcome: "accept" },
});
});
});
test("ring attempt is immediately accepted if recipient is already joined", () => {
withTestScheduler(({ scope, expectObservable, hot }) => {
const { ringAttempts$ } = createCallNotificationLifecycle$({
scope,
...defaultProps,
memberships$: constant(new Epoch([aliceRtcMember])),
sentCallNotification$: hot("-a", {
a: mockRingEvent("$notif1", 30),
}),
});
expectObservable(summarizeRingAttempts$(ringAttempts$)).toBe("-(aA)", {
a: { intent: "audio", recipient: alice.userId },
A: { outcome: "accept" },
});
});
});
test("ring attempt can be declined", () => {
withTestScheduler(({ scope, expectObservable, hot }) => {
const { ringAttempts$ } = createCallNotificationLifecycle$({
scope,
...defaultProps,
sentCallNotification$: hot("-a", {
a: mockRingEvent("$notif1", 30),
}),
receivedDecline$: hot("--d", {
d: [
new MatrixEvent({
type: EventType.RTCDecline,
sender: alice.userId,
content: {
"m.relates_to": {
rel_type: "m.reference",
event_id: "$notif1",
},
},
}),
{} as Room,
undefined,
false,
{} as IRoomTimelineData,
],
}),
});
expectObservable(summarizeRingAttempts$(ringAttempts$)).toBe("-aA", {
a: { intent: "audio", recipient: alice.userId },
A: { outcome: "decline" },
});
});
});
test("ring attempt times out if recipient declines too late", () => {
withTestScheduler(({ scope, expectObservable, hot }) => {
const { ringAttempts$ } = createCallNotificationLifecycle$({
scope,
...defaultProps,
sentCallNotification$: hot("-a", {
a: mockRingEvent("$notif1", 30),
}),
receivedDecline$: hot("100ms d", {
d: [
new MatrixEvent({
type: EventType.RTCDecline,
sender: alice.userId,
content: {
"m.relates_to": {
rel_type: "m.reference",
event_id: "$notif1",
},
},
}),
{} as Room,
undefined,
false,
{} as IRoomTimelineData,
],
}),
});
expectObservable(summarizeRingAttempts$(ringAttempts$)).toBe("-a 29ms A", {
a: { intent: "audio", recipient: alice.userId },
A: { outcome: "timeout" },
});
});
});
test("decline event relating to wrong event is ignored (times out)", () => {
withTestScheduler(({ scope, expectObservable, hot }) => {
const { ringAttempts$ } = createCallNotificationLifecycle$({
scope,
...defaultProps,
sentCallNotification$: hot("-a", {
a: mockRingEvent("$notif1", 30),
}),
receivedDecline$: hot("--d", {
d: [
new MatrixEvent({
type: EventType.RTCDecline,
sender: alice.userId,
content: {
"m.relates_to": {
rel_type: "m.reference",
event_id: "$other", // <---- WRONG
},
},
}),
{} as Room,
undefined,
false,
{} as IRoomTimelineData,
],
}),
});
expectObservable(summarizeRingAttempts$(ringAttempts$)).toBe("-a 29ms A", {
a: { intent: "audio", recipient: alice.userId },
A: { outcome: "timeout" },
});
});
});
test("decline event from wrong sender is ignored (times out)", () => {
withTestScheduler(({ scope, expectObservable, hot }) => {
const { ringAttempts$ } = createCallNotificationLifecycle$({
scope,
...defaultProps,
sentCallNotification$: hot("-a", {
a: mockRingEvent("$notif1", 30),
}),
receivedDecline$: hot("--d", {
d: [
new MatrixEvent({
type: EventType.RTCDecline,
sender: local.userId, // <---- WRONG
content: {
"m.relates_to": {
rel_type: "m.reference",
event_id: "$notif1",
},
},
}),
{} as Room,
undefined,
false,
{} as IRoomTimelineData,
],
}),
});
expectObservable(summarizeRingAttempts$(ringAttempts$)).toBe("-a 29ms A", {
a: { intent: "audio", recipient: alice.userId },
A: { outcome: "timeout" },
});
});
});
src/state/CallViewModel/CallNotificationLifecycle.ts
+99 -109
View File
@@ -10,24 +10,22 @@ import {
type IRTCNotificationContent,
type MatrixRTCSession,
MatrixRTCSessionEvent,
type RTCCallIntent,
} from "matrix-js-sdk/lib/matrixrtc";
import {
combineLatest,
concat,
endWith,
filter,
fromEvent,
ignoreElements,
map,
merge,
NEVER,
type Observable,
of,
pairwise,
startWith,
switchMap,
takeUntil,
timer,
EMPTY,
race,
take,
} from "rxjs";
import {
type EventTimelineSetHandlerMap,
@@ -35,18 +33,26 @@ import {
type Room as MatrixRoom,
RoomEvent,
} from "matrix-js-sdk";
import { logger as rootLogger } from "matrix-js-sdk/lib/logger";
import { type Behavior } from "../Behavior";
import { type Epoch, mapEpoch, type ObservableScope } from "../ObservableScope";
import { type Epoch, type ObservableScope } from "../ObservableScope";
import { type RoomMemberMap } from "./remoteMembers/MatrixMemberMetadata";
export type AutoLeaveReason = "allOthersLeft" | "timeout" | "decline";
export type CallPickupState =
| "unknown"
| "ringing"
| "timeout"
| "decline"
| "success"
| null;
export interface RingAttempt {
intent: RTCCallIntent;
/**
* The user ID of the recipient being rung.
*/
recipient: string;
/**
* The eventual outcome of the ringing attempt. (Emits a single value.)
*/
// TODO: Include a callback for attempting ringing again in case of a timeout
outcome$: Observable<"accept" | "decline" | "timeout">;
}
export type CallNotificationWrapper = {
event_id: string;
@@ -76,6 +82,7 @@ export function createReceivedDecline$(
export interface Props {
scope: ObservableScope;
memberships$: Behavior<Epoch<CallMembership[]>>;
matrixRoomMembers$: Behavior<RoomMemberMap>;
sentCallNotification$: Observable<CallNotificationWrapper | null>;
receivedDecline$: Observable<
Parameters<EventTimelineSetHandlerMap[RoomEvent.Timeline]>
@@ -84,34 +91,86 @@ export interface Props {
localUser: { deviceId: string; userId: string };
}
/**
* @returns two observables:
* `callPickupState$` The current call pickup state of the call.
* - "unknown": The client has not yet sent the notification event. We don't know if it will because it first needs to send its own membership.
* Then we can conclude if we were the first one to join or not.
* - "ringing": The call is ringing on other devices in this room (This client should give audiovisual feedback that this is happening).
* - "timeout": No-one picked up in the defined time this call should be ringing on others devices.
* The call failed. If desired this can be used as a trigger to exit the call.
* - "success": Someone else joined. The call is in a normal state. No audiovisual feedback.
* - null: EC is configured to never show any waiting for answer state.
*
* `autoLeave$` An observable that emits (null) when the call should be automatically left.
* - if options.autoLeaveWhenOthersLeft is set to true it emits when all others left.
* - if options.waitForCallPickup is set to true it emits if noone picked up the ring or if the ring got declined.
* - if options.autoLeaveWhenOthersLeft && options.waitForCallPickup is false it will never emit.
*
*/
export function createCallNotificationLifecycle$({
scope,
memberships$,
matrixRoomMembers$,
sentCallNotification$,
receivedDecline$,
options,
localUser,
}: Props): {
callPickupState$: Behavior<CallPickupState>;
/**
* An observable of attempts to ring the remote participant's devices.
*/
ringAttempts$: Observable<RingAttempt>;
/**
* An observable that emits when the call should be automatically left.
* - if options.autoLeaveWhenOthersLeft is set to true it emits when all others left.
* - if options.waitForCallPickup is set to true it emits if noone picked up the ring or if the ring got declined.
* - if options.autoLeaveWhenOthersLeft && options.waitForCallPickup is false it will never emit.
*/
autoLeave$: Observable<AutoLeaveReason>;
} {
const logger = rootLogger.getChild("[CallNotificationLifecycle]");
let ringAttempts$: Observable<RingAttempt> = NEVER;
if (options.waitForCallPickup)
ringAttempts$ = sentCallNotification$.pipe(
filter(
(
notificationEvent: CallNotificationWrapper | null,
): notificationEvent is CallNotificationWrapper =>
// only care about new events (legacy do not have decline pattern)
notificationEvent?.notification_type === "ring" &&
notificationEvent.lifetime > 0,
),
switchMap((notificationEvent) => {
// We assume that there is only one other user in the room when ringing
// TODO: Respect io.element.functional_members
const recipient = [...matrixRoomMembers$.value.keys()].find(
(userId) => userId !== localUser.userId,
);
if (recipient === undefined) {
logger.warn("No recipient for notification event; not ringing.");
return EMPTY;
}
// Ringing times out after lifetime ms have passed
const timeout$ = timer(notificationEvent.lifetime).pipe(
map(() => "timeout" as const),
);
// Call is accepted when the recipient joins
const accept$ = memberships$.pipe(
filter((ms) => ms.value.some((m) => m.userId === recipient)),
map(() => "accept" as const),
);
// Call is declined when we receive a decline event
const decline$ = receivedDecline$.pipe(
filter(
([event]) =>
event.getRelation()?.rel_type === "m.reference" &&
event.getRelation()?.event_id === notificationEvent.event_id &&
event.getSender() === recipient,
),
map(() => "decline" as const),
);
return of({
intent: notificationEvent["m.call.intent"] ?? "audio",
recipient,
outcome$: race(timeout$, accept$, decline$).pipe(
take(1),
// Make this observable 'hot' to avoid running multiple timers. This
// is not actually a resource leak since there will be at most one
// active ring attempt at any given time.
// eslint-disable-next-line element-call/no-observablescope-leak
scope.share,
),
});
}),
scope.share,
);
const allOthersLeft$ = memberships$.pipe(
pairwise(),
filter(
@@ -122,87 +181,18 @@ export function createCallNotificationLifecycle$({
map(() => {}),
);
/**
* Whether some Matrix user other than ourself is joined to the call.
*/
const someoneElseJoined$ = memberships$.pipe(
mapEpoch((ms) => ms.some((m) => m.userId !== localUser.userId)),
) as Behavior<Epoch<boolean>>;
/**
* The state of the current ringing attempt, if the RTC session is indeed
* ringing the remote participant's devices. Otherwise `null`.
*/
const remoteRingState$: Behavior<"ringing" | "timeout" | "decline" | null> =
scope.behavior(
sentCallNotification$.pipe(
filter(
(notificationEventArgs: CallNotificationWrapper | null) =>
// only care about new events (legacy do not have decline pattern)
notificationEventArgs?.notification_type === "ring",
),
map((e) => e as CallNotificationWrapper),
switchMap((notificationEvent) => {
const lifetimeMs = notificationEvent?.lifetime ?? 0;
return concat(
lifetimeMs === 0
? // If no lifetime, skip the ring state
of(null)
: // Ring until lifetime ms have passed
timer(lifetimeMs).pipe(
ignoreElements(),
startWith("ringing" as const),
),
// The notification lifetime has timed out, meaning ringing has likely
// stopped on all receiving clients.
of("timeout" as const),
// This makes sure we will not drop into the `endWith("decline" as const)` state
NEVER,
).pipe(
takeUntil(
receivedDecline$.pipe(
filter(
([event]) =>
event.getRelation()?.rel_type === "m.reference" &&
event.getRelation()?.event_id ===
notificationEvent.event_id &&
event.getSender() !== localUser.userId &&
callPickupState$.value !== "timeout",
),
),
),
endWith("decline" as const),
);
}),
),
null,
);
const callPickupState$ = scope.behavior(
options.waitForCallPickup === true
? combineLatest(
[someoneElseJoined$, remoteRingState$],
(someoneElseJoined, ring) => {
if (someoneElseJoined.value === true) {
return "success" as const;
}
// Show the ringing state of the most recent ringing attempt.
// as long as we have not yet sent an RTC notification event or noone else joined,
// ring will be null -> callPickupState$ = unknown.
return ring ?? ("unknown" as const);
},
)
: NEVER,
null,
);
const autoLeave$ = merge(
options.autoLeaveWhenOthersLeft === true
? allOthersLeft$.pipe(map(() => "allOthersLeft" as const))
: NEVER,
callPickupState$.pipe(
filter((state) => state === "timeout" || state === "decline"),
ringAttempts$.pipe(
switchMap(({ outcome$ }) =>
outcome$.pipe(
filter((outcome) => outcome === "timeout" || outcome === "decline"),
),
),
),
);
return { autoLeave$, callPickupState$ };
return { ringAttempts$, autoLeave$ };
}
src/state/CallViewModel/CallViewModel.test.ts
+188 -43
View File
@@ -6,7 +6,7 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { test, vi, onTestFinished, it, describe } from "vitest";
import { test, vi, onTestFinished, it, describe, expect } from "vitest";
import {
BehaviorSubject,
combineLatest,
@@ -22,6 +22,7 @@ import { SyncState } from "matrix-js-sdk";
import {
ConnectionState,
type LocalTrackPublication,
type Participant,
type RemoteParticipant,
} from "livekit-client";
import * as ComponentsCore from "@livekit/components-core";
@@ -65,7 +66,7 @@ import {
localParticipant,
withCallViewModel as withCallViewModelInMode,
} from "./CallViewModelTestUtils.ts";
import { MatrixRTCMode } from "../../settings/settings.ts";
import { MatrixRTCMode } from "../../config/ConfigOptions.ts";
import { initializeWidget } from "../../widget.ts";
initializeWidget();
@@ -143,13 +144,13 @@ export interface SpotlightExpandedLayoutSummary {
}
export interface OneOnOneLandscapeLayoutSummary {
type: "one-on-one-landscape";
type: "one-on-one-desktop";
spotlight: string;
pip: string;
}
export interface OneOnOnePortraitLayoutSummary {
type: "one-on-one-portrait";
type: "one-on-one-mobile";
spotlight: string[];
pip?: string;
pipSize: "sm" | "lg";
@@ -204,7 +205,7 @@ function summarizeLayout$(l$: Observable<Layout>): Observable<LayoutSummary> {
pip: pip?.id,
}),
);
case "one-on-one-landscape":
case "one-on-one-desktop":
return combineLatest(
[l.spotlight.media$, l.pip.media$],
(spotlight, pip) => ({
@@ -213,7 +214,7 @@ function summarizeLayout$(l$: Observable<Layout>): Observable<LayoutSummary> {
pip: pip.id,
}),
);
case "one-on-one-portrait":
case "one-on-one-mobile":
return combineLatest(
[
l.spotlight.media$,
@@ -257,11 +258,9 @@ function mockRingEvent(
} as unknown as { event_id: string } & IRTCNotificationContent;
}
describe.each([
[MatrixRTCMode.Legacy],
[MatrixRTCMode.Compatibility],
[MatrixRTCMode.Matrix_2_0],
])("CallViewModel (%s mode)", (mode) => {
const modes = [[MatrixRTCMode.Compatibility], [MatrixRTCMode.Matrix_2_0]];
describe.each(modes)("CallViewModel (%s mode)", (mode) => {
const withCallViewModel = withCallViewModelInMode(mode);
test("participants are retained during a focus switch", () => {
@@ -326,8 +325,8 @@ describe.each([
},
(vm) => {
schedule(modeInputMarbles, {
s: () => vm.setGridMode("spotlight"),
g: () => vm.setGridMode("grid"),
s: () => vm.layoutSwitchVm$.value!.setLayout("spotlight"),
g: () => vm.layoutSwitchVm$.value!.setLayout("grid"),
});
expectObservable(summarizeLayout$(vm.layout$)).toBe(
@@ -436,7 +435,7 @@ describe.each([
expectedLayoutMarbles,
{
a: {
type: "one-on-one-landscape",
type: "one-on-one-desktop",
pip: `${localId}:0`,
spotlight: `${aliceId}:0`,
},
@@ -452,7 +451,7 @@ describe.each([
});
});
test("one-on-one portrait layout shows local tile when video is enabled", () => {
test("one-on-one mobile layout shows local tile when video is enabled", () => {
withTestScheduler(({ behavior, schedule, expectObservable }) => {
// Local participant enables their video, then disables it
const videoInputMarbles = " ny--n";
@@ -479,19 +478,19 @@ describe.each([
expectedLayoutMarbles,
{
a: {
type: "one-on-one-portrait",
type: "one-on-one-mobile",
spotlight: [`${aliceId}:0`],
pip: undefined,
pipSize: "lg",
},
b: {
type: "one-on-one-portrait",
type: "one-on-one-mobile",
spotlight: [`${aliceId}:0`],
pip: `${localId}:0`,
pipSize: "lg",
},
c: {
type: "one-on-one-portrait",
type: "one-on-one-mobile",
spotlight: [`${aliceId}:0`],
pip: `${localId}:0`,
pipSize: "sm",
@@ -503,21 +502,21 @@ describe.each([
});
});
test("one-on-one portrait layout shows name tags in room with 3 members", () => {
withTestScheduler(({ behavior, schedule, expectObservable }) => {
test("one-on-one mobile layout shows name tags in room with 3 members", () => {
withTestScheduler(({ expectObservable }) => {
withCallViewModel(
{
remoteParticipants$: constant([aliceParticipant]),
// Both Alice and Bob are with us in the room
roomMembers: [local, alice, bob],
rtcMembers$: constant([localRtcMember, aliceRtcMember]),
windowSize$: constant({ width: 380, height: 700 }), // Mobile phone in portrait
windowSize$: constant({ width: 380, height: 700 }), // Mobile phone
},
(vm) => {
// Uses one-on-one portrait layout
// Uses one-on-one mobile layout
expectObservable(summarizeLayout$(vm.layout$)).toBe("a", {
a: {
type: "one-on-one-portrait",
type: "one-on-one-mobile",
spotlight: [`${aliceId}:0`],
pip: undefined,
pipSize: "lg",
@@ -531,6 +530,65 @@ describe.each([
});
});
test("landscape mobile layouts show screen shares and group call participants", () => {
withTestScheduler(({ behavior, expectObservable }) => {
// Starts as a one-on-one call, then Alice shares her screen, then Bob
// joins, and finally Alice stops sharing her screen
const participantInputMarbles = " a--b";
const aliceSharingInputMarbles = " ny-n";
// Starts in one-on-one mobile layout, then goes to spotlight layout for
// the screen sharing and group call cases
const expectedLayoutMarbles = " ab-c";
// Whether the layout switch is visible. It should be hidden while in
// one-on-one layout.
const expectedLayoutSwitchMarbles = "ny--";
withCallViewModel(
{
remoteParticipants$: behavior(participantInputMarbles, {
a: [aliceParticipant],
b: [aliceParticipant, bobParticipant],
}),
roomMembers: [local, alice, bob],
rtcMembers$: behavior(participantInputMarbles, {
a: [localRtcMember, aliceRtcMember],
b: [localRtcMember, aliceRtcMember, bobRtcMember],
}),
sharingScreen: new Map([
[aliceParticipant, behavior(aliceSharingInputMarbles, yesNo)],
]),
windowSize$: constant({ width: 700, height: 380 }), // Mobile phone in landscape
},
(vm) => {
expectObservable(summarizeLayout$(vm.layout$)).toBe(
expectedLayoutMarbles,
{
a: {
type: "one-on-one-mobile",
spotlight: [`${aliceId}:0`],
pip: undefined,
pipSize: "sm",
},
b: {
type: "spotlight-landscape",
spotlight: [`${aliceId}:0:screen-share`],
grid: [`${localId}:0`, `${aliceId}:0`],
},
c: {
type: "spotlight-landscape",
spotlight: [`${aliceId}:0`],
grid: [`${localId}:0`, `${bobId}:0`],
},
},
);
expectObservable(
vm.layoutSwitchVm$.pipe(map((vm) => vm !== null)),
).toBe(expectedLayoutSwitchMarbles, yesNo);
},
);
});
});
test("participants stay in the same order unless to appear/disappear", () => {
withTestScheduler(({ behavior, schedule, expectObservable }) => {
const visibilityInputMarbles = "a";
@@ -705,14 +763,14 @@ describe.each([
{
a: {
// This is the expected one-on-one layout for a narrow window
type: "one-on-one-portrait",
type: "one-on-one-mobile",
spotlight: [`${aliceId}:0`],
pip: undefined,
pipSize: "lg",
},
b: {
// In a larger window, expect the normal one-on-one layout
type: "one-on-one-landscape",
// In a larger window, expect the one-on-one desktop layout
type: "one-on-one-desktop",
pip: `${localId}:0`,
spotlight: `${aliceId}:0`,
},
@@ -762,7 +820,9 @@ describe.each([
]),
},
(vm) => {
schedule(modeInputMarbles, { s: () => vm.setGridMode("spotlight") });
schedule(modeInputMarbles, {
s: () => vm.layoutSwitchVm$.value!.setLayout("spotlight"),
});
expectObservable(summarizeLayout$(vm.layout$)).toBe(
expectedLayoutMarbles,
@@ -899,6 +959,41 @@ describe.each([
},
);
// TODO add media to lk mocks
test("onPipMediaOrientationUpdate is called with the spotlight media orientation", () => {
// Set the spy before creating the view model so the initial call is captured
const onPipMediaOrientationUpdate = vi.fn();
window.controls.onPipMediaOrientationUpdate = onPipMediaOrientationUpdate;
onTestFinished(() => {
window.controls.onPipMediaOrientationUpdate = undefined;
});
withTestScheduler(({ behavior }) => {
// Alice starts as a regular participant, then shares her screen, then stops
const aliceSharingInputMarbles = "nyn";
withCallViewModel(
{
remoteParticipants$: constant([aliceParticipant]),
rtcMembers$: constant([localRtcMember, aliceRtcMember]),
sharingScreen: new Map([
[aliceParticipant, behavior(aliceSharingInputMarbles, yesNo)],
]),
},
() => {},
);
});
// Should be called exactly 3 times:
// 1. Initially with "portrait" (Alice is in spotlight as a user, default portrait orientation)
// 2. With "landscape" when Alice starts screen sharing (screen shares always use landscape)
// 3. With "portrait" again when Alice stops screen sharing and returns to user tile
expect(onPipMediaOrientationUpdate).toHaveBeenCalledTimes(3);
expect(onPipMediaOrientationUpdate).toHaveBeenNthCalledWith(1, "portrait");
expect(onPipMediaOrientationUpdate).toHaveBeenNthCalledWith(2, "landscape");
expect(onPipMediaOrientationUpdate).toHaveBeenNthCalledWith(3, "portrait");
});
test("PiP tile in expanded spotlight layout switches speakers without layout shifts", () => {
withTestScheduler(({ behavior, schedule, expectObservable }) => {
// Switch to spotlight immediately
@@ -933,7 +1028,7 @@ describe.each([
},
(vm) => {
schedule(modeInputMarbles, {
s: () => vm.setGridMode("spotlight"),
s: () => vm.layoutSwitchVm$.value!.setLayout("spotlight"),
});
schedule(expandInputMarbles, {
a: () => vm.toggleSpotlightExpanded$.value!(),
@@ -996,10 +1091,14 @@ describe.each([
a: [localRtcMember],
b: [localRtcMember, aliceRtcMember],
}),
videoEnabled: new Map<Participant, Behavior<boolean>>([
[localParticipant, constant(true)],
[aliceParticipant, constant(true)],
]),
},
(vm) => {
schedule(modeInputMarbles, {
s: () => vm.setGridMode("spotlight"),
s: () => vm.layoutSwitchVm$.value!.setLayout("spotlight"),
});
schedule(expandInputMarbles, {
a: () => vm.toggleSpotlightExpanded$.value!(),
@@ -1025,6 +1124,39 @@ describe.each([
});
});
test("expanded spotlight layout hides PiP tile in one-on-one voice call", () => {
withTestScheduler(({ schedule, expectObservable }) => {
withCallViewModel(
{
remoteParticipants$: constant([aliceParticipant]),
roomMembers: [local, alice],
rtcMembers$: constant([localRtcMember, aliceRtcMember]),
videoEnabled: new Map<Participant, Behavior<boolean>>([
[localParticipant, constant(false)],
[aliceParticipant, constant(false)],
]),
},
(vm) => {
schedule("s", {
s: () => vm.layoutSwitchVm$.value!.setLayout("spotlight"),
});
schedule("a", {
a: () => vm.toggleSpotlightExpanded$.value!(),
});
// Layout should show remote tile only
expectObservable(summarizeLayout$(vm.layout$)).toBe("a", {
a: {
type: "spotlight-expanded",
spotlight: [`${aliceId}:0`],
pip: undefined,
},
});
},
);
});
});
test("spotlight remembers whether it's expanded", () => {
withTestScheduler(({ schedule, expectObservable }) => {
// Start in spotlight mode, then switch to grid and back to spotlight a
@@ -1043,8 +1175,8 @@ describe.each([
},
(vm) => {
schedule(modeInputMarbles, {
s: () => vm.setGridMode("spotlight"),
g: () => vm.setGridMode("grid"),
s: () => vm.layoutSwitchVm$.value!.setLayout("spotlight"),
g: () => vm.layoutSwitchVm$.value!.setLayout("grid"),
});
schedule(expandInputMarbles, {
a: () => vm.toggleSpotlightExpanded$.value!(),
@@ -1061,7 +1193,7 @@ describe.each([
b: {
type: "spotlight-expanded",
spotlight: [`${aliceId}:0`],
pip: `${localId}:0`,
pip: undefined,
},
c: {
type: "grid",
@@ -1110,7 +1242,7 @@ describe.each([
]),
},
(vm) => {
vm.setGridMode("grid");
vm.layoutSwitchVm$.value!.setLayout("grid");
expectObservable(summarizeLayout$(vm.layout$)).toBe(
expectedLayoutMarbles,
{
@@ -1120,7 +1252,7 @@ describe.each([
grid: [`${localId}:0`],
},
b: {
type: "one-on-one-landscape",
type: "one-on-one-desktop",
pip: `${localId}:0`,
spotlight: `${aliceId}:0`,
},
@@ -1153,7 +1285,7 @@ describe.each([
}),
},
(vm) => {
vm.setGridMode("grid");
vm.layoutSwitchVm$.value!.setLayout("grid");
expectObservable(summarizeLayout$(vm.layout$)).toBe(
expectedLayoutMarbles,
{
@@ -1163,7 +1295,7 @@ describe.each([
grid: [`${localId}:0`],
},
b: {
type: "one-on-one-landscape",
type: "one-on-one-desktop",
pip: `${localId}:0`,
spotlight: `${aliceId}:0`,
},
@@ -1173,7 +1305,7 @@ describe.each([
grid: [`${localId}:0`, `${aliceId}:0`, `${daveId}:0`],
},
d: {
type: "one-on-one-landscape",
type: "one-on-one-desktop",
pip: `${localId}:0`,
spotlight: `${daveId}:0`,
},
@@ -1385,13 +1517,19 @@ describe.each([
},
});
// Should ring for 30ms and then time out
expectObservable(vm.ringing$).toBe("(ny) 26ms n", yesNo);
expectObservable(vm.ringingVm$).toBe("(ab)", {
a: null,
b: expect.objectContaining({
type: "ringing",
userId: alice.userId,
intent: "audio",
}),
});
// Layout should show placeholder media for the participant we're
// ringing the entire time (even once timed out)
expectObservable(summarizeLayout$(vm.layout$)).toBe("a", {
a: {
type: "one-on-one-landscape",
type: "one-on-one-desktop",
spotlight: `${localId}:0`,
pip: `ringing:${aliceUserId}`,
},
@@ -1425,17 +1563,24 @@ describe.each([
});
// Should ring until Alice joins
expectObservable(vm.ringing$).toBe("(ny) 17ms n", yesNo);
expectObservable(vm.ringingVm$).toBe("(ab) 17ms a", {
a: null,
b: expect.objectContaining({
type: "ringing",
userId: alice.userId,
intent: "audio",
}),
});
// Layout should show placeholder media for the participant we're
// ringing the entire time
expectObservable(summarizeLayout$(vm.layout$)).toBe("a 20ms b", {
a: {
type: "one-on-one-landscape",
type: "one-on-one-desktop",
spotlight: `${localId}:0`,
pip: `ringing:${aliceUserId}`,
},
b: {
type: "one-on-one-landscape",
type: "one-on-one-desktop",
spotlight: `${aliceId}:0`,
pip: `${localId}:0`,
},
src/state/CallViewModel/CallViewModel.ts
+304 -212
View File
@@ -29,7 +29,6 @@ import {
pairwise,
race,
scan,
skipWhile,
startWith,
Subject,
switchAll,
@@ -39,8 +38,9 @@ import {
tap,
throttleTime,
timer,
takeUntil,
} from "rxjs";
import { logger as rootLogger } from "matrix-js-sdk/lib/logger";
import { type Logger, logger as rootLogger } from "matrix-js-sdk/lib/logger";
import {
MembershipManagerEvent,
type LivekitTransportConfig,
@@ -60,17 +60,18 @@ import {
} from "../../utils/observable";
import {
duplicateTiles,
MatrixRTCMode,
playReactionsSound,
showReactions,
} from "../../settings/settings";
import { Config } from "../../config/Config";
import { MatrixRTCMode } from "../../config/ConfigOptions";
import { isFirefox, platform } from "../../Platform";
import { setPipEnabled$ } from "../../controls";
import { TileStore } from "../TileStore";
import { gridLikeLayout } from "../GridLikeLayout";
import { spotlightExpandedLayout } from "../SpotlightExpandedLayout";
import { oneOnOneLandscapeLayout } from "../OneOnOneLandscapeLayout";
import { oneOnOnePortraitLayout } from "../OneOnOnePortraitLayout";
import { oneOnOneDesktopLayout } from "../OneOnOneDesktopLayout";
import { oneOnOneMobileLayout } from "../OneOnOneMobileLayout";
import { pipLayout } from "../PipLayout";
import { type EncryptionSystem } from "../../e2ee/sharedKeyManagement";
import {
@@ -78,7 +79,7 @@ import {
type ReactionInfo,
type ReactionOption,
} from "../../reactions";
import { shallowEquals } from "../../utils/array";
import { shallowEquals as shallowArrayEquals } from "../../utils/array";
import { type MediaDevices } from "../MediaDevices";
import { constant, type Behavior } from "../Behavior";
import { E2eeType } from "../../e2ee/e2eeType";
@@ -88,12 +89,13 @@ import { getUrlParams, HeaderStyle } from "../../UrlParams";
import { type ProcessorState } from "../../livekit/TrackProcessorContext";
import { ElementWidgetActions, widget } from "../../widget";
import {
layoutShallowEquals,
type Alignment,
type GridLayoutMedia,
type Layout,
type LayoutMedia,
type OneOnOneLandscapeLayoutMedia,
type OneOnOnePortraitLayoutMedia,
type OneOnOneDesktopLayoutMedia,
type OneOnOneMobileLayoutMedia,
type SpotlightExpandedLayoutMedia,
type SpotlightLandscapeLayoutMedia,
type SpotlightPortraitLayoutMedia,
@@ -112,6 +114,7 @@ import {
type LocalTransport,
} from "./localMember/LocalTransport.ts";
import {
createKeyRotationSuppressed$,
createMemberships$,
membershipsAndTransports$,
} from "../SessionBehaviors.ts";
@@ -124,10 +127,9 @@ import {
createConnectionManager$,
} from "./remoteMembers/ConnectionManager.ts";
import {
createMatrixLivekitMembers$,
createRemoteMatrixLivekitMembers$,
type LocalMatrixLivekitMember,
type RemoteMatrixLivekitMember,
type MatrixLivekitMember,
} from "./remoteMembers/MatrixLivekitMembers.ts";
import {
type AutoLeaveReason,
@@ -141,7 +143,10 @@ import {
} from "./remoteMembers/MatrixMemberMetadata.ts";
import { Publisher } from "./localMember/Publisher.ts";
import { type Connection } from "./remoteMembers/Connection.ts";
import { createLayoutModeSwitch } from "./LayoutSwitch.ts";
import {
type LayoutSwitchViewModel,
createLayoutSwitchViewModel,
} from "../LayoutSwitchViewModel.ts";
import {
createWrappedUserMedia,
type WrappedUserMediaViewModel,
@@ -158,8 +163,8 @@ import {
createRingingMedia,
type RingingMediaViewModel,
} from "../media/RingingMediaViewModel.ts";
import { type GridTileViewModel } from "../TileViewModel.ts";
const logger = rootLogger.getChild("[CallViewModel]");
//TODO
// Larger rename
// member,membership -> rtcMember
@@ -204,12 +209,11 @@ const smallMobileCallThreshold = 3;
// with the interface
const showFooterMs = 4000;
export type GridMode = "grid" | "spotlight";
export type WindowMode = "normal" | "narrow" | "flat" | "pip";
interface LayoutScanState {
layout: Layout | null;
overflowing: boolean;
tiles: TileStore;
}
@@ -232,9 +236,13 @@ export interface CallViewModel {
// lifecycle
autoLeave$: Observable<AutoLeaveReason>;
/**
* Whether we are ringing a call recipient.
* View model for info relating to ringing, timing out, calling back, etc.
*/
ringing$: Behavior<boolean>;
ringingVm$: Behavior<RingingMediaViewModel | null>;
/**
* Which visual element the ringing status should be shown in.
*/
ringingStatusLocation: "app_bar" | "tile";
/** Observable that emits when the user should leave the call (hangup pressed, widget action, error).
* THIS DOES NOT LEAVE THE CALL YET. The only way to leave the call (send the hangup event) is
* - by ending the scope
@@ -305,11 +313,17 @@ export interface CallViewModel {
participantCount$: Behavior<number>;
/** [lotus] All participants' user media, exposed for the Lotus call-state widget bridge. */
userMedia$: Behavior<WrappedUserMediaViewModel[]>;
/**
* Whether the call has grown large enough that MatrixRTC has stopped rotating the media
* encryption key. While this is true the key in use is still shared with new joiners, but no new
* key is generated when someone joins or leaves.
*/
keyRotationSuppressed$: Behavior<boolean>;
allConnections$: Behavior<ConnectionManagerData>;
/** Participants sorted by livekit room so they can be used in the audio rendering */
livekitRoomItems$: Behavior<LivekitRoomItem[]>;
/** use the layout instead, this is just for the sdk export. */
matrixLivekitMembers$: Behavior<RemoteMatrixLivekitMember[]>;
remoteMatrixLivekitMembers$: Behavior<RemoteMatrixLivekitMember[]>;
localMatrixLivekitMember$: Behavior<LocalMatrixLivekitMember | null>;
/** List of participants raising their hand */
handsRaised$: Behavior<Record<string, RaisedHandInfo>>;
@@ -354,8 +368,7 @@ export interface CallViewModel {
showNameTags$: Behavior<boolean>;
spotlightExpanded$: Behavior<boolean>;
toggleSpotlightExpanded$: Behavior<(() => void) | null>;
gridMode$: Behavior<GridMode>;
setGridMode: (value: GridMode) => void;
layoutSwitchVm$: Behavior<LayoutSwitchViewModel | null>;
// header/footer visibility
showHeader$: Behavior<boolean>;
@@ -365,6 +378,15 @@ export interface CallViewModel {
* and header as overlays.
*/
edgeToEdge$: Behavior<boolean>;
/**
* Whether the call layout is overflowing the interface (causing it to scroll).
*/
overflowing$: Behavior<boolean>;
/**
* Whether modals such as settings and reactions should be accessible at all.
*/
showModals$: Behavior<boolean>;
settingsOpen$: Behavior<boolean>;
setSettingsOpen$: Behavior<(open: boolean) => void>;
@@ -416,6 +438,7 @@ export function createCallViewModel$(
reactionsSubject$: Observable<Record<string, ReactionInfo>>,
trackProcessorState$: Behavior<ProcessorState>,
): CallViewModel {
const logger = rootLogger.getChild("[CallViewModel]");
const client = matrixRoom.client;
const userId = client.getUserId();
const deviceId = client.getDeviceId();
@@ -425,9 +448,17 @@ export function createCallViewModel$(
const livekitKeyProvider = getE2eeKeyProvider(
options.encryptionSystem,
matrixRTCSession,
logger,
);
// matrix_rtc_mode in config.json overrides the user's Developer Settings choice.
// It is validated at config load (src/config/Config.ts) so the cast is safe.
const configMatrixRTCMode = Config.get().matrix_rtc_mode as
| MatrixRTCMode
| undefined;
const matrixRTCMode$ =
options.matrixRTCMode$ ?? constant(MatrixRTCMode.Legacy);
configMatrixRTCMode !== undefined
? constant(configMatrixRTCMode)
: (options.matrixRTCMode$ ?? constant(MatrixRTCMode.Compatibility));
// Each hbar seperates a block of input variables required for the CallViewModel to function.
// The outputs of this block is written under the hbar.
@@ -489,7 +520,6 @@ export function createCallViewModel$(
mode === MatrixRTCMode.Matrix_2_0
? JwtEndpointVersion.Matrix_2_0
: JwtEndpointVersion.Legacy,
useOldestMember: mode === MatrixRTCMode.Legacy,
}),
),
),
@@ -505,9 +535,6 @@ export function createCallViewModel$(
livekitKeyProvider,
getUrlParams().controlledAudioDevices,
options.livekitRoomFactory,
getUrlParams().echoCancellation,
getUrlParams().noiseSuppression,
getUrlParams().autoGainControl,
);
const connectionManager = createConnectionManager$({
@@ -530,13 +557,15 @@ export function createCallViewModel$(
ownMembershipIdentity,
});
const matrixLivekitMembers$: Behavior<Epoch<RemoteMatrixLivekitMember[]>> =
createMatrixLivekitMembers$({
scope: scope,
membershipsWithTransport$:
membershipsAndTransports.membershipsWithTransport$,
connectionManager: connectionManager,
});
const remoteMatrixLivekitMembers$: Behavior<
Epoch<RemoteMatrixLivekitMember[]>
> = createRemoteMatrixLivekitMembers$({
scope: scope,
membershipsWithTransport$:
membershipsAndTransports.membershipsWithTransport$,
connectionManager: connectionManager,
localUser: { userId, deviceId },
});
const connectOptions$ = scope.behavior(
matrixRTCMode$.pipe(
@@ -613,19 +642,12 @@ export function createCallViewModel$(
),
);
// ------------------------------------------------------------------------
// callLifecycle
// TODO if we are in "unknown" state we need a loading rendering (or empty screen)
// Otherwise it looks like we already connected and only than the ringing starts which is weird.
const { callPickupState$, autoLeave$ } = createCallNotificationLifecycle$({
scope: scope,
memberships$: memberships$,
sentCallNotification$: createSentCallNotification$(scope, matrixRTCSession),
receivedDecline$: createReceivedDecline$(matrixRoom),
options: options,
localUser: { userId: userId, deviceId: deviceId },
});
const matrixLivekitMembers$ = scope.behavior(
combineLatest(
[localMatrixLivekitMember$, remoteMatrixLivekitMembers$],
(local, remote) => [...(local === null ? [] : [local]), ...remote.value],
),
);
// ------------------------------------------------------------------------
// matrixMemberMetadataStore
@@ -637,11 +659,26 @@ export function createCallViewModel$(
matrixRoomMembers$,
);
// ------------------------------------------------------------------------
// callLifecycle
// TODO if we are in "unknown" state we need a loading rendering (or empty screen)
// Otherwise it looks like we already connected and only than the ringing starts which is weird.
const { ringAttempts$, autoLeave$ } = createCallNotificationLifecycle$({
scope,
memberships$,
matrixRoomMembers$,
sentCallNotification$: createSentCallNotification$(scope, matrixRTCSession),
receivedDecline$: createReceivedDecline$(matrixRoom),
options,
localUser: { userId, deviceId },
});
const allConnections$ = scope.behavior(
connectionManager.connectionManagerData$.pipe(map((d) => d.value)),
);
const livekitRoomItems$ = scope.behavior(
matrixLivekitMembers$.pipe(
remoteMatrixLivekitMembers$.pipe(
switchMap((members) => {
const a$ = combineLatest(
members.value.map((member) =>
@@ -707,43 +744,20 @@ export function createCallViewModel$(
* List of user media (camera feeds) that we want tiles for.
*/
const userMedia$ = scope.behavior<WrappedUserMediaViewModel[]>(
combineLatest([
localMatrixLivekitMember$,
matrixLivekitMembers$,
duplicateTiles.value$,
]).pipe(
combineLatest([matrixLivekitMembers$, duplicateTiles.value$]).pipe(
// Generate a collection of user media from the list of expected (whether
// present or missing) LiveKit participants.
generateItems(
"CallViewModel userMedia$",
function* ([
localMatrixLivekitMember,
matrixLivekitMembers,
duplicateTiles,
]) {
const computeMediaId = (m: MatrixLivekitMember): string =>
`${m.userId}:${m.membership$.value.deviceId}`;
const localUserMediaId = localMatrixLivekitMember
? computeMediaId(localMatrixLivekitMember)
: undefined;
const localAsArray = localMatrixLivekitMember
? [localMatrixLivekitMember]
: [];
const remoteWithoutLocal = matrixLivekitMembers.value.filter(
(m) => computeMediaId(m) !== localUserMediaId,
);
const allMatrixLivekitMembers = [
...localAsArray,
...remoteWithoutLocal,
];
for (const matrixLivekitMember of allMatrixLivekitMembers) {
const { userId, participant, connection$, membership$ } =
matrixLivekitMember;
const rtcId = membership$.value.rtcBackendIdentity; // rtcBackendIdentity
const mediaId = computeMediaId(matrixLivekitMember);
function* ([members, duplicateTiles]) {
for (const {
userId,
participant,
connection$,
membership$,
} of members) {
const rtcId = membership$.value.rtcBackendIdentity;
const mediaId = `${userId}:${membership$.value.deviceId}`;
for (let dup = 0; dup < 1 + duplicateTiles; dup++) {
yield {
keys: [dup, mediaId, userId, participant, connection$, rtcId],
@@ -769,11 +783,13 @@ export function createCallViewModel$(
pretendToBeDisconnected$: localMembership.reconnecting$,
displayName$: scope.behavior(
matrixMemberMetadataStore
.createDisplayNameBehavior$(userId)
.createDisplayNameBehavior$(scope, userId)
.pipe(map((name) => name ?? userId)),
),
mxcAvatarUrl$:
matrixMemberMetadataStore.createAvatarUrlBehavior$(userId),
mxcAvatarUrl$: matrixMemberMetadataStore.createAvatarUrlBehavior$(
scope,
userId,
),
handRaised$: scope.behavior(
handsRaised$.pipe(map((v) => v[mediaId]?.time ?? null)),
),
@@ -785,49 +801,42 @@ export function createCallViewModel$(
),
);
const ringingMedia$ = scope.behavior<RingingMediaViewModel[]>(
combineLatest([userMedia$, matrixRoomMembers$, callPickupState$]).pipe(
generateItems(
"CallViewModel ringingMedia$",
function* ([userMedia, roomMembers, callPickupState]) {
if (
callPickupState === "ringing" ||
callPickupState === "timeout" ||
callPickupState === "decline"
) {
// TODO: Respect io.element.functional_members
for (const member of roomMembers.values()) {
if (!userMedia.some((vm) => vm.userId === member.userId))
yield {
keys: [member.userId],
data: callPickupState,
};
}
}
},
(scope, pickupState$, userId) =>
createRingingMedia({
id: `ringing:${userId}`,
userId,
displayName$: scope.behavior(
matrixRoomMembers$.pipe(
map((members) => members.get(userId)?.rawDisplayName || userId),
),
),
mxcAvatarUrl$:
matrixMemberMetadataStore.createAvatarUrlBehavior$(userId),
pickupState$,
muteStates,
}),
const ringingMedia$ = scope.behavior<RingingMediaViewModel | null>(
ringAttempts$.pipe(
switchMap(({ intent, recipient, outcome$ }) =>
outcome$.pipe(
startWith("ringing" as const),
generateItems(
"CallViewModel ringingMedia$",
function* (pickupState) {
if (pickupState !== "accept")
yield { keys: [intent, recipient], data: pickupState };
},
(scope, pickupState$, intent, userId) =>
createRingingMedia({
id: `ringing:${userId}`,
userId,
displayName$: scope.behavior(
matrixRoomMembers$.pipe(
map(
(members) =>
members.get(userId)?.rawDisplayName || userId,
),
),
),
mxcAvatarUrl$:
matrixMemberMetadataStore.createAvatarUrlBehavior$(
scope,
userId,
),
pickupState$,
intent,
}),
),
map(([media]) => media ?? null),
),
),
distinctUntilChanged(shallowEquals),
tap((ringingMedia) => {
if (ringingMedia.length > 1)
// Warn that UI may do something unexpected in this case
logger.warn(
`Ringing more than one participant is not supported (ringing ${ringingMedia.map((vm) => vm.userId).join(", ")})`,
);
}),
startWith(null),
),
);
@@ -866,14 +875,15 @@ export function createCallViewModel$(
* multiple devices.
*/
const participantCount$ = scope.behavior(
matrixLivekitMembers$.pipe(map((ms) => ms.value.length)),
matrixLivekitMembers$.pipe(map((ms) => ms.length)),
);
const leaveSoundEffect$ = combineLatest([callPickupState$, userMedia$]).pipe(
// Until the call is successful, do not play a leave sound.
// If callPickupState$ is null, then we always play the sound as it will not conflict with a decline sound.
skipWhile(([c]) => c !== null && c !== "success"),
map(([, userMedia]) => userMedia),
const keyRotationSuppressed$ = createKeyRotationSuppressed$(
scope,
matrixRTCSession,
);
const leaveSoundEffect$ = userMedia$.pipe(
pairwise(),
filter(
([prev, current]) =>
@@ -882,6 +892,9 @@ export function createCallViewModel$(
),
map(() => {}),
throttleTime(THROTTLE_SOUND_EFFECT_MS),
// Avoid doubling up on any auto-leave sounds (e.g. the decline sound),
// which are handled elsewhere
takeUntil(autoLeave$),
);
const userHangup$ = new Subject<void>();
@@ -960,13 +973,13 @@ export function createCallViewModel$(
bins.sort(([, bin1], [, bin2]) => bin1 - bin2).map(([m]) => m),
);
}),
distinctUntilChanged(shallowEquals),
distinctUntilChanged(shallowArrayEquals),
),
);
/**
* Local user media suitable for displaying in a PiP (undefined if not found
* or if user prefers to not see themselves).
* Local user media suitable for displaying in a PiP (undefined if not found,
* video is muted, or if user prefers to not see themselves).
*/
const localUserMediaForPip$ = scope.behavior<
LocalUserMediaViewModel | undefined
@@ -978,8 +991,10 @@ export function createCallViewModel$(
m.type === "user" && m.local,
);
if (!localUserMedia) return of(undefined);
return localUserMedia.alwaysShow$.pipe(
map((alwaysShow) => (alwaysShow ? localUserMedia : undefined)),
return combineLatest(
[localUserMedia.videoEnabled$, localUserMedia.alwaysShow$],
(videoEnabled, alwaysShow) =>
videoEnabled && alwaysShow ? localUserMedia : undefined,
);
}),
),
@@ -991,8 +1006,8 @@ export function createCallViewModel$(
}>(
ringingMedia$.pipe(
switchMap((ringingMedia) => {
if (ringingMedia.length > 0)
return of({ spotlight: ringingMedia, pip$: localUserMediaForPip$ });
if (ringingMedia !== null)
return of({ spotlight: [ringingMedia], pip$: localUserMediaForPip$ });
// [lotus #4] Route the screenshare/spotlight computation through the
// manual-spotlight wrapper. This is byte-for-byte upstream behaviour
@@ -1013,7 +1028,7 @@ export function createCallViewModel$(
const spotlight$ = scope.behavior<MediaViewModel[]>(
spotlightAndPip$.pipe(
map(({ spotlight }) => spotlight),
distinctUntilChanged<MediaViewModel[]>(shallowEquals),
distinctUntilChanged<MediaViewModel[]>(shallowArrayEquals),
),
);
@@ -1067,7 +1082,7 @@ export function createCallViewModel$(
spotlightExpandedToggle$,
);
const { setGridMode, gridMode$ } = createLayoutModeSwitch(
const layoutSwitchVm = createLayoutSwitchViewModel(
scope,
windowMode$,
hasRemoteScreenShares$,
@@ -1119,60 +1134,60 @@ export function createCallViewModel$(
),
);
const oneOnOneLayoutMedia$: Observable<{
const oneOnOneLayoutMedia$: Behavior<{
local: LocalUserMediaViewModel;
remote: UserMediaViewModel | RingingMediaViewModel;
} | null> = combineLatest([userMedia$, screenShares$]).pipe(
switchMap(([userMedia, screenShares]) => {
// One-on-one layout only supports 2 user media, no screen shares
if (userMedia.length <= 2 && screenShares.length === 0) {
const local = userMedia.find(
(vm): vm is WrappedUserMediaViewModel & LocalUserMediaViewModel =>
vm.type === "user" && vm.local,
);
if (local !== undefined) {
const remote = userMedia.find(
(vm): vm is WrappedUserMediaViewModel & RemoteUserMediaViewModel =>
vm.type === "user" && !vm.local,
} | null> = scope.behavior(
combineLatest([userMedia$, screenShares$]).pipe(
switchMap(([userMedia, screenShares]) => {
// One-on-one layout only supports 2 user media, no screen shares
if (userMedia.length <= 2 && screenShares.length === 0) {
const local = userMedia.find(
(vm): vm is WrappedUserMediaViewModel & LocalUserMediaViewModel =>
vm.type === "user" && vm.local,
);
if (remote !== undefined) return of({ local, remote });
// If there's no other user media in the call (could still happen in
// this branch due to the duplicate tiles option), we could possibly
// show ringing media instead
if (userMedia.length === 1)
return ringingMedia$.pipe(
map((ringingMedia) => {
return ringingMedia.length === 1
? {
local,
remote: ringingMedia[0],
}
: null;
}),
if (local !== undefined) {
const remote = userMedia.find(
(
vm,
): vm is WrappedUserMediaViewModel & RemoteUserMediaViewModel =>
vm.type === "user" && !vm.local,
);
}
}
return of(null);
}),
if (remote !== undefined) return of({ local, remote });
// If there's no other user media in the call (could still happen in
// this branch due to the duplicate tiles option), we could possibly
// show ringing media instead
if (userMedia.length === 1)
return ringingMedia$.pipe(
map(
(ringingMedia) =>
ringingMedia && { local, remote: ringingMedia },
),
);
}
}
return of(null);
}),
),
);
const oneOnOneLandscapeLayoutMedia$: Observable<OneOnOneLandscapeLayoutMedia | null> =
const oneOnOneDesktopLayoutMedia$: Observable<OneOnOneDesktopLayoutMedia | null> =
oneOnOneLayoutMedia$.pipe(
map((media) => {
if (media === null) return null;
return media.remote.type === "ringing"
? {
type: "one-on-one-landscape" as const,
type: "one-on-one-desktop" as const,
edgeToEdge: false,
spotlight: media.local,
pip: media.remote,
}
: {
type: "one-on-one-landscape" as const,
type: "one-on-one-desktop" as const,
edgeToEdge: false,
spotlight: media.remote,
pip: media.local,
@@ -1180,13 +1195,13 @@ export function createCallViewModel$(
}),
);
const oneOnOnePortraitLayoutMedia$: Observable<OneOnOnePortraitLayoutMedia | null> =
const oneOnOneMobileLayoutMedia$: Observable<OneOnOneMobileLayoutMedia | null> =
oneOnOneLayoutMedia$.pipe(
switchMap((media) => {
if (media === null) return of(null);
return media.local.videoEnabled$.pipe(
map((videoEnabled) => ({
type: "one-on-one-portrait" as const,
type: "one-on-one-mobile" as const,
edgeToEdge: true as const,
spotlight: media.remote,
pip: videoEnabled ? media.local : undefined,
@@ -1203,6 +1218,34 @@ export function createCallViewModel$(
})),
);
spotlight$
.pipe(
switchMap((media) => {
let layout;
const pipMedia = media[0];
if (pipMedia === undefined) return of(undefined);
switch (pipMedia.type) {
case "user":
layout = pipMedia.videoOrientation$;
break;
case "ringing":
layout = of("landscape" as const);
break;
case "screen share":
layout = of("landscape" as const);
break;
}
return layout;
}),
distinctUntilChanged(),
scope.bind(),
)
.subscribe((orientation) => {
if (orientation === undefined) return;
logger.info("controls api pip orientation updated:", orientation);
window.controls.onPipMediaOrientationUpdate?.(orientation);
});
/**
* The media to be used to produce a layout.
*/
@@ -1211,11 +1254,11 @@ export function createCallViewModel$(
switchMap((windowMode) => {
switch (windowMode) {
case "normal":
return gridMode$.pipe(
switchMap((gridMode) => {
switch (gridMode) {
return layoutSwitchVm.layout$.pipe(
switchMap((layout) => {
switch (layout) {
case "grid":
return oneOnOneLandscapeLayoutMedia$.pipe(
return oneOnOneDesktopLayoutMedia$.pipe(
switchMap((oneOnOne) =>
oneOnOne === null ? gridLayoutMedia$ : of(oneOnOne),
),
@@ -1232,7 +1275,7 @@ export function createCallViewModel$(
}),
);
case "narrow":
return oneOnOnePortraitLayoutMedia$.pipe(
return oneOnOneMobileLayoutMedia$.pipe(
switchMap((oneOnOne) =>
oneOnOne === null
? combineLatest([grid$, spotlight$], (grid, spotlight) =>
@@ -1245,17 +1288,23 @@ export function createCallViewModel$(
),
);
case "flat":
return gridMode$.pipe(
switchMap((gridMode) => {
switch (gridMode) {
case "grid":
// Yes, grid mode actually gets you a "spotlight" layout in
// this window mode.
return spotlightLandscapeLayoutMedia$(true);
case "spotlight":
return spotlightExpandedLayoutMedia$(true);
}
}),
return oneOnOneMobileLayoutMedia$.pipe(
switchMap((oneOnOne) =>
oneOnOne === null
? layoutSwitchVm.layout$.pipe(
switchMap((layout) => {
switch (layout) {
case "grid":
// Yes, grid mode actually gets you a "spotlight" layout in
// this window mode.
return spotlightLandscapeLayoutMedia$(true);
case "spotlight":
return spotlightExpandedLayoutMedia$(true);
}
}),
)
: of(oneOnOne),
),
);
case "pip":
return pipLayoutMedia$;
@@ -1283,8 +1332,8 @@ export function createCallViewModel$(
// indicators. And in one-on-one layout there's no question as to who is
// speaking.
case "spotlight-expanded":
case "one-on-one-landscape":
case "one-on-one-portrait":
case "one-on-one-desktop":
case "one-on-one-mobile":
return false;
default:
return true;
@@ -1296,7 +1345,7 @@ export function createCallViewModel$(
const showNameTags$ = scope.behavior<boolean>(
layoutMedia$.pipe(
switchMap((l) =>
l.type === "pip" || l.type === "one-on-one-portrait"
l.type === "pip" || l.type === "one-on-one-mobile"
? matrixRoomMembers$.pipe(
map(
(members) =>
@@ -1340,6 +1389,22 @@ export function createCallViewModel$(
layoutMedia$.pipe(map(({ edgeToEdge }) => edgeToEdge)),
);
// Only show the layout switch in cases where it has an effect on the layout
const showLayoutSwitch$ = windowMode$.pipe(
switchMap((windowMode) => {
switch (windowMode) {
case "normal":
return of(true);
case "flat":
return oneOnOneLayoutMedia$.pipe(
map((oneOnOne) => oneOnOne === null),
);
default:
return of(false);
}
}),
);
const screenTap$ = new Subject<void>();
const controlsTap$ = new Subject<void>();
const screenHover$ = new Subject<void>();
@@ -1412,6 +1477,11 @@ export function createCallViewModel$(
map((naturallyShowFooter) => naturallyShowFooter && showFooterUrlParams),
),
);
const showModals$ = scope.behavior(
windowMode$.pipe(map((mode) => mode !== "pip")),
);
const settingsOpen$ = new BehaviorSubject(false);
const setSettingsOpen$ = constant((open: boolean) => {
settingsOpen$.next(open);
@@ -1421,7 +1491,7 @@ export function createCallViewModel$(
windowMode$.pipe(
switchMap((mode) => {
// In small windows the header would be too obstructive
if (mode === "pip" || mode === "flat") return of(false);
if (mode === "pip") return of(false);
// In edge-to-edge layouts, couple the visibility of the header
// to that of the footer
return edgeToEdge$.pipe(
@@ -1461,7 +1531,7 @@ export function createCallViewModel$(
// There is a cyclical dependency here: the layout algorithms want to know
// which tiles are on screen, but to know which tiles are on screen we have to
// first render a layout. To deal with this we assume initially that no tiles
// first render a layout. To deal with this we assume initially that all tiles
// are visible, and loop the data back into the layouts with a Subject.
const visibleTiles$ = new Subject<number>();
const setVisibleTiles = (value: number): void => visibleTiles$.next(value);
@@ -1469,7 +1539,7 @@ export function createCallViewModel$(
const layoutInternals$ = scope.behavior<LayoutScanState & { layout: Layout }>(
combineLatest([
layoutMedia$,
visibleTiles$.pipe(startWith(0), distinctUntilChanged()),
visibleTiles$.pipe(startWith(Infinity), distinctUntilChanged()),
]).pipe(
scan<
[LayoutMedia, number],
@@ -1479,6 +1549,8 @@ export function createCallViewModel$(
({ tiles: prevTiles }, [media, visibleTiles]) => {
let layout: Layout;
let newTiles: TileStore;
let pip: GridTileViewModel | undefined;
let overflowing = false;
switch (media.type) {
case "grid":
case "spotlight-landscape":
@@ -1490,6 +1562,7 @@ export function createCallViewModel$(
setVisibleTiles,
prevTiles,
);
overflowing = newTiles.gridTiles.length > visibleTiles;
break;
case "spotlight-expanded":
[layout, newTiles] = spotlightExpandedLayout(
@@ -1498,29 +1571,35 @@ export function createCallViewModel$(
prevTiles,
);
break;
case "one-on-one-landscape":
[layout, newTiles] = oneOnOneLandscapeLayout(
case "one-on-one-desktop":
[layout, newTiles] = oneOnOneDesktopLayout(
media,
landscapePipAlignment$,
prevTiles,
);
pip = layout.pip;
break;
case "one-on-one-portrait":
[layout, newTiles] = oneOnOnePortraitLayout(
case "one-on-one-mobile":
[layout, newTiles] = oneOnOneMobileLayout(
media,
portraitPipSize$,
portraitPipAlignment$,
prevTiles,
);
pip = layout.pip;
break;
case "pip":
[layout, newTiles] = pipLayout(media, prevTiles);
break;
}
return { layout, tiles: newTiles };
for (const tile of newTiles.gridTiles) {
tile.setShowOutline(tile === pip);
}
return { layout, overflowing, tiles: newTiles };
},
{ layout: null, tiles: TileStore.empty() },
{ layout: null, overflowing: false, tiles: TileStore.empty() },
),
),
);
@@ -1529,7 +1608,15 @@ export function createCallViewModel$(
* The layout of tiles in the call interface.
*/
const layout$ = scope.behavior<Layout>(
layoutInternals$.pipe(map(({ layout }) => layout)),
layoutInternals$.pipe(
map(({ layout }) => layout),
// Drop redundant layout updates before they would hit React.
distinctUntilChanged<Layout>(layoutShallowEquals),
),
);
const overflowing$ = scope.behavior<boolean>(
layoutInternals$.pipe(map(({ overflowing }) => overflowing)),
);
/**
@@ -1701,9 +1788,9 @@ export function createCallViewModel$(
return {
autoLeave$: autoLeave$,
ringing$: scope.behavior(
callPickupState$.pipe(map((state) => state === "ringing")),
),
ringingVm$: ringingMedia$,
ringingStatusLocation:
urlParams.header === HeaderStyle.AppBar ? "app_bar" : "tile",
leave$: leave$,
hangup: (): void => userHangup$.next(),
join: localMembership.requestJoinAndPublish,
@@ -1736,6 +1823,7 @@ export function createCallViewModel$(
allConnections$,
participantCount$: participantCount$,
userMedia$,
keyRotationSuppressed$: keyRotationSuppressed$,
handsRaised$: handsRaised$,
reactions$: reactions$,
joinSoundEffect$: joinSoundEffect$,
@@ -1747,12 +1835,13 @@ export function createCallViewModel$(
spotlightExpanded$: spotlightExpanded$,
toggleSpotlightExpanded$: toggleSpotlightExpanded$,
gridMode$: gridMode$,
setGridMode: setGridMode,
layoutSwitchVm$: scope.behavior(
showLayoutSwitch$.pipe(map((show) => (show ? layoutSwitchVm : null))),
),
layout$: layout$,
localMatrixLivekitMember$,
matrixLivekitMembers$: scope.behavior(
matrixLivekitMembers$.pipe(
remoteMatrixLivekitMembers$: scope.behavior(
remoteMatrixLivekitMembers$.pipe(
map((members) => members.value),
tap((v) => {
const listForLogs = v
@@ -1773,9 +1862,11 @@ export function createCallViewModel$(
showNameTags$,
showHeader$: showHeader$,
showFooter$: showFooter$,
showModals$,
settingsOpen$: settingsOpen$,
setSettingsOpen$: setSettingsOpen$,
edgeToEdge$,
overflowing$,
earpieceMode$: earpieceMode$,
audioOutputSwitcher$: audioOutputSwitcher$,
reconnecting$: localMembership.reconnecting$,
@@ -1787,6 +1878,7 @@ export function createCallViewModel$(
function getE2eeKeyProvider(
e2eeSystem: EncryptionSystem,
rtcSession: MatrixRTCSession,
logger: Logger,
): BaseKeyProvider | undefined {
if (e2eeSystem.kind === E2eeType.NONE) return undefined;
src/state/CallViewModel/CallViewModelTestUtils.ts
+1 -4
View File
@@ -56,7 +56,7 @@ import {
import { type Behavior, constant } from "../Behavior";
import { type ProcessorState } from "../../livekit/TrackProcessorContext";
import { type MediaDevices } from "../MediaDevices";
import { type MatrixRTCMode } from "../../settings/settings";
import { type MatrixRTCMode } from "../../config/ConfigOptions";
mockConfig({
livekit: { livekit_service_url: "http://my-default-service-url.com" },
@@ -138,9 +138,6 @@ export function withCallViewModel(mode: MatrixRTCMode) {
public getSyncState(): SyncState {
return syncState;
}
public getAccessToken(): string | null {
return "a-token";
}
})() as Partial<MatrixClient> as MatrixClient,
getMembers: () => roomMembers,
getMembersWithMembership: () => roomMembers,
src/state/CallViewModel/LayoutSwitch.ts
-93
View File
@@ -1,93 +0,0 @@
/*
Copyright 2025 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import {
combineLatest,
map,
Subject,
startWith,
skipWhile,
switchMap,
} from "rxjs";
import { type GridMode, type WindowMode } from "./CallViewModel.ts";
import { constant, type Behavior } from "../Behavior.ts";
import { type ObservableScope } from "../ObservableScope.ts";
/**
* Creates a layout mode switch that allows switching between grid and spotlight modes.
* The actual layout mode might switch automatically to spotlight if there is a
* remote screen share active or if the window mode is flat.
*
* @param scope - The observable scope to manage subscriptions.
* @param windowMode$ - The current window mode.
* @param hasRemoteScreenShares$ - A behavior indicating if there are remote screen shares active.
*/
export function createLayoutModeSwitch(
scope: ObservableScope,
windowMode$: Behavior<WindowMode>,
hasRemoteScreenShares$: Behavior<boolean>,
): {
gridMode$: Behavior<GridMode>;
setGridMode: (value: GridMode) => void;
} {
const userSelection$ = new Subject<GridMode>();
// Callback to set the grid mode desired by the user.
// Notice that this is only a preference, the actual grid mode can be overridden
// if there is a remote screen share active.
const setGridMode = (value: GridMode): void => userSelection$.next(value);
/**
* The natural grid mode - the mode that the grid would prefer to be in,
* not accounting for the user's manual selections.
*/
const naturalGridMode$ = scope.behavior<GridMode>(
combineLatest(
[hasRemoteScreenShares$, windowMode$],
(hasRemoteScreenShares, windowMode) =>
// When there are screen shares or the window is flat (as with a phone
// in landscape orientation), spotlight is a better experience.
// We want screen shares to be big and readable, and we want flipping
// your phone into landscape to be a quick way of maximising the
// spotlight tile.
hasRemoteScreenShares || windowMode === "flat" ? "spotlight" : "grid",
),
);
/**
* The layout mode of the media tile grid.
*/
const gridMode$ = scope.behavior<GridMode>(
// Whenever the user makes a selection, we enter a new mode of behavior:
userSelection$.pipe(
map((selection) => {
if (selection === "grid")
// The user has selected grid mode. Start by respecting their choice,
// but then follow the natural mode again as soon as it matches.
return naturalGridMode$.pipe(
skipWhile((naturalMode) => naturalMode !== selection),
startWith(selection),
);
// The user has selected spotlight mode. If this matches the natural
// mode, then follow the natural mode going forward.
return selection === naturalGridMode$.value
? naturalGridMode$
: constant(selection);
}),
// Initially the mode of behavior is to just follow the natural grid mode.
startWith(naturalGridMode$),
// Switch between each mode of behavior.
switchMap((mode$) => mode$),
),
);
return {
gridMode$,
setGridMode,
};
}
src/state/CallViewModel/localMember/HomeserverConnected.ts
+1 -5
View File
@@ -31,11 +31,6 @@ import { type ObservableScope } from "../../ObservableScope";
import { type Behavior } from "../../Behavior";
import { type NodeStyleEventEmitter } from "../../../utils/test";
/**
* Logger instance (scoped child) for homeserver connection updates.
*/
const logger = rootLogger.getChild("[HomeserverConnected]");
export type HomeserverDisconnectReason = "sync" | "membership" | "probablyLeft";
export interface HomeserverConnected {
@@ -70,6 +65,7 @@ export function createHomeserverConnected$(
Pick<MatrixRTCSession, "membershipStatus" | "probablyLeft">,
gracePeriodMs?: number,
): HomeserverConnected {
const logger = rootLogger.getChild("[HomeserverConnected]");
// Get grace period from parameter or config (default 10000ms)
const graceMs = gracePeriodMs ?? Config.get().sync_disconnect_grace_period_ms;
src/state/CallViewModel/localMember/LocalMember.test.ts
+28 -61
View File
@@ -20,13 +20,12 @@ import {
afterAll,
beforeEach,
} from "vitest";
import { AutoDiscovery } from "matrix-js-sdk/lib/autodiscovery";
import { BehaviorSubject, map, of } from "rxjs";
import { logger } from "matrix-js-sdk/lib/logger";
import { type LocalParticipant, type LocalTrack } from "livekit-client";
import { PosthogAnalytics } from "../../../analytics/PosthogAnalytics";
import { MatrixRTCMode } from "../../../settings/settings";
import { MatrixRTCMode } from "../../../config/ConfigOptions";
import { type HomeserverDisconnectReason } from "./HomeserverConnected";
import {
flushPromises,
@@ -60,7 +59,7 @@ import {
initializeWidget();
const MATRIX_RTC_MODE = MatrixRTCMode.Legacy;
const MATRIX_RTC_MODE = MatrixRTCMode.Compatibility;
const getUrlParams = vi.hoisted(() => vi.fn(() => ({})));
vi.mock("../../../UrlParams", () => ({ getUrlParams }));
vi.mock("@livekit/components-core", () => ({
@@ -72,40 +71,10 @@ vi.mock("@livekit/components-core", () => ({
describe("LocalMembership", () => {
describe("enterRTCSession", () => {
it("It joins the correct Session", () => {
const focusFromOlderMembership = {
type: "livekit",
livekit_service_url: "http://my-oldest-member-service-url.com",
livekit_alias: "my-oldest-member-service-alias",
};
const focusConfigFromWellKnown = {
type: "livekit",
livekit_service_url: "http://my-well-known-service-url.com",
};
const focusConfigFromWellKnown2 = {
type: "livekit",
livekit_service_url: "http://my-well-known-service-url2.com",
};
const clientWellKnown = {
"org.matrix.msc4143.rtc_foci": [
focusConfigFromWellKnown,
focusConfigFromWellKnown2,
],
};
mockConfig({
livekit: { livekit_service_url: "http://my-default-service-url.com" },
});
vi.spyOn(AutoDiscovery, "getRawClientConfig").mockImplementation(
async (domain) => {
if (domain === "example.org") {
return Promise.resolve(clientWellKnown);
}
return Promise.resolve({});
},
);
const mockedSession = vi.mocked({
room: {
roomId: "roomId",
@@ -120,10 +89,6 @@ describe("LocalMembership", () => {
},
},
memberships: [],
getFocusInUse: vi.fn().mockReturnValue(focusFromOlderMembership),
getOldestMembership: vi.fn().mockReturnValue({
getPreferredFoci: vi.fn().mockReturnValue([focusFromOlderMembership]),
}),
joinRTCSession: vi.fn(),
}) as unknown as MatrixRTCSession;
@@ -132,7 +97,7 @@ describe("LocalMembership", () => {
ownMemberMock,
{
livekit_alias: "roomId",
livekit_service_url: "http://my-well-known-service-url.com",
livekit_service_url: "http://my-livekit-service-url.com",
type: "livekit",
},
{
@@ -147,30 +112,24 @@ describe("LocalMembership", () => {
memberId: "@alice:example.org:DEVICE",
userId: "@alice:example.org",
},
[
{
livekit_alias: "roomId",
livekit_service_url: "http://my-well-known-service-url.com",
type: "livekit",
},
],
undefined,
expect.objectContaining({
manageMediaKeys: true,
useLegacyMemberEvents: false,
}),
[],
{
livekit_alias: "roomId",
livekit_service_url: "http://my-livekit-service-url.com",
type: "livekit",
},
expect.objectContaining({ manageMediaKeys: true }),
);
});
it("It should not fail with configuration error if homeserver config has livekit url but not fallback", () => {
mockConfig({});
vi.spyOn(AutoDiscovery, "getRawClientConfig").mockResolvedValue({
"org.matrix.msc4143.rtc_foci": [
{
type: "livekit",
livekit_service_url: "http://my-well-known-service-url.com",
},
],
it("passes keyRotationParticipantLimit from config to joinRTCSession", () => {
mockConfig({
livekit: { livekit_service_url: "http://my-default-service-url.com" },
matrix_rtc_session: {
delayed_leave_event_delay_ms: 0,
network_error_retry_ms: 0,
key_rotation_participant_limit: 50,
},
});
const mockedSession = vi.mocked({
@@ -187,7 +146,6 @@ describe("LocalMembership", () => {
},
},
memberships: [],
getFocusInUse: vi.fn(),
joinRTCSession: vi.fn(),
}) as unknown as MatrixRTCSession;
@@ -196,7 +154,7 @@ describe("LocalMembership", () => {
ownMemberMock,
{
livekit_alias: "roomId",
livekit_service_url: "http://my-well-known-service-url.com",
livekit_service_url: "http://my-livekit-service-url.com",
type: "livekit",
},
{
@@ -204,6 +162,15 @@ describe("LocalMembership", () => {
matrixRTCMode: MATRIX_RTC_MODE,
},
);
expect(mockedSession.joinRTCSession).toHaveBeenLastCalledWith(
expect.any(Object),
[],
expect.any(Object),
expect.objectContaining({
keyRotationParticipantLimit: 50,
}),
);
});
});
src/state/CallViewModel/localMember/LocalMember.ts
+56 -11
View File
@@ -10,6 +10,7 @@ import {
ParticipantEvent,
type LocalParticipant,
type ScreenShareCaptureOptions,
type TrackPublishOptions,
RoomEvent,
MediaDeviceFailure,
} from "livekit-client";
@@ -53,7 +54,15 @@ import {
import { ElementWidgetActions, widget } from "../../../widget.ts";
import { getUrlParams } from "../../../UrlParams.ts";
import { PosthogAnalytics } from "../../../analytics/PosthogAnalytics.ts";
import { MatrixRTCMode } from "../../../settings/settings.ts";
import {
advancedScreenShare,
screenShareResolution,
screenShareFramerate,
screenShareBitrate,
screenShareCodec,
parseResolution,
} from "../../../settings/settings.ts";
import { MatrixRTCMode } from "../../../config/ConfigOptions.ts";
import { Config } from "../../../config/Config.ts";
import {
ConnectionState,
@@ -108,8 +117,6 @@ export type LocalMemberState =
};
/*
* - get well known
* - get oldest membership
* - get transport to use
* - get openId + jwt token
* - wait for createTrack() call
@@ -170,7 +177,7 @@ export const createLocalMembership$ = ({
logger: parentLogger,
muteStates,
matrixRTCSession,
roomId: roomId,
roomId,
}: Props): {
/**
* This request to start audio and video tracks.
@@ -719,6 +726,43 @@ export const createLocalMembership$ = ({
surfaceSwitching: "include",
systemAudio: "include",
};
let publishOptions: TrackPublishOptions | undefined;
if (advancedScreenShare.getValue()) {
// User has advanced screen share settings enabled
const { width, height } = parseResolution(
screenShareResolution.getValue(),
);
const fps = screenShareFramerate.getValue();
const bps = screenShareBitrate.getValue();
const codec = screenShareCodec.getValue();
screenshareSettings.resolution = {
width,
height,
frameRate: fps,
};
publishOptions = {
screenShareEncoding: {
maxBitrate: bps,
maxFramerate: fps,
},
videoCodec: codec,
};
} else {
// Fall back to config.json settings if available
const screenConf = Config.get().media_quality?.screen_share;
if (screenConf?.max_resolution) {
screenshareSettings.resolution = {
width: Math.round((screenConf.max_resolution * 16) / 9),
height: screenConf.max_resolution,
frameRate: screenConf.max_framerate ?? 30,
};
}
}
const targetScreenshareState = !sharingScreen$.value;
logger.info(
`toggleScreenSharing called. Switching ${
@@ -734,7 +778,11 @@ export const createLocalMembership$ = ({
// is still initializing or publishing tracks, because there's no
// technical reason to disallow this. LiveKit will publish if it can.
participant$.value
?.setScreenShareEnabled(targetScreenshareState, screenshareSettings)
?.setScreenShareEnabled(
targetScreenshareState,
screenshareSettings,
publishOptions,
)
.catch(logger.error);
};
}
@@ -802,9 +850,7 @@ export function enterRTCSession(
// have started tracking by the time calls start getting created.
// groupCallOTelMembership?.onJoinCall();
const { features, matrix_rtc_session: matrixRtcSessionConfig } = Config.get();
const useDeviceSessionMemberEvents =
features?.feature_use_device_session_member_events;
const { matrix_rtc_session: matrixRtcSessionConfig } = Config.get();
const { sendNotificationType: notificationType, callIntent } = getUrlParams();
const multiSFU =
matrixRTCMode === MatrixRTCMode.Compatibility ||
@@ -846,9 +892,6 @@ export function enterRTCSession(
notificationType,
callIntent,
manageMediaKeys: encryptMedia,
...(useDeviceSessionMemberEvents !== undefined && {
useLegacyMemberEvents: !useDeviceSessionMemberEvents,
}),
delayedLeaveEventRestartMs:
matrixRtcSessionConfig?.delayed_leave_event_restart_ms,
delayedLeaveEventDelayMs:
@@ -859,6 +902,8 @@ export function enterRTCSession(
makeKeyDelay: matrixRtcSessionConfig?.wait_for_key_rotation_ms,
membershipEventExpiryMs:
matrixRtcSessionConfig?.membership_event_expiry_ms,
keyRotationParticipantLimit:
matrixRtcSessionConfig?.key_rotation_participant_limit,
unstableSendStickyEvents: matrixRTCMode === MatrixRTCMode.Matrix_2_0,
maximumNetworkErrorRetryCount: maximumNetworkErrorRetryCount,
},
src/state/CallViewModel/localMember/LocalTransport.test.ts
+15 -232
View File
@@ -13,7 +13,6 @@ import {
it,
type MockedObject,
vi,
type MockInstance,
} from "vitest";
import {
type CallMembership,
@@ -26,7 +25,6 @@ import {
mockConfig,
flushPromises,
ownMemberMock,
mockRtcMembership,
testScope,
} from "../../../utils/test";
import {
@@ -35,7 +33,7 @@ import {
type LocalTransportWithSFUConfig,
} from "./LocalTransport";
import { constant } from "../../Behavior";
import { Epoch, ObservableScope, trackEpoch } from "../../ObservableScope";
import { Epoch, ObservableScope } from "../../ObservableScope";
import {
MatrixRTCTransportMissingError,
FailToGetOpenIdToken,
@@ -58,13 +56,11 @@ describe("LocalTransport", () => {
const { advertised$, active$ } = createLocalTransport$({
scope: testScope(),
roomId: "!room:example.org",
useOldestMember: false,
memberships$: constant(new Epoch<CallMembership[]>([])),
client: {
// eslint-disable-next-line @typescript-eslint/naming-convention
_unstable_getRTCTransports: async () => Promise.resolve([]),
getAccessToken: vi.fn().mockReturnValue("access_token"),
getDomain: () => "",
getDomain: () => "example.org",
baseUrl: "example.org",
// These won't be called in this error path but satisfy the type
getOpenIdToken: vi.fn(),
@@ -77,9 +73,11 @@ describe("LocalTransport", () => {
await flushPromises();
expect(() => advertised$.value).toThrow(
new MatrixRTCTransportMissingError(""),
new MatrixRTCTransportMissingError("example.org"),
);
expect(() => active$.value).toThrow(
new MatrixRTCTransportMissingError("example.org"),
);
expect(() => active$.value).toThrow(new MatrixRTCTransportMissingError(""));
});
it("throws FailToGetOpenIdToken when OpenID fetch fails", async () => {
@@ -100,13 +98,10 @@ describe("LocalTransport", () => {
const { advertised$, active$ } = createLocalTransport$({
scope,
roomId: "!example_room_id",
useOldestMember: false,
memberships$: constant(new Epoch<CallMembership[]>([])),
client: {
baseUrl: "https://lk.example.org",
// Use empty domain to skip .well-known and use config directly
getDomain: () => "",
getAccessToken: vi.fn().mockReturnValue("access_token"),
baseUrl: "https://example.org",
getDomain: () => "example.org",
// eslint-disable-next-line @typescript-eslint/naming-convention
_unstable_getRTCTransports: async () => Promise.resolve([]),
getOpenIdToken: vi.fn(),
@@ -145,16 +140,14 @@ describe("LocalTransport", () => {
const { advertised$, active$ } = createLocalTransport$({
scope: testScope(),
roomId: "!room:example.org",
useOldestMember: false,
memberships$: constant(new Epoch<CallMembership[]>([])),
client: {
// eslint-disable-next-line @typescript-eslint/naming-convention
_unstable_getRTCTransports: async () => Promise.resolve([]),
getDomain: () => "",
getDomain: () => "example.org",
getOpenIdToken: vi.fn(),
getDeviceId: vi.fn(),
baseUrl: "https://lk.example.org",
getAccessToken: vi.fn().mockReturnValue("access_token"),
baseUrl: "https://example.org",
},
ownMembershipIdentity: ownMemberMock,
forceJwtEndpoint: JwtEndpointVersion.Legacy,
@@ -187,129 +180,6 @@ describe("LocalTransport", () => {
});
});
describe("oldest member mode", () => {
const aliceTransport: LivekitTransportConfig = {
type: "livekit",
livekit_service_url: "https://alice.example.org",
};
const bobTransport: LivekitTransportConfig = {
type: "livekit",
livekit_service_url: "https://bob.example.org",
};
const aliceMembership = mockRtcMembership("@alice:example.org", "AAA", {
fociPreferred: [aliceTransport],
});
const bobMembership = mockRtcMembership("@bob:example.org", "BBB", {
fociPreferred: [bobTransport],
});
let openIdSpy: MockInstance<(typeof openIDSFU)["getSFUConfigWithOpenID"]>;
beforeEach(() => {
openIdSpy = vi
.spyOn(openIDSFU, "getSFUConfigWithOpenID")
.mockResolvedValue(openIdResponse);
});
it("updates active transport when oldest member changes", async () => {
// Initially, Alice is the only member
const memberships$ = new BehaviorSubject([aliceMembership]);
const scope = testScope();
const { advertised$, active$ } = createLocalTransport$({
scope,
roomId: "!example_room_id",
useOldestMember: true,
memberships$: scope.behavior(memberships$.pipe(trackEpoch())),
client: {
getDomain: () => "",
// eslint-disable-next-line @typescript-eslint/naming-convention
_unstable_getRTCTransports: async () => Promise.resolve([]),
getAccessToken: vi.fn().mockReturnValue("access_token"),
getOpenIdToken: vi.fn(),
getDeviceId: vi.fn(),
baseUrl: "https://lk.example.org",
},
ownMembershipIdentity: ownMemberMock,
forceJwtEndpoint: JwtEndpointVersion.Legacy,
delayId$: constant("delay_id_mock"),
});
expect(active$.value).toBe(null);
await flushPromises();
// SFU config should've been fetched
expect(openIdSpy).toHaveBeenCalled();
// Alice's transport should be active and advertised
expect(active$.value?.transport).toStrictEqual(aliceTransport);
expect(advertised$.value).toStrictEqual(aliceTransport);
// Now Bob joins the call, but Alice is still the oldest member
openIdSpy.mockClear();
memberships$.next([aliceMembership, bobMembership]);
await flushPromises();
// No new SFU config should've been fetched
expect(openIdSpy).not.toHaveBeenCalled();
// Alice's transport should still be active and advertised
expect(active$.value?.transport).toStrictEqual(aliceTransport);
expect(advertised$.value).toStrictEqual(aliceTransport);
// Now Bob takes Alice's place as the oldest member
openIdSpy.mockClear();
memberships$.next([bobMembership, aliceMembership]);
// Active transport should reset to null until we have Bob's SFU config
expect(active$.value).toStrictEqual(null);
await flushPromises();
// Bob's SFU config should've been fetched
expect(openIdSpy).toHaveBeenCalled();
// Bob's transport should be active, but Alice's should remain advertised
// (since we don't want the change in oldest member to cause a wave of new
// state events)
expect(active$.value?.transport).toStrictEqual(bobTransport);
expect(advertised$.value).toStrictEqual(aliceTransport);
});
it("advertises preferred transport when no other member exists", async () => {
// Initially, there are no members
const memberships$ = new BehaviorSubject<CallMembership[]>([]);
const scope = testScope();
const { advertised$, active$ } = createLocalTransport$({
scope,
roomId: "!example_room_id",
useOldestMember: true,
memberships$: scope.behavior(memberships$.pipe(trackEpoch())),
client: {
getDomain: () => "",
// eslint-disable-next-line @typescript-eslint/naming-convention
_unstable_getRTCTransports: async () =>
Promise.resolve([aliceTransport]),
getAccessToken: vi.fn().mockReturnValue("access_token"),
getOpenIdToken: vi.fn(),
getDeviceId: vi.fn(),
baseUrl: "https://lk.example.org",
},
ownMembershipIdentity: ownMemberMock,
forceJwtEndpoint: JwtEndpointVersion.Legacy,
delayId$: constant("delay_id_mock"),
});
expect(active$.value).toBe(null);
await flushPromises();
// Our own preferred transport should be advertised
expect(advertised$.value).toStrictEqual(aliceTransport);
// No transport should be active however (there is still no oldest member)
expect(active$.value).toBe(null);
// Now Bob joins the call and becomes the oldest member
memberships$.next([bobMembership]);
await flushPromises();
// We should still advertise our own preferred transport (to avoid
// unnecessary state changes)
expect(advertised$.value).toStrictEqual(aliceTransport);
// Bob's transport should become active
expect(active$.value?.transport).toBe(bobTransport);
});
});
type LocalTransportProps = Parameters<typeof createLocalTransport$>[0];
describe("transport configuration mechanisms", () => {
@@ -324,16 +194,14 @@ describe("LocalTransport", () => {
ownMembershipIdentity: ownMemberMock,
scope: testScope(),
roomId: "!example_room_id",
useOldestMember: false,
forceJwtEndpoint: JwtEndpointVersion.Legacy,
delayId$: constant(null),
memberships$: constant(new Epoch<CallMembership[]>([])),
client: {
baseUrl: "https://example.org",
getDomain: vi.fn().mockReturnValue(""),
getDomain: vi.fn().mockReturnValue("example.org"),
// eslint-disable-next-line @typescript-eslint/naming-convention
_unstable_getRTCTransports: vi.fn().mockResolvedValue([]),
getAccessToken: vi.fn().mockReturnValue("access_token"),
getOpenIdToken: vi.fn(),
getDeviceId: vi.fn(),
},
@@ -421,42 +289,6 @@ describe("LocalTransport", () => {
});
});
it("Should not call _unstable_getRTCTransports in widget mode but use well-known", async () => {
mockConfig({
livekit: { livekit_service_url: "https://do-not-use.lk.example.org" },
});
localTransportOpts.client.getDomain.mockReturnValue("example.org");
fetchMock.getOnce("https://example.org/.well-known/matrix/client", {
"org.matrix.msc4143.rtc_foci": [
{
type: "livekit",
livekit_service_url: "https://use-me.jwt.call.example.org",
},
],
});
localTransportOpts.client.getAccessToken.mockReturnValue(null);
const { advertised$, active$ } =
createLocalTransport$(localTransportOpts);
openIdResolver.resolve?.(openIdResponse);
expect(advertised$.value).toBe(null);
expect(active$.value).toBe(null);
await flushPromises();
expect(
localTransportOpts.client._unstable_getRTCTransports,
).not.toHaveBeenCalled();
const expectedTransport = {
type: "livekit",
livekit_service_url: "https://use-me.jwt.call.example.org",
};
expect(advertised$.value).toStrictEqual(expectedTransport);
});
it("fails fast if the openID request fails for backend config", async () => {
localTransportOpts.client._unstable_getRTCTransports.mockResolvedValue([
{ type: "livekit", livekit_service_url: "https://lk.example.org" },
@@ -469,66 +301,19 @@ describe("LocalTransport", () => {
).rejects.toThrow(expect.any(FailToGetOpenIdToken));
});
it("supports getting transport via well-known", async () => {
localTransportOpts.client.getDomain.mockReturnValue("example.org");
fetchMock.getOnce("https://example.org/.well-known/matrix/client", {
"org.matrix.msc4143.rtc_foci": [
{ type: "livekit", livekit_service_url: "https://lk.example.org" },
],
});
const { advertised$, active$ } =
createLocalTransport$(localTransportOpts);
openIdResolver.resolve?.(openIdResponse);
expect(advertised$.value).toBe(null);
expect(active$.value).toBe(null);
await flushPromises();
const expectedTransport = {
livekit_service_url: "https://lk.example.org",
type: "livekit",
};
expect(advertised$.value).toStrictEqual(expectedTransport);
expect(active$.value).toStrictEqual({
transport: expectedTransport,
sfuConfig: {
jwt: "e30=.eyJzdWIiOiJAbWU6ZXhhbXBsZS5vcmc6QUJDREVGIiwidmlkZW8iOnsicm9vbSI6IiFleGFtcGxlX3Jvb21faWQifX0=.e30=",
livekitAlias: "Akph4alDMhen",
livekitIdentity: "@lk_user:ABCDEF",
url: "https://lk.example.org",
},
});
expect(fetchMock.done()).toEqual(true);
});
it("fails fast if the openId request fails for the well-known config", async () => {
localTransportOpts.client.getDomain.mockReturnValue("example.org");
fetchMock.getOnce("https://example.org/.well-known/matrix/client", {
"org.matrix.msc4143.rtc_foci": [
{ type: "livekit", livekit_service_url: "https://lk.example.org" },
],
});
openIdResolver.reject(
new FailToGetOpenIdToken(new Error("Test driven error")),
);
await expect(async () =>
lastValueFrom(createLocalTransport$(localTransportOpts).active$),
).rejects.toThrow(expect.any(FailToGetOpenIdToken));
});
it("throws if no options are available", async () => {
const { advertised$, active$ } = createLocalTransport$({
scope: testScope(),
ownMembershipIdentity: ownMemberMock,
roomId: "!example_room_id",
useOldestMember: false,
forceJwtEndpoint: JwtEndpointVersion.Legacy,
delayId$: constant(null),
memberships$: constant(new Epoch<CallMembership[]>([])),
client: {
getDomain: () => "",
getDomain: () => "example.org",
baseUrl: "https://example.org",
// eslint-disable-next-line @typescript-eslint/naming-convention
_unstable_getRTCTransports: async () => Promise.resolve([]),
getAccessToken: vi.fn().mockReturnValue("access_token"),
// These won't be called in this error path but satisfy the type
getOpenIdToken: vi.fn(),
getDeviceId: vi.fn(),
@@ -537,10 +322,10 @@ describe("LocalTransport", () => {
await flushPromises();
expect(() => advertised$.value).toThrow(
new MatrixRTCTransportMissingError(""),
new MatrixRTCTransportMissingError("example.org"),
);
expect(() => active$.value).toThrow(
new MatrixRTCTransportMissingError(""),
new MatrixRTCTransportMissingError("example.org"),
);
});
});
@@ -560,16 +345,14 @@ describe("LocalTransport", () => {
ownMembershipIdentity: ownMemberMock,
roomId: "!example_room_id",
// We want multi-sdu
useOldestMember: false,
forceJwtEndpoint: JwtEndpointVersion.Legacy,
delayId$: delayId$,
memberships$: constant(new Epoch<CallMembership[]>([])),
client: {
getDomain: () => "",
getDomain: () => "example.org",
baseUrl: "https://example.org",
// eslint-disable-next-line @typescript-eslint/naming-convention
_unstable_getRTCTransports: async () => Promise.resolve([]),
getAccessToken: vi.fn().mockReturnValue("access_token"),
// These won't be called in this error path but satisfy the type
getOpenIdToken: vi.fn(),
getDeviceId: vi.fn(),
src/state/CallViewModel/localMember/LocalTransport.ts
+7 -152
View File
@@ -7,26 +7,18 @@ Please see LICENSE in the repository root for full details.
import {
type CallMembership,
isLivekitTransportConfig,
type LivekitTransportConfig,
} from "matrix-js-sdk/lib/matrixrtc";
import { type MatrixClient } from "matrix-js-sdk";
import {
catchError,
combineLatest,
distinctUntilChanged,
first,
from,
map,
merge,
type Observable,
of,
startWith,
switchMap,
tap,
} from "rxjs";
import { logger as rootLogger } from "matrix-js-sdk/lib/logger";
import { AutoDiscovery } from "matrix-js-sdk/lib/autodiscovery";
import { logger as rootLogger, type Logger } from "matrix-js-sdk/lib/logger";
import { type CallMembershipIdentityParts } from "matrix-js-sdk/lib/matrixrtc/EncryptionManager";
import { type Behavior } from "../../Behavior.ts";
@@ -46,12 +38,9 @@ import { areLivekitTransportsEqual } from "../remoteMembers/MatrixLivekitMembers
import { customLivekitUrl } from "../../../settings/settings.ts";
import { RtcTransportAutoDiscovery } from "./RtcTransportAutoDiscovery.ts";
const logger = rootLogger.getChild("[LocalTransport]");
/*
* It figures out “which LiveKit focus URL/alias the local user should use,”
* optionally aligning with the oldest member, and ensures the SFU path is primed
* before advertising that choice.
* and ensures the SFU path is primed before advertising that choice.
*/
interface Props {
scope: ObservableScope;
@@ -59,12 +48,11 @@ interface Props {
memberships$: Behavior<Epoch<CallMembership[]>>;
client: Pick<
MatrixClient,
"getDomain" | "baseUrl" | "_unstable_getRTCTransports" | "getAccessToken"
"getDomain" | "baseUrl" | "_unstable_getRTCTransports"
> &
OpenIDClientParts;
// Used by the jwt service to create the livekit room and compute the livekit alias.
roomId: string;
useOldestMember: boolean;
forceJwtEndpoint: JwtEndpointVersion;
delayId$: Behavior<string | null>;
}
@@ -122,8 +110,6 @@ export interface LocalTransport {
/**
* Connects to the JWT service and determines the transports that the local member should use.
*
* @prop useOldestMember Whether to use the same transport as the oldest member.
* This will only update once the first oldest member appears. Will not recompute if the oldest member leaves.
* @prop useOldJwtEndpoint Whether to set forceOldJwtEndpoint on the returned transport and to use the old JWT endpoint.
* This is used when the connection manager needs to know if it has to use the legacy endpoint which implies a string concatenated rtcBackendIdentity.
* (which is expected for non sticky event based rtc member events)
@@ -136,18 +122,14 @@ export const createLocalTransport$ = ({
ownMembershipIdentity,
client,
roomId,
useOldestMember,
forceJwtEndpoint,
delayId$,
}: Props): LocalTransport => {
// The LiveKit transport in use by the oldest RTC membership. `null` when the
// oldest member has no such transport.
const oldestMemberTransport$ = observerOldestMembership$(scope, memberships$);
const logger = rootLogger.getChild("[LocalTransport]");
const transportDiscovery = new RtcTransportAutoDiscovery({
client: client,
resolvedConfig: Config.get(),
wellKnownFetcher: AutoDiscovery.getRawClientConfig.bind(AutoDiscovery),
logger: logger,
});
@@ -190,6 +172,7 @@ export const createLocalTransport$ = ({
roomId,
client,
delayId ?? undefined,
logger,
);
} catch (e) {
logger.error(
@@ -201,18 +184,6 @@ export const createLocalTransport$ = ({
}),
);
if (useOldestMember) {
return observeLocalTransportForOldestMembership(
scope,
oldestMemberTransport$,
preferredTransport$,
client,
ownMembershipIdentity,
roomId,
);
}
// --- Multi-SFU mode ---
// Always publish on and advertise the preferred transport.
return {
advertised$: scope.behavior(
@@ -240,46 +211,6 @@ export const createLocalTransport$ = ({
};
};
/**
* Observes the oldest member in the room and returns the transport that it uses if it is a livekit transport.
* @param scope - The observable scope.
* @param memberships$ - The observable of the call's memberships.'
*/
function observerOldestMembership$(
scope: ObservableScope,
memberships$: Behavior<Epoch<CallMembership[]>>,
): Behavior<LivekitTransportConfig | null> {
return scope.behavior<LivekitTransportConfig | null>(
memberships$.pipe(
map((memberships) => {
const oldestMember = memberships.value[0];
if (oldestMember === undefined) {
logger.info("Oldest member: not found");
return null;
}
const transport = oldestMember.getTransport(oldestMember);
if (transport === undefined) {
logger.warn(
`Oldest member: ${oldestMember.userId}|${oldestMember.deviceId}|${oldestMember.memberId} has no transport`,
);
return null;
}
if (!isLivekitTransportConfig(transport)) {
logger.warn(
`Oldest member: ${oldestMember.userId}|${oldestMember.deviceId}|${oldestMember.memberId} has invalid transport`,
);
return null;
}
logger.info(
"Oldest member: ${oldestMember.userId}|${oldestMember.deviceId}|${oldestMember.memberId} has valid transport",
);
return transport;
}),
distinctUntilChanged(areLivekitTransportsEqual),
),
);
}
/**
* Utility to ensure the user can authenticate with the SFU.
* We will call `getSFUConfigWithOpenID` once per transport here as it's our
@@ -303,10 +234,11 @@ async function doOpenIdAndJWTFromUrl(
roomId: string,
client: Pick<
MatrixClient,
"getDomain" | "baseUrl" | "_unstable_getRTCTransports" | "getAccessToken"
"getDomain" | "baseUrl" | "_unstable_getRTCTransports"
> &
OpenIDClientParts,
delayId?: string,
logger?: Logger,
): Promise<LocalTransportWithSFUConfig> {
const sfuConfig = await getSFUConfigWithOpenID(
client,
@@ -326,83 +258,6 @@ async function doOpenIdAndJWTFromUrl(
};
}
function observeLocalTransportForOldestMembership(
scope: ObservableScope,
oldestMemberTransport$: Behavior<LivekitTransportConfig | null>,
preferredTransport$: Observable<LocalTransportWithSFUConfig>,
client: Pick<
MatrixClient,
"getDomain" | "baseUrl" | "_unstable_getRTCTransports" | "getAccessToken"
> &
OpenIDClientParts,
ownMembershipIdentity: CallMembershipIdentityParts,
roomId: string,
): LocalTransport {
// Ensure we can authenticate with the SFU.
const authenticatedOldestMemberTransport$ = oldestMemberTransport$.pipe(
switchMap((transport) => {
// Oldest member not available -we are first- (or invalid SFU config).
if (transport === null) return of(null);
// Whenever there is transport change we want to revert
// to no transport while we do the authentication.
// So do a from(promise) here to be able to startWith(null)
return from(
doOpenIdAndJWTFromUrl(
transport,
JwtEndpointVersion.Legacy,
ownMembershipIdentity,
roomId,
client,
undefined,
),
).pipe(
catchError((e: unknown) => {
logger.error(
`Failed to authenticate to transport ${transport.livekit_service_url}`,
e,
);
throw mapAuthErrorToUserFriendlyError(e);
}),
startWith(null),
);
}),
);
// --- Oldest member mode ---
return {
// Never update the transport that we advertise in our membership. Just
// take the first valid oldest member or preferred transport that we learn
// about, and stick with that. This avoids unnecessary SFU hops and room
// state changes.
advertised$: scope.behavior(
merge(
authenticatedOldestMemberTransport$.pipe(
map((t) => t?.transport ?? null),
),
preferredTransport$.pipe(map((t) => t.transport)),
).pipe(
first((t) => t !== null),
tap((t) =>
logger.info(`Advertise transport: ${t.livekit_service_url}`),
),
),
null,
),
// Publish on the transport used by the oldest member.
active$: scope.behavior(
authenticatedOldestMemberTransport$.pipe(
tap((t) =>
logger.info(
`Publish on transport: ${t?.transport.livekit_service_url}`,
),
),
),
null,
),
};
}
function mapAuthErrorToUserFriendlyError(e: unknown): Error {
if (
e instanceof FailToGetOpenIdToken ||
src/state/CallViewModel/localMember/RtcTransportAutoDiscovery.test.ts
+16 -96
View File
@@ -13,7 +13,7 @@ import {
type MockedObject,
vi,
} from "vitest";
import { type IClientWellKnown, MatrixError } from "matrix-js-sdk";
import { MatrixError } from "matrix-js-sdk";
import { logger as rootLogger } from "matrix-js-sdk/lib/logger";
import {
type LivekitTransportConfig,
@@ -33,9 +33,9 @@ const backendTransport: LivekitTransportConfig = {
livekit_service_url: "https://backend.example.org",
};
const wellKnownTransport: LivekitTransportConfig = {
const configTransport: LivekitTransportConfig = {
type: "livekit",
livekit_service_url: "https://well-known.example.org",
livekit_service_url: "https://config.example.org",
};
function makeClient(): MockedObject<DiscoveryClient> {
@@ -43,7 +43,6 @@ function makeClient(): MockedObject<DiscoveryClient> {
getDomain: vi.fn().mockReturnValue("example.org"),
baseUrl: "https://matrix.example.org",
_unstable_getRTCTransports: vi.fn().mockResolvedValue([]),
getAccessToken: vi.fn().mockReturnValue("access_token"),
getOpenIdToken: vi.fn(),
getDeviceId: vi.fn(),
} as unknown as MockedObject<DiscoveryClient>;
@@ -59,12 +58,6 @@ function makeResolvedConfig(livekitServiceUrl?: string): ResolvedConfigOptions {
} as ResolvedConfigOptions;
}
function makeWellKnown(rtcFoci?: Transport[]): IClientWellKnown {
return {
"org.matrix.msc4143.rtc_foci": rtcFoci,
} as unknown as IClientWellKnown;
}
describe("RtcTransportAutoDiscovery", () => {
beforeEach(() => {
vi.clearAllMocks();
@@ -75,33 +68,27 @@ describe("RtcTransportAutoDiscovery", () => {
{ transports: [{ type: "not_livekit" }, backendTransport] },
];
it.each(VALID_TEST_CASES)(
"prefers backend transport over well-known and app config $transports",
"prefers backend transport other app config $transports",
async ({ transports }) => {
// it("prefers backend transport over well-known and app config", async () => {
const client = makeClient();
client._unstable_getRTCTransports.mockResolvedValue(transports);
const wellKnownFetcher = vi
.fn<(domain: string) => Promise<IClientWellKnown>>()
.mockResolvedValue(makeWellKnown([wellKnownTransport]));
const discovery = new RtcTransportAutoDiscovery({
client,
resolvedConfig: makeResolvedConfig("https://config.example.org"),
wellKnownFetcher,
resolvedConfig: makeResolvedConfig(configTransport.livekit_service_url),
logger: rootLogger,
});
await expect(
discovery.discoverPreferredTransport(),
).resolves.toStrictEqual(backendTransport);
const discoveredTransport = await discovery.discoverPreferredTransport();
expect(discoveredTransport).toStrictEqual(backendTransport);
expect(discoveredTransport).not.toStrictEqual(configTransport);
expect(client._unstable_getRTCTransports).toHaveBeenCalledTimes(1);
expect(wellKnownFetcher).not.toHaveBeenCalled();
},
);
it("Retries limit_exceeded backend transport over well-known", async () => {
it("Retries limit_exceeded backend transport", async () => {
const client = makeClient();
client._unstable_getRTCTransports
.mockRejectedValueOnce(
@@ -116,14 +103,9 @@ describe("RtcTransportAutoDiscovery", () => {
)
.mockResolvedValue([backendTransport]);
const wellKnownFetcher = vi
.fn<(domain: string) => Promise<IClientWellKnown>>()
.mockResolvedValue(makeWellKnown([wellKnownTransport]));
const discovery = new RtcTransportAutoDiscovery({
client,
resolvedConfig: makeResolvedConfig("https://config.example.org"),
wellKnownFetcher,
logger: rootLogger,
});
@@ -132,7 +114,6 @@ describe("RtcTransportAutoDiscovery", () => {
);
expect(client._unstable_getRTCTransports).toHaveBeenCalledTimes(2);
expect(wellKnownFetcher).not.toHaveBeenCalled();
});
const INVALID_TEST_CASES: Array<{ transports: Transport[] }> = [
@@ -140,91 +121,30 @@ describe("RtcTransportAutoDiscovery", () => {
{ transports: [{ type: "not_livekit" }] },
];
it.each(INVALID_TEST_CASES)(
"falls back to well-known when backend has no (valid) livekit transports $transports",
"falls back to config when backend has no (valid) livekit transports $transports",
async ({ transports }) => {
const client = makeClient();
client._unstable_getRTCTransports.mockResolvedValue(transports);
const wellKnownFetcher = vi
.fn<(domain: string) => Promise<IClientWellKnown>>()
.mockResolvedValue(makeWellKnown([wellKnownTransport]));
const discovery = new RtcTransportAutoDiscovery({
client,
resolvedConfig: makeResolvedConfig("https://config.example.org"),
wellKnownFetcher,
resolvedConfig: makeResolvedConfig(configTransport.livekit_service_url),
logger: rootLogger,
});
await expect(
discovery.discoverPreferredTransport(),
).resolves.toStrictEqual(wellKnownTransport);
expect(wellKnownFetcher).toHaveBeenCalledWith("example.org");
const discoveredTransport = await discovery.discoverPreferredTransport();
expect(discoveredTransport).not.toStrictEqual(backendTransport);
expect(discoveredTransport).toStrictEqual(configTransport);
},
);
it("skips backend discovery in widget mode and uses well-known", async () => {
const client = makeClient();
// widget mode is detected by the absence of an access token
client.getAccessToken.mockReturnValue(null);
const wellKnownFetcher = vi
.fn<(domain: string) => Promise<IClientWellKnown>>()
.mockResolvedValue(makeWellKnown([wellKnownTransport]));
const discovery = new RtcTransportAutoDiscovery({
client,
resolvedConfig: makeResolvedConfig("https://config.example.org"),
wellKnownFetcher,
logger: rootLogger,
});
await expect(discovery.discoverPreferredTransport()).resolves.toStrictEqual(
wellKnownTransport,
);
expect(client._unstable_getRTCTransports).not.toHaveBeenCalled();
expect(wellKnownFetcher).toHaveBeenCalledWith("example.org");
});
it("falls back to app config when backend fails and well-known has no rtc_foci", async () => {
const client = makeClient();
client._unstable_getRTCTransports.mockRejectedValue(
new MatrixError({ errcode: "M_UNKNOWN" }, 404),
);
const wellKnownFetcher = vi
.fn<(domain: string) => Promise<IClientWellKnown>>()
.mockResolvedValue({} as IClientWellKnown);
const discovery = new RtcTransportAutoDiscovery({
client,
resolvedConfig: makeResolvedConfig("https://config.example.org"),
wellKnownFetcher,
logger: rootLogger,
});
await expect(discovery.discoverPreferredTransport()).resolves.toStrictEqual(
{
type: "livekit",
livekit_service_url: "https://config.example.org",
},
);
});
it("returns null when backend, well-known and config are all unavailable", async () => {
it("returns null when backend and config are all unavailable", async () => {
const client = makeClient();
client._unstable_getRTCTransports.mockResolvedValue([]);
const wellKnownFetcher = vi
.fn<(domain: string) => Promise<IClientWellKnown>>()
.mockResolvedValue({} as IClientWellKnown);
const discovery = new RtcTransportAutoDiscovery({
client,
resolvedConfig: makeResolvedConfig(undefined),
wellKnownFetcher,
logger: rootLogger,
});
src/state/CallViewModel/localMember/RtcTransportAutoDiscovery.ts
+15 -84
View File
@@ -8,7 +8,7 @@ import {
isLivekitTransportConfig,
type LivekitTransportConfig,
} from "matrix-js-sdk/lib/matrixrtc";
import { type IClientWellKnown, type MatrixClient } from "matrix-js-sdk";
import { type MatrixClient } from "matrix-js-sdk";
import { type Logger } from "matrix-js-sdk/lib/logger";
import type { ResolvedConfigOptions } from "../../../config/ConfigOptions.ts";
@@ -16,33 +16,27 @@ import { doNetworkOperationWithRetry } from "../../../utils/matrix.ts";
type TransportDiscoveryClient = Pick<
MatrixClient,
"getDomain" | "_unstable_getRTCTransports" | "getAccessToken"
"getDomain" | "_unstable_getRTCTransports"
>;
export interface RtcTransportAutoDiscoveryProps {
client: TransportDiscoveryClient;
resolvedConfig: ResolvedConfigOptions;
wellKnownFetcher: (domain: string) => Promise<IClientWellKnown>;
logger: Logger;
}
export class RtcTransportAutoDiscovery {
private readonly client: TransportDiscoveryClient;
private readonly resolvedConfig: ResolvedConfigOptions;
private readonly wellKnownFetcher: (
domain: string,
) => Promise<IClientWellKnown>;
private readonly logger: Logger;
public constructor({
client,
resolvedConfig,
wellKnownFetcher,
logger,
}: RtcTransportAutoDiscoveryProps) {
this.client = client;
this.resolvedConfig = resolvedConfig;
this.wellKnownFetcher = wellKnownFetcher;
this.logger = logger.getChild("[RtcTransportAutoDiscovery]");
}
@@ -56,21 +50,7 @@ export class RtcTransportAutoDiscovery {
return backendTransport;
}
this.logger.info("No backend transport found, falling back to well-known");
// 2) .well-known transports
const wellKnownTransport = await this.tryWellKnownTransports();
if (wellKnownTransport) {
this.logger.info(
`Found .well-known transport: ${wellKnownTransport.livekit_service_url}`,
);
return wellKnownTransport;
}
this.logger.info(
"No .well-known transport found, falling back to app config",
);
// 3) app config URL
// 2) app config URL
const configTransport = this.tryConfigTransport();
if (configTransport) {
this.logger.info(
@@ -90,72 +70,23 @@ export class RtcTransportAutoDiscovery {
private async tryBackendTransports(): Promise<LivekitTransportConfig | null> {
const client = this.client;
// MSC4143: Attempt to fetch transports from backend.
// TODO: Workaround for an issue in the js-sdk RoomWidgetClient that
// is not yet implementing _unstable_getRTCTransports properly (via widget API new action).
// For now we just skip this call if we are in a widget.
// In widget mode the client is a `RoomWidgetClient` which has no access token (it is using the widget API).
// Could be removed once the js-sdk is fixed (https://github.com/matrix-org/matrix-js-sdk/issues/5245)
const isSPA = !!client.getAccessToken();
if (isSPA && "_unstable_getRTCTransports" in client) {
this.logger.info("First try to use getRTCTransports end point ...");
try {
const transportList = await doNetworkOperationWithRetry(async () =>
client._unstable_getRTCTransports(),
);
const first = transportList.filter(isLivekitTransportConfig)[0];
if (first) {
return first;
} else {
this.logger.info(
`No livekit transport found in getRTCTransports end point`,
transportList,
);
}
} catch (ex) {
this.logger.info(`Failed to use getRTCTransports end point: ${ex}`);
}
} else {
this.logger.debug(`getRTCTransports end point not available`);
}
return null;
}
/**
* Fetches the first rtc_foci from the .well-known/matrix/client.
* This will not throw errors, but instead just log them and return null if the expected config is not found or malformed.
* @private
*/
private async tryWellKnownTransports(): Promise<LivekitTransportConfig | null> {
// Legacy MSC4143 (to be removed) WELL_KNOWN: Prioritize the .well-known/matrix/client, if available.
const client = this.client;
const domain = client.getDomain();
if (domain) {
// we use AutoDiscovery instead of relying on the MatrixClient having already
// been fully configured and started
const wellKnownFoci = await this.wellKnownFetcher(domain);
const fociConfig = wellKnownFoci["org.matrix.msc4143.rtc_foci"];
if (fociConfig) {
if (!Array.isArray(fociConfig)) {
this.logger.warn(
`org.matrix.msc4143.rtc_foci is not an array in .well-known`,
);
} else {
return fociConfig[0];
}
this.logger.info("First try to use getRTCTransports end point ...");
try {
const transportList = await doNetworkOperationWithRetry(async () =>
client._unstable_getRTCTransports(),
);
const first = transportList.find(isLivekitTransportConfig);
if (first) {
return first;
} else {
this.logger.info(
`No .well-known "org.matrix.msc4143.rtc_foci" found for ${domain}`,
wellKnownFoci,
`No livekit transport found in getRTCTransports end point`,
transportList,
);
}
} else {
// Should never happen, but just in case
this.logger.warn(`No domain configured for client`);
} catch (ex) {
this.logger.info(`Failed to use getRTCTransports end point: ${ex}`);
}
return null;
}
src/state/CallViewModel/remoteMembers/Connection.ts
+11 -7
View File
@@ -12,8 +12,9 @@ import {
} from "@livekit/components-core";
import {
ConnectionError,
type Room as LivekitRoom,
ConnectionErrorReason,
type RemoteParticipant,
type Room as LivekitRoom,
} from "livekit-client";
import { type LivekitTransportConfig } from "matrix-js-sdk/lib/matrixrtc";
import { BehaviorSubject, map } from "rxjs";
@@ -30,10 +31,11 @@ import { type ObservableScope } from "../../ObservableScope.ts";
import {
ElementCallError,
InsufficientCapacityError,
LivekitConnectionError,
PeerConnectionTimeoutError,
SFURoomCreationRestrictedError,
UnknownCallError,
} from "../../../utils/errors.ts";
import { type JwtEndpointVersion } from "../localMember/LocalTransport.ts";
export interface ConnectionOpts {
/**
@@ -41,11 +43,6 @@ export interface ConnectionOpts {
* On top the local transport will send additional data to the jwt server to use delayed event delegation.
*/
existingSFUConfig?: SFUConfig;
/**
* For local connections that use the oldest member pattern. here we have not prefetched the sfuConfig
* and hence we need to let the connection do the jwt token fetching.
*/
forceJwtEndpoint?: JwtEndpointVersion;
/** The identity parts to use on this connection */
ownMembershipIdentity: CallMembershipIdentityParts;
/** The media transport to connect to. */
@@ -248,6 +245,13 @@ export class Connection {
// In the first case there will not be a 404, so we are in the second case.
throw new SFURoomCreationRestrictedError();
}
if (e.reason === ConnectionErrorReason.Timeout) {
// Unabled to establish peer connection within the timeout
throw new PeerConnectionTimeoutError();
}
throw new LivekitConnectionError(e);
}
throw e;
}
src/state/CallViewModel/remoteMembers/ConnectionFactory.ts
+77 -41
View File
@@ -12,7 +12,7 @@ import {
type E2EEManagerOptions,
type BaseE2EEManager,
} from "livekit-client";
import { type Logger } from "matrix-js-sdk/lib/logger";
import { logger, type Logger } from "matrix-js-sdk/lib/logger";
// imported as inline to support worker when loaded from a cdn (cross domain)
import E2EEWorker from "livekit-client/e2ee-worker?worker&inline";
import { type CallMembershipIdentityParts } from "matrix-js-sdk/lib/matrixrtc/EncryptionManager";
@@ -27,7 +27,26 @@ import type {
import type { MediaDevices } from "../../MediaDevices.ts";
import type { Behavior } from "../../Behavior.ts";
import type { ProcessorState } from "../../../livekit/TrackProcessorContext.tsx";
import { defaultLiveKitOptions } from "../../../livekit/options.ts";
import { getLiveKitOptions } from "../../../livekit/options.ts";
import {
advancedCamera,
cameraResolution,
cameraFramerate,
cameraBitrate,
cameraCodec,
parseResolution,
echoCancellationSetting,
noiseSuppressionSetting,
autoGainControlSetting,
} from "../../../settings/settings.ts";
// [lotus] Upstream v0.25.0 moved the audio-capture constraints from URL params
// to persisted Settings. The Lotus host (cinny) still drives them per-call via
// URL params (it turns browser noiseSuppression/AGC OFF when the in-source ML
// denoiser is active, so the model gets a raw mic). Re-wire: a constraint is
// enabled only if BOTH the Setting and the URL param allow it. The params
// default to true, so with no params this is byte-for-byte upstream behaviour;
// `?autoGainControl=false` forces AGC off regardless of the stored Setting.
import { getUrlParams } from "../../../UrlParams.ts";
// TODO evaluate if this should be done like the Publisher Factory
export interface ConnectionFactory {
@@ -53,9 +72,6 @@ export class ECConnectionFactory implements ConnectionFactory {
* @param livekitKeyProvider - Optional key provider for end-to-end encryption.
* @param controlledAudioDevices - Option to indicate whether audio output device is controlled externally (native mobile app).
* @param livekitRoomFactory - Optional factory function (for testing) to create LivekitRoom instances. If not provided, a default factory is used.
* @param echoCancellation - Whether to enable echo cancellation for audio capture.
* @param noiseSuppression - Whether to enable noise suppression for audio capture.
* @param autoGainControl - Whether to enable auto gain control for audio capture.
*/
public constructor(
private client: OpenIDClientParts,
@@ -65,27 +81,22 @@ export class ECConnectionFactory implements ConnectionFactory {
livekitKeyProvider: BaseKeyProvider | undefined,
private controlledAudioDevices: boolean,
livekitRoomFactory?: () => LivekitRoom,
echoCancellation: boolean = true,
noiseSuppression: boolean = true,
autoGainControl: boolean = true,
) {
const defaultFactory = (): LivekitRoom =>
new LivekitRoom(
generateRoomOption({
devices: this.devices,
processorState: this.processorState$.value,
e2eeLivekitOptions: livekitKeyProvider && {
keyProvider: livekitKeyProvider,
// It's important that every room use a separate E2EE worker.
// They get confused if given streams from multiple rooms.
worker: new E2EEWorker(),
},
controlledAudioDevices: this.controlledAudioDevices,
echoCancellation,
noiseSuppression,
autoGainControl,
}),
);
const defaultFactory = (): LivekitRoom => {
const roomOptions = generateRoomOption({
devices: this.devices,
processorState: this.processorState$.value,
e2eeLivekitOptions: livekitKeyProvider && {
keyProvider: livekitKeyProvider,
// It's important that every room use a separate E2EE worker.
// They get confused if given streams from multiple rooms.
worker: new E2EEWorker(),
},
controlledAudioDevices: this.controlledAudioDevices,
});
logger.info("[ECConnectionFactory] livekit room options: ", roomOptions);
return new LivekitRoom(roomOptions);
};
this.livekitRoomFactory = livekitRoomFactory ?? defaultFactory;
}
@@ -122,15 +133,13 @@ export class ECConnectionFactory implements ConnectionFactory {
/**
* Generate the initial LiveKit RoomOptions based on the current media devices and processor state.
* Reads audio processing and camera quality settings directly from Settings.
*/
function generateRoomOption({
devices,
processorState,
e2eeLivekitOptions,
controlledAudioDevices,
echoCancellation,
noiseSuppression,
autoGainControl,
}: {
devices: MediaDevices;
processorState: ProcessorState;
@@ -139,23 +148,50 @@ function generateRoomOption({
| { e2eeManager: BaseE2EEManager }
| undefined;
controlledAudioDevices: boolean;
echoCancellation: boolean;
noiseSuppression: boolean;
autoGainControl: boolean;
}): RoomOptions {
const liveKitOptions = getLiveKitOptions();
// Apply advanced camera settings if enabled
let videoCaptureDefaults = {
...liveKitOptions.videoCaptureDefaults,
deviceId: devices.videoInput.selected$.value?.id,
processor: processorState.processor,
};
let publishDefaults = liveKitOptions.publishDefaults;
if (advancedCamera.getValue()) {
const { width, height } = parseResolution(cameraResolution.getValue());
const fps = cameraFramerate.getValue();
const bps = cameraBitrate.getValue();
const codec = cameraCodec.getValue();
videoCaptureDefaults = {
...videoCaptureDefaults,
resolution: { width, height, frameRate: fps },
};
publishDefaults = {
...publishDefaults,
videoEncoding: { maxBitrate: bps, maxFramerate: fps },
videoCodec: codec,
};
}
return {
...defaultLiveKitOptions,
videoCaptureDefaults: {
...defaultLiveKitOptions.videoCaptureDefaults,
deviceId: devices.videoInput.selected$.value?.id,
processor: processorState.processor,
},
...liveKitOptions,
videoCaptureDefaults,
publishDefaults,
audioCaptureDefaults: {
...defaultLiveKitOptions.audioCaptureDefaults,
...liveKitOptions.audioCaptureDefaults,
deviceId: devices.audioInput.selected$.value?.id,
echoCancellation,
noiseSuppression,
autoGainControl,
echoCancellation:
echoCancellationSetting.getValue() &&
getUrlParams().echoCancellation !== false,
noiseSuppression:
noiseSuppressionSetting.getValue() &&
getUrlParams().noiseSuppression !== false,
autoGainControl:
autoGainControlSetting.getValue() &&
getUrlParams().autoGainControl !== false,
},
audioOutput: {
// When using controlled audio devices, we don't want to set the
src/state/CallViewModel/remoteMembers/ECConnectionFactory.test.ts
+133 -15
View File
@@ -22,6 +22,16 @@ import {
} from "../../../utils/test.ts";
import type { ProcessorState } from "../../../livekit/TrackProcessorContext.tsx";
import { constant } from "../../Behavior";
import {
echoCancellationSetting,
noiseSuppressionSetting,
autoGainControlSetting,
advancedCamera,
cameraResolution,
cameraFramerate,
cameraBitrate,
cameraCodec,
} from "../../../settings/settings.ts";
// At the top of your test file, after imports
vi.mock("livekit-client", async (importOriginal) => {
@@ -53,15 +63,19 @@ beforeEach(() => {
describe("ECConnectionFactory - Audio inputs options", () => {
test.each([
{ echo: true, noise: true, agc: true },
{ echo: true, noise: false, agc: false },
{ echo: false, noise: true, agc: false },
{ echo: false, noise: false, agc: true },
{ echo: true, noise: true },
{ echo: true, noise: false },
{ echo: false, noise: true },
{ echo: false, noise: false },
])(
"it sets echoCancellation=$echo, noiseSuppression=$noise, autoGainControl=$agc based on constructor parameters",
({ echo, noise, agc }) => {
"it sets echoCancellation=$echo and noiseSuppression=$noise based on settings",
({ echo, noise }) => {
const RoomConstructor = vi.mocked(LivekitRoom);
// Set audio processing settings
echoCancellationSetting.setValue(echo);
noiseSuppressionSetting.setValue(noise);
const ecConnectionFactory = new ECConnectionFactory(
mockClient,
"!roomid:example.org",
@@ -72,10 +86,6 @@ describe("ECConnectionFactory - Audio inputs options", () => {
}),
undefined,
false,
undefined,
echo,
noise,
agc,
);
ecConnectionFactory.createConnection(
testScope,
@@ -90,7 +100,6 @@ describe("ECConnectionFactory - Audio inputs options", () => {
audioCaptureDefaults: expect.objectContaining({
echoCancellation: echo,
noiseSuppression: noise,
autoGainControl: agc,
}),
}),
);
@@ -102,9 +111,13 @@ describe("ECConnectionFactory - ControlledAudioDevice", () => {
test.each([{ controlled: true }, { controlled: false }])(
"it sets controlledAudioDevice=$controlled then uses deviceId accordingly",
({ controlled }) => {
// test("it sets echoCancellation and noiseSuppression based on constructor parameters", () => {
const RoomConstructor = vi.mocked(LivekitRoom);
// Explicitly set audio settings so the test doesn't depend on defaults
echoCancellationSetting.setValue(true);
noiseSuppressionSetting.setValue(true);
autoGainControlSetting.setValue(true);
const ecConnectionFactory = new ECConnectionFactory(
mockClient,
"!roomid:example.org",
@@ -121,9 +134,6 @@ describe("ECConnectionFactory - ControlledAudioDevice", () => {
}),
undefined,
controlled,
undefined,
false,
false,
);
ecConnectionFactory.createConnection(
testScope,
@@ -144,6 +154,114 @@ describe("ECConnectionFactory - ControlledAudioDevice", () => {
);
});
describe("ECConnectionFactory - Camera quality settings", () => {
test("it uses default video options when advancedCamera is disabled", () => {
const RoomConstructor = vi.mocked(LivekitRoom);
advancedCamera.setValue(false);
const ecConnectionFactory = new ECConnectionFactory(
mockClient,
"!roomid:example.org",
mockMediaDevices({}),
new BehaviorSubject<ProcessorState>({
supported: true,
processor: undefined,
}),
undefined,
false,
);
ecConnectionFactory.createConnection(
testScope,
exampleTransport,
ownMemberMock,
logger,
);
// publishDefaults should use config defaults (vp8), not custom settings
expect(RoomConstructor).toHaveBeenCalledWith(
expect.objectContaining({
publishDefaults: expect.objectContaining({
videoCodec: "vp8",
}),
}),
);
});
test("it applies custom camera resolution, encoding, and codec when advancedCamera is enabled", () => {
const RoomConstructor = vi.mocked(LivekitRoom);
advancedCamera.setValue(true);
cameraResolution.setValue("1920x1080");
cameraFramerate.setValue(60);
cameraBitrate.setValue(4_000_000);
cameraCodec.setValue("vp9");
const ecConnectionFactory = new ECConnectionFactory(
mockClient,
"!roomid:example.org",
mockMediaDevices({}),
new BehaviorSubject<ProcessorState>({
supported: true,
processor: undefined,
}),
undefined,
false,
);
ecConnectionFactory.createConnection(
testScope,
exampleTransport,
ownMemberMock,
logger,
);
expect(RoomConstructor).toHaveBeenCalledWith(
expect.objectContaining({
videoCaptureDefaults: expect.objectContaining({
resolution: { width: 1920, height: 1080, frameRate: 60 },
}),
publishDefaults: expect.objectContaining({
videoEncoding: { maxBitrate: 4_000_000, maxFramerate: 60 },
videoCodec: "vp9",
}),
}),
);
});
test("it applies autoGainControl from settings", () => {
const RoomConstructor = vi.mocked(LivekitRoom);
autoGainControlSetting.setValue(false);
echoCancellationSetting.setValue(true);
noiseSuppressionSetting.setValue(true);
const ecConnectionFactory = new ECConnectionFactory(
mockClient,
"!roomid:example.org",
mockMediaDevices({}),
new BehaviorSubject<ProcessorState>({
supported: true,
processor: undefined,
}),
undefined,
false,
);
ecConnectionFactory.createConnection(
testScope,
exampleTransport,
ownMemberMock,
logger,
);
expect(RoomConstructor).toHaveBeenCalledWith(
expect.objectContaining({
audioCaptureDefaults: expect.objectContaining({
autoGainControl: false,
}),
}),
);
});
});
afterEach(() => {
testScope.end();
fetchMock.reset();
src/state/CallViewModel/remoteMembers/MatrixLivekitMembers.test.ts
+17 -11
View File
@@ -15,7 +15,7 @@ import { BehaviorSubject, combineLatest, map, type Observable } from "rxjs";
import { type IConnectionManager } from "./ConnectionManager.ts";
import {
type RemoteMatrixLivekitMember,
createMatrixLivekitMembers$,
createRemoteMatrixLivekitMembers$,
} from "./MatrixLivekitMembers.ts";
import {
Epoch,
@@ -31,6 +31,7 @@ import {
} from "../../../utils/test.ts";
import { type Connection } from "./Connection.ts";
import { constant } from "../../Behavior.ts";
import { localRtcMember } from "../../../utils/test-fixtures.ts";
let testScope: ObservableScope;
@@ -88,16 +89,17 @@ test("should signal participant not yet connected to livekit", async () => {
mockConnectionManagerData$,
);
const matrixLivekitMember$ = createMatrixLivekitMembers$({
const remoteMatrixLivekitMembers$ = createRemoteMatrixLivekitMembers$({
scope: testScope,
membershipsWithTransport$: testScope.behavior(membershipsWithTransport$),
connectionManager: {
connectionManagerData$: connectionManagerData$,
} as unknown as IConnectionManager,
localUser: localRtcMember,
});
await flushPromises();
expect(matrixLivekitMember$.value.value).toSatisfy(
expect(remoteMatrixLivekitMembers$.value.value).toSatisfy(
(data: RemoteMatrixLivekitMember[]) => {
expect(data.length).toEqual(1);
expect(data[0].membership$.value).toBe(bobMembership);
@@ -157,16 +159,17 @@ test("should signal participant on a connection that is publishing", async () =>
constant(dataWithPublisher),
);
const matrixLivekitMember$ = createMatrixLivekitMembers$({
const remoteMatrixLivekitMembers$ = createRemoteMatrixLivekitMembers$({
scope: testScope,
membershipsWithTransport$: testScope.behavior(membershipsWithTransport$),
connectionManager: {
connectionManagerData$: connectionManagerData$,
} as unknown as IConnectionManager,
localUser: localRtcMember,
});
await flushPromises();
expect(matrixLivekitMember$.value.value).toSatisfy(
expect(remoteMatrixLivekitMembers$.value.value).toSatisfy(
(data: RemoteMatrixLivekitMember[]) => {
expect(data.length).toEqual(1);
expect(data[0].membership$.value).toBe(bobMembership);
@@ -197,15 +200,16 @@ test("should signal participant on a connection that is not publishing", async (
constant(dataWithPublisher),
);
const matrixLivekitMember$ = createMatrixLivekitMembers$({
const remoteMatrixLivekitMembers$ = createRemoteMatrixLivekitMembers$({
scope: testScope,
membershipsWithTransport$: testScope.behavior(membershipsWithTransport$),
connectionManager: {
connectionManagerData$: connectionManagerData$,
} as unknown as IConnectionManager,
localUser: localRtcMember,
});
await flushPromises();
expect(matrixLivekitMember$.value.value).toSatisfy(
expect(remoteMatrixLivekitMembers$.value.value).toSatisfy(
(data: RemoteMatrixLivekitMember[]) => {
expect(data.length).toEqual(1);
expect(data[0].membership$.value).toBe(bobMembership);
@@ -245,15 +249,16 @@ describe("Publication edge case", () => {
constant(connectionWithPublisher),
);
const matrixLivekitMembers$ = createMatrixLivekitMembers$({
const remoteMatrixLivekitMembers$ = createRemoteMatrixLivekitMembers$({
scope: testScope,
membershipsWithTransport$: testScope.behavior(membershipsWithTransport$),
connectionManager: {
connectionManagerData$: connectionManagerData$,
} as unknown as IConnectionManager,
localUser: localRtcMember,
});
await flushPromises();
expect(matrixLivekitMembers$.value.value).toSatisfy(
expect(remoteMatrixLivekitMembers$.value.value).toSatisfy(
(data: RemoteMatrixLivekitMember[]) => {
expect(data.length).toEqual(2);
expect(data[0].membership$.value).toBe(bobMembership);
@@ -303,16 +308,17 @@ test("bob is publishing in the wrong connection", async () => {
connectionsWithPublisher$,
);
const matrixLivekitMember$ = createMatrixLivekitMembers$({
const remoteMatrixLivekitMembers$ = createRemoteMatrixLivekitMembers$({
scope: testScope,
membershipsWithTransport$: testScope.behavior(membershipsWithTransport$),
connectionManager: {
connectionManagerData$: connectionManagerData$,
} as unknown as IConnectionManager,
localUser: localRtcMember,
});
await flushPromises();
expect(matrixLivekitMember$.value.value).toSatisfy(
expect(remoteMatrixLivekitMembers$.value.value).toSatisfy(
(data: RemoteMatrixLivekitMember[]) => {
expect(data.length).toEqual(2);
expect(data[0].membership$.value).toBe(bobMembership);
src/state/CallViewModel/remoteMembers/MatrixLivekitMembers.ts
+13 -3
View File
@@ -62,7 +62,9 @@ interface Props {
Epoch<{ membership: CallMembership; transport?: LivekitTransportConfig }[]>
>;
connectionManager: IConnectionManager;
localUser: { deviceId: string; userId: string };
}
/**
* Combines MatrixRTC and Livekit worlds.
*
@@ -73,13 +75,14 @@ interface Props {
* - out (via public Observable):
* - `remoteMatrixLivekitMember` an observable of MatrixLivekitMember[] to track the remote members and associated livekit data.
*/
export function createMatrixLivekitMembers$({
export function createRemoteMatrixLivekitMembers$({
scope,
membershipsWithTransport$,
connectionManager,
localUser,
}: Props): Behavior<Epoch<RemoteMatrixLivekitMember[]>> {
/**
* Stream of all the call members and their associated livekit data (if available).
* Behavior of all the remote call members and their associated livekit data (if available).
*/
return scope.behavior(
combineLatest([
@@ -91,12 +94,19 @@ export function createMatrixLivekitMembers$({
),
map(([ms, data]) => new Epoch([ms.value, data.value] as const, ms.epoch)),
generateItemsWithEpoch(
"MatrixLivekitMembers",
"RemoteMatrixLivekitMembers",
// Generator function.
// creates an array of `{key, data}[]`
// Each change in the keys (new key) will result in a call to the factory function.
function* ([membershipsWithTransport, managerData]) {
for (const { membership, transport } of membershipsWithTransport) {
// Exclude the local membership
if (
membership.userId === localUser.userId &&
membership.deviceId === localUser.deviceId
)
continue;
const participants = transport
? managerData.getParticipantsForTransport(transport)
: [];
src/state/CallViewModel/remoteMembers/MatrixMemberMetadata.test.ts
+44 -25
View File
@@ -105,7 +105,7 @@ describe("MatrixMemberMetadata", () => {
}
it("should show our own user if present in rtc session and room", () => {
withTestScheduler(({ behavior, expectObservable }) => {
withTestScheduler(({ scope, behavior, expectObservable }) => {
fakeMemberWith({
userId: "@local:example.com",
rawDisplayName: "it's a me",
@@ -118,8 +118,10 @@ describe("MatrixMemberMetadata", () => {
memberships$,
createRoomMembers$(testScope, mockMatrixRoom),
);
const dn$ =
metadataStore.createDisplayNameBehavior$("@local:example.com");
const dn$ = metadataStore.createDisplayNameBehavior$(
scope,
"@local:example.com",
);
expectObservable(dn$).toBe("a", {
a: "it's a me",
@@ -146,7 +148,7 @@ describe("MatrixMemberMetadata", () => {
it("should get displayName for users", () => {
setUpBasicRoom();
withTestScheduler(({ behavior, expectObservable }) => {
withTestScheduler(({ scope, behavior, expectObservable }) => {
const memberships$ = behavior("a", {
a: [
mockRtcMembership("@alice:example.com", "DEVICE1"),
@@ -158,8 +160,10 @@ describe("MatrixMemberMetadata", () => {
memberships$,
createRoomMembers$(testScope, mockMatrixRoom),
);
const aliceDispName$ =
metadataStore.createDisplayNameBehavior$("@alice:example.com");
const aliceDispName$ = metadataStore.createDisplayNameBehavior$(
scope,
"@alice:example.com",
);
expectObservable(aliceDispName$).toBe("a", {
a: "Alice",
@@ -322,7 +326,7 @@ describe("MatrixMemberMetadata", () => {
});
it("should track individual member id with createDisplayNameBehavior", () => {
withTestScheduler(({ behavior, schedule, expectObservable }) => {
withTestScheduler(({ scope, behavior, schedule, expectObservable }) => {
setUpBasicRoom();
const BOB = "@bob:example.com";
const CARL = "@carl:example.com";
@@ -356,8 +360,8 @@ describe("MatrixMemberMetadata", () => {
createRoomMembers$(testScope, mockMatrixRoom),
);
const bob$ = metadataStore.createDisplayNameBehavior$(BOB);
const carl$ = metadataStore.createDisplayNameBehavior$(CARL);
const bob$ = metadataStore.createDisplayNameBehavior$(scope, BOB);
const carl$ = metadataStore.createDisplayNameBehavior$(scope, CARL);
expectObservable(bob$).toBe("abc-", {
a: undefined,
@@ -378,7 +382,7 @@ describe("MatrixMemberMetadata", () => {
});
it("should disambiguate users with invisible characters", () => {
withTestScheduler(({ behavior, expectObservable }) => {
withTestScheduler(({ scope, behavior, expectObservable }) => {
const bobRtcMember = mockRtcMembership("@bob:example.org", "BBBB");
const bobZeroWidthSpaceRtcMember = mockRtcMembership(
"@bob2:example.org",
@@ -411,12 +415,18 @@ describe("MatrixMemberMetadata", () => {
createRoomMembers$(testScope, mockMatrixRoom),
);
const bob$ =
metadataStore.createDisplayNameBehavior$("@bob:example.org");
const bob2$ =
metadataStore.createDisplayNameBehavior$("@bob2:example.org");
const carol$ =
metadataStore.createDisplayNameBehavior$("@carol:example.org");
const bob$ = metadataStore.createDisplayNameBehavior$(
scope,
"@bob:example.org",
);
const bob2$ = metadataStore.createDisplayNameBehavior$(
scope,
"@bob2:example.org",
);
const carol$ = metadataStore.createDisplayNameBehavior$(
scope,
"@carol:example.org",
);
expectObservable(bob$).toBe("ab", {
a: "Bob",
b: "Bob (@bob:example.org)",
@@ -517,7 +527,7 @@ describe("MatrixMemberMetadata", () => {
}
it("should use avatar url from room members", () => {
withTestScheduler(({ behavior, expectObservable }) => {
withTestScheduler(({ scope, behavior, expectObservable }) => {
fakeMemberWith({
userId: "@local:example.com",
});
@@ -536,11 +546,15 @@ describe("MatrixMemberMetadata", () => {
memberships$,
createRoomMembers$(testScope, mockMatrixRoom),
);
const local$ =
metadataStore.createAvatarUrlBehavior$("@local:example.com");
const local$ = metadataStore.createAvatarUrlBehavior$(
scope,
"@local:example.com",
);
const alice$ =
metadataStore.createAvatarUrlBehavior$("@alice:example.com");
const alice$ = metadataStore.createAvatarUrlBehavior$(
scope,
"@alice:example.com",
);
expectObservable(local$).toBe("a", {
a: "mxc://example.com/@local:example.com",
@@ -558,7 +572,7 @@ describe("MatrixMemberMetadata", () => {
});
it("should update on avatar change and user join/leave", () => {
withTestScheduler(({ behavior, schedule, expectObservable }) => {
withTestScheduler(({ scope, behavior, schedule, expectObservable }) => {
fakeMemberWith({ userId: "@carl:example.com" });
fakeMemberWith({ userId: "@bob:example.com" });
const memberships$ = behavior("ab-d", {
@@ -585,9 +599,14 @@ describe("MatrixMemberMetadata", () => {
},
});
const bob$ = metadataStore.createAvatarUrlBehavior$("@bob:example.com");
const carl$ =
metadataStore.createAvatarUrlBehavior$("@carl:example.com");
const bob$ = metadataStore.createAvatarUrlBehavior$(
scope,
"@bob:example.com",
);
const carl$ = metadataStore.createAvatarUrlBehavior$(
scope,
"@carl:example.com",
);
expectObservable(bob$).toBe("a---", {
a: "mxc://example.com/@bob:example.com",
});
src/state/CallViewModel/remoteMembers/MatrixMemberMetadata.ts
+11 -6
View File
@@ -22,8 +22,6 @@ import {
} from "../../../utils/displayname";
import { type Behavior } from "../../Behavior";
const logger = rootLogger.getChild("[MatrixMemberMetadata]");
export type RoomMemberMap = Map<
string,
Pick<RoomMember, "userId" | "getMxcAvatarUrl" | "rawDisplayName">
@@ -67,6 +65,7 @@ export const memberDisplaynames$ = (
memberships$: Behavior<Pick<CallMembership, "userId">[]>,
roomMembers$: Behavior<RoomMemberMap>,
): Behavior<Map<string, string>> => {
const logger = rootLogger.getChild("[MatrixMemberMetadata]");
// This map tracks userIds that at some point needed disambiguation.
// This is a memory leak bound to the number of participants.
// A call application will always increase the memory if there have been more members in a call.
@@ -115,8 +114,14 @@ export const createMatrixMemberMetadata$ = (
memberships$: Behavior<Pick<CallMembership, "userId">[]>,
roomMembers$: Behavior<RoomMemberMap>,
): {
createDisplayNameBehavior$: (userId: string) => Behavior<string | undefined>;
createAvatarUrlBehavior$: (userId: string) => Behavior<string | undefined>;
createDisplayNameBehavior$: (
scope: ObservableScope,
userId: string,
) => Behavior<string | undefined>;
createAvatarUrlBehavior$: (
scope: ObservableScope,
userId: string,
) => Behavior<string | undefined>;
displaynameMap$: Behavior<Map<string, string>>;
avatarMap$: Behavior<Map<string, string | undefined>>;
} => {
@@ -136,13 +141,13 @@ export const createMatrixMemberMetadata$ = (
),
);
return {
createDisplayNameBehavior$: (userId: string) =>
createDisplayNameBehavior$: (scope: ObservableScope, userId: string) =>
scope.behavior(
displaynameMap$.pipe(
map((displaynameMap) => displaynameMap.get(userId)),
),
),
createAvatarUrlBehavior$: (userId: string) =>
createAvatarUrlBehavior$: (scope: ObservableScope, userId: string) =>
scope.behavior(
roomMembers$.pipe(
map((roomMembers) => roomMembers.get(userId)?.getMxcAvatarUrl()),
src/state/CallViewModel/remoteMembers/integration.test.ts
+5 -4
View File
@@ -29,13 +29,13 @@ import {
import { type ProcessorState } from "../../../livekit/TrackProcessorContext.tsx";
import {
areLivekitTransportsEqual,
createMatrixLivekitMembers$,
createRemoteMatrixLivekitMembers$,
type RemoteMatrixLivekitMember,
} from "./MatrixLivekitMembers.ts";
import { createConnectionManager$ } from "./ConnectionManager.ts";
import { membershipsAndTransports$ } from "../../SessionBehaviors.ts";
import { constant } from "../../Behavior.ts";
import { testJWTToken } from "../../../utils/test-fixtures.ts";
import { localRtcMember, testJWTToken } from "../../../utils/test-fixtures.ts";
// Test the integration of ConnectionManager and MatrixLivekitMerger
@@ -130,14 +130,15 @@ test("bob, carl, then bob joining no tracks yet", () => {
ownMembershipIdentity: ownMemberMock,
});
const matrixLivekitMembers$ = createMatrixLivekitMembers$({
const remoteMatrixLivekitMembers$ = createRemoteMatrixLivekitMembers$({
scope: testScope,
membershipsWithTransport$:
membershipsAndTransports.membershipsWithTransport$,
connectionManager,
localUser: localRtcMember,
});
expectObservable(matrixLivekitMembers$).toBe(vMarble, {
expectObservable(remoteMatrixLivekitMembers$).toBe(vMarble, {
a: expect.toSatisfy((e: Epoch<RemoteMatrixLivekitMember[]>) => {
const items = e.value;
expect(items.length).toBe(1);
src/state/CallViewModelWidget.test.ts
+2 -6
View File
@@ -15,7 +15,7 @@ import { constant } from "./Behavior.ts";
import { aliceParticipant, localRtcMember } from "../utils/test-fixtures.ts";
import { ElementWidgetActions, widget } from "../widget.ts";
import { E2eeType } from "../e2ee/e2eeType.ts";
import { MatrixRTCMode } from "../settings/settings.ts";
import { MatrixRTCMode } from "../config/ConfigOptions.ts";
vi.mock("@livekit/components-core", { spy: true });
@@ -35,11 +35,7 @@ vi.mock("../widget", () => ({
},
}));
it.each([
[MatrixRTCMode.Legacy],
[MatrixRTCMode.Compatibility],
[MatrixRTCMode.Matrix_2_0],
])(
it.each([[MatrixRTCMode.Compatibility], [MatrixRTCMode.Matrix_2_0]])(
"expect leave when ElementWidgetActions.HangupCall is called (%s mode)",
async (mode) => {
const pr = Promise.withResolvers<string>();
src/state/IOSControlledAudioOutput.test.ts
+162
View File
@@ -0,0 +1,162 @@
/*
Copyright 2026 Element Corp.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { type Observable, of } from "rxjs";
import { ObservableScope } from "./ObservableScope";
import { constant } from "./Behavior";
import { type SelectedAudioOutputDevice } from "./MediaDevices";
import {
availableOutputDevices$,
type Controls,
type OutputDevice,
outputDevice$,
} from "../controls";
import {
EARPIECE_CONFIG_ID,
IOSControlledAudioOutput,
} from "./IOSControlledAudioOutput";
// `vi.mock` calls are hoisted above all imports, so the static imports below
// already see these mocks. Force the iOS platform so that the virtual earpiece
// is available, and stub the livekit device observer (only subscribed for its
// side effects).
vi.mock("../Platform", () => ({ platform: "ios" }));
vi.mock("@livekit/components-core", () => ({
createMediaDeviceObserver: (): Observable<MediaDeviceInfo[]> => of([]),
}));
// On iOS the host reports a single device for the current route. When output is
// on the loudspeaker it is flagged `forEarpiece`, which makes the controller
// expose a virtual earpiece device.
const SPEAKER: OutputDevice = {
id: "speaker",
name: "Speaker",
isSpeaker: true,
forEarpiece: true,
};
// A connected headset (e.g. Bluetooth) is reported as a plain named device,
// with neither the speaker nor earpiece flag set.
const HEADSET: OutputDevice = {
id: "bt",
name: "AirPods",
};
let testScope: ObservableScope;
beforeEach(() => {
testScope = new ObservableScope();
window.controls = {
onAudioDeviceSelect: vi.fn(),
onOutputDeviceSelect: vi.fn(),
} as unknown as Controls;
});
afterEach(() => {
testScope.end();
});
/**
* Subscribe to the controller's `selected$` and return a getter for the latest
* emitted value.
*/
function latestSelection(
output: InstanceType<typeof IOSControlledAudioOutput>,
): () => SelectedAudioOutputDevice | undefined {
let latest: SelectedAudioOutputDevice | undefined;
output.selected$.subscribe((s) => {
latest = s;
});
return () => latest;
}
describe("Default selection", () => {
it("defaults to the earpiece for voice (audio) calls", () => {
const output = new IOSControlledAudioOutput(
constant(false),
testScope,
"audio",
);
const selected = latestSelection(output);
availableOutputDevices$.next([SPEAKER]);
expect(selected()).toEqual({
id: EARPIECE_CONFIG_ID,
virtualEarpiece: true,
});
expect(window.controls.onAudioDeviceSelect).toHaveBeenLastCalledWith(
EARPIECE_CONFIG_ID,
);
});
it("defaults to the speaker for video calls", () => {
const output = new IOSControlledAudioOutput(
constant(false),
testScope,
"video",
);
const selected = latestSelection(output);
availableOutputDevices$.next([SPEAKER]);
expect(selected()).toEqual({ id: SPEAKER.id, virtualEarpiece: false });
});
it("keeps a headset for voice calls instead of forcing the earpiece", () => {
const output = new IOSControlledAudioOutput(
constant(false),
testScope,
"audio",
);
const selected = latestSelection(output);
// The host proposes the headset as the route (listed first), even though a
// forEarpiece device is also present so the virtual earpiece exists.
availableOutputDevices$.next([HEADSET, SPEAKER]);
expect(selected()).toEqual({ id: HEADSET.id, virtualEarpiece: false });
});
});
describe("Explicit selection", () => {
it("an explicit user selection overrides the earpiece default", () => {
const output = new IOSControlledAudioOutput(
constant(false),
testScope,
"audio",
);
const selected = latestSelection(output);
availableOutputDevices$.next([SPEAKER]);
// Earpiece by default for a voice call...
expect(selected()).toEqual({
id: EARPIECE_CONFIG_ID,
virtualEarpiece: true,
});
// ...until the user explicitly picks the speaker.
output.select(SPEAKER.id);
expect(selected()).toEqual({ id: SPEAKER.id, virtualEarpiece: false });
});
it("a host selection overrides the earpiece default", () => {
const output = new IOSControlledAudioOutput(
constant(false),
testScope,
"audio",
);
const selected = latestSelection(output);
availableOutputDevices$.next([SPEAKER]);
outputDevice$.next(SPEAKER.id);
expect(selected()).toEqual({ id: SPEAKER.id, virtualEarpiece: false });
});
});
src/state/IOSControlledAudioOutput.ts
+35 -2
View File
@@ -8,6 +8,7 @@ Please see LICENSE in the repository root for full details.
import { logger as rootLogger } from "matrix-js-sdk/lib/logger";
import { combineLatest, merge, startWith, Subject, tap } from "rxjs";
import type { RTCCallIntent } from "matrix-js-sdk/lib/matrixrtc";
import {
availableOutputDevices$ as controlledAvailableOutputDevices$,
outputDevice$ as controlledOutputSelection$,
@@ -24,7 +25,7 @@ import {
// This hardcoded id is used in EX ios! It can only be changed in coordination with
// the ios swift team.
const EARPIECE_CONFIG_ID = "earpiece-id";
export const EARPIECE_CONFIG_ID = "earpiece-id";
/**
* A special implementation of audio output that allows the hosting application
@@ -94,7 +95,7 @@ export class IOSControlledAudioOutput implements MediaDevice<
),
],
(available, preferredId) => {
const id = preferredId ?? available.keys().next().value;
const id = preferredId ?? this.chooseDefaultId(available);
return id === undefined
? undefined
: { id, virtualEarpiece: id === EARPIECE_CONFIG_ID };
@@ -106,9 +107,41 @@ export class IOSControlledAudioOutput implements MediaDevice<
),
);
/**
* Chooses the default output device when no explicit selection (from the user
* or the hosting application) has been made yet.
*
* For voice calls (`initialIntent === "audio"`) we want to start on the
* earpiece rather than the speaker, like a regular phone call. We only
* override when the device that would otherwise be the default is the
* speaker: if the host already routed to a headset (e.g. Bluetooth) — which
* is reported as a plain named device, not "speaker"/"earpiece" — we keep it.
* This mirrors the Android behaviour in {@link AndroidControlledAudioOutput}.
*/
private chooseDefaultId(
available: Map<string, AudioOutputDeviceLabel>,
): string | undefined {
const firstId = available.keys().next().value;
if (this.initialIntent === "audio") {
const firstLabel =
firstId !== undefined ? available.get(firstId) : undefined;
if (firstLabel?.type === "speaker") {
for (const [id, label] of available)
if (label.type === "earpiece") {
this.logger.info(
`IOS routing: default to earpiece ${id} instead of speaker for voice call`,
);
return id;
}
}
}
return firstId;
}
public constructor(
private readonly usingNames$: Behavior<boolean>,
private readonly scope: ObservableScope,
private readonly initialIntent: RTCCallIntent | undefined = undefined,
) {
this.selected$.subscribe((device) => {
// Let the hosting application know which output device has been selected.
src/state/CallViewModel/LayoutSwitch.test.ts → src/state/LayoutSwitchViewModel.test.ts
+43 -49
View File
@@ -7,31 +7,31 @@ Please see LICENSE in the repository root for full details.
import { describe, test } from "vitest";
import { createLayoutModeSwitch } from "./LayoutSwitch";
import { testScope, withTestScheduler } from "../../utils/test";
import { createLayoutSwitchViewModel } from "./LayoutSwitchViewModel";
import { testScope, withTestScheduler } from "../utils/test";
function testLayoutSwitch({
windowMode = "n",
hasScreenShares = "n",
userSelection = "",
expectedGridMode,
expectedLayout,
}: {
windowMode?: string;
hasScreenShares?: string;
userSelection?: string;
expectedGridMode: string;
expectedLayout: string;
}): void {
withTestScheduler(({ behavior, schedule, expectObservable }) => {
const { gridMode$, setGridMode } = createLayoutModeSwitch(
const { layout$, setLayout } = createLayoutSwitchViewModel(
testScope(),
behavior(windowMode, { n: "normal", N: "narrow", f: "flat" }),
behavior(hasScreenShares, { y: true, n: false }),
);
schedule(userSelection, {
g: () => setGridMode("grid"),
s: () => setGridMode("spotlight"),
g: () => setLayout("grid"),
s: () => setLayout("spotlight"),
});
expectObservable(gridMode$).toBe(expectedGridMode, {
expectObservable(layout$).toBe(expectedLayout, {
g: "grid",
s: "spotlight",
});
@@ -39,94 +39,88 @@ function testLayoutSwitch({
}
describe("default mode", () => {
test("uses grid layout by default", () =>
test("uses grid layout in normal window", () =>
testLayoutSwitch({
expectedGridMode: "g",
windowMode: " n",
expectedLayout: "g",
}));
test("uses spotlight mode when window mode is flat", () =>
test("uses grid layout in flat window", () =>
testLayoutSwitch({
windowMode: " f",
expectedGridMode: "s",
windowMode: " f",
expectedLayout: "g",
}));
});
test("allows switching modes manually", () =>
testLayoutSwitch({
userSelection: " --sgs",
expectedGridMode: "g-sgs",
userSelection: " --sgs",
expectedLayout: "g-sgs",
}));
test("switches to spotlight mode when there is a remote screen share", () =>
testLayoutSwitch({
hasScreenShares: " n--y",
expectedGridMode: "g--s",
hasScreenShares: "n--y",
expectedLayout: " g--s",
}));
test("can manually switch to grid when there is a screenshare", () =>
testLayoutSwitch({
hasScreenShares: " n-y",
userSelection: " ---g",
expectedGridMode: "g-sg",
hasScreenShares: "n-y",
userSelection: " ---g",
expectedLayout: " g-sg",
}));
test("auto-switches after manually selecting grid", () =>
testLayoutSwitch({
// Two screenshares will happen in sequence. There is a screen share that
// forces spotlight, then the user manually switches back to grid.
hasScreenShares: " n-y-ny",
userSelection: " ---g",
expectedGridMode: "g-sg-s",
hasScreenShares: "n-y-ny",
userSelection: " ---g",
expectedLayout: " g-sg-s",
// If we did want to respect manual selection, the expectation would be: g-sg
}));
test("switches back to grid mode when the remote screen share ends", () =>
testLayoutSwitch({
hasScreenShares: " n--y--n",
expectedGridMode: "g--s--g",
hasScreenShares: "n--y--n",
expectedLayout: " g--s--g",
}));
test("auto-switches to spotlight again after first screen share ends", () =>
testLayoutSwitch({
hasScreenShares: " nyny",
expectedGridMode: "gsgs",
hasScreenShares: "nyny",
expectedLayout: " gsgs",
}));
test("switches manually to grid after screen share while manually in spotlight", () =>
testLayoutSwitch({
// Initially, no one is sharing. Then the user manually switches to spotlight.
// After a screen share starts, the user manually switches to grid.
hasScreenShares: " n-y",
userSelection: " -s-g",
expectedGridMode: "gs-g",
}));
test("auto-switches to spotlight when in flat window mode", () =>
testLayoutSwitch({
// First normal, then narrow, then flat.
windowMode: " nNf",
expectedGridMode: "g-s",
hasScreenShares: "n-y",
userSelection: " -s-g",
expectedLayout: " gs-g",
}));
test("allows switching modes manually when in flat window mode", () =>
testLayoutSwitch({
// Window becomes flat, then user switches to grid and back.
// Window becomes flat, then user switches to spotlight and back.
// Finally the window returns to a normal shape.
windowMode: " nf--n",
userSelection: " --gs",
expectedGridMode: "gsgsg",
windowMode: " nf--n",
userSelection: " --sg",
expectedLayout: "g-sg",
}));
test("stays in spotlight while there are screen shares even when window mode changes", () =>
test("switches to grid when in flat window mode even when there are screen shares", () =>
testLayoutSwitch({
windowMode: " nfn",
hasScreenShares: " y",
expectedGridMode: "s",
windowMode: " nf",
hasScreenShares: "y",
expectedLayout: " sg",
}));
test("ignores end of screen share until window mode returns to normal", () =>
test("ignores screen share until window mode returns to normal", () =>
testLayoutSwitch({
windowMode: " nf-n",
hasScreenShares: " y-n",
expectedGridMode: "s--g",
windowMode: " f-n",
hasScreenShares: "ny-n",
expectedLayout: " g-sg",
}));
src/state/LayoutSwitchViewModel.ts
+99
View File
@@ -0,0 +1,99 @@
/*
Copyright 2025 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import {
combineLatest,
map,
Subject,
startWith,
skipWhile,
switchAll,
} from "rxjs";
import { type WindowMode } from "./CallViewModel/CallViewModel.ts";
import { constant, type Behavior } from "./Behavior.ts";
import { type ObservableScope } from "./ObservableScope.ts";
export type LayoutMode = "spotlight" | "grid";
export interface LayoutSwitchViewModel {
/**
* The layout mode of the call's media tiles.
*/
layout$: Behavior<LayoutMode>;
setLayout: (value: LayoutMode) => void;
}
/**
* Creates a layout mode switch that allows switching between grid and spotlight layouts.
* The actual layout mode might switch automatically to spotlight if there is a
* remote screen share active or if the window mode is flat.
*
* @param scope - The observable scope to manage subscriptions.
* @param windowMode$ - The current window mode.
* @param hasRemoteScreenShares$ - A behavior indicating if there are remote screen shares active.
*/
export function createLayoutSwitchViewModel(
scope: ObservableScope,
windowMode$: Behavior<WindowMode>,
hasRemoteScreenShares$: Behavior<boolean>,
): LayoutSwitchViewModel {
const userSelection$ = new Subject<LayoutMode>();
// Callback to set the layout desired by the user.
// Notice that this is only a preference, the actual layout can be overridden
// if there is a remote screen share active.
const setLayout = (value: LayoutMode): void => userSelection$.next(value);
/**
* The natural layout - the layout that the interface would prefer to be in,
* not accounting for the user's manual selections.
*/
const naturalLayout$ = scope.behavior<LayoutMode>(
combineLatest(
[hasRemoteScreenShares$, windowMode$],
(hasRemoteScreenShares, windowMode) => {
// When the window is flat (as with a phone in landscape orientation),
// grid is preferable as there's usually more than enough horizontal
// space to fit in some grid tiles on the side.
if (windowMode === "flat") return "grid";
// When there are screen shares, spotlight is a better experience. We
// want them to be big and readable.
return hasRemoteScreenShares ? "spotlight" : "grid";
},
),
);
/**
* The layout mode of the call's media tiles.
*/
const layout$ = scope.behavior<LayoutMode>(
// Whenever the user makes a selection, we enter a new mode of behavior:
userSelection$.pipe(
map((selection) => {
if (selection === "grid")
// The user has selected grid. Start by respecting their choice, but
// then follow the natural mode again as soon as it matches.
return naturalLayout$.pipe(
skipWhile((naturalMode) => naturalMode !== selection),
startWith(selection),
);
// The user has selected spotlight. If this matches the natural layout,
// then follow the natural layout going forward.
return selection === naturalLayout$.value
? naturalLayout$
: constant(selection);
}),
// Initially the mode of behavior is to just follow the natural layout.
startWith(naturalLayout$),
// Switch between each mode of behavior.
switchAll(),
),
);
return { layout$, setLayout };
}
src/state/MediaDevices.ts
+5 -1
View File
@@ -376,7 +376,11 @@ export class MediaDevices {
getUrlParams().callIntent,
window.controls,
)
: new IOSControlledAudioOutput(this.usingNames$, this.scope)
: new IOSControlledAudioOutput(
this.usingNames$,
this.scope,
getUrlParams().callIntent,
)
: new AudioOutput(this.usingNames$, this.scope);
public readonly videoInput: MediaDevice<DeviceLabel, SelectedDevice> =
src/state/OneOnOneLandscapeLayout.ts → src/state/OneOnOneDesktopLayout.ts
+6 -6
View File
@@ -10,19 +10,19 @@ import { type BehaviorSubject } from "rxjs";
import {
type Alignment,
type OneOnOneLandscapeLayout,
type OneOnOneLandscapeLayoutMedia,
type OneOnOneDesktopLayout,
type OneOnOneDesktopLayoutMedia,
} from "./layout-types";
import { type TileStore } from "./TileStore";
/**
* Produces a one-on-one landscape layout with the given media.
* Produces a one-on-one desktop layout with the given media.
*/
export function oneOnOneLandscapeLayout(
media: OneOnOneLandscapeLayoutMedia,
export function oneOnOneDesktopLayout(
media: OneOnOneDesktopLayoutMedia,
pipAlignment$: BehaviorSubject<Alignment>,
prevTiles: TileStore,
): [OneOnOneLandscapeLayout, TileStore] {
): [OneOnOneDesktopLayout, TileStore] {
const update = prevTiles.from(2);
update.registerGridTile(media.pip);
update.registerGridTile(media.spotlight);
src/state/OneOnOnePortraitLayout.ts → src/state/OneOnOneMobileLayout.ts
+7 -7
View File
@@ -10,23 +10,23 @@ import { type BehaviorSubject } from "rxjs";
import {
type Alignment,
type OneOnOnePortraitLayout,
type OneOnOnePortraitLayoutMedia,
type OneOnOneMobileLayout,
type OneOnOneMobileLayoutMedia,
} from "./layout-types";
import { type TileStore } from "./TileStore";
import { type Behavior } from "./Behavior";
/**
* Produces a one-on-one portrait layout with the given media.
* Produces a one-on-one mobile layout with the given media.
*/
export function oneOnOnePortraitLayout(
media: OneOnOnePortraitLayoutMedia,
export function oneOnOneMobileLayout(
media: OneOnOneMobileLayoutMedia,
pipSize$: Behavior<"sm" | "lg">,
pipAlignment$: BehaviorSubject<Alignment>,
prevTiles: TileStore,
): [OneOnOnePortraitLayout, TileStore] {
): [OneOnOneMobileLayout, TileStore] {
const update = prevTiles.from(media.pip === undefined ? 0 : 1);
update.registerSpotlight([media.spotlight], true);
update.registerSpotlight([media.spotlight], true, "transparent");
if (media.pip !== undefined) update.registerGridTile(media.pip);
const tiles = update.build();
src/state/PipLayout.ts
+1
View File
@@ -21,6 +21,7 @@ export function pipLayout(
update.registerSpotlight(
media.spotlight,
platform === "desktop" ? false : true,
"transparent",
);
const tiles = update.build();
return [
src/state/ServiceInterruptionsViewModel.ts
+27
View File
@@ -0,0 +1,27 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
export type Service =
| { name: "encryption-keys" }
| { name: "reactions" }
| { name: "screen-sharing" }
| { name: "local-media" }
| {
name: "remote-media";
/**
* The number of users affected, for the purpose of pluralising the
* strings.
*/
count: number;
};
export interface ServiceInterruptionsViewModel {
/**
* A non-empty array of services which are temporarily unavailable.
*/
unavailable: Service[];
}
src/state/SessionBehaviors.test.ts
+41
View File
@@ -0,0 +1,41 @@
/*
Copyright 2025 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { describe, expect, it, vi } from "vitest";
import { MatrixRTCSessionEvent } from "matrix-js-sdk/lib/matrixrtc";
import { EventEmitter } from "events";
import { createKeyRotationSuppressed$ } from "./SessionBehaviors";
import { testScope } from "../utils/test";
describe("SessionBehaviors", () => {
describe("createKeyRotationSuppressed$", () => {
it("emits initial value from isKeyRotationSuppressed and updates when KeyRotationSuppressedChanged event is emitted", () => {
const scope = testScope();
const emitter = new EventEmitter();
const mockSession = Object.assign(emitter, {
on: vi.fn(),
off: vi.fn(),
isKeyRotationSuppressed: false,
});
const keyRotationSuppressed$ = createKeyRotationSuppressed$(
scope,
mockSession as any,
);
expect(keyRotationSuppressed$.value).toBe(false);
emitter.emit(MatrixRTCSessionEvent.KeyRotationSuppressedChanged, true);
expect(keyRotationSuppressed$.value).toBe(true);
emitter.emit(MatrixRTCSessionEvent.KeyRotationSuppressedChanged, false);
expect(keyRotationSuppressed$.value).toBe(false);
});
});
});
src/state/SessionBehaviors.ts
+19
View File
@@ -88,3 +88,22 @@ export const createMemberships$ = (
new Epoch(matrixRTCSession.memberships),
);
};
/**
* Whether the session has grown large enough that MatrixRTC has stopped rotating the media
* encryption key. While this is true the key in use is still shared with new joiners, but no new
* key is generated when someone joins or leaves.
*/
export const createKeyRotationSuppressed$ = (
scope: ObservableScope,
matrixRTCSession: MatrixRTCSession,
): Behavior<boolean> => {
return scope.behavior(
fromEvent(
matrixRTCSession,
MatrixRTCSessionEvent.KeyRotationSuppressedChanged,
(suppressed: boolean) => suppressed,
),
matrixRTCSession.isKeyRotationSuppressed,
);
};
src/state/SpotlightExpandedLayout.ts
+1 -1
View File
@@ -23,7 +23,7 @@ export function spotlightExpandedLayout(
prevTiles: TileStore,
): [SpotlightExpandedLayout, TileStore] {
const update = prevTiles.from(1);
update.registerSpotlight(media.spotlight, true);
update.registerSpotlight(media.spotlight, true, "transparent");
if (media.pip !== undefined) update.registerPipTile(media.pip);
const tiles = update.build();
src/state/TileStore.ts
+31 -7
View File
@@ -15,6 +15,8 @@ import { type MediaViewModel } from "./media/MediaViewModel";
import { type UserMediaViewModel } from "./media/UserMediaViewModel";
import { type RingingMediaViewModel } from "./media/RingingMediaViewModel";
type SpotlightBackground = "solid" | "transparent";
function debugEntries(entries: GridTileData[]): string[] {
return entries.map((e) => e.media.displayName$.value);
}
@@ -39,12 +41,29 @@ class SpotlightTileData {
this.maximised$.next(value);
}
private readonly background$: BehaviorSubject<SpotlightBackground>;
public get background(): SpotlightBackground {
return this.background$.value;
}
public set background(value: SpotlightBackground) {
this.background$.next(value);
}
public readonly vm: SpotlightTileViewModel;
public constructor(media: MediaViewModel[], maximised: boolean) {
public constructor(
media: MediaViewModel[],
maximised: boolean,
background: SpotlightBackground,
) {
this.media$ = new BehaviorSubject(media);
this.maximised$ = new BehaviorSubject(maximised);
this.vm = new SpotlightTileViewModel(this.media$, this.maximised$);
this.background$ = new BehaviorSubject(background);
this.vm = new SpotlightTileViewModel(
this.media$,
this.maximised$,
this.background$,
);
}
}
@@ -131,9 +150,9 @@ export class TileStoreBuilder {
private numGridEntries = 0;
// A sparse array of grid entries which should be kept in the same spots as
// which they appeared in the previous grid
private readonly stationaryGridEntries: GridTileData[] = new Array(
this.prevGrid.length,
);
private readonly stationaryGridEntries: GridTileData[] = Array.from({
length: this.prevGrid.length,
});
// Grid entries which should now enter the visible section of the grid
private readonly visibleGridEntries: GridTileData[] = [];
// Grid entries which should now enter the invisible section of the grid
@@ -157,7 +176,11 @@ export class TileStoreBuilder {
* Sets the contents of the spotlight tile. If this is never called, there
* will be no spotlight tile.
*/
public registerSpotlight(media: MediaViewModel[], maximised: boolean): void {
public registerSpotlight(
media: MediaViewModel[],
maximised: boolean,
background: SpotlightBackground = "solid",
): void {
if (DEBUG_ENABLED)
logger.debug(
`[TileStore, ${this.generation}] register spotlight: ${media.map((m) => m.displayName$.value)}`,
@@ -169,11 +192,12 @@ export class TileStoreBuilder {
// Reuse the previous spotlight tile if it exists
if (this.prevSpotlight === null) {
this.spotlight = new SpotlightTileData(media, maximised);
this.spotlight = new SpotlightTileData(media, maximised, background);
} else {
this.spotlight = this.prevSpotlight;
this.spotlight.media = media;
this.spotlight.maximised = maximised;
this.spotlight.background = background;
}
}
src/state/TileViewModel.ts
+9
View File
@@ -5,6 +5,8 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { BehaviorSubject } from "rxjs";
import { type Behavior } from "./Behavior";
import { type MediaViewModel } from "./media/MediaViewModel";
import { type RingingMediaViewModel } from "./media/RingingMediaViewModel";
@@ -17,18 +19,25 @@ function createId(): string {
export class GridTileViewModel {
public readonly id = createId();
private readonly _showOutline$ = new BehaviorSubject(false);
public readonly showOutline$: Behavior<boolean> = this._showOutline$;
public constructor(
public readonly media$: Behavior<
UserMediaViewModel | RingingMediaViewModel
>,
) {}
public setShowOutline(value: boolean): void {
this._showOutline$.next(value);
}
}
export class SpotlightTileViewModel {
public constructor(
public readonly media$: Behavior<MediaViewModel[]>,
public readonly maximised$: Behavior<boolean>,
public readonly background$: Behavior<"solid" | "transparent">,
) {}
}
src/state/layout-types.test.ts
+74
View File
@@ -0,0 +1,74 @@
/*
Copyright 2026 Element Creations Ltd.
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { test, expect } from "vitest";
import {
type Alignment,
layoutShallowEquals,
type Layout,
} from "./layout-types";
import {
type SpotlightTileViewModel,
type GridTileViewModel,
} from "./TileViewModel";
import { BehaviorSubject } from "rxjs";
const spotlightTile = {} as unknown as SpotlightTileViewModel;
const gridTile = {} as unknown as GridTileViewModel;
const pipAlignment$ = new BehaviorSubject<Alignment>({
inline: "end",
block: "end",
});
const spotlightExpanded: Layout = {
type: "spotlight-expanded",
spotlight: spotlightTile,
pipAlignment$,
};
const spotlightPortrait: Layout = {
type: "spotlight-portrait",
spotlight: spotlightTile,
grid: [gridTile],
setVisibleTiles: () => {},
};
test("layoutShallowEquals considers a layout to be equal to its shallow clone", () =>
expect(layoutShallowEquals(spotlightExpanded, { ...spotlightExpanded })).toBe(
true,
));
test("layoutShallowEquals detects a missing key", () => {
expect(
layoutShallowEquals(spotlightExpanded, {
...spotlightExpanded,
pip: gridTile,
}),
).toBe(false);
expect(
layoutShallowEquals(
{ ...spotlightExpanded, pip: gridTile },
spotlightExpanded,
),
).toBe(false);
});
test("layoutShallowEquals considers grid arrays with equal contents to be equal", () =>
expect(
layoutShallowEquals(spotlightPortrait, {
...spotlightPortrait,
grid: [...spotlightPortrait.grid],
}),
).toBe(true));
test("layoutShallowEquals detects grid arrays with different contents", () =>
expect(
layoutShallowEquals(spotlightPortrait, {
...spotlightPortrait,
grid: [...spotlightPortrait.grid, gridTile],
}),
).toBe(false));
src/state/layout-types.ts
+40 -12
View File
@@ -16,6 +16,7 @@ import {
type SpotlightTileViewModel,
} from "./TileViewModel.ts";
import { type Behavior } from "./Behavior.ts";
import { shallowEquals as arrayShallowEquals } from "../utils/array.ts";
export interface GridLayoutMedia {
type: "grid";
@@ -45,15 +46,15 @@ export interface SpotlightExpandedLayoutMedia {
pip?: UserMediaViewModel;
}
export interface OneOnOneLandscapeLayoutMedia {
type: "one-on-one-landscape";
export interface OneOnOneDesktopLayoutMedia {
type: "one-on-one-desktop";
edgeToEdge: false;
spotlight: UserMediaViewModel;
pip: LocalUserMediaViewModel | RingingMediaViewModel;
}
export interface OneOnOnePortraitLayoutMedia {
type: "one-on-one-portrait";
export interface OneOnOneMobileLayoutMedia {
type: "one-on-one-mobile";
edgeToEdge: true;
spotlight: UserMediaViewModel | RingingMediaViewModel;
pip?: LocalUserMediaViewModel;
@@ -70,8 +71,8 @@ export type LayoutMedia =
| SpotlightLandscapeLayoutMedia
| SpotlightPortraitLayoutMedia
| SpotlightExpandedLayoutMedia
| OneOnOneLandscapeLayoutMedia
| OneOnOnePortraitLayoutMedia
| OneOnOneDesktopLayoutMedia
| OneOnOneMobileLayoutMedia
| PipLayoutMedia;
export interface Alignment {
@@ -108,15 +109,15 @@ export interface SpotlightExpandedLayout {
pipAlignment$: BehaviorSubject<Alignment>;
}
export interface OneOnOneLandscapeLayout {
type: "one-on-one-landscape";
export interface OneOnOneDesktopLayout {
type: "one-on-one-desktop";
spotlight: GridTileViewModel;
pip: GridTileViewModel;
pipAlignment$: BehaviorSubject<Alignment>;
}
export interface OneOnOnePortraitLayout {
type: "one-on-one-portrait";
export interface OneOnOneMobileLayout {
type: "one-on-one-mobile";
spotlight: SpotlightTileViewModel;
pip?: GridTileViewModel;
pipSize$: Behavior<"sm" | "lg">;
@@ -137,6 +138,33 @@ export type Layout =
| SpotlightLandscapeLayout
| SpotlightPortraitLayout
| SpotlightExpandedLayout
| OneOnOneLandscapeLayout
| OneOnOnePortraitLayout
| OneOnOneDesktopLayout
| OneOnOneMobileLayout
| PipLayout;
/**
* Tests whether the top-level properties and array elements of layout `a` are
* equal to those of layout `b`. Useful for deduping redundant layout updates.
*/
export function layoutShallowEquals(a: Layout, b: Layout): boolean {
// If a and b have the same number of keys and every key in a is also in b,
// then they have the same keys.
const aKeys = Object.keys(a);
const bKeys = Object.keys(b);
if (aKeys.length !== bKeys.length) return false;
for (const key of aKeys) {
if (!(key in b)) return false;
// Now check that they have the same values.
const aValue = (a as any)[key];
const bValue = (b as any)[key];
if (Array.isArray(aValue) && Array.isArray(bValue)) {
// Special case for arrays so we can detect when the grid tiles arrays are
// essentially the same.
if (!arrayShallowEquals(aValue, bValue)) return false;
} else if (aValue !== bValue) return false;
}
return true;
}
src/state/media/MemberMediaViewModel.ts
+3 -2
View File
@@ -92,6 +92,7 @@ export function createMemberMedia(
}: MemberMediaInputs,
): BaseMemberMediaViewModel {
const trackBehavior$ = (
scope: ObservableScope,
source: Track.Source,
): Behavior<TrackReference | undefined> =>
scope.behavior(
@@ -102,8 +103,8 @@ export function createMemberMedia(
),
);
const audio$ = trackBehavior$(audioSource);
const video$ = trackBehavior$(videoSource);
const audio$ = trackBehavior$(scope, audioSource);
const video$ = trackBehavior$(scope, videoSource);
return {
...createBaseMedia(inputs),
src/state/media/RingingMediaViewModel.ts
+6 -14
View File
@@ -5,8 +5,9 @@ SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
*/
import { type RTCCallIntent } from "matrix-js-sdk/lib/matrixrtc";
import { type Behavior } from "../Behavior";
import { type MuteStates } from "../MuteStates";
import {
type BaseMediaInputs,
type BaseMediaViewModel,
@@ -20,32 +21,23 @@ import {
export interface RingingMediaViewModel extends BaseMediaViewModel {
type: "ringing";
pickupState$: Behavior<"ringing" | "timeout" | "decline">;
/**
* Whether this media would be expected to have video, were it not simply a
* placeholder.
*/
videoEnabled$: Behavior<boolean>;
intent: RTCCallIntent;
}
export interface RingingMediaInputs extends BaseMediaInputs {
pickupState$: Behavior<"ringing" | "timeout" | "decline">;
/**
* The local user's own mute states.
*/
muteStates: MuteStates;
intent: RTCCallIntent;
}
export function createRingingMedia({
pickupState$,
muteStates,
intent,
...inputs
}: RingingMediaInputs): RingingMediaViewModel {
return {
...createBaseMedia(inputs),
type: "ringing",
pickupState$,
// If our own video is enabled, then this is a video call and we would
// expect remote media to have video as well
videoEnabled$: muteStates.video.enabled$,
intent,
};
}
src/state/media/UserMediaViewModel.ts
+10 -20
View File
@@ -33,7 +33,6 @@ import { type RemoteUserMediaViewModel } from "./RemoteUserMediaViewModel";
import { type ObservableScope } from "../ObservableScope";
import { showConnectionStats } from "../../settings/settings";
import { observeRtpStreamStats$ } from "./observeRtpStreamStats";
import { videoFit$, videoSizeFromParticipant$ } from "../../utils/videoFit.ts";
/**
* A participant's user media (i.e. their microphone and camera feed).
@@ -47,7 +46,7 @@ export interface BaseUserMediaViewModel extends BaseMemberMediaViewModel {
speaking$: Behavior<boolean>;
audioEnabled$: Behavior<boolean>;
videoEnabled$: Behavior<boolean>;
videoFit$: Behavior<"cover" | "contain">;
videoOrientation$: Behavior<"landscape" | "portrait">;
toggleCropVideo: () => void;
/**
* The expected identity of the LiveKit participant. Exposed for debugging.
@@ -62,12 +61,9 @@ export interface BaseUserMediaViewModel extends BaseMemberMediaViewModel {
RTCInboundRtpStreamStats | RTCOutboundRtpStreamStats | undefined
>;
/**
* Set the target dimensions of the HTML element (final dimension after anim).
* This can be used to determine the best video fit (fit to frame / keep ratio).
* @param targetWidth - The target width of the HTML element displaying the video.
* @param targetHeight - The target height of the HTML element displaying the video.
* Set the aspect ratio of the video track to determine the orientation.
*/
setTargetDimensions: (targetWidth: number, targetHeight: number) => void;
setVideoAspectRatio: (ratio: number) => void;
}
export interface BaseUserMediaInputs extends Omit<
@@ -97,12 +93,7 @@ export function createBaseUserMedia(
),
);
const toggleCropVideo$ = new Subject<void>();
// The target size of the video element, used to determine the best video fit.
// The target size is the final size of the HTML element after any animations have completed.
const targetSize$ = new BehaviorSubject<
{ width: number; height: number } | undefined
>(undefined);
const videoAspectRatio$ = new BehaviorSubject(NaN);
return {
...createMemberMedia(scope, {
@@ -129,10 +120,11 @@ export function createBaseUserMedia(
videoEnabled$: scope.behavior(
media$.pipe(map((m) => m?.cameraTrack?.isMuted === false)),
),
videoFit$: videoFit$(
scope,
videoSizeFromParticipant$(participant$),
targetSize$,
videoOrientation$: scope.behavior(
videoAspectRatio$.pipe(
map((aspect) => (aspect > 1 ? "landscape" : "portrait")),
),
"portrait",
),
toggleCropVideo: () => toggleCropVideo$.next(),
rtcBackendIdentity,
@@ -157,8 +149,6 @@ export function createBaseUserMedia(
return observeRtpStreamStats$(p, Track.Source.Camera, statsType);
}),
),
setTargetDimensions: (targetWidth: number, targetHeight: number): void => {
targetSize$.next({ width: targetWidth, height: targetHeight });
},
setVideoAspectRatio: (ratio) => videoAspectRatio$.next(ratio),
};
}
src/state/media/observeRtpStreamStats.ts
+6 -10
View File
@@ -19,10 +19,15 @@ import {
startWith,
switchMap,
map,
share,
} from "rxjs";
import { observeTrackReference$ } from "../observeTrackReference";
// Use a shared timer for all the stats observers so that we don't clog up the
// event loop with hundreds of timers in large calls
const refreshStats$ = interval(1000).pipe(share());
export function observeRtpStreamStats$(
participant: Participant,
source: Track.Source,
@@ -32,7 +37,7 @@ export function observeRtpStreamStats$(
> {
return combineLatest([
observeTrackReference$(participant, source),
interval(1000).pipe(startWith(0)),
refreshStats$.pipe(startWith(0)),
]).pipe(
switchMap(async ([trackReference]) => {
const track = trackReference?.publication?.track;
@@ -67,12 +72,3 @@ export function observeInboundRtpStreamStats$(
map((x) => x as RTCInboundRtpStreamStats | undefined),
);
}
export function observeOutboundRtpStreamStats$(
participant: Participant,
source: Track.Source,
): Observable<RTCOutboundRtpStreamStats | undefined> {
return observeRtpStreamStats$(participant, source, "outbound-rtp").pipe(
map((x) => x as RTCOutboundRtpStreamStats | undefined),
);
}