Compare commits
9
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 jared
|
0aa543cc7d | ||
|
jared
|
cc57292bf4 | ||
|
jared
|
22f9083768 | ||
|
jared
|
d7d75cbba5 | ||
|
jared
|
4be3e92eb6 | ||
  Lotus CIandClaude Opus 5.5
|
a1dd0e254c | ||
|
Lotus CIandClaude Opus 5.5
|
167ffc2440 | ||
|
Lotus CIandClaude Opus 5.5
|
888ed1a66c | ||
|
Lotus CIandClaude Opus 5.5
|
73bf2133dc |
@@ -0,0 +1,80 @@
|
||||
# Lotus fork of Element Call
|
||||
|
||||
This fork is embedded by [Lotus Chat](https://code.lotusguild.org/LotusGuild/cinny)
|
||||
(cinny) as the in-app call frame. Everything it adds is opt-in: without the
|
||||
URL flags and widget actions below it behaves like upstream Element Call.
|
||||
|
||||
The source of truth is `src/lotus/` (the `io.lotus.*` actions are declared in
|
||||
`src/lotus/lotusActions.ts`). Keep this file in step with it: when the host and
|
||||
the fork drift, features fail silently (element-call#43, cinny#260).
|
||||
|
||||
## URL flags
|
||||
|
||||
The host sets these on the widget URL (cinny: `CallEmbed.getWidget`).
|
||||
|
||||
| Flag | Effect |
|
||||
| ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| `lotusCallState=true` | Stream call state to the host: `io.lotus.call_state` (participants, speaking, mute) and `io.lotus.mic_level`. |
|
||||
| `lotusTheme=true` | Apply the host theme; the frame's root `color-scheme` follows it. |
|
||||
| `lotusTransparent=true` | Transparent canvas so the host's wallpaper shows through. Only meaningful with `lotusTheme` (the fork applies it anyway and warns). |
|
||||
| `lotusHostControls=true` | The host draws the call bar: hide Element Call's footer and enable the host-control actions. |
|
||||
| `lotusFrameScreenshare=true` | With `lotusHostControls`: show the frame's own screenshare button, for engines without Capability Delegation (Firefox, Safari) where the host can't start a share. |
|
||||
| `lotusAudioInject=true` | Accept `io.lotus.inject_audio` (soundboard). |
|
||||
| `lotusDenoiseSource=true` | Run ML noise suppression in-source as a LiveKit audio TrackProcessor. |
|
||||
| `lotusModel=rnnoise\|speex\|dtln\|deepfilternet` | Denoise model (default `rnnoise`; falls back to `rnnoise` once if the chosen model fails). |
|
||||
| `lotusGate=true`, `lotusGateThreshold=<dB>` | Noise gate after denoise (default −45 dB). |
|
||||
| `lotusDenoiseFloor=<0..1>` | Denoise mix floor. |
|
||||
| `lotusDenoiseBase=<url>` | Where the denoise worklets / wasm / models are served (same-origin by default: `./denoise/`). |
|
||||
| `sendNotificationType=none` | Upstream param, extended: `none` sends no call-start notification at all (cinny#295; voice channels shouldn't @room-ping). |
|
||||
|
||||
## Widget actions (`io.lotus.*`)
|
||||
|
||||
"host → call" actions must be allow-listed in `LOTUS_TO_WIDGET_ACTIONS`.
|
||||
The table is generated from `src/lotus/lotusActions.ts`; the last column was
|
||||
checked against cinny's `lotus` branch.
|
||||
|
||||
| Action | Direction | Payload / meaning | Handled by cinny (`lotus`) |
|
||||
| -------------------------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------- |
|
||||
| `io.lotus.call_state` | call → host | in-call per-participant speaking / mute state. NOTE: matrix-widget-api `transport.send` is request/response — the host MUST register a handler that replies/acks each one (cinny's `listenAction` does, replying `{}`). If the host has no handler, ClientWidgetApi immediately error-replies "unsupported from-widget action", so the data is silently dropped and each throttled send rejects (caught) — functional miss + log churn, not a hang. | yes |
|
||||
| `io.lotus.focus_participant` | host → call | pin/spotlight (or clear, with userId=null) a participant. | yes |
|
||||
| `io.lotus.inject_audio` | host → call | play an audio clip into the call as a separate published track. | yes |
|
||||
| `io.lotus.set_quality` | host → call | set audio/screenshare encoding quality (bitrate/framerate). | yes |
|
||||
| `io.lotus.decorations` | host → call | per-user avatar-decoration image URLs for in-call tiles. | yes |
|
||||
| `io.lotus.set_deafen` | host → call | deafen remote audio (and optionally mute screenshare audio). | yes |
|
||||
| `io.lotus.request_state` | call → host | sent on (re)registration of a lotus toWidget handler (e.g. decorations) so the host can re-push state that would otherwise only be sent on change (decorations roster, the current focus pin) after an EC-side reconnect/remount. cinny's `resendForkState()` should respond to this the same way it responds to a fresh join. | yes |
|
||||
| `io.lotus.denoise_state` | call → host | real state of the in-source denoise engine — `{ active: boolean, model: string, error?: string }` — sent once the processor attaches (or after the rnnoise fallback also fails), so the host toggle can reflect reality rather than the requested state. | **no** |
|
||||
| `io.lotus.call_summary` | call → host | one-shot end-of-call readout for the local participant — `{ durationMs, reconnects, poorMs, verdict }` (#143) — sent on SFU disconnect or in-call teardown, whichever comes first. | yes |
|
||||
| `io.lotus.set_audio_output` | host → call | select the audio output device `{ deviceId }` (#119). | yes |
|
||||
| `io.lotus.audio_output_state` | call → host | the currently selected output `{ deviceId }` (#119). | **no** |
|
||||
| `io.lotus.screenshare_notice` | call → host | local screenshare reminder `{ kind: "ended" \| "no-frames" \| "alone" }` — window closed, no frames for 15 s, or 30 min of sharing with nobody else in the call (#39). Each fires at most once per share. | yes |
|
||||
| `io.lotus.set_layout` | host → call | switch the call layout `{ layout: "grid" \| "spotlight" }` (cinny #43 — replaces the host clicking EC's hidden layout radio). | yes |
|
||||
| `io.lotus.open_settings` | host → call | open EC's settings modal, or close it with `{ open: false }`; omit `open` to toggle (cinny #43). | yes |
|
||||
| `io.lotus.toggle_reactions` | host → call | toggle the reactions / raise-hand menu (cinny #43). | yes |
|
||||
| `io.lotus.set_screenshare` | host → call | start/stop sharing `{ on?: boolean }` (omit to toggle; cinny #43). Starting calls getDisplayMedia, which needs the user's click: the host sends this with Capability Delegation (`delegate: "display-capture"`), and without it the browser rejects the share. | yes |
|
||||
| `io.lotus.set_hotkeys` | host → call | `{ codes: string[] }` — KeyboardEvent.code values (PTT, deafen) to report while focus is inside this frame (cinny #43). `[]` stops. | yes |
|
||||
| `io.lotus.set_frame_screenshare` | host → call | `{ visible: boolean }` — whether the in-frame screenshare button (`lotusFrameScreenshare`) may show, e.g. false while the room's call policy forbids screensharing (cinny #43). Defaults to visible. | yes |
|
||||
| `io.lotus.prompt_screenshare` | host → call | `{}` — show the "Share your screen?" prompt inside this frame (cinny #43). For hosts that can't delegate the user's click: the prompt's Share button is clicked here, so getDisplayMedia gets its activation. | yes |
|
||||
| `io.lotus.hotkey` | call → host | a watched key went down/up here, or this window's focus changed; see `LotusHotkeyReport` (cinny #43). | yes |
|
||||
| `io.lotus.controls_state` | call → host | `{ screensharing: boolean, layout: "grid" \| "spotlight" \| null, screenshareAction: true, hotkeys: true, frameScreenshare: false, screensharePrompt: boolean, screensharePromptOpen: boolean }` whenever either changes (cinny #43), so the host stops reading EC's DOM for them. Its arrival also tells the host this fork supports the actions above. | yes |
|
||||
| `io.lotus.mic_level` | call → host | local mic level `{ bars: 0 \| 1 \| 2 \| 3 }` (cinny #146), sent only when it changes (≤ 10 Hz); 0 while muted or with no mic. | yes |
|
||||
|
||||
The two unhandled actions are covered by pending cinny PRs (#413 for
|
||||
`denoise_state`; the #310 change for `audio_output_state`).
|
||||
|
||||
## Fork-only behaviour (no flag needed)
|
||||
|
||||
Per-person volume memory, screenshare notices (window closed / no frames /
|
||||
sharing alone), mono mic capture, an SFU refusal reason instead of "Something
|
||||
went wrong", "Try again" for failed connects, and support for running on its
|
||||
own origin (cinny#43).
|
||||
|
||||
## Releasing
|
||||
|
||||
1. Merge the change into `lotus`.
|
||||
2. Bump `version` in `embedded/web/package.json` (`0.25.0-lotus.N`), commit
|
||||
`chore(lotus): 0.25.0-lotus.N` on `lotus`, and push a tag `v0.25.0-lotus.N`.
|
||||
3. CI builds the embedded bundle and, for `v*` tags, publishes
|
||||
`@lotusguild/element-call-embedded` to the Gitea npm registry
|
||||
(`https://code.lotusguild.org/api/packages/LotusGuild/npm/`).
|
||||
4. In cinny, pin the exact version in `package.json` (`npm install --save-exact
|
||||
@lotusguild/element-call-embedded@0.25.0-lotus.N`) and open a PR.
|
||||
@@ -11,6 +11,7 @@ import { of } from "rxjs";
|
||||
|
||||
import { type CallViewModel } from "../state/CallViewModel/CallViewModel";
|
||||
import {
|
||||
canPublishSoundboard,
|
||||
MAX_INJECT_BYTES,
|
||||
parseInjectSource,
|
||||
startLotusAudioInject,
|
||||
@@ -258,3 +259,17 @@ test("parseInjectSource rejects oversized or non-buffer audio", () => {
|
||||
expect(parseInjectSource({ audio: "not bytes" })).toBeNull();
|
||||
expect(parseInjectSource({ audio: new Uint8Array(4) })).toBeNull();
|
||||
});
|
||||
|
||||
test("canPublishSoundboard: refused by a microphone-only grant (EC #47)", () => {
|
||||
expect(canPublishSoundboard(undefined)).toBe(true);
|
||||
expect(
|
||||
canPublishSoundboard({ canPublish: true, canPublishSources: [] }),
|
||||
).toBe(true);
|
||||
expect(
|
||||
canPublishSoundboard({ canPublish: true, canPublishSources: [2] }),
|
||||
).toBe(false);
|
||||
expect(
|
||||
canPublishSoundboard({ canPublish: true, canPublishSources: [2, 0] }),
|
||||
).toBe(true);
|
||||
expect(canPublishSoundboard({ canPublish: false })).toBe(false);
|
||||
});
|
||||
|
||||
@@ -60,6 +60,22 @@ function acquireSharedAudio(): {
|
||||
* No effect unless the host sends the action. Returns a teardown function that
|
||||
* also aborts any clip still playing.
|
||||
*/
|
||||
/**
|
||||
* Pure: whether these LiveKit permissions allow publishing the soundboard's
|
||||
* Unknown-source track (EC #47). No permissions or an empty source list
|
||||
* means unrestricted; otherwise UNKNOWN (TrackSource 0) must be allowed.
|
||||
*/
|
||||
export function canPublishSoundboard(
|
||||
perm:
|
||||
| { canPublish?: boolean; canPublishSources?: readonly number[] }
|
||||
| undefined,
|
||||
): boolean {
|
||||
if (!perm) return true;
|
||||
if (perm.canPublish === false) return false;
|
||||
const sources = perm.canPublishSources ?? [];
|
||||
return sources.length === 0 || sources.includes(0);
|
||||
}
|
||||
|
||||
export function startLotusAudioInject(vm: CallViewModel): () => void {
|
||||
const w = widget;
|
||||
if (!w) return () => undefined;
|
||||
@@ -116,6 +132,19 @@ export function startLotusAudioInject(vm: CallViewModel): () => void {
|
||||
return;
|
||||
}
|
||||
|
||||
// [EC #47] Clips publish as an Unknown-source track. Audio-only rooms get
|
||||
// a token limited to the microphone source, where LiveKit refuses that
|
||||
// publish *after* we replied; say so up front instead of failing silently.
|
||||
if (
|
||||
rooms.some((r) => !canPublishSoundboard(r.localParticipant.permissions))
|
||||
) {
|
||||
w.api.transport.reply(ev.detail, {
|
||||
played: false,
|
||||
reason: "not_permitted",
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
w.api.transport.reply(ev.detail, {});
|
||||
void playInjectedClip(clip, volume, rooms, activeClips).catch((e) =>
|
||||
logger.warn("[lotus] inject_audio failed", e),
|
||||
|
||||
@@ -7,7 +7,7 @@ Please see LICENSE in the repository root for full details.
|
||||
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import { MicLevelQuantizer } from "./lotusMicLevel";
|
||||
import { MicLevelQuantizer, sameRooms } from "./lotusMicLevel";
|
||||
|
||||
describe("MicLevelQuantizer", () => {
|
||||
it("maps RMS to 0–3 bars", () => {
|
||||
@@ -35,3 +35,14 @@ describe("MicLevelQuantizer", () => {
|
||||
expect(q.push(0)).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe("sameRooms (EC #45)", () => {
|
||||
it("compares rooms by identity and order", () => {
|
||||
const a = {};
|
||||
const b = {};
|
||||
expect(sameRooms([a, b], [a, b])).toBe(true);
|
||||
expect(sameRooms([a], [a, b])).toBe(false);
|
||||
expect(sameRooms([a, b], [b, a])).toBe(false);
|
||||
expect(sameRooms([{}], [{}])).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -46,6 +46,10 @@ const micPublication = (
|
||||
: null;
|
||||
};
|
||||
|
||||
/** Pure: the same LiveKit rooms, in the same order (by identity). */
|
||||
export const sameRooms = <T>(a: readonly T[], b: readonly T[]): boolean =>
|
||||
a.length === b.length && a.every((r, i) => r === b[i]);
|
||||
|
||||
const samplers = new WeakMap<
|
||||
CallViewModel,
|
||||
Observable<LocalMicSample | null>
|
||||
@@ -62,10 +66,15 @@ export function observeLocalMicSample$(
|
||||
const cached = samplers.get(vm);
|
||||
if (cached) return cached;
|
||||
const sampler$ = vm.allConnections$.pipe(
|
||||
map((data) => data.getConnections().map((c) => c.livekitRoom)),
|
||||
// [EC #45] allConnections$ re-emits far more often than the set of
|
||||
// LiveKit rooms changes; restarting the tap each time closed the
|
||||
// AudioContext ~1×/s and emitted null, which reset the muted-speech
|
||||
// gate so the "You're muted" nudge could never fire.
|
||||
distinctUntilChanged(sameRooms),
|
||||
switchMap(
|
||||
(data) =>
|
||||
(rooms) =>
|
||||
new Observable<LocalMicSample | null>((subscriber) => {
|
||||
const rooms = data.getConnections().map((c) => c.livekitRoom);
|
||||
let ctx: AudioContext | null = null;
|
||||
let clone: MediaStreamTrack | null = null;
|
||||
let timer: ReturnType<typeof setInterval> | undefined;
|
||||
|
||||
@@ -868,16 +868,19 @@ export function enterRTCSession(
|
||||
};
|
||||
}
|
||||
|
||||
// Calculates `maximumNetworkErrorRetryCount`. The connection is failed if EITHER:
|
||||
// - The /sync loop is unresponsive for > `gracePeriod` ms, or
|
||||
// - A delayed leave event is emitted (after `leaveDelay` ms period).
|
||||
// Note: Use leaveDelay >> gracePeriod for delegated leave events.
|
||||
// Calculates `maximumNetworkErrorRetryCount`: enough retries to outlast the
|
||||
// server's delayed-leave window (see [lotus #46] below).
|
||||
const gracePeriod = Config.get().sync_disconnect_grace_period_ms;
|
||||
const leaveDelay = matrixRtcSessionConfig?.delayed_leave_event_delay_ms;
|
||||
const retryInterval = matrixRtcSessionConfig?.network_error_retry_ms;
|
||||
|
||||
// Math.min is used to account for the respective worst case: /sync not available or leave event emitted.
|
||||
const maxWaitTime = Math.min(gracePeriod, leaveDelay);
|
||||
// [lotus #46] Keep retrying for as long as the homeserver itself would keep
|
||||
// our membership: until its delayed leave fires (`leaveDelay`). Capping at
|
||||
// the 10 s sync grace period made a ~20–35 s homeserver stall end everyone's
|
||||
// call ("Connection lost") even with a 60 s server-side leave delay. The
|
||||
// embedded call has no /sync loop of its own, so the grace period doesn't
|
||||
// describe anything it can observe.
|
||||
const maxWaitTime = Math.max(gracePeriod, leaveDelay);
|
||||
const maximumNetworkErrorRetryCount =
|
||||
Math.ceil(maxWaitTime / retryInterval) + 1;
|
||||
|
||||
|
||||
Reference in New Issue
Block a user