LotusGuild/cinny
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
36343baecc8ec3bda84851a9ba5c969b4f46198e
cinny/HANDOFF_ELEMENT_CALL_FORK.md
T
jared 36343baecc
CI / Build & Quality Checks (push) Successful in 10m27s
Details
CI / Trigger Desktop Build (push) Successful in 25s
Details
call: lint/format cleanup for lotus EC wiring 
Resolve the eslint/prettier failures from the previous commit (non-blocking
in CI, but real): drop the banned `void` operator on fire-and-forget
transport.send().catch() calls, prefix the now-unused _denoiseNativeNS
param, and run prettier on the touched files.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-30 01:52:45 -04:00

40 KiB
Raw Blame History

HANDOFF — Forking & Self-Building Element Call ("Lotus Call")

Audience: a fresh Claude/engineer session with no prior context on this project. Read this top-to-bottom before touching anything. This document is the single source of truth for the Element Call (EC) fork initiative.

Status: PHASE 0–2 IMPLEMENTED (build-verified, not yet live-tested) (2026-06-30). The fork exists, builds, is published, and cinny consumes it (Phase 0/1). All 7 Phase-2 EC features are implemented on the fork's lotus branch, each additive + flag-gated, build+typecheck-clean, per-feature reviewed (+ a holistic multi-agent review), and pushed. None are live-tested yet — every one needs the LOTUS_TESTING.md §D sweep, and the cinny host side must be wired (set flags / send actions / handle call_state) — see §12. See §9 Phase 0/1 results, §10 cutover, §11 Phase-2 seams, §12 Phase-2 status + cinny integration checklist. Created 2026-06 from LotusGuild/cinny.

9. Phase 0 Results (verified 2026-06-29)

Decisions taken with the user: scope = Phase 0 recon; consumption model = private npm package (§5 option 1). Recommended registry = Gitea's built-in npm registry (code.lotusguild.org) — zero new infra.

9.1 Version → tag → commit mapping (LOCKED)

Source Value
cinny package.json pin @element-hq/element-call-embedded@0.20.1
Bundle self-report (VITE_APP_VERSION/appVersion) embedded-v0.20.1
npm registry gitHead for 0.20.1 2d74c48151d9edc01c65a22a91478aac81bf24d0
GitHub tag v0.20.1 → commit 2d74c48… ✅ same commit

→ Fork from upstream tag v0.20.1 (commit 2d74c48). The embedded package version equals the element-call release tag; repo package.json version is 0.0.0 and the real version is stamped at publish time from the tag.

9.2 The shipped npm dist is a CLEAN upstream build

No lotus/denoise/rnnoise strings anywhere in node_modules/@element-hq/element-call-embedded/dist. All Lotus customization (denoise shim) is injected at cinny build time, not baked into the package — so swapping the source does not disturb cinny's denoise injection layer. The ringtone/reaction assets (baduntss, cat, clap, call_declined, …) are upstream EC's own, not ours.

9.3 Build toolchain & mechanism

  • Node 24 (.node-version), pnpm 10.33.0 (packageManager field, via corepack).
  • Build: pnpm run build:embedded = vite build --config vite-embedded.config.ts with NODE_OPTIONS=--max-old-space-size=16384.
  • Output dir is repo-root dist/; CI stages it into embedded/web/dist (the embedded/web/ dir holds the publish template: package.json, README, both LICENSE files).
  • Publish workflow upstream = .github/workflows/publish-embedded-packages.yaml: builds → npm version <tag> --no-git-tag-version → npm publish --provenance --access public to npmjs as @element-hq/element-call-embedded. (Also Android/Maven + iOS/SwiftPM — irrelevant; we are web-only.)

9.4 Build reproduction — PARITY CONFIRMED

Cloned element-call@v0.20.1 to /root/code/element-call (shallow), built with isolated Node 24 / pnpm 10.33.0 (system Node 20 / cinny untouched). Result vs the shipped npm dist:

  • 137 of 147 files byte-identical (same Vite content-hash): all CSS, fonts, wasm, audio, JSON locale files, and IndexedDBWorker.
  • Only 5 JS chunks differ (index, pako.esm, polyfill-force, rust-crypto, spa) — cause isolated to the version define: our local build baked appVersion:\dev`(becauseVITE_APP_VERSIONwas unset) vs the npm build'sappVersion:`embedded-v0.20.1`. index.html` is identical modulo the hashed asset filenames. Benign — our CI sets the version from the git tag, so a tagged CI build will match.

9.5 Fork CI (drafted)

.gitea/workflows/ci.yml is staged in the clone (models cinny's .gitea/workflows/ci.yml + upstream's publish flow). Linux-only (ubuntu-latest) — the Windows worker is for cinny-desktop/Tauri, not the EC web bundle. Build job on PR/push to lotus; publish job on v* tag → @lotusguild/element-call-embedded to the Gitea npm registry (needs secrets.GITEA_NPM_TOKEN).

9.6 Phase 1 — DONE (2026-06-29)

  1. ✅ Fork repo live: code.lotusguild.org/LotusGuild/element-call (public, AGPL), default branch lotus, full history (7018 commits) + tag v0.20.1. Branch lotus = v0.20.1 + 2-file diff (CI workflow + embedded package rename).
  2. ✅ Package published: @lotusguild/element-call-embedded@0.20.1 on the Gitea npm registry (published manually from the version-faithful build while the admin token was available). Publicly readable (unauth npm install works → devs/CI need no token to consume; only publishing needs one).
  3. ✅ cinny wired & built clean (Node 24): .npmrc scope line + package.json dep + vite.config.js viteStaticCopy src. npm install swapped the package (resolved from Gitea), npm run build succeeded, dist/public/element-call/ populated, bundle reports appVersion: embedded-v0.20.1, denoise shim injected + all denoise assets copied (injection layer unchanged). These cinny edits are staged in the working tree, NOT committed/pushed — pushing triggers CI → desktop → deploy, so it's gated on the §D live test (see §10).

9.8 Reproducibility note (important)

A from-source rebuild is NOT byte-identical to upstream's npm tarball. 137/147 files match exactly (CSS, fonts, wasm, audio, worker); the 5 JS chunks (index, pako.esm, polyfill-force, rust-crypto, spa) differ because the rolldown/oxc minifier mangles export names differently across build environments (and the version-define is one input). This is normal and benign — the code is functionally equivalent. Do not chase byte-parity; the §D live call test is the real parity gate.

9.9 Remaining follow-ups (not blocking the cutover)

  • CI publishing: .gitea/workflows/ci.yml publishes on a v* tag but needs (a) a Gitea Actions runner for LotusGuild/element-call, and (b) a durable GITEA_NPM_TOKEN repo secret with package read/write (the admin token used for the manual publish is being deleted, so it was deliberately NOT baked in). Until then, publishing is manual (npm version <tag> in embedded/web → npm publish).
  • Decide rebase cadence vs upstream (0.20.2 / 0.20.3 already out — see §9.1).

9.7 Ready-to-apply artifacts (staged 2026-06-29)

Fork side — already committed on branch lotus in /root/code/element-call (remote lotus = code.lotusguild.org/LotusGuild/element-call.git, push deferred until the repo exists). Minimal 2-file diff vs tag v0.20.1: .gitea/workflows/ci.yml (new) + embedded/web/package.json (rename to @lotusguild/element-call-embedded). Push with: git push -u lotus lotus && git push lotus v0.20.1 (and tag v0.20.1 on our side to trigger the first publish, or push our own v0.20.1 tag).

cinny side — NOT yet applied (applying before the package is published breaks npm ci). Exactly 3 edits + a lockfile regen:

  1. .npmrc — append the scoped-registry line: 
    @lotusguild:registry=https://code.lotusguild.org/api/packages/LotusGuild/npm/
    (CI/auth: //code.lotusguild.org/api/packages/LotusGuild/npm/:_authToken=${GITEA_NPM_TOKEN} — inject via env in CI, do not commit a plaintext token.)
  2. package.json:104 — "@element-hq/element-call-embedded": "0.20.1" → "@lotusguild/element-call-embedded": "0.20.1".
  3. vite.config.js:25 — viteStaticCopy src: node_modules/@element-hq/element-call-embedded/dist → node_modules/@lotusguild/element-call-embedded/dist. stripBase: 4 stays unchanged — node_modules/@lotusguild/element-call-embedded/dist is still exactly 4 leading segments. (Update the comment's path reference too.)
  4. package-lock.json — regenerated by npm install, not hand-edited (drops the registry.npmjs.org/@element-hq/... resolved URL for the Gitea one).

The denoise injection (lotusDenoise() in vite.config.js) is unchanged — it keys off dist/public/element-call/index.html, which our fork's bundle still produces identically (verified: index.html byte-identical modulo asset hashes).

0. TL;DR / The Goal

We embed Element Call (the Matrix group-VoIP/video app) inside Lotus Chat to power voice/video channels. Today we consume Element's pre-compiled npm bundle and can only steer it from the outside (a limited widget API + fragile same-origin DOM hacks). Several in-call problems are unfixable from outside because they live in EC's compiled JS.

We want true ownership: fork element-hq/element-call, build it from source ourselves, host our build, and replace the npm bundle with our fork. Then every in-call behavior becomes editable code.

This requires standing up a brand-new repo and build pipeline for our EC fork.

1. Why fork? (What we cannot fix today)

These came out of live testing and are documented in LOTUS_BUGS.md → "Known Element Call iframe limitations":

Issue What's wrong Why outside-fixes fail
A6 — avatar decorations in-call Our profile-decoration overlays don't appear on in-call video tiles The video grid is rendered by EC's React app inside the iframe. We can only inject overlay DOM (fragile) — we can't make it a first-class part of the tile.
A5 — focus camera / fullscreen during screenshare Can't reliably spotlight a participant's camera while someone screenshares EC's layout logic (screenshare priority, spotlight) is compiled JS we don't control. We currently DOM-click tiles as a hack.
A7 — mic dead after EC's "Reconnect" After EC's own mid-call reconnect, the local mic isn't re-published EC's reconnect/track-republish path is internal. (Partly entangled with our denoise shim — see §6.)
Native theming EC's UI doesn't match Lotus design; we inject CSS hacks Real theming needs source-level component/token changes.
Decorations, custom controls, custom layouts, branding all blocked all require source access

Bottom line: the iframe is same-origin (we self-host it), so we can read and even write its DOM — but we do not own its source, so we can't change its behavior/logic, only poke at its rendered output. Forking removes that wall.

2. How EC is integrated TODAY (the current architecture)

Understand this fully before changing it — the fork must slot into the same integration seams.

2.1 Where the EC bundle comes from

  • npm package: @element-hq/element-call-embedded, pinned to 0.20.1 in cinny/package.json (line ~104).
  • It ships a pre-built dist/. At cinny build time, vite-plugin-static-copy copies that dist/ flat into public/element-call/ (see cinny/vite.config.js, the copyFiles target with rename: { stripBase: 4 } — note the stripBase gotcha documented there; getting this wrong 404s the widget).
  • It is NOT committed to git (git ls-files public/element-call → 0). It's a build artifact materialized from node_modules.

2.2 How EC is loaded & controlled

  • The widget iframe src is same-origin: ${BASE_URL}/public/element-call/index.html?<params> (see cinny/src/app/plugins/call/CallEmbed.ts, getWidget() / getIframe()). Sandbox: allow-forms allow-scripts allow-same-origin allow-popups allow-modals allow-downloads; allow="microphone; camera; display-capture; autoplay; clipboard-write;".
  • Control surface #1 — the official widget API (matrix-widget-api): ClientWidgetApi + a custom CallWidgetDriver. This is the robust, version-stable channel (theme change, hangup, capabilities, timeline events). Files: plugins/call/CallEmbed.ts, plugins/call/CallWidgetDriver.ts, plugins/call/utils.ts (capabilities), plugins/call/CallControl.ts.
  • Control surface #2 — same-origin DOM poking (fragile, version-coupled): reading iframe.contentDocument to detect speakers/mute state and .click()-ing tiles to focus a camera. Files: hooks/useCallSpeakers.ts (reads [data-muted], [data-video-fit]), plugins/call/CallControl.ts (focusCameraParticipant — tile selectors). These selectors break on every EC version bump. A fork lets us replace these hacks with real APIs/props.
  • Control surface #3 — URL params + build-time injection for our denoise shim (see §6).

2.3 Full file inventory (everything that touches EC in cinny)

Plugin / core:

  • src/app/plugins/call/CallEmbed.ts — iframe creation, widget API wiring, theme sync, hangup, load watchdog/self-heal, denoise URL params.
  • src/app/plugins/call/CallControl.ts — control state + DOM-poking (focusCameraParticipant, spotlight).
  • src/app/plugins/call/CallControl.tsx (call-status variant) and features/call-status/CallControl.tsx.
  • src/app/plugins/call/CallWidgetDriver.ts — widget driver (capabilities, event relay).
  • src/app/plugins/call/utils.ts — widget capabilities set.
  • src/app/plugins/call/hooks.ts, index.ts — plugin exports/hooks.
  • src/app/state/callEmbed.ts — jotai atoms for the active embed.

React / UI:

  • src/app/components/CallEmbedProvider.tsx — the big one: incoming-call ring/banner, RTCNotification + RTCDecline listeners, PiP, mute badges, fullscreen, ringtones.
  • src/app/features/call/CallView.tsx — prescreen lobby vs joined (the iframe placement target), load-error recovery UI.
  • src/app/features/call/CallControls.tsx — in-call control bar (mic/cam/deafen/screenshare/fullscreen/more/PiP).
  • src/app/features/call/CallMemberCard.tsx — lobby participant roster (this is where AvatarDecoration works today; in-call grid is EC's).
  • src/app/features/call/PrescreenControls.tsx — join controls.
  • src/app/features/call-status/* — CallStatus.tsx, MemberGlance.tsx (the "Focus camera" menu lives here), LiveChip.tsx.
  • src/app/features/room-nav/RoomNavItem.tsx, features/room/Room.tsx, features/room/RoomViewHeader.tsx, pages/client/space/Space.tsx, pages/CallStatusRenderer.tsx, pages/Router.tsx — call entry points / status surfacing.

Hooks:

  • src/app/hooks/useCallEmbed.ts, useCall.ts, useCallSpeakers.ts (DOM-poking), useCallJoinLeaveSounds.ts, useAfkAutoMute.ts.

Build:

  • cinny/vite.config.js — copyFiles (EC dist copy) + lotusDenoise() plugin (denoise asset copy + index.html shim injection, in closeBundle).

Utils:

  • src/app/utils/ringtones.ts, utils/denoisePipeline.ts, utils/lotusDenoiseUtils.ts.

3. Hosting / infra context (the OTHER repo)

There are two repos:

  1. LotusGuild/cinny (/root/code/cinny) — this Lotus Chat fork. Consumes EC.
  2. LotusGuild/matrix (/root/code/matrix) — the infra/homeserver repo. Subdirs: livekit/ (the SFU EC talks to), deploy/, draupnir/, hookshot/, landing/, matrixbot/, systemd/. Gitea remote code.lotusguild.org/LotusGuild/matrix, branch main.

EC needs a LiveKit SFU + the livekit-jwt-service; those live in matrix/livekit/. A self-hosted EC build must be configured to point at our homeserver (matrix.lotusguild.org / synapse) and our LiveKit. EC's runtime config.json (homeserver, livekit URL, feature flags) is part of what we'll own once we build it ourselves.

Deployment today: chat.lotusguild.org (the cinny web build, which embeds EC at /public/element-call/). cinny-desktop (LotusGuild/cinny-desktop, a Tauri wrapper, bumped by cinny CI) embeds the same.

4. The plan (proposed — confirm with the user before executing)

Decision: YES, create a new repo. LotusGuild/element-call

Rationale: EC is a large standalone app (React + LiveKit client SDK + matrixRTC + its own Vite build + heavy deps). Keep it out of cinny so cinny's build stays clean — cinny keeps consuming a built EC dist/, exactly as today, just sourced from our fork instead of npm.

Phase 0 — Recon (no code)

  • Fork github.com/element-hq/element-call → LotusGuild/element-call on Gitea.
  • Pin to the upstream tag matching 0.20.1 (element-call-embedded 0.20.1's corresponding element-call release) so behavior matches what's shipping now. Verify the embedded-package version ↔ element-call repo tag mapping.
  • Read EC's own build docs: it builds the "embedded" widget bundle (the thing currently published as @element-hq/element-call-embedded). Reproduce that build locally and confirm the output matches public/element-call/ today.
  • License: element-call is AGPL-3.0, same as Lotus Chat — compatible. Our fork must remain AGPL and publish source.

Phase 1 — Reproduce current behavior from our fork (parity, no features)

  • Build our fork's embedded bundle; wire cinny to consume it instead of the npm package (see §5 for the consumption options). Smoke-test: a call works exactly as today (web + desktop), denoise shim still injects, widget API + theme still work. No behavior change yet — this de-risks the swap.

Phase 2 — Replace the outside hacks with source-level features

Tackle the §1 issues in EC's source:

  • A6: render avatar decorations as part of the video-tile component (read decoration data we pass in via widget data / URL param / a small bridge).
  • A5: fix focus/spotlight + screenshare-coexistence in EC's layout code; expose a clean widget action so cinny can trigger it (kill the DOM .click()).
  • A7: fix mic re-publish on reconnect; reconcile with our denoise shim (§6) — ideally move denoise INTO the fork as a real audio-processing step instead of a getUserMedia monkeypatch.
  • Native Lotus theming/branding at the source (kill the injected-CSS hacks).
  • Then retire the DOM-poking in useCallSpeakers.ts / CallControl.ts in favor of real widget messages.

Phase 3 — Maintenance posture

  • Decide rebase cadence vs. upstream element-call releases. Keep customizations isolated (feature flags / minimal-diff patches) to ease rebasing.
  • CI in the new repo builds + publishes the embedded dist as a versioned artifact; cinny CI consumes a pinned version.

5. How cinny should consume the fork (pick one — decide with user)

  1. Private npm package (mirror the current model): our fork's CI publishes @lotusguild/element-call-embedded to a registry; cinny depends on it and viteStaticCopy keeps working almost unchanged. Cleanest swap; needs a registry.
  2. Git submodule + build in cinny CI: add the fork as a submodule, build it during cinny's build, copy its dist/ to public/element-call/. No registry; heavier cinny CI.
  3. CI artifact copy: fork CI uploads a dist tarball; cinny CI downloads a pinned version at build. Decoupled; needs artifact plumbing.

Recommendation: Option 1 — it changes the least in cinny (just swap the package name in package.json + the viteStaticCopy src path) and preserves the clean cinny/EC separation.

6. The denoise shim — critical interaction (don't break this)

Lotus ships ML noise suppression by injecting a same-origin pre-init shim into EC's index.html at build time (cinny vite.config.js → lotusDenoise(), closeBundle). The shim monkeypatches getUserMedia before EC captures the mic and routes audio through RNNoise/Speex/DTLN AudioWorklets, then EC/LiveKit publishes the processed track. It's activated via URL params (lotusDenoise=ml&lotusModel=…&lotusGate=…) set in CallEmbed.ts.

  • Assets copied to public/element-call/denoise/ at build (sapphi RNNoise/Speex/ gate worklets + @workadventure/noise-suppression DTLN tree).
  • Related: utils/denoisePipeline.ts, utils/lotusDenoiseUtils.ts, settings/general/DenoiseTester.tsx, VoiceMessageRecorder.tsx.
  • Known issues: denoise quality is still poor (tracked separately); and the mic-after-reconnect bug (A7) is suspected to involve the shim's getUserMedia patch handing back a stale processed stream when EC re-acquires the mic.

Once we own the fork, the right move is to make denoise a first-class audio-processing stage inside EC (not an index.html monkeypatch) — more robust, survives reconnects, and removes the build-time injection hack. Until then, the fork's index.html must remain injectable the same way, or the shim must be re-homed into the fork.

7. Doc-accuracy notes / corrections for the new session

  • LOTUS_TODO.md (~line 533) calls EC a "cross-origin iframe" — outdated. EC is same-origin today (self-hosted under our domain; iframe.sandbox includes allow-same-origin; we read contentDocument), and as of 2026-06-29 we own the fork's source (@lotusguild/element-call-embedded). The practical point it made still holds until we ship the audio-inject API: LiveKit's LocalAudioTrack lives in EC's module scope, not on window, so cinny can't reach it even same-origin — which is why the in-call soundboard had to be local-playback-only. The fork removes this wall: EC can expose a real io.lotus.inject_audio widget action (Phase 2) that mixes into the published track from inside its own module scope.
  • LOTUS_FEATURES.md documents the EC upgrade history (0.16.3 → 0.19.4 → 0.20.1), the dark-mode CSS injection, and AFK auto-mute — all relevant prior art for what the fork must preserve.
  • LOTUS_TESTING.md §D is the EC regression sweep to re-run after the fork swap (Phase 1 parity check).

8. First actions for the new session

  1. Read this file, then skim §2.3's files in cinny to internalize the seams.
  2. Confirm with the user: new repo name, consumption model (§5), rebase cadence.
  3. Phase 0: fork element-call, map 0.20.1 ↔ element-call tag, reproduce the embedded build locally, diff against public/element-call/.
  4. Phase 1: wire cinny to the fork, run LOTUS_TESTING.md §D parity sweep.
  5. Only then start Phase 2 features (A5/A6/A7, theming, denoise-in-source).

Cross-references: LOTUS_BUGS.md (EC limitations + verify queue), LOTUS_TODO.md (denoise/soundboard constraints), LOTUS_FEATURES.md (EC history), LOTUS_TESTING.md §D (regression sweep). Infra: /root/code/matrix (livekit/, deploy/).

10. Live cutover — the remaining steps (Phase 1 finish)

The fork is published and cinny builds against it locally (§9.6). What's left to go live:

  1. Run LOTUS_TESTING.md §D against a local cinny build (npm run build is already proven; serve dist/ or npm run dev). Verify a real call: join, mic/cam, screenshare, theme sync, denoise on, widget hangup — web first.
  2. Commit the cinny edits (currently staged, uncommitted in the working tree): .npmrc, package.json, package-lock.json, vite.config.js. Suggested message: chore(call): consume self-built @lotusguild/element-call-embedded.
  3. Push to lotus → cinny CI builds, then trigger-desktop bumps cinny-desktop → Tauri release. Re-run §D on cinny-desktop (the path where the old stripBase bug bit — verify the widget loads, not a 404).
  4. Only then start Phase 2 (A5/A6/A7, theming, denoise-in-source).

11. Phase 2 — implementation seams (mapped 2026-06-29)

The exact integration points for each Phase 2 item, found by reading the EC fork

  • cinny source. All of these are media-path / in-call features that cannot be functionally verified without a live Matrix + LiveKit call — implement each as a minimal, feature-flagged, additive diff (no behavior change unless cinny opts in), build-verify the fork (pnpm build:embedded, ~15s) AND cinny (npm run build), then gate shipping on LOTUS_TESTING.md §D.

Shared widget channel (the backbone for #2/#3/#4/#7):

  • EC→cinny: widget.api.transport.send("io.lotus.<x>", data) (see element-call/src/widget.ts).
  • cinny→EC actions: add the action name to the lazyActions allow-list in widget.ts (the array at ~L101) and handle it in EC; cinny sends via this.call.transport.send(...).
  • cinny receives EC→cinny actions via the existing listenAction(type, cb) helper in plugins/call/CallEmbed.ts:626 (auto-replies {} so the transport doesn't time out — same pattern as io.element.device_mute).

#2 mute/speaker events — Source: subscribe to vm.userMedia$ (CallViewModel), per member speaking$ + audioEnabled$ (state/media/UserMediaViewModel.ts:47-48); aggregate and transport.send("io.lotus.call_state", {participants:[{id,speaking,audioEnabled}]}). Mount in room/InCallView.tsx via useEffect guarded by widget !== null. cinny: listenAction("io.lotus.call_state") in CallEmbed.ts, feed hooks/useCallSpeakers.ts → delete its contentDocument [data-muted] / [data-video-fit] scrape. Additive, low risk.

#4 spotlight/focus — EC: add io.lotus.focus_participant to the lazyActions list (widget.ts), drive vm's spotlight (spotlightSpeaker$ / spotlight$ in CallViewModel.ts:898/1001) to pin a given identity, coexisting with hasRemoteScreenShares$ (L1008). cinny: replace CallControl.ts focusCameraParticipant .click() walk with transport.send("io.lotus.focus_participant", {userId}). Additive, low risk.

#3 audio-inject — EC: add io.lotus.inject_audio action; mix an AudioBufferSourceNode into the published mic track. The local publish path is state/CallViewModel/localMember/Publisher.ts + LocalMember.ts (LiveKit localParticipant); create a MediaStreamAudioDestinationNode, mix mic + clip, replaceTrack. cinny soundboard calls the action instead of local-only playback. Medium; touches publish path → live-test carefully.

#1 denoise-in-source — replace the cinny lotusDenoise() getUserMedia monkeypatch with a real processing stage in EC's mic capture (Publisher.ts/LocalMember.ts; note EC has a TrackProcessorContext + BlurBackgroundTransformer precedent in livekit/). EC re-runs it on every (re)publish → fixes A7. Remove vite.config.js lotusDenoise() + URL params in CallEmbed.ts; move denoise/ assets into the fork. Highest value, highest risk — most live testing.

#5 theming — add a Lotus/TDS theme in EC's theme system (src/useTheme.ts + EC theme tokens / CSS); driven by the existing setTheme() channel cinny already calls (CallEmbed.ts:277). Bake transparent background. Delete cinny's applyStyles() injection + background:none !important. Medium.

#6 in-call decorations — render the decoration APNG in EC's tile component (tile/GridTile.tsx); pass slugs via widget member data. cinny already has the decoration data + AvatarDecoration (lobby CallMemberCard.tsx). Medium-Large.

#7 quality controls — set audio maxBitrate via RTCRtpSender.setParameters and screenshare getDisplayMedia constraints in EC's publish path (Publisher.ts); configurable via config.json / a widget message. Keep the server voice-limit-guard as enforcement. Medium.

Rollback: revert the 4 cinny files (restores @element-hq/...@0.20.1 from npmjs). The fork repo/package can stay; nothing else depends on it until pushed.

Local repro/build environment (this session, 2026-06-29)

  • Upstream cloned + our lotus branch at /root/code/element-call (remote lotus → Gitea; origin → github upstream, now un-shallowed/full history).
  • Isolated Node 24.18.0 lives in the session scratchpad (system Node is 20); cinny's .node-version is 24.13.1, so use Node 24 to build cinny too.
  • Build the embedded bundle: in /root/code/element-call, with Node 24 + pnpm 10.33.0 on PATH, VITE_APP_VERSION=embedded-v0.20.1 pnpm run build:embedded → output in dist/; stage to embedded/web/dist before publishing.

12. Phase 2 — IMPLEMENTED on the fork (2026-06-30)

All 7 EC features are on the lotus branch of LotusGuild/element-call, each additive + feature-flagged (a vanilla call with no lotus* params / no Lotus actions behaves exactly like upstream), build + tsc clean, per-feature reviewed (fixes applied) and holistically reviewed. Not yet live-tested — all need the LOTUS_TESTING.md §D sweep.

Fork modules live under element-call/src/lotus/*; mounts are useEffects in src/room/InCallView.tsx. Custom widget actions are in src/lotus/lotusActions.ts (toWidget ones allow-listed in src/widget.ts).

# Feature Enable via EC module
2 Speaker/mute/camera state → host URL lotusCallState=1 lotusCallState.ts (sends io.lotus.call_state)
4 Focus/spotlight a participant (works during screenshare) action `io.lotus.focus_participant {userId null}` lotusFocus.ts + CallViewModel spotlight override
3 Soundboard audio-inject (heard by peers) URL lotusAudioInject=1 + action io.lotus.inject_audio {url,volume?} lotusAudioInject.ts
7 Audio/screenshare quality caps action io.lotus.set_quality {audioMaxBitrate?,screenshareMaxBitrate?,screenshareMaxFramerate?} lotusQuality.ts
5 Transparent bg + Lotus theme URL lotusTransparent=1 / lotusTheme=1 useTheme.ts + index.css
6 In-call avatar decorations action io.lotus.decorations {decorations:{userId:url}} lotusDecorations.ts + MediaView.tsx
1 ML denoise in-source (fixes A7) URL lotusDenoiseSource=1 (+lotusModel,lotusGate,lotusGateThreshold,lotusDenoiseBase) — deliberately NOT the existing lotusDenoise=ml (that drives the host shim; reusing it would double-process) lotusDenoise.ts + lotusDenoiseProcessor.ts

Security hardening applied (holistic audit): lotusDenoiseBase forced same-origin before audioWorklet.addModule (was an arbitrary-code-load vector via a crafted link); audio-inject gated behind lotusAudioInject=1; decoration roster capped. Only https/blob URLs accepted for inject/decoration assets.

12.1 cinny host integration checklist (REQUIRED to light these up)

The EC side is additive and dormant until cinny opts in. Host work needed (in src/app/plugins/call/CallEmbed.ts unless noted):

⚠️ CRITICAL TIMING (protocol audit F1): only send io.lotus.* toWidget actions (#3 focus, #6 decorations, #7 quality, audio-inject) after the call is joined (CallEmbed.onCallJoined / this.joined). Those actions are allow-listed at EC app-init (so preventDefault suppresses the auto-error) but their handlers only mount with InCallView (post-join). Sending earlier leaves the host's transport.send pending until the 10s timeout. Queue and flush on join, or no-op before join.

Also: F3 — the fork implements only rnnoise/speex; cinny's dtln/ deepfilternet selections silently fall back to rnnoise (now logged). Restrict the embedded-call model picker to rnnoise/speex, or implement the others in lotusDenoiseProcessor.ts. F4 — cinny sends lotusNativeNS, which the fork ignores; drop it or wire it in. F7 — no widget capability changes needed; custom actions bypass capability checks.

  1. Set the URL flags on the widget iframe params (the URLSearchParams in CallEmbed): lotusCallState=1, lotusTransparent=1/lotusTheme=1, lotusAudioInject=1 as desired. (Denoise already sets lotusDenoise=ml etc.)
  2. Ack io.lotus.call_state: add listenAction('io.lotus.call_state', …) — without a reply the fork's sends time out every 250ms. Feed the payload into useCallSpeakers and RETIRE its contentDocument DOM scrape.
  3. Send actions via this.call.transport.send(...): io.lotus.focus_participant (replace CallControl.focusCameraParticipant’s .click()), io.lotus.inject_audio (from the soundboard), io.lotus.set_quality (from quality settings), io.lotus.decorations (push the MSC4133 decoration map; resolve mxc→https first).
  4. #1 denoise cutover: once verified, STOP injecting the lotusDenoise() shim in cinny/vite.config.js and remove the index.html injection — the fork now does denoise in-source. Keep shipping the denoise/ assets (the fork loads ./denoise/… at runtime) until those move into the fork build.
  5. Re-run LOTUS_TESTING.md §D for each feature; only then ship.

12.2 Holistic multi-agent review — outstanding follow-ups (non-blocking)

Four aspect-agents reviewed the whole fork. Criticals were fixed in-branch (the denoise restart-silence/A7 bug; the lotusDenoiseBase code-load vector; audio-inject opt-in gate; #6 rendering in the wrong component; #7 simulcast cap). Remaining, deliberately deferred:

  • Denoise H2 (double-processing): if cinny is set to lotusDenoise=ml while ALSO still injecting its build-time getUserMedia shim, audio is denoised twice. The #1 cutover MUST remove the cinny-side injection (it currently has none injected into the iframe — keep it that way). Hard requirement, not code.
  • Denoise M1 (perf): in-source uses non-SIMD rnnoise.wasm; the reference preferred SIMD with detection. Perf-only; add SIMD detection later.
  • dtln/deepfilternet (F3): RESOLVED — all four models (rnnoise/speex/dtln/deepfilternet) are now implemented in lotusDenoiseProcessor.ts (faithful port of cinny's build/lotus-denoise.js pipeline). This also fixed a real bug (the gate worklet name was noiseGate; correct is the hyphenated noise-gate) and added per-model sample rates (DTLN 16 kHz, others 48 kHz), context resume(), and SIMD wasm selection. Still needs live §D testing per model, and depends on cinny shipping the DTLN (denoise/workadventure/) + DeepFilterNet (denoise/deepfilternet/) asset trees (it already does).
  • Rebase-fragility (build agent MED): the CallViewModel spotlight override edits hot upstream lines (renamed spotlightSpeaker$→autoSpotlightSpeaker$). For cheaper future rebases, refactor it into a src/lotus/lotusSpotlight.ts wrapper that takes the upstream stream and returns the overridden one, leaving upstream's definition byte-identical (a single import + two token swaps).
  • Denoise asset coupling (build agent HIGH): the fork loads ./denoise/* shipped by cinny, not by the fork build (documented in the processor). Add an integration smoke-check that GET …/element-call/denoise/rnnoise.wasm == 200, and pin the @sapphi-red/web-noise-suppressor version both repos expect.
  • Unconditional effect registration (build agent LOW): focus/audio-inject/ quality/decorations register widget handlers on every embedded call (true no-ops for a non-Lotus host). Intentional; gate behind a coarse lotus=1 flag if strict zero-footprint is desired.
  • Privacy (security agent): decoration/inject URLs accept any https; ideally restrict to the homeserver media origin host-side. Call-state exposes userId/deviceId/speaking to the (trusted, same-origin) host — documented.

Nothing here blocks the §D live test — but every feature still needs it.

12.3 Safe rollout when prod is the only test environment

Every Phase-2 feature is now dormant by default — with the flags cinny sets today, the fork behaves identically to the parity build (#1 was decoupled onto lotusDenoiseSource=1 so it no longer collides with the host's lotusDenoise=ml shim). This enables a low-risk incremental rollout even without a staging env:

  1. Ship dormant first. Publish the lotus branch (e.g. 0.20.1-lotus.1), bump cinny's pin, deploy. With no Lotus flags set / no Lotus actions sent, this is upstream-equivalent (only inert, holistically-reviewed code runs). "Testing" here = confirm a normal call still works.
  2. Enable ONE feature at a time, each independently revertable:
    • URL-flag features (#2 lotusCallState, #5 lotusTransparent/lotusTheme, #1 lotusDenoiseSource): add the flag in CallEmbed.getWidget, deploy, test that one feature, roll back just that flag if needed.
    • Action features (#3,#4,#6,#7): wire the host send + (for #2) the listenAction ack, gated on join (§12.1 F1).
  3. #1 denoise cutover is a coordinated 2-step (do together): set lotusDenoiseSource=1 AND remove the lotusDenoise() shim injection + lotusDenoise=ml param in cinny — otherwise audio is denoised twice. Roll back = revert both.
  4. Baseline is always upstream-equivalent, so any single feature can be disabled by flipping its flag/send off without touching the rest.

Blocker to step 1: publishing the lotus branch needs a Gitea npm token (the admin token used for the 0.20.1 parity publish was deleted). Either provide a token for a manual npm publish, or stand up the Gitea Actions runner

  • GITEA_NPM_TOKEN secret so a v0.20.1-lotus.1 tag auto-publishes.
Reference in New Issue View Git Blame Copy Permalink