docs: add Element Call fork handoff + tag all EC-FORK references
Captures the plan to fork element-hq/element-call and build it from source for true ownership of the in-call experience (decorations, focus/screenshare, reconnect mic, native theming, call-audio injection) — none of which are fixable against the prebuilt @element-hq/element-call-embedded bundle we ship today. - New HANDOFF_ELEMENT_CALL_FORK.md: self-contained plan for a fresh session (current architecture, full file inventory, phases, new-repo decision, the denoise-shim interaction, doc corrections). - Tagged every related note with [EC-FORK] + links: README (For Developers), LOTUS_BUGS (EC limitations), LOTUS_TODO (soundboard, denoise, soundboard cross-origin correction), LOTUS_FEATURES (call section). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,270 @@
|
||||
# HANDOFF — Forking & Self-Building Element Call ("Lotus Call")
|
||||
|
||||
> **Audience:** a fresh Claude/engineer session with **no prior context** on this
|
||||
> project. Read this top-to-bottom before touching anything. This document is the
|
||||
> single source of truth for the Element Call (EC) fork initiative.
|
||||
>
|
||||
> **Status:** NOT STARTED. This is the planning/handoff artifact. Created
|
||||
> 2026-06 from the Lotus Chat (`LotusGuild/cinny`) repo.
|
||||
|
||||
---
|
||||
|
||||
## 0. TL;DR / The Goal
|
||||
|
||||
We embed **Element Call** (the Matrix group-VoIP/video app) inside Lotus Chat to
|
||||
power voice/video channels. Today we consume Element's **pre-compiled npm
|
||||
bundle** and can only steer it from the outside (a limited widget API + fragile
|
||||
same-origin DOM hacks). Several in-call problems are **unfixable from outside**
|
||||
because they live in EC's compiled JS.
|
||||
|
||||
**We want true ownership: fork `element-hq/element-call`, build it from source
|
||||
ourselves, host our build, and replace the npm bundle with our fork.** Then
|
||||
every in-call behavior becomes editable code.
|
||||
|
||||
**This requires standing up a brand-new repo and build pipeline for our EC fork.**
|
||||
|
||||
---
|
||||
|
||||
## 1. Why fork? (What we cannot fix today)
|
||||
|
||||
These came out of live testing and are documented in `LOTUS_BUGS.md` →
|
||||
"Known Element Call iframe limitations":
|
||||
|
||||
| Issue | What's wrong | Why outside-fixes fail |
|
||||
| :----------------------------------------------------- | :------------------------------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **A6** — avatar decorations in-call | Our profile-decoration overlays don't appear on in-call video tiles | The video grid is rendered by EC's React app inside the iframe. We can only inject overlay DOM (fragile) — we can't make it a first-class part of the tile. |
|
||||
| **A5** — focus camera / fullscreen during screenshare | Can't reliably spotlight a participant's camera while someone screenshares | EC's **layout logic** (screenshare priority, spotlight) is compiled JS we don't control. We currently DOM-click tiles as a hack. |
|
||||
| **A7** — mic dead after EC's "Reconnect" | After EC's own mid-call reconnect, the local mic isn't re-published | EC's reconnect/track-republish path is internal. (Partly entangled with our denoise shim — see §6.) |
|
||||
| Native theming | EC's UI doesn't match Lotus design; we inject CSS hacks | Real theming needs source-level component/token changes. |
|
||||
| Decorations, custom controls, custom layouts, branding | all blocked | all require source access |
|
||||
|
||||
**Bottom line:** the iframe is **same-origin** (we self-host it), so we can read
|
||||
and even write its DOM — but we **do not own its source**, so we can't change its
|
||||
**behavior/logic**, only poke at its rendered output. Forking removes that wall.
|
||||
|
||||
---
|
||||
|
||||
## 2. How EC is integrated TODAY (the current architecture)
|
||||
|
||||
Understand this fully before changing it — the fork must slot into the same
|
||||
integration seams.
|
||||
|
||||
### 2.1 Where the EC bundle comes from
|
||||
|
||||
- npm package: **`@element-hq/element-call-embedded`**, pinned to **`0.20.1`** in
|
||||
`cinny/package.json` (line ~104).
|
||||
- It ships a **pre-built `dist/`**. At cinny build time,
|
||||
`vite-plugin-static-copy` copies that `dist/` flat into
|
||||
**`public/element-call/`** (see `cinny/vite.config.js`, the `copyFiles`
|
||||
target with `rename: { stripBase: 4 }` — note the stripBase gotcha documented
|
||||
there; getting this wrong 404s the widget).
|
||||
- It is **NOT committed** to git (`git ls-files public/element-call` → 0). It's a
|
||||
build artifact materialized from `node_modules`.
|
||||
|
||||
### 2.2 How EC is loaded & controlled
|
||||
|
||||
- The widget iframe `src` is **same-origin**:
|
||||
`${BASE_URL}/public/element-call/index.html?<params>` (see
|
||||
`cinny/src/app/plugins/call/CallEmbed.ts`, `getWidget()` /
|
||||
`getIframe()`). Sandbox: `allow-forms allow-scripts allow-same-origin
|
||||
allow-popups allow-modals allow-downloads`; `allow="microphone; camera;
|
||||
display-capture; autoplay; clipboard-write;"`.
|
||||
- **Control surface #1 — the official widget API** (`matrix-widget-api`):
|
||||
`ClientWidgetApi` + a custom `CallWidgetDriver`. This is the robust,
|
||||
version-stable channel (theme change, hangup, capabilities, timeline events).
|
||||
Files: `plugins/call/CallEmbed.ts`, `plugins/call/CallWidgetDriver.ts`,
|
||||
`plugins/call/utils.ts` (capabilities), `plugins/call/CallControl.ts`.
|
||||
- **Control surface #2 — same-origin DOM poking** (fragile, version-coupled):
|
||||
reading `iframe.contentDocument` to detect speakers/mute state and
|
||||
`.click()`-ing tiles to focus a camera. Files:
|
||||
`hooks/useCallSpeakers.ts` (reads `[data-muted]`, `[data-video-fit]`),
|
||||
`plugins/call/CallControl.ts` (`focusCameraParticipant` — tile selectors).
|
||||
**These selectors break on every EC version bump.** A fork lets us replace
|
||||
these hacks with real APIs/props.
|
||||
- **Control surface #3 — URL params + build-time injection** for our denoise
|
||||
shim (see §6).
|
||||
|
||||
### 2.3 Full file inventory (everything that touches EC in cinny)
|
||||
|
||||
Plugin / core:
|
||||
|
||||
- `src/app/plugins/call/CallEmbed.ts` — iframe creation, widget API wiring, theme sync, hangup, load watchdog/self-heal, denoise URL params.
|
||||
- `src/app/plugins/call/CallControl.ts` — control state + **DOM-poking** (`focusCameraParticipant`, spotlight).
|
||||
- `src/app/plugins/call/CallControl.tsx` _(call-status variant)_ and `features/call-status/CallControl.tsx`.
|
||||
- `src/app/plugins/call/CallWidgetDriver.ts` — widget driver (capabilities, event relay).
|
||||
- `src/app/plugins/call/utils.ts` — widget capabilities set.
|
||||
- `src/app/plugins/call/hooks.ts`, `index.ts` — plugin exports/hooks.
|
||||
- `src/app/state/callEmbed.ts` — jotai atoms for the active embed.
|
||||
|
||||
React / UI:
|
||||
|
||||
- `src/app/components/CallEmbedProvider.tsx` — the big one: incoming-call ring/banner, RTCNotification + **RTCDecline** listeners, PiP, mute badges, fullscreen, ringtones.
|
||||
- `src/app/features/call/CallView.tsx` — prescreen lobby vs joined (the iframe placement target), load-error recovery UI.
|
||||
- `src/app/features/call/CallControls.tsx` — in-call control bar (mic/cam/deafen/screenshare/fullscreen/more/PiP).
|
||||
- `src/app/features/call/CallMemberCard.tsx` — **lobby** participant roster (this is where `AvatarDecoration` works today; in-call grid is EC's).
|
||||
- `src/app/features/call/PrescreenControls.tsx` — join controls.
|
||||
- `src/app/features/call-status/*` — `CallStatus.tsx`, `MemberGlance.tsx` (the "Focus camera" menu lives here), `LiveChip.tsx`.
|
||||
- `src/app/features/room-nav/RoomNavItem.tsx`, `features/room/Room.tsx`, `features/room/RoomViewHeader.tsx`, `pages/client/space/Space.tsx`, `pages/CallStatusRenderer.tsx`, `pages/Router.tsx` — call entry points / status surfacing.
|
||||
|
||||
Hooks:
|
||||
|
||||
- `src/app/hooks/useCallEmbed.ts`, `useCall.ts`, `useCallSpeakers.ts` (DOM-poking), `useCallJoinLeaveSounds.ts`, `useAfkAutoMute.ts`.
|
||||
|
||||
Build:
|
||||
|
||||
- `cinny/vite.config.js` — `copyFiles` (EC dist copy) + `lotusDenoise()` plugin (denoise asset copy + index.html shim injection, in `closeBundle`).
|
||||
|
||||
Utils:
|
||||
|
||||
- `src/app/utils/ringtones.ts`, `utils/denoisePipeline.ts`, `utils/lotusDenoiseUtils.ts`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Hosting / infra context (the OTHER repo)
|
||||
|
||||
There are **two repos**:
|
||||
|
||||
1. **`LotusGuild/cinny`** (`/root/code/cinny`) — this Lotus Chat fork. Consumes EC.
|
||||
2. **`LotusGuild/matrix`** (`/root/code/matrix`) — the **infra/homeserver** repo.
|
||||
Subdirs: `livekit/` (the SFU EC talks to), `deploy/`, `draupnir/`,
|
||||
`hookshot/`, `landing/`, `matrixbot/`, `systemd/`. Gitea remote
|
||||
`code.lotusguild.org/LotusGuild/matrix`, branch `main`.
|
||||
|
||||
EC needs a **LiveKit SFU** + the **livekit-jwt-service**; those live in
|
||||
`matrix/livekit/`. A self-hosted EC build must be configured to point at our
|
||||
homeserver (`matrix.lotusguild.org` / synapse) and our LiveKit. EC's runtime
|
||||
`config.json` (homeserver, livekit URL, feature flags) is part of what we'll own
|
||||
once we build it ourselves.
|
||||
|
||||
Deployment today: `chat.lotusguild.org` (the cinny web build, which embeds EC at
|
||||
`/public/element-call/`). cinny-desktop (`LotusGuild/cinny-desktop`, a Tauri
|
||||
wrapper, bumped by cinny CI) embeds the same.
|
||||
|
||||
---
|
||||
|
||||
## 4. The plan (proposed — confirm with the user before executing)
|
||||
|
||||
### Decision: **YES, create a new repo.** `LotusGuild/element-call`
|
||||
|
||||
Rationale: EC is a large standalone app (React + LiveKit client SDK + matrixRTC +
|
||||
its own Vite build + heavy deps). Keep it out of cinny so cinny's build stays
|
||||
clean — cinny keeps consuming a **built EC `dist/`**, exactly as today, just
|
||||
sourced from **our fork** instead of npm.
|
||||
|
||||
### Phase 0 — Recon (no code)
|
||||
|
||||
- Fork `github.com/element-hq/element-call` → `LotusGuild/element-call` on Gitea.
|
||||
- Pin to the upstream tag matching **0.20.1** (`element-call-embedded` 0.20.1's
|
||||
corresponding `element-call` release) so behavior matches what's shipping now.
|
||||
Verify the embedded-package version ↔ element-call repo tag mapping.
|
||||
- Read EC's own build docs: it builds the "embedded" widget bundle (the thing
|
||||
currently published as `@element-hq/element-call-embedded`). Reproduce that
|
||||
build locally and confirm the output matches `public/element-call/` today.
|
||||
- **License:** element-call is **AGPL-3.0**, same as Lotus Chat — compatible.
|
||||
Our fork must remain AGPL and publish source.
|
||||
|
||||
### Phase 1 — Reproduce current behavior from our fork (parity, no features)
|
||||
|
||||
- Build our fork's embedded bundle; wire cinny to consume it instead of the npm
|
||||
package (see §5 for the consumption options). Smoke-test: a call works exactly
|
||||
as today (web + desktop), denoise shim still injects, widget API + theme still
|
||||
work. **No behavior change yet** — this de-risks the swap.
|
||||
|
||||
### Phase 2 — Replace the outside hacks with source-level features
|
||||
|
||||
Tackle the §1 issues in EC's source:
|
||||
|
||||
- **A6:** render avatar decorations as part of the video-tile component
|
||||
(read decoration data we pass in via widget data / URL param / a small bridge).
|
||||
- **A5:** fix focus/spotlight + screenshare-coexistence in EC's layout code;
|
||||
expose a clean widget action so cinny can trigger it (kill the DOM `.click()`).
|
||||
- **A7:** fix mic re-publish on reconnect; reconcile with our denoise shim (§6) —
|
||||
ideally move denoise INTO the fork as a real audio-processing step instead of a
|
||||
`getUserMedia` monkeypatch.
|
||||
- Native Lotus theming/branding at the source (kill the injected-CSS hacks).
|
||||
- Then retire the DOM-poking in `useCallSpeakers.ts` / `CallControl.ts` in favor
|
||||
of real widget messages.
|
||||
|
||||
### Phase 3 — Maintenance posture
|
||||
|
||||
- Decide rebase cadence vs. upstream element-call releases. Keep customizations
|
||||
isolated (feature flags / minimal-diff patches) to ease rebasing.
|
||||
- CI in the new repo builds + publishes the embedded dist as a versioned
|
||||
artifact; cinny CI consumes a pinned version.
|
||||
|
||||
---
|
||||
|
||||
## 5. How cinny should consume the fork (pick one — decide with user)
|
||||
|
||||
1. **Private npm package** (mirror the current model): our fork's CI publishes
|
||||
`@lotusguild/element-call-embedded` to a registry; cinny depends on it and
|
||||
`viteStaticCopy` keeps working almost unchanged. _Cleanest swap; needs a
|
||||
registry._
|
||||
2. **Git submodule + build in cinny CI:** add the fork as a submodule, build it
|
||||
during cinny's build, copy its `dist/` to `public/element-call/`. _No
|
||||
registry; heavier cinny CI._
|
||||
3. **CI artifact copy:** fork CI uploads a `dist` tarball; cinny CI downloads a
|
||||
pinned version at build. _Decoupled; needs artifact plumbing._
|
||||
|
||||
**Recommendation: Option 1** — it changes the least in cinny (just swap the
|
||||
package name in `package.json` + the `viteStaticCopy` src path) and preserves the
|
||||
clean cinny/EC separation.
|
||||
|
||||
---
|
||||
|
||||
## 6. The denoise shim — critical interaction (don't break this)
|
||||
|
||||
Lotus ships ML noise suppression by **injecting a same-origin pre-init shim into
|
||||
EC's `index.html` at build time** (cinny `vite.config.js` → `lotusDenoise()`,
|
||||
`closeBundle`). The shim monkeypatches `getUserMedia` **before EC captures the
|
||||
mic** and routes audio through RNNoise/Speex/DTLN AudioWorklets, then EC/LiveKit
|
||||
publishes the processed track. It's activated via URL params
|
||||
(`lotusDenoise=ml&lotusModel=…&lotusGate=…`) set in `CallEmbed.ts`.
|
||||
|
||||
- Assets copied to `public/element-call/denoise/` at build (sapphi RNNoise/Speex/
|
||||
gate worklets + `@workadventure/noise-suppression` DTLN tree).
|
||||
- Related: `utils/denoisePipeline.ts`, `utils/lotusDenoiseUtils.ts`,
|
||||
`settings/general/DenoiseTester.tsx`, `VoiceMessageRecorder.tsx`.
|
||||
- **Known issues:** denoise quality is still poor (tracked separately); and the
|
||||
mic-after-reconnect bug (A7) is suspected to involve the shim's getUserMedia
|
||||
patch handing back a stale processed stream when EC re-acquires the mic.
|
||||
|
||||
**Once we own the fork, the right move is to make denoise a first-class
|
||||
audio-processing stage inside EC** (not an index.html monkeypatch) — more robust,
|
||||
survives reconnects, and removes the build-time injection hack. Until then, the
|
||||
fork's `index.html` must remain injectable the same way, or the shim must be
|
||||
re-homed into the fork.
|
||||
|
||||
---
|
||||
|
||||
## 7. Doc-accuracy notes / corrections for the new session
|
||||
|
||||
- `LOTUS_TODO.md` (~line 533) calls EC a **"cross-origin iframe"** — **outdated.**
|
||||
EC is **same-origin** today (self-hosted under our domain;
|
||||
`iframe.sandbox` includes `allow-same-origin`; we read `contentDocument`). The
|
||||
_practical_ point it makes still holds: **LiveKit's `LocalAudioTrack` lives in
|
||||
EC's module scope**, not on `window`, so we can't reach it from cinny even
|
||||
same-origin — which is exactly why the in-call soundboard had to be
|
||||
local-playback-only, and another reason to fork (a fork could expose a real
|
||||
audio-inject API).
|
||||
- `LOTUS_FEATURES.md` documents the EC upgrade history (0.16.3 → 0.19.4 →
|
||||
0.20.1), the dark-mode CSS injection, and AFK auto-mute — all relevant prior
|
||||
art for what the fork must preserve.
|
||||
- `LOTUS_TESTING.md` §D is the **EC regression sweep** to re-run after the fork
|
||||
swap (Phase 1 parity check).
|
||||
|
||||
---
|
||||
|
||||
## 8. First actions for the new session
|
||||
|
||||
1. Read this file, then skim §2.3's files in `cinny` to internalize the seams.
|
||||
2. Confirm with the user: new repo name, consumption model (§5), rebase cadence.
|
||||
3. Phase 0: fork element-call, map 0.20.1 ↔ element-call tag, reproduce the
|
||||
embedded build locally, diff against `public/element-call/`.
|
||||
4. Phase 1: wire cinny to the fork, run `LOTUS_TESTING.md` §D parity sweep.
|
||||
5. Only then start Phase 2 features (A5/A6/A7, theming, denoise-in-source).
|
||||
|
||||
**Cross-references:** `LOTUS_BUGS.md` (EC limitations + verify queue),
|
||||
`LOTUS_TODO.md` (denoise/soundboard constraints), `LOTUS_FEATURES.md` (EC history),
|
||||
`LOTUS_TESTING.md` §D (regression sweep). Infra: `/root/code/matrix` (`livekit/`,
|
||||
`deploy/`).
|
||||
Reference in New Issue
Block a user