LotusGuild/cinny
3
0
Fork 0
Code Issues 27 Pull Requests 5 Actions Packages Projects Releases Wiki Activity

[planning] E2EE / call-key cluster KE-1→4: OTK upload conflict storm, EC MissingKey, malformed algorithm, MatrixRTC delayed-event timeouts — capture first, do not spot-fix #201

New Issue
Open
opened 2026-09-17 23:25:31 -04:00 by jared · 1 comment
jared commented 2026-09-17 23:25:31 -04:00
Owner
Copy Link
Copy Source

Migrated from LOTUS_TODO.md on 2026-09-17 (the file is now reference-only).

🧨 Encryption / E2EE — ⚠️ EXTREME COMPLEXITY · 🧠 PLANNING SESSION REQUIRED

Observed live in prod 2026-06-30 during a 2-person Element Call (E2EE). These span client rust-crypto (matrix-js-sdk@41.7.0) ↔ Synapse ↔ EC MatrixRTC E2EE and are interrelated — do NOT spot-fix. Capture first: run Settings → Developer Tools → Crypto Diagnostics during the next affected call + a synapse-side trace before any fix. (Full runbook was in LOTUS_E2EE_INVESTIGATION.md, now in git history.) None are caused by the EC fork work.

  • KE-1 — OTK upload conflict storm (CRITICAL, root-cause candidate). POST /keys/upload returns 400 M_UNKNOWN: One time key … already exists continuously — the rust-crypto store and Synapse have diverged OTK state (upstream matrix-rust-sdk#5200, OPEN: on the 400 the SDK never marks the request sent → re-uploads forever; not fixed in 41.7.0). Leading web trigger: cinny never calls navigator.storage.persist(), so the IndexedDB crypto store is evictable while the localStorage session survives → device resurrects with a blank store. Buildable preventive fix (no call needed): request persistent storage on login (+ optional multi-tab guard + a 400-loop→recovery prompt). Healing an already-diverged device still needs a clean logout+login.
  • KE-2 — EC media keys not arriving/decrypting → audio/video cut out (CRITICAL). MissingKey … for participant, unexpected encrypted to-device io.element.call.encryption_keys. Almost certainly downstream of KE-1 (broken Olm sessions). This is the "friend's audio cuts out" symptom.
  • KE-3 — Timeline decrypt error: missing algorithm field (HIGH). rust-crypto can't parse a malformed/legacy encrypted event — capture the offending event id + raw content.
  • KE-4 — MatrixRTC delayed-event / membership timeouts (MEDIUM-HIGH). Restart delayed event timed out, repeated msc4157.update_delayed_event — may be partly HS responsiveness; correlate with synapse latency. Same planning session (shares the call-reliability surface).

Manual capture steps: [QA] #184 (§O8).

_Migrated from `LOTUS_TODO.md` on 2026-09-17 (the file is now reference-only)._ ### 🧨 Encryption / E2EE — ⚠️ EXTREME COMPLEXITY · 🧠 PLANNING SESSION REQUIRED Observed live in prod 2026-06-30 during a 2-person **Element Call** (E2EE). These span client rust-crypto (`matrix-js-sdk@41.7.0`) ↔ Synapse ↔ EC MatrixRTC E2EE and are **interrelated** — do NOT spot-fix. **Capture first:** run **Settings → Developer Tools → Crypto Diagnostics** during the next affected call + a synapse-side trace before any fix. (Full runbook was in `LOTUS_E2EE_INVESTIGATION.md`, now in git history.) None are caused by the EC fork work. - **KE-1 — OTK upload conflict storm (CRITICAL, root-cause candidate).** `POST /keys/upload` returns `400 M_UNKNOWN: One time key … already exists` continuously — the rust-crypto store and Synapse have **diverged OTK state** (upstream `matrix-rust-sdk#5200`, OPEN: on the 400 the SDK never marks the request sent → re-uploads forever; **not** fixed in 41.7.0). Leading web trigger: cinny never calls **`navigator.storage.persist()`**, so the IndexedDB crypto store is evictable while the `localStorage` session survives → device resurrects with a blank store. **Buildable preventive fix (no call needed):** request persistent storage on login (+ optional multi-tab guard + a 400-loop→recovery prompt). Healing an already-diverged device still needs a clean logout+login. - **KE-2 — EC media keys not arriving/decrypting → audio/video cut out (CRITICAL).** `MissingKey … for participant`, unexpected encrypted to-device `io.element.call.encryption_keys`. Almost certainly downstream of KE-1 (broken Olm sessions). This is the "friend's audio cuts out" symptom. - **KE-3 — Timeline decrypt error: missing `algorithm` field (HIGH).** rust-crypto can't parse a malformed/legacy encrypted event — capture the offending event id + raw content. - **KE-4 — MatrixRTC delayed-event / membership timeouts (MEDIUM-HIGH).** `Restart delayed event timed out`, repeated `msc4157.update_delayed_event` — may be partly HS responsiveness; correlate with synapse latency. Same planning session (shares the call-reliability surface). Manual capture steps: [QA] #184 (§O8).
jared added the priority: highsecurityarea: callsneeds-human-reviewplanning labels 2026-09-17 23:25:31 -04:00
jared commented 2026-10-24 01:08:48 -04:00
Author
Owner
Copy Link
Copy Source

Investigation, 2026-10-24. Owner decision: build the server alert, a recovery prompt, and the SDK upgrade. Don't spot-fix the crypto.

Evidence (Prometheus synapse_http_server_response_time_seconds_count; the Synapse logs only keep 4 days)

  • KE-1 was real and large: 474k POST /keys/upload → 400 between 2026-05-30 and 2026-07-08, peaking at 63k a day (1,600–3,300 an hour).
    • It's been zero since 2026-07-09, apart from 95 on 2026-09-15; the device can't be identified any more.
    • Upstream matrix-rust-sdk#5200 (retries forever) is still open. Its maintainers' fix is "don't lose the crypto store", which the storage.persist() request already covers.
  • KE-4: delayed-event 5xx are rare (worst hour in 100 days: 37; spikes of 77 and 39 in mid-August). The last 4 days: all ~21.6k succeeded.
  • KE-2: a healthy local two-person E2EE call logs "skipping decryption due to missing key at index 0" and "Received an unexpected encrypted to-device event" at call start, on both the old and new Element Call. These signatures alone are startup noise, not proof of the audio-cut-out bug.

Done

  • Grafana alerts (Matrix → Synapse Health → Matrix Notifications), live:
    • E2EE Key Upload Conflicts: more than 30 per hour, for 15 minutes; the description says how to find the device;
    • Call Delayed-Event Errors: more than 20 5xx per hour.
  • #446, recovery prompt (merged): the app watches /keys/upload. After 3 conflicts in 15 minutes it shows "Encryption on this device is out of sync" → Fix: sign out… (the logout dialog warns about verification and backup first).
  • #447, matrix-js-sdk 41.7 → 44.0 (in CI): crypto-wasm 18.9 and the OAuth port. The upgrade path 41.7 → 44 was tested: same device, old messages still decrypt.
  • element-call #61, upstream v0.25 → v0.28 (in CI): tested in cinny with a real LiveKit, E2EE both ways, and mixed versions both directions.

Still open: KE-3 (malformed algorithm) has no reproduction yet. Leave this issue open for the next real sighting; the alerts now capture KE-1 and KE-4 automatically.

🤖 Generated with Claude Code

**Investigation, 2026-10-24.** Owner decision: build the server alert, a recovery prompt, and the SDK upgrade. Don't spot-fix the crypto. **Evidence** (Prometheus `synapse_http_server_response_time_seconds_count`; the Synapse logs only keep 4 days) - **KE-1 was real and large:** **474k** `POST /keys/upload` → 400 between **2026-05-30 and 2026-07-08**, peaking at 63k a day (1,600–3,300 an hour). - It's been zero since 2026-07-09, apart from **95 on 2026-09-15**; the device can't be identified any more. - Upstream matrix-rust-sdk#5200 (retries forever) is still open. Its maintainers' fix is "don't lose the crypto store", which the `storage.persist()` request already covers. - **KE-4:** delayed-event 5xx are rare (worst hour in 100 days: 37; spikes of 77 and 39 in mid-August). The last 4 days: all ~21.6k succeeded. - **KE-2:** a healthy local two-person E2EE call logs "skipping decryption due to missing key at index 0" and "Received an unexpected encrypted to-device event" at call start, on both the old and new Element Call. **These signatures alone are startup noise, not proof of the audio-cut-out bug.** **Done** - **Grafana alerts** (Matrix → Synapse Health → Matrix Notifications), live: - *E2EE Key Upload Conflicts:* more than 30 per hour, for 15 minutes; the description says how to find the device; - *Call Delayed-Event Errors:* more than 20 5xx per hour. - **#446, recovery prompt** (merged): the app watches `/keys/upload`. After 3 conflicts in 15 minutes it shows "Encryption on this device is out of sync" → **Fix: sign out…** (the logout dialog warns about verification and backup first). - **#447, matrix-js-sdk 41.7 → 44.0** (in CI): crypto-wasm 18.9 and the OAuth port. The upgrade path 41.7 → 44 was tested: same device, old messages still decrypt. - **element-call #61, upstream v0.25 → v0.28** (in CI): tested in cinny with a real LiveKit, E2EE both ways, and mixed versions both directions. **Still open:** KE-3 (malformed `algorithm`) has no reproduction yet. Leave this issue open for the next real sighting; the alerts now capture KE-1 and KE-4 automatically. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
a11y
Accessibility: keyboard, screen reader, contrast, motion
 area: appearance
Client area: appearance
 area: auth-session
Client area: auth-session
 area: build-ci
Client area: build-ci
 area: calls
Client area: calls
 area: desktop
Client area: desktop
 area: media
Client area: media
 area: messaging
Client area: messaging
 area: mobile
Client area: mobile
 area: moderation
Client area: moderation
 area: navigation
Client area: navigation
 area: notifications
Client area: notifications
 area: settings
Client area: settings
 area: threads
Client area: threads
 bug
Something is not working
 dependencies
Third-party package versions and advisories
 docs
README / LOTUS_* docs wrong or missing
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 needs-human-review
performance
Re-render storms, leaks, heavy work on hot paths
 planning
priority: critical
Data loss, security hole, or crash on a main path
 priority: high
Broken feature or serious usability problem
 priority: low
Minor issue or polish
 priority: medium
Wrong behaviour in an edge case or notable degradation
 qa
Manual QA: shipped, needs a human in a real environment
 question
More information is needed
 research
security
XSS, unsafe URLs, data leaks, auth/session
 tech-debt
Code health, dead code, fragile patterns
 ux
Usability or visual inconsistency
 wontfix
This won't be fixed
No labels area: calls needs-human-review planning priority: high security
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
cole (Cole Keller) jared (Jared Vititoe) nathan (Nathan Vititoe)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: LotusGuild/cinny#201