feat(e2ee): tell the user when their device's key uploads are stuck (#201 KE-1) #446

Merged
jared merged 1 commits from feat/key-upload-recovery into lotus 2026-10-24 00:11:29 -04:00
Owner

Part of #201 (KE-1). Owner decision 2026-10-24: build the server alert, a recovery prompt, and the SDK upgrade.

Evidence (Prometheus, synapse_http_server_response_time_seconds_count{servlet="KeyUploadServlet",code="400"}):

  • 474k rejected key uploads between 2026-05-30 and 2026-07-08, peaking at 63k a day (1,600–3,300 an hour);
  • zero since, except 95 on 2026-09-15.

The Synapse logs only go back 4 days, so the device behind the September blip can't be identified any more.

What this does

  • keyUploadWatch observes /keys/upload responses via the client's fetchFn. The SDK's response is untouched; a clone is read in the background.
    • Stuck: 3 or more 400 "One time key … already exists" within 15 minutes.
    • Cleared: any successful upload.
    • Not based on log wording.
  • KeyUploadBanner: a critical strip in the same slot as the sync and clock banners.
    • Says "Encryption on this device is out of sync with the server…".
    • Details gives the plain-language cause and says to check Settings → Devices (verification and backup) first.
    • Fix: sign out… opens the normal logout dialog, which already warns if the device isn't verified or backed up.
    • Hide lasts for this session only.

Server side (already live): two Grafana alerts in Matrix → Synapse Health, routed to Matrix Notifications:

  • E2EE Key Upload Conflicts: more than 30 per hour, for 15 minutes;
  • Call Delayed-Event Errors: more than 20 5xx per hour.

Tests

  • 5 unit tests: endpoint/body matching, the threshold, ageing out, clearing, and the fetch wrapper leaving the SDK's response readable.
  • Playwright against a local Synapse:
    • with /keys/upload forced to the conflict 400, the banner appears after 3 rejections, Details shows the cause, and Fix opens the logout dialog;
    • a healthy login never shows it.
    • The first run caught a crash: the banner sat outside MatrixClientProvider. Fixed.
  • All 1,423 unit tests pass; lint is at 29.

🤖 Generated with Claude Code

https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5

Part of #201 (KE-1). Owner decision 2026-10-24: build the server alert, a recovery prompt, and the SDK upgrade. **Evidence** (Prometheus, `synapse_http_server_response_time_seconds_count{servlet="KeyUploadServlet",code="400"}`): - **474k** rejected key uploads between 2026-05-30 and 2026-07-08, peaking at 63k a day (1,600–3,300 an hour); - zero since, except 95 on 2026-09-15. The Synapse logs only go back 4 days, so the device behind the September blip can't be identified any more. **What this does** - `keyUploadWatch` observes `/keys/upload` responses via the client's `fetchFn`. The SDK's response is untouched; a clone is read in the background. - **Stuck:** 3 or more `400 "One time key … already exists"` within 15 minutes. - **Cleared:** any successful upload. - Not based on log wording. - **`KeyUploadBanner`:** a critical strip in the same slot as the sync and clock banners. - Says "Encryption on this device is out of sync with the server…". - **Details** gives the plain-language cause and says to check Settings → Devices (verification and backup) first. - **Fix: sign out…** opens the normal logout dialog, which already warns if the device isn't verified or backed up. - **Hide** lasts for this session only. **Server side (already live):** two Grafana alerts in Matrix → Synapse Health, routed to Matrix Notifications: - *E2EE Key Upload Conflicts:* more than 30 per hour, for 15 minutes; - *Call Delayed-Event Errors:* more than 20 5xx per hour. **Tests** - 5 unit tests: endpoint/body matching, the threshold, ageing out, clearing, and the fetch wrapper leaving the SDK's response readable. - Playwright against a local Synapse: - with `/keys/upload` forced to the conflict 400, the banner appears after 3 rejections, Details shows the cause, and **Fix** opens the logout dialog; - a healthy login never shows it. - The first run caught a crash: the banner sat outside `MatrixClientProvider`. Fixed. - All 1,423 unit tests pass; lint is at 29. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5
jared added 1 commit 2026-10-23 23:51:19 -04:00
feat(e2ee): tell the user when their device's key uploads are stuck (#201 KE-1)
CI / Build & Quality Checks (push) Skipped
CI / Secret scan (gitleaks) (push) Skipped
CI / Playwright smoke (e2e) (push) Skipped
CI / Build & Quality Checks (pull_request) Successful in 3m41s
CI / Trigger Desktop Build (pull_request) Skipped
CI / Docker image build & smoke test (pull_request) Skipped
CI / Secret scan (gitleaks) (pull_request) Successful in 8s
CI / Playwright smoke (e2e) (pull_request) Successful in 15m54s
c2c43eda99
When the server keeps rejecting this device's /keys/upload with 400 "One
time key ... already exists", the device's encryption store has diverged
and the SDK retries forever (matrix-rust-sdk#5200, still open). Synapse
saw 474k of these from May 30 to Jul 8 2026. The fix is a fresh login, so:

- keyUploadWatch observes /keys/upload responses through the client's
  fetchFn (the response the SDK gets is untouched; a clone is read).
  Three conflicts within 15 min = stuck; any successful upload clears it.
- KeyUploadBanner (critical strip, same slot as the sync/clock banners):
  "Encryption on this device is out of sync" with Details and
  "Fix: sign out…", which opens the normal logout dialog (it already
  warns when the device isn't verified / backed up).

Server side, Grafana now alerts on the same signal (E2EE Key Upload
Conflicts) and on delayed-event 5xx (Call Delayed-Event Errors).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5
jared merged commit bfed37bf50 into lotus 2026-10-24 00:11:29 -04:00
Sign in to join this conversation.
No Reviewers
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: LotusGuild/cinny#446