LotusGuild/cinny
3
0
Fork 0
Code Issues 27 Pull Requests 5 Actions Packages Projects Releases Wiki Activity

Clock-skew warning: compare the homeserver Date header to the local clock and warn once when it is far off #158

New Issue
Closed
opened 2026-09-17 22:53:32 -04:00 by jared · 1 comment
jared commented 2026-09-17 22:53:32 -04:00
Owner
Copy Link
Copy Source

Incident 2026-09-17: a wrong Windows clock made every MatrixRTC call membership look expired locally (own membership "missing → force re-join", "No targets found for sending key", peers' media keys rejected) while Synapse was returning 200 for everything. Nothing in the UI hinted at the cause. Encryption (Olm/Megolm key validity, key backup), delayed events and LiveKit JWTs are all time-sensitive too.

Design ("done properly")

  • Measurement: read the Date response header from ordinary homeserver requests the client already makes (/sync, /versions) — no extra requests. Compute skew = Date.now() − serverDate − (roundTrip / 2) on a fast request only (RTT < 2 s) to avoid mistaking latency for skew; take the median of the last 3 samples before acting. Re-sample on startup and after each reconnect/SyncState PREPARED.
  • Threshold: warn at |skew| > 30 s (calls start failing around a membership-expiry window; Megolm/OTK problems start well before minutes). Hysteresis: clear at < 15 s.
  • UI: one banner (same slot/style as the sync-error / unverified-device banners): "Your computer's clock is 14 minutes ahead of the server. Encrypted messages and voice calls will fail until it is fixed." with a How to fix link (OS-specific one-liner: Windows → Settings → Time & language → Sync now; macOS/Linux equivalents) and Dismiss for 24 h. Direction ("ahead"/"behind") matters — say it.
  • In-call: if a call is joined while skew is over the threshold, show the same text in the call-status area, since that is where the user is looking.
  • Never auto-correct anything, never touch timestamps in messages; this is a warning only. No setting (a wrong clock breaks E2EE — everyone needs to know).
  • Tests: pure helper utils/clockSkew.ts (sample filtering, median, hysteresis, direction wording) unit-tested; the header parse tolerates a missing/invalid header (no warning, no throw).
  • Also consider surfacing the measured skew in Crypto Diagnostics so support conversations can see it.
Incident 2026-09-17: a wrong Windows clock made every MatrixRTC call membership look expired locally (own membership "missing → force re-join", "No targets found for sending key", peers' media keys rejected) while Synapse was returning 200 for everything. Nothing in the UI hinted at the cause. Encryption (Olm/Megolm key validity, key backup), delayed events and LiveKit JWTs are all time-sensitive too. ### Design ("done properly") - **Measurement:** read the `Date` response header from ordinary homeserver requests the client already makes (`/sync`, `/versions`) — no extra requests. Compute `skew = Date.now() − serverDate − (roundTrip / 2)` on a fast request only (RTT < 2 s) to avoid mistaking latency for skew; take the median of the last 3 samples before acting. Re-sample on startup and after each reconnect/`SyncState` PREPARED. - **Threshold:** warn at |skew| > 30 s (calls start failing around a membership-expiry window; Megolm/OTK problems start well before minutes). Hysteresis: clear at < 15 s. - **UI:** one banner (same slot/style as the sync-error / unverified-device banners): "Your computer's clock is **14 minutes ahead** of the server. Encrypted messages and voice calls will fail until it is fixed." with a *How to fix* link (OS-specific one-liner: Windows → Settings → Time & language → Sync now; macOS/Linux equivalents) and *Dismiss for 24 h*. Direction ("ahead"/"behind") matters — say it. - **In-call:** if a call is joined while skew is over the threshold, show the same text in the call-status area, since that is where the user is looking. - **Never** auto-correct anything, never touch timestamps in messages; this is a warning only. No setting (a wrong clock breaks E2EE — everyone needs to know). - **Tests:** pure helper `utils/clockSkew.ts` (sample filtering, median, hysteresis, direction wording) unit-tested; the header parse tolerates a missing/invalid header (no warning, no throw). - Also consider surfacing the measured skew in Crypto Diagnostics so support conversations can see it.
clock-skew-banner.png
276 KiB
clock-skew-call-bar.png
14 KiB
clock-skew-banner.png clock-skew-call-bar.png
jared added this to the Features 2026-Q4 milestone 2026-09-17 22:53:32 -04:00
jared added the enhancementpriority: mediumuxarea: callsarea: auth-session labels 2026-09-17 22:53:32 -04:00
jared self-assigned this 2026-09-17 22:53:32 -04:00
jared commented 2026-09-19 12:57:06 -04:00
Author
Owner
Copy Link
Copy Source

Shipped in 84c906fe, with one deliberate deviation from the design's measurement source:

Measurement — not the Date header. Synapse's CORS policy only exposes Synapse-Trace-Id, Server, ETag, so Date is unreadable from a different origin (chat. vs matrix.) without touching nginx. Instead every live event already carries the answer: origin_server_ts (origin server's stamp) + unsigned.age (OUR server's now − ts when it built the /sync response) = our server's clock at response time, and matrix-js-sdk already computes localTimestamp = Date.now() − age at event construction, so a sample is just localTimestamp − origin_server_ts (download latency = tens of ms of noise; Date has 1 s granularity anyway). No extra requests, no headers, no RTT filtering needed — and it works for federated events too, since age is always computed by our server. Only RoomEvent.Timeline live events count; the SDK already flags cache replays (stale age) as liveEvent=false. The initial network sync qualifies, so a wrong clock is flagged within seconds of startup.

Thresholds as designed: median of the last 5 samples (≥3), warn at |skew| > 30 s, clear below 15 s.

UI: banner in the sync-status slot with direction ("14 minutes ahead of the server" / "3 hours behind the server"), How to fix expands a per-OS one-liner (Windows/macOS/Linux/iOS/Android detected from the UA), Dismiss for 24 h (localStorage). Same line in the call status bar while in a call. Nothing is auto-corrected.
banner
call bar

Verified headless with Playwright's clock skewed +14 min and −3 h (banner, hint, in-call line, dismiss) and in sync (nothing shown); 6 unit tests on the monitor and wording.

Shipped in `84c906fe`, with one deliberate deviation from the design's measurement source: **Measurement** — not the `Date` header. Synapse's CORS policy only exposes `Synapse-Trace-Id, Server, ETag`, so `Date` is unreadable from a different origin (chat. vs matrix.) without touching nginx. Instead every live event already carries the answer: `origin_server_ts` (origin server's stamp) + `unsigned.age` (OUR server's `now − ts` when it built the `/sync` response) = our server's clock at response time, and matrix-js-sdk already computes `localTimestamp = Date.now() − age` at event construction, so a sample is just `localTimestamp − origin_server_ts` (download latency = tens of ms of noise; `Date` has 1 s granularity anyway). No extra requests, no headers, no RTT filtering needed — and it works for federated events too, since `age` is always computed by *our* server. Only `RoomEvent.Timeline` live events count; the SDK already flags cache replays (stale `age`) as `liveEvent=false`. The initial network sync qualifies, so a wrong clock is flagged within seconds of startup. **Thresholds** as designed: median of the last 5 samples (≥3), warn at |skew| > 30 s, clear below 15 s. **UI**: banner in the sync-status slot with direction ("14 minutes ahead of the server" / "3 hours behind the server"), *How to fix* expands a per-OS one-liner (Windows/macOS/Linux/iOS/Android detected from the UA), *Dismiss for 24 h* (localStorage). Same line in the call status bar while in a call. Nothing is auto-corrected. ![banner](https://code.lotusguild.org/attachments/3cad7463-8b2d-4b1b-9136-00771e143bf9) ![call bar](https://code.lotusguild.org/attachments/3875a58d-413f-4d06-bd8d-e7505519bee4) Verified headless with Playwright's clock skewed +14 min and −3 h (banner, hint, in-call line, dismiss) and in sync (nothing shown); 6 unit tests on the monitor and wording.
jared closed this issue 2026-09-19 12:57:06 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
a11y
Accessibility: keyboard, screen reader, contrast, motion
 area: appearance
Client area: appearance
 area: auth-session
Client area: auth-session
 area: build-ci
Client area: build-ci
 area: calls
Client area: calls
 area: desktop
Client area: desktop
 area: media
Client area: media
 area: messaging
Client area: messaging
 area: mobile
Client area: mobile
 area: moderation
Client area: moderation
 area: navigation
Client area: navigation
 area: notifications
Client area: notifications
 area: settings
Client area: settings
 area: threads
Client area: threads
 bug
Something is not working
 dependencies
Third-party package versions and advisories
 docs
README / LOTUS_* docs wrong or missing
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 needs-human-review
performance
Re-render storms, leaks, heavy work on hot paths
 planning
priority: critical
Data loss, security hole, or crash on a main path
 priority: high
Broken feature or serious usability problem
 priority: low
Minor issue or polish
 priority: medium
Wrong behaviour in an edge case or notable degradation
 qa
Manual QA: shipped, needs a human in a real environment
 question
More information is needed
 research
security
XSS, unsafe URLs, data leaks, auth/session
 tech-debt
Code health, dead code, fragile patterns
 ux
Usability or visual inconsistency
 wontfix
This won't be fixed
No labels area: auth-session area: calls enhancement priority: medium ux
Milestone
No items
No Milestone Features 2026-Q4
Projects
Clear projects
No projects
Assignees
Clear assignees
cole (Cole Keller) jared (Jared Vititoe) nathan (Nathan Vititoe)
No Assignees jared
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: LotusGuild/cinny#158