Clock-skew warning: compare the homeserver Date header to the local clock and warn once when it is far off #158

Closed
opened 2026-09-17 22:53:32 -04:00 by jared · 1 comment
Owner

Incident 2026-09-17: a wrong Windows clock made every MatrixRTC call membership look expired locally (own membership "missing → force re-join", "No targets found for sending key", peers' media keys rejected) while Synapse was returning 200 for everything. Nothing in the UI hinted at the cause. Encryption (Olm/Megolm key validity, key backup), delayed events and LiveKit JWTs are all time-sensitive too.

Design ("done properly")

  • Measurement: read the Date response header from ordinary homeserver requests the client already makes (/sync, /versions) — no extra requests. Compute skew = Date.now() − serverDate − (roundTrip / 2) on a fast request only (RTT < 2 s) to avoid mistaking latency for skew; take the median of the last 3 samples before acting. Re-sample on startup and after each reconnect/SyncState PREPARED.
  • Threshold: warn at |skew| > 30 s (calls start failing around a membership-expiry window; Megolm/OTK problems start well before minutes). Hysteresis: clear at < 15 s.
  • UI: one banner (same slot/style as the sync-error / unverified-device banners): "Your computer's clock is 14 minutes ahead of the server. Encrypted messages and voice calls will fail until it is fixed." with a How to fix link (OS-specific one-liner: Windows → Settings → Time & language → Sync now; macOS/Linux equivalents) and Dismiss for 24 h. Direction ("ahead"/"behind") matters — say it.
  • In-call: if a call is joined while skew is over the threshold, show the same text in the call-status area, since that is where the user is looking.
  • Never auto-correct anything, never touch timestamps in messages; this is a warning only. No setting (a wrong clock breaks E2EE — everyone needs to know).
  • Tests: pure helper utils/clockSkew.ts (sample filtering, median, hysteresis, direction wording) unit-tested; the header parse tolerates a missing/invalid header (no warning, no throw).
  • Also consider surfacing the measured skew in Crypto Diagnostics so support conversations can see it.
Incident 2026-09-17: a wrong Windows clock made every MatrixRTC call membership look expired locally (own membership "missing → force re-join", "No targets found for sending key", peers' media keys rejected) while Synapse was returning 200 for everything. Nothing in the UI hinted at the cause. Encryption (Olm/Megolm key validity, key backup), delayed events and LiveKit JWTs are all time-sensitive too. ### Design ("done properly") - **Measurement:** read the `Date` response header from ordinary homeserver requests the client already makes (`/sync`, `/versions`) — no extra requests. Compute `skew = Date.now() − serverDate − (roundTrip / 2)` on a fast request only (RTT < 2 s) to avoid mistaking latency for skew; take the median of the last 3 samples before acting. Re-sample on startup and after each reconnect/`SyncState` PREPARED. - **Threshold:** warn at |skew| > 30 s (calls start failing around a membership-expiry window; Megolm/OTK problems start well before minutes). Hysteresis: clear at < 15 s. - **UI:** one banner (same slot/style as the sync-error / unverified-device banners): "Your computer's clock is **14 minutes ahead** of the server. Encrypted messages and voice calls will fail until it is fixed." with a *How to fix* link (OS-specific one-liner: Windows → Settings → Time & language → Sync now; macOS/Linux equivalents) and *Dismiss for 24 h*. Direction ("ahead"/"behind") matters — say it. - **In-call:** if a call is joined while skew is over the threshold, show the same text in the call-status area, since that is where the user is looking. - **Never** auto-correct anything, never touch timestamps in messages; this is a warning only. No setting (a wrong clock breaks E2EE — everyone needs to know). - **Tests:** pure helper `utils/clockSkew.ts` (sample filtering, median, hysteresis, direction wording) unit-tested; the header parse tolerates a missing/invalid header (no warning, no throw). - Also consider surfacing the measured skew in Crypto Diagnostics so support conversations can see it.
jared added this to the Features 2026-Q4 milestone 2026-09-17 22:53:32 -04:00
jared added the enhancementpriority: mediumuxarea: callsarea: auth-session labels 2026-09-17 22:53:32 -04:00
jared self-assigned this 2026-09-17 22:53:32 -04:00
Author
Owner

Shipped in 84c906fe, with one deliberate deviation from the design's measurement source:

Measurement — not the Date header. Synapse's CORS policy only exposes Synapse-Trace-Id, Server, ETag, so Date is unreadable from a different origin (chat. vs matrix.) without touching nginx. Instead every live event already carries the answer: origin_server_ts (origin server's stamp) + unsigned.age (OUR server's now − ts when it built the /sync response) = our server's clock at response time, and matrix-js-sdk already computes localTimestamp = Date.now() − age at event construction, so a sample is just localTimestamp − origin_server_ts (download latency = tens of ms of noise; Date has 1 s granularity anyway). No extra requests, no headers, no RTT filtering needed — and it works for federated events too, since age is always computed by our server. Only RoomEvent.Timeline live events count; the SDK already flags cache replays (stale age) as liveEvent=false. The initial network sync qualifies, so a wrong clock is flagged within seconds of startup.

Thresholds as designed: median of the last 5 samples (≥3), warn at |skew| > 30 s, clear below 15 s.

UI: banner in the sync-status slot with direction ("14 minutes ahead of the server" / "3 hours behind the server"), How to fix expands a per-OS one-liner (Windows/macOS/Linux/iOS/Android detected from the UA), Dismiss for 24 h (localStorage). Same line in the call status bar while in a call. Nothing is auto-corrected.
banner
call bar

Verified headless with Playwright's clock skewed +14 min and −3 h (banner, hint, in-call line, dismiss) and in sync (nothing shown); 6 unit tests on the monitor and wording.

Shipped in `84c906fe`, with one deliberate deviation from the design's measurement source: **Measurement** — not the `Date` header. Synapse's CORS policy only exposes `Synapse-Trace-Id, Server, ETag`, so `Date` is unreadable from a different origin (chat. vs matrix.) without touching nginx. Instead every live event already carries the answer: `origin_server_ts` (origin server's stamp) + `unsigned.age` (OUR server's `now − ts` when it built the `/sync` response) = our server's clock at response time, and matrix-js-sdk already computes `localTimestamp = Date.now() − age` at event construction, so a sample is just `localTimestamp − origin_server_ts` (download latency = tens of ms of noise; `Date` has 1 s granularity anyway). No extra requests, no headers, no RTT filtering needed — and it works for federated events too, since `age` is always computed by *our* server. Only `RoomEvent.Timeline` live events count; the SDK already flags cache replays (stale `age`) as `liveEvent=false`. The initial network sync qualifies, so a wrong clock is flagged within seconds of startup. **Thresholds** as designed: median of the last 5 samples (≥3), warn at |skew| > 30 s, clear below 15 s. **UI**: banner in the sync-status slot with direction ("14 minutes ahead of the server" / "3 hours behind the server"), *How to fix* expands a per-OS one-liner (Windows/macOS/Linux/iOS/Android detected from the UA), *Dismiss for 24 h* (localStorage). Same line in the call status bar while in a call. Nothing is auto-corrected. ![banner](https://code.lotusguild.org/attachments/3cad7463-8b2d-4b1b-9136-00771e143bf9) ![call bar](https://code.lotusguild.org/attachments/3875a58d-413f-4d06-bd8d-e7505519bee4) Verified headless with Playwright's clock skewed +14 min and −3 h (banner, hint, in-call line, dismiss) and in sync (nothing shown); 6 unit tests on the monitor and wording.
jared closed this issue 2026-09-19 12:57:06 -04:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: LotusGuild/cinny#158