Files
cinny/src/app/utils/clockSkew.ts
T
Lotus CIandClaude Opus 5.5 bb569d69a2
CI / Build & Quality Checks (pull_request) Successful in 3m1s
CI / Trigger Desktop Build (pull_request) Skipped
CI / Docker image build & smoke test (pull_request) Skipped
CI / Secret scan (gitleaks) (pull_request) Successful in 7s
CI / Playwright smoke (e2e) (pull_request) Failing after 11m28s
fix: a stalled server no longer reads as "your clock is ahead"
Incident 2026-09-29: the homeserver's host ran out of memory and stalled for
~2 minutes. The /sync that finally went out carried events whose `age` was
computed ~30 s before it arrived, so every client showed "Your computer's
clock is 30 seconds ahead of the server" while the real problem was the
server (all host clocks were within 0.25 s the whole evening).

The skew estimate was the median of the last 5 samples, and a sample is
local skew + delivery delay, so one late /sync with a handful of events
tripped it.

- Estimate = the LOWEST sample of the last 5 minutes: delay only ever adds,
  so the fastest-delivered event is the truest.
- "Behind" (which a delay can't cause) is reported as soon as there are 3
  samples, like before. "Ahead" must hold across samples received at least
  a minute apart, so a single late burst never trips it.
- Samples are aged on the monotonic clock, and a change of the local clock
  (someone fixing it) resets the measurement, so the warning clears at once.
- Only events stamped by our own homeserver are sampled: a federated event's
  origin_server_ts is the other server's clock.
- Wording: "This device's clock is … Voice calls and encrypted messages can
  fail until it's corrected." / call bar "Device clock … : calls may fail"
  (was "will fail").

Unit tests: the incident (late burst after normal traffic, and a fresh
client whose first samples are all late), mixed slow/fast deliveries,
ahead only after a minute, behind at once, hysteresis, clock fixed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PPmy3tPq869XDW4njjVaKA
2026-09-28 21:40:02 -04:00

191 lines
8.1 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
/**
* [Gitea #158] Local-clock-vs-homeserver skew detection.
*
* Incident 2026-09-17: a wrong Windows clock made every MatrixRTC membership
* look expired locally (calls failed, media keys rejected) while the server
* answered 200 to everything, and nothing in the UI hinted at the cause.
*
* Measurement needs no extra requests and no CORS-exposed headers: every event
* a `/sync` delivers carries `origin_server_ts` (stamped by its origin server)
* and `unsigned.age` (= OUR server's `now − origin_server_ts` at the moment it
* built the response). So `origin_server_ts + age` is our server's clock at
* response time, and `receivedAt − (origin_server_ts + age)` is our skew plus
* the download latency (tens of ms; ignored). matrix-js-sdk already computes
* `localTimestamp = Date.now() − age` at event construction, so a sample is
* simply `localTimestamp − origin_server_ts`.
*
* Only LIVE events count (`RoomEvent.Timeline` data.liveEvent, which the SDK
* already sets false for events replayed from the IndexedDB cache — those carry
* a stale `age` that would read as hours of skew). The initial network sync's
* events qualify, so a wrong clock is flagged within seconds of startup.
*/
export const SKEW_WARN_MS = 30_000;
export const SKEW_CLEAR_MS = 15_000;
export const SKEW_MIN_SAMPLES = 3;
/** Samples older than this are forgotten. */
export const SKEW_WINDOW_MS = 5 * 60 * 1000;
export const SKEW_MAX_SAMPLES = 30;
/**
* "Ahead" must hold across samples received at least this far apart.
*
* Incident 2026-09-29: the homeserver's host ran out of memory and stalled for
* ~2 minutes; the /sync that finally went out carried events whose `age` was
* computed ~30 s before it arrived, so every client read "your clock is 30 s
* ahead" — while the real problem was the server. A late delivery can only make
* the local clock look AHEAD (never behind), so the estimate is the LOWEST
* recent sample (the one delivered fastest), and "ahead" has to persist across
* a minute of fresh samples before it is reported. "Behind" can't come from a
* delay and is reported as soon as there are enough samples.
*/
export const SKEW_AHEAD_SPAN_MS = 60_000;
/**
* Sanity cap on `age`. Old events are still valid samples (the server computes
* `age` at response time, so `ts + age` is its clock regardless of the event's
* own age) — this only rejects garbage.
*/
export const SKEW_MAX_AGE_MS = 30 * 24 * 60 * 60 * 1000;
export type ClockSkewState = {
/** Median of the recent samples, ms; positive = local clock is AHEAD. */
skewMs: number | null;
/** Over the threshold (with hysteresis). */
warning: boolean;
};
/** A wall-clock change larger than this (vs the monotonic clock) resets the samples. */
export const CLOCK_JUMP_MS = 5_000;
type Sample = { skew: number; at: number };
const currentClock = (): { wall: number; mono: number } => {
const wall = Date.now();
const mono = typeof performance !== 'undefined' ? performance.now() : wall;
return { wall, mono };
};
export class ClockSkewMonitor {
private samples: Sample[] = [];
private clockOffset: number | undefined;
private state: ClockSkewState = { skewMs: null, warning: false };
private listeners = new Set<(state: ClockSkewState) => void>();
public getState(): ClockSkewState {
return this.state;
}
public subscribe(cb: (state: ClockSkewState) => void): () => void {
this.listeners.add(cb);
return () => {
this.listeners.delete(cb);
};
}
/**
* Feed one live event. `originServerTs` + `age` come from the event;
* `localTimestamp` is the SDK's `Date.now() − age` at construction; `clock`
* is when the sample was taken: wall clock and a monotonic clock
* (performance.now()), so samples are aged by real elapsed time and a change
* of the local clock (someone fixing it) starts the measurement afresh.
* Only feed events stamped by OUR homeserver: another server's
* `origin_server_ts` carries that server's clock.
* Returns the new state (unchanged object when nothing moved).
*/
public sample(
originServerTs: number,
age: number | undefined,
localTimestamp: number,
clock: { wall: number; mono: number } = currentClock(),
): ClockSkewState {
if (age === undefined || !Number.isFinite(age) || age < 0 || age > SKEW_MAX_AGE_MS) {
return this.state;
}
if (!Number.isFinite(originServerTs) || !Number.isFinite(localTimestamp)) return this.state;
const now = clock.mono;
const offset = clock.wall - clock.mono;
if (this.clockOffset !== undefined && Math.abs(offset - this.clockOffset) > CLOCK_JUMP_MS) {
// The local clock was changed: earlier samples measured the old clock.
this.reset();
}
this.clockOffset = offset;
this.samples.push({ skew: localTimestamp - originServerTs, at: now });
this.samples = this.samples.filter((s) => now - s.at <= SKEW_WINDOW_MS);
if (this.samples.length > SKEW_MAX_SAMPLES) this.samples.shift();
if (this.samples.length < SKEW_MIN_SAMPLES) return this.state;
// Delivery delay only ever adds to a sample: the smallest is the truest.
const skewMs = Math.min(...this.samples.map((s) => s.skew));
const abs = Math.abs(skewMs);
let warning: boolean;
if (this.state.warning) warning = abs >= SKEW_CLEAR_MS;
else if (skewMs < -SKEW_WARN_MS) warning = true;
else if (skewMs > SKEW_WARN_MS) {
// Ahead: only if the fastest-delivered samples stayed high for a minute.
const span = now - Math.min(...this.samples.map((s) => s.at));
warning = span >= SKEW_AHEAD_SPAN_MS;
} else warning = false;
if (skewMs === this.state.skewMs && warning === this.state.warning) return this.state;
this.state = { skewMs, warning };
this.listeners.forEach((cb) => cb(this.state));
return this.state;
}
public reset(): void {
this.samples = [];
this.clockOffset = undefined;
if (this.state.skewMs !== null || this.state.warning) {
this.state = { skewMs: null, warning: false };
this.listeners.forEach((cb) => cb(this.state));
}
}
}
/** "14 minutes ahead" / "2 hours behind" / "45 seconds ahead". */
export const formatSkew = (skewMs: number): string => {
const abs = Math.abs(skewMs);
const dir = skewMs > 0 ? 'ahead' : 'behind';
const unit = (n: number, word: string) => `${n} ${word}${n === 1 ? '' : 's'}`;
if (abs >= 36 * 60 * 60 * 1000)
return `${unit(Math.round(abs / (24 * 60 * 60 * 1000)), 'day')} ${dir}`;
if (abs >= 90 * 60 * 1000) return `${unit(Math.round(abs / (60 * 60 * 1000)), 'hour')} ${dir}`;
if (abs >= 90 * 1000) return `${unit(Math.round(abs / 60_000), 'minute')} ${dir}`;
return `${unit(Math.round(abs / 1000), 'second')} ${dir}`;
};
/** "14 minutes ahead of the server" / "3 hours behind the server". */
export const describeSkewVsServer = (skewMs: number): string =>
formatSkew(skewMs)
.replace(/ ahead$/, ' ahead of the server')
.replace(/ behind$/, ' behind the server');
export type ClockFixPlatform = 'windows' | 'mac' | 'linux' | 'ios' | 'android' | 'other';
export const detectClockFixPlatform = (ua: string): ClockFixPlatform => {
if (/iPhone|iPad|iPod/i.test(ua)) return 'ios';
if (/Android/i.test(ua)) return 'android';
if (/Windows/i.test(ua)) return 'windows';
if (/Mac OS X|Macintosh/i.test(ua)) return 'mac';
if (/Linux|X11/i.test(ua)) return 'linux';
return 'other';
};
export const clockFixHint = (platform: ClockFixPlatform): string => {
switch (platform) {
case 'windows':
return 'Windows: Settings → Time & language → Date & time → turn on "Set time automatically", then Sync now.';
case 'mac':
return 'macOS: System Settings → General → Date & Time → turn on "Set time and date automatically".';
case 'linux':
return "Linux: enable NTP (e.g. `timedatectl set-ntp true`) or your desktop's Date & Time → Automatic.";
case 'ios':
return 'iOS: Settings → General → Date & Time → Set Automatically.';
case 'android':
return 'Android: Settings → System → Date & time → Set time automatically.';
default:
return "Turn on automatic (network) time in your operating system's date & time settings.";
}
};