fix: a stalled server no longer reads as "your clock is ahead"
CI / Build & Quality Checks (pull_request) Successful in 3m1s
CI / Trigger Desktop Build (pull_request) Skipped
CI / Docker image build & smoke test (pull_request) Skipped
CI / Secret scan (gitleaks) (pull_request) Successful in 7s
CI / Playwright smoke (e2e) (pull_request) Failing after 11m28s
CI / Build & Quality Checks (pull_request) Successful in 3m1s
CI / Trigger Desktop Build (pull_request) Skipped
CI / Docker image build & smoke test (pull_request) Skipped
CI / Secret scan (gitleaks) (pull_request) Successful in 7s
CI / Playwright smoke (e2e) (pull_request) Failing after 11m28s
Incident 2026-09-29: the homeserver's host ran out of memory and stalled for ~2 minutes. The /sync that finally went out carried events whose `age` was computed ~30 s before it arrived, so every client showed "Your computer's clock is 30 seconds ahead of the server" while the real problem was the server (all host clocks were within 0.25 s the whole evening). The skew estimate was the median of the last 5 samples, and a sample is local skew + delivery delay, so one late /sync with a handful of events tripped it. - Estimate = the LOWEST sample of the last 5 minutes: delay only ever adds, so the fastest-delivered event is the truest. - "Behind" (which a delay can't cause) is reported as soon as there are 3 samples, like before. "Ahead" must hold across samples received at least a minute apart, so a single late burst never trips it. - Samples are aged on the monotonic clock, and a change of the local clock (someone fixing it) resets the measurement, so the warning clears at once. - Only events stamped by our own homeserver are sampled: a federated event's origin_server_ts is the other server's clock. - Wording: "This device's clock is … Voice calls and encrypted messages can fail until it's corrected." / call bar "Device clock … : calls may fail" (was "will fail"). Unit tests: the incident (late burst after normal traffic, and a fresh client whose first samples are all late), mixed slow/fast deliveries, ahead only after a minute, behind at once, hysteresis, clock fixed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PPmy3tPq869XDW4njjVaKA
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
c0c93213c1
commit
bb569d69a2
+56
-11
@@ -22,8 +22,23 @@
|
||||
|
||||
export const SKEW_WARN_MS = 30_000;
|
||||
export const SKEW_CLEAR_MS = 15_000;
|
||||
export const SKEW_SAMPLES = 5;
|
||||
export const SKEW_MIN_SAMPLES = 3;
|
||||
/** Samples older than this are forgotten. */
|
||||
export const SKEW_WINDOW_MS = 5 * 60 * 1000;
|
||||
export const SKEW_MAX_SAMPLES = 30;
|
||||
/**
|
||||
* "Ahead" must hold across samples received at least this far apart.
|
||||
*
|
||||
* Incident 2026-09-29: the homeserver's host ran out of memory and stalled for
|
||||
* ~2 minutes; the /sync that finally went out carried events whose `age` was
|
||||
* computed ~30 s before it arrived, so every client read "your clock is 30 s
|
||||
* ahead" — while the real problem was the server. A late delivery can only make
|
||||
* the local clock look AHEAD (never behind), so the estimate is the LOWEST
|
||||
* recent sample (the one delivered fastest), and "ahead" has to persist across
|
||||
* a minute of fresh samples before it is reported. "Behind" can't come from a
|
||||
* delay and is reported as soon as there are enough samples.
|
||||
*/
|
||||
export const SKEW_AHEAD_SPAN_MS = 60_000;
|
||||
/**
|
||||
* Sanity cap on `age`. Old events are still valid samples (the server computes
|
||||
* `age` at response time, so `ts + age` is its clock regardless of the event's
|
||||
@@ -38,14 +53,21 @@ export type ClockSkewState = {
|
||||
warning: boolean;
|
||||
};
|
||||
|
||||
const median = (xs: number[]): number => {
|
||||
const s = [...xs].sort((a, b) => a - b);
|
||||
const mid = Math.floor(s.length / 2);
|
||||
return s.length % 2 ? s[mid] : (s[mid - 1] + s[mid]) / 2;
|
||||
/** A wall-clock change larger than this (vs the monotonic clock) resets the samples. */
|
||||
export const CLOCK_JUMP_MS = 5_000;
|
||||
|
||||
type Sample = { skew: number; at: number };
|
||||
|
||||
const currentClock = (): { wall: number; mono: number } => {
|
||||
const wall = Date.now();
|
||||
const mono = typeof performance !== 'undefined' ? performance.now() : wall;
|
||||
return { wall, mono };
|
||||
};
|
||||
|
||||
export class ClockSkewMonitor {
|
||||
private samples: number[] = [];
|
||||
private samples: Sample[] = [];
|
||||
|
||||
private clockOffset: number | undefined;
|
||||
|
||||
private state: ClockSkewState = { skewMs: null, warning: false };
|
||||
|
||||
@@ -64,25 +86,47 @@ export class ClockSkewMonitor {
|
||||
|
||||
/**
|
||||
* Feed one live event. `originServerTs` + `age` come from the event;
|
||||
* `localTimestamp` is the SDK's `Date.now() − age` at construction.
|
||||
* `localTimestamp` is the SDK's `Date.now() − age` at construction; `clock`
|
||||
* is when the sample was taken: wall clock and a monotonic clock
|
||||
* (performance.now()), so samples are aged by real elapsed time and a change
|
||||
* of the local clock (someone fixing it) starts the measurement afresh.
|
||||
* Only feed events stamped by OUR homeserver: another server's
|
||||
* `origin_server_ts` carries that server's clock.
|
||||
* Returns the new state (unchanged object when nothing moved).
|
||||
*/
|
||||
public sample(
|
||||
originServerTs: number,
|
||||
age: number | undefined,
|
||||
localTimestamp: number,
|
||||
clock: { wall: number; mono: number } = currentClock(),
|
||||
): ClockSkewState {
|
||||
if (age === undefined || !Number.isFinite(age) || age < 0 || age > SKEW_MAX_AGE_MS) {
|
||||
return this.state;
|
||||
}
|
||||
if (!Number.isFinite(originServerTs) || !Number.isFinite(localTimestamp)) return this.state;
|
||||
this.samples.push(localTimestamp - originServerTs);
|
||||
if (this.samples.length > SKEW_SAMPLES) this.samples.shift();
|
||||
const now = clock.mono;
|
||||
const offset = clock.wall - clock.mono;
|
||||
if (this.clockOffset !== undefined && Math.abs(offset - this.clockOffset) > CLOCK_JUMP_MS) {
|
||||
// The local clock was changed: earlier samples measured the old clock.
|
||||
this.reset();
|
||||
}
|
||||
this.clockOffset = offset;
|
||||
this.samples.push({ skew: localTimestamp - originServerTs, at: now });
|
||||
this.samples = this.samples.filter((s) => now - s.at <= SKEW_WINDOW_MS);
|
||||
if (this.samples.length > SKEW_MAX_SAMPLES) this.samples.shift();
|
||||
if (this.samples.length < SKEW_MIN_SAMPLES) return this.state;
|
||||
|
||||
const skewMs = median(this.samples);
|
||||
// Delivery delay only ever adds to a sample: the smallest is the truest.
|
||||
const skewMs = Math.min(...this.samples.map((s) => s.skew));
|
||||
const abs = Math.abs(skewMs);
|
||||
const warning = this.state.warning ? abs >= SKEW_CLEAR_MS : abs > SKEW_WARN_MS;
|
||||
let warning: boolean;
|
||||
if (this.state.warning) warning = abs >= SKEW_CLEAR_MS;
|
||||
else if (skewMs < -SKEW_WARN_MS) warning = true;
|
||||
else if (skewMs > SKEW_WARN_MS) {
|
||||
// Ahead: only if the fastest-delivered samples stayed high for a minute.
|
||||
const span = now - Math.min(...this.samples.map((s) => s.at));
|
||||
warning = span >= SKEW_AHEAD_SPAN_MS;
|
||||
} else warning = false;
|
||||
if (skewMs === this.state.skewMs && warning === this.state.warning) return this.state;
|
||||
this.state = { skewMs, warning };
|
||||
this.listeners.forEach((cb) => cb(this.state));
|
||||
@@ -91,6 +135,7 @@ export class ClockSkewMonitor {
|
||||
|
||||
public reset(): void {
|
||||
this.samples = [];
|
||||
this.clockOffset = undefined;
|
||||
if (this.state.skewMs !== null || this.state.warning) {
|
||||
this.state = { skewMs: null, warning: false };
|
||||
this.listeners.forEach((cb) => cb(this.state));
|
||||
|
||||
Reference in New Issue
Block a user