A ~30 s homeserver stall ends everyone's call ("Connection lost — Reconnect"), even with the 60 s delayed-leave window prod configures #46

New Issue

2026-10-05T16:17:01-04:00

jared commented

2026-10-05 16:17:01 -04:00

This is very likely the "everyone dropped out of the call" part of the 2026-09-29 incident. Reproduced locally: two people in the voice room, Synapse frozen with SIGSTOP (connections hang, like the OOM stall), then resumed.

With prod's call config (matrix_rtc_session: { delayed_leave_event_delay_ms: 60000, delayed_leave_event_restart_ms: 15000 }, routed in):

stall 40 s → after resume: server still lists both members (the 60 s delayed leave never fired)
              but the call frame shows "Connection lost · You were disconnected from the call · Reconnect · Close"

With the packaged config (18 s / 4 s), a 60 s stall: after resume the server has fired both delayed leaves, nobody is in room state, and both clients sit on "Connection lost" for at least 3 minutes. No new membership or delayed-event request is ever sent, so they stay there until someone clicks Reconnect.

Why:

GroupCallView.tsx:168-183 turns MatrixRTCSessionEvent.MembershipManagerError into ConnectionLostError.
The SDK's MembershipManager (matrix-js-sdk 42.2.0) throws that error once maximumNetworkErrorRetryCount (10) is used up.
During a stall, each delayed-leave restart is raced against delayedLeaveEventRestartLocalTimeoutMs (2 s). The local timeout counts as an AbortError, which is retried immediately. So 10 retries cost about 20 s, and the manager gives up about 20–35 s after the last successful restart, long before the server would send the 60 s delayed leave.
The higher prod delay therefore buys nothing: the client quits first.
In widget mode every aborted try still leaves its HTTP request pending in the host (cinny#318). Those all land at once on recovery (5 × 404, 7 × 429 measured), after the manager has already stopped.

Fix (fork):

Pass a joinConfig that matches the server window: keep retrying (with backoff, not "immediate") until expectedServerDelayLeaveTs has passed. In practice raise maximumNetworkErrorRetryCount, or make local-timeout retries back off by networkErrorRetryMs.
When the session does give up, retry joining automatically once the homeserver answers again (poll /versions, or listen for the host's sync recovery), rather than waiting for a click.
Test: a dev stack with kill -STOP <synapse> for 40 s, with assertions that the call frame never shows ConnectionLost and the membership is intact afterwards.

Found in the October 2026 client audit (Playwright against a local Synapse + LiveKit stack, plus code reading). Code refs are at cinny 30e2dbcd, cinny-desktop 8f67ec7, element-call 3b0050e8.

This is very likely the "everyone dropped out of the call" part of the 2026-09-29 incident. Reproduced locally: two people in the voice room, Synapse frozen with `SIGSTOP` (connections hang, like the OOM stall), then resumed. **With prod's call config** (`matrix_rtc_session: { delayed_leave_event_delay_ms: 60000, delayed_leave_event_restart_ms: 15000 }`, routed in): ``` stall 40 s → after resume: server still lists both members (the 60 s delayed leave never fired) but the call frame shows "Connection lost · You were disconnected from the call · Reconnect · Close" ``` **With the packaged config** (18 s / 4 s), a 60 s stall: after resume the server has fired both delayed leaves, nobody is in room state, and both clients sit on "Connection lost" for at least 3 minutes. No new membership or delayed-event request is ever sent, so they stay there until someone clicks Reconnect. **Why:** - `GroupCallView.tsx:168-183` turns `MatrixRTCSessionEvent.MembershipManagerError` into `ConnectionLostError`. - The SDK's MembershipManager (matrix-js-sdk 42.2.0) throws that error once `maximumNetworkErrorRetryCount` (10) is used up. - During a stall, each delayed-leave restart is raced against `delayedLeaveEventRestartLocalTimeoutMs` (2 s). The local timeout counts as an `AbortError`, which is retried **immediately**. So 10 retries cost about 20 s, and the manager gives up about 20–35 s after the last successful restart, long before the server would send the 60 s delayed leave. - The higher prod delay therefore buys nothing: the client quits first. - In widget mode every aborted try still leaves its HTTP request pending in the host (cinny#318). Those all land at once on recovery (`5 × 404, 7 × 429` measured), after the manager has already stopped. **Fix (fork):** - Pass a `joinConfig` that matches the server window: keep retrying (with backoff, not "immediate") until `expectedServerDelayLeaveTs` has passed. In practice raise `maximumNetworkErrorRetryCount`, or make local-timeout retries back off by `networkErrorRetryMs`. - When the session does give up, retry joining automatically once the homeserver answers again (poll `/versions`, or listen for the host's sync recovery), rather than waiting for a click. - Test: a dev stack with `kill -STOP <synapse>` for 40 s, with assertions that the call frame never shows ConnectionLost and the membership is intact afterwards. --- _Found in the October 2026 client audit (Playwright against a local Synapse + LiveKit stack, plus code reading). Code refs are at cinny `30e2dbcd`, cinny-desktop `8f67ec7`, element-call `3b0050e8`._