LotusGuild/cinny
3
0
Fork 0
Code Issues 39 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

A stalled homeserver goes unnoticed for up to 110 s: no strip, Kuma isn't asked, and sends just spin (the #124 outage scenario) #317

New Issue
Closed
opened 2026-10-05 15:53:19 -04:00 by jared · 1 comment
jared commented 2026-10-05 15:53:19 -04:00
Owner
Copy Link
Copy Source

#124 (the Kuma status banner) was motivated by the 2026-09-29 incident: Synapse stalled for about 2 minutes (out of memory) and the app gave no useful signal. I reproduced a stall locally by freezing Synapse with SIGSTOP for 90 s. The process keeps its socket, so connections are accepted but never answered, just like a hung server. Then I resumed it with SIGCONT.

What the client showed (sampled every 5 s):

+5 s … +91 s   no strip at all (no "Connection Lost", no status banner)
+16 s … +91 s  a message sent at +10 s: "Sending..." the whole time
after resume   delivered within 1 s (one copy); the row still said "Sending..." 21 s later (#306)

Why nothing appears:

  • matrix-js-sdk gives up on a /sync long-poll only after pollTimeout + BUFFER_PERIOD_MS = 30 s + 80 s = 110 s (sync.js:325). Until then the sync state stays SYNCING, so SyncStatus shows nothing.
  • useServerStatus polls Kuma early only when urgent (= sync lost) becomes true. Otherwise it waits for its 5-minute poll, so an outage under 110 s never asks Kuma.
  • Kuma itself needs two failed 60 s beats before "down", so even with an early ask the banner would be late. The client is the first to know here.

Suggestions:

  • Watchdog: if /sync hasn't returned within pollTimeout + ~10 s, or any send has been pending more than ~15 s, treat it as "connection uncertain". Show a soft strip ("Lotus Chat's server isn't responding… still trying") and poll Kuma immediately. A lightweight GET /_matrix/client/versions with a 5 s timeout tells "server hung" apart from "my network is down".
  • Or start the client with a smaller BUFFER_PERIOD. The SDK option isn't public, but pollTimeout is; e.g. pollTimeout: 20000 with an aborting fetch wrapper.
  • Test: SIGSTOP the dev Synapse (as above) and assert that a strip appears within about 40 s and clears after SIGCONT.

Found in the October 2026 client audit (Playwright against a local Synapse + LiveKit stack, plus code reading). Code refs are at cinny 30e2dbcd, cinny-desktop 8f67ec7, element-call 3b0050e8.

#124 (the Kuma status banner) was motivated by the 2026-09-29 incident: Synapse **stalled** for about 2 minutes (out of memory) and the app gave no useful signal. I reproduced a stall locally by freezing Synapse with `SIGSTOP` for 90 s. The process keeps its socket, so connections are accepted but never answered, just like a hung server. Then I resumed it with `SIGCONT`. **What the client showed (sampled every 5 s):** ``` +5 s … +91 s no strip at all (no "Connection Lost", no status banner) +16 s … +91 s a message sent at +10 s: "Sending..." the whole time after resume delivered within 1 s (one copy); the row still said "Sending..." 21 s later (#306) ``` **Why nothing appears:** - matrix-js-sdk gives up on a `/sync` long-poll only after `pollTimeout + BUFFER_PERIOD_MS` = 30 s + 80 s = **110 s** (`sync.js:325`). Until then the sync state stays `SYNCING`, so `SyncStatus` shows nothing. - `useServerStatus` polls Kuma early only when `urgent` (= sync lost) becomes true. Otherwise it waits for its 5-minute poll, so an outage under 110 s never asks Kuma. - Kuma itself needs two failed 60 s beats before "down", so even with an early ask the banner would be late. The client is the first to know here. **Suggestions:** - **Watchdog:** if `/sync` hasn't returned within `pollTimeout + ~10 s`, or any send has been pending more than ~15 s, treat it as "connection uncertain". Show a soft strip ("Lotus Chat's server isn't responding… still trying") and poll Kuma immediately. A lightweight `GET /_matrix/client/versions` with a 5 s timeout tells "server hung" apart from "my network is down". - Or start the client with a smaller `BUFFER_PERIOD`. The SDK option isn't public, but `pollTimeout` is; e.g. `pollTimeout: 20000` with an aborting fetch wrapper. - Test: SIGSTOP the dev Synapse (as above) and assert that a strip appears within about 40 s and clears after `SIGCONT`. --- _Found in the October 2026 client audit (Playwright against a local Synapse + LiveKit stack, plus code reading). Code refs are at cinny `30e2dbcd`, cinny-desktop `8f67ec7`, element-call `3b0050e8`._
jared added the priority: mediumuxarea: notifications labels 2026-10-05 15:53:19 -04:00
jared commented 2026-10-05 16:17:14 -04:00
Author
Owner
Copy Link
Copy Source

Related, same stall test with people in a call: element-call#46 (a stall of about 30 s ends everyone's call with "Connection lost") and #318 (host request pile-up, and the host still shows "in call").

Related, same stall test with people in a call: element-call#46 (a stall of about 30 s ends everyone's call with "Connection lost") and #318 (host request pile-up, and the host still shows "in call").
jared closed this issue 2026-10-17 21:58:24 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
a11y
Accessibility: keyboard, screen reader, contrast, motion
 area: appearance
Client area: appearance
 area: auth-session
Client area: auth-session
 area: build-ci
Client area: build-ci
 area: calls
Client area: calls
 area: desktop
Client area: desktop
 area: media
Client area: media
 area: messaging
Client area: messaging
 area: mobile
Client area: mobile
 area: moderation
Client area: moderation
 area: navigation
Client area: navigation
 area: notifications
Client area: notifications
 area: settings
Client area: settings
 area: threads
Client area: threads
 bug
Something is not working
 dependencies
Third-party package versions and advisories
 docs
README / LOTUS_* docs wrong or missing
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 needs-human-review
performance
Re-render storms, leaks, heavy work on hot paths
 planning
priority: critical
Data loss, security hole, or crash on a main path
 priority: high
Broken feature or serious usability problem
 priority: low
Minor issue or polish
 priority: medium
Wrong behaviour in an edge case or notable degradation
 qa
Manual QA: shipped, needs a human in a real environment
 question
More information is needed
 research
security
XSS, unsafe URLs, data leaks, auth/session
 tech-debt
Code health, dead code, fragile patterns
 ux
Usability or visual inconsistency
 wontfix
This won't be fixed
No labels area: notifications priority: medium ux
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
cole (Cole Keller) jared (Jared Vititoe) nathan (Nathan Vititoe)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: LotusGuild/cinny#317