docs: record PLP storage fix, Synapse 1.158.0, worker triggers
Synapse and PostgreSQL moved to the plpSSDPool Ceph pool (enterprise SSDs with power-loss protection) after consumer NVMe fsync latency was traced as the cause of dropped calls: slow fsync stalled restart_delayed_event past Element Call's 18s delayed_leave_event_delay_ms, firing the MSC4140 delayed event and clearing every m.call.member at once. INSERT INTO stream_positions went 87.30ms -> 0.462ms mean. Also: - Correct the Synapse version to 1.158.0 (file claimed 1.157.2; the upgrade was performed but never logged - second time) - Correct the push_rules.limits note: the limits are enforced by default in 1.157.2+, so no config block is needed. Verified against the installed synapse/config/push_rules.py - Record worker revisit thresholds. Synapse measures 0.045 cores, 4.5% of its single-process ceiling, so a worker split would add Redis and routing to solve a CPU problem that does not exist - Note that presence is a deliberate, paid-for cost tied to Cinny features - BBR now applied on compute-storage-gpu-01 as well - Remove a duplicated "Server Checklist" header Ceph pool topology and drive-level detail deliberately left out; that belongs with the cluster docs, not the Matrix README. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
@@ -50,7 +50,7 @@ matrix/
|
||||
|
||||
| Service | IP | LXC | RAM | Disk | Versions |
|
||||
|---------|----|-----|-----|------|----------|
|
||||
| Synapse | 10.10.10.29 | 151 | 12GB | 50GB | Synapse 1.157.2, LiveKit 1.9.11, hookshot 7.3.2, coturn latest |
|
||||
| Synapse | 10.10.10.29 | 151 | 12GB | 50GB | Synapse 1.158.0, LiveKit 1.9.11, hookshot 7.3.2, coturn latest |
|
||||
| PostgreSQL 17 | 10.10.10.44 | 109 | 6GB | 80GB | PostgreSQL 17.9 |
|
||||
| Cinny Web | 10.10.10.6 | 106 | 2GB | 8GB | Debian 12, nginx, Node 24, Lotus Cinny fork (custom, tracks `cinnyapp/cinny` main) |
|
||||
| Draupnir | 10.10.10.24 | 110 | 1GB | 10GB | Draupnir v2.9.0, Node.js v22 |
|
||||
@@ -557,6 +557,27 @@ ExecStartPre=-/bin/bash -c 'pkill -x livekit-server; sleep 1'
|
||||
KillMode=control-group
|
||||
```
|
||||
|
||||
### Dropped calls / slow Synapse — fixed by moving to PLP-backed storage (2026-08-16)
|
||||
|
||||
Synapse and PostgreSQL were on Ceph `appPool`, backed by consumer NVMe with **no power-loss
|
||||
protection**. Without PLP, BlueStore's flush-on-every-write forces a real NAND program, so fsync cost
|
||||
tens to hundreds of milliseconds instead of microseconds.
|
||||
|
||||
**The call-kick mechanism:** slow fsync → `restart_delayed_event` stalling ~31 s → exceeding Element
|
||||
Call's 18 s `delayed_leave_event_delay_ms` → Synapse fires the MSC4140 delayed event → every
|
||||
`m.call.member` cleared at once, removing all participants.
|
||||
|
||||
Symptom to recognise: `INSERT INTO stream_positions` — a one-row upsert — was the **top query in
|
||||
`pg_stat_statements` by total time** at 87.30 ms mean. That is not query cost, it is a bare WAL fsync.
|
||||
LXC 109 also showed `procs_blocked 7` (D-state) while Synapse used 0.045 CPU cores.
|
||||
|
||||
**Fix:** LXC 151 and 109 moved to the `plpSSDPool` Ceph pool (enterprise SSDs with PLP, `size=3`).
|
||||
`stream_positions` went **87.30 ms → 0.462 ms**. Pool topology and drive details live with the Ceph
|
||||
cluster docs, not here.
|
||||
|
||||
> If call drops ever return, check DB write latency first:
|
||||
> `rbd perf image iostat --pool plpSSDPool` and the `stream_positions` mean in `pg_stat_statements`.
|
||||
|
||||
### coturn TLS Reset Errors
|
||||
|
||||
Periodic `TLS/TCP socket error: Connection reset by peer` in coturn logs. Normal — clients probe TURN and drop once they establish a direct P2P path.
|
||||
@@ -565,9 +586,11 @@ Periodic `TLS/TCP socket error: Connection reset by peer` in coturn logs. Normal
|
||||
|
||||
`net.ipv4.tcp_congestion_control = bbr` must be set on the Proxmox **host**, not inside an unprivileged LXC. All other sysctl tuning (TCP/UDP buffers, fin_timeout) is applied inside LXC 151.
|
||||
|
||||
Applied 2026-07-26 via `/etc/sysctl.d/99-bbr.conf` on `compute-storage-01` and `large1` (the two
|
||||
nodes the HA affinity rule allows LXC 151 to run on). Because the file lives on the host, **a node
|
||||
that has never hosted 151 will not have it** — check before allowing 151 to migrate somewhere new:
|
||||
Applied 2026-07-26 via `/etc/sysctl.d/99-bbr.conf` on `compute-storage-01` and `large1`, and
|
||||
**2026-08-16 on `compute-storage-gpu-01`** (which was found on `cubic`/`fq_codel` with `tcp_bbr` not
|
||||
even loaded). All three nodes that may host LXC 151 now report `bbr`/`fq`. Because the file lives on
|
||||
the host, **a node that has never hosted 151 will not have it** — check before allowing 151 to migrate
|
||||
somewhere new:
|
||||
|
||||
```bash
|
||||
sysctl net.ipv4.tcp_congestion_control net.core.default_qdisc # want: bbr / fq
|
||||
@@ -643,6 +666,28 @@ It was on the table because peak CPU hit 0.96 cores — the single-process GIL c
|
||||
cleanup, peak is 0.16 cores, so the Redis + generic_workers + `federation_sender` + NPM routing
|
||||
machinery would be operational complexity for no gain. Revisit only if reactor tick returns to ~1ms.
|
||||
|
||||
> **Re-confirmed 2026-08-16.** Synapse now measures **0.045 cores** — 4.5% of its single-process
|
||||
> ceiling, 22× headroom. Workers would add Redis, an `instance_map` and NPM routing to solve a CPU
|
||||
> problem that does not exist, while multiplying the number of processes contending on the same
|
||||
> **fsync-bound** database (see Known Issues → dropped calls / PLP storage). Federating with larger
|
||||
> servers does not change this: the cost of a big federated room is state resolution and DB I/O,
|
||||
> not homeserver CPU.
|
||||
>
|
||||
> **Revisit only when, sustained over a week and *after* the storage fix lands:**
|
||||
> - `rate(process_cpu_seconds_total{job="synapse"}[5m])` > **0.7** cores (today: 0.045), **or**
|
||||
> - reactor tick p99 > **50 ms** *while* DB write latency stays under 2 ms — i.e. genuinely
|
||||
> CPU-bound rather than storage-bound (today: 4.5 ms tick at 0.045 cores = storage-bound), **or**
|
||||
> - `synapse_federation_transaction_queue_pending_pdus` sustained > 100 with CPU near 1.0.
|
||||
>
|
||||
> First worker to add would be `federation_sender`, then a `generic_worker` for `/sync`. Redis and an
|
||||
> `instance_map` are prerequisites for either.
|
||||
|
||||
**`presence: enabled: true` is a real, deliberate cost.** It drives ~18k `presence_stream`
|
||||
insert+delete per window and is one of the larger federation overheads at scale. Disabling it is a
|
||||
standard scaling win — but it would break the presence badges, Discord-style presence tracking, and
|
||||
custom status message in the Cinny fork. It is **paid for on purpose**, not an oversight. Do not
|
||||
"optimize" it away without removing those features first.
|
||||
|
||||
The cost is driven almost entirely by **large federated rooms**, and it scales with the room's
|
||||
*global* member count, not with how much you actually use it. One local user joined to a huge public
|
||||
room makes the server pay that room's full state-resolution cost forever. Because Synapse's caches,
|
||||
@@ -662,12 +707,24 @@ Leaving a space does **not** leave its child rooms; they are independent rooms w
|
||||
|
||||
## Server Checklist
|
||||
|
||||
## Server Checklist
|
||||
|
||||
### Quality of Life
|
||||
- [x] **Upgrade Synapse to v1.155.0** — Done 2026-06-18. LXC 151 was already on Debian 13 Trixie; no OS migration needed.
|
||||
- [x] **Upgrade Synapse to v1.157.1** — Done 2026-07-23. Note: the host was found running **1.156.0**, not the 1.155.0 recorded here — an intermediate upgrade had happened without being logged, so always verify with `dpkg-query -W matrix-synapse-py3` rather than trusting this file. Not a security release (no CVEs between 1.155.0 and 1.157.1; the last advisories were fixed back in 1.152.1), but 1.157.0 fixes an application-service ephemeral-events regression that 1.156.0 shipped — which affects hookshot. No schema-version bump (94 → 94); the three new background updates finished immediately. `experimental_features` (MatrixRTC/Element Call MSCs) was preserved and is unaffected by the v1.157.0 removal of `msc3861`, which we do not use.
|
||||
- [x] **Upgrade Synapse to v1.157.2** — Done 2026-07-28. **Security release** (11 advisories: 6 High / 3 Moderate / 2 Low; no CVE IDs assigned). Most High-severity issues require open federation (we federate) — `/get_missing_events` cross-room leak, relayed-401 crypto-state destruction, spoofed to-device, room-alias redirection; two call out untrusted local users (push-rule exhaustion, alias redirection). Pure apt bump: no `# Upgrading to v1.157.2` notes, no schema migration, Python/PG minimums unchanged, config preserved via `--force-confold`. Restarted clean (NRestarts=0), federation healthy. NB: 1.157.2 adds an **optional** `push_rules.limits` config block (safe defaults: rule_count 10000 / rule_id_length 300 / rule_size 1024) for the push-rule-exhaustion advisory — not required, but worth tuning given untrusted local users. Verify with `dpkg-query -W matrix-synapse-py3`, not this file.
|
||||
|
||||
> **Correction (2026-08-16):** the note above about `push_rules.limits` being an "optional config
|
||||
> block ... worth tuning" is **wrong**, and no action is needed. Reading the installed source
|
||||
> (`synapse/config/push_rules.py`), `PushRulesLimitsConfig` already defaults to `rule_count=10000`,
|
||||
> `rule_id_length=300`, `rule_size=1024`, and `read_config` builds it from `{}` when the block is
|
||||
> absent — so the limits are **enforced by default**. Verified live on LXC 151 with no `push_rules`
|
||||
> block present. Writing the block changes nothing and costs a restart. Only add it to set values
|
||||
> *tighter* than the defaults.
|
||||
|
||||
- [x] **Upgrade Synapse to v1.158.0** — found already installed 2026-08-16 (`1.158.0+trixie1`), while
|
||||
this file still claimed 1.157.2. **The upgrade was not logged by whoever performed it** — the second
|
||||
time this has happened (see the 1.157.1 entry). It is also the current apt candidate, so there is
|
||||
nothing newer to take. Reinforces the standing rule: trust `dpkg-query -W matrix-synapse-py3`, never
|
||||
this file.
|
||||
- [x] Migrate from SQLite to PostgreSQL
|
||||
|
||||
- [x] TURN/STUN server (coturn) for reliable voice/video
|
||||
@@ -698,7 +755,7 @@ Leaving a space does **not** leave its child rooms; they are independent rooms w
|
||||
- [x] TCP retransmit timeout lowered (`tcp_retries2=5`, `tcp_syn_retries=4`, `tcp_keepalive_probes=3`) — stalled outbound federation connections now fail in ~15-30s instead of ~15 min
|
||||
- [x] Unreachable routes added for servers with asymmetric connectivity (can reach us but we can't reach their federation port) — prevents 90s TCP hangs from being added to lag; defined in `/etc/network/interfaces` post-up hooks and survive reboots (bark.lgbt ×2, parodia.dev, chat.ohaa.xyz, matrix.k8ekat.dev)
|
||||
- [x] Stuck `device_lists_remote_resync` entries cleared for dead-server users (@dalite:bark.lgbt, @arndot:matrix.goch.social) — device list resync was firing every 30s
|
||||
- [x] BBR congestion control — applied 2026-07-26 on the Proxmox hosts that can run LXC 151 (`compute-storage-01`, `large1`) via `/etc/sysctl.d/99-bbr.conf` (`net.ipv4.tcp_congestion_control=bbr`, `net.core.default_qdisc=fq`). Persists across reboots. If 151 is ever allowed to run on another node, apply it there too.
|
||||
- [x] BBR congestion control — applied 2026-07-26 on `compute-storage-01` and `large1`, and 2026-08-16 on `compute-storage-gpu-01`, via `/etc/sysctl.d/99-bbr.conf` (`net.ipv4.tcp_congestion_control=bbr`, `net.core.default_qdisc=fq`). Persists across reboots. All three nodes that may host LXC 151 are now covered. If 151 is ever allowed to run on another node, apply it there too.
|
||||
- [x] LXC 109 rootfs grown 30GB → 80GB (2026-07-26). It was at **89% full with only 3.4GB free**, and `pg_wal` shares that volume — PostgreSQL stops outright at 100%. Also a hard prerequisite for any `VACUUM FULL`, which must rewrite the table alongside the original.
|
||||
- [x] **Purged the `#community:matrix.org` space** (`!iMZEhwCvbfeAYUxAjZ:t2l.io`) 2026-07-26 — 81,724,747 rows / 17GB for a directory wrapper with **zero messages** and one local member. Admin API `DELETE /_synapse/admin/v2/rooms/` with `purge:true`, then `VACUUM FULL`. ~2.5 min downtime.
|
||||
- [x] **Compressed room state** with `rust-synapse-compress-state` (built at `/opt/rust-synapse-compress-state/` on LXC 109). Every run verified `New state map matches old one` before commit:
|
||||
@@ -954,7 +1011,7 @@ All commands use the `!` prefix. Run `!help` in any room for the full list.
|
||||
|
||||
| Component | Technology | Version |
|
||||
|-----------|-----------|---------|
|
||||
| Homeserver | Synapse | 1.157.2 |
|
||||
| Homeserver | Synapse | 1.158.0 |
|
||||
| Database | PostgreSQL | 17.9 |
|
||||
| TURN | coturn | latest |
|
||||
| Video/voice calls | LiveKit SFU | 1.9.11 |
|
||||
|
||||
Reference in New Issue
Block a user