docs: record PLP storage fix, Synapse 1.158.0, worker triggers

Synapse and PostgreSQL moved to the plpSSDPool Ceph pool (enterprise SSDs
with power-loss protection) after consumer NVMe fsync latency was traced as
the cause of dropped calls: slow fsync stalled restart_delayed_event past
Element Call's 18s delayed_leave_event_delay_ms, firing the MSC4140 delayed
event and clearing every m.call.member at once.

INSERT INTO stream_positions went 87.30ms -> 0.462ms mean.

Also:
- Correct the Synapse version to 1.158.0 (file claimed 1.157.2; the upgrade
  was performed but never logged - second time)
- Correct the push_rules.limits note: the limits are enforced by default in
  1.157.2+, so no config block is needed. Verified against the installed
  synapse/config/push_rules.py
- Record worker revisit thresholds. Synapse measures 0.045 cores, 4.5% of
  its single-process ceiling, so a worker split would add Redis and routing
  to solve a CPU problem that does not exist
- Note that presence is a deliberate, paid-for cost tied to Cinny features
- BBR now applied on compute-storage-gpu-01 as well
- Remove a duplicated "Server Checklist" header

Ceph pool topology and drive-level detail deliberately left out; that
belongs with the cluster docs, not the Matrix README.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

README.md

@@ -50,7 +50,7 @@ matrix/
 
 | Service | IP | LXC | RAM | Disk | Versions |
 |---------|----|-----|-----|------|----------|
-| Synapse | 10.10.10.29 | 151 | 12GB | 50GB | Synapse 1.157.2, LiveKit 1.9.11, hookshot 7.3.2, coturn latest |
+| Synapse | 10.10.10.29 | 151 | 12GB | 50GB | Synapse 1.158.0, LiveKit 1.9.11, hookshot 7.3.2, coturn latest |
 | PostgreSQL 17 | 10.10.10.44 | 109 | 6GB | 80GB | PostgreSQL 17.9 |
 | Cinny Web | 10.10.10.6 | 106 | 2GB | 8GB | Debian 12, nginx, Node 24, Lotus Cinny fork (custom, tracks `cinnyapp/cinny` main) |
 | Draupnir | 10.10.10.24 | 110 | 1GB | 10GB | Draupnir v2.9.0, Node.js v22 |
@@ -557,6 +557,27 @@ ExecStartPre=-/bin/bash -c 'pkill -x livekit-server; sleep 1'
 KillMode=control-group
 ```
 
+### Dropped calls / slow Synapse — fixed by moving to PLP-backed storage (2026-08-16)
+
+Synapse and PostgreSQL were on Ceph `appPool`, backed by consumer NVMe with **no power-loss
+protection**. Without PLP, BlueStore's flush-on-every-write forces a real NAND program, so fsync cost
+tens to hundreds of milliseconds instead of microseconds.
+
+**The call-kick mechanism:** slow fsync → `restart_delayed_event` stalling ~31 s → exceeding Element
+Call's 18 s `delayed_leave_event_delay_ms` → Synapse fires the MSC4140 delayed event → every
+`m.call.member` cleared at once, removing all participants.
+
+Symptom to recognise: `INSERT INTO stream_positions` — a one-row upsert — was the **top query in
+`pg_stat_statements` by total time** at 87.30 ms mean. That is not query cost, it is a bare WAL fsync.
+LXC 109 also showed `procs_blocked 7` (D-state) while Synapse used 0.045 CPU cores.
+
+**Fix:** LXC 151 and 109 moved to the `plpSSDPool` Ceph pool (enterprise SSDs with PLP, `size=3`).
+`stream_positions` went **87.30 ms → 0.462 ms**. Pool topology and drive details live with the Ceph
+cluster docs, not here.
+
+> If call drops ever return, check DB write latency first:
+> `rbd perf image iostat --pool plpSSDPool` and the `stream_positions` mean in `pg_stat_statements`.
+
 ### coturn TLS Reset Errors
 
 Periodic `TLS/TCP socket error: Connection reset by peer` in coturn logs. Normal — clients probe TURN and drop once they establish a direct P2P path.
@@ -565,9 +586,11 @@ Periodic `TLS/TCP socket error: Connection reset by peer` in coturn logs. Normal
 
 `net.ipv4.tcp_congestion_control = bbr` must be set on the Proxmox **host**, not inside an unprivileged LXC. All other sysctl tuning (TCP/UDP buffers, fin_timeout) is applied inside LXC 151.
 
-Applied 2026-07-26 via `/etc/sysctl.d/99-bbr.conf` on `compute-storage-01` and `large1` (the two
+Applied 2026-07-26 via `/etc/sysctl.d/99-bbr.conf` on `compute-storage-01` and `large1`, and
-nodes the HA affinity rule allows LXC 151 to run on). Because the file lives on the host, **a node
+**2026-08-16 on `compute-storage-gpu-01`** (which was found on `cubic`/`fq_codel` with `tcp_bbr` not
-that has never hosted 151 will not have it** — check before allowing 151 to migrate somewhere new:
+even loaded). All three nodes that may host LXC 151 now report `bbr`/`fq`. Because the file lives on
+the host, **a node that has never hosted 151 will not have it** — check before allowing 151 to migrate
+somewhere new:
 
 ```bash
 sysctl net.ipv4.tcp_congestion_control net.core.default_qdisc   # want: bbr / fq
@@ -643,6 +666,28 @@ It was on the table because peak CPU hit 0.96 cores — the single-process GIL c
 cleanup, peak is 0.16 cores, so the Redis + generic_workers + `federation_sender` + NPM routing
 machinery would be operational complexity for no gain. Revisit only if reactor tick returns to ~1ms.
 
+> **Re-confirmed 2026-08-16.** Synapse now measures **0.045 cores** — 4.5% of its single-process
+> ceiling, 22× headroom. Workers would add Redis, an `instance_map` and NPM routing to solve a CPU
+> problem that does not exist, while multiplying the number of processes contending on the same
+> **fsync-bound** database (see Known Issues → dropped calls / PLP storage). Federating with larger
+> servers does not change this: the cost of a big federated room is state resolution and DB I/O,
+> not homeserver CPU.
+>
+> **Revisit only when, sustained over a week and *after* the storage fix lands:**
+> - `rate(process_cpu_seconds_total{job="synapse"}[5m])` > **0.7** cores (today: 0.045), **or**
+> - reactor tick p99 > **50 ms** *while* DB write latency stays under 2 ms — i.e. genuinely
+>   CPU-bound rather than storage-bound (today: 4.5 ms tick at 0.045 cores = storage-bound), **or**
+> - `synapse_federation_transaction_queue_pending_pdus` sustained > 100 with CPU near 1.0.
+>
+> First worker to add would be `federation_sender`, then a `generic_worker` for `/sync`. Redis and an
+> `instance_map` are prerequisites for either.
+
+**`presence: enabled: true` is a real, deliberate cost.** It drives ~18k `presence_stream`
+insert+delete per window and is one of the larger federation overheads at scale. Disabling it is a
+standard scaling win — but it would break the presence badges, Discord-style presence tracking, and
+custom status message in the Cinny fork. It is **paid for on purpose**, not an oversight. Do not
+"optimize" it away without removing those features first.
+
 The cost is driven almost entirely by **large federated rooms**, and it scales with the room's
 *global* member count, not with how much you actually use it. One local user joined to a huge public
 room makes the server pay that room's full state-resolution cost forever. Because Synapse's caches,
@@ -662,12 +707,24 @@ Leaving a space does **not** leave its child rooms; they are independent rooms w
 
 ## Server Checklist
 
-## Server Checklist
-
 ### Quality of Life
 - [x] **Upgrade Synapse to v1.155.0** — Done 2026-06-18. LXC 151 was already on Debian 13 Trixie; no OS migration needed.
 - [x] **Upgrade Synapse to v1.157.1** — Done 2026-07-23. Note: the host was found running **1.156.0**, not the 1.155.0 recorded here — an intermediate upgrade had happened without being logged, so always verify with `dpkg-query -W matrix-synapse-py3` rather than trusting this file. Not a security release (no CVEs between 1.155.0 and 1.157.1; the last advisories were fixed back in 1.152.1), but 1.157.0 fixes an application-service ephemeral-events regression that 1.156.0 shipped — which affects hookshot. No schema-version bump (94 → 94); the three new background updates finished immediately. `experimental_features` (MatrixRTC/Element Call MSCs) was preserved and is unaffected by the v1.157.0 removal of `msc3861`, which we do not use.
 - [x] **Upgrade Synapse to v1.157.2** — Done 2026-07-28. **Security release** (11 advisories: 6 High / 3 Moderate / 2 Low; no CVE IDs assigned). Most High-severity issues require open federation (we federate) — `/get_missing_events` cross-room leak, relayed-401 crypto-state destruction, spoofed to-device, room-alias redirection; two call out untrusted local users (push-rule exhaustion, alias redirection). Pure apt bump: no `# Upgrading to v1.157.2` notes, no schema migration, Python/PG minimums unchanged, config preserved via `--force-confold`. Restarted clean (NRestarts=0), federation healthy. NB: 1.157.2 adds an **optional** `push_rules.limits` config block (safe defaults: rule_count 10000 / rule_id_length 300 / rule_size 1024) for the push-rule-exhaustion advisory — not required, but worth tuning given untrusted local users. Verify with `dpkg-query -W matrix-synapse-py3`, not this file.
+
+  > **Correction (2026-08-16):** the note above about `push_rules.limits` being an "optional config
+  > block ... worth tuning" is **wrong**, and no action is needed. Reading the installed source
+  > (`synapse/config/push_rules.py`), `PushRulesLimitsConfig` already defaults to `rule_count=10000`,
+  > `rule_id_length=300`, `rule_size=1024`, and `read_config` builds it from `{}` when the block is
+  > absent — so the limits are **enforced by default**. Verified live on LXC 151 with no `push_rules`
+  > block present. Writing the block changes nothing and costs a restart. Only add it to set values
+  > *tighter* than the defaults.
+
+- [x] **Upgrade Synapse to v1.158.0** — found already installed 2026-08-16 (`1.158.0+trixie1`), while
+  this file still claimed 1.157.2. **The upgrade was not logged by whoever performed it** — the second
+  time this has happened (see the 1.157.1 entry). It is also the current apt candidate, so there is
+  nothing newer to take. Reinforces the standing rule: trust `dpkg-query -W matrix-synapse-py3`, never
+  this file.
 - [x] Migrate from SQLite to PostgreSQL
 
 - [x] TURN/STUN server (coturn) for reliable voice/video
@@ -698,7 +755,7 @@ Leaving a space does **not** leave its child rooms; they are independent rooms w
 - [x] TCP retransmit timeout lowered (`tcp_retries2=5`, `tcp_syn_retries=4`, `tcp_keepalive_probes=3`) — stalled outbound federation connections now fail in ~15-30s instead of ~15 min
 - [x] Unreachable routes added for servers with asymmetric connectivity (can reach us but we can't reach their federation port) — prevents 90s TCP hangs from being added to lag; defined in `/etc/network/interfaces` post-up hooks and survive reboots (bark.lgbt ×2, parodia.dev, chat.ohaa.xyz, matrix.k8ekat.dev)
 - [x] Stuck `device_lists_remote_resync` entries cleared for dead-server users (@dalite:bark.lgbt, @arndot:matrix.goch.social) — device list resync was firing every 30s
-- [x] BBR congestion control — applied 2026-07-26 on the Proxmox hosts that can run LXC 151 (`compute-storage-01`, `large1`) via `/etc/sysctl.d/99-bbr.conf` (`net.ipv4.tcp_congestion_control=bbr`, `net.core.default_qdisc=fq`). Persists across reboots. If 151 is ever allowed to run on another node, apply it there too.
+- [x] BBR congestion control — applied 2026-07-26 on `compute-storage-01` and `large1`, and 2026-08-16 on `compute-storage-gpu-01`, via `/etc/sysctl.d/99-bbr.conf` (`net.ipv4.tcp_congestion_control=bbr`, `net.core.default_qdisc=fq`). Persists across reboots. All three nodes that may host LXC 151 are now covered. If 151 is ever allowed to run on another node, apply it there too.
 - [x] LXC 109 rootfs grown 30GB → 80GB (2026-07-26). It was at **89% full with only 3.4GB free**, and `pg_wal` shares that volume — PostgreSQL stops outright at 100%. Also a hard prerequisite for any `VACUUM FULL`, which must rewrite the table alongside the original.
 - [x] **Purged the `#community:matrix.org` space** (`!iMZEhwCvbfeAYUxAjZ:t2l.io`) 2026-07-26 — 81,724,747 rows / 17GB for a directory wrapper with **zero messages** and one local member. Admin API `DELETE /_synapse/admin/v2/rooms/` with `purge:true`, then `VACUUM FULL`. ~2.5 min downtime.
 - [x] **Compressed room state** with `rust-synapse-compress-state` (built at `/opt/rust-synapse-compress-state/` on LXC 109). Every run verified `New state map matches old one` before commit:
@@ -954,7 +1011,7 @@ All commands use the `!` prefix. Run `!help` in any room for the full list.
 
 | Component | Technology | Version |
 |-----------|-----------|---------|
-| Homeserver | Synapse | 1.157.2 |
+| Homeserver | Synapse | 1.158.0 |
 | Database | PostgreSQL | 17.9 |
 | TURN | coturn | latest |
 | Video/voice calls | LiveKit SFU | 1.9.11 |