docs: record Synapse 1.157.2 security upgrade

Upgraded LXC 151 from 1.157.1 to 1.157.2 (packages.matrix.org, Trixie), a
security release fixing 11 advisories (6 High / 3 Moderate / 2 Low; no CVE IDs
assigned). Applies to us: most High-severity issues need open federation (we
federate) and two call out untrusted local users.

Pure apt bump — no upgrade.md notes for v1.157.2, no schema migration,
Python/PostgreSQL minimums unchanged, homeserver.yaml preserved via
--force-confold. Clean restart (NRestarts=0), version endpoint reports 1.157.2,
no errors, federation healthy.

Note: 1.157.2 adds an OPTIONAL push_rules.limits config block (safe defaults)
for the push-rule-exhaustion advisory — captured for later tuning, not required.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

README.md

@@ -50,8 +50,8 @@ matrix/
 
 | Service | IP | LXC | RAM | Disk | Versions |
 |---------|----|-----|-----|------|----------|
-| Synapse | 10.10.10.29 | 151 | 8GB | 50GB | Synapse 1.157.1, LiveKit 1.9.11, hookshot 7.3.2, coturn latest |
+| Synapse | 10.10.10.29 | 151 | 12GB | 50GB | Synapse 1.157.2, LiveKit 1.9.11, hookshot 7.3.2, coturn latest |
-| PostgreSQL 17 | 10.10.10.44 | 109 | 6GB | 30GB | PostgreSQL 17.9 |
+| PostgreSQL 17 | 10.10.10.44 | 109 | 6GB | 80GB | PostgreSQL 17.9 |
 | Cinny Web | 10.10.10.6 | 106 | 2GB | 8GB | Debian 12, nginx, Node 24, Lotus Cinny fork (custom, tracks `cinnyapp/cinny` main) |
 | Draupnir | 10.10.10.24 | 110 | 1GB | 10GB | Draupnir v2.9.0, Node.js v22 |
 | Prometheus | 10.10.10.48 | 118 | — | — | Prometheus — scrapes all Matrix services |
@@ -563,7 +563,100 @@ Periodic `TLS/TCP socket error: Connection reset by peer` in coturn logs. Normal
 
 ### BBR Congestion Control
 
-`net.ipv4.tcp_congestion_control = bbr` must be set on the Proxmox host, not inside an unprivileged LXC. All other sysctl tuning (TCP/UDP buffers, fin_timeout) is applied inside LXC 151.
+`net.ipv4.tcp_congestion_control = bbr` must be set on the Proxmox **host**, not inside an unprivileged LXC. All other sysctl tuning (TCP/UDP buffers, fin_timeout) is applied inside LXC 151.
+
+Applied 2026-07-26 via `/etc/sysctl.d/99-bbr.conf` on `compute-storage-01` and `large1` (the two
+nodes the HA affinity rule allows LXC 151 to run on). Because the file lives on the host, **a node
+that has never hosted 151 will not have it** — check before allowing 151 to migrate somewhere new:
+
+```bash
+sysctl net.ipv4.tcp_congestion_control net.core.default_qdisc   # want: bbr / fq
+```
+
+### ⚠️ `large1` has a failing DIMM — keep Synapse and PostgreSQL off it
+
+`large1` logs **corrected ECC errors** concentrated on one module (**P0 CHANNEL B, DIMM 1** —
+EDAC `mc#0csrow#2channel#1`, 421 corrected errors observed, with the MCE `Over`/overflow flag set
+meaning more occurred than were logged). It has hard-crashed **on its own** twice — 2026-07-24 14:19
+and 2026-07-26 00:20 — with no clean-shutdown markers and no OOM events, which is the classic
+signature of an uncorrectable ECC error.
+
+> Do **not** confuse these with the cluster-wide resets on 2026-07-22 01:56 and 2026-07-24 11:01-11:45,
+> which took down **all six nodes** simultaneously and were power events, not host hardware.
+
+Consequences for Matrix:
+
+- On the 2026-07-26 crash, HA relocated **LXC 109 (PostgreSQL)** and **LXC 151 (Synapse)** to
+  `compute-storage-01`, which has **zero ECC errors across all boots**. They should stay there.
+- The HA rule `ha-rule-43cb668b-becf` in `/etc/pve/ha/rules.cfg` currently weights
+  `compute-storage-01:1,large1:1` — **equal priority, non-strict** — so a future failover can put
+  Synapse straight back onto the bad-RAM node. Re-weighting to `compute-storage-01:2,large1:1` would
+  prefer the healthy node while keeping large1 as a fallback. Not changed automatically because that
+  rule governs 10 resources and `compute-storage-01` has only 46GB RAM vs large1's 128GB.
+- **Never run a large `VACUUM FULL` or state-compression job on a host with failing RAM** — those
+  rewrite the whole table through memory, and a bit-flip becomes silent, backup-propagated corruption.
+- Ceph's **`noout` flag is intentionally left set.** With 10 OSDs on an intermittently-crashing node,
+  `noout` prevents a full 10-OSD rebalance every time it bounces. Do not clear it until large1 is stable.
+
+A replacement DIMM is not currently in budget. Free mitigations worth evaluating: physically removing
+the bad module (96GB in flex mode, or 64GB symmetric dual-channel), kernel page soft-offlining via
+`rasdaemon` (the errors cluster in a small set of physical pages), or a `memmap=` blacklist.
+
+### PostgreSQL `state_groups_state` bloat
+
+Matrix state resolution is the dominant cost on this homeserver, not message volume.
+
+**2026-07-26 cleanup — database went 22GB → 2183MB (−90%):**
+
+| Metric | Before | After |
+|--------|--------|-------|
+| Database size | 22 GB | **2183 MB** |
+| `state_groups_state` | 20 GB | **540 MB** |
+| `state_groups_state` rows | 89,543,584 | **2,325,198** |
+| LXC 109 disk used | 25 G (89%) | **8.5 G (12%)** |
+
+Two independent wins: purging one useless space (17GB) and compressing five large federated
+rooms (a further ~1.3GB). Total downtime across the whole operation was **~3.5 minutes**.
+
+**Runtime effect** (Prometheus, 20-minute window vs. the same window 24h earlier):
+
+| Metric | Before | After |
+|--------|--------|-------|
+| `get_auth_chain_difference_chains` DB time | 0.0508 s/s | **0.0 — never called** |
+| `_get_state_groups_from_groups` DB time | 0.03 s/s | 0.0004 s/s |
+| Reactor tick (avg) | 1.18 ms | **0.52 ms** |
+| Peak CPU | 0.42 cores | 0.16 cores |
+| `stateGroupMembersCache` | 974,953/1M **FULL**, 8,007,530 evictions, 28% hit | 104 entries, **0 evictions** |
+| `*getEvent*` | 61,440/61,440 **FULL**, 546,724 evictions | 286 entries, **0 evictions** |
+
+> Read the "after" cache hit-rates (30-60%) with care: they are *low* only because the caches are
+> now nearly empty and cold-miss dominated. The number that matters is **evictions: 8 million → 0**.
+> `event_cache_size` was left at `30K` — raising it would be pointless when the cache sits at
+> 286/61,440. Tune caches against eviction counts, not hit rates.
+
+> Caveat on the before/after comparison: the two windows are different times of day on a 26-user
+> server, so reactor-tick and CPU deltas are directional rather than exact. The
+> `get_auth_chain_difference_chains` result is structural, not load-dependent — that room is gone.
+
+**Worker split deliberately NOT done.** Synapse remains monolithic (no Redis, no `instance_map`).
+It was on the table because peak CPU hit 0.96 cores — the single-process GIL ceiling. After this
+cleanup, peak is 0.16 cores, so the Redis + generic_workers + `federation_sender` + NPM routing
+machinery would be operational complexity for no gain. Revisit only if reactor tick returns to ~1ms.
+
+The cost is driven almost entirely by **large federated rooms**, and it scales with the room's
+*global* member count, not with how much you actually use it. One local user joined to a huge public
+room makes the server pay that room's full state-resolution cost forever. Because Synapse's caches,
+GIL, DB pool, and PostgreSQL buffer cache are all **global with no per-room fairness**, a single such
+room degrades every other room on the server.
+
+To find the offenders:
+
+```sql
+SELECT room_id, count(*) FROM state_groups_state GROUP BY room_id ORDER BY 2 DESC LIMIT 10;
+```
+
+Spaces (`m.space`) are the worst offenders per byte — they carry huge membership with zero messages.
+Leaving a space does **not** leave its child rooms; they are independent rooms with their own IDs.
 
 ---
 
@@ -574,6 +667,7 @@ Periodic `TLS/TCP socket error: Connection reset by peer` in coturn logs. Normal
 ### Quality of Life
 - [x] **Upgrade Synapse to v1.155.0** — Done 2026-06-18. LXC 151 was already on Debian 13 Trixie; no OS migration needed.
 - [x] **Upgrade Synapse to v1.157.1** — Done 2026-07-23. Note: the host was found running **1.156.0**, not the 1.155.0 recorded here — an intermediate upgrade had happened without being logged, so always verify with `dpkg-query -W matrix-synapse-py3` rather than trusting this file. Not a security release (no CVEs between 1.155.0 and 1.157.1; the last advisories were fixed back in 1.152.1), but 1.157.0 fixes an application-service ephemeral-events regression that 1.156.0 shipped — which affects hookshot. No schema-version bump (94 → 94); the three new background updates finished immediately. `experimental_features` (MatrixRTC/Element Call MSCs) was preserved and is unaffected by the v1.157.0 removal of `msc3861`, which we do not use.
+- [x] **Upgrade Synapse to v1.157.2** — Done 2026-07-28. **Security release** (11 advisories: 6 High / 3 Moderate / 2 Low; no CVE IDs assigned). Most High-severity issues require open federation (we federate) — `/get_missing_events` cross-room leak, relayed-401 crypto-state destruction, spoofed to-device, room-alias redirection; two call out untrusted local users (push-rule exhaustion, alias redirection). Pure apt bump: no `# Upgrading to v1.157.2` notes, no schema migration, Python/PG minimums unchanged, config preserved via `--force-confold`. Restarted clean (NRestarts=0), federation healthy. NB: 1.157.2 adds an **optional** `push_rules.limits` config block (safe defaults: rule_count 10000 / rule_id_length 300 / rule_size 1024) for the push-rule-exhaustion advisory — not required, but worth tuning given untrusted local users. Verify with `dpkg-query -W matrix-synapse-py3`, not this file.
 - [x] Migrate from SQLite to PostgreSQL
 
 - [x] TURN/STUN server (coturn) for reliable voice/video
@@ -593,7 +687,7 @@ Periodic `TLS/TCP socket error: Connection reset by peer` in coturn logs. Normal
 
 ### Performance Tuning
 - [x] PostgreSQL `shared_buffers` → 1500MB, `effective_cache_size`, `work_mem`, checkpoint tuning
-- [x] PostgreSQL `pg_stat_statements` extension installed
+- [x] PostgreSQL `pg_stat_statements` extension installed — **was previously only in `shared_preload_libraries`; the extension itself had never been created in the `synapse` DB, so no query-level stats were ever collected.** Fixed 2026-07-26 with `CREATE EXTENSION pg_stat_statements`. Verify with `psql -d synapse -c '\dx pg_stat_statements'`, not just by grepping the config.
 - [x] PostgreSQL autovacuum tuned per-table (5 high-churn tables), `autovacuum_max_workers` → 5
 - [x] Synapse `event_cache_size` → 30K, per-cache factors tuned
 - [x] sysctl TCP/UDP buffer alignment on LXC 151 (`/etc/sysctl.d/99-matrix-tuning.conf`)
@@ -604,7 +698,23 @@ Periodic `TLS/TCP socket error: Connection reset by peer` in coturn logs. Normal
 - [x] TCP retransmit timeout lowered (`tcp_retries2=5`, `tcp_syn_retries=4`, `tcp_keepalive_probes=3`) — stalled outbound federation connections now fail in ~15-30s instead of ~15 min
 - [x] Unreachable routes added for servers with asymmetric connectivity (can reach us but we can't reach their federation port) — prevents 90s TCP hangs from being added to lag; defined in `/etc/network/interfaces` post-up hooks and survive reboots (bark.lgbt ×2, parodia.dev, chat.ohaa.xyz, matrix.k8ekat.dev)
 - [x] Stuck `device_lists_remote_resync` entries cleared for dead-server users (@dalite:bark.lgbt, @arndot:matrix.goch.social) — device list resync was firing every 30s
-- [ ] BBR congestion control — must be applied on Proxmox host
+- [x] BBR congestion control — applied 2026-07-26 on the Proxmox hosts that can run LXC 151 (`compute-storage-01`, `large1`) via `/etc/sysctl.d/99-bbr.conf` (`net.ipv4.tcp_congestion_control=bbr`, `net.core.default_qdisc=fq`). Persists across reboots. If 151 is ever allowed to run on another node, apply it there too.
+- [x] LXC 109 rootfs grown 30GB → 80GB (2026-07-26). It was at **89% full with only 3.4GB free**, and `pg_wal` shares that volume — PostgreSQL stops outright at 100%. Also a hard prerequisite for any `VACUUM FULL`, which must rewrite the table alongside the original.
+- [x] **Purged the `#community:matrix.org` space** (`!iMZEhwCvbfeAYUxAjZ:t2l.io`) 2026-07-26 — 81,724,747 rows / 17GB for a directory wrapper with **zero messages** and one local member. Admin API `DELETE /_synapse/admin/v2/rooms/` with `purge:true`, then `VACUUM FULL`. ~2.5 min downtime.
+- [x] **Compressed room state** with `rust-synapse-compress-state` (built at `/opt/rust-synapse-compress-state/` on LXC 109). Every run verified `New state map matches old one` before commit:
+
+  | Room | Rows before | After | % |
+  |------|-------------|-------|---|
+  | Jellyfin General Chat | 4,997,570 | 681,538 | 13.6% |
+  | Element X Android | 1,069,387 | 150,138 | 14.0% |
+  | `!fTjMjIzNKEsFlUIiru:neko.dev` | 811,608 | 461,347 | 56.8% |
+  | Cinny community | 258,191 | 142,668 | 55.3% |
+  | Proxmox® | 215,475 | 80,046 | 37.2% |
+
+  > **Do NOT compress `Spam and Stuff`.** Its 107,763 state groups hold only 107,799 rows (~1 row each) — already minimal. The compressor's leveled structure would *add* rows (dry run: **110.16%**) and it refuses to run. High state-group count is not the same as compressible bloat.
+
+- [x] PostgreSQL: `jit=off`, `wal_compression=lz4`, `max_worker_processes` 3→8, `maintenance_work_mem` 256MB→512MB, `shared_buffers` 1500MB→2GB in `/etc/postgresql/17/main/conf.d/synapse_tuning.conf`. Previous version kept as `synapse_tuning.conf.bak-20260726`. **That file is not in this repo — the tuning is unversioned.**
+- [ ] `synchronous_commit = off` — **deliberately NOT applied.** Would be a real write-latency win, but this cluster hard-resets every few hours (HA self-fences on corosync quorum loss + the large1 DIMM), so it would drop up to ~600ms of committed messages *per reset* rather than once in a blue moon. Revisit once the cluster is stable.
 
 ### Auth & SSO
 - [x] Token-based registration
@@ -844,7 +954,7 @@ All commands use the `!` prefix. Run `!help` in any room for the full list.
 
 | Component | Technology | Version |
 |-----------|-----------|---------|
-| Homeserver | Synapse | 1.157.1 |
+| Homeserver | Synapse | 1.157.2 |
 | Database | PostgreSQL | 17.9 |
 | TURN | coturn | latest |
 | Video/voice calls | LiveKit SFU | 1.9.11 |