diff --git a/README.md b/README.md index a4d458d..0c0fca6 100644 --- a/README.md +++ b/README.md @@ -50,8 +50,8 @@ matrix/ | Service | IP | LXC | RAM | Disk | Versions | |---------|----|-----|-----|------|----------| -| Synapse | 10.10.10.29 | 151 | 8GB | 50GB | Synapse 1.157.1, LiveKit 1.9.11, hookshot 7.3.2, coturn latest | -| PostgreSQL 17 | 10.10.10.44 | 109 | 6GB | 30GB | PostgreSQL 17.9 | +| Synapse | 10.10.10.29 | 151 | 12GB | 50GB | Synapse 1.157.2, LiveKit 1.9.11, hookshot 7.3.2, coturn latest | +| PostgreSQL 17 | 10.10.10.44 | 109 | 6GB | 80GB | PostgreSQL 17.9 | | Cinny Web | 10.10.10.6 | 106 | 2GB | 8GB | Debian 12, nginx, Node 24, Lotus Cinny fork (custom, tracks `cinnyapp/cinny` main) | | Draupnir | 10.10.10.24 | 110 | 1GB | 10GB | Draupnir v2.9.0, Node.js v22 | | Prometheus | 10.10.10.48 | 118 | — | — | Prometheus — scrapes all Matrix services | @@ -563,7 +563,100 @@ Periodic `TLS/TCP socket error: Connection reset by peer` in coturn logs. Normal ### BBR Congestion Control -`net.ipv4.tcp_congestion_control = bbr` must be set on the Proxmox host, not inside an unprivileged LXC. All other sysctl tuning (TCP/UDP buffers, fin_timeout) is applied inside LXC 151. +`net.ipv4.tcp_congestion_control = bbr` must be set on the Proxmox **host**, not inside an unprivileged LXC. All other sysctl tuning (TCP/UDP buffers, fin_timeout) is applied inside LXC 151. + +Applied 2026-07-26 via `/etc/sysctl.d/99-bbr.conf` on `compute-storage-01` and `large1` (the two +nodes the HA affinity rule allows LXC 151 to run on). Because the file lives on the host, **a node +that has never hosted 151 will not have it** — check before allowing 151 to migrate somewhere new: + +```bash +sysctl net.ipv4.tcp_congestion_control net.core.default_qdisc # want: bbr / fq +``` + +### ⚠️ `large1` has a failing DIMM — keep Synapse and PostgreSQL off it + +`large1` logs **corrected ECC errors** concentrated on one module (**P0 CHANNEL B, DIMM 1** — +EDAC `mc#0csrow#2channel#1`, 421 corrected errors observed, with the MCE `Over`/overflow flag set +meaning more occurred than were logged). It has hard-crashed **on its own** twice — 2026-07-24 14:19 +and 2026-07-26 00:20 — with no clean-shutdown markers and no OOM events, which is the classic +signature of an uncorrectable ECC error. + +> Do **not** confuse these with the cluster-wide resets on 2026-07-22 01:56 and 2026-07-24 11:01-11:45, +> which took down **all six nodes** simultaneously and were power events, not host hardware. + +Consequences for Matrix: + +- On the 2026-07-26 crash, HA relocated **LXC 109 (PostgreSQL)** and **LXC 151 (Synapse)** to + `compute-storage-01`, which has **zero ECC errors across all boots**. They should stay there. +- The HA rule `ha-rule-43cb668b-becf` in `/etc/pve/ha/rules.cfg` currently weights + `compute-storage-01:1,large1:1` — **equal priority, non-strict** — so a future failover can put + Synapse straight back onto the bad-RAM node. Re-weighting to `compute-storage-01:2,large1:1` would + prefer the healthy node while keeping large1 as a fallback. Not changed automatically because that + rule governs 10 resources and `compute-storage-01` has only 46GB RAM vs large1's 128GB. +- **Never run a large `VACUUM FULL` or state-compression job on a host with failing RAM** — those + rewrite the whole table through memory, and a bit-flip becomes silent, backup-propagated corruption. +- Ceph's **`noout` flag is intentionally left set.** With 10 OSDs on an intermittently-crashing node, + `noout` prevents a full 10-OSD rebalance every time it bounces. Do not clear it until large1 is stable. + +A replacement DIMM is not currently in budget. Free mitigations worth evaluating: physically removing +the bad module (96GB in flex mode, or 64GB symmetric dual-channel), kernel page soft-offlining via +`rasdaemon` (the errors cluster in a small set of physical pages), or a `memmap=` blacklist. + +### PostgreSQL `state_groups_state` bloat + +Matrix state resolution is the dominant cost on this homeserver, not message volume. + +**2026-07-26 cleanup — database went 22GB → 2183MB (−90%):** + +| Metric | Before | After | +|--------|--------|-------| +| Database size | 22 GB | **2183 MB** | +| `state_groups_state` | 20 GB | **540 MB** | +| `state_groups_state` rows | 89,543,584 | **2,325,198** | +| LXC 109 disk used | 25 G (89%) | **8.5 G (12%)** | + +Two independent wins: purging one useless space (17GB) and compressing five large federated +rooms (a further ~1.3GB). Total downtime across the whole operation was **~3.5 minutes**. + +**Runtime effect** (Prometheus, 20-minute window vs. the same window 24h earlier): + +| Metric | Before | After | +|--------|--------|-------| +| `get_auth_chain_difference_chains` DB time | 0.0508 s/s | **0.0 — never called** | +| `_get_state_groups_from_groups` DB time | 0.03 s/s | 0.0004 s/s | +| Reactor tick (avg) | 1.18 ms | **0.52 ms** | +| Peak CPU | 0.42 cores | 0.16 cores | +| `stateGroupMembersCache` | 974,953/1M **FULL**, 8,007,530 evictions, 28% hit | 104 entries, **0 evictions** | +| `*getEvent*` | 61,440/61,440 **FULL**, 546,724 evictions | 286 entries, **0 evictions** | + +> Read the "after" cache hit-rates (30-60%) with care: they are *low* only because the caches are +> now nearly empty and cold-miss dominated. The number that matters is **evictions: 8 million → 0**. +> `event_cache_size` was left at `30K` — raising it would be pointless when the cache sits at +> 286/61,440. Tune caches against eviction counts, not hit rates. + +> Caveat on the before/after comparison: the two windows are different times of day on a 26-user +> server, so reactor-tick and CPU deltas are directional rather than exact. The +> `get_auth_chain_difference_chains` result is structural, not load-dependent — that room is gone. + +**Worker split deliberately NOT done.** Synapse remains monolithic (no Redis, no `instance_map`). +It was on the table because peak CPU hit 0.96 cores — the single-process GIL ceiling. After this +cleanup, peak is 0.16 cores, so the Redis + generic_workers + `federation_sender` + NPM routing +machinery would be operational complexity for no gain. Revisit only if reactor tick returns to ~1ms. + +The cost is driven almost entirely by **large federated rooms**, and it scales with the room's +*global* member count, not with how much you actually use it. One local user joined to a huge public +room makes the server pay that room's full state-resolution cost forever. Because Synapse's caches, +GIL, DB pool, and PostgreSQL buffer cache are all **global with no per-room fairness**, a single such +room degrades every other room on the server. + +To find the offenders: + +```sql +SELECT room_id, count(*) FROM state_groups_state GROUP BY room_id ORDER BY 2 DESC LIMIT 10; +``` + +Spaces (`m.space`) are the worst offenders per byte — they carry huge membership with zero messages. +Leaving a space does **not** leave its child rooms; they are independent rooms with their own IDs. --- @@ -574,6 +667,7 @@ Periodic `TLS/TCP socket error: Connection reset by peer` in coturn logs. Normal ### Quality of Life - [x] **Upgrade Synapse to v1.155.0** — Done 2026-06-18. LXC 151 was already on Debian 13 Trixie; no OS migration needed. - [x] **Upgrade Synapse to v1.157.1** — Done 2026-07-23. Note: the host was found running **1.156.0**, not the 1.155.0 recorded here — an intermediate upgrade had happened without being logged, so always verify with `dpkg-query -W matrix-synapse-py3` rather than trusting this file. Not a security release (no CVEs between 1.155.0 and 1.157.1; the last advisories were fixed back in 1.152.1), but 1.157.0 fixes an application-service ephemeral-events regression that 1.156.0 shipped — which affects hookshot. No schema-version bump (94 → 94); the three new background updates finished immediately. `experimental_features` (MatrixRTC/Element Call MSCs) was preserved and is unaffected by the v1.157.0 removal of `msc3861`, which we do not use. +- [x] **Upgrade Synapse to v1.157.2** — Done 2026-07-28. **Security release** (11 advisories: 6 High / 3 Moderate / 2 Low; no CVE IDs assigned). Most High-severity issues require open federation (we federate) — `/get_missing_events` cross-room leak, relayed-401 crypto-state destruction, spoofed to-device, room-alias redirection; two call out untrusted local users (push-rule exhaustion, alias redirection). Pure apt bump: no `# Upgrading to v1.157.2` notes, no schema migration, Python/PG minimums unchanged, config preserved via `--force-confold`. Restarted clean (NRestarts=0), federation healthy. NB: 1.157.2 adds an **optional** `push_rules.limits` config block (safe defaults: rule_count 10000 / rule_id_length 300 / rule_size 1024) for the push-rule-exhaustion advisory — not required, but worth tuning given untrusted local users. Verify with `dpkg-query -W matrix-synapse-py3`, not this file. - [x] Migrate from SQLite to PostgreSQL - [x] TURN/STUN server (coturn) for reliable voice/video @@ -593,7 +687,7 @@ Periodic `TLS/TCP socket error: Connection reset by peer` in coturn logs. Normal ### Performance Tuning - [x] PostgreSQL `shared_buffers` → 1500MB, `effective_cache_size`, `work_mem`, checkpoint tuning -- [x] PostgreSQL `pg_stat_statements` extension installed +- [x] PostgreSQL `pg_stat_statements` extension installed — **was previously only in `shared_preload_libraries`; the extension itself had never been created in the `synapse` DB, so no query-level stats were ever collected.** Fixed 2026-07-26 with `CREATE EXTENSION pg_stat_statements`. Verify with `psql -d synapse -c '\dx pg_stat_statements'`, not just by grepping the config. - [x] PostgreSQL autovacuum tuned per-table (5 high-churn tables), `autovacuum_max_workers` → 5 - [x] Synapse `event_cache_size` → 30K, per-cache factors tuned - [x] sysctl TCP/UDP buffer alignment on LXC 151 (`/etc/sysctl.d/99-matrix-tuning.conf`) @@ -604,7 +698,23 @@ Periodic `TLS/TCP socket error: Connection reset by peer` in coturn logs. Normal - [x] TCP retransmit timeout lowered (`tcp_retries2=5`, `tcp_syn_retries=4`, `tcp_keepalive_probes=3`) — stalled outbound federation connections now fail in ~15-30s instead of ~15 min - [x] Unreachable routes added for servers with asymmetric connectivity (can reach us but we can't reach their federation port) — prevents 90s TCP hangs from being added to lag; defined in `/etc/network/interfaces` post-up hooks and survive reboots (bark.lgbt ×2, parodia.dev, chat.ohaa.xyz, matrix.k8ekat.dev) - [x] Stuck `device_lists_remote_resync` entries cleared for dead-server users (@dalite:bark.lgbt, @arndot:matrix.goch.social) — device list resync was firing every 30s -- [ ] BBR congestion control — must be applied on Proxmox host +- [x] BBR congestion control — applied 2026-07-26 on the Proxmox hosts that can run LXC 151 (`compute-storage-01`, `large1`) via `/etc/sysctl.d/99-bbr.conf` (`net.ipv4.tcp_congestion_control=bbr`, `net.core.default_qdisc=fq`). Persists across reboots. If 151 is ever allowed to run on another node, apply it there too. +- [x] LXC 109 rootfs grown 30GB → 80GB (2026-07-26). It was at **89% full with only 3.4GB free**, and `pg_wal` shares that volume — PostgreSQL stops outright at 100%. Also a hard prerequisite for any `VACUUM FULL`, which must rewrite the table alongside the original. +- [x] **Purged the `#community:matrix.org` space** (`!iMZEhwCvbfeAYUxAjZ:t2l.io`) 2026-07-26 — 81,724,747 rows / 17GB for a directory wrapper with **zero messages** and one local member. Admin API `DELETE /_synapse/admin/v2/rooms/` with `purge:true`, then `VACUUM FULL`. ~2.5 min downtime. +- [x] **Compressed room state** with `rust-synapse-compress-state` (built at `/opt/rust-synapse-compress-state/` on LXC 109). Every run verified `New state map matches old one` before commit: + + | Room | Rows before | After | % | + |------|-------------|-------|---| + | Jellyfin General Chat | 4,997,570 | 681,538 | 13.6% | + | Element X Android | 1,069,387 | 150,138 | 14.0% | + | `!fTjMjIzNKEsFlUIiru:neko.dev` | 811,608 | 461,347 | 56.8% | + | Cinny community | 258,191 | 142,668 | 55.3% | + | Proxmox® | 215,475 | 80,046 | 37.2% | + + > **Do NOT compress `Spam and Stuff`.** Its 107,763 state groups hold only 107,799 rows (~1 row each) — already minimal. The compressor's leveled structure would *add* rows (dry run: **110.16%**) and it refuses to run. High state-group count is not the same as compressible bloat. + +- [x] PostgreSQL: `jit=off`, `wal_compression=lz4`, `max_worker_processes` 3→8, `maintenance_work_mem` 256MB→512MB, `shared_buffers` 1500MB→2GB in `/etc/postgresql/17/main/conf.d/synapse_tuning.conf`. Previous version kept as `synapse_tuning.conf.bak-20260726`. **That file is not in this repo — the tuning is unversioned.** +- [ ] `synchronous_commit = off` — **deliberately NOT applied.** Would be a real write-latency win, but this cluster hard-resets every few hours (HA self-fences on corosync quorum loss + the large1 DIMM), so it would drop up to ~600ms of committed messages *per reset* rather than once in a blue moon. Revisit once the cluster is stable. ### Auth & SSO - [x] Token-based registration @@ -844,7 +954,7 @@ All commands use the `!` prefix. Run `!help` in any room for the full list. | Component | Technology | Version | |-----------|-----------|---------| -| Homeserver | Synapse | 1.157.1 | +| Homeserver | Synapse | 1.157.2 | | Database | PostgreSQL | 17.9 | | TURN | coturn | latest | | Video/voice calls | LiveKit SFU | 1.9.11 |