From b18aae89d9aba36dc1c07dff78b0b08f79c8bc11 Mon Sep 17 00:00:00 2001 From: Jared Vititoe Date: Wed, 18 Nov 2026 16:15:33 -0500 Subject: [PATCH] docs: remove large1 leftovers; no node names for HA guests large1 was renamed to compute-storage-02 on 2026-11-18. Updates text that still named large1. HA guests no longer carry a fixed node (they move between nodes). Co-Authored-By: Claude Opus 5.5 --- README.md | 27 ++++++++++++++------------- matrixbot/README.md | 4 ++-- 2 files changed, 16 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index 112578a..e7e51d3 100644 --- a/README.md +++ b/README.md @@ -804,7 +804,7 @@ Periodic `TLS/TCP socket error: Connection reset by peer` in coturn logs. Normal `net.ipv4.tcp_congestion_control = bbr` must be set on the Proxmox **host**, not inside an unprivileged LXC. All other sysctl tuning (TCP/UDP buffers, fin_timeout) is applied inside LXC 151. -Applied 2026-07-26 via `/etc/sysctl.d/99-bbr.conf` on `compute-storage-01` and `large1`, and +Applied 2026-07-26 via `/etc/sysctl.d/99-bbr.conf` on `compute-storage-01` and `large1` (now `compute-storage-02`), and **2026-08-16 on `compute-storage-gpu-01`** (which was found on `cubic`/`fq_codel` with `tcp_bbr` not even loaded). All three nodes that may host LXC 151 now report `bbr`/`fq`. Because the file lives on the host, **a node that has never hosted 151 will not have it** — check before allowing 151 to migrate @@ -814,9 +814,9 @@ somewhere new: sysctl net.ipv4.tcp_congestion_control net.core.default_qdisc # want: bbr / fq ``` -### ⚠️ `large1` has a failing DIMM — keep Synapse and PostgreSQL off it +### ⚠️ `compute-storage-02` has a failing DIMM — keep Synapse and PostgreSQL off it -`large1` logs **corrected ECC errors** concentrated on one module (**P0 CHANNEL B, DIMM 1** — +`compute-storage-02` (called `large1` until the 2026-11-18 rename) logs **corrected ECC errors** concentrated on one module (**P0 CHANNEL B, DIMM 1** — EDAC `mc#0csrow#2channel#1`, 421 corrected errors observed, with the MCE `Over`/overflow flag set meaning more occurred than were logged). It has hard-crashed **on its own** twice — 2026-07-24 14:19 and 2026-07-26 00:20 — with no clean-shutdown markers and no OOM events, which is the classic @@ -827,17 +827,18 @@ signature of an uncorrectable ECC error. Consequences for Matrix: -- On the 2026-07-26 crash, HA relocated **LXC 109 (PostgreSQL)** and **LXC 151 (Synapse)** to - `compute-storage-01`, which has **zero ECC errors across all boots**. They should stay there. -- The HA rule `ha-rule-43cb668b-becf` in `/etc/pve/ha/rules.cfg` currently weights - `compute-storage-01:1,large1:1` — **equal priority, non-strict** — so a future failover can put - Synapse straight back onto the bad-RAM node. Re-weighting to `compute-storage-01:2,large1:1` would - prefer the healthy node while keeping large1 as a fallback. Not changed automatically because that - rule governs 10 resources and `compute-storage-01` has only 46GB RAM vs large1's 128GB. +- LXC 109 (PostgreSQL) and LXC 151 (Synapse) are HA guests and move between nodes; keep them + off compute-storage-02 through the HA rule weights, not by pinning. +- The HA rule `ha-rule-43cb668b-becf` in `/etc/pve/ha/rules.cfg` (checked 2026-11-18) weights + `compute-storage-01:1,compute-storage-02:2,storage-01:2`, so the bad-RAM node is one of the two + **preferred** nodes, and a failover can put Synapse or PostgreSQL onto it. Lowering + `compute-storage-02` below `storage-01` would keep it as a fallback only. Not changed automatically, + because that rule governs 9 CTs and a VM. Look up where 151 and 109 run now with + `ha-manager status | grep -E 'ct:(151|109) '`; they move. - **Never run a large `VACUUM FULL` or state-compression job on a host with failing RAM** — those rewrite the whole table through memory, and a bit-flip becomes silent, backup-propagated corruption. - Ceph's **`noout` flag is intentionally left set.** With 10 OSDs on an intermittently-crashing node, - `noout` prevents a full 10-OSD rebalance every time it bounces. Do not clear it until large1 is stable. + `noout` prevents a full 10-OSD rebalance every time it bounces. Do not clear it until compute-storage-02 is stable. A replacement DIMM is not currently in budget. Free mitigations worth evaluating: physically removing the bad module (96GB in flex mode, or 64GB symmetric dual-channel), kernel page soft-offlining via @@ -1017,7 +1018,7 @@ than nightly compression can keep up with. - [x] TCP retransmit timeout lowered (`tcp_retries2=5`, `tcp_syn_retries=4`, `tcp_keepalive_probes=3`) — stalled outbound federation connections now fail in ~15-30s instead of ~15 min - [x] Unreachable routes added for servers with asymmetric connectivity (can reach us but we can't reach their federation port) — prevents 90s TCP hangs from being added to lag; defined in `/etc/network/interfaces` post-up hooks and survive reboots (bark.lgbt ×2, parodia.dev, chat.ohaa.xyz, matrix.k8ekat.dev) - [x] Stuck `device_lists_remote_resync` entries cleared for dead-server users (@dalite:bark.lgbt, @arndot:matrix.goch.social) — device list resync was firing every 30s -- [x] BBR congestion control — applied 2026-07-26 on `compute-storage-01` and `large1`, and 2026-08-16 on `compute-storage-gpu-01`, via `/etc/sysctl.d/99-bbr.conf` (`net.ipv4.tcp_congestion_control=bbr`, `net.core.default_qdisc=fq`). Persists across reboots. All three nodes that may host LXC 151 are now covered. If 151 is ever allowed to run on another node, apply it there too. +- [x] BBR congestion control — applied 2026-07-26 on `compute-storage-01` and `large1` (now `compute-storage-02`), and 2026-08-16 on `compute-storage-gpu-01`, via `/etc/sysctl.d/99-bbr.conf` (`net.ipv4.tcp_congestion_control=bbr`, `net.core.default_qdisc=fq`). Persists across reboots. All three nodes that may host LXC 151 are now covered. If 151 is ever allowed to run on another node, apply it there too. - [x] LXC 109 rootfs grown 30GB → 80GB (2026-07-26). It was at **89% full with only 3.4GB free**, and `pg_wal` shares that volume — PostgreSQL stops outright at 100%. Also a hard prerequisite for any `VACUUM FULL`, which must rewrite the table alongside the original. - [x] **Purged the `#community:matrix.org` space** (`!iMZEhwCvbfeAYUxAjZ:t2l.io`) 2026-07-26 — 81,724,747 rows / 17GB for a directory wrapper with **zero messages** and one local member. Admin API `DELETE /_synapse/admin/v2/rooms/` with `purge:true`, then `VACUUM FULL`. ~2.5 min downtime. - [x] **Compressed room state** with `rust-synapse-compress-state` (built at `/opt/rust-synapse-compress-state/` on LXC 109). Every run verified `New state map matches old one` before commit: @@ -1033,7 +1034,7 @@ than nightly compression can keep up with. > **Do NOT compress `Spam and Stuff`.** Its 107,763 state groups hold only 107,799 rows (~1 row each) — already minimal. The compressor's leveled structure would *add* rows (dry run: **110.16%**) and it refuses to run. High state-group count is not the same as compressible bloat. - [x] PostgreSQL: `jit=off`, `wal_compression=lz4`, `max_worker_processes` 3→8, `maintenance_work_mem` 256MB→512MB, `shared_buffers` 1500MB→2GB in `/etc/postgresql/17/main/conf.d/synapse_tuning.conf`. Previous version kept as `synapse_tuning.conf.bak-20260726`. **That file is not in this repo — the tuning is unversioned.** -- [ ] `synchronous_commit = off` — **deliberately NOT applied.** Would be a real write-latency win, but this cluster hard-resets every few hours (HA self-fences on corosync quorum loss + the large1 DIMM), so it would drop up to ~600ms of committed messages *per reset* rather than once in a blue moon. Revisit once the cluster is stable. +- [ ] `synchronous_commit = off` — **deliberately NOT applied.** Would be a real write-latency win, but this cluster hard-resets every few hours (HA self-fences on corosync quorum loss + the compute-storage-02 DIMM), so it would drop up to ~600ms of committed messages *per reset* rather than once in a blue moon. Revisit once the cluster is stable. ### Auth & SSO - [x] Open registration disabled (`enable_registration: false`, 2026-10-16): accounts come from LLDAP and are created on first sign-in through Lotus SSO (Authelia OIDC, which keeps its own registration on). The old registration token was deleted. diff --git a/matrixbot/README.md b/matrixbot/README.md index 41165b6..74f4bb3 100644 --- a/matrixbot/README.md +++ b/matrixbot/README.md @@ -30,8 +30,8 @@ Matrix bot and server improvements for the Lotus Guild homeserver (`matrix.lotus | Synapse | HA (floats) | 10.10.10.29 | 151 | Homeserver + coturn + LiveKit | | PostgreSQL 17 | HA (floats) | 10.10.10.44 | 109 | Synapse database backend | | NPM | HA (floats) | 10.10.10.27 | 139 | Reverse proxy + landing page | -| Authelia | compute-storage-01 | 10.10.10.36 | 167 | SSO/OIDC provider | -| LLDAP | large1 | 10.10.10.39 | 147 | LDAP user directory | +| Authelia | HA (floats) | 10.10.10.36 | 167 | SSO/OIDC provider | +| LLDAP | HA (floats) | 10.10.10.39 | 147 | LDAP user directory | **Key paths on Synapse LXC (10.10.10.29):** - Synapse config: `/etc/matrix-synapse/homeserver.yaml`