docs: large1 → compute-storage-02; no fixed node for HA guests #69

Merged
jared merged 1 commits from docs/large1-leftovers into main 2026-11-18 16:22:14 -05:00
2 changed files with 16 additions and 15 deletions
Showing only changes of commit b18aae89d9 - Show all commits

No files matched your search

+14 -13
View File
@@ -804,7 +804,7 @@ Periodic `TLS/TCP socket error: Connection reset by peer` in coturn logs. Normal
`net.ipv4.tcp_congestion_control = bbr` must be set on the Proxmox **host**, not inside an unprivileged LXC. All other sysctl tuning (TCP/UDP buffers, fin_timeout) is applied inside LXC 151.
Applied 2026-07-26 via `/etc/sysctl.d/99-bbr.conf` on `compute-storage-01` and `large1`, and
Applied 2026-07-26 via `/etc/sysctl.d/99-bbr.conf` on `compute-storage-01` and `large1` (now `compute-storage-02`), and
**2026-08-16 on `compute-storage-gpu-01`** (which was found on `cubic`/`fq_codel` with `tcp_bbr` not
even loaded). All three nodes that may host LXC 151 now report `bbr`/`fq`. Because the file lives on
the host, **a node that has never hosted 151 will not have it** — check before allowing 151 to migrate
@@ -814,9 +814,9 @@ somewhere new:
sysctl net.ipv4.tcp_congestion_control net.core.default_qdisc # want: bbr / fq
```
### ⚠️ `large1` has a failing DIMM — keep Synapse and PostgreSQL off it
### ⚠️ `compute-storage-02` has a failing DIMM — keep Synapse and PostgreSQL off it
`large1` logs **corrected ECC errors** concentrated on one module (**P0 CHANNEL B, DIMM 1** —
`compute-storage-02` (called `large1` until the 2026-11-18 rename) logs **corrected ECC errors** concentrated on one module (**P0 CHANNEL B, DIMM 1** —
EDAC `mc#0csrow#2channel#1`, 421 corrected errors observed, with the MCE `Over`/overflow flag set
meaning more occurred than were logged). It has hard-crashed **on its own** twice — 2026-07-24 14:19
and 2026-07-26 00:20 — with no clean-shutdown markers and no OOM events, which is the classic
@@ -827,17 +827,18 @@ signature of an uncorrectable ECC error.
Consequences for Matrix:
- On the 2026-07-26 crash, HA relocated **LXC 109 (PostgreSQL)** and **LXC 151 (Synapse)** to
`compute-storage-01`, which has **zero ECC errors across all boots**. They should stay there.
- The HA rule `ha-rule-43cb668b-becf` in `/etc/pve/ha/rules.cfg` currently weights
`compute-storage-01:1,large1:1` — **equal priority, non-strict** — so a future failover can put
Synapse straight back onto the bad-RAM node. Re-weighting to `compute-storage-01:2,large1:1` would
prefer the healthy node while keeping large1 as a fallback. Not changed automatically because that
rule governs 10 resources and `compute-storage-01` has only 46GB RAM vs large1's 128GB.
- LXC 109 (PostgreSQL) and LXC 151 (Synapse) are HA guests and move between nodes; keep them
off compute-storage-02 through the HA rule weights, not by pinning.
- The HA rule `ha-rule-43cb668b-becf` in `/etc/pve/ha/rules.cfg` (checked 2026-11-18) weights
`compute-storage-01:1,compute-storage-02:2,storage-01:2`, so the bad-RAM node is one of the two
**preferred** nodes, and a failover can put Synapse or PostgreSQL onto it. Lowering
`compute-storage-02` below `storage-01` would keep it as a fallback only. Not changed automatically,
because that rule governs 9 CTs and a VM. Look up where 151 and 109 run now with
`ha-manager status | grep -E 'ct:(151|109) '`; they move.
- **Never run a large `VACUUM FULL` or state-compression job on a host with failing RAM** — those
rewrite the whole table through memory, and a bit-flip becomes silent, backup-propagated corruption.
- Ceph's **`noout` flag is intentionally left set.** With 10 OSDs on an intermittently-crashing node,
`noout` prevents a full 10-OSD rebalance every time it bounces. Do not clear it until large1 is stable.
`noout` prevents a full 10-OSD rebalance every time it bounces. Do not clear it until compute-storage-02 is stable.
A replacement DIMM is not currently in budget. Free mitigations worth evaluating: physically removing
the bad module (96GB in flex mode, or 64GB symmetric dual-channel), kernel page soft-offlining via
@@ -1017,7 +1018,7 @@ than nightly compression can keep up with.
- [x] TCP retransmit timeout lowered (`tcp_retries2=5`, `tcp_syn_retries=4`, `tcp_keepalive_probes=3`) — stalled outbound federation connections now fail in ~15-30s instead of ~15 min
- [x] Unreachable routes added for servers with asymmetric connectivity (can reach us but we can't reach their federation port) — prevents 90s TCP hangs from being added to lag; defined in `/etc/network/interfaces` post-up hooks and survive reboots (bark.lgbt ×2, parodia.dev, chat.ohaa.xyz, matrix.k8ekat.dev)
- [x] Stuck `device_lists_remote_resync` entries cleared for dead-server users (@dalite:bark.lgbt, @arndot:matrix.goch.social) — device list resync was firing every 30s
- [x] BBR congestion control — applied 2026-07-26 on `compute-storage-01` and `large1`, and 2026-08-16 on `compute-storage-gpu-01`, via `/etc/sysctl.d/99-bbr.conf` (`net.ipv4.tcp_congestion_control=bbr`, `net.core.default_qdisc=fq`). Persists across reboots. All three nodes that may host LXC 151 are now covered. If 151 is ever allowed to run on another node, apply it there too.
- [x] BBR congestion control — applied 2026-07-26 on `compute-storage-01` and `large1` (now `compute-storage-02`), and 2026-08-16 on `compute-storage-gpu-01`, via `/etc/sysctl.d/99-bbr.conf` (`net.ipv4.tcp_congestion_control=bbr`, `net.core.default_qdisc=fq`). Persists across reboots. All three nodes that may host LXC 151 are now covered. If 151 is ever allowed to run on another node, apply it there too.
- [x] LXC 109 rootfs grown 30GB → 80GB (2026-07-26). It was at **89% full with only 3.4GB free**, and `pg_wal` shares that volume — PostgreSQL stops outright at 100%. Also a hard prerequisite for any `VACUUM FULL`, which must rewrite the table alongside the original.
- [x] **Purged the `#community:matrix.org` space** (`!iMZEhwCvbfeAYUxAjZ:t2l.io`) 2026-07-26 — 81,724,747 rows / 17GB for a directory wrapper with **zero messages** and one local member. Admin API `DELETE /_synapse/admin/v2/rooms/` with `purge:true`, then `VACUUM FULL`. ~2.5 min downtime.
- [x] **Compressed room state** with `rust-synapse-compress-state` (built at `/opt/rust-synapse-compress-state/` on LXC 109). Every run verified `New state map matches old one` before commit:
@@ -1033,7 +1034,7 @@ than nightly compression can keep up with.
> **Do NOT compress `Spam and Stuff`.** Its 107,763 state groups hold only 107,799 rows (~1 row each) — already minimal. The compressor's leveled structure would *add* rows (dry run: **110.16%**) and it refuses to run. High state-group count is not the same as compressible bloat.
- [x] PostgreSQL: `jit=off`, `wal_compression=lz4`, `max_worker_processes` 3→8, `maintenance_work_mem` 256MB→512MB, `shared_buffers` 1500MB→2GB in `/etc/postgresql/17/main/conf.d/synapse_tuning.conf`. Previous version kept as `synapse_tuning.conf.bak-20260726`. **That file is not in this repo — the tuning is unversioned.**
- [ ] `synchronous_commit = off` — **deliberately NOT applied.** Would be a real write-latency win, but this cluster hard-resets every few hours (HA self-fences on corosync quorum loss + the large1 DIMM), so it would drop up to ~600ms of committed messages *per reset* rather than once in a blue moon. Revisit once the cluster is stable.
- [ ] `synchronous_commit = off` — **deliberately NOT applied.** Would be a real write-latency win, but this cluster hard-resets every few hours (HA self-fences on corosync quorum loss + the compute-storage-02 DIMM), so it would drop up to ~600ms of committed messages *per reset* rather than once in a blue moon. Revisit once the cluster is stable.
### Auth & SSO
- [x] Open registration disabled (`enable_registration: false`, 2026-10-16): accounts come from LLDAP and are created on first sign-in through Lotus SSO (Authelia OIDC, which keeps its own registration on). The old registration token was deleted.
+2 -2
View File
@@ -30,8 +30,8 @@ Matrix bot and server improvements for the Lotus Guild homeserver (`matrix.lotus
| Synapse | HA (floats) | 10.10.10.29 | 151 | Homeserver + coturn + LiveKit |
| PostgreSQL 17 | HA (floats) | 10.10.10.44 | 109 | Synapse database backend |
| NPM | HA (floats) | 10.10.10.27 | 139 | Reverse proxy + landing page |
| Authelia | compute-storage-01 | 10.10.10.36 | 167 | SSO/OIDC provider |
| LLDAP | large1 | 10.10.10.39 | 147 | LDAP user directory |
| Authelia | HA (floats) | 10.10.10.36 | 167 | SSO/OIDC provider |
| LLDAP | HA (floats) | 10.10.10.39 | 147 | LDAP user directory |
**Key paths on Synapse LXC (10.10.10.29):**
- Synapse config: `/etc/matrix-synapse/homeserver.yaml`