2026-11-03 19:10:34 -05:00
3 changed files with 64 additions and 0 deletions
@@ -867,6 +867,43 @@ SELECT room_id, count(*) FROM state_groups_state GROUP BY room_id ORDER BY 2 DES
 Spaces (`m.space`) are the worst offenders per byte — they carry huge membership with zero messages.
 Leaving a space does **not** leave its child rooms; they are independent rooms with their own IDs.

+**It grows back.** From 2183 MB after the July cleanup, the database reached 6.2 GB by 2026-11-03
+(`state_groups_state` 2.75 GB, 11 M rows). Almost all of that came from big federated rooms, each with
+one or two local members: *Jellyfin General Chat* (16,754 members, 5.3 M rows), an unnamed 1,213-member
+room (2.4 M), *Homelabbing*, *Element X Android*, the *Cinny community* space.
+
+#### Nightly state compression (CT 109)
+
+`synapse_auto_compressor` (from [rust-synapse-compress-state](https://github.com/matrix-org/rust-synapse-compress-state),
+built from `main`) runs every night at 04:30, after the 03:00 vzdump backup. It does not delete anything:
+it rewrites how a room's state history is stored as smaller deltas.
+
+- **Safety.** It checks that the compressed state matches the original before writing, writes each
+  state group in its own transaction, and skips any chunk that would not shrink.
+- **Progress.** It stores its progress in `state_compressor_*` tables, so each night continues where
+  the last one stopped.
+- **Cost.** No downtime. Synapse's DB p99 stayed at ~10 ms while the initial backlog ran.
+- **First run (2026-11-03).** The initial backlog took ~8 minutes, with 0 Synapse errors.
+  `state_groups_state` went from **11.0 M to 4.25 M rows (−61%)**.
+- **Disk.** The table file stays ~2.77 GB on disk: the freed ~1.75 GB is reused space inside it, so
+  the database stops growing until that is used up. To give it back to the filesystem you need
+  `VACUUM FULL state_groups_state`. That takes an exclusive lock and stalls Synapse for a minute or
+  two, so it is only worth doing if CT 109's disk gets tight (it is at 14%).
+
+Units: `systemd/synapse-state-compressor.{service,timer}` (CT 109 has no auto-deploy hook, so install them by
+hand). Binary: `/usr/local/bin/synapse_auto_compressor`.
+
+```bash
+# on CT 109
+install -m644 systemd/synapse-state-compressor.{service,timer} /etc/systemd/system/
+systemctl daemon-reload && systemctl enable --now synapse-state-compressor.timer
+journalctl -u synapse-state-compressor -n 5     # "Finished running compressor. Saved N rows."
+```
+
+The Grafana alert **Synapse Database Growing Fast** (Matrix → Database) warns if the database grows by
+more than 500 MB in 7 days. That means either the timer stopped running, or a room is growing faster
+than nightly compression can keep up with.
+
 ---

 ## Server Checklist
@@ -0,0 +1,17 @@
+[Unit]
+Description=Compress Synapse state groups (synapse_auto_compressor)
+Documentation=https://github.com/matrix-org/rust-synapse-compress-state
+After=postgresql.service
+
+[Service]
+Type=oneshot
+User=postgres
+# 100 chunks of 500 state groups per night. Progress is stored in the
+# state_compressor_* tables, so each run continues where the last one stopped.
+# Every state group is rewritten in its own transaction, after the compressed
+# state is checked against the original; a chunk that would not shrink is skipped.
+ExecStart=/usr/local/bin/synapse_auto_compressor -p postgresql:///synapse?host=/var/run/postgresql&user=postgres -c 500 -n 100
+Environment=RUST_LOG=info
+Nice=10
+IOSchedulingClass=idle
+TimeoutStartSec=2h
@@ -0,0 +1,10 @@
+[Unit]
+Description=Nightly Synapse state compression (after the 03:00 vzdump backup)
+
+[Timer]
+OnCalendar=*-*-* 04:30
+RandomizedDelaySec=10min
+Persistent=true
+
+[Install]
+WantedBy=timers.target