feat(synapse): nightly state compression on CT 109

The Synapse database grew from 2.2 GB (after the July cleanup) to
6.2 GB, almost all of it state_groups_state from large federated rooms
with one or two local members.

synapse_auto_compressor (matrix-org/rust-synapse-compress-state, built
from main) now runs nightly at 04:30, after the 03:00 vzdump backup.
It verifies compressed state against the original, writes each state
group in its own transaction, skips chunks that would not shrink, and
resumes from stored progress. The initial backlog took ~8 minutes with
no Synapse errors: state_groups_state 11.0 M -> 4.25 M rows (-61%).

Also documents the new "Synapse Database Growing Fast" Grafana alert
(> 500 MB in 7 days).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5

README.md

@@ -867,6 +867,43 @@ SELECT room_id, count(*) FROM state_groups_state GROUP BY room_id ORDER BY 2 DES
 Spaces (`m.space`) are the worst offenders per byte — they carry huge membership with zero messages.
 Leaving a space does **not** leave its child rooms; they are independent rooms with their own IDs.
 
+**It grows back.** From 2183 MB after the July cleanup, the database reached 6.2 GB by 2026-11-03
+(`state_groups_state` 2.75 GB, 11 M rows). Almost all of that came from big federated rooms, each with
+one or two local members: *Jellyfin General Chat* (16,754 members, 5.3 M rows), an unnamed 1,213-member
+room (2.4 M), *Homelabbing*, *Element X Android*, the *Cinny community* space.
+
+#### Nightly state compression (CT 109)
+
+`synapse_auto_compressor` (from [rust-synapse-compress-state](https://github.com/matrix-org/rust-synapse-compress-state),
+built from `main`) runs every night at 04:30, after the 03:00 vzdump backup. It does not delete anything:
+it rewrites how a room's state history is stored as smaller deltas.
+
+- **Safety.** It checks that the compressed state matches the original before writing, writes each
+  state group in its own transaction, and skips any chunk that would not shrink.
+- **Progress.** It stores its progress in `state_compressor_*` tables, so each night continues where
+  the last one stopped.
+- **Cost.** No downtime. Synapse's DB p99 stayed at ~10 ms while the initial backlog ran.
+- **First run (2026-11-03).** The initial backlog took ~8 minutes, with 0 Synapse errors.
+  `state_groups_state` went from **11.0 M to 4.25 M rows (−61%)**.
+- **Disk.** The table file stays ~2.77 GB on disk: the freed ~1.75 GB is reused space inside it, so
+  the database stops growing until that is used up. To give it back to the filesystem you need
+  `VACUUM FULL state_groups_state`. That takes an exclusive lock and stalls Synapse for a minute or
+  two, so it is only worth doing if CT 109's disk gets tight (it is at 14%).
+
+Units: `systemd/synapse-state-compressor.{service,timer}` (CT 109 has no auto-deploy hook, so install them by
+hand). Binary: `/usr/local/bin/synapse_auto_compressor`.
+
+```bash
+# on CT 109
+install -m644 systemd/synapse-state-compressor.{service,timer} /etc/systemd/system/
+systemctl daemon-reload && systemctl enable --now synapse-state-compressor.timer
+journalctl -u synapse-state-compressor -n 5     # "Finished running compressor. Saved N rows."
+```
+
+The Grafana alert **Synapse Database Growing Fast** (Matrix → Database) warns if the database grows by
+more than 500 MB in 7 days. That means either the timer stopped running, or a room is growing faster
+than nightly compression can keep up with.
+
 ---
 
 ## Server Checklist

systemd/synapse-state-compressor.service

@@ -0,0 +1,17 @@
+[Unit]
+Description=Compress Synapse state groups (synapse_auto_compressor)
+Documentation=https://github.com/matrix-org/rust-synapse-compress-state
+After=postgresql.service
+
+[Service]
+Type=oneshot
+User=postgres
+# 100 chunks of 500 state groups per night. Progress is stored in the
+# state_compressor_* tables, so each run continues where the last one stopped.
+# Every state group is rewritten in its own transaction, after the compressed
+# state is checked against the original; a chunk that would not shrink is skipped.
+ExecStart=/usr/local/bin/synapse_auto_compressor -p postgresql:///synapse?host=/var/run/postgresql&user=postgres -c 500 -n 100
+Environment=RUST_LOG=info
+Nice=10
+IOSchedulingClass=idle
+TimeoutStartSec=2h

systemd/synapse-state-compressor.timer

@@ -0,0 +1,10 @@
+[Unit]
+Description=Nightly Synapse state compression (after the 03:00 vzdump backup)
+
+[Timer]
+OnCalendar=*-*-* 04:30
+RandomizedDelaySec=10min
+Persistent=true
+
+[Install]
+WantedBy=timers.target