diff --git a/README.md b/README.md index e91d583..978ca77 100644 --- a/README.md +++ b/README.md @@ -867,6 +867,43 @@ SELECT room_id, count(*) FROM state_groups_state GROUP BY room_id ORDER BY 2 DES Spaces (`m.space`) are the worst offenders per byte — they carry huge membership with zero messages. Leaving a space does **not** leave its child rooms; they are independent rooms with their own IDs. +**It grows back.** From 2183 MB after the July cleanup, the database reached 6.2 GB by 2026-11-03 +(`state_groups_state` 2.75 GB, 11 M rows). Almost all of that came from big federated rooms, each with +one or two local members: *Jellyfin General Chat* (16,754 members, 5.3 M rows), an unnamed 1,213-member +room (2.4 M), *Homelabbing*, *Element X Android*, the *Cinny community* space. + +#### Nightly state compression (CT 109) + +`synapse_auto_compressor` (from [rust-synapse-compress-state](https://github.com/matrix-org/rust-synapse-compress-state), +built from `main`) runs every night at 04:30, after the 03:00 vzdump backup. It does not delete anything: +it rewrites how a room's state history is stored as smaller deltas. + +- **Safety.** It checks that the compressed state matches the original before writing, writes each + state group in its own transaction, and skips any chunk that would not shrink. +- **Progress.** It stores its progress in `state_compressor_*` tables, so each night continues where + the last one stopped. +- **Cost.** No downtime. Synapse's DB p99 stayed at ~10 ms while the initial backlog ran. +- **First run (2026-11-03).** The initial backlog took ~8 minutes, with 0 Synapse errors. + `state_groups_state` went from **11.0 M to 4.25 M rows (−61%)**. +- **Disk.** The table file stays ~2.77 GB on disk: the freed ~1.75 GB is reused space inside it, so + the database stops growing until that is used up. To give it back to the filesystem you need + `VACUUM FULL state_groups_state`. That takes an exclusive lock and stalls Synapse for a minute or + two, so it is only worth doing if CT 109's disk gets tight (it is at 14%). + +Units: `systemd/synapse-state-compressor.{service,timer}` (CT 109 has no auto-deploy hook, so install them by +hand). Binary: `/usr/local/bin/synapse_auto_compressor`. + +```bash +# on CT 109 +install -m644 systemd/synapse-state-compressor.{service,timer} /etc/systemd/system/ +systemctl daemon-reload && systemctl enable --now synapse-state-compressor.timer +journalctl -u synapse-state-compressor -n 5 # "Finished running compressor. Saved N rows." +``` + +The Grafana alert **Synapse Database Growing Fast** (Matrix → Database) warns if the database grows by +more than 500 MB in 7 days. That means either the timer stopped running, or a room is growing faster +than nightly compression can keep up with. + --- ## Server Checklist diff --git a/systemd/synapse-state-compressor.service b/systemd/synapse-state-compressor.service new file mode 100644 index 0000000..9bcde89 --- /dev/null +++ b/systemd/synapse-state-compressor.service @@ -0,0 +1,17 @@ +[Unit] +Description=Compress Synapse state groups (synapse_auto_compressor) +Documentation=https://github.com/matrix-org/rust-synapse-compress-state +After=postgresql.service + +[Service] +Type=oneshot +User=postgres +# 100 chunks of 500 state groups per night. Progress is stored in the +# state_compressor_* tables, so each run continues where the last one stopped. +# Every state group is rewritten in its own transaction, after the compressed +# state is checked against the original; a chunk that would not shrink is skipped. +ExecStart=/usr/local/bin/synapse_auto_compressor -p postgresql:///synapse?host=/var/run/postgresql&user=postgres -c 500 -n 100 +Environment=RUST_LOG=info +Nice=10 +IOSchedulingClass=idle +TimeoutStartSec=2h diff --git a/systemd/synapse-state-compressor.timer b/systemd/synapse-state-compressor.timer new file mode 100644 index 0000000..70ecf30 --- /dev/null +++ b/systemd/synapse-state-compressor.timer @@ -0,0 +1,10 @@ +[Unit] +Description=Nightly Synapse state compression (after the 03:00 vzdump backup) + +[Timer] +OnCalendar=*-*-* 04:30 +RandomizedDelaySec=10min +Persistent=true + +[Install] +WantedBy=timers.target