From b2877323c3891da20af0b87dc865b80e8ebc5f42 Mon Sep 17 00:00:00 2001 From: Lotus CI Date: Tue, 3 Nov 2026 19:08:01 -0500 Subject: [PATCH] feat(synapse): nightly state compression on CT 109 The Synapse database grew from 2.2 GB (after the July cleanup) to 6.2 GB, almost all of it state_groups_state from large federated rooms with one or two local members. synapse_auto_compressor (matrix-org/rust-synapse-compress-state, built from main) now runs nightly at 04:30, after the 03:00 vzdump backup. It verifies compressed state against the original, writes each state group in its own transaction, skips chunks that would not shrink, and resumes from stored progress. The initial backlog took ~8 minutes with no Synapse errors: state_groups_state 11.0 M -> 4.25 M rows (-61%). Also documents the new "Synapse Database Growing Fast" Grafana alert (> 500 MB in 7 days). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5 --- README.md | 37 ++++++++++++++++++++++++ systemd/synapse-state-compressor.service | 17 +++++++++++ systemd/synapse-state-compressor.timer | 10 +++++++ 3 files changed, 64 insertions(+) create mode 100644 systemd/synapse-state-compressor.service create mode 100644 systemd/synapse-state-compressor.timer diff --git a/README.md b/README.md index e91d583..978ca77 100644 --- a/README.md +++ b/README.md @@ -867,6 +867,43 @@ SELECT room_id, count(*) FROM state_groups_state GROUP BY room_id ORDER BY 2 DES Spaces (`m.space`) are the worst offenders per byte — they carry huge membership with zero messages. Leaving a space does **not** leave its child rooms; they are independent rooms with their own IDs. +**It grows back.** From 2183 MB after the July cleanup, the database reached 6.2 GB by 2026-11-03 +(`state_groups_state` 2.75 GB, 11 M rows). Almost all of that came from big federated rooms, each with +one or two local members: *Jellyfin General Chat* (16,754 members, 5.3 M rows), an unnamed 1,213-member +room (2.4 M), *Homelabbing*, *Element X Android*, the *Cinny community* space. + +#### Nightly state compression (CT 109) + +`synapse_auto_compressor` (from [rust-synapse-compress-state](https://github.com/matrix-org/rust-synapse-compress-state), +built from `main`) runs every night at 04:30, after the 03:00 vzdump backup. It does not delete anything: +it rewrites how a room's state history is stored as smaller deltas. + +- **Safety.** It checks that the compressed state matches the original before writing, writes each + state group in its own transaction, and skips any chunk that would not shrink. +- **Progress.** It stores its progress in `state_compressor_*` tables, so each night continues where + the last one stopped. +- **Cost.** No downtime. Synapse's DB p99 stayed at ~10 ms while the initial backlog ran. +- **First run (2026-11-03).** The initial backlog took ~8 minutes, with 0 Synapse errors. + `state_groups_state` went from **11.0 M to 4.25 M rows (−61%)**. +- **Disk.** The table file stays ~2.77 GB on disk: the freed ~1.75 GB is reused space inside it, so + the database stops growing until that is used up. To give it back to the filesystem you need + `VACUUM FULL state_groups_state`. That takes an exclusive lock and stalls Synapse for a minute or + two, so it is only worth doing if CT 109's disk gets tight (it is at 14%). + +Units: `systemd/synapse-state-compressor.{service,timer}` (CT 109 has no auto-deploy hook, so install them by +hand). Binary: `/usr/local/bin/synapse_auto_compressor`. + +```bash +# on CT 109 +install -m644 systemd/synapse-state-compressor.{service,timer} /etc/systemd/system/ +systemctl daemon-reload && systemctl enable --now synapse-state-compressor.timer +journalctl -u synapse-state-compressor -n 5 # "Finished running compressor. Saved N rows." +``` + +The Grafana alert **Synapse Database Growing Fast** (Matrix → Database) warns if the database grows by +more than 500 MB in 7 days. That means either the timer stopped running, or a room is growing faster +than nightly compression can keep up with. + --- ## Server Checklist diff --git a/systemd/synapse-state-compressor.service b/systemd/synapse-state-compressor.service new file mode 100644 index 0000000..9bcde89 --- /dev/null +++ b/systemd/synapse-state-compressor.service @@ -0,0 +1,17 @@ +[Unit] +Description=Compress Synapse state groups (synapse_auto_compressor) +Documentation=https://github.com/matrix-org/rust-synapse-compress-state +After=postgresql.service + +[Service] +Type=oneshot +User=postgres +# 100 chunks of 500 state groups per night. Progress is stored in the +# state_compressor_* tables, so each run continues where the last one stopped. +# Every state group is rewritten in its own transaction, after the compressed +# state is checked against the original; a chunk that would not shrink is skipped. +ExecStart=/usr/local/bin/synapse_auto_compressor -p postgresql:///synapse?host=/var/run/postgresql&user=postgres -c 500 -n 100 +Environment=RUST_LOG=info +Nice=10 +IOSchedulingClass=idle +TimeoutStartSec=2h diff --git a/systemd/synapse-state-compressor.timer b/systemd/synapse-state-compressor.timer new file mode 100644 index 0000000..70ecf30 --- /dev/null +++ b/systemd/synapse-state-compressor.timer @@ -0,0 +1,10 @@ +[Unit] +Description=Nightly Synapse state compression (after the 03:00 vzdump backup) + +[Timer] +OnCalendar=*-*-* 04:30 +RandomizedDelaySec=10min +Persistent=true + +[Install] +WantedBy=timers.target