feat(synapse): nightly state compression on CT 109 #59

Merged
jared merged 1 commits from feat/synapse-state-compressor into main 2026-11-03 19:10:34 -05:00
3 changed files with 64 additions and 0 deletions
+37
View File
@@ -867,6 +867,43 @@ SELECT room_id, count(*) FROM state_groups_state GROUP BY room_id ORDER BY 2 DES
Spaces (`m.space`) are the worst offenders per byte — they carry huge membership with zero messages. Spaces (`m.space`) are the worst offenders per byte — they carry huge membership with zero messages.
Leaving a space does **not** leave its child rooms; they are independent rooms with their own IDs. Leaving a space does **not** leave its child rooms; they are independent rooms with their own IDs.
**It grows back.** From 2183 MB after the July cleanup, the database reached 6.2 GB by 2026-11-03
(`state_groups_state` 2.75 GB, 11 M rows). Almost all of that came from big federated rooms, each with
one or two local members: *Jellyfin General Chat* (16,754 members, 5.3 M rows), an unnamed 1,213-member
room (2.4 M), *Homelabbing*, *Element X Android*, the *Cinny community* space.
#### Nightly state compression (CT 109)
`synapse_auto_compressor` (from [rust-synapse-compress-state](https://github.com/matrix-org/rust-synapse-compress-state),
built from `main`) runs every night at 04:30, after the 03:00 vzdump backup. It does not delete anything:
it rewrites how a room's state history is stored as smaller deltas.
- **Safety.** It checks that the compressed state matches the original before writing, writes each
state group in its own transaction, and skips any chunk that would not shrink.
- **Progress.** It stores its progress in `state_compressor_*` tables, so each night continues where
the last one stopped.
- **Cost.** No downtime. Synapse's DB p99 stayed at ~10 ms while the initial backlog ran.
- **First run (2026-11-03).** The initial backlog took ~8 minutes, with 0 Synapse errors.
`state_groups_state` went from **11.0 M to 4.25 M rows (−61%)**.
- **Disk.** The table file stays ~2.77 GB on disk: the freed ~1.75 GB is reused space inside it, so
the database stops growing until that is used up. To give it back to the filesystem you need
`VACUUM FULL state_groups_state`. That takes an exclusive lock and stalls Synapse for a minute or
two, so it is only worth doing if CT 109's disk gets tight (it is at 14%).
Units: `systemd/synapse-state-compressor.{service,timer}` (CT 109 has no auto-deploy hook, so install them by
hand). Binary: `/usr/local/bin/synapse_auto_compressor`.
```bash
# on CT 109
install -m644 systemd/synapse-state-compressor.{service,timer} /etc/systemd/system/
systemctl daemon-reload && systemctl enable --now synapse-state-compressor.timer
journalctl -u synapse-state-compressor -n 5 # "Finished running compressor. Saved N rows."
```
The Grafana alert **Synapse Database Growing Fast** (Matrix → Database) warns if the database grows by
more than 500 MB in 7 days. That means either the timer stopped running, or a room is growing faster
than nightly compression can keep up with.
--- ---
## Server Checklist ## Server Checklist
+17
View File
@@ -0,0 +1,17 @@
[Unit]
Description=Compress Synapse state groups (synapse_auto_compressor)
Documentation=https://github.com/matrix-org/rust-synapse-compress-state
After=postgresql.service
[Service]
Type=oneshot
User=postgres
# 100 chunks of 500 state groups per night. Progress is stored in the
# state_compressor_* tables, so each run continues where the last one stopped.
# Every state group is rewritten in its own transaction, after the compressed
# state is checked against the original; a chunk that would not shrink is skipped.
ExecStart=/usr/local/bin/synapse_auto_compressor -p postgresql:///synapse?host=/var/run/postgresql&user=postgres -c 500 -n 100
Environment=RUST_LOG=info
Nice=10
IOSchedulingClass=idle
TimeoutStartSec=2h
+10
View File
@@ -0,0 +1,10 @@
[Unit]
Description=Nightly Synapse state compression (after the 03:00 vzdump backup)
[Timer]
OnCalendar=*-*-* 04:30
RandomizedDelaySec=10min
Persistent=true
[Install]
WantedBy=timers.target