feat(synapse): nightly state compression on CT 109 #59

Merged

jared merged 1 commits from feat/synapse-state-compressor into main 2026-11-03 19:10:34 -05:00

Conversation 0 Commits 1 Files Changed 3

+64

jared commented 2026-11-03 19:08:14 -05:00

Owner

Copy Link

Copy Source

The Synapse database grew from 2.2 GB (after the July cleanup) to 6.2 GB. Almost all of it is state_groups_state from large federated rooms with one or two local members: Jellyfin General Chat (16,754 members, 5.3 M rows), an unnamed 1,213-member room, Homelabbing, Element X Android and the Cinny community space.

What

synapse_auto_compressor, from matrix-org/rust-synapse-compress-state (built from main), now runs nightly at 04:30, after the 03:00 vzdump backup. This adds:

• systemd/synapse-state-compressor.{service,timer}, already installed and enabled on CT 109 (it has no auto-deploy hook);
• a README section covering how the compressor works, how to install it, the results, and the disk-space note;
• documentation for the new Grafana alert Synapse Database Growing Fast (> 500 MB in 7 days), which is already created.

Safety (checked in the source before running)

• The compressed state is checked against the original before anything is written (check_that_maps_match).
• Each state group is written in its own transaction.
• Any chunk that would not shrink is skipped.
• Progress lives in state_compressor_* tables, so each night resumes where the last one stopped.

First run

• Backlog: ~8 minutes, done in runs of 100 chunks × 500 groups at nice/idle I/O priority.
• Result: state_groups_state went from 11.0 M to 4.25 M rows (−61%).
• Impact: 0 Synapse errors, and DB query p99 stayed at ~10 ms.
• Disk: the file stays ~2.77 GB, and the freed space is reused internally. VACUUM FULL would return it but needs a lock, and it isn't needed at 14% disk use.

🤖 Generated with Claude Code

https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5

The Synapse database grew from 2.2 GB (after the July cleanup) to **6.2 GB**. Almost all of it is `state_groups_state` from large federated rooms with one or two local members: *Jellyfin General Chat* (16,754 members, 5.3 M rows), an unnamed 1,213-member room, *Homelabbing*, *Element X Android* and the *Cinny community* space. ## What `synapse_auto_compressor`, from matrix-org/rust-synapse-compress-state (built from `main`), now runs nightly at **04:30**, after the 03:00 vzdump backup. This adds: - `systemd/synapse-state-compressor.{service,timer}`, already installed and enabled on CT 109 (it has no auto-deploy hook); - a README section covering how the compressor works, how to install it, the results, and the disk-space note; - documentation for the new Grafana alert **Synapse Database Growing Fast** (> 500 MB in 7 days), which is already created. ## Safety (checked in the source before running) - The compressed state is checked against the original before anything is written (`check_that_maps_match`). - Each state group is written in its own transaction. - Any chunk that would not shrink is skipped. - Progress lives in `state_compressor_*` tables, so each night resumes where the last one stopped. ## First run - **Backlog:** ~8 minutes, done in runs of 100 chunks × 500 groups at nice/idle I/O priority. - **Result:** `state_groups_state` went from **11.0 M to 4.25 M rows (−61%)**. - **Impact:** 0 Synapse errors, and DB query p99 stayed at ~10 ms. - **Disk:** the file stays ~2.77 GB, and the freed space is reused internally. `VACUUM FULL` would return it but needs a lock, and it isn't needed at 14% disk use. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5

jared added 1 commit 2026-11-03 19:08:15 -05:00

feat(synapse): nightly state compression on CT 109 ... b2877323c3

The Synapse database grew from 2.2 GB (after the July cleanup) to
6.2 GB, almost all of it state_groups_state from large federated rooms
with one or two local members.

synapse_auto_compressor (matrix-org/rust-synapse-compress-state, built
from main) now runs nightly at 04:30, after the 03:00 vzdump backup.
It verifies compressed state against the original, writes each state
group in its own transaction, skips chunks that would not shrink, and
resumes from stored progress. The initial backlog took ~8 minutes with
no Synapse errors: state_groups_state 11.0 M -> 4.25 M rows (-61%).

Also documents the new "Synapse Database Growing Fast" Grafana alert
(> 500 MB in 7 days).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5

jared scheduled this pull request to auto merge when all checks succeed 2026-11-03 19:08:38 -05:00

jared merged commit ed1300f146 into main 2026-11-03 19:10:34 -05:00

jared deleted branch feat/synapse-state-compressor 2026-11-03 19:10:34 -05:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

bug

Something is broken

ci

CI / workflows

enhancement

New feature or request

infra

Infrastructure / runners / servers

landing

priority: low

priority: medium

security

Security or privacy

ux

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

cole (Cole Keller) jared (Jared Vititoe) nathan (Nathan Vititoe)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: LotusGuild/matrix#59