diff --git a/README.md b/README.md index 9812767..a4d458d 100644 --- a/README.md +++ b/README.md @@ -50,7 +50,7 @@ matrix/ | Service | IP | LXC | RAM | Disk | Versions | |---------|----|-----|-----|------|----------| -| Synapse | 10.10.10.29 | 151 | 8GB | 50GB | Synapse 1.155.0, LiveKit 1.9.11, hookshot 7.3.2, coturn latest | +| Synapse | 10.10.10.29 | 151 | 8GB | 50GB | Synapse 1.157.1, LiveKit 1.9.11, hookshot 7.3.2, coturn latest | | PostgreSQL 17 | 10.10.10.44 | 109 | 6GB | 30GB | PostgreSQL 17.9 | | Cinny Web | 10.10.10.6 | 106 | 2GB | 8GB | Debian 12, nginx, Node 24, Lotus Cinny fork (custom, tracks `cinnyapp/cinny` main) | | Draupnir | 10.10.10.24 | 110 | 1GB | 10GB | Draupnir v2.9.0, Node.js v22 | @@ -573,6 +573,7 @@ Periodic `TLS/TCP socket error: Connection reset by peer` in coturn logs. Normal ### Quality of Life - [x] **Upgrade Synapse to v1.155.0** — Done 2026-06-18. LXC 151 was already on Debian 13 Trixie; no OS migration needed. +- [x] **Upgrade Synapse to v1.157.1** — Done 2026-07-23. Note: the host was found running **1.156.0**, not the 1.155.0 recorded here — an intermediate upgrade had happened without being logged, so always verify with `dpkg-query -W matrix-synapse-py3` rather than trusting this file. Not a security release (no CVEs between 1.155.0 and 1.157.1; the last advisories were fixed back in 1.152.1), but 1.157.0 fixes an application-service ephemeral-events regression that 1.156.0 shipped — which affects hookshot. No schema-version bump (94 → 94); the three new background updates finished immediately. `experimental_features` (MatrixRTC/Element Call MSCs) was preserved and is unaffected by the v1.157.0 removal of `msc3861`, which we do not use. - [x] Migrate from SQLite to PostgreSQL - [x] TURN/STUN server (coturn) for reliable voice/video @@ -656,6 +657,63 @@ Periodic `TLS/TCP socket error: Connection reset by peer` in coturn logs. Normal --- +## Synapse Upgrade Runbook + +Synapse is installed from the **matrix.org apt repo** (`matrix-synapse-py3`) on **LXC 151** — it is +not containerized, and none of the auto-deploy scripts in `deploy/` touch it. Upgrades are manual. +Access is via the Proxmox host: `ssh compute-storage-01`, then `pct exec 151 -- `. + +**1. Verify the real current version** — this file has been wrong before: +```bash +pct exec 151 -- dpkg-query -W -f='${Version}\n' matrix-synapse-py3 +pct exec 151 -- bash -c 'apt-get update -qq && apt-cache policy matrix-synapse-py3' +``` + +**2. Read upstream upgrade notes** for every version in the gap: +. Look for removed config keys, schema +bumps, and minimum Python/PostgreSQL bumps. Then check ours against them: +```bash +pct exec 151 -- grep -nE 'experimental_features|msc[0-9]+|modules:' /etc/matrix-synapse/homeserver.yaml +pct exec 151 -- python3 -V # needs >= 3.10 +pct exec 109 -- su postgres -c 'psql -tAc "select version()"' # needs >= 14 +``` + +**3. Confirm a backup exists.** Nightly `vzdump` snapshots *all* guests at 03:00 to `mediafs` +(`keep-last=2`), plus a weekly run to `pbs`. Both LXC 151 (Synapse) and 109 (PostgreSQL) are +covered — a rollback needs **both**, since they'd have to be restored to a consistent point: +```bash +pvesm list mediafs | grep -E '\-(151|109)\-' +``` +Take an on-demand backup instead if the nightly one is stale: `vzdump 151 109 --storage mediafs --mode snapshot`. +Note that Synapse does **not** support downgrading once new code has run against the database — +treat every upgrade as one-way in practice, regardless of whether the schema version changed. + +**4. Upgrade.** `--force-confold` is essential: `homeserver.yaml` is a dpkg conffile and we have +modified it heavily, so without this apt may prompt or overwrite it. +```bash +pct exec 151 -- bash -c 'DEBIAN_FRONTEND=noninteractive apt-get install -y \ + -o Dpkg::Options::=--force-confold -o Dpkg::Options::=--force-confdef \ + matrix-synapse-py3=+trixie1' +``` +The package restarts `matrix-synapse` itself. + +**5. Verify:** +```bash +pct exec 151 -- systemctl is-active matrix-synapse +pct exec 151 -- ls /etc/matrix-synapse/*.dpkg-* # must be empty — config was preserved +pct exec 151 -- curl -s http://localhost:8008/_matrix/federation/v1/version +pct exec 151 -- journalctl -u matrix-synapse --since '5 min ago' -p err --no-pager +``` +Then confirm background updates drain (empty result = done) before any *further* upgrade: +```bash +pct exec 109 -- su postgres -c "psql -d synapse -tAc 'select update_name from background_updates'" +``` + +**6. Record it** — update the version in the infra table, the Tech Stack table, and the Server +Checklist in this file. + +--- + ## Monitoring & Observability ### Prometheus Scrape Jobs @@ -786,7 +844,7 @@ All commands use the `!` prefix. Run `!help` in any room for the full list. | Component | Technology | Version | |-----------|-----------|---------| -| Homeserver | Synapse | 1.155.0 | +| Homeserver | Synapse | 1.157.1 | | Database | PostgreSQL | 17.9 | | TURN | coturn | latest | | Video/voice calls | LiveKit SFU | 1.9.11 |