CI: e2e job that starts its own Synapse and runs the UI regression suite (no prod secrets) #220

New Issue

2026-09-18T21:41:07-04:00

jared commented

2026-09-18 21:41:07 -04:00

Today's manual-QA verification scripts (thread panel crash, upload retry, forward partial failure, Invite QR, privacy toggles, read receipts, polls, reminders, notifications, retention, search cache, permalinks, mobile layout…) live as scratch Playwright scripts. Turn them into e2e/*.spec.ts and run them on every push against a homeserver the job starts itself:

pip install "matrix-synapse[url-preview]" on the ubuntu runner (no Docker needed — same recipe as scripts/dev-homeserver.sh: open registration, no rate limits, media, URL previews, MSC4140 delayed events, MSC4143 transports), seed with scripts/dev-seed.py.
The E2EE tier that is currently skipped for lack of E2E_* secrets can run against this local Synapse too — promote e2e from continue-on-error to a hard gate once green.
Fixtures worth keeping from today: login helper, addInitScript stubs for Notification/HTMLMediaElement.play/SW ready, request counters (/typing/, /receipt/, preview_url, media), a crypto-capable second client (.scratch-pw/bob-enc.mjs pattern: matrix-js-sdk + in-memory rust crypto) for encrypted-room cases.
Add a WebKit project (see the Safari issue) and an axe pass (see the a11y issue) to the same job.

Budget: roughly one day. Payoff: every bug found on 2026-09-18 gets a regression test.

Today's manual-QA verification scripts (thread panel crash, upload retry, forward partial failure, Invite QR, privacy toggles, read receipts, polls, reminders, notifications, retention, search cache, permalinks, mobile layout…) live as scratch Playwright scripts. Turn them into `e2e/*.spec.ts` and run them on every push against a homeserver the job starts itself: - `pip install "matrix-synapse[url-preview]"` on the ubuntu runner (no Docker needed — same recipe as `scripts/dev-homeserver.sh`: open registration, no rate limits, media, URL previews, MSC4140 delayed events, MSC4143 transports), seed with `scripts/dev-seed.py`. - The E2EE tier that is currently `skipped` for lack of `E2E_*` secrets can run against this local Synapse too — promote `e2e` from `continue-on-error` to a hard gate once green. - Fixtures worth keeping from today: login helper, `addInitScript` stubs for `Notification`/`HTMLMediaElement.play`/SW `ready`, request counters (`/typing/`, `/receipt/`, `preview_url`, media), a crypto-capable second client (`.scratch-pw/bob-enc.mjs` pattern: matrix-js-sdk + in-memory rust crypto) for encrypted-room cases. - Add a WebKit project (see the Safari issue) and an axe pass (see the a11y issue) to the same job. Budget: roughly one day. Payoff: every bug found on 2026-09-18 gets a regression test.

jared added this to the Features 2026-Q4 milestone 2026-09-18 21:41:07 -04:00

jared added the enhancement priority: medium area: build-ci labels 2026-09-18 21:41:07 -04:00

jared referenced this issue from a commit

2026-09-19 22:38:43 -04:00

ci(e2e): tier-3 regression suite against a Synapse the job starts itself (#220)

jared commented

2026-09-19 22:46:15 -04:00

Done in ef5d06ee and green on the runner (run #1988, e2e job 4625: synapse up on http://localhost:8008 → 13 passed, 3 skipped [the E2EE tier without secrets], 1.4 min).

The e2e job now runs scripts/dev-homeserver.sh start + dev-seed.py before Playwright (installs python3-venv if the runner lacks it) and stops it afterwards; the homeserver is SQLite, open registration, no rate limits, no prod secrets anywhere.
e2e/local-homeserver.spec.ts (+ e2e/localHs.ts helpers) registers its own users/rooms over the CS API and drives the built client: login + send/receive, own-message scroll (#212), /kick toast (#216), upload 413 wording (#213), forward provenance, thread panel + drawer at 1400 px (#218), timeline lightbox (#219), clock-skew banner via page.clock (#158), status save under the presence limit (#226), long-press action sheet on a Pixel 7 emulation (#166). It skips itself when no homeserver answers, so npm run test:e2e still works cold.
Documented in LOTUS_TESTING.md as the third tier. Rule of thumb going forward: whenever a fix is reproduced with a scratch Playwright script, fold it into this spec.

Not in this tier: calls (LiveKit + JWT + guard + an https well-known on :443). The dev script already brings that stack up locally; making it CI-safe (no privileged port) is a follow-up if we want the call regressions gated too.

Done in `ef5d06ee` and **green on the runner** (run #1988, e2e job 4625: `synapse up on http://localhost:8008` → 13 passed, 3 skipped [the E2EE tier without secrets], 1.4 min). - The `e2e` job now runs `scripts/dev-homeserver.sh start` + `dev-seed.py` before Playwright (installs `python3-venv` if the runner lacks it) and stops it afterwards; the homeserver is SQLite, open registration, no rate limits, no prod secrets anywhere. - `e2e/local-homeserver.spec.ts` (+ `e2e/localHs.ts` helpers) registers its own users/rooms over the CS API and drives the built client: login + send/receive, own-message scroll (#212), `/kick` toast (#216), upload 413 wording (#213), forward provenance, thread panel + drawer at 1400 px (#218), timeline lightbox (#219), clock-skew banner via `page.clock` (#158), status save under the presence limit (#226), long-press action sheet on a Pixel 7 emulation (#166). It skips itself when no homeserver answers, so `npm run test:e2e` still works cold. - Documented in `LOTUS_TESTING.md` as the third tier. Rule of thumb going forward: whenever a fix is reproduced with a scratch Playwright script, fold it into this spec. Not in this tier: calls (LiveKit + JWT + guard + an https well-known on :443). The dev script already brings that stack up locally; making it CI-safe (no privileged port) is a follow-up if we want the call regressions gated too.

jared closed this issue

2026-09-19 22:46:15 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: LotusGuild/cinny#220