fix(livekit): graceful restart could restart LiveKit in the middle of a call #54

Merged

jared merged 1 commits from fix/livekit-graceful-restart-detection into main 2026-11-03 00:28:12 -05:00

Conversation 0 Commits 1 Files Changed 2

+26 -6

jared commented 2026-11-03 00:26:09 -05:00

Owner

Copy Link

Copy Source

livekit-graceful-restart.sh is supposed to restart LiveKit only when no call is active. Its check could never see a call.

Bugs

1. ss -tn state established '(…)' | grep -c ESTAB is always 0. With a state filter, ss omits the State column, so there's no "ESTAB" to grep.
2. It counted connections on :7881, which is only LiveKit's ICE-TCP media fallback. Signalling is on :7880, and media is normally UDP.
3. The deploy hook never installed this script when it changed, so fixes to it never reached LXC 151.

It hasn't caused an outage only because nothing has staged a LiveKit restart in the retained logs. The upcoming LiveKit 1.9 → 1.13 upgrade would be its first real use.

Fix

• Wait while LiveKit's own livekit_participant_total (from :6789/metrics) is above 0, or any signalling WebSocket on :7880 is open (ss -H … | wc -l).
• If the metrics can't be read, assume a call is active and wait.
• deploy/lxc151-hookshot.sh installs the script to /usr/local/bin when it changes, after a bash -n syntax check.

Tested

Local stack, with systemctl stubbed out:

Case	Old check	New check
two-person call	0 connections → restart	2 participants, 3 signalling → wait
metrics unreachable	—	wait
idle	restart	restart

🤖 Generated with Claude Code

https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5

`livekit-graceful-restart.sh` is supposed to restart LiveKit only when no call is active. Its check could never see a call. ## Bugs 1. `ss -tn state established '(…)' | grep -c ESTAB` is **always 0**. With a `state` filter, `ss` omits the State column, so there's no "ESTAB" to grep. 2. It counted connections on **:7881**, which is only LiveKit's ICE-TCP media fallback. Signalling is on **:7880**, and media is normally UDP. 3. The deploy hook never installed this script when it changed, so fixes to it never reached LXC 151. It hasn't caused an outage only because nothing has staged a LiveKit restart in the retained logs. The upcoming LiveKit 1.9 → 1.13 upgrade would be its first real use. ## Fix - Wait while LiveKit's own `livekit_participant_total` (from `:6789/metrics`) is above 0, **or** any signalling WebSocket on :7880 is open (`ss -H … | wc -l`). - If the metrics can't be read, assume a call is active and wait. - `deploy/lxc151-hookshot.sh` installs the script to `/usr/local/bin` when it changes, after a `bash -n` syntax check. ## Tested Local stack, with `systemctl` stubbed out: | Case | Old check | New check | |---|---|---| | two-person call | 0 connections → **restart** | 2 participants, 3 signalling → wait | | metrics unreachable | — | wait | | idle | restart | restart | 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5

jared added 1 commit 2026-11-03 00:26:10 -05:00

fix(livekit): graceful restart could restart LiveKit in the middle of a call ... d9f023762e

livekit-graceful-restart.sh decided "no active calls" by counting
established TCP connections on :7881 with `ss ... state established |
grep -c ESTAB`. Two bugs:

- With a `state` filter ss omits the State column, so `grep -c ESTAB`
  is always 0: the check never saw anything.
- :7881 is only LiveKit's ICE-TCP media fallback. Signalling is on
  :7880 and media is normally UDP.

Reproduced in a two-person call on the local stack: the old check
counted 0 and would have restarted. Now it waits while LiveKit's own
livekit_participant_total is above 0 or any signalling WebSocket on
:7880 is open, and also waits if the metrics can't be read.

The deploy hook now installs the script when it changes (it never
did, so edits to it never reached the server).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PCb59uY3UKqDqt7B6hzd5

jared scheduled this pull request to auto merge when all checks succeed 2026-11-03 00:26:21 -05:00

jared merged commit 8a0e16cabd into main 2026-11-03 00:28:12 -05:00

jared deleted branch fix/livekit-graceful-restart-detection 2026-11-03 00:28:13 -05:00

jared referenced this issue from a commit 2026-11-03 00:34:05 -05:00

docs: stop publishing Hookshot webhook IDs; record the Hookshot and call-stack upgrades

jared referenced this pull request 2026-11-03 00:34:13 -05:00

docs: stop publishing Hookshot webhook IDs; record Hookshot + call-stack upgrades #55

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

bug

Something is broken

ci

CI / workflows

enhancement

New feature or request

infra

Infrastructure / runners / servers

landing

priority: low

priority: medium

security

Security or privacy

ux

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

cole (Cole Keller) jared (Jared Vititoe) nathan (Nathan Vititoe)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: LotusGuild/matrix#54