Worker WebSocket connection goes zombie after server restart, never reconnects #1

New Issue

2026-09-05T11:39:53-04:00

jared commented

2026-09-05 11:39:53 -04:00

Summary

When the Pulse server (pulse.service) restarts, connected workers' WebSocket connections can go silently dead without either side noticing, permanently breaking command dispatch to that worker until the worker process is manually restarted.

What happened

pulse-worker-01 (LXC 153) connected via WS on 2026-08-30. The Pulse server (LXC 122) restarted on 2026-09-01 (pulse.service ActiveEnterTimestamp 2026-09-01 13:01:15). The worker's TCP connection to the old server process was dropped, but:

The worker's ws object never received a close or error event, so connectWebSocket()'s reconnect logic (ws.on('close', ...) → setTimeout(() => this.connectWebSocket(), 5000)) never fired.
The worker kept sending HTTP heartbeats successfully (POST /api/workers/heartbeat), so workers.status stayed online in the DB and the dashboard showed it as healthy the whole time.
Any caller hitting POST /api/internal/command for that worker got 400 {"error":"Worker not connected"}, silently and indefinitely, because the server's in-memory workers Map (populated only on WS worker_connect) was reset by the restart and never repopulated for this worker.

This went undetected for 5 days (2026-09-01 to 2026-09-05) and caused a downstream consumer (Gandalf) to treat a monitored host as permanently unreachable, since its reachability check silently depended on this worker connection. Manually restarting pulse-worker.service fixed it immediately (fresh WS connection, worker reconnects fine).

Root cause

Neither side proactively detects a half-open connection:

No native WebSocket ping/pong (the standard ws library pattern using ws.ping() + a per-connection isAlive flag + a periodic terminate() sweep for peers that missed a pong — see https://github.com/websockets/ws#how-to-detect-and-close-broken-connections).
The app already defines a ping/pong message-type protocol (worker replies to ping with sendPong(); server updates last_heartbeat on receiving pong), but grepping the codebase, neither side ever sends a ping message — the handlers exist but are dead code. So this mechanism can never actually detect a stale connection.
The HTTP heartbeat (/api/workers/heartbeat) is independent of the WS connection entirely, so it keeps reporting online even when the WS side is fully dead — there's no cross-check between the two, which is exactly what made this invisible on the dashboard.

Suggested fix

Server: on the workers' WebSocket connections, implement the standard ws heartbeat pattern — setInterval that ws.ping()s each open worker connection (e.g. every 30s) and ws.terminate()s any connection whose isAlive flag wasn't reset by a pong since the last sweep. This uses native WS control frames, not the app-level JSON ping/pong messages.
Worker: add a corresponding ws.on('pong', ...) (or reuse the existing JSON ping/pong path, but actually invoke it on an interval) to detect a silent connection and force-close + reconnect if the server stops responding.
Consider reconciling /api/workers status with actual live WS connection state (e.g. don't report online purely from HTTP heartbeat recency if the worker also expects a live command channel) so the dashboard can't mask this again.

Environment

pulse-server (LXC 122, monitor-02) — pulse.service
pulse-worker-01 (LXC 153, monitor-02) — pulse-worker.service
Found while debugging a Gandalf false-offline report: LotusGuild/gandalf#1

## Summary When the Pulse server (`pulse.service`) restarts, connected workers' WebSocket connections can go silently dead without either side noticing, permanently breaking command dispatch to that worker until the worker process is manually restarted. ## What happened `pulse-worker-01` (LXC 153) connected via WS on 2026-08-30. The Pulse server (LXC 122) restarted on 2026-09-01 (`pulse.service` `ActiveEnterTimestamp` 2026-09-01 13:01:15). The worker's TCP connection to the old server process was dropped, but: - The worker's `ws` object never received a `close` or `error` event, so [`connectWebSocket()`'s reconnect logic](worker/worker.js) (`ws.on('close', ...)` → `setTimeout(() => this.connectWebSocket(), 5000)`) never fired. - The worker kept sending HTTP heartbeats successfully (`POST /api/workers/heartbeat`), so `workers.status` stayed `online` in the DB and the dashboard showed it as healthy the whole time. - Any caller hitting `POST /api/internal/command` for that worker got `400 {"error":"Worker not connected"}`, silently and indefinitely, because the server's in-memory `workers` Map (populated only on WS `worker_connect`) was reset by the restart and never repopulated for this worker. This went undetected for 5 days (2026-09-01 to 2026-09-05) and caused a downstream consumer (Gandalf) to treat a monitored host as permanently unreachable, since its reachability check silently depended on this worker connection. Manually restarting `pulse-worker.service` fixed it immediately (fresh WS connection, worker reconnects fine). ## Root cause Neither side proactively detects a half-open connection: - No native WebSocket ping/pong (the standard `ws` library pattern using `ws.ping()` + a per-connection `isAlive` flag + a periodic `terminate()` sweep for peers that missed a pong — see https://github.com/websockets/ws#how-to-detect-and-close-broken-connections). - The app already defines a `ping`/`pong` message-type protocol (worker replies to `ping` with `sendPong()`; server updates `last_heartbeat` on receiving `pong`), but grepping the codebase, **neither side ever sends a `ping` message** — the handlers exist but are dead code. So this mechanism can never actually detect a stale connection. - The HTTP heartbeat (`/api/workers/heartbeat`) is independent of the WS connection entirely, so it keeps reporting `online` even when the WS side is fully dead — there's no cross-check between the two, which is exactly what made this invisible on the dashboard. ## Suggested fix 1. Server: on the workers' WebSocket connections, implement the standard `ws` heartbeat pattern — `setInterval` that `ws.ping()`s each open worker connection (e.g. every 30s) and `ws.terminate()`s any connection whose `isAlive` flag wasn't reset by a `pong` since the last sweep. This uses native WS control frames, not the app-level JSON `ping`/`pong` messages. 2. Worker: add a corresponding `ws.on('pong', ...)` (or reuse the existing JSON `ping`/`pong` path, but actually invoke it on an interval) to detect a silent connection and force-close + reconnect if the server stops responding. 3. Consider reconciling `/api/workers` status with actual live WS connection state (e.g. don't report `online` purely from HTTP heartbeat recency if the worker also expects a live command channel) so the dashboard can't mask this again. ## Environment - `pulse-server` (LXC 122, monitor-02) — `pulse.service` - `pulse-worker-01` (LXC 153, monitor-02) — `pulse-worker.service` - Found while debugging a Gandalf false-offline report: https://code.lotusguild.org/LotusGuild/gandalf/issues/1

jared referenced this issue from LotusGuild/gandalf

2026-09-05 11:40:06 -04:00

Tickets cutting instantly when devices go offline even during short update periods at scheduled 3am #1

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: LotusGuild/pulse#1