Executions can stay 'running' forever – no server-side timeout or reaper for quick/scheduled/internal commands #7

New Issue

2026-09-08T20:57:20-04:00

jared commented

2026-09-08 20:57:20 -04:00

Overview

For quick commands, scheduled commands and Gandalf internal commands the server relies entirely on the worker sending a command_result. Several ordinary conditions mean that never happens:

Worker at MAX_CONCURRENT_TASKS: executeCommand() logs "rejecting command" and returns without sending any result.
Worker WebSocket closes while a command runs: sendResult() silently drops the message (no queue/retry).
Worker process restarts mid-command.

In all cases the execution shows running indefinitely with a ticking elapsed counter, until someone restarts the server (the "Commands Stuck in Running – restart the server" note in the README is this bug).

Where

worker/worker.js executeCommand() (capacity reject), sendResult().
server.js – waitForCommandResult() only guards workflow steps; direct commands have no server-side timer.

Suggested fix

Worker: send a command_result with success:false, error:'Worker at capacity' instead of dropping; buffer results while disconnected and flush on reconnect.
Server: record timeout on the execution and run a periodic reaper that fails executions older than their timeout (+grace); also fail in-flight executions for a worker when its socket closes.

## Overview For quick commands, scheduled commands and Gandalf internal commands the server relies entirely on the worker sending a `command_result`. Several ordinary conditions mean that never happens: 1. Worker at `MAX_CONCURRENT_TASKS`: `executeCommand()` logs "rejecting command" and **returns without sending any result**. 2. Worker WebSocket closes while a command runs: `sendResult()` silently drops the message (no queue/retry). 3. Worker process restarts mid-command. In all cases the execution shows `running` indefinitely with a ticking elapsed counter, until someone restarts the server (the "Commands Stuck in Running – restart the server" note in the README is this bug). ## Where - `worker/worker.js` `executeCommand()` (capacity reject), `sendResult()`. - `server.js` – `waitForCommandResult()` only guards workflow steps; direct commands have no server-side timer. ## Suggested fix - Worker: send a `command_result` with `success:false, error:'Worker at capacity'` instead of dropping; buffer results while disconnected and flush on reconnect. - Server: record `timeout` on the execution and run a periodic reaper that fails executions older than their timeout (+grace); also fail in-flight executions for a worker when its socket closes.

jared added the bug backend worker executions priority/high labels 2026-09-08 20:57:20 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: LotusGuild/pulse#7