# PULSE - Pipelined Unified Logic & Server Engine

[![Lint](https://code.lotusguild.org/LotusGuild/pulse/actions/workflows/lint.yml/badge.svg)](https://code.lotusguild.org/LotusGuild/pulse/actions?workflow=lint.yml)
[![Test](https://code.lotusguild.org/LotusGuild/pulse/actions/workflows/test.yml/badge.svg)](https://code.lotusguild.org/LotusGuild/pulse/actions?workflow=test.yml)
[![Security](https://code.lotusguild.org/LotusGuild/pulse/actions/workflows/security.yml/badge.svg)](https://code.lotusguild.org/LotusGuild/pulse/actions?workflow=security.yml)

A distributed workflow orchestration platform for managing and executing complex multi-step operations across server clusters through a retro terminal-themed web interface.

> **Security Notice:** This repository is hosted on Gitea and is version-controlled. **Never commit secrets, credentials, passwords, API keys, or any sensitive information to this repo.** All sensitive configuration belongs exclusively in `.env` files which are listed in `.gitignore` and must never be committed. This includes database passwords, worker API keys, webhook secrets, and internal IP details.

**Design System**: [web_template](https://code.lotusguild.org/LotusGuild/web_template) — shared CSS, JS, and layout patterns for all LotusGuild apps

## Styling & Layout

PULSE uses the **LotusGuild Terminal Design System**. The design system is **vendored** into this
repo at `public/web_template/` (`base.css`, `base.js`, `VERSION`) and served from `/web_template/*`,
so the app has no runtime dependency on a sibling checkout. `public/web_template/VERSION` records
the design-system version, upstream short SHA, and sync date; asset URLs are cache-busted with it.

Update the vendored copy with:

```bash
scripts/sync-web-template.sh <path-to-web_template-checkout>
```

The script copies `base.css`/`base.js` as regular files (never symlinks) and rewrites `VERSION`.
Never hand-edit `public/web_template/` — it is excluded from ESLint and overwritten on every sync.
Pulse-local gaps in the design system live in `public/assets/app.css` (currently `.lt-modal-lg`,
`.lt-field-error`, `.is-invalid`) and are candidates for upstreaming.

Reference documentation:

- [`web_template/README.md`](https://code.lotusguild.org/LotusGuild/web_template/src/branch/main/README.md) — full component reference, CSS variables, JS API
- [`web_template/base.css`](https://code.lotusguild.org/LotusGuild/web_template/src/branch/main/base.css) — unified CSS (`.lt-*` classes)
- [`web_template/base.js`](https://code.lotusguild.org/LotusGuild/web_template/src/branch/main/base.js) — `window.lt` utilities (toast, modal, WebSocket helpers, fetch)
- [`web_template/aesthetic_diff.md`](https://code.lotusguild.org/LotusGuild/web_template/src/branch/main/aesthetic_diff.md) — cross-app divergence analysis and convergence guide
- [`web_template/node/middleware.js`](https://code.lotusguild.org/LotusGuild/web_template/src/branch/main/node/middleware.js) — Express auth, CSRF, CSP nonce middleware

![PULSE walkthrough: dashboard, workers, workflows, execution logs with an approval prompt, a multi-worker quick command and the command palette](docs/img/demo.gif)

## Why this exists

I run a six-node Proxmox/Ceph cluster plus a pile of LXC containers. Day-to-day work is the same handful of questions asked of many machines: *what is the disk usage everywhere, restart that service on one node, is it safe to proceed?* SSH-ing into each box does not scale, and a full config-management system is more than I need. PULSE is a small orchestration layer: a server with a web UI, and a lightweight **worker** on each machine that executes what the server dispatches over a WebSocket. It adds what shell loops lack: multi-step **workflows** with conditions and human approval gates, scheduled commands, a full execution history, and an API that other tools call (GANDALF uses it to run network diagnostics).

```mermaid
flowchart LR
  user["Browser<br/>(Authelia SSO)"] -->|HTTPS| srv["PULSE server<br/>Express + EJS + WebSocket"]
  gd["GANDALF"] -->|internal API| srv
  srv --> db[("MariaDB<br/>workflows, executions,<br/>schedules")]
  srv <-->|"WebSocket<br/>commands / results / heartbeat"| w1["worker: node-01"]
  srv <-->|WebSocket| w2["worker: node-02"]
  srv <-->|WebSocket| w3["worker: node-03"]
```

Workflows are plain JSON with five step types: `execute`, `prompt` (pause for a human decision), `wait`, `parse` (pull `KEY=VALUE` lines from output into state) and `route` (branch on parsed state).

## Gallery

**Dashboard**: recent executions at a glance and live worker status.

![Dashboard](docs/img/dashboard.png)

**Workers** report system info, memory, load, uptime and task slots on every heartbeat:

![Workers](docs/img/workers.png)

**Workflows** are reusable multi-step jobs. The editor takes the JSON definition directly:

| Workflow list | Workflow editor |
|---|---|
| ![Workflows](docs/img/workflows.png) | ![Workflow editor](docs/img/workflow-editor.png) |

**Executions**: full history with step-by-step logs, per-command output and durations.

![Executions](docs/img/executions.png)

| Completed run with step logs | A failed command, with its error |
|---|---|
| ![Execution detail](docs/img/execution-detail.png) | ![Failed execution](docs/img/execution-failed.png) |

**Approval gates:** a `prompt` step pauses the workflow until someone chooses an option (here, a rolling restart waits for a human to approve):

![Awaiting approval](docs/img/execution-awaiting-approval.png)

**Quick Command** runs a one-off command on one or many workers at once:

| Compose | Result |
|---|---|
| ![Quick command](docs/img/quick-command.png) | ![Quick command result](docs/img/quick-command-result.png) |

**Scheduler** runs commands on an interval or hourly schedule:

![Scheduler](docs/img/scheduler.png)

**Command palette** (`Ctrl+K`) and **light theme**:

| Command palette | Light theme |
|---|---|
| ![Command palette](docs/img/command-palette.png) | ![Dashboard, light theme](docs/img/dashboard-light.png) |

<sub>Screenshots are generated with Playwright against a throwaway local instance with three local workers running harmless commands (`uptime`, `df`, `echo`). See `scripts/demo/`.</sub>

## Web UI

The UI is server-rendered with EJS. Every route renders `views/pages/<page>.ejs` into the shared
chrome in `views/layout.ejs` (nav, header, WebSocket status dot, theme toggle, command palette).

| Route | Page | View | Page module |
|---|---|---|---|
| `/` | Dashboard | `views/pages/dashboard.ejs` | `public/assets/pages/dashboard.js` |
| `/workers` | Workers | `views/pages/workers.ejs` | `public/assets/pages/workers.js` |
| `/workflows` | Workflows | `views/pages/workflows.ejs` | `public/assets/pages/workflows.js` |
| `/executions` | Executions | `views/pages/executions.ejs` | `public/assets/pages/executions.js` |
| `/quick` | Quick Command | `views/pages/quick.ejs` | `public/assets/pages/quick.js` |
| `/scheduler` | Scheduler | `views/pages/scheduler.ejs` | `public/assets/pages/scheduler.js` |

Scripts load in a fixed order: `/web_template/base.js` → `/assets/app.js` → `/assets/pages/<page>.js`.
`app.js` owns the shell (`window.Pulse`: action registry, event bus, `Pulse.confirm`, formatters,
WebSocket, the single 30 s auto-refresh) and each page module registers itself with:

```js
Pulse.registerPage({ name, init(), refresh(), onEvent(type, data) /* return true if handled */ });
```

Page modules never attach their own listeners for UI actions — they register handlers under their
own action prefix (`dash:`, `wk:`, `wf:`, `ex:`, `qc:`, `sc:`) and the markup wires them up with
`data-action` / `data-change-action` / `data-input-action` / `data-submit-action` attributes that
`app.js` delegates. DOM ids are likewise prefixed per page. All dynamic strings go through
`Pulse.esc`, and destructive actions use the themed `Pulse.confirm` (no native `confirm()`/`alert()`).

**Content Security Policy:** pages are served under a strict nonce-based CSP (helmet), including
`script-src-attr 'none'`. There are **no inline `<script>` blocks without a nonce and no inline
event handler attributes** anywhere in `views/` or `public/assets/`; anything added there must
follow the same rule or the browser will refuse to run it. Set `PULSE_CSP_REPORT_ONLY=1` to switch
the policy to report-only (violations are reported to `/csp-report` and logged) while debugging.

## Overview

PULSE is a centralized workflow execution system designed to orchestrate operations across distributed infrastructure. It provides a powerful web-based interface with a vintage CRT terminal aesthetic for defining, managing, and executing workflows that can span multiple servers, require human interaction, and perform complex automation tasks at scale.

### Key Features

- **🎨 Retro Terminal Interface**: Phosphor green CRT-style interface with scanlines, glow effects, and ASCII art
- **⚡ Quick Command Execution**: Instantly execute commands on any worker with built-in templates and command history
- **📊 Real-Time Worker Monitoring**: Live system metrics including CPU, memory, load average, and active tasks
- **🔄 Interactive Workflow Management**: Define and execute multi-step workflows with conditional logic and user prompts
- **🌐 Distributed Execution**: Run commands across multiple worker nodes simultaneously via WebSocket
- **📈 Execution Tracking**: Comprehensive logging with formatted output, re-run capabilities, and JSON export
- **🔐 SSO Authentication**: Seamless integration with Authelia for enterprise authentication
- **🧹 Auto-Cleanup**: Automatic removal of old executions with configurable retention policies
- **🔔 Terminal Notifications**: Audio beeps and visual toasts for command completion events

## Architecture

PULSE consists of two core components:

### PULSE Server
**Location:** `10.10.10.65` (LXC Container ID: 122)
**Directory:** `/opt/pulse-server`

The central orchestration hub that:
- Hosts the retro terminal web interface
- Manages workflow definitions and execution state
- Coordinates task distribution to worker nodes via WebSocket
- Handles user interactions through Authelia SSO
- Provides real-time status updates and logging
- Stores all data in MariaDB database

**Technology Stack:**
- Node.js 20.x
- Express.js (web framework)
- WebSocket (ws package) for real-time bidirectional communication
- MySQL2 (MariaDB driver)
- Authelia SSO integration

### PULSE Worker
**Example:** `10.10.10.151` (LXC Container ID: 153, hostname: pulse-worker-01)
**Directory:** `/opt/pulse-worker`

Lightweight execution agents that:
- Connect to PULSE server via WebSocket with heartbeat monitoring
- Execute shell commands and report results in real-time
- Provide system metrics (CPU, memory, load, uptime)
- Support concurrent task execution with configurable limits
- Automatically reconnect on connection loss

**Technology Stack:**
- Node.js 20.x
- WebSocket client
- Child process execution
- System metrics collection

```
┌─────────────────────────────────┐
│   PULSE Server (10.10.10.65)   │
│   Terminal Web Interface + API  │
│   ┌───────────┐ ┌──────────┐   │
│   │ MariaDB   │ │ Authelia │   │
│   │ Database  │ │   SSO    │   │
│   └───────────┘ └──────────┘   │
└────────────┬────────────────────┘
             │ WebSocket
    ┌────────┴────────┬───────────┐
    │                 │           │
┌───▼────────┐   ┌───▼────┐  ┌──▼─────┐
│ Worker 1   │   │Worker 2│  │Worker N│
│10.10.10.151│   │  ...   │  │  ...   │
└────────────┘   └────────┘  └────────┘
 LXC Containers in Proxmox with Ceph
```

## Installation

### Prerequisites

- **Node.js 20.x** or higher
- **MariaDB 10.x** or higher
- **Authelia** configured for SSO (optional but recommended)
- **Network Connectivity** between server and workers

### PULSE Server Setup

```bash
# Clone repository
cd /opt
git clone <your-repo-url> pulse-server
cd pulse-server

# Install dependencies
npm install

# Create .env file with configuration
cat > .env << EOF
# Server Configuration
PORT=8080
SECRET_KEY=your-secret-key-here

# MariaDB Configuration
DB_HOST=10.10.10.50
DB_PORT=3306
DB_NAME=pulse
DB_USER=pulse_user
DB_PASSWORD=your-db-password

# Worker API Key (for worker authentication)
WORKER_API_KEY=your-worker-api-key

# Auto-cleanup configuration (optional)
EXECUTION_RETENTION_DAYS=30
EOF

# Create systemd service
cat > /etc/systemd/system/pulse.service << EOF
[Unit]
Description=PULSE Workflow Orchestration Server
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/opt/pulse-server
ExecStart=/usr/bin/node server.js
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

# Start service
systemctl daemon-reload
systemctl enable pulse.service
systemctl start pulse.service
```

### PULSE Worker Setup

```bash
# On each worker node
cd /opt
git clone <your-repo-url> pulse-worker
cd pulse-worker

# Install dependencies
npm install

# Create .env file
cat > .env << EOF
# Worker Configuration
WORKER_NAME=pulse-worker-01
PULSE_SERVER=http://10.10.10.65:8080
PULSE_WS=ws://10.10.10.65:8080
WORKER_API_KEY=your-worker-api-key

# Performance Settings
HEARTBEAT_INTERVAL=30
MAX_CONCURRENT_TASKS=5
EOF

# Create systemd service
cat > /etc/systemd/system/pulse-worker.service << EOF
[Unit]
Description=PULSE Worker Node
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/opt/pulse-worker
ExecStart=/usr/bin/node worker.js
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

# Start service
systemctl daemon-reload
systemctl enable pulse-worker.service
systemctl start pulse-worker.service
```

## Usage

### Quick Command Execution

1. Access PULSE at `http://your-server:8080`
2. Navigate to **⚡ Quick Command** tab
3. Select a worker from the dropdown
4. Use **Templates** for pre-built commands or **History** for recent commands
5. Enter your command and click **Execute**
6. View results in the **Executions** tab

**Built-in Command Templates:**
- System Info: `uname -a`
- Disk Usage: `df -h`
- Memory Usage: `free -h`
- CPU Info: `lscpu`
- Running Processes: `ps aux --sort=-%mem | head -20`
- Network Interfaces: `ip addr show`
- Docker Containers: `docker ps -a`
- System Logs: `tail -n 50 /var/log/syslog`

### Worker Monitoring

The **Workers** tab displays real-time metrics for each worker:
- System information (OS, architecture, CPU cores)
- Memory usage (used/total with percentage)
- Load averages (1m, 5m, 15m)
- System uptime
- Active tasks vs. maximum concurrent capacity

### Execution Management

- **View Details**: Click any execution to see formatted logs with timestamps, status, and output
- **Re-run Command**: Click "Re-run" button in execution details to repeat a command
- **Download Logs**: Export execution data as JSON for auditing
- **Clear Completed**: Bulk delete finished executions
- **Auto-Cleanup**: Executions older than 30 days are automatically removed

### Workflow Creation (Future Feature)

1. Navigate to **Workflows** → **Create New**
2. Define workflow steps using JSON syntax
3. Specify target workers
4. Add interactive prompts where needed
5. Save and execute

## Features in Detail

### Terminal Aesthetic
- Phosphor green (#00ff41) on black (#0a0a0a) color scheme
- CRT scanline animation effect
- Text glow and shadow effects
- ASCII box-drawing characters for borders
- Boot sequence animation on first load
- Hover effects with smooth transitions

### Real-Time Communication
- WebSocket-based bidirectional communication
- Instant command result notifications
- Live worker status updates
- Terminal beep sounds for events
- Toast notifications with visual feedback

### Execution Tracking
- Formatted log display (not raw JSON)
- Color-coded success/failure indicators
- Timestamp and duration for each step
- Scrollable output with syntax highlighting
- Persistent history with pagination
- Load More button for large execution lists

### Security
- Authelia SSO integration for user authentication
- API key authentication for workers
- User session management
- Admin-only operations (worker deletion, workflow management)
- Audit logging for all executions

### Performance
- Automatic cleanup of old executions (configurable retention)
- Pagination for large execution lists (50 at a time)
- Efficient WebSocket connection pooling
- Worker heartbeat monitoring
- Database connection pooling

## Configuration

### Environment Variables

**Server (.env):**
```bash
PORT=8080                           # Server port
SECRET_KEY=<random-string>          # Session secret
DB_HOST=10.10.10.50                # MariaDB host
DB_PORT=3306                        # MariaDB port
DB_NAME=pulse                       # Database name
DB_USER=pulse_user                  # Database user
DB_PASSWORD=<password>              # Database password
WORKER_API_KEY=<api-key>           # Worker authentication key
EXECUTION_RETENTION_DAYS=30         # Auto-cleanup retention (default: 30)

# Web UI (all optional)
APP_NAME=PULSE                      # Header/boot/title app name (default: PULSE)
APP_SUBTITLE=<text>                 # Header subtitle (default: "Worker Orchestration // LotusGuild")
PULSE_CSP_REPORT_ONLY=1             # Serve the CSP report-only instead of enforcing it
PULSE_DEV_READONLY=1                # Local dev guard: disable all background writes (see below)
DISABLE_BACKGROUND_JOBS=1           # Alias for PULSE_DEV_READONLY
```

### Local development against a real database

Running a local instance against the production MariaDB is safe only with the read-only guard on:

```bash
PULSE_DEV_READONLY=1 PORT=8099 HOST=127.0.0.1 npm start
```

`PULSE_DEV_READONLY=1` (alias `DISABLE_BACKGROUND_JOBS=1`) disables every background writer, so a
dev instance can never mutate shared state behind your back: stale-execution recovery at startup,
the old-execution cleanup job (startup call and interval), the scheduled-command processor
(startup call and interval), and the stale-worker offline sweep. Each skip is logged at startup
alongside a `PULSE_DEV_READONLY IS ON` banner. The guard covers background jobs only — API routes
still write, so avoid destructive actions in the UI when you are pointed at production data.

**Worker (.env):**
```bash
WORKER_NAME=pulse-worker-01         # Unique worker name
PULSE_SERVER=http://10.10.10.65:8080   # Server HTTP URL
PULSE_WS=ws://10.10.10.65:8080          # Server WebSocket URL
WORKER_API_KEY=<api-key>            # Must match server key
HEARTBEAT_INTERVAL=30               # Heartbeat seconds (default: 30)
MAX_CONCURRENT_TASKS=5              # Max parallel tasks (default: 5)
```

## Database Schema

PULSE uses MariaDB with the following tables:

| Table | Purpose |
|-------|---------|
| `users` | User accounts synced from Authelia SSO |
| `workers` | Worker node registry with connection metadata |
| `workflows` | Workflow definitions stored as JSON |
| `executions` | Execution history with logs, status, and timestamps |

### `executions` Table Key Columns

| Column | Description |
|--------|-------------|
| `id` | Auto-increment primary key |
| `worker_id` | Foreign key to workers |
| `command` | The command that was executed |
| `status` | `running`, `completed`, `failed` |
| `output` | Command output / log (JSON or text) |
| `created_at` | Execution start timestamp |
| `completed_at` | Execution end timestamp |

### `workers` Table Key Columns

| Column | Description |
|--------|-------------|
| `id` | Auto-increment primary key |
| `name` | Worker name (from `WORKER_NAME` env) |
| `last_seen` | Last heartbeat timestamp |
| `status` | `online`, `offline` |
| `metadata` | JSON blob of system info |

## Troubleshooting

### Worker Not Connecting
```bash
# Check worker service status
systemctl status pulse-worker

# Check worker logs
journalctl -u pulse-worker -n 50 -f

# Verify API key matches server
grep WORKER_API_KEY /opt/pulse-worker/.env
```

### Commands Stuck in "Running"
- This was fixed in recent updates - restart the server:
```bash
systemctl restart pulse.service
```

### Clear All Executions
Use the database directly if needed:
```bash
mysql -h 10.10.10.50 -u pulse_user -p pulse
> DELETE FROM executions WHERE status IN ('completed', 'failed');
```

## Development

### Recent Updates

**Phase 1-6 Improvements:**
- Formatted log display with color-coding
- Worker system metrics monitoring
- Command templates and history
- Re-run and download execution features
- Auto-cleanup and pagination
- Terminal aesthetic refinements
- Audio notifications and visual toasts

See git history for detailed changelog.

### Future Enhancements
- Full workflow system implementation
- Multi-worker command execution
- Scheduled/cron job support
- Execution search and filtering
- Dark/light theme toggle
- Mobile-responsive design
- REST API documentation
- Webhook integrations

## License

MIT License - See LICENSE file for details

---

## CI / CD

| Workflow | Purpose | Triggers |
|---|---|---|
| `lint.yml` | ESLint on all `.js` files | Every push and PR |
| `test.yml` | Jest unit tests (`lib/utils.js`) | Every push and PR |
| `security.yml` | `npm audit --audit-level=high` | Every push, PR, and weekly Monday 6am |
| `deploy` job in `lint.yml` | Calls the `pulse-deploy` webhook on CT122 (10.10.10.65) to pull + restart | Push to `main` only, after lint passes |

Branch protection is enabled on `main` — the `lint.yml` check must pass before any PR can merge.

Tests live in `tests/utils.test.js` and cover the pure utility functions in `lib/utils.js`:
`validateWebhookUrl`, `applyParams`, `evalCondition`, `calculateNextRun`.

---

**PULSE** - Orchestrating your infrastructure, one heartbeat at a time. ⚡

Built with retro terminal aesthetics 🖥️ | Powered by WebSockets 🔌 | Secured by Authelia 🔐