pulse/README.md

# PULSE - Pipelined Unified Logic & Server Engine

A distributed workflow orchestration platform for managing and executing complex multi-step operations across server clusters through a retro terminal-themed web interface.

> **Security Notice:** This repository is hosted on Gitea and is version-controlled. **Never commit secrets, credentials, passwords, API keys, or any sensitive information to this repo.** All sensitive configuration belongs exclusively in `.env` files which are listed in `.gitignore` and must never be committed. This includes database passwords, worker API keys, webhook secrets, and internal IP details.

**Design System**: [web_template](https://code.lotusguild.org/LotusGuild/web_template) — shared CSS, JS, and layout patterns for all LotusGuild apps

## Styling & Layout

PULSE uses the **LotusGuild Terminal Design System**. For all styling, component, and layout documentation see:

- [`web_template/README.md`](https://code.lotusguild.org/LotusGuild/web_template/src/branch/main/README.md) — full component reference, CSS variables, JS API
- [`web_template/base.css`](https://code.lotusguild.org/LotusGuild/web_template/src/branch/main/base.css) — unified CSS (`.lt-*` classes)
- [`web_template/base.js`](https://code.lotusguild.org/LotusGuild/web_template/src/branch/main/base.js) — `window.lt` utilities (toast, modal, WebSocket helpers, fetch)
- [`web_template/aesthetic_diff.md`](https://code.lotusguild.org/LotusGuild/web_template/src/branch/main/aesthetic_diff.md) — cross-app divergence analysis and convergence guide
- [`web_template/node/middleware.js`](https://code.lotusguild.org/LotusGuild/web_template/src/branch/main/node/middleware.js) — Express auth, CSRF, CSP nonce middleware

**Pending convergence items (see aesthetic_diff.md):**
- Extract inline `<style>` from `public/index.html` into `public/style.css` and extend `base.css`
- Use `lt.autoRefresh.start(refreshData, 30000)` instead of raw `setInterval`

## Overview

PULSE is a centralized workflow execution system designed to orchestrate operations across distributed infrastructure. It provides a powerful web-based interface with a vintage CRT terminal aesthetic for defining, managing, and executing workflows that can span multiple servers, require human interaction, and perform complex automation tasks at scale.

### Key Features

- **🎨 Retro Terminal Interface**: Phosphor green CRT-style interface with scanlines, glow effects, and ASCII art
- **⚡ Quick Command Execution**: Instantly execute commands on any worker with built-in templates and command history
- **📊 Real-Time Worker Monitoring**: Live system metrics including CPU, memory, load average, and active tasks
- **🔄 Interactive Workflow Management**: Define and execute multi-step workflows with conditional logic and user prompts
- **🌐 Distributed Execution**: Run commands across multiple worker nodes simultaneously via WebSocket
- **📈 Execution Tracking**: Comprehensive logging with formatted output, re-run capabilities, and JSON export
- **🔐 SSO Authentication**: Seamless integration with Authelia for enterprise authentication
- **🧹 Auto-Cleanup**: Automatic removal of old executions with configurable retention policies
- **🔔 Terminal Notifications**: Audio beeps and visual toasts for command completion events

## Architecture

PULSE consists of two core components:

### PULSE Server
**Location:** `10.10.10.65` (LXC Container ID: 122)
**Directory:** `/opt/pulse-server`

The central orchestration hub that:
- Hosts the retro terminal web interface
- Manages workflow definitions and execution state
- Coordinates task distribution to worker nodes via WebSocket
- Handles user interactions through Authelia SSO
- Provides real-time status updates and logging
- Stores all data in MariaDB database

**Technology Stack:**
- Node.js 20.x
- Express.js (web framework)
- WebSocket (ws package) for real-time bidirectional communication
- MySQL2 (MariaDB driver)
- Authelia SSO integration

### PULSE Worker
**Example:** `10.10.10.151` (LXC Container ID: 153, hostname: pulse-worker-01)
**Directory:** `/opt/pulse-worker`

Lightweight execution agents that:
- Connect to PULSE server via WebSocket with heartbeat monitoring
- Execute shell commands and report results in real-time
- Provide system metrics (CPU, memory, load, uptime)
- Support concurrent task execution with configurable limits
- Automatically reconnect on connection loss

**Technology Stack:**
- Node.js 20.x
- WebSocket client
- Child process execution
- System metrics collection

```
┌─────────────────────────────────┐
│   PULSE Server (10.10.10.65)   │
│   Terminal Web Interface + API  │
│   ┌───────────┐ ┌──────────┐   │
│   │ MariaDB   │ │ Authelia │   │
│   │ Database  │ │   SSO    │   │
│   └───────────┘ └──────────┘   │
└────────────┬────────────────────┘
             │ WebSocket
    ┌────────┴────────┬───────────┐
    │                 │           │
┌───▼────────┐   ┌───▼────┐  ┌──▼─────┐
│ Worker 1   │   │Worker 2│  │Worker N│
│10.10.10.151│   │  ...   │  │  ...   │
└────────────┘   └────────┘  └────────┘
 LXC Containers in Proxmox with Ceph
```

## Installation

### Prerequisites

- **Node.js 20.x** or higher
- **MariaDB 10.x** or higher
- **Authelia** configured for SSO (optional but recommended)
- **Network Connectivity** between server and workers

### PULSE Server Setup

```bash
# Clone repository
cd /opt
git clone <your-repo-url> pulse-server
cd pulse-server

# Install dependencies
npm install

# Create .env file with configuration
cat > .env << EOF
# Server Configuration
PORT=8080
SECRET_KEY=your-secret-key-here

# MariaDB Configuration
DB_HOST=10.10.10.50
DB_PORT=3306
DB_NAME=pulse
DB_USER=pulse_user
DB_PASSWORD=your-db-password

# Worker API Key (for worker authentication)
WORKER_API_KEY=your-worker-api-key

# Auto-cleanup configuration (optional)
EXECUTION_RETENTION_DAYS=30
EOF

# Create systemd service
cat > /etc/systemd/system/pulse.service << EOF
[Unit]
Description=PULSE Workflow Orchestration Server
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/opt/pulse-server
ExecStart=/usr/bin/node server.js
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

# Start service
systemctl daemon-reload
systemctl enable pulse.service
systemctl start pulse.service
```

### PULSE Worker Setup

```bash
# On each worker node
cd /opt
git clone <your-repo-url> pulse-worker
cd pulse-worker

# Install dependencies
npm install

# Create .env file
cat > .env << EOF
# Worker Configuration
WORKER_NAME=pulse-worker-01
PULSE_SERVER=http://10.10.10.65:8080
PULSE_WS=ws://10.10.10.65:8080
WORKER_API_KEY=your-worker-api-key

# Performance Settings
HEARTBEAT_INTERVAL=30
MAX_CONCURRENT_TASKS=5
EOF

# Create systemd service
cat > /etc/systemd/system/pulse-worker.service << EOF
[Unit]
Description=PULSE Worker Node
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/opt/pulse-worker
ExecStart=/usr/bin/node worker.js
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

# Start service
systemctl daemon-reload
systemctl enable pulse-worker.service
systemctl start pulse-worker.service
```

## Usage

### Quick Command Execution

1. Access PULSE at `http://your-server:8080`
2. Navigate to **⚡ Quick Command** tab
3. Select a worker from the dropdown
4. Use **Templates** for pre-built commands or **History** for recent commands
5. Enter your command and click **Execute**
6. View results in the **Executions** tab

**Built-in Command Templates:**
- System Info: `uname -a`
- Disk Usage: `df -h`
- Memory Usage: `free -h`
- CPU Info: `lscpu`
- Running Processes: `ps aux --sort=-%mem | head -20`
- Network Interfaces: `ip addr show`
- Docker Containers: `docker ps -a`
- System Logs: `tail -n 50 /var/log/syslog`

### Worker Monitoring

The **Workers** tab displays real-time metrics for each worker:
- System information (OS, architecture, CPU cores)
- Memory usage (used/total with percentage)
- Load averages (1m, 5m, 15m)
- System uptime
- Active tasks vs. maximum concurrent capacity

### Execution Management

- **View Details**: Click any execution to see formatted logs with timestamps, status, and output
- **Re-run Command**: Click "Re-run" button in execution details to repeat a command
- **Download Logs**: Export execution data as JSON for auditing
- **Clear Completed**: Bulk delete finished executions
- **Auto-Cleanup**: Executions older than 30 days are automatically removed

### Workflow Creation (Future Feature)

1. Navigate to **Workflows** → **Create New**
2. Define workflow steps using JSON syntax
3. Specify target workers
4. Add interactive prompts where needed
5. Save and execute

## Features in Detail

### Terminal Aesthetic
- Phosphor green (#00ff41) on black (#0a0a0a) color scheme
- CRT scanline animation effect
- Text glow and shadow effects
- ASCII box-drawing characters for borders
- Boot sequence animation on first load
- Hover effects with smooth transitions

### Real-Time Communication
- WebSocket-based bidirectional communication
- Instant command result notifications
- Live worker status updates
- Terminal beep sounds for events
- Toast notifications with visual feedback

### Execution Tracking
- Formatted log display (not raw JSON)
- Color-coded success/failure indicators
- Timestamp and duration for each step
- Scrollable output with syntax highlighting
- Persistent history with pagination
- Load More button for large execution lists

### Security
- Authelia SSO integration for user authentication
- API key authentication for workers
- User session management
- Admin-only operations (worker deletion, workflow management)
- Audit logging for all executions

### Performance
- Automatic cleanup of old executions (configurable retention)
- Pagination for large execution lists (50 at a time)
- Efficient WebSocket connection pooling
- Worker heartbeat monitoring
- Database connection pooling

## Configuration

### Environment Variables

**Server (.env):**
```bash
PORT=8080                           # Server port
SECRET_KEY=<random-string>          # Session secret
DB_HOST=10.10.10.50                # MariaDB host
DB_PORT=3306                        # MariaDB port
DB_NAME=pulse                       # Database name
DB_USER=pulse_user                  # Database user
DB_PASSWORD=<password>              # Database password
WORKER_API_KEY=<api-key>           # Worker authentication key
EXECUTION_RETENTION_DAYS=30         # Auto-cleanup retention (default: 30)
```

**Worker (.env):**
```bash
WORKER_NAME=pulse-worker-01         # Unique worker name
PULSE_SERVER=http://10.10.10.65:8080   # Server HTTP URL
PULSE_WS=ws://10.10.10.65:8080          # Server WebSocket URL
WORKER_API_KEY=<api-key>            # Must match server key
HEARTBEAT_INTERVAL=30               # Heartbeat seconds (default: 30)
MAX_CONCURRENT_TASKS=5              # Max parallel tasks (default: 5)
```

## Database Schema

PULSE uses MariaDB with the following tables:

| Table | Purpose |
|-------|---------|
| `users` | User accounts synced from Authelia SSO |
| `workers` | Worker node registry with connection metadata |
| `workflows` | Workflow definitions stored as JSON |
| `executions` | Execution history with logs, status, and timestamps |

### `executions` Table Key Columns

| Column | Description |
|--------|-------------|
| `id` | Auto-increment primary key |
| `worker_id` | Foreign key to workers |
| `command` | The command that was executed |
| `status` | `running`, `completed`, `failed` |
| `output` | Command output / log (JSON or text) |
| `created_at` | Execution start timestamp |
| `completed_at` | Execution end timestamp |

### `workers` Table Key Columns

| Column | Description |
|--------|-------------|
| `id` | Auto-increment primary key |
| `name` | Worker name (from `WORKER_NAME` env) |
| `last_seen` | Last heartbeat timestamp |
| `status` | `online`, `offline` |
| `metadata` | JSON blob of system info |

## Troubleshooting

### Worker Not Connecting
```bash
# Check worker service status
systemctl status pulse-worker

# Check worker logs
journalctl -u pulse-worker -n 50 -f

# Verify API key matches server
grep WORKER_API_KEY /opt/pulse-worker/.env
```

### Commands Stuck in "Running"
- This was fixed in recent updates - restart the server:
```bash
systemctl restart pulse.service
```

### Clear All Executions
Use the database directly if needed:
```bash
mysql -h 10.10.10.50 -u pulse_user -p pulse
> DELETE FROM executions WHERE status IN ('completed', 'failed');
```

## Development

### Recent Updates

**Phase 1-6 Improvements:**
- Formatted log display with color-coding
- Worker system metrics monitoring
- Command templates and history
- Re-run and download execution features
- Auto-cleanup and pagination
- Terminal aesthetic refinements
- Audio notifications and visual toasts

See git history for detailed changelog.

### Future Enhancements
- Full workflow system implementation
- Multi-worker command execution
- Scheduled/cron job support
- Execution search and filtering
- Dark/light theme toggle
- Mobile-responsive design
- REST API documentation
- Webhook integrations

## License

MIT License - See LICENSE file for details

---

**PULSE** - Orchestrating your infrastructure, one heartbeat at a time. ⚡

Built with retro terminal aesthetics 🖥️ | Powered by WebSockets 🔌 | Secured by Authelia 🔐