pulse/README.md

PULSE - Pipelined Unified Logic & Server Engine

A distributed workflow orchestration platform for managing and executing complex multi-step operations across server clusters through an intuitive web interface.

Overview

PULSE is a centralized workflow execution system designed to orchestrate operations across distributed infrastructure. It provides a powerful web-based interface for defining, managing, and executing workflows that can span multiple servers, require human interaction, and perform complex automation tasks at scale.

Key Features

• Interactive Workflow Management: Define and execute multi-step workflows with conditional logic, user prompts, and decision points
• Distributed Execution: Run commands and scripts across multiple worker nodes simultaneously
• High Availability Architecture: Deploy redundant worker nodes in LXC containers with Ceph storage for fault tolerance
• Web-Based Control Center: Intuitive interface for workflow selection, monitoring, and interactive input
• Flexible Worker Pool: Scale horizontally by adding worker nodes as needed
• Real-Time Monitoring: Track workflow progress, view logs, and receive notifications

Architecture

PULSE consists of two core components:

PULSE Server

The central orchestration hub that:

• Hosts the web interface for workflow management
• Manages workflow definitions and execution state
• Coordinates task distribution to worker nodes
• Handles user interactions and input collection
• Provides real-time status updates and logging

PULSE Worker

Lightweight execution agents that:

• Connect to the PULSE server and await task assignments
• Execute commands, scripts, and code on target infrastructure
• Report execution status and results back to the server
• Support multiple concurrent workflow executions
• Automatically reconnect and resume on failure

┌─────────────────────┐
│   PULSE Server      │
│   (Web Interface)   │
└──────────┬──────────┘
           │
    ┌──────┴───────┬──────────────┬──────────────┐
    │              │              │              │
┌───▼────┐    ┌───▼────┐    ┌───▼────┐    ┌───▼────┐
│ Worker │    │ Worker │    │ Worker │    │ Worker │
│ Node 1 │    │ Node 2 │    │ Node 3 │    │ Node N │
└────────┘    └────────┘    └────────┘    └────────┘
     LXC Containers in Proxmox with Ceph

Deployment

Prerequisites

• Proxmox VE Cluster: Hypervisor environment for container deployment
• Ceph Storage: Distributed storage backend for high availability
• LXC Support: Container runtime for worker node deployment
• Network Connectivity: Communication between server and workers

Installation

PULSE Server

# Clone the repository
git clone https://github.com/yourusername/pulse.git
cd pulse

# Install dependencies
npm install  # or pip install -r requirements.txt

# Configure server settings
cp config.example.yml config.yml
nano config.yml

# Start the PULSE server
npm start  # or python server.py

PULSE Worker

# On each worker node (LXC container)
cd pulse-worker

# Install dependencies
npm install  # or pip install -r requirements.txt

# Configure worker connection
cp worker-config.example.yml worker-config.yml
nano worker-config.yml

# Start the worker daemon
npm start  # or python worker.py

High Availability Setup

Deploy multiple worker nodes across Proxmox hosts:

# Create LXC template
pct create 1000 local:vztmpl/ubuntu-22.04-standard_amd64.tar.zst \
  --rootfs ceph-pool:8 \
  --memory 2048 \
  --cores 2 \
  --net0 name=eth0,bridge=vmbr0,ip=dhcp

# Clone for additional workers
pct clone 1000 1001 --full --storage ceph-pool
pct clone 1000 1002 --full --storage ceph-pool
pct clone 1000 1003 --full --storage ceph-pool

# Start all workers
for i in {1000..1003}; do pct start $i; done

Usage

Creating a Workflow

1. Access the PULSE web interface at http://your-server:8080
2. Navigate to Workflows → Create New
3. Define workflow steps using the visual editor or YAML syntax
4. Specify execution targets (specific nodes, groups, or all workers)
5. Add interactive prompts where user input is required
6. Save and activate the workflow

Example Workflow

name: "System Update and Reboot"
description: "Update all servers in the cluster with user confirmation"
steps:
  - name: "Check Current Versions"
    type: "execute"
    targets: ["all"]
    command: "apt list --upgradable"
    
  - name: "User Approval"
    type: "prompt"
    message: "Review available updates. Proceed with installation?"
    options: ["Yes", "No", "Cancel"]
    
  - name: "Install Updates"
    type: "execute"
    targets: ["all"]
    command: "apt-get update && apt-get upgrade -y"
    condition: "prompt_response == 'Yes'"
    
  - name: "Reboot Confirmation"
    type: "prompt"
    message: "Updates complete. Reboot all servers?"
    options: ["Yes", "No"]
    
  - name: "Rolling Reboot"
    type: "execute"
    targets: ["all"]
    command: "reboot"
    strategy: "rolling"
    condition: "prompt_response == 'Yes'"

Running a Workflow

1. Select a workflow from the dashboard
2. Click Execute
3. Monitor progress in real-time
4. Respond to interactive prompts as they appear
5. View detailed logs for each execution step

Configuration

Server Configuration (config.yml)

server:
  host: "0.0.0.0"
  port: 8080
  secret_key: "your-secret-key"

database:
  type: "postgresql"
  host: "localhost"
  port: 5432
  name: "pulse"

workers:
  heartbeat_interval: 30
  timeout: 300
  max_concurrent_tasks: 10

security:
  enable_authentication: true
  require_approval: true

Worker Configuration (worker-config.yml)

worker:
  name: "worker-01"
  server_url: "http://pulse-server:8080"
  api_key: "worker-api-key"
  
resources:
  max_cpu_percent: 80
  max_memory_mb: 1024
  
executor:
  shell: "/bin/bash"
  working_directory: "/tmp/pulse"
  timeout: 3600

Features in Detail

Interactive Workflows

• Pause execution to collect user input via web forms
• Display intermediate results for review
• Conditional branching based on user decisions
• Multi-choice prompts with validation

Mass Execution

• Run commands across all workers simultaneously
• Target specific node groups or individual servers
• Rolling execution for zero-downtime updates
• Parallel and sequential execution strategies

Monitoring & Logging

• Real-time workflow execution dashboard
• Detailed per-step logging and output capture
• Historical execution records and analytics
• Alert notifications for failures or completion

Security

• Role-based access control (RBAC)
• API key authentication for workers
• Workflow approval requirements
• Audit logging for all actions

---

PULSE - Orchestrating your infrastructure, one heartbeat at a time.