gandalf/README.md

GANDALF (Global Advanced Network Detection And Link Facilitator)

Because it shall not let problems pass.

Network monitoring dashboard for the LotusGuild Proxmox cluster. Deployed on LXC 157 (monitor-02 / 10.10.10.9), reachable at gandalf.lotusguild.org.

Design System: web_template: shared CSS, JS, and layout patterns for all LotusGuild apps

Why this exists

A Proxmox/Ceph cluster lives or dies by its network, and network problems are sneaky: a link that flaps, a fibre with weak receive power, a switch port that quietly negotiated 1 Gbps. GANDALF watches the cluster's links from three angles at once (host NIC state from Prometheus, switch ports and LLDP from the UniFi controller, and reachability pings), alerts only on regressions (a link that was up and went down; ports that were never connected are learned as baseline and ignored), and files one ticket per real problem. When something looks wrong, a click on the switch port runs a full on-host diagnostic through PULSE (ethtool, SFP optics, kernel log, LLDP) and explains what it found.

Gallery

Dashboard: alert queue with linked tickets, per-host interface status and the UniFi device inventory.

Topology: every device drawn from its real faceplate or chassis, with each cable leaving the port it is plugged into, styled by medium and speed, with live traffic.

Device pages: search by hostname, IP, MAC, model or interface from the dashboard, then open one page per host, switch, router, PDU or modem. Each interface or port is one row with its state, speed, traffic and neighbour. Expanding a row shows duplex, carrier flaps, error and drop rates, PoE, LLDP, and SFP optical readings (temperature, TX/RX power) with colour-coded thresholds.

Port inspector: each switch and router page has a front-panel diagram. Click a port for its stats, LLDP neighbour and a side-by-side switch/server path check.

One-click diagnostics: GANDALF asks a PULSE worker to SSH to the host on the other end of the link and collect everything in one pass, then analyzes it: carrier, duplex and speed mismatches, SFP power levels, CRC errors, link flaps, recent kernel events and LLDP validation.

Suppressions: timed or manual maintenance windows per host, interface or UniFi device, with a full history.

Light theme:

_{Screenshots come from a throwaway local instance seeded with fictional data (documentation IP ranges), with PULSE replaced by a small mock. See scripts/demo/.}

Styling & Layout

GANDALF uses the LotusGuild Terminal Design System. For all styling, component, and layout documentation see:

• web_template/README.md: full component reference, CSS variables, JS API
• web_template/base.css: unified CSS (.lt-* classes)
• web_template/base.js: window.lt utilities (toast, modal, auto-refresh, fetch helpers)
• web_template/python/base.html: Jinja2 base template
• web_template/python/auth.py: @require_auth decorator pattern

---

Architecture

Two processes share a MariaDB database:

Process	Service	Role
app.py	gandalf.service	Flask web dashboard (gunicorn, port 8000)
monitor.py	gandalf-monitor.service	Background polling daemon

[Prometheus :9090]  ──▶
[UniFi Controller]  ──▶  monitor.py ──▶ MariaDB ◀── app.py ──▶ nginx ──▶ Authelia ──▶ Browser
[Pulse Worker]      ──▶
[SSH / ethtool]     ──▶
[AT&T BGW320]       ──▶

Data Sources

Source	What it provides
Prometheus (10.10.10.48:9090)	Physical NIC link state + traffic/error rates via node_exporter
UniFi API (https://10.10.10.1)	Switch port stats, device status, LLDP neighbor table, PoE data
Pulse Worker	SSH relay: runs ethtool + SFP DOM queries on each Proxmox host
Ping	Reachability for hosts without node_exporter (e.g. PBS)
AT&T BGW320 (http://192.168.1.254)	Fiber light levels, PON and WAN state, WAN counters, scraped read-only from the gateway's status pages (no login)

Monitored Hosts (Prometheus / node_exporter)

Host	Prometheus Instance
large1	10.10.10.2:9100
compute-storage-01	10.10.10.4:9100
monitor-01	10.10.10.8:9100
monitor-02	10.10.10.9:9100
compute-storage-gpu-01	10.10.10.10:9100
storage-01	10.10.10.11:9100

Ping-only (no node_exporter): pbs (10.10.10.3)

---

Pages

Dashboard (/)

• Device search (see Device pages below)
• Active alerts first, then the WAN / Fiber panel, hosts, UniFi devices, and recently resolved alerts
• Real-time host status grid with per-NIC link state (UP / DOWN / degraded)
• WAN / Fiber panel: WAN and PON state, Rx/Tx power, optics temperature, throughput, errors, gateway uptime and last restart, 7-day Rx trend, recent WAN outages
• UniFi device table (switches, APs, gateway)
• Active alerts table with severity, target, consecutive failures, ticket link
• Follow-up flags (see Follow-up below): a ⏳ badge on alerts active for days, a "ticket closed" badge when the ticket was closed but the alert is still active, and a Needs Follow-up panel of tickets left open after their alert cleared. A "TO FOLLOW UP" chip in the status bar counts them all.
• Quick-suppress modal: apply timed or manual suppression from any alert row
• Auto-refreshes every 30 seconds via /api/status + /api/network

Alert history (/events)

Every alert Gandalf raised in the last 24 hours to 90 days, including ones that cleared before anyone saw them: when each started, how long it lasted, failed checks, and its ticket.

• Filter by device (name, port or interface), alert type, severity, and open or cleared. Filters are kept in the URL, so /events?target=large1&range=90d is a shareable link; device pages and tickets link to their own history this way.
• Recurring lists the same alert on the same target more than once in the range, worst first, since a flapping cable or optic shows up here before anyone notices the tickets.
• API: GET /api/events/history?range=24h|7d|30d|90d&type=&target=&severity=&status=all|active|resolved&offset= (100 per page).

Rack (/rack)

The rack front, drawn from the rack units set on cutsheet items, top first.

• Linking items to devices: a cutsheet item can be named after the product ("UDM-Pro") and linked to the live device with its Gandalf device field ("Dream Machine Pro"). Linked UniFi gear with a faceplate is drawn with its real faceplate and live port state (the UPS 2U Pro shows its front only); patch panels get a generic panel with each port's label; anything else is a labelled plate with its live status. Cables may use either name: checks, suggestions, tickets and the device page treat an item and its device as the same thing.
• Drawn fronts: gear without a live faceplate can pick a Front drawing on its cutsheet item: UniFi NVR Pro (a full faceplate, UNVRPRO in static/faceplates.js, laid out from the CC0 NetBox device library image), Sliger CX4712 4U (10 numbered bays), Rosewill RSV-L4500U 4U (perforated fan door), the UniFi blank 24-port keystone patch panel (UACC-Rack-Panel-Patch-Blank-24: keystone jacks appear where cables are documented), or a generic 1U/2U/4U server. The power light shows the linked device's live status. Case drawings live in static/fronts.js.
• Shelves: an item can sit on a shelf item instead of having a rack unit. The shelf draws what is on it side by side (mini PCs, desk switches), each with its status and a link to its page.
• Posts, facing and views: each item hangs on the front posts, the rear posts, or both (full depth), and its ports face the front or the rear. On a 4-post rack a front-post and a rear-post item can share a unit; a full-depth item takes it on both. The page has three views: Front (the front posts), Rear posts (gear on the rear posts seen from the front, i.e. what sits behind the front gear, such as a PDU or a shelf mounted from the rear but facing forward) and Rear (from behind). An item seen from the side it doesn't face shows its back. Items on a shelf take the shelf's posts and facing.
• Shelves to scale: a shelf draws its contents left to right by Position on shelf, at true scale (a 19" panel is 482.6 mm). Gear with a drawing, such as the IceWhale ZimaBoard 832 (traced from IceWhale's labelled photo of its port end), shows its real ports: the onboard RJ45 link lights follow enp2s0 (left) and enp3s0 (right), and the status glow follows the host. The "ZimaBoard 832 + ConnectX-3" drawing adds the single-port SFP+ card lying flat in the side PCIe slot, as in the rack, with a DAC attached; its round dot is Gandalf's link mark for enp1s0 (the slot is PCI bus 1), not an LED on the card. UniFi desk switches (Flex Mini, Flex) on a shelf are drawn with their live faceplate at their real width. Anything else is a labelled box.
• More drawn fronts: the Sliger CX4712 with its red, white or blue perforated front fitted (as in the rack photo, drive cages faint behind the mesh), and the UniFi brush panel (UACC-Rack-Panel-Brush-1U, from Ubiquiti's product images).
• Gear resting on other gear: a shelf item with Front drawing "No shelf: gear resting on what is below" draws no tray, so a switch lying on top of the NVR is a "shelf" in the unit above the NVR whose contents sit directly on it.
• Racks: admins name a rack and set its height with Rack settings on the page, and add more with + Rack (a tab per rack). Renaming a rack moves its items; an empty rack can be deleted. Each item has a rack (empty means the first). Until a rack is saved, racks come from config: racks: [{"name": "Main", "units": 42}, ...], or one rack rack.name (default "Rack 1") with rack.units (default 42); saving the first one carries its items over.
• Every documented port shows its cutsheet state (green matches, red mismatch, grey unconfirmed). Hovering a port lights up every port on the same cable run, through patch panels, and names the cables; clicking opens the switch port or the cable.
• Runs of 3 or more empty units fold into one band; "Show empty units" draws them in full (remembered per browser). Every device is drawn at the full 19" width.
• Download SVG / PNG saves the current rack on a white page for printing or planning.
• Items that would overlap another or run past the top of the rack are listed, not drawn over. Where two claim the same top unit, the one recorded first is drawn. Items without a rack unit are listed below.
• The rack keeps its own dark "hardware" palette in both themes.

Topology (/topology)

The whole network as the switches see it now, top down: the internet, the modem, the gateway, then each switch by how many hops it is from the gateway, the gear and unknown devices below them, and the servers along the bottom. Power gear (PDU, UPS, power strip) sits in a rail on the right.

• What needs a look is listed above the diagram: devices down, degraded or with active alerts, cables down or last seen elsewhere, ports named after the wrong device, and cables with errors. Click one to jump to it. With nothing to report it says all clear. Problem cables are drawn in full colour with a glow and always shown, whatever the filters; healthy cables are drawn quietly.
• A card per device, all the same height so the whole network reads at a glance: name, state (coloured edge and dot), and a picture: UniFi gear shows its real faceplate (the same drawings as the port inspector and rack page), cropped to the ports; servers show the Front drawing of their cutsheet item and a socket per physical NIC, so every cable lands on the NIC it is plugged into (NICs marked unused are dimmed). The internet, modem and gateway sit side by side along the top.
• Cables run like cable management: each leaves the card edge right below its port (a short tail shows the port), drops into a tray above the row it goes to, runs along its own lane (10G on the upper lanes), and drops straight into its socket, so cables never lie on top of each other and only cross at right angles. Devices are placed clear of the cables passing down through their row, and each sits under its fastest cable.
• One line per physical cable, leaving the exact port: a switch-to-switch cable is reported by both switches and drawn once. A server with two links (10G and 1G) has two cables. Colour is the cable type (DAC, multimode fibre, the provider's fibre, an SFP+ module that doesn't say which, copper) and thickness is the speed. A cable that is down is dashed red.
• Last seen: a device with no live cable is drawn on the port it was last seen on, dotted red.
• Unknown devices: something plugged into a switch port that Gandalf can't name is a grey stub, labelled with the port name (or its address), linking to that port. By default each switch's unknown devices are one block with one bundled cable ("10 unknown devices"); click it to show them one by one, or pick "one by one" or "hidden" in the toolbar.
• Port names: a ⚠ marks a port named after one device with a different one plugged in (a port called "Large1 Management" with the AP on it). Generic names ("Port 4", "SFP+ 2", "Uplink") and "Downlink to X" with X on the other end never flag.
• Alerts: a device with active alerts (not suppressed) gets a ring and a count in its worst severity colour, and the NIC it is about is marked.
• Click a device for its page, or a port on a faceplate for that port. Click a cable for its details: both ends, speed, cable type and module, traffic each way and how busy it is, errors and drops, and its cutsheet record. Hovering a device traces its path to the internet.
• Drag to pan, scroll or pinch to zoom, Fit (or 0) to see everything, + and - to zoom. Show unknown devices grouped, one by one or not at all. Server 1G links (off by default) adds each server's slower second cable, its management link; Power gear (off by default) adds the PDU, UPS and power strip. Traffic labels and dimming everything without an alert can be switched too. The filters are remembered per browser. Refreshes every 30 seconds without moving the view.
• API: GET /api/topology returns {nodes, links, warnings, updated} (see topology.py).

Health check (/health)

No login needed. Returns 200 when the database answers and the monitor has completed a check recently, otherwise 503 with the failing check: {"status": "degraded", "checks": {"db": "ok", "monitor": "stale", "monitor_age_s": 742}}. The monitor counts as stalled after five missed polls (at least 10 minutes), so point an uptime monitor at it to catch a hung gandalf-monitor, not just a dead web app.

Device pages (/device/<name>)

One page per host, switch, router, AP, PDU, or modem. Reach them from the search box on the dashboard (/ focuses it), or from any device name. Search matches hostname, IP, MAC, model, type, interface and port names, and LLDP neighbors.

• Overview: status, IP, MAC, model, firmware, uptime, CPU, memory, clients, temperatures, power draw
• Alerts: active and past alerts for this device
• WAN (routers): connection state, speed, latency, availability
• Connected to (hosts): switch ports that see this host over LLDP
• Outlets (PDUs): per-outlet state, power, and auto power-cycle setting
• Port inspector (switches, routers): see below
• Interfaces / Ports: one row each with state, speed, RX/TX rate, and neighbor, filterable. Expanding a host's row shows duplex, auto-negotiation, carrier changes, error and drop rates, traffic bars, PoE, LLDP detail, and the SFP / optical panel (vendor/PN, temperature, voltage, bias, TX/RX power, RX−TX delta)
• WAN / Fiber (modem): the same panel as the dashboard

These replaced the Link Debug page; /links now redirects to the dashboard. Hosts that Prometheus sees but config.json doesn't list still get a page, with their state shown as unknown because they aren't alerted on.

Port inspector (on switch and router device pages)

A realistic front panel, drawn as SVG from the model's real geometry in static/faceplates.js (FACEPLATES). /inspector now redirects to the dashboard. Link to a port with /device/<name>#port-<n>.

SFP cages show what UniFi reports is in them: empty, a fibre optic (two LC sockets), a DAC (cable boot), or an unidentified module. A module raising a receive or transmit fault is outlined in red. The port panel's SFP Module section lists type, part, serial and faults, plus the module's optical readings (TX/RX power, temperature, voltage, bias) when it reports them. Copper DACs don't. Both ends of a DAC report the same cable serial, so a DAC seen on two ports links to its other end.

Cutsheet marks: a port with a cable documented in the cutsheet gets a dot in its corner: green when it matches what Gandalf sees, red for a mismatch (wrong neighbor, no link, missing port, port conflict), grey when it is documented but can't be confirmed (planned, dark, through a panel, or the far end isn't visible). Hovering the port names the cable. "Cutsheet marks" in the legend hides them; the choice is remembered in the browser.

Inventory link: the device page's 📦 button opens the device's record in the inventory (inventory.base_url). Admins set it with ✎ by pasting the record's URL; only URLs on the inventory host are accepted. Until one is set, the button searches the inventory for the model. Tickets include the link too.

Port block colours:

Colour	State
Green	Up, no active PoE
Amber	Up with active PoE draw
Cyan	Uplink port (up)
Grey	Down
White outline	Currently selected

Clicking a port, on the panel or in the port table, opens the detail panel showing:

• Link stats (status, speed, duplex, auto-neg, media type)
• PoE (class, max wattage, current draw, mode)
• Traffic (TX/RX rates)
• Errors/drops per second
• LLDP Neighbor section (system name, port ID, chassis ID, management IPs)
• Path Debug (auto-appears when LLDP system_name matches a known server): two-column comparison of the switch port stats vs. the server NIC stats, including SFP DOM data if the server side has an SFP module

Port neighbors: UniFi's LLDP data identifies a neighbor only by MAC, so Gandalf names it by matching that MAC against host NICs (node_network_info from node_exporter) and UniFi devices, then falling back to management IPs. Servers that don't send LLDP are found through UniFi's wired client table. Uplink ports are skipped there, and neighbors that can't be matched show their MAC. Bonded NICs share a MAC, so only one member's port may be identified.

Path check and diagnostics requirements:

1. The server's node_exporter is scraped, so its NIC MACs are in Prometheus
2. The Pulse worker can SSH to the server
3. Optional: lldpd on the server lets diagnostics confirm the cabling from the server's side

Supported models (keys are UniFi model codes):

Key	Model	Front panel
UDMPRO	Dream Machine Pro	8×RJ45 LAN, RJ45 WAN, SFP+ WAN, SFP+ LAN, screen, drive bay
US24PRO	USW Pro 24 PoE	24×RJ45 in two blocks of 12, 2×SFP+
USL8A	USW Aggregation	8×SFP+ in four stacked pairs
USF5P	USW Flex	5×RJ45, PoE in on port 1, PoE out on 2-5
USMINI	USW Flex Mini	5×RJ45, PoE in on port 1
USPPDUP	USP PDU Pro	20 outlets (4 USB-C, 16 AC), management port 21, surge pass-through ports 22-23
USPDA2B	UPS 2U Pro	Front with touchscreen outlet tiles; rear with 8 outlets, EPO, surge and MGMT ports

Port LEDs use the colors from each model's quick start guide. Other models use plain blocks until they get an entry in FACEPLATES.

Cutsheet (/cutsheet)

Where every cable is meant to go, checked against what Gandalf actually sees. Anyone who can log in can view it; only the admin group can edit, and every change is logged with who made it and when.

• Inventory: things discovery can't see: patch panels, wall jacks, shelves, and rack positions for devices (use a device's exact Gandalf name). Panels and jacks are passive: a cable on a port's front continues through the same port's rear.
• Cables: two endpoints each (device or item, port, and front/rear for panels and jacks), plus ID, type, length, color, status (planned, installed, dark, retired) and notes. Leave the far end blank for a run that ends nowhere yet, such as an unterminated fibre.
• Paths: each cable shows its full run through any panels and jacks, e.g. switch port 13 → panel port 5 → bedroom jack.
• Plan vs. actual: each installed run ending on a switch or router port is compared with that port's discovered neighbor (LLDP, the UniFi client table, or a DAC's serial reported at both ends): matches, wrong neighbor, no link, missing port, or port conflict (two installed cables in one port). Runs that end at a jack are unverified while linked and idle when not. Planned, dark and retired cables aren't checked.
• Undocumented links: live switch ports no cable mentions, with an Add button that pre-fills what Gandalf knows. Each switch port panel also shows its planned cable, or links to add one.
• Labels to make: cable IDs (one per cord end) and panel-port labels not yet stuck on, sized for typing on a handheld label maker. Tick each off once it's applied; Print list prints just this.
• CSV: export, and import that adds or updates cables by ID. A file with any invalid row imports nothing.
• Cables, panels and jacks are included in the dashboard search.
• Trace a cord (admins): pick a panel port and press Start, then unplug that cord at the panel. Gandalf reads every switch port's state from UniFi every 3 seconds and shows the one that drops, with a button to record the cable, then confirms when it's plugged back in. The device on that port loses its link for those seconds. A cord whose far end has nothing linked can't be traced this way.
• Mismatch alerts: the monitor runs the same check every poll. A mismatched run opens a cable_mismatch event on the switch straight away and resolves it once fixed; it becomes a P4 ticket only after cutsheet.mismatch_ticket_hours.

Suppressions (/suppressions)

• Create timed (30 min / 1 hr / 4 hr / 8 hr) or manual suppressions
• Target types: host, interface, UniFi device, or global
• Active suppressions table with one-click removal
• Suppression history (last 50)
• Available targets reference grid (all known hosts + interfaces)

Maintenance: admins get a Maintenance button in the header. It starts a global suppression for 30 min, 1, 2 or 4 hours with a reason (by default "Maintenance"). While it runs, every page shows a banner saying when tickets resume, with End now for admins. Gandalf keeps checking and recording alerts, and anything still failing when maintenance ends gets a ticket as usual. Use it before a UniFi update, a switch swap or recabling.

---

Alert Logic

Ticket Triggers

Condition	Priority
UniFi device offline (≥unifi_failure_threshold consecutive checks)	P2 High
Proxmox host NIC link-down regression (≥2 consecutive checks)	P2 High
Host unreachable via ping (≥host_failure_threshold consecutive checks)	P2 High
Configured host has no metrics in Prometheus (host or node_exporter down, ≥host_failure_threshold consecutive checks)	P2 High
Prometheus unavailable, so host and interface alerting is paused (≥2 consecutive checks)	P2 High
≥3 hosts simultaneously reporting interface or metrics failures	P1 Critical
WAN down: broadband or optical status not Up, or PON not in O5 (≥modem.failure_threshold)	P1 Critical
Modem not responding (≥modem.failure_threshold)	P2 High
Fiber Rx/Tx power or optics temperature in the module's alarm range, or a supply voltage or Tx bias alarm flag (≥modem.failure_threshold)	P2 High
Fiber reading in the module's warning range (≥modem.failure_threshold)	P3
Rx power rx_drift_db or more below its 7-day median (≥modem.failure_threshold)	P3
Modem pages unreadable, e.g. after a firmware update (≥modem.failure_threshold)	P3
UniFi logs that Smart Power auto power-cycled the modem	P3
UniFi logs that the WAN failed multiple times in 24 hours (at most one ticket a day)	P2
An installed cable doesn't match the cutsheet (wrong neighbor, no link, missing port, port conflict) for cutsheet.mismatch_ticket_hours	P4

A failed Pulse ping check counts as "unknown": it neither raises nor resolves host_unreachable.

What a ticket contains

Tickets are Markdown. Besides the alert's own facts and a numbered "What to check" list, each one adds whatever Gandalf knows around the problem:

• Where it is connected: the switch port(s) the host or device plugs into, with link state, speed, PoE and SFP module, linked to that port on the switch's page. A host that has dropped off the network has lost its LLDP entry, so Gandalf keeps the last port it saw each device on (known_uplinks in monitor_state) and shows that, marked "last seen linked there".
• The cutsheet cables ending on the device or port, with their full path through patch panels
• History: how often the same alert fired in the last 7 and 30 days, and when it last cleared; 3 or more in a week is called out as recurring
• Other active alerts on the same device, with their ticket numbers
• Links to the device in Gandalf (from gandalf.public_url)

Titles keep the [target][auto][production]... tag format, which Tinker uses to spot duplicates.

Unused interfaces

An admin can mark a host interface as unused on its device page (Mark unused on a down interface, with an optional reason; In use again to undo). An unused interface:

• never raises a link-down alert, and can't count toward the cluster-wide alert;
• doesn't count toward the host's status, so a host whose only down ports are unused shows as up;
• shows as "unused" (greyed, dashed dot) on the dashboard, the device page and the rack drawings;
• has any open link-down alert closed quietly when it is marked (no recovery note: nothing recovered).

Marks are stored in unused_interfaces (host, interface, reason, who, when).

Recovery notes

When an alert that opened a ticket clears, the monitor adds a comment to that ticket: when it cleared, how long it lasted, how many checks failed, and a link back to Gandalf. Gandalf never closes or changes the status of a ticket; that is left to a person. The one-shot WAN reports (modem power-cycled, WAN unstable) get no note, since they describe something already over.

A note that fails to post is retried every cycle for 24 hours. Posting needs a Tinker API key with read_write scope; if Tinker answers 401 or 403, the monitor logs one error and stops trying until it is restarted.

Follow-up

Gandalf only comments on tickets, so a ticket can stay open long after its problem is fixed, and an alert can sit active for days once everyone has stopped looking. The dashboard flags both:

• Stale alert: active for monitor.stale_alert_days (default 3) or longer. Shown as a ⏳ badge with its age.
• Ticket closed: the alert is still active but its ticket was closed in Tinker, so nobody is tracking it.
• Ticket left open: every alert on a ticket cleared at least monitor.stale_ticket_hours (default 24) ago, but the ticket is still Open, Pending or In Progress. Listed in the Needs Follow-up panel.

The monitor reads the list of open tickets from Tinker's api/tickets_api.php every 15 minutes (one request, read scope only) and saves it. If that list is over 2 hours old, because Tinker or the key stopped working, the ticket flags are left off rather than guessed. Stale alerts are still flagged.

WAN modem notes

The alarm and warning limits are read from the gateway on every poll, so they are never hard-coded here. Those limits are wide (warning at -27.9 dBm on a link that normally sits near -17), so fiber_rx_drift compares each reading to the link's own 7-day median to catch a damaged connector earlier. It stays quiet until rx_drift_min_samples readings exist (about a day at a 5 minute poll).

Outages can be over between polls: two real ones lasted about two minutes against a 5 minute poll. So WAN outages and modem restarts also come from the UniFi system log, and a drop in the modem's uptime records a restart even if nothing logged it. A single short outage only appears in the panel's history.

One ticket per incident. The modem and WAN reports from one check cycle open a single ticket, led by the most urgent (a P2 "WAN modem unreachable" leads, and an RPDU power-cycle seen in the same cycle is listed under "Also seen in this incident"). A report that comes while the modem has an open alert, or one that ended under 30 minutes ago, becomes a comment on that alert's ticket instead of a new ticket. If the comment can't be posted, the report gets its own ticket as before.

UniFi updates. UniFi logs NETWORK_UPDATED (Network app) and FIRMWARE_UPDATED_V2 (device firmware) once an update has finished. Updating the Network app can drop the UDM's WAN, and Smart Power may then power-cycle a modem that was fine. When Gandalf reads one of these entries it:

• comments on modem and WAN tickets first seen from 20 minutes before the update to 15 minutes after, naming the update (for example "UniFi Network 11.0.97 to 11.0.99") as the likely cause;
• opens no new modem or WAN tickets for 15 minutes after the update. A problem still there after that gets its ticket as usual; a power-cycle report during the hold only goes into the event history.

To silence modem alerts during maintenance, create a host suppression with the modem's name, or use Maintenance (below) to pause every ticket.

Baseline Tracking

Interfaces that are down on first observation (unused ports, unplugged cables) are recorded as initial_down and never alerted. Only UP→DOWN regressions generate tickets. Baseline is stored in MariaDB and survives daemon restarts.

Suppression Targets

Type	Suppresses
host	All interface alerts for a named host
interface	A specific NIC on a specific host
unifi_device	A specific UniFi device
all	Everything (global maintenance mode)

Suppressions can be manual (persist until removed) or timed (auto-expire). Expired suppressions are checked at evaluation time: no background cleanup needed.

---

Configuration (config.json)

Shared by both processes. Located in the working directory (/var/www/html/prod/).

{
  "database": {
    "host": "10.10.10.50",
    "port": 3306,
    "user": "gandalf",
    "password": "...",
    "name": "gandalf"
  },
  "prometheus": {
    "url": "http://10.10.10.48:9090"
  },
  "unifi": {
    "controller": "https://10.10.10.1",
    "api_key": "...",
    "site_id": "default"
  },
  "ticket_api": {
    "url": "https://t.lotusguild.org/create_ticket_api.php",
    "api_key": "..."
  },
  "gandalf": {
    "public_url": "https://gandalf.lotusguild.org"
  },
  "pulse": {
    "url": "http://<pulse-host>:<port>",
    "api_key": "...",
    "worker_id": "...",
    "timeout": 45
  },
  "auth": {
    "allowed_groups": ["admin"],
    "trusted_proxies": ["10.10.10.27"]
  },
  "hosts": [
    { "name": "large1",               "prometheus_instance": "10.10.10.2:9100" },
    { "name": "compute-storage-01",   "prometheus_instance": "10.10.10.4:9100" },
    { "name": "monitor-01",           "prometheus_instance": "10.10.10.8:9100" },
    { "name": "monitor-02",           "prometheus_instance": "10.10.10.9:9100" },
    { "name": "compute-storage-gpu-01", "prometheus_instance": "10.10.10.10:9100" },
    { "name": "storage-01",           "prometheus_instance": "10.10.10.11:9100" }
  ],
  "monitor": {
    "poll_interval": 120,
    "failure_threshold": 2,
    "host_failure_threshold": 5,
    "unifi_failure_threshold": 5,
    "cluster_threshold": 3,
    "ping_hosts": [
      { "name": "pbs", "ip": "10.10.10.3" }
    ]
  },
  "cutsheet": {
    "mismatch_ticket_hours": 24
  },
  "modem": {
    "url": "http://192.168.1.254",
    "name": "att-modem"
  }
}

Key Config Fields

Key	Description
database.*	MariaDB credentials (LXC 149 at 10.10.10.50)
prometheus.url	Prometheus base URL
unifi.controller	UniFi controller base URL (HTTPS, self-signed cert ignored)
unifi.api_key	UniFi API key from controller Settings → API
unifi.site_id	UniFi site ID (default: default)
ticket_api.url	Tinker Tickets create_ticket_api.php URL
ticket_api.api_key	Tinker Tickets bearer token. Needs read_write scope to add recovery notes
ticket_api.comment_url	Tinker comment endpoint (default: api/ticket_comment_api.php beside ticket_api.url)
ticket_api.read_url	Tinker ticket read endpoint, used to find tickets left open (default: api/tickets_api.php beside ticket_api.url)
gandalf.public_url	Gandalf's own address, for links in tickets (default: https://gandalf.lotusguild.org)
rack.units, rack.name	The single rack's height and name on /rack (defaults: 42, "Rack 1")
racks	Several racks: [{"name": "...", "units": 42}, ...], in tab order; overrides rack
inventory.base_url	Asset inventory (HomeBox) address; device inventory links must be on this host (default: https://inventory.lotusguild.org)
inventory.search_url	Search link used before a device has a record linked, with {base} and {q} placeholders (default: {base}/items?q={q})
pulse.url	Pulse worker API base URL (for SSH relay)
pulse.worker_id	Which Pulse worker runs ethtool collection
pulse.timeout	Max seconds to wait for SSH collection per host
auth.allowed_groups	Authelia groups that may access Gandalf
auth.trusted_proxies	Addresses or CIDR ranges of the proxy that runs Authelia forward auth (NPM). Gandalf only trusts the Remote-* identity headers on requests from these sources, since anyone else could set them. Required: with no entries, every request is unauthenticated.
hosts	Maps Prometheus instance labels → display hostnames
monitor.poll_interval	Seconds between full check cycles (default: 120)
monitor.failure_threshold	Consecutive failures before creating ticket (default: 2)
monitor.host_failure_threshold	Consecutive failures before ticketing a host that has disappeared, whether it has no metrics or doesn't answer ping (default: 5). A normal reboot takes about 3-5 minutes, so the default rides it out: the host still shows as down on the dashboard, but a ticket opens only after about 8 minutes at the default poll interval. For longer maintenance, suppress the host first.
monitor.unifi_failure_threshold	Consecutive failures before ticketing a UniFi device as offline (default: same as failure_threshold). Set higher than failure_threshold: UniFi switches/APs can take several minutes to rejoin after a firmware-update reboot, so a low threshold cuts tickets for momentary maintenance reboots instead of real failures.
monitor.cluster_threshold	Hosts with failures to trigger cluster-wide P1 (default: 3)
monitor.ping_hosts	Hosts checked only by ping (no node_exporter)
monitor.stale_alert_days	Days an alert can stay active before the dashboard flags it as stale (default: 3)
monitor.stale_ticket_hours	Hours a ticket can stay open after all its alerts cleared before it is listed under Needs Follow-up (default: 24)
cutsheet.mismatch_ticket_hours	Hours a cutsheet mismatch must last before it opens a P4 ticket (default: 24; 0 never tickets). The mismatch shows on the dashboard and the switch's page straight away, so recabling never pages anyone. Suppressing the switch or its port also stops the ticket.
modem.url	AT&T gateway base URL. Omit the modem block to disable modem monitoring. Must be reachable from the monitor host.
modem.name	Name used for the modem's alerts, tickets, and suppressions (default: att-modem)
modem.timeout	Seconds to wait per status page (default: 10)
modem.failure_threshold	Consecutive failures before ticketing a modem alert (default: same as failure_threshold). Raise it if gateway reboots or automatic firmware updates cut tickets.
modem.rx_drift_db	Rx drop below the 7-day median that raises fiber_rx_drift (default: 3.0)
modem.rx_drift_min_samples	Readings required before drift alerts start (default: 288)

---

Deployment (LXC 157)

1. Database: MariaDB LXC 149 (10.10.10.50)

CREATE DATABASE gandalf CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE USER 'gandalf'@'10.10.10.61' IDENTIFIED BY 'your_password';
GRANT ALL PRIVILEGES ON gandalf.* TO 'gandalf'@'10.10.10.61';
FLUSH PRIVILEGES;

Import schema:

mysql -h 10.10.10.50 -u gandalf -p gandalf < schema.sql

2. LXC 157: Install dependencies

pip3 install -r requirements.txt
# Ensure sshpass is available (used by deploy scripts)
apt install sshpass

3. Deploy files

# From dev machine / root/code/gandalf:
for f in app.py db.py monitor.py config.json schema.sql \
          static/style.css static/app.js \
          templates/*.html; do
  sshpass -p 'yourpass' scp -o StrictHostKeyChecking=no \
    "$f" "root@10.10.10.61:/var/www/html/prod/$f"
done
systemctl restart gandalf gandalf-monitor

4. systemd services

gandalf.service (Flask/gunicorn web app):

[Unit]
Description=Gandalf Web Dashboard
After=network.target

[Service]
Type=simple
WorkingDirectory=/var/www/html/prod
ExecStart=/usr/bin/python3 -m gunicorn --workers 1 --bind 127.0.0.1:8000 app:app
Restart=always

[Install]
WantedBy=multi-user.target

gandalf-monitor.service (background polling daemon):

[Unit]
Description=Gandalf Network Monitor Daemon
After=network.target

[Service]
Type=simple
WorkingDirectory=/var/www/html/prod
ExecStart=/usr/bin/python3 monitor.py
Restart=always

[Install]
WantedBy=multi-user.target

5. Authelia rule (LXC 167)

access_control:
  rules:
    - domain: gandalf.lotusguild.org
      policy: one_factor
      subject:
        - group:admin

systemctl restart authelia

6. NPM reverse proxy

• Domain: gandalf.lotusguild.org
• Forward to: http://10.10.10.61:80 (the local nginx, which forwards to gunicorn on 127.0.0.1:8000)
• Forward Auth: Authelia at http://10.10.10.167:9091
• WebSockets: Not required

---

Service Management

# Status
systemctl status gandalf gandalf-monitor

# Logs (live)
journalctl -u gandalf -f
journalctl -u gandalf-monitor -f

# Restart after code or config changes
systemctl restart gandalf gandalf-monitor

---

Troubleshooting

Avatar photo missing (/api/avatar returns 404)

journalctl -u gandalf | grep -i avatar gives the reason: no ldap.host / ldap.bind_dn (missing config), ldap3 not installed (run pip3 install -r requirements.txt), LDAP avatar lookup failed (bind or network), LDAP has no photo (or Remote-User doesn't match a uid), or a cached no photo marker, which expires after ldap.cache_ttl seconds or can be removed from ldap.cache_dir.

Monitor not creating tickets

• Verify config.json → ticket_api.api_key is set and valid
• Check journalctl -u gandalf-monitor for Ticket creation failed lines
• Confirm the Tinker Tickets API is reachable from LXC 157

WAN / Fiber panel missing or showing "unavailable"

• The panel stays hidden until config.json → modem.url is set and the monitor has completed a cycle
• Test from the monitor host: curl -s -o /dev/null -w '%{http_code}\n' http://192.168.1.254/cgi-bin/fiberstat.ha should print 200
• modem_data_invalid means the gateway answered but its pages couldn't be parsed. AT&T updates firmware automatically, so compare the page against the parsers in modem.py (markup confirmed on BGW320-505 firmware 6.35.8)

Device pages show no interface data

• Check gandalf-monitor.service is running and has completed at least one cycle
• Check journalctl -u gandalf-monitor for Prometheus or UniFi errors
• Verify Prometheus is reachable: curl http://10.10.10.48:9090/api/v1/query?query=up

Device page: SFP / optical panel missing

• SFP data requires Pulse worker + SSH access to hosts
• Verify config.json → pulse.* is configured and the Pulse worker is running
• Confirm sshpass + SSH access from the Pulse worker to each Proxmox host
• Only interfaces with physical SFP modules return DOM data (ethtool -m)

Port inspector: path check or Run Diagnostics not appearing

• These need the port's neighbor to be a host with link stats. A port showing only a MAC couldn't be matched
• Check that the server's node_exporter is scraped and reports node_network_info for that NIC
• Hosts missing from config.json are named by IP, so their ports may not match what you expect

Port inspector: front panel shows as a flat list (no layout)

• The switch's model field from UniFi doesn't match any key in FACEPLATES in faceplates.js
• The model is shown on the device page
• Add the model to FACEPLATES in faceplates.js (see Adding a New Device Faceplate)

Baseline re-initializing on every restart

• interface_baseline is stored in the monitor_state DB table; survives restarts
• If it appears to reset: check DB connectivity from the monitor daemon

Interface stuck at "initial_down" forever

• This means the interface was down when the monitor first saw it
• It will begin tracking once it comes up; or manually clear it:

  -- In MariaDB on 10.10.10.50:
  UPDATE monitor_state SET value='{}' WHERE key_name='interface_baseline';
  

  Then restart the monitor: systemctl restart gandalf-monitor

Prometheus data missing for a host

# On the affected host:
systemctl status prometheus-node-exporter
# Verify it's scraped:
curl http://10.10.10.48:9090/api/v1/query?query=up | jq '.data.result[] | select(.metric.job=="node")'

---

Development Notes

File Layout

gandalf/
├── app.py              # Flask web app (routes, auth, API endpoints)
├── monitor.py          # Background daemon (Prometheus, UniFi, Pulse, alert logic)
├── modem.py            # Read-only AT&T gateway status page client and parser
├── devices.py          # Device inventory for search and device pages
├── topology.py         # The topology graph: devices, cables, depth, last-seen links
├── cutsheet.py         # Cutsheet validation, path tracing, plan-vs-actual checks, CSV
├── ticket_text.py      # Markdown for tickets and recovery notes
├── db.py               # Database operations (MariaDB via pymysql, thread-local conn reuse)
├── schema.sql          # Database schema (network_events, suppression_rules, monitor_state)
├── config.json         # Runtime configuration (not committed with secrets)
├── requirements.txt    # Python dependencies
├── static/
│   ├── style.css       # Terminal aesthetic CSS (CRT scanlines, green-on-black)
│   ├── app.js          # Dashboard JS (auto-refresh, events, suppress modal, search, modem panel)
│   ├── device.js       # Device page rendering
│   ├── cutsheet.js     # Cutsheet page
│   ├── events.js       # Alert history page
│   ├── rack.js         # Rack drawing
│   ├── topology.js     # Topology page: layout, cables, pan and zoom
│   ├── faceplates.js   # UniFi faceplates drawn from each model's real layout
│   ├── fronts.js       # Drawn fronts for gear without a live faceplate (NVR, server cases)
│   └── port-inspector.js # Switch front panel, port panel, path check, diagnostics
└── templates/
    ├── base.html       # Shared layout (header, nav, footer)
    ├── index.html      # Dashboard page
    ├── device.html     # Per-device page
    ├── cutsheet.html   # Cutsheet page and its editors
    ├── events.html     # Alert history page
    ├── rack.html       # Rack drawing
    ├── topology.html   # Topology page
    ├── _modem_panel.html # WAN / Fiber panel, shared by the dashboard and modem page
    └── suppressions.html # Suppression management page

Adding a New Monitored Host

1. Install prometheus-node-exporter on the host
2. Add a scrape target to Prometheus config
3. Add an entry to config.json → hosts:

   { "name": "newhost", "prometheus_instance": "10.10.10.X:9100" }
   

4. Restart monitor: systemctl restart gandalf-monitor
5. For SFP DOM / ethtool: ensure the host is SSH-accessible from the Pulse worker

Faceplates included

UniFi gear drawn with live ports: UDM Pro (UDMPRO), USW Pro 24 PoE (US24PRO), USW Aggregation (USL8A), USW Flex (USF5P), USW Flex Mini (USMINI), USP PDU Pro (USPPDUP), UPS 2U Pro (USPDA2B), USW Pro Max 24 PoE (USPM24P, also USPM24) and USW Pro Max 48 PoE (USPM48P, also USPM48), plus the UniFi NVR Pro (UNVRPRO), which UniFi Network doesn't report and so is picked by Front drawing. The Pro Max switches also have Front drawing options, so a new switch draws correctly on the rack before it is adopted. If UniFi reports a Pro Max under a different model code, add it as another alias at the end of static/faceplates.js.

Adding a New Device Faceplate

Find the UniFi model code on the device page, then add an entry to FACEPLATES in static/faceplates.js. Trace coordinates from a front-on image of the device, using the image's pixels as the viewBox:

'MYNEWMODEL': {
  width: 1100, height: 110, frameClass: 'fp-scroll', scrollX: 500,
  label: 'My New Switch',
  legend: [['off', 'no link'], ['amber', '10/100M'], ['green', '1G']],
  roles: { 1: 'RJ45' },
  ports: rj45Pairs([520, 557, 594], { top: 38, bottom: 69, topLabel: 15.5, bottomLabel: 98 }),
  decor() { return fpVents([[90, 500]], 3) + fpQuadScreen(15, 23, 60); },
},

Take LED colors from the model's quick start guide. See the comment at the top of the file for every option.

Database Schema Notes

• cutsheet_items gained rack, shelf, device, mount, chassis, facing and slot columns; they are added automatically on first use
• cutsheet_racks: racks saved on the rack page (name, height, notes)
• unused_interfaces: host interfaces marked unused (no alerts, not counted in host status)
• device_links: each device's inventory URL, by device name; created with the cutsheet tables
• network_events: one row per active event; resolved_at is set when recovered, and recovery_notified once the recovery note is on its ticket (column added automatically at monitor startup)
• suppression_rules: active=FALSE when removed; expires_at checked at query time
• monitor_state: key/value store; interface_baseline, link_stats, and modem_stats are JSON blobs
• monitor_state also holds wan_history (WAN outages, modem restarts) and unifi_log_cursor (how far the UniFi log has been read)
• Database sessions run in UTC (db.py), whatever the server's time zone
• modem_samples: optical readings per poll, kept 90 days; created automatically at monitor startup
• cutsheet_items, cutsheet_cables, cutsheet_port_labels, cutsheet_history: cutsheet records and their change log, created automatically on first use. Cables refer to devices and items by name; renaming an item updates its cables, but a device renamed in UniFi needs its cables edited to match

Security Notes

• XSS prevention: all user-controlled data in dynamically generated HTML uses escHtml() (JS) or Jinja2 auto-escaping (Python). Suppress buttons use data-* attributes + a single delegated click listener rather than inline onclick with interpolated strings.
• Interface name validation: monitor.py validates SSH interface names against ^[a-zA-Z0-9_.@-]+$ before use, and additionally wraps them with shlex.quote() for defense-in-depth.
• DB parameters: all SQL uses parameterised queries via pymysql: no string concatenation into SQL.
• Auth: Authelia enforces admin-only access at the NPM/LXC 167 layer; the Flask app additionally checks the Remote-User and Remote-Groups headers via @require_auth. Those headers are only real when NPM sent the request, because its forward auth overwrites them, so a LAN host connecting to the container directly could otherwise claim any identity. Two layers stop that. The container's nginx site (tracked in pve-infra) blanks them unless the request came from NPM. The app also ignores them unless the source address (nginx's X-Real-IP, or the peer address when gunicorn is reached directly) is in auth.trusted_proxies, and it fails closed when that is unset.

Known Limitations

• Single gunicorn worker (--workers 1): required because db.py uses thread-local connection reuse (one connection per thread). Multiple workers would each have their own connection, which is fine, but the thread-local optimisation only helps within one worker.
• No CSRF tokens on API endpoints: mitigated by Authelia session cookies being SameSite=Strict and the site being admin-only.
• SSH collection via Pulse is synchronous: if Pulse is slow, the entire monitor cycle is delayed. The pulse.timeout config controls the max wait.
• UniFi LLDP data is only as fresh as the last monitor poll (120s default).

---

CI / CD

Workflow	Purpose	Triggers
lint.yml (python-lint)	flake8 on all .py files	Every push and PR
lint.yml (js-lint)	ESLint on static/	Every push and PR
test.yml	pytest: diagnose.py, modem.py, and monitor.py alert logic	Every push and PR
security.yml	bandit -ll (medium+ severity)	Every push, PR, and weekly Monday 6am
deploy job in lint.yml	Calls the gandalf-deploy webhook on CT157 (10.10.10.61)	Push to main only, after both lint jobs pass

Branch protection is enabled on main: both lint jobs must pass before any PR can merge.

Tests live in tests/:

• test_diagnose.py: DiagnosticsRunner static methods (build_ssh_command, parse_output, parse_sysfs_stats, parse_ethtool, and variants)
• test_modem.py: gateway page parsing, optical status, and WAN state
• test_monitor.py: alert logic for dead hosts, Prometheus and Pulse outages, and modem and WAN log events
• test_devices.py: device inventory behind search and device pages
• test_app.py: page and API routes