diff --git a/README.md b/README.md index 55b3c3b..b30b447 100644 --- a/README.md +++ b/README.md @@ -27,17 +27,7 @@ A Proxmox/Ceph cluster lives or dies by its network, and network problems are sn ![Host cards](docs/img/host-cards.png) -**Link debug**: per-interface speed, duplex, carrier flaps, error and drop rates, and **SFP optical readings** (temperature, TX/RX power) with colour-coded thresholds. Here a fibre with weak receive power is flagged while its neighbour is healthy. - -![Link debug](docs/img/link-debug.png) - -| Weak receive power on a fibre link | Healthy fibre link | -|---|---| -| ![SFP warning](docs/img/link-debug-sfp-warning.png) | ![SFP healthy](docs/img/link-debug-sfp.png) | - -Switch ports are shown too, with PoE draw and the LLDP neighbour on each port: - -![Switch ports](docs/img/link-debug-switch.png) +**Device pages**: search by hostname, IP, MAC, model or interface from the dashboard, then open one page per host, switch, router, PDU or modem. Each interface or port is one row with its state, speed, traffic and neighbour. Expanding a row shows duplex, carrier flaps, error and drop rates, PoE, LLDP, and **SFP optical readings** (temperature, TX/RX power) with colour-coded thresholds. **Network inspector**: chassis diagrams for each UniFi switch. Click a port for its stats, LLDP neighbour and a side-by-side switch/server path check. @@ -83,6 +73,7 @@ Two processes share a MariaDB database: [UniFi Controller] ──▶ monitor.py ──▶ MariaDB ◀── app.py ──▶ nginx ──▶ Authelia ──▶ Browser [Pulse Worker] ──▶ [SSH / ethtool] ──▶ +[AT&T BGW320] ──▶ ``` ### Data Sources @@ -93,6 +84,7 @@ Two processes share a MariaDB database: | **UniFi API** (`https://10.10.10.1`) | Switch port stats, device status, LLDP neighbor table, PoE data | | **Pulse Worker** | SSH relay — runs `ethtool` + SFP DOM queries on each Proxmox host | | **Ping** | Reachability for hosts without `node_exporter` (e.g. PBS) | +| **AT&T BGW320** (`http://192.168.1.254`) | Fiber light levels, PON and WAN state, WAN counters, scraped read-only from the gateway's status pages (no login) | ### Monitored Hosts (Prometheus / node_exporter) @@ -112,31 +104,35 @@ Ping-only (no node_exporter): **pbs** (10.10.10.3) ## Pages ### Dashboard (`/`) +- Device search (see Device pages below) +- Active alerts first, then the WAN / Fiber panel, hosts, UniFi devices, topology, and recently resolved alerts - Real-time host status grid with per-NIC link state (UP / DOWN / degraded) -- Network topology diagram (Internet → Gateway → Switches → Hosts) +- Network topology diagram (Internet → Modem → Gateway → Switches → Hosts) +- WAN / Fiber panel: WAN and PON state, Rx/Tx power, optics temperature, throughput, errors, gateway uptime and last restart, 7-day Rx trend, recent WAN outages - UniFi device table (switches, APs, gateway) - Active alerts table with severity, target, consecutive failures, ticket link - Quick-suppress modal: apply timed or manual suppression from any alert row - Auto-refreshes every 30 seconds via `/api/status` + `/api/network` -### Link Debug (`/links`) -Per-interface statistics collected every poll cycle. All panels are collapsible -(click header or use Collapse All / Expand All). Collapse state persists across -page refreshes via `sessionStorage`. +### Device pages (`/device/`) +One page per host, switch, router, AP, PDU, or modem. Reach them from the search box on the +dashboard (`/` focuses it), or from any device name. Search matches hostname, IP, MAC, model, type, +interface and port names, and LLDP neighbors. -**Server NICs** (via Prometheus + SSH/ethtool): -- Speed, duplex, auto-negotiation, link detected -- TX/RX rate bars (bandwidth utilisation % of link capacity) -- TX/RX error and drop rates per second -- Carrier changes (cumulative since boot — watch for flapping) -- **SFP / Optical panel** (when SFP module present): vendor/PN, temp, voltage, - bias current, TX power (dBm), RX power (dBm), RX−TX delta, per-stat bars +- **Overview**: status, IP, MAC, model, firmware, uptime, CPU, memory, clients, temperatures, power draw +- **Alerts**: active and past alerts for this device +- **WAN** (routers): connection state, speed, latency, availability +- **Connected to** (hosts): switch ports that see this host over LLDP +- **Outlets** (PDUs): per-outlet state, power, and auto power-cycle setting +- **Interfaces / Ports**: one row each with state, speed, RX/TX rate, and neighbor, filterable. + Expanding a row shows duplex, auto-negotiation, carrier changes, error and drop rates, + traffic bars, PoE, LLDP detail, and the **SFP / optical panel** (vendor/PN, temperature, voltage, + bias, TX/RX power, RX−TX delta) +- **WAN / Fiber** (modem): the same panel as the dashboard -**UniFi Switch Ports** (via UniFi API): -- Port number badge (`#N`), UPLINK badge, PoE draw badge -- LLDP neighbor line: `→ system_name (port_id)` when neighbor is detected -- PoE class and max wattage line -- Speed, duplex, auto-neg, TX/RX rates, errors, drops +These replaced the Link Debug page; `/links` now redirects to the dashboard. Hosts that Prometheus +sees but `config.json` doesn't list still get a page, with their state shown as unknown because +they aren't alerted on. ### Inspector (`/inspector`) Visual switch chassis diagrams. Each switch is rendered model-accurately using @@ -174,6 +170,7 @@ layout config in the template (`SWITCH_LAYOUTS`). | `USL8A` | UniFi Switch Lite 8 PoE | 8×SFP (2 rows of 4) | | `US24PRO` | UniFi Switch Pro 24 | 24×RJ45 staggered + 2×SFP | | `USPPDUP` | Custom/other | Single-port fallback | +| `UDMPRO` | Dream Machine Pro | 9×RJ45 + 2×SFP+ | | `USMINI` | UniFi Switch Mini | 5-port row | Add new layouts by adding a key to `SWITCH_LAYOUTS` matching the `model` field @@ -197,7 +194,32 @@ returned by the UniFi API for that device. | UniFi device offline (≥`unifi_failure_threshold` consecutive checks) | P2 High | | Proxmox host NIC link-down regression (≥2 consecutive checks) | P2 High | | Host unreachable via ping (≥2 consecutive checks) | P2 High | -| ≥3 hosts simultaneously reporting interface failures | P1 Critical | +| Configured host has no metrics in Prometheus (host or `node_exporter` down, ≥2 consecutive checks) | P2 High | +| Prometheus unavailable, so host and interface alerting is paused (≥2 consecutive checks) | P2 High | +| ≥3 hosts simultaneously reporting interface or metrics failures | P1 Critical | +| WAN down: broadband or optical status not Up, or PON not in O5 (≥`modem.failure_threshold`) | P1 Critical | +| Modem not responding (≥`modem.failure_threshold`) | P2 High | +| Fiber Rx/Tx power or optics temperature in the module's alarm range, or a supply voltage or Tx bias alarm flag (≥`modem.failure_threshold`) | P2 High | +| Fiber reading in the module's warning range (≥`modem.failure_threshold`) | P3 | +| Rx power `rx_drift_db` or more below its 7-day median (≥`modem.failure_threshold`) | P3 | +| Modem pages unreadable, e.g. after a firmware update (≥`modem.failure_threshold`) | P3 | +| UniFi logs that Smart Power auto power-cycled the modem | P3 | +| UniFi logs that the WAN failed multiple times in 24 hours (at most one ticket a day) | P2 | + +A failed Pulse ping check counts as "unknown": it neither raises nor resolves `host_unreachable`. + +### WAN modem notes + +The alarm and warning limits are read from the gateway on every poll, so they are never hard-coded here. +Those limits are wide (warning at -27.9 dBm on a link that normally sits near -17), so `fiber_rx_drift` +compares each reading to the link's own 7-day median to catch a damaged connector earlier. It stays quiet +until `rx_drift_min_samples` readings exist (about a day at a 5 minute poll). + +Outages can be over between polls: two real ones lasted about two minutes against a 5 minute poll. +So WAN outages and modem restarts also come from the UniFi system log, and a drop in the modem's uptime +records a restart even if nothing logged it. A single short outage only appears in the panel's history. + +To silence modem alerts during maintenance, create a `host` suppression with the modem's `name`. ### Baseline Tracking @@ -269,6 +291,10 @@ Shared by both processes. Located in the working directory (`/var/www/html/prod/ "ping_hosts": [ { "name": "pbs", "ip": "10.10.10.3" } ] + }, + "modem": { + "url": "http://192.168.1.254", + "name": "att-modem" } } ``` @@ -293,6 +319,12 @@ Shared by both processes. Located in the working directory (`/var/www/html/prod/ | `monitor.unifi_failure_threshold` | Consecutive failures before ticketing a UniFi device as offline (default: same as `failure_threshold`). Set higher than `failure_threshold` — UniFi switches/APs can take several minutes to rejoin after a firmware-update reboot, so a low threshold cuts tickets for momentary maintenance reboots instead of real failures. | | `monitor.cluster_threshold` | Hosts with failures to trigger cluster-wide P1 (default: 3) | | `monitor.ping_hosts` | Hosts checked only by ping (no node_exporter) | +| `modem.url` | AT&T gateway base URL. Omit the `modem` block to disable modem monitoring. Must be reachable from the monitor host. | +| `modem.name` | Name used for the modem's alerts, tickets, and suppressions (default: `att-modem`) | +| `modem.timeout` | Seconds to wait per status page (default: 10) | +| `modem.failure_threshold` | Consecutive failures before ticketing a modem alert (default: same as `failure_threshold`). Raise it if gateway reboots or automatic firmware updates cut tickets. | +| `modem.rx_drift_db` | Rx drop below the 7-day median that raises `fiber_rx_drift` (default: 3.0) | +| `modem.rx_drift_min_samples` | Readings required before drift alerts start (default: 288) | --- @@ -409,17 +441,28 @@ systemctl restart gandalf gandalf-monitor ## Troubleshooting +### Avatar photo missing (`/api/avatar` returns 404) +`journalctl -u gandalf | grep -i avatar` gives the reason: `no ldap.host / ldap.bind_dn` (missing config), +`ldap3 not installed` (run `pip3 install -r requirements.txt`), `LDAP avatar lookup failed` (bind or network), +`LDAP has no photo` (or `Remote-User` doesn't match a uid), or a cached `no photo` marker, which expires +after `ldap.cache_ttl` seconds or can be removed from `ldap.cache_dir`. + ### Monitor not creating tickets - Verify `config.json → ticket_api.api_key` is set and valid - Check `journalctl -u gandalf-monitor` for `Ticket creation failed` lines - Confirm the Tinker Tickets API is reachable from LXC 157 -### Link Debug shows no data / "Loading…" forever +### WAN / Fiber panel missing or showing "unavailable" +- The panel stays hidden until `config.json → modem.url` is set and the monitor has completed a cycle +- Test from the monitor host: `curl -s -o /dev/null -w '%{http_code}\n' http://192.168.1.254/cgi-bin/fiberstat.ha` should print `200` +- `modem_data_invalid` means the gateway answered but its pages couldn't be parsed. AT&T updates firmware automatically, so compare the page against the parsers in `modem.py` (markup confirmed on BGW320-505 firmware 6.35.8) + +### Device pages show no interface data - Check `gandalf-monitor.service` is running and has completed at least one cycle - Check `journalctl -u gandalf-monitor` for Prometheus or UniFi errors - Verify Prometheus is reachable: `curl http://10.10.10.48:9090/api/v1/query?query=up` -### Link Debug: SFP DOM panel missing +### Device page: SFP / optical panel missing - SFP data requires Pulse worker + SSH access to hosts - Verify `config.json → pulse.*` is configured and the Pulse worker is running - Confirm `sshpass` + SSH access from the Pulse worker to each Proxmox host @@ -467,17 +510,21 @@ curl http://10.10.10.48:9090/api/v1/query?query=up | jq '.data.result[] | select gandalf/ ├── app.py # Flask web app (routes, auth, API endpoints) ├── monitor.py # Background daemon (Prometheus, UniFi, Pulse, alert logic) +├── modem.py # Read-only AT&T gateway status page client and parser +├── devices.py # Device inventory for search and device pages ├── db.py # Database operations (MariaDB via pymysql, thread-local conn reuse) ├── schema.sql # Database schema (network_events, suppression_rules, monitor_state) ├── config.json # Runtime configuration (not committed with secrets) ├── requirements.txt # Python dependencies ├── static/ │ ├── style.css # Terminal aesthetic CSS (CRT scanlines, green-on-black) -│ └── app.js # Dashboard JS (auto-refresh, host grid, events, suppress modal) +│ ├── app.js # Dashboard JS (auto-refresh, events, suppress modal, search, modem panel) +│ └── device.js # Device page rendering └── templates/ ├── base.html # Shared layout (header, nav, footer) ├── index.html # Dashboard page - ├── links.html # Link Debug page (server NICs + UniFi switch ports) + ├── device.html # Per-device page + ├── _modem_panel.html # WAN / Fiber panel, shared by the dashboard and modem page ├── inspector.html # Visual switch inspector + LLDP path debug └── suppressions.html # Suppression management page ``` @@ -511,7 +558,10 @@ under `unifi_switches..model`), then add to `SWITCH_LAYOUTS` in - `network_events`: one row per active event; `resolved_at` is set when recovered - `suppression_rules`: `active=FALSE` when removed; `expires_at` checked at query time -- `monitor_state`: key/value store; `interface_baseline` and `link_stats` are JSON blobs +- `monitor_state`: key/value store; `interface_baseline`, `link_stats`, and `modem_stats` are JSON blobs +- `monitor_state` also holds `wan_history` (WAN outages, modem restarts) and `unifi_log_cursor` (how far the UniFi log has been read) +- Database sessions run in UTC (`db.py`), whatever the server's time zone +- `modem_samples`: optical readings per poll, kept 90 days; created automatically at monitor startup ### Security Notes @@ -547,11 +597,16 @@ under `unifi_switches..model`), then add to `SWITCH_LAYOUTS` in |---|---|---| | `lint.yml` (python-lint) | flake8 on all `.py` files | Every push and PR | | `lint.yml` (js-lint) | ESLint on `static/` | Every push and PR | -| `test.yml` | pytest — 33 tests for `diagnose.py` static methods | Every push and PR | +| `test.yml` | pytest — `diagnose.py`, `modem.py`, and `monitor.py` alert logic | Every push and PR | | `security.yml` | bandit `-ll` (medium+ severity) | Every push, PR, and weekly Monday 6am | | `deploy` job in `lint.yml` | Calls the `gandalf-deploy` webhook on CT157 (10.10.10.61) | Push to `main` only, after both lint jobs pass | Branch protection is enabled on `main` — both lint jobs must pass before any PR can merge. -Tests live in `tests/test_diagnose.py` and cover `DiagnosticsRunner` static methods: -`build_ssh_command`, `parse_output`, `parse_sysfs_stats`, `parse_ethtool`, and variants. +Tests live in `tests/`: +- `test_diagnose.py`: `DiagnosticsRunner` static methods (`build_ssh_command`, `parse_output`, + `parse_sysfs_stats`, `parse_ethtool`, and variants) +- `test_modem.py`: gateway page parsing, optical status, and WAN state +- `test_monitor.py`: alert logic for dead hosts, Prometheus and Pulse outages, and modem and WAN log events +- `test_devices.py`: device inventory behind search and device pages +- `test_app.py`: page and API routes diff --git a/app.py b/app.py index d208004..8d452ae 100644 --- a/app.py +++ b/app.py @@ -18,9 +18,10 @@ import uuid from datetime import datetime, timezone from functools import wraps -from flask import Flask, jsonify, make_response, render_template, request, send_file +from flask import Flask, jsonify, make_response, redirect, render_template, request, send_file, url_for import db +import devices import diagnose from monitor import PulseClient @@ -196,6 +197,18 @@ def _annotate_suppressions(events: list, suppressions: list) -> None: ev['is_suppressed'] = db.check_suppressed(suppressions, 'host', name, detail) +def _state_json(key: str) -> dict: + """Return a monitor_state JSON blob as a dict, or {} if missing or unreadable.""" + raw = db.get_state(key) + if not raw: + return {} + try: + return json.loads(raw) + except Exception as e: + logger.error(f'Failed to parse {key} JSON: {e}') + return {} + + # --------------------------------------------------------------------------- # Page routes # --------------------------------------------------------------------------- @@ -235,8 +248,14 @@ def index(): @app.route('/links') @require_auth def links_page(): - user = _get_user() - return render_template('links.html', user=user) + """Link Debug moved to the per-device pages; keep old bookmarks working.""" + return redirect(url_for('index')) + + +@app.route('/device/') +@require_auth +def device_page(name: str): + return render_template('device.html', user=_get_user(), device_name=name) @app.route('/inspector') @@ -315,6 +334,60 @@ def api_links(): return jsonify({'hosts': {}, 'unifi_switches': {}, 'updated': None}) +@app.route('/api/modem') +@require_auth +def api_modem(): + """Latest WAN modem status plus optical history for the trend chart.""" + try: + hours = max(1, min(int(request.args.get('hours', 168)), 2160)) + except ValueError: + return jsonify({'error': 'hours must be an integer'}), 400 + + raw = db.get_state('modem_stats') + if not raw: + return jsonify({'configured': False}) + try: + data = json.loads(raw) + except Exception as e: + logger.error(f'Failed to parse modem_stats JSON: {e}') + return jsonify({'configured': False}) + + # A history failure shouldn't take the current readings down with it. + try: + # Keep roughly 200 points regardless of range; never finer than the 5 min poll. + data['history'] = db.get_modem_history(hours, max(300, hours * 3600 // 200)) + except Exception as e: + logger.error(f'Failed to load modem history: {e}') + data['history'] = [] + data['wan_history'] = _state_json('wan_history') + data['configured'] = True + return jsonify(data) + + +@app.route('/api/devices') +@require_auth +def api_devices(): + """Device list for the home page search.""" + return jsonify({'devices': devices.build_index( + _state_json('network_snapshot'), _state_json('link_stats'), _state_json('modem_stats'), + )}) + + +@app.route('/api/device/') +@require_auth +def api_device(name: str): + """Everything known about one device, for its detail page.""" + detail = devices.device_detail( + name, _state_json('network_snapshot'), _state_json('link_stats'), _state_json('modem_stats'), + ) + if detail is None: + return jsonify({'error': f'No device named "{name}"'}), 404 + events = db.get_events_for_target(name) + _annotate_suppressions(events, db.get_active_suppressions()) + detail['events'] = events + return jsonify(detail) + + @app.route('/api/events') @require_auth def api_events(): @@ -559,6 +632,7 @@ def api_avatar(): ldap_cfg = _config().get('ldap', {}) if not ldap_cfg.get('host') or not ldap_cfg.get('bind_dn'): + logger.warning('Avatar unavailable: config.json has no ldap.host / ldap.bind_dn') return '', 404 # Build a safe cache filename from the username (alphanumeric + - _ .) @@ -589,6 +663,7 @@ def api_avatar(): # Skip LDAP if we already know this user has no avatar try: if os.path.exists(sentinel) and now - os.path.getmtime(sentinel) < cache_ttl: + logger.info(f'Avatar for {username!r}: cached "no photo" marker is still valid') return '', 404 except OSError: pass @@ -627,6 +702,7 @@ def api_avatar(): pass if not avatar_data or len(avatar_data) < 100: + logger.warning(f'Avatar for {username!r}: LDAP has no photo (or no user with that uid)') with open(sentinel, 'w'): pass return '', 404 diff --git a/db.py b/db.py index 08001ce..1e1a461 100644 --- a/db.py +++ b/db.py @@ -34,6 +34,9 @@ def _new_conn(cfg: dict): cursorclass=pymysql.cursors.DictCursor, connect_timeout=10, charset='utf8mb4', + # All app code treats database times as UTC. Without this, a server in another + # time zone shifts displayed ages and timed suppression expiries by its offset. + init_command="SET time_zone='+00:00'", ) @@ -145,6 +148,24 @@ def upsert_event( return cur.lastrowid, True, 1, None +def get_events_for_target(target_name: str, limit: int = 25) -> list: + """Return a device's events, unresolved first, then most recent.""" + with get_conn() as conn: + with conn.cursor() as cur: + cur.execute( + """SELECT * FROM network_events WHERE target_name=%s + ORDER BY resolved_at IS NOT NULL, COALESCE(resolved_at, last_seen) DESC + LIMIT %s""", + (target_name, limit), + ) + rows = cur.fetchall() + for r in rows: + for k in ('first_seen', 'last_seen', 'resolved_at'): + if r.get(k) and hasattr(r[k], 'isoformat'): + r[k] = r[k].isoformat() + 'Z' + return rows + + def resolve_event(event_type: str, target_name: str, target_detail: str = '') -> None: detail = target_detail or '' with get_conn() as conn: @@ -332,6 +353,80 @@ def purge_old_resolved_events(days: int = 90) -> int: return n +# --------------------------------------------------------------------------- +# WAN modem optical history +# --------------------------------------------------------------------------- + +def ensure_modem_schema() -> None: + """Create the modem history table so deploys need no manual migration.""" + with get_conn() as conn: + with conn.cursor() as cur: + cur.execute( + """CREATE TABLE IF NOT EXISTS modem_samples ( + sampled_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP PRIMARY KEY, + rx_power_dbm DECIMAL(5,1) NULL, + tx_power_dbm DECIMAL(5,1) NULL, + temp_c DECIMAL(5,1) NULL + ) ENGINE=InnoDB""" + ) + + +def record_modem_sample(rx_power_dbm, tx_power_dbm, temp_c) -> None: + with get_conn() as conn: + with conn.cursor() as cur: + cur.execute( + """INSERT INTO modem_samples (sampled_at, rx_power_dbm, tx_power_dbm, temp_c) + VALUES (NOW(), %s, %s, %s) + ON DUPLICATE KEY UPDATE rx_power_dbm=VALUES(rx_power_dbm), + tx_power_dbm=VALUES(tx_power_dbm), temp_c=VALUES(temp_c)""", + (rx_power_dbm, tx_power_dbm, temp_c), + ) + + +def get_modem_rx_values(days: int = 7) -> list: + """Return recent Rx power readings (dBm) for computing a drift baseline.""" + with get_conn() as conn: + with conn.cursor() as cur: + cur.execute( + """SELECT rx_power_dbm FROM modem_samples + WHERE rx_power_dbm IS NOT NULL + AND sampled_at > DATE_SUB(NOW(), INTERVAL %s DAY)""", + (days,), + ) + return [float(r['rx_power_dbm']) for r in cur.fetchall()] + + +def get_modem_history(hours: int = 168, bucket_seconds: int = 3600) -> list: + """Return optical history averaged into time buckets, oldest first.""" + with get_conn() as conn: + with conn.cursor() as cur: + cur.execute( + """SELECT FLOOR(UNIX_TIMESTAMP(sampled_at) / %s) * %s AS ts, + ROUND(AVG(rx_power_dbm), 1) AS rx_avg, + MIN(rx_power_dbm) AS rx_min, + ROUND(AVG(tx_power_dbm), 1) AS tx_avg, + ROUND(AVG(temp_c), 1) AS temp_avg + FROM modem_samples + WHERE sampled_at > DATE_SUB(NOW(), INTERVAL %s HOUR) + GROUP BY ts ORDER BY ts""", + (bucket_seconds, bucket_seconds, hours), + ) + return [ + {k: (float(v) if v is not None else None) for k, v in row.items()} + for row in cur.fetchall() + ] + + +def purge_old_modem_samples(days: int = 90) -> int: + with get_conn() as conn: + with conn.cursor() as cur: + cur.execute( + 'DELETE FROM modem_samples WHERE sampled_at < DATE_SUB(NOW(), INTERVAL %s DAY)', + (days,), + ) + return cur.rowcount + + def check_suppressed(suppressions: list, target_type: str, target_name: str, target_detail: str = '') -> bool: """Check suppression against a pre-loaded list (avoids per-call DB queries).""" for s in suppressions: diff --git a/devices.py b/devices.py new file mode 100644 index 0000000..6502c1c --- /dev/null +++ b/devices.py @@ -0,0 +1,153 @@ +"""Device inventory built from the monitor's cached state. + +Pure functions with no I/O: app.py passes in the JSON blobs the monitor stored, +so search and the per-device pages always agree about what a device is. +""" +import re +from typing import Dict, List, Optional + +# UniFi reports PDUs as switches or APs, so model codes take priority over type. +_PDU_MODELS = {'USPPDUP', 'UP6', 'USPDA2B'} +_KIND_BY_TYPE = { + 'usw': 'switch', 'udm': 'router', 'ugw': 'router', 'uxg': 'router', + 'uap': 'ap', 'usp': 'pdu', +} + + +def unifi_kind(unifi_type: str, model: str) -> str: + if model in _PDU_MODELS: + return 'pdu' + return _KIND_BY_TYPE.get((unifi_type or '').lower(), 'device') + + +def _mac_forms(mac: str) -> List[str]: + """Return a MAC with and without separators, so either spelling is searchable.""" + return [mac, re.sub(r'[^0-9a-fA-F]', '', mac)] if mac else [] + + +def _search_text(*parts) -> str: + flat = [] + for p in parts: + if isinstance(p, (list, tuple, set)): + flat.extend(str(x) for x in p if x) + elif p: + flat.append(str(p)) + return ' '.join(flat).lower() + + +def _port_neighbors(ports: dict) -> List[str]: + return [p['lldp']['system_name'] for p in ports.values() + if (p.get('lldp') or {}).get('system_name')] + + +def build_index(snapshot: dict, link_stats: dict, modem: Optional[dict]) -> List[dict]: + """Return every known device as {name, kind, status, ip, model, ..., search}.""" + snapshot = snapshot or {} + link_stats = link_stats or {} + devices: Dict[str, dict] = {} + + for name, host in (snapshot.get('hosts') or {}).items(): + link_ifaces = (link_stats.get('hosts') or {}).get(name) or {} + ifaces = sorted(set(host.get('interfaces') or {}) | set(link_ifaces)) + devices[name] = { + 'name': name, 'kind': 'host', 'status': host.get('status', 'unknown'), + 'ip': host.get('ip', ''), 'mac': '', 'model': '', 'source': host.get('source', ''), + 'interface_count': len(ifaces), + 'search': _search_text(name, host.get('ip'), 'host server', ifaces), + } + + # Prometheus also sees hosts that aren't in config.json, e.g. LXC app servers. + # They have link stats but no alerting, so their state is reported as unknown. + for name, ifaces in (link_stats.get('hosts') or {}).items(): + if name not in devices: + ip = next((i.get('host_ip') for i in ifaces.values() if i.get('host_ip')), '') + devices[name] = { + 'name': name, 'kind': 'host', 'status': 'unknown', 'ip': ip, 'mac': '', 'model': '', + 'source': 'prometheus', 'interface_count': len(ifaces), + 'search': _search_text(name, ip, 'host server', list(ifaces)), + } + + switches = link_stats.get('unifi_switches') or {} + for d in snapshot.get('unifi') or []: + name = d.get('name', '') + ports = (switches.get(name) or {}).get('ports') or {} + kind = unifi_kind(d.get('type', ''), d.get('model', '')) + devices[name] = { + 'name': name, 'kind': kind, 'status': 'up' if d.get('connected') else 'down', + 'ip': d.get('ip', ''), 'mac': d.get('mac', ''), 'model': d.get('model', ''), + 'source': 'unifi', 'interface_count': len(ports), + 'search': _search_text( + name, d.get('ip'), _mac_forms(d.get('mac', '')), d.get('model'), + d.get('type'), kind, list(ports), _port_neighbors(ports), + ), + } + + # A switch can have link stats while the device list is unavailable this cycle. + for name, sw in switches.items(): + if name not in devices: + ports = sw.get('ports') or {} + devices[name] = { + 'name': name, 'kind': unifi_kind('usw', sw.get('model', '')), 'status': 'unknown', + 'ip': sw.get('ip', ''), 'mac': '', 'model': sw.get('model', ''), + 'source': 'unifi', 'interface_count': len(ports), + 'search': _search_text(name, sw.get('ip'), sw.get('model'), list(ports), _port_neighbors(ports)), + } + + if modem and modem.get('name'): + name = modem['name'] + if not modem.get('reachable'): + status = 'unknown' + else: + status = 'down' if modem.get('wan_problems') else 'up' + devices[name] = { + 'name': name, 'kind': 'modem', 'status': status, + 'ip': modem.get('lan_ip', ''), 'mac': '', 'model': modem.get('model', ''), + 'source': 'modem', 'interface_count': 0, + 'search': _search_text( + name, modem.get('lan_ip'), modem.get('wan_ip'), modem.get('model'), + 'modem gateway fiber wan at&t att', + ), + } + + return sorted(devices.values(), key=lambda d: d['name'].lower()) + + +def uplinks_for_host(name: str, switches: dict) -> List[dict]: + """Find the switch ports whose LLDP neighbor is this host.""" + found = [] + for sw_name, sw in (switches or {}).items(): + for port_name, port in (sw.get('ports') or {}).items(): + lldp = port.get('lldp') or {} + if lldp.get('system_name') == name: + found.append({ + 'switch': sw_name, 'port': port_name, 'port_idx': port.get('port_idx'), + 'host_interface': lldp.get('port_id', ''), 'up': port.get('up', False), + 'speed_mbps': port.get('speed_mbps', 0), + }) + return found + + +def device_detail(name: str, snapshot: dict, link_stats: dict, modem: Optional[dict]) -> Optional[dict]: + """Return everything known about one device, or None if it doesn't exist.""" + match = next((d for d in build_index(snapshot, link_stats, modem) if d['name'] == name), None) + if match is None: + return None + link_stats = link_stats or {} + detail = {k: v for k, v in match.items() if k != 'search'} + switches = link_stats.get('unifi_switches') or {} + + if match['kind'] == 'host': + states = ((snapshot or {}).get('hosts') or {}).get(name, {}).get('interfaces') or {} + link_ifaces = (link_stats.get('hosts') or {}).get(name) or {} + interfaces = {i: {'state': s} for i, s in states.items()} + for iface, stats in link_ifaces.items(): + interfaces.setdefault(iface, {}).update(stats) + detail['interfaces'] = interfaces + detail['uplinks'] = uplinks_for_host(name, switches) + elif match['kind'] == 'modem': + pass # the page renders the shared WAN / Fiber panel from /api/modem + else: + detail['ports'] = (switches.get(name) or {}).get('ports') or {} + detail['info'] = (link_stats.get('unifi_devices') or {}).get(name) or {} + detail['updated'] = link_stats.get('updated') + return detail diff --git a/docs/img/link-debug-sfp-warning.png b/docs/img/link-debug-sfp-warning.png deleted file mode 100644 index b13687e..0000000 Binary files a/docs/img/link-debug-sfp-warning.png and /dev/null differ diff --git a/docs/img/link-debug-sfp.png b/docs/img/link-debug-sfp.png deleted file mode 100644 index 37c1a8d..0000000 Binary files a/docs/img/link-debug-sfp.png and /dev/null differ diff --git a/docs/img/link-debug-switch.png b/docs/img/link-debug-switch.png deleted file mode 100644 index e30983b..0000000 Binary files a/docs/img/link-debug-switch.png and /dev/null differ diff --git a/docs/img/link-debug.png b/docs/img/link-debug.png deleted file mode 100644 index 569d7cc..0000000 Binary files a/docs/img/link-debug.png and /dev/null differ diff --git a/modem.py b/modem.py new file mode 100644 index 0000000..21cd2e6 --- /dev/null +++ b/modem.py @@ -0,0 +1,287 @@ +"""Read-only client for AT&T BGW-series fiber gateways (tested on BGW320-505). + +The gateway has no API, but its status pages are served without a login, so +this scrapes them over HTTP. Only GET requests to the fixed status pages below +are ever made: nothing here can change gateway settings. + +Units come from the gateway's own help text, checked against live readings: + Rx/Tx power tenths of a dBm (-169 -> -16.9 dBm) + Temperature degrees C + Uptime seconds (despite the help text claiming D:H:M:S) + +Deliberately not used, because the page disagrees with itself: + Vcc / Tx Bias readings values do not match the documented scales, so only + their alarm/warning flags are kept + Rx LOS State reads 1 on a healthy link; "Optical WAN Operational + Status" is the reliable signal + Last Change does not count up, despite the help text +""" +import html +import logging +import re +from datetime import datetime, timezone +from typing import Dict, List, Optional + +import requests + +logger = logging.getLogger('gandalf.modem') + +_FIBER_PAGE = '/cgi-bin/fiberstat.ha' +_BROADBAND_PAGE = '/cgi-bin/broadbandstatistics.ha' +_SYSINFO_PAGE = '/cgi-bin/sysinfo.ha' + +_ROW_RE = re.compile(r']*>(.*?)\s*]*>(.*?)', re.I | re.S) +_TAG_RE = re.compile(r'<[^>]+>') +# e.g. "0 (Threshold -292)": alarm flag, then the limit +_DOM_CELL_RE = re.compile(r']*>\s*(\d+)\s*\(Threshold\s+(-?\d+)\)\s*', re.I) + +# Page heading -> (result key, scale to real units; None = reading is unusable) +_DOM_METRICS = { + 'Temperature': ('temperature', 1.0), + 'Vcc': ('vcc', None), + 'Tx Bias': ('tx_bias', None), + 'Tx Power': ('tx_power', 0.1), + 'Rx Power': ('rx_power', 0.1), +} + +_COUNTERS = { + 'rx_bytes': 'Receive Bytes', + 'tx_bytes': 'Transmit Bytes', + 'rx_errors': 'Receive Errors', + 'tx_errors': 'Transmit Errors', + 'rx_drops': 'Receive Drops', + 'tx_drops': 'Transmit Drops', +} + + +class ModemUnreachable(Exception): + """The gateway did not answer (power, cabling, or routing problem).""" + + +class ModemDataInvalid(Exception): + """The gateway answered but the page was not what we can parse. + + AT&T pushes firmware automatically, so this is how a layout change or a + newly login-protected page shows up. + """ + + +def parse_rows(page: str) -> Dict[str, str]: + """Return {label: value} for the page's / rows; the first label wins. + + First wins because labels repeat: "Transmit Packets" appears under both + IPv4 and IPv6, and the IPv4 block comes first. + """ + rows: Dict[str, str] = {} + for label, value in _ROW_RE.findall(page): + key = html.unescape(_TAG_RE.sub('', label)).strip() + if key: + rows.setdefault(key, ' '.join(html.unescape(_TAG_RE.sub('', value)).split())) + return rows + + +def parse_dom_metrics(page: str) -> Dict[str, dict]: + """Parse the fiber page's Temperature / Vcc / Tx Bias / Tx Power / Rx Power tables.""" + metrics: Dict[str, dict] = {} + for heading, (key, scale) in _DOM_METRICS.items(): + m = re.search( + rf'

\s*{re.escape(heading)}(?: |\s)+Currently\s+(-?\d+)\s*

(.*?)', + page, re.I | re.S, + ) + if not m: + continue + cells = _DOM_CELL_RE.findall(m.group(2)) + if len(cells) != 4: # alarm low, alarm high, warning low, warning high + continue + + def limit(raw: str, scale: Optional[float] = scale) -> Optional[float]: + return round(int(raw) * scale, 1) if scale is not None else None + + metrics[key] = { + 'value': limit(m.group(1)), + 'alarm_low': limit(cells[0][1]), + 'alarm_high': limit(cells[1][1]), + 'warn_low': limit(cells[2][1]), + 'warn_high': limit(cells[3][1]), + 'alarm_flag': cells[0][0] != '0' or cells[1][0] != '0', + 'warn_flag': cells[2][0] != '0' or cells[3][0] != '0', + } + return metrics + + +def metric_status(metric: dict) -> str: + """Classify a DOM metric as 'alarm', 'warning', or 'ok'. + + Trusts the module's own flags, and also compares the reading to its limits + in case the flags are latched off. + """ + value = metric.get('value') + + def outside(low: Optional[float], high: Optional[float]) -> bool: + if value is None: + return False + return (low is not None and value <= low) or (high is not None and value >= high) + + if metric.get('alarm_flag') or outside(metric.get('alarm_low'), metric.get('alarm_high')): + return 'alarm' + if metric.get('warn_flag') or outside(metric.get('warn_low'), metric.get('warn_high')): + return 'warning' + return 'ok' + + +def _to_int(raw: Optional[str]) -> Optional[int]: + try: + return int(raw) # type: ignore[arg-type] + except (TypeError, ValueError): + return None + + +def wan_problems(data: dict) -> List[str]: + """Return human-readable reasons the WAN is down; empty when it looks healthy.""" + problems = [] + if (data.get('broadband_state') or '').lower() != 'up': + problems.append(f'broadband connection is {data.get("broadband_state") or "unknown"}') + if (data.get('optical_state') or '').lower() != 'up': + problems.append(f'optical WAN is {data.get("optical_state") or "unknown"}') + pon = data.get('pon_state') + if pon and 'O5' not in pon: # O5 is the PON "operation" state; older firmware may omit the field + problems.append(f'PON link state is {pon}') + return problems + + +def build_status(fiber_page: str, broadband_page: str, sysinfo_page: str = '') -> dict: + """Combine the status pages into one dict. Raises ModemDataInvalid if unusable.""" + fiber = parse_rows(fiber_page) + broadband = parse_rows(broadband_page) + sysinfo = parse_rows(sysinfo_page) + + if 'Optical WAN Operational Status' not in fiber: + raise ModemDataInvalid('fiber page is missing "Optical WAN Operational Status"') + if 'Broadband Connection' not in broadband: + raise ModemDataInvalid('broadband page is missing "Broadband Connection"') + + metrics = parse_dom_metrics(fiber_page) + data = { + 'model': sysinfo.get('Model Number', ''), + 'firmware': sysinfo.get('Software Version', ''), + 'uptime_s': _to_int(sysinfo.get('Time Since Last Reboot')), + 'broadband_state': broadband.get('Broadband Connection', ''), + 'source': broadband.get('Broadband Connection Source', ''), + 'wan_ip': broadband.get('Broadband IPv4 Address', ''), + 'pon_state': broadband.get('PON Link Status', ''), + 'uni_state': broadband.get('UNI Status', ''), + 'line_speed_mbps': _to_int(broadband.get('Current Speed (Mbps)')), + 'optical_state': fiber.get('Optical WAN Operational Status', ''), + 'wavelength': fiber.get('Wave Length', ''), + 'optics_part_no': fiber.get('Vendor PN', ''), + 'optics': metrics, + 'counters': {k: _to_int(broadband.get(label)) for k, label in _COUNTERS.items()}, + } + + # With a healthy link the optics must be readable; if not, the layout changed. + # With a cut fiber they may legitimately be blank, so don't report that as invalid. + if not wan_problems(data) and 'rx_power' not in metrics: + raise ModemDataInvalid('link is up but no Rx power reading was found') + return data + + +def counter_rates(current: dict, previous: Optional[dict], elapsed_s: float) -> Dict[str, Optional[float]]: + """Return per-second rates between counter snapshots. + + A rate is None rather than a bogus value on the first poll, or when a counter + goes backwards because the gateway rebooted. + """ + rates: Dict[str, Optional[float]] = {} + for key, now_val in current.items(): + then = (previous or {}).get(key) + if now_val is None or then is None or elapsed_s <= 0 or now_val < then: + rates[key] = None + else: + rates[key] = (now_val - then) / elapsed_s + return rates + + +class ModemClient: + def __init__(self, cfg: dict): + self.base_url = cfg['url'].rstrip('/') + self.timeout = cfg.get('timeout', 10) + self.session = requests.Session() + + def _get(self, path: str) -> str: + try: + # Protected pages redirect or render a login form, so never follow redirects. + resp = self.session.get(f'{self.base_url}{path}', timeout=self.timeout, allow_redirects=False) + except requests.RequestException as e: + raise ModemUnreachable(f'{path}: {e}') from e + if resp.status_code != 200: + raise ModemDataInvalid(f'{path} returned HTTP {resp.status_code}') + if re.search(r'\s*Login', resp.text, re.I): + raise ModemDataInvalid(f'{path} now requires a login') + return resp.text + + def poll(self) -> dict: + """Return gateway status. Raises ModemUnreachable or ModemDataInvalid.""" + fiber = self._get(_FIBER_PAGE) + broadband = self._get(_BROADBAND_PAGE) + try: + sysinfo = self._get(_SYSINFO_PAGE) + except (ModemUnreachable, ModemDataInvalid) as e: + # Model and uptime are nice to have, not worth failing the poll over. + logger.warning(f'Modem system info unavailable: {e}') + sysinfo = '' + data = build_status(fiber, broadband, sysinfo) + data['updated'] = datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M:%S UTC') + return data + + +# --------------------------------------------------------------------------- +# WAN history: outages and gateway restarts, which are too brief to catch by polling +# --------------------------------------------------------------------------- +LOG_WAN_FAILED = 'NETWORK_WAN_FAILED_2' +LOG_WAN_RESTORED = 'NETWORK_WAN_RESTORED_2' +LOG_WAN_UNSTABLE = 'NETWORK_WAN_FAILED_MULTIPLE_TIMES_2' +LOG_MODEM_RESTARTED = 'SMART_POWER_MODEM_RESTARTED' +WAN_LOG_KEYS = (LOG_WAN_FAILED, LOG_WAN_RESTORED, LOG_WAN_UNSTABLE, LOG_MODEM_RESTARTED) + +_HISTORY_LIMIT = 50 +# The RPDU's log entry and the gateway's calculated boot time can differ by this much. +_RESTART_MATCH_WINDOW_S = 600 + + +def apply_wan_events(history: dict, events: List[dict]) -> List[dict]: + """Fold UniFi log events into *history* in place and return the ones worth a ticket. + + Each event is {'key', 'ts' (epoch seconds), 'device', 'wan'}. + """ + outages = history.setdefault('outages', []) + restarts = history.setdefault('restarts', []) + actionable = [] + for e in sorted(events, key=lambda ev: ev['ts']): + key = e['key'] + if key == LOG_WAN_FAILED: + if not any(o.get('restored_at') is None for o in outages): + outages.append({'failed_at': e['ts'], 'restored_at': None, 'wan': e.get('wan', '')}) + elif key == LOG_WAN_RESTORED: + for o in reversed(outages): + if o.get('restored_at') is None: + o['restored_at'] = e['ts'] + break + elif key == LOG_MODEM_RESTARTED: + restarts.append({'at': e['ts'], 'by': e.get('device') or 'smart power', 'confirmed': False}) + actionable.append(e) + elif key == LOG_WAN_UNSTABLE: + actionable.append(e) + del outages[:-_HISTORY_LIMIT] + del restarts[:-_HISTORY_LIMIT] + return actionable + + +def record_uptime_reset(history: dict, boot_ts: float) -> None: + """Note that the gateway's uptime went backwards, attributing it to a logged restart if possible.""" + restarts = history.setdefault('restarts', []) + for r in reversed(restarts): + if abs(r['at'] - boot_ts) <= _RESTART_MATCH_WINDOW_S: + r['confirmed'] = True + return + restarts.append({'at': boot_ts, 'by': None, 'confirmed': True}) + del restarts[:-_HISTORY_LIMIT] diff --git a/monitor.py b/monitor.py index 79503a5..7522883 100644 --- a/monitor.py +++ b/monitor.py @@ -11,14 +11,17 @@ import json import logging import re import shlex +import statistics import time from datetime import datetime, timezone from typing import Dict, List, Optional +from urllib.parse import urlparse import requests from urllib3.exceptions import InsecureRequestWarning import db +import modem as modem_api logging.basicConfig( @@ -53,7 +56,8 @@ class PrometheusClient: def __init__(self, url: str): self.url = url.rstrip('/') - def query(self, promql: str) -> list: + def _fetch(self, promql: str) -> Optional[list]: + """Return query results, or None if the query itself failed.""" try: resp = requests.get( f'{self.url}/api/v1/query', @@ -64,13 +68,23 @@ class PrometheusClient: data = resp.json() if data.get('status') == 'success': return data['data']['result'] + logger.error(f'Prometheus query unsuccessful ({promql!r}): {data}') except Exception as e: logger.error(f'Prometheus query failed ({promql!r}): {e}') - return [] + return None - def get_interface_states(self) -> Dict[str, Dict[str, bool]]: - """Return {instance: {device: is_up}} for physical interfaces.""" - results = self.query('node_network_up') + def query(self, promql: str) -> list: + return self._fetch(promql) or [] + + def get_interface_states(self) -> Optional[Dict[str, Dict[str, bool]]]: + """Return {instance: {device: is_up}} for physical interfaces. + + Returns None when Prometheus can't answer or has no series at all, so + callers can tell "we can't see anything" apart from "this host is gone". + """ + results = self._fetch('node_network_up') + if not results: + return None hosts: Dict[str, Dict[str, bool]] = {} for r in results: instance = r['metric'].get('instance', '') @@ -85,6 +99,10 @@ class PrometheusClient: # UniFi client # -------------------------------------------------------------------------- class UnifiClient: + # Devices whose port tables are collected: switches and the gateway/router + _PORT_DEVICE_TYPES = ('usw', 'udm', 'ugw', 'uxg') + _SYSTEM_LOG_MAX_PAGES = 10 + def __init__(self, cfg: dict): self.base_url = cfg['controller'] self.site_id = cfg.get('site_id', 'default') @@ -97,6 +115,96 @@ class UnifiClient: 'Accept': 'application/json', } + def get_device_details(self) -> Optional[Dict[str, dict]]: + """Return firmware, load, WAN, and outlet details per device name. + + Uses the v1 stat API, the only one that includes these fields. + """ + try: + url = f'{self.base_url}/proxy/network/api/s/{self.site_id}/stat/device' + resp = self.session.get(url, headers=self.headers, timeout=15) + resp.raise_for_status() + return { + d.get('name') or d.get('mac', 'unknown'): self._device_detail(d) + for d in resp.json().get('data', []) + } + except Exception as e: + logger.error(f'UniFi device detail error: {e}') + return None + + @staticmethod + def _device_detail(d: dict) -> dict: + def number(raw) -> Optional[float]: + try: + return float(raw) + except (TypeError, ValueError): + return None + + stats = d.get('system-stats') or {} + wan = {} + for key in ('wan1', 'wan2'): + w = d.get(key) or {} + if w: + wan[key] = { + 'up': w.get('up'), 'speed_mbps': w.get('speed'), + 'latency_ms': w.get('latency'), 'availability': w.get('availability'), + } + return { + 'firmware': d.get('version', ''), + 'uptime_s': d.get('uptime'), + 'cpu_pct': number(stats.get('cpu')), + 'mem_pct': number(stats.get('mem')), + 'clients': d.get('num_sta'), + 'temps': [ + {'name': t.get('name', ''), 'value_c': number(t.get('value'))} + for t in d.get('temperatures') or [] + ], + 'wan': wan, + 'power_w': number(d.get('outlet_ac_power_consumption')), + 'outlets': [ + { + 'index': o.get('index'), + 'name': o.get('name', ''), + 'on': o.get('relay_state'), + 'auto_power_cycle': o.get('cycle_enabled'), + 'power_w': number(o.get('outlet_power')), + 'current_a': number(o.get('outlet_current')), + } + for o in d.get('outlet_table') or [] + ], + } + + def get_system_events(self, since_ms: int, keys: tuple) -> Optional[List[dict]]: + """Return system log entries with the given keys, or None if the log is unavailable.""" + url = f'{self.base_url}/proxy/network/v2/api/site/{self.site_id}/system-log/all' + found: List[dict] = [] + try: + for page in range(self._SYSTEM_LOG_MAX_PAGES): + resp = self.session.post( + url, headers=self.headers, timeout=20, + json={ + 'timestampFrom': since_ms, 'timestampTo': int(time.time() * 1000), + 'pageNumber': page, 'pageSize': 200, + }, + ) + resp.raise_for_status() + items = resp.json().get('data') or [] + if not items: + break + for e in items: + if e.get('key') in keys and e.get('timestamp', 0) >= since_ms: + params = e.get('parameters') or {} + found.append({ + 'key': e['key'], 'ts': e['timestamp'] / 1000, + 'device': (params.get('DEVICE') or {}).get('name', ''), + 'wan': (params.get('WAN_ID') or {}).get('name', '') + or str((params.get('WAN_ID') or {}).get('id', '')), + }) + return found + except Exception as e: + logger.error(f'UniFi system log error: {e}') + return None + def get_devices(self) -> Optional[List[dict]]: """Return list of UniFi devices, or None if the controller is unreachable.""" try: @@ -122,7 +230,7 @@ class UnifiClient: return None def get_switch_ports(self) -> Optional[Dict[str, dict]]: - """Return per-port stats for all UniFi switches, keyed by switch name. + """Return per-port stats for UniFi switches and routers, keyed by device name. Uses the v1 stat API which includes full port_table data. Returns {switch_name: {'ip': str, 'model': str, 'ports': {port_name: {...}}}}. @@ -134,7 +242,7 @@ class UnifiClient: devices = resp.json().get('data', []) result: Dict[str, dict] = {} for dev in devices: - if dev.get('type', '').lower() != 'usw': + if dev.get('type', '').lower() not in self._PORT_DEVICE_TYPES: continue sw_name = dev.get('name') or dev.get('mac', 'unknown') sw_ip = dev.get('ip', '') @@ -321,13 +429,19 @@ class PulseClient: return self.run_command(command, _retry=False) return None - def ping(self, ip: str, count: int = 3, timeout: int = 2) -> bool: - """Ping *ip* via the Pulse worker. Returns True if host responds.""" + def ping(self, ip: str, count: int = 3, timeout: int = 2) -> Optional[bool]: + """Ping *ip* via the Pulse worker. + + Returns True/False for the host's reachability, or None if Pulse itself + couldn't run the check, which says nothing about the host. + """ ip_q = shlex.quote(ip) output = self.run_command( f'ping -c {count} -W {timeout} {ip_q} >/dev/null 2>&1 && echo REACHABLE || echo UNREACHABLE' ) - return output is not None and output.strip() == 'REACHABLE' + if output is None: + return None + return output.strip() == 'REACHABLE' # -------------------------------------------------------------------------- @@ -615,9 +729,12 @@ class LinkStatsCollector: except Exception as e: logger.warning(f'UniFi switch port collection failed: {e}') + unifi_devices = (self.unifi.get_device_details() or {}) if self.unifi else {} + return { 'hosts': result_hosts, 'unifi_switches': unifi_switches, + 'unifi_devices': unifi_devices, 'updated': datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M:%S UTC'), } @@ -625,8 +742,10 @@ class LinkStatsCollector: """Compute per-port byte/error rates from delta against previous snapshot.""" dt = now - self._prev_unifi_time if self._prev_unifi_time > 0 else 0 - def rate(new_val: int, old_val: int) -> Optional[float]: - if dt <= 0: + def rate(new_val: int, old_val: Optional[int]) -> Optional[float]: + # No earlier reading (first poll, or a new/renamed port) means no rate, + # not "all bytes since boot arrived in this one interval". + if dt <= 0 or old_val is None: return None return max(0.0, (new_val - old_val) / dt) @@ -637,12 +756,12 @@ class LinkStatsCollector: for pname, d in sw_data['ports'].items(): entry = dict(d) prev = prev_ports.get(pname, {}) - entry['tx_bytes_rate'] = rate(d['tx_bytes'], prev.get('tx_bytes', 0)) - entry['rx_bytes_rate'] = rate(d['rx_bytes'], prev.get('rx_bytes', 0)) - entry['tx_errs_rate'] = rate(d['tx_errors'], prev.get('tx_errors', 0)) - entry['rx_errs_rate'] = rate(d['rx_errors'], prev.get('rx_errors', 0)) - entry['tx_drops_rate'] = rate(d['tx_dropped'], prev.get('tx_dropped', 0)) - entry['rx_drops_rate'] = rate(d['rx_dropped'], prev.get('rx_dropped', 0)) + entry['tx_bytes_rate'] = rate(d['tx_bytes'], prev.get('tx_bytes')) + entry['rx_bytes_rate'] = rate(d['rx_bytes'], prev.get('rx_bytes')) + entry['tx_errs_rate'] = rate(d['tx_errors'], prev.get('tx_errors')) + entry['rx_errs_rate'] = rate(d['rx_errors'], prev.get('rx_errors')) + entry['tx_drops_rate'] = rate(d['tx_dropped'], prev.get('tx_dropped')) + entry['rx_drops_rate'] = rate(d['rx_dropped'], prev.get('rx_dropped')) merged_ports[pname] = entry result[sw_name] = {'ip': sw_data['ip'], 'model': sw_data['model'], 'ports': merged_ports} @@ -685,6 +804,27 @@ class NetworkMonitor: self.cluster_thresh = mon.get('cluster_threshold', 3) self.cluster_name = mon.get('cluster_name', CLUSTER_NAME) + # Optional WAN modem (AT&T BGW-series); see modem.py + modem_cfg = self.cfg.get('modem') or {} + self.modem = modem_api.ModemClient(modem_cfg) if modem_cfg.get('url') else None + self.modem_name = modem_cfg.get('name', 'att-modem') + # Modem reboots (including AT&T's automatic firmware updates) take a few + # minutes, so this can be raised independently of failure_threshold. + self.modem_fail_thresh = modem_cfg.get('failure_threshold', self.fail_thresh) + self.rx_drift_db = modem_cfg.get('rx_drift_db', 3.0) + self.rx_drift_min_samples = modem_cfg.get('rx_drift_min_samples', 288) + self.modem_lan_ip = urlparse(modem_cfg['url']).hostname if modem_cfg.get('url') else '' + self._prev_modem_counters: Optional[dict] = None + self._prev_modem_time = 0.0 + if self.modem: + try: + db.ensure_modem_schema() + except Exception as e: + logger.error(f'Could not create modem history table: {e}') + + # Last good Prometheus result, shown (as unknown) during a Prometheus outage + self._last_iface_states: Dict[str, Dict[str, bool]] = {} + # Build Prometheus instance → hostname lookup self._instance_map: Dict[str, str] = { h['prometheus_instance']: h['name'] @@ -744,6 +884,22 @@ class NetworkMonitor: if host_has_regression: hosts_with_regression.append(host) + # A dead host doesn't report its interfaces as down: its series just + # disappear, so a configured host that's absent needs its own check. + for instance, host in self._instance_map.items(): + if instance in states: + db.resolve_event('host_metrics_missing', host, instance) + continue + hosts_with_regression.append(host) + sup = db.check_suppressed(suppressions, 'host', host) + event_id, is_new, consec, ticket_id = db.upsert_event( + 'host_metrics_missing', 'critical', 'prometheus', + host, instance, + f'No metrics from {host} ({instance}): host or node_exporter is down ({_now_utc()})', + ) + if not sup and consec >= self.fail_thresh and not ticket_id: + self._ticket_metrics_missing(event_id, host, instance, consec) + db.set_baseline(new_baseline) # Cluster-wide check – only genuine regressions count @@ -752,7 +908,7 @@ class NetworkMonitor: event_id, is_new, consec, ticket_id = db.upsert_event( 'cluster_network_issue', 'critical', 'prometheus', self.cluster_name, '', - f'{len(hosts_with_regression)} hosts reporting simultaneous interface failures: ' + f'{len(hosts_with_regression)} hosts with simultaneous interface or metrics failures: ' f'{", ".join(hosts_with_regression)}', ) if not sup and (is_new or not ticket_id): @@ -765,7 +921,8 @@ class NetworkMonitor: f'Affected hosts: {", ".join(hosts_with_regression)}\n' f'Detected: {_now_utc()}\n\n' f'{len(hosts_with_regression)} Proxmox hosts simultaneously reported ' - f'interface regressions (link-down on interfaces previously known UP).\n' + f'interface regressions (link-down on interfaces previously known UP) ' + f'or stopped reporting metrics.\n' f'This likely indicates a switch or upstream network failure.\n\n' f'Please check the core and management switches immediately.' ) @@ -775,6 +932,54 @@ class NetworkMonitor: else: db.resolve_event('cluster_network_issue', self.cluster_name, '') + def _ticket_metrics_missing( + self, event_id: int, host: str, instance: str, consec: int + ) -> None: + title = ( + f'[{host}][auto][production][issue][network][single-node] ' + f'No metrics from host' + ) + desc = ( + f'Host Metrics Alert\n{"=" * 40}\n\n' + f'Host: {host}\n' + f'Prometheus instance: {instance}\n' + f'Detected: {_now_utc()}\n' + f'Consecutive check failures: {consec}\n\n' + f'Prometheus is working but has no node_network_up series for {host}.\n' + f'Either the host is down or node_exporter has stopped on it.\n\n' + f'Please check the host power and network connectivity, then ' + f'prometheus-node-exporter on the host.' + ) + tid = self.tickets.create(title, desc, priority='2') + if tid: + db.set_ticket_id(event_id, tid) + + def _process_prometheus_outage(self, suppressions: list) -> None: + """Interface and host alerts can't fire without Prometheus, so say so.""" + sup = db.check_suppressed(suppressions, 'host', 'prometheus') + event_id, is_new, consec, ticket_id = db.upsert_event( + 'monitor_source_unavailable', 'critical', 'prometheus', + 'prometheus', '', + f'Prometheus returned no interface data; host and interface alerting is paused ({_now_utc()})', + ) + if not sup and consec >= self.fail_thresh and not ticket_id: + title = ( + '[prometheus][auto][production][issue][network][cluster-wide] ' + 'Gandalf has no Prometheus data' + ) + desc = ( + f'Monitoring Source Alert\n{"=" * 40}\n\n' + f'Prometheus URL: {self.prom.url}\n' + f'Detected: {_now_utc()}\n' + f'Consecutive check failures: {consec}\n\n' + f'Gandalf could not get node_network_up data from Prometheus.\n' + f'While this lasts, host and interface failures will NOT be detected.\n\n' + f'Please check the Prometheus service and its scrape targets.' + ) + tid = self.tickets.create(title, desc, priority='2') + if tid: + db.set_ticket_id(event_id, tid) + def _ticket_interface( self, event_id: int, host: str, iface: str, consec: int ) -> None: @@ -842,10 +1047,15 @@ class NetworkMonitor: # ------------------------------------------------------------------ # Ping-only hosts (no node_exporter) # ------------------------------------------------------------------ - def _process_ping_hosts(self, suppressions: list, ping_states: Dict[str, bool]) -> None: + def _process_ping_hosts(self, suppressions: list, ping_states: Dict[str, Optional[bool]]) -> None: for h in self.cfg.get('monitor', {}).get('ping_hosts', []): name, ip = h['name'], h['ip'] - reachable = ping_states.get(name, False) + reachable = ping_states.get(name) + + if reachable is None: + # Pulse couldn't run the check. Neither alert nor resolve on no evidence. + logger.warning(f'Ping check for {name} ({ip}) unavailable this cycle') + continue if not reachable: sup = db.check_suppressed(suppressions, 'host', name) @@ -881,26 +1091,301 @@ class NetworkMonitor: if tid: db.set_ticket_id(event_id, tid) + # ------------------------------------------------------------------ + # WAN modem monitoring + # ------------------------------------------------------------------ + _MODEM_METRIC_LABELS = { + 'rx_power': 'Rx power', 'tx_power': 'Tx power', 'temperature': 'Temperature', + 'vcc': 'Supply voltage', 'tx_bias': 'Tx bias', + } + _MODEM_METRIC_UNITS = {'rx_power': ' dBm', 'tx_power': ' dBm', 'temperature': ' C'} + + def _modem_event( + self, suppressions: list, event_type: str, severity: str, detail: str, + description: str, heading: str, priority: str, advice: str, + ) -> None: + name = self.modem_name + sup = db.check_suppressed(suppressions, 'host', name) + event_id, is_new, consec, ticket_id = db.upsert_event( + event_type, severity, 'modem', name, detail, f'{description} ({_now_utc()})', + ) + if not sup and consec >= self.modem_fail_thresh and not ticket_id: + title = f'[{name}][auto][production][issue][network][cluster-wide] {heading}' + desc = ( + f'WAN Modem Alert\n{"=" * 40}\n\n' + f'Modem: {name} ({self.modem.base_url})\n' + f'Detected: {_now_utc()}\n' + f'Consecutive check failures: {consec}\n\n' + f'{description}\n\n{advice}' + ) + tid = self.tickets.create(title, desc, priority=priority) + if tid: + db.set_ticket_id(event_id, tid) + + def _process_modem_failure(self, suppressions: list, error: Exception) -> None: + """Handle a failed poll. Link and optical events stay as they are: we can't see them.""" + name = self.modem_name + if isinstance(error, modem_api.ModemUnreachable): + db.resolve_event('modem_data_invalid', name, '') + self._modem_event( + suppressions, 'modem_unreachable', 'critical', '', + f'WAN modem is not responding: {error}', + 'WAN modem unreachable', '2', + 'Internet access is probably affected too. Check modem power and the ' + 'cable between the modem and the UDM-Pro.', + ) + else: + db.resolve_event('modem_unreachable', name, '') + self._modem_event( + suppressions, 'modem_data_invalid', 'warning', '', + f'WAN modem status could not be read: {error}', + 'WAN modem status unreadable', '3', + 'The modem answered but its pages could not be parsed, so fiber alerting is ' + 'paused. A firmware update may have changed them; see modem.py.', + ) + + def _process_modem(self, data: dict, suppressions: list) -> None: + """Raise and resolve alerts from a successful modem poll.""" + name = self.modem_name + db.resolve_event('modem_unreachable', name, '') + db.resolve_event('modem_data_invalid', name, '') + + problems = modem_api.wan_problems(data) + if problems: + self._modem_event( + suppressions, 'wan_down', 'critical', '', + f'WAN link is down: {"; ".join(problems)}.', + 'WAN link down', '1', + 'Check the fiber at the modem and the AT&T service status. Rx power at or below ' + 'the alarm limit points to a fiber fault rather than a provider outage.', + ) + else: + db.resolve_event('wan_down', name, '') + + for key, metric in data.get('optics', {}).items(): + status = modem_api.metric_status(metric) + if status == 'alarm' and problems and key == 'rx_power': + # wan_down already covers this, and a cut fiber would otherwise open two tickets. + status = 'ok' + label = self._MODEM_METRIC_LABELS.get(key, key) + value = metric.get('value') + reading = f' at {value}{self._MODEM_METRIC_UNITS.get(key, "")}' if value is not None else '' + limits = ( + f' (warning {metric.get("warn_low")} to {metric.get("warn_high")}, ' + f'alarm {metric.get("alarm_low")} to {metric.get("alarm_high")})' + ) if value is not None else '' + advice = ( + 'Inspect and clean the fiber connector at the modem and look for tight bends. ' + 'If that does not help, contact AT&T: the reading is what the modem measures ' + 'on the provider side of the connection.' + ) + + if status == 'alarm': + db.resolve_event('fiber_signal_warning', name, key) + self._modem_event( + suppressions, 'fiber_signal_alarm', 'critical', key, + f'{label} is in the alarm range{reading}{limits}.', + f'Fiber {label} alarm', '2', advice, + ) + elif status == 'warning': + db.resolve_event('fiber_signal_alarm', name, key) + self._modem_event( + suppressions, 'fiber_signal_warning', 'warning', key, + f'{label} is in the warning range{reading}{limits}.', + f'Fiber {label} warning', '3', advice, + ) + else: + db.resolve_event('fiber_signal_alarm', name, key) + db.resolve_event('fiber_signal_warning', name, key) + + self._check_rx_drift(data, suppressions) + + def _check_rx_drift(self, data: dict, suppressions: list) -> None: + """Alert on a sustained Rx drop relative to this link's own recent normal. + + The modem's limits are very wide (warning at -27.9 dBm on a link that sits + around -17), so a damaged connector could lose many dB before they trip. + """ + name = self.modem_name + rx = (data.get('optics', {}).get('rx_power') or {}).get('value') + if rx is None: + return + history = db.get_modem_rx_values(days=7) + if len(history) < self.rx_drift_min_samples: + return # not enough history yet to know what normal looks like + baseline = statistics.median(history) + drop = round(baseline - rx, 1) + if drop >= self.rx_drift_db: + self._modem_event( + suppressions, 'fiber_rx_drift', 'warning', 'rx_power', + f'Rx power is {rx} dBm, {drop} dB below its 7-day median of {baseline:.1f} dBm.', + 'Fiber Rx power dropped', '3', + 'The signal is still in range but has changed. Look for a recently disturbed ' + 'or bent fiber and check the connector at the modem.', + ) + else: + db.resolve_event('fiber_rx_drift', name, 'rx_power') + + def _load_wan_history(self) -> dict: + raw = db.get_state('wan_history') + if raw: + try: + return json.loads(raw) + except ValueError: + logger.error('Failed to parse wan_history JSON; starting over') + return {'outages': [], 'restarts': []} + + def _note_modem_restart(self, data: dict) -> None: + uptime = data.get('uptime_s') + if uptime is None: + return + previous = None + raw = db.get_state('modem_stats') + if raw: + try: + previous = json.loads(raw).get('uptime_s') + except ValueError: + logger.error('Failed to parse modem_stats JSON') + if previous is not None and uptime < previous: + logger.warning(f'Modem restarted: uptime went from {previous}s to {uptime}s') + history = self._load_wan_history() + modem_api.record_uptime_reset(history, time.time() - uptime) + db.set_state('wan_history', history) + + def _one_shot_ticket( + self, suppressions: list, event_type: str, description: str, + heading: str, priority: str, advice: str, + ) -> None: + """Report something that has already happened: open a ticket and resolve the event.""" + name = self.modem_name + event_id, _, _, ticket_id = db.upsert_event(event_type, 'warning', 'unifi', name, '', description) + if not db.check_suppressed(suppressions, 'host', name) and not ticket_id: + tid = self.tickets.create( + f'[{name}][auto][production][issue][network][cluster-wide] {heading}', + f'WAN Event\n{"=" * 40}\n\nModem: {name}\nReported: {_now_utc()}\n\n{description}\n\n{advice}', + priority=priority, + ) + if tid: + db.set_ticket_id(event_id, tid) + db.resolve_event(event_type, name, '') + + def _process_wan_log(self, suppressions: list) -> None: + """Read WAN outages and modem restarts from the UniFi log. + + These can be over in two minutes, so a 5 minute poll of the modem may never see them. + """ + now = time.time() + raw_cursor = db.get_state('unifi_log_cursor') + if raw_cursor is None: + # First run: start from now rather than ticketing old history. + db.set_state('unifi_log_cursor', str(int(now * 1000))) + return + # After long monitor downtime, don't open tickets for stale events. + since_ms = max(int(raw_cursor) + 1, int((now - 86400) * 1000)) + events = self.unifi.get_system_events(since_ms, modem_api.WAN_LOG_KEYS) + if not events: + return # None = log unavailable, so the cursor stays put and it's retried + + history = self._load_wan_history() + actionable = modem_api.apply_wan_events(history, events) + + # UniFi repeats "failed multiple times" per failure, so ticket it once a day at most. + unstable = [e for e in actionable if e['key'] == modem_api.LOG_WAN_UNSTABLE] + ticket_unstable = bool(unstable) and now - history.get('unstable_ticketed_at', 0) >= 86400 + if ticket_unstable: + history['unstable_ticketed_at'] = now + + # Save first: a failed ticket call shouldn't replay these events next cycle. + db.set_state('wan_history', history) + db.set_state('unifi_log_cursor', str(int(max(e['ts'] for e in events) * 1000))) + + for e in actionable: + if e['key'] == modem_api.LOG_MODEM_RESTARTED: + when = datetime.fromtimestamp(e['ts'], timezone.utc).strftime('%Y-%m-%d %H:%M:%S UTC') + self._one_shot_ticket( + suppressions, 'modem_power_cycled', + f'{e.get("device") or "Smart Power"} power-cycled the modem at {when} ' + f'to try to restore the internet connection.', + 'Modem auto power-cycled after internet loss', '3', + 'The internet connection failed badly enough to trigger auto power-cycle. ' + 'Check the WAN / Fiber panel for outages around this time and Rx power for a drop.', + ) + if ticket_unstable: + self._one_shot_ticket( + suppressions, 'wan_unstable', + 'UniFi reports the WAN connection went down multiple times in the last 24 hours.', + 'WAN connection unstable', '2', + 'See the WAN history on the modem page. Repeated drops with healthy Rx power ' + 'point toward the provider or the modem rather than the fiber.', + ) + + def _collect_modem(self, suppressions: list) -> None: + """Poll the modem, then store its status and raise any alerts.""" + try: + data = self.modem.poll() + except (modem_api.ModemUnreachable, modem_api.ModemDataInvalid) as e: + logger.warning(f'Modem poll failed: {e}') + self._process_modem_failure(suppressions, e) + db.set_state('modem_stats', { + 'reachable': False, 'error': str(e), 'name': self.modem_name, + 'lan_ip': self.modem_lan_ip, 'updated': _now_utc(), + }) + return + + # Uptime going backwards between polls means the gateway restarted, even if + # it came back too quickly for any poll to fail. Read before it's overwritten. + self._note_modem_restart(data) + + now = time.monotonic() + data['rates'] = modem_api.counter_rates( + data['counters'], self._prev_modem_counters, now - self._prev_modem_time, + ) + self._prev_modem_counters = data['counters'] + self._prev_modem_time = now + data['reachable'] = True + data['name'] = self.modem_name + data['lan_ip'] = self.modem_lan_ip + data['wan_problems'] = modem_api.wan_problems(data) + data['optics_status'] = {k: modem_api.metric_status(m) for k, m in data['optics'].items()} + db.set_state('modem_stats', data) + + # Alert before recording, so a sudden drop isn't compared against itself. + self._process_modem(data, suppressions) + + optics = data['optics'] + db.record_modem_sample( + (optics.get('rx_power') or {}).get('value'), + (optics.get('tx_power') or {}).get('value'), + (optics.get('temperature') or {}).get('value'), + ) + # ------------------------------------------------------------------ # Snapshot collection (for dashboard) # ------------------------------------------------------------------ def _collect_snapshot( - self, iface_states: Dict[str, Dict[str, bool]], + self, iface_states: Optional[Dict[str, Dict[str, bool]]], unifi_devices: Optional[List[dict]] = None, - ping_states: Optional[Dict[str, bool]] = None, + ping_states: Optional[Dict[str, Optional[bool]]] = None, ) -> dict: # Accept pre-fetched devices; fall back to empty list if unavailable display_unifi = unifi_devices if unifi_devices is not None else [] + # During a Prometheus outage keep showing the hosts, with their state unknown, + # rather than emptying the dashboard or implying that they're down. + prom_ok = iface_states is not None + shown_states = iface_states if prom_ok else self._last_iface_states + hosts = {} - for instance, ifaces in iface_states.items(): + for instance, ifaces in shown_states.items(): if instance not in self._instance_map: continue # skip Prometheus instances not in config (e.g. LXC app servers) host = self._hostname(instance) phys = {k: v for k, v in ifaces.items()} up_count = sum(1 for v in phys.values() if v) total = len(phys) - if total == 0 or up_count == total: + if not prom_ok: + status = 'unknown' + elif total == 0 or up_count == total: status = 'up' elif up_count == 0: status = 'down' @@ -914,13 +1399,23 @@ class NetworkMonitor: 'source': 'prometheus', } + # Configured hosts with no data: down if Prometheus is fine, otherwise unknown + for instance, host in self._instance_map.items(): + if instance not in shown_states: + hosts[host] = { + 'ip': instance.split(':')[0], + 'interfaces': {}, + 'status': 'down' if prom_ok else 'unknown', + 'source': 'prometheus', + } + for h in self.cfg.get('monitor', {}).get('ping_hosts', []): name, ip = h['name'], h['ip'] - reachable = (ping_states or {}).get(name, False) + reachable = (ping_states or {}).get(name) hosts[name] = { 'ip': ip, 'interfaces': {}, - 'status': 'up' if reachable else 'down', + 'status': 'unknown' if reachable is None else ('up' if reachable else 'down'), 'source': 'ping', } @@ -949,13 +1444,15 @@ class NetworkMonitor: unifi_devices = self.unifi.get_devices() # 3a. Ping-only hosts once — shared by snapshot and alert processing - ping_states: Dict[str, bool] = { + ping_states: Dict[str, Optional[bool]] = { h['name']: self.pulse.ping(h['ip']) for h in self.cfg.get('monitor', {}).get('ping_hosts', []) } # 3b. Collect and store snapshot for dashboard snapshot = self._collect_snapshot(iface_states, unifi_devices, ping_states) + if iface_states is not None: + self._last_iface_states = iface_states db.set_state('network_snapshot', snapshot) db.set_state('last_check', _now_utc()) @@ -968,14 +1465,29 @@ class NetworkMonitor: # 5. Process alerts using already-fetched data suppressions = db.get_active_suppressions() - self._process_interfaces(iface_states, suppressions) + if iface_states is None: + self._process_prometheus_outage(suppressions) + else: + db.resolve_event('monitor_source_unavailable', 'prometheus', '') + self._process_interfaces(iface_states, suppressions) self._process_unifi(unifi_devices, suppressions) self._process_ping_hosts(suppressions, ping_states) + # WAN modem: isolated so a failure here can't stop the rest of the cycle + if self.modem: + try: + # Log first, so a restart it reports can explain an uptime reset below + self._process_wan_log(suppressions) + self._collect_modem(suppressions) + except Exception as e: + logger.error(f'Modem collection failed: {e}', exc_info=True) + # Housekeeping: deactivate expired suppressions and purge old resolved events db.cleanup_expired_suppressions() db.purge_old_resolved_events(days=90) + if self.modem: + db.purge_old_modem_samples(days=90) logger.info('Network check cycle complete') diff --git a/schema.sql b/schema.sql index 6248a98..8a76659 100644 --- a/schema.sql +++ b/schema.sql @@ -46,6 +46,15 @@ CREATE TABLE IF NOT EXISTS suppression_rules ( INDEX idx_sup_lookup (active, target_type, target_name, target_detail) ) ENGINE=InnoDB; +-- ── WAN modem optical history (one row per poll) ────────────────────── +-- Also created automatically at monitor startup (db.ensure_modem_schema). +CREATE TABLE IF NOT EXISTS modem_samples ( + sampled_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP PRIMARY KEY, + rx_power_dbm DECIMAL(5,1) NULL, + tx_power_dbm DECIMAL(5,1) NULL, + temp_c DECIMAL(5,1) NULL +) ENGINE=InnoDB; + -- ── Monitor state (key/value store for snapshot + baseline) ─────────── CREATE TABLE IF NOT EXISTS monitor_state ( key_name VARCHAR(100) PRIMARY KEY, diff --git a/scripts/demo/capture.py b/scripts/demo/capture.py index 1a1ccb7..f5cd49d 100644 --- a/scripts/demo/capture.py +++ b/scripts/demo/capture.py @@ -6,6 +6,7 @@ import asyncio import io import pathlib import sys +from urllib.parse import quote from PIL import Image from playwright.async_api import async_playwright @@ -81,18 +82,12 @@ async def shots(): if hb: await pg.locator("#hosts-grid, .host-grid, .hosts-grid").first.screenshot(path=f"{OUT}/host-cards.png") print("host-cards ok") - await go(pg, "/links") - await pg.screenshot(path=f"{OUT}/link-debug.png") - names = await pg.evaluate("[...document.querySelectorAll('.link-host-name')].map(e=>e.textContent.trim())") - print("hosts", names) - for nm, fn in ( - ("large1", "link-debug-sfp-warning"), - ("compute-storage-gpu-01", "link-debug-sfp"), - ("Pro 24 PoE", "link-debug-switch"), - ): - i = next((k for k, n in enumerate(names) if nm in n), None) - if i is not None: - await el(pg, ".link-host-panel", fn, i) + await go(pg, "/device/large1") + await pg.locator('.device-row-toggle[data-row="enp5s0"]').click() + await pg.wait_for_timeout(300) + await pg.screenshot(path=f"{OUT}/device-host.png", full_page=True) + await go(pg, "/device/" + quote("Pro 24 PoE")) + await pg.screenshot(path=f"{OUT}/device-switch.png", full_page=True) await go(pg, "/inspector") await pg.screenshot(path=f"{OUT}/inspector.png") await select_port(pg, "USW-Aggregation", 4) @@ -133,7 +128,7 @@ async def gif(): await snap(3, 600) await pg.mouse.wheel(0, 700) await snap(2, 600) - await go(pg, "/links", 4500) + await go(pg, "/device/large1", 4500) await snap(3) await pg.mouse.wheel(0, 500) await snap(2, 500) diff --git a/static/app.js b/static/app.js index 998db04..cfe0465 100644 --- a/static/app.js +++ b/static/app.js @@ -1,4 +1,6 @@ 'use strict'; +/* exported initDeviceSearch */ +/* global DEVICE_NAME */ // ── Auto-redirect on auth timeout ───────────────────────────────────── // Wraps fetch so a 401 (Authelia session expired) forces a full reload. @@ -40,6 +42,7 @@ async function refreshAll() { const [netResult, statusResult] = await Promise.allSettled([ lt.api.get('/api/network'), lt.api.get('/api/status'), + refreshModem(), ]); if (netResult.status === 'fulfilled') { const net = netResult.value; @@ -170,6 +173,263 @@ function updateTopology(hosts) { }); } +// ── Device search ───────────────────────────────────────────────────── +function deviceUrl(name) { + return `/device/${encodeURIComponent(name)}`; +} + +// All words must match somewhere; the device's name matching ranks first. +function rankDevices(devices, query) { + const words = query.trim().toLowerCase().split(/\s+/).filter(Boolean); + if (!words.length) return []; + const scored = []; + for (const d of devices) { + if (!words.every(w => d.search.includes(w))) continue; + const name = d.name.toLowerCase(); + let score = 3; + if (name === words[0]) score = 0; + else if (name.startsWith(words[0])) score = 1; + else if (name.includes(words[0]) || (d.ip || '').startsWith(words[0])) score = 2; + scored.push({ d, score }); + } + return scored + .sort((a, b) => a.score - b.score || a.d.name.localeCompare(b.d.name)) + .slice(0, 12) + .map(s => s.d); +} + +function initDeviceSearch() { + const input = document.getElementById('device-search-input'); + const results = document.getElementById('device-search-results'); + if (!input || !results) return; + let devices = []; + let matches = []; + let active = 0; + + async function loadDevices() { + try { + devices = (await lt.api.get('/api/devices')).devices || []; + } catch (err) { + console.warn('Device list unavailable', err); + } + } + + function close() { + results.hidden = true; + input.setAttribute('aria-expanded', 'false'); + input.removeAttribute('aria-activedescendant'); + } + + function render() { + matches = rankDevices(devices, input.value); + if (!input.value.trim()) { close(); return; } + active = Math.min(active, Math.max(matches.length - 1, 0)); + results.hidden = false; + input.setAttribute('aria-expanded', 'true'); + if (!matches.length) { + results.innerHTML = '<div class="device-search-empty">No devices match.</div>'; + input.removeAttribute('aria-activedescendant'); + return; + } + results.innerHTML = matches.map((d, i) => ` + <a class="device-search-row${i === active ? ' is-active' : ''}" id="device-opt-${i}" role="option" + aria-selected="${i === active}" href="${lt.escHtml(deviceUrl(d.name))}"> + <span class="dot-${lt.escHtml(d.status)}"></span> + <span class="device-search-name">${lt.escHtml(d.name)}</span> + <span class="device-search-kind">${lt.escHtml(d.kind)}</span> + <span class="device-search-meta">${lt.escHtml(d.ip || '')}</span> + <span class="device-search-meta">${lt.escHtml(d.model || '')}</span> + </a>`).join(''); + input.setAttribute('aria-activedescendant', `device-opt-${active}`); + } + + input.addEventListener('input', () => { active = 0; render(); }); + input.addEventListener('focus', () => { loadDevices().then(render); }); + input.addEventListener('keydown', e => { + if (e.key === 'Escape') { close(); return; } + if (!matches.length) return; + if (e.key === 'ArrowDown') { e.preventDefault(); active = (active + 1) % matches.length; render(); } + if (e.key === 'ArrowUp') { e.preventDefault(); active = (active - 1 + matches.length) % matches.length; render(); } + if (e.key === 'Enter') { e.preventDefault(); window.location.href = deviceUrl(matches[active].name); } + }); + document.addEventListener('click', e => { + if (!e.target.closest('#device-search')) close(); + }); + document.addEventListener('keydown', e => { + const typing = e.target.closest('input, textarea, select, [contenteditable]'); + if (e.key === '/' && !typing && !e.ctrlKey && !e.metaKey) { + e.preventDefault(); + input.focus(); + } + }); + loadDevices(); +} + +// ── WAN modem / fiber panel ─────────────────────────────────────────── +async function refreshModem() { + try { + updateModem(await lt.api.get('/api/modem')); + } catch (err) { + console.warn('Modem data unavailable', err); + } +} + +function _modemUptime(seconds) { + if (seconds === null || seconds === undefined) return '–'; + const d = Math.floor(seconds / 86400); + const h = Math.floor((seconds % 86400) / 3600); + const m = Math.floor((seconds % 3600) / 60); + return d ? `${d}d ${h}h` : `${h}h ${m}m`; +} + +function _modemRate(bytesPerSec) { + if (bytesPerSec === null || bytesPerSec === undefined) return '–'; + const mbps = bytesPerSec * 8 / 1e6; + return mbps >= 1000 ? `${(mbps / 1000).toFixed(2)} Gbps` : `${mbps.toFixed(1)} Mbps`; +} + +function _modemStat(label, value, cls, note) { + return ` + <div class="modem-stat"> + <span class="modem-stat-label">${lt.escHtml(label)}</span> + <span class="modem-stat-value ${cls || ''}">${lt.escHtml(String(value))}</span> + ${note ? `<span class="modem-stat-note">${lt.escHtml(note)}</span>` : ''} + </div>`; +} + +function _modemOptic(label, metric, status, unit) { + if (!metric || metric.value === null || metric.value === undefined) return _modemStat(label, '–'); + const cls = { alarm: 'lt-text-red', warning: 'lt-text-amber', ok: 'lt-text-green' }[status] || ''; + return _modemStat(label, `${metric.value} ${unit}`, cls, `warn ${metric.warn_low} to ${metric.warn_high}`); +} + +function _modemChart(history) { + const points = (history || []).filter(p => p.rx_avg !== null && p.rx_avg !== undefined); + if (points.length < 2) { + return '<div class="modem-chart-empty">Rx power trend appears after a few polls.</div>'; + } + const W = 600, H = 70, PAD = 4; + const values = points.map(p => p.rx_avg); + const low = Math.min(...values) - 0.5, high = Math.max(...values) + 0.5; + const t0 = points[0].ts, tSpan = (points[points.length - 1].ts - t0) || 1; + const coord = p => `${((p.ts - t0) / tSpan * W).toFixed(1)},` + + `${(PAD + (high - p.rx_avg) / (high - low) * (H - 2 * PAD)).toFixed(1)}`; + + // Break the line where samples are missing rather than drawing across an outage + const gaps = points.slice(1).map((p, i) => p.ts - points[i].ts).sort((a, b) => a - b); + const maxGap = gaps[Math.floor(gaps.length / 2)] * 3; + const segments = [[]]; + points.forEach((p, i) => { + if (i && p.ts - points[i - 1].ts > maxGap) segments.push([]); + segments[segments.length - 1].push(coord(p)); + }); + const lines = segments + .map(s => `<polyline points="${s.join(' ')}" fill="none" vector-effect="non-scaling-stroke"/>`) + .join(''); + const summary = `range ${Math.min(...values)} to ${Math.max(...values)} dBm`; + return ` + <div class="modem-chart"> + <div class="modem-chart-title">Rx power, last 7 days <span>${lt.escHtml(summary)}</span></div> + <svg viewBox="0 0 ${W} ${H}" preserveAspectRatio="none" role="img" + aria-label="Rx power trend, ${lt.escHtml(summary)}">${lines}</svg> + </div>`; +} + +function _modemWhen(epochSeconds) { + return new Date(epochSeconds * 1000).toLocaleString([], { dateStyle: 'short', timeStyle: 'short' }); +} + +function _modemLastRestart(history) { + const restarts = (history && history.restarts) || []; + if (!restarts.length) return ''; + const last = restarts[restarts.length - 1]; + return `restarted ${_modemWhen(last.at)}${last.by ? ` by ${last.by}` : ''}`; +} + +function _modemDuration(seconds) { + if (seconds < 60) return `${Math.round(seconds)}s`; + if (seconds < 3600) return `${Math.floor(seconds / 60)}m ${Math.round(seconds % 60)}s`; + return `${Math.floor(seconds / 3600)}h ${Math.floor((seconds % 3600) / 60)}m`; +} + +function _modemOutages(history) { + const outages = ((history && history.outages) || []).slice(-5).reverse(); + if (!outages.length) return ''; + const rows = outages.map(o => { + const length = o.restored_at === null || o.restored_at === undefined + ? '<span class="lt-text-red">ongoing</span>' + : lt.escHtml(_modemDuration(o.restored_at - o.failed_at)); + return `<li><span>${lt.escHtml(_modemWhen(o.failed_at))}</span><span>${length}</span></li>`; + }).join(''); + return ` + <div class="modem-outages"> + <div class="modem-chart-title">Recent WAN outages</div> + <ul>${rows}</ul> + </div>`; +} + +function updateModem(data) { + const section = document.getElementById('modem-section'); + const body = document.getElementById('modem-body'); + if (!section || !body) return; + section.hidden = !data.configured; + if (!data.configured) return; + + // On the dashboard, link to the modem's page. On that page, it would link to itself. + const detailsLink = document.getElementById('modem-device-link'); + if (detailsLink && typeof DEVICE_NAME === 'undefined' && data.name) { + detailsLink.href = deviceUrl(data.name); + detailsLink.hidden = false; + } + + const rxLabel = document.getElementById('topo-modem-rx'); + const topo = document.getElementById('topo-modem'); + const title = document.getElementById('modem-title'); + + if (!data.reachable) { + if (rxLabel) rxLabel.textContent = 'unreachable'; + if (topo) topo.classList.add('topo-v2-status-down'); + body.innerHTML = ` + <div class="modem-error">Modem status unavailable: ${lt.escHtml(data.error || 'unknown error')} + <span class="modem-stat-note">Last attempt ${lt.escHtml(data.updated || '–')}</span></div> + ${_modemChart(data.history)}`; + return; + } + + const problems = data.wan_problems || []; + const optics = data.optics || {}; + const status = data.optics_status || {}; + const counters = data.counters || {}; + const rates = data.rates || {}; + const rx = optics.rx_power ? optics.rx_power.value : null; + + if (topo) topo.classList.toggle('topo-v2-status-down', problems.length > 0); + if (rxLabel) rxLabel.textContent = (rx === null || rx === undefined) ? 'Rx –' : `Rx ${rx} dBm`; + if (title) { + title.textContent = [data.model || 'AT&T Gateway', data.firmware && `fw ${data.firmware}`] + .filter(Boolean).join(' · '); + } + + const errs = (counters.rx_errors || 0) + (counters.tx_errors || 0); + const drops = (counters.rx_drops || 0) + (counters.tx_drops || 0); + body.innerHTML = ` + ${problems.length ? `<div class="modem-error">WAN down: ${lt.escHtml(problems.join('; '))}</div>` : ''} + <div class="modem-stats"> + ${_modemStat('WAN', problems.length ? 'DOWN' : 'UP', problems.length ? 'lt-text-red' : 'lt-text-green', + data.wan_ip || '')} + ${_modemStat('PON', data.pon_state || '–', '', data.wavelength || '')} + ${_modemOptic('Rx power', optics.rx_power, status.rx_power, 'dBm')} + ${_modemOptic('Tx power', optics.tx_power, status.tx_power, 'dBm')} + ${_modemOptic('Optics temp', optics.temperature, status.temperature, '°C')} + ${_modemStat('Throughput', `↓ ${_modemRate(rates.rx_bytes)}`, '', `↑ ${_modemRate(rates.tx_bytes)}`)} + ${_modemStat('Errors / drops', `${errs} / ${drops}`, errs ? 'lt-text-amber' : '', 'since modem reboot')} + ${_modemStat('Uptime', _modemUptime(data.uptime_s), '', _modemLastRestart(data.wan_history))} + </div> + ${_modemChart(data.history)} + ${_modemOutages(data.wan_history)} + <div class="modem-stat-note modem-updated">Updated ${lt.escHtml(data.updated || '–')}</div>`; +} + function updateUnifiTable(devices) { const tbody = document.querySelector('#unifi-table tbody'); if (!tbody || !devices.length) return; @@ -188,7 +448,7 @@ function updateUnifiTable(devices) { return ` <tr class="${statusClass}"> <td><span class="${dotClass}"></span> ${statusText}</td> - <td><strong>${lt.escHtml(d.name)}</strong></td> + <td><a class="device-link" href="${lt.escHtml(deviceUrl(d.name))}"><strong>${lt.escHtml(d.name)}</strong></a></td> <td>${lt.escHtml(d.type)}</td> <td>${lt.escHtml(d.model)}</td> <td>${lt.escHtml(d.ip)}</td> diff --git a/static/device.js b/static/device.js new file mode 100644 index 0000000..8fd0293 --- /dev/null +++ b/static/device.js @@ -0,0 +1,454 @@ +'use strict'; +/* global DEVICE_NAME, deviceUrl, refreshModem, _toIso */ + +// Per-device page: overview, alerts, and a compact interface/port table whose +// rows expand to show everything the old Link Debug page showed. + +const esc = s => lt.escHtml(String(s === null || s === undefined ? '' : s)); + +const _openRows = new Set(); // survives auto-refresh +let _filter = 'all'; +let _query = ''; +let _device = null; + +// ── Formatting ─────────────────────────────────────────────────────── +function fmtRate(bytesPerSec) { + if (bytesPerSec === null || bytesPerSec === undefined) return '–'; + const bps = bytesPerSec * 8; + if (bps < 1e3) return bps.toFixed(0) + ' bps'; + if (bps < 1e6) return (bps / 1e3).toFixed(1) + ' Kbps'; + if (bps < 1e9) return (bps / 1e6).toFixed(2) + ' Mbps'; + return (bps / 1e9).toFixed(3) + ' Gbps'; +} + +function fmtSpeed(mbps) { + if (!mbps) return '–'; + return mbps >= 1000 ? (mbps / 1000).toFixed(0) + ' Gbps' : mbps + ' Mbps'; +} + +function fmtErrors(rate) { + if (rate === null || rate === undefined) return '–'; + if (rate < 0.001) return '<span class="val-good">0 /s</span>'; + return `<span class="val-crit">${rate.toFixed(3)} /s</span>`; +} + +function fmtUnit(value, unit, digits) { + return value === null || value === undefined ? '–' : `${value.toFixed(digits)} ${unit}`; +} + +function fmtUptime(seconds) { + if (seconds === null || seconds === undefined) return '–'; + const d = Math.floor(seconds / 86400), h = Math.floor((seconds % 86400) / 3600); + const m = Math.floor((seconds % 3600) / 60); + return d ? `${d}d ${h}h` : `${h}h ${m}m`; +} + +function ratePct(bytesPerSec, mbps) { + if (!mbps || !bytesPerSec) return 0; + return Math.min(100, (bytesPerSec * 8 / 1e6 / mbps) * 100); +} + +function barClass(pct, isTx) { + if (pct > 85) return 'lt-progress--red'; + if (pct > 65) return 'lt-progress--amber'; + return isTx ? '' : 'lt-progress--cyan'; +} + +// ── SFP/DOM value classification ───────────────────────────────────── +function rxClass(dbm) { + if (dbm === null || dbm === undefined) return 'val-neutral'; + if (dbm < -15) return 'val-crit'; + return dbm < -10 ? 'val-warn' : 'val-good'; +} +function txClass(dbm) { + if (dbm === null || dbm === undefined) return 'val-neutral'; + return dbm < -5 ? 'val-crit' : 'val-good'; +} +function tempClass(c) { + if (c === null || c === undefined) return 'val-neutral'; + if (c > 80) return 'val-crit'; + return c > 70 ? 'val-warn' : 'val-good'; +} +function voltClass(v) { + if (v === null || v === undefined) return 'val-neutral'; + if (v < 3.0 || v > 3.6) return 'val-crit'; + return (v < 3.1 || v > 3.5) ? 'val-warn' : 'val-good'; +} + +function powerFill(cls) { + return cls === 'val-good' ? 'power-ok' : cls === 'val-warn' ? 'power-warn' : 'power-crit'; +} + +function badges(d) { + const out = []; + if ((d.tx_errs_rate || 0) > 0.001 || (d.rx_errs_rate || 0) > 0.001) out.push('<span class="link-alert-badge">ERR</span>'); + if ((d.tx_drops_rate || 0) > 0.001 || (d.rx_drops_rate || 0) > 0.001) out.push('<span class="link-alert-badge link-alert-amber">DROP</span>'); + if ((d.carrier_changes || 0) > 3) out.push('<span class="link-alert-badge link-alert-amber">FLAP</span>'); + return out.join(''); +} + +function stat(label, valueHtml, tooltip) { + const tip = tooltip ? ` data-tooltip="${esc(tooltip)}"` : ''; + return `<div class="link-stat"><span class="link-stat-label"${tip}>${esc(label)}</span>` + + `<span class="link-stat-value">${valueHtml}</span></div>`; +} + +function trafficBars(d) { + const tx = ratePct(d.tx_bytes_rate, d.speed_mbps), rx = ratePct(d.rx_bytes_rate, d.speed_mbps); + const row = (label, pct, rate, isTx) => ` + <div class="traffic-row"> + <span class="traffic-label">${label}</span> + <div class="lt-progress ${barClass(pct, isTx)}"><div class="lt-progress-bar" style="width:${pct}%"></div></div> + <span class="traffic-value">${fmtRate(rate)}</span> + </div>`; + return `<div class="traffic-section">${row('TX', tx, d.tx_bytes_rate, true)}${row('RX', rx, d.rx_bytes_rate, false)}</div>`; +} + +function sfpPanel(s) { + if (!s || !Object.keys(s).length) return ''; + const rx = rxClass(s.rx_power_dbm), tx = txClass(s.tx_power_dbm); + const has = v => v !== null && v !== undefined; + const rxPct = has(s.rx_power_dbm) ? Math.min(100, Math.max(0, (s.rx_power_dbm + 20) / 15 * 100)) : 0; + const txPct = has(s.tx_power_dbm) ? Math.min(100, Math.max(0, (s.tx_power_dbm + 10) / 8 * 100)) : 0; + const item = (label, cls, value, tip, pct) => ` + <div class="sfp-stat"> + <span class="sfp-stat-label" data-tooltip="${esc(tip)}">${label}</span> + <span class="sfp-stat-value ${cls}">${value}</span> + ${pct === undefined ? '' : `<div class="power-row"><div class="power-track"><div class="power-fill ${powerFill(cls)}" style="width:${pct}%"></div></div></div>`} + </div>`; + const vendor = [s.vendor, s.part_no, s.sfp_type, s.connector, s.wavelength_nm && `${s.wavelength_nm} nm`] + .filter(Boolean).map(esc).join(' / '); + return ` + <div class="sfp-panel"> + <div class="sfp-vendor-row">${vendor}</div> + <div class="sfp-grid"> + ${item('Temp', tempClass(s.temp_c), fmtUnit(s.temp_c, '°C', 1), 'Module temperature. Normal: below 70°C.')} + ${item('Voltage', voltClass(s.voltage_v), fmtUnit(s.voltage_v, 'V', 2), 'Supply voltage. Normal: 3.1–3.5 V.')} + ${item('Bias', '', fmtUnit(s.bias_ma, 'mA', 2), 'Laser bias current. High values may mean an aging laser.')} + ${item('TX Power', tx, fmtUnit(s.tx_power_dbm, 'dBm', 2), 'Optical transmit power.', txPct)} + ${item('RX Power', rx, fmtUnit(s.rx_power_dbm, 'dBm', 2), 'Optical receive power. Very low values may mean dirty or damaged fiber.', rxPct)} + ${has(s.rx_power_dbm) && has(s.tx_power_dbm) ? item('RX−TX Δ', '', `${(s.rx_power_dbm - s.tx_power_dbm).toFixed(2)} dBm`, 'Difference between receive and transmit power.') : ''} + </div> + </div>`; +} + +// ── Normalized rows, so hosts and switches share one table ─────────── +function hostRows(interfaces) { + return Object.entries(interfaces) + .sort(([a], [b]) => a.localeCompare(b, undefined, { numeric: true })) + .map(([name, d]) => { + const up = d.state ? d.state === 'up' : d.link_detected !== false; + return { + id: name, label: name, up, data: d, sub: d.port_type || '', + detail: ` + <div class="link-stats-grid"> + ${stat('Duplex', esc(d.duplex || '–'), 'Full = both directions at once. Half can cause collisions.')} + ${stat('Auto-neg', d.auto_neg === null || d.auto_neg === undefined ? '–' : d.auto_neg ? 'On' : 'Off')} + ${stat('Carrier Δ', d.carrier_changes === null || d.carrier_changes === undefined ? '–' : esc(d.carrier_changes), 'Times the link went up or down. High values mean flapping.')} + ${stat('TX Err/s', fmtErrors(d.tx_errs_rate))} + ${stat('RX Err/s', fmtErrors(d.rx_errs_rate))} + ${stat('TX Drop/s', fmtErrors(d.tx_drops_rate))} + ${stat('RX Drop/s', fmtErrors(d.rx_drops_rate))} + </div> + ${trafficBars(d)} + ${sfpPanel(d.sfp)}`, + }; + }); +} + +function portRows(ports) { + return Object.entries(ports) + .sort(([, a], [, b]) => (a.port_idx || 0) - (b.port_idx || 0)) + .map(([name, d]) => { + const neighbor = d.lldp && d.lldp.system_name; + return { + id: name, label: name, num: d.port_idx, up: !!d.up, data: d, + neighbor: d.lldp ? { name: neighbor, port: d.lldp.port_id } : null, + sub: [d.is_uplink ? 'uplink' : '', d.media || '', d.poe_power ? `PoE ${d.poe_power.toFixed(1)} W` : ''].filter(Boolean).join(' · '), + detail: ` + <div class="link-stats-grid"> + ${stat('Duplex', d.full_duplex === null || d.full_duplex === undefined ? '–' : d.full_duplex ? 'Full' : 'Half')} + ${stat('Auto-neg', d.autoneg === null || d.autoneg === undefined ? '–' : d.autoneg ? 'On' : 'Off')} + ${stat('TX Err/s', fmtErrors(d.tx_errs_rate))} + ${stat('RX Err/s', fmtErrors(d.rx_errs_rate))} + ${stat('TX Drop/s', fmtErrors(d.tx_drops_rate))} + ${stat('RX Drop/s', fmtErrors(d.rx_drops_rate))} + ${d.poe_power || d.poe_max_power ? stat('PoE', `${esc(d.poe_class || '–')} · max ${fmtUnit(d.poe_max_power, 'W', 1)}`) : ''} + </div> + ${d.lldp ? `<div class="port-lldp">LLDP: ${esc(d.lldp.system_name || '')} (${esc(d.lldp.port_id || '')})${d.lldp.chassis_id ? ` · ${esc(d.lldp.chassis_id)}` : ''}${(d.lldp.mgmt_ips || []).length ? ` · ${esc(d.lldp.mgmt_ips.join(', '))}` : ''}</div>` : ''} + ${trafficBars(d)}`, + }; + }); +} + +// Down alone isn't a problem: most down ports are simply unused. +function rowHasProblem(r) { + return !!badges(r.data); +} + +function renderTable(rows) { + const q = _query.trim().toLowerCase(); + const shown = rows.filter(r => { + if (_filter === 'up' && !r.up) return false; + if (_filter === 'problems' && !rowHasProblem(r)) return false; + return !q || `${r.label} ${r.num || ''} ${r.sub} ${(r.neighbor && r.neighbor.name) || ''}`.toLowerCase().includes(q); + }); + if (!shown.length) return '<div class="lt-empty-state lt-empty-state--sm"><div class="lt-empty-state-title">Nothing matches this filter</div></div>'; + const isPorts = rows.some(r => r.num !== undefined); + const body = shown.map(r => { + const open = _openRows.has(r.id); + const d = r.data; + const neighbor = r.neighbor + ? (r.neighbor.name ? `<a class="device-link" href="${esc(deviceUrl(r.neighbor.name))}">${esc(r.neighbor.name)}</a>` : esc(r.neighbor.port || '')) + : '–'; + return ` + <tr class="device-row${r.up ? '' : ' device-row--down'}"> + <td><button type="button" class="device-row-toggle" data-row="${esc(r.id)}" aria-expanded="${open}" aria-label="Details for ${esc(r.label)}">${open ? '▾' : '▸'}</button></td> + <td><span class="dot-${r.up ? 'up' : 'down'}"></span></td> + <td class="device-row-name">${r.num !== undefined ? `<span class="port-badge port-badge-num">#${esc(r.num)}</span> ` : ''}${esc(r.label)}<div class="device-row-sub">${esc(r.sub)}</div></td> + <td class="device-num">${r.up ? fmtSpeed(d.speed_mbps) : '–'}</td> + <td class="device-num device-col-wide">${fmtRate(d.rx_bytes_rate)}</td> + <td class="device-num device-col-wide">${fmtRate(d.tx_bytes_rate)}</td> + ${isPorts ? `<td class="device-col-wide">${neighbor}</td>` : ''} + <td>${badges(d) || (r.up ? '<span class="val-good">OK</span>' : '<span class="device-row-sub">down</span>')}</td> + </tr> + ${open ? `<tr class="device-row-detail"><td colspan="${isPorts ? 8 : 7}">${r.detail}</td></tr>` : ''}`; + }).join(''); + return ` + <div class="lt-table-wrap"> + <table class="lt-table device-table"> + <caption class="lt-sr-only">Interfaces</caption> + <thead><tr> + <th><span class="lt-sr-only">Expand</span></th><th><span class="lt-sr-only">State</span></th> + <th>${isPorts ? 'Port' : 'Interface'}</th><th>Speed</th><th class="device-col-wide">RX</th><th class="device-col-wide">TX</th> + ${isPorts ? '<th class="device-col-wide">Neighbor</th>' : ''}<th>Health</th> + </tr></thead> + <tbody>${body}</tbody> + </table> + </div>`; +} + +function section(title, inner, id) { + return ` + <section class="g-section"${id ? ` id="${id}"` : ''}> + <div class="g-section-header"><h2 class="g-section-title">${esc(title)}</h2></div> + <div class="lt-frame"> + <span class="lt-frame-bl">╚</span><span class="lt-frame-br">╝</span> + ${inner} + </div> + </section>`; +} + +function kvGrid(pairs) { + return `<div class="modem-stats">${pairs.filter(([, v]) => v !== null && v !== undefined && v !== '').map(([k, v]) => ` + <div class="modem-stat"><span class="modem-stat-label">${esc(k)}</span><span class="modem-stat-value">${esc(v)}</span></div>`).join('')}</div>`; +} + +function renderOverview(d) { + const info = d.info || {}; + const pct = v => (v === null || v === undefined ? null : `${v.toFixed(0)}%`); + const temps = (info.temps || []).filter(t => t.value_c !== null).map(t => `${t.name} ${t.value_c.toFixed(0)}°C`).join(', '); + return section('Overview', kvGrid([ + ['Status', (d.status || 'unknown').toUpperCase()], + ['Type', d.kind], + ['IP', d.ip], + ['MAC', d.mac], + ['Model', d.model], + ['Firmware', info.firmware], + ['Uptime', info.uptime_s === undefined ? null : fmtUptime(info.uptime_s)], + ['CPU', pct(info.cpu_pct)], + ['Memory', pct(info.mem_pct)], + ['Clients', info.clients], + ['Temperature', temps], + ['Power draw', info.power_w === null || info.power_w === undefined ? null : `${info.power_w.toFixed(0)} W`], + ['Data source', d.source], + ])); +} + +function renderUplinks(d) { + if (!(d.uplinks || []).length) return ''; + const rows = d.uplinks.map(u => ` + <tr> + <td><span class="dot-${u.up ? 'up' : 'down'}"></span></td> + <td>${esc(u.host_interface || '–')}</td> + <td><a class="device-link" href="${esc(deviceUrl(u.switch))}">${esc(u.switch)}</a></td> + <td>#${esc(u.port_idx)} ${esc(u.port)}</td> + <td>${fmtSpeed(u.speed_mbps)}</td> + </tr>`).join(''); + return section('Connected to', `<div class="lt-table-wrap"><table class="lt-table"> + <caption class="lt-sr-only">Switch ports this host connects to, from LLDP</caption> + <thead><tr><th><span class="lt-sr-only">State</span></th><th>Interface</th><th>Switch</th><th>Port</th><th>Speed</th></tr></thead> + <tbody>${rows}</tbody></table></div>`); +} + +function renderWan(d) { + const wan = (d.info && d.info.wan) || {}; + const names = Object.keys(wan); + if (!names.length) return ''; + const rows = names.map(n => { + const w = wan[n]; + return `<tr><td><span class="dot-${w.up ? 'up' : 'down'}"></span></td><td>${esc(n.toUpperCase())}</td> + <td>${fmtSpeed(w.speed_mbps)}</td><td>${w.latency_ms === null || w.latency_ms === undefined ? '–' : `${esc(w.latency_ms)} ms`}</td> + <td>${w.availability === null || w.availability === undefined ? '–' : `${esc(w.availability)}%`}</td></tr>`; + }).join(''); + return section('WAN', `<div class="lt-table-wrap"><table class="lt-table"> + <caption class="lt-sr-only">WAN connections</caption> + <thead><tr><th><span class="lt-sr-only">State</span></th><th>Connection</th><th>Speed</th><th>Latency</th><th>Availability</th></tr></thead> + <tbody>${rows}</tbody></table></div>`); +} + +function renderOutlets(d) { + const outlets = (d.info && d.info.outlets) || []; + if (!outlets.length) return ''; + const rows = outlets.map(o => ` + <tr> + <td><span class="dot-${o.on ? 'up' : 'down'}"></span></td> + <td>#${esc(o.index)}</td> + <td>${esc(o.name || '–')}</td> + <td class="device-num">${o.power_w === null || o.power_w === undefined ? '–' : `${o.power_w.toFixed(1)} W`}</td> + <td>${o.auto_power_cycle ? '<span class="lt-badge">auto power-cycle</span>' : ''}</td> + </tr>`).join(''); + return section('Outlets', `<div class="lt-table-wrap"><table class="lt-table"> + <caption class="lt-sr-only">PDU outlets</caption> + <thead><tr><th><span class="lt-sr-only">State</span></th><th>Outlet</th><th>Name</th><th>Power</th><th></th></tr></thead> + <tbody>${rows}</tbody></table></div>`); +} + +function renderEvents(events) { + const body = !(events || []).length + ? '<div class="lt-empty-state lt-empty-state--sm"><div class="lt-empty-state-icon">✔</div><div class="lt-empty-state-title">No recorded alerts</div></div>' + : `<div class="lt-table-wrap"><table class="lt-table"> + <caption class="lt-sr-only">Alerts for this device</caption> + <thead><tr><th>Sev</th><th>Type</th><th>Detail</th><th>Description</th><th>First seen</th><th>State</th></tr></thead> + <tbody>${events.map(e => ` + <tr class="${e.resolved_at ? '' : `row-${esc(e.severity)}`}"> + <td><span class="lt-badge badge-${e.resolved_at ? 'resolved' : esc(e.severity)}">${esc(e.severity)}</span>${e.is_suppressed && !e.resolved_at ? '<span class="lt-badge badge-suppressed">🔕 sup</span>' : ''}</td> + <td>${esc(e.event_type.replace(/_/g, ' '))}</td> + <td>${esc(e.target_detail || '–')}</td> + <td class="desc-cell">${esc(e.description || '')}</td> + <td class="ts-cell" title="${esc(e.first_seen || '')}">${esc(lt.time.ago(_toIso(e.first_seen)))}</td> + <td>${e.resolved_at ? `resolved ${esc(lt.time.ago(_toIso(e.resolved_at)))}` : '<strong>active</strong>'}</td> + </tr>`).join('')}</tbody></table></div>`; + return section('Alerts', body); +} + +function currentRows() { + if (!_device) return []; + if (_device.ports) return portRows(_device.ports); + if (_device.interfaces) return hostRows(_device.interfaces); + return []; +} + +function renderInterfaces() { + const wrap = document.getElementById('device-table-wrap'); + if (wrap) wrap.innerHTML = renderTable(currentRows()); +} + +function renderInterfaceSection(rows, title) { + if (!rows.length) return ''; + const up = rows.filter(r => r.up).length; + const problems = rows.filter(rowHasProblem).length; + const pill = (value, label) => `<button type="button" class="pill${_filter === value ? ' active' : ''}" data-device-filter="${value}" aria-pressed="${_filter === value}">${label}</button>`; + return section(`${title} · ${up} of ${rows.length} up`, ` + <div class="lt-toolbar device-toolbar"> + <div class="lt-toolbar-left"> + <div class="lt-search"> + <input type="search" class="lt-input lt-search-input lt-search-input--sm" id="device-table-search" + placeholder="Filter…" autocomplete="off" aria-label="Filter ${esc(title.toLowerCase())}" value="${esc(_query)}"> + </div> + <div class="sev-pills" role="group" aria-label="Filter by state"> + ${pill('all', 'All')}${pill('up', 'Up')}${pill('problems', `Errors (${problems})`)} + </div> + </div> + </div> + <div id="device-table-wrap"></div>`); +} + +function render(d) { + _device = d; + document.getElementById('device-dot').className = `dot-${d.status || 'unknown'}`; + document.getElementById('device-subtitle').textContent = + [d.kind, d.ip, d.model].filter(Boolean).join(' · ') + + (d.updated ? ` · updated ${new Date(_toIso(d.updated)).toLocaleTimeString()}` : ''); + + const unifi = d.source === 'unifi'; + document.getElementById('device-actions').innerHTML = ` + ${d.ports && Object.keys(d.ports).length ? `<a class="lt-btn lt-btn-secondary lt-btn-sm" href="/inspector">⬡ Inspector</a>` : ''} + <button class="lt-btn lt-btn-ghost lt-btn-sm btn-suppress" data-sup-type="${unifi ? 'unifi_device' : 'host'}" + data-sup-name="${esc(d.name)}" data-sup-detail="" aria-label="Suppress alerts for ${esc(d.name)}">🔕 Suppress</button>`; + + // Keep focus and caret in the filter box across auto-refresh + const active = document.activeElement && document.activeElement.id === 'device-table-search'; + const caret = active ? document.activeElement.selectionStart : 0; + + const rows = currentRows(); + document.getElementById('device-root').innerHTML = + renderOverview(d) + + renderEvents(d.events) + + renderWan(d) + + renderUplinks(d) + + renderOutlets(d) + + renderInterfaceSection(rows, d.ports ? 'Ports' : 'Interfaces'); + renderInterfaces(); + + if (active) { + const box = document.getElementById('device-table-search'); + box.focus(); + box.setSelectionRange(caret, caret); + } + if (d.kind === 'modem') refreshModem(); +} + +async function loadDevice() { + try { + render(await lt.api.get(`/api/device/${encodeURIComponent(DEVICE_NAME)}`)); + } catch (err) { + if (_device) { + lt.toast.error('Could not refresh device data'); + return; + } + const notFound = /404|No device/.test(err.message || ''); + document.getElementById('device-root').innerHTML = ` + <div class="lt-empty-state"> + <div class="lt-empty-state-icon">✕</div> + <div class="lt-empty-state-title">${notFound ? 'No device with this name' : 'Could not load this device'}</div> + <div class="lt-empty-state-body">${notFound ? 'It may have been renamed, or the monitor may not have seen it yet.' : 'Check that the Gandalf service is healthy, then refresh.'}</div> + </div>`; + } +} + +document.addEventListener('click', e => { + const toggle = e.target.closest('.device-row-toggle'); + if (toggle) { + const id = toggle.dataset.row; + if (_openRows.has(id)) _openRows.delete(id); else _openRows.add(id); + renderInterfaces(); + return; + } + const pill = e.target.closest('[data-device-filter]'); + if (pill) { + _filter = pill.dataset.deviceFilter; + document.querySelectorAll('[data-device-filter]').forEach(p => { + const on = p === pill; + p.classList.toggle('active', on); + p.setAttribute('aria-pressed', String(on)); + }); + renderInterfaces(); + } +}); + +document.addEventListener('input', e => { + if (e.target.id === 'device-table-search') { + _query = e.target.value; + renderInterfaces(); + } +}); + +loadDevice(); +const _deviceInterval = window.gandalfSettings?.refreshInterval ?? 60; +if (_deviceInterval > 0) lt.autoRefresh.start(loadDevice, Math.max(_deviceInterval, 15) * 1000); + +window.onGandalfSettingsChanged = function (s) { + lt.autoRefresh.stop(); + if (s.refreshInterval > 0) lt.autoRefresh.start(loadDevice, Math.max(s.refreshInterval, 15) * 1000); +}; diff --git a/static/style.css b/static/style.css index 592a307..3c64009 100644 --- a/static/style.css +++ b/static/style.css @@ -1064,3 +1064,88 @@ .inspector-panel.open { width: 100%; } .inspector-panel-inner{ width: 100%; } } + +/* ── WAN modem / fiber panel ──────────────────────────────────────── */ +.modem-stats { + display: grid; + grid-template-columns: repeat(4, minmax(0, 1fr)); + gap: 12px; + padding: 12px; +} +@media (max-width: 700px) { + .modem-stats { grid-template-columns: repeat(2, minmax(0, 1fr)); } +} +.modem-stat { display: flex; flex-direction: column; gap: 2px; min-width: 0; } +.modem-stat-label { font-size: .68em; color: var(--text-muted); text-transform: uppercase; letter-spacing: .05em; } +.modem-stat-value { font-size: 1.05em; font-weight: bold; overflow-wrap: anywhere; } +.modem-stat-note { font-size: .7em; color: var(--text-muted); display: block; } +.modem-error { margin: 12px; padding: 8px 10px; border: 1px solid var(--terminal-red, #f44); color: var(--terminal-red, #f44); } +.modem-chart { padding: 0 12px 8px; } +.modem-chart-title { font-size: .72em; color: var(--text-muted); margin-bottom: 4px; } +.modem-chart-title span { margin-left: 8px; opacity: .8; } +.modem-chart svg { display: block; width: 100%; height: 70px; border: 1px solid var(--border-color); } +.modem-chart svg polyline { stroke: var(--terminal-cyan, #0ff); stroke-width: 1.5; } +.modem-chart-empty { padding: 0 12px 8px; font-size: .72em; color: var(--text-muted); } +.modem-updated { padding: 0 12px 10px; } + +/* ── Device search (home page) ────────────────────────────────────── */ +.device-search { position: relative; } +.device-search label { display: block; margin-bottom: 6px; } +.device-search .lt-search { width: 100%; } +.device-search-input { width: 100%; font-size: 1em; padding-top: 10px; padding-bottom: 10px; } +.device-search-results { + position: absolute; left: 0; right: 0; z-index: 50; margin-top: 4px; + background: var(--bg-card, var(--bg-secondary)); border: 1px solid var(--border-color); + max-height: 420px; overflow-y: auto; +} +.device-search-results[hidden] { display: none; } +.device-search-row { + display: grid; grid-template-columns: 14px minmax(0, 1.4fr) 70px minmax(0, 1fr) minmax(0, 1fr); + gap: 10px; align-items: center; padding: 9px 12px; color: inherit; text-decoration: none; + border-bottom: 1px solid var(--border-color); +} +.device-search-row:last-child { border-bottom: 0; } +.device-search-row:hover, .device-search-row.is-active { background: var(--bg-tertiary); } +.device-search-name { font-weight: bold; overflow-wrap: anywhere; } +.device-search-kind { font-size: .72em; color: var(--text-muted); text-transform: uppercase; } +.device-search-meta { font-size: .8em; color: var(--text-muted); overflow-wrap: anywhere; } +.device-search-empty { padding: 12px; color: var(--text-muted); } +@media (max-width: 700px) { + .device-search-row { grid-template-columns: 14px minmax(0, 1fr) auto; } + .device-search-row .device-search-meta { grid-column: 2 / -1; } +} + +/* ── Device page ───────────────────────────────────────────────────── */ +a.device-link { color: inherit; text-decoration: none; border-bottom: 1px dotted var(--text-muted); } +a.device-link:hover { color: var(--text-accent); border-bottom-color: currentColor; } +.device-title { display: flex; align-items: center; gap: 10px; overflow-wrap: anywhere; } +.device-actions { display: flex; gap: 8px; flex-wrap: wrap; align-items: center; } +.device-toolbar { padding: 8px 12px; margin: 0; } +.device-table td { vertical-align: middle; } +.device-num { white-space: nowrap; font-variant-numeric: tabular-nums; } +.device-row--down { opacity: .55; } +.device-row-name { overflow-wrap: anywhere; } +.device-row-sub { font-size: .72em; color: var(--text-muted); } +.device-row-toggle { background: none; border: 0; color: inherit; cursor: pointer; font: inherit; padding: 2px 6px; } +.device-row-detail > td { background: var(--bg-tertiary); padding: 10px 14px; } +.topo-host[data-host] { cursor: pointer; } + +/* WAN outage list in the modem panel */ +.modem-outages { padding: 0 12px 8px; } +.modem-outages ul { list-style: none; margin: 0; padding: 0; border: 1px solid var(--border-color); } +.modem-outages li { display: flex; justify-content: space-between; gap: 12px; padding: 4px 8px; font-size: .8em; } +.modem-outages li + li { border-top: 1px solid var(--border-color); } + +.g-section-actions[hidden] { display: none; } +/* Rows open a detail panel holding the fields that are hidden here on narrow screens */ +.device-row-detail .link-stats-grid { grid-template-columns: repeat(auto-fit, minmax(110px, 1fr)); } +.device-table th:nth-child(-n+2), .device-table td:nth-child(-n+2) { width: 1%; white-space: nowrap; } +@media (max-width: 700px) { + .device-col-wide { display: none; } + .device-table th, .device-table td { padding-left: 6px; padding-right: 6px; } +} +.device-search-row.is-active { box-shadow: inset 3px 0 0 var(--terminal-cyan, #0ff); } +@media (max-width: 700px) { + /* Scroll inside the wrapper rather than squeezing every column to a few letters */ + #events-table { min-width: 760px; } +} diff --git a/templates/_modem_panel.html b/templates/_modem_panel.html new file mode 100644 index 0000000..0455650 --- /dev/null +++ b/templates/_modem_panel.html @@ -0,0 +1,12 @@ +<section class="g-section" id="modem-section" hidden> + <div class="g-section-header"> + <h2 class="g-section-title">WAN / Fiber</h2> + <a class="g-section-actions lt-btn lt-btn-ghost lt-btn-sm" id="modem-device-link" hidden>↗ Details</a> + </div> + <div class="lt-frame"> + <span class="lt-frame-bl">╚</span> + <span class="lt-frame-br">╝</span> + <div class="lt-section-header" id="modem-title">AT&T Gateway</div> + <div id="modem-body" aria-live="polite"></div> + </div> +</section> diff --git a/templates/base.html b/templates/base.html index 9e3b700..9ae9b55 100644 --- a/templates/base.html +++ b/templates/base.html @@ -5,6 +5,7 @@ <meta name="viewport" content="width=device-width, initial-scale=1.0, viewport-fit=cover"> <meta name="theme-color" content="#030508"> <meta name="robots" content="noindex, nofollow"> + <link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 16 16'%3E%3Crect width='16' height='16' rx='3' fill='%23030508'/%3E%3Cpath d='M8 2l5 3v6l-5 3-5-3V5z' fill='none' stroke='%2300e5ff' stroke-width='1.5'/%3E%3C/svg%3E"> <title>{% block title %}GANDALF{% endblock %} — GANDALF @@ -36,9 +37,6 @@ Dashboard - Link Debug Inspector @@ -84,11 +82,6 @@ {% if request.endpoint == 'index' %}aria-current="page"{% endif %}> Dashboard - - Link Debug - @@ -186,7 +179,7 @@ | [ S ] SUPPRESS | - {% elif request.endpoint in ('links_page', 'inspector') %} + {% elif request.endpoint in ('device_page', 'inspector') %} [ R ] REFRESH | {% endif %} @@ -262,7 +255,8 @@ ShortcutAction Ctrl / ⌘ + KCommand palette - RRefresh data (Dashboard / Link Debug / Inspector) + RRefresh data (Dashboard / Device / Inspector) + /Search devices (Dashboard) SQuick-suppress alert (Dashboard) *Open settings ?Show this help @@ -330,7 +324,6 @@ // Command palette lt.cmdPalette.init([ { id: 'nav-dashboard', group: 'Navigate', icon: '~', label: 'Dashboard', action: function() { window.location.href = '/'; } }, - { id: 'nav-links', group: 'Navigate', icon: '↗', label: 'Link Debug', action: function() { window.location.href = '/links'; } }, { id: 'nav-inspector', group: 'Navigate', icon: '⬡', label: 'Inspector', action: function() { window.location.href = '/inspector'; } }, { id: 'nav-suppressions', group: 'Navigate', icon: '🔕', label: 'Suppressions', action: function() { window.location.href = '/suppressions'; } }, { id: 'action-refresh', group: 'Actions', icon: '↻', label: 'Refresh Data', kbd: 'R', action: function() { lt.autoRefresh.now(); } }, diff --git a/templates/device.html b/templates/device.html new file mode 100644 index 0000000..3d28d07 --- /dev/null +++ b/templates/device.html @@ -0,0 +1,25 @@ +{% extends "base.html" %} +{% block title %}{{ device_name }} – GANDALF{% endblock %} + +{% block content %} +
+
+

← Dashboard

+

+ + {{ device_name }} +

+

+
+
+
+ +
Loading device
+ +{% include "_modem_panel.html" %} +{% endblock %} + +{% block scripts %} + + +{% endblock %} diff --git a/templates/index.html b/templates/index.html index 31ec3aa..d00aeb8 100644 --- a/templates/index.html +++ b/templates/index.html @@ -64,6 +64,17 @@ + + +
@@ -166,10 +177,129 @@
- + +{% include "_modem_panel.html" %} + +
-

Network Hosts

+

Hosts

+
+
+ {%- set has_global_sup = suppressions | selectattr('target_type', 'equalto', 'all') | list | length > 0 -%} + {% for name, host in snapshot.hosts.items() %} + {% set suppressed = suppressions | selectattr('target_name', 'equalto', name) | list %} +
+
+
+ + {{ name }} + {% if suppressed or has_global_sup %} + 🔕 + {% endif %} +
+
+ {{ host.ip }} + {{ host.source }} +
+
+ + {% if host.interfaces %} +
+ {% for iface, state in host.interfaces.items() | sort %} +
+ + {{ iface }} + {{ state }} +
+ {% endfor %} +
+ {% else %} +
{% if host.source == 'ping' %}ping-only / no node_exporter{% else %}no metrics from node_exporter{% endif %}
+ {% endif %} + +
+ + + ↗ Details + +
+
+ {% else %} +
+
⌛
+
No host data yet
+
The monitor daemon may still be starting up.
+
+ {% endfor %} +
+
+ + +{% if snapshot.unifi %} +
+
+

UniFi Devices

+
+
+ ╚ + ╝ +
Device Inventory
+
+ + + + + + + + + + + + + + {% for d in snapshot.unifi %} + + + + + + + + + {% endfor %} + +
UniFi network devices
StatusNameTypeModelIPActions
+ + {{ 'ONLINE' if d.connected else 'OFFLINE' }} + {{ d.name }}{{ d.type }}{{ d.model }}{{ d.ip }} + {% if not d.connected %} + + {% endif %} +
+
+
+
+{% endif %} + + +
+
+

Network Topology

@@ -191,10 +321,28 @@ - +
- WAN · 10G SFP+ + AT&T Fiber +
+ + +
+
+ ◈ + BGW320 + AT&T Gateway + Rx – +
+
+ + +
+
+ WAN · Port 9 · 1G
- -
-
-
- -
-
-
-
- {%- set has_global_sup = suppressions | selectattr('target_type', 'equalto', 'all') | list | length > 0 -%} - {% for name, host in snapshot.hosts.items() %} - {% set suppressed = suppressions | selectattr('target_name', 'equalto', name) | list %} -
-
-
- - {{ name }} - {% if suppressed or has_global_sup %} - 🔕 - {% endif %} -
-
- {{ host.ip }} - {{ host.source }} -
-
- - {% if host.interfaces %} -
- {% for iface, state in host.interfaces.items() | sort %} -
- - {{ iface }} - {{ state }} -
- {% endfor %} -
- {% else %} -
ping-only / no node_exporter
- {% endif %} - -
- - - ↗ Links - -
-
- {% else %} -
-
⌛
-
No host data yet
-
The monitor daemon may still be starting up.
-
- {% endfor %} -
- -{% if snapshot.unifi %} -
-
-

UniFi Devices

-
-
- ╚ - ╝ -
Device Inventory
-
- - - - - - - - - - - - - - {% for d in snapshot.unifi %} - - - - - - - - - {% endfor %} - -
UniFi network devices
StatusNameTypeModelIPActions
- - {{ 'ONLINE' if d.connected else 'OFFLINE' }} - {{ d.name }}{{ d.type }}{{ d.model }}{{ d.ip }} - {% if not d.connected %} - - {% endif %} -
-
-
-
-{% endif %} {% if recent_resolved %} @@ -468,6 +499,9 @@ {% block scripts %} -{% endblock %} diff --git a/tests/test_app.py b/tests/test_app.py new file mode 100644 index 0000000..e6f24c0 --- /dev/null +++ b/tests/test_app.py @@ -0,0 +1,93 @@ +"""Tests for app.py routes: database and LDAP are mocked.""" +import json +import os +import sys +from unittest.mock import patch + +import pytest + +sys.path.insert(0, os.path.dirname(os.path.dirname(__file__))) + +import app as gandalf # noqa: E402 + +AUTH = {'Remote-User': 'jared', 'Remote-Name': 'Jared V', 'Remote-Groups': 'admin'} +STATE = { + 'network_snapshot': json.dumps({ + 'hosts': {'large1': {'ip': '10.0.0.2', 'status': 'up', 'source': 'prometheus', 'interfaces': {'eth0': 'up'}}}, + 'unifi': [{'name': 'USW Flex', 'type': 'usw', 'model': 'USF5P', 'ip': '10.0.0.3', 'mac': '', + 'connected': True}], + }), + 'link_stats': json.dumps({'hosts': {}, 'unifi_switches': {}}), +} + + +@pytest.fixture +def client(): + with patch.object(gandalf, 'db') as db: + db.get_state.side_effect = lambda key, default=None: STATE.get(key, default) + db.get_active_events.return_value = [] + db.get_recent_resolved.return_value = [] + db.get_active_suppressions.return_value = [] + db.get_events_for_target.return_value = [] + db.count_active_events.return_value = 0 + db.get_status_summary.return_value = {'critical': 0, 'warning': 0, 'info': 0, 'resolved_24h': 0} + db.check_suppressed.return_value = False + with patch.object(gandalf, '_cfg', {'auth': {'allowed_groups': ['admin']}, 'ticket_api': {}}): + yield gandalf.app.test_client() + + +class TestPagesAndNav: + def test_home_has_search_and_no_link_debug(self, client): + html = client.get('/', headers=AUTH).get_data(as_text=True) + assert 'id="device-search-input"' in html + assert 'Link Debug' not in html + + def test_home_section_order(self, client): + html = client.get('/', headers=AUTH).get_data(as_text=True) + order = [html.index(m) for m in ( + 'id="device-search"', 'Alert Queue', 'WAN / Fiber', 'g-section-title">Hosts<', + 'Device Inventory', 'Network Topology', + )] + assert order == sorted(order) + + def test_old_links_url_redirects_home(self, client): + resp = client.get('/links', headers=AUTH) + assert resp.status_code == 302 and resp.headers['Location'].endswith('/') + + def test_device_page_escapes_name(self, client): + html = client.get('/device/%3Cscript%3Ex', headers=AUTH).get_data(as_text=True) + assert '