Add AT&T modem monitoring and device pages; replace Link Debug
Lint / Python (flake8) (pull_request) Successful in 2m6s
Lint / JS (eslint) (pull_request) Successful in 18s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 2m11s
Test / Python Tests (pytest) (pull_request) Successful in 2m20s
Lint / JS (eslint) (push) Successful in 10s
Lint / Python (flake8) (push) Successful in 1m58s
Lint / Notify on failure (push) Skipped
Security / Python Security (bandit) (push) Successful in 1m58s
Lint / Deploy (push) Successful in 4s
Test / Python Tests (pytest) (push) Successful in 58s

Modem and WAN:
- Poll the BGW320 status pages read-only for light levels, PON and WAN
  state, with a WAN / Fiber panel, 7-day Rx trend, and alerts on WAN down,
  optical limits, and Rx drift
- Read WAN outages and Smart Power restarts from the UniFi system log,
  since outages can be over between 5 minute polls

Device pages:
- Add device search to the dashboard and a page per host, switch, router,
  AP, PDU, or modem
- Move all Link Debug data into expandable rows there; /links redirects
- Collect UDM ports and UniFi device details (firmware, load, WAN, outlets)
- Reorder the dashboard

Monitoring fixes:
- Alert when a configured host stops reporting: its series disappear
  rather than reporting down, so a dead host never raised an alert
- Alert on Prometheus outages instead of blanking the dashboard
- Treat a failed Pulse ping as unknown rather than host unreachable
- Stop new switch ports reporting lifetime bytes as one interval's rate
- Run database sessions in UTC: other server time zones shifted displayed
  ages and timed suppression expiries

Also log every avatar 404 reason, add a favicon, and correct the WAN link
labels (1G on Port 9, not 10G SFP+).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit was merged in pull request #4.
This commit is contained in:
2026-11-04 23:40:17 -05:00
co-authored by Claude Opus 5.5
parent e1716d5228
commit e32499b7f6
25 changed files with 3120 additions and 821 deletions
+92 -37
View File
@@ -27,17 +27,7 @@ A Proxmox/Ceph cluster lives or dies by its network, and network problems are sn
![Host cards](docs/img/host-cards.png)
**Link debug**: per-interface speed, duplex, carrier flaps, error and drop rates, and **SFP optical readings** (temperature, TX/RX power) with colour-coded thresholds. Here a fibre with weak receive power is flagged while its neighbour is healthy.
![Link debug](docs/img/link-debug.png)
| Weak receive power on a fibre link | Healthy fibre link |
|---|---|
| ![SFP warning](docs/img/link-debug-sfp-warning.png) | ![SFP healthy](docs/img/link-debug-sfp.png) |
Switch ports are shown too, with PoE draw and the LLDP neighbour on each port:
![Switch ports](docs/img/link-debug-switch.png)
**Device pages**: search by hostname, IP, MAC, model or interface from the dashboard, then open one page per host, switch, router, PDU or modem. Each interface or port is one row with its state, speed, traffic and neighbour. Expanding a row shows duplex, carrier flaps, error and drop rates, PoE, LLDP, and **SFP optical readings** (temperature, TX/RX power) with colour-coded thresholds.
**Network inspector**: chassis diagrams for each UniFi switch. Click a port for its stats, LLDP neighbour and a side-by-side switch/server path check.
@@ -83,6 +73,7 @@ Two processes share a MariaDB database:
[UniFi Controller] ──▶ monitor.py ──▶ MariaDB ◀── app.py ──▶ nginx ──▶ Authelia ──▶ Browser
[Pulse Worker] ──▶
[SSH / ethtool] ──▶
[AT&T BGW320] ──▶
```
### Data Sources
@@ -93,6 +84,7 @@ Two processes share a MariaDB database:
| **UniFi API** (`https://10.10.10.1`) | Switch port stats, device status, LLDP neighbor table, PoE data |
| **Pulse Worker** | SSH relay — runs `ethtool` + SFP DOM queries on each Proxmox host |
| **Ping** | Reachability for hosts without `node_exporter` (e.g. PBS) |
| **AT&T BGW320** (`http://192.168.1.254`) | Fiber light levels, PON and WAN state, WAN counters, scraped read-only from the gateway's status pages (no login) |
### Monitored Hosts (Prometheus / node_exporter)
@@ -112,31 +104,35 @@ Ping-only (no node_exporter): **pbs** (10.10.10.3)
## Pages
### Dashboard (`/`)
- Device search (see Device pages below)
- Active alerts first, then the WAN / Fiber panel, hosts, UniFi devices, topology, and recently resolved alerts
- Real-time host status grid with per-NIC link state (UP / DOWN / degraded)
- Network topology diagram (Internet → Gateway → Switches → Hosts)
- Network topology diagram (Internet → Modem → Gateway → Switches → Hosts)
- WAN / Fiber panel: WAN and PON state, Rx/Tx power, optics temperature, throughput, errors, gateway uptime and last restart, 7-day Rx trend, recent WAN outages
- UniFi device table (switches, APs, gateway)
- Active alerts table with severity, target, consecutive failures, ticket link
- Quick-suppress modal: apply timed or manual suppression from any alert row
- Auto-refreshes every 30 seconds via `/api/status` + `/api/network`
### Link Debug (`/links`)
Per-interface statistics collected every poll cycle. All panels are collapsible
(click header or use Collapse All / Expand All). Collapse state persists across
page refreshes via `sessionStorage`.
### Device pages (`/device/<name>`)
One page per host, switch, router, AP, PDU, or modem. Reach them from the search box on the
dashboard (`/` focuses it), or from any device name. Search matches hostname, IP, MAC, model, type,
interface and port names, and LLDP neighbors.
**Server NICs** (via Prometheus + SSH/ethtool):
- Speed, duplex, auto-negotiation, link detected
- TX/RX rate bars (bandwidth utilisation % of link capacity)
- TX/RX error and drop rates per second
- Carrier changes (cumulative since boot — watch for flapping)
- **SFP / Optical panel** (when SFP module present): vendor/PN, temp, voltage,
bias current, TX power (dBm), RX power (dBm), RX−TX delta, per-stat bars
- **Overview**: status, IP, MAC, model, firmware, uptime, CPU, memory, clients, temperatures, power draw
- **Alerts**: active and past alerts for this device
- **WAN** (routers): connection state, speed, latency, availability
- **Connected to** (hosts): switch ports that see this host over LLDP
- **Outlets** (PDUs): per-outlet state, power, and auto power-cycle setting
- **Interfaces / Ports**: one row each with state, speed, RX/TX rate, and neighbor, filterable.
Expanding a row shows duplex, auto-negotiation, carrier changes, error and drop rates,
traffic bars, PoE, LLDP detail, and the **SFP / optical panel** (vendor/PN, temperature, voltage,
bias, TX/RX power, RX−TX delta)
- **WAN / Fiber** (modem): the same panel as the dashboard
**UniFi Switch Ports** (via UniFi API):
- Port number badge (`#N`), UPLINK badge, PoE draw badge
- LLDP neighbor line: `→ system_name (port_id)` when neighbor is detected
- PoE class and max wattage line
- Speed, duplex, auto-neg, TX/RX rates, errors, drops
These replaced the Link Debug page; `/links` now redirects to the dashboard. Hosts that Prometheus
sees but `config.json` doesn't list still get a page, with their state shown as unknown because
they aren't alerted on.
### Inspector (`/inspector`)
Visual switch chassis diagrams. Each switch is rendered model-accurately using
@@ -174,6 +170,7 @@ layout config in the template (`SWITCH_LAYOUTS`).
| `USL8A` | UniFi Switch Lite 8 PoE | 8×SFP (2 rows of 4) |
| `US24PRO` | UniFi Switch Pro 24 | 24×RJ45 staggered + 2×SFP |
| `USPPDUP` | Custom/other | Single-port fallback |
| `UDMPRO` | Dream Machine Pro | 9×RJ45 + 2×SFP+ |
| `USMINI` | UniFi Switch Mini | 5-port row |
Add new layouts by adding a key to `SWITCH_LAYOUTS` matching the `model` field
@@ -197,7 +194,32 @@ returned by the UniFi API for that device.
| UniFi device offline (≥`unifi_failure_threshold` consecutive checks) | P2 High |
| Proxmox host NIC link-down regression (≥2 consecutive checks) | P2 High |
| Host unreachable via ping (≥2 consecutive checks) | P2 High |
| ≥3 hosts simultaneously reporting interface failures | P1 Critical |
| Configured host has no metrics in Prometheus (host or `node_exporter` down, ≥2 consecutive checks) | P2 High |
| Prometheus unavailable, so host and interface alerting is paused (≥2 consecutive checks) | P2 High |
| ≥3 hosts simultaneously reporting interface or metrics failures | P1 Critical |
| WAN down: broadband or optical status not Up, or PON not in O5 (≥`modem.failure_threshold`) | P1 Critical |
| Modem not responding (≥`modem.failure_threshold`) | P2 High |
| Fiber Rx/Tx power or optics temperature in the module's alarm range, or a supply voltage or Tx bias alarm flag (≥`modem.failure_threshold`) | P2 High |
| Fiber reading in the module's warning range (≥`modem.failure_threshold`) | P3 |
| Rx power `rx_drift_db` or more below its 7-day median (≥`modem.failure_threshold`) | P3 |
| Modem pages unreadable, e.g. after a firmware update (≥`modem.failure_threshold`) | P3 |
| UniFi logs that Smart Power auto power-cycled the modem | P3 |
| UniFi logs that the WAN failed multiple times in 24 hours (at most one ticket a day) | P2 |
A failed Pulse ping check counts as "unknown": it neither raises nor resolves `host_unreachable`.
### WAN modem notes
The alarm and warning limits are read from the gateway on every poll, so they are never hard-coded here.
Those limits are wide (warning at -27.9 dBm on a link that normally sits near -17), so `fiber_rx_drift`
compares each reading to the link's own 7-day median to catch a damaged connector earlier. It stays quiet
until `rx_drift_min_samples` readings exist (about a day at a 5 minute poll).
Outages can be over between polls: two real ones lasted about two minutes against a 5 minute poll.
So WAN outages and modem restarts also come from the UniFi system log, and a drop in the modem's uptime
records a restart even if nothing logged it. A single short outage only appears in the panel's history.
To silence modem alerts during maintenance, create a `host` suppression with the modem's `name`.
### Baseline Tracking
@@ -269,6 +291,10 @@ Shared by both processes. Located in the working directory (`/var/www/html/prod/
"ping_hosts": [
{ "name": "pbs", "ip": "10.10.10.3" }
]
},
"modem": {
"url": "http://192.168.1.254",
"name": "att-modem"
}
}
```
@@ -293,6 +319,12 @@ Shared by both processes. Located in the working directory (`/var/www/html/prod/
| `monitor.unifi_failure_threshold` | Consecutive failures before ticketing a UniFi device as offline (default: same as `failure_threshold`). Set higher than `failure_threshold` — UniFi switches/APs can take several minutes to rejoin after a firmware-update reboot, so a low threshold cuts tickets for momentary maintenance reboots instead of real failures. |
| `monitor.cluster_threshold` | Hosts with failures to trigger cluster-wide P1 (default: 3) |
| `monitor.ping_hosts` | Hosts checked only by ping (no node_exporter) |
| `modem.url` | AT&T gateway base URL. Omit the `modem` block to disable modem monitoring. Must be reachable from the monitor host. |
| `modem.name` | Name used for the modem's alerts, tickets, and suppressions (default: `att-modem`) |
| `modem.timeout` | Seconds to wait per status page (default: 10) |
| `modem.failure_threshold` | Consecutive failures before ticketing a modem alert (default: same as `failure_threshold`). Raise it if gateway reboots or automatic firmware updates cut tickets. |
| `modem.rx_drift_db` | Rx drop below the 7-day median that raises `fiber_rx_drift` (default: 3.0) |
| `modem.rx_drift_min_samples` | Readings required before drift alerts start (default: 288) |
---
@@ -409,17 +441,28 @@ systemctl restart gandalf gandalf-monitor
## Troubleshooting
### Avatar photo missing (`/api/avatar` returns 404)
`journalctl -u gandalf | grep -i avatar` gives the reason: `no ldap.host / ldap.bind_dn` (missing config),
`ldap3 not installed` (run `pip3 install -r requirements.txt`), `LDAP avatar lookup failed` (bind or network),
`LDAP has no photo` (or `Remote-User` doesn't match a uid), or a cached `no photo` marker, which expires
after `ldap.cache_ttl` seconds or can be removed from `ldap.cache_dir`.
### Monitor not creating tickets
- Verify `config.json → ticket_api.api_key` is set and valid
- Check `journalctl -u gandalf-monitor` for `Ticket creation failed` lines
- Confirm the Tinker Tickets API is reachable from LXC 157
### Link Debug shows no data / "Loading…" forever
### WAN / Fiber panel missing or showing "unavailable"
- The panel stays hidden until `config.json → modem.url` is set and the monitor has completed a cycle
- Test from the monitor host: `curl -s -o /dev/null -w '%{http_code}\n' http://192.168.1.254/cgi-bin/fiberstat.ha` should print `200`
- `modem_data_invalid` means the gateway answered but its pages couldn't be parsed. AT&T updates firmware automatically, so compare the page against the parsers in `modem.py` (markup confirmed on BGW320-505 firmware 6.35.8)
### Device pages show no interface data
- Check `gandalf-monitor.service` is running and has completed at least one cycle
- Check `journalctl -u gandalf-monitor` for Prometheus or UniFi errors
- Verify Prometheus is reachable: `curl http://10.10.10.48:9090/api/v1/query?query=up`
### Link Debug: SFP DOM panel missing
### Device page: SFP / optical panel missing
- SFP data requires Pulse worker + SSH access to hosts
- Verify `config.json → pulse.*` is configured and the Pulse worker is running
- Confirm `sshpass` + SSH access from the Pulse worker to each Proxmox host
@@ -467,17 +510,21 @@ curl http://10.10.10.48:9090/api/v1/query?query=up | jq '.data.result[] | select
gandalf/
├── app.py # Flask web app (routes, auth, API endpoints)
├── monitor.py # Background daemon (Prometheus, UniFi, Pulse, alert logic)
├── modem.py # Read-only AT&T gateway status page client and parser
├── devices.py # Device inventory for search and device pages
├── db.py # Database operations (MariaDB via pymysql, thread-local conn reuse)
├── schema.sql # Database schema (network_events, suppression_rules, monitor_state)
├── config.json # Runtime configuration (not committed with secrets)
├── requirements.txt # Python dependencies
├── static/
│ ├── style.css # Terminal aesthetic CSS (CRT scanlines, green-on-black)
│ └── app.js # Dashboard JS (auto-refresh, host grid, events, suppress modal)
│ ├── app.js # Dashboard JS (auto-refresh, events, suppress modal, search, modem panel)
│ └── device.js # Device page rendering
└── templates/
├── base.html # Shared layout (header, nav, footer)
├── index.html # Dashboard page
├── links.html # Link Debug page (server NICs + UniFi switch ports)
├── device.html # Per-device page
├── _modem_panel.html # WAN / Fiber panel, shared by the dashboard and modem page
├── inspector.html # Visual switch inspector + LLDP path debug
└── suppressions.html # Suppression management page
```
@@ -511,7 +558,10 @@ under `unifi_switches.<switch_name>.model`), then add to `SWITCH_LAYOUTS` in
- `network_events`: one row per active event; `resolved_at` is set when recovered
- `suppression_rules`: `active=FALSE` when removed; `expires_at` checked at query time
- `monitor_state`: key/value store; `interface_baseline` and `link_stats` are JSON blobs
- `monitor_state`: key/value store; `interface_baseline`, `link_stats`, and `modem_stats` are JSON blobs
- `monitor_state` also holds `wan_history` (WAN outages, modem restarts) and `unifi_log_cursor` (how far the UniFi log has been read)
- Database sessions run in UTC (`db.py`), whatever the server's time zone
- `modem_samples`: optical readings per poll, kept 90 days; created automatically at monitor startup
### Security Notes
@@ -547,11 +597,16 @@ under `unifi_switches.<switch_name>.model`), then add to `SWITCH_LAYOUTS` in
|---|---|---|
| `lint.yml` (python-lint) | flake8 on all `.py` files | Every push and PR |
| `lint.yml` (js-lint) | ESLint on `static/` | Every push and PR |
| `test.yml` | pytest — 33 tests for `diagnose.py` static methods | Every push and PR |
| `test.yml` | pytest — `diagnose.py`, `modem.py`, and `monitor.py` alert logic | Every push and PR |
| `security.yml` | bandit `-ll` (medium+ severity) | Every push, PR, and weekly Monday 6am |
| `deploy` job in `lint.yml` | Calls the `gandalf-deploy` webhook on CT157 (10.10.10.61) | Push to `main` only, after both lint jobs pass |
Branch protection is enabled on `main` — both lint jobs must pass before any PR can merge.
Tests live in `tests/test_diagnose.py` and cover `DiagnosticsRunner` static methods:
`build_ssh_command`, `parse_output`, `parse_sysfs_stats`, `parse_ethtool`, and variants.
Tests live in `tests/`:
- `test_diagnose.py`: `DiagnosticsRunner` static methods (`build_ssh_command`, `parse_output`,
`parse_sysfs_stats`, `parse_ethtool`, and variants)
- `test_modem.py`: gateway page parsing, optical status, and WAN state
- `test_monitor.py`: alert logic for dead hosts, Prometheus and Pulse outages, and modem and WAN log events
- `test_devices.py`: device inventory behind search and device pages
- `test_app.py`: page and API routes
+79 -3
View File
@@ -18,9 +18,10 @@ import uuid
from datetime import datetime, timezone
from functools import wraps
from flask import Flask, jsonify, make_response, render_template, request, send_file
from flask import Flask, jsonify, make_response, redirect, render_template, request, send_file, url_for
import db
import devices
import diagnose
from monitor import PulseClient
@@ -196,6 +197,18 @@ def _annotate_suppressions(events: list, suppressions: list) -> None:
ev['is_suppressed'] = db.check_suppressed(suppressions, 'host', name, detail)
def _state_json(key: str) -> dict:
"""Return a monitor_state JSON blob as a dict, or {} if missing or unreadable."""
raw = db.get_state(key)
if not raw:
return {}
try:
return json.loads(raw)
except Exception as e:
logger.error(f'Failed to parse {key} JSON: {e}')
return {}
# ---------------------------------------------------------------------------
# Page routes
# ---------------------------------------------------------------------------
@@ -235,8 +248,14 @@ def index():
@app.route('/links')
@require_auth
def links_page():
user = _get_user()
return render_template('links.html', user=user)
"""Link Debug moved to the per-device pages; keep old bookmarks working."""
return redirect(url_for('index'))
@app.route('/device/<path:name>')
@require_auth
def device_page(name: str):
return render_template('device.html', user=_get_user(), device_name=name)
@app.route('/inspector')
@@ -315,6 +334,60 @@ def api_links():
return jsonify({'hosts': {}, 'unifi_switches': {}, 'updated': None})
@app.route('/api/modem')
@require_auth
def api_modem():
"""Latest WAN modem status plus optical history for the trend chart."""
try:
hours = max(1, min(int(request.args.get('hours', 168)), 2160))
except ValueError:
return jsonify({'error': 'hours must be an integer'}), 400
raw = db.get_state('modem_stats')
if not raw:
return jsonify({'configured': False})
try:
data = json.loads(raw)
except Exception as e:
logger.error(f'Failed to parse modem_stats JSON: {e}')
return jsonify({'configured': False})
# A history failure shouldn't take the current readings down with it.
try:
# Keep roughly 200 points regardless of range; never finer than the 5 min poll.
data['history'] = db.get_modem_history(hours, max(300, hours * 3600 // 200))
except Exception as e:
logger.error(f'Failed to load modem history: {e}')
data['history'] = []
data['wan_history'] = _state_json('wan_history')
data['configured'] = True
return jsonify(data)
@app.route('/api/devices')
@require_auth
def api_devices():
"""Device list for the home page search."""
return jsonify({'devices': devices.build_index(
_state_json('network_snapshot'), _state_json('link_stats'), _state_json('modem_stats'),
)})
@app.route('/api/device/<path:name>')
@require_auth
def api_device(name: str):
"""Everything known about one device, for its detail page."""
detail = devices.device_detail(
name, _state_json('network_snapshot'), _state_json('link_stats'), _state_json('modem_stats'),
)
if detail is None:
return jsonify({'error': f'No device named "{name}"'}), 404
events = db.get_events_for_target(name)
_annotate_suppressions(events, db.get_active_suppressions())
detail['events'] = events
return jsonify(detail)
@app.route('/api/events')
@require_auth
def api_events():
@@ -559,6 +632,7 @@ def api_avatar():
ldap_cfg = _config().get('ldap', {})
if not ldap_cfg.get('host') or not ldap_cfg.get('bind_dn'):
logger.warning('Avatar unavailable: config.json has no ldap.host / ldap.bind_dn')
return '', 404
# Build a safe cache filename from the username (alphanumeric + - _ .)
@@ -589,6 +663,7 @@ def api_avatar():
# Skip LDAP if we already know this user has no avatar
try:
if os.path.exists(sentinel) and now - os.path.getmtime(sentinel) < cache_ttl:
logger.info(f'Avatar for {username!r}: cached "no photo" marker is still valid')
return '', 404
except OSError:
pass
@@ -627,6 +702,7 @@ def api_avatar():
pass
if not avatar_data or len(avatar_data) < 100:
logger.warning(f'Avatar for {username!r}: LDAP has no photo (or no user with that uid)')
with open(sentinel, 'w'):
pass
return '', 404
+95
View File
@@ -34,6 +34,9 @@ def _new_conn(cfg: dict):
cursorclass=pymysql.cursors.DictCursor,
connect_timeout=10,
charset='utf8mb4',
# All app code treats database times as UTC. Without this, a server in another
# time zone shifts displayed ages and timed suppression expiries by its offset.
init_command="SET time_zone='+00:00'",
)
@@ -145,6 +148,24 @@ def upsert_event(
return cur.lastrowid, True, 1, None
def get_events_for_target(target_name: str, limit: int = 25) -> list:
"""Return a device's events, unresolved first, then most recent."""
with get_conn() as conn:
with conn.cursor() as cur:
cur.execute(
"""SELECT * FROM network_events WHERE target_name=%s
ORDER BY resolved_at IS NOT NULL, COALESCE(resolved_at, last_seen) DESC
LIMIT %s""",
(target_name, limit),
)
rows = cur.fetchall()
for r in rows:
for k in ('first_seen', 'last_seen', 'resolved_at'):
if r.get(k) and hasattr(r[k], 'isoformat'):
r[k] = r[k].isoformat() + 'Z'
return rows
def resolve_event(event_type: str, target_name: str, target_detail: str = '') -> None:
detail = target_detail or ''
with get_conn() as conn:
@@ -332,6 +353,80 @@ def purge_old_resolved_events(days: int = 90) -> int:
return n
# ---------------------------------------------------------------------------
# WAN modem optical history
# ---------------------------------------------------------------------------
def ensure_modem_schema() -> None:
"""Create the modem history table so deploys need no manual migration."""
with get_conn() as conn:
with conn.cursor() as cur:
cur.execute(
"""CREATE TABLE IF NOT EXISTS modem_samples (
sampled_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP PRIMARY KEY,
rx_power_dbm DECIMAL(5,1) NULL,
tx_power_dbm DECIMAL(5,1) NULL,
temp_c DECIMAL(5,1) NULL
) ENGINE=InnoDB"""
)
def record_modem_sample(rx_power_dbm, tx_power_dbm, temp_c) -> None:
with get_conn() as conn:
with conn.cursor() as cur:
cur.execute(
"""INSERT INTO modem_samples (sampled_at, rx_power_dbm, tx_power_dbm, temp_c)
VALUES (NOW(), %s, %s, %s)
ON DUPLICATE KEY UPDATE rx_power_dbm=VALUES(rx_power_dbm),
tx_power_dbm=VALUES(tx_power_dbm), temp_c=VALUES(temp_c)""",
(rx_power_dbm, tx_power_dbm, temp_c),
)
def get_modem_rx_values(days: int = 7) -> list:
"""Return recent Rx power readings (dBm) for computing a drift baseline."""
with get_conn() as conn:
with conn.cursor() as cur:
cur.execute(
"""SELECT rx_power_dbm FROM modem_samples
WHERE rx_power_dbm IS NOT NULL
AND sampled_at > DATE_SUB(NOW(), INTERVAL %s DAY)""",
(days,),
)
return [float(r['rx_power_dbm']) for r in cur.fetchall()]
def get_modem_history(hours: int = 168, bucket_seconds: int = 3600) -> list:
"""Return optical history averaged into time buckets, oldest first."""
with get_conn() as conn:
with conn.cursor() as cur:
cur.execute(
"""SELECT FLOOR(UNIX_TIMESTAMP(sampled_at) / %s) * %s AS ts,
ROUND(AVG(rx_power_dbm), 1) AS rx_avg,
MIN(rx_power_dbm) AS rx_min,
ROUND(AVG(tx_power_dbm), 1) AS tx_avg,
ROUND(AVG(temp_c), 1) AS temp_avg
FROM modem_samples
WHERE sampled_at > DATE_SUB(NOW(), INTERVAL %s HOUR)
GROUP BY ts ORDER BY ts""",
(bucket_seconds, bucket_seconds, hours),
)
return [
{k: (float(v) if v is not None else None) for k, v in row.items()}
for row in cur.fetchall()
]
def purge_old_modem_samples(days: int = 90) -> int:
with get_conn() as conn:
with conn.cursor() as cur:
cur.execute(
'DELETE FROM modem_samples WHERE sampled_at < DATE_SUB(NOW(), INTERVAL %s DAY)',
(days,),
)
return cur.rowcount
def check_suppressed(suppressions: list, target_type: str, target_name: str, target_detail: str = '') -> bool:
"""Check suppression against a pre-loaded list (avoids per-call DB queries)."""
for s in suppressions:
+153
View File
@@ -0,0 +1,153 @@
"""Device inventory built from the monitor's cached state.
Pure functions with no I/O: app.py passes in the JSON blobs the monitor stored,
so search and the per-device pages always agree about what a device is.
"""
import re
from typing import Dict, List, Optional
# UniFi reports PDUs as switches or APs, so model codes take priority over type.
_PDU_MODELS = {'USPPDUP', 'UP6', 'USPDA2B'}
_KIND_BY_TYPE = {
'usw': 'switch', 'udm': 'router', 'ugw': 'router', 'uxg': 'router',
'uap': 'ap', 'usp': 'pdu',
}
def unifi_kind(unifi_type: str, model: str) -> str:
if model in _PDU_MODELS:
return 'pdu'
return _KIND_BY_TYPE.get((unifi_type or '').lower(), 'device')
def _mac_forms(mac: str) -> List[str]:
"""Return a MAC with and without separators, so either spelling is searchable."""
return [mac, re.sub(r'[^0-9a-fA-F]', '', mac)] if mac else []
def _search_text(*parts) -> str:
flat = []
for p in parts:
if isinstance(p, (list, tuple, set)):
flat.extend(str(x) for x in p if x)
elif p:
flat.append(str(p))
return ' '.join(flat).lower()
def _port_neighbors(ports: dict) -> List[str]:
return [p['lldp']['system_name'] for p in ports.values()
if (p.get('lldp') or {}).get('system_name')]
def build_index(snapshot: dict, link_stats: dict, modem: Optional[dict]) -> List[dict]:
"""Return every known device as {name, kind, status, ip, model, ..., search}."""
snapshot = snapshot or {}
link_stats = link_stats or {}
devices: Dict[str, dict] = {}
for name, host in (snapshot.get('hosts') or {}).items():
link_ifaces = (link_stats.get('hosts') or {}).get(name) or {}
ifaces = sorted(set(host.get('interfaces') or {}) | set(link_ifaces))
devices[name] = {
'name': name, 'kind': 'host', 'status': host.get('status', 'unknown'),
'ip': host.get('ip', ''), 'mac': '', 'model': '', 'source': host.get('source', ''),
'interface_count': len(ifaces),
'search': _search_text(name, host.get('ip'), 'host server', ifaces),
}
# Prometheus also sees hosts that aren't in config.json, e.g. LXC app servers.
# They have link stats but no alerting, so their state is reported as unknown.
for name, ifaces in (link_stats.get('hosts') or {}).items():
if name not in devices:
ip = next((i.get('host_ip') for i in ifaces.values() if i.get('host_ip')), '')
devices[name] = {
'name': name, 'kind': 'host', 'status': 'unknown', 'ip': ip, 'mac': '', 'model': '',
'source': 'prometheus', 'interface_count': len(ifaces),
'search': _search_text(name, ip, 'host server', list(ifaces)),
}
switches = link_stats.get('unifi_switches') or {}
for d in snapshot.get('unifi') or []:
name = d.get('name', '')
ports = (switches.get(name) or {}).get('ports') or {}
kind = unifi_kind(d.get('type', ''), d.get('model', ''))
devices[name] = {
'name': name, 'kind': kind, 'status': 'up' if d.get('connected') else 'down',
'ip': d.get('ip', ''), 'mac': d.get('mac', ''), 'model': d.get('model', ''),
'source': 'unifi', 'interface_count': len(ports),
'search': _search_text(
name, d.get('ip'), _mac_forms(d.get('mac', '')), d.get('model'),
d.get('type'), kind, list(ports), _port_neighbors(ports),
),
}
# A switch can have link stats while the device list is unavailable this cycle.
for name, sw in switches.items():
if name not in devices:
ports = sw.get('ports') or {}
devices[name] = {
'name': name, 'kind': unifi_kind('usw', sw.get('model', '')), 'status': 'unknown',
'ip': sw.get('ip', ''), 'mac': '', 'model': sw.get('model', ''),
'source': 'unifi', 'interface_count': len(ports),
'search': _search_text(name, sw.get('ip'), sw.get('model'), list(ports), _port_neighbors(ports)),
}
if modem and modem.get('name'):
name = modem['name']
if not modem.get('reachable'):
status = 'unknown'
else:
status = 'down' if modem.get('wan_problems') else 'up'
devices[name] = {
'name': name, 'kind': 'modem', 'status': status,
'ip': modem.get('lan_ip', ''), 'mac': '', 'model': modem.get('model', ''),
'source': 'modem', 'interface_count': 0,
'search': _search_text(
name, modem.get('lan_ip'), modem.get('wan_ip'), modem.get('model'),
'modem gateway fiber wan at&t att',
),
}
return sorted(devices.values(), key=lambda d: d['name'].lower())
def uplinks_for_host(name: str, switches: dict) -> List[dict]:
"""Find the switch ports whose LLDP neighbor is this host."""
found = []
for sw_name, sw in (switches or {}).items():
for port_name, port in (sw.get('ports') or {}).items():
lldp = port.get('lldp') or {}
if lldp.get('system_name') == name:
found.append({
'switch': sw_name, 'port': port_name, 'port_idx': port.get('port_idx'),
'host_interface': lldp.get('port_id', ''), 'up': port.get('up', False),
'speed_mbps': port.get('speed_mbps', 0),
})
return found
def device_detail(name: str, snapshot: dict, link_stats: dict, modem: Optional[dict]) -> Optional[dict]:
"""Return everything known about one device, or None if it doesn't exist."""
match = next((d for d in build_index(snapshot, link_stats, modem) if d['name'] == name), None)
if match is None:
return None
link_stats = link_stats or {}
detail = {k: v for k, v in match.items() if k != 'search'}
switches = link_stats.get('unifi_switches') or {}
if match['kind'] == 'host':
states = ((snapshot or {}).get('hosts') or {}).get(name, {}).get('interfaces') or {}
link_ifaces = (link_stats.get('hosts') or {}).get(name) or {}
interfaces = {i: {'state': s} for i, s in states.items()}
for iface, stats in link_ifaces.items():
interfaces.setdefault(iface, {}).update(stats)
detail['interfaces'] = interfaces
detail['uplinks'] = uplinks_for_host(name, switches)
elif match['kind'] == 'modem':
pass # the page renders the shared WAN / Fiber panel from /api/modem
else:
detail['ports'] = (switches.get(name) or {}).get('ports') or {}
detail['info'] = (link_stats.get('unifi_devices') or {}).get(name) or {}
detail['updated'] = link_stats.get('updated')
return detail
Binary file not shown.

Before

Width:  |  Height:  |  Size: 49 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 71 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 206 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 145 KiB

+287
View File
@@ -0,0 +1,287 @@
"""Read-only client for AT&T BGW-series fiber gateways (tested on BGW320-505).
The gateway has no API, but its status pages are served without a login, so
this scrapes them over HTTP. Only GET requests to the fixed status pages below
are ever made: nothing here can change gateway settings.
Units come from the gateway's own help text, checked against live readings:
Rx/Tx power tenths of a dBm (-169 -> -16.9 dBm)
Temperature degrees C
Uptime seconds (despite the help text claiming D:H:M:S)
Deliberately not used, because the page disagrees with itself:
Vcc / Tx Bias readings values do not match the documented scales, so only
their alarm/warning flags are kept
Rx LOS State reads 1 on a healthy link; "Optical WAN Operational
Status" is the reliable signal
Last Change does not count up, despite the help text
"""
import html
import logging
import re
from datetime import datetime, timezone
from typing import Dict, List, Optional
import requests
logger = logging.getLogger('gandalf.modem')
_FIBER_PAGE = '/cgi-bin/fiberstat.ha'
_BROADBAND_PAGE = '/cgi-bin/broadbandstatistics.ha'
_SYSINFO_PAGE = '/cgi-bin/sysinfo.ha'
_ROW_RE = re.compile(r'<th\b[^>]*>(.*?)</th>\s*<td\b[^>]*>(.*?)</td>', re.I | re.S)
_TAG_RE = re.compile(r'<[^>]+>')
# e.g. "0 (Threshold -292)": alarm flag, then the limit
_DOM_CELL_RE = re.compile(r'<td\b[^>]*>\s*(\d+)\s*\(Threshold\s+(-?\d+)\)\s*</td>', re.I)
# Page heading -> (result key, scale to real units; None = reading is unusable)
_DOM_METRICS = {
'Temperature': ('temperature', 1.0),
'Vcc': ('vcc', None),
'Tx Bias': ('tx_bias', None),
'Tx Power': ('tx_power', 0.1),
'Rx Power': ('rx_power', 0.1),
}
_COUNTERS = {
'rx_bytes': 'Receive Bytes',
'tx_bytes': 'Transmit Bytes',
'rx_errors': 'Receive Errors',
'tx_errors': 'Transmit Errors',
'rx_drops': 'Receive Drops',
'tx_drops': 'Transmit Drops',
}
class ModemUnreachable(Exception):
"""The gateway did not answer (power, cabling, or routing problem)."""
class ModemDataInvalid(Exception):
"""The gateway answered but the page was not what we can parse.
AT&T pushes firmware automatically, so this is how a layout change or a
newly login-protected page shows up.
"""
def parse_rows(page: str) -> Dict[str, str]:
"""Return {label: value} for the page's <th>/<td> rows; the first label wins.
First wins because labels repeat: "Transmit Packets" appears under both
IPv4 and IPv6, and the IPv4 block comes first.
"""
rows: Dict[str, str] = {}
for label, value in _ROW_RE.findall(page):
key = html.unescape(_TAG_RE.sub('', label)).strip()
if key:
rows.setdefault(key, ' '.join(html.unescape(_TAG_RE.sub('', value)).split()))
return rows
def parse_dom_metrics(page: str) -> Dict[str, dict]:
"""Parse the fiber page's Temperature / Vcc / Tx Bias / Tx Power / Rx Power tables."""
metrics: Dict[str, dict] = {}
for heading, (key, scale) in _DOM_METRICS.items():
m = re.search(
rf'<h1>\s*{re.escape(heading)}(?:&nbsp;|\s)+Currently\s+(-?\d+)\s*</h1>(.*?)</table>',
page, re.I | re.S,
)
if not m:
continue
cells = _DOM_CELL_RE.findall(m.group(2))
if len(cells) != 4: # alarm low, alarm high, warning low, warning high
continue
def limit(raw: str, scale: Optional[float] = scale) -> Optional[float]:
return round(int(raw) * scale, 1) if scale is not None else None
metrics[key] = {
'value': limit(m.group(1)),
'alarm_low': limit(cells[0][1]),
'alarm_high': limit(cells[1][1]),
'warn_low': limit(cells[2][1]),
'warn_high': limit(cells[3][1]),
'alarm_flag': cells[0][0] != '0' or cells[1][0] != '0',
'warn_flag': cells[2][0] != '0' or cells[3][0] != '0',
}
return metrics
def metric_status(metric: dict) -> str:
"""Classify a DOM metric as 'alarm', 'warning', or 'ok'.
Trusts the module's own flags, and also compares the reading to its limits
in case the flags are latched off.
"""
value = metric.get('value')
def outside(low: Optional[float], high: Optional[float]) -> bool:
if value is None:
return False
return (low is not None and value <= low) or (high is not None and value >= high)
if metric.get('alarm_flag') or outside(metric.get('alarm_low'), metric.get('alarm_high')):
return 'alarm'
if metric.get('warn_flag') or outside(metric.get('warn_low'), metric.get('warn_high')):
return 'warning'
return 'ok'
def _to_int(raw: Optional[str]) -> Optional[int]:
try:
return int(raw) # type: ignore[arg-type]
except (TypeError, ValueError):
return None
def wan_problems(data: dict) -> List[str]:
"""Return human-readable reasons the WAN is down; empty when it looks healthy."""
problems = []
if (data.get('broadband_state') or '').lower() != 'up':
problems.append(f'broadband connection is {data.get("broadband_state") or "unknown"}')
if (data.get('optical_state') or '').lower() != 'up':
problems.append(f'optical WAN is {data.get("optical_state") or "unknown"}')
pon = data.get('pon_state')
if pon and 'O5' not in pon: # O5 is the PON "operation" state; older firmware may omit the field
problems.append(f'PON link state is {pon}')
return problems
def build_status(fiber_page: str, broadband_page: str, sysinfo_page: str = '') -> dict:
"""Combine the status pages into one dict. Raises ModemDataInvalid if unusable."""
fiber = parse_rows(fiber_page)
broadband = parse_rows(broadband_page)
sysinfo = parse_rows(sysinfo_page)
if 'Optical WAN Operational Status' not in fiber:
raise ModemDataInvalid('fiber page is missing "Optical WAN Operational Status"')
if 'Broadband Connection' not in broadband:
raise ModemDataInvalid('broadband page is missing "Broadband Connection"')
metrics = parse_dom_metrics(fiber_page)
data = {
'model': sysinfo.get('Model Number', ''),
'firmware': sysinfo.get('Software Version', ''),
'uptime_s': _to_int(sysinfo.get('Time Since Last Reboot')),
'broadband_state': broadband.get('Broadband Connection', ''),
'source': broadband.get('Broadband Connection Source', ''),
'wan_ip': broadband.get('Broadband IPv4 Address', ''),
'pon_state': broadband.get('PON Link Status', ''),
'uni_state': broadband.get('UNI Status', ''),
'line_speed_mbps': _to_int(broadband.get('Current Speed (Mbps)')),
'optical_state': fiber.get('Optical WAN Operational Status', ''),
'wavelength': fiber.get('Wave Length', ''),
'optics_part_no': fiber.get('Vendor PN', ''),
'optics': metrics,
'counters': {k: _to_int(broadband.get(label)) for k, label in _COUNTERS.items()},
}
# With a healthy link the optics must be readable; if not, the layout changed.
# With a cut fiber they may legitimately be blank, so don't report that as invalid.
if not wan_problems(data) and 'rx_power' not in metrics:
raise ModemDataInvalid('link is up but no Rx power reading was found')
return data
def counter_rates(current: dict, previous: Optional[dict], elapsed_s: float) -> Dict[str, Optional[float]]:
"""Return per-second rates between counter snapshots.
A rate is None rather than a bogus value on the first poll, or when a counter
goes backwards because the gateway rebooted.
"""
rates: Dict[str, Optional[float]] = {}
for key, now_val in current.items():
then = (previous or {}).get(key)
if now_val is None or then is None or elapsed_s <= 0 or now_val < then:
rates[key] = None
else:
rates[key] = (now_val - then) / elapsed_s
return rates
class ModemClient:
def __init__(self, cfg: dict):
self.base_url = cfg['url'].rstrip('/')
self.timeout = cfg.get('timeout', 10)
self.session = requests.Session()
def _get(self, path: str) -> str:
try:
# Protected pages redirect or render a login form, so never follow redirects.
resp = self.session.get(f'{self.base_url}{path}', timeout=self.timeout, allow_redirects=False)
except requests.RequestException as e:
raise ModemUnreachable(f'{path}: {e}') from e
if resp.status_code != 200:
raise ModemDataInvalid(f'{path} returned HTTP {resp.status_code}')
if re.search(r'<title>\s*Login', resp.text, re.I):
raise ModemDataInvalid(f'{path} now requires a login')
return resp.text
def poll(self) -> dict:
"""Return gateway status. Raises ModemUnreachable or ModemDataInvalid."""
fiber = self._get(_FIBER_PAGE)
broadband = self._get(_BROADBAND_PAGE)
try:
sysinfo = self._get(_SYSINFO_PAGE)
except (ModemUnreachable, ModemDataInvalid) as e:
# Model and uptime are nice to have, not worth failing the poll over.
logger.warning(f'Modem system info unavailable: {e}')
sysinfo = ''
data = build_status(fiber, broadband, sysinfo)
data['updated'] = datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M:%S UTC')
return data
# ---------------------------------------------------------------------------
# WAN history: outages and gateway restarts, which are too brief to catch by polling
# ---------------------------------------------------------------------------
LOG_WAN_FAILED = 'NETWORK_WAN_FAILED_2'
LOG_WAN_RESTORED = 'NETWORK_WAN_RESTORED_2'
LOG_WAN_UNSTABLE = 'NETWORK_WAN_FAILED_MULTIPLE_TIMES_2'
LOG_MODEM_RESTARTED = 'SMART_POWER_MODEM_RESTARTED'
WAN_LOG_KEYS = (LOG_WAN_FAILED, LOG_WAN_RESTORED, LOG_WAN_UNSTABLE, LOG_MODEM_RESTARTED)
_HISTORY_LIMIT = 50
# The RPDU's log entry and the gateway's calculated boot time can differ by this much.
_RESTART_MATCH_WINDOW_S = 600
def apply_wan_events(history: dict, events: List[dict]) -> List[dict]:
"""Fold UniFi log events into *history* in place and return the ones worth a ticket.
Each event is {'key', 'ts' (epoch seconds), 'device', 'wan'}.
"""
outages = history.setdefault('outages', [])
restarts = history.setdefault('restarts', [])
actionable = []
for e in sorted(events, key=lambda ev: ev['ts']):
key = e['key']
if key == LOG_WAN_FAILED:
if not any(o.get('restored_at') is None for o in outages):
outages.append({'failed_at': e['ts'], 'restored_at': None, 'wan': e.get('wan', '')})
elif key == LOG_WAN_RESTORED:
for o in reversed(outages):
if o.get('restored_at') is None:
o['restored_at'] = e['ts']
break
elif key == LOG_MODEM_RESTARTED:
restarts.append({'at': e['ts'], 'by': e.get('device') or 'smart power', 'confirmed': False})
actionable.append(e)
elif key == LOG_WAN_UNSTABLE:
actionable.append(e)
del outages[:-_HISTORY_LIMIT]
del restarts[:-_HISTORY_LIMIT]
return actionable
def record_uptime_reset(history: dict, boot_ts: float) -> None:
"""Note that the gateway's uptime went backwards, attributing it to a logged restart if possible."""
restarts = history.setdefault('restarts', [])
for r in reversed(restarts):
if abs(r['at'] - boot_ts) <= _RESTART_MATCH_WINDOW_S:
r['confirmed'] = True
return
restarts.append({'at': boot_ts, 'by': None, 'confirmed': True})
del restarts[:-_HISTORY_LIMIT]
+541 -29
View File
@@ -11,14 +11,17 @@ import json
import logging
import re
import shlex
import statistics
import time
from datetime import datetime, timezone
from typing import Dict, List, Optional
from urllib.parse import urlparse
import requests
from urllib3.exceptions import InsecureRequestWarning
import db
import modem as modem_api
logging.basicConfig(
@@ -53,7 +56,8 @@ class PrometheusClient:
def __init__(self, url: str):
self.url = url.rstrip('/')
def query(self, promql: str) -> list:
def _fetch(self, promql: str) -> Optional[list]:
"""Return query results, or None if the query itself failed."""
try:
resp = requests.get(
f'{self.url}/api/v1/query',
@@ -64,13 +68,23 @@ class PrometheusClient:
data = resp.json()
if data.get('status') == 'success':
return data['data']['result']
logger.error(f'Prometheus query unsuccessful ({promql!r}): {data}')
except Exception as e:
logger.error(f'Prometheus query failed ({promql!r}): {e}')
return []
return None
def get_interface_states(self) -> Dict[str, Dict[str, bool]]:
"""Return {instance: {device: is_up}} for physical interfaces."""
results = self.query('node_network_up')
def query(self, promql: str) -> list:
return self._fetch(promql) or []
def get_interface_states(self) -> Optional[Dict[str, Dict[str, bool]]]:
"""Return {instance: {device: is_up}} for physical interfaces.
Returns None when Prometheus can't answer or has no series at all, so
callers can tell "we can't see anything" apart from "this host is gone".
"""
results = self._fetch('node_network_up')
if not results:
return None
hosts: Dict[str, Dict[str, bool]] = {}
for r in results:
instance = r['metric'].get('instance', '')
@@ -85,6 +99,10 @@ class PrometheusClient:
# UniFi client
# --------------------------------------------------------------------------
class UnifiClient:
# Devices whose port tables are collected: switches and the gateway/router
_PORT_DEVICE_TYPES = ('usw', 'udm', 'ugw', 'uxg')
_SYSTEM_LOG_MAX_PAGES = 10
def __init__(self, cfg: dict):
self.base_url = cfg['controller']
self.site_id = cfg.get('site_id', 'default')
@@ -97,6 +115,96 @@ class UnifiClient:
'Accept': 'application/json',
}
def get_device_details(self) -> Optional[Dict[str, dict]]:
"""Return firmware, load, WAN, and outlet details per device name.
Uses the v1 stat API, the only one that includes these fields.
"""
try:
url = f'{self.base_url}/proxy/network/api/s/{self.site_id}/stat/device'
resp = self.session.get(url, headers=self.headers, timeout=15)
resp.raise_for_status()
return {
d.get('name') or d.get('mac', 'unknown'): self._device_detail(d)
for d in resp.json().get('data', [])
}
except Exception as e:
logger.error(f'UniFi device detail error: {e}')
return None
@staticmethod
def _device_detail(d: dict) -> dict:
def number(raw) -> Optional[float]:
try:
return float(raw)
except (TypeError, ValueError):
return None
stats = d.get('system-stats') or {}
wan = {}
for key in ('wan1', 'wan2'):
w = d.get(key) or {}
if w:
wan[key] = {
'up': w.get('up'), 'speed_mbps': w.get('speed'),
'latency_ms': w.get('latency'), 'availability': w.get('availability'),
}
return {
'firmware': d.get('version', ''),
'uptime_s': d.get('uptime'),
'cpu_pct': number(stats.get('cpu')),
'mem_pct': number(stats.get('mem')),
'clients': d.get('num_sta'),
'temps': [
{'name': t.get('name', ''), 'value_c': number(t.get('value'))}
for t in d.get('temperatures') or []
],
'wan': wan,
'power_w': number(d.get('outlet_ac_power_consumption')),
'outlets': [
{
'index': o.get('index'),
'name': o.get('name', ''),
'on': o.get('relay_state'),
'auto_power_cycle': o.get('cycle_enabled'),
'power_w': number(o.get('outlet_power')),
'current_a': number(o.get('outlet_current')),
}
for o in d.get('outlet_table') or []
],
}
def get_system_events(self, since_ms: int, keys: tuple) -> Optional[List[dict]]:
"""Return system log entries with the given keys, or None if the log is unavailable."""
url = f'{self.base_url}/proxy/network/v2/api/site/{self.site_id}/system-log/all'
found: List[dict] = []
try:
for page in range(self._SYSTEM_LOG_MAX_PAGES):
resp = self.session.post(
url, headers=self.headers, timeout=20,
json={
'timestampFrom': since_ms, 'timestampTo': int(time.time() * 1000),
'pageNumber': page, 'pageSize': 200,
},
)
resp.raise_for_status()
items = resp.json().get('data') or []
if not items:
break
for e in items:
if e.get('key') in keys and e.get('timestamp', 0) >= since_ms:
params = e.get('parameters') or {}
found.append({
'key': e['key'], 'ts': e['timestamp'] / 1000,
'device': (params.get('DEVICE') or {}).get('name', ''),
'wan': (params.get('WAN_ID') or {}).get('name', '')
or str((params.get('WAN_ID') or {}).get('id', '')),
})
return found
except Exception as e:
logger.error(f'UniFi system log error: {e}')
return None
def get_devices(self) -> Optional[List[dict]]:
"""Return list of UniFi devices, or None if the controller is unreachable."""
try:
@@ -122,7 +230,7 @@ class UnifiClient:
return None
def get_switch_ports(self) -> Optional[Dict[str, dict]]:
"""Return per-port stats for all UniFi switches, keyed by switch name.
"""Return per-port stats for UniFi switches and routers, keyed by device name.
Uses the v1 stat API which includes full port_table data.
Returns {switch_name: {'ip': str, 'model': str, 'ports': {port_name: {...}}}}.
@@ -134,7 +242,7 @@ class UnifiClient:
devices = resp.json().get('data', [])
result: Dict[str, dict] = {}
for dev in devices:
if dev.get('type', '').lower() != 'usw':
if dev.get('type', '').lower() not in self._PORT_DEVICE_TYPES:
continue
sw_name = dev.get('name') or dev.get('mac', 'unknown')
sw_ip = dev.get('ip', '')
@@ -321,13 +429,19 @@ class PulseClient:
return self.run_command(command, _retry=False)
return None
def ping(self, ip: str, count: int = 3, timeout: int = 2) -> bool:
"""Ping *ip* via the Pulse worker. Returns True if host responds."""
def ping(self, ip: str, count: int = 3, timeout: int = 2) -> Optional[bool]:
"""Ping *ip* via the Pulse worker.
Returns True/False for the host's reachability, or None if Pulse itself
couldn't run the check, which says nothing about the host.
"""
ip_q = shlex.quote(ip)
output = self.run_command(
f'ping -c {count} -W {timeout} {ip_q} >/dev/null 2>&1 && echo REACHABLE || echo UNREACHABLE'
)
return output is not None and output.strip() == 'REACHABLE'
if output is None:
return None
return output.strip() == 'REACHABLE'
# --------------------------------------------------------------------------
@@ -615,9 +729,12 @@ class LinkStatsCollector:
except Exception as e:
logger.warning(f'UniFi switch port collection failed: {e}')
unifi_devices = (self.unifi.get_device_details() or {}) if self.unifi else {}
return {
'hosts': result_hosts,
'unifi_switches': unifi_switches,
'unifi_devices': unifi_devices,
'updated': datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M:%S UTC'),
}
@@ -625,8 +742,10 @@ class LinkStatsCollector:
"""Compute per-port byte/error rates from delta against previous snapshot."""
dt = now - self._prev_unifi_time if self._prev_unifi_time > 0 else 0
def rate(new_val: int, old_val: int) -> Optional[float]:
if dt <= 0:
def rate(new_val: int, old_val: Optional[int]) -> Optional[float]:
# No earlier reading (first poll, or a new/renamed port) means no rate,
# not "all bytes since boot arrived in this one interval".
if dt <= 0 or old_val is None:
return None
return max(0.0, (new_val - old_val) / dt)
@@ -637,12 +756,12 @@ class LinkStatsCollector:
for pname, d in sw_data['ports'].items():
entry = dict(d)
prev = prev_ports.get(pname, {})
entry['tx_bytes_rate'] = rate(d['tx_bytes'], prev.get('tx_bytes', 0))
entry['rx_bytes_rate'] = rate(d['rx_bytes'], prev.get('rx_bytes', 0))
entry['tx_errs_rate'] = rate(d['tx_errors'], prev.get('tx_errors', 0))
entry['rx_errs_rate'] = rate(d['rx_errors'], prev.get('rx_errors', 0))
entry['tx_drops_rate'] = rate(d['tx_dropped'], prev.get('tx_dropped', 0))
entry['rx_drops_rate'] = rate(d['rx_dropped'], prev.get('rx_dropped', 0))
entry['tx_bytes_rate'] = rate(d['tx_bytes'], prev.get('tx_bytes'))
entry['rx_bytes_rate'] = rate(d['rx_bytes'], prev.get('rx_bytes'))
entry['tx_errs_rate'] = rate(d['tx_errors'], prev.get('tx_errors'))
entry['rx_errs_rate'] = rate(d['rx_errors'], prev.get('rx_errors'))
entry['tx_drops_rate'] = rate(d['tx_dropped'], prev.get('tx_dropped'))
entry['rx_drops_rate'] = rate(d['rx_dropped'], prev.get('rx_dropped'))
merged_ports[pname] = entry
result[sw_name] = {'ip': sw_data['ip'], 'model': sw_data['model'],
'ports': merged_ports}
@@ -685,6 +804,27 @@ class NetworkMonitor:
self.cluster_thresh = mon.get('cluster_threshold', 3)
self.cluster_name = mon.get('cluster_name', CLUSTER_NAME)
# Optional WAN modem (AT&T BGW-series); see modem.py
modem_cfg = self.cfg.get('modem') or {}
self.modem = modem_api.ModemClient(modem_cfg) if modem_cfg.get('url') else None
self.modem_name = modem_cfg.get('name', 'att-modem')
# Modem reboots (including AT&T's automatic firmware updates) take a few
# minutes, so this can be raised independently of failure_threshold.
self.modem_fail_thresh = modem_cfg.get('failure_threshold', self.fail_thresh)
self.rx_drift_db = modem_cfg.get('rx_drift_db', 3.0)
self.rx_drift_min_samples = modem_cfg.get('rx_drift_min_samples', 288)
self.modem_lan_ip = urlparse(modem_cfg['url']).hostname if modem_cfg.get('url') else ''
self._prev_modem_counters: Optional[dict] = None
self._prev_modem_time = 0.0
if self.modem:
try:
db.ensure_modem_schema()
except Exception as e:
logger.error(f'Could not create modem history table: {e}')
# Last good Prometheus result, shown (as unknown) during a Prometheus outage
self._last_iface_states: Dict[str, Dict[str, bool]] = {}
# Build Prometheus instance → hostname lookup
self._instance_map: Dict[str, str] = {
h['prometheus_instance']: h['name']
@@ -744,6 +884,22 @@ class NetworkMonitor:
if host_has_regression:
hosts_with_regression.append(host)
# A dead host doesn't report its interfaces as down: its series just
# disappear, so a configured host that's absent needs its own check.
for instance, host in self._instance_map.items():
if instance in states:
db.resolve_event('host_metrics_missing', host, instance)
continue
hosts_with_regression.append(host)
sup = db.check_suppressed(suppressions, 'host', host)
event_id, is_new, consec, ticket_id = db.upsert_event(
'host_metrics_missing', 'critical', 'prometheus',
host, instance,
f'No metrics from {host} ({instance}): host or node_exporter is down ({_now_utc()})',
)
if not sup and consec >= self.fail_thresh and not ticket_id:
self._ticket_metrics_missing(event_id, host, instance, consec)
db.set_baseline(new_baseline)
# Cluster-wide check – only genuine regressions count
@@ -752,7 +908,7 @@ class NetworkMonitor:
event_id, is_new, consec, ticket_id = db.upsert_event(
'cluster_network_issue', 'critical', 'prometheus',
self.cluster_name, '',
f'{len(hosts_with_regression)} hosts reporting simultaneous interface failures: '
f'{len(hosts_with_regression)} hosts with simultaneous interface or metrics failures: '
f'{", ".join(hosts_with_regression)}',
)
if not sup and (is_new or not ticket_id):
@@ -765,7 +921,8 @@ class NetworkMonitor:
f'Affected hosts: {", ".join(hosts_with_regression)}\n'
f'Detected: {_now_utc()}\n\n'
f'{len(hosts_with_regression)} Proxmox hosts simultaneously reported '
f'interface regressions (link-down on interfaces previously known UP).\n'
f'interface regressions (link-down on interfaces previously known UP) '
f'or stopped reporting metrics.\n'
f'This likely indicates a switch or upstream network failure.\n\n'
f'Please check the core and management switches immediately.'
)
@@ -775,6 +932,54 @@ class NetworkMonitor:
else:
db.resolve_event('cluster_network_issue', self.cluster_name, '')
def _ticket_metrics_missing(
self, event_id: int, host: str, instance: str, consec: int
) -> None:
title = (
f'[{host}][auto][production][issue][network][single-node] '
f'No metrics from host'
)
desc = (
f'Host Metrics Alert\n{"=" * 40}\n\n'
f'Host: {host}\n'
f'Prometheus instance: {instance}\n'
f'Detected: {_now_utc()}\n'
f'Consecutive check failures: {consec}\n\n'
f'Prometheus is working but has no node_network_up series for {host}.\n'
f'Either the host is down or node_exporter has stopped on it.\n\n'
f'Please check the host power and network connectivity, then '
f'prometheus-node-exporter on the host.'
)
tid = self.tickets.create(title, desc, priority='2')
if tid:
db.set_ticket_id(event_id, tid)
def _process_prometheus_outage(self, suppressions: list) -> None:
"""Interface and host alerts can't fire without Prometheus, so say so."""
sup = db.check_suppressed(suppressions, 'host', 'prometheus')
event_id, is_new, consec, ticket_id = db.upsert_event(
'monitor_source_unavailable', 'critical', 'prometheus',
'prometheus', '',
f'Prometheus returned no interface data; host and interface alerting is paused ({_now_utc()})',
)
if not sup and consec >= self.fail_thresh and not ticket_id:
title = (
'[prometheus][auto][production][issue][network][cluster-wide] '
'Gandalf has no Prometheus data'
)
desc = (
f'Monitoring Source Alert\n{"=" * 40}\n\n'
f'Prometheus URL: {self.prom.url}\n'
f'Detected: {_now_utc()}\n'
f'Consecutive check failures: {consec}\n\n'
f'Gandalf could not get node_network_up data from Prometheus.\n'
f'While this lasts, host and interface failures will NOT be detected.\n\n'
f'Please check the Prometheus service and its scrape targets.'
)
tid = self.tickets.create(title, desc, priority='2')
if tid:
db.set_ticket_id(event_id, tid)
def _ticket_interface(
self, event_id: int, host: str, iface: str, consec: int
) -> None:
@@ -842,10 +1047,15 @@ class NetworkMonitor:
# ------------------------------------------------------------------
# Ping-only hosts (no node_exporter)
# ------------------------------------------------------------------
def _process_ping_hosts(self, suppressions: list, ping_states: Dict[str, bool]) -> None:
def _process_ping_hosts(self, suppressions: list, ping_states: Dict[str, Optional[bool]]) -> None:
for h in self.cfg.get('monitor', {}).get('ping_hosts', []):
name, ip = h['name'], h['ip']
reachable = ping_states.get(name, False)
reachable = ping_states.get(name)
if reachable is None:
# Pulse couldn't run the check. Neither alert nor resolve on no evidence.
logger.warning(f'Ping check for {name} ({ip}) unavailable this cycle')
continue
if not reachable:
sup = db.check_suppressed(suppressions, 'host', name)
@@ -881,26 +1091,301 @@ class NetworkMonitor:
if tid:
db.set_ticket_id(event_id, tid)
# ------------------------------------------------------------------
# WAN modem monitoring
# ------------------------------------------------------------------
_MODEM_METRIC_LABELS = {
'rx_power': 'Rx power', 'tx_power': 'Tx power', 'temperature': 'Temperature',
'vcc': 'Supply voltage', 'tx_bias': 'Tx bias',
}
_MODEM_METRIC_UNITS = {'rx_power': ' dBm', 'tx_power': ' dBm', 'temperature': ' C'}
def _modem_event(
self, suppressions: list, event_type: str, severity: str, detail: str,
description: str, heading: str, priority: str, advice: str,
) -> None:
name = self.modem_name
sup = db.check_suppressed(suppressions, 'host', name)
event_id, is_new, consec, ticket_id = db.upsert_event(
event_type, severity, 'modem', name, detail, f'{description} ({_now_utc()})',
)
if not sup and consec >= self.modem_fail_thresh and not ticket_id:
title = f'[{name}][auto][production][issue][network][cluster-wide] {heading}'
desc = (
f'WAN Modem Alert\n{"=" * 40}\n\n'
f'Modem: {name} ({self.modem.base_url})\n'
f'Detected: {_now_utc()}\n'
f'Consecutive check failures: {consec}\n\n'
f'{description}\n\n{advice}'
)
tid = self.tickets.create(title, desc, priority=priority)
if tid:
db.set_ticket_id(event_id, tid)
def _process_modem_failure(self, suppressions: list, error: Exception) -> None:
"""Handle a failed poll. Link and optical events stay as they are: we can't see them."""
name = self.modem_name
if isinstance(error, modem_api.ModemUnreachable):
db.resolve_event('modem_data_invalid', name, '')
self._modem_event(
suppressions, 'modem_unreachable', 'critical', '',
f'WAN modem is not responding: {error}',
'WAN modem unreachable', '2',
'Internet access is probably affected too. Check modem power and the '
'cable between the modem and the UDM-Pro.',
)
else:
db.resolve_event('modem_unreachable', name, '')
self._modem_event(
suppressions, 'modem_data_invalid', 'warning', '',
f'WAN modem status could not be read: {error}',
'WAN modem status unreadable', '3',
'The modem answered but its pages could not be parsed, so fiber alerting is '
'paused. A firmware update may have changed them; see modem.py.',
)
def _process_modem(self, data: dict, suppressions: list) -> None:
"""Raise and resolve alerts from a successful modem poll."""
name = self.modem_name
db.resolve_event('modem_unreachable', name, '')
db.resolve_event('modem_data_invalid', name, '')
problems = modem_api.wan_problems(data)
if problems:
self._modem_event(
suppressions, 'wan_down', 'critical', '',
f'WAN link is down: {"; ".join(problems)}.',
'WAN link down', '1',
'Check the fiber at the modem and the AT&T service status. Rx power at or below '
'the alarm limit points to a fiber fault rather than a provider outage.',
)
else:
db.resolve_event('wan_down', name, '')
for key, metric in data.get('optics', {}).items():
status = modem_api.metric_status(metric)
if status == 'alarm' and problems and key == 'rx_power':
# wan_down already covers this, and a cut fiber would otherwise open two tickets.
status = 'ok'
label = self._MODEM_METRIC_LABELS.get(key, key)
value = metric.get('value')
reading = f' at {value}{self._MODEM_METRIC_UNITS.get(key, "")}' if value is not None else ''
limits = (
f' (warning {metric.get("warn_low")} to {metric.get("warn_high")}, '
f'alarm {metric.get("alarm_low")} to {metric.get("alarm_high")})'
) if value is not None else ''
advice = (
'Inspect and clean the fiber connector at the modem and look for tight bends. '
'If that does not help, contact AT&T: the reading is what the modem measures '
'on the provider side of the connection.'
)
if status == 'alarm':
db.resolve_event('fiber_signal_warning', name, key)
self._modem_event(
suppressions, 'fiber_signal_alarm', 'critical', key,
f'{label} is in the alarm range{reading}{limits}.',
f'Fiber {label} alarm', '2', advice,
)
elif status == 'warning':
db.resolve_event('fiber_signal_alarm', name, key)
self._modem_event(
suppressions, 'fiber_signal_warning', 'warning', key,
f'{label} is in the warning range{reading}{limits}.',
f'Fiber {label} warning', '3', advice,
)
else:
db.resolve_event('fiber_signal_alarm', name, key)
db.resolve_event('fiber_signal_warning', name, key)
self._check_rx_drift(data, suppressions)
def _check_rx_drift(self, data: dict, suppressions: list) -> None:
"""Alert on a sustained Rx drop relative to this link's own recent normal.
The modem's limits are very wide (warning at -27.9 dBm on a link that sits
around -17), so a damaged connector could lose many dB before they trip.
"""
name = self.modem_name
rx = (data.get('optics', {}).get('rx_power') or {}).get('value')
if rx is None:
return
history = db.get_modem_rx_values(days=7)
if len(history) < self.rx_drift_min_samples:
return # not enough history yet to know what normal looks like
baseline = statistics.median(history)
drop = round(baseline - rx, 1)
if drop >= self.rx_drift_db:
self._modem_event(
suppressions, 'fiber_rx_drift', 'warning', 'rx_power',
f'Rx power is {rx} dBm, {drop} dB below its 7-day median of {baseline:.1f} dBm.',
'Fiber Rx power dropped', '3',
'The signal is still in range but has changed. Look for a recently disturbed '
'or bent fiber and check the connector at the modem.',
)
else:
db.resolve_event('fiber_rx_drift', name, 'rx_power')
def _load_wan_history(self) -> dict:
raw = db.get_state('wan_history')
if raw:
try:
return json.loads(raw)
except ValueError:
logger.error('Failed to parse wan_history JSON; starting over')
return {'outages': [], 'restarts': []}
def _note_modem_restart(self, data: dict) -> None:
uptime = data.get('uptime_s')
if uptime is None:
return
previous = None
raw = db.get_state('modem_stats')
if raw:
try:
previous = json.loads(raw).get('uptime_s')
except ValueError:
logger.error('Failed to parse modem_stats JSON')
if previous is not None and uptime < previous:
logger.warning(f'Modem restarted: uptime went from {previous}s to {uptime}s')
history = self._load_wan_history()
modem_api.record_uptime_reset(history, time.time() - uptime)
db.set_state('wan_history', history)
def _one_shot_ticket(
self, suppressions: list, event_type: str, description: str,
heading: str, priority: str, advice: str,
) -> None:
"""Report something that has already happened: open a ticket and resolve the event."""
name = self.modem_name
event_id, _, _, ticket_id = db.upsert_event(event_type, 'warning', 'unifi', name, '', description)
if not db.check_suppressed(suppressions, 'host', name) and not ticket_id:
tid = self.tickets.create(
f'[{name}][auto][production][issue][network][cluster-wide] {heading}',
f'WAN Event\n{"=" * 40}\n\nModem: {name}\nReported: {_now_utc()}\n\n{description}\n\n{advice}',
priority=priority,
)
if tid:
db.set_ticket_id(event_id, tid)
db.resolve_event(event_type, name, '')
def _process_wan_log(self, suppressions: list) -> None:
"""Read WAN outages and modem restarts from the UniFi log.
These can be over in two minutes, so a 5 minute poll of the modem may never see them.
"""
now = time.time()
raw_cursor = db.get_state('unifi_log_cursor')
if raw_cursor is None:
# First run: start from now rather than ticketing old history.
db.set_state('unifi_log_cursor', str(int(now * 1000)))
return
# After long monitor downtime, don't open tickets for stale events.
since_ms = max(int(raw_cursor) + 1, int((now - 86400) * 1000))
events = self.unifi.get_system_events(since_ms, modem_api.WAN_LOG_KEYS)
if not events:
return # None = log unavailable, so the cursor stays put and it's retried
history = self._load_wan_history()
actionable = modem_api.apply_wan_events(history, events)
# UniFi repeats "failed multiple times" per failure, so ticket it once a day at most.
unstable = [e for e in actionable if e['key'] == modem_api.LOG_WAN_UNSTABLE]
ticket_unstable = bool(unstable) and now - history.get('unstable_ticketed_at', 0) >= 86400
if ticket_unstable:
history['unstable_ticketed_at'] = now
# Save first: a failed ticket call shouldn't replay these events next cycle.
db.set_state('wan_history', history)
db.set_state('unifi_log_cursor', str(int(max(e['ts'] for e in events) * 1000)))
for e in actionable:
if e['key'] == modem_api.LOG_MODEM_RESTARTED:
when = datetime.fromtimestamp(e['ts'], timezone.utc).strftime('%Y-%m-%d %H:%M:%S UTC')
self._one_shot_ticket(
suppressions, 'modem_power_cycled',
f'{e.get("device") or "Smart Power"} power-cycled the modem at {when} '
f'to try to restore the internet connection.',
'Modem auto power-cycled after internet loss', '3',
'The internet connection failed badly enough to trigger auto power-cycle. '
'Check the WAN / Fiber panel for outages around this time and Rx power for a drop.',
)
if ticket_unstable:
self._one_shot_ticket(
suppressions, 'wan_unstable',
'UniFi reports the WAN connection went down multiple times in the last 24 hours.',
'WAN connection unstable', '2',
'See the WAN history on the modem page. Repeated drops with healthy Rx power '
'point toward the provider or the modem rather than the fiber.',
)
def _collect_modem(self, suppressions: list) -> None:
"""Poll the modem, then store its status and raise any alerts."""
try:
data = self.modem.poll()
except (modem_api.ModemUnreachable, modem_api.ModemDataInvalid) as e:
logger.warning(f'Modem poll failed: {e}')
self._process_modem_failure(suppressions, e)
db.set_state('modem_stats', {
'reachable': False, 'error': str(e), 'name': self.modem_name,
'lan_ip': self.modem_lan_ip, 'updated': _now_utc(),
})
return
# Uptime going backwards between polls means the gateway restarted, even if
# it came back too quickly for any poll to fail. Read before it's overwritten.
self._note_modem_restart(data)
now = time.monotonic()
data['rates'] = modem_api.counter_rates(
data['counters'], self._prev_modem_counters, now - self._prev_modem_time,
)
self._prev_modem_counters = data['counters']
self._prev_modem_time = now
data['reachable'] = True
data['name'] = self.modem_name
data['lan_ip'] = self.modem_lan_ip
data['wan_problems'] = modem_api.wan_problems(data)
data['optics_status'] = {k: modem_api.metric_status(m) for k, m in data['optics'].items()}
db.set_state('modem_stats', data)
# Alert before recording, so a sudden drop isn't compared against itself.
self._process_modem(data, suppressions)
optics = data['optics']
db.record_modem_sample(
(optics.get('rx_power') or {}).get('value'),
(optics.get('tx_power') or {}).get('value'),
(optics.get('temperature') or {}).get('value'),
)
# ------------------------------------------------------------------
# Snapshot collection (for dashboard)
# ------------------------------------------------------------------
def _collect_snapshot(
self, iface_states: Dict[str, Dict[str, bool]],
self, iface_states: Optional[Dict[str, Dict[str, bool]]],
unifi_devices: Optional[List[dict]] = None,
ping_states: Optional[Dict[str, bool]] = None,
ping_states: Optional[Dict[str, Optional[bool]]] = None,
) -> dict:
# Accept pre-fetched devices; fall back to empty list if unavailable
display_unifi = unifi_devices if unifi_devices is not None else []
# During a Prometheus outage keep showing the hosts, with their state unknown,
# rather than emptying the dashboard or implying that they're down.
prom_ok = iface_states is not None
shown_states = iface_states if prom_ok else self._last_iface_states
hosts = {}
for instance, ifaces in iface_states.items():
for instance, ifaces in shown_states.items():
if instance not in self._instance_map:
continue # skip Prometheus instances not in config (e.g. LXC app servers)
host = self._hostname(instance)
phys = {k: v for k, v in ifaces.items()}
up_count = sum(1 for v in phys.values() if v)
total = len(phys)
if total == 0 or up_count == total:
if not prom_ok:
status = 'unknown'
elif total == 0 or up_count == total:
status = 'up'
elif up_count == 0:
status = 'down'
@@ -914,13 +1399,23 @@ class NetworkMonitor:
'source': 'prometheus',
}
# Configured hosts with no data: down if Prometheus is fine, otherwise unknown
for instance, host in self._instance_map.items():
if instance not in shown_states:
hosts[host] = {
'ip': instance.split(':')[0],
'interfaces': {},
'status': 'down' if prom_ok else 'unknown',
'source': 'prometheus',
}
for h in self.cfg.get('monitor', {}).get('ping_hosts', []):
name, ip = h['name'], h['ip']
reachable = (ping_states or {}).get(name, False)
reachable = (ping_states or {}).get(name)
hosts[name] = {
'ip': ip,
'interfaces': {},
'status': 'up' if reachable else 'down',
'status': 'unknown' if reachable is None else ('up' if reachable else 'down'),
'source': 'ping',
}
@@ -949,13 +1444,15 @@ class NetworkMonitor:
unifi_devices = self.unifi.get_devices()
# 3a. Ping-only hosts once — shared by snapshot and alert processing
ping_states: Dict[str, bool] = {
ping_states: Dict[str, Optional[bool]] = {
h['name']: self.pulse.ping(h['ip'])
for h in self.cfg.get('monitor', {}).get('ping_hosts', [])
}
# 3b. Collect and store snapshot for dashboard
snapshot = self._collect_snapshot(iface_states, unifi_devices, ping_states)
if iface_states is not None:
self._last_iface_states = iface_states
db.set_state('network_snapshot', snapshot)
db.set_state('last_check', _now_utc())
@@ -968,14 +1465,29 @@ class NetworkMonitor:
# 5. Process alerts using already-fetched data
suppressions = db.get_active_suppressions()
if iface_states is None:
self._process_prometheus_outage(suppressions)
else:
db.resolve_event('monitor_source_unavailable', 'prometheus', '')
self._process_interfaces(iface_states, suppressions)
self._process_unifi(unifi_devices, suppressions)
self._process_ping_hosts(suppressions, ping_states)
# WAN modem: isolated so a failure here can't stop the rest of the cycle
if self.modem:
try:
# Log first, so a restart it reports can explain an uptime reset below
self._process_wan_log(suppressions)
self._collect_modem(suppressions)
except Exception as e:
logger.error(f'Modem collection failed: {e}', exc_info=True)
# Housekeeping: deactivate expired suppressions and purge old resolved events
db.cleanup_expired_suppressions()
db.purge_old_resolved_events(days=90)
if self.modem:
db.purge_old_modem_samples(days=90)
logger.info('Network check cycle complete')
+9
View File
@@ -46,6 +46,15 @@ CREATE TABLE IF NOT EXISTS suppression_rules (
INDEX idx_sup_lookup (active, target_type, target_name, target_detail)
) ENGINE=InnoDB;
-- ── WAN modem optical history (one row per poll) ──────────────────────
-- Also created automatically at monitor startup (db.ensure_modem_schema).
CREATE TABLE IF NOT EXISTS modem_samples (
sampled_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP PRIMARY KEY,
rx_power_dbm DECIMAL(5,1) NULL,
tx_power_dbm DECIMAL(5,1) NULL,
temp_c DECIMAL(5,1) NULL
) ENGINE=InnoDB;
-- ── Monitor state (key/value store for snapshot + baseline) ───────────
CREATE TABLE IF NOT EXISTS monitor_state (
key_name VARCHAR(100) PRIMARY KEY,
+8 -13
View File
@@ -6,6 +6,7 @@ import asyncio
import io
import pathlib
import sys
from urllib.parse import quote
from PIL import Image
from playwright.async_api import async_playwright
@@ -81,18 +82,12 @@ async def shots():
if hb:
await pg.locator("#hosts-grid, .host-grid, .hosts-grid").first.screenshot(path=f"{OUT}/host-cards.png")
print("host-cards ok")
await go(pg, "/links")
await pg.screenshot(path=f"{OUT}/link-debug.png")
names = await pg.evaluate("[...document.querySelectorAll('.link-host-name')].map(e=>e.textContent.trim())")
print("hosts", names)
for nm, fn in (
("large1", "link-debug-sfp-warning"),
("compute-storage-gpu-01", "link-debug-sfp"),
("Pro 24 PoE", "link-debug-switch"),
):
i = next((k for k, n in enumerate(names) if nm in n), None)
if i is not None:
await el(pg, ".link-host-panel", fn, i)
await go(pg, "/device/large1")
await pg.locator('.device-row-toggle[data-row="enp5s0"]').click()
await pg.wait_for_timeout(300)
await pg.screenshot(path=f"{OUT}/device-host.png", full_page=True)
await go(pg, "/device/" + quote("Pro 24 PoE"))
await pg.screenshot(path=f"{OUT}/device-switch.png", full_page=True)
await go(pg, "/inspector")
await pg.screenshot(path=f"{OUT}/inspector.png")
await select_port(pg, "USW-Aggregation", 4)
@@ -133,7 +128,7 @@ async def gif():
await snap(3, 600)
await pg.mouse.wheel(0, 700)
await snap(2, 600)
await go(pg, "/links", 4500)
await go(pg, "/device/large1", 4500)
await snap(3)
await pg.mouse.wheel(0, 500)
await snap(2, 500)
+261 -1
View File
@@ -1,4 +1,6 @@
'use strict';
/* exported initDeviceSearch */
/* global DEVICE_NAME */
// ── Auto-redirect on auth timeout ─────────────────────────────────────
// Wraps fetch so a 401 (Authelia session expired) forces a full reload.
@@ -40,6 +42,7 @@ async function refreshAll() {
const [netResult, statusResult] = await Promise.allSettled([
lt.api.get('/api/network'),
lt.api.get('/api/status'),
refreshModem(),
]);
if (netResult.status === 'fulfilled') {
const net = netResult.value;
@@ -170,6 +173,263 @@ function updateTopology(hosts) {
});
}
// ── Device search ─────────────────────────────────────────────────────
function deviceUrl(name) {
return `/device/${encodeURIComponent(name)}`;
}
// All words must match somewhere; the device's name matching ranks first.
function rankDevices(devices, query) {
const words = query.trim().toLowerCase().split(/\s+/).filter(Boolean);
if (!words.length) return [];
const scored = [];
for (const d of devices) {
if (!words.every(w => d.search.includes(w))) continue;
const name = d.name.toLowerCase();
let score = 3;
if (name === words[0]) score = 0;
else if (name.startsWith(words[0])) score = 1;
else if (name.includes(words[0]) || (d.ip || '').startsWith(words[0])) score = 2;
scored.push({ d, score });
}
return scored
.sort((a, b) => a.score - b.score || a.d.name.localeCompare(b.d.name))
.slice(0, 12)
.map(s => s.d);
}
function initDeviceSearch() {
const input = document.getElementById('device-search-input');
const results = document.getElementById('device-search-results');
if (!input || !results) return;
let devices = [];
let matches = [];
let active = 0;
async function loadDevices() {
try {
devices = (await lt.api.get('/api/devices')).devices || [];
} catch (err) {
console.warn('Device list unavailable', err);
}
}
function close() {
results.hidden = true;
input.setAttribute('aria-expanded', 'false');
input.removeAttribute('aria-activedescendant');
}
function render() {
matches = rankDevices(devices, input.value);
if (!input.value.trim()) { close(); return; }
active = Math.min(active, Math.max(matches.length - 1, 0));
results.hidden = false;
input.setAttribute('aria-expanded', 'true');
if (!matches.length) {
results.innerHTML = '<div class="device-search-empty">No devices match.</div>';
input.removeAttribute('aria-activedescendant');
return;
}
results.innerHTML = matches.map((d, i) => `
<a class="device-search-row${i === active ? ' is-active' : ''}" id="device-opt-${i}" role="option"
aria-selected="${i === active}" href="${lt.escHtml(deviceUrl(d.name))}">
<span class="dot-${lt.escHtml(d.status)}"></span>
<span class="device-search-name">${lt.escHtml(d.name)}</span>
<span class="device-search-kind">${lt.escHtml(d.kind)}</span>
<span class="device-search-meta">${lt.escHtml(d.ip || '')}</span>
<span class="device-search-meta">${lt.escHtml(d.model || '')}</span>
</a>`).join('');
input.setAttribute('aria-activedescendant', `device-opt-${active}`);
}
input.addEventListener('input', () => { active = 0; render(); });
input.addEventListener('focus', () => { loadDevices().then(render); });
input.addEventListener('keydown', e => {
if (e.key === 'Escape') { close(); return; }
if (!matches.length) return;
if (e.key === 'ArrowDown') { e.preventDefault(); active = (active + 1) % matches.length; render(); }
if (e.key === 'ArrowUp') { e.preventDefault(); active = (active - 1 + matches.length) % matches.length; render(); }
if (e.key === 'Enter') { e.preventDefault(); window.location.href = deviceUrl(matches[active].name); }
});
document.addEventListener('click', e => {
if (!e.target.closest('#device-search')) close();
});
document.addEventListener('keydown', e => {
const typing = e.target.closest('input, textarea, select, [contenteditable]');
if (e.key === '/' && !typing && !e.ctrlKey && !e.metaKey) {
e.preventDefault();
input.focus();
}
});
loadDevices();
}
// ── WAN modem / fiber panel ───────────────────────────────────────────
async function refreshModem() {
try {
updateModem(await lt.api.get('/api/modem'));
} catch (err) {
console.warn('Modem data unavailable', err);
}
}
function _modemUptime(seconds) {
if (seconds === null || seconds === undefined) return '–';
const d = Math.floor(seconds / 86400);
const h = Math.floor((seconds % 86400) / 3600);
const m = Math.floor((seconds % 3600) / 60);
return d ? `${d}d ${h}h` : `${h}h ${m}m`;
}
function _modemRate(bytesPerSec) {
if (bytesPerSec === null || bytesPerSec === undefined) return '–';
const mbps = bytesPerSec * 8 / 1e6;
return mbps >= 1000 ? `${(mbps / 1000).toFixed(2)} Gbps` : `${mbps.toFixed(1)} Mbps`;
}
function _modemStat(label, value, cls, note) {
return `
<div class="modem-stat">
<span class="modem-stat-label">${lt.escHtml(label)}</span>
<span class="modem-stat-value ${cls || ''}">${lt.escHtml(String(value))}</span>
${note ? `<span class="modem-stat-note">${lt.escHtml(note)}</span>` : ''}
</div>`;
}
function _modemOptic(label, metric, status, unit) {
if (!metric || metric.value === null || metric.value === undefined) return _modemStat(label, '–');
const cls = { alarm: 'lt-text-red', warning: 'lt-text-amber', ok: 'lt-text-green' }[status] || '';
return _modemStat(label, `${metric.value} ${unit}`, cls, `warn ${metric.warn_low} to ${metric.warn_high}`);
}
function _modemChart(history) {
const points = (history || []).filter(p => p.rx_avg !== null && p.rx_avg !== undefined);
if (points.length < 2) {
return '<div class="modem-chart-empty">Rx power trend appears after a few polls.</div>';
}
const W = 600, H = 70, PAD = 4;
const values = points.map(p => p.rx_avg);
const low = Math.min(...values) - 0.5, high = Math.max(...values) + 0.5;
const t0 = points[0].ts, tSpan = (points[points.length - 1].ts - t0) || 1;
const coord = p => `${((p.ts - t0) / tSpan * W).toFixed(1)},` +
`${(PAD + (high - p.rx_avg) / (high - low) * (H - 2 * PAD)).toFixed(1)}`;
// Break the line where samples are missing rather than drawing across an outage
const gaps = points.slice(1).map((p, i) => p.ts - points[i].ts).sort((a, b) => a - b);
const maxGap = gaps[Math.floor(gaps.length / 2)] * 3;
const segments = [[]];
points.forEach((p, i) => {
if (i && p.ts - points[i - 1].ts > maxGap) segments.push([]);
segments[segments.length - 1].push(coord(p));
});
const lines = segments
.map(s => `<polyline points="${s.join(' ')}" fill="none" vector-effect="non-scaling-stroke"/>`)
.join('');
const summary = `range ${Math.min(...values)} to ${Math.max(...values)} dBm`;
return `
<div class="modem-chart">
<div class="modem-chart-title">Rx power, last 7 days <span>${lt.escHtml(summary)}</span></div>
<svg viewBox="0 0 ${W} ${H}" preserveAspectRatio="none" role="img"
aria-label="Rx power trend, ${lt.escHtml(summary)}">${lines}</svg>
</div>`;
}
function _modemWhen(epochSeconds) {
return new Date(epochSeconds * 1000).toLocaleString([], { dateStyle: 'short', timeStyle: 'short' });
}
function _modemLastRestart(history) {
const restarts = (history && history.restarts) || [];
if (!restarts.length) return '';
const last = restarts[restarts.length - 1];
return `restarted ${_modemWhen(last.at)}${last.by ? ` by ${last.by}` : ''}`;
}
function _modemDuration(seconds) {
if (seconds < 60) return `${Math.round(seconds)}s`;
if (seconds < 3600) return `${Math.floor(seconds / 60)}m ${Math.round(seconds % 60)}s`;
return `${Math.floor(seconds / 3600)}h ${Math.floor((seconds % 3600) / 60)}m`;
}
function _modemOutages(history) {
const outages = ((history && history.outages) || []).slice(-5).reverse();
if (!outages.length) return '';
const rows = outages.map(o => {
const length = o.restored_at === null || o.restored_at === undefined
? '<span class="lt-text-red">ongoing</span>'
: lt.escHtml(_modemDuration(o.restored_at - o.failed_at));
return `<li><span>${lt.escHtml(_modemWhen(o.failed_at))}</span><span>${length}</span></li>`;
}).join('');
return `
<div class="modem-outages">
<div class="modem-chart-title">Recent WAN outages</div>
<ul>${rows}</ul>
</div>`;
}
function updateModem(data) {
const section = document.getElementById('modem-section');
const body = document.getElementById('modem-body');
if (!section || !body) return;
section.hidden = !data.configured;
if (!data.configured) return;
// On the dashboard, link to the modem's page. On that page, it would link to itself.
const detailsLink = document.getElementById('modem-device-link');
if (detailsLink && typeof DEVICE_NAME === 'undefined' && data.name) {
detailsLink.href = deviceUrl(data.name);
detailsLink.hidden = false;
}
const rxLabel = document.getElementById('topo-modem-rx');
const topo = document.getElementById('topo-modem');
const title = document.getElementById('modem-title');
if (!data.reachable) {
if (rxLabel) rxLabel.textContent = 'unreachable';
if (topo) topo.classList.add('topo-v2-status-down');
body.innerHTML = `
<div class="modem-error">Modem status unavailable: ${lt.escHtml(data.error || 'unknown error')}
<span class="modem-stat-note">Last attempt ${lt.escHtml(data.updated || '–')}</span></div>
${_modemChart(data.history)}`;
return;
}
const problems = data.wan_problems || [];
const optics = data.optics || {};
const status = data.optics_status || {};
const counters = data.counters || {};
const rates = data.rates || {};
const rx = optics.rx_power ? optics.rx_power.value : null;
if (topo) topo.classList.toggle('topo-v2-status-down', problems.length > 0);
if (rxLabel) rxLabel.textContent = (rx === null || rx === undefined) ? 'Rx –' : `Rx ${rx} dBm`;
if (title) {
title.textContent = [data.model || 'AT&T Gateway', data.firmware && `fw ${data.firmware}`]
.filter(Boolean).join(' · ');
}
const errs = (counters.rx_errors || 0) + (counters.tx_errors || 0);
const drops = (counters.rx_drops || 0) + (counters.tx_drops || 0);
body.innerHTML = `
${problems.length ? `<div class="modem-error">WAN down: ${lt.escHtml(problems.join('; '))}</div>` : ''}
<div class="modem-stats">
${_modemStat('WAN', problems.length ? 'DOWN' : 'UP', problems.length ? 'lt-text-red' : 'lt-text-green',
data.wan_ip || '')}
${_modemStat('PON', data.pon_state || '–', '', data.wavelength || '')}
${_modemOptic('Rx power', optics.rx_power, status.rx_power, 'dBm')}
${_modemOptic('Tx power', optics.tx_power, status.tx_power, 'dBm')}
${_modemOptic('Optics temp', optics.temperature, status.temperature, '°C')}
${_modemStat('Throughput', `↓ ${_modemRate(rates.rx_bytes)}`, '', `↑ ${_modemRate(rates.tx_bytes)}`)}
${_modemStat('Errors / drops', `${errs} / ${drops}`, errs ? 'lt-text-amber' : '', 'since modem reboot')}
${_modemStat('Uptime', _modemUptime(data.uptime_s), '', _modemLastRestart(data.wan_history))}
</div>
${_modemChart(data.history)}
${_modemOutages(data.wan_history)}
<div class="modem-stat-note modem-updated">Updated ${lt.escHtml(data.updated || '–')}</div>`;
}
function updateUnifiTable(devices) {
const tbody = document.querySelector('#unifi-table tbody');
if (!tbody || !devices.length) return;
@@ -188,7 +448,7 @@ function updateUnifiTable(devices) {
return `
<tr class="${statusClass}">
<td><span class="${dotClass}"></span> ${statusText}</td>
<td><strong>${lt.escHtml(d.name)}</strong></td>
<td><a class="device-link" href="${lt.escHtml(deviceUrl(d.name))}"><strong>${lt.escHtml(d.name)}</strong></a></td>
<td>${lt.escHtml(d.type)}</td>
<td>${lt.escHtml(d.model)}</td>
<td>${lt.escHtml(d.ip)}</td>
+454
View File
@@ -0,0 +1,454 @@
'use strict';
/* global DEVICE_NAME, deviceUrl, refreshModem, _toIso */
// Per-device page: overview, alerts, and a compact interface/port table whose
// rows expand to show everything the old Link Debug page showed.
const esc = s => lt.escHtml(String(s === null || s === undefined ? '' : s));
const _openRows = new Set(); // survives auto-refresh
let _filter = 'all';
let _query = '';
let _device = null;
// ── Formatting ───────────────────────────────────────────────────────
function fmtRate(bytesPerSec) {
if (bytesPerSec === null || bytesPerSec === undefined) return '–';
const bps = bytesPerSec * 8;
if (bps < 1e3) return bps.toFixed(0) + ' bps';
if (bps < 1e6) return (bps / 1e3).toFixed(1) + ' Kbps';
if (bps < 1e9) return (bps / 1e6).toFixed(2) + ' Mbps';
return (bps / 1e9).toFixed(3) + ' Gbps';
}
function fmtSpeed(mbps) {
if (!mbps) return '–';
return mbps >= 1000 ? (mbps / 1000).toFixed(0) + ' Gbps' : mbps + ' Mbps';
}
function fmtErrors(rate) {
if (rate === null || rate === undefined) return '–';
if (rate < 0.001) return '<span class="val-good">0 /s</span>';
return `<span class="val-crit">${rate.toFixed(3)} /s</span>`;
}
function fmtUnit(value, unit, digits) {
return value === null || value === undefined ? '–' : `${value.toFixed(digits)} ${unit}`;
}
function fmtUptime(seconds) {
if (seconds === null || seconds === undefined) return '–';
const d = Math.floor(seconds / 86400), h = Math.floor((seconds % 86400) / 3600);
const m = Math.floor((seconds % 3600) / 60);
return d ? `${d}d ${h}h` : `${h}h ${m}m`;
}
function ratePct(bytesPerSec, mbps) {
if (!mbps || !bytesPerSec) return 0;
return Math.min(100, (bytesPerSec * 8 / 1e6 / mbps) * 100);
}
function barClass(pct, isTx) {
if (pct > 85) return 'lt-progress--red';
if (pct > 65) return 'lt-progress--amber';
return isTx ? '' : 'lt-progress--cyan';
}
// ── SFP/DOM value classification ─────────────────────────────────────
function rxClass(dbm) {
if (dbm === null || dbm === undefined) return 'val-neutral';
if (dbm < -15) return 'val-crit';
return dbm < -10 ? 'val-warn' : 'val-good';
}
function txClass(dbm) {
if (dbm === null || dbm === undefined) return 'val-neutral';
return dbm < -5 ? 'val-crit' : 'val-good';
}
function tempClass(c) {
if (c === null || c === undefined) return 'val-neutral';
if (c > 80) return 'val-crit';
return c > 70 ? 'val-warn' : 'val-good';
}
function voltClass(v) {
if (v === null || v === undefined) return 'val-neutral';
if (v < 3.0 || v > 3.6) return 'val-crit';
return (v < 3.1 || v > 3.5) ? 'val-warn' : 'val-good';
}
function powerFill(cls) {
return cls === 'val-good' ? 'power-ok' : cls === 'val-warn' ? 'power-warn' : 'power-crit';
}
function badges(d) {
const out = [];
if ((d.tx_errs_rate || 0) > 0.001 || (d.rx_errs_rate || 0) > 0.001) out.push('<span class="link-alert-badge">ERR</span>');
if ((d.tx_drops_rate || 0) > 0.001 || (d.rx_drops_rate || 0) > 0.001) out.push('<span class="link-alert-badge link-alert-amber">DROP</span>');
if ((d.carrier_changes || 0) > 3) out.push('<span class="link-alert-badge link-alert-amber">FLAP</span>');
return out.join('');
}
function stat(label, valueHtml, tooltip) {
const tip = tooltip ? ` data-tooltip="${esc(tooltip)}"` : '';
return `<div class="link-stat"><span class="link-stat-label"${tip}>${esc(label)}</span>` +
`<span class="link-stat-value">${valueHtml}</span></div>`;
}
function trafficBars(d) {
const tx = ratePct(d.tx_bytes_rate, d.speed_mbps), rx = ratePct(d.rx_bytes_rate, d.speed_mbps);
const row = (label, pct, rate, isTx) => `
<div class="traffic-row">
<span class="traffic-label">${label}</span>
<div class="lt-progress ${barClass(pct, isTx)}"><div class="lt-progress-bar" style="width:${pct}%"></div></div>
<span class="traffic-value">${fmtRate(rate)}</span>
</div>`;
return `<div class="traffic-section">${row('TX', tx, d.tx_bytes_rate, true)}${row('RX', rx, d.rx_bytes_rate, false)}</div>`;
}
function sfpPanel(s) {
if (!s || !Object.keys(s).length) return '';
const rx = rxClass(s.rx_power_dbm), tx = txClass(s.tx_power_dbm);
const has = v => v !== null && v !== undefined;
const rxPct = has(s.rx_power_dbm) ? Math.min(100, Math.max(0, (s.rx_power_dbm + 20) / 15 * 100)) : 0;
const txPct = has(s.tx_power_dbm) ? Math.min(100, Math.max(0, (s.tx_power_dbm + 10) / 8 * 100)) : 0;
const item = (label, cls, value, tip, pct) => `
<div class="sfp-stat">
<span class="sfp-stat-label" data-tooltip="${esc(tip)}">${label}</span>
<span class="sfp-stat-value ${cls}">${value}</span>
${pct === undefined ? '' : `<div class="power-row"><div class="power-track"><div class="power-fill ${powerFill(cls)}" style="width:${pct}%"></div></div></div>`}
</div>`;
const vendor = [s.vendor, s.part_no, s.sfp_type, s.connector, s.wavelength_nm && `${s.wavelength_nm} nm`]
.filter(Boolean).map(esc).join(' / ');
return `
<div class="sfp-panel">
<div class="sfp-vendor-row">${vendor}</div>
<div class="sfp-grid">
${item('Temp', tempClass(s.temp_c), fmtUnit(s.temp_c, '°C', 1), 'Module temperature. Normal: below 70°C.')}
${item('Voltage', voltClass(s.voltage_v), fmtUnit(s.voltage_v, 'V', 2), 'Supply voltage. Normal: 3.1–3.5 V.')}
${item('Bias', '', fmtUnit(s.bias_ma, 'mA', 2), 'Laser bias current. High values may mean an aging laser.')}
${item('TX Power', tx, fmtUnit(s.tx_power_dbm, 'dBm', 2), 'Optical transmit power.', txPct)}
${item('RX Power', rx, fmtUnit(s.rx_power_dbm, 'dBm', 2), 'Optical receive power. Very low values may mean dirty or damaged fiber.', rxPct)}
${has(s.rx_power_dbm) && has(s.tx_power_dbm) ? item('RX−TX Δ', '', `${(s.rx_power_dbm - s.tx_power_dbm).toFixed(2)} dBm`, 'Difference between receive and transmit power.') : ''}
</div>
</div>`;
}
// ── Normalized rows, so hosts and switches share one table ───────────
function hostRows(interfaces) {
return Object.entries(interfaces)
.sort(([a], [b]) => a.localeCompare(b, undefined, { numeric: true }))
.map(([name, d]) => {
const up = d.state ? d.state === 'up' : d.link_detected !== false;
return {
id: name, label: name, up, data: d, sub: d.port_type || '',
detail: `
<div class="link-stats-grid">
${stat('Duplex', esc(d.duplex || '–'), 'Full = both directions at once. Half can cause collisions.')}
${stat('Auto-neg', d.auto_neg === null || d.auto_neg === undefined ? '–' : d.auto_neg ? 'On' : 'Off')}
${stat('Carrier Δ', d.carrier_changes === null || d.carrier_changes === undefined ? '–' : esc(d.carrier_changes), 'Times the link went up or down. High values mean flapping.')}
${stat('TX Err/s', fmtErrors(d.tx_errs_rate))}
${stat('RX Err/s', fmtErrors(d.rx_errs_rate))}
${stat('TX Drop/s', fmtErrors(d.tx_drops_rate))}
${stat('RX Drop/s', fmtErrors(d.rx_drops_rate))}
</div>
${trafficBars(d)}
${sfpPanel(d.sfp)}`,
};
});
}
function portRows(ports) {
return Object.entries(ports)
.sort(([, a], [, b]) => (a.port_idx || 0) - (b.port_idx || 0))
.map(([name, d]) => {
const neighbor = d.lldp && d.lldp.system_name;
return {
id: name, label: name, num: d.port_idx, up: !!d.up, data: d,
neighbor: d.lldp ? { name: neighbor, port: d.lldp.port_id } : null,
sub: [d.is_uplink ? 'uplink' : '', d.media || '', d.poe_power ? `PoE ${d.poe_power.toFixed(1)} W` : ''].filter(Boolean).join(' · '),
detail: `
<div class="link-stats-grid">
${stat('Duplex', d.full_duplex === null || d.full_duplex === undefined ? '–' : d.full_duplex ? 'Full' : 'Half')}
${stat('Auto-neg', d.autoneg === null || d.autoneg === undefined ? '–' : d.autoneg ? 'On' : 'Off')}
${stat('TX Err/s', fmtErrors(d.tx_errs_rate))}
${stat('RX Err/s', fmtErrors(d.rx_errs_rate))}
${stat('TX Drop/s', fmtErrors(d.tx_drops_rate))}
${stat('RX Drop/s', fmtErrors(d.rx_drops_rate))}
${d.poe_power || d.poe_max_power ? stat('PoE', `${esc(d.poe_class || '–')} · max ${fmtUnit(d.poe_max_power, 'W', 1)}`) : ''}
</div>
${d.lldp ? `<div class="port-lldp">LLDP: ${esc(d.lldp.system_name || '')} (${esc(d.lldp.port_id || '')})${d.lldp.chassis_id ? ` · ${esc(d.lldp.chassis_id)}` : ''}${(d.lldp.mgmt_ips || []).length ? ` · ${esc(d.lldp.mgmt_ips.join(', '))}` : ''}</div>` : ''}
${trafficBars(d)}`,
};
});
}
// Down alone isn't a problem: most down ports are simply unused.
function rowHasProblem(r) {
return !!badges(r.data);
}
function renderTable(rows) {
const q = _query.trim().toLowerCase();
const shown = rows.filter(r => {
if (_filter === 'up' && !r.up) return false;
if (_filter === 'problems' && !rowHasProblem(r)) return false;
return !q || `${r.label} ${r.num || ''} ${r.sub} ${(r.neighbor && r.neighbor.name) || ''}`.toLowerCase().includes(q);
});
if (!shown.length) return '<div class="lt-empty-state lt-empty-state--sm"><div class="lt-empty-state-title">Nothing matches this filter</div></div>';
const isPorts = rows.some(r => r.num !== undefined);
const body = shown.map(r => {
const open = _openRows.has(r.id);
const d = r.data;
const neighbor = r.neighbor
? (r.neighbor.name ? `<a class="device-link" href="${esc(deviceUrl(r.neighbor.name))}">${esc(r.neighbor.name)}</a>` : esc(r.neighbor.port || ''))
: '–';
return `
<tr class="device-row${r.up ? '' : ' device-row--down'}">
<td><button type="button" class="device-row-toggle" data-row="${esc(r.id)}" aria-expanded="${open}" aria-label="Details for ${esc(r.label)}">${open ? '▾' : '▸'}</button></td>
<td><span class="dot-${r.up ? 'up' : 'down'}"></span></td>
<td class="device-row-name">${r.num !== undefined ? `<span class="port-badge port-badge-num">#${esc(r.num)}</span> ` : ''}${esc(r.label)}<div class="device-row-sub">${esc(r.sub)}</div></td>
<td class="device-num">${r.up ? fmtSpeed(d.speed_mbps) : '–'}</td>
<td class="device-num device-col-wide">${fmtRate(d.rx_bytes_rate)}</td>
<td class="device-num device-col-wide">${fmtRate(d.tx_bytes_rate)}</td>
${isPorts ? `<td class="device-col-wide">${neighbor}</td>` : ''}
<td>${badges(d) || (r.up ? '<span class="val-good">OK</span>' : '<span class="device-row-sub">down</span>')}</td>
</tr>
${open ? `<tr class="device-row-detail"><td colspan="${isPorts ? 8 : 7}">${r.detail}</td></tr>` : ''}`;
}).join('');
return `
<div class="lt-table-wrap">
<table class="lt-table device-table">
<caption class="lt-sr-only">Interfaces</caption>
<thead><tr>
<th><span class="lt-sr-only">Expand</span></th><th><span class="lt-sr-only">State</span></th>
<th>${isPorts ? 'Port' : 'Interface'}</th><th>Speed</th><th class="device-col-wide">RX</th><th class="device-col-wide">TX</th>
${isPorts ? '<th class="device-col-wide">Neighbor</th>' : ''}<th>Health</th>
</tr></thead>
<tbody>${body}</tbody>
</table>
</div>`;
}
function section(title, inner, id) {
return `
<section class="g-section"${id ? ` id="${id}"` : ''}>
<div class="g-section-header"><h2 class="g-section-title">${esc(title)}</h2></div>
<div class="lt-frame">
<span class="lt-frame-bl">&#x255A;</span><span class="lt-frame-br">&#x255D;</span>
${inner}
</div>
</section>`;
}
function kvGrid(pairs) {
return `<div class="modem-stats">${pairs.filter(([, v]) => v !== null && v !== undefined && v !== '').map(([k, v]) => `
<div class="modem-stat"><span class="modem-stat-label">${esc(k)}</span><span class="modem-stat-value">${esc(v)}</span></div>`).join('')}</div>`;
}
function renderOverview(d) {
const info = d.info || {};
const pct = v => (v === null || v === undefined ? null : `${v.toFixed(0)}%`);
const temps = (info.temps || []).filter(t => t.value_c !== null).map(t => `${t.name} ${t.value_c.toFixed(0)}°C`).join(', ');
return section('Overview', kvGrid([
['Status', (d.status || 'unknown').toUpperCase()],
['Type', d.kind],
['IP', d.ip],
['MAC', d.mac],
['Model', d.model],
['Firmware', info.firmware],
['Uptime', info.uptime_s === undefined ? null : fmtUptime(info.uptime_s)],
['CPU', pct(info.cpu_pct)],
['Memory', pct(info.mem_pct)],
['Clients', info.clients],
['Temperature', temps],
['Power draw', info.power_w === null || info.power_w === undefined ? null : `${info.power_w.toFixed(0)} W`],
['Data source', d.source],
]));
}
function renderUplinks(d) {
if (!(d.uplinks || []).length) return '';
const rows = d.uplinks.map(u => `
<tr>
<td><span class="dot-${u.up ? 'up' : 'down'}"></span></td>
<td>${esc(u.host_interface || '–')}</td>
<td><a class="device-link" href="${esc(deviceUrl(u.switch))}">${esc(u.switch)}</a></td>
<td>#${esc(u.port_idx)} ${esc(u.port)}</td>
<td>${fmtSpeed(u.speed_mbps)}</td>
</tr>`).join('');
return section('Connected to', `<div class="lt-table-wrap"><table class="lt-table">
<caption class="lt-sr-only">Switch ports this host connects to, from LLDP</caption>
<thead><tr><th><span class="lt-sr-only">State</span></th><th>Interface</th><th>Switch</th><th>Port</th><th>Speed</th></tr></thead>
<tbody>${rows}</tbody></table></div>`);
}
function renderWan(d) {
const wan = (d.info && d.info.wan) || {};
const names = Object.keys(wan);
if (!names.length) return '';
const rows = names.map(n => {
const w = wan[n];
return `<tr><td><span class="dot-${w.up ? 'up' : 'down'}"></span></td><td>${esc(n.toUpperCase())}</td>
<td>${fmtSpeed(w.speed_mbps)}</td><td>${w.latency_ms === null || w.latency_ms === undefined ? '–' : `${esc(w.latency_ms)} ms`}</td>
<td>${w.availability === null || w.availability === undefined ? '–' : `${esc(w.availability)}%`}</td></tr>`;
}).join('');
return section('WAN', `<div class="lt-table-wrap"><table class="lt-table">
<caption class="lt-sr-only">WAN connections</caption>
<thead><tr><th><span class="lt-sr-only">State</span></th><th>Connection</th><th>Speed</th><th>Latency</th><th>Availability</th></tr></thead>
<tbody>${rows}</tbody></table></div>`);
}
function renderOutlets(d) {
const outlets = (d.info && d.info.outlets) || [];
if (!outlets.length) return '';
const rows = outlets.map(o => `
<tr>
<td><span class="dot-${o.on ? 'up' : 'down'}"></span></td>
<td>#${esc(o.index)}</td>
<td>${esc(o.name || '–')}</td>
<td class="device-num">${o.power_w === null || o.power_w === undefined ? '–' : `${o.power_w.toFixed(1)} W`}</td>
<td>${o.auto_power_cycle ? '<span class="lt-badge">auto power-cycle</span>' : ''}</td>
</tr>`).join('');
return section('Outlets', `<div class="lt-table-wrap"><table class="lt-table">
<caption class="lt-sr-only">PDU outlets</caption>
<thead><tr><th><span class="lt-sr-only">State</span></th><th>Outlet</th><th>Name</th><th>Power</th><th></th></tr></thead>
<tbody>${rows}</tbody></table></div>`);
}
function renderEvents(events) {
const body = !(events || []).length
? '<div class="lt-empty-state lt-empty-state--sm"><div class="lt-empty-state-icon">✔</div><div class="lt-empty-state-title">No recorded alerts</div></div>'
: `<div class="lt-table-wrap"><table class="lt-table">
<caption class="lt-sr-only">Alerts for this device</caption>
<thead><tr><th>Sev</th><th>Type</th><th>Detail</th><th>Description</th><th>First seen</th><th>State</th></tr></thead>
<tbody>${events.map(e => `
<tr class="${e.resolved_at ? '' : `row-${esc(e.severity)}`}">
<td><span class="lt-badge badge-${e.resolved_at ? 'resolved' : esc(e.severity)}">${esc(e.severity)}</span>${e.is_suppressed && !e.resolved_at ? '<span class="lt-badge badge-suppressed">🔕 sup</span>' : ''}</td>
<td>${esc(e.event_type.replace(/_/g, ' '))}</td>
<td>${esc(e.target_detail || '–')}</td>
<td class="desc-cell">${esc(e.description || '')}</td>
<td class="ts-cell" title="${esc(e.first_seen || '')}">${esc(lt.time.ago(_toIso(e.first_seen)))}</td>
<td>${e.resolved_at ? `resolved ${esc(lt.time.ago(_toIso(e.resolved_at)))}` : '<strong>active</strong>'}</td>
</tr>`).join('')}</tbody></table></div>`;
return section('Alerts', body);
}
function currentRows() {
if (!_device) return [];
if (_device.ports) return portRows(_device.ports);
if (_device.interfaces) return hostRows(_device.interfaces);
return [];
}
function renderInterfaces() {
const wrap = document.getElementById('device-table-wrap');
if (wrap) wrap.innerHTML = renderTable(currentRows());
}
function renderInterfaceSection(rows, title) {
if (!rows.length) return '';
const up = rows.filter(r => r.up).length;
const problems = rows.filter(rowHasProblem).length;
const pill = (value, label) => `<button type="button" class="pill${_filter === value ? ' active' : ''}" data-device-filter="${value}" aria-pressed="${_filter === value}">${label}</button>`;
return section(`${title} · ${up} of ${rows.length} up`, `
<div class="lt-toolbar device-toolbar">
<div class="lt-toolbar-left">
<div class="lt-search">
<input type="search" class="lt-input lt-search-input lt-search-input--sm" id="device-table-search"
placeholder="Filter…" autocomplete="off" aria-label="Filter ${esc(title.toLowerCase())}" value="${esc(_query)}">
</div>
<div class="sev-pills" role="group" aria-label="Filter by state">
${pill('all', 'All')}${pill('up', 'Up')}${pill('problems', `Errors (${problems})`)}
</div>
</div>
</div>
<div id="device-table-wrap"></div>`);
}
function render(d) {
_device = d;
document.getElementById('device-dot').className = `dot-${d.status || 'unknown'}`;
document.getElementById('device-subtitle').textContent =
[d.kind, d.ip, d.model].filter(Boolean).join(' · ') +
(d.updated ? ` · updated ${new Date(_toIso(d.updated)).toLocaleTimeString()}` : '');
const unifi = d.source === 'unifi';
document.getElementById('device-actions').innerHTML = `
${d.ports && Object.keys(d.ports).length ? `<a class="lt-btn lt-btn-secondary lt-btn-sm" href="/inspector">⬡ Inspector</a>` : ''}
<button class="lt-btn lt-btn-ghost lt-btn-sm btn-suppress" data-sup-type="${unifi ? 'unifi_device' : 'host'}"
data-sup-name="${esc(d.name)}" data-sup-detail="" aria-label="Suppress alerts for ${esc(d.name)}">🔕 Suppress</button>`;
// Keep focus and caret in the filter box across auto-refresh
const active = document.activeElement && document.activeElement.id === 'device-table-search';
const caret = active ? document.activeElement.selectionStart : 0;
const rows = currentRows();
document.getElementById('device-root').innerHTML =
renderOverview(d) +
renderEvents(d.events) +
renderWan(d) +
renderUplinks(d) +
renderOutlets(d) +
renderInterfaceSection(rows, d.ports ? 'Ports' : 'Interfaces');
renderInterfaces();
if (active) {
const box = document.getElementById('device-table-search');
box.focus();
box.setSelectionRange(caret, caret);
}
if (d.kind === 'modem') refreshModem();
}
async function loadDevice() {
try {
render(await lt.api.get(`/api/device/${encodeURIComponent(DEVICE_NAME)}`));
} catch (err) {
if (_device) {
lt.toast.error('Could not refresh device data');
return;
}
const notFound = /404|No device/.test(err.message || '');
document.getElementById('device-root').innerHTML = `
<div class="lt-empty-state">
<div class="lt-empty-state-icon">✕</div>
<div class="lt-empty-state-title">${notFound ? 'No device with this name' : 'Could not load this device'}</div>
<div class="lt-empty-state-body">${notFound ? 'It may have been renamed, or the monitor may not have seen it yet.' : 'Check that the Gandalf service is healthy, then refresh.'}</div>
</div>`;
}
}
document.addEventListener('click', e => {
const toggle = e.target.closest('.device-row-toggle');
if (toggle) {
const id = toggle.dataset.row;
if (_openRows.has(id)) _openRows.delete(id); else _openRows.add(id);
renderInterfaces();
return;
}
const pill = e.target.closest('[data-device-filter]');
if (pill) {
_filter = pill.dataset.deviceFilter;
document.querySelectorAll('[data-device-filter]').forEach(p => {
const on = p === pill;
p.classList.toggle('active', on);
p.setAttribute('aria-pressed', String(on));
});
renderInterfaces();
}
});
document.addEventListener('input', e => {
if (e.target.id === 'device-table-search') {
_query = e.target.value;
renderInterfaces();
}
});
loadDevice();
const _deviceInterval = window.gandalfSettings?.refreshInterval ?? 60;
if (_deviceInterval > 0) lt.autoRefresh.start(loadDevice, Math.max(_deviceInterval, 15) * 1000);
window.onGandalfSettingsChanged = function (s) {
lt.autoRefresh.stop();
if (s.refreshInterval > 0) lt.autoRefresh.start(loadDevice, Math.max(s.refreshInterval, 15) * 1000);
};
+85
View File
@@ -1064,3 +1064,88 @@
.inspector-panel.open { width: 100%; }
.inspector-panel-inner{ width: 100%; }
}
/* ── WAN modem / fiber panel ──────────────────────────────────────── */
.modem-stats {
display: grid;
grid-template-columns: repeat(4, minmax(0, 1fr));
gap: 12px;
padding: 12px;
}
@media (max-width: 700px) {
.modem-stats { grid-template-columns: repeat(2, minmax(0, 1fr)); }
}
.modem-stat { display: flex; flex-direction: column; gap: 2px; min-width: 0; }
.modem-stat-label { font-size: .68em; color: var(--text-muted); text-transform: uppercase; letter-spacing: .05em; }
.modem-stat-value { font-size: 1.05em; font-weight: bold; overflow-wrap: anywhere; }
.modem-stat-note { font-size: .7em; color: var(--text-muted); display: block; }
.modem-error { margin: 12px; padding: 8px 10px; border: 1px solid var(--terminal-red, #f44); color: var(--terminal-red, #f44); }
.modem-chart { padding: 0 12px 8px; }
.modem-chart-title { font-size: .72em; color: var(--text-muted); margin-bottom: 4px; }
.modem-chart-title span { margin-left: 8px; opacity: .8; }
.modem-chart svg { display: block; width: 100%; height: 70px; border: 1px solid var(--border-color); }
.modem-chart svg polyline { stroke: var(--terminal-cyan, #0ff); stroke-width: 1.5; }
.modem-chart-empty { padding: 0 12px 8px; font-size: .72em; color: var(--text-muted); }
.modem-updated { padding: 0 12px 10px; }
/* ── Device search (home page) ────────────────────────────────────── */
.device-search { position: relative; }
.device-search label { display: block; margin-bottom: 6px; }
.device-search .lt-search { width: 100%; }
.device-search-input { width: 100%; font-size: 1em; padding-top: 10px; padding-bottom: 10px; }
.device-search-results {
position: absolute; left: 0; right: 0; z-index: 50; margin-top: 4px;
background: var(--bg-card, var(--bg-secondary)); border: 1px solid var(--border-color);
max-height: 420px; overflow-y: auto;
}
.device-search-results[hidden] { display: none; }
.device-search-row {
display: grid; grid-template-columns: 14px minmax(0, 1.4fr) 70px minmax(0, 1fr) minmax(0, 1fr);
gap: 10px; align-items: center; padding: 9px 12px; color: inherit; text-decoration: none;
border-bottom: 1px solid var(--border-color);
}
.device-search-row:last-child { border-bottom: 0; }
.device-search-row:hover, .device-search-row.is-active { background: var(--bg-tertiary); }
.device-search-name { font-weight: bold; overflow-wrap: anywhere; }
.device-search-kind { font-size: .72em; color: var(--text-muted); text-transform: uppercase; }
.device-search-meta { font-size: .8em; color: var(--text-muted); overflow-wrap: anywhere; }
.device-search-empty { padding: 12px; color: var(--text-muted); }
@media (max-width: 700px) {
.device-search-row { grid-template-columns: 14px minmax(0, 1fr) auto; }
.device-search-row .device-search-meta { grid-column: 2 / -1; }
}
/* ── Device page ───────────────────────────────────────────────────── */
a.device-link { color: inherit; text-decoration: none; border-bottom: 1px dotted var(--text-muted); }
a.device-link:hover { color: var(--text-accent); border-bottom-color: currentColor; }
.device-title { display: flex; align-items: center; gap: 10px; overflow-wrap: anywhere; }
.device-actions { display: flex; gap: 8px; flex-wrap: wrap; align-items: center; }
.device-toolbar { padding: 8px 12px; margin: 0; }
.device-table td { vertical-align: middle; }
.device-num { white-space: nowrap; font-variant-numeric: tabular-nums; }
.device-row--down { opacity: .55; }
.device-row-name { overflow-wrap: anywhere; }
.device-row-sub { font-size: .72em; color: var(--text-muted); }
.device-row-toggle { background: none; border: 0; color: inherit; cursor: pointer; font: inherit; padding: 2px 6px; }
.device-row-detail > td { background: var(--bg-tertiary); padding: 10px 14px; }
.topo-host[data-host] { cursor: pointer; }
/* WAN outage list in the modem panel */
.modem-outages { padding: 0 12px 8px; }
.modem-outages ul { list-style: none; margin: 0; padding: 0; border: 1px solid var(--border-color); }
.modem-outages li { display: flex; justify-content: space-between; gap: 12px; padding: 4px 8px; font-size: .8em; }
.modem-outages li + li { border-top: 1px solid var(--border-color); }
.g-section-actions[hidden] { display: none; }
/* Rows open a detail panel holding the fields that are hidden here on narrow screens */
.device-row-detail .link-stats-grid { grid-template-columns: repeat(auto-fit, minmax(110px, 1fr)); }
.device-table th:nth-child(-n+2), .device-table td:nth-child(-n+2) { width: 1%; white-space: nowrap; }
@media (max-width: 700px) {
.device-col-wide { display: none; }
.device-table th, .device-table td { padding-left: 6px; padding-right: 6px; }
}
.device-search-row.is-active { box-shadow: inset 3px 0 0 var(--terminal-cyan, #0ff); }
@media (max-width: 700px) {
/* Scroll inside the wrapper rather than squeezing every column to a few letters */
#events-table { min-width: 760px; }
}
+12
View File
@@ -0,0 +1,12 @@
<section class="g-section" id="modem-section" hidden>
<div class="g-section-header">
<h2 class="g-section-title">WAN / Fiber</h2>
<a class="g-section-actions lt-btn lt-btn-ghost lt-btn-sm" id="modem-device-link" hidden>↗ Details</a>
</div>
<div class="lt-frame">
<span class="lt-frame-bl">&#x255A;</span>
<span class="lt-frame-br">&#x255D;</span>
<div class="lt-section-header" id="modem-title">AT&amp;T Gateway</div>
<div id="modem-body" aria-live="polite"></div>
</div>
</section>
+4 -11
View File
@@ -5,6 +5,7 @@
<meta name="viewport" content="width=device-width, initial-scale=1.0, viewport-fit=cover">
<meta name="theme-color" content="#030508">
<meta name="robots" content="noindex, nofollow">
<link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 16 16'%3E%3Crect width='16' height='16' rx='3' fill='%23030508'/%3E%3Cpath d='M8 2l5 3v6l-5 3-5-3V5z' fill='none' stroke='%2300e5ff' stroke-width='1.5'/%3E%3C/svg%3E">
<title>{% block title %}GANDALF{% endblock %} — GANDALF</title>
<link rel="preconnect" href="https://fonts.googleapis.com">
@@ -36,9 +37,6 @@
<a href="{{ url_for('index') }}"
class="lt-nav-drawer-link{% if request.endpoint == 'index' %} active{% endif %}"
{% if request.endpoint == 'index' %}aria-current="page"{% endif %}>Dashboard</a>
<a href="{{ url_for('links_page') }}"
class="lt-nav-drawer-link{% if request.endpoint == 'links_page' %} active{% endif %}"
{% if request.endpoint == 'links_page' %}aria-current="page"{% endif %}>Link Debug</a>
<a href="{{ url_for('inspector') }}"
class="lt-nav-drawer-link{% if request.endpoint == 'inspector' %} active{% endif %}"
{% if request.endpoint == 'inspector' %}aria-current="page"{% endif %}>Inspector</a>
@@ -84,11 +82,6 @@
{% if request.endpoint == 'index' %}aria-current="page"{% endif %}>
Dashboard
</a>
<a href="{{ url_for('links_page') }}"
class="lt-nav-link{% if request.endpoint == 'links_page' %} active{% endif %}"
{% if request.endpoint == 'links_page' %}aria-current="page"{% endif %}>
Link Debug
</a>
<a href="{{ url_for('inspector') }}"
class="lt-nav-link{% if request.endpoint == 'inspector' %} active{% endif %}"
{% if request.endpoint == 'inspector' %}aria-current="page"{% endif %}>
@@ -186,7 +179,7 @@
<span class="lt-footer-sep">|</span>
<span class="lt-footer-hint"><span class="lt-footer-key">[ S ]</span> SUPPRESS</span>
<span class="lt-footer-sep">|</span>
{% elif request.endpoint in ('links_page', 'inspector') %}
{% elif request.endpoint in ('device_page', 'inspector') %}
<span class="lt-footer-hint"><span class="lt-footer-key">[ R ]</span> REFRESH</span>
<span class="lt-footer-sep">|</span>
{% endif %}
@@ -262,7 +255,8 @@
<thead><tr><th>Shortcut</th><th>Action</th></tr></thead>
<tbody>
<tr><td>Ctrl / &#x2318; + K</td><td>Command palette</td></tr>
<tr><td>R</td><td>Refresh data (Dashboard / Link Debug / Inspector)</td></tr>
<tr><td>R</td><td>Refresh data (Dashboard / Device / Inspector)</td></tr>
<tr><td>/</td><td>Search devices (Dashboard)</td></tr>
<tr><td>S</td><td>Quick-suppress alert (Dashboard)</td></tr>
<tr><td>*</td><td>Open settings</td></tr>
<tr><td>?</td><td>Show this help</td></tr>
@@ -330,7 +324,6 @@
// Command palette
lt.cmdPalette.init([
{ id: 'nav-dashboard', group: 'Navigate', icon: '~', label: 'Dashboard', action: function() { window.location.href = '/'; } },
{ id: 'nav-links', group: 'Navigate', icon: '↗', label: 'Link Debug', action: function() { window.location.href = '/links'; } },
{ id: 'nav-inspector', group: 'Navigate', icon: '⬡', label: 'Inspector', action: function() { window.location.href = '/inspector'; } },
{ id: 'nav-suppressions', group: 'Navigate', icon: '🔕', label: 'Suppressions', action: function() { window.location.href = '/suppressions'; } },
{ id: 'action-refresh', group: 'Actions', icon: '↻', label: 'Refresh Data', kbd: 'R', action: function() { lt.autoRefresh.now(); } },
+25
View File
@@ -0,0 +1,25 @@
{% extends "base.html" %}
{% block title %}{{ device_name }} – GANDALF{% endblock %}
{% block content %}
<div class="lt-page-header">
<div>
<p class="g-page-sub"><a href="{{ url_for('index') }}">← Dashboard</a></p>
<h1 class="lt-page-title device-title">
<span id="device-dot" class="dot-unknown"></span>
<span>{{ device_name }}</span>
</h1>
<p class="g-page-sub" id="device-subtitle"></p>
</div>
<div id="device-actions" class="device-actions"></div>
</div>
<div id="device-root"><div class="link-loading">Loading device</div></div>
{% include "_modem_panel.html" %}
{% endblock %}
{% block scripts %}
<script>const DEVICE_NAME = {{ device_name | tojson }};</script>
<script src="{{ url_for('static', filename='device.js') }}"></script>
{% endblock %}
+161 -128
View File
@@ -64,6 +64,17 @@
</div>
</div>
<!-- ── Device search ──────────────────────────────────────────────── -->
<section class="g-section device-search" id="device-search" role="search">
<label class="g-section-title" for="device-search-input">Find a device</label>
<div class="lt-search">
<input type="search" class="lt-input lt-search-input device-search-input" id="device-search-input"
placeholder="Hostname, IP, MAC, model, interface… ( / )" autocomplete="off" spellcheck="false"
role="combobox" aria-expanded="false" aria-controls="device-search-results" aria-autocomplete="list">
</div>
<div class="device-search-results" id="device-search-results" role="listbox" hidden></div>
</section>
<!-- ── Active alerts ─────────────────────────────── (above the fold) -->
<section class="g-section">
<div class="g-section-header">
@@ -166,10 +177,129 @@
</div><!-- /.lt-frame -->
</section>
<!-- ── Network topology + host grid ───────────────────────────────── -->
<!-- ── WAN / fiber ──────────────────────────────────────────────────── -->
{% include "_modem_panel.html" %}
<!-- ── Hosts ────────────────────────────────────────────────────────── -->
<section class="g-section">
<div class="g-section-header">
<h2 class="g-section-title">Network Hosts</h2>
<h2 class="g-section-title">Hosts</h2>
</div>
<div class="host-grid" id="host-grid">
{%- set has_global_sup = suppressions | selectattr('target_type', 'equalto', 'all') | list | length > 0 -%}
{% for name, host in snapshot.hosts.items() %}
{% set suppressed = suppressions | selectattr('target_name', 'equalto', name) | list %}
<div class="host-card host-card-{{ host.status }}" data-host="{{ name }}">
<div class="host-card-header">
<div class="host-name-row">
<span class="host-status-dot dot-{{ host.status }}"></span>
<a class="host-name device-link" href="{{ url_for('device_page', name=name) }}">{{ name }}</a>
{% if suppressed or has_global_sup %}
<span class="badge-suppressed" title="Suppressed">🔕</span>
{% endif %}
</div>
<div class="host-meta">
<span class="host-ip">{{ host.ip }}</span>
<span class="host-source source-{{ host.source }}">{{ host.source }}</span>
</div>
</div>
{% if host.interfaces %}
<div class="iface-list">
{% for iface, state in host.interfaces.items() | sort %}
<div class="iface-row">
<span class="iface-dot dot-{{ state }}"></span>
<span class="iface-name">{{ iface }}</span>
<span class="iface-state state-{{ state }}">{{ state }}</span>
</div>
{% endfor %}
</div>
{% else %}
<div class="host-ping-note">{% if host.source == 'ping' %}ping-only / no node_exporter{% else %}no metrics from node_exporter{% endif %}</div>
{% endif %}
<div class="host-actions">
<button class="lt-btn lt-btn-ghost lt-btn-sm btn-suppress"
data-sup-type="host"
data-sup-name="{{ name }}"
data-sup-detail=""
aria-label="Suppress alerts for {{ name }}">
🔕 Suppress
</button>
<a href="{{ url_for('device_page', name=name) }}"
class="lt-btn lt-btn-secondary lt-btn-sm">
↗ Details
</a>
</div>
</div>
{% else %}
<div class="lt-empty-state lt-empty-state--sm">
<div class="lt-empty-state-icon">⌛</div>
<div class="lt-empty-state-title">No host data yet</div>
<div class="lt-empty-state-body">The monitor daemon may still be starting up.</div>
</div>
{% endfor %}
</div>
</section>
<!-- ── UniFi devices ────────────────────────────────────────────────── -->
{% if snapshot.unifi %}
<section class="g-section">
<div class="g-section-header">
<h2 class="g-section-title">UniFi Devices</h2>
</div>
<div class="lt-frame">
<span class="lt-frame-bl">&#x255A;</span>
<span class="lt-frame-br">&#x255D;</span>
<div class="lt-section-header">Device Inventory</div>
<div class="lt-table-wrap">
<table class="lt-table" id="unifi-table">
<caption class="lt-sr-only">UniFi network devices</caption>
<thead>
<tr>
<th>Status</th>
<th>Name</th>
<th>Type</th>
<th>Model</th>
<th>IP</th>
<th>Actions</th>
</tr>
</thead>
<tbody>
{% for d in snapshot.unifi %}
<tr class="{% if not d.connected %}row-critical{% endif %}">
<td>
<span class="dot-{{ 'up' if d.connected else 'down' }}"></span>
{{ 'ONLINE' if d.connected else 'OFFLINE' }}
</td>
<td><a class="device-link" href="{{ url_for('device_page', name=d.name) }}"><strong>{{ d.name }}</strong></a></td>
<td>{{ d.type }}</td>
<td>{{ d.model }}</td>
<td>{{ d.ip }}</td>
<td>
{% if not d.connected %}
<button class="lt-btn lt-btn-ghost lt-btn-sm btn-suppress"
data-sup-type="unifi_device"
data-sup-name="{{ d.name }}"
data-sup-detail=""
aria-label="Suppress alerts for {{ d.name }}">
🔕 Suppress
</button>
{% endif %}
</td>
</tr>
{% endfor %}
</tbody>
</table>
</div>
</div>
</section>
{% endif %}
<!-- ── Network topology ───────────────────────────────── -->
<section class="g-section">
<div class="g-section-header">
<h2 class="g-section-title">Network Topology</h2>
<button type="button" class="topo-collapse-btn" id="topo-toggle-btn"
aria-expanded="true" aria-controls="topo-collapsible-wrap">&#x25B4; Collapse</button>
</div>
@@ -191,10 +321,28 @@
</div>
</div>
<!-- WAN wire: cyan → WAN gradient -->
<!-- Fiber wire: Internet → AT&T gateway -->
<div class="topo-vc">
<div class="topo-vc-wire topo-vc-wire--wan"></div>
<span class="topo-vc-label">WAN · 10G SFP+</span>
<span class="topo-vc-label">AT&amp;T Fiber</span>
</div>
<!-- ══════════════════════════════════════════════════════════════
TIER 1b: AT&T gateway (Rx filled in by updateModem)
══════════════════════════════════════════════════════════ -->
<div class="topo-tier">
<div class="topo-v2-node topo-v2-router" id="topo-modem">
<span class="topo-v2-icon">◈</span>
<span class="topo-v2-label">BGW320</span>
<span class="topo-v2-sub">AT&amp;T Gateway</span>
<span class="topo-v2-sub" id="topo-modem-rx">Rx –</span>
</div>
</div>
<!-- WAN wire: gateway → UDM-Pro -->
<div class="topo-vc">
<div class="topo-vc-wire topo-vc-wire--wan"></div>
<span class="topo-vc-label">WAN · Port 9 · 1G</span>
</div>
<!-- ══════════════════════════════════════════════════════════════
@@ -314,126 +462,9 @@
</div><!-- /topo-v2 -->
</div>
<!-- Host cards -->
<div class="lt-toolbar" id="host-toolbar">
<div class="lt-toolbar-left">
<div class="lt-search">
<input type="search" class="lt-input lt-search-input lt-search-input--sm" id="host-search"
placeholder="Filter hosts…" autocomplete="off" aria-label="Filter hosts">
</div>
</div>
</div>
<div class="host-grid" id="host-grid">
{%- set has_global_sup = suppressions | selectattr('target_type', 'equalto', 'all') | list | length > 0 -%}
{% for name, host in snapshot.hosts.items() %}
{% set suppressed = suppressions | selectattr('target_name', 'equalto', name) | list %}
<div class="host-card host-card-{{ host.status }}" data-host="{{ name }}">
<div class="host-card-header">
<div class="host-name-row">
<span class="host-status-dot dot-{{ host.status }}"></span>
<span class="host-name">{{ name }}</span>
{% if suppressed or has_global_sup %}
<span class="badge-suppressed" title="Suppressed">🔕</span>
{% endif %}
</div>
<div class="host-meta">
<span class="host-ip">{{ host.ip }}</span>
<span class="host-source source-{{ host.source }}">{{ host.source }}</span>
</div>
</div>
{% if host.interfaces %}
<div class="iface-list">
{% for iface, state in host.interfaces.items() | sort %}
<div class="iface-row">
<span class="iface-dot dot-{{ state }}"></span>
<span class="iface-name">{{ iface }}</span>
<span class="iface-state state-{{ state }}">{{ state }}</span>
</div>
{% endfor %}
</div>
{% else %}
<div class="host-ping-note">ping-only / no node_exporter</div>
{% endif %}
<div class="host-actions">
<button class="lt-btn lt-btn-ghost lt-btn-sm btn-suppress"
data-sup-type="host"
data-sup-name="{{ name }}"
data-sup-detail=""
aria-label="Suppress alerts for {{ name }}">
🔕 Suppress
</button>
<a href="{{ url_for('links_page') }}#{{ name }}"
class="lt-btn lt-btn-secondary lt-btn-sm">
↗ Links
</a>
</div>
</div>
{% else %}
<div class="lt-empty-state lt-empty-state--sm">
<div class="lt-empty-state-icon">⌛</div>
<div class="lt-empty-state-title">No host data yet</div>
<div class="lt-empty-state-body">The monitor daemon may still be starting up.</div>
</div>
{% endfor %}
</div>
</div><!-- /#topo-collapsible-wrap -->
</section>
<!-- ── UniFi devices ────────────────────────────────────────────────── -->
{% if snapshot.unifi %}
<section class="g-section">
<div class="g-section-header">
<h2 class="g-section-title">UniFi Devices</h2>
</div>
<div class="lt-frame">
<span class="lt-frame-bl">&#x255A;</span>
<span class="lt-frame-br">&#x255D;</span>
<div class="lt-section-header">Device Inventory</div>
<div class="lt-table-wrap">
<table class="lt-table" id="unifi-table">
<caption class="lt-sr-only">UniFi network devices</caption>
<thead>
<tr>
<th>Status</th>
<th>Name</th>
<th>Type</th>
<th>Model</th>
<th>IP</th>
<th>Actions</th>
</tr>
</thead>
<tbody>
{% for d in snapshot.unifi %}
<tr class="{% if not d.connected %}row-critical{% endif %}">
<td>
<span class="dot-{{ 'up' if d.connected else 'down' }}"></span>
{{ 'ONLINE' if d.connected else 'OFFLINE' }}
</td>
<td><strong>{{ d.name }}</strong></td>
<td>{{ d.type }}</td>
<td>{{ d.model }}</td>
<td>{{ d.ip }}</td>
<td>
{% if not d.connected %}
<button class="lt-btn lt-btn-ghost lt-btn-sm btn-suppress"
data-sup-type="unifi_device"
data-sup-name="{{ d.name }}"
data-sup-detail=""
aria-label="Suppress alerts for {{ d.name }}">
🔕 Suppress
</button>
{% endif %}
</td>
</tr>
{% endfor %}
</tbody>
</table>
</div>
</div>
</section>
{% endif %}
<!-- ── Recently Resolved (last 24h) ───────────────────────────────── -->
{% if recent_resolved %}
@@ -468,6 +499,9 @@
{% block scripts %}
<script>
// Load once even when auto-refresh is set to Off
refreshModem();
// Start auto-refresh using saved settings interval (default 30 s)
const _savedInterval = window.gandalfSettings?.refreshInterval ?? 30;
if (_savedInterval > 0) lt.autoRefresh.start(refreshAll, _savedInterval * 1000);
@@ -560,13 +594,12 @@
new MutationObserver(applyEventsFilter)
.observe(document.getElementById('events-table-wrap'), { childList: true, subtree: true });
// Host grid search filter
document.getElementById('host-search')?.addEventListener('input', function() {
const q = this.value.trim().toLowerCase();
document.querySelectorAll('#host-grid .host-card').forEach(card => {
const name = (card.dataset.host || '').toLowerCase();
card.style.display = (!q || name.includes(q)) ? '' : 'none';
});
initDeviceSearch();
// Topology hosts open their device page
document.getElementById('topology-diagram')?.addEventListener('click', e => {
const node = e.target.closest('.topo-host[data-host]');
if (node) window.location.href = deviceUrl(node.dataset.host);
});
// Stat card clicks — filter events table by severity
+1
View File
@@ -45,6 +45,7 @@ const SWITCH_LAYOUTS = {
},
'USPPDUP': { rows: [[1]] },
'USMINI': { rows: [[1,2,3,4,5]] },
'UDMPRO': { rows: [[1,2,3,4,5,6,7,8,9]], sfp_section: [10,11] },
};
// ── Formatting helpers ───────────────────────────────────────────────────
-598
View File
@@ -1,598 +0,0 @@
{% extends "base.html" %}
{% block title %}Link Debug – GANDALF{% endblock %}
{% block content %}
<div class="lt-page-header">
<div>
<h1 class="lt-page-title">Link Debug</h1>
<p class="g-page-sub">
Per-interface stats: speed, duplex, SFP optical levels, TX/RX rates, errors, and carrier changes.
<span id="links-updated" class="g-page-sub-aside"></span>
</p>
</div>
</div>
<div class="lt-toolbar" id="links-toolbar" style="display:none">
<div class="lt-toolbar-left">
<div class="lt-search">
<input type="search" class="lt-input lt-search-input" id="links-search"
placeholder="Filter by host or switch name…" autocomplete="off"
aria-label="Filter by host or switch name">
</div>
</div>
<div class="lt-toolbar-right">
<button class="lt-btn lt-btn-ghost lt-btn-sm" data-action="collapse-all">Collapse All</button>
<button class="lt-btn lt-btn-ghost lt-btn-sm" data-action="expand-all">Expand All</button>
</div>
</div>
<div id="links-container">
<div class="link-loading">Loading link statistics</div>
</div>
{% endblock %}
{% block scripts %}
<script>
const escHtml = s => lt.escHtml(s);
// ── Formatting helpers ────────────────────────────────────────────
function fmtRate(bytesPerSec) {
if (bytesPerSec === null || bytesPerSec === undefined) return '–';
const bps = bytesPerSec * 8;
if (bps < 1e3) return bps.toFixed(0) + ' bps';
if (bps < 1e6) return (bps/1e3).toFixed(1) + ' Kbps';
if (bps < 1e9) return (bps/1e6).toFixed(2) + ' Mbps';
return (bps/1e9).toFixed(3) + ' Gbps';
}
function fmtRateBar(bytesPerSec, linkSpeedMbps) {
if (!linkSpeedMbps || linkSpeedMbps <= 0) return 0;
const mbps = (bytesPerSec * 8) / 1e6;
return Math.min(100, (mbps / linkSpeedMbps) * 100);
}
function trafficBarClass(pct, isTx) {
if (pct > 85) return 'lt-progress--red';
if (pct > 65) return 'lt-progress--amber';
return isTx ? '' : 'lt-progress--cyan';
}
function fmtSpeed(mbps) {
if (mbps === null || mbps === undefined) return '–';
if (mbps >= 1000) return (mbps/1000).toFixed(0) + ' Gbps';
return mbps + ' Mbps';
}
function fmtDuplex(d) {
if (!d) return '–';
return d.charAt(0).toUpperCase() + d.slice(1);
}
function fmtTemp(c) {
if (c === null || c === undefined) return '–';
return c.toFixed(1) + ' °C';
}
function fmtVoltage(v) {
if (v === null || v === undefined) return '–';
return v.toFixed(2) + ' V';
}
function fmtPower(dbm) {
if (dbm === null || dbm === undefined) return '–';
return dbm.toFixed(2) + ' dBm';
}
function fmtBias(ma) {
if (ma === null || ma === undefined) return '–';
return ma.toFixed(2) + ' mA';
}
function fmtErrors(rate) {
if (rate === null || rate === undefined) return '–';
if (rate < 0.001) return '<span class="val-good">0 /s</span>';
return `<span class="val-crit">${rate.toFixed(3)} /s</span>`;
}
function fmtCarrier(n) {
if (n === null || n === undefined) return '–';
if (n === 0) return '<span class="counter-zero">0</span>';
return `<span class="counter-nonzero">${n}</span>`;
}
// ── SFP/DOM value classification ─────────────────────────────────
function rxPowerClass(dbm) {
if (dbm === null || dbm === undefined) return 'val-neutral';
if (dbm < -15) return 'val-crit';
if (dbm < -10) return 'val-warn';
return 'val-good';
}
function txPowerClass(dbm) {
if (dbm === null || dbm === undefined) return 'val-neutral';
if (dbm < -5) return 'val-crit';
return 'val-good';
}
function tempClass(c) {
if (c === null || c === undefined) return 'val-neutral';
if (c > 80) return 'val-crit';
if (c > 70) return 'val-warn';
return 'val-good';
}
function voltageClass(v) {
if (v === null || v === undefined) return 'val-neutral';
if (v < 3.0 || v > 3.6) return 'val-crit';
if (v < 3.1 || v > 3.5) return 'val-warn';
return 'val-good';
}
function errorBadges(d) {
const badges = [];
if ((d.tx_errs_rate || 0) > 0.001 || (d.rx_errs_rate || 0) > 0.001)
badges.push('<span class="link-alert-badge">ERR</span>');
if ((d.tx_drops_rate || 0) > 0.001 || (d.rx_drops_rate || 0) > 0.001)
badges.push('<span class="link-alert-badge link-alert-amber">DROP</span>');
if ((d.carrier_changes || 0) > 3)
badges.push('<span class="link-alert-badge link-alert-amber">FLAP</span>');
return badges.join('');
}
// ── Render a single server interface card ─────────────────────────
function renderIfaceCard(ifaceName, d) {
const isDown = d.link_detected === false;
const pt = (d.port_type || '').toUpperCase();
const mediaTag = pt === 'FIBRE' || pt === 'SFP' || pt.includes('FIBRE') ? 'type-fibre'
: pt === 'DA' ? 'type-da'
: 'type-copper';
const mediaLabel = d.port_type || '–';
const speedStr = d.speed_mbps ? fmtSpeed(d.speed_mbps) : '–';
const txPct = fmtRateBar(d.tx_bytes_rate, d.speed_mbps);
const rxPct = fmtRateBar(d.rx_bytes_rate, d.speed_mbps);
let sfpHtml = '';
if (d.sfp && Object.keys(d.sfp).length > 0) {
const s = d.sfp;
const rxClass = rxPowerClass(s.rx_power_dbm);
const txClass = txPowerClass(s.tx_power_dbm);
const tmpClass = tempClass(s.temp_c);
const vClass = voltageClass(s.voltage_v);
const rxPct2 = s.rx_power_dbm != null ? Math.min(100, Math.max(0, (s.rx_power_dbm + 20) / 15 * 100)) : 0;
const txPct2 = s.tx_power_dbm != null ? Math.min(100, Math.max(0, (s.tx_power_dbm + 10) / 8 * 100)) : 0;
sfpHtml = `
<div class="sfp-panel">
<div class="sfp-vendor-row">
${s.vendor ? `<span>${escHtml(s.vendor)}</span>` : ''}
${s.part_no ? ` / <span>${escHtml(s.part_no)}</span>` : ''}
</div>
<div class="sfp-grid">
<div class="sfp-stat">
<span class="sfp-stat-label" data-tooltip="SFP module temperature. Normal: below 70°C. Warn: 70–85°C. Critical: above 85°C.">Temp</span>
<span class="sfp-stat-value ${tmpClass}">${fmtTemp(s.temp_c)}</span>
</div>
<div class="sfp-stat">
<span class="sfp-stat-label" data-tooltip="SFP supply voltage. Normal: 3.1–3.5V.">Voltage</span>
<span class="sfp-stat-value ${vClass}">${fmtVoltage(s.voltage_v)}</span>
</div>
<div class="sfp-stat">
<span class="sfp-stat-label" data-tooltip="Laser bias current in mA. High values may indicate end-of-life laser diode.">Bias</span>
<span class="sfp-stat-value">${fmtBias(s.bias_ma)}</span>
</div>
<div class="sfp-stat">
<span class="sfp-stat-label" data-tooltip="Optical transmit power in dBm. Typical good range: -3 to -9 dBm.">TX Power</span>
<span class="sfp-stat-value ${txClass}">${fmtPower(s.tx_power_dbm)}</span>
<div class="power-row">
<div class="power-track"><div class="power-fill ${txClass === 'val-good' ? 'power-ok' : txClass === 'val-warn' ? 'power-warn' : 'power-crit'}" style="width:${txPct2}%"></div></div>
</div>
</div>
<div class="sfp-stat">
<span class="sfp-stat-label" data-tooltip="Optical receive power in dBm. Typical good range: -3 to -18 dBm. Below -20 dBm may indicate dirty/damaged fiber.">RX Power</span>
<span class="sfp-stat-value ${rxClass}">${fmtPower(s.rx_power_dbm)}</span>
<div class="power-row">
<div class="power-track"><div class="power-fill ${rxClass === 'val-good' ? 'power-ok' : rxClass === 'val-warn' ? 'power-warn' : 'power-crit'}" style="width:${rxPct2}%"></div></div>
</div>
</div>
${s.rx_power_dbm != null && s.tx_power_dbm != null ? `
<div class="sfp-stat">
<span class="sfp-stat-label" data-tooltip="Insertion loss: difference between transmit and receive power. Large negative values indicate fiber loss or connector issues.">RX−TX Δ</span>
<span class="sfp-stat-value">${(s.rx_power_dbm - s.tx_power_dbm).toFixed(2)} dBm</span>
</div>` : ''}
</div>
</div>`;
}
return `
<div class="link-iface-card ${isDown ? 'port-down' : ''}">
<div class="link-iface-header">
<span class="link-iface-name">${escHtml(ifaceName)}</span>
<span class="link-iface-speed">${speedStr}</span>
<span class="link-iface-type ${mediaTag}">${escHtml(mediaLabel)}</span>
${errorBadges(d)}
</div>
<div class="link-stats-grid">
<div class="link-stat">
<span class="link-stat-label">Link</span>
<span class="link-stat-value ${isDown ? 'val-crit' : 'val-good'}">${isDown ? 'DOWN' : 'UP'}</span>
</div>
<div class="link-stat">
<span class="link-stat-label" data-tooltip="Full = simultaneous send/receive at full speed. Half = one direction at a time, can cause collisions.">Duplex</span>
<span class="link-stat-value">${fmtDuplex(d.duplex)}</span>
</div>
<div class="link-stat">
<span class="link-stat-label" data-tooltip="Autonegotiation: NIC and switch automatically agree on link speed and duplex mode.">Auto-neg</span>
<span class="link-stat-value">${d.auto_neg == null ? '–' : d.auto_neg ? 'On' : 'Off'}</span>
</div>
<div class="link-stat">
<span class="link-stat-label" data-tooltip="Carrier changes: number of times the link went up or down. High values indicate a flapping or unstable cable/SFP.">Carrier Δ</span>
<span class="link-stat-value">${fmtCarrier(d.carrier_changes)}</span>
</div>
<div class="link-stat">
<span class="link-stat-label" data-tooltip="Transmit errors per second reported by the kernel network driver.">TX Err/s</span>
<span class="link-stat-value">${fmtErrors(d.tx_errs_rate)}</span>
</div>
<div class="link-stat">
<span class="link-stat-label" data-tooltip="Receive errors per second reported by the kernel network driver.">RX Err/s</span>
<span class="link-stat-value">${fmtErrors(d.rx_errs_rate)}</span>
</div>
<div class="link-stat">
<span class="link-stat-label" data-tooltip="Transmit packets dropped per second (ring buffer full or driver overrun).">TX Drop/s</span>
<span class="link-stat-value">${fmtErrors(d.tx_drops_rate)}</span>
</div>
<div class="link-stat">
<span class="link-stat-label" data-tooltip="Receive packets dropped per second (ring buffer full or driver overrun).">RX Drop/s</span>
<span class="link-stat-value">${fmtErrors(d.rx_drops_rate)}</span>
</div>
</div>
<div class="traffic-section">
<div class="traffic-row">
<span class="traffic-label" data-tooltip="Transmit — outgoing traffic from this server">TX</span>
<div class="lt-progress ${trafficBarClass(txPct, true)}"><div class="lt-progress-bar" style="width:${txPct}%"></div></div>
<span class="traffic-value">${fmtRate(d.tx_bytes_rate)}</span>
</div>
<div class="traffic-row">
<span class="traffic-label" data-tooltip="Receive — incoming traffic to this server">RX</span>
<div class="lt-progress ${trafficBarClass(rxPct, false)}"><div class="lt-progress-bar" style="width:${rxPct}%"></div></div>
<span class="traffic-value">${fmtRate(d.rx_bytes_rate)}</span>
</div>
</div>
${sfpHtml}
</div>`;
}
// ── Render a single UniFi switch port card ────────────────────────
function renderPortCard(portName, d) {
const isDown = !d.up;
const speedStr = d.speed_mbps ? fmtSpeed(d.speed_mbps) : '–';
const txPct = fmtRateBar(d.tx_bytes_rate, d.speed_mbps);
const rxPct = fmtRateBar(d.rx_bytes_rate, d.speed_mbps);
const numBadge = d.port_idx != null ? `<span class="port-badge port-badge-num">#${d.port_idx}</span>` : '';
const uplinkBadge = d.is_uplink ? `<span class="port-badge port-badge-uplink">UPLINK</span>` : '';
const poeBadge = d.poe_power ? `<span class="port-badge port-badge-poe">PoE ${d.poe_power.toFixed(1)}W</span>` : '';
const lldpLine = d.lldp ? `<div class="port-lldp">→ ${escHtml(d.lldp.system_name || '')} (${escHtml(d.lldp.port_id || '')})</div>` : '';
const poeLine = d.poe_class ? `<div class="port-poe-info">PoE ${escHtml(d.poe_class)} · max ${d.poe_max_power != null ? d.poe_max_power.toFixed(1)+'W' : '–'}</div>` : '';
return `
<div class="link-iface-card ${isDown ? 'port-down' : ''}">
<div class="link-iface-header">
<span class="link-iface-name">${numBadge} ${escHtml(portName)}</span>
<span class="link-iface-speed">${speedStr}</span>
${uplinkBadge}${poeBadge}
${errorBadges(d)}
</div>
${lldpLine}${poeLine}
<div class="link-stats-grid">
<div class="link-stat">
<span class="link-stat-label">Link</span>
<span class="link-stat-value ${isDown ? 'val-crit' : 'val-good'}">${isDown ? 'DOWN' : 'UP'}</span>
</div>
<div class="link-stat">
<span class="link-stat-label">Duplex</span>
<span class="link-stat-value">${d.full_duplex == null ? '–' : d.full_duplex ? 'Full' : 'Half'}</span>
</div>
<div class="link-stat">
<span class="link-stat-label">Auto-neg</span>
<span class="link-stat-value">${d.autoneg == null ? '–' : d.autoneg ? 'On' : 'Off'}</span>
</div>
<div class="link-stat">
<span class="link-stat-label">TX Err/s</span>
<span class="link-stat-value">${fmtErrors(d.tx_errs_rate)}</span>
</div>
<div class="link-stat">
<span class="link-stat-label">RX Err/s</span>
<span class="link-stat-value">${fmtErrors(d.rx_errs_rate)}</span>
</div>
<div class="link-stat">
<span class="link-stat-label">TX Drop/s</span>
<span class="link-stat-value">${fmtErrors(d.tx_drops_rate)}</span>
</div>
</div>
<div class="traffic-section">
<div class="traffic-row">
<span class="traffic-label">TX</span>
<div class="lt-progress ${trafficBarClass(txPct, true)}"><div class="lt-progress-bar" style="width:${txPct}%"></div></div>
<span class="traffic-value">${fmtRate(d.tx_bytes_rate)}</span>
</div>
<div class="traffic-row">
<span class="traffic-label">RX</span>
<div class="lt-progress ${trafficBarClass(rxPct, false)}"><div class="lt-progress-bar" style="width:${rxPct}%"></div></div>
<span class="traffic-value">${fmtRate(d.rx_bytes_rate)}</span>
</div>
</div>
</div>`;
}
// ── Render all UniFi switches ─────────────────────────────────────
function renderUnifiSwitches(unifiSwitches, dataUpdated) {
if (!unifiSwitches || !Object.keys(unifiSwitches).length) return '';
const updStr = dataUpdated
? new Date(_toIso(dataUpdated)).toLocaleTimeString()
: '';
const html = Object.entries(unifiSwitches).map(([swName, sw]) => {
const ports = sw.ports || {};
const portValues = Object.values(ports);
const portCards = Object.entries(ports)
.sort(([,a],[,b]) => (a.port_idx||0) - (b.port_idx||0))
.map(([pname, d]) => renderPortCard(pname, d)).join('');
const poe_total_w = portValues.reduce((s, p) => s + (p.poe_power || 0), 0);
const poe_max_w = portValues.reduce((s, p) => s + (p.poe_max_power || 0), 0);
const poeLoad = poe_total_w > 0 ? ` · PoE ${poe_total_w.toFixed(1)}W` : '';
// PoE utilisation bar
let poebar = '';
if (poe_total_w > 0 && poe_max_w > 0) {
const pct = Math.min(100, (poe_total_w / poe_max_w) * 100);
const cls = pct > 80 ? 'poe-bar-crit' : pct > 60 ? 'poe-bar-warn' : 'poe-bar-ok';
poebar = `<div class="poe-bar-track"><div class="poe-bar-fill ${cls}" style="width:${pct}%"></div></div>`;
}
return `
<div class="link-host-panel" id="panel-${CSS.escape(swName)}">
<div class="link-host-title" data-action="toggle-panel" role="button" tabindex="0" aria-expanded="true">
<span class="link-host-name">${escHtml(swName)}</span>
<span class="link-host-ip">${escHtml(sw.ip || '')}</span>
<span class="link-host-upd">${escHtml(sw.model || '')}${updStr ? ' · ' + updStr : ''}${poeLoad}</span>
${poebar}
<span class="panel-toggle">[–]</span>
</div>
<div class="link-ifaces-grid">${portCards}</div>
</div>`;
}).join('');
return `<div class="lt-divider lt-divider--unifi"><span class="lt-divider-label lt-divider-label--unifi">UNIFI SWITCH PORTS</span></div>${html}`;
}
// ── Panel collapse / expand ───────────────────────────────────────
function togglePanel(panel) {
panel.classList.toggle('collapsed');
const isCollapsed = panel.classList.contains('collapsed');
const btn = panel.querySelector('.panel-toggle');
const title = panel.querySelector('.link-host-title');
if (btn) btn.textContent = isCollapsed ? '[+]' : '[–]';
if (title) title.setAttribute('aria-expanded', isCollapsed ? 'false' : 'true');
const id = panel.id;
if (id) {
let collapsed = {};
try { collapsed = JSON.parse(sessionStorage.getItem('linksCollapsed') || '{}'); } catch(_) {}
collapsed[id] = panel.classList.contains('collapsed');
try { sessionStorage.setItem('linksCollapsed', JSON.stringify(collapsed)); } catch(_) {}
}
}
function restoreCollapseState() {
let collapsed = {};
try { collapsed = JSON.parse(sessionStorage.getItem('linksCollapsed') || '{}'); } catch(_) {}
for (const [id, isCollapsed] of Object.entries(collapsed)) {
const panel = document.getElementById(id);
if (!panel) continue;
if (isCollapsed) {
panel.classList.add('collapsed');
const btn = panel.querySelector('.panel-toggle');
const title = panel.querySelector('.link-host-title');
if (btn) btn.textContent = '[+]';
if (title) title.setAttribute('aria-expanded', 'false');
}
}
}
// ── Build summary stats header ────────────────────────────────────
function buildLinkSummary(hosts, unifiSwitches) {
let totalIfaces = 0, downIfaces = 0, errIfaces = 0, totalPoe = 0;
for (const ifaces of Object.values(hosts || {})) {
for (const d of Object.values(ifaces)) {
totalIfaces++;
if (d.link_detected === false) downIfaces++;
if ((d.tx_errs_rate || 0) > 0 || (d.rx_errs_rate || 0) > 0) errIfaces++;
}
}
let swTotal = 0, swDown = 0;
for (const sw of Object.values(unifiSwitches || {})) {
for (const p of Object.values(sw.ports || {})) {
totalPoe += p.poe_power || 0;
swTotal++;
if (!p.up) swDown++;
}
}
const allTotal = totalIfaces + swTotal;
const allDown = downIfaces + swDown;
const downCls = allDown > 0 ? 'lt-text-red' : 'lt-text-green';
const errCls = errIfaces > 0 ? 'lt-text-amber' : 'lt-text-green';
const downCardCls = allDown > 0 ? ' lt-stat-card--alert' : '';
const poeCard = totalPoe > 0 ? `
<div class="lt-stat-card">
<span class="lt-stat-icon lt-text-amber" aria-hidden="true">⚡</span>
<div class="lt-stat-info">
<span class="lt-stat-value lt-text-amber">${totalPoe.toFixed(1)}</span>
<span class="lt-stat-label">PoE Load (W)</span>
</div>
</div>` : '';
return `
<div class="lt-stats-grid lt-stats-grid--mb">
<div class="lt-stat-card">
<span class="lt-stat-icon lt-text-cyan" aria-hidden="true">⬡</span>
<div class="lt-stat-info">
<span class="lt-stat-value lt-text-cyan">${allTotal}</span>
<span class="lt-stat-label">Interfaces</span>
</div>
</div>
<div class="lt-stat-card${downCardCls}">
<span class="lt-stat-icon ${downCls}" aria-hidden="true">●</span>
<div class="lt-stat-info">
<span class="lt-stat-value ${downCls}">${allDown}</span>
<span class="lt-stat-label">Ports Down</span>
</div>
</div>
<div class="lt-stat-card">
<span class="lt-stat-icon ${errCls}" aria-hidden="true">▲</span>
<div class="lt-stat-info">
<span class="lt-stat-value ${errCls}">${errIfaces}</span>
<span class="lt-stat-label">With Errors</span>
</div>
</div>
${poeCard}
</div>`;
}
// ── Main render ───────────────────────────────────────────────────
function renderLinks(data) {
const hosts = data.hosts || {};
const unifiSwitches = data.unifi_switches || {};
const parts = [];
parts.push(buildLinkSummary(hosts, unifiSwitches));
parts.push('<div class="link-host-list">');
for (const [hostname, ifaces] of Object.entries(hosts)) {
const ifaceCards = Object.entries(ifaces)
.sort(([a],[b]) => a.localeCompare(b))
.map(([iname, d]) => renderIfaceCard(iname, d)).join('');
const sample = Object.values(ifaces)[0] || {};
const ip = sample.host_ip || '';
const updStr = data.updated
? new Date(_toIso(data.updated)).toLocaleTimeString()
: '';
parts.push(`
<div class="link-host-panel" id="panel-${CSS.escape(hostname)}">
<div class="link-host-title" data-action="toggle-panel" role="button" tabindex="0" aria-expanded="true">
<span class="link-host-name">${escHtml(hostname)}</span>
<span class="link-host-ip">${escHtml(ip)}</span>
<span class="link-host-upd">${updStr}</span>
<span class="panel-toggle">[–]</span>
</div>
<div class="link-ifaces-grid">${ifaceCards}</div>
</div>`);
}
parts.push(renderUnifiSwitches(unifiSwitches, data.updated));
parts.push('</div>');
document.getElementById('links-container').innerHTML = parts.join('');
restoreCollapseState();
document.getElementById('links-toolbar').style.display = '';
applyLinksSearch();
}
// ── Host/switch search filter ─────────────────────────────────────
function applyLinksSearch() {
const q = (document.getElementById('links-search')?.value || '').trim().toLowerCase();
document.querySelectorAll('.link-host-panel').forEach(panel => {
const text = (panel.querySelector('.link-host-name')?.textContent || '').toLowerCase();
panel.style.display = (!q || text.includes(q)) ? '' : 'none';
});
}
function collapseAll() {
document.querySelectorAll('.link-host-panel').forEach(p => {
p.classList.add('collapsed');
const btn = p.querySelector('.panel-toggle');
const title = p.querySelector('.link-host-title');
if (btn) btn.textContent = '[+]';
if (title) title.setAttribute('aria-expanded', 'false');
});
try {
sessionStorage.setItem('linksCollapsed', JSON.stringify(
Object.fromEntries([...document.querySelectorAll('.link-host-panel')].map(p => [p.id, true]))
));
} catch(_) {}
}
function expandAll() {
document.querySelectorAll('.link-host-panel').forEach(p => {
p.classList.remove('collapsed');
const btn = p.querySelector('.panel-toggle');
const title = p.querySelector('.link-host-title');
if (btn) btn.textContent = '[–]';
if (title) title.setAttribute('aria-expanded', 'true');
});
try { sessionStorage.setItem('linksCollapsed', '{}'); } catch(_) {}
}
// ── Stale data warning ────────────────────────────────────────────
function checkLinksStale(updatedStr) {
if (!updatedStr) return;
const age = (Date.now() - new Date(_toIso(updatedStr))) / 1000;
let banner = document.getElementById('links-stale-banner');
if (age > 120) {
if (!banner) {
banner = document.createElement('div');
banner.id = 'links-stale-banner';
banner.className = 'lt-alert lt-alert--warning';
banner.innerHTML = '<span class="lt-alert-icon">⚠</span><div class="lt-alert-body"><div class="lt-alert-msg"></div></div>';
document.getElementById('links-container').prepend(banner);
}
banner.querySelector('.lt-alert-msg').textContent =
`Link data may be stale — last updated ${Math.floor(age/60)}m ago.`;
banner.style.display = '';
} else if (banner) {
banner.style.display = 'none';
}
}
// ── Fetch + render ────────────────────────────────────────────────
async function loadLinks() {
try {
const data = await lt.api.get('/api/links');
if ((!data.hosts || !Object.keys(data.hosts).length) && (!data.unifi_switches || !Object.keys(data.unifi_switches).length)) {
document.getElementById('links-container').innerHTML =
'<div class="link-no-data">No link data yet — monitor has not completed a full cycle.</div>';
return;
}
const updEl = document.getElementById('links-updated');
if (updEl && data.updated) {
updEl.textContent = 'Updated: ' + new Date(_toIso(data.updated)).toLocaleTimeString();
}
renderLinks(data);
checkLinksStale(data.updated);
} catch (e) {
document.getElementById('links-container').innerHTML =
'<div class="error-state">Network error loading link statistics.</div>';
lt.toast.error('Failed to load link statistics');
}
}
loadLinks();
const _linksInterval = window.gandalfSettings?.refreshInterval ?? 60;
if (_linksInterval > 0) lt.autoRefresh.start(loadLinks, Math.max(_linksInterval, 15) * 1000);
window.onGandalfSettingsChanged = function(s) {
lt.autoRefresh.stop();
if (s.refreshInterval > 0) lt.autoRefresh.start(loadLinks, Math.max(s.refreshInterval, 15) * 1000);
};
document.addEventListener('click', e => {
const toggleTitle = e.target.closest('[data-action="toggle-panel"]');
if (toggleTitle) { togglePanel(toggleTitle.closest('.link-host-panel')); return; }
if (e.target.closest('[data-action="collapse-all"]')) { collapseAll(); return; }
if (e.target.closest('[data-action="expand-all"]')) { expandAll(); return; }
});
document.addEventListener('keydown', e => {
if (e.key !== 'Enter' && e.key !== ' ') return;
const toggleTitle = e.target.closest('[data-action="toggle-panel"]');
if (toggleTitle) { e.preventDefault(); togglePanel(toggleTitle.closest('.link-host-panel')); }
});
document.getElementById('links-search')?.addEventListener('input', applyLinksSearch);
</script>
{% endblock %}
+93
View File
@@ -0,0 +1,93 @@
"""Tests for app.py routes: database and LDAP are mocked."""
import json
import os
import sys
from unittest.mock import patch
import pytest
sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
import app as gandalf # noqa: E402
AUTH = {'Remote-User': 'jared', 'Remote-Name': 'Jared V', 'Remote-Groups': 'admin'}
STATE = {
'network_snapshot': json.dumps({
'hosts': {'large1': {'ip': '10.0.0.2', 'status': 'up', 'source': 'prometheus', 'interfaces': {'eth0': 'up'}}},
'unifi': [{'name': 'USW Flex', 'type': 'usw', 'model': 'USF5P', 'ip': '10.0.0.3', 'mac': '',
'connected': True}],
}),
'link_stats': json.dumps({'hosts': {}, 'unifi_switches': {}}),
}
@pytest.fixture
def client():
with patch.object(gandalf, 'db') as db:
db.get_state.side_effect = lambda key, default=None: STATE.get(key, default)
db.get_active_events.return_value = []
db.get_recent_resolved.return_value = []
db.get_active_suppressions.return_value = []
db.get_events_for_target.return_value = []
db.count_active_events.return_value = 0
db.get_status_summary.return_value = {'critical': 0, 'warning': 0, 'info': 0, 'resolved_24h': 0}
db.check_suppressed.return_value = False
with patch.object(gandalf, '_cfg', {'auth': {'allowed_groups': ['admin']}, 'ticket_api': {}}):
yield gandalf.app.test_client()
class TestPagesAndNav:
def test_home_has_search_and_no_link_debug(self, client):
html = client.get('/', headers=AUTH).get_data(as_text=True)
assert 'id="device-search-input"' in html
assert 'Link Debug' not in html
def test_home_section_order(self, client):
html = client.get('/', headers=AUTH).get_data(as_text=True)
order = [html.index(m) for m in (
'id="device-search"', 'Alert Queue', 'WAN / Fiber', 'g-section-title">Hosts<',
'Device Inventory', 'Network Topology',
)]
assert order == sorted(order)
def test_old_links_url_redirects_home(self, client):
resp = client.get('/links', headers=AUTH)
assert resp.status_code == 302 and resp.headers['Location'].endswith('/')
def test_device_page_escapes_name(self, client):
html = client.get('/device/%3Cscript%3Ex', headers=AUTH).get_data(as_text=True)
assert '<script>x' not in html
def test_link_data_api_remains_for_inspector(self, client):
assert client.get('/api/links', headers=AUTH).status_code == 200
class TestDeviceApi:
def test_requires_auth(self, client):
assert client.get('/api/devices').status_code == 401
assert client.get('/api/device/large1').status_code == 401
def test_index_lists_devices(self, client):
names = [d['name'] for d in client.get('/api/devices', headers=AUTH).get_json()['devices']]
assert names == ['large1', 'USW Flex']
def test_name_with_a_space(self, client):
d = client.get('/api/device/USW%20Flex', headers=AUTH).get_json()
assert d['kind'] == 'switch' and d['events'] == []
def test_unknown_device_is_404(self, client):
assert client.get('/api/device/nope', headers=AUTH).status_code == 404
class TestAvatarLogging:
def test_missing_ldap_config_says_why(self, client, caplog):
assert client.get('/api/avatar', headers=AUTH).status_code == 404
assert 'no ldap' in caplog.text
class TestDatabaseTimeZone:
def test_sessions_run_in_utc(self):
import db as real_db
with patch.object(real_db.pymysql, 'connect') as connect:
real_db._new_conn({'host': 'h', 'user': 'u', 'password': 'p', 'name': 'n'})
assert connect.call_args.kwargs['init_command'] == "SET time_zone='+00:00'"
+127
View File
@@ -0,0 +1,127 @@
"""Tests for devices.py: the inventory behind search and device pages."""
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
import devices # noqa: E402
SNAPSHOT = {
'hosts': {
'large1': {'ip': '10.0.0.2', 'status': 'degraded', 'source': 'prometheus',
'interfaces': {'eth0': 'up', 'eth1': 'down'}},
},
'unifi': [
{'name': 'Dream Machine Pro', 'type': 'udm', 'model': 'UDMPRO', 'ip': '203.0.113.5',
'mac': 'aa:bb:cc:dd:ee:ff', 'connected': True},
{'name': '24RJ45-POE', 'type': 'usw', 'model': 'US24PRO', 'ip': '10.0.0.40',
'mac': '11:22:33:44:55:66', 'connected': False},
{'name': 'RPDU', 'type': 'usw', 'model': 'USPPDUP', 'ip': '10.0.0.50', 'mac': '', 'connected': True},
{'name': 'AP', 'type': 'uap', 'model': 'UAL6', 'ip': '10.0.0.60', 'mac': '', 'connected': True},
],
}
LINKS = {
'hosts': {'large1': {'eth0': {'host_ip': '10.0.0.2', 'speed_mbps': 1000}}},
'unifi_switches': {
'24RJ45-POE': {'ip': '10.0.0.40', 'model': 'US24PRO', 'ports': {
'Port 3': {'port_idx': 3, 'up': True, 'speed_mbps': 1000,
'lldp': {'system_name': 'large1', 'port_id': 'eth0'}},
'Port 4': {'port_idx': 4, 'up': False, 'lldp': None},
}},
},
'unifi_devices': {'Dream Machine Pro': {'firmware': '6.0.17', 'wan': {'wan1': {'up': True}}}},
'updated': '2026-11-05 00:00:00 UTC',
}
MODEM = {'name': 'att-modem', 'reachable': True, 'lan_ip': '192.168.1.254', 'wan_ip': '203.0.113.5',
'model': 'BGW320-505', 'wan_problems': []}
def index():
return {d['name']: d for d in devices.build_index(SNAPSHOT, LINKS, MODEM)}
class TestKinds:
def test_kinds(self):
i = index()
assert i['large1']['kind'] == 'host'
assert i['Dream Machine Pro']['kind'] == 'router'
assert i['24RJ45-POE']['kind'] == 'switch'
assert i['AP']['kind'] == 'ap'
assert i['att-modem']['kind'] == 'modem'
def test_pdu_model_beats_reported_type(self):
assert index()['RPDU']['kind'] == 'pdu'
class TestSearchText:
def test_finds_host_by_ip_and_interface(self):
s = index()['large1']['search']
assert '10.0.0.2' in s and 'eth1' in s
def test_mac_with_and_without_separators(self):
s = index()['Dream Machine Pro']['search']
assert 'aa:bb:cc:dd:ee:ff' in s and 'aabbccddeeff' in s
def test_switch_found_by_port_neighbor(self):
assert 'large1' in index()['24RJ45-POE']['search']
def test_modem_found_by_either_address_or_word(self):
s = index()['att-modem']['search']
assert '192.168.1.254' in s and '203.0.113.5' in s and 'fiber' in s
class TestStatus:
def test_offline_unifi_device_is_down(self):
assert index()['24RJ45-POE']['status'] == 'down'
def test_modem_status(self):
def status(**kw):
return {d['name']: d for d in devices.build_index({}, {}, dict(MODEM, **kw))}['att-modem']['status']
assert status() == 'up'
assert status(wan_problems=['optical WAN is Down']) == 'down'
assert status(reachable=False) == 'unknown'
class TestPartialData:
def test_empty_state(self):
assert devices.build_index({}, {}, {}) == []
assert devices.build_index(None, None, None) == []
def test_switch_known_only_from_link_stats(self):
d = {x['name']: x for x in devices.build_index({}, LINKS, None)}['24RJ45-POE']
assert d['status'] == 'unknown' and d['kind'] == 'switch'
class TestUnconfiguredHosts:
LINKS = {'hosts': {'10.0.0.77': {'eth0': {'host_ip': '10.0.0.77', 'speed_mbps': 1000}}}}
def test_host_only_in_link_stats_is_kept(self):
d = {x['name']: x for x in devices.build_index({}, self.LINKS, None)}['10.0.0.77']
assert d['kind'] == 'host' and d['status'] == 'unknown' and d['ip'] == '10.0.0.77'
def test_its_interfaces_are_available(self):
d = devices.device_detail('10.0.0.77', {}, self.LINKS, None)
assert d['interfaces']['eth0']['speed_mbps'] == 1000
class TestDetail:
def test_unknown_device(self):
assert devices.device_detail('nope', SNAPSHOT, LINKS, MODEM) is None
def test_host_merges_state_and_stats(self):
d = devices.device_detail('large1', SNAPSHOT, LINKS, MODEM)
assert d['interfaces']['eth0'] == {'state': 'up', 'host_ip': '10.0.0.2', 'speed_mbps': 1000}
assert d['interfaces']['eth1'] == {'state': 'down'}
assert 'search' not in d
def test_host_shows_switch_port_from_lldp(self):
up = devices.device_detail('large1', SNAPSHOT, LINKS, MODEM)['uplinks']
assert up == [{'switch': '24RJ45-POE', 'port': 'Port 3', 'port_idx': 3,
'host_interface': 'eth0', 'up': True, 'speed_mbps': 1000}]
def test_switch_has_ports(self):
assert set(devices.device_detail('24RJ45-POE', SNAPSHOT, LINKS, MODEM)['ports']) == {'Port 3', 'Port 4'}
def test_router_has_info_even_without_port_data(self):
d = devices.device_detail('Dream Machine Pro', SNAPSHOT, LINKS, MODEM)
assert d['info']['firmware'] == '6.0.17' and d['ports'] == {}
+257
View File
@@ -0,0 +1,257 @@
"""Tests for modem.py parsing. Fixtures copy the BGW320-505 (fw 6.35.8) markup with sanitized values."""
import os
import sys
from unittest.mock import MagicMock
import pytest
import requests
sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
import modem # noqa: E402
def dom_table(heading, current, flags=(0, 0, 0, 0), limits=(-292, -90, -279, -100)):
return f'''
<h1>{heading}&nbsp;&nbsp;Currently {current}</h1>
<table class="grid table100" summary="{heading} table">
<thead><tr><th scope="col" width="30%">&nbsp;</th><th scope="col">Low</th><th scope="col">High</th></tr></thead>
<tbody>
<tr class="a"><td class="heading" scope="row">Alarm</td>
<td>{flags[0]} (Threshold {limits[0]})</td>
<td>{flags[1]} (Threshold {limits[1]})</td>
</tr>
<tr class="b"><td class="heading" scope="row">Warning</td>
<td>{flags[2]} (Threshold {limits[2]})</td>
<td>{flags[3]} (Threshold {limits[3]})</td>
</tr>
</tbody></table>'''
def fiber_page(optical='Up', rx=-169, rx_flags=(0, 0, 0, 0), include_rx=True):
rx_table = dom_table('Rx Power', rx, rx_flags) if include_rx else ''
return f'''<html><title>Fiber Status</title>
<tr><th scope="row" width="47%">Optical WAN Operational Status</th>
<td class="col2">{optical} </td></tr>
<tr><th scope="row">Wave Length</th><td class="col2">1270 nm </td></tr>
<tr><th scope="row">Rx LOS State</th><td class="col2">1</td></tr>
<tr><th scope="row">Vendor PN</th><td class="col2">TESTPN</td></tr>
{dom_table('Temperature', 34, limits=(-50, 95, -45, 90))}
{dom_table('Vcc', 3, limits=(3, 3, 3, 3))}
{dom_table('Tx Bias', 6, limits=(0, 500, 0, 400))}
{dom_table('Tx Power', 33, limits=(-10, 59, 0, 49))}
{rx_table}</html>'''
def broadband_page(connection='Up', pon='OPERATION (O5)'):
return f'''<html><title>Broadband Status</title>
<tr><th scope="row" width="47%">Broadband Connection Source</th><td class="col2">FIBER</td></tr>
<tr><th scope="row" width="47%">Broadband Connection</th>
<td class="col2">
{connection}
</td></tr>
<tr><th scope="row">Broadband IPv4 Address</th><td class="col2">203.0.113.10</td></tr>
<tr class="a"><th scope="row" class="heading a">Current Speed (Mbps)</th><td>10000</td></tr>
<tr><th scope="row">Transmit Bytes</th><td>200</td></tr>
<tr><th scope="row">Receive Bytes</th><td>100</td></tr>
<tr><th scope="row">Receive Errors</th><td>1</td></tr>
<tr><th scope="row">Transmit Errors</th><td>2</td></tr>
<tr><th scope="row">Receive Drops</th><td>3</td></tr>
<tr><th scope="row">Transmit Drops</th><td>4</td></tr>
<tr><th scope="row">Transmit Packets</th><td>7</td></tr>
<tr><th scope="row">Transmit Packets</th><td>999</td></tr>
<tr><th scope="row">PON Link Status</th><td>{pon}</td></tr>
<tr><th scope="row">UNI Status</th><td>up</td></tr></html>'''
SYSINFO = '''<tr><th scope="row">Model Number</th><td class="col2">BGW320-505</td></tr>
<tr><th scope="row">Software Version</th><td class="col2">6.35.8</td></tr>
<tr><th scope="row">Time Since Last Reboot</th><td class="col2">28951</td></tr>'''
class TestBuildStatus:
def test_scales_optical_power_from_tenths_of_dbm(self):
d = modem.build_status(fiber_page(), broadband_page(), SYSINFO)
assert d['optics']['rx_power']['value'] == -16.9
assert d['optics']['tx_power']['value'] == 3.3
assert d['optics']['temperature']['value'] == 34.0
def test_scales_thresholds(self):
rx = modem.build_status(fiber_page(), broadband_page())['optics']['rx_power']
assert (rx['alarm_low'], rx['alarm_high'], rx['warn_low'], rx['warn_high']) == (-29.2, -9.0, -27.9, -10.0)
def test_unit_ambiguous_readings_are_dropped_but_flags_kept(self):
d = modem.build_status(fiber_page(), broadband_page())
assert d['optics']['tx_bias']['value'] is None
assert d['optics']['vcc']['alarm_low'] is None
assert d['optics']['vcc']['alarm_flag'] is False
def test_system_and_wan_fields(self):
d = modem.build_status(fiber_page(), broadband_page(), SYSINFO)
assert d['model'] == 'BGW320-505'
assert d['firmware'] == '6.35.8'
assert d['uptime_s'] == 28951
assert d['broadband_state'] == 'Up'
assert d['line_speed_mbps'] == 10000
assert d['counters'] == {
'rx_bytes': 100, 'tx_bytes': 200, 'rx_errors': 1,
'tx_errors': 2, 'rx_drops': 3, 'tx_drops': 4,
}
def test_repeated_label_keeps_first_value(self):
assert modem.parse_rows(broadband_page())['Transmit Packets'] == '7'
def test_sysinfo_is_optional(self):
d = modem.build_status(fiber_page(), broadband_page(), '')
assert d['model'] == '' and d['uptime_s'] is None
def test_unknown_page_is_invalid(self):
with pytest.raises(modem.ModemDataInvalid):
modem.build_status('<html>nope</html>', broadband_page())
with pytest.raises(modem.ModemDataInvalid):
modem.build_status(fiber_page(), '<html>nope</html>')
def test_up_link_without_rx_power_is_invalid(self):
with pytest.raises(modem.ModemDataInvalid):
modem.build_status(fiber_page(include_rx=False), broadband_page())
def test_cut_fiber_without_rx_power_is_not_invalid(self):
d = modem.build_status(fiber_page(optical='Down', include_rx=False), broadband_page(connection='Down'))
assert 'rx_power' not in d['optics']
assert modem.wan_problems(d)
class TestWanProblems:
def test_healthy_link_has_none(self):
assert modem.wan_problems(modem.build_status(fiber_page(), broadband_page())) == []
@pytest.mark.parametrize('field,value,expected', [
('broadband_state', 'Down', 'broadband connection is Down'),
('optical_state', 'Down', 'optical WAN is Down'),
('pon_state', 'INITIAL (O1)', 'PON link state is INITIAL (O1)'),
])
def test_each_failure_is_reported(self, field, value, expected):
d = {'broadband_state': 'Up', 'optical_state': 'Up', 'pon_state': 'OPERATION (O5)', field: value}
assert modem.wan_problems(d) == [expected]
def test_missing_pon_field_is_not_a_problem(self):
assert modem.wan_problems({'broadband_state': 'Up', 'optical_state': 'Up', 'pon_state': ''}) == []
def test_rx_los_pin_is_ignored(self):
# Real gateways report Rx LOS State=1 on a healthy link (see fixture).
assert modem.wan_problems(modem.build_status(fiber_page(), broadband_page())) == []
class TestMetricStatus:
@pytest.mark.parametrize('rx,expected', [
(-169, 'ok'),
(-279, 'warning'),
(-285, 'warning'),
(-292, 'alarm'),
(-400, 'alarm'),
(-100, 'warning'),
(-90, 'alarm'),
])
def test_rx_thresholds(self, rx, expected):
rx_metric = modem.parse_dom_metrics(fiber_page(rx=rx))['rx_power']
assert modem.metric_status(rx_metric) == expected
def test_module_flags_win_over_reading(self):
alarm = modem.parse_dom_metrics(fiber_page(rx_flags=(1, 0, 0, 0)))['rx_power']
warn = modem.parse_dom_metrics(fiber_page(rx_flags=(0, 0, 1, 0)))['rx_power']
assert modem.metric_status(alarm) == 'alarm'
assert modem.metric_status(warn) == 'warning'
def test_flag_only_metric(self):
assert modem.metric_status(modem.parse_dom_metrics(fiber_page())['vcc']) == 'ok'
class TestCounterRates:
def test_rate_from_delta(self):
assert modem.counter_rates({'rx_bytes': 300}, {'rx_bytes': 100}, 100)['rx_bytes'] == 2.0
def test_first_poll_is_none(self):
assert modem.counter_rates({'rx_bytes': 300}, None, 100)['rx_bytes'] is None
def test_reboot_counter_reset_is_none(self):
assert modem.counter_rates({'rx_bytes': 5}, {'rx_bytes': 100}, 100)['rx_bytes'] is None
class TestClient:
def _client(self, **response):
c = modem.ModemClient({'url': 'http://modem.test/'})
resp = MagicMock(status_code=response.get('status', 200), text=response.get('text', ''))
c.session.get = MagicMock(return_value=resp)
return c
def test_network_failure_is_unreachable(self):
c = self._client()
c.session.get = MagicMock(side_effect=requests.ConnectionError('boom'))
with pytest.raises(modem.ModemUnreachable):
c.poll()
def test_redirect_is_invalid_and_not_followed(self):
c = self._client(status=302)
with pytest.raises(modem.ModemDataInvalid):
c.poll()
assert c.session.get.call_args.kwargs['allow_redirects'] is False
def test_login_page_is_invalid(self):
with pytest.raises(modem.ModemDataInvalid, match='login'):
self._client(text='<html><title>Login</title></html>').poll()
class TestWanHistory:
def ev(self, key, ts, device='RPDU'):
return {'key': key, 'ts': ts, 'device': device, 'wan': 'WAN1'}
def test_outage_opens_and_closes(self):
h = {}
modem.apply_wan_events(h, [self.ev(modem.LOG_WAN_FAILED, 100), self.ev(modem.LOG_WAN_RESTORED, 200)])
assert h['outages'] == [{'failed_at': 100, 'restored_at': 200, 'wan': 'WAN1'}]
def test_events_are_handled_in_time_order(self):
h = {}
modem.apply_wan_events(h, [self.ev(modem.LOG_WAN_RESTORED, 200), self.ev(modem.LOG_WAN_FAILED, 100)])
assert h['outages'][0]['restored_at'] == 200
def test_outage_spanning_two_batches(self):
h = {}
modem.apply_wan_events(h, [self.ev(modem.LOG_WAN_FAILED, 100)])
assert h['outages'][0]['restored_at'] is None
modem.apply_wan_events(h, [self.ev(modem.LOG_WAN_RESTORED, 200)])
assert h['outages'][0]['restored_at'] == 200
def test_repeat_failure_does_not_open_a_second_outage(self):
h = {}
modem.apply_wan_events(h, [self.ev(modem.LOG_WAN_FAILED, 100), self.ev(modem.LOG_WAN_FAILED, 150)])
assert len(h['outages']) == 1
def test_restored_without_failure_is_ignored(self):
h = {}
modem.apply_wan_events(h, [self.ev(modem.LOG_WAN_RESTORED, 200)])
assert h['outages'] == []
def test_blip_is_not_actionable_but_restart_and_instability_are(self):
keys = [modem.LOG_WAN_FAILED, modem.LOG_WAN_RESTORED, modem.LOG_MODEM_RESTARTED, modem.LOG_WAN_UNSTABLE]
got = modem.apply_wan_events({}, [self.ev(k, i) for i, k in enumerate(keys)])
assert [e['key'] for e in got] == [modem.LOG_MODEM_RESTARTED, modem.LOG_WAN_UNSTABLE]
def test_history_is_bounded(self):
h = {}
for i in range(60):
modem.apply_wan_events(h, [
self.ev(modem.LOG_WAN_FAILED, i * 2), self.ev(modem.LOG_WAN_RESTORED, i * 2 + 1),
])
assert len(h['outages']) == 50
def test_uptime_reset_confirms_logged_restart(self):
h = {}
modem.apply_wan_events(h, [self.ev(modem.LOG_MODEM_RESTARTED, 1000)])
modem.record_uptime_reset(h, 1023)
assert h['restarts'] == [{'at': 1000, 'by': 'RPDU', 'confirmed': True}]
def test_unexplained_uptime_reset_is_recorded(self):
h = {}
modem.record_uptime_reset(h, 5000)
assert h['restarts'] == [{'at': 5000, 'by': None, 'confirmed': True}]
+375
View File
@@ -0,0 +1,375 @@
"""Tests for monitor.py alert logic: database and external services are mocked."""
import json
import os
import sys
import time
from unittest.mock import MagicMock, patch
import pytest
sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
import modem as modem_api # noqa: E402
import monitor # noqa: E402
HOST_A, HOST_B = '10.0.0.1:9100', '10.0.0.2:9100'
@pytest.fixture
def db():
with patch.object(monitor, 'db') as mock_db:
mock_db.check_suppressed.return_value = False
mock_db.get_baseline.return_value = {}
# Second consecutive failure and no ticket yet, unless a test overrides it
mock_db.upsert_event.return_value = (1, False, 2, None)
mock_db.get_modem_rx_values.return_value = []
mock_db.get_state.return_value = None
yield mock_db
@pytest.fixture
def mon(db):
m = monitor.NetworkMonitor.__new__(monitor.NetworkMonitor)
m.cfg = {}
m.fail_thresh = 2
m.cluster_thresh = 2
m.cluster_name = 'test-cluster'
m.tickets = MagicMock()
m.tickets.create.return_value = '123'
m.prom = MagicMock(url='http://prom.test')
m._instance_map = {HOST_A: 'alpha', HOST_B: 'beta'}
m._last_iface_states = {}
m.modem = MagicMock(base_url='http://modem.test')
m.modem_name = 'att-modem'
m.modem_lan_ip = '192.168.1.254'
m.modem_fail_thresh = 2
m.rx_drift_db = 3.0
m.rx_drift_min_samples = 288
m._prev_modem_counters = None
m._prev_modem_time = 0.0
return m
def upserted(db):
return [(c.args[0], c.args[3], c.args[4]) for c in db.upsert_event.call_args_list]
def healthy_modem(rx=-16.9, rx_status=None):
def metric(value, alarm_low, alarm_high, warn_low, warn_high):
return {'value': value, 'alarm_low': alarm_low, 'alarm_high': alarm_high,
'warn_low': warn_low, 'warn_high': warn_high,
'alarm_flag': False, 'warn_flag': False}
return {
'broadband_state': 'Up', 'optical_state': 'Up', 'pon_state': 'OPERATION (O5)',
'counters': {'rx_bytes': 1},
'optics': {
'rx_power': metric(rx, -29.2, -9.0, -27.9, -10.0),
'tx_power': metric(3.3, -1.0, 5.9, 0.0, 4.9),
'temperature': metric(34.0, -50.0, 95.0, -45.0, 90.0),
},
}
class TestDeadHostDetection:
def test_host_missing_from_prometheus_alerts_and_tickets(self, mon, db):
mon._process_interfaces({HOST_A: {'eth0': True}}, [])
assert ('host_metrics_missing', 'beta', HOST_B) in upserted(db)
assert 'No metrics' in mon.tickets.create.call_args.args[0]
def test_host_reporting_resolves_the_event(self, mon, db):
mon._process_interfaces({HOST_A: {'eth0': True}, HOST_B: {'eth0': True}}, [])
db.resolve_event.assert_any_call('host_metrics_missing', 'beta', HOST_B)
assert not db.upsert_event.called
def test_no_ticket_before_threshold(self, mon, db):
db.upsert_event.return_value = (1, True, 1, None)
mon._process_interfaces({HOST_A: {'eth0': True}}, [])
assert not mon.tickets.create.called
def test_suppressed_host_gets_no_ticket(self, mon, db):
db.check_suppressed.return_value = True
mon._process_interfaces({HOST_A: {'eth0': True}}, [])
assert not mon.tickets.create.called
def test_missing_hosts_count_toward_cluster_alert(self, mon, db):
mon._instance_map['10.0.0.3:9100'] = 'gamma'
mon._process_interfaces({HOST_A: {'eth0': True}}, [])
assert ('cluster_network_issue', 'test-cluster', '') in upserted(db)
def test_dead_host_shows_as_down_on_dashboard(self, mon):
snap = mon._collect_snapshot({HOST_A: {'eth0': True}})
assert snap['hosts']['alpha']['status'] == 'up'
assert snap['hosts']['beta']['status'] == 'down'
class TestPrometheusOutage:
def test_empty_result_means_unavailable(self):
client = monitor.PrometheusClient('http://prom.test')
with patch.object(client, '_fetch', return_value=[]):
assert client.get_interface_states() is None
def test_failed_query_is_unavailable_not_empty(self):
client = monitor.PrometheusClient('http://prom.test')
with patch.object(monitor.requests, 'get', side_effect=OSError('boom')):
assert client.get_interface_states() is None
assert client.query('up') == []
def test_outage_alerts_about_prometheus_not_each_host(self, mon, db):
mon._process_prometheus_outage([])
assert upserted(db) == [('monitor_source_unavailable', 'prometheus', '')]
def test_dashboard_keeps_hosts_as_unknown(self, mon):
mon._last_iface_states = {HOST_A: {'eth0': True}}
snap = mon._collect_snapshot(None)
assert snap['hosts']['alpha']['status'] == 'unknown'
assert snap['hosts']['alpha']['interfaces'] == {'eth0': 'up'}
assert snap['hosts']['beta']['status'] == 'unknown'
class TestPingUnknown:
def _pulse(self, output):
pulse = monitor.PulseClient({})
pulse.run_command = MagicMock(return_value=output)
return pulse
def test_pulse_failure_is_unknown(self):
assert self._pulse(None).ping('10.0.0.1') is None
def test_reachability(self):
assert self._pulse('REACHABLE\n').ping('10.0.0.1') is True
assert self._pulse('UNREACHABLE\n').ping('10.0.0.1') is False
def test_unknown_neither_alerts_nor_resolves(self, mon, db):
mon.cfg = {'monitor': {'ping_hosts': [{'name': 'pbs', 'ip': '10.0.0.3'}]}}
mon._process_ping_hosts([], {'pbs': None})
assert not db.upsert_event.called
assert not db.resolve_event.called
def test_unreachable_still_alerts(self, mon, db):
mon.cfg = {'monitor': {'ping_hosts': [{'name': 'pbs', 'ip': '10.0.0.3'}]}}
mon._process_ping_hosts([], {'pbs': False})
assert ('host_unreachable', 'pbs', '10.0.0.3') in upserted(db)
class TestUnifiRates:
def _collector(self):
c = monitor.LinkStatsCollector.__new__(monitor.LinkStatsCollector)
c._prev_unifi_time = 100.0
c._prev_unifi = {'sw': {'ports': {'Port 1': {
'tx_bytes': 100, 'rx_bytes': 100, 'tx_errors': 0, 'rx_errors': 0, 'tx_dropped': 0, 'rx_dropped': 0,
}}}}
return c
def test_new_port_has_no_rate_spike(self):
port = {'tx_bytes': 10**12, 'rx_bytes': 10**12, 'tx_errors': 5, 'rx_errors': 5,
'tx_dropped': 5, 'rx_dropped': 5}
raw = {'sw': {'ip': '', 'model': '', 'ports': {'Port 1': dict(port, tx_bytes=300), 'Port 2': port}}}
result = self._collector()._compute_unifi_rates(raw, 200.0)['sw']['ports']
assert result['Port 2']['tx_bytes_rate'] is None
assert result['Port 1']['tx_bytes_rate'] == 2.0
class TestModemAlerts:
def test_healthy_link_raises_nothing(self, mon, db):
mon._process_modem(healthy_modem(), [])
assert not db.upsert_event.called
assert not mon.tickets.create.called
def test_wan_down_is_p1(self, mon, db):
data = healthy_modem()
data['broadband_state'] = 'Down'
mon._process_modem(data, [])
assert ('wan_down', 'att-modem', '') in upserted(db)
assert mon.tickets.create.call_args.kwargs['priority'] == '1'
def test_cut_fiber_opens_only_one_ticket(self, mon, db):
data = healthy_modem(rx=-40.0)
data['optical_state'] = 'Down'
mon._process_modem(data, [])
assert [e[0] for e in upserted(db)] == ['wan_down']
def test_low_rx_warning(self, mon, db):
mon._process_modem(healthy_modem(rx=-28.0), [])
assert ('fiber_signal_warning', 'att-modem', 'rx_power') in upserted(db)
assert mon.tickets.create.call_args.kwargs['priority'] == '3'
db.resolve_event.assert_any_call('fiber_signal_alarm', 'att-modem', 'rx_power')
def test_low_rx_alarm_with_link_up(self, mon, db):
mon._process_modem(healthy_modem(rx=-30.0), [])
assert ('fiber_signal_alarm', 'att-modem', 'rx_power') in upserted(db)
db.resolve_event.assert_any_call('fiber_signal_warning', 'att-modem', 'rx_power')
def test_recovery_resolves_signal_events(self, mon, db):
mon._process_modem(healthy_modem(), [])
db.resolve_event.assert_any_call('fiber_signal_alarm', 'att-modem', 'rx_power')
db.resolve_event.assert_any_call('fiber_signal_warning', 'att-modem', 'rx_power')
db.resolve_event.assert_any_call('wan_down', 'att-modem', '')
def test_suppression_blocks_ticket(self, mon, db):
db.check_suppressed.return_value = True
mon._process_modem(healthy_modem(rx=-30.0), [])
assert upserted(db)
assert not mon.tickets.create.called
def test_drift_needs_enough_history(self, mon, db):
db.get_modem_rx_values.return_value = [-17.0] * 10
mon._process_modem(healthy_modem(rx=-25.0), [])
assert not any(e[0] == 'fiber_rx_drift' for e in upserted(db))
def test_drift_alerts_inside_the_modem_limits(self, mon, db):
db.get_modem_rx_values.return_value = [-17.0] * 300
mon._process_modem(healthy_modem(rx=-21.0), [])
assert ('fiber_rx_drift', 'att-modem', 'rx_power') in upserted(db)
assert 'fiber_signal_warning' not in [e[0] for e in upserted(db)]
def test_small_change_is_not_drift(self, mon, db):
db.get_modem_rx_values.return_value = [-17.0] * 300
mon._process_modem(healthy_modem(rx=-18.0), [])
assert not db.upsert_event.called
db.resolve_event.assert_any_call('fiber_rx_drift', 'att-modem', 'rx_power')
def test_stronger_signal_is_not_drift(self, mon, db):
db.get_modem_rx_values.return_value = [-17.0] * 300
mon._process_modem(healthy_modem(rx=-12.0), [])
assert not db.upsert_event.called
class TestModemCollection:
def test_unreachable_keeps_link_events_open(self, mon, db):
mon.modem.poll.side_effect = modem_api.ModemUnreachable('timeout')
mon._collect_modem([])
assert upserted(db) == [('modem_unreachable', 'att-modem', '')]
assert db.set_state.call_args.args[1]['reachable'] is False
assert not db.record_modem_sample.called
resolved = [c.args[0] for c in db.resolve_event.call_args_list]
assert 'wan_down' not in resolved and 'fiber_signal_alarm' not in resolved
def test_unreadable_page_is_a_warning_not_an_outage(self, mon, db):
mon.modem.poll.side_effect = modem_api.ModemDataInvalid('login required')
mon._collect_modem([])
assert upserted(db) == [('modem_data_invalid', 'att-modem', '')]
assert mon.tickets.create.call_args.kwargs['priority'] == '3'
def test_success_stores_status_and_sample(self, mon, db):
mon.modem.poll.return_value = healthy_modem()
mon._collect_modem([])
assert db.set_state.call_args.args[0] == 'modem_stats'
assert db.set_state.call_args.args[1]['reachable'] is True
db.record_modem_sample.assert_called_once_with(-16.9, 3.3, 34.0)
def test_drift_baseline_excludes_the_current_reading(self, mon, db):
order = []
db.get_modem_rx_values.side_effect = lambda days: order.append('read') or []
db.record_modem_sample.side_effect = lambda *a: order.append('write')
mon.modem.poll.return_value = healthy_modem()
mon._collect_modem([])
assert order == ['read', 'write']
class TestWanLog:
def _mon(self, mon, events):
mon.unifi = MagicMock()
mon.unifi.get_system_events.return_value = events
return mon
def _state(self, db, **values):
db.get_state.side_effect = lambda key, default=None: values.get(key, default)
def test_first_run_starts_from_now_without_tickets(self, mon, db):
self._state(db)
self._mon(mon, [])._process_wan_log([])
assert db.set_state.call_args.args[0] == 'unifi_log_cursor'
assert not mon.unifi.get_system_events.called
def test_log_failure_keeps_the_cursor(self, mon, db):
self._state(db, unifi_log_cursor='1000')
self._mon(mon, None)._process_wan_log([])
assert not db.set_state.called
def test_rpdu_restart_opens_a_ticket_and_resolves(self, mon, db):
self._state(db, unifi_log_cursor=str(int(time.time() * 1000)))
db.upsert_event.return_value = (9, True, 1, None)
self._mon(mon, [{'key': modem_api.LOG_MODEM_RESTARTED, 'ts': time.time(), 'device': 'RPDU', 'wan': ''}])
mon._process_wan_log([])
assert 'power-cycled' in mon.tickets.create.call_args.args[0]
db.resolve_event.assert_called_with('modem_power_cycled', 'att-modem', '')
def test_short_outage_only_records_history(self, mon, db):
now = time.time()
self._state(db, unifi_log_cursor=str(int(now * 1000)))
self._mon(mon, [
{'key': modem_api.LOG_WAN_FAILED, 'ts': now, 'device': 'UDM', 'wan': 'WAN1'},
{'key': modem_api.LOG_WAN_RESTORED, 'ts': now + 90, 'device': 'UDM', 'wan': 'WAN1'},
])
mon._process_wan_log([])
assert not mon.tickets.create.called
saved = dict((c.args[0], c.args[1]) for c in db.set_state.call_args_list)
assert saved['wan_history']['outages'][0]['restored_at'] == now + 90
def test_unstable_wan_tickets_once_a_day(self, mon, db):
now = time.time()
history = json.dumps({'outages': [], 'restarts': [], 'unstable_ticketed_at': now - 3600})
self._state(db, unifi_log_cursor=str(int(now * 1000)), wan_history=history)
self._mon(mon, [{'key': modem_api.LOG_WAN_UNSTABLE, 'ts': now, 'device': 'UDM', 'wan': 'WAN1'}])
mon._process_wan_log([])
assert not mon.tickets.create.called
def test_unstable_wan_tickets_after_a_day(self, mon, db):
now = time.time()
self._state(db, unifi_log_cursor=str(int(now * 1000)))
self._mon(mon, [{'key': modem_api.LOG_WAN_UNSTABLE, 'ts': now, 'device': 'UDM', 'wan': 'WAN1'}])
mon._process_wan_log([])
assert 'unstable' in mon.tickets.create.call_args.args[0]
def test_suppression_blocks_one_shot_ticket(self, mon, db):
now = time.time()
db.check_suppressed.return_value = True
self._state(db, unifi_log_cursor=str(int(now * 1000)))
self._mon(mon, [{'key': modem_api.LOG_MODEM_RESTARTED, 'ts': now, 'device': 'RPDU', 'wan': ''}])
mon._process_wan_log([])
assert not mon.tickets.create.called
def test_uptime_going_backwards_records_a_restart(self, mon, db):
self._state(db, modem_stats=json.dumps({'uptime_s': 5000}))
mon._note_modem_restart({'uptime_s': 60})
saved = dict((c.args[0], c.args[1]) for c in db.set_state.call_args_list)
assert len(saved['wan_history']['restarts']) == 1
def test_uptime_going_forward_records_nothing(self, mon, db):
self._state(db, modem_stats=json.dumps({'uptime_s': 5000}))
mon._note_modem_restart({'uptime_s': 5300})
assert not db.set_state.called
class TestUnifiDeviceDetail:
def test_parses_string_stats_and_outlets(self):
d = monitor.UnifiClient._device_detail({
'version': '7.0', 'uptime': 10, 'system-stats': {'cpu': ' 2.4 ', 'mem': 'bad'},
'outlet_table': [{'index': 13, 'name': 'AT&T Modem', 'relay_state': True,
'cycle_enabled': True, 'outlet_power': '0.000'}],
'wan1': {'up': True, 'speed': 1000, 'latency': 15, 'availability': 100},
})
assert d['cpu_pct'] == 2.4 and d['mem_pct'] is None
assert d['outlets'][0]['auto_power_cycle'] is True and d['outlets'][0]['power_w'] == 0.0
assert d['wan']['wan1']['speed_mbps'] == 1000
assert 'wan2' not in d['wan']
def test_system_log_filters_and_paginates(self):
c = monitor.UnifiClient({'controller': 'https://u.test', 'api_key': 'k'})
page = MagicMock()
page.json.side_effect = [
{'data': [
{'key': modem_api.LOG_MODEM_RESTARTED, 'timestamp': 2000, 'parameters': {'DEVICE': {'name': 'RPDU'}}},
{'key': 'CLIENT_CONNECTED_WIRED_2', 'timestamp': 2000},
{'key': modem_api.LOG_WAN_FAILED, 'timestamp': 500},
]},
{'data': []},
]
c.session.post = MagicMock(return_value=page)
got = c.get_system_events(1000, modem_api.WAN_LOG_KEYS)
assert got == [{'key': modem_api.LOG_MODEM_RESTARTED, 'ts': 2.0, 'device': 'RPDU', 'wan': ''}]
def test_system_log_failure_is_none_not_empty(self):
c = monitor.UnifiClient({'controller': 'https://u.test', 'api_key': 'k'})
c.session.post = MagicMock(side_effect=OSError('boom'))
assert c.get_system_events(0, modem_api.WAN_LOG_KEYS) is None