Compare commits
20
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
1444562a5d | ||
|
|
dd9ba41479 | ||
|
|
3afd85626b | ||
|
|
c2ff072eb2 | ||
|
|
f097e4882c | ||
|
|
0370358363 | ||
|
|
2a9111d8e8 | ||
|
|
de75f71bf8 | ||
|
|
2f4cb05f67 | ||
|
|
fa108b2a1e | ||
|
|
ff425dbaee | ||
|
|
99dee62098 | ||
|
|
4ad331ddfe | ||
|
|
7da4cc26e2 | ||
|
|
73787daaec | ||
|
|
6a717af3f2 | ||
|
|
f1355c4bd8 | ||
|
|
abac6983b2 | ||
|
|
4646d9be90 | ||
|
|
c535bfd9fb |
No files matched your search
@@ -83,7 +83,7 @@ Two processes share a MariaDB database:
|
||||
| Source | What it provides |
|
||||
|---|---|
|
||||
| **Prometheus** (`10.10.10.48:9090`) | Physical NIC link state + traffic/error rates via `node_exporter` |
|
||||
| **UniFi API** (`https://10.10.10.1`) | Switch port stats, device status, LLDP neighbor table, PoE data |
|
||||
| **UniFi API** (`https://10.10.10.1`) | Switch port stats, device status, LLDP neighbor table, PoE data, the WAN health timeline (latency, loss, drops) |
|
||||
| **Pulse Worker** | SSH relay: runs `ethtool` + SFP DOM queries on each Proxmox host |
|
||||
| **Ping** | Reachability for hosts without `node_exporter` (e.g. PBS) |
|
||||
| **AT&T BGW320** (`http://192.168.1.254`) | Fiber light levels, PON and WAN state, WAN counters, scraped read-only from the gateway's status pages (no login) |
|
||||
@@ -92,7 +92,7 @@ Two processes share a MariaDB database:
|
||||
|
||||
| Host | Prometheus Instance |
|
||||
|---|---|
|
||||
| large1 | 10.10.10.2:9100 |
|
||||
| compute-storage-02 | 10.10.10.2:9100 |
|
||||
| compute-storage-01 | 10.10.10.4:9100 |
|
||||
| monitor-01 | 10.10.10.8:9100 |
|
||||
| monitor-02 | 10.10.10.9:9100 |
|
||||
@@ -106,23 +106,37 @@ Ping-only (no node_exporter): **pbs** (10.10.10.3)
|
||||
## Pages
|
||||
|
||||
### Dashboard (`/`)
|
||||
- **Needs attention** at the top gathers what every page would flag into two lists, each line linking to
|
||||
where it is fixed:
|
||||
- *Broken now*: internet trouble in the last hour (see below), active alerts (one line, with the
|
||||
devices), an outlet that is off but feeds a device, frozen outlet readings, and cables not where the
|
||||
cutsheet says.
|
||||
- *Tidy up*: UniFi port names pointing at the wrong device or at an empty port, live links not on the
|
||||
cutsheet, labels to print, powered outlets with no device assigned, and rack items that overlap or sit
|
||||
outside the rack.
|
||||
|
||||
Served by `GET /api/attention` (`attention.py`) and refreshed with the rest of the page.
|
||||
- Device search (see Device pages below)
|
||||
- Active alerts first, then the WAN / Fiber panel, hosts, UniFi devices, and recently resolved alerts
|
||||
- Real-time host status grid with per-NIC link state (UP / DOWN / degraded)
|
||||
- WAN / Fiber panel: WAN and PON state, Rx/Tx power, optics temperature, throughput, errors, gateway uptime and last restart, 7-day Rx trend, recent WAN outages
|
||||
- UniFi device table (switches, APs, gateway)
|
||||
- Active alerts table with severity, target, consecutive failures, ticket link
|
||||
- WAN / Fiber panel: WAN and PON state, Rx/Tx power, optics temperature, throughput, errors, gateway uptime and last restart (just the last restart when the gateway doesn't report uptime), a 24 hour internet health strip from UniFi, 7-day Rx trend, recent WAN outages
|
||||
- Hosts with per-NIC link state; NICs marked unused fold into a "+N unused" line
|
||||
- UniFi devices as a compact list (name and model; offline ones turn red with a suppress button)
|
||||
- Recently resolved alerts, folded until opened
|
||||
- Follow-up flags (see Follow-up below): a ⏳ badge on alerts active for days, a "ticket closed" badge when the
|
||||
ticket was closed but the alert is still active, and a Needs Follow-up panel of tickets left open after their
|
||||
alert cleared. A "TO FOLLOW UP" chip in the status bar counts them all.
|
||||
- Quick-suppress modal: apply timed or manual suppression from any alert row
|
||||
- Auto-refreshes every 30 seconds via `/api/status` + `/api/network`
|
||||
- Quick-suppress modal: apply timed or manual suppression from any alert row or host card
|
||||
- Auto-refreshes every 30 seconds via `/api/status` + `/api/network` + `/api/attention`
|
||||
|
||||
The header spells out the name; the navigation keeps Dashboard, Topology, Rack and Cutsheet in view, with
|
||||
Alert history and Suppressions under **More** (and in the command palette, Ctrl+K). On a phone the drawer
|
||||
holds every page and Start maintenance.
|
||||
|
||||
### Alert history (`/events`)
|
||||
Every alert Gandalf raised in the last 24 hours to 90 days, including ones that cleared before anyone
|
||||
saw them: when each started, how long it lasted, failed checks, and its ticket.
|
||||
- Filter by device (name, port or interface), alert type, severity, and open or cleared. Filters are
|
||||
kept in the URL, so `/events?target=large1&range=90d` is a shareable link; device pages and tickets
|
||||
kept in the URL, so `/events?target=compute-storage-02&range=90d` is a shareable link; device pages and tickets
|
||||
link to their own history this way.
|
||||
- **Recurring** lists the same alert on the same target more than once in the range, worst first, since
|
||||
a flapping cable or optic shows up here before anyone notices the tickets.
|
||||
@@ -185,9 +199,11 @@ The rack front, drawn from the rack units set on cutsheet items, top first.
|
||||
The whole network as the switches see it now, top down: the internet, the modem, the gateway, then
|
||||
each switch by how many hops it is from the gateway, the gear and unknown devices below them, and the
|
||||
servers along the bottom. Power gear (PDU, UPS, power strip) sits in a rail on the right.
|
||||
- **What needs a look** is listed above the diagram: devices down, degraded or with active alerts,
|
||||
cables down or last seen elsewhere, ports named after the wrong device, and cables with errors.
|
||||
Click one to jump to it. With nothing to report it says all clear. Problem cables are drawn in full
|
||||
- **What needs a look** sits above the diagram as one line ("7 to look at" and the first two):
|
||||
devices down, degraded or with active alerts, cables down or last seen elsewhere, ports named after
|
||||
the wrong device, and cables with errors. Open it to list them all and click one to jump to it; it
|
||||
remembers whether it was open. With nothing to report it says all clear. The how-to hides behind the
|
||||
**?** next to the page intro. Problem cables are drawn in full
|
||||
colour with a glow and always shown, whatever the filters; healthy cables are drawn quietly.
|
||||
- **A card per device**, all the same height so the whole network reads at a glance: name, state
|
||||
(coloured edge and dot), and a picture: UniFi gear shows its real faceplate (the same drawings as the
|
||||
@@ -211,7 +227,7 @@ servers along the bottom. Power gear (PDU, UPS, power strip) sits in a rail on t
|
||||
named "Patch panel port 1" (the device beyond the panel is unknown); a stub that already has a name
|
||||
keeps it and adds "via Patch panel port 2". The cable details show where the cutsheet cable goes.
|
||||
- **Port names**: a ⚠ marks a port named after one device with a different one plugged in (a port
|
||||
called "Large1 Management" with the AP on it). Generic names ("Port 4", "SFP+ 2", "Uplink") and
|
||||
called "compute-storage-02 Management" with the AP on it). Generic names ("Port 4", "SFP+ 2", "Uplink") and
|
||||
"Downlink to X" with X on the other end never flag.
|
||||
- **Alerts**: a device with active alerts (not suppressed) gets a ring and a count in its worst
|
||||
severity colour, and the NIC it is about is marked.
|
||||
@@ -339,8 +355,13 @@ entry in `FACEPLATES`.
|
||||
Where every cable is meant to go, checked against what Gandalf actually sees. Anyone who can log in
|
||||
can view it; only the `admin` group can edit, and every change is logged with who made it and when.
|
||||
|
||||
The page has five tabs, each with its count: **Cables**, **To do** (undocumented links, port names
|
||||
in UniFi, labels to make), **Patch panels**, **Inventory** and **History**. A link to a section, cable
|
||||
or item (`/cutsheet#cable-7`, `#cs-portnames-section`) opens the tab that holds it, and
|
||||
`/cutsheet?problems=1` opens the cables with problems. Printing prints every tab.
|
||||
|
||||
- **Inventory**: things discovery can't see: patch panels, wall jacks, shelves, and rack positions
|
||||
for devices (use a device's exact Gandalf name). Panels and jacks are passive: a cable on a
|
||||
for devices (use a device's exact Gandalf name). The rack page draws them. Panels and jacks are passive: a cable on a
|
||||
port's front continues through the same port's rear.
|
||||
- **Cables**: two endpoints each (device or item, port, and front/rear for panels and jacks), plus
|
||||
ID, type, length, color, status (`planned`, `installed`, `dark`, `retired`) and notes. Leave the
|
||||
@@ -477,10 +498,55 @@ Those limits are wide (warning at -27.9 dBm on a link that normally sits near -1
|
||||
compares each reading to the link's own 7-day median to catch a damaged connector earlier. It stays quiet
|
||||
until `rx_drift_min_samples` readings exist (about a day at a 5 minute poll).
|
||||
|
||||
Uptime comes from the System Information page without a login. Firmware 6.40 shows it as D:HH:MM:SS
|
||||
(older firmware sent seconds; both are read). Uptime going backwards between polls records a restart, and
|
||||
the panel shows it (*restarted ... by ...*).
|
||||
|
||||
Outages can be over between polls: two real ones lasted about two minutes against a 5 minute poll.
|
||||
So WAN outages and modem restarts also come from the UniFi system log, and a drop in the modem's uptime
|
||||
records a restart even if nothing logged it. A single short outage only appears in the panel's history.
|
||||
|
||||
Internet quality (slow or lossy rather than down) comes from UniFi too, not from Gandalf's own pings: the
|
||||
UDM rates every 5 minutes as fine, high latency, packet loss or down, which is the timeline its Network app
|
||||
charts. Each cycle the monitor reads the last day of it from the dashboard API (read only) into
|
||||
`monitor_state.wan_quality`. The panel draws it as a 24 hour strip with the current latency to Cloudflare,
|
||||
Google and Microsoft. Needs attention says *Internet: packet loss* (red) when 2 or more of the last 12
|
||||
periods had loss or downtime, *high latency* (amber) when 3 or more had any problem, and lists drops from
|
||||
the last hour. These are dashboard lines, not tickets, and they go once the hour is clean. A copy older
|
||||
than 30 minutes is ignored.
|
||||
|
||||
**With `modem.access_code` set**, the monitor also logs in to the gateway (`ModemAdmin` in `modem.py`)
|
||||
for read-only checks, stored in `monitor_state.modem_checks`:
|
||||
- **NAT sessions** every 30 minutes, from the NAT Table page. Needs attention warns when 80% of the
|
||||
gateway's sessions (32,767 on a BGW320) are in use, since new connections stall once it is full.
|
||||
- **A ping from the gateway itself** to 1.1.1.1, from the Troubleshoot page, whenever UniFi flags the
|
||||
current 5 minutes as slow or measures 150 ms or more (at most every 10 minutes). Compared with UniFi's
|
||||
latency through the gateway at the same time, it says where the delay is: *past the modem* (its own
|
||||
ping is slow too, so the provider), *in the modem* (its ping is fine but ours through it is slow), or
|
||||
*fine by then*. The result goes in the detail of the Internet line in Needs attention and on the
|
||||
WAN / Fiber panel.
|
||||
- **Firewall drops** every 5 minutes, from the Logs page, counted per hour for a day. Most are *FLOOD
|
||||
limit 50pps*: AT&T's firmware caps new incoming connections at 50 per second (visitors to hosted
|
||||
services, peers), with no setting to change it. The log records at most one drop every 2 seconds and
|
||||
keeps about ten minutes, so the counts are a lower bound. Shown on the panel only.
|
||||
- **The gateway's own speed tests** (AT&T runs them from the modem), read with the NAT check; the latest
|
||||
down and up results are shown on the panel as a reference for what the line itself delivers.
|
||||
|
||||
**Restart modem** (admins, WAN / Fiber panel, needs `modem.access_code`): after a confirmation it
|
||||
1. switches the `uptime_kuma` maintenance window on (if that fails it stops and asks whether to go on
|
||||
without it),
|
||||
2. starts Gandalf maintenance (a global suppression for up to 20 minutes, shown in the banner),
|
||||
3. posts the gateway's Restart Device form. If that fails, steps 1 and 2 are undone.
|
||||
|
||||
The monitor then waits for the gateway to answer with its WAN up and an uptime showing it restarted
|
||||
(at least 3 minutes later), switches the Kuma window off, ends the suppression, and records how long it
|
||||
took (`monitor_state.modem_restart`). After 30 minutes it ends both regardless. One restart at a time.
|
||||
|
||||
The login posts md5(access code + nonce), as the gateway's own page does, so the code itself is never sent.
|
||||
The session is kept between checks and renewed when it expires. A refused code is reported in Needs
|
||||
attention and not tried again for 30 minutes, and any other failure pauses the checks for 10 minutes, so
|
||||
the gateway is never retried in a loop. The status polling above doesn't log in and is unaffected.
|
||||
|
||||
**One ticket per incident.** The modem and WAN reports from one check cycle open a single ticket, led by
|
||||
the most urgent (a P2 "WAN modem unreachable" leads, and an RPDU power-cycle seen in the same cycle is listed
|
||||
under "Also seen in this incident"). A report that comes while the modem has an open alert, or one that
|
||||
@@ -558,7 +624,7 @@ Shared by both processes. Located in the working directory (`/var/www/html/prod/
|
||||
"trusted_proxies": ["10.10.10.27"]
|
||||
},
|
||||
"hosts": [
|
||||
{ "name": "large1", "prometheus_instance": "10.10.10.2:9100" },
|
||||
{ "name": "compute-storage-02", "prometheus_instance": "10.10.10.2:9100" },
|
||||
{ "name": "compute-storage-01", "prometheus_instance": "10.10.10.4:9100" },
|
||||
{ "name": "monitor-01", "prometheus_instance": "10.10.10.8:9100" },
|
||||
{ "name": "monitor-02", "prometheus_instance": "10.10.10.9:9100" },
|
||||
@@ -624,6 +690,8 @@ Shared by both processes. Located in the working directory (`/var/www/html/prod/
|
||||
| `modem.failure_threshold` | Consecutive failures before ticketing a modem alert (default: same as `failure_threshold`). Raise it if gateway reboots or automatic firmware updates cut tickets. |
|
||||
| `modem.rx_drift_db` | Rx drop below the 7-day median that raises `fiber_rx_drift` (default: 3.0) |
|
||||
| `modem.rx_drift_min_samples` | Readings required before drift alerts start (default: 288) |
|
||||
| `uptime_kuma.url`, `.token`, `.maintenance_id` | Optional. Lets the Restart modem button switch an Uptime Kuma maintenance window on and off (Kuma 2.x, over its Socket.IO API). `token` is a Kuma session token, which works with a two-factor account and stops working if the Kuma password changes; `username` and `password` can be used instead for an account without two-factor login. Create the window once in Kuma with the *manual* strategy and the monitors a modem restart takes down, pause it, and put its id here. |
|
||||
| `modem.access_code` | Optional. The gateway's Device Access Code (printed on its label). With it, Gandalf logs in to read NAT session counts and to ping from the gateway when the internet is slow (see WAN modem notes). Keep it only in `config.json`, which is git-ignored and mode 600. |
|
||||
|
||||
---
|
||||
|
||||
@@ -808,7 +876,8 @@ curl http://10.10.10.48:9090/api/v1/query?query=up | jq '.data.result[] | select
|
||||
gandalf/
|
||||
├── app.py # Flask web app (routes, auth, API endpoints)
|
||||
├── monitor.py # Background daemon (Prometheus, UniFi, Pulse, alert logic)
|
||||
├── modem.py # Read-only AT&T gateway status page client and parser
|
||||
├── kuma.py # Switches one Uptime Kuma maintenance window on/off (Restart modem)
|
||||
├── modem.py # AT&T gateway client: status pages, plus optional logged-in checks and restart
|
||||
├── devices.py # Device inventory for search and device pages
|
||||
├── topology.py # The topology graph: devices, cables, depth, last-seen links
|
||||
├── cutsheet.py # Cutsheet validation, path tracing, plan-vs-actual checks, CSV
|
||||
@@ -890,7 +959,7 @@ Take LED colors from the model's quick start guide. See the comment at the top o
|
||||
`recovery_notified` once the recovery note is on its ticket (column added automatically at monitor startup)
|
||||
- `suppression_rules`: `active=FALSE` when removed; `expires_at` checked at query time
|
||||
- `monitor_state`: key/value store; `interface_baseline`, `link_stats`, and `modem_stats` are JSON blobs
|
||||
- `monitor_state` also holds `wan_history` (WAN outages, modem restarts) and `unifi_log_cursor` (how far the UniFi log has been read)
|
||||
- `monitor_state` also holds `wan_history` (WAN outages, modem restarts), `wan_quality` (UniFi's 5 minute WAN health timeline), `modem_checks` (NAT sessions, the latest ping from the gateway, hourly firewall drops and its speed test results) and `unifi_log_cursor` (how far the UniFi log has been read)
|
||||
- Database sessions run in UTC (`db.py`), whatever the server's time zone
|
||||
- `modem_samples`: optical readings per poll, kept 90 days; created automatically at monitor startup
|
||||
- `cutsheet_items`, `cutsheet_cables`, `cutsheet_port_labels`, `cutsheet_history`: cutsheet records and
|
||||
@@ -946,7 +1015,7 @@ Branch protection is enabled on `main`: both lint jobs must pass before any PR c
|
||||
Tests live in `tests/`:
|
||||
- `test_diagnose.py`: `DiagnosticsRunner` static methods (`build_ssh_command`, `parse_output`,
|
||||
`parse_sysfs_stats`, `parse_ethtool`, and variants)
|
||||
- `test_modem.py`: gateway page parsing, optical status, and WAN state
|
||||
- `test_modem.py`: gateway page parsing, optical status, WAN state, and the logged-in checks against a fake gateway
|
||||
- `test_monitor.py`: alert logic for dead hosts, Prometheus and Pulse outages, and modem and WAN log events
|
||||
- `test_devices.py`: device inventory behind search and device pages
|
||||
- `test_app.py`: page and API routes
|
||||
@@ -22,10 +22,13 @@ from urllib.parse import quote, urlparse
|
||||
|
||||
from flask import Flask, jsonify, make_response, redirect, render_template, request, send_file, url_for
|
||||
|
||||
import attention
|
||||
import cutsheet
|
||||
import db
|
||||
import devices
|
||||
import diagnose
|
||||
import kuma
|
||||
import modem as modem_api
|
||||
import topology
|
||||
from monitor import PulseClient, UnifiClient
|
||||
|
||||
@@ -554,10 +557,73 @@ def api_modem():
|
||||
logger.error(f'Failed to load modem history: {e}')
|
||||
data['history'] = []
|
||||
data['wan_history'] = _state_json('wan_history')
|
||||
data['wan_quality'] = _state_json('wan_quality') or None
|
||||
data['modem_checks'] = _state_json('modem_checks') or None
|
||||
data['modem_restart'] = _state_json('modem_restart') or None
|
||||
data['can_restart'] = bool((_config().get('modem') or {}).get('access_code'))
|
||||
data['configured'] = True
|
||||
return jsonify(data)
|
||||
|
||||
|
||||
# How long the silence lasts if the monitor never sees the modem come back to end it sooner
|
||||
MODEM_RESTART_SILENCE_MIN = 20
|
||||
|
||||
|
||||
@app.route('/api/modem/restart', methods=['POST'])
|
||||
@require_auth
|
||||
@require_admin
|
||||
def api_modem_restart():
|
||||
"""Restart the AT&T gateway, with alerts silenced first.
|
||||
|
||||
Body: {"confirm": true} and, after a Kuma failure the admin chose to accept,
|
||||
"without_kuma": true. Order: Uptime Kuma maintenance on, Gandalf maintenance on, then
|
||||
the restart; the first two are undone if the restart can't be sent. The monitor ends
|
||||
both once the modem is back (monitor._finish_modem_restart).
|
||||
"""
|
||||
body = _json_body()
|
||||
if body.get('confirm') is not True:
|
||||
return jsonify({'error': 'confirm must be true'}), 400
|
||||
cfg = _config()
|
||||
modem_cfg = cfg.get('modem') or {}
|
||||
if not modem_cfg.get('url') or not modem_cfg.get('access_code'):
|
||||
return jsonify({'error': 'Set modem.access_code in config.json first'}), 400
|
||||
pending = _state_json('modem_restart')
|
||||
if pending and not pending.get('finished_at') and time.time() - pending.get('at', 0) < 1800:
|
||||
return jsonify({'error': 'A restart is already in progress'}), 409
|
||||
|
||||
user = _get_user()['username']
|
||||
kuma_cfg = cfg.get('uptime_kuma') or {}
|
||||
kuma_on = False
|
||||
if kuma.configured(kuma_cfg):
|
||||
try:
|
||||
kuma.set_maintenance(kuma_cfg, True)
|
||||
kuma_on = True
|
||||
except kuma.KumaError as e:
|
||||
logger.warning(f'Uptime Kuma maintenance could not be started: {e}')
|
||||
if not body.get('without_kuma'):
|
||||
return jsonify({'error': f'Uptime Kuma: {e}', 'kuma_failed': True}), 502
|
||||
|
||||
sup_id = db.create_suppression('all', '', '', 'Modem restart from Gandalf', user,
|
||||
MODEM_RESTART_SILENCE_MIN)
|
||||
try:
|
||||
result = modem_api.ModemAdmin(modem_cfg).restart()
|
||||
except (modem_api.ModemUnreachable, modem_api.ModemDataInvalid, modem_api.ModemLoginFailed) as e:
|
||||
logger.error(f'Modem restart failed: {e}')
|
||||
db.deactivate_suppression(sup_id)
|
||||
if kuma_on:
|
||||
try:
|
||||
kuma.set_maintenance(kuma_cfg, False)
|
||||
except kuma.KumaError as ke:
|
||||
logger.error(f'Uptime Kuma maintenance could not be ended: {ke}')
|
||||
return jsonify({'error': f'The modem did not restart: {e}'}), 502
|
||||
|
||||
db.set_state('modem_restart', {'at': time.time(), 'by': user, 'result': result,
|
||||
'suppression_id': sup_id, 'kuma': kuma_on, 'finished_at': None})
|
||||
logger.warning(f'Modem restart requested by {user} ({result}); Kuma maintenance '
|
||||
f'{"on" if kuma_on else "not set"}, suppression #{sup_id}')
|
||||
return jsonify({'ok': True, 'result': result, 'kuma': kuma_on, 'suppression_id': sup_id})
|
||||
|
||||
|
||||
@app.route('/api/devices')
|
||||
@require_auth
|
||||
def api_devices():
|
||||
@@ -1335,6 +1401,33 @@ def api_topology():
|
||||
return jsonify(_topology_graph(events, sheet, feeds))
|
||||
|
||||
|
||||
@app.route('/api/attention')
|
||||
@require_auth
|
||||
def api_attention():
|
||||
"""Everything that needs looking at, from every page, for the dashboard (see attention.build)."""
|
||||
return jsonify(_attention())
|
||||
|
||||
|
||||
def _attention() -> dict:
|
||||
events = db.get_active_events(limit=_PAGE_LIMIT)
|
||||
_annotate_suppressions(events, db.get_active_suppressions())
|
||||
view, racks, feeds = None, {}, []
|
||||
try:
|
||||
view = _cutsheet_view()
|
||||
feeds = db.get_power_feeds()
|
||||
all_racks = _racks(view['items'])
|
||||
for rack in all_racks:
|
||||
here = [i for i in view['items'] if not i.get('shelf')
|
||||
and (i.get('rack') or all_racks[0]['name']) == rack['name']]
|
||||
racks[rack['name']] = cutsheet.rack_layout(here, rack['units'])
|
||||
except Exception as e:
|
||||
logger.error(f'Cutsheet unavailable for the attention list: {e}')
|
||||
sheet = {'items': (view or {}).get('items') or [], 'cables': (view or {}).get('cables') or []}
|
||||
power = _topology_graph(events, sheet, feeds).get('power') or []
|
||||
return attention.build(events, power, view, racks, _state_json('wan_quality'),
|
||||
modem_checks=_state_json('modem_checks'))
|
||||
|
||||
|
||||
def _topology_graph(events: list, sheet: dict, feeds: list) -> dict:
|
||||
return topology.build_graph(
|
||||
_state_json('network_snapshot'), _state_json('link_stats'), _state_json('modem_stats'),
|
||||
|
||||
+215
@@ -0,0 +1,215 @@
|
||||
"""What needs looking at, gathered from every page into one list for the dashboard.
|
||||
|
||||
Two groups: 'broken' is something not working right now; 'tidy' is housekeeping, where
|
||||
the network works but the records or names don't match it. Each entry links to the page
|
||||
where it is fixed. Pure functions, so the rules are easy to test.
|
||||
"""
|
||||
import time
|
||||
from typing import Dict, List, Optional
|
||||
|
||||
import modem
|
||||
|
||||
GROUPS = ('broken', 'tidy')
|
||||
|
||||
|
||||
def _plural(n: int, one: str, many: Optional[str] = None) -> str:
|
||||
return f'{n} {one if n == 1 else (many or one + "s")}'
|
||||
|
||||
|
||||
def _entry(group: str, severity: str, text: str, href: str, detail: str = '') -> dict:
|
||||
return {'group': group, 'severity': severity, 'text': text, 'href': href, 'detail': detail}
|
||||
|
||||
|
||||
def _alerts(events: List[dict]) -> List[dict]:
|
||||
live = [e for e in events if not e.get('is_suppressed') and e.get('severity') != 'info']
|
||||
if not live:
|
||||
return []
|
||||
critical = sum(1 for e in live if e.get('severity') == 'critical')
|
||||
targets = sorted({str(e.get('target_name') or '') for e in live if e.get('target_name')}, key=str.lower)
|
||||
text = _plural(len(live), 'active alert')
|
||||
if critical:
|
||||
text += f' ({critical} critical)'
|
||||
shown = ', '.join(targets[:4]) + (f' and {len(targets) - 4} more' if len(targets) > 4 else '')
|
||||
return [_entry('broken', 'bad' if critical else 'warn', text, '#events-table-wrap', shown)]
|
||||
|
||||
|
||||
def _power(power: List[dict]) -> List[dict]:
|
||||
out = []
|
||||
for src in power or []:
|
||||
name = src.get('node') or ''
|
||||
if src.get('stale_since'):
|
||||
out.append(_entry('broken', 'warn', f'{name}: outlet watts are frozen', '/topology',
|
||||
'UniFi keeps sending the same per-outlet readings; the total is still live. '
|
||||
'Restarting the unit usually clears it.'))
|
||||
for o in src.get('outlets') or []:
|
||||
if not o.get('on') and o.get('target'):
|
||||
out.append(_entry('broken', 'bad', f'{name} outlet {o.get("index")} is off, and it feeds {o["target"]}',
|
||||
'/topology'))
|
||||
idle = []
|
||||
for src in power or []:
|
||||
for o in src.get('outlets') or []:
|
||||
# Drawing real power, but nothing in Gandalf says what is plugged in
|
||||
if o.get('on') and not o.get('target') and o.get('match') != 'cleared' and (o.get('power_w') or 0) >= 5:
|
||||
idle.append(f'{src.get("node")} {o.get("index")}')
|
||||
if idle:
|
||||
out.append(_entry('tidy', 'info', f'{_plural(len(idle), "powered outlet")} with no device assigned',
|
||||
'/topology', ', '.join(idle[:6]) + (' ...' if len(idle) > 6 else '')))
|
||||
return out
|
||||
|
||||
|
||||
def _cutsheet(view: dict) -> List[dict]:
|
||||
out = []
|
||||
cables = [c for c in view.get('cables') or [] if c.get('status') != 'retired']
|
||||
bad = [c for c in cables if c.get('problem')]
|
||||
if bad:
|
||||
names = ', '.join(c.get('label') or f'#{c["id"]}' for c in bad[:5])
|
||||
out.append(_entry('broken', 'bad', f'{_plural(len(bad), "cable")} not where the cutsheet says',
|
||||
'/cutsheet?problems=1#cs-cables-section', names))
|
||||
|
||||
names = view.get('port_names') or []
|
||||
wrong = [r for r in names if r.get('kind') in ('wrong', 'moved')]
|
||||
if wrong:
|
||||
out.append(_entry('tidy', 'warn', f'{_plural(len(wrong), "UniFi port name")} pointing at the wrong device',
|
||||
'/cutsheet#cs-portnames-section',
|
||||
', '.join(f'{r["switch"]} port {r["port_idx"]} "{r["port_name"]}"' for r in wrong[:3])))
|
||||
empty = [r for r in names if r.get('kind') == 'empty']
|
||||
if empty:
|
||||
out.append(_entry('tidy', 'info', f'{_plural(len(empty), "named port")} with nothing plugged in',
|
||||
'/cutsheet#cs-portnames-section',
|
||||
', '.join(f'"{r["port_name"]}"' for r in empty[:4])))
|
||||
|
||||
missing = view.get('suggestions') or []
|
||||
if missing:
|
||||
out.append(_entry('tidy', 'info', f'{_plural(len(missing), "live link")} not on the cutsheet',
|
||||
'/cutsheet#cs-suggest-section'))
|
||||
|
||||
labels = view.get('labels') or {}
|
||||
todo = len(labels.get('cables') or []) + len(labels.get('ports') or [])
|
||||
if todo:
|
||||
out.append(_entry('tidy', 'info', f'{_plural(todo, "label")} to print and stick on',
|
||||
'/cutsheet#cs-labels-section'))
|
||||
return out
|
||||
|
||||
|
||||
def _rack(issues: Dict[str, dict]) -> List[dict]:
|
||||
out = []
|
||||
for rack, r in (issues or {}).items():
|
||||
for o in r.get('overlaps') or []:
|
||||
out.append(_entry('tidy', 'warn', f'{o["name"]} overlaps {", ".join(o["with"])} at {o["units"]}',
|
||||
'/cutsheet#cs-items-section', rack))
|
||||
for o in r.get('outside') or []:
|
||||
out.append(_entry('tidy', 'warn', f'{o["name"]} at {o["units"]} is outside {rack}',
|
||||
'/cutsheet#cs-items-section'))
|
||||
return out
|
||||
|
||||
|
||||
# UniFi's WAN timeline: judge the last hour, and ignore a copy the monitor stopped refreshing
|
||||
_WAN_WINDOW_S = 3600
|
||||
_WAN_STALE_S = 1800
|
||||
_WAN_BAD_PERIODS = 3 # periods with any problem before the list mentions it
|
||||
_WAN_LOSS_PERIODS = 2 # periods with packet loss before it counts as broken
|
||||
|
||||
|
||||
def _ago(seconds: float) -> str:
|
||||
minutes = max(1, round(seconds / 60))
|
||||
return f'{minutes} min ago' if minutes < 60 else f'{round(minutes / 60)} h ago'
|
||||
|
||||
|
||||
_NAT_FULL = 0.8 # share of the gateway's NAT sessions in use before it's worth a line
|
||||
|
||||
|
||||
def _modem_ping_note(checks: Optional[dict], now: float) -> str:
|
||||
"""One sentence on the latest ping from the modem itself, if it ran in the last hour."""
|
||||
ping = (checks or {}).get('ping') or {}
|
||||
if not ping or now - (ping.get('at') or 0) > _WAN_WINDOW_S:
|
||||
return ''
|
||||
when, avg, ours = _ago(now - ping['at']), ping.get('avg_ms'), ping.get('ours_ms')
|
||||
theirs = (f'{round(avg)} ms' if avg is not None else 'no reply') + (
|
||||
f', {ping["loss_pct"]}% lost' if ping.get('loss_pct') else '')
|
||||
ours_txt = f'; ours through it {round(ours)} ms' if ours is not None else ''
|
||||
verdict = {
|
||||
'upstream': "so the delay is past the modem, on the provider's side",
|
||||
'gateway': 'so the delay is in the modem, or between it and the router',
|
||||
'clear': 'both fine by then',
|
||||
}.get(ping.get('verdict'), '')
|
||||
return f"Modem's own ping {when}: {theirs}{ours_txt}, {verdict}. "
|
||||
|
||||
|
||||
def _modem_checks(checks: Optional[dict]) -> List[dict]:
|
||||
out = []
|
||||
nat = (checks or {}).get('nat') or {}
|
||||
if nat.get('available') and nat.get('in_use', 0) / nat['available'] >= _NAT_FULL:
|
||||
pct = round(100 * nat['in_use'] / nat['available'])
|
||||
out.append(_entry('broken', 'warn', f'Modem NAT table {pct}% full',
|
||||
'#modem-section', f'{nat["in_use"]:,} of {nat["available"]:,} sessions in use. '
|
||||
'When it fills, new connections stall or fail.'))
|
||||
error = (checks or {}).get('error') or {}
|
||||
if error.get('login'):
|
||||
out.append(_entry('tidy', 'warn', 'Modem login failed: check modem.access_code in config.json',
|
||||
'#modem-section', 'Gandalf waits 30 minutes before trying again.'))
|
||||
return out
|
||||
|
||||
|
||||
def _wan(quality: Optional[dict], now: float, checks: Optional[dict] = None) -> List[dict]:
|
||||
if not quality or now - (quality.get('updated') or 0) > _WAN_STALE_S:
|
||||
return []
|
||||
out = []
|
||||
periods = quality.get('periods') or []
|
||||
# The last hour is the newest 12 periods (each stamped with its start), so the count
|
||||
# doesn't dip while UniFi is still filling in the current one
|
||||
newest = periods[-1]['ts'] if periods else now
|
||||
recent = modem.quality_counts(periods, newest - _WAN_WINDOW_S + modem.QUALITY_PERIOD_S)
|
||||
total = sum(recent.values())
|
||||
bad = total - recent['ok']
|
||||
day = modem.quality_counts(periods, now - 86400)
|
||||
day_bad = sum(day.values()) - day['ok']
|
||||
latencies = quality.get('latencies') or {}
|
||||
detail = (f'UniFi flagged {day_bad} of the last {sum(day.values())} five-minute periods. '
|
||||
if day_bad else '')
|
||||
if latencies:
|
||||
detail += 'Now: ' + ', '.join(f'{k} {v} ms' for k, v in sorted(latencies.items())) + '. '
|
||||
detail += _modem_ping_note(checks, now)
|
||||
detail += 'See the 24 hour strip on the WAN / Fiber panel.'
|
||||
lossy = recent['loss'] + recent['down']
|
||||
if lossy >= _WAN_LOSS_PERIODS:
|
||||
out.append(_entry('broken', 'bad', f'Internet: packet loss in {lossy} of the last {total} '
|
||||
'five-minute periods', '#modem-section', detail))
|
||||
elif bad >= _WAN_BAD_PERIODS:
|
||||
out.append(_entry('broken', 'warn', f'Internet: high latency in {bad} of the last {total} '
|
||||
'five-minute periods', '#modem-section', detail))
|
||||
drops = [d for d in quality.get('downtime') or [] if d['ts'] >= now - _WAN_WINDOW_S]
|
||||
if len(drops) == 1:
|
||||
out.append(_entry('broken', 'warn', f'Internet dropped for {round(drops[0]["seconds"])} s, '
|
||||
f'{_ago(now - drops[0]["ts"])}', '#modem-section'))
|
||||
elif drops:
|
||||
out.append(_entry('broken', 'warn', f'Internet dropped {len(drops)} times in the last hour, '
|
||||
f'{round(sum(d["seconds"] for d in drops))} s in all',
|
||||
'#modem-section'))
|
||||
return out
|
||||
|
||||
|
||||
def build(events: List[dict], power: List[dict], cutsheet_view: Optional[dict],
|
||||
rack_issues: Optional[Dict[str, dict]] = None, wan_quality: Optional[dict] = None,
|
||||
now: Optional[float] = None, modem_checks: Optional[dict] = None) -> dict:
|
||||
"""{'broken': [entry], 'tidy': [entry]}, worst first within each group.
|
||||
|
||||
events: active alerts, annotated with 'is_suppressed'. power: topology.build_power output.
|
||||
cutsheet_view: the cutsheet API view (cables with 'problem', suggestions, port_names,
|
||||
labels), or None when the cutsheet couldn't be loaded. rack_issues: {rack: rack_layout()}.
|
||||
wan_quality: modem.wan_quality() output as stored by the monitor. modem_checks: the
|
||||
monitor's logged-in modem checks (NAT sessions, ping from the modem). now: for tests.
|
||||
Entry: {group, severity: bad|warn|info, text, href, detail}.
|
||||
"""
|
||||
now = time.time() if now is None else now
|
||||
entries = (_wan(wan_quality, now, modem_checks) + _modem_checks(modem_checks)
|
||||
+ _alerts(events) + _power(power))
|
||||
if cutsheet_view is not None:
|
||||
entries += _cutsheet(cutsheet_view)
|
||||
entries += _rack(rack_issues or {})
|
||||
rank = {'bad': 0, 'warn': 1, 'info': 2}
|
||||
out = {g: [] for g in GROUPS}
|
||||
for e in entries:
|
||||
out[e['group']].append(e)
|
||||
for g in GROUPS:
|
||||
out[g].sort(key=lambda e: rank[e['severity']])
|
||||
return out
|
||||
@@ -0,0 +1,84 @@
|
||||
{
|
||||
"pulse": {
|
||||
"url": "http://10.10.10.65:8080",
|
||||
"api_key": "012b303a324152c509bf5ade6f942cfc21404f68662f01a17001cba9e4486049",
|
||||
"worker_id": "1b11d1b5-4ed0-42df-a6af-8d57fffe1343",
|
||||
"timeout": 45
|
||||
},
|
||||
"unifi": {
|
||||
"controller": "https://10.10.10.1",
|
||||
"api_key": "kyPfIsAVie3hwMD4Bc1MjAu8N7HVPIb8",
|
||||
"site_id": "default"
|
||||
},
|
||||
"prometheus": {
|
||||
"url": "http://10.10.10.48:9090"
|
||||
},
|
||||
"database": {
|
||||
"host": "10.10.10.50",
|
||||
"port": 3306,
|
||||
"user": "gandalf",
|
||||
"password": "Gandalf2026Lotus",
|
||||
"name": "gandalf"
|
||||
},
|
||||
"ticket_api": {
|
||||
"url": "http://10.10.10.45/create_ticket_api.php",
|
||||
"api_key": "5acc5d3c647b84f7c6f59082ce4450ee772e2d1633238b960136f653d20c93af"
|
||||
},
|
||||
"ldap": {
|
||||
"host": "10.10.10.39",
|
||||
"port": 3890,
|
||||
"bind_dn": "uid=gandalf,ou=people,dc=example,dc=com",
|
||||
"bind_pw": "AZ-eUFRxamnlLEELuDy5-Mh6z-2cJbL_YJZnu64Tf0A",
|
||||
"user_base": "ou=people,dc=example,dc=com",
|
||||
"cache_dir": "/tmp/gandalf_avatars",
|
||||
"cache_ttl": 3600
|
||||
},
|
||||
"auth": {
|
||||
"allowed_groups": ["admin"]
|
||||
},
|
||||
"monitor": {
|
||||
"poll_interval": 300,
|
||||
"failure_threshold": 2,
|
||||
"unifi_failure_threshold": 4,
|
||||
"cluster_threshold": 3,
|
||||
"ping_hosts": [],
|
||||
"links_exclude_ips": ["10.10.10.29", "10.10.10.44", "10.10.10.3"]
|
||||
},
|
||||
"hosts": [
|
||||
{
|
||||
"name": "pbs",
|
||||
"ip": "10.10.10.3",
|
||||
"prometheus_instance": "10.10.10.3:9100"
|
||||
},
|
||||
{
|
||||
"name": "large1",
|
||||
"ip": "10.10.10.2",
|
||||
"prometheus_instance": "10.10.10.2:9100"
|
||||
},
|
||||
{
|
||||
"name": "compute-storage-01",
|
||||
"ip": "10.10.10.4",
|
||||
"prometheus_instance": "10.10.10.4:9100"
|
||||
},
|
||||
{
|
||||
"name": "monitor-01",
|
||||
"ip": "10.10.10.8",
|
||||
"prometheus_instance": "10.10.10.8:9100"
|
||||
},
|
||||
{
|
||||
"name": "monitor-02",
|
||||
"ip": "10.10.10.9",
|
||||
"prometheus_instance": "10.10.10.9:9100"
|
||||
},
|
||||
{
|
||||
"name": "compute-storage-gpu-01",
|
||||
"ip": "10.10.10.10",
|
||||
"prometheus_instance": "10.10.10.10:9100"
|
||||
},
|
||||
{
|
||||
"name": "storage-01",
|
||||
"ip": "10.10.10.11",
|
||||
"prometheus_instance": "10.10.10.11:9100"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,74 @@
|
||||
"""Switch one Uptime Kuma maintenance window on and off (Kuma 2.x, over its Socket.IO API).
|
||||
|
||||
Kuma has no REST API for maintenance, so this signs in over the same socket its web UI
|
||||
uses. It only ever resumes or pauses the one window named in config, which is created
|
||||
once in Kuma ("manual" strategy, covering the monitors that a modem restart takes down):
|
||||
|
||||
"uptime_kuma": {"url": "http://10.10.10.25:3001", "token": "...", "maintenance_id": 7}
|
||||
|
||||
`token` is a Kuma session token (works with a two-factor account; it stops working if
|
||||
the Kuma password changes). `username` and `password` are accepted instead for an
|
||||
account without two-factor login. Nothing else in Kuma is read or changed.
|
||||
"""
|
||||
import logging
|
||||
import threading
|
||||
from typing import Callable, Optional
|
||||
|
||||
import socketio
|
||||
|
||||
logger = logging.getLogger('gandalf.kuma')
|
||||
|
||||
|
||||
class KumaError(Exception):
|
||||
"""Kuma couldn't be reached, refused the sign-in, or refused the change."""
|
||||
|
||||
|
||||
def configured(cfg: Optional[dict]) -> bool:
|
||||
cfg = cfg or {}
|
||||
signed_in = bool(cfg.get('token')) or bool(cfg.get('username') and cfg.get('password'))
|
||||
return bool(cfg.get('url') and cfg.get('maintenance_id')) and signed_in
|
||||
|
||||
|
||||
def _sign_in(sio, cfg: dict, timeout: float) -> None:
|
||||
if cfg.get('token'):
|
||||
res = sio.call('loginByToken', cfg['token'], timeout=timeout) or {}
|
||||
if not res.get('ok'):
|
||||
raise KumaError('token rejected (Kuma password changed?)')
|
||||
return
|
||||
res = sio.call('login', {'username': cfg['username'], 'password': cfg['password'], 'token': ''},
|
||||
timeout=timeout) or {}
|
||||
if not res.get('ok'):
|
||||
if res.get('tokenRequired'):
|
||||
raise KumaError('the Kuma account has two-factor login: use uptime_kuma.token instead')
|
||||
raise KumaError(f'login refused: {res.get("msg") or "no reason given"}')
|
||||
|
||||
|
||||
def set_maintenance(cfg: dict, active: bool, client_factory: Optional[Callable] = None) -> None:
|
||||
"""Resume (active=True) or pause the configured maintenance window. Raises KumaError."""
|
||||
timeout = cfg.get('timeout', 10)
|
||||
factory = client_factory or (lambda: socketio.Client(
|
||||
reconnection=False, ssl_verify=cfg.get('verify_ssl', True)))
|
||||
sio = factory()
|
||||
# Kuma registers its event handlers only after it sends 'info', so anything emitted
|
||||
# before that is silently dropped and the call times out.
|
||||
ready = threading.Event()
|
||||
sio.on('info', lambda *args: ready.set())
|
||||
try:
|
||||
sio.connect(cfg['url'], transports=['websocket'], wait_timeout=timeout)
|
||||
if not ready.wait(timeout):
|
||||
raise KumaError('Kuma not ready (no info event after connecting)')
|
||||
_sign_in(sio, cfg, timeout)
|
||||
event = 'resumeMaintenance' if active else 'pauseMaintenance'
|
||||
res = sio.call(event, int(cfg['maintenance_id']), timeout=timeout) or {}
|
||||
if not res.get('ok'):
|
||||
raise KumaError(f'{event} refused: {res.get("msg") or "no reason given"}')
|
||||
logger.info(f'Uptime Kuma maintenance {cfg["maintenance_id"]} {"on" if active else "off"}')
|
||||
except KumaError:
|
||||
raise
|
||||
except Exception as e: # socketio raises its own errors and plain timeouts
|
||||
raise KumaError(f'{type(e).__name__}: {e}') from e
|
||||
finally:
|
||||
try:
|
||||
sio.disconnect()
|
||||
except Exception: # nosec B110 - nothing left to clean up
|
||||
pass
|
||||
@@ -1,13 +1,20 @@
|
||||
"""Read-only client for AT&T BGW-series fiber gateways (tested on BGW320-505).
|
||||
"""Client for AT&T BGW-series fiber gateways (tested on BGW320-505).
|
||||
|
||||
The gateway has no API, but its status pages are served without a login, so
|
||||
this scrapes them over HTTP. Only GET requests to the fixed status pages below
|
||||
are ever made: nothing here can change gateway settings.
|
||||
ModemClient scrapes them over HTTP with GET requests only: it cannot change
|
||||
anything on the gateway.
|
||||
|
||||
With an access code configured, ModemAdmin (at the end) also logs in to read the
|
||||
NAT session counts, the firewall log and speed test history, and to run the
|
||||
Troubleshoot page's ping from the gateway. None of that changes a setting. Its
|
||||
restart() is the one call that does something to the gateway, and it runs only
|
||||
when an admin presses Restart modem.
|
||||
|
||||
Units come from the gateway's own help text, checked against live readings:
|
||||
Rx/Tx power tenths of a dBm (-169 -> -16.9 dBm)
|
||||
Temperature degrees C
|
||||
Uptime seconds (despite the help text claiming D:H:M:S)
|
||||
Uptime D:HH:MM:SS on firmware 6.40 (older firmware sent plain seconds;
|
||||
both are accepted)
|
||||
|
||||
Deliberately not used, because the page disagrees with itself:
|
||||
Vcc / Tx Bias readings values do not match the documented scales, so only
|
||||
@@ -16,9 +23,11 @@ Deliberately not used, because the page disagrees with itself:
|
||||
Status" is the reliable signal
|
||||
Last Change does not count up, despite the help text
|
||||
"""
|
||||
import hashlib
|
||||
import html
|
||||
import logging
|
||||
import re
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from typing import Dict, List, Optional
|
||||
|
||||
@@ -136,6 +145,18 @@ def _to_int(raw: Optional[str]) -> Optional[int]:
|
||||
return None
|
||||
|
||||
|
||||
def parse_uptime(raw: Optional[str]) -> Optional[int]:
|
||||
"""Seconds since the gateway restarted, from 'D:HH:MM:SS' (e.g. 03:06:29:34) or plain seconds."""
|
||||
raw = (raw or '').strip()
|
||||
if re.fullmatch(r'\d+', raw):
|
||||
return int(raw)
|
||||
m = re.fullmatch(r'(\d+):(\d{1,2}):(\d{1,2}):(\d{1,2})', raw)
|
||||
if not m:
|
||||
return None
|
||||
d, h, mi, sec = (int(x) for x in m.groups())
|
||||
return ((d * 24 + h) * 60 + mi) * 60 + sec
|
||||
|
||||
|
||||
def wan_problems(data: dict) -> List[str]:
|
||||
"""Return human-readable reasons the WAN is down; empty when it looks healthy."""
|
||||
problems = []
|
||||
@@ -164,7 +185,7 @@ def build_status(fiber_page: str, broadband_page: str, sysinfo_page: str = '') -
|
||||
data = {
|
||||
'model': sysinfo.get('Model Number', ''),
|
||||
'firmware': sysinfo.get('Software Version', ''),
|
||||
'uptime_s': _to_int(sysinfo.get('Time Since Last Reboot')),
|
||||
'uptime_s': parse_uptime(sysinfo.get('Time Since Last Reboot')),
|
||||
'broadband_state': broadband.get('Broadband Connection', ''),
|
||||
'source': broadband.get('Broadband Connection Source', ''),
|
||||
'wan_ip': broadband.get('Broadband IPv4 Address', ''),
|
||||
@@ -310,3 +331,273 @@ def record_uptime_reset(history: dict, boot_ts: float) -> None:
|
||||
return
|
||||
restarts.append({'at': boot_ts, 'by': None, 'confirmed': True})
|
||||
del restarts[:-_HISTORY_LIMIT]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# WAN quality: UniFi's own 5 minute health timeline (high latency, packet loss, down),
|
||||
# the same data its Network app charts. Read only; Gandalf doesn't measure it again.
|
||||
# ---------------------------------------------------------------------------
|
||||
QUALITY_STATES = ('ok', 'latency', 'loss', 'down') # mildest to worst
|
||||
QUALITY_PERIOD_S = 300
|
||||
|
||||
|
||||
def wan_quality(dashboard: dict, now: float) -> Optional[dict]:
|
||||
"""The primary WAN's health timeline from UniFi's aggregated dashboard, or None.
|
||||
|
||||
Returns {'wan', 'periods': [{'ts', 'state'}], 'downtime': [{'ts', 'seconds'}],
|
||||
'latencies': {service: ms}, 'updated'}. Each period is 5 minutes; its state is the
|
||||
worst flag UniFi set for it.
|
||||
"""
|
||||
details = ((dashboard or {}).get('wan_history') or {}).get('wan_history_details') or []
|
||||
if not details:
|
||||
return None
|
||||
primary = next((d for d in details if d.get('network_group') == 'WAN'), details[0])
|
||||
periods = []
|
||||
for h in primary.get('health_history') or []:
|
||||
if not isinstance(h.get('timestamp'), (int, float)):
|
||||
continue
|
||||
state = ('down' if h.get('wan_downtime') else 'loss' if h.get('packet_loss')
|
||||
else 'latency' if h.get('high_latency') else 'ok')
|
||||
periods.append({'ts': h['timestamp'] / 1000, 'state': state})
|
||||
periods.sort(key=lambda p: p['ts'])
|
||||
downtime = sorted(
|
||||
({'ts': d['timestamp'] / 1000, 'seconds': d.get('downtime_in_seconds') or 0}
|
||||
for d in primary.get('downtime_history') or [] if isinstance(d.get('timestamp'), (int, float))),
|
||||
key=lambda d: d['ts'])
|
||||
latencies = {}
|
||||
for w in ((dashboard.get('wan') or {}).get('wan_details') or []):
|
||||
if w.get('network_group') == primary.get('network_group'):
|
||||
for s in (w.get('stats') or {}).get('service_latencies') or []:
|
||||
if s.get('service_name') and isinstance(s.get('latency'), (int, float)):
|
||||
latencies[s['service_name']] = s['latency']
|
||||
return {'wan': primary.get('network_group') or '', 'periods': periods, 'downtime': downtime,
|
||||
'latencies': latencies, 'updated': now}
|
||||
|
||||
|
||||
def quality_counts(periods: List[dict], since: float) -> Dict[str, int]:
|
||||
"""How many periods starting at or after *since* were in each state."""
|
||||
counts = {s: 0 for s in QUALITY_STATES}
|
||||
for p in periods:
|
||||
if p['ts'] >= since and p['state'] in counts:
|
||||
counts[p['state']] += 1
|
||||
return counts
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Logged-in checks, used only when an access code is configured. Two pages are
|
||||
# read: the NAT table's session counts, and the Troubleshoot page's ping test,
|
||||
# which runs a ping from the gateway itself. Neither changes any setting.
|
||||
# ---------------------------------------------------------------------------
|
||||
_LOGIN_PAGE = '/cgi-bin/login.ha'
|
||||
_NAT_PAGE = '/cgi-bin/nattable.ha'
|
||||
_DIAG_PAGE = '/cgi-bin/diag.ha'
|
||||
_LOGS_PAGE = '/cgi-bin/logs.ha'
|
||||
_SPEED_PAGE = '/cgi-bin/speed.ha'
|
||||
_RESTART_PAGE = '/cgi-bin/restart.ha'
|
||||
_RESTARTING_PAGE = '/cgi-bin/restarting.ha'
|
||||
# Firewall log row: number, ISO time (gateway's local clock), source, destination, protocol, reason
|
||||
_FW_ROW_RE = re.compile(r'<tr[^>]*>\s*<td[^>]*>\s*\d+\s*</td>\s*<td>\s*([^<]+?)\s*</td>\s*<td>\s*([^<]*?)\s*</td>'
|
||||
r'\s*<td>\s*([^<]*?)\s*</td>\s*<td>\s*([^<]*?)\s*</td>\s*<td>\s*([^<]*?)\s*</td>\s*</tr>', re.I)
|
||||
_CELL_RE = re.compile(r'<td\b[^>]*>(.*?)</td>', re.I | re.S)
|
||||
_NONCE_RE = re.compile(r'name="nonce"\s+value="([0-9a-f]+)"', re.I)
|
||||
_PROGRESS_RE = re.compile(r'<textarea[^>]*name="ProgressWindow"[^>]*>(.*?)</textarea>', re.I | re.S)
|
||||
_PING_STATS_RE = re.compile(r'(\d+) packets transmitted, (\d+) (?:packets )?received')
|
||||
_PING_RTT_RE = re.compile(r'min/avg/max\S* = ([\d.]+)/([\d.]+)/([\d.]+)')
|
||||
# After a failed login, leave the gateway alone for this long (it may lock the account)
|
||||
LOGIN_BACKOFF_S = 1800
|
||||
|
||||
|
||||
class ModemLoginFailed(Exception):
|
||||
"""The access code was refused, or the login page wasn't what we expect."""
|
||||
|
||||
|
||||
def login_hash(code: str, nonce: str) -> str:
|
||||
"""What the login page's script posts: md5 of the access code followed by the nonce."""
|
||||
# The gateway's login protocol dictates md5; it isn't used to protect anything here
|
||||
return hashlib.md5((code + nonce).encode(), usedforsecurity=False).hexdigest() # nosec B303 B324
|
||||
|
||||
|
||||
def is_login_page(page: str) -> bool:
|
||||
return 'Access Code Required' in page or bool(re.search(r'<title>\s*Login', page, re.I))
|
||||
|
||||
|
||||
def parse_nat_summary(page: str) -> Optional[Dict[str, int]]:
|
||||
"""{'in_use', 'available'} from the NAT table page, or None if the counts are missing."""
|
||||
rows = {_TAG_RE.sub('', k).strip(): _TAG_RE.sub('', v).strip() for k, v in _ROW_RE.findall(page)}
|
||||
try:
|
||||
return {'in_use': int(rows['Total sessions in use']),
|
||||
'available': int(rows['Total sessions available'])}
|
||||
except (KeyError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def parse_ping_output(text: str) -> Optional[dict]:
|
||||
"""{'sent', 'received', 'loss_pct', 'min_ms', 'avg_ms', 'max_ms'} from busybox ping output,
|
||||
or None until the test has finished. Times are None when every packet was lost."""
|
||||
stats = _PING_STATS_RE.search(text or '')
|
||||
if not stats:
|
||||
return None
|
||||
sent, received = int(stats.group(1)), int(stats.group(2))
|
||||
rtt = _PING_RTT_RE.search(text)
|
||||
lo, avg, hi = (float(x) for x in rtt.groups()) if rtt else (None, None, None)
|
||||
return {'sent': sent, 'received': received,
|
||||
'loss_pct': round(100 * (sent - received) / sent) if sent else 100,
|
||||
'min_ms': lo, 'avg_ms': avg, 'max_ms': hi}
|
||||
|
||||
|
||||
def parse_firewall_log(page: str) -> List[dict]:
|
||||
"""Rows of the gateway's firewall log (Logs page): {'ts', 'src', 'dst', 'proto', 'reason'}.
|
||||
'ts' is the gateway's local time as ISO text, so rows sort by it."""
|
||||
rows = [dict(zip(('ts', 'src', 'dst', 'proto', 'reason'), m.groups())) for m in _FW_ROW_RE.finditer(page)]
|
||||
return sorted((r for r in rows if re.match(r'\d{4}-\d\d-\d\dT', r['ts'])), key=lambda r: r['ts'])
|
||||
|
||||
|
||||
def firewall_kind(reason: str) -> str:
|
||||
"""'flood' (new connections over the gateway's fixed rate limit), 'invalid', or 'other'."""
|
||||
reason = reason.lower()
|
||||
return 'flood' if 'flood' in reason else 'invalid' if 'invalid' in reason else 'other'
|
||||
|
||||
|
||||
def parse_speed_tests(page: str) -> List[dict]:
|
||||
"""The gateway's own speed test history, newest first:
|
||||
{'at' (as the page shows it), 'direction', 'mbps', 'latency_ms', 'ok'}."""
|
||||
i = page.find('Test History')
|
||||
out = []
|
||||
for row in re.findall(r'<tr\b[^>]*>(.*?)</tr>', page[i:] if i >= 0 else '', re.I | re.S):
|
||||
cells = [_TAG_RE.sub('', c).strip() for c in _CELL_RE.findall(row)]
|
||||
if len(cells) < 6 or cells[1] not in ('upstream', 'downstream'):
|
||||
continue
|
||||
try:
|
||||
out.append({'at': cells[0], 'direction': cells[1], 'mbps': round(float(cells[2]), 1),
|
||||
'latency_ms': round(float(cells[4]), 1), 'ok': cells[5].lower() == 'success'})
|
||||
except ValueError:
|
||||
continue
|
||||
return out
|
||||
|
||||
|
||||
class ModemAdmin:
|
||||
"""Logged-in session on the gateway, kept between checks and renewed when it expires.
|
||||
|
||||
Logs in at most once per call, and not at all for LOGIN_BACKOFF_S after a refusal.
|
||||
Raises ModemUnreachable, ModemDataInvalid or ModemLoginFailed.
|
||||
"""
|
||||
|
||||
def __init__(self, cfg: dict, clock=None, sleep=None):
|
||||
self.base_url = cfg['url'].rstrip('/')
|
||||
self._code = cfg['access_code']
|
||||
self.timeout = cfg.get('timeout', 10)
|
||||
self.session = requests.Session()
|
||||
self._clock = clock or time.time
|
||||
self._sleep = sleep or time.sleep
|
||||
self.login_failed_at: Optional[float] = None
|
||||
|
||||
def _request(self, method: str, path: str, timeout: Optional[float] = None, **kw) -> requests.Response:
|
||||
try:
|
||||
return self.session.request(method, f'{self.base_url}{path}', allow_redirects=False,
|
||||
timeout=timeout or self.timeout, **kw)
|
||||
except requests.RequestException as e:
|
||||
raise ModemUnreachable(f'{path}: {e}') from e
|
||||
|
||||
def _login(self) -> None:
|
||||
if self.login_failed_at and self._clock() - self.login_failed_at < LOGIN_BACKOFF_S:
|
||||
raise ModemLoginFailed('waiting after a refused login')
|
||||
page = self._request('GET', _LOGIN_PAGE).text
|
||||
nonce = _NONCE_RE.search(page)
|
||||
if not nonce: # the first visit only sets the session cookie
|
||||
nonce = _NONCE_RE.search(self._request('GET', _LOGIN_PAGE).text)
|
||||
if not nonce:
|
||||
raise ModemDataInvalid('login page has no nonce')
|
||||
resp = self._request('POST', _LOGIN_PAGE, data={
|
||||
'nonce': nonce.group(1), 'password': '*' * len(self._code), 'Continue': 'Continue',
|
||||
'hashpassword': login_hash(self._code, nonce.group(1)),
|
||||
})
|
||||
if resp.status_code != 302 or is_login_page(resp.text):
|
||||
self.login_failed_at = self._clock()
|
||||
raise ModemLoginFailed('the gateway refused the access code')
|
||||
self.login_failed_at = None
|
||||
|
||||
def _page(self, path: str, timeout: Optional[float] = None) -> str:
|
||||
"""A protected page, logging in first if the session has expired."""
|
||||
resp = self._request('GET', path, timeout)
|
||||
if resp.status_code != 200 or is_login_page(resp.text):
|
||||
self._login()
|
||||
resp = self._request('GET', path, timeout)
|
||||
if resp.status_code != 200 or is_login_page(resp.text):
|
||||
raise ModemDataInvalid(f'{path} returned HTTP {resp.status_code} after logging in')
|
||||
return resp.text
|
||||
|
||||
def nat_sessions(self) -> Dict[str, int]:
|
||||
# The page lists every session too, so it can be large (hundreds of KB)
|
||||
counts = parse_nat_summary(self._page(_NAT_PAGE, timeout=max(self.timeout, 30)))
|
||||
if counts is None:
|
||||
raise ModemDataInvalid('NAT table page has no session counts')
|
||||
return counts
|
||||
|
||||
def firewall_log(self) -> List[dict]:
|
||||
return parse_firewall_log(self._page(_LOGS_PAGE))
|
||||
|
||||
def speed_tests(self) -> List[dict]:
|
||||
return parse_speed_tests(self._page(_SPEED_PAGE))
|
||||
|
||||
def restart(self) -> str:
|
||||
"""Restart the gateway from its Restart Device page: the one change this client makes,
|
||||
and only when an admin asks.
|
||||
|
||||
Two steps, as in a browser: posting the form only answers with a redirect to
|
||||
restarting.ha, and loading that page is what starts the restart (firmware 6.40; a
|
||||
post alone leaves the gateway running). Returns 'accepted' once that page says it is
|
||||
restarting, or 'unconfirmed' when the gateway dropped the connection first.
|
||||
"""
|
||||
nonce = _NONCE_RE.search(self._page(_RESTART_PAGE))
|
||||
if not nonce:
|
||||
raise ModemDataInvalid('Restart page has no nonce')
|
||||
timeout = max(self.timeout, 30)
|
||||
try:
|
||||
resp = self.session.post(f'{self.base_url}{_RESTART_PAGE}', allow_redirects=False,
|
||||
timeout=timeout, data={'nonce': nonce.group(1), 'Restart': 'Restart'})
|
||||
if resp.status_code == 302 and _RESTARTING_PAGE in (resp.headers.get('Location') or ''):
|
||||
resp = self.session.get(f'{self.base_url}{_RESTARTING_PAGE}', allow_redirects=False,
|
||||
timeout=timeout)
|
||||
except (requests.ConnectionError, requests.Timeout):
|
||||
return 'unconfirmed'
|
||||
if resp.status_code != 200 or is_login_page(resp.text) or 'restarting' not in resp.text.lower():
|
||||
raise ModemDataInvalid(f'Restart was not started (HTTP {resp.status_code})')
|
||||
return 'accepted'
|
||||
|
||||
def ping(self, target: str, wait_s: float = 30) -> dict:
|
||||
"""Ping *target* from the gateway (its Troubleshoot page; 5 packets) and return the result."""
|
||||
nonce = _NONCE_RE.search(self._page(_DIAG_PAGE))
|
||||
if not nonce:
|
||||
raise ModemDataInvalid('Troubleshoot page has no nonce')
|
||||
resp = self._request('POST', _DIAG_PAGE, timeout=max(self.timeout, 30), data={
|
||||
'nonce': nonce.group(1), 'WebAddress': target, 'protopref': 'IPv4', 'Ping': 'Ping'})
|
||||
deadline = self._clock() + wait_s
|
||||
page = resp.text
|
||||
while True:
|
||||
window = _PROGRESS_RE.search(page)
|
||||
result = parse_ping_output(html.unescape(window.group(1)) if window else '')
|
||||
if result is not None:
|
||||
return dict(result, target=target)
|
||||
if self._clock() >= deadline:
|
||||
raise ModemDataInvalid(f'ping to {target} from the gateway did not finish')
|
||||
self._sleep(3)
|
||||
page = self._page(_DIAG_PAGE)
|
||||
|
||||
|
||||
SLOW_MS = 150 # a ping slower than this, on a link that normally answers in ~15 ms, is "slow"
|
||||
|
||||
|
||||
def ping_verdict(gateway: dict, ours_ms: Optional[float]) -> str:
|
||||
"""Where the delay is, from the gateway's own ping and ours through it at the same time.
|
||||
|
||||
'upstream': the gateway's ping is slow or lossy too, so it's past the gateway (the ISP).
|
||||
'gateway': the gateway's ping is fine but ours through it is slow, so the delay is in
|
||||
the gateway or between it and the router.
|
||||
'clear': both fine; the slow spell had passed by the time of the test.
|
||||
"""
|
||||
avg = gateway.get('avg_ms')
|
||||
if avg is None or avg >= SLOW_MS or (gateway.get('loss_pct') or 0) >= 20:
|
||||
return 'upstream'
|
||||
if ours_ms is not None and ours_ms >= SLOW_MS:
|
||||
return 'gateway'
|
||||
return 'clear'
|
||||
+170
-1
@@ -24,6 +24,7 @@ from urllib3.exceptions import InsecureRequestWarning
|
||||
import db
|
||||
import cutsheet
|
||||
import devices
|
||||
import kuma
|
||||
import modem as modem_api
|
||||
import ticket_text
|
||||
|
||||
@@ -229,6 +230,19 @@ class UnifiClient:
|
||||
logger.error(f'UniFi system log error: {e}')
|
||||
return None
|
||||
|
||||
def get_wan_dashboard(self) -> Optional[dict]:
|
||||
"""UniFi's aggregated dashboard for the last day (WAN health timeline and service
|
||||
latencies, as the Network app charts them), or None if unavailable. Read only."""
|
||||
url = f'{self.base_url}/proxy/network/v2/api/site/{self.site_id}/aggregated-dashboard'
|
||||
try:
|
||||
resp = self.session.get(url, headers=self.headers, timeout=20, params={'historySeconds': 86400})
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
return data if isinstance(data, dict) else None
|
||||
except Exception as e:
|
||||
logger.error(f'UniFi dashboard error: {e}')
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def _update_fields(params: dict) -> dict:
|
||||
"""The app and versions of an update entry; nothing for other entries."""
|
||||
@@ -901,7 +915,7 @@ class LinkStatsCollector:
|
||||
"""
|
||||
Collect full link stats for all Prometheus-monitored hosts.
|
||||
|
||||
*instance_map*: ``{'10.10.10.2:9100': 'large1', ...}``
|
||||
*instance_map*: ``{'10.10.10.2:9100': 'compute-storage-02', ...}``
|
||||
|
||||
Returns a dict suitable for ``db.set_state('link_stats', ...)``.
|
||||
"""
|
||||
@@ -1098,6 +1112,9 @@ class NetworkMonitor:
|
||||
modem_cfg = self.cfg.get('modem') or {}
|
||||
self.modem = modem_api.ModemClient(modem_cfg) if modem_cfg.get('url') else None
|
||||
self.modem_name = modem_cfg.get('name', 'att-modem')
|
||||
# With the gateway's access code, also read its NAT sessions and ping from it
|
||||
self.modem_admin = (modem_api.ModemAdmin(modem_cfg)
|
||||
if modem_cfg.get('url') and modem_cfg.get('access_code') else None)
|
||||
# Modem reboots (including AT&T's automatic firmware updates) take a few
|
||||
# minutes, so this can be raised independently of failure_threshold.
|
||||
self.modem_fail_thresh = modem_cfg.get('failure_threshold', self.fail_thresh)
|
||||
@@ -1950,6 +1967,140 @@ class NetworkMonitor:
|
||||
'point toward the provider or the modem rather than the fiber.',
|
||||
)
|
||||
|
||||
def _collect_wan_quality(self) -> None:
|
||||
"""Store UniFi's WAN health timeline. If UniFi can't be read, the last copy stays
|
||||
and goes stale, and the dashboard stops drawing conclusions from it."""
|
||||
quality = modem_api.wan_quality(self.unifi.get_wan_dashboard(), time.time())
|
||||
if quality is not None:
|
||||
db.set_state('wan_quality', quality)
|
||||
|
||||
_RESTART_SETTLE_S = 180 # the modem answering this soon may not have restarted yet
|
||||
_RESTART_GIVE_UP_S = 1800
|
||||
|
||||
def _finish_modem_restart(self) -> None:
|
||||
"""After the Restart modem button: once the modem answers again with its WAN up (or
|
||||
after 30 minutes regardless), end the Uptime Kuma window and the Gandalf silence the
|
||||
button started, and note how long it took."""
|
||||
raw = db.get_state('modem_restart')
|
||||
try:
|
||||
state = json.loads(raw) if raw else None
|
||||
except ValueError:
|
||||
state = None
|
||||
if not state or state.get('finished_at'):
|
||||
return
|
||||
now = time.time()
|
||||
age = now - state.get('at', now)
|
||||
raw = db.get_state('modem_stats')
|
||||
try:
|
||||
stats = json.loads(raw) if raw else {}
|
||||
except ValueError:
|
||||
stats = {}
|
||||
uptime = stats.get('uptime_s')
|
||||
rebooted = uptime is not None and uptime < age
|
||||
back = (age >= self._RESTART_SETTLE_S and stats.get('reachable')
|
||||
and not stats.get('wan_problems') and (rebooted or uptime is None))
|
||||
if not back and age < self._RESTART_GIVE_UP_S:
|
||||
return
|
||||
if state.get('kuma') and not state.get('kuma_ended'):
|
||||
try:
|
||||
kuma.set_maintenance(self.cfg.get('uptime_kuma') or {}, False)
|
||||
state['kuma_ended'] = True
|
||||
except kuma.KumaError as e:
|
||||
logger.warning(f'Ending the Uptime Kuma window after the modem restart: {e}')
|
||||
if age < self._RESTART_GIVE_UP_S:
|
||||
return # try again next cycle
|
||||
state['kuma_error'] = str(e)
|
||||
if state.get('suppression_id'):
|
||||
db.deactivate_suppression(state['suppression_id'])
|
||||
state.update(finished_at=now, back=bool(back), rebooted=rebooted,
|
||||
back_after_s=round(age) if back else None)
|
||||
db.set_state('modem_restart', state)
|
||||
logger.info(f'Modem restart finished: {"back" if back else "gave up waiting"} after {round(age)} s'
|
||||
f'{", rebooted" if rebooted else ""}')
|
||||
|
||||
_NAT_EVERY_S = 1800 # NAT sessions and the gateway's speed test history
|
||||
_FIREWALL_EVERY_S = 300 # the firewall log only holds about ten minutes
|
||||
_FIREWALL_HOURS = 24
|
||||
_PING_EVERY_S = 600
|
||||
_PING_TARGET = '1.1.1.1'
|
||||
|
||||
def _collect_modem_checks(self) -> None:
|
||||
"""NAT session counts every 30 minutes, and while UniFi says the internet is slow,
|
||||
a ping from the modem itself (at most every 10 minutes) to show where the delay is.
|
||||
Results go to monitor_state.modem_checks for the WAN panel and Needs attention."""
|
||||
admin = getattr(self, 'modem_admin', None)
|
||||
if admin is None:
|
||||
return
|
||||
raw = db.get_state('modem_checks')
|
||||
try:
|
||||
checks = json.loads(raw) if raw else {}
|
||||
except ValueError:
|
||||
checks = {}
|
||||
now = time.time()
|
||||
raw = db.get_state('wan_quality')
|
||||
try:
|
||||
quality = json.loads(raw) if raw else {}
|
||||
except ValueError:
|
||||
quality = {}
|
||||
periods = quality.get('periods') or []
|
||||
latencies = quality.get('latencies') or {}
|
||||
ours = latencies.get('cloudflare', min(latencies.values()) if latencies else None)
|
||||
slow = (bool(periods) and periods[-1].get('state') != 'ok') or (ours or 0) >= modem_api.SLOW_MS
|
||||
|
||||
due_nat = now - (checks.get('nat') or {}).get('at', 0) >= self._NAT_EVERY_S
|
||||
due_fw = now - (checks.get('firewall') or {}).get('checked_at', 0) >= self._FIREWALL_EVERY_S
|
||||
due_ping = slow and now - (checks.get('ping') or {}).get('at', 0) >= self._PING_EVERY_S
|
||||
if not (due_nat or due_fw or due_ping):
|
||||
return
|
||||
if now - (checks.get('error') or {}).get('at', 0) < self._PING_EVERY_S:
|
||||
return # failed recently: don't keep knocking on the gateway
|
||||
try:
|
||||
if due_nat:
|
||||
checks['nat'] = dict(admin.nat_sessions(), at=now)
|
||||
checks['speed'] = self._latest_speed_tests(admin.speed_tests(), now)
|
||||
if due_fw:
|
||||
checks['firewall'] = self._count_firewall_drops(checks.get('firewall'), admin.firewall_log(), now)
|
||||
if due_ping:
|
||||
result = admin.ping(self._PING_TARGET)
|
||||
checks['ping'] = dict(result, at=now, ours_ms=ours,
|
||||
verdict=modem_api.ping_verdict(result, ours))
|
||||
logger.info(f'Modem ping to {self._PING_TARGET}: {result.get("avg_ms")} ms, '
|
||||
f'{result.get("loss_pct")}% loss; ours {ours} ms '
|
||||
f'({checks["ping"]["verdict"]})')
|
||||
checks['error'] = None
|
||||
except (modem_api.ModemUnreachable, modem_api.ModemDataInvalid, modem_api.ModemLoginFailed) as e:
|
||||
logger.warning(f'Modem checks: {e}')
|
||||
checks['error'] = {'message': str(e), 'at': now,
|
||||
'login': isinstance(e, modem_api.ModemLoginFailed)}
|
||||
db.set_state('modem_checks', checks)
|
||||
|
||||
@staticmethod
|
||||
def _latest_speed_tests(tests: list, now: float) -> dict:
|
||||
"""The newest downstream and upstream result from the gateway's own speed tests."""
|
||||
out: dict = {'checked_at': now}
|
||||
for t in tests: # newest first
|
||||
key = 'down' if t['direction'] == 'downstream' else 'up'
|
||||
out.setdefault(key, t)
|
||||
return out
|
||||
|
||||
def _count_firewall_drops(self, previous: Optional[dict], rows: list, now: float) -> dict:
|
||||
"""Add log rows newer than the last one seen to per-hour counts (gateway's local hour).
|
||||
|
||||
The log keeps a few hundred rows and records at most one drop every 2 seconds, so the
|
||||
counts are a lower bound; 'gap' says rows aged out before this read could count them.
|
||||
"""
|
||||
fw = previous or {}
|
||||
hours = dict(fw.get('hours') or {})
|
||||
last = fw.get('last') or ''
|
||||
new = [r for r in rows if r['ts'] > last]
|
||||
for r in new:
|
||||
bucket = hours.setdefault(r['ts'][:13], {'flood': 0, 'invalid': 0, 'other': 0})
|
||||
bucket[modem_api.firewall_kind(r['reason'])] += 1
|
||||
gap = bool(last and rows and rows[0]['ts'] > last)
|
||||
keep = sorted(hours)[-self._FIREWALL_HOURS:]
|
||||
return {'hours': {h: hours[h] for h in keep}, 'last': rows[-1]['ts'] if rows else last,
|
||||
'checked_at': now, 'gap': gap}
|
||||
|
||||
def _collect_modem(self, suppressions: list) -> None:
|
||||
"""Poll the modem, then store its status and raise any alerts."""
|
||||
try:
|
||||
@@ -2135,6 +2286,24 @@ class NetworkMonitor:
|
||||
except Exception as e:
|
||||
logger.error(f'Modem collection failed: {e}', exc_info=True)
|
||||
|
||||
# Internet quality as UniFi measures it (latency, loss, drops): for the dashboard
|
||||
try:
|
||||
self._collect_wan_quality()
|
||||
except Exception as e:
|
||||
logger.error(f'WAN quality collection failed: {e}', exc_info=True)
|
||||
|
||||
# An admin's modem restart: end the silence once the modem is back
|
||||
try:
|
||||
self._finish_modem_restart()
|
||||
except Exception as e:
|
||||
logger.error(f'Finishing the modem restart failed: {e}', exc_info=True)
|
||||
|
||||
# Logged-in modem checks (NAT sessions; a ping from the modem when the internet is slow)
|
||||
try:
|
||||
self._collect_modem_checks()
|
||||
except Exception as e:
|
||||
logger.error(f'Modem checks failed: {e}', exc_info=True)
|
||||
|
||||
# Note on the tickets of anything that cleared this cycle (or failed to post before)
|
||||
try:
|
||||
self._post_recovery_notes()
|
||||
|
||||
@@ -4,3 +4,4 @@ ldap3>=2.9
|
||||
pymysql>=1.1.0
|
||||
requests>=2.31.0
|
||||
urllib3>=2.0.0
|
||||
python-socketio[client]>=5.8
|
||||
+210
-57
@@ -43,6 +43,7 @@ async function refreshAll() {
|
||||
lt.api.get('/api/network'),
|
||||
lt.api.get('/api/status'),
|
||||
refreshModem(),
|
||||
refreshAttention(),
|
||||
]);
|
||||
if (netResult.status === 'fulfilled') {
|
||||
const net = netResult.value;
|
||||
@@ -68,6 +69,40 @@ async function refreshAll() {
|
||||
}
|
||||
}
|
||||
|
||||
// ── Needs attention ───────────────────────────────────────────────────
|
||||
const ATTENTION_GROUPS = [
|
||||
{ key: 'broken', title: 'Broken now', clear: 'Nothing broken' },
|
||||
{ key: 'tidy', title: 'Tidy up', clear: 'Records match the network' },
|
||||
];
|
||||
|
||||
async function refreshAttention() {
|
||||
const body = document.getElementById('attention-body');
|
||||
if (!body) return;
|
||||
try {
|
||||
renderAttention(body, await lt.api.get('/api/attention'));
|
||||
} catch (e) {
|
||||
body.innerHTML = `<p class="attention-empty">Couldn't gather the list: ${lt.escHtml(e.message || e)}</p>`;
|
||||
}
|
||||
}
|
||||
|
||||
function renderAttention(body, data) {
|
||||
const esc = lt.escHtml;
|
||||
body.innerHTML = ATTENTION_GROUPS.map(g => {
|
||||
const rows = data[g.key] || [];
|
||||
const list = rows.length
|
||||
? `<ul class="attention-list">${rows.map(r => `
|
||||
<li class="attention-item attention-${esc(r.severity)}">
|
||||
<a class="attention-link" href="${esc(r.href)}">${esc(r.text)}</a>
|
||||
${r.detail ? `<span class="attention-detail">${esc(r.detail)}</span>` : ''}
|
||||
</li>`).join('')}</ul>`
|
||||
: `<p class="attention-clear">✓ ${g.clear}</p>`;
|
||||
return `<div class="attention-group attention-group-${g.key}">
|
||||
<h3 class="attention-group-title">${g.title}${rows.length ? ` <span class="attention-count">${rows.length}</span>` : ''}</h3>
|
||||
${list}
|
||||
</div>`;
|
||||
}).join('');
|
||||
}
|
||||
|
||||
// Active alerts flagged stale or with a closed ticket, plus tickets left open after clearing
|
||||
function followUpCount(events, staleTickets) {
|
||||
const flagged = events.filter(e => e.severity !== 'info' &&
|
||||
@@ -105,16 +140,6 @@ function updateStatusBar(summary, lastCheck, daemonOk, followUps = 0, hasStaleTi
|
||||
alertBadge.style.display = total ? '' : 'none';
|
||||
}
|
||||
|
||||
// Update stat cards
|
||||
const scCrit = document.getElementById('stat-critical-val');
|
||||
const scWarn = document.getElementById('stat-warning-val');
|
||||
const scRes = document.getElementById('stat-resolved-val');
|
||||
if (scCrit) scCrit.textContent = critCount;
|
||||
if (scWarn) scWarn.textContent = warnCount;
|
||||
if (scRes && summary.resolved_24h !== null && summary.resolved_24h !== undefined) scRes.textContent = summary.resolved_24h;
|
||||
const statCritCard = document.getElementById('stat-critical');
|
||||
if (statCritCard) statCritCard.classList.toggle('lt-stat-card--alert', critCount > 0);
|
||||
|
||||
// Stale data banner: warn if last_check is older than 15 minutes
|
||||
let staleBanner = document.getElementById('stale-banner');
|
||||
if (lastCheck) {
|
||||
@@ -138,9 +163,6 @@ function updateStatusBar(summary, lastCheck, daemonOk, followUps = 0, hasStaleTi
|
||||
}
|
||||
|
||||
function updateHostGrid(hosts) {
|
||||
const scHosts = document.getElementById('stat-hosts-val');
|
||||
if (scHosts) scHosts.textContent = Object.keys(hosts).length;
|
||||
|
||||
for (const [name, host] of Object.entries(hosts)) {
|
||||
const card = document.querySelector(`.host-card[data-host="${CSS.escape(name)}"]`);
|
||||
if (!card) continue;
|
||||
@@ -153,15 +175,17 @@ function updateHostGrid(hosts) {
|
||||
|
||||
const ifaceList = card.querySelector('.iface-list');
|
||||
if (ifaceList && host.interfaces && Object.keys(host.interfaces).length > 0) {
|
||||
ifaceList.innerHTML = Object.entries(host.interfaces)
|
||||
.sort(([a], [b]) => a.localeCompare(b))
|
||||
.map(([iface, state]) => `
|
||||
// NICs marked unused fold into one line, so the card shows what is cabled
|
||||
const all = Object.entries(host.interfaces).sort(([a], [b]) => a.localeCompare(b));
|
||||
const inUse = all.filter(([, state]) => state !== 'unused');
|
||||
const unused = all.length - inUse.length;
|
||||
ifaceList.innerHTML = inUse.map(([iface, state]) => `
|
||||
<div class="iface-row">
|
||||
<span class="iface-dot dot-${state}"></span>
|
||||
<span class="iface-dot dot-${lt.escHtml(state)}"></span>
|
||||
<span class="iface-name">${lt.escHtml(iface)}</span>
|
||||
<span class="iface-state state-${state}">${state}</span>
|
||||
<span class="iface-state state-${lt.escHtml(state)}">${lt.escHtml(state)}</span>
|
||||
</div>
|
||||
`).join('');
|
||||
`).join('') + (unused ? `<div class="iface-unused">+${unused} unused</div>` : '');
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -339,6 +363,58 @@ function _modemLastRestart(history) {
|
||||
return `restarted ${_modemWhen(last.at)}${last.by ? ` by ${last.by}` : ''}`;
|
||||
}
|
||||
|
||||
function _modemUptimeStat(data) {
|
||||
if (data.uptime_s !== null && data.uptime_s !== undefined) {
|
||||
return _modemStat('Uptime', _modemUptime(data.uptime_s), '', _modemLastRestart(data.wan_history));
|
||||
}
|
||||
const restarts = (data.wan_history && data.wan_history.restarts) || [];
|
||||
if (!restarts.length) return '';
|
||||
const last = restarts[restarts.length - 1];
|
||||
return _modemStat('Last restart', _modemWhen(last.at), '', last.by ? `by ${last.by}` : '');
|
||||
}
|
||||
|
||||
const _PING_VERDICTS = {
|
||||
upstream: 'slow past the modem: provider side',
|
||||
gateway: 'modem fine, ours slow: delay in the modem',
|
||||
clear: 'fine',
|
||||
};
|
||||
|
||||
// Logged-in checks: NAT sessions in use, and the latest ping run from the modem itself
|
||||
function _modemChecks(c) {
|
||||
if (!c) return '';
|
||||
const ago = at => lt.time.ago(new Date(at * 1000));
|
||||
let out = '';
|
||||
if (c.nat && c.nat.available) {
|
||||
const pct = Math.round(100 * c.nat.in_use / c.nat.available);
|
||||
out += _modemStat('NAT sessions', `${c.nat.in_use.toLocaleString()} / ${c.nat.available.toLocaleString()}`,
|
||||
pct >= 80 ? 'lt-text-amber' : '', `${pct}% in use, ${ago(c.nat.at)}`);
|
||||
}
|
||||
if (c.ping) {
|
||||
const p = c.ping;
|
||||
const value = p.avg_ms === null || p.avg_ms === undefined ? 'no reply' : `${Math.round(p.avg_ms)} ms`;
|
||||
const verdict = _PING_VERDICTS[p.verdict] || '';
|
||||
out += _modemStat("Modem's own ping", value, p.verdict === 'clear' ? '' : 'lt-text-amber',
|
||||
`to ${p.target}, ${ago(p.at)}${verdict ? `: ${verdict}` : ''}`);
|
||||
}
|
||||
const fw = c.firewall && c.firewall.hours;
|
||||
if (fw && Object.keys(fw).length) {
|
||||
const sum = k => Object.values(fw).reduce((n, h) => n + (h[k] || 0), 0);
|
||||
const flood = sum('flood'), invalid = sum('invalid'), total = flood + invalid + sum('other');
|
||||
out += _modemStat('Firewall drops', `${total.toLocaleString()} in ${Object.keys(fw).length} h`, '',
|
||||
`${flood.toLocaleString()} incoming over the flood limit, ${invalid.toLocaleString()} invalid; ` +
|
||||
'at least (the log keeps 1 per 2 s)');
|
||||
}
|
||||
const sp = c.speed;
|
||||
if (sp && sp.down && sp.up) {
|
||||
out += _modemStat('AT&T speed test', `↓ ${Math.round(sp.down.mbps)} ↑ ${Math.round(sp.up.mbps)} Mbps`, '',
|
||||
`run by the modem ${sp.down.at}, ${Math.round(sp.down.latency_ms)} ms`);
|
||||
}
|
||||
if (c.error && c.error.login) {
|
||||
out += _modemStat('Modem login', 'failed', 'lt-text-red', 'check modem.access_code');
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function _modemDuration(seconds) {
|
||||
if (seconds < 60) return `${Math.round(seconds)}s`;
|
||||
if (seconds < 3600) return `${Math.floor(seconds / 60)}m ${Math.round(seconds % 60)}s`;
|
||||
@@ -361,6 +437,80 @@ function _modemOutages(history) {
|
||||
</div>`;
|
||||
}
|
||||
|
||||
const _QUALITY_LABELS = { ok: 'fine', latency: 'high latency', loss: 'packet loss', down: 'down' };
|
||||
|
||||
// UniFi's own internet health timeline, one block per 5 minutes, worst flag wins
|
||||
function _modemQuality(q) {
|
||||
const periods = (q && q.periods) || [];
|
||||
if (!periods.length) return '';
|
||||
const counts = { ok: 0, latency: 0, loss: 0, down: 0 };
|
||||
periods.forEach(p => { if (p.state in counts) counts[p.state] += 1; });
|
||||
const t0 = periods[0].ts, span = Math.max(periods[periods.length - 1].ts + 300 - t0, 300);
|
||||
const blocks = periods.map(p => {
|
||||
const x = (p.ts - t0) / span * 1000, w = Math.max(300 / span * 1000, 1);
|
||||
const tip = `${_modemWhen(p.ts)}: ${_QUALITY_LABELS[p.state] || p.state}`;
|
||||
return `<rect class="wq-${lt.escHtml(p.state)}" x="${x.toFixed(1)}" y="0" width="${w.toFixed(2)}" height="20">` +
|
||||
`<title>${lt.escHtml(tip)}</title></rect>`;
|
||||
}).join('');
|
||||
const flagged = ['latency', 'loss', 'down'].filter(s => counts[s])
|
||||
.map(s => `${_QUALITY_LABELS[s]} ${counts[s]}`).join(', ');
|
||||
const summary = flagged ? `${flagged} of ${periods.length} periods` : 'no problems';
|
||||
const lat = Object.entries(q.latencies || {}).sort()
|
||||
.map(([k, v]) => `${lt.escHtml(k)} ${lt.escHtml(String(v))} ms`).join(', ');
|
||||
const legend = ['latency', 'loss', 'down']
|
||||
.map(s => `<span><i class="wq-key wq-${s}"></i>${_QUALITY_LABELS[s]}</span>`).join('');
|
||||
return `
|
||||
<div class="modem-chart modem-quality">
|
||||
<div class="modem-chart-title">Internet, last 24 hours (from UniFi) <span>${lt.escHtml(summary)}</span></div>
|
||||
<svg viewBox="0 0 1000 20" preserveAspectRatio="none" role="img"
|
||||
aria-label="Internet health over the last 24 hours: ${lt.escHtml(summary)}">${blocks}</svg>
|
||||
<div class="wq-foot"><span>${lt.escHtml(_modemWhen(t0))}</span>
|
||||
<span class="wq-legend">${legend}</span><span>now</span></div>
|
||||
${lat ? `<div class="modem-stat-note">Latency now: ${lat}</div>` : ''}
|
||||
</div>`;
|
||||
}
|
||||
|
||||
// A restart from the Restart modem button: in progress, or finished in the last hour
|
||||
function _modemRestartNote(r) {
|
||||
if (!r || !r.at) return '';
|
||||
const who = r.by ? ` by ${r.by}` : '';
|
||||
if (!r.finished_at) {
|
||||
return `<div class="modem-restart-note">Restart requested${lt.escHtml(who)} ${lt.escHtml(_modemWhen(r.at))}. ` +
|
||||
`Alerts are silenced${r.kuma ? ' here and in Uptime Kuma' : ''} until the modem is back.</div>`;
|
||||
}
|
||||
if (Date.now() / 1000 - r.finished_at > 3600) return '';
|
||||
const how = r.back ? `back after ${_modemDuration(r.back_after_s)}` : 'did not come back within 30 minutes';
|
||||
return `<div class="modem-restart-note">Restarted${lt.escHtml(who)} ${lt.escHtml(_modemWhen(r.at))}: ` +
|
||||
`${lt.escHtml(how)}. Alerts are on again.</div>`;
|
||||
}
|
||||
|
||||
async function restartModem(btn) {
|
||||
const msg = 'Restart the AT&T modem?\n\nThe internet will be down for about 3 to 5 minutes. ' +
|
||||
'Gandalf alerts and the Uptime Kuma maintenance window are switched on first, and switched off ' +
|
||||
'again when the modem is back.';
|
||||
if (!window.confirm(msg)) return;
|
||||
btn.disabled = true;
|
||||
try {
|
||||
let res;
|
||||
try {
|
||||
res = await lt.api.post('/api/modem/restart', { confirm: true });
|
||||
} catch (err) {
|
||||
if (!String(err.message).startsWith('Uptime Kuma:')) throw err;
|
||||
if (!window.confirm(`${err.message}\n\nRestart anyway? Uptime Kuma will report the outage.`)) {
|
||||
btn.disabled = false;
|
||||
return;
|
||||
}
|
||||
res = await lt.api.post('/api/modem/restart', { confirm: true, without_kuma: true });
|
||||
}
|
||||
showToast(res.result === 'accepted' ? 'Modem restarting' : 'Restart sent; the modem dropped the connection',
|
||||
'success');
|
||||
window.location.reload();
|
||||
} catch (err) {
|
||||
btn.disabled = false;
|
||||
showToast(err.message || 'Could not restart the modem', 'error');
|
||||
}
|
||||
}
|
||||
|
||||
function updateModem(data) {
|
||||
const section = document.getElementById('modem-section');
|
||||
const body = document.getElementById('modem-body');
|
||||
@@ -368,6 +518,18 @@ function updateModem(data) {
|
||||
section.hidden = !data.configured;
|
||||
if (!data.configured) return;
|
||||
|
||||
const restartBtn = document.getElementById('modem-restart-btn');
|
||||
if (restartBtn) {
|
||||
const pending = data.modem_restart && !data.modem_restart.finished_at;
|
||||
restartBtn.hidden = !data.can_restart;
|
||||
restartBtn.disabled = Boolean(pending);
|
||||
if (!restartBtn.dataset.bound) {
|
||||
restartBtn.dataset.bound = '1';
|
||||
restartBtn.addEventListener('click', () => restartModem(restartBtn));
|
||||
}
|
||||
}
|
||||
const restartNote = _modemRestartNote(data.modem_restart);
|
||||
|
||||
// On the dashboard, link to the modem's page. On that page, it would link to itself.
|
||||
const detailsLink = document.getElementById('modem-device-link');
|
||||
if (detailsLink && typeof DEVICE_NAME === 'undefined' && data.name) {
|
||||
@@ -378,9 +540,10 @@ function updateModem(data) {
|
||||
const title = document.getElementById('modem-title');
|
||||
|
||||
if (!data.reachable) {
|
||||
body.innerHTML = `
|
||||
body.innerHTML = `${restartNote}
|
||||
<div class="modem-error">Modem status unavailable: ${lt.escHtml(data.error || 'unknown error')}
|
||||
<span class="modem-stat-note">Last attempt ${lt.escHtml(data.updated || '-')}</span></div>
|
||||
${_modemQuality(data.wan_quality)}
|
||||
${_modemChart(data.history)}`;
|
||||
return;
|
||||
}
|
||||
@@ -398,7 +561,7 @@ function updateModem(data) {
|
||||
|
||||
const errs = (counters.rx_errors || 0) + (counters.tx_errors || 0);
|
||||
const drops = (counters.rx_drops || 0) + (counters.tx_drops || 0);
|
||||
body.innerHTML = `
|
||||
body.innerHTML = `${restartNote}
|
||||
${problems.length ? `<div class="modem-error">WAN down: ${lt.escHtml(problems.join('; '))}</div>` : ''}
|
||||
<div class="modem-stats">
|
||||
${_modemStat('WAN', problems.length ? 'DOWN' : 'UP', problems.length ? 'lt-text-red' : 'lt-text-green',
|
||||
@@ -409,38 +572,28 @@ function updateModem(data) {
|
||||
${_modemOptic('Optics temp', optics.temperature, status.temperature, '°C')}
|
||||
${_modemStat('Throughput', `↓ ${_modemRate(rates.rx_bytes)}`, '', `↑ ${_modemRate(rates.tx_bytes)}`)}
|
||||
${_modemStat('Errors / drops', `${errs} / ${drops}`, errs ? 'lt-text-amber' : '', 'since modem reboot')}
|
||||
${_modemStat('Uptime', _modemUptime(data.uptime_s), '', _modemLastRestart(data.wan_history))}
|
||||
${_modemUptimeStat(data)}
|
||||
${_modemChecks(data.modem_checks)}
|
||||
</div>
|
||||
${_modemQuality(data.wan_quality)}
|
||||
${_modemChart(data.history)}
|
||||
${_modemOutages(data.wan_history)}
|
||||
<div class="modem-stat-note modem-updated">Updated ${lt.escHtml(data.updated || '-')}</div>`;
|
||||
}
|
||||
|
||||
function updateUnifiTable(devices) {
|
||||
const tbody = document.querySelector('#unifi-table tbody');
|
||||
if (!tbody || !devices.length) return;
|
||||
|
||||
tbody.innerHTML = devices.map(d => {
|
||||
const statusClass = d.connected ? '' : 'row-critical';
|
||||
const dotClass = d.connected ? 'dot-up' : 'dot-down';
|
||||
const statusText = d.connected ? 'ONLINE' : 'OFFLINE';
|
||||
const suppressBtn = !d.connected
|
||||
? `<button class="lt-btn lt-btn-ghost lt-btn-sm btn-suppress"
|
||||
data-sup-type="unifi_device"
|
||||
data-sup-name="${lt.escHtml(d.name)}"
|
||||
data-sup-detail=""
|
||||
aria-label="Suppress alerts for ${lt.escHtml(d.name)}">🔕 Suppress</button>`
|
||||
: '';
|
||||
return `
|
||||
<tr class="${statusClass}">
|
||||
<td><span class="${dotClass}"></span> ${statusText}</td>
|
||||
<td><a class="device-link" href="${lt.escHtml(deviceUrl(d.name))}"><strong>${lt.escHtml(d.name)}</strong></a></td>
|
||||
<td>${lt.escHtml(d.type)}</td>
|
||||
<td>${lt.escHtml(d.model)}</td>
|
||||
<td>${lt.escHtml(d.ip)}</td>
|
||||
<td>${suppressBtn}</td>
|
||||
</tr>`;
|
||||
}).join('');
|
||||
const grid = document.getElementById('unifi-grid');
|
||||
if (!grid || !devices.length) return;
|
||||
const esc = lt.escHtml;
|
||||
grid.innerHTML = devices.map(d => `
|
||||
<li class="unifi-chip${d.connected ? '' : ' unifi-chip-down'}">
|
||||
<span class="${d.connected ? 'dot-up' : 'dot-down'}" title="${d.connected ? 'Online' : 'Offline'}"></span>
|
||||
<a class="device-link unifi-chip-name" href="${esc(deviceUrl(d.name))}">${esc(d.name)}</a>
|
||||
<span class="unifi-chip-model">${esc(d.model)}</span>
|
||||
${d.connected ? '' : `<button class="lt-btn lt-btn-ghost lt-btn-sm btn-suppress" data-sup-type="unifi_device"
|
||||
data-sup-name="${esc(d.name)}" data-sup-detail="" title="Suppress alerts for ${esc(d.name)}"
|
||||
aria-label="Suppress alerts for ${esc(d.name)}">🔕</button>`}
|
||||
</li>`).join('');
|
||||
}
|
||||
|
||||
function updateEventsTable(events, totalActive) {
|
||||
@@ -479,16 +632,16 @@ function updateEventsTable(events, totalActive) {
|
||||
: '';
|
||||
return `
|
||||
<tr class="row-${e.severity}${e.is_suppressed ? ' row-suppressed' : ''}">
|
||||
<td><span class="lt-badge badge-${e.severity}">${e.severity}</span>${supBadge}${staleBadge}${closedBadge}</td>
|
||||
<td>${lt.escHtml(e.event_type.replace(/_/g,' '))}</td>
|
||||
<td><strong>${lt.escHtml(e.target_name)}</strong></td>
|
||||
<td>${lt.escHtml(e.target_detail || '-')}</td>
|
||||
<td class="desc-cell" title="${lt.escHtml(e.description || '')}">${lt.escHtml((e.description||'').substring(0,60))}${(e.description||'').length>60?'…':''}</td>
|
||||
<td class="ts-cell" title="${lt.escHtml(e.first_seen||'')}">${lt.time.ago(_toIso(e.first_seen))}</td>
|
||||
<td class="ts-cell" title="${lt.escHtml(e.last_seen||'')}">${lt.time.ago(_toIso(e.last_seen))}</td>
|
||||
<td>${e.consecutive_failures}</td>
|
||||
<td>${ticket}</td>
|
||||
<td>
|
||||
<td data-label="Sev"><span class="lt-badge badge-${e.severity}">${e.severity}</span>${supBadge}${staleBadge}${closedBadge}</td>
|
||||
<td data-label="Type">${lt.escHtml(e.event_type.replace(/_/g,' '))}</td>
|
||||
<td data-label="Target"><strong>${lt.escHtml(e.target_name)}</strong></td>
|
||||
<td data-label="Detail">${lt.escHtml(e.target_detail || '-')}</td>
|
||||
<td data-label="Description" class="desc-cell" title="${lt.escHtml(e.description || '')}">${lt.escHtml((e.description||'').substring(0,60))}${(e.description||'').length>60?'…':''}</td>
|
||||
<td data-label="First seen" class="ts-cell" title="${lt.escHtml(e.first_seen||'')}">${lt.time.ago(_toIso(e.first_seen))}</td>
|
||||
<td data-label="Last seen" class="ts-cell" title="${lt.escHtml(e.last_seen||'')}">${lt.time.ago(_toIso(e.last_seen))}</td>
|
||||
<td data-label="Failures">${e.consecutive_failures}</td>
|
||||
<td data-label="Ticket">${ticket}</td>
|
||||
<td data-label="Actions">
|
||||
<button class="lt-btn lt-btn-ghost lt-btn-sm btn-suppress"
|
||||
data-sup-type="${lt.escHtml(supType)}"
|
||||
data-sup-name="${lt.escHtml(e.target_name)}"
|
||||
@@ -501,7 +654,7 @@ function updateEventsTable(events, totalActive) {
|
||||
wrap.innerHTML = `
|
||||
${countNotice}
|
||||
<div class="lt-table-wrap">
|
||||
<table class="lt-table" id="events-table">
|
||||
<table class="lt-table lt-table-responsive" id="events-table">
|
||||
<caption class="lt-sr-only">Active network alerts</caption>
|
||||
<thead>
|
||||
<tr>
|
||||
|
||||
@@ -488,6 +488,7 @@
|
||||
const date = value instanceof Date ? value : new Date(value);
|
||||
if (isNaN(date)) return '-';
|
||||
const diff = Math.floor((Date.now() - date.getTime()) / 1000);
|
||||
if (diff < 5) return 'just now'; // also a clock a little ahead of this browser's
|
||||
if (diff < 60) return diff + 's ago';
|
||||
if (diff < 3600) return Math.floor(diff / 60) + 'm ago';
|
||||
if (diff < 86400) return Math.floor(diff / 3600) + 'h ago';
|
||||
|
||||
+70
-42
@@ -67,16 +67,19 @@ function renderSummary() {
|
||||
const problems = cables.filter(c => c.problem).length;
|
||||
const matches = cables.filter(c => c.verdict === 'match').length;
|
||||
const labels = _cs.labels.cables.length + _cs.labels.ports.length;
|
||||
const chip = (n, text, cls, target) =>
|
||||
`<a class="cs-chip ${cls}" href="#${target}"><b>${n}</b> ${esc(text)}</a>`;
|
||||
document.getElementById('cs-summary').innerHTML = [
|
||||
chip(cables.length, 'cables', '', 'cs-cables-section'),
|
||||
chip(matches, 'match what Gandalf sees', 'cs-chip-ok', 'cs-cables-section'),
|
||||
chip(problems, problems === 1 ? 'problem' : 'problems', problems ? 'cs-chip-bad' : '', 'cs-cables-section'),
|
||||
chip(_cs.suggestions.length, 'undocumented links', _cs.suggestions.length ? 'cs-chip-warn' : '', 'cs-suggest-section'),
|
||||
chip(portFixes(), 'port names to fix', portFixes() ? 'cs-chip-warn' : '', 'cs-portnames-section'),
|
||||
chip(labels, 'labels to make', labels ? 'cs-chip-warn' : '', 'cs-labels-section'),
|
||||
].join('');
|
||||
document.getElementById('cs-summary').innerHTML =
|
||||
`<b class="cs-ok">${matches}</b> of ${cables.length} match what Gandalf sees`
|
||||
+ (problems ? ` · <b class="cs-bad">${problems}</b> ${problems === 1 ? 'problem' : 'problems'}` : '');
|
||||
// Each tab says how much is in it; To do counts what needs doing
|
||||
const todo = _cs.suggestions.length + portFixes() + labels;
|
||||
const tabCount = (id, n, cls) => {
|
||||
const el = document.getElementById(`cs-tabcount-${id}`);
|
||||
el.textContent = n ? String(n) : '';
|
||||
el.className = `cs-tab-count${cls ? ` ${cls}` : ''}`;
|
||||
};
|
||||
tabCount('cables', cables.length, problems ? 'cs-tab-count-bad' : '');
|
||||
tabCount('todo', todo, todo ? 'cs-tab-count-warn' : '');
|
||||
tabCount('items', _cs.items.length, '');
|
||||
}
|
||||
|
||||
function cableMatches(c, words, status, problemsOnly) {
|
||||
@@ -209,6 +212,9 @@ function farEnd(c, name, port, side) {
|
||||
return endText(c, end);
|
||||
}
|
||||
|
||||
// Panels whose every port is shown, not just the ones with something on them
|
||||
const _panelsShowAll = new Set();
|
||||
|
||||
function renderPanels() {
|
||||
const panels = _cs.items.filter(i => i.kind === 'patch_panel').sort((a, b) => natural(a.name, b.name));
|
||||
const wrap = document.getElementById('cs-panels');
|
||||
@@ -220,10 +226,14 @@ function renderPanels() {
|
||||
for (const l of _cs.port_labels) labels[`${l.item_id}|${l.port}`] = l;
|
||||
wrap.innerHTML = panels.map(p => {
|
||||
const rows = [];
|
||||
const all = _panelsShowAll.has(p.id);
|
||||
let used = 0;
|
||||
for (let n = 1; n <= (p.port_count || 0); n++) {
|
||||
const front = cableAt(p.name, n, 'front');
|
||||
const rear = cableAt(p.name, n, 'rear');
|
||||
const lab = labels[`${p.id}|${n}`];
|
||||
if (front || rear || lab) used += 1;
|
||||
else if (!all) continue;
|
||||
const cell = (c, side) => c
|
||||
? `<a class="cs-id" href="#cable-${esc(c.id)}">${esc(c.label || 'unlabeled')}</a> → ${farEnd(c, p.name, n, side)}`
|
||||
: '<span class="val-neutral">-</span>';
|
||||
@@ -236,17 +246,24 @@ function renderPanels() {
|
||||
<td>${cell(rear, 'rear')}</td>
|
||||
</tr>`);
|
||||
}
|
||||
return `
|
||||
<div class="lt-frame cs-panel" id="item-${esc(p.id)}">
|
||||
<span class="lt-frame-bl">╚</span><span class="lt-frame-br">╝</span>
|
||||
<div class="lt-section-header">${esc(p.name)}${p.rack_u ? ` · U${esc(p.rack_u)}` : ''}${p.location ? ` · ${esc(p.location)}` : ''}</div>
|
||||
const total = p.port_count || 0;
|
||||
const toggle = total && used < total
|
||||
? `<button type="button" class="cs-link-btn no-print" data-action="panel-all" data-id="${esc(p.id)}" aria-expanded="${all}">${all ? `Show only the ${used} in use` : `Show all ${total} ports`}</button>`
|
||||
: '';
|
||||
const table = rows.length ? `
|
||||
<div class="lt-table-wrap">
|
||||
<table class="lt-table lt-table-sm">
|
||||
<caption class="lt-sr-only">Ports on ${esc(p.name)}</caption>
|
||||
<thead><tr><th>Port</th><th>Label</th><th>Front (patch cord)</th><th>Rear (run)</th></tr></thead>
|
||||
<tbody>${rows.join('')}</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>` : '';
|
||||
return `
|
||||
<div class="lt-frame cs-panel" id="item-${esc(p.id)}">
|
||||
<span class="lt-frame-bl">╚</span><span class="lt-frame-br">╝</span>
|
||||
<div class="lt-section-header">${esc(p.name)}${p.rack_u ? ` · U${esc(p.rack_u)}` : ''}${p.location ? ` · ${esc(p.location)}` : ''}</div>
|
||||
<p class="cs-panel-use">${used} of ${total} ports have a cable or label${used ? '' : '; nothing documented yet'}. ${toggle}</p>
|
||||
${table}
|
||||
</div>`;
|
||||
}).join('');
|
||||
}
|
||||
@@ -282,27 +299,6 @@ function renderLabels() {
|
||||
${unlabeled.length ? `<p class="cs-hint">${unlabeled.length} more cable(s) have no ID yet.</p>` : ''}`;
|
||||
}
|
||||
|
||||
function renderRack() {
|
||||
const placed = _cs.items.filter(i => i.rack_u).sort((a, b) => (b.rack_u + (b.rack_height || 1)) - (a.rack_u + (a.rack_height || 1)));
|
||||
const wrap = document.getElementById('cs-rack');
|
||||
if (!placed.length) {
|
||||
wrap.innerHTML = emptyState('🗄️', 'No rack positions recorded', CS_CAN_EDIT ? 'Set a rack unit on a panel, shelf or device in the inventory.' : '');
|
||||
return;
|
||||
}
|
||||
wrap.innerHTML = `
|
||||
<div class="cs-rack">
|
||||
${placed.map(i => {
|
||||
const top = i.rack_u + (i.rack_height || 1) - 1;
|
||||
return `
|
||||
<div class="cs-rack-unit" style="--u:${Math.min(i.rack_height || 1, 8)}">
|
||||
<span class="cs-rack-u">${i.rack_height > 1 ? `U${esc(top)}-${esc(i.rack_u)}` : `U${esc(i.rack_u)}`}</span>
|
||||
<span class="cs-rack-name">${deviceLink(i.name)}</span>
|
||||
<span class="cs-rack-kind">${esc(KIND_TEXT[i.kind] || i.kind)}</span>
|
||||
</div>`;
|
||||
}).join('')}
|
||||
</div>`;
|
||||
}
|
||||
|
||||
function rackText(i) {
|
||||
if (i.shelf) return `on ${esc(i.shelf)}${i.slot ? `, position ${esc(i.slot)}` : ''}`;
|
||||
if (!i.rack_u) return '-';
|
||||
@@ -380,7 +376,6 @@ function render() {
|
||||
renderPortNames();
|
||||
renderPanels();
|
||||
renderLabels();
|
||||
renderRack();
|
||||
renderItems();
|
||||
renderHistory();
|
||||
renderDatalists();
|
||||
@@ -396,13 +391,39 @@ async function load() {
|
||||
return;
|
||||
}
|
||||
render();
|
||||
const hash = window.location.hash;
|
||||
if (hash && /^#(cable|item)-\d+$/.test(hash)) {
|
||||
const el = document.getElementById(hash.slice(1));
|
||||
if (el) { el.classList.add('cs-flash'); el.scrollIntoView({ block: 'center' }); }
|
||||
}
|
||||
reveal(window.location.hash);
|
||||
}
|
||||
|
||||
// Open the tab holding a section, cable or item (links from the dashboard, the rack and
|
||||
// other tabs point at them), then scroll to it. Cables and items flash so they're easy to spot.
|
||||
function reveal(hash) {
|
||||
if (!hash || !/^#[\w-]+$/.test(hash)) return false;
|
||||
const el = document.getElementById(hash.slice(1));
|
||||
if (!el) return false;
|
||||
const panel = el.closest('.lt-tab-panel');
|
||||
if (panel && !panel.classList.contains('active')) lt.tabs.switch(panel.id);
|
||||
if (el.classList.contains('lt-tab-panel')) return true;
|
||||
if (/^#(cable|item)-\d+$/.test(hash)) {
|
||||
el.classList.remove('cs-flash');
|
||||
void el.offsetWidth;
|
||||
el.classList.add('cs-flash');
|
||||
el.scrollIntoView({ block: 'center' });
|
||||
} else {
|
||||
el.scrollIntoView({ block: 'start' });
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
window.addEventListener('hashchange', () => reveal(window.location.hash));
|
||||
document.addEventListener('click', e => {
|
||||
// A link to something on another tab: switch tabs even when the hash is already set
|
||||
const a = e.target.closest('a[href^="#"]');
|
||||
if (a && a.getAttribute('href').length > 1 && reveal(a.getAttribute('href'))) {
|
||||
e.preventDefault();
|
||||
history.replaceState(null, '', a.getAttribute('href'));
|
||||
}
|
||||
});
|
||||
|
||||
// ── Editing ─────────────────────────────────────────────────────────────
|
||||
function fillPorts(end) {
|
||||
const device = document.querySelector(`#cs-cable-form [name="${end}_device"]`).value;
|
||||
@@ -706,6 +727,11 @@ document.addEventListener('click', e => {
|
||||
else if (action === 'edit-item') openItem(_cs.items.find(i => String(i.id) === btn.dataset.id));
|
||||
else if (action === 'edit-port-label') openPortLabel(btn.dataset.item, btn.dataset.port);
|
||||
else if (action === 'print-labels') printLabels();
|
||||
else if (action === 'panel-all') {
|
||||
const id = Number(btn.dataset.id);
|
||||
if (_panelsShowAll.has(id)) _panelsShowAll.delete(id); else _panelsShowAll.add(id);
|
||||
renderPanels();
|
||||
}
|
||||
else if (action === 'trace-start') startTrace();
|
||||
else if (action === 'trace-stop') { stopTrace(); traceStatus('Stopped.'); }
|
||||
else if (action === 'trace-record') {
|
||||
@@ -757,6 +783,8 @@ function openFromQuery() {
|
||||
document.getElementById('cs-filter').addEventListener('input', () => _cs && renderCables());
|
||||
document.getElementById('cs-status-filter').addEventListener('change', () => _cs && renderCables());
|
||||
document.getElementById('cs-problems-only').addEventListener('change', () => _cs && renderCables());
|
||||
// The dashboard's "Needs attention" list links here with ?problems=1
|
||||
if (new URLSearchParams(window.location.search).get('problems')) document.getElementById('cs-problems-only').checked = true;
|
||||
if (CS_CAN_EDIT) {
|
||||
document.getElementById('cs-cable-form').addEventListener('submit', saveCable);
|
||||
document.getElementById('cs-item-form').addEventListener('submit', saveItem);
|
||||
|
||||
+191
-17
@@ -177,13 +177,18 @@
|
||||
text-transform: uppercase;
|
||||
letter-spacing: .08em;
|
||||
}
|
||||
/* A count. Neutral by default; red only where it counts problems (active alerts, follow-ups). */
|
||||
.g-section-badge {
|
||||
font-size: .72em;
|
||||
font-weight: bold;
|
||||
color: var(--accent-red);
|
||||
border: 1px solid var(--accent-red);
|
||||
color: var(--text-muted);
|
||||
border: 1px solid var(--border-color);
|
||||
padding: 1px 7px;
|
||||
}
|
||||
.g-section-badge-alert {
|
||||
color: var(--accent-red);
|
||||
border-color: var(--accent-red);
|
||||
}
|
||||
.g-section-badge-resolved {
|
||||
font-size: .68em;
|
||||
color: var(--text-muted);
|
||||
@@ -330,7 +335,6 @@
|
||||
letter-spacing: .04em;
|
||||
font-weight: bold;
|
||||
}
|
||||
.source-prometheus { color: var(--orange); border-color: var(--orange-dim); }
|
||||
.source-ping { color: var(--cyan); border-color: var(--cyan-dim); }
|
||||
.iface-list { border-top: 1px solid var(--border-color); padding-top: 6px; margin-bottom: 8px; }
|
||||
.iface-row { display: flex; align-items: center; gap: 6px; padding: 2px 0; }
|
||||
@@ -345,8 +349,9 @@
|
||||
/* ── Form layout helpers (used in suppressions.html form grids) ───── */
|
||||
.form-row { display: flex; gap: 12px; flex-wrap: wrap; margin-bottom: 14px; }
|
||||
.form-row-align { align-items: flex-end; }
|
||||
.form-row > .lt-form-group { flex: 1 1 220px; }
|
||||
.form-group-wide { flex: 3; }
|
||||
.form-group-submit { flex: 0 0 auto; min-width: unset; }
|
||||
.form-group-submit, .form-row > .form-group-submit { flex: 0 0 auto; min-width: unset; }
|
||||
.required { color: var(--red); }
|
||||
|
||||
/* ── Duration pills ───────────────────────────────────────────────── */
|
||||
@@ -844,6 +849,18 @@
|
||||
.modem-chart svg { display: block; width: 100%; height: 70px; border: 1px solid var(--border-color); }
|
||||
.modem-chart svg polyline { stroke: var(--terminal-cyan, #0ff); stroke-width: 1.5; }
|
||||
.modem-chart-empty { padding: 0 12px 8px; font-size: .72em; color: var(--text-muted); }
|
||||
.modem-quality svg { height: 20px; }
|
||||
.wq-ok { fill: var(--accent-green-dim, rgba(0, 255, 0, .25)); }
|
||||
.wq-latency { fill: var(--accent-amber); }
|
||||
.wq-loss { fill: var(--accent-red); }
|
||||
.wq-down { fill: var(--text-primary, #fff); }
|
||||
.wq-foot { display: flex; justify-content: space-between; gap: 8px; flex-wrap: wrap;
|
||||
font-size: .68em; color: var(--text-muted); margin-top: 3px; }
|
||||
.wq-legend { display: inline-flex; gap: 10px; flex-wrap: wrap; }
|
||||
.wq-key { display: inline-block; width: 9px; height: 9px; margin-right: 4px; vertical-align: -1px; }
|
||||
.wq-key.wq-latency { background: var(--accent-amber); }
|
||||
.wq-key.wq-loss { background: var(--accent-red); }
|
||||
.wq-key.wq-down { background: var(--text-primary, #fff); }
|
||||
.modem-updated { padding: 0 12px 10px; }
|
||||
|
||||
/* ── Device search (home page) ────────────────────────────────────── */
|
||||
@@ -902,10 +919,6 @@ a.device-link:hover { color: var(--text-accent); border-bottom-color: current
|
||||
.device-table th, .device-table td { padding-left: 6px; padding-right: 6px; }
|
||||
}
|
||||
.device-search-row.is-active { box-shadow: inset 3px 0 0 var(--terminal-cyan, #0ff); }
|
||||
@media (max-width: 700px) {
|
||||
/* Scroll inside the wrapper rather than squeezing every column to a few letters */
|
||||
#events-table { min-width: 760px; }
|
||||
}
|
||||
|
||||
/* ── Port inspector on the device page ────────────────────────────── */
|
||||
.device-inspector { display: grid; grid-template-columns: minmax(0, 1fr) auto; gap: 14px 16px; align-items: start; }
|
||||
@@ -1094,7 +1107,6 @@ polygon.fp-led-off { fill: #6b6c71; }
|
||||
.cs-trace-status { margin-top: 10px; font-size: .88rem; line-height: 1.6; }
|
||||
.cs-trace-found { font-size: 1rem; margin-bottom: 6px; }
|
||||
.cs-actions { display: flex; flex-wrap: wrap; gap: 8px; align-items: center; }
|
||||
.cs-summary { display: flex; flex-wrap: wrap; gap: 8px; margin: 0 0 18px; }
|
||||
.cs-chip { padding: 5px 10px; border: 1px solid var(--border-color, #234); border-radius: 3px; font-size: .82rem;
|
||||
color: inherit; text-decoration: none; }
|
||||
.cs-chip b { font-size: .95rem; }
|
||||
@@ -1102,6 +1114,16 @@ polygon.fp-led-off { fill: #6b6c71; }
|
||||
.cs-chip-bad { border-color: var(--status-critical, #e0312b); }
|
||||
.cs-chip-bad b { color: var(--status-critical, #e0312b); }
|
||||
.cs-chip-warn b { color: var(--accent-amber, #f0a020); }
|
||||
.cs-summary { font-size: .82rem; color: var(--text-secondary); margin: 0 0 10px; }
|
||||
.cs-summary .cs-ok { color: var(--accent-green); }
|
||||
.cs-summary .cs-bad { color: var(--accent-red); }
|
||||
/* Tabs: wrap on phones rather than scroll the page sideways */
|
||||
.cs-tabs { flex-wrap: wrap; }
|
||||
.cs-tab-count { color: var(--text-secondary); font-weight: 400; margin-left: 2px; }
|
||||
.cs-tab-count:empty { display: none; }
|
||||
.cs-tab-count-bad { color: var(--accent-red); }
|
||||
.cs-tab-count-warn { color: var(--accent-amber); }
|
||||
.cs-panel-use { font-size: .8rem; color: var(--text-secondary); margin: 6px 12px 8px; }
|
||||
.cs-portname-group { margin: 0 0 12px; }
|
||||
.cs-portname-group > summary { cursor: pointer; list-style-position: inside; margin-bottom: 8px; }
|
||||
.cs-portname-group code { white-space: nowrap; }
|
||||
@@ -1130,12 +1152,6 @@ polygon.fp-led-off { fill: #6b6c71; }
|
||||
.cs-label-text { font-family: var(--font-mono, monospace); font-size: 1.05rem; font-weight: 700; letter-spacing: .05em; }
|
||||
.cs-label-copies { font-size: .8rem; color: var(--text-muted, #8aa); }
|
||||
.cs-label-where { grid-column: 1 / -1; font-size: .74rem; color: var(--text-muted, #8aa); }
|
||||
.cs-rack { display: flex; flex-direction: column; gap: 3px; max-width: 560px; }
|
||||
.cs-rack-unit { display: grid; grid-template-columns: 70px 1fr auto; gap: 10px; align-items: center;
|
||||
min-height: calc(var(--u, 1) * 26px); padding: 4px 10px; border: 1px solid var(--border-color, #234);
|
||||
border-radius: 2px; background: rgba(255,255,255,.02); }
|
||||
.cs-rack-u { font-family: var(--font-mono, monospace); font-size: .78rem; color: var(--text-muted, #8aa); }
|
||||
.cs-rack-kind { font-size: .74rem; color: var(--text-muted, #8aa); }
|
||||
.cs-end { border: 1px solid var(--border-color, #234); border-radius: 3px; padding: 4px 12px 0; margin: 0 0 12px; }
|
||||
.cs-end legend { font-size: .78rem; padding: 0 6px; color: var(--text-muted, #8aa); text-transform: uppercase; }
|
||||
.cs-side[hidden] { display: none; }
|
||||
@@ -1153,11 +1169,11 @@ polygon.fp-led-off { fill: #6b6c71; }
|
||||
.cs-table td.cs-id { font-size: 1rem; }
|
||||
.cs-row-problem { box-shadow: inset 3px 0 0 var(--status-critical, #e0312b); }
|
||||
.cs-row-problem td:first-child { box-shadow: none; }
|
||||
.cs-rack-unit { grid-template-columns: 56px 1fr; }
|
||||
.cs-rack-kind { display: none; }
|
||||
}
|
||||
@media print {
|
||||
.no-print, .lt-header, .lt-nav-drawer, .lt-nav-drawer-overlay { display: none !important; }
|
||||
/* Printing shows every tab, or only the labels */
|
||||
.lt-tab-panel { display: block !important; }
|
||||
body.cs-print-labels .g-section:not(#cs-labels-section),
|
||||
body.cs-print-labels .lt-page-header, body.cs-print-labels .cs-summary { display: none !important; }
|
||||
.cs-label-card { break-inside: avoid; border-color: #000; color: #000; }
|
||||
@@ -1680,3 +1696,161 @@ body.fp-marks-off .fp-svg .fp-mark { display: none; }
|
||||
.tp-pop select { width: 100%; }
|
||||
.tp-pop-hint { color: var(--text-muted); font-size: .9em; margin: 6px 0 8px; }
|
||||
.tp-pop-actions { display: flex; justify-content: flex-end; gap: 6px; }
|
||||
|
||||
/* ── Needs attention (dashboard) ─────────────────────────────────── */
|
||||
.attention-groups { display: grid; grid-template-columns: 1fr 1fr; gap: 14px; }
|
||||
.attention-group {
|
||||
background: var(--bg-card);
|
||||
border: 1px solid var(--border-color);
|
||||
padding: 10px 14px 12px;
|
||||
min-width: 0;
|
||||
}
|
||||
.attention-group-title {
|
||||
font-size: .78em;
|
||||
font-weight: bold;
|
||||
letter-spacing: .08em;
|
||||
text-transform: uppercase;
|
||||
color: var(--text-secondary);
|
||||
margin: 0 0 8px;
|
||||
}
|
||||
.attention-group-broken .attention-group-title { color: var(--accent-red); }
|
||||
.attention-count {
|
||||
display: inline-block;
|
||||
min-width: 1.6em;
|
||||
padding: 0 5px;
|
||||
margin-left: 4px;
|
||||
border: 1px solid currentColor;
|
||||
text-align: center;
|
||||
font-size: .9em;
|
||||
}
|
||||
.attention-list { list-style: none; margin: 0; padding: 0; }
|
||||
.attention-item {
|
||||
position: relative;
|
||||
padding: 5px 0 5px 16px;
|
||||
border-top: 1px solid var(--border-color);
|
||||
font-size: .86em;
|
||||
}
|
||||
.attention-item:first-child { border-top: 0; }
|
||||
.attention-item::before {
|
||||
content: '';
|
||||
position: absolute;
|
||||
left: 2px;
|
||||
top: .85em;
|
||||
width: 7px;
|
||||
height: 7px;
|
||||
border-radius: 50%;
|
||||
background: var(--text-muted);
|
||||
}
|
||||
.attention-bad::before { background: var(--accent-red); }
|
||||
.attention-warn::before { background: var(--accent-amber); }
|
||||
.attention-info::before { background: var(--accent-cyan); }
|
||||
.attention-link { color: var(--text-primary); text-decoration: none; }
|
||||
.attention-link:hover, .attention-link:focus-visible { color: var(--accent-cyan); text-decoration: underline; }
|
||||
.attention-detail {
|
||||
display: block;
|
||||
color: var(--text-muted);
|
||||
font-size: .9em;
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
.attention-clear { color: var(--accent-green); font-size: .86em; margin: 0; }
|
||||
.attention-empty { color: var(--text-muted); font-size: .86em; margin: 0; }
|
||||
@media (max-width: 768px) {
|
||||
.attention-groups { grid-template-columns: 1fr; }
|
||||
}
|
||||
|
||||
/* ── Brand: what GANDALF stands for ──────────────────────────────── */
|
||||
.g-brand-expansion { letter-spacing: .08em; line-height: 1.35; }
|
||||
.g-brand-expansion b { color: var(--accent-cyan); font-weight: 700; }
|
||||
.g-help-about { font-size: .86em; color: var(--text-secondary); margin: 0 0 12px; line-height: 1.5; }
|
||||
.g-help-about b { color: var(--accent-cyan); }
|
||||
/* A tab holding one section already says what it is: skip the repeated heading */
|
||||
.lt-tab-panel > .g-section:only-child > .g-section-header { display: none; }
|
||||
|
||||
/* ── Dashboard: calmer host cards and device list ────────────────── */
|
||||
.host-suppress { margin-left: auto; padding: 0 6px; opacity: .45; }
|
||||
.host-suppress:hover, .host-suppress:focus-visible { opacity: 1; }
|
||||
.iface-unused { font-size: .74em; color: var(--text-muted); padding: 2px 0 0 14px; }
|
||||
.unifi-grid {
|
||||
list-style: none; margin: 0; padding: 0;
|
||||
display: grid; grid-template-columns: repeat(auto-fill, minmax(220px, 1fr)); gap: 6px;
|
||||
}
|
||||
.unifi-chip {
|
||||
display: flex; align-items: center; gap: 8px; min-width: 0;
|
||||
padding: 6px 10px; border: 1px solid var(--border-color); background: var(--bg-card);
|
||||
font-size: .84em;
|
||||
}
|
||||
.unifi-chip-down { border-color: var(--accent-red); }
|
||||
.unifi-chip-name { color: var(--text-primary); white-space: nowrap; overflow: hidden; text-overflow: ellipsis; }
|
||||
.unifi-chip-model { margin-left: auto; color: var(--text-muted); font-size: .86em; }
|
||||
.unifi-chip .btn-suppress { padding: 0 6px; }
|
||||
.resolved-fold > summary { cursor: pointer; list-style: none; }
|
||||
.resolved-fold > summary::-webkit-details-marker { display: none; }
|
||||
.resolved-fold > summary::after { content: '▸ show'; margin-left: auto; font-size: .72em; color: var(--text-muted); }
|
||||
.resolved-fold[open] > summary::after { content: '▾ hide'; }
|
||||
|
||||
/* ── Navigation: More menu ───────────────────────────────────────── */
|
||||
.g-nav-more { position: relative; }
|
||||
.g-nav-more > summary { list-style: none; cursor: pointer; }
|
||||
.g-nav-more > summary::-webkit-details-marker { display: none; }
|
||||
.g-nav-more-menu {
|
||||
position: absolute; top: calc(100% + 6px); left: 0; z-index: 60;
|
||||
display: flex; flex-direction: column; min-width: 180px;
|
||||
background: var(--bg-secondary); border: 1px solid var(--border-color);
|
||||
box-shadow: 0 8px 24px rgba(0,0,0,.35);
|
||||
}
|
||||
.g-nav-more-link {
|
||||
padding: 9px 14px; font-size: .72rem; letter-spacing: .12em; text-transform: uppercase;
|
||||
color: var(--text-secondary); text-decoration: none;
|
||||
}
|
||||
.g-nav-more-link:hover, .g-nav-more-link:focus-visible { color: var(--accent-cyan); background: rgba(0,212,255,.05); }
|
||||
.g-nav-more-link.active { color: var(--accent-orange); }
|
||||
.g-drawer-group {
|
||||
display: block; margin: 14px 0 4px; padding: 0 16px; font-size: .66rem;
|
||||
letter-spacing: .14em; text-transform: uppercase; color: var(--text-muted);
|
||||
}
|
||||
|
||||
/* ── Page intros: the how-to sits behind a small "?" ─────────────── */
|
||||
.g-howto { display: inline; }
|
||||
.g-howto > summary {
|
||||
display: inline-flex; align-items: center; justify-content: center;
|
||||
width: 1.5em; height: 1.5em; margin-left: 4px; cursor: pointer; list-style: none;
|
||||
border: 1px solid var(--border-color); border-radius: 50%; color: var(--accent-cyan); font-size: .9em;
|
||||
}
|
||||
.g-howto > summary::-webkit-details-marker { display: none; }
|
||||
.g-howto[open] > span { display: block; margin-top: 6px; color: var(--text-secondary); max-width: 70ch; }
|
||||
.tp-summary-toggle { background: none; border: 0; padding: 0; font: inherit; font-weight: 700; cursor: pointer; color: inherit; }
|
||||
.tp-summary-bad .tp-summary-toggle { color: var(--red); }
|
||||
.tp-summary-warn .tp-summary-toggle { color: var(--amber); }
|
||||
.tp-summary-text { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; min-width: 0; flex: 1 1 0; }
|
||||
|
||||
/* ── Phone header: room for the GANDALF name ─────────────────────── */
|
||||
@media (max-width: 767px) {
|
||||
.lt-header-user, .lt-cmd-hint-btn, .lt-header-right [data-action="start-maintenance"] { display: none; }
|
||||
}
|
||||
.g-drawer-action { background: none; border: 0; width: 100%; text-align: left; font: inherit; cursor: pointer; }
|
||||
|
||||
/* ── Footer: the CFG and HELP buttons read like the keyboard hints beside them ── */
|
||||
button.lt-footer-hint {
|
||||
background: none; border: 0; padding: 0; margin: 0;
|
||||
font: inherit; color: inherit; letter-spacing: inherit; text-transform: inherit; cursor: pointer;
|
||||
}
|
||||
button.lt-footer-hint:hover, button.lt-footer-hint:focus-visible { color: var(--accent-cyan, var(--cyan)); }
|
||||
|
||||
/* ── Footer at the bottom of the window on short pages, not halfway up ── */
|
||||
body { display: flex; flex-direction: column; }
|
||||
body > .lt-main { flex: 1 0 auto; width: 100%; }
|
||||
body > .lt-footer { flex-shrink: 0; }
|
||||
|
||||
/* ── Active alerts: room for the filter's hint; full-width cards on phones ── */
|
||||
#events-search { min-width: 300px; }
|
||||
@media (max-width: 767px) {
|
||||
#events-search { min-width: 0; }
|
||||
#events-table, #events-table tbody { display: block; width: 100%; }
|
||||
#events-table td { max-width: none; }
|
||||
#events-table td.desc-cell { white-space: normal; text-align: right; }
|
||||
}
|
||||
|
||||
/* ── WAN / Fiber: restart button and progress note ── */
|
||||
.modem-actions { display: inline-flex; gap: 6px; }
|
||||
.modem-restart-note { margin: 8px 12px 0; padding: 6px 10px; font-size: .8em;
|
||||
border: 1px solid var(--accent-amber); color: var(--text-primary); }
|
||||
+13
-2
@@ -8,6 +8,7 @@ const PER_MM = 1100 / 482.6;
|
||||
const NODE_GAP = 90;
|
||||
const REFRESH_MS = 30000;
|
||||
const FILTER_KEY = 'gandalf.topology.filters';
|
||||
const PROBLEMS_KEY = 'gandalf.topology.problemsOpen';
|
||||
const DEFAULT_FILTERS = { stubs: 'grouped', power: true, traffic: true, mgmt: false, alerts: false };
|
||||
const STUB_MODES = ['grouped', 'all', 'none'];
|
||||
|
||||
@@ -20,6 +21,7 @@ const state = {
|
||||
expanded: new Set(), // switches whose unknown devices are shown one by one
|
||||
selected: null,
|
||||
pinned: null, // the device a touch screen tapped to trace
|
||||
problemsOpen: (() => { try { return localStorage.getItem(PROBLEMS_KEY) === '1'; } catch (_) { return false; } })(),
|
||||
};
|
||||
// Whether the last press was a mouse, pen or finger: touch has no hover, so taps trace instead
|
||||
let lastPointer = 'mouse';
|
||||
@@ -1323,6 +1325,12 @@ function initEvents() {
|
||||
if (e.key === 'Escape') closeOutletEditor();
|
||||
});
|
||||
document.getElementById('tp-summary').addEventListener('click', e => {
|
||||
if (e.target.closest('[data-action="toggle-problems"]')) {
|
||||
state.problemsOpen = !state.problemsOpen;
|
||||
try { localStorage.setItem(PROBLEMS_KEY, state.problemsOpen ? '1' : '0'); } catch (_) { /* private mode */ }
|
||||
renderSummary();
|
||||
return;
|
||||
}
|
||||
const chip = e.target.closest('.tp-problem-chip');
|
||||
if (!chip) return;
|
||||
if (chip.dataset.focusNode) focusOn('node', chip.dataset.focusNode);
|
||||
@@ -1422,8 +1430,11 @@ function renderSummary() {
|
||||
return;
|
||||
}
|
||||
box.className = `tp-summary ${problems.some(p => p.severity === 'bad') ? 'tp-summary-bad' : 'tp-summary-warn'}`;
|
||||
box.innerHTML = `<span class="tp-summary-head">⚠ ${problems.length} to look at</span>
|
||||
${problems.map(p => `<button type="button" class="tp-problem-chip tp-chip-${p.severity}" data-focus-${p.kind}="${esc(p.id)}">${esc(p.text)}</button>`).join('')}`;
|
||||
// One line by default: the dashboard already lists these; open it to jump to each on the map
|
||||
const open = state.problemsOpen;
|
||||
box.innerHTML = `<button type="button" class="tp-summary-head tp-summary-toggle" data-action="toggle-problems" aria-expanded="${open}">⚠ ${problems.length} to look at ${open ? '▾' : '▸'}</button>
|
||||
${open ? '' : `<span class="tp-summary-text">${esc(problems.slice(0, 2).map(p => p.text).join(' · '))}${problems.length > 2 ? ' ...' : ''}</span>`}
|
||||
${!open ? '' : problems.map(p => `<button type="button" class="tp-problem-chip tp-chip-${p.severity}" data-focus-${p.kind}="${esc(p.id)}">${esc(p.text)}</button>`).join('')}`;
|
||||
}
|
||||
|
||||
// Centre the view on a device or cable and show what is wrong with it
|
||||
|
||||
@@ -1,7 +1,13 @@
|
||||
<section class="g-section" id="modem-section" hidden>
|
||||
<div class="g-section-header">
|
||||
<h2 class="g-section-title">WAN / Fiber</h2>
|
||||
<a class="g-section-actions lt-btn lt-btn-ghost lt-btn-sm" id="modem-device-link" hidden>↗ Details</a>
|
||||
<span class="g-section-actions modem-actions">
|
||||
{% if user.groups and 'admin' in user.groups %}
|
||||
<button type="button" class="lt-btn lt-btn-ghost lt-btn-sm" id="modem-restart-btn" hidden
|
||||
title="Silence alerts (Gandalf and Uptime Kuma), then restart the modem">⟳ Restart modem</button>
|
||||
{% endif %}
|
||||
<a class="lt-btn lt-btn-ghost lt-btn-sm" id="modem-device-link" hidden>↗ Details</a>
|
||||
</span>
|
||||
</div>
|
||||
<div class="lt-frame">
|
||||
<span class="lt-frame-bl">╚</span>
|
||||
|
||||
+56
-42
@@ -5,6 +5,7 @@
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0, viewport-fit=cover">
|
||||
<meta name="theme-color" content="#030508">
|
||||
<meta name="robots" content="noindex, nofollow">
|
||||
<meta name="description" content="GANDALF: Global Advanced Network Detection And Link Facilitator">
|
||||
<link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 16 16'%3E%3Crect width='16' height='16' rx='3' fill='%23030508'/%3E%3Cpath d='M8 2l5 3v6l-5 3-5-3V5z' fill='none' stroke='%2300e5ff' stroke-width='1.5'/%3E%3C/svg%3E">
|
||||
<title>{% block title %}Network Monitor{% endblock %} | GANDALF</title>
|
||||
|
||||
@@ -30,29 +31,33 @@
|
||||
<!-- MOBILE NAV DRAWER -->
|
||||
<div id="lt-nav-drawer" class="lt-nav-drawer" aria-hidden="true" role="dialog" aria-modal="true" aria-label="Navigation menu">
|
||||
<div class="lt-nav-drawer-header">
|
||||
<span class="lt-brand-title">GANDALF</span>
|
||||
<span class="lt-brand-title" title="Global Advanced Network Detection And Link Facilitator">GANDALF</span>
|
||||
<button type="button" class="lt-nav-drawer-close" id="lt-nav-drawer-close" aria-label="Close navigation">✕</button>
|
||||
</div>
|
||||
<nav class="lt-nav-drawer-links" aria-label="Mobile navigation">
|
||||
<a href="{{ url_for('index') }}"
|
||||
class="lt-nav-drawer-link{% if request.endpoint == 'index' %} active{% endif %}"
|
||||
{% if request.endpoint == 'index' %}aria-current="page"{% endif %}>Dashboard</a>
|
||||
<a href="{{ url_for('events_page') }}"
|
||||
class="lt-nav-drawer-link{% if request.endpoint == 'events_page' %} active{% endif %}"
|
||||
{% if request.endpoint == 'events_page' %}aria-current="page"{% endif %}>History</a>
|
||||
<a href="{{ url_for('cutsheet_page') }}"
|
||||
class="lt-nav-drawer-link{% if request.endpoint == 'cutsheet_page' %} active{% endif %}"
|
||||
{% if request.endpoint == 'cutsheet_page' %}aria-current="page"{% endif %}>Cutsheet</a>
|
||||
<a href="{{ url_for('rack_page') }}"
|
||||
class="lt-nav-drawer-link{% if request.endpoint == 'rack_page' %} active{% endif %}"
|
||||
{% if request.endpoint == 'rack_page' %}aria-current="page"{% endif %}>Rack</a>
|
||||
<a href="{{ url_for('topology_page') }}"
|
||||
class="lt-nav-drawer-link{% if request.endpoint == 'topology_page' %} active{% endif %}"
|
||||
{% if request.endpoint == 'topology_page' %}aria-current="page"{% endif %}>Topology</a>
|
||||
<a href="{{ url_for('rack_page') }}"
|
||||
class="lt-nav-drawer-link{% if request.endpoint == 'rack_page' %} active{% endif %}"
|
||||
{% if request.endpoint == 'rack_page' %}aria-current="page"{% endif %}>Rack</a>
|
||||
<a href="{{ url_for('cutsheet_page') }}"
|
||||
class="lt-nav-drawer-link{% if request.endpoint == 'cutsheet_page' %} active{% endif %}"
|
||||
{% if request.endpoint == 'cutsheet_page' %}aria-current="page"{% endif %}>Cutsheet</a>
|
||||
<span class="g-drawer-group">More</span>
|
||||
<a href="{{ url_for('events_page') }}"
|
||||
class="lt-nav-drawer-link{% if request.endpoint == 'events_page' %} active{% endif %}"
|
||||
{% if request.endpoint == 'events_page' %}aria-current="page"{% endif %}>Alert history</a>
|
||||
{% if user.groups and 'admin' in user.groups %}
|
||||
<a href="{{ url_for('suppressions_page') }}"
|
||||
class="lt-nav-drawer-link{% if request.endpoint == 'suppressions_page' %} active{% endif %}"
|
||||
{% if request.endpoint == 'suppressions_page' %}aria-current="page"{% endif %}>Suppressions</a>
|
||||
{% if not maintenance %}
|
||||
<button type="button" class="lt-nav-drawer-link g-drawer-action" data-action="start-maintenance">Start maintenance</button>
|
||||
{% endif %}
|
||||
{% endif %}
|
||||
</nav>
|
||||
</div>
|
||||
@@ -81,43 +86,37 @@
|
||||
class="lt-brand-title lt-glitch"
|
||||
data-text="GANDALF"
|
||||
aria-label="GANDALF home">GANDALF</a>
|
||||
<span class="lt-brand-subtitle">Network Monitor // LotusGuild</span>
|
||||
<span class="lt-brand-subtitle g-brand-expansion" title="GANDALF: Global Advanced Network Detection And Link Facilitator"><b>G</b>lobal <b>A</b>dvanced <b>N</b>etwork<br><b>D</b>etection <b>A</b>nd <b>L</b>ink <b>F</b>acilitator</span>
|
||||
</div>
|
||||
|
||||
<!-- Desktop nav -->
|
||||
<!-- Desktop nav: the four pages used daily, the rest under More -->
|
||||
<nav class="lt-nav" aria-label="Main navigation">
|
||||
<a href="{{ url_for('index') }}"
|
||||
class="lt-nav-link{% if request.endpoint == 'index' %} active{% endif %}"
|
||||
{% if request.endpoint == 'index' %}aria-current="page"{% endif %}>
|
||||
Dashboard
|
||||
</a>
|
||||
<a href="{{ url_for('events_page') }}"
|
||||
class="lt-nav-link{% if request.endpoint == 'events_page' %} active{% endif %}"
|
||||
{% if request.endpoint == 'events_page' %}aria-current="page"{% endif %}>
|
||||
History
|
||||
</a>
|
||||
<a href="{{ url_for('cutsheet_page') }}"
|
||||
class="lt-nav-link{% if request.endpoint == 'cutsheet_page' %} active{% endif %}"
|
||||
{% if request.endpoint == 'cutsheet_page' %}aria-current="page"{% endif %}>
|
||||
Cutsheet
|
||||
</a>
|
||||
<a href="{{ url_for('rack_page') }}"
|
||||
class="lt-nav-link{% if request.endpoint == 'rack_page' %} active{% endif %}"
|
||||
{% if request.endpoint == 'rack_page' %}aria-current="page"{% endif %}>
|
||||
Rack
|
||||
</a>
|
||||
{% if request.endpoint == 'index' %}aria-current="page"{% endif %}>Dashboard</a>
|
||||
<a href="{{ url_for('topology_page') }}"
|
||||
class="lt-nav-link{% if request.endpoint == 'topology_page' %} active{% endif %}"
|
||||
{% if request.endpoint == 'topology_page' %}aria-current="page"{% endif %}>
|
||||
Topology
|
||||
</a>
|
||||
{% if user.groups and 'admin' in user.groups %}
|
||||
<a href="{{ url_for('suppressions_page') }}"
|
||||
class="lt-nav-link{% if request.endpoint == 'suppressions_page' %} active{% endif %}"
|
||||
{% if request.endpoint == 'suppressions_page' %}aria-current="page"{% endif %}>
|
||||
Suppressions
|
||||
</a>
|
||||
{% endif %}
|
||||
{% if request.endpoint == 'topology_page' %}aria-current="page"{% endif %}>Topology</a>
|
||||
<a href="{{ url_for('rack_page') }}"
|
||||
class="lt-nav-link{% if request.endpoint == 'rack_page' %} active{% endif %}"
|
||||
{% if request.endpoint == 'rack_page' %}aria-current="page"{% endif %}>Rack</a>
|
||||
<a href="{{ url_for('cutsheet_page') }}"
|
||||
class="lt-nav-link{% if request.endpoint == 'cutsheet_page' %} active{% endif %}"
|
||||
{% if request.endpoint == 'cutsheet_page' %}aria-current="page"{% endif %}>Cutsheet</a>
|
||||
{% set _more = request.endpoint in ('events_page', 'suppressions_page') %}
|
||||
<details class="g-nav-more">
|
||||
<summary class="lt-nav-link{% if _more %} active{% endif %}">More ▾</summary>
|
||||
<div class="g-nav-more-menu">
|
||||
<a href="{{ url_for('events_page') }}"
|
||||
class="g-nav-more-link{% if request.endpoint == 'events_page' %} active{% endif %}"
|
||||
{% if request.endpoint == 'events_page' %}aria-current="page"{% endif %}>Alert history</a>
|
||||
{% if user.groups and 'admin' in user.groups %}
|
||||
<a href="{{ url_for('suppressions_page') }}"
|
||||
class="g-nav-more-link{% if request.endpoint == 'suppressions_page' %} active{% endif %}"
|
||||
{% if request.endpoint == 'suppressions_page' %}aria-current="page"{% endif %}>Suppressions</a>
|
||||
{% endif %}
|
||||
</div>
|
||||
</details>
|
||||
</nav>
|
||||
</div>
|
||||
|
||||
@@ -229,7 +228,7 @@
|
||||
<span class="lt-footer-sep">|</span>
|
||||
<button type="button" class="lt-footer-hint" data-action="show-keyboard-help"><span class="lt-footer-key">[ ? ]</span> HELP</button>
|
||||
</nav>
|
||||
<span>GANDALF | TDS v1.2</span>
|
||||
<span>GANDALF: Global Advanced Network Detection And Link Facilitator | TDS v1.2</span>
|
||||
</footer>
|
||||
|
||||
{% if user.groups and 'admin' in user.groups and not maintenance %}
|
||||
@@ -291,7 +290,7 @@
|
||||
</div>
|
||||
<div class="lt-form-group" id="sup-name-group">
|
||||
<label class="lt-label" for="sup-name">Target Name</label>
|
||||
<input type="text" class="lt-input" id="sup-name" name="target_name" placeholder="e.g. large1">
|
||||
<input type="text" class="lt-input" id="sup-name" name="target_name" placeholder="e.g. compute-storage-02">
|
||||
</div>
|
||||
<div class="lt-form-group" id="sup-detail-group" style="display:none">
|
||||
<label class="lt-label" for="sup-detail">Interface <span class="lt-field-hint">(interface type only)</span></label>
|
||||
@@ -331,6 +330,7 @@
|
||||
<button type="button" class="lt-modal-close" data-modal-close aria-label="Close">✕</button>
|
||||
</div>
|
||||
<div class="lt-modal-body">
|
||||
<p class="g-help-about"><strong>GANDALF</strong>: <b>G</b>lobal <b>A</b>dvanced <b>N</b>etwork <b>D</b>etection <b>A</b>nd <b>L</b>ink <b>F</b>acilitator. It watches the UniFi switches and the hosts' links, works out what is plugged in where, and raises a ticket when something breaks.</p>
|
||||
<table class="lt-table">
|
||||
<thead><tr><th>Shortcut</th><th>Action</th></tr></thead>
|
||||
<tbody>
|
||||
@@ -417,6 +417,20 @@
|
||||
]);
|
||||
}
|
||||
|
||||
// ── More menu: close on a click elsewhere or Escape ───────────────────
|
||||
document.addEventListener('click', function(e) {
|
||||
document.querySelectorAll('.g-nav-more[open]').forEach(function(d) {
|
||||
if (!d.contains(e.target)) d.removeAttribute('open');
|
||||
});
|
||||
});
|
||||
document.addEventListener('keydown', function(e) {
|
||||
if (e.key !== 'Escape') return;
|
||||
document.querySelectorAll('.g-nav-more[open]').forEach(function(d) {
|
||||
d.removeAttribute('open');
|
||||
d.querySelector('summary').focus();
|
||||
});
|
||||
});
|
||||
|
||||
// ── Global footer + key actions ───────────────────────────────────────
|
||||
document.addEventListener('click', function(e) {
|
||||
const btn = e.target.closest('[data-action]');
|
||||
|
||||
+108
-100
@@ -5,7 +5,7 @@
|
||||
<div class="lt-page-header">
|
||||
<div>
|
||||
<h1 class="lt-page-title">Cutsheet</h1>
|
||||
<p class="g-page-sub">Where every cable is meant to go, checked against what Gandalf sees on the network.</p>
|
||||
<p class="g-page-sub">Where every cable is meant to go, checked against the live network.</p>
|
||||
</div>
|
||||
<div class="cs-actions no-print">
|
||||
{% if can_edit %}
|
||||
@@ -17,111 +17,119 @@
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div id="cs-summary" class="cs-summary" aria-live="polite"></div>
|
||||
<div class="lt-tabs cs-tabs no-print" role="tablist" aria-label="Cutsheet sections">
|
||||
<button type="button" class="lt-tab active" role="tab" id="cs-tabbtn-cables" data-tab="cs-tab-cables" aria-controls="cs-tab-cables" aria-selected="true">Cables <span class="cs-tab-count" id="cs-tabcount-cables"></span></button>
|
||||
<button type="button" class="lt-tab" role="tab" id="cs-tabbtn-todo" data-tab="cs-tab-todo" aria-controls="cs-tab-todo" aria-selected="false">To do <span class="cs-tab-count" id="cs-tabcount-todo"></span></button>
|
||||
<button type="button" class="lt-tab" role="tab" id="cs-tabbtn-panels" data-tab="cs-tab-panels" aria-controls="cs-tab-panels" aria-selected="false">Patch panels</button>
|
||||
<button type="button" class="lt-tab" role="tab" id="cs-tabbtn-items" data-tab="cs-tab-items" aria-controls="cs-tab-items" aria-selected="false">Inventory <span class="cs-tab-count" id="cs-tabcount-items"></span></button>
|
||||
<button type="button" class="lt-tab" role="tab" id="cs-tabbtn-history" data-tab="cs-tab-history" aria-controls="cs-tab-history" aria-selected="false">History</button>
|
||||
</div>
|
||||
|
||||
<section class="g-section" id="cs-cables-section">
|
||||
<div class="g-section-header">
|
||||
<h2 class="g-section-title">Cables</h2>
|
||||
<span class="g-section-badge" id="cs-cable-count">0</span>
|
||||
</div>
|
||||
<div class="cs-filters no-print">
|
||||
<input type="search" class="lt-input" id="cs-filter" placeholder="Filter by label, device, port or note" aria-label="Filter cables">
|
||||
<select class="lt-select" id="cs-status-filter" aria-label="Filter by status">
|
||||
<option value="">All statuses</option>
|
||||
<option value="installed">Installed</option>
|
||||
<option value="planned">Planned</option>
|
||||
<option value="dark">Dark</option>
|
||||
<option value="retired">Retired</option>
|
||||
</select>
|
||||
<label class="cs-check"><input type="checkbox" id="cs-problems-only"> Problems only</label>
|
||||
</div>
|
||||
<div id="cs-cables"></div>
|
||||
</section>
|
||||
|
||||
<section class="g-section no-print" id="cs-suggest-section">
|
||||
<div class="g-section-header">
|
||||
<h2 class="g-section-title">Undocumented links</h2>
|
||||
<span class="g-section-badge" id="cs-suggest-count">0</span>
|
||||
</div>
|
||||
<p class="cs-hint">Switch ports with a live link that no cable record mentions. Adding one pre-fills what Gandalf already knows; insert the patch panel hop if the cable runs through one.</p>
|
||||
<div id="cs-suggestions"></div>
|
||||
</section>
|
||||
|
||||
<section class="g-section no-print" id="cs-portnames-section">
|
||||
<div class="g-section-header">
|
||||
<h2 class="g-section-title">Port names in UniFi</h2>
|
||||
<span class="g-section-badge" id="cs-portnames-count">0</span>
|
||||
</div>
|
||||
<p class="cs-hint">Switch port names that no longer fit what is plugged in, as a checklist. Rename them in UniFi: Gandalf only reports them and never changes UniFi.</p>
|
||||
<div id="cs-portnames"></div>
|
||||
</section>
|
||||
|
||||
<section class="g-section" id="cs-panels-section">
|
||||
<div class="g-section-header">
|
||||
<h2 class="g-section-title">Patch panels</h2>
|
||||
</div>
|
||||
{% if can_edit %}
|
||||
<div class="lt-frame cs-trace no-print" id="cs-trace">
|
||||
<span class="lt-frame-bl">╚</span><span class="lt-frame-br">╝</span>
|
||||
<div class="lt-section-header">Trace a cord</div>
|
||||
<div class="lt-section-body">
|
||||
<p class="cs-hint">Find which switch port a panel port's cord goes to: start, then unplug that cord at the panel.
|
||||
Gandalf watches every switch port and shows the one that drops. Plug it back in afterwards. The device on it
|
||||
loses its link for those few seconds.</p>
|
||||
<div class="cs-trace-form">
|
||||
<div class="lt-form-group">
|
||||
<label class="lt-label" for="cs-trace-panel">Panel</label>
|
||||
<select class="lt-select" id="cs-trace-panel"></select>
|
||||
</div>
|
||||
<div class="lt-form-group">
|
||||
<label class="lt-label" for="cs-trace-port">Port</label>
|
||||
<input class="lt-input" id="cs-trace-port" type="number" min="1" max="512" value="1">
|
||||
</div>
|
||||
<div class="lt-form-group cs-trace-buttons">
|
||||
<button type="button" class="lt-btn lt-btn-primary lt-btn-sm" data-action="trace-start">Start</button>
|
||||
<button type="button" class="lt-btn lt-btn-secondary lt-btn-sm" data-action="trace-stop" hidden>Stop</button>
|
||||
</div>
|
||||
</div>
|
||||
<div id="cs-trace-status" class="cs-trace-status" aria-live="polite"></div>
|
||||
<div class="lt-tab-panel active" id="cs-tab-cables" role="tabpanel" aria-labelledby="cs-tabbtn-cables">
|
||||
<section class="g-section" id="cs-cables-section">
|
||||
<div class="g-section-header">
|
||||
<h2 class="g-section-title">Cables</h2>
|
||||
<span class="g-section-badge" id="cs-cable-count">0</span>
|
||||
</div>
|
||||
</div>
|
||||
{% endif %}
|
||||
<div id="cs-panels"></div>
|
||||
</section>
|
||||
<p class="cs-summary" id="cs-summary" aria-live="polite"></p>
|
||||
<div class="cs-filters no-print">
|
||||
<input type="search" class="lt-input" id="cs-filter" placeholder="Filter by label, device, port or note" aria-label="Filter cables">
|
||||
<select class="lt-select" id="cs-status-filter" aria-label="Filter by status">
|
||||
<option value="">All statuses</option>
|
||||
<option value="installed">Installed</option>
|
||||
<option value="planned">Planned</option>
|
||||
<option value="dark">Dark</option>
|
||||
<option value="retired">Retired</option>
|
||||
</select>
|
||||
<label class="cs-check"><input type="checkbox" id="cs-problems-only"> Problems only</label>
|
||||
</div>
|
||||
<div id="cs-cables"></div>
|
||||
</section>
|
||||
</div>
|
||||
|
||||
<section class="g-section" id="cs-labels-section">
|
||||
<div class="g-section-header">
|
||||
<h2 class="g-section-title">Labels to make</h2>
|
||||
<span class="g-section-badge" id="cs-label-count">0</span>
|
||||
<button type="button" class="lt-btn lt-btn-secondary lt-btn-sm no-print" data-action="print-labels">Print list</button>
|
||||
</div>
|
||||
<p class="cs-hint">Type these on the label maker. Cable IDs go on both ends of the cord; panel labels go on the port. Tick each one once it's stuck on.</p>
|
||||
<div id="cs-labels"></div>
|
||||
</section>
|
||||
<div class="lt-tab-panel" id="cs-tab-todo" role="tabpanel" aria-labelledby="cs-tabbtn-todo">
|
||||
<section class="g-section no-print" id="cs-suggest-section">
|
||||
<div class="g-section-header">
|
||||
<h2 class="g-section-title">Undocumented links</h2>
|
||||
<span class="g-section-badge" id="cs-suggest-count">0</span>
|
||||
</div>
|
||||
<p class="cs-hint">Live switch ports no cable record mentions. Add pre-fills what Gandalf knows.</p>
|
||||
<div id="cs-suggestions"></div>
|
||||
</section>
|
||||
|
||||
<section class="g-section" id="cs-rack-section">
|
||||
<div class="g-section-header">
|
||||
<h2 class="g-section-title">Rack</h2>
|
||||
</div>
|
||||
<div id="cs-rack"></div>
|
||||
<p class="g-section-more"><a href="/rack">Open the rack drawing →</a></p>
|
||||
</section>
|
||||
<section class="g-section no-print" id="cs-portnames-section">
|
||||
<div class="g-section-header">
|
||||
<h2 class="g-section-title">Port names in UniFi</h2>
|
||||
<span class="g-section-badge" id="cs-portnames-count">0</span>
|
||||
</div>
|
||||
<p class="cs-hint">Names that no longer fit what is plugged in. Rename them in UniFi; Gandalf never changes UniFi.</p>
|
||||
<div id="cs-portnames"></div>
|
||||
</section>
|
||||
|
||||
<section class="g-section" id="cs-items-section">
|
||||
<div class="g-section-header">
|
||||
<h2 class="g-section-title">Inventory</h2>
|
||||
<span class="g-section-badge" id="cs-item-count">0</span>
|
||||
</div>
|
||||
<p class="cs-hint">Things Gandalf can't discover: patch panels, wall jacks, shelves, and rack positions for devices.</p>
|
||||
<div id="cs-items"></div>
|
||||
</section>
|
||||
<section class="g-section" id="cs-labels-section">
|
||||
<div class="g-section-header">
|
||||
<h2 class="g-section-title">Labels to make</h2>
|
||||
<span class="g-section-badge" id="cs-label-count">0</span>
|
||||
<button type="button" class="lt-btn lt-btn-secondary lt-btn-sm no-print" data-action="print-labels">Print list</button>
|
||||
</div>
|
||||
<p class="cs-hint">Cable IDs go on both ends of the cord, panel labels on the port. Tick each once it's stuck on.</p>
|
||||
<div id="cs-labels"></div>
|
||||
</section>
|
||||
</div>
|
||||
|
||||
<section class="g-section no-print" id="cs-history-section">
|
||||
<div class="g-section-header">
|
||||
<h2 class="g-section-title">History</h2>
|
||||
</div>
|
||||
<div id="cs-history"></div>
|
||||
</section>
|
||||
<div class="lt-tab-panel" id="cs-tab-panels" role="tabpanel" aria-labelledby="cs-tabbtn-panels">
|
||||
<section class="g-section" id="cs-panels-section">
|
||||
<div class="g-section-header">
|
||||
<h2 class="g-section-title">Patch panels</h2>
|
||||
</div>
|
||||
{% if can_edit %}
|
||||
<div class="lt-frame cs-trace no-print" id="cs-trace">
|
||||
<span class="lt-frame-bl">╚</span><span class="lt-frame-br">╝</span>
|
||||
<div class="lt-section-header">Trace a cord</div>
|
||||
<div class="lt-section-body">
|
||||
<p class="cs-hint">Press Start, then unplug the panel port's cord: the switch port that drops is where it goes.
|
||||
The device on it loses its link until you plug it back in.</p>
|
||||
<div class="cs-trace-form">
|
||||
<div class="lt-form-group">
|
||||
<label class="lt-label" for="cs-trace-panel">Panel</label>
|
||||
<select class="lt-select" id="cs-trace-panel"></select>
|
||||
</div>
|
||||
<div class="lt-form-group">
|
||||
<label class="lt-label" for="cs-trace-port">Port</label>
|
||||
<input class="lt-input" id="cs-trace-port" type="number" min="1" max="512" value="1">
|
||||
</div>
|
||||
<div class="lt-form-group cs-trace-buttons">
|
||||
<button type="button" class="lt-btn lt-btn-primary lt-btn-sm" data-action="trace-start">Start</button>
|
||||
<button type="button" class="lt-btn lt-btn-secondary lt-btn-sm" data-action="trace-stop" hidden>Stop</button>
|
||||
</div>
|
||||
</div>
|
||||
<div id="cs-trace-status" class="cs-trace-status" aria-live="polite"></div>
|
||||
</div>
|
||||
</div>
|
||||
{% endif %}
|
||||
<div id="cs-panels"></div>
|
||||
</section>
|
||||
</div>
|
||||
|
||||
<div class="lt-tab-panel" id="cs-tab-items" role="tabpanel" aria-labelledby="cs-tabbtn-items">
|
||||
<section class="g-section" id="cs-items-section">
|
||||
<div class="g-section-header">
|
||||
<h2 class="g-section-title">Inventory</h2>
|
||||
<span class="g-section-badge" id="cs-item-count">0</span>
|
||||
</div>
|
||||
<p class="cs-hint">Things Gandalf can't discover: patch panels, wall jacks, shelves and rack positions. <a href="/rack">See them in the rack drawing →</a></p>
|
||||
<div id="cs-items"></div>
|
||||
</section>
|
||||
</div>
|
||||
|
||||
<div class="lt-tab-panel" id="cs-tab-history" role="tabpanel" aria-labelledby="cs-tabbtn-history">
|
||||
<section class="g-section no-print" id="cs-history-section">
|
||||
<div class="g-section-header">
|
||||
<h2 class="g-section-title">History</h2>
|
||||
</div>
|
||||
<div id="cs-history"></div>
|
||||
</section>
|
||||
</div>
|
||||
|
||||
<datalist id="cs-device-list"></datalist>
|
||||
<datalist id="cs-a-port-list"></datalist>
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
<div class="lt-page-header">
|
||||
<div>
|
||||
<h1 class="lt-page-title">Alert History</h1>
|
||||
<p class="g-page-sub">Every alert Gandalf raised, including the ones that cleared on their own. Resolved alerts are kept 90 days.</p>
|
||||
<p class="g-page-sub">Every alert raised, including ones that cleared on their own; kept 90 days.</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
+46
-131
@@ -30,43 +30,13 @@
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- ── Stats summary cards ──────────────────────────────────────────── -->
|
||||
<div class="lt-stats-grid">
|
||||
<div class="lt-stat-card{% if summary.critical %} lt-stat-card--alert{% endif %}"
|
||||
id="stat-critical" role="button" tabindex="0"
|
||||
data-stat-filter="critical" aria-label="{{ summary.critical or 0 }} critical alerts"
|
||||
aria-controls="events-table-wrap">
|
||||
<span class="lt-stat-icon lt-text-red" aria-hidden="true">●</span>
|
||||
<div class="lt-stat-info">
|
||||
<span class="lt-stat-value lt-text-red" id="stat-critical-val">{{ summary.critical or 0 }}</span>
|
||||
<span class="lt-stat-label">Critical</span>
|
||||
</div>
|
||||
<!-- ── Needs attention: every page's problems in one place ──────────── -->
|
||||
<section class="g-section attention" id="attention" aria-labelledby="attention-title">
|
||||
<div class="g-section-header"><h2 class="g-section-title" id="attention-title">Needs attention</h2></div>
|
||||
<div class="attention-groups" id="attention-body" aria-live="polite">
|
||||
<p class="attention-empty">Checking every page…</p>
|
||||
</div>
|
||||
<div class="lt-stat-card"
|
||||
id="stat-warning" role="button" tabindex="0"
|
||||
data-stat-filter="warning" aria-label="{{ summary.warning or 0 }} warning alerts"
|
||||
aria-controls="events-table-wrap">
|
||||
<span class="lt-stat-icon lt-text-amber" aria-hidden="true">●</span>
|
||||
<div class="lt-stat-info">
|
||||
<span class="lt-stat-value lt-text-amber" id="stat-warning-val">{{ summary.warning or 0 }}</span>
|
||||
<span class="lt-stat-label">Warning</span>
|
||||
</div>
|
||||
</div>
|
||||
<div class="lt-stat-card" id="stat-hosts" aria-label="Monitored hosts">
|
||||
<span class="lt-stat-icon lt-text-cyan" aria-hidden="true">⬡</span>
|
||||
<div class="lt-stat-info">
|
||||
<span class="lt-stat-value lt-text-cyan" id="stat-hosts-val">{{ snapshot.hosts | length }}</span>
|
||||
<span class="lt-stat-label">Hosts</span>
|
||||
</div>
|
||||
</div>
|
||||
<div class="lt-stat-card" id="stat-resolved" aria-label="{{ recent_resolved | length }} alerts resolved in last 24 hours">
|
||||
<span class="lt-stat-icon lt-text-green" aria-hidden="true">✔</span>
|
||||
<div class="lt-stat-info">
|
||||
<span class="lt-stat-value lt-text-green" id="stat-resolved-val">{{ recent_resolved | length }}</span>
|
||||
<span class="lt-stat-label">Resolved 24h</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- ── Device search ──────────────────────────────────────────────── -->
|
||||
<section class="g-section device-search" id="device-search" role="search">
|
||||
@@ -83,7 +53,7 @@
|
||||
<section class="g-section">
|
||||
<div class="g-section-header">
|
||||
<h2 class="g-section-title">Active Alerts</h2>
|
||||
<span class="g-section-badge" id="alert-count-badge"
|
||||
<span class="g-section-badge g-section-badge-alert" id="alert-count-badge"
|
||||
{% if not summary.critical and not summary.warning %}style="display:none"{% endif %}>{{ (summary.critical or 0) + (summary.warning or 0) }}</span>
|
||||
</div>
|
||||
<div class="lt-toolbar">
|
||||
@@ -110,7 +80,7 @@
|
||||
<div class="pagination-notice">Showing {{ events|length }} of {{ total_active }} active alerts. Use the search box to filter, or <a href="/api/events?limit=1000" target="_blank" rel="noopener">export all as JSON</a></div>
|
||||
{% endif %}
|
||||
<div class="lt-table-wrap">
|
||||
<table class="lt-table" id="events-table">
|
||||
<table class="lt-table lt-table-responsive" id="events-table">
|
||||
<caption class="lt-sr-only">Active network alerts</caption>
|
||||
<thead>
|
||||
<tr>
|
||||
@@ -130,30 +100,30 @@
|
||||
{% for e in events %}
|
||||
{% if e.severity != 'info' %}
|
||||
<tr class="row-{{ e.severity }}{% if e.is_suppressed %} row-suppressed{% endif %}">
|
||||
<td>
|
||||
<td data-label="Sev">
|
||||
<span class="lt-badge badge-{{ e.severity }}">{{ e.severity }}</span>
|
||||
{% if e.is_suppressed %}<span class="lt-badge badge-suppressed" title="Alert suppressed">🔕 sup</span>{% endif %}
|
||||
{% if e.stale_days is not none %}<span class="lt-badge badge-stale" title="Active for {{ e.stale_days }} days with no fix yet">⏳ {{ e.stale_days }}d</span>{% endif %}
|
||||
{% if e.ticket_closed %}<span class="lt-badge badge-ticket-closed" title="Ticket #{{ e.ticket_id }} was closed in Tinker, but this alert is still active">ticket closed</span>{% endif %}
|
||||
</td>
|
||||
<td>{{ e.event_type | replace('_', ' ') }}</td>
|
||||
<td><strong>{{ e.target_name }}</strong></td>
|
||||
<td>{{ e.target_detail or '-' }}</td>
|
||||
<td class="desc-cell" title="{{ e.description | e }}">{{ e.description | truncate(60) }}</td>
|
||||
<td class="ts-cell" title="{{ e.first_seen }}">
|
||||
<td data-label="Type">{{ e.event_type | replace('_', ' ') }}</td>
|
||||
<td data-label="Target"><strong>{{ e.target_name }}</strong></td>
|
||||
<td data-label="Detail">{{ e.target_detail or '-' }}</td>
|
||||
<td data-label="Description" class="desc-cell" title="{{ e.description | e }}">{{ e.description | truncate(60) }}</td>
|
||||
<td data-label="First seen" class="ts-cell" title="{{ e.first_seen }}">
|
||||
<span class="event-age" data-ts="{{ e.first_seen }}">{{ e.first_seen }}</span>
|
||||
</td>
|
||||
<td class="ts-cell" title="{{ e.last_seen }}">
|
||||
<td data-label="Last seen" class="ts-cell" title="{{ e.last_seen }}">
|
||||
<span class="event-age" data-ts="{{ e.last_seen }}">{{ e.last_seen }}</span>
|
||||
</td>
|
||||
<td>{{ e.consecutive_failures }}</td>
|
||||
<td>
|
||||
<td data-label="Failures">{{ e.consecutive_failures }}</td>
|
||||
<td data-label="Ticket">
|
||||
{% if e.ticket_id %}
|
||||
<a href="{{ config.ticket_api.web_url }}{{ e.ticket_id }}" target="_blank"
|
||||
class="ticket-link">#{{ e.ticket_id }}</a>
|
||||
{% else %}-{% endif %}
|
||||
</td>
|
||||
<td>
|
||||
<td data-label="Actions">
|
||||
<button class="lt-btn lt-btn-ghost lt-btn-sm btn-suppress"
|
||||
data-sup-type="{{ 'unifi_device' if e.event_type == 'unifi_device_offline' else 'interface' if e.event_type == 'interface_down' else 'host' }}"
|
||||
data-sup-name="{{ e.target_name }}"
|
||||
@@ -187,7 +157,7 @@
|
||||
<section class="g-section" id="followup"{% if not stale_tickets %} hidden{% endif %}>
|
||||
<div class="g-section-header">
|
||||
<h2 class="g-section-title">Needs Follow-up</h2>
|
||||
<span class="g-section-badge" id="followup-badge">{{ stale_tickets | length }}</span>
|
||||
<span class="g-section-badge g-section-badge-alert" id="followup-badge">{{ stale_tickets | length }}</span>
|
||||
</div>
|
||||
<p class="followup-intro">These tickets are still open in Tinker, but their alert cleared over a day ago. Close them if the problem is fixed.</p>
|
||||
<div class="lt-frame">
|
||||
@@ -236,40 +206,32 @@
|
||||
{% if suppressed or has_global_sup %}
|
||||
<span class="badge-suppressed" title="Suppressed">🔕</span>
|
||||
{% endif %}
|
||||
<button class="lt-btn lt-btn-ghost lt-btn-sm btn-suppress host-suppress"
|
||||
data-sup-type="host" data-sup-name="{{ name }}" data-sup-detail=""
|
||||
title="Suppress alerts for {{ name }}" aria-label="Suppress alerts for {{ name }}">🔕</button>
|
||||
</div>
|
||||
<div class="host-meta">
|
||||
<span class="host-ip">{{ host.ip }}</span>
|
||||
<span class="host-source source-{{ host.source }}">{{ host.source }}</span>
|
||||
{% if host.source != 'prometheus' %}<span class="host-source source-{{ host.source }}">{{ host.source }}</span>{% endif %}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{% if host.interfaces %}
|
||||
{% set in_use = host.interfaces | dictsort | rejectattr(1, 'equalto', 'unused') | list %}
|
||||
{% set unused = (host.interfaces | length) - (in_use | length) %}
|
||||
<div class="iface-list">
|
||||
{% for iface, state in host.interfaces.items() | sort %}
|
||||
{% for iface, state in in_use %}
|
||||
<div class="iface-row">
|
||||
<span class="iface-dot dot-{{ state }}"></span>
|
||||
<span class="iface-name">{{ iface }}</span>
|
||||
<span class="iface-state state-{{ state }}">{{ state }}</span>
|
||||
</div>
|
||||
{% endfor %}
|
||||
{% if unused %}<div class="iface-unused">+{{ unused }} unused</div>{% endif %}
|
||||
</div>
|
||||
{% else %}
|
||||
<div class="host-ping-note">{% if host.source == 'ping' %}ping-only / no node_exporter{% else %}no metrics from node_exporter{% endif %}</div>
|
||||
{% endif %}
|
||||
|
||||
<div class="host-actions">
|
||||
<button class="lt-btn lt-btn-ghost lt-btn-sm btn-suppress"
|
||||
data-sup-type="host"
|
||||
data-sup-name="{{ name }}"
|
||||
data-sup-detail=""
|
||||
aria-label="Suppress alerts for {{ name }}">
|
||||
🔕 Suppress
|
||||
</button>
|
||||
<a href="{{ url_for('device_page', name=name) }}"
|
||||
class="lt-btn lt-btn-secondary lt-btn-sm">
|
||||
↗ Details
|
||||
</a>
|
||||
</div>
|
||||
</div>
|
||||
{% else %}
|
||||
<div class="lt-empty-state lt-empty-state--sm">
|
||||
@@ -287,61 +249,30 @@
|
||||
<div class="g-section-header">
|
||||
<h2 class="g-section-title">UniFi Devices</h2>
|
||||
</div>
|
||||
<div class="lt-frame">
|
||||
<span class="lt-frame-bl">╚</span>
|
||||
<span class="lt-frame-br">╝</span>
|
||||
<div class="lt-section-header">Device Inventory</div>
|
||||
<div class="lt-table-wrap">
|
||||
<table class="lt-table" id="unifi-table">
|
||||
<caption class="lt-sr-only">UniFi network devices</caption>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Status</th>
|
||||
<th>Name</th>
|
||||
<th>Type</th>
|
||||
<th>Model</th>
|
||||
<th>IP</th>
|
||||
<th>Actions</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for d in snapshot.unifi %}
|
||||
<tr class="{% if not d.connected %}row-critical{% endif %}">
|
||||
<td>
|
||||
<span class="dot-{{ 'up' if d.connected else 'down' }}"></span>
|
||||
{{ 'ONLINE' if d.connected else 'OFFLINE' }}
|
||||
</td>
|
||||
<td><a class="device-link" href="{{ url_for('device_page', name=d.name) }}"><strong>{{ d.name }}</strong></a></td>
|
||||
<td>{{ d.type }}</td>
|
||||
<td>{{ d.model }}</td>
|
||||
<td>{{ d.ip }}</td>
|
||||
<td>
|
||||
{% if not d.connected %}
|
||||
<button class="lt-btn lt-btn-ghost lt-btn-sm btn-suppress"
|
||||
data-sup-type="unifi_device"
|
||||
data-sup-name="{{ d.name }}"
|
||||
data-sup-detail=""
|
||||
aria-label="Suppress alerts for {{ d.name }}">
|
||||
🔕 Suppress
|
||||
</button>
|
||||
{% endif %}
|
||||
</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
<ul class="unifi-grid" id="unifi-grid">
|
||||
{% for d in snapshot.unifi %}
|
||||
<li class="unifi-chip{% if not d.connected %} unifi-chip-down{% endif %}">
|
||||
<span class="dot-{{ 'up' if d.connected else 'down' }}" title="{{ 'Online' if d.connected else 'Offline' }}"></span>
|
||||
<a class="device-link unifi-chip-name" href="{{ url_for('device_page', name=d.name) }}">{{ d.name }}</a>
|
||||
<span class="unifi-chip-model">{{ d.model }}</span>
|
||||
{% if not d.connected %}
|
||||
<button class="lt-btn lt-btn-ghost lt-btn-sm btn-suppress" data-sup-type="unifi_device"
|
||||
data-sup-name="{{ d.name }}" data-sup-detail="" title="Suppress alerts for {{ d.name }}"
|
||||
aria-label="Suppress alerts for {{ d.name }}">🔕</button>
|
||||
{% endif %}
|
||||
</li>
|
||||
{% endfor %}
|
||||
</ul>
|
||||
</section>
|
||||
{% endif %}
|
||||
|
||||
<!-- ── Recently Resolved (last 24h) ───────────────────────────────── -->
|
||||
{% if recent_resolved %}
|
||||
<section class="g-section">
|
||||
<div class="g-section-header">
|
||||
<details class="g-section resolved-fold">
|
||||
<summary class="g-section-header">
|
||||
<h2 class="g-section-title">Recently Resolved</h2>
|
||||
<span class="g-section-badge g-section-badge-resolved">{{ recent_resolved | length }} in last 24h</span>
|
||||
</div>
|
||||
</summary>
|
||||
<div class="lt-timeline">
|
||||
{% for e in recent_resolved %}
|
||||
{%- set dot_cls = 'lt-timeline-item--green' if e.severity == 'info' else 'lt-timeline-item--dim' -%}
|
||||
@@ -361,7 +292,7 @@
|
||||
</div>
|
||||
{% endfor %}
|
||||
</div>
|
||||
</section>
|
||||
</details>
|
||||
{% endif %}
|
||||
|
||||
{% endblock %}
|
||||
@@ -370,6 +301,7 @@
|
||||
<script>
|
||||
// Load once even when auto-refresh is set to Off
|
||||
refreshModem();
|
||||
refreshAttention();
|
||||
|
||||
// Start auto-refresh using saved settings interval (default 30 s)
|
||||
const _savedInterval = window.gandalfSettings?.refreshInterval ?? 30;
|
||||
@@ -440,22 +372,5 @@
|
||||
.observe(document.getElementById('events-table-wrap'), { childList: true, subtree: true });
|
||||
|
||||
initDeviceSearch();
|
||||
|
||||
// Stat card clicks: filter events table by severity
|
||||
document.querySelectorAll('.lt-stat-card[data-stat-filter]').forEach(card => {
|
||||
card.addEventListener('click', () => {
|
||||
const sev = card.dataset.statFilter;
|
||||
document.querySelectorAll('.sev-pills .pill').forEach(p => {
|
||||
p.classList.remove('active');
|
||||
p.setAttribute('aria-pressed', 'false');
|
||||
});
|
||||
const matchPill = document.querySelector(`.sev-pills .pill[data-sev="${sev}"]`);
|
||||
if (matchPill) { matchPill.classList.add('active'); matchPill.setAttribute('aria-pressed', 'true'); }
|
||||
_filterSev = sev;
|
||||
applyEventsFilter();
|
||||
document.getElementById('events-table-wrap')?.scrollIntoView({ behavior: 'smooth', block: 'start' });
|
||||
});
|
||||
card.addEventListener('keydown', e => { if (e.key === 'Enter' || e.key === ' ') { e.preventDefault(); card.click(); } });
|
||||
});
|
||||
</script>
|
||||
{% endblock %}
|
||||
+2
-1
@@ -5,7 +5,8 @@
|
||||
<div class="lt-page-header">
|
||||
<div>
|
||||
<h1 class="lt-page-title">Rack</h1>
|
||||
<p class="g-page-sub">The rack front, drawn from the rack units on the cutsheet with live port state. Hover a port to light up every port on the same cable run; click it to open the port or the cable.</p>
|
||||
<div class="g-page-sub">The rack, drawn from the cutsheet with live port state.
|
||||
<details class="g-howto"><summary aria-label="How to use this page">?</summary><span>Hover a port to light up every port on the same cable run; click it to open the port or the cable.</span></details></div>
|
||||
</div>
|
||||
<div class="cs-actions no-print">
|
||||
<button type="button" class="lt-btn lt-btn-secondary lt-btn-sm" data-export="svg">Download SVG</button>
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
<div class="lt-page-header">
|
||||
<div>
|
||||
<h1 class="lt-page-title">Alert Suppressions</h1>
|
||||
<p class="g-page-sub">Manage maintenance windows and per-target alert suppression rules.</p>
|
||||
<p class="g-page-sub">Maintenance windows and alert suppression rules.</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -50,7 +50,7 @@
|
||||
<div class="lt-form-group form-group-wide">
|
||||
<label class="lt-label" for="s-reason">Reason <span class="required">*</span></label>
|
||||
<input type="text" class="lt-input" id="s-reason" name="reason"
|
||||
placeholder="e.g. Planned switch maintenance, replacing SFP on large1/enp43s0"
|
||||
placeholder="e.g. Planned switch maintenance, replacing SFP on compute-storage-02/enp43s0"
|
||||
required aria-required="true">
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -5,7 +5,8 @@
|
||||
<div class="lt-page-header">
|
||||
<div>
|
||||
<h1 class="lt-page-title">Topology</h1>
|
||||
<p class="g-page-sub">Every device and cable as the switches see them right now. Drag to pan, scroll or pinch to zoom. Hover a device to trace its path to the internet and its power (on a touch screen, tap it once to trace and again to open it); click a cable for its details, or a group of unknown devices to open it.</p>
|
||||
<div class="g-page-sub">Every device and cable, as the switches see them now.
|
||||
<details class="g-howto"><summary aria-label="How to use this page">?</summary><span>Drag to pan; scroll or pinch to zoom. Hover a device to trace its path to the internet and its power (on a touch screen, tap once to trace and again to open it). Click a cable for its details, or a group of unknown devices to open it.</span></details></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
+148
-3
@@ -4,7 +4,7 @@ import time
|
||||
from datetime import datetime, timedelta, timezone
|
||||
import os
|
||||
import sys
|
||||
from unittest.mock import patch
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import pytest
|
||||
|
||||
@@ -95,10 +95,12 @@ class TestPagesAndNav:
|
||||
def test_home_section_order(self, client):
|
||||
html = client.get('/', headers=AUTH).get_data(as_text=True)
|
||||
order = [html.index(m) for m in (
|
||||
'id="device-search"', 'Alert Queue', 'WAN / Fiber', 'g-section-title">Hosts<',
|
||||
'Device Inventory',
|
||||
'id="attention"', 'id="device-search"', 'Alert Queue', 'WAN / Fiber', 'g-section-title">Hosts<',
|
||||
'id="unifi-grid"',
|
||||
)]
|
||||
assert order == sorted(order)
|
||||
# The needs-attention list replaced the stat cards
|
||||
assert 'lt-stat-card' not in html
|
||||
|
||||
def test_dashboard_topology_moved_to_its_own_page(self, client):
|
||||
html = client.get('/', headers=AUTH).get_data(as_text=True)
|
||||
@@ -792,3 +794,146 @@ class TestPowerFeedApi:
|
||||
resp = client.put('/api/topology/power/RPDU/7', data='target=large1', headers=AUTH,
|
||||
content_type='application/x-www-form-urlencoded')
|
||||
assert resp.status_code == 400
|
||||
|
||||
|
||||
class TestAttention:
|
||||
def test_needs_auth(self, client):
|
||||
assert client.get('/api/attention').status_code == 401
|
||||
|
||||
def test_gathers_every_page(self, client):
|
||||
gandalf._cutsheet_schema_ready = True
|
||||
ev = {'id': 1, 'event_type': 'host_down', 'severity': 'critical', 'target_name': 'large1',
|
||||
'target_detail': '', 'description': ''}
|
||||
with patch.object(gandalf.db, 'cutsheet_load', return_value={'items': [], 'cables': [], 'port_labels': []}), \
|
||||
patch.object(gandalf.db, 'get_active_events', return_value=[ev]), \
|
||||
patch.object(gandalf.db, 'get_power_feeds', return_value=[]), \
|
||||
patch.object(gandalf.db, 'cutsheet_racks', return_value=[]), \
|
||||
patch.object(gandalf.db, 'check_suppressed', return_value=False):
|
||||
data = client.get('/api/attention', headers=AUTH).get_json()
|
||||
assert set(data) == {'broken', 'tidy'}
|
||||
assert data['broken'][0]['text'] == '1 active alert (1 critical)'
|
||||
|
||||
def test_survives_a_cutsheet_failure(self, client):
|
||||
gandalf._cutsheet_schema_ready = True
|
||||
with patch.object(gandalf.db, 'cutsheet_load', side_effect=RuntimeError('db down')):
|
||||
resp = client.get('/api/attention', headers=AUTH)
|
||||
assert resp.status_code == 200
|
||||
assert set(resp.get_json()) == {'broken', 'tidy'}
|
||||
|
||||
def test_internet_quality_from_unifi(self, client):
|
||||
now = time.time()
|
||||
quality = {'periods': [{'ts': now - (12 - i) * 300, 'state': 'loss'} for i in range(12)],
|
||||
'downtime': [], 'latencies': {}, 'updated': now}
|
||||
state = {'wan_quality': json.dumps(quality), 'modem_stats': json.dumps({'reachable': True})}
|
||||
with patch.object(gandalf.db, 'get_state', side_effect=lambda k, d=None: state.get(k, d)), \
|
||||
patch.object(gandalf.db, 'get_modem_history', return_value=[]), \
|
||||
patch.object(gandalf.db, 'cutsheet_load', side_effect=RuntimeError('db down')):
|
||||
data = client.get('/api/attention', headers=AUTH).get_json()
|
||||
modem = client.get('/api/modem', headers=AUTH).get_json()
|
||||
assert data['broken'][0]['text'] == 'Internet: packet loss in 12 of the last 12 five-minute periods'
|
||||
assert len(modem['wan_quality']['periods']) == 12
|
||||
|
||||
def test_dashboard_has_the_list_and_says_what_gandalf_stands_for(self, client):
|
||||
html = client.get('/', headers=AUTH).get_data(as_text=True)
|
||||
assert 'id="attention-body"' in html
|
||||
assert 'Global Advanced Network Detection And Link Facilitator' in html
|
||||
|
||||
|
||||
class TestCalmDashboard:
|
||||
def test_unused_nics_fold_into_one_line(self, client):
|
||||
snap = json.dumps({'hosts': {'large1': {'ip': '10.0.0.2', 'status': 'up', 'source': 'prometheus',
|
||||
'interfaces': {'eth0': 'up', 'eth1': 'unused', 'eth2': 'unused'}}},
|
||||
'unifi': []})
|
||||
state = dict(STATE, network_snapshot=snap)
|
||||
with patch.object(gandalf.db, 'get_state', side_effect=lambda k, d=None: state.get(k, d)):
|
||||
html = client.get('/', headers=AUTH).get_data(as_text=True)
|
||||
assert '>eth0<' in html and '>eth1<' not in html
|
||||
assert '+2 unused' in html
|
||||
|
||||
def test_source_tag_only_for_hosts_without_node_exporter(self, client):
|
||||
snap = json.dumps({'hosts': {
|
||||
'large1': {'ip': '10.0.0.2', 'status': 'up', 'source': 'prometheus', 'interfaces': {'eth0': 'up'}},
|
||||
'pbs': {'ip': '10.0.0.3', 'status': 'up', 'source': 'ping', 'interfaces': {}}}, 'unifi': []})
|
||||
state = dict(STATE, network_snapshot=snap)
|
||||
with patch.object(gandalf.db, 'get_state', side_effect=lambda k, d=None: state.get(k, d)):
|
||||
html = client.get('/', headers=AUTH).get_data(as_text=True)
|
||||
assert 'source-prometheus' not in html
|
||||
assert '<span class="host-source source-ping">ping</span>' in html
|
||||
|
||||
def test_alert_table_turns_into_cards_on_phones(self, client):
|
||||
ev = {'id': 1, 'event_type': 'host_down', 'severity': 'critical', 'target_name': 'pbs',
|
||||
'target_detail': '', 'description': 'pbs is down', 'consecutive_failures': 3,
|
||||
'first_seen': '2026-11-15 01:00:00', 'last_seen': '2026-11-15 02:00:00'}
|
||||
with patch.object(gandalf.db, 'get_active_events', return_value=[ev]):
|
||||
html = client.get('/', headers=AUTH).get_data(as_text=True)
|
||||
assert 'class="lt-table lt-table-responsive" id="events-table"' in html
|
||||
assert 'data-label="Target"><strong>pbs</strong>' in html
|
||||
|
||||
def test_resolved_alerts_are_folded(self, client):
|
||||
ev = {'target_name': 'large1', 'target_detail': '', 'severity': 'warning', 'event_type': 'host_down',
|
||||
'first_seen': '2026-11-15 01:00:00 UTC', 'resolved_at': '2026-11-15 02:00:00 UTC'}
|
||||
with patch.object(gandalf.db, 'get_recent_resolved', return_value=[ev]):
|
||||
html = client.get('/', headers=AUTH).get_data(as_text=True)
|
||||
assert '<details class="g-section resolved-fold">' in html
|
||||
|
||||
|
||||
class TestModemRestart:
|
||||
CFG = dict(CONFIG, modem={'url': 'http://192.0.2.254', 'access_code': 'secret'},
|
||||
uptime_kuma={'url': 'http://kuma.test', 'username': 'u', 'password': 'p', 'maintenance_id': 7})
|
||||
|
||||
def _post(self, client, body=None, headers=AUTH, cfg=None, kuma_error=None, restart=None):
|
||||
admin = MagicMock()
|
||||
if isinstance(restart, Exception):
|
||||
admin.restart.side_effect = restart
|
||||
else:
|
||||
admin.restart.return_value = restart or 'accepted'
|
||||
with patch.object(gandalf, '_cfg', cfg or self.CFG), \
|
||||
patch.object(gandalf.kuma, 'set_maintenance', side_effect=kuma_error) as km, \
|
||||
patch.object(gandalf.modem_api, 'ModemAdmin', return_value=admin):
|
||||
gandalf.db.create_suppression.return_value = 42
|
||||
resp = client.post('/api/modem/restart', json=body if body is not None else {'confirm': True},
|
||||
headers=headers)
|
||||
return resp, km, admin
|
||||
|
||||
def test_admin_only(self, client):
|
||||
resp, _, admin = self._post(client, headers=USER)
|
||||
assert resp.status_code == 403 and not admin.restart.called
|
||||
|
||||
def test_needs_confirm(self, client):
|
||||
resp, km, admin = self._post(client, body={})
|
||||
assert resp.status_code == 400 and not km.called and not admin.restart.called
|
||||
|
||||
def test_needs_the_access_code(self, client):
|
||||
resp, _, admin = self._post(client, cfg=dict(CONFIG, modem={'url': 'http://192.0.2.254'}))
|
||||
assert resp.status_code == 400 and 'access_code' in resp.get_json()['error']
|
||||
|
||||
def test_silences_then_restarts(self, client):
|
||||
resp, km, admin = self._post(client)
|
||||
assert resp.status_code == 200
|
||||
assert resp.get_json() == {'ok': True, 'result': 'accepted', 'kuma': True, 'suppression_id': 42}
|
||||
km.assert_called_once_with(self.CFG['uptime_kuma'], True)
|
||||
args = gandalf.db.create_suppression.call_args.args
|
||||
assert args[:3] == ('all', '', '') and args[4] == 'jared' and args[5] == gandalf.MODEM_RESTART_SILENCE_MIN
|
||||
admin.restart.assert_called_once_with()
|
||||
key, state = gandalf.db.set_state.call_args.args
|
||||
assert key == 'modem_restart' and state['suppression_id'] == 42 and state['kuma'] is True
|
||||
|
||||
def test_kuma_failure_stops_unless_accepted(self, client):
|
||||
resp, _, admin = self._post(client, kuma_error=gandalf.kuma.KumaError('login refused'))
|
||||
assert resp.status_code == 502 and resp.get_json()['kuma_failed'] and not admin.restart.called
|
||||
resp, _, admin = self._post(client, body={'confirm': True, 'without_kuma': True},
|
||||
kuma_error=gandalf.kuma.KumaError('login refused'))
|
||||
assert resp.status_code == 200 and resp.get_json()['kuma'] is False and admin.restart.called
|
||||
|
||||
def test_failed_restart_undoes_the_silence(self, client):
|
||||
resp, km, _ = self._post(client, restart=gandalf.modem_api.ModemUnreachable('timeout'))
|
||||
assert resp.status_code == 502
|
||||
gandalf.db.deactivate_suppression.assert_called_once_with(42)
|
||||
assert km.call_args_list[-1].args == (self.CFG['uptime_kuma'], False)
|
||||
|
||||
def test_one_at_a_time(self, client):
|
||||
pending = json.dumps({'at': time.time() - 60, 'finished_at': None})
|
||||
state = {'modem_restart': pending}
|
||||
with patch.object(gandalf.db, 'get_state', side_effect=lambda k, d=None: state.get(k, d)):
|
||||
resp, _, admin = self._post(client)
|
||||
assert resp.status_code == 409 and not admin.restart.called
|
||||
@@ -0,0 +1,183 @@
|
||||
"""The dashboard's "Needs attention" list (attention.py)."""
|
||||
import attention
|
||||
|
||||
|
||||
def _alert(target, severity='critical', **kw):
|
||||
return dict({'target_name': target, 'severity': severity, 'is_suppressed': False}, **kw)
|
||||
|
||||
|
||||
def _outlet(index, on=True, target=None, power_w=10.0, match=None):
|
||||
return {'index': index, 'on': on, 'target': target, 'power_w': power_w,
|
||||
'match': match or ('name' if target else 'none')}
|
||||
|
||||
|
||||
class TestAlerts:
|
||||
def test_one_line_for_all_alerts_with_the_targets(self):
|
||||
out = attention.build([_alert('pbs'), _alert('24RJ45-POE', 'warning')], [], None)
|
||||
[line] = out['broken']
|
||||
assert line['text'] == '2 active alerts (1 critical)'
|
||||
assert line['severity'] == 'bad'
|
||||
assert line['detail'] == '24RJ45-POE, pbs'
|
||||
assert line['href'] == '#events-table-wrap'
|
||||
|
||||
def test_suppressed_and_info_alerts_are_left_out(self):
|
||||
events = [_alert('pbs', is_suppressed=True), _alert('large1', 'info')]
|
||||
assert attention.build(events, [], None)['broken'] == []
|
||||
|
||||
def test_warnings_only_is_amber(self):
|
||||
[line] = attention.build([_alert('x', 'warning')], [], None)['broken']
|
||||
assert line['severity'] == 'warn' and line['text'] == '1 active alert'
|
||||
|
||||
|
||||
class TestPower:
|
||||
def test_frozen_readings_and_an_off_outlet_feeding_a_device(self):
|
||||
power = [{'node': 'RPDU', 'stale_since': '2026-11-15 05:26:54 UTC',
|
||||
'outlets': [_outlet(19, on=False, target='pbs'), _outlet(3, on=False)]}]
|
||||
broken = attention.build([], power, None)['broken']
|
||||
assert [b['text'] for b in broken] == ['RPDU outlet 19 is off, and it feeds pbs',
|
||||
'RPDU: outlet watts are frozen']
|
||||
|
||||
def test_powered_outlets_with_nothing_assigned(self):
|
||||
power = [{'node': 'RPDU', 'outlets': [
|
||||
_outlet(1, target='large1'), _outlet(2, power_w=40), _outlet(3, power_w=1),
|
||||
_outlet(4, power_w=40, match='cleared'), _outlet(5, on=False, power_w=0)]}]
|
||||
[line] = attention.build([], power, None)['tidy']
|
||||
assert line['text'] == '1 powered outlet with no device assigned'
|
||||
assert line['detail'] == 'RPDU 2'
|
||||
|
||||
|
||||
class TestCutsheet:
|
||||
VIEW = {
|
||||
'cables': [{'id': 1, 'label': 'C01', 'status': 'installed', 'problem': True},
|
||||
{'id': 2, 'label': 'C02', 'status': 'retired', 'problem': True},
|
||||
{'id': 3, 'label': 'C03', 'status': 'installed', 'problem': False}],
|
||||
'suggestions': [{}, {}, {}],
|
||||
'port_names': [{'kind': 'wrong', 'switch': 'POE', 'port_idx': 11, 'port_name': 'Large1 Management'},
|
||||
{'kind': 'empty', 'switch': 'POE', 'port_idx': 20, 'port_name': 'WAP'},
|
||||
{'kind': 'unnamed', 'switch': 'POE', 'port_idx': 4, 'port_name': 'Port 4'}],
|
||||
'labels': {'cables': [{}, {}], 'ports': [{}]},
|
||||
}
|
||||
|
||||
def test_every_cutsheet_todo(self):
|
||||
out = attention.build([], [], self.VIEW)
|
||||
assert [b['text'] for b in out['broken']] == ['1 cable not where the cutsheet says']
|
||||
assert out['broken'][0]['detail'] == 'C01'
|
||||
assert out['broken'][0]['href'].startswith('/cutsheet?problems=1')
|
||||
tidy = {t['text']: t for t in out['tidy']}
|
||||
assert set(tidy) == {'1 UniFi port name pointing at the wrong device', '1 named port with nothing plugged in',
|
||||
'3 live links not on the cutsheet', '3 labels to print and stick on'}
|
||||
assert tidy['1 UniFi port name pointing at the wrong device']['detail'] == 'POE port 11 "Large1 Management"'
|
||||
# Worst first
|
||||
assert out['tidy'][0]['severity'] == 'warn'
|
||||
|
||||
def test_unnamed_ports_alone_are_not_worth_a_line(self):
|
||||
view = {'port_names': [{'kind': 'unnamed', 'switch': 'POE', 'port_idx': 4, 'port_name': 'Port 4'}]}
|
||||
assert attention.build([], [], view) == {'broken': [], 'tidy': []}
|
||||
|
||||
|
||||
class TestRack:
|
||||
def test_overlaps_and_items_outside_the_rack(self):
|
||||
racks = {'Rack 1': {'overlaps': [{'name': 'NAS', 'units': 'U5-6', 'with': ['storage-01']}],
|
||||
'outside': [{'name': 'Shelf', 'units': 'U43'}]}}
|
||||
texts = [t['text'] for t in attention.build([], [], None, racks)['tidy']]
|
||||
assert texts == ['NAS overlaps storage-01 at U5-6', 'Shelf at U43 is outside Rack 1']
|
||||
|
||||
def test_nothing_to_report(self):
|
||||
assert attention.build([], [], {}, {}) == {'broken': [], 'tidy': []}
|
||||
|
||||
|
||||
NOW = 1_000_000.0
|
||||
|
||||
|
||||
def _quality(states, downtime=(), latencies=None, updated=NOW):
|
||||
"""states: newest last, one per 5 minutes ending at NOW."""
|
||||
n = len(states)
|
||||
return {'periods': [{'ts': NOW - (n - i) * 300, 'state': s} for i, s in enumerate(states)],
|
||||
'downtime': [{'ts': NOW - ago, 'seconds': s} for ago, s in downtime],
|
||||
'latencies': latencies or {}, 'updated': updated}
|
||||
|
||||
|
||||
def _wan(q):
|
||||
return [e for e in attention.build([], [], None, wan_quality=q, now=NOW)['broken']
|
||||
if e['text'].startswith('Internet')]
|
||||
|
||||
|
||||
class TestWan:
|
||||
def test_healthy_says_nothing(self):
|
||||
assert _wan(_quality(['ok'] * 288)) == []
|
||||
|
||||
def test_latency_in_the_last_hour(self):
|
||||
[e] = _wan(_quality(['ok'] * 276 + ['latency', 'ok'] * 3 + ['latency'] * 3 + ['ok'] * 3,
|
||||
latencies={'google': 26, 'cloudflare': 12}))
|
||||
assert e['severity'] == 'warn' and e['href'] == '#modem-section'
|
||||
assert e['text'] == 'Internet: high latency in 6 of the last 12 five-minute periods'
|
||||
assert 'UniFi flagged 6 of the last 288' in e['detail']
|
||||
assert 'Now: cloudflare 12 ms, google 26 ms' in e['detail']
|
||||
|
||||
def test_a_couple_of_slow_periods_are_not_worth_a_line(self):
|
||||
assert _wan(_quality(['ok'] * 10 + ['latency'] * 2)) == []
|
||||
|
||||
def test_old_trouble_has_cleared(self):
|
||||
assert _wan(_quality(['latency'] * 100 + ['ok'] * 12)) == []
|
||||
|
||||
def test_packet_loss_is_broken(self):
|
||||
[e] = _wan(_quality(['ok'] * 9 + ['loss', 'latency', 'down']))
|
||||
assert e['severity'] == 'bad'
|
||||
assert e['text'] == 'Internet: packet loss in 2 of the last 12 five-minute periods'
|
||||
|
||||
def test_a_drop_in_the_last_hour(self):
|
||||
[e] = _wan(_quality(['ok'] * 12, downtime=[(900, 90), (7200, 30)]))
|
||||
assert e['text'] == 'Internet dropped for 90 s, 15 min ago'
|
||||
|
||||
def test_several_drops(self):
|
||||
[e] = _wan(_quality(['ok'] * 12, downtime=[(900, 90), (1800, 30)]))
|
||||
assert e['text'] == 'Internet dropped 2 times in the last hour, 120 s in all'
|
||||
|
||||
def test_stale_copy_is_ignored(self):
|
||||
assert _wan(_quality(['loss'] * 12, updated=NOW - 3600)) == []
|
||||
|
||||
def test_none_is_fine(self):
|
||||
assert _wan(None) == [] and _wan({}) == []
|
||||
|
||||
def test_internet_comes_before_alerts_of_the_same_severity(self):
|
||||
out = attention.build([_alert('x', 'warning')], [], None,
|
||||
wan_quality=_quality(['latency'] * 12), now=NOW)['broken']
|
||||
assert out[0]['text'].startswith('Internet')
|
||||
|
||||
|
||||
class TestModemChecks:
|
||||
def _broken(self, checks, quality=None):
|
||||
return attention.build([], [], None, wan_quality=quality, now=NOW, modem_checks=checks)
|
||||
|
||||
def test_gateway_ping_explains_the_latency(self):
|
||||
checks = {'ping': {'at': NOW - 120, 'avg_ms': 20.4, 'loss_pct': 0, 'ours_ms': 1043, 'verdict': 'gateway'}}
|
||||
[e] = self._broken(checks, _quality(['latency'] * 12))['broken']
|
||||
assert ("Modem's own ping 2 min ago: 20 ms; ours through it 1043 ms, "
|
||||
'so the delay is in the modem, or between it and the router.') in e['detail']
|
||||
|
||||
def test_upstream_verdict(self):
|
||||
checks = {'ping': {'at': NOW - 60, 'avg_ms': 900.0, 'loss_pct': 20, 'ours_ms': 1000, 'verdict': 'upstream'}}
|
||||
[e] = self._broken(checks, _quality(['latency'] * 12))['broken']
|
||||
assert ("900 ms, 20% lost; ours through it 1000 ms, "
|
||||
"so the delay is past the modem, on the provider's side") in e['detail']
|
||||
|
||||
def test_old_ping_is_left_out(self):
|
||||
checks = {'ping': {'at': NOW - 7200, 'avg_ms': 20.0, 'verdict': 'clear'}}
|
||||
[e] = self._broken(checks, _quality(['latency'] * 12))['broken']
|
||||
assert "Modem's own ping" not in e['detail']
|
||||
|
||||
def test_nat_nearly_full(self):
|
||||
[e] = self._broken({'nat': {'in_use': 30000, 'available': 32767}})['broken']
|
||||
assert e['text'] == 'Modem NAT table 92% full' and e['severity'] == 'warn'
|
||||
assert '30,000 of 32,767' in e['detail']
|
||||
|
||||
def test_nat_normal_says_nothing(self):
|
||||
out = self._broken({'nat': {'in_use': 2858, 'available': 32767}})
|
||||
assert out == {'broken': [], 'tidy': []}
|
||||
|
||||
def test_login_failure_is_housekeeping(self):
|
||||
[e] = self._broken({'error': {'login': True, 'message': 'refused', 'at': NOW}})['tidy']
|
||||
assert e['text'].startswith('Modem login failed')
|
||||
|
||||
def test_other_errors_stay_quiet(self):
|
||||
assert self._broken({'error': {'login': False, 'message': 'timeout', 'at': NOW}})['tidy'] == []
|
||||
@@ -0,0 +1,91 @@
|
||||
"""kuma.py: switching the Uptime Kuma maintenance window, against a fake Socket.IO client."""
|
||||
import os
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
|
||||
import kuma # noqa: E402
|
||||
|
||||
CFG = {'url': 'http://kuma.test:3001', 'token': 'tok', 'maintenance_id': 7}
|
||||
PASSWORD_CFG = {'url': 'http://kuma.test:3001', 'username': 'gandalf', 'password': 'pw', 'maintenance_id': 7}
|
||||
|
||||
|
||||
class FakeSocket:
|
||||
def __init__(self, answers=None, connect_error=None, sends_info=True):
|
||||
self.answers = answers or {}
|
||||
self.connect_error = connect_error
|
||||
self.sends_info = sends_info
|
||||
self.calls, self.disconnected, self.handlers = [], False, {}
|
||||
|
||||
def on(self, event, handler):
|
||||
self.handlers[event] = handler
|
||||
|
||||
def connect(self, url, **kw):
|
||||
if self.connect_error:
|
||||
raise self.connect_error
|
||||
self.url = url
|
||||
if self.sends_info: # Kuma's first event; its handlers exist only after it
|
||||
self.handlers['info']({'serverTimezone': 'America/New_York'})
|
||||
|
||||
def call(self, event, data=None, timeout=None):
|
||||
self.calls.append((event, data))
|
||||
return self.answers.get(event, {'ok': True})
|
||||
|
||||
def disconnect(self):
|
||||
self.disconnected = True
|
||||
|
||||
|
||||
def run(active=True, cfg=CFG, **kw):
|
||||
sock = FakeSocket(**kw)
|
||||
kuma.set_maintenance(dict(cfg, timeout=0.05), active, client_factory=lambda: sock)
|
||||
return sock
|
||||
|
||||
|
||||
class TestKuma:
|
||||
def test_configured_needs_url_window_and_a_way_to_sign_in(self):
|
||||
assert kuma.configured(CFG) and kuma.configured(PASSWORD_CFG)
|
||||
assert not kuma.configured(dict(CFG, token=''))
|
||||
assert not kuma.configured(dict(CFG, maintenance_id=None))
|
||||
assert not kuma.configured(dict(PASSWORD_CFG, password=''))
|
||||
assert not kuma.configured(None)
|
||||
|
||||
def test_token_sign_in_then_switches_the_window_on(self):
|
||||
sock = run(True)
|
||||
assert sock.calls == [('loginByToken', 'tok'), ('resumeMaintenance', 7)]
|
||||
assert sock.disconnected and sock.url == CFG['url']
|
||||
|
||||
def test_rejected_token(self):
|
||||
with pytest.raises(kuma.KumaError, match='token rejected'):
|
||||
run(answers={'loginByToken': {'ok': False}})
|
||||
|
||||
def test_waits_for_kuma_to_be_ready(self):
|
||||
sock = FakeSocket(sends_info=False)
|
||||
with pytest.raises(kuma.KumaError, match='not ready'):
|
||||
kuma.set_maintenance(dict(CFG, timeout=0.05), True, client_factory=lambda: sock)
|
||||
assert sock.calls == [] and sock.disconnected # nothing sent before info
|
||||
|
||||
def test_password_sign_in(self):
|
||||
sock = run(True, cfg=PASSWORD_CFG)
|
||||
assert sock.calls[0] == ('login', {'username': 'gandalf', 'password': 'pw', 'token': ''})
|
||||
|
||||
def test_pause(self):
|
||||
assert run(False).calls[-1] == ('pauseMaintenance', 7)
|
||||
|
||||
def test_refused_password(self):
|
||||
with pytest.raises(kuma.KumaError, match='login refused: Incorrect username or password'):
|
||||
run(cfg=PASSWORD_CFG, answers={'login': {'ok': False, 'msg': 'Incorrect username or password.'}})
|
||||
|
||||
def test_two_factor_account_needs_the_token(self):
|
||||
with pytest.raises(kuma.KumaError, match='use uptime_kuma.token'):
|
||||
run(cfg=PASSWORD_CFG, answers={'login': {'ok': False, 'tokenRequired': True}})
|
||||
|
||||
def test_refused_change(self):
|
||||
with pytest.raises(kuma.KumaError, match='resumeMaintenance refused'):
|
||||
run(answers={'resumeMaintenance': {'ok': False, 'msg': 'not found'}})
|
||||
|
||||
def test_unreachable(self):
|
||||
sock = FakeSocket(connect_error=ConnectionError('refused'))
|
||||
with pytest.raises(kuma.KumaError, match='ConnectionError'):
|
||||
kuma.set_maintenance(dict(CFG, timeout=0.05), True, client_factory=lambda: sock)
|
||||
assert sock.disconnected
|
||||
@@ -255,3 +255,306 @@ class TestWanHistory:
|
||||
h = {}
|
||||
modem.record_uptime_reset(h, 5000)
|
||||
assert h['restarts'] == [{'at': 5000, 'by': None, 'confirmed': True}]
|
||||
|
||||
|
||||
def _dashboard(health, downtime=(), latencies=None, group='WAN'):
|
||||
return {
|
||||
'wan_history': {'wan_history_details': [{
|
||||
'network_group': group,
|
||||
'health_history': [dict({'timestamp': ts * 1000, 'high_latency': False, 'packet_loss': False,
|
||||
'wan_downtime': False}, **flags) for ts, flags in health],
|
||||
'downtime_history': [{'timestamp': ts * 1000, 'downtime_in_seconds': s} for ts, s in downtime],
|
||||
}]},
|
||||
'wan': {'wan_details': [{'network_group': group, 'stats': {'service_latencies': [
|
||||
{'service_name': k, 'latency': v} for k, v in (latencies or {}).items()]}}]},
|
||||
}
|
||||
|
||||
|
||||
class TestWanQuality:
|
||||
def test_worst_flag_wins_and_periods_are_in_time_order(self):
|
||||
q = modem.wan_quality(_dashboard([
|
||||
(600, {'high_latency': True, 'packet_loss': True}),
|
||||
(0, {}),
|
||||
(300, {'high_latency': True}),
|
||||
(900, {'wan_downtime': True, 'packet_loss': True}),
|
||||
]), now=1000)
|
||||
assert [p['state'] for p in q['periods']] == ['ok', 'latency', 'loss', 'down']
|
||||
assert [p['ts'] for p in q['periods']] == [0, 300, 600, 900]
|
||||
assert q['updated'] == 1000 and q['wan'] == 'WAN'
|
||||
|
||||
def test_downtime_and_latencies(self):
|
||||
q = modem.wan_quality(_dashboard([(0, {})], downtime=[(500, 90)],
|
||||
latencies={'cloudflare': 12, 'google': 26}), now=1)
|
||||
assert q['downtime'] == [{'ts': 500, 'seconds': 90}]
|
||||
assert q['latencies'] == {'cloudflare': 12, 'google': 26}
|
||||
|
||||
def test_prefers_the_primary_wan(self):
|
||||
d = _dashboard([(0, {'high_latency': True})], group='WAN2')
|
||||
d['wan_history']['wan_history_details'].append(
|
||||
_dashboard([(0, {})])['wan_history']['wan_history_details'][0])
|
||||
assert modem.wan_quality(d, 0)['periods'][0]['state'] == 'ok'
|
||||
|
||||
@pytest.mark.parametrize('dashboard', [None, {}, {'wan_history': {}}, {'wan_history': {'wan_history_details': []}}])
|
||||
def test_missing_history_is_none(self, dashboard):
|
||||
assert modem.wan_quality(dashboard, 0) is None
|
||||
|
||||
def test_counts_since(self):
|
||||
periods = [{'ts': 0, 'state': 'latency'}, {'ts': 300, 'state': 'ok'}, {'ts': 600, 'state': 'loss'}]
|
||||
assert modem.quality_counts(periods, 300) == {'ok': 1, 'latency': 0, 'loss': 1, 'down': 0}
|
||||
|
||||
|
||||
# Trimmed from a BGW320-505 (firmware 6.40.4)
|
||||
NAT_PAGE = '''<table class="table60" summary="This table displays a summary of session information.">
|
||||
<tr>
|
||||
<th scope="row" nowrap="nowrap">Total sessions available</th>
|
||||
<td>32767
|
||||
</td></tr>
|
||||
<tr>
|
||||
<th scope="row">Total sessions in use</th>
|
||||
<td>2858
|
||||
</td></tr></table>'''
|
||||
LOGIN_FORM = ('<h1>Access Code Required</h1><form>'
|
||||
'<input type="hidden" name="nonce" value="0a1b2c3d4e5f60718293a4b5c6d7e8f9" />')
|
||||
LOGIN_NO_NONCE = '<h1>Access Code Required</h1><strong>Cookies must be enabled</strong>'
|
||||
DIAG_FORM = ('<form method="post" action="/cgi-bin/diag.ha">'
|
||||
'<input type="hidden" name="nonce" value="ffee00112233445566778899aabbccdd" />')
|
||||
PING_DONE = '''PING 1.1.1.1 (1.1.1.1): 56 data bytes
|
||||
64 bytes from 1.1.1.1: seq=0 ttl=59 time=24.697 ms
|
||||
64 bytes from 1.1.1.1: seq=4 ttl=59 time=12.392 ms
|
||||
|
||||
--- 1.1.1.1 ping statistics ---
|
||||
5 packets transmitted, 5 packets received, 0% packet loss
|
||||
round-trip min/avg/max = 12.392/20.501/32.697 ms
|
||||
test done'''
|
||||
|
||||
|
||||
def _diag(window):
|
||||
return DIAG_FORM + f'<textarea id="progress" name="ProgressWindow" rows="15">{window}</textarea></form>'
|
||||
|
||||
|
||||
class TestModemPages:
|
||||
def test_login_hash_matches_the_page_script(self):
|
||||
# The page posts hex_md5(password + nonce)
|
||||
assert modem.login_hash('abc', '123') == 'e99a18c428cb38d5f260853678922e03'
|
||||
|
||||
def test_login_page_is_recognised(self):
|
||||
assert modem.is_login_page(LOGIN_FORM) and modem.is_login_page(LOGIN_NO_NONCE)
|
||||
assert not modem.is_login_page(NAT_PAGE)
|
||||
|
||||
def test_nat_summary(self):
|
||||
assert modem.parse_nat_summary(NAT_PAGE) == {'in_use': 2858, 'available': 32767}
|
||||
assert modem.parse_nat_summary('<table></table>') is None
|
||||
|
||||
def test_ping_output(self):
|
||||
assert modem.parse_ping_output(PING_DONE) == {
|
||||
'sent': 5, 'received': 5, 'loss_pct': 0, 'min_ms': 12.392, 'avg_ms': 20.501, 'max_ms': 32.697}
|
||||
|
||||
def test_ping_still_running_is_none(self):
|
||||
assert modem.parse_ping_output('PING 1.1.1.1 (1.1.1.1): 56 data bytes\n64 bytes from') is None
|
||||
|
||||
def test_ping_all_lost(self):
|
||||
r = modem.parse_ping_output('5 packets transmitted, 0 packets received, 100% packet loss')
|
||||
assert r['loss_pct'] == 100 and r['avg_ms'] is None
|
||||
|
||||
@pytest.mark.parametrize('gateway,ours,expected', [
|
||||
({'avg_ms': 900.0, 'loss_pct': 0}, 1000, 'upstream'),
|
||||
({'avg_ms': 20.0, 'loss_pct': 40}, 1000, 'upstream'),
|
||||
({'avg_ms': None, 'loss_pct': 100}, None, 'upstream'),
|
||||
({'avg_ms': 20.0, 'loss_pct': 0}, 1000, 'gateway'),
|
||||
({'avg_ms': 20.0, 'loss_pct': 0}, 30, 'clear'),
|
||||
({'avg_ms': 20.0, 'loss_pct': 0}, None, 'clear'),
|
||||
])
|
||||
def test_verdict(self, gateway, ours, expected):
|
||||
assert modem.ping_verdict(gateway, ours) == expected
|
||||
|
||||
|
||||
class FakeGateway:
|
||||
"""Answers ModemAdmin's requests like the BGW320 does."""
|
||||
|
||||
def __init__(self, code='secret', ping_pages=None):
|
||||
self.code, self.logged_in, self.calls = code, False, []
|
||||
self.cookie = self.pinged = False
|
||||
self.ping_pages = list(ping_pages or [_diag(PING_DONE)])
|
||||
|
||||
def __call__(self, method, url, **kw):
|
||||
path = url.split('192.0.2.254', 1)[1]
|
||||
self.calls.append((method, path))
|
||||
if path == '/cgi-bin/login.ha' and method == 'GET':
|
||||
first, self.cookie = not self.cookie, True
|
||||
return MagicMock(status_code=200, text=LOGIN_NO_NONCE if first else LOGIN_FORM)
|
||||
if path == '/cgi-bin/login.ha':
|
||||
ok = kw['data']['hashpassword'] == modem.login_hash(self.code, '0a1b2c3d4e5f60718293a4b5c6d7e8f9')
|
||||
assert set(kw['data']['password']) == {'*'} # the code itself is never sent
|
||||
self.logged_in = ok
|
||||
return MagicMock(status_code=302 if ok else 200, text='' if ok else LOGIN_FORM)
|
||||
if not self.logged_in:
|
||||
return MagicMock(status_code=200, text=LOGIN_FORM)
|
||||
if path == '/cgi-bin/restart.ha' and method == 'GET':
|
||||
return MagicMock(status_code=200, text=DIAG_FORM.replace('diag.ha', 'restart.ha'))
|
||||
if path == '/cgi-bin/nattable.ha':
|
||||
return MagicMock(status_code=200, text=NAT_PAGE)
|
||||
if path == '/cgi-bin/diag.ha' and method == 'POST':
|
||||
assert kw['data']['WebAddress'] == '1.1.1.1' and kw['data']['Ping'] == 'Ping'
|
||||
self.pinged = True
|
||||
return MagicMock(status_code=200, text=_diag(''))
|
||||
if path == '/cgi-bin/diag.ha':
|
||||
# Before the test the window is empty; after it, each refresh shows more output
|
||||
page = self.ping_pages.pop(0) if self.pinged else _diag('')
|
||||
return MagicMock(status_code=200, text=page)
|
||||
raise AssertionError(path)
|
||||
|
||||
|
||||
class TestModemAdmin:
|
||||
def _admin(self, gw, now=None):
|
||||
clock = MagicMock(return_value=now or 1000.0)
|
||||
a = modem.ModemAdmin({'url': 'http://192.0.2.254', 'access_code': 'secret'},
|
||||
clock=clock, sleep=lambda s: None)
|
||||
a.session.request = MagicMock(side_effect=gw)
|
||||
return a, clock
|
||||
|
||||
def test_logs_in_when_needed_and_reads_nat(self):
|
||||
gw = FakeGateway()
|
||||
a, _ = self._admin(gw)
|
||||
assert a.nat_sessions() == {'in_use': 2858, 'available': 32767}
|
||||
assert ('POST', '/cgi-bin/login.ha') in gw.calls
|
||||
gw.calls.clear()
|
||||
a.nat_sessions() # session still good: no second login
|
||||
assert ('POST', '/cgi-bin/login.ha') not in gw.calls
|
||||
|
||||
def test_expired_session_logs_in_again(self):
|
||||
gw = FakeGateway()
|
||||
a, _ = self._admin(gw)
|
||||
a.nat_sessions()
|
||||
gw.logged_in = False
|
||||
assert a.nat_sessions()['in_use'] == 2858
|
||||
|
||||
def test_refused_code_backs_off(self):
|
||||
gw = FakeGateway(code='other')
|
||||
a, clock = self._admin(gw)
|
||||
with pytest.raises(modem.ModemLoginFailed):
|
||||
a.nat_sessions()
|
||||
gw.calls.clear()
|
||||
clock.return_value = 1000.0 + modem.LOGIN_BACKOFF_S - 1
|
||||
with pytest.raises(modem.ModemLoginFailed):
|
||||
a.nat_sessions()
|
||||
assert ('POST', '/cgi-bin/login.ha') not in gw.calls # didn't even try
|
||||
|
||||
def test_ping_from_the_gateway_waits_for_the_result(self):
|
||||
gw = FakeGateway(ping_pages=[_diag('PING 1.1.1.1 (1.1.1.1): 56 data bytes'), _diag(PING_DONE)])
|
||||
a, _ = self._admin(gw)
|
||||
r = a.ping('1.1.1.1')
|
||||
assert r['avg_ms'] == 20.501 and r['target'] == '1.1.1.1'
|
||||
|
||||
def test_ping_that_never_finishes(self):
|
||||
gw = FakeGateway(ping_pages=[_diag('PING ...')] * 50)
|
||||
a, clock = self._admin(gw)
|
||||
clock.side_effect = [1000.0] + [2000.0] * 100
|
||||
with pytest.raises(modem.ModemDataInvalid, match='did not finish'):
|
||||
a.ping('1.1.1.1')
|
||||
|
||||
def test_unreachable(self):
|
||||
a, _ = self._admin(FakeGateway())
|
||||
a.session.request = MagicMock(side_effect=requests.ConnectionError('boom'))
|
||||
with pytest.raises(modem.ModemUnreachable):
|
||||
a.nat_sessions()
|
||||
|
||||
|
||||
FW_LOG = '''<table><tr><th>No.</th><th>Date/Time</th></tr>
|
||||
<tr class="b"><td class="heading" scope="row">2</td><td>2026-11-19T18:32:04.984186</td><td>198.51.100.20</td>
|
||||
<td>203.0.113.7</td><td>TCP</td><td>FLOOD limit 50pps burst 100</td></tr>
|
||||
<tr class="a"><td class="heading" scope="row">1</td><td>2026-11-19T18:32:02.983868</td><td>203.0.113.7</td>
|
||||
<td>198.51.100.30</td><td>TCP</td><td>Invalid IP Packet</td></tr>
|
||||
</table>'''
|
||||
SPEED_PAGE = '''<h2>Test History</h2><table><thead><th scope="col">Test Completion Time</th></thead><tbody>
|
||||
<tr class="b"><td scope="row" class="heading" width="47%">
|
||||
11/06/2026 08:46:05</td>
|
||||
<td>upstream</td>
|
||||
<td> 1242.810000</td>
|
||||
<td> 28</td>
|
||||
<td> 19.000000</td>
|
||||
<td> Success</td></tr>
|
||||
<tr class="a"><td scope="row" class="heading" width="47%">
|
||||
11/06/2026 08:45:57</td>
|
||||
<td>downstream</td>
|
||||
<td> 1167.400000</td>
|
||||
<td> 28</td>
|
||||
<td> 28.000000</td>
|
||||
<td> Success</td></tr>
|
||||
</tbody></table>'''
|
||||
|
||||
|
||||
class TestModemExtras:
|
||||
@pytest.mark.parametrize('raw,expected', [
|
||||
('03:06:29:34', ((3 * 24 + 6) * 60 + 29) * 60 + 34), # firmware 6.40
|
||||
('28951', 28951), # older firmware
|
||||
('0:00:05:00', 300),
|
||||
('', None), (None, None), ('soon', None),
|
||||
])
|
||||
def test_uptime_formats(self, raw, expected):
|
||||
assert modem.parse_uptime(raw) == expected
|
||||
|
||||
def test_status_reads_dhms_uptime(self):
|
||||
sysinfo = SYSINFO.replace('<td class="col2">28951</td>', '<td class="col2">03:06:29:34</td>')
|
||||
assert sysinfo != SYSINFO
|
||||
assert modem.build_status(fiber_page(), broadband_page(), sysinfo)['uptime_s'] == 282574
|
||||
|
||||
def test_firewall_log_oldest_first(self):
|
||||
rows = modem.parse_firewall_log(FW_LOG)
|
||||
assert [r['ts'] for r in rows] == ['2026-11-19T18:32:02.983868', '2026-11-19T18:32:04.984186']
|
||||
assert rows[1]['reason'] == 'FLOOD limit 50pps burst 100' and rows[1]['proto'] == 'TCP'
|
||||
|
||||
@pytest.mark.parametrize('reason,kind', [
|
||||
('FLOOD limit 50pps burst 100', 'flood'), ('Invalid IP Packet', 'invalid'), ('Port scan', 'other')])
|
||||
def test_firewall_kind(self, reason, kind):
|
||||
assert modem.firewall_kind(reason) == kind
|
||||
|
||||
def test_speed_tests(self):
|
||||
assert modem.parse_speed_tests(SPEED_PAGE) == [
|
||||
{'at': '11/06/2026 08:46:05', 'direction': 'upstream', 'mbps': 1242.8, 'latency_ms': 19.0, 'ok': True},
|
||||
{'at': '11/06/2026 08:45:57', 'direction': 'downstream', 'mbps': 1167.4, 'latency_ms': 28.0, 'ok': True},
|
||||
]
|
||||
|
||||
def test_speed_tests_missing(self):
|
||||
assert modem.parse_speed_tests('<html></html>') == []
|
||||
|
||||
|
||||
class TestModemRestart:
|
||||
def _admin(self):
|
||||
gw = FakeGateway()
|
||||
a = modem.ModemAdmin({'url': 'http://192.0.2.254', 'access_code': 'secret'}, sleep=lambda s: None)
|
||||
a.session.request = MagicMock(side_effect=gw)
|
||||
return a, gw
|
||||
|
||||
REDIRECT = MagicMock(status_code=302, headers={'Location': '/cgi-bin/restarting.ha'}, text='')
|
||||
RESTARTING = MagicMock(status_code=200, text='<title>Restarting</title> Your AT&T BGW320-505 is restarting.')
|
||||
|
||||
def test_posts_the_form_then_loads_the_restarting_page(self):
|
||||
# Firmware 6.40: the post only redirects; loading restarting.ha starts the restart
|
||||
a, gw = self._admin()
|
||||
a.session.post = MagicMock(return_value=self.REDIRECT)
|
||||
a.session.get = MagicMock(return_value=self.RESTARTING)
|
||||
assert a.restart() == 'accepted'
|
||||
assert a.session.post.call_args.kwargs['data'] == {'nonce': 'ffee00112233445566778899aabbccdd',
|
||||
'Restart': 'Restart'}
|
||||
assert a.session.post.call_args.args[0].endswith('/cgi-bin/restart.ha')
|
||||
assert a.session.get.call_args.args[0].endswith('/cgi-bin/restarting.ha')
|
||||
assert ('POST', '/cgi-bin/login.ha') in gw.calls # logged in to read the form
|
||||
|
||||
def test_redirect_alone_is_not_a_restart(self):
|
||||
a, _ = self._admin()
|
||||
a.session.post = MagicMock(return_value=self.REDIRECT)
|
||||
a.session.get = MagicMock(return_value=MagicMock(status_code=200, text='<title>Device</title>'))
|
||||
with pytest.raises(modem.ModemDataInvalid, match='not started'):
|
||||
a.restart()
|
||||
|
||||
def test_connection_dropped_mid_restart(self):
|
||||
a, _ = self._admin()
|
||||
a.session.post = MagicMock(return_value=self.REDIRECT)
|
||||
a.session.get = MagicMock(side_effect=requests.ConnectionError('reset'))
|
||||
assert a.restart() == 'unconfirmed'
|
||||
|
||||
def test_refused(self):
|
||||
a, _ = self._admin()
|
||||
a.session.post = MagicMock(return_value=MagicMock(status_code=200, text=LOGIN_FORM))
|
||||
with pytest.raises(modem.ModemDataInvalid, match='not started'):
|
||||
a.restart()
|
||||
@@ -1018,3 +1018,190 @@ class TestSyncOpenTickets:
|
||||
mon._open_tickets_synced -= monitor.OPEN_TICKETS_SYNC_S + 1
|
||||
mon._sync_open_tickets()
|
||||
assert mon.tickets.open_tickets.call_count == 2
|
||||
|
||||
|
||||
class TestWanQualityCollection:
|
||||
def test_stores_the_timeline(self, mon, db):
|
||||
mon.unifi = MagicMock()
|
||||
mon.unifi.get_wan_dashboard.return_value = {'wan_history': {'wan_history_details': [
|
||||
{'network_group': 'WAN', 'health_history': [{'timestamp': 1000, 'high_latency': True}]}]}}
|
||||
mon._collect_wan_quality()
|
||||
key, value = db.set_state.call_args.args
|
||||
assert key == 'wan_quality'
|
||||
assert value['periods'] == [{'ts': 1.0, 'state': 'latency'}]
|
||||
|
||||
def test_unavailable_keeps_the_last_copy(self, mon, db):
|
||||
mon.unifi = MagicMock()
|
||||
mon.unifi.get_wan_dashboard.return_value = None
|
||||
mon._collect_wan_quality()
|
||||
assert not db.set_state.called
|
||||
|
||||
def test_client_reads_the_day(self):
|
||||
c = monitor.UnifiClient({'controller': 'https://udm.test', 'api_key': 'k'})
|
||||
c.session.get = MagicMock(return_value=MagicMock(json=lambda: {'wan_history': {}}))
|
||||
assert c.get_wan_dashboard() == {'wan_history': {}}
|
||||
url = c.session.get.call_args.args[0]
|
||||
assert url.endswith('/proxy/network/v2/api/site/default/aggregated-dashboard')
|
||||
assert c.session.get.call_args.kwargs['params'] == {'historySeconds': 86400}
|
||||
|
||||
def test_client_error_is_none(self):
|
||||
c = monitor.UnifiClient({'controller': 'https://udm.test', 'api_key': 'k'})
|
||||
c.session.get = MagicMock(side_effect=RuntimeError('boom'))
|
||||
assert c.get_wan_dashboard() is None
|
||||
|
||||
|
||||
class TestModemChecks:
|
||||
NOW = 1_000_000.0
|
||||
|
||||
def _run(self, mon, db, checks=None, quality=None, admin=None):
|
||||
state = {'modem_checks': json.dumps(checks) if checks is not None else None,
|
||||
'wan_quality': json.dumps(quality) if quality is not None else None}
|
||||
db.get_state.side_effect = lambda k, d=None: state.get(k, d)
|
||||
db.set_state.reset_mock()
|
||||
mon.modem_admin = admin or MagicMock()
|
||||
if admin is None:
|
||||
mon.modem_admin.nat_sessions.return_value = {'in_use': 2858, 'available': 32767}
|
||||
mon.modem_admin.ping.return_value = {'sent': 5, 'received': 5, 'loss_pct': 0, 'min_ms': 12.0,
|
||||
'avg_ms': 20.0, 'max_ms': 32.0, 'target': '1.1.1.1'}
|
||||
with patch.object(monitor.time, 'time', return_value=self.NOW):
|
||||
mon._collect_modem_checks()
|
||||
saved = [c.args[1] for c in db.set_state.call_args_list if c.args[0] == 'modem_checks']
|
||||
return saved[-1] if saved else None
|
||||
|
||||
def _slow(self):
|
||||
return {'periods': [{'ts': self.NOW - 300, 'state': 'latency'}], 'latencies': {'cloudflare': 1043}}
|
||||
|
||||
def test_no_access_code_does_nothing(self, mon, db):
|
||||
mon.modem_admin = None
|
||||
mon._collect_modem_checks()
|
||||
assert not db.set_state.called
|
||||
|
||||
def test_nat_every_half_hour(self, mon, db):
|
||||
out = self._run(mon, db)
|
||||
assert out['nat'] == {'in_use': 2858, 'available': 32767, 'at': self.NOW}
|
||||
assert 'ping' not in out # the internet is fine, so no ping
|
||||
recent = {'nat': {'at': self.NOW - 60}, 'firewall': {'checked_at': self.NOW - 60}}
|
||||
assert self._run(mon, db, checks=recent) is None
|
||||
|
||||
def test_slow_internet_pings_from_the_modem(self, mon, db):
|
||||
out = self._run(mon, db, checks={'nat': {'at': self.NOW - 60}}, quality=self._slow())
|
||||
assert out['ping']['avg_ms'] == 20.0 and out['ping']['ours_ms'] == 1043
|
||||
assert out['ping']['verdict'] == 'gateway'
|
||||
mon.modem_admin.ping.assert_called_once_with('1.1.1.1')
|
||||
|
||||
def test_ping_at_most_every_ten_minutes(self, mon, db):
|
||||
checks = {'nat': {'at': self.NOW - 60}, 'ping': {'at': self.NOW - 300},
|
||||
'firewall': {'checked_at': self.NOW - 60}}
|
||||
assert self._run(mon, db, checks=checks, quality=self._slow()) is None
|
||||
|
||||
def test_failure_is_recorded_and_paused(self, mon, db):
|
||||
admin = MagicMock()
|
||||
admin.nat_sessions.side_effect = modem_api.ModemLoginFailed('refused')
|
||||
out = self._run(mon, db, admin=admin)
|
||||
assert out['error']['login'] is True and out['error']['at'] == self.NOW
|
||||
admin.nat_sessions.reset_mock()
|
||||
db.set_state.reset_mock()
|
||||
assert self._run(mon, db, checks=out, admin=admin) is None
|
||||
assert not admin.nat_sessions.called
|
||||
|
||||
def test_success_clears_the_error(self, mon, db):
|
||||
out = self._run(mon, db, checks={'error': {'message': 'x', 'at': self.NOW - 3600, 'login': True}})
|
||||
assert out['error'] is None
|
||||
|
||||
|
||||
class TestModemExtrasCollection:
|
||||
def _row(self, ts, reason='FLOOD limit 50pps burst 100'):
|
||||
return {'ts': ts, 'src': 'a', 'dst': 'b', 'proto': 'TCP', 'reason': reason}
|
||||
|
||||
def test_counts_only_new_rows_per_hour(self, mon):
|
||||
first = mon._count_firewall_drops(None, [self._row('2026-11-19T17:59:58.1'),
|
||||
self._row('2026-11-19T18:00:01.2', 'Invalid IP Packet')], 100.0)
|
||||
assert first['hours'] == {'2026-11-19T17': {'flood': 1, 'invalid': 0, 'other': 0},
|
||||
'2026-11-19T18': {'flood': 0, 'invalid': 1, 'other': 0}}
|
||||
assert first['last'] == '2026-11-19T18:00:01.2' and first['gap'] is False
|
||||
# The next read overlaps the last one: only the new row counts
|
||||
second = mon._count_firewall_drops(first, [self._row('2026-11-19T18:00:01.2', 'Invalid IP Packet'),
|
||||
self._row('2026-11-19T18:00:03.0')], 400.0)
|
||||
assert second['hours']['2026-11-19T18'] == {'flood': 1, 'invalid': 1, 'other': 0}
|
||||
assert second['checked_at'] == 400.0
|
||||
|
||||
def test_gap_when_the_log_rolled_over(self, mon):
|
||||
prev = {'hours': {}, 'last': '2026-11-19T18:00:00.0'}
|
||||
out = mon._count_firewall_drops(prev, [self._row('2026-11-19T18:20:00.0')], 1.0)
|
||||
assert out['gap'] is True
|
||||
|
||||
def test_keeps_a_day(self, mon):
|
||||
prev = {'hours': {f'2026-11-18T{h:02d}': {'flood': 1, 'invalid': 0, 'other': 0} for h in range(24)},
|
||||
'last': '2026-11-18T23:59:59.0'}
|
||||
out = mon._count_firewall_drops(prev, [self._row('2026-11-19T00:00:01.0')], 1.0)
|
||||
assert len(out['hours']) == 24 and '2026-11-18T00' not in out['hours']
|
||||
|
||||
def test_latest_speed_tests(self, mon):
|
||||
tests = [{'direction': 'upstream', 'mbps': 1242.8, 'at': 'b'},
|
||||
{'direction': 'downstream', 'mbps': 1167.4, 'at': 'b'},
|
||||
{'direction': 'upstream', 'mbps': 1.0, 'at': 'a'}]
|
||||
out = mon._latest_speed_tests(tests, 5.0)
|
||||
assert out['up']['mbps'] == 1242.8 and out['down']['mbps'] == 1167.4 and out['checked_at'] == 5.0
|
||||
|
||||
def test_scheduled_with_the_other_checks(self, mon, db):
|
||||
admin = MagicMock()
|
||||
admin.nat_sessions.return_value = {'in_use': 1, 'available': 10}
|
||||
admin.speed_tests.return_value = []
|
||||
admin.firewall_log.return_value = [self._row('2026-11-19T18:00:00.0')]
|
||||
mon.modem_admin = admin
|
||||
db.get_state.side_effect = lambda k, d=None: None
|
||||
mon._collect_modem_checks()
|
||||
saved = [c.args[1] for c in db.set_state.call_args_list if c.args[0] == 'modem_checks'][-1]
|
||||
assert saved['firewall']['hours'] == {'2026-11-19T18': {'flood': 1, 'invalid': 0, 'other': 0}}
|
||||
assert 'speed' in saved and admin.firewall_log.called
|
||||
|
||||
|
||||
class TestFinishModemRestart:
|
||||
NOW = 1_000_000.0
|
||||
|
||||
def _run(self, mon, db, restart, stats, kuma_error=None):
|
||||
state = {'modem_restart': json.dumps(restart), 'modem_stats': json.dumps(stats)}
|
||||
db.get_state.side_effect = lambda k, d=None: state.get(k, d)
|
||||
db.set_state.reset_mock()
|
||||
mon.cfg = {'uptime_kuma': {'url': 'http://kuma.test'}}
|
||||
with patch.object(monitor.time, 'time', return_value=self.NOW), \
|
||||
patch.object(monitor.kuma, 'set_maintenance', side_effect=kuma_error) as km:
|
||||
mon._finish_modem_restart()
|
||||
saved = [c.args[1] for c in db.set_state.call_args_list if c.args[0] == 'modem_restart']
|
||||
return (saved[-1] if saved else None), km
|
||||
|
||||
def _restart(self, ago, **kw):
|
||||
return dict({'at': self.NOW - ago, 'by': 'jared', 'suppression_id': 9, 'kuma': True,
|
||||
'finished_at': None}, **kw)
|
||||
|
||||
def test_back_ends_kuma_and_the_silence(self, mon, db):
|
||||
out, km = self._run(mon, db, self._restart(300), {'reachable': True, 'wan_problems': [], 'uptime_s': 200})
|
||||
km.assert_called_once_with({'url': 'http://kuma.test'}, False)
|
||||
db.deactivate_suppression.assert_called_once_with(9)
|
||||
assert out['back'] and out['rebooted'] and out['back_after_s'] == 300 and out['finished_at'] == self.NOW
|
||||
|
||||
def test_waits_while_the_modem_is_down(self, mon, db):
|
||||
out, km = self._run(mon, db, self._restart(300), {'reachable': False})
|
||||
assert out is None and not km.called and not db.deactivate_suppression.called
|
||||
|
||||
def test_too_soon_to_tell(self, mon, db):
|
||||
out, _ = self._run(mon, db, self._restart(60), {'reachable': True, 'wan_problems': [], 'uptime_s': 30})
|
||||
assert out is None
|
||||
|
||||
def test_answering_without_having_rebooted_is_not_back(self, mon, db):
|
||||
out, _ = self._run(mon, db, self._restart(300), {'reachable': True, 'wan_problems': [], 'uptime_s': 99999})
|
||||
assert out is None
|
||||
|
||||
def test_gives_up_after_half_an_hour(self, mon, db):
|
||||
out, km = self._run(mon, db, self._restart(1900), {'reachable': False})
|
||||
assert out['back'] is False and out['back_after_s'] is None and km.called
|
||||
db.deactivate_suppression.assert_called_once_with(9)
|
||||
|
||||
def test_kuma_failure_retries_next_cycle(self, mon, db):
|
||||
out, _ = self._run(mon, db, self._restart(300), {'reachable': True, 'wan_problems': [], 'uptime_s': 200},
|
||||
kuma_error=monitor.kuma.KumaError('down'))
|
||||
assert out is None and not db.deactivate_suppression.called
|
||||
|
||||
def test_finished_restart_is_left_alone(self, mon, db):
|
||||
out, km = self._run(mon, db, self._restart(300, finished_at=1.0), {'reachable': True})
|
||||
assert out is None and not km.called
|
||||
+1
-1
@@ -62,7 +62,7 @@ def link(text: str, url: str) -> str:
|
||||
|
||||
|
||||
def port_text(switch: str, port_name: str, port: dict) -> str:
|
||||
"""'8SFP-AGG port 7 "Large1": up at 10 Gbps, DAC'"""
|
||||
"""'8SFP-AGG port 7 "compute-storage-02": up at 10 Gbps, DAC'"""
|
||||
idx = port.get('port_idx')
|
||||
text = f'{switch} port {idx}' if idx is not None else f'{switch} {port_name}'
|
||||
if port_name and idx is not None and port_name != f'Port {idx}':
|
||||
|
||||
+2
-2
@@ -25,7 +25,7 @@ _TOKEN_RE = re.compile(r'[a-z0-9]+')
|
||||
|
||||
|
||||
def _norm(name: str) -> str:
|
||||
"""A name with case, spaces, dashes and underscores ignored: "Large1" == "large-1"."""
|
||||
"""A name with case, spaces, dashes and underscores ignored: "Storage-01" == "storage_01"."""
|
||||
return ''.join(_TOKEN_RE.findall((name or '').lower()))
|
||||
|
||||
|
||||
@@ -114,7 +114,7 @@ def name_mismatch(port_name: str, neighbour: str, own: str, known: Dict[str, str
|
||||
"""A port named after a different device than the one plugged into it.
|
||||
|
||||
known: {normalised name: display name} of every device. Runs of up to four words of
|
||||
the port name are compared with them, so "Large1 Management" names large1 and
|
||||
the port name are compared with them, so "Storage-01 Management" names storage-01 and
|
||||
"Downlink to 24RJ45-POE" names 24RJ45-POE. Ports named only "Port 4" never match.
|
||||
"""
|
||||
if not port_name or _is_generic_port_name(port_name):
|
||||
|
||||
Reference in new issue
Block a user