Compare commits

...
Author SHA1 Message Date
jaredandClaude Opus 5.5 1444562a5d Kuma: sign in with a session token, wait for Kuma to be ready
Lint / JS (eslint) (push) Successful in 10s
Lint / Python (flake8) (push) Successful in 58s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 1m1s
Lint / Python (flake8) (pull_request) Successful in 57s
Lint / JS (eslint) (pull_request) Successful in 8s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Test / Python Tests (pytest) (push) Successful in 1m34s
Test / Python Tests (pytest) (pull_request) Successful in 1m45s
Security / Python Security (bandit) (pull_request) Successful in 59s
The Kuma account has two-factor login, so a password login always ends
in tokenRequired. Prod config now holds a Kuma session token instead
(uptime_kuma = {url, token, maintenance_id}).

- Sign in with loginByToken; a refusal says the token was rejected
  (Kuma password changed?). username/password stay as a fallback for an
  account without two-factor login.
- Wait for Kuma's first 'info' event before emitting: Kuma registers
  its handlers only after sending it, so an earlier emit is dropped and
  times out now and then.
- configured() needs url, maintenance_id and a token (or username and
  password).

Tested from prod against Kuma 2.5.5: signed in and paused window 7,
which was already paused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 20:59:37 -05:00
jared dd9ba41479 Merge pull request 'Restart modem button: silence Gandalf and Uptime Kuma, restart, end both' (#53) from modem-restart into main
Lint / JS (eslint) (push) Successful in 9s
Lint / Python (flake8) (push) Successful in 52s
Lint / Notify on failure (push) Skipped
Security / Python Security (bandit) (push) Successful in 1m6s
Test / Python Tests (pytest) (push) Successful in 1m15s
Lint / Deploy (push) Successful in 12s
2026-11-19 20:39:02 -05:00
jaredandClaude Opus 5.5 3afd85626b Modem restart: load restarting.ha, which is what starts it
Lint / JS (eslint) (push) Successful in 8s
Lint / Python (flake8) (push) Successful in 1m2s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 1m3s
Lint / Python (flake8) (pull_request) Successful in 2m7s
Test / Python Tests (pytest) (push) Successful in 2m20s
Lint / JS (eslint) (pull_request) Successful in 15s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 57s
Test / Python Tests (pytest) (pull_request) Successful in 1m28s
Tried on the real BGW320 (firmware 6.40.4): posting the Restart form only
answers 302 to /cgi-bin/restarting.ha, and the modem keeps running (its
uptime carried on). Loading that page, as a browser does, starts the
restart ("restarting ... approximately 155 seconds"). restart() now
follows the redirect and only reports 'accepted' when that page says it
is restarting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 20:34:50 -05:00
jaredandClaude Opus 5.5 c2ff072eb2 Restart modem button: silence Gandalf and Uptime Kuma, restart, end both
Lint / Python (flake8) (push) Failing after 15s
Lint / JS (eslint) (push) Successful in 1m43s
Lint / Deploy (push) Skipped
Test / Python Tests (pytest) (push) Failing after 21s
Security / Python Security (bandit) (push) Successful in 2m3s
Lint / Python (flake8) (pull_request) Failing after 15s
Lint / JS (eslint) (pull_request) Failing after 15s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 2m17s
Lint / Notify on failure (push) Successful in 3s
Test / Python Tests (pytest) (pull_request) Successful in 2m58s
Admins get a Restart modem button on the WAN / Fiber panel (needs
modem.access_code). After a confirmation it:

1. switches an Uptime Kuma maintenance window on (new kuma.py: logs in
   over Kuma 2's Socket.IO API and resumes/pauses one pre-made manual
   window, uptime_kuma.maintenance_id); if Kuma fails it stops and asks
   whether to go on without it,
2. starts Gandalf maintenance (global suppression, 20 minutes at most),
3. posts the gateway's Restart Device form (ModemAdmin.restart), undoing
   1 and 2 if that fails.

The monitor waits for the modem to answer with its WAN up and an uptime
showing it restarted, then switches Kuma off, ends the suppression and
records how long it took; after 30 minutes it ends both regardless. The
panel shows the restart in progress and the result for an hour.

New dependency: python-socketio[client].

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 19:27:25 -05:00
jared f097e4882c Merge pull request 'Modem: read uptime again, firewall drop counts, AT&T speed tests' (#52) from modem-extras into main
Lint / JS (eslint) (push) Successful in 9s
Lint / Python (flake8) (push) Successful in 53s
Lint / Notify on failure (push) Skipped
Security / Python Security (bandit) (push) Successful in 59s
Lint / Deploy (push) Successful in 11s
Test / Python Tests (pytest) (push) Successful in 49s
2026-11-19 18:52:08 -05:00
jaredandClaude Opus 5.5 0370358363 Modem: read uptime again, firewall drop counts, AT&T speed tests
Lint / JS (eslint) (push) Successful in 24s
Lint / Python (flake8) (push) Successful in 1m2s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 1m26s
Test / Python Tests (pytest) (push) Successful in 1m57s
Lint / Python (flake8) (pull_request) Successful in 1m25s
Lint / JS (eslint) (pull_request) Successful in 24s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 1m9s
Test / Python Tests (pytest) (pull_request) Successful in 1m14s
- Uptime: firmware 6.40 shows "Time Since Last Reboot" as D:HH:MM:SS,
  which the parser (expecting seconds) read as nothing, so uptime was
  blank and restart detection never fired. Both formats are read now.
- Firewall drops (logged in, every 5 minutes): the Logs page counted
  per hour for a day, split into incoming connections over the
  firmware's fixed flood limit (50 per second) and invalid packets. A
  lower bound, since the log keeps one entry per 2 s. Panel only.
- The modem's own speed test history (logged in, every 30 minutes):
  latest down and up results on the panel.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 18:50:37 -05:00
jared 2a9111d8e8 Merge pull request 'Modem access code: NAT sessions, and a ping from the modem when the internet is slow' (#51) from modem-login into main
Lint / JS (eslint) (push) Successful in 13s
Lint / Python (flake8) (push) Successful in 1m1s
Lint / Notify on failure (push) Skipped
Security / Python Security (bandit) (push) Successful in 57s
Lint / Deploy (push) Successful in 6s
Test / Python Tests (pytest) (push) Successful in 54s
2026-11-19 18:38:54 -05:00
jaredandClaude Opus 5.5 de75f71bf8 Modem access code: NAT sessions, and a ping from the modem when slow
Lint / JS (eslint) (push) Successful in 1m11s
Lint / Python (flake8) (push) Successful in 1m55s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 54s
Lint / Python (flake8) (pull_request) Successful in 53s
Test / Python Tests (pytest) (push) Successful in 1m5s
Lint / JS (eslint) (pull_request) Successful in 11s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 55s
Test / Python Tests (pytest) (pull_request) Successful in 1m3s
With modem.access_code in config.json, the monitor logs in to the
BGW320 (md5 of code + nonce, as its own page does) for two read-only
checks, stored in monitor_state.modem_checks:

- NAT sessions in use every 30 minutes; Needs attention warns at 80%.
- While UniFi says the internet is slow, a ping from the modem itself
  to 1.1.1.1 (at most every 10 minutes). Compared with UniFi's latency
  through the modem, it says whether the delay is past the modem (the
  provider), in the modem, or gone. The Internet line in Needs
  attention and the WAN / Fiber panel show the result.

A refused code is reported and not retried for 30 minutes; other
failures pause the checks for 10. The unauthenticated status polling
is unchanged. Tested against the real gateway and a fake one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 18:32:31 -05:00
jared 2f4cb05f67 Merge pull request 'docs: replace the old "Large1" port-label examples' (#50) from docs/large1-leftovers into main
Lint / JS (eslint) (push) Successful in 9s
Lint / Python (flake8) (push) Successful in 54s
Lint / Notify on failure (push) Skipped
Security / Python Security (bandit) (push) Successful in 1m3s
Test / Python Tests (pytest) (push) Successful in 1m43s
Lint / Deploy (push) Successful in 5s
2026-11-18 16:18:30 -05:00
jaredandClaude Opus 5.5 fa108b2a1e docs: remove large1 leftovers; no node names for HA guests
Lint / JS (eslint) (push) Successful in 10s
Lint / Python (flake8) (push) Successful in 43s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 1m44s
Test / Python Tests (pytest) (push) Successful in 1m27s
Lint / JS (eslint) (pull_request) Successful in 8s
Lint / Python (flake8) (pull_request) Successful in 1m9s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 1m9s
Test / Python Tests (pytest) (pull_request) Successful in 1m12s
large1 was renamed to compute-storage-02 on 2026-11-18. Updates text that still
named large1. HA guests no longer carry a fixed node (they move between nodes).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 16:15:31 -05:00
jared ff425dbaee Merge pull request 'Rename large1 to compute-storage-02 (README, placeholders, docstring)' (#49) from rename/large1-to-compute-storage-02 into main
Lint / JS (eslint) (push) Successful in 11s
Lint / Python (flake8) (push) Successful in 1m4s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Successful in 16s
Security / Python Security (bandit) (push) Successful in 1m11s
Test / Python Tests (pytest) (push) Successful in 1m31s
2026-11-18 15:41:08 -05:00
jaredandClaude Opus 5.5 99dee62098 Rename large1 to compute-storage-02 (README, placeholders, docstring)
Lint / Python (flake8) (push) Successful in 1m24s
Lint / JS (eslint) (push) Successful in 13s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 1m25s
Test / Python Tests (pytest) (push) Successful in 1m35s
Lint / JS (eslint) (pull_request) Successful in 9s
Lint / Python (flake8) (pull_request) Successful in 59s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 1m14s
Test / Python Tests (pytest) (pull_request) Successful in 1m8s
Text only. Prod config.json and DB history are updated during the window.
Test fixtures keep large1 on purpose.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 14:30:56 -05:00
jared 4ad331ddfe Merge pull request 'Polish: phone alert cards, quieter badges, footer, small layout fixes' (#48) from polish into main
Lint / JS (eslint) (push) Successful in 8s
Lint / Python (flake8) (push) Successful in 1m2s
Lint / Notify on failure (push) Skipped
Security / Python Security (bandit) (push) Successful in 1m17s
Lint / Deploy (push) Successful in 12s
Test / Python Tests (pytest) (push) Successful in 1m28s
2026-11-18 01:49:34 -05:00
jaredandClaude Opus 5.5 7da4cc26e2 Polish: phone alert cards, quieter badges, footer, small layout fixes
Lint / JS (eslint) (push) Successful in 14s
Lint / Python (flake8) (push) Successful in 1m17s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Lint / Python (flake8) (pull_request) Successful in 57s
Security / Python Security (bandit) (push) Successful in 1m0s
Test / Python Tests (pytest) (push) Successful in 1m56s
Security / Python Security (bandit) (pull_request) Successful in 1m52s
Lint / JS (eslint) (pull_request) Successful in 8s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Test / Python Tests (pytest) (pull_request) Successful in 1m25s
- Active alerts on the dashboard become one card per alert on phones
  instead of a table squeezed to a letter per line.
- Section count badges are neutral; red is kept for active alerts and
  tickets to follow up, so "History 0" no longer looks like a problem.
- Host cards only tag the data source when it isn't node_exporter
  (a ping-only host), instead of "prometheus" on every card.
- WAN / Fiber: the BGW320 doesn't report uptime here, so show the last
  restart instead of a permanent "Uptime -".
- Footer: CFG and HELP read like the hints beside them rather than grey
  browser buttons, and the footer sits at the bottom of short pages.
- Wider alert filter and suppression name fields so their hints fit.
- "just now" instead of a negative age when a clock runs a little ahead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 01:36:20 -05:00
22 changed files with 1512 additions and 62 deletions

No files matched your search

+47 -8
View File
@@ -92,7 +92,7 @@ Two processes share a MariaDB database:
| Host | Prometheus Instance |
|---|---|
| large1 | 10.10.10.2:9100 |
| compute-storage-02 | 10.10.10.2:9100 |
| compute-storage-01 | 10.10.10.4:9100 |
| monitor-01 | 10.10.10.8:9100 |
| monitor-02 | 10.10.10.9:9100 |
@@ -118,7 +118,7 @@ Ping-only (no node_exporter): **pbs** (10.10.10.3)
Served by `GET /api/attention` (`attention.py`) and refreshed with the rest of the page.
- Device search (see Device pages below)
- Active alerts table with severity, target, consecutive failures, ticket link
- WAN / Fiber panel: WAN and PON state, Rx/Tx power, optics temperature, throughput, errors, gateway uptime and last restart, a 24 hour internet health strip from UniFi, 7-day Rx trend, recent WAN outages
- WAN / Fiber panel: WAN and PON state, Rx/Tx power, optics temperature, throughput, errors, gateway uptime and last restart (just the last restart when the gateway doesn't report uptime), a 24 hour internet health strip from UniFi, 7-day Rx trend, recent WAN outages
- Hosts with per-NIC link state; NICs marked unused fold into a "+N unused" line
- UniFi devices as a compact list (name and model; offline ones turn red with a suppress button)
- Recently resolved alerts, folded until opened
@@ -136,7 +136,7 @@ holds every page and Start maintenance.
Every alert Gandalf raised in the last 24 hours to 90 days, including ones that cleared before anyone
saw them: when each started, how long it lasted, failed checks, and its ticket.
- Filter by device (name, port or interface), alert type, severity, and open or cleared. Filters are
kept in the URL, so `/events?target=large1&range=90d` is a shareable link; device pages and tickets
kept in the URL, so `/events?target=compute-storage-02&range=90d` is a shareable link; device pages and tickets
link to their own history this way.
- **Recurring** lists the same alert on the same target more than once in the range, worst first, since
a flapping cable or optic shows up here before anyone notices the tickets.
@@ -227,7 +227,7 @@ servers along the bottom. Power gear (PDU, UPS, power strip) sits in a rail on t
named "Patch panel port 1" (the device beyond the panel is unknown); a stub that already has a name
keeps it and adds "via Patch panel port 2". The cable details show where the cutsheet cable goes.
- **Port names**: a ⚠ marks a port named after one device with a different one plugged in (a port
called "Large1 Management" with the AP on it). Generic names ("Port 4", "SFP+ 2", "Uplink") and
called "compute-storage-02 Management" with the AP on it). Generic names ("Port 4", "SFP+ 2", "Uplink") and
"Downlink to X" with X on the other end never flag.
- **Alerts**: a device with active alerts (not suppressed) gets a ring and a count in its worst
severity colour, and the NIC it is about is marked.
@@ -498,6 +498,10 @@ Those limits are wide (warning at -27.9 dBm on a link that normally sits near -1
compares each reading to the link's own 7-day median to catch a damaged connector earlier. It stays quiet
until `rx_drift_min_samples` readings exist (about a day at a 5 minute poll).
Uptime comes from the System Information page without a login. Firmware 6.40 shows it as D:HH:MM:SS
(older firmware sent seconds; both are read). Uptime going backwards between polls records a restart, and
the panel shows it (*restarted ... by ...*).
Outages can be over between polls: two real ones lasted about two minutes against a 5 minute poll.
So WAN outages and modem restarts also come from the UniFi system log, and a drop in the modem's uptime
records a restart even if nothing logged it. A single short outage only appears in the panel's history.
@@ -511,6 +515,38 @@ periods had loss or downtime, *high latency* (amber) when 3 or more had any prob
the last hour. These are dashboard lines, not tickets, and they go once the hour is clean. A copy older
than 30 minutes is ignored.
**With `modem.access_code` set**, the monitor also logs in to the gateway (`ModemAdmin` in `modem.py`)
for read-only checks, stored in `monitor_state.modem_checks`:
- **NAT sessions** every 30 minutes, from the NAT Table page. Needs attention warns when 80% of the
gateway's sessions (32,767 on a BGW320) are in use, since new connections stall once it is full.
- **A ping from the gateway itself** to 1.1.1.1, from the Troubleshoot page, whenever UniFi flags the
current 5 minutes as slow or measures 150 ms or more (at most every 10 minutes). Compared with UniFi's
latency through the gateway at the same time, it says where the delay is: *past the modem* (its own
ping is slow too, so the provider), *in the modem* (its ping is fine but ours through it is slow), or
*fine by then*. The result goes in the detail of the Internet line in Needs attention and on the
WAN / Fiber panel.
- **Firewall drops** every 5 minutes, from the Logs page, counted per hour for a day. Most are *FLOOD
limit 50pps*: AT&T's firmware caps new incoming connections at 50 per second (visitors to hosted
services, peers), with no setting to change it. The log records at most one drop every 2 seconds and
keeps about ten minutes, so the counts are a lower bound. Shown on the panel only.
- **The gateway's own speed tests** (AT&T runs them from the modem), read with the NAT check; the latest
down and up results are shown on the panel as a reference for what the line itself delivers.
**Restart modem** (admins, WAN / Fiber panel, needs `modem.access_code`): after a confirmation it
1. switches the `uptime_kuma` maintenance window on (if that fails it stops and asks whether to go on
without it),
2. starts Gandalf maintenance (a global suppression for up to 20 minutes, shown in the banner),
3. posts the gateway's Restart Device form. If that fails, steps 1 and 2 are undone.
The monitor then waits for the gateway to answer with its WAN up and an uptime showing it restarted
(at least 3 minutes later), switches the Kuma window off, ends the suppression, and records how long it
took (`monitor_state.modem_restart`). After 30 minutes it ends both regardless. One restart at a time.
The login posts md5(access code + nonce), as the gateway's own page does, so the code itself is never sent.
The session is kept between checks and renewed when it expires. A refused code is reported in Needs
attention and not tried again for 30 minutes, and any other failure pauses the checks for 10 minutes, so
the gateway is never retried in a loop. The status polling above doesn't log in and is unaffected.
**One ticket per incident.** The modem and WAN reports from one check cycle open a single ticket, led by
the most urgent (a P2 "WAN modem unreachable" leads, and an RPDU power-cycle seen in the same cycle is listed
under "Also seen in this incident"). A report that comes while the modem has an open alert, or one that
@@ -588,7 +624,7 @@ Shared by both processes. Located in the working directory (`/var/www/html/prod/
"trusted_proxies": ["10.10.10.27"]
},
"hosts": [
{ "name": "large1", "prometheus_instance": "10.10.10.2:9100" },
{ "name": "compute-storage-02", "prometheus_instance": "10.10.10.2:9100" },
{ "name": "compute-storage-01", "prometheus_instance": "10.10.10.4:9100" },
{ "name": "monitor-01", "prometheus_instance": "10.10.10.8:9100" },
{ "name": "monitor-02", "prometheus_instance": "10.10.10.9:9100" },
@@ -654,6 +690,8 @@ Shared by both processes. Located in the working directory (`/var/www/html/prod/
| `modem.failure_threshold` | Consecutive failures before ticketing a modem alert (default: same as `failure_threshold`). Raise it if gateway reboots or automatic firmware updates cut tickets. |
| `modem.rx_drift_db` | Rx drop below the 7-day median that raises `fiber_rx_drift` (default: 3.0) |
| `modem.rx_drift_min_samples` | Readings required before drift alerts start (default: 288) |
| `uptime_kuma.url`, `.token`, `.maintenance_id` | Optional. Lets the Restart modem button switch an Uptime Kuma maintenance window on and off (Kuma 2.x, over its Socket.IO API). `token` is a Kuma session token, which works with a two-factor account and stops working if the Kuma password changes; `username` and `password` can be used instead for an account without two-factor login. Create the window once in Kuma with the *manual* strategy and the monitors a modem restart takes down, pause it, and put its id here. |
| `modem.access_code` | Optional. The gateway's Device Access Code (printed on its label). With it, Gandalf logs in to read NAT session counts and to ping from the gateway when the internet is slow (see WAN modem notes). Keep it only in `config.json`, which is git-ignored and mode 600. |
---
@@ -838,7 +876,8 @@ curl http://10.10.10.48:9090/api/v1/query?query=up | jq '.data.result[] | select
gandalf/
├── app.py # Flask web app (routes, auth, API endpoints)
├── monitor.py # Background daemon (Prometheus, UniFi, Pulse, alert logic)
├── modem.py # Read-only AT&T gateway status page client and parser
├── kuma.py # Switches one Uptime Kuma maintenance window on/off (Restart modem)
├── modem.py # AT&T gateway client: status pages, plus optional logged-in checks and restart
├── devices.py # Device inventory for search and device pages
├── topology.py # The topology graph: devices, cables, depth, last-seen links
├── cutsheet.py # Cutsheet validation, path tracing, plan-vs-actual checks, CSV
@@ -920,7 +959,7 @@ Take LED colors from the model's quick start guide. See the comment at the top o
`recovery_notified` once the recovery note is on its ticket (column added automatically at monitor startup)
- `suppression_rules`: `active=FALSE` when removed; `expires_at` checked at query time
- `monitor_state`: key/value store; `interface_baseline`, `link_stats`, and `modem_stats` are JSON blobs
- `monitor_state` also holds `wan_history` (WAN outages, modem restarts), `wan_quality` (UniFi's 5 minute WAN health timeline) and `unifi_log_cursor` (how far the UniFi log has been read)
- `monitor_state` also holds `wan_history` (WAN outages, modem restarts), `wan_quality` (UniFi's 5 minute WAN health timeline), `modem_checks` (NAT sessions, the latest ping from the gateway, hourly firewall drops and its speed test results) and `unifi_log_cursor` (how far the UniFi log has been read)
- Database sessions run in UTC (`db.py`), whatever the server's time zone
- `modem_samples`: optical readings per poll, kept 90 days; created automatically at monitor startup
- `cutsheet_items`, `cutsheet_cables`, `cutsheet_port_labels`, `cutsheet_history`: cutsheet records and
@@ -976,7 +1015,7 @@ Branch protection is enabled on `main`: both lint jobs must pass before any PR c
Tests live in `tests/`:
- `test_diagnose.py`: `DiagnosticsRunner` static methods (`build_ssh_command`, `parse_output`,
`parse_sysfs_stats`, `parse_ethtool`, and variants)
- `test_modem.py`: gateway page parsing, optical status, and WAN state
- `test_modem.py`: gateway page parsing, optical status, WAN state, and the logged-in checks against a fake gateway
- `test_monitor.py`: alert logic for dead hosts, Prometheus and Pulse outages, and modem and WAN log events
- `test_devices.py`: device inventory behind search and device pages
- `test_app.py`: page and API routes
+66 -1
View File
@@ -27,6 +27,8 @@ import cutsheet
import db
import devices
import diagnose
import kuma
import modem as modem_api
import topology
from monitor import PulseClient, UnifiClient
@@ -556,10 +558,72 @@ def api_modem():
data['history'] = []
data['wan_history'] = _state_json('wan_history')
data['wan_quality'] = _state_json('wan_quality') or None
data['modem_checks'] = _state_json('modem_checks') or None
data['modem_restart'] = _state_json('modem_restart') or None
data['can_restart'] = bool((_config().get('modem') or {}).get('access_code'))
data['configured'] = True
return jsonify(data)
# How long the silence lasts if the monitor never sees the modem come back to end it sooner
MODEM_RESTART_SILENCE_MIN = 20
@app.route('/api/modem/restart', methods=['POST'])
@require_auth
@require_admin
def api_modem_restart():
"""Restart the AT&T gateway, with alerts silenced first.
Body: {"confirm": true} and, after a Kuma failure the admin chose to accept,
"without_kuma": true. Order: Uptime Kuma maintenance on, Gandalf maintenance on, then
the restart; the first two are undone if the restart can't be sent. The monitor ends
both once the modem is back (monitor._finish_modem_restart).
"""
body = _json_body()
if body.get('confirm') is not True:
return jsonify({'error': 'confirm must be true'}), 400
cfg = _config()
modem_cfg = cfg.get('modem') or {}
if not modem_cfg.get('url') or not modem_cfg.get('access_code'):
return jsonify({'error': 'Set modem.access_code in config.json first'}), 400
pending = _state_json('modem_restart')
if pending and not pending.get('finished_at') and time.time() - pending.get('at', 0) < 1800:
return jsonify({'error': 'A restart is already in progress'}), 409
user = _get_user()['username']
kuma_cfg = cfg.get('uptime_kuma') or {}
kuma_on = False
if kuma.configured(kuma_cfg):
try:
kuma.set_maintenance(kuma_cfg, True)
kuma_on = True
except kuma.KumaError as e:
logger.warning(f'Uptime Kuma maintenance could not be started: {e}')
if not body.get('without_kuma'):
return jsonify({'error': f'Uptime Kuma: {e}', 'kuma_failed': True}), 502
sup_id = db.create_suppression('all', '', '', 'Modem restart from Gandalf', user,
MODEM_RESTART_SILENCE_MIN)
try:
result = modem_api.ModemAdmin(modem_cfg).restart()
except (modem_api.ModemUnreachable, modem_api.ModemDataInvalid, modem_api.ModemLoginFailed) as e:
logger.error(f'Modem restart failed: {e}')
db.deactivate_suppression(sup_id)
if kuma_on:
try:
kuma.set_maintenance(kuma_cfg, False)
except kuma.KumaError as ke:
logger.error(f'Uptime Kuma maintenance could not be ended: {ke}')
return jsonify({'error': f'The modem did not restart: {e}'}), 502
db.set_state('modem_restart', {'at': time.time(), 'by': user, 'result': result,
'suppression_id': sup_id, 'kuma': kuma_on, 'finished_at': None})
logger.warning(f'Modem restart requested by {user} ({result}); Kuma maintenance '
f'{"on" if kuma_on else "not set"}, suppression #{sup_id}')
return jsonify({'ok': True, 'result': result, 'kuma': kuma_on, 'suppression_id': sup_id})
@app.route('/api/devices')
@require_auth
def api_devices():
@@ -1360,7 +1424,8 @@ def _attention() -> dict:
logger.error(f'Cutsheet unavailable for the attention list: {e}')
sheet = {'items': (view or {}).get('items') or [], 'cables': (view or {}).get('cables') or []}
power = _topology_graph(events, sheet, feeds).get('power') or []
return attention.build(events, power, view, racks, _state_json('wan_quality'))
return attention.build(events, power, view, racks, _state_json('wan_quality'),
modem_checks=_state_json('modem_checks'))
def _topology_graph(events: list, sheet: dict, feeds: list) -> dict:
+43 -4
View File
@@ -115,7 +115,42 @@ def _ago(seconds: float) -> str:
return f'{minutes} min ago' if minutes < 60 else f'{round(minutes / 60)} h ago'
def _wan(quality: Optional[dict], now: float) -> List[dict]:
_NAT_FULL = 0.8 # share of the gateway's NAT sessions in use before it's worth a line
def _modem_ping_note(checks: Optional[dict], now: float) -> str:
"""One sentence on the latest ping from the modem itself, if it ran in the last hour."""
ping = (checks or {}).get('ping') or {}
if not ping or now - (ping.get('at') or 0) > _WAN_WINDOW_S:
return ''
when, avg, ours = _ago(now - ping['at']), ping.get('avg_ms'), ping.get('ours_ms')
theirs = (f'{round(avg)} ms' if avg is not None else 'no reply') + (
f', {ping["loss_pct"]}% lost' if ping.get('loss_pct') else '')
ours_txt = f'; ours through it {round(ours)} ms' if ours is not None else ''
verdict = {
'upstream': "so the delay is past the modem, on the provider's side",
'gateway': 'so the delay is in the modem, or between it and the router',
'clear': 'both fine by then',
}.get(ping.get('verdict'), '')
return f"Modem's own ping {when}: {theirs}{ours_txt}, {verdict}. "
def _modem_checks(checks: Optional[dict]) -> List[dict]:
out = []
nat = (checks or {}).get('nat') or {}
if nat.get('available') and nat.get('in_use', 0) / nat['available'] >= _NAT_FULL:
pct = round(100 * nat['in_use'] / nat['available'])
out.append(_entry('broken', 'warn', f'Modem NAT table {pct}% full',
'#modem-section', f'{nat["in_use"]:,} of {nat["available"]:,} sessions in use. '
'When it fills, new connections stall or fail.'))
error = (checks or {}).get('error') or {}
if error.get('login'):
out.append(_entry('tidy', 'warn', 'Modem login failed: check modem.access_code in config.json',
'#modem-section', 'Gandalf waits 30 minutes before trying again.'))
return out
def _wan(quality: Optional[dict], now: float, checks: Optional[dict] = None) -> List[dict]:
if not quality or now - (quality.get('updated') or 0) > _WAN_STALE_S:
return []
out = []
@@ -133,6 +168,7 @@ def _wan(quality: Optional[dict], now: float) -> List[dict]:
if day_bad else '')
if latencies:
detail += 'Now: ' + ', '.join(f'{k} {v} ms' for k, v in sorted(latencies.items())) + '. '
detail += _modem_ping_note(checks, now)
detail += 'See the 24 hour strip on the WAN / Fiber panel.'
lossy = recent['loss'] + recent['down']
if lossy >= _WAN_LOSS_PERIODS:
@@ -154,16 +190,19 @@ def _wan(quality: Optional[dict], now: float) -> List[dict]:
def build(events: List[dict], power: List[dict], cutsheet_view: Optional[dict],
rack_issues: Optional[Dict[str, dict]] = None, wan_quality: Optional[dict] = None,
now: Optional[float] = None) -> dict:
now: Optional[float] = None, modem_checks: Optional[dict] = None) -> dict:
"""{'broken': [entry], 'tidy': [entry]}, worst first within each group.
events: active alerts, annotated with 'is_suppressed'. power: topology.build_power output.
cutsheet_view: the cutsheet API view (cables with 'problem', suggestions, port_names,
labels), or None when the cutsheet couldn't be loaded. rack_issues: {rack: rack_layout()}.
wan_quality: modem.wan_quality() output as stored by the monitor. now: for tests.
wan_quality: modem.wan_quality() output as stored by the monitor. modem_checks: the
monitor's logged-in modem checks (NAT sessions, ping from the modem). now: for tests.
Entry: {group, severity: bad|warn|info, text, href, detail}.
"""
entries = _wan(wan_quality, time.time() if now is None else now) + _alerts(events) + _power(power)
now = time.time() if now is None else now
entries = (_wan(wan_quality, now, modem_checks) + _modem_checks(modem_checks)
+ _alerts(events) + _power(power))
if cutsheet_view is not None:
entries += _cutsheet(cutsheet_view)
entries += _rack(rack_issues or {})
+84
View File
@@ -0,0 +1,84 @@
{
"pulse": {
"url": "http://10.10.10.65:8080",
"api_key": "012b303a324152c509bf5ade6f942cfc21404f68662f01a17001cba9e4486049",
"worker_id": "1b11d1b5-4ed0-42df-a6af-8d57fffe1343",
"timeout": 45
},
"unifi": {
"controller": "https://10.10.10.1",
"api_key": "kyPfIsAVie3hwMD4Bc1MjAu8N7HVPIb8",
"site_id": "default"
},
"prometheus": {
"url": "http://10.10.10.48:9090"
},
"database": {
"host": "10.10.10.50",
"port": 3306,
"user": "gandalf",
"password": "Gandalf2026Lotus",
"name": "gandalf"
},
"ticket_api": {
"url": "http://10.10.10.45/create_ticket_api.php",
"api_key": "5acc5d3c647b84f7c6f59082ce4450ee772e2d1633238b960136f653d20c93af"
},
"ldap": {
"host": "10.10.10.39",
"port": 3890,
"bind_dn": "uid=gandalf,ou=people,dc=example,dc=com",
"bind_pw": "AZ-eUFRxamnlLEELuDy5-Mh6z-2cJbL_YJZnu64Tf0A",
"user_base": "ou=people,dc=example,dc=com",
"cache_dir": "/tmp/gandalf_avatars",
"cache_ttl": 3600
},
"auth": {
"allowed_groups": ["admin"]
},
"monitor": {
"poll_interval": 300,
"failure_threshold": 2,
"unifi_failure_threshold": 4,
"cluster_threshold": 3,
"ping_hosts": [],
"links_exclude_ips": ["10.10.10.29", "10.10.10.44", "10.10.10.3"]
},
"hosts": [
{
"name": "pbs",
"ip": "10.10.10.3",
"prometheus_instance": "10.10.10.3:9100"
},
{
"name": "large1",
"ip": "10.10.10.2",
"prometheus_instance": "10.10.10.2:9100"
},
{
"name": "compute-storage-01",
"ip": "10.10.10.4",
"prometheus_instance": "10.10.10.4:9100"
},
{
"name": "monitor-01",
"ip": "10.10.10.8",
"prometheus_instance": "10.10.10.8:9100"
},
{
"name": "monitor-02",
"ip": "10.10.10.9",
"prometheus_instance": "10.10.10.9:9100"
},
{
"name": "compute-storage-gpu-01",
"ip": "10.10.10.10",
"prometheus_instance": "10.10.10.10:9100"
},
{
"name": "storage-01",
"ip": "10.10.10.11",
"prometheus_instance": "10.10.10.11:9100"
}
]
}
+74
View File
@@ -0,0 +1,74 @@
"""Switch one Uptime Kuma maintenance window on and off (Kuma 2.x, over its Socket.IO API).
Kuma has no REST API for maintenance, so this signs in over the same socket its web UI
uses. It only ever resumes or pauses the one window named in config, which is created
once in Kuma ("manual" strategy, covering the monitors that a modem restart takes down):
"uptime_kuma": {"url": "http://10.10.10.25:3001", "token": "...", "maintenance_id": 7}
`token` is a Kuma session token (works with a two-factor account; it stops working if
the Kuma password changes). `username` and `password` are accepted instead for an
account without two-factor login. Nothing else in Kuma is read or changed.
"""
import logging
import threading
from typing import Callable, Optional
import socketio
logger = logging.getLogger('gandalf.kuma')
class KumaError(Exception):
"""Kuma couldn't be reached, refused the sign-in, or refused the change."""
def configured(cfg: Optional[dict]) -> bool:
cfg = cfg or {}
signed_in = bool(cfg.get('token')) or bool(cfg.get('username') and cfg.get('password'))
return bool(cfg.get('url') and cfg.get('maintenance_id')) and signed_in
def _sign_in(sio, cfg: dict, timeout: float) -> None:
if cfg.get('token'):
res = sio.call('loginByToken', cfg['token'], timeout=timeout) or {}
if not res.get('ok'):
raise KumaError('token rejected (Kuma password changed?)')
return
res = sio.call('login', {'username': cfg['username'], 'password': cfg['password'], 'token': ''},
timeout=timeout) or {}
if not res.get('ok'):
if res.get('tokenRequired'):
raise KumaError('the Kuma account has two-factor login: use uptime_kuma.token instead')
raise KumaError(f'login refused: {res.get("msg") or "no reason given"}')
def set_maintenance(cfg: dict, active: bool, client_factory: Optional[Callable] = None) -> None:
"""Resume (active=True) or pause the configured maintenance window. Raises KumaError."""
timeout = cfg.get('timeout', 10)
factory = client_factory or (lambda: socketio.Client(
reconnection=False, ssl_verify=cfg.get('verify_ssl', True)))
sio = factory()
# Kuma registers its event handlers only after it sends 'info', so anything emitted
# before that is silently dropped and the call times out.
ready = threading.Event()
sio.on('info', lambda *args: ready.set())
try:
sio.connect(cfg['url'], transports=['websocket'], wait_timeout=timeout)
if not ready.wait(timeout):
raise KumaError('Kuma not ready (no info event after connecting)')
_sign_in(sio, cfg, timeout)
event = 'resumeMaintenance' if active else 'pauseMaintenance'
res = sio.call(event, int(cfg['maintenance_id']), timeout=timeout) or {}
if not res.get('ok'):
raise KumaError(f'{event} refused: {res.get("msg") or "no reason given"}')
logger.info(f'Uptime Kuma maintenance {cfg["maintenance_id"]} {"on" if active else "off"}')
except KumaError:
raise
except Exception as e: # socketio raises its own errors and plain timeouts
raise KumaError(f'{type(e).__name__}: {e}') from e
finally:
try:
sio.disconnect()
except Exception: # nosec B110 - nothing left to clean up
pass
+246 -5
View File
@@ -1,13 +1,20 @@
"""Read-only client for AT&T BGW-series fiber gateways (tested on BGW320-505).
"""Client for AT&T BGW-series fiber gateways (tested on BGW320-505).
The gateway has no API, but its status pages are served without a login, so
this scrapes them over HTTP. Only GET requests to the fixed status pages below
are ever made: nothing here can change gateway settings.
ModemClient scrapes them over HTTP with GET requests only: it cannot change
anything on the gateway.
With an access code configured, ModemAdmin (at the end) also logs in to read the
NAT session counts, the firewall log and speed test history, and to run the
Troubleshoot page's ping from the gateway. None of that changes a setting. Its
restart() is the one call that does something to the gateway, and it runs only
when an admin presses Restart modem.
Units come from the gateway's own help text, checked against live readings:
Rx/Tx power tenths of a dBm (-169 -> -16.9 dBm)
Temperature degrees C
Uptime seconds (despite the help text claiming D:H:M:S)
Uptime D:HH:MM:SS on firmware 6.40 (older firmware sent plain seconds;
both are accepted)
Deliberately not used, because the page disagrees with itself:
Vcc / Tx Bias readings values do not match the documented scales, so only
@@ -16,9 +23,11 @@ Deliberately not used, because the page disagrees with itself:
Status" is the reliable signal
Last Change does not count up, despite the help text
"""
import hashlib
import html
import logging
import re
import time
from datetime import datetime, timezone
from typing import Dict, List, Optional
@@ -136,6 +145,18 @@ def _to_int(raw: Optional[str]) -> Optional[int]:
return None
def parse_uptime(raw: Optional[str]) -> Optional[int]:
"""Seconds since the gateway restarted, from 'D:HH:MM:SS' (e.g. 03:06:29:34) or plain seconds."""
raw = (raw or '').strip()
if re.fullmatch(r'\d+', raw):
return int(raw)
m = re.fullmatch(r'(\d+):(\d{1,2}):(\d{1,2}):(\d{1,2})', raw)
if not m:
return None
d, h, mi, sec = (int(x) for x in m.groups())
return ((d * 24 + h) * 60 + mi) * 60 + sec
def wan_problems(data: dict) -> List[str]:
"""Return human-readable reasons the WAN is down; empty when it looks healthy."""
problems = []
@@ -164,7 +185,7 @@ def build_status(fiber_page: str, broadband_page: str, sysinfo_page: str = '') -
data = {
'model': sysinfo.get('Model Number', ''),
'firmware': sysinfo.get('Software Version', ''),
'uptime_s': _to_int(sysinfo.get('Time Since Last Reboot')),
'uptime_s': parse_uptime(sysinfo.get('Time Since Last Reboot')),
'broadband_state': broadband.get('Broadband Connection', ''),
'source': broadband.get('Broadband Connection Source', ''),
'wan_ip': broadband.get('Broadband IPv4 Address', ''),
@@ -360,3 +381,223 @@ def quality_counts(periods: List[dict], since: float) -> Dict[str, int]:
if p['ts'] >= since and p['state'] in counts:
counts[p['state']] += 1
return counts
# ---------------------------------------------------------------------------
# Logged-in checks, used only when an access code is configured. Two pages are
# read: the NAT table's session counts, and the Troubleshoot page's ping test,
# which runs a ping from the gateway itself. Neither changes any setting.
# ---------------------------------------------------------------------------
_LOGIN_PAGE = '/cgi-bin/login.ha'
_NAT_PAGE = '/cgi-bin/nattable.ha'
_DIAG_PAGE = '/cgi-bin/diag.ha'
_LOGS_PAGE = '/cgi-bin/logs.ha'
_SPEED_PAGE = '/cgi-bin/speed.ha'
_RESTART_PAGE = '/cgi-bin/restart.ha'
_RESTARTING_PAGE = '/cgi-bin/restarting.ha'
# Firewall log row: number, ISO time (gateway's local clock), source, destination, protocol, reason
_FW_ROW_RE = re.compile(r'<tr[^>]*>\s*<td[^>]*>\s*\d+\s*</td>\s*<td>\s*([^<]+?)\s*</td>\s*<td>\s*([^<]*?)\s*</td>'
r'\s*<td>\s*([^<]*?)\s*</td>\s*<td>\s*([^<]*?)\s*</td>\s*<td>\s*([^<]*?)\s*</td>\s*</tr>', re.I)
_CELL_RE = re.compile(r'<td\b[^>]*>(.*?)</td>', re.I | re.S)
_NONCE_RE = re.compile(r'name="nonce"\s+value="([0-9a-f]+)"', re.I)
_PROGRESS_RE = re.compile(r'<textarea[^>]*name="ProgressWindow"[^>]*>(.*?)</textarea>', re.I | re.S)
_PING_STATS_RE = re.compile(r'(\d+) packets transmitted, (\d+) (?:packets )?received')
_PING_RTT_RE = re.compile(r'min/avg/max\S* = ([\d.]+)/([\d.]+)/([\d.]+)')
# After a failed login, leave the gateway alone for this long (it may lock the account)
LOGIN_BACKOFF_S = 1800
class ModemLoginFailed(Exception):
"""The access code was refused, or the login page wasn't what we expect."""
def login_hash(code: str, nonce: str) -> str:
"""What the login page's script posts: md5 of the access code followed by the nonce."""
# The gateway's login protocol dictates md5; it isn't used to protect anything here
return hashlib.md5((code + nonce).encode(), usedforsecurity=False).hexdigest() # nosec B303 B324
def is_login_page(page: str) -> bool:
return 'Access Code Required' in page or bool(re.search(r'<title>\s*Login', page, re.I))
def parse_nat_summary(page: str) -> Optional[Dict[str, int]]:
"""{'in_use', 'available'} from the NAT table page, or None if the counts are missing."""
rows = {_TAG_RE.sub('', k).strip(): _TAG_RE.sub('', v).strip() for k, v in _ROW_RE.findall(page)}
try:
return {'in_use': int(rows['Total sessions in use']),
'available': int(rows['Total sessions available'])}
except (KeyError, ValueError):
return None
def parse_ping_output(text: str) -> Optional[dict]:
"""{'sent', 'received', 'loss_pct', 'min_ms', 'avg_ms', 'max_ms'} from busybox ping output,
or None until the test has finished. Times are None when every packet was lost."""
stats = _PING_STATS_RE.search(text or '')
if not stats:
return None
sent, received = int(stats.group(1)), int(stats.group(2))
rtt = _PING_RTT_RE.search(text)
lo, avg, hi = (float(x) for x in rtt.groups()) if rtt else (None, None, None)
return {'sent': sent, 'received': received,
'loss_pct': round(100 * (sent - received) / sent) if sent else 100,
'min_ms': lo, 'avg_ms': avg, 'max_ms': hi}
def parse_firewall_log(page: str) -> List[dict]:
"""Rows of the gateway's firewall log (Logs page): {'ts', 'src', 'dst', 'proto', 'reason'}.
'ts' is the gateway's local time as ISO text, so rows sort by it."""
rows = [dict(zip(('ts', 'src', 'dst', 'proto', 'reason'), m.groups())) for m in _FW_ROW_RE.finditer(page)]
return sorted((r for r in rows if re.match(r'\d{4}-\d\d-\d\dT', r['ts'])), key=lambda r: r['ts'])
def firewall_kind(reason: str) -> str:
"""'flood' (new connections over the gateway's fixed rate limit), 'invalid', or 'other'."""
reason = reason.lower()
return 'flood' if 'flood' in reason else 'invalid' if 'invalid' in reason else 'other'
def parse_speed_tests(page: str) -> List[dict]:
"""The gateway's own speed test history, newest first:
{'at' (as the page shows it), 'direction', 'mbps', 'latency_ms', 'ok'}."""
i = page.find('Test History')
out = []
for row in re.findall(r'<tr\b[^>]*>(.*?)</tr>', page[i:] if i >= 0 else '', re.I | re.S):
cells = [_TAG_RE.sub('', c).strip() for c in _CELL_RE.findall(row)]
if len(cells) < 6 or cells[1] not in ('upstream', 'downstream'):
continue
try:
out.append({'at': cells[0], 'direction': cells[1], 'mbps': round(float(cells[2]), 1),
'latency_ms': round(float(cells[4]), 1), 'ok': cells[5].lower() == 'success'})
except ValueError:
continue
return out
class ModemAdmin:
"""Logged-in session on the gateway, kept between checks and renewed when it expires.
Logs in at most once per call, and not at all for LOGIN_BACKOFF_S after a refusal.
Raises ModemUnreachable, ModemDataInvalid or ModemLoginFailed.
"""
def __init__(self, cfg: dict, clock=None, sleep=None):
self.base_url = cfg['url'].rstrip('/')
self._code = cfg['access_code']
self.timeout = cfg.get('timeout', 10)
self.session = requests.Session()
self._clock = clock or time.time
self._sleep = sleep or time.sleep
self.login_failed_at: Optional[float] = None
def _request(self, method: str, path: str, timeout: Optional[float] = None, **kw) -> requests.Response:
try:
return self.session.request(method, f'{self.base_url}{path}', allow_redirects=False,
timeout=timeout or self.timeout, **kw)
except requests.RequestException as e:
raise ModemUnreachable(f'{path}: {e}') from e
def _login(self) -> None:
if self.login_failed_at and self._clock() - self.login_failed_at < LOGIN_BACKOFF_S:
raise ModemLoginFailed('waiting after a refused login')
page = self._request('GET', _LOGIN_PAGE).text
nonce = _NONCE_RE.search(page)
if not nonce: # the first visit only sets the session cookie
nonce = _NONCE_RE.search(self._request('GET', _LOGIN_PAGE).text)
if not nonce:
raise ModemDataInvalid('login page has no nonce')
resp = self._request('POST', _LOGIN_PAGE, data={
'nonce': nonce.group(1), 'password': '*' * len(self._code), 'Continue': 'Continue',
'hashpassword': login_hash(self._code, nonce.group(1)),
})
if resp.status_code != 302 or is_login_page(resp.text):
self.login_failed_at = self._clock()
raise ModemLoginFailed('the gateway refused the access code')
self.login_failed_at = None
def _page(self, path: str, timeout: Optional[float] = None) -> str:
"""A protected page, logging in first if the session has expired."""
resp = self._request('GET', path, timeout)
if resp.status_code != 200 or is_login_page(resp.text):
self._login()
resp = self._request('GET', path, timeout)
if resp.status_code != 200 or is_login_page(resp.text):
raise ModemDataInvalid(f'{path} returned HTTP {resp.status_code} after logging in')
return resp.text
def nat_sessions(self) -> Dict[str, int]:
# The page lists every session too, so it can be large (hundreds of KB)
counts = parse_nat_summary(self._page(_NAT_PAGE, timeout=max(self.timeout, 30)))
if counts is None:
raise ModemDataInvalid('NAT table page has no session counts')
return counts
def firewall_log(self) -> List[dict]:
return parse_firewall_log(self._page(_LOGS_PAGE))
def speed_tests(self) -> List[dict]:
return parse_speed_tests(self._page(_SPEED_PAGE))
def restart(self) -> str:
"""Restart the gateway from its Restart Device page: the one change this client makes,
and only when an admin asks.
Two steps, as in a browser: posting the form only answers with a redirect to
restarting.ha, and loading that page is what starts the restart (firmware 6.40; a
post alone leaves the gateway running). Returns 'accepted' once that page says it is
restarting, or 'unconfirmed' when the gateway dropped the connection first.
"""
nonce = _NONCE_RE.search(self._page(_RESTART_PAGE))
if not nonce:
raise ModemDataInvalid('Restart page has no nonce')
timeout = max(self.timeout, 30)
try:
resp = self.session.post(f'{self.base_url}{_RESTART_PAGE}', allow_redirects=False,
timeout=timeout, data={'nonce': nonce.group(1), 'Restart': 'Restart'})
if resp.status_code == 302 and _RESTARTING_PAGE in (resp.headers.get('Location') or ''):
resp = self.session.get(f'{self.base_url}{_RESTARTING_PAGE}', allow_redirects=False,
timeout=timeout)
except (requests.ConnectionError, requests.Timeout):
return 'unconfirmed'
if resp.status_code != 200 or is_login_page(resp.text) or 'restarting' not in resp.text.lower():
raise ModemDataInvalid(f'Restart was not started (HTTP {resp.status_code})')
return 'accepted'
def ping(self, target: str, wait_s: float = 30) -> dict:
"""Ping *target* from the gateway (its Troubleshoot page; 5 packets) and return the result."""
nonce = _NONCE_RE.search(self._page(_DIAG_PAGE))
if not nonce:
raise ModemDataInvalid('Troubleshoot page has no nonce')
resp = self._request('POST', _DIAG_PAGE, timeout=max(self.timeout, 30), data={
'nonce': nonce.group(1), 'WebAddress': target, 'protopref': 'IPv4', 'Ping': 'Ping'})
deadline = self._clock() + wait_s
page = resp.text
while True:
window = _PROGRESS_RE.search(page)
result = parse_ping_output(html.unescape(window.group(1)) if window else '')
if result is not None:
return dict(result, target=target)
if self._clock() >= deadline:
raise ModemDataInvalid(f'ping to {target} from the gateway did not finish')
self._sleep(3)
page = self._page(_DIAG_PAGE)
SLOW_MS = 150 # a ping slower than this, on a link that normally answers in ~15 ms, is "slow"
def ping_verdict(gateway: dict, ours_ms: Optional[float]) -> str:
"""Where the delay is, from the gateway's own ping and ours through it at the same time.
'upstream': the gateway's ping is slow or lossy too, so it's past the gateway (the ISP).
'gateway': the gateway's ping is fine but ours through it is slow, so the delay is in
the gateway or between it and the router.
'clear': both fine; the slow spell had passed by the time of the test.
"""
avg = gateway.get('avg_ms')
if avg is None or avg >= SLOW_MS or (gateway.get('loss_pct') or 0) >= 20:
return 'upstream'
if ours_ms is not None and ours_ms >= SLOW_MS:
return 'gateway'
return 'clear'
+144 -1
View File
@@ -24,6 +24,7 @@ from urllib3.exceptions import InsecureRequestWarning
import db
import cutsheet
import devices
import kuma
import modem as modem_api
import ticket_text
@@ -914,7 +915,7 @@ class LinkStatsCollector:
"""
Collect full link stats for all Prometheus-monitored hosts.
*instance_map*: ``{'10.10.10.2:9100': 'large1', ...}``
*instance_map*: ``{'10.10.10.2:9100': 'compute-storage-02', ...}``
Returns a dict suitable for ``db.set_state('link_stats', ...)``.
"""
@@ -1111,6 +1112,9 @@ class NetworkMonitor:
modem_cfg = self.cfg.get('modem') or {}
self.modem = modem_api.ModemClient(modem_cfg) if modem_cfg.get('url') else None
self.modem_name = modem_cfg.get('name', 'att-modem')
# With the gateway's access code, also read its NAT sessions and ping from it
self.modem_admin = (modem_api.ModemAdmin(modem_cfg)
if modem_cfg.get('url') and modem_cfg.get('access_code') else None)
# Modem reboots (including AT&T's automatic firmware updates) take a few
# minutes, so this can be raised independently of failure_threshold.
self.modem_fail_thresh = modem_cfg.get('failure_threshold', self.fail_thresh)
@@ -1970,6 +1974,133 @@ class NetworkMonitor:
if quality is not None:
db.set_state('wan_quality', quality)
_RESTART_SETTLE_S = 180 # the modem answering this soon may not have restarted yet
_RESTART_GIVE_UP_S = 1800
def _finish_modem_restart(self) -> None:
"""After the Restart modem button: once the modem answers again with its WAN up (or
after 30 minutes regardless), end the Uptime Kuma window and the Gandalf silence the
button started, and note how long it took."""
raw = db.get_state('modem_restart')
try:
state = json.loads(raw) if raw else None
except ValueError:
state = None
if not state or state.get('finished_at'):
return
now = time.time()
age = now - state.get('at', now)
raw = db.get_state('modem_stats')
try:
stats = json.loads(raw) if raw else {}
except ValueError:
stats = {}
uptime = stats.get('uptime_s')
rebooted = uptime is not None and uptime < age
back = (age >= self._RESTART_SETTLE_S and stats.get('reachable')
and not stats.get('wan_problems') and (rebooted or uptime is None))
if not back and age < self._RESTART_GIVE_UP_S:
return
if state.get('kuma') and not state.get('kuma_ended'):
try:
kuma.set_maintenance(self.cfg.get('uptime_kuma') or {}, False)
state['kuma_ended'] = True
except kuma.KumaError as e:
logger.warning(f'Ending the Uptime Kuma window after the modem restart: {e}')
if age < self._RESTART_GIVE_UP_S:
return # try again next cycle
state['kuma_error'] = str(e)
if state.get('suppression_id'):
db.deactivate_suppression(state['suppression_id'])
state.update(finished_at=now, back=bool(back), rebooted=rebooted,
back_after_s=round(age) if back else None)
db.set_state('modem_restart', state)
logger.info(f'Modem restart finished: {"back" if back else "gave up waiting"} after {round(age)} s'
f'{", rebooted" if rebooted else ""}')
_NAT_EVERY_S = 1800 # NAT sessions and the gateway's speed test history
_FIREWALL_EVERY_S = 300 # the firewall log only holds about ten minutes
_FIREWALL_HOURS = 24
_PING_EVERY_S = 600
_PING_TARGET = '1.1.1.1'
def _collect_modem_checks(self) -> None:
"""NAT session counts every 30 minutes, and while UniFi says the internet is slow,
a ping from the modem itself (at most every 10 minutes) to show where the delay is.
Results go to monitor_state.modem_checks for the WAN panel and Needs attention."""
admin = getattr(self, 'modem_admin', None)
if admin is None:
return
raw = db.get_state('modem_checks')
try:
checks = json.loads(raw) if raw else {}
except ValueError:
checks = {}
now = time.time()
raw = db.get_state('wan_quality')
try:
quality = json.loads(raw) if raw else {}
except ValueError:
quality = {}
periods = quality.get('periods') or []
latencies = quality.get('latencies') or {}
ours = latencies.get('cloudflare', min(latencies.values()) if latencies else None)
slow = (bool(periods) and periods[-1].get('state') != 'ok') or (ours or 0) >= modem_api.SLOW_MS
due_nat = now - (checks.get('nat') or {}).get('at', 0) >= self._NAT_EVERY_S
due_fw = now - (checks.get('firewall') or {}).get('checked_at', 0) >= self._FIREWALL_EVERY_S
due_ping = slow and now - (checks.get('ping') or {}).get('at', 0) >= self._PING_EVERY_S
if not (due_nat or due_fw or due_ping):
return
if now - (checks.get('error') or {}).get('at', 0) < self._PING_EVERY_S:
return # failed recently: don't keep knocking on the gateway
try:
if due_nat:
checks['nat'] = dict(admin.nat_sessions(), at=now)
checks['speed'] = self._latest_speed_tests(admin.speed_tests(), now)
if due_fw:
checks['firewall'] = self._count_firewall_drops(checks.get('firewall'), admin.firewall_log(), now)
if due_ping:
result = admin.ping(self._PING_TARGET)
checks['ping'] = dict(result, at=now, ours_ms=ours,
verdict=modem_api.ping_verdict(result, ours))
logger.info(f'Modem ping to {self._PING_TARGET}: {result.get("avg_ms")} ms, '
f'{result.get("loss_pct")}% loss; ours {ours} ms '
f'({checks["ping"]["verdict"]})')
checks['error'] = None
except (modem_api.ModemUnreachable, modem_api.ModemDataInvalid, modem_api.ModemLoginFailed) as e:
logger.warning(f'Modem checks: {e}')
checks['error'] = {'message': str(e), 'at': now,
'login': isinstance(e, modem_api.ModemLoginFailed)}
db.set_state('modem_checks', checks)
@staticmethod
def _latest_speed_tests(tests: list, now: float) -> dict:
"""The newest downstream and upstream result from the gateway's own speed tests."""
out: dict = {'checked_at': now}
for t in tests: # newest first
key = 'down' if t['direction'] == 'downstream' else 'up'
out.setdefault(key, t)
return out
def _count_firewall_drops(self, previous: Optional[dict], rows: list, now: float) -> dict:
"""Add log rows newer than the last one seen to per-hour counts (gateway's local hour).
The log keeps a few hundred rows and records at most one drop every 2 seconds, so the
counts are a lower bound; 'gap' says rows aged out before this read could count them.
"""
fw = previous or {}
hours = dict(fw.get('hours') or {})
last = fw.get('last') or ''
new = [r for r in rows if r['ts'] > last]
for r in new:
bucket = hours.setdefault(r['ts'][:13], {'flood': 0, 'invalid': 0, 'other': 0})
bucket[modem_api.firewall_kind(r['reason'])] += 1
gap = bool(last and rows and rows[0]['ts'] > last)
keep = sorted(hours)[-self._FIREWALL_HOURS:]
return {'hours': {h: hours[h] for h in keep}, 'last': rows[-1]['ts'] if rows else last,
'checked_at': now, 'gap': gap}
def _collect_modem(self, suppressions: list) -> None:
"""Poll the modem, then store its status and raise any alerts."""
try:
@@ -2161,6 +2292,18 @@ class NetworkMonitor:
except Exception as e:
logger.error(f'WAN quality collection failed: {e}', exc_info=True)
# An admin's modem restart: end the silence once the modem is back
try:
self._finish_modem_restart()
except Exception as e:
logger.error(f'Finishing the modem restart failed: {e}', exc_info=True)
# Logged-in modem checks (NAT sessions; a ping from the modem when the internet is slow)
try:
self._collect_modem_checks()
except Exception as e:
logger.error(f'Modem checks failed: {e}', exc_info=True)
# Note on the tickets of anything that cleared this cycle (or failed to post before)
try:
self._post_recovery_notes()
+1
View File
@@ -4,3 +4,4 @@ ldap3>=2.9
pymysql>=1.1.0
requests>=2.31.0
urllib3>=2.0.0
python-socketio[client]>=5.8
+120 -14
View File
@@ -363,6 +363,58 @@ function _modemLastRestart(history) {
return `restarted ${_modemWhen(last.at)}${last.by ? ` by ${last.by}` : ''}`;
}
function _modemUptimeStat(data) {
if (data.uptime_s !== null && data.uptime_s !== undefined) {
return _modemStat('Uptime', _modemUptime(data.uptime_s), '', _modemLastRestart(data.wan_history));
}
const restarts = (data.wan_history && data.wan_history.restarts) || [];
if (!restarts.length) return '';
const last = restarts[restarts.length - 1];
return _modemStat('Last restart', _modemWhen(last.at), '', last.by ? `by ${last.by}` : '');
}
const _PING_VERDICTS = {
upstream: 'slow past the modem: provider side',
gateway: 'modem fine, ours slow: delay in the modem',
clear: 'fine',
};
// Logged-in checks: NAT sessions in use, and the latest ping run from the modem itself
function _modemChecks(c) {
if (!c) return '';
const ago = at => lt.time.ago(new Date(at * 1000));
let out = '';
if (c.nat && c.nat.available) {
const pct = Math.round(100 * c.nat.in_use / c.nat.available);
out += _modemStat('NAT sessions', `${c.nat.in_use.toLocaleString()} / ${c.nat.available.toLocaleString()}`,
pct >= 80 ? 'lt-text-amber' : '', `${pct}% in use, ${ago(c.nat.at)}`);
}
if (c.ping) {
const p = c.ping;
const value = p.avg_ms === null || p.avg_ms === undefined ? 'no reply' : `${Math.round(p.avg_ms)} ms`;
const verdict = _PING_VERDICTS[p.verdict] || '';
out += _modemStat("Modem's own ping", value, p.verdict === 'clear' ? '' : 'lt-text-amber',
`to ${p.target}, ${ago(p.at)}${verdict ? `: ${verdict}` : ''}`);
}
const fw = c.firewall && c.firewall.hours;
if (fw && Object.keys(fw).length) {
const sum = k => Object.values(fw).reduce((n, h) => n + (h[k] || 0), 0);
const flood = sum('flood'), invalid = sum('invalid'), total = flood + invalid + sum('other');
out += _modemStat('Firewall drops', `${total.toLocaleString()} in ${Object.keys(fw).length} h`, '',
`${flood.toLocaleString()} incoming over the flood limit, ${invalid.toLocaleString()} invalid; ` +
'at least (the log keeps 1 per 2 s)');
}
const sp = c.speed;
if (sp && sp.down && sp.up) {
out += _modemStat('AT&T speed test', `↓ ${Math.round(sp.down.mbps)} ↑ ${Math.round(sp.up.mbps)} Mbps`, '',
`run by the modem ${sp.down.at}, ${Math.round(sp.down.latency_ms)} ms`);
}
if (c.error && c.error.login) {
out += _modemStat('Modem login', 'failed', 'lt-text-red', 'check modem.access_code');
}
return out;
}
function _modemDuration(seconds) {
if (seconds < 60) return `${Math.round(seconds)}s`;
if (seconds < 3600) return `${Math.floor(seconds / 60)}m ${Math.round(seconds % 60)}s`;
@@ -418,6 +470,47 @@ function _modemQuality(q) {
</div>`;
}
// A restart from the Restart modem button: in progress, or finished in the last hour
function _modemRestartNote(r) {
if (!r || !r.at) return '';
const who = r.by ? ` by ${r.by}` : '';
if (!r.finished_at) {
return `<div class="modem-restart-note">Restart requested${lt.escHtml(who)} ${lt.escHtml(_modemWhen(r.at))}. ` +
`Alerts are silenced${r.kuma ? ' here and in Uptime Kuma' : ''} until the modem is back.</div>`;
}
if (Date.now() / 1000 - r.finished_at > 3600) return '';
const how = r.back ? `back after ${_modemDuration(r.back_after_s)}` : 'did not come back within 30 minutes';
return `<div class="modem-restart-note">Restarted${lt.escHtml(who)} ${lt.escHtml(_modemWhen(r.at))}: ` +
`${lt.escHtml(how)}. Alerts are on again.</div>`;
}
async function restartModem(btn) {
const msg = 'Restart the AT&T modem?\n\nThe internet will be down for about 3 to 5 minutes. ' +
'Gandalf alerts and the Uptime Kuma maintenance window are switched on first, and switched off ' +
'again when the modem is back.';
if (!window.confirm(msg)) return;
btn.disabled = true;
try {
let res;
try {
res = await lt.api.post('/api/modem/restart', { confirm: true });
} catch (err) {
if (!String(err.message).startsWith('Uptime Kuma:')) throw err;
if (!window.confirm(`${err.message}\n\nRestart anyway? Uptime Kuma will report the outage.`)) {
btn.disabled = false;
return;
}
res = await lt.api.post('/api/modem/restart', { confirm: true, without_kuma: true });
}
showToast(res.result === 'accepted' ? 'Modem restarting' : 'Restart sent; the modem dropped the connection',
'success');
window.location.reload();
} catch (err) {
btn.disabled = false;
showToast(err.message || 'Could not restart the modem', 'error');
}
}
function updateModem(data) {
const section = document.getElementById('modem-section');
const body = document.getElementById('modem-body');
@@ -425,6 +518,18 @@ function updateModem(data) {
section.hidden = !data.configured;
if (!data.configured) return;
const restartBtn = document.getElementById('modem-restart-btn');
if (restartBtn) {
const pending = data.modem_restart && !data.modem_restart.finished_at;
restartBtn.hidden = !data.can_restart;
restartBtn.disabled = Boolean(pending);
if (!restartBtn.dataset.bound) {
restartBtn.dataset.bound = '1';
restartBtn.addEventListener('click', () => restartModem(restartBtn));
}
}
const restartNote = _modemRestartNote(data.modem_restart);
// On the dashboard, link to the modem's page. On that page, it would link to itself.
const detailsLink = document.getElementById('modem-device-link');
if (detailsLink && typeof DEVICE_NAME === 'undefined' && data.name) {
@@ -435,7 +540,7 @@ function updateModem(data) {
const title = document.getElementById('modem-title');
if (!data.reachable) {
body.innerHTML = `
body.innerHTML = `${restartNote}
<div class="modem-error">Modem status unavailable: ${lt.escHtml(data.error || 'unknown error')}
<span class="modem-stat-note">Last attempt ${lt.escHtml(data.updated || '-')}</span></div>
${_modemQuality(data.wan_quality)}
@@ -456,7 +561,7 @@ function updateModem(data) {
const errs = (counters.rx_errors || 0) + (counters.tx_errors || 0);
const drops = (counters.rx_drops || 0) + (counters.tx_drops || 0);
body.innerHTML = `
body.innerHTML = `${restartNote}
${problems.length ? `<div class="modem-error">WAN down: ${lt.escHtml(problems.join('; '))}</div>` : ''}
<div class="modem-stats">
${_modemStat('WAN', problems.length ? 'DOWN' : 'UP', problems.length ? 'lt-text-red' : 'lt-text-green',
@@ -467,7 +572,8 @@ function updateModem(data) {
${_modemOptic('Optics temp', optics.temperature, status.temperature, '°C')}
${_modemStat('Throughput', `↓ ${_modemRate(rates.rx_bytes)}`, '', `↑ ${_modemRate(rates.tx_bytes)}`)}
${_modemStat('Errors / drops', `${errs} / ${drops}`, errs ? 'lt-text-amber' : '', 'since modem reboot')}
${_modemStat('Uptime', _modemUptime(data.uptime_s), '', _modemLastRestart(data.wan_history))}
${_modemUptimeStat(data)}
${_modemChecks(data.modem_checks)}
</div>
${_modemQuality(data.wan_quality)}
${_modemChart(data.history)}
@@ -526,16 +632,16 @@ function updateEventsTable(events, totalActive) {
: '';
return `
<tr class="row-${e.severity}${e.is_suppressed ? ' row-suppressed' : ''}">
<td><span class="lt-badge badge-${e.severity}">${e.severity}</span>${supBadge}${staleBadge}${closedBadge}</td>
<td>${lt.escHtml(e.event_type.replace(/_/g,' '))}</td>
<td><strong>${lt.escHtml(e.target_name)}</strong></td>
<td>${lt.escHtml(e.target_detail || '-')}</td>
<td class="desc-cell" title="${lt.escHtml(e.description || '')}">${lt.escHtml((e.description||'').substring(0,60))}${(e.description||'').length>60?'…':''}</td>
<td class="ts-cell" title="${lt.escHtml(e.first_seen||'')}">${lt.time.ago(_toIso(e.first_seen))}</td>
<td class="ts-cell" title="${lt.escHtml(e.last_seen||'')}">${lt.time.ago(_toIso(e.last_seen))}</td>
<td>${e.consecutive_failures}</td>
<td>${ticket}</td>
<td>
<td data-label="Sev"><span class="lt-badge badge-${e.severity}">${e.severity}</span>${supBadge}${staleBadge}${closedBadge}</td>
<td data-label="Type">${lt.escHtml(e.event_type.replace(/_/g,' '))}</td>
<td data-label="Target"><strong>${lt.escHtml(e.target_name)}</strong></td>
<td data-label="Detail">${lt.escHtml(e.target_detail || '-')}</td>
<td data-label="Description" class="desc-cell" title="${lt.escHtml(e.description || '')}">${lt.escHtml((e.description||'').substring(0,60))}${(e.description||'').length>60?'…':''}</td>
<td data-label="First seen" class="ts-cell" title="${lt.escHtml(e.first_seen||'')}">${lt.time.ago(_toIso(e.first_seen))}</td>
<td data-label="Last seen" class="ts-cell" title="${lt.escHtml(e.last_seen||'')}">${lt.time.ago(_toIso(e.last_seen))}</td>
<td data-label="Failures">${e.consecutive_failures}</td>
<td data-label="Ticket">${ticket}</td>
<td data-label="Actions">
<button class="lt-btn lt-btn-ghost lt-btn-sm btn-suppress"
data-sup-type="${lt.escHtml(supType)}"
data-sup-name="${lt.escHtml(e.target_name)}"
@@ -548,7 +654,7 @@ function updateEventsTable(events, totalActive) {
wrap.innerHTML = `
${countNotice}
<div class="lt-table-wrap">
<table class="lt-table" id="events-table">
<table class="lt-table lt-table-responsive" id="events-table">
<caption class="lt-sr-only">Active network alerts</caption>
<thead>
<tr>
+1
View File
@@ -488,6 +488,7 @@
const date = value instanceof Date ? value : new Date(value);
if (isNaN(date)) return '-';
const diff = Math.floor((Date.now() - date.getTime()) / 1000);
if (diff < 5) return 'just now'; // also a clock a little ahead of this browser's
if (diff < 60) return diff + 's ago';
if (diff < 3600) return Math.floor(diff / 60) + 'm ago';
if (diff < 86400) return Math.floor(diff / 3600) + 'h ago';
+35 -8
View File
@@ -177,13 +177,18 @@
text-transform: uppercase;
letter-spacing: .08em;
}
/* A count. Neutral by default; red only where it counts problems (active alerts, follow-ups). */
.g-section-badge {
font-size: .72em;
font-weight: bold;
color: var(--accent-red);
border: 1px solid var(--accent-red);
color: var(--text-muted);
border: 1px solid var(--border-color);
padding: 1px 7px;
}
.g-section-badge-alert {
color: var(--accent-red);
border-color: var(--accent-red);
}
.g-section-badge-resolved {
font-size: .68em;
color: var(--text-muted);
@@ -330,7 +335,6 @@
letter-spacing: .04em;
font-weight: bold;
}
.source-prometheus { color: var(--orange); border-color: var(--orange-dim); }
.source-ping { color: var(--cyan); border-color: var(--cyan-dim); }
.iface-list { border-top: 1px solid var(--border-color); padding-top: 6px; margin-bottom: 8px; }
.iface-row { display: flex; align-items: center; gap: 6px; padding: 2px 0; }
@@ -345,8 +349,9 @@
/* ── Form layout helpers (used in suppressions.html form grids) ───── */
.form-row { display: flex; gap: 12px; flex-wrap: wrap; margin-bottom: 14px; }
.form-row-align { align-items: flex-end; }
.form-row > .lt-form-group { flex: 1 1 220px; }
.form-group-wide { flex: 3; }
.form-group-submit { flex: 0 0 auto; min-width: unset; }
.form-group-submit, .form-row > .form-group-submit { flex: 0 0 auto; min-width: unset; }
.required { color: var(--red); }
/* ── Duration pills ───────────────────────────────────────────────── */
@@ -914,10 +919,6 @@ a.device-link:hover { color: var(--text-accent); border-bottom-color: current
.device-table th, .device-table td { padding-left: 6px; padding-right: 6px; }
}
.device-search-row.is-active { box-shadow: inset 3px 0 0 var(--terminal-cyan, #0ff); }
@media (max-width: 700px) {
/* Scroll inside the wrapper rather than squeezing every column to a few letters */
#events-table { min-width: 760px; }
}
/* ── Port inspector on the device page ────────────────────────────── */
.device-inspector { display: grid; grid-template-columns: minmax(0, 1fr) auto; gap: 14px 16px; align-items: start; }
@@ -1827,3 +1828,29 @@ body.fp-marks-off .fp-svg .fp-mark { display: none; }
.lt-header-user, .lt-cmd-hint-btn, .lt-header-right [data-action="start-maintenance"] { display: none; }
}
.g-drawer-action { background: none; border: 0; width: 100%; text-align: left; font: inherit; cursor: pointer; }
/* ── Footer: the CFG and HELP buttons read like the keyboard hints beside them ── */
button.lt-footer-hint {
background: none; border: 0; padding: 0; margin: 0;
font: inherit; color: inherit; letter-spacing: inherit; text-transform: inherit; cursor: pointer;
}
button.lt-footer-hint:hover, button.lt-footer-hint:focus-visible { color: var(--accent-cyan, var(--cyan)); }
/* ── Footer at the bottom of the window on short pages, not halfway up ── */
body { display: flex; flex-direction: column; }
body > .lt-main { flex: 1 0 auto; width: 100%; }
body > .lt-footer { flex-shrink: 0; }
/* ── Active alerts: room for the filter's hint; full-width cards on phones ── */
#events-search { min-width: 300px; }
@media (max-width: 767px) {
#events-search { min-width: 0; }
#events-table, #events-table tbody { display: block; width: 100%; }
#events-table td { max-width: none; }
#events-table td.desc-cell { white-space: normal; text-align: right; }
}
/* ── WAN / Fiber: restart button and progress note ── */
.modem-actions { display: inline-flex; gap: 6px; }
.modem-restart-note { margin: 8px 12px 0; padding: 6px 10px; font-size: .8em;
border: 1px solid var(--accent-amber); color: var(--text-primary); }
+7 -1
View File
@@ -1,7 +1,13 @@
<section class="g-section" id="modem-section" hidden>
<div class="g-section-header">
<h2 class="g-section-title">WAN / Fiber</h2>
<a class="g-section-actions lt-btn lt-btn-ghost lt-btn-sm" id="modem-device-link" hidden>↗ Details</a>
<span class="g-section-actions modem-actions">
{% if user.groups and 'admin' in user.groups %}
<button type="button" class="lt-btn lt-btn-ghost lt-btn-sm" id="modem-restart-btn" hidden
title="Silence alerts (Gandalf and Uptime Kuma), then restart the modem">⟳ Restart modem</button>
{% endif %}
<a class="lt-btn lt-btn-ghost lt-btn-sm" id="modem-device-link" hidden>↗ Details</a>
</span>
</div>
<div class="lt-frame">
<span class="lt-frame-bl">&#x255A;</span>
+1 -1
View File
@@ -290,7 +290,7 @@
</div>
<div class="lt-form-group" id="sup-name-group">
<label class="lt-label" for="sup-name">Target Name</label>
<input type="text" class="lt-input" id="sup-name" name="target_name" placeholder="e.g. large1">
<input type="text" class="lt-input" id="sup-name" name="target_name" placeholder="e.g. compute-storage-02">
</div>
<div class="lt-form-group" id="sup-detail-group" style="display:none">
<label class="lt-label" for="sup-detail">Interface <span class="lt-field-hint">(interface type only)</span></label>
+14 -14
View File
@@ -53,7 +53,7 @@
<section class="g-section">
<div class="g-section-header">
<h2 class="g-section-title">Active Alerts</h2>
<span class="g-section-badge" id="alert-count-badge"
<span class="g-section-badge g-section-badge-alert" id="alert-count-badge"
{% if not summary.critical and not summary.warning %}style="display:none"{% endif %}>{{ (summary.critical or 0) + (summary.warning or 0) }}</span>
</div>
<div class="lt-toolbar">
@@ -80,7 +80,7 @@
<div class="pagination-notice">Showing {{ events|length }} of {{ total_active }} active alerts. Use the search box to filter, or <a href="/api/events?limit=1000" target="_blank" rel="noopener">export all as JSON</a></div>
{% endif %}
<div class="lt-table-wrap">
<table class="lt-table" id="events-table">
<table class="lt-table lt-table-responsive" id="events-table">
<caption class="lt-sr-only">Active network alerts</caption>
<thead>
<tr>
@@ -100,30 +100,30 @@
{% for e in events %}
{% if e.severity != 'info' %}
<tr class="row-{{ e.severity }}{% if e.is_suppressed %} row-suppressed{% endif %}">
<td>
<td data-label="Sev">
<span class="lt-badge badge-{{ e.severity }}">{{ e.severity }}</span>
{% if e.is_suppressed %}<span class="lt-badge badge-suppressed" title="Alert suppressed">🔕 sup</span>{% endif %}
{% if e.stale_days is not none %}<span class="lt-badge badge-stale" title="Active for {{ e.stale_days }} days with no fix yet">⏳ {{ e.stale_days }}d</span>{% endif %}
{% if e.ticket_closed %}<span class="lt-badge badge-ticket-closed" title="Ticket #{{ e.ticket_id }} was closed in Tinker, but this alert is still active">ticket closed</span>{% endif %}
</td>
<td>{{ e.event_type | replace('_', ' ') }}</td>
<td><strong>{{ e.target_name }}</strong></td>
<td>{{ e.target_detail or '-' }}</td>
<td class="desc-cell" title="{{ e.description | e }}">{{ e.description | truncate(60) }}</td>
<td class="ts-cell" title="{{ e.first_seen }}">
<td data-label="Type">{{ e.event_type | replace('_', ' ') }}</td>
<td data-label="Target"><strong>{{ e.target_name }}</strong></td>
<td data-label="Detail">{{ e.target_detail or '-' }}</td>
<td data-label="Description" class="desc-cell" title="{{ e.description | e }}">{{ e.description | truncate(60) }}</td>
<td data-label="First seen" class="ts-cell" title="{{ e.first_seen }}">
<span class="event-age" data-ts="{{ e.first_seen }}">{{ e.first_seen }}</span>
</td>
<td class="ts-cell" title="{{ e.last_seen }}">
<td data-label="Last seen" class="ts-cell" title="{{ e.last_seen }}">
<span class="event-age" data-ts="{{ e.last_seen }}">{{ e.last_seen }}</span>
</td>
<td>{{ e.consecutive_failures }}</td>
<td>
<td data-label="Failures">{{ e.consecutive_failures }}</td>
<td data-label="Ticket">
{% if e.ticket_id %}
<a href="{{ config.ticket_api.web_url }}{{ e.ticket_id }}" target="_blank"
class="ticket-link">#{{ e.ticket_id }}</a>
{% else %}-{% endif %}
</td>
<td>
<td data-label="Actions">
<button class="lt-btn lt-btn-ghost lt-btn-sm btn-suppress"
data-sup-type="{{ 'unifi_device' if e.event_type == 'unifi_device_offline' else 'interface' if e.event_type == 'interface_down' else 'host' }}"
data-sup-name="{{ e.target_name }}"
@@ -157,7 +157,7 @@
<section class="g-section" id="followup"{% if not stale_tickets %} hidden{% endif %}>
<div class="g-section-header">
<h2 class="g-section-title">Needs Follow-up</h2>
<span class="g-section-badge" id="followup-badge">{{ stale_tickets | length }}</span>
<span class="g-section-badge g-section-badge-alert" id="followup-badge">{{ stale_tickets | length }}</span>
</div>
<p class="followup-intro">These tickets are still open in Tinker, but their alert cleared over a day ago. Close them if the problem is fixed.</p>
<div class="lt-frame">
@@ -212,7 +212,7 @@
</div>
<div class="host-meta">
<span class="host-ip">{{ host.ip }}</span>
<span class="host-source source-{{ host.source }}">{{ host.source }}</span>
{% if host.source != 'prometheus' %}<span class="host-source source-{{ host.source }}">{{ host.source }}</span>{% endif %}
</div>
</div>
+1 -1
View File
@@ -50,7 +50,7 @@
<div class="lt-form-group form-group-wide">
<label class="lt-label" for="s-reason">Reason <span class="required">*</span></label>
<input type="text" class="lt-input" id="s-reason" name="reason"
placeholder="e.g. Planned switch maintenance, replacing SFP on large1/enp43s0"
placeholder="e.g. Planned switch maintenance, replacing SFP on compute-storage-02/enp43s0"
required aria-required="true">
</div>
</div>
+82 -1
View File
@@ -4,7 +4,7 @@ import time
from datetime import datetime, timedelta, timezone
import os
import sys
from unittest.mock import patch
from unittest.mock import MagicMock, patch
import pytest
@@ -850,9 +850,90 @@ class TestCalmDashboard:
assert '>eth0<' in html and '>eth1<' not in html
assert '+2 unused' in html
def test_source_tag_only_for_hosts_without_node_exporter(self, client):
snap = json.dumps({'hosts': {
'large1': {'ip': '10.0.0.2', 'status': 'up', 'source': 'prometheus', 'interfaces': {'eth0': 'up'}},
'pbs': {'ip': '10.0.0.3', 'status': 'up', 'source': 'ping', 'interfaces': {}}}, 'unifi': []})
state = dict(STATE, network_snapshot=snap)
with patch.object(gandalf.db, 'get_state', side_effect=lambda k, d=None: state.get(k, d)):
html = client.get('/', headers=AUTH).get_data(as_text=True)
assert 'source-prometheus' not in html
assert '<span class="host-source source-ping">ping</span>' in html
def test_alert_table_turns_into_cards_on_phones(self, client):
ev = {'id': 1, 'event_type': 'host_down', 'severity': 'critical', 'target_name': 'pbs',
'target_detail': '', 'description': 'pbs is down', 'consecutive_failures': 3,
'first_seen': '2026-11-15 01:00:00', 'last_seen': '2026-11-15 02:00:00'}
with patch.object(gandalf.db, 'get_active_events', return_value=[ev]):
html = client.get('/', headers=AUTH).get_data(as_text=True)
assert 'class="lt-table lt-table-responsive" id="events-table"' in html
assert 'data-label="Target"><strong>pbs</strong>' in html
def test_resolved_alerts_are_folded(self, client):
ev = {'target_name': 'large1', 'target_detail': '', 'severity': 'warning', 'event_type': 'host_down',
'first_seen': '2026-11-15 01:00:00 UTC', 'resolved_at': '2026-11-15 02:00:00 UTC'}
with patch.object(gandalf.db, 'get_recent_resolved', return_value=[ev]):
html = client.get('/', headers=AUTH).get_data(as_text=True)
assert '<details class="g-section resolved-fold">' in html
class TestModemRestart:
CFG = dict(CONFIG, modem={'url': 'http://192.0.2.254', 'access_code': 'secret'},
uptime_kuma={'url': 'http://kuma.test', 'username': 'u', 'password': 'p', 'maintenance_id': 7})
def _post(self, client, body=None, headers=AUTH, cfg=None, kuma_error=None, restart=None):
admin = MagicMock()
if isinstance(restart, Exception):
admin.restart.side_effect = restart
else:
admin.restart.return_value = restart or 'accepted'
with patch.object(gandalf, '_cfg', cfg or self.CFG), \
patch.object(gandalf.kuma, 'set_maintenance', side_effect=kuma_error) as km, \
patch.object(gandalf.modem_api, 'ModemAdmin', return_value=admin):
gandalf.db.create_suppression.return_value = 42
resp = client.post('/api/modem/restart', json=body if body is not None else {'confirm': True},
headers=headers)
return resp, km, admin
def test_admin_only(self, client):
resp, _, admin = self._post(client, headers=USER)
assert resp.status_code == 403 and not admin.restart.called
def test_needs_confirm(self, client):
resp, km, admin = self._post(client, body={})
assert resp.status_code == 400 and not km.called and not admin.restart.called
def test_needs_the_access_code(self, client):
resp, _, admin = self._post(client, cfg=dict(CONFIG, modem={'url': 'http://192.0.2.254'}))
assert resp.status_code == 400 and 'access_code' in resp.get_json()['error']
def test_silences_then_restarts(self, client):
resp, km, admin = self._post(client)
assert resp.status_code == 200
assert resp.get_json() == {'ok': True, 'result': 'accepted', 'kuma': True, 'suppression_id': 42}
km.assert_called_once_with(self.CFG['uptime_kuma'], True)
args = gandalf.db.create_suppression.call_args.args
assert args[:3] == ('all', '', '') and args[4] == 'jared' and args[5] == gandalf.MODEM_RESTART_SILENCE_MIN
admin.restart.assert_called_once_with()
key, state = gandalf.db.set_state.call_args.args
assert key == 'modem_restart' and state['suppression_id'] == 42 and state['kuma'] is True
def test_kuma_failure_stops_unless_accepted(self, client):
resp, _, admin = self._post(client, kuma_error=gandalf.kuma.KumaError('login refused'))
assert resp.status_code == 502 and resp.get_json()['kuma_failed'] and not admin.restart.called
resp, _, admin = self._post(client, body={'confirm': True, 'without_kuma': True},
kuma_error=gandalf.kuma.KumaError('login refused'))
assert resp.status_code == 200 and resp.get_json()['kuma'] is False and admin.restart.called
def test_failed_restart_undoes_the_silence(self, client):
resp, km, _ = self._post(client, restart=gandalf.modem_api.ModemUnreachable('timeout'))
assert resp.status_code == 502
gandalf.db.deactivate_suppression.assert_called_once_with(42)
assert km.call_args_list[-1].args == (self.CFG['uptime_kuma'], False)
def test_one_at_a_time(self, client):
pending = json.dumps({'at': time.time() - 60, 'finished_at': None})
state = {'modem_restart': pending}
with patch.object(gandalf.db, 'get_state', side_effect=lambda k, d=None: state.get(k, d)):
resp, _, admin = self._post(client)
assert resp.status_code == 409 and not admin.restart.called
+38
View File
@@ -143,3 +143,41 @@ class TestWan:
out = attention.build([_alert('x', 'warning')], [], None,
wan_quality=_quality(['latency'] * 12), now=NOW)['broken']
assert out[0]['text'].startswith('Internet')
class TestModemChecks:
def _broken(self, checks, quality=None):
return attention.build([], [], None, wan_quality=quality, now=NOW, modem_checks=checks)
def test_gateway_ping_explains_the_latency(self):
checks = {'ping': {'at': NOW - 120, 'avg_ms': 20.4, 'loss_pct': 0, 'ours_ms': 1043, 'verdict': 'gateway'}}
[e] = self._broken(checks, _quality(['latency'] * 12))['broken']
assert ("Modem's own ping 2 min ago: 20 ms; ours through it 1043 ms, "
'so the delay is in the modem, or between it and the router.') in e['detail']
def test_upstream_verdict(self):
checks = {'ping': {'at': NOW - 60, 'avg_ms': 900.0, 'loss_pct': 20, 'ours_ms': 1000, 'verdict': 'upstream'}}
[e] = self._broken(checks, _quality(['latency'] * 12))['broken']
assert ("900 ms, 20% lost; ours through it 1000 ms, "
"so the delay is past the modem, on the provider's side") in e['detail']
def test_old_ping_is_left_out(self):
checks = {'ping': {'at': NOW - 7200, 'avg_ms': 20.0, 'verdict': 'clear'}}
[e] = self._broken(checks, _quality(['latency'] * 12))['broken']
assert "Modem's own ping" not in e['detail']
def test_nat_nearly_full(self):
[e] = self._broken({'nat': {'in_use': 30000, 'available': 32767}})['broken']
assert e['text'] == 'Modem NAT table 92% full' and e['severity'] == 'warn'
assert '30,000 of 32,767' in e['detail']
def test_nat_normal_says_nothing(self):
out = self._broken({'nat': {'in_use': 2858, 'available': 32767}})
assert out == {'broken': [], 'tidy': []}
def test_login_failure_is_housekeeping(self):
[e] = self._broken({'error': {'login': True, 'message': 'refused', 'at': NOW}})['tidy']
assert e['text'].startswith('Modem login failed')
def test_other_errors_stay_quiet(self):
assert self._broken({'error': {'login': False, 'message': 'timeout', 'at': NOW}})['tidy'] == []
+91
View File
@@ -0,0 +1,91 @@
"""kuma.py: switching the Uptime Kuma maintenance window, against a fake Socket.IO client."""
import os
import sys
import pytest
sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
import kuma # noqa: E402
CFG = {'url': 'http://kuma.test:3001', 'token': 'tok', 'maintenance_id': 7}
PASSWORD_CFG = {'url': 'http://kuma.test:3001', 'username': 'gandalf', 'password': 'pw', 'maintenance_id': 7}
class FakeSocket:
def __init__(self, answers=None, connect_error=None, sends_info=True):
self.answers = answers or {}
self.connect_error = connect_error
self.sends_info = sends_info
self.calls, self.disconnected, self.handlers = [], False, {}
def on(self, event, handler):
self.handlers[event] = handler
def connect(self, url, **kw):
if self.connect_error:
raise self.connect_error
self.url = url
if self.sends_info: # Kuma's first event; its handlers exist only after it
self.handlers['info']({'serverTimezone': 'America/New_York'})
def call(self, event, data=None, timeout=None):
self.calls.append((event, data))
return self.answers.get(event, {'ok': True})
def disconnect(self):
self.disconnected = True
def run(active=True, cfg=CFG, **kw):
sock = FakeSocket(**kw)
kuma.set_maintenance(dict(cfg, timeout=0.05), active, client_factory=lambda: sock)
return sock
class TestKuma:
def test_configured_needs_url_window_and_a_way_to_sign_in(self):
assert kuma.configured(CFG) and kuma.configured(PASSWORD_CFG)
assert not kuma.configured(dict(CFG, token=''))
assert not kuma.configured(dict(CFG, maintenance_id=None))
assert not kuma.configured(dict(PASSWORD_CFG, password=''))
assert not kuma.configured(None)
def test_token_sign_in_then_switches_the_window_on(self):
sock = run(True)
assert sock.calls == [('loginByToken', 'tok'), ('resumeMaintenance', 7)]
assert sock.disconnected and sock.url == CFG['url']
def test_rejected_token(self):
with pytest.raises(kuma.KumaError, match='token rejected'):
run(answers={'loginByToken': {'ok': False}})
def test_waits_for_kuma_to_be_ready(self):
sock = FakeSocket(sends_info=False)
with pytest.raises(kuma.KumaError, match='not ready'):
kuma.set_maintenance(dict(CFG, timeout=0.05), True, client_factory=lambda: sock)
assert sock.calls == [] and sock.disconnected # nothing sent before info
def test_password_sign_in(self):
sock = run(True, cfg=PASSWORD_CFG)
assert sock.calls[0] == ('login', {'username': 'gandalf', 'password': 'pw', 'token': ''})
def test_pause(self):
assert run(False).calls[-1] == ('pauseMaintenance', 7)
def test_refused_password(self):
with pytest.raises(kuma.KumaError, match='login refused: Incorrect username or password'):
run(cfg=PASSWORD_CFG, answers={'login': {'ok': False, 'msg': 'Incorrect username or password.'}})
def test_two_factor_account_needs_the_token(self):
with pytest.raises(kuma.KumaError, match='use uptime_kuma.token'):
run(cfg=PASSWORD_CFG, answers={'login': {'ok': False, 'tokenRequired': True}})
def test_refused_change(self):
with pytest.raises(kuma.KumaError, match='resumeMaintenance refused'):
run(answers={'resumeMaintenance': {'ok': False, 'msg': 'not found'}})
def test_unreachable(self):
sock = FakeSocket(connect_error=ConnectionError('refused'))
with pytest.raises(kuma.KumaError, match='ConnectionError'):
kuma.set_maintenance(dict(CFG, timeout=0.05), True, client_factory=lambda: sock)
assert sock.disconnected
+257
View File
@@ -301,3 +301,260 @@ class TestWanQuality:
def test_counts_since(self):
periods = [{'ts': 0, 'state': 'latency'}, {'ts': 300, 'state': 'ok'}, {'ts': 600, 'state': 'loss'}]
assert modem.quality_counts(periods, 300) == {'ok': 1, 'latency': 0, 'loss': 1, 'down': 0}
# Trimmed from a BGW320-505 (firmware 6.40.4)
NAT_PAGE = '''<table class="table60" summary="This table displays a summary of session information.">
<tr>
<th scope="row" nowrap="nowrap">Total sessions available</th>
<td>32767
</td></tr>
<tr>
<th scope="row">Total sessions in use</th>
<td>2858
</td></tr></table>'''
LOGIN_FORM = ('<h1>Access Code Required</h1><form>'
'<input type="hidden" name="nonce" value="0a1b2c3d4e5f60718293a4b5c6d7e8f9" />')
LOGIN_NO_NONCE = '<h1>Access Code Required</h1><strong>Cookies must be enabled</strong>'
DIAG_FORM = ('<form method="post" action="/cgi-bin/diag.ha">'
'<input type="hidden" name="nonce" value="ffee00112233445566778899aabbccdd" />')
PING_DONE = '''PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: seq=0 ttl=59 time=24.697 ms
64 bytes from 1.1.1.1: seq=4 ttl=59 time=12.392 ms
--- 1.1.1.1 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 12.392/20.501/32.697 ms
test done'''
def _diag(window):
return DIAG_FORM + f'<textarea id="progress" name="ProgressWindow" rows="15">{window}</textarea></form>'
class TestModemPages:
def test_login_hash_matches_the_page_script(self):
# The page posts hex_md5(password + nonce)
assert modem.login_hash('abc', '123') == 'e99a18c428cb38d5f260853678922e03'
def test_login_page_is_recognised(self):
assert modem.is_login_page(LOGIN_FORM) and modem.is_login_page(LOGIN_NO_NONCE)
assert not modem.is_login_page(NAT_PAGE)
def test_nat_summary(self):
assert modem.parse_nat_summary(NAT_PAGE) == {'in_use': 2858, 'available': 32767}
assert modem.parse_nat_summary('<table></table>') is None
def test_ping_output(self):
assert modem.parse_ping_output(PING_DONE) == {
'sent': 5, 'received': 5, 'loss_pct': 0, 'min_ms': 12.392, 'avg_ms': 20.501, 'max_ms': 32.697}
def test_ping_still_running_is_none(self):
assert modem.parse_ping_output('PING 1.1.1.1 (1.1.1.1): 56 data bytes\n64 bytes from') is None
def test_ping_all_lost(self):
r = modem.parse_ping_output('5 packets transmitted, 0 packets received, 100% packet loss')
assert r['loss_pct'] == 100 and r['avg_ms'] is None
@pytest.mark.parametrize('gateway,ours,expected', [
({'avg_ms': 900.0, 'loss_pct': 0}, 1000, 'upstream'),
({'avg_ms': 20.0, 'loss_pct': 40}, 1000, 'upstream'),
({'avg_ms': None, 'loss_pct': 100}, None, 'upstream'),
({'avg_ms': 20.0, 'loss_pct': 0}, 1000, 'gateway'),
({'avg_ms': 20.0, 'loss_pct': 0}, 30, 'clear'),
({'avg_ms': 20.0, 'loss_pct': 0}, None, 'clear'),
])
def test_verdict(self, gateway, ours, expected):
assert modem.ping_verdict(gateway, ours) == expected
class FakeGateway:
"""Answers ModemAdmin's requests like the BGW320 does."""
def __init__(self, code='secret', ping_pages=None):
self.code, self.logged_in, self.calls = code, False, []
self.cookie = self.pinged = False
self.ping_pages = list(ping_pages or [_diag(PING_DONE)])
def __call__(self, method, url, **kw):
path = url.split('192.0.2.254', 1)[1]
self.calls.append((method, path))
if path == '/cgi-bin/login.ha' and method == 'GET':
first, self.cookie = not self.cookie, True
return MagicMock(status_code=200, text=LOGIN_NO_NONCE if first else LOGIN_FORM)
if path == '/cgi-bin/login.ha':
ok = kw['data']['hashpassword'] == modem.login_hash(self.code, '0a1b2c3d4e5f60718293a4b5c6d7e8f9')
assert set(kw['data']['password']) == {'*'} # the code itself is never sent
self.logged_in = ok
return MagicMock(status_code=302 if ok else 200, text='' if ok else LOGIN_FORM)
if not self.logged_in:
return MagicMock(status_code=200, text=LOGIN_FORM)
if path == '/cgi-bin/restart.ha' and method == 'GET':
return MagicMock(status_code=200, text=DIAG_FORM.replace('diag.ha', 'restart.ha'))
if path == '/cgi-bin/nattable.ha':
return MagicMock(status_code=200, text=NAT_PAGE)
if path == '/cgi-bin/diag.ha' and method == 'POST':
assert kw['data']['WebAddress'] == '1.1.1.1' and kw['data']['Ping'] == 'Ping'
self.pinged = True
return MagicMock(status_code=200, text=_diag(''))
if path == '/cgi-bin/diag.ha':
# Before the test the window is empty; after it, each refresh shows more output
page = self.ping_pages.pop(0) if self.pinged else _diag('')
return MagicMock(status_code=200, text=page)
raise AssertionError(path)
class TestModemAdmin:
def _admin(self, gw, now=None):
clock = MagicMock(return_value=now or 1000.0)
a = modem.ModemAdmin({'url': 'http://192.0.2.254', 'access_code': 'secret'},
clock=clock, sleep=lambda s: None)
a.session.request = MagicMock(side_effect=gw)
return a, clock
def test_logs_in_when_needed_and_reads_nat(self):
gw = FakeGateway()
a, _ = self._admin(gw)
assert a.nat_sessions() == {'in_use': 2858, 'available': 32767}
assert ('POST', '/cgi-bin/login.ha') in gw.calls
gw.calls.clear()
a.nat_sessions() # session still good: no second login
assert ('POST', '/cgi-bin/login.ha') not in gw.calls
def test_expired_session_logs_in_again(self):
gw = FakeGateway()
a, _ = self._admin(gw)
a.nat_sessions()
gw.logged_in = False
assert a.nat_sessions()['in_use'] == 2858
def test_refused_code_backs_off(self):
gw = FakeGateway(code='other')
a, clock = self._admin(gw)
with pytest.raises(modem.ModemLoginFailed):
a.nat_sessions()
gw.calls.clear()
clock.return_value = 1000.0 + modem.LOGIN_BACKOFF_S - 1
with pytest.raises(modem.ModemLoginFailed):
a.nat_sessions()
assert ('POST', '/cgi-bin/login.ha') not in gw.calls # didn't even try
def test_ping_from_the_gateway_waits_for_the_result(self):
gw = FakeGateway(ping_pages=[_diag('PING 1.1.1.1 (1.1.1.1): 56 data bytes'), _diag(PING_DONE)])
a, _ = self._admin(gw)
r = a.ping('1.1.1.1')
assert r['avg_ms'] == 20.501 and r['target'] == '1.1.1.1'
def test_ping_that_never_finishes(self):
gw = FakeGateway(ping_pages=[_diag('PING ...')] * 50)
a, clock = self._admin(gw)
clock.side_effect = [1000.0] + [2000.0] * 100
with pytest.raises(modem.ModemDataInvalid, match='did not finish'):
a.ping('1.1.1.1')
def test_unreachable(self):
a, _ = self._admin(FakeGateway())
a.session.request = MagicMock(side_effect=requests.ConnectionError('boom'))
with pytest.raises(modem.ModemUnreachable):
a.nat_sessions()
FW_LOG = '''<table><tr><th>No.</th><th>Date/Time</th></tr>
<tr class="b"><td class="heading" scope="row">2</td><td>2026-11-19T18:32:04.984186</td><td>198.51.100.20</td>
<td>203.0.113.7</td><td>TCP</td><td>FLOOD limit 50pps burst 100</td></tr>
<tr class="a"><td class="heading" scope="row">1</td><td>2026-11-19T18:32:02.983868</td><td>203.0.113.7</td>
<td>198.51.100.30</td><td>TCP</td><td>Invalid IP Packet</td></tr>
</table>'''
SPEED_PAGE = '''<h2>Test History</h2><table><thead><th scope="col">Test Completion Time</th></thead><tbody>
<tr class="b"><td scope="row" class="heading" width="47%">
11/06/2026 08:46:05</td>
<td>upstream</td>
<td> 1242.810000</td>
<td> 28</td>
<td> 19.000000</td>
<td> Success</td></tr>
<tr class="a"><td scope="row" class="heading" width="47%">
11/06/2026 08:45:57</td>
<td>downstream</td>
<td> 1167.400000</td>
<td> 28</td>
<td> 28.000000</td>
<td> Success</td></tr>
</tbody></table>'''
class TestModemExtras:
@pytest.mark.parametrize('raw,expected', [
('03:06:29:34', ((3 * 24 + 6) * 60 + 29) * 60 + 34), # firmware 6.40
('28951', 28951), # older firmware
('0:00:05:00', 300),
('', None), (None, None), ('soon', None),
])
def test_uptime_formats(self, raw, expected):
assert modem.parse_uptime(raw) == expected
def test_status_reads_dhms_uptime(self):
sysinfo = SYSINFO.replace('<td class="col2">28951</td>', '<td class="col2">03:06:29:34</td>')
assert sysinfo != SYSINFO
assert modem.build_status(fiber_page(), broadband_page(), sysinfo)['uptime_s'] == 282574
def test_firewall_log_oldest_first(self):
rows = modem.parse_firewall_log(FW_LOG)
assert [r['ts'] for r in rows] == ['2026-11-19T18:32:02.983868', '2026-11-19T18:32:04.984186']
assert rows[1]['reason'] == 'FLOOD limit 50pps burst 100' and rows[1]['proto'] == 'TCP'
@pytest.mark.parametrize('reason,kind', [
('FLOOD limit 50pps burst 100', 'flood'), ('Invalid IP Packet', 'invalid'), ('Port scan', 'other')])
def test_firewall_kind(self, reason, kind):
assert modem.firewall_kind(reason) == kind
def test_speed_tests(self):
assert modem.parse_speed_tests(SPEED_PAGE) == [
{'at': '11/06/2026 08:46:05', 'direction': 'upstream', 'mbps': 1242.8, 'latency_ms': 19.0, 'ok': True},
{'at': '11/06/2026 08:45:57', 'direction': 'downstream', 'mbps': 1167.4, 'latency_ms': 28.0, 'ok': True},
]
def test_speed_tests_missing(self):
assert modem.parse_speed_tests('<html></html>') == []
class TestModemRestart:
def _admin(self):
gw = FakeGateway()
a = modem.ModemAdmin({'url': 'http://192.0.2.254', 'access_code': 'secret'}, sleep=lambda s: None)
a.session.request = MagicMock(side_effect=gw)
return a, gw
REDIRECT = MagicMock(status_code=302, headers={'Location': '/cgi-bin/restarting.ha'}, text='')
RESTARTING = MagicMock(status_code=200, text='<title>Restarting</title> Your AT&T BGW320-505 is restarting.')
def test_posts_the_form_then_loads_the_restarting_page(self):
# Firmware 6.40: the post only redirects; loading restarting.ha starts the restart
a, gw = self._admin()
a.session.post = MagicMock(return_value=self.REDIRECT)
a.session.get = MagicMock(return_value=self.RESTARTING)
assert a.restart() == 'accepted'
assert a.session.post.call_args.kwargs['data'] == {'nonce': 'ffee00112233445566778899aabbccdd',
'Restart': 'Restart'}
assert a.session.post.call_args.args[0].endswith('/cgi-bin/restart.ha')
assert a.session.get.call_args.args[0].endswith('/cgi-bin/restarting.ha')
assert ('POST', '/cgi-bin/login.ha') in gw.calls # logged in to read the form
def test_redirect_alone_is_not_a_restart(self):
a, _ = self._admin()
a.session.post = MagicMock(return_value=self.REDIRECT)
a.session.get = MagicMock(return_value=MagicMock(status_code=200, text='<title>Device</title>'))
with pytest.raises(modem.ModemDataInvalid, match='not started'):
a.restart()
def test_connection_dropped_mid_restart(self):
a, _ = self._admin()
a.session.post = MagicMock(return_value=self.REDIRECT)
a.session.get = MagicMock(side_effect=requests.ConnectionError('reset'))
assert a.restart() == 'unconfirmed'
def test_refused(self):
a, _ = self._admin()
a.session.post = MagicMock(return_value=MagicMock(status_code=200, text=LOGIN_FORM))
with pytest.raises(modem.ModemDataInvalid, match='not started'):
a.restart()
+157
View File
@@ -1048,3 +1048,160 @@ class TestWanQualityCollection:
c = monitor.UnifiClient({'controller': 'https://udm.test', 'api_key': 'k'})
c.session.get = MagicMock(side_effect=RuntimeError('boom'))
assert c.get_wan_dashboard() is None
class TestModemChecks:
NOW = 1_000_000.0
def _run(self, mon, db, checks=None, quality=None, admin=None):
state = {'modem_checks': json.dumps(checks) if checks is not None else None,
'wan_quality': json.dumps(quality) if quality is not None else None}
db.get_state.side_effect = lambda k, d=None: state.get(k, d)
db.set_state.reset_mock()
mon.modem_admin = admin or MagicMock()
if admin is None:
mon.modem_admin.nat_sessions.return_value = {'in_use': 2858, 'available': 32767}
mon.modem_admin.ping.return_value = {'sent': 5, 'received': 5, 'loss_pct': 0, 'min_ms': 12.0,
'avg_ms': 20.0, 'max_ms': 32.0, 'target': '1.1.1.1'}
with patch.object(monitor.time, 'time', return_value=self.NOW):
mon._collect_modem_checks()
saved = [c.args[1] for c in db.set_state.call_args_list if c.args[0] == 'modem_checks']
return saved[-1] if saved else None
def _slow(self):
return {'periods': [{'ts': self.NOW - 300, 'state': 'latency'}], 'latencies': {'cloudflare': 1043}}
def test_no_access_code_does_nothing(self, mon, db):
mon.modem_admin = None
mon._collect_modem_checks()
assert not db.set_state.called
def test_nat_every_half_hour(self, mon, db):
out = self._run(mon, db)
assert out['nat'] == {'in_use': 2858, 'available': 32767, 'at': self.NOW}
assert 'ping' not in out # the internet is fine, so no ping
recent = {'nat': {'at': self.NOW - 60}, 'firewall': {'checked_at': self.NOW - 60}}
assert self._run(mon, db, checks=recent) is None
def test_slow_internet_pings_from_the_modem(self, mon, db):
out = self._run(mon, db, checks={'nat': {'at': self.NOW - 60}}, quality=self._slow())
assert out['ping']['avg_ms'] == 20.0 and out['ping']['ours_ms'] == 1043
assert out['ping']['verdict'] == 'gateway'
mon.modem_admin.ping.assert_called_once_with('1.1.1.1')
def test_ping_at_most_every_ten_minutes(self, mon, db):
checks = {'nat': {'at': self.NOW - 60}, 'ping': {'at': self.NOW - 300},
'firewall': {'checked_at': self.NOW - 60}}
assert self._run(mon, db, checks=checks, quality=self._slow()) is None
def test_failure_is_recorded_and_paused(self, mon, db):
admin = MagicMock()
admin.nat_sessions.side_effect = modem_api.ModemLoginFailed('refused')
out = self._run(mon, db, admin=admin)
assert out['error']['login'] is True and out['error']['at'] == self.NOW
admin.nat_sessions.reset_mock()
db.set_state.reset_mock()
assert self._run(mon, db, checks=out, admin=admin) is None
assert not admin.nat_sessions.called
def test_success_clears_the_error(self, mon, db):
out = self._run(mon, db, checks={'error': {'message': 'x', 'at': self.NOW - 3600, 'login': True}})
assert out['error'] is None
class TestModemExtrasCollection:
def _row(self, ts, reason='FLOOD limit 50pps burst 100'):
return {'ts': ts, 'src': 'a', 'dst': 'b', 'proto': 'TCP', 'reason': reason}
def test_counts_only_new_rows_per_hour(self, mon):
first = mon._count_firewall_drops(None, [self._row('2026-11-19T17:59:58.1'),
self._row('2026-11-19T18:00:01.2', 'Invalid IP Packet')], 100.0)
assert first['hours'] == {'2026-11-19T17': {'flood': 1, 'invalid': 0, 'other': 0},
'2026-11-19T18': {'flood': 0, 'invalid': 1, 'other': 0}}
assert first['last'] == '2026-11-19T18:00:01.2' and first['gap'] is False
# The next read overlaps the last one: only the new row counts
second = mon._count_firewall_drops(first, [self._row('2026-11-19T18:00:01.2', 'Invalid IP Packet'),
self._row('2026-11-19T18:00:03.0')], 400.0)
assert second['hours']['2026-11-19T18'] == {'flood': 1, 'invalid': 1, 'other': 0}
assert second['checked_at'] == 400.0
def test_gap_when_the_log_rolled_over(self, mon):
prev = {'hours': {}, 'last': '2026-11-19T18:00:00.0'}
out = mon._count_firewall_drops(prev, [self._row('2026-11-19T18:20:00.0')], 1.0)
assert out['gap'] is True
def test_keeps_a_day(self, mon):
prev = {'hours': {f'2026-11-18T{h:02d}': {'flood': 1, 'invalid': 0, 'other': 0} for h in range(24)},
'last': '2026-11-18T23:59:59.0'}
out = mon._count_firewall_drops(prev, [self._row('2026-11-19T00:00:01.0')], 1.0)
assert len(out['hours']) == 24 and '2026-11-18T00' not in out['hours']
def test_latest_speed_tests(self, mon):
tests = [{'direction': 'upstream', 'mbps': 1242.8, 'at': 'b'},
{'direction': 'downstream', 'mbps': 1167.4, 'at': 'b'},
{'direction': 'upstream', 'mbps': 1.0, 'at': 'a'}]
out = mon._latest_speed_tests(tests, 5.0)
assert out['up']['mbps'] == 1242.8 and out['down']['mbps'] == 1167.4 and out['checked_at'] == 5.0
def test_scheduled_with_the_other_checks(self, mon, db):
admin = MagicMock()
admin.nat_sessions.return_value = {'in_use': 1, 'available': 10}
admin.speed_tests.return_value = []
admin.firewall_log.return_value = [self._row('2026-11-19T18:00:00.0')]
mon.modem_admin = admin
db.get_state.side_effect = lambda k, d=None: None
mon._collect_modem_checks()
saved = [c.args[1] for c in db.set_state.call_args_list if c.args[0] == 'modem_checks'][-1]
assert saved['firewall']['hours'] == {'2026-11-19T18': {'flood': 1, 'invalid': 0, 'other': 0}}
assert 'speed' in saved and admin.firewall_log.called
class TestFinishModemRestart:
NOW = 1_000_000.0
def _run(self, mon, db, restart, stats, kuma_error=None):
state = {'modem_restart': json.dumps(restart), 'modem_stats': json.dumps(stats)}
db.get_state.side_effect = lambda k, d=None: state.get(k, d)
db.set_state.reset_mock()
mon.cfg = {'uptime_kuma': {'url': 'http://kuma.test'}}
with patch.object(monitor.time, 'time', return_value=self.NOW), \
patch.object(monitor.kuma, 'set_maintenance', side_effect=kuma_error) as km:
mon._finish_modem_restart()
saved = [c.args[1] for c in db.set_state.call_args_list if c.args[0] == 'modem_restart']
return (saved[-1] if saved else None), km
def _restart(self, ago, **kw):
return dict({'at': self.NOW - ago, 'by': 'jared', 'suppression_id': 9, 'kuma': True,
'finished_at': None}, **kw)
def test_back_ends_kuma_and_the_silence(self, mon, db):
out, km = self._run(mon, db, self._restart(300), {'reachable': True, 'wan_problems': [], 'uptime_s': 200})
km.assert_called_once_with({'url': 'http://kuma.test'}, False)
db.deactivate_suppression.assert_called_once_with(9)
assert out['back'] and out['rebooted'] and out['back_after_s'] == 300 and out['finished_at'] == self.NOW
def test_waits_while_the_modem_is_down(self, mon, db):
out, km = self._run(mon, db, self._restart(300), {'reachable': False})
assert out is None and not km.called and not db.deactivate_suppression.called
def test_too_soon_to_tell(self, mon, db):
out, _ = self._run(mon, db, self._restart(60), {'reachable': True, 'wan_problems': [], 'uptime_s': 30})
assert out is None
def test_answering_without_having_rebooted_is_not_back(self, mon, db):
out, _ = self._run(mon, db, self._restart(300), {'reachable': True, 'wan_problems': [], 'uptime_s': 99999})
assert out is None
def test_gives_up_after_half_an_hour(self, mon, db):
out, km = self._run(mon, db, self._restart(1900), {'reachable': False})
assert out['back'] is False and out['back_after_s'] is None and km.called
db.deactivate_suppression.assert_called_once_with(9)
def test_kuma_failure_retries_next_cycle(self, mon, db):
out, _ = self._run(mon, db, self._restart(300), {'reachable': True, 'wan_problems': [], 'uptime_s': 200},
kuma_error=monitor.kuma.KumaError('down'))
assert out is None and not db.deactivate_suppression.called
def test_finished_restart_is_left_alone(self, mon, db):
out, km = self._run(mon, db, self._restart(300, finished_at=1.0), {'reachable': True})
assert out is None and not km.called
+1 -1
View File
@@ -62,7 +62,7 @@ def link(text: str, url: str) -> str:
def port_text(switch: str, port_name: str, port: dict) -> str:
"""'8SFP-AGG port 7 "Large1": up at 10 Gbps, DAC'"""
"""'8SFP-AGG port 7 "compute-storage-02": up at 10 Gbps, DAC'"""
idx = port.get('port_idx')
text = f'{switch} port {idx}' if idx is not None else f'{switch} {port_name}'
if port_name and idx is not None and port_name != f'Port {idx}':
+2 -2
View File
@@ -25,7 +25,7 @@ _TOKEN_RE = re.compile(r'[a-z0-9]+')
def _norm(name: str) -> str:
"""A name with case, spaces, dashes and underscores ignored: "Large1" == "large-1"."""
"""A name with case, spaces, dashes and underscores ignored: "Storage-01" == "storage_01"."""
return ''.join(_TOKEN_RE.findall((name or '').lower()))
@@ -114,7 +114,7 @@ def name_mismatch(port_name: str, neighbour: str, own: str, known: Dict[str, str
"""A port named after a different device than the one plugged into it.
known: {normalised name: display name} of every device. Runs of up to four words of
the port name are compared with them, so "Large1 Management" names large1 and
the port name are compared with them, so "Storage-01 Management" names storage-01 and
"Downlink to 24RJ45-POE" names 24RJ45-POE. Ports named only "Port 4" never match.
"""
if not port_name or _is_generic_port_name(port_name):