Compare commits
7
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
1444562a5d | ||
|
|
dd9ba41479 | ||
|
|
3afd85626b | ||
|
|
c2ff072eb2 | ||
|
|
f097e4882c | ||
|
|
0370358363 | ||
|
|
2a9111d8e8 |
No files matched your search
@@ -498,6 +498,10 @@ Those limits are wide (warning at -27.9 dBm on a link that normally sits near -1
|
|||||||
compares each reading to the link's own 7-day median to catch a damaged connector earlier. It stays quiet
|
compares each reading to the link's own 7-day median to catch a damaged connector earlier. It stays quiet
|
||||||
until `rx_drift_min_samples` readings exist (about a day at a 5 minute poll).
|
until `rx_drift_min_samples` readings exist (about a day at a 5 minute poll).
|
||||||
|
|
||||||
|
Uptime comes from the System Information page without a login. Firmware 6.40 shows it as D:HH:MM:SS
|
||||||
|
(older firmware sent seconds; both are read). Uptime going backwards between polls records a restart, and
|
||||||
|
the panel shows it (*restarted ... by ...*).
|
||||||
|
|
||||||
Outages can be over between polls: two real ones lasted about two minutes against a 5 minute poll.
|
Outages can be over between polls: two real ones lasted about two minutes against a 5 minute poll.
|
||||||
So WAN outages and modem restarts also come from the UniFi system log, and a drop in the modem's uptime
|
So WAN outages and modem restarts also come from the UniFi system log, and a drop in the modem's uptime
|
||||||
records a restart even if nothing logged it. A single short outage only appears in the panel's history.
|
records a restart even if nothing logged it. A single short outage only appears in the panel's history.
|
||||||
@@ -512,7 +516,7 @@ the last hour. These are dashboard lines, not tickets, and they go once the hour
|
|||||||
than 30 minutes is ignored.
|
than 30 minutes is ignored.
|
||||||
|
|
||||||
**With `modem.access_code` set**, the monitor also logs in to the gateway (`ModemAdmin` in `modem.py`)
|
**With `modem.access_code` set**, the monitor also logs in to the gateway (`ModemAdmin` in `modem.py`)
|
||||||
for two read-only checks, stored in `monitor_state.modem_checks`:
|
for read-only checks, stored in `monitor_state.modem_checks`:
|
||||||
- **NAT sessions** every 30 minutes, from the NAT Table page. Needs attention warns when 80% of the
|
- **NAT sessions** every 30 minutes, from the NAT Table page. Needs attention warns when 80% of the
|
||||||
gateway's sessions (32,767 on a BGW320) are in use, since new connections stall once it is full.
|
gateway's sessions (32,767 on a BGW320) are in use, since new connections stall once it is full.
|
||||||
- **A ping from the gateway itself** to 1.1.1.1, from the Troubleshoot page, whenever UniFi flags the
|
- **A ping from the gateway itself** to 1.1.1.1, from the Troubleshoot page, whenever UniFi flags the
|
||||||
@@ -521,6 +525,22 @@ for two read-only checks, stored in `monitor_state.modem_checks`:
|
|||||||
ping is slow too, so the provider), *in the modem* (its ping is fine but ours through it is slow), or
|
ping is slow too, so the provider), *in the modem* (its ping is fine but ours through it is slow), or
|
||||||
*fine by then*. The result goes in the detail of the Internet line in Needs attention and on the
|
*fine by then*. The result goes in the detail of the Internet line in Needs attention and on the
|
||||||
WAN / Fiber panel.
|
WAN / Fiber panel.
|
||||||
|
- **Firewall drops** every 5 minutes, from the Logs page, counted per hour for a day. Most are *FLOOD
|
||||||
|
limit 50pps*: AT&T's firmware caps new incoming connections at 50 per second (visitors to hosted
|
||||||
|
services, peers), with no setting to change it. The log records at most one drop every 2 seconds and
|
||||||
|
keeps about ten minutes, so the counts are a lower bound. Shown on the panel only.
|
||||||
|
- **The gateway's own speed tests** (AT&T runs them from the modem), read with the NAT check; the latest
|
||||||
|
down and up results are shown on the panel as a reference for what the line itself delivers.
|
||||||
|
|
||||||
|
**Restart modem** (admins, WAN / Fiber panel, needs `modem.access_code`): after a confirmation it
|
||||||
|
1. switches the `uptime_kuma` maintenance window on (if that fails it stops and asks whether to go on
|
||||||
|
without it),
|
||||||
|
2. starts Gandalf maintenance (a global suppression for up to 20 minutes, shown in the banner),
|
||||||
|
3. posts the gateway's Restart Device form. If that fails, steps 1 and 2 are undone.
|
||||||
|
|
||||||
|
The monitor then waits for the gateway to answer with its WAN up and an uptime showing it restarted
|
||||||
|
(at least 3 minutes later), switches the Kuma window off, ends the suppression, and records how long it
|
||||||
|
took (`monitor_state.modem_restart`). After 30 minutes it ends both regardless. One restart at a time.
|
||||||
|
|
||||||
The login posts md5(access code + nonce), as the gateway's own page does, so the code itself is never sent.
|
The login posts md5(access code + nonce), as the gateway's own page does, so the code itself is never sent.
|
||||||
The session is kept between checks and renewed when it expires. A refused code is reported in Needs
|
The session is kept between checks and renewed when it expires. A refused code is reported in Needs
|
||||||
@@ -670,6 +690,7 @@ Shared by both processes. Located in the working directory (`/var/www/html/prod/
|
|||||||
| `modem.failure_threshold` | Consecutive failures before ticketing a modem alert (default: same as `failure_threshold`). Raise it if gateway reboots or automatic firmware updates cut tickets. |
|
| `modem.failure_threshold` | Consecutive failures before ticketing a modem alert (default: same as `failure_threshold`). Raise it if gateway reboots or automatic firmware updates cut tickets. |
|
||||||
| `modem.rx_drift_db` | Rx drop below the 7-day median that raises `fiber_rx_drift` (default: 3.0) |
|
| `modem.rx_drift_db` | Rx drop below the 7-day median that raises `fiber_rx_drift` (default: 3.0) |
|
||||||
| `modem.rx_drift_min_samples` | Readings required before drift alerts start (default: 288) |
|
| `modem.rx_drift_min_samples` | Readings required before drift alerts start (default: 288) |
|
||||||
|
| `uptime_kuma.url`, `.token`, `.maintenance_id` | Optional. Lets the Restart modem button switch an Uptime Kuma maintenance window on and off (Kuma 2.x, over its Socket.IO API). `token` is a Kuma session token, which works with a two-factor account and stops working if the Kuma password changes; `username` and `password` can be used instead for an account without two-factor login. Create the window once in Kuma with the *manual* strategy and the monitors a modem restart takes down, pause it, and put its id here. |
|
||||||
| `modem.access_code` | Optional. The gateway's Device Access Code (printed on its label). With it, Gandalf logs in to read NAT session counts and to ping from the gateway when the internet is slow (see WAN modem notes). Keep it only in `config.json`, which is git-ignored and mode 600. |
|
| `modem.access_code` | Optional. The gateway's Device Access Code (printed on its label). With it, Gandalf logs in to read NAT session counts and to ping from the gateway when the internet is slow (see WAN modem notes). Keep it only in `config.json`, which is git-ignored and mode 600. |
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -855,7 +876,8 @@ curl http://10.10.10.48:9090/api/v1/query?query=up | jq '.data.result[] | select
|
|||||||
gandalf/
|
gandalf/
|
||||||
├── app.py # Flask web app (routes, auth, API endpoints)
|
├── app.py # Flask web app (routes, auth, API endpoints)
|
||||||
├── monitor.py # Background daemon (Prometheus, UniFi, Pulse, alert logic)
|
├── monitor.py # Background daemon (Prometheus, UniFi, Pulse, alert logic)
|
||||||
├── modem.py # AT&T gateway client: status pages, plus optional logged-in NAT and ping checks
|
├── kuma.py # Switches one Uptime Kuma maintenance window on/off (Restart modem)
|
||||||
|
├── modem.py # AT&T gateway client: status pages, plus optional logged-in checks and restart
|
||||||
├── devices.py # Device inventory for search and device pages
|
├── devices.py # Device inventory for search and device pages
|
||||||
├── topology.py # The topology graph: devices, cables, depth, last-seen links
|
├── topology.py # The topology graph: devices, cables, depth, last-seen links
|
||||||
├── cutsheet.py # Cutsheet validation, path tracing, plan-vs-actual checks, CSV
|
├── cutsheet.py # Cutsheet validation, path tracing, plan-vs-actual checks, CSV
|
||||||
@@ -937,7 +959,7 @@ Take LED colors from the model's quick start guide. See the comment at the top o
|
|||||||
`recovery_notified` once the recovery note is on its ticket (column added automatically at monitor startup)
|
`recovery_notified` once the recovery note is on its ticket (column added automatically at monitor startup)
|
||||||
- `suppression_rules`: `active=FALSE` when removed; `expires_at` checked at query time
|
- `suppression_rules`: `active=FALSE` when removed; `expires_at` checked at query time
|
||||||
- `monitor_state`: key/value store; `interface_baseline`, `link_stats`, and `modem_stats` are JSON blobs
|
- `monitor_state`: key/value store; `interface_baseline`, `link_stats`, and `modem_stats` are JSON blobs
|
||||||
- `monitor_state` also holds `wan_history` (WAN outages, modem restarts), `wan_quality` (UniFi's 5 minute WAN health timeline), `modem_checks` (NAT sessions and the latest ping from the gateway) and `unifi_log_cursor` (how far the UniFi log has been read)
|
- `monitor_state` also holds `wan_history` (WAN outages, modem restarts), `wan_quality` (UniFi's 5 minute WAN health timeline), `modem_checks` (NAT sessions, the latest ping from the gateway, hourly firewall drops and its speed test results) and `unifi_log_cursor` (how far the UniFi log has been read)
|
||||||
- Database sessions run in UTC (`db.py`), whatever the server's time zone
|
- Database sessions run in UTC (`db.py`), whatever the server's time zone
|
||||||
- `modem_samples`: optical readings per poll, kept 90 days; created automatically at monitor startup
|
- `modem_samples`: optical readings per poll, kept 90 days; created automatically at monitor startup
|
||||||
- `cutsheet_items`, `cutsheet_cables`, `cutsheet_port_labels`, `cutsheet_history`: cutsheet records and
|
- `cutsheet_items`, `cutsheet_cables`, `cutsheet_port_labels`, `cutsheet_history`: cutsheet records and
|
||||||
|
|||||||
@@ -27,6 +27,8 @@ import cutsheet
|
|||||||
import db
|
import db
|
||||||
import devices
|
import devices
|
||||||
import diagnose
|
import diagnose
|
||||||
|
import kuma
|
||||||
|
import modem as modem_api
|
||||||
import topology
|
import topology
|
||||||
from monitor import PulseClient, UnifiClient
|
from monitor import PulseClient, UnifiClient
|
||||||
|
|
||||||
@@ -557,10 +559,71 @@ def api_modem():
|
|||||||
data['wan_history'] = _state_json('wan_history')
|
data['wan_history'] = _state_json('wan_history')
|
||||||
data['wan_quality'] = _state_json('wan_quality') or None
|
data['wan_quality'] = _state_json('wan_quality') or None
|
||||||
data['modem_checks'] = _state_json('modem_checks') or None
|
data['modem_checks'] = _state_json('modem_checks') or None
|
||||||
|
data['modem_restart'] = _state_json('modem_restart') or None
|
||||||
|
data['can_restart'] = bool((_config().get('modem') or {}).get('access_code'))
|
||||||
data['configured'] = True
|
data['configured'] = True
|
||||||
return jsonify(data)
|
return jsonify(data)
|
||||||
|
|
||||||
|
|
||||||
|
# How long the silence lasts if the monitor never sees the modem come back to end it sooner
|
||||||
|
MODEM_RESTART_SILENCE_MIN = 20
|
||||||
|
|
||||||
|
|
||||||
|
@app.route('/api/modem/restart', methods=['POST'])
|
||||||
|
@require_auth
|
||||||
|
@require_admin
|
||||||
|
def api_modem_restart():
|
||||||
|
"""Restart the AT&T gateway, with alerts silenced first.
|
||||||
|
|
||||||
|
Body: {"confirm": true} and, after a Kuma failure the admin chose to accept,
|
||||||
|
"without_kuma": true. Order: Uptime Kuma maintenance on, Gandalf maintenance on, then
|
||||||
|
the restart; the first two are undone if the restart can't be sent. The monitor ends
|
||||||
|
both once the modem is back (monitor._finish_modem_restart).
|
||||||
|
"""
|
||||||
|
body = _json_body()
|
||||||
|
if body.get('confirm') is not True:
|
||||||
|
return jsonify({'error': 'confirm must be true'}), 400
|
||||||
|
cfg = _config()
|
||||||
|
modem_cfg = cfg.get('modem') or {}
|
||||||
|
if not modem_cfg.get('url') or not modem_cfg.get('access_code'):
|
||||||
|
return jsonify({'error': 'Set modem.access_code in config.json first'}), 400
|
||||||
|
pending = _state_json('modem_restart')
|
||||||
|
if pending and not pending.get('finished_at') and time.time() - pending.get('at', 0) < 1800:
|
||||||
|
return jsonify({'error': 'A restart is already in progress'}), 409
|
||||||
|
|
||||||
|
user = _get_user()['username']
|
||||||
|
kuma_cfg = cfg.get('uptime_kuma') or {}
|
||||||
|
kuma_on = False
|
||||||
|
if kuma.configured(kuma_cfg):
|
||||||
|
try:
|
||||||
|
kuma.set_maintenance(kuma_cfg, True)
|
||||||
|
kuma_on = True
|
||||||
|
except kuma.KumaError as e:
|
||||||
|
logger.warning(f'Uptime Kuma maintenance could not be started: {e}')
|
||||||
|
if not body.get('without_kuma'):
|
||||||
|
return jsonify({'error': f'Uptime Kuma: {e}', 'kuma_failed': True}), 502
|
||||||
|
|
||||||
|
sup_id = db.create_suppression('all', '', '', 'Modem restart from Gandalf', user,
|
||||||
|
MODEM_RESTART_SILENCE_MIN)
|
||||||
|
try:
|
||||||
|
result = modem_api.ModemAdmin(modem_cfg).restart()
|
||||||
|
except (modem_api.ModemUnreachable, modem_api.ModemDataInvalid, modem_api.ModemLoginFailed) as e:
|
||||||
|
logger.error(f'Modem restart failed: {e}')
|
||||||
|
db.deactivate_suppression(sup_id)
|
||||||
|
if kuma_on:
|
||||||
|
try:
|
||||||
|
kuma.set_maintenance(kuma_cfg, False)
|
||||||
|
except kuma.KumaError as ke:
|
||||||
|
logger.error(f'Uptime Kuma maintenance could not be ended: {ke}')
|
||||||
|
return jsonify({'error': f'The modem did not restart: {e}'}), 502
|
||||||
|
|
||||||
|
db.set_state('modem_restart', {'at': time.time(), 'by': user, 'result': result,
|
||||||
|
'suppression_id': sup_id, 'kuma': kuma_on, 'finished_at': None})
|
||||||
|
logger.warning(f'Modem restart requested by {user} ({result}); Kuma maintenance '
|
||||||
|
f'{"on" if kuma_on else "not set"}, suppression #{sup_id}')
|
||||||
|
return jsonify({'ok': True, 'result': result, 'kuma': kuma_on, 'suppression_id': sup_id})
|
||||||
|
|
||||||
|
|
||||||
@app.route('/api/devices')
|
@app.route('/api/devices')
|
||||||
@require_auth
|
@require_auth
|
||||||
def api_devices():
|
def api_devices():
|
||||||
|
|||||||
@@ -0,0 +1,74 @@
|
|||||||
|
"""Switch one Uptime Kuma maintenance window on and off (Kuma 2.x, over its Socket.IO API).
|
||||||
|
|
||||||
|
Kuma has no REST API for maintenance, so this signs in over the same socket its web UI
|
||||||
|
uses. It only ever resumes or pauses the one window named in config, which is created
|
||||||
|
once in Kuma ("manual" strategy, covering the monitors that a modem restart takes down):
|
||||||
|
|
||||||
|
"uptime_kuma": {"url": "http://10.10.10.25:3001", "token": "...", "maintenance_id": 7}
|
||||||
|
|
||||||
|
`token` is a Kuma session token (works with a two-factor account; it stops working if
|
||||||
|
the Kuma password changes). `username` and `password` are accepted instead for an
|
||||||
|
account without two-factor login. Nothing else in Kuma is read or changed.
|
||||||
|
"""
|
||||||
|
import logging
|
||||||
|
import threading
|
||||||
|
from typing import Callable, Optional
|
||||||
|
|
||||||
|
import socketio
|
||||||
|
|
||||||
|
logger = logging.getLogger('gandalf.kuma')
|
||||||
|
|
||||||
|
|
||||||
|
class KumaError(Exception):
|
||||||
|
"""Kuma couldn't be reached, refused the sign-in, or refused the change."""
|
||||||
|
|
||||||
|
|
||||||
|
def configured(cfg: Optional[dict]) -> bool:
|
||||||
|
cfg = cfg or {}
|
||||||
|
signed_in = bool(cfg.get('token')) or bool(cfg.get('username') and cfg.get('password'))
|
||||||
|
return bool(cfg.get('url') and cfg.get('maintenance_id')) and signed_in
|
||||||
|
|
||||||
|
|
||||||
|
def _sign_in(sio, cfg: dict, timeout: float) -> None:
|
||||||
|
if cfg.get('token'):
|
||||||
|
res = sio.call('loginByToken', cfg['token'], timeout=timeout) or {}
|
||||||
|
if not res.get('ok'):
|
||||||
|
raise KumaError('token rejected (Kuma password changed?)')
|
||||||
|
return
|
||||||
|
res = sio.call('login', {'username': cfg['username'], 'password': cfg['password'], 'token': ''},
|
||||||
|
timeout=timeout) or {}
|
||||||
|
if not res.get('ok'):
|
||||||
|
if res.get('tokenRequired'):
|
||||||
|
raise KumaError('the Kuma account has two-factor login: use uptime_kuma.token instead')
|
||||||
|
raise KumaError(f'login refused: {res.get("msg") or "no reason given"}')
|
||||||
|
|
||||||
|
|
||||||
|
def set_maintenance(cfg: dict, active: bool, client_factory: Optional[Callable] = None) -> None:
|
||||||
|
"""Resume (active=True) or pause the configured maintenance window. Raises KumaError."""
|
||||||
|
timeout = cfg.get('timeout', 10)
|
||||||
|
factory = client_factory or (lambda: socketio.Client(
|
||||||
|
reconnection=False, ssl_verify=cfg.get('verify_ssl', True)))
|
||||||
|
sio = factory()
|
||||||
|
# Kuma registers its event handlers only after it sends 'info', so anything emitted
|
||||||
|
# before that is silently dropped and the call times out.
|
||||||
|
ready = threading.Event()
|
||||||
|
sio.on('info', lambda *args: ready.set())
|
||||||
|
try:
|
||||||
|
sio.connect(cfg['url'], transports=['websocket'], wait_timeout=timeout)
|
||||||
|
if not ready.wait(timeout):
|
||||||
|
raise KumaError('Kuma not ready (no info event after connecting)')
|
||||||
|
_sign_in(sio, cfg, timeout)
|
||||||
|
event = 'resumeMaintenance' if active else 'pauseMaintenance'
|
||||||
|
res = sio.call(event, int(cfg['maintenance_id']), timeout=timeout) or {}
|
||||||
|
if not res.get('ok'):
|
||||||
|
raise KumaError(f'{event} refused: {res.get("msg") or "no reason given"}')
|
||||||
|
logger.info(f'Uptime Kuma maintenance {cfg["maintenance_id"]} {"on" if active else "off"}')
|
||||||
|
except KumaError:
|
||||||
|
raise
|
||||||
|
except Exception as e: # socketio raises its own errors and plain timeouts
|
||||||
|
raise KumaError(f'{type(e).__name__}: {e}') from e
|
||||||
|
finally:
|
||||||
|
try:
|
||||||
|
sio.disconnect()
|
||||||
|
except Exception: # nosec B110 - nothing left to clean up
|
||||||
|
pass
|
||||||
@@ -1,17 +1,20 @@
|
|||||||
"""Read-only client for AT&T BGW-series fiber gateways (tested on BGW320-505).
|
"""Client for AT&T BGW-series fiber gateways (tested on BGW320-505).
|
||||||
|
|
||||||
The gateway has no API, but its status pages are served without a login, so
|
The gateway has no API, but its status pages are served without a login, so
|
||||||
this scrapes them over HTTP. Only GET requests to the fixed status pages below
|
ModemClient scrapes them over HTTP with GET requests only: it cannot change
|
||||||
are ever made: nothing here can change gateway settings.
|
anything on the gateway.
|
||||||
|
|
||||||
With an access code configured, ModemAdmin (at the end) also logs in to read the
|
With an access code configured, ModemAdmin (at the end) also logs in to read the
|
||||||
NAT session counts and to run the Troubleshoot page's ping from the gateway.
|
NAT session counts, the firewall log and speed test history, and to run the
|
||||||
Those are a login, page reads and a ping test: still no setting is changed.
|
Troubleshoot page's ping from the gateway. None of that changes a setting. Its
|
||||||
|
restart() is the one call that does something to the gateway, and it runs only
|
||||||
|
when an admin presses Restart modem.
|
||||||
|
|
||||||
Units come from the gateway's own help text, checked against live readings:
|
Units come from the gateway's own help text, checked against live readings:
|
||||||
Rx/Tx power tenths of a dBm (-169 -> -16.9 dBm)
|
Rx/Tx power tenths of a dBm (-169 -> -16.9 dBm)
|
||||||
Temperature degrees C
|
Temperature degrees C
|
||||||
Uptime seconds (despite the help text claiming D:H:M:S)
|
Uptime D:HH:MM:SS on firmware 6.40 (older firmware sent plain seconds;
|
||||||
|
both are accepted)
|
||||||
|
|
||||||
Deliberately not used, because the page disagrees with itself:
|
Deliberately not used, because the page disagrees with itself:
|
||||||
Vcc / Tx Bias readings values do not match the documented scales, so only
|
Vcc / Tx Bias readings values do not match the documented scales, so only
|
||||||
@@ -142,6 +145,18 @@ def _to_int(raw: Optional[str]) -> Optional[int]:
|
|||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def parse_uptime(raw: Optional[str]) -> Optional[int]:
|
||||||
|
"""Seconds since the gateway restarted, from 'D:HH:MM:SS' (e.g. 03:06:29:34) or plain seconds."""
|
||||||
|
raw = (raw or '').strip()
|
||||||
|
if re.fullmatch(r'\d+', raw):
|
||||||
|
return int(raw)
|
||||||
|
m = re.fullmatch(r'(\d+):(\d{1,2}):(\d{1,2}):(\d{1,2})', raw)
|
||||||
|
if not m:
|
||||||
|
return None
|
||||||
|
d, h, mi, sec = (int(x) for x in m.groups())
|
||||||
|
return ((d * 24 + h) * 60 + mi) * 60 + sec
|
||||||
|
|
||||||
|
|
||||||
def wan_problems(data: dict) -> List[str]:
|
def wan_problems(data: dict) -> List[str]:
|
||||||
"""Return human-readable reasons the WAN is down; empty when it looks healthy."""
|
"""Return human-readable reasons the WAN is down; empty when it looks healthy."""
|
||||||
problems = []
|
problems = []
|
||||||
@@ -170,7 +185,7 @@ def build_status(fiber_page: str, broadband_page: str, sysinfo_page: str = '') -
|
|||||||
data = {
|
data = {
|
||||||
'model': sysinfo.get('Model Number', ''),
|
'model': sysinfo.get('Model Number', ''),
|
||||||
'firmware': sysinfo.get('Software Version', ''),
|
'firmware': sysinfo.get('Software Version', ''),
|
||||||
'uptime_s': _to_int(sysinfo.get('Time Since Last Reboot')),
|
'uptime_s': parse_uptime(sysinfo.get('Time Since Last Reboot')),
|
||||||
'broadband_state': broadband.get('Broadband Connection', ''),
|
'broadband_state': broadband.get('Broadband Connection', ''),
|
||||||
'source': broadband.get('Broadband Connection Source', ''),
|
'source': broadband.get('Broadband Connection Source', ''),
|
||||||
'wan_ip': broadband.get('Broadband IPv4 Address', ''),
|
'wan_ip': broadband.get('Broadband IPv4 Address', ''),
|
||||||
@@ -376,6 +391,14 @@ def quality_counts(periods: List[dict], since: float) -> Dict[str, int]:
|
|||||||
_LOGIN_PAGE = '/cgi-bin/login.ha'
|
_LOGIN_PAGE = '/cgi-bin/login.ha'
|
||||||
_NAT_PAGE = '/cgi-bin/nattable.ha'
|
_NAT_PAGE = '/cgi-bin/nattable.ha'
|
||||||
_DIAG_PAGE = '/cgi-bin/diag.ha'
|
_DIAG_PAGE = '/cgi-bin/diag.ha'
|
||||||
|
_LOGS_PAGE = '/cgi-bin/logs.ha'
|
||||||
|
_SPEED_PAGE = '/cgi-bin/speed.ha'
|
||||||
|
_RESTART_PAGE = '/cgi-bin/restart.ha'
|
||||||
|
_RESTARTING_PAGE = '/cgi-bin/restarting.ha'
|
||||||
|
# Firewall log row: number, ISO time (gateway's local clock), source, destination, protocol, reason
|
||||||
|
_FW_ROW_RE = re.compile(r'<tr[^>]*>\s*<td[^>]*>\s*\d+\s*</td>\s*<td>\s*([^<]+?)\s*</td>\s*<td>\s*([^<]*?)\s*</td>'
|
||||||
|
r'\s*<td>\s*([^<]*?)\s*</td>\s*<td>\s*([^<]*?)\s*</td>\s*<td>\s*([^<]*?)\s*</td>\s*</tr>', re.I)
|
||||||
|
_CELL_RE = re.compile(r'<td\b[^>]*>(.*?)</td>', re.I | re.S)
|
||||||
_NONCE_RE = re.compile(r'name="nonce"\s+value="([0-9a-f]+)"', re.I)
|
_NONCE_RE = re.compile(r'name="nonce"\s+value="([0-9a-f]+)"', re.I)
|
||||||
_PROGRESS_RE = re.compile(r'<textarea[^>]*name="ProgressWindow"[^>]*>(.*?)</textarea>', re.I | re.S)
|
_PROGRESS_RE = re.compile(r'<textarea[^>]*name="ProgressWindow"[^>]*>(.*?)</textarea>', re.I | re.S)
|
||||||
_PING_STATS_RE = re.compile(r'(\d+) packets transmitted, (\d+) (?:packets )?received')
|
_PING_STATS_RE = re.compile(r'(\d+) packets transmitted, (\d+) (?:packets )?received')
|
||||||
@@ -422,6 +445,36 @@ def parse_ping_output(text: str) -> Optional[dict]:
|
|||||||
'min_ms': lo, 'avg_ms': avg, 'max_ms': hi}
|
'min_ms': lo, 'avg_ms': avg, 'max_ms': hi}
|
||||||
|
|
||||||
|
|
||||||
|
def parse_firewall_log(page: str) -> List[dict]:
|
||||||
|
"""Rows of the gateway's firewall log (Logs page): {'ts', 'src', 'dst', 'proto', 'reason'}.
|
||||||
|
'ts' is the gateway's local time as ISO text, so rows sort by it."""
|
||||||
|
rows = [dict(zip(('ts', 'src', 'dst', 'proto', 'reason'), m.groups())) for m in _FW_ROW_RE.finditer(page)]
|
||||||
|
return sorted((r for r in rows if re.match(r'\d{4}-\d\d-\d\dT', r['ts'])), key=lambda r: r['ts'])
|
||||||
|
|
||||||
|
|
||||||
|
def firewall_kind(reason: str) -> str:
|
||||||
|
"""'flood' (new connections over the gateway's fixed rate limit), 'invalid', or 'other'."""
|
||||||
|
reason = reason.lower()
|
||||||
|
return 'flood' if 'flood' in reason else 'invalid' if 'invalid' in reason else 'other'
|
||||||
|
|
||||||
|
|
||||||
|
def parse_speed_tests(page: str) -> List[dict]:
|
||||||
|
"""The gateway's own speed test history, newest first:
|
||||||
|
{'at' (as the page shows it), 'direction', 'mbps', 'latency_ms', 'ok'}."""
|
||||||
|
i = page.find('Test History')
|
||||||
|
out = []
|
||||||
|
for row in re.findall(r'<tr\b[^>]*>(.*?)</tr>', page[i:] if i >= 0 else '', re.I | re.S):
|
||||||
|
cells = [_TAG_RE.sub('', c).strip() for c in _CELL_RE.findall(row)]
|
||||||
|
if len(cells) < 6 or cells[1] not in ('upstream', 'downstream'):
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
out.append({'at': cells[0], 'direction': cells[1], 'mbps': round(float(cells[2]), 1),
|
||||||
|
'latency_ms': round(float(cells[4]), 1), 'ok': cells[5].lower() == 'success'})
|
||||||
|
except ValueError:
|
||||||
|
continue
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
class ModemAdmin:
|
class ModemAdmin:
|
||||||
"""Logged-in session on the gateway, kept between checks and renewed when it expires.
|
"""Logged-in session on the gateway, kept between checks and renewed when it expires.
|
||||||
|
|
||||||
@@ -480,6 +533,37 @@ class ModemAdmin:
|
|||||||
raise ModemDataInvalid('NAT table page has no session counts')
|
raise ModemDataInvalid('NAT table page has no session counts')
|
||||||
return counts
|
return counts
|
||||||
|
|
||||||
|
def firewall_log(self) -> List[dict]:
|
||||||
|
return parse_firewall_log(self._page(_LOGS_PAGE))
|
||||||
|
|
||||||
|
def speed_tests(self) -> List[dict]:
|
||||||
|
return parse_speed_tests(self._page(_SPEED_PAGE))
|
||||||
|
|
||||||
|
def restart(self) -> str:
|
||||||
|
"""Restart the gateway from its Restart Device page: the one change this client makes,
|
||||||
|
and only when an admin asks.
|
||||||
|
|
||||||
|
Two steps, as in a browser: posting the form only answers with a redirect to
|
||||||
|
restarting.ha, and loading that page is what starts the restart (firmware 6.40; a
|
||||||
|
post alone leaves the gateway running). Returns 'accepted' once that page says it is
|
||||||
|
restarting, or 'unconfirmed' when the gateway dropped the connection first.
|
||||||
|
"""
|
||||||
|
nonce = _NONCE_RE.search(self._page(_RESTART_PAGE))
|
||||||
|
if not nonce:
|
||||||
|
raise ModemDataInvalid('Restart page has no nonce')
|
||||||
|
timeout = max(self.timeout, 30)
|
||||||
|
try:
|
||||||
|
resp = self.session.post(f'{self.base_url}{_RESTART_PAGE}', allow_redirects=False,
|
||||||
|
timeout=timeout, data={'nonce': nonce.group(1), 'Restart': 'Restart'})
|
||||||
|
if resp.status_code == 302 and _RESTARTING_PAGE in (resp.headers.get('Location') or ''):
|
||||||
|
resp = self.session.get(f'{self.base_url}{_RESTARTING_PAGE}', allow_redirects=False,
|
||||||
|
timeout=timeout)
|
||||||
|
except (requests.ConnectionError, requests.Timeout):
|
||||||
|
return 'unconfirmed'
|
||||||
|
if resp.status_code != 200 or is_login_page(resp.text) or 'restarting' not in resp.text.lower():
|
||||||
|
raise ModemDataInvalid(f'Restart was not started (HTTP {resp.status_code})')
|
||||||
|
return 'accepted'
|
||||||
|
|
||||||
def ping(self, target: str, wait_s: float = 30) -> dict:
|
def ping(self, target: str, wait_s: float = 30) -> dict:
|
||||||
"""Ping *target* from the gateway (its Troubleshoot page; 5 packets) and return the result."""
|
"""Ping *target* from the gateway (its Troubleshoot page; 5 packets) and return the result."""
|
||||||
nonce = _NONCE_RE.search(self._page(_DIAG_PAGE))
|
nonce = _NONCE_RE.search(self._page(_DIAG_PAGE))
|
||||||
|
|||||||
+86
-2
@@ -24,6 +24,7 @@ from urllib3.exceptions import InsecureRequestWarning
|
|||||||
import db
|
import db
|
||||||
import cutsheet
|
import cutsheet
|
||||||
import devices
|
import devices
|
||||||
|
import kuma
|
||||||
import modem as modem_api
|
import modem as modem_api
|
||||||
import ticket_text
|
import ticket_text
|
||||||
|
|
||||||
@@ -1973,7 +1974,53 @@ class NetworkMonitor:
|
|||||||
if quality is not None:
|
if quality is not None:
|
||||||
db.set_state('wan_quality', quality)
|
db.set_state('wan_quality', quality)
|
||||||
|
|
||||||
_NAT_EVERY_S = 1800
|
_RESTART_SETTLE_S = 180 # the modem answering this soon may not have restarted yet
|
||||||
|
_RESTART_GIVE_UP_S = 1800
|
||||||
|
|
||||||
|
def _finish_modem_restart(self) -> None:
|
||||||
|
"""After the Restart modem button: once the modem answers again with its WAN up (or
|
||||||
|
after 30 minutes regardless), end the Uptime Kuma window and the Gandalf silence the
|
||||||
|
button started, and note how long it took."""
|
||||||
|
raw = db.get_state('modem_restart')
|
||||||
|
try:
|
||||||
|
state = json.loads(raw) if raw else None
|
||||||
|
except ValueError:
|
||||||
|
state = None
|
||||||
|
if not state or state.get('finished_at'):
|
||||||
|
return
|
||||||
|
now = time.time()
|
||||||
|
age = now - state.get('at', now)
|
||||||
|
raw = db.get_state('modem_stats')
|
||||||
|
try:
|
||||||
|
stats = json.loads(raw) if raw else {}
|
||||||
|
except ValueError:
|
||||||
|
stats = {}
|
||||||
|
uptime = stats.get('uptime_s')
|
||||||
|
rebooted = uptime is not None and uptime < age
|
||||||
|
back = (age >= self._RESTART_SETTLE_S and stats.get('reachable')
|
||||||
|
and not stats.get('wan_problems') and (rebooted or uptime is None))
|
||||||
|
if not back and age < self._RESTART_GIVE_UP_S:
|
||||||
|
return
|
||||||
|
if state.get('kuma') and not state.get('kuma_ended'):
|
||||||
|
try:
|
||||||
|
kuma.set_maintenance(self.cfg.get('uptime_kuma') or {}, False)
|
||||||
|
state['kuma_ended'] = True
|
||||||
|
except kuma.KumaError as e:
|
||||||
|
logger.warning(f'Ending the Uptime Kuma window after the modem restart: {e}')
|
||||||
|
if age < self._RESTART_GIVE_UP_S:
|
||||||
|
return # try again next cycle
|
||||||
|
state['kuma_error'] = str(e)
|
||||||
|
if state.get('suppression_id'):
|
||||||
|
db.deactivate_suppression(state['suppression_id'])
|
||||||
|
state.update(finished_at=now, back=bool(back), rebooted=rebooted,
|
||||||
|
back_after_s=round(age) if back else None)
|
||||||
|
db.set_state('modem_restart', state)
|
||||||
|
logger.info(f'Modem restart finished: {"back" if back else "gave up waiting"} after {round(age)} s'
|
||||||
|
f'{", rebooted" if rebooted else ""}')
|
||||||
|
|
||||||
|
_NAT_EVERY_S = 1800 # NAT sessions and the gateway's speed test history
|
||||||
|
_FIREWALL_EVERY_S = 300 # the firewall log only holds about ten minutes
|
||||||
|
_FIREWALL_HOURS = 24
|
||||||
_PING_EVERY_S = 600
|
_PING_EVERY_S = 600
|
||||||
_PING_TARGET = '1.1.1.1'
|
_PING_TARGET = '1.1.1.1'
|
||||||
|
|
||||||
@@ -2001,14 +2048,18 @@ class NetworkMonitor:
|
|||||||
slow = (bool(periods) and periods[-1].get('state') != 'ok') or (ours or 0) >= modem_api.SLOW_MS
|
slow = (bool(periods) and periods[-1].get('state') != 'ok') or (ours or 0) >= modem_api.SLOW_MS
|
||||||
|
|
||||||
due_nat = now - (checks.get('nat') or {}).get('at', 0) >= self._NAT_EVERY_S
|
due_nat = now - (checks.get('nat') or {}).get('at', 0) >= self._NAT_EVERY_S
|
||||||
|
due_fw = now - (checks.get('firewall') or {}).get('checked_at', 0) >= self._FIREWALL_EVERY_S
|
||||||
due_ping = slow and now - (checks.get('ping') or {}).get('at', 0) >= self._PING_EVERY_S
|
due_ping = slow and now - (checks.get('ping') or {}).get('at', 0) >= self._PING_EVERY_S
|
||||||
if not (due_nat or due_ping):
|
if not (due_nat or due_fw or due_ping):
|
||||||
return
|
return
|
||||||
if now - (checks.get('error') or {}).get('at', 0) < self._PING_EVERY_S:
|
if now - (checks.get('error') or {}).get('at', 0) < self._PING_EVERY_S:
|
||||||
return # failed recently: don't keep knocking on the gateway
|
return # failed recently: don't keep knocking on the gateway
|
||||||
try:
|
try:
|
||||||
if due_nat:
|
if due_nat:
|
||||||
checks['nat'] = dict(admin.nat_sessions(), at=now)
|
checks['nat'] = dict(admin.nat_sessions(), at=now)
|
||||||
|
checks['speed'] = self._latest_speed_tests(admin.speed_tests(), now)
|
||||||
|
if due_fw:
|
||||||
|
checks['firewall'] = self._count_firewall_drops(checks.get('firewall'), admin.firewall_log(), now)
|
||||||
if due_ping:
|
if due_ping:
|
||||||
result = admin.ping(self._PING_TARGET)
|
result = admin.ping(self._PING_TARGET)
|
||||||
checks['ping'] = dict(result, at=now, ours_ms=ours,
|
checks['ping'] = dict(result, at=now, ours_ms=ours,
|
||||||
@@ -2023,6 +2074,33 @@ class NetworkMonitor:
|
|||||||
'login': isinstance(e, modem_api.ModemLoginFailed)}
|
'login': isinstance(e, modem_api.ModemLoginFailed)}
|
||||||
db.set_state('modem_checks', checks)
|
db.set_state('modem_checks', checks)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _latest_speed_tests(tests: list, now: float) -> dict:
|
||||||
|
"""The newest downstream and upstream result from the gateway's own speed tests."""
|
||||||
|
out: dict = {'checked_at': now}
|
||||||
|
for t in tests: # newest first
|
||||||
|
key = 'down' if t['direction'] == 'downstream' else 'up'
|
||||||
|
out.setdefault(key, t)
|
||||||
|
return out
|
||||||
|
|
||||||
|
def _count_firewall_drops(self, previous: Optional[dict], rows: list, now: float) -> dict:
|
||||||
|
"""Add log rows newer than the last one seen to per-hour counts (gateway's local hour).
|
||||||
|
|
||||||
|
The log keeps a few hundred rows and records at most one drop every 2 seconds, so the
|
||||||
|
counts are a lower bound; 'gap' says rows aged out before this read could count them.
|
||||||
|
"""
|
||||||
|
fw = previous or {}
|
||||||
|
hours = dict(fw.get('hours') or {})
|
||||||
|
last = fw.get('last') or ''
|
||||||
|
new = [r for r in rows if r['ts'] > last]
|
||||||
|
for r in new:
|
||||||
|
bucket = hours.setdefault(r['ts'][:13], {'flood': 0, 'invalid': 0, 'other': 0})
|
||||||
|
bucket[modem_api.firewall_kind(r['reason'])] += 1
|
||||||
|
gap = bool(last and rows and rows[0]['ts'] > last)
|
||||||
|
keep = sorted(hours)[-self._FIREWALL_HOURS:]
|
||||||
|
return {'hours': {h: hours[h] for h in keep}, 'last': rows[-1]['ts'] if rows else last,
|
||||||
|
'checked_at': now, 'gap': gap}
|
||||||
|
|
||||||
def _collect_modem(self, suppressions: list) -> None:
|
def _collect_modem(self, suppressions: list) -> None:
|
||||||
"""Poll the modem, then store its status and raise any alerts."""
|
"""Poll the modem, then store its status and raise any alerts."""
|
||||||
try:
|
try:
|
||||||
@@ -2214,6 +2292,12 @@ class NetworkMonitor:
|
|||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f'WAN quality collection failed: {e}', exc_info=True)
|
logger.error(f'WAN quality collection failed: {e}', exc_info=True)
|
||||||
|
|
||||||
|
# An admin's modem restart: end the silence once the modem is back
|
||||||
|
try:
|
||||||
|
self._finish_modem_restart()
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f'Finishing the modem restart failed: {e}', exc_info=True)
|
||||||
|
|
||||||
# Logged-in modem checks (NAT sessions; a ping from the modem when the internet is slow)
|
# Logged-in modem checks (NAT sessions; a ping from the modem when the internet is slow)
|
||||||
try:
|
try:
|
||||||
self._collect_modem_checks()
|
self._collect_modem_checks()
|
||||||
|
|||||||
@@ -4,3 +4,4 @@ ldap3>=2.9
|
|||||||
pymysql>=1.1.0
|
pymysql>=1.1.0
|
||||||
requests>=2.31.0
|
requests>=2.31.0
|
||||||
urllib3>=2.0.0
|
urllib3>=2.0.0
|
||||||
|
python-socketio[client]>=5.8
|
||||||
+68
-2
@@ -396,6 +396,19 @@ function _modemChecks(c) {
|
|||||||
out += _modemStat("Modem's own ping", value, p.verdict === 'clear' ? '' : 'lt-text-amber',
|
out += _modemStat("Modem's own ping", value, p.verdict === 'clear' ? '' : 'lt-text-amber',
|
||||||
`to ${p.target}, ${ago(p.at)}${verdict ? `: ${verdict}` : ''}`);
|
`to ${p.target}, ${ago(p.at)}${verdict ? `: ${verdict}` : ''}`);
|
||||||
}
|
}
|
||||||
|
const fw = c.firewall && c.firewall.hours;
|
||||||
|
if (fw && Object.keys(fw).length) {
|
||||||
|
const sum = k => Object.values(fw).reduce((n, h) => n + (h[k] || 0), 0);
|
||||||
|
const flood = sum('flood'), invalid = sum('invalid'), total = flood + invalid + sum('other');
|
||||||
|
out += _modemStat('Firewall drops', `${total.toLocaleString()} in ${Object.keys(fw).length} h`, '',
|
||||||
|
`${flood.toLocaleString()} incoming over the flood limit, ${invalid.toLocaleString()} invalid; ` +
|
||||||
|
'at least (the log keeps 1 per 2 s)');
|
||||||
|
}
|
||||||
|
const sp = c.speed;
|
||||||
|
if (sp && sp.down && sp.up) {
|
||||||
|
out += _modemStat('AT&T speed test', `↓ ${Math.round(sp.down.mbps)} ↑ ${Math.round(sp.up.mbps)} Mbps`, '',
|
||||||
|
`run by the modem ${sp.down.at}, ${Math.round(sp.down.latency_ms)} ms`);
|
||||||
|
}
|
||||||
if (c.error && c.error.login) {
|
if (c.error && c.error.login) {
|
||||||
out += _modemStat('Modem login', 'failed', 'lt-text-red', 'check modem.access_code');
|
out += _modemStat('Modem login', 'failed', 'lt-text-red', 'check modem.access_code');
|
||||||
}
|
}
|
||||||
@@ -457,6 +470,47 @@ function _modemQuality(q) {
|
|||||||
</div>`;
|
</div>`;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// A restart from the Restart modem button: in progress, or finished in the last hour
|
||||||
|
function _modemRestartNote(r) {
|
||||||
|
if (!r || !r.at) return '';
|
||||||
|
const who = r.by ? ` by ${r.by}` : '';
|
||||||
|
if (!r.finished_at) {
|
||||||
|
return `<div class="modem-restart-note">Restart requested${lt.escHtml(who)} ${lt.escHtml(_modemWhen(r.at))}. ` +
|
||||||
|
`Alerts are silenced${r.kuma ? ' here and in Uptime Kuma' : ''} until the modem is back.</div>`;
|
||||||
|
}
|
||||||
|
if (Date.now() / 1000 - r.finished_at > 3600) return '';
|
||||||
|
const how = r.back ? `back after ${_modemDuration(r.back_after_s)}` : 'did not come back within 30 minutes';
|
||||||
|
return `<div class="modem-restart-note">Restarted${lt.escHtml(who)} ${lt.escHtml(_modemWhen(r.at))}: ` +
|
||||||
|
`${lt.escHtml(how)}. Alerts are on again.</div>`;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function restartModem(btn) {
|
||||||
|
const msg = 'Restart the AT&T modem?\n\nThe internet will be down for about 3 to 5 minutes. ' +
|
||||||
|
'Gandalf alerts and the Uptime Kuma maintenance window are switched on first, and switched off ' +
|
||||||
|
'again when the modem is back.';
|
||||||
|
if (!window.confirm(msg)) return;
|
||||||
|
btn.disabled = true;
|
||||||
|
try {
|
||||||
|
let res;
|
||||||
|
try {
|
||||||
|
res = await lt.api.post('/api/modem/restart', { confirm: true });
|
||||||
|
} catch (err) {
|
||||||
|
if (!String(err.message).startsWith('Uptime Kuma:')) throw err;
|
||||||
|
if (!window.confirm(`${err.message}\n\nRestart anyway? Uptime Kuma will report the outage.`)) {
|
||||||
|
btn.disabled = false;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
res = await lt.api.post('/api/modem/restart', { confirm: true, without_kuma: true });
|
||||||
|
}
|
||||||
|
showToast(res.result === 'accepted' ? 'Modem restarting' : 'Restart sent; the modem dropped the connection',
|
||||||
|
'success');
|
||||||
|
window.location.reload();
|
||||||
|
} catch (err) {
|
||||||
|
btn.disabled = false;
|
||||||
|
showToast(err.message || 'Could not restart the modem', 'error');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
function updateModem(data) {
|
function updateModem(data) {
|
||||||
const section = document.getElementById('modem-section');
|
const section = document.getElementById('modem-section');
|
||||||
const body = document.getElementById('modem-body');
|
const body = document.getElementById('modem-body');
|
||||||
@@ -464,6 +518,18 @@ function updateModem(data) {
|
|||||||
section.hidden = !data.configured;
|
section.hidden = !data.configured;
|
||||||
if (!data.configured) return;
|
if (!data.configured) return;
|
||||||
|
|
||||||
|
const restartBtn = document.getElementById('modem-restart-btn');
|
||||||
|
if (restartBtn) {
|
||||||
|
const pending = data.modem_restart && !data.modem_restart.finished_at;
|
||||||
|
restartBtn.hidden = !data.can_restart;
|
||||||
|
restartBtn.disabled = Boolean(pending);
|
||||||
|
if (!restartBtn.dataset.bound) {
|
||||||
|
restartBtn.dataset.bound = '1';
|
||||||
|
restartBtn.addEventListener('click', () => restartModem(restartBtn));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
const restartNote = _modemRestartNote(data.modem_restart);
|
||||||
|
|
||||||
// On the dashboard, link to the modem's page. On that page, it would link to itself.
|
// On the dashboard, link to the modem's page. On that page, it would link to itself.
|
||||||
const detailsLink = document.getElementById('modem-device-link');
|
const detailsLink = document.getElementById('modem-device-link');
|
||||||
if (detailsLink && typeof DEVICE_NAME === 'undefined' && data.name) {
|
if (detailsLink && typeof DEVICE_NAME === 'undefined' && data.name) {
|
||||||
@@ -474,7 +540,7 @@ function updateModem(data) {
|
|||||||
const title = document.getElementById('modem-title');
|
const title = document.getElementById('modem-title');
|
||||||
|
|
||||||
if (!data.reachable) {
|
if (!data.reachable) {
|
||||||
body.innerHTML = `
|
body.innerHTML = `${restartNote}
|
||||||
<div class="modem-error">Modem status unavailable: ${lt.escHtml(data.error || 'unknown error')}
|
<div class="modem-error">Modem status unavailable: ${lt.escHtml(data.error || 'unknown error')}
|
||||||
<span class="modem-stat-note">Last attempt ${lt.escHtml(data.updated || '-')}</span></div>
|
<span class="modem-stat-note">Last attempt ${lt.escHtml(data.updated || '-')}</span></div>
|
||||||
${_modemQuality(data.wan_quality)}
|
${_modemQuality(data.wan_quality)}
|
||||||
@@ -495,7 +561,7 @@ function updateModem(data) {
|
|||||||
|
|
||||||
const errs = (counters.rx_errors || 0) + (counters.tx_errors || 0);
|
const errs = (counters.rx_errors || 0) + (counters.tx_errors || 0);
|
||||||
const drops = (counters.rx_drops || 0) + (counters.tx_drops || 0);
|
const drops = (counters.rx_drops || 0) + (counters.tx_drops || 0);
|
||||||
body.innerHTML = `
|
body.innerHTML = `${restartNote}
|
||||||
${problems.length ? `<div class="modem-error">WAN down: ${lt.escHtml(problems.join('; '))}</div>` : ''}
|
${problems.length ? `<div class="modem-error">WAN down: ${lt.escHtml(problems.join('; '))}</div>` : ''}
|
||||||
<div class="modem-stats">
|
<div class="modem-stats">
|
||||||
${_modemStat('WAN', problems.length ? 'DOWN' : 'UP', problems.length ? 'lt-text-red' : 'lt-text-green',
|
${_modemStat('WAN', problems.length ? 'DOWN' : 'UP', problems.length ? 'lt-text-red' : 'lt-text-green',
|
||||||
|
|||||||
@@ -1849,3 +1849,8 @@ body > .lt-footer { flex-shrink: 0; }
|
|||||||
#events-table td { max-width: none; }
|
#events-table td { max-width: none; }
|
||||||
#events-table td.desc-cell { white-space: normal; text-align: right; }
|
#events-table td.desc-cell { white-space: normal; text-align: right; }
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* ── WAN / Fiber: restart button and progress note ── */
|
||||||
|
.modem-actions { display: inline-flex; gap: 6px; }
|
||||||
|
.modem-restart-note { margin: 8px 12px 0; padding: 6px 10px; font-size: .8em;
|
||||||
|
border: 1px solid var(--accent-amber); color: var(--text-primary); }
|
||||||
@@ -1,7 +1,13 @@
|
|||||||
<section class="g-section" id="modem-section" hidden>
|
<section class="g-section" id="modem-section" hidden>
|
||||||
<div class="g-section-header">
|
<div class="g-section-header">
|
||||||
<h2 class="g-section-title">WAN / Fiber</h2>
|
<h2 class="g-section-title">WAN / Fiber</h2>
|
||||||
<a class="g-section-actions lt-btn lt-btn-ghost lt-btn-sm" id="modem-device-link" hidden>↗ Details</a>
|
<span class="g-section-actions modem-actions">
|
||||||
|
{% if user.groups and 'admin' in user.groups %}
|
||||||
|
<button type="button" class="lt-btn lt-btn-ghost lt-btn-sm" id="modem-restart-btn" hidden
|
||||||
|
title="Silence alerts (Gandalf and Uptime Kuma), then restart the modem">⟳ Restart modem</button>
|
||||||
|
{% endif %}
|
||||||
|
<a class="lt-btn lt-btn-ghost lt-btn-sm" id="modem-device-link" hidden>↗ Details</a>
|
||||||
|
</span>
|
||||||
</div>
|
</div>
|
||||||
<div class="lt-frame">
|
<div class="lt-frame">
|
||||||
<span class="lt-frame-bl">╚</span>
|
<span class="lt-frame-bl">╚</span>
|
||||||
|
|||||||
+63
-1
@@ -4,7 +4,7 @@ import time
|
|||||||
from datetime import datetime, timedelta, timezone
|
from datetime import datetime, timedelta, timezone
|
||||||
import os
|
import os
|
||||||
import sys
|
import sys
|
||||||
from unittest.mock import patch
|
from unittest.mock import MagicMock, patch
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
@@ -875,3 +875,65 @@ class TestCalmDashboard:
|
|||||||
with patch.object(gandalf.db, 'get_recent_resolved', return_value=[ev]):
|
with patch.object(gandalf.db, 'get_recent_resolved', return_value=[ev]):
|
||||||
html = client.get('/', headers=AUTH).get_data(as_text=True)
|
html = client.get('/', headers=AUTH).get_data(as_text=True)
|
||||||
assert '<details class="g-section resolved-fold">' in html
|
assert '<details class="g-section resolved-fold">' in html
|
||||||
|
|
||||||
|
|
||||||
|
class TestModemRestart:
|
||||||
|
CFG = dict(CONFIG, modem={'url': 'http://192.0.2.254', 'access_code': 'secret'},
|
||||||
|
uptime_kuma={'url': 'http://kuma.test', 'username': 'u', 'password': 'p', 'maintenance_id': 7})
|
||||||
|
|
||||||
|
def _post(self, client, body=None, headers=AUTH, cfg=None, kuma_error=None, restart=None):
|
||||||
|
admin = MagicMock()
|
||||||
|
if isinstance(restart, Exception):
|
||||||
|
admin.restart.side_effect = restart
|
||||||
|
else:
|
||||||
|
admin.restart.return_value = restart or 'accepted'
|
||||||
|
with patch.object(gandalf, '_cfg', cfg or self.CFG), \
|
||||||
|
patch.object(gandalf.kuma, 'set_maintenance', side_effect=kuma_error) as km, \
|
||||||
|
patch.object(gandalf.modem_api, 'ModemAdmin', return_value=admin):
|
||||||
|
gandalf.db.create_suppression.return_value = 42
|
||||||
|
resp = client.post('/api/modem/restart', json=body if body is not None else {'confirm': True},
|
||||||
|
headers=headers)
|
||||||
|
return resp, km, admin
|
||||||
|
|
||||||
|
def test_admin_only(self, client):
|
||||||
|
resp, _, admin = self._post(client, headers=USER)
|
||||||
|
assert resp.status_code == 403 and not admin.restart.called
|
||||||
|
|
||||||
|
def test_needs_confirm(self, client):
|
||||||
|
resp, km, admin = self._post(client, body={})
|
||||||
|
assert resp.status_code == 400 and not km.called and not admin.restart.called
|
||||||
|
|
||||||
|
def test_needs_the_access_code(self, client):
|
||||||
|
resp, _, admin = self._post(client, cfg=dict(CONFIG, modem={'url': 'http://192.0.2.254'}))
|
||||||
|
assert resp.status_code == 400 and 'access_code' in resp.get_json()['error']
|
||||||
|
|
||||||
|
def test_silences_then_restarts(self, client):
|
||||||
|
resp, km, admin = self._post(client)
|
||||||
|
assert resp.status_code == 200
|
||||||
|
assert resp.get_json() == {'ok': True, 'result': 'accepted', 'kuma': True, 'suppression_id': 42}
|
||||||
|
km.assert_called_once_with(self.CFG['uptime_kuma'], True)
|
||||||
|
args = gandalf.db.create_suppression.call_args.args
|
||||||
|
assert args[:3] == ('all', '', '') and args[4] == 'jared' and args[5] == gandalf.MODEM_RESTART_SILENCE_MIN
|
||||||
|
admin.restart.assert_called_once_with()
|
||||||
|
key, state = gandalf.db.set_state.call_args.args
|
||||||
|
assert key == 'modem_restart' and state['suppression_id'] == 42 and state['kuma'] is True
|
||||||
|
|
||||||
|
def test_kuma_failure_stops_unless_accepted(self, client):
|
||||||
|
resp, _, admin = self._post(client, kuma_error=gandalf.kuma.KumaError('login refused'))
|
||||||
|
assert resp.status_code == 502 and resp.get_json()['kuma_failed'] and not admin.restart.called
|
||||||
|
resp, _, admin = self._post(client, body={'confirm': True, 'without_kuma': True},
|
||||||
|
kuma_error=gandalf.kuma.KumaError('login refused'))
|
||||||
|
assert resp.status_code == 200 and resp.get_json()['kuma'] is False and admin.restart.called
|
||||||
|
|
||||||
|
def test_failed_restart_undoes_the_silence(self, client):
|
||||||
|
resp, km, _ = self._post(client, restart=gandalf.modem_api.ModemUnreachable('timeout'))
|
||||||
|
assert resp.status_code == 502
|
||||||
|
gandalf.db.deactivate_suppression.assert_called_once_with(42)
|
||||||
|
assert km.call_args_list[-1].args == (self.CFG['uptime_kuma'], False)
|
||||||
|
|
||||||
|
def test_one_at_a_time(self, client):
|
||||||
|
pending = json.dumps({'at': time.time() - 60, 'finished_at': None})
|
||||||
|
state = {'modem_restart': pending}
|
||||||
|
with patch.object(gandalf.db, 'get_state', side_effect=lambda k, d=None: state.get(k, d)):
|
||||||
|
resp, _, admin = self._post(client)
|
||||||
|
assert resp.status_code == 409 and not admin.restart.called
|
||||||
@@ -0,0 +1,91 @@
|
|||||||
|
"""kuma.py: switching the Uptime Kuma maintenance window, against a fake Socket.IO client."""
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
|
||||||
|
import kuma # noqa: E402
|
||||||
|
|
||||||
|
CFG = {'url': 'http://kuma.test:3001', 'token': 'tok', 'maintenance_id': 7}
|
||||||
|
PASSWORD_CFG = {'url': 'http://kuma.test:3001', 'username': 'gandalf', 'password': 'pw', 'maintenance_id': 7}
|
||||||
|
|
||||||
|
|
||||||
|
class FakeSocket:
|
||||||
|
def __init__(self, answers=None, connect_error=None, sends_info=True):
|
||||||
|
self.answers = answers or {}
|
||||||
|
self.connect_error = connect_error
|
||||||
|
self.sends_info = sends_info
|
||||||
|
self.calls, self.disconnected, self.handlers = [], False, {}
|
||||||
|
|
||||||
|
def on(self, event, handler):
|
||||||
|
self.handlers[event] = handler
|
||||||
|
|
||||||
|
def connect(self, url, **kw):
|
||||||
|
if self.connect_error:
|
||||||
|
raise self.connect_error
|
||||||
|
self.url = url
|
||||||
|
if self.sends_info: # Kuma's first event; its handlers exist only after it
|
||||||
|
self.handlers['info']({'serverTimezone': 'America/New_York'})
|
||||||
|
|
||||||
|
def call(self, event, data=None, timeout=None):
|
||||||
|
self.calls.append((event, data))
|
||||||
|
return self.answers.get(event, {'ok': True})
|
||||||
|
|
||||||
|
def disconnect(self):
|
||||||
|
self.disconnected = True
|
||||||
|
|
||||||
|
|
||||||
|
def run(active=True, cfg=CFG, **kw):
|
||||||
|
sock = FakeSocket(**kw)
|
||||||
|
kuma.set_maintenance(dict(cfg, timeout=0.05), active, client_factory=lambda: sock)
|
||||||
|
return sock
|
||||||
|
|
||||||
|
|
||||||
|
class TestKuma:
|
||||||
|
def test_configured_needs_url_window_and_a_way_to_sign_in(self):
|
||||||
|
assert kuma.configured(CFG) and kuma.configured(PASSWORD_CFG)
|
||||||
|
assert not kuma.configured(dict(CFG, token=''))
|
||||||
|
assert not kuma.configured(dict(CFG, maintenance_id=None))
|
||||||
|
assert not kuma.configured(dict(PASSWORD_CFG, password=''))
|
||||||
|
assert not kuma.configured(None)
|
||||||
|
|
||||||
|
def test_token_sign_in_then_switches_the_window_on(self):
|
||||||
|
sock = run(True)
|
||||||
|
assert sock.calls == [('loginByToken', 'tok'), ('resumeMaintenance', 7)]
|
||||||
|
assert sock.disconnected and sock.url == CFG['url']
|
||||||
|
|
||||||
|
def test_rejected_token(self):
|
||||||
|
with pytest.raises(kuma.KumaError, match='token rejected'):
|
||||||
|
run(answers={'loginByToken': {'ok': False}})
|
||||||
|
|
||||||
|
def test_waits_for_kuma_to_be_ready(self):
|
||||||
|
sock = FakeSocket(sends_info=False)
|
||||||
|
with pytest.raises(kuma.KumaError, match='not ready'):
|
||||||
|
kuma.set_maintenance(dict(CFG, timeout=0.05), True, client_factory=lambda: sock)
|
||||||
|
assert sock.calls == [] and sock.disconnected # nothing sent before info
|
||||||
|
|
||||||
|
def test_password_sign_in(self):
|
||||||
|
sock = run(True, cfg=PASSWORD_CFG)
|
||||||
|
assert sock.calls[0] == ('login', {'username': 'gandalf', 'password': 'pw', 'token': ''})
|
||||||
|
|
||||||
|
def test_pause(self):
|
||||||
|
assert run(False).calls[-1] == ('pauseMaintenance', 7)
|
||||||
|
|
||||||
|
def test_refused_password(self):
|
||||||
|
with pytest.raises(kuma.KumaError, match='login refused: Incorrect username or password'):
|
||||||
|
run(cfg=PASSWORD_CFG, answers={'login': {'ok': False, 'msg': 'Incorrect username or password.'}})
|
||||||
|
|
||||||
|
def test_two_factor_account_needs_the_token(self):
|
||||||
|
with pytest.raises(kuma.KumaError, match='use uptime_kuma.token'):
|
||||||
|
run(cfg=PASSWORD_CFG, answers={'login': {'ok': False, 'tokenRequired': True}})
|
||||||
|
|
||||||
|
def test_refused_change(self):
|
||||||
|
with pytest.raises(kuma.KumaError, match='resumeMaintenance refused'):
|
||||||
|
run(answers={'resumeMaintenance': {'ok': False, 'msg': 'not found'}})
|
||||||
|
|
||||||
|
def test_unreachable(self):
|
||||||
|
sock = FakeSocket(connect_error=ConnectionError('refused'))
|
||||||
|
with pytest.raises(kuma.KumaError, match='ConnectionError'):
|
||||||
|
kuma.set_maintenance(dict(CFG, timeout=0.05), True, client_factory=lambda: sock)
|
||||||
|
assert sock.disconnected
|
||||||
@@ -389,6 +389,8 @@ class FakeGateway:
|
|||||||
return MagicMock(status_code=302 if ok else 200, text='' if ok else LOGIN_FORM)
|
return MagicMock(status_code=302 if ok else 200, text='' if ok else LOGIN_FORM)
|
||||||
if not self.logged_in:
|
if not self.logged_in:
|
||||||
return MagicMock(status_code=200, text=LOGIN_FORM)
|
return MagicMock(status_code=200, text=LOGIN_FORM)
|
||||||
|
if path == '/cgi-bin/restart.ha' and method == 'GET':
|
||||||
|
return MagicMock(status_code=200, text=DIAG_FORM.replace('diag.ha', 'restart.ha'))
|
||||||
if path == '/cgi-bin/nattable.ha':
|
if path == '/cgi-bin/nattable.ha':
|
||||||
return MagicMock(status_code=200, text=NAT_PAGE)
|
return MagicMock(status_code=200, text=NAT_PAGE)
|
||||||
if path == '/cgi-bin/diag.ha' and method == 'POST':
|
if path == '/cgi-bin/diag.ha' and method == 'POST':
|
||||||
@@ -455,3 +457,104 @@ class TestModemAdmin:
|
|||||||
a.session.request = MagicMock(side_effect=requests.ConnectionError('boom'))
|
a.session.request = MagicMock(side_effect=requests.ConnectionError('boom'))
|
||||||
with pytest.raises(modem.ModemUnreachable):
|
with pytest.raises(modem.ModemUnreachable):
|
||||||
a.nat_sessions()
|
a.nat_sessions()
|
||||||
|
|
||||||
|
|
||||||
|
FW_LOG = '''<table><tr><th>No.</th><th>Date/Time</th></tr>
|
||||||
|
<tr class="b"><td class="heading" scope="row">2</td><td>2026-11-19T18:32:04.984186</td><td>198.51.100.20</td>
|
||||||
|
<td>203.0.113.7</td><td>TCP</td><td>FLOOD limit 50pps burst 100</td></tr>
|
||||||
|
<tr class="a"><td class="heading" scope="row">1</td><td>2026-11-19T18:32:02.983868</td><td>203.0.113.7</td>
|
||||||
|
<td>198.51.100.30</td><td>TCP</td><td>Invalid IP Packet</td></tr>
|
||||||
|
</table>'''
|
||||||
|
SPEED_PAGE = '''<h2>Test History</h2><table><thead><th scope="col">Test Completion Time</th></thead><tbody>
|
||||||
|
<tr class="b"><td scope="row" class="heading" width="47%">
|
||||||
|
11/06/2026 08:46:05</td>
|
||||||
|
<td>upstream</td>
|
||||||
|
<td> 1242.810000</td>
|
||||||
|
<td> 28</td>
|
||||||
|
<td> 19.000000</td>
|
||||||
|
<td> Success</td></tr>
|
||||||
|
<tr class="a"><td scope="row" class="heading" width="47%">
|
||||||
|
11/06/2026 08:45:57</td>
|
||||||
|
<td>downstream</td>
|
||||||
|
<td> 1167.400000</td>
|
||||||
|
<td> 28</td>
|
||||||
|
<td> 28.000000</td>
|
||||||
|
<td> Success</td></tr>
|
||||||
|
</tbody></table>'''
|
||||||
|
|
||||||
|
|
||||||
|
class TestModemExtras:
|
||||||
|
@pytest.mark.parametrize('raw,expected', [
|
||||||
|
('03:06:29:34', ((3 * 24 + 6) * 60 + 29) * 60 + 34), # firmware 6.40
|
||||||
|
('28951', 28951), # older firmware
|
||||||
|
('0:00:05:00', 300),
|
||||||
|
('', None), (None, None), ('soon', None),
|
||||||
|
])
|
||||||
|
def test_uptime_formats(self, raw, expected):
|
||||||
|
assert modem.parse_uptime(raw) == expected
|
||||||
|
|
||||||
|
def test_status_reads_dhms_uptime(self):
|
||||||
|
sysinfo = SYSINFO.replace('<td class="col2">28951</td>', '<td class="col2">03:06:29:34</td>')
|
||||||
|
assert sysinfo != SYSINFO
|
||||||
|
assert modem.build_status(fiber_page(), broadband_page(), sysinfo)['uptime_s'] == 282574
|
||||||
|
|
||||||
|
def test_firewall_log_oldest_first(self):
|
||||||
|
rows = modem.parse_firewall_log(FW_LOG)
|
||||||
|
assert [r['ts'] for r in rows] == ['2026-11-19T18:32:02.983868', '2026-11-19T18:32:04.984186']
|
||||||
|
assert rows[1]['reason'] == 'FLOOD limit 50pps burst 100' and rows[1]['proto'] == 'TCP'
|
||||||
|
|
||||||
|
@pytest.mark.parametrize('reason,kind', [
|
||||||
|
('FLOOD limit 50pps burst 100', 'flood'), ('Invalid IP Packet', 'invalid'), ('Port scan', 'other')])
|
||||||
|
def test_firewall_kind(self, reason, kind):
|
||||||
|
assert modem.firewall_kind(reason) == kind
|
||||||
|
|
||||||
|
def test_speed_tests(self):
|
||||||
|
assert modem.parse_speed_tests(SPEED_PAGE) == [
|
||||||
|
{'at': '11/06/2026 08:46:05', 'direction': 'upstream', 'mbps': 1242.8, 'latency_ms': 19.0, 'ok': True},
|
||||||
|
{'at': '11/06/2026 08:45:57', 'direction': 'downstream', 'mbps': 1167.4, 'latency_ms': 28.0, 'ok': True},
|
||||||
|
]
|
||||||
|
|
||||||
|
def test_speed_tests_missing(self):
|
||||||
|
assert modem.parse_speed_tests('<html></html>') == []
|
||||||
|
|
||||||
|
|
||||||
|
class TestModemRestart:
|
||||||
|
def _admin(self):
|
||||||
|
gw = FakeGateway()
|
||||||
|
a = modem.ModemAdmin({'url': 'http://192.0.2.254', 'access_code': 'secret'}, sleep=lambda s: None)
|
||||||
|
a.session.request = MagicMock(side_effect=gw)
|
||||||
|
return a, gw
|
||||||
|
|
||||||
|
REDIRECT = MagicMock(status_code=302, headers={'Location': '/cgi-bin/restarting.ha'}, text='')
|
||||||
|
RESTARTING = MagicMock(status_code=200, text='<title>Restarting</title> Your AT&T BGW320-505 is restarting.')
|
||||||
|
|
||||||
|
def test_posts_the_form_then_loads_the_restarting_page(self):
|
||||||
|
# Firmware 6.40: the post only redirects; loading restarting.ha starts the restart
|
||||||
|
a, gw = self._admin()
|
||||||
|
a.session.post = MagicMock(return_value=self.REDIRECT)
|
||||||
|
a.session.get = MagicMock(return_value=self.RESTARTING)
|
||||||
|
assert a.restart() == 'accepted'
|
||||||
|
assert a.session.post.call_args.kwargs['data'] == {'nonce': 'ffee00112233445566778899aabbccdd',
|
||||||
|
'Restart': 'Restart'}
|
||||||
|
assert a.session.post.call_args.args[0].endswith('/cgi-bin/restart.ha')
|
||||||
|
assert a.session.get.call_args.args[0].endswith('/cgi-bin/restarting.ha')
|
||||||
|
assert ('POST', '/cgi-bin/login.ha') in gw.calls # logged in to read the form
|
||||||
|
|
||||||
|
def test_redirect_alone_is_not_a_restart(self):
|
||||||
|
a, _ = self._admin()
|
||||||
|
a.session.post = MagicMock(return_value=self.REDIRECT)
|
||||||
|
a.session.get = MagicMock(return_value=MagicMock(status_code=200, text='<title>Device</title>'))
|
||||||
|
with pytest.raises(modem.ModemDataInvalid, match='not started'):
|
||||||
|
a.restart()
|
||||||
|
|
||||||
|
def test_connection_dropped_mid_restart(self):
|
||||||
|
a, _ = self._admin()
|
||||||
|
a.session.post = MagicMock(return_value=self.REDIRECT)
|
||||||
|
a.session.get = MagicMock(side_effect=requests.ConnectionError('reset'))
|
||||||
|
assert a.restart() == 'unconfirmed'
|
||||||
|
|
||||||
|
def test_refused(self):
|
||||||
|
a, _ = self._admin()
|
||||||
|
a.session.post = MagicMock(return_value=MagicMock(status_code=200, text=LOGIN_FORM))
|
||||||
|
with pytest.raises(modem.ModemDataInvalid, match='not started'):
|
||||||
|
a.restart()
|
||||||
+102
-2
@@ -1080,7 +1080,8 @@ class TestModemChecks:
|
|||||||
out = self._run(mon, db)
|
out = self._run(mon, db)
|
||||||
assert out['nat'] == {'in_use': 2858, 'available': 32767, 'at': self.NOW}
|
assert out['nat'] == {'in_use': 2858, 'available': 32767, 'at': self.NOW}
|
||||||
assert 'ping' not in out # the internet is fine, so no ping
|
assert 'ping' not in out # the internet is fine, so no ping
|
||||||
assert self._run(mon, db, checks={'nat': {'at': self.NOW - 60}}) is None
|
recent = {'nat': {'at': self.NOW - 60}, 'firewall': {'checked_at': self.NOW - 60}}
|
||||||
|
assert self._run(mon, db, checks=recent) is None
|
||||||
|
|
||||||
def test_slow_internet_pings_from_the_modem(self, mon, db):
|
def test_slow_internet_pings_from_the_modem(self, mon, db):
|
||||||
out = self._run(mon, db, checks={'nat': {'at': self.NOW - 60}}, quality=self._slow())
|
out = self._run(mon, db, checks={'nat': {'at': self.NOW - 60}}, quality=self._slow())
|
||||||
@@ -1089,7 +1090,8 @@ class TestModemChecks:
|
|||||||
mon.modem_admin.ping.assert_called_once_with('1.1.1.1')
|
mon.modem_admin.ping.assert_called_once_with('1.1.1.1')
|
||||||
|
|
||||||
def test_ping_at_most_every_ten_minutes(self, mon, db):
|
def test_ping_at_most_every_ten_minutes(self, mon, db):
|
||||||
checks = {'nat': {'at': self.NOW - 60}, 'ping': {'at': self.NOW - 300}}
|
checks = {'nat': {'at': self.NOW - 60}, 'ping': {'at': self.NOW - 300},
|
||||||
|
'firewall': {'checked_at': self.NOW - 60}}
|
||||||
assert self._run(mon, db, checks=checks, quality=self._slow()) is None
|
assert self._run(mon, db, checks=checks, quality=self._slow()) is None
|
||||||
|
|
||||||
def test_failure_is_recorded_and_paused(self, mon, db):
|
def test_failure_is_recorded_and_paused(self, mon, db):
|
||||||
@@ -1105,3 +1107,101 @@ class TestModemChecks:
|
|||||||
def test_success_clears_the_error(self, mon, db):
|
def test_success_clears_the_error(self, mon, db):
|
||||||
out = self._run(mon, db, checks={'error': {'message': 'x', 'at': self.NOW - 3600, 'login': True}})
|
out = self._run(mon, db, checks={'error': {'message': 'x', 'at': self.NOW - 3600, 'login': True}})
|
||||||
assert out['error'] is None
|
assert out['error'] is None
|
||||||
|
|
||||||
|
|
||||||
|
class TestModemExtrasCollection:
|
||||||
|
def _row(self, ts, reason='FLOOD limit 50pps burst 100'):
|
||||||
|
return {'ts': ts, 'src': 'a', 'dst': 'b', 'proto': 'TCP', 'reason': reason}
|
||||||
|
|
||||||
|
def test_counts_only_new_rows_per_hour(self, mon):
|
||||||
|
first = mon._count_firewall_drops(None, [self._row('2026-11-19T17:59:58.1'),
|
||||||
|
self._row('2026-11-19T18:00:01.2', 'Invalid IP Packet')], 100.0)
|
||||||
|
assert first['hours'] == {'2026-11-19T17': {'flood': 1, 'invalid': 0, 'other': 0},
|
||||||
|
'2026-11-19T18': {'flood': 0, 'invalid': 1, 'other': 0}}
|
||||||
|
assert first['last'] == '2026-11-19T18:00:01.2' and first['gap'] is False
|
||||||
|
# The next read overlaps the last one: only the new row counts
|
||||||
|
second = mon._count_firewall_drops(first, [self._row('2026-11-19T18:00:01.2', 'Invalid IP Packet'),
|
||||||
|
self._row('2026-11-19T18:00:03.0')], 400.0)
|
||||||
|
assert second['hours']['2026-11-19T18'] == {'flood': 1, 'invalid': 1, 'other': 0}
|
||||||
|
assert second['checked_at'] == 400.0
|
||||||
|
|
||||||
|
def test_gap_when_the_log_rolled_over(self, mon):
|
||||||
|
prev = {'hours': {}, 'last': '2026-11-19T18:00:00.0'}
|
||||||
|
out = mon._count_firewall_drops(prev, [self._row('2026-11-19T18:20:00.0')], 1.0)
|
||||||
|
assert out['gap'] is True
|
||||||
|
|
||||||
|
def test_keeps_a_day(self, mon):
|
||||||
|
prev = {'hours': {f'2026-11-18T{h:02d}': {'flood': 1, 'invalid': 0, 'other': 0} for h in range(24)},
|
||||||
|
'last': '2026-11-18T23:59:59.0'}
|
||||||
|
out = mon._count_firewall_drops(prev, [self._row('2026-11-19T00:00:01.0')], 1.0)
|
||||||
|
assert len(out['hours']) == 24 and '2026-11-18T00' not in out['hours']
|
||||||
|
|
||||||
|
def test_latest_speed_tests(self, mon):
|
||||||
|
tests = [{'direction': 'upstream', 'mbps': 1242.8, 'at': 'b'},
|
||||||
|
{'direction': 'downstream', 'mbps': 1167.4, 'at': 'b'},
|
||||||
|
{'direction': 'upstream', 'mbps': 1.0, 'at': 'a'}]
|
||||||
|
out = mon._latest_speed_tests(tests, 5.0)
|
||||||
|
assert out['up']['mbps'] == 1242.8 and out['down']['mbps'] == 1167.4 and out['checked_at'] == 5.0
|
||||||
|
|
||||||
|
def test_scheduled_with_the_other_checks(self, mon, db):
|
||||||
|
admin = MagicMock()
|
||||||
|
admin.nat_sessions.return_value = {'in_use': 1, 'available': 10}
|
||||||
|
admin.speed_tests.return_value = []
|
||||||
|
admin.firewall_log.return_value = [self._row('2026-11-19T18:00:00.0')]
|
||||||
|
mon.modem_admin = admin
|
||||||
|
db.get_state.side_effect = lambda k, d=None: None
|
||||||
|
mon._collect_modem_checks()
|
||||||
|
saved = [c.args[1] for c in db.set_state.call_args_list if c.args[0] == 'modem_checks'][-1]
|
||||||
|
assert saved['firewall']['hours'] == {'2026-11-19T18': {'flood': 1, 'invalid': 0, 'other': 0}}
|
||||||
|
assert 'speed' in saved and admin.firewall_log.called
|
||||||
|
|
||||||
|
|
||||||
|
class TestFinishModemRestart:
|
||||||
|
NOW = 1_000_000.0
|
||||||
|
|
||||||
|
def _run(self, mon, db, restart, stats, kuma_error=None):
|
||||||
|
state = {'modem_restart': json.dumps(restart), 'modem_stats': json.dumps(stats)}
|
||||||
|
db.get_state.side_effect = lambda k, d=None: state.get(k, d)
|
||||||
|
db.set_state.reset_mock()
|
||||||
|
mon.cfg = {'uptime_kuma': {'url': 'http://kuma.test'}}
|
||||||
|
with patch.object(monitor.time, 'time', return_value=self.NOW), \
|
||||||
|
patch.object(monitor.kuma, 'set_maintenance', side_effect=kuma_error) as km:
|
||||||
|
mon._finish_modem_restart()
|
||||||
|
saved = [c.args[1] for c in db.set_state.call_args_list if c.args[0] == 'modem_restart']
|
||||||
|
return (saved[-1] if saved else None), km
|
||||||
|
|
||||||
|
def _restart(self, ago, **kw):
|
||||||
|
return dict({'at': self.NOW - ago, 'by': 'jared', 'suppression_id': 9, 'kuma': True,
|
||||||
|
'finished_at': None}, **kw)
|
||||||
|
|
||||||
|
def test_back_ends_kuma_and_the_silence(self, mon, db):
|
||||||
|
out, km = self._run(mon, db, self._restart(300), {'reachable': True, 'wan_problems': [], 'uptime_s': 200})
|
||||||
|
km.assert_called_once_with({'url': 'http://kuma.test'}, False)
|
||||||
|
db.deactivate_suppression.assert_called_once_with(9)
|
||||||
|
assert out['back'] and out['rebooted'] and out['back_after_s'] == 300 and out['finished_at'] == self.NOW
|
||||||
|
|
||||||
|
def test_waits_while_the_modem_is_down(self, mon, db):
|
||||||
|
out, km = self._run(mon, db, self._restart(300), {'reachable': False})
|
||||||
|
assert out is None and not km.called and not db.deactivate_suppression.called
|
||||||
|
|
||||||
|
def test_too_soon_to_tell(self, mon, db):
|
||||||
|
out, _ = self._run(mon, db, self._restart(60), {'reachable': True, 'wan_problems': [], 'uptime_s': 30})
|
||||||
|
assert out is None
|
||||||
|
|
||||||
|
def test_answering_without_having_rebooted_is_not_back(self, mon, db):
|
||||||
|
out, _ = self._run(mon, db, self._restart(300), {'reachable': True, 'wan_problems': [], 'uptime_s': 99999})
|
||||||
|
assert out is None
|
||||||
|
|
||||||
|
def test_gives_up_after_half_an_hour(self, mon, db):
|
||||||
|
out, km = self._run(mon, db, self._restart(1900), {'reachable': False})
|
||||||
|
assert out['back'] is False and out['back_after_s'] is None and km.called
|
||||||
|
db.deactivate_suppression.assert_called_once_with(9)
|
||||||
|
|
||||||
|
def test_kuma_failure_retries_next_cycle(self, mon, db):
|
||||||
|
out, _ = self._run(mon, db, self._restart(300), {'reachable': True, 'wan_problems': [], 'uptime_s': 200},
|
||||||
|
kuma_error=monitor.kuma.KumaError('down'))
|
||||||
|
assert out is None and not db.deactivate_suppression.called
|
||||||
|
|
||||||
|
def test_finished_restart_is_left_alone(self, mon, db):
|
||||||
|
out, km = self._run(mon, db, self._restart(300, finished_at=1.0), {'reachable': True})
|
||||||
|
assert out is None and not km.called
|
||||||
Reference in new issue
Block a user