Compare commits
6
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
de75f71bf8 | ||
|
|
2f4cb05f67 | ||
|
|
fa108b2a1e | ||
|
|
ff425dbaee | ||
|
|
99dee62098 | ||
|
|
4ad331ddfe |
No files matched your search
@@ -92,7 +92,7 @@ Two processes share a MariaDB database:
|
||||
|
||||
| Host | Prometheus Instance |
|
||||
|---|---|
|
||||
| large1 | 10.10.10.2:9100 |
|
||||
| compute-storage-02 | 10.10.10.2:9100 |
|
||||
| compute-storage-01 | 10.10.10.4:9100 |
|
||||
| monitor-01 | 10.10.10.8:9100 |
|
||||
| monitor-02 | 10.10.10.9:9100 |
|
||||
@@ -136,7 +136,7 @@ holds every page and Start maintenance.
|
||||
Every alert Gandalf raised in the last 24 hours to 90 days, including ones that cleared before anyone
|
||||
saw them: when each started, how long it lasted, failed checks, and its ticket.
|
||||
- Filter by device (name, port or interface), alert type, severity, and open or cleared. Filters are
|
||||
kept in the URL, so `/events?target=large1&range=90d` is a shareable link; device pages and tickets
|
||||
kept in the URL, so `/events?target=compute-storage-02&range=90d` is a shareable link; device pages and tickets
|
||||
link to their own history this way.
|
||||
- **Recurring** lists the same alert on the same target more than once in the range, worst first, since
|
||||
a flapping cable or optic shows up here before anyone notices the tickets.
|
||||
@@ -227,7 +227,7 @@ servers along the bottom. Power gear (PDU, UPS, power strip) sits in a rail on t
|
||||
named "Patch panel port 1" (the device beyond the panel is unknown); a stub that already has a name
|
||||
keeps it and adds "via Patch panel port 2". The cable details show where the cutsheet cable goes.
|
||||
- **Port names**: a ⚠ marks a port named after one device with a different one plugged in (a port
|
||||
called "Large1 Management" with the AP on it). Generic names ("Port 4", "SFP+ 2", "Uplink") and
|
||||
called "compute-storage-02 Management" with the AP on it). Generic names ("Port 4", "SFP+ 2", "Uplink") and
|
||||
"Downlink to X" with X on the other end never flag.
|
||||
- **Alerts**: a device with active alerts (not suppressed) gets a ring and a count in its worst
|
||||
severity colour, and the NIC it is about is marked.
|
||||
@@ -511,6 +511,22 @@ periods had loss or downtime, *high latency* (amber) when 3 or more had any prob
|
||||
the last hour. These are dashboard lines, not tickets, and they go once the hour is clean. A copy older
|
||||
than 30 minutes is ignored.
|
||||
|
||||
**With `modem.access_code` set**, the monitor also logs in to the gateway (`ModemAdmin` in `modem.py`)
|
||||
for two read-only checks, stored in `monitor_state.modem_checks`:
|
||||
- **NAT sessions** every 30 minutes, from the NAT Table page. Needs attention warns when 80% of the
|
||||
gateway's sessions (32,767 on a BGW320) are in use, since new connections stall once it is full.
|
||||
- **A ping from the gateway itself** to 1.1.1.1, from the Troubleshoot page, whenever UniFi flags the
|
||||
current 5 minutes as slow or measures 150 ms or more (at most every 10 minutes). Compared with UniFi's
|
||||
latency through the gateway at the same time, it says where the delay is: *past the modem* (its own
|
||||
ping is slow too, so the provider), *in the modem* (its ping is fine but ours through it is slow), or
|
||||
*fine by then*. The result goes in the detail of the Internet line in Needs attention and on the
|
||||
WAN / Fiber panel.
|
||||
|
||||
The login posts md5(access code + nonce), as the gateway's own page does, so the code itself is never sent.
|
||||
The session is kept between checks and renewed when it expires. A refused code is reported in Needs
|
||||
attention and not tried again for 30 minutes, and any other failure pauses the checks for 10 minutes, so
|
||||
the gateway is never retried in a loop. The status polling above doesn't log in and is unaffected.
|
||||
|
||||
**One ticket per incident.** The modem and WAN reports from one check cycle open a single ticket, led by
|
||||
the most urgent (a P2 "WAN modem unreachable" leads, and an RPDU power-cycle seen in the same cycle is listed
|
||||
under "Also seen in this incident"). A report that comes while the modem has an open alert, or one that
|
||||
@@ -588,7 +604,7 @@ Shared by both processes. Located in the working directory (`/var/www/html/prod/
|
||||
"trusted_proxies": ["10.10.10.27"]
|
||||
},
|
||||
"hosts": [
|
||||
{ "name": "large1", "prometheus_instance": "10.10.10.2:9100" },
|
||||
{ "name": "compute-storage-02", "prometheus_instance": "10.10.10.2:9100" },
|
||||
{ "name": "compute-storage-01", "prometheus_instance": "10.10.10.4:9100" },
|
||||
{ "name": "monitor-01", "prometheus_instance": "10.10.10.8:9100" },
|
||||
{ "name": "monitor-02", "prometheus_instance": "10.10.10.9:9100" },
|
||||
@@ -654,6 +670,7 @@ Shared by both processes. Located in the working directory (`/var/www/html/prod/
|
||||
| `modem.failure_threshold` | Consecutive failures before ticketing a modem alert (default: same as `failure_threshold`). Raise it if gateway reboots or automatic firmware updates cut tickets. |
|
||||
| `modem.rx_drift_db` | Rx drop below the 7-day median that raises `fiber_rx_drift` (default: 3.0) |
|
||||
| `modem.rx_drift_min_samples` | Readings required before drift alerts start (default: 288) |
|
||||
| `modem.access_code` | Optional. The gateway's Device Access Code (printed on its label). With it, Gandalf logs in to read NAT session counts and to ping from the gateway when the internet is slow (see WAN modem notes). Keep it only in `config.json`, which is git-ignored and mode 600. |
|
||||
|
||||
---
|
||||
|
||||
@@ -838,7 +855,7 @@ curl http://10.10.10.48:9090/api/v1/query?query=up | jq '.data.result[] | select
|
||||
gandalf/
|
||||
├── app.py # Flask web app (routes, auth, API endpoints)
|
||||
├── monitor.py # Background daemon (Prometheus, UniFi, Pulse, alert logic)
|
||||
├── modem.py # Read-only AT&T gateway status page client and parser
|
||||
├── modem.py # AT&T gateway client: status pages, plus optional logged-in NAT and ping checks
|
||||
├── devices.py # Device inventory for search and device pages
|
||||
├── topology.py # The topology graph: devices, cables, depth, last-seen links
|
||||
├── cutsheet.py # Cutsheet validation, path tracing, plan-vs-actual checks, CSV
|
||||
@@ -920,7 +937,7 @@ Take LED colors from the model's quick start guide. See the comment at the top o
|
||||
`recovery_notified` once the recovery note is on its ticket (column added automatically at monitor startup)
|
||||
- `suppression_rules`: `active=FALSE` when removed; `expires_at` checked at query time
|
||||
- `monitor_state`: key/value store; `interface_baseline`, `link_stats`, and `modem_stats` are JSON blobs
|
||||
- `monitor_state` also holds `wan_history` (WAN outages, modem restarts), `wan_quality` (UniFi's 5 minute WAN health timeline) and `unifi_log_cursor` (how far the UniFi log has been read)
|
||||
- `monitor_state` also holds `wan_history` (WAN outages, modem restarts), `wan_quality` (UniFi's 5 minute WAN health timeline), `modem_checks` (NAT sessions and the latest ping from the gateway) and `unifi_log_cursor` (how far the UniFi log has been read)
|
||||
- Database sessions run in UTC (`db.py`), whatever the server's time zone
|
||||
- `modem_samples`: optical readings per poll, kept 90 days; created automatically at monitor startup
|
||||
- `cutsheet_items`, `cutsheet_cables`, `cutsheet_port_labels`, `cutsheet_history`: cutsheet records and
|
||||
@@ -976,7 +993,7 @@ Branch protection is enabled on `main`: both lint jobs must pass before any PR c
|
||||
Tests live in `tests/`:
|
||||
- `test_diagnose.py`: `DiagnosticsRunner` static methods (`build_ssh_command`, `parse_output`,
|
||||
`parse_sysfs_stats`, `parse_ethtool`, and variants)
|
||||
- `test_modem.py`: gateway page parsing, optical status, and WAN state
|
||||
- `test_modem.py`: gateway page parsing, optical status, WAN state, and the logged-in checks against a fake gateway
|
||||
- `test_monitor.py`: alert logic for dead hosts, Prometheus and Pulse outages, and modem and WAN log events
|
||||
- `test_devices.py`: device inventory behind search and device pages
|
||||
- `test_app.py`: page and API routes
|
||||
@@ -556,6 +556,7 @@ def api_modem():
|
||||
data['history'] = []
|
||||
data['wan_history'] = _state_json('wan_history')
|
||||
data['wan_quality'] = _state_json('wan_quality') or None
|
||||
data['modem_checks'] = _state_json('modem_checks') or None
|
||||
data['configured'] = True
|
||||
return jsonify(data)
|
||||
|
||||
@@ -1360,7 +1361,8 @@ def _attention() -> dict:
|
||||
logger.error(f'Cutsheet unavailable for the attention list: {e}')
|
||||
sheet = {'items': (view or {}).get('items') or [], 'cables': (view or {}).get('cables') or []}
|
||||
power = _topology_graph(events, sheet, feeds).get('power') or []
|
||||
return attention.build(events, power, view, racks, _state_json('wan_quality'))
|
||||
return attention.build(events, power, view, racks, _state_json('wan_quality'),
|
||||
modem_checks=_state_json('modem_checks'))
|
||||
|
||||
|
||||
def _topology_graph(events: list, sheet: dict, feeds: list) -> dict:
|
||||
|
||||
+43
-4
@@ -115,7 +115,42 @@ def _ago(seconds: float) -> str:
|
||||
return f'{minutes} min ago' if minutes < 60 else f'{round(minutes / 60)} h ago'
|
||||
|
||||
|
||||
def _wan(quality: Optional[dict], now: float) -> List[dict]:
|
||||
_NAT_FULL = 0.8 # share of the gateway's NAT sessions in use before it's worth a line
|
||||
|
||||
|
||||
def _modem_ping_note(checks: Optional[dict], now: float) -> str:
|
||||
"""One sentence on the latest ping from the modem itself, if it ran in the last hour."""
|
||||
ping = (checks or {}).get('ping') or {}
|
||||
if not ping or now - (ping.get('at') or 0) > _WAN_WINDOW_S:
|
||||
return ''
|
||||
when, avg, ours = _ago(now - ping['at']), ping.get('avg_ms'), ping.get('ours_ms')
|
||||
theirs = (f'{round(avg)} ms' if avg is not None else 'no reply') + (
|
||||
f', {ping["loss_pct"]}% lost' if ping.get('loss_pct') else '')
|
||||
ours_txt = f'; ours through it {round(ours)} ms' if ours is not None else ''
|
||||
verdict = {
|
||||
'upstream': "so the delay is past the modem, on the provider's side",
|
||||
'gateway': 'so the delay is in the modem, or between it and the router',
|
||||
'clear': 'both fine by then',
|
||||
}.get(ping.get('verdict'), '')
|
||||
return f"Modem's own ping {when}: {theirs}{ours_txt}, {verdict}. "
|
||||
|
||||
|
||||
def _modem_checks(checks: Optional[dict]) -> List[dict]:
|
||||
out = []
|
||||
nat = (checks or {}).get('nat') or {}
|
||||
if nat.get('available') and nat.get('in_use', 0) / nat['available'] >= _NAT_FULL:
|
||||
pct = round(100 * nat['in_use'] / nat['available'])
|
||||
out.append(_entry('broken', 'warn', f'Modem NAT table {pct}% full',
|
||||
'#modem-section', f'{nat["in_use"]:,} of {nat["available"]:,} sessions in use. '
|
||||
'When it fills, new connections stall or fail.'))
|
||||
error = (checks or {}).get('error') or {}
|
||||
if error.get('login'):
|
||||
out.append(_entry('tidy', 'warn', 'Modem login failed: check modem.access_code in config.json',
|
||||
'#modem-section', 'Gandalf waits 30 minutes before trying again.'))
|
||||
return out
|
||||
|
||||
|
||||
def _wan(quality: Optional[dict], now: float, checks: Optional[dict] = None) -> List[dict]:
|
||||
if not quality or now - (quality.get('updated') or 0) > _WAN_STALE_S:
|
||||
return []
|
||||
out = []
|
||||
@@ -133,6 +168,7 @@ def _wan(quality: Optional[dict], now: float) -> List[dict]:
|
||||
if day_bad else '')
|
||||
if latencies:
|
||||
detail += 'Now: ' + ', '.join(f'{k} {v} ms' for k, v in sorted(latencies.items())) + '. '
|
||||
detail += _modem_ping_note(checks, now)
|
||||
detail += 'See the 24 hour strip on the WAN / Fiber panel.'
|
||||
lossy = recent['loss'] + recent['down']
|
||||
if lossy >= _WAN_LOSS_PERIODS:
|
||||
@@ -154,16 +190,19 @@ def _wan(quality: Optional[dict], now: float) -> List[dict]:
|
||||
|
||||
def build(events: List[dict], power: List[dict], cutsheet_view: Optional[dict],
|
||||
rack_issues: Optional[Dict[str, dict]] = None, wan_quality: Optional[dict] = None,
|
||||
now: Optional[float] = None) -> dict:
|
||||
now: Optional[float] = None, modem_checks: Optional[dict] = None) -> dict:
|
||||
"""{'broken': [entry], 'tidy': [entry]}, worst first within each group.
|
||||
|
||||
events: active alerts, annotated with 'is_suppressed'. power: topology.build_power output.
|
||||
cutsheet_view: the cutsheet API view (cables with 'problem', suggestions, port_names,
|
||||
labels), or None when the cutsheet couldn't be loaded. rack_issues: {rack: rack_layout()}.
|
||||
wan_quality: modem.wan_quality() output as stored by the monitor. now: for tests.
|
||||
wan_quality: modem.wan_quality() output as stored by the monitor. modem_checks: the
|
||||
monitor's logged-in modem checks (NAT sessions, ping from the modem). now: for tests.
|
||||
Entry: {group, severity: bad|warn|info, text, href, detail}.
|
||||
"""
|
||||
entries = _wan(wan_quality, time.time() if now is None else now) + _alerts(events) + _power(power)
|
||||
now = time.time() if now is None else now
|
||||
entries = (_wan(wan_quality, now, modem_checks) + _modem_checks(modem_checks)
|
||||
+ _alerts(events) + _power(power))
|
||||
if cutsheet_view is not None:
|
||||
entries += _cutsheet(cutsheet_view)
|
||||
entries += _rack(rack_issues or {})
|
||||
|
||||
@@ -0,0 +1,84 @@
|
||||
{
|
||||
"pulse": {
|
||||
"url": "http://10.10.10.65:8080",
|
||||
"api_key": "012b303a324152c509bf5ade6f942cfc21404f68662f01a17001cba9e4486049",
|
||||
"worker_id": "1b11d1b5-4ed0-42df-a6af-8d57fffe1343",
|
||||
"timeout": 45
|
||||
},
|
||||
"unifi": {
|
||||
"controller": "https://10.10.10.1",
|
||||
"api_key": "kyPfIsAVie3hwMD4Bc1MjAu8N7HVPIb8",
|
||||
"site_id": "default"
|
||||
},
|
||||
"prometheus": {
|
||||
"url": "http://10.10.10.48:9090"
|
||||
},
|
||||
"database": {
|
||||
"host": "10.10.10.50",
|
||||
"port": 3306,
|
||||
"user": "gandalf",
|
||||
"password": "Gandalf2026Lotus",
|
||||
"name": "gandalf"
|
||||
},
|
||||
"ticket_api": {
|
||||
"url": "http://10.10.10.45/create_ticket_api.php",
|
||||
"api_key": "5acc5d3c647b84f7c6f59082ce4450ee772e2d1633238b960136f653d20c93af"
|
||||
},
|
||||
"ldap": {
|
||||
"host": "10.10.10.39",
|
||||
"port": 3890,
|
||||
"bind_dn": "uid=gandalf,ou=people,dc=example,dc=com",
|
||||
"bind_pw": "AZ-eUFRxamnlLEELuDy5-Mh6z-2cJbL_YJZnu64Tf0A",
|
||||
"user_base": "ou=people,dc=example,dc=com",
|
||||
"cache_dir": "/tmp/gandalf_avatars",
|
||||
"cache_ttl": 3600
|
||||
},
|
||||
"auth": {
|
||||
"allowed_groups": ["admin"]
|
||||
},
|
||||
"monitor": {
|
||||
"poll_interval": 300,
|
||||
"failure_threshold": 2,
|
||||
"unifi_failure_threshold": 4,
|
||||
"cluster_threshold": 3,
|
||||
"ping_hosts": [],
|
||||
"links_exclude_ips": ["10.10.10.29", "10.10.10.44", "10.10.10.3"]
|
||||
},
|
||||
"hosts": [
|
||||
{
|
||||
"name": "pbs",
|
||||
"ip": "10.10.10.3",
|
||||
"prometheus_instance": "10.10.10.3:9100"
|
||||
},
|
||||
{
|
||||
"name": "large1",
|
||||
"ip": "10.10.10.2",
|
||||
"prometheus_instance": "10.10.10.2:9100"
|
||||
},
|
||||
{
|
||||
"name": "compute-storage-01",
|
||||
"ip": "10.10.10.4",
|
||||
"prometheus_instance": "10.10.10.4:9100"
|
||||
},
|
||||
{
|
||||
"name": "monitor-01",
|
||||
"ip": "10.10.10.8",
|
||||
"prometheus_instance": "10.10.10.8:9100"
|
||||
},
|
||||
{
|
||||
"name": "monitor-02",
|
||||
"ip": "10.10.10.9",
|
||||
"prometheus_instance": "10.10.10.9:9100"
|
||||
},
|
||||
{
|
||||
"name": "compute-storage-gpu-01",
|
||||
"ip": "10.10.10.10",
|
||||
"prometheus_instance": "10.10.10.10:9100"
|
||||
},
|
||||
{
|
||||
"name": "storage-01",
|
||||
"ip": "10.10.10.11",
|
||||
"prometheus_instance": "10.10.10.11:9100"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -4,6 +4,10 @@ The gateway has no API, but its status pages are served without a login, so
|
||||
this scrapes them over HTTP. Only GET requests to the fixed status pages below
|
||||
are ever made: nothing here can change gateway settings.
|
||||
|
||||
With an access code configured, ModemAdmin (at the end) also logs in to read the
|
||||
NAT session counts and to run the Troubleshoot page's ping from the gateway.
|
||||
Those are a login, page reads and a ping test: still no setting is changed.
|
||||
|
||||
Units come from the gateway's own help text, checked against live readings:
|
||||
Rx/Tx power tenths of a dBm (-169 -> -16.9 dBm)
|
||||
Temperature degrees C
|
||||
@@ -16,9 +20,11 @@ Deliberately not used, because the page disagrees with itself:
|
||||
Status" is the reliable signal
|
||||
Last Change does not count up, despite the help text
|
||||
"""
|
||||
import hashlib
|
||||
import html
|
||||
import logging
|
||||
import re
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from typing import Dict, List, Optional
|
||||
|
||||
@@ -360,3 +366,154 @@ def quality_counts(periods: List[dict], since: float) -> Dict[str, int]:
|
||||
if p['ts'] >= since and p['state'] in counts:
|
||||
counts[p['state']] += 1
|
||||
return counts
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Logged-in checks, used only when an access code is configured. Two pages are
|
||||
# read: the NAT table's session counts, and the Troubleshoot page's ping test,
|
||||
# which runs a ping from the gateway itself. Neither changes any setting.
|
||||
# ---------------------------------------------------------------------------
|
||||
_LOGIN_PAGE = '/cgi-bin/login.ha'
|
||||
_NAT_PAGE = '/cgi-bin/nattable.ha'
|
||||
_DIAG_PAGE = '/cgi-bin/diag.ha'
|
||||
_NONCE_RE = re.compile(r'name="nonce"\s+value="([0-9a-f]+)"', re.I)
|
||||
_PROGRESS_RE = re.compile(r'<textarea[^>]*name="ProgressWindow"[^>]*>(.*?)</textarea>', re.I | re.S)
|
||||
_PING_STATS_RE = re.compile(r'(\d+) packets transmitted, (\d+) (?:packets )?received')
|
||||
_PING_RTT_RE = re.compile(r'min/avg/max\S* = ([\d.]+)/([\d.]+)/([\d.]+)')
|
||||
# After a failed login, leave the gateway alone for this long (it may lock the account)
|
||||
LOGIN_BACKOFF_S = 1800
|
||||
|
||||
|
||||
class ModemLoginFailed(Exception):
|
||||
"""The access code was refused, or the login page wasn't what we expect."""
|
||||
|
||||
|
||||
def login_hash(code: str, nonce: str) -> str:
|
||||
"""What the login page's script posts: md5 of the access code followed by the nonce."""
|
||||
# The gateway's login protocol dictates md5; it isn't used to protect anything here
|
||||
return hashlib.md5((code + nonce).encode(), usedforsecurity=False).hexdigest() # nosec B303 B324
|
||||
|
||||
|
||||
def is_login_page(page: str) -> bool:
|
||||
return 'Access Code Required' in page or bool(re.search(r'<title>\s*Login', page, re.I))
|
||||
|
||||
|
||||
def parse_nat_summary(page: str) -> Optional[Dict[str, int]]:
|
||||
"""{'in_use', 'available'} from the NAT table page, or None if the counts are missing."""
|
||||
rows = {_TAG_RE.sub('', k).strip(): _TAG_RE.sub('', v).strip() for k, v in _ROW_RE.findall(page)}
|
||||
try:
|
||||
return {'in_use': int(rows['Total sessions in use']),
|
||||
'available': int(rows['Total sessions available'])}
|
||||
except (KeyError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def parse_ping_output(text: str) -> Optional[dict]:
|
||||
"""{'sent', 'received', 'loss_pct', 'min_ms', 'avg_ms', 'max_ms'} from busybox ping output,
|
||||
or None until the test has finished. Times are None when every packet was lost."""
|
||||
stats = _PING_STATS_RE.search(text or '')
|
||||
if not stats:
|
||||
return None
|
||||
sent, received = int(stats.group(1)), int(stats.group(2))
|
||||
rtt = _PING_RTT_RE.search(text)
|
||||
lo, avg, hi = (float(x) for x in rtt.groups()) if rtt else (None, None, None)
|
||||
return {'sent': sent, 'received': received,
|
||||
'loss_pct': round(100 * (sent - received) / sent) if sent else 100,
|
||||
'min_ms': lo, 'avg_ms': avg, 'max_ms': hi}
|
||||
|
||||
|
||||
class ModemAdmin:
|
||||
"""Logged-in session on the gateway, kept between checks and renewed when it expires.
|
||||
|
||||
Logs in at most once per call, and not at all for LOGIN_BACKOFF_S after a refusal.
|
||||
Raises ModemUnreachable, ModemDataInvalid or ModemLoginFailed.
|
||||
"""
|
||||
|
||||
def __init__(self, cfg: dict, clock=None, sleep=None):
|
||||
self.base_url = cfg['url'].rstrip('/')
|
||||
self._code = cfg['access_code']
|
||||
self.timeout = cfg.get('timeout', 10)
|
||||
self.session = requests.Session()
|
||||
self._clock = clock or time.time
|
||||
self._sleep = sleep or time.sleep
|
||||
self.login_failed_at: Optional[float] = None
|
||||
|
||||
def _request(self, method: str, path: str, timeout: Optional[float] = None, **kw) -> requests.Response:
|
||||
try:
|
||||
return self.session.request(method, f'{self.base_url}{path}', allow_redirects=False,
|
||||
timeout=timeout or self.timeout, **kw)
|
||||
except requests.RequestException as e:
|
||||
raise ModemUnreachable(f'{path}: {e}') from e
|
||||
|
||||
def _login(self) -> None:
|
||||
if self.login_failed_at and self._clock() - self.login_failed_at < LOGIN_BACKOFF_S:
|
||||
raise ModemLoginFailed('waiting after a refused login')
|
||||
page = self._request('GET', _LOGIN_PAGE).text
|
||||
nonce = _NONCE_RE.search(page)
|
||||
if not nonce: # the first visit only sets the session cookie
|
||||
nonce = _NONCE_RE.search(self._request('GET', _LOGIN_PAGE).text)
|
||||
if not nonce:
|
||||
raise ModemDataInvalid('login page has no nonce')
|
||||
resp = self._request('POST', _LOGIN_PAGE, data={
|
||||
'nonce': nonce.group(1), 'password': '*' * len(self._code), 'Continue': 'Continue',
|
||||
'hashpassword': login_hash(self._code, nonce.group(1)),
|
||||
})
|
||||
if resp.status_code != 302 or is_login_page(resp.text):
|
||||
self.login_failed_at = self._clock()
|
||||
raise ModemLoginFailed('the gateway refused the access code')
|
||||
self.login_failed_at = None
|
||||
|
||||
def _page(self, path: str, timeout: Optional[float] = None) -> str:
|
||||
"""A protected page, logging in first if the session has expired."""
|
||||
resp = self._request('GET', path, timeout)
|
||||
if resp.status_code != 200 or is_login_page(resp.text):
|
||||
self._login()
|
||||
resp = self._request('GET', path, timeout)
|
||||
if resp.status_code != 200 or is_login_page(resp.text):
|
||||
raise ModemDataInvalid(f'{path} returned HTTP {resp.status_code} after logging in')
|
||||
return resp.text
|
||||
|
||||
def nat_sessions(self) -> Dict[str, int]:
|
||||
# The page lists every session too, so it can be large (hundreds of KB)
|
||||
counts = parse_nat_summary(self._page(_NAT_PAGE, timeout=max(self.timeout, 30)))
|
||||
if counts is None:
|
||||
raise ModemDataInvalid('NAT table page has no session counts')
|
||||
return counts
|
||||
|
||||
def ping(self, target: str, wait_s: float = 30) -> dict:
|
||||
"""Ping *target* from the gateway (its Troubleshoot page; 5 packets) and return the result."""
|
||||
nonce = _NONCE_RE.search(self._page(_DIAG_PAGE))
|
||||
if not nonce:
|
||||
raise ModemDataInvalid('Troubleshoot page has no nonce')
|
||||
resp = self._request('POST', _DIAG_PAGE, timeout=max(self.timeout, 30), data={
|
||||
'nonce': nonce.group(1), 'WebAddress': target, 'protopref': 'IPv4', 'Ping': 'Ping'})
|
||||
deadline = self._clock() + wait_s
|
||||
page = resp.text
|
||||
while True:
|
||||
window = _PROGRESS_RE.search(page)
|
||||
result = parse_ping_output(html.unescape(window.group(1)) if window else '')
|
||||
if result is not None:
|
||||
return dict(result, target=target)
|
||||
if self._clock() >= deadline:
|
||||
raise ModemDataInvalid(f'ping to {target} from the gateway did not finish')
|
||||
self._sleep(3)
|
||||
page = self._page(_DIAG_PAGE)
|
||||
|
||||
|
||||
SLOW_MS = 150 # a ping slower than this, on a link that normally answers in ~15 ms, is "slow"
|
||||
|
||||
|
||||
def ping_verdict(gateway: dict, ours_ms: Optional[float]) -> str:
|
||||
"""Where the delay is, from the gateway's own ping and ours through it at the same time.
|
||||
|
||||
'upstream': the gateway's ping is slow or lossy too, so it's past the gateway (the ISP).
|
||||
'gateway': the gateway's ping is fine but ours through it is slow, so the delay is in
|
||||
the gateway or between it and the router.
|
||||
'clear': both fine; the slow spell had passed by the time of the test.
|
||||
"""
|
||||
avg = gateway.get('avg_ms')
|
||||
if avg is None or avg >= SLOW_MS or (gateway.get('loss_pct') or 0) >= 20:
|
||||
return 'upstream'
|
||||
if ours_ms is not None and ours_ms >= SLOW_MS:
|
||||
return 'gateway'
|
||||
return 'clear'
|
||||
+60
-1
@@ -914,7 +914,7 @@ class LinkStatsCollector:
|
||||
"""
|
||||
Collect full link stats for all Prometheus-monitored hosts.
|
||||
|
||||
*instance_map*: ``{'10.10.10.2:9100': 'large1', ...}``
|
||||
*instance_map*: ``{'10.10.10.2:9100': 'compute-storage-02', ...}``
|
||||
|
||||
Returns a dict suitable for ``db.set_state('link_stats', ...)``.
|
||||
"""
|
||||
@@ -1111,6 +1111,9 @@ class NetworkMonitor:
|
||||
modem_cfg = self.cfg.get('modem') or {}
|
||||
self.modem = modem_api.ModemClient(modem_cfg) if modem_cfg.get('url') else None
|
||||
self.modem_name = modem_cfg.get('name', 'att-modem')
|
||||
# With the gateway's access code, also read its NAT sessions and ping from it
|
||||
self.modem_admin = (modem_api.ModemAdmin(modem_cfg)
|
||||
if modem_cfg.get('url') and modem_cfg.get('access_code') else None)
|
||||
# Modem reboots (including AT&T's automatic firmware updates) take a few
|
||||
# minutes, so this can be raised independently of failure_threshold.
|
||||
self.modem_fail_thresh = modem_cfg.get('failure_threshold', self.fail_thresh)
|
||||
@@ -1970,6 +1973,56 @@ class NetworkMonitor:
|
||||
if quality is not None:
|
||||
db.set_state('wan_quality', quality)
|
||||
|
||||
_NAT_EVERY_S = 1800
|
||||
_PING_EVERY_S = 600
|
||||
_PING_TARGET = '1.1.1.1'
|
||||
|
||||
def _collect_modem_checks(self) -> None:
|
||||
"""NAT session counts every 30 minutes, and while UniFi says the internet is slow,
|
||||
a ping from the modem itself (at most every 10 minutes) to show where the delay is.
|
||||
Results go to monitor_state.modem_checks for the WAN panel and Needs attention."""
|
||||
admin = getattr(self, 'modem_admin', None)
|
||||
if admin is None:
|
||||
return
|
||||
raw = db.get_state('modem_checks')
|
||||
try:
|
||||
checks = json.loads(raw) if raw else {}
|
||||
except ValueError:
|
||||
checks = {}
|
||||
now = time.time()
|
||||
raw = db.get_state('wan_quality')
|
||||
try:
|
||||
quality = json.loads(raw) if raw else {}
|
||||
except ValueError:
|
||||
quality = {}
|
||||
periods = quality.get('periods') or []
|
||||
latencies = quality.get('latencies') or {}
|
||||
ours = latencies.get('cloudflare', min(latencies.values()) if latencies else None)
|
||||
slow = (bool(periods) and periods[-1].get('state') != 'ok') or (ours or 0) >= modem_api.SLOW_MS
|
||||
|
||||
due_nat = now - (checks.get('nat') or {}).get('at', 0) >= self._NAT_EVERY_S
|
||||
due_ping = slow and now - (checks.get('ping') or {}).get('at', 0) >= self._PING_EVERY_S
|
||||
if not (due_nat or due_ping):
|
||||
return
|
||||
if now - (checks.get('error') or {}).get('at', 0) < self._PING_EVERY_S:
|
||||
return # failed recently: don't keep knocking on the gateway
|
||||
try:
|
||||
if due_nat:
|
||||
checks['nat'] = dict(admin.nat_sessions(), at=now)
|
||||
if due_ping:
|
||||
result = admin.ping(self._PING_TARGET)
|
||||
checks['ping'] = dict(result, at=now, ours_ms=ours,
|
||||
verdict=modem_api.ping_verdict(result, ours))
|
||||
logger.info(f'Modem ping to {self._PING_TARGET}: {result.get("avg_ms")} ms, '
|
||||
f'{result.get("loss_pct")}% loss; ours {ours} ms '
|
||||
f'({checks["ping"]["verdict"]})')
|
||||
checks['error'] = None
|
||||
except (modem_api.ModemUnreachable, modem_api.ModemDataInvalid, modem_api.ModemLoginFailed) as e:
|
||||
logger.warning(f'Modem checks: {e}')
|
||||
checks['error'] = {'message': str(e), 'at': now,
|
||||
'login': isinstance(e, modem_api.ModemLoginFailed)}
|
||||
db.set_state('modem_checks', checks)
|
||||
|
||||
def _collect_modem(self, suppressions: list) -> None:
|
||||
"""Poll the modem, then store its status and raise any alerts."""
|
||||
try:
|
||||
@@ -2161,6 +2214,12 @@ class NetworkMonitor:
|
||||
except Exception as e:
|
||||
logger.error(f'WAN quality collection failed: {e}', exc_info=True)
|
||||
|
||||
# Logged-in modem checks (NAT sessions; a ping from the modem when the internet is slow)
|
||||
try:
|
||||
self._collect_modem_checks()
|
||||
except Exception as e:
|
||||
logger.error(f'Modem checks failed: {e}', exc_info=True)
|
||||
|
||||
# Note on the tickets of anything that cleared this cycle (or failed to post before)
|
||||
try:
|
||||
self._post_recovery_notes()
|
||||
|
||||
@@ -373,6 +373,35 @@ function _modemUptimeStat(data) {
|
||||
return _modemStat('Last restart', _modemWhen(last.at), '', last.by ? `by ${last.by}` : '');
|
||||
}
|
||||
|
||||
const _PING_VERDICTS = {
|
||||
upstream: 'slow past the modem: provider side',
|
||||
gateway: 'modem fine, ours slow: delay in the modem',
|
||||
clear: 'fine',
|
||||
};
|
||||
|
||||
// Logged-in checks: NAT sessions in use, and the latest ping run from the modem itself
|
||||
function _modemChecks(c) {
|
||||
if (!c) return '';
|
||||
const ago = at => lt.time.ago(new Date(at * 1000));
|
||||
let out = '';
|
||||
if (c.nat && c.nat.available) {
|
||||
const pct = Math.round(100 * c.nat.in_use / c.nat.available);
|
||||
out += _modemStat('NAT sessions', `${c.nat.in_use.toLocaleString()} / ${c.nat.available.toLocaleString()}`,
|
||||
pct >= 80 ? 'lt-text-amber' : '', `${pct}% in use, ${ago(c.nat.at)}`);
|
||||
}
|
||||
if (c.ping) {
|
||||
const p = c.ping;
|
||||
const value = p.avg_ms === null || p.avg_ms === undefined ? 'no reply' : `${Math.round(p.avg_ms)} ms`;
|
||||
const verdict = _PING_VERDICTS[p.verdict] || '';
|
||||
out += _modemStat("Modem's own ping", value, p.verdict === 'clear' ? '' : 'lt-text-amber',
|
||||
`to ${p.target}, ${ago(p.at)}${verdict ? `: ${verdict}` : ''}`);
|
||||
}
|
||||
if (c.error && c.error.login) {
|
||||
out += _modemStat('Modem login', 'failed', 'lt-text-red', 'check modem.access_code');
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function _modemDuration(seconds) {
|
||||
if (seconds < 60) return `${Math.round(seconds)}s`;
|
||||
if (seconds < 3600) return `${Math.floor(seconds / 60)}m ${Math.round(seconds % 60)}s`;
|
||||
@@ -478,6 +507,7 @@ function updateModem(data) {
|
||||
${_modemStat('Throughput', `↓ ${_modemRate(rates.rx_bytes)}`, '', `↑ ${_modemRate(rates.tx_bytes)}`)}
|
||||
${_modemStat('Errors / drops', `${errs} / ${drops}`, errs ? 'lt-text-amber' : '', 'since modem reboot')}
|
||||
${_modemUptimeStat(data)}
|
||||
${_modemChecks(data.modem_checks)}
|
||||
</div>
|
||||
${_modemQuality(data.wan_quality)}
|
||||
${_modemChart(data.history)}
|
||||
|
||||
+1
-1
@@ -290,7 +290,7 @@
|
||||
</div>
|
||||
<div class="lt-form-group" id="sup-name-group">
|
||||
<label class="lt-label" for="sup-name">Target Name</label>
|
||||
<input type="text" class="lt-input" id="sup-name" name="target_name" placeholder="e.g. large1">
|
||||
<input type="text" class="lt-input" id="sup-name" name="target_name" placeholder="e.g. compute-storage-02">
|
||||
</div>
|
||||
<div class="lt-form-group" id="sup-detail-group" style="display:none">
|
||||
<label class="lt-label" for="sup-detail">Interface <span class="lt-field-hint">(interface type only)</span></label>
|
||||
|
||||
@@ -50,7 +50,7 @@
|
||||
<div class="lt-form-group form-group-wide">
|
||||
<label class="lt-label" for="s-reason">Reason <span class="required">*</span></label>
|
||||
<input type="text" class="lt-input" id="s-reason" name="reason"
|
||||
placeholder="e.g. Planned switch maintenance, replacing SFP on large1/enp43s0"
|
||||
placeholder="e.g. Planned switch maintenance, replacing SFP on compute-storage-02/enp43s0"
|
||||
required aria-required="true">
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -143,3 +143,41 @@ class TestWan:
|
||||
out = attention.build([_alert('x', 'warning')], [], None,
|
||||
wan_quality=_quality(['latency'] * 12), now=NOW)['broken']
|
||||
assert out[0]['text'].startswith('Internet')
|
||||
|
||||
|
||||
class TestModemChecks:
|
||||
def _broken(self, checks, quality=None):
|
||||
return attention.build([], [], None, wan_quality=quality, now=NOW, modem_checks=checks)
|
||||
|
||||
def test_gateway_ping_explains_the_latency(self):
|
||||
checks = {'ping': {'at': NOW - 120, 'avg_ms': 20.4, 'loss_pct': 0, 'ours_ms': 1043, 'verdict': 'gateway'}}
|
||||
[e] = self._broken(checks, _quality(['latency'] * 12))['broken']
|
||||
assert ("Modem's own ping 2 min ago: 20 ms; ours through it 1043 ms, "
|
||||
'so the delay is in the modem, or between it and the router.') in e['detail']
|
||||
|
||||
def test_upstream_verdict(self):
|
||||
checks = {'ping': {'at': NOW - 60, 'avg_ms': 900.0, 'loss_pct': 20, 'ours_ms': 1000, 'verdict': 'upstream'}}
|
||||
[e] = self._broken(checks, _quality(['latency'] * 12))['broken']
|
||||
assert ("900 ms, 20% lost; ours through it 1000 ms, "
|
||||
"so the delay is past the modem, on the provider's side") in e['detail']
|
||||
|
||||
def test_old_ping_is_left_out(self):
|
||||
checks = {'ping': {'at': NOW - 7200, 'avg_ms': 20.0, 'verdict': 'clear'}}
|
||||
[e] = self._broken(checks, _quality(['latency'] * 12))['broken']
|
||||
assert "Modem's own ping" not in e['detail']
|
||||
|
||||
def test_nat_nearly_full(self):
|
||||
[e] = self._broken({'nat': {'in_use': 30000, 'available': 32767}})['broken']
|
||||
assert e['text'] == 'Modem NAT table 92% full' and e['severity'] == 'warn'
|
||||
assert '30,000 of 32,767' in e['detail']
|
||||
|
||||
def test_nat_normal_says_nothing(self):
|
||||
out = self._broken({'nat': {'in_use': 2858, 'available': 32767}})
|
||||
assert out == {'broken': [], 'tidy': []}
|
||||
|
||||
def test_login_failure_is_housekeeping(self):
|
||||
[e] = self._broken({'error': {'login': True, 'message': 'refused', 'at': NOW}})['tidy']
|
||||
assert e['text'].startswith('Modem login failed')
|
||||
|
||||
def test_other_errors_stay_quiet(self):
|
||||
assert self._broken({'error': {'login': False, 'message': 'timeout', 'at': NOW}})['tidy'] == []
|
||||
@@ -301,3 +301,157 @@ class TestWanQuality:
|
||||
def test_counts_since(self):
|
||||
periods = [{'ts': 0, 'state': 'latency'}, {'ts': 300, 'state': 'ok'}, {'ts': 600, 'state': 'loss'}]
|
||||
assert modem.quality_counts(periods, 300) == {'ok': 1, 'latency': 0, 'loss': 1, 'down': 0}
|
||||
|
||||
|
||||
# Trimmed from a BGW320-505 (firmware 6.40.4)
|
||||
NAT_PAGE = '''<table class="table60" summary="This table displays a summary of session information.">
|
||||
<tr>
|
||||
<th scope="row" nowrap="nowrap">Total sessions available</th>
|
||||
<td>32767
|
||||
</td></tr>
|
||||
<tr>
|
||||
<th scope="row">Total sessions in use</th>
|
||||
<td>2858
|
||||
</td></tr></table>'''
|
||||
LOGIN_FORM = ('<h1>Access Code Required</h1><form>'
|
||||
'<input type="hidden" name="nonce" value="0a1b2c3d4e5f60718293a4b5c6d7e8f9" />')
|
||||
LOGIN_NO_NONCE = '<h1>Access Code Required</h1><strong>Cookies must be enabled</strong>'
|
||||
DIAG_FORM = ('<form method="post" action="/cgi-bin/diag.ha">'
|
||||
'<input type="hidden" name="nonce" value="ffee00112233445566778899aabbccdd" />')
|
||||
PING_DONE = '''PING 1.1.1.1 (1.1.1.1): 56 data bytes
|
||||
64 bytes from 1.1.1.1: seq=0 ttl=59 time=24.697 ms
|
||||
64 bytes from 1.1.1.1: seq=4 ttl=59 time=12.392 ms
|
||||
|
||||
--- 1.1.1.1 ping statistics ---
|
||||
5 packets transmitted, 5 packets received, 0% packet loss
|
||||
round-trip min/avg/max = 12.392/20.501/32.697 ms
|
||||
test done'''
|
||||
|
||||
|
||||
def _diag(window):
|
||||
return DIAG_FORM + f'<textarea id="progress" name="ProgressWindow" rows="15">{window}</textarea></form>'
|
||||
|
||||
|
||||
class TestModemPages:
|
||||
def test_login_hash_matches_the_page_script(self):
|
||||
# The page posts hex_md5(password + nonce)
|
||||
assert modem.login_hash('abc', '123') == 'e99a18c428cb38d5f260853678922e03'
|
||||
|
||||
def test_login_page_is_recognised(self):
|
||||
assert modem.is_login_page(LOGIN_FORM) and modem.is_login_page(LOGIN_NO_NONCE)
|
||||
assert not modem.is_login_page(NAT_PAGE)
|
||||
|
||||
def test_nat_summary(self):
|
||||
assert modem.parse_nat_summary(NAT_PAGE) == {'in_use': 2858, 'available': 32767}
|
||||
assert modem.parse_nat_summary('<table></table>') is None
|
||||
|
||||
def test_ping_output(self):
|
||||
assert modem.parse_ping_output(PING_DONE) == {
|
||||
'sent': 5, 'received': 5, 'loss_pct': 0, 'min_ms': 12.392, 'avg_ms': 20.501, 'max_ms': 32.697}
|
||||
|
||||
def test_ping_still_running_is_none(self):
|
||||
assert modem.parse_ping_output('PING 1.1.1.1 (1.1.1.1): 56 data bytes\n64 bytes from') is None
|
||||
|
||||
def test_ping_all_lost(self):
|
||||
r = modem.parse_ping_output('5 packets transmitted, 0 packets received, 100% packet loss')
|
||||
assert r['loss_pct'] == 100 and r['avg_ms'] is None
|
||||
|
||||
@pytest.mark.parametrize('gateway,ours,expected', [
|
||||
({'avg_ms': 900.0, 'loss_pct': 0}, 1000, 'upstream'),
|
||||
({'avg_ms': 20.0, 'loss_pct': 40}, 1000, 'upstream'),
|
||||
({'avg_ms': None, 'loss_pct': 100}, None, 'upstream'),
|
||||
({'avg_ms': 20.0, 'loss_pct': 0}, 1000, 'gateway'),
|
||||
({'avg_ms': 20.0, 'loss_pct': 0}, 30, 'clear'),
|
||||
({'avg_ms': 20.0, 'loss_pct': 0}, None, 'clear'),
|
||||
])
|
||||
def test_verdict(self, gateway, ours, expected):
|
||||
assert modem.ping_verdict(gateway, ours) == expected
|
||||
|
||||
|
||||
class FakeGateway:
|
||||
"""Answers ModemAdmin's requests like the BGW320 does."""
|
||||
|
||||
def __init__(self, code='secret', ping_pages=None):
|
||||
self.code, self.logged_in, self.calls = code, False, []
|
||||
self.cookie = self.pinged = False
|
||||
self.ping_pages = list(ping_pages or [_diag(PING_DONE)])
|
||||
|
||||
def __call__(self, method, url, **kw):
|
||||
path = url.split('192.0.2.254', 1)[1]
|
||||
self.calls.append((method, path))
|
||||
if path == '/cgi-bin/login.ha' and method == 'GET':
|
||||
first, self.cookie = not self.cookie, True
|
||||
return MagicMock(status_code=200, text=LOGIN_NO_NONCE if first else LOGIN_FORM)
|
||||
if path == '/cgi-bin/login.ha':
|
||||
ok = kw['data']['hashpassword'] == modem.login_hash(self.code, '0a1b2c3d4e5f60718293a4b5c6d7e8f9')
|
||||
assert set(kw['data']['password']) == {'*'} # the code itself is never sent
|
||||
self.logged_in = ok
|
||||
return MagicMock(status_code=302 if ok else 200, text='' if ok else LOGIN_FORM)
|
||||
if not self.logged_in:
|
||||
return MagicMock(status_code=200, text=LOGIN_FORM)
|
||||
if path == '/cgi-bin/nattable.ha':
|
||||
return MagicMock(status_code=200, text=NAT_PAGE)
|
||||
if path == '/cgi-bin/diag.ha' and method == 'POST':
|
||||
assert kw['data']['WebAddress'] == '1.1.1.1' and kw['data']['Ping'] == 'Ping'
|
||||
self.pinged = True
|
||||
return MagicMock(status_code=200, text=_diag(''))
|
||||
if path == '/cgi-bin/diag.ha':
|
||||
# Before the test the window is empty; after it, each refresh shows more output
|
||||
page = self.ping_pages.pop(0) if self.pinged else _diag('')
|
||||
return MagicMock(status_code=200, text=page)
|
||||
raise AssertionError(path)
|
||||
|
||||
|
||||
class TestModemAdmin:
|
||||
def _admin(self, gw, now=None):
|
||||
clock = MagicMock(return_value=now or 1000.0)
|
||||
a = modem.ModemAdmin({'url': 'http://192.0.2.254', 'access_code': 'secret'},
|
||||
clock=clock, sleep=lambda s: None)
|
||||
a.session.request = MagicMock(side_effect=gw)
|
||||
return a, clock
|
||||
|
||||
def test_logs_in_when_needed_and_reads_nat(self):
|
||||
gw = FakeGateway()
|
||||
a, _ = self._admin(gw)
|
||||
assert a.nat_sessions() == {'in_use': 2858, 'available': 32767}
|
||||
assert ('POST', '/cgi-bin/login.ha') in gw.calls
|
||||
gw.calls.clear()
|
||||
a.nat_sessions() # session still good: no second login
|
||||
assert ('POST', '/cgi-bin/login.ha') not in gw.calls
|
||||
|
||||
def test_expired_session_logs_in_again(self):
|
||||
gw = FakeGateway()
|
||||
a, _ = self._admin(gw)
|
||||
a.nat_sessions()
|
||||
gw.logged_in = False
|
||||
assert a.nat_sessions()['in_use'] == 2858
|
||||
|
||||
def test_refused_code_backs_off(self):
|
||||
gw = FakeGateway(code='other')
|
||||
a, clock = self._admin(gw)
|
||||
with pytest.raises(modem.ModemLoginFailed):
|
||||
a.nat_sessions()
|
||||
gw.calls.clear()
|
||||
clock.return_value = 1000.0 + modem.LOGIN_BACKOFF_S - 1
|
||||
with pytest.raises(modem.ModemLoginFailed):
|
||||
a.nat_sessions()
|
||||
assert ('POST', '/cgi-bin/login.ha') not in gw.calls # didn't even try
|
||||
|
||||
def test_ping_from_the_gateway_waits_for_the_result(self):
|
||||
gw = FakeGateway(ping_pages=[_diag('PING 1.1.1.1 (1.1.1.1): 56 data bytes'), _diag(PING_DONE)])
|
||||
a, _ = self._admin(gw)
|
||||
r = a.ping('1.1.1.1')
|
||||
assert r['avg_ms'] == 20.501 and r['target'] == '1.1.1.1'
|
||||
|
||||
def test_ping_that_never_finishes(self):
|
||||
gw = FakeGateway(ping_pages=[_diag('PING ...')] * 50)
|
||||
a, clock = self._admin(gw)
|
||||
clock.side_effect = [1000.0] + [2000.0] * 100
|
||||
with pytest.raises(modem.ModemDataInvalid, match='did not finish'):
|
||||
a.ping('1.1.1.1')
|
||||
|
||||
def test_unreachable(self):
|
||||
a, _ = self._admin(FakeGateway())
|
||||
a.session.request = MagicMock(side_effect=requests.ConnectionError('boom'))
|
||||
with pytest.raises(modem.ModemUnreachable):
|
||||
a.nat_sessions()
|
||||
@@ -1048,3 +1048,60 @@ class TestWanQualityCollection:
|
||||
c = monitor.UnifiClient({'controller': 'https://udm.test', 'api_key': 'k'})
|
||||
c.session.get = MagicMock(side_effect=RuntimeError('boom'))
|
||||
assert c.get_wan_dashboard() is None
|
||||
|
||||
|
||||
class TestModemChecks:
|
||||
NOW = 1_000_000.0
|
||||
|
||||
def _run(self, mon, db, checks=None, quality=None, admin=None):
|
||||
state = {'modem_checks': json.dumps(checks) if checks is not None else None,
|
||||
'wan_quality': json.dumps(quality) if quality is not None else None}
|
||||
db.get_state.side_effect = lambda k, d=None: state.get(k, d)
|
||||
db.set_state.reset_mock()
|
||||
mon.modem_admin = admin or MagicMock()
|
||||
if admin is None:
|
||||
mon.modem_admin.nat_sessions.return_value = {'in_use': 2858, 'available': 32767}
|
||||
mon.modem_admin.ping.return_value = {'sent': 5, 'received': 5, 'loss_pct': 0, 'min_ms': 12.0,
|
||||
'avg_ms': 20.0, 'max_ms': 32.0, 'target': '1.1.1.1'}
|
||||
with patch.object(monitor.time, 'time', return_value=self.NOW):
|
||||
mon._collect_modem_checks()
|
||||
saved = [c.args[1] for c in db.set_state.call_args_list if c.args[0] == 'modem_checks']
|
||||
return saved[-1] if saved else None
|
||||
|
||||
def _slow(self):
|
||||
return {'periods': [{'ts': self.NOW - 300, 'state': 'latency'}], 'latencies': {'cloudflare': 1043}}
|
||||
|
||||
def test_no_access_code_does_nothing(self, mon, db):
|
||||
mon.modem_admin = None
|
||||
mon._collect_modem_checks()
|
||||
assert not db.set_state.called
|
||||
|
||||
def test_nat_every_half_hour(self, mon, db):
|
||||
out = self._run(mon, db)
|
||||
assert out['nat'] == {'in_use': 2858, 'available': 32767, 'at': self.NOW}
|
||||
assert 'ping' not in out # the internet is fine, so no ping
|
||||
assert self._run(mon, db, checks={'nat': {'at': self.NOW - 60}}) is None
|
||||
|
||||
def test_slow_internet_pings_from_the_modem(self, mon, db):
|
||||
out = self._run(mon, db, checks={'nat': {'at': self.NOW - 60}}, quality=self._slow())
|
||||
assert out['ping']['avg_ms'] == 20.0 and out['ping']['ours_ms'] == 1043
|
||||
assert out['ping']['verdict'] == 'gateway'
|
||||
mon.modem_admin.ping.assert_called_once_with('1.1.1.1')
|
||||
|
||||
def test_ping_at_most_every_ten_minutes(self, mon, db):
|
||||
checks = {'nat': {'at': self.NOW - 60}, 'ping': {'at': self.NOW - 300}}
|
||||
assert self._run(mon, db, checks=checks, quality=self._slow()) is None
|
||||
|
||||
def test_failure_is_recorded_and_paused(self, mon, db):
|
||||
admin = MagicMock()
|
||||
admin.nat_sessions.side_effect = modem_api.ModemLoginFailed('refused')
|
||||
out = self._run(mon, db, admin=admin)
|
||||
assert out['error']['login'] is True and out['error']['at'] == self.NOW
|
||||
admin.nat_sessions.reset_mock()
|
||||
db.set_state.reset_mock()
|
||||
assert self._run(mon, db, checks=out, admin=admin) is None
|
||||
assert not admin.nat_sessions.called
|
||||
|
||||
def test_success_clears_the_error(self, mon, db):
|
||||
out = self._run(mon, db, checks={'error': {'message': 'x', 'at': self.NOW - 3600, 'login': True}})
|
||||
assert out['error'] is None
|
||||
+1
-1
@@ -62,7 +62,7 @@ def link(text: str, url: str) -> str:
|
||||
|
||||
|
||||
def port_text(switch: str, port_name: str, port: dict) -> str:
|
||||
"""'8SFP-AGG port 7 "Large1": up at 10 Gbps, DAC'"""
|
||||
"""'8SFP-AGG port 7 "compute-storage-02": up at 10 Gbps, DAC'"""
|
||||
idx = port.get('port_idx')
|
||||
text = f'{switch} port {idx}' if idx is not None else f'{switch} {port_name}'
|
||||
if port_name and idx is not None and port_name != f'Port {idx}':
|
||||
|
||||
+2
-2
@@ -25,7 +25,7 @@ _TOKEN_RE = re.compile(r'[a-z0-9]+')
|
||||
|
||||
|
||||
def _norm(name: str) -> str:
|
||||
"""A name with case, spaces, dashes and underscores ignored: "Large1" == "large-1"."""
|
||||
"""A name with case, spaces, dashes and underscores ignored: "Storage-01" == "storage_01"."""
|
||||
return ''.join(_TOKEN_RE.findall((name or '').lower()))
|
||||
|
||||
|
||||
@@ -114,7 +114,7 @@ def name_mismatch(port_name: str, neighbour: str, own: str, known: Dict[str, str
|
||||
"""A port named after a different device than the one plugged into it.
|
||||
|
||||
known: {normalised name: display name} of every device. Runs of up to four words of
|
||||
the port name are compared with them, so "Large1 Management" names large1 and
|
||||
the port name are compared with them, so "Storage-01 Management" names storage-01 and
|
||||
"Downlink to 24RJ45-POE" names 24RJ45-POE. Ports named only "Port 4" never match.
|
||||
"""
|
||||
if not port_name or _is_generic_port_name(port_name):
|
||||
|
||||
Reference in new issue
Block a user