Compare commits

..
3 Commits
Author SHA1 Message Date
jared f097e4882c Merge pull request 'Modem: read uptime again, firewall drop counts, AT&T speed tests' (#52) from modem-extras into main
Lint / Python (flake8) (push) Waiting to run
Lint / JS (eslint) (push) Waiting to run
Security / Python Security (bandit) (push) Waiting to run
Test / Python Tests (pytest) (push) Waiting to run
2026-11-19 18:52:08 -05:00
jaredandClaude Opus 5.5 0370358363 Modem: read uptime again, firewall drop counts, AT&T speed tests
Lint / JS (eslint) (pull_request) Waiting to run
Security / Python Security (bandit) (pull_request) Waiting to run
Test / Python Tests (pytest) (pull_request) Waiting to run
Lint / JS (eslint) (push) Successful in 24s
Lint / Python (flake8) (push) Successful in 1m2s
Test / Python Tests (pytest) (push) In progress
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 1m26s
Lint / Python (flake8) (pull_request) In progress
- Uptime: firmware 6.40 shows "Time Since Last Reboot" as D:HH:MM:SS,
  which the parser (expecting seconds) read as nothing, so uptime was
  blank and restart detection never fired. Both formats are read now.
- Firewall drops (logged in, every 5 minutes): the Logs page counted
  per hour for a day, split into incoming connections over the
  firmware's fixed flood limit (50 per second) and invalid packets. A
  lower bound, since the log keeps one entry per 2 s. Panel only.
- The modem's own speed test history (logged in, every 30 minutes):
  latest down and up results on the panel.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 18:50:37 -05:00
jared 2a9111d8e8 Merge pull request 'Modem access code: NAT sessions, and a ping from the modem when the internet is slow' (#51) from modem-login into main
Lint / JS (eslint) (push) Successful in 13s
Lint / Python (flake8) (push) Successful in 1m1s
Lint / Notify on failure (push) Skipped
Security / Python Security (bandit) (push) Successful in 57s
Lint / Deploy (push) Successful in 6s
Test / Python Tests (pytest) (push) Successful in 54s
2026-11-19 18:38:54 -05:00
6 changed files with 229 additions and 9 deletions

No files matched your search

+12 -2
View File
@@ -498,6 +498,10 @@ Those limits are wide (warning at -27.9 dBm on a link that normally sits near -1
compares each reading to the link's own 7-day median to catch a damaged connector earlier. It stays quiet
until `rx_drift_min_samples` readings exist (about a day at a 5 minute poll).
Uptime comes from the System Information page without a login. Firmware 6.40 shows it as D:HH:MM:SS
(older firmware sent seconds; both are read). Uptime going backwards between polls records a restart, and
the panel shows it (*restarted ... by ...*).
Outages can be over between polls: two real ones lasted about two minutes against a 5 minute poll.
So WAN outages and modem restarts also come from the UniFi system log, and a drop in the modem's uptime
records a restart even if nothing logged it. A single short outage only appears in the panel's history.
@@ -512,7 +516,7 @@ the last hour. These are dashboard lines, not tickets, and they go once the hour
than 30 minutes is ignored.
**With `modem.access_code` set**, the monitor also logs in to the gateway (`ModemAdmin` in `modem.py`)
for two read-only checks, stored in `monitor_state.modem_checks`:
for read-only checks, stored in `monitor_state.modem_checks`:
- **NAT sessions** every 30 minutes, from the NAT Table page. Needs attention warns when 80% of the
gateway's sessions (32,767 on a BGW320) are in use, since new connections stall once it is full.
- **A ping from the gateway itself** to 1.1.1.1, from the Troubleshoot page, whenever UniFi flags the
@@ -521,6 +525,12 @@ for two read-only checks, stored in `monitor_state.modem_checks`:
ping is slow too, so the provider), *in the modem* (its ping is fine but ours through it is slow), or
*fine by then*. The result goes in the detail of the Internet line in Needs attention and on the
WAN / Fiber panel.
- **Firewall drops** every 5 minutes, from the Logs page, counted per hour for a day. Most are *FLOOD
limit 50pps*: AT&T's firmware caps new incoming connections at 50 per second (visitors to hosted
services, peers), with no setting to change it. The log records at most one drop every 2 seconds and
keeps about ten minutes, so the counts are a lower bound. Shown on the panel only.
- **The gateway's own speed tests** (AT&T runs them from the modem), read with the NAT check; the latest
down and up results are shown on the panel as a reference for what the line itself delivers.
The login posts md5(access code + nonce), as the gateway's own page does, so the code itself is never sent.
The session is kept between checks and renewed when it expires. A refused code is reported in Needs
@@ -937,7 +947,7 @@ Take LED colors from the model's quick start guide. See the comment at the top o
`recovery_notified` once the recovery note is on its ticket (column added automatically at monitor startup)
- `suppression_rules`: `active=FALSE` when removed; `expires_at` checked at query time
- `monitor_state`: key/value store; `interface_baseline`, `link_stats`, and `modem_stats` are JSON blobs
- `monitor_state` also holds `wan_history` (WAN outages, modem restarts), `wan_quality` (UniFi's 5 minute WAN health timeline), `modem_checks` (NAT sessions and the latest ping from the gateway) and `unifi_log_cursor` (how far the UniFi log has been read)
- `monitor_state` also holds `wan_history` (WAN outages, modem restarts), `wan_quality` (UniFi's 5 minute WAN health timeline), `modem_checks` (NAT sessions, the latest ping from the gateway, hourly firewall drops and its speed test results) and `unifi_log_cursor` (how far the UniFi log has been read)
- Database sessions run in UTC (`db.py`), whatever the server's time zone
- `modem_samples`: optical readings per poll, kept 90 days; created automatically at monitor startup
- `cutsheet_items`, `cutsheet_cables`, `cutsheet_port_labels`, `cutsheet_history`: cutsheet records and
+59 -3
View File
@@ -5,13 +5,15 @@ this scrapes them over HTTP. Only GET requests to the fixed status pages below
are ever made: nothing here can change gateway settings.
With an access code configured, ModemAdmin (at the end) also logs in to read the
NAT session counts and to run the Troubleshoot page's ping from the gateway.
NAT session counts, the firewall log and speed test history, and to run the
Troubleshoot page's ping from the gateway.
Those are a login, page reads and a ping test: still no setting is changed.
Units come from the gateway's own help text, checked against live readings:
Rx/Tx power tenths of a dBm (-169 -> -16.9 dBm)
Temperature degrees C
Uptime seconds (despite the help text claiming D:H:M:S)
Uptime D:HH:MM:SS on firmware 6.40 (older firmware sent plain seconds;
both are accepted)
Deliberately not used, because the page disagrees with itself:
Vcc / Tx Bias readings values do not match the documented scales, so only
@@ -142,6 +144,18 @@ def _to_int(raw: Optional[str]) -> Optional[int]:
return None
def parse_uptime(raw: Optional[str]) -> Optional[int]:
"""Seconds since the gateway restarted, from 'D:HH:MM:SS' (e.g. 03:06:29:34) or plain seconds."""
raw = (raw or '').strip()
if re.fullmatch(r'\d+', raw):
return int(raw)
m = re.fullmatch(r'(\d+):(\d{1,2}):(\d{1,2}):(\d{1,2})', raw)
if not m:
return None
d, h, mi, sec = (int(x) for x in m.groups())
return ((d * 24 + h) * 60 + mi) * 60 + sec
def wan_problems(data: dict) -> List[str]:
"""Return human-readable reasons the WAN is down; empty when it looks healthy."""
problems = []
@@ -170,7 +184,7 @@ def build_status(fiber_page: str, broadband_page: str, sysinfo_page: str = '') -
data = {
'model': sysinfo.get('Model Number', ''),
'firmware': sysinfo.get('Software Version', ''),
'uptime_s': _to_int(sysinfo.get('Time Since Last Reboot')),
'uptime_s': parse_uptime(sysinfo.get('Time Since Last Reboot')),
'broadband_state': broadband.get('Broadband Connection', ''),
'source': broadband.get('Broadband Connection Source', ''),
'wan_ip': broadband.get('Broadband IPv4 Address', ''),
@@ -376,6 +390,12 @@ def quality_counts(periods: List[dict], since: float) -> Dict[str, int]:
_LOGIN_PAGE = '/cgi-bin/login.ha'
_NAT_PAGE = '/cgi-bin/nattable.ha'
_DIAG_PAGE = '/cgi-bin/diag.ha'
_LOGS_PAGE = '/cgi-bin/logs.ha'
_SPEED_PAGE = '/cgi-bin/speed.ha'
# Firewall log row: number, ISO time (gateway's local clock), source, destination, protocol, reason
_FW_ROW_RE = re.compile(r'<tr[^>]*>\s*<td[^>]*>\s*\d+\s*</td>\s*<td>\s*([^<]+?)\s*</td>\s*<td>\s*([^<]*?)\s*</td>'
r'\s*<td>\s*([^<]*?)\s*</td>\s*<td>\s*([^<]*?)\s*</td>\s*<td>\s*([^<]*?)\s*</td>\s*</tr>', re.I)
_CELL_RE = re.compile(r'<td\b[^>]*>(.*?)</td>', re.I | re.S)
_NONCE_RE = re.compile(r'name="nonce"\s+value="([0-9a-f]+)"', re.I)
_PROGRESS_RE = re.compile(r'<textarea[^>]*name="ProgressWindow"[^>]*>(.*?)</textarea>', re.I | re.S)
_PING_STATS_RE = re.compile(r'(\d+) packets transmitted, (\d+) (?:packets )?received')
@@ -422,6 +442,36 @@ def parse_ping_output(text: str) -> Optional[dict]:
'min_ms': lo, 'avg_ms': avg, 'max_ms': hi}
def parse_firewall_log(page: str) -> List[dict]:
"""Rows of the gateway's firewall log (Logs page): {'ts', 'src', 'dst', 'proto', 'reason'}.
'ts' is the gateway's local time as ISO text, so rows sort by it."""
rows = [dict(zip(('ts', 'src', 'dst', 'proto', 'reason'), m.groups())) for m in _FW_ROW_RE.finditer(page)]
return sorted((r for r in rows if re.match(r'\d{4}-\d\d-\d\dT', r['ts'])), key=lambda r: r['ts'])
def firewall_kind(reason: str) -> str:
"""'flood' (new connections over the gateway's fixed rate limit), 'invalid', or 'other'."""
reason = reason.lower()
return 'flood' if 'flood' in reason else 'invalid' if 'invalid' in reason else 'other'
def parse_speed_tests(page: str) -> List[dict]:
"""The gateway's own speed test history, newest first:
{'at' (as the page shows it), 'direction', 'mbps', 'latency_ms', 'ok'}."""
i = page.find('Test History')
out = []
for row in re.findall(r'<tr\b[^>]*>(.*?)</tr>', page[i:] if i >= 0 else '', re.I | re.S):
cells = [_TAG_RE.sub('', c).strip() for c in _CELL_RE.findall(row)]
if len(cells) < 6 or cells[1] not in ('upstream', 'downstream'):
continue
try:
out.append({'at': cells[0], 'direction': cells[1], 'mbps': round(float(cells[2]), 1),
'latency_ms': round(float(cells[4]), 1), 'ok': cells[5].lower() == 'success'})
except ValueError:
continue
return out
class ModemAdmin:
"""Logged-in session on the gateway, kept between checks and renewed when it expires.
@@ -480,6 +530,12 @@ class ModemAdmin:
raise ModemDataInvalid('NAT table page has no session counts')
return counts
def firewall_log(self) -> List[dict]:
return parse_firewall_log(self._page(_LOGS_PAGE))
def speed_tests(self) -> List[dict]:
return parse_speed_tests(self._page(_SPEED_PAGE))
def ping(self, target: str, wait_s: float = 30) -> dict:
"""Ping *target* from the gateway (its Troubleshoot page; 5 packets) and return the result."""
nonce = _NONCE_RE.search(self._page(_DIAG_PAGE))
+35 -2
View File
@@ -1973,7 +1973,9 @@ class NetworkMonitor:
if quality is not None:
db.set_state('wan_quality', quality)
_NAT_EVERY_S = 1800
_NAT_EVERY_S = 1800 # NAT sessions and the gateway's speed test history
_FIREWALL_EVERY_S = 300 # the firewall log only holds about ten minutes
_FIREWALL_HOURS = 24
_PING_EVERY_S = 600
_PING_TARGET = '1.1.1.1'
@@ -2001,14 +2003,18 @@ class NetworkMonitor:
slow = (bool(periods) and periods[-1].get('state') != 'ok') or (ours or 0) >= modem_api.SLOW_MS
due_nat = now - (checks.get('nat') or {}).get('at', 0) >= self._NAT_EVERY_S
due_fw = now - (checks.get('firewall') or {}).get('checked_at', 0) >= self._FIREWALL_EVERY_S
due_ping = slow and now - (checks.get('ping') or {}).get('at', 0) >= self._PING_EVERY_S
if not (due_nat or due_ping):
if not (due_nat or due_fw or due_ping):
return
if now - (checks.get('error') or {}).get('at', 0) < self._PING_EVERY_S:
return # failed recently: don't keep knocking on the gateway
try:
if due_nat:
checks['nat'] = dict(admin.nat_sessions(), at=now)
checks['speed'] = self._latest_speed_tests(admin.speed_tests(), now)
if due_fw:
checks['firewall'] = self._count_firewall_drops(checks.get('firewall'), admin.firewall_log(), now)
if due_ping:
result = admin.ping(self._PING_TARGET)
checks['ping'] = dict(result, at=now, ours_ms=ours,
@@ -2023,6 +2029,33 @@ class NetworkMonitor:
'login': isinstance(e, modem_api.ModemLoginFailed)}
db.set_state('modem_checks', checks)
@staticmethod
def _latest_speed_tests(tests: list, now: float) -> dict:
"""The newest downstream and upstream result from the gateway's own speed tests."""
out: dict = {'checked_at': now}
for t in tests: # newest first
key = 'down' if t['direction'] == 'downstream' else 'up'
out.setdefault(key, t)
return out
def _count_firewall_drops(self, previous: Optional[dict], rows: list, now: float) -> dict:
"""Add log rows newer than the last one seen to per-hour counts (gateway's local hour).
The log keeps a few hundred rows and records at most one drop every 2 seconds, so the
counts are a lower bound; 'gap' says rows aged out before this read could count them.
"""
fw = previous or {}
hours = dict(fw.get('hours') or {})
last = fw.get('last') or ''
new = [r for r in rows if r['ts'] > last]
for r in new:
bucket = hours.setdefault(r['ts'][:13], {'flood': 0, 'invalid': 0, 'other': 0})
bucket[modem_api.firewall_kind(r['reason'])] += 1
gap = bool(last and rows and rows[0]['ts'] > last)
keep = sorted(hours)[-self._FIREWALL_HOURS:]
return {'hours': {h: hours[h] for h in keep}, 'last': rows[-1]['ts'] if rows else last,
'checked_at': now, 'gap': gap}
def _collect_modem(self, suppressions: list) -> None:
"""Poll the modem, then store its status and raise any alerts."""
try:
+13
View File
@@ -396,6 +396,19 @@ function _modemChecks(c) {
out += _modemStat("Modem's own ping", value, p.verdict === 'clear' ? '' : 'lt-text-amber',
`to ${p.target}, ${ago(p.at)}${verdict ? `: ${verdict}` : ''}`);
}
const fw = c.firewall && c.firewall.hours;
if (fw && Object.keys(fw).length) {
const sum = k => Object.values(fw).reduce((n, h) => n + (h[k] || 0), 0);
const flood = sum('flood'), invalid = sum('invalid'), total = flood + invalid + sum('other');
out += _modemStat('Firewall drops', `${total.toLocaleString()} in ${Object.keys(fw).length} h`, '',
`${flood.toLocaleString()} incoming over the flood limit, ${invalid.toLocaleString()} invalid; ` +
'at least (the log keeps 1 per 2 s)');
}
const sp = c.speed;
if (sp && sp.down && sp.up) {
out += _modemStat('AT&T speed test', `↓ ${Math.round(sp.down.mbps)} ↑ ${Math.round(sp.up.mbps)} Mbps`, '',
`run by the modem ${sp.down.at}, ${Math.round(sp.down.latency_ms)} ms`);
}
if (c.error && c.error.login) {
out += _modemStat('Modem login', 'failed', 'lt-text-red', 'check modem.access_code');
}
+59
View File
@@ -455,3 +455,62 @@ class TestModemAdmin:
a.session.request = MagicMock(side_effect=requests.ConnectionError('boom'))
with pytest.raises(modem.ModemUnreachable):
a.nat_sessions()
FW_LOG = '''<table><tr><th>No.</th><th>Date/Time</th></tr>
<tr class="b"><td class="heading" scope="row">2</td><td>2026-11-19T18:32:04.984186</td><td>198.51.100.20</td>
<td>203.0.113.7</td><td>TCP</td><td>FLOOD limit 50pps burst 100</td></tr>
<tr class="a"><td class="heading" scope="row">1</td><td>2026-11-19T18:32:02.983868</td><td>203.0.113.7</td>
<td>198.51.100.30</td><td>TCP</td><td>Invalid IP Packet</td></tr>
</table>'''
SPEED_PAGE = '''<h2>Test History</h2><table><thead><th scope="col">Test Completion Time</th></thead><tbody>
<tr class="b"><td scope="row" class="heading" width="47%">
11/06/2026 08:46:05</td>
<td>upstream</td>
<td> 1242.810000</td>
<td> 28</td>
<td> 19.000000</td>
<td> Success</td></tr>
<tr class="a"><td scope="row" class="heading" width="47%">
11/06/2026 08:45:57</td>
<td>downstream</td>
<td> 1167.400000</td>
<td> 28</td>
<td> 28.000000</td>
<td> Success</td></tr>
</tbody></table>'''
class TestModemExtras:
@pytest.mark.parametrize('raw,expected', [
('03:06:29:34', ((3 * 24 + 6) * 60 + 29) * 60 + 34), # firmware 6.40
('28951', 28951), # older firmware
('0:00:05:00', 300),
('', None), (None, None), ('soon', None),
])
def test_uptime_formats(self, raw, expected):
assert modem.parse_uptime(raw) == expected
def test_status_reads_dhms_uptime(self):
sysinfo = SYSINFO.replace('<td class="col2">28951</td>', '<td class="col2">03:06:29:34</td>')
assert sysinfo != SYSINFO
assert modem.build_status(fiber_page(), broadband_page(), sysinfo)['uptime_s'] == 282574
def test_firewall_log_oldest_first(self):
rows = modem.parse_firewall_log(FW_LOG)
assert [r['ts'] for r in rows] == ['2026-11-19T18:32:02.983868', '2026-11-19T18:32:04.984186']
assert rows[1]['reason'] == 'FLOOD limit 50pps burst 100' and rows[1]['proto'] == 'TCP'
@pytest.mark.parametrize('reason,kind', [
('FLOOD limit 50pps burst 100', 'flood'), ('Invalid IP Packet', 'invalid'), ('Port scan', 'other')])
def test_firewall_kind(self, reason, kind):
assert modem.firewall_kind(reason) == kind
def test_speed_tests(self):
assert modem.parse_speed_tests(SPEED_PAGE) == [
{'at': '11/06/2026 08:46:05', 'direction': 'upstream', 'mbps': 1242.8, 'latency_ms': 19.0, 'ok': True},
{'at': '11/06/2026 08:45:57', 'direction': 'downstream', 'mbps': 1167.4, 'latency_ms': 28.0, 'ok': True},
]
def test_speed_tests_missing(self):
assert modem.parse_speed_tests('<html></html>') == []
+51 -2
View File
@@ -1080,7 +1080,8 @@ class TestModemChecks:
out = self._run(mon, db)
assert out['nat'] == {'in_use': 2858, 'available': 32767, 'at': self.NOW}
assert 'ping' not in out # the internet is fine, so no ping
assert self._run(mon, db, checks={'nat': {'at': self.NOW - 60}}) is None
recent = {'nat': {'at': self.NOW - 60}, 'firewall': {'checked_at': self.NOW - 60}}
assert self._run(mon, db, checks=recent) is None
def test_slow_internet_pings_from_the_modem(self, mon, db):
out = self._run(mon, db, checks={'nat': {'at': self.NOW - 60}}, quality=self._slow())
@@ -1089,7 +1090,8 @@ class TestModemChecks:
mon.modem_admin.ping.assert_called_once_with('1.1.1.1')
def test_ping_at_most_every_ten_minutes(self, mon, db):
checks = {'nat': {'at': self.NOW - 60}, 'ping': {'at': self.NOW - 300}}
checks = {'nat': {'at': self.NOW - 60}, 'ping': {'at': self.NOW - 300},
'firewall': {'checked_at': self.NOW - 60}}
assert self._run(mon, db, checks=checks, quality=self._slow()) is None
def test_failure_is_recorded_and_paused(self, mon, db):
@@ -1105,3 +1107,50 @@ class TestModemChecks:
def test_success_clears_the_error(self, mon, db):
out = self._run(mon, db, checks={'error': {'message': 'x', 'at': self.NOW - 3600, 'login': True}})
assert out['error'] is None
class TestModemExtrasCollection:
def _row(self, ts, reason='FLOOD limit 50pps burst 100'):
return {'ts': ts, 'src': 'a', 'dst': 'b', 'proto': 'TCP', 'reason': reason}
def test_counts_only_new_rows_per_hour(self, mon):
first = mon._count_firewall_drops(None, [self._row('2026-11-19T17:59:58.1'),
self._row('2026-11-19T18:00:01.2', 'Invalid IP Packet')], 100.0)
assert first['hours'] == {'2026-11-19T17': {'flood': 1, 'invalid': 0, 'other': 0},
'2026-11-19T18': {'flood': 0, 'invalid': 1, 'other': 0}}
assert first['last'] == '2026-11-19T18:00:01.2' and first['gap'] is False
# The next read overlaps the last one: only the new row counts
second = mon._count_firewall_drops(first, [self._row('2026-11-19T18:00:01.2', 'Invalid IP Packet'),
self._row('2026-11-19T18:00:03.0')], 400.0)
assert second['hours']['2026-11-19T18'] == {'flood': 1, 'invalid': 1, 'other': 0}
assert second['checked_at'] == 400.0
def test_gap_when_the_log_rolled_over(self, mon):
prev = {'hours': {}, 'last': '2026-11-19T18:00:00.0'}
out = mon._count_firewall_drops(prev, [self._row('2026-11-19T18:20:00.0')], 1.0)
assert out['gap'] is True
def test_keeps_a_day(self, mon):
prev = {'hours': {f'2026-11-18T{h:02d}': {'flood': 1, 'invalid': 0, 'other': 0} for h in range(24)},
'last': '2026-11-18T23:59:59.0'}
out = mon._count_firewall_drops(prev, [self._row('2026-11-19T00:00:01.0')], 1.0)
assert len(out['hours']) == 24 and '2026-11-18T00' not in out['hours']
def test_latest_speed_tests(self, mon):
tests = [{'direction': 'upstream', 'mbps': 1242.8, 'at': 'b'},
{'direction': 'downstream', 'mbps': 1167.4, 'at': 'b'},
{'direction': 'upstream', 'mbps': 1.0, 'at': 'a'}]
out = mon._latest_speed_tests(tests, 5.0)
assert out['up']['mbps'] == 1242.8 and out['down']['mbps'] == 1167.4 and out['checked_at'] == 5.0
def test_scheduled_with_the_other_checks(self, mon, db):
admin = MagicMock()
admin.nat_sessions.return_value = {'in_use': 1, 'available': 10}
admin.speed_tests.return_value = []
admin.firewall_log.return_value = [self._row('2026-11-19T18:00:00.0')]
mon.modem_admin = admin
db.get_state.side_effect = lambda k, d=None: None
mon._collect_modem_checks()
saved = [c.args[1] for c in db.set_state.call_args_list if c.args[0] == 'modem_checks'][-1]
assert saved['firewall']['hours'] == {'2026-11-19T18': {'flood': 1, 'invalid': 0, 'other': 0}}
assert 'speed' in saved and admin.firewall_log.called