WAN tickets: one per incident, notice UniFi updates, maintenance button
Lint / JS (eslint) (push) Successful in 42s
Lint / Python (flake8) (push) Successful in 1m35s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 1m3s
Test / Python Tests (pytest) (push) Successful in 1m5s
Lint / Python (flake8) (pull_request) Successful in 58s
Lint / JS (eslint) (pull_request) Successful in 8s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 1m1s
Test / Python Tests (pytest) (pull_request) Successful in 1m10s

A UniFi Network update on 2026-11-12 dropped the UDM's WAN, Smart Power
power-cycled the modem, and Gandalf opened two tickets for the one event.

- Modem and WAN reports from one check cycle open a single ticket, led by
  the most urgent, with the rest listed under "Also seen in this incident".
  A report while the modem has an open alert, or one that ended under 30
  minutes ago, becomes a comment on that ticket instead.
- UniFi update log entries (NETWORK_UPDATED, FIRMWARE_UPDATED_V2 and the
  like) add a note to modem/WAN tickets from 20 minutes before the update
  to 15 after, and hold new modem/WAN tickets for 15 minutes.
- A Maintenance button for admins starts a global suppression for 30 min
  to 4 hours, with a banner on every page and End now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-12 12:53:22 -05:00
co-authored by Claude Opus 5.5
parent 6b03fe7b44
commit 379e3be365
11 changed files with 541 additions and 28 deletions
+25
View File
@@ -242,6 +242,31 @@ LOG_WAN_RESTORED = 'NETWORK_WAN_RESTORED_2'
LOG_WAN_UNSTABLE = 'NETWORK_WAN_FAILED_MULTIPLE_TIMES_2'
LOG_MODEM_RESTARTED = 'SMART_POWER_MODEM_RESTARTED'
WAN_LOG_KEYS = (LOG_WAN_FAILED, LOG_WAN_RESTORED, LOG_WAN_UNSTABLE, LOG_MODEM_RESTARTED)
# An app or firmware update finishing. The UDM often drops its WAN while the Network app
# updates, and Smart Power may then power-cycle a modem that was fine.
LOG_NETWORK_UPDATED = 'NETWORK_UPDATED'
LOG_FIRMWARE_UPDATED = 'FIRMWARE_UPDATED_V2'
_UPDATE_KEY_RE = re.compile(r'[A-Z_]+_(?:UPDATED|UPGRADED)(?:_V\d+)?')
def is_update_key(key: str) -> bool:
"""True for log keys that mean an update finished (other app names follow the same pattern)."""
return key in (LOG_NETWORK_UPDATED, LOG_FIRMWARE_UPDATED) or (
bool(_UPDATE_KEY_RE.fullmatch(key or '')) and 'AVAILABLE' not in key
and not key.startswith('MADE_CHANGES'))
def wanted_log_key(key: str) -> bool:
return key in WAN_LOG_KEYS or is_update_key(key)
def update_label(event: dict) -> str:
"""'UniFi Network 11.0.97 to 11.0.99' from an update log event."""
what = event.get('app') or event.get('device') or 'UniFi'
before, after = event.get('previous') or '', event.get('version') or ''
if before and after:
return f'{what} {before} to {after}'
return f'{what} {after}'.strip()
_HISTORY_LIMIT = 50
# The RPDU's log entry and the gateway's calculated boot time can differ by this much.