Compare commits

...
Author SHA1 Message Date
jaredandClaude Opus 5.5 0921e82c54 Dashboard: flag stale alerts and tickets left open
Lint / JS (eslint) (push) Successful in 19s
Lint / Python (flake8) (push) Successful in 1m7s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 52s
Test / Python Tests (pytest) (push) Successful in 44s
Lint / JS (eslint) (pull_request) Successful in 7s
Lint / Python (flake8) (pull_request) Successful in 50s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 52s
Test / Python Tests (pytest) (pull_request) Successful in 1m35s
Gandalf only comments on tickets, so a ticket can stay open long after its
alert cleared (pbs's ticket sat open for a month), and an alert can stay
active for days once nobody is looking.

- The monitor reads Tinker's open ticket list (tickets_api.php, read scope)
  every 15 minutes and saves it. A failed read keeps the previous list.
- Active alerts get a stale badge after monitor.stale_alert_days (3), and a
  "ticket closed" badge when their ticket was closed while still active.
- A Needs Follow-up panel lists open tickets whose every alert cleared at
  least monitor.stale_ticket_hours (24) ago.
- A "TO FOLLOW UP" status chip counts both. If the open ticket list is over
  2 hours old, the ticket flags are left off rather than guessed.
- Replaced an — in the alert count notice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 15:57:58 -05:00
jared 7b80633088 Merge pull request 'WAN tickets: one per incident, notice UniFi updates, maintenance button' (#36) from wan-incidents into main
Lint / JS (eslint) (push) Successful in 14s
Lint / Python (flake8) (push) Successful in 1m37s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Successful in 16s
Security / Python Security (bandit) (push) Successful in 1m47s
Test / Python Tests (pytest) (push) Successful in 44s
2026-11-12 13:46:31 -05:00
jaredandClaude Opus 5.5 379e3be365 WAN tickets: one per incident, notice UniFi updates, maintenance button
Lint / JS (eslint) (push) Successful in 42s
Lint / Python (flake8) (push) Successful in 1m35s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 1m3s
Test / Python Tests (pytest) (push) Successful in 1m5s
Lint / Python (flake8) (pull_request) Successful in 58s
Lint / JS (eslint) (pull_request) Successful in 8s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 1m1s
Test / Python Tests (pytest) (pull_request) Successful in 1m10s
A UniFi Network update on 2026-11-12 dropped the UDM's WAN, Smart Power
power-cycled the modem, and Gandalf opened two tickets for the one event.

- Modem and WAN reports from one check cycle open a single ticket, led by
  the most urgent, with the rest listed under "Also seen in this incident".
  A report while the modem has an open alert, or one that ended under 30
  minutes ago, becomes a comment on that ticket instead.
- UniFi update log entries (NETWORK_UPDATED, FIRMWARE_UPDATED_V2 and the
  like) add a note to modem/WAN tickets from 20 minutes before the update
  to 15 after, and hold new modem/WAN tickets for 15 minutes.
- A Maintenance button for admins starts a global suppression for 30 min
  to 4 hours, with a banner on every page and End now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 12:53:22 -05:00
13 changed files with 1003 additions and 32 deletions

No files matched your search

+43 -1
View File
@@ -111,6 +111,9 @@ Ping-only (no node_exporter): **pbs** (10.10.10.3)
- WAN / Fiber panel: WAN and PON state, Rx/Tx power, optics temperature, throughput, errors, gateway uptime and last restart, 7-day Rx trend, recent WAN outages
- UniFi device table (switches, APs, gateway)
- Active alerts table with severity, target, consecutive failures, ticket link
- Follow-up flags (see Follow-up below): a ⏳ badge on alerts active for days, a "ticket closed" badge when the
ticket was closed but the alert is still active, and a Needs Follow-up panel of tickets left open after their
alert cleared. A "TO FOLLOW UP" chip in the status bar counts them all.
- Quick-suppress modal: apply timed or manual suppression from any alert row
- Auto-refreshes every 30 seconds via `/api/status` + `/api/network`
@@ -309,6 +312,12 @@ can view it; only the `admin` group can edit, and every change is logged with wh
- Suppression history (last 50)
- Available targets reference grid (all known hosts + interfaces)
**Maintenance**: admins get a **Maintenance** button in the header. It starts a global suppression for
30 min, 1, 2 or 4 hours with a reason (by default "Maintenance"). While it runs, every page shows a banner
saying when tickets resume, with **End now** for admins. Gandalf keeps checking and recording alerts, and
anything still failing when maintenance ends gets a ticket as usual. Use it before a UniFi update, a switch
swap or recabling.
---
## Alert Logic
@@ -375,6 +384,20 @@ A note that fails to post is retried every cycle for 24 hours. Posting needs a T
`read_write` scope; if Tinker answers 401 or 403, the monitor logs one error and stops trying until it is
restarted.
### Follow-up
Gandalf only comments on tickets, so a ticket can stay open long after its problem is fixed, and an alert can
sit active for days once everyone has stopped looking. The dashboard flags both:
- **Stale alert**: active for `monitor.stale_alert_days` (default 3) or longer. Shown as a ⏳ badge with its age.
- **Ticket closed**: the alert is still active but its ticket was closed in Tinker, so nobody is tracking it.
- **Ticket left open**: every alert on a ticket cleared at least `monitor.stale_ticket_hours` (default 24) ago,
but the ticket is still Open, Pending or In Progress. Listed in the Needs Follow-up panel.
The monitor reads the list of open tickets from Tinker's `api/tickets_api.php` every 15 minutes (one request,
read scope only) and saves it. If that list is over 2 hours old, because Tinker or the key stopped working,
the ticket flags are left off rather than guessed. Stale alerts are still flagged.
### WAN modem notes
The alarm and warning limits are read from the gateway on every poll, so they are never hard-coded here.
@@ -386,7 +409,23 @@ Outages can be over between polls: two real ones lasted about two minutes agains
So WAN outages and modem restarts also come from the UniFi system log, and a drop in the modem's uptime
records a restart even if nothing logged it. A single short outage only appears in the panel's history.
To silence modem alerts during maintenance, create a `host` suppression with the modem's `name`.
**One ticket per incident.** The modem and WAN reports from one check cycle open a single ticket, led by
the most urgent (a P2 "WAN modem unreachable" leads, and an RPDU power-cycle seen in the same cycle is listed
under "Also seen in this incident"). A report that comes while the modem has an open alert, or one that
ended under 30 minutes ago, becomes a comment on that alert's ticket instead of a new ticket. If the
comment can't be posted, the report gets its own ticket as before.
**UniFi updates.** UniFi logs `NETWORK_UPDATED` (Network app) and `FIRMWARE_UPDATED_V2` (device firmware)
once an update has finished. Updating the Network app can drop the UDM's WAN, and Smart Power may then
power-cycle a modem that was fine. When Gandalf reads one of these entries it:
- comments on modem and WAN tickets first seen from 20 minutes before the update to 15 minutes after,
naming the update (for example "UniFi Network 11.0.97 to 11.0.99") as the likely cause;
- opens no new modem or WAN tickets for 15 minutes after the update. A problem still there after that gets
its ticket as usual; a power-cycle report during the hold only goes into the event history.
To silence modem alerts during maintenance, create a `host` suppression with the modem's `name`, or use
**Maintenance** (below) to pause every ticket.
### Baseline Tracking
@@ -486,6 +525,7 @@ Shared by both processes. Located in the working directory (`/var/www/html/prod/
| `ticket_api.url` | Tinker Tickets `create_ticket_api.php` URL |
| `ticket_api.api_key` | Tinker Tickets bearer token. Needs `read_write` scope to add recovery notes |
| `ticket_api.comment_url` | Tinker comment endpoint (default: `api/ticket_comment_api.php` beside `ticket_api.url`) |
| `ticket_api.read_url` | Tinker ticket read endpoint, used to find tickets left open (default: `api/tickets_api.php` beside `ticket_api.url`) |
| `gandalf.public_url` | Gandalf's own address, for links in tickets (default: `https://gandalf.lotusguild.org`) |
| `rack.units`, `rack.name` | The single rack's height and name on `/rack` (defaults: 42, "Rack 1") |
| `racks` | Several racks: `[{"name": "...", "units": 42}, ...]`, in tab order; overrides `rack` |
@@ -503,6 +543,8 @@ Shared by both processes. Located in the working directory (`/var/www/html/prod/
| `monitor.unifi_failure_threshold` | Consecutive failures before ticketing a UniFi device as offline (default: same as `failure_threshold`). Set higher than `failure_threshold`: UniFi switches/APs can take several minutes to rejoin after a firmware-update reboot, so a low threshold cuts tickets for momentary maintenance reboots instead of real failures. |
| `monitor.cluster_threshold` | Hosts with failures to trigger cluster-wide P1 (default: 3) |
| `monitor.ping_hosts` | Hosts checked only by ping (no node_exporter) |
| `monitor.stale_alert_days` | Days an alert can stay active before the dashboard flags it as stale (default: 3) |
| `monitor.stale_ticket_hours` | Hours a ticket can stay open after all its alerts cleared before it is listed under Needs Follow-up (default: 24) |
| `cutsheet.mismatch_ticket_hours` | Hours a cutsheet mismatch must last before it opens a P4 ticket (default: 24; `0` never tickets). The mismatch shows on the dashboard and the switch's page straight away, so recabling never pages anyone. Suppressing the switch or its port also stops the ticket. |
| `modem.url` | AT&T gateway base URL. Omit the `modem` block to disable modem monitoring. Must be reachable from the monitor host. |
| `modem.name` | Name used for the modem's alerts, tickets, and suppressions (default: `att-modem`) |
+84
View File
@@ -60,6 +60,19 @@ def inject_config():
}
@app.context_processor
def inject_maintenance():
"""The active global suppression (maintenance), shown as a banner on every page."""
try:
active = [s for s in db.get_active_suppressions() if s.get('target_type') == 'all']
except Exception as e:
logger.error(f'Reading maintenance state failed: {e}')
active = []
# Of several, the one that ends last decides when tickets resume; no expiry beats any
active.sort(key=lambda s: (s.get('expires_at') is None, s.get('expires_at') or ''))
return {'maintenance': active[-1] if active else None}
# In-memory diagnostic job store { job_id: { status, result, created_at } }
_diag_jobs: dict = {}
_diag_lock = threading.Lock()
@@ -261,6 +274,72 @@ def _annotate_suppressions(events: list, suppressions: list) -> None:
ev['is_suppressed'] = db.check_suppressed(suppressions, 'host', name, detail)
# The monitor refreshes the open ticket list every 15 minutes; older than this and
# it has stopped working, so nothing is flagged rather than flagging on stale data.
_OPEN_TICKETS_MAX_AGE_S = 2 * 3600
def _stale_settings() -> tuple:
"""(days an alert can stay active before it counts as stale,
hours a ticket can stay open after its alert cleared)."""
mon = _config().get('monitor', {})
try:
days = float(mon.get('stale_alert_days', 3))
hours = int(mon.get('stale_ticket_hours', 24))
except (TypeError, ValueError):
days, hours = 3.0, 24
return days, hours
def _open_tickets() -> Optional[dict]:
"""{ticket_id: status} of open Tinker tickets, or None when the list is missing or out of date."""
try:
raw = db.get_state('open_tickets')
data = json.loads(raw) if raw else None
if not data or time.time() - float(data.get('checked_at', 0)) > _OPEN_TICKETS_MAX_AGE_S:
return None
return {str(k): v for k, v in (data.get('tickets') or {}).items()}
except Exception as e:
logger.error(f'Reading open ticket list failed: {e}')
return None
def _annotate_staleness(events: list, open_tickets: Optional[dict]) -> None:
"""Add stale_days (set when an alert has been active too long) and ticket_closed
(its ticket was closed in Tinker while the alert is still active) to each event."""
days, _ = _stale_settings()
now = datetime.now(timezone.utc)
for ev in events:
ev['stale_days'] = None
ev['ticket_closed'] = False
try:
first = datetime.fromisoformat(str(ev.get('first_seen', '')).replace('Z', '+00:00'))
if first.tzinfo is None:
first = first.replace(tzinfo=timezone.utc)
age = (now - first).total_seconds() / 86400
if age >= days:
ev['stale_days'] = int(age)
except ValueError:
pass
tid = ev.get('ticket_id')
if tid and open_tickets is not None and str(tid) not in open_tickets:
ev['ticket_closed'] = True
def _stale_view(events: list) -> list:
"""Flag stale active alerts in place; return tickets left open after their alerts cleared."""
open_tickets = _open_tickets()
_annotate_staleness(events, open_tickets)
if open_tickets is None:
return []
_, hours = _stale_settings()
try:
return db.get_cleared_tickets(list(open_tickets), older_than_hours=hours)
except Exception as e:
logger.error(f'Finding tickets left open failed: {e}')
return []
def _state_json(key: str) -> dict:
"""Return a monitor_state JSON blob as a dict, or {} if missing or unreadable."""
raw = db.get_state(key)
@@ -294,6 +373,7 @@ def index():
snapshot = {}
suppressions = db.get_active_suppressions()
_annotate_suppressions(events, suppressions)
stale_tickets = _stale_view(events)
recent_resolved = db.get_recent_resolved(hours=24, limit=10)
return render_template(
'index.html',
@@ -305,6 +385,8 @@ def index():
last_check=last_check,
suppressions=suppressions,
recent_resolved=recent_resolved,
stale_tickets=stale_tickets,
stale_alerts=sum(1 for e in events if e.get('stale_days') is not None or e.get('ticket_closed')),
daemon_ok=_daemon_ok(last_check),
)
@@ -406,11 +488,13 @@ def api_status():
active = db.get_active_events(limit=_PAGE_LIMIT)
suppressions = db.get_active_suppressions()
_annotate_suppressions(active, suppressions)
stale_tickets = _stale_view(active)
last_check = db.get_state('last_check', 'Never')
return jsonify({
'summary': db.get_status_summary(),
'last_check': last_check,
'events': active,
'stale_tickets': stale_tickets,
'total_active': db.count_active_events(),
'daemon_ok': _daemon_ok(last_check),
})
+73 -1
View File
@@ -4,7 +4,7 @@ import logging
import threading
from contextlib import contextmanager
from datetime import datetime, timedelta, timezone
from typing import Optional
from typing import List, Optional
import pymysql
import pymysql.cursors
@@ -198,6 +198,35 @@ def set_ticket_id(event_id: int, ticket_id: str) -> None:
)
def get_incident_ticket(target_name: str, within_s: int) -> Optional[str]:
"""The ticket of *target_name*'s newest event that is still open or ended within *within_s*."""
with get_conn() as conn:
with conn.cursor() as cur:
cur.execute(
"""SELECT ticket_id FROM network_events
WHERE target_name=%s AND ticket_id IS NOT NULL AND ticket_id <> ''
AND (resolved_at IS NULL OR resolved_at >= NOW() - INTERVAL %s SECOND)
ORDER BY first_seen DESC, id DESC LIMIT 1""",
(target_name, int(within_s)),
)
row = cur.fetchone()
return row['ticket_id'] if row else None
def get_ticket_ids_between(target_name: str, start: datetime, end: datetime) -> List[str]:
"""Tickets of *target_name*'s events first seen between *start* and *end* (UTC), oldest first."""
with get_conn() as conn:
with conn.cursor() as cur:
cur.execute(
"""SELECT ticket_id, MIN(first_seen) AS first FROM network_events
WHERE target_name=%s AND ticket_id IS NOT NULL AND ticket_id <> ''
AND first_seen BETWEEN %s AND %s
GROUP BY ticket_id ORDER BY first""",
(target_name, start, end),
)
return [r['ticket_id'] for r in cur.fetchall()]
def get_active_events(limit: int = 200, offset: int = 0) -> list:
with get_conn() as conn:
with conn.cursor() as cur:
@@ -218,6 +247,49 @@ def get_active_events(limit: int = 200, offset: int = 0) -> list:
return rows
def get_cleared_tickets(open_ids: List[str], older_than_hours: int = 24) -> list:
"""Of the given open tickets, those whose every alert cleared at least `older_than_hours` ago.
One row per ticket, describing its most recent alert, newest clearance first:
{ticket_id, event_type, target_name, target_detail, severity, first_seen, cleared_at, events}
"""
ids = sorted({str(i) for i in open_ids if i})
if not ids:
return []
placeholders = ','.join(['%s'] * len(ids))
with get_conn() as conn:
with conn.cursor() as cur:
cur.execute(
f"""SELECT ticket_id, event_type, target_name, target_detail, severity,
first_seen, resolved_at
FROM network_events WHERE ticket_id IN ({placeholders})
ORDER BY first_seen""", # nosec B608: placeholders are only %s markers
ids,
)
rows = cur.fetchall()
cur.execute('SELECT NOW() AS now')
now = cur.fetchone()['now']
by_ticket: dict = {}
for r in rows:
by_ticket.setdefault(str(r['ticket_id']), []).append(r)
out = []
for tid, events in by_ticket.items():
if any(e['resolved_at'] is None for e in events):
continue
cleared = max(e['resolved_at'] for e in events)
if (now - cleared).total_seconds() < older_than_hours * 3600:
continue
last = events[-1]
out.append({
'ticket_id': tid, 'event_type': last['event_type'], 'target_name': last['target_name'],
'target_detail': last['target_detail'], 'severity': last['severity'],
'first_seen': events[0]['first_seen'].isoformat() + 'Z',
'cleared_at': cleared.isoformat() + 'Z', 'events': len(events),
})
out.sort(key=lambda r: r['cleared_at'], reverse=True)
return out
def count_active_events() -> int:
"""Return count of all unresolved events (for pagination)."""
with get_conn() as conn:
+25
View File
@@ -242,6 +242,31 @@ LOG_WAN_RESTORED = 'NETWORK_WAN_RESTORED_2'
LOG_WAN_UNSTABLE = 'NETWORK_WAN_FAILED_MULTIPLE_TIMES_2'
LOG_MODEM_RESTARTED = 'SMART_POWER_MODEM_RESTARTED'
WAN_LOG_KEYS = (LOG_WAN_FAILED, LOG_WAN_RESTORED, LOG_WAN_UNSTABLE, LOG_MODEM_RESTARTED)
# An app or firmware update finishing. The UDM often drops its WAN while the Network app
# updates, and Smart Power may then power-cycle a modem that was fine.
LOG_NETWORK_UPDATED = 'NETWORK_UPDATED'
LOG_FIRMWARE_UPDATED = 'FIRMWARE_UPDATED_V2'
_UPDATE_KEY_RE = re.compile(r'[A-Z_]+_(?:UPDATED|UPGRADED)(?:_V\d+)?')
def is_update_key(key: str) -> bool:
"""True for log keys that mean an update finished (other app names follow the same pattern)."""
return key in (LOG_NETWORK_UPDATED, LOG_FIRMWARE_UPDATED) or (
bool(_UPDATE_KEY_RE.fullmatch(key or '')) and 'AVAILABLE' not in key
and not key.startswith('MADE_CHANGES'))
def wanted_log_key(key: str) -> bool:
return key in WAN_LOG_KEYS or is_update_key(key)
def update_label(event: dict) -> str:
"""'UniFi Network 11.0.97 to 11.0.99' from an update log event."""
what = event.get('app') or event.get('device') or 'UniFi'
before, after = event.get('previous') or '', event.get('version') or ''
if before and after:
return f'{what} {before} to {after}'
return f'{what} {after}'.strip()
_HISTORY_LIMIT = 50
# The RPDU's log entry and the gateway's calculated boot time can differ by this much.
+199 -25
View File
@@ -13,7 +13,8 @@ import re
import shlex
import statistics
import time
from datetime import datetime, timezone
from contextlib import contextmanager
from datetime import datetime, timedelta, timezone
from typing import Dict, List, Optional
from urllib.parse import urljoin, urlparse
@@ -192,8 +193,12 @@ class UnifiClient:
logger.error(f'UniFi client list error: {e}')
return []
def get_system_events(self, since_ms: int, keys: tuple) -> Optional[List[dict]]:
"""Return system log entries with the given keys, or None if the log is unavailable."""
def get_system_events(self, since_ms: int, keys) -> Optional[List[dict]]:
"""Return system log entries with the given keys, or None if the log is unavailable.
keys: a tuple of log keys, or a function that takes a key and says whether to keep it.
"""
wanted = keys if callable(keys) else keys.__contains__
url = f'{self.base_url}/proxy/network/v2/api/site/{self.site_id}/system-log/all'
found: List[dict] = []
try:
@@ -210,19 +215,29 @@ class UnifiClient:
if not items:
break
for e in items:
if e.get('key') in keys and e.get('timestamp', 0) >= since_ms:
if wanted(e.get('key') or '') and e.get('timestamp', 0) >= since_ms:
params = e.get('parameters') or {}
found.append({
'key': e['key'], 'ts': e['timestamp'] / 1000,
'device': (params.get('DEVICE') or {}).get('name', ''),
'wan': (params.get('WAN_ID') or {}).get('name', '')
or str((params.get('WAN_ID') or {}).get('id', '')),
**self._update_fields(params),
})
return found
except Exception as e:
logger.error(f'UniFi system log error: {e}')
return None
@staticmethod
def _update_fields(params: dict) -> dict:
"""The app and versions of an update entry; nothing for other entries."""
def text(key):
value = params.get(key)
return str(value.get('name') or value.get('id') or '') if isinstance(value, dict) else str(value or '')
fields = {'app': text('APPLICATION'), 'version': text('VERSION'), 'previous': text('PREVIOUS_VERSION')}
return {k: v for k, v in fields.items() if v}
def get_devices(self) -> Optional[List[dict]]:
"""Return list of UniFi devices, or None if the controller is unreachable."""
try:
@@ -393,6 +408,8 @@ class TicketClient:
self.comment_url = cfg.get('comment_url') or (
urljoin(self.url, 'api/ticket_comment_api.php') if self.url else '')
self.comments_denied = False
self.read_url = cfg.get('read_url') or (
urljoin(self.url, 'api/tickets_api.php') if self.url else '')
# Titles whose create matched a Closed ticket Tinker couldn't reopen: {title: (ticket_id, when)}
self._closed_matches: Dict[str, tuple] = {}
@@ -427,6 +444,41 @@ class TicketClient:
logger.error(f'Ticket comment failed for #{ticket_id}: {e}')
return False
# Statuses that mean someone still has a ticket on their list
OPEN_STATUSES = 'Open,Pending,In Progress'
_OPEN_TICKETS_MAX_PAGES = 20
def open_tickets(self) -> Optional[Dict[str, str]]:
"""{ticket_id: status} for every ticket not yet closed, or None if Tinker can't say.
Read only: this needs the key's read scope and never changes a ticket.
"""
if not self.api_key or not self.read_url:
return None
found: Dict[str, str] = {}
try:
for page in range(1, self._OPEN_TICKETS_MAX_PAGES + 1):
resp = requests.get(
self.read_url,
params={'status': self.OPEN_STATUSES, 'limit': 100, 'page': page},
headers={'Authorization': f'Bearer {self.api_key}'},
timeout=15,
)
resp.raise_for_status()
data = resp.json()
if not data.get('success'):
logger.warning(f'Unexpected ticket list response: {str(data)[:200]}')
return None
for t in data.get('tickets') or []:
found[str(t.get('ticket_id'))] = t.get('status') or ''
if page >= int(data.get('pages') or 1):
return found
logger.warning('Open ticket list is longer than expected; using the first pages only')
return found
except Exception as e:
logger.error(f'Reading open tickets failed: {e}')
return None
# How long to wait before asking again after a create matched an unreopenable Closed ticket.
# Each ask adds a "recurred" comment to that ticket, so once a poll would pile them up.
CLOSED_MATCH_RETRY_S = 24 * 3600
@@ -965,12 +1017,25 @@ CLUSTER_NAME = 'proxmox-cluster'
DEFAULT_PUBLIC_URL = 'https://gandalf.lotusguild.org'
# Reports of something already over: the ticket is the whole story, no recovery note
ONE_SHOT_EVENTS = ('modem_power_cycled', 'wan_unstable')
# Modem and WAN reports this close together are one incident, so they share one ticket
WAN_INCIDENT_WINDOW_S = 1800
# UniFi logs an update when it finishes, after any WAN trouble it caused. So tickets opened
# this long before the entry get a note, and new modem/WAN tickets wait this long after it.
UPDATE_LOOKBACK_S = 1200
UPDATE_HOLD_S = 900
UNIFI_UPDATE_STATE = 'unifi_update'
# Ports a host or device was last seen on, kept so a ticket for a dead host
# (whose link, and so its LLDP entry, is gone) can still say where it plugs in
KNOWN_UPLINKS_STATE = 'known_uplinks'
# Which tickets are still open in Tinker, so the dashboard can flag ones left open
# after their alert cleared. Refreshed this often; one list request per refresh.
OPEN_TICKETS_STATE = 'open_tickets'
OPEN_TICKETS_SYNC_S = 900
class NetworkMonitor:
_open_tickets_synced = 0.0
def __init__(self):
with open('config.json') as f:
self.cfg = json.load(f)
@@ -1223,6 +1288,20 @@ class NetworkMonitor:
elif self.tickets.comments_denied:
break
def _sync_open_tickets(self) -> None:
"""Save which tickets are still open in Tinker, at most every OPEN_TICKETS_SYNC_S.
On a failed read the previous list stays, with its old checked_at, so the
dashboard can tell it is out of date rather than treat every ticket as closed.
"""
now = time.time()
if now - self._open_tickets_synced < OPEN_TICKETS_SYNC_S:
return
self._open_tickets_synced = now
found = self.tickets.open_tickets()
if found is not None:
db.set_state(OPEN_TICKETS_STATE, {'checked_at': now, 'tickets': found})
# ------------------------------------------------------------------
# Interface monitoring (Prometheus)
# ------------------------------------------------------------------
@@ -1543,14 +1622,95 @@ class NetworkMonitor:
event_type, severity, 'modem', name, detail, f'{description} ({_now_utc()})',
)
if not sup and consec >= self.modem_fail_thresh and not ticket_id:
self._open_ticket(
event_id, event_type, name, detail,
title=f'[{name}][auto][production][issue][network][cluster-wide] {heading}',
heading='WAN Modem Alert',
facts=[('Modem', f'{name} ({self.modem.base_url})'), ('Detected', _now_utc()),
('Consecutive check failures', str(consec))],
summary=description, checks=[advice], priority=priority,
)
self._queue_wan_ticket({
'event_id': event_id, 'event_type': event_type, 'detail': detail,
'title': f'[{name}][auto][production][issue][network][cluster-wide] {heading}',
'name': heading, 'heading': 'WAN Modem Alert',
'facts': [('Modem', f'{name} ({self.modem.base_url})'), ('Detected', _now_utc()),
('Consecutive check failures', str(consec))],
'summary': description, 'checks': [advice], 'priority': priority, 'recovers': True,
})
# ------------------------------------------------------------------
# Modem and WAN tickets: one per incident, held around UniFi updates
# ------------------------------------------------------------------
@contextmanager
def _wan_batch(self):
"""Collect the cycle's modem and WAN tickets so related ones are opened together."""
self._wan_pending = []
try:
yield
finally:
pending, self._wan_pending = self._wan_pending, None
if pending:
self._open_wan_tickets(pending)
def _queue_wan_ticket(self, item: dict) -> None:
held = self._update_hold()
if held:
logger.info(f'Not opening a ticket for {item["event_type"]} yet: {held} just finished')
return
if getattr(self, '_wan_pending', None) is not None:
self._wan_pending.append(item)
else:
self._open_wan_tickets([item])
def _update_hold(self) -> Optional[str]:
"""The update that modem and WAN tickets are waiting out, if any."""
raw = db.get_state(UNIFI_UPDATE_STATE)
if not raw:
return None
try:
update = json.loads(raw)
except ValueError:
return None
if update.get('hold_until', 0) <= time.time():
return None
return update.get('label') or 'a UniFi update'
def _open_wan_tickets(self, items: List[dict]) -> None:
"""Comment on the incident's ticket if one is recent, else open one ticket for all of *items*."""
# Most urgent first; at equal priority an ongoing problem leads over a past event
items = sorted(items, key=lambda i: (int(i['priority']), not i['recovers']))
incident = db.get_incident_ticket(self.modem_name, WAN_INCIDENT_WINDOW_S)
if incident:
left = []
for item in items:
text = ticket_text.incident_comment(item['name'], item['summary'], item['checks'], _now_utc())
if self.tickets.comment(incident, text):
db.set_ticket_id(item['event_id'], incident)
else:
left.append(item) # comments failing must not lose the report
items = left
if not items:
return
first, others = items[0], items[1:]
sections = []
if first['recovers']:
sections = self._ticket_sections(first['event_id'], first['event_type'], self.modem_name, first['detail'])
if others:
sections.append(('Also seen in this incident', [f'- **{o["name"]}:** {o["summary"]}' for o in others]))
desc = ticket_text.description(first['heading'], first['facts'], first['summary'], first['checks'],
sections, recovers=any(i['recovers'] for i in items))
tid = self.tickets.create(first['title'], desc, priority=first['priority'])
if tid:
for item in items:
db.set_ticket_id(item['event_id'], tid)
def _note_unifi_update(self, event: dict) -> None:
"""Hold new modem/WAN tickets for a while and explain the ones opened just before."""
label = modem_api.update_label(event)
when = datetime.fromtimestamp(event['ts'], timezone.utc)
logger.info(f'UniFi update finished: {label} at {when:%Y-%m-%d %H:%M:%S} UTC')
if event['ts'] + UPDATE_HOLD_S > time.time():
db.set_state(UNIFI_UPDATE_STATE, {'label': label, 'at': event['ts'],
'hold_until': event['ts'] + UPDATE_HOLD_S})
naive = when.replace(tzinfo=None)
tickets = db.get_ticket_ids_between(
self.modem_name, naive - timedelta(seconds=UPDATE_LOOKBACK_S), naive + timedelta(seconds=UPDATE_HOLD_S))
for tid in tickets:
self.tickets.comment(tid, ticket_text.update_comment(
label, f'{when:%Y-%m-%d %H:%M} UTC', UPDATE_LOOKBACK_S // 60, UPDATE_HOLD_S // 60))
def _process_modem_failure(self, suppressions: list, error: Exception) -> None:
"""Handle a failed poll. Link and optical events stay as they are: we can't see them."""
@@ -1690,14 +1850,13 @@ class NetworkMonitor:
name = self.modem_name
event_id, _, _, ticket_id = db.upsert_event(event_type, 'warning', 'unifi', name, '', description)
if not db.check_suppressed(suppressions, 'host', name) and not ticket_id:
tid = self.tickets.create(
f'[{name}][auto][production][issue][network][cluster-wide] {heading}',
ticket_text.description('WAN Event', [('Modem', name), ('Reported', _now_utc())],
description, [advice], recovers=False),
priority=priority,
)
if tid:
db.set_ticket_id(event_id, tid)
self._queue_wan_ticket({
'event_id': event_id, 'event_type': event_type, 'detail': '',
'title': f'[{name}][auto][production][issue][network][cluster-wide] {heading}',
'name': heading, 'heading': 'WAN Event',
'facts': [('Modem', name), ('Reported', _now_utc())],
'summary': description, 'checks': [advice], 'priority': priority, 'recovers': False,
})
db.resolve_event(event_type, name, '')
def _process_wan_log(self, suppressions: list) -> None:
@@ -1713,12 +1872,21 @@ class NetworkMonitor:
return
# After long monitor downtime, don't open tickets for stale events.
since_ms = max(int(raw_cursor) + 1, int((now - 86400) * 1000))
events = self.unifi.get_system_events(since_ms, modem_api.WAN_LOG_KEYS)
events = self.unifi.get_system_events(since_ms, modem_api.wanted_log_key)
if not events:
return # None = log unavailable, so the cursor stays put and it's retried
# Updates first: one in this batch explains (and holds) the WAN events beside it
updates = [e for e in events if modem_api.is_update_key(e['key'])]
for e in sorted(updates, key=lambda ev: ev['ts']):
try:
self._note_unifi_update(e)
except Exception as ex:
logger.error(f'Noting UniFi update failed: {ex}', exc_info=True)
history = self._load_wan_history()
actionable = modem_api.apply_wan_events(history, events)
actionable = modem_api.apply_wan_events(
history, [e for e in events if not modem_api.is_update_key(e['key'])])
# UniFi repeats "failed multiple times" per failure, so ticket it once a day at most.
unstable = [e for e in actionable if e['key'] == modem_api.LOG_WAN_UNSTABLE]
@@ -1927,9 +2095,11 @@ class NetworkMonitor:
# WAN modem: isolated so a failure here can't stop the rest of the cycle
if self.modem:
try:
# Log first, so a restart it reports can explain an uptime reset below
self._process_wan_log(suppressions)
self._collect_modem(suppressions)
# Log first, so a restart it reports can explain an uptime reset below.
# Tickets open when the batch ends, so one incident gets one ticket.
with self._wan_batch():
self._process_wan_log(suppressions)
self._collect_modem(suppressions)
except Exception as e:
logger.error(f'Modem collection failed: {e}', exc_info=True)
@@ -1938,6 +2108,10 @@ class NetworkMonitor:
self._post_recovery_notes()
except Exception as e:
logger.error(f'Posting recovery notes failed: {e}', exc_info=True)
try:
self._sync_open_tickets()
except Exception as e:
logger.error(f'Syncing open tickets failed: {e}', exc_info=True)
# Housekeeping: deactivate expired suppressions and purge old resolved events
db.cleanup_expired_suppressions()
+108 -4
View File
@@ -55,7 +55,10 @@ async function refreshAll() {
if (statusResult.status === 'fulfilled') {
const status = statusResult.value;
updateEventsTable(status.events || [], status.total_active);
updateStatusBar(status.summary || {}, status.last_check || '', status.daemon_ok);
updateFollowUp(status.stale_tickets || []);
updateStatusBar(status.summary || {}, status.last_check || '', status.daemon_ok,
followUpCount(status.events || [], status.stale_tickets || []),
(status.stale_tickets || []).length > 0);
if (typeof window.gandalfNotifUpdate === 'function') window.gandalfNotifUpdate(status.events || []);
} else {
showToast('Status data unavailable', 'warning');
@@ -66,7 +69,14 @@ async function refreshAll() {
}
}
function updateStatusBar(summary, lastCheck, daemonOk) {
// Active alerts flagged stale or with a closed ticket, plus tickets left open after clearing
function followUpCount(events, staleTickets) {
const flagged = events.filter(e => e.severity !== 'info' &&
((e.stale_days !== null && e.stale_days !== undefined) || e.ticket_closed)).length;
return flagged + staleTickets.length;
}
function updateStatusBar(summary, lastCheck, daemonOk, followUps = 0, hasStaleTickets = false) {
const bar = document.querySelector('.status-chips');
if (!bar) return;
const chips = [];
@@ -74,6 +84,10 @@ function updateStatusBar(summary, lastCheck, daemonOk) {
if (summary.critical) chips.push(`<span class="chip chip-critical">● ${summary.critical} CRITICAL</span>`);
if (summary.warning) chips.push(`<span class="chip chip-warning">● ${summary.warning} WARNING</span>`);
if (!summary.critical && !summary.warning && daemonOk !== false) chips.push('<span class="chip chip-ok">✔ ALL SYSTEMS NOMINAL</span>');
if (followUps) {
const href = hasStaleTickets ? '#followup' : '#events-table-wrap';
chips.push(`<a class="chip chip-warning chip-followup" href="${href}">⏳ ${followUps} TO FOLLOW UP</a>`);
}
bar.innerHTML = chips.join('');
const lc = document.getElementById('last-check');
@@ -469,7 +483,7 @@ function updateEventsTable(events, totalActive) {
const truncated = totalActive !== null && totalActive !== undefined && totalActive > active.length;
const countNotice = truncated
? `<div class="pagination-notice">Showing ${active.length} of ${totalActive} active alerts &mdash; <a href="/api/events?limit=1000">view all via API</a></div>`
? `<div class="pagination-notice">Showing ${active.length} of ${totalActive} active alerts: <a href="/api/events?limit=1000">view all via API</a></div>`
: '';
const rows = active.map(e => {
@@ -485,9 +499,15 @@ function updateEventsTable(events, totalActive) {
const supBadge = e.is_suppressed
? `<span class="lt-badge badge-suppressed" title="Alert suppressed">🔕 sup</span>`
: '';
const staleBadge = (e.stale_days !== null && e.stale_days !== undefined)
? `<span class="lt-badge badge-stale" title="Active for ${Number(e.stale_days)} days with no fix yet">⏳ ${Number(e.stale_days)}d</span>`
: '';
const closedBadge = e.ticket_closed
? `<span class="lt-badge badge-ticket-closed" title="Ticket #${lt.escHtml(String(e.ticket_id))} was closed in Tinker, but this alert is still active">ticket closed</span>`
: '';
return `
<tr class="row-${e.severity}${e.is_suppressed ? ' row-suppressed' : ''}">
<td><span class="lt-badge badge-${e.severity}">${e.severity}</span>${supBadge}</td>
<td><span class="lt-badge badge-${e.severity}">${e.severity}</span>${supBadge}${staleBadge}${closedBadge}</td>
<td>${lt.escHtml(e.event_type.replace(/_/g,' '))}</td>
<td><strong>${lt.escHtml(e.target_name)}</strong></td>
<td>${lt.escHtml(e.target_detail || '-')}</td>
@@ -522,6 +542,28 @@ function updateEventsTable(events, totalActive) {
</div>`;
}
// ── Needs follow-up: tickets left open after their alert cleared ──────
function updateFollowUp(tickets) {
const section = document.getElementById('followup');
const tbody = document.querySelector('#followup-table tbody');
if (!section || !tbody) return;
section.hidden = !tickets.length;
const badge = document.getElementById('followup-badge');
if (badge) badge.textContent = tickets.length;
const ticketBase = (typeof GANDALF_CONFIG !== 'undefined' && GANDALF_CONFIG.ticket_web_url)
? GANDALF_CONFIG.ticket_web_url : 'http://t.lotusguild.org/ticket/';
tbody.innerHTML = tickets.map(t => `
<tr>
<td><a href="${lt.escHtml(ticketBase)}${lt.escHtml(String(t.ticket_id))}" target="_blank" rel="noopener"
class="ticket-link">#${lt.escHtml(String(t.ticket_id))}</a></td>
<td><strong>${lt.escHtml(t.target_name || '')}</strong></td>
<td>${lt.escHtml((t.event_type || '').replace(/_/g, ' '))}${t.target_detail
? ` <span class="lt-text-muted">${lt.escHtml(t.target_detail)}</span>` : ''}</td>
<td class="ts-cell" title="${lt.escHtml(t.cleared_at || '')}">${lt.time.ago(_toIso(t.cleared_at))}</td>
<td>${Number(t.events) || 0}</td>
</tr>`).join('');
}
// ── Suppression modal ─────────────────────────────────────────────────
function openSuppressModal(type, name, detail) {
const modal = document.getElementById('suppress-modal');
@@ -648,3 +690,65 @@ document.addEventListener('click', e => {
);
}
});
// ── Maintenance: pause every ticket for a while (a global suppression with an expiry) ──
(function () {
const until = document.querySelector('.maint-banner .maint-until');
if (until) {
const t = new Date(until.getAttribute('datetime'));
if (!isNaN(t)) {
const sameDay = t.toDateString() === new Date().toDateString();
until.textContent = sameDay
? t.toLocaleTimeString([], { hour: 'numeric', minute: '2-digit' })
: t.toLocaleString([], { weekday: 'short', hour: 'numeric', minute: '2-digit' });
until.title = t.toLocaleString();
}
}
let minutes = 60;
document.addEventListener('click', async e => {
if (e.target.closest('[data-action="start-maintenance"]')) {
lt.modal.open('maint-modal');
document.getElementById('maint-reason')?.focus();
return;
}
const pill = e.target.closest('.maint-pill');
if (pill) {
minutes = parseInt(pill.dataset.minutes, 10);
document.querySelectorAll('.maint-pill').forEach(p => {
const on = p === pill;
p.classList.toggle('active', on);
p.setAttribute('aria-pressed', String(on));
});
return;
}
const end = e.target.closest('[data-action="end-maintenance"]');
if (end) {
end.disabled = true;
try {
await lt.api.delete(`/api/suppressions/${encodeURIComponent(end.dataset.supId)}`);
window.location.reload();
} catch (err) {
end.disabled = false;
showToast(err.message || 'Could not end maintenance', 'error');
}
}
});
document.getElementById('maint-form')?.addEventListener('submit', async e => {
e.preventDefault();
const reason = (document.getElementById('maint-reason').value || '').trim();
if (!reason) { showToast('Reason is required', 'error'); return; }
const submit = e.target.querySelector('[type="submit"]');
submit.disabled = true;
try {
await lt.api.post('/api/suppressions', {
target_type: 'all', target_name: '', reason, expires_minutes: minutes,
});
window.location.reload();
} catch (err) {
submit.disabled = false;
showToast(err.message || 'Could not start maintenance', 'error');
}
});
})();
+22
View File
@@ -229,6 +229,12 @@
.badge-resolved { color: var(--text-muted); border-color: var(--border-color); text-decoration: line-through; }
.badge-suppressed { font-size: .9em; padding: 0; border: none; color: var(--text-muted); }
.badge-purple { color: var(--accent-purple); border-color: var(--accent-purple); }
.badge-stale { color: var(--amber, var(--orange)); border-color: transparent; margin-left: 4px; white-space: nowrap; }
.badge-ticket-closed { color: var(--accent-purple); border-color: var(--accent-purple); margin-left: 4px; white-space: nowrap; }
.chip-followup { text-decoration: none; cursor: pointer; }
.chip-followup:hover, .chip-followup:focus-visible { background: rgba(255,107,0,.12); }
.followup-intro { color: var(--text-muted); font-size: .85em; margin: 0 0 8px; }
#followup-table td { white-space: nowrap; }
/* ── Table row state colors ───────────────────────────────────────── */
.lt-table tr.row-critical td { background: rgba(255,45,85,.04); }
@@ -383,6 +389,22 @@
.pill:hover { border-color: var(--green); color: var(--green); background: var(--green-dim); }
.pill.active,
.pill-manual.active { border-color: var(--amber); color: var(--amber); background: var(--amber-dim); text-shadow: var(--glow-amber); }
/* Maintenance modal pills: same look, own class so the suppress modal's handler skips them */
.maint-pill { padding: 3px 10px; border: 1px solid var(--border-color); background: transparent;
font-family: var(--font); font-size: .72em; font-weight: bold; cursor: pointer;
color: var(--text-muted); transition: all .15s; letter-spacing: .04em; }
.maint-pill:hover { border-color: var(--green); color: var(--green); background: var(--green-dim); }
.maint-pill.active { border-color: var(--amber); color: var(--amber); background: var(--amber-dim); text-shadow: var(--glow-amber); }
.maint-intro { margin: 0 0 12px; }
/* Maintenance banner: every page while a global suppression is active */
.maint-banner {
display: flex; align-items: center; justify-content: space-between; gap: 12px; flex-wrap: wrap;
margin-bottom: 16px; padding: 8px 14px; border: 1px solid var(--amber); background: var(--amber-dim);
color: var(--amber); font-size: .82em;
}
.maint-banner strong { letter-spacing: .06em; text-transform: uppercase; }
.maint-reason { color: var(--text-muted); }
/* ── Available targets grid (suppressions page) ───────────────────── */
.targets-grid { display: grid; grid-template-columns: repeat(auto-fill, minmax(180px, 1fr)); gap: 10px; }
+57 -1
View File
@@ -125,6 +125,10 @@
<span class="lt-header-user">{{ _uname }}</span>
{% if user.groups and 'admin' in user.groups %}
<span class="lt-badge lt-badge-admin" aria-label="Administrator">ADMIN</span>
{% if not maintenance %}
<button type="button" class="lt-btn lt-btn-ghost lt-btn-sm" data-action="start-maintenance"
title="Pause all tickets while you work on the network">Maintenance</button>
{% endif %}
{% endif %}
<!-- Notification bell: shows active monitoring alerts -->
@@ -184,6 +188,20 @@
<!-- MAIN CONTENT -->
<main class="lt-main lt-container" id="main-content">
{% if maintenance %}
<div class="maint-banner" role="status">
<span class="maint-banner-text">
<strong>Maintenance:</strong> Gandalf is not opening tickets
{% if maintenance.expires_at %}until <time class="maint-until" datetime="{{ maintenance.expires_at }}">{{ maintenance.expires_at }}</time>{% else %}until it is ended{% endif %}.
Alerts are still recorded, and anything still failing afterwards gets a ticket.
<span class="maint-reason">{{ maintenance.reason }}{% if maintenance.suppressed_by %} ({{ maintenance.suppressed_by }}){% endif %}</span>
</span>
{% if user.groups and 'admin' in user.groups %}
<button type="button" class="lt-btn lt-btn-secondary lt-btn-sm" data-action="end-maintenance"
data-sup-id="{{ maintenance.id }}">End now</button>
{% endif %}
</div>
{% endif %}
{% block content %}{% endblock %}
</main>
@@ -203,9 +221,47 @@
<span class="lt-footer-sep">|</span>
<button type="button" class="lt-footer-hint" data-action="show-keyboard-help"><span class="lt-footer-key">[ ? ]</span> HELP</button>
</nav>
<span>GANDALF &mdash; TDS v1.2</span>
<span>GANDALF | TDS v1.2</span>
</footer>
{% if user.groups and 'admin' in user.groups and not maintenance %}
<!-- MAINTENANCE MODAL: pause every ticket for a while -->
<div id="maint-modal" class="lt-modal-overlay"
role="dialog" aria-modal="true" aria-labelledby="maint-modal-title" aria-hidden="true">
<div class="lt-modal">
<div class="lt-modal-header">
<h3 class="lt-modal-title" id="maint-modal-title">Start maintenance</h3>
<button type="button" class="lt-modal-close" data-modal-close aria-label="Close">&#x2715;</button>
</div>
<form id="maint-form">
<div class="lt-modal-body">
<p class="lt-field-hint maint-intro">Gandalf keeps checking and recording alerts but opens no tickets
until this ends. Anything still failing afterwards gets a ticket as usual.</p>
<div class="lt-form-group">
<label class="lt-label" for="maint-reason">Reason <span class="required">*</span></label>
<input type="text" class="lt-input" id="maint-reason" name="reason" maxlength="500"
value="Maintenance" required aria-required="true"
placeholder="e.g. UniFi Network update">
</div>
<div class="lt-form-group lt-form-group--last">
<span class="lt-label" id="maint-len-label">How long</span>
<div class="duration-pills" role="group" aria-labelledby="maint-len-label">
<button type="button" class="maint-pill" data-minutes="30" aria-pressed="false">30 min</button>
<button type="button" class="maint-pill active" data-minutes="60" aria-pressed="true">1 hr</button>
<button type="button" class="maint-pill" data-minutes="120" aria-pressed="false">2 hr</button>
<button type="button" class="maint-pill" data-minutes="240" aria-pressed="false">4 hr</button>
</div>
</div>
</div>
<div class="lt-modal-footer">
<button type="button" class="lt-btn lt-btn-secondary" data-modal-close>Cancel</button>
<button type="submit" class="lt-btn lt-btn-primary">Start</button>
</div>
</form>
</div>
</div>
{% endif %}
<!-- QUICK-SUPPRESS MODAL: available on all pages via [S] shortcut -->
<div id="suppress-modal" class="lt-modal-overlay"
role="dialog" aria-modal="true" aria-labelledby="suppress-modal-title" aria-hidden="true">
+39
View File
@@ -18,6 +18,10 @@
{% if daemon_ok and not summary.critical and not summary.warning %}
<span class="chip chip-ok">✔ ALL SYSTEMS NOMINAL</span>
{% endif %}
{% set followups = (stale_tickets | length) + (stale_alerts or 0) %}
{% if followups %}
<a class="chip chip-warning chip-followup" href="{{ '#followup' if stale_tickets else '#events-table-wrap' }}">⏳ {{ followups }} TO FOLLOW UP</a>
{% endif %}
</div>
<div class="status-meta">
<span class="last-check" id="last-check" data-tooltip="Last monitor poll timestamp" data-tooltip-pos="bottom">{{ last_check }}</span>
@@ -129,6 +133,8 @@
<td>
<span class="lt-badge badge-{{ e.severity }}">{{ e.severity }}</span>
{% if e.is_suppressed %}<span class="lt-badge badge-suppressed" title="Alert suppressed">🔕 sup</span>{% endif %}
{% if e.stale_days is not none %}<span class="lt-badge badge-stale" title="Active for {{ e.stale_days }} days with no fix yet">⏳ {{ e.stale_days }}d</span>{% endif %}
{% if e.ticket_closed %}<span class="lt-badge badge-ticket-closed" title="Ticket #{{ e.ticket_id }} was closed in Tinker, but this alert is still active">ticket closed</span>{% endif %}
</td>
<td>{{ e.event_type | replace('_', ' ') }}</td>
<td><strong>{{ e.target_name }}</strong></td>
@@ -177,6 +183,39 @@
</div><!-- /.lt-frame -->
</section>
<!-- ── Needs follow-up: tickets left open after their alert cleared ── -->
<section class="g-section" id="followup"{% if not stale_tickets %} hidden{% endif %}>
<div class="g-section-header">
<h2 class="g-section-title">Needs Follow-up</h2>
<span class="g-section-badge" id="followup-badge">{{ stale_tickets | length }}</span>
</div>
<p class="followup-intro">These tickets are still open in Tinker, but their alert cleared over a day ago. Close them if the problem is fixed.</p>
<div class="lt-frame">
<span class="lt-frame-bl">&#x255A;</span>
<span class="lt-frame-br">&#x255D;</span>
<div class="lt-section-header">Tickets Left Open</div>
<div class="lt-table-wrap">
<table class="lt-table" id="followup-table">
<caption class="lt-sr-only">Tickets still open after their alert cleared</caption>
<thead>
<tr><th>Ticket</th><th>Target</th><th>Alert</th><th>Cleared</th><th>Alerts on ticket</th></tr>
</thead>
<tbody>
{% for t in stale_tickets %}
<tr>
<td><a href="{{ config.ticket_api.web_url }}{{ t.ticket_id }}" target="_blank" rel="noopener" class="ticket-link">#{{ t.ticket_id }}</a></td>
<td><strong>{{ t.target_name }}</strong></td>
<td>{{ t.event_type | replace('_', ' ') }}{% if t.target_detail %} <span class="lt-text-muted">{{ t.target_detail }}</span>{% endif %}</td>
<td class="ts-cell" title="{{ t.cleared_at }}"><span class="event-age" data-ts="{{ t.cleared_at }}">{{ t.cleared_at }}</span></td>
<td>{{ t.events }}</td>
</tr>
{% endfor %}
</tbody>
</table>
</div>
</div>
</section>
<!-- ── WAN / fiber ──────────────────────────────────────────────────── -->
{% include "_modem_panel.html" %}
+93
View File
@@ -1,5 +1,7 @@
"""Tests for app.py routes: database and LDAP are mocked."""
import json
import time
from datetime import datetime, timedelta, timezone
import os
import sys
from unittest.mock import patch
@@ -610,3 +612,94 @@ class TestUnusedInterfaces:
gandalf.db.get_unused_interfaces.return_value = {'large1': {'enp36s0': {'reason': 'spare'}}}
data = client.get('/api/device/large1', headers=AUTH).get_json()
assert data['unused_interfaces'] == {'enp36s0': {'reason': 'spare'}}
class TestMaintenance:
SUP = {'id': 7, 'target_type': 'all', 'target_name': '', 'reason': 'UniFi Network update',
'suppressed_by': 'jared', 'expires_at': '2026-11-12T20:00:00Z'}
def test_no_banner_and_a_start_button_by_default(self, client):
html = client.get('/', headers=AUTH).get_data(as_text=True)
assert 'maint-banner' not in html
assert 'data-action="start-maintenance"' in html and 'id="maint-modal"' in html
def test_banner_on_every_page_while_on(self, client):
gandalf.db.get_active_suppressions.return_value = [
{'id': 3, 'target_type': 'host', 'target_name': 'large1', 'reason': 'x', 'expires_at': None},
self.SUP,
]
for page in ('/', '/events', '/rack'):
html = client.get(page, headers=AUTH).get_data(as_text=True)
assert 'class="maint-banner"' in html, page
assert 'datetime="2026-11-12T20:00:00Z"' in html and 'UniFi Network update' in html
assert 'data-sup-id="7"' in html and 'data-action="start-maintenance"' not in html
def test_open_ended_maintenance_wins(self, client):
gandalf.db.get_active_suppressions.return_value = [
self.SUP, dict(self.SUP, id=9, expires_at=None, reason='Rewiring')]
html = client.get('/', headers=AUTH).get_data(as_text=True)
assert 'until it is ended' in html and 'data-sup-id="9"' in html
def test_viewers_see_the_banner_without_buttons(self, client):
gandalf.db.get_active_suppressions.return_value = [self.SUP]
viewer = dict(AUTH, **{'Remote-Groups': 'staff'})
with patch.object(gandalf, '_cfg', dict(CONFIG, auth=dict(CONFIG['auth'], allowed_groups=['admin', 'staff']))):
html = client.get('/', headers=viewer).get_data(as_text=True)
assert 'class="maint-banner"' in html and 'data-action="end-maintenance"' not in html
def test_a_db_error_means_no_banner(self, client):
gandalf.db.get_active_suppressions.side_effect = RuntimeError('db down')
assert gandalf.inject_maintenance() == {'maintenance': None}
class TestFollowUp:
def _state(self, tickets, age_s=60):
state = dict(STATE, open_tickets=json.dumps({'checked_at': time.time() - age_s, 'tickets': tickets}))
gandalf.db.get_state.side_effect = lambda key, default=None: state.get(key, default)
def _event(self, days_old, ticket_id=None):
first = (datetime.now(timezone.utc) - timedelta(days=days_old)).strftime('%Y-%m-%dT%H:%M:%SZ')
return {'id': 1, 'event_type': 'interface_down', 'severity': 'critical', 'target_name': 'micro1',
'target_detail': 'enp3s0', 'description': 'down', 'first_seen': first, 'last_seen': first,
'consecutive_failures': 3, 'ticket_id': ticket_id}
def test_old_alert_is_stale_and_closed_ticket_is_flagged(self, client):
self._state({'9': 'Open'})
gandalf.db.get_active_events.return_value = [self._event(5, '8'), self._event(1, '9')]
gandalf.db.get_cleared_tickets.return_value = []
events = client.get('/api/status', headers=AUTH).get_json()['events']
assert events[0]['stale_days'] == 5 and events[0]['ticket_closed'] is True
assert events[1]['stale_days'] is None and events[1]['ticket_closed'] is False
def test_threshold_comes_from_config(self, client):
self._state({})
gandalf.db.get_active_events.return_value = [self._event(1.5)]
gandalf.db.get_cleared_tickets.return_value = []
cfg = dict(CONFIG, monitor={'stale_alert_days': 1})
with patch.object(gandalf, '_cfg', cfg):
events = client.get('/api/status', headers=AUTH).get_json()['events']
assert events[0]['stale_days'] == 1
def test_tickets_left_open_are_listed(self, client):
self._state({'281331508': 'Open'})
left = [{'ticket_id': '281331508', 'event_type': 'host_unreachable', 'target_name': 'pbs',
'target_detail': '', 'severity': 'critical', 'first_seen': '2026-10-14T12:02:05Z',
'cleared_at': '2026-11-04T22:12:36Z', 'events': 3}]
gandalf.db.get_cleared_tickets.return_value = left
assert client.get('/api/status', headers=AUTH).get_json()['stale_tickets'] == left
gandalf.db.get_cleared_tickets.assert_called_with(['281331508'], older_than_hours=24)
html = client.get('/', headers=AUTH).get_data(as_text=True)
assert '#281331508' in html and 'TO FOLLOW UP' in html
assert '<section class="g-section" id="followup">' in html
def test_out_of_date_list_flags_nothing(self, client):
self._state({}, age_s=3 * 3600)
gandalf.db.get_active_events.return_value = [self._event(1, '8')]
body = client.get('/api/status', headers=AUTH).get_json()
assert body['stale_tickets'] == [] and body['events'][0]['ticket_closed'] is False
gandalf.db.get_cleared_tickets.assert_not_called()
def test_no_list_yet_hides_the_panel(self, client):
html = client.get('/', headers=AUTH).get_data(as_text=True)
assert '<section class="g-section" id="followup" hidden>' in html
assert 'TO FOLLOW UP' not in html
+64
View File
@@ -0,0 +1,64 @@
"""Tests for db.py query helpers that do work beyond the SQL: the connection is faked."""
import os
import sys
from contextlib import contextmanager
from datetime import datetime, timedelta
from unittest.mock import MagicMock, patch
sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
import db # noqa: E402
NOW = datetime(2026, 11, 14, 12, 0, 0)
def _fake_conn(rows):
cur = MagicMock()
cur.fetchall.return_value = rows
cur.fetchone.return_value = {'now': NOW}
conn = MagicMock()
conn.cursor.return_value.__enter__.return_value = cur
@contextmanager
def get_conn():
yield conn
return patch.object(db, 'get_conn', get_conn), cur
def _event(tid, hours_ago_first, hours_ago_resolved, etype='host_unreachable', target='pbs'):
return {
'ticket_id': tid, 'event_type': etype, 'target_name': target, 'target_detail': '',
'severity': 'critical', 'first_seen': NOW - timedelta(hours=hours_ago_first),
'resolved_at': None if hours_ago_resolved is None else NOW - timedelta(hours=hours_ago_resolved),
}
class TestGetClearedTickets:
def test_no_ids_skips_the_query(self):
patcher, cur = _fake_conn([])
with patcher:
assert db.get_cleared_tickets([]) == []
cur.execute.assert_not_called()
def test_only_tickets_whose_every_alert_cleared_long_enough_ago(self):
rows = [
_event('1', 300, 250), _event('1', 100, 48), # two outages, both long cleared
_event('2', 50, 40), _event('2', 5, None), # still active
_event('3', 10, 2), # cleared too recently
]
patcher, cur = _fake_conn(rows)
with patcher:
out = db.get_cleared_tickets(['3', '1', '2', '1'], older_than_hours=24)
assert [r['ticket_id'] for r in out] == ['1']
assert out[0]['events'] == 2
assert out[0]['cleared_at'] == (NOW - timedelta(hours=48)).isoformat() + 'Z'
assert out[0]['first_seen'] == (NOW - timedelta(hours=300)).isoformat() + 'Z'
assert cur.execute.call_args_list[0].args[1] == ['1', '2', '3']
def test_newest_clearance_first(self):
rows = [_event('1', 400, 300), _event('2', 100, 30, target='micro1')]
patcher, _ = _fake_conn(rows)
with patcher:
out = db.get_cleared_tickets(['1', '2'])
assert [r['ticket_id'] for r in out] == ['2', '1']
assert out[0]['target_name'] == 'micro1'
+174
View File
@@ -31,6 +31,8 @@ def db():
mock_db.cutsheet_load.return_value = {'items': [], 'cables': [], 'port_labels': []}
mock_db.get_pending_recovery_notes.return_value = []
mock_db.get_device_link.return_value = ''
mock_db.get_incident_ticket.return_value = None
mock_db.get_ticket_ids_between.return_value = []
yield mock_db
@@ -381,6 +383,47 @@ class TestWanLog:
mon._process_wan_log([])
assert not mon.tickets.create.called
def test_unifi_update_is_not_a_wan_event(self, mon, db):
now = time.time()
self._state(db, unifi_log_cursor=str(int(now * 1000)))
self._mon(mon, [{'key': 'NETWORK_UPDATED', 'ts': now, 'device': '', 'wan': '',
'app': 'UniFi Network', 'previous': '11.0.97', 'version': '11.0.99'}])
mon._process_wan_log([])
assert not mon.tickets.create.called
saved = dict((c.args[0], c.args[1]) for c in db.set_state.call_args_list)
assert saved[monitor.UNIFI_UPDATE_STATE]['label'] == 'UniFi Network 11.0.97 to 11.0.99'
assert saved['wan_history'] == {'outages': [], 'restarts': []}
def test_update_comments_on_tickets_opened_just_before(self, mon, db):
now = time.time()
self._state(db, unifi_log_cursor=str(int(now * 1000)))
db.get_ticket_ids_between.return_value = ['207521068']
self._mon(mon, [{'key': 'NETWORK_UPDATED', 'ts': now, 'device': '', 'wan': '',
'app': 'UniFi Network', 'previous': '11.0.97', 'version': '11.0.99'}])
mon._process_wan_log([])
start, end = db.get_ticket_ids_between.call_args.args[1:]
assert (end - start).total_seconds() == monitor.UPDATE_LOOKBACK_S + monitor.UPDATE_HOLD_S
tid, text = mon.tickets.comment.call_args.args
assert tid == '207521068' and 'UniFi Network 11.0.97 to 11.0.99' in text
def test_update_in_the_same_read_holds_the_power_cycle_ticket(self, mon, db):
now = time.time()
state = {'unifi_log_cursor': str(int(now * 1000))}
db.get_state.side_effect = lambda key, default=None: state.get(key, default)
db.set_state.side_effect = lambda key, value: state.__setitem__(
key, json.dumps(value) if isinstance(value, dict) else value)
self._mon(mon, [
{'key': modem_api.LOG_MODEM_RESTARTED, 'ts': now - 600, 'device': 'RPDU', 'wan': ''},
{'key': 'NETWORK_UPDATED', 'ts': now, 'device': '', 'wan': '', 'app': 'UniFi Network'},
])
mon._process_wan_log([])
assert not mon.tickets.create.called
db.resolve_event.assert_called_with('modem_power_cycled', 'att-modem', '')
def test_hold_ends(self, mon, db):
self._state(db, unifi_update=json.dumps({'label': 'x', 'hold_until': time.time() - 1}))
assert mon._update_hold() is None
def test_uptime_going_backwards_records_a_restart(self, mon, db):
self._state(db, modem_stats=json.dumps({'uptime_s': 5000}))
mon._note_modem_restart({'uptime_s': 60})
@@ -393,6 +436,55 @@ class TestWanLog:
assert not db.set_state.called
class TestWanIncident:
"""Last night's sequence: the RPDU power-cycles the modem in the cycle the modem goes unreachable."""
def _cycle(self, mon, db):
db.upsert_event.side_effect = [(30, True, 1, None), (29, False, 2, None)]
mon.modem.poll.side_effect = modem_api.ModemUnreachable('Network is unreachable')
with mon._wan_batch():
mon._one_shot_ticket([], 'modem_power_cycled', 'RPDU power-cycled the modem.',
'Modem auto power-cycled after internet loss', '3', 'Check the WAN panel.')
mon._collect_modem([])
def test_one_ticket_led_by_the_most_urgent(self, mon, db):
self._cycle(mon, db)
assert mon.tickets.create.call_count == 1
title, desc = mon.tickets.create.call_args.args
assert 'WAN modem unreachable' in title
assert mon.tickets.create.call_args.kwargs['priority'] == '2'
assert '### Also seen in this incident' in desc and 'power-cycled' in desc
assert sorted(c.args for c in db.set_ticket_id.call_args_list) == [(29, '123'), (30, '123')]
def test_recent_incident_gets_a_comment_instead(self, mon, db):
db.get_incident_ticket.return_value = '555'
mon.tickets.comment.return_value = True
self._cycle(mon, db)
assert not mon.tickets.create.called
assert [c.args[0] for c in mon.tickets.comment.call_args_list] == ['555', '555']
assert 'Also seen: WAN modem unreachable' in mon.tickets.comment.call_args_list[0].args[1]
assert sorted(c.args for c in db.set_ticket_id.call_args_list) == [(29, '555'), (30, '555')]
db.get_incident_ticket.assert_called_with('att-modem', monitor.WAN_INCIDENT_WINDOW_S)
def test_failed_comment_falls_back_to_a_ticket(self, mon, db):
db.get_incident_ticket.return_value = '555'
mon.tickets.comment.return_value = False
self._cycle(mon, db)
assert mon.tickets.create.call_count == 1
def test_outside_a_batch_tickets_open_at_once(self, mon, db):
mon._one_shot_ticket([], 'wan_unstable', 'Unstable.', 'WAN connection unstable', '2', 'Look.')
assert 'unstable' in mon.tickets.create.call_args.args[0]
assert 'no recovery note' in mon.tickets.create.call_args.args[1]
def test_update_hold_skips_new_tickets(self, mon, db):
db.get_state.side_effect = lambda key, default=None: (
json.dumps({'label': 'UniFi Network 11.0.99', 'hold_until': time.time() + 60})
if key == monitor.UNIFI_UPDATE_STATE else default)
self._cycle(mon, db)
assert not mon.tickets.create.called and not mon.tickets.comment.called
class TestUnifiDeviceDetail:
def test_parses_string_stats_and_outlets(self):
d = monitor.UnifiClient._device_detail({
@@ -421,6 +513,31 @@ class TestUnifiDeviceDetail:
got = c.get_system_events(1000, modem_api.WAN_LOG_KEYS)
assert got == [{'key': modem_api.LOG_MODEM_RESTARTED, 'ts': 2.0, 'device': 'RPDU', 'wan': ''}]
def test_system_log_reads_update_versions(self):
c = monitor.UnifiClient({'controller': 'https://u.test', 'api_key': 'k'})
page = MagicMock()
page.json.side_effect = [
{'data': [
{'key': 'NETWORK_UPDATED', 'timestamp': 3000, 'parameters': {
'APPLICATION': 'UniFi Network', 'PREVIOUS_VERSION': '11.0.97', 'VERSION': '11.0.99'}},
{'key': 'FIRMWARE_UPDATE_AVAILABLE_V2', 'timestamp': 3000},
{'key': 'MADE_CHANGES_ONE_SETTING_OBJECT', 'timestamp': 3000},
]},
{'data': []},
]
c.session.post = MagicMock(return_value=page)
got = c.get_system_events(1000, modem_api.wanted_log_key)
assert got == [{'key': 'NETWORK_UPDATED', 'ts': 3.0, 'device': '', 'wan': '',
'app': 'UniFi Network', 'previous': '11.0.97', 'version': '11.0.99'}]
@pytest.mark.parametrize('key,expected', [
('NETWORK_UPDATED', True), ('FIRMWARE_UPDATED_V2', True), ('PROTECT_UPDATED', True),
('FIRMWARE_UPDATE_AVAILABLE_V2', False), ('MADE_CHANGES_ONE_SETTING_OBJECT', False),
('NETWORK_WAN_FAILED_2', False),
])
def test_update_keys(self, key, expected):
assert modem_api.is_update_key(key) is expected
def test_system_log_failure_is_none_not_empty(self):
c = monitor.UnifiClient({'controller': 'https://u.test', 'api_key': 'k'})
c.session.post = MagicMock(side_effect=OSError('boom'))
@@ -796,3 +913,60 @@ class TestTicketClientCreate:
with self._post({'success': True, 'ticket_id': '8', 'action': 'none'}):
assert c.create('t', 'd') == '8'
assert 't' not in c._closed_matches
class TestTicketClientOpenTickets:
def _client(self):
return monitor.TicketClient({'url': 'http://t.test/create_ticket_api.php', 'api_key': 'k'})
def _page(self, tickets, pages):
resp = MagicMock(status_code=200)
resp.json.return_value = {'success': True, 'tickets': tickets, 'page': 1, 'pages': pages}
return resp
def test_read_endpoint_sits_beside_create(self):
assert self._client().read_url == 'http://t.test/api/tickets_api.php'
def test_reads_every_page_of_open_statuses(self):
pages = [self._page([{'ticket_id': '1', 'status': 'Open'}], 2),
self._page([{'ticket_id': 2, 'status': 'Pending'}], 2)]
with patch.object(monitor.requests, 'get', side_effect=pages) as get:
assert self._client().open_tickets() == {'1': 'Open', '2': 'Pending'}
assert get.call_count == 2
params = get.call_args.kwargs['params']
assert params['status'] == 'Open,Pending,In Progress' and params['page'] == 2
assert get.call_args.kwargs['headers'] == {'Authorization': 'Bearer k'}
def test_failure_means_unknown_not_empty(self):
with patch.object(monitor.requests, 'get', side_effect=OSError('down')):
assert self._client().open_tickets() is None
bad = MagicMock(status_code=200)
bad.json.return_value = {'success': False, 'error': 'nope'}
with patch.object(monitor.requests, 'get', return_value=bad):
assert self._client().open_tickets() is None
def test_unconfigured(self):
assert monitor.TicketClient({}).open_tickets() is None
class TestSyncOpenTickets:
def test_saves_the_list_then_waits(self, mon, db):
mon.tickets.open_tickets.return_value = {'7': 'Open'}
mon._sync_open_tickets()
mon._sync_open_tickets()
assert mon.tickets.open_tickets.call_count == 1
key, value = db.set_state.call_args.args
assert key == monitor.OPEN_TICKETS_STATE and value['tickets'] == {'7': 'Open'}
assert value['checked_at'] > 0
def test_failed_read_keeps_the_previous_list(self, mon, db):
mon.tickets.open_tickets.return_value = None
mon._sync_open_tickets()
db.set_state.assert_not_called()
def test_refreshes_after_the_interval(self, mon, db):
mon.tickets.open_tickets.return_value = {}
mon._sync_open_tickets()
mon._open_tickets_synced -= monitor.OPEN_TICKETS_SYNC_S + 1
mon._sync_open_tickets()
assert mon.tickets.open_tickets.call_count == 2
+22
View File
@@ -147,3 +147,25 @@ def recovery_comment(event: dict, now: datetime, url: str = '') -> str:
lines += ['', '_Gandalf leaves this ticket open. Close it once you know why it happened, '
'or leave it open if it needs follow-up._']
return '\n'.join(lines)
def incident_comment(name: str, summary: str, checks: Sequence[str], when: str) -> str:
"""A further report that belongs to the incident this ticket is already about."""
lines = [f'**Also seen: {name}** ({when})', '', summary.strip()]
if checks:
lines += ['', '**What to check**'] + [f'{n}. {check}' for n, check in enumerate(checks, 1)]
lines += ['', '_Gandalf added this here instead of opening another ticket, since it is part of '
'the same incident._']
return '\n'.join(lines)
def update_comment(label: str, when: str, lookback_min: int, hold_min: int) -> str:
"""Note on a ticket that a UniFi update finished around the time it was opened."""
return '\n'.join([
f'**UniFi update:** {label} finished at {when}.', '',
f'This ticket was opened within {lookback_min} minutes of that update, which UniFi logs once '
f'it is done. Updating the Network app or the gateway firmware can drop the WAN, and Smart '
f'Power may then power-cycle the modem, so the update is the likely cause.', '',
f'_Gandalf holds new modem and WAN tickets for {hold_min} minutes after an update; anything '
f'still wrong after that gets its own ticket._',
])