Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
0921e82c54 | ||
|
|
7b80633088 | ||
|
|
379e3be365 | ||
|
|
6b03fe7b44 |
No files matched your search
@@ -111,6 +111,9 @@ Ping-only (no node_exporter): **pbs** (10.10.10.3)
|
||||
- WAN / Fiber panel: WAN and PON state, Rx/Tx power, optics temperature, throughput, errors, gateway uptime and last restart, 7-day Rx trend, recent WAN outages
|
||||
- UniFi device table (switches, APs, gateway)
|
||||
- Active alerts table with severity, target, consecutive failures, ticket link
|
||||
- Follow-up flags (see Follow-up below): a ⏳ badge on alerts active for days, a "ticket closed" badge when the
|
||||
ticket was closed but the alert is still active, and a Needs Follow-up panel of tickets left open after their
|
||||
alert cleared. A "TO FOLLOW UP" chip in the status bar counts them all.
|
||||
- Quick-suppress modal: apply timed or manual suppression from any alert row
|
||||
- Auto-refreshes every 30 seconds via `/api/status` + `/api/network`
|
||||
|
||||
@@ -309,6 +312,12 @@ can view it; only the `admin` group can edit, and every change is logged with wh
|
||||
- Suppression history (last 50)
|
||||
- Available targets reference grid (all known hosts + interfaces)
|
||||
|
||||
**Maintenance**: admins get a **Maintenance** button in the header. It starts a global suppression for
|
||||
30 min, 1, 2 or 4 hours with a reason (by default "Maintenance"). While it runs, every page shows a banner
|
||||
saying when tickets resume, with **End now** for admins. Gandalf keeps checking and recording alerts, and
|
||||
anything still failing when maintenance ends gets a ticket as usual. Use it before a UniFi update, a switch
|
||||
swap or recabling.
|
||||
|
||||
---
|
||||
|
||||
## Alert Logic
|
||||
@@ -375,6 +384,20 @@ A note that fails to post is retried every cycle for 24 hours. Posting needs a T
|
||||
`read_write` scope; if Tinker answers 401 or 403, the monitor logs one error and stops trying until it is
|
||||
restarted.
|
||||
|
||||
### Follow-up
|
||||
|
||||
Gandalf only comments on tickets, so a ticket can stay open long after its problem is fixed, and an alert can
|
||||
sit active for days once everyone has stopped looking. The dashboard flags both:
|
||||
|
||||
- **Stale alert**: active for `monitor.stale_alert_days` (default 3) or longer. Shown as a ⏳ badge with its age.
|
||||
- **Ticket closed**: the alert is still active but its ticket was closed in Tinker, so nobody is tracking it.
|
||||
- **Ticket left open**: every alert on a ticket cleared at least `monitor.stale_ticket_hours` (default 24) ago,
|
||||
but the ticket is still Open, Pending or In Progress. Listed in the Needs Follow-up panel.
|
||||
|
||||
The monitor reads the list of open tickets from Tinker's `api/tickets_api.php` every 15 minutes (one request,
|
||||
read scope only) and saves it. If that list is over 2 hours old, because Tinker or the key stopped working,
|
||||
the ticket flags are left off rather than guessed. Stale alerts are still flagged.
|
||||
|
||||
### WAN modem notes
|
||||
|
||||
The alarm and warning limits are read from the gateway on every poll, so they are never hard-coded here.
|
||||
@@ -386,7 +409,23 @@ Outages can be over between polls: two real ones lasted about two minutes agains
|
||||
So WAN outages and modem restarts also come from the UniFi system log, and a drop in the modem's uptime
|
||||
records a restart even if nothing logged it. A single short outage only appears in the panel's history.
|
||||
|
||||
To silence modem alerts during maintenance, create a `host` suppression with the modem's `name`.
|
||||
**One ticket per incident.** The modem and WAN reports from one check cycle open a single ticket, led by
|
||||
the most urgent (a P2 "WAN modem unreachable" leads, and an RPDU power-cycle seen in the same cycle is listed
|
||||
under "Also seen in this incident"). A report that comes while the modem has an open alert, or one that
|
||||
ended under 30 minutes ago, becomes a comment on that alert's ticket instead of a new ticket. If the
|
||||
comment can't be posted, the report gets its own ticket as before.
|
||||
|
||||
**UniFi updates.** UniFi logs `NETWORK_UPDATED` (Network app) and `FIRMWARE_UPDATED_V2` (device firmware)
|
||||
once an update has finished. Updating the Network app can drop the UDM's WAN, and Smart Power may then
|
||||
power-cycle a modem that was fine. When Gandalf reads one of these entries it:
|
||||
|
||||
- comments on modem and WAN tickets first seen from 20 minutes before the update to 15 minutes after,
|
||||
naming the update (for example "UniFi Network 11.0.97 to 11.0.99") as the likely cause;
|
||||
- opens no new modem or WAN tickets for 15 minutes after the update. A problem still there after that gets
|
||||
its ticket as usual; a power-cycle report during the hold only goes into the event history.
|
||||
|
||||
To silence modem alerts during maintenance, create a `host` suppression with the modem's `name`, or use
|
||||
**Maintenance** (below) to pause every ticket.
|
||||
|
||||
### Baseline Tracking
|
||||
|
||||
@@ -486,6 +525,7 @@ Shared by both processes. Located in the working directory (`/var/www/html/prod/
|
||||
| `ticket_api.url` | Tinker Tickets `create_ticket_api.php` URL |
|
||||
| `ticket_api.api_key` | Tinker Tickets bearer token. Needs `read_write` scope to add recovery notes |
|
||||
| `ticket_api.comment_url` | Tinker comment endpoint (default: `api/ticket_comment_api.php` beside `ticket_api.url`) |
|
||||
| `ticket_api.read_url` | Tinker ticket read endpoint, used to find tickets left open (default: `api/tickets_api.php` beside `ticket_api.url`) |
|
||||
| `gandalf.public_url` | Gandalf's own address, for links in tickets (default: `https://gandalf.lotusguild.org`) |
|
||||
| `rack.units`, `rack.name` | The single rack's height and name on `/rack` (defaults: 42, "Rack 1") |
|
||||
| `racks` | Several racks: `[{"name": "...", "units": 42}, ...]`, in tab order; overrides `rack` |
|
||||
@@ -503,6 +543,8 @@ Shared by both processes. Located in the working directory (`/var/www/html/prod/
|
||||
| `monitor.unifi_failure_threshold` | Consecutive failures before ticketing a UniFi device as offline (default: same as `failure_threshold`). Set higher than `failure_threshold`: UniFi switches/APs can take several minutes to rejoin after a firmware-update reboot, so a low threshold cuts tickets for momentary maintenance reboots instead of real failures. |
|
||||
| `monitor.cluster_threshold` | Hosts with failures to trigger cluster-wide P1 (default: 3) |
|
||||
| `monitor.ping_hosts` | Hosts checked only by ping (no node_exporter) |
|
||||
| `monitor.stale_alert_days` | Days an alert can stay active before the dashboard flags it as stale (default: 3) |
|
||||
| `monitor.stale_ticket_hours` | Hours a ticket can stay open after all its alerts cleared before it is listed under Needs Follow-up (default: 24) |
|
||||
| `cutsheet.mismatch_ticket_hours` | Hours a cutsheet mismatch must last before it opens a P4 ticket (default: 24; `0` never tickets). The mismatch shows on the dashboard and the switch's page straight away, so recabling never pages anyone. Suppressing the switch or its port also stops the ticket. |
|
||||
| `modem.url` | AT&T gateway base URL. Omit the `modem` block to disable modem monitoring. Must be reachable from the monitor host. |
|
||||
| `modem.name` | Name used for the modem's alerts, tickets, and suppressions (default: `att-modem`) |
|
||||
|
||||
@@ -60,6 +60,19 @@ def inject_config():
|
||||
}
|
||||
|
||||
|
||||
@app.context_processor
|
||||
def inject_maintenance():
|
||||
"""The active global suppression (maintenance), shown as a banner on every page."""
|
||||
try:
|
||||
active = [s for s in db.get_active_suppressions() if s.get('target_type') == 'all']
|
||||
except Exception as e:
|
||||
logger.error(f'Reading maintenance state failed: {e}')
|
||||
active = []
|
||||
# Of several, the one that ends last decides when tickets resume; no expiry beats any
|
||||
active.sort(key=lambda s: (s.get('expires_at') is None, s.get('expires_at') or ''))
|
||||
return {'maintenance': active[-1] if active else None}
|
||||
|
||||
|
||||
# In-memory diagnostic job store { job_id: { status, result, created_at } }
|
||||
_diag_jobs: dict = {}
|
||||
_diag_lock = threading.Lock()
|
||||
@@ -261,6 +274,72 @@ def _annotate_suppressions(events: list, suppressions: list) -> None:
|
||||
ev['is_suppressed'] = db.check_suppressed(suppressions, 'host', name, detail)
|
||||
|
||||
|
||||
# The monitor refreshes the open ticket list every 15 minutes; older than this and
|
||||
# it has stopped working, so nothing is flagged rather than flagging on stale data.
|
||||
_OPEN_TICKETS_MAX_AGE_S = 2 * 3600
|
||||
|
||||
|
||||
def _stale_settings() -> tuple:
|
||||
"""(days an alert can stay active before it counts as stale,
|
||||
hours a ticket can stay open after its alert cleared)."""
|
||||
mon = _config().get('monitor', {})
|
||||
try:
|
||||
days = float(mon.get('stale_alert_days', 3))
|
||||
hours = int(mon.get('stale_ticket_hours', 24))
|
||||
except (TypeError, ValueError):
|
||||
days, hours = 3.0, 24
|
||||
return days, hours
|
||||
|
||||
|
||||
def _open_tickets() -> Optional[dict]:
|
||||
"""{ticket_id: status} of open Tinker tickets, or None when the list is missing or out of date."""
|
||||
try:
|
||||
raw = db.get_state('open_tickets')
|
||||
data = json.loads(raw) if raw else None
|
||||
if not data or time.time() - float(data.get('checked_at', 0)) > _OPEN_TICKETS_MAX_AGE_S:
|
||||
return None
|
||||
return {str(k): v for k, v in (data.get('tickets') or {}).items()}
|
||||
except Exception as e:
|
||||
logger.error(f'Reading open ticket list failed: {e}')
|
||||
return None
|
||||
|
||||
|
||||
def _annotate_staleness(events: list, open_tickets: Optional[dict]) -> None:
|
||||
"""Add stale_days (set when an alert has been active too long) and ticket_closed
|
||||
(its ticket was closed in Tinker while the alert is still active) to each event."""
|
||||
days, _ = _stale_settings()
|
||||
now = datetime.now(timezone.utc)
|
||||
for ev in events:
|
||||
ev['stale_days'] = None
|
||||
ev['ticket_closed'] = False
|
||||
try:
|
||||
first = datetime.fromisoformat(str(ev.get('first_seen', '')).replace('Z', '+00:00'))
|
||||
if first.tzinfo is None:
|
||||
first = first.replace(tzinfo=timezone.utc)
|
||||
age = (now - first).total_seconds() / 86400
|
||||
if age >= days:
|
||||
ev['stale_days'] = int(age)
|
||||
except ValueError:
|
||||
pass
|
||||
tid = ev.get('ticket_id')
|
||||
if tid and open_tickets is not None and str(tid) not in open_tickets:
|
||||
ev['ticket_closed'] = True
|
||||
|
||||
|
||||
def _stale_view(events: list) -> list:
|
||||
"""Flag stale active alerts in place; return tickets left open after their alerts cleared."""
|
||||
open_tickets = _open_tickets()
|
||||
_annotate_staleness(events, open_tickets)
|
||||
if open_tickets is None:
|
||||
return []
|
||||
_, hours = _stale_settings()
|
||||
try:
|
||||
return db.get_cleared_tickets(list(open_tickets), older_than_hours=hours)
|
||||
except Exception as e:
|
||||
logger.error(f'Finding tickets left open failed: {e}')
|
||||
return []
|
||||
|
||||
|
||||
def _state_json(key: str) -> dict:
|
||||
"""Return a monitor_state JSON blob as a dict, or {} if missing or unreadable."""
|
||||
raw = db.get_state(key)
|
||||
@@ -294,6 +373,7 @@ def index():
|
||||
snapshot = {}
|
||||
suppressions = db.get_active_suppressions()
|
||||
_annotate_suppressions(events, suppressions)
|
||||
stale_tickets = _stale_view(events)
|
||||
recent_resolved = db.get_recent_resolved(hours=24, limit=10)
|
||||
return render_template(
|
||||
'index.html',
|
||||
@@ -305,6 +385,8 @@ def index():
|
||||
last_check=last_check,
|
||||
suppressions=suppressions,
|
||||
recent_resolved=recent_resolved,
|
||||
stale_tickets=stale_tickets,
|
||||
stale_alerts=sum(1 for e in events if e.get('stale_days') is not None or e.get('ticket_closed')),
|
||||
daemon_ok=_daemon_ok(last_check),
|
||||
)
|
||||
|
||||
@@ -406,11 +488,13 @@ def api_status():
|
||||
active = db.get_active_events(limit=_PAGE_LIMIT)
|
||||
suppressions = db.get_active_suppressions()
|
||||
_annotate_suppressions(active, suppressions)
|
||||
stale_tickets = _stale_view(active)
|
||||
last_check = db.get_state('last_check', 'Never')
|
||||
return jsonify({
|
||||
'summary': db.get_status_summary(),
|
||||
'last_check': last_check,
|
||||
'events': active,
|
||||
'stale_tickets': stale_tickets,
|
||||
'total_active': db.count_active_events(),
|
||||
'daemon_ok': _daemon_ok(last_check),
|
||||
})
|
||||
|
||||
@@ -4,7 +4,7 @@ import logging
|
||||
import threading
|
||||
from contextlib import contextmanager
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Optional
|
||||
from typing import List, Optional
|
||||
|
||||
import pymysql
|
||||
import pymysql.cursors
|
||||
@@ -198,6 +198,35 @@ def set_ticket_id(event_id: int, ticket_id: str) -> None:
|
||||
)
|
||||
|
||||
|
||||
def get_incident_ticket(target_name: str, within_s: int) -> Optional[str]:
|
||||
"""The ticket of *target_name*'s newest event that is still open or ended within *within_s*."""
|
||||
with get_conn() as conn:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
"""SELECT ticket_id FROM network_events
|
||||
WHERE target_name=%s AND ticket_id IS NOT NULL AND ticket_id <> ''
|
||||
AND (resolved_at IS NULL OR resolved_at >= NOW() - INTERVAL %s SECOND)
|
||||
ORDER BY first_seen DESC, id DESC LIMIT 1""",
|
||||
(target_name, int(within_s)),
|
||||
)
|
||||
row = cur.fetchone()
|
||||
return row['ticket_id'] if row else None
|
||||
|
||||
|
||||
def get_ticket_ids_between(target_name: str, start: datetime, end: datetime) -> List[str]:
|
||||
"""Tickets of *target_name*'s events first seen between *start* and *end* (UTC), oldest first."""
|
||||
with get_conn() as conn:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
"""SELECT ticket_id, MIN(first_seen) AS first FROM network_events
|
||||
WHERE target_name=%s AND ticket_id IS NOT NULL AND ticket_id <> ''
|
||||
AND first_seen BETWEEN %s AND %s
|
||||
GROUP BY ticket_id ORDER BY first""",
|
||||
(target_name, start, end),
|
||||
)
|
||||
return [r['ticket_id'] for r in cur.fetchall()]
|
||||
|
||||
|
||||
def get_active_events(limit: int = 200, offset: int = 0) -> list:
|
||||
with get_conn() as conn:
|
||||
with conn.cursor() as cur:
|
||||
@@ -218,6 +247,49 @@ def get_active_events(limit: int = 200, offset: int = 0) -> list:
|
||||
return rows
|
||||
|
||||
|
||||
def get_cleared_tickets(open_ids: List[str], older_than_hours: int = 24) -> list:
|
||||
"""Of the given open tickets, those whose every alert cleared at least `older_than_hours` ago.
|
||||
|
||||
One row per ticket, describing its most recent alert, newest clearance first:
|
||||
{ticket_id, event_type, target_name, target_detail, severity, first_seen, cleared_at, events}
|
||||
"""
|
||||
ids = sorted({str(i) for i in open_ids if i})
|
||||
if not ids:
|
||||
return []
|
||||
placeholders = ','.join(['%s'] * len(ids))
|
||||
with get_conn() as conn:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
f"""SELECT ticket_id, event_type, target_name, target_detail, severity,
|
||||
first_seen, resolved_at
|
||||
FROM network_events WHERE ticket_id IN ({placeholders})
|
||||
ORDER BY first_seen""", # nosec B608: placeholders are only %s markers
|
||||
ids,
|
||||
)
|
||||
rows = cur.fetchall()
|
||||
cur.execute('SELECT NOW() AS now')
|
||||
now = cur.fetchone()['now']
|
||||
by_ticket: dict = {}
|
||||
for r in rows:
|
||||
by_ticket.setdefault(str(r['ticket_id']), []).append(r)
|
||||
out = []
|
||||
for tid, events in by_ticket.items():
|
||||
if any(e['resolved_at'] is None for e in events):
|
||||
continue
|
||||
cleared = max(e['resolved_at'] for e in events)
|
||||
if (now - cleared).total_seconds() < older_than_hours * 3600:
|
||||
continue
|
||||
last = events[-1]
|
||||
out.append({
|
||||
'ticket_id': tid, 'event_type': last['event_type'], 'target_name': last['target_name'],
|
||||
'target_detail': last['target_detail'], 'severity': last['severity'],
|
||||
'first_seen': events[0]['first_seen'].isoformat() + 'Z',
|
||||
'cleared_at': cleared.isoformat() + 'Z', 'events': len(events),
|
||||
})
|
||||
out.sort(key=lambda r: r['cleared_at'], reverse=True)
|
||||
return out
|
||||
|
||||
|
||||
def count_active_events() -> int:
|
||||
"""Return count of all unresolved events (for pagination)."""
|
||||
with get_conn() as conn:
|
||||
|
||||
@@ -242,6 +242,31 @@ LOG_WAN_RESTORED = 'NETWORK_WAN_RESTORED_2'
|
||||
LOG_WAN_UNSTABLE = 'NETWORK_WAN_FAILED_MULTIPLE_TIMES_2'
|
||||
LOG_MODEM_RESTARTED = 'SMART_POWER_MODEM_RESTARTED'
|
||||
WAN_LOG_KEYS = (LOG_WAN_FAILED, LOG_WAN_RESTORED, LOG_WAN_UNSTABLE, LOG_MODEM_RESTARTED)
|
||||
# An app or firmware update finishing. The UDM often drops its WAN while the Network app
|
||||
# updates, and Smart Power may then power-cycle a modem that was fine.
|
||||
LOG_NETWORK_UPDATED = 'NETWORK_UPDATED'
|
||||
LOG_FIRMWARE_UPDATED = 'FIRMWARE_UPDATED_V2'
|
||||
_UPDATE_KEY_RE = re.compile(r'[A-Z_]+_(?:UPDATED|UPGRADED)(?:_V\d+)?')
|
||||
|
||||
|
||||
def is_update_key(key: str) -> bool:
|
||||
"""True for log keys that mean an update finished (other app names follow the same pattern)."""
|
||||
return key in (LOG_NETWORK_UPDATED, LOG_FIRMWARE_UPDATED) or (
|
||||
bool(_UPDATE_KEY_RE.fullmatch(key or '')) and 'AVAILABLE' not in key
|
||||
and not key.startswith('MADE_CHANGES'))
|
||||
|
||||
|
||||
def wanted_log_key(key: str) -> bool:
|
||||
return key in WAN_LOG_KEYS or is_update_key(key)
|
||||
|
||||
|
||||
def update_label(event: dict) -> str:
|
||||
"""'UniFi Network 11.0.97 to 11.0.99' from an update log event."""
|
||||
what = event.get('app') or event.get('device') or 'UniFi'
|
||||
before, after = event.get('previous') or '', event.get('version') or ''
|
||||
if before and after:
|
||||
return f'{what} {before} to {after}'
|
||||
return f'{what} {after}'.strip()
|
||||
|
||||
_HISTORY_LIMIT = 50
|
||||
# The RPDU's log entry and the gateway's calculated boot time can differ by this much.
|
||||
|
||||
+199
-25
@@ -13,7 +13,8 @@ import re
|
||||
import shlex
|
||||
import statistics
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from contextlib import contextmanager
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Dict, List, Optional
|
||||
from urllib.parse import urljoin, urlparse
|
||||
|
||||
@@ -192,8 +193,12 @@ class UnifiClient:
|
||||
logger.error(f'UniFi client list error: {e}')
|
||||
return []
|
||||
|
||||
def get_system_events(self, since_ms: int, keys: tuple) -> Optional[List[dict]]:
|
||||
"""Return system log entries with the given keys, or None if the log is unavailable."""
|
||||
def get_system_events(self, since_ms: int, keys) -> Optional[List[dict]]:
|
||||
"""Return system log entries with the given keys, or None if the log is unavailable.
|
||||
|
||||
keys: a tuple of log keys, or a function that takes a key and says whether to keep it.
|
||||
"""
|
||||
wanted = keys if callable(keys) else keys.__contains__
|
||||
url = f'{self.base_url}/proxy/network/v2/api/site/{self.site_id}/system-log/all'
|
||||
found: List[dict] = []
|
||||
try:
|
||||
@@ -210,19 +215,29 @@ class UnifiClient:
|
||||
if not items:
|
||||
break
|
||||
for e in items:
|
||||
if e.get('key') in keys and e.get('timestamp', 0) >= since_ms:
|
||||
if wanted(e.get('key') or '') and e.get('timestamp', 0) >= since_ms:
|
||||
params = e.get('parameters') or {}
|
||||
found.append({
|
||||
'key': e['key'], 'ts': e['timestamp'] / 1000,
|
||||
'device': (params.get('DEVICE') or {}).get('name', ''),
|
||||
'wan': (params.get('WAN_ID') or {}).get('name', '')
|
||||
or str((params.get('WAN_ID') or {}).get('id', '')),
|
||||
**self._update_fields(params),
|
||||
})
|
||||
return found
|
||||
except Exception as e:
|
||||
logger.error(f'UniFi system log error: {e}')
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def _update_fields(params: dict) -> dict:
|
||||
"""The app and versions of an update entry; nothing for other entries."""
|
||||
def text(key):
|
||||
value = params.get(key)
|
||||
return str(value.get('name') or value.get('id') or '') if isinstance(value, dict) else str(value or '')
|
||||
fields = {'app': text('APPLICATION'), 'version': text('VERSION'), 'previous': text('PREVIOUS_VERSION')}
|
||||
return {k: v for k, v in fields.items() if v}
|
||||
|
||||
def get_devices(self) -> Optional[List[dict]]:
|
||||
"""Return list of UniFi devices, or None if the controller is unreachable."""
|
||||
try:
|
||||
@@ -393,6 +408,8 @@ class TicketClient:
|
||||
self.comment_url = cfg.get('comment_url') or (
|
||||
urljoin(self.url, 'api/ticket_comment_api.php') if self.url else '')
|
||||
self.comments_denied = False
|
||||
self.read_url = cfg.get('read_url') or (
|
||||
urljoin(self.url, 'api/tickets_api.php') if self.url else '')
|
||||
# Titles whose create matched a Closed ticket Tinker couldn't reopen: {title: (ticket_id, when)}
|
||||
self._closed_matches: Dict[str, tuple] = {}
|
||||
|
||||
@@ -427,6 +444,41 @@ class TicketClient:
|
||||
logger.error(f'Ticket comment failed for #{ticket_id}: {e}')
|
||||
return False
|
||||
|
||||
# Statuses that mean someone still has a ticket on their list
|
||||
OPEN_STATUSES = 'Open,Pending,In Progress'
|
||||
_OPEN_TICKETS_MAX_PAGES = 20
|
||||
|
||||
def open_tickets(self) -> Optional[Dict[str, str]]:
|
||||
"""{ticket_id: status} for every ticket not yet closed, or None if Tinker can't say.
|
||||
|
||||
Read only: this needs the key's read scope and never changes a ticket.
|
||||
"""
|
||||
if not self.api_key or not self.read_url:
|
||||
return None
|
||||
found: Dict[str, str] = {}
|
||||
try:
|
||||
for page in range(1, self._OPEN_TICKETS_MAX_PAGES + 1):
|
||||
resp = requests.get(
|
||||
self.read_url,
|
||||
params={'status': self.OPEN_STATUSES, 'limit': 100, 'page': page},
|
||||
headers={'Authorization': f'Bearer {self.api_key}'},
|
||||
timeout=15,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
if not data.get('success'):
|
||||
logger.warning(f'Unexpected ticket list response: {str(data)[:200]}')
|
||||
return None
|
||||
for t in data.get('tickets') or []:
|
||||
found[str(t.get('ticket_id'))] = t.get('status') or ''
|
||||
if page >= int(data.get('pages') or 1):
|
||||
return found
|
||||
logger.warning('Open ticket list is longer than expected; using the first pages only')
|
||||
return found
|
||||
except Exception as e:
|
||||
logger.error(f'Reading open tickets failed: {e}')
|
||||
return None
|
||||
|
||||
# How long to wait before asking again after a create matched an unreopenable Closed ticket.
|
||||
# Each ask adds a "recurred" comment to that ticket, so once a poll would pile them up.
|
||||
CLOSED_MATCH_RETRY_S = 24 * 3600
|
||||
@@ -965,12 +1017,25 @@ CLUSTER_NAME = 'proxmox-cluster'
|
||||
DEFAULT_PUBLIC_URL = 'https://gandalf.lotusguild.org'
|
||||
# Reports of something already over: the ticket is the whole story, no recovery note
|
||||
ONE_SHOT_EVENTS = ('modem_power_cycled', 'wan_unstable')
|
||||
# Modem and WAN reports this close together are one incident, so they share one ticket
|
||||
WAN_INCIDENT_WINDOW_S = 1800
|
||||
# UniFi logs an update when it finishes, after any WAN trouble it caused. So tickets opened
|
||||
# this long before the entry get a note, and new modem/WAN tickets wait this long after it.
|
||||
UPDATE_LOOKBACK_S = 1200
|
||||
UPDATE_HOLD_S = 900
|
||||
UNIFI_UPDATE_STATE = 'unifi_update'
|
||||
# Ports a host or device was last seen on, kept so a ticket for a dead host
|
||||
# (whose link, and so its LLDP entry, is gone) can still say where it plugs in
|
||||
KNOWN_UPLINKS_STATE = 'known_uplinks'
|
||||
# Which tickets are still open in Tinker, so the dashboard can flag ones left open
|
||||
# after their alert cleared. Refreshed this often; one list request per refresh.
|
||||
OPEN_TICKETS_STATE = 'open_tickets'
|
||||
OPEN_TICKETS_SYNC_S = 900
|
||||
|
||||
|
||||
class NetworkMonitor:
|
||||
_open_tickets_synced = 0.0
|
||||
|
||||
def __init__(self):
|
||||
with open('config.json') as f:
|
||||
self.cfg = json.load(f)
|
||||
@@ -1223,6 +1288,20 @@ class NetworkMonitor:
|
||||
elif self.tickets.comments_denied:
|
||||
break
|
||||
|
||||
def _sync_open_tickets(self) -> None:
|
||||
"""Save which tickets are still open in Tinker, at most every OPEN_TICKETS_SYNC_S.
|
||||
|
||||
On a failed read the previous list stays, with its old checked_at, so the
|
||||
dashboard can tell it is out of date rather than treat every ticket as closed.
|
||||
"""
|
||||
now = time.time()
|
||||
if now - self._open_tickets_synced < OPEN_TICKETS_SYNC_S:
|
||||
return
|
||||
self._open_tickets_synced = now
|
||||
found = self.tickets.open_tickets()
|
||||
if found is not None:
|
||||
db.set_state(OPEN_TICKETS_STATE, {'checked_at': now, 'tickets': found})
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Interface monitoring (Prometheus)
|
||||
# ------------------------------------------------------------------
|
||||
@@ -1543,14 +1622,95 @@ class NetworkMonitor:
|
||||
event_type, severity, 'modem', name, detail, f'{description} ({_now_utc()})',
|
||||
)
|
||||
if not sup and consec >= self.modem_fail_thresh and not ticket_id:
|
||||
self._open_ticket(
|
||||
event_id, event_type, name, detail,
|
||||
title=f'[{name}][auto][production][issue][network][cluster-wide] {heading}',
|
||||
heading='WAN Modem Alert',
|
||||
facts=[('Modem', f'{name} ({self.modem.base_url})'), ('Detected', _now_utc()),
|
||||
('Consecutive check failures', str(consec))],
|
||||
summary=description, checks=[advice], priority=priority,
|
||||
)
|
||||
self._queue_wan_ticket({
|
||||
'event_id': event_id, 'event_type': event_type, 'detail': detail,
|
||||
'title': f'[{name}][auto][production][issue][network][cluster-wide] {heading}',
|
||||
'name': heading, 'heading': 'WAN Modem Alert',
|
||||
'facts': [('Modem', f'{name} ({self.modem.base_url})'), ('Detected', _now_utc()),
|
||||
('Consecutive check failures', str(consec))],
|
||||
'summary': description, 'checks': [advice], 'priority': priority, 'recovers': True,
|
||||
})
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Modem and WAN tickets: one per incident, held around UniFi updates
|
||||
# ------------------------------------------------------------------
|
||||
@contextmanager
|
||||
def _wan_batch(self):
|
||||
"""Collect the cycle's modem and WAN tickets so related ones are opened together."""
|
||||
self._wan_pending = []
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
pending, self._wan_pending = self._wan_pending, None
|
||||
if pending:
|
||||
self._open_wan_tickets(pending)
|
||||
|
||||
def _queue_wan_ticket(self, item: dict) -> None:
|
||||
held = self._update_hold()
|
||||
if held:
|
||||
logger.info(f'Not opening a ticket for {item["event_type"]} yet: {held} just finished')
|
||||
return
|
||||
if getattr(self, '_wan_pending', None) is not None:
|
||||
self._wan_pending.append(item)
|
||||
else:
|
||||
self._open_wan_tickets([item])
|
||||
|
||||
def _update_hold(self) -> Optional[str]:
|
||||
"""The update that modem and WAN tickets are waiting out, if any."""
|
||||
raw = db.get_state(UNIFI_UPDATE_STATE)
|
||||
if not raw:
|
||||
return None
|
||||
try:
|
||||
update = json.loads(raw)
|
||||
except ValueError:
|
||||
return None
|
||||
if update.get('hold_until', 0) <= time.time():
|
||||
return None
|
||||
return update.get('label') or 'a UniFi update'
|
||||
|
||||
def _open_wan_tickets(self, items: List[dict]) -> None:
|
||||
"""Comment on the incident's ticket if one is recent, else open one ticket for all of *items*."""
|
||||
# Most urgent first; at equal priority an ongoing problem leads over a past event
|
||||
items = sorted(items, key=lambda i: (int(i['priority']), not i['recovers']))
|
||||
incident = db.get_incident_ticket(self.modem_name, WAN_INCIDENT_WINDOW_S)
|
||||
if incident:
|
||||
left = []
|
||||
for item in items:
|
||||
text = ticket_text.incident_comment(item['name'], item['summary'], item['checks'], _now_utc())
|
||||
if self.tickets.comment(incident, text):
|
||||
db.set_ticket_id(item['event_id'], incident)
|
||||
else:
|
||||
left.append(item) # comments failing must not lose the report
|
||||
items = left
|
||||
if not items:
|
||||
return
|
||||
first, others = items[0], items[1:]
|
||||
sections = []
|
||||
if first['recovers']:
|
||||
sections = self._ticket_sections(first['event_id'], first['event_type'], self.modem_name, first['detail'])
|
||||
if others:
|
||||
sections.append(('Also seen in this incident', [f'- **{o["name"]}:** {o["summary"]}' for o in others]))
|
||||
desc = ticket_text.description(first['heading'], first['facts'], first['summary'], first['checks'],
|
||||
sections, recovers=any(i['recovers'] for i in items))
|
||||
tid = self.tickets.create(first['title'], desc, priority=first['priority'])
|
||||
if tid:
|
||||
for item in items:
|
||||
db.set_ticket_id(item['event_id'], tid)
|
||||
|
||||
def _note_unifi_update(self, event: dict) -> None:
|
||||
"""Hold new modem/WAN tickets for a while and explain the ones opened just before."""
|
||||
label = modem_api.update_label(event)
|
||||
when = datetime.fromtimestamp(event['ts'], timezone.utc)
|
||||
logger.info(f'UniFi update finished: {label} at {when:%Y-%m-%d %H:%M:%S} UTC')
|
||||
if event['ts'] + UPDATE_HOLD_S > time.time():
|
||||
db.set_state(UNIFI_UPDATE_STATE, {'label': label, 'at': event['ts'],
|
||||
'hold_until': event['ts'] + UPDATE_HOLD_S})
|
||||
naive = when.replace(tzinfo=None)
|
||||
tickets = db.get_ticket_ids_between(
|
||||
self.modem_name, naive - timedelta(seconds=UPDATE_LOOKBACK_S), naive + timedelta(seconds=UPDATE_HOLD_S))
|
||||
for tid in tickets:
|
||||
self.tickets.comment(tid, ticket_text.update_comment(
|
||||
label, f'{when:%Y-%m-%d %H:%M} UTC', UPDATE_LOOKBACK_S // 60, UPDATE_HOLD_S // 60))
|
||||
|
||||
def _process_modem_failure(self, suppressions: list, error: Exception) -> None:
|
||||
"""Handle a failed poll. Link and optical events stay as they are: we can't see them."""
|
||||
@@ -1690,14 +1850,13 @@ class NetworkMonitor:
|
||||
name = self.modem_name
|
||||
event_id, _, _, ticket_id = db.upsert_event(event_type, 'warning', 'unifi', name, '', description)
|
||||
if not db.check_suppressed(suppressions, 'host', name) and not ticket_id:
|
||||
tid = self.tickets.create(
|
||||
f'[{name}][auto][production][issue][network][cluster-wide] {heading}',
|
||||
ticket_text.description('WAN Event', [('Modem', name), ('Reported', _now_utc())],
|
||||
description, [advice], recovers=False),
|
||||
priority=priority,
|
||||
)
|
||||
if tid:
|
||||
db.set_ticket_id(event_id, tid)
|
||||
self._queue_wan_ticket({
|
||||
'event_id': event_id, 'event_type': event_type, 'detail': '',
|
||||
'title': f'[{name}][auto][production][issue][network][cluster-wide] {heading}',
|
||||
'name': heading, 'heading': 'WAN Event',
|
||||
'facts': [('Modem', name), ('Reported', _now_utc())],
|
||||
'summary': description, 'checks': [advice], 'priority': priority, 'recovers': False,
|
||||
})
|
||||
db.resolve_event(event_type, name, '')
|
||||
|
||||
def _process_wan_log(self, suppressions: list) -> None:
|
||||
@@ -1713,12 +1872,21 @@ class NetworkMonitor:
|
||||
return
|
||||
# After long monitor downtime, don't open tickets for stale events.
|
||||
since_ms = max(int(raw_cursor) + 1, int((now - 86400) * 1000))
|
||||
events = self.unifi.get_system_events(since_ms, modem_api.WAN_LOG_KEYS)
|
||||
events = self.unifi.get_system_events(since_ms, modem_api.wanted_log_key)
|
||||
if not events:
|
||||
return # None = log unavailable, so the cursor stays put and it's retried
|
||||
|
||||
# Updates first: one in this batch explains (and holds) the WAN events beside it
|
||||
updates = [e for e in events if modem_api.is_update_key(e['key'])]
|
||||
for e in sorted(updates, key=lambda ev: ev['ts']):
|
||||
try:
|
||||
self._note_unifi_update(e)
|
||||
except Exception as ex:
|
||||
logger.error(f'Noting UniFi update failed: {ex}', exc_info=True)
|
||||
|
||||
history = self._load_wan_history()
|
||||
actionable = modem_api.apply_wan_events(history, events)
|
||||
actionable = modem_api.apply_wan_events(
|
||||
history, [e for e in events if not modem_api.is_update_key(e['key'])])
|
||||
|
||||
# UniFi repeats "failed multiple times" per failure, so ticket it once a day at most.
|
||||
unstable = [e for e in actionable if e['key'] == modem_api.LOG_WAN_UNSTABLE]
|
||||
@@ -1927,9 +2095,11 @@ class NetworkMonitor:
|
||||
# WAN modem: isolated so a failure here can't stop the rest of the cycle
|
||||
if self.modem:
|
||||
try:
|
||||
# Log first, so a restart it reports can explain an uptime reset below
|
||||
self._process_wan_log(suppressions)
|
||||
self._collect_modem(suppressions)
|
||||
# Log first, so a restart it reports can explain an uptime reset below.
|
||||
# Tickets open when the batch ends, so one incident gets one ticket.
|
||||
with self._wan_batch():
|
||||
self._process_wan_log(suppressions)
|
||||
self._collect_modem(suppressions)
|
||||
except Exception as e:
|
||||
logger.error(f'Modem collection failed: {e}', exc_info=True)
|
||||
|
||||
@@ -1938,6 +2108,10 @@ class NetworkMonitor:
|
||||
self._post_recovery_notes()
|
||||
except Exception as e:
|
||||
logger.error(f'Posting recovery notes failed: {e}', exc_info=True)
|
||||
try:
|
||||
self._sync_open_tickets()
|
||||
except Exception as e:
|
||||
logger.error(f'Syncing open tickets failed: {e}', exc_info=True)
|
||||
|
||||
# Housekeeping: deactivate expired suppressions and purge old resolved events
|
||||
db.cleanup_expired_suppressions()
|
||||
|
||||
+108
-4
@@ -55,7 +55,10 @@ async function refreshAll() {
|
||||
if (statusResult.status === 'fulfilled') {
|
||||
const status = statusResult.value;
|
||||
updateEventsTable(status.events || [], status.total_active);
|
||||
updateStatusBar(status.summary || {}, status.last_check || '', status.daemon_ok);
|
||||
updateFollowUp(status.stale_tickets || []);
|
||||
updateStatusBar(status.summary || {}, status.last_check || '', status.daemon_ok,
|
||||
followUpCount(status.events || [], status.stale_tickets || []),
|
||||
(status.stale_tickets || []).length > 0);
|
||||
if (typeof window.gandalfNotifUpdate === 'function') window.gandalfNotifUpdate(status.events || []);
|
||||
} else {
|
||||
showToast('Status data unavailable', 'warning');
|
||||
@@ -66,7 +69,14 @@ async function refreshAll() {
|
||||
}
|
||||
}
|
||||
|
||||
function updateStatusBar(summary, lastCheck, daemonOk) {
|
||||
// Active alerts flagged stale or with a closed ticket, plus tickets left open after clearing
|
||||
function followUpCount(events, staleTickets) {
|
||||
const flagged = events.filter(e => e.severity !== 'info' &&
|
||||
((e.stale_days !== null && e.stale_days !== undefined) || e.ticket_closed)).length;
|
||||
return flagged + staleTickets.length;
|
||||
}
|
||||
|
||||
function updateStatusBar(summary, lastCheck, daemonOk, followUps = 0, hasStaleTickets = false) {
|
||||
const bar = document.querySelector('.status-chips');
|
||||
if (!bar) return;
|
||||
const chips = [];
|
||||
@@ -74,6 +84,10 @@ function updateStatusBar(summary, lastCheck, daemonOk) {
|
||||
if (summary.critical) chips.push(`<span class="chip chip-critical">● ${summary.critical} CRITICAL</span>`);
|
||||
if (summary.warning) chips.push(`<span class="chip chip-warning">● ${summary.warning} WARNING</span>`);
|
||||
if (!summary.critical && !summary.warning && daemonOk !== false) chips.push('<span class="chip chip-ok">✔ ALL SYSTEMS NOMINAL</span>');
|
||||
if (followUps) {
|
||||
const href = hasStaleTickets ? '#followup' : '#events-table-wrap';
|
||||
chips.push(`<a class="chip chip-warning chip-followup" href="${href}">⏳ ${followUps} TO FOLLOW UP</a>`);
|
||||
}
|
||||
bar.innerHTML = chips.join('');
|
||||
|
||||
const lc = document.getElementById('last-check');
|
||||
@@ -469,7 +483,7 @@ function updateEventsTable(events, totalActive) {
|
||||
|
||||
const truncated = totalActive !== null && totalActive !== undefined && totalActive > active.length;
|
||||
const countNotice = truncated
|
||||
? `<div class="pagination-notice">Showing ${active.length} of ${totalActive} active alerts — <a href="/api/events?limit=1000">view all via API</a></div>`
|
||||
? `<div class="pagination-notice">Showing ${active.length} of ${totalActive} active alerts: <a href="/api/events?limit=1000">view all via API</a></div>`
|
||||
: '';
|
||||
|
||||
const rows = active.map(e => {
|
||||
@@ -485,9 +499,15 @@ function updateEventsTable(events, totalActive) {
|
||||
const supBadge = e.is_suppressed
|
||||
? `<span class="lt-badge badge-suppressed" title="Alert suppressed">🔕 sup</span>`
|
||||
: '';
|
||||
const staleBadge = (e.stale_days !== null && e.stale_days !== undefined)
|
||||
? `<span class="lt-badge badge-stale" title="Active for ${Number(e.stale_days)} days with no fix yet">⏳ ${Number(e.stale_days)}d</span>`
|
||||
: '';
|
||||
const closedBadge = e.ticket_closed
|
||||
? `<span class="lt-badge badge-ticket-closed" title="Ticket #${lt.escHtml(String(e.ticket_id))} was closed in Tinker, but this alert is still active">ticket closed</span>`
|
||||
: '';
|
||||
return `
|
||||
<tr class="row-${e.severity}${e.is_suppressed ? ' row-suppressed' : ''}">
|
||||
<td><span class="lt-badge badge-${e.severity}">${e.severity}</span>${supBadge}</td>
|
||||
<td><span class="lt-badge badge-${e.severity}">${e.severity}</span>${supBadge}${staleBadge}${closedBadge}</td>
|
||||
<td>${lt.escHtml(e.event_type.replace(/_/g,' '))}</td>
|
||||
<td><strong>${lt.escHtml(e.target_name)}</strong></td>
|
||||
<td>${lt.escHtml(e.target_detail || '-')}</td>
|
||||
@@ -522,6 +542,28 @@ function updateEventsTable(events, totalActive) {
|
||||
</div>`;
|
||||
}
|
||||
|
||||
// ── Needs follow-up: tickets left open after their alert cleared ──────
|
||||
function updateFollowUp(tickets) {
|
||||
const section = document.getElementById('followup');
|
||||
const tbody = document.querySelector('#followup-table tbody');
|
||||
if (!section || !tbody) return;
|
||||
section.hidden = !tickets.length;
|
||||
const badge = document.getElementById('followup-badge');
|
||||
if (badge) badge.textContent = tickets.length;
|
||||
const ticketBase = (typeof GANDALF_CONFIG !== 'undefined' && GANDALF_CONFIG.ticket_web_url)
|
||||
? GANDALF_CONFIG.ticket_web_url : 'http://t.lotusguild.org/ticket/';
|
||||
tbody.innerHTML = tickets.map(t => `
|
||||
<tr>
|
||||
<td><a href="${lt.escHtml(ticketBase)}${lt.escHtml(String(t.ticket_id))}" target="_blank" rel="noopener"
|
||||
class="ticket-link">#${lt.escHtml(String(t.ticket_id))}</a></td>
|
||||
<td><strong>${lt.escHtml(t.target_name || '')}</strong></td>
|
||||
<td>${lt.escHtml((t.event_type || '').replace(/_/g, ' '))}${t.target_detail
|
||||
? ` <span class="lt-text-muted">${lt.escHtml(t.target_detail)}</span>` : ''}</td>
|
||||
<td class="ts-cell" title="${lt.escHtml(t.cleared_at || '')}">${lt.time.ago(_toIso(t.cleared_at))}</td>
|
||||
<td>${Number(t.events) || 0}</td>
|
||||
</tr>`).join('');
|
||||
}
|
||||
|
||||
// ── Suppression modal ─────────────────────────────────────────────────
|
||||
function openSuppressModal(type, name, detail) {
|
||||
const modal = document.getElementById('suppress-modal');
|
||||
@@ -648,3 +690,65 @@ document.addEventListener('click', e => {
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
// ── Maintenance: pause every ticket for a while (a global suppression with an expiry) ──
|
||||
(function () {
|
||||
const until = document.querySelector('.maint-banner .maint-until');
|
||||
if (until) {
|
||||
const t = new Date(until.getAttribute('datetime'));
|
||||
if (!isNaN(t)) {
|
||||
const sameDay = t.toDateString() === new Date().toDateString();
|
||||
until.textContent = sameDay
|
||||
? t.toLocaleTimeString([], { hour: 'numeric', minute: '2-digit' })
|
||||
: t.toLocaleString([], { weekday: 'short', hour: 'numeric', minute: '2-digit' });
|
||||
until.title = t.toLocaleString();
|
||||
}
|
||||
}
|
||||
|
||||
let minutes = 60;
|
||||
document.addEventListener('click', async e => {
|
||||
if (e.target.closest('[data-action="start-maintenance"]')) {
|
||||
lt.modal.open('maint-modal');
|
||||
document.getElementById('maint-reason')?.focus();
|
||||
return;
|
||||
}
|
||||
const pill = e.target.closest('.maint-pill');
|
||||
if (pill) {
|
||||
minutes = parseInt(pill.dataset.minutes, 10);
|
||||
document.querySelectorAll('.maint-pill').forEach(p => {
|
||||
const on = p === pill;
|
||||
p.classList.toggle('active', on);
|
||||
p.setAttribute('aria-pressed', String(on));
|
||||
});
|
||||
return;
|
||||
}
|
||||
const end = e.target.closest('[data-action="end-maintenance"]');
|
||||
if (end) {
|
||||
end.disabled = true;
|
||||
try {
|
||||
await lt.api.delete(`/api/suppressions/${encodeURIComponent(end.dataset.supId)}`);
|
||||
window.location.reload();
|
||||
} catch (err) {
|
||||
end.disabled = false;
|
||||
showToast(err.message || 'Could not end maintenance', 'error');
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
document.getElementById('maint-form')?.addEventListener('submit', async e => {
|
||||
e.preventDefault();
|
||||
const reason = (document.getElementById('maint-reason').value || '').trim();
|
||||
if (!reason) { showToast('Reason is required', 'error'); return; }
|
||||
const submit = e.target.querySelector('[type="submit"]');
|
||||
submit.disabled = true;
|
||||
try {
|
||||
await lt.api.post('/api/suppressions', {
|
||||
target_type: 'all', target_name: '', reason, expires_minutes: minutes,
|
||||
});
|
||||
window.location.reload();
|
||||
} catch (err) {
|
||||
submit.disabled = false;
|
||||
showToast(err.message || 'Could not start maintenance', 'error');
|
||||
}
|
||||
});
|
||||
})();
|
||||
@@ -229,6 +229,12 @@
|
||||
.badge-resolved { color: var(--text-muted); border-color: var(--border-color); text-decoration: line-through; }
|
||||
.badge-suppressed { font-size: .9em; padding: 0; border: none; color: var(--text-muted); }
|
||||
.badge-purple { color: var(--accent-purple); border-color: var(--accent-purple); }
|
||||
.badge-stale { color: var(--amber, var(--orange)); border-color: transparent; margin-left: 4px; white-space: nowrap; }
|
||||
.badge-ticket-closed { color: var(--accent-purple); border-color: var(--accent-purple); margin-left: 4px; white-space: nowrap; }
|
||||
.chip-followup { text-decoration: none; cursor: pointer; }
|
||||
.chip-followup:hover, .chip-followup:focus-visible { background: rgba(255,107,0,.12); }
|
||||
.followup-intro { color: var(--text-muted); font-size: .85em; margin: 0 0 8px; }
|
||||
#followup-table td { white-space: nowrap; }
|
||||
|
||||
/* ── Table row state colors ───────────────────────────────────────── */
|
||||
.lt-table tr.row-critical td { background: rgba(255,45,85,.04); }
|
||||
@@ -383,6 +389,22 @@
|
||||
.pill:hover { border-color: var(--green); color: var(--green); background: var(--green-dim); }
|
||||
.pill.active,
|
||||
.pill-manual.active { border-color: var(--amber); color: var(--amber); background: var(--amber-dim); text-shadow: var(--glow-amber); }
|
||||
/* Maintenance modal pills: same look, own class so the suppress modal's handler skips them */
|
||||
.maint-pill { padding: 3px 10px; border: 1px solid var(--border-color); background: transparent;
|
||||
font-family: var(--font); font-size: .72em; font-weight: bold; cursor: pointer;
|
||||
color: var(--text-muted); transition: all .15s; letter-spacing: .04em; }
|
||||
.maint-pill:hover { border-color: var(--green); color: var(--green); background: var(--green-dim); }
|
||||
.maint-pill.active { border-color: var(--amber); color: var(--amber); background: var(--amber-dim); text-shadow: var(--glow-amber); }
|
||||
.maint-intro { margin: 0 0 12px; }
|
||||
|
||||
/* Maintenance banner: every page while a global suppression is active */
|
||||
.maint-banner {
|
||||
display: flex; align-items: center; justify-content: space-between; gap: 12px; flex-wrap: wrap;
|
||||
margin-bottom: 16px; padding: 8px 14px; border: 1px solid var(--amber); background: var(--amber-dim);
|
||||
color: var(--amber); font-size: .82em;
|
||||
}
|
||||
.maint-banner strong { letter-spacing: .06em; text-transform: uppercase; }
|
||||
.maint-reason { color: var(--text-muted); }
|
||||
|
||||
/* ── Available targets grid (suppressions page) ───────────────────── */
|
||||
.targets-grid { display: grid; grid-template-columns: repeat(auto-fill, minmax(180px, 1fr)); gap: 10px; }
|
||||
|
||||
+57
-1
@@ -125,6 +125,10 @@
|
||||
<span class="lt-header-user">{{ _uname }}</span>
|
||||
{% if user.groups and 'admin' in user.groups %}
|
||||
<span class="lt-badge lt-badge-admin" aria-label="Administrator">ADMIN</span>
|
||||
{% if not maintenance %}
|
||||
<button type="button" class="lt-btn lt-btn-ghost lt-btn-sm" data-action="start-maintenance"
|
||||
title="Pause all tickets while you work on the network">Maintenance</button>
|
||||
{% endif %}
|
||||
{% endif %}
|
||||
|
||||
<!-- Notification bell: shows active monitoring alerts -->
|
||||
@@ -184,6 +188,20 @@
|
||||
|
||||
<!-- MAIN CONTENT -->
|
||||
<main class="lt-main lt-container" id="main-content">
|
||||
{% if maintenance %}
|
||||
<div class="maint-banner" role="status">
|
||||
<span class="maint-banner-text">
|
||||
<strong>Maintenance:</strong> Gandalf is not opening tickets
|
||||
{% if maintenance.expires_at %}until <time class="maint-until" datetime="{{ maintenance.expires_at }}">{{ maintenance.expires_at }}</time>{% else %}until it is ended{% endif %}.
|
||||
Alerts are still recorded, and anything still failing afterwards gets a ticket.
|
||||
<span class="maint-reason">{{ maintenance.reason }}{% if maintenance.suppressed_by %} ({{ maintenance.suppressed_by }}){% endif %}</span>
|
||||
</span>
|
||||
{% if user.groups and 'admin' in user.groups %}
|
||||
<button type="button" class="lt-btn lt-btn-secondary lt-btn-sm" data-action="end-maintenance"
|
||||
data-sup-id="{{ maintenance.id }}">End now</button>
|
||||
{% endif %}
|
||||
</div>
|
||||
{% endif %}
|
||||
{% block content %}{% endblock %}
|
||||
</main>
|
||||
|
||||
@@ -203,9 +221,47 @@
|
||||
<span class="lt-footer-sep">|</span>
|
||||
<button type="button" class="lt-footer-hint" data-action="show-keyboard-help"><span class="lt-footer-key">[ ? ]</span> HELP</button>
|
||||
</nav>
|
||||
<span>GANDALF — TDS v1.2</span>
|
||||
<span>GANDALF | TDS v1.2</span>
|
||||
</footer>
|
||||
|
||||
{% if user.groups and 'admin' in user.groups and not maintenance %}
|
||||
<!-- MAINTENANCE MODAL: pause every ticket for a while -->
|
||||
<div id="maint-modal" class="lt-modal-overlay"
|
||||
role="dialog" aria-modal="true" aria-labelledby="maint-modal-title" aria-hidden="true">
|
||||
<div class="lt-modal">
|
||||
<div class="lt-modal-header">
|
||||
<h3 class="lt-modal-title" id="maint-modal-title">Start maintenance</h3>
|
||||
<button type="button" class="lt-modal-close" data-modal-close aria-label="Close">✕</button>
|
||||
</div>
|
||||
<form id="maint-form">
|
||||
<div class="lt-modal-body">
|
||||
<p class="lt-field-hint maint-intro">Gandalf keeps checking and recording alerts but opens no tickets
|
||||
until this ends. Anything still failing afterwards gets a ticket as usual.</p>
|
||||
<div class="lt-form-group">
|
||||
<label class="lt-label" for="maint-reason">Reason <span class="required">*</span></label>
|
||||
<input type="text" class="lt-input" id="maint-reason" name="reason" maxlength="500"
|
||||
value="Maintenance" required aria-required="true"
|
||||
placeholder="e.g. UniFi Network update">
|
||||
</div>
|
||||
<div class="lt-form-group lt-form-group--last">
|
||||
<span class="lt-label" id="maint-len-label">How long</span>
|
||||
<div class="duration-pills" role="group" aria-labelledby="maint-len-label">
|
||||
<button type="button" class="maint-pill" data-minutes="30" aria-pressed="false">30 min</button>
|
||||
<button type="button" class="maint-pill active" data-minutes="60" aria-pressed="true">1 hr</button>
|
||||
<button type="button" class="maint-pill" data-minutes="120" aria-pressed="false">2 hr</button>
|
||||
<button type="button" class="maint-pill" data-minutes="240" aria-pressed="false">4 hr</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="lt-modal-footer">
|
||||
<button type="button" class="lt-btn lt-btn-secondary" data-modal-close>Cancel</button>
|
||||
<button type="submit" class="lt-btn lt-btn-primary">Start</button>
|
||||
</div>
|
||||
</form>
|
||||
</div>
|
||||
</div>
|
||||
{% endif %}
|
||||
|
||||
<!-- QUICK-SUPPRESS MODAL: available on all pages via [S] shortcut -->
|
||||
<div id="suppress-modal" class="lt-modal-overlay"
|
||||
role="dialog" aria-modal="true" aria-labelledby="suppress-modal-title" aria-hidden="true">
|
||||
|
||||
@@ -18,6 +18,10 @@
|
||||
{% if daemon_ok and not summary.critical and not summary.warning %}
|
||||
<span class="chip chip-ok">✔ ALL SYSTEMS NOMINAL</span>
|
||||
{% endif %}
|
||||
{% set followups = (stale_tickets | length) + (stale_alerts or 0) %}
|
||||
{% if followups %}
|
||||
<a class="chip chip-warning chip-followup" href="{{ '#followup' if stale_tickets else '#events-table-wrap' }}">⏳ {{ followups }} TO FOLLOW UP</a>
|
||||
{% endif %}
|
||||
</div>
|
||||
<div class="status-meta">
|
||||
<span class="last-check" id="last-check" data-tooltip="Last monitor poll timestamp" data-tooltip-pos="bottom">{{ last_check }}</span>
|
||||
@@ -129,6 +133,8 @@
|
||||
<td>
|
||||
<span class="lt-badge badge-{{ e.severity }}">{{ e.severity }}</span>
|
||||
{% if e.is_suppressed %}<span class="lt-badge badge-suppressed" title="Alert suppressed">🔕 sup</span>{% endif %}
|
||||
{% if e.stale_days is not none %}<span class="lt-badge badge-stale" title="Active for {{ e.stale_days }} days with no fix yet">⏳ {{ e.stale_days }}d</span>{% endif %}
|
||||
{% if e.ticket_closed %}<span class="lt-badge badge-ticket-closed" title="Ticket #{{ e.ticket_id }} was closed in Tinker, but this alert is still active">ticket closed</span>{% endif %}
|
||||
</td>
|
||||
<td>{{ e.event_type | replace('_', ' ') }}</td>
|
||||
<td><strong>{{ e.target_name }}</strong></td>
|
||||
@@ -177,6 +183,39 @@
|
||||
</div><!-- /.lt-frame -->
|
||||
</section>
|
||||
|
||||
<!-- ── Needs follow-up: tickets left open after their alert cleared ── -->
|
||||
<section class="g-section" id="followup"{% if not stale_tickets %} hidden{% endif %}>
|
||||
<div class="g-section-header">
|
||||
<h2 class="g-section-title">Needs Follow-up</h2>
|
||||
<span class="g-section-badge" id="followup-badge">{{ stale_tickets | length }}</span>
|
||||
</div>
|
||||
<p class="followup-intro">These tickets are still open in Tinker, but their alert cleared over a day ago. Close them if the problem is fixed.</p>
|
||||
<div class="lt-frame">
|
||||
<span class="lt-frame-bl">╚</span>
|
||||
<span class="lt-frame-br">╝</span>
|
||||
<div class="lt-section-header">Tickets Left Open</div>
|
||||
<div class="lt-table-wrap">
|
||||
<table class="lt-table" id="followup-table">
|
||||
<caption class="lt-sr-only">Tickets still open after their alert cleared</caption>
|
||||
<thead>
|
||||
<tr><th>Ticket</th><th>Target</th><th>Alert</th><th>Cleared</th><th>Alerts on ticket</th></tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for t in stale_tickets %}
|
||||
<tr>
|
||||
<td><a href="{{ config.ticket_api.web_url }}{{ t.ticket_id }}" target="_blank" rel="noopener" class="ticket-link">#{{ t.ticket_id }}</a></td>
|
||||
<td><strong>{{ t.target_name }}</strong></td>
|
||||
<td>{{ t.event_type | replace('_', ' ') }}{% if t.target_detail %} <span class="lt-text-muted">{{ t.target_detail }}</span>{% endif %}</td>
|
||||
<td class="ts-cell" title="{{ t.cleared_at }}"><span class="event-age" data-ts="{{ t.cleared_at }}">{{ t.cleared_at }}</span></td>
|
||||
<td>{{ t.events }}</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- ── WAN / fiber ──────────────────────────────────────────────────── -->
|
||||
{% include "_modem_panel.html" %}
|
||||
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
"""Tests for app.py routes: database and LDAP are mocked."""
|
||||
import json
|
||||
import time
|
||||
from datetime import datetime, timedelta, timezone
|
||||
import os
|
||||
import sys
|
||||
from unittest.mock import patch
|
||||
@@ -610,3 +612,94 @@ class TestUnusedInterfaces:
|
||||
gandalf.db.get_unused_interfaces.return_value = {'large1': {'enp36s0': {'reason': 'spare'}}}
|
||||
data = client.get('/api/device/large1', headers=AUTH).get_json()
|
||||
assert data['unused_interfaces'] == {'enp36s0': {'reason': 'spare'}}
|
||||
|
||||
|
||||
class TestMaintenance:
|
||||
SUP = {'id': 7, 'target_type': 'all', 'target_name': '', 'reason': 'UniFi Network update',
|
||||
'suppressed_by': 'jared', 'expires_at': '2026-11-12T20:00:00Z'}
|
||||
|
||||
def test_no_banner_and_a_start_button_by_default(self, client):
|
||||
html = client.get('/', headers=AUTH).get_data(as_text=True)
|
||||
assert 'maint-banner' not in html
|
||||
assert 'data-action="start-maintenance"' in html and 'id="maint-modal"' in html
|
||||
|
||||
def test_banner_on_every_page_while_on(self, client):
|
||||
gandalf.db.get_active_suppressions.return_value = [
|
||||
{'id': 3, 'target_type': 'host', 'target_name': 'large1', 'reason': 'x', 'expires_at': None},
|
||||
self.SUP,
|
||||
]
|
||||
for page in ('/', '/events', '/rack'):
|
||||
html = client.get(page, headers=AUTH).get_data(as_text=True)
|
||||
assert 'class="maint-banner"' in html, page
|
||||
assert 'datetime="2026-11-12T20:00:00Z"' in html and 'UniFi Network update' in html
|
||||
assert 'data-sup-id="7"' in html and 'data-action="start-maintenance"' not in html
|
||||
|
||||
def test_open_ended_maintenance_wins(self, client):
|
||||
gandalf.db.get_active_suppressions.return_value = [
|
||||
self.SUP, dict(self.SUP, id=9, expires_at=None, reason='Rewiring')]
|
||||
html = client.get('/', headers=AUTH).get_data(as_text=True)
|
||||
assert 'until it is ended' in html and 'data-sup-id="9"' in html
|
||||
|
||||
def test_viewers_see_the_banner_without_buttons(self, client):
|
||||
gandalf.db.get_active_suppressions.return_value = [self.SUP]
|
||||
viewer = dict(AUTH, **{'Remote-Groups': 'staff'})
|
||||
with patch.object(gandalf, '_cfg', dict(CONFIG, auth=dict(CONFIG['auth'], allowed_groups=['admin', 'staff']))):
|
||||
html = client.get('/', headers=viewer).get_data(as_text=True)
|
||||
assert 'class="maint-banner"' in html and 'data-action="end-maintenance"' not in html
|
||||
|
||||
def test_a_db_error_means_no_banner(self, client):
|
||||
gandalf.db.get_active_suppressions.side_effect = RuntimeError('db down')
|
||||
assert gandalf.inject_maintenance() == {'maintenance': None}
|
||||
|
||||
|
||||
class TestFollowUp:
|
||||
def _state(self, tickets, age_s=60):
|
||||
state = dict(STATE, open_tickets=json.dumps({'checked_at': time.time() - age_s, 'tickets': tickets}))
|
||||
gandalf.db.get_state.side_effect = lambda key, default=None: state.get(key, default)
|
||||
|
||||
def _event(self, days_old, ticket_id=None):
|
||||
first = (datetime.now(timezone.utc) - timedelta(days=days_old)).strftime('%Y-%m-%dT%H:%M:%SZ')
|
||||
return {'id': 1, 'event_type': 'interface_down', 'severity': 'critical', 'target_name': 'micro1',
|
||||
'target_detail': 'enp3s0', 'description': 'down', 'first_seen': first, 'last_seen': first,
|
||||
'consecutive_failures': 3, 'ticket_id': ticket_id}
|
||||
|
||||
def test_old_alert_is_stale_and_closed_ticket_is_flagged(self, client):
|
||||
self._state({'9': 'Open'})
|
||||
gandalf.db.get_active_events.return_value = [self._event(5, '8'), self._event(1, '9')]
|
||||
gandalf.db.get_cleared_tickets.return_value = []
|
||||
events = client.get('/api/status', headers=AUTH).get_json()['events']
|
||||
assert events[0]['stale_days'] == 5 and events[0]['ticket_closed'] is True
|
||||
assert events[1]['stale_days'] is None and events[1]['ticket_closed'] is False
|
||||
|
||||
def test_threshold_comes_from_config(self, client):
|
||||
self._state({})
|
||||
gandalf.db.get_active_events.return_value = [self._event(1.5)]
|
||||
gandalf.db.get_cleared_tickets.return_value = []
|
||||
cfg = dict(CONFIG, monitor={'stale_alert_days': 1})
|
||||
with patch.object(gandalf, '_cfg', cfg):
|
||||
events = client.get('/api/status', headers=AUTH).get_json()['events']
|
||||
assert events[0]['stale_days'] == 1
|
||||
|
||||
def test_tickets_left_open_are_listed(self, client):
|
||||
self._state({'281331508': 'Open'})
|
||||
left = [{'ticket_id': '281331508', 'event_type': 'host_unreachable', 'target_name': 'pbs',
|
||||
'target_detail': '', 'severity': 'critical', 'first_seen': '2026-10-14T12:02:05Z',
|
||||
'cleared_at': '2026-11-04T22:12:36Z', 'events': 3}]
|
||||
gandalf.db.get_cleared_tickets.return_value = left
|
||||
assert client.get('/api/status', headers=AUTH).get_json()['stale_tickets'] == left
|
||||
gandalf.db.get_cleared_tickets.assert_called_with(['281331508'], older_than_hours=24)
|
||||
html = client.get('/', headers=AUTH).get_data(as_text=True)
|
||||
assert '#281331508' in html and 'TO FOLLOW UP' in html
|
||||
assert '<section class="g-section" id="followup">' in html
|
||||
|
||||
def test_out_of_date_list_flags_nothing(self, client):
|
||||
self._state({}, age_s=3 * 3600)
|
||||
gandalf.db.get_active_events.return_value = [self._event(1, '8')]
|
||||
body = client.get('/api/status', headers=AUTH).get_json()
|
||||
assert body['stale_tickets'] == [] and body['events'][0]['ticket_closed'] is False
|
||||
gandalf.db.get_cleared_tickets.assert_not_called()
|
||||
|
||||
def test_no_list_yet_hides_the_panel(self, client):
|
||||
html = client.get('/', headers=AUTH).get_data(as_text=True)
|
||||
assert '<section class="g-section" id="followup" hidden>' in html
|
||||
assert 'TO FOLLOW UP' not in html
|
||||
@@ -0,0 +1,64 @@
|
||||
"""Tests for db.py query helpers that do work beyond the SQL: the connection is faked."""
|
||||
import os
|
||||
import sys
|
||||
from contextlib import contextmanager
|
||||
from datetime import datetime, timedelta
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
|
||||
|
||||
import db # noqa: E402
|
||||
|
||||
NOW = datetime(2026, 11, 14, 12, 0, 0)
|
||||
|
||||
|
||||
def _fake_conn(rows):
|
||||
cur = MagicMock()
|
||||
cur.fetchall.return_value = rows
|
||||
cur.fetchone.return_value = {'now': NOW}
|
||||
conn = MagicMock()
|
||||
conn.cursor.return_value.__enter__.return_value = cur
|
||||
|
||||
@contextmanager
|
||||
def get_conn():
|
||||
yield conn
|
||||
return patch.object(db, 'get_conn', get_conn), cur
|
||||
|
||||
|
||||
def _event(tid, hours_ago_first, hours_ago_resolved, etype='host_unreachable', target='pbs'):
|
||||
return {
|
||||
'ticket_id': tid, 'event_type': etype, 'target_name': target, 'target_detail': '',
|
||||
'severity': 'critical', 'first_seen': NOW - timedelta(hours=hours_ago_first),
|
||||
'resolved_at': None if hours_ago_resolved is None else NOW - timedelta(hours=hours_ago_resolved),
|
||||
}
|
||||
|
||||
|
||||
class TestGetClearedTickets:
|
||||
def test_no_ids_skips_the_query(self):
|
||||
patcher, cur = _fake_conn([])
|
||||
with patcher:
|
||||
assert db.get_cleared_tickets([]) == []
|
||||
cur.execute.assert_not_called()
|
||||
|
||||
def test_only_tickets_whose_every_alert_cleared_long_enough_ago(self):
|
||||
rows = [
|
||||
_event('1', 300, 250), _event('1', 100, 48), # two outages, both long cleared
|
||||
_event('2', 50, 40), _event('2', 5, None), # still active
|
||||
_event('3', 10, 2), # cleared too recently
|
||||
]
|
||||
patcher, cur = _fake_conn(rows)
|
||||
with patcher:
|
||||
out = db.get_cleared_tickets(['3', '1', '2', '1'], older_than_hours=24)
|
||||
assert [r['ticket_id'] for r in out] == ['1']
|
||||
assert out[0]['events'] == 2
|
||||
assert out[0]['cleared_at'] == (NOW - timedelta(hours=48)).isoformat() + 'Z'
|
||||
assert out[0]['first_seen'] == (NOW - timedelta(hours=300)).isoformat() + 'Z'
|
||||
assert cur.execute.call_args_list[0].args[1] == ['1', '2', '3']
|
||||
|
||||
def test_newest_clearance_first(self):
|
||||
rows = [_event('1', 400, 300), _event('2', 100, 30, target='micro1')]
|
||||
patcher, _ = _fake_conn(rows)
|
||||
with patcher:
|
||||
out = db.get_cleared_tickets(['1', '2'])
|
||||
assert [r['ticket_id'] for r in out] == ['2', '1']
|
||||
assert out[0]['target_name'] == 'micro1'
|
||||
@@ -31,6 +31,8 @@ def db():
|
||||
mock_db.cutsheet_load.return_value = {'items': [], 'cables': [], 'port_labels': []}
|
||||
mock_db.get_pending_recovery_notes.return_value = []
|
||||
mock_db.get_device_link.return_value = ''
|
||||
mock_db.get_incident_ticket.return_value = None
|
||||
mock_db.get_ticket_ids_between.return_value = []
|
||||
yield mock_db
|
||||
|
||||
|
||||
@@ -381,6 +383,47 @@ class TestWanLog:
|
||||
mon._process_wan_log([])
|
||||
assert not mon.tickets.create.called
|
||||
|
||||
def test_unifi_update_is_not_a_wan_event(self, mon, db):
|
||||
now = time.time()
|
||||
self._state(db, unifi_log_cursor=str(int(now * 1000)))
|
||||
self._mon(mon, [{'key': 'NETWORK_UPDATED', 'ts': now, 'device': '', 'wan': '',
|
||||
'app': 'UniFi Network', 'previous': '11.0.97', 'version': '11.0.99'}])
|
||||
mon._process_wan_log([])
|
||||
assert not mon.tickets.create.called
|
||||
saved = dict((c.args[0], c.args[1]) for c in db.set_state.call_args_list)
|
||||
assert saved[monitor.UNIFI_UPDATE_STATE]['label'] == 'UniFi Network 11.0.97 to 11.0.99'
|
||||
assert saved['wan_history'] == {'outages': [], 'restarts': []}
|
||||
|
||||
def test_update_comments_on_tickets_opened_just_before(self, mon, db):
|
||||
now = time.time()
|
||||
self._state(db, unifi_log_cursor=str(int(now * 1000)))
|
||||
db.get_ticket_ids_between.return_value = ['207521068']
|
||||
self._mon(mon, [{'key': 'NETWORK_UPDATED', 'ts': now, 'device': '', 'wan': '',
|
||||
'app': 'UniFi Network', 'previous': '11.0.97', 'version': '11.0.99'}])
|
||||
mon._process_wan_log([])
|
||||
start, end = db.get_ticket_ids_between.call_args.args[1:]
|
||||
assert (end - start).total_seconds() == monitor.UPDATE_LOOKBACK_S + monitor.UPDATE_HOLD_S
|
||||
tid, text = mon.tickets.comment.call_args.args
|
||||
assert tid == '207521068' and 'UniFi Network 11.0.97 to 11.0.99' in text
|
||||
|
||||
def test_update_in_the_same_read_holds_the_power_cycle_ticket(self, mon, db):
|
||||
now = time.time()
|
||||
state = {'unifi_log_cursor': str(int(now * 1000))}
|
||||
db.get_state.side_effect = lambda key, default=None: state.get(key, default)
|
||||
db.set_state.side_effect = lambda key, value: state.__setitem__(
|
||||
key, json.dumps(value) if isinstance(value, dict) else value)
|
||||
self._mon(mon, [
|
||||
{'key': modem_api.LOG_MODEM_RESTARTED, 'ts': now - 600, 'device': 'RPDU', 'wan': ''},
|
||||
{'key': 'NETWORK_UPDATED', 'ts': now, 'device': '', 'wan': '', 'app': 'UniFi Network'},
|
||||
])
|
||||
mon._process_wan_log([])
|
||||
assert not mon.tickets.create.called
|
||||
db.resolve_event.assert_called_with('modem_power_cycled', 'att-modem', '')
|
||||
|
||||
def test_hold_ends(self, mon, db):
|
||||
self._state(db, unifi_update=json.dumps({'label': 'x', 'hold_until': time.time() - 1}))
|
||||
assert mon._update_hold() is None
|
||||
|
||||
def test_uptime_going_backwards_records_a_restart(self, mon, db):
|
||||
self._state(db, modem_stats=json.dumps({'uptime_s': 5000}))
|
||||
mon._note_modem_restart({'uptime_s': 60})
|
||||
@@ -393,6 +436,55 @@ class TestWanLog:
|
||||
assert not db.set_state.called
|
||||
|
||||
|
||||
class TestWanIncident:
|
||||
"""Last night's sequence: the RPDU power-cycles the modem in the cycle the modem goes unreachable."""
|
||||
|
||||
def _cycle(self, mon, db):
|
||||
db.upsert_event.side_effect = [(30, True, 1, None), (29, False, 2, None)]
|
||||
mon.modem.poll.side_effect = modem_api.ModemUnreachable('Network is unreachable')
|
||||
with mon._wan_batch():
|
||||
mon._one_shot_ticket([], 'modem_power_cycled', 'RPDU power-cycled the modem.',
|
||||
'Modem auto power-cycled after internet loss', '3', 'Check the WAN panel.')
|
||||
mon._collect_modem([])
|
||||
|
||||
def test_one_ticket_led_by_the_most_urgent(self, mon, db):
|
||||
self._cycle(mon, db)
|
||||
assert mon.tickets.create.call_count == 1
|
||||
title, desc = mon.tickets.create.call_args.args
|
||||
assert 'WAN modem unreachable' in title
|
||||
assert mon.tickets.create.call_args.kwargs['priority'] == '2'
|
||||
assert '### Also seen in this incident' in desc and 'power-cycled' in desc
|
||||
assert sorted(c.args for c in db.set_ticket_id.call_args_list) == [(29, '123'), (30, '123')]
|
||||
|
||||
def test_recent_incident_gets_a_comment_instead(self, mon, db):
|
||||
db.get_incident_ticket.return_value = '555'
|
||||
mon.tickets.comment.return_value = True
|
||||
self._cycle(mon, db)
|
||||
assert not mon.tickets.create.called
|
||||
assert [c.args[0] for c in mon.tickets.comment.call_args_list] == ['555', '555']
|
||||
assert 'Also seen: WAN modem unreachable' in mon.tickets.comment.call_args_list[0].args[1]
|
||||
assert sorted(c.args for c in db.set_ticket_id.call_args_list) == [(29, '555'), (30, '555')]
|
||||
db.get_incident_ticket.assert_called_with('att-modem', monitor.WAN_INCIDENT_WINDOW_S)
|
||||
|
||||
def test_failed_comment_falls_back_to_a_ticket(self, mon, db):
|
||||
db.get_incident_ticket.return_value = '555'
|
||||
mon.tickets.comment.return_value = False
|
||||
self._cycle(mon, db)
|
||||
assert mon.tickets.create.call_count == 1
|
||||
|
||||
def test_outside_a_batch_tickets_open_at_once(self, mon, db):
|
||||
mon._one_shot_ticket([], 'wan_unstable', 'Unstable.', 'WAN connection unstable', '2', 'Look.')
|
||||
assert 'unstable' in mon.tickets.create.call_args.args[0]
|
||||
assert 'no recovery note' in mon.tickets.create.call_args.args[1]
|
||||
|
||||
def test_update_hold_skips_new_tickets(self, mon, db):
|
||||
db.get_state.side_effect = lambda key, default=None: (
|
||||
json.dumps({'label': 'UniFi Network 11.0.99', 'hold_until': time.time() + 60})
|
||||
if key == monitor.UNIFI_UPDATE_STATE else default)
|
||||
self._cycle(mon, db)
|
||||
assert not mon.tickets.create.called and not mon.tickets.comment.called
|
||||
|
||||
|
||||
class TestUnifiDeviceDetail:
|
||||
def test_parses_string_stats_and_outlets(self):
|
||||
d = monitor.UnifiClient._device_detail({
|
||||
@@ -421,6 +513,31 @@ class TestUnifiDeviceDetail:
|
||||
got = c.get_system_events(1000, modem_api.WAN_LOG_KEYS)
|
||||
assert got == [{'key': modem_api.LOG_MODEM_RESTARTED, 'ts': 2.0, 'device': 'RPDU', 'wan': ''}]
|
||||
|
||||
def test_system_log_reads_update_versions(self):
|
||||
c = monitor.UnifiClient({'controller': 'https://u.test', 'api_key': 'k'})
|
||||
page = MagicMock()
|
||||
page.json.side_effect = [
|
||||
{'data': [
|
||||
{'key': 'NETWORK_UPDATED', 'timestamp': 3000, 'parameters': {
|
||||
'APPLICATION': 'UniFi Network', 'PREVIOUS_VERSION': '11.0.97', 'VERSION': '11.0.99'}},
|
||||
{'key': 'FIRMWARE_UPDATE_AVAILABLE_V2', 'timestamp': 3000},
|
||||
{'key': 'MADE_CHANGES_ONE_SETTING_OBJECT', 'timestamp': 3000},
|
||||
]},
|
||||
{'data': []},
|
||||
]
|
||||
c.session.post = MagicMock(return_value=page)
|
||||
got = c.get_system_events(1000, modem_api.wanted_log_key)
|
||||
assert got == [{'key': 'NETWORK_UPDATED', 'ts': 3.0, 'device': '', 'wan': '',
|
||||
'app': 'UniFi Network', 'previous': '11.0.97', 'version': '11.0.99'}]
|
||||
|
||||
@pytest.mark.parametrize('key,expected', [
|
||||
('NETWORK_UPDATED', True), ('FIRMWARE_UPDATED_V2', True), ('PROTECT_UPDATED', True),
|
||||
('FIRMWARE_UPDATE_AVAILABLE_V2', False), ('MADE_CHANGES_ONE_SETTING_OBJECT', False),
|
||||
('NETWORK_WAN_FAILED_2', False),
|
||||
])
|
||||
def test_update_keys(self, key, expected):
|
||||
assert modem_api.is_update_key(key) is expected
|
||||
|
||||
def test_system_log_failure_is_none_not_empty(self):
|
||||
c = monitor.UnifiClient({'controller': 'https://u.test', 'api_key': 'k'})
|
||||
c.session.post = MagicMock(side_effect=OSError('boom'))
|
||||
@@ -796,3 +913,60 @@ class TestTicketClientCreate:
|
||||
with self._post({'success': True, 'ticket_id': '8', 'action': 'none'}):
|
||||
assert c.create('t', 'd') == '8'
|
||||
assert 't' not in c._closed_matches
|
||||
|
||||
|
||||
class TestTicketClientOpenTickets:
|
||||
def _client(self):
|
||||
return monitor.TicketClient({'url': 'http://t.test/create_ticket_api.php', 'api_key': 'k'})
|
||||
|
||||
def _page(self, tickets, pages):
|
||||
resp = MagicMock(status_code=200)
|
||||
resp.json.return_value = {'success': True, 'tickets': tickets, 'page': 1, 'pages': pages}
|
||||
return resp
|
||||
|
||||
def test_read_endpoint_sits_beside_create(self):
|
||||
assert self._client().read_url == 'http://t.test/api/tickets_api.php'
|
||||
|
||||
def test_reads_every_page_of_open_statuses(self):
|
||||
pages = [self._page([{'ticket_id': '1', 'status': 'Open'}], 2),
|
||||
self._page([{'ticket_id': 2, 'status': 'Pending'}], 2)]
|
||||
with patch.object(monitor.requests, 'get', side_effect=pages) as get:
|
||||
assert self._client().open_tickets() == {'1': 'Open', '2': 'Pending'}
|
||||
assert get.call_count == 2
|
||||
params = get.call_args.kwargs['params']
|
||||
assert params['status'] == 'Open,Pending,In Progress' and params['page'] == 2
|
||||
assert get.call_args.kwargs['headers'] == {'Authorization': 'Bearer k'}
|
||||
|
||||
def test_failure_means_unknown_not_empty(self):
|
||||
with patch.object(monitor.requests, 'get', side_effect=OSError('down')):
|
||||
assert self._client().open_tickets() is None
|
||||
bad = MagicMock(status_code=200)
|
||||
bad.json.return_value = {'success': False, 'error': 'nope'}
|
||||
with patch.object(monitor.requests, 'get', return_value=bad):
|
||||
assert self._client().open_tickets() is None
|
||||
|
||||
def test_unconfigured(self):
|
||||
assert monitor.TicketClient({}).open_tickets() is None
|
||||
|
||||
|
||||
class TestSyncOpenTickets:
|
||||
def test_saves_the_list_then_waits(self, mon, db):
|
||||
mon.tickets.open_tickets.return_value = {'7': 'Open'}
|
||||
mon._sync_open_tickets()
|
||||
mon._sync_open_tickets()
|
||||
assert mon.tickets.open_tickets.call_count == 1
|
||||
key, value = db.set_state.call_args.args
|
||||
assert key == monitor.OPEN_TICKETS_STATE and value['tickets'] == {'7': 'Open'}
|
||||
assert value['checked_at'] > 0
|
||||
|
||||
def test_failed_read_keeps_the_previous_list(self, mon, db):
|
||||
mon.tickets.open_tickets.return_value = None
|
||||
mon._sync_open_tickets()
|
||||
db.set_state.assert_not_called()
|
||||
|
||||
def test_refreshes_after_the_interval(self, mon, db):
|
||||
mon.tickets.open_tickets.return_value = {}
|
||||
mon._sync_open_tickets()
|
||||
mon._open_tickets_synced -= monitor.OPEN_TICKETS_SYNC_S + 1
|
||||
mon._sync_open_tickets()
|
||||
assert mon.tickets.open_tickets.call_count == 2
|
||||
@@ -147,3 +147,25 @@ def recovery_comment(event: dict, now: datetime, url: str = '') -> str:
|
||||
lines += ['', '_Gandalf leaves this ticket open. Close it once you know why it happened, '
|
||||
'or leave it open if it needs follow-up._']
|
||||
return '\n'.join(lines)
|
||||
|
||||
|
||||
def incident_comment(name: str, summary: str, checks: Sequence[str], when: str) -> str:
|
||||
"""A further report that belongs to the incident this ticket is already about."""
|
||||
lines = [f'**Also seen: {name}** ({when})', '', summary.strip()]
|
||||
if checks:
|
||||
lines += ['', '**What to check**'] + [f'{n}. {check}' for n, check in enumerate(checks, 1)]
|
||||
lines += ['', '_Gandalf added this here instead of opening another ticket, since it is part of '
|
||||
'the same incident._']
|
||||
return '\n'.join(lines)
|
||||
|
||||
|
||||
def update_comment(label: str, when: str, lookback_min: int, hold_min: int) -> str:
|
||||
"""Note on a ticket that a UniFi update finished around the time it was opened."""
|
||||
return '\n'.join([
|
||||
f'**UniFi update:** {label} finished at {when}.', '',
|
||||
f'This ticket was opened within {lookback_min} minutes of that update, which UniFi logs once '
|
||||
f'it is done. Updating the Network app or the gateway firmware can drop the WAN, and Smart '
|
||||
f'Power may then power-cycle the modem, so the update is the likely cause.', '',
|
||||
f'_Gandalf holds new modem and WAN tickets for {hold_min} minutes after an update; anything '
|
||||
f'still wrong after that gets its own ticket._',
|
||||
])
|
||||
Reference in new issue
Block a user