Compare commits
7
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f655836446 | ||
|
|
ee2708dccb | ||
|
|
c262e5ecc8 | ||
|
|
29119c2235 | ||
|
|
9d561db703 | ||
|
|
9e2a2b987c | ||
|
|
437f7a415f |
No files matched your search
@@ -114,6 +114,36 @@ Ping-only (no node_exporter): **pbs** (10.10.10.3)
|
||||
- Quick-suppress modal: apply timed or manual suppression from any alert row
|
||||
- Auto-refreshes every 30 seconds via `/api/status` + `/api/network`
|
||||
|
||||
### Alert history (`/events`)
|
||||
Every alert Gandalf raised in the last 24 hours to 90 days, including ones that cleared before anyone
|
||||
saw them: when each started, how long it lasted, failed checks, and its ticket.
|
||||
- Filter by device (name, port or interface), alert type, severity, and open or cleared. Filters are
|
||||
kept in the URL, so `/events?target=large1&range=90d` is a shareable link; device pages and tickets
|
||||
link to their own history this way.
|
||||
- **Recurring** lists the same alert on the same target more than once in the range, worst first, since
|
||||
a flapping cable or optic shows up here before anyone notices the tickets.
|
||||
- API: `GET /api/events/history?range=24h|7d|30d|90d&type=&target=&severity=&status=all|active|resolved&offset=`
|
||||
(100 per page).
|
||||
|
||||
### Rack (`/rack`)
|
||||
The rack front, drawn from the rack units set on cutsheet items (`rack.units`, default 42U, top first).
|
||||
- An item named after a UniFi switch or gateway with a faceplate is drawn with its real faceplate and
|
||||
live port state; patch panels get a generic panel with each port's label; anything else is a labelled
|
||||
blank plate (desk switches and devices without a faceplate included).
|
||||
- Every documented port shows its cutsheet state (green matches, red mismatch, grey unconfirmed).
|
||||
Hovering a port lights up every port on the same cable run, through patch panels, and names the
|
||||
cables; clicking opens the switch port or the cable.
|
||||
- Items that would overlap another or run past the top of the rack are listed, not drawn over. Where two
|
||||
claim the same top unit, the one recorded first is drawn. Items without a rack unit are listed below.
|
||||
- Adding the new switch: give it a cutsheet item with its rack unit and it appears; it is drawn as a
|
||||
plate until its model has a faceplate in `static/faceplates.js`.
|
||||
|
||||
### Health check (`/health`)
|
||||
No login needed. Returns 200 when the database answers and the monitor has completed a check recently,
|
||||
otherwise 503 with the failing check: `{"status": "degraded", "checks": {"db": "ok", "monitor": "stale",
|
||||
"monitor_age_s": 742}}`. The monitor counts as stalled after five missed polls (at least 10 minutes),
|
||||
so point an uptime monitor at it to catch a hung `gandalf-monitor`, not just a dead web app.
|
||||
|
||||
### Device pages (`/device/<name>`)
|
||||
One page per host, switch, router, AP, PDU, or modem. Reach them from the search box on the
|
||||
dashboard (`/` focuses it), or from any device name. Search matches hostname, IP, MAC, model, type,
|
||||
@@ -145,6 +175,17 @@ The port panel's **SFP Module** section lists type, part, serial and faults, plu
|
||||
optical readings (TX/RX power, temperature, voltage, bias) when it reports them. Copper DACs don't.
|
||||
Both ends of a DAC report the same cable serial, so a DAC seen on two ports links to its other end.
|
||||
|
||||
**Cutsheet marks**: a port with a cable documented in the cutsheet gets a dot in its corner: green when
|
||||
it matches what Gandalf sees, red for a mismatch (wrong neighbor, no link, missing port, port conflict),
|
||||
grey when it is documented but can't be confirmed (planned, dark, through a panel, or the far end isn't
|
||||
visible). Hovering the port names the cable. "Cutsheet marks" in the legend hides them; the choice is
|
||||
remembered in the browser.
|
||||
|
||||
**Inventory link**: the device page's 📦 button opens the device's record in the inventory
|
||||
(`inventory.base_url`). Admins set it with ✎ by pasting the record's URL; only URLs on the inventory
|
||||
host are accepted. Until one is set, the button searches the inventory for the model. Tickets include
|
||||
the link too.
|
||||
|
||||
**Port block colours:**
|
||||
| Colour | State |
|
||||
|---|---|
|
||||
@@ -255,6 +296,34 @@ can view it; only the `admin` group can edit, and every change is logged with wh
|
||||
|
||||
A failed Pulse ping check counts as "unknown": it neither raises nor resolves `host_unreachable`.
|
||||
|
||||
### What a ticket contains
|
||||
|
||||
Tickets are Markdown. Besides the alert's own facts and a numbered "What to check" list, each one adds
|
||||
whatever Gandalf knows around the problem:
|
||||
|
||||
- **Where it is connected**: the switch port(s) the host or device plugs into, with link state, speed,
|
||||
PoE and SFP module, linked to that port on the switch's page. A host that has dropped off the network
|
||||
has lost its LLDP entry, so Gandalf keeps the last port it saw each device on (`known_uplinks` in
|
||||
`monitor_state`) and shows that, marked "last seen linked there".
|
||||
- The **cutsheet** cables ending on the device or port, with their full path through patch panels
|
||||
- **History**: how often the same alert fired in the last 7 and 30 days, and when it last cleared;
|
||||
3 or more in a week is called out as recurring
|
||||
- **Other active alerts** on the same device, with their ticket numbers
|
||||
- **Links** to the device in Gandalf (from `gandalf.public_url`)
|
||||
|
||||
Titles keep the `[target][auto][production]...` tag format, which Tinker uses to spot duplicates.
|
||||
|
||||
### Recovery notes
|
||||
|
||||
When an alert that opened a ticket clears, the monitor adds a comment to that ticket: when it cleared, how
|
||||
long it lasted, how many checks failed, and a link back to Gandalf. **Gandalf never closes or changes the
|
||||
status of a ticket**; that is left to a person. The one-shot WAN reports (modem power-cycled, WAN unstable)
|
||||
get no note, since they describe something already over.
|
||||
|
||||
A note that fails to post is retried every cycle for 24 hours. Posting needs a Tinker API key with
|
||||
`read_write` scope; if Tinker answers 401 or 403, the monitor logs one error and stops trying until it is
|
||||
restarted.
|
||||
|
||||
### WAN modem notes
|
||||
|
||||
The alarm and warning limits are read from the gateway on every poll, so they are never hard-coded here.
|
||||
@@ -310,9 +379,12 @@ Shared by both processes. Located in the working directory (`/var/www/html/prod/
|
||||
"site_id": "default"
|
||||
},
|
||||
"ticket_api": {
|
||||
"url": "https://t.lotusguild.org/api/tickets",
|
||||
"url": "https://t.lotusguild.org/create_ticket_api.php",
|
||||
"api_key": "..."
|
||||
},
|
||||
"gandalf": {
|
||||
"public_url": "https://gandalf.lotusguild.org"
|
||||
},
|
||||
"pulse": {
|
||||
"url": "http://<pulse-host>:<port>",
|
||||
"api_key": "...",
|
||||
@@ -360,7 +432,13 @@ Shared by both processes. Located in the working directory (`/var/www/html/prod/
|
||||
| `unifi.controller` | UniFi controller base URL (HTTPS, self-signed cert ignored) |
|
||||
| `unifi.api_key` | UniFi API key from controller Settings → API |
|
||||
| `unifi.site_id` | UniFi site ID (default: `default`) |
|
||||
| `ticket_api.api_key` | Tinker Tickets bearer token |
|
||||
| `ticket_api.url` | Tinker Tickets `create_ticket_api.php` URL |
|
||||
| `ticket_api.api_key` | Tinker Tickets bearer token. Needs `read_write` scope to add recovery notes |
|
||||
| `ticket_api.comment_url` | Tinker comment endpoint (default: `api/ticket_comment_api.php` beside `ticket_api.url`) |
|
||||
| `gandalf.public_url` | Gandalf's own address, for links in tickets (default: `https://gandalf.lotusguild.org`) |
|
||||
| `rack.units` | Height of the rack drawn on `/rack` (default: 42) |
|
||||
| `inventory.base_url` | Asset inventory (HomeBox) address; device inventory links must be on this host (default: `https://inventory.lotusguild.org`) |
|
||||
| `inventory.search_url` | Search link used before a device has a record linked, with `{base}` and `{q}` placeholders (default: `{base}/items?q={q}`) |
|
||||
| `pulse.url` | Pulse worker API base URL (for SSH relay) |
|
||||
| `pulse.worker_id` | Which Pulse worker runs ethtool collection |
|
||||
| `pulse.timeout` | Max seconds to wait for SSH collection per host |
|
||||
@@ -567,6 +645,7 @@ gandalf/
|
||||
├── modem.py # Read-only AT&T gateway status page client and parser
|
||||
├── devices.py # Device inventory for search and device pages
|
||||
├── cutsheet.py # Cutsheet validation, path tracing, plan-vs-actual checks, CSV
|
||||
├── ticket_text.py # Markdown for tickets and recovery notes
|
||||
├── db.py # Database operations (MariaDB via pymysql, thread-local conn reuse)
|
||||
├── schema.sql # Database schema (network_events, suppression_rules, monitor_state)
|
||||
├── config.json # Runtime configuration (not committed with secrets)
|
||||
@@ -576,12 +655,16 @@ gandalf/
|
||||
│ ├── app.js # Dashboard JS (auto-refresh, events, suppress modal, search, modem panel)
|
||||
│ ├── device.js # Device page rendering
|
||||
│ ├── cutsheet.js # Cutsheet page
|
||||
│ ├── events.js # Alert history page
|
||||
│ ├── rack.js # Rack drawing
|
||||
│ └── port-inspector.js # Switch front panel, port panel, path check, diagnostics
|
||||
└── templates/
|
||||
├── base.html # Shared layout (header, nav, footer)
|
||||
├── index.html # Dashboard page
|
||||
├── device.html # Per-device page
|
||||
├── cutsheet.html # Cutsheet page and its editors
|
||||
├── events.html # Alert history page
|
||||
├── rack.html # Rack drawing
|
||||
├── _modem_panel.html # WAN / Fiber panel, shared by the dashboard and modem page
|
||||
└── suppressions.html # Suppression management page
|
||||
```
|
||||
@@ -617,7 +700,9 @@ Take LED colors from the model's quick start guide. See the comment at the top o
|
||||
|
||||
### Database Schema Notes
|
||||
|
||||
- `network_events`: one row per active event; `resolved_at` is set when recovered
|
||||
- `device_links`: each device's inventory URL, by device name; created with the cutsheet tables
|
||||
- `network_events`: one row per active event; `resolved_at` is set when recovered, and
|
||||
`recovery_notified` once the recovery note is on its ticket (column added automatically at monitor startup)
|
||||
- `suppression_rules`: `active=FALSE` when removed; `expires_at` checked at query time
|
||||
- `monitor_state`: key/value store; `interface_baseline`, `link_stats`, and `modem_stats` are JSON blobs
|
||||
- `monitor_state` also holds `wan_history` (WAN outages, modem restarts) and `unifi_log_cursor` (how far the UniFi log has been read)
|
||||
|
||||
@@ -17,6 +17,8 @@ import time
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from functools import wraps
|
||||
from typing import Optional
|
||||
from urllib.parse import quote, urlparse
|
||||
|
||||
from flask import Flask, jsonify, make_response, redirect, render_template, request, send_file, url_for
|
||||
|
||||
@@ -104,15 +106,34 @@ def add_security_headers(response):
|
||||
return response
|
||||
|
||||
|
||||
def _daemon_ok(last_check: str) -> bool:
|
||||
"""Return True if monitor last checked within 20 minutes."""
|
||||
def _monitor_stale_after() -> int:
|
||||
"""Seconds without a completed check before the monitor counts as stalled.
|
||||
|
||||
Five missed polls, and never under 10 minutes: one slow cycle (an SSH timeout
|
||||
to a dead host) shouldn't trip it, but a hung monitor is caught in minutes.
|
||||
"""
|
||||
try:
|
||||
poll = int(_config().get('monitor', {}).get('poll_interval', 120))
|
||||
except (TypeError, ValueError):
|
||||
poll = 120
|
||||
return max(600, poll * 5)
|
||||
|
||||
|
||||
def _monitor_age(last_check: str) -> Optional[float]:
|
||||
"""Seconds since the monitor's last check, or None if it never ran or can't be read."""
|
||||
if not last_check or last_check == 'Never':
|
||||
return False
|
||||
return None
|
||||
try:
|
||||
ts = datetime.strptime(last_check, '%Y-%m-%d %H:%M:%S UTC').replace(tzinfo=timezone.utc)
|
||||
return (datetime.now(timezone.utc) - ts).total_seconds() < 1200
|
||||
except Exception:
|
||||
return False
|
||||
except ValueError:
|
||||
return None
|
||||
return (datetime.now(timezone.utc) - ts).total_seconds()
|
||||
|
||||
|
||||
def _daemon_ok(last_check: str) -> bool:
|
||||
"""True if the monitor checked recently enough (see _monitor_stale_after)."""
|
||||
age = _monitor_age(last_check)
|
||||
return age is not None and age < _monitor_stale_after()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -298,7 +319,8 @@ def links_page():
|
||||
@app.route('/device/<path:name>')
|
||||
@require_auth
|
||||
def device_page(name: str):
|
||||
return render_template('device.html', user=_get_user(), device_name=name)
|
||||
user = _get_user()
|
||||
return render_template('device.html', user=user, device_name=name, can_edit='admin' in user['groups'])
|
||||
|
||||
|
||||
@app.route('/inspector')
|
||||
@@ -308,6 +330,50 @@ def inspector():
|
||||
return redirect(url_for('index'))
|
||||
|
||||
|
||||
@app.route('/events')
|
||||
@require_auth
|
||||
def events_page():
|
||||
return render_template('events.html', user=_get_user())
|
||||
|
||||
|
||||
_HISTORY_RANGES = {'24h': 24, '7d': 24 * 7, '30d': 24 * 30, '90d': 24 * 90}
|
||||
_HISTORY_PAGE = 100
|
||||
|
||||
|
||||
@app.route('/api/events/history')
|
||||
@require_auth
|
||||
def api_events_history():
|
||||
"""Every alert open at some point in a time window, with summaries.
|
||||
|
||||
Query: range (24h|7d|30d|90d), type, target (substring of name or detail),
|
||||
severity, status (all|active|resolved), offset.
|
||||
"""
|
||||
args = request.args
|
||||
hours = _HISTORY_RANGES.get(args.get('range', '7d'))
|
||||
if hours is None:
|
||||
return jsonify({'error': f'range must be one of {", ".join(_HISTORY_RANGES)}'}), 400
|
||||
severity = args.get('severity', '')
|
||||
if severity not in ('', 'critical', 'warning', 'info'):
|
||||
return jsonify({'error': 'severity must be critical, warning or info'}), 400
|
||||
status = args.get('status', 'all')
|
||||
if status not in ('all', 'active', 'resolved'):
|
||||
return jsonify({'error': 'status must be all, active or resolved'}), 400
|
||||
try:
|
||||
offset = max(int(args.get('offset', 0)), 0)
|
||||
except ValueError:
|
||||
return jsonify({'error': 'offset must be an integer'}), 400
|
||||
event_type = args.get('type', '')[:60]
|
||||
target = args.get('target', '').strip()[:255]
|
||||
|
||||
result = db.get_events_history(hours, event_type, target, severity, status,
|
||||
limit=_HISTORY_PAGE, offset=offset)
|
||||
result['flappers'] = db.get_flappers(hours, event_type, target, severity)
|
||||
result['types'] = db.get_event_types()
|
||||
result['offset'] = offset
|
||||
result['page_size'] = _HISTORY_PAGE
|
||||
return jsonify(result)
|
||||
|
||||
|
||||
@app.route('/suppressions')
|
||||
@require_auth
|
||||
@require_admin
|
||||
@@ -461,6 +527,7 @@ def api_device(name: str):
|
||||
except Exception as e:
|
||||
logger.error(f'Cutsheet unavailable for {name}: {e}')
|
||||
detail['cutsheet'] = []
|
||||
detail['inventory'] = _inventory_for(name, detail.get('model') or '')
|
||||
return jsonify(detail)
|
||||
|
||||
|
||||
@@ -936,6 +1003,118 @@ def api_cutsheet():
|
||||
return jsonify(view)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Inventory (HomeBox) links
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_INVENTORY_DEFAULT = 'https://inventory.lotusguild.org'
|
||||
|
||||
|
||||
def _inventory_base() -> str:
|
||||
return ((_config().get('inventory') or {}).get('base_url') or _INVENTORY_DEFAULT).rstrip('/')
|
||||
|
||||
|
||||
def _inventory_for(name: str, model: str) -> dict:
|
||||
"""The device's inventory record link, or a search to find it."""
|
||||
base = _inventory_base()
|
||||
template = (_config().get('inventory') or {}).get('search_url') or '{base}/items?q={q}'
|
||||
try:
|
||||
_ensure_cutsheet_schema()
|
||||
url = db.get_device_link(name)
|
||||
except Exception as e:
|
||||
logger.error(f'Inventory link unavailable for {name}: {e}')
|
||||
url = ''
|
||||
return {'url': url, 'search_url': template.format(base=base, q=quote(model or name, safe=''))}
|
||||
|
||||
|
||||
def _validate_inventory_url(value) -> str:
|
||||
"""Only links into the configured inventory site: this URL is shown as a link to every viewer."""
|
||||
url = str(value or '').strip()
|
||||
if not url:
|
||||
return ''
|
||||
if len(url) > 512:
|
||||
raise cutsheet.ValidationError('Inventory link is too long')
|
||||
base = urlparse(_inventory_base())
|
||||
got = urlparse(url)
|
||||
if got.scheme != 'https' or got.netloc.lower() != base.netloc.lower() or got.username or got.password:
|
||||
raise cutsheet.ValidationError(f'Inventory link must start with {base.scheme}://{base.netloc}/')
|
||||
return url
|
||||
|
||||
|
||||
@app.route('/api/device/<path:name>/inventory', methods=['PUT'])
|
||||
@require_auth
|
||||
@require_admin
|
||||
@_cutsheet_write
|
||||
def api_device_inventory(name: str):
|
||||
"""Set or clear (empty url) the device's inventory link."""
|
||||
if len(name) > 255:
|
||||
raise cutsheet.ValidationError('Device name is too long')
|
||||
url = _validate_inventory_url(_json_body().get('url'))
|
||||
db.set_device_link(name, url, _get_user()['username'])
|
||||
logger.info(f'{_get_user()["username"]} set inventory link for {name}: {url or "(removed)"}')
|
||||
return jsonify({'success': True, 'url': url})
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Rack elevation
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@app.route('/rack')
|
||||
@require_auth
|
||||
def rack_page():
|
||||
user = _get_user()
|
||||
return render_template('rack.html', user=user, can_edit='admin' in user['groups'])
|
||||
|
||||
|
||||
def _rack_units() -> int:
|
||||
try:
|
||||
return max(1, min(int((_config().get('rack') or {}).get('units', 42)), 60))
|
||||
except (TypeError, ValueError):
|
||||
return 42
|
||||
|
||||
|
||||
@app.route('/api/rack')
|
||||
@require_auth
|
||||
def api_rack():
|
||||
"""The rack front: placed cutsheet items, live switch ports, and each port's cutsheet state."""
|
||||
view = _cutsheet_view()
|
||||
link_stats = _state_json('link_stats')
|
||||
switches = link_stats.get('unifi_switches') or {}
|
||||
unifi_info = link_stats.get('unifi_devices') or {}
|
||||
snapshot = _state_json('network_snapshot')
|
||||
online = {d['name']: d.get('connected', True) for d in snapshot.get('unifi') or []}
|
||||
known = set(switches) | set(online) | set(snapshot.get('hosts') or {})
|
||||
|
||||
for cable in view['cables']:
|
||||
cable['run'] = [h['cable_id'] for h in cable['path'] if 'cable_id' in h]
|
||||
marks = cutsheet.port_marks(view['cables'])
|
||||
labels: dict = {}
|
||||
for pl in view['port_labels']:
|
||||
labels.setdefault(pl['item_name'], {})[str(pl['port'])] = pl['label']
|
||||
|
||||
units = _rack_units()
|
||||
layout = cutsheet.rack_layout(view['items'], units)
|
||||
out = []
|
||||
for item in layout['placed']:
|
||||
name = item['name']
|
||||
entry = {k: item.get(k) for k in ('id', 'name', 'kind', 'port_count', 'rack_u', 'rack_height',
|
||||
'rack_top', 'location', 'notes')}
|
||||
entry['url'] = f'/device/{quote(name, safe="")}' if name in known else f'/cutsheet#item-{item["id"]}'
|
||||
entry['ports'] = {port: m for (dev, port), m in marks.items() if dev == name}
|
||||
entry['labels'] = labels.get(name, {})
|
||||
if name in switches:
|
||||
sw = switches[name]
|
||||
entry['switch'] = {'model': sw.get('model') or '', 'ports': sw.get('ports') or {},
|
||||
'outlets': (unifi_info.get(name) or {}).get('outlets') or [],
|
||||
'online': online.get(name, True)}
|
||||
out.append(entry)
|
||||
return jsonify({
|
||||
'units': units, 'items': out, 'overlaps': layout['overlaps'], 'outside': layout['outside'],
|
||||
'unplaced': sorted(i['name'] for i in view['items'] if not i.get('rack_u')),
|
||||
'updated': link_stats.get('updated'),
|
||||
})
|
||||
|
||||
|
||||
@app.route('/api/cutsheet/cables', methods=['POST'])
|
||||
@require_auth
|
||||
@require_admin
|
||||
@@ -1090,19 +1269,19 @@ def health():
|
||||
checks['db'] = 'error'
|
||||
overall = 'degraded'
|
||||
|
||||
# Monitor freshness: fail if last_check is older than 20 minutes
|
||||
# Monitor freshness: a hung or crashed monitor stops updating last_check
|
||||
try:
|
||||
last_check = db.get_state('last_check', '')
|
||||
if last_check:
|
||||
ts = datetime.strptime(last_check, '%Y-%m-%d %H:%M:%S UTC').replace(tzinfo=timezone.utc)
|
||||
age_s = (datetime.now(timezone.utc) - ts).total_seconds()
|
||||
if age_s > 1200:
|
||||
checks['monitor'] = 'stale'
|
||||
overall = 'degraded'
|
||||
else:
|
||||
checks['monitor'] = 'ok'
|
||||
else:
|
||||
age_s = _monitor_age(last_check)
|
||||
if not last_check:
|
||||
checks['monitor'] = 'no data yet'
|
||||
elif age_s is None or age_s > _monitor_stale_after():
|
||||
checks['monitor'] = 'stale'
|
||||
checks['monitor_age_s'] = int(age_s) if age_s is not None else None
|
||||
overall = 'degraded'
|
||||
else:
|
||||
checks['monitor'] = 'ok'
|
||||
checks['monitor_age_s'] = int(age_s)
|
||||
except Exception as e:
|
||||
logger.error(f'Health check monitor error: {e}')
|
||||
checks['monitor'] = 'error'
|
||||
|
||||
+94
-1
@@ -473,6 +473,37 @@ def parse_csv(text: str, items: Dict[str, dict], max_rows: int = 2000) -> Tuple[
|
||||
|
||||
# ── Problems, for the monitor ───────────────────────────────────────────────
|
||||
|
||||
def path_text(hops: list) -> str:
|
||||
return ' → '.join(describe_node(h) for h in hops if not isinstance(h, int))
|
||||
|
||||
|
||||
def runs_for(device: str, cables: List[dict], items: Dict[str, dict], switches: dict,
|
||||
port: Optional[str] = None) -> List[dict]:
|
||||
"""The live cables ending on a device (or one of its ports), each with its full run.
|
||||
|
||||
Returns [{'cable_id', 'label', 'port', 'status', 'verdict', 'detail', 'path'}].
|
||||
"""
|
||||
results = None
|
||||
out = []
|
||||
for c in sorted(cables, key=lambda c: _natural(c.get('label') or '')):
|
||||
if not _live(c):
|
||||
continue
|
||||
for end in ('a', 'b'):
|
||||
if not _same(c.get(f'{end}_device') or '', device):
|
||||
continue
|
||||
if port is not None and str(c.get(f'{end}_port') or '') != str(port):
|
||||
continue
|
||||
if results is None:
|
||||
results = evaluate(cables, items, switches or {})
|
||||
r = results[c['id']]
|
||||
out.append({
|
||||
'cable_id': c['id'], 'label': c.get('label') or f'#{c["id"]}',
|
||||
'port': c.get(f'{end}_port') or '', 'status': c['status'],
|
||||
'verdict': r['verdict'], 'detail': r['detail'], 'path': path_text(r['path']['hops']),
|
||||
})
|
||||
return out
|
||||
|
||||
|
||||
def problems(cables: List[dict], items: Dict[str, dict], switches: dict) -> List[dict]:
|
||||
"""One entry per installed run whose check found a problem.
|
||||
|
||||
@@ -508,6 +539,68 @@ def problems(cables: List[dict], items: Dict[str, dict], switches: dict) -> List
|
||||
'port': switch_end[1] if switch_end else lead['a_port'],
|
||||
'verdict': r['verdict'],
|
||||
'detail': r['detail'],
|
||||
'path': ' → '.join(describe_node(h) for h in hops if not isinstance(h, int)),
|
||||
'path': path_text(hops),
|
||||
})
|
||||
return found
|
||||
|
||||
|
||||
# ── Rack elevation ──────────────────────────────────────────────────────────
|
||||
|
||||
def rack_layout(items: List[dict], units: int) -> dict:
|
||||
"""Place items that have a rack unit, top of the rack first.
|
||||
|
||||
An item that would share a unit with one already placed, or runs past the
|
||||
top of the rack, is reported rather than drawn over another.
|
||||
Returns {'placed': [item + 'rack_top'], 'overlaps': [{'name', 'units', 'with'}], 'outside': [name]}.
|
||||
"""
|
||||
taken: Dict[int, str] = {}
|
||||
placed, overlaps, outside = [], [], []
|
||||
racked = [i for i in items if i.get('rack_u')]
|
||||
# Top first; where two claim the same top unit, the one recorded first is drawn
|
||||
for item in sorted(racked, key=lambda i: (-(i['rack_u'] + (i.get('rack_height') or 1)), i.get('id') or 0)):
|
||||
low = item['rack_u']
|
||||
high = low + (item.get('rack_height') or 1) - 1
|
||||
span = f'U{low}' if high == low else f'U{low}–{high}'
|
||||
if high > units:
|
||||
outside.append({'name': item['name'], 'units': span})
|
||||
continue
|
||||
clash = sorted({taken[u] for u in range(low, high + 1) if u in taken})
|
||||
if clash:
|
||||
overlaps.append({'name': item['name'], 'units': span, 'with': clash})
|
||||
continue
|
||||
for u in range(low, high + 1):
|
||||
taken[u] = item['name']
|
||||
placed.append(dict(item, rack_top=high))
|
||||
return {'placed': placed, 'overlaps': overlaps, 'outside': outside}
|
||||
|
||||
|
||||
_MARK_RANK = {'bad': 3, 'doc': 2, 'ok': 1}
|
||||
|
||||
|
||||
def port_marks(cables: List[dict]) -> Dict[tuple, dict]:
|
||||
"""Per (device, port): the worst check of the live cables plugged in there.
|
||||
|
||||
`cables` are evaluated cables (with 'verdict', 'detail' and 'run', the IDs of
|
||||
every cable in its run). Returns {(device, port): {'mark', 'text', 'cable_id', 'run'}},
|
||||
where mark is ok (matches), bad (a problem) or doc (documented, not confirmed).
|
||||
"""
|
||||
out: Dict[tuple, dict] = {}
|
||||
for c in cables:
|
||||
if not _live(c):
|
||||
continue
|
||||
mark = 'bad' if c['verdict'] in PROBLEM_VERDICTS else ('ok' if c['verdict'] == 'match' else 'doc')
|
||||
label = c.get('label') or f'#{c["id"]}'
|
||||
for end in ('a', 'b'):
|
||||
device, port = c.get(f'{end}_device'), c.get(f'{end}_port')
|
||||
if not device or not port:
|
||||
continue
|
||||
entry = out.setdefault((device, str(port)), {'mark': mark, 'cable_id': c['id'], 'run': [], 'by_detail': {}})
|
||||
if _MARK_RANK[mark] > _MARK_RANK[entry['mark']]:
|
||||
entry.update(mark=mark, cable_id=c['id'])
|
||||
entry['by_detail'].setdefault(c['detail'], []).append(label)
|
||||
entry['run'] = sorted(set(entry['run']) | set(c.get('run') or [c['id']]))
|
||||
for entry in out.values():
|
||||
# A panel port's front and rear cables share one check: say it once
|
||||
entry['text'] = '; '.join(f'cable{"s" if len(labels) > 1 else ""} {", ".join(labels)}: {detail}'
|
||||
for detail, labels in entry.pop('by_detail').items())
|
||||
return out
|
||||
@@ -348,6 +348,175 @@ def cleanup_expired_suppressions() -> int:
|
||||
return n
|
||||
|
||||
|
||||
def ensure_events_schema() -> None:
|
||||
"""Add columns newer than the original events table, so deploys need no manual migration."""
|
||||
with get_conn() as conn:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute("SHOW COLUMNS FROM network_events LIKE 'recovery_notified'")
|
||||
if cur.fetchone():
|
||||
return
|
||||
cur.execute(
|
||||
'ALTER TABLE network_events ADD COLUMN recovery_notified TINYINT(1) NOT NULL DEFAULT 0'
|
||||
)
|
||||
# Events that cleared before this existed are history, not owed a comment
|
||||
cur.execute(
|
||||
'UPDATE network_events SET recovery_notified=1, last_seen=last_seen WHERE resolved_at IS NOT NULL'
|
||||
)
|
||||
|
||||
|
||||
def get_pending_recovery_notes(hours: int = 24, exclude_types: tuple = ()) -> list:
|
||||
"""Resolved events with a ticket whose recovery comment hasn't been posted yet."""
|
||||
sql = """SELECT id, event_type, target_name, target_detail, ticket_id,
|
||||
first_seen, resolved_at, consecutive_failures
|
||||
FROM network_events
|
||||
WHERE resolved_at IS NOT NULL AND ticket_id IS NOT NULL AND recovery_notified=0
|
||||
AND resolved_at > DATE_SUB(NOW(), INTERVAL %s HOUR)"""
|
||||
params: list = [hours]
|
||||
if exclude_types:
|
||||
sql += f' AND event_type NOT IN ({", ".join(["%s"] * len(exclude_types))})' # nosec B608 - placeholders only
|
||||
params.extend(exclude_types)
|
||||
with get_conn() as conn:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(sql + ' ORDER BY resolved_at', params)
|
||||
return list(cur.fetchall())
|
||||
|
||||
|
||||
def mark_recovery_notified(event_id: int) -> None:
|
||||
with get_conn() as conn:
|
||||
with conn.cursor() as cur:
|
||||
# last_seen updates itself on any write; keep it as the last failing check
|
||||
cur.execute(
|
||||
'UPDATE network_events SET recovery_notified=1, last_seen=last_seen WHERE id=%s',
|
||||
(event_id,),
|
||||
)
|
||||
|
||||
|
||||
def event_stats(event_type: str, target_name: str, target_detail: str = '') -> dict:
|
||||
"""How often one alert has fired before, for a ticket's history section.
|
||||
|
||||
Returns {'last_7d', 'last_30d', 'last_first_seen', 'last_resolved_at'} over resolved events.
|
||||
"""
|
||||
key = (event_type, target_name, target_detail or '')
|
||||
with get_conn() as conn:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
"""SELECT COALESCE(SUM(first_seen > DATE_SUB(NOW(), INTERVAL 7 DAY)), 0) AS last_7d,
|
||||
COUNT(*) AS last_30d
|
||||
FROM network_events
|
||||
WHERE event_type=%s AND target_name=%s AND target_detail=%s
|
||||
AND resolved_at IS NOT NULL AND first_seen > DATE_SUB(NOW(), INTERVAL 30 DAY)""",
|
||||
key,
|
||||
)
|
||||
counts = cur.fetchone() or {}
|
||||
cur.execute(
|
||||
"""SELECT first_seen, resolved_at FROM network_events
|
||||
WHERE event_type=%s AND target_name=%s AND target_detail=%s
|
||||
AND resolved_at IS NOT NULL ORDER BY resolved_at DESC LIMIT 1""",
|
||||
key,
|
||||
)
|
||||
last = cur.fetchone() or {}
|
||||
return {
|
||||
'last_7d': int(counts.get('last_7d') or 0),
|
||||
'last_30d': int(counts.get('last_30d') or 0),
|
||||
'last_first_seen': last.get('first_seen'),
|
||||
'last_resolved_at': last.get('resolved_at'),
|
||||
}
|
||||
|
||||
|
||||
def _history_where(since_hours: int, event_type: str = '', target: str = '',
|
||||
severity: str = '', status: str = 'all') -> tuple:
|
||||
"""WHERE clause shared by the history list and its summaries.
|
||||
|
||||
An event is in range if it was open at any point in the window: it started
|
||||
in it, or it was still open when the window began.
|
||||
"""
|
||||
clauses = ['(first_seen > DATE_SUB(NOW(), INTERVAL %s HOUR) OR '
|
||||
'resolved_at IS NULL OR resolved_at > DATE_SUB(NOW(), INTERVAL %s HOUR))']
|
||||
params: list = [since_hours, since_hours]
|
||||
if event_type:
|
||||
clauses.append('event_type=%s')
|
||||
params.append(event_type)
|
||||
if severity:
|
||||
clauses.append('severity=%s')
|
||||
params.append(severity)
|
||||
if target:
|
||||
like = '%' + target.replace('\\', '\\\\').replace('%', '\\%').replace('_', '\\_') + '%'
|
||||
clauses.append('(target_name LIKE %s OR target_detail LIKE %s)')
|
||||
params.extend([like, like])
|
||||
if status == 'active':
|
||||
clauses.append('resolved_at IS NULL')
|
||||
elif status == 'resolved':
|
||||
clauses.append('resolved_at IS NOT NULL')
|
||||
return ' AND '.join(clauses), params
|
||||
|
||||
|
||||
def _iso(rows: list, keys=('first_seen', 'last_seen', 'resolved_at')) -> list:
|
||||
for r in rows:
|
||||
for k in keys:
|
||||
if r.get(k) and hasattr(r[k], 'isoformat'):
|
||||
r[k] = r[k].isoformat() + 'Z'
|
||||
return rows
|
||||
|
||||
|
||||
def get_events_history(since_hours: int, event_type: str = '', target: str = '', severity: str = '',
|
||||
status: str = 'all', limit: int = 100, offset: int = 0) -> dict:
|
||||
"""Events open at any time in the last `since_hours`, newest first, with totals.
|
||||
|
||||
Returns {'events', 'total', 'by_type': {type: count}}.
|
||||
"""
|
||||
where, params = _history_where(since_hours, event_type, target, severity, status)
|
||||
with get_conn() as conn:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
f"""SELECT id, event_type, severity, source_type, target_name, target_detail, description,
|
||||
first_seen, last_seen, resolved_at, consecutive_failures, ticket_id,
|
||||
TIMESTAMPDIFF(SECOND, first_seen, COALESCE(resolved_at, NOW())) AS duration_s
|
||||
FROM network_events WHERE {where}
|
||||
ORDER BY first_seen DESC, id DESC LIMIT %s OFFSET %s""", # nosec B608 - fixed clauses
|
||||
params + [limit, offset],
|
||||
)
|
||||
events = _iso(list(cur.fetchall()))
|
||||
cur.execute(
|
||||
f"""SELECT event_type, COUNT(*) AS n
|
||||
FROM network_events WHERE {where} GROUP BY event_type ORDER BY n DESC""", # nosec B608
|
||||
params,
|
||||
)
|
||||
groups = cur.fetchall()
|
||||
return {
|
||||
'events': events,
|
||||
'total': sum(int(g['n']) for g in groups),
|
||||
'by_type': {g['event_type']: int(g['n']) for g in groups},
|
||||
}
|
||||
|
||||
|
||||
def get_flappers(since_hours: int, event_type: str = '', target: str = '', severity: str = '',
|
||||
min_count: int = 2, limit: int = 10) -> list:
|
||||
"""The alerts that fired most often in the window: the ones worth a root cause."""
|
||||
where, params = _history_where(since_hours, event_type, target, severity)
|
||||
with get_conn() as conn:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
f"""SELECT event_type, target_name, target_detail, COUNT(*) AS n,
|
||||
SUM(TIMESTAMPDIFF(SECOND, first_seen, COALESCE(resolved_at, NOW()))) AS total_s,
|
||||
MAX(first_seen) AS last_seen, SUM(resolved_at IS NULL) AS open_now
|
||||
FROM network_events WHERE {where}
|
||||
GROUP BY event_type, target_name, target_detail
|
||||
HAVING n >= %s ORDER BY n DESC, last_seen DESC LIMIT %s""", # nosec B608 - fixed clauses
|
||||
params + [min_count, limit],
|
||||
)
|
||||
rows = list(cur.fetchall())
|
||||
for r in rows:
|
||||
r['n'], r['total_s'], r['open_now'] = int(r['n']), int(r['total_s'] or 0), int(r['open_now'] or 0)
|
||||
return _iso(rows, ('last_seen',))
|
||||
|
||||
|
||||
def get_event_types() -> list:
|
||||
with get_conn() as conn:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute('SELECT DISTINCT event_type FROM network_events ORDER BY event_type')
|
||||
return [r['event_type'] for r in cur.fetchall()]
|
||||
|
||||
|
||||
def purge_old_resolved_events(days: int = 90) -> int:
|
||||
"""Delete resolved events older than `days` days. Returns count deleted."""
|
||||
with get_conn() as conn:
|
||||
@@ -553,8 +722,38 @@ _CUTSHEET_SCHEMA = (
|
||||
INDEX idx_hist_time (changed_at),
|
||||
INDEX idx_hist_entity (entity, entity_id)
|
||||
) ENGINE=InnoDB""",
|
||||
# Where a device lives in the asset inventory (HomeBox), set by hand on its Gandalf page
|
||||
"""CREATE TABLE IF NOT EXISTS device_links (
|
||||
name VARCHAR(255) NOT NULL PRIMARY KEY,
|
||||
inventory_url VARCHAR(512) NOT NULL,
|
||||
updated_by VARCHAR(255) NOT NULL,
|
||||
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
|
||||
) ENGINE=InnoDB""",
|
||||
)
|
||||
|
||||
|
||||
def get_device_link(name: str) -> str:
|
||||
"""The device's inventory URL, or ''."""
|
||||
with get_conn() as conn:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute('SELECT inventory_url FROM device_links WHERE name=%s', (name,))
|
||||
row = cur.fetchone()
|
||||
return row['inventory_url'] if row else ''
|
||||
|
||||
|
||||
def set_device_link(name: str, inventory_url: str, user: str) -> None:
|
||||
"""Save a device's inventory URL; an empty URL removes it."""
|
||||
with get_conn() as conn:
|
||||
with conn.cursor() as cur:
|
||||
if not inventory_url:
|
||||
cur.execute('DELETE FROM device_links WHERE name=%s', (name,))
|
||||
return
|
||||
cur.execute(
|
||||
"""INSERT INTO device_links (name, inventory_url, updated_by) VALUES (%s, %s, %s)
|
||||
ON DUPLICATE KEY UPDATE inventory_url=VALUES(inventory_url), updated_by=VALUES(updated_by)""",
|
||||
(name, inventory_url, user),
|
||||
)
|
||||
|
||||
_ITEM_FIELDS = ('name', 'kind', 'port_count', 'location', 'rack_u', 'rack_height', 'notes')
|
||||
_CABLE_FIELDS = ('label', 'a_device', 'a_port', 'a_side', 'b_device', 'b_port', 'b_side',
|
||||
'cable_type', 'length_m', 'color', 'status', 'label_applied', 'notes')
|
||||
|
||||
+353
-128
@@ -15,7 +15,7 @@ import statistics
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from typing import Dict, List, Optional
|
||||
from urllib.parse import urlparse
|
||||
from urllib.parse import urljoin, urlparse
|
||||
|
||||
import requests
|
||||
from urllib3.exceptions import InsecureRequestWarning
|
||||
@@ -24,6 +24,7 @@ import db
|
||||
import cutsheet
|
||||
import devices
|
||||
import modem as modem_api
|
||||
import ticket_text
|
||||
|
||||
|
||||
logging.basicConfig(
|
||||
@@ -388,6 +389,41 @@ class TicketClient:
|
||||
def __init__(self, cfg: dict):
|
||||
self.url = cfg.get('url', '')
|
||||
self.api_key = cfg.get('api_key', '')
|
||||
# Tinker serves its other API scripts under api/ beside create_ticket_api.php
|
||||
self.comment_url = cfg.get('comment_url') or (
|
||||
urljoin(self.url, 'api/ticket_comment_api.php') if self.url else '')
|
||||
self.comments_denied = False
|
||||
|
||||
def comment(self, ticket_id: str, text: str) -> bool:
|
||||
"""Add a Markdown comment to a ticket. Never raises; False means try again later."""
|
||||
if not self.api_key or not self.comment_url or self.comments_denied:
|
||||
return False
|
||||
try:
|
||||
resp = requests.post(
|
||||
self.comment_url,
|
||||
json={'ticket_id': str(ticket_id), 'comment_text': text, 'markdown_enabled': True},
|
||||
headers={'Authorization': f'Bearer {self.api_key}'},
|
||||
timeout=15,
|
||||
)
|
||||
if resp.status_code in (401, 403):
|
||||
# Retrying every cycle can't fix a key without write access, so stop until restart
|
||||
self.comments_denied = True
|
||||
logger.error(
|
||||
f'Ticket API refused a comment on #{ticket_id} (HTTP {resp.status_code}). '
|
||||
f'The API key needs read_write scope; recovery notes are off until the monitor restarts.'
|
||||
)
|
||||
return False
|
||||
if resp.status_code == 404:
|
||||
logger.warning(f'Ticket #{ticket_id} no longer exists; skipping its recovery note')
|
||||
return True
|
||||
resp.raise_for_status()
|
||||
if resp.json().get('success'):
|
||||
logger.info(f'Commented on ticket #{ticket_id}')
|
||||
return True
|
||||
logger.warning(f'Unexpected comment API response for #{ticket_id}: {resp.text[:200]}')
|
||||
except Exception as e:
|
||||
logger.error(f'Ticket comment failed for #{ticket_id}: {e}')
|
||||
return False
|
||||
|
||||
def create(self, title: str, description: str, priority: str = '2') -> Optional[str]:
|
||||
if not self.api_key or not self.url:
|
||||
@@ -893,10 +929,21 @@ def _now_utc() -> str:
|
||||
return datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M:%S UTC')
|
||||
|
||||
|
||||
def _utcnow() -> datetime:
|
||||
"""Naive UTC, to compare with database times."""
|
||||
return datetime.now(timezone.utc).replace(tzinfo=None)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Monitor
|
||||
# --------------------------------------------------------------------------
|
||||
CLUSTER_NAME = 'proxmox-cluster'
|
||||
DEFAULT_PUBLIC_URL = 'https://gandalf.lotusguild.org'
|
||||
# Reports of something already over: the ticket is the whole story, no recovery note
|
||||
ONE_SHOT_EVENTS = ('modem_power_cycled', 'wan_unstable')
|
||||
# Ports a host or device was last seen on, kept so a ticket for a dead host
|
||||
# (whose link, and so its LLDP entry, is gone) can still say where it plugs in
|
||||
KNOWN_UPLINKS_STATE = 'known_uplinks'
|
||||
|
||||
|
||||
class NetworkMonitor:
|
||||
@@ -950,6 +997,16 @@ class NetworkMonitor:
|
||||
db.ensure_cutsheet_schema()
|
||||
except Exception as e:
|
||||
logger.error(f'Could not create cutsheet tables: {e}')
|
||||
try:
|
||||
db.ensure_events_schema()
|
||||
except Exception as e:
|
||||
logger.error(f'Could not update the events table: {e}')
|
||||
|
||||
# Absolute links in tickets point here
|
||||
self.public_url = ((self.cfg.get('gandalf') or {}).get('public_url')
|
||||
or DEFAULT_PUBLIC_URL).rstrip('/')
|
||||
self._link_data: Optional[dict] = None
|
||||
self._known_uplinks: Optional[dict] = None
|
||||
|
||||
# Last good Prometheus result, shown (as unknown) during a Prometheus outage
|
||||
self._last_iface_states: Dict[str, Dict[str, bool]] = {}
|
||||
@@ -964,6 +1021,174 @@ class NetworkMonitor:
|
||||
def _hostname(self, instance: str) -> str:
|
||||
return self._instance_map.get(instance, instance.split(':')[0])
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Tickets: context, and notes when an alert clears
|
||||
# ------------------------------------------------------------------
|
||||
def _base_url(self) -> str:
|
||||
return getattr(self, 'public_url', DEFAULT_PUBLIC_URL)
|
||||
|
||||
def _open_ticket(
|
||||
self, event_id: int, event_type: str, target: str, detail: str, *,
|
||||
title: str, heading: str, facts: list, summary: str, checks: list, priority: str,
|
||||
host: Optional[str] = None, port: Optional[tuple] = None,
|
||||
) -> None:
|
||||
"""Open a ticket with the alert's facts plus everything Gandalf knows around it.
|
||||
|
||||
host: a device whose switch ports to list. port: (switch, port_idx) the alert is about.
|
||||
"""
|
||||
sections = self._ticket_sections(event_id, event_type, target, detail, host=host, port=port)
|
||||
desc = ticket_text.description(heading, facts, summary, checks, sections)
|
||||
tid = self.tickets.create(title, desc, priority=priority)
|
||||
if tid:
|
||||
db.set_ticket_id(event_id, tid)
|
||||
|
||||
def _ticket_sections(self, event_id: int, event_type: str, target: str, detail: str,
|
||||
host: Optional[str] = None, port: Optional[tuple] = None) -> list:
|
||||
"""Each section is optional: a failure loses that section, never the ticket."""
|
||||
sections = []
|
||||
for title, build in (
|
||||
('Where it is connected', lambda: self._where_lines(event_type, detail, host, port)),
|
||||
('History', lambda: ticket_text.history_lines(db.event_stats(event_type, target, detail), _utcnow())),
|
||||
('Other active alerts here', lambda: self._other_alert_lines(event_id, {target, host})),
|
||||
('Links', lambda: self._link_lines(host or (port[0] if port else None))),
|
||||
):
|
||||
try:
|
||||
sections.append((title, build()))
|
||||
except Exception as e:
|
||||
logger.error(f'Ticket section "{title}" for {target} failed: {e}', exc_info=True)
|
||||
return sections
|
||||
|
||||
def _switches(self) -> dict:
|
||||
return (getattr(self, '_link_data', None) or {}).get('unifi_switches') or {}
|
||||
|
||||
def _remember_uplinks(self, link_data: dict) -> None:
|
||||
"""Record where every LLDP-visible device is plugged in, for when it disappears."""
|
||||
known = self._load_known_uplinks()
|
||||
switches = link_data.get('unifi_switches') or {}
|
||||
names = {(p.get('lldp') or {}).get('system_name')
|
||||
for sw in switches.values() for p in (sw.get('ports') or {}).values()}
|
||||
now = _now_utc()
|
||||
changed = False
|
||||
for name in sorted(n for n in names if n):
|
||||
ups = devices.uplinks_for_host(name, switches)
|
||||
entry = {'seen': now, 'uplinks': [
|
||||
{k: u[k] for k in ('switch', 'port', 'port_idx', 'host_interface')} for u in ups]}
|
||||
if (known.get(name) or {}).get('uplinks') != entry['uplinks']:
|
||||
changed = True
|
||||
known[name] = entry
|
||||
self._known_uplinks = known
|
||||
# Write when wiring changes, and hourly so "last seen" stays roughly right
|
||||
last_saved = getattr(self, '_known_uplinks_saved', 0.0)
|
||||
if changed or time.time() - last_saved > 3600:
|
||||
db.set_state(KNOWN_UPLINKS_STATE, known)
|
||||
self._known_uplinks_saved = time.time()
|
||||
|
||||
def _load_known_uplinks(self) -> dict:
|
||||
if getattr(self, '_known_uplinks', None) is None:
|
||||
self._known_uplinks = {}
|
||||
raw = db.get_state(KNOWN_UPLINKS_STATE)
|
||||
if raw:
|
||||
try:
|
||||
self._known_uplinks = json.loads(raw)
|
||||
except ValueError:
|
||||
logger.error('Failed to parse known_uplinks JSON; starting over')
|
||||
return self._known_uplinks
|
||||
|
||||
def _uplinks_for(self, name: str, switches: dict) -> tuple:
|
||||
"""(uplinks, None) if the device is linked now, else (last known uplinks, when seen)."""
|
||||
current = devices.uplinks_for_host(name, switches)
|
||||
if current:
|
||||
return current, None
|
||||
entry = self._load_known_uplinks().get(name) or {}
|
||||
return entry.get('uplinks') or [], entry.get('seen')
|
||||
|
||||
def _where_lines(self, event_type: str, detail: str, host: Optional[str], port: Optional[tuple]) -> list:
|
||||
base, switches = self._base_url(), self._switches()
|
||||
lines = []
|
||||
if host:
|
||||
ups, seen = self._uplinks_for(host, switches)
|
||||
if event_type == 'interface_down':
|
||||
ups = [u for u in ups if u.get('host_interface') == detail] or ups
|
||||
for u in ups:
|
||||
live = (switches.get(u['switch'], {}).get('ports') or {}).get(u['port'])
|
||||
text = ticket_text.port_text(u['switch'], u['port'], live or dict(u, up=False))
|
||||
iface = f'`{u["host_interface"]}` → ' if u.get('host_interface') else ''
|
||||
lines.append(f'- {iface}' + ticket_text.link(
|
||||
text, ticket_text.device_url(base, u['switch'], u.get('port_idx'))))
|
||||
if seen:
|
||||
lines[-1] += f' _(last seen linked there {seen})_'
|
||||
if not ups:
|
||||
lines.append('- Gandalf has not seen this device on a switch port (no LLDP or client entry).')
|
||||
if port:
|
||||
switch, idx = port
|
||||
live = next((dict(p, name=n) for n, p in (switches.get(switch, {}).get('ports') or {}).items()
|
||||
if str(p.get('port_idx')) == str(idx)), None)
|
||||
if live:
|
||||
lines.append('- ' + ticket_text.link(ticket_text.port_text(switch, live['name'], live),
|
||||
ticket_text.device_url(base, switch, idx)))
|
||||
device = host or (port[0] if port else None)
|
||||
if device:
|
||||
data = db.cutsheet_load()
|
||||
items = {i['name']: i for i in data['items']}
|
||||
runs = cutsheet.runs_for(device, data['cables'], items, switches,
|
||||
port=str(port[1]) if port else None)
|
||||
for r in runs:
|
||||
label = ticket_text.link(f'**{r["label"]}**', ticket_text.cutsheet_url(base, r['cable_id']))
|
||||
lines.append(f'- Cable {label}: {r["path"]}. Cutsheet check: '
|
||||
f'{cutsheet.VERDICTS.get(r["verdict"], r["verdict"]).lower()}.')
|
||||
if not runs:
|
||||
lines.append('- ' + ticket_text.link('Not on the cutsheet yet', f'{base}/cutsheet' if base else ''))
|
||||
return lines
|
||||
|
||||
def _other_alert_lines(self, event_id: int, names: set) -> list:
|
||||
lines = []
|
||||
for name in sorted(n for n in names if n):
|
||||
for e in db.get_events_for_target(name, limit=25):
|
||||
if e.get('resolved_at') or e.get('id') == event_id:
|
||||
continue
|
||||
what = e['event_type'].replace('_', ' ')
|
||||
where = f'{e["target_name"]} {e.get("target_detail") or ""}'.strip()
|
||||
ticket = f' (ticket #{e["ticket_id"]})' if e.get('ticket_id') else ''
|
||||
lines.append(f'- {what} on {where} since {e.get("first_seen", "?")}{ticket}')
|
||||
return lines[:10]
|
||||
|
||||
def _link_lines(self, device: Optional[str]) -> list:
|
||||
base = self._base_url()
|
||||
if not base:
|
||||
return []
|
||||
lines = []
|
||||
if device:
|
||||
lines.append(f'- {ticket_text.link(f"{device} in Gandalf", ticket_text.device_url(base, device))}')
|
||||
lines.append(f'- {ticket_text.link(f"Alert history for {device}", ticket_text.history_url(base, device))}')
|
||||
inventory = db.get_device_link(device)
|
||||
if inventory:
|
||||
lines.append(f'- {ticket_text.link(f"{device} in the inventory", inventory)}')
|
||||
lines.append(f'- {ticket_text.link("Gandalf dashboard", base + "/")}')
|
||||
return lines
|
||||
|
||||
def _event_url(self, event: dict) -> str:
|
||||
base = self._base_url()
|
||||
if not base:
|
||||
return ''
|
||||
if event['event_type'] == 'cable_mismatch':
|
||||
cable_id = str(event.get('target_detail') or '').replace('cable ', '')
|
||||
return ticket_text.cutsheet_url(base, cable_id)
|
||||
if event['event_type'] in ('cluster_network_issue', 'monitor_source_unavailable'):
|
||||
return base + '/'
|
||||
return ticket_text.device_url(base, event['target_name'])
|
||||
|
||||
def _post_recovery_notes(self) -> None:
|
||||
"""Comment on the tickets of alerts that have cleared. Failures retry next cycle (for 24 h)."""
|
||||
if self.tickets.comments_denied:
|
||||
return
|
||||
now = _utcnow()
|
||||
for event in db.get_pending_recovery_notes(hours=24, exclude_types=ONE_SHOT_EVENTS):
|
||||
text = ticket_text.recovery_comment(event, now, self._event_url(event))
|
||||
if self.tickets.comment(event['ticket_id'], text):
|
||||
db.mark_recovery_notified(event['id'])
|
||||
elif self.tickets.comments_denied:
|
||||
break
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Interface monitoring (Prometheus)
|
||||
# ------------------------------------------------------------------
|
||||
@@ -1041,47 +1266,54 @@ class NetworkMonitor:
|
||||
f'{", ".join(hosts_with_regression)}',
|
||||
)
|
||||
if not sup and (is_new or not ticket_id):
|
||||
title = (
|
||||
f'[{self.cluster_name}][auto][production][issue][network][cluster-wide] '
|
||||
f'Multiple hosts reporting interface failures'
|
||||
)
|
||||
desc = (
|
||||
f'Cluster Network Alert\n{"=" * 40}\n\n'
|
||||
f'Affected hosts: {", ".join(hosts_with_regression)}\n'
|
||||
f'Detected: {_now_utc()}\n\n'
|
||||
f'{len(hosts_with_regression)} Proxmox hosts simultaneously reported '
|
||||
f'interface regressions (link-down on interfaces previously known UP) '
|
||||
f'or stopped reporting metrics.\n'
|
||||
f'This likely indicates a switch or upstream network failure.\n\n'
|
||||
f'Please check the core and management switches immediately.'
|
||||
)
|
||||
tid = self.tickets.create(title, desc, priority='1')
|
||||
if tid:
|
||||
db.set_ticket_id(event_id, tid)
|
||||
self._ticket_cluster(event_id, hosts_with_regression)
|
||||
else:
|
||||
db.resolve_event('cluster_network_issue', self.cluster_name, '')
|
||||
|
||||
def _ticket_cluster(self, event_id: int, hosts: List[str]) -> None:
|
||||
switches = self._switches()
|
||||
# The switches these hosts hang off are the likeliest shared cause
|
||||
shared = sorted({u['switch'] for h in hosts for u in self._uplinks_for(h, switches)[0]})
|
||||
self._open_ticket(
|
||||
event_id, 'cluster_network_issue', self.cluster_name, '',
|
||||
title=(f'[{self.cluster_name}][auto][production][issue][network][cluster-wide] '
|
||||
f'Multiple hosts reporting interface failures'),
|
||||
heading='Cluster Network Alert',
|
||||
facts=[('Affected hosts', ', '.join(hosts)),
|
||||
('Switches they connect through', ', '.join(shared)),
|
||||
('Detected', _now_utc())],
|
||||
summary=(f'{len(hosts)} Proxmox hosts reported interface regressions (link-down on '
|
||||
f'interfaces previously known up) or stopped reporting metrics at the same time. '
|
||||
f'This usually means a switch or upstream network failure rather than the hosts.'),
|
||||
checks=['Check the core and management switches (power, uplinks, recent firmware updates).',
|
||||
'Check whether the hosts share a PDU or circuit that lost power.',
|
||||
'Look at the per-host tickets and the Gandalf dashboard for which links went down first.'],
|
||||
priority='1',
|
||||
)
|
||||
|
||||
def _ticket_metrics_missing(
|
||||
self, event_id: int, host: str, instance: str, consec: int
|
||||
) -> None:
|
||||
title = (
|
||||
f'[{host}][auto][production][issue][network][single-node] '
|
||||
f'No metrics from host'
|
||||
self._open_ticket(
|
||||
event_id, 'host_metrics_missing', host, instance,
|
||||
title=f'[{host}][auto][production][issue][network][single-node] No metrics from host',
|
||||
heading='Host Metrics Alert',
|
||||
facts=[('Host', host), ('Prometheus instance', instance), ('Detected', _now_utc()),
|
||||
('Consecutive check failures', str(consec)),
|
||||
('Down for at least', ticket_text.duration(consec * self._poll_seconds()))],
|
||||
summary=(f'Prometheus is working but has no node_network_up series for {host}. '
|
||||
f'Either the host is down or node_exporter has stopped on it. '
|
||||
f'Gandalf waits {self.host_fail_thresh} checks before opening this, so a normal '
|
||||
f'reboot should not cause it.'),
|
||||
checks=[f'Is {host} powered on and reachable (Proxmox UI, ping, console)?',
|
||||
'If it is up, check `systemctl status prometheus-node-exporter` on the host.',
|
||||
'If it is down, check its switch port below: a dark port with no PoE or link '
|
||||
'points to power or cabling.'],
|
||||
priority='2', host=host,
|
||||
)
|
||||
desc = (
|
||||
f'Host Metrics Alert\n{"=" * 40}\n\n'
|
||||
f'Host: {host}\n'
|
||||
f'Prometheus instance: {instance}\n'
|
||||
f'Detected: {_now_utc()}\n'
|
||||
f'Consecutive check failures: {consec}\n\n'
|
||||
f'Prometheus is working but has no node_network_up series for {host}.\n'
|
||||
f'Either the host is down or node_exporter has stopped on it.\n\n'
|
||||
f'Please check the host power and network connectivity, then '
|
||||
f'prometheus-node-exporter on the host.'
|
||||
)
|
||||
tid = self.tickets.create(title, desc, priority='2')
|
||||
if tid:
|
||||
db.set_ticket_id(event_id, tid)
|
||||
|
||||
def _poll_seconds(self) -> int:
|
||||
return int(getattr(self, 'poll_interval', 120) or 120)
|
||||
|
||||
def _process_prometheus_outage(self, suppressions: list) -> None:
|
||||
"""Interface and host alerts can't fire without Prometheus, so say so."""
|
||||
@@ -1092,44 +1324,38 @@ class NetworkMonitor:
|
||||
f'Prometheus returned no interface data; host and interface alerting is paused ({_now_utc()})',
|
||||
)
|
||||
if not sup and consec >= self.fail_thresh and not ticket_id:
|
||||
title = (
|
||||
'[prometheus][auto][production][issue][network][cluster-wide] '
|
||||
'Gandalf has no Prometheus data'
|
||||
self._open_ticket(
|
||||
event_id, 'monitor_source_unavailable', 'prometheus', '',
|
||||
title='[prometheus][auto][production][issue][network][cluster-wide] Gandalf has no Prometheus data',
|
||||
heading='Monitoring Source Alert',
|
||||
facts=[('Prometheus URL', self.prom.url), ('Detected', _now_utc()),
|
||||
('Consecutive check failures', str(consec))],
|
||||
summary=('Gandalf could not get node_network_up data from Prometheus. While this lasts, '
|
||||
'host and interface failures will **not** be detected. UniFi, ping and modem '
|
||||
'checks keep running.'),
|
||||
checks=['Is the Prometheus service running on its host?',
|
||||
'Open Prometheus → Status → Targets and check the node exporters are being scraped.',
|
||||
'Check disk space on the Prometheus host (a full disk stops ingestion).'],
|
||||
priority='2',
|
||||
)
|
||||
desc = (
|
||||
f'Monitoring Source Alert\n{"=" * 40}\n\n'
|
||||
f'Prometheus URL: {self.prom.url}\n'
|
||||
f'Detected: {_now_utc()}\n'
|
||||
f'Consecutive check failures: {consec}\n\n'
|
||||
f'Gandalf could not get node_network_up data from Prometheus.\n'
|
||||
f'While this lasts, host and interface failures will NOT be detected.\n\n'
|
||||
f'Please check the Prometheus service and its scrape targets.'
|
||||
)
|
||||
tid = self.tickets.create(title, desc, priority='2')
|
||||
if tid:
|
||||
db.set_ticket_id(event_id, tid)
|
||||
|
||||
def _ticket_interface(
|
||||
self, event_id: int, host: str, iface: str, consec: int
|
||||
) -> None:
|
||||
title = (
|
||||
f'[{host}][auto][production][issue][network][single-node] '
|
||||
f'Interface {iface} link-down'
|
||||
self._open_ticket(
|
||||
event_id, 'interface_down', host, iface,
|
||||
title=f'[{host}][auto][production][issue][network][single-node] Interface {iface} link-down',
|
||||
heading='Network Interface Alert',
|
||||
facts=[('Host', host), ('Interface', f'`{iface}`'), ('Detected', _now_utc()),
|
||||
('Consecutive check failures', str(consec))],
|
||||
summary=(f'Interface {iface} on {host} was up and is now reporting link-down via '
|
||||
f'Prometheus node_exporter. {host} itself is still reporting, so it may be '
|
||||
f'reachable over its other interfaces.'),
|
||||
checks=['Check the switch port below: is it down on the switch side too?',
|
||||
'Reseat or swap the cable or DAC; for an optic, check the module for a fault.',
|
||||
f'On {host}, check `ip link show {iface}` and `dmesg` for NIC or driver errors.'],
|
||||
priority='2', host=host,
|
||||
)
|
||||
desc = (
|
||||
f'Network Interface Alert\n{"=" * 40}\n\n'
|
||||
f'Host: {host}\n'
|
||||
f'Interface: {iface}\n'
|
||||
f'Detected: {_now_utc()}\n'
|
||||
f'Consecutive check failures: {consec}\n\n'
|
||||
f'Interface {iface} on {host} is reporting link-down state via '
|
||||
f'Prometheus node_exporter.\n\n'
|
||||
f'Note: {host} may still be reachable via its other network interface.\n'
|
||||
f'Please inspect the cable/SFP/switch port for {host}/{iface}.'
|
||||
)
|
||||
tid = self.tickets.create(title, desc, priority='2')
|
||||
if tid:
|
||||
db.set_ticket_id(event_id, tid)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# UniFi device monitoring
|
||||
@@ -1155,23 +1381,20 @@ class NetworkMonitor:
|
||||
|
||||
def _ticket_unifi(self, event_id: int, device: dict) -> None:
|
||||
name = device['name']
|
||||
title = (
|
||||
f'[{name}][auto][production][issue][network][single-node] '
|
||||
f'UniFi device offline'
|
||||
self._open_ticket(
|
||||
event_id, 'unifi_device_offline', name, device.get('type', ''),
|
||||
title=f'[{name}][auto][production][issue][network][single-node] UniFi device offline',
|
||||
heading='UniFi Device Alert',
|
||||
facts=[('Device', name), ('Type', device.get('type', 'unknown')), ('Model', device.get('model', '')),
|
||||
('Last known IP', device.get('ip', 'unknown')), ('Detected', _now_utc())],
|
||||
summary=(f'The UniFi controller reports {name} as offline. Gandalf waits '
|
||||
f'{self.unifi_fail_thresh} checks first, so a firmware-update reboot should not cause this.'),
|
||||
checks=['Check the device for power (status LED) and, if it is PoE powered, the PoE '
|
||||
'budget on the port below.',
|
||||
'Check the uplink cable and the switch port it plugs into.',
|
||||
'In the UniFi controller, check whether the device is stuck adopting or upgrading.'],
|
||||
priority='2', host=name,
|
||||
)
|
||||
desc = (
|
||||
f'UniFi Device Alert\n{"=" * 40}\n\n'
|
||||
f'Device: {name}\n'
|
||||
f'Type: {device.get("type","unknown")}\n'
|
||||
f'Model: {device.get("model","")}\n'
|
||||
f'Last Known IP: {device.get("ip","unknown")}\n'
|
||||
f'Detected: {_now_utc()}\n\n'
|
||||
f'The UniFi device {name} is offline per the UniFi controller.\n'
|
||||
f'Please check power and cable connectivity.'
|
||||
)
|
||||
tid = self.tickets.create(title, desc, priority='2')
|
||||
if tid:
|
||||
db.set_ticket_id(event_id, tid)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Cutsheet: documented cabling vs. what discovery sees
|
||||
@@ -1206,24 +1429,20 @@ class NetworkMonitor:
|
||||
db.resolve_event('cable_mismatch', target_name, target_detail)
|
||||
|
||||
def _ticket_cable_mismatch(self, event_id: int, p: dict, hours: float) -> None:
|
||||
title = (
|
||||
f'[{p["switch"]}][auto][production][issue][network][single-node] '
|
||||
f'Cable {p["label"]} does not match the cutsheet'
|
||||
url = ticket_text.cutsheet_url(self._base_url(), p['cable_id'])
|
||||
self._open_ticket(
|
||||
event_id, 'cable_mismatch', p['switch'], p['key'],
|
||||
title=(f'[{p["switch"]}][auto][production][issue][network][single-node] '
|
||||
f'Cable {p["label"]} does not match the cutsheet'),
|
||||
heading='Cutsheet Mismatch',
|
||||
facts=[('Cable', ticket_text.link(p['label'], url)), ('Switch port', f'{p["switch"]} port {p["port"]}'),
|
||||
('Problem', p['detail']), ('Documented path', p['path']), ('Detected', _now_utc())],
|
||||
summary=f'This has not matched the cutsheet for at least {hours:g} hours.',
|
||||
checks=['If the cable was moved on purpose, update its record in the cutsheet.',
|
||||
'Otherwise, follow the label and check it is plugged into the documented port '
|
||||
'(Trace mode on the cutsheet page can find which port a cord is in).'],
|
||||
priority='4', port=(p['switch'], p['port']),
|
||||
)
|
||||
desc = (
|
||||
f'Cutsheet Mismatch\n{"=" * 40}\n\n'
|
||||
f'Cable: {p["label"]}\n'
|
||||
f'Switch port: {p["switch"]} port {p["port"]}\n'
|
||||
f'Problem: {p["detail"]}\n'
|
||||
f'Documented path: {p["path"]}\n'
|
||||
f'Detected: {_now_utc()}\n\n'
|
||||
f'This has not matched for at least {hours:g} hours.\n\n'
|
||||
f'Either the cable was moved (update the cutsheet in Gandalf: /cutsheet#cable-{p["cable_id"]}) '
|
||||
f'or it is plugged into the wrong place.'
|
||||
)
|
||||
tid = self.tickets.create(title, desc, priority='4')
|
||||
if tid:
|
||||
db.set_ticket_id(event_id, tid)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Ping-only hosts (no node_exporter)
|
||||
@@ -1253,24 +1472,20 @@ class NetworkMonitor:
|
||||
def _ticket_unreachable(
|
||||
self, event_id: int, name: str, ip: str, consec: int
|
||||
) -> None:
|
||||
title = (
|
||||
f'[{name}][auto][production][issue][network][single-node] '
|
||||
f'Host unreachable'
|
||||
self._open_ticket(
|
||||
event_id, 'host_unreachable', name, ip,
|
||||
title=f'[{name}][auto][production][issue][network][single-node] Host unreachable',
|
||||
heading='Host Reachability Alert',
|
||||
facts=[('Host', name), ('IP', ip), ('Detected', _now_utc()),
|
||||
('Consecutive check failures', str(consec)),
|
||||
('Down for at least', ticket_text.duration(consec * self._poll_seconds()))],
|
||||
summary=(f'{name} ({ip}) is not answering ping from the Pulse worker Gandalf uses. '
|
||||
f'It has no node_exporter, so ping is the only check Gandalf has for it.'),
|
||||
checks=[f'Is {name} powered on? Check its console or management interface.',
|
||||
'Check its switch port below for link and PoE.',
|
||||
'If the port is up, check the IP and firewall on the device itself.'],
|
||||
priority='2', host=name,
|
||||
)
|
||||
desc = (
|
||||
f'Host Reachability Alert\n{"=" * 40}\n\n'
|
||||
f'Host: {name}\n'
|
||||
f'IP: {ip}\n'
|
||||
f'Detected: {_now_utc()}\n'
|
||||
f'Consecutive check failures: {consec}\n\n'
|
||||
f'Host {name} ({ip}) is not responding to ping from the Gandalf monitor.\n'
|
||||
f'This host does not have a Prometheus node_exporter, so interface-level '
|
||||
f'detail is unavailable.\n\n'
|
||||
f'Please check the host power, management interface, and network connectivity.'
|
||||
)
|
||||
tid = self.tickets.create(title, desc, priority='2')
|
||||
if tid:
|
||||
db.set_ticket_id(event_id, tid)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# WAN modem monitoring
|
||||
@@ -1291,17 +1506,14 @@ class NetworkMonitor:
|
||||
event_type, severity, 'modem', name, detail, f'{description} ({_now_utc()})',
|
||||
)
|
||||
if not sup and consec >= self.modem_fail_thresh and not ticket_id:
|
||||
title = f'[{name}][auto][production][issue][network][cluster-wide] {heading}'
|
||||
desc = (
|
||||
f'WAN Modem Alert\n{"=" * 40}\n\n'
|
||||
f'Modem: {name} ({self.modem.base_url})\n'
|
||||
f'Detected: {_now_utc()}\n'
|
||||
f'Consecutive check failures: {consec}\n\n'
|
||||
f'{description}\n\n{advice}'
|
||||
self._open_ticket(
|
||||
event_id, event_type, name, detail,
|
||||
title=f'[{name}][auto][production][issue][network][cluster-wide] {heading}',
|
||||
heading='WAN Modem Alert',
|
||||
facts=[('Modem', f'{name} ({self.modem.base_url})'), ('Detected', _now_utc()),
|
||||
('Consecutive check failures', str(consec))],
|
||||
summary=description, checks=[advice], priority=priority,
|
||||
)
|
||||
tid = self.tickets.create(title, desc, priority=priority)
|
||||
if tid:
|
||||
db.set_ticket_id(event_id, tid)
|
||||
|
||||
def _process_modem_failure(self, suppressions: list, error: Exception) -> None:
|
||||
"""Handle a failed poll. Link and optical events stay as they are: we can't see them."""
|
||||
@@ -1443,7 +1655,8 @@ class NetworkMonitor:
|
||||
if not db.check_suppressed(suppressions, 'host', name) and not ticket_id:
|
||||
tid = self.tickets.create(
|
||||
f'[{name}][auto][production][issue][network][cluster-wide] {heading}',
|
||||
f'WAN Event\n{"=" * 40}\n\nModem: {name}\nReported: {_now_utc()}\n\n{description}\n\n{advice}',
|
||||
ticket_text.description('WAN Event', [('Modem', name), ('Reported', _now_utc())],
|
||||
description, [advice], recovers=False),
|
||||
priority=priority,
|
||||
)
|
||||
if tid:
|
||||
@@ -1645,6 +1858,12 @@ class NetworkMonitor:
|
||||
db.set_state('link_stats', link_data)
|
||||
except Exception as e:
|
||||
logger.error(f'Link stats collection failed: {e}', exc_info=True)
|
||||
if link_data is not None:
|
||||
self._link_data = link_data
|
||||
try:
|
||||
self._remember_uplinks(link_data)
|
||||
except Exception as e:
|
||||
logger.error(f'Recording known uplinks failed: {e}', exc_info=True)
|
||||
|
||||
# 5. Process alerts using already-fetched data
|
||||
suppressions = db.get_active_suppressions()
|
||||
@@ -1673,6 +1892,12 @@ class NetworkMonitor:
|
||||
except Exception as e:
|
||||
logger.error(f'Modem collection failed: {e}', exc_info=True)
|
||||
|
||||
# Note on the tickets of anything that cleared this cycle (or failed to post before)
|
||||
try:
|
||||
self._post_recovery_notes()
|
||||
except Exception as e:
|
||||
logger.error(f'Posting recovery notes failed: {e}', exc_info=True)
|
||||
|
||||
# Housekeeping: deactivate expired suppressions and purge old resolved events
|
||||
db.cleanup_expired_suppressions()
|
||||
db.purge_old_resolved_events(days=90)
|
||||
|
||||
+2
-1
@@ -21,8 +21,9 @@ CREATE TABLE IF NOT EXISTS network_events (
|
||||
resolved_at TIMESTAMP NULL,
|
||||
consecutive_failures INT NOT NULL DEFAULT 1,
|
||||
ticket_id VARCHAR(20) NULL,
|
||||
recovery_notified TINYINT(1) NOT NULL DEFAULT 0, -- recovery comment posted on ticket_id
|
||||
|
||||
INDEX idx_active (resolved_at),
|
||||
INDEX idx_active (resolved_at),
|
||||
INDEX idx_target (target_name, target_detail),
|
||||
INDEX idx_type (event_type),
|
||||
-- Compound index for hot upsert_event lookup: event_type+target+resolved_at
|
||||
|
||||
+45
-3
@@ -1,5 +1,5 @@
|
||||
'use strict';
|
||||
/* global DEVICE_NAME, deviceUrl, refreshModem, _toIso, fmtRate, fmtErrors, renderPortInspector, selectPortByIdx, highlightSelectedPort, FACEPLATES */
|
||||
/* global DEVICE_NAME, CAN_EDIT, deviceUrl, refreshModem, _toIso, fmtRate, fmtErrors, renderPortInspector, selectPortByIdx, highlightSelectedPort, FACEPLATES */
|
||||
|
||||
// Per-device page: overview, alerts, and a compact interface/port table. Host
|
||||
// rows expand in place; switch and router rows open the port inspector panel.
|
||||
@@ -300,7 +300,8 @@ function renderEvents(events) {
|
||||
<td class="ts-cell" title="${esc(e.first_seen || '')}">${esc(lt.time.ago(_toIso(e.first_seen)))}</td>
|
||||
<td>${e.resolved_at ? `resolved ${esc(lt.time.ago(_toIso(e.resolved_at)))}` : '<strong>active</strong>'}</td>
|
||||
</tr>`).join('')}</tbody></table></div>`;
|
||||
return section('Alerts', body);
|
||||
const more = `<p class="g-section-more"><a href="/events?target=${encodeURIComponent(DEVICE_NAME)}&range=90d">Full alert history for ${esc(DEVICE_NAME)} →</a></p>`;
|
||||
return section('Alerts', body + more);
|
||||
}
|
||||
|
||||
function currentRows() {
|
||||
@@ -331,6 +332,47 @@ function updateInterfaceSection(rows, title, hasChassis) {
|
||||
document.getElementById('device-error-count').textContent = rows.filter(rowHasProblem).length;
|
||||
}
|
||||
|
||||
// ── Inventory (HomeBox) link ────────────────────────────────────────────
|
||||
function inventoryActions(d) {
|
||||
const inv = d.inventory || {};
|
||||
const edit = CAN_EDIT
|
||||
? `<button type="button" class="lt-btn lt-btn-ghost lt-btn-sm" data-action="edit-inventory"
|
||||
aria-label="${inv.url ? 'Change' : 'Set'} the inventory link for ${esc(d.name)}">✎</button>` : '';
|
||||
if (inv.url) {
|
||||
return `<a class="lt-btn lt-btn-secondary lt-btn-sm" href="${esc(inv.url)}" target="_blank" rel="noopener">📦 Inventory ↗</a>${edit}`;
|
||||
}
|
||||
if (!inv.search_url) return edit;
|
||||
return `<a class="lt-btn lt-btn-ghost lt-btn-sm" href="${esc(inv.search_url)}" target="_blank" rel="noopener"
|
||||
title="No inventory record is linked yet">📦 Find in inventory ↗</a>${edit}`;
|
||||
}
|
||||
|
||||
function openInventoryEditor() {
|
||||
const inv = (_device && _device.inventory) || {};
|
||||
const form = document.getElementById('inventory-form');
|
||||
form.url.value = inv.url || '';
|
||||
document.getElementById('inventory-search').href = inv.search_url || '#';
|
||||
lt.modal.open('inventory-modal');
|
||||
form.url.focus();
|
||||
}
|
||||
|
||||
async function saveInventory(e) {
|
||||
e.preventDefault();
|
||||
try {
|
||||
await lt.api.put(`/api/device/${encodeURIComponent(DEVICE_NAME)}/inventory`, { url: e.target.url.value.trim() });
|
||||
lt.modal.close('inventory-modal');
|
||||
lt.toast.success(e.target.url.value.trim() ? 'Inventory link saved' : 'Inventory link removed');
|
||||
await loadDevice();
|
||||
} catch (err) {
|
||||
lt.toast.error(err.message || 'Save failed');
|
||||
}
|
||||
}
|
||||
|
||||
document.addEventListener('click', e => {
|
||||
if (e.target.closest('[data-action="edit-inventory"]')) openInventoryEditor();
|
||||
});
|
||||
const _inventoryForm = document.getElementById('inventory-form');
|
||||
if (_inventoryForm) _inventoryForm.addEventListener('submit', saveInventory);
|
||||
|
||||
function render(d) {
|
||||
_device = d;
|
||||
document.getElementById('device-dot').className = `dot-${d.status || 'unknown'}`;
|
||||
@@ -339,7 +381,7 @@ function render(d) {
|
||||
(d.updated ? ` · updated ${new Date(_toIso(d.updated)).toLocaleTimeString()}` : '');
|
||||
|
||||
const unifi = d.source === 'unifi';
|
||||
document.getElementById('device-actions').innerHTML = `
|
||||
document.getElementById('device-actions').innerHTML = inventoryActions(d) + `
|
||||
<button class="lt-btn lt-btn-ghost lt-btn-sm btn-suppress" data-sup-type="${unifi ? 'unifi_device' : 'host'}"
|
||||
data-sup-name="${esc(d.name)}" data-sup-detail="" aria-label="Suppress alerts for ${esc(d.name)}">🔕 Suppress</button>`;
|
||||
|
||||
|
||||
@@ -0,0 +1,218 @@
|
||||
'use strict';
|
||||
/* global deviceUrl, _toIso */
|
||||
// Alert history page: renders /api/events/history with filters kept in the URL.
|
||||
|
||||
const esc = s => lt.escHtml(s == null ? '' : String(s));
|
||||
const FILTERS = ['target', 'range', 'type', 'severity', 'status'];
|
||||
const DEFAULTS = { target: '', range: '7d', type: '', severity: '', status: 'all' };
|
||||
const RANGE_TEXT = { '24h': '24 hours', '7d': '7 days', '30d': '30 days', '90d': '90 days' };
|
||||
|
||||
let _offset = 0;
|
||||
let _loadSeq = 0;
|
||||
|
||||
function typeText(t) {
|
||||
return String(t || '').replace(/_/g, ' ');
|
||||
}
|
||||
|
||||
function duration(seconds) {
|
||||
let s = Math.max(0, Math.floor(seconds || 0));
|
||||
if (s < 60) return `${s} s`;
|
||||
let m = Math.floor(s / 60);
|
||||
if (m < 60) return `${m} min`;
|
||||
let h = Math.floor(m / 60);
|
||||
m %= 60;
|
||||
if (h < 48) return m ? `${h} h ${m} min` : `${h} h`;
|
||||
const d = Math.floor(h / 24);
|
||||
h %= 24;
|
||||
return h ? `${d} d ${h} h` : `${d} d`;
|
||||
}
|
||||
|
||||
function when(iso) {
|
||||
if (!iso) return '–';
|
||||
const d = new Date(_toIso(iso));
|
||||
if (isNaN(d)) return esc(iso);
|
||||
const text = d.toLocaleString([], { month: 'short', day: 'numeric', hour: '2-digit', minute: '2-digit' });
|
||||
return `<span title="${esc(d.toLocaleString())}">${esc(text)}</span>`;
|
||||
}
|
||||
|
||||
function ticketLink(id) {
|
||||
if (!id) return '<span class="val-neutral">–</span>';
|
||||
const base = (typeof GANDALF_CONFIG !== 'undefined' && GANDALF_CONFIG.ticket_web_url) || '';
|
||||
return base ? `<a href="${esc(base + id)}" target="_blank" rel="noopener">#${esc(id)}</a>` : `#${esc(id)}`;
|
||||
}
|
||||
|
||||
function filters() {
|
||||
const f = {};
|
||||
FILTERS.forEach(k => { f[k] = document.getElementById(`ev-${k}`).value.trim(); });
|
||||
return f;
|
||||
}
|
||||
|
||||
function readUrl() {
|
||||
const q = new URLSearchParams(location.search);
|
||||
FILTERS.forEach(k => {
|
||||
const el = document.getElementById(`ev-${k}`);
|
||||
const v = q.get(k);
|
||||
if (v == null) return;
|
||||
if (el.tagName === 'SELECT' && ![...el.options].some(o => o.value === v)) {
|
||||
// An alert type from a link may not be in the list until data arrives
|
||||
if (k === 'type') el.add(new Option(typeText(v), v));
|
||||
else return;
|
||||
}
|
||||
el.value = v;
|
||||
});
|
||||
}
|
||||
|
||||
function writeUrl(f) {
|
||||
const q = new URLSearchParams();
|
||||
FILTERS.forEach(k => { if (f[k] && f[k] !== DEFAULTS[k]) q.set(k, f[k]); });
|
||||
const qs = q.toString();
|
||||
history.replaceState(null, '', qs ? `?${qs}` : location.pathname);
|
||||
document.getElementById('ev-clear').hidden = !FILTERS.some(k => k !== 'range' && f[k] && f[k] !== DEFAULTS[k]);
|
||||
}
|
||||
|
||||
function setFilter(changes) {
|
||||
Object.entries(changes).forEach(([k, v]) => {
|
||||
const el = document.getElementById(`ev-${k}`);
|
||||
if (el.tagName === 'SELECT' && v && ![...el.options].some(o => o.value === v)) el.add(new Option(typeText(v), v));
|
||||
el.value = v;
|
||||
});
|
||||
_offset = 0;
|
||||
load();
|
||||
}
|
||||
|
||||
function fillTypes(types) {
|
||||
const sel = document.getElementById('ev-type');
|
||||
const current = sel.value;
|
||||
sel.innerHTML = '<option value="">All alert types</option>' +
|
||||
types.map(t => `<option value="${esc(t)}">${esc(typeText(t))}</option>`).join('');
|
||||
if (current && !types.includes(current)) sel.add(new Option(typeText(current), current));
|
||||
sel.value = current;
|
||||
}
|
||||
|
||||
function renderSummary(data, f) {
|
||||
const open = data.events.filter(e => !e.resolved_at).length;
|
||||
const chips = [
|
||||
`<span class="cs-chip"><b>${data.total}</b> alert${data.total === 1 ? '' : 's'} in the last ${esc(RANGE_TEXT[f.range] || f.range)}</span>`,
|
||||
];
|
||||
if (open && data.offset === 0) chips.push(`<span class="cs-chip cs-chip-bad"><b>${open}</b> open now</span>`);
|
||||
Object.entries(data.by_type || {}).forEach(([t, n]) => {
|
||||
const on = f.type === t;
|
||||
chips.push(`<button type="button" class="cs-chip ev-type-chip${on ? ' active' : ''}" data-type="${esc(on ? '' : t)}"
|
||||
aria-pressed="${on}"><b>${n}</b> ${esc(typeText(t))}</button>`);
|
||||
});
|
||||
document.getElementById('ev-summary').innerHTML = chips.join('');
|
||||
}
|
||||
|
||||
function renderFlappers(rows) {
|
||||
const sec = document.getElementById('ev-flappers-section');
|
||||
sec.hidden = !rows.length;
|
||||
if (!rows.length) return;
|
||||
document.getElementById('ev-flappers').innerHTML = `<div class="lt-table-wrap"><table class="lt-table ev-cards">
|
||||
<caption class="lt-sr-only">Alerts that fired more than once</caption>
|
||||
<thead><tr><th>Alert</th><th>Target</th><th>Times</th><th>Total time</th><th>Last started</th><th></th></tr></thead>
|
||||
<tbody>${rows.map(r => `
|
||||
<tr>
|
||||
<td class="ev-c-type">${esc(typeText(r.event_type))}${r.open_now ? ' <span class="lt-badge badge-critical">open</span>' : ''}</td>
|
||||
<td class="ev-c-target"><a class="device-link" href="${esc(deviceUrl(r.target_name))}">${esc(r.target_name)}</a>
|
||||
${r.target_detail ? `<span class="cs-port">${esc(r.target_detail)}</span>` : ''}</td>
|
||||
<td data-label="Times"><strong>${r.n}</strong></td>
|
||||
<td data-label="Total">${esc(duration(r.total_s))}</td>
|
||||
<td class="ts-cell" data-label="Last">${when(r.last_seen)}</td>
|
||||
<td class="ev-c-action"><button type="button" class="lt-btn lt-btn-secondary lt-btn-sm" data-show-type="${esc(r.event_type)}"
|
||||
data-show-target="${esc(r.target_name)}">Show these</button></td>
|
||||
</tr>`).join('')}</tbody></table></div>`;
|
||||
}
|
||||
|
||||
function renderTable(data) {
|
||||
document.getElementById('ev-count').textContent = data.total;
|
||||
const wrap = document.getElementById('ev-table');
|
||||
if (!data.events.length) {
|
||||
wrap.innerHTML = '<div class="lt-empty-state lt-empty-state--sm"><div class="lt-empty-state-icon">✔</div>' +
|
||||
'<div class="lt-empty-state-title">No alerts match</div></div>';
|
||||
document.getElementById('ev-pager').innerHTML = '';
|
||||
return;
|
||||
}
|
||||
wrap.innerHTML = `<div class="lt-table-wrap"><table class="lt-table ev-table ev-cards">
|
||||
<caption class="lt-sr-only">Alert history</caption>
|
||||
<thead><tr><th>State</th><th>Type</th><th>Target</th><th>Started</th><th>Lasted</th>
|
||||
<th title="Consecutive failed checks">Checks</th><th>Ticket</th><th>Description</th></tr></thead>
|
||||
<tbody>${data.events.map(e => {
|
||||
const open = !e.resolved_at;
|
||||
return `<tr class="${open ? `row-${esc(e.severity)}` : ''}">
|
||||
<td class="ev-c-state"><span class="lt-badge badge-${open ? esc(e.severity) : 'neutral'}">${open ? 'open' : 'cleared'}</span></td>
|
||||
<td class="ev-c-type"><button type="button" class="ev-link" data-show-type="${esc(e.event_type)}"
|
||||
title="Only show ${esc(typeText(e.event_type))}">${esc(typeText(e.event_type))}</button></td>
|
||||
<td class="ev-c-target"><a class="device-link" href="${esc(deviceUrl(e.target_name))}">${esc(e.target_name)}</a>
|
||||
${e.target_detail ? `<span class="cs-port">${esc(e.target_detail)}</span>` : ''}</td>
|
||||
<td class="ts-cell" data-label="Started">${when(e.first_seen)}</td>
|
||||
<td data-label="Lasted">${esc(duration(e.duration_s))}${open ? ' <span class="val-neutral">so far</span>' : ''}</td>
|
||||
<td data-label="Checks">${esc(e.consecutive_failures)}</td>
|
||||
<td data-label="Ticket">${ticketLink(e.ticket_id)}</td>
|
||||
<td class="desc-cell ev-c-desc" title="${esc(e.description || '')}">${esc(e.description || '')}</td>
|
||||
</tr>`;
|
||||
}).join('')}</tbody></table></div>`;
|
||||
|
||||
const end = data.offset + data.events.length;
|
||||
const prev = data.offset > 0;
|
||||
const next = end < data.total;
|
||||
document.getElementById('ev-pager').innerHTML = (prev || next) ? `
|
||||
<button type="button" class="lt-btn lt-btn-secondary lt-btn-sm" data-page="${data.offset - data.page_size}" ${prev ? '' : 'disabled'}>← Newer</button>
|
||||
<span>${data.offset + 1}–${end} of ${data.total}</span>
|
||||
<button type="button" class="lt-btn lt-btn-secondary lt-btn-sm" data-page="${end}" ${next ? '' : 'disabled'}>Older →</button>` : '';
|
||||
}
|
||||
|
||||
async function load() {
|
||||
const f = filters();
|
||||
writeUrl(f);
|
||||
const q = new URLSearchParams();
|
||||
FILTERS.forEach(k => { if (f[k]) q.set(k, f[k]); });
|
||||
if (_offset) q.set('offset', _offset);
|
||||
const seq = ++_loadSeq;
|
||||
try {
|
||||
const data = await lt.api.get(`/api/events/history?${q}`);
|
||||
if (seq !== _loadSeq) return; // a newer filter change is already loading
|
||||
fillTypes(data.types || []);
|
||||
renderSummary(data, f);
|
||||
renderFlappers(data.flappers || []);
|
||||
renderTable(data);
|
||||
} catch (err) {
|
||||
if (seq !== _loadSeq) return;
|
||||
document.getElementById('ev-table').innerHTML =
|
||||
`<div class="lt-empty-state lt-empty-state--sm"><div class="lt-empty-state-title">Could not load alerts</div>
|
||||
<div class="lt-empty-state-text">${esc(err.message || err)}</div></div>`;
|
||||
}
|
||||
}
|
||||
|
||||
let _typing = null;
|
||||
document.getElementById('ev-target').addEventListener('input', () => {
|
||||
clearTimeout(_typing);
|
||||
_typing = setTimeout(() => { _offset = 0; load(); }, 300);
|
||||
});
|
||||
document.getElementById('ev-filters').addEventListener('change', e => {
|
||||
if (e.target.id === 'ev-target') return;
|
||||
_offset = 0;
|
||||
load();
|
||||
});
|
||||
document.getElementById('ev-filters').addEventListener('submit', e => { e.preventDefault(); _offset = 0; load(); });
|
||||
document.getElementById('ev-clear').addEventListener('click', () => {
|
||||
setFilter({ target: '', type: '', severity: '', status: 'all' });
|
||||
});
|
||||
document.addEventListener('click', e => {
|
||||
const typeChip = e.target.closest('.ev-type-chip');
|
||||
if (typeChip) return setFilter({ type: typeChip.dataset.type });
|
||||
const show = e.target.closest('[data-show-type]');
|
||||
if (show) {
|
||||
const changes = { type: show.dataset.showType };
|
||||
if (show.dataset.showTarget) changes.target = show.dataset.showTarget;
|
||||
return setFilter(changes);
|
||||
}
|
||||
const page = e.target.closest('[data-page]');
|
||||
if (page && !page.disabled) {
|
||||
_offset = Math.max(0, parseInt(page.dataset.page, 10) || 0);
|
||||
load();
|
||||
document.getElementById('ev-count').scrollIntoView({ block: 'nearest' });
|
||||
}
|
||||
});
|
||||
|
||||
readUrl();
|
||||
load();
|
||||
+29
-7
@@ -107,7 +107,13 @@ function portLabelSvg(p, port) {
|
||||
return `<text class="fp-label" x="${x}" y="${y}" text-anchor="middle"${size}>${p.text || p.idx}</text>${bolt}`;
|
||||
}
|
||||
|
||||
function faceplatePortSvg(p, port, swName, title) {
|
||||
// Cutsheet status dot in a port's corner: ok (matches), bad (mismatch), doc (documented, not confirmed)
|
||||
function cutsheetMarkSvg(mark, x, y) {
|
||||
if (!mark) return '';
|
||||
return `<circle class="fp-mark fp-mark-${mark}" cx="${x}" cy="${y}" r="3.2"/>`;
|
||||
}
|
||||
|
||||
function faceplatePortSvg(p, port, swName, title, mark) {
|
||||
// The drawn port number can differ from the index UniFi reports (see anyPort)
|
||||
const idx = (p.anyPort && port) ? port.port_idx : p.idx;
|
||||
const attrs = `class="fp-port${port && port.up ? ' fp-up' : ''}" data-switch="${lt.escHtml(swName)}" data-port-idx="${idx}"
|
||||
@@ -117,6 +123,7 @@ function faceplatePortSvg(p, port, swName, title) {
|
||||
<title>${title}</title>
|
||||
${sfpSvg(port, p.w, p.h)}
|
||||
<rect class="fp-hit" x="-1" y="-1" width="${p.w + 2}" height="${p.h + 2}" rx="2"/>
|
||||
${cutsheetMarkSvg(mark, p.w - 3, 3)}
|
||||
</g>
|
||||
${sfpLedSvg(p, port)}${portLabelSvg(p, port)}`;
|
||||
}
|
||||
@@ -125,6 +132,7 @@ function faceplatePortSvg(p, port, swName, title) {
|
||||
<title>${title}</title>
|
||||
<g transform="scale(1 ${p.flip ? -1 : 1})">${rj45Svg(port, p)}</g>
|
||||
<rect class="fp-hit" x="-18" y="-${p.flip ? 11 : 17}" width="36" height="28" rx="2"/>
|
||||
${cutsheetMarkSvg(mark, 14, p.flip ? 13 : -13)}
|
||||
</g>${portLabelSvg(p, port)}`;
|
||||
}
|
||||
|
||||
@@ -469,9 +477,17 @@ function faceplateLegendSwatch(cls) {
|
||||
return `<span class="fp-legend-led fp-led-${cls}"></span>`;
|
||||
}
|
||||
|
||||
function faceplateLegendHtml(spec) {
|
||||
const items = spec.legend.map(([cls, text]) =>
|
||||
const CUTSHEET_MARK_LEGEND = [['ok', 'Matches cutsheet'], ['bad', 'Cutsheet mismatch'], ['doc', 'Documented, unconfirmed']];
|
||||
|
||||
function faceplateLegendHtml(spec, marks) {
|
||||
let items = spec.legend.map(([cls, text]) =>
|
||||
`<div class="chassis-legend-item">${faceplateLegendSwatch(cls)}${text}</div>`).join('');
|
||||
if (marks && marks.size) {
|
||||
items += CUTSHEET_MARK_LEGEND.filter(([m]) => marks.has(m)).map(([m, text]) =>
|
||||
`<div class="chassis-legend-item fp-mark-key"><svg class="fp-legend-icon" viewBox="0 0 8 8"><circle class="fp-mark fp-mark-${m}" cx="4" cy="4" r="3.2"/></svg>${text}</div>`).join('');
|
||||
items += `<label class="chassis-legend-item fp-mark-toggle"><input type="checkbox" data-action="toggle-cutsheet-marks"
|
||||
${document.body.classList.contains('fp-marks-off') ? '' : 'checked'}> Cutsheet marks</label>`;
|
||||
}
|
||||
const hasDetail = spec.ports.length || spec.outlets;
|
||||
return `<div class="chassis-legend">
|
||||
${items}
|
||||
@@ -479,14 +495,20 @@ function faceplateLegendHtml(spec) {
|
||||
</div>`;
|
||||
}
|
||||
|
||||
// ctx: { swName, portMap, outlets, online, titleFor(port, idx, role) }
|
||||
// ctx: { swName, portMap, outlets, online, titleFor(port, idx, role), cutsheetFor?(idx) -> {mark, text} }
|
||||
function renderFaceplate(model, ctx) {
|
||||
const spec = FACEPLATES[model];
|
||||
const anyPort = Object.values(ctx.portMap)[0];
|
||||
const marks = new Set();
|
||||
const ports = spec.ports.map(p => {
|
||||
const port = ctx.portMap[p.idx] || (p.anyPort ? anyPort : undefined);
|
||||
const title = ctx.titleFor(port, p.idx, spec.roles && spec.roles[p.idx]);
|
||||
return faceplatePortSvg(p, port, ctx.swName, title);
|
||||
let title = ctx.titleFor(port, p.idx, spec.roles && spec.roles[p.idx]);
|
||||
const cs = ctx.cutsheetFor ? ctx.cutsheetFor((p.anyPort && port) ? port.port_idx : p.idx) : null;
|
||||
if (cs) {
|
||||
marks.add(cs.mark);
|
||||
title += lt.escHtml(` · ${cs.text}`);
|
||||
}
|
||||
return faceplatePortSvg(p, port, ctx.swName, title, cs && cs.mark);
|
||||
}).join('');
|
||||
const outletData = Object.fromEntries((ctx.outlets || []).map(o => [o.index, o]));
|
||||
const outlets = (spec.outlets || []).map(o => faceplateOutletSvg(o, outletData[o.idx])).join('');
|
||||
@@ -510,7 +532,7 @@ function renderFaceplate(model, ctx) {
|
||||
${ports}
|
||||
</svg>
|
||||
</div>
|
||||
</div>${faceplateLegendHtml(spec)}`;
|
||||
</div>${faceplateLegendHtml(spec, marks)}`;
|
||||
}
|
||||
|
||||
// Where an fp-scroll panel should start: its port area
|
||||
|
||||
@@ -112,6 +112,32 @@ function cutsheetPanelHtml(swName, idx) {
|
||||
}).join('');
|
||||
}
|
||||
|
||||
// The faceplate dot for one port: the worst state of the cables documented on it
|
||||
const MARK_RANK = { bad: 3, doc: 2, ok: 1 };
|
||||
|
||||
function cutsheetMarkFor(idx) {
|
||||
const entries = ((_apiData && _apiData.cutsheet) || []).filter(e => e.port === String(idx) && e.status !== 'retired');
|
||||
if (!entries.length) return null;
|
||||
let best = null;
|
||||
entries.forEach(e => {
|
||||
const mark = CUTSHEET_PROBLEM.has(e.verdict) ? 'bad' : (e.verdict === 'match' ? 'ok' : 'doc');
|
||||
if (!best || MARK_RANK[mark] > MARK_RANK[best.mark]) {
|
||||
best = { mark, text: `cable ${e.label || 'unlabeled'}: ${e.detail}` };
|
||||
}
|
||||
});
|
||||
return best;
|
||||
}
|
||||
|
||||
document.addEventListener('change', e => {
|
||||
if (!e.target.matches('[data-action="toggle-cutsheet-marks"]')) return;
|
||||
const off = !e.target.checked;
|
||||
document.body.classList.toggle('fp-marks-off', off);
|
||||
try { localStorage.setItem('gandalf.cutsheetMarks', off ? 'off' : 'on'); } catch (_) { /* private mode */ }
|
||||
});
|
||||
try {
|
||||
if (localStorage.getItem('gandalf.cutsheetMarks') === 'off') document.body.classList.add('fp-marks-off');
|
||||
} catch (_) { /* private mode */ }
|
||||
|
||||
// ── SFP module section of the port panel ────────────────────────────────
|
||||
function sfpPanelHtml(sfp) {
|
||||
if (!sfp) return '';
|
||||
@@ -206,6 +232,7 @@ function renderChassis(swName, sw) {
|
||||
if (FACEPLATES[model]) {
|
||||
return `<div class="inspector-chassis">${header}${renderFaceplate(model, {
|
||||
swName, portMap, outlets: sw.outlets, online: sw.online, titleFor: faceplatePortTitle,
|
||||
cutsheetFor: cutsheetMarkFor,
|
||||
})}</div>`;
|
||||
}
|
||||
|
||||
|
||||
+244
@@ -0,0 +1,244 @@
|
||||
'use strict';
|
||||
/* global FACEPLATES, renderFaceplate */
|
||||
// Rack page: draws the rack front from /api/rack. Switches use their real faceplates,
|
||||
// patch panels a generic panel; hovering a port lights up every port on the same cable run.
|
||||
|
||||
const esc = s => lt.escHtml(s == null ? '' : String(s));
|
||||
const RATIO = 10.5; // a 19" rack's width to one unit's height
|
||||
const KIND_TEXT = { patch_panel: 'Patch panel', wall_jack: 'Wall jack', device: 'Device', shelf: 'Shelf', other: 'Other' };
|
||||
|
||||
let _rack = null;
|
||||
|
||||
function speedText(mbps) {
|
||||
if (!mbps) return '';
|
||||
return mbps >= 1000 ? `${mbps / 1000}G` : `${mbps}M`;
|
||||
}
|
||||
|
||||
function portMapOf(ports) {
|
||||
const map = {};
|
||||
Object.entries(ports || {}).forEach(([name, p]) => {
|
||||
if (p.port_idx != null) map[p.port_idx] = Object.assign({ name }, p);
|
||||
});
|
||||
return map;
|
||||
}
|
||||
|
||||
function portTitle(port, idx) {
|
||||
const parts = [port && port.name && port.name !== `Port ${idx}` ? `Port ${idx} "${port.name}"` : `Port ${idx}`];
|
||||
if (port) {
|
||||
parts.push(port.up ? (speedText(port.speed_mbps) || 'up') : 'down');
|
||||
const nb = port.lldp && (port.lldp.system_name || port.lldp.chassis_id);
|
||||
if (nb) parts.push(`→ ${nb}`);
|
||||
}
|
||||
return esc(parts.join(' · '));
|
||||
}
|
||||
|
||||
function isRackFaceplate(model) {
|
||||
const spec = FACEPLATES[model];
|
||||
return !!spec && !/fp-desk/.test(spec.frameClass || '');
|
||||
}
|
||||
|
||||
// ── Item drawings ───────────────────────────────────────────────────────
|
||||
function switchHtml(item) {
|
||||
const sw = item.switch;
|
||||
return renderFaceplate(sw.model, {
|
||||
swName: item.name, portMap: portMapOf(sw.ports), outlets: sw.outlets, online: sw.online,
|
||||
titleFor: (port, idx) => portTitle(port, idx),
|
||||
cutsheetFor: idx => {
|
||||
const m = item.ports[String(idx)];
|
||||
return m ? { mark: m.mark, text: m.text } : null;
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
function panelSvg(item) {
|
||||
const count = Math.max(1, item.port_count || 24);
|
||||
const units = item.rack_height || 1;
|
||||
const rows = Math.max(1, Math.min(units * 2, Math.ceil(count / 24)));
|
||||
const perRow = Math.ceil(count / rows);
|
||||
const W = 1100, H = 105 * units, ear = 70;
|
||||
const pitch = Math.min(38, (W - ear * 2 - 40) / perRow);
|
||||
const left = (W - pitch * perRow) / 2;
|
||||
const rowH = (H - 20) / rows;
|
||||
// Label characters that fit under one port at the label's font size (9px, ~5.6px a character)
|
||||
const fit = Math.max(3, Math.floor((pitch - 2) / 5.6));
|
||||
const short = t => (t.length > fit ? `${t.slice(0, fit - 1)}…` : t);
|
||||
const ports = [];
|
||||
for (let n = 1; n <= count; n++) {
|
||||
const r = Math.floor((n - 1) / perRow), c = (n - 1) % perRow;
|
||||
const x = left + c * pitch + pitch / 2, y = 10 + r * rowH + rowH / 2;
|
||||
const m = item.ports[String(n)];
|
||||
const label = item.labels[String(n)] || '';
|
||||
const title = [`Port ${n}`, label && `"${label}"`, m ? m.text : 'not on the cutsheet'].filter(Boolean).join(' · ');
|
||||
const attrs = m ? ` data-run="${m.run.join(' ')}" data-cable="${m.cable_id}" tabindex="0" role="link"` : '';
|
||||
ports.push(`<g class="rack-pp-port${m ? ` rack-pp-${m.mark}` : ''}"${attrs} transform="translate(${x} ${y})">
|
||||
<title>${esc(title)}</title>
|
||||
<text class="rack-pp-num" y="-17" text-anchor="middle">${n}</text>
|
||||
<rect class="rack-pp-jack" x="-12" y="-10" width="24" height="20" rx="2"/>
|
||||
<rect class="rack-pp-hit" x="${-pitch / 2}" y="-28" width="${pitch}" height="56"/>
|
||||
${label ? `<text class="rack-pp-label" y="24" text-anchor="middle">${esc(short(label))}</text>` : ''}
|
||||
</g>`);
|
||||
}
|
||||
return `<svg class="rack-svg" viewBox="0 0 ${W} ${H}" preserveAspectRatio="xMidYMid meet" role="group"
|
||||
aria-label="${esc(item.name)} patch panel">
|
||||
<rect class="rack-pp-body" width="${W}" height="${H}" rx="3"/>
|
||||
<text class="rack-pp-name" x="${ear - 50}" y="${H / 2 + 5}">${esc(item.name)}</text>
|
||||
${ports.join('')}
|
||||
</svg>`;
|
||||
}
|
||||
|
||||
function plateHtml(item) {
|
||||
const sub = [KIND_TEXT[item.kind] || item.kind, item.location].filter(Boolean).join(' · ');
|
||||
return `<div class="rack-plate">
|
||||
<a class="device-link rack-plate-name" href="${esc(item.url)}">${esc(item.name)}</a>
|
||||
<span class="rack-plate-sub">${esc(sub)}</span>
|
||||
</div>`;
|
||||
}
|
||||
|
||||
function itemHtml(item) {
|
||||
let body;
|
||||
if (item.switch && isRackFaceplate(item.switch.model)) body = switchHtml(item);
|
||||
else if (item.kind === 'patch_panel') body = panelSvg(item);
|
||||
else body = plateHtml(item);
|
||||
return `<div class="rack-item rack-kind-${esc(item.kind)}" data-item="${esc(item.id)}">${body}</div>`;
|
||||
}
|
||||
|
||||
function uLabels(top, bottom) {
|
||||
const spans = [];
|
||||
for (let u = top; u >= bottom; u--) spans.push(`<span>${u}</span>`);
|
||||
return `<div class="rack-u" aria-hidden="true">${spans.join('')}</div>`;
|
||||
}
|
||||
|
||||
function row(top, bottom, inner, cls) {
|
||||
const n = top - bottom + 1;
|
||||
return `<div class="rack-row ${cls}" style="--n:${n}">
|
||||
${uLabels(top, bottom)}
|
||||
<div class="rack-slot" style="aspect-ratio:${RATIO} / ${n}">${inner}</div>
|
||||
</div>`;
|
||||
}
|
||||
|
||||
// ── Page ────────────────────────────────────────────────────────────────
|
||||
function render(data) {
|
||||
_rack = data;
|
||||
const byTop = Object.fromEntries(data.items.map(i => [i.rack_top, i]));
|
||||
const rows = [];
|
||||
let u = data.units;
|
||||
while (u >= 1) {
|
||||
const item = byTop[u];
|
||||
if (item) {
|
||||
rows.push(row(u, item.rack_u, itemHtml(item), 'rack-row-item'));
|
||||
u = item.rack_u - 1;
|
||||
continue;
|
||||
}
|
||||
let low = u;
|
||||
while (low - 1 >= 1 && !byTop[low - 1]) low--;
|
||||
rows.push(row(u, low, '', 'rack-row-empty'));
|
||||
u = low - 1;
|
||||
}
|
||||
document.getElementById('rack').innerHTML = rows.join('');
|
||||
document.getElementById('rack-count').textContent = data.items.length;
|
||||
|
||||
// Faceplate ports carry their cable run too, for hover and click
|
||||
document.querySelectorAll('.rack-item .fp-port').forEach(g => {
|
||||
const item = data.items.find(i => i.name === g.dataset.switch);
|
||||
const m = item && item.ports[g.dataset.portIdx];
|
||||
if (m) {
|
||||
g.dataset.run = m.run.join(' ');
|
||||
g.dataset.cable = m.cable_id;
|
||||
}
|
||||
});
|
||||
|
||||
const notes = [];
|
||||
data.overlaps.forEach(o => notes.push(`<strong>${esc(o.name)}</strong> (${esc(o.units)}) overlaps ${esc(o.with.join(', '))} and is not drawn.`));
|
||||
data.outside.forEach(o => notes.push(`<strong>${esc(o.name)}</strong> (${esc(o.units)}) is above the top of a ${data.units}U rack and is not drawn.`));
|
||||
const warn = document.getElementById('rack-warnings');
|
||||
warn.hidden = !notes.length;
|
||||
warn.innerHTML = notes.length ? `<ul>${notes.map(n => `<li>${n}</li>`).join('')}</ul>
|
||||
<p>Fix the rack unit on the <a href="/cutsheet#cs-items-section">cutsheet</a>.</p>` : '';
|
||||
|
||||
const unplaced = document.getElementById('rack-unplaced');
|
||||
unplaced.innerHTML = data.unplaced.length
|
||||
? `Not in the drawing (no rack unit set): ${data.unplaced.map(n => `<span class="cs-chip">${esc(n)}</span>`).join(' ')}`
|
||||
: '';
|
||||
|
||||
if (!data.items.length) {
|
||||
document.getElementById('rack').insertAdjacentHTML('afterbegin',
|
||||
`<div class="lt-empty-state lt-empty-state--sm rack-empty"><div class="lt-empty-state-title">Nothing placed yet</div>
|
||||
<div class="lt-empty-state-text">Give panels, shelves and devices a rack unit on the <a href="/cutsheet">cutsheet</a> and they appear here.</div></div>`);
|
||||
}
|
||||
}
|
||||
|
||||
function readout(el) {
|
||||
const out = document.getElementById('rack-readout');
|
||||
if (!el) {
|
||||
out.textContent = 'Hover a port to follow its cable.';
|
||||
return;
|
||||
}
|
||||
const title = el.querySelector('title');
|
||||
out.textContent = title ? title.textContent : '';
|
||||
}
|
||||
|
||||
function highlight(el) {
|
||||
const ids = el && el.dataset.run ? el.dataset.run.split(' ') : [];
|
||||
document.querySelectorAll('#rack [data-run]').forEach(x => {
|
||||
x.classList.toggle('rack-hl', ids.length > 0 && x.dataset.run.split(' ').some(i => ids.includes(i)));
|
||||
});
|
||||
}
|
||||
|
||||
function portAt(target) {
|
||||
return target.closest && target.closest('#rack .fp-port, #rack .rack-pp-port');
|
||||
}
|
||||
|
||||
function onEnter(e) {
|
||||
const el = portAt(e.target);
|
||||
if (!el) return;
|
||||
highlight(el);
|
||||
readout(el);
|
||||
}
|
||||
|
||||
function onLeave(e) {
|
||||
const el = portAt(e.target);
|
||||
if (el && !el.contains(e.relatedTarget)) {
|
||||
highlight(null);
|
||||
readout(null);
|
||||
}
|
||||
}
|
||||
|
||||
function openPort(el) {
|
||||
if (el.classList.contains('fp-port')) {
|
||||
location.href = `/device/${encodeURIComponent(el.dataset.switch)}#port-${encodeURIComponent(el.dataset.portIdx)}`;
|
||||
} else if (el.dataset.cable) {
|
||||
location.href = `/cutsheet#cable-${encodeURIComponent(el.dataset.cable)}`;
|
||||
}
|
||||
}
|
||||
|
||||
const rackEl = document.getElementById('rack');
|
||||
rackEl.addEventListener('mouseover', onEnter);
|
||||
rackEl.addEventListener('mouseout', onLeave);
|
||||
rackEl.addEventListener('focusin', onEnter);
|
||||
rackEl.addEventListener('focusout', onLeave);
|
||||
rackEl.addEventListener('click', e => {
|
||||
const el = portAt(e.target);
|
||||
if (el) openPort(el);
|
||||
});
|
||||
rackEl.addEventListener('keydown', e => {
|
||||
const el = portAt(e.target);
|
||||
if (el && (e.key === 'Enter' || e.key === ' ')) {
|
||||
e.preventDefault();
|
||||
openPort(el);
|
||||
}
|
||||
});
|
||||
|
||||
async function load() {
|
||||
try {
|
||||
render(await lt.api.get('/api/rack'));
|
||||
} catch (err) {
|
||||
if (_rack) return; // keep the last drawing on a failed refresh
|
||||
document.getElementById('rack').innerHTML = `<div class="lt-empty-state lt-empty-state--sm">
|
||||
<div class="lt-empty-state-title">Could not load the rack</div>
|
||||
<div class="lt-empty-state-text">${esc(err.message || err)}</div></div>`;
|
||||
}
|
||||
}
|
||||
|
||||
readout(null);
|
||||
load();
|
||||
setInterval(() => { if (!document.hidden && !document.querySelector('#rack .rack-hl')) load(); }, 60000);
|
||||
@@ -1393,3 +1393,96 @@ polygon.fp-led-off { fill: #6b6c71; }
|
||||
.cs-label-card { break-inside: avoid; border-color: #000; color: #000; }
|
||||
.lt-table, .lt-table td, .lt-table th { color: #000 !important; }
|
||||
}
|
||||
|
||||
/* ── Alert history (/events) ───────────────────────────────────────── */
|
||||
.ev-filters .lt-select { flex: 0 1 170px; }
|
||||
.ev-type-chip { background: none; cursor: pointer; font: inherit; font-size: .82rem; }
|
||||
.ev-type-chip:hover, .ev-type-chip:focus-visible { border-color: var(--accent-cyan); }
|
||||
.ev-type-chip.active { border-color: var(--accent-cyan); background: var(--accent-cyan-dim); }
|
||||
.ev-link { background: none; border: 0; padding: 0; color: inherit; font: inherit; cursor: pointer;
|
||||
text-decoration: underline dotted; text-underline-offset: 3px; text-align: left; }
|
||||
.ev-link:hover, .ev-link:focus-visible { color: var(--accent-cyan); }
|
||||
.ev-table td { vertical-align: top; }
|
||||
.ev-table .cs-port { display: block; }
|
||||
.ev-pager { display: flex; gap: 12px; align-items: center; justify-content: center; margin-top: 12px; font-size: .85rem; }
|
||||
.g-section-more { margin: 10px 12px 4px; font-size: .85rem; }
|
||||
.ev-table .lt-badge, .ev-c-type { white-space: nowrap; }
|
||||
|
||||
/* Phones: each alert becomes a card instead of a squeezed table row */
|
||||
@media (max-width: 760px) {
|
||||
.ev-cards thead { display: none; }
|
||||
.ev-cards, .ev-cards tbody { display: block; width: 100%; }
|
||||
.ev-cards tr { display: flex; flex-wrap: wrap; align-items: baseline; gap: 4px 14px; padding: 10px 12px;
|
||||
border-bottom: 1px solid var(--border-color); }
|
||||
.ev-cards td { display: block; padding: 0; border: 0; max-width: none; white-space: normal; font-size: .82rem; }
|
||||
.ev-cards td[data-label]::before { content: attr(data-label) " "; color: var(--text-muted); font-size: .72rem;
|
||||
text-transform: uppercase; letter-spacing: .05em; }
|
||||
.ev-cards .ev-c-target { flex: 1 1 100%; order: -1; font-size: .95rem; }
|
||||
.ev-cards .ev-c-target .cs-port { display: inline; margin-left: 6px; }
|
||||
.ev-cards .ev-c-desc { flex: 1 1 100%; color: var(--text-muted); white-space: normal; overflow: visible; }
|
||||
.ev-cards .ev-c-action { flex: 1 1 100%; }
|
||||
}
|
||||
|
||||
/* ── Cutsheet marks on faceplates ──────────────────────────────────── */
|
||||
.fp-mark { stroke: #101418; stroke-width: 1.2; pointer-events: none; }
|
||||
.fp-mark-ok { fill: var(--accent-green, #00ff88); }
|
||||
.fp-mark-bad { fill: var(--status-critical, #e0312b); }
|
||||
.fp-mark-doc { fill: #9aa7b4; }
|
||||
body.fp-marks-off .fp-svg .fp-mark { display: none; }
|
||||
.fp-mark-toggle { cursor: pointer; gap: 6px; }
|
||||
.fp-mark-toggle input { margin: 0; }
|
||||
|
||||
/* ── Rack (/rack) ──────────────────────────────────────────────────── */
|
||||
.rack { max-width: 1000px; display: flex; flex-direction: column; gap: 2px; padding: 10px 12px 10px 0;
|
||||
background: #0b0f14; border: 1px solid var(--border-color); border-radius: 4px; }
|
||||
.rack-row { display: grid; grid-template-columns: 34px 1fr; }
|
||||
.rack-u { display: flex; flex-direction: column; font-size: .62rem; color: var(--text-muted); text-align: right;
|
||||
padding-right: 6px; font-family: var(--font-mono, monospace); }
|
||||
.rack-u span { flex: 1; display: flex; align-items: center; justify-content: flex-end; }
|
||||
.rack-slot { position: relative; min-width: 0; }
|
||||
.rack-row-empty .rack-slot { background: repeating-linear-gradient(to bottom, transparent 0 calc(100% / var(--n) - 1px),
|
||||
rgba(255,255,255,.05) calc(100% / var(--n) - 1px) calc(100% / var(--n))); border: 1px dashed rgba(255,255,255,.06); }
|
||||
.rack-item, .rack-item .fp-wrap, .rack-item .fp-frame { height: 100%; }
|
||||
.rack-item .fp-wrap { padding: 0; }
|
||||
.rack-item .fp-frame { box-shadow: none; overflow: hidden; }
|
||||
.rack-item .fp-svg { width: 100%; height: 100%; }
|
||||
.rack-item .chassis-legend { display: none; }
|
||||
.rack-svg { display: block; width: 100%; height: 100%; }
|
||||
.rack-plate { height: 100%; display: flex; align-items: center; gap: 14px; padding: 0 18px; border-radius: 3px;
|
||||
background: linear-gradient(#232a33, #1a2028); border: 1px solid #2d3743; }
|
||||
.rack-plate-name { font-weight: 700; }
|
||||
.rack-plate-sub { font-size: .78rem; color: var(--text-muted); }
|
||||
.rack-kind-shelf .rack-plate { background: repeating-linear-gradient(90deg, #1b2129 0 10px, #161b22 10px 20px); }
|
||||
|
||||
.rack-pp-body { fill: #1d2229; stroke: #39424e; }
|
||||
.rack-pp-name { fill: #9aa7b4; font-size: 15px; font-family: var(--font-mono, monospace); }
|
||||
.rack-pp-num { fill: #7d8a97; font-size: 10px; }
|
||||
.rack-pp-label { fill: #c9d3dc; font-size: 9px; }
|
||||
.rack-pp-jack { fill: #0c0f13; stroke: #4a5560; stroke-width: 1.5; }
|
||||
.rack-pp-ok .rack-pp-jack { stroke: var(--accent-green); }
|
||||
.rack-pp-bad .rack-pp-jack { stroke: var(--status-critical, #e0312b); stroke-width: 2.5; }
|
||||
.rack-pp-doc .rack-pp-jack { stroke: #9aa7b4; }
|
||||
.rack-pp-hit { fill: transparent; }
|
||||
.rack-pp-port[data-cable] { cursor: pointer; outline: none; }
|
||||
|
||||
/* A hovered run: every port on it */
|
||||
.rack-pp-port.rack-hl .rack-pp-jack { fill: rgba(0,212,255,.35); stroke: var(--accent-cyan); stroke-width: 3; }
|
||||
.rack .fp-port.rack-hl .fp-hit { stroke: var(--accent-cyan); stroke-width: 4; fill: rgba(0,212,255,.3); }
|
||||
|
||||
.rack-readout { min-height: 1.4em; margin: 0 0 8px; font-size: .85rem; color: var(--accent-cyan); }
|
||||
.rack-legend { display: flex; flex-wrap: wrap; gap: 14px; font-size: .75rem; color: var(--text-muted); margin-bottom: 8px; }
|
||||
.rack-key { display: inline-block; width: 10px; height: 10px; border-radius: 50%; margin-right: 6px; vertical-align: -1px; }
|
||||
.rack-key-ok { background: var(--accent-green); }
|
||||
.rack-key-bad { background: var(--status-critical, #e0312b); }
|
||||
.rack-key-doc { background: #9aa7b4; }
|
||||
.rack-warnings { border: 1px solid var(--status-critical, #e0312b); padding: 8px 14px; margin-bottom: 14px; font-size: .85rem; }
|
||||
.rack-warnings ul { margin: 0 0 6px; padding-left: 18px; }
|
||||
.rack-warnings p { margin: 0; }
|
||||
.rack-unplaced { margin-top: 12px; display: flex; flex-wrap: wrap; gap: 6px; align-items: center; }
|
||||
.rack-empty { margin: 12px; }
|
||||
@media (max-width: 600px) {
|
||||
.rack-row { grid-template-columns: 22px 1fr; }
|
||||
.rack-u { font-size: .5rem; padding-right: 3px; }
|
||||
.rack-plate { padding: 0 8px; gap: 8px; }
|
||||
.rack-plate-sub { display: none; }
|
||||
}
|
||||
+19
-1
@@ -6,7 +6,7 @@
|
||||
<meta name="theme-color" content="#030508">
|
||||
<meta name="robots" content="noindex, nofollow">
|
||||
<link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 16 16'%3E%3Crect width='16' height='16' rx='3' fill='%23030508'/%3E%3Cpath d='M8 2l5 3v6l-5 3-5-3V5z' fill='none' stroke='%2300e5ff' stroke-width='1.5'/%3E%3C/svg%3E">
|
||||
<title>{% block title %}GANDALF{% endblock %} — GANDALF</title>
|
||||
<title>{% block title %}Network Monitor{% endblock %} — GANDALF</title>
|
||||
|
||||
<link rel="preconnect" href="https://fonts.googleapis.com">
|
||||
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
|
||||
@@ -37,9 +37,15 @@
|
||||
<a href="{{ url_for('index') }}"
|
||||
class="lt-nav-drawer-link{% if request.endpoint == 'index' %} active{% endif %}"
|
||||
{% if request.endpoint == 'index' %}aria-current="page"{% endif %}>Dashboard</a>
|
||||
<a href="{{ url_for('events_page') }}"
|
||||
class="lt-nav-drawer-link{% if request.endpoint == 'events_page' %} active{% endif %}"
|
||||
{% if request.endpoint == 'events_page' %}aria-current="page"{% endif %}>History</a>
|
||||
<a href="{{ url_for('cutsheet_page') }}"
|
||||
class="lt-nav-drawer-link{% if request.endpoint == 'cutsheet_page' %} active{% endif %}"
|
||||
{% if request.endpoint == 'cutsheet_page' %}aria-current="page"{% endif %}>Cutsheet</a>
|
||||
<a href="{{ url_for('rack_page') }}"
|
||||
class="lt-nav-drawer-link{% if request.endpoint == 'rack_page' %} active{% endif %}"
|
||||
{% if request.endpoint == 'rack_page' %}aria-current="page"{% endif %}>Rack</a>
|
||||
{% if user.groups and 'admin' in user.groups %}
|
||||
<a href="{{ url_for('suppressions_page') }}"
|
||||
class="lt-nav-drawer-link{% if request.endpoint == 'suppressions_page' %} active{% endif %}"
|
||||
@@ -82,11 +88,21 @@
|
||||
{% if request.endpoint == 'index' %}aria-current="page"{% endif %}>
|
||||
Dashboard
|
||||
</a>
|
||||
<a href="{{ url_for('events_page') }}"
|
||||
class="lt-nav-link{% if request.endpoint == 'events_page' %} active{% endif %}"
|
||||
{% if request.endpoint == 'events_page' %}aria-current="page"{% endif %}>
|
||||
History
|
||||
</a>
|
||||
<a href="{{ url_for('cutsheet_page') }}"
|
||||
class="lt-nav-link{% if request.endpoint == 'cutsheet_page' %} active{% endif %}"
|
||||
{% if request.endpoint == 'cutsheet_page' %}aria-current="page"{% endif %}>
|
||||
Cutsheet
|
||||
</a>
|
||||
<a href="{{ url_for('rack_page') }}"
|
||||
class="lt-nav-link{% if request.endpoint == 'rack_page' %} active{% endif %}"
|
||||
{% if request.endpoint == 'rack_page' %}aria-current="page"{% endif %}>
|
||||
Rack
|
||||
</a>
|
||||
{% if user.groups and 'admin' in user.groups %}
|
||||
<a href="{{ url_for('suppressions_page') }}"
|
||||
class="lt-nav-link{% if request.endpoint == 'suppressions_page' %} active{% endif %}"
|
||||
@@ -324,7 +340,9 @@
|
||||
// Command palette
|
||||
lt.cmdPalette.init([
|
||||
{ id: 'nav-dashboard', group: 'Navigate', icon: '~', label: 'Dashboard', action: function() { window.location.href = '/'; } },
|
||||
{ id: 'nav-history', group: 'Navigate', icon: '⏱', label: 'Alert History', action: function() { window.location.href = '/events'; } },
|
||||
{ id: 'nav-cutsheet', group: 'Navigate', icon: '🔌', label: 'Cutsheet', action: function() { window.location.href = '/cutsheet'; } },
|
||||
{ id: 'nav-rack', group: 'Navigate', icon: '🗄', label: 'Rack', action: function() { window.location.href = '/rack'; } },
|
||||
{ id: 'nav-suppressions', group: 'Navigate', icon: '🔕', label: 'Suppressions', action: function() { window.location.href = '/suppressions'; } },
|
||||
{ id: 'action-refresh', group: 'Actions', icon: '↻', label: 'Refresh Data', kbd: 'R', action: function() { lt.autoRefresh.now(); } },
|
||||
{ id: 'action-suppress', group: 'Actions', icon: '🔕', label: 'New Suppression', kbd: 'S', action: function() { if (typeof openSuppressModal === 'function') openSuppressModal('host','',''); else window.location.href='/suppressions'; } },
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
{% extends "base.html" %}
|
||||
{% block title %}Cutsheet – GANDALF{% endblock %}
|
||||
{% block title %}Cutsheet{% endblock %}
|
||||
|
||||
{% block content %}
|
||||
<div class="lt-page-header">
|
||||
@@ -95,6 +95,7 @@
|
||||
<h2 class="g-section-title">Rack</h2>
|
||||
</div>
|
||||
<div id="cs-rack"></div>
|
||||
<p class="g-section-more"><a href="/rack">Open the rack drawing →</a></p>
|
||||
</section>
|
||||
|
||||
<section class="g-section" id="cs-items-section">
|
||||
|
||||
+29
-2
@@ -1,5 +1,5 @@
|
||||
{% extends "base.html" %}
|
||||
{% block title %}{{ device_name }} – GANDALF{% endblock %}
|
||||
{% block title %}{{ device_name }}{% endblock %}
|
||||
|
||||
{% block content %}
|
||||
<div class="lt-page-header">
|
||||
@@ -46,10 +46,37 @@
|
||||
</section>
|
||||
|
||||
{% include "_modem_panel.html" %}
|
||||
|
||||
{% if can_edit %}
|
||||
<div id="inventory-modal" class="lt-modal-overlay" role="dialog" aria-modal="true" aria-labelledby="inventory-title" aria-hidden="true">
|
||||
<div class="lt-modal">
|
||||
<div class="lt-modal-header">
|
||||
<h3 class="lt-modal-title" id="inventory-title">Inventory link</h3>
|
||||
<button type="button" class="lt-modal-close" data-modal-close aria-label="Close">✕</button>
|
||||
</div>
|
||||
<form id="inventory-form">
|
||||
<div class="lt-modal-body">
|
||||
<p class="cs-hint">Open this device's record in the inventory and paste its address here. Leave it empty to remove the link.</p>
|
||||
<div class="lt-form-group">
|
||||
<label class="lt-label" for="inventory-url">Inventory record URL</label>
|
||||
<input class="lt-input" id="inventory-url" name="url" type="url" inputmode="url" autocomplete="off"
|
||||
placeholder="https://inventory.lotusguild.org/item/…">
|
||||
</div>
|
||||
<p class="cs-hint"><a id="inventory-search" href="#" target="_blank" rel="noopener">Search the inventory for this model ↗</a></p>
|
||||
</div>
|
||||
<div class="lt-modal-footer">
|
||||
<button type="button" class="lt-btn lt-btn-secondary" data-modal-close>Cancel</button>
|
||||
<button type="submit" class="lt-btn lt-btn-primary">Save</button>
|
||||
</div>
|
||||
</form>
|
||||
</div>
|
||||
</div>
|
||||
{% endif %}
|
||||
{% endblock %}
|
||||
|
||||
{% block scripts %}
|
||||
<script>const DEVICE_NAME = {{ device_name | tojson }};</script>
|
||||
<script>const DEVICE_NAME = {{ device_name | tojson }};
|
||||
const CAN_EDIT = {{ 'true' if can_edit else 'false' }};</script>
|
||||
<script src="{{ url_for('static', filename='faceplates.js') }}"></script>
|
||||
<script src="{{ url_for('static', filename='port-inspector.js') }}"></script>
|
||||
<script src="{{ url_for('static', filename='device.js') }}"></script>
|
||||
|
||||
@@ -0,0 +1,66 @@
|
||||
{% extends "base.html" %}
|
||||
{% block title %}Alert History{% endblock %}
|
||||
|
||||
{% block content %}
|
||||
<div class="lt-page-header">
|
||||
<div>
|
||||
<h1 class="lt-page-title">Alert History</h1>
|
||||
<p class="g-page-sub">Every alert Gandalf raised, including the ones that cleared on their own. Resolved alerts are kept 90 days.</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<form class="cs-filters ev-filters" id="ev-filters" role="search" aria-label="Filter alert history">
|
||||
<input type="search" class="lt-input" id="ev-target" name="target" placeholder="Device, host, port or interface"
|
||||
aria-label="Filter by device" autocomplete="off">
|
||||
<select class="lt-select" id="ev-range" name="range" aria-label="Time range">
|
||||
<option value="24h">Last 24 hours</option>
|
||||
<option value="7d" selected>Last 7 days</option>
|
||||
<option value="30d">Last 30 days</option>
|
||||
<option value="90d">Last 90 days</option>
|
||||
</select>
|
||||
<select class="lt-select" id="ev-type" name="type" aria-label="Alert type">
|
||||
<option value="">All alert types</option>
|
||||
</select>
|
||||
<select class="lt-select" id="ev-severity" name="severity" aria-label="Severity">
|
||||
<option value="">All severities</option>
|
||||
<option value="critical">Critical</option>
|
||||
<option value="warning">Warning</option>
|
||||
<option value="info">Info</option>
|
||||
</select>
|
||||
<select class="lt-select" id="ev-status" name="status" aria-label="State">
|
||||
<option value="all">Open and cleared</option>
|
||||
<option value="active">Open now</option>
|
||||
<option value="resolved">Cleared</option>
|
||||
</select>
|
||||
<button type="button" class="lt-btn lt-btn-secondary lt-btn-sm" id="ev-clear" hidden>Clear filters</button>
|
||||
</form>
|
||||
|
||||
<div id="ev-summary" class="cs-summary" aria-live="polite"></div>
|
||||
|
||||
<section class="g-section" id="ev-flappers-section" hidden>
|
||||
<div class="g-section-header">
|
||||
<h2 class="g-section-title">Recurring</h2>
|
||||
</div>
|
||||
<p class="cs-hint">The same alert on the same target more than once in this range. Something that keeps coming back usually has one cause worth finding.</p>
|
||||
<div class="lt-frame">
|
||||
<span class="lt-frame-bl">╚</span><span class="lt-frame-br">╝</span>
|
||||
<div id="ev-flappers"></div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="g-section">
|
||||
<div class="g-section-header">
|
||||
<h2 class="g-section-title">Alerts</h2>
|
||||
<span class="g-section-badge" id="ev-count">0</span>
|
||||
</div>
|
||||
<div class="lt-frame">
|
||||
<span class="lt-frame-bl">╚</span><span class="lt-frame-br">╝</span>
|
||||
<div id="ev-table"><div class="lt-empty-state lt-empty-state--sm"><div class="lt-empty-state-title">Loading…</div></div></div>
|
||||
</div>
|
||||
<div class="ev-pager" id="ev-pager"></div>
|
||||
</section>
|
||||
{% endblock %}
|
||||
|
||||
{% block scripts %}
|
||||
<script src="{{ url_for('static', filename='events.js') }}"></script>
|
||||
{% endblock %}
|
||||
@@ -1,5 +1,5 @@
|
||||
{% extends "base.html" %}
|
||||
{% block title %}Dashboard – GANDALF{% endblock %}
|
||||
{% block title %}Dashboard{% endblock %}
|
||||
|
||||
{% block content %}
|
||||
|
||||
|
||||
@@ -0,0 +1,38 @@
|
||||
{% extends "base.html" %}
|
||||
{% block title %}Rack{% endblock %}
|
||||
|
||||
{% block content %}
|
||||
<div class="lt-page-header">
|
||||
<div>
|
||||
<h1 class="lt-page-title">Rack</h1>
|
||||
<p class="g-page-sub">The rack front, drawn from the rack units on the cutsheet with live port state. Hover a port to light up every port on the same cable run; click it to open the port or the cable.</p>
|
||||
</div>
|
||||
{% if can_edit %}
|
||||
<div class="cs-actions no-print">
|
||||
<a class="lt-btn lt-btn-secondary lt-btn-sm" href="/cutsheet#cs-items-section">Edit positions</a>
|
||||
</div>
|
||||
{% endif %}
|
||||
</div>
|
||||
|
||||
<div class="rack-warnings" id="rack-warnings" role="alert" hidden></div>
|
||||
|
||||
<section class="g-section">
|
||||
<div class="g-section-header">
|
||||
<h2 class="g-section-title">Front</h2>
|
||||
<span class="g-section-badge" id="rack-count">0</span>
|
||||
</div>
|
||||
<p class="rack-readout" id="rack-readout" aria-live="polite"></p>
|
||||
<div class="rack-legend" aria-hidden="true">
|
||||
<span><i class="rack-key rack-key-ok"></i>matches cutsheet</span>
|
||||
<span><i class="rack-key rack-key-bad"></i>mismatch</span>
|
||||
<span><i class="rack-key rack-key-doc"></i>documented, unconfirmed</span>
|
||||
</div>
|
||||
<div class="rack" id="rack"><div class="link-loading">Loading rack</div></div>
|
||||
<p class="cs-hint rack-unplaced" id="rack-unplaced"></p>
|
||||
</section>
|
||||
{% endblock %}
|
||||
|
||||
{% block scripts %}
|
||||
<script src="{{ url_for('static', filename='faceplates.js') }}"></script>
|
||||
<script src="{{ url_for('static', filename='rack.js') }}"></script>
|
||||
{% endblock %}
|
||||
@@ -1,5 +1,5 @@
|
||||
{% extends "base.html" %}
|
||||
{% block title %}Suppressions – GANDALF{% endblock %}
|
||||
{% block title %}Suppressions{% endblock %}
|
||||
|
||||
{% block content %}
|
||||
|
||||
|
||||
@@ -35,6 +35,7 @@ def client():
|
||||
db.count_active_events.return_value = 0
|
||||
db.get_status_summary.return_value = {'critical': 0, 'warning': 0, 'info': 0, 'resolved_24h': 0}
|
||||
db.check_suppressed.return_value = False
|
||||
db.get_device_link.return_value = ''
|
||||
with patch.object(gandalf, '_cfg', CONFIG):
|
||||
yield gandalf.app.test_client()
|
||||
|
||||
@@ -316,3 +317,154 @@ class TestCutsheetLivePorts:
|
||||
def test_admin_only(self, cs_client):
|
||||
client, _ = cs_client
|
||||
assert client.get('/api/cutsheet/live-ports', headers=USER).status_code == 403
|
||||
|
||||
|
||||
class TestAlertHistory:
|
||||
HISTORY = {'events': [], 'total': 0, 'by_type': {}}
|
||||
|
||||
def _db(self):
|
||||
db = gandalf.db
|
||||
db.get_events_history.return_value = dict(self.HISTORY)
|
||||
db.get_flappers.return_value = []
|
||||
db.get_event_types.return_value = ['host_unreachable']
|
||||
return db
|
||||
|
||||
def test_page_and_nav(self, client):
|
||||
html = client.get('/events', headers=AUTH).get_data(as_text=True)
|
||||
assert 'Alert History' in html and 'events.js' in html
|
||||
assert 'href="/events"' in client.get('/', headers=AUTH).get_data(as_text=True)
|
||||
|
||||
def test_defaults_to_seven_days(self, client):
|
||||
db = self._db()
|
||||
data = client.get('/api/events/history', headers=AUTH).get_json()
|
||||
assert db.get_events_history.call_args.args == (168, '', '', '', 'all')
|
||||
assert data['types'] == ['host_unreachable'] and data['page_size'] == 100
|
||||
|
||||
def test_filters_are_passed_through(self, client):
|
||||
db = self._db()
|
||||
client.get('/api/events/history?range=90d&type=interface_down&target=+large1+&severity=critical'
|
||||
'&status=resolved&offset=200', headers=AUTH)
|
||||
assert db.get_events_history.call_args.args == (2160, 'interface_down', 'large1', 'critical', 'resolved')
|
||||
assert db.get_events_history.call_args.kwargs == {'limit': 100, 'offset': 200}
|
||||
assert db.get_flappers.call_args.args == (2160, 'interface_down', 'large1', 'critical')
|
||||
|
||||
@pytest.mark.parametrize('query', ['range=1y', 'severity=bad', 'status=open', 'offset=x'])
|
||||
def test_bad_filters_are_rejected(self, client, query):
|
||||
self._db()
|
||||
assert client.get(f'/api/events/history?{query}', headers=AUTH).status_code == 400
|
||||
|
||||
def test_needs_login(self, client):
|
||||
assert client.get('/api/events/history').status_code == 401
|
||||
|
||||
|
||||
class TestHealth:
|
||||
def _health(self, client, age_s, poll=120):
|
||||
from datetime import datetime, timedelta, timezone
|
||||
stamp = (datetime.now(timezone.utc) - timedelta(seconds=age_s)).strftime('%Y-%m-%d %H:%M:%S UTC')
|
||||
gandalf.db.get_state.side_effect = lambda key, default=None: stamp if key == 'last_check' else default
|
||||
with patch.object(gandalf, '_cfg', dict(CONFIG, monitor={'poll_interval': poll})):
|
||||
return client.get('/health')
|
||||
|
||||
def test_recent_check_is_healthy(self, client):
|
||||
resp = self._health(client, 150)
|
||||
assert resp.status_code == 200 and resp.get_json()['checks']['monitor'] == 'ok'
|
||||
|
||||
def test_stalled_monitor_fails_after_ten_minutes(self, client):
|
||||
resp = self._health(client, 660)
|
||||
assert resp.status_code == 503
|
||||
assert resp.get_json()['checks'] == {'db': 'ok', 'monitor': 'stale', 'monitor_age_s': 660}
|
||||
|
||||
def test_limit_follows_a_long_poll_interval(self, client):
|
||||
assert self._health(client, 1000, poll=300).status_code == 200
|
||||
assert self._health(client, 1600, poll=300).status_code == 503
|
||||
|
||||
|
||||
class TestPageTitles:
|
||||
@pytest.mark.parametrize('path,title', [('/', 'Dashboard — GANDALF'), ('/events', 'Alert History — GANDALF'),
|
||||
('/cutsheet', 'Cutsheet — GANDALF')])
|
||||
def test_name_appears_once(self, client, path, title):
|
||||
gandalf._cutsheet_schema_ready = True
|
||||
html = client.get(path, headers=AUTH).get_data(as_text=True)
|
||||
assert f'<title>{title}</title>' in html
|
||||
|
||||
|
||||
class TestInventoryLinks:
|
||||
def _put(self, client, url, headers=AUTH, name='USW Flex'):
|
||||
gandalf._cutsheet_schema_ready = True
|
||||
return client.put(f'/api/device/{name}/inventory', json={'url': url}, headers=headers)
|
||||
|
||||
def test_device_api_has_link_and_search(self, client):
|
||||
gandalf._cutsheet_schema_ready = True
|
||||
gandalf.db.get_device_link.return_value = 'https://inventory.lotusguild.org/item/abc'
|
||||
inv = client.get('/api/device/USW Flex', headers=AUTH).get_json()['inventory']
|
||||
assert inv == {'url': 'https://inventory.lotusguild.org/item/abc',
|
||||
'search_url': 'https://inventory.lotusguild.org/items?q=USF5P'}
|
||||
|
||||
def test_admin_sets_and_clears(self, client):
|
||||
assert self._put(client, ' https://inventory.lotusguild.org/item/abc ').status_code == 200
|
||||
gandalf.db.set_device_link.assert_called_with('USW Flex', 'https://inventory.lotusguild.org/item/abc', 'jared')
|
||||
assert self._put(client, '').status_code == 200
|
||||
gandalf.db.set_device_link.assert_called_with('USW Flex', '', 'jared')
|
||||
|
||||
@pytest.mark.parametrize('url', ['http://inventory.lotusguild.org/item/a', 'https://evil.example/item/a',
|
||||
'https://inventory.lotusguild.org.evil.example/x', 'javascript:alert(1)',
|
||||
'https://user:pw@inventory.lotusguild.org/x',
|
||||
'https://inventory.lotusguild.org/' + 'a' * 600])
|
||||
def test_only_inventory_links_are_accepted(self, client, url):
|
||||
assert self._put(client, url).status_code == 400
|
||||
assert not gandalf.db.set_device_link.called
|
||||
|
||||
def test_configured_host(self, client):
|
||||
cfg = dict(CONFIG, inventory={'base_url': 'https://homebox.example'})
|
||||
with patch.object(gandalf, '_cfg', cfg):
|
||||
assert self._put(client, 'https://homebox.example/item/1').status_code == 200
|
||||
assert self._put(client, 'https://inventory.lotusguild.org/item/1').status_code == 400
|
||||
|
||||
def test_needs_admin_and_json(self, client):
|
||||
user = dict(AUTH, **{'Remote-Groups': 'staff'})
|
||||
with patch.object(gandalf, '_cfg', dict(CONFIG, auth=dict(CONFIG['auth'], allowed_groups=['admin', 'staff']))):
|
||||
assert self._put(client, 'https://inventory.lotusguild.org/x', headers=user).status_code == 403
|
||||
gandalf._cutsheet_schema_ready = True
|
||||
resp = client.put('/api/device/USW Flex/inventory', data='url=x', headers=AUTH)
|
||||
assert resp.status_code == 400
|
||||
|
||||
|
||||
class TestRackApi:
|
||||
def test_rack_combines_items_switches_and_marks(self, client):
|
||||
gandalf._cutsheet_schema_ready = True
|
||||
db = gandalf.db
|
||||
db.cutsheet_load.return_value = {
|
||||
'items': [
|
||||
{'id': 1, 'name': 'USW Flex', 'kind': 'device', 'port_count': None, 'rack_u': 20, 'rack_height': 1,
|
||||
'location': '', 'notes': ''},
|
||||
{'id': 2, 'name': 'PP1', 'kind': 'patch_panel', 'port_count': 24, 'rack_u': 21, 'rack_height': 1,
|
||||
'location': '', 'notes': ''},
|
||||
{'id': 3, 'name': 'Shelf', 'kind': 'shelf', 'port_count': None, 'rack_u': 20, 'rack_height': 1,
|
||||
'location': '', 'notes': ''},
|
||||
{'id': 4, 'name': 'Loose', 'kind': 'other', 'port_count': None, 'rack_u': None, 'rack_height': None,
|
||||
'location': '', 'notes': ''},
|
||||
],
|
||||
'cables': [{'id': 7, 'label': 'C07', 'a_device': 'USW Flex', 'a_port': '2', 'a_side': '',
|
||||
'b_device': 'PP1', 'b_port': '5', 'b_side': 'front', 'status': 'installed',
|
||||
'cable_type': '', 'length_m': None, 'color': '', 'notes': '', 'label_applied': False}],
|
||||
'port_labels': [{'item_id': 2, 'item_name': 'PP1', 'port': '5', 'label': 'OFFICE', 'applied': False}],
|
||||
}
|
||||
data = client.get('/api/rack', headers=AUTH).get_json()
|
||||
assert data['units'] == 42 and data['unplaced'] == ['Loose']
|
||||
assert [i['name'] for i in data['items']] == ['PP1', 'USW Flex']
|
||||
assert data['overlaps'] == [{'name': 'Shelf', 'units': 'U20', 'with': ['USW Flex']}]
|
||||
pp, sw = data['items']
|
||||
assert pp['labels'] == {'5': 'OFFICE'} and pp['url'] == '/cutsheet#item-2'
|
||||
assert pp['ports']['5']['run'] == [7] and pp['ports']['5']['cable_id'] == 7
|
||||
assert sw['url'] == '/device/USW%20Flex' and 'switch' not in sw # no port data in this snapshot
|
||||
assert sw['ports']['2']['mark'] in ('ok', 'bad', 'doc')
|
||||
|
||||
def test_rack_page_and_auth(self, client):
|
||||
assert 'rack.js' in client.get('/rack', headers=AUTH).get_data(as_text=True)
|
||||
assert client.get('/api/rack').status_code == 401
|
||||
|
||||
def test_units_from_config(self, client):
|
||||
gandalf._cutsheet_schema_ready = True
|
||||
gandalf.db.cutsheet_load.return_value = {'items': [], 'cables': [], 'port_labels': []}
|
||||
with patch.object(gandalf, '_cfg', dict(CONFIG, rack={'units': 24})):
|
||||
assert client.get('/api/rack', headers=AUTH).get_json()['units'] == 24
|
||||
@@ -248,3 +248,51 @@ class TestProblems:
|
||||
cables = [cable(1, ('sw', '13'), ('tv', 'eth0')), cable(2, ('sw', '13'), ('pc', 'eth0'))]
|
||||
found = cs.problems(cables, ITEMS, switch([port(13, seen='tv')]))
|
||||
assert sorted(p['cable_id'] for p in found) == [1, 2]
|
||||
|
||||
|
||||
class TestRackLayout:
|
||||
_ids = {}
|
||||
|
||||
def _item(self, name, u, h=None):
|
||||
# IDs in creation order: on a tie for the same top unit, the earlier record is drawn
|
||||
return {'id': self._ids.setdefault(name, len(self._ids) + 1), 'name': name, 'kind': 'device',
|
||||
'rack_u': u, 'rack_height': h}
|
||||
|
||||
def test_top_first_with_top_unit(self):
|
||||
out = cs.rack_layout([self._item('a', 1), self._item('b', 10, 2), {'name': 'loose', 'rack_u': None}], 42)
|
||||
assert [(i['name'], i['rack_top']) for i in out['placed']] == [('b', 11), ('a', 1)]
|
||||
assert out['overlaps'] == [] and out['outside'] == []
|
||||
|
||||
def test_overlap_is_reported_not_drawn(self):
|
||||
out = cs.rack_layout([self._item('ups', 1, 2), self._item('shelf', 2)], 42)
|
||||
assert [i['name'] for i in out['placed']] == ['ups']
|
||||
assert out['overlaps'] == [{'name': 'shelf', 'units': 'U2', 'with': ['ups']}]
|
||||
|
||||
def test_past_the_top(self):
|
||||
out = cs.rack_layout([self._item('tall', 41, 3)], 42)
|
||||
assert out['placed'] == [] and out['outside'] == [{'name': 'tall', 'units': 'U41–43'}]
|
||||
|
||||
|
||||
class TestPortMarks:
|
||||
def test_worst_cable_wins_and_runs_merge(self):
|
||||
cables = [
|
||||
dict(cable(1, ('sw', '3'), ('PP1', '1', 'front'), label='C1'), verdict='match', detail='ok', run=[1, 2]),
|
||||
dict(cable(2, ('PP1', '1', 'rear'), ('jack', '1')), verdict='wrong_neighbor', detail='bad', run=[1, 2]),
|
||||
dict(cable(3, ('sw', '4'), ('x',)), verdict='planned', detail='Planned', run=[3]),
|
||||
dict(cable(4, ('sw', '5'), ('y',), status='retired'), verdict='retired', detail='', run=[4]),
|
||||
]
|
||||
marks = cs.port_marks(cables)
|
||||
assert marks[('sw', '3')]['mark'] == 'ok'
|
||||
assert marks[('PP1', '1')]['mark'] == 'bad' and marks[('PP1', '1')]['cable_id'] == 2
|
||||
assert marks[('PP1', '1')]['run'] == [1, 2]
|
||||
assert marks[('PP1', '1')]['text'] == 'cable C1: ok; cable #2: bad'
|
||||
assert marks[('sw', '4')]['text'] == 'cable #3: Planned'
|
||||
assert marks[('sw', '4')]['mark'] == 'doc'
|
||||
assert ('sw', '5') not in marks and ('x', '') not in marks
|
||||
|
||||
def test_shared_check_is_said_once(self):
|
||||
cables = [
|
||||
dict(cable(1, ('sw', '3'), ('PP1', '1', 'front'), label='C1'), verdict='idle', detail='Idle', run=[1, 2]),
|
||||
dict(cable(2, ('PP1', '1', 'rear'), ('jack', '1')), verdict='idle', detail='Idle', run=[1, 2]),
|
||||
]
|
||||
assert cs.port_marks(cables)[('PP1', '1')]['text'] == 'cables C1, #2: Idle'
|
||||
@@ -3,6 +3,7 @@ import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
from datetime import timedelta
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import pytest
|
||||
@@ -24,6 +25,12 @@ def db():
|
||||
mock_db.upsert_event.return_value = (1, False, 2, None)
|
||||
mock_db.get_modem_rx_values.return_value = []
|
||||
mock_db.get_state.return_value = None
|
||||
mock_db.event_stats.return_value = {'last_7d': 0, 'last_30d': 0,
|
||||
'last_first_seen': None, 'last_resolved_at': None}
|
||||
mock_db.get_events_for_target.return_value = []
|
||||
mock_db.cutsheet_load.return_value = {'items': [], 'cables': [], 'port_labels': []}
|
||||
mock_db.get_pending_recovery_notes.return_value = []
|
||||
mock_db.get_device_link.return_value = ''
|
||||
yield mock_db
|
||||
|
||||
|
||||
@@ -560,3 +567,184 @@ class TestCutsheetMismatch:
|
||||
assert not db.upsert_event.called
|
||||
resolved = {c.args for c in db.resolve_event.call_args_list}
|
||||
assert ('cable_mismatch', 'sw', 'cable 7') in resolved and ('cable_mismatch', 'other', 'cable 9') in resolved
|
||||
|
||||
|
||||
class TestTicketContext:
|
||||
SWITCHES = {'agg': {'ports': {
|
||||
'Large1': {'port_idx': 7, 'up': True, 'speed_mbps': 10000,
|
||||
'lldp': {'system_name': 'large1', 'port_id': 'enp1s0f0'},
|
||||
'sfp': {'found': True, 'kind': 'dac', 'part': 'DAC-10G'}},
|
||||
'Port 3': {'port_idx': 3, 'up': False, 'speed_mbps': 0},
|
||||
}}}
|
||||
|
||||
def _mon(self, mon, switches=None):
|
||||
mon.public_url = 'https://gandalf.test'
|
||||
mon.poll_interval = 120
|
||||
mon._link_data = {'unifi_switches': switches if switches is not None else self.SWITCHES}
|
||||
return mon
|
||||
|
||||
def _desc(self, mon):
|
||||
return mon.tickets.create.call_args.args[1]
|
||||
|
||||
def test_interface_ticket_names_the_switch_port_and_links(self, mon, db):
|
||||
self._mon(mon)._ticket_interface(1, 'large1', 'enp1s0f0', 2)
|
||||
desc = self._desc(mon)
|
||||
assert '### Where it is connected' in desc
|
||||
assert '`enp1s0f0` → [agg port 7 "Large1": up at 10 Gbps, module: DAC DAC-10G]' in desc
|
||||
assert '(https://gandalf.test/device/agg#port-7)' in desc
|
||||
assert 'https://gandalf.test/device/large1' in desc
|
||||
assert 'Opened automatically by Gandalf' in desc
|
||||
assert 'inventory' not in desc
|
||||
|
||||
def test_inventory_link_when_set(self, mon, db):
|
||||
db.get_device_link.return_value = 'https://inv.test/item/abc'
|
||||
self._mon(mon)._ticket_interface(1, 'large1', 'enp1s0f0', 2)
|
||||
assert '- [large1 in the inventory](https://inv.test/item/abc)' in self._desc(mon)
|
||||
db.get_device_link.assert_called_with('large1')
|
||||
|
||||
def test_dead_host_uses_the_last_known_port(self, mon, db):
|
||||
mon._known_uplinks = {'large1': {'seen': '2026-11-10 10:00:00 UTC', 'uplinks': [
|
||||
{'switch': 'agg', 'port': 'Port 3', 'port_idx': 3, 'host_interface': 'eno1'}]}}
|
||||
self._mon(mon)._ticket_metrics_missing(1, 'micro1', 'x:9100', 5)
|
||||
assert 'not seen this device on a switch port' in self._desc(mon)
|
||||
mon.tickets.create.reset_mock()
|
||||
unplugged = {'agg': {'ports': {'Port 3': {'port_idx': 3, 'up': False}}}}
|
||||
self._mon(mon, unplugged)._ticket_unreachable(1, 'large1', '10.0.0.9', 5)
|
||||
desc = self._desc(mon)
|
||||
assert 'agg port 3: down' in desc
|
||||
assert 'last seen linked there 2026-11-10 10:00:00 UTC' in desc
|
||||
assert 'Down for at least' in desc and '10 min' in desc
|
||||
|
||||
def test_cutsheet_runs_and_history(self, mon, db):
|
||||
db.cutsheet_load.return_value = {'items': [], 'port_labels': [], 'cables': [
|
||||
{'id': 4, 'label': 'C04', 'a_device': 'agg', 'a_port': '7', 'a_side': '',
|
||||
'b_device': 'large1', 'b_port': 'enp1s0f0', 'b_side': '', 'status': 'installed'}]}
|
||||
db.event_stats.return_value = {'last_7d': 3, 'last_30d': 5,
|
||||
'last_first_seen': monitor._utcnow(), 'last_resolved_at': monitor._utcnow()}
|
||||
self._mon(mon)._ticket_interface(1, 'large1', 'enp1s0f0', 2)
|
||||
desc = self._desc(mon)
|
||||
assert '[**C04**](https://gandalf.test/cutsheet#cable-4): agg port 7 → large1 enp1s0f0' in desc
|
||||
assert 'Happened 3 times in the last 7 days and 5 in the last 30' in desc
|
||||
assert 'This keeps coming back' in desc
|
||||
|
||||
def test_other_open_alerts_are_listed(self, mon, db):
|
||||
db.get_events_for_target.return_value = [
|
||||
{'id': 1, 'event_type': 'interface_down', 'target_name': 'large1', 'target_detail': 'eno1'},
|
||||
{'id': 2, 'event_type': 'interface_down', 'target_name': 'large1', 'target_detail': 'eno2',
|
||||
'first_seen': 'T', 'ticket_id': '55'},
|
||||
{'id': 3, 'event_type': 'host_unreachable', 'target_name': 'large1', 'resolved_at': 'R'},
|
||||
]
|
||||
self._mon(mon)._ticket_interface(1, 'large1', 'eno1', 2)
|
||||
desc = self._desc(mon)
|
||||
assert '- interface down on large1 eno2 since T (ticket #55)' in desc
|
||||
assert 'eno1 since' not in desc and 'host unreachable' not in desc
|
||||
|
||||
def test_a_failing_section_still_opens_the_ticket(self, mon, db):
|
||||
db.event_stats.side_effect = RuntimeError('db gone')
|
||||
self._mon(mon)._ticket_interface(1, 'large1', 'enp1s0f0', 2)
|
||||
desc = self._desc(mon)
|
||||
assert '### History' not in desc and '### Where it is connected' in desc
|
||||
db.set_ticket_id.assert_called_with(1, '123')
|
||||
|
||||
def test_cable_mismatch_shows_the_switch_port(self, mon, db):
|
||||
self._mon(mon)._ticket_cable_mismatch(9, {
|
||||
'switch': 'agg', 'port': '7', 'cable_id': 4, 'label': 'C04', 'key': 'cable 4',
|
||||
'detail': 'A different device is connected', 'path': 'agg port 7 → tv'}, 24)
|
||||
desc = self._desc(mon)
|
||||
assert '[C04](https://gandalf.test/cutsheet#cable-4)' in desc
|
||||
assert '[agg port 7 "Large1": up at 10 Gbps' in desc
|
||||
|
||||
def test_titles_keep_the_tag_format(self, mon, db):
|
||||
mon.unifi_fail_thresh = 3
|
||||
self._mon(mon)._ticket_unifi(1, {'name': 'ap1', 'type': 'uap', 'model': 'U6', 'ip': '1.2.3.4'})
|
||||
assert mon.tickets.create.call_args.args[0] == \
|
||||
'[ap1][auto][production][issue][network][single-node] UniFi device offline'
|
||||
|
||||
def test_remember_uplinks_saves_only_on_change(self, mon, db):
|
||||
self._mon(mon)
|
||||
mon._known_uplinks = None
|
||||
mon._remember_uplinks({'unifi_switches': self.SWITCHES})
|
||||
saved = db.set_state.call_args.args
|
||||
assert saved[0] == 'known_uplinks'
|
||||
assert saved[1]['large1']['uplinks'] == [
|
||||
{'switch': 'agg', 'port': 'Large1', 'port_idx': 7, 'host_interface': 'enp1s0f0'}]
|
||||
db.set_state.reset_mock()
|
||||
mon._remember_uplinks({'unifi_switches': self.SWITCHES})
|
||||
assert not db.set_state.called
|
||||
|
||||
|
||||
class TestRecoveryNotes:
|
||||
def _event(self, **kw):
|
||||
now = monitor._utcnow()
|
||||
e = {'id': 8, 'event_type': 'host_metrics_missing', 'target_name': 'micro1', 'target_detail': 'x:9100',
|
||||
'ticket_id': '993', 'first_seen': now - timedelta(minutes=12), 'resolved_at': now,
|
||||
'consecutive_failures': 6}
|
||||
e.update(kw)
|
||||
return e
|
||||
|
||||
def test_cleared_alert_comments_and_marks(self, mon, db):
|
||||
mon.public_url = 'https://gandalf.test'
|
||||
mon.tickets.comments_denied = False
|
||||
mon.tickets.comment.return_value = True
|
||||
db.get_pending_recovery_notes.return_value = [self._event()]
|
||||
mon._post_recovery_notes()
|
||||
ticket_id, text = mon.tickets.comment.call_args.args
|
||||
assert ticket_id == '993'
|
||||
assert 'Cleared' in text and '12 min' in text and 'Failed checks:** 6' in text
|
||||
assert 'https://gandalf.test/device/micro1' in text
|
||||
db.mark_recovery_notified.assert_called_once_with(8)
|
||||
assert db.get_pending_recovery_notes.call_args.kwargs['exclude_types'] == monitor.ONE_SHOT_EVENTS
|
||||
|
||||
def test_failed_comment_is_left_for_next_cycle(self, mon, db):
|
||||
mon.tickets.comments_denied = False
|
||||
mon.tickets.comment.return_value = False
|
||||
db.get_pending_recovery_notes.return_value = [self._event(), self._event(id=9)]
|
||||
mon._post_recovery_notes()
|
||||
assert not db.mark_recovery_notified.called
|
||||
assert mon.tickets.comment.call_count == 2
|
||||
|
||||
def test_denied_key_stops_trying(self, mon, db):
|
||||
mon.tickets.comments_denied = True
|
||||
mon._post_recovery_notes()
|
||||
assert not db.get_pending_recovery_notes.called
|
||||
|
||||
def test_cable_mismatch_links_the_cutsheet(self, mon, db):
|
||||
mon.public_url = 'https://gandalf.test'
|
||||
url = mon._event_url({'event_type': 'cable_mismatch', 'target_name': 'sw', 'target_detail': 'cable 7'})
|
||||
assert url == 'https://gandalf.test/cutsheet#cable-7'
|
||||
|
||||
|
||||
class TestTicketClientComment:
|
||||
def _client(self):
|
||||
return monitor.TicketClient({'url': 'http://t.test/create_ticket_api.php', 'api_key': 'k'})
|
||||
|
||||
def test_comment_endpoint_sits_beside_create(self):
|
||||
assert self._client().comment_url == 'http://t.test/api/ticket_comment_api.php'
|
||||
sub = monitor.TicketClient({'url': 'http://t.test/tinker/create_ticket_api.php', 'api_key': 'k'})
|
||||
assert sub.comment_url == 'http://t.test/tinker/api/ticket_comment_api.php'
|
||||
|
||||
def test_payload(self):
|
||||
c = self._client()
|
||||
resp = MagicMock(status_code=200)
|
||||
resp.json.return_value = {'success': True, 'comment_id': 1}
|
||||
with patch.object(monitor.requests, 'post', return_value=resp) as post:
|
||||
assert c.comment('42', 'hi') is True
|
||||
assert post.call_args.kwargs['json'] == {'ticket_id': '42', 'comment_text': 'hi', 'markdown_enabled': True}
|
||||
assert post.call_args.kwargs['headers'] == {'Authorization': 'Bearer k'}
|
||||
|
||||
def test_forbidden_disables_comments(self):
|
||||
c = self._client()
|
||||
with patch.object(monitor.requests, 'post', return_value=MagicMock(status_code=403)) as post:
|
||||
assert c.comment('42', 'hi') is False
|
||||
assert c.comment('43', 'hi') is False
|
||||
assert post.call_count == 1 and c.comments_denied
|
||||
|
||||
def test_missing_ticket_counts_as_done(self):
|
||||
with patch.object(monitor.requests, 'post', return_value=MagicMock(status_code=404)):
|
||||
assert self._client().comment('42', 'hi') is True
|
||||
|
||||
def test_network_error_is_retryable(self):
|
||||
c = self._client()
|
||||
with patch.object(monitor.requests, 'post', side_effect=OSError('down')):
|
||||
assert c.comment('42', 'hi') is False
|
||||
assert not c.comments_denied
|
||||
@@ -0,0 +1,64 @@
|
||||
"""Tests for ticket_text.py: the Markdown Gandalf puts in tickets."""
|
||||
import os
|
||||
import sys
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
|
||||
|
||||
import ticket_text as tt # noqa: E402
|
||||
|
||||
NOW = datetime(2026, 11, 10, 18, 0)
|
||||
|
||||
|
||||
def test_duration():
|
||||
assert tt.duration(42) == '42 s'
|
||||
assert tt.duration(600) == '10 min'
|
||||
assert tt.duration(3600) == '1 h'
|
||||
assert tt.duration(3 * 3600 + 120) == '3 h 2 min'
|
||||
assert tt.duration(3 * 86400 + 7200) == '3 d 2 h'
|
||||
|
||||
|
||||
def test_speed():
|
||||
assert tt.speed(100) == '100 Mbps'
|
||||
assert tt.speed(2500) == '2.5 Gbps'
|
||||
assert tt.speed(10000) == '10 Gbps'
|
||||
assert tt.speed(0) == ''
|
||||
|
||||
|
||||
def test_device_url_escapes_names():
|
||||
assert tt.device_url('https://g', 'USW Flex Mini', 3) == 'https://g/device/USW%20Flex%20Mini#port-3'
|
||||
assert tt.device_url('', 'x') == ''
|
||||
|
||||
|
||||
def test_port_text():
|
||||
assert tt.port_text('sw', 'Port 4', {'port_idx': 4, 'up': True, 'speed_mbps': 1000, 'poe_power': 4.21}) == \
|
||||
'sw port 4: up at 1 Gbps, PoE 4.2 W'
|
||||
assert tt.port_text('sw', 'Cam', {'port_idx': 9, 'up': False,
|
||||
'sfp': {'found': True, 'kind': 'optic', 'part': 'SR', 'rx_fault': True}}) == \
|
||||
'sw port 9 "Cam": down, module: optic SR, **module reports a fault**'
|
||||
|
||||
|
||||
def test_description_skips_empty_facts_and_sections():
|
||||
text = tt.description('Alert', [('Host', 'a'), ('Blank', '')], 'Summary.', ['One', 'Two'],
|
||||
[('Empty', []), ('Links', ['- x'])])
|
||||
assert text.startswith('## Alert\n\n- **Host:** a\n\nSummary.')
|
||||
assert 'Blank' not in text and '### Empty' not in text
|
||||
assert '1. One\n2. Two' in text and '### Links\n- x' in text
|
||||
|
||||
|
||||
def test_history_lines():
|
||||
assert tt.history_lines({'last_7d': 0, 'last_30d': 0}, NOW) == ['- First time in the last 30 days.']
|
||||
lines = tt.history_lines({'last_7d': 1, 'last_30d': 2, 'last_first_seen': NOW - timedelta(hours=26),
|
||||
'last_resolved_at': NOW - timedelta(hours=25, minutes=55)}, NOW)
|
||||
assert lines[0].startswith('- Happened 1 time in the last 7 days and 2')
|
||||
assert lines[1] == '- Last cleared 2026-11-09 16:05 UTC (25 h 55 min ago), after 5 min.'
|
||||
|
||||
|
||||
def test_recovery_comment():
|
||||
text = tt.recovery_comment({'event_type': 'interface_down', 'target_name': 'large1', 'target_detail': 'eno1',
|
||||
'first_seen': NOW - timedelta(minutes=7), 'resolved_at': NOW,
|
||||
'consecutive_failures': 3}, NOW, 'https://g/device/large1')
|
||||
assert text.startswith('**✅ Cleared**: interface down on large1 eno1.')
|
||||
assert '- **Lasted:** 7 min (first seen 2026-11-10 17:53 UTC)' in text
|
||||
assert '- **Gandalf:** https://g/device/large1' in text
|
||||
assert 'leaves this ticket open' in text
|
||||
+149
@@ -0,0 +1,149 @@
|
||||
"""Markdown for the tickets Gandalf opens in Tinker Tickets, and the notes it adds to them.
|
||||
|
||||
Pure formatting only: monitor.py gathers the facts and passes them in.
|
||||
"""
|
||||
from datetime import datetime, timezone
|
||||
from typing import List, Optional, Sequence, Tuple
|
||||
from urllib.parse import quote
|
||||
|
||||
Section = Tuple[str, List[str]]
|
||||
|
||||
|
||||
def duration(seconds) -> str:
|
||||
seconds = max(0, int(seconds))
|
||||
if seconds < 60:
|
||||
return f'{seconds} s'
|
||||
minutes = seconds // 60
|
||||
if minutes < 60:
|
||||
return f'{minutes} min'
|
||||
hours, minutes = divmod(minutes, 60)
|
||||
if hours < 48:
|
||||
return f'{hours} h {minutes} min' if minutes else f'{hours} h'
|
||||
days, hours = divmod(hours, 24)
|
||||
return f'{days} d {hours} h' if hours else f'{days} d'
|
||||
|
||||
|
||||
def utc(dt: Optional[datetime]) -> str:
|
||||
"""Database times are naive UTC."""
|
||||
if dt is None:
|
||||
return 'unknown'
|
||||
if dt.tzinfo is not None:
|
||||
dt = dt.astimezone(timezone.utc).replace(tzinfo=None)
|
||||
return dt.strftime('%Y-%m-%d %H:%M UTC')
|
||||
|
||||
|
||||
def speed(mbps) -> str:
|
||||
if not mbps:
|
||||
return ''
|
||||
mbps = int(mbps)
|
||||
if mbps >= 1000:
|
||||
gbps = mbps / 1000
|
||||
return f'{gbps:g} Gbps'
|
||||
return f'{mbps} Mbps'
|
||||
|
||||
|
||||
def device_url(base: str, name: str, port_idx=None) -> str:
|
||||
if not base or not name:
|
||||
return ''
|
||||
url = f'{base}/device/{quote(name, safe="")}'
|
||||
return f'{url}#port-{port_idx}' if port_idx not in (None, '') else url
|
||||
|
||||
|
||||
def history_url(base: str, target: str) -> str:
|
||||
return f'{base}/events?target={quote(target, safe="")}&range=90d' if base and target else ''
|
||||
|
||||
|
||||
def cutsheet_url(base: str, cable_id) -> str:
|
||||
return f'{base}/cutsheet#cable-{cable_id}' if base else ''
|
||||
|
||||
|
||||
def link(text: str, url: str) -> str:
|
||||
return f'[{text}]({url})' if url else text
|
||||
|
||||
|
||||
def port_text(switch: str, port_name: str, port: dict) -> str:
|
||||
"""'8SFP-AGG port 7 "Large1": up at 10 Gbps, DAC'"""
|
||||
idx = port.get('port_idx')
|
||||
text = f'{switch} port {idx}' if idx is not None else f'{switch} {port_name}'
|
||||
if port_name and idx is not None and port_name != f'Port {idx}':
|
||||
text += f' "{port_name}"'
|
||||
state = []
|
||||
if port.get('up'):
|
||||
rate = speed(port.get('speed_mbps'))
|
||||
state.append(f'up at {rate}' if rate else 'up')
|
||||
else:
|
||||
state.append('down')
|
||||
if port.get('poe_power'):
|
||||
state.append(f'PoE {float(port["poe_power"]):.1f} W')
|
||||
sfp = port.get('sfp') or {}
|
||||
if sfp.get('found'):
|
||||
kind = 'DAC' if sfp.get('kind') == 'dac' else sfp.get('kind', '')
|
||||
module = ' '.join(p for p in (kind, sfp.get('part', '')) if p)
|
||||
if module:
|
||||
state.append(f'module: {module}')
|
||||
if sfp.get('rx_fault') or sfp.get('tx_fault'):
|
||||
state.append('**module reports a fault**')
|
||||
return f'{text}: {", ".join(state)}'
|
||||
|
||||
|
||||
def description(heading: str, facts: Sequence[Tuple[str, str]], summary: str,
|
||||
checks: Sequence[str], sections: Sequence[Section] = (), recovers: bool = True) -> str:
|
||||
lines = [f'## {heading}', '']
|
||||
lines += [f'- **{key}:** {value}' for key, value in facts if value not in (None, '')]
|
||||
lines += ['', summary.strip()]
|
||||
if checks:
|
||||
lines += ['', '### What to check']
|
||||
lines += [f'{n}. {check}' for n, check in enumerate(checks, 1)]
|
||||
for title, body in sections:
|
||||
if body:
|
||||
lines += ['', f'### {title}'] + list(body)
|
||||
if recovers:
|
||||
lines += ['', '_Opened automatically by Gandalf. It adds a note here when the problem clears, '
|
||||
'but leaves closing the ticket to you._']
|
||||
else:
|
||||
lines += ['', '_Reported automatically by Gandalf. This already happened, so there will be no '
|
||||
'recovery note._']
|
||||
return '\n'.join(lines)
|
||||
|
||||
|
||||
def history_lines(stats: dict, now: datetime) -> List[str]:
|
||||
week, month = stats.get('last_7d', 0), stats.get('last_30d', 0)
|
||||
if not month:
|
||||
lines = ['- First time in the last 30 days.']
|
||||
else:
|
||||
lines = [f'- Happened {week} time{"s" if week != 1 else ""} in the last 7 days '
|
||||
f'and {month} in the last 30 (not counting this one).']
|
||||
if week >= 3:
|
||||
lines.append('- **This keeps coming back**: look for a cause that comes and goes '
|
||||
'(a loose cable, a failing optic, power, or a scheduled job).')
|
||||
first, resolved = stats.get('last_first_seen'), stats.get('last_resolved_at')
|
||||
if resolved:
|
||||
took = duration((resolved - first).total_seconds()) if first else 'unknown'
|
||||
ago = duration((now - resolved).total_seconds())
|
||||
lines.append(f'- Last cleared {utc(resolved)} ({ago} ago), after {took}.')
|
||||
return lines
|
||||
|
||||
|
||||
ISSUE_FIXED = {
|
||||
'cable_mismatch': 'The cable now matches the cutsheet (or the cutsheet was updated).',
|
||||
'monitor_source_unavailable': 'Prometheus is returning data again; host and interface alerting has resumed.',
|
||||
}
|
||||
|
||||
|
||||
def recovery_comment(event: dict, now: datetime, url: str = '') -> str:
|
||||
"""The note Gandalf adds to a ticket when its alert clears."""
|
||||
first, resolved = event.get('first_seen'), event.get('resolved_at') or now
|
||||
target = event['target_name'] + (f' {event["target_detail"]}' if event.get('target_detail') else '')
|
||||
lines = [f'**✅ Cleared**: {event["event_type"].replace("_", " ")} on {target}.', '']
|
||||
lines.append(f'- **Cleared:** {utc(resolved)}')
|
||||
if first:
|
||||
lines.append(f'- **Lasted:** {duration((resolved - first).total_seconds())} '
|
||||
f'(first seen {utc(first)})')
|
||||
if event.get('consecutive_failures'):
|
||||
lines.append(f'- **Failed checks:** {event["consecutive_failures"]}')
|
||||
if url:
|
||||
lines.append(f'- **Gandalf:** {url}')
|
||||
lines += ['', ISSUE_FIXED.get(event['event_type'], 'Gandalf no longer sees the problem.')]
|
||||
lines += ['', '_Gandalf leaves this ticket open. Close it once you know why it happened, '
|
||||
'or leave it open if it needs follow-up._']
|
||||
return '\n'.join(lines)
|
||||
Reference in new issue
Block a user