Compare commits

...
Author SHA1 Message Date
jaredandClaude Opus 5.5 f655836446 Add a rack drawing (/rack)
Lint / JS (eslint) (push) Successful in 7s
Lint / Python (flake8) (push) Successful in 50s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 50s
Test / Python Tests (pytest) (push) Successful in 1m2s
Lint / Python (flake8) (pull_request) Successful in 52s
Lint / JS (eslint) (pull_request) Successful in 7s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 1m42s
Test / Python Tests (pytest) (pull_request) Successful in 2m2s
The rack front is drawn from the rack units on cutsheet items. Switches
with a faceplate use it, with live port state; patch panels get a generic
panel with port labels; everything else is a labelled plate. Each
documented port shows its cutsheet state, and hovering a port lights up
every port on the same cable run (through patch panels) and names the
cables. Clicking opens the switch port or the cable.

Overlapping items and items past the top of the rack are listed instead
of drawn over another; where two claim the same top unit, the first one
recorded wins. rack.units sets the height (default 42).

/api/rack builds on the cutsheet view; cutsheet.rack_layout and
cutsheet.port_marks hold the placement and per-port logic.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 20:59:30 -05:00
jared ee2708dccb Merge pull request 'Inventory links, cutsheet marks on faceplates, single page titles' (#22) from inventory-and-faceplate-marks into main
Lint / JS (eslint) (push) Successful in 13s
Lint / Python (flake8) (push) Successful in 1m45s
Lint / Notify on failure (push) Skipped
Security / Python Security (bandit) (push) Successful in 1m32s
Lint / Deploy (push) Successful in 4s
Test / Python Tests (pytest) (push) Successful in 46s
2026-11-10 20:50:26 -05:00
jaredandClaude Opus 5.5 c262e5ecc8 Inventory links, cutsheet marks on faceplates, and single page titles
Lint / JS (eslint) (push) Successful in 7s
Lint / Python (flake8) (push) Successful in 54s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 1m1s
Lint / JS (eslint) (pull_request) Successful in 19s
Test / Python Tests (pytest) (push) Successful in 2m6s
Security / Python Security (bandit) (pull_request) Successful in 1m18s
Lint / Python (flake8) (pull_request) Successful in 53s
Test / Python Tests (pytest) (pull_request) Successful in 1m1s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Device pages get an Inventory button: it opens the device's HomeBox record
once an admin pastes its URL (stored in the new device_links table; only
URLs on inventory.base_url are accepted), and searches the inventory for
the model until then. Tickets link the record too.

Faceplate ports with a documented cable get a corner dot: green matches,
red mismatch, grey documented but unconfirmed. The port's tooltip names
the cable, and a legend toggle (remembered per browser) hides them.

Page titles no longer repeat GANDALF ("Alert History - GANDALF - GANDALF").

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 20:47:14 -05:00
jared 29119c2235 Merge pull request 'Alert history page and a tighter monitor health check' (#21) from events-history into main
Lint / JS (eslint) (push) Successful in 9s
Lint / Python (flake8) (push) Successful in 1m19s
Lint / Notify on failure (push) Skipped
Security / Python Security (bandit) (push) Successful in 55s
Lint / Deploy (push) Successful in 4s
Test / Python Tests (pytest) (push) Successful in 48s
2026-11-10 20:37:45 -05:00
jaredandClaude Opus 5.5 9d561db703 Add an alert history page and tie /health staleness to the poll interval
Lint / JS (eslint) (push) Successful in 6s
Lint / Python (flake8) (push) Successful in 1m56s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 2m3s
Test / Python Tests (pytest) (push) Successful in 1m0s
Lint / Python (flake8) (pull_request) Successful in 48s
Lint / JS (eslint) (pull_request) Successful in 6s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 2m7s
Test / Python Tests (pytest) (pull_request) Successful in 2m6s
/events lists every alert from the last 24 hours to 90 days, open or
cleared, with how long each lasted, failed checks and its ticket. Filters
(device, type, severity, state, range) live in the URL, so device pages
and tickets link straight to a device's history. A "Recurring" table
lists alerts that fired more than once on the same target. On phones the
rows become cards.

/api/events/history backs it (100 per page). The existing /api/events is
unchanged.

/health and the dashboard's monitor indicator now call the monitor
stalled after five missed polls (at least 10 minutes) instead of a fixed
20, and /health reports the monitor's age.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 20:29:09 -05:00
jared 9e2a2b987c Merge pull request 'Recovery notes and fuller context on tickets' (#20) from ticket-recovery-and-context into main
Lint / JS (eslint) (push) Successful in 30s
Lint / Python (flake8) (push) Successful in 1m17s
Lint / Notify on failure (push) Skipped
Security / Python Security (bandit) (push) Successful in 58s
Lint / Deploy (push) Successful in 4s
Test / Python Tests (pytest) (push) Successful in 1m18s
2026-11-10 20:09:03 -05:00
jaredandClaude Opus 5.5 437f7a415f Add recovery notes and fuller context to tickets
Lint / JS (eslint) (push) Successful in 7s
Lint / Python (flake8) (push) Successful in 1m26s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 1m30s
Lint / Python (flake8) (pull_request) Successful in 52s
Test / Python Tests (pytest) (push) Successful in 1m4s
Lint / JS (eslint) (pull_request) Successful in 7s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 1m28s
Test / Python Tests (pytest) (pull_request) Successful in 2m0s
When an alert that opened a ticket clears, the monitor now comments on the
ticket (when it cleared, how long it lasted, failed checks, a link back).
It never closes or changes a ticket's status. Failed notes retry each
cycle for 24 hours; a 401/403 from Tinker stops them until restart.

Tickets are now Markdown with a "What to check" list and, where known:
the switch port the device plugs into (kept from the last time it was
seen, so a dead host still shows it), cutsheet cables and their path,
how often the alert fired in the last 7/30 days, other open alerts on the
device, and links into Gandalf (gandalf.public_url). Titles are unchanged
so Tinker's duplicate detection still works.

network_events gains recovery_notified, added at monitor startup; rows
that cleared before the upgrade count as already notified.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 19:44:50 -05:00
24 changed files with 2345 additions and 166 deletions

No files matched your search

+88 -3
View File
@@ -114,6 +114,36 @@ Ping-only (no node_exporter): **pbs** (10.10.10.3)
- Quick-suppress modal: apply timed or manual suppression from any alert row
- Auto-refreshes every 30 seconds via `/api/status` + `/api/network`
### Alert history (`/events`)
Every alert Gandalf raised in the last 24 hours to 90 days, including ones that cleared before anyone
saw them: when each started, how long it lasted, failed checks, and its ticket.
- Filter by device (name, port or interface), alert type, severity, and open or cleared. Filters are
kept in the URL, so `/events?target=large1&range=90d` is a shareable link; device pages and tickets
link to their own history this way.
- **Recurring** lists the same alert on the same target more than once in the range, worst first, since
a flapping cable or optic shows up here before anyone notices the tickets.
- API: `GET /api/events/history?range=24h|7d|30d|90d&type=&target=&severity=&status=all|active|resolved&offset=`
(100 per page).
### Rack (`/rack`)
The rack front, drawn from the rack units set on cutsheet items (`rack.units`, default 42U, top first).
- An item named after a UniFi switch or gateway with a faceplate is drawn with its real faceplate and
live port state; patch panels get a generic panel with each port's label; anything else is a labelled
blank plate (desk switches and devices without a faceplate included).
- Every documented port shows its cutsheet state (green matches, red mismatch, grey unconfirmed).
Hovering a port lights up every port on the same cable run, through patch panels, and names the
cables; clicking opens the switch port or the cable.
- Items that would overlap another or run past the top of the rack are listed, not drawn over. Where two
claim the same top unit, the one recorded first is drawn. Items without a rack unit are listed below.
- Adding the new switch: give it a cutsheet item with its rack unit and it appears; it is drawn as a
plate until its model has a faceplate in `static/faceplates.js`.
### Health check (`/health`)
No login needed. Returns 200 when the database answers and the monitor has completed a check recently,
otherwise 503 with the failing check: `{"status": "degraded", "checks": {"db": "ok", "monitor": "stale",
"monitor_age_s": 742}}`. The monitor counts as stalled after five missed polls (at least 10 minutes),
so point an uptime monitor at it to catch a hung `gandalf-monitor`, not just a dead web app.
### Device pages (`/device/<name>`)
One page per host, switch, router, AP, PDU, or modem. Reach them from the search box on the
dashboard (`/` focuses it), or from any device name. Search matches hostname, IP, MAC, model, type,
@@ -145,6 +175,17 @@ The port panel's **SFP Module** section lists type, part, serial and faults, plu
optical readings (TX/RX power, temperature, voltage, bias) when it reports them. Copper DACs don't.
Both ends of a DAC report the same cable serial, so a DAC seen on two ports links to its other end.
**Cutsheet marks**: a port with a cable documented in the cutsheet gets a dot in its corner: green when
it matches what Gandalf sees, red for a mismatch (wrong neighbor, no link, missing port, port conflict),
grey when it is documented but can't be confirmed (planned, dark, through a panel, or the far end isn't
visible). Hovering the port names the cable. "Cutsheet marks" in the legend hides them; the choice is
remembered in the browser.
**Inventory link**: the device page's 📦 button opens the device's record in the inventory
(`inventory.base_url`). Admins set it with ✎ by pasting the record's URL; only URLs on the inventory
host are accepted. Until one is set, the button searches the inventory for the model. Tickets include
the link too.
**Port block colours:**
| Colour | State |
|---|---|
@@ -255,6 +296,34 @@ can view it; only the `admin` group can edit, and every change is logged with wh
A failed Pulse ping check counts as "unknown": it neither raises nor resolves `host_unreachable`.
### What a ticket contains
Tickets are Markdown. Besides the alert's own facts and a numbered "What to check" list, each one adds
whatever Gandalf knows around the problem:
- **Where it is connected**: the switch port(s) the host or device plugs into, with link state, speed,
PoE and SFP module, linked to that port on the switch's page. A host that has dropped off the network
has lost its LLDP entry, so Gandalf keeps the last port it saw each device on (`known_uplinks` in
`monitor_state`) and shows that, marked "last seen linked there".
- The **cutsheet** cables ending on the device or port, with their full path through patch panels
- **History**: how often the same alert fired in the last 7 and 30 days, and when it last cleared;
3 or more in a week is called out as recurring
- **Other active alerts** on the same device, with their ticket numbers
- **Links** to the device in Gandalf (from `gandalf.public_url`)
Titles keep the `[target][auto][production]...` tag format, which Tinker uses to spot duplicates.
### Recovery notes
When an alert that opened a ticket clears, the monitor adds a comment to that ticket: when it cleared, how
long it lasted, how many checks failed, and a link back to Gandalf. **Gandalf never closes or changes the
status of a ticket**; that is left to a person. The one-shot WAN reports (modem power-cycled, WAN unstable)
get no note, since they describe something already over.
A note that fails to post is retried every cycle for 24 hours. Posting needs a Tinker API key with
`read_write` scope; if Tinker answers 401 or 403, the monitor logs one error and stops trying until it is
restarted.
### WAN modem notes
The alarm and warning limits are read from the gateway on every poll, so they are never hard-coded here.
@@ -310,9 +379,12 @@ Shared by both processes. Located in the working directory (`/var/www/html/prod/
"site_id": "default"
},
"ticket_api": {
"url": "https://t.lotusguild.org/api/tickets",
"url": "https://t.lotusguild.org/create_ticket_api.php",
"api_key": "..."
},
"gandalf": {
"public_url": "https://gandalf.lotusguild.org"
},
"pulse": {
"url": "http://<pulse-host>:<port>",
"api_key": "...",
@@ -360,7 +432,13 @@ Shared by both processes. Located in the working directory (`/var/www/html/prod/
| `unifi.controller` | UniFi controller base URL (HTTPS, self-signed cert ignored) |
| `unifi.api_key` | UniFi API key from controller Settings → API |
| `unifi.site_id` | UniFi site ID (default: `default`) |
| `ticket_api.api_key` | Tinker Tickets bearer token |
| `ticket_api.url` | Tinker Tickets `create_ticket_api.php` URL |
| `ticket_api.api_key` | Tinker Tickets bearer token. Needs `read_write` scope to add recovery notes |
| `ticket_api.comment_url` | Tinker comment endpoint (default: `api/ticket_comment_api.php` beside `ticket_api.url`) |
| `gandalf.public_url` | Gandalf's own address, for links in tickets (default: `https://gandalf.lotusguild.org`) |
| `rack.units` | Height of the rack drawn on `/rack` (default: 42) |
| `inventory.base_url` | Asset inventory (HomeBox) address; device inventory links must be on this host (default: `https://inventory.lotusguild.org`) |
| `inventory.search_url` | Search link used before a device has a record linked, with `{base}` and `{q}` placeholders (default: `{base}/items?q={q}`) |
| `pulse.url` | Pulse worker API base URL (for SSH relay) |
| `pulse.worker_id` | Which Pulse worker runs ethtool collection |
| `pulse.timeout` | Max seconds to wait for SSH collection per host |
@@ -567,6 +645,7 @@ gandalf/
├── modem.py # Read-only AT&T gateway status page client and parser
├── devices.py # Device inventory for search and device pages
├── cutsheet.py # Cutsheet validation, path tracing, plan-vs-actual checks, CSV
├── ticket_text.py # Markdown for tickets and recovery notes
├── db.py # Database operations (MariaDB via pymysql, thread-local conn reuse)
├── schema.sql # Database schema (network_events, suppression_rules, monitor_state)
├── config.json # Runtime configuration (not committed with secrets)
@@ -576,12 +655,16 @@ gandalf/
│ ├── app.js # Dashboard JS (auto-refresh, events, suppress modal, search, modem panel)
│ ├── device.js # Device page rendering
│ ├── cutsheet.js # Cutsheet page
│ ├── events.js # Alert history page
│ ├── rack.js # Rack drawing
│ └── port-inspector.js # Switch front panel, port panel, path check, diagnostics
└── templates/
├── base.html # Shared layout (header, nav, footer)
├── index.html # Dashboard page
├── device.html # Per-device page
├── cutsheet.html # Cutsheet page and its editors
├── events.html # Alert history page
├── rack.html # Rack drawing
├── _modem_panel.html # WAN / Fiber panel, shared by the dashboard and modem page
└── suppressions.html # Suppression management page
```
@@ -617,7 +700,9 @@ Take LED colors from the model's quick start guide. See the comment at the top o
### Database Schema Notes
- `network_events`: one row per active event; `resolved_at` is set when recovered
- `device_links`: each device's inventory URL, by device name; created with the cutsheet tables
- `network_events`: one row per active event; `resolved_at` is set when recovered, and
`recovery_notified` once the recovery note is on its ticket (column added automatically at monitor startup)
- `suppression_rules`: `active=FALSE` when removed; `expires_at` checked at query time
- `monitor_state`: key/value store; `interface_baseline`, `link_stats`, and `modem_stats` are JSON blobs
- `monitor_state` also holds `wan_history` (WAN outages, modem restarts) and `unifi_log_cursor` (how far the UniFi log has been read)
+196 -17
View File
@@ -17,6 +17,8 @@ import time
import uuid
from datetime import datetime, timezone
from functools import wraps
from typing import Optional
from urllib.parse import quote, urlparse
from flask import Flask, jsonify, make_response, redirect, render_template, request, send_file, url_for
@@ -104,15 +106,34 @@ def add_security_headers(response):
return response
def _daemon_ok(last_check: str) -> bool:
"""Return True if monitor last checked within 20 minutes."""
def _monitor_stale_after() -> int:
"""Seconds without a completed check before the monitor counts as stalled.
Five missed polls, and never under 10 minutes: one slow cycle (an SSH timeout
to a dead host) shouldn't trip it, but a hung monitor is caught in minutes.
"""
try:
poll = int(_config().get('monitor', {}).get('poll_interval', 120))
except (TypeError, ValueError):
poll = 120
return max(600, poll * 5)
def _monitor_age(last_check: str) -> Optional[float]:
"""Seconds since the monitor's last check, or None if it never ran or can't be read."""
if not last_check or last_check == 'Never':
return False
return None
try:
ts = datetime.strptime(last_check, '%Y-%m-%d %H:%M:%S UTC').replace(tzinfo=timezone.utc)
return (datetime.now(timezone.utc) - ts).total_seconds() < 1200
except Exception:
return False
except ValueError:
return None
return (datetime.now(timezone.utc) - ts).total_seconds()
def _daemon_ok(last_check: str) -> bool:
"""True if the monitor checked recently enough (see _monitor_stale_after)."""
age = _monitor_age(last_check)
return age is not None and age < _monitor_stale_after()
# ---------------------------------------------------------------------------
@@ -298,7 +319,8 @@ def links_page():
@app.route('/device/<path:name>')
@require_auth
def device_page(name: str):
return render_template('device.html', user=_get_user(), device_name=name)
user = _get_user()
return render_template('device.html', user=user, device_name=name, can_edit='admin' in user['groups'])
@app.route('/inspector')
@@ -308,6 +330,50 @@ def inspector():
return redirect(url_for('index'))
@app.route('/events')
@require_auth
def events_page():
return render_template('events.html', user=_get_user())
_HISTORY_RANGES = {'24h': 24, '7d': 24 * 7, '30d': 24 * 30, '90d': 24 * 90}
_HISTORY_PAGE = 100
@app.route('/api/events/history')
@require_auth
def api_events_history():
"""Every alert open at some point in a time window, with summaries.
Query: range (24h|7d|30d|90d), type, target (substring of name or detail),
severity, status (all|active|resolved), offset.
"""
args = request.args
hours = _HISTORY_RANGES.get(args.get('range', '7d'))
if hours is None:
return jsonify({'error': f'range must be one of {", ".join(_HISTORY_RANGES)}'}), 400
severity = args.get('severity', '')
if severity not in ('', 'critical', 'warning', 'info'):
return jsonify({'error': 'severity must be critical, warning or info'}), 400
status = args.get('status', 'all')
if status not in ('all', 'active', 'resolved'):
return jsonify({'error': 'status must be all, active or resolved'}), 400
try:
offset = max(int(args.get('offset', 0)), 0)
except ValueError:
return jsonify({'error': 'offset must be an integer'}), 400
event_type = args.get('type', '')[:60]
target = args.get('target', '').strip()[:255]
result = db.get_events_history(hours, event_type, target, severity, status,
limit=_HISTORY_PAGE, offset=offset)
result['flappers'] = db.get_flappers(hours, event_type, target, severity)
result['types'] = db.get_event_types()
result['offset'] = offset
result['page_size'] = _HISTORY_PAGE
return jsonify(result)
@app.route('/suppressions')
@require_auth
@require_admin
@@ -461,6 +527,7 @@ def api_device(name: str):
except Exception as e:
logger.error(f'Cutsheet unavailable for {name}: {e}')
detail['cutsheet'] = []
detail['inventory'] = _inventory_for(name, detail.get('model') or '')
return jsonify(detail)
@@ -936,6 +1003,118 @@ def api_cutsheet():
return jsonify(view)
# ---------------------------------------------------------------------------
# Inventory (HomeBox) links
# ---------------------------------------------------------------------------
_INVENTORY_DEFAULT = 'https://inventory.lotusguild.org'
def _inventory_base() -> str:
return ((_config().get('inventory') or {}).get('base_url') or _INVENTORY_DEFAULT).rstrip('/')
def _inventory_for(name: str, model: str) -> dict:
"""The device's inventory record link, or a search to find it."""
base = _inventory_base()
template = (_config().get('inventory') or {}).get('search_url') or '{base}/items?q={q}'
try:
_ensure_cutsheet_schema()
url = db.get_device_link(name)
except Exception as e:
logger.error(f'Inventory link unavailable for {name}: {e}')
url = ''
return {'url': url, 'search_url': template.format(base=base, q=quote(model or name, safe=''))}
def _validate_inventory_url(value) -> str:
"""Only links into the configured inventory site: this URL is shown as a link to every viewer."""
url = str(value or '').strip()
if not url:
return ''
if len(url) > 512:
raise cutsheet.ValidationError('Inventory link is too long')
base = urlparse(_inventory_base())
got = urlparse(url)
if got.scheme != 'https' or got.netloc.lower() != base.netloc.lower() or got.username or got.password:
raise cutsheet.ValidationError(f'Inventory link must start with {base.scheme}://{base.netloc}/')
return url
@app.route('/api/device/<path:name>/inventory', methods=['PUT'])
@require_auth
@require_admin
@_cutsheet_write
def api_device_inventory(name: str):
"""Set or clear (empty url) the device's inventory link."""
if len(name) > 255:
raise cutsheet.ValidationError('Device name is too long')
url = _validate_inventory_url(_json_body().get('url'))
db.set_device_link(name, url, _get_user()['username'])
logger.info(f'{_get_user()["username"]} set inventory link for {name}: {url or "(removed)"}')
return jsonify({'success': True, 'url': url})
# ---------------------------------------------------------------------------
# Rack elevation
# ---------------------------------------------------------------------------
@app.route('/rack')
@require_auth
def rack_page():
user = _get_user()
return render_template('rack.html', user=user, can_edit='admin' in user['groups'])
def _rack_units() -> int:
try:
return max(1, min(int((_config().get('rack') or {}).get('units', 42)), 60))
except (TypeError, ValueError):
return 42
@app.route('/api/rack')
@require_auth
def api_rack():
"""The rack front: placed cutsheet items, live switch ports, and each port's cutsheet state."""
view = _cutsheet_view()
link_stats = _state_json('link_stats')
switches = link_stats.get('unifi_switches') or {}
unifi_info = link_stats.get('unifi_devices') or {}
snapshot = _state_json('network_snapshot')
online = {d['name']: d.get('connected', True) for d in snapshot.get('unifi') or []}
known = set(switches) | set(online) | set(snapshot.get('hosts') or {})
for cable in view['cables']:
cable['run'] = [h['cable_id'] for h in cable['path'] if 'cable_id' in h]
marks = cutsheet.port_marks(view['cables'])
labels: dict = {}
for pl in view['port_labels']:
labels.setdefault(pl['item_name'], {})[str(pl['port'])] = pl['label']
units = _rack_units()
layout = cutsheet.rack_layout(view['items'], units)
out = []
for item in layout['placed']:
name = item['name']
entry = {k: item.get(k) for k in ('id', 'name', 'kind', 'port_count', 'rack_u', 'rack_height',
'rack_top', 'location', 'notes')}
entry['url'] = f'/device/{quote(name, safe="")}' if name in known else f'/cutsheet#item-{item["id"]}'
entry['ports'] = {port: m for (dev, port), m in marks.items() if dev == name}
entry['labels'] = labels.get(name, {})
if name in switches:
sw = switches[name]
entry['switch'] = {'model': sw.get('model') or '', 'ports': sw.get('ports') or {},
'outlets': (unifi_info.get(name) or {}).get('outlets') or [],
'online': online.get(name, True)}
out.append(entry)
return jsonify({
'units': units, 'items': out, 'overlaps': layout['overlaps'], 'outside': layout['outside'],
'unplaced': sorted(i['name'] for i in view['items'] if not i.get('rack_u')),
'updated': link_stats.get('updated'),
})
@app.route('/api/cutsheet/cables', methods=['POST'])
@require_auth
@require_admin
@@ -1090,19 +1269,19 @@ def health():
checks['db'] = 'error'
overall = 'degraded'
# Monitor freshness: fail if last_check is older than 20 minutes
# Monitor freshness: a hung or crashed monitor stops updating last_check
try:
last_check = db.get_state('last_check', '')
if last_check:
ts = datetime.strptime(last_check, '%Y-%m-%d %H:%M:%S UTC').replace(tzinfo=timezone.utc)
age_s = (datetime.now(timezone.utc) - ts).total_seconds()
if age_s > 1200:
checks['monitor'] = 'stale'
overall = 'degraded'
else:
checks['monitor'] = 'ok'
else:
age_s = _monitor_age(last_check)
if not last_check:
checks['monitor'] = 'no data yet'
elif age_s is None or age_s > _monitor_stale_after():
checks['monitor'] = 'stale'
checks['monitor_age_s'] = int(age_s) if age_s is not None else None
overall = 'degraded'
else:
checks['monitor'] = 'ok'
checks['monitor_age_s'] = int(age_s)
except Exception as e:
logger.error(f'Health check monitor error: {e}')
checks['monitor'] = 'error'
+94 -1
View File
@@ -473,6 +473,37 @@ def parse_csv(text: str, items: Dict[str, dict], max_rows: int = 2000) -> Tuple[
# ── Problems, for the monitor ───────────────────────────────────────────────
def path_text(hops: list) -> str:
return ' → '.join(describe_node(h) for h in hops if not isinstance(h, int))
def runs_for(device: str, cables: List[dict], items: Dict[str, dict], switches: dict,
port: Optional[str] = None) -> List[dict]:
"""The live cables ending on a device (or one of its ports), each with its full run.
Returns [{'cable_id', 'label', 'port', 'status', 'verdict', 'detail', 'path'}].
"""
results = None
out = []
for c in sorted(cables, key=lambda c: _natural(c.get('label') or '')):
if not _live(c):
continue
for end in ('a', 'b'):
if not _same(c.get(f'{end}_device') or '', device):
continue
if port is not None and str(c.get(f'{end}_port') or '') != str(port):
continue
if results is None:
results = evaluate(cables, items, switches or {})
r = results[c['id']]
out.append({
'cable_id': c['id'], 'label': c.get('label') or f'#{c["id"]}',
'port': c.get(f'{end}_port') or '', 'status': c['status'],
'verdict': r['verdict'], 'detail': r['detail'], 'path': path_text(r['path']['hops']),
})
return out
def problems(cables: List[dict], items: Dict[str, dict], switches: dict) -> List[dict]:
"""One entry per installed run whose check found a problem.
@@ -508,6 +539,68 @@ def problems(cables: List[dict], items: Dict[str, dict], switches: dict) -> List
'port': switch_end[1] if switch_end else lead['a_port'],
'verdict': r['verdict'],
'detail': r['detail'],
'path': ' → '.join(describe_node(h) for h in hops if not isinstance(h, int)),
'path': path_text(hops),
})
return found
# ── Rack elevation ──────────────────────────────────────────────────────────
def rack_layout(items: List[dict], units: int) -> dict:
"""Place items that have a rack unit, top of the rack first.
An item that would share a unit with one already placed, or runs past the
top of the rack, is reported rather than drawn over another.
Returns {'placed': [item + 'rack_top'], 'overlaps': [{'name', 'units', 'with'}], 'outside': [name]}.
"""
taken: Dict[int, str] = {}
placed, overlaps, outside = [], [], []
racked = [i for i in items if i.get('rack_u')]
# Top first; where two claim the same top unit, the one recorded first is drawn
for item in sorted(racked, key=lambda i: (-(i['rack_u'] + (i.get('rack_height') or 1)), i.get('id') or 0)):
low = item['rack_u']
high = low + (item.get('rack_height') or 1) - 1
span = f'U{low}' if high == low else f'U{low}–{high}'
if high > units:
outside.append({'name': item['name'], 'units': span})
continue
clash = sorted({taken[u] for u in range(low, high + 1) if u in taken})
if clash:
overlaps.append({'name': item['name'], 'units': span, 'with': clash})
continue
for u in range(low, high + 1):
taken[u] = item['name']
placed.append(dict(item, rack_top=high))
return {'placed': placed, 'overlaps': overlaps, 'outside': outside}
_MARK_RANK = {'bad': 3, 'doc': 2, 'ok': 1}
def port_marks(cables: List[dict]) -> Dict[tuple, dict]:
"""Per (device, port): the worst check of the live cables plugged in there.
`cables` are evaluated cables (with 'verdict', 'detail' and 'run', the IDs of
every cable in its run). Returns {(device, port): {'mark', 'text', 'cable_id', 'run'}},
where mark is ok (matches), bad (a problem) or doc (documented, not confirmed).
"""
out: Dict[tuple, dict] = {}
for c in cables:
if not _live(c):
continue
mark = 'bad' if c['verdict'] in PROBLEM_VERDICTS else ('ok' if c['verdict'] == 'match' else 'doc')
label = c.get('label') or f'#{c["id"]}'
for end in ('a', 'b'):
device, port = c.get(f'{end}_device'), c.get(f'{end}_port')
if not device or not port:
continue
entry = out.setdefault((device, str(port)), {'mark': mark, 'cable_id': c['id'], 'run': [], 'by_detail': {}})
if _MARK_RANK[mark] > _MARK_RANK[entry['mark']]:
entry.update(mark=mark, cable_id=c['id'])
entry['by_detail'].setdefault(c['detail'], []).append(label)
entry['run'] = sorted(set(entry['run']) | set(c.get('run') or [c['id']]))
for entry in out.values():
# A panel port's front and rear cables share one check: say it once
entry['text'] = '; '.join(f'cable{"s" if len(labels) > 1 else ""} {", ".join(labels)}: {detail}'
for detail, labels in entry.pop('by_detail').items())
return out
+199
View File
@@ -348,6 +348,175 @@ def cleanup_expired_suppressions() -> int:
return n
def ensure_events_schema() -> None:
"""Add columns newer than the original events table, so deploys need no manual migration."""
with get_conn() as conn:
with conn.cursor() as cur:
cur.execute("SHOW COLUMNS FROM network_events LIKE 'recovery_notified'")
if cur.fetchone():
return
cur.execute(
'ALTER TABLE network_events ADD COLUMN recovery_notified TINYINT(1) NOT NULL DEFAULT 0'
)
# Events that cleared before this existed are history, not owed a comment
cur.execute(
'UPDATE network_events SET recovery_notified=1, last_seen=last_seen WHERE resolved_at IS NOT NULL'
)
def get_pending_recovery_notes(hours: int = 24, exclude_types: tuple = ()) -> list:
"""Resolved events with a ticket whose recovery comment hasn't been posted yet."""
sql = """SELECT id, event_type, target_name, target_detail, ticket_id,
first_seen, resolved_at, consecutive_failures
FROM network_events
WHERE resolved_at IS NOT NULL AND ticket_id IS NOT NULL AND recovery_notified=0
AND resolved_at > DATE_SUB(NOW(), INTERVAL %s HOUR)"""
params: list = [hours]
if exclude_types:
sql += f' AND event_type NOT IN ({", ".join(["%s"] * len(exclude_types))})' # nosec B608 - placeholders only
params.extend(exclude_types)
with get_conn() as conn:
with conn.cursor() as cur:
cur.execute(sql + ' ORDER BY resolved_at', params)
return list(cur.fetchall())
def mark_recovery_notified(event_id: int) -> None:
with get_conn() as conn:
with conn.cursor() as cur:
# last_seen updates itself on any write; keep it as the last failing check
cur.execute(
'UPDATE network_events SET recovery_notified=1, last_seen=last_seen WHERE id=%s',
(event_id,),
)
def event_stats(event_type: str, target_name: str, target_detail: str = '') -> dict:
"""How often one alert has fired before, for a ticket's history section.
Returns {'last_7d', 'last_30d', 'last_first_seen', 'last_resolved_at'} over resolved events.
"""
key = (event_type, target_name, target_detail or '')
with get_conn() as conn:
with conn.cursor() as cur:
cur.execute(
"""SELECT COALESCE(SUM(first_seen > DATE_SUB(NOW(), INTERVAL 7 DAY)), 0) AS last_7d,
COUNT(*) AS last_30d
FROM network_events
WHERE event_type=%s AND target_name=%s AND target_detail=%s
AND resolved_at IS NOT NULL AND first_seen > DATE_SUB(NOW(), INTERVAL 30 DAY)""",
key,
)
counts = cur.fetchone() or {}
cur.execute(
"""SELECT first_seen, resolved_at FROM network_events
WHERE event_type=%s AND target_name=%s AND target_detail=%s
AND resolved_at IS NOT NULL ORDER BY resolved_at DESC LIMIT 1""",
key,
)
last = cur.fetchone() or {}
return {
'last_7d': int(counts.get('last_7d') or 0),
'last_30d': int(counts.get('last_30d') or 0),
'last_first_seen': last.get('first_seen'),
'last_resolved_at': last.get('resolved_at'),
}
def _history_where(since_hours: int, event_type: str = '', target: str = '',
severity: str = '', status: str = 'all') -> tuple:
"""WHERE clause shared by the history list and its summaries.
An event is in range if it was open at any point in the window: it started
in it, or it was still open when the window began.
"""
clauses = ['(first_seen > DATE_SUB(NOW(), INTERVAL %s HOUR) OR '
'resolved_at IS NULL OR resolved_at > DATE_SUB(NOW(), INTERVAL %s HOUR))']
params: list = [since_hours, since_hours]
if event_type:
clauses.append('event_type=%s')
params.append(event_type)
if severity:
clauses.append('severity=%s')
params.append(severity)
if target:
like = '%' + target.replace('\\', '\\\\').replace('%', '\\%').replace('_', '\\_') + '%'
clauses.append('(target_name LIKE %s OR target_detail LIKE %s)')
params.extend([like, like])
if status == 'active':
clauses.append('resolved_at IS NULL')
elif status == 'resolved':
clauses.append('resolved_at IS NOT NULL')
return ' AND '.join(clauses), params
def _iso(rows: list, keys=('first_seen', 'last_seen', 'resolved_at')) -> list:
for r in rows:
for k in keys:
if r.get(k) and hasattr(r[k], 'isoformat'):
r[k] = r[k].isoformat() + 'Z'
return rows
def get_events_history(since_hours: int, event_type: str = '', target: str = '', severity: str = '',
status: str = 'all', limit: int = 100, offset: int = 0) -> dict:
"""Events open at any time in the last `since_hours`, newest first, with totals.
Returns {'events', 'total', 'by_type': {type: count}}.
"""
where, params = _history_where(since_hours, event_type, target, severity, status)
with get_conn() as conn:
with conn.cursor() as cur:
cur.execute(
f"""SELECT id, event_type, severity, source_type, target_name, target_detail, description,
first_seen, last_seen, resolved_at, consecutive_failures, ticket_id,
TIMESTAMPDIFF(SECOND, first_seen, COALESCE(resolved_at, NOW())) AS duration_s
FROM network_events WHERE {where}
ORDER BY first_seen DESC, id DESC LIMIT %s OFFSET %s""", # nosec B608 - fixed clauses
params + [limit, offset],
)
events = _iso(list(cur.fetchall()))
cur.execute(
f"""SELECT event_type, COUNT(*) AS n
FROM network_events WHERE {where} GROUP BY event_type ORDER BY n DESC""", # nosec B608
params,
)
groups = cur.fetchall()
return {
'events': events,
'total': sum(int(g['n']) for g in groups),
'by_type': {g['event_type']: int(g['n']) for g in groups},
}
def get_flappers(since_hours: int, event_type: str = '', target: str = '', severity: str = '',
min_count: int = 2, limit: int = 10) -> list:
"""The alerts that fired most often in the window: the ones worth a root cause."""
where, params = _history_where(since_hours, event_type, target, severity)
with get_conn() as conn:
with conn.cursor() as cur:
cur.execute(
f"""SELECT event_type, target_name, target_detail, COUNT(*) AS n,
SUM(TIMESTAMPDIFF(SECOND, first_seen, COALESCE(resolved_at, NOW()))) AS total_s,
MAX(first_seen) AS last_seen, SUM(resolved_at IS NULL) AS open_now
FROM network_events WHERE {where}
GROUP BY event_type, target_name, target_detail
HAVING n >= %s ORDER BY n DESC, last_seen DESC LIMIT %s""", # nosec B608 - fixed clauses
params + [min_count, limit],
)
rows = list(cur.fetchall())
for r in rows:
r['n'], r['total_s'], r['open_now'] = int(r['n']), int(r['total_s'] or 0), int(r['open_now'] or 0)
return _iso(rows, ('last_seen',))
def get_event_types() -> list:
with get_conn() as conn:
with conn.cursor() as cur:
cur.execute('SELECT DISTINCT event_type FROM network_events ORDER BY event_type')
return [r['event_type'] for r in cur.fetchall()]
def purge_old_resolved_events(days: int = 90) -> int:
"""Delete resolved events older than `days` days. Returns count deleted."""
with get_conn() as conn:
@@ -553,8 +722,38 @@ _CUTSHEET_SCHEMA = (
INDEX idx_hist_time (changed_at),
INDEX idx_hist_entity (entity, entity_id)
) ENGINE=InnoDB""",
# Where a device lives in the asset inventory (HomeBox), set by hand on its Gandalf page
"""CREATE TABLE IF NOT EXISTS device_links (
name VARCHAR(255) NOT NULL PRIMARY KEY,
inventory_url VARCHAR(512) NOT NULL,
updated_by VARCHAR(255) NOT NULL,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
) ENGINE=InnoDB""",
)
def get_device_link(name: str) -> str:
"""The device's inventory URL, or ''."""
with get_conn() as conn:
with conn.cursor() as cur:
cur.execute('SELECT inventory_url FROM device_links WHERE name=%s', (name,))
row = cur.fetchone()
return row['inventory_url'] if row else ''
def set_device_link(name: str, inventory_url: str, user: str) -> None:
"""Save a device's inventory URL; an empty URL removes it."""
with get_conn() as conn:
with conn.cursor() as cur:
if not inventory_url:
cur.execute('DELETE FROM device_links WHERE name=%s', (name,))
return
cur.execute(
"""INSERT INTO device_links (name, inventory_url, updated_by) VALUES (%s, %s, %s)
ON DUPLICATE KEY UPDATE inventory_url=VALUES(inventory_url), updated_by=VALUES(updated_by)""",
(name, inventory_url, user),
)
_ITEM_FIELDS = ('name', 'kind', 'port_count', 'location', 'rack_u', 'rack_height', 'notes')
_CABLE_FIELDS = ('label', 'a_device', 'a_port', 'a_side', 'b_device', 'b_port', 'b_side',
'cable_type', 'length_m', 'color', 'status', 'label_applied', 'notes')
+353 -128
View File
@@ -15,7 +15,7 @@ import statistics
import time
from datetime import datetime, timezone
from typing import Dict, List, Optional
from urllib.parse import urlparse
from urllib.parse import urljoin, urlparse
import requests
from urllib3.exceptions import InsecureRequestWarning
@@ -24,6 +24,7 @@ import db
import cutsheet
import devices
import modem as modem_api
import ticket_text
logging.basicConfig(
@@ -388,6 +389,41 @@ class TicketClient:
def __init__(self, cfg: dict):
self.url = cfg.get('url', '')
self.api_key = cfg.get('api_key', '')
# Tinker serves its other API scripts under api/ beside create_ticket_api.php
self.comment_url = cfg.get('comment_url') or (
urljoin(self.url, 'api/ticket_comment_api.php') if self.url else '')
self.comments_denied = False
def comment(self, ticket_id: str, text: str) -> bool:
"""Add a Markdown comment to a ticket. Never raises; False means try again later."""
if not self.api_key or not self.comment_url or self.comments_denied:
return False
try:
resp = requests.post(
self.comment_url,
json={'ticket_id': str(ticket_id), 'comment_text': text, 'markdown_enabled': True},
headers={'Authorization': f'Bearer {self.api_key}'},
timeout=15,
)
if resp.status_code in (401, 403):
# Retrying every cycle can't fix a key without write access, so stop until restart
self.comments_denied = True
logger.error(
f'Ticket API refused a comment on #{ticket_id} (HTTP {resp.status_code}). '
f'The API key needs read_write scope; recovery notes are off until the monitor restarts.'
)
return False
if resp.status_code == 404:
logger.warning(f'Ticket #{ticket_id} no longer exists; skipping its recovery note')
return True
resp.raise_for_status()
if resp.json().get('success'):
logger.info(f'Commented on ticket #{ticket_id}')
return True
logger.warning(f'Unexpected comment API response for #{ticket_id}: {resp.text[:200]}')
except Exception as e:
logger.error(f'Ticket comment failed for #{ticket_id}: {e}')
return False
def create(self, title: str, description: str, priority: str = '2') -> Optional[str]:
if not self.api_key or not self.url:
@@ -893,10 +929,21 @@ def _now_utc() -> str:
return datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M:%S UTC')
def _utcnow() -> datetime:
"""Naive UTC, to compare with database times."""
return datetime.now(timezone.utc).replace(tzinfo=None)
# --------------------------------------------------------------------------
# Monitor
# --------------------------------------------------------------------------
CLUSTER_NAME = 'proxmox-cluster'
DEFAULT_PUBLIC_URL = 'https://gandalf.lotusguild.org'
# Reports of something already over: the ticket is the whole story, no recovery note
ONE_SHOT_EVENTS = ('modem_power_cycled', 'wan_unstable')
# Ports a host or device was last seen on, kept so a ticket for a dead host
# (whose link, and so its LLDP entry, is gone) can still say where it plugs in
KNOWN_UPLINKS_STATE = 'known_uplinks'
class NetworkMonitor:
@@ -950,6 +997,16 @@ class NetworkMonitor:
db.ensure_cutsheet_schema()
except Exception as e:
logger.error(f'Could not create cutsheet tables: {e}')
try:
db.ensure_events_schema()
except Exception as e:
logger.error(f'Could not update the events table: {e}')
# Absolute links in tickets point here
self.public_url = ((self.cfg.get('gandalf') or {}).get('public_url')
or DEFAULT_PUBLIC_URL).rstrip('/')
self._link_data: Optional[dict] = None
self._known_uplinks: Optional[dict] = None
# Last good Prometheus result, shown (as unknown) during a Prometheus outage
self._last_iface_states: Dict[str, Dict[str, bool]] = {}
@@ -964,6 +1021,174 @@ class NetworkMonitor:
def _hostname(self, instance: str) -> str:
return self._instance_map.get(instance, instance.split(':')[0])
# ------------------------------------------------------------------
# Tickets: context, and notes when an alert clears
# ------------------------------------------------------------------
def _base_url(self) -> str:
return getattr(self, 'public_url', DEFAULT_PUBLIC_URL)
def _open_ticket(
self, event_id: int, event_type: str, target: str, detail: str, *,
title: str, heading: str, facts: list, summary: str, checks: list, priority: str,
host: Optional[str] = None, port: Optional[tuple] = None,
) -> None:
"""Open a ticket with the alert's facts plus everything Gandalf knows around it.
host: a device whose switch ports to list. port: (switch, port_idx) the alert is about.
"""
sections = self._ticket_sections(event_id, event_type, target, detail, host=host, port=port)
desc = ticket_text.description(heading, facts, summary, checks, sections)
tid = self.tickets.create(title, desc, priority=priority)
if tid:
db.set_ticket_id(event_id, tid)
def _ticket_sections(self, event_id: int, event_type: str, target: str, detail: str,
host: Optional[str] = None, port: Optional[tuple] = None) -> list:
"""Each section is optional: a failure loses that section, never the ticket."""
sections = []
for title, build in (
('Where it is connected', lambda: self._where_lines(event_type, detail, host, port)),
('History', lambda: ticket_text.history_lines(db.event_stats(event_type, target, detail), _utcnow())),
('Other active alerts here', lambda: self._other_alert_lines(event_id, {target, host})),
('Links', lambda: self._link_lines(host or (port[0] if port else None))),
):
try:
sections.append((title, build()))
except Exception as e:
logger.error(f'Ticket section "{title}" for {target} failed: {e}', exc_info=True)
return sections
def _switches(self) -> dict:
return (getattr(self, '_link_data', None) or {}).get('unifi_switches') or {}
def _remember_uplinks(self, link_data: dict) -> None:
"""Record where every LLDP-visible device is plugged in, for when it disappears."""
known = self._load_known_uplinks()
switches = link_data.get('unifi_switches') or {}
names = {(p.get('lldp') or {}).get('system_name')
for sw in switches.values() for p in (sw.get('ports') or {}).values()}
now = _now_utc()
changed = False
for name in sorted(n for n in names if n):
ups = devices.uplinks_for_host(name, switches)
entry = {'seen': now, 'uplinks': [
{k: u[k] for k in ('switch', 'port', 'port_idx', 'host_interface')} for u in ups]}
if (known.get(name) or {}).get('uplinks') != entry['uplinks']:
changed = True
known[name] = entry
self._known_uplinks = known
# Write when wiring changes, and hourly so "last seen" stays roughly right
last_saved = getattr(self, '_known_uplinks_saved', 0.0)
if changed or time.time() - last_saved > 3600:
db.set_state(KNOWN_UPLINKS_STATE, known)
self._known_uplinks_saved = time.time()
def _load_known_uplinks(self) -> dict:
if getattr(self, '_known_uplinks', None) is None:
self._known_uplinks = {}
raw = db.get_state(KNOWN_UPLINKS_STATE)
if raw:
try:
self._known_uplinks = json.loads(raw)
except ValueError:
logger.error('Failed to parse known_uplinks JSON; starting over')
return self._known_uplinks
def _uplinks_for(self, name: str, switches: dict) -> tuple:
"""(uplinks, None) if the device is linked now, else (last known uplinks, when seen)."""
current = devices.uplinks_for_host(name, switches)
if current:
return current, None
entry = self._load_known_uplinks().get(name) or {}
return entry.get('uplinks') or [], entry.get('seen')
def _where_lines(self, event_type: str, detail: str, host: Optional[str], port: Optional[tuple]) -> list:
base, switches = self._base_url(), self._switches()
lines = []
if host:
ups, seen = self._uplinks_for(host, switches)
if event_type == 'interface_down':
ups = [u for u in ups if u.get('host_interface') == detail] or ups
for u in ups:
live = (switches.get(u['switch'], {}).get('ports') or {}).get(u['port'])
text = ticket_text.port_text(u['switch'], u['port'], live or dict(u, up=False))
iface = f'`{u["host_interface"]}` → ' if u.get('host_interface') else ''
lines.append(f'- {iface}' + ticket_text.link(
text, ticket_text.device_url(base, u['switch'], u.get('port_idx'))))
if seen:
lines[-1] += f' _(last seen linked there {seen})_'
if not ups:
lines.append('- Gandalf has not seen this device on a switch port (no LLDP or client entry).')
if port:
switch, idx = port
live = next((dict(p, name=n) for n, p in (switches.get(switch, {}).get('ports') or {}).items()
if str(p.get('port_idx')) == str(idx)), None)
if live:
lines.append('- ' + ticket_text.link(ticket_text.port_text(switch, live['name'], live),
ticket_text.device_url(base, switch, idx)))
device = host or (port[0] if port else None)
if device:
data = db.cutsheet_load()
items = {i['name']: i for i in data['items']}
runs = cutsheet.runs_for(device, data['cables'], items, switches,
port=str(port[1]) if port else None)
for r in runs:
label = ticket_text.link(f'**{r["label"]}**', ticket_text.cutsheet_url(base, r['cable_id']))
lines.append(f'- Cable {label}: {r["path"]}. Cutsheet check: '
f'{cutsheet.VERDICTS.get(r["verdict"], r["verdict"]).lower()}.')
if not runs:
lines.append('- ' + ticket_text.link('Not on the cutsheet yet', f'{base}/cutsheet' if base else ''))
return lines
def _other_alert_lines(self, event_id: int, names: set) -> list:
lines = []
for name in sorted(n for n in names if n):
for e in db.get_events_for_target(name, limit=25):
if e.get('resolved_at') or e.get('id') == event_id:
continue
what = e['event_type'].replace('_', ' ')
where = f'{e["target_name"]} {e.get("target_detail") or ""}'.strip()
ticket = f' (ticket #{e["ticket_id"]})' if e.get('ticket_id') else ''
lines.append(f'- {what} on {where} since {e.get("first_seen", "?")}{ticket}')
return lines[:10]
def _link_lines(self, device: Optional[str]) -> list:
base = self._base_url()
if not base:
return []
lines = []
if device:
lines.append(f'- {ticket_text.link(f"{device} in Gandalf", ticket_text.device_url(base, device))}')
lines.append(f'- {ticket_text.link(f"Alert history for {device}", ticket_text.history_url(base, device))}')
inventory = db.get_device_link(device)
if inventory:
lines.append(f'- {ticket_text.link(f"{device} in the inventory", inventory)}')
lines.append(f'- {ticket_text.link("Gandalf dashboard", base + "/")}')
return lines
def _event_url(self, event: dict) -> str:
base = self._base_url()
if not base:
return ''
if event['event_type'] == 'cable_mismatch':
cable_id = str(event.get('target_detail') or '').replace('cable ', '')
return ticket_text.cutsheet_url(base, cable_id)
if event['event_type'] in ('cluster_network_issue', 'monitor_source_unavailable'):
return base + '/'
return ticket_text.device_url(base, event['target_name'])
def _post_recovery_notes(self) -> None:
"""Comment on the tickets of alerts that have cleared. Failures retry next cycle (for 24 h)."""
if self.tickets.comments_denied:
return
now = _utcnow()
for event in db.get_pending_recovery_notes(hours=24, exclude_types=ONE_SHOT_EVENTS):
text = ticket_text.recovery_comment(event, now, self._event_url(event))
if self.tickets.comment(event['ticket_id'], text):
db.mark_recovery_notified(event['id'])
elif self.tickets.comments_denied:
break
# ------------------------------------------------------------------
# Interface monitoring (Prometheus)
# ------------------------------------------------------------------
@@ -1041,47 +1266,54 @@ class NetworkMonitor:
f'{", ".join(hosts_with_regression)}',
)
if not sup and (is_new or not ticket_id):
title = (
f'[{self.cluster_name}][auto][production][issue][network][cluster-wide] '
f'Multiple hosts reporting interface failures'
)
desc = (
f'Cluster Network Alert\n{"=" * 40}\n\n'
f'Affected hosts: {", ".join(hosts_with_regression)}\n'
f'Detected: {_now_utc()}\n\n'
f'{len(hosts_with_regression)} Proxmox hosts simultaneously reported '
f'interface regressions (link-down on interfaces previously known UP) '
f'or stopped reporting metrics.\n'
f'This likely indicates a switch or upstream network failure.\n\n'
f'Please check the core and management switches immediately.'
)
tid = self.tickets.create(title, desc, priority='1')
if tid:
db.set_ticket_id(event_id, tid)
self._ticket_cluster(event_id, hosts_with_regression)
else:
db.resolve_event('cluster_network_issue', self.cluster_name, '')
def _ticket_cluster(self, event_id: int, hosts: List[str]) -> None:
switches = self._switches()
# The switches these hosts hang off are the likeliest shared cause
shared = sorted({u['switch'] for h in hosts for u in self._uplinks_for(h, switches)[0]})
self._open_ticket(
event_id, 'cluster_network_issue', self.cluster_name, '',
title=(f'[{self.cluster_name}][auto][production][issue][network][cluster-wide] '
f'Multiple hosts reporting interface failures'),
heading='Cluster Network Alert',
facts=[('Affected hosts', ', '.join(hosts)),
('Switches they connect through', ', '.join(shared)),
('Detected', _now_utc())],
summary=(f'{len(hosts)} Proxmox hosts reported interface regressions (link-down on '
f'interfaces previously known up) or stopped reporting metrics at the same time. '
f'This usually means a switch or upstream network failure rather than the hosts.'),
checks=['Check the core and management switches (power, uplinks, recent firmware updates).',
'Check whether the hosts share a PDU or circuit that lost power.',
'Look at the per-host tickets and the Gandalf dashboard for which links went down first.'],
priority='1',
)
def _ticket_metrics_missing(
self, event_id: int, host: str, instance: str, consec: int
) -> None:
title = (
f'[{host}][auto][production][issue][network][single-node] '
f'No metrics from host'
self._open_ticket(
event_id, 'host_metrics_missing', host, instance,
title=f'[{host}][auto][production][issue][network][single-node] No metrics from host',
heading='Host Metrics Alert',
facts=[('Host', host), ('Prometheus instance', instance), ('Detected', _now_utc()),
('Consecutive check failures', str(consec)),
('Down for at least', ticket_text.duration(consec * self._poll_seconds()))],
summary=(f'Prometheus is working but has no node_network_up series for {host}. '
f'Either the host is down or node_exporter has stopped on it. '
f'Gandalf waits {self.host_fail_thresh} checks before opening this, so a normal '
f'reboot should not cause it.'),
checks=[f'Is {host} powered on and reachable (Proxmox UI, ping, console)?',
'If it is up, check `systemctl status prometheus-node-exporter` on the host.',
'If it is down, check its switch port below: a dark port with no PoE or link '
'points to power or cabling.'],
priority='2', host=host,
)
desc = (
f'Host Metrics Alert\n{"=" * 40}\n\n'
f'Host: {host}\n'
f'Prometheus instance: {instance}\n'
f'Detected: {_now_utc()}\n'
f'Consecutive check failures: {consec}\n\n'
f'Prometheus is working but has no node_network_up series for {host}.\n'
f'Either the host is down or node_exporter has stopped on it.\n\n'
f'Please check the host power and network connectivity, then '
f'prometheus-node-exporter on the host.'
)
tid = self.tickets.create(title, desc, priority='2')
if tid:
db.set_ticket_id(event_id, tid)
def _poll_seconds(self) -> int:
return int(getattr(self, 'poll_interval', 120) or 120)
def _process_prometheus_outage(self, suppressions: list) -> None:
"""Interface and host alerts can't fire without Prometheus, so say so."""
@@ -1092,44 +1324,38 @@ class NetworkMonitor:
f'Prometheus returned no interface data; host and interface alerting is paused ({_now_utc()})',
)
if not sup and consec >= self.fail_thresh and not ticket_id:
title = (
'[prometheus][auto][production][issue][network][cluster-wide] '
'Gandalf has no Prometheus data'
self._open_ticket(
event_id, 'monitor_source_unavailable', 'prometheus', '',
title='[prometheus][auto][production][issue][network][cluster-wide] Gandalf has no Prometheus data',
heading='Monitoring Source Alert',
facts=[('Prometheus URL', self.prom.url), ('Detected', _now_utc()),
('Consecutive check failures', str(consec))],
summary=('Gandalf could not get node_network_up data from Prometheus. While this lasts, '
'host and interface failures will **not** be detected. UniFi, ping and modem '
'checks keep running.'),
checks=['Is the Prometheus service running on its host?',
'Open Prometheus → Status → Targets and check the node exporters are being scraped.',
'Check disk space on the Prometheus host (a full disk stops ingestion).'],
priority='2',
)
desc = (
f'Monitoring Source Alert\n{"=" * 40}\n\n'
f'Prometheus URL: {self.prom.url}\n'
f'Detected: {_now_utc()}\n'
f'Consecutive check failures: {consec}\n\n'
f'Gandalf could not get node_network_up data from Prometheus.\n'
f'While this lasts, host and interface failures will NOT be detected.\n\n'
f'Please check the Prometheus service and its scrape targets.'
)
tid = self.tickets.create(title, desc, priority='2')
if tid:
db.set_ticket_id(event_id, tid)
def _ticket_interface(
self, event_id: int, host: str, iface: str, consec: int
) -> None:
title = (
f'[{host}][auto][production][issue][network][single-node] '
f'Interface {iface} link-down'
self._open_ticket(
event_id, 'interface_down', host, iface,
title=f'[{host}][auto][production][issue][network][single-node] Interface {iface} link-down',
heading='Network Interface Alert',
facts=[('Host', host), ('Interface', f'`{iface}`'), ('Detected', _now_utc()),
('Consecutive check failures', str(consec))],
summary=(f'Interface {iface} on {host} was up and is now reporting link-down via '
f'Prometheus node_exporter. {host} itself is still reporting, so it may be '
f'reachable over its other interfaces.'),
checks=['Check the switch port below: is it down on the switch side too?',
'Reseat or swap the cable or DAC; for an optic, check the module for a fault.',
f'On {host}, check `ip link show {iface}` and `dmesg` for NIC or driver errors.'],
priority='2', host=host,
)
desc = (
f'Network Interface Alert\n{"=" * 40}\n\n'
f'Host: {host}\n'
f'Interface: {iface}\n'
f'Detected: {_now_utc()}\n'
f'Consecutive check failures: {consec}\n\n'
f'Interface {iface} on {host} is reporting link-down state via '
f'Prometheus node_exporter.\n\n'
f'Note: {host} may still be reachable via its other network interface.\n'
f'Please inspect the cable/SFP/switch port for {host}/{iface}.'
)
tid = self.tickets.create(title, desc, priority='2')
if tid:
db.set_ticket_id(event_id, tid)
# ------------------------------------------------------------------
# UniFi device monitoring
@@ -1155,23 +1381,20 @@ class NetworkMonitor:
def _ticket_unifi(self, event_id: int, device: dict) -> None:
name = device['name']
title = (
f'[{name}][auto][production][issue][network][single-node] '
f'UniFi device offline'
self._open_ticket(
event_id, 'unifi_device_offline', name, device.get('type', ''),
title=f'[{name}][auto][production][issue][network][single-node] UniFi device offline',
heading='UniFi Device Alert',
facts=[('Device', name), ('Type', device.get('type', 'unknown')), ('Model', device.get('model', '')),
('Last known IP', device.get('ip', 'unknown')), ('Detected', _now_utc())],
summary=(f'The UniFi controller reports {name} as offline. Gandalf waits '
f'{self.unifi_fail_thresh} checks first, so a firmware-update reboot should not cause this.'),
checks=['Check the device for power (status LED) and, if it is PoE powered, the PoE '
'budget on the port below.',
'Check the uplink cable and the switch port it plugs into.',
'In the UniFi controller, check whether the device is stuck adopting or upgrading.'],
priority='2', host=name,
)
desc = (
f'UniFi Device Alert\n{"=" * 40}\n\n'
f'Device: {name}\n'
f'Type: {device.get("type","unknown")}\n'
f'Model: {device.get("model","")}\n'
f'Last Known IP: {device.get("ip","unknown")}\n'
f'Detected: {_now_utc()}\n\n'
f'The UniFi device {name} is offline per the UniFi controller.\n'
f'Please check power and cable connectivity.'
)
tid = self.tickets.create(title, desc, priority='2')
if tid:
db.set_ticket_id(event_id, tid)
# ------------------------------------------------------------------
# Cutsheet: documented cabling vs. what discovery sees
@@ -1206,24 +1429,20 @@ class NetworkMonitor:
db.resolve_event('cable_mismatch', target_name, target_detail)
def _ticket_cable_mismatch(self, event_id: int, p: dict, hours: float) -> None:
title = (
f'[{p["switch"]}][auto][production][issue][network][single-node] '
f'Cable {p["label"]} does not match the cutsheet'
url = ticket_text.cutsheet_url(self._base_url(), p['cable_id'])
self._open_ticket(
event_id, 'cable_mismatch', p['switch'], p['key'],
title=(f'[{p["switch"]}][auto][production][issue][network][single-node] '
f'Cable {p["label"]} does not match the cutsheet'),
heading='Cutsheet Mismatch',
facts=[('Cable', ticket_text.link(p['label'], url)), ('Switch port', f'{p["switch"]} port {p["port"]}'),
('Problem', p['detail']), ('Documented path', p['path']), ('Detected', _now_utc())],
summary=f'This has not matched the cutsheet for at least {hours:g} hours.',
checks=['If the cable was moved on purpose, update its record in the cutsheet.',
'Otherwise, follow the label and check it is plugged into the documented port '
'(Trace mode on the cutsheet page can find which port a cord is in).'],
priority='4', port=(p['switch'], p['port']),
)
desc = (
f'Cutsheet Mismatch\n{"=" * 40}\n\n'
f'Cable: {p["label"]}\n'
f'Switch port: {p["switch"]} port {p["port"]}\n'
f'Problem: {p["detail"]}\n'
f'Documented path: {p["path"]}\n'
f'Detected: {_now_utc()}\n\n'
f'This has not matched for at least {hours:g} hours.\n\n'
f'Either the cable was moved (update the cutsheet in Gandalf: /cutsheet#cable-{p["cable_id"]}) '
f'or it is plugged into the wrong place.'
)
tid = self.tickets.create(title, desc, priority='4')
if tid:
db.set_ticket_id(event_id, tid)
# ------------------------------------------------------------------
# Ping-only hosts (no node_exporter)
@@ -1253,24 +1472,20 @@ class NetworkMonitor:
def _ticket_unreachable(
self, event_id: int, name: str, ip: str, consec: int
) -> None:
title = (
f'[{name}][auto][production][issue][network][single-node] '
f'Host unreachable'
self._open_ticket(
event_id, 'host_unreachable', name, ip,
title=f'[{name}][auto][production][issue][network][single-node] Host unreachable',
heading='Host Reachability Alert',
facts=[('Host', name), ('IP', ip), ('Detected', _now_utc()),
('Consecutive check failures', str(consec)),
('Down for at least', ticket_text.duration(consec * self._poll_seconds()))],
summary=(f'{name} ({ip}) is not answering ping from the Pulse worker Gandalf uses. '
f'It has no node_exporter, so ping is the only check Gandalf has for it.'),
checks=[f'Is {name} powered on? Check its console or management interface.',
'Check its switch port below for link and PoE.',
'If the port is up, check the IP and firewall on the device itself.'],
priority='2', host=name,
)
desc = (
f'Host Reachability Alert\n{"=" * 40}\n\n'
f'Host: {name}\n'
f'IP: {ip}\n'
f'Detected: {_now_utc()}\n'
f'Consecutive check failures: {consec}\n\n'
f'Host {name} ({ip}) is not responding to ping from the Gandalf monitor.\n'
f'This host does not have a Prometheus node_exporter, so interface-level '
f'detail is unavailable.\n\n'
f'Please check the host power, management interface, and network connectivity.'
)
tid = self.tickets.create(title, desc, priority='2')
if tid:
db.set_ticket_id(event_id, tid)
# ------------------------------------------------------------------
# WAN modem monitoring
@@ -1291,17 +1506,14 @@ class NetworkMonitor:
event_type, severity, 'modem', name, detail, f'{description} ({_now_utc()})',
)
if not sup and consec >= self.modem_fail_thresh and not ticket_id:
title = f'[{name}][auto][production][issue][network][cluster-wide] {heading}'
desc = (
f'WAN Modem Alert\n{"=" * 40}\n\n'
f'Modem: {name} ({self.modem.base_url})\n'
f'Detected: {_now_utc()}\n'
f'Consecutive check failures: {consec}\n\n'
f'{description}\n\n{advice}'
self._open_ticket(
event_id, event_type, name, detail,
title=f'[{name}][auto][production][issue][network][cluster-wide] {heading}',
heading='WAN Modem Alert',
facts=[('Modem', f'{name} ({self.modem.base_url})'), ('Detected', _now_utc()),
('Consecutive check failures', str(consec))],
summary=description, checks=[advice], priority=priority,
)
tid = self.tickets.create(title, desc, priority=priority)
if tid:
db.set_ticket_id(event_id, tid)
def _process_modem_failure(self, suppressions: list, error: Exception) -> None:
"""Handle a failed poll. Link and optical events stay as they are: we can't see them."""
@@ -1443,7 +1655,8 @@ class NetworkMonitor:
if not db.check_suppressed(suppressions, 'host', name) and not ticket_id:
tid = self.tickets.create(
f'[{name}][auto][production][issue][network][cluster-wide] {heading}',
f'WAN Event\n{"=" * 40}\n\nModem: {name}\nReported: {_now_utc()}\n\n{description}\n\n{advice}',
ticket_text.description('WAN Event', [('Modem', name), ('Reported', _now_utc())],
description, [advice], recovers=False),
priority=priority,
)
if tid:
@@ -1645,6 +1858,12 @@ class NetworkMonitor:
db.set_state('link_stats', link_data)
except Exception as e:
logger.error(f'Link stats collection failed: {e}', exc_info=True)
if link_data is not None:
self._link_data = link_data
try:
self._remember_uplinks(link_data)
except Exception as e:
logger.error(f'Recording known uplinks failed: {e}', exc_info=True)
# 5. Process alerts using already-fetched data
suppressions = db.get_active_suppressions()
@@ -1673,6 +1892,12 @@ class NetworkMonitor:
except Exception as e:
logger.error(f'Modem collection failed: {e}', exc_info=True)
# Note on the tickets of anything that cleared this cycle (or failed to post before)
try:
self._post_recovery_notes()
except Exception as e:
logger.error(f'Posting recovery notes failed: {e}', exc_info=True)
# Housekeeping: deactivate expired suppressions and purge old resolved events
db.cleanup_expired_suppressions()
db.purge_old_resolved_events(days=90)
+2 -1
View File
@@ -21,8 +21,9 @@ CREATE TABLE IF NOT EXISTS network_events (
resolved_at TIMESTAMP NULL,
consecutive_failures INT NOT NULL DEFAULT 1,
ticket_id VARCHAR(20) NULL,
recovery_notified TINYINT(1) NOT NULL DEFAULT 0, -- recovery comment posted on ticket_id
INDEX idx_active (resolved_at),
INDEX idx_active (resolved_at),
INDEX idx_target (target_name, target_detail),
INDEX idx_type (event_type),
-- Compound index for hot upsert_event lookup: event_type+target+resolved_at
+45 -3
View File
@@ -1,5 +1,5 @@
'use strict';
/* global DEVICE_NAME, deviceUrl, refreshModem, _toIso, fmtRate, fmtErrors, renderPortInspector, selectPortByIdx, highlightSelectedPort, FACEPLATES */
/* global DEVICE_NAME, CAN_EDIT, deviceUrl, refreshModem, _toIso, fmtRate, fmtErrors, renderPortInspector, selectPortByIdx, highlightSelectedPort, FACEPLATES */
// Per-device page: overview, alerts, and a compact interface/port table. Host
// rows expand in place; switch and router rows open the port inspector panel.
@@ -300,7 +300,8 @@ function renderEvents(events) {
<td class="ts-cell" title="${esc(e.first_seen || '')}">${esc(lt.time.ago(_toIso(e.first_seen)))}</td>
<td>${e.resolved_at ? `resolved ${esc(lt.time.ago(_toIso(e.resolved_at)))}` : '<strong>active</strong>'}</td>
</tr>`).join('')}</tbody></table></div>`;
return section('Alerts', body);
const more = `<p class="g-section-more"><a href="/events?target=${encodeURIComponent(DEVICE_NAME)}&amp;range=90d">Full alert history for ${esc(DEVICE_NAME)} →</a></p>`;
return section('Alerts', body + more);
}
function currentRows() {
@@ -331,6 +332,47 @@ function updateInterfaceSection(rows, title, hasChassis) {
document.getElementById('device-error-count').textContent = rows.filter(rowHasProblem).length;
}
// ── Inventory (HomeBox) link ────────────────────────────────────────────
function inventoryActions(d) {
const inv = d.inventory || {};
const edit = CAN_EDIT
? `<button type="button" class="lt-btn lt-btn-ghost lt-btn-sm" data-action="edit-inventory"
aria-label="${inv.url ? 'Change' : 'Set'} the inventory link for ${esc(d.name)}">✎</button>` : '';
if (inv.url) {
return `<a class="lt-btn lt-btn-secondary lt-btn-sm" href="${esc(inv.url)}" target="_blank" rel="noopener">📦 Inventory ↗</a>${edit}`;
}
if (!inv.search_url) return edit;
return `<a class="lt-btn lt-btn-ghost lt-btn-sm" href="${esc(inv.search_url)}" target="_blank" rel="noopener"
title="No inventory record is linked yet">📦 Find in inventory ↗</a>${edit}`;
}
function openInventoryEditor() {
const inv = (_device && _device.inventory) || {};
const form = document.getElementById('inventory-form');
form.url.value = inv.url || '';
document.getElementById('inventory-search').href = inv.search_url || '#';
lt.modal.open('inventory-modal');
form.url.focus();
}
async function saveInventory(e) {
e.preventDefault();
try {
await lt.api.put(`/api/device/${encodeURIComponent(DEVICE_NAME)}/inventory`, { url: e.target.url.value.trim() });
lt.modal.close('inventory-modal');
lt.toast.success(e.target.url.value.trim() ? 'Inventory link saved' : 'Inventory link removed');
await loadDevice();
} catch (err) {
lt.toast.error(err.message || 'Save failed');
}
}
document.addEventListener('click', e => {
if (e.target.closest('[data-action="edit-inventory"]')) openInventoryEditor();
});
const _inventoryForm = document.getElementById('inventory-form');
if (_inventoryForm) _inventoryForm.addEventListener('submit', saveInventory);
function render(d) {
_device = d;
document.getElementById('device-dot').className = `dot-${d.status || 'unknown'}`;
@@ -339,7 +381,7 @@ function render(d) {
(d.updated ? ` · updated ${new Date(_toIso(d.updated)).toLocaleTimeString()}` : '');
const unifi = d.source === 'unifi';
document.getElementById('device-actions').innerHTML = `
document.getElementById('device-actions').innerHTML = inventoryActions(d) + `
<button class="lt-btn lt-btn-ghost lt-btn-sm btn-suppress" data-sup-type="${unifi ? 'unifi_device' : 'host'}"
data-sup-name="${esc(d.name)}" data-sup-detail="" aria-label="Suppress alerts for ${esc(d.name)}">🔕 Suppress</button>`;
+218
View File
@@ -0,0 +1,218 @@
'use strict';
/* global deviceUrl, _toIso */
// Alert history page: renders /api/events/history with filters kept in the URL.
const esc = s => lt.escHtml(s == null ? '' : String(s));
const FILTERS = ['target', 'range', 'type', 'severity', 'status'];
const DEFAULTS = { target: '', range: '7d', type: '', severity: '', status: 'all' };
const RANGE_TEXT = { '24h': '24 hours', '7d': '7 days', '30d': '30 days', '90d': '90 days' };
let _offset = 0;
let _loadSeq = 0;
function typeText(t) {
return String(t || '').replace(/_/g, ' ');
}
function duration(seconds) {
let s = Math.max(0, Math.floor(seconds || 0));
if (s < 60) return `${s} s`;
let m = Math.floor(s / 60);
if (m < 60) return `${m} min`;
let h = Math.floor(m / 60);
m %= 60;
if (h < 48) return m ? `${h} h ${m} min` : `${h} h`;
const d = Math.floor(h / 24);
h %= 24;
return h ? `${d} d ${h} h` : `${d} d`;
}
function when(iso) {
if (!iso) return '–';
const d = new Date(_toIso(iso));
if (isNaN(d)) return esc(iso);
const text = d.toLocaleString([], { month: 'short', day: 'numeric', hour: '2-digit', minute: '2-digit' });
return `<span title="${esc(d.toLocaleString())}">${esc(text)}</span>`;
}
function ticketLink(id) {
if (!id) return '<span class="val-neutral">–</span>';
const base = (typeof GANDALF_CONFIG !== 'undefined' && GANDALF_CONFIG.ticket_web_url) || '';
return base ? `<a href="${esc(base + id)}" target="_blank" rel="noopener">#${esc(id)}</a>` : `#${esc(id)}`;
}
function filters() {
const f = {};
FILTERS.forEach(k => { f[k] = document.getElementById(`ev-${k}`).value.trim(); });
return f;
}
function readUrl() {
const q = new URLSearchParams(location.search);
FILTERS.forEach(k => {
const el = document.getElementById(`ev-${k}`);
const v = q.get(k);
if (v == null) return;
if (el.tagName === 'SELECT' && ![...el.options].some(o => o.value === v)) {
// An alert type from a link may not be in the list until data arrives
if (k === 'type') el.add(new Option(typeText(v), v));
else return;
}
el.value = v;
});
}
function writeUrl(f) {
const q = new URLSearchParams();
FILTERS.forEach(k => { if (f[k] && f[k] !== DEFAULTS[k]) q.set(k, f[k]); });
const qs = q.toString();
history.replaceState(null, '', qs ? `?${qs}` : location.pathname);
document.getElementById('ev-clear').hidden = !FILTERS.some(k => k !== 'range' && f[k] && f[k] !== DEFAULTS[k]);
}
function setFilter(changes) {
Object.entries(changes).forEach(([k, v]) => {
const el = document.getElementById(`ev-${k}`);
if (el.tagName === 'SELECT' && v && ![...el.options].some(o => o.value === v)) el.add(new Option(typeText(v), v));
el.value = v;
});
_offset = 0;
load();
}
function fillTypes(types) {
const sel = document.getElementById('ev-type');
const current = sel.value;
sel.innerHTML = '<option value="">All alert types</option>' +
types.map(t => `<option value="${esc(t)}">${esc(typeText(t))}</option>`).join('');
if (current && !types.includes(current)) sel.add(new Option(typeText(current), current));
sel.value = current;
}
function renderSummary(data, f) {
const open = data.events.filter(e => !e.resolved_at).length;
const chips = [
`<span class="cs-chip"><b>${data.total}</b> alert${data.total === 1 ? '' : 's'} in the last ${esc(RANGE_TEXT[f.range] || f.range)}</span>`,
];
if (open && data.offset === 0) chips.push(`<span class="cs-chip cs-chip-bad"><b>${open}</b> open now</span>`);
Object.entries(data.by_type || {}).forEach(([t, n]) => {
const on = f.type === t;
chips.push(`<button type="button" class="cs-chip ev-type-chip${on ? ' active' : ''}" data-type="${esc(on ? '' : t)}"
aria-pressed="${on}"><b>${n}</b> ${esc(typeText(t))}</button>`);
});
document.getElementById('ev-summary').innerHTML = chips.join('');
}
function renderFlappers(rows) {
const sec = document.getElementById('ev-flappers-section');
sec.hidden = !rows.length;
if (!rows.length) return;
document.getElementById('ev-flappers').innerHTML = `<div class="lt-table-wrap"><table class="lt-table ev-cards">
<caption class="lt-sr-only">Alerts that fired more than once</caption>
<thead><tr><th>Alert</th><th>Target</th><th>Times</th><th>Total time</th><th>Last started</th><th></th></tr></thead>
<tbody>${rows.map(r => `
<tr>
<td class="ev-c-type">${esc(typeText(r.event_type))}${r.open_now ? ' <span class="lt-badge badge-critical">open</span>' : ''}</td>
<td class="ev-c-target"><a class="device-link" href="${esc(deviceUrl(r.target_name))}">${esc(r.target_name)}</a>
${r.target_detail ? `<span class="cs-port">${esc(r.target_detail)}</span>` : ''}</td>
<td data-label="Times"><strong>${r.n}</strong></td>
<td data-label="Total">${esc(duration(r.total_s))}</td>
<td class="ts-cell" data-label="Last">${when(r.last_seen)}</td>
<td class="ev-c-action"><button type="button" class="lt-btn lt-btn-secondary lt-btn-sm" data-show-type="${esc(r.event_type)}"
data-show-target="${esc(r.target_name)}">Show these</button></td>
</tr>`).join('')}</tbody></table></div>`;
}
function renderTable(data) {
document.getElementById('ev-count').textContent = data.total;
const wrap = document.getElementById('ev-table');
if (!data.events.length) {
wrap.innerHTML = '<div class="lt-empty-state lt-empty-state--sm"><div class="lt-empty-state-icon">✔</div>' +
'<div class="lt-empty-state-title">No alerts match</div></div>';
document.getElementById('ev-pager').innerHTML = '';
return;
}
wrap.innerHTML = `<div class="lt-table-wrap"><table class="lt-table ev-table ev-cards">
<caption class="lt-sr-only">Alert history</caption>
<thead><tr><th>State</th><th>Type</th><th>Target</th><th>Started</th><th>Lasted</th>
<th title="Consecutive failed checks">Checks</th><th>Ticket</th><th>Description</th></tr></thead>
<tbody>${data.events.map(e => {
const open = !e.resolved_at;
return `<tr class="${open ? `row-${esc(e.severity)}` : ''}">
<td class="ev-c-state"><span class="lt-badge badge-${open ? esc(e.severity) : 'neutral'}">${open ? 'open' : 'cleared'}</span></td>
<td class="ev-c-type"><button type="button" class="ev-link" data-show-type="${esc(e.event_type)}"
title="Only show ${esc(typeText(e.event_type))}">${esc(typeText(e.event_type))}</button></td>
<td class="ev-c-target"><a class="device-link" href="${esc(deviceUrl(e.target_name))}">${esc(e.target_name)}</a>
${e.target_detail ? `<span class="cs-port">${esc(e.target_detail)}</span>` : ''}</td>
<td class="ts-cell" data-label="Started">${when(e.first_seen)}</td>
<td data-label="Lasted">${esc(duration(e.duration_s))}${open ? ' <span class="val-neutral">so far</span>' : ''}</td>
<td data-label="Checks">${esc(e.consecutive_failures)}</td>
<td data-label="Ticket">${ticketLink(e.ticket_id)}</td>
<td class="desc-cell ev-c-desc" title="${esc(e.description || '')}">${esc(e.description || '')}</td>
</tr>`;
}).join('')}</tbody></table></div>`;
const end = data.offset + data.events.length;
const prev = data.offset > 0;
const next = end < data.total;
document.getElementById('ev-pager').innerHTML = (prev || next) ? `
<button type="button" class="lt-btn lt-btn-secondary lt-btn-sm" data-page="${data.offset - data.page_size}" ${prev ? '' : 'disabled'}>← Newer</button>
<span>${data.offset + 1}–${end} of ${data.total}</span>
<button type="button" class="lt-btn lt-btn-secondary lt-btn-sm" data-page="${end}" ${next ? '' : 'disabled'}>Older →</button>` : '';
}
async function load() {
const f = filters();
writeUrl(f);
const q = new URLSearchParams();
FILTERS.forEach(k => { if (f[k]) q.set(k, f[k]); });
if (_offset) q.set('offset', _offset);
const seq = ++_loadSeq;
try {
const data = await lt.api.get(`/api/events/history?${q}`);
if (seq !== _loadSeq) return; // a newer filter change is already loading
fillTypes(data.types || []);
renderSummary(data, f);
renderFlappers(data.flappers || []);
renderTable(data);
} catch (err) {
if (seq !== _loadSeq) return;
document.getElementById('ev-table').innerHTML =
`<div class="lt-empty-state lt-empty-state--sm"><div class="lt-empty-state-title">Could not load alerts</div>
<div class="lt-empty-state-text">${esc(err.message || err)}</div></div>`;
}
}
let _typing = null;
document.getElementById('ev-target').addEventListener('input', () => {
clearTimeout(_typing);
_typing = setTimeout(() => { _offset = 0; load(); }, 300);
});
document.getElementById('ev-filters').addEventListener('change', e => {
if (e.target.id === 'ev-target') return;
_offset = 0;
load();
});
document.getElementById('ev-filters').addEventListener('submit', e => { e.preventDefault(); _offset = 0; load(); });
document.getElementById('ev-clear').addEventListener('click', () => {
setFilter({ target: '', type: '', severity: '', status: 'all' });
});
document.addEventListener('click', e => {
const typeChip = e.target.closest('.ev-type-chip');
if (typeChip) return setFilter({ type: typeChip.dataset.type });
const show = e.target.closest('[data-show-type]');
if (show) {
const changes = { type: show.dataset.showType };
if (show.dataset.showTarget) changes.target = show.dataset.showTarget;
return setFilter(changes);
}
const page = e.target.closest('[data-page]');
if (page && !page.disabled) {
_offset = Math.max(0, parseInt(page.dataset.page, 10) || 0);
load();
document.getElementById('ev-count').scrollIntoView({ block: 'nearest' });
}
});
readUrl();
load();
+29 -7
View File
@@ -107,7 +107,13 @@ function portLabelSvg(p, port) {
return `<text class="fp-label" x="${x}" y="${y}" text-anchor="middle"${size}>${p.text || p.idx}</text>${bolt}`;
}
function faceplatePortSvg(p, port, swName, title) {
// Cutsheet status dot in a port's corner: ok (matches), bad (mismatch), doc (documented, not confirmed)
function cutsheetMarkSvg(mark, x, y) {
if (!mark) return '';
return `<circle class="fp-mark fp-mark-${mark}" cx="${x}" cy="${y}" r="3.2"/>`;
}
function faceplatePortSvg(p, port, swName, title, mark) {
// The drawn port number can differ from the index UniFi reports (see anyPort)
const idx = (p.anyPort && port) ? port.port_idx : p.idx;
const attrs = `class="fp-port${port && port.up ? ' fp-up' : ''}" data-switch="${lt.escHtml(swName)}" data-port-idx="${idx}"
@@ -117,6 +123,7 @@ function faceplatePortSvg(p, port, swName, title) {
<title>${title}</title>
${sfpSvg(port, p.w, p.h)}
<rect class="fp-hit" x="-1" y="-1" width="${p.w + 2}" height="${p.h + 2}" rx="2"/>
${cutsheetMarkSvg(mark, p.w - 3, 3)}
</g>
${sfpLedSvg(p, port)}${portLabelSvg(p, port)}`;
}
@@ -125,6 +132,7 @@ function faceplatePortSvg(p, port, swName, title) {
<title>${title}</title>
<g transform="scale(1 ${p.flip ? -1 : 1})">${rj45Svg(port, p)}</g>
<rect class="fp-hit" x="-18" y="-${p.flip ? 11 : 17}" width="36" height="28" rx="2"/>
${cutsheetMarkSvg(mark, 14, p.flip ? 13 : -13)}
</g>${portLabelSvg(p, port)}`;
}
@@ -469,9 +477,17 @@ function faceplateLegendSwatch(cls) {
return `<span class="fp-legend-led fp-led-${cls}"></span>`;
}
function faceplateLegendHtml(spec) {
const items = spec.legend.map(([cls, text]) =>
const CUTSHEET_MARK_LEGEND = [['ok', 'Matches cutsheet'], ['bad', 'Cutsheet mismatch'], ['doc', 'Documented, unconfirmed']];
function faceplateLegendHtml(spec, marks) {
let items = spec.legend.map(([cls, text]) =>
`<div class="chassis-legend-item">${faceplateLegendSwatch(cls)}${text}</div>`).join('');
if (marks && marks.size) {
items += CUTSHEET_MARK_LEGEND.filter(([m]) => marks.has(m)).map(([m, text]) =>
`<div class="chassis-legend-item fp-mark-key"><svg class="fp-legend-icon" viewBox="0 0 8 8"><circle class="fp-mark fp-mark-${m}" cx="4" cy="4" r="3.2"/></svg>${text}</div>`).join('');
items += `<label class="chassis-legend-item fp-mark-toggle"><input type="checkbox" data-action="toggle-cutsheet-marks"
${document.body.classList.contains('fp-marks-off') ? '' : 'checked'}> Cutsheet marks</label>`;
}
const hasDetail = spec.ports.length || spec.outlets;
return `<div class="chassis-legend">
${items}
@@ -479,14 +495,20 @@ function faceplateLegendHtml(spec) {
</div>`;
}
// ctx: { swName, portMap, outlets, online, titleFor(port, idx, role) }
// ctx: { swName, portMap, outlets, online, titleFor(port, idx, role), cutsheetFor?(idx) -> {mark, text} }
function renderFaceplate(model, ctx) {
const spec = FACEPLATES[model];
const anyPort = Object.values(ctx.portMap)[0];
const marks = new Set();
const ports = spec.ports.map(p => {
const port = ctx.portMap[p.idx] || (p.anyPort ? anyPort : undefined);
const title = ctx.titleFor(port, p.idx, spec.roles && spec.roles[p.idx]);
return faceplatePortSvg(p, port, ctx.swName, title);
let title = ctx.titleFor(port, p.idx, spec.roles && spec.roles[p.idx]);
const cs = ctx.cutsheetFor ? ctx.cutsheetFor((p.anyPort && port) ? port.port_idx : p.idx) : null;
if (cs) {
marks.add(cs.mark);
title += lt.escHtml(` · ${cs.text}`);
}
return faceplatePortSvg(p, port, ctx.swName, title, cs && cs.mark);
}).join('');
const outletData = Object.fromEntries((ctx.outlets || []).map(o => [o.index, o]));
const outlets = (spec.outlets || []).map(o => faceplateOutletSvg(o, outletData[o.idx])).join('');
@@ -510,7 +532,7 @@ function renderFaceplate(model, ctx) {
${ports}
</svg>
</div>
</div>${faceplateLegendHtml(spec)}`;
</div>${faceplateLegendHtml(spec, marks)}`;
}
// Where an fp-scroll panel should start: its port area
+27
View File
@@ -112,6 +112,32 @@ function cutsheetPanelHtml(swName, idx) {
}).join('');
}
// The faceplate dot for one port: the worst state of the cables documented on it
const MARK_RANK = { bad: 3, doc: 2, ok: 1 };
function cutsheetMarkFor(idx) {
const entries = ((_apiData && _apiData.cutsheet) || []).filter(e => e.port === String(idx) && e.status !== 'retired');
if (!entries.length) return null;
let best = null;
entries.forEach(e => {
const mark = CUTSHEET_PROBLEM.has(e.verdict) ? 'bad' : (e.verdict === 'match' ? 'ok' : 'doc');
if (!best || MARK_RANK[mark] > MARK_RANK[best.mark]) {
best = { mark, text: `cable ${e.label || 'unlabeled'}: ${e.detail}` };
}
});
return best;
}
document.addEventListener('change', e => {
if (!e.target.matches('[data-action="toggle-cutsheet-marks"]')) return;
const off = !e.target.checked;
document.body.classList.toggle('fp-marks-off', off);
try { localStorage.setItem('gandalf.cutsheetMarks', off ? 'off' : 'on'); } catch (_) { /* private mode */ }
});
try {
if (localStorage.getItem('gandalf.cutsheetMarks') === 'off') document.body.classList.add('fp-marks-off');
} catch (_) { /* private mode */ }
// ── SFP module section of the port panel ────────────────────────────────
function sfpPanelHtml(sfp) {
if (!sfp) return '';
@@ -206,6 +232,7 @@ function renderChassis(swName, sw) {
if (FACEPLATES[model]) {
return `<div class="inspector-chassis">${header}${renderFaceplate(model, {
swName, portMap, outlets: sw.outlets, online: sw.online, titleFor: faceplatePortTitle,
cutsheetFor: cutsheetMarkFor,
})}</div>`;
}
+244
View File
@@ -0,0 +1,244 @@
'use strict';
/* global FACEPLATES, renderFaceplate */
// Rack page: draws the rack front from /api/rack. Switches use their real faceplates,
// patch panels a generic panel; hovering a port lights up every port on the same cable run.
const esc = s => lt.escHtml(s == null ? '' : String(s));
const RATIO = 10.5; // a 19" rack's width to one unit's height
const KIND_TEXT = { patch_panel: 'Patch panel', wall_jack: 'Wall jack', device: 'Device', shelf: 'Shelf', other: 'Other' };
let _rack = null;
function speedText(mbps) {
if (!mbps) return '';
return mbps >= 1000 ? `${mbps / 1000}G` : `${mbps}M`;
}
function portMapOf(ports) {
const map = {};
Object.entries(ports || {}).forEach(([name, p]) => {
if (p.port_idx != null) map[p.port_idx] = Object.assign({ name }, p);
});
return map;
}
function portTitle(port, idx) {
const parts = [port && port.name && port.name !== `Port ${idx}` ? `Port ${idx} "${port.name}"` : `Port ${idx}`];
if (port) {
parts.push(port.up ? (speedText(port.speed_mbps) || 'up') : 'down');
const nb = port.lldp && (port.lldp.system_name || port.lldp.chassis_id);
if (nb) parts.push(`→ ${nb}`);
}
return esc(parts.join(' · '));
}
function isRackFaceplate(model) {
const spec = FACEPLATES[model];
return !!spec && !/fp-desk/.test(spec.frameClass || '');
}
// ── Item drawings ───────────────────────────────────────────────────────
function switchHtml(item) {
const sw = item.switch;
return renderFaceplate(sw.model, {
swName: item.name, portMap: portMapOf(sw.ports), outlets: sw.outlets, online: sw.online,
titleFor: (port, idx) => portTitle(port, idx),
cutsheetFor: idx => {
const m = item.ports[String(idx)];
return m ? { mark: m.mark, text: m.text } : null;
},
});
}
function panelSvg(item) {
const count = Math.max(1, item.port_count || 24);
const units = item.rack_height || 1;
const rows = Math.max(1, Math.min(units * 2, Math.ceil(count / 24)));
const perRow = Math.ceil(count / rows);
const W = 1100, H = 105 * units, ear = 70;
const pitch = Math.min(38, (W - ear * 2 - 40) / perRow);
const left = (W - pitch * perRow) / 2;
const rowH = (H - 20) / rows;
// Label characters that fit under one port at the label's font size (9px, ~5.6px a character)
const fit = Math.max(3, Math.floor((pitch - 2) / 5.6));
const short = t => (t.length > fit ? `${t.slice(0, fit - 1)}…` : t);
const ports = [];
for (let n = 1; n <= count; n++) {
const r = Math.floor((n - 1) / perRow), c = (n - 1) % perRow;
const x = left + c * pitch + pitch / 2, y = 10 + r * rowH + rowH / 2;
const m = item.ports[String(n)];
const label = item.labels[String(n)] || '';
const title = [`Port ${n}`, label && `"${label}"`, m ? m.text : 'not on the cutsheet'].filter(Boolean).join(' · ');
const attrs = m ? ` data-run="${m.run.join(' ')}" data-cable="${m.cable_id}" tabindex="0" role="link"` : '';
ports.push(`<g class="rack-pp-port${m ? ` rack-pp-${m.mark}` : ''}"${attrs} transform="translate(${x} ${y})">
<title>${esc(title)}</title>
<text class="rack-pp-num" y="-17" text-anchor="middle">${n}</text>
<rect class="rack-pp-jack" x="-12" y="-10" width="24" height="20" rx="2"/>
<rect class="rack-pp-hit" x="${-pitch / 2}" y="-28" width="${pitch}" height="56"/>
${label ? `<text class="rack-pp-label" y="24" text-anchor="middle">${esc(short(label))}</text>` : ''}
</g>`);
}
return `<svg class="rack-svg" viewBox="0 0 ${W} ${H}" preserveAspectRatio="xMidYMid meet" role="group"
aria-label="${esc(item.name)} patch panel">
<rect class="rack-pp-body" width="${W}" height="${H}" rx="3"/>
<text class="rack-pp-name" x="${ear - 50}" y="${H / 2 + 5}">${esc(item.name)}</text>
${ports.join('')}
</svg>`;
}
function plateHtml(item) {
const sub = [KIND_TEXT[item.kind] || item.kind, item.location].filter(Boolean).join(' · ');
return `<div class="rack-plate">
<a class="device-link rack-plate-name" href="${esc(item.url)}">${esc(item.name)}</a>
<span class="rack-plate-sub">${esc(sub)}</span>
</div>`;
}
function itemHtml(item) {
let body;
if (item.switch && isRackFaceplate(item.switch.model)) body = switchHtml(item);
else if (item.kind === 'patch_panel') body = panelSvg(item);
else body = plateHtml(item);
return `<div class="rack-item rack-kind-${esc(item.kind)}" data-item="${esc(item.id)}">${body}</div>`;
}
function uLabels(top, bottom) {
const spans = [];
for (let u = top; u >= bottom; u--) spans.push(`<span>${u}</span>`);
return `<div class="rack-u" aria-hidden="true">${spans.join('')}</div>`;
}
function row(top, bottom, inner, cls) {
const n = top - bottom + 1;
return `<div class="rack-row ${cls}" style="--n:${n}">
${uLabels(top, bottom)}
<div class="rack-slot" style="aspect-ratio:${RATIO} / ${n}">${inner}</div>
</div>`;
}
// ── Page ────────────────────────────────────────────────────────────────
function render(data) {
_rack = data;
const byTop = Object.fromEntries(data.items.map(i => [i.rack_top, i]));
const rows = [];
let u = data.units;
while (u >= 1) {
const item = byTop[u];
if (item) {
rows.push(row(u, item.rack_u, itemHtml(item), 'rack-row-item'));
u = item.rack_u - 1;
continue;
}
let low = u;
while (low - 1 >= 1 && !byTop[low - 1]) low--;
rows.push(row(u, low, '', 'rack-row-empty'));
u = low - 1;
}
document.getElementById('rack').innerHTML = rows.join('');
document.getElementById('rack-count').textContent = data.items.length;
// Faceplate ports carry their cable run too, for hover and click
document.querySelectorAll('.rack-item .fp-port').forEach(g => {
const item = data.items.find(i => i.name === g.dataset.switch);
const m = item && item.ports[g.dataset.portIdx];
if (m) {
g.dataset.run = m.run.join(' ');
g.dataset.cable = m.cable_id;
}
});
const notes = [];
data.overlaps.forEach(o => notes.push(`<strong>${esc(o.name)}</strong> (${esc(o.units)}) overlaps ${esc(o.with.join(', '))} and is not drawn.`));
data.outside.forEach(o => notes.push(`<strong>${esc(o.name)}</strong> (${esc(o.units)}) is above the top of a ${data.units}U rack and is not drawn.`));
const warn = document.getElementById('rack-warnings');
warn.hidden = !notes.length;
warn.innerHTML = notes.length ? `<ul>${notes.map(n => `<li>${n}</li>`).join('')}</ul>
<p>Fix the rack unit on the <a href="/cutsheet#cs-items-section">cutsheet</a>.</p>` : '';
const unplaced = document.getElementById('rack-unplaced');
unplaced.innerHTML = data.unplaced.length
? `Not in the drawing (no rack unit set): ${data.unplaced.map(n => `<span class="cs-chip">${esc(n)}</span>`).join(' ')}`
: '';
if (!data.items.length) {
document.getElementById('rack').insertAdjacentHTML('afterbegin',
`<div class="lt-empty-state lt-empty-state--sm rack-empty"><div class="lt-empty-state-title">Nothing placed yet</div>
<div class="lt-empty-state-text">Give panels, shelves and devices a rack unit on the <a href="/cutsheet">cutsheet</a> and they appear here.</div></div>`);
}
}
function readout(el) {
const out = document.getElementById('rack-readout');
if (!el) {
out.textContent = 'Hover a port to follow its cable.';
return;
}
const title = el.querySelector('title');
out.textContent = title ? title.textContent : '';
}
function highlight(el) {
const ids = el && el.dataset.run ? el.dataset.run.split(' ') : [];
document.querySelectorAll('#rack [data-run]').forEach(x => {
x.classList.toggle('rack-hl', ids.length > 0 && x.dataset.run.split(' ').some(i => ids.includes(i)));
});
}
function portAt(target) {
return target.closest && target.closest('#rack .fp-port, #rack .rack-pp-port');
}
function onEnter(e) {
const el = portAt(e.target);
if (!el) return;
highlight(el);
readout(el);
}
function onLeave(e) {
const el = portAt(e.target);
if (el && !el.contains(e.relatedTarget)) {
highlight(null);
readout(null);
}
}
function openPort(el) {
if (el.classList.contains('fp-port')) {
location.href = `/device/${encodeURIComponent(el.dataset.switch)}#port-${encodeURIComponent(el.dataset.portIdx)}`;
} else if (el.dataset.cable) {
location.href = `/cutsheet#cable-${encodeURIComponent(el.dataset.cable)}`;
}
}
const rackEl = document.getElementById('rack');
rackEl.addEventListener('mouseover', onEnter);
rackEl.addEventListener('mouseout', onLeave);
rackEl.addEventListener('focusin', onEnter);
rackEl.addEventListener('focusout', onLeave);
rackEl.addEventListener('click', e => {
const el = portAt(e.target);
if (el) openPort(el);
});
rackEl.addEventListener('keydown', e => {
const el = portAt(e.target);
if (el && (e.key === 'Enter' || e.key === ' ')) {
e.preventDefault();
openPort(el);
}
});
async function load() {
try {
render(await lt.api.get('/api/rack'));
} catch (err) {
if (_rack) return; // keep the last drawing on a failed refresh
document.getElementById('rack').innerHTML = `<div class="lt-empty-state lt-empty-state--sm">
<div class="lt-empty-state-title">Could not load the rack</div>
<div class="lt-empty-state-text">${esc(err.message || err)}</div></div>`;
}
}
readout(null);
load();
setInterval(() => { if (!document.hidden && !document.querySelector('#rack .rack-hl')) load(); }, 60000);
+93
View File
@@ -1393,3 +1393,96 @@ polygon.fp-led-off { fill: #6b6c71; }
.cs-label-card { break-inside: avoid; border-color: #000; color: #000; }
.lt-table, .lt-table td, .lt-table th { color: #000 !important; }
}
/* ── Alert history (/events) ───────────────────────────────────────── */
.ev-filters .lt-select { flex: 0 1 170px; }
.ev-type-chip { background: none; cursor: pointer; font: inherit; font-size: .82rem; }
.ev-type-chip:hover, .ev-type-chip:focus-visible { border-color: var(--accent-cyan); }
.ev-type-chip.active { border-color: var(--accent-cyan); background: var(--accent-cyan-dim); }
.ev-link { background: none; border: 0; padding: 0; color: inherit; font: inherit; cursor: pointer;
text-decoration: underline dotted; text-underline-offset: 3px; text-align: left; }
.ev-link:hover, .ev-link:focus-visible { color: var(--accent-cyan); }
.ev-table td { vertical-align: top; }
.ev-table .cs-port { display: block; }
.ev-pager { display: flex; gap: 12px; align-items: center; justify-content: center; margin-top: 12px; font-size: .85rem; }
.g-section-more { margin: 10px 12px 4px; font-size: .85rem; }
.ev-table .lt-badge, .ev-c-type { white-space: nowrap; }
/* Phones: each alert becomes a card instead of a squeezed table row */
@media (max-width: 760px) {
.ev-cards thead { display: none; }
.ev-cards, .ev-cards tbody { display: block; width: 100%; }
.ev-cards tr { display: flex; flex-wrap: wrap; align-items: baseline; gap: 4px 14px; padding: 10px 12px;
border-bottom: 1px solid var(--border-color); }
.ev-cards td { display: block; padding: 0; border: 0; max-width: none; white-space: normal; font-size: .82rem; }
.ev-cards td[data-label]::before { content: attr(data-label) " "; color: var(--text-muted); font-size: .72rem;
text-transform: uppercase; letter-spacing: .05em; }
.ev-cards .ev-c-target { flex: 1 1 100%; order: -1; font-size: .95rem; }
.ev-cards .ev-c-target .cs-port { display: inline; margin-left: 6px; }
.ev-cards .ev-c-desc { flex: 1 1 100%; color: var(--text-muted); white-space: normal; overflow: visible; }
.ev-cards .ev-c-action { flex: 1 1 100%; }
}
/* ── Cutsheet marks on faceplates ──────────────────────────────────── */
.fp-mark { stroke: #101418; stroke-width: 1.2; pointer-events: none; }
.fp-mark-ok { fill: var(--accent-green, #00ff88); }
.fp-mark-bad { fill: var(--status-critical, #e0312b); }
.fp-mark-doc { fill: #9aa7b4; }
body.fp-marks-off .fp-svg .fp-mark { display: none; }
.fp-mark-toggle { cursor: pointer; gap: 6px; }
.fp-mark-toggle input { margin: 0; }
/* ── Rack (/rack) ──────────────────────────────────────────────────── */
.rack { max-width: 1000px; display: flex; flex-direction: column; gap: 2px; padding: 10px 12px 10px 0;
background: #0b0f14; border: 1px solid var(--border-color); border-radius: 4px; }
.rack-row { display: grid; grid-template-columns: 34px 1fr; }
.rack-u { display: flex; flex-direction: column; font-size: .62rem; color: var(--text-muted); text-align: right;
padding-right: 6px; font-family: var(--font-mono, monospace); }
.rack-u span { flex: 1; display: flex; align-items: center; justify-content: flex-end; }
.rack-slot { position: relative; min-width: 0; }
.rack-row-empty .rack-slot { background: repeating-linear-gradient(to bottom, transparent 0 calc(100% / var(--n) - 1px),
rgba(255,255,255,.05) calc(100% / var(--n) - 1px) calc(100% / var(--n))); border: 1px dashed rgba(255,255,255,.06); }
.rack-item, .rack-item .fp-wrap, .rack-item .fp-frame { height: 100%; }
.rack-item .fp-wrap { padding: 0; }
.rack-item .fp-frame { box-shadow: none; overflow: hidden; }
.rack-item .fp-svg { width: 100%; height: 100%; }
.rack-item .chassis-legend { display: none; }
.rack-svg { display: block; width: 100%; height: 100%; }
.rack-plate { height: 100%; display: flex; align-items: center; gap: 14px; padding: 0 18px; border-radius: 3px;
background: linear-gradient(#232a33, #1a2028); border: 1px solid #2d3743; }
.rack-plate-name { font-weight: 700; }
.rack-plate-sub { font-size: .78rem; color: var(--text-muted); }
.rack-kind-shelf .rack-plate { background: repeating-linear-gradient(90deg, #1b2129 0 10px, #161b22 10px 20px); }
.rack-pp-body { fill: #1d2229; stroke: #39424e; }
.rack-pp-name { fill: #9aa7b4; font-size: 15px; font-family: var(--font-mono, monospace); }
.rack-pp-num { fill: #7d8a97; font-size: 10px; }
.rack-pp-label { fill: #c9d3dc; font-size: 9px; }
.rack-pp-jack { fill: #0c0f13; stroke: #4a5560; stroke-width: 1.5; }
.rack-pp-ok .rack-pp-jack { stroke: var(--accent-green); }
.rack-pp-bad .rack-pp-jack { stroke: var(--status-critical, #e0312b); stroke-width: 2.5; }
.rack-pp-doc .rack-pp-jack { stroke: #9aa7b4; }
.rack-pp-hit { fill: transparent; }
.rack-pp-port[data-cable] { cursor: pointer; outline: none; }
/* A hovered run: every port on it */
.rack-pp-port.rack-hl .rack-pp-jack { fill: rgba(0,212,255,.35); stroke: var(--accent-cyan); stroke-width: 3; }
.rack .fp-port.rack-hl .fp-hit { stroke: var(--accent-cyan); stroke-width: 4; fill: rgba(0,212,255,.3); }
.rack-readout { min-height: 1.4em; margin: 0 0 8px; font-size: .85rem; color: var(--accent-cyan); }
.rack-legend { display: flex; flex-wrap: wrap; gap: 14px; font-size: .75rem; color: var(--text-muted); margin-bottom: 8px; }
.rack-key { display: inline-block; width: 10px; height: 10px; border-radius: 50%; margin-right: 6px; vertical-align: -1px; }
.rack-key-ok { background: var(--accent-green); }
.rack-key-bad { background: var(--status-critical, #e0312b); }
.rack-key-doc { background: #9aa7b4; }
.rack-warnings { border: 1px solid var(--status-critical, #e0312b); padding: 8px 14px; margin-bottom: 14px; font-size: .85rem; }
.rack-warnings ul { margin: 0 0 6px; padding-left: 18px; }
.rack-warnings p { margin: 0; }
.rack-unplaced { margin-top: 12px; display: flex; flex-wrap: wrap; gap: 6px; align-items: center; }
.rack-empty { margin: 12px; }
@media (max-width: 600px) {
.rack-row { grid-template-columns: 22px 1fr; }
.rack-u { font-size: .5rem; padding-right: 3px; }
.rack-plate { padding: 0 8px; gap: 8px; }
.rack-plate-sub { display: none; }
}
+19 -1
View File
@@ -6,7 +6,7 @@
<meta name="theme-color" content="#030508">
<meta name="robots" content="noindex, nofollow">
<link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 16 16'%3E%3Crect width='16' height='16' rx='3' fill='%23030508'/%3E%3Cpath d='M8 2l5 3v6l-5 3-5-3V5z' fill='none' stroke='%2300e5ff' stroke-width='1.5'/%3E%3C/svg%3E">
<title>{% block title %}GANDALF{% endblock %} — GANDALF</title>
<title>{% block title %}Network Monitor{% endblock %} — GANDALF</title>
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
@@ -37,9 +37,15 @@
<a href="{{ url_for('index') }}"
class="lt-nav-drawer-link{% if request.endpoint == 'index' %} active{% endif %}"
{% if request.endpoint == 'index' %}aria-current="page"{% endif %}>Dashboard</a>
<a href="{{ url_for('events_page') }}"
class="lt-nav-drawer-link{% if request.endpoint == 'events_page' %} active{% endif %}"
{% if request.endpoint == 'events_page' %}aria-current="page"{% endif %}>History</a>
<a href="{{ url_for('cutsheet_page') }}"
class="lt-nav-drawer-link{% if request.endpoint == 'cutsheet_page' %} active{% endif %}"
{% if request.endpoint == 'cutsheet_page' %}aria-current="page"{% endif %}>Cutsheet</a>
<a href="{{ url_for('rack_page') }}"
class="lt-nav-drawer-link{% if request.endpoint == 'rack_page' %} active{% endif %}"
{% if request.endpoint == 'rack_page' %}aria-current="page"{% endif %}>Rack</a>
{% if user.groups and 'admin' in user.groups %}
<a href="{{ url_for('suppressions_page') }}"
class="lt-nav-drawer-link{% if request.endpoint == 'suppressions_page' %} active{% endif %}"
@@ -82,11 +88,21 @@
{% if request.endpoint == 'index' %}aria-current="page"{% endif %}>
Dashboard
</a>
<a href="{{ url_for('events_page') }}"
class="lt-nav-link{% if request.endpoint == 'events_page' %} active{% endif %}"
{% if request.endpoint == 'events_page' %}aria-current="page"{% endif %}>
History
</a>
<a href="{{ url_for('cutsheet_page') }}"
class="lt-nav-link{% if request.endpoint == 'cutsheet_page' %} active{% endif %}"
{% if request.endpoint == 'cutsheet_page' %}aria-current="page"{% endif %}>
Cutsheet
</a>
<a href="{{ url_for('rack_page') }}"
class="lt-nav-link{% if request.endpoint == 'rack_page' %} active{% endif %}"
{% if request.endpoint == 'rack_page' %}aria-current="page"{% endif %}>
Rack
</a>
{% if user.groups and 'admin' in user.groups %}
<a href="{{ url_for('suppressions_page') }}"
class="lt-nav-link{% if request.endpoint == 'suppressions_page' %} active{% endif %}"
@@ -324,7 +340,9 @@
// Command palette
lt.cmdPalette.init([
{ id: 'nav-dashboard', group: 'Navigate', icon: '~', label: 'Dashboard', action: function() { window.location.href = '/'; } },
{ id: 'nav-history', group: 'Navigate', icon: '⏱', label: 'Alert History', action: function() { window.location.href = '/events'; } },
{ id: 'nav-cutsheet', group: 'Navigate', icon: '🔌', label: 'Cutsheet', action: function() { window.location.href = '/cutsheet'; } },
{ id: 'nav-rack', group: 'Navigate', icon: '🗄', label: 'Rack', action: function() { window.location.href = '/rack'; } },
{ id: 'nav-suppressions', group: 'Navigate', icon: '🔕', label: 'Suppressions', action: function() { window.location.href = '/suppressions'; } },
{ id: 'action-refresh', group: 'Actions', icon: '↻', label: 'Refresh Data', kbd: 'R', action: function() { lt.autoRefresh.now(); } },
{ id: 'action-suppress', group: 'Actions', icon: '🔕', label: 'New Suppression', kbd: 'S', action: function() { if (typeof openSuppressModal === 'function') openSuppressModal('host','',''); else window.location.href='/suppressions'; } },
+2 -1
View File
@@ -1,5 +1,5 @@
{% extends "base.html" %}
{% block title %}Cutsheet – GANDALF{% endblock %}
{% block title %}Cutsheet{% endblock %}
{% block content %}
<div class="lt-page-header">
@@ -95,6 +95,7 @@
<h2 class="g-section-title">Rack</h2>
</div>
<div id="cs-rack"></div>
<p class="g-section-more"><a href="/rack">Open the rack drawing →</a></p>
</section>
<section class="g-section" id="cs-items-section">
+29 -2
View File
@@ -1,5 +1,5 @@
{% extends "base.html" %}
{% block title %}{{ device_name }} – GANDALF{% endblock %}
{% block title %}{{ device_name }}{% endblock %}
{% block content %}
<div class="lt-page-header">
@@ -46,10 +46,37 @@
</section>
{% include "_modem_panel.html" %}
{% if can_edit %}
<div id="inventory-modal" class="lt-modal-overlay" role="dialog" aria-modal="true" aria-labelledby="inventory-title" aria-hidden="true">
<div class="lt-modal">
<div class="lt-modal-header">
<h3 class="lt-modal-title" id="inventory-title">Inventory link</h3>
<button type="button" class="lt-modal-close" data-modal-close aria-label="Close">&#x2715;</button>
</div>
<form id="inventory-form">
<div class="lt-modal-body">
<p class="cs-hint">Open this device's record in the inventory and paste its address here. Leave it empty to remove the link.</p>
<div class="lt-form-group">
<label class="lt-label" for="inventory-url">Inventory record URL</label>
<input class="lt-input" id="inventory-url" name="url" type="url" inputmode="url" autocomplete="off"
placeholder="https://inventory.lotusguild.org/item/…">
</div>
<p class="cs-hint"><a id="inventory-search" href="#" target="_blank" rel="noopener">Search the inventory for this model ↗</a></p>
</div>
<div class="lt-modal-footer">
<button type="button" class="lt-btn lt-btn-secondary" data-modal-close>Cancel</button>
<button type="submit" class="lt-btn lt-btn-primary">Save</button>
</div>
</form>
</div>
</div>
{% endif %}
{% endblock %}
{% block scripts %}
<script>const DEVICE_NAME = {{ device_name | tojson }};</script>
<script>const DEVICE_NAME = {{ device_name | tojson }};
const CAN_EDIT = {{ 'true' if can_edit else 'false' }};</script>
<script src="{{ url_for('static', filename='faceplates.js') }}"></script>
<script src="{{ url_for('static', filename='port-inspector.js') }}"></script>
<script src="{{ url_for('static', filename='device.js') }}"></script>
+66
View File
@@ -0,0 +1,66 @@
{% extends "base.html" %}
{% block title %}Alert History{% endblock %}
{% block content %}
<div class="lt-page-header">
<div>
<h1 class="lt-page-title">Alert History</h1>
<p class="g-page-sub">Every alert Gandalf raised, including the ones that cleared on their own. Resolved alerts are kept 90 days.</p>
</div>
</div>
<form class="cs-filters ev-filters" id="ev-filters" role="search" aria-label="Filter alert history">
<input type="search" class="lt-input" id="ev-target" name="target" placeholder="Device, host, port or interface"
aria-label="Filter by device" autocomplete="off">
<select class="lt-select" id="ev-range" name="range" aria-label="Time range">
<option value="24h">Last 24 hours</option>
<option value="7d" selected>Last 7 days</option>
<option value="30d">Last 30 days</option>
<option value="90d">Last 90 days</option>
</select>
<select class="lt-select" id="ev-type" name="type" aria-label="Alert type">
<option value="">All alert types</option>
</select>
<select class="lt-select" id="ev-severity" name="severity" aria-label="Severity">
<option value="">All severities</option>
<option value="critical">Critical</option>
<option value="warning">Warning</option>
<option value="info">Info</option>
</select>
<select class="lt-select" id="ev-status" name="status" aria-label="State">
<option value="all">Open and cleared</option>
<option value="active">Open now</option>
<option value="resolved">Cleared</option>
</select>
<button type="button" class="lt-btn lt-btn-secondary lt-btn-sm" id="ev-clear" hidden>Clear filters</button>
</form>
<div id="ev-summary" class="cs-summary" aria-live="polite"></div>
<section class="g-section" id="ev-flappers-section" hidden>
<div class="g-section-header">
<h2 class="g-section-title">Recurring</h2>
</div>
<p class="cs-hint">The same alert on the same target more than once in this range. Something that keeps coming back usually has one cause worth finding.</p>
<div class="lt-frame">
<span class="lt-frame-bl">&#x255A;</span><span class="lt-frame-br">&#x255D;</span>
<div id="ev-flappers"></div>
</div>
</section>
<section class="g-section">
<div class="g-section-header">
<h2 class="g-section-title">Alerts</h2>
<span class="g-section-badge" id="ev-count">0</span>
</div>
<div class="lt-frame">
<span class="lt-frame-bl">&#x255A;</span><span class="lt-frame-br">&#x255D;</span>
<div id="ev-table"><div class="lt-empty-state lt-empty-state--sm"><div class="lt-empty-state-title">Loading…</div></div></div>
</div>
<div class="ev-pager" id="ev-pager"></div>
</section>
{% endblock %}
{% block scripts %}
<script src="{{ url_for('static', filename='events.js') }}"></script>
{% endblock %}
+1 -1
View File
@@ -1,5 +1,5 @@
{% extends "base.html" %}
{% block title %}Dashboard – GANDALF{% endblock %}
{% block title %}Dashboard{% endblock %}
{% block content %}
+38
View File
@@ -0,0 +1,38 @@
{% extends "base.html" %}
{% block title %}Rack{% endblock %}
{% block content %}
<div class="lt-page-header">
<div>
<h1 class="lt-page-title">Rack</h1>
<p class="g-page-sub">The rack front, drawn from the rack units on the cutsheet with live port state. Hover a port to light up every port on the same cable run; click it to open the port or the cable.</p>
</div>
{% if can_edit %}
<div class="cs-actions no-print">
<a class="lt-btn lt-btn-secondary lt-btn-sm" href="/cutsheet#cs-items-section">Edit positions</a>
</div>
{% endif %}
</div>
<div class="rack-warnings" id="rack-warnings" role="alert" hidden></div>
<section class="g-section">
<div class="g-section-header">
<h2 class="g-section-title">Front</h2>
<span class="g-section-badge" id="rack-count">0</span>
</div>
<p class="rack-readout" id="rack-readout" aria-live="polite"></p>
<div class="rack-legend" aria-hidden="true">
<span><i class="rack-key rack-key-ok"></i>matches cutsheet</span>
<span><i class="rack-key rack-key-bad"></i>mismatch</span>
<span><i class="rack-key rack-key-doc"></i>documented, unconfirmed</span>
</div>
<div class="rack" id="rack"><div class="link-loading">Loading rack</div></div>
<p class="cs-hint rack-unplaced" id="rack-unplaced"></p>
</section>
{% endblock %}
{% block scripts %}
<script src="{{ url_for('static', filename='faceplates.js') }}"></script>
<script src="{{ url_for('static', filename='rack.js') }}"></script>
{% endblock %}
+1 -1
View File
@@ -1,5 +1,5 @@
{% extends "base.html" %}
{% block title %}Suppressions – GANDALF{% endblock %}
{% block title %}Suppressions{% endblock %}
{% block content %}
+152
View File
@@ -35,6 +35,7 @@ def client():
db.count_active_events.return_value = 0
db.get_status_summary.return_value = {'critical': 0, 'warning': 0, 'info': 0, 'resolved_24h': 0}
db.check_suppressed.return_value = False
db.get_device_link.return_value = ''
with patch.object(gandalf, '_cfg', CONFIG):
yield gandalf.app.test_client()
@@ -316,3 +317,154 @@ class TestCutsheetLivePorts:
def test_admin_only(self, cs_client):
client, _ = cs_client
assert client.get('/api/cutsheet/live-ports', headers=USER).status_code == 403
class TestAlertHistory:
HISTORY = {'events': [], 'total': 0, 'by_type': {}}
def _db(self):
db = gandalf.db
db.get_events_history.return_value = dict(self.HISTORY)
db.get_flappers.return_value = []
db.get_event_types.return_value = ['host_unreachable']
return db
def test_page_and_nav(self, client):
html = client.get('/events', headers=AUTH).get_data(as_text=True)
assert 'Alert History' in html and 'events.js' in html
assert 'href="/events"' in client.get('/', headers=AUTH).get_data(as_text=True)
def test_defaults_to_seven_days(self, client):
db = self._db()
data = client.get('/api/events/history', headers=AUTH).get_json()
assert db.get_events_history.call_args.args == (168, '', '', '', 'all')
assert data['types'] == ['host_unreachable'] and data['page_size'] == 100
def test_filters_are_passed_through(self, client):
db = self._db()
client.get('/api/events/history?range=90d&type=interface_down&target=+large1+&severity=critical'
'&status=resolved&offset=200', headers=AUTH)
assert db.get_events_history.call_args.args == (2160, 'interface_down', 'large1', 'critical', 'resolved')
assert db.get_events_history.call_args.kwargs == {'limit': 100, 'offset': 200}
assert db.get_flappers.call_args.args == (2160, 'interface_down', 'large1', 'critical')
@pytest.mark.parametrize('query', ['range=1y', 'severity=bad', 'status=open', 'offset=x'])
def test_bad_filters_are_rejected(self, client, query):
self._db()
assert client.get(f'/api/events/history?{query}', headers=AUTH).status_code == 400
def test_needs_login(self, client):
assert client.get('/api/events/history').status_code == 401
class TestHealth:
def _health(self, client, age_s, poll=120):
from datetime import datetime, timedelta, timezone
stamp = (datetime.now(timezone.utc) - timedelta(seconds=age_s)).strftime('%Y-%m-%d %H:%M:%S UTC')
gandalf.db.get_state.side_effect = lambda key, default=None: stamp if key == 'last_check' else default
with patch.object(gandalf, '_cfg', dict(CONFIG, monitor={'poll_interval': poll})):
return client.get('/health')
def test_recent_check_is_healthy(self, client):
resp = self._health(client, 150)
assert resp.status_code == 200 and resp.get_json()['checks']['monitor'] == 'ok'
def test_stalled_monitor_fails_after_ten_minutes(self, client):
resp = self._health(client, 660)
assert resp.status_code == 503
assert resp.get_json()['checks'] == {'db': 'ok', 'monitor': 'stale', 'monitor_age_s': 660}
def test_limit_follows_a_long_poll_interval(self, client):
assert self._health(client, 1000, poll=300).status_code == 200
assert self._health(client, 1600, poll=300).status_code == 503
class TestPageTitles:
@pytest.mark.parametrize('path,title', [('/', 'Dashboard — GANDALF'), ('/events', 'Alert History — GANDALF'),
('/cutsheet', 'Cutsheet — GANDALF')])
def test_name_appears_once(self, client, path, title):
gandalf._cutsheet_schema_ready = True
html = client.get(path, headers=AUTH).get_data(as_text=True)
assert f'<title>{title}</title>' in html
class TestInventoryLinks:
def _put(self, client, url, headers=AUTH, name='USW Flex'):
gandalf._cutsheet_schema_ready = True
return client.put(f'/api/device/{name}/inventory', json={'url': url}, headers=headers)
def test_device_api_has_link_and_search(self, client):
gandalf._cutsheet_schema_ready = True
gandalf.db.get_device_link.return_value = 'https://inventory.lotusguild.org/item/abc'
inv = client.get('/api/device/USW Flex', headers=AUTH).get_json()['inventory']
assert inv == {'url': 'https://inventory.lotusguild.org/item/abc',
'search_url': 'https://inventory.lotusguild.org/items?q=USF5P'}
def test_admin_sets_and_clears(self, client):
assert self._put(client, ' https://inventory.lotusguild.org/item/abc ').status_code == 200
gandalf.db.set_device_link.assert_called_with('USW Flex', 'https://inventory.lotusguild.org/item/abc', 'jared')
assert self._put(client, '').status_code == 200
gandalf.db.set_device_link.assert_called_with('USW Flex', '', 'jared')
@pytest.mark.parametrize('url', ['http://inventory.lotusguild.org/item/a', 'https://evil.example/item/a',
'https://inventory.lotusguild.org.evil.example/x', 'javascript:alert(1)',
'https://user:pw@inventory.lotusguild.org/x',
'https://inventory.lotusguild.org/' + 'a' * 600])
def test_only_inventory_links_are_accepted(self, client, url):
assert self._put(client, url).status_code == 400
assert not gandalf.db.set_device_link.called
def test_configured_host(self, client):
cfg = dict(CONFIG, inventory={'base_url': 'https://homebox.example'})
with patch.object(gandalf, '_cfg', cfg):
assert self._put(client, 'https://homebox.example/item/1').status_code == 200
assert self._put(client, 'https://inventory.lotusguild.org/item/1').status_code == 400
def test_needs_admin_and_json(self, client):
user = dict(AUTH, **{'Remote-Groups': 'staff'})
with patch.object(gandalf, '_cfg', dict(CONFIG, auth=dict(CONFIG['auth'], allowed_groups=['admin', 'staff']))):
assert self._put(client, 'https://inventory.lotusguild.org/x', headers=user).status_code == 403
gandalf._cutsheet_schema_ready = True
resp = client.put('/api/device/USW Flex/inventory', data='url=x', headers=AUTH)
assert resp.status_code == 400
class TestRackApi:
def test_rack_combines_items_switches_and_marks(self, client):
gandalf._cutsheet_schema_ready = True
db = gandalf.db
db.cutsheet_load.return_value = {
'items': [
{'id': 1, 'name': 'USW Flex', 'kind': 'device', 'port_count': None, 'rack_u': 20, 'rack_height': 1,
'location': '', 'notes': ''},
{'id': 2, 'name': 'PP1', 'kind': 'patch_panel', 'port_count': 24, 'rack_u': 21, 'rack_height': 1,
'location': '', 'notes': ''},
{'id': 3, 'name': 'Shelf', 'kind': 'shelf', 'port_count': None, 'rack_u': 20, 'rack_height': 1,
'location': '', 'notes': ''},
{'id': 4, 'name': 'Loose', 'kind': 'other', 'port_count': None, 'rack_u': None, 'rack_height': None,
'location': '', 'notes': ''},
],
'cables': [{'id': 7, 'label': 'C07', 'a_device': 'USW Flex', 'a_port': '2', 'a_side': '',
'b_device': 'PP1', 'b_port': '5', 'b_side': 'front', 'status': 'installed',
'cable_type': '', 'length_m': None, 'color': '', 'notes': '', 'label_applied': False}],
'port_labels': [{'item_id': 2, 'item_name': 'PP1', 'port': '5', 'label': 'OFFICE', 'applied': False}],
}
data = client.get('/api/rack', headers=AUTH).get_json()
assert data['units'] == 42 and data['unplaced'] == ['Loose']
assert [i['name'] for i in data['items']] == ['PP1', 'USW Flex']
assert data['overlaps'] == [{'name': 'Shelf', 'units': 'U20', 'with': ['USW Flex']}]
pp, sw = data['items']
assert pp['labels'] == {'5': 'OFFICE'} and pp['url'] == '/cutsheet#item-2'
assert pp['ports']['5']['run'] == [7] and pp['ports']['5']['cable_id'] == 7
assert sw['url'] == '/device/USW%20Flex' and 'switch' not in sw # no port data in this snapshot
assert sw['ports']['2']['mark'] in ('ok', 'bad', 'doc')
def test_rack_page_and_auth(self, client):
assert 'rack.js' in client.get('/rack', headers=AUTH).get_data(as_text=True)
assert client.get('/api/rack').status_code == 401
def test_units_from_config(self, client):
gandalf._cutsheet_schema_ready = True
gandalf.db.cutsheet_load.return_value = {'items': [], 'cables': [], 'port_labels': []}
with patch.object(gandalf, '_cfg', dict(CONFIG, rack={'units': 24})):
assert client.get('/api/rack', headers=AUTH).get_json()['units'] == 24
+48
View File
@@ -248,3 +248,51 @@ class TestProblems:
cables = [cable(1, ('sw', '13'), ('tv', 'eth0')), cable(2, ('sw', '13'), ('pc', 'eth0'))]
found = cs.problems(cables, ITEMS, switch([port(13, seen='tv')]))
assert sorted(p['cable_id'] for p in found) == [1, 2]
class TestRackLayout:
_ids = {}
def _item(self, name, u, h=None):
# IDs in creation order: on a tie for the same top unit, the earlier record is drawn
return {'id': self._ids.setdefault(name, len(self._ids) + 1), 'name': name, 'kind': 'device',
'rack_u': u, 'rack_height': h}
def test_top_first_with_top_unit(self):
out = cs.rack_layout([self._item('a', 1), self._item('b', 10, 2), {'name': 'loose', 'rack_u': None}], 42)
assert [(i['name'], i['rack_top']) for i in out['placed']] == [('b', 11), ('a', 1)]
assert out['overlaps'] == [] and out['outside'] == []
def test_overlap_is_reported_not_drawn(self):
out = cs.rack_layout([self._item('ups', 1, 2), self._item('shelf', 2)], 42)
assert [i['name'] for i in out['placed']] == ['ups']
assert out['overlaps'] == [{'name': 'shelf', 'units': 'U2', 'with': ['ups']}]
def test_past_the_top(self):
out = cs.rack_layout([self._item('tall', 41, 3)], 42)
assert out['placed'] == [] and out['outside'] == [{'name': 'tall', 'units': 'U41–43'}]
class TestPortMarks:
def test_worst_cable_wins_and_runs_merge(self):
cables = [
dict(cable(1, ('sw', '3'), ('PP1', '1', 'front'), label='C1'), verdict='match', detail='ok', run=[1, 2]),
dict(cable(2, ('PP1', '1', 'rear'), ('jack', '1')), verdict='wrong_neighbor', detail='bad', run=[1, 2]),
dict(cable(3, ('sw', '4'), ('x',)), verdict='planned', detail='Planned', run=[3]),
dict(cable(4, ('sw', '5'), ('y',), status='retired'), verdict='retired', detail='', run=[4]),
]
marks = cs.port_marks(cables)
assert marks[('sw', '3')]['mark'] == 'ok'
assert marks[('PP1', '1')]['mark'] == 'bad' and marks[('PP1', '1')]['cable_id'] == 2
assert marks[('PP1', '1')]['run'] == [1, 2]
assert marks[('PP1', '1')]['text'] == 'cable C1: ok; cable #2: bad'
assert marks[('sw', '4')]['text'] == 'cable #3: Planned'
assert marks[('sw', '4')]['mark'] == 'doc'
assert ('sw', '5') not in marks and ('x', '') not in marks
def test_shared_check_is_said_once(self):
cables = [
dict(cable(1, ('sw', '3'), ('PP1', '1', 'front'), label='C1'), verdict='idle', detail='Idle', run=[1, 2]),
dict(cable(2, ('PP1', '1', 'rear'), ('jack', '1')), verdict='idle', detail='Idle', run=[1, 2]),
]
assert cs.port_marks(cables)[('PP1', '1')]['text'] == 'cables C1, #2: Idle'
+188
View File
@@ -3,6 +3,7 @@ import json
import os
import sys
import time
from datetime import timedelta
from unittest.mock import MagicMock, patch
import pytest
@@ -24,6 +25,12 @@ def db():
mock_db.upsert_event.return_value = (1, False, 2, None)
mock_db.get_modem_rx_values.return_value = []
mock_db.get_state.return_value = None
mock_db.event_stats.return_value = {'last_7d': 0, 'last_30d': 0,
'last_first_seen': None, 'last_resolved_at': None}
mock_db.get_events_for_target.return_value = []
mock_db.cutsheet_load.return_value = {'items': [], 'cables': [], 'port_labels': []}
mock_db.get_pending_recovery_notes.return_value = []
mock_db.get_device_link.return_value = ''
yield mock_db
@@ -560,3 +567,184 @@ class TestCutsheetMismatch:
assert not db.upsert_event.called
resolved = {c.args for c in db.resolve_event.call_args_list}
assert ('cable_mismatch', 'sw', 'cable 7') in resolved and ('cable_mismatch', 'other', 'cable 9') in resolved
class TestTicketContext:
SWITCHES = {'agg': {'ports': {
'Large1': {'port_idx': 7, 'up': True, 'speed_mbps': 10000,
'lldp': {'system_name': 'large1', 'port_id': 'enp1s0f0'},
'sfp': {'found': True, 'kind': 'dac', 'part': 'DAC-10G'}},
'Port 3': {'port_idx': 3, 'up': False, 'speed_mbps': 0},
}}}
def _mon(self, mon, switches=None):
mon.public_url = 'https://gandalf.test'
mon.poll_interval = 120
mon._link_data = {'unifi_switches': switches if switches is not None else self.SWITCHES}
return mon
def _desc(self, mon):
return mon.tickets.create.call_args.args[1]
def test_interface_ticket_names_the_switch_port_and_links(self, mon, db):
self._mon(mon)._ticket_interface(1, 'large1', 'enp1s0f0', 2)
desc = self._desc(mon)
assert '### Where it is connected' in desc
assert '`enp1s0f0` → [agg port 7 "Large1": up at 10 Gbps, module: DAC DAC-10G]' in desc
assert '(https://gandalf.test/device/agg#port-7)' in desc
assert 'https://gandalf.test/device/large1' in desc
assert 'Opened automatically by Gandalf' in desc
assert 'inventory' not in desc
def test_inventory_link_when_set(self, mon, db):
db.get_device_link.return_value = 'https://inv.test/item/abc'
self._mon(mon)._ticket_interface(1, 'large1', 'enp1s0f0', 2)
assert '- [large1 in the inventory](https://inv.test/item/abc)' in self._desc(mon)
db.get_device_link.assert_called_with('large1')
def test_dead_host_uses_the_last_known_port(self, mon, db):
mon._known_uplinks = {'large1': {'seen': '2026-11-10 10:00:00 UTC', 'uplinks': [
{'switch': 'agg', 'port': 'Port 3', 'port_idx': 3, 'host_interface': 'eno1'}]}}
self._mon(mon)._ticket_metrics_missing(1, 'micro1', 'x:9100', 5)
assert 'not seen this device on a switch port' in self._desc(mon)
mon.tickets.create.reset_mock()
unplugged = {'agg': {'ports': {'Port 3': {'port_idx': 3, 'up': False}}}}
self._mon(mon, unplugged)._ticket_unreachable(1, 'large1', '10.0.0.9', 5)
desc = self._desc(mon)
assert 'agg port 3: down' in desc
assert 'last seen linked there 2026-11-10 10:00:00 UTC' in desc
assert 'Down for at least' in desc and '10 min' in desc
def test_cutsheet_runs_and_history(self, mon, db):
db.cutsheet_load.return_value = {'items': [], 'port_labels': [], 'cables': [
{'id': 4, 'label': 'C04', 'a_device': 'agg', 'a_port': '7', 'a_side': '',
'b_device': 'large1', 'b_port': 'enp1s0f0', 'b_side': '', 'status': 'installed'}]}
db.event_stats.return_value = {'last_7d': 3, 'last_30d': 5,
'last_first_seen': monitor._utcnow(), 'last_resolved_at': monitor._utcnow()}
self._mon(mon)._ticket_interface(1, 'large1', 'enp1s0f0', 2)
desc = self._desc(mon)
assert '[**C04**](https://gandalf.test/cutsheet#cable-4): agg port 7 → large1 enp1s0f0' in desc
assert 'Happened 3 times in the last 7 days and 5 in the last 30' in desc
assert 'This keeps coming back' in desc
def test_other_open_alerts_are_listed(self, mon, db):
db.get_events_for_target.return_value = [
{'id': 1, 'event_type': 'interface_down', 'target_name': 'large1', 'target_detail': 'eno1'},
{'id': 2, 'event_type': 'interface_down', 'target_name': 'large1', 'target_detail': 'eno2',
'first_seen': 'T', 'ticket_id': '55'},
{'id': 3, 'event_type': 'host_unreachable', 'target_name': 'large1', 'resolved_at': 'R'},
]
self._mon(mon)._ticket_interface(1, 'large1', 'eno1', 2)
desc = self._desc(mon)
assert '- interface down on large1 eno2 since T (ticket #55)' in desc
assert 'eno1 since' not in desc and 'host unreachable' not in desc
def test_a_failing_section_still_opens_the_ticket(self, mon, db):
db.event_stats.side_effect = RuntimeError('db gone')
self._mon(mon)._ticket_interface(1, 'large1', 'enp1s0f0', 2)
desc = self._desc(mon)
assert '### History' not in desc and '### Where it is connected' in desc
db.set_ticket_id.assert_called_with(1, '123')
def test_cable_mismatch_shows_the_switch_port(self, mon, db):
self._mon(mon)._ticket_cable_mismatch(9, {
'switch': 'agg', 'port': '7', 'cable_id': 4, 'label': 'C04', 'key': 'cable 4',
'detail': 'A different device is connected', 'path': 'agg port 7 → tv'}, 24)
desc = self._desc(mon)
assert '[C04](https://gandalf.test/cutsheet#cable-4)' in desc
assert '[agg port 7 "Large1": up at 10 Gbps' in desc
def test_titles_keep_the_tag_format(self, mon, db):
mon.unifi_fail_thresh = 3
self._mon(mon)._ticket_unifi(1, {'name': 'ap1', 'type': 'uap', 'model': 'U6', 'ip': '1.2.3.4'})
assert mon.tickets.create.call_args.args[0] == \
'[ap1][auto][production][issue][network][single-node] UniFi device offline'
def test_remember_uplinks_saves_only_on_change(self, mon, db):
self._mon(mon)
mon._known_uplinks = None
mon._remember_uplinks({'unifi_switches': self.SWITCHES})
saved = db.set_state.call_args.args
assert saved[0] == 'known_uplinks'
assert saved[1]['large1']['uplinks'] == [
{'switch': 'agg', 'port': 'Large1', 'port_idx': 7, 'host_interface': 'enp1s0f0'}]
db.set_state.reset_mock()
mon._remember_uplinks({'unifi_switches': self.SWITCHES})
assert not db.set_state.called
class TestRecoveryNotes:
def _event(self, **kw):
now = monitor._utcnow()
e = {'id': 8, 'event_type': 'host_metrics_missing', 'target_name': 'micro1', 'target_detail': 'x:9100',
'ticket_id': '993', 'first_seen': now - timedelta(minutes=12), 'resolved_at': now,
'consecutive_failures': 6}
e.update(kw)
return e
def test_cleared_alert_comments_and_marks(self, mon, db):
mon.public_url = 'https://gandalf.test'
mon.tickets.comments_denied = False
mon.tickets.comment.return_value = True
db.get_pending_recovery_notes.return_value = [self._event()]
mon._post_recovery_notes()
ticket_id, text = mon.tickets.comment.call_args.args
assert ticket_id == '993'
assert 'Cleared' in text and '12 min' in text and 'Failed checks:** 6' in text
assert 'https://gandalf.test/device/micro1' in text
db.mark_recovery_notified.assert_called_once_with(8)
assert db.get_pending_recovery_notes.call_args.kwargs['exclude_types'] == monitor.ONE_SHOT_EVENTS
def test_failed_comment_is_left_for_next_cycle(self, mon, db):
mon.tickets.comments_denied = False
mon.tickets.comment.return_value = False
db.get_pending_recovery_notes.return_value = [self._event(), self._event(id=9)]
mon._post_recovery_notes()
assert not db.mark_recovery_notified.called
assert mon.tickets.comment.call_count == 2
def test_denied_key_stops_trying(self, mon, db):
mon.tickets.comments_denied = True
mon._post_recovery_notes()
assert not db.get_pending_recovery_notes.called
def test_cable_mismatch_links_the_cutsheet(self, mon, db):
mon.public_url = 'https://gandalf.test'
url = mon._event_url({'event_type': 'cable_mismatch', 'target_name': 'sw', 'target_detail': 'cable 7'})
assert url == 'https://gandalf.test/cutsheet#cable-7'
class TestTicketClientComment:
def _client(self):
return monitor.TicketClient({'url': 'http://t.test/create_ticket_api.php', 'api_key': 'k'})
def test_comment_endpoint_sits_beside_create(self):
assert self._client().comment_url == 'http://t.test/api/ticket_comment_api.php'
sub = monitor.TicketClient({'url': 'http://t.test/tinker/create_ticket_api.php', 'api_key': 'k'})
assert sub.comment_url == 'http://t.test/tinker/api/ticket_comment_api.php'
def test_payload(self):
c = self._client()
resp = MagicMock(status_code=200)
resp.json.return_value = {'success': True, 'comment_id': 1}
with patch.object(monitor.requests, 'post', return_value=resp) as post:
assert c.comment('42', 'hi') is True
assert post.call_args.kwargs['json'] == {'ticket_id': '42', 'comment_text': 'hi', 'markdown_enabled': True}
assert post.call_args.kwargs['headers'] == {'Authorization': 'Bearer k'}
def test_forbidden_disables_comments(self):
c = self._client()
with patch.object(monitor.requests, 'post', return_value=MagicMock(status_code=403)) as post:
assert c.comment('42', 'hi') is False
assert c.comment('43', 'hi') is False
assert post.call_count == 1 and c.comments_denied
def test_missing_ticket_counts_as_done(self):
with patch.object(monitor.requests, 'post', return_value=MagicMock(status_code=404)):
assert self._client().comment('42', 'hi') is True
def test_network_error_is_retryable(self):
c = self._client()
with patch.object(monitor.requests, 'post', side_effect=OSError('down')):
assert c.comment('42', 'hi') is False
assert not c.comments_denied
+64
View File
@@ -0,0 +1,64 @@
"""Tests for ticket_text.py: the Markdown Gandalf puts in tickets."""
import os
import sys
from datetime import datetime, timedelta
sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
import ticket_text as tt # noqa: E402
NOW = datetime(2026, 11, 10, 18, 0)
def test_duration():
assert tt.duration(42) == '42 s'
assert tt.duration(600) == '10 min'
assert tt.duration(3600) == '1 h'
assert tt.duration(3 * 3600 + 120) == '3 h 2 min'
assert tt.duration(3 * 86400 + 7200) == '3 d 2 h'
def test_speed():
assert tt.speed(100) == '100 Mbps'
assert tt.speed(2500) == '2.5 Gbps'
assert tt.speed(10000) == '10 Gbps'
assert tt.speed(0) == ''
def test_device_url_escapes_names():
assert tt.device_url('https://g', 'USW Flex Mini', 3) == 'https://g/device/USW%20Flex%20Mini#port-3'
assert tt.device_url('', 'x') == ''
def test_port_text():
assert tt.port_text('sw', 'Port 4', {'port_idx': 4, 'up': True, 'speed_mbps': 1000, 'poe_power': 4.21}) == \
'sw port 4: up at 1 Gbps, PoE 4.2 W'
assert tt.port_text('sw', 'Cam', {'port_idx': 9, 'up': False,
'sfp': {'found': True, 'kind': 'optic', 'part': 'SR', 'rx_fault': True}}) == \
'sw port 9 "Cam": down, module: optic SR, **module reports a fault**'
def test_description_skips_empty_facts_and_sections():
text = tt.description('Alert', [('Host', 'a'), ('Blank', '')], 'Summary.', ['One', 'Two'],
[('Empty', []), ('Links', ['- x'])])
assert text.startswith('## Alert\n\n- **Host:** a\n\nSummary.')
assert 'Blank' not in text and '### Empty' not in text
assert '1. One\n2. Two' in text and '### Links\n- x' in text
def test_history_lines():
assert tt.history_lines({'last_7d': 0, 'last_30d': 0}, NOW) == ['- First time in the last 30 days.']
lines = tt.history_lines({'last_7d': 1, 'last_30d': 2, 'last_first_seen': NOW - timedelta(hours=26),
'last_resolved_at': NOW - timedelta(hours=25, minutes=55)}, NOW)
assert lines[0].startswith('- Happened 1 time in the last 7 days and 2')
assert lines[1] == '- Last cleared 2026-11-09 16:05 UTC (25 h 55 min ago), after 5 min.'
def test_recovery_comment():
text = tt.recovery_comment({'event_type': 'interface_down', 'target_name': 'large1', 'target_detail': 'eno1',
'first_seen': NOW - timedelta(minutes=7), 'resolved_at': NOW,
'consecutive_failures': 3}, NOW, 'https://g/device/large1')
assert text.startswith('**✅ Cleared**: interface down on large1 eno1.')
assert '- **Lasted:** 7 min (first seen 2026-11-10 17:53 UTC)' in text
assert '- **Gandalf:** https://g/device/large1' in text
assert 'leaves this ticket open' in text
+149
View File
@@ -0,0 +1,149 @@
"""Markdown for the tickets Gandalf opens in Tinker Tickets, and the notes it adds to them.
Pure formatting only: monitor.py gathers the facts and passes them in.
"""
from datetime import datetime, timezone
from typing import List, Optional, Sequence, Tuple
from urllib.parse import quote
Section = Tuple[str, List[str]]
def duration(seconds) -> str:
seconds = max(0, int(seconds))
if seconds < 60:
return f'{seconds} s'
minutes = seconds // 60
if minutes < 60:
return f'{minutes} min'
hours, minutes = divmod(minutes, 60)
if hours < 48:
return f'{hours} h {minutes} min' if minutes else f'{hours} h'
days, hours = divmod(hours, 24)
return f'{days} d {hours} h' if hours else f'{days} d'
def utc(dt: Optional[datetime]) -> str:
"""Database times are naive UTC."""
if dt is None:
return 'unknown'
if dt.tzinfo is not None:
dt = dt.astimezone(timezone.utc).replace(tzinfo=None)
return dt.strftime('%Y-%m-%d %H:%M UTC')
def speed(mbps) -> str:
if not mbps:
return ''
mbps = int(mbps)
if mbps >= 1000:
gbps = mbps / 1000
return f'{gbps:g} Gbps'
return f'{mbps} Mbps'
def device_url(base: str, name: str, port_idx=None) -> str:
if not base or not name:
return ''
url = f'{base}/device/{quote(name, safe="")}'
return f'{url}#port-{port_idx}' if port_idx not in (None, '') else url
def history_url(base: str, target: str) -> str:
return f'{base}/events?target={quote(target, safe="")}&range=90d' if base and target else ''
def cutsheet_url(base: str, cable_id) -> str:
return f'{base}/cutsheet#cable-{cable_id}' if base else ''
def link(text: str, url: str) -> str:
return f'[{text}]({url})' if url else text
def port_text(switch: str, port_name: str, port: dict) -> str:
"""'8SFP-AGG port 7 "Large1": up at 10 Gbps, DAC'"""
idx = port.get('port_idx')
text = f'{switch} port {idx}' if idx is not None else f'{switch} {port_name}'
if port_name and idx is not None and port_name != f'Port {idx}':
text += f' "{port_name}"'
state = []
if port.get('up'):
rate = speed(port.get('speed_mbps'))
state.append(f'up at {rate}' if rate else 'up')
else:
state.append('down')
if port.get('poe_power'):
state.append(f'PoE {float(port["poe_power"]):.1f} W')
sfp = port.get('sfp') or {}
if sfp.get('found'):
kind = 'DAC' if sfp.get('kind') == 'dac' else sfp.get('kind', '')
module = ' '.join(p for p in (kind, sfp.get('part', '')) if p)
if module:
state.append(f'module: {module}')
if sfp.get('rx_fault') or sfp.get('tx_fault'):
state.append('**module reports a fault**')
return f'{text}: {", ".join(state)}'
def description(heading: str, facts: Sequence[Tuple[str, str]], summary: str,
checks: Sequence[str], sections: Sequence[Section] = (), recovers: bool = True) -> str:
lines = [f'## {heading}', '']
lines += [f'- **{key}:** {value}' for key, value in facts if value not in (None, '')]
lines += ['', summary.strip()]
if checks:
lines += ['', '### What to check']
lines += [f'{n}. {check}' for n, check in enumerate(checks, 1)]
for title, body in sections:
if body:
lines += ['', f'### {title}'] + list(body)
if recovers:
lines += ['', '_Opened automatically by Gandalf. It adds a note here when the problem clears, '
'but leaves closing the ticket to you._']
else:
lines += ['', '_Reported automatically by Gandalf. This already happened, so there will be no '
'recovery note._']
return '\n'.join(lines)
def history_lines(stats: dict, now: datetime) -> List[str]:
week, month = stats.get('last_7d', 0), stats.get('last_30d', 0)
if not month:
lines = ['- First time in the last 30 days.']
else:
lines = [f'- Happened {week} time{"s" if week != 1 else ""} in the last 7 days '
f'and {month} in the last 30 (not counting this one).']
if week >= 3:
lines.append('- **This keeps coming back**: look for a cause that comes and goes '
'(a loose cable, a failing optic, power, or a scheduled job).')
first, resolved = stats.get('last_first_seen'), stats.get('last_resolved_at')
if resolved:
took = duration((resolved - first).total_seconds()) if first else 'unknown'
ago = duration((now - resolved).total_seconds())
lines.append(f'- Last cleared {utc(resolved)} ({ago} ago), after {took}.')
return lines
ISSUE_FIXED = {
'cable_mismatch': 'The cable now matches the cutsheet (or the cutsheet was updated).',
'monitor_source_unavailable': 'Prometheus is returning data again; host and interface alerting has resumed.',
}
def recovery_comment(event: dict, now: datetime, url: str = '') -> str:
"""The note Gandalf adds to a ticket when its alert clears."""
first, resolved = event.get('first_seen'), event.get('resolved_at') or now
target = event['target_name'] + (f' {event["target_detail"]}' if event.get('target_detail') else '')
lines = [f'**✅ Cleared**: {event["event_type"].replace("_", " ")} on {target}.', '']
lines.append(f'- **Cleared:** {utc(resolved)}')
if first:
lines.append(f'- **Lasted:** {duration((resolved - first).total_seconds())} '
f'(first seen {utc(first)})')
if event.get('consecutive_failures'):
lines.append(f'- **Failed checks:** {event["consecutive_failures"]}')
if url:
lines.append(f'- **Gandalf:** {url}')
lines += ['', ISSUE_FIXED.get(event['event_type'], 'Gandalf no longer sees the problem.')]
lines += ['', '_Gandalf leaves this ticket open. Close it once you know why it happened, '
'or leave it open if it needs follow-up._']
return '\n'.join(lines)