Compare commits

..
Author SHA1 Message Date
jaredandClaude Opus 5.5 379e3be365 WAN tickets: one per incident, notice UniFi updates, maintenance button
Lint / JS (eslint) (push) Successful in 42s
Lint / Python (flake8) (push) Successful in 1m35s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 1m3s
Test / Python Tests (pytest) (push) Successful in 1m5s
Lint / Python (flake8) (pull_request) Successful in 58s
Lint / JS (eslint) (pull_request) Successful in 8s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 1m1s
Test / Python Tests (pytest) (pull_request) Successful in 1m10s
A UniFi Network update on 2026-11-12 dropped the UDM's WAN, Smart Power
power-cycled the modem, and Gandalf opened two tickets for the one event.

- Modem and WAN reports from one check cycle open a single ticket, led by
  the most urgent, with the rest listed under "Also seen in this incident".
  A report while the modem has an open alert, or one that ended under 30
  minutes ago, becomes a comment on that ticket instead.
- UniFi update log entries (NETWORK_UPDATED, FIRMWARE_UPDATED_V2 and the
  like) add a note to modem/WAN tickets from 20 minutes before the update
  to 15 after, and hold new modem/WAN tickets for 15 minutes.
- A Maintenance button for admins starts a global suppression for 30 min
  to 4 hours, with a banner on every page and End now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 12:53:22 -05:00
jared 6b03fe7b44 Merge pull request 'Mark host interfaces as unused' (#35) from unused-interfaces into main
Lint / JS (eslint) (push) Successful in 8s
Lint / Python (flake8) (push) Successful in 54s
Lint / Notify on failure (push) Skipped
Security / Python Security (bandit) (push) Successful in 52s
Lint / Deploy (push) Successful in 4s
Test / Python Tests (pytest) (push) Successful in 1m15s
2026-11-11 17:50:14 -05:00
jaredandClaude Opus 5.5 3834546f38 Mark host interfaces as unused
Lint / JS (eslint) (push) Successful in 44s
Lint / Python (flake8) (push) Successful in 1m40s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 1m18s
Lint / JS (eslint) (pull_request) Successful in 10s
Security / Python Security (bandit) (pull_request) Successful in 1m13s
Test / Python Tests (pytest) (pull_request) Successful in 1m11s
Lint / Python (flake8) (pull_request) Successful in 50s
Test / Python Tests (pytest) (push) Successful in 1m1s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Ports that are never meant to be up (compute-storage-01 nic1-3, large1's
spare NIC and USB adapter) kept their hosts "degraded" and were the source
of tonight's false alerts. An admin can now mark an interface unused from
the device page, with an optional reason. Unused interfaces raise no
link-down alerts, don't count toward the cluster-wide alert or the host's
status, and show greyed as "unused" on the dashboard, device page and rack.
Marking one closes any open link-down alert on it without a recovery note.

New table unused_interfaces, created with the cutsheet tables.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 17:19:45 -05:00
jared aeb7a89e81 Merge pull request 'Rack: Flex Mini on the NVR, coloured Sliger fronts, UniFi brush panel' (#32) from loose-gear-on-top into main
Lint / Python (flake8) (push) Successful in 47s
Lint / JS (eslint) (push) Successful in 7s
Lint / Notify on failure (push) Skipped
Security / Python Security (bandit) (push) Successful in 2m26s
Test / Python Tests (pytest) (push) Successful in 2m44s
Lint / Deploy (push) Successful in 7s
2026-11-11 02:09:38 -05:00
jared 9bdf5d9eed Merge pull request 'Tickets: don't adopt a Closed ticket Tinker couldn't reopen' (#34) from tinker-create-action into main
Lint / Python (flake8) (push) Successful in 47s
Lint / JS (eslint) (push) Successful in 9s
Lint / Notify on failure (push) Skipped
Security / Python Security (bandit) (push) Successful in 1m39s
Test / Python Tests (pytest) (push) Successful in 1m49s
Lint / Deploy (push) Successful in 24s
2026-11-11 02:01:10 -05:00
jaredandClaude Opus 5.5 cbdae54d70 Rack: brush panel opening runs nearly the full width between the ears
Lint / Python (flake8) (push) Successful in 58s
Lint / JS (eslint) (push) Successful in 7s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 1m6s
Test / Python Tests (pytest) (push) Successful in 1m7s
Lint / JS (eslint) (pull_request) Successful in 9s
Lint / Python (flake8) (pull_request) Successful in 1m51s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 1m45s
Test / Python Tests (pytest) (pull_request) Successful in 1m1s
As in Ubiquiti's product image and the rack photo.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 01:57:19 -05:00
jaredandClaude Opus 5.5 36cbdd061d Rack: brush panel holds up at rack size
Lint / Python (flake8) (push) Successful in 1m2s
Lint / JS (eslint) (push) Successful in 10s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Lint / JS (eslint) (pull_request) Successful in 7s
Lint / Python (flake8) (pull_request) Successful in 1m26s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (push) Successful in 1m55s
Test / Python Tests (pytest) (push) Successful in 1m24s
Security / Python Security (bandit) (pull_request) Successful in 49s
Test / Python Tests (pytest) (pull_request) Successful in 1m10s
The bristles were hundreds of hard strands, which aliased into flat
stripes at normal rack size. Each row is now shaded root to tip with a
faint strand texture over it, so the panel looks the same at any size.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 01:56:24 -05:00
jaredandClaude Opus 5.5 6e6150f9b9 Rack: rack handles on the Sliger ears
Lint / JS (eslint) (push) Successful in 13s
Lint / Python (flake8) (push) Successful in 1m50s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 1m17s
Test / Python Tests (pytest) (push) Successful in 1m39s
Lint / Python (flake8) (pull_request) Successful in 1m28s
Lint / JS (eslint) (pull_request) Successful in 17s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 1m26s
Test / Python Tests (pytest) (pull_request) Successful in 1m6s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 01:55:12 -05:00
jaredandClaude Opus 5.5 7408b1feca Rack: Sliger fronts in red, white and blue, and the UniFi brush panel
Lint / JS (eslint) (push) Successful in 7s
Lint / Python (flake8) (push) Successful in 53s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 1m0s
Lint / Python (flake8) (pull_request) Successful in 52s
Test / Python Tests (pytest) (push) Successful in 1m7s
Lint / JS (eslint) (pull_request) Successful in 10s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 1m52s
Test / Python Tests (pytest) (pull_request) Successful in 1m51s
From the rack photo on the wiki, the three Sligers have their coloured
perforated fronts fitted, not the faceplate-off view: Front drawings
"Sliger CX4712 4U, red/white/blue front" draw the hex-perforated panel
(about 20 rows of holes, the drive cages faint behind it), black rack ears,
the power button with Gandalf's status ring, and the USB recess.

UniFi UACC-Rack-Panel-Brush-1U is drawn from Ubiquiti's product images:
silver face, ear plates with rounded slots, a recessed chamfered frame and
two rows of bristles meeting at a seam.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 01:51:29 -05:00
jared 64ef4222d0 Merge remote-tracking branch 'origin/main' into loose-gear-on-top 2026-11-11 01:48:37 -05:00
jaredandClaude Opus 5.5 3bc8df6dc7 Tests: wrap a long line
Lint / JS (eslint) (push) Successful in 25s
Lint / Python (flake8) (push) Successful in 1m53s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 1m28s
Lint / Python (flake8) (pull_request) Successful in 58s
Test / Python Tests (pytest) (push) Successful in 1m9s
Lint / JS (eslint) (pull_request) Successful in 6s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 1m1s
Test / Python Tests (pytest) (pull_request) Successful in 1m6s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 01:08:52 -05:00
jaredandClaude Opus 5.5 d8e827c88b Rack: gear resting on other gear, and live desk switches on shelves
Lint / Python (flake8) (push) Failing after 47s
Lint / JS (eslint) (push) Successful in 17s
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 1m26s
Lint / Notify on failure (push) Successful in 7s
Test / Python Tests (pytest) (push) Successful in 2m2s
A USW Flex Mini lies on top of the NVR. A shelf item with Front drawing
"No shelf: gear resting on what is below" draws no tray or ears, so its
contents sit directly on the item under it.

UniFi desk switches on a shelf (Flex Mini, Flex) are drawn with their live
faceplate, scaled to their real width from Ubiquiti's tech specs (107 mm and
122.5 mm), with working ports. faceplateSvg() returns a faceplate's SVG on
its own so it can be placed inside the shelf drawing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 01:08:42 -05:00
17 changed files with 966 additions and 76 deletions

No files matched your search

+44 -2
View File
@@ -154,7 +154,14 @@ The rack front, drawn from the rack units set on cutsheet items, top first.
enp2s0 (left) and enp3s0 (right), and the status glow follows the host. The "ZimaBoard 832 +
ConnectX-3" drawing adds the single-port SFP+ card lying flat in the side PCIe slot, as in the rack,
with a DAC attached; its round dot is Gandalf's link mark for enp1s0 (the slot is PCI bus 1), not an
LED on the card. Anything else is a labelled box.
LED on the card. UniFi desk switches (Flex Mini, Flex) on a shelf are drawn with their live faceplate
at their real width. Anything else is a labelled box.
- **More drawn fronts**: the Sliger CX4712 with its red, white or blue perforated front fitted (as in
the rack photo, drive cages faint behind the mesh), and the UniFi brush panel
(UACC-Rack-Panel-Brush-1U, from Ubiquiti's product images).
- **Gear resting on other gear**: a shelf item with Front drawing "No shelf: gear resting on what is
below" draws no tray, so a switch lying on top of the NVR is a "shelf" in the unit above the NVR whose
contents sit directly on it.
- **Racks**: admins name a rack and set its height with **Rack settings** on the page, and add more with
**+ Rack** (a tab per rack). Renaming a rack moves its items; an empty rack can be deleted. Each item
has a rack (empty means the first). Until a rack is saved, racks come from config:
@@ -302,6 +309,12 @@ can view it; only the `admin` group can edit, and every change is logged with wh
- Suppression history (last 50)
- Available targets reference grid (all known hosts + interfaces)
**Maintenance**: admins get a **Maintenance** button in the header. It starts a global suppression for
30 min, 1, 2 or 4 hours with a reason (by default "Maintenance"). While it runs, every page shows a banner
saying when tickets resume, with **End now** for admins. Gandalf keeps checking and recording alerts, and
anything still failing when maintenance ends gets a ticket as usual. Use it before a UniFi update, a switch
swap or recabling.
---
## Alert Logic
@@ -345,6 +358,18 @@ whatever Gandalf knows around the problem:
Titles keep the `[target][auto][production]...` tag format, which Tinker uses to spot duplicates.
### Unused interfaces
An admin can mark a host interface as unused on its device page (**Mark unused** on a down interface,
with an optional reason; **In use again** to undo). An unused interface:
- never raises a link-down alert, and can't count toward the cluster-wide alert;
- doesn't count toward the host's status, so a host whose only down ports are unused shows as up;
- shows as "unused" (greyed, dashed dot) on the dashboard, the device page and the rack drawings;
- has any open link-down alert closed quietly when it is marked (no recovery note: nothing recovered).
Marks are stored in `unused_interfaces` (host, interface, reason, who, when).
### Recovery notes
When an alert that opened a ticket clears, the monitor adds a comment to that ticket: when it cleared, how
@@ -367,7 +392,23 @@ Outages can be over between polls: two real ones lasted about two minutes agains
So WAN outages and modem restarts also come from the UniFi system log, and a drop in the modem's uptime
records a restart even if nothing logged it. A single short outage only appears in the panel's history.
To silence modem alerts during maintenance, create a `host` suppression with the modem's `name`.
**One ticket per incident.** The modem and WAN reports from one check cycle open a single ticket, led by
the most urgent (a P2 "WAN modem unreachable" leads, and an RPDU power-cycle seen in the same cycle is listed
under "Also seen in this incident"). A report that comes while the modem has an open alert, or one that
ended under 30 minutes ago, becomes a comment on that alert's ticket instead of a new ticket. If the
comment can't be posted, the report gets its own ticket as before.
**UniFi updates.** UniFi logs `NETWORK_UPDATED` (Network app) and `FIRMWARE_UPDATED_V2` (device firmware)
once an update has finished. Updating the Network app can drop the UDM's WAN, and Smart Power may then
power-cycle a modem that was fine. When Gandalf reads one of these entries it:
- comments on modem and WAN tickets first seen from 20 minutes before the update to 15 minutes after,
naming the update (for example "UniFi Network 11.0.97 to 11.0.99") as the likely cause;
- opens no new modem or WAN tickets for 15 minutes after the update. A problem still there after that gets
its ticket as usual; a power-cycle report during the hold only goes into the event history.
To silence modem alerts during maintenance, create a `host` suppression with the modem's `name`, or use
**Maintenance** (below) to pause every ticket.
### Baseline Tracking
@@ -746,6 +787,7 @@ Take LED colors from the model's quick start guide. See the comment at the top o
- `cutsheet_items` gained `rack`, `shelf`, `device`, `mount`, `chassis`, `facing` and `slot` columns; they are added automatically on first use
- `cutsheet_racks`: racks saved on the rack page (name, height, notes)
- `unused_interfaces`: host interfaces marked unused (no alerts, not counted in host status)
- `device_links`: each device's inventory URL, by device name; created with the cutsheet tables
- `network_events`: one row per active event; `resolved_at` is set when recovered, and
`recovery_notified` once the recovery note is on its ticket (column added automatically at monitor startup)
+53 -3
View File
@@ -60,6 +60,19 @@ def inject_config():
}
@app.context_processor
def inject_maintenance():
"""The active global suppression (maintenance), shown as a banner on every page."""
try:
active = [s for s in db.get_active_suppressions() if s.get('target_type') == 'all']
except Exception as e:
logger.error(f'Reading maintenance state failed: {e}')
active = []
# Of several, the one that ends last decides when tickets resume; no expiry beats any
active.sort(key=lambda s: (s.get('expires_at') is None, s.get('expires_at') or ''))
return {'maintenance': active[-1] if active else None}
# In-memory diagnostic job store { job_id: { status, result, created_at } }
_diag_jobs: dict = {}
_diag_lock = threading.Lock()
@@ -528,6 +541,12 @@ def api_device(name: str):
logger.error(f'Cutsheet unavailable for {name}: {e}')
detail['cutsheet'] = []
detail['inventory'] = _inventory_for(name, detail.get('model') or '')
try:
_ensure_cutsheet_schema()
detail['unused_interfaces'] = db.get_unused_interfaces().get(name, {})
except Exception as e:
logger.error(f'Unused interfaces unavailable for {name}: {e}')
detail['unused_interfaces'] = {}
return jsonify(detail)
@@ -1114,8 +1133,8 @@ def _physical_interfaces(name: str, snapshot: dict) -> list:
Drawn fronts light one RJ45 per NIC in this order (first NIC on the leftmost jack).
"""
ifaces = ((snapshot.get('hosts') or {}).get(name) or {}).get('interfaces') or {}
return [{'name': n, 'up': str(state).lower() == 'up'} for n, state in sorted(ifaces.items())
if not n.startswith(_VIRTUAL_IFACE)]
return [{'name': n, 'up': str(state).lower() == 'up', 'unused': str(state).lower() == 'unused'}
for n, state in sorted(ifaces.items()) if not n.startswith(_VIRTUAL_IFACE)]
def _device_status(name: str, snapshot: dict) -> str:
@@ -1210,7 +1229,8 @@ def api_rack():
'status': _device_status(dev(c), snapshot) if dev(c) in known else '',
'model': next((u.get('model') for u in snapshot.get('unifi') or [] if u.get('name') == dev(c)), '')
or '', 'notes': c.get('notes') or '', 'chassis': c.get('chassis') or '',
'slot': c.get('slot'), 'interfaces': _physical_interfaces(dev(c), snapshot)}
'slot': c.get('slot'), 'interfaces': _physical_interfaces(dev(c), snapshot),
'device': dev(c), 'ports': (switches.get(dev(c)) or {}).get('ports') or {}}
for c in sorted(on_shelf.get(name, []), key=lambda c: (c.get('slot') or 999, c['name'].lower()))
]
out.append(entry)
@@ -1256,6 +1276,36 @@ def api_cutsheet_delete_rack(rack_id: int):
return jsonify({'success': True})
# ---------------------------------------------------------------------------
# Unused interfaces
# ---------------------------------------------------------------------------
_IFACE_RE = re.compile(r'^[A-Za-z0-9_.:@-]{1,64}$')
@app.route('/api/device/<path:name>/interfaces/<iface>/unused', methods=['PUT', 'DELETE'])
@require_auth
@require_admin
@_cutsheet_write
def api_interface_unused(name: str, iface: str):
"""Mark a host interface unused (PUT {reason}) or in use again (DELETE).
Unused interfaces raise no link-down alerts and don't count toward the host's status;
marking one quietly closes any link-down alert it has open.
"""
if len(name) > 255 or not _IFACE_RE.match(iface):
raise cutsheet.ValidationError('Not a valid host or interface name')
user = _get_user()['username']
if request.method == 'DELETE':
found = db.clear_unused_interface(name, iface)
logger.info(f'{user} marked {name} {iface} in use again')
return jsonify({'success': True, 'changed': found})
reason = str(_json_body().get('reason') or '').strip()[:255]
db.set_unused_interface(name, iface, reason, user)
logger.info(f'{user} marked {name} {iface} unused: {reason or "(no reason)"}')
return jsonify({'success': True})
@app.route('/api/cutsheet/cables', methods=['POST'])
@require_auth
@require_admin
+6 -1
View File
@@ -37,10 +37,15 @@ CHASSIS = {
'uacc-patch-24': 'UniFi Patch Panel Blank 24 (UACC-Rack-Panel-Patch-Blank-24)',
'usw-pro-max-24-poe': 'UniFi Switch Pro Max 24 PoE (until UniFi reports it)',
'usw-pro-max-48-poe': 'UniFi Switch Pro Max 48 PoE (until UniFi reports it)',
'sliger-cx4712': 'Sliger CX4712 4U (10-bay)',
'sliger-cx4712': 'Sliger CX4712 4U (10-bay, faceplate off)',
'sliger-cx4712-red': 'Sliger CX4712 4U, red front',
'sliger-cx4712-white': 'Sliger CX4712 4U, white front',
'sliger-cx4712-blue': 'Sliger CX4712 4U, blue front',
'uacc-brush-1u': 'UniFi Rack Panel Brush 1U (UACC-Rack-Panel-Brush-1U)',
'rosewill-rsv-l4500u': 'Rosewill RSV-L4500U 4U (15-bay)',
'zimaboard-832': 'IceWhale ZimaBoard 832 (shelf gear)',
'zimaboard-832-cx3': 'IceWhale ZimaBoard 832 + ConnectX-3 SFP+ in PCIe (shelf gear)',
'resting': 'No shelf: gear resting on what is below (for a shelf item)',
'server-1u': 'Generic 1U server',
'server-2u': 'Generic 2U server',
'server-4u': 'Generic 4U server',
+83 -1
View File
@@ -4,7 +4,7 @@ import logging
import threading
from contextlib import contextmanager
from datetime import datetime, timedelta, timezone
from typing import Optional
from typing import List, Optional
import pymysql
import pymysql.cursors
@@ -198,6 +198,35 @@ def set_ticket_id(event_id: int, ticket_id: str) -> None:
)
def get_incident_ticket(target_name: str, within_s: int) -> Optional[str]:
"""The ticket of *target_name*'s newest event that is still open or ended within *within_s*."""
with get_conn() as conn:
with conn.cursor() as cur:
cur.execute(
"""SELECT ticket_id FROM network_events
WHERE target_name=%s AND ticket_id IS NOT NULL AND ticket_id <> ''
AND (resolved_at IS NULL OR resolved_at >= NOW() - INTERVAL %s SECOND)
ORDER BY first_seen DESC, id DESC LIMIT 1""",
(target_name, int(within_s)),
)
row = cur.fetchone()
return row['ticket_id'] if row else None
def get_ticket_ids_between(target_name: str, start: datetime, end: datetime) -> List[str]:
"""Tickets of *target_name*'s events first seen between *start* and *end* (UTC), oldest first."""
with get_conn() as conn:
with conn.cursor() as cur:
cur.execute(
"""SELECT ticket_id, MIN(first_seen) AS first FROM network_events
WHERE target_name=%s AND ticket_id IS NOT NULL AND ticket_id <> ''
AND first_seen BETWEEN %s AND %s
GROUP BY ticket_id ORDER BY first""",
(target_name, start, end),
)
return [r['ticket_id'] for r in cur.fetchall()]
def get_active_events(limit: int = 200, offset: int = 0) -> list:
with get_conn() as conn:
with conn.cursor() as cur:
@@ -738,6 +767,15 @@ _CUTSHEET_SCHEMA = (
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
UNIQUE KEY uq_rack_name (name)
) ENGINE=InnoDB""",
# Host interfaces an admin has marked as not in use: no alerts, not counted in host status
"""CREATE TABLE IF NOT EXISTS unused_interfaces (
host VARCHAR(255) NOT NULL,
iface VARCHAR(64) NOT NULL,
reason VARCHAR(255) NOT NULL DEFAULT '',
set_by VARCHAR(255) NOT NULL,
set_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (host, iface)
) ENGINE=InnoDB""",
# Where a device lives in the asset inventory (HomeBox), set by hand on its Gandalf page
"""CREATE TABLE IF NOT EXISTS device_links (
name VARCHAR(255) NOT NULL PRIMARY KEY,
@@ -1107,3 +1145,47 @@ def cutsheet_delete_rack(rack_id: int, user: str) -> bool:
cur.execute('DELETE FROM cutsheet_racks WHERE id=%s', (rack_id,))
_cs_log(cur, user, 'rack', rack_id, 'delete', f'Removed rack {before["name"]}', _cs_jsonable(before), None)
return True
# ---------------------------------------------------------------------------
# Unused interfaces: ports an admin says aren't meant to be up
# ---------------------------------------------------------------------------
def get_unused_interfaces() -> dict:
"""{host: {iface: {'reason', 'set_by', 'set_at'}}}"""
with get_conn() as conn:
with conn.cursor() as cur:
cur.execute('SELECT host, iface, reason, set_by, set_at FROM unused_interfaces')
out: dict = {}
for r in cur.fetchall():
at = r['set_at'].isoformat() + 'Z' if hasattr(r['set_at'], 'isoformat') else r['set_at']
out.setdefault(r['host'], {})[r['iface']] = {'reason': r['reason'], 'set_by': r['set_by'],
'set_at': at}
return out
def set_unused_interface(host: str, iface: str, reason: str, user: str) -> None:
"""Mark an interface unused and quietly close any link-down alert it has open.
The alert is resolved without a recovery note: nothing recovered, the port just isn't used.
"""
with get_conn() as conn:
with conn.cursor() as cur:
cur.execute(
"""INSERT INTO unused_interfaces (host, iface, reason, set_by) VALUES (%s, %s, %s, %s)
ON DUPLICATE KEY UPDATE reason=VALUES(reason), set_by=VALUES(set_by)""",
(host, iface, reason, user),
)
cur.execute(
"""UPDATE network_events SET resolved_at=NOW(), recovery_notified=1, last_seen=last_seen
WHERE event_type='interface_down' AND target_name=%s AND target_detail=%s
AND resolved_at IS NULL""",
(host, iface),
)
def clear_unused_interface(host: str, iface: str) -> bool:
with get_conn() as conn:
with conn.cursor() as cur:
cur.execute('DELETE FROM unused_interfaces WHERE host=%s AND iface=%s', (host, iface))
return cur.rowcount > 0
+25
View File
@@ -242,6 +242,31 @@ LOG_WAN_RESTORED = 'NETWORK_WAN_RESTORED_2'
LOG_WAN_UNSTABLE = 'NETWORK_WAN_FAILED_MULTIPLE_TIMES_2'
LOG_MODEM_RESTARTED = 'SMART_POWER_MODEM_RESTARTED'
WAN_LOG_KEYS = (LOG_WAN_FAILED, LOG_WAN_RESTORED, LOG_WAN_UNSTABLE, LOG_MODEM_RESTARTED)
# An app or firmware update finishing. The UDM often drops its WAN while the Network app
# updates, and Smart Power may then power-cycle a modem that was fine.
LOG_NETWORK_UPDATED = 'NETWORK_UPDATED'
LOG_FIRMWARE_UPDATED = 'FIRMWARE_UPDATED_V2'
_UPDATE_KEY_RE = re.compile(r'[A-Z_]+_(?:UPDATED|UPGRADED)(?:_V\d+)?')
def is_update_key(key: str) -> bool:
"""True for log keys that mean an update finished (other app names follow the same pattern)."""
return key in (LOG_NETWORK_UPDATED, LOG_FIRMWARE_UPDATED) or (
bool(_UPDATE_KEY_RE.fullmatch(key or '')) and 'AVAILABLE' not in key
and not key.startswith('MADE_CHANGES'))
def wanted_log_key(key: str) -> bool:
return key in WAN_LOG_KEYS or is_update_key(key)
def update_label(event: dict) -> str:
"""'UniFi Network 11.0.97 to 11.0.99' from an update log event."""
what = event.get('app') or event.get('device') or 'UniFi'
before, after = event.get('previous') or '', event.get('version') or ''
if before and after:
return f'{what} {before} to {after}'
return f'{what} {after}'.strip()
_HISTORY_LIMIT = 50
# The RPDU's log entry and the gateway's calculated boot time can differ by this much.
+158 -28
View File
@@ -13,7 +13,8 @@ import re
import shlex
import statistics
import time
from datetime import datetime, timezone
from contextlib import contextmanager
from datetime import datetime, timedelta, timezone
from typing import Dict, List, Optional
from urllib.parse import urljoin, urlparse
@@ -192,8 +193,12 @@ class UnifiClient:
logger.error(f'UniFi client list error: {e}')
return []
def get_system_events(self, since_ms: int, keys: tuple) -> Optional[List[dict]]:
"""Return system log entries with the given keys, or None if the log is unavailable."""
def get_system_events(self, since_ms: int, keys) -> Optional[List[dict]]:
"""Return system log entries with the given keys, or None if the log is unavailable.
keys: a tuple of log keys, or a function that takes a key and says whether to keep it.
"""
wanted = keys if callable(keys) else keys.__contains__
url = f'{self.base_url}/proxy/network/v2/api/site/{self.site_id}/system-log/all'
found: List[dict] = []
try:
@@ -210,19 +215,29 @@ class UnifiClient:
if not items:
break
for e in items:
if e.get('key') in keys and e.get('timestamp', 0) >= since_ms:
if wanted(e.get('key') or '') and e.get('timestamp', 0) >= since_ms:
params = e.get('parameters') or {}
found.append({
'key': e['key'], 'ts': e['timestamp'] / 1000,
'device': (params.get('DEVICE') or {}).get('name', ''),
'wan': (params.get('WAN_ID') or {}).get('name', '')
or str((params.get('WAN_ID') or {}).get('id', '')),
**self._update_fields(params),
})
return found
except Exception as e:
logger.error(f'UniFi system log error: {e}')
return None
@staticmethod
def _update_fields(params: dict) -> dict:
"""The app and versions of an update entry; nothing for other entries."""
def text(key):
value = params.get(key)
return str(value.get('name') or value.get('id') or '') if isinstance(value, dict) else str(value or '')
fields = {'app': text('APPLICATION'), 'version': text('VERSION'), 'previous': text('PREVIOUS_VERSION')}
return {k: v for k, v in fields.items() if v}
def get_devices(self) -> Optional[List[dict]]:
"""Return list of UniFi devices, or None if the controller is unreachable."""
try:
@@ -965,6 +980,13 @@ CLUSTER_NAME = 'proxmox-cluster'
DEFAULT_PUBLIC_URL = 'https://gandalf.lotusguild.org'
# Reports of something already over: the ticket is the whole story, no recovery note
ONE_SHOT_EVENTS = ('modem_power_cycled', 'wan_unstable')
# Modem and WAN reports this close together are one incident, so they share one ticket
WAN_INCIDENT_WINDOW_S = 1800
# UniFi logs an update when it finishes, after any WAN trouble it caused. So tickets opened
# this long before the entry get a note, and new modem/WAN tickets wait this long after it.
UPDATE_LOOKBACK_S = 1200
UPDATE_HOLD_S = 900
UNIFI_UPDATE_STATE = 'unifi_update'
# Ports a host or device was last seen on, kept so a ticket for a dead host
# (whose link, and so its LLDP entry, is gone) can still say where it plugs in
KNOWN_UPLINKS_STATE = 'known_uplinks'
@@ -1045,6 +1067,16 @@ class NetworkMonitor:
def _hostname(self, instance: str) -> str:
return self._instance_map.get(instance, instance.split(':')[0])
def _unused_for(self, host: str) -> dict:
return (getattr(self, '_unused', None) or {}).get(host) or {}
def _load_unused(self) -> None:
"""Reload the interfaces marked unused (once a cycle; keeps the last list on error)."""
try:
self._unused = db.get_unused_interfaces()
except Exception as e:
logger.error(f'Could not load unused interfaces: {e}')
# ------------------------------------------------------------------
# Tickets: context, and notes when an alert clears
# ------------------------------------------------------------------
@@ -1229,7 +1261,10 @@ class NetworkMonitor:
new_baseline.setdefault(host, {})
host_has_regression = False
unused = self._unused_for(host)
for iface, is_up in ifaces.items():
if iface in unused:
continue # marked unused: no alerts, and it can't count as a regression
prev = baseline.get(host, {}).get(iface) # 'up', 'initial_down', or None
if is_up:
@@ -1530,14 +1565,95 @@ class NetworkMonitor:
event_type, severity, 'modem', name, detail, f'{description} ({_now_utc()})',
)
if not sup and consec >= self.modem_fail_thresh and not ticket_id:
self._open_ticket(
event_id, event_type, name, detail,
title=f'[{name}][auto][production][issue][network][cluster-wide] {heading}',
heading='WAN Modem Alert',
facts=[('Modem', f'{name} ({self.modem.base_url})'), ('Detected', _now_utc()),
('Consecutive check failures', str(consec))],
summary=description, checks=[advice], priority=priority,
)
self._queue_wan_ticket({
'event_id': event_id, 'event_type': event_type, 'detail': detail,
'title': f'[{name}][auto][production][issue][network][cluster-wide] {heading}',
'name': heading, 'heading': 'WAN Modem Alert',
'facts': [('Modem', f'{name} ({self.modem.base_url})'), ('Detected', _now_utc()),
('Consecutive check failures', str(consec))],
'summary': description, 'checks': [advice], 'priority': priority, 'recovers': True,
})
# ------------------------------------------------------------------
# Modem and WAN tickets: one per incident, held around UniFi updates
# ------------------------------------------------------------------
@contextmanager
def _wan_batch(self):
"""Collect the cycle's modem and WAN tickets so related ones are opened together."""
self._wan_pending = []
try:
yield
finally:
pending, self._wan_pending = self._wan_pending, None
if pending:
self._open_wan_tickets(pending)
def _queue_wan_ticket(self, item: dict) -> None:
held = self._update_hold()
if held:
logger.info(f'Not opening a ticket for {item["event_type"]} yet: {held} just finished')
return
if getattr(self, '_wan_pending', None) is not None:
self._wan_pending.append(item)
else:
self._open_wan_tickets([item])
def _update_hold(self) -> Optional[str]:
"""The update that modem and WAN tickets are waiting out, if any."""
raw = db.get_state(UNIFI_UPDATE_STATE)
if not raw:
return None
try:
update = json.loads(raw)
except ValueError:
return None
if update.get('hold_until', 0) <= time.time():
return None
return update.get('label') or 'a UniFi update'
def _open_wan_tickets(self, items: List[dict]) -> None:
"""Comment on the incident's ticket if one is recent, else open one ticket for all of *items*."""
# Most urgent first; at equal priority an ongoing problem leads over a past event
items = sorted(items, key=lambda i: (int(i['priority']), not i['recovers']))
incident = db.get_incident_ticket(self.modem_name, WAN_INCIDENT_WINDOW_S)
if incident:
left = []
for item in items:
text = ticket_text.incident_comment(item['name'], item['summary'], item['checks'], _now_utc())
if self.tickets.comment(incident, text):
db.set_ticket_id(item['event_id'], incident)
else:
left.append(item) # comments failing must not lose the report
items = left
if not items:
return
first, others = items[0], items[1:]
sections = []
if first['recovers']:
sections = self._ticket_sections(first['event_id'], first['event_type'], self.modem_name, first['detail'])
if others:
sections.append(('Also seen in this incident', [f'- **{o["name"]}:** {o["summary"]}' for o in others]))
desc = ticket_text.description(first['heading'], first['facts'], first['summary'], first['checks'],
sections, recovers=any(i['recovers'] for i in items))
tid = self.tickets.create(first['title'], desc, priority=first['priority'])
if tid:
for item in items:
db.set_ticket_id(item['event_id'], tid)
def _note_unifi_update(self, event: dict) -> None:
"""Hold new modem/WAN tickets for a while and explain the ones opened just before."""
label = modem_api.update_label(event)
when = datetime.fromtimestamp(event['ts'], timezone.utc)
logger.info(f'UniFi update finished: {label} at {when:%Y-%m-%d %H:%M:%S} UTC')
if event['ts'] + UPDATE_HOLD_S > time.time():
db.set_state(UNIFI_UPDATE_STATE, {'label': label, 'at': event['ts'],
'hold_until': event['ts'] + UPDATE_HOLD_S})
naive = when.replace(tzinfo=None)
tickets = db.get_ticket_ids_between(
self.modem_name, naive - timedelta(seconds=UPDATE_LOOKBACK_S), naive + timedelta(seconds=UPDATE_HOLD_S))
for tid in tickets:
self.tickets.comment(tid, ticket_text.update_comment(
label, f'{when:%Y-%m-%d %H:%M} UTC', UPDATE_LOOKBACK_S // 60, UPDATE_HOLD_S // 60))
def _process_modem_failure(self, suppressions: list, error: Exception) -> None:
"""Handle a failed poll. Link and optical events stay as they are: we can't see them."""
@@ -1677,14 +1793,13 @@ class NetworkMonitor:
name = self.modem_name
event_id, _, _, ticket_id = db.upsert_event(event_type, 'warning', 'unifi', name, '', description)
if not db.check_suppressed(suppressions, 'host', name) and not ticket_id:
tid = self.tickets.create(
f'[{name}][auto][production][issue][network][cluster-wide] {heading}',
ticket_text.description('WAN Event', [('Modem', name), ('Reported', _now_utc())],
description, [advice], recovers=False),
priority=priority,
)
if tid:
db.set_ticket_id(event_id, tid)
self._queue_wan_ticket({
'event_id': event_id, 'event_type': event_type, 'detail': '',
'title': f'[{name}][auto][production][issue][network][cluster-wide] {heading}',
'name': heading, 'heading': 'WAN Event',
'facts': [('Modem', name), ('Reported', _now_utc())],
'summary': description, 'checks': [advice], 'priority': priority, 'recovers': False,
})
db.resolve_event(event_type, name, '')
def _process_wan_log(self, suppressions: list) -> None:
@@ -1700,12 +1815,21 @@ class NetworkMonitor:
return
# After long monitor downtime, don't open tickets for stale events.
since_ms = max(int(raw_cursor) + 1, int((now - 86400) * 1000))
events = self.unifi.get_system_events(since_ms, modem_api.WAN_LOG_KEYS)
events = self.unifi.get_system_events(since_ms, modem_api.wanted_log_key)
if not events:
return # None = log unavailable, so the cursor stays put and it's retried
# Updates first: one in this batch explains (and holds) the WAN events beside it
updates = [e for e in events if modem_api.is_update_key(e['key'])]
for e in sorted(updates, key=lambda ev: ev['ts']):
try:
self._note_unifi_update(e)
except Exception as ex:
logger.error(f'Noting UniFi update failed: {ex}', exc_info=True)
history = self._load_wan_history()
actionable = modem_api.apply_wan_events(history, events)
actionable = modem_api.apply_wan_events(
history, [e for e in events if not modem_api.is_update_key(e['key'])])
# UniFi repeats "failed multiple times" per failure, so ticket it once a day at most.
unstable = [e for e in actionable if e['key'] == modem_api.LOG_WAN_UNSTABLE]
@@ -1799,8 +1923,11 @@ class NetworkMonitor:
continue # skip Prometheus instances not in config (e.g. LXC app servers)
host = self._hostname(instance)
phys = {k: v for k, v in ifaces.items()}
up_count = sum(1 for v in phys.values() if v)
total = len(phys)
# Interfaces marked unused don't count toward the host's status
unused = self._unused_for(host)
counted = {k: v for k, v in phys.items() if k not in unused}
up_count = sum(1 for v in counted.values() if v)
total = len(counted)
if not prom_ok:
status = 'unknown'
elif total == 0 or up_count == total:
@@ -1812,7 +1939,7 @@ class NetworkMonitor:
hosts[host] = {
'ip': instance.split(':')[0],
'interfaces': {k: ('up' if v else 'down') for k, v in phys.items()},
'interfaces': {k: ('unused' if k in unused else ('up' if v else 'down')) for k, v in phys.items()},
'status': status,
'source': 'prometheus',
}
@@ -1855,6 +1982,7 @@ class NetworkMonitor:
while True:
try:
logger.info('Starting network check cycle')
self._load_unused()
# 1. Fetch interface states once: shared by snapshot and alert processing
iface_states = self.prom.get_interface_states()
@@ -1910,9 +2038,11 @@ class NetworkMonitor:
# WAN modem: isolated so a failure here can't stop the rest of the cycle
if self.modem:
try:
# Log first, so a restart it reports can explain an uptime reset below
self._process_wan_log(suppressions)
self._collect_modem(suppressions)
# Log first, so a restart it reports can explain an uptime reset below.
# Tickets open when the batch ends, so one incident gets one ticket.
with self._wan_batch():
self._process_wan_log(suppressions)
self._collect_modem(suppressions)
except Exception as e:
logger.error(f'Modem collection failed: {e}', exc_info=True)
+62
View File
@@ -648,3 +648,65 @@ document.addEventListener('click', e => {
);
}
});
// ── Maintenance: pause every ticket for a while (a global suppression with an expiry) ──
(function () {
const until = document.querySelector('.maint-banner .maint-until');
if (until) {
const t = new Date(until.getAttribute('datetime'));
if (!isNaN(t)) {
const sameDay = t.toDateString() === new Date().toDateString();
until.textContent = sameDay
? t.toLocaleTimeString([], { hour: 'numeric', minute: '2-digit' })
: t.toLocaleString([], { weekday: 'short', hour: 'numeric', minute: '2-digit' });
until.title = t.toLocaleString();
}
}
let minutes = 60;
document.addEventListener('click', async e => {
if (e.target.closest('[data-action="start-maintenance"]')) {
lt.modal.open('maint-modal');
document.getElementById('maint-reason')?.focus();
return;
}
const pill = e.target.closest('.maint-pill');
if (pill) {
minutes = parseInt(pill.dataset.minutes, 10);
document.querySelectorAll('.maint-pill').forEach(p => {
const on = p === pill;
p.classList.toggle('active', on);
p.setAttribute('aria-pressed', String(on));
});
return;
}
const end = e.target.closest('[data-action="end-maintenance"]');
if (end) {
end.disabled = true;
try {
await lt.api.delete(`/api/suppressions/${encodeURIComponent(end.dataset.supId)}`);
window.location.reload();
} catch (err) {
end.disabled = false;
showToast(err.message || 'Could not end maintenance', 'error');
}
}
});
document.getElementById('maint-form')?.addEventListener('submit', async e => {
e.preventDefault();
const reason = (document.getElementById('maint-reason').value || '').trim();
if (!reason) { showToast('Reason is required', 'error'); return; }
const submit = e.target.querySelector('[type="submit"]');
submit.disabled = true;
try {
await lt.api.post('/api/suppressions', {
target_type: 'all', target_name: '', reason, expires_minutes: minutes,
});
window.location.reload();
} catch (err) {
submit.disabled = false;
showToast(err.message || 'Could not start maintenance', 'error');
}
});
})();
+45 -7
View File
@@ -122,9 +122,11 @@ function hostRows(interfaces) {
return Object.entries(interfaces)
.sort(([a], [b]) => a.localeCompare(b, undefined, { numeric: true }))
.map(([name, d]) => {
const up = d.state ? d.state === 'up' : d.link_detected !== false;
const mark = (_device.unused_interfaces || {})[name];
const unused = !!mark || d.state === 'unused';
const up = !unused && (d.state ? d.state === 'up' : d.link_detected !== false);
return {
id: name, label: name, up, data: d, sub: d.port_type || '',
id: name, label: name, up, unused, unusedReason: mark ? mark.reason : '', data: d, sub: d.port_type || '',
detail: `
<div class="link-stats-grid">
${stat('Duplex', esc(d.duplex || '-'), 'Full = both directions at once. Half can cause collisions.')}
@@ -154,7 +156,21 @@ function portRows(ports) {
}
function rowHasProblem(r) {
return !!badges(r.data);
return !r.unused && !!badges(r.data);
}
// The Health cell: an unused interface says so (with an admin toggle) instead of reporting down
function healthCell(r) {
if (r.unused) {
const tip = r.unusedReason ? ` title="${esc(r.unusedReason)}"` : '';
const btn = CAN_EDIT ? ` <button type="button" class="lt-btn lt-btn-ghost lt-btn-xs" data-unused="0" data-iface="${esc(r.id)}">In use again</button>` : '';
return `<span class="lt-badge badge-neutral"${tip}>unused</span>${btn}`;
}
const state = badges(r.data) || (r.up ? '<span class="val-good">OK</span>' : '<span class="device-row-sub">down</span>');
const btn = CAN_EDIT && !r.up && r.num === undefined
? ` <button type="button" class="lt-btn lt-btn-ghost lt-btn-xs" data-unused="1" data-iface="${esc(r.id)}"
title="Not meant to be up: stop alerting and stop counting it against the host">Mark unused</button>` : '';
return state + btn;
}
function renderTable(rows) {
@@ -173,15 +189,15 @@ function renderTable(rows) {
? (r.neighbor.name ? `<a class="device-link" href="${esc(deviceUrl(r.neighbor.name))}">${esc(r.neighbor.name)}</a>` : esc(r.neighbor.chassis || r.neighbor.port || ''))
: '-';
return `
<tr class="device-row${r.up ? '' : ' device-row--down'}"${isPorts ? ` data-port-idx="${esc(r.num)}"` : ''}>
<tr class="device-row${r.unused ? ' device-row--unused' : (r.up ? '' : ' device-row--down')}"${isPorts ? ` data-port-idx="${esc(r.num)}"` : ''}>
<td><button type="button" class="device-row-toggle" data-row="${esc(r.id)}"${isPorts ? '' : ` aria-expanded="${open}"`} aria-label="Details for ${esc(r.label)}">${open ? '▾' : '▸'}</button></td>
<td><span class="dot-${r.up ? 'up' : 'down'}"></span></td>
<td><span class="dot-${r.unused ? 'unused' : (r.up ? 'up' : 'down')}"></span></td>
<td class="device-row-name">${r.num !== undefined ? `<span class="port-badge port-badge-num">#${esc(r.num)}</span> ` : ''}${esc(r.label)}<div class="device-row-sub">${esc(r.sub)}</div></td>
<td class="device-num">${r.up ? fmtSpeed(d.speed_mbps) : '-'}</td>
<td class="device-num device-col-wide">${fmtRate(d.rx_bytes_rate)}</td>
<td class="device-num device-col-wide">${fmtRate(d.tx_bytes_rate)}</td>
${isPorts ? `<td class="device-col-wide">${neighbor}</td>` : ''}
<td>${badges(d) || (r.up ? '<span class="val-good">OK</span>' : '<span class="device-row-sub">down</span>')}</td>
<td>${healthCell(r)}</td>
</tr>
${open ? `<tr class="device-row-detail"><td colspan="7">${r.detail}</td></tr>` : ''}`;
}).join('');
@@ -326,8 +342,10 @@ function updateInterfaceSection(rows, title, hasChassis) {
document.getElementById('device-ports-title').textContent = 'Front panel';
return;
}
const inUse = rows.filter(r => !r.unused);
const unusedCount = rows.length - inUse.length;
document.getElementById('device-ports-title').textContent =
`${title} · ${rows.filter(r => r.up).length} of ${rows.length} up`;
`${title} · ${inUse.filter(r => r.up).length} of ${inUse.length} up${unusedCount ? ` · ${unusedCount} unused` : ''}`;
document.getElementById('device-table-search').setAttribute('aria-label', `Filter ${title.toLowerCase()}`);
document.getElementById('device-error-count').textContent = rows.filter(rowHasProblem).length;
}
@@ -369,7 +387,27 @@ async function saveInventory(e) {
document.addEventListener('click', e => {
if (e.target.closest('[data-action="edit-inventory"]')) openInventoryEditor();
const mark = e.target.closest('[data-unused]');
if (mark) toggleUnused(mark.dataset.iface, mark.dataset.unused === '1');
});
async function toggleUnused(iface, unused) {
const url = `/api/device/${encodeURIComponent(DEVICE_NAME)}/interfaces/${encodeURIComponent(iface)}/unused`;
try {
if (unused) {
const reason = window.prompt(`Why isn't ${iface} in use? (optional, e.g. "spare, never cabled")`, '');
if (reason === null) return;
await lt.api.put(url, { reason });
lt.toast.success(`${iface} marked unused: it won't alert or count against ${DEVICE_NAME}`);
} else {
await lt.api.delete(url);
lt.toast.success(`${iface} is in use again`);
}
await loadDevice();
} catch (err) {
lt.toast.error(err.message || 'Save failed');
}
}
const _inventoryForm = document.getElementById('inventory-form');
if (_inventoryForm) _inventoryForm.addEventListener('submit', saveInventory);
+12 -6
View File
@@ -595,6 +595,15 @@ function faceplateLegendHtml(spec, marks) {
// ctx: { swName, portMap, outlets, online, titleFor(port, idx, role), cutsheetFor?(idx) -> {mark, text},
// rack? (draw only the front, spec.rackView, as mounted in a rack) }
function renderFaceplate(model, ctx) {
const { svg, marks } = faceplateSvg(model, ctx);
return `<div class="fp-wrap">
<div class="fp-frame ${FACEPLATES[model].frameClass}">${svg}</div>
</div>${faceplateLegendHtml(FACEPLATES[model], marks)}`;
}
// The faceplate's SVG on its own, for drawing inside another picture (a shelf). `attrs` adds
// attributes to the <svg> (x, y, width, height to place it). Returns { svg, marks }.
function faceplateSvg(model, ctx, attrs = '') {
const spec = FACEPLATES[model];
const anyPort = Object.values(ctx.portMap)[0];
const marks = new Set();
@@ -612,9 +621,7 @@ function renderFaceplate(model, ctx) {
const outlets = (spec.outlets || []).map(o => faceplateOutletSvg(o, outletData[o.idx])).join('');
const body = spec.noBody ? ''
: `<rect class="fp-body" width="${spec.width}" height="${spec.height}" rx="3"/>`;
return `<div class="fp-wrap">
<div class="fp-frame ${spec.frameClass}">
<svg class="fp-svg" viewBox="${((ctx.rack && spec.rackView) || [0, 0, spec.width, spec.height]).join(' ')}" role="group" aria-label="${lt.escHtml(spec.label)} front panel">
const svg = `<svg class="fp-svg" ${attrs} viewBox="${((ctx.rack && spec.rackView) || [0, 0, spec.width, spec.height]).join(' ')}" role="group" aria-label="${lt.escHtml(spec.label)} front panel">
<defs>
<linearGradient id="fp-metal" x1="0" y1="0" x2="0" y2="1">
<stop offset="0" stop-color="#e4e4e6"/><stop offset=".08" stop-color="#d6d6d8"/>
@@ -628,9 +635,8 @@ function renderFaceplate(model, ctx) {
${spec.decor({ online: ctx.online })}
${outlets}
${ports}
</svg>
</div>
</div>${faceplateLegendHtml(spec, marks)}`;
</svg>`;
return { svg, marks };
}
// Where an fp-scroll panel should start: its port area
+146 -23
View File
@@ -1,5 +1,5 @@
'use strict';
/* exported RACK_FRONTS, CHASSIS_FACEPLATES, rackFrontSvg, shelfSvg, SHELF_GEAR */
/* exported RACK_FRONTS, CHASSIS_FACEPLATES, rackFrontSvg, shelfSvg, SHELF_GEAR, DESK_FACEPLATES */
// Drawn fronts for rack gear Gandalf can't read a faceplate from: custom-built servers. Chosen per cutsheet item ("Front drawing"). Each one
// shows the device's live status on its power light when the item is linked to a device.
//
@@ -106,12 +106,107 @@ const RACK_FRONTS = {
},
},
// Sliger CX4712 with its coloured front fitted, as in the rack photo (wiki: rack-2026.jpg):
// a hex-perforated panel over most of the face, the drive cages faintly visible through it,
// black rack ears, and a power button over two USB ports at the top right.
'sliger-cx4712-red': sligerFront('#c8161d', '#e9e5e0'),
'sliger-cx4712-white': sligerFront('#eceae6', '#9aa0a6'),
'sliger-cx4712-blue': sligerFront('#1f5fae', '#e9e5e0'),
// Ubiquiti UACC-Rack-Panel-Brush-1U, from Ubiquiti's product images (store.ui.com, UACC-Rack-Panel
// variant Brush 1U): a light silver face with separate ear plates (two rounded slots each), a
// recessed chamfered frame around the opening (shadowed along the top, bright bevel along the
// bottom), and two rows of black bristles meeting at a seam along the middle.
'uacc-brush-1u': {
label: 'UniFi Rack Panel Brush 1U (UACC-Rack-Panel-Brush-1U)', units: 1, width: 1100, height: 105,
draw() {
// Bristles: each row is shaded root to tip (dark where it leaves the frame, lighter where the
// two rows meet), with a faint strand texture on top. Strong individual strands alias into hard
// stripes at rack size, so the texture stays low-contrast and the shading carries the look.
// The opening runs almost the full width between the ear plates, as in Ubiquiti's image
const strands = (y0, y1, seed) => Array.from({ length: 172 }, (_, i) => {
const n = (i * 7919 + seed * 104729) % 97;
return `<rect x="${60 + i * 5.7 + (n % 5) * 0.4}" y="${y0 + (n % 4)}" width="${1.4 + (n % 3) * 0.5}"
height="${y1 - y0 - (n % 4)}" class="ub-strand" style="opacity:${0.18 + (n % 5) * 0.05}"/>`;
}).join('');
const slot = x => `<rect class="ua-ear-slot" x="${x - 7}" y="22" width="14" height="10" rx="5"/>
<rect class="ua-ear-slot" x="${x - 7}" y="73" width="14" height="10" rx="5"/>`;
return `<defs><linearGradient id="fp-metal" x1="0" y1="0" x2="0" y2="1">
<stop offset="0" stop-color="#e4e4e6"/><stop offset=".08" stop-color="#d6d6d8"/>
<stop offset=".92" stop-color="#cfcfd2"/><stop offset="1" stop-color="#a9a9ad"/>
</linearGradient>
<linearGradient id="ub-row-top" x1="0" y1="0" x2="0" y2="1">
<stop offset="0" stop-color="#08090a"/><stop offset=".7" stop-color="#1f2125"/><stop offset="1" stop-color="#34373c"/>
</linearGradient>
<linearGradient id="ub-row-bottom" x1="0" y1="0" x2="0" y2="1">
<stop offset="0" stop-color="#34373c"/><stop offset=".3" stop-color="#1f2125"/><stop offset="1" stop-color="#08090a"/>
</linearGradient></defs>
<rect class="fp-body" width="1100" height="105" rx="3"/>
<rect class="ua-ear-plate" x="0" y="1" width="32" height="103" rx="3"/>
<rect class="ua-ear-plate" x="1068" y="1" width="32" height="103" rx="3"/>
<rect class="ua-ear-seam" x="31" y="1" width="1.5" height="103"/>
<rect class="ua-ear-seam" x="1067.5" y="1" width="1.5" height="103"/>
${slot(16)}${slot(1084)}
<rect class="ub-frame" x="44" y="20" width="1012" height="66" rx="7"/>
<rect class="ub-frame-shadow" x="48" y="24" width="1004" height="7" rx="3"/>
<rect class="ub-frame-bevel" x="48" y="77" width="1004" height="5" rx="2.5"/>
<rect class="ub-opening" x="56" y="30" width="988" height="47" rx="2"/>
<rect x="56" y="30" width="988" height="23.5" fill="url(#ub-row-top)"/>
<rect x="56" y="53.5" width="988" height="23.5" fill="url(#ub-row-bottom)"/>
${strands(31, 53, 1)}${strands(54, 76, 2)}
<rect class="ub-seam" x="56" y="53" width="988" height="1"/>`;
},
},
// Generic servers, for gear without a specific drawing
'server-1u': genericServer(1, 4),
'server-2u': genericServer(2, 12),
'server-4u': genericServer(4, 24),
};
// A rack handle on a Sliger ear: a vertical bar standing off the ear on two mounts
function sligerHandle(x) {
const top = 70, bottom = 350;
return `<rect class="rf-sl-mount" x="${x - 2}" y="${top - 14}" width="30" height="22" rx="4"/>
<rect class="rf-sl-mount" x="${x - 2}" y="${bottom - 8}" width="30" height="22" rx="4"/>
<rect class="rf-sl-handle" x="${x}" y="${top - 6}" width="26" height="${bottom - top + 12}" rx="13"/>
<rect class="rf-sl-handle-shine" x="${x + 5}" y="${top + 6}" width="5" height="${bottom - top - 12}" rx="2.5"/>`;
}
function sligerFront(colour, hole) {
const W = 1100, H = 420;
// Hex perforations: a pattern of hexagons cut out of the panel through a mask
const hex = (cx, cy, r) => Array.from({ length: 6 }, (_, i) => {
const a = Math.PI / 6 + i * Math.PI / 3;
return `${(cx + r * Math.cos(a)).toFixed(1)},${(cy + r * Math.sin(a)).toFixed(1)}`;
}).join(' ');
// Hole size and pitch from the rack photo: about 20 rows of holes over the panel's height
const r = 6.9, dx = 18, dy = 15.6;
const holes = [];
for (let row = 0, y = 46; y < H - 34; row++, y += dy) {
for (let x = 82 + (row % 2 ? dx / 2 : 0); x < 925; x += dx) holes.push(`<polygon points="${hex(x, y, r)}"/>`);
}
const bays = Array.from({ length: 10 }, (_, i) => `<rect class="rf-sl-bay" x="${140 + i * 74}" y="120" width="44" height="250" rx="3" style="fill:${hole}"/>`).join('');
// The cages show only faintly through the mesh
return {
label: 'Sliger CX4712 4U (10-bay)', units: 4, width: W, height: H,
draw({ status }) {
const id = `sl-${colour.slice(1)}`;
return `<defs><mask id="${id}"><rect width="${W}" height="${H}" fill="#fff"/><g fill="#000">${holes.join('')}</g></mask></defs>
<rect class="rf-sl-inside" x="70" y="30" width="870" height="${H - 60}"/>${bays}
<rect class="rf-sl-ear" x="0" y="0" width="70" height="${H}" rx="4"/>
<rect class="rf-sl-ear" x="${W - 70}" y="0" width="70" height="${H}" rx="4"/>
${sligerHandle(14)}${sligerHandle(W - 40)}
<rect x="52" y="10" width="${W - 104}" height="${H - 20}" rx="8" style="fill:${colour}" mask="url(#${id})"/>
<rect class="rf-sl-edge" x="52" y="10" width="${W - 104}" height="${H - 20}" rx="8"/>
<circle class="rf-sl-button" cx="988" cy="88" r="17"/>
<circle class="rf-power-ring" cx="988" cy="88" r="12" style="stroke:${frontLed(status)}"/>
<rect class="rf-sl-io" x="966" y="138" width="44" height="74" rx="4"/>
${usbA(977, 146)}${usbA(977, 168)}${usbC(980, 192)}`;
},
};
}
function genericServer(units, bays) {
const H = 105 * units;
return {
@@ -256,42 +351,70 @@ const SHELF_GEAR = {
},
};
// UniFi desk gear drawn from its live faceplate when it sits on a shelf: the real width of the
// face (Ubiquiti tech specs) and the span of that face in the faceplate drawing, to scale it.
const DESK_FACEPLATES = {
USMINI: { mm: 107, body: [2, 661], label: 'UniFi Switch Flex Mini' }, // 107 x 70 x 21 mm
USF5P: { mm: 122.5, body: [66, 488], label: 'UniFi Switch Flex' }, // 122.5 x 107.1 x 28 mm
};
// A shelf at true scale (1100 units = 482.6 mm, a 19" panel) with what sits on it, left to right.
// Gear with a SHELF_GEAR drawing is drawn to its real width; anything else is a labelled box.
function shelfSvg(item, esc) {
// Gear with a SHELF_GEAR drawing, or UniFi desk gear with a live faceplate, is drawn to its real
// width; anything else is a labelled box. A shelf with Front drawing "resting" has no tray: its gear
// lies directly on whatever is below it (a switch on top of the NVR).
function shelfSvg(item, esc, faceplate) {
const units = item.rack_height || 1;
const W = 1100, H = 105 * units, perMm = W / 482.6;
const floor = H - 20;
const resting = item.chassis === 'resting';
const floor = resting ? H - 3 : H - 20;
let x = 64;
const things = (item.contents || []).map(c => {
const gear = SHELF_GEAR[c.chassis];
const w = gear ? gear.mm * perMm : 150;
const k = gear ? w / gear.width : 1;
const h = gear ? gear.height * k : 58;
const desk = !gear && faceplate && DESK_FACEPLATES[c.model];
let w, h, body;
if (gear) {
w = gear.mm * perMm;
const k = w / gear.width;
h = gear.height * k;
body = `<g transform="translate(${x} ${floor - h}) scale(${k})">${gear.draw(c)}</g>`;
} else if (desk) {
// Scale the faceplate so its body spans the switch's real width, then place the whole drawing
const spec = faceplate.spec(c.model);
const k = (desk.mm * perMm) / (desk.body[1] - desk.body[0]);
w = desk.mm * perMm;
h = spec.height * k;
body = faceplate.svg(c, `x="${x - desk.body[0] * k}" y="${floor - h}" width="${spec.width * k}" height="${h}"`);
} else {
w = 150;
h = 58;
body = `<rect class="sh-box" x="${x}" y="${floor - h}" width="${w}" height="${h}" rx="4"/>
<text class="sh-box-name" x="${x + w / 2}" y="${floor - h / 2 + 5}" text-anchor="middle">${esc(c.name)}</text>`;
}
const top = floor - h;
const nics = (c.interfaces || []).map(n => `${n.name} ${n.up ? 'up' : 'down'}`).join(', ');
const tip = [c.name, gear ? gear.label : c.kind, c.status, nics, gear && gear.specs].filter(Boolean).join(' · ');
const body = gear
? `<g transform="translate(${x} ${top}) scale(${k})">${gear.draw(c)}</g>`
: `<rect class="sh-box" x="${x}" y="${top}" width="${w}" height="${h}" rx="4"/>
<text class="sh-box-name" x="${x + w / 2}" y="${top + h / 2 + 5}" text-anchor="middle">${esc(c.name)}</text>`;
const out = `<a class="sh-thing" href="${esc(c.url)}"><title>${esc(tip)}</title>
const nics = (c.interfaces || []).map(n => `${n.name} ${n.unused ? 'unused' : (n.up ? 'up' : 'down')}`).join(', ');
const label = gear ? gear.label : (desk ? desk.label : c.kind);
const tip = [c.name, label, c.status, nics, gear && gear.specs].filter(Boolean).join(' · ');
// A live faceplate's ports are links of their own, so it isn't wrapped in one
const open = desk ? '<g class="sh-thing">' : `<a class="sh-thing" href="${esc(c.url)}">`;
const tagY = resting ? top - 16 : floor + 5;
const out = `${open}<title>${esc(tip)}</title>
${body}
<rect class="sh-hit" x="${x - 4}" y="${top - 4}" width="${w + 8}" height="${h + 8}" rx="4"/>
<rect class="sh-tag" x="${x + w / 2 - 46}" y="${floor + 5}" width="92" height="12" rx="2"/>
<text class="sh-tag-text" x="${x + w / 2}" y="${floor + 14}" text-anchor="middle">${esc(c.name)}</text>
</a>`;
${desk ? '' : `<rect class="sh-hit" x="${x - 4}" y="${top - 4}" width="${w + 8}" height="${h + 8}" rx="4"/>`}
<rect class="sh-tag" x="${x + w / 2 - 46}" y="${tagY}" width="92" height="12" rx="2"/>
<text class="sh-tag-text" x="${x + w / 2}" y="${tagY + 9}" text-anchor="middle">${esc(c.name)}</text>
${desk ? '</g>' : '</a>'}`;
x += w + 26;
return out;
}).join('');
const tray = resting ? '' : `${ears(W, H, 'rf-ear-dark')}
<rect class="sh-back" x="36" y="6" width="${W - 72}" height="${floor - 6}"/>
<rect class="sh-floor" x="30" y="${floor}" width="${W - 60}" height="${H - floor - 2}" rx="2"/>
<rect class="sh-lip" x="30" y="${floor}" width="${W - 60}" height="3"/>`;
return `<svg class="rack-svg rack-shelf-svg" viewBox="0 0 ${W} ${H}" preserveAspectRatio="none" role="group"
aria-label="${esc(item.name)}">
<title>${esc(item.name)}</title>
${ears(W, H, 'rf-ear-dark')}
<rect class="sh-back" x="36" y="6" width="${W - 72}" height="${floor - 6}"/>
<rect class="sh-floor" x="30" y="${floor}" width="${W - 60}" height="${H - floor - 2}" rx="2"/>
<rect class="sh-lip" x="30" y="${floor}" width="${W - 60}" height="3"/>
${things || `<text class="sh-empty" x="${W / 2}" y="${floor - 12}" text-anchor="middle">empty shelf</text>`}
${tray}
${things || `<text class="sh-empty" x="${W / 2}" y="${floor - 12}" text-anchor="middle">${resting ? 'nothing here' : 'empty shelf'}</text>`}
<text class="sh-name" x="46" y="18">${esc(item.name)}</text>
</svg>`;
}
+11 -2
View File
@@ -1,5 +1,5 @@
'use strict';
/* global FACEPLATES, renderFaceplate, RACK_CAN_EDIT, RACK_FRONTS, CHASSIS_FACEPLATES, rackFrontSvg, shelfSvg */
/* global FACEPLATES, renderFaceplate, faceplateSvg, RACK_CAN_EDIT, RACK_FRONTS, CHASSIS_FACEPLATES, rackFrontSvg, shelfSvg */
// Rack page: draws the rack front from /api/rack. Switches use their real faceplates,
// patch panels a generic panel; hovering a port lights up every port on the same cable run.
@@ -145,6 +145,15 @@ function plateHtml(item) {
</div>`;
}
// Live faceplates for UniFi desk gear sitting on a shelf (a Flex Mini on top of the NVR)
const shelfFaceplates = {
spec: model => FACEPLATES[model],
svg: (c, attrs) => faceplateSvg(c.model, {
swName: c.device || c.name, portMap: portMapOf(c.ports), outlets: [], online: c.status === 'up',
titleFor: (port, idx) => portTitle(port, idx),
}, attrs).svg,
};
function backHtml(item) {
// A full-depth item seen from the rear: its back, not its faceplate
return `<div class="rack-plate rack-back">
@@ -161,7 +170,7 @@ function itemHtml(item) {
else if (item.chassis && RACK_FRONTS[item.chassis]) body = rackFrontSvg(item.chassis, item);
else if (item.kind === 'patch_panel' && item.chassis === 'uacc-patch-24') body = uaccPanelSvg(item);
else if (item.kind === 'patch_panel') body = panelSvg(item);
else if (item.kind === 'shelf') body = shelfSvg(item, esc);
else if (item.kind === 'shelf') body = shelfSvg(item, esc, shelfFaceplates);
else body = plateHtml(item);
return `<div class="rack-item rack-kind-${esc(item.kind)}" data-item="${esc(item.id)}">${body}</div>`;
}
+42
View File
@@ -383,6 +383,22 @@
.pill:hover { border-color: var(--green); color: var(--green); background: var(--green-dim); }
.pill.active,
.pill-manual.active { border-color: var(--amber); color: var(--amber); background: var(--amber-dim); text-shadow: var(--glow-amber); }
/* Maintenance modal pills: same look, own class so the suppress modal's handler skips them */
.maint-pill { padding: 3px 10px; border: 1px solid var(--border-color); background: transparent;
font-family: var(--font); font-size: .72em; font-weight: bold; cursor: pointer;
color: var(--text-muted); transition: all .15s; letter-spacing: .04em; }
.maint-pill:hover { border-color: var(--green); color: var(--green); background: var(--green-dim); }
.maint-pill.active { border-color: var(--amber); color: var(--amber); background: var(--amber-dim); text-shadow: var(--glow-amber); }
.maint-intro { margin: 0 0 12px; }
/* Maintenance banner: every page while a global suppression is active */
.maint-banner {
display: flex; align-items: center; justify-content: space-between; gap: 12px; flex-wrap: wrap;
margin-bottom: 16px; padding: 8px 14px; border: 1px solid var(--amber); background: var(--amber-dim);
color: var(--amber); font-size: .82em;
}
.maint-banner strong { letter-spacing: .06em; text-transform: uppercase; }
.maint-reason { color: var(--text-muted); }
/* ── Available targets grid (suppressions page) ───────────────────── */
.targets-grid { display: grid; grid-template-columns: repeat(auto-fill, minmax(180px, 1fr)); gap: 10px; }
@@ -1630,3 +1646,29 @@ body.fp-marks-off .fp-svg .fp-mark { display: none; }
.cx-tab { fill: none; stroke: #c9ccd0; stroke-width: 9; stroke-linejoin: round; }
.cx-link { stroke: #101418; stroke-width: 3; }
.fp-speedbar { fill: #b3b5ba; }
.rf-sl-inside { fill: #141619; }
.rf-sl-ear { fill: #17181a; }
.rf-sl-edge { fill: none; stroke: rgba(0,0,0,.35); stroke-width: 2; }
.rf-sl-button { fill: #c9ccd0; stroke: #8a8e93; stroke-width: 2; }
.rf-sl-bay { opacity: .45; }
.rf-sl-io { fill: #0e0f11; }
.rf-sl-mount { fill: #0c0d0f; }
.rf-sl-handle { fill: #3a3e44; stroke: #0a0b0c; stroke-width: 2; }
.rf-sl-handle-shine { fill: #6b717a; }
.ua-ear-plate { fill: #d3d5d8; stroke: #b2b4b8; stroke-width: 1; }
.ua-ear-seam { fill: #9fa1a6; }
.ua-ear-slot { fill: #2a2c30; }
.ub-frame { fill: #bfc1c5; stroke: #a5a7ac; stroke-width: 1; }
.ub-frame-shadow { fill: #8c8e93; }
.ub-frame-bevel { fill: #f1f2f3; }
.ub-opening { fill: #0d0e10; }
.ub-seam { fill: #050506; opacity: .8; }
.ub-strand { fill: #000; }
/* Interfaces marked unused: greyed, never "down" */
.dot-unused { display: inline-block; width: 8px; height: 8px; border: 1px dashed var(--text-muted);
background: transparent; flex-shrink: 0; vertical-align: middle; }
.state-unused { color: var(--text-muted); }
.device-row--unused td { opacity: .6; }
.device-row--unused td:last-child { opacity: 1; }
.lt-btn-xs { padding: 1px 6px; font-size: .68rem; margin-left: 6px; }
+57 -1
View File
@@ -125,6 +125,10 @@
<span class="lt-header-user">{{ _uname }}</span>
{% if user.groups and 'admin' in user.groups %}
<span class="lt-badge lt-badge-admin" aria-label="Administrator">ADMIN</span>
{% if not maintenance %}
<button type="button" class="lt-btn lt-btn-ghost lt-btn-sm" data-action="start-maintenance"
title="Pause all tickets while you work on the network">Maintenance</button>
{% endif %}
{% endif %}
<!-- Notification bell: shows active monitoring alerts -->
@@ -184,6 +188,20 @@
<!-- MAIN CONTENT -->
<main class="lt-main lt-container" id="main-content">
{% if maintenance %}
<div class="maint-banner" role="status">
<span class="maint-banner-text">
<strong>Maintenance:</strong> Gandalf is not opening tickets
{% if maintenance.expires_at %}until <time class="maint-until" datetime="{{ maintenance.expires_at }}">{{ maintenance.expires_at }}</time>{% else %}until it is ended{% endif %}.
Alerts are still recorded, and anything still failing afterwards gets a ticket.
<span class="maint-reason">{{ maintenance.reason }}{% if maintenance.suppressed_by %} ({{ maintenance.suppressed_by }}){% endif %}</span>
</span>
{% if user.groups and 'admin' in user.groups %}
<button type="button" class="lt-btn lt-btn-secondary lt-btn-sm" data-action="end-maintenance"
data-sup-id="{{ maintenance.id }}">End now</button>
{% endif %}
</div>
{% endif %}
{% block content %}{% endblock %}
</main>
@@ -203,9 +221,47 @@
<span class="lt-footer-sep">|</span>
<button type="button" class="lt-footer-hint" data-action="show-keyboard-help"><span class="lt-footer-key">[ ? ]</span> HELP</button>
</nav>
<span>GANDALF &mdash; TDS v1.2</span>
<span>GANDALF | TDS v1.2</span>
</footer>
{% if user.groups and 'admin' in user.groups and not maintenance %}
<!-- MAINTENANCE MODAL: pause every ticket for a while -->
<div id="maint-modal" class="lt-modal-overlay"
role="dialog" aria-modal="true" aria-labelledby="maint-modal-title" aria-hidden="true">
<div class="lt-modal">
<div class="lt-modal-header">
<h3 class="lt-modal-title" id="maint-modal-title">Start maintenance</h3>
<button type="button" class="lt-modal-close" data-modal-close aria-label="Close">&#x2715;</button>
</div>
<form id="maint-form">
<div class="lt-modal-body">
<p class="lt-field-hint maint-intro">Gandalf keeps checking and recording alerts but opens no tickets
until this ends. Anything still failing afterwards gets a ticket as usual.</p>
<div class="lt-form-group">
<label class="lt-label" for="maint-reason">Reason <span class="required">*</span></label>
<input type="text" class="lt-input" id="maint-reason" name="reason" maxlength="500"
value="Maintenance" required aria-required="true"
placeholder="e.g. UniFi Network update">
</div>
<div class="lt-form-group lt-form-group--last">
<span class="lt-label" id="maint-len-label">How long</span>
<div class="duration-pills" role="group" aria-labelledby="maint-len-label">
<button type="button" class="maint-pill" data-minutes="30" aria-pressed="false">30 min</button>
<button type="button" class="maint-pill active" data-minutes="60" aria-pressed="true">1 hr</button>
<button type="button" class="maint-pill" data-minutes="120" aria-pressed="false">2 hr</button>
<button type="button" class="maint-pill" data-minutes="240" aria-pressed="false">4 hr</button>
</div>
</div>
</div>
<div class="lt-modal-footer">
<button type="button" class="lt-btn lt-btn-secondary" data-modal-close>Cancel</button>
<button type="submit" class="lt-btn lt-btn-primary">Start</button>
</div>
</form>
</div>
</div>
{% endif %}
<!-- QUICK-SUPPRESS MODAL: available on all pages via [S] shortcut -->
<div id="suppress-modal" class="lt-modal-overlay"
role="dialog" aria-modal="true" aria-labelledby="suppress-modal-title" aria-hidden="true">
+67 -1
View File
@@ -37,6 +37,7 @@ def client():
db.check_suppressed.return_value = False
db.get_device_link.return_value = ''
db.cutsheet_racks.return_value = []
db.get_unused_interfaces.return_value = {}
with patch.object(gandalf, '_cfg', CONFIG):
yield gandalf.app.test_client()
@@ -551,7 +552,8 @@ class TestRackFacesAndSettings:
things = data['items'][0]['contents']
assert [(c['name'], c['slot'], c['chassis']) for c in things] == [('b-box', 1, 'zimaboard-832'),
('large1', 2, '')]
assert things[1]['interfaces'] == [{'name': 'eth0', 'up': True}]
assert things[1]['interfaces'] == [{'name': 'eth0', 'up': True, 'unused': False}]
assert things[0]['device'] == 'b-box' and things[0]['ports'] == {}
def test_saved_racks_replace_config(self, client):
self._items()
@@ -582,3 +584,67 @@ class TestRackFacesAndSettings:
assert client.put('/api/cutsheet/racks/1', json={'name': 'x', 'units': 2}, headers=user).status_code == 403
gandalf.db.cutsheet_delete_rack.return_value = False
assert client.delete('/api/cutsheet/racks/9', headers=AUTH).status_code == 404
class TestUnusedInterfaces:
def _url(self, iface='nic1'):
gandalf._cutsheet_schema_ready = True
return f'/api/device/compute-storage-01/interfaces/{iface}/unused'
def test_mark_and_clear(self, client):
assert client.put(self._url(), json={'reason': ' never cabled '}, headers=AUTH).status_code == 200
gandalf.db.set_unused_interface.assert_called_with('compute-storage-01', 'nic1', 'never cabled', 'jared')
gandalf.db.clear_unused_interface.return_value = True
resp = client.delete(self._url(), headers=AUTH)
assert resp.get_json() == {'success': True, 'changed': True}
def test_needs_admin_json_and_a_sane_name(self, client):
user = dict(AUTH, **{'Remote-Groups': 'staff'})
with patch.object(gandalf, '_cfg', dict(CONFIG, auth=dict(CONFIG['auth'], allowed_groups=['admin', 'staff']))):
assert client.put(self._url(), json={}, headers=user).status_code == 403
assert client.put(self._url(), data='x', headers=AUTH).status_code == 400
assert client.put(self._url('bad name!'), json={}, headers=AUTH).status_code in (400, 404)
def test_device_api_lists_them(self, client):
gandalf._cutsheet_schema_ready = True
gandalf.db.get_unused_interfaces.return_value = {'large1': {'enp36s0': {'reason': 'spare'}}}
data = client.get('/api/device/large1', headers=AUTH).get_json()
assert data['unused_interfaces'] == {'enp36s0': {'reason': 'spare'}}
class TestMaintenance:
SUP = {'id': 7, 'target_type': 'all', 'target_name': '', 'reason': 'UniFi Network update',
'suppressed_by': 'jared', 'expires_at': '2026-11-12T20:00:00Z'}
def test_no_banner_and_a_start_button_by_default(self, client):
html = client.get('/', headers=AUTH).get_data(as_text=True)
assert 'maint-banner' not in html
assert 'data-action="start-maintenance"' in html and 'id="maint-modal"' in html
def test_banner_on_every_page_while_on(self, client):
gandalf.db.get_active_suppressions.return_value = [
{'id': 3, 'target_type': 'host', 'target_name': 'large1', 'reason': 'x', 'expires_at': None},
self.SUP,
]
for page in ('/', '/events', '/rack'):
html = client.get(page, headers=AUTH).get_data(as_text=True)
assert 'class="maint-banner"' in html, page
assert 'datetime="2026-11-12T20:00:00Z"' in html and 'UniFi Network update' in html
assert 'data-sup-id="7"' in html and 'data-action="start-maintenance"' not in html
def test_open_ended_maintenance_wins(self, client):
gandalf.db.get_active_suppressions.return_value = [
self.SUP, dict(self.SUP, id=9, expires_at=None, reason='Rewiring')]
html = client.get('/', headers=AUTH).get_data(as_text=True)
assert 'until it is ended' in html and 'data-sup-id="9"' in html
def test_viewers_see_the_banner_without_buttons(self, client):
gandalf.db.get_active_suppressions.return_value = [self.SUP]
viewer = dict(AUTH, **{'Remote-Groups': 'staff'})
with patch.object(gandalf, '_cfg', dict(CONFIG, auth=dict(CONFIG['auth'], allowed_groups=['admin', 'staff']))):
html = client.get('/', headers=viewer).get_data(as_text=True)
assert 'class="maint-banner"' in html and 'data-action="end-maintenance"' not in html
def test_a_db_error_means_no_banner(self, client):
gandalf.db.get_active_suppressions.side_effect = RuntimeError('db down')
assert gandalf.inject_maintenance() == {'maintenance': None}
+3 -1
View File
@@ -369,7 +369,9 @@ class TestChassis:
def test_every_drawing_has_a_label(self):
assert all(cs.CHASSIS.values())
assert {'sliger-cx4712', 'rosewill-rsv-l4500u', 'unvr-pro', 'uacc-patch-24',
'usw-pro-max-24-poe', 'usw-pro-max-48-poe', 'zimaboard-832', 'zimaboard-832-cx3'} <= set(cs.CHASSIS)
'usw-pro-max-24-poe', 'usw-pro-max-48-poe', 'zimaboard-832', 'zimaboard-832-cx3',
'resting', 'uacc-brush-1u', 'sliger-cx4712-red', 'sliger-cx4712-white',
'sliger-cx4712-blue'} <= set(cs.CHASSIS)
class TestFacingAndSlots:
+130
View File
@@ -31,6 +31,8 @@ def db():
mock_db.cutsheet_load.return_value = {'items': [], 'cables': [], 'port_labels': []}
mock_db.get_pending_recovery_notes.return_value = []
mock_db.get_device_link.return_value = ''
mock_db.get_incident_ticket.return_value = None
mock_db.get_ticket_ids_between.return_value = []
yield mock_db
@@ -117,6 +119,19 @@ class TestDeadHostDetection:
mon._process_interfaces({HOST_A: {'eth0': True}}, [])
assert ('cluster_network_issue', 'test-cluster', '') in upserted(db)
def test_unused_interface_never_alerts(self, mon, db):
mon._unused = {'alpha': {'eth1': {'reason': 'spare'}}}
db.get_baseline.return_value = {'alpha': {'eth0': 'up', 'eth1': 'up'}, 'beta': {'eth0': 'up'}}
mon._process_interfaces({HOST_A: {'eth0': True, 'eth1': False}, HOST_B: {'eth0': True}}, [])
assert ('interface_down', 'alpha', 'eth1') not in upserted(db)
assert not mon.tickets.create.called
def test_unused_interface_does_not_degrade_the_host(self, mon):
mon._unused = {'alpha': {'eth1': {}}}
snap = mon._collect_snapshot({HOST_A: {'eth0': True, 'eth1': False}, HOST_B: {'eth0': True}})
assert snap['hosts']['alpha']['status'] == 'up'
assert snap['hosts']['alpha']['interfaces'] == {'eth0': 'up', 'eth1': 'unused'}
def test_dead_host_shows_as_down_on_dashboard(self, mon):
snap = mon._collect_snapshot({HOST_A: {'eth0': True}})
assert snap['hosts']['alpha']['status'] == 'up'
@@ -368,6 +383,47 @@ class TestWanLog:
mon._process_wan_log([])
assert not mon.tickets.create.called
def test_unifi_update_is_not_a_wan_event(self, mon, db):
now = time.time()
self._state(db, unifi_log_cursor=str(int(now * 1000)))
self._mon(mon, [{'key': 'NETWORK_UPDATED', 'ts': now, 'device': '', 'wan': '',
'app': 'UniFi Network', 'previous': '11.0.97', 'version': '11.0.99'}])
mon._process_wan_log([])
assert not mon.tickets.create.called
saved = dict((c.args[0], c.args[1]) for c in db.set_state.call_args_list)
assert saved[monitor.UNIFI_UPDATE_STATE]['label'] == 'UniFi Network 11.0.97 to 11.0.99'
assert saved['wan_history'] == {'outages': [], 'restarts': []}
def test_update_comments_on_tickets_opened_just_before(self, mon, db):
now = time.time()
self._state(db, unifi_log_cursor=str(int(now * 1000)))
db.get_ticket_ids_between.return_value = ['207521068']
self._mon(mon, [{'key': 'NETWORK_UPDATED', 'ts': now, 'device': '', 'wan': '',
'app': 'UniFi Network', 'previous': '11.0.97', 'version': '11.0.99'}])
mon._process_wan_log([])
start, end = db.get_ticket_ids_between.call_args.args[1:]
assert (end - start).total_seconds() == monitor.UPDATE_LOOKBACK_S + monitor.UPDATE_HOLD_S
tid, text = mon.tickets.comment.call_args.args
assert tid == '207521068' and 'UniFi Network 11.0.97 to 11.0.99' in text
def test_update_in_the_same_read_holds_the_power_cycle_ticket(self, mon, db):
now = time.time()
state = {'unifi_log_cursor': str(int(now * 1000))}
db.get_state.side_effect = lambda key, default=None: state.get(key, default)
db.set_state.side_effect = lambda key, value: state.__setitem__(
key, json.dumps(value) if isinstance(value, dict) else value)
self._mon(mon, [
{'key': modem_api.LOG_MODEM_RESTARTED, 'ts': now - 600, 'device': 'RPDU', 'wan': ''},
{'key': 'NETWORK_UPDATED', 'ts': now, 'device': '', 'wan': '', 'app': 'UniFi Network'},
])
mon._process_wan_log([])
assert not mon.tickets.create.called
db.resolve_event.assert_called_with('modem_power_cycled', 'att-modem', '')
def test_hold_ends(self, mon, db):
self._state(db, unifi_update=json.dumps({'label': 'x', 'hold_until': time.time() - 1}))
assert mon._update_hold() is None
def test_uptime_going_backwards_records_a_restart(self, mon, db):
self._state(db, modem_stats=json.dumps({'uptime_s': 5000}))
mon._note_modem_restart({'uptime_s': 60})
@@ -380,6 +436,55 @@ class TestWanLog:
assert not db.set_state.called
class TestWanIncident:
"""Last night's sequence: the RPDU power-cycles the modem in the cycle the modem goes unreachable."""
def _cycle(self, mon, db):
db.upsert_event.side_effect = [(30, True, 1, None), (29, False, 2, None)]
mon.modem.poll.side_effect = modem_api.ModemUnreachable('Network is unreachable')
with mon._wan_batch():
mon._one_shot_ticket([], 'modem_power_cycled', 'RPDU power-cycled the modem.',
'Modem auto power-cycled after internet loss', '3', 'Check the WAN panel.')
mon._collect_modem([])
def test_one_ticket_led_by_the_most_urgent(self, mon, db):
self._cycle(mon, db)
assert mon.tickets.create.call_count == 1
title, desc = mon.tickets.create.call_args.args
assert 'WAN modem unreachable' in title
assert mon.tickets.create.call_args.kwargs['priority'] == '2'
assert '### Also seen in this incident' in desc and 'power-cycled' in desc
assert sorted(c.args for c in db.set_ticket_id.call_args_list) == [(29, '123'), (30, '123')]
def test_recent_incident_gets_a_comment_instead(self, mon, db):
db.get_incident_ticket.return_value = '555'
mon.tickets.comment.return_value = True
self._cycle(mon, db)
assert not mon.tickets.create.called
assert [c.args[0] for c in mon.tickets.comment.call_args_list] == ['555', '555']
assert 'Also seen: WAN modem unreachable' in mon.tickets.comment.call_args_list[0].args[1]
assert sorted(c.args for c in db.set_ticket_id.call_args_list) == [(29, '555'), (30, '555')]
db.get_incident_ticket.assert_called_with('att-modem', monitor.WAN_INCIDENT_WINDOW_S)
def test_failed_comment_falls_back_to_a_ticket(self, mon, db):
db.get_incident_ticket.return_value = '555'
mon.tickets.comment.return_value = False
self._cycle(mon, db)
assert mon.tickets.create.call_count == 1
def test_outside_a_batch_tickets_open_at_once(self, mon, db):
mon._one_shot_ticket([], 'wan_unstable', 'Unstable.', 'WAN connection unstable', '2', 'Look.')
assert 'unstable' in mon.tickets.create.call_args.args[0]
assert 'no recovery note' in mon.tickets.create.call_args.args[1]
def test_update_hold_skips_new_tickets(self, mon, db):
db.get_state.side_effect = lambda key, default=None: (
json.dumps({'label': 'UniFi Network 11.0.99', 'hold_until': time.time() + 60})
if key == monitor.UNIFI_UPDATE_STATE else default)
self._cycle(mon, db)
assert not mon.tickets.create.called and not mon.tickets.comment.called
class TestUnifiDeviceDetail:
def test_parses_string_stats_and_outlets(self):
d = monitor.UnifiClient._device_detail({
@@ -408,6 +513,31 @@ class TestUnifiDeviceDetail:
got = c.get_system_events(1000, modem_api.WAN_LOG_KEYS)
assert got == [{'key': modem_api.LOG_MODEM_RESTARTED, 'ts': 2.0, 'device': 'RPDU', 'wan': ''}]
def test_system_log_reads_update_versions(self):
c = monitor.UnifiClient({'controller': 'https://u.test', 'api_key': 'k'})
page = MagicMock()
page.json.side_effect = [
{'data': [
{'key': 'NETWORK_UPDATED', 'timestamp': 3000, 'parameters': {
'APPLICATION': 'UniFi Network', 'PREVIOUS_VERSION': '11.0.97', 'VERSION': '11.0.99'}},
{'key': 'FIRMWARE_UPDATE_AVAILABLE_V2', 'timestamp': 3000},
{'key': 'MADE_CHANGES_ONE_SETTING_OBJECT', 'timestamp': 3000},
]},
{'data': []},
]
c.session.post = MagicMock(return_value=page)
got = c.get_system_events(1000, modem_api.wanted_log_key)
assert got == [{'key': 'NETWORK_UPDATED', 'ts': 3.0, 'device': '', 'wan': '',
'app': 'UniFi Network', 'previous': '11.0.97', 'version': '11.0.99'}]
@pytest.mark.parametrize('key,expected', [
('NETWORK_UPDATED', True), ('FIRMWARE_UPDATED_V2', True), ('PROTECT_UPDATED', True),
('FIRMWARE_UPDATE_AVAILABLE_V2', False), ('MADE_CHANGES_ONE_SETTING_OBJECT', False),
('NETWORK_WAN_FAILED_2', False),
])
def test_update_keys(self, key, expected):
assert modem_api.is_update_key(key) is expected
def test_system_log_failure_is_none_not_empty(self):
c = monitor.UnifiClient({'controller': 'https://u.test', 'api_key': 'k'})
c.session.post = MagicMock(side_effect=OSError('boom'))
+22
View File
@@ -147,3 +147,25 @@ def recovery_comment(event: dict, now: datetime, url: str = '') -> str:
lines += ['', '_Gandalf leaves this ticket open. Close it once you know why it happened, '
'or leave it open if it needs follow-up._']
return '\n'.join(lines)
def incident_comment(name: str, summary: str, checks: Sequence[str], when: str) -> str:
"""A further report that belongs to the incident this ticket is already about."""
lines = [f'**Also seen: {name}** ({when})', '', summary.strip()]
if checks:
lines += ['', '**What to check**'] + [f'{n}. {check}' for n, check in enumerate(checks, 1)]
lines += ['', '_Gandalf added this here instead of opening another ticket, since it is part of '
'the same incident._']
return '\n'.join(lines)
def update_comment(label: str, when: str, lookback_min: int, hold_min: int) -> str:
"""Note on a ticket that a UniFi update finished around the time it was opened."""
return '\n'.join([
f'**UniFi update:** {label} finished at {when}.', '',
f'This ticket was opened within {lookback_min} minutes of that update, which UniFi logs once '
f'it is done. Updating the Network app or the gateway firmware can drop the WAN, and Smart '
f'Power may then power-cycle the modem, so the update is the likely cause.', '',
f'_Gandalf holds new modem and WAN tickets for {hold_min} minutes after an update; anything '
f'still wrong after that gets its own ticket._',
])