Compare commits

...
Author SHA1 Message Date
jaredandClaude Sonnet 5 c64853f643 Fix CI: pass --break-system-packages to pip3 installs
Lint / Python (flake8) (push) Successful in 34s
Lint / JS (eslint) (push) Successful in 5s
Lint / Notify on failure (push) Skipped
Security / Python Security (bandit) (push) Successful in 41s
Test / Python Tests (pytest) (push) Successful in 49s
Lint / Deploy (push) Successful in 8s
The runner's Debian image update enforces PEP 668, so system-wide pip3
installs (flake8, bandit, pytest, requirements.txt) were failing before
any job logic ran, blocking lint/test/security and the deploy job.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UB2D82eYNzD1bnMHhFPjRr
2026-09-04 21:10:38 -04:00
jaredandClaude Sonnet 5 42cf754796 Add separate unifi_failure_threshold to avoid false tickets on reboots
Lint / Python (flake8) (push) Failing after 1m4s
Lint / JS (eslint) (push) Successful in 5s
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Failing after 37s
Test / Python Tests (pytest) (push) Failing after 33s
Lint / Notify on failure (push) Successful in 2s
UniFi switches/APs take several minutes to rejoin after a firmware-update
reboot (observed AP taking 12 min on the scheduled 3am update), longer than
the 2-consecutive-check (~4 min) threshold shared with host link-down
detection. Give UniFi device offline detection its own, more forgiving
threshold so momentary maintenance reboots don't cut tickets.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UB2D82eYNzD1bnMHhFPjRr
2026-09-04 20:59:43 -04:00
jaredandClaude Sonnet 4.6 28fd5aad14 Guard ticket creation against duplicates using event's existing ticket_id
Lint / Python (flake8) (push) Failing after 1m11s
Lint / JS (eslint) (push) Successful in 6s
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Failing after 32s
Test / Python Tests (pytest) (push) Failing after 34s
Lint / Notify on failure (push) Successful in 5s
upsert_event now returns ticket_id (4th element) so callers can skip
ticket creation when one already exists. This prevents calling the ticket
API every poll cycle for ongoing issues while still retrying if the
previous creation attempt failed (ticket_id stays NULL until success).

Cluster events use (is_new or not ticket_id) so they too get retried
on failure rather than relying solely on is_new.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-14 11:09:50 -04:00
jaredandClaude Sonnet 4.6 65536d58c5 Fix misleading docstring on _purge_old_jobs_loop
Lint / Python (flake8) (push) Skipped
Lint / JS (eslint) (push) Skipped
The comment claimed the function "runs daily event purge" — that
housekeeping is done by monitor.py's main loop, not here.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-14 11:06:28 -04:00
jaredandClaude Sonnet 4.6 0e852ebd11 Fix ping-only hosts polled twice per cycle with inconsistent parameters
Lint / Python (flake8) (push) Skipped
Lint / JS (eslint) (push) Skipped
_collect_snapshot called pulse.ping(count=1) independently from
_process_ping_hosts which called pulse.ping(count=3). This doubled
network load and could show a host as 'up' in the dashboard while
simultaneously firing an 'unreachable' alert, or vice versa.

Now ping_states is computed once in run() using the alert-quality
parameters (count=3) and shared by both snapshot and alert processing.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-13 23:13:43 -04:00
jaredandClaude Sonnet 4.6 8aaaa89f03 Replace deprecated datetime.utcnow() with datetime.now(timezone.utc)
Lint / Python (flake8) (push) Skipped
Lint / JS (eslint) (push) Skipped
datetime.utcnow() is deprecated in Python 3.12 and removed in 3.13.
Replace all four call sites with timezone-aware equivalents so the
codebase is ready for Python 3.12+.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-13 15:34:41 -04:00
jaredandClaude Sonnet 4.6 7467236ffd Add ESLint config enforcing no-undef and eqeqeq
Lint / Python (flake8) (push) Skipped
Lint / JS (eslint) (push) Skipped
Without a config file, ESLint was running with no-undef disabled, meaning
undefined variable references in static/app.js were silently ignored.
Add .eslintrc.json with no-undef: error and eqeqeq: error so CI actually
catches JS bugs.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-13 15:33:26 -04:00
jaredandClaude Sonnet 4.6 dc16a583a5 Fix inspector auto-refresh ignoring 'Off' setting on page load
Lint / Python (flake8) (push) Skipped
Lint / JS (eslint) (push) Skipped
Same ?? / || issue as the previous fix in index.html and links.html.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-13 13:20:42 -04:00
jaredandClaude Sonnet 4.6 12ef20901a Fix auto-refresh ignoring 'Off' setting on page load
Lint / Python (flake8) (push) Skipped
Lint / JS (eslint) (push) Skipped
Using || 30 / || 60 as a fallback treats refreshInterval=0 (Off) as
falsy and replaces it with the default, causing auto-refresh to start
even when the user saved 'Off'. Replace with nullish coalescing (??)
so only null/undefined triggers the default.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-13 13:19:44 -04:00
jaredandClaude Sonnet 4.6 0693ad56f9 Fix monitor loop double-sleep on error; add grep -F regression test
Lint / Python (flake8) (push) Skipped
Lint / JS (eslint) (push) Skipped
On exception the monitor slept 30s inside the except block then fell
through to time.sleep(poll_interval), giving a 150s recovery gap instead
of 30s. Adding continue after the error sleep fixes this.

Also adds a regression test asserting dmesg filtering uses grep -F --
so a future refactor cannot silently reintroduce the regex wildcard bug.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-13 13:16:43 -04:00
jaredandClaude Sonnet 4.6 c455c5d4b9 Fix suppression annotation for interface_down not checking host-level rules
Lint / Python (flake8) (push) Skipped
Lint / JS (eslint) (push) Skipped
monitor.py checks both 'interface' and 'host' suppressions for interface_down
events, but _annotate_suppressions only checked 'interface'. A host-level
suppression would silently suppress tickets but not mark the table row as
suppressed in the UI.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-13 13:14:46 -04:00
jaredandClaude Sonnet 4.6 13e6839cdf Show suppressed badge on host cards during global maintenance windows
Lint / Python (flake8) (push) Skipped
Lint / JS (eslint) (push) Skipped
Global suppressions (target_type='all') have an empty target_name, so
the selectattr filter never matched them, leaving no visual indicator
when a global maintenance window was active. Pre-compute has_global_sup
before the host loop and OR it into the badge condition.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-13 13:12:25 -04:00
jaredandClaude Sonnet 4.6 e7c83eff2b Use grep -F in dmesg filter to prevent interface name treated as regex
grep {iface} treats dots and other special chars as regex metacharacters.
Switch to grep -F -- {iface} for fixed-string matching and to prevent
a leading dash in the interface name from being parsed as a grep flag.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-13 11:12:02 -04:00
jaredandClaude Sonnet 4.6 5c2a92eaee Remove unused is_new parameter from ticket helper methods
After fixing the is_new guard bug, is_new is no longer used inside
_ticket_interface, _ticket_unifi, or _ticket_unreachable. Drop it from
their signatures and call sites.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-13 11:10:32 -04:00
jaredandClaude Sonnet 4.6 f942e96c24 Fix ticket_id never stored when fail_thresh>1; guard sessionStorage JSON.parse
Lint / Python (flake8) (push) Skipped
Lint / JS (eslint) (push) Skipped
monitor.py: _ticket_interface/_ticket_unifi/_ticket_unreachable all used
`if tid and is_new` to guard db.set_ticket_id(). Since is_new is True only
on the first upsert (consec=1) but tickets are created at consec>=fail_thresh
(default 2), is_new is always False when the ticket is created, so the
ticket link never appeared in the UI. Changed to `if tid:`.

links.html: JSON.parse(sessionStorage.getItem(...)) in togglePanel and
restoreCollapseState had no try-catch. Corrupt/stale session storage would
throw an uncaught SyntaxError. Also wrapped all sessionStorage.setItem
calls in try-catch to defend against storage-full / private-browsing errors.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-11 23:45:20 -04:00
jaredandClaude Sonnet 4.6 8173b9dc7f Cancel in-flight diagnostic poll when user selects a new port
Lint / Python (flake8) (push) Skipped
Lint / JS (eslint) (push) Skipped
Previously switching ports while a diagnostic was running left the
setInterval timer active, causing the result to be written into the
old (now detached) DOM elements and never shown to the user.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-11 23:26:53 -04:00
jaredandClaude Sonnet 4.6 2896692d93 Fix empty-object false negative in links page no-data check
Lint / Python (flake8) (push) Skipped
Lint / JS (eslint) (push) Skipped
The check `!data.hosts && !data.unifi_switches` never caught empty
objects `{}`, which are truthy. Replace with Object.keys length checks
so the friendly "no data yet" banner renders when both collections
are empty.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-11 23:21:50 -04:00
jaredandClaude Sonnet 4.6 c2492bbfe8 Use shlex.quote for remote_cmd in build_ssh_command
Lint / Python (flake8) (push) Skipped
Lint / JS (eslint) (push) Skipped
Matches the pattern already used in monitor.py's _ssh_batch(); prevents
quoting breakage if shlex.quote(iface) emits single-quoted tokens inside
the remote command string.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-11 23:17:11 -04:00
jaredandClaude Sonnet 4.6 33b214eac0 fix: escape ticket_id text content in dynamic events table
Lint / Python (flake8) (push) Skipped
Lint / JS (eslint) (push) Skipped
ticket_id was already escaped in the href attribute but the visible
text (#<id>) used the raw value in an innerHTML template literal.
Apply lt.escHtml() for defense-in-depth against a compromised ticket API.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-11 23:02:09 -04:00
jaredandClaude Sonnet 4.6 0331c2bd00 fix: add aria-required to s-reason field in suppressions form
Lint / Python (flake8) (push) Skipped
Lint / JS (eslint) (push) Skipped
The reason input had `required` for browser validation but was missing
`aria-required="true"`, so screen readers did not announce it as required.
Matches the fix already applied to the equivalent field in base.html.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-11 15:11:05 -04:00
jaredandClaude Sonnet 4.6 2710eaa118 fix: SSH shell quoting bug breaks ethtool collection; ticket_id KeyError
Lint / Python (flake8) (push) Skipped
Lint / JS (eslint) (push) Skipped
monitor.py _ssh_batch(): the remote command was wrapped in double-quotes
(f'root@{ip} "{shell_cmd}"') but shell_cmd itself contains double-quoted
echo sentinels ("___IFACE:eth0___"). When Pulse's shell parses the full
ssh invocation, the nested double-quotes cause mis-parsing — the remote
command is split incorrectly, silently breaking all ethtool/SFP DOM
collection. Fix: use shlex.quote(shell_cmd) so the entire remote command
is single-quoted, leaving inner double-quotes untouched.

TicketClient.create(): data['ticket_id'] raises KeyError if the Tinker
Tickets API returns success=true without a ticket_id field (malformed
response). Use data.get('ticket_id') with an explicit warning log.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-11 13:41:09 -04:00
jaredandClaude Sonnet 4.6 9c011cfce1 fix: UTC timezone suffix missing from all isoformat() timestamp outputs
db.py returned all datetime columns (first_seen, last_seen, resolved_at,
created_at, expires_at) as bare ISO strings like "2026-03-14T14:14:21"
with no timezone marker. Per the ECMAScript spec, new Date() on a
datetime string without timezone treats it as LOCAL time, not UTC.
This made lt.time.ago() and stale-detection wrong for any user whose
browser is not in UTC — event ages and stale warnings would be off by
the client's UTC offset.

monitor.py had the same issue on the network_snapshot 'updated' field.

Fix: append 'Z' to all isoformat() calls (UTC datetimes confirmed by
MySQL server timezone and _now_utc() pattern used throughout codebase).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-11 13:28:49 -04:00
jaredandClaude Sonnet 4.6 182cc8c451 fix: inspector.html stale/updated timestamp broken date parsing
Lint / Python (flake8) (push) Skipped
Lint / JS (eslint) (push) Skipped
Same bug as was just fixed in links.html: data.updated is stored as
"YYYY-MM-DD HH:MM:SS UTC" by monitor.py, so appending 'Z' produced
"…UTCZ" — an invalid date. The stale-data warning and Updated timestamp
in Inspector were silently showing "Invalid Date" and the stale overlay
never fired. Fixed to use _toIso() (already global via app.js).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-11 13:25:17 -04:00
jaredandClaude Sonnet 4.6 0d4f209c79 fix: admin-only suppression enforcement, links.html broken date parsing
Lint / Python (flake8) (push) Skipped
Lint / JS (eslint) (push) Skipped
Security: add require_admin decorator; apply to POST/DELETE /api/suppressions
and /suppressions page. Previously any user in allowed_groups could create or
delete suppressions even though the nav restricts the UI to admins.

Bug: links.html "Updated:" timestamp and stale-warning both produced
Invalid Date because the raw "YYYY-MM-DD HH:MM:SS UTC" string was appended
with 'Z' instead of being normalised through _toIso(). Fix both call sites to
use _toIso(), and remove the now-redundant local _toIso redefinition.

Style: use `with open(sentinel, 'w'): pass` consistently (was open().close()
at avatar JPEG validation path).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-11 13:03:37 -04:00
15 changed files with 151 additions and 77 deletions
+21
View File
@@ -0,0 +1,21 @@
{
"env": {
"browser": true,
"es2021": true
},
"globals": {
"lt": "readonly",
"GANDALF_CONFIG": "readonly",
"CSS": "readonly"
},
"rules": {
"no-undef": "error",
"no-unused-vars": ["warn", { "argsIgnorePattern": "^_", "varsIgnorePattern": "^_" }],
"no-console": "off",
"eqeqeq": ["error", "always", { "null": "ignore" }]
},
"parserOptions": {
"ecmaVersion": 2021,
"sourceType": "script"
}
}
+1 -1
View File
@@ -17,7 +17,7 @@ jobs:
run: |
apt-get update -qq
apt-get install -y -qq python3 python3-pip
pip3 install flake8
pip3 install --break-system-packages flake8
- name: Run flake8
run: flake8 . --exclude=__pycache__,.git
+1 -1
View File
@@ -19,7 +19,7 @@ jobs:
run: |
apt-get update -qq
apt-get install -y -qq python3 python3-pip
pip3 install bandit
pip3 install --break-system-packages bandit
- name: Run bandit
run: bandit -r . --exclude .git,__pycache__,node_modules -ll
+2 -2
View File
@@ -17,8 +17,8 @@ jobs:
run: |
apt-get update -qq
apt-get install -y -qq python3 python3-pip
pip3 install pytest pytest-cov
pip3 install -r requirements.txt --quiet
pip3 install --break-system-packages pytest pytest-cov
pip3 install --break-system-packages -r requirements.txt --quiet
- name: Run pytest with coverage
run: python3 -m pytest tests/ -v --cov=. --cov-report=term-missing --cov-config=.coveragerc
+3 -1
View File
@@ -148,7 +148,7 @@ returned by the UniFi API for that device.
| Condition | Priority |
|---|---|
| UniFi device offline (≥2 consecutive checks) | P2 High |
| UniFi device offline (≥`unifi_failure_threshold` consecutive checks) | P2 High |
| Proxmox host NIC link-down regression (≥2 consecutive checks) | P2 High |
| Host unreachable via ping (≥2 consecutive checks) | P2 High |
| ≥3 hosts simultaneously reporting interface failures | P1 Critical |
@@ -218,6 +218,7 @@ Shared by both processes. Located in the working directory (`/var/www/html/prod/
"monitor": {
"poll_interval": 120,
"failure_threshold": 2,
"unifi_failure_threshold": 5,
"cluster_threshold": 3,
"ping_hosts": [
{ "name": "pbs", "ip": "10.10.10.3" }
@@ -243,6 +244,7 @@ Shared by both processes. Located in the working directory (`/var/www/html/prod/
| `hosts` | Maps Prometheus instance labels → display hostnames |
| `monitor.poll_interval` | Seconds between full check cycles (default: 120) |
| `monitor.failure_threshold` | Consecutive failures before creating ticket (default: 2) |
| `monitor.unifi_failure_threshold` | Consecutive failures before ticketing a UniFi device as offline (default: same as `failure_threshold`). Set higher than `failure_threshold` — UniFi switches/APs can take several minutes to rejoin after a firmware-update reboot, so a low threshold cuts tickets for momentary maintenance reboots instead of real failures. |
| `monitor.cluster_threshold` | Hosts with failures to trigger cluster-wide P1 (default: 3) |
| `monitor.ping_hosts` | Hosts checked only by ping (no node_exporter) |
+36 -12
View File
@@ -64,7 +64,7 @@ _diag_rate: dict = {}
def _purge_old_jobs_loop():
"""Background thread: remove stale diag jobs and run daily event purge."""
"""Background thread: remove stale diagnostic jobs and mark stuck ones done."""
while True:
time.sleep(120)
cutoff = time.time() - 600
@@ -155,6 +155,17 @@ def require_auth(f):
return wrapper
def require_admin(f):
"""Decorator: require require_auth AND membership in the 'admin' group."""
@wraps(f)
def wrapper(*args, **kwargs):
user = _get_user()
if 'admin' not in user.get('groups', []):
return jsonify({'error': 'Admin access required'}), 403
return f(*args, **kwargs)
return wrapper
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
@@ -163,17 +174,26 @@ _PAGE_LIMIT = 200 # max events returned per request
def _annotate_suppressions(events: list, suppressions: list) -> None:
"""Annotate each event dict in-place with an is_suppressed bool."""
"""Annotate each event dict in-place with an is_suppressed bool.
Mirrors the suppression check order in monitor.py exactly:
interface_down → interface OR host
unifi_device_* → unifi_device
everything else → host
"""
for ev in events:
sup_type = (
'unifi_device' if ev.get('event_type') == 'unifi_device_offline'
else 'interface' if ev.get('event_type') == 'interface_down'
else 'host'
)
ev['is_suppressed'] = db.check_suppressed(
suppressions, sup_type,
ev.get('target_name', ''), ev.get('target_detail', '') or '',
)
etype = ev.get('event_type', '')
name = ev.get('target_name', '')
detail = ev.get('target_detail', '') or ''
if etype == 'interface_down':
ev['is_suppressed'] = (
db.check_suppressed(suppressions, 'interface', name, detail) or
db.check_suppressed(suppressions, 'host', name)
)
elif etype == 'unifi_device_offline':
ev['is_suppressed'] = db.check_suppressed(suppressions, 'unifi_device', name, detail)
else:
ev['is_suppressed'] = db.check_suppressed(suppressions, 'host', name, detail)
# ---------------------------------------------------------------------------
@@ -228,6 +248,7 @@ def inspector():
@app.route('/suppressions')
@require_auth
@require_admin
def suppressions_page():
user = _get_user()
active = db.get_active_suppressions()
@@ -323,6 +344,7 @@ def api_get_suppressions():
@app.route('/api/suppressions', methods=['POST'])
@require_auth
@require_admin
def api_create_suppression():
user = _get_user()
data = request.get_json(silent=True) or {}
@@ -371,6 +393,7 @@ def api_create_suppression():
@app.route('/api/suppressions/<int:sup_id>', methods=['DELETE'])
@require_auth
@require_admin
def api_delete_suppression(sup_id: int):
user = _get_user()
db.deactivate_suppression(sup_id)
@@ -612,7 +635,8 @@ def api_avatar():
avatar_data = avatar_data.encode('latin-1')
if avatar_data[:3] != b'\xFF\xD8\xFF':
logger.warning(f'Non-JPEG avatar data for {username}')
open(sentinel, 'w').close()
with open(sentinel, 'w'):
pass
return '', 404
with open(cache_file, 'wb') as f:
+10 -10
View File
@@ -3,7 +3,7 @@ import json
import logging
import threading
from contextlib import contextmanager
from datetime import datetime, timedelta
from datetime import datetime, timedelta, timezone
from typing import Optional
import pymysql
@@ -114,12 +114,12 @@ def upsert_event(
target_detail: str,
description: str,
) -> tuple:
"""Insert or update a network event. Returns (id, is_new, consecutive_failures)."""
"""Insert or update a network event. Returns (id, is_new, consecutive_failures, ticket_id)."""
detail = target_detail or ''
with get_conn() as conn:
with conn.cursor() as cur:
cur.execute(
"""SELECT id, consecutive_failures FROM network_events
"""SELECT id, consecutive_failures, ticket_id FROM network_events
WHERE event_type=%s AND target_name=%s AND target_detail=%s
AND resolved_at IS NULL LIMIT 1""",
(event_type, target_name, detail),
@@ -134,7 +134,7 @@ def upsert_event(
WHERE id=%s""",
(new_count, description, existing['id']),
)
return existing['id'], False, new_count
return existing['id'], False, new_count, existing.get('ticket_id')
else:
cur.execute(
"""INSERT INTO network_events
@@ -142,7 +142,7 @@ def upsert_event(
VALUES (%s, %s, %s, %s, %s, %s)""",
(event_type, severity, source_type, target_name, detail, description),
)
return cur.lastrowid, True, 1
return cur.lastrowid, True, 1, None
def resolve_event(event_type: str, target_name: str, target_detail: str = '') -> None:
@@ -182,7 +182,7 @@ def get_active_events(limit: int = 200, offset: int = 0) -> list:
for r in rows:
for k in ('first_seen', 'last_seen'):
if r.get(k) and hasattr(r[k], 'isoformat'):
r[k] = r[k].isoformat()
r[k] = r[k].isoformat() + 'Z'
return rows
@@ -210,7 +210,7 @@ def get_recent_resolved(hours: int = 24, limit: int = 50) -> list:
for r in rows:
for k in ('first_seen', 'last_seen', 'resolved_at'):
if r.get(k) and hasattr(r[k], 'isoformat'):
r[k] = r[k].isoformat()
r[k] = r[k].isoformat() + 'Z'
return rows
@@ -252,7 +252,7 @@ def get_active_suppressions() -> list:
for r in rows:
for k in ('created_at', 'expires_at'):
if r.get(k) and hasattr(r[k], 'isoformat'):
r[k] = r[k].isoformat()
r[k] = r[k].isoformat() + 'Z'
return rows
@@ -267,7 +267,7 @@ def get_suppression_history(limit: int = 50) -> list:
for r in rows:
for k in ('created_at', 'expires_at'):
if r.get(k) and hasattr(r[k], 'isoformat'):
r[k] = r[k].isoformat()
r[k] = r[k].isoformat() + 'Z'
return rows
@@ -281,7 +281,7 @@ def create_suppression(
) -> int:
expires_at = None
if expires_minutes:
expires_at = datetime.utcnow() + timedelta(minutes=int(expires_minutes))
expires_at = datetime.now(timezone.utc) + timedelta(minutes=int(expires_minutes))
with get_conn() as conn:
with conn.cursor() as cur:
cur.execute(
+2 -2
View File
@@ -68,7 +68,7 @@ class DiagnosticsRunner:
f' echo "=== ip_route ===";'
f' ip route show dev {q} 2>/dev/null;'
f' echo "=== dmesg ===";'
f' dmesg 2>/dev/null | grep {q} | tail -50;'
f' dmesg 2>/dev/null | grep -F -- {q} | tail -50;'
f' echo "=== lldpctl ===";'
f' lldpctl 2>/dev/null || echo "lldpd not running";'
f' echo "=== end ==="'
@@ -78,7 +78,7 @@ class DiagnosticsRunner:
f'ssh -o StrictHostKeyChecking=accept-new -o ConnectTimeout=5 '
f'-o BatchMode=yes -o LogLevel=ERROR '
f'-o ServerAliveInterval=10 -o ServerAliveCountMax=2 '
f'root@{ip_q} \'{remote_cmd}\''
f'root@{ip_q} {shlex.quote(remote_cmd)}'
)
# ------------------------------------------------------------------
+46 -30
View File
@@ -12,7 +12,7 @@ import logging
import re
import shlex
import time
from datetime import datetime
from datetime import datetime, timezone
from typing import Dict, List, Optional
import requests
@@ -215,7 +215,10 @@ class TicketClient:
resp.raise_for_status()
data = resp.json()
if data.get('success'):
tid = data['ticket_id']
tid = data.get('ticket_id')
if not tid:
logger.warning(f'Ticket API success but no ticket_id in response: {data}')
return None
logger.info(f'Created ticket #{tid}: {title}')
return tid
if data.get('existing_ticket_id'):
@@ -377,7 +380,7 @@ class LinkStatsCollector:
f'ssh -o StrictHostKeyChecking=accept-new -o ConnectTimeout=5 '
f'-o BatchMode=yes -o LogLevel=ERROR '
f'-o ServerAliveInterval=10 -o ServerAliveCountMax=2 '
f'root@{ip} "{shell_cmd}"'
f'root@{ip} {shlex.quote(shell_cmd)}'
)
output = self.pulse.run_command(ssh_cmd)
if output is None:
@@ -615,7 +618,7 @@ class LinkStatsCollector:
return {
'hosts': result_hosts,
'unifi_switches': unifi_switches,
'updated': datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S UTC'),
'updated': datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M:%S UTC'),
}
def _compute_unifi_rates(self, raw: Dict[str, dict], now: float) -> Dict[str, dict]:
@@ -650,7 +653,7 @@ class LinkStatsCollector:
# Helpers
# --------------------------------------------------------------------------
def _now_utc() -> str:
return datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S UTC')
return datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M:%S UTC')
# --------------------------------------------------------------------------
@@ -674,6 +677,11 @@ class NetworkMonitor:
mon = self.cfg.get('monitor', {})
self.poll_interval = mon.get('poll_interval', 120)
self.fail_thresh = mon.get('failure_threshold', 2)
# UniFi gear (switches/APs) can take several minutes to rejoin after a
# firmware-update reboot, much longer than a Proxmox host link flap.
# Use a separate, more forgiving threshold so those momentary reboots
# don't cut tickets, while still catching genuine device failures.
self.unifi_fail_thresh = mon.get('unifi_failure_threshold', self.fail_thresh)
self.cluster_thresh = mon.get('cluster_threshold', 3)
self.cluster_name = mon.get('cluster_name', CLUSTER_NAME)
@@ -725,13 +733,13 @@ class NetworkMonitor:
db.check_suppressed(suppressions, 'interface', host, iface) or
db.check_suppressed(suppressions, 'host', host)
)
event_id, is_new, consec = db.upsert_event(
event_id, is_new, consec, ticket_id = db.upsert_event(
'interface_down', 'critical', 'prometheus',
host, iface,
f'Interface {iface} on {host} went link-down ({_now_utc()})',
)
if not sup and consec >= self.fail_thresh:
self._ticket_interface(event_id, is_new, host, iface, consec)
if not sup and consec >= self.fail_thresh and not ticket_id:
self._ticket_interface(event_id, host, iface, consec)
if host_has_regression:
hosts_with_regression.append(host)
@@ -741,13 +749,13 @@ class NetworkMonitor:
# Cluster-wide check only genuine regressions count
if len(hosts_with_regression) >= self.cluster_thresh:
sup = db.check_suppressed(suppressions, 'all', '')
event_id, is_new, consec = db.upsert_event(
event_id, is_new, consec, ticket_id = db.upsert_event(
'cluster_network_issue', 'critical', 'prometheus',
self.cluster_name, '',
f'{len(hosts_with_regression)} hosts reporting simultaneous interface failures: '
f'{", ".join(hosts_with_regression)}',
)
if not sup and is_new:
if not sup and (is_new or not ticket_id):
title = (
f'[{self.cluster_name}][auto][production][issue][network][cluster-wide] '
f'Multiple hosts reporting interface failures'
@@ -768,7 +776,7 @@ class NetworkMonitor:
db.resolve_event('cluster_network_issue', self.cluster_name, '')
def _ticket_interface(
self, event_id: int, is_new: bool, host: str, iface: str, consec: int
self, event_id: int, host: str, iface: str, consec: int
) -> None:
title = (
f'[{host}][auto][production][issue][network][single-node] '
@@ -786,7 +794,7 @@ class NetworkMonitor:
f'Please inspect the cable/SFP/switch port for {host}/{iface}.'
)
tid = self.tickets.create(title, desc, priority='2')
if tid and is_new:
if tid:
db.set_ticket_id(event_id, tid)
# ------------------------------------------------------------------
@@ -801,17 +809,17 @@ class NetworkMonitor:
name = d['name']
if not d['connected']:
sup = db.check_suppressed(suppressions, 'unifi_device', name)
event_id, is_new, consec = db.upsert_event(
event_id, is_new, consec, ticket_id = db.upsert_event(
'unifi_device_offline', 'critical', 'unifi',
name, d.get('type', ''),
f'UniFi {name} ({d.get("ip","")}) offline ({_now_utc()})',
)
if not sup and consec >= self.fail_thresh:
self._ticket_unifi(event_id, is_new, d)
if not sup and consec >= self.unifi_fail_thresh and not ticket_id:
self._ticket_unifi(event_id, d)
else:
db.resolve_event('unifi_device_offline', name, d.get('type', ''))
def _ticket_unifi(self, event_id: int, is_new: bool, device: dict) -> None:
def _ticket_unifi(self, event_id: int, device: dict) -> None:
name = device['name']
title = (
f'[{name}][auto][production][issue][network][single-node] '
@@ -828,31 +836,31 @@ class NetworkMonitor:
f'Please check power and cable connectivity.'
)
tid = self.tickets.create(title, desc, priority='2')
if tid and is_new:
if tid:
db.set_ticket_id(event_id, tid)
# ------------------------------------------------------------------
# Ping-only hosts (no node_exporter)
# ------------------------------------------------------------------
def _process_ping_hosts(self, suppressions: list) -> None:
def _process_ping_hosts(self, suppressions: list, ping_states: Dict[str, bool]) -> None:
for h in self.cfg.get('monitor', {}).get('ping_hosts', []):
name, ip = h['name'], h['ip']
reachable = self.pulse.ping(ip)
reachable = ping_states.get(name, False)
if not reachable:
sup = db.check_suppressed(suppressions, 'host', name)
event_id, is_new, consec = db.upsert_event(
event_id, is_new, consec, ticket_id = db.upsert_event(
'host_unreachable', 'critical', 'ping',
name, ip,
f'Host {name} ({ip}) unreachable via ping ({_now_utc()})',
)
if not sup and consec >= self.fail_thresh:
self._ticket_unreachable(event_id, is_new, name, ip, consec)
if not sup and consec >= self.fail_thresh and not ticket_id:
self._ticket_unreachable(event_id, name, ip, consec)
else:
db.resolve_event('host_unreachable', name, ip)
def _ticket_unreachable(
self, event_id: int, is_new: bool, name: str, ip: str, consec: int
self, event_id: int, name: str, ip: str, consec: int
) -> None:
title = (
f'[{name}][auto][production][issue][network][single-node] '
@@ -870,7 +878,7 @@ class NetworkMonitor:
f'Please check the host power, management interface, and network connectivity.'
)
tid = self.tickets.create(title, desc, priority='2')
if tid and is_new:
if tid:
db.set_ticket_id(event_id, tid)
# ------------------------------------------------------------------
@@ -879,6 +887,7 @@ class NetworkMonitor:
def _collect_snapshot(
self, iface_states: Dict[str, Dict[str, bool]],
unifi_devices: Optional[List[dict]] = None,
ping_states: Optional[Dict[str, bool]] = None,
) -> dict:
# Accept pre-fetched devices; fall back to empty list if unavailable
display_unifi = unifi_devices if unifi_devices is not None else []
@@ -907,7 +916,7 @@ class NetworkMonitor:
for h in self.cfg.get('monitor', {}).get('ping_hosts', []):
name, ip = h['name'], h['ip']
reachable = self.pulse.ping(ip, count=1, timeout=2)
reachable = (ping_states or {}).get(name, False)
hosts[name] = {
'ip': ip,
'interfaces': {},
@@ -918,7 +927,7 @@ class NetworkMonitor:
return {
'hosts': hosts,
'unifi': display_unifi,
'updated': datetime.utcnow().isoformat(),
'updated': datetime.now(timezone.utc).isoformat().replace('+00:00', 'Z'),
}
# ------------------------------------------------------------------
@@ -927,7 +936,7 @@ class NetworkMonitor:
def run(self) -> None:
logger.info(
f'Gandalf monitor started poll_interval={self.poll_interval}s '
f'fail_thresh={self.fail_thresh}'
f'fail_thresh={self.fail_thresh} unifi_fail_thresh={self.unifi_fail_thresh}'
)
while True:
try:
@@ -939,8 +948,14 @@ class NetworkMonitor:
# 2. Fetch UniFi devices once — used by both snapshot and alert processing
unifi_devices = self.unifi.get_devices()
# 3. Collect and store snapshot for dashboard
snapshot = self._collect_snapshot(iface_states, unifi_devices)
# 3a. Ping-only hosts once — shared by snapshot and alert processing
ping_states: Dict[str, bool] = {
h['name']: self.pulse.ping(h['ip'])
for h in self.cfg.get('monitor', {}).get('ping_hosts', [])
}
# 3b. Collect and store snapshot for dashboard
snapshot = self._collect_snapshot(iface_states, unifi_devices, ping_states)
db.set_state('network_snapshot', snapshot)
db.set_state('last_check', _now_utc())
@@ -956,7 +971,7 @@ class NetworkMonitor:
self._process_interfaces(iface_states, suppressions)
self._process_unifi(unifi_devices, suppressions)
self._process_ping_hosts(suppressions)
self._process_ping_hosts(suppressions, ping_states)
# Housekeeping: deactivate expired suppressions and purge old resolved events
db.cleanup_expired_suppressions()
@@ -967,6 +982,7 @@ class NetworkMonitor:
except Exception as e:
logger.error(f'Monitor loop error: {e}', exc_info=True)
time.sleep(30)
continue
time.sleep(self.poll_interval)
+1 -1
View File
@@ -220,7 +220,7 @@ function updateEventsTable(events, totalActive) {
? GANDALF_CONFIG.ticket_web_url : 'http://t.lotusguild.org/ticket/';
const ticket = e.ticket_id
? `<a href="${lt.escHtml(ticketBase)}${lt.escHtml(String(e.ticket_id))}" target="_blank"
class="ticket-link">#${e.ticket_id}</a>`
class="ticket-link">#${lt.escHtml(String(e.ticket_id))}</a>`
: '';
const supBadge = e.is_suppressed
? `<span class="lt-badge badge-suppressed" title="Alert suppressed">🔕 sup</span>`
+3 -2
View File
@@ -324,6 +324,7 @@
</div>
</div>
<div class="host-grid" id="host-grid">
{%- set has_global_sup = suppressions | selectattr('target_type', 'equalto', 'all') | list | length > 0 -%}
{% for name, host in snapshot.hosts.items() %}
{% set suppressed = suppressions | selectattr('target_name', 'equalto', name) | list %}
<div class="host-card host-card-{{ host.status }}" data-host="{{ name }}">
@@ -331,7 +332,7 @@
<div class="host-name-row">
<span class="host-status-dot dot-{{ host.status }}"></span>
<span class="host-name">{{ name }}</span>
{% if suppressed %}
{% if suppressed or has_global_sup %}
<span class="badge-suppressed" title="Suppressed">🔕</span>
{% endif %}
</div>
@@ -468,7 +469,7 @@
{% block scripts %}
<script>
// Start auto-refresh using saved settings interval (default 30 s)
const _savedInterval = (window.gandalfSettings && window.gandalfSettings.refreshInterval) || 30;
const _savedInterval = window.gandalfSettings?.refreshInterval ?? 30;
if (_savedInterval > 0) lt.autoRefresh.start(refreshAll, _savedInterval * 1000);
// When settings change, restart auto-refresh with new interval
+3 -2
View File
@@ -218,6 +218,7 @@ let _apiData = null;
function selectPort(el) {
const swName = el.dataset.switch;
const idx = parseInt(el.dataset.portIdx, 10);
if (_diagPollTimer) { clearInterval(_diagPollTimer); _diagPollTimer = null; }
document.querySelectorAll('.switch-port-block.selected')
.forEach(e => e.classList.remove('selected'));
el.classList.add('selected');
@@ -428,7 +429,7 @@ function renderInspector(data) {
const updEl = document.getElementById('inspector-updated');
if (updEl && data.updated) {
const updMs = new Date(data.updated + (data.updated.includes('Z') ? '' : 'Z'));
const updMs = new Date(_toIso(data.updated));
const ageMin = (Date.now() - updMs) / 60000;
const timeStr = updMs.toLocaleTimeString();
if (ageMin > 15) {
@@ -472,7 +473,7 @@ async function loadInspector() {
}
loadInspector();
const _inspInterval = (window.gandalfSettings && window.gandalfSettings.refreshInterval) || 60;
const _inspInterval = window.gandalfSettings?.refreshInterval ?? 60;
if (_inspInterval > 0) lt.autoRefresh.start(loadInspector, Math.max(_inspInterval, 15) * 1000);
window.onGandalfSettingsChanged = function(s) {
+15 -12
View File
@@ -36,7 +36,6 @@
{% block scripts %}
<script>
const escHtml = s => lt.escHtml(s);
const _toIso = s => s ? s.replace(' UTC', 'Z').replace(' ', 'T') : s;
// ── Formatting helpers ────────────────────────────────────────────
function fmtRate(bytesPerSec) {
@@ -373,14 +372,16 @@ function togglePanel(panel) {
if (title) title.setAttribute('aria-expanded', isCollapsed ? 'false' : 'true');
const id = panel.id;
if (id) {
const collapsed = JSON.parse(sessionStorage.getItem('linksCollapsed') || '{}');
let collapsed = {};
try { collapsed = JSON.parse(sessionStorage.getItem('linksCollapsed') || '{}'); } catch(_) {}
collapsed[id] = panel.classList.contains('collapsed');
sessionStorage.setItem('linksCollapsed', JSON.stringify(collapsed));
try { sessionStorage.setItem('linksCollapsed', JSON.stringify(collapsed)); } catch(_) {}
}
}
function restoreCollapseState() {
const collapsed = JSON.parse(sessionStorage.getItem('linksCollapsed') || '{}');
let collapsed = {};
try { collapsed = JSON.parse(sessionStorage.getItem('linksCollapsed') || '{}'); } catch(_) {}
for (const [id, isCollapsed] of Object.entries(collapsed)) {
const panel = document.getElementById(id);
if (!panel) continue;
@@ -508,9 +509,11 @@ function collapseAll() {
if (btn) btn.textContent = '[+]';
if (title) title.setAttribute('aria-expanded', 'false');
});
sessionStorage.setItem('linksCollapsed', JSON.stringify(
Object.fromEntries([...document.querySelectorAll('.link-host-panel')].map(p => [p.id, true]))
));
try {
sessionStorage.setItem('linksCollapsed', JSON.stringify(
Object.fromEntries([...document.querySelectorAll('.link-host-panel')].map(p => [p.id, true]))
));
} catch(_) {}
}
function expandAll() {
@@ -521,13 +524,13 @@ function expandAll() {
if (btn) btn.textContent = '[]';
if (title) title.setAttribute('aria-expanded', 'true');
});
sessionStorage.setItem('linksCollapsed', '{}');
try { sessionStorage.setItem('linksCollapsed', '{}'); } catch(_) {}
}
// ── Stale data warning ────────────────────────────────────────────
function checkLinksStale(updatedStr) {
if (!updatedStr) return;
const age = (Date.now() - new Date(updatedStr + (updatedStr.includes('Z') ? '' : 'Z'))) / 1000;
const age = (Date.now() - new Date(_toIso(updatedStr))) / 1000;
let banner = document.getElementById('links-stale-banner');
if (age > 120) {
if (!banner) {
@@ -549,14 +552,14 @@ function checkLinksStale(updatedStr) {
async function loadLinks() {
try {
const data = await lt.api.get('/api/links');
if (!data.hosts && !data.unifi_switches) {
if ((!data.hosts || !Object.keys(data.hosts).length) && (!data.unifi_switches || !Object.keys(data.unifi_switches).length)) {
document.getElementById('links-container').innerHTML =
'<div class="link-no-data">No link data yet — monitor has not completed a full cycle.</div>';
return;
}
const updEl = document.getElementById('links-updated');
if (updEl && data.updated) {
updEl.textContent = 'Updated: ' + new Date(data.updated + (data.updated.includes('Z') ? '' : 'Z')).toLocaleTimeString();
updEl.textContent = 'Updated: ' + new Date(_toIso(data.updated)).toLocaleTimeString();
}
renderLinks(data);
checkLinksStale(data.updated);
@@ -568,7 +571,7 @@ async function loadLinks() {
}
loadLinks();
const _linksInterval = (window.gandalfSettings && window.gandalfSettings.refreshInterval) || 60;
const _linksInterval = window.gandalfSettings?.refreshInterval ?? 60;
if (_linksInterval > 0) lt.autoRefresh.start(loadLinks, Math.max(_linksInterval, 15) * 1000);
window.onGandalfSettingsChanged = function(s) {
+1 -1
View File
@@ -51,7 +51,7 @@
<label class="lt-label" for="s-reason">Reason <span class="required">*</span></label>
<input type="text" class="lt-input" id="s-reason" name="reason"
placeholder="e.g. Planned switch maintenance, replacing SFP on large1/enp43s0"
required>
required aria-required="true">
</div>
</div>
+6
View File
@@ -36,6 +36,12 @@ class TestBuildSshCommand:
cmd = DiagnosticsRunner.build_ssh_command('10.0.0.1', 'eth0')
assert 'ethtool' in cmd
def test_dmesg_uses_fixed_string_grep(self):
# grep -F prevents iface names with dots (e.g. eth0.1) being treated as
# regex wildcards; -- prevents leading - from being parsed as a flag
cmd = DiagnosticsRunner.build_ssh_command('10.0.0.1', 'eth0')
assert 'grep -F --' in cmd
# ── parse_output ─────────────────────────────────────────────────────────────