Commit Graph
43 Commits
Author SHA1 Message Date
jaredandClaude Opus 5.5 aa7341af06 Topology: a live page drawn from the real gear
Lint / JS (eslint) (push) Successful in 21s
Lint / Python (flake8) (push) Successful in 1m20s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 1m0s
Test / Python Tests (pytest) (push) Successful in 1m48s
Lint / Python (flake8) (pull_request) Successful in 1m44s
Lint / JS (eslint) (pull_request) Successful in 30s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 1m42s
Test / Python Tests (pytest) (pull_request) Successful in 1m16s
The dashboard's hand-drawn diagram was out of date. /topology replaces it
with the network as the switches report it, every device drawn to scale
from its faceplate or cutsheet front, and one cable per physical link
leaving the port it is plugged into.

- topology.py builds the graph: switch-to-switch cables seen from both ends
  are merged, servers get one cable per NIC, the gateway's internet port
  goes to the modem and on to the internet, unknown neighbours are stubs,
  and gear with no live cable is drawn where it was last seen.
- Ports named after a different device than the one plugged in are flagged.
- Cables show medium, speed, state and traffic; click one for its details,
  hover a device to trace its path to the internet; pan, zoom and filters.
- The monitor now keeps host_macs (MAC to host NIC) in link_stats.
- The old dashboard diagram and its JS and CSS are removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 19:30:59 -05:00
jaredandClaude Opus 5.5 0921e82c54 Dashboard: flag stale alerts and tickets left open
Lint / JS (eslint) (push) Successful in 19s
Lint / Python (flake8) (push) Successful in 1m7s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 52s
Test / Python Tests (pytest) (push) Successful in 44s
Lint / JS (eslint) (pull_request) Successful in 7s
Lint / Python (flake8) (pull_request) Successful in 50s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 52s
Test / Python Tests (pytest) (pull_request) Successful in 1m35s
Gandalf only comments on tickets, so a ticket can stay open long after its
alert cleared (pbs's ticket sat open for a month), and an alert can stay
active for days once nobody is looking.

- The monitor reads Tinker's open ticket list (tickets_api.php, read scope)
  every 15 minutes and saves it. A failed read keeps the previous list.
- Active alerts get a stale badge after monitor.stale_alert_days (3), and a
  "ticket closed" badge when their ticket was closed while still active.
- A Needs Follow-up panel lists open tickets whose every alert cleared at
  least monitor.stale_ticket_hours (24) ago.
- A "TO FOLLOW UP" status chip counts both. If the open ticket list is over
  2 hours old, the ticket flags are left off rather than guessed.
- Replaced an &mdash; in the alert count notice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 15:57:58 -05:00
jaredandClaude Opus 5.5 379e3be365 WAN tickets: one per incident, notice UniFi updates, maintenance button
Lint / JS (eslint) (push) Successful in 42s
Lint / Python (flake8) (push) Successful in 1m35s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 1m3s
Test / Python Tests (pytest) (push) Successful in 1m5s
Lint / Python (flake8) (pull_request) Successful in 58s
Lint / JS (eslint) (pull_request) Successful in 8s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 1m1s
Test / Python Tests (pytest) (pull_request) Successful in 1m10s
A UniFi Network update on 2026-11-12 dropped the UDM's WAN, Smart Power
power-cycled the modem, and Gandalf opened two tickets for the one event.

- Modem and WAN reports from one check cycle open a single ticket, led by
  the most urgent, with the rest listed under "Also seen in this incident".
  A report while the modem has an open alert, or one that ended under 30
  minutes ago, becomes a comment on that ticket instead.
- UniFi update log entries (NETWORK_UPDATED, FIRMWARE_UPDATED_V2 and the
  like) add a note to modem/WAN tickets from 20 minutes before the update
  to 15 after, and hold new modem/WAN tickets for 15 minutes.
- A Maintenance button for admins starts a global suppression for 30 min
  to 4 hours, with a banner on every page and End now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 12:53:22 -05:00
jaredandClaude Opus 5.5 3834546f38 Mark host interfaces as unused
Lint / JS (eslint) (push) Successful in 44s
Lint / Python (flake8) (push) Successful in 1m40s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 1m18s
Lint / JS (eslint) (pull_request) Successful in 10s
Security / Python Security (bandit) (pull_request) Successful in 1m13s
Test / Python Tests (pytest) (pull_request) Successful in 1m11s
Lint / Python (flake8) (pull_request) Successful in 50s
Test / Python Tests (pytest) (push) Successful in 1m1s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Ports that are never meant to be up (compute-storage-01 nic1-3, large1's
spare NIC and USB adapter) kept their hosts "degraded" and were the source
of tonight's false alerts. An admin can now mark an interface unused from
the device page, with an optional reason. Unused interfaces raise no
link-down alerts, don't count toward the cluster-wide alert or the host's
status, and show greyed as "unused" on the dashboard, device page and rack.
Marking one closes any open link-down alert on it without a recovery note.

New table unused_interfaces, created with the cutsheet tables.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 17:19:45 -05:00
jaredandClaude Opus 5.5 d234e36e3f Tickets: don't adopt a Closed ticket Tinker couldn't reopen
Lint / JS (eslint) (push) Successful in 14s
Lint / Python (flake8) (push) Successful in 57s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 55s
Lint / Python (flake8) (pull_request) Successful in 1m31s
Test / Python Tests (pytest) (push) Successful in 1m51s
Lint / JS (eslint) (pull_request) Successful in 7s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 1m22s
Test / Python Tests (pytest) (pull_request) Successful in 1m46s
Tinker's create response says what it did (action). For recurrence_noted it
matched a Closed ticket it could not reopen unattended: that ticket isn't
tracking the alert (this is what hid micro1 enp3s0 behind an old reboot
ticket), so Gandalf now logs it and keeps no ticket for the alert. Each such
create adds a "recurred" comment to the closed ticket, so the same title is
retried once a day rather than every poll. Other actions are logged.

See tinker_tickets#116 and #115.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 01:52:53 -05:00
jaredandClaude Opus 5.5 8edaa78ad7 Revert #31: open link-down alerts from a monitor glitch were revived and ticketed
Lint / JS (eslint) (push) Successful in 12s
Lint / Python (flake8) (push) Successful in 1m47s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 1m15s
Lint / JS (eslint) (pull_request) Successful in 7s
Lint / Python (flake8) (pull_request) Successful in 48s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Test / Python Tests (pytest) (push) Successful in 1m41s
Security / Python Security (bandit) (pull_request) Successful in 1m32s
Test / Python Tests (pytest) (pull_request) Successful in 1m6s
#31 treated any open interface_down alert as previously up. Production had
six such alerts all created in the same second on 2026-03-02 (a monitor
glitch, on ports that were never in use: large1 enp36s0 and a USB adapter,
compute-storage-01 nic1-3, micro1 enp3s0). Reviving them opened per-host
tickets and a P1 cluster-wide ticket. Back to the previous behaviour while a
safer fix is worked out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 01:11:00 -05:00
jaredandClaude Opus 5.5 d07787ee38 Monitor: keep checking interfaces whose link-down alert is still open
Lint / JS (eslint) (push) Successful in 8s
Lint / Python (flake8) (push) Successful in 49s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 53s
Test / Python Tests (pytest) (push) Successful in 1m3s
Lint / Python (flake8) (pull_request) Successful in 55s
Lint / JS (eslint) (pull_request) Successful in 6s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 1m22s
Test / Python Tests (pytest) (pull_request) Successful in 1m27s
After the interface baseline is reset, a port that is already down reads as
"first seen down", which neither alerts nor resolves. An alert opened before
the reset was then stuck open without updates: micro1's enp3s0 had sat at
one failed check since March with no ticket. An interface with an open
interface_down alert is now treated as previously up, so the alert keeps
counting, tickets and resolves as normal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 01:02:53 -05:00
jaredandClaude Opus 5.5 619a2c913d Remove em and en dashes from the UI, code and docs
Lint / JS (eslint) (push) Successful in 7s
Lint / Python (flake8) (push) Successful in 57s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 57s
Lint / JS (eslint) (pull_request) Successful in 10s
Lint / Python (flake8) (pull_request) Successful in 51s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 1m0s
Test / Python Tests (pytest) (push) Successful in 1m3s
Test / Python Tests (pytest) (pull_request) Successful in 52s
Page titles read "Rack | GANDALF", ranges use a hyphen (U26-42,
1-100 of 300), empty cells show "-", and dash-separated phrases use a
colon, comma or full stop. In the shared base.js only the on-screen strings
changed (boot line, placeholders, offline banner and toast).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 23:37:56 -05:00
jaredandClaude Opus 5.5 c262e5ecc8 Inventory links, cutsheet marks on faceplates, and single page titles
Lint / JS (eslint) (push) Successful in 7s
Lint / Python (flake8) (push) Successful in 54s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 1m1s
Lint / JS (eslint) (pull_request) Successful in 19s
Test / Python Tests (pytest) (push) Successful in 2m6s
Security / Python Security (bandit) (pull_request) Successful in 1m18s
Lint / Python (flake8) (pull_request) Successful in 53s
Test / Python Tests (pytest) (pull_request) Successful in 1m1s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Device pages get an Inventory button: it opens the device's HomeBox record
once an admin pastes its URL (stored in the new device_links table; only
URLs on inventory.base_url are accepted), and searches the inventory for
the model until then. Tickets link the record too.

Faceplate ports with a documented cable get a corner dot: green matches,
red mismatch, grey documented but unconfirmed. The port's tooltip names
the cable, and a legend toggle (remembered per browser) hides them.

Page titles no longer repeat GANDALF ("Alert History - GANDALF - GANDALF").

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 20:47:14 -05:00
jaredandClaude Opus 5.5 9d561db703 Add an alert history page and tie /health staleness to the poll interval
Lint / JS (eslint) (push) Successful in 6s
Lint / Python (flake8) (push) Successful in 1m56s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 2m3s
Test / Python Tests (pytest) (push) Successful in 1m0s
Lint / Python (flake8) (pull_request) Successful in 48s
Lint / JS (eslint) (pull_request) Successful in 6s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 2m7s
Test / Python Tests (pytest) (pull_request) Successful in 2m6s
/events lists every alert from the last 24 hours to 90 days, open or
cleared, with how long each lasted, failed checks and its ticket. Filters
(device, type, severity, state, range) live in the URL, so device pages
and tickets link straight to a device's history. A "Recurring" table
lists alerts that fired more than once on the same target. On phones the
rows become cards.

/api/events/history backs it (100 per page). The existing /api/events is
unchanged.

/health and the dashboard's monitor indicator now call the monitor
stalled after five missed polls (at least 10 minutes) instead of a fixed
20, and /health reports the monitor's age.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 20:29:09 -05:00
jaredandClaude Opus 5.5 437f7a415f Add recovery notes and fuller context to tickets
Lint / JS (eslint) (push) Successful in 7s
Lint / Python (flake8) (push) Successful in 1m26s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 1m30s
Lint / Python (flake8) (pull_request) Successful in 52s
Test / Python Tests (pytest) (push) Successful in 1m4s
Lint / JS (eslint) (pull_request) Successful in 7s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 1m28s
Test / Python Tests (pytest) (pull_request) Successful in 2m0s
When an alert that opened a ticket clears, the monitor now comments on the
ticket (when it cleared, how long it lasted, failed checks, a link back).
It never closes or changes a ticket's status. Failed notes retry each
cycle for 24 hours; a 401/403 from Tinker stops them until restart.

Tickets are now Markdown with a "What to check" list and, where known:
the switch port the device plugs into (kept from the last time it was
seen, so a dead host still shows it), cutsheet cables and their path,
how often the alert fired in the last 7/30 days, other open alerts on the
device, and links into Gandalf (gandalf.public_url). Titles are unchanged
so Tinker's duplicate detection still works.

network_events gains recovery_notified, added at monitor startup; rows
that cleared before the upgrade count as already notified.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 19:44:50 -05:00
jaredandClaude Opus 5.5 86aa29b378 Cutsheet trace mode and mismatch alerts
Lint / Python (flake8) (push) Successful in 1m8s
Security / Python Security (bandit) (push) Successful in 56s
Test / Python Tests (pytest) (push) Successful in 43s
Lint / Python (flake8) (pull_request) Successful in 50s
Lint / JS (eslint) (pull_request) Successful in 30s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 55s
Test / Python Tests (pytest) (pull_request) Successful in 50s
Lint / JS (eslint) (push) Successful in 16m17s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Trace a cord: pick a panel port, press Start, and unplug the cord at the
panel. The page polls a new admin-only endpoint that reads every switch
port's state from UniFi (cached 2 s), shows the one port that dropped
with a button to record the cable, and confirms when it's plugged back
in. Several ports dropping at once, or none within 3 minutes, is
explained rather than guessed at.

Mismatch alerts: the monitor runs the cutsheet check every poll. Each
mismatched installed run (wrong neighbor, no link, missing port, port
conflict) keeps one cable_mismatch event on its switch, resolved when
fixed, and opens a P4 ticket after cutsheet.mismatch_ticket_hours
(default 24, 0 never). Suppressing the switch or port stops the ticket.

Also fixes buttons with the hidden attribute still showing, which left a
Delete button on the new-cable form.

Closes #12

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 16:54:57 -05:00
jaredandClaude Opus 5.5 7e588e5b02 Keep every switch port and show SFP modules
Security / Python Security (bandit) (push) Successful in 1m50s
Test / Python Tests (pytest) (push) Successful in 1m52s
Lint / Python (flake8) (pull_request) Successful in 1m20s
Lint / Python (flake8) (push) Successful in 1m16s
Lint / JS (eslint) (pull_request) Successful in 10s
Lint / JS (eslint) (push) Successful in 15s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 1m5s
Test / Python Tests (pytest) (pull_request) Successful in 1m7s
Ports were keyed by their UniFi name, so when two ports shared a name the
later one replaced the earlier. The 24-port switch lost port 13 this way
(ports 13 and 14 are both "Office TV"). A repeated name is now kept as
"<name> (Port N)".

Traffic rates are now matched to the previous poll by port number rather
than name. Names are re-read from UniFi every poll, so a port renamed
there keeps its rates instead of losing them for a cycle.

Collect what UniFi reports about each SFP cage: whether a module is
present, its type (DAC or optic), part, serial, fault flags, and DOM
readings when the module has them. The faceplate draws empty cages,
optics, DACs and unidentified modules differently and outlines a faulted
module in red. The port panel gets an SFP Module section. Both ends of a
DAC report the same cable serial, so each end links to the other.

Closes #7
Closes #8

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 14:37:08 -05:00
jaredandClaude Opus 5.5 178a03e898 Give hosts longer before ticketing them as gone
Lint / JS (eslint) (push) Successful in 16s
Lint / Python (flake8) (push) Successful in 1m23s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 1m49s
Lint / Python (flake8) (pull_request) Successful in 54s
Test / Python Tests (pytest) (push) Successful in 1m35s
Lint / JS (eslint) (pull_request) Successful in 12s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 1m42s
Test / Python Tests (pytest) (pull_request) Successful in 2m3s
A host's node_exporter stops as soon as a shutdown begins and a normal
reboot takes about 3-5 minutes, so the 2-check failure_threshold cut a
ticket for planned reboots. Add monitor.host_failure_threshold, default 5,
for hosts with no metrics and ping-only hosts. Interface link-down alerts
keep the shorter threshold, and the dashboard still shows the host as down
straight away.

Closes #14

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 19:35:50 -05:00
jaredandClaude Opus 5.5 bdcfb76c25 Move the port inspector onto device pages
Lint / JS (eslint) (push) Successful in 1m14s
Lint / Python (flake8) (push) Successful in 2m12s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 1m3s
Test / Python Tests (pytest) (push) Successful in 1m11s
Lint / Python (flake8) (pull_request) Successful in 1m1s
Lint / JS (eslint) (pull_request) Successful in 10s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 56s
Test / Python Tests (pytest) (pull_request) Successful in 1m5s
Replace the all-switches Inspector page with a per-device front panel,
port table and detail panel. /inspector now redirects to the dashboard,
and /device/<name>#port-<n> links directly to a port.

Restore neighbor names after UniFi's LLDP change: read local_port_idx
and resolve neighbors by MAC using Prometheus NIC data, UniFi devices
and the wired client table. This also brings back Run Diagnostics on
server ports, which depends on a named neighbor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 10:11:11 -05:00
jaredandClaude Opus 5.5 e32499b7f6 Add AT&T modem monitoring and device pages; replace Link Debug
Lint / Python (flake8) (pull_request) Successful in 2m6s
Lint / JS (eslint) (pull_request) Successful in 18s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 2m11s
Test / Python Tests (pytest) (pull_request) Successful in 2m20s
Lint / JS (eslint) (push) Successful in 10s
Lint / Python (flake8) (push) Successful in 1m58s
Lint / Notify on failure (push) Skipped
Security / Python Security (bandit) (push) Successful in 1m58s
Lint / Deploy (push) Successful in 4s
Test / Python Tests (pytest) (push) Successful in 58s
Modem and WAN:
- Poll the BGW320 status pages read-only for light levels, PON and WAN
  state, with a WAN / Fiber panel, 7-day Rx trend, and alerts on WAN down,
  optical limits, and Rx drift
- Read WAN outages and Smart Power restarts from the UniFi system log,
  since outages can be over between 5 minute polls

Device pages:
- Add device search to the dashboard and a page per host, switch, router,
  AP, PDU, or modem
- Move all Link Debug data into expandable rows there; /links redirects
- Collect UDM ports and UniFi device details (firmware, load, WAN, outlets)
- Reorder the dashboard

Monitoring fixes:
- Alert when a configured host stops reporting: its series disappear
  rather than reporting down, so a dead host never raised an alert
- Alert on Prometheus outages instead of blanking the dashboard
- Treat a failed Pulse ping as unknown rather than host unreachable
- Stop new switch ports reporting lifetime bytes as one interval's rate
- Run database sessions in UTC: other server time zones shifted displayed
  ages and timed suppression expiries

Also log every avatar 404 reason, add a favicon, and correct the WAN link
labels (1G on Port 9, not 10G SFP+).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 23:40:17 -05:00
jaredandClaude Sonnet 5 42cf754796 Add separate unifi_failure_threshold to avoid false tickets on reboots
Lint / Python (flake8) (push) Failing after 1m4s
Lint / JS (eslint) (push) Successful in 5s
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Failing after 37s
Test / Python Tests (pytest) (push) Failing after 33s
Lint / Notify on failure (push) Successful in 2s
UniFi switches/APs take several minutes to rejoin after a firmware-update
reboot (observed AP taking 12 min on the scheduled 3am update), longer than
the 2-consecutive-check (~4 min) threshold shared with host link-down
detection. Give UniFi device offline detection its own, more forgiving
threshold so momentary maintenance reboots don't cut tickets.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UB2D82eYNzD1bnMHhFPjRr
2026-09-04 20:59:43 -04:00
jaredandClaude Sonnet 4.6 28fd5aad14 Guard ticket creation against duplicates using event's existing ticket_id
Lint / Python (flake8) (push) Failing after 1m11s
Lint / JS (eslint) (push) Successful in 6s
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Failing after 32s
Test / Python Tests (pytest) (push) Failing after 34s
Lint / Notify on failure (push) Successful in 5s
upsert_event now returns ticket_id (4th element) so callers can skip
ticket creation when one already exists. This prevents calling the ticket
API every poll cycle for ongoing issues while still retrying if the
previous creation attempt failed (ticket_id stays NULL until success).

Cluster events use (is_new or not ticket_id) so they too get retried
on failure rather than relying solely on is_new.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-14 11:09:50 -04:00
jaredandClaude Sonnet 4.6 0e852ebd11 Fix ping-only hosts polled twice per cycle with inconsistent parameters
Lint / Python (flake8) (push) Skipped
Lint / JS (eslint) (push) Skipped
_collect_snapshot called pulse.ping(count=1) independently from
_process_ping_hosts which called pulse.ping(count=3). This doubled
network load and could show a host as 'up' in the dashboard while
simultaneously firing an 'unreachable' alert, or vice versa.

Now ping_states is computed once in run() using the alert-quality
parameters (count=3) and shared by both snapshot and alert processing.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-13 23:13:43 -04:00
jaredandClaude Sonnet 4.6 8aaaa89f03 Replace deprecated datetime.utcnow() with datetime.now(timezone.utc)
Lint / Python (flake8) (push) Skipped
Lint / JS (eslint) (push) Skipped
datetime.utcnow() is deprecated in Python 3.12 and removed in 3.13.
Replace all four call sites with timezone-aware equivalents so the
codebase is ready for Python 3.12+.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-13 15:34:41 -04:00
jaredandClaude Sonnet 4.6 0693ad56f9 Fix monitor loop double-sleep on error; add grep -F regression test
Lint / Python (flake8) (push) Skipped
Lint / JS (eslint) (push) Skipped
On exception the monitor slept 30s inside the except block then fell
through to time.sleep(poll_interval), giving a 150s recovery gap instead
of 30s. Adding continue after the error sleep fixes this.

Also adds a regression test asserting dmesg filtering uses grep -F --
so a future refactor cannot silently reintroduce the regex wildcard bug.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-13 13:16:43 -04:00
jaredandClaude Sonnet 4.6 5c2a92eaee Remove unused is_new parameter from ticket helper methods
After fixing the is_new guard bug, is_new is no longer used inside
_ticket_interface, _ticket_unifi, or _ticket_unreachable. Drop it from
their signatures and call sites.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-13 11:10:32 -04:00
jaredandClaude Sonnet 4.6 f942e96c24 Fix ticket_id never stored when fail_thresh>1; guard sessionStorage JSON.parse
Lint / Python (flake8) (push) Skipped
Lint / JS (eslint) (push) Skipped
monitor.py: _ticket_interface/_ticket_unifi/_ticket_unreachable all used
`if tid and is_new` to guard db.set_ticket_id(). Since is_new is True only
on the first upsert (consec=1) but tickets are created at consec>=fail_thresh
(default 2), is_new is always False when the ticket is created, so the
ticket link never appeared in the UI. Changed to `if tid:`.

links.html: JSON.parse(sessionStorage.getItem(...)) in togglePanel and
restoreCollapseState had no try-catch. Corrupt/stale session storage would
throw an uncaught SyntaxError. Also wrapped all sessionStorage.setItem
calls in try-catch to defend against storage-full / private-browsing errors.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-11 23:45:20 -04:00
jaredandClaude Sonnet 4.6 2710eaa118 fix: SSH shell quoting bug breaks ethtool collection; ticket_id KeyError
Lint / Python (flake8) (push) Skipped
Lint / JS (eslint) (push) Skipped
monitor.py _ssh_batch(): the remote command was wrapped in double-quotes
(f'root@{ip} "{shell_cmd}"') but shell_cmd itself contains double-quoted
echo sentinels ("___IFACE:eth0___"). When Pulse's shell parses the full
ssh invocation, the nested double-quotes cause mis-parsing — the remote
command is split incorrectly, silently breaking all ethtool/SFP DOM
collection. Fix: use shlex.quote(shell_cmd) so the entire remote command
is single-quoted, leaving inner double-quotes untouched.

TicketClient.create(): data['ticket_id'] raises KeyError if the Tinker
Tickets API returns success=true without a ticket_id field (malformed
response). Use data.get('ticket_id') with an explicit warning log.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-11 13:41:09 -04:00
jaredandClaude Sonnet 4.6 9c011cfce1 fix: UTC timezone suffix missing from all isoformat() timestamp outputs
db.py returned all datetime columns (first_seen, last_seen, resolved_at,
created_at, expires_at) as bare ISO strings like "2026-03-14T14:14:21"
with no timezone marker. Per the ECMAScript spec, new Date() on a
datetime string without timezone treats it as LOCAL time, not UTC.
This made lt.time.ago() and stale-detection wrong for any user whose
browser is not in UTC — event ages and stale warnings would be off by
the client's UTC offset.

monitor.py had the same issue on the network_snapshot 'updated' field.

Fix: append 'Z' to all isoformat() calls (UTC datetimes confirmed by
MySQL server timezone and _now_utc() pattern used throughout codebase).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-11 13:28:49 -04:00
jaredandClaude Sonnet 4.6 d3616badb2 fix: LLDP input validation, mgmt_ip early validation, poll timer cleanup, monitor backoff
- app.py: validate server_name from LLDP with fullmatch before use in logs/lookups (prevents log injection)
- app.py: validate each mgmt_ip candidate before assigning host_ip (avoids assigning non-IP string that then fails later check)
- app.py: log actual exception in link_stats JSON parse error
- inspector.html: clear _diagPollTimer in closePanel() so timer doesn't orphan when panel is closed mid-poll
- monitor.py: sleep 30s after a monitor loop exception before resuming normal poll interval

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-11 08:45:28 -04:00
jaredandClaude Sonnet 4.6 705d06ca5b fix: diagnostic rate limiting, lock-held ownership check, iface name length cap
- app.py: add per-user diagnostic rate limit (5/min) enforced atomically under _diag_lock
- app.py: move diagnostic job ownership check inside _diag_lock to close TOCTOU window; snapshot result before releasing lock
- monitor.py: cap interface name regex to 15 chars (Linux IFNAMSIZ limit)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-11 08:42:50 -04:00
jaredandClaude Sonnet 4.6 1daa9ab475 security: harden exception exposure, SSL config, and Pulse response parsing
- app.py: replace raw str(e) in diagnostic _run() with generic client message; log internally only
- app.py: /health endpoint no longer leaks exception strings to unauthenticated callers; errors logged server-side
- monitor.py: UniFi SSL verification now defaults True, configurable via config.json unifi.verify_ssl; urllib3 warning suppression scoped to verify=False only (removed global disable)
- monitor.py: Pulse execution_id extracted with .get() + explicit None check to avoid KeyError on malformed response
- monitor.py: interface name regex drops '@' (not a valid kernel interface char) to match app.py and fix inconsistency

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-11 08:40:25 -04:00
jaredandClaude Sonnet 4.6 17d273d792 arch+security: route all server contact through Pulse, harden SSH
Architecture:
- Remove direct subprocess ping from Gandalf; add PulseClient.ping()
  which runs the ping via the Pulse worker instead
- Remove standalone ping() function and subprocess import from monitor.py
- Add self.pulse alias to NetworkMonitor for convenience
- Both _process_ping_hosts() and snapshot builder now use self.pulse.ping()

Security:
- Change StrictHostKeyChecking=no → accept-new in both SSH command
  builders (monitor.py _ssh_batch, diagnose.py build_ssh_command).
  The Pulse worker's known_hosts is now authoritative; host keys are
  recorded on first connection and verified on all subsequent ones.
  MITM attacks after initial key exchange are now detectable.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-10 23:58:16 -04:00
jaredandClaude Sonnet 4.6 032d0db81e Add LDAP avatar photos, UX polish, and TDS component upgrades
Lint / Python (flake8) (push) Skipped
Lint / JS (eslint) (push) Skipped
- Add /api/avatar endpoint querying lldap for user jpegPhoto; disk cache
  with sentinel pattern avoids repeat LDAP hits for users without photos
- Add ldap3 dependency and ldap config block to config.json
- Wire lt-avatar img overlay in base.html with capture-phase error
  fallback (lt-avatar-img-err) to reveal initials when image is absent
- Fix lt-avatar CSS shim: position:relative + absolute inset on img
  (local base.css was missing these; added to style.css)
- Replace all empty-state paragraphs with proper lt-empty-state markup
  (icon + title + body) across index, suppressions, inspector, app.js
- Add lt-spinner--cyan next to refresh button; shows during refreshAll()
- Replace inspector panel-section-title with lt-divider throughout
- Add data-tooltip attributes to SFP DOM metrics, TX/RX/Carrier/Duplex/
  Auto-neg/Error labels in links.html and inspector panel
- Add tooltips to events table column headers (Sev, First Seen, Failures)
- Fix links.html host panel timestamp (was reading sample.updated which
  is always undefined; now uses data.updated)
- Fix UniFi status text casing (Online→ONLINE to match server render)
- Remove dead topo-status-* class manipulation from updateTopology()
- Always render alert-count-badge; toggle display:none when count is 0
- Fix double UniFi get_devices() call in monitor.py run loop
- Fix chip-critical animation (was using green pulse-glow; now red)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-30 21:09:56 -04:00
jaredandClaude Sonnet 4.6 1767c12f1b Fix field name mismatches, add events filter, in-place suppression refresh
- links.html: fix all field name bugs (auto_negotiation→autoneg, full_duplex,
  tx/rx_errors/drops_per_sec→_rate, tx/rx_bytes_per_sec→_rate, poe_total_w/poe_max_w
  computed from ports, renderUnifiSwitches uses top-level updated timestamp)
- suppressions.html: in-place DOM refresh after create/remove (no page reload),
  datalist autocomplete for target names, form reset after submit
- inspector.html: ESC key closes detail panel via lt.keys.on
- index.html: events filter bar with search input + severity pills (All/Critical/Warning),
  MutationObserver re-applies filter after dynamic updates
- style.css: g-section-actions, events-filter-bar, sev-pills layout
- app.js/db.py/monitor.py: carry forward prior session fixes (Promise.allSettled,
  daemon_ok, stale connection handling, double Prometheus call, self.cfg fix)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-19 23:35:02 -04:00
jaredandClaude Sonnet 4.6 eb9fb3f88f Exclude LXC IPs from link stats collection
Add links_exclude_ips to monitor config; collect() skips any Prometheus
instance whose IP is in that list, preventing LXC containers from
appearing on the links/inspector pages as phantom hosts.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-17 20:39:47 -04:00
jaredandClaude Sonnet 4.6 ee0ea8fd47 Fix host filtering: only show/monitor configured hosts; add PBS
- _collect_snapshot() and _process_interfaces() now skip any Prometheus
  instance not explicitly listed in config.json hosts[]. LXC app servers
  (postgresql, matrix, etc.) report node_exporter metrics but are not
  infrastructure hosts Gandalf should display or alert on.
- Add PBS (10.10.10.3) to config hosts[] with prometheus_instance;
  remove from ping_hosts (node_exporter already running on PBS, now
  added to Prometheus scrape config as job pbs-node).
- The _instance_map membership check is now consistent across snapshot,
  alerting, and ethtool SSH collection.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-17 17:17:40 -04:00
jaredandClaude Sonnet 4.6 95ec211037 Fix: only SSH into explicitly configured hosts for ethtool collection
LinkStatsCollector.collect() was SSHing into every host reporting
node_network_* metrics to Prometheus, including unrelated app servers
like postgresql and matrix. Add instance_map membership check so ethtool
collection via Pulse only runs on hosts defined in config.json.

Prometheus metrics (traffic rates, errors) are still collected for all
instances — only the SSH/ethtool step is gated.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-15 18:35:21 -04:00
jaredandClaude Sonnet 4.6 253bc568d6 Improve Pulse execution reliability: retry logic, better logging, SSH hardening
monitor.py / diagnose.py PulseClient.run_command:
- Add automatic single retry on submit failure, explicit Pulse failure
  (status=failed/timed_out), and poll timeout — handles transient SSH
  or Pulse hiccups without dropping the whole collection cycle
- Log execution_id and full Pulse URL on every failure so failed runs
  can be found in the Pulse UI immediately
- Handle 'timed_out' and 'cancelled' Pulse statuses explicitly (previously
  only 'failed' was caught; others would spin until local deadline)
- Poll every 2s instead of 1s to reduce Pulse API chatter

SSH command options (_ssh_batch + diagnose.py):
- Add BatchMode=yes: aborts immediately instead of hanging on a
  password prompt if key auth fails
- Add ServerAliveInterval=10 ServerAliveCountMax=2: SSH detects a
  hung remote command within ~20s instead of sitting silent until the
  45s Pulse timeout expires

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-15 09:19:07 -04:00
jaredandClaude Sonnet 4.6 34758768a4 New features: stale banner, tab title alerts, health checks, DB housekeeping
static/app.js:
- Browser tab title updates to show alert count: '(3 CRIT) GANDALF' or '(2 WARN) GANDALF'
- Stale monitoring banner: injected into .main if last_check > 15 min old,
  warns operator that the monitor daemon may be down

static/style.css:
- .stale-banner: amber top-border warning strip

app.py:
- /health now checks DB connectivity and monitor freshness (last_check age)
  Returns 503 + degraded status if DB unreachable or monitor stale >20min

db.py:
- cleanup_expired_suppressions(): marks time-limited suppressions inactive when
  expires_at <= NOW() (was only filtered in SELECTs, never marked inactive)
- purge_old_resolved_events(days=90): deletes old resolved events to prevent
  unbounded table growth

monitor.py:
- Calls cleanup_expired_suppressions() and purge_old_resolved_events() each cycle

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-14 21:35:32 -04:00
jaredandClaude Sonnet 4.6 a7c3f97252 De-hardcode ticket URL and cluster name; improve diagnostic polling UX
app.py:
- Context processor injects config.ticket_api.web_url into all templates
  (falls back to 'http://t.lotusguild.org/ticket/' if not set in config)

templates/base.html:
- Inject GANDALF_CONFIG JS global with ticket_web_url before app.js loads

static/app.js:
- Use GANDALF_CONFIG.ticket_web_url instead of hardcoded domain

templates/index.html:
- Use {{ config.ticket_api.web_url }} Jinja var instead of hardcoded domain

monitor.py:
- CLUSTER_NAME constant kept as default; NetworkMonitor now reads cluster_name
  from config monitor.cluster_name, falling back to the constant
- All CLUSTER_NAME references inside class methods replaced with self.cluster_name

templates/inspector.html:
- pollDiagnostic() .catch() now clears interval and shows error message instead
  of silently ignoring network failures during active polling

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-14 14:31:57 -04:00
jaredandClaude Sonnet 4.6 df2a2cc48c Optimize suppression checks: load once per cycle, add error logging
db.py:
- Add check_suppressed(suppressions, ...) for in-memory suppression lookups
  against pre-loaded list (eliminates N*M DB queries per monitoring cycle)
- get_baseline(): log error instead of silently swallowing JSON parse failure

monitor.py:
- Load active suppressions once per cycle at the top of the alert loop
- Pass suppressions list to _process_interfaces, _process_unifi, _process_ping_hosts
- Replace all db.is_suppressed() calls with db.check_suppressed(suppressions, ...)
- Reduces DB queries from 100-600+ per cycle down to 1

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-14 14:13:54 -04:00
jaredandClaude Sonnet 4.6 300b1cc68c Security and reliability fixes: input validation, logging, job cleanup
- C5: Validate host_ip (IPv4 check) and iface (allowlist regex) before SSH command builder
- H6: Upgrade Pulse failure logging from debug to error so operators see outages
- M6: Replace per-request O(n) purge with background daemon thread (runs every 2 min)
- M7: Background thread marks jobs stuck in 'running' > 5 min as errored

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-12 17:30:50 -04:00
jaredandClaude Sonnet 4.6 18112541c8 feat: deep link diagnostics via Pulse SSH
Adds comprehensive per-port link troubleshooting triggered from the
Inspector panel when a port has an LLDP-identified server counterpart.

- diagnose.py: DiagnosticsRunner with 15-section SSH command (carrier,
  operstate, sysfs counters, ethtool, ethtool -i/-a/-g/-S/-m, ip link,
  ip addr, ip route, dmesg, lldpctl); parsers for all sections; health
  analyzer with 14 check codes (NO_CARRIER, HALF_DUPLEX, SPEED_MISMATCH,
  SFP_RX_CRITICAL, CARRIER_FLAPPING, CRC_ERRORS_HIGH, LLDP_MISMATCH, etc.)
- monitor.py: PulseClient now tracks last_execution_id so callers can
  link back to the raw Pulse execution URL
- app.py: POST /api/diagnose + GET /api/diagnose/<job_id> with daemon
  thread background execution and 10-minute in-memory job store
- inspector.html: "Run Link Diagnostics" button (shown only when LLDP
  host is resolvable); full results panel: health banner, physical layer,
  SFP/DOM with power bars, NIC error counters, collapsible ethtool -S,
  flow control/ring buffers, driver info, LLDP 2-col validation,
  collapsible dmesg, switch port summary, "View in Pulse" link
- style.css: all .diag-* CSS classes with terminal aesthetic

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-03 16:03:54 -05:00
jaredandClaude Sonnet 4.6 468dbda7a1 feat: inspector page, link debug enhancements, security hardening
- Add /inspector page: visual model-accurate switch chassis diagrams
  (USF5P, USL8A, US24PRO, USPPDUP, USMINI), clickable port blocks
  with color coding (green=up, amber=PoE, cyan=uplink, grey=down),
  detail panel with stats/PoE/LLDP, LLDP-based path debug side-by-side

- Link Debug: port number badges (#N), LLDP neighbor line, PoE class/max,
  collapsible host/switch panels with sessionStorage persistence

- monitor.py: collect LLDP neighbor map + PoE class/max/mode per switch
  port; PulseClient uses requests.Session() for HTTP keep-alive; add
  shlex.quote() around interface names (defense-in-depth)

- Security: suppress buttons use data-* attrs + delegated click handler
  instead of inline onclick with Jinja2 variable interpolation; remove
  | safe filter from user-controlled fields in suppressions.html;
  setDuration() takes explicit el param instead of implicit event global

- db.py: thread-local connection reuse with ping(reconnect=True) to
  avoid a new TCP handshake per query

- .gitignore: add config.json (contains credentials), __pycache__

- README: full rewrite covering architecture, all 4 pages, alert logic,
  config reference, deployment, troubleshooting, security notes

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-03 15:39:48 -05:00
jaredandClaude Sonnet 4.6 c8e183fc4f feat: terminal aesthetic rewrite + link debug page
- Full dark terminal aesthetic (Pulse/TinkerTickets style):
  - #0a0a0a background, #00ff41 green, #ffb000 amber, #00ffff cyan
  - CRT scanline overlay, phosphor glow, ASCII corner pseudoelements
  - Bracket-notation badges [CRITICAL], monospace font throughout
  - style.css, base.html, index.html, suppressions.html all rewritten

- New Link Debug page (/links, /api/links):
  - Per-host, per-interface cards with speed/duplex/port type/auto-neg
  - Traffic bars (TX cyan, RX green) with rate labels
  - Error/drop counters, carrier change history
  - SFP/DOM optical panel: vendor, temp, voltage, bias, TX/RX power dBm bars
  - RX-TX delta shown; color-coded warn/crit thresholds
  - Auto-refresh every 60s, anchor-jump to #hostname

- LinkStatsCollector in monitor.py:
  - SSHes to each host (one connection, all ifaces batched)
  - Parses ethtool + ethtool -m (SFP DOM) output
  - Merges with Prometheus traffic/error/carrier metrics
  - Stores as link_stats in monitor_state table

- config.json: added ssh section for ethtool collection
- app.js: terminal chip style consistency (uppercase, ● bullet)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-02 12:43:11 -05:00
jaredandClaude Sonnet 4.6 04a5182259 Complete rewrite: full-featured network monitoring dashboard
- Two-service architecture: Flask web app (gandalf.service) + background
  polling daemon (gandalf-monitor.service)
- Monitor polls Prometheus node_network_up for physical NIC states on all
  6 hypervisors (added storage-01 at 10.10.10.11:9100)
- UniFi API monitoring for switches, APs, and gateway device status
- Ping reachability for hosts without node_exporter (pbs only now)
- Smart baseline: interfaces first seen as down are never alerted on;
  only UP→DOWN regressions trigger tickets
- Cluster-wide P1 ticket when 3+ hosts have genuine simultaneous
  interface regressions (guards against false positives on startup)
- Tinker Tickets integration with 24-hour hash-based deduplication
- Alert suppression: manual toggle or timed windows (30m/1h/4h/8h)
- Authelia SSO via forward-auth headers, admin group required
- Network topology: Internet → UDM-Pro → Agg Switch (10G DAC) →
  PoE Switch (10G DAC) → Hosts
- MariaDB schema, suppression management UI, host/interface cards

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-01 23:03:18 -05:00