Dashboard: flag stale alerts and tickets left open
Lint / JS (eslint) (push) Successful in 19s
Lint / Python (flake8) (push) Successful in 1m7s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 52s
Test / Python Tests (pytest) (push) Successful in 44s
Lint / JS (eslint) (pull_request) Successful in 7s
Lint / Python (flake8) (pull_request) Successful in 50s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 52s
Test / Python Tests (pytest) (pull_request) Successful in 1m35s

Gandalf only comments on tickets, so a ticket can stay open long after its
alert cleared (pbs's ticket sat open for a month), and an alert can stay
active for days once nobody is looking.

- The monitor reads Tinker's open ticket list (tickets_api.php, read scope)
  every 15 minutes and saves it. A failed read keeps the previous list.
- Active alerts get a stale badge after monitor.stale_alert_days (3), and a
  "ticket closed" badge when their ticket was closed while still active.
- A Needs Follow-up panel lists open tickets whose every alert cleared at
  least monitor.stale_ticket_hours (24) ago.
- A "TO FOLLOW UP" status chip counts both. If the open ticket list is over
  2 hours old, the ticket flags are left off rather than guessed.
- Replaced an — in the alert count notice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
jaredandClaude Opus 5.5 committed 2026-11-14 15:57:58 -05:00
1 parent 7b80633088
commit 0921e82c54
10 files changed
+462 -4

No files matched your search

+20
View File
@@ -111,6 +111,9 @@ Ping-only (no node_exporter): **pbs** (10.10.10.3)
- WAN / Fiber panel: WAN and PON state, Rx/Tx power, optics temperature, throughput, errors, gateway uptime and last restart, 7-day Rx trend, recent WAN outages
- UniFi device table (switches, APs, gateway)
- Active alerts table with severity, target, consecutive failures, ticket link
- Follow-up flags (see Follow-up below): a ⏳ badge on alerts active for days, a "ticket closed" badge when the
ticket was closed but the alert is still active, and a Needs Follow-up panel of tickets left open after their
alert cleared. A "TO FOLLOW UP" chip in the status bar counts them all.
- Quick-suppress modal: apply timed or manual suppression from any alert row
- Auto-refreshes every 30 seconds via `/api/status` + `/api/network`
@@ -381,6 +384,20 @@ A note that fails to post is retried every cycle for 24 hours. Posting needs a T
`read_write` scope; if Tinker answers 401 or 403, the monitor logs one error and stops trying until it is
restarted.
### Follow-up
Gandalf only comments on tickets, so a ticket can stay open long after its problem is fixed, and an alert can
sit active for days once everyone has stopped looking. The dashboard flags both:
- **Stale alert**: active for `monitor.stale_alert_days` (default 3) or longer. Shown as a ⏳ badge with its age.
- **Ticket closed**: the alert is still active but its ticket was closed in Tinker, so nobody is tracking it.
- **Ticket left open**: every alert on a ticket cleared at least `monitor.stale_ticket_hours` (default 24) ago,
but the ticket is still Open, Pending or In Progress. Listed in the Needs Follow-up panel.
The monitor reads the list of open tickets from Tinker's `api/tickets_api.php` every 15 minutes (one request,
read scope only) and saves it. If that list is over 2 hours old, because Tinker or the key stopped working,
the ticket flags are left off rather than guessed. Stale alerts are still flagged.
### WAN modem notes
The alarm and warning limits are read from the gateway on every poll, so they are never hard-coded here.
@@ -508,6 +525,7 @@ Shared by both processes. Located in the working directory (`/var/www/html/prod/
| `ticket_api.url` | Tinker Tickets `create_ticket_api.php` URL |
| `ticket_api.api_key` | Tinker Tickets bearer token. Needs `read_write` scope to add recovery notes |
| `ticket_api.comment_url` | Tinker comment endpoint (default: `api/ticket_comment_api.php` beside `ticket_api.url`) |
| `ticket_api.read_url` | Tinker ticket read endpoint, used to find tickets left open (default: `api/tickets_api.php` beside `ticket_api.url`) |
| `gandalf.public_url` | Gandalf's own address, for links in tickets (default: `https://gandalf.lotusguild.org`) |
| `rack.units`, `rack.name` | The single rack's height and name on `/rack` (defaults: 42, "Rack 1") |
| `racks` | Several racks: `[{"name": "...", "units": 42}, ...]`, in tab order; overrides `rack` |
@@ -525,6 +543,8 @@ Shared by both processes. Located in the working directory (`/var/www/html/prod/
| `monitor.unifi_failure_threshold` | Consecutive failures before ticketing a UniFi device as offline (default: same as `failure_threshold`). Set higher than `failure_threshold`: UniFi switches/APs can take several minutes to rejoin after a firmware-update reboot, so a low threshold cuts tickets for momentary maintenance reboots instead of real failures. |
| `monitor.cluster_threshold` | Hosts with failures to trigger cluster-wide P1 (default: 3) |
| `monitor.ping_hosts` | Hosts checked only by ping (no node_exporter) |
| `monitor.stale_alert_days` | Days an alert can stay active before the dashboard flags it as stale (default: 3) |
| `monitor.stale_ticket_hours` | Hours a ticket can stay open after all its alerts cleared before it is listed under Needs Follow-up (default: 24) |
| `cutsheet.mismatch_ticket_hours` | Hours a cutsheet mismatch must last before it opens a P4 ticket (default: 24; `0` never tickets). The mismatch shows on the dashboard and the switch's page straight away, so recabling never pages anyone. Suppressing the switch or its port also stops the ticket. |
| `modem.url` | AT&T gateway base URL. Omit the `modem` block to disable modem monitoring. Must be reachable from the monitor host. |
| `modem.name` | Name used for the modem's alerts, tickets, and suppressions (default: `att-modem`) |