Dashboard: flag stale alerts and tickets left open
Lint / JS (eslint) (push) Successful in 19s
Lint / Python (flake8) (push) Successful in 1m7s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 52s
Test / Python Tests (pytest) (push) Successful in 44s
Lint / JS (eslint) (pull_request) Successful in 7s
Lint / Python (flake8) (pull_request) Successful in 50s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 52s
Test / Python Tests (pytest) (pull_request) Successful in 1m35s
Lint / JS (eslint) (push) Successful in 19s
Lint / Python (flake8) (push) Successful in 1m7s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 52s
Test / Python Tests (pytest) (push) Successful in 44s
Lint / JS (eslint) (pull_request) Successful in 7s
Lint / Python (flake8) (pull_request) Successful in 50s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 52s
Test / Python Tests (pytest) (pull_request) Successful in 1m35s
Gandalf only comments on tickets, so a ticket can stay open long after its alert cleared (pbs's ticket sat open for a month), and an alert can stay active for days once nobody is looking. - The monitor reads Tinker's open ticket list (tickets_api.php, read scope) every 15 minutes and saves it. A failed read keeps the previous list. - Active alerts get a stale badge after monitor.stale_alert_days (3), and a "ticket closed" badge when their ticket was closed while still active. - A Needs Follow-up panel lists open tickets whose every alert cleared at least monitor.stale_ticket_hours (24) ago. - A "TO FOLLOW UP" status chip counts both. If the open ticket list is over 2 hours old, the ticket flags are left off rather than guessed. - Replaced an — in the alert count notice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
7b80633088
commit
0921e82c54
10 files changed
+462
-4
No files matched your search
@@ -111,6 +111,9 @@ Ping-only (no node_exporter): **pbs** (10.10.10.3)
|
||||
- WAN / Fiber panel: WAN and PON state, Rx/Tx power, optics temperature, throughput, errors, gateway uptime and last restart, 7-day Rx trend, recent WAN outages
|
||||
- UniFi device table (switches, APs, gateway)
|
||||
- Active alerts table with severity, target, consecutive failures, ticket link
|
||||
- Follow-up flags (see Follow-up below): a ⏳ badge on alerts active for days, a "ticket closed" badge when the
|
||||
ticket was closed but the alert is still active, and a Needs Follow-up panel of tickets left open after their
|
||||
alert cleared. A "TO FOLLOW UP" chip in the status bar counts them all.
|
||||
- Quick-suppress modal: apply timed or manual suppression from any alert row
|
||||
- Auto-refreshes every 30 seconds via `/api/status` + `/api/network`
|
||||
|
||||
@@ -381,6 +384,20 @@ A note that fails to post is retried every cycle for 24 hours. Posting needs a T
|
||||
`read_write` scope; if Tinker answers 401 or 403, the monitor logs one error and stops trying until it is
|
||||
restarted.
|
||||
|
||||
### Follow-up
|
||||
|
||||
Gandalf only comments on tickets, so a ticket can stay open long after its problem is fixed, and an alert can
|
||||
sit active for days once everyone has stopped looking. The dashboard flags both:
|
||||
|
||||
- **Stale alert**: active for `monitor.stale_alert_days` (default 3) or longer. Shown as a ⏳ badge with its age.
|
||||
- **Ticket closed**: the alert is still active but its ticket was closed in Tinker, so nobody is tracking it.
|
||||
- **Ticket left open**: every alert on a ticket cleared at least `monitor.stale_ticket_hours` (default 24) ago,
|
||||
but the ticket is still Open, Pending or In Progress. Listed in the Needs Follow-up panel.
|
||||
|
||||
The monitor reads the list of open tickets from Tinker's `api/tickets_api.php` every 15 minutes (one request,
|
||||
read scope only) and saves it. If that list is over 2 hours old, because Tinker or the key stopped working,
|
||||
the ticket flags are left off rather than guessed. Stale alerts are still flagged.
|
||||
|
||||
### WAN modem notes
|
||||
|
||||
The alarm and warning limits are read from the gateway on every poll, so they are never hard-coded here.
|
||||
@@ -508,6 +525,7 @@ Shared by both processes. Located in the working directory (`/var/www/html/prod/
|
||||
| `ticket_api.url` | Tinker Tickets `create_ticket_api.php` URL |
|
||||
| `ticket_api.api_key` | Tinker Tickets bearer token. Needs `read_write` scope to add recovery notes |
|
||||
| `ticket_api.comment_url` | Tinker comment endpoint (default: `api/ticket_comment_api.php` beside `ticket_api.url`) |
|
||||
| `ticket_api.read_url` | Tinker ticket read endpoint, used to find tickets left open (default: `api/tickets_api.php` beside `ticket_api.url`) |
|
||||
| `gandalf.public_url` | Gandalf's own address, for links in tickets (default: `https://gandalf.lotusguild.org`) |
|
||||
| `rack.units`, `rack.name` | The single rack's height and name on `/rack` (defaults: 42, "Rack 1") |
|
||||
| `racks` | Several racks: `[{"name": "...", "units": 42}, ...]`, in tab order; overrides `rack` |
|
||||
@@ -525,6 +543,8 @@ Shared by both processes. Located in the working directory (`/var/www/html/prod/
|
||||
| `monitor.unifi_failure_threshold` | Consecutive failures before ticketing a UniFi device as offline (default: same as `failure_threshold`). Set higher than `failure_threshold`: UniFi switches/APs can take several minutes to rejoin after a firmware-update reboot, so a low threshold cuts tickets for momentary maintenance reboots instead of real failures. |
|
||||
| `monitor.cluster_threshold` | Hosts with failures to trigger cluster-wide P1 (default: 3) |
|
||||
| `monitor.ping_hosts` | Hosts checked only by ping (no node_exporter) |
|
||||
| `monitor.stale_alert_days` | Days an alert can stay active before the dashboard flags it as stale (default: 3) |
|
||||
| `monitor.stale_ticket_hours` | Hours a ticket can stay open after all its alerts cleared before it is listed under Needs Follow-up (default: 24) |
|
||||
| `cutsheet.mismatch_ticket_hours` | Hours a cutsheet mismatch must last before it opens a P4 ticket (default: 24; `0` never tickets). The mismatch shows on the dashboard and the switch's page straight away, so recabling never pages anyone. Suppressing the switch or its port also stops the ticket. |
|
||||
| `modem.url` | AT&T gateway base URL. Omit the `modem` block to disable modem monitoring. Must be reachable from the monitor host. |
|
||||
| `modem.name` | Name used for the modem's alerts, tickets, and suppressions (default: `att-modem`) |
|
||||
|
||||
Reference in new issue
Block a user