Dashboard: flag stale alerts and tickets left open
Lint / JS (eslint) (push) Successful in 19s
Lint / Python (flake8) (push) Successful in 1m7s
Lint / Notify on failure (push) Skipped
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Successful in 52s
Test / Python Tests (pytest) (push) Successful in 44s
Lint / JS (eslint) (pull_request) Successful in 7s
Lint / Python (flake8) (pull_request) Successful in 50s
Lint / Notify on failure (pull_request) Skipped
Lint / Deploy (pull_request) Skipped
Security / Python Security (bandit) (pull_request) Successful in 52s
Test / Python Tests (pytest) (pull_request) Successful in 1m35s

Gandalf only comments on tickets, so a ticket can stay open long after its
alert cleared (pbs's ticket sat open for a month), and an alert can stay
active for days once nobody is looking.

- The monitor reads Tinker's open ticket list (tickets_api.php, read scope)
  every 15 minutes and saves it. A failed read keeps the previous list.
- Active alerts get a stale badge after monitor.stale_alert_days (3), and a
  "ticket closed" badge when their ticket was closed while still active.
- A Needs Follow-up panel lists open tickets whose every alert cleared at
  least monitor.stale_ticket_hours (24) ago.
- A "TO FOLLOW UP" status chip counts both. If the open ticket list is over
  2 hours old, the ticket flags are left off rather than guessed.
- Replaced an — in the alert count notice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-14 15:57:58 -05:00
co-authored by Claude Opus 5.5
parent 7b80633088
commit 0921e82c54
10 changed files with 462 additions and 4 deletions
+61
View File
@@ -408,6 +408,8 @@ class TicketClient:
self.comment_url = cfg.get('comment_url') or (
urljoin(self.url, 'api/ticket_comment_api.php') if self.url else '')
self.comments_denied = False
self.read_url = cfg.get('read_url') or (
urljoin(self.url, 'api/tickets_api.php') if self.url else '')
# Titles whose create matched a Closed ticket Tinker couldn't reopen: {title: (ticket_id, when)}
self._closed_matches: Dict[str, tuple] = {}
@@ -442,6 +444,41 @@ class TicketClient:
logger.error(f'Ticket comment failed for #{ticket_id}: {e}')
return False
# Statuses that mean someone still has a ticket on their list
OPEN_STATUSES = 'Open,Pending,In Progress'
_OPEN_TICKETS_MAX_PAGES = 20
def open_tickets(self) -> Optional[Dict[str, str]]:
"""{ticket_id: status} for every ticket not yet closed, or None if Tinker can't say.
Read only: this needs the key's read scope and never changes a ticket.
"""
if not self.api_key or not self.read_url:
return None
found: Dict[str, str] = {}
try:
for page in range(1, self._OPEN_TICKETS_MAX_PAGES + 1):
resp = requests.get(
self.read_url,
params={'status': self.OPEN_STATUSES, 'limit': 100, 'page': page},
headers={'Authorization': f'Bearer {self.api_key}'},
timeout=15,
)
resp.raise_for_status()
data = resp.json()
if not data.get('success'):
logger.warning(f'Unexpected ticket list response: {str(data)[:200]}')
return None
for t in data.get('tickets') or []:
found[str(t.get('ticket_id'))] = t.get('status') or ''
if page >= int(data.get('pages') or 1):
return found
logger.warning('Open ticket list is longer than expected; using the first pages only')
return found
except Exception as e:
logger.error(f'Reading open tickets failed: {e}')
return None
# How long to wait before asking again after a create matched an unreopenable Closed ticket.
# Each ask adds a "recurred" comment to that ticket, so once a poll would pile them up.
CLOSED_MATCH_RETRY_S = 24 * 3600
@@ -990,9 +1027,15 @@ UNIFI_UPDATE_STATE = 'unifi_update'
# Ports a host or device was last seen on, kept so a ticket for a dead host
# (whose link, and so its LLDP entry, is gone) can still say where it plugs in
KNOWN_UPLINKS_STATE = 'known_uplinks'
# Which tickets are still open in Tinker, so the dashboard can flag ones left open
# after their alert cleared. Refreshed this often; one list request per refresh.
OPEN_TICKETS_STATE = 'open_tickets'
OPEN_TICKETS_SYNC_S = 900
class NetworkMonitor:
_open_tickets_synced = 0.0
def __init__(self):
with open('config.json') as f:
self.cfg = json.load(f)
@@ -1245,6 +1288,20 @@ class NetworkMonitor:
elif self.tickets.comments_denied:
break
def _sync_open_tickets(self) -> None:
"""Save which tickets are still open in Tinker, at most every OPEN_TICKETS_SYNC_S.
On a failed read the previous list stays, with its old checked_at, so the
dashboard can tell it is out of date rather than treat every ticket as closed.
"""
now = time.time()
if now - self._open_tickets_synced < OPEN_TICKETS_SYNC_S:
return
self._open_tickets_synced = now
found = self.tickets.open_tickets()
if found is not None:
db.set_state(OPEN_TICKETS_STATE, {'checked_at': now, 'tickets': found})
# ------------------------------------------------------------------
# Interface monitoring (Prometheus)
# ------------------------------------------------------------------
@@ -2051,6 +2108,10 @@ class NetworkMonitor:
self._post_recovery_notes()
except Exception as e:
logger.error(f'Posting recovery notes failed: {e}', exc_info=True)
try:
self._sync_open_tickets()
except Exception as e:
logger.error(f'Syncing open tickets failed: {e}', exc_info=True)
# Housekeeping: deactivate expired suppressions and purge old resolved events
db.cleanup_expired_suppressions()