Recovery notes and fuller context on tickets #20

Merged

jared merged 1 commits from ticket-recovery-and-context into main 2026-11-10 20:09:04 -05:00

Conversation 0 Commits 1 Files Changed 8

+886 -133

jared commented 2026-11-10 19:45:09 -05:00

Owner

Copy Link

Copy Source

What changes

Recovery notes. When an alert that opened a ticket clears, the monitor adds a comment to that ticket. The comment says when it cleared, how long it lasted, how many checks failed, and links back to Gandalf. Gandalf never closes or changes the status of a ticket; that is still up to a person.

• The one-shot WAN reports (modem power-cycled, WAN unstable) get no note, because they describe something that is already over.
• A note that fails to post is retried every cycle for 24 hours.
• A 401 or 403 from Tinker logs one error and stops notes until the monitor restarts. The API key needs read_write scope.

Fuller tickets. Every ticket is now Markdown, with a numbered "What to check" list and these sections:

• Where it is connected: the switch port the host or device plugs into, with its state, speed, PoE and SFP module, linked to that port in Gandalf. A dead host has lost its LLDP entry, so the monitor now keeps the last port it saw each device on (known_uplinks in monitor_state) and shows that instead.
• Cutsheet cables ending on the device, with their full path through patch panels.
• History: how many times the same alert fired in the last 7 and 30 days, and when it last cleared. Three or more in a week is called out.
• Other active alerts on the same device, with their ticket numbers.
• Links into Gandalf, built from gandalf.public_url. It defaults to the public hostname.

Titles are unchanged, because Tinker's duplicate detection relies on the tag format.

Schema. network_events.recovery_notified is added automatically when the monitor starts. Rows that cleared before the upgrade are marked as already notified, so old tickets get no comments.

Example (made-up names)

Interface-down ticket

## Network Interface Alert

- **Host:** server2
- **Interface:** `enp1s0f0`
- **Detected:** 2026-11-11 00:44:16 UTC
- **Consecutive check failures:** 2

Interface enp1s0f0 on server2 was up and is now reporting link-down via Prometheus node_exporter. server2 itself is still reporting, so it may be reachable over its other interfaces.

### What to check
1. Check the switch port below: is it down on the switch side too?
2. Reseat or swap the cable or DAC; for an optic, check the module for a fault.
3. On server2, check `ip link show enp1s0f0` and `dmesg` for NIC or driver errors.

### Where it is connected
- `enp1s0f0` → [switch-a port 7 "Server2 10G": down, module: DAC DAC-10G-1M](https://gandalf.example/device/switch-a#port-7) _(last seen linked there 2026-11-10 21:56:10 UTC)_
- Cable [**C05**](https://gandalf.example/cutsheet#cable-5): switch-a port 7 → PP1 port 7 (front) → PP1 port 7 (rear) → server2 enp1s0f0. Cutsheet check: planned device is not linked.

### History
- Happened 3 times in the last 7 days and 4 in the last 30 (not counting this one).
- **This keeps coming back**: look for a cause that comes and goes (a loose cable, a failing optic, power, or a scheduled job).
- Last cleared 2026-11-09 00:40 UTC (2 d ago), after 5 min.

### Other active alerts here
- interface down on server2 eno1 since 2026-11-10T21:58:00Z (ticket #1041)

### Links
- [server2 in Gandalf](https://gandalf.example/device/server2)
- [Gandalf dashboard](https://gandalf.example/)

_Opened automatically by Gandalf. It adds a note here when the problem clears, but leaves closing the ticket to you._

Recovery note

**✅ Cleared**: interface down on server2 enp1s0f0.

- **Cleared:** 2026-11-11 00:44 UTC
- **Lasted:** 14 min (first seen 2026-11-11 00:30 UTC)
- **Failed checks:** 7
- **Gandalf:** https://gandalf.example/device/server2

Gandalf no longer sees the problem.

_Gandalf leaves this ticket open. Close it once you know why it happened, or leave it open if it needs follow-up._

Testing

• 314 tests pass (24 new): ticket context, the last-known-port fallback, a failing section that still opens the ticket, recovery notes and their retry, 403 handling, the comment payload and endpoint, and the text helpers.
• Against a local MariaDB: the migration from the old schema (old resolved rows count as notified, and it can run twice), the pending-notes query excluding one-shot types, last_seen left unchanged by the notified flag, and event_stats.
• flake8 and bandit are clean.

After deploy

I'll confirm the column was added and that the first real recovery note shows up in Tinker.

🤖 Generated with Claude Code

## What changes **Recovery notes.** When an alert that opened a ticket clears, the monitor adds a comment to that ticket. The comment says when it cleared, how long it lasted, how many checks failed, and links back to Gandalf. Gandalf **never closes or changes the status** of a ticket; that is still up to a person. - The one-shot WAN reports (modem power-cycled, WAN unstable) get no note, because they describe something that is already over. - A note that fails to post is retried every cycle for 24 hours. - A 401 or 403 from Tinker logs one error and stops notes until the monitor restarts. The API key needs `read_write` scope. **Fuller tickets.** Every ticket is now Markdown, with a numbered "What to check" list and these sections: - **Where it is connected**: the switch port the host or device plugs into, with its state, speed, PoE and SFP module, linked to that port in Gandalf. A dead host has lost its LLDP entry, so the monitor now keeps the last port it saw each device on (`known_uplinks` in `monitor_state`) and shows that instead. - Cutsheet cables ending on the device, with their full path through patch panels. - **History**: how many times the same alert fired in the last 7 and 30 days, and when it last cleared. Three or more in a week is called out. - **Other active alerts** on the same device, with their ticket numbers. - **Links** into Gandalf, built from `gandalf.public_url`. It defaults to the public hostname. Titles are unchanged, because Tinker's duplicate detection relies on the tag format. **Schema.** `network_events.recovery_notified` is added automatically when the monitor starts. Rows that cleared before the upgrade are marked as already notified, so old tickets get no comments. ## Example (made-up names) <details><summary>Interface-down ticket</summary> ```markdown ## Network Interface Alert - **Host:** server2 - **Interface:** `enp1s0f0` - **Detected:** 2026-11-11 00:44:16 UTC - **Consecutive check failures:** 2 Interface enp1s0f0 on server2 was up and is now reporting link-down via Prometheus node_exporter. server2 itself is still reporting, so it may be reachable over its other interfaces. ### What to check 1. Check the switch port below: is it down on the switch side too? 2. Reseat or swap the cable or DAC; for an optic, check the module for a fault. 3. On server2, check `ip link show enp1s0f0` and `dmesg` for NIC or driver errors. ### Where it is connected - `enp1s0f0` → [switch-a port 7 "Server2 10G": down, module: DAC DAC-10G-1M](https://gandalf.example/device/switch-a#port-7) _(last seen linked there 2026-11-10 21:56:10 UTC)_ - Cable [**C05**](https://gandalf.example/cutsheet#cable-5): switch-a port 7 → PP1 port 7 (front) → PP1 port 7 (rear) → server2 enp1s0f0. Cutsheet check: planned device is not linked. ### History - Happened 3 times in the last 7 days and 4 in the last 30 (not counting this one). - **This keeps coming back**: look for a cause that comes and goes (a loose cable, a failing optic, power, or a scheduled job). - Last cleared 2026-11-09 00:40 UTC (2 d ago), after 5 min. ### Other active alerts here - interface down on server2 eno1 since 2026-11-10T21:58:00Z (ticket #1041) ### Links - [server2 in Gandalf](https://gandalf.example/device/server2) - [Gandalf dashboard](https://gandalf.example/) _Opened automatically by Gandalf. It adds a note here when the problem clears, but leaves closing the ticket to you._ ``` </details> <details><summary>Recovery note</summary> ```markdown **✅ Cleared**: interface down on server2 enp1s0f0. - **Cleared:** 2026-11-11 00:44 UTC - **Lasted:** 14 min (first seen 2026-11-11 00:30 UTC) - **Failed checks:** 7 - **Gandalf:** https://gandalf.example/device/server2 Gandalf no longer sees the problem. _Gandalf leaves this ticket open. Close it once you know why it happened, or leave it open if it needs follow-up._ ``` </details> ## Testing - 314 tests pass (24 new): ticket context, the last-known-port fallback, a failing section that still opens the ticket, recovery notes and their retry, 403 handling, the comment payload and endpoint, and the text helpers. - Against a local MariaDB: the migration from the old schema (old resolved rows count as notified, and it can run twice), the pending-notes query excluding one-shot types, `last_seen` left unchanged by the notified flag, and `event_stats`. - flake8 and bandit are clean. ## After deploy I'll confirm the column was added and that the first real recovery note shows up in Tinker. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

jared added 1 commit 2026-11-10 19:45:10 -05:00

Add recovery notes and fuller context to tickets ... 437f7a415f

When an alert that opened a ticket clears, the monitor now comments on the
ticket (when it cleared, how long it lasted, failed checks, a link back).
It never closes or changes a ticket's status. Failed notes retry each
cycle for 24 hours; a 401/403 from Tinker stops them until restart.

Tickets are now Markdown with a "What to check" list and, where known:
the switch port the device plugs into (kept from the last time it was
seen, so a dead host still shows it), cutsheet cables and their path,
how often the alert fired in the last 7/30 days, other open alerts on the
device, and links into Gandalf (gandalf.public_url). Titles are unchanged
so Tinker's duplicate detection still works.

network_events gains recovery_notified, added at monitor startup; rows
that cleared before the upgrade count as already notified.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

jared merged commit 9e2a2b987c into main 2026-11-10 20:09:04 -05:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

cole (Cole Keller) jared (Jared Vititoe) nathan (Nathan Vititoe)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: LotusGuild/gandalf#20