Network dedup hash ignores the issue: every single-host network alert for a host lands on one ticket #116

New Issue

2026-11-11T01:13:51-05:00

jared commented

2026-11-11 01:13:51 -05:00

Summary

generateTicketHash() in create_ticket_api.php keys every network ticket on the following fields only:

source_type (auto)
issue_category (network)
issue_subtype, which is always '' for this category
environment tags ([production], [single-node] / [cluster-wide])
hostname, the first [...] tag

The network branch never sets issue_subtype. Every automated single-host network alert for a host therefore hashes the same, whatever the problem is: a host reboot, a NIC link going down, a switch port, a cable mismatch, a modem fault. One host gets one network ticket, forever.

This is the same shape of bug that #112's fix addressed for Ceph HEALTH_WARN, where unrelated warnings collapsed onto one ticket.

Real-world impact (2026-11-11)

Gandalf filed [micro1][auto][production][issue][network][single-node] Interface enp3s0 link-down. It matched 993441773, [micro1]...[single-node] No metrics from host, a closed false alarm about a reboot from 2026-11-06.

The two alerts have nothing in common, but they hash identically. Because 993441773 is Closed and can't be reopened unattended (#115), Tinker:

posted "Issue recurred, but the ticket could not be reopened automatically" on the unrelated reboot ticket;
returned success: true with action: recurrence_noted and ticket_id: 993441773.

Gandalf recorded that ID as the ticket for the interface alert. The real problem got no visible ticket, and the reboot ticket gained a misleading comment.

The same thing happens to every Gandalf alert type for any host that has ever had a network ticket.

Who sends `network` tickets

Gandalf (monitor.py): [<target>][auto][production][issue][network][single-node|cluster-wide] <heading>. The headings are stable for a given problem and differ between problems:

No metrics from host
Host unreachable
Interface <iface> link-down
UniFi device offline
Cable <label> does not match the cutsheet
WAN link down
Fiber <metric> alarm / Fiber <metric> warning
Fiber Rx power dropped
WAN modem unreachable / WAN modem status unreadable
Multiple hosts reporting interface failures (cluster-wide)

hwmonDaemon: [host][auto][network] - <issue>[single-node][production][issue]. Its network issues are fixed sentences ("Management network is unreachable", "Ceph network is unreachable"), so keying on the text is stable for it too.

Suggested fix

In the network branch, set issue_subtype from the title text outside the bracketed tags, normalised. This mirrors how #112 keys HEALTH_WARN on its text:

} elseif (stripos($title, 'network') !== false) {
    $issueCategory = 'network';
    // One ticket per kind of network problem per host, not one per host.
    $heading = trim(preg_replace('/\[[^\]]*\]/', '', $title), " -\t");
    $slug = trim(preg_replace('/[^a-z0-9]+/', '_', strtolower($heading)), '_');
    if ($slug !== '') {
        $issueSubtype = 'net_' . substr($slug, 0, 80);
    }
}

Points to decide:

Keep digits. They identify the problem (enp3s0, cable C07). Unlike HEALTH_WARN counts, none of these headings carry changing numbers. A check_code from Gandalf, as #112 added for Ceph, would be an even steadier key if preferred; Gandalf can send one.
Hash change is one-time. Existing open network tickets get a new hash, so the next recurrence of each opens a new ticket instead of updating the old one. A migration could recompute hash for open network tickets to avoid that.
Tests: the same host with two different headings gives two hashes. The same host and heading, sent repeatedly, gives one hash. hwmonDaemon's management-network vs ceph-network alerts get separate hashes.

#115: recurrences onto a Closed ticket that can't reopen are swallowed. That made this case worse: the wrong match landed on a closed ticket and vanished.
Gandalf side (no Tinker change needed): Gandalf currently ignores action in the create response. It should log, and probably not store, a ticket ID returned with action: recurrence_noted.

🤖 Generated with Claude Code

## Summary `generateTicketHash()` in `create_ticket_api.php` keys every `network` ticket on the following fields only: - `source_type` (auto) - `issue_category` (`network`) - `issue_subtype`, which is always `''` for this category - environment tags (`[production]`, `[single-node]` / `[cluster-wide]`) - `hostname`, the first `[...]` tag The `network` branch never sets `issue_subtype`. Every automated single-host network alert for a host therefore hashes the same, whatever the problem is: a host reboot, a NIC link going down, a switch port, a cable mismatch, a modem fault. **One host gets one network ticket, forever.** This is the same shape of bug that #112's fix addressed for Ceph HEALTH_WARN, where unrelated warnings collapsed onto one ticket. ## Real-world impact (2026-11-11) Gandalf filed `[micro1][auto][production][issue][network][single-node] Interface enp3s0 link-down`. It matched **993441773**, `[micro1]...[single-node] No metrics from host`, a closed false alarm about a reboot from 2026-11-06. The two alerts have nothing in common, but they hash identically. Because 993441773 is Closed and can't be reopened unattended (#115), Tinker: - posted "Issue recurred, but the ticket could not be reopened automatically" on the unrelated reboot ticket; - returned `success: true` with `action: recurrence_noted` and `ticket_id: 993441773`. Gandalf recorded that ID as the ticket for the interface alert. The real problem got no visible ticket, and the reboot ticket gained a misleading comment. The same thing happens to every Gandalf alert type for any host that has ever had a network ticket. ## Who sends `network` tickets **Gandalf** (`monitor.py`): `[<target>][auto][production][issue][network][single-node|cluster-wide] <heading>`. The headings are stable for a given problem and differ between problems: - `No metrics from host` - `Host unreachable` - `Interface <iface> link-down` - `UniFi device offline` - `Cable <label> does not match the cutsheet` - `WAN link down` - `Fiber <metric> alarm` / `Fiber <metric> warning` - `Fiber Rx power dropped` - `WAN modem unreachable` / `WAN modem status unreadable` - `Multiple hosts reporting interface failures` (cluster-wide) **hwmonDaemon:** `[host][auto][network] - <issue>[single-node][production][issue]`. Its network issues are fixed sentences ("Management network is unreachable", "Ceph network is unreachable"), so keying on the text is stable for it too. ## Suggested fix In the `network` branch, set `issue_subtype` from the title text outside the bracketed tags, normalised. This mirrors how #112 keys HEALTH_WARN on its text: ```php } elseif (stripos($title, 'network') !== false) { $issueCategory = 'network'; // One ticket per kind of network problem per host, not one per host. $heading = trim(preg_replace('/\[[^\]]*\]/', '', $title), " -\t"); $slug = trim(preg_replace('/[^a-z0-9]+/', '_', strtolower($heading)), '_'); if ($slug !== '') { $issueSubtype = 'net_' . substr($slug, 0, 80); } } ``` Points to decide: - **Keep digits.** They identify the problem (`enp3s0`, cable `C07`). Unlike HEALTH_WARN counts, none of these headings carry changing numbers. A `check_code` from Gandalf, as #112 added for Ceph, would be an even steadier key if preferred; Gandalf can send one. - **Hash change is one-time.** Existing open network tickets get a new hash, so the next recurrence of each opens a new ticket instead of updating the old one. A migration could recompute `hash` for open network tickets to avoid that. - **Tests:** the same host with two different headings gives two hashes. The same host and heading, sent repeatedly, gives one hash. hwmonDaemon's management-network vs ceph-network alerts get separate hashes. ## Related - **#115:** recurrences onto a Closed ticket that can't reopen are swallowed. That made this case worse: the wrong match landed on a closed ticket and vanished. - **Gandalf side** (no Tinker change needed): Gandalf currently ignores `action` in the create response. It should log, and probably not store, a ticket ID returned with `action: recurrence_noted`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

jared referenced this issue from a commit

2026-11-11 01:40:49 -05:00

Key network dedup tickets on the problem, not just the host (#116)

jared referenced a pull request that will close this issue

2026-11-11 01:41:56 -05:00

Merge development into main: network dedup per problem (#116) #117

jared referenced this issue from a commit

2026-11-11 01:43:55 -05:00

Merge development into main: network dedup per problem (#116)

jared closed this issue

2026-11-11 01:44:06 -05:00

jared commented

2026-11-11 01:47:31 -05:00

Fixed in aad3e8e (PR #117) and deployed to prod (a03fd64).

Fix

Automated network tickets are now keyed on host + check_code when it is sent (it must name the instance, e.g. iface_down:enp3s0), otherwise on host + the title text outside the tags, with digits kept.
Manual tickets and every other category hash exactly as before.
The hash logic now lives in helpers/TicketHash.php.

Rehash: I ran scripts/rehash_network_tickets.php on prod. It re-keyed the one open network ticket, pbs 281331508, with 0 conflicts. A second dry run showed it already current.

Cleanup of past tickets. The audit log shows the bug had already done more damage than the original report:

200098273, opened 2026-09-28 by hwmonDaemon as "[large1] Management network is unreachable". Its title was overwritten by Gandalf's bogus enp36s0 link-down alert, and it was then closed as that false alarm.
448664107, opened 2026-09-28 as "[compute-storage-01] Ceph network is unreachable". It was overwritten three times (nic1, then nic2, then nic3) and closed the same way.

I restored both original titles and recorded the change in the audit log. Both stay Closed, with a comment explaining what happened. Their original descriptions were overwritten and are lost. If hwmonDaemon still sees either problem, its next report opens a new ticket.

993441773 (micro1 reboot): the misleading recurrence comment is kept as history, with a reply explaining it.

The Gandalf follow-up from the issue (don't store an ID returned with action: recurrence_noted) is still open.

🤖 Generated with Claude Code

Fixed in aad3e8e (PR #117) and deployed to prod (a03fd64). **Fix** - Automated network tickets are now keyed on host + `check_code` when it is sent (it must name the instance, e.g. `iface_down:enp3s0`), otherwise on host + the title text outside the tags, with digits kept. - Manual tickets and every other category hash exactly as before. - The hash logic now lives in `helpers/TicketHash.php`. **Rehash:** I ran `scripts/rehash_network_tickets.php` on prod. It re-keyed the one open network ticket, pbs 281331508, with 0 conflicts. A second dry run showed it already current. **Cleanup of past tickets.** The audit log shows the bug had already done more damage than the original report: - **200098273**, opened 2026-09-28 by hwmonDaemon as "[large1] Management network is unreachable". Its title was overwritten by Gandalf's bogus `enp36s0 link-down` alert, and it was then closed as that false alarm. - **448664107**, opened 2026-09-28 as "[compute-storage-01] Ceph network is unreachable". It was overwritten three times (nic1, then nic2, then nic3) and closed the same way. I restored both original titles and recorded the change in the audit log. Both stay Closed, with a comment explaining what happened. Their original descriptions were overwritten and are lost. If hwmonDaemon still sees either problem, its next report opens a new ticket. - **993441773** (micro1 reboot): the misleading recurrence comment is kept as history, with a reply explaining it. The Gandalf follow-up from the issue (don't store an ID returned with `action: recurrence_noted`) is still open. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: LotusGuild/tinker_tickets#116

Network dedup hash ignores the issue: every single-host network alert for a host lands on one ticket #116

Summary

Real-world impact (2026-11-11)

Who sends network tickets

Suggested fix

Related

Who sends `network` tickets