LotusGuild/gandalf
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Tickets cutting instantly when devices go offline even during short update periods at scheduled 3am #1

New Issue
Closed
opened 2026-09-04 09:17:29 -04:00 by jared · 1 comment
jared commented 2026-09-04 09:17:29 -04:00
Owner
Copy Link
Copy Source

http://t.lotusguild.org/ticket/886841157

http://t.lotusguild.org/ticket/886841157
jared commented 2026-09-05 11:40:06 -04:00
Author
Owner
Copy Link
Copy Source

Root cause: the AP took 12 minutes to rejoin after the scheduled 3am firmware update, crossing the shared failure_threshold=2 (~4 min at the 120s poll interval) used for both server NIC and UniFi device offline detection — so a plain maintenance reboot cut a real ticket.

Fix (deployed): added a separate unifi_failure_threshold (set to 5, ~10 min) used only for unifi_device_offline ticketing, decoupled from the host/interface failure_threshold=2. Merged in 42cf754, live on CT157 (gandalf-monitor startup log now shows fail_thresh=2 unifi_fail_thresh=5).

Also fixed in passing: CI (flake8/bandit/pytest) had been failing on every push due to the runner's Debian image enforcing PEP 668 on pip3 install, which was silently blocking the deploy webhook too (c64853f).

Separately found and fixed while investigating: pbs was being reported offline permanently due to a missing pulse config block in production plus a stale Pulse worker WebSocket connection — filed as LotusGuild/pulse#1, now resolved.

Closing this out.

Root cause: the AP took 12 minutes to rejoin after the scheduled 3am firmware update, crossing the shared `failure_threshold=2` (~4 min at the 120s poll interval) used for both server NIC and UniFi device offline detection — so a plain maintenance reboot cut a real ticket. Fix (deployed): added a separate `unifi_failure_threshold` (set to 5, ~10 min) used only for `unifi_device_offline` ticketing, decoupled from the host/interface `failure_threshold=2`. Merged in 42cf754, live on CT157 (`gandalf-monitor` startup log now shows `fail_thresh=2 unifi_fail_thresh=5`). Also fixed in passing: CI (flake8/bandit/pytest) had been failing on every push due to the runner's Debian image enforcing PEP 668 on `pip3 install`, which was silently blocking the deploy webhook too (c64853f). Separately found and fixed while investigating: `pbs` was being reported offline permanently due to a missing `pulse` config block in production plus a stale Pulse worker WebSocket connection — filed as LotusGuild/pulse#1, now resolved. Closing this out.
jared closed this issue 2026-09-05 11:40:07 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
cole (Cole Keller) jared (Jared Vititoe) nathan (Nathan Vititoe)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: LotusGuild/gandalf#1