Compare commits

..
Author SHA1 Message Date
jaredandClaude Sonnet 5 c64853f643 Fix CI: pass --break-system-packages to pip3 installs
Lint / Python (flake8) (push) Successful in 34s
Lint / JS (eslint) (push) Successful in 5s
Lint / Notify on failure (push) Skipped
Security / Python Security (bandit) (push) Successful in 41s
Test / Python Tests (pytest) (push) Successful in 49s
Lint / Deploy (push) Successful in 8s
The runner's Debian image update enforces PEP 668, so system-wide pip3
installs (flake8, bandit, pytest, requirements.txt) were failing before
any job logic ran, blocking lint/test/security and the deploy job.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UB2D82eYNzD1bnMHhFPjRr
2026-09-04 21:10:38 -04:00
jaredandClaude Sonnet 5 42cf754796 Add separate unifi_failure_threshold to avoid false tickets on reboots
Lint / Python (flake8) (push) Failing after 1m4s
Lint / JS (eslint) (push) Successful in 5s
Lint / Deploy (push) Skipped
Security / Python Security (bandit) (push) Failing after 37s
Test / Python Tests (pytest) (push) Failing after 33s
Lint / Notify on failure (push) Successful in 2s
UniFi switches/APs take several minutes to rejoin after a firmware-update
reboot (observed AP taking 12 min on the scheduled 3am update), longer than
the 2-consecutive-check (~4 min) threshold shared with host link-down
detection. Give UniFi device offline detection its own, more forgiving
threshold so momentary maintenance reboots don't cut tickets.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UB2D82eYNzD1bnMHhFPjRr
2026-09-04 20:59:43 -04:00
5 changed files with 14 additions and 7 deletions
+1 -1
View File
@@ -17,7 +17,7 @@ jobs:
run: |
apt-get update -qq
apt-get install -y -qq python3 python3-pip
pip3 install flake8
pip3 install --break-system-packages flake8
- name: Run flake8
run: flake8 . --exclude=__pycache__,.git
+1 -1
View File
@@ -19,7 +19,7 @@ jobs:
run: |
apt-get update -qq
apt-get install -y -qq python3 python3-pip
pip3 install bandit
pip3 install --break-system-packages bandit
- name: Run bandit
run: bandit -r . --exclude .git,__pycache__,node_modules -ll
+2 -2
View File
@@ -17,8 +17,8 @@ jobs:
run: |
apt-get update -qq
apt-get install -y -qq python3 python3-pip
pip3 install pytest pytest-cov
pip3 install -r requirements.txt --quiet
pip3 install --break-system-packages pytest pytest-cov
pip3 install --break-system-packages -r requirements.txt --quiet
- name: Run pytest with coverage
run: python3 -m pytest tests/ -v --cov=. --cov-report=term-missing --cov-config=.coveragerc
+3 -1
View File
@@ -148,7 +148,7 @@ returned by the UniFi API for that device.
| Condition | Priority |
|---|---|
| UniFi device offline (≥2 consecutive checks) | P2 High |
| UniFi device offline (≥`unifi_failure_threshold` consecutive checks) | P2 High |
| Proxmox host NIC link-down regression (≥2 consecutive checks) | P2 High |
| Host unreachable via ping (≥2 consecutive checks) | P2 High |
| ≥3 hosts simultaneously reporting interface failures | P1 Critical |
@@ -218,6 +218,7 @@ Shared by both processes. Located in the working directory (`/var/www/html/prod/
"monitor": {
"poll_interval": 120,
"failure_threshold": 2,
"unifi_failure_threshold": 5,
"cluster_threshold": 3,
"ping_hosts": [
{ "name": "pbs", "ip": "10.10.10.3" }
@@ -243,6 +244,7 @@ Shared by both processes. Located in the working directory (`/var/www/html/prod/
| `hosts` | Maps Prometheus instance labels → display hostnames |
| `monitor.poll_interval` | Seconds between full check cycles (default: 120) |
| `monitor.failure_threshold` | Consecutive failures before creating ticket (default: 2) |
| `monitor.unifi_failure_threshold` | Consecutive failures before ticketing a UniFi device as offline (default: same as `failure_threshold`). Set higher than `failure_threshold` — UniFi switches/APs can take several minutes to rejoin after a firmware-update reboot, so a low threshold cuts tickets for momentary maintenance reboots instead of real failures. |
| `monitor.cluster_threshold` | Hosts with failures to trigger cluster-wide P1 (default: 3) |
| `monitor.ping_hosts` | Hosts checked only by ping (no node_exporter) |
+7 -2
View File
@@ -677,6 +677,11 @@ class NetworkMonitor:
mon = self.cfg.get('monitor', {})
self.poll_interval = mon.get('poll_interval', 120)
self.fail_thresh = mon.get('failure_threshold', 2)
# UniFi gear (switches/APs) can take several minutes to rejoin after a
# firmware-update reboot, much longer than a Proxmox host link flap.
# Use a separate, more forgiving threshold so those momentary reboots
# don't cut tickets, while still catching genuine device failures.
self.unifi_fail_thresh = mon.get('unifi_failure_threshold', self.fail_thresh)
self.cluster_thresh = mon.get('cluster_threshold', 3)
self.cluster_name = mon.get('cluster_name', CLUSTER_NAME)
@@ -809,7 +814,7 @@ class NetworkMonitor:
name, d.get('type', ''),
f'UniFi {name} ({d.get("ip","")}) offline ({_now_utc()})',
)
if not sup and consec >= self.fail_thresh and not ticket_id:
if not sup and consec >= self.unifi_fail_thresh and not ticket_id:
self._ticket_unifi(event_id, d)
else:
db.resolve_event('unifi_device_offline', name, d.get('type', ''))
@@ -931,7 +936,7 @@ class NetworkMonitor:
def run(self) -> None:
logger.info(
f'Gandalf monitor started poll_interval={self.poll_interval}s '
f'fail_thresh={self.fail_thresh}'
f'fail_thresh={self.fail_thresh} unifi_fail_thresh={self.unifi_fail_thresh}'
)
while True:
try: