_check_cpu_usage took a single 1-second psutil sample per run, so a momentary
spike raised a CPU ticket -- and because the ticket API reopens a closed ticket
on a matching alert, the same CPU ticket flapped open/closed on every hourly run.
Ticketing is now gated on the 15-minute load average normalized per core (a true
"sustained" signal): a transient spike barely moves the 15-min load, so it no
longer raises or reopens a ticket. The instantaneous percentage is still reported
(dry-run summary + description) and the measured sustained load is shown in the
ticket description; the title stays value-free so it doesn't churn each run. Falls
back to the instantaneous sample only if os.getloadavg() is unavailable.
Adds 5 regression tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The LXC storage check runs in every mode, but its results were never
printed in the --dry-run summary, so a --dry-run appeared to skip LXC
storage entirely. Add an "LXC Storage:" section to the summary
(per-container usage %, a warning flag over the LXC_WARNING threshold, and
the issue count) via a small, testable helper `_format_lxc_dry_run()`.
Adds 4 regression tests.
Closes#23
Ref: #23
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two monitoring-logic fixes so hwmonDaemon stops raising spurious hardware
tickets on a Ceph-backed cluster with PBS backups:
- _check_system_drive_indicators: count dmesg Buffer I/O / block errors
per-line and skip non-physical / network-backed devices (Ceph RBD, dm-,
loop). RBD Buffer I/O errors are storage-connectivity events (e.g. a Ceph
mon-session blip), not local drive faults, so they must not raise a
CRITICAL drive alert. Consistent with the existing _is_physical_disk
exclusion. (Fired a false "CRITICAL: Buffer I/O errors (75 occurrences)"
ticket where all 75 were on rbd devices.)
- _get_attribute_thresholds: drop the Power_On_Hours warning/critical
threshold. Drive age alone is not a failure; with 2-3x Ceph redundancy
and PBS backups we run drives to hard-failure rather than replace on age.
The value is still recorded (history/description/new-drive logic) but no
longer generates a ticket. Real failure-predictors (reallocated/pending/
uncorrectable/CRC, self-test, trends) are unchanged.
Adds 7 regression tests (TestSystemDriveIndicators, TestAttributeThresholds).
All 98 tests pass; flake8 clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Debian runner image marks its Python as externally managed, so
`pip3 install` fails with error: externally-managed-environment, breaking
the test/lint/security workflows at the install step (unrelated to the code
under test). Add --break-system-packages; the runner is ephemeral so a
system-wide install is fine.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
_determine_ticket_priority returned P1 for EVERY issue in a scan whenever the
report had >= PRIORITY_ESCALATION_THRESHOLD criticals, because the critical
count is global to the health report. A benign P4 "high storage usage" or
"CPU usage" alert on a node that also had 3+ critical drive failures was
created as P1, flooding P1s.
Split the severity cascade into _base_issue_priority() and only apply the
multi-critical "cluster risk" P1 bump to issues that are themselves
critical/high-severity (base P1/P2). Routine warnings (P3) and normal alerts
(P4/P5) keep their base priority. Critical issues still escalate to P1 under
multi-critical conditions as before.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add 42 new tests covering _get_manufacturer_profile (Seagate/WD/Samsung/Toshiba/
OOS/Ridata/unknown), _should_monitor_attribute, _get_attribute_thresholds,
_get_issue_type, _get_impact_level, and _categorize_issue.
Toshiba MG08 prefix matching and the raised High_Fly_Writes/Command_Timeout
thresholds introduced in the previous commit are now covered by tests.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove unused MagicMock import (F401)
- Move sys.path.insert above third-party imports and add noqa: E402
to the two imports that must follow it (E402)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Investigated all 7 pending drive tickets in the ticketing DB. Identified
3 confirmed false positives and 1 parsing bug. Implemented manufacturer-
specific SMART profiles and a systemic substring-match fix.
Changes:
- Seagate: disable Seek_Error_Rate (packed counter), add High_Fly_Writes
profile threshold (100/500 vs the old 1/5), disable Command_Timeout
(packed 3-part 48-bit format on Exos series)
- Western Digital: disable Command_Timeout (same packed format)
- Toshiba: new profile covering MG04-MG10 enterprise and MQ01-MQ04
consumer series; disable Raw/Seek counters, keep Command_Timeout with
raised thresholds (1000/5000) since MG-series uses a real simple count;
add model-prefix detection so MG08ACP16TE etc. match without "TOSHIBA"
in the model string
- OOS: add OOS14000G alias (fleet has both 12TB and 14TB variants);
replace billion-scale Command_Timeout threshold with monitor:False
- Samsung: disable Program_Fail_Cnt_Total (attr 181, vendor-encoded),
Erase_Fail_Count_Chip (attrs 172/176, chip-level internal counter),
Program_Fail_Count_Chip (attr 171); disable generic Erase_Fail_Count
and Program_Fail_Count to prevent bleed-through from _Chip lines
Bug fixes:
- Fix substring match: 'Erase_Fail_Count' was matching
'Erase_Fail_Count_Chip' lines in both the first-pass and main attribute
loops. Changed to token-boundary check (attr + ' ') in both places.
- Add 32-bit overflow guard: raw SMART values > 0xFFFFFFFF are skipped
at threshold comparison. Catches 0xFFFFFFFFFFFF sentinel values from
unrecognized drives (was generating Critical Program_Fail_Cnt_Total
tickets with value 281474976710655).
BASE_SMART_THRESHOLDS:
- High_Fly_Writes: 1/5 -> 100/500
- Program_Fail_Cnt_Total: 1/5 -> 50/200
- Erase_Fail_Count_Total: 1/5 -> 50/200
Global filtered_issues: removed Seek_Error_Rate and Command_Timeout
(now handled per-profile); Raw_Read_Error_Rate kept as catch-all.
Verified with --dry-run on all 4 servers: compute-storage-01, large1,
compute-storage-gpu-01, pbs. Only legitimate issues surface.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- LXC/ZFS storage usage percentages: "usage high: 80.1%" → "usage high"
- LXC "high storage usage: 80.1% on /mnt" → "high storage usage on /mnt"
- Ceph BlueStore OSD counts: "2 OSD(s) experiencing slow" →
"OSD(s) experiencing slow" (count fluctuates every run)
These changing values embedded in titles were triggering a "Title updated"
comment on every hourly run even though nothing meaningfully changed.
Values are fully retained in the ticket description.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Power_On_Hours and other SMART counters embedded in the issue string
were included verbatim in the ticket title. Since the count increments
every hour, the title was "new" on every run, triggering a title-update
comment every single cycle (307 spam comments on two tickets).
Strip ': Warning <attr>: <N>' / ': Critical <attr>: <N>' suffixes from
the title before building the ticket payload. Counter values are still
fully captured in the ticket description.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
All multi-line ASCII art blocks in _generate_ticket_description were
regular strings, not f-strings — so {hostname}, {'━' * box_width},
{cluster_health}, etc. were sent as literal template text instead of
rendered values. Added f prefix to all affected triple-quoted strings:
banner, executive_summary, DRIVE SPECIFICATIONS, DRIVE TIMELINE,
SMART STATUS, PARTITION, CEPH CLUSTER STATUS, CPU STATUS,
NETWORK STATUS, CONTAINER STORAGE.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- lint.yml: add notify-failure Matrix alert job
- test.yml: new workflow running pytest with pytest-cov for coverage
- .coveragerc: omit tests and site-packages from coverage
- .gitignore: ignore __pycache__ and .pyc files
- tests/test_hwmon.py: 49 unit tests covering SystemHealthMonitor
(temperature parsing, service monitoring, disk usage, metric collection,
dry run behaviour); uses unittest.mock to isolate from env/filesystem
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds .gitea/workflows/lint.yml running flake8 with .flake8 config.
Removes unused sys/urllib.request imports (F401).
Removes f prefix from 52 f-strings that had no placeholders (F541).
Auto-fixes trailing whitespace in blank lines (W293) via autopep8.
Fixes over-indentation in LXC storage check try block (E117).
Config ignores F841 (unused locals) and E501 (long lines).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Device paths like /dev/sdg are assigned by the kernel at boot and can
change after hot-swaps or reboots, causing duplicate tickets for the
same physical drive under a new letter.
Changes:
- _detect_issues(): issue strings now use serial number (e.g.
"Drive Z4ZC4B6R has SMART issues: ...") falling back to device
path only if smartctl cannot return a serial
- _create_tickets_for_issues(): capacity lookup resolves serial →
device via the details cache instead of regex on the issue string;
serial is included in the API payload as a dedicated field
- _generate_detailed_description(): drive lookup uses serial match
instead of /dev/ regex
The tinker_tickets API uses the serial field in the dedup hash so
the same physical drive always maps to the same ticket regardless
of which /dev/sdX letter it occupies.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Monitors ZFS pool status/usage and failed PBS tasks (backup, GC, sync).
Includes configurable thresholds (PBS_ZFS_WARNING/CRITICAL), Prometheus
metrics (hwmon_pbs_*), dry-run summary, issue categorization, and
priority classification. Enabled via PBS_ENABLED=true in .env config.
Fixes: #5
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Lightweight /health endpoint returns JSON with status, hostname, and
last check timestamp. Runs as daemon thread, activated via --health-server
flag or HEALTH_SERVER_ENABLED=true in .env config.
Fixes: #21
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Runs SMART checks concurrently (up to 8 workers) instead of
sequentially, significantly reducing check time on multi-drive systems.
Results are collected and processed in original disk order.
Fixes: #22
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Checks availability of required (smartctl, lsblk) and optional (nvme,
ceph, pct, dmidecode) tools at startup. Guards all tool-dependent code
sections to skip gracefully with informative log messages instead of
crashing. Also fixes pre-existing indentation bug in LXC exception handler.
Fixes: #19
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Explains that the ticket API deduplicates using SHA-256 hash of
(category + tags + hostname + device), not description/timestamp.
Clarifies the 24-hour dedup window and cluster-wide hostname exclusion.
Fixes: #18
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Ridata drives are known unreliable hardware. Instead of skipping them
with no notification, flag as REPLACEMENT_NEEDED and create tickets
recommending replacement.
Resolves #13
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use regex pattern matching instead of split()[N] indexing for parsing
pct df output. This is more robust against variations in column
formatting and whitespace.
Resolves #11
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add per-run cache for _get_drive_details() results. Each drive is
queried once via smartctl -i and the result is reused across SMART
health checks and ticket creation.
Resolves #15
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace dual-method detection (lsblk + glob scanning) with single
lsblk -p call that returns full device paths directly. Adds timeout,
returns sorted results for consistency.
Resolves #14
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
With 50 drives checked hourly over 30 days, history data can reach ~36MB
which exceeded the old 10MB limit causing constant file churn. Increase
to 50MB and make configurable via HISTORY_MAX_BYTES in .env.
Resolves #12
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Change default log level from DEBUG to INFO to reduce noise during
hourly execution. Add --verbose/-v CLI flag to enable DEBUG logging
when needed for troubleshooting.
Resolves #16
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wrap all int() conversions in try/except to handle malformed .env values
gracefully. Validate TICKET_API_KEY is not empty or placeholder value,
logging a warning instead of raising to preserve dry-run compatibility.
Resolves #17
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add escape function to sanitize backslashes, double quotes, and newlines
in label values per Prometheus text format spec. Prevents corrupted
metrics output from model names or paths containing these characters.
Resolves #10
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add timeout=30 to smartctl -i calls in _get_drive_details() and
_check_disk_firmware(), and dmidecode in _check_memory_usage().
Add TimeoutExpired handler in _check_disk_firmware(). Prevents
potential hangs when drives or system tools become unresponsive.
Resolves #9
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace digits[:2] truncation with regex extraction of complete number.
Previously "123°C" would be parsed as 12 instead of 123.
Resolves #8
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Check both file existence AND size > 0 before opening in r+ mode.
Previously, an empty file (0 bytes) would be opened in r+ mode, causing
json.load() to fail on the empty content.
Resolves #7
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move NEW_DRIVE_HOURS_THRESHOLD (720h) and SMART_ERROR_RECENT_HOURS (168h)
from inline literals to configurable CONFIG entries with .env support.
Resolves #20
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Exclude manufacturer operation counters (Seek_Error_Rate,
Command_Timeout, Raw_Read_Error_Rate) from critical issue
count to prevent false P1 escalation
- Fix missing space after [ceph] tag in ticket titles
Before: [hostname][auto][ceph]Ceph HEALTH_WARN
After: [hostname][auto][ceph] Ceph HEALTH_WARN
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add comprehensive Ceph cluster health monitoring
- Check cluster health status (HEALTH_OK/WARN/ERR)
- Monitor cluster usage with configurable thresholds
- Track OSD status (up/down) per node
- Separate cluster-wide vs node-specific issues
- Cluster-wide ticket deduplication
- Add [cluster-wide] scope tag for Ceph issues
- Cluster-wide issues deduplicate across all nodes
- Node-specific issues (OSD down) include hostname
- Add Prometheus metrics export
- export_prometheus_metrics() method
- write_prometheus_metrics() for textfile collector
- --metrics CLI flag to output metrics to stdout
- --export-json CLI flag to export health report as JSON
- Add Grafana dashboard template (grafana-dashboard.json)
- Add .gitignore
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add P5 (LOW) priority for informational/minimal impact alerts
- Expand ISSUE_PRIORITIES from 7 to 40+ comprehensive mappings
- Fix TICKET_TYPES to match tinker_tickets API (Issue, Problem, Task,
Maintenance, Upgrade, Install, Request)
- Fix TICKET_CATEGORIES to only Hardware and Software
- Add P1 escalation logic via _count_critical_issues() helper
- Rewrite _determine_ticket_priority() with full P1-P5 support
- Add CONFIG options: INCLUDE_INFO_TICKETS, PRIORITY_ESCALATION_THRESHOLD
- Filter INFO-level alerts from ticket creation by default
- Update _categorize_issue() to use valid ticket types
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Added intelligent categorization to match tickets with correct category and type
instead of defaulting everything to Hardware/Problem.
Changes:
- Added TICKET_CATEGORIES and TICKET_TYPES mappings for API consistency
- Created _categorize_issue() method to determine proper classification:
Hardware Issues:
- SMART/drive/disk errors → Hardware + Incident (critical/failed)
- SMART warnings → Hardware + Problem (needs investigation)
Software Issues:
- LXC/container/storage usage/CPU → Software category
- Critical levels → Software + Incident (service degradation)
- Warning levels → Software + Problem (preventive investigation)
Network Issues:
- Network failures/unreachable → Network + Incident
- Network warnings → Network + Problem
- Updated ticket creation to use _categorize_issue() and _determine_ticket_priority()
- Tickets now have correct tags: [incident] vs [problem] instead of always [maintenance]
- Category field in API payload now matches issue type (Hardware/Software/Network)
- Type field in API payload now reflects actual situation (Incident/Problem/Task)
Examples:
- "LXC storage usage >80%" → Software + Problem
- "Critical SMART errors" → Hardware + Incident
- "High CPU usage" → Software + Problem
- "Network unreachable" → Network + Incident
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Fixed all box alignment issues and improved visual consistency:
- Standardized box width to 78 chars across all sections
- Unified field width calculations (62 chars for values)
- Fixed executive summary box with proper dynamic width
- Fixed drive specifications box alignment
- Fixed drive timeline box with proper field widths
- Fixed SMART status box and improved temperature handling (None check)
- Fixed SMART attributes box with consistent widths
- Improved partition boxes:
- 50-char usage meter (2% per block) instead of 20-char
- Added percentage display next to meter
- Truncate long mountpoints in header to prevent overflow
- Consistent field widths across all fields
- Fixed firmware alerts box alignment
All boxes now use consistent Unicode box-drawing characters (┏━┓┃┗┛│)
with proper width calculations for perfect alignment.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Problem: Drive capacity was being extracted but never inserted into ticket titles.
The drive_size variable was calculated from drive details but omitted from the
ticket_title string construction.
Solution: Added drive_size to ticket title format between category and issue.
Example ticket titles now show:
- Before: "[hostname][auto][hardware]Drive /dev/sda has SMART issues..."
- After: "[hostname][auto][hardware][16.0 TB] Drive /dev/sda has SMART issues..."
This makes it easier to identify which drives need attention at a glance.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Added comprehensive manual execution section with both:
- Direct execution from local file (python3 hwmonDaemon.py)
- Remote execution matching systemd service (one-liner download+exec)
Both modes include dry-run and normal execution examples for testing
and production use.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Problem: Seagate drives were triggering tickets for "Critical Seek_Error_Rate"
and "Critical Command_Timeout" even though these are operation counters used by
the manufacturer, not actual errors.
Solution: Added filtering in _detect_issues() method to skip known manufacturer
operation counters:
- Seek_Error_Rate (Seagate/WD operation counter)
- Command_Timeout (OOS/Seagate operation counter)
- Raw_Read_Error_Rate (Seagate/WD operation counter)
These attributes are already correctly excluded from monitoring in manufacturer
profiles, but were still appearing in smart_issues list. This fix prevents them
from creating tickets while still catching legitimate SMART errors.
Changes:
- hwmonDaemon.py:1351-1378 - Added operation counter filtering in _detect_issues()
- Added debug logging when filtering manufacturer counters
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Critical fixes implemented:
- Add 10MB storage limit with automatic cleanup of old history files
- Add file locking (fcntl) to prevent race conditions in history writes
- Disable SMART monitoring for unreliable Ridata drives
- Fix bare except clause in _read_ecc_count() to properly catch errors
- Add timeouts to all network and subprocess calls (10s for API, 30s for subprocess)
- Fix unchecked regex in ticket creation to prevent AttributeError
- Add JSON decode error handling for ticket API responses
Service configuration improvements:
- hwmon.timer: Reduce jitter from 300s to 60s, add Persistent=true
- hwmon.service: Add Restart=on-failure, TimeoutStartSec=300, logging to journal
These changes improve reliability, prevent hung processes, eliminate race
conditions, and add proper error handling throughout the daemon.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
jaredandClaude Opus 4.8