fix: stop false-positive [hardware] tickets in a redundant Ceph fleet #24

2026-07-15T15:25:57-04:00

jared commented

2026-07-15 15:25:57 -04:00

Two monitoring-logic fixes so hwmonDaemon stops raising spurious hardware
tickets on a Ceph-backed cluster with PBS backups:

_check_system_drive_indicators: count dmesg Buffer I/O / block errors
per-line and skip non-physical / network-backed devices (Ceph RBD, dm-,
loop). RBD Buffer I/O errors are storage-connectivity events (e.g. a Ceph
mon-session blip), not local drive faults, so they must not raise a
CRITICAL drive alert. Consistent with the existing _is_physical_disk
exclusion. (Fired a false "CRITICAL: Buffer I/O errors (75 occurrences)"
ticket where all 75 were on rbd devices.)
_get_attribute_thresholds: drop the Power_On_Hours warning/critical
threshold. Drive age alone is not a failure; with 2-3x Ceph redundancy
and PBS backups we run drives to hard-failure rather than replace on age.
The value is still recorded (history/description/new-drive logic) but no
longer generates a ticket. Real failure-predictors (reallocated/pending/
uncorrectable/CRC, self-test, trends) are unchanged.

Adds 7 regression tests (TestSystemDriveIndicators, TestAttributeThresholds).
All 98 tests pass; flake8 clean.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

Two monitoring-logic fixes so hwmonDaemon stops raising spurious hardware tickets on a Ceph-backed cluster with PBS backups: - _check_system_drive_indicators: count dmesg Buffer I/O / block errors per-line and skip non-physical / network-backed devices (Ceph RBD, dm-, loop). RBD Buffer I/O errors are storage-connectivity events (e.g. a Ceph mon-session blip), not local drive faults, so they must not raise a CRITICAL drive alert. Consistent with the existing _is_physical_disk exclusion. (Fired a false "CRITICAL: Buffer I/O errors (75 occurrences)" ticket where all 75 were on rbd devices.) - _get_attribute_thresholds: drop the Power_On_Hours warning/critical threshold. Drive age alone is not a failure; with 2-3x Ceph redundancy and PBS backups we run drives to hard-failure rather than replace on age. The value is still recorded (history/description/new-drive logic) but no longer generates a ticket. Real failure-predictors (reallocated/pending/ uncorrectable/CRC, self-test, trends) are unchanged. Adds 7 regression tests (TestSystemDriveIndicators, TestAttributeThresholds). All 98 tests pass; flake8 clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

jared added 1 commit 2026-07-15 15:25:58 -04:00

fix: stop false-positive [hardware] tickets in a redundant Ceph fleet

Lint / Python (flake8) (push) Successful in 38s

Details

Security / Python Security (bandit) (push) Successful in 39s

Details

Test / Python Tests (pytest) (push) Successful in 51s

Details

Lint / Notify on failure (push) Has been skipped

Details

Lint / Python (flake8) (pull_request) Successful in 1m22s

Details

Security / Python Security (bandit) (pull_request) Successful in 36s

Details

Test / Python Tests (pytest) (pull_request) Successful in 44s

Details

Lint / Notify on failure (pull_request) Has been skipped

Details

2c6b8c1294

Two monitoring-logic fixes so hwmonDaemon stops raising spurious hardware
tickets on a Ceph-backed cluster with PBS backups:

- _check_system_drive_indicators: count dmesg Buffer I/O / block errors
  per-line and skip non-physical / network-backed devices (Ceph RBD, dm-,
  loop). RBD Buffer I/O errors are storage-connectivity events (e.g. a Ceph
  mon-session blip), not local drive faults, so they must not raise a
  CRITICAL drive alert. Consistent with the existing _is_physical_disk
  exclusion. (Fired a false "CRITICAL: Buffer I/O errors (75 occurrences)"
  ticket where all 75 were on rbd devices.)

- _get_attribute_thresholds: drop the Power_On_Hours warning/critical
  threshold. Drive age alone is not a failure; with 2-3x Ceph redundancy
  and PBS backups we run drives to hard-failure rather than replace on age.
  The value is still recorded (history/description/new-drive logic) but no
  longer generates a ticket. Real failure-predictors (reallocated/pending/
  uncorrectable/CRC, self-test, trends) are unchanged.

Adds 7 regression tests (TestSystemDriveIndicators, TestAttributeThresholds).
All 98 tests pass; flake8 clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

jared merged commit 061625fc3a into main

2026-07-15 15:26:10 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: LotusGuild/hwmonDaemon#24