Add MIT license and .gitignore; stop tracking local AI notes; fix security CI pip install
Lint / Python (flake8) (push) Failing after 1m3s
Security / Python Security (bandit) (push) Failing after 49s

This commit is contained in:
2026-10-03 00:16:58 -04:00
parent f53f0c5b40
commit 0110ded15f
4 changed files with 44 additions and 235 deletions
+18
View File
@@ -0,0 +1,18 @@
# Python
__pycache__/
*.py[cod]
.venv/
venv/
*.egg-info/
# Local environment / secrets
.env
*.log
*.bak
*.swp
.DS_Store
# AI assistant notes (kept locally, not published)
Claude.md
CLAUDE.md
.claude/
-230
View File
@@ -1,230 +0,0 @@
# Ceph OSD Replacement Analyzer - Project Documentation
## Project Overview
**Purpose**: Intelligent analysis tool for identifying optimal Ceph OSD replacement candidates across an entire cluster by analyzing health metrics, capacity optimization potential, and cluster resilience factors.
**Type**: Python 3 CLI tool for Ceph storage cluster maintenance
**Target Users**: Storage administrators, DevOps engineers, and infrastructure teams managing Ceph clusters
## Architecture
### Core Components
1. **Data Collection Layer** ([ceph_osd_analyzer.py:34-172](ceph_osd_analyzer.py#L34-L172))
- Executes Ceph commands locally and via SSH
- Retrieves SMART data from all cluster nodes
- Handles both local `ceph device query-daemon-health-metrics` and remote `smartctl` fallback
- Device path resolution with dm-device mapping support
2. **Analysis Engine** ([ceph_osd_analyzer.py:173-357](ceph_osd_analyzer.py#L173-L357))
- SMART health parsing for HDD and NVMe devices
- Capacity optimization scoring
- Cluster resilience impact calculation
- Multi-factor weighted scoring system
3. **Reporting System** ([ceph_osd_analyzer.py:361-525](ceph_osd_analyzer.py#L361-L525))
- Color-coded console output
- Top 15 ranked replacement candidates
- Summary by device class (HDD/NVMe)
- Per-host analysis breakdown
### Key Design Decisions
**Remote SMART Data Collection**: The script uses SSH to gather SMART data from all cluster nodes, not just the local node. This is critical because OSDs are distributed across multiple physical hosts.
**Fallback Strategy**: Primary method uses `ceph device query-daemon-health-metrics`, with automatic fallback to direct `smartctl` queries via SSH if Ceph's built-in metrics are unavailable.
**Device Mapping**: Handles complex storage configurations including device-mapper devices, resolving them to physical drives using `lsblk` and symlink resolution.
**Weighted Scoring**: 60% health, 30% capacity optimization, 10% resilience - prioritizes failing drives while considering operational efficiency.
## Scoring Algorithm
### Health Score (60% weight)
**HDD Metrics** ([ceph_osd_analyzer.py:183-236](ceph_osd_analyzer.py#L183-L236)):
- Reallocated sectors (ID 5): -20 points for any presence
- Spin retry count (ID 10): -15 points
- Pending sectors (ID 197): -25 points (critical indicator)
- Uncorrectable sectors (ID 198): -30 points (critical)
- Temperature (ID 190/194): -10 points if >60°C
- Age (ID 9): -15 points if >5 years
**NVMe Metrics** ([ceph_osd_analyzer.py:239-267](ceph_osd_analyzer.py#L239-L267)):
- Available spare: penalized if <50%
- Percentage used: -30 points if >80%
- Media errors: -25 points for any errors
- Temperature: -10 points if >70°C
### Capacity Score (30% weight)
([ceph_osd_analyzer.py:271-311](ceph_osd_analyzer.py#L271-L311))
- **Small drives prioritized**: <2TB = +40 points (maximum capacity gain)
- **Medium drives**: 2-5TB = +30 points, 5-10TB = +15 points
- **High utilization penalty**: >70% = -15 points (migration complexity)
- **Host balance bonus**: +15 points if below host average weight
### Resilience Score (10% weight)
([ceph_osd_analyzer.py:313-357](ceph_osd_analyzer.py#L313-L357))
- Hosts with >20% above average OSD count: +20 points
- Presence of down OSDs on same host: +15 points (hardware issues)
## Usage Patterns
### One-Line Execution (Recommended)
```bash
sudo python3 -c "import urllib.request; exec(urllib.request.urlopen('http://10.10.10.63:3000/LotusGuild/analyzeOSDs/raw/branch/main/ceph_osd_analyzer.py').read().decode())" --debug --class hdd
```
**Why**: Always uses latest version, no local installation, integrates easily into automation.
### Command-Line Options
- `--class [hdd|nvme]`: Filter by device type
- `--min-size N`: Minimum OSD size in TB
- `--debug`: Enable verbose debugging output
### Typical Workflow
1. Run analysis during maintenance window
2. Identify top 3-5 candidates with scores >70
3. Review health issues and capacity gains
4. Plan replacement based on available hardware
5. Execute OSD out/destroy/replace operations
## Dependencies
### Required Packages
- Python 3.6+ (standard library only, no external dependencies)
- `smartmontools` package (`smartctl` binary)
- SSH access configured between all cluster nodes
### Required Permissions
- Ceph admin keyring access
- `sudo` privileges for SMART data retrieval
- SSH key-based authentication to all OSD hosts
### Ceph Commands Used
- `ceph osd tree -f json`: Cluster topology
- `ceph osd df -f json`: Disk usage statistics
- `ceph osd metadata osd.N -f json`: OSD device information
- `ceph device query-daemon-health-metrics osd.N`: SMART data
## Output Interpretation
### Replacement Score Ranges
- **70-100** (RED): Critical - immediate replacement recommended
- **50-69** (YELLOW): High priority - plan replacement soon
- **30-49**: Medium priority - next upgrade cycle
- **0-29** (GREEN): Low priority - healthy drives
### Health Score Ranges
- **80-100** (GREEN): Excellent condition
- **60-79** (YELLOW): Monitor for issues
- **40-59**: Fair - multiple concerns
- **0-39** (RED): Critical - replace urgently
## Common Issues & Solutions
### "No SMART data available"
- **Cause**: Missing `smartmontools` or insufficient permissions
- **Solution**: `apt install smartmontools` and verify sudo access
### SSH Timeout Errors
- **Cause**: Node unreachable or SSH keys not configured
- **Solution**: Verify connectivity with `ssh -o ConnectTimeout=5 <host> hostname`
### Device Path Resolution Failures
- **Cause**: Non-standard OSD deployment or encryption
- **Solution**: Enable `--debug` to see device resolution attempts
### dm-device Mapping Issues
- **Cause**: LVM or LUKS encrypted OSDs
- **Solution**: Script automatically resolves via `lsblk -no pkname`
## Development Notes
### Code Structure
- **Single file design**: Easier to execute remotely via `exec()`
- **Minimal dependencies**: Uses only Python standard library
- **Color-coded output**: ANSI escape codes for terminal display
- **Debug mode**: Comprehensive logging when `--debug` enabled
### Notable Functions
**`run_command()`** ([ceph_osd_analyzer.py:34-56](ceph_osd_analyzer.py#L34-L56)): Universal command executor with SSH support and JSON parsing
**`get_device_path_for_osd()`** ([ceph_osd_analyzer.py:84-122](ceph_osd_analyzer.py#L84-L122)): Complex device resolution logic handling metadata, symlinks, and dm-devices
**`get_smart_data_remote()`** ([ceph_osd_analyzer.py:124-145](ceph_osd_analyzer.py#L124-L145)): Remote SMART data collection with device type detection
**`parse_smart_health()`** ([ceph_osd_analyzer.py:173-269](ceph_osd_analyzer.py#L173-L269)): SMART attribute parsing with device-class-specific logic
### Future Enhancement Opportunities
1. **Parallel data collection**: Use threading for faster cluster-wide analysis
2. **Historical trending**: Track scores over time to predict failures
3. **JSON output mode**: For integration with monitoring systems
4. **Cost-benefit analysis**: Factor in replacement drive costs
5. **PG rebalance impact**: Estimate data movement required
## Security Considerations
### Permissions Required
- Root access for `smartctl` execution
- SSH access to all OSD hosts
- Ceph admin keyring (read-only sufficient)
### Network Requirements
- Script assumes SSH connectivity between nodes
- No outbound internet access required (internal-only tool)
- Hardcoded internal git server URL: `http://10.10.10.63:3000`
### SSH Configuration
- Uses `-o StrictHostKeyChecking=no` for automated execution
- 5-second connection timeout to handle unreachable nodes
- Assumes key-based authentication is configured
## Related Infrastructure
**Internal Git Server**: `http://10.10.10.63:3000/LotusGuild/analyzeOSDs`
**Related Projects**:
- hwmonDaemon: Hardware monitoring daemon for continuous health checks
- Other LotusGuild infrastructure automation tools
## Maintenance
### Version Control
- Maintained in internal git repository
- One-line execution always pulls from `main` branch
- No formal versioning; latest commit is production
### Testing Checklist
- [ ] Test on cluster with mixed HDD/NVMe OSDs
- [ ] Verify SSH connectivity to all hosts
- [ ] Confirm SMART data retrieval for both device types
- [ ] Validate dm-device resolution on encrypted OSDs
- [ ] Check output formatting with various terminal widths
- [ ] Test `--class` and `--min-size` filtering
## Performance Characteristics
**Execution Time**: ~5-15 seconds per OSD depending on cluster size and SSH latency
**Bottlenecks**:
- Serial OSD processing (parallelization would help)
- SSH round-trip times for SMART data
- SMART data parsing can be slow for unresponsive drives
**Resource Usage**: Minimal CPU/memory, I/O bound on SSH operations
**Intended Audience**: LotusGuild infrastructure team
**Support**: Submit issues or pull requests to internal git repository
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2025-2026 Jared Vititoe
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+5 -5
View File
@@ -61,22 +61,22 @@ The script leverages multiple Ceph commands:
Run directly from your internal git server: Run directly from your internal git server:
```bash ```bash
sudo python3 -c "import urllib.request; exec(urllib.request.urlopen('http://10.10.10.63:3000/LotusGuild/analyzeOSDs/raw/branch/main/ceph_osd_analyzer.py').read().decode())" sudo python3 -c "import urllib.request; exec(urllib.request.urlopen('https://code.lotusguild.org/LotusGuild/analyzeOSDs/raw/branch/main/ceph_osd_analyzer.py').read().decode())"
``` ```
Run directly from internal git server with debug enabled: Run directly from internal git server with debug enabled:
```bash ```bash
sudo python3 -c "import urllib.request; exec(urllib.request.urlopen('http://10.10.10.63:3000/LotusGuild/analyzeOSDs/raw/branch/main/ceph_osd_analyzer.py').read().decode())" --debug sudo python3 -c "import urllib.request; exec(urllib.request.urlopen('https://code.lotusguild.org/LotusGuild/analyzeOSDs/raw/branch/main/ceph_osd_analyzer.py').read().decode())" --debug
``` ```
Most common execution Most common execution
```bash ```bash
sudo python3 -c "import urllib.request; exec(urllib.request.urlopen('http://10.10.10.63:3000/LotusGuild/analyzeOSDs/raw/branch/main/ceph_osd_analyzer.py').read().decode())" --debug --class hdd sudo python3 -c "import urllib.request; exec(urllib.request.urlopen('https://code.lotusguild.org/LotusGuild/analyzeOSDs/raw/branch/main/ceph_osd_analyzer.py').read().decode())" --debug --class hdd
``` ```
### Traditional Installation (not recommended) ### Traditional Installation (not recommended)
```bash ```bash
# Clone repository # Clone repository
git clone http://10.10.10.63:3000/LotusGuild/analyzeOSDs.git git clone https://code.lotusguild.org/LotusGuild/analyzeOSDs.git
cd analyzeOSDs cd analyzeOSDs
# Make executable # Make executable
@@ -218,7 +218,7 @@ Internal use only - LotusGuild infrastructure tools.
## Related Tools ## Related Tools
- [hwmonDaemon](http://10.10.10.63:3000/LotusGuild/hwmonDaemon) - Hardware monitoring daemon - [hwmonDaemon](https://code.lotusguild.org/LotusGuild/hwmonDaemon) - Hardware monitoring daemon
- Other LotusGuild infrastructure tools - Other LotusGuild infrastructure tools
## Changelog ## Changelog