From 38aa53c71ee08b30572aba75bea37b8b5b6c754d Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 31 Dec 2025 21:25:49 +0000 Subject: [PATCH] docs: update CLAUDE.md and README.md with recent improvements Updated both documentation files to reflect recent enhancements and fixes: CLAUDE.md Changes: - Added comprehensive Core Architecture section detailing main scripts and lib/ components - Documented lib/circuit_breaker.sh and lib/response_analyzer.sh modular architecture - Added detailed Exit Conditions and Thresholds section with circuit breaker thresholds - Documented advanced two-stage error detection process to eliminate false positives - Documented multi-line error matching for accurate stuck loop detection - Added Recent Improvements section highlighting v0.9.0 circuit breaker enhancements - Updated Global Installation section to include lib/ directory and ralph-import command - Test coverage details: 13 error detection + 9 stuck loop tests README.md Changes: - Updated "What's Working Now" section with circuit breaker enhancements - Added "Recent Improvements" section highlighting v0.9.0 updates - Updated test count from 75 to 97 tests (75 core + 13 error detection + 9 stuck loop) - Enhanced circuit breaker description with two-stage filtering details - Added circuit breaker thresholds to Exit Thresholds section - Updated test commands to include error detection and stuck loop test scripts - Updated current test status with specialized test file counts Key improvements documented: - Multi-line error matching fix for detect_stuck_loop function - JSON field false positive elimination (e.g., "is_error": false) - Two-stage error filtering for accurate error detection - Installation fix for lib/ directory components - 22 new tests added for circuit breaker functionality These updates ensure documentation accurately reflects the current state of the codebase following PR #6 (circuit-breaker false positives fix) and PR #4 (installation lib/ directory fix). --- CLAUDE.md | 88 +++++++++++++++++++++++++++++++++++++++++++++++++++++-- README.md | 48 +++++++++++++++++++++++------- 2 files changed, 122 insertions(+), 14 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 7b72417..0d4c2c2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -8,12 +8,32 @@ This is the Ralph for Claude Code repository - an autonomous AI development loop ## Core Architecture -The system consists of four main bash scripts that work together: +The system consists of four main bash scripts and a modular library system: + +### Main Scripts 1. **ralph_loop.sh** - The main autonomous loop that executes Claude Code repeatedly 2. **ralph_monitor.sh** - Live monitoring dashboard for tracking loop status 3. **setup.sh** - Project initialization script for new Ralph projects 4. **create_files.sh** - Bootstrap script that creates the entire Ralph system +5. **ralph_import.sh** - PRD/specification import tool that converts documents to Ralph format + +### Library Components (lib/) + +The system uses a modular architecture with reusable components in the `lib/` directory: + +1. **lib/circuit_breaker.sh** - Circuit breaker pattern implementation + - Prevents runaway loops by detecting stagnation + - Three states: CLOSED (normal), HALF_OPEN (monitoring), OPEN (halted) + - Configurable thresholds for no-progress and error detection + - Automatic state transitions and recovery + +2. **lib/response_analyzer.sh** - Intelligent response analysis + - Analyzes Claude Code output for completion signals + - Detects test-only loops and stuck error patterns + - Two-stage error filtering to eliminate false positives + - Multi-line error matching for accurate stuck loop detection + - Confidence scoring for exit decisions ## Key Commands @@ -115,9 +135,16 @@ Templates in `templates/` provide starting points for new projects: ## Global Installation Ralph installs to: -- **Commands**: `~/.local/bin/` (ralph, ralph-monitor, ralph-setup) +- **Commands**: `~/.local/bin/` (ralph, ralph-monitor, ralph-setup, ralph-import) - **Templates**: `~/.ralph/templates/` -- **Scripts**: `~/.ralph/` (ralph_loop.sh, ralph_monitor.sh, setup.sh) +- **Scripts**: `~/.ralph/` (ralph_loop.sh, ralph_monitor.sh, setup.sh, ralph_import.sh) +- **Libraries**: `~/.ralph/lib/` (circuit_breaker.sh, response_analyzer.sh) + +After installation, the following global commands are available: +- `ralph` - Start the autonomous development loop +- `ralph-monitor` - Launch the monitoring dashboard +- `ralph-setup` - Create a new Ralph-managed project +- `ralph-import` - Import PRD/specification documents to Ralph format ## Integration Points @@ -130,11 +157,66 @@ Ralph integrates with: ## Exit Conditions and Thresholds +Ralph uses multiple mechanisms to detect when to exit: + +### Exit Detection Thresholds - `MAX_CONSECUTIVE_TEST_LOOPS=3` - Exit if too many test-only iterations - `MAX_CONSECUTIVE_DONE_SIGNALS=2` - Exit on repeated completion signals - `TEST_PERCENTAGE_THRESHOLD=30%` - Flag if testing dominates recent loops - Completion detection via @fix_plan.md checklist items +### Circuit Breaker Thresholds +- `CB_NO_PROGRESS_THRESHOLD=3` - Open circuit after 3 loops with no file changes +- `CB_SAME_ERROR_THRESHOLD=5` - Open circuit after 5 loops with repeated errors +- `CB_OUTPUT_DECLINE_THRESHOLD=70%` - Open circuit if output declines by >70% + +### Error Detection + +Ralph uses advanced error detection with two-stage filtering to eliminate false positives: + +**Stage 1: JSON Field Filtering** +- Filters out JSON field patterns like `"is_error": false` that contain the word "error" but aren't actual errors +- Pattern: `grep -v '"[^"]*error[^"]*":'` + +**Stage 2: Actual Error Detection** +- Detects real error messages in specific contexts: + - Error prefixes: `Error:`, `ERROR:`, `error:` + - Context-specific errors: `]: error`, `Link: error` + - Error occurrences: `Error occurred`, `failed with error` + - Exceptions: `Exception`, `Fatal`, `FATAL` +- Pattern: `grep -cE '(^Error:|^ERROR:|^error:|\]: error|Link: error|Error occurred|failed with error|[Ee]xception|Fatal|FATAL)'` + +**Multi-line Error Matching** +- Detects stuck loops by verifying ALL error lines appear in ALL recent history files +- Uses literal fixed-string matching (`grep -qF`) to avoid regex edge cases +- Prevents false negatives when multiple distinct errors occur simultaneously + +## Recent Improvements + +### Circuit Breaker Enhancements (v0.9.0) + +**Multi-line Error Matching Fix** +- Fixed critical bug in `detect_stuck_loop` function where only the first error line was checked when multiple distinct errors occurred +- Now verifies ALL error lines appear in ALL recent history files for accurate stuck loop detection +- Uses nested loop checking with `grep -qF` for literal fixed-string matching + +**JSON Field False Positive Elimination** +- Implemented two-stage error filtering to avoid counting JSON field names as errors +- Stage 1 filters out patterns like `"is_error": false` that contain "error" as a field name +- Stage 2 detects actual error messages in specific contexts +- Aligned patterns between `response_analyzer.sh` and `ralph_loop.sh` for consistent behavior + +**Test Coverage** +- Added comprehensive test suite for error detection and stuck loop scenarios +- 13/13 error detection tests passing +- 9/9 stuck loop detection tests passing (including multi-error scenarios) +- Tests validate both single and multiple simultaneous recurring errors + +### Installation Improvements +- Added `lib/` directory to installation process for modular architecture +- Fixed issue where `response_analyzer.sh` and `circuit_breaker.sh` were not being copied during global installation +- All library components now properly installed to `~/.ralph/lib/` + ## Feature Development Quality Standards **CRITICAL**: All new features MUST meet the following mandatory requirements before being considered complete. diff --git a/README.md b/README.md index 8daa685..df0a078 100644 --- a/README.md +++ b/README.md @@ -20,12 +20,22 @@ Ralph is an implementation of the Geoffrey Huntley's technique for Claude Code t ### What's Working Now ✅ - Autonomous development loops with intelligent exit detection - Rate limiting with hourly reset (100 calls/hour, configurable) -- Circuit breaker prevents runaway loops -- Response analyzer with semantic understanding +- Circuit breaker with advanced error detection (prevents runaway loops) +- Response analyzer with semantic understanding and two-stage error filtering +- Multi-line error matching for accurate stuck loop detection - 5-hour API limit handling with user prompts - tmux integration for live monitoring - PRD import functionality -- 75 passing tests covering critical paths +- 97 passing tests covering critical paths (13 error detection + 9 stuck loop + 75 core tests) + +### Recent Improvements 🎉 + +**v0.9.0 - Circuit Breaker Enhancements** +- ✅ Fixed multi-line error matching in stuck loop detection +- ✅ Eliminated JSON field false positives (e.g., `"is_error": false`) +- ✅ Added two-stage error filtering for accurate detection +- ✅ Comprehensive test suite: 22 new tests for error detection +- ✅ Fixed installation to include lib/ directory components ### In Progress 🚧 - Expanding test coverage (60% → 90%+) @@ -51,7 +61,7 @@ Ralph is an implementation of the Geoffrey Huntley's technique for Claude Code t - **⏱️ Configurable Timeouts** - Set execution timeout for Claude Code operations (1-120 minutes) - **🔍 Verbose Progress Mode** - Optional detailed progress updates during execution - **🧠 Response Analyzer** - AI-powered analysis of Claude Code responses with semantic understanding -- **🔌 Circuit Breaker** - Smart error detection and recovery with automatic retry logic +- **🔌 Circuit Breaker** - Advanced error detection with two-stage filtering, multi-line error matching, and automatic recovery - **✅ Test Coverage** - 75 comprehensive tests with 60%+ code coverage (target: 90%+) ## 🚀 Quick Start @@ -205,10 +215,12 @@ ralph --status ``` The circuit breaker automatically: -- Detects API errors and rate limit issues -- Opens circuit after 5 consecutive failures -- Gradually recovers with half-open state -- Provides detailed error tracking and logging +- Detects API errors and rate limit issues with advanced two-stage filtering +- Opens circuit after 3 loops with no progress or 5 loops with same errors +- Eliminates false positives from JSON fields containing "error" +- Accurately detects stuck loops with multi-line error matching +- Gradually recovers with half-open monitoring state +- Provides detailed error tracking and logging with state history ### Claude API 5-Hour Limit @@ -255,12 +267,21 @@ ralph --monitor --verbose --timeout 30 ### Exit Thresholds Modify these variables in `~/.ralph/ralph_loop.sh`: + +**Exit Detection Thresholds:** ```bash MAX_CONSECUTIVE_TEST_LOOPS=3 # Exit after 3 test-only loops MAX_CONSECUTIVE_DONE_SIGNALS=2 # Exit after 2 "done" signals TEST_PERCENTAGE_THRESHOLD=30 # Flag if 30%+ loops are test-only ``` +**Circuit Breaker Thresholds:** +```bash +CB_NO_PROGRESS_THRESHOLD=3 # Open circuit after 3 loops with no file changes +CB_SAME_ERROR_THRESHOLD=5 # Open circuit after 5 loops with repeated errors +CB_OUTPUT_DECLINE_THRESHOLD=70 # Open circuit if output declines by >70% +``` + ## 📁 Project Structure Ralph creates a standardized structure for each project: @@ -318,7 +339,7 @@ If you want to run the test suite: # Install BATS testing framework npm install -g bats bats-support bats-assert -# Run all tests (75 tests) +# Run all tests (97 tests) bats tests/ # Run specific test suites @@ -326,13 +347,18 @@ bats tests/unit/test_rate_limiting.bats bats tests/unit/test_exit_detection.bats bats tests/integration/test_loop_execution.bats bats tests/integration/test_edge_cases.bats + +# Run error detection and circuit breaker tests +./tests/test_error_detection.sh +./tests/test_stuck_loop_detection.sh ``` Current test status: -- **75 tests** across 4 test files -- **100% pass rate** (75/75 passing) +- **97 tests** across 6 test files (75 core + 13 error detection + 9 stuck loop) +- **100% pass rate** (97/97 passing) - **~60% code coverage** (target: 90%+) - Comprehensive unit and integration tests +- Specialized tests for error detection and circuit breaker functionality ### Installing tmux