* refactor(naming): remove @ prefix from on-disk filenames BREAKING CHANGE: Renames @fix_plan.md → fix_plan.md and @AGENT.md → AGENT.md This change improves POSIX compliance and compatibility with command-line tools. The @ prefix was originally used to avoid naming conflicts, but with the .ralph/ folder structure introduced in v0.10.0, this convention is no longer necessary. Changes: - Update all scripts to use new naming (fix_plan.md, AGENT.md) - Update migration script to handle both old and new naming conventions - Update templates to use new naming - Update all documentation references - Update all 420 tests to expect new naming (TDD approach) Migration: - Existing projects with @-prefixed files will be automatically renamed when running ralph-migrate - Projects already using the new naming will continue to work unchanged * docs(claude): update file naming conventions section * fix(migrate): prevent orphaned @-prefixed files during migration When both root/@fix_plan.md and .ralph/@fix_plan.md exist, the root file now takes priority and the .ralph/@fix_plan.md is removed (backup exists). This prevents orphaned legacy files after migration. --------- Co-authored-by: Test User <test@example.com>
1016 lines
24 KiB
Markdown
1016 lines
24 KiB
Markdown
# Testing Guide for Ralph
|
|
|
|
This guide provides comprehensive documentation for the Ralph test suite, helping contributors understand how to run, write, and maintain tests.
|
|
|
|
**Current Status**: 276 tests | 100% pass rate | CI/CD via GitHub Actions
|
|
|
|
---
|
|
|
|
## Table of Contents
|
|
|
|
1. [Quick Start](#quick-start)
|
|
2. [Test Organization](#test-organization)
|
|
3. [Writing Tests](#writing-tests)
|
|
4. [Test Helpers](#test-helpers)
|
|
5. [Coverage Requirements](#coverage-requirements)
|
|
6. [CI/CD Integration](#cicd-integration)
|
|
7. [Troubleshooting](#troubleshooting)
|
|
|
|
---
|
|
|
|
## Quick Start
|
|
|
|
### Prerequisites
|
|
|
|
Ensure you have the following installed:
|
|
|
|
```bash
|
|
# Node.js 18+ and npm
|
|
node --version # Should show v18+
|
|
npm --version
|
|
|
|
# jq for JSON processing
|
|
jq --version # Used by test fixtures
|
|
|
|
# git for integration tests
|
|
git --version
|
|
```
|
|
|
|
### Install Test Dependencies
|
|
|
|
```bash
|
|
npm install
|
|
```
|
|
|
|
This installs:
|
|
- **bats** (v1.12.0) - Bash Automated Testing System
|
|
- **bats-assert** - Assertion library
|
|
- **bats-support** - Support functions
|
|
|
|
### Run All Tests
|
|
|
|
```bash
|
|
# Run the complete test suite (unit + integration)
|
|
npm test
|
|
|
|
# Expected output:
|
|
# 1..276
|
|
# ok 1 - ...
|
|
# ok 2 - ...
|
|
# ...
|
|
# 276 tests, 0 failures
|
|
```
|
|
|
|
### Run Tests by Category
|
|
|
|
```bash
|
|
# Unit tests only (fast, isolated function tests)
|
|
npm run test:unit
|
|
|
|
# Integration tests only (component interaction tests)
|
|
npm run test:integration
|
|
|
|
# E2E tests only (full workflow tests)
|
|
npm run test:e2e
|
|
```
|
|
|
|
### Run Individual Test Files
|
|
|
|
```bash
|
|
# Run a specific test file
|
|
bats tests/unit/test_rate_limiting.bats
|
|
|
|
# Run with verbose output for debugging
|
|
bats --verbose-run tests/unit/test_cli_parsing.bats
|
|
|
|
# Run a single test by pattern (partial match)
|
|
bats tests/unit/test_rate_limiting.bats --filter "can_make_call"
|
|
```
|
|
|
|
---
|
|
|
|
## Test Organization
|
|
|
|
### Directory Structure
|
|
|
|
```
|
|
tests/
|
|
├── unit/ # Isolated function tests
|
|
│ ├── test_rate_limiting.bats # Rate limiting behavior (15 tests)
|
|
│ ├── test_exit_detection.bats # Exit signal detection (20 tests)
|
|
│ ├── test_cli_parsing.bats # CLI argument parsing (27 tests)
|
|
│ ├── test_cli_modern.bats # Modern CLI features (29 tests)
|
|
│ ├── test_json_parsing.bats # JSON output parsing (36 tests)
|
|
│ └── test_session_continuity.bats # Session lifecycle (26 tests)
|
|
│
|
|
├── integration/ # Component interaction tests
|
|
│ ├── test_loop_execution.bats # Main loop behavior (20 tests)
|
|
│ ├── test_edge_cases.bats # Edge case handling (20 tests)
|
|
│ ├── test_installation.bats # Global install workflow (14 tests)
|
|
│ ├── test_project_setup.bats # Project setup (setup.sh) (36 tests)
|
|
│ └── test_prd_import.bats # PRD import workflow (33 tests)
|
|
│
|
|
├── e2e/ # End-to-end tests (planned)
|
|
│
|
|
└── helpers/ # Shared test utilities
|
|
├── test_helper.bash # Assertions and setup functions
|
|
├── mocks.bash # Mock functions for external commands
|
|
└── fixtures.bash # Sample data generators
|
|
```
|
|
|
|
### Test Categories
|
|
|
|
| Category | Purpose | Execution Speed | Dependencies |
|
|
|----------|---------|-----------------|--------------|
|
|
| **Unit** | Test individual functions in isolation | Fast (<1s per file) | None (uses mocks) |
|
|
| **Integration** | Test component interactions | Medium (1-5s per file) | Real git, filesystem |
|
|
| **E2E** | Test complete workflows | Slow (>5s per file) | Full environment |
|
|
|
|
### Naming Conventions
|
|
|
|
- **Test files**: `test_<component_name>.bats`
|
|
- **Test functions**: Descriptive sentences: `@test "can_make_call returns success when under limit"`
|
|
- **Location**: Place tests in `unit/` or `integration/` based on scope
|
|
|
|
---
|
|
|
|
## Writing Tests
|
|
|
|
### BATS Fundamentals
|
|
|
|
BATS (Bash Automated Testing System) is our testing framework. Each `.bats` file contains test cases that run in isolated subshells.
|
|
|
|
#### Basic Test Structure
|
|
|
|
```bash
|
|
#!/usr/bin/env bats
|
|
# Description of what this file tests
|
|
|
|
# Load helper functions (required)
|
|
load '../helpers/test_helper'
|
|
|
|
# Setup runs before EACH test
|
|
setup() {
|
|
export TEST_TEMP_DIR="$(mktemp -d)"
|
|
cd "$TEST_TEMP_DIR"
|
|
# Initialize test environment...
|
|
}
|
|
|
|
# Teardown runs after EACH test
|
|
teardown() {
|
|
cd /
|
|
rm -rf "$TEST_TEMP_DIR"
|
|
}
|
|
|
|
# Test case syntax: @test "description" { commands }
|
|
@test "descriptive name of what is being tested" {
|
|
# Arrange: set up test conditions
|
|
echo "50" > "$CALL_COUNT_FILE"
|
|
|
|
# Act: run the command being tested
|
|
run my_function
|
|
|
|
# Assert: verify the results
|
|
assert_success
|
|
assert_equal "$output" "expected output"
|
|
}
|
|
```
|
|
|
|
#### The `run` Command
|
|
|
|
The `run` command captures output and exit status:
|
|
|
|
```bash
|
|
@test "example using run command" {
|
|
run ls /nonexistent
|
|
|
|
# $status contains exit code (0 = success)
|
|
echo "Exit code was: $status"
|
|
|
|
# $output contains stdout + stderr
|
|
echo "Output was: $output"
|
|
|
|
# Assert on these values
|
|
assert_failure # Expect non-zero exit
|
|
[[ "$output" == *"No such"* ]] # Check output contains text
|
|
}
|
|
```
|
|
|
|
### Example: Unit Test
|
|
|
|
From `tests/unit/test_rate_limiting.bats`:
|
|
|
|
```bash
|
|
#!/usr/bin/env bats
|
|
# Unit Tests for Rate Limiting Logic
|
|
|
|
load '../helpers/test_helper'
|
|
|
|
setup() {
|
|
source "$(dirname "$BATS_TEST_FILENAME")/../helpers/test_helper.bash"
|
|
|
|
export MAX_CALLS_PER_HOUR=100
|
|
export CALL_COUNT_FILE=".call_count"
|
|
export TEST_TEMP_DIR="$(mktemp -d /tmp/ralph-test.XXXXXX)"
|
|
cd "$TEST_TEMP_DIR"
|
|
|
|
echo "0" > "$CALL_COUNT_FILE"
|
|
}
|
|
|
|
teardown() {
|
|
cd /
|
|
rm -rf "$TEST_TEMP_DIR"
|
|
}
|
|
|
|
# Define the function being tested (extracted from production code)
|
|
can_make_call() {
|
|
local calls_made=0
|
|
[[ -f "$CALL_COUNT_FILE" ]] && calls_made=$(cat "$CALL_COUNT_FILE")
|
|
[[ $calls_made -ge $MAX_CALLS_PER_HOUR ]] && return 1
|
|
return 0
|
|
}
|
|
|
|
@test "can_make_call returns success when under limit" {
|
|
echo "50" > "$CALL_COUNT_FILE"
|
|
run can_make_call
|
|
assert_success
|
|
}
|
|
|
|
@test "can_make_call returns failure when at limit" {
|
|
echo "100" > "$CALL_COUNT_FILE"
|
|
run can_make_call
|
|
assert_failure
|
|
}
|
|
```
|
|
|
|
### Example: Integration Test
|
|
|
|
From `tests/integration/test_project_setup.bats`:
|
|
|
|
```bash
|
|
#!/usr/bin/env bats
|
|
# Integration tests for setup.sh project initialization
|
|
|
|
load '../helpers/test_helper'
|
|
load '../helpers/fixtures'
|
|
|
|
SETUP_SCRIPT="${BATS_TEST_DIRNAME}/../../setup.sh"
|
|
|
|
setup() {
|
|
export TEST_TEMP_DIR="$(mktemp -d)"
|
|
export HOME="$TEST_TEMP_DIR/home"
|
|
mkdir -p "$HOME/.ralph/templates"
|
|
|
|
# Copy real templates for integration testing
|
|
cp -r "${BATS_TEST_DIRNAME}/../../templates/"* "$HOME/.ralph/templates/"
|
|
|
|
cd "$TEST_TEMP_DIR"
|
|
}
|
|
|
|
teardown() {
|
|
cd /
|
|
rm -rf "$TEST_TEMP_DIR"
|
|
}
|
|
|
|
@test "setup.sh creates project directory with correct structure" {
|
|
run bash "$SETUP_SCRIPT" "test-project"
|
|
|
|
assert_success
|
|
assert_dir_exists "$TEST_TEMP_DIR/test-project"
|
|
assert_dir_exists "$TEST_TEMP_DIR/test-project/specs"
|
|
assert_dir_exists "$TEST_TEMP_DIR/test-project/src"
|
|
assert_dir_exists "$TEST_TEMP_DIR/test-project/logs"
|
|
}
|
|
|
|
@test "setup.sh initializes git repository" {
|
|
bash "$SETUP_SCRIPT" "test-project"
|
|
|
|
cd "$TEST_TEMP_DIR/test-project"
|
|
[[ -d ".git" ]]
|
|
|
|
run git log --oneline -1
|
|
assert_success
|
|
[[ "$output" == *"Initial commit"* ]]
|
|
}
|
|
```
|
|
|
|
### Example: Testing with Mocks
|
|
|
|
When testing functions that call external commands:
|
|
|
|
```bash
|
|
#!/usr/bin/env bats
|
|
|
|
load '../helpers/test_helper'
|
|
load '../helpers/mocks'
|
|
|
|
setup() {
|
|
source "$(dirname "$BATS_TEST_FILENAME")/../helpers/mocks.bash"
|
|
setup_mocks # Replace git, tmux, etc. with mocks
|
|
|
|
export TEST_TEMP_DIR="$(mktemp -d)"
|
|
cd "$TEST_TEMP_DIR"
|
|
}
|
|
|
|
teardown() {
|
|
teardown_mocks # Restore original commands
|
|
cd /
|
|
rm -rf "$TEST_TEMP_DIR"
|
|
}
|
|
|
|
@test "function handles git unavailable gracefully" {
|
|
# Configure mock to simulate git not installed
|
|
export MOCK_GIT_AVAILABLE=false
|
|
|
|
run function_that_uses_git
|
|
|
|
assert_failure
|
|
[[ "$output" == *"git: command not found"* ]]
|
|
}
|
|
|
|
@test "function uses Claude Code successfully" {
|
|
# Configure successful mock response
|
|
export MOCK_CLAUDE_SUCCESS=true
|
|
export MOCK_CLAUDE_OUTPUT="Task completed"
|
|
|
|
run function_that_calls_claude
|
|
|
|
assert_success
|
|
[[ "$output" == *"Task completed"* ]]
|
|
}
|
|
```
|
|
|
|
### Best Practices
|
|
|
|
1. **Test One Thing**: Each test should verify a single behavior
|
|
```bash
|
|
# Good: focused test
|
|
@test "increment counter increases value by 1" { ... }
|
|
|
|
# Bad: multiple behaviors
|
|
@test "counter increments and respects limit and resets hourly" { ... }
|
|
```
|
|
|
|
2. **Descriptive Names**: Tests should read as documentation
|
|
```bash
|
|
# Good: clear intent
|
|
@test "can_make_call returns failure when at limit"
|
|
|
|
# Bad: unclear
|
|
@test "test limit"
|
|
```
|
|
|
|
3. **Isolate Tests**: Each test should set up its own state
|
|
```bash
|
|
setup() {
|
|
export TEST_TEMP_DIR="$(mktemp -d)" # Fresh directory each test
|
|
cd "$TEST_TEMP_DIR"
|
|
}
|
|
```
|
|
|
|
4. **Clean Up**: Always restore state in teardown
|
|
```bash
|
|
teardown() {
|
|
teardown_mocks # Restore mocked commands
|
|
cd /
|
|
rm -rf "$TEST_TEMP_DIR" # Clean up files
|
|
}
|
|
```
|
|
|
|
5. **Use Helpers**: Don't duplicate setup/assertion code
|
|
```bash
|
|
# Good: use provided helpers
|
|
assert_file_exists "output.txt"
|
|
assert_valid_json "data.json"
|
|
|
|
# Bad: inline checks
|
|
[[ -f "output.txt" ]] || fail "File missing"
|
|
```
|
|
|
|
---
|
|
|
|
## Test Helpers
|
|
|
|
### test_helper.bash
|
|
|
|
Located at `tests/helpers/test_helper.bash`, provides core utilities:
|
|
|
|
#### Assertion Functions
|
|
|
|
```bash
|
|
# Exit status assertions
|
|
assert_success # Assert $status == 0
|
|
assert_failure # Assert $status != 0
|
|
|
|
# Value assertions
|
|
assert_equal "$actual" "$expected" # Compare two values
|
|
assert_output "expected text" # Compare $output exactly
|
|
|
|
# File assertions
|
|
assert_file_exists "path/to/file" # File must exist
|
|
assert_file_not_exists "path/to/file" # File must NOT exist
|
|
assert_dir_exists "path/to/dir" # Directory must exist
|
|
|
|
# JSON assertions
|
|
assert_valid_json "file.json" # Validate JSON syntax
|
|
get_json_field "file.json" "field" # Extract field value
|
|
```
|
|
|
|
#### Setup Utilities
|
|
|
|
```bash
|
|
# Provided environment variables (set in setup)
|
|
$TEST_TEMP_DIR # Unique temp directory for this test
|
|
$PROMPT_FILE # "PROMPT.md"
|
|
$LOG_DIR # "logs"
|
|
$STATUS_FILE # "status.json"
|
|
$CALL_COUNT_FILE # ".call_count"
|
|
$EXIT_SIGNALS_FILE # ".exit_signals"
|
|
|
|
# Mock data creation
|
|
create_mock_prompt # Create sample PROMPT.md
|
|
create_mock_fix_plan 5 2 # Create fix_plan.md (5 total, 2 completed)
|
|
create_mock_status 1 42 100 # Create status.json (loop 1, 42 calls, 100 max)
|
|
create_mock_exit_signals 0 2 0 # Create exit signals (0 test, 2 done, 0 complete)
|
|
```
|
|
|
|
#### Date Mocking
|
|
|
|
```bash
|
|
# Mock date for deterministic tests
|
|
mock_date "2025093012" # Set fixed date
|
|
# ... run tests ...
|
|
restore_date # Restore system date
|
|
```
|
|
|
|
### mocks.bash
|
|
|
|
Located at `tests/helpers/mocks.bash`, provides mock implementations:
|
|
|
|
#### Available Mocks
|
|
|
|
```bash
|
|
# Claude Code CLI mock
|
|
mock_claude_code() # Configurable via MOCK_CLAUDE_* vars
|
|
MOCK_CLAUDE_SUCCESS=true|false
|
|
MOCK_CLAUDE_OUTPUT="response text"
|
|
MOCK_CLAUDE_EXIT_CODE=0
|
|
|
|
# tmux mock (terminal multiplexer)
|
|
mock_tmux() # Configurable via MOCK_TMUX_* vars
|
|
MOCK_TMUX_AVAILABLE=true|false
|
|
|
|
# git mock
|
|
mock_git() # Configurable via MOCK_GIT_* vars
|
|
MOCK_GIT_AVAILABLE=true|false
|
|
MOCK_GIT_REPO=true|false
|
|
|
|
# Other mocks
|
|
mock_notify_send() # Desktop notifications
|
|
mock_osascript() # macOS notifications
|
|
mock_stat() # File statistics
|
|
mock_timeout() # Command timeout
|
|
```
|
|
|
|
#### Using Mocks
|
|
|
|
```bash
|
|
setup() {
|
|
source ".../helpers/mocks.bash"
|
|
setup_mocks # Install all mocks
|
|
}
|
|
|
|
teardown() {
|
|
teardown_mocks # Remove all mocks
|
|
}
|
|
|
|
@test "example with mock configuration" {
|
|
# Configure mock behavior
|
|
export MOCK_CLAUDE_SUCCESS=true
|
|
export MOCK_CLAUDE_OUTPUT='{"status": "complete"}'
|
|
|
|
run my_function_that_calls_claude
|
|
|
|
assert_success
|
|
}
|
|
```
|
|
|
|
### fixtures.bash
|
|
|
|
Located at `tests/helpers/fixtures.bash`, provides sample data:
|
|
|
|
#### PRD Fixtures
|
|
|
|
```bash
|
|
# Create sample PRD documents
|
|
create_sample_prd_md "output.md" # Markdown PRD
|
|
create_sample_prd_txt "output.txt" # Plain text PRD
|
|
create_sample_prd_json "output.json" # JSON PRD
|
|
```
|
|
|
|
#### Project Fixtures
|
|
|
|
```bash
|
|
# Create sample Ralph project files
|
|
create_sample_prompt "PROMPT.md"
|
|
create_sample_fix_plan "fix_plan.md" 10 3 # 10 tasks, 3 completed
|
|
create_sample_agent_md "AGENT.md"
|
|
|
|
# Create complete project structure
|
|
create_test_project "project-name"
|
|
# Creates: PROMPT.md, fix_plan.md, AGENT.md, specs/, src/, logs/, etc.
|
|
```
|
|
|
|
#### Output Fixtures
|
|
|
|
```bash
|
|
# Create sample Claude outputs
|
|
create_sample_claude_output_success "output.log" # Successful run
|
|
create_sample_claude_output_error "output.log" # Error response
|
|
create_sample_claude_output_limit "output.log" # Rate limit hit
|
|
|
|
# Create sample status files
|
|
create_sample_status_running "status.json"
|
|
create_sample_status_completed "status.json"
|
|
create_sample_progress_executing "progress.json"
|
|
```
|
|
|
|
---
|
|
|
|
## Coverage Requirements
|
|
|
|
### Quality Gates
|
|
|
|
| Metric | Requirement | Enforcement |
|
|
|--------|-------------|-------------|
|
|
| **Test Pass Rate** | 100% | **Blocking** - CI fails on any test failure |
|
|
| **Coverage Target** | 85%+ | Informational only |
|
|
|
|
### Why Coverage Is Informational
|
|
|
|
Bash code coverage with kcov has fundamental limitations:
|
|
|
|
> **Technical Limitation**: kcov uses LD_PRELOAD to trace execution, but cannot instrument subprocesses spawned by bats. Each test runs in a subprocess that kcov cannot follow.
|
|
>
|
|
> Reference: [bats-core/bats-core#15](https://github.com/bats-core/bats-core/issues/15)
|
|
|
|
**Result**: Reported coverage percentages are lower than actual coverage. **Test pass rate (100%) is the enforced quality gate.**
|
|
|
|
### Running Coverage Locally
|
|
|
|
```bash
|
|
# Install kcov (Ubuntu/Debian)
|
|
sudo apt-get install kcov
|
|
|
|
# Or build from source
|
|
git clone https://github.com/SimonKagstrom/kcov.git
|
|
cd kcov && mkdir build && cd build
|
|
cmake .. && make && sudo make install
|
|
|
|
# Run tests with coverage
|
|
mkdir -p coverage
|
|
kcov --include-path="$(pwd)/ralph_loop.sh,$(pwd)/lib" \
|
|
coverage/ \
|
|
bats tests/unit/
|
|
|
|
# View report
|
|
open coverage/index.html # macOS
|
|
xdg-open coverage/index.html # Linux
|
|
```
|
|
|
|
### Coverage Best Practices
|
|
|
|
1. **Prioritize Critical Paths**: Test the main loop, exit detection, circuit breaker
|
|
2. **Test Error Conditions**: Verify graceful handling of failures
|
|
3. **Don't Chase 100%**: Quality over quantity
|
|
4. **New Features Need Tests**: All PRs introducing features must include tests
|
|
|
|
---
|
|
|
|
## CI/CD Integration
|
|
|
|
### GitHub Actions Pipeline
|
|
|
|
The test workflow is defined in `.github/workflows/test.yml`:
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ GitHub Actions Pipeline │
|
|
├─────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ Triggers: push (main, develop), PR (main) │
|
|
│ │
|
|
│ ┌─────────────────┐ ┌─────────────────┐ │
|
|
│ │ test job │────▶│ coverage job │ │
|
|
│ └────────┬────────┘ └────────┬────────┘ │
|
|
│ │ │ │
|
|
│ • Checkout repo • Build kcov from source │
|
|
│ • Setup Node.js 18 • Run tests with coverage │
|
|
│ • Install deps (jq) • Parse coverage results │
|
|
│ • Run unit tests • Check threshold (disabled) │
|
|
│ • Run integration • Upload artifacts │
|
|
│ • Generate summary • Upload to Codecov (optional) │
|
|
│ │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
### Workflow Stages
|
|
|
|
#### 1. Test Job (Required)
|
|
|
|
```yaml
|
|
test:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@v3
|
|
- uses: actions/setup-node@v3
|
|
with:
|
|
node-version: '18'
|
|
- run: npm install && sudo apt-get install -y jq
|
|
- run: npm run test:unit # Must pass
|
|
- run: npm run test:integration # Allowed to fail (|| true)
|
|
- run: npm run test:e2e # Allowed to fail (|| true)
|
|
```
|
|
|
|
#### 2. Coverage Job (Informational)
|
|
|
|
```yaml
|
|
coverage:
|
|
runs-on: ubuntu-latest
|
|
needs: test # Only runs after test passes
|
|
env:
|
|
COVERAGE_THRESHOLD: 0 # Disabled
|
|
```
|
|
|
|
### Viewing CI Results
|
|
|
|
1. **GitHub Actions tab**: See workflow runs and logs
|
|
2. **Step Summary**: Test results appear in PR summary
|
|
3. **Coverage Artifacts**: Downloadable for 7 days
|
|
4. **Codecov** (optional): Interactive coverage reports
|
|
|
|
### Local vs CI Differences
|
|
|
|
| Aspect | Local | CI |
|
|
|--------|-------|-----|
|
|
| Environment | Your machine | ubuntu-latest container |
|
|
| Node version | Your installed version | v18 (specified) |
|
|
| Dependencies | Cached | Fresh install |
|
|
| Coverage | Optional | Automatic |
|
|
| Artifacts | Manual | Auto-uploaded |
|
|
|
|
### Reproducing CI Failures
|
|
|
|
```bash
|
|
# Match CI environment
|
|
nvm use 18
|
|
npm ci # Clean install (not npm install)
|
|
|
|
# Run tests in CI order
|
|
npm run test:unit
|
|
npm run test:integration
|
|
npm run test:e2e
|
|
|
|
# Check for environment-specific issues
|
|
uname -a # OS differences
|
|
bash --version # Bash version
|
|
```
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
### Test Failures
|
|
|
|
#### Reading BATS Output
|
|
|
|
```bash
|
|
# Verbose output shows each test
|
|
bats --verbose-run tests/unit/test_rate_limiting.bats
|
|
|
|
# TAP format for parsing
|
|
bats --tap tests/unit/test_rate_limiting.bats
|
|
|
|
# Timing information
|
|
bats --timing tests/unit/test_rate_limiting.bats
|
|
```
|
|
|
|
#### Understanding Failure Messages
|
|
|
|
```
|
|
not ok 3 - can_make_call returns success when under limit
|
|
# (in test file tests/unit/test_rate_limiting.bats, line 58)
|
|
# `assert_success' failed
|
|
# Expected success but got status 1
|
|
# Output: Error: file not found
|
|
```
|
|
|
|
- **Line 58**: Where the assertion failed
|
|
- **assert_success failed**: Exit code wasn't 0
|
|
- **status 1**: Actual exit code
|
|
- **Output**: What the command printed
|
|
|
|
#### Debugging Steps
|
|
|
|
1. **Run single test**:
|
|
```bash
|
|
bats tests/unit/test_rate_limiting.bats --filter "can_make_call"
|
|
```
|
|
|
|
2. **Add debug output**:
|
|
```bash
|
|
@test "debugging example" {
|
|
echo "Before command" >&3 # Print to stdout during test
|
|
|
|
run my_function
|
|
|
|
echo "Status: $status" >&3
|
|
echo "Output: $output" >&3
|
|
|
|
assert_success
|
|
}
|
|
```
|
|
|
|
3. **Use set -x for tracing**:
|
|
```bash
|
|
@test "trace example" {
|
|
set -x # Enable bash tracing
|
|
run my_function
|
|
set +x # Disable tracing
|
|
}
|
|
```
|
|
|
|
4. **Preserve temp directory**:
|
|
```bash
|
|
teardown() {
|
|
echo "Temp dir: $TEST_TEMP_DIR" >&3
|
|
# Comment out cleanup to inspect:
|
|
# rm -rf "$TEST_TEMP_DIR"
|
|
}
|
|
```
|
|
|
|
### Mock Issues
|
|
|
|
#### Mock Not Being Called
|
|
|
|
```bash
|
|
# Verify setup_mocks was called
|
|
setup() {
|
|
source "$(dirname "$BATS_TEST_FILENAME")/../helpers/mocks.bash"
|
|
setup_mocks # Must call this!
|
|
}
|
|
|
|
# Verify function is exported
|
|
type git # Should show "git is a function"
|
|
```
|
|
|
|
#### Wrong Mock Response
|
|
|
|
```bash
|
|
# Check environment variables
|
|
@test "debug mock" {
|
|
echo "MOCK_CLAUDE_SUCCESS: $MOCK_CLAUDE_SUCCESS" >&3
|
|
echo "MOCK_CLAUDE_OUTPUT: $MOCK_CLAUDE_OUTPUT" >&3
|
|
|
|
# Set explicitly if needed
|
|
export MOCK_CLAUDE_SUCCESS=true
|
|
export MOCK_CLAUDE_OUTPUT="expected response"
|
|
}
|
|
```
|
|
|
|
#### Mock Cleanup Issues
|
|
|
|
```bash
|
|
# Always clean up in teardown
|
|
teardown() {
|
|
teardown_mocks # Restore original commands
|
|
unset MOCK_CLAUDE_SUCCESS
|
|
unset MOCK_CLAUDE_OUTPUT
|
|
}
|
|
```
|
|
|
|
### JSON Parsing Errors
|
|
|
|
#### Invalid JSON in Fixtures
|
|
|
|
```bash
|
|
# Validate fixture output
|
|
@test "debug json" {
|
|
create_sample_status_running "status.json"
|
|
|
|
# Validate JSON is valid
|
|
run jq empty "status.json"
|
|
assert_success
|
|
|
|
# Show content if invalid
|
|
if [[ $status -ne 0 ]]; then
|
|
cat "status.json" >&3
|
|
fi
|
|
}
|
|
```
|
|
|
|
#### Missing jq
|
|
|
|
```bash
|
|
# Check jq is available
|
|
which jq || echo "jq not installed"
|
|
|
|
# Install if missing
|
|
# Ubuntu/Debian
|
|
sudo apt-get install jq
|
|
|
|
# macOS
|
|
brew install jq
|
|
```
|
|
|
|
### File Permission Errors
|
|
|
|
#### Temp Directory Issues
|
|
|
|
```bash
|
|
# Ensure temp dir is writable
|
|
setup() {
|
|
export TEST_TEMP_DIR="$(mktemp -d)"
|
|
[[ -w "$TEST_TEMP_DIR" ]] || fail "Cannot write to temp dir"
|
|
}
|
|
```
|
|
|
|
#### Read-Only Filesystem
|
|
|
|
```bash
|
|
# Use system temp location
|
|
export BATS_TEST_TMPDIR="${TMPDIR:-/tmp}/bats-ralph-$$"
|
|
```
|
|
|
|
### CI/CD Failures
|
|
|
|
#### Tests Pass Locally, Fail in CI
|
|
|
|
1. **Check environment differences**:
|
|
```bash
|
|
# CI uses ubuntu-latest
|
|
uname -a
|
|
bash --version
|
|
```
|
|
|
|
2. **Check for hardcoded paths**:
|
|
```bash
|
|
# Bad: hardcoded path
|
|
source "/home/user/ralph/lib/utils.sh"
|
|
|
|
# Good: relative path
|
|
source "$(dirname "$BATS_TEST_FILENAME")/../../lib/utils.sh"
|
|
```
|
|
|
|
3. **Check for timing issues**:
|
|
```bash
|
|
# Add explicit waits if needed
|
|
sleep 1
|
|
```
|
|
|
|
#### Coverage Threshold Failures
|
|
|
|
```bash
|
|
# Check current threshold
|
|
grep COVERAGE_THRESHOLD .github/workflows/test.yml
|
|
|
|
# Threshold is set to 0 (disabled)
|
|
# If enabled, review coverage report
|
|
```
|
|
|
|
### Getting Help
|
|
|
|
1. **Check existing tests**: Look at similar tests in the suite for patterns
|
|
2. **BATS documentation**: https://bats-core.readthedocs.io/
|
|
3. **GitHub Issues**: Report test infrastructure issues at https://github.com/frankbria/ralph-claude-code/issues
|
|
|
|
---
|
|
|
|
## Appendices
|
|
|
|
### Appendix A: BATS Quick Reference
|
|
|
|
```bash
|
|
# Test file header
|
|
#!/usr/bin/env bats
|
|
load '../helpers/test_helper'
|
|
|
|
# Lifecycle hooks
|
|
setup() { } # Before each test
|
|
teardown() { } # After each test
|
|
setup_file() { } # Before all tests in file
|
|
teardown_file() { } # After all tests in file
|
|
|
|
# Test definition
|
|
@test "description" {
|
|
# Arrange, Act, Assert
|
|
}
|
|
|
|
# The run command
|
|
run command arg1 arg2
|
|
# Sets: $status (exit code), $output (stdout+stderr)
|
|
|
|
# Skip tests
|
|
@test "skipped test" {
|
|
skip "reason for skipping"
|
|
}
|
|
|
|
# Conditional skip
|
|
@test "conditional skip" {
|
|
[[ -z "$CI" ]] || skip "Only runs locally"
|
|
}
|
|
```
|
|
|
|
### Appendix B: Common Patterns
|
|
|
|
#### Testing Exit Codes
|
|
|
|
```bash
|
|
@test "command succeeds" {
|
|
run my_command
|
|
assert_success
|
|
}
|
|
|
|
@test "command fails with specific code" {
|
|
run my_command --invalid
|
|
[[ $status -eq 2 ]] # Specific exit code
|
|
}
|
|
```
|
|
|
|
#### Testing Output Content
|
|
|
|
```bash
|
|
@test "output contains expected text" {
|
|
run my_command
|
|
[[ "$output" == *"expected"* ]]
|
|
}
|
|
|
|
@test "output matches regex" {
|
|
run my_command
|
|
[[ "$output" =~ ^[0-9]+$ ]] # Matches digits
|
|
}
|
|
```
|
|
|
|
#### Testing File Creation
|
|
|
|
```bash
|
|
@test "command creates file" {
|
|
run my_command
|
|
assert_file_exists "output.txt"
|
|
}
|
|
|
|
@test "file contains expected content" {
|
|
run my_command
|
|
[[ "$(cat output.txt)" == "expected content" ]]
|
|
}
|
|
```
|
|
|
|
#### Testing JSON Output
|
|
|
|
```bash
|
|
@test "produces valid JSON" {
|
|
run my_command
|
|
echo "$output" | jq empty # Validates JSON
|
|
}
|
|
|
|
@test "JSON has expected field" {
|
|
run my_command
|
|
value=$(echo "$output" | jq -r '.status')
|
|
[[ "$value" == "success" ]]
|
|
}
|
|
```
|
|
|
|
### Appendix C: Contributing Tests
|
|
|
|
#### Adding New Test Files
|
|
|
|
1. Create file in appropriate directory:
|
|
```bash
|
|
touch tests/unit/test_my_feature.bats
|
|
chmod +x tests/unit/test_my_feature.bats
|
|
```
|
|
|
|
2. Use standard header:
|
|
```bash
|
|
#!/usr/bin/env bats
|
|
# Unit tests for my feature
|
|
|
|
load '../helpers/test_helper'
|
|
```
|
|
|
|
3. Verify tests run:
|
|
```bash
|
|
bats tests/unit/test_my_feature.bats
|
|
```
|
|
|
|
4. Update documentation if needed
|
|
|
|
#### Test Review Checklist
|
|
|
|
- [ ] Tests have descriptive names
|
|
- [ ] Each test verifies one behavior
|
|
- [ ] Tests clean up after themselves
|
|
- [ ] Mocks are properly set up and torn down
|
|
- [ ] No hardcoded paths
|
|
- [ ] Tests pass in isolation
|
|
- [ ] Tests pass in CI environment
|