v1.4.0: Major Skills Expansion - 75 Total Skills
This commit is contained in:
admin
303
skills/superpowers/docs/testing.md
Normal file
303
skills/superpowers/docs/testing.md
Normal file
@@ -0,0 +1,303 @@
|
||||
# Testing Superpowers Skills
|
||||
|
||||
This document describes how to test Superpowers skills, particularly the integration tests for complex skills like `subagent-driven-development`.
|
||||
|
||||
## Overview
|
||||
|
||||
Testing skills that involve subagents, workflows, and complex interactions requires running actual Claude Code sessions in headless mode and verifying their behavior through session transcripts.
|
||||
|
||||
## Test Structure
|
||||
|
||||
```
|
||||
tests/
|
||||
├── claude-code/
|
||||
│ ├── test-helpers.sh # Shared test utilities
|
||||
│ ├── test-subagent-driven-development-integration.sh
|
||||
│ ├── analyze-token-usage.py # Token analysis tool
|
||||
│ └── run-skill-tests.sh # Test runner (if exists)
|
||||
```
|
||||
|
||||
## Running Tests
|
||||
|
||||
### Integration Tests
|
||||
|
||||
Integration tests execute real Claude Code sessions with actual skills:
|
||||
|
||||
```bash
|
||||
# Run the subagent-driven-development integration test
|
||||
cd tests/claude-code
|
||||
./test-subagent-driven-development-integration.sh
|
||||
```
|
||||
|
||||
**Note:** Integration tests can take 10-30 minutes as they execute real implementation plans with multiple subagents.
|
||||
|
||||
### Requirements
|
||||
|
||||
- Must run from the **superpowers plugin directory** (not from temp directories)
|
||||
- Claude Code must be installed and available as `claude` command
|
||||
- Local dev marketplace must be enabled: `"superpowers@superpowers-dev": true` in `~/.claude/settings.json`
|
||||
|
||||
## Integration Test: subagent-driven-development
|
||||
|
||||
### What It Tests
|
||||
|
||||
The integration test verifies the `subagent-driven-development` skill correctly:
|
||||
|
||||
1. **Plan Loading**: Reads the plan once at the beginning
|
||||
2. **Full Task Text**: Provides complete task descriptions to subagents (doesn't make them read files)
|
||||
3. **Self-Review**: Ensures subagents perform self-review before reporting
|
||||
4. **Review Order**: Runs spec compliance review before code quality review
|
||||
5. **Review Loops**: Uses review loops when issues are found
|
||||
6. **Independent Verification**: Spec reviewer reads code independently, doesn't trust implementer reports
|
||||
|
||||
### How It Works
|
||||
|
||||
1. **Setup**: Creates a temporary Node.js project with a minimal implementation plan
|
||||
2. **Execution**: Runs Claude Code in headless mode with the skill
|
||||
3. **Verification**: Parses the session transcript (`.jsonl` file) to verify:
|
||||
- Skill tool was invoked
|
||||
- Subagents were dispatched (Task tool)
|
||||
- TodoWrite was used for tracking
|
||||
- Implementation files were created
|
||||
- Tests pass
|
||||
- Git commits show proper workflow
|
||||
4. **Token Analysis**: Shows token usage breakdown by subagent
|
||||
|
||||
### Test Output
|
||||
|
||||
```
|
||||
========================================
|
||||
Integration Test: subagent-driven-development
|
||||
========================================
|
||||
|
||||
Test project: /tmp/tmp.xyz123
|
||||
|
||||
=== Verification Tests ===
|
||||
|
||||
Test 1: Skill tool invoked...
|
||||
[PASS] subagent-driven-development skill was invoked
|
||||
|
||||
Test 2: Subagents dispatched...
|
||||
[PASS] 7 subagents dispatched
|
||||
|
||||
Test 3: Task tracking...
|
||||
[PASS] TodoWrite used 5 time(s)
|
||||
|
||||
Test 6: Implementation verification...
|
||||
[PASS] src/math.js created
|
||||
[PASS] add function exists
|
||||
[PASS] multiply function exists
|
||||
[PASS] test/math.test.js created
|
||||
[PASS] Tests pass
|
||||
|
||||
Test 7: Git commit history...
|
||||
[PASS] Multiple commits created (3 total)
|
||||
|
||||
Test 8: No extra features added...
|
||||
[PASS] No extra features added
|
||||
|
||||
=========================================
|
||||
Token Usage Analysis
|
||||
=========================================
|
||||
|
||||
Usage Breakdown:
|
||||
----------------------------------------------------------------------------------------------------
|
||||
Agent Description Msgs Input Output Cache Cost
|
||||
----------------------------------------------------------------------------------------------------
|
||||
main Main session (coordinator) 34 27 3,996 1,213,703 $ 4.09
|
||||
3380c209 implementing Task 1: Create Add Function 1 2 787 24,989 $ 0.09
|
||||
34b00fde implementing Task 2: Create Multiply Function 1 4 644 25,114 $ 0.09
|
||||
3801a732 reviewing whether an implementation matches... 1 5 703 25,742 $ 0.09
|
||||
4c142934 doing a final code review... 1 6 854 25,319 $ 0.09
|
||||
5f017a42 a code reviewer. Review Task 2... 1 6 504 22,949 $ 0.08
|
||||
a6b7fbe4 a code reviewer. Review Task 1... 1 6 515 22,534 $ 0.08
|
||||
f15837c0 reviewing whether an implementation matches... 1 6 416 22,485 $ 0.07
|
||||
----------------------------------------------------------------------------------------------------
|
||||
|
||||
TOTALS:
|
||||
Total messages: 41
|
||||
Input tokens: 62
|
||||
Output tokens: 8,419
|
||||
Cache creation tokens: 132,742
|
||||
Cache read tokens: 1,382,835
|
||||
|
||||
Total input (incl cache): 1,515,639
|
||||
Total tokens: 1,524,058
|
||||
|
||||
Estimated cost: $4.67
|
||||
(at $3/$15 per M tokens for input/output)
|
||||
|
||||
========================================
|
||||
Test Summary
|
||||
========================================
|
||||
|
||||
STATUS: PASSED
|
||||
```
|
||||
|
||||
## Token Analysis Tool
|
||||
|
||||
### Usage
|
||||
|
||||
Analyze token usage from any Claude Code session:
|
||||
|
||||
```bash
|
||||
python3 tests/claude-code/analyze-token-usage.py ~/.claude/projects/<project-dir>/<session-id>.jsonl
|
||||
```
|
||||
|
||||
### Finding Session Files
|
||||
|
||||
Session transcripts are stored in `~/.claude/projects/` with the working directory path encoded:
|
||||
|
||||
```bash
|
||||
# Example for /Users/jesse/Documents/GitHub/superpowers/superpowers
|
||||
SESSION_DIR="$HOME/.claude/projects/-Users-jesse-Documents-GitHub-superpowers-superpowers"
|
||||
|
||||
# Find recent sessions
|
||||
ls -lt "$SESSION_DIR"/*.jsonl | head -5
|
||||
```
|
||||
|
||||
### What It Shows
|
||||
|
||||
- **Main session usage**: Token usage by the coordinator (you or main Claude instance)
|
||||
- **Per-subagent breakdown**: Each Task invocation with:
|
||||
- Agent ID
|
||||
- Description (extracted from prompt)
|
||||
- Message count
|
||||
- Input/output tokens
|
||||
- Cache usage
|
||||
- Estimated cost
|
||||
- **Totals**: Overall token usage and cost estimate
|
||||
|
||||
### Understanding the Output
|
||||
|
||||
- **High cache reads**: Good - means prompt caching is working
|
||||
- **High input tokens on main**: Expected - coordinator has full context
|
||||
- **Similar costs per subagent**: Expected - each gets similar task complexity
|
||||
- **Cost per task**: Typical range is $0.05-$0.15 per subagent depending on task
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Skills Not Loading
|
||||
|
||||
**Problem**: Skill not found when running headless tests
|
||||
|
||||
**Solutions**:
|
||||
1. Ensure you're running FROM the superpowers directory: `cd /path/to/superpowers && tests/...`
|
||||
2. Check `~/.claude/settings.json` has `"superpowers@superpowers-dev": true` in `enabledPlugins`
|
||||
3. Verify skill exists in `skills/` directory
|
||||
|
||||
### Permission Errors
|
||||
|
||||
**Problem**: Claude blocked from writing files or accessing directories
|
||||
|
||||
**Solutions**:
|
||||
1. Use `--permission-mode bypassPermissions` flag
|
||||
2. Use `--add-dir /path/to/temp/dir` to grant access to test directories
|
||||
3. Check file permissions on test directories
|
||||
|
||||
### Test Timeouts
|
||||
|
||||
**Problem**: Test takes too long and times out
|
||||
|
||||
**Solutions**:
|
||||
1. Increase timeout: `timeout 1800 claude ...` (30 minutes)
|
||||
2. Check for infinite loops in skill logic
|
||||
3. Review subagent task complexity
|
||||
|
||||
### Session File Not Found
|
||||
|
||||
**Problem**: Can't find session transcript after test run
|
||||
|
||||
**Solutions**:
|
||||
1. Check the correct project directory in `~/.claude/projects/`
|
||||
2. Use `find ~/.claude/projects -name "*.jsonl" -mmin -60` to find recent sessions
|
||||
3. Verify test actually ran (check for errors in test output)
|
||||
|
||||
## Writing New Integration Tests
|
||||
|
||||
### Template
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
source "$SCRIPT_DIR/test-helpers.sh"
|
||||
|
||||
# Create test project
|
||||
TEST_PROJECT=$(create_test_project)
|
||||
trap "cleanup_test_project $TEST_PROJECT" EXIT
|
||||
|
||||
# Set up test files...
|
||||
cd "$TEST_PROJECT"
|
||||
|
||||
# Run Claude with skill
|
||||
PROMPT="Your test prompt here"
|
||||
cd "$SCRIPT_DIR/../.." && timeout 1800 claude -p "$PROMPT" \
|
||||
--allowed-tools=all \
|
||||
--add-dir "$TEST_PROJECT" \
|
||||
--permission-mode bypassPermissions \
|
||||
2>&1 | tee output.txt
|
||||
|
||||
# Find and analyze session
|
||||
WORKING_DIR_ESCAPED=$(echo "$SCRIPT_DIR/../.." | sed 's/\\//-/g' | sed 's/^-//')
|
||||
SESSION_DIR="$HOME/.claude/projects/$WORKING_DIR_ESCAPED"
|
||||
SESSION_FILE=$(find "$SESSION_DIR" -name "*.jsonl" -type f -mmin -60 | sort -r | head -1)
|
||||
|
||||
# Verify behavior by parsing session transcript
|
||||
if grep -q '"name":"Skill".*"skill":"your-skill-name"' "$SESSION_FILE"; then
|
||||
echo "[PASS] Skill was invoked"
|
||||
fi
|
||||
|
||||
# Show token analysis
|
||||
python3 "$SCRIPT_DIR/analyze-token-usage.py" "$SESSION_FILE"
|
||||
```
|
||||
|
||||
### Best Practices
|
||||
|
||||
1. **Always cleanup**: Use trap to cleanup temp directories
|
||||
2. **Parse transcripts**: Don't grep user-facing output - parse the `.jsonl` session file
|
||||
3. **Grant permissions**: Use `--permission-mode bypassPermissions` and `--add-dir`
|
||||
4. **Run from plugin dir**: Skills only load when running from the superpowers directory
|
||||
5. **Show token usage**: Always include token analysis for cost visibility
|
||||
6. **Test real behavior**: Verify actual files created, tests passing, commits made
|
||||
|
||||
## Session Transcript Format
|
||||
|
||||
Session transcripts are JSONL (JSON Lines) files where each line is a JSON object representing a message or tool result.
|
||||
|
||||
### Key Fields
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "assistant",
|
||||
"message": {
|
||||
"content": [...],
|
||||
"usage": {
|
||||
"input_tokens": 27,
|
||||
"output_tokens": 3996,
|
||||
"cache_read_input_tokens": 1213703
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Tool Results
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "user",
|
||||
"toolUseResult": {
|
||||
"agentId": "3380c209",
|
||||
"usage": {
|
||||
"input_tokens": 2,
|
||||
"output_tokens": 787,
|
||||
"cache_read_input_tokens": 24989
|
||||
},
|
||||
"prompt": "You are implementing Task 1...",
|
||||
"content": [{"type": "text", "text": "..."}]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The `agentId` field links to subagent sessions, and the `usage` field contains token usage for that specific subagent invocation.
|
||||
Reference in New Issue
Block a user