iomgaa
6bdb802f01
- Add all claude skills (brainstorming, commit, debugging, TDD, etc.) - Add claude hooks (pre-commit-guard, post-edit-quality) - Add research templates (experiment plan, research brief, etc.) - Add claude tools (arxiv/semantic_scholar/openalex fetch, wiki, exa) - Add TRM4 reference implementation as algorithm fidelity baseline - Add research-wiki content (plans, index, graph, query_pack) - Update .gitignore to exclude .graphify_version runtime state
343 lines
12 KiB
Markdown
343 lines
12 KiB
Markdown
---
|
||
name: systematic-debugging
|
||
description: Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes
|
||
---
|
||
|
||
# Systematic Debugging
|
||
|
||
## Overview
|
||
|
||
Random fixes waste time and create new bugs. Quick patches mask underlying issues.
|
||
|
||
**Core principle:** ALWAYS find root cause before attempting fixes. Symptom fixes are failure.
|
||
|
||
**Violating the letter of this process is violating the spirit of debugging.**
|
||
|
||
## The Iron Law
|
||
|
||
```
|
||
NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST
|
||
```
|
||
|
||
If you haven't completed Phase 1, you cannot propose fixes.
|
||
|
||
## When to Use
|
||
|
||
Use for ANY technical issue:
|
||
- Test failures
|
||
- Bugs in production
|
||
- Unexpected behavior
|
||
- Performance problems
|
||
- Build failures
|
||
- Integration issues
|
||
|
||
**Use this ESPECIALLY when:**
|
||
- Under time pressure (emergencies make guessing tempting)
|
||
- "Just one quick fix" seems obvious
|
||
- You've already tried multiple fixes
|
||
- Previous fix didn't work
|
||
- You don't fully understand the issue
|
||
|
||
**Don't skip when:**
|
||
- Issue seems simple (simple bugs have root causes too)
|
||
- You're in a hurry (rushing guarantees rework)
|
||
- Manager wants it fixed NOW (systematic is faster than thrashing)
|
||
|
||
## The Five Phases
|
||
|
||
You MUST complete each phase before proceeding to the next.
|
||
|
||
### Phase 1: Root Cause Investigation
|
||
|
||
**BEFORE attempting ANY fix:**
|
||
|
||
1. **Read Error Messages Carefully**
|
||
- Don't skip past errors or warnings
|
||
- They often contain the exact solution
|
||
- Read stack traces completely
|
||
- Note line numbers, file paths, error codes
|
||
|
||
2. **Reproduce Consistently**
|
||
- Can you trigger it reliably?
|
||
- What are the exact steps?
|
||
- Does it happen every time?
|
||
- If not reproducible → gather more data, don't guess
|
||
|
||
3. **Check Recent Changes**
|
||
- What changed that could cause this?
|
||
- Git diff, recent commits
|
||
- New dependencies, config changes
|
||
- Environmental differences
|
||
|
||
**代码检索（graphify）**：定位前先 `/graphify . --update` 刷新图谱，再用 graphify 追调用关系与影响面，而非盲读文件：
|
||
- `conda run -n chs graphify path "<A>" "<B>"` 追依赖/调用链
|
||
- `conda run -n chs graphify affected "<出问题的节点>"` 反向影响面（改它会波及谁）
|
||
- `conda run -n chs graphify explain "<节点>"` 看某节点及邻居
|
||
- 图谱未建（`graph file not found`）则提示人类先 `/graphify .`，未建图前降级常规阅读、不阻断。
|
||
|
||
4. **Gather Evidence in Multi-Component Systems**
|
||
|
||
**WHEN system has multiple components (CI → build → signing, API → service → database):**
|
||
|
||
**BEFORE proposing fixes, add diagnostic instrumentation:**
|
||
```
|
||
For EACH component boundary:
|
||
- Log what data enters component
|
||
- Log what data exits component
|
||
- Verify environment/config propagation
|
||
- Check state at each layer
|
||
|
||
Run once to gather evidence showing WHERE it breaks
|
||
THEN analyze evidence to identify failing component
|
||
THEN investigate that specific component
|
||
```
|
||
|
||
**Example (multi-layer system):**
|
||
```bash
|
||
# Layer 1: Workflow
|
||
echo "=== Secrets available in workflow: ==="
|
||
echo "IDENTITY: ${IDENTITY:+SET}${IDENTITY:-UNSET}"
|
||
|
||
# Layer 2: Build script
|
||
echo "=== Env vars in build script: ==="
|
||
env | grep IDENTITY || echo "IDENTITY not in environment"
|
||
|
||
# Layer 3: Signing script
|
||
echo "=== Keychain state: ==="
|
||
security list-keychains
|
||
security find-identity -v
|
||
|
||
# Layer 4: Actual signing
|
||
codesign --sign "$IDENTITY" --verbose=4 "$APP"
|
||
```
|
||
|
||
**This reveals:** Which layer fails (secrets → workflow ✓, workflow → build ✗)
|
||
|
||
5. **Trace Data Flow**
|
||
|
||
**WHEN error is deep in call stack:**
|
||
|
||
See `root-cause-tracing.md` in this directory for the complete backward tracing technique.
|
||
|
||
**Quick version:**
|
||
- Where does bad value originate?
|
||
- What called this with bad value?
|
||
- Keep tracing up until you find the source
|
||
- Fix at source, not at symptom
|
||
|
||
### Phase 2: Log Diagnosis
|
||
|
||
**Query SqliteRunStore for runtime evidence:**
|
||
|
||
1. **Read Wiki Schema Directory**
|
||
- Check `research-wiki/schemas/` for all log tables in this project
|
||
- Identify which tables are relevant to the current bug
|
||
|
||
2. **Query Event Stream**
|
||
- Query `_events` table: what events occurred around the time of the bug
|
||
- `SELECT * FROM _events WHERE timestamp > '<bug_time>' ORDER BY timestamp LIMIT 20`
|
||
- Focus on event_type = error/warning events
|
||
|
||
3. **Query Custom Tables**
|
||
- Query relevant metrics tables: are trends abnormal?
|
||
- Compare data across recent runs: is there a regression?
|
||
|
||
4. **Output**
|
||
- Hypotheses backed by log data (not just code-reading guesses)
|
||
- Specific data evidence to support each hypothesis
|
||
|
||
### Phase 3: Pattern Analysis
|
||
|
||
**Find the pattern before fixing:**
|
||
|
||
1. **Find Working Examples**
|
||
- Locate similar working code in same codebase
|
||
- What works that's similar to what's broken?
|
||
|
||
2. **Compare Against References**
|
||
- If implementing pattern, read reference implementation COMPLETELY
|
||
- Don't skim - read every line
|
||
- Understand the pattern fully before applying
|
||
|
||
3. **Identify Differences**
|
||
- What's different between working and broken?
|
||
- List every difference, however small
|
||
- Don't assume "that can't matter"
|
||
|
||
4. **Understand Dependencies**
|
||
- What other components does this need?
|
||
- What settings, config, environment?
|
||
- What assumptions does it make?
|
||
|
||
### Phase 4: Hypothesis and Testing
|
||
|
||
**Scientific method:**
|
||
|
||
1. **Form Single Hypothesis**
|
||
- State clearly: "I think X is the root cause because Y"
|
||
- Write it down
|
||
- Be specific, not vague
|
||
|
||
2. **Test Minimally**
|
||
- Make the SMALLEST possible change to test hypothesis
|
||
- One variable at a time
|
||
- Don't fix multiple things at once
|
||
|
||
3. **Verify Before Continuing**
|
||
- Did it work? Yes → Phase 4
|
||
- Didn't work? Form NEW hypothesis
|
||
- DON'T add more fixes on top
|
||
|
||
4. **When You Don't Know**
|
||
- Say "I don't understand X"
|
||
- Don't pretend to know
|
||
- Ask for help
|
||
- Research more
|
||
|
||
### Phase 5: Implementation
|
||
|
||
**Fix the root cause, not the symptom:**
|
||
|
||
1. **Create Failing Test Case**
|
||
- Simplest possible reproduction
|
||
- Automated test if possible
|
||
- One-off test script if no framework
|
||
- MUST have before fixing
|
||
- Use the `test-driven-development` skill for writing proper failing tests
|
||
|
||
2. **Implement Single Fix**
|
||
- Address the root cause identified
|
||
- ONE change at a time
|
||
- No "while I'm here" improvements
|
||
- No bundled refactoring
|
||
|
||
3. **Verify Fix**
|
||
- Test passes now?
|
||
- No other tests broken?
|
||
- Issue actually resolved?
|
||
|
||
4. **If Fix Doesn't Work**
|
||
- STOP
|
||
- Count: How many fixes have you tried?
|
||
- If < 3: Return to Phase 1, re-analyze with new information
|
||
- **If ≥ 3: STOP and question the architecture (step 5 below)**
|
||
- DON'T attempt Fix #4 without architectural discussion
|
||
|
||
5. **If 3+ Fixes Failed: Question Architecture**
|
||
|
||
**Pattern indicating architectural problem:**
|
||
- Each fix reveals new shared state/coupling/problem in different place
|
||
- Fixes require "massive refactoring" to implement
|
||
- Each fix creates new symptoms elsewhere
|
||
|
||
**STOP and question fundamentals:**
|
||
- Is this pattern fundamentally sound?
|
||
- Are we "sticking with it through sheer inertia"?
|
||
- Should we refactor architecture vs. continue fixing symptoms?
|
||
|
||
**Discuss with your human partner before attempting more fixes**
|
||
|
||
This is NOT a failed hypothesis - this is a wrong architecture.
|
||
|
||
## Red Flags - STOP and Follow Process
|
||
|
||
If you catch yourself thinking:
|
||
- "Quick fix for now, investigate later"
|
||
- "Just try changing X and see if it works"
|
||
- "Add multiple changes, run tests"
|
||
- "Skip the test, I'll manually verify"
|
||
- "It's probably X, let me fix that"
|
||
- "I don't fully understand but this might work"
|
||
- "Pattern says X but I'll adapt it differently"
|
||
- "Here are the main problems: [lists fixes without investigation]"
|
||
- Proposing solutions before tracing data flow
|
||
- **"One more fix attempt" (when already tried 2+)**
|
||
- **Each fix reveals new problem in different place**
|
||
- Commenting out tests or skipping validation to make errors go away
|
||
- Patching symptoms without understanding root cause
|
||
|
||
**ALL of these mean: STOP. Return to Phase 1.**
|
||
|
||
**If 3+ fixes failed:** Question the architecture (see Phase 4.5)
|
||
|
||
## your human partner's Signals You're Doing It Wrong
|
||
|
||
**Watch for these redirections:**
|
||
- "Is that not happening?" - You assumed without verifying
|
||
- "Will it show us...?" - You should have added evidence gathering
|
||
- "Stop guessing" - You're proposing fixes without understanding
|
||
- "Ultrathink this" - Question fundamentals, not just symptoms
|
||
- "We're stuck?" (frustrated) - Your approach isn't working
|
||
|
||
**When you see these:** STOP. Return to Phase 1.
|
||
|
||
## Common Rationalizations
|
||
|
||
| Excuse | Reality |
|
||
|--------|---------|
|
||
| "Issue is simple, don't need process" | Simple issues have root causes too. Process is fast for simple bugs. |
|
||
| "Emergency, no time for process" | Systematic debugging is FASTER than guess-and-check thrashing. |
|
||
| "Just try this first, then investigate" | First fix sets the pattern. Do it right from the start. |
|
||
| "I'll write test after confirming fix works" | Untested fixes don't stick. Test first proves it. |
|
||
| "Multiple fixes at once saves time" | Can't isolate what worked. Causes new bugs. |
|
||
| "Reference too long, I'll adapt the pattern" | Partial understanding guarantees bugs. Read it completely. |
|
||
| "I see the problem, let me fix it" | Seeing symptoms ≠ understanding root cause. |
|
||
| "One more fix attempt" (after 2+ failures) | 3+ failures = architectural problem. Question pattern, don't fix again. |
|
||
|
||
## Quick Reference
|
||
|
||
| Phase | Key Activities | Success Criteria |
|
||
|-------|---------------|------------------|
|
||
| **1. Root Cause** | Read errors, reproduce, check changes, gather evidence | Understand WHAT and WHY |
|
||
| **2. Log Diagnosis** | Query Wiki schemas, _events, custom tables | Data-backed hypotheses |
|
||
| **3. Pattern** | Find working examples, compare | Identify differences |
|
||
| **4. Hypothesis** | Form theory, test minimally | Confirmed or new hypothesis |
|
||
| **5. Implementation** | Create test, fix, verify | Bug resolved, tests pass |
|
||
|
||
## When Process Reveals "No Root Cause"
|
||
|
||
If systematic investigation reveals issue is truly environmental, timing-dependent, or external:
|
||
|
||
1. You've completed the process
|
||
2. Document what you investigated
|
||
3. Implement appropriate handling (retry, timeout, error message)
|
||
4. Add monitoring/logging for future investigation
|
||
|
||
**But:** 95% of "no root cause" cases are incomplete investigation.
|
||
|
||
## Supporting Techniques
|
||
|
||
These techniques are part of systematic debugging and available in this directory:
|
||
|
||
- **`root-cause-tracing.md`** - Trace bugs backward through call stack to find original trigger
|
||
- **`defense-in-depth.md`** - Add validation at multiple layers after finding root cause
|
||
- **`condition-based-waiting.md`** - Replace arbitrary timeouts with condition polling
|
||
|
||
**Related skills:**
|
||
- **test-driven-development** - For creating failing test case (Phase 4, Step 1)
|
||
- **verification-before-completion** - Verify fix worked before claiming success
|
||
|
||
## Real-World Impact
|
||
|
||
From debugging sessions:
|
||
- Systematic approach: 15-30 minutes to fix
|
||
- Random fixes approach: 2-3 hours of thrashing
|
||
- First-time fix rate: 95% vs 40%
|
||
- New bugs introduced: Near zero vs common
|
||
|
||
## Wiki Integration
|
||
|
||
**Precondition**: `research-wiki/` directory exists (skip this section entirely if it does not).
|
||
|
||
**Trigger**: When you reach an actionable bug conclusion, especially one worth preserving for later tasks.
|
||
|
||
**Type**: `debugging`
|
||
|
||
**Severity**: `critical` / `important` / `minor`
|
||
|
||
**Steps**:
|
||
1. Run `.claude/tools/research_wiki.py add_entity research-wiki/ --type finding --id <slug> --title "<issue title>"` to create the finding entity
|
||
2. Append the symptoms, root cause, verification method, fix, and impact scope to the generated page
|
||
3. If the finding reveals a design, plan, or implementation defect, run `.claude/tools/research_wiki.py add_edge research-wiki/ --from "finding:<id>" --to "<target-type>:<id>" --type reveals --evidence "..."`
|
||
4. Run `.claude/tools/research_wiki.py rebuild_index research-wiki/`
|