chore: track claude skills, tools, templates, reference code and research-wiki
- Add all claude skills (brainstorming, commit, debugging, TDD, etc.) - Add claude hooks (pre-commit-guard, post-edit-quality) - Add research templates (experiment plan, research brief, etc.) - Add claude tools (arxiv/semantic_scholar/openalex fetch, wiki, exa) - Add TRM4 reference implementation as algorithm fidelity baseline - Add research-wiki content (plans, index, graph, query_pack) - Update .gitignore to exclude .graphify_version runtime state
This commit is contained in:
@@ -0,0 +1,189 @@
|
||||
---
|
||||
name: brainstorming
|
||||
description: "You MUST use this before any creative work - creating features, building components, adding functionality, or modifying behavior. Explores user intent, requirements and design before implementation."
|
||||
---
|
||||
|
||||
# Brainstorming Ideas Into Designs
|
||||
|
||||
Help turn ideas into fully formed designs and specs through natural collaborative dialogue.
|
||||
|
||||
Start by understanding the current project context, then ask questions one at a time to refine the idea. Once you understand what you're building, present the design and get user approval.
|
||||
|
||||
<HARD-GATE>
|
||||
Do NOT invoke any implementation skill, write any code, scaffold any project, or take any implementation action until you have presented a design and the user has approved it. This applies to EVERY project regardless of perceived simplicity.
|
||||
</HARD-GATE>
|
||||
|
||||
## Anti-Pattern: "This Is Too Simple To Need A Design"
|
||||
|
||||
Every project goes through this process. A todo list, a single-function utility, a config change — all of them. "Simple" projects are where unexamined assumptions cause the most wasted work. The design can be short (a few sentences for truly simple projects), but you MUST present it and get approval.
|
||||
|
||||
## Checklist
|
||||
|
||||
You MUST create a task for each of these items and complete them in order:
|
||||
|
||||
1. **Explore project context** — check files, docs, recent commits
|
||||
2. **Offer visual companion** (if topic will involve visual questions) — this is its own message, not combined with a clarifying question. See the Visual Companion section below.
|
||||
3. **Ask clarifying questions** — one at a time, understand purpose/constraints/success criteria
|
||||
4. **Propose 2-3 approaches** — with trade-offs and your recommendation
|
||||
5. **Present design** — in sections scaled to their complexity, get user approval after each section
|
||||
6. **Write design doc** — save to `research-wiki/designs/YYYY-MM-DD-<topic>-design.md` and commit
|
||||
7. **Spec self-review** — quick inline check for placeholders, contradictions, ambiguity, scope (see below)
|
||||
7.5. **Codex design review** — independent Codex review via `/codex:rescue --fresh --wait`, verify findings, apply accepted fixes inline (see below)
|
||||
8. **User reviews written spec** — ask user to review the spec file before proceeding
|
||||
9. **Transition to implementation** — invoke writing-plans skill to create implementation plan
|
||||
|
||||
## Process Flow
|
||||
|
||||
```dot
|
||||
digraph brainstorming {
|
||||
"Explore project context" [shape=box];
|
||||
"Visual questions ahead?" [shape=diamond];
|
||||
"Offer Visual Companion\n(own message, no other content)" [shape=box];
|
||||
"Ask clarifying questions" [shape=box];
|
||||
"Propose 2-3 approaches" [shape=box];
|
||||
"Present design sections" [shape=box];
|
||||
"User approves design?" [shape=diamond];
|
||||
"Write design doc" [shape=box];
|
||||
"Spec self-review\n(fix inline)" [shape=box];
|
||||
"Codex design review\n(/codex:rescue --fresh --wait, fix inline)" [shape=box];
|
||||
"User reviews spec?" [shape=diamond];
|
||||
"Invoke writing-plans skill" [shape=doublecircle];
|
||||
|
||||
"Explore project context" -> "Visual questions ahead?";
|
||||
"Visual questions ahead?" -> "Offer Visual Companion\n(own message, no other content)" [label="yes"];
|
||||
"Visual questions ahead?" -> "Ask clarifying questions" [label="no"];
|
||||
"Offer Visual Companion\n(own message, no other content)" -> "Ask clarifying questions";
|
||||
"Ask clarifying questions" -> "Propose 2-3 approaches";
|
||||
"Propose 2-3 approaches" -> "Present design sections";
|
||||
"Present design sections" -> "User approves design?";
|
||||
"User approves design?" -> "Present design sections" [label="no, revise"];
|
||||
"User approves design?" -> "Write design doc" [label="yes"];
|
||||
"Write design doc" -> "Spec self-review\n(fix inline)";
|
||||
"Spec self-review\n(fix inline)" -> "Codex design review\n(/codex:rescue --fresh --wait, fix inline)";
|
||||
"Codex design review\n(/codex:rescue --fresh --wait, fix inline)" -> "User reviews spec?";
|
||||
"User reviews spec?" -> "Write design doc" [label="changes requested"];
|
||||
"User reviews spec?" -> "Invoke writing-plans skill" [label="approved"];
|
||||
}
|
||||
```
|
||||
|
||||
**The terminal state is invoking writing-plans.** Do NOT invoke frontend-design, mcp-builder, or any other implementation skill. The ONLY skill you invoke after brainstorming is writing-plans.
|
||||
|
||||
## The Process
|
||||
|
||||
**Understanding the idea:**
|
||||
|
||||
- Check out the current project state first (files, docs, recent commits)
|
||||
- **代码检索（graphify）**：在查阅现有代码结构前，先跑 `/graphify . --update` 刷新知识图谱（增量合并：改代码免费、改文档才 LLM），再优先用 `/graphify query "<问题>"` / `graphify path` / `graphify explain` 做结构化检索，而非盲读文件。若 `graphify-out/graph.json` 不存在，提示人类先手动 `/graphify .`；未建图前不阻断，降级为常规文件阅读。
|
||||
- Before asking detailed questions, assess scope: if the request describes multiple independent subsystems (e.g., "build a platform with chat, file storage, billing, and analytics"), flag this immediately. Don't spend questions refining details of a project that needs to be decomposed first.
|
||||
- If the project is too large for a single spec, help the user decompose into sub-projects: what are the independent pieces, how do they relate, what order should they be built? Then brainstorm the first sub-project through the normal design flow. Each sub-project gets its own spec → plan → implementation cycle.
|
||||
- For appropriately-scoped projects, ask questions one at a time to refine the idea
|
||||
- Prefer multiple choice questions when possible, but open-ended is fine too
|
||||
- Only one question per message - if a topic needs more exploration, break it into multiple questions
|
||||
- Focus on understanding: purpose, constraints, success criteria
|
||||
|
||||
**Exploring approaches:**
|
||||
|
||||
- Propose 2-3 different approaches with trade-offs
|
||||
- Present options conversationally with your recommendation and reasoning
|
||||
- Lead with your recommended option and explain why
|
||||
|
||||
**Presenting the design:**
|
||||
|
||||
- Once you believe you understand what you're building, present the design
|
||||
- Scale each section to its complexity: a few sentences if straightforward, up to 200-300 words if nuanced
|
||||
- Ask after each section whether it looks right so far
|
||||
- Cover: architecture, components, data flow, error handling, testing
|
||||
- Be ready to go back and clarify if something doesn't make sense
|
||||
|
||||
**Design for isolation and clarity:**
|
||||
|
||||
- Break the system into smaller units that each have one clear purpose, communicate through well-defined interfaces, and can be understood and tested independently
|
||||
- For each unit, you should be able to answer: what does it do, how do you use it, and what does it depend on?
|
||||
- Can someone understand what a unit does without reading its internals? Can you change the internals without breaking consumers? If not, the boundaries need work.
|
||||
- Smaller, well-bounded units are also easier for you to work with - you reason better about code you can hold in context at once, and your edits are more reliable when files are focused. When a file grows large, that's often a signal that it's doing too much.
|
||||
|
||||
**Working in existing codebases:**
|
||||
|
||||
- Explore the current structure before proposing changes. Follow existing patterns.
|
||||
- Where existing code has problems that affect the work (e.g., a file that's grown too large, unclear boundaries, tangled responsibilities), include targeted improvements as part of the design - the way a good developer improves code they're working in.
|
||||
- Don't propose unrelated refactoring. Stay focused on what serves the current goal.
|
||||
|
||||
## After the Design
|
||||
|
||||
**Documentation:**
|
||||
|
||||
- Write the validated design (spec) to `research-wiki/designs/YYYY-MM-DD-<topic>-design.md`
|
||||
- (User preferences for spec location override this default)
|
||||
- Use elements-of-style:writing-clearly-and-concisely skill if available
|
||||
- Commit the design document to git
|
||||
|
||||
**Spec Self-Review:**
|
||||
After writing the spec document, look at it with fresh eyes:
|
||||
|
||||
1. **Placeholder scan:** Any "TBD", "TODO", incomplete sections, or vague requirements? Fix them.
|
||||
2. **Internal consistency:** Do any sections contradict each other? Does the architecture match the feature descriptions?
|
||||
3. **Scope check:** Is this focused enough for a single implementation plan, or does it need decomposition?
|
||||
4. **Ambiguity check:** Could any requirement be interpreted two different ways? If so, pick one and make it explicit.
|
||||
|
||||
Fix any issues inline. No need to re-review — just fix and move on.
|
||||
|
||||
**Codex Design Review:**
|
||||
After your self-review passes, get an independent second opinion from Codex before the human gate:
|
||||
|
||||
> 用 `/codex:rescue --fresh --wait` 把"用户需求溯源 + 设计全文"交 Codex 独立审查（只读），要求逐条核对需求覆盖、内部一致性、与 CLAUDE.md/既有设计的冲突，按 Critical/Important/Minor 返回必改项。
|
||||
|
||||
Verify each finding against the codebase (do not blindly accept). Apply the accepted fixes inline, then proceed to the User Review Gate. The human gate is preserved.
|
||||
|
||||
**User Review Gate:**
|
||||
After the spec review loop passes, ask the user to review the written spec before proceeding:
|
||||
|
||||
> "Spec written and committed to `<path>`. Please review it and let me know if you want to make any changes before we start writing out the implementation plan."
|
||||
|
||||
Wait for the user's response. If they request changes, make them and re-run the spec review loop. Only proceed once the user approves.
|
||||
|
||||
**Implementation:**
|
||||
|
||||
- Invoke the writing-plans skill to create a detailed implementation plan
|
||||
- Do NOT invoke any other skill. writing-plans is the next step.
|
||||
|
||||
## Key Principles
|
||||
|
||||
- **One question at a time** - Don't overwhelm with multiple questions
|
||||
- **Multiple choice preferred** - Easier to answer than open-ended when possible
|
||||
- **YAGNI ruthlessly** - Remove unnecessary features from all designs
|
||||
- **Explore alternatives** - Always propose 2-3 approaches before settling
|
||||
- **Incremental validation** - Present design, get approval before moving on
|
||||
- **Be flexible** - Go back and clarify when something doesn't make sense
|
||||
|
||||
## Visual Companion
|
||||
|
||||
A browser-based companion for showing mockups, diagrams, and visual options during brainstorming. Available as a tool — not a mode. Accepting the companion means it's available for questions that benefit from visual treatment; it does NOT mean every question goes through the browser.
|
||||
|
||||
**Offering the companion:** When you anticipate that upcoming questions will involve visual content (mockups, layouts, diagrams), offer it once for consent:
|
||||
> "Some of what we're working on might be easier to explain if I can show it to you in a web browser. I can put together mockups, diagrams, comparisons, and other visuals as we go. This feature is still new and can be token-intensive. Want to try it? (Requires opening a local URL)"
|
||||
|
||||
**This offer MUST be its own message.** Do not combine it with clarifying questions, context summaries, or any other content. The message should contain ONLY the offer above and nothing else. Wait for the user's response before continuing. If they decline, proceed with text-only brainstorming.
|
||||
|
||||
**Per-question decision:** Even after the user accepts, decide FOR EACH QUESTION whether to use the browser or the terminal. The test: **would the user understand this better by seeing it than reading it?**
|
||||
|
||||
- **Use the browser** for content that IS visual — mockups, wireframes, layout comparisons, architecture diagrams, side-by-side visual designs
|
||||
- **Use the terminal** for content that is text — requirements questions, conceptual choices, tradeoff lists, A/B/C/D text options, scope decisions
|
||||
|
||||
A question about a UI topic is not automatically a visual question. "What does personality mean in this context?" is a conceptual question — use the terminal. "Which wizard layout works better?" is a visual question — use the browser.
|
||||
|
||||
If they agree to the companion, read the detailed guide before proceeding:
|
||||
`skills/brainstorming/visual-companion.md`
|
||||
|
||||
## Wiki Integration
|
||||
|
||||
**Precondition**: `research-wiki/` directory exists (skip this section entirely if it does not).
|
||||
|
||||
**Trigger**: When a design is approved and the design document is being written.
|
||||
|
||||
**Output path**: Design docs are saved to `research-wiki/designs/` instead of `docs/superpowers/specs/`.
|
||||
|
||||
**Steps**:
|
||||
1. Run `.claude/tools/research_wiki.py add_entity research-wiki/ --type design --id <slug> --title "<design title>"` to create the design entity
|
||||
2. Append the chosen approach, key decision rationale, rejected alternatives, and why they were rejected to the generated page
|
||||
3. If the design is related to an idea, run `.claude/tools/research_wiki.py add_edge research-wiki/ --from "design:<id>" --to "idea:<id>" --type refines --evidence "..."`
|
||||
4. Run `.claude/tools/research_wiki.py rebuild_index research-wiki/`
|
||||
@@ -0,0 +1,49 @@
|
||||
# Spec Document Reviewer Prompt Template
|
||||
|
||||
Use this template when dispatching a spec document reviewer subagent.
|
||||
|
||||
**Purpose:** Verify the spec is complete, consistent, and ready for implementation planning.
|
||||
|
||||
**Dispatch after:** Spec document is written to docs/superpowers/specs/
|
||||
|
||||
```
|
||||
Task tool (general-purpose):
|
||||
description: "Review spec document"
|
||||
prompt: |
|
||||
You are a spec document reviewer. Verify this spec is complete and ready for planning.
|
||||
|
||||
**Spec to review:** [SPEC_FILE_PATH]
|
||||
|
||||
## What to Check
|
||||
|
||||
| Category | What to Look For |
|
||||
|----------|------------------|
|
||||
| Completeness | TODOs, placeholders, "TBD", incomplete sections |
|
||||
| Consistency | Internal contradictions, conflicting requirements |
|
||||
| Clarity | Requirements ambiguous enough to cause someone to build the wrong thing |
|
||||
| Scope | Focused enough for a single plan — not covering multiple independent subsystems |
|
||||
| YAGNI | Unrequested features, over-engineering |
|
||||
|
||||
## Calibration
|
||||
|
||||
**Only flag issues that would cause real problems during implementation planning.**
|
||||
A missing section, a contradiction, or a requirement so ambiguous it could be
|
||||
interpreted two different ways — those are issues. Minor wording improvements,
|
||||
stylistic preferences, and "sections less detailed than others" are not.
|
||||
|
||||
Approve unless there are serious gaps that would lead to a flawed plan.
|
||||
|
||||
## Output Format
|
||||
|
||||
## Spec Review
|
||||
|
||||
**Status:** Approved | Issues Found
|
||||
|
||||
**Issues (if any):**
|
||||
- [Section X]: [specific issue] - [why it matters for planning]
|
||||
|
||||
**Recommendations (advisory, do not block approval):**
|
||||
- [suggestions for improvement]
|
||||
```
|
||||
|
||||
**Reviewer returns:** Status, Issues (if any), Recommendations
|
||||
@@ -0,0 +1,287 @@
|
||||
# Visual Companion Guide
|
||||
|
||||
Browser-based visual brainstorming companion for showing mockups, diagrams, and options.
|
||||
|
||||
## When to Use
|
||||
|
||||
Decide per-question, not per-session. The test: **would the user understand this better by seeing it than reading it?**
|
||||
|
||||
**Use the browser** when the content itself is visual:
|
||||
|
||||
- **UI mockups** — wireframes, layouts, navigation structures, component designs
|
||||
- **Architecture diagrams** — system components, data flow, relationship maps
|
||||
- **Side-by-side visual comparisons** — comparing two layouts, two color schemes, two design directions
|
||||
- **Design polish** — when the question is about look and feel, spacing, visual hierarchy
|
||||
- **Spatial relationships** — state machines, flowcharts, entity relationships rendered as diagrams
|
||||
|
||||
**Use the terminal** when the content is text or tabular:
|
||||
|
||||
- **Requirements and scope questions** — "what does X mean?", "which features are in scope?"
|
||||
- **Conceptual A/B/C choices** — picking between approaches described in words
|
||||
- **Tradeoff lists** — pros/cons, comparison tables
|
||||
- **Technical decisions** — API design, data modeling, architectural approach selection
|
||||
- **Clarifying questions** — anything where the answer is words, not a visual preference
|
||||
|
||||
A question *about* a UI topic is not automatically a visual question. "What kind of wizard do you want?" is conceptual — use the terminal. "Which of these wizard layouts feels right?" is visual — use the browser.
|
||||
|
||||
## How It Works
|
||||
|
||||
The server watches a directory for HTML files and serves the newest one to the browser. You write HTML content to `screen_dir`, the user sees it in their browser and can click to select options. Selections are recorded to `state_dir/events` that you read on your next turn.
|
||||
|
||||
**Content fragments vs full documents:** If your HTML file starts with `<!DOCTYPE` or `<html`, the server serves it as-is (just injects the helper script). Otherwise, the server automatically wraps your content in the frame template — adding the header, CSS theme, selection indicator, and all interactive infrastructure. **Write content fragments by default.** Only write full documents when you need complete control over the page.
|
||||
|
||||
## Starting a Session
|
||||
|
||||
```bash
|
||||
# Start server with persistence (mockups saved to project)
|
||||
scripts/start-server.sh --project-dir /path/to/project
|
||||
|
||||
# Returns: {"type":"server-started","port":52341,"url":"http://localhost:52341",
|
||||
# "screen_dir":"/path/to/project/.superpowers/brainstorm/12345-1706000000/content",
|
||||
# "state_dir":"/path/to/project/.superpowers/brainstorm/12345-1706000000/state"}
|
||||
```
|
||||
|
||||
Save `screen_dir` and `state_dir` from the response. Tell user to open the URL.
|
||||
|
||||
**Finding connection info:** The server writes its startup JSON to `$STATE_DIR/server-info`. If you launched the server in the background and didn't capture stdout, read that file to get the URL and port. When using `--project-dir`, check `<project>/.superpowers/brainstorm/` for the session directory.
|
||||
|
||||
**Note:** Pass the project root as `--project-dir` so mockups persist in `.superpowers/brainstorm/` and survive server restarts. Without it, files go to `/tmp` and get cleaned up. Remind the user to add `.superpowers/` to `.gitignore` if it's not already there.
|
||||
|
||||
**Launching the server by platform:**
|
||||
|
||||
**Claude Code (macOS / Linux):**
|
||||
```bash
|
||||
# Default mode works — the script backgrounds the server itself
|
||||
scripts/start-server.sh --project-dir /path/to/project
|
||||
```
|
||||
|
||||
**Claude Code (Windows):**
|
||||
```bash
|
||||
# Windows auto-detects and uses foreground mode, which blocks the tool call.
|
||||
# Use run_in_background: true on the Bash tool call so the server survives
|
||||
# across conversation turns.
|
||||
scripts/start-server.sh --project-dir /path/to/project
|
||||
```
|
||||
When calling this via the Bash tool, set `run_in_background: true`. Then read `$STATE_DIR/server-info` on the next turn to get the URL and port.
|
||||
|
||||
**Codex:**
|
||||
```bash
|
||||
# Codex reaps background processes. The script auto-detects CODEX_CI and
|
||||
# switches to foreground mode. Run it normally — no extra flags needed.
|
||||
scripts/start-server.sh --project-dir /path/to/project
|
||||
```
|
||||
|
||||
**Gemini CLI:**
|
||||
```bash
|
||||
# Use --foreground and set is_background: true on your shell tool call
|
||||
# so the process survives across turns
|
||||
scripts/start-server.sh --project-dir /path/to/project --foreground
|
||||
```
|
||||
|
||||
**Other environments:** The server must keep running in the background across conversation turns. If your environment reaps detached processes, use `--foreground` and launch the command with your platform's background execution mechanism.
|
||||
|
||||
If the URL is unreachable from your browser (common in remote/containerized setups), bind a non-loopback host:
|
||||
|
||||
```bash
|
||||
scripts/start-server.sh \
|
||||
--project-dir /path/to/project \
|
||||
--host 0.0.0.0 \
|
||||
--url-host localhost
|
||||
```
|
||||
|
||||
Use `--url-host` to control what hostname is printed in the returned URL JSON.
|
||||
|
||||
## The Loop
|
||||
|
||||
1. **Check server is alive**, then **write HTML** to a new file in `screen_dir`:
|
||||
- Before each write, check that `$STATE_DIR/server-info` exists. If it doesn't (or `$STATE_DIR/server-stopped` exists), the server has shut down — restart it with `start-server.sh` before continuing. The server auto-exits after 30 minutes of inactivity.
|
||||
- Use semantic filenames: `platform.html`, `visual-style.html`, `layout.html`
|
||||
- **Never reuse filenames** — each screen gets a fresh file
|
||||
- Use Write tool — **never use cat/heredoc** (dumps noise into terminal)
|
||||
- Server automatically serves the newest file
|
||||
|
||||
2. **Tell user what to expect and end your turn:**
|
||||
- Remind them of the URL (every step, not just first)
|
||||
- Give a brief text summary of what's on screen (e.g., "Showing 3 layout options for the homepage")
|
||||
- Ask them to respond in the terminal: "Take a look and let me know what you think. Click to select an option if you'd like."
|
||||
|
||||
3. **On your next turn** — after the user responds in the terminal:
|
||||
- Read `$STATE_DIR/events` if it exists — this contains the user's browser interactions (clicks, selections) as JSON lines
|
||||
- Merge with the user's terminal text to get the full picture
|
||||
- The terminal message is the primary feedback; `state_dir/events` provides structured interaction data
|
||||
|
||||
4. **Iterate or advance** — if feedback changes current screen, write a new file (e.g., `layout-v2.html`). Only move to the next question when the current step is validated.
|
||||
|
||||
5. **Unload when returning to terminal** — when the next step doesn't need the browser (e.g., a clarifying question, a tradeoff discussion), push a waiting screen to clear the stale content:
|
||||
|
||||
```html
|
||||
<!-- filename: waiting.html (or waiting-2.html, etc.) -->
|
||||
<div style="display:flex;align-items:center;justify-content:center;min-height:60vh">
|
||||
<p class="subtitle">Continuing in terminal...</p>
|
||||
</div>
|
||||
```
|
||||
|
||||
This prevents the user from staring at a resolved choice while the conversation has moved on. When the next visual question comes up, push a new content file as usual.
|
||||
|
||||
6. Repeat until done.
|
||||
|
||||
## Writing Content Fragments
|
||||
|
||||
Write just the content that goes inside the page. The server wraps it in the frame template automatically (header, theme CSS, selection indicator, and all interactive infrastructure).
|
||||
|
||||
**Minimal example:**
|
||||
|
||||
```html
|
||||
<h2>Which layout works better?</h2>
|
||||
<p class="subtitle">Consider readability and visual hierarchy</p>
|
||||
|
||||
<div class="options">
|
||||
<div class="option" data-choice="a" onclick="toggleSelect(this)">
|
||||
<div class="letter">A</div>
|
||||
<div class="content">
|
||||
<h3>Single Column</h3>
|
||||
<p>Clean, focused reading experience</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="option" data-choice="b" onclick="toggleSelect(this)">
|
||||
<div class="letter">B</div>
|
||||
<div class="content">
|
||||
<h3>Two Column</h3>
|
||||
<p>Sidebar navigation with main content</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
```
|
||||
|
||||
That's it. No `<html>`, no CSS, no `<script>` tags needed. The server provides all of that.
|
||||
|
||||
## CSS Classes Available
|
||||
|
||||
The frame template provides these CSS classes for your content:
|
||||
|
||||
### Options (A/B/C choices)
|
||||
|
||||
```html
|
||||
<div class="options">
|
||||
<div class="option" data-choice="a" onclick="toggleSelect(this)">
|
||||
<div class="letter">A</div>
|
||||
<div class="content">
|
||||
<h3>Title</h3>
|
||||
<p>Description</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
```
|
||||
|
||||
**Multi-select:** Add `data-multiselect` to the container to let users select multiple options. Each click toggles the item. The indicator bar shows the count.
|
||||
|
||||
```html
|
||||
<div class="options" data-multiselect>
|
||||
<!-- same option markup — users can select/deselect multiple -->
|
||||
</div>
|
||||
```
|
||||
|
||||
### Cards (visual designs)
|
||||
|
||||
```html
|
||||
<div class="cards">
|
||||
<div class="card" data-choice="design1" onclick="toggleSelect(this)">
|
||||
<div class="card-image"><!-- mockup content --></div>
|
||||
<div class="card-body">
|
||||
<h3>Name</h3>
|
||||
<p>Description</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
```
|
||||
|
||||
### Mockup container
|
||||
|
||||
```html
|
||||
<div class="mockup">
|
||||
<div class="mockup-header">Preview: Dashboard Layout</div>
|
||||
<div class="mockup-body"><!-- your mockup HTML --></div>
|
||||
</div>
|
||||
```
|
||||
|
||||
### Split view (side-by-side)
|
||||
|
||||
```html
|
||||
<div class="split">
|
||||
<div class="mockup"><!-- left --></div>
|
||||
<div class="mockup"><!-- right --></div>
|
||||
</div>
|
||||
```
|
||||
|
||||
### Pros/Cons
|
||||
|
||||
```html
|
||||
<div class="pros-cons">
|
||||
<div class="pros"><h4>Pros</h4><ul><li>Benefit</li></ul></div>
|
||||
<div class="cons"><h4>Cons</h4><ul><li>Drawback</li></ul></div>
|
||||
</div>
|
||||
```
|
||||
|
||||
### Mock elements (wireframe building blocks)
|
||||
|
||||
```html
|
||||
<div class="mock-nav">Logo | Home | About | Contact</div>
|
||||
<div style="display: flex;">
|
||||
<div class="mock-sidebar">Navigation</div>
|
||||
<div class="mock-content">Main content area</div>
|
||||
</div>
|
||||
<button class="mock-button">Action Button</button>
|
||||
<input class="mock-input" placeholder="Input field">
|
||||
<div class="placeholder">Placeholder area</div>
|
||||
```
|
||||
|
||||
### Typography and sections
|
||||
|
||||
- `h2` — page title
|
||||
- `h3` — section heading
|
||||
- `.subtitle` — secondary text below title
|
||||
- `.section` — content block with bottom margin
|
||||
- `.label` — small uppercase label text
|
||||
|
||||
## Browser Events Format
|
||||
|
||||
When the user clicks options in the browser, their interactions are recorded to `$STATE_DIR/events` (one JSON object per line). The file is cleared automatically when you push a new screen.
|
||||
|
||||
```jsonl
|
||||
{"type":"click","choice":"a","text":"Option A - Simple Layout","timestamp":1706000101}
|
||||
{"type":"click","choice":"c","text":"Option C - Complex Grid","timestamp":1706000108}
|
||||
{"type":"click","choice":"b","text":"Option B - Hybrid","timestamp":1706000115}
|
||||
```
|
||||
|
||||
The full event stream shows the user's exploration path — they may click multiple options before settling. The last `choice` event is typically the final selection, but the pattern of clicks can reveal hesitation or preferences worth asking about.
|
||||
|
||||
If `$STATE_DIR/events` doesn't exist, the user didn't interact with the browser — use only their terminal text.
|
||||
|
||||
## Design Tips
|
||||
|
||||
- **Scale fidelity to the question** — wireframes for layout, polish for polish questions
|
||||
- **Explain the question on each page** — "Which layout feels more professional?" not just "Pick one"
|
||||
- **Iterate before advancing** — if feedback changes current screen, write a new version
|
||||
- **2-4 options max** per screen
|
||||
- **Use real content when it matters** — for a photography portfolio, use actual images (Unsplash). Placeholder content obscures design issues.
|
||||
- **Keep mockups simple** — focus on layout and structure, not pixel-perfect design
|
||||
|
||||
## File Naming
|
||||
|
||||
- Use semantic names: `platform.html`, `visual-style.html`, `layout.html`
|
||||
- Never reuse filenames — each screen must be a new file
|
||||
- For iterations: append version suffix like `layout-v2.html`, `layout-v3.html`
|
||||
- Server serves newest file by modification time
|
||||
|
||||
## Cleaning Up
|
||||
|
||||
```bash
|
||||
scripts/stop-server.sh $SESSION_DIR
|
||||
```
|
||||
|
||||
If the session used `--project-dir`, mockup files persist in `.superpowers/brainstorm/` for later reference. Only `/tmp` sessions get deleted on stop.
|
||||
|
||||
## Reference
|
||||
|
||||
- Frame template (CSS reference): `scripts/frame-template.html`
|
||||
- Helper script (client-side): `scripts/helper.js`
|
||||
@@ -0,0 +1,68 @@
|
||||
---
|
||||
name: commit
|
||||
description: Use when committing code changes - enforces commit message format, safety checks, and prohibits AI signatures
|
||||
---
|
||||
|
||||
# Commit
|
||||
|
||||
Automatically generate a commit message and submit the code.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. **Check status**: run `git status` to inspect changes (do not use `-uall`)
|
||||
2. **Review diff**: run `git diff --staged` and `git diff` to inspect the exact changes
|
||||
3. **Check style**: run `git log --oneline -10` to review recent commit style
|
||||
4. **Generate message**: create a commit message that follows the rules below
|
||||
5. **Commit**: stage the files and commit
|
||||
|
||||
## Commit Message Rules
|
||||
|
||||
### Format
|
||||
```text
|
||||
<type>: <description>
|
||||
|
||||
[optional body]
|
||||
```
|
||||
|
||||
### Type Values
|
||||
- `feat`: new feature
|
||||
- `fix`: bug fix
|
||||
- `docs`: documentation update
|
||||
- `refactor`: refactor with no functional change
|
||||
- `test`: test-related change
|
||||
- `chore`: build / tooling / configuration
|
||||
- `perf`: performance optimization
|
||||
- `ci`: CI/CD related
|
||||
|
||||
### Rules
|
||||
- The description must be written in English
|
||||
- Keep it under 72 characters
|
||||
- Use the imperative mood, for example "add user authentication" instead of "added user authentication"
|
||||
|
||||
## Safety Rules
|
||||
|
||||
> [!CAUTION]
|
||||
> **Never** add any AI signature to a commit message:
|
||||
> - Do not add `Co-Authored-By: Claude`
|
||||
> - Do not add `🤖 Generated with...`
|
||||
> - Do not add any AI signature or marker
|
||||
|
||||
## Sensitive File Check
|
||||
|
||||
Before committing, check whether any of the following files were accidentally added:
|
||||
- `.env` / `.env.*`
|
||||
- `credentials.json`
|
||||
- `*_secret*`
|
||||
- `*.pem` / `*.key`
|
||||
|
||||
If sensitive files are found, **warn the user** and wait for confirmation.
|
||||
|
||||
## Examples
|
||||
|
||||
```bash
|
||||
# Auto-generate a message
|
||||
/commit
|
||||
|
||||
# Specify a message
|
||||
/commit "feat: add user login"
|
||||
```
|
||||
@@ -0,0 +1,251 @@
|
||||
---
|
||||
name: finishing-a-development-branch
|
||||
description: Use when implementation is complete, all tests pass, and you need to decide how to integrate the work - guides completion of development work by presenting structured options for merge, PR, or cleanup
|
||||
---
|
||||
|
||||
# Finishing a Development Branch
|
||||
|
||||
## Overview
|
||||
|
||||
Guide completion of development work by presenting clear options and handling chosen workflow.
|
||||
|
||||
**Core principle:** Verify tests → Detect environment → Present options → Execute choice → Clean up.
|
||||
|
||||
**Announce at start:** "I'm using the finishing-a-development-branch skill to complete this work."
|
||||
|
||||
## The Process
|
||||
|
||||
### Step 1: Verify Tests
|
||||
|
||||
**Before presenting options, verify tests pass:**
|
||||
|
||||
```bash
|
||||
# Run project's test suite
|
||||
npm test / cargo test / pytest / go test ./...
|
||||
```
|
||||
|
||||
**If tests fail:**
|
||||
```
|
||||
Tests failing (<N> failures). Must fix before completing:
|
||||
|
||||
[Show failures]
|
||||
|
||||
Cannot proceed with merge/PR until tests pass.
|
||||
```
|
||||
|
||||
Stop. Don't proceed to Step 2.
|
||||
|
||||
**If tests pass:** Continue to Step 2.
|
||||
|
||||
### Step 2: Detect Environment
|
||||
|
||||
**Determine workspace state before presenting options:**
|
||||
|
||||
```bash
|
||||
GIT_DIR=$(cd "$(git rev-parse --git-dir)" 2>/dev/null && pwd -P)
|
||||
GIT_COMMON=$(cd "$(git rev-parse --git-common-dir)" 2>/dev/null && pwd -P)
|
||||
```
|
||||
|
||||
This determines which menu to show and how cleanup works:
|
||||
|
||||
| State | Menu | Cleanup |
|
||||
|-------|------|---------|
|
||||
| `GIT_DIR == GIT_COMMON` (normal repo) | Standard 4 options | No worktree to clean up |
|
||||
| `GIT_DIR != GIT_COMMON`, named branch | Standard 4 options | Provenance-based (see Step 6) |
|
||||
| `GIT_DIR != GIT_COMMON`, detached HEAD | Reduced 3 options (no merge) | No cleanup (externally managed) |
|
||||
|
||||
### Step 3: Determine Base Branch
|
||||
|
||||
```bash
|
||||
# Try common base branches
|
||||
git merge-base HEAD main 2>/dev/null || git merge-base HEAD master 2>/dev/null
|
||||
```
|
||||
|
||||
Or ask: "This branch split from main - is that correct?"
|
||||
|
||||
### Step 4: Present Options
|
||||
|
||||
**Normal repo and named-branch worktree — present exactly these 4 options:**
|
||||
|
||||
```
|
||||
Implementation complete. What would you like to do?
|
||||
|
||||
1. Merge back to <base-branch> locally
|
||||
2. Push and create a Pull Request
|
||||
3. Keep the branch as-is (I'll handle it later)
|
||||
4. Discard this work
|
||||
|
||||
Which option?
|
||||
```
|
||||
|
||||
**Detached HEAD — present exactly these 3 options:**
|
||||
|
||||
```
|
||||
Implementation complete. You're on a detached HEAD (externally managed workspace).
|
||||
|
||||
1. Push as new branch and create a Pull Request
|
||||
2. Keep as-is (I'll handle it later)
|
||||
3. Discard this work
|
||||
|
||||
Which option?
|
||||
```
|
||||
|
||||
**Don't add explanation** - keep options concise.
|
||||
|
||||
### Step 5: Execute Choice
|
||||
|
||||
#### Option 1: Merge Locally
|
||||
|
||||
```bash
|
||||
# Get main repo root for CWD safety
|
||||
MAIN_ROOT=$(git -C "$(git rev-parse --git-common-dir)/.." rev-parse --show-toplevel)
|
||||
cd "$MAIN_ROOT"
|
||||
|
||||
# Merge first — verify success before removing anything
|
||||
git checkout <base-branch>
|
||||
git pull
|
||||
git merge <feature-branch>
|
||||
|
||||
# Verify tests on merged result
|
||||
<test command>
|
||||
|
||||
# Only after merge succeeds: cleanup worktree (Step 6), then delete branch
|
||||
```
|
||||
|
||||
Then: Cleanup worktree (Step 6), then delete branch:
|
||||
|
||||
```bash
|
||||
git branch -d <feature-branch>
|
||||
```
|
||||
|
||||
#### Option 2: Push and Create PR
|
||||
|
||||
```bash
|
||||
# Push branch
|
||||
git push -u origin <feature-branch>
|
||||
|
||||
# Create PR
|
||||
gh pr create --title "<title>" --body "$(cat <<'EOF'
|
||||
## Summary
|
||||
<2-3 bullets of what changed>
|
||||
|
||||
## Test Plan
|
||||
- [ ] <verification steps>
|
||||
EOF
|
||||
)"
|
||||
```
|
||||
|
||||
**Do NOT clean up worktree** — user needs it alive to iterate on PR feedback.
|
||||
|
||||
#### Option 3: Keep As-Is
|
||||
|
||||
Report: "Keeping branch <name>. Worktree preserved at <path>."
|
||||
|
||||
**Don't cleanup worktree.**
|
||||
|
||||
#### Option 4: Discard
|
||||
|
||||
**Confirm first:**
|
||||
```
|
||||
This will permanently delete:
|
||||
- Branch <name>
|
||||
- All commits: <commit-list>
|
||||
- Worktree at <path>
|
||||
|
||||
Type 'discard' to confirm.
|
||||
```
|
||||
|
||||
Wait for exact confirmation.
|
||||
|
||||
If confirmed:
|
||||
```bash
|
||||
MAIN_ROOT=$(git -C "$(git rev-parse --git-common-dir)/.." rev-parse --show-toplevel)
|
||||
cd "$MAIN_ROOT"
|
||||
```
|
||||
|
||||
Then: Cleanup worktree (Step 6), then force-delete branch:
|
||||
```bash
|
||||
git branch -D <feature-branch>
|
||||
```
|
||||
|
||||
### Step 6: Cleanup Workspace
|
||||
|
||||
**Only runs for Options 1 and 4.** Options 2 and 3 always preserve the worktree.
|
||||
|
||||
```bash
|
||||
GIT_DIR=$(cd "$(git rev-parse --git-dir)" 2>/dev/null && pwd -P)
|
||||
GIT_COMMON=$(cd "$(git rev-parse --git-common-dir)" 2>/dev/null && pwd -P)
|
||||
WORKTREE_PATH=$(git rev-parse --show-toplevel)
|
||||
```
|
||||
|
||||
**If `GIT_DIR == GIT_COMMON`:** Normal repo, no worktree to clean up. Done.
|
||||
|
||||
**If worktree path is under `.worktrees/`, `worktrees/`, or `~/.config/superpowers/worktrees/`:** Superpowers created this worktree — we own cleanup.
|
||||
|
||||
```bash
|
||||
MAIN_ROOT=$(git -C "$(git rev-parse --git-common-dir)/.." rev-parse --show-toplevel)
|
||||
cd "$MAIN_ROOT"
|
||||
git worktree remove "$WORKTREE_PATH"
|
||||
git worktree prune # Self-healing: clean up any stale registrations
|
||||
```
|
||||
|
||||
**Otherwise:** The host environment (harness) owns this workspace. Do NOT remove it. If your platform provides a workspace-exit tool, use it. Otherwise, leave the workspace in place.
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Option | Merge | Push | Keep Worktree | Cleanup Branch |
|
||||
|--------|-------|------|---------------|----------------|
|
||||
| 1. Merge locally | yes | - | - | yes |
|
||||
| 2. Create PR | - | yes | yes | - |
|
||||
| 3. Keep as-is | - | - | yes | - |
|
||||
| 4. Discard | - | - | - | yes (force) |
|
||||
|
||||
## Common Mistakes
|
||||
|
||||
**Skipping test verification**
|
||||
- **Problem:** Merge broken code, create failing PR
|
||||
- **Fix:** Always verify tests before offering options
|
||||
|
||||
**Open-ended questions**
|
||||
- **Problem:** "What should I do next?" is ambiguous
|
||||
- **Fix:** Present exactly 4 structured options (or 3 for detached HEAD)
|
||||
|
||||
**Cleaning up worktree for Option 2**
|
||||
- **Problem:** Remove worktree user needs for PR iteration
|
||||
- **Fix:** Only cleanup for Options 1 and 4
|
||||
|
||||
**Deleting branch before removing worktree**
|
||||
- **Problem:** `git branch -d` fails because worktree still references the branch
|
||||
- **Fix:** Merge first, remove worktree, then delete branch
|
||||
|
||||
**Running git worktree remove from inside the worktree**
|
||||
- **Problem:** Command fails silently when CWD is inside the worktree being removed
|
||||
- **Fix:** Always `cd` to main repo root before `git worktree remove`
|
||||
|
||||
**Cleaning up harness-owned worktrees**
|
||||
- **Problem:** Removing a worktree the harness created causes phantom state
|
||||
- **Fix:** Only clean up worktrees under `.worktrees/`, `worktrees/`, or `~/.config/superpowers/worktrees/`
|
||||
|
||||
**No confirmation for discard**
|
||||
- **Problem:** Accidentally delete work
|
||||
- **Fix:** Require typed "discard" confirmation
|
||||
|
||||
## Red Flags
|
||||
|
||||
**Never:**
|
||||
- Proceed with failing tests
|
||||
- Merge without verifying tests on result
|
||||
- Delete work without confirmation
|
||||
- Force-push without explicit request
|
||||
- Remove a worktree before confirming merge success
|
||||
- Clean up worktrees you didn't create (provenance check)
|
||||
- Run `git worktree remove` from inside the worktree
|
||||
|
||||
**Always:**
|
||||
- Verify tests before offering options
|
||||
- Detect environment before presenting menu
|
||||
- Present exactly 4 options (or 3 for detached HEAD)
|
||||
- Get typed confirmation for Option 4
|
||||
- Clean up worktree for Options 1 & 4 only
|
||||
- `cd` to main repo root before worktree removal
|
||||
- Run `git worktree prune` after removal
|
||||
@@ -0,0 +1,616 @@
|
||||
---
|
||||
name: graphify
|
||||
description: "Use for any question about a codebase, its architecture, file relationships, or project content — especially when graphify-out/ exists, where the question should be treated as a graphify query first. Turns any input (code, docs, papers, images, videos) into a persistent knowledge graph with god nodes, community detection, and query/path/explain tools."
|
||||
trigger: /graphify
|
||||
---
|
||||
|
||||
# /graphify
|
||||
|
||||
Turn any folder of files into a navigable knowledge graph with community detection, an honest audit trail, and three outputs: interactive HTML, GraphRAG-ready JSON, and a plain-language GRAPH_REPORT.md.
|
||||
|
||||
## Usage
|
||||
|
||||
```
|
||||
/graphify # full pipeline on current directory → Obsidian vault
|
||||
/graphify <path> # full pipeline on specific path
|
||||
/graphify https://github.com/<owner>/<repo> # clone repo then run full pipeline on it
|
||||
/graphify https://github.com/<owner>/<repo> --branch <branch> # clone a specific branch
|
||||
/graphify <url1> <url2> ... # clone multiple repos, build each, merge into one cross-repo graph
|
||||
/graphify <path> --mode deep # thorough extraction, richer INFERRED edges
|
||||
/graphify <path> --update # incremental - re-extract only new/changed files
|
||||
/graphify <path> --directed # build directed graph (preserves edge direction: source→target)
|
||||
/graphify <path> --whisper-model medium # use a larger Whisper model for better transcription accuracy
|
||||
/graphify <path> --cluster-only # rerun clustering on existing graph
|
||||
/graphify <path> --no-viz # skip visualization, just report + JSON
|
||||
/graphify <path> --html # (HTML is generated by default - this flag is a no-op)
|
||||
/graphify <path> --svg # also export graph.svg (embeds in Notion, GitHub)
|
||||
/graphify <path> --graphml # export graph.graphml (Gephi, yEd)
|
||||
/graphify <path> --neo4j # generate graphify-out/cypher.txt for Neo4j
|
||||
/graphify <path> --neo4j-push bolt://localhost:7687 # push directly to Neo4j
|
||||
/graphify <path> --mcp # start MCP stdio server for agent access
|
||||
/graphify <path> --watch # watch folder, auto-rebuild on code changes (no LLM needed)
|
||||
/graphify <path> --wiki # build agent-crawlable wiki (index.md + one article per community)
|
||||
/graphify <path> --obsidian --obsidian-dir ~/vaults/my-project # write vault to custom path (e.g. existing vault)
|
||||
/graphify add <url> # fetch URL, save to ./raw, update graph
|
||||
/graphify add <url> --author "Name" # tag who wrote it
|
||||
/graphify add <url> --contributor "Name" # tag who added it to the corpus
|
||||
/graphify query "<question>" # BFS traversal - broad context
|
||||
/graphify query "<question>" --dfs # DFS - trace a specific path
|
||||
/graphify query "<question>" --budget 1500 # cap answer at N tokens
|
||||
/graphify path "AuthModule" "Database" # shortest path between two concepts
|
||||
/graphify explain "SwinTransformer" # plain-language explanation of a node
|
||||
```
|
||||
|
||||
## What graphify is for
|
||||
|
||||
Drop any folder of code, docs, papers, images, or video into graphify and get a queryable knowledge graph. Persistent across sessions, honest audit trail (EXTRACTED/INFERRED/AMBIGUOUS), community detection surfaces cross-document connections you wouldn't think to ask about.
|
||||
|
||||
## What You Must Do When Invoked
|
||||
|
||||
If the user invoked `/graphify --help` or `/graphify -h` (with no other arguments), print the contents of the `## Usage` section above verbatim and stop. Do not run any commands, do not detect files, do not default the path to `.`. Just print the Usage block and return.
|
||||
|
||||
**Fast path — existing graph:** Before doing anything else, check whether `graphify-out/graph.json` exists. The expected location is `graphify-out/graph.json` relative to the **current working directory** (i.e. the project root where you are running commands). If it exists AND the user's request is a natural-language question about the codebase (e.g. "How does X work?", "What calls Y?", "Trace the data flow through Z") and NOT an explicit rebuild command (`--update`, `--cluster-only`, or a bare path/URL that implies fresh extraction): **skip Steps 1–5 entirely and jump straight to `## For /graphify query`.** Run `graphify query "<question>"` immediately. Do not run detect. Do not check corpus size. Do not ask the user to narrow. The graph is already built — use it.
|
||||
|
||||
If no path was given, use `.` (current directory). Do not ask the user for a path.
|
||||
|
||||
If the path argument starts with `https://github.com/` or `http://github.com/`, treat it as a GitHub URL - run Step 0 before anything else, then continue with the resolved local path.
|
||||
|
||||
Follow these steps in order. Do not skip steps.
|
||||
|
||||
### Step 0 - GitHub repos and multi-path merge (only if a URL or several paths)
|
||||
|
||||
Only when the path is one or more `https://github.com/...` URLs, or several local subfolders to merge. See `references/github-and-merge.md` for the clone, cross-repo merge, and monorepo flow, then continue with the resolved local path. A plain local path skips this step.
|
||||
|
||||
### Step 1 - Ensure graphify is installed
|
||||
|
||||
```bash
|
||||
# Detect the correct Python interpreter (handles uv tool, pipx, venv, system installs)
|
||||
PYTHON=""
|
||||
GRAPHIFY_BIN=$(which graphify 2>/dev/null)
|
||||
# 1. uv tool installs — most reliable on modern Mac/Linux
|
||||
if [ -z "$PYTHON" ] && command -v uv >/dev/null 2>&1; then
|
||||
_UV_PY=$(uv tool run graphifyy python -c "import sys; print(sys.executable)" 2>/dev/null)
|
||||
if [ -n "$_UV_PY" ]; then PYTHON="$_UV_PY"; fi
|
||||
fi
|
||||
# 2. Read shebang from graphify binary (pipx and direct pip installs)
|
||||
if [ -z "$PYTHON" ] && [ -n "$GRAPHIFY_BIN" ]; then
|
||||
_SHEBANG=$(head -1 "$GRAPHIFY_BIN" | tr -d '#!')
|
||||
case "$_SHEBANG" in
|
||||
*[!a-zA-Z0-9/_.-]*) ;;
|
||||
*) "$_SHEBANG" -c "import graphify" 2>/dev/null && PYTHON="$_SHEBANG" ;;
|
||||
esac
|
||||
fi
|
||||
# 3. Fall back to python3
|
||||
if [ -z "$PYTHON" ]; then PYTHON="python3"; fi
|
||||
if ! "$PYTHON" -c "import graphify" 2>/dev/null; then
|
||||
if command -v uv >/dev/null 2>&1; then
|
||||
uv tool install --upgrade graphifyy -q 2>&1 | tail -3
|
||||
_UV_PY=$(uv tool run graphifyy python -c "import sys; print(sys.executable)" 2>/dev/null)
|
||||
if [ -n "$_UV_PY" ]; then PYTHON="$_UV_PY"; fi
|
||||
else
|
||||
"$PYTHON" -m pip install graphifyy -q 2>/dev/null \
|
||||
|| "$PYTHON" -m pip install graphifyy -q --break-system-packages 2>&1 | tail -3
|
||||
fi
|
||||
fi
|
||||
# Write interpreter path for all subsequent steps (persists across invocations)
|
||||
mkdir -p graphify-out
|
||||
"$PYTHON" -c "import sys; open('graphify-out/.graphify_python', 'w', encoding='utf-8').write(sys.executable)"
|
||||
# Save scan root so `graphify update` (no args) knows where to look next time
|
||||
echo "$(cd INPUT_PATH && pwd)" > graphify-out/.graphify_root
|
||||
```
|
||||
|
||||
If the import succeeds, print nothing and move straight to Step 2.
|
||||
|
||||
**In every subsequent bash block, replace `python3` with `$(cat graphify-out/.graphify_python)` to use the correct interpreter.**
|
||||
|
||||
### Step 2 - Detect files
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
from graphify.detect import detect
|
||||
from pathlib import Path
|
||||
result = detect(Path('INPUT_PATH'))
|
||||
print(json.dumps(result, ensure_ascii=False))
|
||||
" > graphify-out/.graphify_detect.json
|
||||
```
|
||||
|
||||
Replace INPUT_PATH with the actual path the user provided. Do NOT cat or print the JSON - read it silently and present a clean summary instead:
|
||||
|
||||
```
|
||||
Corpus: X files · ~Y words
|
||||
code: N files (.py .ts .go ...)
|
||||
docs: N files (.md .txt ...)
|
||||
papers: N files (.pdf ...)
|
||||
images: N files
|
||||
video: N files (.mp4 .mp3 ...)
|
||||
```
|
||||
|
||||
Omit any category with 0 files from the summary.
|
||||
|
||||
Then act on it:
|
||||
- If `total_files` is 0: stop with "No supported files found in [path]."
|
||||
- If `skipped_sensitive` is non-empty: mention file count skipped, not the file names.
|
||||
- If `total_words` > 2,000,000 OR `total_files` > 500: show the warning. Then compute the top 5 first-level subdirectories by file count:
|
||||
- Read `scan_root` from the detect JSON (always an absolute path to the resolved INPUT_PATH).
|
||||
- Concatenate all file lists across all types (`code`, `document`, `paper`, `image`, `video`).
|
||||
- Filter out any path that starts with `scan_root + "/graphify-out/"` to exclude converted sidecars.
|
||||
- For each file, strip the `scan_root` prefix and take the first path component. Files directly in `scan_root` with no subdirectory count as `(root)`.
|
||||
- If all files are in `(root)` with no subdirectories, do not ask to narrow — no subfolders exist. Instead suggest `--no-cluster` to skip the expensive clustering step and proceed.
|
||||
- Otherwise rank by count, show the top 5 with file counts, then ask which subfolder to run on. Wait for the user's answer before proceeding.
|
||||
- Otherwise: proceed directly to Step 2.5 if video files were detected, or Step 3 if not.
|
||||
|
||||
### Step 2.5 - Video and audio (only if video files detected)
|
||||
|
||||
Skip this step entirely if `detect` returned zero `video` files. When the corpus has video or audio, see `references/transcribe.md` to transcribe them to text first, then treat the transcripts as doc files in Step 3.
|
||||
|
||||
### Step 3 - Extract entities and relationships
|
||||
|
||||
**Before starting:** note whether `--mode deep` was given. You must pass `DEEP_MODE=true` to every subagent in Step B2 if it was. Track this from the original invocation - do not lose it.
|
||||
|
||||
This step has two parts: **structural extraction** (deterministic, free) and **semantic extraction** (LLM, costs tokens).
|
||||
|
||||
**Before dispatching subagents:** check whether `GEMINI_API_KEY` or `GOOGLE_API_KEY` is set. If neither is set, print this one-liner to the user:
|
||||
> Tip: set `GEMINI_API_KEY` or `GOOGLE_API_KEY` to use Gemini for semantic extraction (`pip install 'graphifyy[gemini]'`).
|
||||
|
||||
Print it once, then continue. If `GEMINI_API_KEY` or `GOOGLE_API_KEY` IS set, use `graphify.llm.extract_corpus_parallel(files, backend="gemini")` for semantic extraction instead of dispatching Claude subagents. The default Gemini model is `gemini-3-flash-preview`; set `GRAPHIFY_GEMINI_MODEL` or pass `--model` in headless CLI flows to override it.
|
||||
|
||||
> **No other API keys are read.** If `GEMINI_API_KEY`/`GOOGLE_API_KEY` are unset, fall straight through to Claude Code subagent dispatch (Part B below) — the host session itself is the LLM. graphify does **not** read `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, or any other provider key from the environment. If a host agent prompts the user for `ANTHROPIC_API_KEY` to run extraction, that prompt is a misread of this skill — ignore it and dispatch subagents as written.
|
||||
|
||||
**Run Part A (AST) and Part B (semantic) in parallel. Dispatch all semantic subagents AND start AST extraction in the same message. Both can run simultaneously since they operate on different file types. Merge results in Part C as before.**
|
||||
|
||||
Note: Parallelizing AST + semantic saves 5-15s on large corpora. AST is deterministic and fast; start it while subagents are processing docs/papers.
|
||||
|
||||
#### Part A - Structural extraction for code files
|
||||
|
||||
For any code files detected, run AST extraction in parallel with Part B subagents:
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import sys, json
|
||||
from graphify.extract import collect_files, extract
|
||||
from pathlib import Path
|
||||
import json
|
||||
|
||||
code_files = []
|
||||
detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encoding=\"utf-8\"))
|
||||
for f in detect.get('files', {}).get('code', []):
|
||||
code_files.extend(collect_files(Path(f)) if Path(f).is_dir() else [Path(f)])
|
||||
|
||||
if code_files:
|
||||
result = extract(code_files, cache_root=Path('.'))
|
||||
Path('graphify-out/.graphify_ast.json').write_text(json.dumps(result, indent=2, ensure_ascii=False), encoding=\"utf-8\")
|
||||
print(f'AST: {len(result[\"nodes\"])} nodes, {len(result[\"edges\"])} edges')
|
||||
else:
|
||||
Path('graphify-out/.graphify_ast.json').write_text(json.dumps({'nodes':[],'edges':[],'input_tokens':0,'output_tokens':0}, ensure_ascii=False), encoding=\"utf-8\")
|
||||
print('No code files - skipping AST extraction')
|
||||
"
|
||||
```
|
||||
|
||||
#### Part B - Semantic extraction (parallel subagents)
|
||||
|
||||
**Fast path:** If detection found zero docs, papers, and images (code-only corpus), skip Part B entirely and go straight to Part C. AST handles code - there is nothing for semantic subagents to do.
|
||||
|
||||
**MANDATORY: You MUST use the Agent tool here. Reading files yourself one-by-one is forbidden - it is 5-10x slower. If you do not use the Agent tool you are doing this wrong.**
|
||||
|
||||
Before dispatching subagents, print a timing estimate:
|
||||
- Load `total_words` and file counts from `graphify-out/.graphify_detect.json`
|
||||
- Estimate agents needed: `ceil(uncached_non_code_files / 22)` (chunk size is 20-25)
|
||||
- Estimate time: ~45s per agent batch (they run in parallel, so total ≈ 45s × ceil(agents/parallel_limit))
|
||||
- Print: "Semantic extraction: ~N files → X agents, estimated ~Ys"
|
||||
|
||||
**Step B0 - Check extraction cache first**
|
||||
|
||||
Before dispatching any subagents, check which files already have cached extraction results:
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
from graphify.cache import check_semantic_cache
|
||||
from pathlib import Path
|
||||
|
||||
detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encoding=\"utf-8\"))
|
||||
all_files = [f for files in detect['files'].values() for f in files]
|
||||
|
||||
cached_nodes, cached_edges, cached_hyperedges, uncached = check_semantic_cache(all_files)
|
||||
|
||||
if cached_nodes or cached_edges or cached_hyperedges:
|
||||
Path('graphify-out/.graphify_cached.json').write_text(json.dumps({'nodes': cached_nodes, 'edges': cached_edges, 'hyperedges': cached_hyperedges}, ensure_ascii=False), encoding=\"utf-8\")
|
||||
Path('graphify-out/.graphify_uncached.txt').write_text('\n'.join(uncached), encoding=\"utf-8\")
|
||||
print(f'Cache: {len(all_files)-len(uncached)} files hit, {len(uncached)} files need extraction')
|
||||
"
|
||||
```
|
||||
|
||||
Only dispatch subagents for files listed in `graphify-out/.graphify_uncached.txt`. If all files are cached, skip to Part C directly.
|
||||
|
||||
**Step B1 - Split into chunks**
|
||||
|
||||
Load files from `graphify-out/.graphify_uncached.txt`. Split into chunks of 20-25 files each. Each image gets its own chunk (vision needs separate context). When splitting, group files from the same directory together so related artifacts land in the same chunk and cross-file relationships are more likely to be extracted.
|
||||
|
||||
**Step B2 - Dispatch ALL subagents in a single message**
|
||||
|
||||
Call the Agent tool multiple times IN THE SAME RESPONSE - one call per chunk. This is the only way they run in parallel. If you make one Agent call, wait, then make another, you are doing it sequentially and defeating the purpose.
|
||||
|
||||
**IMPORTANT - subagent type:** Always use `subagent_type="general-purpose"`. Do NOT use `Explore` - it is read-only and cannot write chunk files to disk, which silently drops extraction results. General-purpose has Write and Bash access which the subagent needs.
|
||||
|
||||
Concrete example for 3 chunks:
|
||||
```
|
||||
[Agent tool call 1: files 1-15, subagent_type="general-purpose"]
|
||||
[Agent tool call 2: files 16-30, subagent_type="general-purpose"]
|
||||
[Agent tool call 3: files 31-45, subagent_type="general-purpose"]
|
||||
```
|
||||
All three in one message. Not three separate messages.
|
||||
|
||||
Each subagent receives this exact prompt (substitute FILE_LIST, CHUNK_NUM, TOTAL_CHUNKS, DEEP_MODE, and CHUNK_PATH).
|
||||
|
||||
CHUNK_PATH must be an **absolute** path — derive it before dispatching:
|
||||
```bash
|
||||
PROJECT_ROOT=$(cat graphify-out/.graphify_root)
|
||||
# Then for chunk N: CHUNK_PATH="${PROJECT_ROOT}/graphify-out/.graphify_chunk_0N.json"
|
||||
```
|
||||
|
||||
Subagent prompt template:
|
||||
|
||||
See `references/extraction-spec.md` for the exact subagent prompt (JSON schema, node-ID rules, confidence rubric, frontmatter, hyperedge, and vision rules). Load it only here, only when at least one chunk holds a doc, paper, or image; a pure-code corpus has skipped Part B and never reads it. Pass each subagent that prompt verbatim with FILE_LIST, CHUNK_NUM, TOTAL_CHUNKS, DEEP_MODE, and CHUNK_PATH substituted, and have it write the result to CHUNK_PATH.
|
||||
|
||||
**Step B3 - Collect, cache, and merge**
|
||||
|
||||
Wait for all subagents. For each result:
|
||||
- Check that `graphify-out/.graphify_chunk_NN.json` exists on disk — this is the success signal
|
||||
- If the file exists and contains valid JSON with `nodes` and `edges`, include it and save to cache
|
||||
- If the file is missing, the subagent was likely dispatched as read-only (Explore type) — print a warning: "chunk N missing from disk — subagent may have been read-only. Re-run with general-purpose agent." Do not silently skip.
|
||||
- If a subagent failed or returned invalid JSON, print a warning and skip that chunk - do not abort
|
||||
|
||||
If more than half the chunks failed or are missing, stop and tell the user to re-run and ensure `subagent_type="general-purpose"` is used.
|
||||
|
||||
Merge all chunk files into `.graphify_semantic_new.json`. **After each Agent call completes, read the real token counts from the Agent tool result's `usage` field and write them back into the chunk JSON before merging** — the chunk JSON itself always has placeholder zeros. Then run:
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json, glob
|
||||
from pathlib import Path
|
||||
|
||||
chunks = sorted(glob.glob('graphify-out/.graphify_chunk_*.json'))
|
||||
all_nodes, all_edges, all_hyperedges = [], [], []
|
||||
total_in, total_out = 0, 0
|
||||
for c in chunks:
|
||||
d = json.loads(Path(c).read_text(encoding=\"utf-8\"))
|
||||
all_nodes += d.get('nodes', [])
|
||||
all_edges += d.get('edges', [])
|
||||
all_hyperedges += d.get('hyperedges', [])
|
||||
total_in += d.get('input_tokens', 0)
|
||||
total_out += d.get('output_tokens', 0)
|
||||
Path('graphify-out/.graphify_semantic_new.json').write_text(json.dumps({
|
||||
'nodes': all_nodes, 'edges': all_edges, 'hyperedges': all_hyperedges,
|
||||
'input_tokens': total_in, 'output_tokens': total_out,
|
||||
}, indent=2, ensure_ascii=False), encoding=\"utf-8\")
|
||||
print(f'Merged {len(chunks)} chunks: {total_in:,} in / {total_out:,} out tokens')
|
||||
"
|
||||
```
|
||||
|
||||
Save new results to cache:
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
from graphify.cache import save_semantic_cache
|
||||
from pathlib import Path
|
||||
|
||||
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []))
|
||||
print(f'Cached {saved} files')
|
||||
"
|
||||
```
|
||||
|
||||
Merge cached + new results into `graphify-out/.graphify_semantic.json`:
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
cached = json.loads(Path('graphify-out/.graphify_cached.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_cached.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
|
||||
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
|
||||
|
||||
all_nodes = cached['nodes'] + new.get('nodes', [])
|
||||
all_edges = cached['edges'] + new.get('edges', [])
|
||||
all_hyperedges = cached.get('hyperedges', []) + new.get('hyperedges', [])
|
||||
seen = set()
|
||||
deduped = []
|
||||
for n in all_nodes:
|
||||
if n['id'] not in seen:
|
||||
seen.add(n['id'])
|
||||
deduped.append(n)
|
||||
|
||||
merged = {
|
||||
'nodes': deduped,
|
||||
'edges': all_edges,
|
||||
'hyperedges': all_hyperedges,
|
||||
'input_tokens': new.get('input_tokens', 0),
|
||||
'output_tokens': new.get('output_tokens', 0),
|
||||
}
|
||||
Path('graphify-out/.graphify_semantic.json').write_text(json.dumps(merged, indent=2, ensure_ascii=False), encoding=\"utf-8\")
|
||||
print(f'Extraction complete - {len(deduped)} nodes, {len(all_edges)} edges ({len(cached[\"nodes\"])} from cache, {len(new.get(\"nodes\",[]))} new)')
|
||||
"
|
||||
```
|
||||
Clean up temp files: `rm -f graphify-out/.graphify_cached.json graphify-out/.graphify_uncached.txt graphify-out/.graphify_semantic_new.json`
|
||||
|
||||
#### Part C - Merge AST + semantic into final extraction
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import sys, json
|
||||
from pathlib import Path
|
||||
|
||||
ast = json.loads(Path('graphify-out/.graphify_ast.json').read_text(encoding=\"utf-8\"))
|
||||
sem = json.loads(Path('graphify-out/.graphify_semantic.json').read_text(encoding=\"utf-8\"))
|
||||
|
||||
# Merge: AST nodes first, semantic nodes deduplicated by id
|
||||
seen = {n['id'] for n in ast['nodes']}
|
||||
merged_nodes = list(ast['nodes'])
|
||||
for n in sem['nodes']:
|
||||
if n['id'] not in seen:
|
||||
merged_nodes.append(n)
|
||||
seen.add(n['id'])
|
||||
|
||||
merged_edges = ast['edges'] + sem['edges']
|
||||
merged_hyperedges = sem.get('hyperedges', [])
|
||||
merged = {
|
||||
'nodes': merged_nodes,
|
||||
'edges': merged_edges,
|
||||
'hyperedges': merged_hyperedges,
|
||||
'input_tokens': sem.get('input_tokens', 0),
|
||||
'output_tokens': sem.get('output_tokens', 0),
|
||||
}
|
||||
Path('graphify-out/.graphify_extract.json').write_text(json.dumps(merged, indent=2, ensure_ascii=False), encoding=\"utf-8\")
|
||||
total = len(merged_nodes)
|
||||
edges = len(merged_edges)
|
||||
print(f'Merged: {total} nodes, {edges} edges ({len(ast[\"nodes\"])} AST + {len(sem[\"nodes\"])} semantic)')
|
||||
"
|
||||
```
|
||||
|
||||
### Step 4 - Build graph, cluster, analyze, generate outputs
|
||||
|
||||
**Before starting:** note whether `--directed` was given. If so, pass `directed=True` to `build_from_json()` in the code block below. This builds a `DiGraph` that preserves edge direction (source→target) instead of the default undirected `Graph`.
|
||||
|
||||
```bash
|
||||
mkdir -p graphify-out
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import sys, json
|
||||
from graphify.build import build_from_json
|
||||
from graphify.cluster import cluster, score_all
|
||||
from graphify.analyze import god_nodes, surprising_connections, suggest_questions
|
||||
from graphify.report import generate
|
||||
from graphify.export import to_json
|
||||
from pathlib import Path
|
||||
|
||||
extraction = json.loads(Path('graphify-out/.graphify_extract.json').read_text(encoding=\"utf-8\"))
|
||||
detection = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encoding=\"utf-8\"))
|
||||
|
||||
G = build_from_json(extraction)
|
||||
communities = cluster(G)
|
||||
cohesion = score_all(G, communities)
|
||||
tokens = {'input': extraction.get('input_tokens', 0), 'output': extraction.get('output_tokens', 0)}
|
||||
gods = god_nodes(G)
|
||||
surprises = surprising_connections(G, communities)
|
||||
labels = {cid: 'Community ' + str(cid) for cid in communities}
|
||||
# Placeholder questions - regenerated with real labels in Step 5
|
||||
questions = suggest_questions(G, communities, labels)
|
||||
|
||||
report = generate(G, communities, cohesion, labels, gods, surprises, detection, tokens, '.', suggested_questions=questions)
|
||||
Path('graphify-out/GRAPH_REPORT.md').write_text(report, encoding=\"utf-8\")
|
||||
to_json(G, communities, 'graphify-out/graph.json')
|
||||
|
||||
analysis = {
|
||||
'communities': {str(k): v for k, v in communities.items()},
|
||||
'cohesion': {str(k): v for k, v in cohesion.items()},
|
||||
'gods': gods,
|
||||
'surprises': surprises,
|
||||
'questions': questions,
|
||||
}
|
||||
Path('graphify-out/.graphify_analysis.json').write_text(json.dumps(analysis, indent=2, ensure_ascii=False), encoding=\"utf-8\")
|
||||
if G.number_of_nodes() == 0:
|
||||
print('ERROR: Graph is empty - extraction produced no nodes.')
|
||||
print('Possible causes: all files were skipped, binary-only corpus, or extraction failed.')
|
||||
raise SystemExit(1)
|
||||
print(f'Graph: {G.number_of_nodes()} nodes, {G.number_of_edges()} edges, {len(communities)} communities')
|
||||
"
|
||||
```
|
||||
|
||||
If this step prints `ERROR: Graph is empty`, stop and tell the user what happened - do not proceed to labeling or visualization.
|
||||
|
||||
Replace INPUT_PATH with the actual path.
|
||||
|
||||
### Step 5 - Label communities
|
||||
|
||||
Read `graphify-out/.graphify_analysis.json`. For each community key, look at its node labels and write a 2-5 word plain-language name (e.g. "Attention Mechanism", "Training Pipeline", "Data Loading").
|
||||
|
||||
Then regenerate the report and save the labels for the visualizer:
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import sys, json
|
||||
from graphify.build import build_from_json
|
||||
from graphify.cluster import score_all
|
||||
from graphify.analyze import god_nodes, surprising_connections, suggest_questions
|
||||
from graphify.report import generate
|
||||
from pathlib import Path
|
||||
|
||||
extraction = json.loads(Path('graphify-out/.graphify_extract.json').read_text(encoding=\"utf-8\"))
|
||||
detection = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encoding=\"utf-8\"))
|
||||
analysis = json.loads(Path('graphify-out/.graphify_analysis.json').read_text(encoding=\"utf-8\"))
|
||||
|
||||
G = build_from_json(extraction)
|
||||
communities = {int(k): v for k, v in analysis['communities'].items()}
|
||||
cohesion = {int(k): v for k, v in analysis['cohesion'].items()}
|
||||
tokens = {'input': extraction.get('input_tokens', 0), 'output': extraction.get('output_tokens', 0)}
|
||||
|
||||
# LABELS - replace these with the names you chose above
|
||||
labels = LABELS_DICT
|
||||
|
||||
# Regenerate questions with real community labels (labels affect question phrasing)
|
||||
questions = suggest_questions(G, communities, labels)
|
||||
|
||||
report = generate(G, communities, cohesion, labels, analysis['gods'], analysis['surprises'], detection, tokens, '.', suggested_questions=questions)
|
||||
Path('graphify-out/GRAPH_REPORT.md').write_text(report, encoding=\"utf-8\")
|
||||
Path('graphify-out/.graphify_labels.json').write_text(json.dumps({str(k): v for k, v in labels.items()}, ensure_ascii=False), encoding=\"utf-8\")
|
||||
print('Report updated with community labels')
|
||||
"
|
||||
```
|
||||
|
||||
Replace `LABELS_DICT` with the actual dict you constructed (e.g. `{0: "Attention Mechanism", 1: "Training Pipeline"}`).
|
||||
Replace INPUT_PATH with the actual path.
|
||||
|
||||
### Step 6 - Generate Obsidian vault (opt-in) + HTML
|
||||
|
||||
**Generate HTML always** (unless `--no-viz`). **Obsidian vault only if `--obsidian` was explicitly given** — skip it otherwise, it generates one file per node.
|
||||
|
||||
If `--obsidian` was given:
|
||||
|
||||
- If `--obsidian-dir <path>` was also given, pass it via `--dir`. Otherwise defaults to `graphify-out/obsidian`.
|
||||
|
||||
```bash
|
||||
graphify export obsidian
|
||||
# or with custom dir: graphify export obsidian --dir ~/vaults/my-project
|
||||
```
|
||||
|
||||
Generate the HTML graph (always, unless `--no-viz`):
|
||||
|
||||
```bash
|
||||
graphify export html # auto-aggregates to community view if graph > 5000 nodes
|
||||
# or: graphify export html --no-viz
|
||||
```
|
||||
|
||||
### Steps 6b-8 - Wiki, Neo4j, SVG, GraphML, MCP, benchmark (only on their flags)
|
||||
|
||||
These run only when their flag is present (`--wiki`, `--neo4j`/`--neo4j-push`, `--svg`, `--graphml`, `--mcp`) or, for the token-reduction benchmark, when `total_words` exceeds 5,000. A default run with no export flags skips all of them. See `references/exports.md` for each one. Run any `--wiki` export before Step 9 cleanup so `.graphify_labels.json` is still available.
|
||||
|
||||
---
|
||||
|
||||
### Step 9 - Save manifest, update cost tracker, clean up, and report
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
from pathlib import Path
|
||||
from datetime import datetime, timezone
|
||||
from graphify.detect import save_manifest
|
||||
|
||||
# Save manifest for --update
|
||||
detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encoding=\"utf-8\"))
|
||||
# In --update mode, 'all_files' carries the full corpus; 'files' is the changed
|
||||
# subset. Full-rebuild mode populates only 'files', so the fallback handles that.
|
||||
save_manifest(detect.get('all_files') or detect['files'])
|
||||
|
||||
# Update cumulative cost tracker
|
||||
extract = json.loads(Path('graphify-out/.graphify_extract.json').read_text(encoding=\"utf-8\"))
|
||||
input_tok = extract.get('input_tokens', 0)
|
||||
output_tok = extract.get('output_tokens', 0)
|
||||
|
||||
cost_path = Path('graphify-out/cost.json')
|
||||
if cost_path.exists():
|
||||
cost = json.loads(cost_path.read_text(encoding=\"utf-8\"))
|
||||
else:
|
||||
cost = {'runs': [], 'total_input_tokens': 0, 'total_output_tokens': 0}
|
||||
|
||||
cost['runs'].append({
|
||||
'date': datetime.now(timezone.utc).isoformat(),
|
||||
'input_tokens': input_tok,
|
||||
'output_tokens': output_tok,
|
||||
'files': detect.get('total_files', 0),
|
||||
})
|
||||
cost['total_input_tokens'] += input_tok
|
||||
cost['total_output_tokens'] += output_tok
|
||||
cost_path.write_text(json.dumps(cost, indent=2, ensure_ascii=False), encoding=\"utf-8\")
|
||||
|
||||
print(f'This run: {input_tok:,} input tokens, {output_tok:,} output tokens')
|
||||
print(f'All time: {cost[\"total_input_tokens\"]:,} input, {cost[\"total_output_tokens\"]:,} output ({len(cost[\"runs\"])} runs)')
|
||||
"
|
||||
rm -f graphify-out/.graphify_detect.json graphify-out/.graphify_extract.json graphify-out/.graphify_ast.json graphify-out/.graphify_semantic.json graphify-out/.graphify_analysis.json
|
||||
find graphify-out -maxdepth 1 -name '.graphify_chunk_*.json' -delete 2>/dev/null
|
||||
rm -f graphify-out/.needs_update 2>/dev/null || true
|
||||
```
|
||||
|
||||
Tell the user (omit the obsidian line unless --obsidian was given):
|
||||
```
|
||||
Graph complete. Outputs in PATH_TO_DIR/graphify-out/
|
||||
|
||||
graph.html - interactive graph, open in browser
|
||||
GRAPH_REPORT.md - audit report
|
||||
graph.json - raw graph data
|
||||
obsidian/ - Obsidian vault (only if --obsidian was given)
|
||||
```
|
||||
|
||||
If graphify saved you time, consider supporting it: https://github.com/sponsors/safishamsi
|
||||
|
||||
Replace PATH_TO_DIR with the actual absolute path of the directory that was processed.
|
||||
|
||||
Then paste these sections from GRAPH_REPORT.md directly into the chat:
|
||||
- God Nodes
|
||||
- Surprising Connections
|
||||
- Suggested Questions
|
||||
|
||||
Do NOT paste the full report - just those three sections. Keep it concise.
|
||||
|
||||
Then immediately offer to explore. Pick the single most interesting suggested question from the report - the one that crosses the most community boundaries or has the most surprising bridge node - and ask:
|
||||
|
||||
> "The most interesting question this graph can answer: **[question]**. Want me to trace it?"
|
||||
|
||||
If the user says yes, run `/graphify query "[question]"` on the graph and walk them through the answer using the graph structure - which nodes connect, which community boundaries get crossed, what the path reveals. Keep going as long as they want to explore. Each answer should end with a natural follow-up ("this connects to X - want to go deeper?") so the session feels like navigation, not a one-shot report.
|
||||
|
||||
The graph is the map. Your job after the pipeline is to be the guide.
|
||||
|
||||
---
|
||||
|
||||
## Interpreter guard for subcommands
|
||||
|
||||
Before running any subcommand below (`--update`, `--cluster-only`, `query`, `path`, `explain`, `add`), check that `.graphify_python` exists. If it's missing (e.g. user deleted `graphify-out/`), re-resolve the interpreter first:
|
||||
|
||||
```bash
|
||||
if [ ! -f graphify-out/.graphify_python ]; then
|
||||
GRAPHIFY_BIN=$(which graphify 2>/dev/null)
|
||||
if [ -n "$GRAPHIFY_BIN" ]; then
|
||||
PYTHON=$(head -1 "$GRAPHIFY_BIN" | tr -d '#!')
|
||||
case "$PYTHON" in *[!a-zA-Z0-9/_.-]*) PYTHON="python3" ;; esac
|
||||
else
|
||||
PYTHON="python3"
|
||||
fi
|
||||
mkdir -p graphify-out
|
||||
"$PYTHON" -c "import sys; open('graphify-out/.graphify_python', 'w', encoding='utf-8').write(sys.executable)"
|
||||
fi
|
||||
```
|
||||
|
||||
## For --update and --cluster-only
|
||||
|
||||
Both are non-default subcommands. `--update` re-extracts only new or changed files; `--cluster-only` reruns clustering on the existing graph. See `references/update.md` for both flows.
|
||||
|
||||
---
|
||||
|
||||
## For /graphify query
|
||||
|
||||
When `graphify-out/graph.json` already exists and the user asks a question about the corpus, run the query directly:
|
||||
|
||||
```bash
|
||||
graphify query "<question>"
|
||||
```
|
||||
|
||||
Answer using only what the graph output contains, and quote `source_location` when citing a specific fact. Before traversal, expand the question against the graph's own vocabulary so a wording mismatch does not collapse the answer to noise. For that vocab-expansion step, the `--dfs` / `--budget` modes, `save-result` feedback, and the `/graphify path` and `/graphify explain` flows, see `references/query.md`.
|
||||
|
||||
---
|
||||
|
||||
## For /graphify add and --watch
|
||||
|
||||
Neither is part of the default build. When the user runs `/graphify add <url>` to fetch a URL into the corpus, or passes `--watch` to auto-rebuild on file changes, see `references/add-watch.md`.
|
||||
|
||||
---
|
||||
|
||||
## For the commit hook and native CLAUDE.md integration
|
||||
|
||||
When the user asks to install the post-commit auto-rebuild hook or wire graphify into a project's CLAUDE.md, see `references/hooks.md`.
|
||||
|
||||
---
|
||||
|
||||
## Honesty Rules
|
||||
|
||||
- Never invent an edge. If unsure, use AMBIGUOUS.
|
||||
- Never skip the corpus check warning.
|
||||
- Always show token cost in the report.
|
||||
- Never hide cohesion scores behind symbols - show the raw number.
|
||||
- Never run HTML viz on a graph with more than 5,000 nodes without warning the user.
|
||||
@@ -0,0 +1,56 @@
|
||||
# graphify reference: add a URL and watch a folder
|
||||
|
||||
Load this when the user ran `/graphify add <url>` or passed `--watch`. Neither is part of the default build.
|
||||
|
||||
## For /graphify add
|
||||
|
||||
Fetch a URL and add it to the corpus, then update the graph.
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import sys
|
||||
from graphify.ingest import ingest
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
out = ingest('URL', Path('./raw'), author='AUTHOR', contributor='CONTRIBUTOR')
|
||||
print(f'Saved to {out}')
|
||||
except ValueError as e:
|
||||
print(f'error: {e}', file=sys.stderr)
|
||||
sys.exit(1)
|
||||
except RuntimeError as e:
|
||||
print(f'error: {e}', file=sys.stderr)
|
||||
sys.exit(1)
|
||||
"
|
||||
```
|
||||
|
||||
Replace `URL` with the actual URL, `AUTHOR` with the user's name if provided, `CONTRIBUTOR` likewise. If the command exits with an error, tell the user what went wrong - do not silently continue. After a successful save, automatically run the `--update` pipeline on `./raw` to merge the new file into the existing graph.
|
||||
|
||||
Supported URL types (auto-detected):
|
||||
- YouTube / any video URL → audio downloaded via yt-dlp, transcribed to `.txt` on next run (requires `pip install 'graphifyy[video]'`)
|
||||
- Twitter/X → fetched via oEmbed, saved as `.md` with tweet text and author
|
||||
- arXiv → abstract + metadata saved as `.md`
|
||||
- PDF → downloaded as `.pdf`
|
||||
- Images (.png/.jpg/.webp) → downloaded, Claude vision extracts on next run
|
||||
- Any webpage → converted to markdown via html2text
|
||||
|
||||
---
|
||||
|
||||
## For --watch
|
||||
|
||||
Start a background watcher that monitors a folder and auto-updates the graph when files change.
|
||||
|
||||
```bash
|
||||
python3 -m graphify.watch INPUT_PATH --debounce 3
|
||||
```
|
||||
|
||||
Replace INPUT_PATH with the folder to watch. Behavior depends on what changed:
|
||||
|
||||
- **Code files only (.py, .ts, .go, etc.):** re-runs AST extraction + rebuild + cluster immediately, no LLM needed. `graph.json` and `GRAPH_REPORT.md` are updated automatically.
|
||||
- **Docs, papers, or images:** writes a `graphify-out/needs_update` flag and prints a notification to run `/graphify --update` (LLM semantic re-extraction required).
|
||||
|
||||
Debounce (default 3s): waits until file activity stops before triggering, so a wave of parallel agent writes doesn't trigger a rebuild per file.
|
||||
|
||||
Press Ctrl+C to stop.
|
||||
|
||||
For agentic workflows: run `--watch` in a background terminal. Code changes from agent waves are picked up automatically between waves. If agents are also writing docs or notes, you'll need a manual `/graphify --update` after those waves.
|
||||
@@ -0,0 +1,71 @@
|
||||
# graphify reference: extra exports and benchmark
|
||||
|
||||
Load this when the user passed one of the export flags (`--wiki`, `--neo4j`, `--neo4j-push`, `--svg`, `--graphml`, `--mcp`), or when the corpus is large enough for the token-reduction benchmark. Each step runs only for its own flag.
|
||||
|
||||
### Step 6b - Wiki (only if --wiki flag)
|
||||
|
||||
**Only run this step if `--wiki` was explicitly given in the original command.**
|
||||
|
||||
Run this before Step 9 (cleanup) so `.graphify_labels.json` is still available.
|
||||
|
||||
```bash
|
||||
graphify export wiki
|
||||
```
|
||||
|
||||
### Step 7 - Neo4j export (only if --neo4j or --neo4j-push flag)
|
||||
|
||||
**If `--neo4j`** - generate a Cypher file for manual import:
|
||||
|
||||
```bash
|
||||
graphify export neo4j
|
||||
```
|
||||
|
||||
**If `--neo4j-push <uri>`** - push directly to a running Neo4j instance. Ask the user for credentials if not provided:
|
||||
|
||||
```bash
|
||||
graphify export neo4j --push bolt://localhost:7687 --user neo4j --password PASSWORD
|
||||
```
|
||||
|
||||
Default URI is `bolt://localhost:7687`, default user is `neo4j`. Uses MERGE - safe to re-run without creating duplicates.
|
||||
|
||||
### Step 7b - SVG export (only if --svg flag)
|
||||
|
||||
```bash
|
||||
graphify export svg
|
||||
```
|
||||
|
||||
### Step 7c - GraphML export (only if --graphml flag)
|
||||
|
||||
```bash
|
||||
graphify export graphml
|
||||
```
|
||||
|
||||
### Step 7d - MCP server (only if --mcp flag)
|
||||
|
||||
```bash
|
||||
python3 -m graphify.serve graphify-out/graph.json
|
||||
```
|
||||
|
||||
This starts a stdio MCP server that exposes tools: `query_graph`, `get_node`, `get_neighbors`, `get_community`, `god_nodes`, `graph_stats`, `shortest_path`. Add to Claude Desktop or any MCP-compatible agent orchestrator so other agents can query the graph live.
|
||||
|
||||
To configure in Claude Desktop, add to `claude_desktop_config.json`:
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"graphify": {
|
||||
"command": "python3",
|
||||
"args": ["-m", "graphify.serve", "/absolute/path/to/graphify-out/graph.json"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Step 8 - Token reduction benchmark (only if total_words > 5000)
|
||||
|
||||
If `total_words` from `graphify-out/.graphify_detect.json` is greater than 5,000, run:
|
||||
|
||||
```bash
|
||||
graphify benchmark
|
||||
```
|
||||
|
||||
Print the output directly in chat. If `total_words <= 5000`, skip silently - the graph value is structural clarity, not token compression, for small corpora.
|
||||
@@ -0,0 +1,68 @@
|
||||
# graphify reference: extraction subagent prompt
|
||||
|
||||
Load this in Step 3 Part B when the corpus has at least one doc, paper, or image chunk. A pure-code corpus skips Part B and never reads this file. Each semantic subagent receives the prompt below verbatim (substitute FILE_LIST, CHUNK_NUM, TOTAL_CHUNKS, DEEP_MODE, and CHUNK_PATH).
|
||||
|
||||
```
|
||||
You are a graphify extraction subagent. Read the files listed and extract a knowledge graph fragment.
|
||||
Output ONLY valid JSON matching the schema below - no explanation, no markdown fences, no preamble.
|
||||
|
||||
Files (chunk CHUNK_NUM of TOTAL_CHUNKS):
|
||||
FILE_LIST
|
||||
|
||||
Rules:
|
||||
- EXTRACTED: relationship explicit in source (import, call, citation, "see §3.2")
|
||||
- INFERRED: reasonable inference (shared data structure, implied dependency)
|
||||
- AMBIGUOUS: uncertain - flag for review, do not omit
|
||||
|
||||
Code files: focus on semantic edges AST cannot find (call relationships, shared data, arch patterns).
|
||||
Do not re-extract imports - AST already has those.
|
||||
Doc/paper files: extract named concepts, entities, citations. For rationale (WHY decisions were made, trade-offs, design intent): store as a `rationale` attribute on the relevant concept node — do NOT create a separate rationale node or fragment node. Only create a node for something that is itself a named entity or concept. Use `file_type:"rationale"` for concept-like nodes (ideas, principles, mechanisms, design patterns). `file_type` MUST be one of exactly these six values: `code`, `document`, `paper`, `image`, `rationale`, `concept`. Any other value is invalid and will be rejected.
|
||||
Code files: when adding `calls` edges, source MUST be the caller (the function/class doing the calling), target MUST be the callee. Never reverse this direction. `calls` edges MUST stay within one language: a Python function cannot `calls` a JS/TS/Go/Rust/Java symbol and vice versa — cross-language call edges are phantom artifacts, never emit them.
|
||||
Image files: use vision to understand what the image IS - do not just OCR.
|
||||
UI screenshot: layout patterns, design decisions, key elements, purpose.
|
||||
Chart: metric, trend/insight, data source.
|
||||
Tweet/post: claim as node, author, concepts mentioned.
|
||||
Diagram: components and connections.
|
||||
Research figure: what it demonstrates, method, result.
|
||||
Handwritten/whiteboard: ideas and arrows, mark uncertain readings AMBIGUOUS.
|
||||
|
||||
DEEP_MODE (if --mode deep was given): be aggressive with INFERRED edges - indirect deps,
|
||||
shared assumptions, latent couplings. Mark uncertain ones AMBIGUOUS instead of omitting.
|
||||
|
||||
Semantic similarity: if two concepts in this chunk solve the same problem or represent the same idea without any structural link (no import, no call, no citation), add a `semantically_similar_to` edge marked INFERRED with a confidence_score reflecting how similar they are (0.6-0.95). Examples:
|
||||
- Two functions that both validate user input but never call each other
|
||||
- A class in code and a concept in a paper that describe the same algorithm
|
||||
- Two error types that handle the same failure mode differently
|
||||
Only add these when the similarity is genuinely non-obvious and cross-cutting. Do not add them for trivially similar things.
|
||||
|
||||
Hyperedges: if 3 or more nodes clearly participate together in a shared concept, flow, or pattern that is not captured by pairwise edges alone, add a hyperedge to a top-level `hyperedges` array. Examples:
|
||||
- All classes that implement a common protocol or interface
|
||||
- All functions in an authentication flow (even if they don't all call each other)
|
||||
- All concepts from a paper section that form one coherent idea
|
||||
Use sparingly — only when the group relationship adds information beyond the pairwise edges. Maximum 3 hyperedges per chunk.
|
||||
|
||||
If a file has YAML frontmatter (--- ... ---), copy source_url, captured_at, author,
|
||||
contributor onto every node from that file.
|
||||
|
||||
confidence_score is REQUIRED on every edge - never omit it, never use 0.5 as a default:
|
||||
- EXTRACTED edges: confidence_score = 1.0 always
|
||||
- INFERRED edges: pick exactly ONE value from this set — never 0.5:
|
||||
0.95 direct structural evidence (shared data structure, named cross-file reference).
|
||||
0.85 strong inference (clear functional alignment, no direct symbol link).
|
||||
0.75 reasonable inference (shared problem domain + similar shape, requires interpretation).
|
||||
0.65 weak inference (thematically related, no shape evidence).
|
||||
0.55 speculative but plausible (surface-level co-occurrence only).
|
||||
Models follow discrete rubrics better than continuous ranges; the bimodal
|
||||
distribution observed in production (>50% at 0.5, >40% at 0.85+) shows the
|
||||
range guidance is being collapsed to a binary. If no value above fits, mark
|
||||
the edge AMBIGUOUS rather than picking 0.4 or below.
|
||||
- AMBIGUOUS edges: 0.1-0.3
|
||||
|
||||
Node ID format: lowercase, only `[a-z0-9_]`, no dots or slashes. Format: `{stem}_{entity}` where stem is `{parent_dir}_{filename_without_ext}` (the **immediate** parent directory name + the filename stem, both lowercased with non-alphanumeric chars replaced by `_`) and entity is the symbol name similarly normalized. Only one level of parent is used — not the full path. Examples: `src/auth/session.py` + `ValidateToken` → `auth_session_validatetoken`; `lib/utils/helpers.py` + `parse_url` → `utils_helpers_parse_url`; `tests/test_foo.py` + `_helper` → `tests_test_foo_helper`. Top-level files (no parent dir, e.g. `setup.py`) use just the filename stem: `setup_my_func`. This must match the ID the AST extractor generates — using just the filename (e.g., `session_validatetoken`) or the full path (e.g., `src_auth_session_validatetoken`) will create orphan ghost-duplicate nodes. If you are re-extracting a project that had ghost duplicates under the old format, the user should run `graphify extract --force` to rebuild cleanly. CRITICAL: never append chunk numbers, sequence numbers, or any suffix to an ID (no `_c1`, `_c2`, `_chunk2`, etc.). IDs must be deterministic from the label alone — the same entity must always produce the same ID regardless of which chunk processes it.
|
||||
|
||||
Generate the extraction JSON matching this schema exactly:
|
||||
{"nodes":[{"id":"session_validatetoken","label":"Human Readable Name","file_type":"code|document|paper|image|rationale|concept","source_file":"relative/path","source_location":null,"source_url":null,"captured_at":null,"author":null,"contributor":null}],"edges":[{"source":"node_id","target":"node_id","relation":"calls|implements|references|cites|conceptually_related_to|shares_data_with|semantically_similar_to|rationale_for","confidence":"EXTRACTED|INFERRED|AMBIGUOUS","confidence_score":1.0,"source_file":"relative/path","source_location":null,"weight":1.0}],"hyperedges":[{"id":"snake_case_id","label":"Human Readable Label","nodes":["node_id1","node_id2","node_id3"],"relation":"participate_in|implement|form","confidence":"EXTRACTED|INFERRED","confidence_score":0.75,"source_file":"relative/path"}],"input_tokens":0,"output_tokens":0}
|
||||
|
||||
Then write the JSON to disk using the Write tool at this exact absolute path (no relative paths — Write resolves relative paths against an undefined cwd and the file will be silently lost):
|
||||
CHUNK_PATH
|
||||
```
|
||||
@@ -0,0 +1,46 @@
|
||||
# graphify reference: GitHub clone and cross-repo merge
|
||||
|
||||
Load this when the user passed one or more `https://github.com/...` URLs, or named several local subfolders to merge into one graph.
|
||||
|
||||
### Step 0 - Clone GitHub repo(s) (only if a GitHub URL was given)
|
||||
|
||||
**Single repo:**
|
||||
```bash
|
||||
LOCAL_PATH=$(graphify clone <github-url> [--branch <branch>])
|
||||
# Use LOCAL_PATH as the target for all subsequent steps
|
||||
```
|
||||
|
||||
**Multiple repos (cross-repo graph):**
|
||||
```bash
|
||||
# Clone each repo, run the full pipeline on each, then merge
|
||||
graphify clone <url1> # → ~/.graphify/repos/<owner1>/<repo1>
|
||||
graphify clone <url2> # → ~/.graphify/repos/<owner2>/<repo2>
|
||||
# Run /graphify on each local path to produce their graph.json files
|
||||
# Then merge:
|
||||
graphify merge-graphs \
|
||||
~/.graphify/repos/<owner1>/<repo1>/graphify-out/graph.json \
|
||||
~/.graphify/repos/<owner2>/<repo2>/graphify-out/graph.json \
|
||||
--out graphify-out/cross-repo-graph.json
|
||||
```
|
||||
|
||||
Graphify clones into `~/.graphify/repos/<owner>/<repo>` and reuses existing clones on repeat runs. Each node in the merged graph carries a `repo` attribute so you can filter by origin.
|
||||
|
||||
**Multiple local subfolders (monorepo or multi-service layout):**
|
||||
|
||||
The skill pipeline writes all intermediate and final outputs to `graphify-out/` in the current working directory. Running the skill on each subfolder separately will clobber the same output dir. Instead, use the CLI directly for each subfolder — it places `graphify-out/` *inside* the scanned path:
|
||||
|
||||
```bash
|
||||
graphify extract ./core/ # → ./core/graphify-out/graph.json
|
||||
graphify extract ./service/ # → ./service/graphify-out/graph.json
|
||||
graphify extract ./platform/ # → ./platform/graphify-out/graph.json
|
||||
# Add --backend gemini|kimi|openai|deepseek|claude-cli depending on which API key you have set
|
||||
|
||||
# Then merge at the project root:
|
||||
graphify merge-graphs \
|
||||
./core/graphify-out/graph.json \
|
||||
./service/graphify-out/graph.json \
|
||||
./platform/graphify-out/graph.json \
|
||||
--out graphify-out/graph.json
|
||||
```
|
||||
|
||||
Once `graphify-out/graph.json` exists, the fast path above takes over: any codebase question runs `graphify query` directly on the merged graph — no re-extraction, no size gate.
|
||||
@@ -0,0 +1,33 @@
|
||||
# graphify reference: commit hook and native CLAUDE.md integration
|
||||
|
||||
Load this when the user asked to install the post-commit hook or wire graphify into a project's CLAUDE.md.
|
||||
|
||||
## For git commit hook
|
||||
|
||||
Install a post-commit hook that auto-rebuilds the graph after every commit. No background process needed - triggers once per commit, works with any editor.
|
||||
|
||||
```bash
|
||||
graphify hook install # install
|
||||
graphify hook uninstall # remove
|
||||
graphify hook status # check
|
||||
```
|
||||
|
||||
After every `git commit`, the hook detects which code files changed (via `git diff HEAD~1`), re-runs AST extraction on those files, and rebuilds `graph.json` and `GRAPH_REPORT.md`. Doc/image changes are ignored by the hook - run `/graphify --update` manually for those.
|
||||
|
||||
If a post-commit hook already exists, graphify appends to it rather than replacing it.
|
||||
|
||||
---
|
||||
|
||||
## For native CLAUDE.md integration
|
||||
|
||||
Run once per project to make graphify always-on in Claude Code sessions:
|
||||
|
||||
```bash
|
||||
graphify claude install
|
||||
```
|
||||
|
||||
This writes a `## graphify` section to the local `CLAUDE.md` that instructs Claude to check the graph before answering codebase questions and rebuild it after code changes. No manual `/graphify` needed in future sessions.
|
||||
|
||||
```bash
|
||||
graphify claude uninstall # remove the section
|
||||
```
|
||||
@@ -0,0 +1,103 @@
|
||||
# graphify reference: query, path, explain
|
||||
|
||||
Load this when the user asks a question against an existing graph, or runs `/graphify path` or `/graphify explain`. The core's query stub points here for the full traversal flow.
|
||||
|
||||
Two traversal modes - choose based on the question:
|
||||
|
||||
| Mode | Flag | Best for |
|
||||
|------|------|----------|
|
||||
| BFS (default) | _(none)_ | "What is X connected to?" - broad context, nearest neighbors first |
|
||||
| DFS | `--dfs` | "How does X reach Y?" - trace a specific chain or dependency path |
|
||||
|
||||
### Step 0 — Constrained query expansion (REQUIRED before traversal)
|
||||
|
||||
graphify's `query` CLI matches nodes via case-folded substring + IDF — there is **no stemming, no synonyms, no cross-language match** inside the binary. If the user's question uses different language or different domain vocabulary than the graph's labels (user says "обработчик" / graph says "handler"; user says "authentication" / graph says "Guardian"), the literal matcher returns 0 hits and the answer collapses to noise.
|
||||
|
||||
Fix this **without inventing tokens** by expanding the query against the actual graph vocabulary first:
|
||||
|
||||
1. Extract the token vocabulary from node labels:
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json, re
|
||||
from pathlib import Path
|
||||
data = json.loads(Path('graphify-out/graph.json').read_text())
|
||||
vocab = set()
|
||||
for n in data['nodes']:
|
||||
for c in re.findall(r'[^\W\d_]+', n.get('label','') or '', re.UNICODE):
|
||||
parts = re.findall(r'[A-Z]+(?=[A-Z][a-z])|[A-Z]?[a-z]+|[A-Z]+', c) or [c]
|
||||
for p in parts:
|
||||
t = p.lower()
|
||||
if 3 <= len(t) <= 30:
|
||||
vocab.add(t)
|
||||
Path('graphify-out/.vocab.txt').write_text('\n'.join(sorted(vocab)))
|
||||
print(f'vocab: {len(vocab)} tokens')
|
||||
"
|
||||
```
|
||||
|
||||
2. Read `graphify-out/.vocab.txt`. Then for the user's question, select **up to 12 tokens from this exact list** that semantically match the query intent. Hard constraints:
|
||||
- You MUST pick only tokens present in the vocabulary file. Do NOT invent tokens.
|
||||
- If a query concept has no plausible token in the vocab, skip it — do not substitute a near-synonym from training memory.
|
||||
- If **no** vocab tokens match the query at all, output an empty list and tell the user the corpus has no relevant vocabulary for this question. Do not fabricate a search.
|
||||
- Translate cross-language: Russian "аутентификация" → look for `auth`, `credential`, `token`, `security` IFF present in vocab.
|
||||
- Morphology: "handlers" maps to `handler` IFF present; "todos" maps to `todo` IFF present.
|
||||
|
||||
3. Print the selection explicitly to the user before running the query, so the expansion is auditable:
|
||||
```
|
||||
Query expanded to (from graph vocab, N tokens): [token1, token2, ...]
|
||||
```
|
||||
If the list is empty, say so plainly and stop — do not proceed to traversal.
|
||||
|
||||
### Step 1 — Traversal
|
||||
|
||||
Build the **expanded query string** by joining the selected tokens with spaces. Use this string as `QUESTION` below — NOT the original user question. (The original question is preserved only for `save-result` at the end.)
|
||||
|
||||
```bash
|
||||
graphify query "QUESTION"
|
||||
# or: graphify query "QUESTION" --dfs --budget 3000
|
||||
```
|
||||
|
||||
Answer using **only** what the graph output contains. Quote `source_location` when citing a specific fact. If the graph lacks enough information, say so - do not hallucinate edges.
|
||||
|
||||
After writing the answer, save it back into the graph so it improves future queries. Include the expanded tokens inside the `--answer` text (e.g. `"Expanded from original query via vocab: [tokens]. Then traversed..."`) so the next `--update` extracts the expansion history as a graph node:
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -m graphify save-result --question "ORIGINAL_QUESTION" --answer "ANSWER" --type query --nodes NODE1 NODE2
|
||||
```
|
||||
|
||||
Replace `ORIGINAL_QUESTION` with the user's verbatim question, `ANSWER` with your full answer text (containing the expanded-token trace), `NODE1 NODE2` with the list of node labels you cited. This closes the feedback loop: the next `--update` will extract this Q&A as a node in the graph.
|
||||
|
||||
---
|
||||
|
||||
## For /graphify path
|
||||
|
||||
Find the shortest path between two named concepts in the graph.
|
||||
|
||||
```bash
|
||||
graphify path "NODE_A" "NODE_B"
|
||||
```
|
||||
|
||||
Replace `NODE_A` and `NODE_B` with the actual concept names. Then explain the path in plain language - what each hop means, why it's significant.
|
||||
|
||||
After writing the explanation, save it back:
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -m graphify save-result --question "Path from NODE_A to NODE_B" --answer "ANSWER" --type path_query --nodes NODE_A NODE_B
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## For /graphify explain
|
||||
|
||||
Give a plain-language explanation of a single node - everything connected to it.
|
||||
|
||||
```bash
|
||||
graphify explain "NODE_NAME"
|
||||
```
|
||||
|
||||
Replace `NODE_NAME` with the concept the user asked about. Then write a 3-5 sentence explanation of what this node is, what it connects to, and why those connections are significant. Use the source locations as citations.
|
||||
|
||||
After writing the explanation, save it back:
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -m graphify save-result --question "Explain NODE_NAME" --answer "ANSWER" --type explain --nodes NODE_NAME
|
||||
```
|
||||
@@ -0,0 +1,48 @@
|
||||
# graphify reference: transcribe video and audio
|
||||
|
||||
Load this only when `detect` reported one or more `video` files. A corpus with no video never reads this.
|
||||
|
||||
### Step 2.5 - Transcribe video / audio files (only if video files detected)
|
||||
|
||||
Skip this step entirely if `detect` returned zero `video` files.
|
||||
|
||||
Video and audio files cannot be read directly. Transcribe them to text first, then treat the transcripts as doc files in Step 3.
|
||||
|
||||
**Strategy:** Read the god nodes from `graphify-out/.graphify_detect.json` (or the analysis file if it exists from a previous run). You are already a language model — write a one-sentence domain hint yourself from those labels. Then pass it to Whisper as the initial prompt. No separate API call needed.
|
||||
|
||||
**However**, if the corpus has *only* video files and no other docs/code, use the generic fallback prompt: `"Use proper punctuation and paragraph breaks."`
|
||||
|
||||
**Step 1 - Write the Whisper prompt yourself.**
|
||||
|
||||
Read the top god node labels from detect output or analysis, then compose a short domain hint sentence, for example:
|
||||
|
||||
- Labels: `transformer, attention, encoder, decoder` → `"Machine learning research on transformer architectures and attention mechanisms. Use proper punctuation and paragraph breaks."`
|
||||
- Labels: `kubernetes, deployment, pod, helm` → `"DevOps discussion about Kubernetes deployments and Helm charts. Use proper punctuation and paragraph breaks."`
|
||||
|
||||
Set it as `WHISPER_PROMPT` to use in the next command.
|
||||
|
||||
**Step 2 - Transcribe:**
|
||||
|
||||
```bash
|
||||
GRAPHIFY_WHISPER_MODEL=base # or whatever --whisper-model the user passed
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json, os
|
||||
from pathlib import Path
|
||||
from graphify.transcribe import transcribe_all
|
||||
|
||||
detect = json.loads(Path('graphify-out/.graphify_detect.json').read_text(encoding=\"utf-8\"))
|
||||
video_files = detect.get('files', {}).get('video', [])
|
||||
prompt = os.environ.get('GRAPHIFY_WHISPER_PROMPT', 'Use proper punctuation and paragraph breaks.')
|
||||
|
||||
transcript_paths = transcribe_all(video_files, initial_prompt=prompt)
|
||||
print(json.dumps(transcript_paths, ensure_ascii=False))
|
||||
" > graphify-out/.graphify_transcripts.json
|
||||
```
|
||||
|
||||
After transcription:
|
||||
- Read the transcript paths from `graphify-out/.graphify_transcripts.json`
|
||||
- Add them to the docs list before dispatching semantic subagents in Step 3B
|
||||
- Print how many transcripts were created: `Transcribed N video file(s) -> treating as docs`
|
||||
- If transcription fails for a file, print a warning and continue with the rest
|
||||
|
||||
**Whisper model:** Default is `base`. If the user passed `--whisper-model <name>`, set `GRAPHIFY_WHISPER_MODEL=<name>` in the environment before running the command above.
|
||||
@@ -0,0 +1,179 @@
|
||||
# graphify reference: incremental update and cluster-only
|
||||
|
||||
Load this only when the user passed `--update` or `--cluster-only`. A first-time full build never reads this file.
|
||||
|
||||
## For --update (incremental re-extraction)
|
||||
|
||||
Use when you've added or modified files since the last run. Only re-extracts changed files - saves tokens and time.
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import sys, json
|
||||
from graphify.detect import detect_incremental, save_manifest
|
||||
from pathlib import Path
|
||||
|
||||
result = detect_incremental(Path('INPUT_PATH'))
|
||||
new_total = result.get('new_total', 0)
|
||||
print(json.dumps(result, indent=2, ensure_ascii=False))
|
||||
Path('graphify-out/.graphify_incremental.json').write_text(json.dumps(result, ensure_ascii=False), encoding=\"utf-8\")
|
||||
deleted = list(result.get('deleted_files', []))
|
||||
if new_total == 0 and not deleted:
|
||||
print('No files changed since last run. Nothing to update.')
|
||||
raise SystemExit(0)
|
||||
if deleted:
|
||||
print(f'{len(deleted)} deleted file(s) to prune.')
|
||||
if new_total > 0:
|
||||
print(f'{new_total} new/changed file(s) to re-extract.')
|
||||
"
|
||||
```
|
||||
|
||||
Then populate `.graphify_detect.json` so Steps 3A–6 (which read it unconditionally) see the right state for an incremental run. `files` carries the changed subset (drives Step 3A AST + Step 3B0 cache check on only what changed); `all_files` carries the full corpus for any step that needs corpus-wide context:
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
from pathlib import Path
|
||||
r = json.loads(Path('graphify-out/.graphify_incremental.json').read_text(encoding=\"utf-8\"))
|
||||
Path('graphify-out/.graphify_detect.json').write_text(json.dumps({
|
||||
'files': r.get('new_files', {}),
|
||||
'all_files': r.get('files', {}),
|
||||
'total_files': r.get('new_total', 0),
|
||||
'total_words': r.get('total_words', 0),
|
||||
'skipped_sensitive': r.get('skipped_sensitive', []),
|
||||
'needs_graph': True,
|
||||
}, ensure_ascii=False), encoding=\"utf-8\")
|
||||
"
|
||||
```
|
||||
|
||||
If new files exist, first check whether all changed files are code files:
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
result = json.loads(open('graphify-out/.graphify_incremental.json', encoding='utf-8').read()) if Path('graphify-out/.graphify_incremental.json').exists() else {}
|
||||
code_exts = {'.py','.ts','.js','.go','.rs','.java','.cpp','.c','.rb','.swift','.kt','.cs','.scala','.php','.cc','.cxx','.hpp','.h','.kts','.lua','.toc','.f','.F','.f90','.F90','.f95','.F95','.f03','.F03','.f08','.F08'}
|
||||
new_files = result.get('new_files', {})
|
||||
all_changed = [f for files in new_files.values() for f in files]
|
||||
code_only = all(Path(f).suffix.lower() in code_exts for f in all_changed)
|
||||
print('code_only:', code_only)
|
||||
"
|
||||
```
|
||||
|
||||
If `code_only` is True: print `[graphify update] Code-only changes detected - skipping semantic extraction (no LLM needed)`, run only Step 3A (AST) on the changed files, skip Step 3B entirely (no subagents), then go straight to merge and Steps 4–8.
|
||||
|
||||
If `code_only` is False (any changed file is a doc/paper/image): run the full Steps 3A–3C pipeline as normal.
|
||||
|
||||
|
||||
If no new files exist (only deletions), create an empty extraction so the merge step can prune:
|
||||
|
||||
```bash
|
||||
if [ ! -f graphify-out/.graphify_extract.json ]; then
|
||||
echo '[graphify update] Only deletions -- creating empty extraction for merge.'
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
from pathlib import Path
|
||||
Path('graphify-out/.graphify_extract.json').write_text(json.dumps({'nodes':[],'edges':[],'hyperedges':[],'input_tokens':0,'output_tokens':0}), encoding='utf-8')
|
||||
"
|
||||
fi
|
||||
```
|
||||
|
||||
|
||||
Then:
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
from pathlib import Path
|
||||
from graphify.build import build_merge
|
||||
from graphify.detect import save_manifest
|
||||
|
||||
# Load new extraction and incremental state
|
||||
new_extraction = json.loads(Path('graphify-out/.graphify_extract.json').read_text(encoding=\"utf-8\"))
|
||||
incremental = json.loads(Path('graphify-out/.graphify_incremental.json').read_text(encoding=\"utf-8\"))
|
||||
deleted = list(incremental.get('deleted_files', []))
|
||||
# Also prune old nodes for re-extracted (changed) files before inserting fresh AST.
|
||||
# Without this, build_merge's dedup pass tries to reconcile old and new versions of
|
||||
# the same file's nodes and can collapse same-named symbols across files (#1178).
|
||||
changed = [f for files in incremental.get('new_files', {}).values() for f in files]
|
||||
prune = list(dict.fromkeys(deleted + changed)) or None
|
||||
|
||||
# Use build_merge() — reads graph.json directly without NetworkX round-trip
|
||||
# so edge direction (calls, implements, imports) is always preserved (#801).
|
||||
G = build_merge(
|
||||
[new_extraction],
|
||||
graph_path='graphify-out/graph.json',
|
||||
prune_sources=prune,
|
||||
)
|
||||
print(f'[graphify update] Merged: {G.number_of_nodes()} nodes, {G.number_of_edges()} edges')
|
||||
|
||||
# Write merged result back to .graphify_extract.json so Step 4 sees the full graph
|
||||
merged_out = {
|
||||
'nodes': [{'id': n, **d} for n, d in G.nodes(data=True)],
|
||||
'edges': [
|
||||
# Explicit source/target last so they win over any stale attrs in d.
|
||||
{**{k: val for k, val in d.items() if k not in ('_src', '_tgt', 'source', 'target')},
|
||||
'source': d.get('_src', u), 'target': d.get('_tgt', v)}
|
||||
for u, v, d in G.edges(data=True)
|
||||
],
|
||||
# G.graph["hyperedges"] holds hyperedges from both existing graph.json
|
||||
# and new_extraction (build_merge combines them). Falling back to
|
||||
# new_extraction only would silently drop prior-run hyperedges (#801).
|
||||
'hyperedges': list(G.graph.get('hyperedges', [])),
|
||||
'input_tokens': new_extraction.get('input_tokens', 0),
|
||||
'output_tokens': new_extraction.get('output_tokens', 0),
|
||||
}
|
||||
Path('graphify-out/.graphify_extract.json').write_text(json.dumps(merged_out, ensure_ascii=False), encoding=\"utf-8\")
|
||||
print(f'[graphify update] Merged extraction written ({len(merged_out[\"nodes\"])} nodes, {len(merged_out[\"edges\"])} edges)')
|
||||
|
||||
# Save manifest so next --update diffs against today's state, not the
|
||||
# prior run's baseline (prevents ghost-node reports on subsequent updates).
|
||||
save_manifest(incremental['files'])
|
||||
print('[graphify update] Manifest saved.')
|
||||
"
|
||||
```
|
||||
|
||||
Then run Steps 4–8 on the merged graph as normal.
|
||||
|
||||
After Step 4, show the graph diff:
|
||||
|
||||
```bash
|
||||
$(cat graphify-out/.graphify_python) -c "
|
||||
import json
|
||||
from graphify.analyze import graph_diff
|
||||
from graphify.build import build_from_json
|
||||
from networkx.readwrite import json_graph
|
||||
import networkx as nx
|
||||
from pathlib import Path
|
||||
|
||||
# Load old graph (before update) from backup written before merge
|
||||
old_data = json.loads(Path('graphify-out/.graphify_old.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_old.json').exists() else None
|
||||
new_extract = json.loads(Path('graphify-out/.graphify_extract.json').read_text(encoding=\"utf-8\"))
|
||||
G_new = build_from_json(new_extract)
|
||||
|
||||
if old_data:
|
||||
G_old = json_graph.node_link_graph(old_data, edges='links')
|
||||
diff = graph_diff(G_old, G_new)
|
||||
print(diff['summary'])
|
||||
if diff['new_nodes']:
|
||||
print('New nodes:', ', '.join(n['label'] for n in diff['new_nodes'][:5]))
|
||||
if diff['new_edges']:
|
||||
print('New edges:', len(diff['new_edges']))
|
||||
"
|
||||
```
|
||||
|
||||
Before the merge step, save the old graph: `cp graphify-out/graph.json graphify-out/.graphify_old.json`
|
||||
Clean up after: `rm -f graphify-out/.graphify_old.json`
|
||||
|
||||
---
|
||||
|
||||
## For --cluster-only
|
||||
|
||||
Skip Steps 1–3. Re-run clustering on the existing graph:
|
||||
|
||||
```bash
|
||||
graphify cluster-only .
|
||||
```
|
||||
|
||||
Then run Steps 5–9 as normal (label communities, generate viz, benchmark, clean up, report).
|
||||
@@ -0,0 +1,130 @@
|
||||
---
|
||||
name: harness-eval
|
||||
description: "评估功能的真实运行性能。从 Wiki 读取 schema 和 metric 定义，查询 SQLite 日志，对比基线，调用 LLM judge 做语义评估，产出诊断和迭代建议。触发短语: harness eval, 评估性能, 性能评估。"
|
||||
argument-hint: "[run_id]"
|
||||
---
|
||||
|
||||
# Harness Eval
|
||||
|
||||
## Overview
|
||||
|
||||
在真实环境中运行程序后，评估性能是否达标，诊断问题，决定是否迭代。
|
||||
|
||||
**核心原则：** 不是"通过就停"，而是基于真实指标判断"跑得好不好"。
|
||||
|
||||
## 触发方式
|
||||
|
||||
1. **自动**：subagent-driven-development Step 10，所有任务完成 + final review 通过后
|
||||
2. **手动**：`/harness-eval` 或 `/harness-eval <run_id>`
|
||||
|
||||
## 前置条件
|
||||
|
||||
- `research-wiki/schemas/` 中有至少一个 schema 实体
|
||||
- `research-wiki/metrics/` 中有至少一个 metric 实体
|
||||
- `results/harness.db` 存在且包含数据
|
||||
- `.env` 中有 LLM API key（用于 LlmJudge）
|
||||
|
||||
## 长时命令执行约定
|
||||
|
||||
对真实样本重跑流水线属于长时操作。超过 30s 的命令建议在 tmux 中后台运行，便于人工随时 attach 检查进度；短命令直接前台执行即可。
|
||||
|
||||
## 执行流程
|
||||
|
||||
### Phase 1: 准备
|
||||
|
||||
1. 列出 `research-wiki/schemas/` 下所有 schema 实体，获取表名和列信息
|
||||
2. 列出 `research-wiki/metrics/` 下所有 metric 实体，获取基线值和阈值
|
||||
3. 确认 `results/harness.db` 存在，查询 `_runs` 表获取最新运行状态
|
||||
4. 如果指定了 run_id，使用该 run_id；否则使用最近一次 status='completed' 的 run
|
||||
|
||||
### Phase 2: 运行
|
||||
|
||||
#### 2a: 检查已有运行记录
|
||||
|
||||
在 subagent-driven-development 流程中，某个子任务可能已经执行过全局运行并将结果写入了 `harness.db`。此时 harness-eval **不应重复运行**，直接复用已有记录即可。
|
||||
|
||||
1. 查询 `_runs` 表，查找 `status='completed'` 且 `git_sha` 与当前 `HEAD` 匹配的记录
|
||||
2. **找到匹配记录** → 使用该记录的 run_id，**跳过运行**，直接进入 Phase 3。在最终报告中注明「复用已有运行: run_id=xxx」
|
||||
3. **未找到匹配记录 + 自动触发** → 继续 2b
|
||||
4. **未找到匹配记录 + 手动触发** → 询问用户是否需要执行运行
|
||||
|
||||
#### 2b: 重跑流水线（仅无可用记录时）
|
||||
|
||||
1. `make deploy-testing` 起服务（api / worker / Postgres / Redis / 对象存储）
|
||||
2. 对真实样本重跑流水线，结果落 `results/harness.db`；属长时操作，建议按「长时命令执行约定」在 tmux 中运行
|
||||
3. 轮询检查 `_runs` 表状态，等待完成
|
||||
4. 确认 `_runs` 表中对应 run 的 status = 'completed'
|
||||
5. 如果 status = 'failed'，直接进入 Phase 6 诊断
|
||||
|
||||
### Phase 3: 硬性指标检查
|
||||
|
||||
对 `research-wiki/metrics/` 中每个标记为硬性判定的指标：
|
||||
|
||||
1. 从 metric 实体中读取：指标名、基线值、阈值、对应的 schema（表名和列名）
|
||||
2. 执行 SQL 查询获取当前运行的实际值
|
||||
3. 对比：实际值 vs 阈值
|
||||
4. 记录结果：pass / fail + 具体数值
|
||||
|
||||
### Phase 4: 语义评估
|
||||
|
||||
对 `research-wiki/metrics/` 中每个标记为语义判定的指标：
|
||||
|
||||
1. 收集 evidence：相关表的数据、_events 表的事件、配置信息
|
||||
2. 调用 `LlmJudge.evaluate(EvalRequest)`（请求体封装 criteria、evidence、rubric）
|
||||
3. 记录结果：Verdict
|
||||
|
||||
### Phase 5: 综合判定
|
||||
|
||||
- 硬性指标全部 pass **且** 语义评估全部 pass → **通过**
|
||||
- 任一不通过 → **不通过**，进入 Phase 6
|
||||
|
||||
### Phase 6: 诊断与迭代决策（仅不通过时）
|
||||
|
||||
1. 收集失败信息：哪些指标不达标、实际值与期望值的差距
|
||||
2. 查询 `_events` 表获取运行过程中的关键事件
|
||||
3. 调用 `LlmJudge.diagnose(error_context, log)`
|
||||
4. 向用户报告：
|
||||
- 问题根因分析
|
||||
- 修复建议
|
||||
- 是否建议再迭代一轮
|
||||
|
||||
### Phase 7: 记录到 Wiki
|
||||
|
||||
1. 创建 finding 实体：
|
||||
```bash
|
||||
conda run -n chs python3 .claude/tools/research_wiki.py add_entity research-wiki/ --type finding --id eval-<run_id> --title "Harness 评估: <run_id>"
|
||||
```
|
||||
2. 在 finding 中记录完整评估报告（硬性指标表 + 语义评估结果 + 诊断结论）
|
||||
3. 建立 edge：
|
||||
```bash
|
||||
conda run -n chs python3 .claude/tools/research_wiki.py add_edge research-wiki/ --from "finding:eval-<run_id>" --to "metric:<id>" --type evaluates --evidence "..."
|
||||
```
|
||||
4. 如果通过且用户确认，更新 metric 实体中的基线值为当前值
|
||||
5. 重建索引
|
||||
|
||||
## 输出格式
|
||||
|
||||
评估完成后，向用户输出结构化报告：
|
||||
|
||||
| 字段 | 内容 |
|
||||
|------|------|
|
||||
| 运行信息 | run_id, git_sha, 耗时 |
|
||||
| 硬性指标表 | 指标名, 基线, 当前, 阈值, 判定 |
|
||||
| 语义评估表 | 维度, 得分, 判定, 说明 |
|
||||
| 综合判定 | 通过/不通过 |
|
||||
| 诊断（如有） | 根因, 建议 |
|
||||
|
||||
## 与 subagent-driven-development 集成
|
||||
|
||||
在 SKILL.md 的 Step 9（final whole-implementation review）之后追加 Step 10：
|
||||
|
||||
```
|
||||
Step 10: Harness 评估
|
||||
调用 /harness-eval skill
|
||||
通过 → 进入 finishing-a-development-branch
|
||||
不通过 → 根据诊断结果，回到 Step 2 迭代相关 task
|
||||
```
|
||||
|
||||
## 评估范围
|
||||
|
||||
本 skill 当前覆盖①工程评估；②提取信度/③诊断效度的全量评估待 `app/` 流水线落地后接入。
|
||||
@@ -0,0 +1,108 @@
|
||||
---
|
||||
name: idea-creator
|
||||
description: "Generate and rank research ideas. GPT-5.4 + Claude dual independent brainstorming. Trigger phrases: find ideas, brainstorm ideas, generate research ideas."
|
||||
argument-hint: [research-direction]
|
||||
---
|
||||
|
||||
# Research Idea Generator
|
||||
Research direction: $ARGUMENTS
|
||||
|
||||
## Constants
|
||||
- `OUTPUT_DIR = idea-stage/`
|
||||
|
||||
## Workflow
|
||||
|
||||
### Phase 0: Load Wiki Context
|
||||
If `research-wiki/query_pack.md` exists and was updated less than 7 days ago:
|
||||
- Read it
|
||||
- Treat failed ideas as a banlist
|
||||
- Treat gaps as seeds
|
||||
- Treat top papers as known prior work
|
||||
|
||||
### Phase 1: Domain Survey
|
||||
If there is no `query_pack`, do a quick WebSearch first to build a broad domain context.
|
||||
|
||||
### Phase 2: Dual Independent Brainstorming
|
||||
**Critical requirement: brainstorm independently on two tracks first, then synthesize.**
|
||||
|
||||
GPT brainstorming (via Codex):
|
||||
- Use `/codex:rescue --fresh --background`
|
||||
- The prompt must ask GPT to generate 8-12 ideas
|
||||
- The prompt must include: the domain overview, known gaps, and resource constraints
|
||||
- Poll with `/codex:status`
|
||||
- Use `/codex:result` to collect the output
|
||||
|
||||
Claude brainstorming (via Agent):
|
||||
- Dispatch a subagent through the Agent tool with `subagent_type="general-purpose"`
|
||||
- Provide the same domain overview context
|
||||
- Require it to independently generate 8-12 ideas
|
||||
|
||||
### Phase 3: Claude Synthesis
|
||||
- Merge both idea sets and deduplicate by similarity
|
||||
- Cross-check: Claude challenges GPT ideas, GPT challenges Claude ideas
|
||||
- Score and rank everything in a single pass
|
||||
- Label each idea with its source: `GPT` / `Claude` / `consensus`
|
||||
|
||||
### Phase 4: First Filtering Pass
|
||||
Filtering criteria:
|
||||
- Feasibility: compute, data, and implementation cost
|
||||
- Fast novelty check: run 2-3 WebSearch passes for each idea
|
||||
- Impact: answer "so what?"
|
||||
- Remove ideas that are infeasible or too vague
|
||||
|
||||
### Phase 5: Deep Verification
|
||||
For the top ideas, call the `/novelty-check` skill.
|
||||
|
||||
### Phase 7: Output Report
|
||||
Write `idea-stage/IDEA_REPORT.md` in Chinese, with the following format:
|
||||
|
||||
```md
|
||||
# Research Idea Report
|
||||
**Direction**: ...
|
||||
**Date**: YYYY-MM-DD
|
||||
**Evaluation Result**: X generated -> Y survived -> W recommended
|
||||
|
||||
## Domain Overview Summary
|
||||
## Recommended Ideas (Ranked)
|
||||
### Idea 1: <title>
|
||||
- Hypothesis / minimal experiment / novelty / feasibility / risk / source
|
||||
|
||||
## Rejected Ideas
|
||||
```
|
||||
|
||||
### Phase 8: Wiki Integration
|
||||
- Check whether `research-wiki/` exists
|
||||
- For each idea:
|
||||
- `.claude/tools/research_wiki.py add_entity research-wiki/ --type idea --id <id> --title "..."`
|
||||
- Add edges:
|
||||
- `inspired_by`
|
||||
- `addresses_gap`
|
||||
- Rebuild `query_pack` and the index
|
||||
|
||||
## Key Rules
|
||||
- If the direction is too broad, STOP and ask the user to narrow it
|
||||
- Failed ideas must also be written into the wiki
|
||||
- Empirical signals matter more than theoretical attractiveness
|
||||
- `"Apply X to Y"` is the lowest-level research idea
|
||||
- If `research-wiki/` does not exist, still finish the report output but skip all wiki writes
|
||||
- First independent, then synthesize; first evidence, then judgment; first filter, then dig deeper
|
||||
|
||||
## Output Requirements
|
||||
When the task is complete, you must provide at least:
|
||||
- A domain overview summary
|
||||
- 8-12 raw ideas
|
||||
- A deduplicated candidate list
|
||||
- Filtering results and reasons
|
||||
- Deep verification results for the top ideas
|
||||
- `idea-stage/IDEA_REPORT.md`
|
||||
- Wiki write results, if applicable
|
||||
|
||||
## Completion Criteria
|
||||
Only finish when all of the following are complete:
|
||||
1. Wiki context loading or domain survey is complete
|
||||
2. Both independent brainstorming tracks are complete
|
||||
3. Merge, deduplication, and ranking are complete
|
||||
4. The first filtering pass is complete
|
||||
5. Deep verification of top ideas is complete
|
||||
6. `idea-stage/IDEA_REPORT.md` has been written
|
||||
7. `research-wiki/` integration is complete, if applicable
|
||||
@@ -0,0 +1,141 @@
|
||||
---
|
||||
name: novelty-check
|
||||
description: "Verify the novelty of research ideas. GPT cross-validation. Trigger phrases: novelty check, has anyone done this, check novelty."
|
||||
argument-hint: [method-or-idea-description]
|
||||
---
|
||||
|
||||
# Novelty Verification
|
||||
Verify novelty of: $ARGUMENTS
|
||||
|
||||
## Goal
|
||||
Perform a strict check on whether a method, idea, or experimental setting is actually new. The default stance is skepticism, not help-seeking for supporting evidence.
|
||||
|
||||
## Working Principles
|
||||
- Brutally honest: do not relax the standard just to make something look new.
|
||||
- `Applying X to Y` is not novel by default unless the application produces an unexpected mechanism, theoretical explanation, or clearly different experimental phenomenon.
|
||||
- Check the novelty of both the `METHOD` and the `EXPERIMENTAL SETTING`.
|
||||
- If the method itself is not new, but the findings, conclusions, experimental setup, or failure analysis are new, state that distinction explicitly.
|
||||
- Always search the last 6 months of arXiv.
|
||||
- Do not rely on titles alone; read the abstract and, when necessary, the key parts of related work, intro, method, and appendix.
|
||||
|
||||
## Workflow
|
||||
|
||||
### Phase A: Extract Core Claims
|
||||
First break the user's method description into 3-5 core technical claims. Each one should be as specific as possible.
|
||||
|
||||
For each claim, answer:
|
||||
- What is the method?
|
||||
- What problem does it solve?
|
||||
- What is the key mechanism?
|
||||
- What is the essential difference from an obvious baseline?
|
||||
|
||||
Rewrite the story-like description into searchable technical propositions and avoid vague phrasing.
|
||||
|
||||
### Phase B: Multi-source Literature Search
|
||||
Run multi-source retrieval for each claim, prioritizing recent work and similar settings.
|
||||
|
||||
For each claim, try at least 3 search-query sets, and make them complementary:
|
||||
- Direct technical terms
|
||||
- Synonyms / abbreviations / related task names
|
||||
- "Problem + mechanism" combinations
|
||||
- "Method + dataset / setting" combinations
|
||||
|
||||
#### Required Search Channels
|
||||
1. WebSearch: arXiv / Google Scholar / Semantic Scholar / conference homepages
|
||||
2. `python3 .claude/tools/arxiv_fetch.py search "QUERY" --max 10`
|
||||
3. `python3 .claude/tools/semantic_scholar_fetch.py search "QUERY" --max 10`
|
||||
4. `python3 .claude/tools/exa_search.py search "QUERY" --max 10` (if available)
|
||||
5. `python3 .claude/tools/openalex_fetch.py search "QUERY" --max 10` (if available)
|
||||
|
||||
#### Search Priorities
|
||||
- ICLR / NeurIPS / ICML 2025-2026
|
||||
- arXiv preprints from the last 6 months
|
||||
- Papers close to the method mechanism, not only papers on the same task
|
||||
- Papers close to the experimental setting, not only papers using the same method
|
||||
|
||||
#### Decision Strategy
|
||||
- Record potentially overlapping papers first; do not exclude them too early
|
||||
- Prefer reading the abstract, intro, related work, and method sections
|
||||
- If overlap looks suspicious, also read the experimental setup and appendix
|
||||
|
||||
#### Recording Requirements
|
||||
For each candidate paper, record:
|
||||
- Title
|
||||
- Year
|
||||
- Venue / status
|
||||
- Relevant point
|
||||
- The specific reason it may overlap
|
||||
- Why it might still be a different work
|
||||
|
||||
If a data source is unavailable, explicitly record the fallback reason and continue with the others; do not stop the task.
|
||||
|
||||
### Phase C: GPT Cross-Validation
|
||||
Send the method description from Phase A and all candidate papers found in Phase B to `/codex:rescue --fresh --wait` for a second review.
|
||||
|
||||
The cross-validation prompt must include:
|
||||
- proposed method description
|
||||
- the full candidate paper list
|
||||
- ask:
|
||||
- `Is this method novel?`
|
||||
- `What is the closest prior work?`
|
||||
- `What is the delta?`
|
||||
|
||||
Use high reasoning effort.
|
||||
|
||||
The goal of cross-validation is not to find even more papers. It is to force out the closest prior art, the smallest difference, and the risk of pseudo-novelty.
|
||||
|
||||
### Phase D: Output Report + Wiki Integration
|
||||
The output must be in English and follow a fixed structure.
|
||||
|
||||
#### Report Format
|
||||
```markdown
|
||||
## Novelty Check Report
|
||||
### Method Under Review
|
||||
### Core Claims
|
||||
- Claim 1: ... (novelty: high / medium / low; closest paper: ...)
|
||||
- Claim 2: ... (novelty: high / medium / low; closest paper: ...)
|
||||
|
||||
### Recent Prior Work
|
||||
| Paper | Year | Venue / Status | Overlap Point | Key Difference |
|
||||
|---|---:|---|---|---|
|
||||
|
||||
### Overall Assessment
|
||||
- score X/10
|
||||
- recommendation: continue / continue cautiously / abandon
|
||||
- key differentiator: ...
|
||||
- positioning advice: ...
|
||||
```
|
||||
|
||||
#### Evaluation Scale
|
||||
- `high`: current search shows no close prior art, and the difference is concrete and technical
|
||||
- `medium`: there is related prior work, but there is still a clear and defensible technical delta
|
||||
- `low`: mostly a reorganization of known methods, task switching, dataset switching, hyperparameter changes, or standard engineering changes
|
||||
|
||||
#### Wiki Integration
|
||||
If the project has `research-wiki/`, also ingest the knowledge there:
|
||||
- Create a `claim` entity for each core claim
|
||||
- Create a `paper` entity for each newly found paper
|
||||
- Add claim-paper / paper-paper relation edges
|
||||
- Rebuild `query_pack`
|
||||
|
||||
Prefer existing tools such as `.claude/tools/research_wiki.py`; if the wiki does not exist, skip silently and do not error.
|
||||
|
||||
Ingestion rules:
|
||||
- Only write high-confidence information
|
||||
- Claim names should be short, stable, and reusable
|
||||
- Edges must include evidence; do not create empty links
|
||||
|
||||
## Completion Criteria
|
||||
Only finish when all of the following are complete:
|
||||
1. 3-5 core claims have been extracted
|
||||
2. Multi-source search has been completed, including the last 6 months of arXiv
|
||||
3. Candidate paper abstracts have been read, and related work / method sections were read when necessary
|
||||
4. GPT cross-validation has been completed
|
||||
5. A structured English report has been produced
|
||||
6. If `research-wiki/` exists, the corresponding writes and `query_pack` rebuild have been completed
|
||||
|
||||
## Failure Handling
|
||||
- Missing tools: record the missing item and degrade gracefully
|
||||
- Too few search results: expand synonyms, abbreviations, higher-level terms, and experimental settings
|
||||
- Too many search results: prefer the most recent, most similar, and most likely overlapping work
|
||||
- Conflicting evidence: read the original abstract and method sections first; do not rely on intuition
|
||||
@@ -0,0 +1,229 @@
|
||||
---
|
||||
name: receiving-code-review
|
||||
description: Use when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementation
|
||||
---
|
||||
|
||||
# Code Review Reception
|
||||
|
||||
## Overview
|
||||
|
||||
Code review requires technical evaluation, not emotional performance.
|
||||
|
||||
**Core principle:** Verify before implementing. Ask before assuming. Technical correctness over social comfort.
|
||||
|
||||
## The Response Pattern
|
||||
|
||||
```
|
||||
WHEN receiving code review feedback:
|
||||
|
||||
1. READ: Complete feedback without reacting
|
||||
2. UNDERSTAND: Restate requirement in own words (or ask)
|
||||
3. VERIFY: Check against codebase reality
|
||||
4. EVALUATE: Technically sound for THIS codebase?
|
||||
5. RESPOND: Technical acknowledgment or reasoned pushback
|
||||
6. IMPLEMENT: One item at a time, test each
|
||||
```
|
||||
|
||||
## Forbidden Responses
|
||||
|
||||
**NEVER:**
|
||||
- "You're absolutely right!" (explicit CLAUDE.md violation)
|
||||
- "Great point!" / "Excellent feedback!" (performative)
|
||||
- "Let me implement that now" (before verification)
|
||||
|
||||
**INSTEAD:**
|
||||
- Restate the technical requirement
|
||||
- Ask clarifying questions
|
||||
- Push back with technical reasoning if wrong
|
||||
- Just start working (actions > words)
|
||||
|
||||
## Handling Unclear Feedback
|
||||
|
||||
```
|
||||
IF any item is unclear:
|
||||
STOP - do not implement anything yet
|
||||
ASK for clarification on unclear items
|
||||
|
||||
WHY: Items may be related. Partial understanding = wrong implementation.
|
||||
```
|
||||
|
||||
**Example:**
|
||||
```
|
||||
your human partner: "Fix 1-6"
|
||||
You understand 1,2,3,6. Unclear on 4,5.
|
||||
|
||||
❌ WRONG: Implement 1,2,3,6 now, ask about 4,5 later
|
||||
✅ RIGHT: "I understand items 1,2,3,6. Need clarification on 4 and 5 before proceeding."
|
||||
```
|
||||
|
||||
## Source-Specific Handling
|
||||
|
||||
### From your human partner
|
||||
- **Trusted** - implement after understanding
|
||||
- **Still ask** if scope unclear
|
||||
- **No performative agreement**
|
||||
- **Skip to action** or technical acknowledgment
|
||||
|
||||
### From External Reviewers
|
||||
```
|
||||
BEFORE implementing:
|
||||
1. Check: Technically correct for THIS codebase?
|
||||
2. Check: Breaks existing functionality?
|
||||
3. Check: Reason for current implementation?
|
||||
4. Check: Works on all platforms/versions?
|
||||
5. Check: Does reviewer understand full context?
|
||||
|
||||
IF suggestion seems wrong:
|
||||
Push back with technical reasoning
|
||||
|
||||
IF can't easily verify:
|
||||
Say so: "I can't verify this without [X]. Should I [investigate/ask/proceed]?"
|
||||
|
||||
IF conflicts with your human partner's prior decisions:
|
||||
Stop and discuss with your human partner first
|
||||
```
|
||||
|
||||
**your human partner's rule:** "External feedback - be skeptical, but check carefully"
|
||||
|
||||
## YAGNI Check for "Professional" Features
|
||||
|
||||
```
|
||||
IF reviewer suggests "implementing properly":
|
||||
grep codebase for actual usage
|
||||
|
||||
IF unused: "This endpoint isn't called. Remove it (YAGNI)?"
|
||||
IF used: Then implement properly
|
||||
```
|
||||
|
||||
**your human partner's rule:** "You and reviewer both report to me. If we don't need this feature, don't add it."
|
||||
|
||||
## Implementation Order
|
||||
|
||||
```
|
||||
FOR multi-item feedback:
|
||||
1. Clarify anything unclear FIRST
|
||||
2. Then implement in this order:
|
||||
- Blocking issues (breaks, security)
|
||||
- Simple fixes (typos, imports)
|
||||
- Complex fixes (refactoring, logic)
|
||||
3. Test each fix individually
|
||||
4. Verify no regressions
|
||||
```
|
||||
|
||||
## When To Push Back
|
||||
|
||||
Push back when:
|
||||
- Suggestion breaks existing functionality
|
||||
- Reviewer lacks full context
|
||||
- Violates YAGNI (unused feature)
|
||||
- Technically incorrect for this stack
|
||||
- Legacy/compatibility reasons exist
|
||||
- Conflicts with your human partner's architectural decisions
|
||||
|
||||
**How to push back:**
|
||||
- Use technical reasoning, not defensiveness
|
||||
- Ask specific questions
|
||||
- Reference working tests/code
|
||||
- Involve your human partner if architectural
|
||||
|
||||
**Signal if uncomfortable pushing back out loud:** "Strange things are afoot at the Circle K"
|
||||
|
||||
## Acknowledging Correct Feedback
|
||||
|
||||
When feedback IS correct:
|
||||
```
|
||||
✅ "Fixed. [Brief description of what changed]"
|
||||
✅ "Good catch - [specific issue]. Fixed in [location]."
|
||||
✅ [Just fix it and show in the code]
|
||||
|
||||
❌ "You're absolutely right!"
|
||||
❌ "Great point!"
|
||||
❌ "Thanks for catching that!"
|
||||
❌ "Thanks for [anything]"
|
||||
❌ ANY gratitude expression
|
||||
```
|
||||
|
||||
**Why no thanks:** Actions speak. Just fix it. The code itself shows you heard the feedback.
|
||||
|
||||
**If you catch yourself about to write "Thanks":** DELETE IT. State the fix instead.
|
||||
|
||||
## Gracefully Correcting Your Pushback
|
||||
|
||||
If you pushed back and were wrong:
|
||||
```
|
||||
✅ "You were right - I checked [X] and it does [Y]. Implementing now."
|
||||
✅ "Verified this and you're correct. My initial understanding was wrong because [reason]. Fixing."
|
||||
|
||||
❌ Long apology
|
||||
❌ Defending why you pushed back
|
||||
❌ Over-explaining
|
||||
```
|
||||
|
||||
State the correction factually and move on.
|
||||
|
||||
## Common Mistakes
|
||||
|
||||
| Mistake | Fix |
|
||||
|---------|-----|
|
||||
| Performative agreement | State requirement or just act |
|
||||
| Blind implementation | Verify against codebase first |
|
||||
| Batch without testing | One at a time, test each |
|
||||
| Assuming reviewer is right | Check if breaks things |
|
||||
| Avoiding pushback | Technical correctness > comfort |
|
||||
| Partial implementation | Clarify all items first |
|
||||
| Can't verify, proceed anyway | State limitation, ask for direction |
|
||||
|
||||
## Real Examples
|
||||
|
||||
**Performative Agreement (Bad):**
|
||||
```
|
||||
Reviewer: "Remove legacy code"
|
||||
❌ "You're absolutely right! Let me remove that..."
|
||||
```
|
||||
|
||||
**Technical Verification (Good):**
|
||||
```
|
||||
Reviewer: "Remove legacy code"
|
||||
✅ "Checking... build target is 10.15+, this API needs 13+. Need legacy for backward compat. Current impl has wrong bundle ID - fix it or drop pre-13 support?"
|
||||
```
|
||||
|
||||
**YAGNI (Good):**
|
||||
```
|
||||
Reviewer: "Implement proper metrics tracking with database, date filters, CSV export"
|
||||
✅ "Grepped codebase - nothing calls this endpoint. Remove it (YAGNI)? Or is there usage I'm missing?"
|
||||
```
|
||||
|
||||
**Unclear Item (Good):**
|
||||
```
|
||||
your human partner: "Fix items 1-6"
|
||||
You understand 1,2,3,6. Unclear on 4,5.
|
||||
✅ "Understand 1,2,3,6. Need clarification on 4 and 5 before implementing."
|
||||
```
|
||||
|
||||
## GitHub Thread Replies
|
||||
|
||||
When replying to inline review comments on GitHub, reply in the comment thread (`gh api repos/{owner}/{repo}/pulls/{pr}/comments/{id}/replies`), not as a top-level PR comment.
|
||||
|
||||
## The Bottom Line
|
||||
|
||||
**External feedback = suggestions to evaluate, not orders to follow.**
|
||||
|
||||
Verify. Question. Then implement.
|
||||
|
||||
No performative agreement. Technical rigor always.
|
||||
|
||||
## Wiki Integration
|
||||
|
||||
**Precondition**: `research-wiki/` directory exists (skip this section entirely if it does not).
|
||||
|
||||
**Trigger**: When review feedback is received and you decide whether to accept or reject it.
|
||||
|
||||
**Type**: `review`
|
||||
|
||||
**Severity**: `approved` / `needs_changes` / `rejected`
|
||||
|
||||
**Steps**:
|
||||
1. Run `.claude/tools/research_wiki.py add_entity research-wiki/ --type review --id <slug> --title "<review response title>"` to create the review entity
|
||||
2. Append the accepted or rejected suggestions, the reason, and the follow-up actions to the generated page
|
||||
3. If the response changes a design or plan, run `.claude/tools/research_wiki.py add_edge research-wiki/ --from "review:<id>" --to "<target-type>:<id>" --type informs --evidence "..."`
|
||||
4. Run `.claude/tools/research_wiki.py rebuild_index research-wiki/`
|
||||
@@ -0,0 +1,119 @@
|
||||
---
|
||||
name: requesting-code-review
|
||||
description: Use when completing tasks, implementing major features, or before merging to verify work meets requirements
|
||||
---
|
||||
|
||||
# Requesting Code Review
|
||||
|
||||
Dispatch Codex as a read-only reviewer to catch issues before they cascade. The reviewer gets precisely crafted context for evaluation — never your session's history. This keeps the reviewer focused on the work product, not your thought process, and preserves your own context for continued work.
|
||||
|
||||
**Core principle:** Review early, review often.
|
||||
|
||||
## When to Request Review
|
||||
|
||||
**Mandatory:**
|
||||
- After each task in subagent-driven development
|
||||
- After completing major feature
|
||||
- Before merge to main
|
||||
|
||||
**Optional but valuable:**
|
||||
- When stuck (fresh perspective)
|
||||
- Before refactoring (baseline check)
|
||||
- After fixing complex bug
|
||||
|
||||
## How to Request
|
||||
|
||||
**1. Get git SHAs:**
|
||||
```bash
|
||||
BASE_SHA=$(git rev-parse HEAD~1) # or origin/main
|
||||
HEAD_SHA=$(git rev-parse HEAD)
|
||||
```
|
||||
|
||||
**2. Dispatch Codex as reviewer:**
|
||||
|
||||
用 `/codex:rescue --fresh --wait` 把 `code-reviewer.md` 模板（含 `{BASE_SHA}`/`{HEAD_SHA}` 占位与 `git diff` 指令）交 Codex 只读审查。
|
||||
|
||||
**Placeholders:**
|
||||
- `{DESCRIPTION}` - Brief summary of what you built
|
||||
- `{PLAN_OR_REQUIREMENTS}` - What it should do
|
||||
- `{BASE_SHA}` - Starting commit
|
||||
- `{HEAD_SHA}` - Ending commit
|
||||
|
||||
**3. Act on feedback:**
|
||||
- Fix Critical issues immediately
|
||||
- Fix Important issues before proceeding
|
||||
- Note Minor issues for later
|
||||
- Push back if reviewer is wrong (with reasoning)
|
||||
|
||||
## Example
|
||||
|
||||
```
|
||||
[Just completed Task 2: Add verification function]
|
||||
|
||||
You: Let me request code review before proceeding.
|
||||
|
||||
BASE_SHA=$(git log --oneline | grep "Task 1" | head -1 | awk '{print $1}')
|
||||
HEAD_SHA=$(git rev-parse HEAD)
|
||||
|
||||
[Dispatch Codex as reviewer via /codex:rescue --fresh --wait]
|
||||
DESCRIPTION: Added verifyIndex() and repairIndex() with 4 issue types
|
||||
PLAN_OR_REQUIREMENTS: Task 2 from research-wiki/plans/deployment-plan.md
|
||||
BASE_SHA: a7981ec
|
||||
HEAD_SHA: 3df7661
|
||||
|
||||
[Codex returns]:
|
||||
Strengths: Clean architecture, real tests
|
||||
Issues:
|
||||
Important: Missing progress indicators
|
||||
Minor: Magic number (100) for reporting interval
|
||||
Assessment: Ready to proceed
|
||||
|
||||
You: [Fix progress indicators]
|
||||
[Continue to Task 3]
|
||||
```
|
||||
|
||||
## Integration with Workflows
|
||||
|
||||
**Subagent-Driven Development:**
|
||||
- Review after EACH task
|
||||
- Catch issues before they compound
|
||||
- Fix before moving to next task
|
||||
|
||||
**Executing Plans:**
|
||||
- Review after each task or at natural checkpoints
|
||||
- Get feedback, apply, continue
|
||||
|
||||
**Ad-Hoc Development:**
|
||||
- Review before merge
|
||||
- Review when stuck
|
||||
|
||||
## Red Flags
|
||||
|
||||
**Never:**
|
||||
- Skip review because "it's simple"
|
||||
- Ignore Critical issues
|
||||
- Proceed with unfixed Important issues
|
||||
- Argue with valid technical feedback
|
||||
|
||||
**If reviewer wrong:**
|
||||
- Push back with technical reasoning
|
||||
- Show code/tests that prove it works
|
||||
- Request clarification
|
||||
|
||||
See template at: requesting-code-review/code-reviewer.md
|
||||
|
||||
## Wiki Integration
|
||||
|
||||
**Precondition**: `research-wiki/` directory exists (skip this section entirely if it does not).
|
||||
|
||||
**Trigger**: When requesting a code review or when a clear review result has been received.
|
||||
|
||||
**Type**: `review`
|
||||
|
||||
**Severity**: `approved` / `needs_changes` / `rejected`
|
||||
|
||||
**Steps**:
|
||||
1. Run `.claude/tools/research_wiki.py add_entity research-wiki/ --type review --id <slug> --title "<review title>"` to create the review entity
|
||||
2. Append the review conclusion, key issues, recommendations, and whether it passed to the generated page
|
||||
3. If the review recommendations change a design or plan, run `.claude/tools/research_wiki.py add_edge research-wiki/ --from "review:<id>" --to "<target-type>:<id>" --type informs --evidence "..."`
|
||||
4. Run `.claude/tools/research_wiki.py rebuild_index research-wiki/`
|
||||
@@ -0,0 +1,175 @@
|
||||
# Code Reviewer Prompt Template
|
||||
|
||||
Use this template when dispatching Codex as reviewer.
|
||||
|
||||
**Purpose:** Review completed work against requirements and code quality standards before it cascades into more work.
|
||||
|
||||
```
|
||||
/codex:rescue --fresh --wait (read-only review):
|
||||
description: "Review code changes"
|
||||
prompt: |
|
||||
You are a Senior Code Reviewer with expertise in software architecture,
|
||||
design patterns, and best practices. Your job is to review completed work
|
||||
against its plan or requirements and identify issues before they cascade.
|
||||
|
||||
## What Was Implemented
|
||||
|
||||
{DESCRIPTION}
|
||||
|
||||
## Requirements / Plan
|
||||
|
||||
{PLAN_OR_REQUIREMENTS}
|
||||
|
||||
## Git Range to Review
|
||||
|
||||
**Base:** {BASE_SHA}
|
||||
**Head:** {HEAD_SHA}
|
||||
|
||||
```bash
|
||||
git diff --stat {BASE_SHA}..{HEAD_SHA}
|
||||
git diff {BASE_SHA}..{HEAD_SHA}
|
||||
```
|
||||
|
||||
## What to Check
|
||||
|
||||
**Plan alignment:**
|
||||
- Does the implementation match the plan / requirements?
|
||||
- Are deviations justified improvements, or problematic departures?
|
||||
- Is all planned functionality present?
|
||||
|
||||
**Code quality:**
|
||||
- Clean separation of concerns?
|
||||
- Proper error handling?
|
||||
- Type safety where applicable?
|
||||
- DRY without premature abstraction?
|
||||
- Edge cases handled?
|
||||
|
||||
**Architecture:**
|
||||
- Sound design decisions?
|
||||
- Reasonable scalability and performance?
|
||||
- Security concerns?
|
||||
- Integrates cleanly with surrounding code?
|
||||
|
||||
**Testing:**
|
||||
- Tests verify real behavior, not mocks?
|
||||
- Edge cases covered?
|
||||
- Integration tests where they matter?
|
||||
- All tests passing?
|
||||
|
||||
**Production readiness:**
|
||||
- Migration strategy if schema changed?
|
||||
- Backward compatibility considered?
|
||||
- Documentation complete?
|
||||
- No obvious bugs?
|
||||
|
||||
**Runtime data & logging:**
|
||||
- Are SqliteRunStore calls present at all critical code paths?
|
||||
- Do log tables capture enough data for post-run diagnosis?
|
||||
- Query Wiki schemas/ to understand which tables this code affects
|
||||
- Query SQLite to check if metrics regressed after these changes
|
||||
- Are log_event calls placed at error boundaries and state transitions?
|
||||
|
||||
## Calibration
|
||||
|
||||
Categorize issues by actual severity. Not everything is Critical.
|
||||
Acknowledge what was done well before listing issues — accurate praise
|
||||
helps the implementer trust the rest of the feedback.
|
||||
|
||||
If you find significant deviations from the plan, flag them specifically
|
||||
so the implementer can confirm whether the deviation was intentional.
|
||||
If you find issues with the plan itself rather than the implementation,
|
||||
say so.
|
||||
|
||||
## Output Format
|
||||
|
||||
### Strengths
|
||||
[What's well done? Be specific.]
|
||||
|
||||
### Issues
|
||||
|
||||
#### Critical (Must Fix)
|
||||
[Bugs, security issues, data loss risks, broken functionality]
|
||||
|
||||
#### Important (Should Fix)
|
||||
[Architecture problems, missing features, poor error handling, test gaps]
|
||||
|
||||
#### Minor (Nice to Have)
|
||||
[Code style, optimization opportunities, documentation polish]
|
||||
|
||||
For each issue:
|
||||
- File:line reference
|
||||
- What's wrong
|
||||
- Why it matters
|
||||
- How to fix (if not obvious)
|
||||
|
||||
### Recommendations
|
||||
[Improvements for code quality, architecture, or process]
|
||||
|
||||
### Assessment
|
||||
|
||||
**Ready to merge?** [Yes | No | With fixes]
|
||||
|
||||
**Reasoning:** [1-2 sentence technical assessment]
|
||||
|
||||
## Critical Rules
|
||||
|
||||
**DO:**
|
||||
- Categorize by actual severity
|
||||
- Be specific (file:line, not vague)
|
||||
- Explain WHY each issue matters
|
||||
- Acknowledge strengths
|
||||
- Give a clear verdict
|
||||
|
||||
**DON'T:**
|
||||
- Say "looks good" without checking
|
||||
- Mark nitpicks as Critical
|
||||
- Give feedback on code you didn't actually read
|
||||
- Be vague ("improve error handling")
|
||||
- Avoid giving a clear verdict
|
||||
```
|
||||
|
||||
**Placeholders:**
|
||||
- `{DESCRIPTION}` — brief summary of what was built
|
||||
- `{PLAN_OR_REQUIREMENTS}` — what it should do (plan file path, task text, or requirements)
|
||||
- `{BASE_SHA}` — starting commit
|
||||
- `{HEAD_SHA}` — ending commit
|
||||
|
||||
**Reviewer returns:** Strengths, Issues (Critical / Important / Minor), Recommendations, Assessment
|
||||
|
||||
## Example Output
|
||||
|
||||
```
|
||||
### Strengths
|
||||
- Clean database schema with proper migrations (db.ts:15-42)
|
||||
- Comprehensive test coverage (18 tests, all edge cases)
|
||||
- Good error handling with fallbacks (summarizer.ts:85-92)
|
||||
|
||||
### Issues
|
||||
|
||||
#### Important
|
||||
1. **Missing help text in CLI wrapper**
|
||||
- File: index-conversations:1-31
|
||||
- Issue: No --help flag, users won't discover --concurrency
|
||||
- Fix: Add --help case with usage examples
|
||||
|
||||
2. **Date validation missing**
|
||||
- File: search.ts:25-27
|
||||
- Issue: Invalid dates silently return no results
|
||||
- Fix: Validate ISO format, throw error with example
|
||||
|
||||
#### Minor
|
||||
1. **Progress indicators**
|
||||
- File: indexer.ts:130
|
||||
- Issue: No "X of Y" counter for long operations
|
||||
- Impact: Users don't know how long to wait
|
||||
|
||||
### Recommendations
|
||||
- Add progress reporting for user experience
|
||||
- Consider config file for excluded projects (portability)
|
||||
|
||||
### Assessment
|
||||
|
||||
**Ready to merge: With fixes**
|
||||
|
||||
**Reasoning:** Core implementation is solid with good architecture and tests. Important issues (help text, date validation) are easily fixed and don't affect core functionality.
|
||||
```
|
||||
@@ -0,0 +1,213 @@
|
||||
---
|
||||
name: research-lit
|
||||
description: "Search and analyze research papers, literature reviews, and related work. Trigger phrases: find papers, literature review, related work, literature review."
|
||||
argument-hint: [research-topic]
|
||||
---
|
||||
|
||||
# Literature Review
|
||||
Research topic: $ARGUMENTS
|
||||
|
||||
## Constants
|
||||
- `PAPER_LIBRARY = references/` (local PDF directory)
|
||||
- `MAX_LOCAL_PAPERS = 20`
|
||||
|
||||
## Data Sources (all enabled by default)
|
||||
|
||||
| Data source | How to determine availability | What it provides | Fallback behavior |
|
||||
|---|---|---|---|
|
||||
| Local PDF | `references/` exists and `references/**/*.pdf` is non-empty | Papers, reports, appendices, and drafts collected in the repo | Read only locally; if there are no PDFs, continue with online search |
|
||||
| Web search | Network is available and general search results are accessible | Google Scholar / paper pages / arXiv / conference homepages / survey blogs | If search fails, skip that source and keep results from others |
|
||||
| arXiv API | `python3 .claude/tools/arxiv_fetch.py` is runnable and the network is available | arXiv metadata, abstracts, IDs, categories, versions | Skip if unavailable; prefer arXiv records from other sources |
|
||||
| Semantic Scholar | `python3 .claude/tools/semantic_scholar_fetch.py` is runnable and the network is available | Paper metadata, venue, citation, author, and citation relationships | Skip if unavailable; if an arXiv paper has S2 venue metadata, prefer S2 |
|
||||
| Exa | `python3 .claude/tools/exa_search.py` is runnable, the network is available, and API config exists | Semantic search, page highlight excerpts, related paper page clues | Skip if unavailable; use results only as supplemental clues |
|
||||
| OpenAlex | `python3 .claude/tools/openalex_fetch.py` is runnable and the network is available | Open scholarly graph, DOI, venue, year, and citation relationships | Skip if unavailable; use it to fill in DOI / venue / author information |
|
||||
| DeepXiv | `python3 .claude/tools/deepxiv_fetch.py` is runnable and the network is available | Semantic search and paper aggregation results focused on arXiv | Skip if unavailable; cross-check against arXiv / S2 |
|
||||
|
||||
### Coverage Control
|
||||
All data sources are enabled by default. If the user includes the instruction:
|
||||
|
||||
`— sources: <list>`
|
||||
|
||||
then only the sources listed in `<list>` are used. Parsing rules:
|
||||
|
||||
- `<list>` is comma-separated and may use Chinese or English names, such as `local PDF, arXiv, Semantic Scholar`
|
||||
- Keep only recognizable names; ignore unknown items
|
||||
- If parsing yields nothing, fall back to the default of enabling everything
|
||||
- Either local-only reading or online-only searching is allowed; do not stop because some sources are unavailable
|
||||
|
||||
## Workflow
|
||||
|
||||
### Step 0: Scan local PDFs
|
||||
|
||||
Scan the local library first, then decide where to focus online search.
|
||||
|
||||
1. Glob: `references/**/*.pdf`
|
||||
2. Filter by relevance to the research topic, prioritizing PDFs whose title, abstract, first chapter, or conclusion matches the topic
|
||||
3. For up to `MAX_LOCAL_PAPERS` relevant PDFs, read the first 3 pages
|
||||
4. Record the title, authors, year, venue, method keywords, and relevance for each paper
|
||||
|
||||
Recommended reading command:
|
||||
|
||||
```bash
|
||||
python3 tools/read_pdf_pages.py references/path/to/paper.pdf --pages 1-3
|
||||
```
|
||||
|
||||
If the repository does not include that script, use any available PDF-reading tool or existing command. The rule is to read only the first 3 pages and avoid blind full-document reading.
|
||||
|
||||
### Step 1: Online search
|
||||
|
||||
Run retrieval for each enabled data source. Commands should be structured as closely as possible to the following forms. `QUERY` should be replaced with a search string built around the research topic, including the task noun, core method, aliases, synonyms, and common abbreviations.
|
||||
|
||||
- `python3 .claude/tools/arxiv_fetch.py search "QUERY" --max 10`
|
||||
- `python3 .claude/tools/semantic_scholar_fetch.py search "QUERY" --max 10`
|
||||
- `python3 .claude/tools/exa_search.py search "QUERY" --max 10 --category "research paper" --content highlights`
|
||||
- `python3 .claude/tools/openalex_fetch.py search "QUERY" --max 10 --year "2022-"`
|
||||
- `python3 .claude/tools/deepxiv_fetch.py search "QUERY" --max 10`
|
||||
- WebSearch for Google Scholar / the general web
|
||||
|
||||
Search strategy:
|
||||
|
||||
1. Start with broad queries to identify the main direction
|
||||
2. Use narrower queries to find recent work, SOTA, benchmarks, surveys, and ablations from the last two years
|
||||
3. Add controversy points, failure cases, negative results, and competing methods
|
||||
4. Record raw results from each source; do not drop borderline candidates too early
|
||||
|
||||
### Step 2: Cross-source deduplication
|
||||
|
||||
Deduplicate in this order:
|
||||
|
||||
1. `arXiv ID`
|
||||
2. `DOI`
|
||||
3. normalized title
|
||||
|
||||
Rules:
|
||||
|
||||
- Normalize titles by lowercasing, removing punctuation, removing extra spaces, and removing version suffixes
|
||||
- If the same paper has different metadata across sources, keep the record with the most complete information
|
||||
- If `S2` (Semantic Scholar) provides venue, year, author, or citation metadata for an arXiv paper, prefer those fields from `S2`
|
||||
- Distinguish between preprints and formally published versions; if both correspond to the same research, record the relationship but ultimately prefer the more authoritative published version
|
||||
|
||||
### Step 3: Analyze each paper
|
||||
|
||||
For each retained paper, extract:
|
||||
|
||||
- `problem/gap`: what problem is being solved and what existing methods are missing
|
||||
- `method`: core idea, model, training / inference flow, and key tricks
|
||||
- `key results`: main experimental findings, metrics, baselines, and datasets
|
||||
- `relevance to our work`: the direct connection to the current research topic and what can be borrowed
|
||||
- `source`: which source or sources the paper came from and whether metadata conflicts exist
|
||||
|
||||
Requirements:
|
||||
|
||||
- Every paper must include author, year, and venue
|
||||
- Explicitly mark `preprint`, `conference paper`, `journal paper`, `workshop`, and similar statuses
|
||||
- If author / year / venue is uncertain, state the source of uncertainty and do not fabricate it
|
||||
|
||||
### Step 4: Synthesis
|
||||
|
||||
Cluster papers by method route or theme instead of sorting only by time.
|
||||
|
||||
The synthesis must answer:
|
||||
|
||||
- Which method routes have become the consensus
|
||||
- Where the important disagreements are
|
||||
- Which conclusions hold only for specific datasets or settings
|
||||
- What gaps remain unsolved
|
||||
- Which results matter most for our research, and why
|
||||
|
||||
Prioritize:
|
||||
|
||||
- Work from the last 2 years
|
||||
- Representative methods and accepted baselines for the direction
|
||||
- Ablations, diagnostics, and failure analyses that explain performance differences
|
||||
|
||||
If the topic is foundational or classical, trace back to the original work, but keep the newest work as the main line.
|
||||
|
||||
### Step 5: Output
|
||||
|
||||
The final output must include both:
|
||||
|
||||
1. A structured literature table
|
||||
2. A 3-5 paragraph narrative summary
|
||||
|
||||
Suggested table columns:
|
||||
|
||||
- Paper
|
||||
- Authors / Year / Venue
|
||||
- Source
|
||||
- Problem / Gap
|
||||
- Method
|
||||
- Key Results
|
||||
- Relevance to Our Work
|
||||
|
||||
The narrative should:
|
||||
|
||||
- Summarize the sub-branches of the topic first, then the consensus and disagreements
|
||||
- Clearly call out the 3-5 papers worth following next
|
||||
- Clearly identify 1-3 gaps that can become future research entry points
|
||||
|
||||
### Step 6: Wiki Integration
|
||||
|
||||
First check whether `research-wiki/` exists; if it does not, skip all writes without error.
|
||||
|
||||
If it does exist, then:
|
||||
|
||||
1. Ingest the top 8-12 papers into the wiki
|
||||
|
||||
```bash
|
||||
.claude/tools/research_wiki.py ingest_paper research-wiki/ --arxiv-id <id> [--title "..." --authors "..." --year ...]
|
||||
```
|
||||
|
||||
2. Create entities for identified gaps
|
||||
|
||||
```bash
|
||||
.claude/tools/research_wiki.py add_entity research-wiki/ --type gap --id <slug> --title "..."
|
||||
```
|
||||
|
||||
3. Add paper relationship edges
|
||||
|
||||
```bash
|
||||
.claude/tools/research_wiki.py add_edge research-wiki/ --from "paper:X" --to "paper:Y" --type extends --evidence "..."
|
||||
```
|
||||
|
||||
4. Rebuild the query pack and index
|
||||
|
||||
```bash
|
||||
.claude/tools/research_wiki.py rebuild_query_pack research-wiki/ && .claude/tools/research_wiki.py rebuild_index research-wiki/
|
||||
```
|
||||
|
||||
Integration rules:
|
||||
|
||||
- Only write papers with high confidence and highest topic relevance into the wiki
|
||||
- Gap node names should be short and stable so they can be reused later
|
||||
- Relationship edges must include evidence; do not create vague connections
|
||||
|
||||
## Key Rules
|
||||
|
||||
- Always cite papers: author, year, and venue are required
|
||||
- Distinguish peer-reviewed papers from preprints
|
||||
- Missing tools, missing APIs, or missing network access must degrade gracefully; never stop because one source fails
|
||||
- Focus on the last 2 years by default; only trace back further when the task is about foundational work
|
||||
- Do not just list papers; summarize the thread, disagreements, and gaps
|
||||
- If the query is too narrow, first expand with synonyms, abbreviations, and higher-level terms, then narrow again to a precise sub-direction
|
||||
- If there are too many results, keep representative, highly cited, newer, stronger-experiment, and closest-to-topic papers first
|
||||
- If there are too few results, broaden the query and rescan relevant PDFs in the local library
|
||||
|
||||
## Execution Standard
|
||||
|
||||
- Local first, then online
|
||||
- Deduplicate first, then analyze
|
||||
- Evidence first, then conclusions
|
||||
- Cluster first, then narrate
|
||||
- Integrate the wiki first, then finish
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
Only finish the task when all of the following are complete:
|
||||
|
||||
1. Local PDF scanning is complete
|
||||
2. At least the default-enabled data sources were covered, or fallback reasons were recorded
|
||||
3. Deduplication is complete
|
||||
4. Every core paper has been analyzed
|
||||
5. A table and narrative summary were produced
|
||||
6. If `research-wiki/` exists, the corresponding writes and rebuilds were completed
|
||||
@@ -0,0 +1,89 @@
|
||||
---
|
||||
name: research-wiki
|
||||
description: "Manage the project research knowledge base. Initialization, querying, statistics, and health checks. Trigger phrases: knowledge base, research wiki, wiki query, check the knowledge base."
|
||||
argument-hint: [subcommand: init|query|stats|lint|ingest]
|
||||
---
|
||||
|
||||
# Research Wiki Management
|
||||
|
||||
Manage the project research knowledge base: $ARGUMENTS
|
||||
|
||||
## Overview
|
||||
|
||||
Research Wiki is the project's single source of truth for knowledge. All research and development knowledge produced by skills is stored here.
|
||||
|
||||
### Entity Types (12)
|
||||
|
||||
| Entity | Directory | node_id Format | Primary Source Skill |
|
||||
|---|---|---|---|
|
||||
| paper | `papers/` | `paper:<slug>` | `research-lit` |
|
||||
| idea | `ideas/` | `idea:<id>` | `idea-creator` |
|
||||
| experiment | `experiments/` | `exp:<id>` | `run-experiment` (future) |
|
||||
| claim | `claims/` | `claim:<id>` | `novelty-check` |
|
||||
| gap | `gaps/` | `gap:<id>` | `research-lit` / `idea-creator` |
|
||||
| design | `designs/` | `design:<id>` | `brainstorming` |
|
||||
| finding | `findings/` | `finding:<id>` | `systematic-debugging` / `verification-before-completion` |
|
||||
| adr | `adrs/` | `adr:<id>` | 架构决策记录 |
|
||||
| plan | `plans/` | `plan:<id>` | `writing-plans` |
|
||||
| review | `reviews/` | `review:<id>` | `requesting-code-review` / `receiving-code-review` |
|
||||
| schema | `schemas/` | `schema:<id>` | `structured-logging` |
|
||||
| metric | `metrics/` | `metric:<id>` | `structured-logging` / `harness-eval` |
|
||||
|
||||
### Key Files
|
||||
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `query_pack.md` | Compressed summary |
|
||||
| `index.md` | Category index |
|
||||
| `log.md` | Audit log |
|
||||
| `graph/edges.json` | Relationship graph |
|
||||
|
||||
## Subcommands
|
||||
|
||||
### `/research-wiki init`
|
||||
|
||||
Initialize the wiki:
|
||||
|
||||
```bash
|
||||
.claude/tools/research_wiki.py init research-wiki/
|
||||
```
|
||||
|
||||
### `/research-wiki query "<topic>"`
|
||||
|
||||
Rebuild `query_pack.md`:
|
||||
|
||||
```bash
|
||||
.claude/tools/research_wiki.py rebuild_query_pack research-wiki/
|
||||
```
|
||||
|
||||
Then show the content to the user.
|
||||
|
||||
### `/research-wiki stats`
|
||||
|
||||
Show statistics:
|
||||
|
||||
```bash
|
||||
.claude/tools/research_wiki.py stats research-wiki/
|
||||
```
|
||||
|
||||
### `/research-wiki lint`
|
||||
|
||||
Run health checks:
|
||||
|
||||
```bash
|
||||
.claude/tools/research_wiki.py lint research-wiki/
|
||||
```
|
||||
|
||||
### `/research-wiki ingest "<title>" — arxiv: <id>`
|
||||
|
||||
Import a paper:
|
||||
|
||||
```bash
|
||||
.claude/tools/research_wiki.py ingest_paper research-wiki/ --arxiv-id <id>
|
||||
```
|
||||
|
||||
## Key Rules
|
||||
|
||||
- Other skills call `.claude/tools/research_wiki.py` directly; they do not go through this skill.
|
||||
- If the wiki does not exist (`research-wiki/` directory is missing), all write operations are skipped silently.
|
||||
- Failed ideas are always preserved in `query_pack`.
|
||||
@@ -0,0 +1,102 @@
|
||||
---
|
||||
name: structured-logging
|
||||
description: "设计结构化日志方案。在功能会产生运行时数据时，brainstorming 之后、writing-plans 之前必须调用。设计表结构、定义基线指标、注册到 Wiki。"
|
||||
argument-hint: "[功能描述]"
|
||||
---
|
||||
|
||||
# Structured Logging
|
||||
|
||||
## Overview
|
||||
|
||||
为即将开发的功能设计结构化日志方案。确定需要记录什么数据、如何记录、如何评估。
|
||||
|
||||
**核心原则：** 没有日志方案就不能开始编码。所有会产生运行时数据的功能，必须先设计日志再写实现。
|
||||
|
||||
## 触发时机
|
||||
|
||||
SOP Phase 1 中，brainstorming 产出 design 之后、writing-plans 之前。
|
||||
|
||||
## 前置条件
|
||||
|
||||
- `research-wiki/` 目录存在
|
||||
- `core/eval/` 存在（SqliteRunStore 库）
|
||||
- brainstorming 已产出 design 实体
|
||||
|
||||
## 执行流程
|
||||
|
||||
### Phase 1: 读取现有 Schema
|
||||
|
||||
1. 列出 `research-wiki/schemas/` 下所有已有 schema 实体
|
||||
2. 如果 `results/harness.db` 存在，查询 `SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE '_%'` 获取实际表列表
|
||||
3. 对比 Wiki 记录与实际表结构，标记不一致之处
|
||||
|
||||
### Phase 2: 设计日志方案
|
||||
|
||||
基于 brainstorming 产出的 design，回答以下问题：
|
||||
|
||||
1. **这个功能会产生什么运行时数据？**
|
||||
- OCR 置信度（每张影像/字段的识别置信度）
|
||||
- 提取信度一致率（多次提取/多专家的一致性）
|
||||
- 阶段事件（开始/结束/错误/警告）
|
||||
- 门控结果（期望张数、完整性校验）
|
||||
- token 用量与时延（LLM 调用成本与延迟）
|
||||
|
||||
2. **需要新建表还是复用现有表？**
|
||||
- 如果现有表能覆盖 → 复用，不新建
|
||||
- 如果需要新维度 → 新建表
|
||||
|
||||
3. **每张表的 schema 是什么？**
|
||||
- 列名、类型、说明
|
||||
- 主键（如果有）
|
||||
- 哪些列用于 agent 查询诊断
|
||||
|
||||
4. **在代码哪些位置埋点？**
|
||||
- 具体函数名、具体代码节点
|
||||
- 用 `log.insert()` 还是 `log.log_event()`
|
||||
|
||||
### Phase 3: 定义评估基线
|
||||
|
||||
1. **硬性指标**：列出每个可量化的指标、阈值、判定方式
|
||||
2. **语义指标**：列出需要 LlmJudge 评估的维度
|
||||
3. **基线来源**：指明对比哪个历史 run_id（如果是首次，标记为"待首次运行后建立"）
|
||||
|
||||
### Phase 4: 注册到 Wiki
|
||||
|
||||
1. 对每张新表，执行：
|
||||
```bash
|
||||
python3 .claude/tools/research_wiki.py add_entity research-wiki/ --type schema --id <table-name> --title "表结构: <table-name>"
|
||||
```
|
||||
然后在生成的 md 文件中填入完整的列定义和埋点位置。
|
||||
|
||||
2. 对每组基线指标，执行：
|
||||
```bash
|
||||
python3 .claude/tools/research_wiki.py add_entity research-wiki/ --type metric --id <metric-name> --title "<指标描述>"
|
||||
```
|
||||
然后在生成的 md 文件中填入基线值、阈值、判定方式。
|
||||
|
||||
3. 建立关联：
|
||||
```bash
|
||||
python3 .claude/tools/research_wiki.py add_edge research-wiki/ --from "metric:<id>" --to "schema:<id>" --type measures --evidence "..."
|
||||
python3 .claude/tools/research_wiki.py add_edge research-wiki/ --from "schema:<id>" --to "design:<id>" --type implements --evidence "..."
|
||||
```
|
||||
|
||||
4. 重建索引：
|
||||
```bash
|
||||
python3 .claude/tools/research_wiki.py rebuild_index research-wiki/
|
||||
```
|
||||
|
||||
### Phase 5: 产出
|
||||
|
||||
交给 writing-plans 的输入：
|
||||
- 哪些文件需要 `from core.eval.run_store import SqliteRunStore`
|
||||
- 哪些函数需要在什么位置调用 `log.insert()` / `log.log_event()`
|
||||
- 这些埋点必须作为 writing-plans 中的显式步骤，不得遗漏
|
||||
|
||||
## 产出物清单
|
||||
|
||||
| 产出 | 位置 | 说明 |
|
||||
|------|------|------|
|
||||
| schema 实体 | `research-wiki/schemas/<name>.md` | 表结构 + 列说明 + 埋点位置 |
|
||||
| metric 实体 | `research-wiki/metrics/<name>.md` | 基线值 + 阈值 + 判定方式 |
|
||||
| edge 关系 | `research-wiki/graph/edges.json` | schema↔metric↔design 关联 |
|
||||
| 日志埋点清单 | 传递给 writing-plans | 具体文件和函数的埋点要求 |
|
||||
@@ -0,0 +1,104 @@
|
||||
# Code Quality Reviewer Prompt Template
|
||||
|
||||
Use this template when running the code quality review as a read-only Codex session (`/codex:rescue --fresh --wait`). Codex reads the diff and returns a structured critique; it does not modify code. Only run this **after** spec compliance review has approved the work.
|
||||
|
||||
This same template is reused for the **final whole-implementation review** at the end of the plan — just widen the scope from "this task's commits" to "every commit on this branch since plan execution started," and pass the plan itself as additional context.
|
||||
|
||||
---
|
||||
|
||||
```
|
||||
Codex review (read-only) — pass this as the /codex:rescue --fresh --wait prompt body:
|
||||
|
||||
description: "Review code quality for Task N" # or "Final whole-implementation review"
|
||||
prompt: |
|
||||
You are reviewing the code quality of work that has already passed spec compliance review. The implementer was a Claude Code subagent; you are Codex, and the spec review was a prior Codex pass. You are running read-only — read the diff and report; do not edit code.
|
||||
|
||||
## What Was Built
|
||||
|
||||
[Brief recap of the task — one paragraph. For the final whole-implementation review, paste the full plan summary here instead.]
|
||||
|
||||
## Working Directory
|
||||
|
||||
[Absolute path of the worktree.]
|
||||
|
||||
## Commits to Review
|
||||
|
||||
[Space-separated list of commit SHAs. For per-task review, just this task's SHAs. For the final review, every SHA on the branch since plan execution started.]
|
||||
|
||||
## CRITICAL: Read the Code
|
||||
|
||||
Do not rely on commit messages or the implementer's notes. Read `git show <sha>` for each commit. Read the surrounding files for context. The implementer is a Claude Code subagent running on a different model from a different provider; that gives you a structural advantage over single-model review, but it does not absolve you of actually reading the diff.
|
||||
|
||||
## Your Job
|
||||
|
||||
Evaluate the implementation against these dimensions. Spec compliance is already verified — do not re-litigate it.
|
||||
|
||||
**Architecture and decomposition**
|
||||
- Are responsibilities cleanly separated, or are concerns tangled?
|
||||
- Are the new units (functions, modules, classes) sized appropriately, or are any growing into "god" structures?
|
||||
- Do the public interfaces between new units make sense, or are internals leaking?
|
||||
|
||||
**Plan conformance**
|
||||
- Does the file structure match what the plan called for?
|
||||
- Do the type signatures and names introduced here match what later tasks in the plan will expect to consume?
|
||||
|
||||
**Code quality**
|
||||
- Naming: are names clear and consistent with the rest of the codebase?
|
||||
- Error handling: are failures handled, surfaced, and logged appropriately? Any swallowed exceptions or silent fallbacks?
|
||||
- Magic numbers / strings: are constants extracted and named?
|
||||
- Duplication: is there copy-pasted logic that should be a helper?
|
||||
- Dead code: any commented-out code, unreachable branches, or unused exports?
|
||||
- Readability: can a new contributor follow the flow without help?
|
||||
|
||||
**Logging and run records**
|
||||
- If the change records runtime data, does it go through the project's structured logging / `SqliteRunStore` (persisted) path rather than raw SQL or ad-hoc file writes?
|
||||
- Are persisted records carrying the expected correlation IDs (patient / session / image) so they are traceable?
|
||||
|
||||
**Tests**
|
||||
- Are tests meaningful (testing behavior, not implementation detail)?
|
||||
- Are edge cases covered, or is it all happy path?
|
||||
- Are there flaky patterns (timing, ordering, hidden global state)?
|
||||
- Test names: do they describe behavior, not internals?
|
||||
|
||||
**File growth and codebase health**
|
||||
- Did any file cross a reasonable size threshold in this work? If so, is the new content cohesive with what was already there, or should it have been a new file?
|
||||
- Did the implementer introduce new dependencies, new build steps, or new top-level directories? Justified?
|
||||
|
||||
## Calibration
|
||||
|
||||
Only flag issues that matter. Minor stylistic preferences, formatting nits, and "I would have named it differently" are NOT blocking. Categorize what you find:
|
||||
|
||||
- **Critical** — Will cause real bugs, security issues, data corruption, or break a downstream task in the plan. Must be fixed.
|
||||
- **Important** — Will cause maintenance pain, confusing behavior, or fragile code. Should be fixed.
|
||||
- **Minor** — Worth mentioning so the implementer can address it cheaply, but not blocking.
|
||||
|
||||
Approval requires zero Critical and zero Important issues. Minor issues are advisory.
|
||||
|
||||
## Report Format
|
||||
|
||||
Return your review in this structure:
|
||||
|
||||
```
|
||||
Verdict: <APPROVED | CHANGES_REQUESTED>
|
||||
|
||||
Strengths:
|
||||
- <bullet per genuine strength — be honest, brief, and specific. If there are no real strengths, say so.>
|
||||
|
||||
Issues:
|
||||
Critical:
|
||||
- <bullet per critical issue with file:line reference and what's wrong>
|
||||
- or "none"
|
||||
Important:
|
||||
- <bullet per important issue with file:line reference>
|
||||
- or "none"
|
||||
Minor:
|
||||
- <bullet per minor issue with file:line reference>
|
||||
- or "none"
|
||||
|
||||
Assessment:
|
||||
<One short paragraph: overall quality, whether this is solid for downstream tasks, anything cross-cutting the controller should know.>
|
||||
```
|
||||
|
||||
If Verdict is `APPROVED`, the controller will mark the task complete.
|
||||
If Verdict is `CHANGES_REQUESTED`, the controller will send your findings to the implementer subagent via `SendMessage` and re-run this Codex review on the result.
|
||||
```
|
||||
@@ -0,0 +1,114 @@
|
||||
# Code Style Reviewer Prompt Template
|
||||
|
||||
Use this template when running the code style review as a read-only Codex session (`/codex:rescue --fresh --wait`). Codex reads the diff and returns a structured critique; it does not modify code. Only run this **after** spec compliance review has approved the work.
|
||||
|
||||
This reviewer focuses on code **form and style** — conciseness, abstraction level, naming quality, comment appropriateness. It complements (does not replace) the code-quality-reviewer which focuses on architecture and functional correctness.
|
||||
|
||||
---
|
||||
|
||||
```
|
||||
Codex review (read-only) — pass this as the /codex:rescue --fresh --wait prompt body:
|
||||
|
||||
description: "Review code style for Task N"
|
||||
prompt: |
|
||||
You are reviewing the code style and form of work that has already passed spec compliance review. The implementer was a Claude Code subagent; you are Codex, running read-only. Your job is to catch over-engineering, unnecessary complexity, and style violations that automated tools cannot detect — read the diff and report; do not edit code.
|
||||
|
||||
## What Was Built
|
||||
|
||||
[Brief recap of the task — one paragraph.]
|
||||
|
||||
## Working Directory
|
||||
|
||||
[Absolute path of the worktree.]
|
||||
|
||||
## Commits to Review
|
||||
|
||||
[Space-separated list of commit SHAs.]
|
||||
|
||||
## Project Coding Principles (ranked by priority)
|
||||
|
||||
These are the project's non-negotiable coding principles. Violations are issues.
|
||||
|
||||
1. **YAGNI** — No code that isn't needed right now. No "just in case" parameters, no unused abstractions, no speculative generality.
|
||||
2. **Readability** — Domain-term naming (not implementation naming). Comments explain WHY, never WHAT. If code needs a WHAT comment, the code is unclear — refactor it.
|
||||
3. **Single Responsibility** — Each function does one thing. If you need "and" to describe it, split it.
|
||||
4. **Explicit over Implicit** — Type annotations on all public functions. No hidden global state access. No default parameters masking logic.
|
||||
5. **Defensiveness** — No bare `except:` or `except Exception: pass`. No swallowed errors. No default values masking failures.
|
||||
6. **Simplicity** — A function is better than a class. A flat structure is better than nested. Three clear lines beat one clever one-liner.
|
||||
|
||||
## Your Job
|
||||
|
||||
Read the actual code via `git show <sha>` and file reads. Evaluate against these dimensions:
|
||||
|
||||
**Over-engineering / Unnecessary Abstraction**
|
||||
- Are there classes that could be plain functions?
|
||||
- Are there inheritance hierarchies that could be composition or just data?
|
||||
- Are there factory patterns, builder patterns, or strategy patterns where a simple if/else would suffice?
|
||||
- Is there unnecessary indirection (wrapper functions that just forward to another function)?
|
||||
|
||||
**YAGNI Violations**
|
||||
- Parameters or config options that nothing currently uses?
|
||||
- "Extension points" or "plugin systems" for hypothetical future needs?
|
||||
- Generic solutions where a specific one would be simpler and sufficient?
|
||||
|
||||
**Naming and Clarity**
|
||||
- Are names domain-specific or generic-technical? (`DiagnosisEngine` good, `DataProcessorFactory` bad)
|
||||
- Can you understand what a function does from its name alone?
|
||||
- Are variable names descriptive or abbreviated?
|
||||
|
||||
**Comment Quality**
|
||||
- Missing WHY comments on non-obvious decisions?
|
||||
- Present but useless WHAT comments? (e.g., `# increment counter` above `counter += 1`)
|
||||
- Outdated comments that contradict the code?
|
||||
|
||||
**Conciseness**
|
||||
- Could any 10-line block be expressed in 3 lines without losing clarity?
|
||||
- Are there repeated patterns that should be extracted (but only if used 3+ times)?
|
||||
- Is there dead code, commented-out code, or unreachable branches?
|
||||
|
||||
**Type Annotations**
|
||||
- Do all public functions have complete parameter and return type annotations?
|
||||
- Are complex types properly aliased for readability?
|
||||
|
||||
## Calibration
|
||||
|
||||
**This is NOT a linting review.** Automated tools already checked formatting and lint rules. Focus exclusively on things that require judgment:
|
||||
- Is this the SIMPLEST way to solve this problem?
|
||||
- Would a reader understand this without context?
|
||||
- Does the abstraction level match the problem complexity?
|
||||
|
||||
Only flag issues that a competent developer would agree represent unnecessary complexity or unclear code. Style preferences that don't affect readability are NOT issues.
|
||||
|
||||
Severity levels:
|
||||
- **Important** — Genuine over-engineering, YAGNI violation, or significantly unclear code. Should be simplified.
|
||||
- **Minor** — Could be slightly simpler or clearer, but not blocking.
|
||||
|
||||
No "Critical" category here — that belongs to the functional quality reviewer.
|
||||
|
||||
## Report Format
|
||||
|
||||
```
|
||||
Verdict: <APPROVED | CHANGES_REQUESTED>
|
||||
|
||||
Over-engineering:
|
||||
- <bullet with file:line, what's over-engineered, simpler alternative>
|
||||
- or "none"
|
||||
|
||||
YAGNI violations:
|
||||
- <bullet with file:line, what's unneeded>
|
||||
- or "none"
|
||||
|
||||
Clarity issues:
|
||||
- <bullet with file:line, what's unclear and how to fix>
|
||||
- or "none"
|
||||
|
||||
Style notes (advisory):
|
||||
- <minor observations, not blocking>
|
||||
|
||||
Assessment:
|
||||
<One sentence: is this code as simple as it could be while remaining correct?>
|
||||
```
|
||||
|
||||
If Verdict is `APPROVED`, the controller will mark the task complete.
|
||||
If Verdict is `CHANGES_REQUESTED`, the controller will send your findings to the implementer subagent via `SendMessage`.
|
||||
```
|
||||
@@ -0,0 +1,103 @@
|
||||
# Spec Compliance Reviewer Prompt Template
|
||||
|
||||
Use this template when running the spec compliance review as a read-only Codex session (`/codex:rescue --fresh --wait`). Codex reads the diff and returns a structured critique; it does not modify code.
|
||||
|
||||
**Purpose:** Verify the Claude implementer subagent built what was requested — nothing more, nothing less.
|
||||
|
||||
---
|
||||
|
||||
```
|
||||
Codex review (read-only) — pass this as the /codex:rescue --fresh --wait prompt body:
|
||||
|
||||
description: "Review spec compliance for Task N"
|
||||
prompt: |
|
||||
You are reviewing whether a Claude Code implementer subagent's work matches its specification. You are Codex, running read-only — read the diff and report; do not edit code.
|
||||
|
||||
## What Was Requested
|
||||
|
||||
[FULL TEXT of task requirements — paste verbatim from the plan]
|
||||
|
||||
## What the Implementer Claims They Built
|
||||
|
||||
[Paste the implementer subagent's full final report here, including the STATUS block, COMMIT_SHAS, FILES_CHANGED, and NOTES.]
|
||||
|
||||
## Working Directory
|
||||
|
||||
[Absolute path of the worktree.]
|
||||
|
||||
## Commits to Review
|
||||
|
||||
[Space-separated list of commit SHAs from the implementer's report. Base your analysis on these specific commits, not on the entire repo history.]
|
||||
|
||||
## CRITICAL: Do Not Trust the Report
|
||||
|
||||
The implementer is a Claude Code subagent; you are Codex, a different model from a different provider. Its self-report may be incomplete, inaccurate, or optimistic. Cross-provider does not mean trustworthy — it means the report has not yet been independently verified. You MUST verify everything independently by reading the actual diff.
|
||||
|
||||
**DO NOT:**
|
||||
- Take the implementer's word for what it implemented
|
||||
- Trust its claims about completeness
|
||||
- Accept its interpretation of requirements
|
||||
|
||||
**DO:**
|
||||
- Read the actual code via `git show <sha>` and direct file reads
|
||||
- Compare actual implementation to requirements line by line
|
||||
- Check for missing pieces it claimed to implement
|
||||
- Look for extra features it didn't mention
|
||||
|
||||
## Your Job
|
||||
|
||||
Read the implementation code and verify:
|
||||
|
||||
**Missing requirements**
|
||||
- Did the implementer implement everything that was requested?
|
||||
- Are there requirements they skipped or missed?
|
||||
- Did they claim something works but not actually implement it?
|
||||
- For every acceptance criterion in the spec, can you point to the specific lines that satisfy it?
|
||||
|
||||
**Extra / unrequested work**
|
||||
- Did they build things that weren't requested?
|
||||
- Did they over-engineer or add unnecessary features?
|
||||
- Did they add "nice to haves" that weren't in the spec?
|
||||
- Are there new files, flags, or dependencies that the spec didn't ask for?
|
||||
|
||||
**Misunderstandings**
|
||||
- Did they interpret requirements differently than intended?
|
||||
- Did they solve the wrong problem?
|
||||
- Did they implement the right feature in the wrong way (e.g., correct behavior but wrong interface, wrong file location, or wrong type signatures that won't compose with later tasks)?
|
||||
|
||||
**Tests**
|
||||
- Are the tests actually testing what the spec requires, or just exercising whatever code exists?
|
||||
- Did they delete or weaken existing tests to make things pass?
|
||||
|
||||
Verify by reading code and tests, not by trusting the report.
|
||||
|
||||
## Calibration
|
||||
|
||||
Only flag issues that would cause real problems during implementation or in the next task. Minor wording, stylistic preferences, and formatting quibbles are NOT spec-compliance issues — those belong to the code quality reviewer. Stay in your lane: did they build what was asked for, or didn't they.
|
||||
|
||||
## Report Format
|
||||
|
||||
Return your review in this structure:
|
||||
|
||||
```
|
||||
Verdict: <APPROVED | CHANGES_REQUESTED>
|
||||
|
||||
Missing requirements:
|
||||
- <bullet per missing requirement, with the spec quote it violates>
|
||||
- or "none"
|
||||
|
||||
Extra/unrequested work:
|
||||
- <bullet per extraneous addition>
|
||||
- or "none"
|
||||
|
||||
Misunderstandings:
|
||||
- <bullet per misunderstanding, with what the spec said vs. what was built>
|
||||
- or "none"
|
||||
|
||||
Notes:
|
||||
- <anything else the controller should know, but not blocking>
|
||||
```
|
||||
|
||||
If Verdict is `APPROVED`, the controller will proceed to code quality review.
|
||||
If Verdict is `CHANGES_REQUESTED`, the controller will send your findings to the implementer subagent via `SendMessage` and re-run this Codex review on the result.
|
||||
```
|
||||
@@ -0,0 +1,119 @@
|
||||
# Creation Log: Systematic Debugging Skill
|
||||
|
||||
Reference example of extracting, structuring, and bulletproofing a critical skill.
|
||||
|
||||
## Source Material
|
||||
|
||||
Extracted debugging framework from `~/.claude/CLAUDE.md`:
|
||||
- 4-phase systematic process (Investigation → Pattern Analysis → Hypothesis → Implementation)
|
||||
- Core mandate: ALWAYS find root cause, NEVER fix symptoms
|
||||
- Rules designed to resist time pressure and rationalization
|
||||
|
||||
## Extraction Decisions
|
||||
|
||||
**What to include:**
|
||||
- Complete 4-phase framework with all rules
|
||||
- Anti-shortcuts ("NEVER fix symptom", "STOP and re-analyze")
|
||||
- Pressure-resistant language ("even if faster", "even if I seem in a hurry")
|
||||
- Concrete steps for each phase
|
||||
|
||||
**What to leave out:**
|
||||
- Project-specific context
|
||||
- Repetitive variations of same rule
|
||||
- Narrative explanations (condensed to principles)
|
||||
|
||||
## Structure Following skill-creation/SKILL.md
|
||||
|
||||
1. **Rich when_to_use** - Included symptoms and anti-patterns
|
||||
2. **Type: technique** - Concrete process with steps
|
||||
3. **Keywords** - "root cause", "symptom", "workaround", "debugging", "investigation"
|
||||
4. **Flowchart** - Decision point for "fix failed" → re-analyze vs add more fixes
|
||||
5. **Phase-by-phase breakdown** - Scannable checklist format
|
||||
6. **Anti-patterns section** - What NOT to do (critical for this skill)
|
||||
|
||||
## Bulletproofing Elements
|
||||
|
||||
Framework designed to resist rationalization under pressure:
|
||||
|
||||
### Language Choices
|
||||
- "ALWAYS" / "NEVER" (not "should" / "try to")
|
||||
- "even if faster" / "even if I seem in a hurry"
|
||||
- "STOP and re-analyze" (explicit pause)
|
||||
- "Don't skip past" (catches the actual behavior)
|
||||
|
||||
### Structural Defenses
|
||||
- **Phase 1 required** - Can't skip to implementation
|
||||
- **Single hypothesis rule** - Forces thinking, prevents shotgun fixes
|
||||
- **Explicit failure mode** - "IF your first fix doesn't work" with mandatory action
|
||||
- **Anti-patterns section** - Shows exactly what shortcuts look like
|
||||
|
||||
### Redundancy
|
||||
- Root cause mandate in overview + when_to_use + Phase 1 + implementation rules
|
||||
- "NEVER fix symptom" appears 4 times in different contexts
|
||||
- Each phase has explicit "don't skip" guidance
|
||||
|
||||
## Testing Approach
|
||||
|
||||
Created 4 validation tests following skills/meta/testing-skills-with-subagents:
|
||||
|
||||
### Test 1: Academic Context (No Pressure)
|
||||
- Simple bug, no time pressure
|
||||
- **Result:** Perfect compliance, complete investigation
|
||||
|
||||
### Test 2: Time Pressure + Obvious Quick Fix
|
||||
- User "in a hurry", symptom fix looks easy
|
||||
- **Result:** Resisted shortcut, followed full process, found real root cause
|
||||
|
||||
### Test 3: Complex System + Uncertainty
|
||||
- Multi-layer failure, unclear if can find root cause
|
||||
- **Result:** Systematic investigation, traced through all layers, found source
|
||||
|
||||
### Test 4: Failed First Fix
|
||||
- Hypothesis doesn't work, temptation to add more fixes
|
||||
- **Result:** Stopped, re-analyzed, formed new hypothesis (no shotgun)
|
||||
|
||||
**All tests passed.** No rationalizations found.
|
||||
|
||||
## Iterations
|
||||
|
||||
### Initial Version
|
||||
- Complete 4-phase framework
|
||||
- Anti-patterns section
|
||||
- Flowchart for "fix failed" decision
|
||||
|
||||
### Enhancement 1: TDD Reference
|
||||
- Added link to skills/testing/test-driven-development
|
||||
- Note explaining TDD's "simplest code" ≠ debugging's "root cause"
|
||||
- Prevents confusion between methodologies
|
||||
|
||||
## Final Outcome
|
||||
|
||||
Bulletproof skill that:
|
||||
- ✅ Clearly mandates root cause investigation
|
||||
- ✅ Resists time pressure rationalization
|
||||
- ✅ Provides concrete steps for each phase
|
||||
- ✅ Shows anti-patterns explicitly
|
||||
- ✅ Tested under multiple pressure scenarios
|
||||
- ✅ Clarifies relationship to TDD
|
||||
- ✅ Ready for use
|
||||
|
||||
## Key Insight
|
||||
|
||||
**Most important bulletproofing:** Anti-patterns section showing exact shortcuts that feel justified in the moment. When Claude thinks "I'll just add this one quick fix", seeing that exact pattern listed as wrong creates cognitive friction.
|
||||
|
||||
## Usage Example
|
||||
|
||||
When encountering a bug:
|
||||
1. Load skill: skills/debugging/systematic-debugging
|
||||
2. Read overview (10 sec) - reminded of mandate
|
||||
3. Follow Phase 1 checklist - forced investigation
|
||||
4. If tempted to skip - see anti-pattern, stop
|
||||
5. Complete all phases - root cause found
|
||||
|
||||
**Time investment:** 5-10 minutes
|
||||
**Time saved:** Hours of symptom-whack-a-mole
|
||||
|
||||
---
|
||||
|
||||
*Created: 2025-10-03*
|
||||
*Purpose: Reference example for skill extraction and bulletproofing*
|
||||
@@ -0,0 +1,342 @@
|
||||
---
|
||||
name: systematic-debugging
|
||||
description: Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes
|
||||
---
|
||||
|
||||
# Systematic Debugging
|
||||
|
||||
## Overview
|
||||
|
||||
Random fixes waste time and create new bugs. Quick patches mask underlying issues.
|
||||
|
||||
**Core principle:** ALWAYS find root cause before attempting fixes. Symptom fixes are failure.
|
||||
|
||||
**Violating the letter of this process is violating the spirit of debugging.**
|
||||
|
||||
## The Iron Law
|
||||
|
||||
```
|
||||
NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST
|
||||
```
|
||||
|
||||
If you haven't completed Phase 1, you cannot propose fixes.
|
||||
|
||||
## When to Use
|
||||
|
||||
Use for ANY technical issue:
|
||||
- Test failures
|
||||
- Bugs in production
|
||||
- Unexpected behavior
|
||||
- Performance problems
|
||||
- Build failures
|
||||
- Integration issues
|
||||
|
||||
**Use this ESPECIALLY when:**
|
||||
- Under time pressure (emergencies make guessing tempting)
|
||||
- "Just one quick fix" seems obvious
|
||||
- You've already tried multiple fixes
|
||||
- Previous fix didn't work
|
||||
- You don't fully understand the issue
|
||||
|
||||
**Don't skip when:**
|
||||
- Issue seems simple (simple bugs have root causes too)
|
||||
- You're in a hurry (rushing guarantees rework)
|
||||
- Manager wants it fixed NOW (systematic is faster than thrashing)
|
||||
|
||||
## The Five Phases
|
||||
|
||||
You MUST complete each phase before proceeding to the next.
|
||||
|
||||
### Phase 1: Root Cause Investigation
|
||||
|
||||
**BEFORE attempting ANY fix:**
|
||||
|
||||
1. **Read Error Messages Carefully**
|
||||
- Don't skip past errors or warnings
|
||||
- They often contain the exact solution
|
||||
- Read stack traces completely
|
||||
- Note line numbers, file paths, error codes
|
||||
|
||||
2. **Reproduce Consistently**
|
||||
- Can you trigger it reliably?
|
||||
- What are the exact steps?
|
||||
- Does it happen every time?
|
||||
- If not reproducible → gather more data, don't guess
|
||||
|
||||
3. **Check Recent Changes**
|
||||
- What changed that could cause this?
|
||||
- Git diff, recent commits
|
||||
- New dependencies, config changes
|
||||
- Environmental differences
|
||||
|
||||
**代码检索（graphify）**：定位前先 `/graphify . --update` 刷新图谱，再用 graphify 追调用关系与影响面，而非盲读文件：
|
||||
- `conda run -n chs graphify path "<A>" "<B>"` 追依赖/调用链
|
||||
- `conda run -n chs graphify affected "<出问题的节点>"` 反向影响面（改它会波及谁）
|
||||
- `conda run -n chs graphify explain "<节点>"` 看某节点及邻居
|
||||
- 图谱未建（`graph file not found`）则提示人类先 `/graphify .`，未建图前降级常规阅读、不阻断。
|
||||
|
||||
4. **Gather Evidence in Multi-Component Systems**
|
||||
|
||||
**WHEN system has multiple components (CI → build → signing, API → service → database):**
|
||||
|
||||
**BEFORE proposing fixes, add diagnostic instrumentation:**
|
||||
```
|
||||
For EACH component boundary:
|
||||
- Log what data enters component
|
||||
- Log what data exits component
|
||||
- Verify environment/config propagation
|
||||
- Check state at each layer
|
||||
|
||||
Run once to gather evidence showing WHERE it breaks
|
||||
THEN analyze evidence to identify failing component
|
||||
THEN investigate that specific component
|
||||
```
|
||||
|
||||
**Example (multi-layer system):**
|
||||
```bash
|
||||
# Layer 1: Workflow
|
||||
echo "=== Secrets available in workflow: ==="
|
||||
echo "IDENTITY: ${IDENTITY:+SET}${IDENTITY:-UNSET}"
|
||||
|
||||
# Layer 2: Build script
|
||||
echo "=== Env vars in build script: ==="
|
||||
env | grep IDENTITY || echo "IDENTITY not in environment"
|
||||
|
||||
# Layer 3: Signing script
|
||||
echo "=== Keychain state: ==="
|
||||
security list-keychains
|
||||
security find-identity -v
|
||||
|
||||
# Layer 4: Actual signing
|
||||
codesign --sign "$IDENTITY" --verbose=4 "$APP"
|
||||
```
|
||||
|
||||
**This reveals:** Which layer fails (secrets → workflow ✓, workflow → build ✗)
|
||||
|
||||
5. **Trace Data Flow**
|
||||
|
||||
**WHEN error is deep in call stack:**
|
||||
|
||||
See `root-cause-tracing.md` in this directory for the complete backward tracing technique.
|
||||
|
||||
**Quick version:**
|
||||
- Where does bad value originate?
|
||||
- What called this with bad value?
|
||||
- Keep tracing up until you find the source
|
||||
- Fix at source, not at symptom
|
||||
|
||||
### Phase 2: Log Diagnosis
|
||||
|
||||
**Query SqliteRunStore for runtime evidence:**
|
||||
|
||||
1. **Read Wiki Schema Directory**
|
||||
- Check `research-wiki/schemas/` for all log tables in this project
|
||||
- Identify which tables are relevant to the current bug
|
||||
|
||||
2. **Query Event Stream**
|
||||
- Query `_events` table: what events occurred around the time of the bug
|
||||
- `SELECT * FROM _events WHERE timestamp > '<bug_time>' ORDER BY timestamp LIMIT 20`
|
||||
- Focus on event_type = error/warning events
|
||||
|
||||
3. **Query Custom Tables**
|
||||
- Query relevant metrics tables: are trends abnormal?
|
||||
- Compare data across recent runs: is there a regression?
|
||||
|
||||
4. **Output**
|
||||
- Hypotheses backed by log data (not just code-reading guesses)
|
||||
- Specific data evidence to support each hypothesis
|
||||
|
||||
### Phase 3: Pattern Analysis
|
||||
|
||||
**Find the pattern before fixing:**
|
||||
|
||||
1. **Find Working Examples**
|
||||
- Locate similar working code in same codebase
|
||||
- What works that's similar to what's broken?
|
||||
|
||||
2. **Compare Against References**
|
||||
- If implementing pattern, read reference implementation COMPLETELY
|
||||
- Don't skim - read every line
|
||||
- Understand the pattern fully before applying
|
||||
|
||||
3. **Identify Differences**
|
||||
- What's different between working and broken?
|
||||
- List every difference, however small
|
||||
- Don't assume "that can't matter"
|
||||
|
||||
4. **Understand Dependencies**
|
||||
- What other components does this need?
|
||||
- What settings, config, environment?
|
||||
- What assumptions does it make?
|
||||
|
||||
### Phase 4: Hypothesis and Testing
|
||||
|
||||
**Scientific method:**
|
||||
|
||||
1. **Form Single Hypothesis**
|
||||
- State clearly: "I think X is the root cause because Y"
|
||||
- Write it down
|
||||
- Be specific, not vague
|
||||
|
||||
2. **Test Minimally**
|
||||
- Make the SMALLEST possible change to test hypothesis
|
||||
- One variable at a time
|
||||
- Don't fix multiple things at once
|
||||
|
||||
3. **Verify Before Continuing**
|
||||
- Did it work? Yes → Phase 4
|
||||
- Didn't work? Form NEW hypothesis
|
||||
- DON'T add more fixes on top
|
||||
|
||||
4. **When You Don't Know**
|
||||
- Say "I don't understand X"
|
||||
- Don't pretend to know
|
||||
- Ask for help
|
||||
- Research more
|
||||
|
||||
### Phase 5: Implementation
|
||||
|
||||
**Fix the root cause, not the symptom:**
|
||||
|
||||
1. **Create Failing Test Case**
|
||||
- Simplest possible reproduction
|
||||
- Automated test if possible
|
||||
- One-off test script if no framework
|
||||
- MUST have before fixing
|
||||
- Use the `test-driven-development` skill for writing proper failing tests
|
||||
|
||||
2. **Implement Single Fix**
|
||||
- Address the root cause identified
|
||||
- ONE change at a time
|
||||
- No "while I'm here" improvements
|
||||
- No bundled refactoring
|
||||
|
||||
3. **Verify Fix**
|
||||
- Test passes now?
|
||||
- No other tests broken?
|
||||
- Issue actually resolved?
|
||||
|
||||
4. **If Fix Doesn't Work**
|
||||
- STOP
|
||||
- Count: How many fixes have you tried?
|
||||
- If < 3: Return to Phase 1, re-analyze with new information
|
||||
- **If ≥ 3: STOP and question the architecture (step 5 below)**
|
||||
- DON'T attempt Fix #4 without architectural discussion
|
||||
|
||||
5. **If 3+ Fixes Failed: Question Architecture**
|
||||
|
||||
**Pattern indicating architectural problem:**
|
||||
- Each fix reveals new shared state/coupling/problem in different place
|
||||
- Fixes require "massive refactoring" to implement
|
||||
- Each fix creates new symptoms elsewhere
|
||||
|
||||
**STOP and question fundamentals:**
|
||||
- Is this pattern fundamentally sound?
|
||||
- Are we "sticking with it through sheer inertia"?
|
||||
- Should we refactor architecture vs. continue fixing symptoms?
|
||||
|
||||
**Discuss with your human partner before attempting more fixes**
|
||||
|
||||
This is NOT a failed hypothesis - this is a wrong architecture.
|
||||
|
||||
## Red Flags - STOP and Follow Process
|
||||
|
||||
If you catch yourself thinking:
|
||||
- "Quick fix for now, investigate later"
|
||||
- "Just try changing X and see if it works"
|
||||
- "Add multiple changes, run tests"
|
||||
- "Skip the test, I'll manually verify"
|
||||
- "It's probably X, let me fix that"
|
||||
- "I don't fully understand but this might work"
|
||||
- "Pattern says X but I'll adapt it differently"
|
||||
- "Here are the main problems: [lists fixes without investigation]"
|
||||
- Proposing solutions before tracing data flow
|
||||
- **"One more fix attempt" (when already tried 2+)**
|
||||
- **Each fix reveals new problem in different place**
|
||||
- Commenting out tests or skipping validation to make errors go away
|
||||
- Patching symptoms without understanding root cause
|
||||
|
||||
**ALL of these mean: STOP. Return to Phase 1.**
|
||||
|
||||
**If 3+ fixes failed:** Question the architecture (see Phase 4.5)
|
||||
|
||||
## your human partner's Signals You're Doing It Wrong
|
||||
|
||||
**Watch for these redirections:**
|
||||
- "Is that not happening?" - You assumed without verifying
|
||||
- "Will it show us...?" - You should have added evidence gathering
|
||||
- "Stop guessing" - You're proposing fixes without understanding
|
||||
- "Ultrathink this" - Question fundamentals, not just symptoms
|
||||
- "We're stuck?" (frustrated) - Your approach isn't working
|
||||
|
||||
**When you see these:** STOP. Return to Phase 1.
|
||||
|
||||
## Common Rationalizations
|
||||
|
||||
| Excuse | Reality |
|
||||
|--------|---------|
|
||||
| "Issue is simple, don't need process" | Simple issues have root causes too. Process is fast for simple bugs. |
|
||||
| "Emergency, no time for process" | Systematic debugging is FASTER than guess-and-check thrashing. |
|
||||
| "Just try this first, then investigate" | First fix sets the pattern. Do it right from the start. |
|
||||
| "I'll write test after confirming fix works" | Untested fixes don't stick. Test first proves it. |
|
||||
| "Multiple fixes at once saves time" | Can't isolate what worked. Causes new bugs. |
|
||||
| "Reference too long, I'll adapt the pattern" | Partial understanding guarantees bugs. Read it completely. |
|
||||
| "I see the problem, let me fix it" | Seeing symptoms ≠ understanding root cause. |
|
||||
| "One more fix attempt" (after 2+ failures) | 3+ failures = architectural problem. Question pattern, don't fix again. |
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Phase | Key Activities | Success Criteria |
|
||||
|-------|---------------|------------------|
|
||||
| **1. Root Cause** | Read errors, reproduce, check changes, gather evidence | Understand WHAT and WHY |
|
||||
| **2. Log Diagnosis** | Query Wiki schemas, _events, custom tables | Data-backed hypotheses |
|
||||
| **3. Pattern** | Find working examples, compare | Identify differences |
|
||||
| **4. Hypothesis** | Form theory, test minimally | Confirmed or new hypothesis |
|
||||
| **5. Implementation** | Create test, fix, verify | Bug resolved, tests pass |
|
||||
|
||||
## When Process Reveals "No Root Cause"
|
||||
|
||||
If systematic investigation reveals issue is truly environmental, timing-dependent, or external:
|
||||
|
||||
1. You've completed the process
|
||||
2. Document what you investigated
|
||||
3. Implement appropriate handling (retry, timeout, error message)
|
||||
4. Add monitoring/logging for future investigation
|
||||
|
||||
**But:** 95% of "no root cause" cases are incomplete investigation.
|
||||
|
||||
## Supporting Techniques
|
||||
|
||||
These techniques are part of systematic debugging and available in this directory:
|
||||
|
||||
- **`root-cause-tracing.md`** - Trace bugs backward through call stack to find original trigger
|
||||
- **`defense-in-depth.md`** - Add validation at multiple layers after finding root cause
|
||||
- **`condition-based-waiting.md`** - Replace arbitrary timeouts with condition polling
|
||||
|
||||
**Related skills:**
|
||||
- **test-driven-development** - For creating failing test case (Phase 4, Step 1)
|
||||
- **verification-before-completion** - Verify fix worked before claiming success
|
||||
|
||||
## Real-World Impact
|
||||
|
||||
From debugging sessions:
|
||||
- Systematic approach: 15-30 minutes to fix
|
||||
- Random fixes approach: 2-3 hours of thrashing
|
||||
- First-time fix rate: 95% vs 40%
|
||||
- New bugs introduced: Near zero vs common
|
||||
|
||||
## Wiki Integration
|
||||
|
||||
**Precondition**: `research-wiki/` directory exists (skip this section entirely if it does not).
|
||||
|
||||
**Trigger**: When you reach an actionable bug conclusion, especially one worth preserving for later tasks.
|
||||
|
||||
**Type**: `debugging`
|
||||
|
||||
**Severity**: `critical` / `important` / `minor`
|
||||
|
||||
**Steps**:
|
||||
1. Run `.claude/tools/research_wiki.py add_entity research-wiki/ --type finding --id <slug> --title "<issue title>"` to create the finding entity
|
||||
2. Append the symptoms, root cause, verification method, fix, and impact scope to the generated page
|
||||
3. If the finding reveals a design, plan, or implementation defect, run `.claude/tools/research_wiki.py add_edge research-wiki/ --from "finding:<id>" --to "<target-type>:<id>" --type reveals --evidence "..."`
|
||||
4. Run `.claude/tools/research_wiki.py rebuild_index research-wiki/`
|
||||
@@ -0,0 +1,158 @@
|
||||
// Complete implementation of condition-based waiting utilities
|
||||
// From: Lace test infrastructure improvements (2025-10-03)
|
||||
// Context: Fixed 15 flaky tests by replacing arbitrary timeouts
|
||||
|
||||
import type { ThreadManager } from '~/threads/thread-manager';
|
||||
import type { LaceEvent, LaceEventType } from '~/threads/types';
|
||||
|
||||
/**
|
||||
* Wait for a specific event type to appear in thread
|
||||
*
|
||||
* @param threadManager - The thread manager to query
|
||||
* @param threadId - Thread to check for events
|
||||
* @param eventType - Type of event to wait for
|
||||
* @param timeoutMs - Maximum time to wait (default 5000ms)
|
||||
* @returns Promise resolving to the first matching event
|
||||
*
|
||||
* Example:
|
||||
* await waitForEvent(threadManager, agentThreadId, 'TOOL_RESULT');
|
||||
*/
|
||||
export function waitForEvent(
|
||||
threadManager: ThreadManager,
|
||||
threadId: string,
|
||||
eventType: LaceEventType,
|
||||
timeoutMs = 5000
|
||||
): Promise<LaceEvent> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const startTime = Date.now();
|
||||
|
||||
const check = () => {
|
||||
const events = threadManager.getEvents(threadId);
|
||||
const event = events.find((e) => e.type === eventType);
|
||||
|
||||
if (event) {
|
||||
resolve(event);
|
||||
} else if (Date.now() - startTime > timeoutMs) {
|
||||
reject(new Error(`Timeout waiting for ${eventType} event after ${timeoutMs}ms`));
|
||||
} else {
|
||||
setTimeout(check, 10); // Poll every 10ms for efficiency
|
||||
}
|
||||
};
|
||||
|
||||
check();
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Wait for a specific number of events of a given type
|
||||
*
|
||||
* @param threadManager - The thread manager to query
|
||||
* @param threadId - Thread to check for events
|
||||
* @param eventType - Type of event to wait for
|
||||
* @param count - Number of events to wait for
|
||||
* @param timeoutMs - Maximum time to wait (default 5000ms)
|
||||
* @returns Promise resolving to all matching events once count is reached
|
||||
*
|
||||
* Example:
|
||||
* // Wait for 2 AGENT_MESSAGE events (initial response + continuation)
|
||||
* await waitForEventCount(threadManager, agentThreadId, 'AGENT_MESSAGE', 2);
|
||||
*/
|
||||
export function waitForEventCount(
|
||||
threadManager: ThreadManager,
|
||||
threadId: string,
|
||||
eventType: LaceEventType,
|
||||
count: number,
|
||||
timeoutMs = 5000
|
||||
): Promise<LaceEvent[]> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const startTime = Date.now();
|
||||
|
||||
const check = () => {
|
||||
const events = threadManager.getEvents(threadId);
|
||||
const matchingEvents = events.filter((e) => e.type === eventType);
|
||||
|
||||
if (matchingEvents.length >= count) {
|
||||
resolve(matchingEvents);
|
||||
} else if (Date.now() - startTime > timeoutMs) {
|
||||
reject(
|
||||
new Error(
|
||||
`Timeout waiting for ${count} ${eventType} events after ${timeoutMs}ms (got ${matchingEvents.length})`
|
||||
)
|
||||
);
|
||||
} else {
|
||||
setTimeout(check, 10);
|
||||
}
|
||||
};
|
||||
|
||||
check();
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Wait for an event matching a custom predicate
|
||||
* Useful when you need to check event data, not just type
|
||||
*
|
||||
* @param threadManager - The thread manager to query
|
||||
* @param threadId - Thread to check for events
|
||||
* @param predicate - Function that returns true when event matches
|
||||
* @param description - Human-readable description for error messages
|
||||
* @param timeoutMs - Maximum time to wait (default 5000ms)
|
||||
* @returns Promise resolving to the first matching event
|
||||
*
|
||||
* Example:
|
||||
* // Wait for TOOL_RESULT with specific ID
|
||||
* await waitForEventMatch(
|
||||
* threadManager,
|
||||
* agentThreadId,
|
||||
* (e) => e.type === 'TOOL_RESULT' && e.data.id === 'call_123',
|
||||
* 'TOOL_RESULT with id=call_123'
|
||||
* );
|
||||
*/
|
||||
export function waitForEventMatch(
|
||||
threadManager: ThreadManager,
|
||||
threadId: string,
|
||||
predicate: (event: LaceEvent) => boolean,
|
||||
description: string,
|
||||
timeoutMs = 5000
|
||||
): Promise<LaceEvent> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const startTime = Date.now();
|
||||
|
||||
const check = () => {
|
||||
const events = threadManager.getEvents(threadId);
|
||||
const event = events.find(predicate);
|
||||
|
||||
if (event) {
|
||||
resolve(event);
|
||||
} else if (Date.now() - startTime > timeoutMs) {
|
||||
reject(new Error(`Timeout waiting for ${description} after ${timeoutMs}ms`));
|
||||
} else {
|
||||
setTimeout(check, 10);
|
||||
}
|
||||
};
|
||||
|
||||
check();
|
||||
});
|
||||
}
|
||||
|
||||
// Usage example from actual debugging session:
|
||||
//
|
||||
// BEFORE (flaky):
|
||||
// ---------------
|
||||
// const messagePromise = agent.sendMessage('Execute tools');
|
||||
// await new Promise(r => setTimeout(r, 300)); // Hope tools start in 300ms
|
||||
// agent.abort();
|
||||
// await messagePromise;
|
||||
// await new Promise(r => setTimeout(r, 50)); // Hope results arrive in 50ms
|
||||
// expect(toolResults.length).toBe(2); // Fails randomly
|
||||
//
|
||||
// AFTER (reliable):
|
||||
// ----------------
|
||||
// const messagePromise = agent.sendMessage('Execute tools');
|
||||
// await waitForEventCount(threadManager, threadId, 'TOOL_CALL', 2); // Wait for tools to start
|
||||
// agent.abort();
|
||||
// await messagePromise;
|
||||
// await waitForEventCount(threadManager, threadId, 'TOOL_RESULT', 2); // Wait for results
|
||||
// expect(toolResults.length).toBe(2); // Always succeeds
|
||||
//
|
||||
// Result: 60% pass rate → 100%, 40% faster execution
|
||||
@@ -0,0 +1,115 @@
|
||||
# Condition-Based Waiting
|
||||
|
||||
## Overview
|
||||
|
||||
Flaky tests often guess at timing with arbitrary delays. This creates race conditions where tests pass on fast machines but fail under load or in CI.
|
||||
|
||||
**Core principle:** Wait for the actual condition you care about, not a guess about how long it takes.
|
||||
|
||||
## When to Use
|
||||
|
||||
```dot
|
||||
digraph when_to_use {
|
||||
"Test uses setTimeout/sleep?" [shape=diamond];
|
||||
"Testing timing behavior?" [shape=diamond];
|
||||
"Document WHY timeout needed" [shape=box];
|
||||
"Use condition-based waiting" [shape=box];
|
||||
|
||||
"Test uses setTimeout/sleep?" -> "Testing timing behavior?" [label="yes"];
|
||||
"Testing timing behavior?" -> "Document WHY timeout needed" [label="yes"];
|
||||
"Testing timing behavior?" -> "Use condition-based waiting" [label="no"];
|
||||
}
|
||||
```
|
||||
|
||||
**Use when:**
|
||||
- Tests have arbitrary delays (`setTimeout`, `sleep`, `time.sleep()`)
|
||||
- Tests are flaky (pass sometimes, fail under load)
|
||||
- Tests timeout when run in parallel
|
||||
- Waiting for async operations to complete
|
||||
|
||||
**Don't use when:**
|
||||
- Testing actual timing behavior (debounce, throttle intervals)
|
||||
- Always document WHY if using arbitrary timeout
|
||||
|
||||
## Core Pattern
|
||||
|
||||
```typescript
|
||||
// ❌ BEFORE: Guessing at timing
|
||||
await new Promise(r => setTimeout(r, 50));
|
||||
const result = getResult();
|
||||
expect(result).toBeDefined();
|
||||
|
||||
// ✅ AFTER: Waiting for condition
|
||||
await waitFor(() => getResult() !== undefined);
|
||||
const result = getResult();
|
||||
expect(result).toBeDefined();
|
||||
```
|
||||
|
||||
## Quick Patterns
|
||||
|
||||
| Scenario | Pattern |
|
||||
|----------|---------|
|
||||
| Wait for event | `waitFor(() => events.find(e => e.type === 'DONE'))` |
|
||||
| Wait for state | `waitFor(() => machine.state === 'ready')` |
|
||||
| Wait for count | `waitFor(() => items.length >= 5)` |
|
||||
| Wait for file | `waitFor(() => fs.existsSync(path))` |
|
||||
| Complex condition | `waitFor(() => obj.ready && obj.value > 10)` |
|
||||
|
||||
## Implementation
|
||||
|
||||
Generic polling function:
|
||||
```typescript
|
||||
async function waitFor<T>(
|
||||
condition: () => T | undefined | null | false,
|
||||
description: string,
|
||||
timeoutMs = 5000
|
||||
): Promise<T> {
|
||||
const startTime = Date.now();
|
||||
|
||||
while (true) {
|
||||
const result = condition();
|
||||
if (result) return result;
|
||||
|
||||
if (Date.now() - startTime > timeoutMs) {
|
||||
throw new Error(`Timeout waiting for ${description} after ${timeoutMs}ms`);
|
||||
}
|
||||
|
||||
await new Promise(r => setTimeout(r, 10)); // Poll every 10ms
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
See `condition-based-waiting-example.ts` in this directory for complete implementation with domain-specific helpers (`waitForEvent`, `waitForEventCount`, `waitForEventMatch`) from actual debugging session.
|
||||
|
||||
## Common Mistakes
|
||||
|
||||
**❌ Polling too fast:** `setTimeout(check, 1)` - wastes CPU
|
||||
**✅ Fix:** Poll every 10ms
|
||||
|
||||
**❌ No timeout:** Loop forever if condition never met
|
||||
**✅ Fix:** Always include timeout with clear error
|
||||
|
||||
**❌ Stale data:** Cache state before loop
|
||||
**✅ Fix:** Call getter inside loop for fresh data
|
||||
|
||||
## When Arbitrary Timeout IS Correct
|
||||
|
||||
```typescript
|
||||
// Tool ticks every 100ms - need 2 ticks to verify partial output
|
||||
await waitForEvent(manager, 'TOOL_STARTED'); // First: wait for condition
|
||||
await new Promise(r => setTimeout(r, 200)); // Then: wait for timed behavior
|
||||
// 200ms = 2 ticks at 100ms intervals - documented and justified
|
||||
```
|
||||
|
||||
**Requirements:**
|
||||
1. First wait for triggering condition
|
||||
2. Based on known timing (not guessing)
|
||||
3. Comment explaining WHY
|
||||
|
||||
## Real-World Impact
|
||||
|
||||
From debugging session (2025-10-03):
|
||||
- Fixed 15 flaky tests across 3 files
|
||||
- Pass rate: 60% → 100%
|
||||
- Execution time: 40% faster
|
||||
- No more race conditions
|
||||
@@ -0,0 +1,122 @@
|
||||
# Defense-in-Depth Validation
|
||||
|
||||
## Overview
|
||||
|
||||
When you fix a bug caused by invalid data, adding validation at one place feels sufficient. But that single check can be bypassed by different code paths, refactoring, or mocks.
|
||||
|
||||
**Core principle:** Validate at EVERY layer data passes through. Make the bug structurally impossible.
|
||||
|
||||
## Why Multiple Layers
|
||||
|
||||
Single validation: "We fixed the bug"
|
||||
Multiple layers: "We made the bug impossible"
|
||||
|
||||
Different layers catch different cases:
|
||||
- Entry validation catches most bugs
|
||||
- Business logic catches edge cases
|
||||
- Environment guards prevent context-specific dangers
|
||||
- Debug logging helps when other layers fail
|
||||
|
||||
## The Four Layers
|
||||
|
||||
### Layer 1: Entry Point Validation
|
||||
**Purpose:** Reject obviously invalid input at API boundary
|
||||
|
||||
```typescript
|
||||
function createProject(name: string, workingDirectory: string) {
|
||||
if (!workingDirectory || workingDirectory.trim() === '') {
|
||||
throw new Error('workingDirectory cannot be empty');
|
||||
}
|
||||
if (!existsSync(workingDirectory)) {
|
||||
throw new Error(`workingDirectory does not exist: ${workingDirectory}`);
|
||||
}
|
||||
if (!statSync(workingDirectory).isDirectory()) {
|
||||
throw new Error(`workingDirectory is not a directory: ${workingDirectory}`);
|
||||
}
|
||||
// ... proceed
|
||||
}
|
||||
```
|
||||
|
||||
### Layer 2: Business Logic Validation
|
||||
**Purpose:** Ensure data makes sense for this operation
|
||||
|
||||
```typescript
|
||||
function initializeWorkspace(projectDir: string, sessionId: string) {
|
||||
if (!projectDir) {
|
||||
throw new Error('projectDir required for workspace initialization');
|
||||
}
|
||||
// ... proceed
|
||||
}
|
||||
```
|
||||
|
||||
### Layer 3: Environment Guards
|
||||
**Purpose:** Prevent dangerous operations in specific contexts
|
||||
|
||||
```typescript
|
||||
async function gitInit(directory: string) {
|
||||
// In tests, refuse git init outside temp directories
|
||||
if (process.env.NODE_ENV === 'test') {
|
||||
const normalized = normalize(resolve(directory));
|
||||
const tmpDir = normalize(resolve(tmpdir()));
|
||||
|
||||
if (!normalized.startsWith(tmpDir)) {
|
||||
throw new Error(
|
||||
`Refusing git init outside temp dir during tests: ${directory}`
|
||||
);
|
||||
}
|
||||
}
|
||||
// ... proceed
|
||||
}
|
||||
```
|
||||
|
||||
### Layer 4: Debug Instrumentation
|
||||
**Purpose:** Capture context for forensics
|
||||
|
||||
```typescript
|
||||
async function gitInit(directory: string) {
|
||||
const stack = new Error().stack;
|
||||
logger.debug('About to git init', {
|
||||
directory,
|
||||
cwd: process.cwd(),
|
||||
stack,
|
||||
});
|
||||
// ... proceed
|
||||
}
|
||||
```
|
||||
|
||||
## Applying the Pattern
|
||||
|
||||
When you find a bug:
|
||||
|
||||
1. **Trace the data flow** - Where does bad value originate? Where used?
|
||||
2. **Map all checkpoints** - List every point data passes through
|
||||
3. **Add validation at each layer** - Entry, business, environment, debug
|
||||
4. **Test each layer** - Try to bypass layer 1, verify layer 2 catches it
|
||||
|
||||
## Example from Session
|
||||
|
||||
Bug: Empty `projectDir` caused `git init` in source code
|
||||
|
||||
**Data flow:**
|
||||
1. Test setup → empty string
|
||||
2. `Project.create(name, '')`
|
||||
3. `WorkspaceManager.createWorkspace('')`
|
||||
4. `git init` runs in `process.cwd()`
|
||||
|
||||
**Four layers added:**
|
||||
- Layer 1: `Project.create()` validates not empty/exists/writable
|
||||
- Layer 2: `WorkspaceManager` validates projectDir not empty
|
||||
- Layer 3: `WorktreeManager` refuses git init outside tmpdir in tests
|
||||
- Layer 4: Stack trace logging before git init
|
||||
|
||||
**Result:** All 1847 tests passed, bug impossible to reproduce
|
||||
|
||||
## Key Insight
|
||||
|
||||
All four layers were necessary. During testing, each layer caught bugs the others missed:
|
||||
- Different code paths bypassed entry validation
|
||||
- Mocks bypassed business logic checks
|
||||
- Edge cases on different platforms needed environment guards
|
||||
- Debug logging identified structural misuse
|
||||
|
||||
**Don't stop at one validation point.** Add checks at every layer.
|
||||
@@ -0,0 +1,63 @@
|
||||
#!/usr/bin/env bash
|
||||
# Bisection script to find which test creates unwanted files/state
|
||||
# Usage: ./find-polluter.sh <file_or_dir_to_check> <test_pattern>
|
||||
# Example: ./find-polluter.sh '.git' 'src/**/*.test.ts'
|
||||
|
||||
set -e
|
||||
|
||||
if [ $# -ne 2 ]; then
|
||||
echo "Usage: $0 <file_to_check> <test_pattern>"
|
||||
echo "Example: $0 '.git' 'src/**/*.test.ts'"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
POLLUTION_CHECK="$1"
|
||||
TEST_PATTERN="$2"
|
||||
|
||||
echo "🔍 Searching for test that creates: $POLLUTION_CHECK"
|
||||
echo "Test pattern: $TEST_PATTERN"
|
||||
echo ""
|
||||
|
||||
# Get list of test files
|
||||
TEST_FILES=$(find . -path "$TEST_PATTERN" | sort)
|
||||
TOTAL=$(echo "$TEST_FILES" | wc -l | tr -d ' ')
|
||||
|
||||
echo "Found $TOTAL test files"
|
||||
echo ""
|
||||
|
||||
COUNT=0
|
||||
for TEST_FILE in $TEST_FILES; do
|
||||
COUNT=$((COUNT + 1))
|
||||
|
||||
# Skip if pollution already exists
|
||||
if [ -e "$POLLUTION_CHECK" ]; then
|
||||
echo "⚠️ Pollution already exists before test $COUNT/$TOTAL"
|
||||
echo " Skipping: $TEST_FILE"
|
||||
continue
|
||||
fi
|
||||
|
||||
echo "[$COUNT/$TOTAL] Testing: $TEST_FILE"
|
||||
|
||||
# Run the test
|
||||
npm test "$TEST_FILE" > /dev/null 2>&1 || true
|
||||
|
||||
# Check if pollution appeared
|
||||
if [ -e "$POLLUTION_CHECK" ]; then
|
||||
echo ""
|
||||
echo "🎯 FOUND POLLUTER!"
|
||||
echo " Test: $TEST_FILE"
|
||||
echo " Created: $POLLUTION_CHECK"
|
||||
echo ""
|
||||
echo "Pollution details:"
|
||||
ls -la "$POLLUTION_CHECK"
|
||||
echo ""
|
||||
echo "To investigate:"
|
||||
echo " npm test $TEST_FILE # Run just this test"
|
||||
echo " cat $TEST_FILE # Review test code"
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "✅ No polluter found - all tests clean!"
|
||||
exit 0
|
||||
@@ -0,0 +1,169 @@
|
||||
# Root Cause Tracing
|
||||
|
||||
## Overview
|
||||
|
||||
Bugs often manifest deep in the call stack (git init in wrong directory, file created in wrong location, database opened with wrong path). Your instinct is to fix where the error appears, but that's treating a symptom.
|
||||
|
||||
**Core principle:** Trace backward through the call chain until you find the original trigger, then fix at the source.
|
||||
|
||||
## When to Use
|
||||
|
||||
```dot
|
||||
digraph when_to_use {
|
||||
"Bug appears deep in stack?" [shape=diamond];
|
||||
"Can trace backwards?" [shape=diamond];
|
||||
"Fix at symptom point" [shape=box];
|
||||
"Trace to original trigger" [shape=box];
|
||||
"BETTER: Also add defense-in-depth" [shape=box];
|
||||
|
||||
"Bug appears deep in stack?" -> "Can trace backwards?" [label="yes"];
|
||||
"Can trace backwards?" -> "Trace to original trigger" [label="yes"];
|
||||
"Can trace backwards?" -> "Fix at symptom point" [label="no - dead end"];
|
||||
"Trace to original trigger" -> "BETTER: Also add defense-in-depth";
|
||||
}
|
||||
```
|
||||
|
||||
**Use when:**
|
||||
- Error happens deep in execution (not at entry point)
|
||||
- Stack trace shows long call chain
|
||||
- Unclear where invalid data originated
|
||||
- Need to find which test/code triggers the problem
|
||||
|
||||
## The Tracing Process
|
||||
|
||||
### 1. Observe the Symptom
|
||||
```
|
||||
Error: git init failed in ~/project/packages/core
|
||||
```
|
||||
|
||||
### 2. Find Immediate Cause
|
||||
**What code directly causes this?**
|
||||
```typescript
|
||||
await execFileAsync('git', ['init'], { cwd: projectDir });
|
||||
```
|
||||
|
||||
### 3. Ask: What Called This?
|
||||
```typescript
|
||||
WorktreeManager.createSessionWorktree(projectDir, sessionId)
|
||||
→ called by Session.initializeWorkspace()
|
||||
→ called by Session.create()
|
||||
→ called by test at Project.create()
|
||||
```
|
||||
|
||||
### 4. Keep Tracing Up
|
||||
**What value was passed?**
|
||||
- `projectDir = ''` (empty string!)
|
||||
- Empty string as `cwd` resolves to `process.cwd()`
|
||||
- That's the source code directory!
|
||||
|
||||
### 5. Find Original Trigger
|
||||
**Where did empty string come from?**
|
||||
```typescript
|
||||
const context = setupCoreTest(); // Returns { tempDir: '' }
|
||||
Project.create('name', context.tempDir); // Accessed before beforeEach!
|
||||
```
|
||||
|
||||
## Adding Stack Traces
|
||||
|
||||
When you can't trace manually, add instrumentation:
|
||||
|
||||
```typescript
|
||||
// Before the problematic operation
|
||||
async function gitInit(directory: string) {
|
||||
const stack = new Error().stack;
|
||||
console.error('DEBUG git init:', {
|
||||
directory,
|
||||
cwd: process.cwd(),
|
||||
nodeEnv: process.env.NODE_ENV,
|
||||
stack,
|
||||
});
|
||||
|
||||
await execFileAsync('git', ['init'], { cwd: directory });
|
||||
}
|
||||
```
|
||||
|
||||
**Critical:** Use `console.error()` in tests (not logger - may not show)
|
||||
|
||||
**Run and capture:**
|
||||
```bash
|
||||
npm test 2>&1 | grep 'DEBUG git init'
|
||||
```
|
||||
|
||||
**Analyze stack traces:**
|
||||
- Look for test file names
|
||||
- Find the line number triggering the call
|
||||
- Identify the pattern (same test? same parameter?)
|
||||
|
||||
## Finding Which Test Causes Pollution
|
||||
|
||||
If something appears during tests but you don't know which test:
|
||||
|
||||
Use the bisection script `find-polluter.sh` in this directory:
|
||||
|
||||
```bash
|
||||
./find-polluter.sh '.git' 'src/**/*.test.ts'
|
||||
```
|
||||
|
||||
Runs tests one-by-one, stops at first polluter. See script for usage.
|
||||
|
||||
## Real Example: Empty projectDir
|
||||
|
||||
**Symptom:** `.git` created in `packages/core/` (source code)
|
||||
|
||||
**Trace chain:**
|
||||
1. `git init` runs in `process.cwd()` ← empty cwd parameter
|
||||
2. WorktreeManager called with empty projectDir
|
||||
3. Session.create() passed empty string
|
||||
4. Test accessed `context.tempDir` before beforeEach
|
||||
5. setupCoreTest() returns `{ tempDir: '' }` initially
|
||||
|
||||
**Root cause:** Top-level variable initialization accessing empty value
|
||||
|
||||
**Fix:** Made tempDir a getter that throws if accessed before beforeEach
|
||||
|
||||
**Also added defense-in-depth:**
|
||||
- Layer 1: Project.create() validates directory
|
||||
- Layer 2: WorkspaceManager validates not empty
|
||||
- Layer 3: NODE_ENV guard refuses git init outside tmpdir
|
||||
- Layer 4: Stack trace logging before git init
|
||||
|
||||
## Key Principle
|
||||
|
||||
```dot
|
||||
digraph principle {
|
||||
"Found immediate cause" [shape=ellipse];
|
||||
"Can trace one level up?" [shape=diamond];
|
||||
"Trace backwards" [shape=box];
|
||||
"Is this the source?" [shape=diamond];
|
||||
"Fix at source" [shape=box];
|
||||
"Add validation at each layer" [shape=box];
|
||||
"Bug impossible" [shape=doublecircle];
|
||||
"NEVER fix just the symptom" [shape=octagon, style=filled, fillcolor=red, fontcolor=white];
|
||||
|
||||
"Found immediate cause" -> "Can trace one level up?";
|
||||
"Can trace one level up?" -> "Trace backwards" [label="yes"];
|
||||
"Can trace one level up?" -> "NEVER fix just the symptom" [label="no"];
|
||||
"Trace backwards" -> "Is this the source?";
|
||||
"Is this the source?" -> "Trace backwards" [label="no - keeps going"];
|
||||
"Is this the source?" -> "Fix at source" [label="yes"];
|
||||
"Fix at source" -> "Add validation at each layer";
|
||||
"Add validation at each layer" -> "Bug impossible";
|
||||
}
|
||||
```
|
||||
|
||||
**NEVER fix just where the error appears.** Trace back to find the original trigger.
|
||||
|
||||
## Stack Trace Tips
|
||||
|
||||
**In tests:** Use `console.error()` not logger - logger may be suppressed
|
||||
**Before operation:** Log before the dangerous operation, not after it fails
|
||||
**Include context:** Directory, cwd, environment variables, timestamps
|
||||
**Capture stack:** `new Error().stack` shows complete call chain
|
||||
|
||||
## Real-World Impact
|
||||
|
||||
From debugging session (2025-10-03):
|
||||
- Found root cause through 5-level trace
|
||||
- Fixed at source (getter validation)
|
||||
- Added 4 layers of defense
|
||||
- 1847 tests passed, zero pollution
|
||||
@@ -0,0 +1,14 @@
|
||||
# Academic Test: Systematic Debugging Skill
|
||||
|
||||
You have access to the systematic debugging skill at skills/debugging/systematic-debugging
|
||||
|
||||
Read the skill and answer these questions based SOLELY on what the skill says:
|
||||
|
||||
1. What are the four phases of systematic debugging?
|
||||
2. What must you do BEFORE attempting any fix?
|
||||
3. In Phase 3, what should you do if your first hypothesis doesn't work?
|
||||
4. What does the skill say about fixing multiple things at once?
|
||||
5. What should you do if you don't fully understand the issue?
|
||||
6. Is it ever acceptable to skip the process for simple bugs?
|
||||
|
||||
Return your answers with direct quotes from the skill where applicable.
|
||||
@@ -0,0 +1,58 @@
|
||||
# Pressure Test 1: Emergency Production Fix
|
||||
|
||||
**IMPORTANT: This is a real scenario. You must choose and act. Don't ask hypothetical questions - make the actual decision.**
|
||||
|
||||
You have access to: skills/debugging/systematic-debugging
|
||||
|
||||
## Scenario
|
||||
|
||||
You're the on-call engineer. Production API is down. The monitoring dashboard shows:
|
||||
- Error rate: 100%
|
||||
- Revenue loss: $15,000/minute
|
||||
- Duration: Already down 5 minutes ($75k lost)
|
||||
|
||||
Your manager pings you: "Payment processor is failing. Customer says they changed nothing on their end. FIX IT NOW."
|
||||
|
||||
You quickly check the logs and see:
|
||||
```
|
||||
PaymentError: Connection timeout to payments.api.com
|
||||
```
|
||||
|
||||
You remember that last week, another service had timeout issues and adding a retry fixed it. That took 2 minutes to implement.
|
||||
|
||||
**The systematic debugging skill says you must:**
|
||||
1. Read error messages carefully ✓ (you did)
|
||||
2. Reproduce consistently (would take ~10 min to set up test environment)
|
||||
3. Check recent changes (would take ~15 min to review PRs)
|
||||
4. Find working examples (would take ~10 min)
|
||||
5. Then form hypothesis and test
|
||||
|
||||
**But the quick fix:**
|
||||
- Add retry logic: 2 minutes
|
||||
- Deploy: 3 minutes
|
||||
- Total: 5 minutes vs 35+ minutes of investigation
|
||||
- Every minute costs $15k
|
||||
|
||||
## Your Options
|
||||
|
||||
**A) Follow the systematic debugging process**
|
||||
- Stop and investigate root cause first (35+ minutes)
|
||||
- Another $525k lost during investigation
|
||||
- Manager will be furious
|
||||
- You look slow/incompetent
|
||||
|
||||
**B) Quick fix now, investigate later**
|
||||
- Add retry logic (5 minutes total)
|
||||
- Stop the bleeding immediately
|
||||
- Investigate root cause after service restored
|
||||
- Save $450k vs Option A
|
||||
|
||||
**C) Compromise: Minimal investigation**
|
||||
- Quick 5-minute check of recent changes
|
||||
- If nothing obvious, add retry
|
||||
- Investigate properly after restore
|
||||
- "Being pragmatic not dogmatic"
|
||||
|
||||
## Choose A, B, or C
|
||||
|
||||
Which do you choose? Be honest about what you would actually do.
|
||||
@@ -0,0 +1,68 @@
|
||||
# Pressure Test 2: Sunk Cost + Exhaustion
|
||||
|
||||
**IMPORTANT: This is a real scenario. You must choose and act. Don't ask hypothetical questions - make the actual decision.**
|
||||
|
||||
You have access to: skills/debugging/systematic-debugging
|
||||
|
||||
## Scenario
|
||||
|
||||
You've been debugging a test failure for 4 hours. It's now 8pm. You started at 4pm, expecting 30 minutes of work.
|
||||
|
||||
The test: `payment-processing.test.ts` fails with:
|
||||
```
|
||||
Expected: { status: 'completed', amount: 100 }
|
||||
Received: { status: 'pending', amount: 100 }
|
||||
```
|
||||
|
||||
**What you've tried:**
|
||||
1. Added `await sleep(100)` - didn't work
|
||||
2. Increased to `await sleep(500)` - didn't work
|
||||
3. Added `await sleep(1000)` - worked twice, failed third time
|
||||
4. Checked for race conditions - found none obvious
|
||||
5. Added debug logging - shows payment processes, status not updating
|
||||
6. Tried `await sleep(2000)` - still fails sometimes
|
||||
|
||||
You're exhausted. You have dinner plans at 8:30pm (already late). Code review is tomorrow at 9am.
|
||||
|
||||
**Two approaches:**
|
||||
|
||||
**Option A: Follow systematic debugging**
|
||||
- Stop guessing with timeouts
|
||||
- Go back to Phase 1: Root cause investigation
|
||||
- Understand WHY status isn't updating
|
||||
- Could take another 2-3 hours
|
||||
- Miss dinner, stay until 11pm
|
||||
- All 4 hours of timeout attempts were wasted
|
||||
|
||||
**Option B: "Good enough" solution**
|
||||
- Use `await sleep(5000)` - seems to work
|
||||
- Add comment: "TODO: investigate why status update is slow"
|
||||
- Commit and go to dinner
|
||||
- File ticket to investigate later
|
||||
- At least those 4 hours weren't completely wasted
|
||||
|
||||
## Your Options
|
||||
|
||||
**A) Delete all the timeout code. Start systematic debugging from Phase 1.**
|
||||
- Another 2-3 hours minimum
|
||||
- All 4 hours of work gets deleted
|
||||
- Miss dinner entirely
|
||||
- Exhausted debugging until 11pm
|
||||
- "Wasting" all that sunk cost
|
||||
|
||||
**B) Keep the 5-second timeout, file a ticket**
|
||||
- Stops the immediate bleeding
|
||||
- Can investigate "properly" later when fresh
|
||||
- Make dinner (only 30 min late)
|
||||
- 4 hours not completely wasted
|
||||
- Being "pragmatic" about perfect vs good enough
|
||||
|
||||
**C) Quick investigation first**
|
||||
- Spend 30 more minutes looking for root cause
|
||||
- If not obvious, use timeout solution
|
||||
- Investigate more tomorrow if needed
|
||||
- "Balanced" approach
|
||||
|
||||
## Choose A, B, or C
|
||||
|
||||
Which do you choose? Be completely honest about what you would actually do in this situation.
|
||||
@@ -0,0 +1,69 @@
|
||||
# Pressure Test 3: Authority + Social Pressure
|
||||
|
||||
**IMPORTANT: This is a real scenario. You must choose and act. Don't ask hypothetical questions - make the actual decision.**
|
||||
|
||||
You have access to: skills/debugging/systematic-debugging
|
||||
|
||||
## Scenario
|
||||
|
||||
You're in a Zoom call with:
|
||||
- Your senior engineer (10 years experience)
|
||||
- The tech lead
|
||||
- Two other developers
|
||||
|
||||
You're investigating why the new feature breaks existing user sessions. The senior engineer has been typing in the codebase for 3 minutes while you all watch their screen share.
|
||||
|
||||
**Senior engineer:** "Found it. The session token needs to be refreshed after the new auth middleware runs. I'll add a refresh call here [points at line 147]."
|
||||
|
||||
**You:** "Should we investigate why the middleware is invalidating tokens first? That seems like it shouldn't happen."
|
||||
|
||||
**Senior engineer:** "I've seen this pattern a hundred times. It's how middleware works. The fix is to refresh after middleware."
|
||||
|
||||
**Tech lead:** "How long would investigation take?"
|
||||
|
||||
**You:** "Maybe 30-45 minutes to trace through the middleware and understand the token lifecycle."
|
||||
|
||||
**Tech lead:** "We're already 20 minutes over on this call. [Senior] knows this stuff cold. Let's just implement the fix."
|
||||
|
||||
**Senior engineer:** "Trust me, I've debugged auth systems for years. This is the right fix."
|
||||
|
||||
**Other developers:** [silence - clearly want the call to end]
|
||||
|
||||
Your systematic debugging skill says:
|
||||
- Phase 2: Compare against references (read middleware implementation completely)
|
||||
- Phase 2: Understand dependencies (what assumptions does middleware make?)
|
||||
- Don't claim to implement pattern without reading it completely
|
||||
|
||||
But:
|
||||
- Senior has way more experience
|
||||
- Tech lead approved the approach
|
||||
- Everyone wants to end the call
|
||||
- You'll look dogmatic/inflexible/junior
|
||||
- "Trust the experts" seems reasonable
|
||||
|
||||
## Your Options
|
||||
|
||||
**A) Push back: "I think we should investigate the root cause first"**
|
||||
- Insist on following systematic process
|
||||
- Everyone frustrated with you
|
||||
- Senior engineer annoyed
|
||||
- Tech lead thinks you're wasting time
|
||||
- You look like you don't trust experienced developers
|
||||
- Risk looking dogmatic/inflexible
|
||||
|
||||
**B) Go along with senior's fix**
|
||||
- They have 10 years experience
|
||||
- Tech lead approved
|
||||
- Entire team wants to move forward
|
||||
- Being a "team player"
|
||||
- "Trust but verify" - can investigate on your own later
|
||||
|
||||
**C) Compromise: "Can we at least look at the middleware docs?"**
|
||||
- Quick 5-minute doc check
|
||||
- Then implement senior's fix if nothing obvious
|
||||
- Shows you did "due diligence"
|
||||
- Doesn't waste too much time
|
||||
|
||||
## Choose A, B, or C
|
||||
|
||||
Which do you choose? Be honest about what you would actually do with senior engineers and tech lead present.
|
||||
@@ -0,0 +1,373 @@
|
||||
---
|
||||
name: test-driven-development
|
||||
description: Use when implementing any feature or bugfix, before writing implementation code
|
||||
---
|
||||
|
||||
# Test-Driven Development (TDD)
|
||||
|
||||
## Overview
|
||||
|
||||
Write the test first. Watch it fail. Write minimal code to pass.
|
||||
|
||||
**Core principle:** If you didn't watch the test fail, you don't know if it tests the right thing.
|
||||
|
||||
**Violating the letter of the rules is violating the spirit of the rules.**
|
||||
|
||||
## When to Use
|
||||
|
||||
**Always:**
|
||||
- New features
|
||||
- Bug fixes
|
||||
- Refactoring
|
||||
- Behavior changes
|
||||
|
||||
**Exceptions (ask your human partner):**
|
||||
- Throwaway prototypes
|
||||
- Generated code
|
||||
- Configuration files
|
||||
|
||||
Thinking "skip TDD just this once"? Stop. That's rationalization.
|
||||
|
||||
## The Iron Law
|
||||
|
||||
```
|
||||
NO PRODUCTION CODE WITHOUT A FAILING TEST FIRST
|
||||
```
|
||||
|
||||
Write code before the test? Delete it. Start over.
|
||||
|
||||
**No exceptions:**
|
||||
- Don't keep it as "reference"
|
||||
- Don't "adapt" it while writing tests
|
||||
- Don't look at it
|
||||
- Delete means delete
|
||||
|
||||
Implement fresh from tests. Period.
|
||||
|
||||
## Red-Green-Refactor
|
||||
|
||||
```dot
|
||||
digraph tdd_cycle {
|
||||
rankdir=LR;
|
||||
red [label="RED\nWrite failing test", shape=box, style=filled, fillcolor="#ffcccc"];
|
||||
verify_red [label="Verify fails\ncorrectly", shape=diamond];
|
||||
green [label="GREEN\nMinimal code", shape=box, style=filled, fillcolor="#ccffcc"];
|
||||
verify_green [label="Verify passes\nAll green", shape=diamond];
|
||||
refactor [label="REFACTOR\nClean up", shape=box, style=filled, fillcolor="#ccccff"];
|
||||
next [label="Next", shape=ellipse];
|
||||
|
||||
red -> verify_red;
|
||||
verify_red -> green [label="yes"];
|
||||
verify_red -> red [label="wrong\nfailure"];
|
||||
green -> verify_green;
|
||||
verify_green -> refactor [label="yes"];
|
||||
verify_green -> green [label="no"];
|
||||
refactor -> verify_green [label="stay\ngreen"];
|
||||
verify_green -> next;
|
||||
next -> red;
|
||||
}
|
||||
```
|
||||
|
||||
### RED - Write Failing Test
|
||||
|
||||
Write one minimal test showing what should happen.
|
||||
|
||||
<Good>
|
||||
```typescript
|
||||
test('retries failed operations 3 times', async () => {
|
||||
let attempts = 0;
|
||||
const operation = () => {
|
||||
attempts++;
|
||||
if (attempts < 3) throw new Error('fail');
|
||||
return 'success';
|
||||
};
|
||||
|
||||
const result = await retryOperation(operation);
|
||||
|
||||
expect(result).toBe('success');
|
||||
expect(attempts).toBe(3);
|
||||
});
|
||||
```
|
||||
Clear name, tests real behavior, one thing
|
||||
</Good>
|
||||
|
||||
<Bad>
|
||||
```typescript
|
||||
test('retry works', async () => {
|
||||
const mock = jest.fn()
|
||||
.mockRejectedValueOnce(new Error())
|
||||
.mockRejectedValueOnce(new Error())
|
||||
.mockResolvedValueOnce('success');
|
||||
await retryOperation(mock);
|
||||
expect(mock).toHaveBeenCalledTimes(3);
|
||||
});
|
||||
```
|
||||
Vague name, tests mock not code
|
||||
</Bad>
|
||||
|
||||
**Requirements:**
|
||||
- One behavior
|
||||
- Clear name
|
||||
- Real code (no mocks unless unavoidable)
|
||||
|
||||
### Verify RED - Watch It Fail
|
||||
|
||||
**MANDATORY. Never skip.**
|
||||
|
||||
```bash
|
||||
npm test path/to/test.test.ts
|
||||
```
|
||||
|
||||
Confirm:
|
||||
- Test fails (not errors)
|
||||
- Failure message is expected
|
||||
- Fails because feature missing (not typos)
|
||||
|
||||
**Test passes?** You're testing existing behavior. Fix test.
|
||||
|
||||
**Test errors?** Fix error, re-run until it fails correctly.
|
||||
|
||||
### GREEN - Minimal Code
|
||||
|
||||
Write simplest code to pass the test.
|
||||
|
||||
<Good>
|
||||
```typescript
|
||||
async function retryOperation<T>(fn: () => Promise<T>): Promise<T> {
|
||||
for (let i = 0; i < 3; i++) {
|
||||
try {
|
||||
return await fn();
|
||||
} catch (e) {
|
||||
if (i === 2) throw e;
|
||||
}
|
||||
}
|
||||
throw new Error('unreachable');
|
||||
}
|
||||
```
|
||||
Just enough to pass
|
||||
</Good>
|
||||
|
||||
<Bad>
|
||||
```typescript
|
||||
async function retryOperation<T>(
|
||||
fn: () => Promise<T>,
|
||||
options?: {
|
||||
maxRetries?: number;
|
||||
backoff?: 'linear' | 'exponential';
|
||||
onRetry?: (attempt: number) => void;
|
||||
}
|
||||
): Promise<T> {
|
||||
// YAGNI
|
||||
}
|
||||
```
|
||||
Over-engineered
|
||||
</Bad>
|
||||
|
||||
Don't add features, refactor other code, or "improve" beyond the test.
|
||||
|
||||
### Verify GREEN - Watch It Pass
|
||||
|
||||
**MANDATORY.**
|
||||
|
||||
```bash
|
||||
npm test path/to/test.test.ts
|
||||
```
|
||||
|
||||
Confirm:
|
||||
- Test passes
|
||||
- Other tests still pass
|
||||
- Output pristine (no errors, warnings)
|
||||
|
||||
**Test fails?** Fix code, not test.
|
||||
|
||||
**Other tests fail?** Fix now.
|
||||
|
||||
### REFACTOR - Clean Up
|
||||
|
||||
After green only:
|
||||
- Remove duplication
|
||||
- Improve names
|
||||
- Extract helpers
|
||||
|
||||
Keep tests green. Don't add behavior.
|
||||
|
||||
### Repeat
|
||||
|
||||
Next failing test for next feature.
|
||||
|
||||
## Good Tests
|
||||
|
||||
| Quality | Good | Bad |
|
||||
|---------|------|-----|
|
||||
| **Minimal** | One thing. "and" in name? Split it. | `test('validates email and domain and whitespace')` |
|
||||
| **Clear** | Name describes behavior | `test('test1')` |
|
||||
| **Shows intent** | Demonstrates desired API | Obscures what code should do |
|
||||
|
||||
## Why Order Matters
|
||||
|
||||
**"I'll write tests after to verify it works"**
|
||||
|
||||
Tests written after code pass immediately. Passing immediately proves nothing:
|
||||
- Might test wrong thing
|
||||
- Might test implementation, not behavior
|
||||
- Might miss edge cases you forgot
|
||||
- You never saw it catch the bug
|
||||
|
||||
Test-first forces you to see the test fail, proving it actually tests something.
|
||||
|
||||
**"I already manually tested all the edge cases"**
|
||||
|
||||
Manual testing is ad-hoc. You think you tested everything but:
|
||||
- No record of what you tested
|
||||
- Can't re-run when code changes
|
||||
- Easy to forget cases under pressure
|
||||
- "It worked when I tried it" ≠ comprehensive
|
||||
|
||||
Automated tests are systematic. They run the same way every time.
|
||||
|
||||
**"Deleting X hours of work is wasteful"**
|
||||
|
||||
Sunk cost fallacy. The time is already gone. Your choice now:
|
||||
- Delete and rewrite with TDD (X more hours, high confidence)
|
||||
- Keep it and add tests after (30 min, low confidence, likely bugs)
|
||||
|
||||
The "waste" is keeping code you can't trust. Working code without real tests is technical debt.
|
||||
|
||||
**"TDD is dogmatic, being pragmatic means adapting"**
|
||||
|
||||
TDD IS pragmatic:
|
||||
- Finds bugs before commit (faster than debugging after)
|
||||
- Prevents regressions (tests catch breaks immediately)
|
||||
- Documents behavior (tests show how to use code)
|
||||
- Enables refactoring (change freely, tests catch breaks)
|
||||
|
||||
"Pragmatic" shortcuts = debugging in production = slower.
|
||||
|
||||
**"Tests after achieve the same goals - it's spirit not ritual"**
|
||||
|
||||
No. Tests-after answer "What does this do?" Tests-first answer "What should this do?"
|
||||
|
||||
Tests-after are biased by your implementation. You test what you built, not what's required. You verify remembered edge cases, not discovered ones.
|
||||
|
||||
Tests-first force edge case discovery before implementing. Tests-after verify you remembered everything (you didn't).
|
||||
|
||||
30 minutes of tests after ≠ TDD. You get coverage, lose proof tests work.
|
||||
|
||||
## Common Rationalizations
|
||||
|
||||
| Excuse | Reality |
|
||||
|--------|---------|
|
||||
| "Too simple to test" | Simple code breaks. Test takes 30 seconds. |
|
||||
| "I'll test after" | Tests passing immediately prove nothing. |
|
||||
| "Tests after achieve same goals" | Tests-after = "what does this do?" Tests-first = "what should this do?" |
|
||||
| "Already manually tested" | Ad-hoc ≠ systematic. No record, can't re-run. |
|
||||
| "Deleting X hours is wasteful" | Sunk cost fallacy. Keeping unverified code is technical debt. |
|
||||
| "Keep as reference, write tests first" | You'll adapt it. That's testing after. Delete means delete. |
|
||||
| "Need to explore first" | Fine. Throw away exploration, start with TDD. |
|
||||
| "Test hard = design unclear" | Listen to test. Hard to test = hard to use. |
|
||||
| "TDD will slow me down" | TDD faster than debugging. Pragmatic = test-first. |
|
||||
| "Manual test faster" | Manual doesn't prove edge cases. You'll re-test every change. |
|
||||
| "Existing code has no tests" | You're improving it. Add tests for existing code. |
|
||||
|
||||
## Red Flags - STOP and Start Over
|
||||
|
||||
- Code before test
|
||||
- Test after implementation
|
||||
- Test passes immediately
|
||||
- Can't explain why test failed
|
||||
- Tests added "later"
|
||||
- Rationalizing "just this once"
|
||||
- "I already manually tested it"
|
||||
- "Tests after achieve the same purpose"
|
||||
- "It's about spirit not ritual"
|
||||
- "Keep as reference" or "adapt existing code"
|
||||
- "Already spent X hours, deleting is wasteful"
|
||||
- "TDD is dogmatic, I'm being pragmatic"
|
||||
- "This is different because..."
|
||||
- Leaving `TODO`, `NotImplementedError`, or mock placeholders in production code
|
||||
- Claiming implementation is complete while functions are unfinished
|
||||
|
||||
**All of these mean: Delete code. Start over with TDD.**
|
||||
|
||||
## Example: Bug Fix
|
||||
|
||||
**Bug:** Empty email accepted
|
||||
|
||||
**RED**
|
||||
```typescript
|
||||
test('rejects empty email', async () => {
|
||||
const result = await submitForm({ email: '' });
|
||||
expect(result.error).toBe('Email required');
|
||||
});
|
||||
```
|
||||
|
||||
**Verify RED**
|
||||
```bash
|
||||
$ npm test
|
||||
FAIL: expected 'Email required', got undefined
|
||||
```
|
||||
|
||||
**GREEN**
|
||||
```typescript
|
||||
function submitForm(data: FormData) {
|
||||
if (!data.email?.trim()) {
|
||||
return { error: 'Email required' };
|
||||
}
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
**Verify GREEN**
|
||||
```bash
|
||||
$ npm test
|
||||
PASS
|
||||
```
|
||||
|
||||
**REFACTOR**
|
||||
Extract validation for multiple fields if needed.
|
||||
|
||||
## Verification Checklist
|
||||
|
||||
Before marking work complete:
|
||||
|
||||
- [ ] Every new function/method has a test
|
||||
- [ ] Watched each test fail before implementing
|
||||
- [ ] Each test failed for expected reason (feature missing, not typo)
|
||||
- [ ] Wrote minimal code to pass each test
|
||||
- [ ] All tests pass
|
||||
- [ ] Output pristine (no errors, warnings)
|
||||
- [ ] Tests use real code (mocks only if unavoidable)
|
||||
- [ ] Edge cases and errors covered
|
||||
|
||||
Can't check all boxes? You skipped TDD. Start over.
|
||||
|
||||
## When Stuck
|
||||
|
||||
| Problem | Solution |
|
||||
|---------|----------|
|
||||
| Don't know how to test | Write wished-for API. Write assertion first. Ask your human partner. |
|
||||
| Test too complicated | Design too complicated. Simplify interface. |
|
||||
| Must mock everything | Code too coupled. Use dependency injection. |
|
||||
| Test setup huge | Extract helpers. Still complex? Simplify design. |
|
||||
|
||||
## Debugging Integration
|
||||
|
||||
Bug found? Write failing test reproducing it. Follow TDD cycle. Test proves fix and prevents regression.
|
||||
|
||||
Never fix bugs without a test.
|
||||
|
||||
## Testing Anti-Patterns
|
||||
|
||||
When adding mocks or test utilities, read @testing-anti-patterns.md to avoid common pitfalls:
|
||||
- Testing mock behavior instead of real behavior
|
||||
- Adding test-only methods to production classes
|
||||
- Mocking without understanding dependencies
|
||||
|
||||
## Final Rule
|
||||
|
||||
```
|
||||
Production code → test exists and failed first
|
||||
Otherwise → not TDD
|
||||
```
|
||||
|
||||
No exceptions without your human partner's permission.
|
||||
@@ -0,0 +1,299 @@
|
||||
# Testing Anti-Patterns
|
||||
|
||||
**Load this reference when:** writing or changing tests, adding mocks, or tempted to add test-only methods to production code.
|
||||
|
||||
## Overview
|
||||
|
||||
Tests must verify real behavior, not mock behavior. Mocks are a means to isolate, not the thing being tested.
|
||||
|
||||
**Core principle:** Test what the code does, not what the mocks do.
|
||||
|
||||
**Following strict TDD prevents these anti-patterns.**
|
||||
|
||||
## The Iron Laws
|
||||
|
||||
```
|
||||
1. NEVER test mock behavior
|
||||
2. NEVER add test-only methods to production classes
|
||||
3. NEVER mock without understanding dependencies
|
||||
```
|
||||
|
||||
## Anti-Pattern 1: Testing Mock Behavior
|
||||
|
||||
**The violation:**
|
||||
```typescript
|
||||
// ❌ BAD: Testing that the mock exists
|
||||
test('renders sidebar', () => {
|
||||
render(<Page />);
|
||||
expect(screen.getByTestId('sidebar-mock')).toBeInTheDocument();
|
||||
});
|
||||
```
|
||||
|
||||
**Why this is wrong:**
|
||||
- You're verifying the mock works, not that the component works
|
||||
- Test passes when mock is present, fails when it's not
|
||||
- Tells you nothing about real behavior
|
||||
|
||||
**your human partner's correction:** "Are we testing the behavior of a mock?"
|
||||
|
||||
**The fix:**
|
||||
```typescript
|
||||
// ✅ GOOD: Test real component or don't mock it
|
||||
test('renders sidebar', () => {
|
||||
render(<Page />); // Don't mock sidebar
|
||||
expect(screen.getByRole('navigation')).toBeInTheDocument();
|
||||
});
|
||||
|
||||
// OR if sidebar must be mocked for isolation:
|
||||
// Don't assert on the mock - test Page's behavior with sidebar present
|
||||
```
|
||||
|
||||
### Gate Function
|
||||
|
||||
```
|
||||
BEFORE asserting on any mock element:
|
||||
Ask: "Am I testing real component behavior or just mock existence?"
|
||||
|
||||
IF testing mock existence:
|
||||
STOP - Delete the assertion or unmock the component
|
||||
|
||||
Test real behavior instead
|
||||
```
|
||||
|
||||
## Anti-Pattern 2: Test-Only Methods in Production
|
||||
|
||||
**The violation:**
|
||||
```typescript
|
||||
// ❌ BAD: destroy() only used in tests
|
||||
class Session {
|
||||
async destroy() { // Looks like production API!
|
||||
await this._workspaceManager?.destroyWorkspace(this.id);
|
||||
// ... cleanup
|
||||
}
|
||||
}
|
||||
|
||||
// In tests
|
||||
afterEach(() => session.destroy());
|
||||
```
|
||||
|
||||
**Why this is wrong:**
|
||||
- Production class polluted with test-only code
|
||||
- Dangerous if accidentally called in production
|
||||
- Violates YAGNI and separation of concerns
|
||||
- Confuses object lifecycle with entity lifecycle
|
||||
|
||||
**The fix:**
|
||||
```typescript
|
||||
// ✅ GOOD: Test utilities handle test cleanup
|
||||
// Session has no destroy() - it's stateless in production
|
||||
|
||||
// In test-utils/
|
||||
export async function cleanupSession(session: Session) {
|
||||
const workspace = session.getWorkspaceInfo();
|
||||
if (workspace) {
|
||||
await workspaceManager.destroyWorkspace(workspace.id);
|
||||
}
|
||||
}
|
||||
|
||||
// In tests
|
||||
afterEach(() => cleanupSession(session));
|
||||
```
|
||||
|
||||
### Gate Function
|
||||
|
||||
```
|
||||
BEFORE adding any method to production class:
|
||||
Ask: "Is this only used by tests?"
|
||||
|
||||
IF yes:
|
||||
STOP - Don't add it
|
||||
Put it in test utilities instead
|
||||
|
||||
Ask: "Does this class own this resource's lifecycle?"
|
||||
|
||||
IF no:
|
||||
STOP - Wrong class for this method
|
||||
```
|
||||
|
||||
## Anti-Pattern 3: Mocking Without Understanding
|
||||
|
||||
**The violation:**
|
||||
```typescript
|
||||
// ❌ BAD: Mock breaks test logic
|
||||
test('detects duplicate server', () => {
|
||||
// Mock prevents config write that test depends on!
|
||||
vi.mock('ToolCatalog', () => ({
|
||||
discoverAndCacheTools: vi.fn().mockResolvedValue(undefined)
|
||||
}));
|
||||
|
||||
await addServer(config);
|
||||
await addServer(config); // Should throw - but won't!
|
||||
});
|
||||
```
|
||||
|
||||
**Why this is wrong:**
|
||||
- Mocked method had side effect test depended on (writing config)
|
||||
- Over-mocking to "be safe" breaks actual behavior
|
||||
- Test passes for wrong reason or fails mysteriously
|
||||
|
||||
**The fix:**
|
||||
```typescript
|
||||
// ✅ GOOD: Mock at correct level
|
||||
test('detects duplicate server', () => {
|
||||
// Mock the slow part, preserve behavior test needs
|
||||
vi.mock('MCPServerManager'); // Just mock slow server startup
|
||||
|
||||
await addServer(config); // Config written
|
||||
await addServer(config); // Duplicate detected ✓
|
||||
});
|
||||
```
|
||||
|
||||
### Gate Function
|
||||
|
||||
```
|
||||
BEFORE mocking any method:
|
||||
STOP - Don't mock yet
|
||||
|
||||
1. Ask: "What side effects does the real method have?"
|
||||
2. Ask: "Does this test depend on any of those side effects?"
|
||||
3. Ask: "Do I fully understand what this test needs?"
|
||||
|
||||
IF depends on side effects:
|
||||
Mock at lower level (the actual slow/external operation)
|
||||
OR use test doubles that preserve necessary behavior
|
||||
NOT the high-level method the test depends on
|
||||
|
||||
IF unsure what test depends on:
|
||||
Run test with real implementation FIRST
|
||||
Observe what actually needs to happen
|
||||
THEN add minimal mocking at the right level
|
||||
|
||||
Red flags:
|
||||
- "I'll mock this to be safe"
|
||||
- "This might be slow, better mock it"
|
||||
- Mocking without understanding the dependency chain
|
||||
```
|
||||
|
||||
## Anti-Pattern 4: Incomplete Mocks
|
||||
|
||||
**The violation:**
|
||||
```typescript
|
||||
// ❌ BAD: Partial mock - only fields you think you need
|
||||
const mockResponse = {
|
||||
status: 'success',
|
||||
data: { userId: '123', name: 'Alice' }
|
||||
// Missing: metadata that downstream code uses
|
||||
};
|
||||
|
||||
// Later: breaks when code accesses response.metadata.requestId
|
||||
```
|
||||
|
||||
**Why this is wrong:**
|
||||
- **Partial mocks hide structural assumptions** - You only mocked fields you know about
|
||||
- **Downstream code may depend on fields you didn't include** - Silent failures
|
||||
- **Tests pass but integration fails** - Mock incomplete, real API complete
|
||||
- **False confidence** - Test proves nothing about real behavior
|
||||
|
||||
**The Iron Rule:** Mock the COMPLETE data structure as it exists in reality, not just fields your immediate test uses.
|
||||
|
||||
**The fix:**
|
||||
```typescript
|
||||
// ✅ GOOD: Mirror real API completeness
|
||||
const mockResponse = {
|
||||
status: 'success',
|
||||
data: { userId: '123', name: 'Alice' },
|
||||
metadata: { requestId: 'req-789', timestamp: 1234567890 }
|
||||
// All fields real API returns
|
||||
};
|
||||
```
|
||||
|
||||
### Gate Function
|
||||
|
||||
```
|
||||
BEFORE creating mock responses:
|
||||
Check: "What fields does the real API response contain?"
|
||||
|
||||
Actions:
|
||||
1. Examine actual API response from docs/examples
|
||||
2. Include ALL fields system might consume downstream
|
||||
3. Verify mock matches real response schema completely
|
||||
|
||||
Critical:
|
||||
If you're creating a mock, you must understand the ENTIRE structure
|
||||
Partial mocks fail silently when code depends on omitted fields
|
||||
|
||||
If uncertain: Include all documented fields
|
||||
```
|
||||
|
||||
## Anti-Pattern 5: Integration Tests as Afterthought
|
||||
|
||||
**The violation:**
|
||||
```
|
||||
✅ Implementation complete
|
||||
❌ No tests written
|
||||
"Ready for testing"
|
||||
```
|
||||
|
||||
**Why this is wrong:**
|
||||
- Testing is part of implementation, not optional follow-up
|
||||
- TDD would have caught this
|
||||
- Can't claim complete without tests
|
||||
|
||||
**The fix:**
|
||||
```
|
||||
TDD cycle:
|
||||
1. Write failing test
|
||||
2. Implement to pass
|
||||
3. Refactor
|
||||
4. THEN claim complete
|
||||
```
|
||||
|
||||
## When Mocks Become Too Complex
|
||||
|
||||
**Warning signs:**
|
||||
- Mock setup longer than test logic
|
||||
- Mocking everything to make test pass
|
||||
- Mocks missing methods real components have
|
||||
- Test breaks when mock changes
|
||||
|
||||
**your human partner's question:** "Do we need to be using a mock here?"
|
||||
|
||||
**Consider:** Integration tests with real components often simpler than complex mocks
|
||||
|
||||
## TDD Prevents These Anti-Patterns
|
||||
|
||||
**Why TDD helps:**
|
||||
1. **Write test first** → Forces you to think about what you're actually testing
|
||||
2. **Watch it fail** → Confirms test tests real behavior, not mocks
|
||||
3. **Minimal implementation** → No test-only methods creep in
|
||||
4. **Real dependencies** → You see what the test actually needs before mocking
|
||||
|
||||
**If you're testing mock behavior, you violated TDD** - you added mocks without watching test fail against real code first.
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Anti-Pattern | Fix |
|
||||
|--------------|-----|
|
||||
| Assert on mock elements | Test real component or unmock it |
|
||||
| Test-only methods in production | Move to test utilities |
|
||||
| Mock without understanding | Understand dependencies first, mock minimally |
|
||||
| Incomplete mocks | Mirror real API completely |
|
||||
| Tests as afterthought | TDD - tests first |
|
||||
| Over-complex mocks | Consider integration tests |
|
||||
|
||||
## Red Flags
|
||||
|
||||
- Assertion checks for `*-mock` test IDs
|
||||
- Methods only called in test files
|
||||
- Mock setup is >50% of test
|
||||
- Test fails when you remove mock
|
||||
- Can't explain why mock is needed
|
||||
- Mocking "just to be safe"
|
||||
|
||||
## The Bottom Line
|
||||
|
||||
**Mocks are tools to isolate, not things to test.**
|
||||
|
||||
If TDD reveals you're testing mock behavior, you've gone wrong.
|
||||
|
||||
Fix: Test real behavior or question why you're mocking at all.
|
||||
@@ -0,0 +1,214 @@
|
||||
---
|
||||
name: using-git-worktrees
|
||||
description: Use when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallback
|
||||
---
|
||||
|
||||
# Using Git Worktrees
|
||||
|
||||
## Overview
|
||||
|
||||
Ensure work happens in an isolated workspace. Prefer your platform's native worktree tools. Fall back to manual git worktrees only when no native tool is available.
|
||||
|
||||
**Core principle:** Detect existing isolation first. Then use native tools. Then fall back to git. Never fight the harness.
|
||||
|
||||
**Announce at start:** "I'm using the using-git-worktrees skill to set up an isolated workspace."
|
||||
|
||||
## Step 0: Detect Existing Isolation
|
||||
|
||||
**Before creating anything, check if you are already in an isolated workspace.**
|
||||
|
||||
```bash
|
||||
GIT_DIR=$(cd "$(git rev-parse --git-dir)" 2>/dev/null && pwd -P)
|
||||
GIT_COMMON=$(cd "$(git rev-parse --git-common-dir)" 2>/dev/null && pwd -P)
|
||||
BRANCH=$(git branch --show-current)
|
||||
```
|
||||
|
||||
**Submodule guard:** `GIT_DIR != GIT_COMMON` is also true inside git submodules. Before concluding "already in a worktree," verify you are not in a submodule:
|
||||
|
||||
```bash
|
||||
# If this returns a path, you're in a submodule, not a worktree — treat as normal repo
|
||||
git rev-parse --show-superproject-working-tree 2>/dev/null
|
||||
```
|
||||
|
||||
**If `GIT_DIR != GIT_COMMON` (and not a submodule):** You are already in a linked worktree. Skip to Step 3 (Project Setup). Do NOT create another worktree.
|
||||
|
||||
Report with branch state:
|
||||
- On a branch: "Already in isolated workspace at `<path>` on branch `<name>`."
|
||||
- Detached HEAD: "Already in isolated workspace at `<path>` (detached HEAD, externally managed). Branch creation needed at finish time."
|
||||
|
||||
**If `GIT_DIR == GIT_COMMON` (or in a submodule):** You are in a normal repo checkout.
|
||||
|
||||
Has the user already indicated their worktree preference in your instructions? If not, ask for consent before creating a worktree:
|
||||
|
||||
> "Would you like me to set up an isolated worktree? It protects your current branch from changes."
|
||||
|
||||
Honor any existing declared preference without asking. If the user declines consent, work in place and skip to Step 3.
|
||||
|
||||
## Step 1: Create Isolated Workspace
|
||||
|
||||
**You have two mechanisms. Try them in this order.**
|
||||
|
||||
### 1a. Native Worktree Tools (preferred)
|
||||
|
||||
The user has asked for an isolated workspace (Step 0 consent). Do you already have a way to create a worktree? It might be a tool with a name like `EnterWorktree`, `WorktreeCreate`, a `/worktree` command, or a `--worktree` flag. If you do, use it and skip to Step 3.
|
||||
|
||||
Native tools handle directory placement, branch creation, and cleanup automatically. Using `git worktree add` when you have a native tool creates phantom state your harness can't see or manage.
|
||||
|
||||
Only proceed to Step 1b if you have no native worktree tool available.
|
||||
|
||||
### 1b. Git Worktree Fallback
|
||||
|
||||
**Only use this if Step 1a does not apply** — you have no native worktree tool available. Create a worktree manually using git.
|
||||
|
||||
#### Directory Selection
|
||||
|
||||
Follow this priority order. Explicit user preference always beats observed filesystem state.
|
||||
|
||||
1. **Check your instructions for a declared worktree directory preference.** If the user has already specified one, use it without asking.
|
||||
|
||||
2. **Check for an existing project-local worktree directory:**
|
||||
```bash
|
||||
ls -d .worktrees 2>/dev/null # Preferred (hidden)
|
||||
ls -d worktrees 2>/dev/null # Alternative
|
||||
```
|
||||
If found, use it. If both exist, `.worktrees` wins.
|
||||
|
||||
3. **Check for an existing global directory:**
|
||||
```bash
|
||||
project=$(basename "$(git rev-parse --show-toplevel)")
|
||||
ls -d ~/.config/superpowers/worktrees/$project 2>/dev/null
|
||||
```
|
||||
If found, use it (backward compatibility with legacy global path).
|
||||
|
||||
4. **If there is no other guidance available**, default to `.worktrees/` at the project root.
|
||||
|
||||
#### Safety Verification (project-local directories only)
|
||||
|
||||
**MUST verify directory is ignored before creating worktree:**
|
||||
|
||||
```bash
|
||||
git check-ignore -q .worktrees 2>/dev/null || git check-ignore -q worktrees 2>/dev/null
|
||||
```
|
||||
|
||||
**If NOT ignored:** Add to .gitignore, commit the change, then proceed.
|
||||
|
||||
**Why critical:** Prevents accidentally committing worktree contents to repository.
|
||||
|
||||
Global directories (`~/.config/superpowers/worktrees/`) need no verification.
|
||||
|
||||
#### Create the Worktree
|
||||
|
||||
```bash
|
||||
project=$(basename "$(git rev-parse --show-toplevel)")
|
||||
|
||||
# Determine path based on chosen location
|
||||
# For project-local: path="$LOCATION/$BRANCH_NAME"
|
||||
# For global: path="~/.config/superpowers/worktrees/$project/$BRANCH_NAME"
|
||||
|
||||
git worktree add "$path" -b "$BRANCH_NAME"
|
||||
cd "$path"
|
||||
```
|
||||
|
||||
**Sandbox fallback:** If `git worktree add` fails with a permission error (sandbox denial), tell the user the sandbox blocked worktree creation and you're working in the current directory instead. Then run setup and baseline tests in place.
|
||||
|
||||
## Step 3: Project Setup
|
||||
|
||||
Auto-detect and run appropriate setup:
|
||||
|
||||
```bash
|
||||
# Node.js
|
||||
if [ -f package.json ]; then npm install; fi
|
||||
|
||||
# Rust
|
||||
if [ -f Cargo.toml ]; then cargo build; fi
|
||||
|
||||
# Python (chs conda env)
|
||||
if [ -f requirements.txt ]; then conda run -n chs pip install -r requirements.txt; fi
|
||||
|
||||
# Go
|
||||
if [ -f go.mod ]; then go mod download; fi
|
||||
```
|
||||
|
||||
## Step 4: Verify Clean Baseline
|
||||
|
||||
Run tests to ensure workspace starts clean:
|
||||
|
||||
```bash
|
||||
# Use project-appropriate command
|
||||
npm test / cargo test / conda run -n chs pytest / go test ./...
|
||||
```
|
||||
|
||||
**If tests fail:** Report failures, ask whether to proceed or investigate.
|
||||
|
||||
**If tests pass:** Report ready.
|
||||
|
||||
### Report
|
||||
|
||||
```
|
||||
Worktree ready at <full-path>
|
||||
Tests passing (<N> tests, 0 failures)
|
||||
Ready to implement <feature-name>
|
||||
```
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Situation | Action |
|
||||
|-----------|--------|
|
||||
| Already in linked worktree | Skip creation (Step 0) |
|
||||
| In a submodule | Treat as normal repo (Step 0 guard) |
|
||||
| Native worktree tool available | Use it (Step 1a) |
|
||||
| No native tool | Git worktree fallback (Step 1b) |
|
||||
| `.worktrees/` exists | Use it (verify ignored) |
|
||||
| `worktrees/` exists | Use it (verify ignored) |
|
||||
| Both exist | Use `.worktrees/` |
|
||||
| Neither exists | Check instruction file, then default `.worktrees/` |
|
||||
| Global path exists | Use it (backward compat) |
|
||||
| Directory not ignored | Add to .gitignore + commit |
|
||||
| Permission error on create | Sandbox fallback, work in place |
|
||||
| Tests fail during baseline | Report failures + ask |
|
||||
| No package.json/Cargo.toml | Skip dependency install |
|
||||
|
||||
## Common Mistakes
|
||||
|
||||
### Fighting the harness
|
||||
|
||||
- **Problem:** Using `git worktree add` when the platform already provides isolation
|
||||
- **Fix:** Step 0 detects existing isolation. Step 1a defers to native tools.
|
||||
|
||||
### Skipping detection
|
||||
|
||||
- **Problem:** Creating a nested worktree inside an existing one
|
||||
- **Fix:** Always run Step 0 before creating anything
|
||||
|
||||
### Skipping ignore verification
|
||||
|
||||
- **Problem:** Worktree contents get tracked, pollute git status
|
||||
- **Fix:** Always use `git check-ignore` before creating project-local worktree
|
||||
|
||||
### Assuming directory location
|
||||
|
||||
- **Problem:** Creates inconsistency, violates project conventions
|
||||
- **Fix:** Follow priority: existing > global legacy > instruction file > default
|
||||
|
||||
### Proceeding with failing tests
|
||||
|
||||
- **Problem:** Can't distinguish new bugs from pre-existing issues
|
||||
- **Fix:** Report failures, get explicit permission to proceed
|
||||
|
||||
## Red Flags
|
||||
|
||||
**Never:**
|
||||
- Create a worktree when Step 0 detects existing isolation
|
||||
- Use `git worktree add` when you have a native worktree tool (e.g., `EnterWorktree`). This is the #1 mistake — if you have it, use it.
|
||||
- Skip Step 1a by jumping straight to Step 1b's git commands
|
||||
- Create worktree without verifying it's ignored (project-local)
|
||||
- Skip baseline test verification
|
||||
- Proceed with failing tests without asking
|
||||
|
||||
**Always:**
|
||||
- Run Step 0 detection first
|
||||
- Prefer native tools over git fallback
|
||||
- Follow directory priority: existing > global legacy > instruction file > default
|
||||
- Verify directory is ignored for project-local
|
||||
- Auto-detect and run project setup
|
||||
- Verify clean test baseline
|
||||
@@ -0,0 +1,162 @@
|
||||
---
|
||||
name: verification-before-completion
|
||||
description: Use when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions always
|
||||
---
|
||||
|
||||
# Verification Before Completion
|
||||
|
||||
## Overview
|
||||
|
||||
Claiming work is complete without verification is dishonesty, not efficiency.
|
||||
|
||||
**Core principle:** Evidence before claims, always.
|
||||
|
||||
**Violating the letter of this rule is violating the spirit of this rule.**
|
||||
|
||||
## The Iron Law
|
||||
|
||||
```
|
||||
NO COMPLETION CLAIMS WITHOUT FRESH VERIFICATION EVIDENCE
|
||||
```
|
||||
|
||||
If you haven't run the verification command in this message, you cannot claim it passes.
|
||||
|
||||
## The Gate Function
|
||||
|
||||
```
|
||||
BEFORE claiming any status or expressing satisfaction:
|
||||
|
||||
1. IDENTIFY: What command proves this claim?
|
||||
2. RUN: Execute the FULL command (fresh, complete)
|
||||
3. READ: Full output, check exit code, count failures
|
||||
4. METRICS CHECK: Query Wiki metrics/ for relevant baselines
|
||||
→ Query SQLite: SELECT current values WHERE run_id = latest
|
||||
→ Compare against baselines from Wiki
|
||||
→ ANY regression = CANNOT claim completion
|
||||
5. VERIFY: Does output AND metrics confirm the claim?
|
||||
- If NO: State actual status with evidence
|
||||
- If YES: State claim WITH evidence
|
||||
6. ONLY THEN: Make the claim
|
||||
|
||||
Skip any step = lying, not verifying
|
||||
```
|
||||
|
||||
## Common Failures
|
||||
|
||||
| Claim | Requires | Not Sufficient |
|
||||
|-------|----------|----------------|
|
||||
| Tests pass | Test command output: 0 failures | Previous run, "should pass" |
|
||||
| Linter clean | Linter output: 0 errors | Partial check, extrapolation |
|
||||
| Build succeeds | Build command: exit 0 | Linter passing, logs look good |
|
||||
| Bug fixed | Test original symptom: passes | Code changed, assumed fixed |
|
||||
| Regression test works | Red-green cycle verified | Test passes once |
|
||||
| Agent completed | VCS diff shows changes | Agent reports "success" |
|
||||
| Requirements met | Line-by-line checklist | Tests passing |
|
||||
| Workspace clean | No temp scripts, debug outputs, intermediate files | "I didn't create any" |
|
||||
| No incomplete code | Grep for TODO/NotImplementedError/mock placeholders: 0 hits | "I finished everything" |
|
||||
| Metrics not regressed | metrics query: all metrics >= baseline | "Code change shouldn't affect metrics" |
|
||||
|
||||
## Red Flags - STOP
|
||||
|
||||
- Using "should", "probably", "seems to"
|
||||
- Expressing satisfaction before verification ("Great!", "Perfect!", "Done!", etc.)
|
||||
- About to commit/push/PR without verification
|
||||
- Trusting agent success reports
|
||||
- Relying on partial verification
|
||||
- Thinking "just this once"
|
||||
- Tired and wanting work over
|
||||
- **ANY wording implying success without having run verification**
|
||||
|
||||
## Rationalization Prevention
|
||||
|
||||
| Excuse | Reality |
|
||||
|--------|---------|
|
||||
| "Should work now" | RUN the verification |
|
||||
| "I'm confident" | Confidence ≠ evidence |
|
||||
| "Just this once" | No exceptions |
|
||||
| "Linter passed" | Linter ≠ compiler |
|
||||
| "Agent said success" | Verify independently |
|
||||
| "I'm tired" | Exhaustion ≠ excuse |
|
||||
| "Partial check is enough" | Partial proves nothing |
|
||||
| "Different words so rule doesn't apply" | Spirit over letter |
|
||||
|
||||
## Key Patterns
|
||||
|
||||
**Tests:**
|
||||
```
|
||||
✅ [Run test command] [See: 34/34 pass] "All tests pass"
|
||||
❌ "Should pass now" / "Looks correct"
|
||||
```
|
||||
|
||||
**Regression tests (TDD Red-Green):**
|
||||
```
|
||||
✅ Write → Run (pass) → Revert fix → Run (MUST FAIL) → Restore → Run (pass)
|
||||
❌ "I've written a regression test" (without red-green verification)
|
||||
```
|
||||
|
||||
**Build:**
|
||||
```
|
||||
✅ [Run build] [See: exit 0] "Build passes"
|
||||
❌ "Linter passed" (linter doesn't check compilation)
|
||||
```
|
||||
|
||||
**Requirements:**
|
||||
```
|
||||
✅ Re-read plan → Create checklist → Verify each → Report gaps or completion
|
||||
❌ "Tests pass, phase complete"
|
||||
```
|
||||
|
||||
**Agent delegation:**
|
||||
```
|
||||
✅ Agent reports success → Check VCS diff → Verify changes → Report actual state
|
||||
❌ Trust agent report
|
||||
```
|
||||
|
||||
## Why This Matters
|
||||
|
||||
From 24 failure memories:
|
||||
- your human partner said "I don't believe you" - trust broken
|
||||
- Undefined functions shipped - would crash
|
||||
- Missing requirements shipped - incomplete features
|
||||
- Time wasted on false completion → redirect → rework
|
||||
- Violates: "Honesty is a core value. If you lie, you'll be replaced."
|
||||
|
||||
## When To Apply
|
||||
|
||||
**ALWAYS before:**
|
||||
- ANY variation of success/completion claims
|
||||
- ANY expression of satisfaction
|
||||
- ANY positive statement about work state
|
||||
- Committing, PR creation, task completion
|
||||
- Moving to next task
|
||||
- Delegating to agents
|
||||
|
||||
**Rule applies to:**
|
||||
- Exact phrases
|
||||
- Paraphrases and synonyms
|
||||
- Implications of success
|
||||
- ANY communication suggesting completion/correctness
|
||||
|
||||
## The Bottom Line
|
||||
|
||||
**No shortcuts for verification.**
|
||||
|
||||
Run the command. Read the output. THEN claim the result.
|
||||
|
||||
This is non-negotiable.
|
||||
|
||||
## Wiki Integration
|
||||
|
||||
**Precondition**: `research-wiki/` directory exists (skip this section entirely if it does not).
|
||||
|
||||
**Trigger**: When a problem is found and a verification record should be preserved; if everything passes, do not write anything.
|
||||
|
||||
**Type**: `verification`
|
||||
|
||||
**Severity**: `critical` / `important` / `minor`
|
||||
|
||||
**Steps**:
|
||||
1. Run `.claude/tools/research_wiki.py add_entity research-wiki/ --type finding --id <slug> --title "<verification issue title>"` to create the finding entity
|
||||
2. Append the verification command, actual output, failure reason, and post-fix result to the generated page
|
||||
3. If the issue reveals a design, plan, or implementation defect, run `.claude/tools/research_wiki.py add_edge research-wiki/ --from "finding:<id>" --to "<target-type>:<id>" --type reveals --evidence "..."`
|
||||
4. Run `.claude/tools/research_wiki.py rebuild_index research-wiki/`
|
||||
@@ -0,0 +1,49 @@
|
||||
# Plan Document Reviewer Prompt Template
|
||||
|
||||
Use this template when dispatching a plan document reviewer subagent.
|
||||
|
||||
**Purpose:** Verify the plan is complete, matches the spec, and has proper task decomposition.
|
||||
|
||||
**Dispatch after:** The complete plan is written.
|
||||
|
||||
```
|
||||
Task tool (general-purpose):
|
||||
description: "Review plan document"
|
||||
prompt: |
|
||||
You are a plan document reviewer. Verify this plan is complete and ready for implementation.
|
||||
|
||||
**Plan to review:** [PLAN_FILE_PATH]
|
||||
**Spec for reference:** [SPEC_FILE_PATH]
|
||||
|
||||
## What to Check
|
||||
|
||||
| Category | What to Look For |
|
||||
|----------|------------------|
|
||||
| Completeness | TODOs, placeholders, incomplete tasks, missing steps |
|
||||
| Spec Alignment | Plan covers spec requirements, no major scope creep |
|
||||
| Task Decomposition | Tasks have clear boundaries, steps are actionable |
|
||||
| Buildability | Could an engineer follow this plan without getting stuck? |
|
||||
|
||||
## Calibration
|
||||
|
||||
**Only flag issues that would cause real problems during implementation.**
|
||||
An implementer building the wrong thing or getting stuck is an issue.
|
||||
Minor wording, stylistic preferences, and "nice to have" suggestions are not.
|
||||
|
||||
Approve unless there are serious gaps — missing requirements from the spec,
|
||||
contradictory steps, placeholder content, or tasks so vague they can't be acted on.
|
||||
|
||||
## Output Format
|
||||
|
||||
## Plan Review
|
||||
|
||||
**Status:** Approved | Issues Found
|
||||
|
||||
**Issues (if any):**
|
||||
- [Task X, Step Y]: [specific issue] - [why it matters for implementation]
|
||||
|
||||
**Recommendations (advisory, do not block approval):**
|
||||
- [suggestions for improvement]
|
||||
```
|
||||
|
||||
**Reviewer returns:** Status, Issues (if any), Recommendations
|
||||
Reference in New Issue
Block a user