iomgaa
6bdb802f01
- Add all claude skills (brainstorming, commit, debugging, TDD, etc.) - Add claude hooks (pre-commit-guard, post-edit-quality) - Add research templates (experiment plan, research brief, etc.) - Add claude tools (arxiv/semantic_scholar/openalex fetch, wiki, exa) - Add TRM4 reference implementation as algorithm fidelity baseline - Add research-wiki content (plans, index, graph, query_pack) - Update .gitignore to exclude .graphify_version runtime state
142 lines
6.0 KiB
Markdown
142 lines
6.0 KiB
Markdown
---
|
|
name: novelty-check
|
|
description: "Verify the novelty of research ideas. GPT cross-validation. Trigger phrases: novelty check, has anyone done this, check novelty."
|
|
argument-hint: [method-or-idea-description]
|
|
---
|
|
|
|
# Novelty Verification
|
|
Verify novelty of: $ARGUMENTS
|
|
|
|
## Goal
|
|
Perform a strict check on whether a method, idea, or experimental setting is actually new. The default stance is skepticism, not help-seeking for supporting evidence.
|
|
|
|
## Working Principles
|
|
- Brutally honest: do not relax the standard just to make something look new.
|
|
- `Applying X to Y` is not novel by default unless the application produces an unexpected mechanism, theoretical explanation, or clearly different experimental phenomenon.
|
|
- Check the novelty of both the `METHOD` and the `EXPERIMENTAL SETTING`.
|
|
- If the method itself is not new, but the findings, conclusions, experimental setup, or failure analysis are new, state that distinction explicitly.
|
|
- Always search the last 6 months of arXiv.
|
|
- Do not rely on titles alone; read the abstract and, when necessary, the key parts of related work, intro, method, and appendix.
|
|
|
|
## Workflow
|
|
|
|
### Phase A: Extract Core Claims
|
|
First break the user's method description into 3-5 core technical claims. Each one should be as specific as possible.
|
|
|
|
For each claim, answer:
|
|
- What is the method?
|
|
- What problem does it solve?
|
|
- What is the key mechanism?
|
|
- What is the essential difference from an obvious baseline?
|
|
|
|
Rewrite the story-like description into searchable technical propositions and avoid vague phrasing.
|
|
|
|
### Phase B: Multi-source Literature Search
|
|
Run multi-source retrieval for each claim, prioritizing recent work and similar settings.
|
|
|
|
For each claim, try at least 3 search-query sets, and make them complementary:
|
|
- Direct technical terms
|
|
- Synonyms / abbreviations / related task names
|
|
- "Problem + mechanism" combinations
|
|
- "Method + dataset / setting" combinations
|
|
|
|
#### Required Search Channels
|
|
1. WebSearch: arXiv / Google Scholar / Semantic Scholar / conference homepages
|
|
2. `python3 .claude/tools/arxiv_fetch.py search "QUERY" --max 10`
|
|
3. `python3 .claude/tools/semantic_scholar_fetch.py search "QUERY" --max 10`
|
|
4. `python3 .claude/tools/exa_search.py search "QUERY" --max 10` (if available)
|
|
5. `python3 .claude/tools/openalex_fetch.py search "QUERY" --max 10` (if available)
|
|
|
|
#### Search Priorities
|
|
- ICLR / NeurIPS / ICML 2025-2026
|
|
- arXiv preprints from the last 6 months
|
|
- Papers close to the method mechanism, not only papers on the same task
|
|
- Papers close to the experimental setting, not only papers using the same method
|
|
|
|
#### Decision Strategy
|
|
- Record potentially overlapping papers first; do not exclude them too early
|
|
- Prefer reading the abstract, intro, related work, and method sections
|
|
- If overlap looks suspicious, also read the experimental setup and appendix
|
|
|
|
#### Recording Requirements
|
|
For each candidate paper, record:
|
|
- Title
|
|
- Year
|
|
- Venue / status
|
|
- Relevant point
|
|
- The specific reason it may overlap
|
|
- Why it might still be a different work
|
|
|
|
If a data source is unavailable, explicitly record the fallback reason and continue with the others; do not stop the task.
|
|
|
|
### Phase C: GPT Cross-Validation
|
|
Send the method description from Phase A and all candidate papers found in Phase B to `/codex:rescue --fresh --wait` for a second review.
|
|
|
|
The cross-validation prompt must include:
|
|
- proposed method description
|
|
- the full candidate paper list
|
|
- ask:
|
|
- `Is this method novel?`
|
|
- `What is the closest prior work?`
|
|
- `What is the delta?`
|
|
|
|
Use high reasoning effort.
|
|
|
|
The goal of cross-validation is not to find even more papers. It is to force out the closest prior art, the smallest difference, and the risk of pseudo-novelty.
|
|
|
|
### Phase D: Output Report + Wiki Integration
|
|
The output must be in English and follow a fixed structure.
|
|
|
|
#### Report Format
|
|
```markdown
|
|
## Novelty Check Report
|
|
### Method Under Review
|
|
### Core Claims
|
|
- Claim 1: ... (novelty: high / medium / low; closest paper: ...)
|
|
- Claim 2: ... (novelty: high / medium / low; closest paper: ...)
|
|
|
|
### Recent Prior Work
|
|
| Paper | Year | Venue / Status | Overlap Point | Key Difference |
|
|
|---|---:|---|---|---|
|
|
|
|
### Overall Assessment
|
|
- score X/10
|
|
- recommendation: continue / continue cautiously / abandon
|
|
- key differentiator: ...
|
|
- positioning advice: ...
|
|
```
|
|
|
|
#### Evaluation Scale
|
|
- `high`: current search shows no close prior art, and the difference is concrete and technical
|
|
- `medium`: there is related prior work, but there is still a clear and defensible technical delta
|
|
- `low`: mostly a reorganization of known methods, task switching, dataset switching, hyperparameter changes, or standard engineering changes
|
|
|
|
#### Wiki Integration
|
|
If the project has `research-wiki/`, also ingest the knowledge there:
|
|
- Create a `claim` entity for each core claim
|
|
- Create a `paper` entity for each newly found paper
|
|
- Add claim-paper / paper-paper relation edges
|
|
- Rebuild `query_pack`
|
|
|
|
Prefer existing tools such as `.claude/tools/research_wiki.py`; if the wiki does not exist, skip silently and do not error.
|
|
|
|
Ingestion rules:
|
|
- Only write high-confidence information
|
|
- Claim names should be short, stable, and reusable
|
|
- Edges must include evidence; do not create empty links
|
|
|
|
## Completion Criteria
|
|
Only finish when all of the following are complete:
|
|
1. 3-5 core claims have been extracted
|
|
2. Multi-source search has been completed, including the last 6 months of arXiv
|
|
3. Candidate paper abstracts have been read, and related work / method sections were read when necessary
|
|
4. GPT cross-validation has been completed
|
|
5. A structured English report has been produced
|
|
6. If `research-wiki/` exists, the corresponding writes and `query_pack` rebuild have been completed
|
|
|
|
## Failure Handling
|
|
- Missing tools: record the missing item and degrade gracefully
|
|
- Too few search results: expand synonyms, abbreviations, higher-level terms, and experimental settings
|
|
- Too many search results: prefer the most recent, most similar, and most likely overlapping work
|
|
- Conflicting evidence: read the original abstract and method sections first; do not rely on intuition
|