research-pipeline

Full Research Pipeline: Idea → Experiments → Submission

⏱ External cadence: non-judgmental heartbeat only. An overnight /loop / CronCreate heartbeat may wake, detect a stalled phase (no progress, dead process, blocked on a freed resource) and nudge it forward — it may NEVER decide the work is good (paper good enough, proof holds, claim supported). Every such verdict stays on its own skill's internal cadence and terminates in the cross-model jury. A heartbeat may say "keep going," never "good enough." See shared-references/external-cadence.md (overnight-pipeline rule + stall detection & forced structural pivot). At heartbeat startup, touch the run state first each tick and register this run with the watchdog loop type (so a silent death surfaces as STALE); unregister on completion. The watchdog only detects — it never acquits. Each tick also record the new-finding count via the iteration_log.py helper (resolve through the canonical .aris/tools → tools → $ARIS_REPO/tools → $ARIS_REPO/tools via ~/.aris/repo chain, integration-contract §2; warn-and-skip if unresolved): python3 "$ITER_LOG" note <root> <run_id> <phase> <n>. On the returned pivot=structural (stale ≥ 2) the nudge must change a STRUCTURAL constraint and pick an untried direction; on pivot=human (stale ≥ 4) flag for attention. Counting only — never a quality verdict.

End-to-end autonomous research workflow for: $ARGUMENTS

Constants

• AUTO_PROCEED = true — When true, every selection checkpoint is informational: report the choice and continue in the same turn. When false, ask for explicit user confirmation and end the turn at the checkpoint.

• ARXIV_DOWNLOAD = false — When true, /research-lit downloads the top relevant arXiv PDFs during literature survey. When false (default), only fetches metadata via arXiv API. Passed through to /idea-discovery → /research-lit.

• HUMAN_CHECKPOINT = false — When true, the auto-review loops (Stage 3) pause after each round's review to let you see the score and provide custom modification instructions before fixes are implemented. When false (default), loops run fully autonomously. Passed through to /auto-review-loop.

• REVIEWER_DIFFICULTY = medium — How adversarial the reviewer is. medium (default): standard MCP review. hard: adds reviewer memory + debate protocol. nightmare: GPT reads repo directly via codex exec + memory + debate. Passed through to /auto-review-loop.

• CODE_REVIEW = true — GPT-5.6-Sol xhigh reviews experiment code before deployment. Catches logic bugs before wasting GPU hours. Set false to skip. Passed through to /experiment-bridge.

• BASE_REPO = false — GitHub repo URL to use as base codebase. When set, /experiment-bridge clones the repo first and implements experiments on top of it. When false (default), writes code from scratch or reuses existing project files. Passed through to /experiment-bridge.

• COMPACT = false — When true, generates compact summary files for short-context models and session recovery. Passed through to /idea-discovery and /experiment-bridge.

• AUTO_WRITE = false — When true, automatically invoke Workflow 3 (/paper-writing) after Stage 4. Requires VENUE to be set. When false (default), Stage 4 generates NARRATIVE_REPORT.md and stops — user invokes /paper-writing manually.

• VENUE = ICLR — Target venue for paper writing (Stage 5). Only used when AUTO_WRITE=true. Options: ICLR, NeurIPS, ICML, CVPR, ACL, AAAI, ACM, IEEE_CONF, IEEE_JOURNAL.

• RENDER_HTML = true — When true (default), auto-render NARRATIVE_REPORT.md to HTML at Stage 4 completion via /render-html. Uses --no-review (this is an internal handoff doc to /paper-writing, not a reviewer-facing final artifact — the upstream Stage 3 auto-review loop already cross-model-reviewed the claims). Set false to skip, or pass — render html: false. Non-blocking: if /render-html fails or Codex MCP is unavailable, log the failure and continue — the HTML view is a nice-to-have, not a Stage 4 prerequisite.

• RESUMABLE = true — When true (default), the pipeline records per-stage state to .aris/runs/<run_id>.json so a crashed/interrupted run can resume via /research-pipeline — resume <run_id> instead of restarting. Stage status splits done (executor finished writing) from accepted (the stage's cross-model gate / deterministic verifier passed); resume re-validates any done-but-unaccepted stage. See shared-references/resumable-runs.md.

💡 Override via argument, e.g., /research-pipeline "topic" — AUTO_PROCEED: false, human checkpoint: true, difficulty: nightmare, code review: false, base repo: https://github.com/org/project, auto_write: true, venue: NeurIPS.

Checkpoint execution rule

Resolve AUTO_PROCEED once from $ARGUMENTS before Stage 1 and pass that resolved value to nested workflows.

• AUTO_PROCEED=true is non-blocking. A checkpoint is a progress update, not a question. State the result and the automatically selected next action, then continue executing in the same turn. Do not ask for confirmation, request user input, sleep, wait for silence, or end the turn at a checkpoint.
• AUTO_PROCEED=false is blocking. Present the options, ask the user, and end the turn. Resume only after an explicit reply.

Never implement auto-proceed as “ask, then continue if there is no response.” Once a turn ends, silence cannot resume the pipeline. The user can still interrupt a non-blocking run at any time.

This rule governs only AUTO_PROCEED-controlled selection checkpoints. If the user explicitly enables a Feishu interactive gate, that external approval or reply is an intentional blocking exception; wait for that user-controlled gate rather than treating it as a silence timeout. Feishu off/push-only modes remain non-blocking under AUTO_PROCEED=true.

Overview

This skill chains the entire research lifecycle into a single pipeline:

/idea-discovery → /experiment-bridge → /auto-review-loop → /paper-writing (optional)
├── Workflow 1 ──┤├── Workflow 1.5 ──┤├── Workflow 2 ───┤ ├── Workflow 3 ──┤

It orchestrates up to four major workflows in sequence. Workflow 3 (paper writing) is optional and controlled by AUTO_WRITE.

Resumable runs (— resume <run_id>)

This pipeline is long and can fail mid-run; it tracks per-stage state via run_state.py so you can resume instead of restarting (see shared-references/resumable-runs.md). Skip this whole section if RESUMABLE = false.

Resolve the helper via the canonical chain (integration-contract §2): .aris/tools/run_state.py → tools/run_state.py → $ARIS_REPO/tools/run_state.py → $ARIS_REPO/tools/run_state.py via ~/.aris/repo (warn-and-skip if unresolved — never block the pipeline).

Phases, in order: idea-discovery, experiment-bridge, auto-review-loop, summary, paper-writing.

• At start: if — resume <run_id> was passed, run run_state.py resume <root> <run_id> — it prints the first non-accepted phase; begin the pipeline at that stage (re-run a running/failed stage; re-audit a done-but-unaccepted stage). Otherwise derive <run_id> from the direction slug + date and run_state.py start <root> <run_id> --phases "idea-discovery,experiment-bridge,auto-review-loop,summary,paper-writing".

• Per stage: set <run_id> <phase> running on entry; set <run_id> <phase> done --artifact <path> once the stage's artifact is written.

• Mark accepted ONLY after the stage's gate passes — never on the executor's own say-so (run_state.py accept requires a recorded verdict id + reviewer):

  phase	what sets accepted	record as reviewer
  idea-discovery	Gate 1 cross-model jury / novelty-check passed	codex-gpt-5.6-sol + thread id
  experiment-bridge	experiments actually ran (jobs completed) — deterministic	deterministic:experiment-bridge
  auto-review-loop	the loop hit its positive STOP (score>=6 AND verdict∈{ready,almost} — codex's verdict)	codex-gpt-5.6-sol + final review trace id
  summary	NARRATIVE_REPORT.md written (+ rendered if RENDER_HTML) — deterministic	deterministic:summary
  paper-writing	submission audits passed (verify_paper_audits.sh exit 0) — deterministic	deterministic:verify_paper_audits.sh

If AUTO_WRITE = false (default), paper-writing is not part of this run: after summary is accepted, set <run_id> paper-writing skipped so resume reports COMPLETE instead of pointing forever at a pending stage. Record each accept verdict_id as a durable handle — the codex thread/trace id, or the path/sha of the deterministic verifier's report (e.g. the verify_paper_audits.sh output JSON) — not just the reviewer label.

A stage left done (gate failed/ambiguous, or the run crashed before the gate) is re-validated on the next resume — the acceptance obligation is never skipped.

Overnight heartbeat: stall detection → forced structural pivot

Only when an unattended heartbeat is driving this run (overnight /loop / CronCreate). Skip otherwise. Doctrine + rationale: shared-references/external-cadence.md → "Stall detection & forced structural pivot". This is a Type-A signal — it counts findings and changes direction, never judges quality.

Resolve the helper via the canonical chain (integration-contract §2), warn-and-skip if unresolved (never block the run):

ITER_LOG=".aris/tools/iteration_log.py"
[ -f "$ITER_LOG" ] || ITER_LOG="tools/iteration_log.py"
[ -f "$ITER_LOG" ] || ITER_LOG="${ARIS_REPO:-}/tools/iteration_log.py"
[ -f "$ITER_LOG" ] || { [ -z "${ARIS_REPO:-}" ] && [ -f "$HOME/.aris/repo" ] && ARIS_REPO="$(cat "$HOME/.aris/repo" 2>/dev/null)"; } || true
[ -f "$ITER_LOG" ] || ITER_LOG="${ARIS_REPO:-}/tools/iteration_log.py"
[ -f "$ITER_LOG" ] || { echo "WARN: iteration_log.py not resolved; skipping stall detection" >&2; ITER_LOG=""; }

Then, each heartbeat tick, record how many concrete new findings the current stage produced and read the returned pivot:

[ -n "$ITER_LOG" ] && python3 "$ITER_LOG" note "$ROOT" "$RUN_ID" "$STAGE" "$N_NEW_FINDINGS"
# → {"stale_count": N, "pivot": "none|structural|human"}

Act on pivot:

• none — keep going.
• structural (stale ≥ 2) — the next nudge must change a structural constraint (frame / objective / data / representation), not a tactical parameter, and pick a direction different from every one already tried. Record the chosen frame so future ticks can avoid it: python3 "$ITER_LOG" note "$ROOT" "$RUN_ID" "$STAGE" 0 --direction "<the new frame>".
• human (stale ≥ 4) — stop nudging blindly; flag for human attention (escalate, do not silently abandon).

The heartbeat may say "keep going / change direction," never "good enough" — every quality verdict still terminates in the cross-model jury (acceptance-gate.md).

Pipeline

Stage 1: Idea Discovery (Workflow 1)

If RESEARCH_BRIEF.md exists in the project root, it will be automatically loaded as detailed context (replaces one-line prompt). See templates/RESEARCH_BRIEF_TEMPLATE.md.

Invoke the idea discovery pipeline:

/idea-discovery "$ARGUMENTS" — AUTO_PROCEED: $AUTO_PROCEED

This internally runs: /research-lit → /idea-creator → /novelty-check → /research-review

Output: idea-stage/IDEA_REPORT.md with ranked, validated, pilot-tested ideas.

🚦 Gate 1 — Idea Selection:

After idea-stage/IDEA_REPORT.md is generated, present the top ideas.

If AUTO_PROCEED=true (non-blocking): report the selection and continue immediately in the same turn. Do not phrase the update as a question:

📋 Idea Discovery complete. Top ideas:

1. [Idea 1 title] — Pilot: POSITIVE (+X%), Novelty: CONFIRMED
2. [Idea 2 title] — Pilot: WEAK POSITIVE (+Y%), Novelty: CONFIRMED
3. [Idea 3 title] — Pilot: NEGATIVE, eliminated

AUTO_PROCEED: selected Idea 1 — [title]. Continuing to Stage 2.

If AUTO_PROCEED=false (blocking): present the same ranking, ask Recommended: Idea 1. Shall I proceed with implementation?, then end the turn. The user may:

• Approve the idea → proceed to Stage 2. /experiment-bridge reads refine-logs/EXPERIMENT_PLAN.md already generated by /idea-discovery.
• Request changes (e.g., "combine Idea 1 and 3", "focus more on X") → update the idea prompt with user feedback, re-run /idea-discovery with refined constraints, and present again.
• Reject all ideas → collect feedback on what's missing, re-run Stage 1 with adjusted research direction. Repeat until the user commits to an idea.
• Stop here → save current state to idea-stage/IDEA_REPORT.md for future reference.

⚠️ This gate waits for user confirmation when AUTO_PROCEED=false. When true, it auto-proceeds after presenting results. The rest of the pipeline (Stages 2-3) is expensive (GPU time + multiple review rounds), so set AUTO_PROCEED=false if you want a final review checkpoint before committing GPU resources.

Stage 2: Experiment Bridge (Workflow 1.5)

Once the idea is selected (automatically or by the user), delegate implementation and deployment to /experiment-bridge:

/experiment-bridge "$CHOSEN_IDEA_TITLE" — code review: $CODE_REVIEW, base repo: $BASE_REPO, compact: $COMPACT

💡 Queue routing is automatic: /experiment-bridge Phase 4 routes each milestone by job count — ≤5 jobs → /run-experiment, ≥10 jobs or teacher→student phase dependencies → /experiment-queue (with OOM retry, wave gating, crash-safe state). No manual override is needed.

What this does (fully autonomous):

1. Parses refine-logs/EXPERIMENT_PLAN.md — extracts milestones, run order, compute budget
2. Implements experiment code — extends pilot to full scale, follows existing codebase conventions
3. Cross-model code review — GPT-5.6-Sol xhigh reviews the implementation for logic bugs, incorrect metrics, and ground-truth misuse before any GPU time is spent
4. Sanity check — runs the smallest experiment first to verify the environment; auto-debugs failures (up to 3 attempts, with /codex:rescue fallback)
5. Deploys full experiments — auto-routes by job count (≤5 → /run-experiment, ≥10 → /experiment-queue with OOM retry, wave gating, crash-safe state)
6. Collects initial results — parses outputs, updates refine-logs/EXPERIMENT_TRACKER.md, runs /training-check if W&B is configured
7. Auto-plans ablations via /ablation-planner if main results are positive

Output:

• refine-logs/EXPERIMENT_RESULTS.md — structured results by milestone
• refine-logs/EXPERIMENT_TRACKER.md — updated run-by-run status
• EXPERIMENT_LOG.md (when COMPACT=true) — session-recovery-friendly log

Monitor progress (while experiments run):

/monitor-experiment [server]

Wait for /experiment-bridge to complete and report its handoff summary before proceeding.

Stage 3: Auto Review Loop (Workflow 2)

Once initial results are in, start the autonomous improvement loop:

/auto-review-loop "$ARGUMENTS — [chosen idea title], difficulty: $REVIEWER_DIFFICULTY"

What this does (up to 4 rounds):

1. GPT-5.6-Sol xhigh reviews the work (score, weaknesses, minimum fixes)
2. Claude Code implements fixes (code changes, new experiments, reframing)
3. Deploy fixes, collect new results
4. Re-review → repeat until (score ≥ 6/10 AND verdict ∈ {ready, almost}) or 4 rounds reached

Output: review-stage/AUTO_REVIEW.md with full review history and final assessment.

Stage 4: Research Summary & Writing Handoff

After the auto-review loop completes, prepare the handoff for paper writing.

Step 1: Write a final research status report (same as before).

Step 2: Generate NARRATIVE_REPORT.md from:

• IDEA_REPORT.md (chosen idea, hypothesis, novelty justification)
• Implementation details from the repo
• Experiment configs and final results
• AUTO_REVIEW.md (review history, weaknesses fixed, remaining limitations)

The narrative report must contain:

• Problem statement and core claim
• Method summary
• Key quantitative results with evidence for each claim
• Figure/table inventory (which exist, which need manual creation)
• Limitations and remaining follow-up items

Output: NARRATIVE_REPORT.md + research pipeline report.

# Research Pipeline Report

**Direction**: $ARGUMENTS
**Chosen Idea**: [title]
**Date**: [start] → [end]
**Pipeline**: idea-discovery → experiment-bridge → auto-review-loop

## Journey Summary
- Ideas generated: X → filtered to Y → piloted Z → chose 1
- Implementation: [brief description of what was built]
- Experiments: [number of GPU experiments, total compute time]
- Review rounds: N/4, final score: X/10

## Writing Handoff
- NARRATIVE_REPORT.md: ✅ generated
- Venue: [VENUE or "not set — run /paper-writing manually"]
- Manual figures needed: [list or "none"]

## Remaining TODOs (if any)
- [items flagged by reviewer that weren't addressed]

Stage 5: Paper Writing (Workflow 3 — Optional)

Skip this stage if AUTO_WRITE=false (default). Present the /paper-writing command for manual use:

📝 Research complete. To write the paper:
/paper-writing "NARRATIVE_REPORT.md" — venue: ICLR, AUTO_PROCEED: $AUTO_PROCEED

If AUTO_WRITE=true:

🚦 Gate 2 — Writing Checkpoint:

📝 Research pipeline complete. Ready for Workflow 3.

- Venue: [VENUE]
- Input: NARRATIVE_REPORT.md
- Manual figures required: [list or none]
- Next step: /paper-writing "NARRATIVE_REPORT.md" — venue: [VENUE], AUTO_PROCEED: $AUTO_PROCEED

Proceeding with paper writing...

Checks before proceeding:

• If VENUE is missing → stop and ask. Do NOT silently use a default venue.
• If manual figures are required → pause and list them. Wait for user to add them.

Then invoke:

/paper-writing "NARRATIVE_REPORT.md" — venue: $VENUE, AUTO_PROCEED: $AUTO_PROCEED

Pass the resolved AUTO_PROCEED explicitly so Workflow 3 cannot silently fall back to its own default mode.

This delegates to Workflow 3 which handles its own phases: /paper-plan → /paper-figure → /paper-write → /paper-compile → /auto-paper-improvement-loop

When Workflow 3 finishes, update the pipeline report with:

• Paper writing completion status
• Final PDF path (paper/main.pdf)
• Improvement scores (round 0 → round N)
• Remaining issues

Output: paper/ directory with LaTeX source, compiled PDF, and PAPER_IMPROVEMENT_LOG.md.

Render HTML view (auto, when RENDER_HTML = true)

After Stage 4 finalizes NARRATIVE_REPORT.md (before paper writing branches), invoke /render-html on the narrative report:

/render-html "NARRATIVE_REPORT.md" --no-review

--no-review is intentional: this is an internal handoff doc, not reviewer-facing — the claims it summarizes were already cross-model-reviewed in Stage 3's /auto-review-loop. Output: NARRATIVE_REPORT.html next to the MD, with embedded source SHA256.

Non-blocking: if /render-html fails (helper missing, file write error, etc.), log the failure and continue Stage 4 — the HTML view is a convenience artifact, not a pipeline prerequisite.

Skip this step if RENDER_HTML = false.

Output Protocols

Follow these shared protocols for all output files:

• Output Versioning Protocol — write timestamped file first, then copy to fixed name
• Output Manifest Protocol — log every output to MANIFEST.md
• Output Language Protocol — respect the project's language setting

Key Rules

• Large file handling: If the Write tool fails due to file size, immediately retry using Bash (cat << 'EOF' > file) to write in chunks. Do NOT ask the user for permission — just do it silently.

• The Stage 1 checkpoint is controlled by AUTO_PROCEED. When false, do not proceed without user confirmation. When true, report the top selection and continue in the same turn without asking or waiting.

• Stages 2-3 can run autonomously once the idea is selected. This is the "sleep and wake up to results" part.

• If Stage 3 ends at round 4 without positive assessment, stop and report remaining issues. Do not loop forever.

• Budget awareness: Track total GPU-hours across the pipeline. Flag if approaching user-defined limits.

• Documentation: Every stage updates its own output file. The full history should be self-contained.

• Fail gracefully: If any stage fails (no good ideas, experiments crash, review loop stuck), report clearly and suggest alternatives rather than forcing forward.

Typical Timeline

Sweet spot: Run Stage 1 in the evening, launch Stage 2-3 before bed, wake up to a reviewed paper.