Skip to content

CLAUDE.md

Latest commit

 

History

History
143 lines (130 loc) · 8.33 KB

CLAUDE.md

File metadata and controls

143 lines (130 loc) · 8.33 KB
Raw

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

llmXive is an automated platform for scientific discovery: a registry of specialist LLM agents — with occasional human guidance — advances ideas from a one-paragraph brainstorm to a peer-reviewed paper, committing every artifact, review, and decision to git. Two pipelines (research and paper) drive each project through a ~34-state lifecycle scaffolded with GitHub Spec Kit.

Governance lives in the constitution (.specify/memory/constitution.md) — six principles: Single Source of Truth, Verified Accuracy, Real-World Testing, Cost Effectiveness (Free-First), Fail Fast, and Convergent Review.

Repository Architecture

  • src/llmxive/: the Python package — agents, backends (Dartmouth Chat + local transformers), pipeline/state machinery, claims verification layer (specs 016–020), librarian, convergence engine, audits. Installable via pip install -e ".[dev]" (Python ≥ 3.11).
  • agents/registry.yaml: the SINGLE source of truth for agent → prompt / model / backend / budget assignments (~50 agents). Prompts live in agents/prompts/. All registry models must be free (paid_opt_in: false; enforced by tests/unit/test_config_consistency.py).
  • projects/: one directory per research project (PROJ-###-descriptive-name) holding idea/, specs, code, data, paper, and reviews for that project.
  • state/: canonical machine state — run logs, claims registry, librarian + judge caches, convergence cache, revisions.
  • specs/: Spec Kit feature specs for llmXive itself (the platform); the highest-numbered spec is the most recent work.
  • web/ is the dashboard source; docs/ is its deployed copy, regenerated wholesale from web/ by the Pages workflow — never hand-edit docs/. web/data/projects.json is built by src/llmxive/web_data.py from canonical state.
  • papers/, notes/, infra/, scripts/: papers + audit artifacts, dated engineering notes, deployment helpers, maintenance scripts.

Key Workflows

Project status management

  • Every reviewable stage runs the SAME identify → revise → re-review convergence cycle — ONE shared mechanism (llmxive.convergence.run_convergence), never re-implemented per stage (Constitution I + VI). The three rounds: R1 every reviewer raises concerns and a non-accept review MUST carry an actionable item (a "revise" with zero action items is rejected + resubmitted); R2 the reviser addresses every concern; R3 each reviewer signs off ONLY on whether its own R1 concerns were addressed — closed-set: re-review does NOT inject new concerns (a genuinely new issue is carried forward to the next stage's R1), which is what guarantees convergence within the cap. The deterministic backstops (fabricated-citation / unresolved-claim / spec-quality) still hard-block at convergence. Advancement requires unanimous panel acceptance within the 3-round per-step cap, else adaptive kickback to the prior stage. The bar is two-tier (Constitution 1.2.0): review stages (research_review/paper_review) require zero open concerns, while doc-authoring stages (specified/planned/tasked) may advance on writing-level-only residue (a requirement-or-worse concern always kicks back — the science gate is never relaxed). The tasked gate takes its verdict from the engine's closed-set panel (NOT a separate open-set re-analyze). There is NO accumulated point system; do not re-introduce one. Human and simulated-personality reviews are advisory inputs routed through stage-aware triage.
  • Review records are written under projects/<id>/reviews/research/ (research) and projects/<id>/paper/reviews/ (paper) as <reviewer_name>__<YYYY-MM-DD>__<stage>.md with YAML frontmatter validated against the review-record contract.
  • Review-stage dispatch is verdict-coverage-gated (spec 023): a complete, current verdict set is evaluated directly (no reviewer re-dispatch); an unconsumed revision_spec_path dispatches llmxive_implementer; advancement decisions are PERSISTED by the graph (never reduced to a stage name). Auto-revision rounds cap at advancement.MAX_REVISION_ROUNDS → honest terminal.
  • Human input is required for EXACTLY ONE thing: the publication DOI sign-off (a maintainer-vote GitHub issue parsed by the signoff-poll lane / llmxive.integrations.signoff_gate). Everything else is handled autonomously and NEVER parks at human_input_needed (graph.py has zero return Stage.HUMAN_INPUT_NEEDED). Infeasible ideas archive + bounded re-brainstorm → validator_rejected terminal. When the execution OR convergence fix-loop exhausts its per-tier round cap it ESCALATES the model tier (execution_status model_tier ladder: registered default → openai.gpt-oss-120b → optional LLMXIVE_EXECUTION_PAID_TIERS via the paid opt-in + credit guard) and retries the cap; when ALL tiers exhaust it routes back to planned/paper_planned with a DETERMINISTIC (no-LLM) re-plan report (artifacts produced + failing commands / unresolved panel concerns + an "adjust the approach" note) the planner ingests. The execution fix-loop also self-heals data-contract bugs: a cross-script schema mismatch (CSV columns, a missing intermediate file) is detected from the failure, the REAL produced artifact's schema is read, and the implementer is shown the exact mismatch + producer to fix (execution/data_contract.py, sibling to shared_contract.py). Engine failures file deduped GitHub issues and stay schedulable.
  • Every paper's compile/restyle/audit outcome lives in state/paper_status/<id>.json (spec 023 / FR-022): silent fallbacks are rejected at write time; audit defects feed a bounded repair loop through the same revision machinery.

Claims and verified accuracy

  • Every factual claim in generated artifacts is detected, registered, resolved against a real source, and cached (src/llmxive/claims/, state/claims/; specs 016–020). Empirical results must trace to harness-signed receipts (src/llmxive/results/) — never hallucinated.
  • All references must be validated against the live source (download/fetch, not memory) before commit.
  • The dedicated execution stage (src/llmxive/execution/stage.py) RUNS each project's analysis end-to-end and gates research_complete on real artifacts. GPU-bound runs the free CPU CI cannot satisfy are offloaded to Kaggle's free GPU (src/llmxive/execution/offload.py; issue #367) instead of being parked: an in-flight offload records an async tri-state in execution_status that NEVER bumps fix_rounds, so it polls to completion without escalating to human_input_needed. Gated on the KAGGLE_API_TOKEN secret (inert without it).

Testing

  • pytest tests/unit tests/contract tests/integration — offline suites.
  • LLMXIVE_REAL_TESTS=1 pytest tests/real_call — real-call suites (Constitution III: no mocks as the primary path). Heavy modules carry the slow marker; the per-PR CI gate runs -m "not slow", nightly runs all.
  • Real LLM calls need DARTMOUTH_CHAT_API_KEY; without Dartmouth access, the local transformers backend exercises the identical router path (e.g. LLMXIVE_JUDGE_TEST_BACKEND=local).
  • Prompt changes (agents/prompts/**) are gated by the promptfoo eval in eval/promptfoo/ (.github/workflows/prompt-eval.yml), whose assertions reuse the production parsers.

Working with the Repository

  • Single Source of Truth: before adding any function/config/prompt, search for an existing equivalent; modify in place, delete dead code, or refactor to a shared helper (Constitution I).
  • When code changes, update the corresponding docs (README, docstrings, website text) in the same commit; when dependencies change, update pyproject.toml.
  • Run the full verification suite before pushing; if tests fail repeatedly, fix the code — do not weaken the tests.
  • Use GitHub issues for all project tracking and communication.

For additional context about technologies to be used, project structure, shell commands, and other important information, read the current plan: specs/023-pipeline-e2e-completion/plan.md.