Experience-Driven Research Loop

Status: Proposed

Audience: maintainers and contributors implementing the next architecture phase

Scope: turn verified Research Runs into reusable experience that measurably improves later Research Runs

Owner: AI-Researcher maintainers

Last updated: 2026-07-31

Implementation and validation documents: Experience Gain next-round plan, Experience Gain V3 Phase A specification, Verified Research Memory implementation plan, Memory effectiveness evaluation protocol, and the historical V2 one-layer VQ protocol

Authority: CONTEXT.md is authoritative for domain identity and terminology. Durable Research Runtime and Stage Continuation is authoritative for Research Run lifecycle, restart, effect, artifact, budget, reuse, and terminal-completion semantics. This document governs the experience, verification, and causal feedback loop. Verified Research Memory is authoritative for Knowledge Candidate/Family/Record, Procedure Record, lifecycle, Knowledge/Evidence/Recall Input Snapshots, Decision Intent, Evidence Card, Recall Context, Recall Decision Outcome, distillation, retrieval, and memory evaluation. This document remains authoritative for Hypothesis, Experiment Attempt, Observation, Verification, and Experience Record; where older text conflicts with an authoritative source, that source prevails.

Context-Aware Tool Use and Governed Tool Effects is authoritative for Capability Catalog, Tool Interaction, Tool Decision Record, model-visible exposure, and ToolFeedback. ToolFeedback may contribute evidence to an Experiment Attempt Observation but never replaces that domain record.

1. Decision

AI-Researcher will add a closed research loop with one governing rule:

Generated output is not knowledge. Only independently verified observations may influence future research decisions.

The first implementation will deepen the existing runtime, evaluation, and memory Modules instead of adding another parallel agent framework. The loop will:

retrieve relevant, verified experience before a governed Intervention decision;
execute the existing research stages;
evaluate actual artifacts and metrics with a task-specific evaluator;
persist the Experiment Attempt and its provenance as immutable experience;
promote reusable knowledge only through an explicit verification policy;
expose the verified result with citations to a later Intervention decision, either in the same Research Run or a later Research Run.

The initial storage Implementation will use SQLite for structured records and the filesystem for large artifacts. Chroma remains a derived semantic index, not the source of truth. Redis, Neo4j, model-weight updates, and distributed execution are deliberately deferred.

2. Why this design is needed

The current repository already has most of the raw ingredients:

ordered research stages and artifact guardrails;
Research Run status, hooks, heartbeats, restart support, and traces;
Docker-backed execution;
structural evaluation;
session, episodic, semantic, code, paper, and tool memory primitives.

They do not yet form a trustworthy learning loop.

2.1 Artifact writers and validators disagree

The runtime guardrails and the flow writers do not currently share one schema:

Stage	Writer output	Guardrail expectation	Consequence
`implement`	manifest nested under `project_manifest`	top-level `exists` and `key_paths`	a real writer result can be rejected
`submit`	`submission_report`	`submit_result`	a real submission can be rejected

The supervisor tests construct JSON in the guardrail format directly, so they do not exercise the real writer-to-validator path. A versioned artifact contract is therefore the prerequisite for every later phase.

2.2 Evaluation is structural, not independent

The current evaluation Adapter converts plan and final_output into evidence. The evidence metric then uses token overlap to decide whether a claim is supported. This permits an output to support itself. An empty claim list also passes evidence coverage.

This evaluator is still useful as a structural quality check, but it cannot be the verification gate for scientific knowledge.

2.3 Memory is not connected to future decisions

MemoryStore keeps episodes in a process-local list. Episode retrieval is a substring match. MemoryAwareMetaChain records generic completion summaries, but the main research flows do not retrieve those episodes before hypothesis or plan generation. Consolidation currently copies summaries into fact-shaped objects.

The missing behavior is not another memory class. It is the verified transition:

experiment -> independent evaluation -> immutable experience
           -> comparative distillation disposition
           -> cited recall in a later typed Decision Point

2.4 Iteration is text-driven

The current refinement loop detects success through a literal "fully_correct": true substring. Later analysis/refinement steps continue to refer to the original submission result and do not always re-submit and re-evaluate the modified code.

Scientific progress must instead be driven by typed metrics, comparable budgets, evaluator versions, and artifact digests.

2.5 The two entry flows duplicate orchestration

run_infer_idea.py and run_infer_plan.py contain near-duplicate long pipelines. Adding recall, evaluation, persistence, and promotion separately to both would reduce Locality and create contract drift. The common pipeline must move behind one deep Module.

3. Goals and non-goals

3.1 Goals

Make every stage artifact conform to the same versioned schema used by its writer and validator.
Make independent evaluation a hard requirement for completed research and reusable knowledge.
Persist successful, neutral, negative, failed, and invalid Experiment Attempts across processes.
Preserve complete provenance from recalled knowledge to experiment artifacts and evaluator output.
Retrieve only scoped, reproducible, budgeted context for a later decision.
Support memory-off, record-only, recall-only, and closed-loop modes.
Make retries and restarts idempotent.
Measure whether prior verified experience improves later governed Intervention decisions and Research Run outcomes.
Keep the first implementation runnable without Redis, Neo4j, or a hosted vector database.

3.2 Non-goals for v1

Updating foundation-model weights.
Multi-machine orchestration.
General-purpose conversation memory.
Cross-user knowledge sharing.
Autonomous publication or submission.
Best-first tree search across many hypotheses.
Replacing Scientist-Bench.
Treating an LLM judge as ground truth when an executable evaluator exists.

Multi-Hypothesis candidate-search policy is intentionally deferred until a single Hypothesis can complete the verified loop reliably; this does not defer the shipped governed Intervention proposal/execution mechanism.

4. Domain language

The canonical, normative definitions live in CONTEXT.md. The summaries below are maintained for readability and may not narrow those definitions.

Research Run

The durable top-level execution of one canonical request under pinned workflow, continuation policy, model/tool configuration, Evaluation Contract, Budget Envelope, and retention/execution policy. A Research Run may contain multiple Experiment Attempts and Research Context Snapshot generations.

Hypothesis

A falsifiable proposed change with an expected effect, target metric, and conditions under which the claim is expected to hold.

Experiment Attempt

One execution of a Hypothesis bound to an immutable canonical AttemptSpec: Research Run/spec and exact Research Context Snapshot generation; source and dataset identities; Intervention; seed and allocation; Evaluation Contract; decision-input/Recall citations; environment/code/container/model/tool digests; and Intervention Catalog digest. Its Trial Provenance must match that AttemptSpec.

Intervention

The schema-valid executable configuration deliberately varied for one Experiment Attempt under a pinned Intervention Catalog.

Trial Provenance

The immutable source, Intervention, execution configuration, dataset, environment, Evaluation Contract, and evidence identity for an Experiment Attempt, bound to the canonical AttemptSpec digest.

Observation

A raw result produced by executing an Experiment Attempt. It includes metrics, artifact references, logs, exit status, and environment data. An Observation is not yet trusted.

Evaluation Contract

The task-specific definition of required artifacts, evaluator entrypoint, metrics, directions, baselines, repetitions, timeouts, and validity rules.

Verification Record

The immutable result of applying an Evaluation Contract to an Observation. It is written by the evaluator, not by the research agent.

Experience Record

An immutable record combining a Hypothesis, Experiment Attempt, Observation, Verification Record, analysis, and provenance. Invalid Experiment Attempts remain useful audit records but cannot become Knowledge.

Knowledge Record

A versioned conditional action-effect rule backed by comparable, independently verified Experience Records. Positive and negative results may both support a Knowledge Candidate; invalid or unverifiable results may not. Full semantics and lifecycle are owned by the Verified Research Memory design.

Recall Context

An immutable, bounded set of Evidence Cards selected for one typed Decision Intent under a Recall Input Snapshot. Full semantics are owned by the Verified Research Memory design.

Improvement Cycle

The transition from Recall Context through Hypothesis, Experiment Attempt, evaluation, recording, and promotion to a later Intervention decision in the same or a later Research Run.

5. Architecture vocabulary

This document uses the following architecture terms consistently:

Module — an Interface plus its Implementation.
Interface — everything callers must know, including invariants, configuration, ordering, and error modes.
Seam — where behavior can vary without editing the caller.
Adapter — a concrete Implementation satisfying an Interface at a Seam.
Depth — the leverage hidden behind a small Interface.
Leverage — behavior callers receive without learning internal details.
Locality — change and verification concentrated in one place.

6. Required invariants

These invariants are release blockers, not recommendations.

One artifact contract. Every writer output must validate against the exact versioned schema consumed by the runtime.
No self-evidence. Plans, analyses, judge prose, and final responses are never independent evidence for their own claims.
Evaluator authority. Only an evaluator satisfying the Evaluation Contract may write verified metrics or a Verification Record.
Completion follows the full durable chain. A Research Run is not COMPLETED until every completion-required valid Verification Record and durable Experience Record, one Recall Decision Outcome for every Run-owned Decision Intent, one Experience Distillation Receipt for every completion-required Experience, and the operational settlement barrier are committed. A receipt may queue or defer later comparative distillation. A valid negative Verification may complete; invalid evidence may not.
Immutable raw history. Hypotheses, Observations, and Verification Records are append-only. Corrections use superseding records.
Traceable knowledge. Every Knowledge Record names all source experience IDs, evaluator versions, artifact digests, and promotion policy version.
Comparable experiments. Primary metric, direction, baseline, budget, seed, evaluator/dataset versions, canonical AttemptSpec digest, and matching Trial Provenance source/config/evidence digests are required. A mixed code_revision is not an execution identity.
Private evaluation isolation. Agents cannot read private labels or evaluator-only data.
Scoped recall. Retrieval must enforce task/domain/dataset scope and must record the memory snapshot and ranking scores used.
Idempotent restart. Re-driving stable Research Run/Activity identities cannot duplicate an Experiment Attempt, Verification, Experience Record, or Experience Distillation Receipt.
Governed reuse identity. Reuse follows the durable design's stage policy and content + governance digests. Recall-dependent decision stages are Run-local, and Attempt-scoped Observation/Verification/Experience may never be reused as evidence for a new Experiment Attempt.
Bounded context. Recall has explicit item and token budgets. More matching memories do not imply a larger prompt.

7. Target flow

flowchart TD
    Q["Task and goal"] --> P["Prepare, survey, and method review"]
    P --> S["Freeze Research Context Snapshot"]
    S --> R["Recall for a typed Decision Point"]
    R --> H["Make a governed hypothesis/intervention decision"]
    H --> X["Implement and execute"]
    X --> O["Persist raw observation"]
    O --> V["Run independent evaluator"]
    V --> D{"Valid verification?"}
    D -- "No" --> A["Record invalid or failed experience"]
    D -- "Yes" --> E["Record verified experience"]
    E --> G{"Experience ingestion receipt"}
    G -- "Defer / not required" --> N["Experience durable; current run may continue"]
    G -- "Queue" --> N
    N -. "Asynchronous" .-> B["Later comparative distillation batch"]
    B --> J{"Distillation Decision"}
    J -- "Reject / defer" --> N
    J -- "Promote" --> K["Write governed cited memory record"]
    A --> C{"Budget remains?"}
    N --> C
    C -- "Yes" --> R
    C -- "No" --> T["Terminal Research Run result"]

The evaluator is intentionally outside the agent-controlled execution path. The agent may propose metrics, but it may not declare them verified. The comparative distillation branch may execute asynchronously; Research Run completion waits for the durable ingestion receipt, not for global promotion.

8. Deep Modules and Interfaces

The current repository exposes the following migration Interface. It is a compatibility Adapter behind Runtime Activities, not the target application ordering authority. LongTaskRuntime and StageContinuation own ordering; target memory Interfaces are KnowledgeDistiller.distill() and DecisionPointRetriever.recall().

class ExperienceLoop:
    def before_run(self, request: RecallRequest) -> RecallContext:
        """Return a bounded, cited snapshot for the next decision."""

    def after_run(self, completion: RunCompletion) -> LoopOutcome:
        """Evaluate, record, promote, and return the next action."""

This keeps old entrypoints compatible during migration. New callers do not use this Interface to schedule workflow stages. The following historical Modules explain the path that the current implementation is migrating from.

8.1 Stage Contract Module

Location: research_agent/runtime/artifacts.py

Responsibility: own all stage artifact schemas, serialization, validation, and legacy normalization.

Proposed envelope:

class StageArtifact(BaseModel):
    schema_version: Literal["1"]
    run_id: str
    task_id: str
    stage: StageName
    status: Literal["completed", "failed", "invalid"]
    created_at: datetime
    payload: dict[str, Any]
    artifact_refs: list[ArtifactRef] = []

Each stage has a typed payload model. File names remain compatible during migration, but writers and validators must import the same model. Guardrails validate semantics beyond presence, for example:

referenced files exist;
the main script is inside the experiment workspace;
required metrics are finite;
artifact hashes match;
the submission exit status is recorded.

Legacy artifacts are read through a LegacyStageArtifactAdapter; new writers never emit legacy shapes.

8.2 Experiment Ledger Module

Location: research_agent/inno/experience/ledger.py

Interface:

class ExperimentLedger(Protocol):
    def append_attempt(self, attempt: ExperimentAttempt) -> None: ...
    def append_observation(self, observation: Observation) -> None: ...
    def append_verification(self, verification: VerificationRecord) -> None: ...
    def append_knowledge(self, knowledge: KnowledgeRecord) -> None: ...
    def get_experience(self, experience_id: str) -> ExperienceRecord: ...
    def query(self, query: ExperienceQuery) -> list[ExperienceRecord]: ...
    def snapshot_id(self) -> str: ...

Adapters:

SQLiteExperimentLedger for real local Research Runs;
InMemoryExperimentLedger for contract and loop tests.

SQLite runs in WAL mode. Structured records live in the database; large logs, models, datasets, figures, and papers stay in the filesystem and are referenced by relative path plus SHA-256 digest.

The ledger is the canonical source of truth. Chroma can be deleted and rebuilt from it.

8.3 Evaluation Module

Location: research_agent/inno/experience/evaluation.py

Interface:

class Verifier(Protocol):
    def verify(
        self,
        contract: EvaluationContract,
        observation: Observation,
    ) -> VerificationRecord: ...

Adapters:

CommandVerifier executes a task evaluator in an isolated subprocess;
CallableVerifier runs deterministic Python evaluators in tests and small local tasks.

The current GoalDrivenEvaluator remains a StructuralEvaluator. It may reject incomplete plans or missing provenance, but it cannot promote Knowledge.

An Evaluation Contract is stored with the benchmark task:

schema_version: 1
contract_id: vq/one_layer_vq@1
task_id: one_layer_vq
entrypoint: python evaluate.py --attempt-dir "{attempt_dir}"
timeout_seconds: 900
repetitions: 3
required_artifacts:
  - metrics.json
  - run.log
primary_metric:
  name: fid
  direction: minimize
baseline: 42.10
validity:
  require_finite_metrics: true
  max_failed_repetitions: 0
private_data_dir: evaluator_private/

The evaluator receives the attempt directory and evaluator-only data. It writes a machine-readable result. Its stdout is diagnostic, not the authoritative metric.

8.4 Legacy Knowledge Gate Module (compatibility only)

Location: research_agent/inno/experience/knowledge.py

Interface:

class KnowledgeGate:
    def decide(
        self,
        experience: ExperienceRecord,
        related: list[ExperienceRecord],
    ) -> PromotionDecision:
        ...

The v1 policy is deterministic. Promotion requires:

a valid Verification Record;
all required artifacts and digests;
a known baseline and metric direction;
no unresolved validity violation;
a lesson that states conditions and evidence;
no direct contradiction with a higher-confidence record.

A verified failure may be promoted as negative Knowledge when it identifies a reproducible dead end. An execution failure without valid evaluation remains Experience only.

An LLM may draft a lesson, but the deterministic policy decides whether the record is eligible and preserves all citations.

This single-Experience Interface is permitted only in legacy_recall. It does not supply decision_memory or frozen_memory; comparative distillation, candidate isolation, lifecycle, and promotion are specified by Verified Research Memory.

8.5 Legacy Experience Retrieval Module (compatibility only)

Location: research_agent/inno/experience/retrieval.py

Interface:

class ExperienceRetriever(Protocol):
    def recall(self, request: RecallRequest) -> RecallContext: ...

Adapters:

KeywordExperienceRetriever provides an offline, deterministic baseline;
ChromaExperienceRetriever adds semantic candidate retrieval.

Both use the same deterministic final ranker:

score =
    relevance
  + outcome_utility
  + provenance_quality
  + recency
  + novelty
  - redundancy

The ranker filters scope before scoring and uses stable tie-breaking. Every returned item contains a citation ID, source experience IDs, score breakdown, and short lesson. Raw transcripts are not injected by default.

These Adapters remain baselines. Active and confirmatory paths use the Decision-Point Retrieval Interface and acceptance gates in Verified Research Memory; a hash-derived vector is never a production semantic Adapter.

8.6 Research Stage Adapters

Location: research_agent/runtime/research_pipeline.py

The legacy shared pipeline is decomposed into Runtime Activity Adapters for prepare, survey, method review, decision, materialize/execute, verify, record, and distillation. StageContinuation.plan() owns their pure ordering and LongTaskRuntime owns durable lifecycle; no stage Adapter is a second coordinator.

Idea-based and plan-based entry become strategy Implementations:

class ResearchIntentStrategy(Protocol):
    def build_hypothesis(self, request: RunRequest, recall: RecallContext) -> Hypothesis:
        ...


class ReferenceIdeationStrategy:
    ...


class ProvidedIdeaStrategy:
    ...

The two CLI files become thin application Adapters that parse arguments, select a strategy, and use only the public Runtime Interface. The migration-only LegacyFlowStageAdapter may temporarily wrap the old pipeline as one coarse Activity, but the durable plan removes MasterRuntime as an authority. RuntimeHooks remains an observer Seam and is not used as a durable knowledge commit path.

9. Data model

All persisted models include schema_version and reject unknown enum values. JSON serialization is stable and timestamps are UTC.

9.1 Hypothesis

Field	Required	Meaning
`hypothesis_id`	yes	deterministic UUID or content-derived ID
`task_id`	yes	evaluation task
`statement`	yes	falsifiable proposed effect
`mechanism`	yes	why the effect is expected
`expected_metric`	yes	metric and direction
`conditions`	yes	dataset/model/environment assumptions
`parent_experience_ids`	no	recalled sources used
`citations`	no	papers, code, or memory citations

9.2 Experiment Attempt

Field	Required	Meaning
`attempt_id`	yes	idempotency key
`run_id` / `run_spec_digest`	yes	immutable Research Run lineage
`snapshot_generation/ref/digest`	yes	exact Research Context Snapshot
`hypothesis_ref/payload_digest`	yes	tested Hypothesis
`intervention_ref/digest`	yes	governed executable change
`source/dataset_identity_digests`	yes	comparable code/data identity
`seed`	yes	deterministic seed
`attempt_allocation`	yes	immutable calls/tokens/wall/GPU allocation
`evaluation_contract_ref/digest`	yes	verifier contract
`decision_input_ref/digest/citations`	yes	exact Recall/decision context
`environment/code/container/model/tool_digests`	yes	execution identity
`intervention_catalog_digest`	yes	allowed decision surface
`attempt_spec_digest`	yes	canonical digest over all preceding fields

9.3 Trial Provenance

Field	Required	Meaning
`provenance_id` / `attempt_id`	yes	immutable lineage
`attempt_spec_digest`	yes	exact canonical AttemptSpec
`actual_execution_identity_digests`	yes	all spec-bound source/config/data/environment fields observed by runner
`manipulation_status`	yes	`baseline
`execution_receipt/evidence_bundle_digest`	yes	immutable physical evidence

9.4 Observation

Field	Required	Meaning
`observation_id`	yes	immutable ID
`attempt_id`	yes	source Experiment Attempt
`exit_code`	yes	execution result
`metrics`	yes	unverified raw metric observations
`artifact_refs`	yes	path, media type, size, and digest
`started_at` / `completed_at`	yes	timing
`environment_fingerprint`	yes	Python, packages, GPU, OS, container
`error`	no	structured failure information

9.5 Verification Record

Field	Required	Meaning
`verification_id`	yes	immutable ID
`observation_ref/digest`	yes	exact immutable Observation evaluated
`attempt_spec_digest` / `provenance_id`	yes	exact execution contract and Trial Provenance
`contract_id` / `contract_version`	yes	evaluator identity
`evaluator_digest`	yes	exact evaluator code
`valid`	yes	whether evaluation is trustworthy
`passed`	yes	contract success result
`outcome`	yes	positive, neutral, negative, or invalid
`verified_metrics`	yes	evaluator-produced metrics
`baseline_comparison`	yes	delta and direction
`violations`	yes	failed validity rules
`evidence_refs`	yes	evaluator outputs and artifact citations

9.6 Legacy Knowledge Record (schema version 1; audit/compatibility only)

Field	Required	Meaning
`knowledge_id`	yes	immutable ID
`scope`	yes	task, domain, dataset, model family
`lesson`	yes	concise reusable abstraction
`conditions`	yes	where the lesson applies
`outcome`	yes	positive or negative
`confidence`	yes	deterministic policy result
`source_experience_ids`	yes	provenance
`promotion_policy_version`	yes	reproducibility
`supersedes`	no	correction lineage

10. Persistence layout

This is the historical pre-Durable-Runtime layout. Durable Runtime exclusively owns Research Run state and artifact lifecycle; the Experiment Ledger stores artifact references and MUST NOT create a competing research_runs authority table. Current Knowledge/Recall schema revision 3 is specified in the Verified Research Memory implementation plan.

Historical project-level state:

.ai_researcher/
  experience.sqlite3
  semantic_index/
  artifacts/
    <run_id>/
      <iteration_id>/
        observation.json
        verification.json
        metrics.json
        run.log
        ...

The illustrated artifacts/ tree is Runtime-owned storage, not an Experience Ledger transaction domain. New ledger rows keep only immutable refs/digests.

Historical v1 SQLite tables:

hypotheses
experiment_attempts
observations
metric_observations
verification_records
experience_records
knowledge_records
knowledge_sources
recall_snapshots
recall_items
promotion_decisions

Foreign keys are enabled. Raw history tables reject updates and deletes through the public Interface. Retractions and superseding Knowledge Records are append-only.

11. Experience-loop overlay inside a Research Run

This section does not define a second lifecycle or status union. Run acceptance, snapshot preparation, Continuation Decisions, waits, retry/cancel/failure, and terminal ordering are owned by the durable design.

11.1 After snapshot freeze, before an Attempt decision

The accepted Research Run first completes or imports its pinned Research Context Snapshot through prepare → survey → method_review → freeze_context.
Before diagnose/intervention_plan opens the next Experiment Attempt, build and persist the sanitized DecisionContractView, then preallocate a typed DecisionIntent binding task scope, Decision Point, allowed-action digest, and logical artifact slot.
Resolve the exact RecallInputSnapshot, construct a DecisionRecallRequest, retrieve only verified/scoped/applicable records, and persist/read back the bounded DecisionRecallContext.
Generate or select the Hypothesis/Intervention with explicit source Experience IDs and a Manipulation Check, then close the Intent with one terminal RecallDecisionOutcome, including blocked/empty/degraded/rejected/ no-op/failure cases. The legacy RecallRequest is not used on this path.

A retrieval failure is a typed decision-stage result. empty is reserved for a healthy search with no eligible evidence. A pinned policy may continue only with an explicit degraded Recall Context for an allowed infrastructure/Adapter fallback; invalid request, private exposure, corruption, contamination, or identity mismatch is blocked and must wait/fail. None may silently change the Research Run lifecycle or bypass private-data and governance checks.

11.2 Open and execute the Experiment Attempt

Commit the Hypothesis and immutable canonical AttemptSpec before execution.
Persist typed stage artifacts and every external Effect/Physical Invocation through the durable Runtime; Recall-dependent decision stages are Run-local.
Keep private evaluator data isolated; the research agent may see only public metrics allowed by the Evaluation Contract.
After an execution receipt, atomically persist Trial Provenance matching all AttemptSpec fields and the raw Observation.

11.3 Verify, record, promote, and continue

Run the independent Verifier against Observation, Trial Provenance, and the canonical AttemptSpec digest/all pinned fields.
Persist the Verification Record.
Build and append the Experience Record.
Confirm every Run-owned Decision Intent has its unique terminal Recall Decision Outcome; the Outcome remains owned by its Runtime Activity even if no Experiment Attempt was opened.
Persist one Experience Distillation Receipt for each completion-required Experience: queued for comparison, deferred ineligible, or not required. It is not a promotion decision; comparative distillation may run later as a dedicated Runtime Activity.
Feed these committed records back to StageContinuation.plan(), which may open another Experiment Attempt, wait, or propose a terminal disposition under the pinned workflow and budgets.
Permit COMPLETED only after all required Verification/Experience pairs, every Run-owned Recall Decision Outcome, the corresponding per-Experience Distillation Receipts, and the durable design's operational settlement barrier. The canonical lifecycle union remains the one defined there.

A crash replays only the missing suffix: after the atomic Trial-Provenance/ Observation commit, verification is next; after Verification, Experience and its Distillation Receipt are next; completion also reconciles any missing Run-owned Recall Decision Outcome. Stable identities and fencing prevent duplicates.

12. Configuration and CLI

Proposed settings:

memory:
  mode: decision_memory   # off | record_only | legacy_recall | memory_shadow | decision_memory | frozen_memory
  store_path: .ai_researcher/experience.sqlite3
  knowledge_policy: policies/knowledge-v1.yaml
  retrieval_policy: policies/retrieval-v1.yaml
  knowledge_snapshot: latest-development
  max_recall_items: 3
  max_recall_tokens: 1200

evaluation:
  contract: benchmark/evaluators/vq/one_layer_vq.yaml
  require_verification_for_completion: true

budget_envelope:
  max_attempts: 3
  stop_on_primary_metric: true

Target CLI flags:

--memory-mode
--experience-store
--evaluation-contract
--knowledge-policy
--retrieval-policy
--knowledge-snapshot

--experience-mode remains a one-release compatibility alias for legacy behavior and cannot be combined with --memory-mode. Defaults during rollout:

existing entrypoints: record_only;
new writer/retrieval path: memory_shadow;
after memory acceptance: decision_memory for supported development tasks;
confirmatory evaluation: exact frozen_memory snapshot;
unsupported tasks fail closed when verification is required and no contract exists.

13. Cache correctness

Current cache identity is too coarse for trustworthy A/B evaluation. Cache and reuse identity are therefore delegated to the durable design, not a second experience-layer formula. Its stage key binds:

stage_contract_version / stage_name / canonical_scope_digest / input_digest
/ workflow_digest / continuation_policy_digest / runtime_adapter_contract_digest
/ reuse_governance_digest

AttemptSpec separately binds source, Intervention/config, dataset, environment, model/tool, Evaluation Contract, and decision-input/Recall digests. A mixed code_revision is compatibility metadata only and never a cache, provenance, or comparability identity. Run-local Recall-dependent decision stages may replay only under exact inputs; Attempt-scoped Observation/Verification/Experience is never reused as evidence for a new Experiment Attempt.

Unattended Research Runs never prompt interactively about stale cache entries. Cache policy is explicit: reuse, refresh, or disabled, but may never widen the stage reuse policy.

Memory-on and memory-off experiments use isolated namespaces and governance digests so no cross-arm state leaks.

14. Observability

The runtime emits these additional events:

recall_started
recall_completed
recall_failed
attempt_recorded
observation_recorded
verification_started
verification_completed
verification_failed
experience_recorded
knowledge_promoted
knowledge_rejected
iteration_scheduled
loop_completed

Required event fields:

Research Run, iteration, Experiment Attempt, and task IDs;
memory snapshot ID;
evaluation contract and evaluator versions;
artifact IDs and digests;
duration and resource usage;
terminal status and reason.

Secrets, full prompts containing credentials, private evaluation data, and raw personal information are redacted before events are written.

15. Success metrics

The north-star measurement is Experience Gain:

Experience Gain =
  score(closed-loop with prior verified experience)
  - score(same model, task, seed policy, and budget without recall)

Supporting metrics:

valid verification rate;
repeated-failure rate;
hypothesis redundancy rate;
successful experiments per GPU-hour;
successful experiments per million tokens;
Recall Context precision;
percentage of recalled lessons cited in decisions;
Knowledge Writer promotion precision and recall;
contradiction and retraction rate;
Research Run completion and restart-recovery rates;
Pass@k and variance across repeated Research Runs.

No claim of self-improvement is made until Experience Gain is positive across multiple tasks and repeated trials.

16. Test strategy

The Interface of each deep Module is the test surface.

16.1 Stage contract tests

Feed every real writer output into validate_stage_artifacts.
Cover all stages, not hand-written guardrail fixtures.
Reproduce and fix the current implement and submit mismatches.
Round-trip each versioned schema through JSON.
Verify legacy artifacts normalize but new writers never emit legacy shapes.

16.2 Ledger contract suite

Run the same contract tests against in-memory and SQLite Adapters:

append and reload after process restart;
idempotent writes;
immutable raw history;
superseding Knowledge Records;
concurrent writers;
foreign-key and digest validation;
stable snapshot IDs.

16.3 Evaluation trust tests

Empty claims do not imply scientific success.
A plan or final output cannot support its own claim.
Only evidence with provenance and a digest is accepted.
Metric direction and baseline are required.
Private labels are not visible in RunContext.
Invalid, timed-out, or non-finite results cannot be promoted.
Evaluator version changes produce a new Verification Record.

16.4 Retrieval tests

task and dataset scope filters;
positive and negative lesson retrieval;
no cross-task leakage by default;
deterministic ranking and tie-breaking;
item and token budget enforcement;
complete citation IDs and score breakdown;
index rebuild from the canonical ledger.

16.5 Closed-loop integration test

Create one deterministic local task requiring no LLM, Docker, network, or GPU:

iteration one produces a verified negative result;
the negative lesson is promoted;
iteration two receives that lesson with a citation;
the strategy changes the Hypothesis;
the evaluator returns a better primary metric;
the ledger shows exact lineage and no duplicate records.

Additional cases:

verification failure does not promote;
crash after Observation resumes at verification;
crash after verification resumes at Experience/Distillation Receipt commit;
memory-off and memory-on caches remain isolated;
both provided-idea and reference-ideation strategies cross the same pipeline Interface.

16.6 External evaluation

CI runs deterministic unit, contract, and fake-loop tests. A manually triggered or scheduled workflow runs a small real Docker experiment. GPU benchmarks remain outside normal PR CI until a dedicated runner exists.

17. Implementation plan

Sections 17–19 preserve the original pre-Phase-A roadmap and file map for design history; they are not current implementation instructions. Shipped status is recorded in the linked Phase A specification, response-surface calibration in the next-round plan, current memory work in the Verified Research Memory implementation/evaluation documents, and Runtime replacement/removal work in the durable implementation plan. In particular, the durable plan supersedes any older suggestion to make MasterRuntime itself the durable authority.

Each slice is independently reviewable and must leave the test suite green.

Phase 0 — Make existing contracts truthful

Deliverables

Add versioned stage artifact models.
Update all real writers and guardrails to use them.
Add writer-to-validator contract tests.
Fix implement and submit schema mismatches.
Separate structural evaluation from scientific verification terminology.
Remove self-generated plan/final text from independent evidence.
Make cache policy non-interactive for unattended Research Runs.

Exit criteria

Every stage written by a fixture flow passes its real guardrail.
No test fabricates a different schema from the writer.
A Research Run cannot be marked COMPLETED before every completion invariant in the durable design passes.

Phase 1 — Persist independently verified experience

Deliverables

Add domain models and schema versioning.
Add SQLite and in-memory ledger Adapters.
Add Evaluation Contract loading and validation.
Add callable and subprocess Verifier Adapters.
Record Experiment Attempts, Observations, Verification Records, and provenance.
Add one deterministic evaluator fixture.

Exit criteria

Records survive process restart.
Repeated commits are idempotent.
The evaluator is the sole writer of verified metrics.
Failed and invalid Experiment Attempts are retained but cannot become Knowledge.

Phase 2 — Close the single-hypothesis loop

Deliverables

Add deterministic Knowledge Gate.
Add keyword and Chroma retrieval Adapters.
Add Recall Context snapshots and citations.
Add ExperienceLoop.before_run and after_run.
Extract the shared research pipeline.
Integrate both entry strategies.
Add lifecycle events and configuration modes.

Exit criteria

The deterministic two-iteration integration test passes.
A later iteration can prove which verified experiences influenced it.
Restart at every durable transition commits exactly once.
Recall stays within scope and budget.

Phase 3 — Demonstrate Experience Gain

Deliverables

Define Evaluation Contracts for a representative Scientist-Bench subset.
Add memory-off versus closed-loop benchmark runner.
Execute repeated trials with identical model and budget settings.
Publish result artifacts, cost, variance, and failure analysis.
Add a ResearchClawBench Adapter after the internal subset is stable.

Exit criteria

Positive Experience Gain on more than one task.
Lower repeated-failure rate without materially increasing invalid results.
Results can be reproduced from stored config, code, data, evaluator, and memory snapshot IDs.

Phase 4 — Multi-Hypothesis candidate search policy

Phase A already ships governed single-decision Intervention proposal/execution. Only after Phase 3 may the system add best-first/bandit selection across multiple Hypotheses:

generate multiple Hypotheses;
evaluate candidates under a shared budget;
add best-first, bandit, or other search policy Adapter;
retain complete parent/child lineage;
compare search gain separately from memory gain.

18. File-level change map

File or Module	Action
`research_agent/runtime/artifacts.py`	new versioned stage contracts
`research_agent/runtime/criteria.py`	validate typed artifacts and verification state
`research_agent/runtime/master.py`	make verified completion explicit and resumable
`research_agent/runtime/research_pipeline.py`	new shared pipeline Module
`research_agent/runtime/context.py`	add Recall Context and lineage IDs
`research_agent/runtime/hooks.py`	emit closed-loop lifecycle events
`research_agent/inno/experience/models.py`	new domain models
`research_agent/inno/experience/ledger.py`	ledger Interface and SQLite Adapter
`research_agent/inno/experience/evaluation.py`	Evaluation Contract and Verifier Adapters
`research_agent/inno/experience/knowledge.py`	promotion policy and Knowledge construction
`research_agent/inno/experience/retrieval.py`	scoped recall and index Adapters
`research_agent/inno/experience/loop.py`	deep external closed-loop Interface
`research_agent/inno/evals/evaluator.py`	retain and rename structural checks
`research_agent/inno/evals/adapter.py`	remove self-evidence and emit typed observations
`research_agent/inno/evals/trace.py`	add artifact/evidence citations and lineage
`research_agent/inno/workflow/flowcache.py`	content-addressed cache identity
`research_agent/inno/memory/store.py`	keep session memory; stop treating it as durable experience
`research_agent/run_infer_idea.py`	thin CLI Adapter using reference-ideation strategy
`research_agent/run_infer_plan.py`	thin CLI Adapter using provided-idea strategy
`tests/test_runtime/`	real writer-to-validator and resume tests
`tests/test_experience/`	ledger, evaluation, retrieval, gate, and loop contract suites
`benchmark/evaluators/`	task Evaluation Contracts and evaluator code

19. Pull request sequence

Recommended tracer-bullet sequence:

Stage artifact contracts: typed artifacts, real writer contract tests, and current mismatch fixes.
Evaluation trust: structural/scientific split and self-evidence removal.
Experience models and ledger: SQLite persistence with contract suite.
Evaluation Contract: deterministic task evaluator and Verification Record.
Knowledge gate: verified positive/negative promotion with citations.
Recall: scoped retrieval, snapshots, budgets, and cache identity.
Closed-loop tracer bullet: one deterministic task across two iterations.
Shared pipeline extraction: move both entry strategies behind one Interface.
Scientist-Bench subset: memory-off versus closed-loop comparison.
Search policy: only after measurable single-loop improvement.

Do not combine the first eight items into one large rewrite. Each pull request must add an Interface-level test that survives later Implementations.

20. Rejected alternatives

Add Redis and Neo4j first

Rejected for v1. They add operational complexity without fixing verification, schema consistency, or provenance. SQLite and Chroma are sufficient to prove the loop.

Store full conversations as long-term knowledge

Rejected. Transcripts are low-density experience and may contain unsupported claims. Store structured, cited lessons backed by verified experiments.

Let the judge agent decide what becomes knowledge

Rejected. The judge may help interpret an experiment, but promotion eligibility must depend on an independent Verification Record and deterministic policy.

Add tree search before persistence and evaluation

Rejected. Parallel search multiplies invalid experiments when the observation and verification contracts are not trustworthy.

Replace all existing memory code

Rejected. Existing session and RAG memory remain useful. Durable experimental experience gets a distinct domain model so conversation state is not confused with verified knowledge.

21. Definition of done for v1

The experience-driven loop is v1-complete only when all statements below are demonstrably true:

Real flow writers and runtime validators share one versioned contract.
A task-specific evaluator independently verifies actual artifacts and metrics.
Research Run completion requires every completion-required persisted valid Verification/Experience pair, one Recall Decision Outcome per Run-owned Decision Intent, one Experience Distillation Receipt per completion-required Experience, and the operational settlement barrier.
Every Experiment Attempt, including failures, survives restart with complete AttemptSpec-bound Trial Provenance.
Only eligible, verified and comparable Experience can support active Knowledge.
A later Research Run can retrieve a bounded cited Recall Context for one typed Decision Intent under a pinned Recall Input Snapshot.
The deterministic two-iteration test proves end-to-end feedback.
Memory-on/off Research Runs have isolated caches and reproducible configuration.
The existing test suite and new contract suites pass in CI.
The claim made by a benchmark report is no stronger than the passed level in the Memory effectiveness evaluation protocol.

Until these conditions hold, documentation and releases should describe memory as experimental infrastructure rather than self-improvement.

FilesExpand file tree

experience-driven-research-loop.md

Latest commit

History

experience-driven-research-loop.md

File metadata and controls

Experience-Driven Research Loop

1. Decision

2. Why this design is needed

2.1 Artifact writers and validators disagree

2.2 Evaluation is structural, not independent

2.3 Memory is not connected to future decisions

2.4 Iteration is text-driven

2.5 The two entry flows duplicate orchestration

3. Goals and non-goals

3.1 Goals

3.2 Non-goals for v1

4. Domain language

Research Run

Hypothesis

Experiment Attempt

Intervention

Trial Provenance

Observation

Evaluation Contract

Verification Record

Experience Record

Knowledge Record

Recall Context

Improvement Cycle

5. Architecture vocabulary

6. Required invariants

7. Target flow

8. Deep Modules and Interfaces

8.1 Stage Contract Module

8.2 Experiment Ledger Module

8.3 Evaluation Module

8.4 Legacy Knowledge Gate Module (compatibility only)

8.5 Legacy Experience Retrieval Module (compatibility only)

8.6 Research Stage Adapters

9. Data model

9.1 Hypothesis

9.2 Experiment Attempt

9.3 Trial Provenance

9.4 Observation

9.5 Verification Record

9.6 Legacy Knowledge Record (schema version 1; audit/compatibility only)

10. Persistence layout

11. Experience-loop overlay inside a Research Run

11.1 After snapshot freeze, before an Attempt decision

11.2 Open and execute the Experiment Attempt

11.3 Verify, record, promote, and continue

12. Configuration and CLI

13. Cache correctness

14. Observability

15. Success metrics

16. Test strategy

16.1 Stage contract tests

16.2 Ledger contract suite

16.3 Evaluation trust tests

16.4 Retrieval tests

16.5 Closed-loop integration test

16.6 External evaluation

17. Implementation plan

Phase 0 — Make existing contracts truthful

Phase 1 — Persist independently verified experience

Phase 2 — Close the single-hypothesis loop

Phase 3 — Demonstrate Experience Gain

Phase 4 — Multi-Hypothesis candidate search policy

18. File-level change map

19. Pull request sequence

20. Rejected alternatives

Add Redis and Neo4j first

Store full conversations as long-term knowledge

Let the judge agent decide what becomes knowledge

Add tree search before persistence and evaluation

Replace all existing memory code

21. Definition of done for v1