Implement governed experience gain v3

Author: LIHUA919

.gitignore

@@ -25,6 +25,17 @@ research_agent/evaluation/**/data/
 research_agent/cache/*
 research_agent/cache_*/*
 
+benchmark/data/
+benchmark/results/one_layer_vq_real_smoke/*/
+
+# Generated experiment state
+/benchmark/runs/
+/workplace_paper/
+/cache_soak*/
+/terminal_tmp/
+/soak_*.log
+/soak_*.pid
+
 research_agent/.env
 
 research_agent/*_tmp/*

CONTEXT.md

@@ -0,0 +1,78 @@
+# Verified Experimentation
+
+This context names the durable records and decisions used to test whether verified experience improves later research attempts. The detailed governing design remains `docs/design/experience-driven-research-loop.md`.
+
+## Language
+
+**Hypothesis**:
+A falsifiable proposed change with an expected metric effect and stated applicability conditions.
+_Avoid_: Idea, suggestion
+
+**Experiment Attempt**:
+One execution of a Hypothesis under a fixed dataset, source, Intervention, seed, budget, environment, and Evaluation Contract.
+_Avoid_: Retry, run
+
+**Intervention**:
+The schema-valid, executable configuration deliberately varied for one Experiment Attempt.
+_Avoid_: Tweak, prompt change, arbitrary config
+
+**Intervention Catalog**:
+The versioned allowlist of mutable knobs, permitted values, protected settings, and no-op policy for one decision point.
+_Avoid_: Config file, parameter dump
+
+**Manipulation Check**:
+The recorded determination of whether an Intervention changed the intended executable configuration and was observed by the runner.
+_Avoid_: Recall count, changed prompt
+
+**Trial Provenance**:
+The immutable identity of the source, Intervention, execution configuration, dataset plan, environment, Evaluation Contract, and resulting evidence for an Experiment Attempt.
+_Avoid_: Code revision, metadata
+
+**Observation**:
+The untrusted raw result, artifacts, metrics, timing, and exit status produced by an Experiment Attempt.
+_Avoid_: Result, finding
+
+**Evaluation Contract**:
+The task-specific, versioned rules that define required evidence, evaluator identity, metrics, validity, and success.
+_Avoid_: Test config, prompt rubric
+
+**Verification Record**:
+The immutable result of applying an Evaluation Contract to an Observation.
+_Avoid_: Judge response, score
+
+**Experience Record**:
+The immutable combination of a Hypothesis, Experiment Attempt, Observation, Verification Record, and analysis.
+_Avoid_: Memory
+
+**Knowledge Record**:
+A reusable abstraction backed by one or more verified Experience Records.
+_Avoid_: Reflection, summary
+
+**Recall Context**:
+A bounded, cited snapshot of Knowledge Records selected for one decision.
+_Avoid_: Chat history, full memory
+
+## Relationships
+
+- A **Hypothesis** has one or more **Experiment Attempts**.
+- Each **Experiment Attempt** executes exactly one **Intervention**.
+- An **Intervention Catalog** governs every valid **Intervention** at its decision point.
+- A **Manipulation Check** compares an **Intervention** with the preceding executable configuration.
+- **Trial Provenance** identifies the complete execution conditions of one **Experiment Attempt**.
+- An **Experiment Attempt** produces one **Observation**.
+- An **Evaluation Contract** produces one **Verification Record** for an **Observation**.
+- An **Experience Record** combines the preceding records without changing them.
+- A **Knowledge Record** cites one or more verified **Experience Records**.
+- A **Recall Context** supplies selected **Knowledge Records** to a later Intervention decision.
+
+## Example dialogue
+
+> **Developer:** “Treatment recalled two Knowledge Records. Does that pass the Manipulation Check?”
+>
+> **Domain expert:** “No. The Experiment Attempt must execute a different schema-valid Intervention, and its Trial Provenance must show that the runner observed the same configuration digest.”
+
+## Flagged ambiguities
+
+- “run” previously meant both a top-level Research Run and a single training execution. Use **Experiment Attempt** for the latter.
+- “code revision” previously mixed generated artifacts with source identity. Use **Trial Provenance** and its separate source and evidence digests.
+- “memory gain” previously conflated retrieval with behavior change. Use **Manipulation Check** for behavior change and reserve Experience Gain for verified outcome differences.

README.md

@@ -268,7 +268,7 @@ pytest -q
 
 Current local baseline:
 
-- 290 tests passing
+- 490 tests passing
 - 46 dynamically registered tools
 - 5 dynamically registered agents
 
@@ -301,11 +301,30 @@ The checked-in deterministic contract under
 `benchmark/evaluators/deterministic_score/` is a local integration fixture, not
 evidence of improvement on Scientist-Bench.
 
+The task-aligned `one_layer_vq` evidence contract under
+`benchmark/evaluators/one_layer_vq_smoke/` independently recomputes codebook
+utilization, perplexity, reconstruction MSE, and PSNR from raw arrays, and
+verifies the canonical CIFAR-10 test-image prefix by pixel digest. The local
+real-data runner is under `benchmark/real_smoke/one_layer_vq/`. It is an
+execution/evidence smoke contract with a zero baseline, not a scientific
+improvement threshold. The checked three-seed method-smoke report found no mean
+utilization improvement for the SimVQ-style variant. See
+[`benchmark/ONE_LAYER_VQ_REAL_TEST.md`](benchmark/ONE_LAYER_VQ_REAL_TEST.md) for
+the calibration and paired no-recall/recall protocol.
+
+In `closed-loop` mode, each Experiment Attempt receives an isolated stage and
+agent cache under `<cache>/attempts/iteration-NNN`. Verified Recall Context is
+rendered as cited decision guidance for the next attempt, so a retry cannot
+silently reuse the previous attempt's completed implementation and submission
+stages. Evaluator inputs are copied into immutable per-attempt evidence
+snapshots before verification, preventing a later attempt from overwriting an
+earlier Observation's artifacts.
+
 ## Remaining Validation Roadmap
 
-1. Add task-specific Evaluation Contracts for a representative
-   Scientist-Bench subset.
-2. Run paired memory-off and closed-loop trials on more than one real task.
+1. Calibrate versioned task-specific Evaluation Contracts on independent real
+   runs for a representative Scientist-Bench subset.
+2. Run paired no-recall and closed-loop trials on more than one real task.
 3. Add scheduled Docker/GPU benchmark execution outside normal pull-request CI.
 4. Add a ResearchClawBench Adapter after the internal subset is stable.
 5. Validate memory gain separately from the implemented candidate-search gain.