Skip to content

Commit bbb94d1

Browse files
committed
docs: add governed research architecture plans
1 parent e4ac154 commit bbb94d1

16 files changed

Lines changed: 17211 additions & 324 deletions

‎CONTEXT.md‎

Lines changed: 480 additions & 6 deletions
Large diffs are not rendered by default.

‎README.md‎

Lines changed: 65 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -57,11 +57,13 @@ selection problems:
5757
The purpose of memory is not to remember everything. It is to enable
5858
association, abstraction, and ultimately generalization.
5959

60-
Raw messages and artifacts are useful as experience, but they should not all
61-
become durable knowledge. The intended progression is:
60+
Raw messages and artifacts may supply evidence, but they should not all become
61+
durable knowledge. The intended governed progression is:
6262

6363
```text
64-
run events -> episodes -> reflection -> consolidated facts -> reusable knowledge
64+
immutable evidence -> comparative verification -> untrusted candidate
65+
-> governed semantic/procedural record -> bounded decision-point recall
66+
-> recorded use -> independently verified outcome
6567
```
6668

6769
### Verification is the boundary between output and knowledge
@@ -84,8 +86,10 @@ This fork extends the upstream research workflow with:
8486
- **Skill architecture** — discoverable `SKILL.md` bundles, lazy loading, JSON
8587
Schema tool descriptions, semantic search, lifecycle events, and A2A Agent
8688
Card export.
87-
- **Memory primitives** — typed session state, agent namespaces, episode
88-
storage, append-only event logs, RAG-backed memory, and fact consolidation.
89+
- **Legacy memory primitives** — typed session state, agent namespaces, episode
90+
storage, event logs, and source RAG. These are working/session/reference
91+
mechanics; their summaries and consolidated facts are not trusted Research
92+
Knowledge.
8993
- **Isolated experimentation** — Docker and browser environments for code,
9094
dataset, training, and evaluation workflows.
9195

@@ -94,11 +98,11 @@ The conceptual loop maps to the current implementation as follows:
9498
| Loop stage | Current implementation | Maturity |
9599
| --- | --- | --- |
96100
| Experience | Run traces, stage artifacts, event logs, agent episodes | Implemented |
97-
| Reflection | Judge feedback, analysis stage, memory consolidation | Partial |
101+
| Reflection | Judge feedback, analysis stage, legacy consolidation | Partial; candidate input only |
98102
| Hypothesis | Idea, survey, and planning agents | Implemented |
99103
| Experiment | Docker-backed implementation and training workflow | Implemented, environment-dependent |
100104
| Evaluation | Stage guardrails, goal-driven metrics, judge reports | Implemented |
101-
| Knowledge | RAG memory, consolidated facts, code/paper/tool memory | Partial |
105+
| Knowledge | Verified Experience ledger and proposed governed Knowledge/Procedure records | Ledger implemented; governed memory proposed |
102106
| Feedback | Reuse of retrieved memory in later decisions | Opt-in; not yet a fully automatic cross-run loop |
103107

104108
## Project Status
@@ -113,6 +117,12 @@ feedback across runs is also still partial. The current focus is making each
113117
transition observable, resumable, and verifiable before closing the
114118
self-improvement loop.
115119

120+
In particular, the current supervisor/restart primitives do **not** yet provide
121+
the proposed durable semantic continuation contract (RPO=0 committed state,
122+
fenced takeover, effect reconciliation, and stage-level resume). Phase A does
123+
provide governed executable Interventions and Trial Provenance, but no accepted
124+
experiment has yet demonstrated an AI-for-AI capability gain.
125+
116126
## Project Structure
117127

118128
```text
@@ -217,6 +227,11 @@ ai-researcher agent \
217227

218228
Skills are modular tool bundles discovered from `SKILL.md` manifests:
219229

230+
Today, production Agents still send stage-static callable lists;
231+
`search_tools()` below is not yet used to construct the next production model
232+
request. The governed per-turn migration is specified in the
233+
[Context-Aware Tool Use Design](docs/design/context-aware-tool-use.md).
234+
220235
```python
221236
from research_agent.inno.skills import skill_registry
222237

@@ -235,7 +250,8 @@ experiment planning, and memory tools.
235250

236251
## Memory
237252

238-
Memory can be added to MetaChain without changing its core loop:
253+
The following compatibility example adds process-level/session memory to
254+
MetaChain without changing its core loop:
239255

240256
```python
241257
from research_agent.inno.core import MetaChain
@@ -253,8 +269,11 @@ store.add_episode(
253269
)
254270
```
255271

256-
Memory is intentionally opt-in while retrieval quality, consolidation, and
257-
cross-run feedback are being hardened.
272+
This legacy Interface is intentionally opt-in. Episodes and consolidated facts
273+
from it cannot enter the trusted Knowledge Snapshot or confirmatory Recall
274+
Context. The target system uses comparative distillation, typed Decision Points,
275+
bounded Evidence Cards, Recall Decision Outcomes, and independently verified outcome
276+
gates.
258277

259278
## Validation
260279

@@ -275,15 +294,20 @@ Current local baseline:
275294
Some dependency deprecation warnings remain and are tracked separately from
276295
functional test failures.
277296

278-
## Verified Experience Loop (Experimental)
297+
## Verified Experience Loop (Experimental; legacy recall compatibility)
279298

280299
Both research entrypoints support four explicit modes:
281300

282-
- `off` — preserve the legacy one-run behavior;
283-
- `record` — independently verify and persist the attempt;
284-
- `recall` — retrieve scoped verified knowledge without recording;
285-
- `closed-loop` — recall, run, verify, promote eligible knowledge, and iterate
286-
within the configured budget.
301+
- `off` — execute one legacy Research Run without Experience Loop recall/recording;
302+
- `record` — independently verify and persist the Experiment Attempt;
303+
- `recall` — retrieve scoped legacy schema-v1 lesson records without recording;
304+
- `closed-loop` — use that legacy recall path, run, independently verify, apply
305+
the legacy KnowledgeGate, and iterate within the configured budget.
306+
307+
These modes do **not** yet implement canonical Verified Research Memory.
308+
Schema-v1 lessons/`RecallContext` map to the implementation plan's
309+
`legacy_recall` compatibility profile; they are not governed Knowledge Records,
310+
Evidence Cards, or canonical Decision-Intent Recall Contexts.
287311

288312
Recording modes require a task-specific evaluator contract:
289313

@@ -310,28 +334,44 @@ execution/evidence smoke contract with a zero baseline, not a scientific
310334
improvement threshold. The checked three-seed method-smoke report found no mean
311335
utilization improvement for the SimVQ-style variant. See
312336
[`docs/implementation/one-layer-vq-real-test.md`](docs/implementation/one-layer-vq-real-test.md) for
313-
the calibration and paired no-recall/recall protocol.
337+
the mechanics calibration and retired V2 paired protocol; current confirmatory
338+
gates live in the memory effectiveness evaluation protocol above.
314339

315340
In `closed-loop` mode, each Experiment Attempt receives an isolated stage and
316-
agent cache under `<cache>/attempts/iteration-NNN`. Verified Recall Context is
317-
rendered as cited decision guidance for the next attempt, so a retry cannot
318-
silently reuse the previous attempt's completed implementation and submission
319-
stages. Evaluator inputs are copied into immutable per-attempt evidence
320-
snapshots before verification, preventing a later attempt from overwriting an
321-
earlier Observation's artifacts.
341+
agent cache under `<cache>/attempts/iteration-NNN`. The legacy schema-v1
342+
`RecallContext` is rendered as cited compatibility guidance for the next
343+
Experiment Attempt, so a later
344+
Experiment Attempt cannot silently reuse the previous Experiment Attempt's
345+
completed implementation and submission stages. Evaluator inputs are copied
346+
into immutable per-attempt evidence snapshots before verification, preventing a
347+
later Experiment Attempt from overwriting an earlier Observation's artifacts.
322348

323349
## Remaining Validation Roadmap
324350

325351
1. Calibrate versioned task-specific Evaluation Contracts on independent real
326352
runs for a representative Scientist-Bench subset.
327-
2. Run paired no-recall and closed-loop trials on more than one real task.
353+
2. Run paired no-recall and closed-loop Research Run arms on more than one real task.
328354
3. Add scheduled Docker/GPU benchmark execution outside normal pull-request CI.
329355
4. Add a ResearchClawBench Adapter after the internal subset is stable.
330-
5. Validate memory gain separately from the implemented candidate-search gain.
356+
5. Validate memory gain separately from the implemented governed Intervention
357+
proposal/execution mechanism; neither gain is yet established by accepted
358+
scientific evidence.
331359

332360
The implementation contracts, module seams, rollout phases, and acceptance
333361
criteria are defined in the
334362
[Experience-Driven Research Loop Design](docs/design/experience-driven-research-loop.md).
363+
Canonical domain terms are defined in the [domain glossary](CONTEXT.md).
364+
The governed memory data plane is defined in the
365+
[Verified Research Memory Design](docs/design/verified-research-memory.md), its
366+
[implementation plan](docs/implementation/verified-research-memory-plan.md), and
367+
[effectiveness evaluation protocol](docs/implementation/memory-effectiveness-evaluation.md).
368+
The proposed durable execution and continuation architecture is defined in the
369+
[Durable Research Runtime Design](docs/design/durable-research-runtime.md) and its
370+
[implementation/acceptance plan](docs/implementation/durable-research-runtime-plan.md).
371+
Per-turn capability selection, governed tool Effects, migration, and acceptance
372+
are defined in the
373+
[Context-Aware Tool Use Design](docs/design/context-aware-tool-use.md) and its
374+
[implementation plan](docs/implementation/context-aware-tool-use-plan.md).
335375
The complete design and implementation document index is maintained in
336376
[`docs/README.md`](docs/README.md).
337377

‎docs/README.md‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
# Project documentation
22

33
This directory is the canonical home for maintained project documentation.
4+
The repository-wide canonical domain language is maintained in
5+
[`../CONTEXT.md`](../CONTEXT.md).
46

57
## Sections
68

@@ -15,10 +17,11 @@ This directory is the canonical home for maintained project documentation.
1517
documents in `docs/implementation/`; do not add planning documents to
1618
feature or benchmark source directories.
1719
2. Use lowercase kebab-case filenames.
18-
3. Start each document with its status, scope, owner, and last-updated date.
20+
3. Start each non-index document with its status, scope, owner, and last-updated
21+
date. Directory `README.md` index files are exempt.
1922
4. Link a design document to its implementation documents and link each
2023
implementation document back to the governing design.
2124
5. Update the relevant directory index when adding, replacing, or retiring a
2225
document.
2326
6. Keep experiment output and generated reports out of `docs/`; documentation
24-
may link to evidence stored under `benchmark/`.
27+
may link to evidence stored under `benchmark/`.

‎docs/design/README.md‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,15 @@
11
# Design documents
22

3-
Design documents define architecture, domain language, invariants, interfaces,
4-
and major technical decisions.
3+
Design documents use the canonical domain language in
4+
[`../../CONTEXT.md`](../../CONTEXT.md) and define architecture, invariants,
5+
interfaces, and major technical decisions.
56

67
| Document | Status | Purpose |
78
| --- | --- | --- |
8-
| [Experience-Driven Research Loop](experience-driven-research-loop.md) | Proposed | Governing design for turning verified research runs into reusable experience |
9+
| [Experience-Driven Research Loop](experience-driven-research-loop.md) | Proposed | Governing design for turning verified Research Runs into reusable experience |
10+
| [Durable Research Runtime and Stage Continuation](durable-research-runtime.md) | Proposed | Governing design for recoverable long tasks and Research-Run-scoped continuation |
11+
| [Verified Research Memory](verified-research-memory.md) | Proposed | Governing memory data-plane design for evidence distillation, lifecycle, decision-point recall, use trace, and Knowledge Snapshot |
12+
| [Context-Aware Tool Use and Governed Tool Effects](context-aware-tool-use.md) | Proposed | Governing design for deciding when and how to expose and execute tools |
913

1014
Implementation plans and code-level specifications live in
1115
[`../implementation/`](../implementation/README.md).

0 commit comments

Comments
 (0)