Skip to content

Commit ff49905

Browse files
CKwin26CKwin26
committed
Productize Research Forge v1 scientific execution
1 parent 28ee2ff commit ff49905

249 files changed

Lines changed: 84777 additions & 1030 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎docs/Research_Forge_现有功能与GitHub能力总表_2026-08-01.txt‎

Lines changed: 392 additions & 0 deletions
Large diffs are not rendered by default.

‎docs/benchmark-prediction-hidden-target-acceptance-2026-08-01.md‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
# benchmark_prediction_v1 真实隐藏目标验收
2+
3+
日期：2026-08-01
4+
5+
## 结论
6+
7+
`benchmark_prediction_v1` 已在真实外部任务 OpenML Task 39（Sonar v1）上完成隐藏目标验收。候选容器只接收 187 个带标签训练样本和 21 个不带标签测试样本；测试标签只进入随后启动的独立评估容器。两个容器均断网、只读根文件系统、非 root 运行，并使用彼此分离的只读输入挂载。
8+
9+
该结果把 `benchmark_prediction_v1` 的“候选提交与隐藏目标隔离”能力提升为窄范围 C4 真实案例证据。执行者仍为同一运营方，因此不构成 C5 外部独立验证。
10+
11+
## 数据与边界
12+
13+
- OpenML Task：39
14+
- Dataset：Sonar，dataset 40，version 1
15+
- OpenML MD5：`3ab630fbbfe25ab48b9bb47ce5759203`
16+
- 冻结材料化数据 SHA-256：`3765885e439f4576574f6de266b9179e807601eae4f671ca4cea3d6190b3768e`
17+
- 训练样本：187
18+
- 隐藏测试样本：21
19+
- 隐藏目标 SHA-256：`de1586e5aba2001cb03aeef357f95b3016384da06469ffd69fc62379d9375c3d`
20+
- 候选输入中存在目标文件：否
21+
22+
## 结果
23+
24+
| Arm | 冻结算法 | Accuracy | Submission SHA-256 |
25+
|---|---|---:|---|
26+
| baseline | majority class | 0.5714285714 | `20e6fb56ed159f71101cb3214775f3ff59fb677cde5470ba8b6be52cb98856f8` |
27+
| treatment | nearest centroid | 0.9047619048 | `e526c6f189fadf52a5a1adfe3edc3bab13e9ed3c0f776351bb428b38bd358184` |
28+
29+
- Beneficial effect：`+0.3333333333`
30+
- 冻结阈值：`0.0`
31+
- Verdict：`supported`
32+
- 独立评估容器与冻结参考评估器：两个 arm 均完全一致
33+
- 报告：验收运行生成的 `acceptance-report.json`（运行产物不提交到仓库）
34+
- 报告 SHA-256：`4dac07e63beb960f840f753a5c90974df9c627882afb626379fb251a3a90ca36`
35+
36+
## 平台落点
37+
38+
- `research_forge/benchmarks/hidden_prediction.py`
39+
- `scripts/run_hidden_prediction_acceptance.py`
40+
- `tests/test_hidden_prediction_acceptance.py`
41+
- `research_forge/profiles/benchmark_prediction.py`
42+
43+
## 允许声称与边界
44+
45+
允许声称：Research Forge 能在一个真实 OpenML 分类任务中，把候选提交与隐藏测试标签隔离，并由独立评估器重新计算主指标。
46+
47+
不能声称：该结果不证明所有 benchmark、回归任务或外部排行榜均受支持，也不是由外部组织执行的 C5 独立复现。科学结论只适用于冻结的 Task 39 split、两种确定性算法和已登记指标。

‎docs/clean-room-replay-acceptance-2026-07-31.md‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
# Controlled clean-room replay acceptance — 2026-07-31
2+
3+
## Scope
4+
5+
This acceptance run validates the operational clean-room launcher and two
6+
sealed reproduction packages. It does **not** claim an external organization,
7+
an independent cloud account, KMS/HSM receipt signing, RF-E2, C5, or scientific
8+
generalization beyond the controlled paired-comparison fixture.
9+
10+
Each candidate worker was a newly created `python:3.12-slim` Docker container
11+
with:
12+
13+
- `--network none`;
14+
- a read-only root filesystem;
15+
- all Linux capabilities dropped and `no-new-privileges` enabled;
16+
- a temporary `/tmp` filesystem;
17+
- only the signed reproduction package mounted read-only as scientific input;
18+
- one empty evidence directory mounted writeable;
19+
- no repository, source project, Workflow database, cache, credential,
20+
signing key, Docker socket, or original control-plane endpoint mounted.
21+
22+
The embedded runner independently recomputed the paired baseline mean,
23+
treatment mean, effect, interval, verdict, sample IDs, and completed RunCell
24+
counts from a hashed input asset. The host comparator then applied the frozen
25+
policy to the unsigned worker result.
26+
27+
## Results
28+
29+
| Case | Sealed archive SHA-256 | Manifest SHA-256 | Worker report SHA-256 | Result |
30+
|---|---|---|---|---|
31+
| A | `802bde8c18bfe53e3db706ed9e6fc1ec1a882bb1a6142705cd0ec7d86ffb226f` | `93a40f094b4290ad612c842d398e4c21b7aa7d6a7b6866b283ad8992f3e373d6` | `1b54aace238375a577d13c42154926ea713be32d7a93581e67bcb4569a884c81` | comparison passed; coverage full; all clean-room invariants true |
32+
| B | `8195a67ce99ab57f090ef58fdc0c7abe00162b99746cadca7d34e5a7c54f4d40` | `a2c2e1450dda1ef7767c4f3aa3232960478695f1f92f661100ff9253fddf7d17` | `01b1437dc4800870ec3c361d40f3d33df8aedf34fe57d5329c61b9170acc0dae` | comparison passed; coverage full; all clean-room invariants true |
33+
34+
The acceptance summary was written to an operator-selected directory outside
35+
the repository. Generated packages, logs, reports, and candidate outputs are
36+
intentionally not committed.
37+
38+
## Evidence grade
39+
40+
Both runs remain `RF-E1_package_verified` and `rf_e2_awarded=false`. The
41+
explicit blocker is: no separately operated verifier receipt backed by an
42+
independent KMS, HSM, or external organization. The worker report is unsigned
43+
by design, and the local control plane cannot promote it by self-attestation.
44+
45+
## Re-run
46+
47+
```powershell
48+
python scripts/run_clean_room_acceptance.py <stage3-completion.zip> <external-output-directory>
49+
```
50+
51+
The reusable launcher is `research_forge.clean_room_replay`, and the sealed
52+
example runner is `examples/clean-room-replay/replay.py`.

‎docs/drawio-runtime-revalidation-2026-07-31.md‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
# draw.io Desktop runtime revalidation — 2026-07-31
2+
3+
## Scope
4+
5+
This is a local runtime acceptance record for the Research Forge
6+
`drawio_backend.export_drawio` path. It does not validate next-ai-draw-io MCP
7+
editing, collaboration, or any scientific claim.
8+
9+
## Environment
10+
11+
- Operating system: Windows
12+
- draw.io Desktop: `30.3.14`
13+
- Executable: user-local draw.io Desktop installation
14+
- Backend: `research_forge.drawio_backend.export_drawio`
15+
- Export format: SVG
16+
- GPU mode: disabled
17+
- Electron profile: isolated per-export temporary user-data directory
18+
- Electron sandbox: enabled; `--no-sandbox` was not used
19+
20+
## Input and result
21+
22+
- Input: an existing Research Forge governed-workflow `.drawio` figure
23+
- Input SHA-256:
24+
`565c8a6fcf663122742261a3299915bd92695b97011cc9c5cf468fb308d55428`
25+
- Output: SVG, 139,968 bytes
26+
- Output SHA-256:
27+
`555f10e9996840315b282aa796ebac21bbbe1807b00d22c547dd6e8b0d914f9d`
28+
- Result: passed
29+
30+
The first revalidation attempt exposed two Windows-specific defects: Electron
31+
could not create its default cache in the managed workspace, and the launcher
32+
could return before the child process materialized the output. The backend was
33+
changed to use an isolated temporary profile and to wait for a non-empty
34+
artifact after process exit. A minimal generated diagram and the existing
35+
paper diagram both exported successfully after the change.
36+
37+
## Reproduction command
38+
39+
```powershell
40+
python -c "from pathlib import Path; from research_forge.drawio_backend import export_drawio; from research_forge.storage import sha256_file; p=export_drawio(Path(r'<input.drawio>'), Path(r'<output.svg>'), format='svg'); print(p.stat().st_size, sha256_file(p))"
41+
```
42+
43+
The generated SVG is a runtime artifact and is intentionally not committed.
44+
The durable acceptance evidence is this record plus the delayed-artifact
45+
regression test in `tests/test_drawio_backend.py`.

‎docs/existing-python-project-real-case-validation-2026-07-31.md‎

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
# Existing Python project real-case validation — 2026-07-31
2+
3+
## Scope
4+
5+
This acceptance validates a narrow Research Forge capability: an owner can
6+
allow-list a slice of an existing Python project, freeze the files and the
7+
expected metric boundary, and replay that sealed package in fresh offline
8+
Docker workspaces. It does **not** claim that arbitrary Python repositories
9+
can already be converted into a complete two-arm scientific experiment.
10+
11+
Runtime entrypoint:
12+
13+
```text
14+
research_forge.existing_project_replay
15+
scripts/run_existing_project_replay.py
16+
```
17+
18+
Deterministic controls:
19+
20+
- only explicitly listed files enter the package;
21+
- secret-bearing names, parent traversal and symlinks are rejected;
22+
- the host source path is not written to the package or report;
23+
- package identity is computed from the spec and file hashes, not time;
24+
- the input mount and container root are read-only;
25+
- the output mount is separate and writable;
26+
- networking, added devices and Docker socket mounts are disabled;
27+
- metrics come from a frozen parser, not from an agent interpretation;
28+
- every case runs twice in separate workspaces;
29+
- both replays must match the frozen historical expectation and each other.
30+
31+
## Real case 1 — stock disclosure extractor
32+
33+
Source class: pre-existing user stock-research repository.
34+
35+
Frozen slice:
36+
37+
- deterministic annual-report extraction implementation;
38+
- its package initializer;
39+
- its existing disclosure-extractor test module.
40+
41+
Observed result:
42+
43+
```json
44+
{
45+
"case_id": "stock-disclosure-extractor-real-project-v1",
46+
"status": "verified",
47+
"tests_run": 5,
48+
"pass_rate": 1.0,
49+
"historical_match": true,
50+
"cross_replay_equal": true,
51+
"clean_workspace_count": 2,
52+
"package_sha256": "4ed6d143856ed736294d4048ab5b0f5230c4a232a3b7919db0150c6864e65964"
53+
}
54+
```
55+
56+
## Real case 2 — adviser adapter inventory
57+
58+
Source class: pre-existing user investment-adviser repository.
59+
60+
Frozen slice:
61+
62+
- the existing adapter inventory implementation;
63+
- its package initializer;
64+
- its existing adapter-inventory test module.
65+
66+
Observed result:
67+
68+
```json
69+
{
70+
"case_id": "advisor-adapter-inventory-real-project-v1",
71+
"status": "verified",
72+
"tests_run": 1,
73+
"pass_rate": 1.0,
74+
"historical_match": true,
75+
"cross_replay_equal": true,
76+
"clean_workspace_count": 2,
77+
"package_sha256": "373717c5c465b19e28aa62e81e6b20167f663e864f1e05b8bb0e81be3fbb3bc9"
78+
}
79+
```
80+
81+
Both cases used image ID:
82+
83+
```text
84+
sha256:57cd7c3a7a273101a6485ba99423ee568157882804b1124b4dd04266317710de
85+
```
86+
87+
## Claim boundary
88+
89+
These are **real-case, same-host isolated replays**. They establish C4 for
90+
the narrow sealed-project replay utility. They are not C5 independent
91+
validation because the same Research Forge installation orchestrated both
92+
containers. They also do not yet prove that the full
93+
`existing_python_project_v1` Stage 2 → Stage 3 contract/build/verdict path is
94+
C4. Independent workers, external receipt verification and scientific
95+
baseline/treatment replay remain separate acceptance items.

‎docs/experiment-profile-bundles.md‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
# Experiment Profile Bundles
2+
3+
Stage 3 consists of a generic evidence-governance kernel and certified,
4+
versioned Experiment Profile Bundles. A Profile is not an application domain.
5+
It freezes a compatible combination of:
6+
7+
`Design + Outcome + Estimand + Estimator + Inference + Missingness +
8+
Multiplicity + Verdict policy`.
9+
10+
Reusable components live under `research_forge/profiles/components`. Formal
11+
execution accepts only combinations registered as a certified Bundle. Agents
12+
cannot assemble an untested statistical procedure at runtime.
13+
14+
## Current support
15+
16+
| Profile | Outcome and inference | Formal status |
17+
|---|---|---|
18+
| `computational_paired_comparison_v1` | Historical paired summary mean and normal approximation | Legacy frozen |
19+
| `computational_paired_comparison_v2` | Sample-level paired continuous effect; frozen cluster bootstrap | Certified |
20+
| `paired_binary_independent_v1` | Paired risk difference; exact McNemar over independent pairs | Certified |
21+
| `paired_binary_clustered_v1` | Paired risk difference; frozen cluster bootstrap | Certified |
22+
23+
The v1 Profile is immutable. Moving from v1 to v2 requires a new Research
24+
Contract and Scientific Successor; historical runs, results and verdicts are
25+
not silently reinterpreted.
26+
27+
## Modern paired result schema
28+
29+
Modern Profiles require `record_layout=summary_with_analysis_rows`. Each arm
30+
still reports its aggregate metric, denominator and sample IDs, and also emits
31+
the sample-level rows used by the independent evaluator:
32+
33+
```json
34+
{
35+
"accuracy": 0.75,
36+
"denominator": 2,
37+
"sample_ids": ["u1", "u2"],
38+
"analysis_rows": [
39+
{"sample_id": "u1", "value": 1, "task_id": "task-a"},
40+
{"sample_id": "u2", "value": 0, "task_id": "task-a"}
41+
]
42+
}
43+
```
44+
45+
For clustered Profiles, both arms must bind every pair to the same non-empty
46+
cluster. Pair omissions, duplicate identifiers, non-binary values and cluster
47+
disagreement block qualification.
48+
49+
## Frozen inference and verdicts
50+
51+
Bootstrap method, resampling unit, number of resamples, random seed and
52+
confidence level are Research Contract inputs. They cannot be chosen after
53+
results are visible. Modern superiority verdicts use the registered confidence
54+
interval boundary, not only a favorable point estimate.
55+
56+
Each certified Profile has typed contract validation, a deterministic Run Plan
57+
compiler, sample-level analysis construction and qualification, an estimator,
58+
inference, golden-vector assurance, an independent effect cross-check, a
59+
reproduction comparator and explicit cross-version reuse rules.
60+
61+
The reproduction control plane remains Profile-agnostic. It verifies and
62+
executes a frozen package, then delegates scientific result equivalence to the
63+
Profile comparator. RF-E1/RF-E2 evidence grades describe Research Forge
64+
assurance, not an international certification standard.

‎docs/github-hosted-public-replay-2026-08-01.md‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
# GitHub-hosted public replay acceptance — 2026-08-01
2+
3+
## Outcome
4+
5+
The public replay workflow ran on a fresh GitHub-hosted `ubuntu-latest`
6+
runner, downloaded both public Research Forge v1 acceptance packages, checked
7+
their frozen SHA-256 digests, executed both standalone standard-library
8+
verifiers, emitted a machine-readable receipt, and created a GitHub artifact
9+
provenance attestation for that receipt.
10+
11+
- Workflow run:
12+
<https://github.com/CKwin26/Auto-Agentic-Research-Forge/actions/runs/30652669858>
13+
- Source commit:
14+
`28ee2ffbb688528b70b3bf505876177e595863c6`
15+
- Workflow:
16+
`.github/workflows/public-replay-verification.yml`
17+
- Runner environment: `github-hosted ubuntu-latest`
18+
- Run result: success
19+
20+
## Replayed results
21+
22+
Successful package:
23+
24+
- SHA-256:
25+
`8c788149ac3a892e8e83d6f3d217d7671d50d507094611317ca273d7943a0eab`
26+
- Verifier: `passed: true`
27+
- Profile: `benchmark_prediction_v1`
28+
- Verdict: `supported`
29+
30+
Correctly blocked package:
31+
32+
- SHA-256:
33+
`7bb31c15bf45e6b832cbdc299315b4922b40d8524d75bf715677a88d4a78d227`
34+
- Verifier: `passed: true`
35+
- Protocol status: `blocked`
36+
- Run Specification count: `0`
37+
- Scientific Verdict created: `false`
38+
39+
## Durable receipt and attestation
40+
41+
- Public receipt:
42+
<https://github.com/CKwin26/Auto-Agentic-Research-Forge/releases/download/research-forge-v1-capability-acceptance-2026-08-01/external-host-replay-receipt.json>
43+
- Receipt SHA-256:
44+
`2917e281a1ad241290a3fe43642e2e0458a3a4b1ccd4ece89d1a9f924909e7d4`
45+
- Public offline attestation bundle:
46+
<https://github.com/CKwin26/Auto-Agentic-Research-Forge/releases/download/research-forge-v1-capability-acceptance-2026-08-01/sha256-2917e281a1ad241290a3fe43642e2e0458a3a4b1ccd4ece89d1a9f924909e7d4.jsonl>
47+
- Bundle SHA-256:
48+
`44787a156462ef604cb5e6c3e6872e1d3662c09ec4f32af5ff1581480b28b372`
49+
50+
`gh attestation verify` succeeded while enforcing repository
51+
`CKwin26/Auto-Agentic-Research-Forge` and signer workflow
52+
`CKwin26/Auto-Agentic-Research-Forge/.github/workflows/public-replay-verification.yml`.
53+
The verified certificate records the GitHub-hosted runner, `main` ref, source
54+
commit, run URL, workflow identity, and a Sigstore transparency-log timestamp.
55+
56+
## Claim boundary
57+
58+
This is real external-host execution and cryptographically verifiable artifact
59+
provenance. It is not an independent scientific-operator review because the
60+
workflow belongs to the same repository owner. The frozen receipt therefore
61+
states:
62+
63+
```json
64+
{
65+
"independent_scientific_operator": false,
66+
"c5_awarded": false
67+
}
68+
```
69+
70+
Research Forge may claim a signed GitHub-hosted replay receipt. It may not
71+
claim C5 independent reproduction until a separately controlled operator
72+
reviews the package, recomputes the metric, and signs an independent receipt.
73+
The independent handoff is published as
74+
<https://github.com/CKwin26/Auto-Agentic-Research-Forge/issues/6>.

0 commit comments

Comments
 (0)