Skip to content

Commit 03f28cd

Browse files
authored
Add ECP-7 Batch 23 phased replay study (#24)
Document the sealed phased replay-gradient experiment and its valid negative result.
1 parent 789127a commit 03f28cd

12 files changed

Lines changed: 346 additions & 16 deletions

File tree

‎README.md‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ The protocol uses no words or alphabet, but four meaning-free local symbols. For
2626
See [`docs/results-ecp6.md`](docs/results-ecp6.md) for the conclusion, [`docs/protocol-specification-ecp6.md`](docs/protocol-specification-ecp6.md) for the wire format and [`evidence/ecp6/report.md`](evidence/ecp6/report.md) for the compact confirmatory evidence.
2727

2828
ECP-7 development now tests how much of that result depends on the explicit
29-
one-factor-per-slot architecture. Its first twenty-two sealed batches are valid
29+
one-factor-per-slot architecture. Its first twenty-three sealed batches are valid
3030
development results but none passes the full gate. Batch 15 established the
3131
strong position-aware base: at 30,000 optimization steps it reaches 83.46% train
3232
exactness, 82.59% validation and 83.37% translator validation while using
@@ -46,7 +46,9 @@ exactness and injectivity. Batch 21 restricted replay to errors shared by all
4646
but its target error pool grew and train exactness still failed. The ECP-7
4747
confirmatory test remains sealed. Batch 22 routed shared-error replay only into
4848
senders; this reduced shared errors but increased cross-link fragmentation and
49-
regressed validation to 83.51%. See
49+
regressed validation to 83.51%. Batch 23 restored joint replay after the
50+
sender-only warmup. Validation remained 83.50%; total fragmented errors fell,
51+
but the shared-error pool immediately regrew and injectivity still failed. See
5052
[`docs/development-log-ecp7.md`](docs/development-log-ecp7.md).
5153

5254
## Build on this work
@@ -109,6 +111,7 @@ Do not translate those artifacts in place. New experiment configurations and all
109111
- [`config/ecp7-b20-development.yaml`](config/ecp7-b20-development.yaml) — late ordinary task replay on population-hard training meanings.
110112
- [`config/ecp7-b21-development.yaml`](config/ecp7-b21-development.yaml) — Batch 20 replay restricted to meanings failed by all 16 population links.
111113
- [`config/ecp7-b22-development.yaml`](config/ecp7-b22-development.yaml) — Batch 21 replay with receiver-parameter gradients disabled only on the replay branch.
114+
- [`config/ecp7-b23-development.yaml`](config/ecp7-b23-development.yaml) — Batch 22 sender-only replay through step 20,000 followed by restored joint replay gradients.
112115
- [`config/ecp7-positive-control-development.yaml`](config/ecp7-positive-control-development.yaml) — ECP-6 positive control on the ECP-7 split.
113116
- [`config/ecp0.yaml`](config/ecp0.yaml) — machine-readable configuration from step 1.
114117
- [`config/ecp1.yaml`](config/ecp1.yaml) — frozen configuration of the population trial.

‎config/ecp7-b23-development.yaml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
extends: ecp7-b22-development.yaml
2+
3+
experiment:
4+
title: ECP-7 phased population-shared hard replay
5+
arm: phased_population_shared_hard_replay
6+
objective: >-
7+
Test whether a sender-only replay warmup followed by restored joint replay
8+
gradients can preserve early shared-error separation while allowing
9+
receivers to realign with the moved sender codes.
10+
11+
training:
12+
global_hard_meaning_replay:
13+
enable_receiver_parameter_gradients_after_step: 20000

‎docs/AI_AGENT_START.md‎

Lines changed: 16 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ python3.12 -m venv .venv
4141
.venv/bin/ecp6 --config config/ecp6.yaml validate
4242
```
4343

44-
Expected baseline: 114 passing tests and split sizes `14336/1024/1024`.
44+
Expected baseline: 117 passing tests and split sizes `14336/1024/1024`.
4545

4646
## 4. Reproduce ECP-6
4747

@@ -71,7 +71,7 @@ This reruns a known experiment. It must not be presented as a new independent co
7171

7272
## 5. Continue ECP-7 scientifically
7373

74-
Do not modify ECP-0 through ECP-6 or the completed ECP-7 Batch 1 through Batch 22
74+
Do not modify ECP-0 through ECP-6 or the completed ECP-7 Batch 1 through Batch 23
7575
configs. ECP7-B1-I collapsed to 130–139 hard messages. ECP7-B2-I improved its
7676
soft objective but collapsed further to 85–104 hard messages. ECP7-B3-I applied
7777
that loss to straight-through hard messages and improved to 585–972 messages,
@@ -135,11 +135,16 @@ loss. Its shared-error pool fell from 1,745 to 1,513 at selection, but the total
135135
any-link pool grew to 3,197, sender agreement fell to 92.06% and validation
136136
regressed to 83.51%. Train thresholds and injectivity still failed, and the
137137
confirmatory ECP-7 test remains sealed.
138+
ECP7-B23-I restored joint replay gradients after the sender-only warmup. The
139+
shared-error pool reached 1,414 during sender-only routing, then jumped to 1,984
140+
at the first joint boundary and reached 2,085 at selection. The any-link pool
141+
fell to 2,660, but validation remained 83.50%, train exactness and injectivity
142+
still failed, and the confirmatory ECP-7 test remains sealed.
138143

139144
Continue with this sequence:
140145

141146
1. Read `docs/research-design-ecp7.md` and `docs/development-log-ecp7.md`.
142-
2. Define exactly one ECP7-B23 intervention and its failure criterion.
147+
2. Define exactly one ECP7-B24 intervention and its failure criterion.
143148
3. Register the variant and immutable configuration hashes before training.
144149
4. Add tests for every new invariant.
145150
5. Use only `smoke` and `develop`; keep the ECP-7 test split sealed.
@@ -152,16 +157,16 @@ Continue with this sequence:
152157

153158
## 6. Recommended next experiment
154159

155-
Batch 22 confirms that sender-only replay reduces shared errors, but receivers
156-
cannot follow the moving codes quickly enough: any-link failures rise and
157-
agreement falls. Batch 21 shows the opposite joint-routing behavior. The clean
158-
next question is whether routing should be phased rather than permanent. A clean
159-
ECP7-B23 progression is:
160+
Batch 23 confirms that restoring joint replay after sender-only warmup promptly
161+
recreates shared errors. Batch 22 confirms that staying sender-only instead
162+
creates cross-link fragmentation. The clean remaining routing question is
163+
whether receivers can catch up without letting the replay branch move sender
164+
codes again. A clean ECP7-B24 progression is:
160165

161-
- inherit the complete Batch 22 implementation, Batch 21 predicate and Batch 15
166+
- inherit the complete Batch 23 implementation, Batch 21 predicate and Batch 15
162167
base;
163-
- keep replay sender-only only during its registered steps 15,000–20,000
164-
warmup, then restore receiver-parameter replay gradients after step 20,000;
168+
- keep replay sender-only during steps 15,000–20,000, then make the additional
169+
replay branch receiver-only by detaching sender messages after step 20,000;
165170
- keep ordinary base-task updates to both senders and receivers unchanged;
166171
- keep weight `0.25`, the 64-meaning replay batch, 200-step refresh,
167172
seed-derived sampler, predicate, temperature and horizon unchanged;

‎docs/development-log-ecp7.md‎

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ this log.
3030
| ECP7-B20-I | Intervention | Late global replay of population-hard training meanings | Position-aware MLP | valid negative; new-best validation 83.45%, train and injectivity fail |
3131
| ECP7-B21-I | Intervention | B20 replay restricted to meanings failed by all 16 links | Position-aware MLP | valid negative; new-best validation 83.63%, shared-error pool grows |
3232
| ECP7-B22-I | Intervention | B21 replay gradients routed to senders but not receiver parameters | Position-aware MLP | valid negative; shared errors fall but link-specific fragmentation rises |
33+
| ECP7-B23-I | Intervention | B22 sender-only replay warmup followed by restored joint replay gradients | Position-aware MLP | valid negative; joint phase recreates shared errors without improving accuracy |
3334

3435
## Batch 1 preregistration
3536

@@ -1773,3 +1774,104 @@ gradients for replay after the registered warmup. This tests whether an initial
17731774
code-separation phase followed by joint decoder catch-up can combine B22's
17741775
shared-error reduction with B21's population alignment. No other parameter,
17751776
predicate or mechanism may change.
1777+
1778+
## Batch 23 preregistration
1779+
1780+
Preregistered: July 18, 2026 at 16:51:26 UTC<br>
1781+
Seed: `11`<br>
1782+
Maximum population steps: `30,000`<br>
1783+
Minimum selection steps: `15,000`<br>
1784+
Early-stopping patience: `5,000` steps<br>
1785+
Hard-meaning predicate: all `16` current population links have an incorrect factor<br>
1786+
Replay receiver-parameter gradients: **disabled through step 20,000; enabled from step 20,001**<br>
1787+
Ordinary-task receiver-parameter gradients: **enabled throughout**<br>
1788+
Replay weight: `0` through step `15,000`, linear to `0.25` at step `20,000`, then hold<br>
1789+
Replay batch size: `64` meanings, uniform with replacement<br>
1790+
Training-only pool refresh interval: `200` steps<br>
1791+
Replay sampler seed offset: `+1211`<br>
1792+
Raw configuration SHA-256: `8e0f7ed9b4d9a487bf0700be85e922142aafa3f8ccadf28326227bace6b78015`<br>
1793+
Effective configuration SHA-256: `60f46c913a95077a9f40df695adf6bb9b8788f581c83a12deb35c1f9ca015fb9`<br>
1794+
Split-SHA-256: `4947058c75ab07cb43a87eb82776b12cb2a7e2eeba7114de110d3b852cbc64cd`<br>
1795+
Test unsealed: **no**
1796+
1797+
ECP7-B23-I inherits Batch 22 and changes only the replay receiver-gradient
1798+
routing schedule. Its additional replay branch is sender-only during the
1799+
registered weight warmup and becomes joint after step 20,000. The ordinary
1800+
task remains joint throughout. An invariant test fixes the inclusive boundary:
1801+
receiver parameters are excluded at step 20,000 and included at step 20,001.
1802+
The positive control is rerun, and no alternative switch point, routing,
1803+
predicate or accompanying intervention is admitted into this batch.
1804+
1805+
## Batch 23 results
1806+
1807+
Positive-control run: `runs/ecp7-batch23-control-development/20260718T165354Z-ecp7-development`<br>
1808+
Intervention run: `runs/ecp7-batch23-intervention-development/20260718T165354Z-ecp7-development`<br>
1809+
Test unsealed: **no**
1810+
1811+
| Metric | Positive control | ECP7-B23-I |
1812+
|---|---:|---:|
1813+
| Population train, mean | 100% | **83.7197%** |
1814+
| Population train, worst link | 100% | 82.7148% |
1815+
| Population validation, mean | 100% | **83.4961%** |
1816+
| Population validation, worst link | 100% | 82.3242% |
1817+
| Universal translator, validation | 100% | **83.6182%** |
1818+
| Exact sender-message agreement | 100% | 92.31% |
1819+
| Unique messages per sender | 15,360 | 12,705–13,199 |
1820+
| Collision meanings per sender | 0 | 2,161–2,655 |
1821+
| Message entropy | 13.91 bits | 13.54–13.61 bits |
1822+
| Development gate | pass | **fail** |
1823+
1824+
The positive control passed every gate at 100%. ECP7-B23-I passes validation and
1825+
translator thresholds but fails both train thresholds and sender injectivity.
1826+
It does not improve a primary performance metric over Batch 21. The joint gate
1827+
fails and confirmatory access remains unauthorized.
1828+
1829+
All 101 optimization records through step 20,000 match Batch 22 exactly. The
1830+
routing invariant is also visible in every history record: replay receiver
1831+
gradients are disabled through step 20,000 and enabled from the next recorded
1832+
boundary at step 20,200. The all-link pool starts at 1,745 meanings, reaches a
1833+
minimum of 1,414 at step 18,200, and contains 1,579 meanings when the sender-only
1834+
phase ends at step 20,000.
1835+
1836+
Restoring joint replay immediately reverses that separation. The all-link pool
1837+
jumps to 1,984 at step 20,200, reaches 2,085 at the selected step 23,200, and
1838+
ends at 1,979 when patience stops training at step 28,200. The selected
1839+
checkpoint diagnostic confirms 2,085 meanings failed by all 16 links. It has
1840+
2,660 meanings failed by at least one link, lower than Batch 21's 2,755 and
1841+
Batch 22's 3,197, while the mean number of failed links among those meanings is
1842+
14.04. The receiver catch-up phase therefore reduces fragmented errors but
1843+
reconcentrates them into population-shared ambiguity.
1844+
1845+
Compared with Batch 21, mean validation falls by 0.13 percentage points,
1846+
worst-link validation by 0.29 points, translator validation by 0.02 points and
1847+
sender agreement by 0.66 points. Mean train is effectively unchanged. Batch 21
1848+
retains the strongest validation and agreement results. Compared with Batch 22,
1849+
B23 recovers 0.25 points of sender agreement and 0.02 points of translator
1850+
validation, but mean validation is effectively unchanged and the shared-error
1851+
pool is substantially larger.
1852+
1853+
Selected sender codebooks contain 12,960, 12,870, 12,705 and 13,199 unique
1854+
messages, with 2,400, 2,490, 2,655 and 2,161 collision meanings. Population
1855+
validation factor accuracies were `[99.87%, 99.91%, 98.44%, 85.25%]`;
1856+
universal-translator factor accuracies were
1857+
`[100%, 100%, 98.44%, 85.18%]`. The persistent size and texture bottlenecks are
1858+
unchanged.
1859+
1860+
Both sealed analyses report 65 matching artifact hashes, 16,384 validation-only
1861+
episodes, no confirmatory-test keys, valid local alphabets, and exactly 14
1862+
declared bits for every logged message.
1863+
1864+
## Batch 23 decision
1865+
1866+
ECP7-B23-I is a valid negative result. A sender-only warmup followed by joint
1867+
replay does not combine the advantages of B21 and B22: joint replay promptly
1868+
recreates shared errors, while accuracy and injectivity remain unchanged. The
1869+
confirmatory split remains sealed.
1870+
1871+
The routing schedule itself is now sufficiently isolated: permanent joint,
1872+
permanent sender-only and sender-only-to-joint replay all fail differently. A
1873+
future ECP7-B24 may preserve the Batch 23 schedule but make the second replay
1874+
phase receiver-only rather than joint. Detaching replay messages after step
1875+
20,000 would let receivers catch up to the sender-only code movement without
1876+
allowing the replay branch to move those codes again. The ordinary task must
1877+
remain joint throughout, and no other setting may change.

‎docs/research-design-ecp7.md‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -972,3 +972,44 @@ sealed-test requirement fails. No alternative gradient route, predicate,
972972
coefficient, replay batch, refresh, schedule, architecture, base loss,
973973
optimizer, duration, data, translator or seed may be tried inside Batch 22.
974974
Failure is recorded without confirmatory test access.
975+
976+
## Batch 23 preregistration — Phased shared-error replay routing
977+
978+
Status: **preregistered for sealed development**<br>
979+
Preregistered: July 18, 2026 at 16:51:26 UTC<br>
980+
Configuration: `config/ecp7-b23-development.yaml`<br>
981+
Raw configuration SHA-256: `8e0f7ed9b4d9a487bf0700be85e922142aafa3f8ccadf28326227bace6b78015`<br>
982+
Effective configuration SHA-256: `60f46c913a95077a9f40df695adf6bb9b8788f581c83a12deb35c1f9ca015fb9`
983+
984+
Batch 21's joint replay preserved population alignment but grew the shared-error
985+
pool. Batch 22's sender-only replay reduced that pool but increased fragmented
986+
any-link errors and reduced sender agreement. Batch 23 isolates whether the two
987+
routing modes are useful in sequence: first move ambiguous sender codes against
988+
fixed receivers, then allow receivers to catch up.
989+
990+
ECP7-B23-I inherits the complete Batch 22 implementation, Batch 21 predicate,
991+
Batch 20 replay mechanism, Batch 15 architecture, ordinary task batch,
992+
utilization objective, optimizer, data, temperature schedule, coefficient,
993+
replay batch, sampler, refresh, warmup, 30,000-step horizon, selection,
994+
translator and thresholds. The sole change is:
995+
996+
- the additional hard-replay branch keeps receiver parameters fixed through
997+
step 20,000 and restores receiver-parameter gradients from step 20,001.
998+
999+
The replay loss remains zero through step 15,000 and warms linearly through
1000+
step 20,000. The ordinary base task updates both senders and receivers at every
1001+
step. Mining still uses hard training predictions only. The complete
1002+
optimization trajectory through step 15,000 must match B22, B21, B20 and B15
1003+
exactly; through step 20,000 it must match B22.
1004+
1005+
The batch contains exactly two seed-11 runs:
1006+
1007+
1. the unchanged ECP-6 positive control;
1008+
2. ECP7-B23-I with phased population-shared hard replay.
1009+
1010+
All existing validity and development gates remain unchanged. The intervention
1011+
fails if any train, validation, translator, injectivity, channel-integrity or
1012+
sealed-test requirement fails. No alternative switch step, gradient route,
1013+
predicate, coefficient, replay batch, refresh, architecture, base loss,
1014+
optimizer, duration, data, translator or seed may be tried inside Batch 23.
1015+
Failure is recorded without confirmatory test access.

‎evidence/ecp7-development/README.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,13 @@ any-link pool grew to 3,197 and validation regressed to 83.51%. Sender-only
123123
routing therefore trades shared ambiguity for cross-link fragmentation. Exact
124124
identities and hashes are in `batch22-manifest.json`.
125125

126+
Batch 23 kept the Batch 22 sender-only warmup through step 20,000 and then
127+
restored joint replay gradients. The shared-error pool fell to 1,414 during the
128+
sender-only phase but jumped to 1,984 at the first joint phase boundary and
129+
reached 2,085 at selection. The any-link pool improved to 2,660, but validation
130+
remained 83.50% and injectivity still failed. Exact identities and hashes are in
131+
`batch23-manifest.json`.
132+
126133
Reproduce the development arms with:
127134

128135
```bash
Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
{
2+
"batch": "ECP7-B23",
3+
"confirmatory_test_unsealed": false,
4+
"config_effective_sha256": "60f46c913a95077a9f40df695adf6bb9b8788f581c83a12deb35c1f9ca015fb9",
5+
"config_raw_sha256": "8e0f7ed9b4d9a487bf0700be85e922142aafa3f8ccadf28326227bace6b78015",
6+
"control": {
7+
"artifact_hashes": {
8+
"effective_config.yaml": "4bfb976293f117d3ae35f790e4fd6371628be95d91febb334bc275dc5d6b66d4",
9+
"posthoc-analysis.json": "c00039440f19eed367c76239cfe81cdfc11b473e054f52cf6c84dfcb860466d9",
10+
"seed-11/metrics.json": "512fe2d262d16ecf4fa6ea4992250f1cd7e0c2647d7a16f5395267c835466d24",
11+
"summary.json": "527c8f7488bb795ed0866a5a62e817e78c5eee75052ef6bd9236201cd0cd8f3b"
12+
},
13+
"development_gate_passed": true,
14+
"population_train_mean_exact_match": 1.0,
15+
"population_validation_mean_exact_match": 1.0,
16+
"run_id": "20260718T165354Z-ecp7-development",
17+
"universal_translator_validation_exact_match": 1.0,
18+
"unique_messages_per_sender": [15360, 15360, 15360, 15360]
19+
},
20+
"intervention": {
21+
"artifact_hashes": {
22+
"effective_config.yaml": "7570f38b6af4fd25d92d8a79b9dbc2837f1d91c5cc089ba2f6b213cbb9583362",
23+
"posthoc-analysis.json": "1faf5ca496445a77017ee078e5830ed9ec574e2af18ef70aa03e3c2f2c0d618b",
24+
"seed-11/metrics.json": "3d97e0c07aaaf32747dda4a14a4181799bd3f094313f7a75d791132d9d5f72d1",
25+
"summary.json": "c045382367e7a4b476ad277de6526760039e44e087dab4d7cdbd46876821bb93"
26+
},
27+
"best_code_utilization_loss": -0.7546836137771606,
28+
"best_global_hard_meaning_replay_loss": 0.12714391946792603,
29+
"best_global_hard_meaning_replay_pool_size": 2085,
30+
"best_global_hard_meaning_replay_weight": 0.25,
31+
"best_step": 23200,
32+
"collision_meanings_per_sender": [2400, 2490, 2655, 2161],
33+
"development_gate_passed": false,
34+
"exact_sender_message_agreement": 0.923068576388889,
35+
"final_hard_meaning_replay_pool_size": 1979,
36+
"final_step": 28200,
37+
"initial_hard_meaning_replay_pool_size": 1745,
38+
"message_entropy_bits_per_sender": [13.57551840805443, 13.56497916977656, 13.537597236165908, 13.607817711443227],
39+
"minimum_hard_meaning_replay_pool_size": 1414,
40+
"minimum_hard_meaning_replay_pool_step": 18200,
41+
"population_train_mean_exact_match": 0.8371974400111607,
42+
"population_train_worst_link_exact_match": 0.8271484375,
43+
"population_validation_factor_accuracy": [0.9986572265625, 0.9991455078125, 0.984375, 0.8525390625],
44+
"population_validation_mean_exact_match": 0.8349609375,
45+
"population_validation_worst_link_exact_match": 0.8232421875,
46+
"replay_receiver_parameter_gradients_after_step": 20000,
47+
"run_id": "20260718T165354Z-ecp7-development",
48+
"selected_all_links_wrong_meaning_count": 2085,
49+
"selected_any_link_wrong_meaning_count": 2660,
50+
"selected_mean_incorrect_links_per_hard_meaning": 14.038722038269043,
51+
"universal_translator_validation_exact_match": 0.836181640625,
52+
"unique_messages_per_sender": [12960, 12870, 12705, 13199]
53+
},
54+
"seed": 11,
55+
"split_artifact_sha256": "676195997ee0c77d5c9565bae045dbb938b432a2e4e74b666233b0a41f924718",
56+
"split_sha256": "4947058c75ab07cb43a87eb82776b12cb2a7e2eeba7114de110d3b852cbc64cd"
57+
}

0 commit comments

Comments
 (0)