Skip to content

Commit 7bd24e4

Browse files
authored
Record ECP-7 Batch 19 bounded replay pulse (#20)
1 parent d161f3f commit 7bd24e4

12 files changed

Lines changed: 348 additions & 26 deletions

File tree

‎README.md‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ The protocol uses no words or alphabet, but four meaning-free local symbols. For
2626
See [`docs/results-ecp6.md`](docs/results-ecp6.md) for the conclusion, [`docs/protocol-specification-ecp6.md`](docs/protocol-specification-ecp6.md) for the wire format and [`evidence/ecp6/report.md`](evidence/ecp6/report.md) for the compact confirmatory evidence.
2727

2828
ECP-7 development now tests how much of that result depends on the explicit
29-
one-factor-per-slot architecture. Its first eighteen sealed batches are valid
29+
one-factor-per-slot architecture. Its first nineteen sealed batches are valid
3030
development results but none passes the full gate. Batch 15 remains strongest:
3131
the position-aware design at 30,000 optimization steps reaches 83.46% train
3232
exactness, 82.59% validation and 83.37% translator validation while using
@@ -36,7 +36,9 @@ added late worst-factor pressure and regressed to 76.46% validation. Batch 17
3636
replayed globally mined training collisions and modestly improved code use, but
3737
still regressed to 77.09% validation. Batch 18 reduced replay to task-loss scale,
3838
recovering 80.71% validation and a new-best 84.06% translator score, but train
39-
exactness and injectivity still failed. The ECP-6 positive controls remain
39+
exactness and injectivity still failed. Batch 19 bounded replay to a pulse and
40+
reached 82.04% validation plus a new-best 80.57% worst-link validation, but
41+
again failed train exactness and injectivity. The ECP-6 positive controls remain
4042
perfect and the ECP-7 confirmatory test remains sealed. See
4143
[`docs/development-log-ecp7.md`](docs/development-log-ecp7.md).
4244

@@ -96,6 +98,7 @@ Do not translate those artifacts in place. New experiment configurations and all
9698
- [`config/ecp7-b16-development.yaml`](config/ecp7-b16-development.yaml) — late normalized factor-minimax pressure on the Batch 15 base.
9799
- [`config/ecp7-b17-development.yaml`](config/ecp7-b17-development.yaml) — late training-only global collision-pair replay on the Batch 15 base.
98100
- [`config/ecp7-b18-development.yaml`](config/ecp7-b18-development.yaml) — the Batch 17 replay mechanism with final weight reduced to 0.1.
101+
- [`config/ecp7-b19-development.yaml`](config/ecp7-b19-development.yaml) — the Batch 18 replay mechanism decayed back to zero after step 20,000.
99102
- [`config/ecp7-positive-control-development.yaml`](config/ecp7-positive-control-development.yaml) — ECP-6 positive control on the ECP-7 split.
100103
- [`config/ecp0.yaml`](config/ecp0.yaml) — machine-readable configuration from step 1.
101104
- [`config/ecp1.yaml`](config/ecp1.yaml) — frozen configuration of the population trial.

‎config/ecp7-b19-development.yaml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
extends: ecp7-b18-development.yaml
2+
3+
experiment:
4+
title: ECP-7 bounded global collision-replay pulse
5+
arm: bounded_global_collision_replay_pulse
6+
objective: >-
7+
Test whether decaying the Batch 18 replay coefficient after its early
8+
collision-reduction phase preserves code-use gains while restoring an
9+
unperturbed final semantic-refinement phase.
10+
11+
training:
12+
global_collision_replay:
13+
weight_decay:
14+
enabled: true
15+
start_step: 20000
16+
end_step: 25000
17+
final_weight: 0.0

‎docs/AI_AGENT_START.md‎

Lines changed: 29 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ python3.12 -m venv .venv
4141
.venv/bin/ecp6 --config config/ecp6.yaml validate
4242
```
4343

44-
Expected baseline: 102 passing tests and split sizes `14336/1024/1024`.
44+
Expected baseline: 105 passing tests and split sizes `14336/1024/1024`.
4545

4646
## 4. Reproduce ECP-6
4747

@@ -71,7 +71,7 @@ This reruns a known experiment. It must not be presented as a new independent co
7171

7272
## 5. Continue ECP-7 scientifically
7373

74-
Do not modify ECP-0 through ECP-6 or the completed ECP-7 Batch 1 through Batch 18
74+
Do not modify ECP-0 through ECP-6 or the completed ECP-7 Batch 1 through Batch 19
7575
configs. ECP7-B1-I collapsed to 130–139 hard messages. ECP7-B2-I improved its
7676
soft objective but collapsed further to 85–104 hard messages. ECP7-B3-I applied
7777
that loss to straight-through hard messages and improved to 585–972 messages,
@@ -114,11 +114,16 @@ ECP7-B18-I reduced replay weight to `0.1` and recovered 84.08% train, 80.71%
114114
validation and a new-best 84.06% translator validation. Validation and
115115
translator gates pass, but train and injectivity do not. Batch 15 retains the
116116
best validation result, and the confirmatory ECP-7 test remains sealed.
117+
ECP7-B19-I decayed replay back to zero from steps 20,000–25,000. It reached
118+
83.74% train, 82.04% validation, a new-best 80.57% worst-link validation and
119+
83.50% translator validation. The bounded pulse improved cross-link balance but
120+
still failed train thresholds and injectivity. Batch 15 retains the best mean
121+
validation result, and the confirmatory ECP-7 test remains sealed.
117122

118123
Continue with this sequence:
119124

120125
1. Read `docs/research-design-ecp7.md` and `docs/development-log-ecp7.md`.
121-
2. Define exactly one ECP7-B19 intervention and its failure criterion.
126+
2. Define exactly one ECP7-B20 intervention and its failure criterion.
122127
3. Register the variant and immutable configuration hashes before training.
123128
4. Add tests for every new invariant.
124129
5. Use only `smoke` and `develop`; keep the ECP-7 test split sealed.
@@ -131,26 +136,27 @@ Continue with this sequence:
131136

132137
## 6. Recommended next experiment
133138

134-
Batch 18 shows that replay at task-loss scale avoids B17's early collapse and
135-
improves train and translator scores, but validation peaks at step 22,400 and
136-
then declines while replay remains active. The clean next question is whether a
137-
bounded replay pulse can preserve early collision reduction and then return to
138-
unperturbed semantic refinement. A clean ECP7-B19 progression is:
139-
140-
- keep the same world and bit budget;
141-
- keep the complete B15 architecture, horizon and base losses;
142-
- keep the complete B17 training-only collision mining, replay batch,
143-
temperature and refresh mechanism;
144-
- keep replay weight `0` through step 15,000, warm only to `0.1` at step 20,000,
145-
then decay linearly to `0` at step 25,000 and hold;
146-
- measure injectivity, validation composition and new-reader induction;
147-
- rerun the ECP-6 architecture as the frozen positive control.
148-
149-
Continue to use training meanings only for pair mining. Do not change the
150-
mining source, pair sampler, architecture, base utilization, optimizer, data,
151-
temperature, duration, validation cadence or development thresholds, and do
152-
not combine the schedule change with factor reweighting, variable-length
153-
messages or negotiation.
139+
Batches 17–19 show that collision-pair replay changes code use but does not
140+
stably separate the remaining collisions. Batch 19 improves validation only
141+
after replay is removed, while known-meaning train exactness remains near 84%.
142+
The clean next question is whether the residual error is concentrated in a
143+
sparse set of hard training meanings that uniform task batches do not repair.
144+
A clean ECP7-B20 progression is:
145+
146+
- keep the same world, bit budget, Batch 15 architecture and 30,000-step horizon;
147+
- do not enable collision replay or factor reweighting;
148+
- periodically evaluate only the training codebook and deterministically mine
149+
meanings reconstructed incorrectly by the current population;
150+
- add ordinary reconstruction updates on one fixed, registered number of mined
151+
meanings after step 15,000, using a separate seed-derived sampler;
152+
- keep the ordinary random task batch and all base objectives unchanged;
153+
- measure train exactness, injectivity, validation composition and new-reader
154+
induction, and rerun the frozen ECP-6 positive control.
155+
156+
Preregister the mining score, refresh interval, replay batch size, coefficient
157+
and schedule before training. Do not combine hard-meaning replay with collision
158+
pressure, a new architecture, factor-specific weights, optimizer changes,
159+
longer training, variable-length messages or negotiation.
154160

155161
## 7. Definition of done for any contribution
156162

‎docs/development-log-ecp7.md‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ this log.
2626
| ECP7-B16-I | Intervention | B15 with late normalized factor minimax | Position-aware MLP | completed; gate failed |
2727
| ECP7-B17-I | Intervention | B15 with late global collision-pair replay | Position-aware MLP | completed; gate failed |
2828
| ECP7-B18-I | Intervention | B17 replay with final weight 0.1 | Position-aware MLP | completed; gate failed |
29+
| ECP7-B19-I | Intervention | B18 with replay decayed to zero after step 20,000 | Position-aware MLP | valid negative; validation 82.04%, train and injectivity fail |
2930

3031
## Batch 1 preregistration
3132

@@ -1353,3 +1354,103 @@ it at zero thereafter. This tests whether a bounded replay pulse can preserve
13531354
the early collision reduction while returning the final phase to pure B15 task
13541355
and utilization refinement. Mining, pair batch, refresh, architecture, base
13551356
losses, optimizer, duration, data, translator and thresholds must not change.
1357+
1358+
## Batch 19 preregistration
1359+
1360+
Preregistered: July 18, 2026 at 15:15:36 UTC<br>
1361+
Seed: `11`<br>
1362+
Maximum population steps: `30,000`<br>
1363+
Minimum selection steps: `15,000`<br>
1364+
Early-stopping patience: `5,000` steps<br>
1365+
Replay weight: `0` through step `15,000`, linear to `0.1` at step `20,000`, linear to `0` at step `25,000`, then hold<br>
1366+
Replay pairs per sender and update: `32`<br>
1367+
Training-codebook refresh interval: `200` steps<br>
1368+
Relaxed replay temperature: `1.0`<br>
1369+
Raw configuration SHA-256: `dab2cd0dadbadb47a10e2c45a720611dc91d140d3783f0ca1a537e4e8a8549c0`<br>
1370+
Effective configuration SHA-256: `cbd144a5fce451315a7d5fb322a11f2f29c32514fc22fbe06f46285aef677d13`<br>
1371+
Split-SHA-256: `4947058c75ab07cb43a87eb82776b12cb2a7e2eeba7114de110d3b852cbc64cd`<br>
1372+
Test unsealed: **no**
1373+
1374+
ECP7-B19-I is identical to Batch 18 through step 20,000. It then changes only
1375+
the replay coefficient, decaying from `0.1` to zero by step 25,000. Mining
1376+
continues for diagnostics, but the final 5,000 steps receive no replay gradient
1377+
or sampler draws. This directly tests whether the early code-use gain can be
1378+
retained while the final phase returns to the unchanged B15 objectives. The
1379+
positive control is rerun, and no alternative pulse or accompanying
1380+
intervention is admitted into this batch.
1381+
1382+
## Batch 19 results
1383+
1384+
Positive-control run: `runs/ecp7-batch19-control-development/20260718T151853Z-ecp7-development`<br>
1385+
Intervention run: `runs/ecp7-batch19-intervention-development/20260718T151853Z-ecp7-development`<br>
1386+
Test unsealed: **no**
1387+
1388+
| Metric | Positive control | ECP7-B19-I |
1389+
|---|---:|---:|
1390+
| Population train, mean | 100% | **83.7385%** |
1391+
| Population train, worst link | 100% | 82.7148% |
1392+
| Population validation, mean | 100% | **82.0435%** |
1393+
| Population validation, worst link | 100% | **80.5664%** |
1394+
| Universal translator, validation | 100% | **83.4961%** |
1395+
| Exact sender-message agreement | 100% | 91.81% |
1396+
| Unique messages per sender | 15,360 | 12,709–13,200 |
1397+
| Collision meanings per sender | 0 | 2,160–2,651 |
1398+
| Message entropy | 13.91 bits | 13.54–13.61 bits |
1399+
| Development gate | pass | **fail** |
1400+
1401+
The positive control again passed every gate at 100%. ECP7-B19-I passed the
1402+
validation and universal-translator thresholds, and its 80.57% worst-link
1403+
validation is the strongest ECP-7 worst-link result so far. It still failed
1404+
both known-meaning train thresholds and every sender remained non-injective.
1405+
The joint gate therefore fails and confirmatory access remains unauthorized.
1406+
1407+
All 101 logged trajectory records through step 20,000 match Batch 18 exactly.
1408+
Replay then decayed as registered: weight was `0.06` at step 22,000, `0.02` at
1409+
step 24,000 and zero from step 25,000 onward. At zero weight the logged replay
1410+
loss is also zero and the sampler consumes no draws; deterministic full-codebook
1411+
mining remains visible only through diagnostic collision-pair counts.
1412+
1413+
Validation rose from 81.59% at the end of the decay to the selected 82.04% at
1414+
step 26,200, with replay fully inactive. It later fluctuated and finished at
1415+
81.20% at step 30,000. The selected checkpoint had task loss `0.06103`,
1416+
utilization loss `-0.72858`, and mined collision-pair counts of 2,674, 2,645,
1417+
2,921 and 2,352. The run reached the registered maximum horizon rather than an
1418+
early stop.
1419+
1420+
Compared with Batch 18, mean validation improved by 1.33 percentage points and
1421+
sender agreement improved from 90.13% to 91.81%. Train exactness fell by 0.34
1422+
points, translator validation fell by 0.56 points, and three of four senders
1423+
had more collision meanings. Compared with Batch 15, the pulse slightly reduced
1424+
collisions for three senders and produced a stronger worst validation link, but
1425+
mean validation remained 0.54 points lower and train exactness remained lower.
1426+
The selected sender codebooks contain 12,855, 12,886, 12,709 and 13,200 unique
1427+
messages, with 2,505, 2,474, 2,651 and 2,160 collision meanings.
1428+
1429+
Population validation factor accuracies were
1430+
`[97.93%, 99.73%, 98.44%, 85.30%]`; universal-translator factor accuracies were
1431+
`[100%, 100%, 98.44%, 85.06%]`. The bounded pulse therefore improves population
1432+
color generalization and cross-link balance after replay is removed, but does
1433+
not resolve the persistent texture errors or hard-code collisions.
1434+
1435+
Both sealed analyses report 65 matching artifact hashes, 16,384 validation-only
1436+
episodes, no confirmatory-test keys, valid local alphabets, and exactly 14
1437+
declared bits for every logged message.
1438+
1439+
## Batch 19 decision
1440+
1441+
ECP7-B19-I is a valid negative result under the joint gate. A bounded replay
1442+
pulse is less disruptive to validation than persistent replay, and its
1443+
replay-free tail improves the worst population link, but the collision benefit
1444+
does not persist and the train bottleneck remains essentially unchanged. Batch
1445+
15 retains the strongest mean validation, while Batch 18 retains the strongest
1446+
train and translator values. The confirmatory split remains sealed.
1447+
1448+
A future ECP7-B20 should stop tuning collision-replay schedules. A clean next
1449+
causal test is deterministic global hard-meaning task replay on the unchanged
1450+
Batch 15 base: mine only training meanings that the current population
1451+
reconstructs incorrectly, then add ordinary sender-receiver reconstruction
1452+
updates on a registered bounded sample. This directly tests whether the
1453+
remaining train error is a sparse hard-example problem without adding a
1454+
meaning-to-slot binding or a code-distance surrogate. Collision replay, factor
1455+
reweighting, architecture, optimizer, horizon, data, translator and thresholds
1456+
must remain unchanged or absent as appropriate.

‎docs/research-design-ecp7.md‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -796,3 +796,45 @@ sealed-test requirement fails. No alternative coefficient, schedule, mining
796796
source, pair sampler, refresh, temperature, architecture, duration, optimizer,
797797
data, translator or seed may be tried inside Batch 18. Failure is recorded
798798
without confirmatory test access.
799+
800+
## Batch 19 preregistration — Bounded collision-replay pulse
801+
802+
Status: **preregistered for sealed development**<br>
803+
Preregistered: July 18, 2026 at 15:15:36 UTC<br>
804+
Configuration: `config/ecp7-b19-development.yaml`<br>
805+
Raw configuration SHA-256: `dab2cd0dadbadb47a10e2c45a720611dc91d140d3783f0ca1a537e4e8a8549c0`<br>
806+
Effective configuration SHA-256: `cbd144a5fce451315a7d5fb322a11f2f29c32514fc22fbe06f46285aef677d13`
807+
808+
Batch 18 confirmed that replay at task-loss scale avoids B17's early collapse,
809+
but validation peaked at step 22,400 and then regressed while weight `0.1`
810+
remained active. Batch 19 tests whether replay is useful as a bounded codebook
811+
diversification pulse rather than a permanent late objective.
812+
813+
ECP7-B19-I inherits the complete Batch 18 architecture, training-only mining,
814+
seed-derived pair sampler, 32-pair sender batches, 200-step refresh interval,
815+
relaxed temperature, base losses, optimizer, data, duration, selection,
816+
translator and thresholds. It is identical to B18 through step 20,000. The sole
817+
change is a coefficient decay:
818+
819+
- replay weight remains `0` through step `15,000`;
820+
- replay weight rises linearly to `0.1` at step `20,000`;
821+
- replay weight falls linearly to `0` over steps `20,000–25,000`;
822+
- replay weight remains `0` through step `30,000`.
823+
824+
Training-codebook collision mining continues at evaluation boundaries after
825+
step 25,000 for diagnostics, but contributes no gradient and consumes no replay
826+
sampler draws. The final optimization phase therefore contains only the B15
827+
task and utilization objectives. The complete trajectory through step 20,000
828+
must match B18 exactly.
829+
830+
The batch contains exactly two seed-11 runs:
831+
832+
1. the unchanged ECP-6 positive control;
833+
2. ECP7-B19-I with one bounded global collision-replay pulse.
834+
835+
All existing validity and development gates remain unchanged. The intervention
836+
fails if any train, validation, translator, injectivity, channel-integrity or
837+
sealed-test requirement fails. No alternative decay, endpoint, mining, pair
838+
sampler, architecture, base loss, optimizer, duration, data, translator or seed
839+
may be tried inside Batch 19. Failure is recorded without confirmatory test
840+
access.

‎evidence/ecp7-development/README.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,12 @@ exactness, 80.71% validation and a new-best 84.06% translator validation, but
9898
still failed train thresholds and injectivity. Exact identities and hashes are
9999
in `batch18-manifest.json`.
100100

101+
Batch 19 decayed that replay weight back to zero after step 20,000. It reached
102+
83.74% train exactness, 82.04% validation, a new-best 80.57% worst-link
103+
validation and 83.50% translator validation. The bounded pulse improved
104+
cross-link balance but still failed train thresholds and injectivity. Exact
105+
identities and hashes are in `batch19-manifest.json`.
106+
101107
Reproduce the development arms with:
102108

103109
```bash
Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
{
2+
"batch": "ECP7-B19",
3+
"confirmatory_test_unsealed": false,
4+
"config_effective_sha256": "cbd144a5fce451315a7d5fb322a11f2f29c32514fc22fbe06f46285aef677d13",
5+
"config_raw_sha256": "dab2cd0dadbadb47a10e2c45a720611dc91d140d3783f0ca1a537e4e8a8549c0",
6+
"control": {
7+
"artifact_hashes": {
8+
"effective_config.yaml": "4bfb976293f117d3ae35f790e4fd6371628be95d91febb334bc275dc5d6b66d4",
9+
"posthoc-analysis.json": "d38450c328b0528137d7873144338300f6542c54f28ed1a096329a5404bb13ac",
10+
"seed-11/metrics.json": "3fa1aa30d7795153e24622d5d4f519b5f86b178fa79bb2b7a39c75c875016927",
11+
"summary.json": "cfea6b314e74d04a0236e7eaf61ac70f9945f3e15513d343f19267b4093d7f46"
12+
},
13+
"development_gate_passed": true,
14+
"population_train_mean_exact_match": 1.0,
15+
"population_validation_mean_exact_match": 1.0,
16+
"run_id": "20260718T151853Z-ecp7-development",
17+
"universal_translator_validation_exact_match": 1.0,
18+
"unique_messages_per_sender": [15360, 15360, 15360, 15360]
19+
},
20+
"intervention": {
21+
"artifact_hashes": {
22+
"effective_config.yaml": "05bfddf2185783338a4c9fc626b06f7dcb335d883380b0d5f14c5117eb897e76",
23+
"posthoc-analysis.json": "e6954381e530120522873543d0e89a454d191e2e05e878fa1ee7b81c04bf661e",
24+
"seed-11/metrics.json": "26d40d118482038fca8bb73930a646babd7c1ae8c7a41178254cd5d8d2528b15",
25+
"summary.json": "965b1ca352583378eedfdacb72651d8656bda495f5307b952540ad592e2fc668"
26+
},
27+
"best_code_utilization_loss": -0.7285780310630798,
28+
"best_global_collision_replay_loss": 0.0,
29+
"best_global_collision_replay_pair_counts": [2674, 2645, 2921, 2352],
30+
"best_global_collision_replay_weight": 0.0,
31+
"best_step": 26200,
32+
"collision_meanings_per_sender": [2505, 2474, 2651, 2160],
33+
"development_gate_passed": false,
34+
"exact_sender_message_agreement": 0.9180881076388889,
35+
"final_step": 30000,
36+
"message_entropy_bits_per_sender": [13.56302604477656, 13.567062503109893, 13.538167215820996, 13.60794791977656],
37+
"population_train_mean_exact_match": 0.8373849051339286,
38+
"population_train_worst_link_exact_match": 0.8271484375,
39+
"population_validation_factor_accuracy": [0.97930908203125, 0.99725341796875, 0.984375, 0.85302734375],
40+
"population_validation_mean_exact_match": 0.8204345703125,
41+
"population_validation_worst_link_exact_match": 0.8056640625,
42+
"run_id": "20260718T151853Z-ecp7-development",
43+
"universal_translator_validation_exact_match": 0.8349609375,
44+
"unique_messages_per_sender": [12855, 12886, 12709, 13200]
45+
},
46+
"seed": 11,
47+
"split_artifact_sha256": "676195997ee0c77d5c9565bae045dbb938b432a2e4e74b666233b0a41f924718",
48+
"split_sha256": "4947058c75ab07cb43a87eb82776b12cb2a7e2eeba7114de110d3b852cbc64cd"
49+
}

0 commit comments

Comments
 (0)