Skip to content

Commit 7aa23e0

Browse files
authored
Record ECP-7 Batch 21 shared-error replay (#22)
1 parent 077a563 commit 7aa23e0

12 files changed

Lines changed: 325 additions & 29 deletions

File tree

‎README.md‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ The protocol uses no words or alphabet, but four meaning-free local symbols. For
2626
See [`docs/results-ecp6.md`](docs/results-ecp6.md) for the conclusion, [`docs/protocol-specification-ecp6.md`](docs/protocol-specification-ecp6.md) for the wire format and [`evidence/ecp6/report.md`](evidence/ecp6/report.md) for the compact confirmatory evidence.
2727

2828
ECP-7 development now tests how much of that result depends on the explicit
29-
one-factor-per-slot architecture. Its first twenty sealed batches are valid
29+
one-factor-per-slot architecture. Its first twenty-one sealed batches are valid
3030
development results but none passes the full gate. Batch 15 established the
3131
strong position-aware base: at 30,000 optimization steps it reaches 83.46% train
3232
exactness, 82.59% validation and 83.37% translator validation while using
@@ -41,7 +41,10 @@ reached 82.04% validation plus a new-best 80.57% worst-link validation, but
4141
again failed train exactness and injectivity. The ECP-6 positive controls remain
4242
perfect. Batch 20 replayed population-hard training meanings and established
4343
new-best 83.45% mean and 82.13% worst-link validation, but again failed train
44-
exactness and injectivity. The ECP-7 confirmatory test remains sealed. See
44+
exactness and injectivity. Batch 21 restricted replay to errors shared by all
45+
16 links and improved validation again to 83.63% mean and 82.62% worst-link,
46+
but its target error pool grew and train exactness still failed. The ECP-7
47+
confirmatory test remains sealed. See
4548
[`docs/development-log-ecp7.md`](docs/development-log-ecp7.md).
4649

4750
## Build on this work
@@ -102,6 +105,7 @@ Do not translate those artifacts in place. New experiment configurations and all
102105
- [`config/ecp7-b18-development.yaml`](config/ecp7-b18-development.yaml) — the Batch 17 replay mechanism with final weight reduced to 0.1.
103106
- [`config/ecp7-b19-development.yaml`](config/ecp7-b19-development.yaml) — the Batch 18 replay mechanism decayed back to zero after step 20,000.
104107
- [`config/ecp7-b20-development.yaml`](config/ecp7-b20-development.yaml) — late ordinary task replay on population-hard training meanings.
108+
- [`config/ecp7-b21-development.yaml`](config/ecp7-b21-development.yaml) — Batch 20 replay restricted to meanings failed by all 16 population links.
105109
- [`config/ecp7-positive-control-development.yaml`](config/ecp7-positive-control-development.yaml) — ECP-6 positive control on the ECP-7 split.
106110
- [`config/ecp0.yaml`](config/ecp0.yaml) — machine-readable configuration from step 1.
107111
- [`config/ecp1.yaml`](config/ecp1.yaml) — frozen configuration of the population trial.

‎config/ecp7-b21-development.yaml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
extends: ecp7-b20-development.yaml
2+
3+
experiment:
4+
title: ECP-7 population-shared hard-meaning task replay
5+
arm: population_shared_hard_meaning_task_replay
6+
objective: >-
7+
Test whether keeping the complete Batch 20 replay mechanism but mining only
8+
training meanings failed by all 16 current sender-receiver links can repair
9+
the population-shared error concentration without changing replay scale.
10+
11+
training:
12+
global_hard_meaning_replay:
13+
minimum_failed_links: 16

‎docs/AI_AGENT_START.md‎

Lines changed: 26 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ python3.12 -m venv .venv
4141
.venv/bin/ecp6 --config config/ecp6.yaml validate
4242
```
4343

44-
Expected baseline: 109 passing tests and split sizes `14336/1024/1024`.
44+
Expected baseline: 111 passing tests and split sizes `14336/1024/1024`.
4545

4646
## 4. Reproduce ECP-6
4747

@@ -71,7 +71,7 @@ This reruns a known experiment. It must not be presented as a new independent co
7171

7272
## 5. Continue ECP-7 scientifically
7373

74-
Do not modify ECP-0 through ECP-6 or the completed ECP-7 Batch 1 through Batch 20
74+
Do not modify ECP-0 through ECP-6 or the completed ECP-7 Batch 1 through Batch 21
7575
configs. ECP7-B1-I collapsed to 130–139 hard messages. ECP7-B2-I improved its
7676
soft objective but collapsed further to 85–104 hard messages. ECP7-B3-I applied
7777
that loss to straight-through hard messages and improved to 585–972 messages,
@@ -126,11 +126,15 @@ population link. It reached 83.77% train, a new-best 83.45% mean and 82.13%
126126
worst-link validation, and 83.96% translator validation. The pool shrank but its
127127
remaining errors became more population-wide, so train thresholds and
128128
injectivity still failed. The confirmatory ECP-7 test remains sealed.
129+
ECP7-B21-I restricted the same replay budget to meanings failed by all 16
130+
population links. It reached a new-best 83.63% mean and 82.62% worst-link
131+
validation and 92.96% sender agreement, but mean train remained 83.71% and the
132+
target shared-error pool grew. The confirmatory ECP-7 test remains sealed.
129133

130134
Continue with this sequence:
131135

132136
1. Read `docs/research-design-ecp7.md` and `docs/development-log-ecp7.md`.
133-
2. Define exactly one ECP7-B21 intervention and its failure criterion.
137+
2. Define exactly one ECP7-B22 intervention and its failure criterion.
134138
3. Register the variant and immutable configuration hashes before training.
135139
4. Add tests for every new invariant.
136140
5. Use only `smoke` and `develop`; keep the ECP-7 test split sealed.
@@ -143,25 +147,27 @@ Continue with this sequence:
143147

144148
## 6. Recommended next experiment
145149

146-
Batch 20 shows that ordinary task replay on any-link errors improves
147-
generalization and cross-link balance, but mostly removes link-specific errors.
148-
At its selected checkpoint, 1,937 of 2,848 hard meanings fail for all 16 links,
149-
up from 1,469 of 3,406 at the Batch 15 checkpoint. The clean next question is
150-
whether the fixed replay budget should target only those population-shared
151-
errors. A clean ECP7-B21 progression is:
152-
153-
- inherit the complete Batch 20 implementation and Batch 15 base;
154-
- change only the mining predicate from at least one failed link to all 16
155-
failed links;
156-
- keep weight `0.25`, the 64-meaning uniform replay batch, 200-step refresh,
157-
seed-derived sampler, warmup and horizon unchanged;
150+
Batch 21 directly targets population-shared errors, but its all-link pool grows
151+
from 1,745 to 1,997 at the selected checkpoint while validation and sender
152+
agreement improve. This suggests that joint replay updates align receivers and
153+
senders around persistent ambiguous codes rather than forcing senders to move.
154+
The clean next question is whether replay gradients must be sender-only. A clean
155+
ECP7-B22 progression is:
156+
157+
- inherit the complete Batch 21 predicate, Batch 20 replay mechanism and Batch
158+
15 base;
159+
- change only replay-loss gradient routing so receiver parameters are treated as
160+
fixed during the additional replay forward pass;
161+
- keep ordinary base-task updates to both senders and receivers unchanged;
162+
- keep weight `0.25`, the 64-meaning replay batch, 200-step refresh,
163+
seed-derived sampler, warmup, temperature and horizon unchanged;
158164
- continue to mine and replay training meanings only;
159-
- measure train exactness, injectivity, validation composition and new-reader
160-
induction, and rerun the frozen ECP-6 positive control.
165+
- measure shared-error pool size, train exactness, injectivity, validation and
166+
new-reader induction, and rerun the frozen ECP-6 positive control.
161167

162-
Do not combine the predicate change with another coefficient, graded sampling,
163-
collision pressure, a new architecture, factor-specific weights, optimizer
164-
changes, longer training, variable-length messages or negotiation.
168+
Do not combine gradient routing with another predicate, coefficient, graded
169+
sampling, collision pressure, a new architecture, factor-specific weights,
170+
optimizer changes, longer training, variable-length messages or negotiation.
165171

166172
## 7. Definition of done for any contribution
167173

‎docs/development-log-ecp7.md‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ this log.
2828
| ECP7-B18-I | Intervention | B17 replay with final weight 0.1 | Position-aware MLP | completed; gate failed |
2929
| ECP7-B19-I | Intervention | B18 with replay decayed to zero after step 20,000 | Position-aware MLP | valid negative; validation 82.04%, train and injectivity fail |
3030
| ECP7-B20-I | Intervention | Late global replay of population-hard training meanings | Position-aware MLP | valid negative; new-best validation 83.45%, train and injectivity fail |
31+
| ECP7-B21-I | Intervention | B20 replay restricted to meanings failed by all 16 links | Position-aware MLP | valid negative; new-best validation 83.63%, shared-error pool grows |
3132

3233
## Batch 1 preregistration
3334

@@ -1566,3 +1567,106 @@ the same registered replay budget exclusively on population-shared errors. No
15661567
coefficient, replay batch, refresh, schedule, architecture, base loss,
15671568
optimizer, horizon, data, translator or threshold change may accompany that
15681569
predicate test.
1570+
1571+
## Batch 21 preregistration
1572+
1573+
Preregistered: July 18, 2026 at 16:05:33 UTC<br>
1574+
Seed: `11`<br>
1575+
Maximum population steps: `30,000`<br>
1576+
Minimum selection steps: `15,000`<br>
1577+
Early-stopping patience: `5,000` steps<br>
1578+
Hard-meaning predicate: all `16` current population links have an incorrect factor<br>
1579+
Replay weight: `0` through step `15,000`, linear to `0.25` at step `20,000`, then hold<br>
1580+
Replay batch size: `64` meanings, uniform with replacement<br>
1581+
Training-only pool refresh interval: `200` steps<br>
1582+
Replay sampler seed offset: `+1211`<br>
1583+
Raw configuration SHA-256: `d2d8d8bf3d87aef42b533bfd33eea4033af99444ec00b1cffe3d6f63c5970a5b`<br>
1584+
Effective configuration SHA-256: `913fa521edb8138b8e2319ef52b1ee1533ce13592f53303250d97ef217af0f5b`<br>
1585+
Split-SHA-256: `4947058c75ab07cb43a87eb82776b12cb2a7e2eeba7114de110d3b852cbc64cd`<br>
1586+
Test unsealed: **no**
1587+
1588+
ECP7-B21-I inherits Batch 20 and changes only the failed-link threshold from the
1589+
implicit default `1` to the complete population count `16`. The base random task
1590+
sequence, replay sampler, replay scale and every other setting remain unchanged.
1591+
This directly tests whether the B20 pool's population-wide error concentration
1592+
requires a narrower curriculum. The positive control is rerun, and no
1593+
alternative threshold or accompanying intervention is admitted into this batch.
1594+
1595+
## Batch 21 results
1596+
1597+
Positive-control run: `runs/ecp7-batch21-control-development/20260718T160742Z-ecp7-development`<br>
1598+
Intervention run: `runs/ecp7-batch21-intervention-development/20260718T160742Z-ecp7-development`<br>
1599+
Test unsealed: **no**
1600+
1601+
| Metric | Positive control | ECP7-B21-I |
1602+
|---|---:|---:|
1603+
| Population train, mean | 100% | **83.7062%** |
1604+
| Population train, worst link | 100% | **82.7916%** |
1605+
| Population validation, mean | 100% | **83.6304%** |
1606+
| Population validation, worst link | 100% | **82.6172%** |
1607+
| Universal translator, validation | 100% | **83.6426%** |
1608+
| Exact sender-message agreement | 100% | **92.96%** |
1609+
| Unique messages per sender | 15,360 | 12,720–13,200 |
1610+
| Collision meanings per sender | 0 | 2,160–2,640 |
1611+
| Message entropy | 13.91 bits | 13.54–13.61 bits |
1612+
| Development gate | pass | **fail** |
1613+
1614+
The positive control again passed every gate at 100%. ECP7-B21-I establishes
1615+
new-best ECP-7 mean validation, worst-link validation, worst-link train and
1616+
sender agreement. It passes validation and translator thresholds, but mean and
1617+
worst-link train remain far below their registered thresholds and every sender
1618+
remains non-injective. The joint gate fails and confirmatory access remains
1619+
unauthorized.
1620+
1621+
All 76 optimization records through step 15,000 match Batch 20 exactly across
1622+
the 30 shared non-pool fields. The registered predicate produces the expected
1623+
different first pool at step 15,000: 1,745 meanings fail for all 16 links. That
1624+
pool briefly falls to 1,504 at step 15,600, then grows as replay weight rises,
1625+
reaching 2,031 at full weight on step 20,000.
1626+
1627+
The selected checkpoint was step 29,400 with 1,997 all-link failures. Task loss
1628+
was `0.07461`, hard-meaning replay loss `0.11821`, replay weight `0.25`, and
1629+
utilization loss `-0.75199`. The replay contribution was approximately `0.0296`,
1630+
well below the ordinary task loss. The run reached step 30,000, finishing with
1631+
2,016 all-link failures and 83.18% validation.
1632+
1633+
Compared with Batch 20, mean train fell by 0.07 percentage points and translator
1634+
validation by 0.32 points. Mean validation improved by 0.18 points, worst-link
1635+
validation by 0.49 points, worst-link train by 0.08 points and sender agreement
1636+
by 0.57 points. The narrower curriculum therefore continues the cross-link
1637+
alignment effect without improving mean known-meaning reconstruction.
1638+
1639+
A selected-checkpoint training diagnostic confirms the tradeoff. Batch 21 has
1640+
2,755 meanings failed by at least one link, fewer than Batch 20's 2,848, but
1641+
1,997 fail for all links, more than Batch 20's 1,937. A hard meaning now fails
1642+
for 13.57 links on average, up from 13.07. The all-link replay pool grows 14.4%
1643+
from its step-15,000 value and its selected hard loss is `0.34190`. Directing
1644+
more ordinary joint sender-receiver optimization at shared errors therefore
1645+
reinforces population alignment around the remaining ambiguous mappings rather
1646+
than separating them.
1647+
1648+
Selected sender codebooks contain 12,840, 12,870, 12,720 and 13,200 unique
1649+
messages, with 2,520, 2,490, 2,640 and 2,160 collision meanings. Population
1650+
validation factor accuracies were `[99.95%, 99.95%, 98.44%, 85.29%]`;
1651+
universal-translator factor accuracies were
1652+
`[100%, 100%, 98.44%, 85.21%]`. Population color and shape are effectively
1653+
complete, while the same size and texture bottlenecks remain.
1654+
1655+
Both sealed analyses report 65 matching artifact hashes, 16,384 validation-only
1656+
episodes, no confirmatory-test keys, valid local alphabets, and exactly 14
1657+
declared bits for every logged message.
1658+
1659+
## Batch 21 decision
1660+
1661+
ECP7-B21-I is a valid negative result. All-link replay slightly improves the
1662+
strongest validation result, but directly falsifies the hypothesis that a
1663+
narrower shared-error curriculum will repair train exactness: its target pool
1664+
grows and becomes more dominant. The confirmatory split remains sealed.
1665+
1666+
A future ECP7-B22 may keep Batch 21's predicate, replay batch, weight, schedule
1667+
and every Batch 15 setting while stopping replay-loss gradients into receiver
1668+
parameters. Ordinary base-task updates would continue training both sides; only
1669+
the additional all-link replay loss would update senders through temporarily
1670+
fixed receivers. This isolates whether joint coadaptation prevents ambiguous
1671+
sender codes from moving toward already learned decoder distinctions. No other
1672+
parameter or mechanism may change in that test.

‎docs/research-design-ecp7.md‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -888,3 +888,45 @@ sealed-test requirement fails. No alternative predicate, score, replay weight,
888888
batch size, refresh, schedule, architecture, base loss, optimizer, duration,
889889
data, translator or seed may be tried inside Batch 20. Failure is recorded
890890
without confirmatory test access.
891+
892+
## Batch 21 preregistration — Population-shared hard-meaning replay
893+
894+
Status: **preregistered for sealed development**<br>
895+
Preregistered: July 18, 2026 at 16:05:33 UTC<br>
896+
Configuration: `config/ecp7-b21-development.yaml`<br>
897+
Raw configuration SHA-256: `d2d8d8bf3d87aef42b533bfd33eea4033af99444ec00b1cffe3d6f63c5970a5b`<br>
898+
Effective configuration SHA-256: `913fa521edb8138b8e2319ef52b1ee1533ce13592f53303250d97ef217af0f5b`
899+
900+
Batch 20 reduced the any-link hard pool and established the strongest ECP-7
901+
validation result, but its remaining error became more population-wide. At the
902+
selected checkpoint, 1,937 of 2,848 hard meanings failed for all 16 links,
903+
compared with 1,469 of 3,406 at the selected Batch 15 checkpoint. Batch 21 tests
904+
whether replay should spend its fixed budget exclusively on those shared
905+
errors.
906+
907+
ECP7-B21-I inherits the complete Batch 20 implementation, Batch 15 architecture,
908+
ordinary random task batch, utilization objective, optimizer, data, temperature
909+
schedule, hard-replay coefficient, batch size, sampler, refresh, warmup,
910+
30,000-step horizon, selection, translator and thresholds. The sole change is:
911+
912+
- a training meaning enters the replay pool only when all 16 current
913+
sender-receiver links reconstruct at least one factor incorrectly, instead of
914+
when at least one link fails.
915+
916+
Mining still uses hard predictions over training meanings only. Replay still
917+
samples 64 pool meanings uniformly with replacement, applies the ordinary
918+
all-link reconstruction loss, warms from weight zero at step 15,000 to `0.25`
919+
at step 20,000, and then holds. The complete trajectory through step 15,000
920+
must match Batch 20 and Batch 15 exactly.
921+
922+
The batch contains exactly two seed-11 runs:
923+
924+
1. the unchanged ECP-6 positive control;
925+
2. ECP7-B21-I with population-shared hard-meaning replay.
926+
927+
All existing validity and development gates remain unchanged. The intervention
928+
fails if any train, validation, translator, injectivity, channel-integrity or
929+
sealed-test requirement fails. No alternative failed-link threshold, graded
930+
score, coefficient, replay batch, refresh, schedule, architecture, base loss,
931+
optimizer, duration, data, translator or seed may be tried inside Batch 21.
932+
Failure is recorded without confirmatory test access.

‎evidence/ecp7-development/README.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -111,6 +111,12 @@ shrunk but concentrated into population-wide failures, so train thresholds and
111111
injectivity still failed. Exact identities and hashes are in
112112
`batch20-manifest.json`.
113113

114+
Batch 21 restricted the unchanged replay budget to meanings failed by all 16
115+
population links. It reached a new-best 83.63% mean and 82.62% worst-link
116+
validation plus 92.96% sender agreement, but mean train remained 83.71% and the
117+
target shared-error pool grew. Exact identities and hashes are in
118+
`batch21-manifest.json`.
119+
114120
Reproduce the development arms with:
115121

116122
```bash

0 commit comments

Comments
 (0)