Skip to content

Commit 5d28858

Browse files
authored
Record sealed ECP-7 deep decoder batch (#13)
Add the deep shared decoder, preregister and record sealed Batch 12, and update the evidence-backed fresh-agent handoff. The confirmatory ECP-7 test remains sealed.
1 parent 59110ff commit 5d28858

11 files changed

Lines changed: 275 additions & 21 deletions

File tree

‎README.md‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -26,15 +26,16 @@ The protocol uses no words or alphabet, but four meaning-free local symbols. For
2626
See [`docs/results-ecp6.md`](docs/results-ecp6.md) for the conclusion, [`docs/protocol-specification-ecp6.md`](docs/protocol-specification-ecp6.md) for the wire format and [`evidence/ecp6/report.md`](evidence/ecp6/report.md) for the compact confirmatory evidence.
2727

2828
ECP-7 development now tests how much of that result depends on the explicit
29-
one-factor-per-slot architecture. Its first eleven sealed batches are valid
29+
one-factor-per-slot architecture. Its first twelve sealed batches are valid
3030
development results but none passes the full gate. Batch 10 remains strongest,
3131
extending the position-aware design to 15,000 optimization steps. It reaches
3232
82.08% train exactness, 72.98% validation and 75.15% translator validation while
3333
using 12,358–12,906 hard messages per sender. The translator threshold now passes,
3434
but the registered train, validation and injectivity thresholds do not. Batch
35-
11's utilization-weight decay regressed all principal measures. The ECP-6
36-
positive controls remain perfect and the ECP-7 confirmatory test remains sealed.
37-
See [`docs/development-log-ecp7.md`](docs/development-log-ecp7.md).
35+
11's utilization-weight decay regressed all principal measures, and Batch 12's
36+
deeper decoder regressed further. The ECP-6 positive controls remain perfect and
37+
the ECP-7 confirmatory test remains sealed. See
38+
[`docs/development-log-ecp7.md`](docs/development-log-ecp7.md).
3839

3940
## Build on this work
4041

@@ -85,6 +86,7 @@ Do not translate those artifacts in place. New experiment configurations and all
8586
- [`config/ecp7-b9-development.yaml`](config/ecp7-b9-development.yaml) — generic position-aware MLP decoding on the parallel sender.
8687
- [`config/ecp7-b10-development.yaml`](config/ecp7-b10-development.yaml) — the Batch 9 design with an isolated extended optimization horizon.
8788
- [`config/ecp7-b11-development.yaml`](config/ecp7-b11-development.yaml) — late utilization-weight decay on the Batch 10 base.
89+
- [`config/ecp7-b12-development.yaml`](config/ecp7-b12-development.yaml) — one additional generic shared decoder layer on the Batch 10 base.
8890
- [`config/ecp7-positive-control-development.yaml`](config/ecp7-positive-control-development.yaml) — ECP-6 positive control on the ECP-7 split.
8991
- [`config/ecp0.yaml`](config/ecp0.yaml) — machine-readable configuration from step 1.
9092
- [`config/ecp1.yaml`](config/ecp1.yaml) — frozen configuration of the population trial.

‎config/ecp7-b12-development.yaml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
extends: ecp7-b10-development.yaml
2+
3+
experiment:
4+
title: ECP-7 extended optimization with a deep position-aware decoder
5+
arm: extended_optimization_deep_position_aware_decoder
6+
objective: >-
7+
Test whether one additional generic shared hidden layer lets receivers and
8+
the isolated translator resolve the residual color and size distinctions
9+
without introducing factor-specific paths or bindings.
10+
11+
agents:
12+
receiver:
13+
family: deep_position_aware_mlp_receiver
14+
translator:
15+
family: deep_position_aware_mlp_receiver

‎docs/AI_AGENT_START.md‎

Lines changed: 17 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ python3.12 -m venv .venv
4141
.venv/bin/ecp6 --config config/ecp6.yaml validate
4242
```
4343

44-
Expected baseline: 79 passing tests and split sizes `14336/1024/1024`.
44+
Expected baseline: 82 passing tests and split sizes `14336/1024/1024`.
4545

4646
## 4. Reproduce ECP-6
4747

@@ -71,7 +71,7 @@ This reruns a known experiment. It must not be presented as a new independent co
7171

7272
## 5. Continue ECP-7 scientifically
7373

74-
Do not modify ECP-0 through ECP-6 or the completed ECP-7 Batch 1 through Batch 11
74+
Do not modify ECP-0 through ECP-6 or the completed ECP-7 Batch 1 through Batch 12
7575
configs. ECP7-B1-I collapsed to 130–139 hard messages. ECP7-B2-I improved its
7676
soft objective but collapsed further to 85–104 hard messages. ECP7-B3-I applied
7777
that loss to straight-through hard messages and improved to 585–972 messages,
@@ -95,12 +95,14 @@ schedule and extended population optimization to 15,000 steps. It reached
9595
12,358–12,906 messages. The translator gate passed, but train, validation and
9696
injectivity did not. ECP7-B11-I decayed late utilization pressure and regressed
9797
to 79.09% train, 72.22% validation and 72.41% translator validation. B10 remains
98-
the strongest base. The confirmatory ECP-7 test is still sealed.
98+
the strongest base. ECP7-B12-I added one shared decoder layer and regressed
99+
further to 71.53% train, 47.98% validation and 63.04% translator validation.
100+
The confirmatory ECP-7 test is still sealed.
99101

100102
Continue with this sequence:
101103

102104
1. Read `docs/research-design-ecp7.md` and `docs/development-log-ecp7.md`.
103-
2. Define exactly one ECP7-B12 intervention and its failure criterion.
105+
2. Define exactly one ECP7-B13 intervention and its failure criterion.
104106
3. Register the variant and immutable configuration hashes before training.
105107
4. Add tests for every new invariant.
106108
5. Use only `smoke` and `develop`; keep the ECP-7 test split sealed.
@@ -113,23 +115,23 @@ Continue with this sequence:
113115

114116
## 6. Recommended next experiment
115117

116-
Batch 10 is the strongest weak-structure result. Batch 11 showed that reducing
117-
late utilization pressure harms task accuracy, code use and sender agreement.
118-
B10's one-hidden-layer population decoder and fresh translator both retain a
119-
size weakness, while color is the other residual population error. The most
120-
informative next step is one generic decoder-capacity change. A clean ECP7-B12
121-
progression is:
118+
Batch 10 is the strongest weak-structure result. Batch 11 ruled out reduced late
119+
utilization pressure, and Batch 12 ruled out a deeper shared decoder. B10 still
120+
generates all four message positions from one shallow shared sender context.
121+
The most informative next step is the symmetric sender-capacity test. A clean
122+
ECP7-B13 progression is:
122123

123124
- keep the same world and bit budget;
124125
- return to the complete B10 training and loss schedule;
125-
- preserve the flattened position-aware input and shared representation;
126-
- add exactly one second shared hidden layer to both receiver and translator;
126+
- keep the B10 one-hidden-layer receiver and translator;
127+
- add exactly one second shared hidden layer to the bounded parallel sender
128+
before all four slot heads;
127129
- measure injectivity, validation composition and new-reader induction;
128130
- rerun the ECP-6 architecture as the frozen positive control.
129131

130-
Do not add a factor-specific input, binding matrix, slot selection or loss
131-
change. Do not change the sender, hidden width, data, temperature, duration,
132-
validation cadence or development thresholds, and do not combine decoder depth
132+
Do not add a factor-specific sender path, binding matrix, slot selection or loss
133+
change. Do not change the decoder, hidden width, data, temperature, duration,
134+
validation cadence or development thresholds, and do not combine sender depth
133135
with variable-length messages or negotiation. That would make its effect
134136
uninterpretable.
135137

‎docs/development-log-ecp7.md‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -787,3 +787,72 @@ complete B10 schedule and add exactly one second generic shared hidden layer to
787787
both the position-aware receiver and isolated translator. It must preserve the
788788
same flattened positional input and must not add factor-specific paths, binding,
789789
loss, duration, data or threshold changes.
790+
791+
## Batch 12 preregistration
792+
793+
Date: July 18, 2026<br>
794+
Seed: `11`<br>
795+
Maximum population steps: `15,000`<br>
796+
Temperature schedule steps: `5,000`, then hold at `0.8`<br>
797+
Decoder change: one additional shared `hidden_dim × hidden_dim` layer<br>
798+
Split-SHA-256: `4947058c75ab07cb43a87eb82776b12cb2a7e2eeba7114de110d3b852cbc64cd`<br>
799+
Test unsealed: **no**
800+
801+
ECP7-B12-I returns to every Batch 10 training and loss setting, then changes
802+
only the receiver and translator depth. The four factor heads still consume one
803+
shared representation of all four embedded positions. There is no binding or
804+
factor-specific input path. The unchanged positive control is rerun, and no
805+
alternative decoder variant is admitted into this batch.
806+
807+
## Batch 12 results
808+
809+
Positive-control run: `runs/ecp7-batch12-control-development/20260718T130109Z-ecp7-development`<br>
810+
Intervention run: `runs/ecp7-batch12-intervention-development/20260718T130109Z-ecp7-development`<br>
811+
Test unsealed: **no**
812+
813+
| Metric | Positive control | ECP7-B12-I |
814+
|---|---:|---:|
815+
| Population train, mean | 100% | 71.5297% |
816+
| Population train, worst link | 100% | 66.6016% |
817+
| Population validation, mean | 100% | 47.9797% |
818+
| Population validation, worst link | 100% | 39.7461% |
819+
| Universal translator, validation | 100% | 63.0371% |
820+
| Exact sender-message agreement | 100% | 88.64% |
821+
| Unique messages per sender | 15,360 | 10,119–11,404 |
822+
| Collision meanings per sender | 0 | 3,956–5,241 |
823+
| Message entropy | 13.91 bits | 13.11–13.33 bits |
824+
| Development gate | pass | **fail** |
825+
826+
The positive control passed every gate at 100%. ECP7-B12-I failed train,
827+
validation, translator and injectivity. It does not authorize confirmatory
828+
access.
829+
830+
Relative to B10, the second decoder layer reduced train exactness from 82.0836%
831+
to 71.5297%, validation from 72.9797% to 47.9797%, and translator validation from
832+
75.1465% to 63.0371%. Code use and sender agreement also fell. The added generic
833+
decoder depth therefore made the emergent protocol harder, not easier, to learn.
834+
835+
Population validation factor accuracies were
836+
`[77.07%, 86.05%, 100%, 77.05%]`; universal-translator factor accuracies were
837+
`[97.17%, 87.50%, 100%, 75.90%]`. The deeper shared representation did not
838+
resolve the residual factors consistently across receivers and a new reader.
839+
840+
The selected checkpoint was step 10,000 with task loss `0.1539` and utilization
841+
loss `-0.7153`. Validation subsequently regressed, so the registered patience
842+
rule stopped population training at step 13,000.
843+
844+
Both sealed analyses report 65 matching artifact hashes, 16,384 validation-only
845+
episodes, no confirmatory-test keys, valid local alphabets, and exactly 14
846+
declared bits for every logged message.
847+
848+
## Batch 12 decision
849+
850+
ECP7-B12-I is rejected. Decoder depth is not the B10 bottleneck. Batch 10 remains
851+
the strongest weak-structure base, and the confirmatory split remains sealed.
852+
853+
The remaining capacity hypothesis is sender-side: B10 generates all four slots
854+
from one shallow shared context, and receiver depth has now been ruled out. A
855+
future ECP7-B13 may return to the complete B10 model and add exactly one second
856+
generic shared hidden layer to the bounded parallel sender before all four slot
857+
heads. It must retain the B10 one-layer receiver and translator and must not add
858+
factor-specific sender paths, loss, width, duration, data or threshold changes.

‎docs/research-design-ecp7.md‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -519,3 +519,40 @@ All existing validity and development gates remain unchanged. Failure is
519519
recorded without confirmatory test access. No alternative final weight, decay
520520
shape, decay boundary, duration, architecture, loss, optimizer or seed may be
521521
tried inside Batch 11.
522+
523+
## Batch 12 preregistration — Deep position-aware decoder
524+
525+
Status: **preregistered for sealed development**<br>
526+
Preregistered: July 18, 2026<br>
527+
Configuration: `config/ecp7-b12-development.yaml`<br>
528+
Raw configuration SHA-256: `4cc37488453b756fb43b4782d0e3e93cf051815a367d1852047fcc34fa29d5ce`<br>
529+
Effective configuration SHA-256: `798a6380ab144e5db63c8177d3dd70edf76e66b660b3cbb355f5beabe86b27b2`
530+
531+
Batch 11 showed that reducing late utilization pressure worsened task accuracy,
532+
code use and sender agreement. Batch 10 therefore remains the base. Its shared
533+
one-hidden-layer decoder and independently trained translator both retained a
534+
size weakness, while color was the other residual population error. Batch 12
535+
tests whether the generic decoder is still too shallow for the entangled code.
536+
537+
ECP7-B12-I returns to the complete B10 training and loss schedule. It keeps the
538+
shared symbol embedding, flattened four-position input and first generic hidden
539+
projection, then adds exactly one second `hidden_dim × hidden_dim` projection
540+
with `tanh`. All four factor heads read that same second shared representation.
541+
The isolated translator uses the identical deeper family.
542+
543+
The decoder still has no factor-specific input path, factor-specific hidden
544+
projection, binding matrix, slot selector or hard factor-to-slot assignment.
545+
Hidden width, sender, losses, optimizer, data, 15,000-step duration, original
546+
5,000-step temperature annealing, selection window and thresholds remain
547+
unchanged from B10. Smoke runs are mechanical only and may not authorize
548+
tuning.
549+
550+
The batch contains exactly two seed-11 runs:
551+
552+
1. the unchanged ECP-6 positive control;
553+
2. ECP7-B12-I with the deeper shared position-aware receiver and translator.
554+
555+
All existing validity and development gates remain unchanged. Failure is
556+
recorded without confirmatory test access. No alternative depth, activation,
557+
width, residual connection, sender, loss, duration, optimizer or seed may be
558+
tried inside Batch 12.

‎evidence/ecp7-development/README.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,11 @@ Batch 11 decayed code-utilization weight after step 5,000. It regressed to
6363
11,893–12,323 messages. Batch 10 therefore remains the strongest base. Exact
6464
identities and hashes are in `batch11-manifest.json`.
6565

66+
Batch 12 added one generic shared hidden layer to the receiver and translator.
67+
It regressed sharply to 71.53% train exactness, 47.98% validation and 63.04%
68+
translator validation. Decoder depth is therefore rejected as the B10
69+
bottleneck. Exact identities and hashes are in `batch12-manifest.json`.
70+
6671
Reproduce the development arms with:
6772

6873
```bash
Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
{
2+
"batch": "ECP7-B12",
3+
"confirmatory_test_unsealed": false,
4+
"control": {
5+
"artifact_hashes": {
6+
"effective_config.yaml": "4bfb976293f117d3ae35f790e4fd6371628be95d91febb334bc275dc5d6b66d4",
7+
"posthoc-analysis.json": "2d92231db6e55352d15ae772fa8bee116a1c725db1aa31216a4cb0b5aa9ec270",
8+
"seed-11/metrics.json": "b32329ddd65833a0f0814144ed7c6b0cdffca1100829c61638c9d5ea3a58cff6",
9+
"summary.json": "34f1828bc95891a08bf234032a08696c495af55c27e26b607a2e0557e25f2e6b"
10+
},
11+
"development_gate_passed": true,
12+
"population_train_mean_exact_match": 1.0,
13+
"population_validation_mean_exact_match": 1.0,
14+
"run_id": "20260718T130109Z-ecp7-development",
15+
"universal_translator_validation_exact_match": 1.0,
16+
"unique_messages_per_sender": [15360, 15360, 15360, 15360]
17+
},
18+
"intervention": {
19+
"artifact_hashes": {
20+
"effective_config.yaml": "27017456afb7c224537735095dc1b031bc66fae8e7cf195ba088db9b208f3b82",
21+
"posthoc-analysis.json": "b6714ce0414acc540ba386c0ecfb0ea9d6789a413743713dc83c560a3e310812",
22+
"seed-11/metrics.json": "b03f9b13b0b03e3a1dd3d9bb81015e4f1ccde1e38eccd793bc76de813f217d51",
23+
"summary.json": "1dd591a537f57015bff6792bbcbad70563292f16ff66e174bc20df73d757f518"
24+
},
25+
"best_code_utilization_loss": -0.7153493165969849,
26+
"best_step": 10000,
27+
"collision_meanings_per_sender": [4171, 5241, 4380, 3956],
28+
"development_gate_passed": false,
29+
"exact_sender_message_agreement": 0.8863823784722223,
30+
"message_entropy_bits_per_sender": [13.30507490700547, 13.107133180956183, 13.260048045675191, 13.333167991315648],
31+
"population_train_mean_exact_match": 0.7152971540178572,
32+
"population_train_worst_link_exact_match": 0.666015625,
33+
"population_validation_factor_accuracy": [0.77069091796875, 0.8604736328125, 1.0, 0.7705078125],
34+
"population_validation_mean_exact_match": 0.47979736328125,
35+
"population_validation_worst_link_exact_match": 0.3974609375,
36+
"run_id": "20260718T130109Z-ecp7-development",
37+
"universal_translator_validation_exact_match": 0.63037109375,
38+
"unique_messages_per_sender": [11189, 10119, 10980, 11404]
39+
},
40+
"seed": 11,
41+
"split_artifact_sha256": "676195997ee0c77d5c9565bae045dbb938b432a2e4e74b666233b0a41f924718",
42+
"split_sha256": "4947058c75ab07cb43a87eb82776b12cb2a7e2eeba7114de110d3b852cbc64cd"
43+
}

‎src/ai_taal/models.py‎

Lines changed: 20 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -746,7 +746,7 @@ def __init__(self, spec: ModelSpec) -> None:
746746
nn.Linear(spec.hidden_dim, size) for size in spec.factor_sizes
747747
)
748748

749-
def forward(self, message: Tensor) -> tuple[Tensor, ...]:
749+
def _message_representation(self, message: Tensor) -> Tensor:
750750
if message.ndim == 2:
751751
embedded = self.symbol_embedding(message)
752752
elif message.ndim == 3:
@@ -756,7 +756,23 @@ def forward(self, message: Tensor) -> tuple[Tensor, ...]:
756756
"Message must have shape [batch,length] or [batch,length,vocab]."
757757
)
758758
flattened = embedded.reshape(embedded.shape[0], -1)
759-
representation = torch.tanh(self.message_projection(flattened))
759+
return torch.tanh(self.message_projection(flattened))
760+
761+
def forward(self, message: Tensor) -> tuple[Tensor, ...]:
762+
representation = self._message_representation(message)
763+
return tuple(head(representation) for head in self.factor_heads)
764+
765+
766+
class DeepPositionAwareMLPReceiver(PositionAwareMLPReceiver):
767+
"""Add one generic shared hidden layer before all factor heads."""
768+
769+
def __init__(self, spec: ModelSpec) -> None:
770+
super().__init__(spec)
771+
self.hidden_projection = nn.Linear(spec.hidden_dim, spec.hidden_dim)
772+
773+
def forward(self, message: Tensor) -> tuple[Tensor, ...]:
774+
representation = self._message_representation(message)
775+
representation = torch.tanh(self.hidden_projection(representation))
760776
return tuple(head(representation) for head in self.factor_heads)
761777

762778

@@ -825,6 +841,8 @@ def make_receiver(spec: ModelSpec) -> ReceiverModel:
825841
return Receiver(spec)
826842
if spec.receiver_family == "position_aware_mlp_receiver":
827843
return PositionAwareMLPReceiver(spec)
844+
if spec.receiver_family == "deep_position_aware_mlp_receiver":
845+
return DeepPositionAwareMLPReceiver(spec)
828846
if spec.receiver_family == "factorized_permutation_slot_receiver":
829847
return FactorizedPermutationSlotReceiver(spec)
830848
raise ValueError(f"Unknown receiver architecture: {spec.receiver_family}")

‎tests/conftest.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,3 +98,8 @@ def ecp7_b10_config():
9898
@pytest.fixture(scope="session")
9999
def ecp7_b11_config():
100100
return load_config(PROJECT_ROOT / "config" / "ecp7-b11-development.yaml")
101+
102+
103+
@pytest.fixture(scope="session")
104+
def ecp7_b12_config():
105+
return load_config(PROJECT_ROOT / "config" / "ecp7-b12-development.yaml")

‎tests/test_config.py‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -432,3 +432,26 @@ def test_code_utilization_weight_decay_must_be_bounded(ecp7_b11_config):
432432
] = 1.0
433433
with pytest.raises(ConfigError, match="below its initial weight"):
434434
validate_config(excessive_final)
435+
436+
437+
def test_ecp7_b12_changes_only_the_shared_decoder_depth(
438+
ecp7_b10_config, ecp7_b12_config
439+
):
440+
assert ecp7_b12_config["world"] == ecp7_b10_config["world"]
441+
assert ecp7_b12_config["dataset"] == ecp7_b10_config["dataset"]
442+
assert ecp7_b12_config["channel"] == ecp7_b10_config["channel"]
443+
assert ecp7_b12_config["training"] == ecp7_b10_config["training"]
444+
assert ecp7_b12_config["agents"]["sender"] == ecp7_b10_config["agents"][
445+
"sender"
446+
]
447+
assert ecp7_b12_config["agents"]["population"] == ecp7_b10_config["agents"][
448+
"population"
449+
]
450+
assert ecp7_b12_config["agents"]["receiver"] == {
451+
"family": "deep_position_aware_mlp_receiver"
452+
}
453+
b10_translator = deepcopy(ecp7_b10_config["agents"]["translator"])
454+
b12_translator = deepcopy(ecp7_b12_config["agents"]["translator"])
455+
assert b10_translator.pop("family") == "position_aware_mlp_receiver"
456+
assert b12_translator.pop("family") == "deep_position_aware_mlp_receiver"
457+
assert b12_translator == b10_translator

0 commit comments

Comments
 (0)