Skip to content

Commit ac99792

Browse files
authored
Record sealed ECP-7 context-invariance batch (#9)
Add and evaluate algebraic context invariance on the parallel sender while keeping the confirmatory split sealed.
1 parent 4cd6ceb commit ac99792

11 files changed

Lines changed: 254 additions & 18 deletions

File tree

‎README.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -26,13 +26,13 @@ The protocol uses no words or alphabet, but four meaning-free local symbols. For
2626
See [`docs/results-ecp6.md`](docs/results-ecp6.md) for the conclusion, [`docs/protocol-specification-ecp6.md`](docs/protocol-specification-ecp6.md) for the wire format and [`evidence/ecp6/report.md`](evidence/ecp6/report.md) for the compact confirmatory evidence.
2727

2828
ECP-7 development now tests how much of that result depends on the explicit
29-
one-factor-per-slot architecture. Its first seven sealed batches are valid
29+
one-factor-per-slot architecture. Its first eight sealed batches are valid
3030
negative results. Straight-through hard utilization remains the best weakly
3131
structured population-validation result at 1.92%. A parallel joint-context
3232
sender increased code use to 3,118–3,415 messages and train exactness to 9.60%,
33-
but shape remained below chance and validation stayed at 0.51%. The ECP-6
34-
positive controls remained perfect. The ECP-7 confirmatory test remains sealed.
35-
See
33+
but shape remained below chance and validation stayed at 0.51%. Adding relaxed
34+
context invariance reduced that gain. The ECP-6 positive controls remained
35+
perfect. The ECP-7 confirmatory test remains sealed. See
3636
[`docs/development-log-ecp7.md`](docs/development-log-ecp7.md).
3737

3838
## Build on this work
@@ -80,6 +80,7 @@ Do not translate those artifacts in place. New experiment configurations and all
8080
- [`config/ecp7-b5-development.yaml`](config/ecp7-b5-development.yaml) — straight-through sender consensus on the Batch 3 base.
8181
- [`config/ecp7-b6-development.yaml`](config/ecp7-b6-development.yaml) — normalized minimax factor reconstruction on the Batch 3 base.
8282
- [`config/ecp7-b7-development.yaml`](config/ecp7-b7-development.yaml) — bounded parallel generation from one shared joint context.
83+
- [`config/ecp7-b8-development.yaml`](config/ecp7-b8-development.yaml) — algebraic context invariance on the parallel sender.
8384
- [`config/ecp7-positive-control-development.yaml`](config/ecp7-positive-control-development.yaml) — ECP-6 positive control on the ECP-7 split.
8485
- [`config/ecp0.yaml`](config/ecp0.yaml) — machine-readable configuration from step 1.
8586
- [`config/ecp1.yaml`](config/ecp1.yaml) — frozen configuration of the population trial.

‎config/ecp7-b8-development.yaml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
extends: ecp7-b7-development.yaml
2+
3+
experiment:
4+
title: ECP-7 parallel sender with context-invariant transitions
5+
arm: parallel_sender_algebraic_consistency
6+
objective: >-
7+
Test whether making the same atomic meaning change produce the same relaxed
8+
message displacement across training contexts converts the parallel
9+
sender's memorized distinctions into a compositional protocol.
10+
11+
training:
12+
algebraic_consistency:
13+
enabled: true
14+
weight: 1.0
15+
warmup_steps: 800
16+
relaxed_temperature: 1.0
17+
batch_size: 32
18+
quadruple_pool_size: 8192

‎docs/AI_AGENT_START.md‎

Lines changed: 14 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ python3.12 -m venv .venv
4141
.venv/bin/ecp6 --config config/ecp6.yaml validate
4242
```
4343

44-
Expected baseline: 68 passing tests and split sizes `14336/1024/1024`.
44+
Expected baseline: 70 passing tests and split sizes `14336/1024/1024`.
4545

4646
## 4. Reproduce ECP-6
4747

@@ -71,7 +71,7 @@ This reruns a known experiment. It must not be presented as a new independent co
7171

7272
## 5. Continue ECP-7 scientifically
7373

74-
Do not modify ECP-0 through ECP-6 or the completed ECP-7 Batch 1 through Batch 7
74+
Do not modify ECP-0 through ECP-6 or the completed ECP-7 Batch 1 through Batch 8
7575
configs. ECP7-B1-I collapsed to 130–139 hard messages. ECP7-B2-I improved its
7676
soft objective but collapsed further to 85–104 hard messages. ECP7-B3-I applied
7777
that loss to straight-through hard messages and improved to 585–972 messages,
@@ -84,12 +84,14 @@ remained near chance and validation reached only 0.46%. All paired ECP-6
8484
positive controls stayed perfect. ECP7-B7-I replaced autoregression with a
8585
parallel joint-context sender. It increased train exactness to 9.60% and used
8686
3,118–3,415 messages, but shape stayed below chance and validation reached only
87-
0.51%. The confirmatory ECP-7 test is still sealed.
87+
0.51%. ECP7-B8-I added context-invariant atomic transitions, but regressed to
88+
857–1,135 messages and 0.32% validation. The confirmatory ECP-7 test is still
89+
sealed.
8890

8991
Continue with this sequence:
9092

9193
1. Read `docs/research-design-ecp7.md` and `docs/development-log-ecp7.md`.
92-
2. Define exactly one ECP7-B8 intervention and its failure criterion.
94+
2. Define exactly one ECP7-B9 intervention and its failure criterion.
9395
3. Register the variant and immutable configuration hashes before training.
9496
4. Add tests for every new invariant.
9597
5. Use only `smoke` and `develop`; keep the ECP-7 test split sealed.
@@ -103,20 +105,20 @@ Continue with this sequence:
103105
## 6. Recommended next experiment
104106

105107
Batch 7 is the strongest optimization result but still memorizes rather than
106-
composes: train exactness and code-space use increased sharply, while held-out
107-
color-shape exactness remained low. The most informative next step is one
108-
context-invariance objective on the parallel sender. A clean ECP7-B8
109-
progression is:
108+
composes. Batch 8 context invariance suppressed that gain. B7 messages change
109+
when shape changes, and its fresh universal translator decodes shape better
110+
than its jointly trained recurrent receivers. The most informative next step is
111+
one generic decoder change. A clean ECP7-B9 progression is:
110112

111113
- keep the same world and bit budget;
112114
- keep the joint autoregressive sender and generic isolated receiver;
113-
- keep the B7 sender and receiver, then add the existing algebraic consistency
114-
loss for identical atomic factor transitions across different contexts;
115+
- keep the B7 sender and losses, then replace the GRU final-state decoder with
116+
one position-aware MLP over all four embedded message slots;
115117
- measure injectivity, validation composition and new-reader induction;
116118
- rerun the ECP-6 architecture as the frozen positive control.
117119

118-
Do not restore the Batch 4, Batch 5 or Batch 6 terms, and do not combine
119-
context-invariance with receiver changes, architectural resizing, longer
120+
Do not restore the Batch 4 through Batch 6 or Batch 8 terms, and do not combine
121+
the receiver change with sender changes, architectural resizing, longer
120122
training, variable-length messages or negotiation. That would make its effect
121123
uninterpretable.
122124

‎docs/development-log-ecp7.md‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -490,3 +490,75 @@ context-invariance objective: the existing algebraic consistency loss for the
490490
same atomic factor transition in two different training contexts. This directly
491491
targets composition rather than memorization and assigns no factor to a message
492492
slot. It must not restore B4–B6 losses or change the receiver simultaneously.
493+
494+
## Batch 8 preregistration
495+
496+
Date: July 18, 2026<br>
497+
Seed: `11`<br>
498+
Maximum population steps: `5,000`<br>
499+
Split-SHA-256: `4947058c75ab07cb43a87eb82776b12cb2a7e2eeba7114de110d3b852cbc64cd`<br>
500+
Test unsealed: **no**
501+
502+
ECP7-B8-I retains the complete Batch 7 architecture and loss set, then activates
503+
one context-invariance term. It matches two training-only instances of the same
504+
atomic factor transition and minimizes the squared difference between their
505+
relaxed message displacements. Its fixed construction and hyperparameters are
506+
recorded in `docs/research-design-ecp7.md`. No factor is assigned to a channel
507+
slot. The unchanged positive control is rerun. Smoke is mechanical only and
508+
cannot authorize tuning.
509+
510+
## Batch 8 results
511+
512+
Positive-control run: `runs/ecp7-batch8-control-development/20260718T120358Z-ecp7-development`<br>
513+
Intervention run: `runs/ecp7-batch8-intervention-development/20260718T120358Z-ecp7-development`<br>
514+
Test unsealed: **no**
515+
516+
| Metric | Positive control | ECP7-B8-I |
517+
|---|---:|---:|
518+
| Population train, mean | 100% | 0.5349% |
519+
| Population train, worst link | 100% | 0.4464% |
520+
| Population validation, mean | 100% | 0.3235% |
521+
| Population validation, worst link | 100% | 0.0977% |
522+
| Universal translator, validation | 100% | 0.6836% |
523+
| Exact sender-message agreement | 100% | 1.18% |
524+
| Unique messages per sender | 15,360 | 857–1,135 |
525+
| Collision meanings per sender | 0 | 14,225–14,503 |
526+
| Message entropy | 13.91 bits | 8.23–8.92 bits |
527+
| Development gate | pass | **fail** |
528+
529+
The positive control again validated the batch at 100%. ECP7-B8-I failed the
530+
population, validation, translator and injectivity gates. It stopped at step
531+
2,200 under the unchanged early-stopping rule, with step 600 selected.
532+
533+
At that checkpoint, algebraic consistency loss was `0.05265` at its scheduled
534+
weight of `0.75`, and the hard-utilization loss was `-0.7321`. The surrogate
535+
could be reduced, but it did not create a useful discrete compositional code.
536+
Compared with B7, training exactness fell from 9.5965% to 0.5349%, unique
537+
messages fell from 3,118–3,415 to 857–1,135, population validation fell from
538+
0.5066% to 0.3235%, and translator validation fell from 2.7588% to 0.6836%.
539+
540+
Population validation factor accuracies were approximately
541+
`[7.63%, 6.57%, 89.04%, 96.64%]`. The context objective therefore suppressed
542+
the strong B7 color signal and left shape near chance. Universal-translator
543+
factor accuracies of `[14.60%, 10.30%, 98.12%, 99.02%]` again show that a fresh
544+
decoder extracts more color and shape information than the jointly trained
545+
population receivers, but far below the registered gate.
546+
547+
Both sealed analyses report 65 matching artifact hashes, 16,384 validation-only
548+
episodes, no confirmatory-test keys, valid local alphabets, and exactly 14
549+
declared bits for every logged message.
550+
551+
## Batch 8 decision
552+
553+
ECP7-B8-I is recorded as a negative development result and is not retained.
554+
Relaxed transition invariance competed with the discrete task and utilization
555+
objectives instead of converting B7's distinctions into composition. The
556+
confirmatory split remains sealed.
557+
558+
B7 remains the strongest optimization base. Its shape minimal pairs change
559+
approximately 1.11–1.13 message positions and concentrate about 64% of that
560+
change in one position, while its post-hoc universal translator decodes shape
561+
better than the jointly trained receivers. A future ECP7-B9 may keep the B7
562+
sender and losses but replace the recurrent final-state receiver with one
563+
generic position-aware MLP over all four embedded slots. This tests decoder
564+
access without factor-to-slot binding and must not add another loss.

‎docs/research-design-ecp7.md‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -356,3 +356,44 @@ All existing validity and development gates remain unchanged. Failure is
356356
recorded without confirmatory test access. No receiver change, hidden-size
357357
change, extra loss, longer duration or alternative parallel architecture may be
358358
tried inside Batch 7.
359+
360+
## Batch 8 preregistration — Algebraic context invariance
361+
362+
Status: **preregistered for sealed development**<br>
363+
Preregistered: July 18, 2026<br>
364+
Configuration: `config/ecp7-b8-development.yaml`<br>
365+
Raw configuration SHA-256: `3eac0d25bf636d4a1aeb6161ad479a4a78f0a3e88788a0274c154c21ff192829`<br>
366+
Effective configuration SHA-256: `6e8f9ea0e968693bd9f67a4ca1180ab447c6e2d6c2fbc6b9f9a66582aa02fa3c`
367+
368+
Batch 7 greatly improved training exactness and code-space use, but those
369+
distinctions did not compose across the held-out color-shape matching. Batch 8
370+
retains the entire B7 architecture and loss set, then activates one existing
371+
context-invariance objective that was previously disabled.
372+
373+
Every training quadruple contains meanings `A`, `B`, `A'`, and `B'`. The
374+
transitions `A→A'` and `B→B'` change the same factor value in two different
375+
contexts, and all four meanings belong to the training split. For relaxed
376+
message distributions `m`, the objective is:
377+
378+
`L_alg = mean ||(m(A') - m(A)) - (m(B') - m(B))||²`
379+
380+
This encourages the same atomic semantic change to produce the same message
381+
displacement regardless of the other factor values. It uses factor identities
382+
only to construct matching transitions. It does not specify a target message,
383+
symbol, channel slot, factor-to-slot binding or codebook.
384+
385+
The loss weight is fixed at `1.0` with an 800-step linear warmup, relaxed
386+
temperature `1.0`, 32 quadruples per update and a deterministic pool of 8,192
387+
training-only quadruples. These values match the dormant inherited settings;
388+
only `enabled` and `weight` change from B7. The parallel sender, generic
389+
receiver, hard-utilization loss, channel, split, training duration, selection
390+
and translator remain unchanged.
391+
392+
The batch contains exactly two seed-11 runs:
393+
394+
1. the unchanged ECP-6 positive control;
395+
2. ECP7-B8-I with algebraic context invariance on the B7 parallel sender.
396+
397+
All existing validity and development gates remain unchanged. Failure is
398+
recorded without confirmatory test access. No alternative quadruple sampling,
399+
weight, warmup, receiver, architecture or duration may be tried inside Batch 8.

‎evidence/ecp7-development/README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,10 @@ sender. It reached 3,118–3,415 messages, 9.60% train exactness and 45.64% colo
4343
accuracy, but shape remained below chance and validation stayed at 0.51%. Exact
4444
identities and hashes are in `batch7-manifest.json`.
4545

46+
Batch 8 added algebraic context invariance to Batch 7. The relaxed surrogate
47+
improved, but the discrete protocol regressed to 857–1,135 messages and 0.32%
48+
validation. Exact identities and hashes are in `batch8-manifest.json`.
49+
4650
Reproduce the development arms with:
4751

4852
```bash
Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
{
2+
"batch": "ECP7-B8",
3+
"confirmatory_test_unsealed": false,
4+
"control": {
5+
"artifact_hashes": {
6+
"effective_config.yaml": "4bfb976293f117d3ae35f790e4fd6371628be95d91febb334bc275dc5d6b66d4",
7+
"posthoc-analysis.json": "f22ba2bf1042a97deb5274f03fab22d70678d1418986927764ced363634e0846",
8+
"seed-11/metrics.json": "e9bde51415a531c0ad32f7caf921048840dff9c1f6d55b1ad4635292ff668b60",
9+
"summary.json": "ad2301a93c46c175d2b38687dfe7b66e8b75a14e02b8971fdf18a8a30d3273ae"
10+
},
11+
"development_gate_passed": true,
12+
"population_train_mean_exact_match": 1.0,
13+
"population_validation_mean_exact_match": 1.0,
14+
"run_id": "20260718T120358Z-ecp7-development",
15+
"universal_translator_validation_exact_match": 1.0,
16+
"unique_messages_per_sender": [15360, 15360, 15360, 15360]
17+
},
18+
"intervention": {
19+
"artifact_hashes": {
20+
"effective_config.yaml": "15fa04bb33b321b2ca8c29f01da84ff8bbcdd70e85ecbe96c08d6a51cca84cef",
21+
"posthoc-analysis.json": "4f1af42cbd75044cd7149a729a4869d81b55a3c3e8b929da0a08570d0e608c60",
22+
"seed-11/metrics.json": "945cf063cb1300029175a590452d7e2f39e1ff068b6eb423892357f15581c745",
23+
"summary.json": "7fefb6774ba89d3300aaa2138640057484181591940ce918c27349a5e1fa17ff"
24+
},
25+
"best_algebraic_consistency_loss": 0.05264976993203163,
26+
"best_code_utilization_loss": -0.7321186661720276,
27+
"best_step": 600,
28+
"development_gate_passed": false,
29+
"exact_sender_message_agreement": 0.011827256944444446,
30+
"message_entropy_bits_per_sender": [8.229443275271354, 8.409431758360435, 8.922917128759757, 8.81365004262187],
31+
"population_train_mean_exact_match": 0.005349295479910714,
32+
"population_validation_factor_accuracy": [0.0762939453125, 0.065673828125, 0.890380859375, 0.9664306640625],
33+
"population_validation_mean_exact_match": 0.00323486328125,
34+
"run_id": "20260718T120358Z-ecp7-development",
35+
"universal_translator_validation_exact_match": 0.0068359375,
36+
"unique_messages_per_sender": [857, 895, 1123, 1135]
37+
},
38+
"seed": 11,
39+
"split_artifact_sha256": "676195997ee0c77d5c9565bae045dbb938b432a2e4e74b666233b0a41f924718",
40+
"split_sha256": "4947058c75ab07cb43a87eb82776b12cb2a7e2eeba7114de110d3b852cbc64cd"
41+
}

‎src/ai_taal/training.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@
1919
PopulationSystem,
2020
Receiver,
2121
ReceiverModel,
22-
Sender,
22+
SenderModel,
2323
make_receiver,
2424
make_sender,
2525
)
@@ -756,7 +756,7 @@ def build_algebraic_quadruples(
756756

757757

758758
def algebraic_consistency_loss(
759-
sender: Sender, quadruples: Tensor, *, temperature: float
759+
sender: SenderModel, quadruples: Tensor, *, temperature: float
760760
) -> Tensor:
761761
"""Make the same factor change context-invariant in message distribution."""
762762
if quadruples.ndim != 3 or quadruples.shape[1:] != (4, 4):

‎tests/conftest.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,3 +78,8 @@ def ecp7_b6_config():
7878
@pytest.fixture(scope="session")
7979
def ecp7_b7_config():
8080
return load_config(PROJECT_ROOT / "config" / "ecp7-b7-development.yaml")
81+
82+
83+
@pytest.fixture(scope="session")
84+
def ecp7_b8_config():
85+
return load_config(PROJECT_ROOT / "config" / "ecp7-b8-development.yaml")

‎tests/test_config.py‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -297,3 +297,34 @@ def test_ecp7_b7_changes_only_the_sender_generation_architecture(
297297
assert "joint_message_collision" not in b7_training
298298
assert "sender_message_consensus" not in b7_training
299299
assert "factor_minimax" not in b7_training
300+
301+
302+
def test_ecp7_b8_adds_only_algebraic_context_invariance(
303+
ecp7_b7_config, ecp7_b8_config
304+
):
305+
assert ecp7_b8_config["world"] == ecp7_b7_config["world"]
306+
assert ecp7_b8_config["dataset"] == ecp7_b7_config["dataset"]
307+
assert ecp7_b8_config["channel"] == ecp7_b7_config["channel"]
308+
assert ecp7_b8_config["agents"] == ecp7_b7_config["agents"]
309+
b7_training = deepcopy(ecp7_b7_config["training"])
310+
b8_training = deepcopy(ecp7_b8_config["training"])
311+
assert b7_training.pop("algebraic_consistency") == {
312+
"enabled": False,
313+
"weight": 0.0,
314+
"warmup_steps": 800,
315+
"relaxed_temperature": 1.0,
316+
"batch_size": 32,
317+
"quadruple_pool_size": 8192,
318+
}
319+
assert b8_training.pop("algebraic_consistency") == {
320+
"enabled": True,
321+
"weight": 1.0,
322+
"warmup_steps": 800,
323+
"relaxed_temperature": 1.0,
324+
"batch_size": 32,
325+
"quadruple_pool_size": 8192,
326+
}
327+
assert b8_training == b7_training
328+
assert "joint_message_collision" not in b8_training
329+
assert "sender_message_consensus" not in b8_training
330+
assert "factor_minimax" not in b8_training

0 commit comments

Comments
 (0)