Skip to content

Commit b9333cb

Browse files
authored
Record sealed ECP-7 joint-collision batch (#5)
Add and evaluate the factor-agnostic straight-through joint-message collision intervention while keeping the confirmatory split sealed.
1 parent e4814a1 commit b9333cb

12 files changed

Lines changed: 301 additions & 20 deletions

File tree

‎README.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -26,12 +26,11 @@ The protocol uses no words or alphabet, but four meaning-free local symbols. For
2626
See [`docs/results-ecp6.md`](docs/results-ecp6.md) for the conclusion, [`docs/protocol-specification-ecp6.md`](docs/protocol-specification-ecp6.md) for the wire format and [`evidence/ecp6/report.md`](evidence/ecp6/report.md) for the compact confirmatory evidence.
2727

2828
ECP-7 development now tests how much of that result depends on the explicit
29-
one-factor-per-slot architecture. Its first three sealed batches are valid
30-
negative results. A generic sender collapsed to 130–139 messages, soft
31-
utilization reduced this to 85–104, and straight-through hard utilization
32-
improved it to 585–972 messages and 1.92% validation. The ECP-6 positive
33-
controls remained perfect. The direction is informative but still far below
34-
the development gate, so the ECP-7 confirmatory test remains sealed. See
29+
one-factor-per-slot architecture. Its first four sealed batches are valid
30+
negative results. Straight-through hard utilization remains the best weakly
31+
structured variant at 585–972 messages and 1.92% validation. A direct minibatch
32+
collision penalty regressed to 426–579 messages and 0.42%. The ECP-6 positive
33+
controls remained perfect. The ECP-7 confirmatory test remains sealed. See
3534
[`docs/development-log-ecp7.md`](docs/development-log-ecp7.md).
3635

3736
## Build on this work
@@ -75,6 +74,7 @@ Do not translate those artifacts in place. New experiment configurations and all
7574
- [`config/ecp7-development.yaml`](config/ecp7-development.yaml) — sealed weak-structure intervention configuration.
7675
- [`config/ecp7-b2-development.yaml`](config/ecp7-b2-development.yaml) — factor-agnostic soft code-utilization intervention.
7776
- [`config/ecp7-b3-development.yaml`](config/ecp7-b3-development.yaml) — the same utilization formula on straight-through hard messages.
77+
- [`config/ecp7-b4-development.yaml`](config/ecp7-b4-development.yaml) — direct joint-message collision pressure on top of Batch 3.
7878
- [`config/ecp7-positive-control-development.yaml`](config/ecp7-positive-control-development.yaml) — ECP-6 positive control on the ECP-7 split.
7979
- [`config/ecp0.yaml`](config/ecp0.yaml) — machine-readable configuration from step 1.
8080
- [`config/ecp1.yaml`](config/ecp1.yaml) — frozen configuration of the population trial.

‎config/ecp7-b4-development.yaml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
extends: ecp7-b3-development.yaml
2+
3+
experiment:
4+
title: ECP-7 direct joint-message collision development
5+
arm: straight_through_joint_collision
6+
objective: >-
7+
Test whether directly penalizing complete straight-through message
8+
collisions between distinct inputs closes the higher-order utilization gap
9+
left by marginal entropy and pairwise slot independence.
10+
11+
training:
12+
joint_message_collision:
13+
enabled: true
14+
weight: 1.0
15+
warmup_steps: 400

‎docs/AI_AGENT_START.md‎

Lines changed: 17 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ python3.12 -m venv .venv
4141
.venv/bin/ecp6 --config config/ecp6.yaml validate
4242
```
4343

44-
Expected baseline: 54 passing tests and split sizes `14336/1024/1024`.
44+
Expected baseline: 58 passing tests and split sizes `14336/1024/1024`.
4545

4646
## 4. Reproduce ECP-6
4747

@@ -71,17 +71,19 @@ This reruns a known experiment. It must not be presented as a new independent co
7171

7272
## 5. Continue ECP-7 scientifically
7373

74-
Do not modify ECP-0 through ECP-6 or the completed ECP-7 Batch 1 through Batch 3
74+
Do not modify ECP-0 through ECP-6 or the completed ECP-7 Batch 1 through Batch 4
7575
configs. ECP7-B1-I collapsed to 130–139 hard messages. ECP7-B2-I improved its
7676
soft objective but collapsed further to 85–104 hard messages. ECP7-B3-I applied
7777
that loss to straight-through hard messages and improved to 585–972 messages,
78-
but still reached only 1.92% validation. All paired ECP-6 positive controls
79-
stayed perfect. The confirmatory ECP-7 test is still sealed.
78+
but still reached only 1.92% validation. ECP7-B4-I added direct minibatch
79+
collision pressure and regressed to 426–579 messages and 0.42% validation. All
80+
paired ECP-6 positive controls stayed perfect. The confirmatory ECP-7 test is
81+
still sealed.
8082

8183
Continue with this sequence:
8284

8385
1. Read `docs/research-design-ecp7.md` and `docs/development-log-ecp7.md`.
84-
2. Define exactly one ECP7-B4 intervention and its failure criterion.
86+
2. Define exactly one ECP7-B5 intervention and its failure criterion.
8587
3. Register the variant and immutable configuration hashes before training.
8688
4. Add tests for every new invariant.
8789
5. Use only `smoke` and `develop`; keep the ECP-7 test split sealed.
@@ -94,21 +96,22 @@ Continue with this sequence:
9496

9597
## 6. Recommended next experiment
9698

97-
Batch 3 partly closed the soft-to-hard gap, but marginal balance and pairwise
98-
independence still allowed many complete-message collisions. The most
99-
informative next step is one direct joint-collision penalty. A clean ECP7-B4
100-
progression is:
99+
Batch 3 remains the strongest weak-structure variant; Batch 4 showed that sparse
100+
minibatch collision pressure is not a useful proxy for global occupancy. The
101+
four B3 senders also agreed on only 2.47% of complete messages. The most
102+
informative next step is one factor-agnostic sender-consensus loss. A clean
103+
ECP7-B5 progression is:
101104

102105
- keep the same world and bit budget;
103106
- keep the joint autoregressive sender and generic isolated receiver;
104-
- retain the full Batch 3 objective and add one loss equal to the average
105-
number of other distinct inputs with the same straight-through full message;
107+
- return to Batch 3 and add one straight-through loss that aligns sender
108+
symbols for the same input without assigning factors to slots;
106109
- measure injectivity, validation composition and new-reader induction;
107110
- rerun the ECP-6 architecture as the frozen positive control.
108111

109-
Do not combine this with architectural resizing, changes to the existing loss
110-
weight, longer training, variable-length messages or negotiation. That would
111-
make any change from Batch 3 uninterpretable.
112+
Do not retain the Batch 4 collision term or combine consensus with architectural
113+
resizing, longer training, variable-length messages or negotiation. That would
114+
make the sender-consensus effect uninterpretable.
112115

113116
## 7. Definition of done for any contribution
114117

‎docs/development-log-ecp7.md‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ this log.
1111
| ECP7-B1-I | Intervention | Joint-context bounded autoregressive | Generic sequence encoder | completed; gate failed |
1212
| ECP7-B2-I | Intervention | B1 sender plus factor-agnostic code utilization | Generic sequence encoder | completed; gate failed |
1313
| ECP7-B3-I | Intervention | B2 loss on straight-through hard messages | Generic sequence encoder | completed; gate failed |
14+
| ECP7-B4-I | Intervention | B3 plus direct full-message collision loss | Generic sequence encoder | completed; gate failed |
1415

1516
No other ECP-7 variant has been trained.
1617

@@ -207,3 +208,63 @@ confirmatory split remains sealed and no frozen ECP-7 configuration is created.
207208
A future ECP7-B4 may add one direct straight-through full-message collision
208209
penalty while retaining all Batch 3 settings. This targets joint collisions
209210
without assigning semantic factors to slots.
211+
212+
## Batch 4 preregistration
213+
214+
Date: July 18, 2026<br>
215+
Seed: `11`<br>
216+
Maximum population steps: `5,000`<br>
217+
Split-SHA-256: `4947058c75ab07cb43a87eb82776b12cb2a7e2eeba7114de110d3b852cbc64cd`<br>
218+
Test unsealed: **no**
219+
220+
ECP7-B4-I retains all Batch 3 settings and adds one factor-agnostic loss: the
221+
average number of distinct minibatch inputs sharing the same complete
222+
straight-through message. Its exact formula, weight and warmup are recorded in
223+
`docs/research-design-ecp7.md`. The unchanged positive control is rerun. Smoke
224+
cannot authorize tuning.
225+
226+
## Batch 4 results
227+
228+
Positive-control run: `runs/ecp7-batch4-control-development/20260718T111008Z-ecp7-development`<br>
229+
Intervention run: `runs/ecp7-batch4-intervention-development/20260718T111141Z-ecp7-development`<br>
230+
Test unsealed: **no**
231+
232+
| Metric | Positive control | ECP7-B4-I |
233+
|---|---:|---:|
234+
| Population train, mean | 100% | 0.3204% |
235+
| Population train, worst link | 100% | 0.2860% |
236+
| Population validation, mean | 100% | 0.4211% |
237+
| Population validation, worst link | 100% | 0.2930% |
238+
| Universal translator, validation | 100% | 0.6104% |
239+
| Unique messages per sender | 15,360 | 426–579 |
240+
| Collision meanings per sender | 0 | 14,781–14,934 |
241+
| Message entropy | 13.91 bits | 7.46–7.84 bits |
242+
| Development gate | pass | **fail** |
243+
244+
The positive control again validated the batch at 100%. ECP7-B4-I failed every
245+
performance and injectivity gate and regressed relative to Batch 3.
246+
247+
At the selected step 800 checkpoint, the average number of distinct colliding
248+
partners in the current minibatch was `0.03125`, while the retained hard
249+
utilization loss was `-0.7213`. Despite that low local collision count, global
250+
evaluation found only 426–579 messages across 15,360 accessible meanings. The
251+
minibatch signal was therefore too sparse and local to represent global
252+
codebook occupancy.
253+
254+
Population validation dropped from Batch 3's 1.9226% to 0.4211%; message entropy
255+
dropped from 8.20–8.73 to 7.46–7.84 bits. Factor accuracies were approximately
256+
`[6.48%, 6.41%, 92.13%, 83.79%]`, so the added loss also disrupted the two
257+
factors previously learned almost exactly. Changing its weight after seeing
258+
this outcome is not permitted inside Batch 4.
259+
260+
Both sealed analyses report 65 matching artifact hashes, 16,384 validation-only
261+
episodes, no confirmatory-test keys, valid local alphabets, and exactly 14
262+
declared bits for every logged message.
263+
264+
## Batch 4 decision
265+
266+
ECP7-B4-I is recorded as a negative development result and is not retained. The
267+
confirmatory split remains sealed. The strongest weak-structure variant remains
268+
ECP7-B3-I. A future ECP7-B5 may add one straight-through sender-consensus loss
269+
to B3, testing whether a shared factor-agnostic convention helps receivers
270+
without restoring semantic slot assignments.

‎docs/research-design-ecp7.md‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -199,3 +199,41 @@ unchanged. B3 must pass population, validation, translator, injectivity, channel
199199
and sealed-test checks together. A failure remains a development result and
200200
does not authorize confirmatory test access. No alternative estimator, weight
201201
or training schedule may be tried inside Batch 3.
202+
203+
## Batch 4 preregistration — Direct joint-message collisions
204+
205+
Status: **preregistered for sealed development**<br>
206+
Preregistered: July 18, 2026<br>
207+
Configuration: `config/ecp7-b4-development.yaml`<br>
208+
Raw configuration SHA-256: `6229c5cd13c796a9ce297b9c38cc2ff1a2090a2a22a5678ec7afe38aa0bd6955`<br>
209+
Effective configuration SHA-256: `4cffdb040ba358d7416e0643595c4e11fadc0fdffd5c0120761c4890a8c3f10e`
210+
211+
Batch 3 improved marginal hard utilization but retained thousands of complete
212+
message collisions. Batch 4 asks whether directly penalizing those joint
213+
collisions produces an injective and compositional code without a semantic
214+
factor-to-slot binding.
215+
216+
For every sender and minibatch, the new loss is the average number of other
217+
distinct inputs with the same complete straight-through message:
218+
219+
`L_joint = (1/B) × sum_i sum_j!=i 1[x_i != x_j] × product_s <m_i,s, m_j,s>`
220+
221+
Because the forward messages are one-hot, the slot dot product is one exactly
222+
when both symbols match, and the product is one exactly when the full messages
223+
match. Identical sampled inputs are excluded; no factor label, factor identity
224+
or desired code is provided. Straight-through gradients flow back through the
225+
hard messages.
226+
227+
The joint-collision weight is fixed at `1.0` with a 400-step linear warmup. This
228+
is the only Batch 4 addition. The complete Batch 3 utilization objective,
229+
architecture, task loss, population, split, 14-bit channel, 5,000-step limit,
230+
early stopping and translator remain unchanged.
231+
232+
The batch contains exactly two seed-11 runs:
233+
234+
1. the unchanged ECP-6 positive control;
235+
2. ECP7-B4-I with the added direct joint-collision loss.
236+
237+
All existing validity and development gates remain unchanged. Failure is
238+
recorded without confirmatory test access. No alternative collision weight,
239+
batch size, architecture or duration may be tried inside Batch 4.

‎evidence/ecp7-development/README.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,11 @@ It remained below the gate but improved to 585–972 hard messages, 8.20–8.73
2424
bits of entropy and 1.92% validation. Exact identities and hashes are in
2525
`batch3-manifest.json`.
2626

27+
Batch 4 added direct minibatch full-message collision pressure. It regressed to
28+
426–579 messages and 0.42% validation, showing that the sparse local signal was
29+
not a useful proxy for global codebook occupancy. Exact identities and hashes
30+
are in `batch4-manifest.json`.
31+
2732
Reproduce the development arms with:
2833

2934
```bash
Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
{
2+
"batch": "ECP7-B4",
3+
"confirmatory_test_unsealed": false,
4+
"control": {
5+
"artifact_hashes": {
6+
"effective_config.yaml": "4bfb976293f117d3ae35f790e4fd6371628be95d91febb334bc275dc5d6b66d4",
7+
"posthoc-analysis.json": "4c028786125509cb33a081662d12c28743cc212d13161a2360948c82a46e4ec1",
8+
"seed-11/metrics.json": "7af4742b93a1a68dd0603e5647cd7c33161a41af24a5e40deb95b7fb997efb04",
9+
"summary.json": "2aa1a8fcf8d791a6fe4ef90f669b0564e6251cfd800476b178f63c464d9053fe"
10+
},
11+
"development_gate_passed": true,
12+
"population_train_mean_exact_match": 1.0,
13+
"population_validation_mean_exact_match": 1.0,
14+
"run_id": "20260718T111008Z-ecp7-development",
15+
"universal_translator_validation_exact_match": 1.0,
16+
"unique_messages_per_sender": [15360, 15360, 15360, 15360]
17+
},
18+
"intervention": {
19+
"artifact_hashes": {
20+
"effective_config.yaml": "fd98dbb08a8cf11bd6c5e6d05f24c5950f48f11cec2e39024e901abd67bd9c9f",
21+
"posthoc-analysis.json": "67fbb886d49e328c0643990b7b5d013c2fcb2831a4cdee0e0645a6082f7adb0f",
22+
"seed-11/metrics.json": "f280df0ab86a0ba7555b6384bd370e3765f419d39f66f71c0a8a76704c79b297",
23+
"summary.json": "1539a99f35f80467eafb5e1c7e90169c7724c862de7f9a45ced2e467b10f1e6a"
24+
},
25+
"best_code_utilization_loss": -0.7212955355644226,
26+
"best_joint_message_collision_loss": 0.03125,
27+
"best_step": 800,
28+
"development_gate_passed": false,
29+
"message_entropy_bits_per_sender": [7.514687306501297, 7.837134515331, 7.501597202083582, 7.462425878518177],
30+
"population_train_mean_exact_match": 0.003204345703125,
31+
"population_validation_factor_accuracy": [0.06475830078125, 0.06414794921875, 0.9212646484375, 0.837890625],
32+
"population_validation_mean_exact_match": 0.00421142578125,
33+
"run_id": "20260718T111141Z-ecp7-development",
34+
"universal_translator_validation_exact_match": 0.006103515625,
35+
"unique_messages_per_sender": [449, 579, 498, 426]
36+
},
37+
"seed": 11,
38+
"split_artifact_sha256": "676195997ee0c77d5c9565bae045dbb938b432a2e4e74b666233b0a41f924718",
39+
"split_sha256": "4947058c75ab07cb43a87eb82776b12cb2a7e2eeba7114de110d3b852cbc64cd"
40+
}

‎src/ai_taal/config.py‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -144,6 +144,13 @@ def validate_config(config: dict[str, Any]) -> None:
144144
"Code-utilization message_source must be relaxed or straight_through."
145145
)
146146

147+
joint_collision = training.get("joint_message_collision", {})
148+
if joint_collision.get("enabled", False):
149+
if joint_collision.get("weight", -1) < 0:
150+
raise ConfigError("Joint-collision weight cannot be negative.")
151+
if joint_collision.get("warmup_steps", 0) < 1:
152+
raise ConfigError("Joint-collision warmup must be positive.")
153+
147154

148155
def file_sha256(path: str | Path) -> str:
149156
digest = hashlib.sha256()

‎src/ai_taal/training.py‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -203,6 +203,12 @@ def train_population_system(
203203
raise ValueError(
204204
"Code-utilization message source must be relaxed or straight_through."
205205
)
206+
joint_collision_config = training.get("joint_message_collision", {})
207+
joint_collision_enabled = bool(joint_collision_config.get("enabled", False))
208+
if joint_collision_enabled and float(
209+
joint_collision_config.get("weight", 0.0)
210+
) < 0:
211+
raise ValueError("Joint-collision weight cannot be negative.")
206212
if population_config["pairing"] != "all_senders_all_receivers":
207213
raise ValueError(
208214
"Population training supports all_senders_all_receivers only."
@@ -341,12 +347,28 @@ def train_population_system(
341347
utilization_weight = float(utilization_config["weight"]) * min(
342348
step / utilization_warmup, 1.0
343349
)
350+
joint_collision_loss = torch.zeros((), device=device)
351+
joint_collision_weight = 0.0
352+
if joint_collision_enabled:
353+
joint_collision_loss = torch.stack(
354+
[
355+
straight_through_joint_collision_loss(message, batch)
356+
for message in messages
357+
]
358+
).mean()
359+
joint_collision_warmup = max(
360+
int(joint_collision_config.get("warmup_steps", 1)), 1
361+
)
362+
joint_collision_weight = float(
363+
joint_collision_config["weight"]
364+
) * min(step / joint_collision_warmup, 1.0)
344365
loss = (
345366
task_loss
346367
+ consistency_weight * consistency_loss
347368
+ slot_consensus_weight * slot_consensus_loss
348369
+ atom_consensus_weight * atom_consensus_loss
349370
+ utilization_weight * utilization_loss
371+
+ joint_collision_weight * joint_collision_loss
350372
)
351373
optimizer.zero_grad(set_to_none=True)
352374
loss.backward()
@@ -379,6 +401,10 @@ def train_population_system(
379401
"atom_code_consensus_weight": float(atom_consensus_weight),
380402
"code_utilization_loss": float(utilization_loss.detach().cpu()),
381403
"code_utilization_weight": float(utilization_weight),
404+
"joint_message_collision_loss": float(
405+
joint_collision_loss.detach().cpu()
406+
),
407+
"joint_message_collision_weight": float(joint_collision_weight),
382408
"temperature": float(temperature),
383409
"train_mean_exact": train_metrics["mean_exact_match"],
384410
"train_worst_pair_exact": train_metrics["worst_pair_exact_match"],
@@ -594,6 +620,23 @@ def factor_agnostic_code_utilization_loss(
594620
return determinism - utilization + independence_weight * independence
595621

596622

623+
def straight_through_joint_collision_loss(
624+
message_distributions: Tensor, meanings: Tensor
625+
) -> Tensor:
626+
"""Average colliding distinct inputs per item in a straight-through batch."""
627+
if message_distributions.ndim != 3:
628+
raise ValueError("Joint-collision messages must have shape [batch,slots,vocab].")
629+
if meanings.ndim != 2 or meanings.shape[0] != message_distributions.shape[0]:
630+
raise ValueError("Joint-collision meanings must match the message batch.")
631+
per_slot_agreement = torch.einsum(
632+
"bsv,csv->bcs", message_distributions, message_distributions
633+
)
634+
full_message_agreement = per_slot_agreement.prod(dim=-1)
635+
same_input = meanings[:, None, :].eq(meanings[None, :, :]).all(dim=-1)
636+
distinct_input = (~same_input).to(full_message_agreement.dtype)
637+
return (full_message_agreement * distinct_input).sum() / meanings.shape[0]
638+
639+
597640
def build_algebraic_quadruples(
598641
train_meanings: tuple[Meaning, ...],
599642
*,

‎tests/conftest.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,3 +58,8 @@ def ecp7_b2_config():
5858
@pytest.fixture(scope="session")
5959
def ecp7_b3_config():
6060
return load_config(PROJECT_ROOT / "config" / "ecp7-b3-development.yaml")
61+
62+
63+
@pytest.fixture(scope="session")
64+
def ecp7_b4_config():
65+
return load_config(PROJECT_ROOT / "config" / "ecp7-b4-development.yaml")

0 commit comments

Comments
 (0)