Skip to content

Commit c270f60

Browse files
authored
Record sealed ECP-7 sender-consensus batch (#6)
Add and evaluate the factor-agnostic straight-through sender-consensus intervention while keeping the confirmatory split sealed.
1 parent b9333cb commit c270f60

12 files changed

Lines changed: 330 additions & 20 deletions

File tree

‎README.md‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -26,11 +26,12 @@ The protocol uses no words or alphabet, but four meaning-free local symbols. For
2626
See [`docs/results-ecp6.md`](docs/results-ecp6.md) for the conclusion, [`docs/protocol-specification-ecp6.md`](docs/protocol-specification-ecp6.md) for the wire format and [`evidence/ecp6/report.md`](evidence/ecp6/report.md) for the compact confirmatory evidence.
2727

2828
ECP-7 development now tests how much of that result depends on the explicit
29-
one-factor-per-slot architecture. Its first four sealed batches are valid
29+
one-factor-per-slot architecture. Its first five sealed batches are valid
3030
negative results. Straight-through hard utilization remains the best weakly
31-
structured variant at 585–972 messages and 1.92% validation. A direct minibatch
32-
collision penalty regressed to 426–579 messages and 0.42%. The ECP-6 positive
33-
controls remained perfect. The ECP-7 confirmatory test remains sealed. See
31+
structured variant at 585–972 messages and 1.92% validation. Direct collision
32+
pressure regressed, while sender consensus raised exact agreement to 44.02% but
33+
collapsed the code to 169–231 messages. The ECP-6 positive controls remained
34+
perfect. The ECP-7 confirmatory test remains sealed. See
3435
[`docs/development-log-ecp7.md`](docs/development-log-ecp7.md).
3536

3637
## Build on this work
@@ -75,6 +76,7 @@ Do not translate those artifacts in place. New experiment configurations and all
7576
- [`config/ecp7-b2-development.yaml`](config/ecp7-b2-development.yaml) — factor-agnostic soft code-utilization intervention.
7677
- [`config/ecp7-b3-development.yaml`](config/ecp7-b3-development.yaml) — the same utilization formula on straight-through hard messages.
7778
- [`config/ecp7-b4-development.yaml`](config/ecp7-b4-development.yaml) — direct joint-message collision pressure on top of Batch 3.
79+
- [`config/ecp7-b5-development.yaml`](config/ecp7-b5-development.yaml) — straight-through sender consensus on the Batch 3 base.
7880
- [`config/ecp7-positive-control-development.yaml`](config/ecp7-positive-control-development.yaml) — ECP-6 positive control on the ECP-7 split.
7981
- [`config/ecp0.yaml`](config/ecp0.yaml) — machine-readable configuration from step 1.
8082
- [`config/ecp1.yaml`](config/ecp1.yaml) — frozen configuration of the population trial.

‎config/ecp7-b5-development.yaml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
extends: ecp7-b3-development.yaml
2+
3+
experiment:
4+
title: ECP-7 straight-through sender-consensus development
5+
arm: straight_through_sender_consensus
6+
objective: >-
7+
Test whether factor-agnostic pairwise agreement between independent senders
8+
creates one shared hard-message convention while the Batch 3 utilization
9+
objective preserves codebook diversity.
10+
11+
training:
12+
sender_message_consensus:
13+
enabled: true
14+
weight: 1.0
15+
warmup_steps: 400

‎docs/AI_AGENT_START.md‎

Lines changed: 20 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ python3.12 -m venv .venv
4141
.venv/bin/ecp6 --config config/ecp6.yaml validate
4242
```
4343

44-
Expected baseline: 58 passing tests and split sizes `14336/1024/1024`.
44+
Expected baseline: 62 passing tests and split sizes `14336/1024/1024`.
4545

4646
## 4. Reproduce ECP-6
4747

@@ -71,19 +71,21 @@ This reruns a known experiment. It must not be presented as a new independent co
7171

7272
## 5. Continue ECP-7 scientifically
7373

74-
Do not modify ECP-0 through ECP-6 or the completed ECP-7 Batch 1 through Batch 4
74+
Do not modify ECP-0 through ECP-6 or the completed ECP-7 Batch 1 through Batch 5
7575
configs. ECP7-B1-I collapsed to 130–139 hard messages. ECP7-B2-I improved its
7676
soft objective but collapsed further to 85–104 hard messages. ECP7-B3-I applied
7777
that loss to straight-through hard messages and improved to 585–972 messages,
7878
but still reached only 1.92% validation. ECP7-B4-I added direct minibatch
79-
collision pressure and regressed to 426–579 messages and 0.42% validation. All
80-
paired ECP-6 positive controls stayed perfect. The confirmatory ECP-7 test is
81-
still sealed.
79+
collision pressure and regressed to 426–579 messages and 0.42% validation.
80+
ECP7-B5-I raised exact inter-sender agreement from 2.47% to 44.02%, but did so
81+
through a shared collapse to 169–231 messages and 0.48% validation. All paired
82+
ECP-6 positive controls stayed perfect. The confirmatory ECP-7 test is still
83+
sealed.
8284

8385
Continue with this sequence:
8486

8587
1. Read `docs/research-design-ecp7.md` and `docs/development-log-ecp7.md`.
86-
2. Define exactly one ECP7-B5 intervention and its failure criterion.
88+
2. Define exactly one ECP7-B6 intervention and its failure criterion.
8789
3. Register the variant and immutable configuration hashes before training.
8890
4. Add tests for every new invariant.
8991
5. Use only `smoke` and `develop`; keep the ECP-7 test split sealed.
@@ -96,22 +98,24 @@ Continue with this sequence:
9698

9799
## 6. Recommended next experiment
98100

99-
Batch 3 remains the strongest weak-structure variant; Batch 4 showed that sparse
100-
minibatch collision pressure is not a useful proxy for global occupancy. The
101-
four B3 senders also agreed on only 2.47% of complete messages. The most
102-
informative next step is one factor-agnostic sender-consensus loss. A clean
103-
ECP7-B5 progression is:
101+
Batch 3 remains the strongest weak-structure variant. Batch 4 showed that sparse
102+
minibatch collision pressure is not a useful proxy for global occupancy, while
103+
Batch 5 showed that agreement pressure creates a shared collapse. Across these
104+
batches, color and especially shape remain weak while size and texture are much
105+
easier. The most informative next step is one minimax factor-reconstruction
106+
objective. A clean ECP7-B6 progression is:
104107

105108
- keep the same world and bit budget;
106109
- keep the joint autoregressive sender and generic isolated receiver;
107-
- return to Batch 3 and add one straight-through loss that aligns sender
108-
symbols for the same input without assigning factors to slots;
110+
- return to Batch 3 and add one fixed objective that emphasizes the currently
111+
worst factor-level cross-entropy without assigning factors to slots;
109112
- measure injectivity, validation composition and new-reader induction;
110113
- rerun the ECP-6 architecture as the frozen positive control.
111114

112-
Do not retain the Batch 4 collision term or combine consensus with architectural
113-
resizing, longer training, variable-length messages or negotiation. That would
114-
make the sender-consensus effect uninterpretable.
115+
Do not retain the Batch 4 collision or Batch 5 consensus terms, and do not
116+
combine factor balancing with architectural resizing, longer training,
117+
variable-length messages or negotiation. That would make its effect
118+
uninterpretable.
115119

116120
## 7. Definition of done for any contribution
117121

‎docs/development-log-ecp7.md‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -268,3 +268,74 @@ confirmatory split remains sealed. The strongest weak-structure variant remains
268268
ECP7-B3-I. A future ECP7-B5 may add one straight-through sender-consensus loss
269269
to B3, testing whether a shared factor-agnostic convention helps receivers
270270
without restoring semantic slot assignments.
271+
272+
## Batch 5 preregistration
273+
274+
Date: July 18, 2026<br>
275+
Seed: `11`<br>
276+
Maximum population steps: `5,000`<br>
277+
Split-SHA-256: `4947058c75ab07cb43a87eb82776b12cb2a7e2eeba7114de110d3b852cbc64cd`<br>
278+
Test unsealed: **no**
279+
280+
ECP7-B5-I returns to the Batch 3 configuration and adds exactly one
281+
factor-agnostic loss: mean straight-through symbol disagreement between every
282+
sender pair on the same minibatch. Its fixed formula, weight and warmup are
283+
recorded in `docs/research-design-ecp7.md`. The Batch 4 collision loss is not
284+
retained. The unchanged positive control is rerun. Smoke is mechanical only
285+
and cannot authorize tuning.
286+
287+
## Batch 5 results
288+
289+
Positive-control run: `runs/ecp7-batch5-control-development/20260718T112234Z-ecp7-development`<br>
290+
Intervention run: `runs/ecp7-batch5-intervention-development/20260718T112234Z-ecp7-development`<br>
291+
Test unsealed: **no**
292+
293+
| Metric | Positive control | ECP7-B5-I |
294+
|---|---:|---:|
295+
| Population train, mean | 100% | 0.1818% |
296+
| Population train, worst link | 100% | 0.1256% |
297+
| Population validation, mean | 100% | 0.4761% |
298+
| Population validation, worst link | 100% | 0.1953% |
299+
| Universal translator, validation | 100% | 0.5859% |
300+
| Exact sender-message agreement | 100% | 44.02% |
301+
| Unique messages per sender | 15,360 | 169–231 |
302+
| Collision meanings per sender | 0 | 15,129–15,191 |
303+
| Message entropy | 13.91 bits | 6.02–6.13 bits |
304+
| Development gate | pass | **fail** |
305+
306+
The positive control again validated the batch at 100%. ECP7-B5-I failed the
307+
population, validation, translator and injectivity gates. It stopped at step
308+
2,000 under the unchanged early-stopping rule, with step 400 selected.
309+
310+
The intervention answered its narrow question: exact complete-message
311+
agreement between sender pairs increased from Batch 3's 2.47% to 44.02%.
312+
However, the shared convention was mostly a shared collapse. Unique messages
313+
fell from Batch 3's 585–972 to 169–231 and entropy fell from 8.20–8.73 to
314+
6.02–6.13 bits. At the selected checkpoint, sender-consensus loss was `0.2673`
315+
and hard-utilization loss was only `-0.5350`, compared with Batch 3's selected
316+
hard-utilization value of approximately `-0.7075`.
317+
318+
Population validation factor accuracies were approximately
319+
`[6.43%, 6.31%, 64.42%, 78.00%]`. Consensus therefore did not recover color or
320+
shape and also weakened size and texture. Population validation fell from
321+
Batch 3's 1.9226% to 0.4761%; translator validation fell from 2.2705% to
322+
0.5859%.
323+
324+
Both sealed analyses report 65 matching artifact hashes, 16,384 validation-only
325+
episodes, no confirmatory-test keys, valid local alphabets, and exactly 14
326+
declared bits for every logged message.
327+
328+
## Batch 5 decision
329+
330+
ECP7-B5-I is recorded as a negative development result and is not retained. It
331+
shows that inter-sender agreement is not sufficient: without a stronger
332+
semantic pressure, independent senders can agree by discarding distinctions.
333+
The confirmatory split remains sealed and ECP7-B3-I remains the strongest
334+
weak-structure variant.
335+
336+
Across B3–B5, color and especially shape remain the persistent bottleneck while
337+
size and texture are easier to learn. A future ECP7-B6 may return to B3 and add
338+
one minimax factor-reconstruction objective that emphasizes the currently
339+
worst decoded factor without assigning any factor to a channel slot. It must
340+
not retain the rejected B4 or B5 losses or change the architecture at the same
341+
time.

‎docs/research-design-ecp7.md‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -237,3 +237,45 @@ The batch contains exactly two seed-11 runs:
237237
All existing validity and development gates remain unchanged. Failure is
238238
recorded without confirmatory test access. No alternative collision weight,
239239
batch size, architecture or duration may be tried inside Batch 4.
240+
241+
## Batch 5 preregistration — Straight-through sender consensus
242+
243+
Status: **preregistered for sealed development**<br>
244+
Preregistered: July 18, 2026<br>
245+
Configuration: `config/ecp7-b5-development.yaml`<br>
246+
Raw configuration SHA-256: `3333e3e150e242b0984e79a2923ffe94d68cee7d004ae51f4914097917503db7`<br>
247+
Effective configuration SHA-256: `4c2648efa2719d1cd3e0bdf96a2cbea21ea697280b74bbce82ba45371bbc0ed9`
248+
249+
Batch 4 regressed relative to Batch 3, so Batch 5 returns to the stronger
250+
ECP7-B3-I base and does not retain the joint-collision loss. Batch 3's four
251+
senders used substantially more of the hard code space than earlier batches,
252+
but their complete-message agreement on the accessible meanings was only
253+
approximately 2.47%. Batch 5 asks whether one shared factor-agnostic convention
254+
helps generic receivers learn the protocol.
255+
256+
For the same training minibatch, the new objective compares each unordered
257+
pair of senders. It averages their straight-through one-hot symbol disagreement
258+
over inputs and message slots, then averages over all sender pairs:
259+
260+
`L_sender = mean_(a<b,x,s) [1 - <m_a(x,s), m_b(x,s)>]`
261+
262+
The forward value is zero when two senders emit the same hard symbol and one
263+
when they differ. Gradients flow through the straight-through Gumbel estimator.
264+
The objective receives no factor labels, factor identities, target slots,
265+
desired symbols or privileged codebook. Per-sender hard utilization from Batch
266+
3 remains active to oppose agreement through collapse.
267+
268+
The sender-consensus weight is fixed at `1.0` with a 400-step linear warmup.
269+
This is the only Batch 5 addition. Architecture, task loss, Batch 3 hard
270+
utilization, population, split, 14-bit channel, 5,000-step limit, early
271+
stopping and translator remain unchanged. The rejected Batch 4 collision loss
272+
is absent.
273+
274+
The batch contains exactly two seed-11 runs:
275+
276+
1. the unchanged ECP-6 positive control;
277+
2. ECP7-B5-I with straight-through sender consensus on the Batch 3 base.
278+
279+
All existing validity and development gates remain unchanged. Failure is
280+
recorded without confirmatory test access. No alternative consensus weight,
281+
warmup, reduction, architecture or duration may be tried inside Batch 5.

‎evidence/ecp7-development/README.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,11 @@ Batch 4 added direct minibatch full-message collision pressure. It regressed to
2929
not a useful proxy for global codebook occupancy. Exact identities and hashes
3030
are in `batch4-manifest.json`.
3131

32+
Batch 5 aligned straight-through symbols across independent senders on the
33+
Batch 3 base. Exact complete-message agreement rose from 2.47% to 44.02%, but
34+
the shared protocol collapsed to 169–231 messages and 0.48% validation. Exact
35+
identities and hashes are in `batch5-manifest.json`.
36+
3237
Reproduce the development arms with:
3338

3439
```bash
Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
{
2+
"batch": "ECP7-B5",
3+
"confirmatory_test_unsealed": false,
4+
"control": {
5+
"artifact_hashes": {
6+
"effective_config.yaml": "4bfb976293f117d3ae35f790e4fd6371628be95d91febb334bc275dc5d6b66d4",
7+
"posthoc-analysis.json": "c1584664fb80be9aead8cc0f475f25f7406aaab127a647e0e56ab6e0df95bfb3",
8+
"seed-11/metrics.json": "2dc9534c05f386328a12e4a332dc1efbb4f73eebbb3485ae31494de88fde5bed",
9+
"summary.json": "b0bbf9350b501962f340ea7c001d18fe6c7b00dea1dc154e0b5e4bc43b1884fb"
10+
},
11+
"development_gate_passed": true,
12+
"population_train_mean_exact_match": 1.0,
13+
"population_validation_mean_exact_match": 1.0,
14+
"run_id": "20260718T112234Z-ecp7-development",
15+
"universal_translator_validation_exact_match": 1.0,
16+
"unique_messages_per_sender": [15360, 15360, 15360, 15360]
17+
},
18+
"intervention": {
19+
"artifact_hashes": {
20+
"effective_config.yaml": "fa92c6924d3809c85f1517037deadc55db133ce403be00267fc0adb7f141586c",
21+
"posthoc-analysis.json": "27fbe964c56e74309bbf42e6e795ba38091615f1575c9966b7f6ae90b7241d8a",
22+
"seed-11/metrics.json": "357d08228f47614f3c97d84cff414a18fccc23fd28b5fadbf2b8e175094c0504",
23+
"summary.json": "0b3435a7c3d8712c252feba1cf46a19cafdcd328b4e9db8fc692062f35d6b6a9"
24+
},
25+
"best_code_utilization_loss": -0.5349538326263428,
26+
"best_sender_message_consensus_loss": 0.2672525942325592,
27+
"best_step": 400,
28+
"development_gate_passed": false,
29+
"exact_sender_message_agreement": 0.44023437499999996,
30+
"message_entropy_bits_per_sender": [6.125398440530093, 6.019520019497614, 6.091599411068266, 6.106033809745002],
31+
"population_train_mean_exact_match": 0.0018179757254464285,
32+
"population_validation_factor_accuracy": [0.06427001953125, 0.0631103515625, 0.6441650390625, 0.77996826171875],
33+
"population_validation_mean_exact_match": 0.0047607421875,
34+
"run_id": "20260718T112234Z-ecp7-development",
35+
"universal_translator_validation_exact_match": 0.005859375,
36+
"unique_messages_per_sender": [194, 199, 231, 169]
37+
},
38+
"seed": 11,
39+
"split_artifact_sha256": "676195997ee0c77d5c9565bae045dbb938b432a2e4e74b666233b0a41f924718",
40+
"split_sha256": "4947058c75ab07cb43a87eb82776b12cb2a7e2eeba7114de110d3b852cbc64cd"
41+
}

‎src/ai_taal/config.py‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -151,6 +151,13 @@ def validate_config(config: dict[str, Any]) -> None:
151151
if joint_collision.get("warmup_steps", 0) < 1:
152152
raise ConfigError("Joint-collision warmup must be positive.")
153153

154+
sender_consensus = training.get("sender_message_consensus", {})
155+
if sender_consensus.get("enabled", False):
156+
if sender_consensus.get("weight", -1) < 0:
157+
raise ConfigError("Sender-message consensus weight cannot be negative.")
158+
if sender_consensus.get("warmup_steps", 0) < 1:
159+
raise ConfigError("Sender-message consensus warmup must be positive.")
160+
154161

155162
def file_sha256(path: str | Path) -> str:
156163
digest = hashlib.sha256()

‎src/ai_taal/training.py‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -209,6 +209,12 @@ def train_population_system(
209209
joint_collision_config.get("weight", 0.0)
210210
) < 0:
211211
raise ValueError("Joint-collision weight cannot be negative.")
212+
sender_consensus_config = training.get("sender_message_consensus", {})
213+
sender_consensus_enabled = bool(sender_consensus_config.get("enabled", False))
214+
if sender_consensus_enabled and float(
215+
sender_consensus_config.get("weight", 0.0)
216+
) < 0:
217+
raise ValueError("Sender-message consensus weight cannot be negative.")
212218
if population_config["pairing"] != "all_senders_all_receivers":
213219
raise ValueError(
214220
"Population training supports all_senders_all_receivers only."
@@ -362,13 +368,24 @@ def train_population_system(
362368
joint_collision_weight = float(
363369
joint_collision_config["weight"]
364370
) * min(step / joint_collision_warmup, 1.0)
371+
sender_consensus_loss = torch.zeros((), device=device)
372+
sender_consensus_weight = 0.0
373+
if sender_consensus_enabled:
374+
sender_consensus_loss = straight_through_sender_consensus_loss(messages)
375+
sender_consensus_warmup = max(
376+
int(sender_consensus_config.get("warmup_steps", 1)), 1
377+
)
378+
sender_consensus_weight = float(
379+
sender_consensus_config["weight"]
380+
) * min(step / sender_consensus_warmup, 1.0)
365381
loss = (
366382
task_loss
367383
+ consistency_weight * consistency_loss
368384
+ slot_consensus_weight * slot_consensus_loss
369385
+ atom_consensus_weight * atom_consensus_loss
370386
+ utilization_weight * utilization_loss
371387
+ joint_collision_weight * joint_collision_loss
388+
+ sender_consensus_weight * sender_consensus_loss
372389
)
373390
optimizer.zero_grad(set_to_none=True)
374391
loss.backward()
@@ -405,6 +422,12 @@ def train_population_system(
405422
joint_collision_loss.detach().cpu()
406423
),
407424
"joint_message_collision_weight": float(joint_collision_weight),
425+
"sender_message_consensus_loss": float(
426+
sender_consensus_loss.detach().cpu()
427+
),
428+
"sender_message_consensus_weight": float(
429+
sender_consensus_weight
430+
),
408431
"temperature": float(temperature),
409432
"train_mean_exact": train_metrics["mean_exact_match"],
410433
"train_worst_pair_exact": train_metrics["worst_pair_exact_match"],
@@ -637,6 +660,30 @@ def straight_through_joint_collision_loss(
637660
return (full_message_agreement * distinct_input).sum() / meanings.shape[0]
638661

639662

663+
def straight_through_sender_consensus_loss(
664+
message_distributions: list[Tensor],
665+
) -> Tensor:
666+
"""Align complete straight-through messages across independent senders."""
667+
if len(message_distributions) < 2:
668+
raise ValueError("Sender consensus requires at least two senders.")
669+
reference_shape = message_distributions[0].shape
670+
if len(reference_shape) != 3:
671+
raise ValueError(
672+
"Sender-consensus messages must have shape [batch,slots,vocab]."
673+
)
674+
if any(
675+
message.shape != reference_shape for message in message_distributions[1:]
676+
):
677+
raise ValueError("Sender-consensus messages must share one shape.")
678+
679+
pair_losses = []
680+
for left_index, left in enumerate(message_distributions):
681+
for right in message_distributions[left_index + 1 :]:
682+
symbol_agreement = (left * right).sum(dim=-1)
683+
pair_losses.append(1.0 - symbol_agreement.mean())
684+
return torch.stack(pair_losses).mean()
685+
686+
640687
def build_algebraic_quadruples(
641688
train_meanings: tuple[Meaning, ...],
642689
*,

‎tests/conftest.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,3 +63,8 @@ def ecp7_b3_config():
6363
@pytest.fixture(scope="session")
6464
def ecp7_b4_config():
6565
return load_config(PROJECT_ROOT / "config" / "ecp7-b4-development.yaml")
66+
67+
68+
@pytest.fixture(scope="session")
69+
def ecp7_b5_config():
70+
return load_config(PROJECT_ROOT / "config" / "ecp7-b5-development.yaml")

0 commit comments

Comments
 (0)