Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,9 @@ This repository is a reproducible research project, not a generic chat applicati
Read these files in order before changing code:

1. `README.md`
2. `docs/resultaten-ecp6.md`
3. `docs/protocolspecificatie-ecp6.md`
4. `docs/onderzoeksopzet-ecp6.md`
2. `docs/results-ecp6.md`
3. `docs/protocol-specification-ecp6.md`
4. `docs/research-design-ecp6.md`
5. `docs/AI_AGENT_START.md`

Then run:
Expand Down Expand Up @@ -40,6 +40,7 @@ python3.12 -m venv .venv
- Run the full test suite and the relevant config validation before committing.
- Keep existing ECP-0 through ECP-6 configurations reproducible.
- Document new experiments in three stages: preregistration, sealed development log, confirmatory results.
- Write new configurations, documentation, reports and user-facing output in English.
- When proposing ECP-7, prefer a single falsifiable intervention over several simultaneous architecture changes.

The recommended next research question is whether compositional structure still emerges when the explicit one-factor-per-slot architectural bias is weakened.
163 changes: 86 additions & 77 deletions README.md

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions docs/AI_AGENT_START.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,10 +78,10 @@ Do not modify ECP-6. Use this sequence:
3. Create a genuinely new deterministic holdout or world seed.
4. Add tests for every new invariant before training.
5. Use only `smoke` and `develop`; keep the ECP-7 test split sealed.
6. Record all tried variants in `docs/ontwikkellog-ecp7.md`.
6. Record all tried variants in `docs/development-log-ecp7.md`.
7. Select one final design using train and validation only.
8. Create `config/ecp7.yaml` with status `frozen_for_confirmatory` and a UTC freeze time.
9. Record config and split SHA-256 values in `docs/onderzoeksopzet-ecp7.md`.
9. Record config and split SHA-256 values in `docs/research-design-ecp7.md`.
10. Only then run the predefined seeds once with `experiment --unseal-test`.
11. Run posthoc integrity analysis and publish both successes and failures.

Expand Down
31 changes: 31 additions & 0 deletions docs/development-log-ecp1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# Development log ECP-1

All variants below used only training and validation data. The new ECP-1 test split remained sealed during these design choices.

| Variant | Pairings per step | Known mean | Known worst | Validation mean | Conclusion |
|---|---:|---:|---:|---:|---|
| One pair at random | 1 | 92.7% | 91.4% | 61.2% | too few direct pair updates |
| B: one sender, all receivers | 4 | 95.9% | 95.1% | 57.3% | more stable, but known accuracy is insufficient |
| C: all senders, all receivers | 16 | 99.2% | 98.8% | 77.6% | frozen for the confirmatory experiment |

## Variant A

Each cycle used all sixteen pairs once, but only one pair per training step. At the best checkpoint each specific pair had only a fraction of the learning interactions of an ECP-0 pair. The agents did develop a shared convention, but the known reconstruction remained insufficient.

Local, untracked artifact: `runs/20260718T000531Z-ecp1-development/report.md`.

## Variant B

One selected sender sent the same message per step to all four receivers, giving receivers a lot of experience, but each sender still less direct updates. The known performance improved, while validation did not improve.

Local, untracked artifact: `runs/20260718T000811Z-ecp1-development/report.md`.

## Variant C

Each batch ran through all sixteen sender-receiver pairs, and all losses were averaged. There was no leader, shared weight, or extra semantic label. Each pair received the same amount of training exposure as ECP-0.

The universal receiver achieved test-free 78.3% validation. A fresh receiver achieved 74.2% validation with 512 out of 768 training concepts and 77.9% with all meanings.

Local, untracked artifact: `runs/20260718T001224Z-ecp1-development/report.md`.

After this run, architecture, learning settings, thresholds and data split have been frozen. Subsequent changes will get a new experiment ID.
117 changes: 117 additions & 0 deletions docs/development-log-ecp2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
# Development log ECP-2

The confirmatory ECP-2 test set remained sealed throughout this work. All performance below comes exclusively from training and the separate compositional validation split.

## Variant A - algebraic consistency pressure

Seed: 11<br>
Architecture: four independent channels × four independent receivers<br>
Channel: four symbols from a vocabulary of sixteen, 16 bits<br>
Split: `7a227e22d86f00d8ab5078690de37de794774cdab636e7db374c5adbd0985126`

| Variant | Weight | Known mean | Worst known pair | Compositional validation | Universal translator validation |
|---|---:|---:|---:|---:|---:|
| Control | 0 | 97.0% | 96.5% | **9.5%** | **13.9%** |
| Light | 0.25 | 95.8% | 95.1% | 4.6% | 10.0% |
| Medium | 1.0 | 59.9% | 35.2% | 2.6% | 3.5% |
| Strong | 4.0 | 2.0% | 1.6% | 1.4% | 2.3% |

Local artifacts are underneath `runs/ecp2-variants/{control,light,medium,strong}/20260718T005403Z-ecp2-development/`.

### Conclusion

The intervention is rejected. No weight improves the compositional validation and only the control approaches the known performance threshold. The test set is not opened.

Regularization revealed an informative degeneration. Under strong pressure, size and texture are reconstructed perfectly on the validation set, while color and shape can fall to chance level. Because the algebraic requirement is minimized on average across all changes, the sender can satisfy it cheaply by omitting the most difficult factors. A higher task-loss weight would merely rebalance the two objectives and would not remove this fundamental escape route.

Under the predefined fallback, development continues with cultural transmission, which selects for learnability by introducing new agents instead of directly imposing a desired geometry on messages.

## Variant B - Cultural transmission by agent replacement

The population retains four channels and four receivers. After a fixed interval, one sender and one receiver are completely re-initialized. The remaining six agents form the temporary cultural memory. After four replacements, no original agent is present; checkpoints prior to that complete turnover are not selectable.

Three preselected intervals are compared with seed 11 and the same sealed test set:

| Variant | Steps between replacements | First complete turnover |
|---|---:|---:|
| Fast | 400 | Step 1,600 |
| Medium | 600 | Step 2,400 |
| Slow | 800 | Step 3,200 |

The training budget remains 5,000 steps. Selection first uses the known performance thresholds and then only the compositional validation. Algebraic regularization is out in these three variants.

### First outcome

| Variant | Known mean | Worst known pair | Compositional validation | Universal translator validation |
|---|---:|---:|---:|---:|
| Fast, continuous | 97.4% | 93.9% | **19.5%** | **18.4%** |
| Medium, continuous | 95.2% | 92.1% | 10.0% | 12.7% |
| Slow, continuous | 94.1% | 90.8% | 10.1% | 11.9% |

The fast variant almost doubles the compositional validation compared to the 9.5% control, but continues to introduce new agents. As a result, a recently replaced link is always the weakest. The following pre-defined refinement separates transmission and consolidation:

| Variant | Interval | Maximum replacements | Turnovers | Consolidation after last replacement |
|---|---:|---:|---:|---:|
| One fast turnover | 400 | 4 | 1 | 3,400 steps |
| Two fast turnovers | 400 | 8 | 2 | 1,800 steps |
| One medium turnover | 600 | 4 | 1 | 2,600 steps |

The final test remains sealed for this refinement.

### Consolidation outcome

| Variant | Known mean | Worst known pair | Compositional validation | Universal translator validation |
|---|---:|---:|---:|---:|
| One fast turnover | 98.8% | 97.5% | 10.3% | 15.0% |
| Two fast turnovers | **99.3%** | **98.8%** | **19.9%** | **25.2%** |
| One medium turnover | 97.2% | 96.6% | 7.3% | 10.4% |

Two fast turnovers form the best cultural variant. It meets the known-performance thresholds and doubles compositional validation relative to the control, but remains well below the development targets of 80% population and 70% translator accuracy. Cultural transmission is retained as the best weakly structured intervention, but not as the confirmatory ECP-2 candidate.

## Variant C — Learned permutation slots

If the consolidation variants fail to reach the compositional validation threshold, one structurally stronger architecture is tested. The sender contains four discrete message slots and internally chooses a hard one-to-one permutation between the four meaning factors and those slots.

The following are not defined in advance:

- which factor is linked to which slot;
- which of the sixteen symbols represents a factor value;
- whether different channels choose the same internal permutation or symbols;
- how receivers decode the four symbols.

This is an explicit compositional architectural bias and a stricter procedure than algebraic or cultural pressure. The variant is valuable as a working model and as a positive control, but a success should not be described as completely uninfluenced linguistic emergence.

The slot variant is first trained with seed 11, without algebraic regularization or agent replacement. The preselected development thresholds are 97% mean known accuracy, 95% for the worst pair, 80% compositional validation, and 70% universal-translation validation.

### First final outcome and consensus refinement

The first fully trained slot population achieves training/validation without test access:

- 100% known reconstruction for each sender-receiver pair;
- 69.4% average compositional validation;
- 30.5% for the worst compositional pair;
- 50.6% universal translation validation.

The four senders chose three different factor-slot permutations. Two independently converged on the same permutation; the other two chose different ones. One meaning-free refinement is therefore tested: minimize differences among the four soft binding matrices and make each matrix sharp. Consensus does not specify a factor-slot mapping in advance; any of the 24 permutations can win.

Preselected consensus weights:

| Variant | Consensus weight | Sharpness weight |
|---|---:|---:|
| Light | 0.1 | 0.1 |
| Medium | 1.0 | 0.1 |
| Strong | 5.0 | 0.1 |

The final test remains sealed. When validation scores are nearly equal, the lower consensus weight wins.

### Consensus outcome and final choice

| Variant | Known mean | Worst known pair | Compositional validation | Worst validation pair | Universal translator validation |
|---|---:|---:|---:|---:|---:|
| Light, 0.1 | 100% | 100% | 76.9% | 51.6% | 50.6% |
| Medium, 1.0 | 100% | 100% | 90.0% | 82.0% | 95.3% |
| **Strong, 5.0** | **100%** | **100%** | **97.0%** | **93.8%** | **95.3%** |

Weight 5.0 wins unequivocally and becomes the confirmatory ECP-2 candidate. All four channels chose the same internally determined permutation `[color, texture, shape, size]`; the final soft binding probabilities have maxima between 0.968 and 0.986. Across the 896 training and validation meanings, the population uses 896 unique messages, has zero collisions, a minimum-pair distance of exactly one, and topographic similarity 1.0.

The confirmatory intervention is accompanied by a control using exactly the same permutation-slot architecture, split, seeds, and training budget, but without binding consensus. Both configurations are frozen before any ECP-2 test access.
44 changes: 44 additions & 0 deletions docs/development-log-ecp3.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# Development log ECP-3

The confirmatory ECP-3 test split remained sealed throughout all work described below.

## Failure mode in ECP-2

ECP-2 learned a shared factor-slot binding within each seed, but only two of five seeds met every threshold. The weak seeds no longer had a structural slot-binding problem; they remained stuck with non-unique or insufficiently stable symbols for individual factor values. As a result, even known meanings in seeds 37 and 53 could not be fully distinguished.

## Preselected repair

ECP-3 replaces the neural slot heads with a freely learned injective symbol allocation for each factor. The sender may still choose any symbol for each value, but two values within the same factor may not coincide. The existing free factor-slot permutation is retained. In addition to binding consensus, the same meaning-free population consensus is applied to the soft atomic codebooks.

One development seed has been used: seed 11. The training budget, population size, receiver architecture and channel capacity are kept equal to ECP-2.

## Result on training and compositional validation

| Metric | Outcome |
|---|---:|
| Known mean | 100.0% |
| Worst known sender-receiver pair | 100.0% |
| Fully new validation pairs | 100.0% |
| Worst validation pair | 100.0% |
| Universal translator on validation | 100.0% |
| New receiver with 128 examples | 94.3% |
| New receiver with 512 examples | 99.8% |
| New receiver with 768 examples | 100.0% |

All four channels chose the same slot order `[shape, texture, size, color]`. Their exact atomic codebooks did not match everywhere, but every code remained injective and all sixteen sender-receiver links decoded both dialect variants flawlessly.

For each of the four channels, across the 896 training and validation meanings:

- 896 unique messages and zero collisions;
- message entropy of `9.807` bits, equal to the source entropy of this subset;
- mean message distance of exactly one for a one-factor meaning change;
- full concentration of such a change in one position;
- topographic Spearman correlation of `1.0`.

The technical smoke test, checkpoint cycle, isolated processes, shuffled-message control, and consistent symbol permutation all passed. All 29 tests passed.

## Conclusion

No further refinement is allowed before the confirmatory test. The development variant achieves every goal without opening the sealed split and is therefore frozen unchanged as the ECP-3 intervention. An otherwise identical control disables only slot-binding and atomic-code consensus.

Development run: `runs/ecp3-development/20260718T020325Z-ecp3-development`.
50 changes: 50 additions & 0 deletions docs/development-log-ecp4.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# Development log ECP-4

The confirmatory ECP-4 test split remained technically sealed throughout development.

## Implementation

ECP-4 adds two prespecified components:

1. `MinimalPermutationSlotSender`: a freely learned hard permutation for each factor within a local alphabet of exactly the required size;
2. `FactorizedPermutationSlotReceiver`: a freely learned but hard-isolated message slot for each factor output.

The transport metadata, configuration validation, and episode schemas now support protocol-specific bit lengths. Each episode is also checked against the active configuration for exact message length, bit length, and symbol range.

## Technical verification

The tests verify that:

- the four local codebooks exactly `8×8`, `8×8`, `4×4` and `4×4` are hard permutations;
- each symbol actually sent falls within the local factor alphabet;
- a factor output remains unchanged when an unselected slot changes;
- minimum sender and factor receiver run separately through checkpoints and isolated processes;
- `3+3+2+2 = 10` bits exactly equals the source entropy;
- the explicit validation and test splits use exactly the final sixteen unused pairs.

All 34 tests pass.

## Sealed development

Develop run: `runs/ecp4-development/20260718T060943Z-ecp4-development`<br>
Seed: 11

| Metric | Outcome |
|---|---:|
| Known mean | 100.0% |
| Worst known sender-receiver pair | 100.0% |
| Fully new validation pairs | 100.0% |
| Worst validation pair | 100.0% |
| Universal translator on validation | 100.0% |
| New receiver, 32 examples | 100.0% |
| New receiver, 128 examples | 100.0% |
| New receiver, 512 examples | 100.0% |
| New receiver, 768 examples | 100.0% |

All four channels chose the same internally determined slot order `[size, texture, color, shape]`. Message agreement between every sender pair is 100%. Across the 896 training and validation meanings, each sender uses 896 unique messages without collisions and has topographic similarity 1.0.

The best condition was already selected at step 400. No 12-bit intermediate variant is therefore required: the direct 10-bit variant meets every sealed-development criterion.

## Conclusion

The 10-bit variant is frozen without further modification. The paired control retains the same factorized receiver but uses the 16-bit injective ECP-3 channel. No confirmatory test pair was evaluated before this choice.
34 changes: 34 additions & 0 deletions docs/development-log-ecp5.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# Development log ECP-5

The ECP-5 test set remained sealed during development.

## Cause analysis ECP-4

All five 10-bit populations communicated without error. However, four universal translators remained at 9.4–12.5%. Their symbol heads were not the problem: the hard receiver binding assigned factors to the wrong slots. The same error also occurred twice in the 16-bit control.

## Exact calibration

The new procedure calculates a `4×4` matrix of empirical mutual information and scores all 24 permutations. A test with a hidden synthetic slot order recovers the exact inverse binding and freezes it. All 35 tests pass.

## Sealed development run

Run: `runs/ecp5-development-final/20260718T062526Z-ecp5-development`<br>
Seed: 11

| Metric | Outcome |
|---|---:|
| Population known | 100% |
| Worst known pair | 100% |
| New size-texture validation | 100% |
| Worst validation pair | 100% |
| Calibrated universal translator | 100% |
| New receiver with 32 examples | 100% |
| New receiver with 128 examples | 100% |
| New receiver with 256 examples | 100% |
| New receiver with 512 examples | 100% |

The chosen slot order by factor was `[2,3,0,1]`. The total mutual-information score was exactly 10.0 bits; the runner-up scored 8.0 bits. The choice is therefore not marginal. Even with only 32 examples, the correct permutation remained the clear winner.

## Conclusion

Calibration was frozen without further variants. With `binding_calibration.enabled`, no ECP-5 test pair was evaluated before this choice.
36 changes: 36 additions & 0 deletions docs/development-log-ecp6.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# Development log ECP-6

The ECP-6 test set remained sealed throughout all work described below.

## Scale invariants

The code is generalized from the original factor sizes `[8,8,4,4]` to `[16,16,8,8]`. Configuration validation, meaning IDs, schemas, factor-local alphabets, and baselines now follow the actual factor sizes. The complete test suite has 39 passing tests.

The first smoke attempt revealed one geometric scaling problem: topographic similarity materialized all `n(n-1)/2` pairs. The attempt was canceled before completion and without test access. The measurement now uses up to 1,000,000 deterministic, uniformly sampled pairs. A second smoke run completed normally and logged validation only.

## Sealed development run

Run: `runs/20260718T065639Z-ecp6-development`<br>
Seed: 11<br>
Test unsealed: no

| Metric | Outcome |
|---|---:|
| Population known, mean | 100% |
| Worst known link | 100% |
| New color-shape validation, mean | 100% |
| Worst validation link | 100% |
| Universal translator on train | 100% |
| Universal translator on validation | 100% |
| Transfer with 32 examples, validation | 81.25% |
| Transfer with 128 examples, validation | 100% |
| Transfer with 512 examples, validation | 100% |
| Transfer with 2,048 examples, validation | 100% |

Binding calibration selected the slot order `[1,0,3,2]`. The total mutual-information score was exactly 14.0 bits; the runner-up scored 8.0 bits. All four channels encoded the 15,360 accessible training and validation meanings.

The development run wrote 16,384 episodes: `1,024 validation meanings × 4 senders × 4 receivers`. No test episode was written.

## Freezing decision

Development provided no reason for an additional variant. The world, explicit holdout pairs, 14-bit channel, training budgets, binding calibration, transfer budgets, topographic sampling rule, five seeds, and success criteria are therefore frozen unchanged for confirmation.
Loading