feat(selection): penalise uneven context spread across agents

[lfnothias]

Implements the alternative you proposed in #161: rather than compacting an agent's context, put selection pressure on workflows that concentrate context on one agent instead of distributing it.

Change

qd_score already reads (1 - w)·quality + w·novelty − λ_len·length_penalty. A dispersity term is subtracted alongside it with its own lambda, so the structure is unchanged:

(1 - w)·quality + w·novelty − λ_len·length_penalty − λ_disp·context_dispersity

context_dispersity_lambda defaults to 0.0. With the default, scores are bit-identical and no extra work is done.

The measure

_context_dispersity is the coefficient of variation of the per-agent context lengths, divided by sqrt(n - 1). That divisor is the coefficient's maximum for n non-negative values, attained only when one agent holds everything, so the term is exactly 1 at total concentration and strictly below it elsewhere.

The normalisation matters. A plain clipped coefficient of variation saturates well before the extreme: with four agents both [10, 10, 10, 900] and [1, 1, 1, 900] exceed 1 and clip to the same value, leaving evolution no gradient in exactly the severe regime the issue is about. Normalised, they read 0.957 and 0.995. The clip is retained regardless, because at total concentration floating point pushes the raw quotient marginally above 1.

The signal

An agent's final context length is the input-token count of the last step it recorded. sources/utils/agent_context.py reads it from the per-agent memory files that cost accounting already walks, and EvolutionEngine attaches the lengths to the run. SelectionPressure reads them off the run object the same way _genotype_chars reads run.code, so the selector performs no filesystem access. A memory file that is missing, truncated or unreadable simply does not contribute, because a ranking term must never abort an iteration.

That read is gated on the lambda being positive. Recovering the lengths costs a full parse of every agent memory file, and those files are largest for precisely the high-context runs this term targets — several megabytes each on a real run. When the penalty is off, nothing is read.

Three limitations worth stating plainly

The measure is scale free, which is what "dispersity between agents" asks for, but it means absolute size is not penalised. [10, 10, 10, 900] and [1000, 1000, 1000, 90000] score identically, and four agents each holding 900k tokens score 0. Concretely, starting from [866000, 5000, 5000, 5000] (dispersity 0.977) and padding the three small agents drives dispersity to 0 while total context grows from 881k to 3.46M tokens. The pressure can be satisfied by raising the floor rather than lowering the ceiling. The existing length penalty covers genotype characters, not context. If you want absolute growth discouraged too, a companion term such as − λ_abs · clip((max_context − baseline)/baseline, 0, 1) would do it, and it stays consistent with your objection to compaction because it is a visible term in qd_score rather than a lossy cut. I would rather you decide that than assume it.

Second, because the measure is normalised per n, it is mildly entangled with agent count: under neutral spreading its expectation decays roughly as 1/sqrt(n - 1), so at λ_disp = 0.05 an eight-agent workflow pays about 0.009 less than a two-agent one. This is inherent to any [0, 1]-normalised concentration measure and I know of no clean fix, but it is not something you asked for.

Third, qd_score also gates archive admission via admit_threshold. At λ_disp = 0.1 a novel but concentrated workflow sitting at qd = 0.375 falls to 0.275 and is refused admission at the default threshold of 0.3. Since context concentration is orthogonal to the behavioural descriptor, that is a small leak in the archive's "keep novel behaviours" contract. At 0.05 the maximum penalty is 0.05 and cannot flip anything not already at the bar, which is why I would suggest 0.05 as the first value worth trying, at parity with length_penalty_lambda.

Adding a trivial agent to game the mean does not help, incidentally: near-empty agents push dispersity up, not down.

Tests and docs

tests/context_dispersity_test.py covers the measure (zero for evenly shared context at any scale and below two agents; growing with concentration; exactly 1 at total concentration; retaining a gradient in the severe regime; the scale-free limitation asserted rather than hidden) and the wiring end to end: that _validate_open_ended derives dispersity from run.agent_context_lengths, that a concentrated run ranks below an evenly spread one, that a run with no recorded lengths is unpenalised, and that an admitted member keeps its dispersity through the archive recompute rather than being silently rescored as balanced. Replacing the wiring with a constant fails three of them.

The knob is documented in docs/reference/configuration.md, including the caveat that it penalises unevenness and not total size.

Suite: 214 passed, 4 skipped. The three astra_exporter_test failures are pre-existing on mimosa_v2 and fixed separately in #177.

[lfnothias]

Added a small cleanup commit: agent_context.py no longer carries a single_agent filename prefix. It matched nothing — both factories write agent memory as task_{agent_name}.json, and the single-agent run saves itself as task_single_agent.json, so task_ already covers it. The single_agent_* name that does exist is the run folder, not a memory file. A test pins that the single-agent memory file is still read.

The same dead prefix exists on mimosa_v2 in pricing.py and is removed separately in #178, so neither pull request reaches into the other's files.

[lfnothias]

A brief note on external evidence that has appeared since this was filed. Lee et al. (arXiv:2607.15524) co-evolve a model with its agent harness on ML research tasks and attribute the gains primarily to task-specific management of inter-agent information flow rather than longer reasoning traces: the converged harnesses impose a task-dependent sparsity pattern on what passes between agents, while normalised output tokens stay roughly flat across iterations and cache usage and inference cost drop substantially. That is the property this penalty selects for, and independent support for the position taken in #161 that the productive pressure is on how context is distributed across agents, not on cutting traces mid-run.

It also bears on the absolute-context question left open in #161: in their runs total trace length barely moves while the flow structure changes, which suggests the scale-free measure captures the operative part.

[Fosowl]

Thanks for the careful writeup, and especially for stating the limitations plainly. That said, I'm going to close this one. We already have enough ablations in flight to isolate the variables we care about, and on the substance I'm not convinced by the premise: a more even context spread across agents doesn't imply a better workflow. Worse, a penalty like this can actively work against exploration, putting selection pressure against workflows that legitimately concentrate context and keeping suboptimal but well-spread candidates alive. That's the opposite of what we want from the search.

On Lee et al. (arXiv:2607.15524): I read it, and it doesn't support this penalty. RHI rewrites the harness only, with a fixed model (§6.1: complementary to, not a replacement for, a stronger model). What it optimizes is which information flows between agents, via contracts. This penalty optimizes evenness of context length across agents. Those are different things, and arguably opposed: the sparse pattern the paper describes (heavy orchestrator, narrow specialists) is exactly what this penalty would punish. The "tokens stay flat" claim also only holds for two of the three models; the paper calls the third inconclusive.