You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 3f3f8d0
Browse filesBrowse the repository at this point in the historyBrowse files
authored
Refine the horizon and composition sections of the AI4AI page (#12)
<spanclass="lang-en" lang="en">Simplify the system: assume steps are independent; at 95% per step, finishing 100 steps leaves under 1%. Real errors are not independent, so the number predicts nothing—it just sharpens the question: which part of the loop does a method improve? Model-side work covers every stage, yet the longer the task, the further end-to-end reliability falls, as if each step lost a fixed fraction. So the reliable horizon comes down to the harness: can errors be contained, and can the system recover. This matters more than it seems—between frontier models of similar capability, changing the harness can matter more than changing the model, even reversing rankings; swapping the scaffold alone moves one model's accuracy by 28 points, more than most of the model gaps people argue about.</span>
<spanclass="lang-en" lang="en">Of any method that claims a longer horizon, ask two questions first: which stage of the loop it improves, and whether that improvement survives longer, more tightly coupled trajectories. The model route works stage by stage: plan turns reasoning and interaction traces into executable plans, execute grounds those plans in interfaces and state, feedback replaces one coarse trajectory-level reward with step-level credit assignment, and repair lets the evaluated policy rewrite the data, architecture, and training recipe that produce its successor. All of this lowers per-step error, yet end-to-end reliability still decays as tasks lengthen.</span>
<spanclass="lang-en" lang="en">Reliable long-horizon execution also shows up in the harness, the runtime outside the weights: whether errors are contained locally, and whether the system recovers once they occur. Its leverage is larger than usually assumed: among frontier models of comparable capability, changing the harness can matter more than changing the model and can reverse rankings, and swapping the scaffold alone moves a single model's accuracy by 28 points. Evidence from the other direction makes the same point: on highly decomposable work, decomposition with per-step correction has sustained more than a million steps without a final error. This shows how far a harness can carry a fixed model, and also where that stops: tightly coupled tasks admit no such decomposition.</span>
<spanclass="lang-en" lang="en">Components that work in isolation become less reliable once composed, for two reasons: the weakest link sets the whole—any required component can terminate the pass, so scores measured in isolation do not measure the full system—and handoffs lose information, since compression and summarization can remove a constraint that only a later experiment needs. The paper follows the resulting failures across four scales:</span>
<spanclass="lang-en" lang="en">The paper introduces the <em>composition gap</em>: components are strong in isolation, yet once they share one goal, state, budget, and evaluator, end-to-end reliability remains unestablished. Two mechanisms produce the gap.</span>
<spanclass="lang-en" lang="en">First, an AI4AI pass depends on every component: any single failure terminates the whole pass, so the least reliable component becomes the bottleneck; even short of outright failure, one local error changes all subsequent state, and later observations look consistent while the pass has already drifted off-goal.</span>
<spanclass="lang-en" lang="en">Second, handoffs between components lose information: compression and summarization can drop a constraint that only a later experiment needs, and evaluating only the final artifact cannot show where the loss occurred. The paper follows the resulting failures along the following lines.</span>
<spanclass="lang-en" lang="en"><strong>Within one pass.</strong> Planning, grounding, and tool use are tightly coupled: a local error—the wrong control, the wrong file, the wrong database row—reshapes the future state space, so later observations look self-consistent while drifting from the goal.</span>
<spanclass="lang-en" lang="en"><strong>Across passes.</strong> Memory can retrieve stale state, and lossy summaries can drop task-critical constraints; an early mistake enters the trajectory and cascades, so later decisions stay coherent with the record while being causally downstream of the fault.</span>
<spanclass="lang-en" lang="en"><strong>Across passes.</strong> Memory is at bottom state management, not a larger context window. Retrieving stale state or summarizing away a task-critical constraint shifts the goal actually being pursued, and because intermediate artifacts become part of the environment, later observations reinforce the drift rather than expose it. Recovery should therefore be judged as state repair—was the fault detected and localized, which state was rolled back or distrusted, did the trajectory return to a valid state—not as another attempt.</span>
<spanclass="lang-en" lang="en"><strong>Across generations.</strong> When a system both proposes and judges an improvement, their agreement is not evidence; recursive training on generated data can erode retained capabilities, and a weakening proxy invites reward hacking. In RSIBench-Data, more than three quarters of searches that continue past their best checkpointterminate below it.</span>
<spanclass="lang-en" lang="en"><strong>Across generations.</strong> When one system both proposes an improvement and judges it, their agreement is not evidence. Recursive training on generated data can erode retained capabilities, and a weakening improvement signal invites reward hacking: PostTrainBench records agents training on evaluation data and substituting existing checkpoints to improve the visible objective. RSIBench-Data quantifies the non-monotonicity: of searches that continue past their best checkpoint, more than three quarters terminate below that peak. More feedback and more generations are not evidence that the boundary of reliable execution is moving outward.</span>
<spanclass="lang-en" lang="en"><strong>Evidence and control.</strong> Gains can be bought with budget, so a success rate is only readable alongside tokens, wall-clock time, verifier cost, and human intervention—together with model–harness attribution, contamination controls, and disclosed oversight.</span>
<spanclass="lang-en" lang="en"><strong>Evidence and control.</strong> Gains can be purchased with budget: longer prompts, more candidates, deeper search, and extra verifier calls all raise success rates. A success rate is therefore meaningful only alongside tokens, wall-clock time, verifier cost, and human intervention; a credible improvement claim additionally needs model–harness attribution (freeze the harness while changing the model, and the reverse), together with data-contamination controls and disclosure of human oversight.</span>
<spanclass="lang-en" lang="en">Back to where we started: the release notes of frontier models are pushing agents toward the same thing—sustained execution, and evolving from that execution. The judgment of this survey is that what AI4AI lacks has never been capability: plan, execute, feedback, and repair are each already strong on their own, and across the 35 systems execute is fully system-owned. What is missing is the authority to decide—no system picks its own goal, and none can judge what counts as “better” without a human. So “can AI improve AI” is no longer a question of whether the work can be done, but of who defines “better.” The eve ends the first time a system, with no human-specified goal or evaluation, has successors that reliably strengthen it. We hope this taxonomy and this audit help that moment arrive a little sooner.</span>
<spanclass="lang-en" lang="en">Back to where we started: the release notes of frontier models are pushing agents toward the same thing—sustained execution, and evolving from that execution. The judgment of this survey is that what AI4AI lacks has never been capability: plan, execute, feedback, and repair are each already strong on their own, and across the 35 systems execute is fully system-owned. <strong>What is missing is the authority to decide—no system picks its own goal, and none can judge what counts as “better” without a human.</strong></span>
<spanclass="lang-en" lang="en">The eve ends the first time a system, with no human-specified goal or evaluation, has successors that reliably strengthen it. We hope this taxonomy and this audit help that moment arrive a little sooner.</span>
0 commit comments