Recursive language models (RLMs) let a fixed, bounded-context model answer queries over inputs far larger than its context window: the model treats its prompt as an external variable, decomposes it in code, and recursively calls itself on the pieces arXiv preprint. Whether recursion succeeds depends not on the model a[...] the harness that governs how it decomposes the input, when it stops, and how it recovers from errors. Today that harness is supplied either by fine-tuning the model or by an expert hand-engineerin[...]
Note
This repository contains inference code for SH-RLMs with support for various sandbox environments. Open-source contributions are welcome. This repository is maintained by the authors of the pape[...]
This repository separates core runtime, harness specifications, training code, and supporting tools so you can find the part you need quickly.
Top-level layout (annotated):
.gitattributes repository attributes
.github/ GitHub workflows and CI configs
.gitignore ignore rules
AGENTS.md agent-related documentation
Makefile common tasks (install, check, quickstart)
pyproject.toml packaging, dependencies, and extras
uv.lock lockfile for the uv environment
configs/ optional configuration files
docs/ additional documentation
examples/ example scripts and small demos
paper/ paper source and assets
patches/ auxiliary patches
rlm/ core RLM runtime package (clients, environments, logger, utils)
shrlm/ self-harness implementation and spec (surfaces, runner, harness_identity, optimization)
training/ training harness, environments, and examples (rlm-train, oolong env)
visualizer/ web visualizer (Node.js + shadcn/ui) for run logs
tests/ unit tests and test utilities
How it fits together: The rlm/ package implements the runtime client, REPL environments, logging, and utilities used when running RLMs. The shrlm/ package provides the self-harness specification (S1–S10), structural checks (I1–I3), and helpers to build and serialize harnesses for reproducible experiments. The training/ directory holds training harnesses and example environments (including oolong) that depend on the core packages. The visualizer/ is a separate frontend that reads JSONL logs written by RLMLogger in order to show call graphs, code, and sub-call traces.
Where to look next:
- rlm/Readme.md — runtime usage and REPL environments
- shrlm/README.md — self-harness specification and examples
- pyproject.toml — packaging, extras for sandboxes (modal, prime, ipython)
New experiment calls save provider-returned reasoning beside their traces and meta attempts. See saved reasoning outputs for locations, cache behavior, and availability meanings.
The Self-Harness experiment (the optimization loop in shrlm/experiment/) is driven by
examples/run_experiment.py, which loads one profile from one TOML in configs/.
Every experiment parameter lives in that TOML — the code hardcodes none of them — and the
loaded config is hashed into an identity that is stamped into the output directory.
Re-invoking the same command resumes an interrupted experiment exactly; a changed config
refuses to resume, so every distinct configuration gets its own fresh --out-dir.
The smoke, Qwen, and Ox configurations run on OpenRouter and require
OPENROUTER_API_KEY in the environment (a .env file at the repo root works). The
Azure Kimi-K2.5 configuration instead requires:
echo 'AZURE_API_KEY=...' >> .env
echo 'AZURE_FOUNDRY_ENDPOINT=https://<your-resource>.services.ai.azure.com' >> .env
# Verify the deployment's pay-as-you-go rates in the Azure portal match the
# config's [pricing] tables ($0.60 in / $3.00 out per 1M), then attest them --
# Azure returns no per-response cost, so cost is synthesized from these rates
# and the launcher refuses any live spend until the attestation matches:
export SHRLM_VERIFIED_PRICING='0.6/3.0'The config's model value must equal your Foundry deployment name (the catalog
default is Kimi-K2.5). In every configuration the launcher refuses to start — and
spends nothing — if a required credential or attestation is missing.
Note (applies to every Azure Foundry deployment, Kimi-K2.5 included): an empty
completion arriving with finish_reason='length' is classified once as token-limit
exhaustion (TokenLimitExceededError -> resource_terminated, spend kept) instead of
entering the empty-content retry ladder — a length cut is deterministic for the prompt,
so a retry only re-bills the same outcome. Empty completions with other finish reasons
keep the retry ladder as before.
The SHRLM_VERIFIED_PRICING attestation is per config, matched against the
selected config's [pricing.list_price]:
| Config | Attestation |
|---|---|
configs/experiment_kimiK25.toml, configs/experiment_oolong.toml (Kimi-K2.5) |
SHRLM_VERIFIED_PRICING='0.60/3.00' |
configs/experiment_oolong_gptoss.toml (gpt-oss-120b) |
SHRLM_VERIFIED_PRICING='0.15/0.6' |
Paid pytest items are marked live and are deselected (never executed) unless
SHRLM_RUN_LIVE=1 is exported and CI is unset; once selected, module-level skip
gates still name any missing credential or attestation in -rs output.
SHRLM_EXPERIMENT_CONFIG=<path-to-toml> points every live tier — the client live
class, the smoke live round, both recursion tiers, and
examples/experiment_smoke.py via --config — at that config; unset, each tier
keeps its own default (the shipped smoke profile, configs/experiment_kimiK25.toml,
or configs/experiment_oolong.toml).
The gpt-oss live gate ladder (run the steps in order; each must pass before the next spends more):
export AZURE_API_KEY=... AZURE_FOUNDRY_ENDPOINT=https://<your-resource>.services.ai.azure.com
export SHRLM_RUN_LIVE=1 SHRLM_VERIFIED_PRICING='0.15/0.6' # verify in the portal first
export SHRLM_EXPERIMENT_CONFIG=configs/experiment_oolong_gptoss.toml
# 1. Probe (~7 calls, < $0.05): reasoning contract, sampling args, cost synthesis
uv run python examples/experiment_smoke.py --probe \
--config configs/experiment_oolong_gptoss.toml --out-dir ./smoke_gptoss
# 2. Client live (~4 calls, < $0.05): azure_gptoss rows must show PASSED in -rs
uv run pytest -rs -m live tests/clients/test_azure_foundry.py
# 3. Recursion live (4 runs x $0.40 max, budgeted separately -- up to $1.60 on
# top of the $5 smoke-tier ceiling): sub-calls, token and wall-time columns
uv run pytest -rs -m live tests/experiment/test_oolong_recursion_live.py -s
# 4. Smoke live: the whole loop, three roles, cost provenance
uv run python examples/experiment_smoke.py --live \
--config configs/experiment_oolong_gptoss.toml --out-dir ./smoke_gptoss_live
# 5. Dry run ($0): identity, model @ rate, probe verdict round-trip
uv run python examples/run_experiment.py --dry-run \
--config configs/experiment_oolong_gptoss.toml \
--out-dir ./experiment_oolong_gptoss --probe-json ./smoke_gptoss/probe.json| Smoke | Qwen (main) | Ox | Kimi-K2.5 (Azure) | |
|---|---|---|---|---|
| Config file | configs/experiment.toml |
configs/experiment.toml |
configs/experiment_ox.toml |
configs/experiment_kimiK25.toml |
| Profile | smoke |
full (default) |
full (default) |
full (default) |
| Model (all 3 roles) | qwen/qwen3-30b-a3b-instruct-2507 |
qwen/qwen3-30b-a3b-instruct-2507 |
stealth/ox-alpha (free stealth model) |
Kimi-K2.5 deployment on Azure AI Foundry |
| Scale | tiny: 3 held-in / 3 held-out instances, 1 round, 2 candidates | 24 / 40 instances, up to 3 rounds, 4 candidates | same instances as Qwen, up to 16 rounds | same instances as Qwen, up to 3 rounds |
| Purpose | end-to-end plumbing check for cents | the 2026-08-23 comparison run | free-model comparison arm; bounded by wall clock and patience, not spend |
Azure comparison arm; cost synthesized from [pricing] (no provider cost field) |
The smoke profile is the same TOML as the Qwen run with the [smoke.*] tables at the
bottom overriding scale counts only (instance counts, repetitions, candidate width,
rounds, spend caps); every other semantic is identical. The Ox config is a fork of the
Qwen TOML that changes the role models, loop.t = 16, max_output_tokens (ox-alpha is a
reasoning model; see the comment in [decoding]), zeroes the [pricing] tables (the
model bills $0), and gives the attribution/proposal caches _ox-suffixed paths so two
experiments running concurrently from this directory never append to the same cache file.
The Kimi config is likewise a fork of the Qwen TOML that switches the three roles to the
azure_foundry backend, restores the Kimi-K2.5 instant-mode decoding defaults
(temperature 0.6, top_p 0.95, no top_k/min_p), sets both [pricing] tables to the Azure
rate ($0.60/$3.00 per 1M — these arm the spend breaker, since Azure reports no cost), and
uses _kimi-suffixed cache paths.
Append --dry-run to any of these first: it checks credentials, loads the config, and
prints the identity and per-role backends without creating anything or spending anything.
# Smoke: end-to-end plumbing check (small, but spends real money)
uv run python examples/run_experiment.py --profile smoke --out-dir ./experiment_smoke
# Qwen: the main comparison run
uv run python examples/run_experiment.py --out-dir ./experiment_qwen_
# Ox: the stealth/ox-alpha free-model arm
uv run python examples/run_experiment.py --config configs/experiment_ox.toml --out-dir ./experiment_ox
# Azure: KimiK-2.5
uv run python examples/run_experiment.py --config configs/experiment_kimiK25.toml --out-dir ./experiment_kimiInterrupting with Ctrl-C loses nothing — re-run the same command and the experiment resumes at its exact stage boundary. The Qwen and Ox experiments can run concurrently from the same checkout: they share nothing but the (read-only) datasets and the API key.
Proposal generation now checks edited candidates locally and can repair failed
members once while retaining valid siblings; repair can move the same failed
pattern to another eligible, unoccupied surface. Diagnoses cite operations and
verification limits, and the proposer sees the relevant caller and consumer code.
The proposer selects one intervention per surface before writing replacements.
Each proposal compares actual execution with the unresolved operation and explains
why the change addresses its cause. Task-derived reasoning asks which information
must survive and which conditions the final computation must enforce.
The host encodes literal instruction text once; models no longer escape template braces.
A shared capability description explains what each surface can observe, when it
runs, and what it can change. New S8 parsing/aggregation and S5 recovery routes
require admitted operations from held-in traces. A 32,000-character evidence
budget allocates complete core examples across distinct mechanisms before adding
context or contrasts; attribution also preserves complete code and coordinates.
Rejected candidates with comparable recorded quality gains remain visible as
potentially promising in a separately bounded 12,000-character history. Verifiers
declare their primary quality metric, direction, precision and terminal values;
missing or incomparable measurements remain unassessed. History separates claims,
observed retry/redirect/skill-load events, and measured batch outcomes. Revisited
interventions require a prior reference and revision explanation. Identical
rejected evaluated harnesses under the same incumbent are refused before paid calls.
Exact outcomes and regressions remain visible, and v=1 provides no causal guarantee.
Validation still uses one baseline and one combined candidate on held-out data.
See the proposal contract,
metric definitions, and
OOLONG-Pairs investigation.
The methodology note
records these changes for a future paper update.
Nothing needs to be run to get analysis output. After every executed round the orchestrator refreshes aggregation snapshots under the experiment directory:
<out_dir>/
config.json # profile + config identity hash for this experiment
splits/ # the seeded dataset partitions
opt/round_NN/ # per-round artifacts: runs, manifests, proposals, ledgers
analysis/<UTC-stamp>/ # one snapshot per analysis batch, e.g. 20260823T210400Z/
surface_activity.csv
incumbent_quality.csv
...
provenance.json # identity, code revision, source-artifact hashes
published.json # written LAST -- a snapshot without it is incomplete; ignore it
plots/ # figures; re-renderable without touching the frozen CSVs
Snapshots are append-only: a re-run allocates a new UTC-stamped directory rather than
overwriting the previous one, and only directories containing published.json are ever
selected as "latest" by the plotting tools. The post-round refresh is strictly one-way —
the optimization loop never reads analysis/ back.
The same aggregations can be re-run by hand at any time (each allocates a fresh
snapshot): python -m shrlm.experiment.surface_activity <out_dir>,
...incumbent_quality, ...collapse_and_attribution, ...pattern_frequency_diff, and
the plot_* counterparts. The cost/time report is
python -m shrlm.experiment.report <out_dir> --profile <profile> --config <toml>.
Surface activity credits each bundled edit when its linked combined candidate
is promoted, so batch promotions appear on all affected surfaces. Incumbent
quality uses the current round's score of the retained harness: the candidate
score on promotion, otherwise the fresh baseline score. Unscored comparisons
stay blank; earlier scores are never carried forward. The cost report labels
optimization runs and selects their projection basis using [loop].environment
from the supplied config; evaluation results keep their recorded environments.
One caveat for any tool that resolves the config from an out-dir: the default TOML is
configs/experiment.toml, and a config that no longer hashes to the identity recorded in
<out_dir>/config.json resolves to "unknown" (deliberately — no numbers are invented
from a config the experiment never ran under). For the Ox experiment, always pass
--config configs/experiment_ox.toml where the tool accepts it.
The rlm/ package (runtime setup, REPL environments -- local, IPython, Docker, Modal,
Prime -- model providers, trajectory logging, and the visualizer) is documented in
rlm/Readme.md. The self-harness specification (surfaces S1-S10,
invariants I1-I3) is documented in shrlm/README.md.
- [Dec '25] Recursive Language Models arXiv
- [Oct '25] Recursive Language Models Blogpost
- [Jun '26] Self-Harness: Harnesses That Improve Themselves
If you use this code or repository in your research, please cite: