Skip to content

Commit 8bea8f6

Browse files
docs: opt-in W&B/Weave runbook for AppSprout Crashlanded benchmarks (#68)
* docs: document opt-in W&B/Weave for AppSprout Crashlanded runs Keep open-source RLE wandb-optional: AppSprout benchmark recipe only, no --wandb hook implied for run_scenario.py. Co-authored-by: Jason <jkbennitt@users.noreply.github.com> * docs: use a stable heading slug for the W&B runbook Avoid '&' in the README heading so the CONTRIBUTING anchor is unambiguous. Co-authored-by: Jason <jkbennitt@users.noreply.github.com> * docs: note Weave is optional and not in the tracking extra uv sync --extra tracking installs wandb + huggingface-hub only. weave is a separate import, not locked, and stays a no-op if missing. Co-authored-by: Jason <jkbennitt@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Jason <jkbennitt@users.noreply.github.com>
1 parent 6144802 commit 8bea8f6

2 files changed

Lines changed: 31 additions & 1 deletion

File tree

‎CONTRIBUTING.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -203,7 +203,7 @@ docker/ # HeadlessRim Dockerfile, compose, entrypoint
203203
- [felix-agent-sdk](https://github.com/AppSprout-dev/felix-agent-sdk) >= 0.3.0
204204
- [RIMAPI](https://github.com/IlyaChichkov/RIMAPI) C# mod
205205
- httpx, pydantic >= 2.0, pyyaml
206-
- Optional: wandb, huggingface-hub (`uv sync --extra tracking`)
206+
- Optional: wandb, huggingface-hub (`uv sync --extra tracking`). W&B stays opt-in (`--wandb` on `run_benchmark.py` only). Weave is a separate optional import (not in the extra or lockfile); see [README — Weights and Biases (opt-in)](README.md#weights-and-biases-opt-in).
207207

208208
## Questions?
209209

‎README.md‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -228,6 +228,36 @@ python scripts/visualize_results.py results/ --all
228228
python scripts/analyze_spread.py --spread-dir results/spread
229229
```
230230

231+
### Weights and Biases (opt-in)
232+
233+
W&B is **strictly opt-in**. Open-source RLE does not require `wandb`, does not enable it by default, and degrades gracefully if the package is missing or unauthenticated. Third parties who omit `--wandb` need no W&B setup.
234+
235+
AppSprout Crashlanded **benchmark** runs should use W&B / Weave. That is an AppSprout ops policy, not a core default and not a required dependency.
236+
237+
The optional `tracking` extra installs `wandb` and `huggingface-hub` only. `uv sync --extra tracking` enables W&B. Weave tracing is a separate optional import in `wandb_logger.py` and is not in `uv.lock`; install it yourself (`uv pip install weave`) if you want traces, or expand the extra later. Missing `weave` is a no-op.
238+
239+
`scripts/run_scenario.py` has **no `--wandb` hook** today. Do not assume a single-scenario run will log to W&B unless that hook is added later. This recipe is for `run_benchmark.py` only.
240+
241+
```bash
242+
# Optional extra — not part of the default install (wandb + huggingface-hub)
243+
uv sync --extra tracking
244+
245+
# Weave traces only if weave is installed separately (not in the tracking extra)
246+
# uv pip install weave
247+
248+
# Authenticate only if this machine is not already logged in.
249+
# Never commit a W&B key. WANDB_API_KEY may be set in the environment
250+
# (do not print it).
251+
wandb login
252+
253+
python scripts/run_benchmark.py \
254+
--wandb \
255+
--no-think --no-pause \
256+
--ticks 10 --tick-interval 30 --seed 42 \
257+
--output results/crashlanded \
258+
--harness felix --harness baseline
259+
```
260+
231261
## Benchmark Results
232262

233263
**Live board — `spread-2026-09-06`, scoring 1.2, harness × model** — Crashlanded, 10 ticks, seed 42, N=1. **Content-first — not statistically valid (no confidence intervals). Winners advance to N=4; N=4 is not published.** Ranked by mean composite across the run. Featured numbers live on [rle.appsprout.dev](https://rle.appsprout.dev) and the [HF card](https://huggingface.co/datasets/AppSprout/rle-benchmarks) (`runs/spread-2026-09-06`).

0 commit comments

Comments
 (0)