Skip to content

README.md

Latest commit

 

History

History
91 lines (69 loc) · 5.54 KB

README.md

File metadata and controls

91 lines (69 loc) · 5.54 KB
Raw

procgen-generalization-gap

Code and data for What Does a ProcGen Generalization Gap Measure? Action Rules, Residual Entropy, and the Missing Random Floor (Abhisek Keshari, arXiv:2609.32532).

A generalization gap in RL (return on training levels minus return on held-out levels) is usually reported without a reference point. This repository measures one: the return of a uniform-random policy on the same levels under the same evaluation harness. Results cover eight ProcGen environments with PPO at 8M steps (three games extended to 25M):

  • The test-time action rule decides which policy is measured. In miner, the sampled policy scores 5.1× the floor on held-out levels while its argmax scores below the floor in every run. Greedy evaluation places two environments significantly below the floor.
  • Raw policy entropy over-calls non-convergence. It flags six of eight environments, but 32–66% of that entropy lies on actions with identical effects.
  • Against the floor, five of eight sampled policies are clearly above it on held-out levels, and heist's cannot be distinguished from it.

Code, per-episode evaluation returns and analysis scripts are all included. The scripts here recompute every number, table and figure in the paper from the files in results/.

Contents

Path What it is
train_ppo_procgen.py, encoders/ PPO agent (CleanRL ProcGen configuration) and the IMPALA / multiscale-FPN encoders
scripts/eval_gap_curves.py evaluation harness: seeded level draws, balanced per-slot episode counting, burn-in
scripts/tmlr_eval.py evaluates final checkpoints under sampled, greedy and (exploratory) merged-argmax rules on both level sets and three seeded draws, with entropy on the visited states
scripts/tmlr_floor.py uniform-random floor on the same draws
scripts/measurements/ shared harness, per-state action-equivalence probe (merged_entropy.py) and effective-action classes (effective_actions.py)
scripts/tmlr_aggregate.py, tmlr_stats.py, tmlr_entropy.py, tmlr_tables.py, tmlr_cobbe.py, tmlr_merged_stats.py, tmlr_carry.py, tmlr_fig1.py aggregation, statistics, tables, normalized scores, exploratory analysis, appendix tables, Figure 1
results/tmlr/raw/ every evaluation cell (one row per checkpoint × level set × rule × draw) with all 128 episode returns (returns_json)
results/tmlr/T2_floor.csv uniform-random floor, per draw, with episode returns
results/tmlr/T4_plan.md the statistical analysis plan, written before any aggregate was computed
results/tmlr/T4_addendum_merged_argmax.md the exploratory merged-argmax addendum, written before that analysis was run
results/tmlr/*.csv, tables/ aggregated results, statistics and the generated tables
results/measurements/ effective-action partitions and the earlier single-draw evaluation cells used for the exact-reproduction check
results/cka* representation-diagnostic source values (appendix)

Notes on the included documents

The statistics are a pre-specified confirmatory re-analysis, not a pre-registration: after an initial single-draw analysis of the same checkpoints, the analysis plan (results/tmlr/T4_plan.md) fixed the tests, families, margins and stop conditions, and the three-draw re-analysis (which includes the original draw) was run under it. results/tmlr/T4_plan.md and results/tmlr/T4_addendum_merged_argmax.md are included as written before the corresponding analyses were run; only file paths were updated to this repository's layout. In these and the acceptance reports, "v1" denotes an earlier evaluation of the same checkpoints (one seeded level draw per run, in results/measurements/), which the current evaluation reproduces exactly (results/tmlr/T1_acceptance.md, T2_acceptance.md).

Install

pip install -r requirements.txt

The scripts use python3 (set PY to override). Tested with Python 3.10, torch 2.5, gym 0.26.2, procgen 0.10.7, gymnasium 1.x.

Reproduce the tables and figures (no checkpoints needed)

bash scripts/reproduce_tables.sh

This recomputes the run-level results, the pre-specified statistics, the entropy screen, all tables (results/tmlr/tables/), the normalized scores, the exploratory merged-argmax analysis and Figure 1 from the included per-episode returns.

Re-run the evaluation (needs checkpoints)

Model checkpoints are not included because of their size. With checkpoints laid out as <dir>/{8M,25M}/ppo_<env>_<encoder>_s<seed>/final.pt:

export PROCGEN_CKPT_DIR=<dir>
python scripts/tmlr_floor.py                                    # uniform-random floor
python scripts/tmlr_eval.py --budget 8M  --env heist --tag 8M   # one environment; see --help
python scripts/tmlr_eval.py --budget 8M --rules greedy_merged --prefix M1 --tag m  # exploratory rule

Run tmlr_eval.py once per environment and budget (it appends to results/tmlr/raw/ and skips finished cells, so runs can be split or resumed). Evaluation is deterministic given the seeds recorded in every output row (level-draw seed, torch seed, reservoir seed). Training: python train_ppo_procgen.py --help (optional experiment tracking with --track needs the Weights & Biases client).

Citation

@article{keshari2026procgengap,
  title   = {What Does a {ProcGen} Generalization Gap Measure? Action Rules, Residual Entropy, and the Missing Random Floor},
  author  = {Keshari, Abhisek},
  journal = {arXiv preprint arXiv:2609.32532},
  year    = {2026}
}

License

MIT (see LICENSE).