Policy Gradient Steering

Code for the experiments in the Policy Gradient Steering paper. The release is organized around independently runnable paper workflows:

configs/   One fixed configuration per experiment script
scripts/   Public experiment entrypoints
src/       Shared implementations and the Light-MALib compatibility layer
results/   Generated data, tables, and figures (ignored by Git)

Scripts beginning with main_ reproduce main-paper experiments; scripts beginning with appendix_ reproduce appendix experiments. Each script loads the configuration with the same filename and writes its outputs to results/<domain>/<script-name>/.

Installation

The project uses Python 3.11 and uv:

uv sync

Install the optional chess dependencies with:

uv sync --extra chess

Football additionally requires Google Research Football. The repository includes the policy checkpoints used by the experiments. These checkpoints are committed intentionally so that the released football workflows use the exact policies evaluated in the paper.

The src/light_malib/ compatibility layer is taken unchanged from the GRF_MARL repository accompanying the MARL paper. The Policy Gradient Steering implementation is maintained separately under src/policy_gradient_steering/.

Running experiments

Run a complete workflow without arguments, for example:

uv run -m scripts.gridworld.main_rollouts
uv run -m scripts.chess.main_composition
uv run -m scripts.football.main_self_play
uv run -m scripts.football.main_self_play_offline

Use --dry-run to inspect the resolved configuration without running an experiment, or --smoke for a small structural check. Existing output directories are not reused unless --resume is passed.

The available workflows are:

Gridworld
  main_rollouts
  main_demonstrations
  main_parameterizations
  appendix_steering_visualization
  appendix_data_budget
  appendix_optim_steps
  appendix_optim_strength
  appendix_normalization
  appendix_site

Chess
  main_composition
  main_cross_layer
  main_models
  appendix_layers
  appendix_data_budget
  appendix_optim_steps
  appendix_parameterization

Football
  main_self_play
  main_self_play_offline
  main_pass
  main_pass_cross_play
  main_behavior_description
  appendix_behavior_description
  appendix_critic
  appendix_data_budget
  appendix_gamma
  appendix_off_policy
  appendix_role
  opponent_conditioned_goals

For any name above, run uv run -m scripts.<domain>.<name>.