Skip to content

README.en.md

Latest commit

 

History

History
56 lines (45 loc) · 1.98 KB

README.en.md

File metadata and controls

56 lines (45 loc) · 1.98 KB
Raw

ppo-from-scratch

🇺🇸 English | 🇧🇷 Português

PPO (Proximal Policy Optimization, Clip variant) implemented from scratch in PyTorch, with GAE and a component-ablation study.

Overview

A modular, on-policy PPO implementation in four decoupled units: ActorCritic (network), RolloutBuffer (storage + GAE), PPO (clipped objective + update), and PPOTrainer (collect-rollout → update loop). Includes an ablation study measuring the effect of removing GAE, clipping, and the entropy bonus.

Architecture

  • ActorCritic (ppo/networks.py) — shared trunk, policy + value heads.
  • RolloutBuffer (ppo/rollout_buffer.py) — on-policy rollout; GAE/returns; minibatches. use_gae flag.
  • PPO (ppo/ppo.py) — act, update (PPO-clip objective). use_clipping, use_entropy flags.
  • PPOTrainer (ppo/trainer.py) — on-policy loop + metrics (History).

The three ablation flags live in a single configurable PPO/buffer — no duplicated code.

Installation (uv)

uv sync --extra dev

Usage

import gymnasium as gym
from ppo import PPO, RolloutBuffer, PPOTrainer

env = gym.make("CartPole-v1")
obs_dim = env.observation_space.shape[0]; n = env.action_space.n
agent = PPO(obs_dim, n)
buf = RolloutBuffer(size=2048, obs_dim=obs_dim)
trainer = PPOTrainer(agent, env, buf, seed=0)
hist = trainer.train(total_steps=100_000, rollout_len=2048)
print(trainer.evaluate(num_episodes=10))

Experiments

uv run python experiments/run_cartpole.py      # validation (PPO reaches ~414/500)
uv run python experiments/run_lunarlander.py   # full PPO
uv run python experiments/run_ablation.py      # ablation (GAE/clip/entropy)

Results in results/. Docs: docs/ — including the ablation study, where removing GAE proved the most harmful.

Tests

uv run pytest

Includes exact numerical tests of GAE and the clipped objective.

License

MIT — see LICENSE.