Orbit Wars — RL Training Project

A self-play RL agent for the Orbit Wars Kaggle competition. Players send fleets to capture orbiting planets; most ships at turn 500 wins (full rules in Game Specification below).

The agent is an entity-transformer policy trained with PPO self-play on a vectorised GPU env, warm-started from behaviour cloning of top-leaderboard replays. The hard part has never been capturing planets — it's holding them against forward-projecting planner opponents. Most of the work below is the search for a training signal that produces retention instead of spray-and-churn.

Replay

Turns 104–144 of episode 83428941, Saheb vs Johnny Hyland. Click the GIF for the full interactive replay with pause, scrubbing, speed, and turn-by-turn controls.

System at a glance

The project has four separable layers:

1. features.py turns a variable-size observation into padded planet, fleet, global, and source-target tensors. Geometry that should be exact—arrival direction, travel time, sun safety, projected garrison—is computed explicitly; strategic preferences are left to the network.
2. model.py embeds those entities with a shared transformer, then scores actions independently for every owned source planet and candidate target planet.
3. torch_env.py simulates hundreds of games in parallel on one device. train_torch.py collects self-play rollouts and ppo.py applies clipped PPO with GAE.
4. eval.py runs deterministic, both-seat panels against fixed opponents. export_agent.py packages a checkpoint as a standalone Kaggle agent.

flowchart LR
    O[Kaggle observation] --> F[Feature extraction]
    F --> P[Planet tokens]
    F --> L[Fleet tokens]
    F --> G[Global token]
    P & L & G --> T[3-layer entity transformer]
    T --> E[Owned-source embeddings]
    F --> X[36-D source-target features]
    E & X --> H[Target / fire / ship heads]
    T --> V[Value head]
    H --> M[Legality masks]
    M --> D[Orbital-intercept decoder]
    D --> A[Launch actions]
    A --> R[Vectorised GPU environment]
    R --> O

Loading

Policy architecture

The default model uses 96-dimensional entity embeddings, three transformer blocks, and four attention heads. Its inputs are checkpoint-versioned because the feature set evolved during the project:

Input	Default shape	Contents
Planet	48 × 116	20 current-state features + 24-step projected owner/garrison timeline
Fleet	256 × 13	Kinematics, ownership, destination/ETA, and threat context
Global	15	Step, mode, angular velocity, and aggregate economy; optional 48-D economy timeline
Source-target	16 × 48 × 36	Arrival geometry, ownership, capture cost, contest/support, ROI, and counterfactual features

Each owned planet is an action slot. For each slot, the network produces:

• a target distribution over real planets;
• a target-conditioned fire/NOOP decision;
• a target-conditioned ship-size distribution, unless the checkpoint uses binary NOOP/COMMIT mode;
• a shared state value for PPO.

The network selects what planet to act from and target. The decoder computes how to aim using orbital intercept geometry rather than learning an angle bin. Action masks enforce only environment legality and the explicit discipline stored with the checkpoint. See orbit_wars_rl/model.py, orbit_wars_rl/features.py, and the feature audit for the exact contract.

Training system

• Algorithm: clipped PPO + GAE over synchronous rollouts from the vectorised Torch environment.
• Opponents: live self-play, historical checkpoints, and optional fixed heuristics; PFSP can bias the pool toward opponents near the policy's current frontier.
• Initialization/stability: runs may start from behaviour cloning, resume a PPO checkpoint, or anchor to a frozen best policy with KL regularisation and a promotion gate.
• Action spaces: legacy absolute/fractional ship bins, semantic intent sizing, and binary NOOP/COMMIT are checkpoint-compatible experiment variants—not interchangeable eval flags.
• Observability: checkpoints persist model/action configuration; training exposes passive mechanism metrics and optional Weights & Biases logging.

The strongest post-competition run uses pure self-play, sparse terminal reward, projected-future features, and the binary action space with minimal hardcoded commit gates. That is a result of the experiments below, not a claim that every checkpoint in the repository shares that setup. The full experiment ledger is docs/training.md.

Evaluation contract

Local win rate is an opponent-specific diagnostic, not a leaderboard-score estimate. Small 16-game runs are used only to catch obvious regressions; decisions use a fixed 128-seed panel with the learned agent in both seats. The opponent ladder also has to move as the policy improves: Zach is a smoke test, Ajay became saturated, Yijie is the current mid-tier verdict, and Ender remains the top-tier wall.

Two-player and four-player results are kept separate. The learned policy was developed primarily in 2p; the competition submissions route 4p games to the ajay heuristic because 2p strength did not transfer reliably to free-for-all play. Final checks run through the official Kaggle environment, fail on any agent error, and report seat asymmetry alongside aggregate win rate.

Research progression

The work moved through three failure modes rather than one linear recipe:

Problem	Intervention	What survived testing
Opening paralysis and weak conversion	reward shaping, BC, target-conditioned heads	Per-target conditioning and correct geometry were necessary; shaping alone was not stable.
Spray/churn self-play equilibrium	masks, opponent pools, teacher anchors	Pools slowed drift, but projected-future state made bad launches learnably visible.
Binary policy plateau	richer sizing, counterfactual features, gate audits	Hardcoded commit gates—not binary actions themselves—were deleting valuable reinforcement and pincer moves.

Detailed chronology and failed arms are retained in docs/training.md, docs/writeup_lessons.md, and archive/docs/training-till-submission.md.

Results

Final pre-submission eval of the best exported agents (2026-06-24, 256-game panels on the GCP eval box, 0 draws across all panels). Full tables + raw logs are archived in archive/cleanup_2026-07/results/eval_results.md.

2p head-to-head win rate (256 games = 128 seeds × 2 seats):

Agent	vs Ajay (~1200 LB)	vs Debatreya (~1300 LB)
stgpr1 0.5M (spray)	57.4%	59.4%
presres1 1.5M (~1000 LB ref)	54.3%	53.1%
presres1 0.5M (decisive)	53.9%	51.6%

stgpr1's higher WR is consistent with its spray/churn style, which inflates head-to-head WR without reflecting cleaner positional play — a caveat for interpretation, not a cert problem.

4p FFA mixed-field (seat-rotated, win-rate = 1st-place share = the LB metric):

Agent	Win-rate	Mean place
ajay	39.1%	1.64
producer_v2 (prior 4p incumbent)	28.1%	1.73
stgpr1 (our 2p neural agent)	18.8%	1.81
deb	14.1%	1.88

Verdict held against both producer variants → 4p slot = ajay (a heuristic), since 2p strength does not predict 4p and our 2p-trained neural agent only places 3rd–4th there.

Cross-eval vs the held-out opponent set — both submitted agents against public heuristics and our own past-best checkpoints (full 256-game both-seats panels, same as the certs):

Opponent	presres1 (decisive)	stgpr1 (spray)
zach	99.6%	99.6%
hellburner	97.3%	98.0%
h1043 (lb1043)	98.4%	98.4%
h1166 (lb1166 peak)	89.5%	93.8%
pool_lb1084 / lb1138 / lb1152	95.7 / 85.5 / 88.3%	97.7 / 90.6 / 91.0%
past selves rev38 / rev53b / rev31 / rev32b	94 / 91 / 93 / 93%	97 / 91 / 95 / 97%

Both sweep the held-out set — every public heuristic (85–99%) and every prior self (91–97%) — clear absolute progress with no forgetting, seats near-symmetric (|asym| ≤ 5.5pp). stgpr1 edges ahead on most opponents, but that's the same spray/churn WR-inflation, not cleaner play. Full per-seat table + the historical corrpack3e panel + the N=6 diversity matrix: archive/cleanup_2026-07/results/eval_results.md.

Current state

The current experiment state and the evidence behind it are tracked in docs/training.md; competition-era decisions are preserved in archive/docs/training-till-submission.md.

• Two agents submitted (2026-06-24, both daily slots): presres1 0.5M "decisive" and stgpr1 0.5M "spray", each routing 2p → neural agent, 4p → ajay. See docs/submissions.md.
• Two bug fixes banked in code but not in a competition agent: phantom-neutral-production and path-obstruction. Every retrain (presfix1/pathobs1/stgpr2) drifted back into the spray Nash within ~2M steps, and applying the fixes at inference-only on a frozen agent measured net-negative (~−1.5 to −2pp) — both genuinely need a retrain to cash in.
• Key open finding: phantom-fix / --min-ship-bin / path-obstruction-veto are all the wrong lever class — they clean the opening, then self-play retrains the agent back into flooding + non-holding. The next move is structural (an opponent/curriculum that punishes spray and forces holding), steered by launch_rate → ~0.04 / peel↓ / hold↑ rather than by spray-inflated Ajay WR.

Post-submission progress (2026-07)

The competition ended; the project continued as a study of the top-100 writeups (docs/writeup_lessons.md), applying one lesson at a time. The first structural lesson — projected-future timeline features (per-planet ownership/garrison rolled 24 steps forward, the most universal ingredient across winner writeups) — plus a 10–20× step budget answered the open finding above:

• tl100m (2026-07-12): 100M steps from scratch, pure self-play, sparse ±1 reward, no shaping — 74.6% vs Ajay (best panel 77.7%), past the shaped lineage's 57.4%, and clean: launch discipline (launch_rate 0.09) was learned, not masked in. The spray Nash that every competition-era retrain fell back into simply doesn't form when bad launches are visible to the critic the step they happen. Run record: docs/training.md.

• binarygates100m_l4 (2026-07-18): 100M from scratch on a binary NOOP/COMMIT action space with the hand-tuned commit gates released (--binary-commit-gates minimal) — 98.0% vs Ajay and 14.1% vs Yijie (rank 13, 1640 Elo). Best result on both metrics, and ~2× the prior Yijie best (~7%). The finding: the binary lineage's earlier Yijie regression was the hardcoded commit gates, not the binary action space itself. The gates computed a verdict from features the model already saw, then deleted ~80% of the commit options it might have disagreed with — no multi-source pincers, no pre-emptive reinforcement. Releasing them recovered strong play while also lifting Ajay to the best ever. Yijie is now the verdict metric — Ajay saturates ~77–80% and is blind to what beats us. The top-10 wall is unchanged: still 0/32 vs Ender, wiped 100%. Follow-on arms (long-horizon economy features + gamma 0.999; a 1.44M capacity jump to the winners' ~1.2M shape) test whether the ~15% Yijie plateau is the ceiling of this config.

The previously reported 256/256 against each submitted endpoint was invalid. The local wrapper used __file__, which kaggle_environments does not define when it executes a path agent; the old evaluator then counted the errored opponent as a win. The identical aggregate statistics against two different neural payloads were the tell. The evaluator now fails closed on non-DONE agent status, and submitted-opponent checks use the archived standalone neural_agent.py payloads whose hashes match the final tarballs.

A corrected full-panel audit of the best current checkpoint (exact-marginal binary 40.108M, 80.5% vs Ajay) completed against the hash-validated final payloads:

Submitted opponent	Overall	Current agent as seat 0	Current agent as seat 1
presres1 (decisive)	179/256 (69.9%)	66.4%	73.4%
stgpr1 (spray)	164/256 (64.1%)	55.5%	72.7%

The current policy is genuinely stronger than both submitted agents, but it does not sweep them. The 17.2pp seat gap against stgpr1 is also a real limitation hidden by the invalid 256/256 result. Experiment history, passive metrics, and the audit trail live in docs/training.md.

Reproduce the core workflow

Python 3.11 is the safest local choice for the current Torch and Kaggle environment stack.

python3.11 -m venv .venv
source .venv/bin/activate
pip install -r orbit_wars_rl/requirements.txt
bash setup/install_orbit_wars.sh

Run a small CPU smoke train before launching a GPU experiment:

python orbit_wars_rl/train_torch.py \
  --device cpu \
  --num-envs 8 \
  --rollout-steps 8 \
  --total-steps 64 \
  --no-wandb

Evaluate a checkpoint against the fixed 128-seed panel from both seats (256 games total):

CUDA_VISIBLE_DEVICES="" python orbit_wars_rl/eval.py \
  --checkpoint <checkpoint.pt> \
  --opponent opponents/candidate_ajay_1200.py \
  --panel \
  --target-decode

Export the same policy contract for Kaggle:

python orbit_wars_rl/export_agent.py \
  --checkpoint <checkpoint.pt> \
  --output main.py \
  --target-decode

Evaluation and export infer feature widths and action-mode settings from the checkpoint. Do not override reinforcement gates or decoding semantics unless you are deliberately running an ablation. The evaluator fails closed when an agent errors; this matters because an earlier wrapper bug produced invalid 256/256 results by treating errored opponents as losses.

For full training launches, checkpoint sync, monitoring, and teardown, use docs/commands.md and the provider runbooks. Cloud commands are intentionally kept out of this README because they encode machine-specific paths and cost-control rules.

Repo navigation

Where	What
orbit_wars_rl/	All active RL code (training, env, model, eval, export)
opponents/	Eval + training opponents (Ajay, Zach, Debatreya, producers; orbit_lite/ dep)
archive/cleanup_2026-07/results/	Final competition eval panels + raw cert/FFA logs
final_submissions/	Hash-audited competition checkpoints and submission archives
seed_checkpoints/	Resume points uploaded to training instances
docs/commands.md	Copy-paste command reference (start here for ops)
docs/training.md	Post-competition experiment ledger and current conclusions
docs/feature_audit.md	Exact learned-vs-computed feature boundary
archive/docs/training-till-submission.md	Full run history + key config (through first submission)
docs/submissions.md	Submission log with Kaggle IDs and checkpoint paths
docs/GCP_RUNBOOK.md · docs/JARVIS_RUNBOOK.md	GPU instance launch / monitor / teardown
AGENTS.md	Repository map, current baselines, and operating constraints
gpu_run_artifacts/	Training scripts, watchers, synced checkpoints (gitignored)

---

Game Specification

Conquer planets rotating around a sun in continuous 2D space. A real-time strategy game for 2 or 4 players.

Overview

Players start with a single home planet and compete to control the map by sending fleets to capture neutral and enemy planets. The board is a 100x100 continuous space with a sun at the center. Planets orbit the sun, comets fly through on elliptical trajectories, and fleets travel in straight lines. The game lasts 500 turns. The player with the most total ships (on planets + in fleets) at the end wins.

Board Layout

• Board: 100x100 continuous space, origin at top-left.
• Sun: Centered at (50, 50) with radius 10. Fleets that cross the sun are destroyed.
• Symmetry: All planets and comets are placed with 4-fold mirror symmetry around the center: (x, y), (100-x, y), (x, 100-y), (100-x, 100-y). This ensures fairness regardless of starting position.

Planets

Each planet is represented as [id, owner, x, y, radius, ships, production].

• owner: Player ID (0-3), or -1 for neutral.
• radius: Determined by production: 1 + ln(production). Higher production planets are physically larger.
• production: Integer from 1 to 5. Each turn, an owned planet generates this many ships.
• ships: Current garrison. Starts between 5 and 99 (skewed toward lower values).

Planet Types

• Orbiting planets: Planets whose orbital_radius + planet_radius < 50 rotate around the sun at a constant angular velocity (0.025-0.05 radians/turn, randomized per game). Use initial_planets and angular_velocity from the observation to predict their positions.
• Static planets: Planets further from the center do not rotate.

The map contains 20-40 planets (5-10 symmetric groups of 4). At least 3 groups are guaranteed to be static, and at least one group is guaranteed to be orbiting.

Home Planets

One symmetric group is randomly chosen as the starting planets. In a 2-player game, players start on diagonally opposite planets (Q1 and Q4). In a 4-player game, each player gets one planet from the group. Home planets start with 10 ships.

Fleets

Each fleet is represented as [id, owner, x, y, angle, from_planet_id, ships].

• angle: Direction of travel in radians.
• ships: Number of ships in the fleet (does not change during travel).

Fleet Speed

Fleet speed scales with size on a logarithmic curve:

speed = 1.0 + (maxSpeed - 1.0) * (log(ships) / log(1000)) ^ 1.5

• 1 ship moves at 1.0 units/turn.
• Larger fleets move faster, approaching the maximum speed (default 6.0).
• A fleet of ~500 ships moves at ~5, and ~1000 ships reaches the max.

Fleet Movement

Fleets travel in a straight line at their computed speed each turn. A fleet is removed if it:

• Goes out of bounds (leaves the 100x100 playing field).
• Crosses the sun (path segment comes within the sun's radius).
• Collides with any planet (path segment comes within the planet's radius). This triggers combat.

Collision detection is continuous -- the entire path segment from old to new position is checked, not just the endpoint.

Fleet Launch

Each turn, your agent returns a list of moves: [from_planet_id, direction_angle, num_ships].

• You can only launch from planets you own.
• You cannot launch more ships than the planet currently has.
• The fleet spawns just outside the planet's radius in the given direction.
• You can issue multiple launches from the same or different planets in a single turn.

Comets

Comets are temporary extra-solar objects that fly through the board on highly elliptical orbits around the sun. They spawn in groups of 4 (one per quadrant) at steps 50, 150, 250, 350, and 450.

• Radius: 1.0 (fixed).
• Production: 1 ship/turn when owned.
• Starting ships: Random, skewed low (minimum of 4 rolls from 1-99). All 4 comets in a group share the same starting ship count.
• Speed: Configurable via cometSpeed (default 4.0 units/turn).
• Identification: Check comet_planet_ids in the observation to see which planet IDs are comets. Comets also appear in the planets array and follow all normal planet rules (capture, production, fleet launch, combat).

When a comet leaves the board, it is removed along with any ships garrisoned on it. Comets are removed before fleet launches each turn, so you cannot launch from a departing comet.

The comets observation field contains comet group data including paths (the full trajectory for each comet) and path_index (current position along the path), which can be used to predict future comet positions.

Turn Order

Each turn executes in this order:

1. Comet expiration: Remove comets that have left the board.
2. Comet spawning: Spawn new comet groups at designated steps.
3. Fleet launch: Process all player actions, creating new fleets.
4. Production: All owned planets (including comets) generate ships.
5. Fleet movement: Move all fleets along their headings. Check for out-of-bounds, sun collision, and planet collision. Fleets that hit planets are queued for combat.
6. Planet rotation & comet movement: Orbiting planets rotate, comets advance along their paths. Any fleet caught by a moving planet/comet is swept into combat with it.
7. Combat resolution: Resolve all queued planet combats.

Combat

When one or more fleets collide with a planet (either by flying into it or being swept by a moving planet), combat is resolved:

1. All arriving fleets are grouped by owner. Ships from the same owner are summed.
2. The largest attacking force fights the second largest. The difference in ships survives.
3. If there is a surviving attacker:
   • If the attacker is the same owner as the planet, the surviving ships are added to the garrison.
   • If the attacker is a different owner, the surviving ships fight the garrison. If the attackers exceed the garrison, the planet changes ownership and the garrison becomes the surplus.
4. If two attackers tie, all attacking ships are destroyed (no survivors).

Scoring and Termination

The game ends when:

• Step limit reached: 500 turns.
• Elimination: Only one player (or zero) remains with any planets or fleets.

Final score = total ships on owned planets + total ships in owned fleets. Highest score wins.

Observation Reference

Field	Type	Description
planets	[[id, owner, x, y, radius, ships, production], ...]	All planets including comets
fleets	[[id, owner, x, y, angle, from_planet_id, ships], ...]	All active fleets
player	int	Your player ID (0-3)
angular_velocity	float	Planet rotation speed (radians/turn)
initial_planets	[[id, owner, x, y, radius, ships, production], ...]	Planet positions at game start
comets	[{planet_ids, paths, path_index}, ...]	Active comet group data
comet_planet_ids	[int, ...]	Planet IDs that are comets
remainingOverageTime	float	Remaining overage time budget (seconds)

Action Format

Return a list of moves:

[[from_planet_id, direction_angle, num_ships], ...]

• from_planet_id: ID of a planet you own.
• direction_angle: Angle in radians (0 = right, pi/2 = down).
• num_ships: Integer number of ships to send.

Return an empty list [] to take no action.

Agent Convenience

The module exports named tuples for easier field access:

from kaggle_environments.envs.orbit_wars.orbit_wars import Planet, Fleet, CENTER, ROTATION_RADIUS_LIMIT

def agent(obs):
    planets = [Planet(*p) for p in obs.get("planets", [])]
    fleets = [Fleet(*f) for f in obs.get("fleets", [])]
    player = obs.get("player", 0)

    for p in planets:
        print(p.id, p.owner, p.x, p.y, p.radius, p.ships, p.production)

    return []  # list of [from_planet_id, angle, num_ships]

Configuration

Parameter	Default	Description
episodeSteps	500	Maximum number of turns
actTimeout	1	Seconds per turn
shipSpeed	6.0	Maximum fleet speed
sunRadius	10.0	Radius of the sun
boardSize	100.0	Board dimensions
cometSpeed	4.0	Comet speed (units/turn)