Reinforcement Learning Algorithm Comparison

This repository contains the final code for comparing several reinforcement learning algorithms on Gymnasium control environments. The implemented algorithms include DQN, PPO, SAC, and TD3. The project can run baseline experiments, perform one-parameter hyperparameter sweeps, generate plots, and save numerical results for later analysis.

Project Structure

RL_Final_Code/
+-- algorithms/
|   +-- dqn.py
|   +-- ppo_cartpole.py
|   +-- ppo_pendulum.py
|   +-- sac_2018.py
|   +-- td3.py
+-- compare.py
+-- experiment_config.py
+-- main.py
+-- requirements.txt
+-- run.ipynb
+-- utils.py
+-- results/

Implemented Algorithms

• DQN: for discrete-action environments.
• PPO: supports both discrete and continuous action spaces through separate implementations.
• SAC: for continuous-action environments.
• TD3: for continuous-action environments.

The main script automatically skips incompatible algorithm-environment pairs. For example, DQN is skipped on continuous-action environments, while SAC and TD3 are skipped on discrete-action environments.

Supported Environments

The default configuration uses:

• CartPole-v1
• Acrobot-v1
• Pendulum-v1

Installation

Create and activate a Python environment, then install the required packages:

pip install -r requirements.txt

The project was written for Gymnasium, PyTorch, NumPy, pandas, matplotlib, and tqdm. The exact package versions are listed in requirements.txt.

Configuration

All main experiment settings are defined in experiment_config.py.

Important options include:

• ALGORITHMS_TO_RUN: algorithms selected for execution.
• ENV_NAMES_TO_RUN: environments selected for execution.
• RUN_TUNING: whether to run hyperparameter tuning after the baseline.
• TOTAL_TIMESTEPS: number of environment interaction steps per seed.
• SEEDS: random seeds used for repeated runs.
• Algorithm-specific hyperparameters and tuning value lists.

To run a shorter test, reduce TOTAL_TIMESTEPS, use fewer seeds, or set RUN_TUNING = False.

Running Experiments

From the project root directory, run:

python main.py

Alternatively, open run.ipynb and execute the notebook cells. The notebook is mainly a convenient wrapper for running the same experiment pipeline and inspecting results.

Outputs

Experiment outputs are saved under:

results/<environment>/<algorithm>/

Each algorithm-environment folder may contain:

• summary.csv: summary statistics for baseline and tuning runs.
• episode_returns.csv: per-seed episode return curves.
• efficiency_metrics.csv: timing and sample-efficiency related metrics.
• all_returns.npz: saved return arrays.
• figures/: baseline, tuning, and tuned-vs-baseline plots.

Trained model weights are saved under:

results/models/

Experiment Pipeline

The main workflow in main.py is:

1. Read the selected environments and algorithms from experiment_config.py.
2. Detect whether each environment has a discrete or continuous action space.
3. Skip incompatible algorithm-environment combinations.
4. Run baseline training across all selected seeds.
5. Plot and save baseline learning curves.
6. If enabled, run one-parameter hyperparameter tuning.
7. Plot tuning curves and compare the best tuned configuration with the baseline.
8. Save numerical summaries and per-episode results.

Reproducibility

The code sets random seeds for Python, NumPy, and PyTorch through utils.set_seed. Results may still vary slightly across hardware, operating systems, and PyTorch versions, especially when GPU acceleration is used.

Notes

• Running the full default configuration can take a long time because it includes multiple environments, multiple algorithms, three seeds, and several hyperparameter sweeps.
• The results/ directory already contains generated experiment outputs and trained model files.
• The __pycache__/ and .ipynb_checkpoints/ folders are generated automatically and are not required to understand or run the code.