RBE 577 Project 3: Reinforcement Learning Agents for Lunar Lander and KUKA

Author: Filippo Marcantoni
Course: RBE 577 - Machine Learning for Robotics
Institution: Worcester Polytechnic Institute

---

Overview

This repository contains two main reinforcement learning implementations for Project 3:

• A2C/: A single-process implementation of REINFORCE and Advantage Actor-Critic (A2C) on the OpenAI Gym LunarLander-v2 environment.
• A3C/: An asynchronous Advantage Actor-Critic (A3C) implementation for a PyBullet KUKA pick-and-place task.

The code includes training, evaluation, checkpoint saving, plotting, and video export for the agent policies.

The project report is available as RBE577_ML_Project3_Report.pdf.

---

A2C Subproject

Overview

The A2C/ folder implements two policy-gradient methods for the continuous control benchmark LunarLander-v2:

• REINFORCE: Monte Carlo policy gradient with reward-to-go and baseline normalization.
• A2C: Actor-Critic using a learned value function for advantage estimation.

The implementation supports training from configuration JSON files, automatic checkpointing, and reward-history plotting.

Key Features

• Shared train.py entry point for both reinforce and a2c algorithms.
• Actor and critic network modules implemented in actor.py and critic.py.
• Observation normalization and environment compatibility between Gym/Gymnasium.
• Evaluation script with deterministic policy execution and optional video export.

Code Structure

A2C/
├── actor.py
├── critic.py
├── compute_objectives.py
├── config_a2c.json
├── config_reinforce.json
├── config.json
├── eval.py
├── train.py
├── utils.py
├── checkpoints/
│   ├── best_actor_critic.pt
│   └── best_reinforce_actor.pt
├── plots/
└── videos/

Configuration

Training and evaluation are configured using JSON files:

• A2C/config_reinforce.json — REINFORCE settings
• A2C/config_a2c.json — A2C settings

Common options include:

• env_id: environment ID (LunarLander-v2)
• algorithm: reinforce or a2c
• actor_lr, critic_lr
• gamma
• hidden_dim
• num_episodes
• max_ep_steps
• checkpoint_path
• plot_filename

How to Run

1. Train REINFORCE

cd A2C
python train.py --config config_reinforce.json

2. Train A2C

cd A2C
python train.py --config config_a2c.json

3. Evaluate a checkpoint

cd A2C
python eval.py --config config_a2c.json --checkpoint checkpoints/best_actor_critic.pt --episodes 500

4. Generate evaluation videos

cd A2C
python eval.py --config config_reinforce.json --checkpoint checkpoints/best_reinforce_actor.pt --episodes 1 --video reinforce_video1.mp4 --video-seed 101
python eval.py --config config_reinforce.json --checkpoint checkpoints/best_reinforce_actor.pt --episodes 1 --video reinforce_video2.mp4 --video-seed 102
python eval.py --config config_reinforce.json --checkpoint checkpoints/best_reinforce_actor.pt --episodes 1 --video reinforce_video3.mp4 --video-seed 103

Video output is written to A2C/videos/.

---

A3C Subproject

Overview

The A3C/ folder implements the asynchronous advantage actor-critic algorithm for a PyBullet KUKA pick-and-place environment.

The implementation launches multiple worker processes via PyTorch multiprocessing and trains a shared actor-critic network on image-based observations.

Key Features

• Asynchronous worker processes with shared global parameters.
• CNN-based image preprocessing for the KUKA PyBullet environment.
• Checkpoint saving to A3C/models/.
• Deterministic evaluation and optional success/failure video recording.

Code Structure

A3C/
├── main.py
├── eval.py
├── plot_a3c.py
├── config/
│   ├── a3c.yaml
│   └── common.yaml
├── helpers/
│   ├── __init__.py
│   ├── config.py
│   ├── logger.py
│   ├── metrics.py
│   └── utils.py
├── lib/
│   └── a3c/
│       ├── __init__.py
│       ├── agent.py
│       ├── model.py
│       ├── objectives.py
│       ├── shared_optim.py
│       └── train.py
├── logs/
├── models/
│   ├── a3c_kuka_model_ep1000.pth
│   ├── a3c_kuka_model_ep10000.pth
│   ├── ...
│   └── a3c_kuka_model_final.pth
├── plots/
└── videos/

Configuration

The A3C configuration is split into:

• A3C/config/common.yaml — environment, camera, and device settings
• A3C/config/a3c.yaml — A3C training hyperparameters and model architecture settings

Important parameters include:

• hyperparameters.num_workers
• hyperparameters.max_episodes
• hyperparameters.lr
• network.shared_layers
• network.actor_hidden_layers
• network.critic_hidden_layers
• env.name
• device

How to Run

1. Train A3C

cd A3C
python main.py

2. Evaluate a trained A3C checkpoint

cd A3C
python eval.py --checkpoint models/a3c_kuka_model_final.pth --episodes 100 --success-threshold 0.5

3. Save evaluation video

cd A3C
python eval.py --checkpoint models/a3c_kuka_model_final.pth --episodes 100 --success-threshold 0.5 --video a3c_eval.mp4 --video-successes 3 --video-failures 3

---

Environment Setup

This project is designed to run in a Python environment with PyTorch, Gym, PyBullet, and standard scientific packages.

A recommended Conda setup is:

conda create -n rl-agent python=3.8 -y
conda activate rl-agent
python -m pip install "pip==23.0.1" "setuptools==65.5.0" "wheel==0.38.4"
conda install -y numpy matplotlib ipython ffmpeg
pip install "imageio[ffmpeg]"
pip install torch==2.4.1+cu121 torchvision==0.19.1+cu121 torchaudio==2.4.1+cu121 \
  --index-url https://download.pytorch.org/whl/cu121
pip install gym==0.21.0 Box2D pygame "pyglet<2"
pip install pybullet==3.2.6 pyyaml

If CUDA is unavailable, use the CPU fallback by setting device: "cpu" in A3C/config/common.yaml or passing --device cpu to A3C/eval.py.

---

Notes

• A2C/train.py saves checkpoints and reward plots automatically in A2C/checkpoints/ and A2C/plots/.
• A3C/main.py saves the final shared policy in A3C/models/a3c_kuka_model_final.pth.
• A3C/eval.py reports success rate, average reward, and average episode length for deterministic evaluation runs.
• The final project report is in RBE577_ML_Project3_Report.pdf.