🐟 FishRL

FishRL is a collection of popular deep reinforcement learning algorithms implemented in PyTorch, designed for robotics and embodied AI research. It supports both state-based and visual (RGB) observations, and provides clean, reference implementations of methods ranging from PPO and SAC to more advanced approaches such as Dreamer v3.

In addition to algorithm implementations, FishRL focuses on the following practical aspects:

• Unified environment interface. FishRL supports multiple robotics and embodied AI environments through a unified interface. This interface can be reused to quickly test new algorithms across different simulation environments with minimal additional engineering for research purposes.

• Efficient benchmarking. For large-scale benchmarking, FishRL implements optimized training pipelines and common acceleration techniques. Inspired by LeanRL, FishRL typically achieves 3–10× speedup compared to naïve PyTorch implementations, and is competitive with JAX-based implementations in terms of training throughput.

• Single-file, readable implementations. Following the philosophy of CleanRL, FishRL keeps each algorithm in a single, self-contained file whenever possible, while sharing common RL utilities. This design emphasizes readability and ease of modification, making it convenient to understand, debug, and extend the algorithms.

Installation

Recommended Setup

conda create -n fishrl python=3.11 -y && conda activate fishrl
uv pip install -e ".[dmc,benchmark]"

The benchmark option specifies the packages version for the reproducibility of benchmarking results.

Advanced Setup

Gymnasium-Robotics environment

uv pip install -e ".[gymnasium_robotics]"

ManiSkill environment

uv pip install -e ".[maniskill]"

Humanoid-bench environment

cd third_party && git clone --depth 1 https://github.com/Fisher-Wang/humanoid-bench && cd ..
uv pip install -e ".[humanoid_bench]" -e third_party/humanoid-bench

IsaacLab 2.3.1 environment

cd third_party && git clone --depth 1 --branch v2.3.1 https://github.com/isaac-sim/IsaacLab.git IsaacLab231 && cd ..
sed -i 's/gymnasium==1\.2\.0/gymnasium/g' third_party/IsaacLab231/source/isaaclab/setup.py
uv pip install -e ".[isaaclab]" -e "third_party/IsaacLab231/source/isaaclab" -e "third_party/IsaacLab231/source/isaaclab_tasks"

IsaacGymEnvs environment

conda create -n fishrl_gym python=3.8 -y && conda activate fishrl_gym
cd third_party
wget https://developer.nvidia.com/isaac-gym-preview-4 \
    && tar -xf isaac-gym-preview-4 \
    && rm isaac-gym-preview-4
find isaacgym/python -type f -name "*.py" -exec sed -i 's/np\.float/np.float32/g' {} +
uv pip install isaacgym/python
git clone --depth 1 https://github.com/isaac-sim/IsaacGymEnvs && cd IsaacGymEnvs
uv pip install -e .
cd ../..
uv pip install networkx==2.1

Get Started

To run the Dreamerv3 agent on the DMControl Walker-walk task with rgb observation, you can use the following command:

python dm3.py --env-id=dmc/walker-walk-v0 --obs-mode=rgb

For more examples, please refer to script.sh.

Algorithms Implemented

Algorithm	File	Recommended Acceleration Option
PPO	ppo.py	TODO
DDPG	ddpg.py	--compile --cudagraph
SAC	sac.py	--compile --cudagraph
DrQ v2	drqv2.py	--compile --cudagraph
Dreamer v1	dm1.py	TODO
Dreamer v3	dm3.py	--compile --amp

Acknowledgments

FishRL is inspired by CleanRL and SheepRL.

Its Dreamer v1 and Dreamer v3 implementation has referred to NaturalDreamer, SimpleDreamer, SheepRL, dreamerv3-torch, and dreamerv3.

Its acceleration techniques have referred to LeanRL.

FishRL's previous name before the public release was RVRL.