README.md

Crux.jl

Deep RL library with concise implementations of popular algorithms. Implemented using Flux.jl and fits into the POMDPs.jl interface.

Supports CPU and GPU computation and implements deep reinforcement learning, imitation learning, batch RL, adversarial RL, and continual learning algorithms. See the documentation for more details.

Reinforcement Learning

• Deep Q-Learning (DQN)
  • Prioritized Experience Replay
• Soft Q-Learning
• REINFORCE
• Proximal Policy Optimization (PPO)
• Lagrange-Constrained PPO
• Advantage Actor Critic (A2C)
• Deep Deterministic Policy Gradient (DDPG)
• Twin Delayed DDPG (TD3)
• Soft Actor Critic (SAC)

Imitation Learning

• Behavioral Cloning
• Generative Adversarial Imitation Learning (GAIL) w/ On-Policy and Off-Policy Versions
• Adversarial Value Moment Imitation Learning (AdVIL)
• Adversarial Reward-moment Imitation Learning (AdRIL)
• Soft Q Imitation Learning (SQIL)
• Adversarial Soft Advantage Fitting (ASAF)
• Inverse Q-Learning (IQLearn)

Batch RL

• Batch Soft Actor Critic (BatchSAC)
• Conservative Q-Learning (CQL)

Adversarial RL

• Robust Adversarial RL (RARL)

Continual Learning

• Experience Replay

Installation

To install the package, run:

] add Crux

Usage

Basic Example (Pure Julia)

A minimal example using DQN to solve a GridWorld problem:

using Crux

mdp = SimpleGridWorld()
S = state_space(mdp)

A() = DiscreteNetwork(Chain(Dense(2, 8, relu), Dense(8, 4)), actions(mdp))

solver = DQN(π=A(), S=S, N=100_000)
policy = solve(solver, mdp)

See the documentation for more examples and details.

Gym Environments

For OpenAI Gym environments like CartPole, install POMDPGym.jl:

] add https://github.com/ancorso/POMDPGym.jl

Note: POMDPGym requires Python with Gymnasium installed (pip install gymnasium).

using Crux, POMDPGym

mdp = GymPOMDP(:CartPole)
as = actions(mdp)
S = state_space(mdp)

A() = DiscreteNetwork(Chain(Dense(dim(S)..., 64, relu), Dense(64, length(as))), as)

solver = REINFORCE(S=S, π=A())
policy = solve(solver, mdp)

See the installation documentation for more details on how to install POMDPGym for more environment.