Welcome to the documentation of my Reinforcement Learning library!#

This is a personal project to learn about reinforcement learning! As such, it is not meant to be a comprehensive library, but rather a learning tool for myself.

Introduction#

One can find the repository here.

Package Installation#

To install the package, clone the repository and install using pip:

git clone https://github.com/axeldinh/rlib.git
cd rlib
pip install -e .

To perform the tests, run the following commands:

pip install pytest
cd rlib
pytest

However, tests are automatically performed on push using Github Actions.

Documentation Generation#

To generate the documentation, run the following commands:

cd docs
make html

Note that the documentation is generated using Sphinx. The documentation can be found in the docs/build/html directory.

Usage#

The package can then be simply imported using:

import rlib

The code can be decomposed into two main parts Learning and Agents.

Learning contains the different algorithms which have been implemented while Agents contains the different agents which can be used to interact with the environment.

To use them on a gymnasium environment, the user must choose an algorithm along with the different parameters the agents should take.

For example, to use the DeepQLearning algorithm on the gymnasium CartPole-v1 environment, as the action space is discrete, the agent should be a MLP. This can be done using the following code:

from rlib.learning import EvolutionStrategy

env_kwargs = {'id': 'CartPole-v1'}
agent_kwargs = {'hidden_sizes': [64, 64]}

model = EvolutionStrategy(env_kwargs=env_kwargs, agent_kwargs=agent_kwargs, save_folder="EvStrat_CartPole-v1")

Note that only dictionaries are passed to the algorithms and agents. This allows the user to pass any parameters to the environment and the agent, while allowing a full saving of the model. Finally, the model can be trained using the following code:

model.train()

And the plots and videos generated by the model can be saved using:

model.save_plots()
model.save_videos()

Finally, a Flappy Bird environment has been implemented using PyGame. This environment can be used using the following code:

from rlib.learning import EvolutionStrategy
import rlib.envs

env_kwargs = {'id': 'FlappyBird-v0'}
agent_kwargs = {'hidden_sizes': [64, 64]}

model = EvolutionStrategy(env_kwargs=env_kwargs, agent_kwargs=agent_kwargs, save_folder="EvStrat_FlappyBird-v0", env=FlappyBird)

For more detail on the environment, see Environments

Examples#

Here is a showcase of the algorithms used on different environments:

Q-Learning on MountainCar-v0	Deep Q-Learning on LunarLander-v2	Evolution Strategy on FlappyBird-v0	Deep Deterministic Policy Gradient on HalfCheetah-v4	Proximal Policy Optimization on BipedalWalker-v3

Limitations#

Many things are still missing from the package, but the main ones are:

GPU support, this could be easily implemented by sending the tensors to the GPU using torch.cuda() when needed. This would allow the package to be used on more complex environments, and with more complex agents, such CNNs.
CNN support, this could be implemented by adding a CNN class in rlib.agents and adding an automatic detection of the type of agent in get_agent.