Most RL libraries optimize for code reuse — shared base classes, registries, layers of abstraction — so understanding one algorithm means jumping across a dozen files. This project makes the opposite bet: every algorithm lives in one self-contained script you can read end to end, duplication accepted as the price of clarity. The PPO Atari file is ~340 lines, yet still reproduces published numbers.
What Sets It Apart
- Debugging is reading, not tracing. The training loop, network, and hyperparameters sit in a single file, so you follow logic top to bottom instead of chasing call stacks through inheritance chains.
- Reproducibility is first-class. Each variant is benchmarked against published results in the Open RL Benchmark, with tracked W&B runs and learning curves you can inspect rather than trust.
- Breadth without a framework. 9+ algorithms (PPO, DQN, C51, SAC, DDPG, TD3, PPG, RND, Qdagger) span classic control, Atari, MuJoCo, and procgen.
- Scales when needed. Built-in wandb/TensorBoard logging plus AWS Batch support to launch thousands of experiments.
Who It's For and Tradeoffs
Great fit if you are a researcher or student who needs to understand, modify, or fork one algorithm in isolation — the readability makes it a teaching staple (Hugging Face's Deep RL course builds on it) and a fast base for prototyping new ideas. Look elsewhere if you want a production framework with shared abstractions and plug-and-play components: the deliberate duplication means a bug fix or new feature has to be copied into every script it touches.
