Most reinforcement learning codebases force a trade-off: research frameworks expose every knob but bury you in boilerplate, while batteries-included libraries hide the algorithm so you can't actually modify it. The interesting move here is refusing that choice — the same 30+ algorithms are reachable through a one-call high-level interface or a fully procedural one where you wire up collector, buffer, and policy yourself.
What Sets It Apart
- Two APIs, one engine: prototype with a high-level builder, then drop to the procedural layer when you need to control the training loop — without switching libraries.
- Clean algorithm taxonomy: an explicit split between
AlgorithmandPolicy, and between on-policy, off-policy, and offline families, makes it readable enough to fork rather than just consume. - Reproducibility as a feature: every algorithm ships with a full training test, and MuJoCo results are benchmarked to match published baselines — so a regression shows up in CI, not in your paper.
- Built for real experiments: synchronous/asynchronous vectorized environments, EnvPool acceleration, multi-GPU training, RNN-style training for POMDPs, and multi-agent support.
Great Fit / Look Elsewhere
A great fit if you research or extend RL methods in pure PyTorch and want type-hinted, tested baselines you can read and modify — DQN variants, PPO/TRPO/SAC/TD3, and offline methods like CQL and BCQ are all in one place. Look elsewhere if you want single-file copy-paste implementations (CleanRL), large-scale distributed RL across a cluster (Ray RLlib), or JAX-level training throughput; Tianshou optimizes for clarity and breadth on a single machine, not horizontal scale.
