AIAny
Icon for item

ProtoMotions3

GPU‑accelerated framework for training physically simulated humanoid characters and robots using reinforcement learning and motion imitation. Provides a modular multi‑backend simulator stack, large‑scale multi‑GPU training recipes, built‑in motion retargeting and an ONNX deployment pathway to real robots.

Introduction

Physically simulated humanoid learning is costly to prototype: you need scalable simulators, motion retargeting, multi‑GPU training pipelines and a clean path to deploy to hardware. ProtoMotions3 addresses that gap by packaging a GPU‑first, modular framework that lets researchers scale motion imitation and RL for digital humans and humanoid robots end-to-end.

What Sets It Apart
  • GPU-native multi-backend support: run at scale across NVIDIA Newton (beta), IsaacGym/IsaacLab, Genesis and MuJoCo (CPU) backends, making sim2sim and backend swaps straightforward.
  • Large-scale motion learning recipes: example claim — train SMPL character on the full AMASS dataset (~40+ hours) in ~12 hours on 4 A100s; demonstrated runs with 24 A100s and 13K motions per GPU for extreme scaling. This means you can move from single-motion tests to dataset-scale imitation without reengineering the pipeline.
  • Integrated retargeting and deployment: PyRoki-based retargeting tools and an ONNX export flow let you retarget entire motion datasets and export policies (observation computation baked in) for zero‑shot transfer to hardware like Unitree G1.
  • Modular, research-friendly primitives: observation/reward/control components are pure tensor kernels or small classes, enabling rapid composition of new tasks, environments and custom simulators with minimal glue.
  • Built-in algorithms and generative policies: implementations include MaskedMimic, AMP, ASE and PPO; supports generative policy concepts (discrete latent priors, PEFT adapters) for higher-level behavior selection.
Who It's For and Tradeoffs

Great fit if you need to train physics‑based humanoid control at dataset scale, want a unified pipeline from retargeting to sim training to on‑robot deployment, or need to compare policies across multiple physics engines. It favors teams with GPU resources (multi‑A100 class machines) and those comfortable integrating simulation assets and robot specifications.

Look elsewhere if you only need lightweight motion imitation on a single CPU/GPU, require a minimal runtime with no external simulator dependencies, or cannot accept the engineering overhead of setting up multi‑GPU distributed runs and retargeting pipelines.

Where It Fits

ProtoMotions3 sits between research‑oriented physics simulators and applied robot deployment stacks: it reduces the plumbing for large‑scale motion learning while retaining low‑level control of observations, rewards and simulator backends, making it a practical bridge from academic motion datasets to real robot controllers.

Information

  • Websitegithub.com
  • OrganizationsNVIDIA Research (NVlabs)
  • AuthorsChen Tessler, Yifeng Jiang, Xue Bin Peng, Erwin Coumans, Yi Shi, Haotian Zhang, Davis Rempe, Gal Chechik, Sanja Fidler
  • Published date2024/09/24

Categories

More Items

Hugging Face

Provides about 10,000 OpenUSD objects for robotics simulation, including articulated and static assets with physics properties and robot-interaction annotations.

Hugging Face

Pairs metric-scale 3D scans of real environments with egocentric task recordings, estimated body and hand motion, and hierarchical annotations for physical AI research. The first release contains 126 scenes; the dataset is licensed CC BY-NC 4.0.

Transfers hand and scene dynamics from an action-conditioned spatial world model into a dexterous manipulation policy during training. At deployment, the policy uses RGB, language, and robot state without depth input or explicit future prediction.