AIAny
Icon for item

Embodied.cpp: A Portable Inference Runtime of Embodied AI Models on Heterogeneous Robots

Provides a portable C++ inference runtime to deploy embodied AI models (vision–language–action and world–action) on heterogeneous robot hardware, enabling latency-first batch-1 closed-loop control. Key features include modular multi-rate layers, fused low-latency inference, and extensible head/IO plugins.

Introduction

The deployment gap for embodied agents is practical: models combine perception, multi-rate prediction, and action control but existing runtimes assume request–response LLM serving and fail to meet closed-loop latency and interface needs. This work reframes deployment as a runtime contract problem — the system must support components running at different refresh rates, deterministic low-latency batch-1 inference on heterogeneous edge hardware, and extensible multimodal I/O beyond token streams.

Key Findings
  • Shared execution path and five-layer architecture: input adapters, sequence builders, backbone execution, head plugins, and deployment adapters — this modularization turns model-specific glue code into pluggable runtime pieces. So what: new embodied variants can be adopted without rewriting the whole stack.
  • Latency-first fused inference and multi-rate scheduling target real-time control: evaluated VLA deployments achieved closed-loop success (HY-VLA 100.0%, pi0.5 91.0%) while respecting per-step timing constraints. So what: the runtime preserves task behavior while optimizing for low-latency control rather than throughput.
  • Quantized C++ blocks (GGUF Q4_K) substantially reduce memory for WAM components: a LingBot-VA Transformer block dropped resident weight memory from 312.2 MiB to 88.1 MiB with MAE < 3.3×10^-2 and cosine similarity > 0.9997. So what: constrained edge devices can host larger embodied modules with little functional drift.
Who it's for and trade-offs

Great fit if you need to run multimodal embodied models (VLA/WAM) on robots or edge GPUs/NPUs and care about predictable, low-latency closed-loop control and modular extensibility. Look elsewhere if your primary goal is high-throughput batched serving for text-only LLM workloads or you require full end-to-end support for models not yet integrated (the repo documents supported families but broader model support is evolving). The runtime reduces memory and adapts to heterogeneous backends, but actual latency and peak memory still depend on backbone size, input complexity (e.g., multi-view video), and action-chunking choices.

Information

  • Websitearxiv.org
  • OrganizationsSoutheast University, Nanjing University, Microsoft Research, Institute for AI Industry Research (AIR), Tsinghua University
  • AuthorsLing Xu, Chuyu Han, Borui Li, Hao Wu, Shiqi Jiang, Ting Cao, Chuanyou Li, Sheng Zhong, Shuai Wang
  • Published date2026/07/02

More Items

Studies scaling language models with parallel Transformer branches and introduces Joint-KV to share key-value caches across branches. Experiments report improved modeling quality at comparable training compute and faster long-context decoding.

Evaluates how well models and coding agents optimize operators for real Ascend NPU workloads, accounting for both speedups and token cost. Test cases preserve tensor shapes and data types from real model execution contexts.

Hugging Face

Provides first-person video of skilled manual work with synchronized motion-sensor data for robot learning and dexterous manipulation research. The first release spans 10,124 hours from 12,227 contributors, with task-level annotations.