Most action tokenizers evaluate fidelity with pointwise reconstruction metrics (e.g., MSE), but such measures miss whether relative action adjustments across contexts are preserved after compression. This paper argues that preserving relational structure between actions—what changes are near vs. far in physical space—is critical for downstream autoregressive vision-language-action (VLA) policies. The core insight is that tokenizers should be judged not only by per-step error but by whether decoded actions maintain local physical distance rankings from the originals.
Key Findings
- Physical Rank Consistency (PRC): a new metric that measures how well tokenization preserves local physical distance orderings after reconstruction; complements pointwise metrics by capturing relational fidelity.
- ActionPiece tokenizer: jointly supervises encoder representations and quantization so that encoder distances, quantized feature distances, and codeword assignment distributions respect near-far orderings. This explicitly enforces preservation of local action relationships during discretization.
- Empirical gains: under the same Qwen3-VL-4B policy training setup, ActionPiece attains 94.8% on LIBERO, 68.8% on unseen LIBERO-Plus, 71.9% on SimplerEnv, and 51.5% across VLA-Arena L0–L2, with ablations showing PRC and quantization regularization jointly improve policy success.
- Practical behavior: produces discrete action tokens compatible with standard autoregressive policy learning and execution via a frozen decoder, avoiding undefined token decodings.
Who it's for and trade-offs
Great fit if you train or evaluate autoregressive VLA policies and care about preserving fine-grained, context-dependent action adjustments rather than only minimizing reconstruction error. The approach is especially relevant for embodied-robotics research where relational fidelity among nearby actions affects policy outcomes. Look elsewhere if your downstream stack tolerates arbitrary per-step reconstruction noise (e.g., diffusion-based continuous controllers) or if you need extremely simple, per-dimension binning for tight real-time constraints—ActionPiece adds representational and quantization supervision that increases tokenizer design complexity.
Where it fits
ActionPiece sits between purely reconstruction-driven quantizers and heuristic discretization schemes: it keeps the decoder frozen for execution (total decodability) while explicitly supervising relational structure, making it suitable for research and development of autoregressive VLA policies that require both compact discrete tokens and preserved physical action relationships.