Most avatar systems split “what to say” from “how to move,” producing speech first and running a separate motion model over the finished audio. Motion‑Omni shows that co‑adapting the language, speech, and motion pathways recovers tight audio–motion alignment and removes the need for a costly two‑stage cascade. The paper provides a scalable pseudo‑labeling pipeline and a unified evaluation protocol for open‑ended spoken dialogue with full‑body motion.
Key Findings
- End-to-end joint generation: The model directly outputs speech plus explicit facial, hand, upper‑body and lower‑body motion from the hidden states that produce the audio, enabling single-pass, jointly optimized inference.
- Scalable supervision: A model‑agnostic pseudo‑labeling pipeline produced 422,856 quality‑ranked speech–motion pairs (≈1,402 hours), letting motion supervision scale without end‑to‑end paired recordings.
- Practical performance: Instantiated with a Qwen2.5‑7B‑Instruct backbone, Motion‑Omni‑Q7 runs at RTF 0.78 (≈5.4× faster than the teacher cascade), stays within ~2% of the same‑audio teacher on reference‑free motion metrics, and achieves a 2.62% WER for its speech output.
- Unified evaluation: The authors release SwDA‑500 and present the first public evaluation protocol for stochastic open‑ended full‑body spoken dialogue that unifies rendering, automatic metrics, human evaluation, and latency measurement.
Who it's for & trade-offs
Great fit if you need a single, low‑latency system that must produce tightly synchronized audio and full‑body motion for interactive avatars, virtual agents, or embodied conversational interfaces. Look elsewhere if you require a modular pipeline for independent reuse of speech and motion models, or if you cannot provide a motion teacher or the compute to co‑train and fine‑tune multiple pathways — Motion‑Omni’s gains rely on large pseudo‑labeled supervision and joint optimization, and its motion quality depends on the chosen motion teacher.