Generates synchronized spoken dialogue and explicit full-body co-speech motion (facial expressions, hands, upper- and lower-body) end-to-end from the same hidden states, replacing the speech-then-motion cascade. Trains with a scalable pseudo-labeling pipeline (422,856 ranked pairs) and supports real-time inference (RTF 0.78) while matching teacher motion metrics within ~2%.
Haojun Zhang, Yi Zou +8Affiliation: Xini Ding, Hao Li, Shuchang Zhou, Xianming Liu, Shiyu Huang‡, Affiliation: XPeng Inc.
Progressively prunes and distills audio encoders for speech LLMs to cut inference cost while preserving decoder-facing embeddings, using behavioral probes, representation alignment, cross-scale distillation and LoRA finetuning; reports reduced macro-error on Chinese–English benchmarks.