AI Video Papers2026
Generates enterable omnimodal world-model rollouts that follow continuous 6-DoF camera control while jointly producing 720p video, environmental sound, music and speech. Uses dataset-level motion calibration, a specialized data engine, progressive training and autoregressive post-training to support long-horizon first- and third-person interaction.
AI Audio2026
Generates low-latency, instruction-driven English and Chinese speech for voice cloning, voice design, and directed performances; supports real-time streaming, reference-free voice creation, and reference-guided cloning. Open-weight PyTorch model released under a research/non-commercial license with GPU recommendations.
AI Dataset2026
Provides 22.7 hours of read Amharic speech (7,405 clips, 320 speakers) for ASR, collected via a crowdsourced Telegram bot and peer-validated; speaker- and prompt-disjoint train/validation/test splits, 16 kHz audio under CC BY 4.0.
Provides a streaming dual-brain memory for real-time speech agents: an informational left brain for factual retrieval and an affective right brain for persona/emotion, achieving high top-5 accuracy while keeping retrieval latency within VAD budgets (~134 ms).
Generates synchronized spoken dialogue and explicit full-body co-speech motion (facial expressions, hands, upper- and lower-body) end-to-end from the same hidden states, replacing the speech-then-motion cascade. Trains with a scalable pseudo-labeling pipeline (422,856 ranked pairs) and supports real-time inference (RTF 0.78) while matching teacher motion metrics within ~2%.
AI Dataset2026
Provides 315,000 pairwise human-preference votes comparing 15 English TTS models over 300 operational prompts, with 4,500 high‑quality audio renders and structured vote/pair/prompt records for training or evaluating preference/reward models. Metadata under CC-BY-4.0; audio use governed by model providers' terms.
AI Dataset2026
Provides a dual-channel, channel-separated sample (8.9 hours) and access path to a 1,000‑hour English conversational corpus for commercial and research use. Delivers 48 kHz per-speaker audio, word-level machine transcripts, and per-speaker metadata designed for full‑duplex/turn-taking and ASR/ TTS research.
AI Agent Papers2026
Unifies continuous streaming perception, full‑duplex real‑time interaction, and agentic tool use across video, audio, and text. Proposes a Cerebellum–Brain split—streaming Thinker‑Talker cerebellum for low‑latency interaction plus a Brain for complex reasoning—and is released with models, code, and data.
AI Audio2026
Provides low-latency, true-streaming automatic speech recognition that emits append-only committed transcripts to avoid partial-result rollbacks. Supports configurable decoding chunks (80 ms–2 s), optimized for Chinese and English, and offers vLLM and transformers backends for real-time deployment.
Haojun Zhang, Yi Zou +8Affiliation: Xini Ding, Hao Li, Shuchang Zhou, Xianming Liu, Shiyu Huang‡, Affiliation: XPeng Inc.
Progressively prunes and distills audio encoders for speech LLMs to cut inference cost while preserving decoder-facing embeddings, using behavioral probes, representation alignment, cross-scale distillation and LoRA finetuning; reports reduced macro-error on Chinese–English benchmarks.
Provides a realtime audio–language foundation model for fully duplex spoken interaction that listens, interprets acoustic cues, reasons, and speaks in parallel. Key features include Deep Perception for acoustic understanding, Seamless Duplex for synchronized streams, and Think‑While‑Speaking for parallel private reasoning; reports strong benchmark results.
AI Audio2026
Encodes real audio into YuE2 semantic tokens and ships a matching NAR-branch LoRA so the YuE2 decoder renders realistic latents—enables tokenizing recordings, training artist LoRAs, and generating songs or covers. Includes an 8-layer tokenizer head and a rank-32 decoder LoRA; requires the YuE2 base models and GPU resources.