Generates multi‑speaker speech and environmental audio from textual instructions or a reference clip, supporting zero‑shot voice cloning and detailed scene/specification control. Combines a cleaned, captioned dataset with a VAE-based multimodal generator, reward-conditioned quality control, and staged training to improve expressiveness and multi-audio modeling.
AI Audio2026
Generates and edits speech from natural-language instructions plus optional reference audio, supporting zero-shot TTS, content/acoustic/paralinguistic edits, enhancement, and source separation. Open-source 1.5B-parameter base model with a 4-step distilled AuK‑Flash for faster inference.
AI Audio2026
Zero-shot multilingual text-to-speech checkpoint for speech generation and voice cloning with a compact footprint. Features an ~170M-parameter main model plus a bundled ~120M-parameter codec decoder, with primary support for Chinese and English; other languages show more variable quality and long/noisy references reduce fidelity.
AI Model2026
Open-weights LLM fine-tuned for phone-based voice agents that prioritizes low latency and reliable tool/function calling. Based on NVIDIA Nemotron 3 Nano (30B total, 3.5B active), supports very long contexts (262,144 tokens) and recommends temperature=0 with thinking disabled for deployment.
AI Audio2026
Generates low-latency, instruction-driven English and Chinese speech for voice cloning, voice design, and directed performances; supports real-time streaming, reference-free voice creation, and reference-guided cloning. Open-weight PyTorch model released under a research/non-commercial license with GPU recommendations.
Provides a streaming dual-brain memory for real-time speech agents: an informational left brain for factual retrieval and an affective right brain for persona/emotion, achieving high top-5 accuracy while keeping retrieval latency within VAD budgets (~134 ms).
Generates synchronized spoken dialogue and explicit full-body co-speech motion (facial expressions, hands, upper- and lower-body) end-to-end from the same hidden states, replacing the speech-then-motion cascade. Trains with a scalable pseudo-labeling pipeline (422,856 ranked pairs) and supports real-time inference (RTF 0.78) while matching teacher motion metrics within ~2%.
AI Audio2026
Ruibin Yuan, Hanfeng Lin +56Multimodal Art Projection (MAP), Hong Kong University of Science and Technology (HKUST)
Generates full songs from lyrics and a style prompt, producing vocals and accompaniment and exporting editable symbolic scores for melody/chord control. Uses symbolic planning with agentic multi-turn editing and runs local 48 kHz inference on a 24GB GPU.
Provides a realtime audio–language foundation model for fully duplex spoken interaction that listens, interprets acoustic cues, reasons, and speaks in parallel. Key features include Deep Perception for acoustic understanding, Seamless Duplex for synchronized streams, and Think‑While‑Speaking for parallel private reasoning; reports strong benchmark results.
AI Audio2026
Performs speaker diarization (who spoke when) for live and recorded audio using an open-weight, 100M-parameter streaming-capable model that supports up to eight anonymous speaker channels, overlapping speech, chunked processing, and configurable latency for ASR integration.
AI Dataset2026
Provides 3,451 hours (2,051,810 clips) of AI‑generated 48 kHz Turkish speech with transcripts, spoken forms and per‑clip voice descriptions for TTS and ASR development. Includes 2,752 designed voices and is licensed CC BY 4.0 / CC BY‑SA 4.0 (attribution to PatientDesk AI required).