Bet that one neural net, scaled with HPC, could transcribe both English and Mandarin without hand-built pipelines — reaching human-competitive accuracy by training fast enough to iterate on architecture in days, not weeks.
AI Dataset2026
Provides open ASR and TTS speech data for 24 Sub‑Saharan African languages to train and evaluate speech models. Includes ~1,250 hours of transcribed ASR and ~235 hours of single‑speaker TTS with train/validation/test/unlabeled splits and mixed CC-BY licenses.
AI Audio2026
Generates controllable multilingual speech from text with nine predefined timbres and custom-voice control; supports voice design, quick voice cloning and low-latency streaming (first audio packet after a single character), suitable for real-time TTS and voice-design workflows.
Zero-shot TTS for expressive long-form monologue and multi-speaker dialogue, designed to preserve acoustic consistency, conversational coherence, and affective continuity. Trained on SwanData-Speech and using a 25 Hz VAE, pause-aware text conditioning, and a flow-matching DiT with DiffusionNFT fine-tuning.
Generates synchronized, streaming spatial audio from panoramic video and text prompts using a causal autoregressive diffusion transformer. Combines Spatial Video-Audio Contrastive (SVAC) alignment and online direct preference optimization (ODPO) to improve spatial perception, plus an automated annotation pipeline and public demos.
Provides a comprehensive benchmark for instruction-based audio editing across seven audio modalities and eight operation types, with 2,000 high-fidelity samples and a rubric that decomposes tasks into 17,741 verifiable criteria for multi-dimensional evaluation.
Continuously records egocentric visual and audio streams into a lightweight streaming memory that organizes experiences into current, short-term, and long-term tiers and retrieves multimodal evidence to answer queries about past events. Built for on-device use (smartphones/AI glasses) with dynamic retrieval routing.
Retrieves short speech segments from MEG recordings with a compact interpretable neural decoder trained against wav2vec 2.0 embeddings, and maps decoder weights to cortical source space to reveal which acoustic and linguistic features drive retrieval.
Supervises audio reasoning by generating per-sample, audio-grounded rubrics that evolve with model rollouts and serve as reinforcement-learning rewards, improving perception and adaptive multi-step reasoning while avoiding reward saturation.
Generates multi‑speaker speech and environmental audio from textual instructions or a reference clip, supporting zero‑shot voice cloning and detailed scene/specification control. Combines a cleaned, captioned dataset with a VAE-based multimodal generator, reward-conditioned quality control, and staged training to improve expressiveness and multi-audio modeling.
Provides a streaming dual-brain memory for real-time speech agents: an informational left brain for factual retrieval and an affective right brain for persona/emotion, achieving high top-5 accuracy while keeping retrieval latency within VAD budgets (~134 ms).