Bringing discrete audio tokens and a dedicated audio encoder into a 30B Mixture-of-Experts text LLM enables unified handling of audio and text tasks without regressing on text intelligence. The design aims to let one model answer audio understanding queries, transcribe and translate speech, synthesize speech, and generate audio while keeping the backbone's reasoning, alignment, and long-context behavior.
Key Capabilities
- Unified modality support: Extends a Nemotron-Cascade-2 30B MoE backbone with an expanded vocabulary for discrete audio tokens and audio encoders to support audio-in / audio-out workflows (ASR, speech translation, audio understanding, text-to-speech, text-to-audio, speech-to-speech).
- Two inference pathways: Optimized recipes for vLLM-based streaming/offline inference and Hugging Face / transformers-based runs; recommends vLLM for offline audio QA and provides scripts for both server and batch use.
- Large context & thinking mode: Supports up to 1,000,000-token context and a special "thinking" mode (internal reasoning delimited by
<think>tags) plus an instruct (non-thinking) mode. - Practical engineering: Provides conversion and inference scripts (vLLM, Megatron-LM native inference, transformers >=4.53), and recommended sampling settings per audio task to balance fidelity and speed.
Who it's for and tradeoffs
Great fit if you need a single, locally runnable model that covers both advanced text reasoning and a wide range of audio tasks (ASR, translation, TTS, TTA, S2S) and can integrate with vLLM or HF transformers pipelines. Look elsewhere if you require a permissive commercial license (this release uses NVIDIA's OneWay Noncommercial License) or extremely small, low-latency edge deployments—the model is large (30B MoE with ~3B activated params) and optimized for multi-GPU / tensor-parallel setups. The release includes recommended inference setups and benchmark notes, but reproducing exact quality may require the provided XCodec assets and enhancement VAE for highest audio fidelity.
