Multilingual speech pipelines often trade off between lightweight ASR systems and heavyweight multi-modal LLMs that handle punctuation, context, and translation. This model takes the middle path: a compact ~2B-parameter speech-aware LLM trained on large public and synthetic corpora to deliver higher-quality transcripts (with punctuation and truecasing) and direct speech↔text translation without stitching separate components.
Key Capabilities
- Modality-aligned speech-aware LLM: a CTC-based conformer encoder feeds a downsampled projector into an intermediate LLM checkpoint, enabling unified handling of ASR and speech translation while keeping inference tractable for sub-8B setups. This means you can get translation outputs and punctuated transcripts from the same model prompt.
- Dual-head CTC + frame importance sampling: character and BPE heads with posterior-weighted pooling improve token alignment and multilingual accuracy—particularly useful for languages with different orthographic systems (e.g., Japanese Katakana support alongside European character sets).
- Practical production features: punctuation and truecasing across supported languages (including German noun capitalization) via prompt control, keyword-list biasing for better named-entity recognition, plus variants that add speaker attribution and word-level timestamps or a non-autoregressive high-throughput option.
- Integration-ready: supported in transformers and vLLM, with example recipes for speculative decoding, finetuning, and deployment to endpoints (Azure-compatible). The model is released under Apache-2.0 for commercial and research use.
Who it's for — trade-offs and fit
Great fit if you need: enterprise-grade multilingual ASR or speech translation with built-in punctuation/truecasing and keyword biasing; teams that want a single model to handle both transcription and translation workflows; and users who can run mid-sized models on GPU-backed infrastructure (or via hosted endpoints). Look elsewhere if: you must run on very low-resource edge devices (2B parameters still requires substantial memory/compute), need guaranteed word-level timestamps and diarization without the plus variant, or require models trained on proprietary/private audio (fine-tuning will be necessary). Also expect general LLM caveats: the model can hallucinate in open-ended generation and should be constrained to ASR/AST prompts for safer outputs.
Where this sits in the stack
Unlike dedicated ASR decoders or massive multimodal models, this model optimizes for the speech→text interface: it avoids a separate language-only pipeline by modality aligning a speech encoder to an LLM backbone, which reduces engineering complexity when you need translation and well-formatted transcripts from the same input. For highest throughput use cases, consider the non-autoregressive variant; for speaker-aware transcripts, choose the ".plus" variant.
