Multi-speaker long-form audio typically needs separate ASR and diarization pipelines that are error-prone and hard to keep aligned. MOSS-Transcribe-Diarize collapses those steps: a single end-to-end model produces timestamped text with consistent anonymous speaker tags (e.g. [S01]) and optional acoustic event annotations in one generation pass, which simplifies pipelines and reduces error accumulation when processing meetings, podcasts, interviews and long videos.
Key Capabilities
- Joint transcription + diarization: produces time-aligned segments with anonymous speaker labels in a single pass, so you avoid chaining independent ASR and diarization systems and the label-matching step.
- Long-form and multimodal input: audio frontend uses 16 kHz, 80 mel bins and 30s chunking, enabling robust operation on long recordings and video audio tracks.
- Promptable and hotword-aware: supports custom instruction prompts and hotwords to bias recognition toward domain-specific terms, useful for meetings or industry vocabulary.
- Practical serving options: distributed-friendly (vLLM) and SGLang Omni integration with examples for high-throughput inference and verbose/json output formats for parsed segments.
What Sets It Apart
- Architecture: Qwen3-style causal text backbone + Whisper-Medium encoder with an audio-text bridge (temporal merge + MLP) that directly replaces audio placeholders, enabling end-to-end audio→text with diarization output formatting. This design trades separate clustering pipelines for a single-generation approach that keeps speaker labels consistent across long contexts.
- Evaluation highlights: on benchmark splits reported by the authors, the 0.9B variant achieves strong cpCER/CER numbers (e.g., AISHELL‑4 cpCER ~15.83, Podcast CER ~5.97 cpCER ~7.37, Movies CER ~6.36 cpCER ~12.76), showing competitive accuracy for multi-speaker long-form tasks.
- Output format and tooling: produces a compact canonical string format ([start][Sxx]text[end]) and includes helper utilities, a subtitle web app, and scripts for vLLM/sglang serving and batch processing, which reduces engineering overhead for integration.
Who it's for — and trade-offs
Great fit if you need a single model to transcribe and diarize long multi-speaker audio (meetings, podcasts, interviews) with easy deployment options and prompt/hotword control. It is also suitable when you want parsed segments (verbose_json) directly from an inference endpoint.
Look elsewhere if you require verified speaker identity (this model emits anonymous labels only), extremely low-latency streaming on tiny devices, or if you cannot run custom remote code (the Hugging Face repo requires trust_remote_code=True). The model benefits from GPU resources for long recordings and may require raising max_new_tokens for very long sessions. License: Apache-2.0.
