Most streaming ASR systems either accumulate unbounded context or accept drift over very long streams. This project keeps memory and latency bounded for unlimited-length audio by running token emission on a fixed clock (80/120/160 ms) and maintaining a 30s rolling KV cache with exact RoPE re-basing, so you can run continuous real-time transcription without slow memory growth or gradual accuracy loss.
Key Capabilities
- Low-latency tokenized output: emits one text token per configurable audio clock step (12.5 / 8.3 / 6.25 decisions/sec). So what: you get fine-grained partial results suitable for live captions, streaming UIs, and low-latency agent pipelines.
- Unlimited-length, bounded-memory transcription: a 30s rolling KV window prevents RoPE drift and keeps memory/latency constant. So what: supports 24/7 continuous transcription jobs without model state blowup or periodic resets.
- Configurable delay and semantic VAD: per-session target_delay_ms (e.g., 240–560 ms depending on clock) and frame-level semantic VAD that distinguishes thinking pauses, stuttering, and true end-of-turn. So what: you can trade a small, controlled latency for higher final accuracy and better turn segmentation compared with pure acoustic VAD.
- Competitive accuracy for Chinese and English: reported metrics include aishell1 CER 1.75, aishell4 CER 2.893, librispeech clean WER 3.042, librispeech other WER 6.808. So what: accuracy is in the practical range for production ASR in both languages while retaining streaming guarantees.
Who it's for and trade-offs
Great fit if you need continuous, real-time speech-to-text for live services (streaming captions, voice agents, monitoring) and cannot afford unbounded memory growth or periodic re-initialization. It is also suitable when bilingual (zh/en) coverage and frame-level semantic signals (VAD) are useful.
Look elsewhere if you need the absolute best offline WER on long-form English benchmarks and are willing to run large-context offline decoders; some offline-focused models with huge context windows can slightly outperform on certain Librispeech splits. Also note this release is a preview ASR base: semantic perception beyond transcription is still under active development.
How it works (concise)
Architecture mixes a Voxtral realtime causal audio tower (32 layers, hidden 1280) with a Qwen2.5-3B-Instruct decoder and trained audio projector/frame-length embeddings. It ships as a merged checkpoint (bfloat16, ~8.17 GB weights) and provides two inference paths: simulated torch streaming and a vLLM-based real-time service with a 30s rolling KV window for 24/7 operation. Runtime constraints and a merged-weight deployment model mean the packaged checkpoint and matching inference code are required for correct streaming behavior.
