Why this matters Most practical speech LLM deployments are bottlenecked by always-on audio tokenizers and encoders that occupy memory and add latency. Naively removing encoder layers perturbs the representations the frozen language decoder expects, causing deletion or premature stops. The core insight of this work is that a progressive, behavior-driven pruning schedule plus targeted representation- and policy-level distillation can remove encoder depth while restoring decoder-facing embeddings and transcript fidelity.
Key Findings
- Progressive pruning with short behavioral probes selects layer combinations that minimize downstream errors, and outperforms direct (one-shot) pruning in the authors' experiments. This reduces the risk of deletions and early end-of-sequence artifacts.
- Cross-scale distillation and representation alignment (regressing intermediate latents and matching scales) restore the pruned student's embeddings so the frozen LM backbone continues to decode reliably.
- Attention LoRA adapters and tied output embedding finetuning are used while keeping the language-model backbone frozen, allowing efficient adaptation with fewer trainable parameters.
- Empirical results: compressing Qwen3-ASR-0.6B from 18→16 audio-encoder layers lowered macro-average error from 5.61% to 5.27%; an 18→14 model reached 5.75% while cutting ~20.7% of audio-tower parameters. Under a matched recipe, a 1.7B teacher yielded 5.55% mean error vs. 8.45% for self-distillation.
Who it fits and trade-offs
Great fit if you need to reduce on-device or always-on tokenizer/encoder memory and latency for speech–LM pipelines but cannot retrain or change the frozen language backbone. The method is practical when you can run a distillation pipeline with transcript-consistency filtering and LoRA-style low-rank adaptation.
Look elsewhere if you require a fully end-to-end retrain of the LM backbone, or if your target setup cannot support any finetuning/adapters: the approach assumes the LM remains frozen and relies on targeted student-side adaptation. Also note that accuracy gains vary across benchmarks, so evaluate on your specific languages/domains before deployment.
