Layer dropout vanished from many modern LLM recipes because of concerns it harms accuracy. This paper shows the opposite is possible: with careful choices of layer-density distribution, time schedule, and a few optimizer/hyperparameter adjustments, layer dropout reduces training compute and yields models that can be elastically accelerated at inference time with minimal accuracy loss.
Key Findings
- Best-practice configuration (Increasing Layer Distribution + Decreasing Time Schedule, “ILD+DTS”) produces models that match or beat dense baselines at the same training FLOPs, enabling roughly 20–25% non-embedding FLOPs savings so training can be cheaper or faster.
- Practical hyperparameter rules: scale activations during training by 1/ρ (ρ = layer density) to keep activation magnitudes stable; prefer per-sequence, block-level dropout granularity for better generalization; tune optimizer settings when increasing dropout to avoid degradation.
- Post-training elasticity: models trained with layer dropout are robust to zero-shot layer skipping, early-exit strategies, and self-speculative (draft-and-verify) decoding, producing up to ~1.4–1.5× end-to-end inference speedups with negligible loss on standard benchmarks.
- Empirical scale: conclusions hold across a large sweep (≈2400 runs) spanning models from 271M to 8.2B parameters and datasets up to 160B tokens, with all pretraining experiments run on Cerebras CS-3 hardware.
Who it's for and trade-offs
Great fit if you train foundation or production LLMs and need to reduce training FLOPs or want inference-time elasticity without post-hoc retraining. The approach requires extra hyperparameter tuning (layer density schedule, optimizer tweaks) and careful validation; earlier negative reports about dropout reflect poor configuration rather than an inherent impossibility. Not ideal if you require single-epoch pretraining regimes where dropout has been shown to hurt, or if you want plug-and-play results without any retuning. The paper does not fully explore attention-head or neuron-level dropping, non-transformer architectures (e.g., MoE), or produce a formal scaling law for maximum safe dropout rates.