Why this matters Nemotron 3.5 Lightning targets a common production tradeoff: deliver the capacity of a large model while only running a fraction of parameters per token so agentic systems can handle very high call volumes with low latency. By combining a hybrid Mamba‑2 + MoE + attention architecture with multi-token prediction and external draft models, Lightning is tuned for execution-focused agent workloads rather than only benchmark peak scores.
Key Capabilities
- Efficient MoE execution: 30B total parameters with ~3B active per token—router-based expert selection gives the throughput of a small model with the capacity of a much larger one, lowering compute per request.
- Speculative decoding and MTP: built-in Multi-Token Prediction (MTP) plus optional draft models (DSpark, DFlash) let deployments trade accuracy and decoder latency for up to multi× output speed in high-concurrency or DGX Spark scenarios.
- Full-precision reference weights: BF16 release intended as the starting point for post-training (SFT, RL, distillation) and producing quantized variants (NVFP4, W4A16, GGUF); NVFP4 is recommended for inference-optimized deployments.
- Long-context support: validated up to 1M tokens on multi-GPU/Blackwell setups (single H100 memory-bound to ~256K in BF16), with recipes for vLLM and SGLang serving backends.
- Open data and licensing: weights, training recipes, and selected datasets released under OpenMDW-1.1 to enable research, customization, and reproducibility.
Who it's for and trade-offs
Great fit if you need an open, customizable model for high-throughput agentic systems or domain adaptation—teams that will post-train, distill, or produce quantized checkpoints for specific inference hardware. It’s also suitable for long-context research and integrating into multi-model routing via NeMo Switchyard. Look elsewhere if you require an out-of-the-box lowest-latency production endpoint in heterogeneous hardware environments without doing quantization or deployment engineering—Lightning BF16 is memory- and GPU-intensive; use the NVFP4 release or other inference-optimized checkpoints for latency-first production.
Practical notes
Training and post-training used very large synthetic and curated corpora (>20T tokens pretraining, detailed post-training mixes). Recommended sampling: temperature 1.0, top_p 0.95. Primary runtimes and validated stacks include vLLM, SGLang, and PyTorch on NVIDIA Ampere/Hopper/Blackwell hardware.
