Why this matters
Many production teams face a trade-off: keep a very large MoE model for peak accuracy, or compress it for feasible, high-throughput serving. This model targets the middle ground — preserving Nemotron‑3‑Super’s reasoning and long-context strengths while cutting the active inference budget and improving real-world throughput and concurrency.
Key Capabilities
- Architecture and scale: hybrid Mamba2‑Transformer LatentMoE design; compressed from ~120.7B total / 12.8B active to 75.3B total / 9.3B active parameters.
- Inference optimizations: Iterative Puzzle compression + knowledge distillation + RL recovery + post-training quantization (NVFP4 for Blackwell, FP8 for Hopper) and continued MTP training to raise speculative decoding efficiency.
- Performance outcomes: ~2× higher server throughput on a single 8×B200 node at matched user-throughput constraints and sustainable 1M-token single‑H100 concurrency rising from 1 to 8 requests in tested configs.
- Reasoning & capabilities: retains strong scores across reasoning, coding, multilingual and long-context benchmarks (examples: MMLU-Pro ~82, long-context RULER @1M ~93.2).
- Operational features: Multi-Token Prediction (MTP) support, up to 1M token context (default HF config: 256k), recommended runtime engines include vLLM and Hugging Face Transformers with trust_remote_code enabled.
Who it's for and trade-offs
Great fit if:
- You need a high-quality reasoning/chat model that is tuned for interactive, high-concurrency serving and long contexts (RAG, agents, multi-turn technical assistants).
- Your infra targets NVIDIA GPUs (Blackwell/Hopper/H100) and you can use NVFP4/FP8 optimized checkpoints and vLLM or optimized TF runtimes.
Look elsewhere if:
- You require a fully open permissive license (this uses OpenMDW‑1.1 which has specific governing terms).
- Your deployment must avoid vendor-provided toolchains or trust_remote_code usage, or you cannot allocate the GPU memory needed for very long contexts at scale.
Deployment notes
- Serving examples: vLLM serve commands and Transformers examples are provided; recommended tensor-parallel sizes and MTP speculative token settings are documented in the model card.
- Hardware: tested on NVIDIA H100/B200 and optimized for Blackwell/Hopper microarchitectures with NVFP4/FP8 variants.
- Safety & governance: released under OpenMDW‑1.1; NVIDIA provides safety, bias and privacy subcards and recommends V‑model testing and risk audits before commercial use.
