Aggressive 4-bit NVFP4 quantization makes a 250B Mixture-of-Experts model practical to serve: this pack compresses both weights and activations into a packed compressed-tensors format optimized for vLLM and Blackwell-class GPUs while aiming to keep MoE routing behavior intact.
What Sets It Apart
- NVFP4 (W4A4) with group_size=16, packed for direct vLLM serving — both weights and activations use 4-bit float quantization, reducing the model footprint from BF16’s 500.6 GB to 153.3 GB. This is a real-world storage/transfer and memory reduction for very large MoE models.
- MoE-preserving quantization pipeline — uses Nota AI’s MoE-specialized calibration and two published techniques (DREAM-MoE, SRA-MoE) to align routing decisions during quantization, which helps retain downstream quality for MoE architectures.
- Near-original performance on benchmarks — aggregated scores remain very close to BF16 (avg ~81.35 vs 81.57 across provided benchmarks), with some tasks slightly better or worse; demonstrates that low-bit MoE quantization can be practical for many inference workloads.
- Deployment-focused packaging — compressed-tensors format and vLLM integration enable serving with tensor-parallel setups; includes configuration notes and a recommended vLLM command to run on Blackwell hardware.
Who It's For and Trade-offs
- Great fit if you run large MoE LLM inference at scale and have access to Blackwell-class GPUs (e.g., B200/GB200) — NVFP4 requires FP4 tensor cores not available on prior NVIDIA architectures.
- Good when you need substantially smaller weight footprints and lower memory bandwidth demands while preserving MoE routing behavior for conversational or long-context text-generation workloads.
- Look elsewhere if you lack Blackwell hardware, need an unrestricted open-source license, or require bit-exact BF16 behavior — this release is distributed under the Upstage Solar License, which imposes naming, attribution, and license-inclusion requirements for derivatives.
Practical notes: expect slightly dataset-dependent quality regressions on some narrow benchmarks; plan tensor-parallel sizing and vLLM configuration for multi-GPU serving. The pack is explicitly a quantized derivative of Upstage’s Solar Open2 250B and targets deployment rather than model-development experiments.
