JEV-35B-NVFP4 shows how targeted 4-bit floating-point quantization can make a specialized decision model practical to run on a single medium-sized GPU without rebuilding the whole model. By keeping attention, vision, routers and the decision head in bf16 and only quantizing the routed expert MLPs to NVFP4, the checkpoint reduces weight size from ~72 GB to ~25.6 GB and GPU memory for weights from ~66.5 GiB to ~23.3 GiB while retaining most of the bf16 behaviour (≈96.6% identical top-1 Decision Index answers).
What Sets It Apart
- Sparse decision head design: fine-tuned from Qwen3.5-35B-A3B, the model reads the final hidden token and emits logits for up to 264 slots; system-level calibration (per-kind temperatures) produces well-calibrated probabilities for choice (2–256), score (0–5) and noul (yes/no). This makes it a drop-in decision API rather than a token generator.
- NVFP4 expert quantization: routed experts (40 layers × 256 experts, 10,240 expert MLPs) are stored in NVIDIA NVFP4 (ModelOpt layout) while other components remain bf16. This yields a ~65% reduction in GPU weight memory and a 25.6 GB downloadable weights bundle suitable for one 32–48 GB GPU.
- vLLM-first deployment: packaged with a vLLM serve script and a dedicated POST /v1/decide route (System 1 as a LoRA adapter for the decision head), enabling low-latency, single-request decisions (median text latency ~215–227 ms on a B200 depending on build).
- Measured trade-offs: Decision Index 0.3 on the public suite is 59.18 (bf16 59.73, −0.57); vision rebuild score 71.23 (bf16 72.08, −0.85). Per-task differences are generally small; top-1 answers remain highly consistent with bf16.
Who it's for and trade-offs
Great fit if you need a compact, deployable decision API that returns calibrated probabilities for bounded options (routing, reranking, tool selection, simple automation) and you want to run it on a single 32–48 GB GPU. It’s useful for production services that prefer a small, deterministic decision endpoint (vLLM + /v1/decide) instead of generation. Look elsewhere if your workload requires maximum bf16 accuracy for the routed experts (small but measurable drops vs bf16), if you must run on GPUs without tested NVFP4 kernels (only B200/B300/GB200 NVFP4 MoE kernels were validated in the repo), or if you need strong multilingual support or high-stakes decisions without additional confidence gating. The model is English-centric and the NVFP4 performance was validated primarily on SM100 (B200) hardware.
Additional notes: the repository includes the NVFP4-quantized safetensors, quant config and calibrated activation ranges, the System 1 LoRA adapter and a 264-slot fp32 decision head; license is Apache-2.0. Deployment instructions use vLLM and the provided serve.sh to expose /v1/decide.
