Non-uniform quantization matters because uniform low-bit quantizations waste precision on insensitive tensors or overcompress sensitive ones. This release pairs GSQ (a learned scalar quantizer) with RCO (a gradient-based budgeted allocator) to pick one quantization type per weight tensor, producing standard GGUF files that preserve downstream task accuracy at dramatically reduced sizes.
Key capabilities
- Per-tensor allocation: RCO searches for a budget-respecting assignment of quant types across tensors; GSQ produces the low-bit scalar quantizations used in the database.
- Multiple operating points: published GGUF files at ~2.50, 2.75, 3.00, and 3.50 bpw (8.4–11.8 GB) plus an mmproj BF16 vision encoder (0.9 GB) for multimodal workloads.
- Task-level fidelity: the 3.50 bpw IQ3_S build is reported as task-lossless on AIME25 and LiveCodeBench v6 and within ~0.5 points on GPQA-Diamond while reducing model size by ~4.6× versus BF16.
- Deployable formats: outputs are standard GGUF and run without modification in common local runtimes (llama.cpp, Ollama, LM Studio); optional -mtp builds include a speculative decoding head.
- Reproducibility: each GGUF ships per-tensor allocation dumps and an importance matrix used in search, enabling audit of the exact quantization assignment.
Who it's for and tradeoffs
Great fit if you need small, high-quality local weights for inference or research—deployable on CPU/GPU runtimes that accept GGUF and useful for multimodal use when paired with the provided BF16 mmproj. Look elsewhere if you require original BF16 weights, need a different base model, or cannot accept any BF16 components (the multimodal projector remains BF16). The build process and tooling (GSQ, RCO) are research-grade and tuned for accuracy; reproducing or adapting the search requires access to the provided artifacts and compute to run RCO/GSQ on a per-tensor database.
