Why this matters
Careful low-bit quantization can retain near-BF16 quality for many benchmarks while reducing download and VRAM costs by multiple factors — but uniform formats waste precision on insensitive tensors. This release applies a gradient-based, budget-aware per-tensor assignment (RCO) over a GSQ-generated database of quantized tensor variants to produce standard GGUF files that run unmodified in common local runtimes, yielding practical mixed-precision LLM builds for inference and multimodal use.
What Sets It Apart
- Per-tensor mixed precision under an exact total-size constraint — RCO optimizes a discrete assignment of GGUF quant types for each weight tensor on a smooth Riemannian manifold so the overall average bit-width meets the target exactly. This concentrates bits where the loss is most sensitive.
- Deployable scalar formats — GSQ produces accurate low-bit scalar quantizations (2–3 bits) that remain compatible with GGUF and common inference engines (llama.cpp, Ollama, LM Studio), avoiding custom vector formats that complicate deployment.
- MoE-aware allocation and shipped artifacts — the search covers 352 tensors (304 dense + 48 fused routed-expert matrices) and produces two-shard GGUF builds; the large per-layer n-gram embedding shard is held at a fixed lookup format and can remain memory-mapped on disk to lower resident VRAM needs.
- Multimodal support — the repository includes a shared BF16 vision encoder/projector (mmproj) converted from the base checkpoint for image-text usage.
Key quantitative tradeoffs (practical guidance)
- Three operating points: Q2_0 (2.40 bpw, 66.4 GB total, shard1 37.6 GB), IQ2_XS (2.50 bpw, 68.0 GB), IQ3_XXS (3.00 bpw, 75.8 GB). Keep shard 1 resident; shard 2 (28.8 GB n-gram table) can be memory-mapped on SSD.
- Performance vs quality: Q2_0 is optimized for throughput (≈3.4× prompt throughput vs IQ2_XS, much lower decode latency), while IQ3_XXS matches the BF16 baseline on AIME25 and reaches ~99.4% of task average at ~1/4 the BF16 size.
- Runtime compatibility: files are standard GGUF and run in llama.cpp, Ollama, LM Studio; use -lm mmap --lazy-mode on to avoid loading the n-gram shard into VRAM.
Who it’s for and tradeoffs
Great fit if you need to run Qwen3.8-Flash-Next locally with constrained GPU/VRAM and want a supported, deployable GGUF that balances throughput and quality via per-tensor allocation. Choose Q2_0 for highest throughput/lowest latency, IQ3_XXS when reasoning quality parity matters, and IQ2_XS for an intermediate point.
Look elsewhere if you require per-expert (not per-layer) precision control, custom vector-quant formats for absolute top accuracy at a given bit budget, or a single-file BF16 build (this repo focuses on compressed GGUF shards and mixed-precision deployment).
