Most PTQ methods for LLMs treat all output channels equally by building reconstruction losses only from activation statistics. That assumption hides a key signal: gradient covariance contains layer- and channel-level sensitivity information that dramatically changes which weights matter for downstream performance. KronQ mines that signal with a Kronecker-factored Hessian approximation, producing a practical PTQ pipeline that stabilizes extreme low-bit quantization for very large models.
Key Findings
- Bidirectional incoherence processing: extends input-side random rotations to the output dimension using gradient covariance, reducing weight-magnitude variance across both input and output axes. This lowers reconstruction error concentration and makes subsequent uniform quantizers less prone to catastrophic degradation.
- Sensitivity-driven mixed-precision allocation: derives a new layer/inter-layer sensitivity metric from traces of the activation and gradient Hessians, enabling practical allocation of bit-widths across layers so that limited bits are placed where they most reduce loss.
- Empirical robustness at extreme bit budgets: kronQ achieves stable 2-bit weight-only quantization on LLaMA-3-70B with 7.93 perplexity on WikiText-2, whereas baseline second-order PTQ methods (GPTQ/GPTAQ) diverge or produce degenerate results (>2000 PPL) in that setting. The method generalizes to mixed-precision allocations and improves reconstruction across standard LLM benchmarks.
- Practical PTQ integration: KronQ is designed as a post-training step compatible with existing weight-only PTQ workflows; the extra cost is estimation of gradient covariance and performing bidirectional rotations, which are calibrated operations rather than full retraining.
Who it's for and tradeoffs
Great fit if you need to compress very large LLMs without retraining and want to push to 2–4 bit weight-only quantization while preserving perplexity and downstream behavior. KronQ is valuable for ML engineers and researchers who can run calibration passes to collect gradient statistics and who can apply small pre-/post-processing (rotations) at deployment. Look elsewhere if you cannot afford any extra calibration or runtime preprocessing on weights, if you require hardware-specific integer-only pipelines that disallow weight rotations, or if you plan to perform full quantization-aware training instead of PTQ. The main tradeoff is modest extra computation and calibration to estimate gradient covariance in exchange for much higher stability at extreme low-bit regimes.