The core insight: putting a genuine 27B-class reasoning model on phones and thin laptops becomes practical by storing nearly all weights as binary sign codes with group-wise FP16 scales, not by throwing away capabilities. That trade — 14.2× smaller footprint at roughly 89.5% average FP16 performance in “thinking” benchmarks — is what makes long-context, locally-run 27B workflows feasible.
What Sets It Apart
- True end-to-end binary weights (binary g128): every language-model weight is stored as a sign bit plus one FP16 group scale (1.125 bits/weight), yielding a deployed language-model footprint of ~3.9 GB. This is the most aggressive, non-escape-hatch low-bit operating point in the Bonsai family.
- Long on-device context: hybrid-attention backbone (≈75% linear attention) plus a near-lossless 4-bit KV-cache enables up to 262K tokens of context without impractical memory spikes on laptops and reasonable peaks on high-end phones.
- Cross-platform low-bit kernels: custom implementations for Apple MLX (Python/Swift) and CUDA/llama.cpp that consume packed sign codes directly (weights are never fully expanded to FP16 in memory), reducing runtime bandwidth and memory pressure.
- Speculative decoding support: ships with a DSpark drafter layer trained against the low-bit target, offering lossless speculative decoding and measured end-to-end speedups on GPU serving paths.
- Measured trade-offs: in thinking-mode benchmarks the 1-bit build scores ~76.11 average vs ~85.07 FP16 (≈89.5%), retaining strong math (≈91.7) and coding (≈81.9) ability while giving up some margin in instruction-following and agentic benchmarks.
Who It's For and Trade-offs
Great fit if you need 27B-quality reasoning or long-document agents on local hardware (high-end phones, mainstream laptops, or single-GPU servers) and must minimize resident memory. Good for privacy-sensitive, offline, or low-latency on-device deployments and for single-GPU serving where weight bandwidth limits matter.
Look elsewhere if top-tier FP16 parity is essential (the ternary 2-bit Bonsai or full FP16 builds retain higher absolute scores) or if your target workloads are heavily agentic long-horizon coding tasks — those remain areas of relative weakness. Also expect phone throughput and sustained generation to be thermally constrained.
Where It Fits
Bonsai 27B (1-bit) sits between conventional quantized builds and full-precision models as a density-optimized operating point: it delivers a 27B-class reasoning experience at a multi-gigabyte deployed size that fits per-app memory budgets on modern high-end phones and common laptops, enabling use cases (local long-context agents, private on-device inference) that were previously impractical at this capability tier.
How It Works (brief)
Weights are packed as sign codes in groups of 128 with a shared FP16 scale (binary g128). Custom low-bit GEMM kernels apply group scales inside the matrix multiply so the model is never expanded back into dense FP16 in memory. A 4-bit KV cache and optional DSpark drafter are used to keep context capacity and decode speed practical without regressing quality beyond the measured trade-offs.
