Why this matters
Compressing a 27B-class reasoning model into a ~3.9 GB deployable pack changes what ‘‘on-device’’ means: long-context (262K tokens) reasoning and multi-hundred-token interactive throughput become practical on laptops, single commodity GPUs, and even phones. Bonsai-27B deliberately pushes to a true 1.125 bits/weight binary format (GGUF Q1_0_g128) to minimize both stored footprint and weight traffic during decoding while preserving most of the FP16 reasoning ability.
Key Capabilities
- Tiny deployed footprint: ~3.9 GB language-model weights (1-bit sign + group FP16 scale, effective 1.125 bpw), enabling 27B-class models on mainstream laptops and single GPUs.
- Long context on-device: 262K-token full-context capability enabled by a hybrid-attention backbone (~75% linear attention) and 4-bit KV-cache quantization.
- Quality vs size: measured thinking-mode average 76.11 (≈89.5% of the Qwen3.6-27B FP16 baseline) with strong retention on math (91.66) and coding (81.88).
- Serving optimizations: GGUF Q1_0_g128 native pack consumed directly by custom low-bit kernels in a PrismML fork of llama.cpp (CUDA + Metal) and an optional DSpark speculative-decoding drafter for faster CUDA serving (measured 1.37x speedup on H100).
- Cross-platform throughput: runs on Apple Metal (e.g., ~44 tok/s on M5 Pro) and CUDA (H100 ~105 tok/s), with energy-efficient on-device decoding (~0.275 mWh/token on M5 Pro with DSpark).
- Licensing and components: Apache-2.0 license; optional ternary build (≈5.9 GB) for higher quality and companion MLX/MLX-Swift runtimes for Apple devices and phones.
Who it's for — and the trade-offs
Great fit if you need a real 27B-class model to run locally for privacy-sensitive, offline, or long-document workflows (researchers, developers building local agents, single-GPU inference deployments, or mobile prototypes). Bonsai is especially attractive when device memory is the gating factor: it brings full 27B reasoning to devices that cannot host FP16 or conventional
