Running very large Mixture-of-Experts (MoE) models locally usually demands huge disk space, long conversions and specialized runtimes. This colibrì-format container removes the conversion step: it supplies token-identical int4 weights and auxiliary files so GLM-5.2 (744B MoE) can be executed by streaming experts from NVMe, enabling inference on machines with modest RAM.
What Sets It Apart
- Pre-converted, engine-native format: weights are stored as packed int4 nibbles plus per-row float32 scales using the exact math that colibrì's C kernels expect, avoiding local FP8→int4 conversion time and rounding differences.
- Expert streaming for low memory footprint: routed experts are kept on disk and streamed on demand, allowing a ~25 GB RAM inference footprint on consumer hardware while hosting ~21,504 experts.
- MTP multi-token-prediction shard included: supports lossless speculative decoding for higher throughput without changing model semantics.
- Not interoperable with common quant formats: this is neither GGUF nor AWQ/GPTQ — it only works with the colibrì engine.
Who It's For and Trade-offs
Great fit if you need to run GLM-5.2 locally without spending days converting a 756 GB FP8 checkpoint and you can provide a fast local NVMe (ext4) volume. The container is MIT-licensed and is convenient for experimentation and inference on CPUs (AVX2) or WSL2 setups. Look elsewhere if you require GGUF/AWQ/GPTQ tooling, GPU-native formats, network-mounted storage, or a portable single-file model: the design expects local NVMe, the colibrì runtime, and ~400 GB free disk. Also, this repo is an engine-specific derivative—operational details (engine build, exact runtime flags) remain with colibrì's project docs.
