Most inference systems assume a whole model must fit into fast memory. Colibri reverses that constraint: it treats storage, RAM and VRAM as a unified inference hierarchy and streams the routed experts on demand so only the dense core stays resident. That design lets very large MoE models answer on machines that could never hold them in RAM, trading raw latency for accessibility and reproducible, measurable placement policies.
What Sets It Apart
- Placement-first architecture: routing decisions and model semantics stay the same regardless of whether an expert is computed from VRAM, RAM or disk — placement only affects speed, not correctness. This preserves fidelity while enabling aggressive offload strategies.
- Experts streamed on demand: per-layer LRU caches, a learned pinned hot-set and an optional GPU expert tier let the engine adapt to a workload; a small dense footprint (int4/int8) remains resident in RAM while thousands of experts live on disk and are read in one pread when needed.
- Tiny, dependency-free engines: one C source file per model family, no BLAS or Python at runtime, and optional backends for Vulkan, CUDA and Metal. The launcher and tooling handle machine detection, model placement planning, and resumable downloads.
- Measured, reproducible approach: benchmarks and tooling are first-class; the project prioritizes end-to-end, hardware-aware measurements and keeps quality checks when using low-precision containers (int4/int8/fp8).
Who It's For & Trade-offs
Great fit if you want to run frontier MoE or large multimodal models (examples in the catalog include GLM-5.2 744B, Inkling 975B, Kimi K3 2.8T and Qwen3.6 35B) on local or heterogeneous hardware without needing a multi-GPU server. It is also suited to researchers exploring expert placement, storage-I/O overlap, and inference-side optimizations. Look elsewhere if your primary constraint is single-request low latency (cold disk access can be slow), if you require fully resident GPU inference for peak throughput, or if you cannot tolerate the engineering complexity of measuring and tuning placement and caches. Performance depends heavily on NVMe bandwidth, RAM/VRAM budgets and the working set locality of your workloads.
How It Works (brief)
A small dense core (attention, shared experts, embeddings) is kept resident in RAM at low precision; routed experts live on disk and are streamed per-token with multi-tier caching. The setup inspects hardware (RAM, disk, GPUs), recommends a model and build, and can run with the network off once model files are local. The project exposes OpenAI-/Anthropic-compatible APIs, a web dashboard (chat, System One decisions, expert atlas), and tools for benchmarking and machine-aware tuning.
