DwarfStar 4 targets a narrow but increasingly important niche: running routed-expert large models locally or on commodity multi‑GPU servers without relying on a general-purpose GGUF runner. As model weights and compressed KV caches grow, practical local inference becomes possible on high‑memory laptops and workstations — DwarfStar 4 trades generality for an optimized, end-to-end inference path tuned for DeepSeek V4 Flash (and a tested subset of GLM 5.2).
What Sets It Apart
- Model-specialized path: deliberately not a generic GGUF loader — tensor layouts, quant mixes, and optional MTP/DSpark support are expected and exploited for speed and correctness.
- Routed-expert quant strategy: asymmetric 2-bit quantization (IQ2_XXS / Q2_K mix) that quantizes routed experts but keeps other tensors higher-precision to preserve quality under aggressive compression.
- SSD streaming for experts: lets machines with insufficient RAM still run Flash/PRO models by caching routed experts on NVMe and dynamically loading them, with configurable expert-cache budgets.
- Parallel and distributed modes: pipeline parallelism (layer-slicing), tensor parallelism (RDMA/TCP across Macs or CUDA tensor-parallel across GPUs), and CUDA multi-GPU topologies for production multi-user serving.
- Server and agent integration: OpenAI/Anthropic/Responses-compatible HTTP server, disk-backed KV cache with exact DSML tool-call replay, native coding agent and CLI for low-latency local workflows.
- Practical tooling: imatrix-backed quant tunes, quality/speed benchmarking, ds4-eval capability suite, and utilities for GGUF/DSpark/MTP workflows.
Who it's for — and tradeoffs
Great fit if you run or plan to run DeepSeek V4 Flash/PRO or validated GLM 5.2 weights locally (Macs with 96–512 GB, DGX Spark, Strix Halo) and need a highly tuned, model-specific inference runtime with SSD streaming and multi‑host parallelism. It is useful for local coding agents, multi-user on-prem serving, and experiments that require exact tool-call replay and large context KV caching. Look elsewhere if you need a general-purpose GGUF runner that supports arbitrary models, a fully stable production release (the project moves fast and is labeled beta), or a lightweight client-only SDK — DwarfStar intentionally prioritizes DeepSeek-specific optimizations over broad model compatibility.
Where it fits compared to similar tooling
- Compared to llama.cpp: DwarfStar owes much to llama.cpp's kernels/quant approaches but implements a self-contained C inference path specialized for Flash/PRO and GLM 5.2 layouts rather than a generic GGUF runner.
- Compared to vLLM-style servers: DwarfStar focuses on routed-expert workloads, SSD streaming, and tight model-specific correctness (tool-call replay, DSML handling) rather than being a general GPU inference orchestration layer.
Key implementation notes
- Imatrix and calibration tooling are provided to produce high-quality quant variants tuned for routed MoE shapes.
- DSpark speculative decoding and MTP are supported as opt-in experimental paths to reduce generation passes when beneficial.
- Disk KV snapshot format and exact DSML replay are engineered to keep stateless API clients aligned with live checkpoints across restarts.
Overall, DwarfStar 4 is a pragmatic, hardware-aware inference engine for teams and users who need to run DeepSeek‑family models locally or across trusted multi‑host setups and are willing to accept the project's narrow model compatibility and active development status.
