Why this matters
Making a 35B Mixture‑of‑Experts model runnable on a single consumer GPU removes a major hardware barrier for people who need strong reasoning, math and code capabilities without a multi‑GPU cluster. This 2‑bit build reduces the on‑disk size to 12.3 GB while retaining near‑FP8 quality on most benchmarks, so you can experiment, serve, and evaluate a Qwen‑class MoE model on commodity hardware.
Key Capabilities
- Size vs quality tradeoff: uses Escha's "eschamoe" mixed 2/3‑bit quantization for experts and int8 for dense layers to cut the footprint to 12.3 GB while preserving broad capability parity with a much larger FP8 baseline across knowledge, math, and commonsense tasks; long‑horizon code generation shows the largest measurable gap.
- Serveability and runtimes: packaged to run with Escha runtimes (SGLang for concurrency, structured output, and tool calling; ZML for a single‑binary, no‑Python fast path). Exposes an OpenAI‑compatible /v1 HTTP API for easy integration with existing clients.
- Practical GPU targets and perf: designed to run on a single NVIDIA GPU — minimum 16 GB (trade concurrency or context) and recommended 24 GB; detailed launch recipes and CUDA‑graph tuning produce high single‑stream and batched throughput figures across Ampere→Blackwell GPUs.
- Features for reasoning workflows: supports a thinking mode that emits a separate reasoning trace, configurable per request; the release documents thinking budgets and token limits used in evaluation.
Who it's for and tradeoffs
Great fit if you want to run a 35B MoE model locally for text, code, or reasoning tasks without a multi‑GPU setup, or if you need an OpenAI‑compatible local endpoint for development and benchmarking. It is also useful for benchmarking quantization and serving strategies on consumer hardware.
Look elsewhere if you need: multi‑GPU FP8/BF16 fidelity, the vision tower (this checkpoint is text‑only), or a Windows/WSL2 or non‑glibc (glibc ≥ 2.28 required) environment. Operational notes: the runtime is distributed separately (escha‑runtime‑qwen3moe), SGLang requires Python 3.12 and a pinned torch 2.9.x, and ZML offers a no‑Python single‑user binary with different latency/throughput tradeoffs. The model weights and tokenizer are Apache‑2.0 licensed.
