Why this matters
Compressing a 27B dense model into ~10 GB changes the deployment trade-offs: you can host a full-parameter LLM with large-context capability on a single consumer GPU instead of a multi‑GPU server. Escha-W2 uses a mixed 2/3-bit per-projection scheme (average 2.469 bits/weight) and an int8 embedding/head to reach that size while retaining near-FP8 quality on the suite of benchmarks the authors ran.
Key Capabilities
- Dense 27B preserved in quantized form: weights occupy ~10.15 GB on disk while preserving full parameter count, enabling single‑card hosting of weights + KV cache for a 64k context on a 24 GB GPU with tuned settings. This makes long-context experiments and local inference practical on fewer resources.
- Low-bit quantization with measured quality parity: benchmarked against an FP8 reference on Commonsense-6, GPQA-Diamond and LiveCodeBench v6; reported deltas are small (commonsense avg +1.29 pp, GPQA −0.51 pp, LiveCodeBench +1.65 pp) and overall claims no measurable loss on the tested axes.
- Production-focused runtime and interface: shipped to run with a companion SGLang runtime (escha-runtime-qwen3dense) exposing an OpenAI-compatible HTTP API and expectable tuning knobs for memory, CUDA-graph capture, attention backend and thinking-mode template behavior.
- Long-context engineering: defaults target CTXLEN=65,536 with documented recipes to expand to ~131k tokens on tuned 24 GB configs and guidance on memory fractions, KV dtype, and concurrency budgeting.
Who it's for and trade-offs
Great fit if you need a full-parameter Qwen3.8-quality model on a single consumer GPU, want to experiment with very long contexts locally, or must reduce download/storage costs without a measured hit on reasoning or code benchmarks. The repo is weight-only and assumes use of Escha's SGLang runtime—expect to install pinned CUDA/PyTorch and the runtime wheel.
Look elsewhere if you need an out-of-the-box transformers/PyTorch path with standard attention backends (the quantized format and kernels are runtime-specific), require official multimodal image tower weights (this checkpoint is text-only), or cannot accept the operational constraints of a custom runtime (SGLang wheel, specific torch ABI, GPU driver/tooling requirements). Also test quality on your workload: published benchmarks are encouraging but quantized models can vary by backend and task.
Practical decision points
- Storage & memory: 10.15 GB weights plus tokenizer/config; on 24 GB cards the shipped defaults target one 64k stream or multiple shorter streams — follow the provided MEM/CTXLEN/MAXREQ tuning recipes.
- Runtime lock-in: the model is measured and served with Escha’s SGLang runtime; reproducing reported numbers on other backends is not guaranteed.
- Use-case fit: particularly useful for local inference, long-context retrieval/probing, or constrained-hosting scenarios. For teams that prioritize standard PyTorch serving or multimodal vision, consider alternative checkpoints or the unquantized base.
