A 145B parameter count usually suggests a multi-GPU setup, but sparse routing changes what must be active at once. This release keeps only a small subset of its experts in GPU memory and streams others from host RAM, making local use possible without fitting every weight in VRAM. The trade-off is that “one GPU” does not mean “GPU memory alone.”
Key Capabilities
- Sparse activation: Of 512 routed experts per layer, 10 are selected per token, alongside a shared expert; about 10.12B parameters are active per token. This cuts per-token computation, though the full model still requires substantial storage and data movement.
- Latent expert design: Expert inputs are compressed before routing and computation. The model combines this with 48 hybrid-attention layers and a 256K context, while keeping the shared projections at full width.
- Measured local performance: The NVFP4 build reports 98.3 tok/s agent decoding and 3,165.7 tok/s 16K prefill on an RTX 5090 D 24 GB. With expert residency enabled, the reported setup uses 23.25 GiB of VRAM plus 38.27 GiB of host RAM, so the headline GPU size is not the total memory requirement.
Who It’s For
A good fit for researchers and developers exploring large sparse models, local agent workloads, or inference under a constrained GPU budget—especially if they have ample system RAM and are comfortable with a specialized runtime. Look elsewhere if you need a self-contained model that fits entirely in 24 GB of VRAM, broad plug-and-play runtime support, or unrestricted commercial deployment: the release has a dual-license context, and some commercial uses require separate permission from Edge0 and/or the upstream Qwen rights holder.
