A small checkpoint’s parameter count does not tell the whole compute story: this model reuses its middle layers three times, so local inference must account for a looped architecture as well as the memory needed to hold its weights. These GGUF builds make that trade-off adjustable, from compact quantizations to a larger, higher-fidelity option.
Key Capabilities
- A range of quantization sizes: The listed builds span 7.76 GB to 22.87 GB. The Q4_K_XL variant is 17.63 GB and reports the same top token as the Q8 reference in 98.0% of comparisons, offering a practical balance between memory use and output similarity.
- Text and image input: The underlying model processes images and text together and produces text, making it useful for local assistants that need visual understanding as well as coding, tool use, or conversational tasks.
- A looped MoE design: The 24-layer model repeats its middle 12 layers three times with shared weights. It has 26.2B total parameters and 3.4B active parameters before accounting for the additional computation from looping; the model card describes this as about 5.4B effective active parameters per token.
- Runtime choices matter: Unsloth provides builds intended for Unsloth Desktop and separate uploads for mainline llama.cpp. Support for looped models is still developing across local runtimes, so performance and compatibility can differ by software and hardware.
Who It Suits
Great fit if you want to run a multimodal assistant locally and need to choose a memory-versus-fidelity trade-off, or are testing local coding and agent workflows. Look elsewhere if you need consistent performance across runtimes, stronger factual recall than a compact model provides, or reliably high-quality non-English output; verify results and sandbox tool use, especially before allowing destructive actions.