Local inference workflows increasingly demand models that are both quantized for efficiency and feature-complete for multimodal and long-context use. Qwythos-9B-v2-GGUF packages the Qwythos-9B-v2 SFT + FTPO improvements into GGUF artifacts targeted at llama.cpp, Ollama, LM Studio and other GGUF runtimes, trading a few GB of disk for deterministic, long-context and MTP-capable local inference.
Key Capabilities
- Quantization variants and sizes: multiple GGUF quant files from Q4_K_M (~5.34 GiB) up to BF16 (~16.69 GiB), plus MTP-enabled variants that keep MTP matrices at Q8_0. This makes it straightforward to pick a size/quality point for single‑GPU or constrained desktop inference.
- Long-context support: YaRN rope-scaling for a 1,048,576-token context window (configurable via -c up to 1048576), with guidance on expected KV-cache resource needs across GPU classes.
- Looping fix & reasoning preservation: FTPO (Final-Token Preference Optimization) reduced greedy/low-temperature repetition from 6.7% to 0% while keeping reasoning benchmarks at-or-above the base Qwythos levels; greedy decoding can be used deterministically in many cases.
- Multimodal option: an optional BF16 mmproj (0.86 GiB) pairs with any text quant to enable image input via the inherited Qwen3.5 vision tower (vision tower was not fine-tuned in v2; behavior matches base Qwen3.5-9B).
Who it's for + Trade-offs
Great fit if you need a deployable 9B-class model that runs locally on GGUF runtimes, want explicit quantization choices (Q4–Q8–BF16), need support for MTP speculative decoding, or require extremely long context windows for retrieval-augmented workflows. Look elsewhere if you need a rigorously fine-tuned vision encoder (the vision tower is inherited and not v2-fine-tuned), need official vendor support, or cannot accept the storage/memory footprint of higher-precision variants.
Where it fits
This release sits between raw Qwen3.5 weights and larger commercial hosted models: it preserves Qwen3.5-compatible multimodal plumbing and adds a targeted degeneration fix (FTPO) plus pragmatic quant files for local GGUF runtimes. For deterministic, repeat-free local runs at small disk budgets, start with Q4_K_M; for near-lossless fidelity, use Q8_0 or BF16.
