Most speculative-decoding systems trade fidelity for speed or need an extra drafter model in memory. MTPLX takes a different approach: it uses the target model's own MTP heads to draft multiple tokens, verifies them in a single batched forward pass, and applies exact rejection sampling with the Leviathan–Chen residual correction so outputs are statistically identical to autoregressive decoding while running substantially faster on Apple Silicon (measured ~1.6× on a 16 GB M4 Mac mini and ~2.24× on an M5 Max).
What Sets It Apart
- Native MTP path (no external drafter): the drafter lives inside the model's MTP heads, so there is no second model consuming RAM or changing the distribution. This keeps memory use predictable and avoids distillation artifacts.
- Exactness-preserving acceptance math: uses probability-ratio acceptance + (p−q)+ residual correction, empirically verified (logits_max_abs_diff = 0.0) against single-token AR decoding—so temperature/top_p behave identically, just faster.
- Apple Silicon-first engineering: MLX-native runtime with Metal-optimized verify kernels and a compiled verify path (up to 32k compiled verify window on supported builds) tuned for macOS and modern Apple chips.
- Full product surface: a native Mac app that auto-tunes draft depth per machine, a CLI that serves an OpenAI/Anthropic-compatible local API, a Forge workflow to build and validate MTP adapters from Hugging Face repos, plus benchmarking and observability (tokens/sec, acceptance rates, verify waterfall).
Who it's for & trade-offs
Great fit if you want to run reasonably large LLMs locally on Apple Silicon and need higher token throughput without changing model behavior (coding, agents, RAG frontends that call a local server). The tool automatically recommends quantized builds (e.g., Qwen 3.8 Optimized Speed) and tunes decode depth for your Mac.
Look elsewhere if you need Linux-first GPU solutions (vLLM is the suggested Linux alternative), if your target checkpoints lack MTP heads and you cannot convert them, or if your machine cannot meet the unified-memory requirements for very large AR-only checkpoints (some checkpoints require very large unified memory to load). MTPLX also refuses untrusted remote inference code for retrieval models unless explicitly opted in.
Where it fits
Positions itself between local inference runtimes and full server deployments: it is a developer- and power-user oriented on-device inference and serving stack optimized for macOS/Apple Silicon rather than a general CUDA-backed data-center runtime.
How it works (brief)
Per cycle the MTP head drafts K tokens, the runtime verifies all K in one batched forward pass, and each position is accepted or resampled using exact rejection sampling with a residual-correction term. That yields the same sampling distribution as autoregressive single-token decoding but amortizes the verify cost across multiple tokens, improving throughput. The project remains MLX-native, integrates with Hugging Face model artifacts, and provides Forge to convert/measure/verify adapters on real hardware rather than assuming theoretical gains.
