When an agent asks several questions about the same history, ordinary inference servers can spend much of their time recomputing or restoring that shared context. The key insight here is to treat those questions as branches of one computation: reuse the state without changing the result each question would get on its own.
Key Findings
- Branching preserves standalone results: ForkAttention tracks request and branch relationships in GPU-resident metadata, so questions share the context while remaining isolated from sibling branches. In exact arithmetic, each branch matches an independent forward pass.
- Hybrid models need state-aware execution: The method handles both standard attention and recurrent Gated DeltaNet layers, where each question must inherit the correct recurrent state. This extends shared-prefix execution beyond conventional attention-only models.
- The performance gain depends on shared work: Evaluations report up to 9.5× lower latency and 8.5× higher throughput than the best baseline on multi-question workloads. The largest gains occur when many questions reuse a long state; single-question or compute-bound workloads leave less work to eliminate.
Method and Trade-offs
Requests are represented as logical trees but run as a flattened token stream. GPU indices direct each question to its own shared state, avoiding repeated context computation and state copying. Floating-point results may differ from a float32 reference at rounding level, so the paper evaluates numerical deviation as well as speed.
A good fit for latency-sensitive agent systems that score or answer multiple questions against the same history, especially with hybrid recurrent-attention models. Look elsewhere if the workload is general-purpose text generation or rarely shares context: the reported gains target decision-style inference and should not be assumed for unrelated serving workloads.
