Most autoregressive video diffusion pipelines train on ground-truth contexts or on their own rollouts but still treat the historical key–value cache as a frozen past, leaving a gap: future losses cannot teach earlier generations how to write more useful context for later frames. That missing supervision—what the authors call the "historical context-gradient gap"—is why many native autoregressive generators struggle to preserve subject identity, scene layout, and temporal stability when extrapolating far beyond their training window.
Key Findings
- Restoring missing supervision without full backpropagation: SGF runs a no-gradient autoregressive rollout that mimics inference and records the generated context plus noisy latents at a sampled denoising exit; a second parallel pass recomputes KV representations and causal attention so future-to-context losses can update how earlier latents are written. This gives memory-writing supervision while avoiding the compute of full serial backprop.
- Large extrapolation from tiny windows: In experiments, SGF trained on very short (≈5s) windows can extrapolate video continuations lasting minutes, improving subject identity preservation, background/layout consistency, and temporal stability compared to standard Self Forcing.
- Training- and deployment-friendly: The two-pass design matches inference rollout behavior in the first pass (reducing train–inference mismatch) while keeping the heavy gradient computation localized to sampled exits, making it more practical to integrate into existing autoregressive video diffusion setups.
Who it's for and tradeoffs
Great fit if you develop or research autoregressive video generation and need better long-horizon extrapolation from limited context—especially when preserving identity and scene consistency matters. Look elsewhere if you require an out-of-the-box, zero-infrastructure solution: SGF adds complexity (a two-pass training loop and exit-step bookkeeping) and requires modifications to model training and memory/cache handling. It also focuses on improving native autoregressive objectives rather than substituting alternative architectures (e.g., retrieval-conditioned or latent diffusion approaches) that might be preferable for some production constraints.
Method note
The core trick is decoupling the forward rollout (no gradients) from a targeted gradient-aware reconstruction at selected denoising exits. By recomputing context KV pairs and causal attention in parallel for those exits, SGF channels future-latent losses back into how the model writes context, effectively teaching the model to create more useful causal memory without the prohibitive cost of end-to-end rollout backpropagation.