Why this matters
Real-time, open-ended video editing under streaming conditions is challenging because deployed systems cannot access future frames, must run with bounded compute and low latency, and must avoid accumulating temporal drift that breaks identity or motion consistency. The core insight of this paper is that combining causal, chunk-wise autoregressive inference with targeted distillation and long-horizon supervision can close the quality gap between streaming editors and stronger offline systems while enabling practical, single-GPU throughput.
Key Findings
- Real-time streaming quality: a 16B-parameter autoregressive diffusion editor delivers instruction-guided video-to-video edits while operating causally on incoming frames, achieving an end-to-end throughput of ~30.19 FPS at 720×1280 on a single Nvidia B200 GPU. This demonstrates that high-quality editing can be practical under strict latency and memory constraints.
- Few-step distilled generator: Source-Anchored Distribution Matching Distillation (SA-DMD) distills iterative diffusion denoising into a two-step generator that retains source fidelity by anchoring distillation to temporally aligned source chunks, enabling low-latency inference without large quality loss.
- Robustness to autoregressive errors: chunk-wise autoregressive adaptation (sliding temporal window with bounded KV cache) plus Long-Horizon Autoregressive Distillation expose the model to model-generated history and directly supervise states affected by accumulated errors, reducing drift across long streams.
- Empirical competitiveness: across automatic and human evaluations and a long-video benchmark (LongV2VBench), the method substantially outperforms prior streaming editors and is competitive with strong offline editors on both local and global editing tasks.
Who it’s for and tradeoffs
Great fit if you need: low-latency, instruction-driven editing on live or streaming video where future frames are unavailable; a system that must preserve subject identity and complex temporal dynamics across long durations; research into practical real-time generative video models.
Look elsewhere if: you require ultra-high-resolution offline editing without latency constraints (offline diffusion editors may still produce higher-fidelity per-frame results), have strict memory limits that preclude a 16B model, or need an architecture-agnostic plugin—this work is a full-stack model+training+distillation recipe rather than a lightweight client library.
Where it fits
This paper sits at the intersection of generative video modeling and systems-oriented distillation: it narrows the gap between offline iterative diffusion editors and streaming, causal editors by engineering training-time adaptations (clean-history teacher forcing → resampling forcing) and a source-aware distillation target. It is primarily a research/engineering contribution for teams building real-time video editing pipelines.
Method details (concise)
- Model stack: MLLM-based condition encoder, causal video VAE, and a multimodal diffusion transformer (16B params) trained first bidirectionally, then converted to chunk-wise causal inference.
- Chunk-wise inference: attention is bidirectional within each chunk and causal across chunks; a sliding window retains recent chunks plus the first chunk as a global sink, bounding temporal memory and per-chunk compute.
- Distillation & long-horizon supervision: SA-DMD supervises the distillation target along text-conditioning and source-fidelity axes using aligned source chunks; Long-Horizon Autoregressive Distillation performs segmented optimization over extended rollouts to directly correct accumulated autoregressive errors.
Taken together, these design choices yield a practical pipeline that balances editing quality, source fidelity, and real-time constraints while exposing reasonable tradeoffs for deployment on a single inference GPU.