Most autoregressive video generators rely on bounded context windows, which causes essential historical details—object identities, spatial layouts, event states—to drop out long before they are no longer relevant. This paper argues that treating retention as an explicit memory problem (what to store, how to update it, when to read it) is the key to reliable long-horizon, autoregressive video generation. The survey reframes diverse techniques under a unified operational view and pinpoints where current methods fail under closed-loop rollouts.
Key Findings
- Memory as a design lens: effective memory requires not just capacity but accuracy, accessibility, and causal influence; representations range from raw visual windows and KV caches to compressed latents, 3D priors, and retrieval banks, each trading fidelity for efficiency.
- Lifecycle operations matter: writing, reading, updating, managing, and integrating memory are distinct problems that demand specialized mechanisms (salience scoring, routers, retrieval, state-conditioned updates) and matching training strategies to avoid drift.
- Learning & evaluation gaps: self-rollout training, trustworthy state-updating supervision, and standardized evaluation tasks that reveal true long-term retention remain underdeveloped; many methods optimize short-horizon metrics that mask memory failure.
- Practical trade-offs: approaches that extend context (large KV caches or rolling windows) scale poorly; structured or compressed memories reduce footprint but need robust update rules to prevent inconsistency.
Who it's for and trade-offs
Great fit if you research long-horizon video generation, world-models, or memory-augmented generative systems and need a structured taxonomy plus pointers to recent mechanisms and benchmarks. Look elsewhere if you only need short-video synthesis or fidelity-focused diffusion recipes—this survey emphasizes temporal persistence, evaluation design, and memory-specific training over implementation details or single-model engineering.