A language model’s rich video-related representations do not automatically help a video generator: the generator can ignore them, and useful features for describing a scene may still fail to predict what happens next. This work treats effective latent use as a prerequisite, then trains for factor-level editing and sparse-input temporal inference as distinct goals.
Key Findings
- Grounding before specialization: The diffusion transformer is first trained with only frozen MLLM latents as its conditioning signal, making it more reliant on those representations rather than native conditioning routes.
- Control through paired changes: Paired videos identify a factor to change and content to preserve; factor-span replacement trains the generator to make that targeted change without unnecessarily altering other elements.
- Prediction through the generator: Text-only and text-plus-complete-video observations share the same video target. With the generator frozen, the MLLM and connector are optimized so limited observations become useful for video generation while full-observation reconstruction remains available.
- Three evaluated tasks: The paper reports improved results over its baselines on sparse-to-video generation, factor-aware editing, and multi-shot generation.
Method in Brief
The three stages address different failure modes in sequence: whether the generator uses MLLM features at all, whether those features support selective control, and whether they can infer temporal content from incomplete observations. This separation is the method’s central design choice; it avoids assuming that stronger general-purpose representations alone will yield controllable or predictive video generation.
Who It Suits—and the Trade-off
A useful read for researchers working on video diffusion, multimodal representations, cinematic editing, or generation from sparse guidance. The evidence is focused on the paper’s evaluated tasks and baselines; readers looking for a general-purpose video model, a released implementation, or broad independent validation should not infer those from the reported experiments alone.