AIAny
Icon for item

Video models are zero-shot learners and reasoners

Argues a single web-scale generative video model handles vision tasks zero-shot the way LLMs handle language. Probes Veo 3 on segmentation, edge detection, image editing, physical and affordance reasoning, and puzzles like maze solving and symmetry.

Introduction

GPT-3 made a quiet point that took the field years to absorb: scale a single next-token predictor on enough text and task-specific models start to feel like a detour. This paper asks whether video generation is now at the same inflection point — whether predicting the next frame, at web scale, quietly bundles together the perception and reasoning skills that vision research has spent a decade building one specialist model at a time.

Key Findings
  • A single frozen model (Veo 3) is shown solving tasks it was never trained for, just by conditioning on a prompt and an input image — no fine-tuning, no task heads. The work treats prompting a video model as the visual analogue of prompting an LLM.
  • The capability sweep is unusually broad: low-level perception (segmentation, edge detection), pixel-level editing, physical-property and affordance reasoning, and abstract visual puzzles like maze navigation and symmetry recognition all fall out of the same generative objective.
  • The framing matters more than any single benchmark: it reframes a video model not as a clip generator but as a candidate unified vision foundation model, the way GPT-style models became unified language models.
Why This Reframes the Debate

Most of computer vision is still organized around bespoke architectures per task. The provocation here is that emergent, zero-shot generality — the thing that made LLMs eat NLP's task zoo — may already be appearing in video. If the trajectory holds, the interesting question shifts from "which model for segmentation" to "which prompt."

Who Should Read It

Great fit if you work on vision foundation models, generative video, or are tracking where the next paradigm shift in perception comes from — the chains-of-frames-as-reasoning angle is the takeaway. Look elsewhere if you need production-ready accuracy today: this is an existence-and-direction argument about emergent ability, not a claim that a prompted video model beats tuned specialists on any given metric.

Information

  • Websitearxiv.org
  • OrganizationsGoogle DeepMind
  • AuthorsThaddäus Wiedemer, Yuxuan Li, Paul Vicol, Shixiang Shane Gu, Nick Matarese, Kevin Swersky, Been Kim, Priyank Jaini, Robert Geirhos
  • Published date2025/09/24

More Items

Rewrites text-to-image prompts with reinforcement learning that aligns policy updates to semantic spans and pools log-probability shifts within each span to weight token losses. Improves prompt engineering for frozen generators without extra renders (≈9.5% on GenEval, 7.6% on T2I-CompBench).

Provides a large corpus of runnable 3D programs and a multi-agent harness that generates, executes, measures, visually judges, and repairs executable 3D code to produce editable, verifiable 3D scenes. Includes ~269K runnable programs, 2.03M render–caption pairs, and an automated execution+repair pipeline.

Post-training pipeline that boosts game-control transfer by fine-tuning on 6,044 curated gameplay trajectories and then continuing training with PPO across 16 synthetic 2D/3D environments, extending the keyboard/mouse action space with relative pointer motion, sustained inputs, and composite actions.