AIAny
Icon for item

JVT: Just Video Transformer makes pixel-space video diffusion simple

Generates videos directly in pixel space without a pretrained autoencoder or cascaded refinement stages. It can jointly produce RGB frames and disparity maps, helping generated scenes follow target camera motion.

Introduction

Removing the autoencoder from video diffusion is not just an architectural simplification: it also lets the model generate geometry-aligned outputs alongside color. The trade-off is that the Transformer must model high-dimensional pixels directly, so the paper focuses on how to make this setup practical without specialized U-Net machinery.

Key Findings
  • A large-patch DiT-style Transformer reaches competitive results without a pretrained autoencoder, auxiliary detailers, cascaded stages, or U-Net skip connections. On the reported benchmarks, its class-conditioned UCF-101 FVD is 136.5, and register guidance lowers it to 129.2.
  • Patch embedding, register tokens, and adaptive layer normalization are examined as concrete design choices. The results favor patch-first spatiotemporal tokenization and standard AdaLN, while registers can add gains on more diverse data.
  • Pixel-space outputs share a common grid, making joint RGB-disparity generation straightforward. On RealEstate10K, the RGB-disparity variant improves translation-following metrics over RGB-only JVT (RTA@30: 98.4 vs. 96.0; AUC@30: 88.6 vs. 84.1), with a small FVD increase from 27.0 to 27.5.
Who It Suits—and the Trade-off

A good fit for researchers studying video diffusion, pixel-aligned multimodal generation, or camera-conditioned scene synthesis. The evidence is based on relatively short clips and 256×256 video benchmarks; camera-conditioned experiments use eight-frame clips, and training relies on depth targets supplied by a frozen reconstruction model. Those seeking evidence for long, high-resolution generation should treat these results as an early design study rather than a settled scaling recipe.

Information

  • Websitearxiv.org
  • OrganizationsZhejiang University, Shanghai Innovation Institute, Shanghai AI Laboratory
  • AuthorsJianjun Zhou, Hua Geng, Haoyu Guo
  • Published date2026/11/13

More Items

Speeds up lossless speculative decoding for vision-language models by focusing drafter training on visually dependent tokens. Sparse attention also reduces redundant work on long video inputs; reported speedups reach 3.58×.

Turns screen observations into keyboard and mouse actions to complete long-horizon game tasks. Sparse reasoning, persistent memory, and online reinforcement learning support real-time interaction and multi-step decisions.

Reduces autoregressive image-generation steps by scaling each spatial position’s quantized-code capacity alongside spatial compression. The approach reaches up to effective 64× compression while retaining strong generation performance.