Removing the autoencoder from video diffusion is not just an architectural simplification: it also lets the model generate geometry-aligned outputs alongside color. The trade-off is that the Transformer must model high-dimensional pixels directly, so the paper focuses on how to make this setup practical without specialized U-Net machinery.
Key Findings
- A large-patch DiT-style Transformer reaches competitive results without a pretrained autoencoder, auxiliary detailers, cascaded stages, or U-Net skip connections. On the reported benchmarks, its class-conditioned UCF-101 FVD is 136.5, and register guidance lowers it to 129.2.
- Patch embedding, register tokens, and adaptive layer normalization are examined as concrete design choices. The results favor patch-first spatiotemporal tokenization and standard AdaLN, while registers can add gains on more diverse data.
- Pixel-space outputs share a common grid, making joint RGB-disparity generation straightforward. On RealEstate10K, the RGB-disparity variant improves translation-following metrics over RGB-only JVT (RTA@30: 98.4 vs. 96.0; AUC@30: 88.6 vs. 84.1), with a small FVD increase from 27.0 to 27.5.
Who It Suits—and the Trade-off
A good fit for researchers studying video diffusion, pixel-aligned multimodal generation, or camera-conditioned scene synthesis. The evidence is based on relatively short clips and 256×256 video benchmarks; camera-conditioned experiments use eight-frame clips, and training relies on depth targets supplied by a frozen reconstruction model. Those seeking evidence for long, high-resolution generation should treat these results as an early design study rather than a settled scaling recipe.