Most visual generators evolve appearance-centric latents and treat geometry as an afterthought; GAE flips this assumption by putting 3D inductive bias directly into the generated state. The core insight is that a shared, geometry-native latent — one that decodes natively to both appearance and geometry — makes it much easier for standard generative models to produce cross-view consistent output.
Key Findings
- A two-stage recipe: (1) train a codec that compresses a frozen geometry backbone’s multi-level features into one compact bottleneck decodable to RGB and geometry; (2) freeze the codec and train a DiT-style conditional flow in that latent space. This separates representation design from flow training.
- Quantitative gains: replacing a conventional latent with GAE reduces FVD by 12.7% on RealEstate10K and 23.1% on DL3DV, and halves camera-trajectory error on RealEstate10K — showing improvement in both perceptual quality and independent 3D coherence metrics.
- Latent structuring matters: GAE organizes the bottleneck at token and relational levels (token alignment to local features and pairwise similarity matching to preserve relational geometry), forcing a single compact state to represent both appearance and cross-view geometry.
Who it's for & tradeoffs
Great fit if you train conditional generative models for camera-controlled video, reference-conditioned novel-view synthesis, or any task where cross-view geometric consistency matters. It’s also useful when you can afford a perception backbone and a two-stage training pipeline. Look elsewhere if you need a zero-shot, single-stage generator without a frozen geometry backbone, or when minimal implementation complexity is the top priority.
Where it fits
GAE is a representational complement to recent flow- and diffusion-based generators: instead of adding geometry outputs to an appearance-first latent, it reparameterizes a geometry foundation model’s features into a generator-friendly latent. Practically, this makes perception and generation share the same interface, improving consistency without changing the downstream flow architecture.