Humans build spatial understanding by observing changes caused by actions and integrating those changes over time. Many VLM training setups focus on static spatial questions and therefore provide weak supervision for state transitions; this paper flips that perspective and trains models directly on interaction trajectories that pair a preceding observation, an action, and the resulting observation, creating explicit supervision for local transitions and their composition over longer horizons.
Key Findings
- Interaction-centric curriculum: Organizes learning into three levels — passive world-state transitions (L1), active self-state transitions (L2), and long-horizon interaction trajectories (L3) — so the model first masters local transitions before learning to integrate them.
- LSI-108K dataset: A curated mix of simulated and real interaction trajectories aligned to the three curriculum levels, enabling supervised signals that static QA datasets lack.
- Two-stage training: Applies Supervised Fine-Tuning (SFT) on L1/L2 to learn local transitions, then On-Policy Distillation (OPD) where a teacher with segment-level transition descriptions guides a student’s on-policy chain-of-thought, improving sequence integration over long trajectories.
- Empirical gains: Consistent improvements across multiple VLMs and spatial benchmarks, with large per-benchmark uplifts and cross-benchmark generalization, showing the approach helps both local transition accuracy and long-horizon reasoning.
Who it's for and tradeoffs
Great fit if you research multimodal/embodied reasoning or develop VLMs for robotics, navigation, or interactive perception — especially when temporal state updates matter. Look elsewhere if your task is purely static visual QA or you cannot supply interaction-like supervision; OPD requires privileged segment annotations for the teacher and longer training pipelines compared to standard SFT.
Where it fits
Positions itself between static spatial QA datasets and full reinforcement-learning pipelines: it uses supervised signals from interaction data (not reward-driven exploration) to teach state transitions, so it is more data-efficient than RL but more temporally focused than static VQA training.
Short method sketch
The pipeline constructs local transition tasks from trajectories and trains via SFT; for long trajectories, a teacher branch produces segment-level transition descriptions that supervise a student producing on-policy chain-of-thoughts, enabling the student to learn to compose transitions across segments.