Most current benchmarks judge physical plausibility only from model outputs; they don’t test whether a model reached that output via law-grounded reasoning. Apple-π reframes video-model evaluation so the generated video itself becomes a visible reasoning trace and forces models to demonstrate law-grounded scientific thinking across perception, formulation, and deduction.
Key Findings
- Stage-resolved protocol: The benchmark enforces a three-stage scientific reasoning pipeline (Perception → Formulation → Deduction). This separation reveals where models fail rather than only whether they fail.
- Dataset design: Orchard contains 400 videos across ten canonical classical-mechanics tasks, split into single-law tasks for confounder-free diagnosis and multi-law tasks to probe state-transfer and compositional generalization.
- Hybrid evaluation: Combines MLLM-based subjective scoring with physics-law-grounded objective metrics, enabling both human-aligned judgments and strict law-checking for final states/trajectories.
- Empirical takeaway: Across 11 evaluated video models, the best model scored 0.473, exposing persistent Perception→Formulation→Deduction bottlenecks, weak multi-law transfer, and a notable Sim-to-Real gap.
What Sets It Apart
- Diagnoses reasoning stages, not just final outputs: By design, Apple-π isolates perception errors (extracting state from frames), formulation errors (translating perceived state into physics descriptions/variables), and deduction errors (simulating outcomes under physical laws). That means you can tell whether a model can't see, can't formalize, or can't simulate.
- Uses generated video as evidence of internal reasoning: The benchmark treats videos produced by models under chain-of-frames prompting as the model’s visible reasoning trace, allowing evaluation of intermediate steps instead of opaque latent computations.
- Mixed scoring to balance realism and rigor: MLLM scoring captures plausibility and human-like explanations, while objective physics checks verify conservation laws, kinematics, and quantitative end-states.
Who It's For and Trade-offs
Great fit if you want to: benchmark multimodal video models on physics-aware reasoning; diagnose whether failures stem from perception, symbolic formulation, or simulation; or develop models that must obey explicit physical constraints. Look elsewhere if your priority is large-scale natural video understanding without a focus on physical law, or if you need benchmarks centered on richly annotated real-world human activities rather than controlled mechanics scenarios. Trade-offs: Apple-π emphasizes controlled mechanics tasks (Orchard’s 400 clips), so its findings are diagnostic for physical-intelligence claims but less representative of unconstrained, diverse real-world video corpora.
Where It Fits
Apple-π sits between output-level realism benchmarks and fully symbolic physics engines: it targets multimodal video generators and reasoning models that claim to internalize physical laws and provides tooling to quantify and localize their shortcomings. Use it when the research question centers on law-grounded simulation and interpretability of multimodal reasoning.
Methodological Notes
Key methodological choices include chain-of-frames prompting with infographic-annotated first frames to steer perception, explicit task splits (single-law vs multi-law) to probe compositionality, and a hybrid evaluation suite combining LLM-graded explanations with automated physics checks (e.g., trajectory consistency, conservation constraints). The benchmark emphasizes reproducible stage-level diagnostics over single-number leaderboards.
