Why this matters now Modern embodied and multimodal AI models are increasingly evaluated on real-world, long-horizon tasks that combine perception, instruction understanding, planning and fine-grained control. Games are a controllable, diverse, and measurable testbed for those capabilities, but prior corpora either lack temporal instruction structure, cover few titles, or rely on high-variance online rollouts. GameHorizon's core insight is to unify data, annotation, and evaluation across multiple temporal horizons so offline benchmarks better predict real gameplay ability.
Key Findings
- Multi-component design: the suite pairs GameHorizon-Annotator (an automated bottom-up pipeline that produces short-, medium-, and long-horizon natural-language instructions), GameHorizon-Data (large-scale gameplay recordings aligned with frames, keyboard-mouse actions, and multi-horizon instructions), and GameHorizon-Bench (reproducible offline and stepwise online testing).
- Scale and density: the released dataset centers on ~5,000 hours of human gameplay with dense temporal alignment of instructions and actions, enabling standardized MCQ-style offline evaluation and stepwise verification in environments.
- Reproducible, diagnostic evaluation: the offline track uses thousands of standardized questions across three primary tasks (single-horizon action, multi-horizon instruction decomposition, cross-horizon consistency) and diagnostic variants; the online track decomposes long-horizon objectives into verifiable short-horizon subtasks, allowing error localization and validation of offline scores against actual gameplay.
- Empirical signal: evaluations over many models and invocations reveal a clear hierarchy of task difficulty (short < medium < long horizons) and substantial capability differences across model families.
Who this is for and trade-offs
Great fit if you need a reproducible, temporally structured benchmark and large-scale aligned gameplay data to evaluate or train multimodal, embodied, or RL-capable models. The suite is especially useful for research that links offline predictive metrics to online performance and for diagnosing stepwise failures in long-horizon tasks. Look elsewhere if you require direct RL online training loops with environment wrappers tailored to a single game engine, or if you need extremely low-level simulator access for frame-perfect control: GameHorizon emphasizes standardized evaluation and annotation density over bespoke environment API depth.