AIAny
Icon for item

Gamer: A Generalist Vision-Language-Action Foundation Model for Long-Horizon Real-Time Game Agent

Turns screen observations into keyboard and mouse actions to complete long-horizon game tasks. Sparse reasoning, persistent memory, and online reinforcement learning support real-time interaction and multi-step decisions.

Introduction

A useful lesson from Gamer is that general-purpose interactive skills may emerge from data that looks anything but sophisticated: ordinary keyboard-and-mouse play. The paper’s central bet is that scaling these trajectories, then teaching an agent when to reason and how to preserve progress, can bridge short-context models and tasks that unfold over many steps.

Key Findings
  • Training follows three stages: continued pre-training on interaction trajectories, instruction tuning with sparse reasoning and memory annotations, and online reinforcement learning. This progression moves from learning how to control an interface to carrying out task-level plans.
  • Memory summaries connect trajectory segments across context windows, while temporal-aware reinforcement learning assigns credit across those segments. This targets a key long-horizon failure mode: losing objectives or repeating mistakes as interaction history grows.
  • The reported Honkai: Star Rail evaluations show gains from continued pre-training through supervised tuning to online RL: two side-quest scores reach 80.0% and 83.6%, and the minigame macro-average reaches 49.7% after RL. These results are task-specific evidence, not a universal measure of game-playing ability.
  • A streaming inference system reuses context and dispatches actions before decoding fully finishes. The authors report reducing request latency from 1,306 ms to 65.12 ms on a specified H200 workload, an implementation detail that matters because delayed actions can undermine real-time control.
Who It’s For—and the Trade-Off

Great fit for researchers studying vision-language-action agents, long-horizon reinforcement learning, or real-time control through standard input devices. The shared screen-and-input interface is a practical route to cross-game experiments. Look elsewhere if you need a general-purpose desktop agent, broad independent benchmarks, or a ready-to-use model: the reported evaluations focus on selected games, and strong results do not by themselves establish transfer to arbitrary interfaces or physical environments.

Information

  • Websitearxiv.org
  • OrganizationsmiHoYo Honkai AI R&D Team
  • AuthorsRun Luo, Kuan Zhang, Qi Deng, Yukun Chen, Zhihao Yang, Yufei Zhang, Renrong Hu, Yi Zhang, Guiyu Ma, Tianyu Fan …
  • Published date2026/11/13

More Items

Generates videos directly in pixel space without a pretrained autoencoder or cascaded refinement stages. It can jointly produce RGB frames and disparity maps, helping generated scenes follow target camera motion.

Reduces Transformer parameter count by repeatedly applying shared blocks, then combines their updates with geometric decay and learned attention. On ClimbMix, it beats standard baselines in validation loss with fewer Transformer parameters.

Speeds up lossless speculative decoding for vision-language models by focusing drafter training on visually dependent tokens. Sparse attention also reduces redundant work on long video inputs; reported speedups reach 3.58×.