Modern games expose agents to visually rich, dynamic states that supervised demonstrations alone rarely cover. This work argues that a focused post-training stage—combining supervised initialization with targeted reinforcement learning in programmatically verifiable synthetic environments—can bridge that gap and meaningfully improve cross-game transfer.
Key Findings
- Synthetic multi-environment RL improves transfer: training with Player-1-Gym (16 randomized 2D/3D environments) yields substantial gains over supervised-finetuned (SFT) policies, showing that curated, verifiable probes of control demands teach behaviors that generalize to real games.
- Action-space extensions matter: adding relative pointer motion, sustained inputs, and composite keyboard+mouse actions lets the agent learn richer control primitives; during RL the agent increases use of composite actions, indicating policy-level reliance on those primitives.
- Empirical improvements: Player-1-9B and Player-1-35B-A3B improve MineDojo-300 success over their SFT policies by 27.8 and 17.7 percentage points respectively; Player-1-9B reaches 38.33% success versus 12.63% for the strongest evaluated baseline, and also improves results on ViZDoom-10 and GameWorld.
- Broader synthetic coverage helps: wider variety in environment dynamics and observations during RL correlates with better transfer to held-out game benchmarks, suggesting environment diversity is a key lever.
Who it's for and trade-offs
Great fit if you want to push a vision-and-input-conditioned agent beyond imitation: teams building game-playing agents, computer-use models that must handle mouse+keyboard, or researchers studying transfer between synthetic probes and real interactive tasks. Look elsewhere if you need a plug-and-play supervised-only model or if you cannot afford the RL engineering and compute overhead; the approach requires environment engineering, reward specification, and multi-environment RL tuning (PPO).
Method highlights
- Supervised initialization on 6,044 curated gameplay trajectories that pair recorded low-level actions with task instructions and visual reasoning.
- Player-1-Gym: a suite of 16 engineered 2D/3D environments with programmatic rewards, randomized observations and control dynamics to support multi-environment PPO training.
- Policy-level insights: the agent increases composite/sustained inputs during RL, indicating learned exploitation of extended action primitives; broader env coverage improves downstream transfer more than narrow or single-env RL runs.
Taken together, the paper presents a practical post-training recipe: use focused supervised data to bootstrap low-level control, then use diverse, verifiable synthetic environments to let RL grow generalizable control habits that transfer across real games.