Before this paper, reinforcement learning that worked tended to rely on hand-engineered features; the moment you fed an agent raw pixels, the combination of noise, delayed rewards, and correlated, non-stationary data broke it. The lasting insight here is that two pragmatic fixes — experience replay (storing transitions and sampling them at random) and a convolutional network reading the screen directly — were enough to make deep learning and RL finally cooperate at scale.
Key Findings
- A single network architecture and hyperparameter set learned seven different Atari 2600 games from the same 84x84 pixel input, with no per-game tuning — evidence the approach generalizes rather than overfits one task.
- Experience replay was the load-bearing trick: random sampling broke the temporal correlations that destabilize naive online Q-learning, turning a notoriously divergent setup into something trainable.
- It outperformed prior methods on six of seven games and beat a human expert on three, the first credible demonstration that an agent could learn competent control end-to-end from vision alone.
Who Should Read It, and the Caveats
Great fit if you want the conceptual origin of the deep RL wave — DQN, Rainbow, and most pixel-based agents trace back here, and the writing is short and readable. Look elsewhere for state-of-the-art numbers: this is the 2013 NIPS workshop version, predating the larger 2015 Nature paper that added a target network and full 49-game results. It is also sample-hungry and value-based only, so it offers little to practitioners focused on policy-gradient or continuous-control methods.
