AIAny

Mastering the game of Go with deep neural networks and tree search

Combines a policy network (to narrow move choices) and a value network (to score board positions) with Monte Carlo tree search, cutting Go's vast search space enough to beat top programs 99.8% of the time and the European champion 5-0.

Introduction

For decades Go was AI's white whale — a 19x19 board has more legal positions than atoms in the observable universe, so the brute-force search that cracked chess simply doesn't fit. AlphaGo's breakthrough wasn't a faster search; it was teaching two neural networks to do the pruning and judging that humans do by intuition, then letting tree search explore only the moves worth exploring.

Key Findings
  • A policy network trained on 30 million human moves predicts likely next moves, collapsing the branching factor so search wastes no time on hopeless lines.
  • A value network estimates who is winning from a raw board position — replacing the hand-crafted evaluation functions that had stumped Go programmers for years.
  • Both networks plug into Monte Carlo tree search, so the system reasons about consequences instead of pattern-matching alone. The full AlphaGo won 99.8% of games against other Go programs and beat European champion Fan Hui 5-0 — the first time a program defeated a human professional on a full board without handicap.
  • Self-play reinforcement learning improved the policy network beyond its human teachers, hinting that imitation was only the starting point.
Who It's For and the Trade-offs

Essential reading if you care about how learned evaluation plus search beats pure search, or about the lineage that runs through AlphaZero and MuZero. Be clear-eyed about the limits: this AlphaGo bootstrapped from human game records, so it inherited human bias and needed that supervised scaffolding. The 2017 follow-up, AlphaGo Zero, dropped human data entirely and learned from self-play alone — surpassing this version and making the case that the supervised stage was a crutch, not a requirement.

Information

  • Websitewww.academia.edu
  • OrganizationsGoogle DeepMind
  • AuthorsDavid Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot
  • Published date2016/01/27

More Items

Analyzes adversarial weaknesses of World-Action Models (WAMs) via BadWAM, a framework that crafts visual perturbations to decouple a model’s imagined future from its executed actions. Introduces two attack modes—action-only (disruptive) and imagination-preserving (stealthy)—and shows large drops in closed-loop task success (e.g., 96.5%→43.1%).

Converts completed on-policy trajectories into natural-language 'hindsight skills' and converts the skill-induced action probability shifts into a dense token-level on-policy distillation signal, jointly optimized with outcome-based RL to improve sample efficiency and long-horizon agent behavior.

Uses pretrained multimodal LLMs as zero-shot, training-free reward models for text-to-image RL by scoring how well the original text prompt can be recovered from a generated image via image-conditioned prompt log-likelihood; includes a Self-SpectraReward closed-loop variant.