AIAny

Mastering the game of Go with deep neural networks and tree search

Combines a policy network (to narrow move choices) and a value network (to score board positions) with Monte Carlo tree search, cutting Go's vast search space enough to beat top programs 99.8% of the time and the European champion 5-0.

Introduction

For decades Go was AI's white whale — a 19x19 board has more legal positions than atoms in the observable universe, so the brute-force search that cracked chess simply doesn't fit. AlphaGo's breakthrough wasn't a faster search; it was teaching two neural networks to do the pruning and judging that humans do by intuition, then letting tree search explore only the moves worth exploring.

Key Findings
  • A policy network trained on 30 million human moves predicts likely next moves, collapsing the branching factor so search wastes no time on hopeless lines.
  • A value network estimates who is winning from a raw board position — replacing the hand-crafted evaluation functions that had stumped Go programmers for years.
  • Both networks plug into Monte Carlo tree search, so the system reasons about consequences instead of pattern-matching alone. The full AlphaGo won 99.8% of games against other Go programs and beat European champion Fan Hui 5-0 — the first time a program defeated a human professional on a full board without handicap.
  • Self-play reinforcement learning improved the policy network beyond its human teachers, hinting that imitation was only the starting point.
Who It's For and the Trade-offs

Essential reading if you care about how learned evaluation plus search beats pure search, or about the lineage that runs through AlphaZero and MuZero. Be clear-eyed about the limits: this AlphaGo bootstrapped from human game records, so it inherited human bias and needed that supervised scaffolding. The 2017 follow-up, AlphaGo Zero, dropped human data entirely and learned from self-play alone — surpassing this version and making the case that the supervised stage was a crutch, not a requirement.

Information

  • Websitewww.academia.edu
  • OrganizationsGoogle DeepMind
  • AuthorsDavid Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot
  • Published date2016/01/27

More Items

Turns each research paper into a training environment to generate verifiable research plans by synthesizing questions from goals/background and deriving evaluation criteria from methods/experiments. Key features: four-stage extraction that reduces criterion leakage to 3.7%, a two-stage rubric-centered training (self-distillation then GRPO), and the PaperGym-20k corpus with two held-out benchmarks.

Analyzes on-policy distillation for LLM fine-tuning, shows teacher token-level supervision is often noisy and not the main driver of gains, and introduces OPSA, a supervision-free, entropy-adaptive method that suppresses low-probability tokens to improve downstream accuracy.

Trains LLM agents to proactively edit and manage their working context for long-horizon tasks using an expanded toolset (planning, long-term memory, soft offloading) and a fine-grained RL algorithm that identifies critical edits and assigns action-level credit. Improves accuracy while keeping contexts compact on long-context QA and deep search.