Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models

Improves language-model problem solving by generating candidate answers, retaining verified successes, and fine-tuning on them. Repeats this loop with an expectation-maximization interpretation instead of collecting more human-written solutions.

Visit Website

Introduction

A model can learn from answers it already knows how to sample: verification turns occasional successes into reusable supervision. The insight is that better problem solving need not start with a stronger teacher, provided correctness is cheap to check.

Key Findings

PaLM 2 experiments on MATH and APPS favor verified model-generated solutions over human-solution fine-tuning.
Restarting each fine-tuning round from the base model improves transfer compared with continually updating the previous checkpoint.
More rounds can hurt: APPS regresses after the first iteration, making held-out validation essential.

Who Should Read It

Great fit if you study self-improvement with answer keys or executable tests. Look elsewhere if feedback is subjective or reliable verification is unavailable; sampling and filtering still require compute, prompts, and a reward mechanism.

Back

Information

Websitearxiv.org
OrganizationsGoogle DeepMind, Mila
AuthorsAvi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J. Liu, James Harrison, Jaehoon Lee, Kelvin Xu …
Published date2023/12/11

More Items

AI Agent Papers2025

DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research

Rulin Shao, Akari Asai +19University of Washington, Allen Institute for AI +4

Trains an 8B model to search, synthesize evidence, and write cited research reports. Its reinforcement-learning rewards use evolving rubrics grounded in retrieved evidence and contrasting model responses.

paper ai-agent rl research web-search+1

AI Agent Papers2025

Tongyi DeepResearch Technical Report

Baixuan Li, Bo Zhang +54Tongyi Lab, Alibaba Group

Trains long-horizon web researchers by combining agentic mid-training, synthetic trajectories, and reinforcement learning. A 30.5B-parameter model activates 3.3B parameters per token and learns to plan, search, and synthesize evidence.

paper ai-agent LLM RL research+4

Large Language Model Papers2025

Understanding R1-Zero-Like Training: A Critical Perspective

Zichen Liu, Changyu Chen +6Sea AI Lab, National University of Singapore +1

Explains how base-model behavior and GRPO normalization distort claims about reasoning gains. Introduces Dr. GRPO to reduce length bias, with a 7B training recipe reaching 43.3% on AIME 2024.

paper LLM RL reasoning math