Most zero-data LLM self-play methods hit a ceiling in open-ended (unverifiable) tasks because a fixed proxy judge can only reward what it already knows to prefer. J-Zero addresses this by co-evolving the Judge alongside a Challenger (task generator) and Solver (answerer), so the evaluator's capability grows with the agents and continues to provide learning signal across many iterations.
Key Findings
- Empirical gains: J-Zero improves average performance over baselines, notably by ~4.2 points on verifiable tasks and ~8.0 points on unverifiable benchmarks, and maintains monotonic improvement through at least 10 iterations instead of plateauing after 1–2 iterations.
- Judge training without external labels: preference pairs are derived inside the loop via role-asymmetry (Solver responses preferred over Challenger answers) and subtask-amplification (divide-and-conquer Solver outputs preferred over one-shot answers), avoiding reliance on the Judge's own scores.
- Broad applicability: the method outperforms prior zero-data approaches across multiple benchmarks and model scales, showing the Judge is the bottleneck for sustained self-improvement.
How it works
- Adversarial co-evolution: the Challenger is trained to produce harder tasks (minimizing Judge reward), the Solver to maximize Judge reward, and the Judge is updated on Bradley–Terry loss using constructed preference pairs that have known ordering by design.
- Two preference sources: (1) role asymmetry supplies early reliable signals because the Solver is optimized to answer while the Challenger is not, and (2) subtask amplification supplies stronger supervision by decomposing tasks and recombining subtask solutions.
Who it's for and tradeoffs
Great fit if you research autonomous LLM training, self-play, or evaluation-robust methods for open-ended generation and want a zero-human-labels workflow. Look elsewhere if you require human-calibrated evaluation, rely on strict ground-truth labels for safety-critical deployment, or cannot afford the compute for iterative self-play and judge updates.