Most scalable RL approaches for LLMs rely on deterministically verifiable reward signals (e.g., math/coding). This paper's core insight is that you can make open-ended generation verifiable by transforming the task into a proxy environment that generates its own ground-truth labels via internal rules: Reinforcement Learning with Self-Verifiable Rewards (RLSVR).
Key Findings
- RLSVR formalizes task transformation as a way to produce deterministic, rule-based rewards from latent environment variables, removing the need for human annotation or learned reward models.
- SpyRL instantiates RLSVR via information-asymmetric self-play: one agent (the spy) sees a degraded input while others see the full input; all produce outputs and then vote to identify the spy. Because the spy identity is set by the environment, votes yield fully verifiable rewards linked to output quality.
- Empirical gains: on Qwen3-8B SpyRL achieved 75.4% and 77.3% win rates on summarization and creative writing respectively, and improved mathematical reasoning performance for Qwen3-4B and 8B by 8.97% and 6.16% across seven benchmarks.
- Outputs receiving more suspicion votes correlate with lower intrinsic quality, supporting the validity of the induced rewards.
Who it's for and tradeoffs
Great fit if you want to scale LLM self-improvement for open-ended generation without human labeling or learned reward models, and can design asymmetric-information transformations of your tasks. Look elsewhere if your task cannot be meaningfully partitioned into asymmetric observations, if voting-based verification cannot reflect the target metric, or if compute/engineering costs of multi-agent self-play are prohibitive.
How it works (brief)
RLSVR defines a transformation Φ that maps the original task into a proxy environment with a latent variable z sampled by the environment. SpyRL operationalizes this by giving most agents full inputs and one agent a degraded input; agents produce outputs and vote on who is the spy. The environment then computes deterministic rewards from z and the vote outcome, enabling standard RL optimization (e.g., policy updates) using verifiable rewards rather than subjective judges.