Most RL successes for reasoning still rely on verifiable ground-truth rewards, which become scarce as task complexity grows. Co-RL shows a different path: independent models can supply decorrelated supervision to one another, producing a usable learning signal without external labels and avoiding the self-reinforcing biases that doom self-rewarding methods.
Key Findings
- Cross-agent pseudo-rewards: Each agent samples multiple completions and a cohort-majority pseudo-answer is used as the reward target for other agents, breaking the policy-reward feedback loop common to self-rewarding RL.
- Diversity matters: Mixing model families, sizes, and rephrased prompts reduces correlated errors, preserves behavioral diversity, and mitigates training collapse—so ensembles improve both accuracy and robustness rather than homogenizing failures.
- Empirical gains: Co-RL yields average improvements of 3.0–8.6% across seven text-only reasoning benchmarks for LLMs and 2.3–7.2% across four multimodal benchmarks for VLMs, outperforming prior label-free baselines and matching or exceeding some supervised runs.
- Efficiency vs scale: Under controlled evaluation, Co-RL outperforms prior multi-agent methods while using fewer agents, showing the method scales effectively with cohort diversity rather than sheer agent count.
Who This Is For (and Tradeoffs)
Great fit if you want to improve reasoning ability of LLMs or VLMs without access to verified labels, especially when you can assemble heterogeneous models or checkpoints for a training cohort. Look elsewhere if you only have a single model instance or cannot afford the compute to run multiple decoupled agents in parallel—Co-RL’s benefits rely on decorrelated errors across a diverse cohort. Also consider supervised RL when high-quality labeled rewards are readily available and cheaper than maintaining a cohort.
How It Works (brief)
Agents share no parameters; training couples them only via rewards. For an unlabeled prompt, each agent generates multiple answers, the cohort forms a majority-voted pseudo-answer, and one agent’s completions are rewarded against another agent’s pseudo-answer. Rewards drive policy updates (e.g., GRPO or REINFORCE++). The framework is lightweight: no external judge or learned reward model is required, and all agents act as both learners and supervisors within the same run.