AIAny
Icon for item

Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL

Uses cooperative multi-agent RL where multiple decoupled models provide peer-derived pseudo-rewards to each other, enabling unsupervised improvements in reasoning; increases cohort diversity to reduce correlated errors and avoid training collapse, showing consistent gains across text and multimodal benchmarks.

Introduction

Most RL successes for reasoning still rely on verifiable ground-truth rewards, which become scarce as task complexity grows. Co-RL shows a different path: independent models can supply decorrelated supervision to one another, producing a usable learning signal without external labels and avoiding the self-reinforcing biases that doom self-rewarding methods.

Key Findings
  • Cross-agent pseudo-rewards: Each agent samples multiple completions and a cohort-majority pseudo-answer is used as the reward target for other agents, breaking the policy-reward feedback loop common to self-rewarding RL.
  • Diversity matters: Mixing model families, sizes, and rephrased prompts reduces correlated errors, preserves behavioral diversity, and mitigates training collapse—so ensembles improve both accuracy and robustness rather than homogenizing failures.
  • Empirical gains: Co-RL yields average improvements of 3.0–8.6% across seven text-only reasoning benchmarks for LLMs and 2.3–7.2% across four multimodal benchmarks for VLMs, outperforming prior label-free baselines and matching or exceeding some supervised runs.
  • Efficiency vs scale: Under controlled evaluation, Co-RL outperforms prior multi-agent methods while using fewer agents, showing the method scales effectively with cohort diversity rather than sheer agent count.
Who This Is For (and Tradeoffs)

Great fit if you want to improve reasoning ability of LLMs or VLMs without access to verified labels, especially when you can assemble heterogeneous models or checkpoints for a training cohort. Look elsewhere if you only have a single model instance or cannot afford the compute to run multiple decoupled agents in parallel—Co-RL’s benefits rely on decorrelated errors across a diverse cohort. Also consider supervised RL when high-quality labeled rewards are readily available and cheaper than maintaining a cohort.

How It Works (brief)

Agents share no parameters; training couples them only via rewards. For an unlabeled prompt, each agent generates multiple answers, the cohort forms a majority-voted pseudo-answer, and one agent’s completions are rewarded against another agent’s pseudo-answer. Rewards drive policy updates (e.g., GRPO or REINFORCE++). The framework is lightweight: no external judge or learned reward model is required, and all agents act as both learners and supervisors within the same run.

Information

  • Websitearxiv.org
  • OrganizationsUniversity of Exeter, Independent Researcher, {yijiangli, nuno}@ucsd.edu
  • AuthorsYunhao Yang, Yuexin Bian, Yunjie Tian, Di Fu, Tianjin Huang, Yuanyuan Shi, Ziang Xiao, Nuno Vasconcelos, Yijiang Li
  • Published date2026/08/18

More Items

Trains AI reviewers to identify substantive, evidence-based concerns about papers. Its reinforcement-learning objective balances the precision of critiques with their coverage of known high-quality concerns.

Generates coordinated portfolio weights directly from multi-asset market histories, then refines them with portfolio-level reinforcement learning. In cryptocurrency futures tests, refinement improved net annualized return while reducing maximum drawdown.

Turns screen observations into keyboard and mouse actions to complete long-horizon game tasks. Sparse reasoning, persistent memory, and online reinforcement learning support real-time interaction and multi-step decisions.