A useful AI review is not simply a long list of possible objections: unsupported criticism can be as damaging as missed flaws. This work treats review quality as a balance between making sound, substantive points and covering the important concerns a paper raises—a framing that rewards careful criticism rather than volume.
Key Findings
- The authors build PaperSound-23K from 1,000 ICLR submissions and 23,000 human-written and LLM-generated review points, labeled for soundness and substantiveness. This gives training a more specific target than imitating whole human reviews.
- PaperCritic-4B is trained with reinforcement learning using review-level F1 as its reward. On the paper-review test set, it reaches 45.1 F1, compared with 23.7 for the best reported conventionally trained 4B baselines, and exceeds the listed frontier models.
- The review-focused training transfers beyond ICLR submissions: results improve on Nature-family paper reviews and on code review, legal contract review, and story feedback. This suggests the learned skill may extend to evaluating claims and evidence across domains.
Who It’s For—and the Trade-Off
A good fit for researchers studying AI-assisted peer review or evaluating methods for producing evidence-grounded critiques. It is not a complete substitute for human review: it focuses on soundness, not novelty, clarity, interestingness, or accept/reject decisions, and its recall measures coverage of known reference concerns rather than every valid issue.
How the Objective Works
A generated review point counts toward precision only when its premises are supported and its concern is substantive. Recall measures how many distinct high-quality reference concerns are covered. Their harmonic mean rewards balance, so a model cannot compensate for unreliable criticism with sheer coverage—or for narrow coverage with a few excellent points.