Reproducing an experiment often tests how well an agent can guess what the authors meant, not just whether the released code works. ReproBridge’s central insight is that those recovered decisions—how to configure, launch, and measure a run—are reusable research knowledge, and preserving them can make reproduction both more reliable and less costly.
Key Findings
- Across 1,029 tasks drawn from 36 papers, four evaluated models improved from 76.4% mean success with raw code to 92.6% with refined artifacts; average downstream API spending fell by 19.7%. This suggests better handoffs can reduce the gap between stronger and weaker coding agents.
- The handoff combines a repaired codebase, environment setup, experiment launchers, and a guide that records decisions and paper–code discrepancies. Removing any component lowered success, with the largest drops from removing code or environment support.
- A separate Auditor checks whether reported measurements are supported by execution records and outputs. This makes the benchmark’s success criterion stricter than simply producing a plausible number, though automated auditing cannot catch every violation.
Who It’s For—and the Trade-off
A good fit for researchers and agent developers who want experiments to be independently rerun, compared, or handed to another model without rediscovering operational details. The results apply to the selected papers, tasks, and agents—not to all ML research—and artifact construction, review, and verification have costs not included in the downstream API savings. A reproducible handoff also cannot guarantee that the original experiment or its assumptions are scientifically sound.