Most automatic scores or task completions do not by themselves prove that an AI agent actually produced a trustworthy scientific discovery. The paper's central insight is operational: convert a discovery claim into a set of reproducible, adversarially designed recovery and feedback tests that can be audited deterministically. That shift makes the negative side decidable (did the agent recover the target under controlled conditions?) and provides a structured, evidence-based language for distinguishing useful method effects from overfitting, extra search, or oracle adaptation.
Key Findings
- Protocol design: DCP structures certification into gated checks — Gate 1 (sealed evaluation improvement), Gate 2 (matched-agent recovery with withheld target history), and an optional Gate 3 (average effect of truthful feedback vs a neutral policy). This makes claims testable under specific experimental constraints.
- Core requirements: certification requires adequate controls, zero observed recoveries in certain baseline conditions, and a finite-sample upper bound on recovery rate in a fresh registered episode; valid recoveries must supply a recovery witness and can trigger a Core veto.
- Verifiability: the authors provide an LLM-free, deterministic offline verifier (dcp-audit) that replays frozen evidence and checks integrity without calling models or the Web.
- Empirical audits: two controlled audits (SQLite optimization and virtual catalyst control) produced zero recoveries in 96 baseline episodes (upper bound 0.0468) and paired results showing truthful feedback recoveries versus neutral-policy failures, demonstrating the protocol’s ability to separate effects in practice.
Who it's for and tradeoffs
Great fit if you are designing, evaluating, or regulating AI-driven research pipelines and need an evidence-first audit that distinguishes genuine method effects from search or oracle artifacts. It is practical for teams that can register episodes, freeze evidence, and run reproducible replay checks. Look elsewhere if you need lightweight, purely score-based benchmarks (DCP imposes experimental controls and registration overhead) or if you cannot provide sealed evaluations / reproducible evidence for legal or operational reasons.
Where it fits
DCP targets the evaluation gap between task-completion benchmarks and claims of novel scientific discovery. It complements benchmarks that measure capability by adding a controlled, adversarial audit layer that focuses on recoverability, controls, and feedback effects rather than only final scores.
How it works (brief)
A producer registers an episode and sealed evaluation; matched agents run with shared starting information while the target research history is withheld. Methods that meet numeric targets must provide recovery witnesses. The Core checks ensure zero baseline recoveries and bound recovery probability; optional evidence measures the average effect of truthful feedback after independent null calibration. An offline verifier deterministically replays the bundled evidence to reproduce the audit decision without model calls.