A strong result on a fixed retrieval benchmark does not guarantee that an agent can research a question on the live Web: evidence may be scattered across sites, formats, and interactive pages. This work tests that harder end-to-end capability—and finds that even the best of 12 frontier models answers fewer than half of the benchmark tasks correctly.
Key Findings
- OpenDS-World contains 200 independently verified tasks requiring agents to discover and connect evidence from multiple sources, rather than retrieve from a prepared corpus. This makes the benchmark a test of research execution, not just answer generation.
- The top model reaches 39.0% accuracy. Human attempts score 21.1% across 1,000 attempts, underscoring that these tasks can be difficult even when performed by people.
- The analysis tracks tool use, search cost, stopping behavior, and failure recovery. Together, these measures help distinguish weak answers caused by poor reasoning from failures in navigation or search management.
- The benchmark spans varied modalities, file formats, and interactive access requirements, making tool coordination and sustained search central parts of the evaluation.
Who It’s For—and the Trade-off
A good fit for researchers developing web-browsing agents or evaluating whether they can carry out multi-step research beyond static RAG settings. It is less suitable for measuring general language ability or isolated retrieval quality: performance depends heavily on the tool environment, web access, and the ability to manage a long search process.
Evaluation Design
Tasks require agents to explore the live Web using search and browser interaction, process heterogeneous evidence, and arrive at a uniquely supported factual answer. Independent human verification checks that the evidence is valid and the answer is unambiguous. The resulting scores should therefore be read as an evaluation of the complete research workflow, not of a model in isolation.
