A search tool does not automatically make a model a good investigator. The useful distinction here is between finding a hidden fact and checking it once found: short answers keep grading simple while the search itself remains demanding.
Key Findings
- In the paper's evaluation, GPT-4o reached 0.6% accuracy without browsing and 1.9% with browsing; tool access alone left most tasks unsolved.
- Deep Research reached 51.5%, with the important caveat that it was trained on data targeting this type of task.
- More browsing effort and repeated attempts improved results, making the benchmark useful for studying the accuracy-versus-compute tradeoff.
- Questions combine several constraints, rewarding search reformulation and evidence checking rather than a single successful lookup.
Who Should Read It
Great fit if you evaluate persistent search agents and want an automatically gradable stress test. Look elsewhere if your main concern is report quality, ambiguous user requests, or typical everyday search: a difficult short-answer benchmark does not establish performance on those tasks.