A polished research report can still rest on weak evidence. The useful distinction here is between producing a convincing synthesis and retrieving sources that actually support its claims: stronger performance on one does not guarantee stronger performance on the other.
Key Findings
- The 100 tasks span 22 fields and both English and Chinese. Their topic allocation follows analysis of 96,147 user queries, connecting the test distribution to observed demand rather than a single technical domain.
- RACE adapts criteria and weights for completeness, depth, instruction following, and readability, then scores against a reference report. Its scores are relative comparisons, so a score near 50 is not equivalent to answering half the task correctly.
- FACT deduplicates statement-source pairs and checks whether retrieved pages support them. Reporting both citation accuracy and effective citation count reveals the tradeoff between collecting more evidence and citing it reliably.
- The study finds that strong report-quality scores do not ensure the best citation accuracy. Its human evaluation on 50 Chinese tasks also supports the value of adaptive reference-based assessment over direct scoring.
Who Should Read It
Great fit if you build research agents and need to distinguish synthesis quality from evidence quality. Look elsewhere if you need a large benchmark of arbitrary agent actions: this has only 100 report-oriented tasks, and LLM judging, reference choice, changing web pages, and uneven domain coverage all constrain how broadly to interpret the results.