A correct-looking API call does not establish that an agent completed its task. The central insight is to assess tools by the capabilities they add and the cost of using them, rather than treating tool access as an automatic improvement.
Key Findings
- A shared definition covers external executable programs, separating perception, action, and computation to clarify which capability is missing.
- Five of nine surveyed API benchmarks lack executable evaluation, making call matching an incomplete measure of real task success.
- Compute comparisons reveal task-dependent gains; training and inference costs can change which method looks preferable.
Who Should Read It
Great fit if you design tool-using agents or benchmark them and need a framework for choosing evaluation criteria. Look elsewhere if you need an implementation tutorial or a current leaderboard: this March 2024 survey synthesizes heterogeneous studies, so its comparisons require attention to task and baseline differences.