Most agent benchmarks are expensive because every multi-step run is executed to completion. EarlyEval rests on a simple, practical insight: an agent's eventual success or failure is often apparent well before the final step. By detecting strong behavioral indicators of success or failure in partial trajectories, you can stop runs early and avoid redundant environment interactions while preserving evaluation fidelity.
Key Findings
- Substantial per-task savings: across SWE-bench Verified, TerminalBench and Toolathlon EarlyEval eliminated roughly 13%–26% of execution steps and reduced input tokens by up to 44.1% (output tokens reduced up to 29.4%). So what: repeat evaluation passes become materially cheaper without redesigning benchmarks.
- High early-prediction fidelity: operating points attained 89%–97% prediction accuracy while keeping per-agent resolve-rate shifts to about one to two percentage points on average. So what: aggregate metrics and agent rankings remain reliable for most practical thresholds.
- Robust leaderboard preservation: on SWE-bench Verified a leave-one-agent-out setup yielded a Spearman ρ≈0.991 across 16 agents, indicating early-stopped evaluations largely reproduce full-run rankings. So what: comparative evaluations and research iteration cycles stay meaningful even when many runs are truncated.
- Lightweight, practical machinery: the system trains two LightGBM classifiers (success and failure) on behavioral, textual, and reference-solution features and applies a calibrated dual-threshold halting rule at each step. So what: negligible per-step inference overhead and compatibility with benchmarks that publish outcome-labeled trajectories.
Who it's for and tradeoffs
Great fit if you run frequent agentic benchmark passes or operate under constrained evaluation budgets and you have historical, outcome-labeled run trajectories to train on. EarlyEval complements benchmark-distillation approaches by cutting cost inside each task rather than removing tasks.
Look elsewhere or proceed cautiously if your benchmark lacks labeled past trajectories, if rare/counterintuitive success modes matter (EarlyEval can prematurely stop atypical long-horizon recoveries), or if absolute per-task correctness is mandatory for every single run. There is also a calibration tradeoff: earlier stopping increases compute savings but raises the risk of mispredictions and small metric perturbations.
How it works (brief)
Collect outcome-labeled trajectories from diverse agents on the target benchmark, extract multi-step features from the partial trajectory (behavioral signals, text context, optional reference-solution cues), and train two ensemble classifiers that estimate success and failure confidences at each step. During evaluation, evaluate both classifiers each step and terminate the run as soon as either confidence crosses its calibrated threshold, recording the predicted outcome instead of executing remaining steps. The framework is designed to rely primarily on reference-free behavioral signals so it can operate even when gold solutions are not available.