Why this matters Many existing terminal benchmarks treat success as a binary final outcome and focus on short tasks, which understates progress on open-ended workflows and starves agents of learning signals. Long-Horizon-Terminal-Bench flips that design: by decomposing 46 long-horizon tasks into graded subtasks it converts sparse final rewards into dense intermediate feedback, forcing agents to solve extended planning, stateful iteration, and debugging rather than one-shot answers.
Key Findings
- Task suite and scale: 46 tasks across nine categories (experiment reproduction, software engineering, multimodal analysis, interactive games, scientific computing, etc.), where individual tasks typically require hundreds of episodes and minutes-to-hours of wall-clock execution. This makes the benchmark substantially more demanding than prior terminal-based suites.
- Resource profile: Evaluated agents consumed on average 9.9M tokens per task, with roughly 231 episodes and 85.3 minutes of execution time per run—quantitative anchors for expected compute and evaluation budgets.
- Performance gap: Even top-performing models achieve modest pass rates (best: 15.2% pass@1 at a 0.95 partial-reward threshold; 10.9% at a perfect 1.0 threshold). Mean pass rates across tested models are 4.3% and 1.7% under those thresholds, respectively—clear evidence of large room for improvement in long-horizon agent capability.
- Diagnostic value: Dense grading yields partial-credit signals that expose intermediate progress and common failure modes (e.g., planning collapse, context truncation, iterative debugging failures), enabling more informative analysis than binary terminal checks.
Who it's for and trade-offs
Great fit if you are developing or evaluating agents that must sustain long planning horizons, manage long contexts, or perform iterative debugging and multi-step workflows—especially research on agent architectures, memory, and evaluation protocols. The benchmark is also useful for system builders to quantify real-world compute and token costs for long-horizon runs. Look elsewhere if you need quick, low-cost evaluations or single-step task suites: the benchmark's demands (multi-hour runs, millions of tokens, hundreds of episodes) make it costly to run at scale and less suitable for rapid iteration or low-resource settings.