Most agent benchmarks test well-specified tasks; real-world vendor engineers must instead clarify ambiguous asks, reveal unstated constraints, and ship systems that actually improve customer metrics. FDE-Gym reframes evaluation around these last-mile challenges by embedding tasks in simulated but production-like customer environments and grading delivered systems by measurable business uplift.
Key Findings
- Realistic scope: 95 end-to-end delivery tasks drawn from 20 customer scenarios across 18 sectors; agents interact with scripted stakeholders and modify production-like systems rather than just outputting text.
- Evaluation metric: normalized business uplift (0 = no change, 1 = expert reference) with penalties for stakeholder messages, trial runs, and waiting time to discourage excessive interaction.
- Empirical gap: across 13 model configurations the best mean score is 0.56 and only 14/95 tasks are fully resolved (>=0.8 with no incident), showing major failures beyond information gathering.
- Bottlenecks: discovering unstated constraints is the primary failure mode; even when key facts are obtained, many solutions still fall short of the reference uplift.
Who it's for
Great fit if you research agent reliability, long-horizon delivery, or evaluation metrics that tie systems to business outcomes; also useful for teams building evaluation harnesses or training forward-deployed engineers. Look elsewhere if you need plug-and-play datasets for supervised model training or tasks limited to short, well-specified prompts—FDE-Gym focuses on interactive, systems-level delivery and requires running the provided harness and scenarios.
Availability and reproducibility: the harness and experiment runner are open-source; original tasks and some examples are available via the project's distribution, while many reference solutions and variants are held out for evaluation integrity.
