Real-world AI delivery fails when agents solve the wrong problem, miss unstated constraints, or produce systems that don't hold up in production. FDE-Gym reframes agent evaluation around that last mile: agents enter realistic customer workspaces, discover what matters, make engineering decisions, and ship systems that are replayed on held-out production traffic.
What Sets It Apart
- Task realism and scale — 95 tasks derived from 20 real delivery projects across 18 sectors, rebuilt as working customer environments (documents, services, scripted stakeholders). This goes beyond closed, well-specified benchmarks by forcing information discovery and trade-off decisions.
- Business-centered scoring — deliveries are scored as normalized business uplift (0 = do-nothing baseline, 1 = expert reference) with explicit penalties for messages, trial runs, and waiting time to discourage excessive or careless stakeholder interaction.
- Open harness and evaluated variants — the workspace harness and experiment runner are open-source; the dataset includes the original 20 cases and graders, while 75 variants and reference solutions are held out for leaderboard evaluation.
- Measured gap to experts — experiments show current model configurations reach modest performance (best mean ~0.56 in realistic settings) and only resolve a small fraction of tasks at expert-level quality, highlighting the challenge of discovering unstated constraints and delivering production-grade systems.
Who It's For and Tradeoffs
Great fit if you need an evaluation that stresses real-world delivery skills: research on agent-driven engineering, teams building autonomous delivery tools, or studies of human–AI collaboration in production systems. It forces information elicitation, risk-aware decision making, and end-to-end deployment robustness. Look elsewhere if you only need isolated algorithmic benchmarks (e.g., single-turn reasoning or synthetic tasks) or if you require fully public reference solutions and all variant data—many variants and reference solutions are intentionally held out to support leaderboard evaluation.
