Long-horizon consistency — preserving purposeful, adaptive behavior across many interdependent steps and delayed feedback — is a core obstacle for autonomous LLM agents that most short benchmarks miss. This work supplies a persistent, economically grounded simulation that forces agents to manage sourcing, listings, pricing, cash flow, and delayed order lifecycles over a full year, making delayed consequences and revisit logic first-class evaluation criteria.
Key Findings
- Realistic, long-horizon setup: a 365-day, order-level simulator grounded in 98,843 real e-commerce product records and instrumented with 26 merchant-visible tools that let agents modify catalogs, place orders, and inspect supplier/order states. This design couples promptly observable upstream supplier events with delayed downstream order outcomes, creating mixed-latency feedback.
- Broad evaluation matrix: eight LLMs under two agent frameworks (a minimal controller/ReAct-style baseline and a richer agent with planning, memory, and skill integrations) across 48 runs. Each run tracks per-order lifecycles, cash flow, penalties, and reputation.
- Quantified gap to humans: the best LLM configuration reached only 27.3% of the mean final net assets achieved by human participants, highlighting severe deficits in long-horizon planning, evidence aggregation, and adaptive recovery.
- Practical stress tests: the benchmark exposes failures in revisiting earlier decisions, handling supplier disruptions, balancing inventory/cash constraints, and summarizing long histories effectively for future decisions.
Who it's for and tradeoffs
Great fit if you need a reproducible stress test for agent-level planning, memory/summarization strategies, tool-use policies, or research on mixed-latency feedback adaptation in economically grounded settings. The simulator is valuable for comparing agent architectures, controller designs, and memory compression strategies under realistic business constraints.
Look elsewhere if your goal is immediate production evaluation of conversational quality or short-horizon tasks: the environment is domain-specific (seller-side e-commerce) and emphasizes long-term economic metrics over single-turn language metrics. Also, because it is a simulation with many domain-specific assumptions, transferred conclusions should be validated on real deployment data before production decisions.