A simulator that predicts the next click can still fail badly over a shopper’s full journey: preferences shift, sessions accumulate, and small behavioral errors compound. The key contribution here is to test fidelity at those longer horizons—and to compare simulated policy effects with a real production experiment, not just historical actions.
Key Findings
- The benchmark pairs real shopping trajectories with an interactive environment: 64,729 sessions from 11,010 users, covering 566,102 products and 13 action types. This gives researchers a setting for studying decisions beyond isolated clicks.
- Its five-level evaluation spans output validity, local decisions, sequential behavior, personalization, and outcomes. The progression helps identify whether a simulator’s errors arise in basic responses or accumulate across interactions.
- Offline validation compares simulated ranking-policy effects with a controlled production A/B test. This adds a practical calibration check: a simulator can match user actions yet still misestimate how a policy change affects click-through.
- On-Policy Trajectory Distillation (OPTD) uses trajectory-level intent to provide dense teacher targets on student-generated rollouts. In the reported evaluation, it achieved the strongest overall simulator fidelity and most closely reproduced the online policy effect among the evaluated simulators.
Method and Scope
OPTD addresses a mismatch in ordinary training: models are often trained on logged behavior but evaluated on their own generated trajectories. Trajectory-level intent supplies broader context while token-level supervision guides the model along those generated rollouts. The benchmark’s cross-session data also makes it possible to test whether preferences and behavioral tendencies persist across repeated visits.
Who It’s For—and the Trade-Off
A good fit for researchers evaluating agentic recommendation, user simulation, or offline policy comparisons who need long-horizon behavioral tests. The production-grounded setting is valuable but domain-specific: results on one e-commerce environment do not establish fidelity across other platforms, populations, or shopping contexts. Teams seeking a general-purpose simulator benchmark may need additional environments and user groups.
