A simulator can make inaccurate predictions and still recommend the right design—or its errors can flip the ranking and make that recommendation unsafe. The difficult question is not whether a model notices a flaw, but whether it understands when that flaw matters to the decision.
Key Findings
- Decision quality is the test, not flaw detection. The benchmark separates recognizing a simulator limitation from correctly deciding whether to follow its recommendation.
- Matched workload pairs isolate the judgment. Candidate designs, the limitation, and the simulator’s recommendation stay the same, while a workload perturbation changes whether a reference simulator supports that recommendation. This makes the distinction between harmless prediction error and decision-changing error explicit.
- Review turns help, but are not a guaranteed fix. Self-correction can restore valid choices that models initially abandoned, yet it also introduces new mistakes for some models.
- The results expose a blind spot in systems research with LLMs. Evaluating reasoning only by whether a model identifies a modeling limitation can miss whether it makes sound design decisions under that limitation.
Who It’s For—and the Trade-off
A useful read for researchers building LLM-driven systems research agents or relying on simulation to compare LLM-serving designs: it offers a targeted way to evaluate whether model judgments preserve valid recommendations. Its evidence is specific to the studied LLM-serving design problems and thirteen evaluated models, so it does not establish how models handle every simulator or engineering domain.