Why this matters CUAs record actions, states, and reasoning as they operate across GUIs and web platforms; deciding whether a trajectory actually fulfills an instruction is core to evaluation, dataset curation, and RL training. Existing manual verification is too slow to scale, and off-the-shelf vision-language judges show consistent failure modes. The paper's central insight is that judge reliability—not just raw model capability—drives whether automated verification can replace costly human annotation at training scale.
Key Findings
- Systematic leniency bias: many state-of-the-art VLM judges frequently label failed runs as successes, creating a high false-success rate that distorts downstream evaluation and reward signals.
- Benchmark design: OSReward bundles a realistic, human-verified corpus of CUA trajectories, plus two focused suites—OSReward-Hard for genuinely challenging cases and OSReward-Multi for fine-grained efficiency/alignment scoring—so evaluations stress practical failure modes.
- Data + models: OS-Shepherd-100K is a reasoning-annotated training corpus; OS-Shepherd models (9B and 35B) are trained on it and provide low-cost, stable reward judgments that match commercial judges at roughly 30–60× lower inference cost on the presented frontier.
- Analysis and robustness: the study includes input ablations, ensembling experiments, and held-out transfer tests that reveal when judges fail and when de-biasing generalizes to unseen benchmarks.
What this means (so what?)
- For teams building CUA evaluation pipelines, relying on off-the-shelf VLM judges risks introducing systematic mislabels that can corrupt dataset labels and RL reward targets.
- The OSReward benchmark lets practitioners measure these failure modes explicitly and compare judges under standardized, cross-platform conditions rather than toy or synthetic tasks.
- OS-Shepherd provides a practical midground: open reward models that are affordable enough to run at training scale while achieving judge-level accuracy close to commercial systems, lowering the cost barrier for academic and industrial experimentation.
Suitable for and tradeoffs
Great fit if you need a realistic stress-test for automated trajectory judges, want a labeled corpus for training reward/judge models, or aim to replace costly human verification in RL pipelines. Look elsewhere if you require end-to-end production deployment guidance (this is a benchmark and model-release paper, not a turnkey deployment stack) or if your domain diverges strongly from GUI/web CUA interactions (results may not transfer to unrelated environments).