A web agent can follow plausible clicks and still fail the user’s request. This benchmark focuses on the outcome that matters: whether the real website ends up satisfying a Korean-language task. That makes it a useful complement to datasets that score agents on recorded actions or predict the next step offline.
What Sets It Apart
- Live task completion: Agents start from a URL and interact with real websites; evaluation checks the result against the request. This tests end-to-end completion, not just action prediction.
- Broad task and site coverage: The 312 tasks span 104 website hostnames and include constrained search, ranking, information lookup, and multi-page workflows. The range helps expose failures beyond a single site or interaction pattern.
- Human-created Korean tasks: Two native Korean annotators wrote the requests and recorded browser demonstrations, contributing about 480 annotator-hours. This supports evaluation on Korean-language instructions and interfaces rather than translated tasks alone.
- Difficulty reflects trajectory length: Easy, medium, and hard labels are based on the length of human reference trajectories. Task categories can overlap, and their allocation is an exploratory taxonomy rather than a mutually exclusive partition.
Who It’s For—and the Trade-off
A good fit for researchers comparing Korean-capable web agents on realistic, outcome-based tasks. It is less suitable when you need a stable offline replay set: live pages can change, and the repository requires Hugging Face access. For controlled next-action prediction, the suite’s offline benchmark is a better match.