AIAny
Icon for item

Ko-Online-Mind2Web

Evaluates whether web agents can complete tasks on live Korean websites. Its 312 human-written tasks span 104 sites, and success is judged by the final website outcome rather than a single prescribed interaction path.

Introduction

A web agent can follow plausible clicks and still fail the user’s request. This benchmark focuses on the outcome that matters: whether the real website ends up satisfying a Korean-language task. That makes it a useful complement to datasets that score agents on recorded actions or predict the next step offline.

What Sets It Apart
  • Live task completion: Agents start from a URL and interact with real websites; evaluation checks the result against the request. This tests end-to-end completion, not just action prediction.
  • Broad task and site coverage: The 312 tasks span 104 website hostnames and include constrained search, ranking, information lookup, and multi-page workflows. The range helps expose failures beyond a single site or interaction pattern.
  • Human-created Korean tasks: Two native Korean annotators wrote the requests and recorded browser demonstrations, contributing about 480 annotator-hours. This supports evaluation on Korean-language instructions and interfaces rather than translated tasks alone.
  • Difficulty reflects trajectory length: Easy, medium, and hard labels are based on the length of human reference trajectories. Task categories can overlap, and their allocation is an exploratory taxonomy rather than a mutually exclusive partition.
Who It’s For—and the Trade-off

A good fit for researchers comparing Korean-capable web agents on realistic, outcome-based tasks. It is less suitable when you need a stable offline replay set: live pages can change, and the repository requires Hugging Face access. For controlled next-action prediction, the suite’s offline benchmark is a better match.

Information

  • Websitehuggingface.co
  • OrganizationsKAIST
  • AuthorsYerin Nam, Minkyung Cho, Minkyu Kim, Seungwoo Song, Hangyeol Yoo, Gyun Lee, KyungTae Lim
  • Published date2026/11/16

Categories

More Items

Hugging Face

Provides paired text and synthetic American English speech in two voices for training lightweight TTS models. The 17,371 clips include phoneme-level annotations, with both voices reading each sentence.

Hugging Face

Evaluates offline action prediction for Korean web agents across 549 tasks on 144 websites, pairing screenshots and action histories with human-recorded next-step targets.

Hugging Face

Evaluates whether GUI agents can locate interface elements from Korean instructions and website screenshots. Its 3,319 human-annotated examples span 142 websites, providing a focused test of visual grounding on Korean web interfaces.