AIAny
Icon for item

xlangai/osworld_v2_tasks

Provides the gated, official OSWorld 2.0 Python task class files (task_*.py) required to run the benchmark; distributed via a Hugging Face gated dataset to reduce benchmark leakage. Download requires accepting gated access on Hugging Face.

Introduction

Long-horizon computer-use benchmarks depend on authoritative, non-leaking task implementations. This gated Hugging Face dataset houses the official OSWorld 2.0 Python task classes that define environment logic and evaluators used across benchmark runs, so that agent evaluations remain comparable and resistant to accidentally leaked solutions.

What Sets It Apart
  • Canonical task classes: Contains the root-level task_*.py implementations that reproduce the benchmark’s environment, input artifacts, and evaluators — not just example data. This ensures evaluation parity across runs.
  • Gated distribution to prevent leakage: The files are intentionally distributed behind a Hugging Face gated dataset to reduce the chance that evaluated agents can find task answers or internal evaluator behavior during execution. This design choice prioritizes benchmark integrity over open ease-of-access.
  • Integration-ready format: The dataset provides JSON-formatted task files and is commonly used alongside the OSWorld codebase and evaluation tooling; consumers typically integrate these task classes into local benchmark runners or evaluation pipelines.
Who It's For

Great fit if you run or reproduce OSWorld 2.0 evaluations, develop agents that interact with long-horizon GUI/web workflows, or need the authoritative task logic for research-grade comparisons. Look elsewhere if you only need example scenarios or lightweight synthetic tasks — this gated package contains the official task implementations and expects you to follow the project’s benchmark release policies and access workflow.

Information

Categories

More Items

Hugging Face

Pairs metric-scale 3D scans of real environments with egocentric task recordings, estimated body and hand motion, and hierarchical annotations for physical AI research. The first release contains 126 scenes; the dataset is licensed CC BY-NC 4.0.

Turns LLMs’ physical knowledge into video training data using simulator constraints, physical verification, and iterative correction. It also repairs physical errors in generated videos and releases 100K videos spanning 2,500 physical events.

Hugging Face

Tests whether AI agents can interpret complex visual files, synthesize information across documents, and produce finished professional deliverables. Its 10 public tasks span distinct domains and require multimodal perception and multi-step tool use.