Evaluates whether AI systems can independently carry out project-level scientific research by progressively removing human methodological guidance across 60 tasks in 11 domains. Built with expert review, sandbox execution, and multi-agent–model scoring to measure innovation and autonomous experimental execution.
Turns embodied navigation into 2D visual prompting where a vision-language model selects image pixels that are projected to 3D actions; adds selective chain-of-thought, compressed anchor-trajectory memory, and a two-level alignment objective to improve sample and runtime efficiency.
Fine-tunes long-horizon LLM agents with evolution strategies so full-model updates run at inference-level GPU memory. Emphasizes trajectory-level credit via black-box rewards, online prompt–parameter co-evolution, and a cosine decay for perturbation scale to balance exploration and adaptation; suited for limited-GPU settings.
Turns natural-language PLC requirements into verified, runnable IEC 61131-3 Structured Text by driving a closed loop of generation, compilation, deployment, and behavioral verification on a live OpenPLC runtime. The verification-gated harness forces inputs, traces execution, repairs failures, and renders ladder diagrams plus process simulation to raise dynamic runtime pass rates.
Converts 200+ hours of expert Figma screen recordings into 3,469 Playwright-MCP action trajectories for training and evaluating vision-language and GUI agents; includes 126 long‑horizon tasks, phase labels, a 10‑skill taxonomy, and is CC‑BY‑4.0 licensed.
Proposes FACET, a framework that synthesizes verifiable terminal tasks by reconstructing scenario intent and grounding instruction, solution, and verifier in a shared executable container state. Key features include environment-first generation, execution-based validation, and targeted repair to preserve source intent and cross-artifact consistency.
Wraps static, hand-built environments with a programmable plug-in harness that reshapes environment behavior without changing underlying logic. EnvRigger automates diagnosis and synthesis of harness components from agent failure trajectories, validating edits via fresh rollouts to improve agent success and efficiency.
Provides 1,021.64 hours across 597 CAD/BIM workflows with synchronized screen recordings and interaction logs; each workflow includes video, timestamped input events, task specs, source files, final outputs, and evaluation rubrics for training or evaluating desktop CAD agents.
Proposes “Graph Engineering”: using explicit, dynamic graphs to represent tasks, agents, tools, and system state so LLM-based agent systems can coordinate, persist, and evolve. Surveys principles, methods, applications, and curates related resources.
Open-weights LLM fine-tuned for phone-based voice agents that prioritizes low latency and reliable tool/function calling. Based on NVIDIA Nemotron 3 Nano (30B total, 3.5B active), supports very long contexts (262,144 tokens) and recommends temperature=0 with thinking disabled for deployment.
Assesses mobile planning agents' ability to call tools, plan long-horizon workflows, and coordinate sub-agents in realistic, interactive phone scenarios via a stateful executable sandbox. Covers 13 domains, 212 tools, evidence-based verification, and tests memory, skill usage, permission and runtime constraints.
Evaluates whether tool-using LLM agents reliably complete stateful business workflows via 507 executable agent–tool–user tasks across retail, travel, auto insurance, neobank, and IT/HR consulting. Provides browsable Parquet tables for tasks, scenarios, and agent instructions; v1.0 is intended for evaluation-only.