AI Dataset2026
Compares 30 frontier LLMs generating static SVG markup from 500 prompts using 1,355,161 human votes across three leaderboards (Preference, Coherence, Alignment); provides raw SVGs, 768×768 rasterized PNGs, and per-comparison human vote records under a CC-BY-4.0 prompt license.
AI Dataset2026
Measures how autonomous AI agents learn via long-horizon, feedback-rich executable tasks; publishes 51 public tasks from a 134-task suite and provides SForge, a two-container evaluation harness for iterative 12+ hour runs to track learning trajectories.
AI Dataset2026
Provides 300 adversarial "metacognitive-trap" problems to measure whether LLMs notice and recover from their own reasoning errors. Combines multiple-choice vulnerability tests with free-form adapter-gain evaluation and ships per-model metacognition adapters for frozen-base probing.
AI Dataset2026
Provides 2,165 historical natural-history page scans paired with ~99.95% expert transcriptions and pixel-aligned PAGE XML layout ground truth for OCR and layout evaluation; multilingual (EN/FR/DE/LA), CC-BY 3.0.
Zhilin Wang, Han Song +14University of Science and Technology of China, The Chinese University of Hong Kong +6
Provides a benchmark and protocol to evaluate agents that iteratively edit executable policies under a fixed interaction budget, recording full execution–feedback–revise trajectories. Built from compact RL environments with trajectory-level diagnostics and hidden held-out validation.
Embodied AI2026
Evaluates generalist robot manipulation policies across simulation and real-world settings using 42 sim tasks and 18 real tasks; measures generalization, memory, precision, long-horizon execution and open-vocabulary instruction following, and provides a cloud-accessible real-world evaluation system with XPolicyLab integration and a public leaderboard.
AI Dataset2026
Provides fixed-seed benchmark instances (prompts and agent-visible inputs) for ASI-Bench to run reproducible evaluations of LLM agents on scientific tasks. Includes four matched prompt levels (B1–B4) across 60 project-level tasks in 11 domains; excludes reference answers and private scorers; Apache-2.0 licensed.
AI Agent Papers2026
A 365-day, order-level simulation benchmark for evaluating long-term coherence of LLM agents in seller-side e-commerce. Grounded in 98,843 real product records and 26 interactive tools, it pairs prompt upstream supplier signals with delayed downstream order outcomes to stress planning, memory, and tool use over long horizons.
Evaluates how large language models fabricate user attributes in personalization and whether model self-monitoring is a reliable signal. Introduces MirageBench (150 personas, 6 personalization tasks, judge-validated faithfulness taxonomy) and a 12-model leaderboard revealing pervasive over-inference and a 'Self-Monitoring Inversion'.
AI Dataset2026
A living, crowdsourced dataset of hard-to-translate examples (text, images, audio, video) paired with handcrafted verification rules that flag concrete MT failures. LTBv1 contains 3,456 peer-reviewed examples across many language pairs and accepts ongoing contributions.
AI Dataset2026
Provides 315,000 pairwise human-preference votes comparing 15 English TTS models over 300 operational prompts, with 4,500 high‑quality audio renders and structured vote/pair/prompt records for training or evaluating preference/reward models. Metadata under CC-BY-4.0; audio use governed by model providers' terms.
Provides a realtime audio–language foundation model for fully duplex spoken interaction that listens, interprets acoustic cues, reasons, and speaks in parallel. Key features include Deep Perception for acoustic understanding, Seamless Duplex for synchronized streams, and Think‑While‑Speaking for parallel private reasoning; reports strong benchmark results.