AI Dataset2018
Alex Wang, Amanpreet Singh +4New York University, Paul G. Allen School of Computer Science & Engineering, University of Washington +1
A multi-task English NLU benchmark for evaluating models across nine tasks (acceptability, sentiment, paraphrase, similarity, and various NLI setups), with a diagnostic evaluation set and an online leaderboard to compare generalization and transfer learning.
Showed that fine-tuning a GPT model on public GitHub code yields a capable program synthesizer, and introduced HumanEval — the docstring-to-function benchmark that still anchors code-generation evaluation. A production variant powers GitHub Copilot.
AI Agent Papers2023
Tests whether browser agents complete 812 tasks across realistic shopping, forum, development, and content-management sites. Evaluates final outcomes rather than matching a reference sequence of clicks.
AI Dataset2023
Benchmark for evaluating general AI assistants with 466 short, real-world questions that require tool use, multimodality and reasoning; provides a public dev set and a withheld test set used for leaderboard evaluation.
AI Dataset2023
A multiple-choice benchmark for evaluating LLM understanding in Traditional Chinese across 66 subjects (elementary to professional). Contains ~22K verified questions covering STEM, humanities, social sciences and Taiwan-specific topics, with standardized splits and model leaderboards under an MIT license.
AI Agent Papers2024
Tests whether web agents can use visual evidence to finish 910 tasks across shopping, classifieds, and social forums. Execution-based checks distinguish successful actions from plausible explanations.
AI Agent Papers2024
Explains when language models benefit from external programs, comparing tool selection, use, and creation. Evaluates performance gains against compute costs and identifies gaps in tool-use benchmarks.
AI Agent Papers2024
Tests whether agents can finish 369 tasks in real desktop applications, using executable checks of the final state. The original study found a wide gap between agent and human success rates.
AI Dataset2024
Hisami Suzuki, Satoru Katsumata +4LLM-jp (Center for Large Language Model Research and Development, National Institute of Informatics), National Institute of Informatics
Provides manually curated Japanese instruction pairs (questions and safe reference answers) for improving LLM output safety, covering broad harm categories and regionally sensitive cases. Includes English meta-tags and standard splits for benchmarking and fine-tuning.
AI Dataset2024
A 12K-question multiple-choice benchmark that raises difficulty by expanding choices to ten and adding reasoning-focused problems. Designed for robust LLM evaluation with cross-domain coverage, expert curation, and emphasis on chain-of-thought performance.
AI Agent2024
Orchestrates configurable deep-research agent workflows that combine LLMs, web search, and MCP tools to produce structured research reports and evaluation outputs. Supports LangGraph Studio, multiple model providers (OpenAI, Anthropic, local models), and Deep Research Bench evaluation for benchmarked comparisons.
AI Dataset2024
A fine-grained, domain-labeled web corpus for NLP and LLM research that decomposes a deduplicated web crawl into 67 domains via iterative URL/content labeling and coarse→fine recall; includes per-domain statistics, validation subsets, and domain-benchmark analysis tools.