AI Leaderboard2023
Ranks LLMs by real production token usage, not benchmarks. Aggregates traffic from millions of users hitting 400+ models through one API — sliced by model, lab market share, tool-call frequency, and image volume, updated weekly.
AI Leaderboard2023
Public leaderboard ranking LLMs and multimodal models across 70+ datasets — reasoning, knowledge, coding, math, and long-context. Blends open-source and proprietary benchmarks into one comparative view spanning GPT-4, Claude, Qwen, and InternLM.
AI Infra2020
Unified framework for few-shot evaluation of generative language models across 60+ academic benchmarks. Supports multiple model backends (Hugging Face, vLLM, APIs, local servers), configurable prompts/YAML configs, and reproducible exports for leaderboards and research comparisons.
AI Leaderboard2023
Blind side-by-side voting site where users send one prompt to two anonymous chat models, pick the winner, and millions of votes become Elo rankings across text, coding, vision, image, and video. Crowd preference, not static benchmarks, decides the order.
AI Leaderboard2023
Runs one-command evaluation of vision-language models across 80+ multimodal benchmarks, handling data download, inference, and metric scoring in a single pass. Supports 220+ LMMs; adding a new model means writing one generate_inner() function.
AI Dataset2024
Provides leaderboard-ready test splits for the Open ASR Leaderboard: converts unsafe custom loaders to Parquet, sorts samples by audio length, and packages eight ESB test sets (LibriSpeech, Common Voice, GigaSpeech, SPGISpeech, etc.) for reproducible ASR benchmarking.
AI Dataset2025
Multi‑modal closed-ended academic benchmark with 2,500 multiple-choice and short-answer exam questions spanning math, natural sciences, and humanities for automated grading. Curated by subject-matter experts, released under MIT, and includes a canary string to help prevent dataset leakage into model training.
AI Dataset2025
A human-verified subset of 500 SWE-bench test cases for evaluating models that resolve GitHub issues into PRs using unit-test verification. Contains problem statements and base commits (pre-fix) for reproducible unit-test based evaluation; suitable for benchmarking code-fix and issue-resolution capabilities.
AI Dataset2025
Benchmark dataset for evaluating agents on long-horizon software-engineering tasks (repo-level patches, test-driven fixes). Includes golden patches, related tests, and problem statements in parquet format; aimed at agent debugging and code-modification evaluation but requires full test environments.
AI Train2026
A challenge repository for training the best language model that fits inside a 16,000,000‑byte (16MB) submission artifact; provides baseline training code, FineWeb bpb evaluation, a public leaderboard, and compute-grant instructions for short 8×H100 runs.
AI Dataset2026
Benchmarks document-parsing systems on real-world enterprise PDFs and images—evaluates tables, charts, content faithfulness, semantic formatting, and visual grounding with human-verified, rule-level tests. Ships with ~2,000 pages, ~169K test rules, and an open evaluation framework for end-to-end pipeline scoring.
AI Dataset2026
Benchmark dataset for evaluating clinician-facing chat assistants: physician-authored conversations plus rubric items, use-case and difficulty labels, specialty metadata, and a built-in canary to reduce benchmark contamination. Hosted on Hugging Face under an MIT license.