Benchmarks LLM agents on realistic legal work by packaging lawyer-style assignments with client materials and expert, per-deliverable rubrics. Includes an execution harness to run, score, and compare agents across a large, evolving task set spanning multiple practice areas.
Provides a diagnostic suite that audits video-understanding benchmarks to find samples solvable without visual or temporal input, filters those shortcuts, and produces a distilled video-native testbed that reveals major capability gaps in current Video-LLMs.
Aligns chart images with plotting code, underlying tables, summaries, and reasoning-based QA for chart extraction and interpretation. Specialized subsets add human-verified, multi-panel, grounded, and multilingual examples.
Provides an end-to-end platform to evaluate, observe, protect, and optimize LLM and AI agent deployments. Integrates OpenTelemetry tracing, 50+ evaluation metrics, agent simulations, an OpenAI‑compatible gateway, and guardrails; self‑hostable under Apache 2.0.
Audits deployed AI agents for governance and operational misalignment, running up to 60 inspections that detect deception, manipulation, fabrication, unpredictability and opacity; outputs JSON/Markdown scorecards, a graded A–F score, and optional Pro reports and dashboards.
A collection of biology-focused 'mystery' tasks for benchmarking model performance on biomedical reasoning, evidence synthesis, and problem solving; curated by Anthropic and hosted on Hugging Face, designed for granular evaluation of scientific decision-making.
A prompt-only mixture of ~478k prompts designed to support antidoom-style generation and preference-data pipelines for reducing model repetition (doom loops). Prompts are stripped of answers and labels and sourced from many public datasets so it’s usable for FTPO/adapter generation but not for supervised QA evaluation.
Provides a county-harmonized corpus of U.S. municipal and county ordinance text (≈2.21M chunks) labeled for function, substantive indicator, and topic to support legal NLP, retrieval, and comparative local-law research. Includes model-assigned labels and continuous scorers (opacity, paternalism, enforcement discretion) plus coverage metadata; not exhaustive or a substitute for legal advice.
Collection of hands-on workshop materials and sample code from Anthropic's "Code with Claude" series, covering Claude Managed Agents, memory (Dreaming Service), eval-driven agent development, and multi-agent patterns. Not maintained and not accepting contributions.
A small image-folder dataset for multimodal/vision model safety benchmarking, containing under 1,000 curated images with annotations to exercise safety-related model behaviours; licensed CC BY 4.0 and hosted on HuggingFace.
Diagnoses physical plausibility in generated videos by scoring adherence to 13 observable laws. It pairs benchmark prompts and model outputs with quality-controlled human ratings for fine-grained evaluation.
Trains reusable natural-language 'skills' for frozen LLM agents by optimizing the skill document in text-space — using trajectory-driven edits, validation-gated updates, and deployable best_skill.md artifacts. Multi-backend, zero inference-time cost at deployment, designed for iterative, validation-led skill improvement.