Provides IdeaGene-Bench, a dataset and evaluation suite for scientific-lineage reasoning and lineage-grounded idea generation, representing papers as minimal, typed Idea Genome objects and GenomeDiffs that record inheritance, mutation, loss, import and novel insertion. Includes 1,961 lineage traces, IG-Exam (42 task types) and IG-Arena with a Population-Evolution Score for generation.
AI Dataset2026
Provides fixed-seed benchmark instances (prompts and agent-visible inputs) for ASI-Bench to run reproducible evaluations of LLM agents on scientific tasks. Includes four matched prompt levels (B1–B4) across 60 project-level tasks in 11 domains; excludes reference answers and private scorers; Apache-2.0 licensed.
Evaluates multimodal context learning across grounding, new information application, and knowledge acquisition using a 3,443-instance benchmark spanning science, finance, long documents, spatial reasoning, and web VQA; finds current multimodal models perform poorly (best score 0.2847) and analyzes failure modes.
AI Dataset2026
Generated instance set (seed 31415) for ASI‑Bench: includes four matched prompt variants, agent-visible inputs, reference artifacts, and instance metadata for 60 project-scale scientific research tasks across 11 domains; intended for evaluating autonomous research agents. Licensed Apache‑2.0.
AI Model2026
Finetunes Qwen3.6‑35B using an adversarial generator–critic loop that synthesizes ~10,000 verifiable “frontier” tasks to boost scientific research, long‑horizon reasoning, coding and tool use; supports an extended 262,144‑token context and common serving stacks.
AI Agent Papers2026
Evaluates whether AI systems can independently carry out project-level scientific research by progressively removing human methodological guidance across 60 tasks in 11 domains. Built with expert review, sandbox execution, and multi-agent–model scoring to measure innovation and autonomous experimental execution.
AI Agent Papers2026
Evaluates whether coding agents can modify real scientific software while preserving domain-specific scientific contracts. Contains 119 repository-level tasks across 98 GitHub projects and 20 scientific domains, measures reproducible edits in pinned Docker images, and analyzes recurring failure modes.
AI Agent Papers2026
Evaluates AI agents' ability to complete end-to-end scientific workflows by releasing and assessing 97 tasks from a 300-task FrontierChallenge suite across chemistry, materials, life science, and electrochemistry. Finds that top agent configurations achieved only a 20.6% pass rate despite high partial scores, revealing a gap between partial progress/confident completion claims and actual complete scientific deliverables.