Scientific progress is cumulative: new proposals inherit, recombine, fix, or discard components of prior work. Existing benchmarks rarely test whether models can trace that inheritance or generate proposals that coherently fit into a lineage. IdeaGene-Bench (IG-Bench) reframes scientific artifacts as structured "Idea Genomes" and formalizes lineage edits with GenomeDiffs, enabling both closed-form lineage reasoning and lineage-conditioned generation.
Key Findings
- Dataset scale and structure: 1,961 gold lineage traces, 1,085 curated Idea Genome objects, and 920 pairwise GenomeDiff records across 10 scientific domains. IG-Exam provides 42 task types and 1,029 evaluation instances for explicit reasoning. IG-Arena evaluates generation with a Population-Evolution Score (PES) that rewards correct inheritance, meaningful variation, and selection value for future research.
- Diagnostic results: Evaluations on 14 LLM-based systems reveal a substantial compositional bottleneck—top systems reach only 27.3% exact accuracy on lineage reasoning, and adding structured lineage context reshuffles model rankings rather than uniformly improving performance.
- Evaluation design: IG-Bench separates reasoning (abstraction, tracing, evolutionary attribution, verification) from generation (lineage-conditioned insertion), emphasizing both symbolically grounded annotations and human-curated lineage traces to assess proposal plausibility within evolving research populations.
Who it's for and trade-offs
Great fit if you want to measure whether an LLM can (1) infer what components a paper inherited or dropped, (2) explain evolutionary changes between works, or (3) generate proposals that plausibly extend a given research lineage. Useful for researchers building evaluation suites, model developers diagnosing compositional weaknesses, and meta-science studies of idea diffusion. Look elsewhere if you need fully automated, large-scale extraction from raw corpora (IG-Bench relies on curated Idea Genome objects and manual lineage annotation) or if your focus is low-level code synthesis rather than high-level conceptual inheritance.
Where it fits
IG-Bench complements standard NLP and generation benchmarks by targeting the lineage dimension of scientific progress—it evaluates whether models can reason about inheritance semantics and produce descendants that meaningfully belong to a population, rather than only optimizing fluency or task-specific metrics. As such, it is a diagnostic resource for research on controllable, explainable scientific generation and for improving models' compositional abstraction over idea components.