Most ML research agents combine a reasoning backbone with an orchestration harness, yet still fail because the field’s operational know‑how — the pragmatic, repository‑level details that make methods actually work — is missing from an agent’s context. This paper’s core insight is simple but consequential: extract and verify that know‑how into compact, discoverable skills so agents stop rediscovering the same engineering tricks and misconfigurations across tasks. The authors implement this idea end‑to‑end (DisCo) and produce a large, verified skill corpus (AREX‑Skill Library) to quantify the effect.
Key Findings
- Distillation at scale: task‑agnostic distillation across 1,000 widely used ML repositories yields the AREX‑Skill Library with 5,000+ verified skills organized into 20 areas and 178 capability families — so what? agents can hold many skills yet read only the few relevant to a task, keeping context small.
- Task‑oriented distillation: on‑demand skill creation tailors operational knowledge to a concrete research problem, reducing wasted trial‑and‑error during experiments.
- Verified impact: holding model backbone (GPT‑5.5), harness, and execution budget constant, adding skills raises scores by large margins on multiple benchmarks (e.g., +134.3% MLE‑bench, +34.4% PaperBench) — so what? the gains come from supplying execution‑ready context rather than larger models or more compute.
- Skill design: each skill packages a short SKILL.md (interface), references (substrate), and scripts (execution), enabling progressive disclosure and automated checks so agents can safely consume thousands of skills.
Who it's for and tradeoffs
Great fit if you build or evaluate autonomous research agents, maintain agent knowledge bases, or need repeatable, repository‑level operating context for ML experiments. The approach accelerates agent workflows and reduces wasted runs by encoding common APIs, pitfalls, and pipeline settings. Look elsewhere if your tasks require always‑fresh repository content without a maintenance pipeline: skills can go stale across releases, and safe ingestion requires auditing (security and supply‑chain concerns). Building and verifying large skill corpora also demands upfront engineering and verification budgets.
Where It Fits
This work sits between model improvements and orchestration engineering: instead of changing the reasoning backbone or the harness, it supplies the missing operational layer that maps methods and tools to runnable procedures. It complements retrieval/RAG approaches by packaging executable and verified operational context rather than raw docs.
How It Works (brief)
Distillation pipelines parse repos, extract candidate operational items, generate concise SKILL.md interfaces backed by references and scripts, and apply automated verification and repair. Distillation runs both ahead‑of‑time (task‑agnostic library) and on demand (task‑oriented skills), with a router that selects relevant skill graphs at runtime.