Most prior work builds procedural skills from agent trajectories or documentation, both of which have limits: trajectories require environment interactions and docs often lack executable evidence. This paper argues that source code is a complementary, high-density reservoir of procedural knowledge that needs no prior agent experience yet provides concrete implementation evidence.
Key Findings
- Code2Skill pipeline: selects code units (functions, methods, entry points), extracts typed skills (atomic, composite, recurring-pattern), challenges candidate records via source-body-blind reconstruction, and accepts records after source-aware comparison and deduplication. This pipeline emphasizes verifiability over mere summarization.
- Scale and artifact: applied to 19,769 popular, actively maintained GitHub repositories to produce CodeSkillBank with 1,006,822 accepted skill records, each annotated with workflows, boundaries, provenance, inputs/outputs, invariants, failure modes and source evidence.
- Empirical impact: across 72 protocol-matched evaluations (nine model settings, eight benchmarks), augmenting models with retrieved CodeSkillBank skills improved average performance by 11.7% over matched baselines and outperformed trajectory-derived skill banks on shared benchmarks. Skills synthesized from tested AI-generated code showed pass rates comparable to human-written code.
Who it's for and tradeoffs
Great fit if you need grounded, implementation-anchored procedural knowledge for coding agents, tool-enabled LLMs, or agent planners and want verifiable evidence tied to real repositories. Look elsewhere if your target behaviors are heavily environment-specific (hardware sensors, closed-system state) or depend on runtime interactions not captured in repository code; the approach abstracts from runtime contexts and may miss deployment-specific nuances.
Where it fits
This work sits between documentation-derived skill summarization and trajectory-derived imitation: it provides transferable procedural building blocks before agents accumulate interaction experience, and it scales with the growth of public (and AI-generated) code.
Methodological notes
The pipeline prioritizes automated verification (blind reconstruction + source-aware comparison) to limit hallucinated or under-specified skills. Evaluation focuses on protocol-matched downstream tasks and cross-benchmark comparisons to measure practical utility rather than only intrinsic metrics.