Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills

Key points

• Problem: ML research agents combine a model with a harness that handles orchestration, memory, verification, and iterative refinement, but the operational knowledge of which methods and tools to use, when, and how (the know-how that separates knowing a method from making it work) stays outside the agent. This knowledge is abundant in repositories and papers, but it is written for human readers and is too large to load during a task. Today experts turn it into skills and tools by hand, so its cost scales with expert labor.
• Input and output: In creator mode, DisCo takes an anchor (the object that starts skill construction), either a source such as a repository or paper or one concrete task, and returns a verified skill graph (the skills distilled from one source plus their routing, dependency, and composition links) plus a construction record of the evidence used, the checks performed, and any unresolved gaps. In researcher mode, it takes a research task made of a problem, data, an environment (tools and budget), and a target, and produces artifacts such as code, models, experimental results, and reports. This is a system paper that builds the DisCo agent and a skill library and tests them on 4 benchmarks.
• Method: A skill has three layers: SKILL.md, read up front (goals, key concepts, tool usage, known failure modes), references/, opened only when needed (API documentation, algorithmic detail, parameter configurations), and scripts/, executable wrappers with defined inputs and outputs. Skill distillation (reworking declarative source knowledge into skills an agent can use) runs the same four stages whether the anchor is a source or a task: scope the capabilities, ground them in evidence, construct a candidate skill graph, and verify it while repairing it where possible. During research, a router narrows a request from area to family to repository skill graph, and the agent starts from the graph's entry skill and loads only the branch it needs.
• Implementation: Skill graphs for 1,000 repositories were built with GPT-5.5 and GPT-5.6-sol at xhigh reasoning effort, with an average allocation of about $40 per repository, and the snapshot organizes 5,353 skills into 20 areas and 178 capability families. For the experiments, the authors built 636 skills from 153 prior papers of the 20 target papers in PaperBench (a paper replication benchmark), one graph per competition for MLE-bench (an ML engineering competition benchmark), and one graph per benchmark for FrontierCS (open-ended computer science problems with unknown optima) and PassNet (graph compiler pass generation). According to its README, the GitHub repository holds the skill library (the skills folder), the DisCo CLI (command-line tool) distributed as an npm package (the cli folder), installation and workflow documentation, and example session exports.
• Results: With the Codex harness, the GPT-5.5 backbone, and the running budget held fixed and only skill access changed, the Any Medal score on all 75 MLE-bench competitions rose from 31.11% to 72.89% (134.3% relative), above 64.44%, the strongest of the public baselines that Table 1 copies from the official MLE-bench leaderboard. The average PaperBench replication score over 20 papers rose from 29.45 to 39.59, with gains on 18 of the 20 papers. The FrontierCS Agent Track score over 188 tasks rose from 70.63 to 77.14, and the main PassNet score (AS Score) over 200 samples rose from 1.343 to 1.5313.
• Limits: Two PaperBench papers scored lower with skills (sample-specific-masks from 57.11 to 52.04 and stay-on-topic from 32.31 to 27.79), and the authors point to a possible retrieval-precision failure in which retrieved skill content may occasionally distract from a strategy the base agent would find on its own. The comparison matches the running budget, and the one-time skill construction budget (for MLE-bench, up to 24 GPU-hours per task) is kept separate and excluded from both conditions. The 1,000 repositories are a curated snapshot of part of the commonly used ML software, chosen by open-source visibility (how well known they are in the open-source ecosystem) and practical use, with GitHub stars as one signal.

Paper links

External research summaries. These are not HDATF publications or measured product results.