RepoMaster: Autonomous Exploration and Understanding of GitHub Repositories for Complex Task Solving

Key points

• Problem: The authors argue that when solving real tasks by adapting GitHub repositories, simply following the README seldom works, since READMEs can be vague, incomplete or even wrong. Repositories often have tangled files, classes and functions, and their information exceeds the context limits of most LLMs. The authors note that existing frameworks such as OpenHands and SWE-Agent provide no mechanism for grasping repository structure, tracing dependencies and selecting what enters the context.
• Input and output: This is an agent system paper with an implementation and released code. The input is a natural-language task such as removing scratches from an old photo, with input file paths and output requirements, and the repository is either found on GitHub by RepoMaster or fixed by the benchmark. The output is Python scripts written and run on top of that repository, plus result files saved to the required folder, such as a restored image or a Kaggle submission file.
• Method: RepoMaster first extracts the task intent and key entities, searches GitHub, assesses candidates by README and star count, and validates the top three. For the chosen repository it walks only the .py files with an AST (abstract syntax tree) to build a hierarchical code tree (HCT), a function call graph (FCG) and a module dependency graph (MDG), scores each module by summing six features (dependency centrality via personalized PageRank, complexity, usage, naming semantics, docstrings and Git history), adds normalized method counts and call frequencies to score classes and picks the top classes as core components, and builds the initial context from the README, summaries of key modules, core class code and a list of other paths. During execution it reads more through code view, dependency tracing and search tools, and when inputs exceed the per-interaction token limit it shrinks the context by keeping AST subtrees for code, a few highly relevant chunks for large or unstructured files such as .txt and .csv, and only the head and tail of execution logs.
• Implementation: RepoMaster is built on the multi-agent dialog platform AutoGen, caps each interaction at 8000 tokens, puts summaries of the top 20 modules and 10 key classes in the initial context, and splits unstructured text files into 1000-token chunks keeping the 4 most relevant. The compared LLMs are GPT-4o, Claude 3.5 Sonnet and DeepSeek V3, and the benchmarks are MLE-R, which turns 22 Kaggle tasks from MLE-Bench into repository-based tasks, and the authors' new GitTaskBench with 18 repositories and 54 tasks. The official repository README releases launcher.py for a web interface and the terminal, with backend modes such as a deep search agent and a repository agent, and states that it needs an OpenAI API key, plus Serper and Jina API keys for deep search.
• Results: On MLE-R, RepoMaster with Claude 3.5 Sonnet reached a 95.45% valid submission rate and a 27.27% medal rate, against 45.45% and 4.55% for OpenHands with the same model. On GitTaskBench with Claude 3.5 Sonnet, execution completion and task pass rates were 75.92% and 62.96% for RepoMaster versus 53.70% and 40.74% for OpenHands, with 154k versus 2883k tokens per task. In the GPT-4o ablation, removing all three mechanisms dropped the task pass rate from 40.74% to 24.07%.
• Limits: The paper has no separate limitations section, and its structural analysis covers only .py files. MLE-R repositories were picked in advance with RepoMaster's own search procedure because the other general agent frameworks have no automatic repository retrieval, GitTaskBench is the authors' own benchmark, built after adding downloadable resources and critical information missing from the original repositories, and the ablation used GPT-4o only. Appendix Table 5 reports execution completion and task pass rates of 48.15% and 24.07% for OpenHands with Claude 3.5 Sonnet on GitTaskBench, which differ from those in Table 2.

Paper links

External research summaries. These are not HDATF publications or measured product results.