RoboPhD: Evolving Diverse Complex Agents Under Tight Evaluation Budgets

Key points

• Problem: Systems such as GEPA and Autoresearch let LLMs iteratively improve prompts, code, and agent architectures from evaluation feedback, but the authors state that no systematic comparison under identical tasks, budgets, and evaluation infrastructure exists. Evaluations may be expensive (for ARC-AGI, even at a cap of $0.25 per problem, 1,500 evaluations can cost up to $375) or the pool of examples may be small, so how the evaluation budget is allocated determines which agents can be found, and existing methods split it between training and validation, where, per the authors, validation evaluations inform selection but do not help improve the next candidate.
• Input and output: This is a method paper with an implementation and experiments. RoboPhD takes an objective, task background, a seed agent (code and prompts), a pool of training examples, an evaluator that returns a score and diagnostics, and an evaluation budget, and searches for the agent with the highest expected score on unseen test examples. The output is an evolved agent program; on ARC-AGI, for example, a 22-line seed agent became a 1,013-line multi-strategy system.
• Method: Each iteration, 3 agents compete on 20 freshly sampled training examples and pairwise comparisons update Elo ratings (K = 32), so every evaluation goes to training-time competition with no validation split. The evolution AI, which designs the new agents (here Claude Code with Opus 4.6), receives the Elo rankings, comparative error reports that point to problems where agents diverge, and per-problem diagnostics that the agents print() themselves, creates a new agent, and refines it in the same session against data from a prior iteration (Deep Focus). Six diversity mechanisms, such as a 200-point Elo penalty for clones whose predictions match a competitor, are included, and a 1,500-evaluation budget yields about 21 iterations.
• Implementation: With a budget of 1,500 evaluations, the experiments cover four tasks: ARC-AGI (the solver, the model that solves the task, is Gemini 3.1 Flash Lite; 400 training and 400 test problems), Can't Be Late (a Python strategy class with no LLM that decides cloud spot instance use), Text2SQL on BIRD (solver Claude Haiku 4.5, co-evolving a database analysis script and an agent), and DocFinQA (solver GPT-4.1-mini with text-embedding-3-small, over 10-K annual reports filed with the SEC that average 123K words). The baselines, which receive the same diagnostics, are GEPA v0.1.1 with Opus 4.6 through the Claude API, an Autoresearch adaptation generalized to arbitrary artifacts with Opus 4.6 through Claude Code, and a King-of-the-Hill variant that pits the current champion against one new challenger each iteration. Per its README, the official andborth/RoboPhD repository releases the optimize_anything() API, per-domain examples that include the four tasks of the paper, and an engine option to choose GEPA or Autoresearch, and evolution requires the Claude Code command-line interface (CLI).
• Results: On the test sets, default RoboPhD beat both GEPA and Autoresearch on three of four tasks: ARC-AGI 65.8% versus 58.5% and 54.2%, Text2SQL 64.5% versus 60.4% and 60.7%, and DocFinQA 50.4% versus 40.0% and 48.2%. On Can't Be Late, Autoresearch scored best at -87.6 against -90.7 for RoboPhD, and the King-of-the-Hill variant beat the default on ARC-AGI with 67.0%. Turning off Deep Focus lowered all four scores (DocFinQA from 50.4% to 41.2%), and shrinking the validation set from 200 to 100 raised test scores in all eight GEPA and Autoresearch comparisons.
• Limits: On the simplest task, Can't Be Late (no LLM calls, solvable in under 90 lines), Autoresearch won, and default RoboPhD and the King-of-the-Hill variant were very close, which the authors say warrants further study. In the simulation by the authors, when three agents one accuracy point apart are compared on 20 problems, the best one ranks first only 45.0% of the time against a 33.3% random baseline, so selection relies on Elo accumulating weak signals over iterations, and non-determinism in LLM responses may cause variation across runs. During development, the Autoresearch adaptation found an exploit that read future spot availability from the internal state of the simulator and scored -85.4, so the authors redesigned the task API and stress careful sandboxing between the evaluator and the agent.

Paper links

External research summaries. These are not HDATF publications or measured product results.