Auto Research with Specialist Agents Develops Effective and Non-Trivial Training Recipes

Key points

• Problem: Prior work covers repository-editing agents, ML experiment agents, and evaluator-driven discovery separately, and evaluator-driven search still targets algorithms, infrastructure, or fixed parameter and architecture spaces. The authors ask whether a propose, measure, and revise loop driven by external measurement can run over full real training recipes without human intervention, and whether its trajectory can be kept as an auditable record.
• Input and output: This is a system paper that builds a specialist-agent harness and runs it in three environments. The input is a training-recipe environment that fixes the editable files, the scored metric, legal failures, and the evaluator, plus a public starting recipe: Parameter Golf scores validation bpb (bits per byte, lower is better), NanoChat-D12 scores CORE (higher is better), and CIFAR-10 Airbench96 scores training time at 0.96 accuracy or above. The output is a trajectory that keeps, for every trial, the hypothesis, the executable code edit, the evaluator-measured score and status, and failure labels, along with the improved final recipes.
• Method: An agent reads the shared lineage (the cross-trial record of hypotheses, diffs, scores, runtimes, statuses, and crash summaries), proposes a hypothesis, edits the code, and submits a trial; an external evaluator records the score and status, and the next agent receives that result. Specialist roles split the recipe surface, with 10 in Parameter Golf, including architecture, optimization, quantization, tokenizer, and test-time training, and 5 each in NanoChat-D12 and CIFAR-10 Airbench96. Scores are measured outside the editable recipe, which blocks reward hacking such as printing a better score or reporting a fake runtime.
• Implementation: All reported runs use Claude Opus 4.7 for every role, and each specialist session is one Claude Agent SDK call whose defaults are an 8000-token thinking budget and at most 200 tool-use turns. Agents use MCP (Model Context Protocol) tools such as submit_trial, syntax_check, size_project, and rebase_to; the Write tool is left out of the allowed list, and a hook blocks Bash write commands so Bash is read-only. The official repository README lists a task-agnostic harness (agent_core), the three environment packages, single-agent and generic-agent comparison packages, and frozen run records per experiment, and states that running it needs an eight-GPU node and an Anthropic API key.
• Results: Across 1,197 trials in three headline runs, Parameter Golf validation bpb fell from 1.0810 to 1.072210 (0.81%), NanoChat-D12 CORE rose from 0.1618 to 0.2244 (38.7%), and CIFAR-10 Airbench96 training time fell from 26.3560 s to 25.1464 s (4.59%) while keeping the 0.96 accuracy gate. In the first 200 Parameter Golf trials, the role swarm with lineage reached 1.073142 with 16 valid improvements, the 10 generic agents reached 1.074495 with 10, the single generalist reached 1.075384 with 14, and the run without lineage reached 1.077413 with 3. Removing lineage raised the share of trials that hit the evaluation-budget cap from 19.0% to 61.5%.
• Limits: The authors place the observed boundary at combining, transferring, and repairing known techniques, and they state that paradigm-level inventions such as a replacement for the original Transformer lie outside the recorded trials. The loop fits settings where failures come back as trusted, compact feedback and fits less well where evidence is subjective or hard to verify automatically. The compute upper bound is 4,000 H100-hours for 1,500 Parameter Golf trials and 2,400 H100-hours for 200 NanoChat-D12 trials, with setup and calibration compute outside these totals.

Paper links

External research summaries. These are not HDATF publications or measured product results.