The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvements

Key points

• Problem: Scientific findings are trusted when they can be reproduced, so automated science needs automated reproducibility: reimplementing an experiment from its description so that it reproduces the reported outcome. Prior reproducibility benchmarks evaluate on a wide set of often unrelated papers, so the authors build a benchmark that measures reproduction of a chain of cumulative research findings within one task, speeding up LLM training.
• Input and output: This is a benchmark paper. In each of the 19 tasks the agent receives the previous record's GPT-2 training script (train_gpt2.py), optionally with hints at three levels (pseudocode, a text description, or a paper-like mini-paper), and returns a new training script that reaches a FineWeb validation loss of 3.28 faster. The score is FSR (fraction of speedup recovered), the share of the human next record's training-time reduction that the agent recovers, and a record optimization setting without hints asks the agent to find its own improvements.
• Method: The authors also release a search scaffold, a program that calls an LLM iteratively to find a solution, that generalizes AIDE. Each node in the search tree is a solution holding the training script and a results file, and each step has the coding assistant Aider make diff-based code edits (implementation), runs the code on an 8xH100 node (execution), and summarizes the output in natural language (analysis). Varying the branching factors and debug probability gives five search variants, Flat (best-of-M), Tree, Forest, AIDE, and Multi-AIDE, which are compared under the same budget of search steps.
• Implementation: For each of the 19 records, DeepSeek-R1, o3-mini, Gemini-2.5-Pro, and Claude-3.7-Sonnet were run with five search variants, six hint regimes, and three seeds, for 6,840 agent runs in total; each solution had a runtime cap of 60 minutes, and an agent run took about 10 hours on average. The hints were drafted by R1, manually verified, and edited where necessary, and the search used N0 = 3, N = 3, pdebug = 0.5, Dmax = 5, and a budget of M = 20 nodes. Per its README, the official facebookresearch/llm-speedrunner repository releases the scaffold code (core), starter workspaces for the tasks, the hint data for each level, Slurm launch scripts for the paper's experiments, and the analysis notebooks.
• Results: Without hints, the speedup recovered by every agent averaged at most 20% of the human solutions. The best model, o3-mini, reached FSR 0.40 to 0.43 across the five search variants with pseudocode hints, and with all three hints combined it reached 0.46 on Multi-AIDE but only 0.24 to 0.39 on the other four variants, below its pseudocode-only scores. In a cumulative run where o3-mini built each record on its own earlier code, it recovered about 60% on the first task, dropped to about 20% on the next, and reproduced no speedup by the third record.
• Limits: The authors note that the hints are short, manually defined, and handed directly to the agent, leaving a more realistic setup where the agent retrieves external knowledge through function calling and memory for future work. Many records may have been published before the tested models' knowledge cut-off, so the models may have seen the solutions in training, and the authors note that separating memorization from generalization may become more necessary as models begin to saturate the benchmark. The benchmark mainly tests reproduction rather than innovation, and the authors argue that agents which generalize to future advances in machine learning as a whole will need harder tasks spanning multi-file codebases, metrics beyond training time, and distributed training beyond a single node.

Paper links

External research summaries. These are not HDATF publications or measured product results.