Towards Execution-Grounded Automated AI Research

Key points

• Problem: Earlier studies found that research ideas from LLMs often look convincing but turn out ineffective once human researchers execute them. Grounding ideas in execution requires implementing and running any idea written in natural language automatically and at scale, and it was unclear whether this is feasible and whether LLMs can learn from the execution results.
• Input and output: The executor takes a batch of natural language ideas, generates a code diff for each idea and applies it to the baseline code, runs GPU experiments, and returns each idea's benchmark performance. The two environments are nanoGPT, which cuts the time for a 124M GPT-2 pre-trained on FineWeb to reach validation loss 3.28 on 8 H100 GPUs (during search, training time is fixed at 25 minutes and the reciprocal of the validation loss is the score), and GRPO, which trains Qwen2.5-Math-1.5B on MATH (a math problem dataset) to raise the best MATH validation accuracy within a fixed training time. This is an empirical study that builds the executor and then compares evolutionary search and RL on top of it.
• Method: In the executor, the Implementer gives the idea and the baseline code to a code LLM, gets 10 diffs in parallel, feeds the patch log back for up to 2 rounds of revision when a patch fails and uses the first diff that applies; the Scheduler assigns GPU resources and the Worker uploads experiment results to wandb. Evolutionary search fills the first epoch with new ideas and then starts with 50% exploitation, which feeds ideas that beat the baseline into the prompt to produce variants, and 50% exploration, which asks for new ideas different from earlier ones, shifting toward exploitation over epochs. RL trains Qwen3-30B-A3B with GRPO using the execution result (accuracy or 1/loss) as reward and gives reward 0 to ideas that fail to run.
• Implementation: Evolutionary search ran Claude-4.5-Opus, Claude-4.5-Sonnet and GPT-5 in the self-execution setting, where one model serves as both the ideator that proposes ideas and the executor that implements them, with 50 ideas per epoch on GRPO and 80 on nanoGPT, and RL used group sizes of 256 (GRPO) and 128 (nanoGPT), which means 256 or 1024 GPUs in parallel per batch. When each model implemented 50 of its own ideas, the nanoGPT completion rate was 96% for Claude-4.5-Opus, 90% for Claude-4.5-Sonnet and 64% for GPT-5, and with GPT-5 implementing the ideas of 4 other ideators it ranged from 42% to 78%. Per the README, the official GitHub repository holds the two environments (env/grpo, env/nanogpt) and the evolutionary search code (agent/), the 6 idea trajectories for three models on two environments are released separately as Hugging Face datasets, and users must implement their own executor API that launches the GPU jobs.
• Results: Within 10 epochs, evolutionary search with Claude-4.5-Sonnet found a recipe reaching 69.4% validation accuracy on GRPO (baseline 48.0%, best student in the Stanford CS336 graduate LLM class 68.8%), and with Claude-4.5-Opus a nanoGPT recipe reaching loss 3.28 in 19.7 minutes (baseline 35.9 minutes), while the best human record on the nanoGPT speedrun leaderboard is under 2.1 minutes. Comparing the first 3 epochs with GPT-5 on nanoGPT against best-of-N under the same sampling budget (picking the best of N samples, N of 80, 160 and 240), evolutionary search was ahead from epoch 1 onward, and among the three models only Claude-4.5-Opus showed a clear scaling curve while Claude-4.5-Sonnet and GPT-5 tended to saturate early. RL on Qwen3-30B-A3B raised the average accuracy on GRPO from 0.253 to 0.343 while the max reward fluctuated without a clear upward trend, and on nanoGPT the two common ideas grew from 51 at epoch 0 to 119 at epoch 68 out of the 128 ideas sampled per epoch.
• Limits: The authors state that whether ideas chosen at small scale still help at larger scale or on other datasets lies outside what their current procedure tests. The scope is bounded by the execution agent: many promising ideas failed to execute and add noise to the reward, and in most cases execution errors came from ideas that need complicated changes or installing external packages unsupported in the execution environment. Only effectiveness was used as reward, novelty assessment is outside the scope, and the attempts to stop the RL collapse (dynamic prompt, length reward, similarity penalty) showed no clear gain in early epochs and were stopped early.

Paper links

External research summaries. These are not HDATF publications or measured product results.