AIM: Agentic Idea Management for Automated Research

Key points

• Problem: In automated research, implementation and evaluation are expensive, so a limited experimental budget has to be allocated across possible approaches. Idea-driven search, which selects ideas first and delegates implementation to a solver, must organize a growing pool of ideas and evidence, select which ideas to implement, and keep the solver's code faithful to the selected idea. Existing methods typically rely on fixed rules such as UCB or MCTS (Monte Carlo Tree Search) or on LLM judgments without explicit estimates, and when the code does not realize the idea, its score and lessons are attributed to the wrong idea.
• Input and output: AIM takes one AutoLab task (a natural-language instruction, a containerized environment, a deliberately weak baseline codebase, and an evaluation script) together with a research brief generated from the task description using Claude Code. When the experimental budget is spent, it returns the highest-scoring executable solution among those evaluated as the final output. This is a system paper that implements the framework, tests it on 10 benchmark tasks, and adds a theoretical analysis of when idea-level search pays off.
• Method: Agentic Surrogate follows the surrogate role in Bayesian optimization, regrouping the whole idea pool into 2 to 5 semantic clusters at every iteration and producing ordinal estimates of promise for clusters and ideas. Agentic Acquisition uses two LLM calls to assign each branch an explore or exploit action at both the cluster and the idea level, sends the chosen ideas to parallel Solvers, and then generates new ideas from extracted lessons with one of four modes: score-guided refinement, cross-pollination, error-guided repair, or novel idea generation. Solution Auditor removes results flagged as trivial, task_mismatch, or reward_hacking from the history and lessons, and when idea_mismatch is the only issue it rewrites the idea to match the implementation and attaches the score to that idea.
• Implementation: All methods use Gemini-3.1-Pro-Preview as the backbone LLM, implementation is mainly done by the Gemini Deep Solver from ScientistOne, and the initial pool of about 10 ideas comes from the ScientistOne Ideator. The budget is at most 300 executions and 6 hours for System Optimization, 60 executions and 24 hours for Model Development, and 60 executions and 12 hours for CUDA, and the Resource Planner launches 5 Solver branches in the first iteration and then sets between 3 and 10 branches per iteration. The appendix gives pseudocode for the full pipeline and the prompt templates for every LLM-driven operator.
• Results: On the five AutoLab System Optimization tasks, AIM averaged 67.0%, 1.6 points above ScientistOne (65.4%), the strongest idea-driven baseline, and 3.7 points above AdaEvolve (63.3%), the strongest solution-driven baseline. On the five Model Development and CUDA tasks, AIM averaged 55.8%, 4.9 points above ScientistOne (50.9%), and the authors report that AIM reached ScientistOne's best score up to 3.1 times faster in wall-clock time. In the Flash Attention ablation, removing any single component scored below the full AIM's 90.5%, and removing the Agentic Surrogate caused the largest drop, to 85.9%.
• Limits: The authors state that the assumption behind the theoretical analysis, that the implementation faithfully realizes the selected idea, could be slightly idealized and that real solver agents may produce incomplete or incorrect implementations. AIM tends to need more executions to reach its best score and generally uses a larger token budget, and on the data selection task Data Select IFEval it scored 61.8, below AdaEvolve (82.7). The authors write that isolated containerized environments and the Solution Auditor reduce risk, that some risk remains, and that human review is needed before applying autonomous research systems to consequential scientific or engineering settings.

Paper links

External research summaries. These are not HDATF publications or measured product results.