The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search

Key points

• Problem: The AI Scientist-v1 needed a human-written baseline code template for every new topic, and it refined experiments linearly, each step building directly on the previous one, so hypothesis exploration stayed shallow. The authors observed that v1's automated research often ended in short-sighted experimentation.
• Input and output: It takes a broad topic prompt such as a workshop description, writes research ideas as JSON (title, short hypothesis, related work, abstract, experiments, risk factors and limitations), and after experiment code and result figures returns a complete manuscript written in LaTeX. This is a system paper about an implemented and tested system.
• Method: During ideation an LLM checks novelty and related work with literature search tools such as Semantic Scholar. The experiment progress manager splits experimentation into four stages (preliminary investigation, hyperparameter tuning, research agenda execution, ablation) and passes the best node chosen by an LLM evaluator (an LLM that compares candidate nodes) to the next stage as its root. At each node an LLM writes a plan and Python code that is executed; nodes with execution errors or figure problems flagged by the VLM are marked buggy and picked for debugging with a predefined probability, and otherwise non-buggy nodes are picked best-first under LLM guidance and refined in parallel.
• Implementation: Code generation used Claude 3.5 Sonnet (v2) and the LLM and VLM feedback agents used GPT-4o; the node budget was 21 for stage 1 and 12 each for stages 2 to 4 with at most 1 hour of runtime per node, and in the authors' experience one paper usually took from several hours up to the 15-hour limit. Datasets were downloaded from the Hugging Face Hub with datasets.load_dataset whenever possible, which the authors call somewhat ad-hoc because support differs across dataset repositories, and the dataset for the pest detection paper was downloaded from Kaggle by the authors. Per the official GitHub README, the release includes an ideation script, a tree search launch script with its configuration file (bfts_config.yaml), and the tree search component is built on top of AIDE (an earlier system that combines LLM code generation with tree search).
• Results: The authors submitted 3 manuscripts produced by the system to the ICLR 2025 ICBINB workshop, where they went through blind review among the 43 total submissions. One paper on compositional regularization scored an average of 6.33 (6, 6 and 7), passed the acceptance threshold and ranked roughly in the top 45% of submissions, while the other two, on label noise and pest detection, were rejected. As agreed beforehand, the accepted paper was withdrawn after review.
• Limits: The acceptance was at workshop level for 1 of 3 papers, and the authors note that workshop acceptance rates (typically 60 to 80%) are higher than main conference tracks (20 to 30%). Humans picked 3 ideas from two rounds of about 20 ideas each generated by the system, chose the best manuscript among the seeds for each idea, and downloaded the pest detection dataset from Kaggle and cut it to 1/10 of its size, and the study asked whether at least one paper could pass review rather than measuring a success rate. In the authors' own review none of the 3 papers met main-track standards, and the accepted paper showed a wrong figure caption and a potential dataset overlap: regenerating the data 10 times with the generated code using addition and multiplication, digits 0 to 9, 1,000 training and 200 test samples gave on average about 57% of test examples overlapping the training set, while the original setup used digits 1 to 9.

Paper links

External research summaries. These are not HDATF publications or measured product results.