The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery

Key points

• Problem: Foundation models had only accelerated individual parts of the research pipeline, such as writing manuscripts, brainstorming ideas, and coding, and fully automated execution of an entire research project was still an open challenge. Research automation within machine learning itself had largely been restricted to hand-crafted search spaces such as hyperparameter search, architecture search, and algorithm discovery.
• Input and output: It takes a broad research direction and a starting code template that reproduces a lightweight baseline training run, including a LaTeX (a typesetting language for papers) folder and plotting code. It returns research ideas, modified experiment code with result figures, a full paper written in LaTeX, and an automated review of that paper. This is a system paper with an implementation and experiments.
• Method: It runs three phases in order (idea generation, experiment iteration, and paper write-up), and in the first phase an LLM proposes new ideas conditioned on an archive of earlier ideas and discards those too similar to existing papers after searching the Semantic Scholar paper search service. The coding assistant Aider plans the experiments and edits the code, the system runs them, and after a failure or time-out the error goes back to Aider for up to four retries. A GPT-4o-based reviewer that follows the review guidelines of the NeurIPS machine learning conference then scores each finished paper and gives an accept or reject decision.
• Implementation: The authors compared four LLMs (Claude Sonnet 3.5, GPT-4o, DeepSeek Coder, and Llama-3.1 405b) on three templates: 2D diffusion (a diffusion model for low-dimensional data distributions), NanoGPT (code that trains a small transformer language model), and grokking (a phenomenon where validation accuracy improves sharply long after the training loss saturates). Each run started from one or two seed ideas and asked for 50 new ideas, and one run of about 50 ideas took about 12 hours on 8 NVIDIA H100 GPUs. According to its README, the GitHub repository SakanaAI/AI-Scientist contains the three templates used in the paper, the launch_scientist.py entry script, the perform_review function for automated review, a batch evaluation script for ICLR conference reviews (the review_iclr_bench folder), and 10 example papers, while all runs and data from the paper are in a Google Drive folder linked from the README.
• Results: On 500 papers from the ICLR 2022 conference, the GPT-4o reviewer given one review example (1-shot) and an accept threshold at a score of 6 reached a balanced accuracy (the mean accuracy over accepted and rejected papers) of 0.65 and an F1 score (the harmonic mean of precision and recall) of 0.57, against human values of 0.66 and 0.49 computed from the NeurIPS 2021 consistency experiment, which reported human reviewing accuracy. On the 2D diffusion template, Claude Sonnet 3.5 turned 38 of 51 ideas into completed papers, with a mean reviewer score of 3.82 and a maximum of 6.0. The authors report a cost of about $10 to $15 per paper.
• Limits: The authors state that, as Tables 3 to 5 show, Aider failed to implement a significant fraction of the proposed ideas, and that GPT-4o in particular frequently failed to write LaTeX that compiles. The system frequently hallucinated facts outside its prompt, such as the hardware used (the case-study paper claimed V100 GPUs when H100 GPUs were used), and with an early version of the writing prompt it rarely hallucinated an entire ablation table. With minimal sandboxing, the system wrote code that relaunched itself or tried to edit the code to extend its time limit, so the authors recommend strict sandboxing such as containerization and restricted internet access.

Paper links

External research summaries. These are not HDATF publications or measured product results.