Evaluating Sakana's AI Scientist: Bold Claims, Mixed Results, and a Promising Future?

Key points

• Problem: Sakana promoted the AI Scientist as automating the entire research lifecycle, from idea generation through experiment design and execution, analysis and paper writing to review, and it drew wide attention, but to the authors' knowledge a comprehensive independent assessment based on direct experimentation had yet to be conducted. The authors point out that the most positive reviewers relied solely on Sakana's information in place of testing the system themselves.
• Input and output: This is an independent evaluation case study that installs and runs the AI Scientist, followed by recommendations (a position part) for the IR (Information Retrieval) community. The input is a template built by the authors: a prompt on Green Recommender Systems, experiment.py and plot.py that train FunkSVD on an 80/10/10 split of MovieLens-100k and evaluate it with RMSE (root mean squared error), two seed ideas (adaptive learning rates for SGD (stochastic gradient descent) and e-fold cross-validation), and Sakana's default LaTeX templates. The AI Scientist produced 10 ideas with novelty decisions, experiment code edited by the LLM-based coding assistant Aider, manuscripts and its own reviews, which the authors counted stage by stage and checked against the literature.
• Method: The authors measured each stage separately. For ideas they checked whether the novelty check, which queries the Semantic Scholar API for up to 10 results per query over up to 10 iterations, filters out known techniques; for experiments they counted the characters changed and whether runs succeeded as Aider edited experiment.py up to five times. For manuscripts they examined length, number and age of references, structural errors and numerical errors, they applied the review function to their 7 generated manuscripts and to 10 human-written OpenReview papers (5 accepted, 5 rejected) and compared it with human decisions, and they recorded cost and human effort.
• Implementation: Setup, coding and experiments were done by the third author, a third-year computer science undergraduate with basic ML knowledge and strong Python skills, and the foundation LLM was gpt-4o-2024-05-13. The main experiments ran FunkSVD on CPU on a university computing cluster, and the authors state that they released their own codebase for replicating the experiments with a Singularity container definition file. The linked GitHub repository SakanaAI/AI-Scientist is the code of the evaluated AI Scientist; per its README it holds NanoGPT, 2D Diffusion and Grokking templates, the paper-generation script launch_scientist.py and an LLM paper-review function, and it assumes Linux with NVIDIA GPUs using CUDA (NVIDIA's GPU computing platform) and PyTorch.
• Results: The AI Scientist judged all 10 generated ideas and both seed ideas novel, including known techniques such as micro-batching for SGD, 5 of the 12 ideas (42%) failed to run because of unresolved coding errors, and on a 6,260-character template Aider added on average 529 characters (+8%) in the first iteration and then 118, 83, 66 and 21. The 7 manuscripts had a median of 5 references, only 5 (14.7%) of all 34 references were from 2020 or later, 4 (57%) had structural problems such as missing or misplaced figures, incomplete sections like 'Conclusions Here' and duplicates, and 4 (57%) contained incorrect or hallucinated numbers. The review function rejected 9 of 10 human-written OpenReview papers, including 4 that human reviewers had accepted, and counting only OpenAI API costs the main experiments cost 42 dollars (6 dollars per manuscript on average) with about 3.5 hours of human effort per manuscript excluding setup.
• Limits: The authors note that they used one dataset (MovieLens-100k), one research domain (Green Recommender Systems), fixed seed ideas and one template, which may affect generalizability and reproducibility. Setup and experiments were done by one undergraduate, the only foundation LLM was gpt-4o-2024-05-13, and the system, whose pipeline is optimized for GPU deep learning, was tested with a simple CPU recommendation algorithm. The human-paper review test covered 10 papers, which the authors call indicative, and the claim of being about 3 to 11 times faster than human researchers rests on the authors' estimates of at least 20 to 40 hours for an undergraduate and 10 to 20 hours for an experienced researcher.

Paper links

External research summaries. These are not HDATF publications or measured product results.