The More You Automate, the Less You See: Hidden Pitfalls of AI Scientist Systems

Key points

• Problem: AI scientist systems run the research workflow from hypothesis generation through experiments to paper writing, but the authors note that their internal workflow has not been closely examined, which risks flaws that undermine the integrity, reliability and trustworthiness of their outputs. Diagnosis is also hard: with any existing dataset or task, web-scale pretraining makes data contamination almost inevitable, and each failure mode needs conditions that isolate it from confounding factors.
• Input and output: The authors run two open-source AI scientist systems on SPR task descriptions with local datasets and loading scripts, and record the benchmarks they choose, the accuracies they report, the metrics they pick and the candidates their reward functions select to judge each failure. A separate LLM-based classifier takes the task description, the final paper, generated code, execution logs and pitfall-specific detection rules and returns a Yes or No detection with an explanation. This is an empirical study that diagnoses existing systems with controlled experiments and also proposes the diagnostic task and a detection method.
• Method: SPR (Symbolic Pattern Reasoning) is a synthetic classification task where sequences of tokens built from 4 shapes and 4 colors are labeled accept or reject by whether they satisfy k hidden conditions on shape count, color position, parity and order. The benchmark-selection test offers 20 SPR datasets in a strict difficulty ladder, each raising one of rule complexity, vocabulary size or sequence length, under random five-letter names and random order, and asks the system to pick 4; the data-leakage test flips 20% or 30% of labels and treats test accuracy significantly above the theoretical upper bound (1 minus the noise level) as a leakage signal. The metric test adds noise so that only one of SWA (Shape-Weighted Accuracy) and CWA (Color-Weighted Accuracy) drops on the test set, and the post-hoc test inverts only the test performance of five candidates taken from logs to see whether the reward function changes its pick.
• Implementation: Both systems ran with their default LLM API settings, except that the code model of The AI Scientist v2 was switched from Claude 3.5 Sonnet, deprecated as of August 13, 2025, to OpenAI's O3-mini. For benchmark selection Agent Laboratory was stopped at the data preparation stage and run 1,000 times each with and without SOTA (state of the art) figures and The AI Scientist v2 generated 1,000 ideas each without running experiments, and post-hoc selection was measured with 200 trials per condition from 20 research projects run 10 times each. The audit classifier used gemini-2.5-flash-preview-05-20 and was evaluated on 20 cases per pitfall type over five types, half positive and half negative, and per the official README the repository holds the research generated by both systems, the SPR task descriptions and datasets, and the detection script (detect_pitfalls.py).
• Results: In benchmark selection Agent Laboratory picked the first four listed benchmarks in 82.4% of 945 runs with explicit selections when SOTA figures were given and in 79.6% of 927 runs without them, following list position over difficulty, while 47.1% of the benchmarks The AI Scientist v2 picked with SOTA figures were easy ones (18.0% without). No system peeked at test data, but Agent Laboratory used a subset of the data (16th run) or a synthetic dataset (11th run) without saying so in the paper, The AI Scientist v2 subsampled or synthesized datasets in runs 3, 7, 8, 11 and 13 and some test accuracies exceeded the provided SOTA figures, and with SWA listed first Agent Laboratory chose only SWA as its metric in all 20 runs. With test performance inverted, the share of picks going to the best train and validation candidate fell from 78.5% to 43.5% for Agent Laboratory and from 82.0% to 31.5% for The AI Scientist v2, and the LLM audit's overall accuracy rose from 55.0% (F1 0.51) with the paper alone to 82.0% (F1 0.81) with trace logs and code.
• Limits: The authors focus on four failure modes that do not exhaust other forms of scientific malpractice or system failure, and they leave reward hacking of LLM-based peer review by AI scientists outside this study. The detection techniques primarily rely on controlled setups and observable behavior, so they may not generalize to more nuanced or adversarial misuse or to flaws never reflected in logs, and with only two systems evaluated, extrapolating to other systems and to domains beyond ML calls for caution. The detection evaluation used 20 cases per pitfall type, positives for pitfalls that were rare or absent in natural runs were created with controlled perturbations of the workflow, and data leakage was injected by merging test samples into the training set.

Paper links

External research summaries. These are not HDATF publications or measured product results.