SoundnessBench: Can Your AI Scientist Really Tell Good Research Ideas from Bad Ones?

Key points

• Problem: Recent AI scientist frameworks can automate research from hypothesis generation to experiment execution, but few benchmarks directly test whether a model can judge the methodological soundness of a proposal before it is executed. Existing benchmarks such as MLE-Bench and PaperBench mainly score execution, and a few idea-evaluation benchmarks score novelty and impact. The authors argue that without a strong first-gate filter, agents risk automatically pursuing flawed hypotheses and wasting compute and time.
• Input and output: This is a benchmark paper. The evaluated model receives a research proposal (hypothesis and experiment design) with results removed and returns, in JSON, a high or low soundness label, a confidence from 1 to 5, and a step-by-step justification. Ground-truth labels come from reviewer soundness sub-scores at ICLR (the International Conference on Learning Representations): a mean of 3 or more is high, 2 or less is low, and middle scores are excluded.
• Method: The construction pipeline takes ICLR submissions from 2022 to 2026, drops desk rejects (papers rejected before review), and keeps only papers with mean reviewer confidence of at least 3 and a standard deviation of normalized soundness scores below 0.15. Gemini 2.5 Pro extracts the abstract, related work, risk factors, hypothesis, and experiment design from each PDF in near-verbatim form while removing experimental results, direct outcome claims, and acceptance cues. Each extracted proposal is split into atomic claims that are checked against source passages found with BM25 keyword ranking, candidates with a support ratio at or above a threshold of 0.7 are kept, and 66.93% of candidates in the authors' current pool passed this check.
• Implementation: The final benchmark holds 1,099 proposals (458 low, 641 high) across 16 ML subfields, selected after processing over 35,209 initial ICLR submissions and 137,940 reviews, and the first page of the paper points to this dataset on Hugging Face. During construction, GPT-5.4 decomposes atomic claims and Gemini-2.5-Flash verifies them. Of the 12 evaluated models, closed models run through provider APIs and open models run with vLLM (an LLM serving library) on two NVIDIA H200 GPUs, and unless otherwise noted all models use temperature 0.2 and max_tokens 8192.
• Results: Under the standard prompt, the 12 models rated low-soundness proposals as high 74.0% of the time on average, and the two models with the highest rate, LLaMA-3.3-70B and GPT-4o, did so for 98.0% and 94.5%. Switching to the tested strict prompt that defaults to low cut this mean rate from 74.0% to 19.9%, but mean high-soundness recall also dropped from 91.8% to 36.1% and mean Macro F1 (the unweighted mean of the two class F1 scores) fell from 54.9 to 49.3. On an ICLR 2026-only split evaluated with models whose training cutoffs precede the ICLR 2026 submission period, low-soundness proposals were still rated high 77.47% of the time on average, close to 73.88% for the same models on the full dataset.
• Limits: The authors state that the labels come from reviewers who saw the full paper and results, so they are imperfect proxies for judging a results-masked proposal. The scope is ML research from ICLR only, and because the source corpus is public, training-data contamination cannot be fully controlled, a concern that the ICLR 2026-only split and the identifier-removal analysis reduce. The human audit is a preliminary check of 60 proposals, and only two prompts, standard and strict, were tested.

Paper links

External research summaries. These are not HDATF publications or measured product results.