Are LLMs Ready for Scientific Discovery? A Capability-Oriented Benchmark for AI Scientists

Key points

• Problem: Existing benchmarks such as DataSciBench, KramaBench, and ScienceAgentBench mainly check whether an LLM can generate and execute analysis code or pipelines that reach ground-truth answers or task-specific success criteria. The authors argue that such evaluation can mistake successful execution for scientific validity and hides behind one aggregate score whether a model fails at basic description or at causal inference. Scientific analysis supports claims such as description, hypothesis exploration, statistical inference, prediction, causal estimation, and mechanistic explanation, each with different assumptions and validity criteria.
• Input and output: This is a benchmark paper. Each item consists of a domain, a task type, a dataset with metadata, a question, and a ground-truth answer, and the model reads the dataset and answers in a multiple-choice (MCQ) or open-ended (OEQ) format. Base models are evaluated zero-shot, every model returns a JSON response with reasoning and answer fields, only the answer is scored, and the reasoning is used for error annotation.
• Method: The authors turn the six-question framework of Leek and Peng into six capabilities (descriptive, exploratory, inferential, predictive, causal, mechanistic) and build items across five domains (biology, chemistry, environment, geography, physics). SDA-Synth, the synthetic part, uses scenarios proposed by GPT-4o to build a semantic DAG (directed acyclic graph) of variables with mechanism equations, adds noise and distractor columns to generate datasets, and keeps 50 (10 per domain) after execution replay, range checks, and expert review. Questions come from templates built by four agents (Taxonomist, Extractor, Generator, Verifier), answers come from a deterministic derivation procedure per template, and MCQ distractors come from four perturbation strategies on variables or intermediate computations.
• Scale and implementation: SDA-Real, the real-data part, has 527 items built from real datasets in seed resources such as KramaBench, BLADE (a benchmark for data-driven science agents), and QRData, chosen from 24 candidate datasets and benchmarks, and SDA-Synth keeps 6,000 items by taking 200 per domain and task pair from the 6,231 that passed validation. Validation removed 779 candidates, a manual audit of 307 templates and 2,000 sampled items filtered out 324 more, and SDA-Synth is split 60% for fine-tuning training and 40% for testing. The 15 evaluated LLMs include closed-source models such as GPT-5.4, Claude Sonnet 4.6, and Gemini 3.1 Pro, open-source models such as DeepSeek V3.2 and Qwen 3.5-397B-A17B, and two Llama-3.1-8B variants fine-tuned with LoRA (Low-Rank Adaptation) on the SDA-Synth training split.
• Results: On the SDA-Synth test split, the top OEQ accuracies were 58.33% for GPT-5.4, 56.88% for Gemini 3.1 Pro, and 55.79% for Claude Sonnet 4.6, and Llama-3.1-8B fine-tuned with the error-type objective approached this range at 55.33%. By task type, Descriptive tasks were easiest and accuracy dropped on Inferential and Causal tasks, and by domain the average OEQ accuracy across LLMs was highest for Environment at 46.9% and lowest for Biology, whose Predictive score was 22.5%. In the error analysis, which labels the first error in execution, the normalized Function Error rate peaked at 37.7% on Predictive tasks and the Relationship Error rate reached 32.1% on Mechanistic tasks.
• Limits: The authors state that the synthetic semantic graphs of SDA-Synth, though validated by domain experts, may not capture real experimental confounders and measurement artifacts, and that the MCQ format can induce superficial pattern matching in some models. Because the error taxonomy labels the first error in execution, it may undercount cascading failures. Because four-option MCQ has a 25% chance floor, MCQ and OEQ accuracies are not directly comparable, and the full SDA-Real results are deferred to supplementary material C.

Paper links

External research summaries. These are not HDATF publications or measured product results.