Autonomous Scientific Discovery via Iterative Meta-Reflection

Key points

• Problem: Most existing LLM discovery systems treat each hypothesis-generation step independently, so they lack a step that surveys what has already been found and reasons about what remains unknown. Classical structure learning can only express edges between variable pairs, and full-pipeline AI scientist systems need initial research objectives or seed hypotheses to guide their search.
• Input and output: By default the system receives only tabular observation data (optionally with images) and no research question, and outputs a list of validated discoveries, each with a natural-language statement, test code and statistical evidence. This is a method paper that also builds a new benchmark, iNatDisco, from iNaturalist observations. iNatDisco-800 has 800 observations of eight species with nine literature patterns, and iNatDisco-50K has 50,000 observations of 9,776 species with 12 patterns.
• Method: In PROPOSE, an LLM agent reads a data summary, the current claims and the latest guidance, and outputs a hypothesis plus Python code that tests it. EVALUATE uses seven statistical tools, including correlation, group difference, prediction, clustering, stratified re-tests and VLM (vision-language model) visual attribute tests, tunes the code on a training split, runs it once on a held-out split, and accepts a claim only if both splits show an effect size of at least 0.2 and p of at most 0.05. REFLECT analyzes accepted and rejected claims together and returns guidance on gaps, recurring confounds and compound hypotheses; in the experiments it ran every five iterations.
• Implementation: The main text names Claude Sonnet 4.6 as the default LLM, while the appendix model comparison and baseline description name Claude Sonnet 4.5; the same base LLM also serves as the VLM for reading images, except the DeepSeek V4 Pro run in the model comparison, which was text only. The default setting runs 100 iterations per experiment three times for mean and standard deviation, the counterfactual experiment used 50 iterations, and the iteration-scaling experiment ranged from 10 to 100 iterations. All experiments took about 86 completed runs, about 8,250 API calls, an estimated cost of about 220 US dollars at Sonnet pricing and about 72 hours of wall-clock time.
• Results: On iNatDisco-800, DiscoPER recovered 8 of 9 literature patterns with a 72.7% support rate, while HeurekaBench-like and ExperiGen-like baselines re-implemented in the same framework found 3 and LLM-assisted classical causal discovery found at most 1. On iNatDisco-50K it found 8 of 12 with a 74.2% support rate, against 6 for the version without REFLECT. On the counterfactual iNatDisco-800-CF data, where five known ecological relationships were reversed in the tabular data, most proposed hypotheses were rejected by held-out validation and the accepted discoveries followed the modified data instead of the LLM's prior knowledge.
• Limits: The authors state that DiscoPER can only find what the input data supports and that citizen-science data carries known biases, so human verification of any proposed discovery is essential. Valid patterns outside the iNatDisco answer list can exist; recall is judged by an LLM judge (Claude Sonnet 4.5) that counts a partial match (score 1) or better as a discovery, and when the authors manually inspected all exact-match (score 2) judgments in the primary experiments (iNatDisco-800, iNatDisco-50K), 95% agreed. On a synthetic image benchmark it found 3 of 8 patterns, and the authors identify statistical validation of visual features as the bottleneck.

Paper links

External research summaries. These are not HDATF publications or measured product results.