Generating Literature-Driven Scientific Theories at Scale

Key points

• Problem: Automated scientific discovery has focused on agents that generate experiments, while systems for higher-level activities such as theory building, which consolidates many empirical results into generalizable laws, remain underexplored. Prior work on law discovery has largely induced quantitative or qualitative laws from structured data rather than from the research literature.
• Input and output: It takes a user-written theory query and a corpus of papers and returns a set of theories. Each theory has a name and a description and holds tuples of a law (a qualitative relation or an equation), its scope (the conditions under which the law is expected to hold), and empirical evidence typically drawn from the input corpus. This is a system paper that implements THEORIZER and compares four generation conditions experimentally.
• Method: THEORIZER rewrites the theory query as a literature search query for PaperFinder from AI2 (the Allen Institute for Artificial Intelligence) to retrieve up to K relevant papers (100 in the experiments), takes open-access PDFs located through Semantic Scholar, converts them to full text with an OCR (optical character recognition) based pipeline, and, if fewer than K papers come back from the initial search, backfills the corpus with relevant papers referenced in the retrieved full texts. It then builds an extraction schema for the query, has an inexpensive extraction model fill the schema for each paper as JSON records, and has the generation model synthesize theories from the pooled records, after which a self-reflection step refines each theory for internal consistency, evidence attribution, and specificity. The four compared conditions pair literature-supported generation, which uses this evidence, and parametric generation, which uses only the model's internal knowledge, with an accuracy-focused or a novelty-focused objective.
• Implementation: Generation uses GPT-4.1, whose reported knowledge cutoff is June 2024, per-paper evidence extraction uses GPT-5-mini, and evaluation mainly uses Claude Sonnet 4.5. From 50 randomly sampled papers at NLP (natural language processing) and AI venues (ACL 2023, EMNLP 2023, AAAI 2023, NeurIPS 2024), one general and one specific query were generated automatically per paper, and the resulting 100 queries drew on 13,744 papers to produce 2,856 theories, and evaluation was done per law rather than per theory, with 790 to 1187 laws per condition in the Table 1 LLM-as-a-judge evaluation. According to the README, the src folder of the official GitHub repository allenai/asta-theorizer holds the web interface and API server, the core prompts, and four evaluation scripts, the example-theories folder holds a small toy set and the theory data from the paper, and literature-supported generation needs a separately installed local copy of Asta PaperFinder.
• Results: In the Table 1 LLM-as-a-judge evaluation (an LLM acting as the grader), where Claude Sonnet 4.5 rated each law on a 1 to 10 scale, accuracy-focused literature-supported laws scored higher than parametric ones on empirical support (consistency with existing evidence), 5.8 versus 3.9, and on specificity, 6.5 versus 5.3, while the novelty score was 5.8 for both. In the Table 2 backtesting against 4,554 papers published in the six months from July 2025, precision was similarly high in the accuracy-focused condition, 0.88 for parametric and 0.90 for literature-supported laws, and 0.34 and 0.61 in the novelty-focused condition. The authors note that the literature-supported method is almost 7 times as expensive as a simple LLM approach, and the Table 4 estimate of LLM API costs is $0.26 per theory for literature-supported and $0.04 for parametric generation.
• Limits: Backtesting has low recall: in Table 2 the share of laws with at least one recent paper that tests their predictions was 0.45 to 0.51 in the accuracy-focused condition and 0.04 for parametric and 0.16 for literature-supported laws in the novelty-focused condition, and the authors note that the literature's positive-result bias, under which negative results often go unreported, may make contradicting evidence harder to find. The qualified novelty analysis of Table 3, separate from the Table 1 LLM-as-a-judge scores, measured the shares of seven novelty types against the 66 to 71 papers retrieved on average for each theory query instead of the full literature, on 100 randomly selected laws per condition, and in this analysis literature-supported laws were generally rated as less novel than parametric ones. The authors state that generation is currently limited to fields whose literature is mainly open access and downloadable automatically, such as AI and NLP, and that generation and especially evaluation are currently somewhat expensive, which limits scaling.

Paper links

External research summaries. These are not HDATF publications or measured product results.