AI Research Agents Narrow Scientific Exploration

Key points

• Problem: The prompts and reasoning procedures of AI research agent frameworks often instruct agents to produce novel, high-impact, unconventional ideas, yet the ability to generate ideas at scale does not necessarily imply broader scientific exploration. Existing evaluations mainly assess whether individual ideas are interesting, novel, feasible, or executable, and reveal much less about how repeated AI-assisted ideation shapes the broader landscape of scientific exploration.
• Input and output: In each run, one agent framework paired with one LLM receives a seed set of five papers from the same research area (one anchor paper and four related papers, titles and abstracts only) and produces one research idea with a research question and methods. The authors compare the collected ideas with human-authored papers from the same areas and with follow-on human papers that cite the seed papers, using four measures. This is a large-scale empirical study that analyzes 219,655 ideas.
• Method: Research areas are built by grouping papers by how much they cite the same prior literature (bibliographic coupling). The four measures are exploration breadth (the mean pairwise cosine distance among ideas in the same area), exploration distance (the cosine distance from an idea to the centroid of its five seed papers), frontier alignment (the share of the top 10% most frequent keywords in the next year's human papers that the ideas cover), and potential scientific impact (the mean normalized citation score, relative to papers of the same area and year, of the 20 semantically nearest human papers). Ideas and human papers are brought into the same format by having Gemma-4-31B-IT extract research questions, methods, and keywords, and are then embedded with Qwen3-Embedding-4B.
• Implementation: From a Semantic Scholar index of 149,405,218 papers, the authors drew 155 research areas in 12 broad fields and 11,520 seed sets, and ran five frameworks (Zero-shot, AIScientist, ResearchAgent, AgentLaboratory, and Co-Scientist) with four open-weight LLMs (Gemma-4-31B-IT, Llama-3.1-8B, Hermes-4-14B, and Qwen3-35B-A3B) for 230,400 runs. Because of budget limits, GPT-5.4 ran only on randomly sampled 2022 seed sets for 2,400 runs, and of the 232,800 runs in total, 219,655 valid ideas (94.4%) were analyzed. All frameworks except Zero-shot could search a locally deployed Semantic Scholar database restricted to papers available when the seed papers were published, AIScientist was run only through its ideation stage, and Co-Scientist was implemented from its published description.
• Results: Exploration breadth within the same research area was 0.554 for AI ideas against 0.599 for human papers, 7.5% lower, and exploration distance from the seed literature was 0.322 for AI ideas against 0.410 for follow-on human papers. AI ideas covered 28.5% of next-year frontier keywords against 36.5% for follow-on human papers, and their potential impact score was 0.387 against 0.492, 21.3% lower, with GPT-5.4 the only LLM whose ideas were comparable to human research on impact (0.472 against 0.464). Only 10.5% of AI ideas contained research questions absent from the seed literature, while 90.4% introduced new methods.
• Limits: Because of budget limits GPT-5.4 ran only 2,400 times on part of the 2022 seed sets, and the seed context was set to five papers because most frameworks are constrained by the context windows of current LLMs, with only titles and abstracts included. Information leakage from LLM pretraining on papers published after the seed literature was mitigated by restricting retrieval to papers available at the time, and since AI ideas still showed lower frontier alignment, the authors consider the reported differences likely conservative. Potential impact is a proxy based on the citations of semantically nearby human papers, and in a leave-one-out check its Spearman correlation with a target paper's own citation score was 0.155, which the authors note explains only part of the variation in citation outcomes.

Paper links

External research summaries. These are not HDATF publications or measured product results.