Many Heads Are Better Than One: Improved Scientific Idea Generation by A LLM-Based Multi-Agent System

Key points

• Problem: Earlier work such as AI Scientist, ResearchTown, and HypoGen uses LLM agents to simulate scientific idea generation, but the authors note that these efforts either rely on a single agent or build multi-agent setups on an oversimplified collaboration framework and unrealistic data such as manually crafted profiles and synthetic collaboration networks, missing the dynamic relationships of real scientific teams. The authors ask whether simulating how real scientists form teams and collaborate yields more novel ideas, and how collaboration conditions such as team size, freshness, and research diversity affect novelty.
• Input and output: This is a system paper presenting the multi-agent system VirSci, whose input is a research ecosystem built from real academic data at a chosen time point. The ecosystem consists of a database of papers published before that point, a database of papers published after it, an author knowledge bank of scientist profiles, and an adjacency matrix of collaboration counts between scientists. The output is one abstract written by a team that a randomly sampled team leader assembles, with Introduction, Objective, Methods, Expected Results, and Conclusion sections.
• Method: The team leader forms a team by using the adjacency matrix, with 1 added to every collaboration count, as invitation probabilities, and the team then takes turns speaking for K turns in each step of topic discussion, idea generation, novelty assessment, and abstract generation. During discussions, agents can retrieve the author knowledge bank with RAG (retrieval-augmented generation) to hear from relevant scientists outside the team, which the authors call the Invitation Mechanism. Idea generation keeps the three ideas with the highest confidence, and in novelty assessment the agents compare each idea with past papers and select one by vote.
• Implementation: The system is built on the multi-agent framework AgentScope and calls GPT-4o through a public API and the open-weight LLaMA3.1-8b and LLaMA3.1-70b through Ollama, with LLaMA3.1-8b as the default. The data are the AMiner Computer Science dataset filtered to 156 authors and 178,197 papers and Open Academic Graph 3.1 filtered to 3,169 authors and 201,131 papers. The top level of the GitHub repository holds an agentscope-main folder that defines the customized scientist agent, a sci_platform folder with run.py as the main entry, and a preprocess_data folder, and the README points to Google Drive for the preprocessed AMiner Computer Science data.
• Results: With GPT-4o agents, VirSci scores a CD (Contemporary Dissimilarity, the average distance between the generated abstract and its 5 most similar contemporary abstracts, where smaller means closer to new work) of 0.34, a CI (Contemporary Impact, the average citation count of those abstracts) of 3.78, and a novelty score of 5.24 on a 10-point scale from ten PhD students, ahead of HypoGen (0.36, 3.10, 4.78) and AI Scientist (0.38, 3.22, 4.94), and with LLaMA3.1-8b and LLaMA3.1-70b agents it also leads on every metric in the table, including the LLM review score. In this comparison, NanoGPT is added as the initial discussion topic for HypoGen and VirSci to match AI Scientist's research direction, inference cost is matched with 4 members and 5 turns for VirSci and 12 turns for HypoGen, and paper retrieval for the metrics uses the Semantic Scholar API for all three systems. In the collaboration experiments comparing team sizes of 1, 4, 8, 10, and 100, the overall novelty metric ON (Overall Novelty) peaks at a team size of 8 and around 5 discussion turns, and also when about 50% of members are new collaborators.
• Limits: In large research communities multiple teams often work on related projects together or separately and researchers frequently join several teams at once, and the authors state that the system simplifies these complexities; they also list biases inherited from the underlying LLM and training data and the risk of plausible but incorrect scientific claims. Because of computational constraints, the evaluation focuses primarily on generated abstracts rather than full texts, and on 200 abstracts ON correlates with human ratings at a Pearson coefficient of 0.52. VirSci covers the idea generation phase, and an example abstract in the appendix includes a results sentence saying randomized controlled trials with 500 children reduced failure rates by 42%.

Paper links

External research summaries. These are not HDATF publications or measured product results.