IdeaScientist: Orchestrating Agents for Grounded Scientific Ideation

Key points

• Problem: Existing autoresearch systems couple solution proposal with implementation, experimentation and writing, so it is hard to tell whether success comes from strong ideas or strong execution. A research proposal has no directly verifiable reward, and a single proposal-level rubric gives weak credit assignment (telling which stage earned the score) and invites reward hacking.
• Input and output: This is an agent system for research ideation. The input is a problem definition (the task input and desired output) plus a challenge that current solutions fail to meet, and the output is a research proposal in a fixed schema covering the problem definition, key novelty, related work, method, evaluation plan, falsifiable predictions and limitations. The proposal carries no measured results and cites prior work inline by database id.
• Method: An orchestrator keeps the research plan and calls three sub-agents: a gap finder identifies the shared limitation of closely related work, an innovator adapts a mechanism that solved a similar challenge in another field, and a report writer turns that intuition into a complete proposal. Each role is trained separately with asynchronous GRPO (Group Relative Policy Optimization, an RL method that compares rewards within each question's group of rollouts) against a reference artifact reconstructed from a human paper. The reward weights by 0.85 a rubric score gated by structural, task-compliance, citation and tool-validity checks (the mean of a reference-free quality score, a reference-grounded score and citation F1 for overlap in cited papers) and by 0.15 a rule-based compliance term.
• Implementation: The Svalbard Idea Vault holds 4.7 million papers (2.8 million, or 60%, with full text), namely arXiv papers from all subject areas published from 1986 through June 18, 2026 plus missing papers found one hop through the references of papers collected on or after January 1, 2026, decomposed into 2.77 million (problem definition, challenge, research gap, intuition, proposal) tuples. The 14,977 machine learning methodology papers published on or after January 1, 2026 were split into 14,500 training, 200 validation and 277 test papers, and one LoRA (Low-Rank Adaptation, a fine-tuning method that trains small adapters while keeping the original weights) adapter per role was trained on Qwen3.6-27B. Per the README, the GitHub repository holds the ideation harness, the vault construction code, the RL reward and the evaluation, the vault and the train, validation and test partition sit in a Hugging Face dataset, and the ablation arms, the human study and the LaTeX table builders are left out.
• Results: On the overall score (the mean of the novelty average and the proposal-quality average) given by a Qwen3.6-27B judge on the 277 test papers, IdeaScientist-Trained on Qwen3.6-27B reached 74.6%, 14.0% above DeepScientist (60.6%), the best open-source baseline on the same backbone, and 5.9% above the general-purpose agent Claude Code with Claude-4.8-Opus (68.7%) and 5.1% above Codex with GPT-5.4 (69.5%). Held-out validation rewards per role rose from 56.6% to 66.7% for the gap finder, from 58.8% to 69.5% for the innovator and from 78.6% to 89.1% for the report writer. The gains of IdeaScientist-Base came mainly from novelty, and its average-novelty margin over the strongest baseline on the same backbone peaked at 23.1% on Qwen3.6-27B.
• Limits: Evaluation happens before any experiment, as an LLM-judged match to directions that human papers took after the cutoff, and on the 50 proposals used to validate the judge, two human experts agreed at a Pearson r = 0.56. In the blind pairwise human study on 50 papers, DeepScientist was the only exception to the statistically significant wins of IdeaScientist-Trained, whose more frequent wins over IdeaScientist-Base were also not significant. Training and test papers are limited to machine learning methodology papers, and a title-only probe of 5 models on 10 post-cutoff papers found no recall, but the authors list residual model-memory contamination as a limitation of the retrospective evaluation.

Paper links

External research summaries. These are not HDATF publications or measured product results.