Idea2Plan: Exploring AI-Powered Research Planning

Key points

• Problem: Scientists tend to produce more promising ideas than they can pursue, many remain unexplored due to time constraints, and turning an idea into a testable plan can take many days or even weeks. Recent work has focused on other stages such as idea generation, literature review, and experiment execution, and has treated research planning as an intermediate step without explicit evaluation (CodeScientist) or evaluated plans only on high-level dimensions such as clarity and novelty (MLR-Bench). Evaluation faces data contamination, since an LLM may have already seen the idea through a published paper, and the fact that several plans can be equally valid for the same idea.
• Input and output: This is a benchmark paper that contributes a task definition, an automatic benchmark construction pipeline, and a benchmark for validating LLM judges. The input is a research idea rewritten in the first person from the original paper's abstract with method names and experimental results removed, and the output is a research plan with five sections: Introduction, Key Literature, Methods, Initial Experimental Design, and Resources/Compliance/Ethical Considerations. An LLM judge answers section-wise yes/no rubric questions built from a reference plan extracted from the original paper, and the Planning Score is the share of questions satisfied in each section, averaged over the five sections.
• Method: From the ICML 2025 Spotlight and Oral papers, the authors drop papers whose arXiv versions predate the most recent training data cutoff of the tested LLMs (GPT-5) and randomly pick 200 of the remaining papers as the test set and 30 as the dev set. o4-mini (reasoning=high) extracts the idea from the abstract and the reference plan from the full paper, and rubric questions ask about high-level design choices that any reasonable plan should address instead of specific terms absent from the original idea. Plans are generated four ways: Naïve (a short prompt with section titles), 0-shot (general instructions added), 1-shot (one fixed example plan added), and a ReAct agent that can call an arXiv search tool and a read tool up to five times each.
• Implementation: The tested LLMs are GPT-4.1, GPT-4.1-mini, o4-mini, GPT-5, and GPT-5-mini, plus the open-source DeepSeek-V3, DeepSeek-R1, and Phi-4, with three runs per idea for each model and setting and the mean reported. The judge, o4-mini (reasoning=high), was chosen after comparing candidates on Idea2Plan JudgeEval, where two human annotators independently graded the rubric questions of 30 plans (88.4% agreement, Cohen's κ 0.768), and it grades each section of a plan separately. The search tool uses Bing Search restricted to arXiv and blocks papers published after January 30, 2025 (the ICML 2025 deadline), the target paper, and papers citing it, while the read tool returns an o4-mini summary in a fixed template of a paper fetched through the arXiv API.
• Results: On the test set, GPT-5 had the highest average Planning Score in all four settings (Naïve 62.0, 0-shot 63.0, 1-shot 63.2, ReAct 61.9), and GPT-5 and GPT-5-mini achieved pairwise win rates over 90% against the other models. The effect of ReAct varied by model: GPT-4.1 scored 40.8 with 1-shot and 47.4 with ReAct, while GPT-5 scored 63.2 and 61.9, and the authors report that models sometimes incorporated irrelevant retrieved details into plans. On an appendix benchmark of 100 ICML 2026 papers, standalone GPT-5.5 scored 62.0 and GPT-5.5 in the Codex harness with an MCP (Model Context Protocol) paper search tool scored 75.1, and the upper bound in Table 28 (a plan written with the original paper in hand) was 85.8.
• Limits: The plan template follows AI paper conventions and drops components an LLM cannot easily determine, such as timelines, logistics, or human expertise, and the Nature Mental Health benchmark cuts the template to four sections. Because the rubric measures coverage of key elements, longer outputs may gain an advantage, and GPT-4.1 and o4-mini scored higher when allowed longer outputs. All candidate judge models showed a leniency bias with more false positives than false negatives, and human checks of the extracted plans and rubrics covered 8 papers, one chosen by each of eight experts (section means above 4.0 on a 5-point scale).

Paper links

External research summaries. These are not HDATF publications or measured product results.