Why LLMs Aren't Scientists Yet: Lessons from Four Autonomous Research Attempts

Key points

• Problem: The authors ask whether state-of-the-art reasoning LLMs can go from a research idea to a paper with a high degree of autonomy, minimal code scaffolding and the most basic tools. Various AI Scientist systems predefine domain-specific workflows or frame the problem in a system-specific way, Sakana's tree-search approach would have required complex meta-orchestration, and AlphaEvolve requires a verification metric set in advance by a human expert. The scope was limited to machine learning research because its experiments can be done entirely digitally.
• Input and output: The system takes two papers from one subdomain, generates an idea, and aims to move through hypotheses, an experiment plan, code and experiment outputs to an ML paper manuscript. The paper is a case study of four attempts with the authors' own system: starting from more than 135 papers in three domains (world models, multi-agent RL, AI safety and alignment), four ideas were selected for end-to-end attempts.
• Method: Six agents split the research stages: Idea Generation, Hypotheses Generation, Experiments Planning, Experimental Output Evaluation, Revision and Paper Outlining. Claude Code takes the plan.md and agent.md files from the Experiments Planning Agent, writes the code and runs it on Modal (a cloud execution platform); every artifact stays as a file in a per-idea repository, and each agent receives that repository state in its prompt and works with four tools: read file, write file, list files and llm search. The Experimental Output Evaluation Agent primarily checks hypothesis implementation fidelity and statistical validity, and when it signals failure the Revision Agent chooses between revising the idea, revising the hypotheses suite or requesting LLM mentor feedback.
• Implementation: All six agents used Gemini 2.5 Pro, which the authors chose primarily for its long context, and Claude Code handled code implementation and paper writing with Claude Opus 4.1 and Claude Sonnet 4. Each domain started from about 45 to 50 papers and produced up to 15 ideas, which were shortlisted with four zero-shot reviewer prompts run on Claude Opus 4.1; the first three ideas tried were chosen with feedback from the seed papers' authors, and the last one through an internal review informed by the earlier attempts. Per the official README, the repository holds four idea folders (three failures, one success) and prompts for idea generation, hypotheses generation, experiment planning and paper creation, and the authors state they released only prompts and selected outputs instead of the full system architecture.
• Results: Three of the four attempts failed at the execution or evaluation stage, and only AS-1 (semantic entropy for jailbreak detection) was accepted at Agents4Science 2025, a venue requiring AI first authors that accepted 48 of 254 valid submissions. The six failure modes the authors list are falling back to training data defaults such as outdated libraries, implementation drift, memory and context loss over long tasks, overexcitement, insufficient domain intelligence and weak scientific taste; for example, in WM-1, which tried to add differentiable tree search to a stochastic world model, Claude Code hit time limits and swapped the tree search for an actor-critic approach, and the experiment ran with one seed. From these they propose four design principles: Start Abstract, Ground Later; Verify Everything; Plan For Failure and Recovery; and Log Everything.
• Limits: The authors state that the scope is limited to ML research run entirely in software, that four ideas across three subdomains were each pursued with a single implementation instead of a best-of-N approach, and that because conference submission was the main goal, design changes were not recorded as systematic ablations, so the changes behind one success and three failures are hard to isolate. The failure modes come from qualitative observation, and the release covers prompts and selected outputs instead of the full system architecture. AS-1's experimental design, execution and writing were rated at least 95% AI, yet humans intervened in idea review, paper writing and meta-prompting during execution, and most failed experiments were stopped by human decision.

Paper links

External research summaries. These are not HDATF publications or measured product results.