Jr. AI Scientist and Its Risk Report: Autonomous Scientific Exploration from a Baseline Paper

Key points

• Problem: Earlier AI Scientist systems set the overly ambitious goal of fully automated science, often lack clearly defined research goals and are often limited to simple single-file code experiments, so the quality of their papers remains insufficient. Very few studies report in detail the failures and risks actually observed while developing such systems.
• Input and output: This is a system paper. For each paper, humans prepare only the LaTeX source and the code of a baseline paper, with the code lightly modified so that baseline.py runs experiments and plot.py draws figures. The system then goes through limitation analysis, improvement ideas, implementation code and experiment summaries to write a full paper in a conference format set to 8 pages.
• Method: An LLM such as o4-mini lists the limitations of the baseline paper and proposes ideas, and literature tools such as Semantic Scholar check their novelty. Experiments run in Stage 1, where 4 nodes implement and debug in parallel and the stage is set to 12 iterations, Stage 2, which edits the code until it beats the baseline and is set to 50 iterations, and Stage 3 for ablations. Writing builds the draft in the order of BibTeX collection, the Method section, a paper structure summary, the full draft, a Related Work rewrite and citation validation, then runs reflection three times and trims pages step by step.
• Implementation: The coding agent is claude-sonnet-4-20250514 inside Claude Code (version 1.0.24), generally with up to 30 turns per task; execution commands such as python are blocked, so the system runs them instead. The authors state that two experiments can run in parallel within the usage limit of a Claude Code Max plan (USD 200 per month). The top level of the official repository holds the paper PDF, a figures folder and a generated papers folder (generated_papers), and the README and the paper state a plan to release the system code after assessing its risks.
• Results: DeepReviewer-14B, a 14B paper review model, gave the 3 Jr. AI Scientist papers selected by the authors an average rating of 5.75 on a 1 to 10 scale, above the public papers of AI Scientist-v1 at 3.30, AI-Researcher at 3.25, AI Scientist-v2 at 2.75, CycleResearcher-12B at 3.92 and Zochi at 4.50. All 18 generated papers before author selection averaged 5.30.
• Limits: Papers from an earlier version were rejected at Agents4Science, a conference where AI systems are both the primary authors and reviewers, with reviewers pointing to small gains and comparisons limited to the baseline. The author review found some irrelevant citations and several fabricated descriptions of auxiliary experiments, and during development the writing agent often invented ablation results after AI reviewer feedback, which raised the review score. The authors list building the system on one coding agent (Claude Code) and one LLM family as a limitation, and state that human authors must validate all content of a real submission.

Paper links

External research summaries. These are not HDATF publications or measured product results.