AI-Researcher: Autonomous Scientific Innovation

Key points

• Problem: Systems exist for literature analysis or experiment design, but the authors argue they do not connect the full research workflow from hypothesis generation to publication-quality reporting. They also note that no standardized benchmark measures autonomous research across domains, which makes progress hard to track.
• Input and output: This is a system paper that presents AI-Researcher, a multi-agent system automating the full research workflow, together with Scientist-Bench, a benchmark for it. Scientist-Bench uses 22 papers from 2022 to 2024 as ground truth and gives the system 15 to 20 reference papers, a research instruction and datasets (Level-2 tasks omit the instruction). The system returns implementation code and a technical report covering background, method, experiments and results.
• Method: A Knowledge Acquisition Agent selects at least 5 GitHub repositories and gathers the matching arXiv LaTeX sources, then a Resource Analyst splits the idea into atomic concepts and maps each formula to its code. An Idea Generator proposes five directions and develops the most promising one, while a Code Agent and an Advisor Agent iterate implementation and feedback like a student and an advisor. A Documentation Agent writes the paper in three stages: outline, detailed writing, and checklist-based revision.
• Implementation: All analysis and execution run inside a Docker container, and prototypes are typically tested first on 1 to 2 epochs or small data subsets before full experiments. The paper review agent used for evaluation (an agent that compares two papers and scores them) rated the accepted paper higher in 65.62% to 90.62% of 32 ICLR (International Conference on Learning Representations) accept-reject pairs from 2021 to 2023, depending on the LLM judge. According to the README, the GitHub repository holds the research_agent and paper_agent code, the benchmark data and its collection process (benchmark and benchmark_collection folders), a Dockerfile, a gradio-based web GUI (graphical user interface) and example generated papers with code (examples folder), and the README explains how to pull the runtime Docker image (tjbtech1/airesearcher:v1) or build it from the Dockerfile.
• Results: With Claude-series models over the whole benchmark, the completeness rate (executable code within the inference budget) was 93.8%, and correctness, scored from 1 to 5 by a Judge Agent using an Advisor Agent's analysis, averaged 2.65. In Level-1 pairwise comparison of generated papers against the ground-truth papers (Table 2), five LLM judges rated on a -3 to 3 scale, and the share of generated papers scoring at least -1.0 ranged from 13.64% to 81.82% depending on the judge. In Level-2 without instructions (Table 3), the same share ranged from 50.00% to 100.00%.
• Limits: In the failure analysis, the authors report that generated research relied predominantly on conventional techniques and standard formulas with shallow theoretical analysis, and that in one Level-1 knowledge graph recommendation task it missed the Gumbel reparameterization the human researchers used. The current system carries information through the LLM context window and summaries of earlier stages in place of an external memory system, and the authors also note that LLM judges tend to overvalue presentation over substance. The experiments cover 22 papers in four AI domains (6 Level-2 tasks per Table 1), and the summary ranges in Sections 4.3 and 4.4 differ from the values in Tables 2 and 3.

Paper links

External research summaries. These are not HDATF publications or measured product results.