CycleResearcher: Improving Automated Research via Automated Review

Key points

• Problem: Agents for research automation predominantly rely on commercial LLMs, so the full research process cannot be effectively modeled as a policy optimization problem for reinforcement learning (RL). The authors argue that generating and refining research outputs that meet peer review standards remains largely unresolved, and that few efforts integrate the iterative feedback needed for academic soundness and novelty.
• Input and output: CycleResearcher takes a bib file containing all references and their abstracts, alternates between generating outlines and main text, and produces a complete LaTeX paper from motivation and idea through experimental design, simulated results, and conclusion. CycleReviewer takes a complete paper, generates strengths and weaknesses, soundness, presentation, and contribution scores, and an overall score from 1 to 10 for multiple simulated reviewers, and outputs their average as the final evaluation. This is a method paper that contributes a training framework, two datasets, and the trained models.
• Method: It mimics the research, review, and refinement cycle with iterative SimPO (Simple Preference Optimization, which learns from preference pairs with a length-normalized reward and no reference model), and to speed up training it has the model fabricate experimental results instead of running real experiments. It keeps only the reference sections of 4,152 recent machine learning papers from arXiv as the knowledge base, samples three papers from CycleResearcher at temperature 0.4, and builds preference pairs from the outputs with the highest and lowest average CycleReviewer scores. Training adds a negative log-likelihood (NLL) loss to the SimPO loss for stability, samples one-third of the full dataset each round to reduce overfitting, and repeats the process with each new model.
• Implementation: CycleReviewer was trained from Mistral-Large-2 with LoRA-GA (low-rank adaptation with gradient approximation) on an 8x H100 80G cluster for 12 epochs on Review-5k. CycleResearcher comes in three sizes, 12B, 72B, and 123B, trained from Mistral-Nemo-12B, Qwen2.5-Instruct-72B, and Mistral-Large-2 123B on 8x H100 GPUs with Research-14k for 12,000 steps. Review-5k holds 4,991 ICLR 2024 papers with over 16,000 reviewer comments, and Research-14k collected 14,911 papers accepted at major international machine learning conferences from 2022 to 2024, which after filtering were split into 12,696 training samples and 802 test samples.
• Results: On the Review-5k test set, where the average score of the other reviewers of the same paper stands in for the ground truth, CycleReviewer (123B) reached a Proxy MSE (mean squared error) of 1.43 and a Proxy MAE (mean absolute error) of 0.92, lower than 2.34 and 1.16 for individual human reviewers, and its decision accuracy was 74.24% against 75.40% for individual human reviewers. Scored by CycleReviewer, CycleResearcher-12B papers averaged 5.36 (accept rate 35.13%), above arXiv preprints at 5.24 and below conference accepted papers at 5.69, while AI Scientist papers generated with GPT-4o and Claude-3.5 averaged 4.31. In a human evaluation where three NLP experts each read 10 CycleResearcher-12B papers chosen with N=100 rejection sampling and 10 AI Scientist papers, the overall scores were 4.8 and 3.6, and the CycleResearcher score stayed below ICLR 2024 submissions at 5.5 and accepted papers at 6.4.
• Limits: The authors state that the experimental results in the papers generated in this work were fabricated by CycleResearcher, which limits the ability to produce verifiable and reproducible scientific knowledge, and the one paper with real experiments is in Appendix I, where AI Scientist implemented the experiments with GPT-o1. Human researchers or other agents are still needed to execute the experiments the models design, the current implementation is primarily optimized for machine learning research, and CycleResearcher's knowledge extends only to April 2024 while CycleReviewer is an offline model whose knowledge ends in January 2024, which makes judging the novelty of newer papers difficult. The authors also note a risk of reward hacking, where the policy model exploits loopholes in the reward model to raise rewards without genuinely improving research quality.

Paper links

External research summaries. These are not HDATF publications or measured product results.