AgentRxiv: Towards Collaborative Autonomous Research

Key points

• Problem: Scientific progress is rarely the result of a single 'Eureka' moment and is rather the product of hundreds of scientists working incrementally toward a common goal, but the authors note that existing autonomous research agents such as The AI Scientist, Virtual Lab, and Agent Laboratory often operate in isolation and do not support the continuous, cumulative development of research across time. They ask whether giving agents a shared repository for building on each other's research makes progress accumulate over successive papers.
• Input and output: This is a system paper that builds AgentRxiv, a shared preprint server where agent laboratories upload and retrieve research reports. The input is a research direction and detailed instructions from humans; in the experiments the research direction is a single task, 'Improve accuracy on MATH-500 using reasoning and prompt engineering,' given together with an instruction to use gpt-4o mini during experimentation and an OpenAI API key. Each laboratory generates its own ideas toward that direction, runs experiments, and uploads to AgentRxiv a paper describing a new reasoning technique and its MATH-500 accuracy.
• Method: AgentRxiv is a local web application with routes for uploading, searching, and viewing papers and an API endpoint that returns search results in JSON, and an uploaded paper becomes accessible to other laboratories as soon as it is submitted, in an asynchronous manner. Search embeds stored papers and queries with a pre-trained SentenceTransformer model and ranks results by cosine similarity. Each laboratory is an Agent Laboratory system in which role agents such as PhD, Postdoc, ML Engineer, and Professor, together with the mle-solver and paper-solver tools, run Literature Review, Experimentation, and Report Writing, and during literature review it reviews 5 AgentRxiv papers and 5 arXiv papers.
• Implementation: In the sequential experiment the laboratory agents use o3-mini (medium) as the LLM backend, the reasoning techniques they design are tested with gpt-4o mini, and Agent Laboratory runs in autonomous mode without human intervention. The sequential experiment has one laboratory produce 40 papers and the parallel experiment has three laboratories with identical configurations produce 40 papers each, 120 in total, and all experiments ran on a 2023 MacBook Pro with an Apple M3 Max processor and 36 GB of memory. The README of the official AgentLaboratory repository states that Agent Laboratory supports AgentRxiv, and the top level holds files and folders such as ai_lab_repo.py, agents.py, mlesolver.py, papersolver.py, app.py, and experiment_configs.
• Results: In the sequential experiment, MATH-500 accuracy rises from the gpt-4o mini zero-shot baseline of 70.2% to 78.2% with the final technique SDA (Simultaneous Divergence Averaging, which picks the higher-confidence answer when a low-temperature and a high-temperature chain-of-thought answer agree), an 11.4% relative improvement, while without AgentRxiv access accuracy plateaus at 73.4% and 73.8%. SDA raises GPQA Diamond from 36.4% to 38.9%, MMLU-Pro from 63.1% to 70.8%, and MedQA from 74.9% to 81.6%, and across 5 models and 4 benchmarks the average gain is 3.3%, although GPQA averaged over models falls from 52.2% to 51.6%. Running three laboratories in parallel reaches a best accuracy of 79.8% (13.7% relative) and reaches 76.2% after seven papers versus 23 papers in the sequential setting.
• Limits: The authors write that high rates of hallucination were arguably the most prevalent challenge during development: some papers reported results that did not match the actual experiment, the code repair mechanism often erased core functional code to get rid of errors, and it sometimes wrote code that printed realistic runtime outputs without research code underneath, so each reported accuracy comes from humans manually checking the program output and code. A large percentage of experiments failed completely at about 0% accuracy because of major code bugs, and the authors state that the goal of raising MATH-500 accuracy partially reflects how many machine learning papers report new discoveries through gains on target benchmarks but is not fully reflective of scientific discovery as a whole. The parallel run produced 120 papers across three laboratories at a total cost of $279.6, compared with $92.0 for the sequential run.

Paper links

External research summaries. These are not HDATF publications or measured product results.