aiXiv: A Next-Generation Open Access Ecosystem for Scientific Discovery Generated by AI Scientists

Key points

• Problem: LLM agents can now write research proposals and papers and even perform peer review, yet traditional journals that still rely heavily on human peer review remain reluctant to accept AI-generated research and struggle to scale with growing submissions, and preprint servers often lack rigorous quality control. The Agents4Science conference, where AI acts as both author and reviewer, has no revision or rebuttal stage, and most existing LLM review systems treat review as a one-shot process.
• Input and output: An AI scientist or a human researcher submits a research proposal (problem statement, motivation, methodology, and planned experiments) or a full paper through API and MCP interfaces (MCP being a protocol that lets agents call external tools), and LLM review agents return structured feedback on novelty, technical soundness, clarity, feasibility, and potential impact. A revised version can be resubmitted, and a submission is published with a DOI (a persistent identifier for the paper) when at least three of the five LLMs on the review panel vote accept. This is a system paper that builds the platform and tests its review accuracy and the effect of revisions.
• Method: In Single Review Mode, one review agent writes strengths, weaknesses, and suggestions under four criteria (methodological quality, novelty and significance, clarity and organization, and feasibility and planning), and RAG (retrieval-augmented generation) adds literature retrieved through the Semantic Scholar API. In Meta Review Mode, an Area Chair or Editor agent identifies the subfields of a submission, creates 3 to 5 domain-specific reviewer agents for each subfield, and synthesizes their reports into a decision letter. The prompt injection check extracts font size, color, character positions, and encoding from the PDF, runs rule-based scanning and LLM semantic verification, and filters out documents whose risk score exceeds a predefined threshold before they reach the review model.
• Implementation: aiXiv runs as a public website (aixiv.science), where an API and MCP layer connects agents in author, reviewer, and meta-reviewer roles to the platform and users can like, comment on, and discuss submissions. The accuracy of the pairwise mode, which picks the better of two documents, was measured on 235 and 163 pairs of accepted and rejected papers from the DeepReview test data for ICLR (International Conference on Learning Representations) 2024 and 2025, after dropping papers with mean reviewer ratings between 5 and 6, and on 500 pairs of ICLR papers converted into proposal format. Revision effects were tested on proposals that the AI Scientist generated, 50 per topic on NanoGPT, 2dDiffusion, and Grokking, deduplicated at an 80% cosine similarity threshold (28, 27, and 29 per topic in Table 5), and on 10 papers written by the AI Scientist.
• Results: With RAG, Claude-sonnet-4 reached pairwise accuracies of 81.70% on ICLR 2024 and 79.75% on ICLR 2025 for papers, and 77.91% on ICLR 2024 for proposals. The prompt injection check reached 84.8% accuracy on a synthetic set built by adding attacks to 35% of 105 arXiv papers, which yielded 36 adversarial papers, and 87.9% on a real-world suspicious sample set. Under five-LLM voting, the mean acceptance rate of 10 AI Scientist papers went from 10% before revision to 70% after, and proposals in all three topics went from 0% before revision to a mean of 45.2% after.
• Limits: The authors acknowledge that AI models may still produce fluent yet factually incorrect content, that review bias can remain even with multiple models, and that simple LLM majority voting may lack objectivity. They also note that experimental validation is currently restricted to simulated environments and virtual agent interactions, which limits generalization, especially in domains that need real-world experiments or physical constraints. The effect of RAG was mixed across models, and GPT-4.1's proposal pairwise accuracy on ICLR 2024 fell from 75.05% to 69.73% with RAG.

Paper links

External research summaries. These are not HDATF publications or measured product results.