PaperOrchestra: A Multi-Agent Framework for Automated AI Research Paper Writing

Key points

• Problem: The writing modules of research automation agents such as AI Scientist-v2 rely on artifacts produced earlier in their own pipelines, which makes them hard to use as standalone tools for writing a manuscript from ideas and experimental logs that people provide. Such agents rely on simple keyword searches, write shallow literature reviews with few citations, and limit visuals to code-generated data plots, and there was no standardized benchmark for evaluating automated paper writing on its own.
• Input and output: It takes an idea summary of the method and contributions, an experimental log of results and ablations, the conference LaTeX template, and the conference guidelines, and it uses figures when they are given and generates them when they are not. It returns the LaTeX source together with the compiled PDF. This paper presents an implemented multi-agent framework together with PaperWritingBench, a benchmark for evaluating it.
• Method: The Outline Agent turns the materials into a JSON outline with a visualization plan, a literature search strategy, and a section-level writing plan, and the Plotting Agent and the Literature Review Agent then produce the figures and the Introduction and Related Work in parallel. For the literature review, an LLM with web search finds candidate papers, the Semantic Scholar API keeps those with a fuzzy title match, an available abstract, and a date before the cutoff, and when writing the Introduction and Related Work the agent must cite at least 90% of the gathered pool. The Section Writing Agent builds tables from the numbers in the experimental log and writes the remaining sections, and the Content Refinement Agent revises the draft with simulated reviews from AgentReview and keeps a revision when the overall score rises, or when it ties and the net change in sub-scores is zero or positive.
• Implementation: Gemini-3.1-Pro is fixed as the writing backbone for PaperOrchestra and every baseline, Gemini-3-Flash with Google Search grounding handles literature discovery, and one paper takes about 60 to 70 LLM calls, with a mean processing time of 39.6 minutes measured on a random subset of 10 papers with a single worker. PaperWritingBench was built from 100 accepted papers each from CVPR (Computer Vision and Pattern Recognition) 2025 and ICLR (International Conference on Learning Representations) 2025 by converting them to Markdown with MinerU and extracting figures and captions with PDFFigures 2.0, then reverse-engineering idea summaries (a high-level Sparse version and a Dense version that keeps formal definitions and equations) and experimental logs with Gemini-3.1-Pro, and removing authors, titles, citations, and URLs. Per its README, the official GitHub repository google-research/paper-orchestra contains the paper_writing_cli.py entry script, the agent code (methods), the automated evaluation scripts (autoraters), conference LaTeX templates, and a Streamlit demo, and the README states that the dataset used for paper generation is not in the repository and will be released separately later.
• Results: In the standard sparse idea setting, side-by-side judgments by 11 AI researchers on 40 papers (20 per venue, 180 comparisons) gave PaperOrchestra a win rate margin over the AI baselines (Single Agent and AI Scientist-v2) of 50% to 68% for literature review quality and 14% to 38% for overall manuscript quality. In the same setting, PaperOrchestra with the original figures (PlotOff) reached ScholarPeer simulated acceptance rates of 84% on CVPR and 81% on ICLR, 13% and 9% above the strongest autonomous baseline, against 86% and 94% for the human-written originals. The PlotOff manuscripts averaged 45.73 to 47.98 citations, against roughly 59 in the originals and an average of 9 to 14 for the autonomous baselines.
• Limits: The authors state that relying on the external PaperBanana for figures limits direct control over hallucinated figure content, that the refinement step revises drafts from structured LLM-generated feedback while a human-in-the-loop system in which researchers steer drafts through natural language critiques is left for future work, and that a benchmark built from published papers carries a risk of pre-training data contamination. For literature review quality, the correlation between the autoraters and human judgments was low, with Pearson 0.0868 for Gemini-3.1-Pro and 0.2764 for GPT-5. A gap to human-written papers remains, and under the AI Scientist-v2 Reviewer the simulated acceptance rate of PaperOrchestra (PlotOff) was 48.00% on CVPR and 22.00% on ICLR, against 71.00% and 63.00% for the originals.

Paper links

External research summaries. These are not HDATF publications or measured product results.