AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery

Key points

• Problem: Autonomous research agents must find the right papers to explore prior knowledge and gather evidence for claims, but the decisive evidence often sits in method details, ablation tables, figure captions, appendices, and citation chains rather than titles or abstracts. Agents have made strong progress on long-horizon web-search benchmarks such as GAIA (a benchmark for general AI assistants) and BrowseComp, but whether an agent can search a large up-to-date scientific corpus, read full papers in depth, verify fine-grained conditions, and decide when its search is complete remains largely unmeasured.
• Input and output: This is a benchmark paper. An agent takes a natural-language query, searches and reads a corpus of over 3 million full-text arXiv papers through the search tool of the DeepXiv academic platform, and returns either the single paper that meets every condition or no paper for Deep Research, and the full set of qualifying papers for Wide Research. Deep Research is scored by exact-match accuracy and Wide Research by the IoU (intersection over union) between the predicted and gold sets.
• Method: Deep Research items preferentially start from under-exposed, high-quality computer science (CS) papers with typically 10 to 100 citations; clues mined from the full text are fuzzified and pruned to those needed to single out the target paper, and no-answer items perturb one core constraint. A Deep item is dropped if GPT-5.4 query rewrites surface the target, if a Claude 4.6-Sonnet or Gemini-3-Flash agent solves it, or if a human finds it within 10 minutes. Wide Research builds queries from attributes shared by topic-level candidate papers, searches 31,734 more candidates with LLM-written queries to supplement missing answers, and then sets the gold answers through full-text judgments by three LLMs and expert spot checks.
• Scale and implementation: The benchmark has 1,000 queries (600 Deep, 400 Wide) across eight CS areas. 10% of Deep queries have no answer, and Wide queries cover 3,692 gold papers in total, 9.23 per query on average. Evaluation runs a ReAct agent, which alternates reasoning and tool calls, for at most 30 turns with the DeepXiv search tool; per its README, the GitHub repository (CherYou/AutoResearchBench) holds batch inference code, DeepXiv and web search tools, Deep and Wide scoring scripts, and a decryption script, while the benchmark data sits as an obfuscated file in a Hugging Face dataset repository.
• Results: Among models evaluated with the same ReAct agent and DeepXiv search tool on the full benchmark, the best Deep Research accuracy was 9.39% by Claude-Opus-4.6 and the best Wide Research IoU was 9.31% by Gemini-3.1-Pro-Preview. GPT-5.4 reached 7.44% with 6.1 turns on average, while DeepSeek-V3.2 used more turns, 28.8, and stayed at 4.21%. For the four models in the search-tool comparison, replacing DeepXiv with open-web search lowered average Deep accuracy from 5.42% to 3.97%.
• Limits: The authors state that the benchmark targets computer science papers in a fixed corpus and text-based search, leaving other sciences, multimodal evidence, and evolving literature to future work. They note that exhaustive Wide answer sets may still be hard to build at the boundary of a large corpus, and a human audit of sampled out-of-gold predictions from the three top models found that 96% violated at least one query constraint. The three costly end-to-end research systems were evaluated on a random sample of 50 queries, scoring 11/50 for GPT Deep Research, 7/50 for AI-studio Gemini-3.1-Pro, and 0/50 for Alphaxiv on Deep Research.

Paper links

External research summaries. These are not HDATF publications or measured product results.