How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks

Key points

• Problem: Existing research-agent benchmarks either use simulated worlds or, when grounded in real science, restrict themselves to verifiable settings, most commonly machine learning and software engineering tasks with a well-defined target. Nearly all of them anchor scoring at the endpoint, so a single number hides where and why an agent broke down. Among prior failure diagnoses, expert case studies cannot generalize across systems or tasks, categorization efforts come from small corpora, and trace-level analyses such as MAST, TRAIL and Who&When annotate conversational or tool-call traces, so failures outside those traces stay invisible.
• Input and output: This is a benchmark paired with a failure taxonomy. The agent receives only the Premise and Tension extracted from a published paper, works alone through six stages in a sandbox with code execution, and delivers a decision.json plus a report.md with an appended Peer Review section. On the evaluation side, a judge reads the whole trajectory and outputs a stage-by-stage analysis.md with ARFT pattern labels.
• Method: From 5,878 candidate papers the authors built 100 runnable tasks: 70 open-ended discovery tasks with no metric to aim at, and 30 target-anchored optimization tasks with a human SOTA (state of the art) or a computable metric. Three experts induced ARFT bottom-up from full trajectories, and it sorts 45 patterns by the stage where a failure surfaces (A to F, plus a cross-stage layer X) and by four root causes.
• Implementation: The eight agents are Claude Code driving opus-4.8, claude-sonnet-5, qwen3.7-max, glm-5.2, minimax-m3 and deepseek-v4-pro, Codex driving gpt-5-mini, and Gemini CLI driving gemini-3.5-flash; the 800 trajectories hold 73k tool calls at 92.3 steps per episode on average. The judge is a Claude Code agent running claude-opus-5 that opens the code, execution logs, final reports and data files itself. Per the README, the GitHub repository holds the task-construction pipelines and the agent-as-a-judge code and leaves out the rollout infrastructure, while the paper states that tasks, trajectories and annotations are released and lists a Hugging Face dataset as the data address.
• Results: On 50 trajectories with human expert labels, the agent judge reached a Cohen's kappa (an agreement measure) of 0.75 at the pattern level and 0.83 at the taxonomy level, against 0.53 and 0.62 for a single-call LLM judge that reads only the transcript. Across 800 analyses the taxonomy recorded 12,712 hits; Grounding & Faithfulness (31.0%), Scientific Integrity & Alignment (33.5%) and Cognitive Depth & Adaptability (27.6%) together made up 92.1%, and Engineering Robustness 7.9%. The most frequent pattern, F.4, appeared in 660 of 800 analyses (82.5%), and total hits across the eight models ranged from 1,396 for opus-4.8 to 1,818 for qwen3.7-max.
• Limits: ARFT comes from trajectories of eight combinations on this task suite, and the authors present the failure frequencies as indicative. All runs had fixed wall-clock and token budgets, so patterns such as premature termination and shallow search may correlate with resource pressure, and data contamination (the source papers appearing in model training data) cannot be fully excluded. Judge validation is reported only in aggregate over the 50-trajectory sample, and whether orchestration-level interventions can close the metacognitive gap is left as an open question.

Paper links

External research summaries. These are not HDATF publications or measured product results.