AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite

Key points

• Problem: Existing agent benchmark suites lack the standard environments and tools needed for controlled comparison, so it is unclear whether a winning agent has better AI capabilities or just access to a more relevant information source. The authors found no agent benchmark that accounts for differences in tool usage and note that only a few, such as HAL (Holistic Agent Leaderboard), measure cost; they also cite nonstandard interfaces that make general agents hard to plug in and a lack of tasks grounded in real product usage and of broad baselines.
• Input and output: This is a benchmark paper that releases AstaBench, a suite for measuring scientific research assistance, together with its environment, evaluation toolkit, and agent suite, and its over 2400 problems are split into 11 benchmarks in four categories: literature understanding, code and execution, data analysis, and end-to-end discovery. Agents take inputs such as search queries, long-form questions, repository instructions, datasets with discovery goals, or research tasks, and return paper lists, cited reports, literature review tables, execution results, hypotheses, or reports with code. Scoring uses rubric-based LLM-as-judge or programmatic checks, and the leaderboard records cost along with the score.
• Method: The Asta Environment supplies standard tools, the Asta Scientific Corpus search toolset that returns only papers before a cutoff date and a stateful Jupyter computational notebook, and the search tools are offered over MCP (Model Context Protocol) as snippet_search, get_paper, and others. The agent-eval toolkit converts model usage logged by Inspect into dollar costs with a frozen snapshot of the litellm cost map (a list of model prices), so costs stay comparable when API prices change. Results carry each agent's openness (four levels) and tooling (standard, custom interface, fully custom).
• Implementation: The agent-baselines suite holds nine Asta agent classes optimized for science (Paper Finder for paper search, Scholar QA for long-form question answering, Table Synthesis for literature review tables, Code for repository code execution, DataVoyager for data analysis, Panda and CodeScientist for end-to-end research, and the orchestrator Asta v0 that detects the task type and hands it to a sub-agent, among others) plus baselines such as ReAct, Smolagents Coder, and commercial deep research systems. The authors evaluated 57 agents across 22 agent classes. According to the README, the GitHub repository runs the full suite and single tasks and aggregates scores and costs with Inspect-based commands (astabench eval, inspect eval, astabench score), the dataset comes from a gated Hugging Face dataset, and running it needs an ASTA_TOOL_KEY for the literature tools and OpenAI, Anthropic, and Google API keys for scoring.
• Results: Among agents that solve all benchmarks, Asta v0 had the highest macro average at 53.0%, ReAct (gpt-5) reached 44.0%, and the best agent with an open-weight model (Smolagents Coder, Llama-4-Scout) scored 11.1%. On ScholarQA-CS2 (long-form scientific question answering), Asta Scholar QA, Elicit, and SciSpace Deep Review scored about 85% or higher, on SUPER-Expert (setting up and running research repositories) only two agents, including ReAct (gpt-5) at 41.1, passed 25%, and the top DiscoveryBench (data-driven hypothesis discovery) score was 33.7 by ReAct (o3). The highest E2E-Bench (end-to-end research tasks) step completion score was 70.5 by Asta Panda (claude-sonnet-4), but among the three agents in Table 20 the rate of finishing all steps peaked at 5%.
• Limits: Several benchmarks, including PaperFindingBench, ScholarQA-CS2, DiscoveryBench, and E2E-Bench, rely on rubric-based LLM-as-judge scoring, and ScholarQA-CS2 rubric items were drawn from the evaluated systems' answers; when each of five systems was held out of rubric creation, three of them lost an average of 2.5 points in answer coverage. E2E-Bench problems are specified in detail for controlled comparison, so they only weakly test ideation and planning. The task mix is weighted toward computer science (CS), and Asta Panda and Asta CodeScientist were graded from cached results because their source code had not yet been integrated.

Paper links

External research summaries. These are not HDATF publications or measured product results.