HDATF
简体中文
开始交流
菜单

论文档案库

按编号从大到小列出全部收录论文及摘要。

收录论文 1,430资料更新日期 Research Monitoring基准测试
  1. 1293Research AgentsarXiv:2407.10362

    LAB-Bench: Measuring Capabilities of Language Models for Biology Research

    • Problem: Most science benchmarks ask textbook-style recall questions, and benchmarks that measure whether a system can carry out practical research work in a specific domain are rare. The authors argue that building AI systems that assist biology research first needs a benchmark for practical skills such as literature lookup, protocol troubleshooting, and sequence manipulation.
    • Input and output: This is a benchmark and dataset that measures practical biology research work with 2,457 multiple-choice questions. A model receives a question and options about literature, figures, tables, databases, DNA and protein sequences, protocols, or cloning scenarios and returns one answer letter, and every question carries an extra option saying there is not enough information to answer.
    • Method: LitQA2 (literature QA), SuppQA (supplementary material QA), FigQA (figure QA), TableQA (table QA), ProtocolQA (protocol troubleshooting), and CloningScenarios (real-world cloning workflows) were written manually by the authors and contracted biology experts, while most subtasks of SeqQA (sequence QA) and DbQA (database QA) were generated programmatically. Scores are reported as accuracy (correct over all questions), precision (correct over attempted questions), and coverage (share of questions attempted). About 80% of each subtask is released as a Hugging Face dataset and 20% is kept private to monitor training-data contamination.
    • Implementation: Claude 3.5 Sonnet, Claude 3 Opus, Claude 3 Haiku, Gemini 1.5 Pro, GPT-4o, GPT-4-turbo, and Llama 3 70B Instruct were called through APIs without tools using a 0-shot chain-of-thought prompt, and results are averaged over three runs per model. Llama 3 70B has no vision capability and was left out of FigQA and TableQA, and only 25 of the 41 CloningScenarios questions were short enough for its prompt. The human baseline comes from PhD-level biology researchers who covered 69% of the questions in quizzes of 20 to 140 questions and could use the internet and DNA sequence software.
    • Results: Human experts beat the models in most categories, and on TableQA, where models came closest, Claude 3.5 Sonnet reached 0.83 accuracy (humans 0.84) and 0.90 precision (humans 0.87). On LitQA2 the major frontier models often declined to answer, so Claude 3.5 Sonnet's accuracy was only 0.06, and on FigQA every model except Claude 3.5 Sonnet had near-random precision. On CloningScenarios, humans reached 0.60 accuracy and the best models (Claude 3.5 Sonnet and GPT-4o) 0.28.
    • Limits: The authors note that plausible distractors are hard to design and that stronger models appear to raise scores by eliminating unlikely options and guessing instead of reasoning. In a small open-answer run on 10 CloningScenarios questions, Claude 3.5 Sonnet and GPT-4o both reached only 0.20 accuracy. The authors also note that only the humans could use tools, and CloningScenarios questions take 10 to 60 minutes each, so the human baseline there may be too low.
    原文摘要PDF
  2. 1292Research AgentsarXiv:2609.09219

    Scores Alone Do Not Prove Discovery: The Discovery Certification Protocol for Auditing AI Research Agents

    • Problem: When an AI research agent reports a high-scoring output after experiments, the score alone does not show that a discovery was made. A sealed test (held-out evaluation data) establishes the output's utility, but whether a matched agent could reach the same score from the initial data, public material, and its own reasoning, and whether truthful experimental feedback improves the chance of reaching it, are separate measurement questions. The authors turn these three questions about one outcome into an executable audit.
    • Input and output: This is a framework paper that proposes an audit protocol for research outcomes and demonstrates it on controlled and real-data tasks. The input is an outcome claim (the final artifact, a registered model and budget, background knowledge and starting observations, Web content seen by the target run, and validity and recovery rules); the output is a record bundle with two decisions, Core and Evidence, each set to Pass, Refuted, Inconclusive, or Not evaluated. A deterministic verifier recomputes the decisions offline from the frozen records.
    • Method: Gate 1 checks, with a lower confidence bound on sealed evaluation, that the output beats the baseline by at least a preset smallest useful gain (δmin). Gate 2 runs a registered number of fresh challenger agents that receive the background knowledge, starting observations, and Web content seen by the target run but no run history or new measurements; any valid result from any method at or above the recovery line (a threshold equal to the target score x minus tolerance ε) refutes Core, and zero recoveries yield a Clopper-Pearson upper bound on the recovery probability. Optional Gate 3 compares paired branches from a shared checkpoint that receive truthful measurement feedback or neutral messages, and Evidence requires a lower confidence bound on the effect plus a null-policy equivalence check.
    • Implementation: The audited agents and challengers are DeepSeek models run through a registered command-line interface (CLI): deepseek-v4-flash for SQLite-Web, deepseek-v4-pro for the virtual catalyst, and deepseek-flash for the real-data tasks, and real-data sessions expose the Read, Write, Edit, Glob, and Grep tools, with shell and native Web tools disabled. The Yacht and Ionosphere audits each registered 96 challengers, 45 file-channel controls, 30 feedback pairs, and 60 null pairs. According to the paper, the supplement contains the installable dcp-audit, task-specific evidence producers, the reusable dcp-harness, and three real-data decision bundles.
    • Results: On real data, Yacht and Ionosphere passed Core with outcome scores of 0.7677 and 0.6551; their best scores across 96 challengers, 0.4641 and 0.4171, stayed below the recovery lines of 0.7177 and 0.6051, and the recovery upper bound was 0.0468. From a shared checkpoint, branches with truthful feedback reached the target level in 9 and 16 of 30 trials while neutral branches reached it in none, yet Evidence was Inconclusive for both tasks. In controlled cases, SQLite-Web (88.55% less traffic-weighted work) and virtual catalyst control (sealed optimum reached) passed both Core and Evidence, and in the developmental knapsack case Core was refuted when a challenger produced 2 results above the recovery line.
    • Limits: The authors state that every reported pass is a local kernel decision from records collected and replayed locally (formal_certificate_issued=false), and formal issuance needs review and countersigning by an independent authority. The recovery bound concerns one fresh episode from the registered model, policy, information, and budget distribution, and at an illustrative recovery rate of 0.01, a 96-episode audit would find an existing route with probability 0.6190. The Core and Evidence passes for Dry Bean are an exploratory reanalysis under a revised policy recorded after Gate 2 scoring.
    原文摘要PDF
  3. 1291Research AgentsarXiv:2607.22682

    A Vocabulary for Multi-Agent Automated Research Systems

    • Problem: Recent multi-agent LLM systems automate research end to end (reading prior work, proposing ideas, writing code, running experiments, and interpreting results), but they differ from one another in many design choices at once, such as communication topology, cross-run memory, initialization, and the evaluator. A claim that a multi-agent system or an autoresearcher is better therefore stays ambiguous until one knows which choice changed, and there was no common language for describing these systems.
    • Input and output: The vocabulary takes the description of an automated research system (its paper), maps the design onto a problem tuple and a system tuple, and produces a map of which coordinates the system sets and which it leaves generic. One run is recorded as a trajectory from the input task to the returned artifact, and because agents, operations, and initialization may be stochastic, repeated runs on the same task yield a distribution over trajectories. This is a conceptual paper that defines the vocabulary and adds case analyses applying it to 10 existing systems.
    • Method: A system is written as an eight-element tuple of the agent set, the operation universe (tools, skills, Model Context Protocol (MCP) server calls, and so on), the communication structure, the capability assignment, the shared state, the control policy, the initialization, and the evaluator. Shared state is split into within-run state that is wiped when the run ends, the external environment the agents act on, and state carried across runs, and the control policy is split into routing (who acts next and what they do), stopping (when the run ends), meta-control that changes the structure mid-run (spawn, kill, grant, revoke, consolidate, and others), and an exploration term for stochasticity. Each step's action in a trajectory is classified as an operation call, a message, a structural change, or a halt, which turns when to communicate or when to add an agent into a choice made by the policy.
    • Argument: The author frames a system as search that maximizes a proxy evaluator score under a budget, and notes that the same agents and tools can score higher under more search, such as Best-of-N, while every other coordinate stays the same. The paper defines an overfitting tax, where ranking more trajectories makes it more likely that the top one is a trajectory the evaluator overrates, and cites MLR-Bench reporting fabricated or invalidated results in 8 of 10 audited coding-agent tasks (around 80%). Taste is split into generative taste, the rate of proposing novel trajectories before any score is seen, and evaluative taste, the gap between the proxy score and true quality, and the author suggests raising evaluator integrity (the structural protections that keep the score close to true quality) before crediting a stronger generator.
    • Case studies: Mapping AIRA2 (an autoresearcher for ML research benchmarks), AlphaEvolve, Glia, AAR (Automated Alignment Researcher), AI Scientist-v2, MetaGPT, EvoX, ml-intern, SimpleTES, and Engram onto the tuple shows, for example, that the core of Glia is a capability assignment that leaves its Supervisor with an empty operation set, while AIRA2 sets a cross-run population store and meta-control. In this map no system changes capabilities during a run, every system leaves the exploration term at the default temperature, and EvoX is the only system that rewrites its control policy at run time. Per its README, the author's GitHub repository BardiyaAk/multi-agent-research-vocabulary releases a static project page with an explainer of the eight-element tuple, structure views, and a comparison table for the 10 case studies, along with tools/coords.json, the source of the comparison data, and tools/build-data.py, which converts it.
    • Limits: The author maps each system from its own paper's description, so the coding is a descriptive map and differs from a validated measurement. A marked coordinate means that the paper sets it and is distinct from evidence that the coordinate improved performance. Generative taste stays qualitative until a novelty measure exists, and true quality is uncomputable in general, so it remains a design target rather than a measured value.
    公开代码原文摘要PDF
  4. 1290Research AgentsarXiv:2607.11079

    Are LLMs Ready for Scientific Discovery? A Capability-Oriented Benchmark for AI Scientists

    • Problem: Existing benchmarks such as DataSciBench, KramaBench, and ScienceAgentBench mainly check whether an LLM can generate and execute analysis code or pipelines that reach ground-truth answers or task-specific success criteria. The authors argue that such evaluation can mistake successful execution for scientific validity and hides behind one aggregate score whether a model fails at basic description or at causal inference. Scientific analysis supports claims such as description, hypothesis exploration, statistical inference, prediction, causal estimation, and mechanistic explanation, each with different assumptions and validity criteria.
    • Input and output: This is a benchmark paper. Each item consists of a domain, a task type, a dataset with metadata, a question, and a ground-truth answer, and the model reads the dataset and answers in a multiple-choice (MCQ) or open-ended (OEQ) format. Base models are evaluated zero-shot, every model returns a JSON response with reasoning and answer fields, only the answer is scored, and the reasoning is used for error annotation.
    • Method: The authors turn the six-question framework of Leek and Peng into six capabilities (descriptive, exploratory, inferential, predictive, causal, mechanistic) and build items across five domains (biology, chemistry, environment, geography, physics). SDA-Synth, the synthetic part, uses scenarios proposed by GPT-4o to build a semantic DAG (directed acyclic graph) of variables with mechanism equations, adds noise and distractor columns to generate datasets, and keeps 50 (10 per domain) after execution replay, range checks, and expert review. Questions come from templates built by four agents (Taxonomist, Extractor, Generator, Verifier), answers come from a deterministic derivation procedure per template, and MCQ distractors come from four perturbation strategies on variables or intermediate computations.
    • Scale and implementation: SDA-Real, the real-data part, has 527 items built from real datasets in seed resources such as KramaBench, BLADE (a benchmark for data-driven science agents), and QRData, chosen from 24 candidate datasets and benchmarks, and SDA-Synth keeps 6,000 items by taking 200 per domain and task pair from the 6,231 that passed validation. Validation removed 779 candidates, a manual audit of 307 templates and 2,000 sampled items filtered out 324 more, and SDA-Synth is split 60% for fine-tuning training and 40% for testing. The 15 evaluated LLMs include closed-source models such as GPT-5.4, Claude Sonnet 4.6, and Gemini 3.1 Pro, open-source models such as DeepSeek V3.2 and Qwen 3.5-397B-A17B, and two Llama-3.1-8B variants fine-tuned with LoRA (Low-Rank Adaptation) on the SDA-Synth training split.
    • Results: On the SDA-Synth test split, the top OEQ accuracies were 58.33% for GPT-5.4, 56.88% for Gemini 3.1 Pro, and 55.79% for Claude Sonnet 4.6, and Llama-3.1-8B fine-tuned with the error-type objective approached this range at 55.33%. By task type, Descriptive tasks were easiest and accuracy dropped on Inferential and Causal tasks, and by domain the average OEQ accuracy across LLMs was highest for Environment at 46.9% and lowest for Biology, whose Predictive score was 22.5%. In the error analysis, which labels the first error in execution, the normalized Function Error rate peaked at 37.7% on Predictive tasks and the Relationship Error rate reached 32.1% on Mechanistic tasks.
    • Limits: The authors state that the synthetic semantic graphs of SDA-Synth, though validated by domain experts, may not capture real experimental confounders and measurement artifacts, and that the MCQ format can induce superficial pattern matching in some models. Because the error taxonomy labels the first error in execution, it may undercount cascading failures. Because four-option MCQ has a 25% chance floor, MCQ and OEQ accuracies are not directly comparable, and the full SDA-Real results are deferred to supplementary material C.
    原文摘要PDF
  5. 1289Research AgentsarXiv:2605.30329

    SoundnessBench: Can Your AI Scientist Really Tell Good Research Ideas from Bad Ones?

    • Problem: Recent AI scientist frameworks can automate research from hypothesis generation to experiment execution, but few benchmarks directly test whether a model can judge the methodological soundness of a proposal before it is executed. Existing benchmarks such as MLE-Bench and PaperBench mainly score execution, and a few idea-evaluation benchmarks score novelty and impact. The authors argue that without a strong first-gate filter, agents risk automatically pursuing flawed hypotheses and wasting compute and time.
    • Input and output: This is a benchmark paper. The evaluated model receives a research proposal (hypothesis and experiment design) with results removed and returns, in JSON, a high or low soundness label, a confidence from 1 to 5, and a step-by-step justification. Ground-truth labels come from reviewer soundness sub-scores at ICLR (the International Conference on Learning Representations): a mean of 3 or more is high, 2 or less is low, and middle scores are excluded.
    • Method: The construction pipeline takes ICLR submissions from 2022 to 2026, drops desk rejects (papers rejected before review), and keeps only papers with mean reviewer confidence of at least 3 and a standard deviation of normalized soundness scores below 0.15. Gemini 2.5 Pro extracts the abstract, related work, risk factors, hypothesis, and experiment design from each PDF in near-verbatim form while removing experimental results, direct outcome claims, and acceptance cues. Each extracted proposal is split into atomic claims that are checked against source passages found with BM25 keyword ranking, candidates with a support ratio at or above a threshold of 0.7 are kept, and 66.93% of candidates in the authors' current pool passed this check.
    • Implementation: The final benchmark holds 1,099 proposals (458 low, 641 high) across 16 ML subfields, selected after processing over 35,209 initial ICLR submissions and 137,940 reviews, and the first page of the paper points to this dataset on Hugging Face. During construction, GPT-5.4 decomposes atomic claims and Gemini-2.5-Flash verifies them. Of the 12 evaluated models, closed models run through provider APIs and open models run with vLLM (an LLM serving library) on two NVIDIA H200 GPUs, and unless otherwise noted all models use temperature 0.2 and max_tokens 8192.
    • Results: Under the standard prompt, the 12 models rated low-soundness proposals as high 74.0% of the time on average, and the two models with the highest rate, LLaMA-3.3-70B and GPT-4o, did so for 98.0% and 94.5%. Switching to the tested strict prompt that defaults to low cut this mean rate from 74.0% to 19.9%, but mean high-soundness recall also dropped from 91.8% to 36.1% and mean Macro F1 (the unweighted mean of the two class F1 scores) fell from 54.9 to 49.3. On an ICLR 2026-only split evaluated with models whose training cutoffs precede the ICLR 2026 submission period, low-soundness proposals were still rated high 77.47% of the time on average, close to 73.88% for the same models on the full dataset.
    • Limits: The authors state that the labels come from reviewers who saw the full paper and results, so they are imperfect proxies for judging a results-masked proposal. The scope is ML research from ICLR only, and because the source corpus is public, training-data contamination cannot be fully controlled, a concern that the ICLR 2026-only split and the identifier-removal analysis reduce. The human audit is a preliminary check of 60 proposals, and only two prompts, standard and strict, were tested.
    原文摘要PDF
  6. 1288Research AgentsarXiv:2605.23204

    AutoResearch AI: Towards AI-Powered Research Automation for Scientific Discovery

    • Problem: AI research systems now reach into long workflows that run from literature grounding through hypotheses, experiments, validation, and reporting, yet they differ in autonomy, domain, execution environment, validation mechanism, and human oversight, so they are hard to compare in one frame. The authors note that pipeline breadth can be mistaken for scientific autonomy, and that current systems are strong at search, drafting, and coding but much weaker at validation, rejection, and reproducibility.
    • Scope: This paper is a survey. Table 1 groups surveys, workflow-level systems, and open-source projects and marks the autonomy level of each system and project, and Table 2 sorts evaluation instruments by evaluative role into discovery, experimentation and verification, deep research and synthesis, and review and provenance. The official repository (Mr-Tieguigui/Autoresearch) is, per its README, a static project page that shows the survey figures, made of index.html and a static folder with styles, scripts, and figures.
    • Taxonomy: The authors define five levels from L0 (human only) to L4 (AI autonomous) by how humans and AI split control, execution, and validation, and call the human-steered L1 and L2 region Vibe Research. They split L2 into single-step execution (L2-S), interactive workflow automation (L2-I), and pipeline automation under human verification (L2-P), and use a conservative rule that places each system at the lowest level consistent with its demonstrated workflow role.
    • Workflow stages: The authors split the technical foundations into five stages, literature grounding, hypothesis formation and planning, experimentation and tool use, feedback and validation, and reporting, and sort the recurring approaches within each stage. For example, grounding has four regimes (search-centered, evidence-centered, structure-centered, literature-memory), and validation has three (execution-coupled reruns, critique-mediated validation, expert- or temporally-grounded validation).
    • Evaluation and domains: The authors propose five evaluation dimensions, novelty, validity, impact, reliability, and provenance, so that outputs are judged on scientific credibility instead of task completion alone. Across domains, they place computational and formal sciences, whose artifacts are executable and replayable, at the highest level currently observed, and medicine and clinical research and economics and social sciences in early-to-middle L2. They conclude that no domain yet gives convincing evidence of robust L3 or L4 scientific autonomy.
    • Limits: Table 1 is a selection of structurally important works rather than a full catalog, and its levels show the primary autonomy regime suggested by each work's workflow scope, execution capability, and degree of human oversight rather than a performance ranking. The authors state that novelty still has no agreed measurable definition, so current evaluations lean on weak proxies such as dissimilarity from prior papers, LLM-as-a-judge, or human expert judgment.
    公开代码原文摘要PDF
  7. 1287Research AgentsarXiv:2605.03042

    ARIS: Autonomous Research via Adversarial Multi-Agent Collaboration

    • Problem: The authors see plausible unsupported success as the central failure in long-horizon research workflows, where results may be misreported or claims may outrun their evidence. Among existing systems such as AI Scientist, AI Scientist v2, and Agent Laboratory, many use the same or closely related model families for execution and review, which can leave correlated errors uncaught, their workflows are tightly coupled end to end, which makes stages hard to replace or resume, and few provide explicit system-level checks on experimental integrity and manuscript quality.
    • Input and output: The full pipeline takes a research direction and moves through a ranked idea report (IDEA_REPORT.md), an experiment log (EXPERIMENT_LOG.md), and a narrative report to a compiled paper PDF, and the rebuttal workflow takes a paper with its reviews and returns a paste-ready rebuttal. Skills pass results to each other through such plain-text files, so work can resume from saved artifacts. This is a technical report on the design and early deployment of an implemented open-source system, and a controlled comparative evaluation is left for future work.
    • Method: The executor produces an artifact such as code, a manuscript section, or an experiment design, a reviewer that the recommended configuration draws from a different model family returns a rubric score and action items, and the loop ends when the score exceeds a threshold (default 6/10) and all critical items are resolved, or when it reaches the maximum number of rounds (default 4). The reviewer receives file paths instead of the executor's summary and reads the artifacts directly, with a configurable access scope (document only, plus result files, or the whole repository) and context policy (a fresh thread each round or memory across rounds). Auditing runs in three stages: an integrity audit in which a cross-model reviewer examines the evaluation code and outputs (model-derived reference labels, self-normalized scores, numbers that differ from the actual result files, and more), a claim ledger that rates results as supported, partial, or invalidated, and a fresh zero-context reviewer that checks the manuscript's numbers against the raw result files.
    • Implementation: As of v0.4 (April 2026), ARIS (Autonomous Research via Adversarial Multi-Agent Collaboration) has more than 65 Markdown SKILL.md skills, 5 workflows, and 6 MCP (Model Context Protocol) bridges (Codex, Oracle, Claude, Gemini, MiniMax, and llm-chat), it was tested on 3 executors (Claude Code, Codex CLI, and Cursor), and its research wiki builds a lightweight knowledge graph from four entity types (papers, ideas, experiments, and claims) with 8 relationship types. The default pairing is a Claude-family executor with a GPT-family reviewer, or the reverse, with reviews in the current implementation routed to GPT-5.4 through Codex MCP, and the system includes effort presets (lite, balanced, max, beast), a FigureSpec renderer that turns JSON into SVG (Scalable Vector Graphics) figures, a standalone Rust command-line interface (CLI) called ARIS-Code, and a prototype meta-optimize loop that proposes SKILL.md patches from usage logs and shows the user only those the reviewer scores at least 7/10. Per its README, the official GitHub repository wanshuiyin/Auto-claude-code-research-in-sleep contains skills, mcp-servers, tools, and templates folders, installer scripts, and Claude Code and Codex CLI plugins, and the README describes a later state that adds Workflows 5 to 7 and new reviewer bridges.
    • Results: Instead of a controlled comparison, the authors document one overnight run in which, over about eight hours, the system completed 4 review and revision rounds, launched more than 20 GPU experiments, and removed unsupported claims while the internal reviewer score rose from 5.0 to 7.5/10. For the deployment footprint, they report growth from 21 core skills at release to more than 65, with 4 GPU backends (local, SSH (Secure Shell) remote servers, Vast.ai, Modal), 9 venue template families, and 30 or more community-contributed skills. The authors state that the reported outcomes are observational and cannot be causally attributed to ARIS alone, and that this single trajectory should not be read as evidence that cross-family review beats same-family review.
    • Limits: The authors state that ARIS cannot guarantee that any output is correct, novel, or scientifically sound, and that the three-stage audit can catch common integrity failures but is an advisory safety net, distinct from a formal verification system that would detect every error, inconsistency, or fabrication. The review loop can amplify reviewer biases by overfitting to the reviewer model's preferences, and over-iteration past diminishing returns can degrade paper quality. Repository-level review may send source code to external LLM APIs, so the authors advise against enabling it on repositories with sensitive code or secrets unless an approved local-only review path is available, and local-only reviewer routing is still planned.
    公开代码原文摘要PDF
  8. 1286Research AgentsarXiv:2604.25256

    AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery

    • Problem: Autonomous research agents must find the right papers to explore prior knowledge and gather evidence for claims, but the decisive evidence often sits in method details, ablation tables, figure captions, appendices, and citation chains rather than titles or abstracts. Agents have made strong progress on long-horizon web-search benchmarks such as GAIA (a benchmark for general AI assistants) and BrowseComp, but whether an agent can search a large up-to-date scientific corpus, read full papers in depth, verify fine-grained conditions, and decide when its search is complete remains largely unmeasured.
    • Input and output: This is a benchmark paper. An agent takes a natural-language query, searches and reads a corpus of over 3 million full-text arXiv papers through the search tool of the DeepXiv academic platform, and returns either the single paper that meets every condition or no paper for Deep Research, and the full set of qualifying papers for Wide Research. Deep Research is scored by exact-match accuracy and Wide Research by the IoU (intersection over union) between the predicted and gold sets.
    • Method: Deep Research items preferentially start from under-exposed, high-quality computer science (CS) papers with typically 10 to 100 citations; clues mined from the full text are fuzzified and pruned to those needed to single out the target paper, and no-answer items perturb one core constraint. A Deep item is dropped if GPT-5.4 query rewrites surface the target, if a Claude 4.6-Sonnet or Gemini-3-Flash agent solves it, or if a human finds it within 10 minutes. Wide Research builds queries from attributes shared by topic-level candidate papers, searches 31,734 more candidates with LLM-written queries to supplement missing answers, and then sets the gold answers through full-text judgments by three LLMs and expert spot checks.
    • Scale and implementation: The benchmark has 1,000 queries (600 Deep, 400 Wide) across eight CS areas. 10% of Deep queries have no answer, and Wide queries cover 3,692 gold papers in total, 9.23 per query on average. Evaluation runs a ReAct agent, which alternates reasoning and tool calls, for at most 30 turns with the DeepXiv search tool; per its README, the GitHub repository (CherYou/AutoResearchBench) holds batch inference code, DeepXiv and web search tools, Deep and Wide scoring scripts, and a decryption script, while the benchmark data sits as an obfuscated file in a Hugging Face dataset repository.
    • Results: Among models evaluated with the same ReAct agent and DeepXiv search tool on the full benchmark, the best Deep Research accuracy was 9.39% by Claude-Opus-4.6 and the best Wide Research IoU was 9.31% by Gemini-3.1-Pro-Preview. GPT-5.4 reached 7.44% with 6.1 turns on average, while DeepSeek-V3.2 used more turns, 28.8, and stayed at 4.21%. For the four models in the search-tool comparison, replacing DeepXiv with open-web search lowered average Deep accuracy from 5.42% to 3.97%.
    • Limits: The authors state that the benchmark targets computer science papers in a fixed corpus and text-based search, leaving other sciences, multimodal evidence, and evolving literature to future work. They note that exhaustive Wide answer sets may still be hard to build at the boundary of a large corpus, and a human audit of sampled out-of-gold predictions from the three top models found that 96% violated at least one query constraint. The three costly end-to-end research systems were evaluated on a random sample of 50 queries, scoring 11/50 for GPT Deep Research, 7/50 for AI-studio Gemini-3.1-Pro, and 0/50 for Alphaxiv on Deep Research.
    公开代码原文摘要PDF
  9. 1285Research AgentsarXiv:2604.05018

    PaperOrchestra: A Multi-Agent Framework for Automated AI Research Paper Writing

    • Problem: The writing modules of research automation agents such as AI Scientist-v2 rely on artifacts produced earlier in their own pipelines, which makes them hard to use as standalone tools for writing a manuscript from ideas and experimental logs that people provide. Such agents rely on simple keyword searches, write shallow literature reviews with few citations, and limit visuals to code-generated data plots, and there was no standardized benchmark for evaluating automated paper writing on its own.
    • Input and output: It takes an idea summary of the method and contributions, an experimental log of results and ablations, the conference LaTeX template, and the conference guidelines, and it uses figures when they are given and generates them when they are not. It returns the LaTeX source together with the compiled PDF. This paper presents an implemented multi-agent framework together with PaperWritingBench, a benchmark for evaluating it.
    • Method: The Outline Agent turns the materials into a JSON outline with a visualization plan, a literature search strategy, and a section-level writing plan, and the Plotting Agent and the Literature Review Agent then produce the figures and the Introduction and Related Work in parallel. For the literature review, an LLM with web search finds candidate papers, the Semantic Scholar API keeps those with a fuzzy title match, an available abstract, and a date before the cutoff, and when writing the Introduction and Related Work the agent must cite at least 90% of the gathered pool. The Section Writing Agent builds tables from the numbers in the experimental log and writes the remaining sections, and the Content Refinement Agent revises the draft with simulated reviews from AgentReview and keeps a revision when the overall score rises, or when it ties and the net change in sub-scores is zero or positive.
    • Implementation: Gemini-3.1-Pro is fixed as the writing backbone for PaperOrchestra and every baseline, Gemini-3-Flash with Google Search grounding handles literature discovery, and one paper takes about 60 to 70 LLM calls, with a mean processing time of 39.6 minutes measured on a random subset of 10 papers with a single worker. PaperWritingBench was built from 100 accepted papers each from CVPR (Computer Vision and Pattern Recognition) 2025 and ICLR (International Conference on Learning Representations) 2025 by converting them to Markdown with MinerU and extracting figures and captions with PDFFigures 2.0, then reverse-engineering idea summaries (a high-level Sparse version and a Dense version that keeps formal definitions and equations) and experimental logs with Gemini-3.1-Pro, and removing authors, titles, citations, and URLs. Per its README, the official GitHub repository google-research/paper-orchestra contains the paper_writing_cli.py entry script, the agent code (methods), the automated evaluation scripts (autoraters), conference LaTeX templates, and a Streamlit demo, and the README states that the dataset used for paper generation is not in the repository and will be released separately later.
    • Results: In the standard sparse idea setting, side-by-side judgments by 11 AI researchers on 40 papers (20 per venue, 180 comparisons) gave PaperOrchestra a win rate margin over the AI baselines (Single Agent and AI Scientist-v2) of 50% to 68% for literature review quality and 14% to 38% for overall manuscript quality. In the same setting, PaperOrchestra with the original figures (PlotOff) reached ScholarPeer simulated acceptance rates of 84% on CVPR and 81% on ICLR, 13% and 9% above the strongest autonomous baseline, against 86% and 94% for the human-written originals. The PlotOff manuscripts averaged 45.73 to 47.98 citations, against roughly 59 in the originals and an average of 9 to 14 for the autonomous baselines.
    • Limits: The authors state that relying on the external PaperBanana for figures limits direct control over hallucinated figure content, that the refinement step revises drafts from structured LLM-generated feedback while a human-in-the-loop system in which researchers steer drafts through natural language critiques is left for future work, and that a benchmark built from published papers carries a risk of pre-training data contamination. For literature review quality, the correlation between the autoraters and human judgments was low, with Pearson 0.0868 for Gemini-3.1-Pro and 0.2764 for GPT-5. A gap to human-written papers remains, and under the AI Scientist-v2 Reviewer the simulated acceptance rate of PaperOrchestra (PlotOff) was 48.00% on CVPR and 22.00% on ICLR, against 71.00% and 63.00% for the originals.
    公开代码原文摘要PDF
  10. 1284Research AgentsarXiv:2508.15126

    aiXiv: A Next-Generation Open Access Ecosystem for Scientific Discovery Generated by AI Scientists

    • Problem: LLM agents can now write research proposals and papers and even perform peer review, yet traditional journals that still rely heavily on human peer review remain reluctant to accept AI-generated research and struggle to scale with growing submissions, and preprint servers often lack rigorous quality control. The Agents4Science conference, where AI acts as both author and reviewer, has no revision or rebuttal stage, and most existing LLM review systems treat review as a one-shot process.
    • Input and output: An AI scientist or a human researcher submits a research proposal (problem statement, motivation, methodology, and planned experiments) or a full paper through API and MCP interfaces (MCP being a protocol that lets agents call external tools), and LLM review agents return structured feedback on novelty, technical soundness, clarity, feasibility, and potential impact. A revised version can be resubmitted, and a submission is published with a DOI (a persistent identifier for the paper) when at least three of the five LLMs on the review panel vote accept. This is a system paper that builds the platform and tests its review accuracy and the effect of revisions.
    • Method: In Single Review Mode, one review agent writes strengths, weaknesses, and suggestions under four criteria (methodological quality, novelty and significance, clarity and organization, and feasibility and planning), and RAG (retrieval-augmented generation) adds literature retrieved through the Semantic Scholar API. In Meta Review Mode, an Area Chair or Editor agent identifies the subfields of a submission, creates 3 to 5 domain-specific reviewer agents for each subfield, and synthesizes their reports into a decision letter. The prompt injection check extracts font size, color, character positions, and encoding from the PDF, runs rule-based scanning and LLM semantic verification, and filters out documents whose risk score exceeds a predefined threshold before they reach the review model.
    • Implementation: aiXiv runs as a public website (aixiv.science), where an API and MCP layer connects agents in author, reviewer, and meta-reviewer roles to the platform and users can like, comment on, and discuss submissions. The accuracy of the pairwise mode, which picks the better of two documents, was measured on 235 and 163 pairs of accepted and rejected papers from the DeepReview test data for ICLR (International Conference on Learning Representations) 2024 and 2025, after dropping papers with mean reviewer ratings between 5 and 6, and on 500 pairs of ICLR papers converted into proposal format. Revision effects were tested on proposals that the AI Scientist generated, 50 per topic on NanoGPT, 2dDiffusion, and Grokking, deduplicated at an 80% cosine similarity threshold (28, 27, and 29 per topic in Table 5), and on 10 papers written by the AI Scientist.
    • Results: With RAG, Claude-sonnet-4 reached pairwise accuracies of 81.70% on ICLR 2024 and 79.75% on ICLR 2025 for papers, and 77.91% on ICLR 2024 for proposals. The prompt injection check reached 84.8% accuracy on a synthetic set built by adding attacks to 35% of 105 arXiv papers, which yielded 36 adversarial papers, and 87.9% on a real-world suspicious sample set. Under five-LLM voting, the mean acceptance rate of 10 AI Scientist papers went from 10% before revision to 70% after, and proposals in all three topics went from 0% before revision to a mean of 45.2% after.
    • Limits: The authors acknowledge that AI models may still produce fluent yet factually incorrect content, that review bias can remain even with multiple models, and that simple LLM majority voting may lack objectivity. They also note that experimental validation is currently restricted to simulated environments and virtual agent interactions, which limits generalization, especially in domains that need real-world experiments or physical constraints. The effect of RAG was mixed across models, and GPT-4.1's proposal pairwise accuracy on ICLR 2024 fell from 75.05% to 69.73% with RAG.
    原文摘要PDF
  11. 1283Research AgentsarXiv:2507.18074

    Neural Architecture Discovery via Autonomous Evolution

    • Problem: Whether LLM agents can complete hard long-horizon tasks that advance AI research itself remains largely unexplored. Neural Architecture Search (NAS) only selects within human-defined spaces. The authors name three obstacles in architecture design: LLMs have limited built-in architecture knowledge, a single constraint violation in a large codebase silently corrupts training, and quality can only be measured by costly pre-training from scratch and benchmark runs.
    • Input and output: This is a system paper with an implementation and experiments. ASI-ARCH takes summaries of past architectures, experimental analyses, and literature knowledge (cognition), writes the motivation and code of a new linear attention architecture in one pass, and stores the loss, benchmark scores, and analysis from actually training it as input for the next cycle.
    • Method: A proposal from the Researcher enters training only after a novelty check, in which an LLM judges it against the 5 most similar past motivations found by embedding, and a code sanity check, and when training fails from an implementation error a debugging agent fixes the code from the error log. Fitness averages the sigmoid-transformed loss and benchmark improvements over the baseline, focused on differences within 10%, with an LLM judge score. The Analyzer compares each architecture with parent and sibling nodes to estimate the effect of individual design choices, and a pool of the top 50 architectures plus literature-derived cognition entries ground the next proposal.
    • Implementation: The exploration stage trains 20M-parameter models on 1B tokens and evaluates them on 500 samples per benchmark, and the verification stage scales selected candidates to 340M parameters on 1B tokens and top candidates to 1.3B parameters on 15B tokens. The planner uses o3 and GPT-4.1, the checker and analysis use o3, training runs and debugging use GPT-4.1, and roughly 100 linear-attention papers were distilled into about 150 cognition entries searched through OpenSearch. Per its README, the official repository (GAIR-NLP/ASI-Arch) holds the pipeline, a MongoDB-based architecture database, and a cognition base built on retrieval-augmented generation (RAG), and links the list of discovered architectures to a project page.
    • Results: In the 1,773 experiments (about 10,000 GPU hours) of the 20M-scale exploration, 1,350 candidates beat the DeltaNet baseline on both quantitative and qualitative metrics, and of about 150 re-verified at 340M scale, 105 reached state-of-the-art results. At 1.3B, the best model beat DeltaNet by +0.97 points on the overall average of 14 accuracy metrics, nearly three times the +0.34-point gain of Mamba2 over DeltaNet. With the same DeepSeek-V4-Flash proposal model, a 120-attempt budget, and the 20M training setup, the successful-design rate was 37.4% for ASI-ARCH and 8.3% for OpenEvolve, an open-source evolutionary coding agent.
    • Limits: The search ran on linear attention with DeltaNet as the baseline, and the 105 state-of-the-art results are judged under the authors' controlled 340M evaluation. In the 1.3B efficiency comparison on the same implementation and hardware, the five ASI-ARCH models peak at 16.28G to 22.75G of memory, above the 11.44G of DeltaNet. At 340M, transferred rules gave +2.05 points on GLA (Gated Linear Attention) but smaller gains of +0.35 on the state-space model Mamba2 and +0.14 on HGRN2 (a gated linear recurrent model), and the authors leave family-specific adapters to future work.
    公开代码原文摘要PDF
  12. 1282ReasoningarXiv:2507.16812

    MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoning

    • Problem: Open reasoning research has focused primarily on mathematics and coding, which offer abundant datasets, established benchmarks, and clear verification, and scientific reasoning remains significantly underdeveloped, particularly in the open-source community. Many science benchmarks are multiple-choice and post-training datasets often follow that format, and the authors observe that models trained on such data show inflated multiple-choice scores but struggle with computational problems, while decontamination (removing benchmark questions from training data), which typically relies on n-gram overlap or embedding similarity, is easily circumvented by minor variations in phrasing or structure. Reference answers in many science datasets are scraped from the web or generated by LLMs, which makes their factual accuracy hard to guarantee, and data distilled directly as long CoT (written-out reasoning) from DeepSeek-R1 is often prone to overthinking, which hurts training and inference efficiency.
    • Input and output: It takes textbook PDFs and public science datasets (NaturalReasoning and Nemotron-Science) and returns two supervised fine-tuning (SFT) datasets of questions with step-by-step solutions (TextbookReasoning and MegaScience), an evaluation system covering 15 benchmarks, and Llama3.1, Qwen2.5, and Qwen3 models trained on the data. This is a dataset paper with a data construction pipeline and training experiments.
    • Method: For TextbookReasoning, olmOCR (a PDF text recognition tool) converts textbooks to text, and for each 4,096-token chunk Llama3.3-70B-Instruct extracts question and answer pairs under a high-standard criterion (questions that demand multi-step reasoning, with complete solutions in the source) and a low-standard criterion (any complete question and answer), followed by MinHash deduplication (similarity threshold 0.6), refinement by DeepSeek-V3 against the source text, and filtering of defective pairs. For decontamination, BGE-large-en-v1.5 embeddings find the 5 most similar benchmark questions for each question, and Llama3.3-70B-Instruct judges whether each pair is a paraphrase, removing the question if any pair is. For MegaScience, response length, difficulty (16 responses from Qwen2.5-7B-Instruct scored 0 to 10 by Qwen2.5-32B-Instruct), and random selection are compared per dataset through SFT on Qwen2.5-7B, and DeepSeek-V3 writes new step-by-step solutions for every source except TextbookReasoning.
    • Implementation: TextbookReasoning used 12.8k books (including 6,057 in computer science and 2,305 in medicine and biology) after removing crawled books whose metadata marked them as restricted for public access and filtering out books below university level with a subject and level classification by Llama3.3-70B-Instruct, and its 945,683 extracted pairs became 651,840 after deduplication, filtering, and decontamination. MegaScience combines 436.4k from NaturalReasoning, 173.3k from Nemotron-Science, and 651.8k from TextbookReasoning into 1261.5k instances, and Qwen2.5, Qwen3, and Llama3.1 base models were fine-tuned on it with LLaMA-Factory (a fine-tuning tool) at a learning rate of 5e-6, batch size 512, and 3 epochs. Per its README, the official GitHub repository GAIR-NLP/MegaScience contains step-by-step data_process scripts from question and answer extraction on olmOCR output through decontamination, reference answer extraction, and final formatting, along with the LLaMA-Factory training configuration, while the two datasets and the trained models are on Hugging Face and the evaluation code is in a separate GitHub repository, lm-open-science-evaluation.
    • Results: In Table 6, which compares Qwen2.5-7B models trained on different open datasets with Qwen2.5-7B-Instruct, the MegaScience model was best on 7 of the 14 benchmarks used (MMLU, Massive Multitask Language Understanding, was excluded), averaged 61.01 against 58.80 for Qwen2.5-7B-Instruct, and scored 48.75 on SciBench and 40.23 on OlympicArena. With Qwen3 compared in non-thinking mode, the Qwen3-30B-A3B MegaScience model averaged 73.86 against 70.62 for the official Instruct model, and Llama3.1-8B also led 51.07 to 49.67, but for Qwen2.5-1.5B the official Instruct model was ahead by 2.99. The LLM-based decontamination found benchmark leakage in existing datasets: 19K questions in SCP-116K, 66K in NaturalReasoning, and 164K in Nemotron-Science.
    • Limits: The authors state that finding one data mixture that is optimal for every base model remains hard, and that weaker models such as the Llama series or small Qwen2.5 models trained on difficult data like MegaScience without supplemental foundational data or easier warmup training give shorter responses with more repetition. Data selection was validated with Qwen2.5-7B as a proxy model, which ties the results to that model's capacity, and in Table 7 the math average of the MegaScience models fell below the official Instruct models for Llama3.1-8B, Qwen2.5-1.5B, Qwen3-1.7B, and Qwen3-4B. Reinforcement learning (RL) is left for future work, and experiments on compressing long CoT into shorter formats were postponed because of compute limits.
    公开代码原文摘要PDF
  13. 1281Research AgentsarXiv:2501.03916

    Dolphin: Moving Towards Closed-loop Auto-research through Thinking, Practice, and Feedback

    • Problem: In auto-research, judging whether AI-generated ideas actually work is hard. Most prior studies only had humans or LLMs rate idea quality, and AI-Scientist ran experiments on simple self-constructed datasets that limit comparison with existing methods in the same field. Prior idea-generation work also lacked a feedback mechanism that passes experimental results back into idea generation.
    • Input and output: This is a system paper with an implementation and experiments. DOLPHIN takes a research topic (for example, 3D classification) and reference code, reads the titles and abstracts of related papers to propose ideas, modifies the code to follow an experimental plan and runs it, and feeds ideas that improved performance into the idea-generation prompt of the next loop.
    • Method: An LLM extracts the task attributes of the topic (model inputs, outputs, and so on) and scores papers found through Semantic Scholar from 1 to 10 to keep only relevant ones. An idea is dropped as redundant when its maximum cosine similarity to the idea bank exceeds the 0.8 threshold and must pass an LLM novelty check, and when code fails, custom-code function names, lines, and code are pulled from the exception traceback to build a local code structure before debugging. Results are sorted into improvement, maintenance, and decline against the reference code, and ideas similar to earlier ones that failed to improve performance are filtered out in the next loop.
    • Implementation: Idea generation and result analysis use gpt-4o-2024-08-06, code editing and debugging use deepseek-v2.5 served through ollama, and aider calls the LLM agents. The total number of retrieved papers is 50, each loop generates 20 ideas, and debugging runs at most 5 times. Per its README and file list, the GitHub repository (InternScience/Dolphin) holds the launch_dolphin scripts, dolphin_utils, examples, and docs, and the data for the three classification tasks is downloaded from a Google Drive link.
    • Results: With two loops (40 ideas) per task, the best idea raised ModelNet40 OA (overall accuracy) from 91.0% for the authors' PointNet implementation to 93.9%, CIFAR-100 accuracy from 81.2% to 82.0%, and SST-2 accuracy from 91.0% to 92.5%, and the ModelNet40 result is comparable to the 93.8% of the human-designed GPSFormer. On the insult detection task of MLE-bench, which is built from Kaggle competitions, DOLPHIN raised code generated by AIDE (a code generation framework) from 81.0 to 84.7. In the per-loop analysis, the share of successfully executed ideas that improved performance rose from 2/7 in Loop 1 to 4/8 in Loop 3.
    • Limits: The authors state that knowledge the LLM retained from training data may leak into idea generation, and that using only titles and abstracts can miss the technical details of papers. Current LLM coding ability is too weak for project-level code, so DOLPHIN cannot yet verify complex tasks. The 3D comparison excludes pre-training and voting, and the experiments span three classification tasks (point cloud, image, sentence) and three MLE-bench tasks.
    公开代码原文摘要PDF
  14. 1280Research AgentsarXiv:2610.04074

    IdeaScientist: Orchestrating Agents for Grounded Scientific Ideation

    • Problem: Existing autoresearch systems couple solution proposal with implementation, experimentation and writing, so it is hard to tell whether success comes from strong ideas or strong execution. A research proposal has no directly verifiable reward, and a single proposal-level rubric gives weak credit assignment (telling which stage earned the score) and invites reward hacking.
    • Input and output: This is an agent system for research ideation. The input is a problem definition (the task input and desired output) plus a challenge that current solutions fail to meet, and the output is a research proposal in a fixed schema covering the problem definition, key novelty, related work, method, evaluation plan, falsifiable predictions and limitations. The proposal carries no measured results and cites prior work inline by database id.
    • Method: An orchestrator keeps the research plan and calls three sub-agents: a gap finder identifies the shared limitation of closely related work, an innovator adapts a mechanism that solved a similar challenge in another field, and a report writer turns that intuition into a complete proposal. Each role is trained separately with asynchronous GRPO (Group Relative Policy Optimization, an RL method that compares rewards within each question's group of rollouts) against a reference artifact reconstructed from a human paper. The reward weights by 0.85 a rubric score gated by structural, task-compliance, citation and tool-validity checks (the mean of a reference-free quality score, a reference-grounded score and citation F1 for overlap in cited papers) and by 0.15 a rule-based compliance term.
    • Implementation: The Svalbard Idea Vault holds 4.7 million papers (2.8 million, or 60%, with full text), namely arXiv papers from all subject areas published from 1986 through June 18, 2026 plus missing papers found one hop through the references of papers collected on or after January 1, 2026, decomposed into 2.77 million (problem definition, challenge, research gap, intuition, proposal) tuples. The 14,977 machine learning methodology papers published on or after January 1, 2026 were split into 14,500 training, 200 validation and 277 test papers, and one LoRA (Low-Rank Adaptation, a fine-tuning method that trains small adapters while keeping the original weights) adapter per role was trained on Qwen3.6-27B. Per the README, the GitHub repository holds the ideation harness, the vault construction code, the RL reward and the evaluation, the vault and the train, validation and test partition sit in a Hugging Face dataset, and the ablation arms, the human study and the LaTeX table builders are left out.
    • Results: On the overall score (the mean of the novelty average and the proposal-quality average) given by a Qwen3.6-27B judge on the 277 test papers, IdeaScientist-Trained on Qwen3.6-27B reached 74.6%, 14.0% above DeepScientist (60.6%), the best open-source baseline on the same backbone, and 5.9% above the general-purpose agent Claude Code with Claude-4.8-Opus (68.7%) and 5.1% above Codex with GPT-5.4 (69.5%). Held-out validation rewards per role rose from 56.6% to 66.7% for the gap finder, from 58.8% to 69.5% for the innovator and from 78.6% to 89.1% for the report writer. The gains of IdeaScientist-Base came mainly from novelty, and its average-novelty margin over the strongest baseline on the same backbone peaked at 23.1% on Qwen3.6-27B.
    • Limits: Evaluation happens before any experiment, as an LLM-judged match to directions that human papers took after the cutoff, and on the 50 proposals used to validate the judge, two human experts agreed at a Pearson r = 0.56. In the blind pairwise human study on 50 papers, DeepScientist was the only exception to the statistically significant wins of IdeaScientist-Trained, whose more frequent wins over IdeaScientist-Base were also not significant. Training and test papers are limited to machine learning methodology papers, and a title-only probe of 5 models on 10 post-cutoff papers found no recall, but the authors list residual model-memory contamination as a limitation of the retrospective evaluation.
    公开代码原文摘要PDF
  15. 1279Research AgentsarXiv:2609.38445

    AIM: Agentic Idea Management for Automated Research

    • Problem: In automated research, implementation and evaluation are expensive, so a limited experimental budget has to be allocated across possible approaches. Idea-driven search, which selects ideas first and delegates implementation to a solver, must organize a growing pool of ideas and evidence, select which ideas to implement, and keep the solver's code faithful to the selected idea. Existing methods typically rely on fixed rules such as UCB or MCTS (Monte Carlo Tree Search) or on LLM judgments without explicit estimates, and when the code does not realize the idea, its score and lessons are attributed to the wrong idea.
    • Input and output: AIM takes one AutoLab task (a natural-language instruction, a containerized environment, a deliberately weak baseline codebase, and an evaluation script) together with a research brief generated from the task description using Claude Code. When the experimental budget is spent, it returns the highest-scoring executable solution among those evaluated as the final output. This is a system paper that implements the framework, tests it on 10 benchmark tasks, and adds a theoretical analysis of when idea-level search pays off.
    • Method: Agentic Surrogate follows the surrogate role in Bayesian optimization, regrouping the whole idea pool into 2 to 5 semantic clusters at every iteration and producing ordinal estimates of promise for clusters and ideas. Agentic Acquisition uses two LLM calls to assign each branch an explore or exploit action at both the cluster and the idea level, sends the chosen ideas to parallel Solvers, and then generates new ideas from extracted lessons with one of four modes: score-guided refinement, cross-pollination, error-guided repair, or novel idea generation. Solution Auditor removes results flagged as trivial, task_mismatch, or reward_hacking from the history and lessons, and when idea_mismatch is the only issue it rewrites the idea to match the implementation and attaches the score to that idea.
    • Implementation: All methods use Gemini-3.1-Pro-Preview as the backbone LLM, implementation is mainly done by the Gemini Deep Solver from ScientistOne, and the initial pool of about 10 ideas comes from the ScientistOne Ideator. The budget is at most 300 executions and 6 hours for System Optimization, 60 executions and 24 hours for Model Development, and 60 executions and 12 hours for CUDA, and the Resource Planner launches 5 Solver branches in the first iteration and then sets between 3 and 10 branches per iteration. The appendix gives pseudocode for the full pipeline and the prompt templates for every LLM-driven operator.
    • Results: On the five AutoLab System Optimization tasks, AIM averaged 67.0%, 1.6 points above ScientistOne (65.4%), the strongest idea-driven baseline, and 3.7 points above AdaEvolve (63.3%), the strongest solution-driven baseline. On the five Model Development and CUDA tasks, AIM averaged 55.8%, 4.9 points above ScientistOne (50.9%), and the authors report that AIM reached ScientistOne's best score up to 3.1 times faster in wall-clock time. In the Flash Attention ablation, removing any single component scored below the full AIM's 90.5%, and removing the Agentic Surrogate caused the largest drop, to 85.9%.
    • Limits: The authors state that the assumption behind the theoretical analysis, that the implementation faithfully realizes the selected idea, could be slightly idealized and that real solver agents may produce incomplete or incorrect implementations. AIM tends to need more executions to reach its best score and generally uses a larger token budget, and on the data selection task Data Select IFEval it scored 61.8, below AdaEvolve (82.7). The authors write that isolated containerized environments and the Solution Auditor reduce risk, that some risk remains, and that human review is needed before applying autonomous research systems to consequential scientific or engineering settings.
    原文摘要PDF
  16. 1278Research AgentsarXiv:2609.00365

    Dr. Claw: An AI Scientist Workspace for Vibe Research

    • Problem: Command-line (CLI) coding agents such as Claude Code and Gemini CLI read and write files and sustain long sessions, yet end-to-end research still splits across chat tools, IDEs (integrated development environments), terminals and writing environments. These agents optimize execution rather than control, so the plan, intermediate decisions and artifacts that make a research process reviewable get scattered or lost, and humans have few explicit takeover points.
    • Input and output: Given goals, constraints and acceptance criteria in natural language, the system produces a structured research brief and a dependency-aware task plan, writes execution outputs into the Artifact Store, updates task states in the Task Graph and keeps all interventions in the Decision Log and Execution Trace. This is a system paper: the workspace keeps an existing command-line coding agent as the executor and adds state, control and audit layers on top. The authors call this split, where humans own research direction, evaluation criteria and final acceptance while AI handles retrieval, coding, running, summarizing and drafting, Vibe Research.
    • Method: Four state objects (Task Graph, Artifact Store, Decision Log, Execution Trace) form the workflow state, and each interaction is formalized as a state transition from the current state, an action and the observed feedback. The system has three layers (Interaction, Orchestration, Execution) so the executor can be switched without changing workflow semantics or the audit view, and each task goes through Plan, Execute, Verify and Write-back. Skills reach a task through a stage-skill map keyed on the task's stage and type, keyword detection over user instructions, or manual invocation, and every action must belong to the action set allowed by the current permission policy.
    • Implementation: The deployed skill catalogue holds 171 skills, of which 58 are mapped to five research stages (survey, ideation, experiment, publication, promotion). Moving to a new domain means editing one JSON map instead of code, backends attach through Claude and Codex SDK (software development kit) adapters, and pipeline mutations are written to persistent state first and then broadcast to the interface over WebSocket. Per the official README, the release includes a React and Vite frontend, a Node.js and Express backend, executor integrations for Claude Code, Gemini CLI, Codex and OpenRouter, a Chat flow that generates the research brief and task list with a skill, a Research Lab to review them, and Auto Research for one-click sequential task execution.
    • Results: Against a bare agent using the same executor (codex provider, gpt-5.4) on three medical tasks (melanoma and nevus classification on Derm7pt and a clinical-note risk baseline), pooled completeness, the share of 21 research best-practice elements present in the output files, was 0.873 for the bare agent and 0.952 for Dr. Claw (two wins, one tie), and the whole gap came from research hygiene elements (basic parts of a research report such as a limitations section, subgroup analysis and real literature citations), whose pooled pass rates rose from 0.33 to 1.00, 0.33 to 1.00 and 0.00 to 0.67. In a demonstration that injected a wrong-path error into one Derm7pt mini-project, the project recovered to a result with 0.892 accuracy while keeping all 5 earlier files. In a retrospective study of seven AI PhD researchers, Dr. Claw was associated with shorter completion-time bands and the highest output-quality ratings of the three conditions compared with working without AI or with general web and desktop AI assistants, and the experience score difference was significant in every pairwise comparison.
    • Limits: The authors call the evaluation a small exploratory pilot with few tasks and participants: with one run per task the pooled completeness gain of +0.079 has a 95% bootstrap CI (confidence interval) of [-0.00, +0.14] that includes zero, and the human study collected the control conditions as retrospective reports in a hybrid design. Holding the executor fixed adds the task graph, state objects and skill library as one bundle, so no single component's effect can be isolated; the comparison target is the bare executor Dr. Claw wraps instead of a state-of-the-art orchestration framework; and completeness counts which of 21 elements are present, so correctness needs separate review. Tasks come from one domain (medical), Dr. Claw was slower than the bare agent, and skill invocation is guaranteed only up to a suggestion in the task prompt, so the reference audit did not fire on the clinical-note task.
    公开代码原文摘要PDF
  17. 1277Research AgentsarXiv:2608.14905

    How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks

    • Problem: Existing research-agent benchmarks either use simulated worlds or, when grounded in real science, restrict themselves to verifiable settings, most commonly machine learning and software engineering tasks with a well-defined target. Nearly all of them anchor scoring at the endpoint, so a single number hides where and why an agent broke down. Among prior failure diagnoses, expert case studies cannot generalize across systems or tasks, categorization efforts come from small corpora, and trace-level analyses such as MAST, TRAIL and Who&When annotate conversational or tool-call traces, so failures outside those traces stay invisible.
    • Input and output: This is a benchmark paired with a failure taxonomy. The agent receives only the Premise and Tension extracted from a published paper, works alone through six stages in a sandbox with code execution, and delivers a decision.json plus a report.md with an appended Peer Review section. On the evaluation side, a judge reads the whole trajectory and outputs a stage-by-stage analysis.md with ARFT pattern labels.
    • Method: From 5,878 candidate papers the authors built 100 runnable tasks: 70 open-ended discovery tasks with no metric to aim at, and 30 target-anchored optimization tasks with a human SOTA (state of the art) or a computable metric. Three experts induced ARFT bottom-up from full trajectories, and it sorts 45 patterns by the stage where a failure surfaces (A to F, plus a cross-stage layer X) and by four root causes.
    • Implementation: The eight agents are Claude Code driving opus-4.8, claude-sonnet-5, qwen3.7-max, glm-5.2, minimax-m3 and deepseek-v4-pro, Codex driving gpt-5-mini, and Gemini CLI driving gemini-3.5-flash; the 800 trajectories hold 73k tool calls at 92.3 steps per episode on average. The judge is a Claude Code agent running claude-opus-5 that opens the code, execution logs, final reports and data files itself. Per the README, the GitHub repository holds the task-construction pipelines and the agent-as-a-judge code and leaves out the rollout infrastructure, while the paper states that tasks, trajectories and annotations are released and lists a Hugging Face dataset as the data address.
    • Results: On 50 trajectories with human expert labels, the agent judge reached a Cohen's kappa (an agreement measure) of 0.75 at the pattern level and 0.83 at the taxonomy level, against 0.53 and 0.62 for a single-call LLM judge that reads only the transcript. Across 800 analyses the taxonomy recorded 12,712 hits; Grounding & Faithfulness (31.0%), Scientific Integrity & Alignment (33.5%) and Cognitive Depth & Adaptability (27.6%) together made up 92.1%, and Engineering Robustness 7.9%. The most frequent pattern, F.4, appeared in 660 of 800 analyses (82.5%), and total hits across the eight models ranged from 1,396 for opus-4.8 to 1,818 for qwen3.7-max.
    • Limits: ARFT comes from trajectories of eight combinations on this task suite, and the authors present the failure frequencies as indicative. All runs had fixed wall-clock and token budgets, so patterns such as premature termination and shallow search may correlate with resource pressure, and data contamination (the source papers appearing in model training data) cannot be fully excluded. Judge validation is reported only in aggregate over the 50-trajectory sample, and whether orchestration-level interventions can close the metacognitive gap is left as an open question.
    公开代码原文摘要PDF
  18. 1276Research AgentsarXiv:2608.12564

    Scaling Automatic Research Agents via World Models

    • Problem: Training AutoResearch agents (LLM agents that take a research question, form an idea, implement the experiment, analyze the outcome, and iterate) with RL needs many trajectories. Generation shares compute through batching, but every candidate solution needs its own isolated sandbox that loads data and trains models on real GPUs, so execution cost grows linearly with the number of trajectories and becomes the dominant bottleneck for scaling RL.
    • Input and output: This is an RL training method paper with convergence theory and experiments. In the AutoResearch tasks, the world model takes the task description and the agent's solution code, predicts the execution outcome, and aggregates strict JSON verdicts from eight checks such as imports and names, API calls, file and column references, and the compute budget into a score. The final output is the post-trained agent policy, and AutoResearch evaluation uses the percentile on real Kaggle competition leaderboards.
    • Method: About 10% of groups are anchor groups graded by both the world model and real execution, and these score pairs form the thin stream of ground truth kept during training (the anchor signal). Online Debiasing fits a monotone map to the score pairs with isotonic regression (regression constrained to a non-decreasing function) to remove the world model's bias, and Inverse-Variance Denoising combines the policy gradients of the two group types weighted by inverse variance to suppress noise. The authors prove that bias and noise each add an error term to the convergence bound and that the two corrections shrink these terms, strictly improving the convergence guarantee.
    • Implementation: In the AutoResearch experiments, Qwen3.5-4B and Qwen3.5-9B agents are trained with GRPO (group-relative policy optimization), and the world model is the same backbone used through prompting only and kept fixed during training. Training uses 45 MLE-Dojo competitions (Kaggle machine learning competitions wrapped as interactive environments), evaluation uses 14 held-out competitions and 60 tasks from DSBench (a data science benchmark built on Kaggle modeling tasks), and the vision-language-action (VLA) experiments use the MiniVLA-1B policy with the progress model Robometer as the world model. According to the official repository README, the release contains the method in numpy (wmrl/), the MLE-Dojo and DSBench pipeline (ml_research/), the VLA track (embodied/), and tests; it is a clean reimplementation of the production stack used for the paper and implements only WMRL itself.
    • Results: WMRL cut training compute relative to real-execution GRPO from 883 to 286 GPU-hours at 4B and from 1174 to 349 GPU-hours at 9B, a 3.1x and 3.4x reduction. Average leaderboard percentile at 9B rose from 18.8 to 21.6 on MLE-Dojo and from 31.2 to 32.8 on DSBench, and on both benchmark averages the 9B WMRL agent scored above Nemotron-120B-A12B (20.5, 31.7) while the 4B WMRL agent scored above Kimi-48B-A3B. On LIBERO-Long (a simulated robot-arm manipulation benchmark), the overall success rate rose from 37.4% after SFT to 41.2% with WMRL, a 3.8 point gain.
    • Limits: The authors assume tasks whose outcomes are predictable from the agent's solution and where a small stream of real execution stays available as an anchor, and they note that when outcomes hinge on randomness that no reading of the solution reveals, WMRL degrades toward standard GRPO. The convergence guarantee relies on the assumption that the bias acts as a monotone distortion of the score, the AutoResearch experiments cover two Qwen3.5 scales (4B and 9B), and the MLE-Dojo tasks were chosen from the 20 smallest competitions by dataset size per category to fit the sandbox.
    公开代码原文摘要PDF
  19. 1275Research AgentsarXiv:2605.26029

    CausaLab: A Scalable Environment for Interactive Causal Discovery Toward AI Scientists

    • Problem: Existing LLM causal reasoning benchmarks generally turn causal graphs, datasets or narratives into question answering and classification tasks and usually provide the variables, evidence, graph or query up front, leaving the 'causal parrot' concern that models succeed through memorized causal facts or linguistic cues. The interactive Auto-Bench primarily asks whether an agent discovers a hidden DAG (directed acyclic graph), and the authors argue that evaluation should also measure whether the discovered mechanism transfers to a new instance.
    • Input and output: This is a benchmark environment for interactive causal discovery by LLM agents. In each episode the agent receives prior measurement records generated by a randomly sampled hidden SCM, intervenes on one controllable property of a manipulator crystal at a time within a fixed budget and observes the results, and then predicts the resonance frequency of a separate reactor crystal governed by the same SCM. At every step it emits a DSL record with five fields, memory, thought, past data, hypothesis and the next planned experiment, and the hypothesis is parsed into a graph, a frequency equation and coefficients for scoring.
    • Method: A DAG is sampled over the observable properties and frequency, root nodes take exogenous source values and the other variables are computed in topological order with linear or quadratic structural equations; the graph, equations and coefficients are shared by the records, the manipulator crystal and the reactor crystal, while controllable-property base values differ. Interventions are shift-style, replacing the variable's base term with v and keeping parent contributions, in place of hard do(X=v), and some graph families add a hidden disturbance H that is resampled after each intervention and added to a subset of variables. Scoring reports reactor frequency accuracy separately from graph precision, recall and F1, SHD (structural Hamming distance, the count of missing, extra and reversed edges), frequency-coefficient F1 and root-node identification.
    • Implementation: The authors built a new SCM generator, intervention and reactor mechanics, DSL logging and scoring code on top of the 2D game-style DiscoveryWorld environment, and a DSL record that fails the schema check is re-prompted up to two times with the parser error. The main suite ran four models, GPT-5-mini, GPT-5.2-high, Qwen3.5-Thinking and Qwen3.5-Non-thinking, on graphs with 3 to 7 nodes, with up to 50 topologies per (graph size, model) cell and one run per task, at temperature 0.1 and a fixed budget of 2 observations and 4(k-1) interventions for a k-node graph. Observation-intervention scaling used GPT-5-mini and GPT-5.2-high on 4-node and 6-node graphs, and the other follow-ups primarily used GPT-5-mini.
    • Results: With the 50 four-node topologies fixed, replacing the linear mechanism with a hard-quadratic one cut GPT-5-mini accuracy from 48% to 24% and frequency-coefficient F1 from 0.589 to 0.251, while root-node F1 rose from 0.559 to 0.829. For GPT-5.2-high on 6-node graphs, 20 observations with no intervention gave 92% accuracy and 0.471 all-edge F1, while 5 observations plus 20 interventions gave 80% and 0.799, and a Golden intervention chain that leaves a small set of candidate graphs explaining the data (Markov equivalence class) raised GPT-5-mini 4-node accuracy from 48% to 90% but lowered all-edge F1 from 0.793 to 0.728. Among the four models GPT-5.2-high had the best accuracy and lowest SHD at every graph size, yet it reached 64% accuracy and SHD 4.761 at 7 nodes, and adding one verification step before answering raised GPT-5-mini 4-node accuracy from 48% to 60%.
    • Limits: The authors frame CausaLab as a controlled synthetic benchmark whose 3 to 7 node SCMs with mostly linear mechanisms and one hard-quadratic extension cover part of the causal structures, functional forms, latent variables and measurement processes found in real scientific domains. Experiments cover a finite set of LLM agents, prompts and interaction budgets, performance may differ with other models, tool interfaces, decoding policies or longer exploration budgets, and the main analyses focus on final predictions and final hypotheses. Interventions are shift-style in place of perfect hard-do operations, the main suite ran each task once, and the authors name overgeneralizing success on templated SCM tasks to real scientific, medical or policy settings as the main risk.
    原文摘要PDF
  20. 1274Research AgentsarXiv:2605.22878

    SciAtlas: A Computable Atlas of Science for Knowledge-Grounded AI Research

    • Problem: Most AI scientists gather scientific knowledge through retrieval procedures designed around individual workflows, and even when agents iterate, these procedures usually leave a transient collection of documents used for one task. The authors see this as a lack of breadth, depth and standardization, and note that existing scientific knowledge graphs are often bounded by particular disciplines, information objects or schemas, which limits their shared use across research workflows.
    • Input and output: This is a system paper that builds a scholarly knowledge graph, a shared retrieval mechanism and three research workflows. It takes a research object such as a topic, research question, seed paper or idea, grounds it in seed nodes of the graph, and returns ranked paper and author pools. Research Development Trajectory (RDT) and Research Opportunity Discovery (ROD) mainly use the paper pool and Innovation Assessment (IA) also uses the author pool, producing respectively a review outline organized by period and research thread with section-level evidence packs, a motivation, mechanism, falsifiable hypothesis and phased experimental plan, and reviewer-specific assessments, a meta-review and revision advice.
    • Method: Under one schema, the graph centres on papers and holds five layers: citation and publication relations, normalized keywords with co-occurrence relations, a topic-to-domain disciplinary hierarchy, authorship, affiliation and co-authorship relations, and peer-review and evaluation signals attached where available. For retrieval, an LLM parser splits the input into typed query units such as concepts, papers, authors and topics, symbolic indexes over names and identifiers and approximate nearest-neighbour indexes over dense embeddings pick seed nodes, and relevance propagates over a query-centred local graph expanded for a fixed number of hops by mixing a restart term back to the seeds with diffusion from neighbours until scores converge or reach the iteration limit. On top of this, RDT diagnoses missing periods and research threads to repeat retrieval rounds, ROD condenses field bottlenecks into a Research Structure Signature to find same-field and cross-domain mechanisms, and IA splits an idea into motivation, method, experiment and impact claims to build claim-level evidence, reviewer profiles and rubrics.
    • Implementation: The primary source is OpenAlex, and the normative layer was filled by an automated search agent that located public peer-review pages and attached them to papers. The resulting graph spans 26 disciplines and links more than 157M entities through 3B relational triples, including over 43M papers, and configurable LLM components in the three workflows and the foundation evaluations used GPT-5.5. Per the README, the GitHub repository holds a pip-installable client and CLI for the hosted SciAtlas API, workflow code for literature review, idea evaluation and idea generation, and an Agent Skill pack for coding agents such as Codex and Claude Code, while graph retrieval runs through the hosted API with a personal API token.
    • Results: On SurveyLens, made of 1,000 human-written surveys in 10 disciplines, SciAtlas-assisted RDT reached a macro-average Rubric Score of 4.82 against 4.67 for the strongest baseline, SurveyForge, and on LiveIdeaBench, made of 1,180 keywords across 22 domains, ROD scored a mean of 7.38 against 7.12 for the strongest baseline, AI-Scientist. On SciAtlasEval-Idea with 349 ideas, where three LLM judges compared reports pairwise, IA had mean win rates of 99.6% against CoT (chain-of-thought prompting), 96.6% against RAG (retrieval-augmented generation), 74.5% against DeepReviewer, 78.5% against NoveltyAgent and 68.2% against ScholarEval. On 200 SciAtlasEval-HQR (Heterogeneous Query-Object Retrieval) cases with direct query-target edges masked, nDCG@20 (ranking quality of the top 20 results) was 0.51, nearly three times the 0.18 of the strongest baseline, and TaxoBench paper-retrieval F1 (the harmonic mean of precision and recall) was 32.8 against 21.8 for the strongest baseline.
    • Limits: The authors note that the graph is built by an automated pipeline but updated periodically, so emerging work may enter late, coverage depends on the underlying scholarly records, and the availability of expertise and evaluation signals can vary across disciplines and venues. The study focuses on knowledge-intensive stages and leaves closed-loop experimental validation out of scope, and representing procedures, conditions, measurements and outcomes as explicit graph objects remains future work, so the claims concern scientific-context construction and knowledge-grounded decision support. The graph's scale demands substantial storage, indexing and retrieval infrastructure, so propagation is restricted to query-centred local neighbourhoods; SciAtlasEval was curated by the authors, and correctness gains on the single-paper PubMedQA and SciFact tasks of ScholarQABench were comparatively small.
    公开代码原文摘要PDF
  21. 1273Research AgentsarXiv:2604.28158

    Intern-Atlas: A Methodological Evolution Graph as Research Infrastructure for AI Scientists

    • Problem: Scholarly platforms such as Google Scholar, Semantic Scholar, and OpenAlex treat the paper as the unit and link papers by citations, so whether a citation extends a method, resolves its bottleneck, or is mere background stays buried in text. The authors argue that humans reconstruct this lineage in their heads, while AI research agents cannot reliably reconstruct method evolution from unstructured text.
    • Input and output: This is a dataset paper that also proposes three operators running on top of the graph. The input is 1,030,314 AI papers from 1965 to 2025, and the output is a graph with 8,155 method nodes, 9,545 aliases, and 9,410,201 typed edges, where every non-background causal edge carries verbatim quotes as evidence. Given a method name, an idea description, or a research question, the operators return evolution chains, five-dimension scores, or a new research proposal with evidence.
    • Method: An LLM classifier reads the text around each citation and labels the edge with one of seven types (extends, improves, replaces, adapts, uses_component, compares, background), and the first four form the strong-causal subgraph used for lineage search. Each causal edge gets bottleneck, mechanism, and trade-off spans quoted verbatim from the citing paper plus a confidence, and a deterministic checker discards edges whose quotes fail substring match or whose endpoints violate publication-year order. Lineage search uses SGT-MCTS (Self-Guided Temporal Monte Carlo tree search), which adds an edge-confidence and year-gap prior to UCT (the MCTS selection rule that balances high-scoring and under-visited branches), idea evaluation computes five dimension scores from graph statistics instead of LLM judgment, and idea generation requires every proposal to cite the stored bottleneck quote of a real edge.
    • Implementation: PDFs are parsed with the Nougat v1.0 parser, with the GROBID v0.8.0 parser as the fallback for the 3.7% of documents where Nougat fails, and extraction reads only the Introduction, Method, and Related Work sections of each citing paper. Both extraction phases use Qwen3.6-35B-A3B at temperature 0, with a separate audit model used only in the reliability benchmarks. The authors state that they release the graph and pipeline as open infrastructure.
    • Results: On a benchmark built from 30 surveys (2,268 nodes, 1,462 edges, 133 chains), Intern-Atlas reached a node match ratio of 91.0%, an edge reachable ratio of 89.7%, and a path semantic correctness of 92.0%, and SGT-MCTS reached a node recall of 84.8 against 44.9 for the strongest baseline, Beam@10 (beam search with width 10). On 1,200 papers, mean idea scores followed the strata order of 8.48 for top-tier conferences, 7.83 for core conferences, 6.85 for workshops, and 5.84 for rejected submissions, and on 100 ideas rated by 10 experts the overall Spearman correlation was 0.81 against 0.58 for a pure LLM judge. Ideas generated for 100 research questions scored 7.20 overall, 1.02 points above the best of four baselines (Semantic Scholar, 6.18), and won 88.0%, 82.0%, and 81.0% of blind expert pairwise comparisons against No-KB (generation without external knowledge), OpenAlex, and BM25 RAG (generation grounded in papers found by BM25 keyword retrieval).
    • Limits: The authors report Phase-1 edge-type accuracy of 70.4% for the production model and 93.0% for the audit model, reflecting how hard fine-grained relations such as extends versus improves are to separate. The 14-axis bottleneck taxonomy is fixed at release, the temporal coherence function (the weight given to publication-year gaps) is calibrated on post-2015 AI literature and needs recalibration for fields with different research cadences, and the generated ideas were scored with the same Intern-Atlas evaluation pipeline.
    原文摘要PDF
  22. 1272Information RetrievalarXiv:2604.20857

    DiagramBank: A Quality-Audited Dataset of Scientific Schematic Diagrams with Multi-Level Document Context

    • Problem: Schematic diagrams in papers convey methods, workflows, and system structure, yet existing scientific-figure datasets often mix them with plots, screenshots, and photos and often omit the body text that explains each figure. The authors argue for a resource that identifies schematic figures and keeps paper and figure context, provenance, and filtering uncertainty together.
    • Input and output: This is a dataset paper, and the authors limit the contribution to the resource itself. The input is PDFs and metadata of AI/ML venue papers hosted on OpenReview, and the output is records for 57,100 schematic diagrams. Each record carries the figure image, paper title, abstract, caption, body paragraphs citing the figure, venue and year, provenance, and filtering labels.
    • Method: PDFs and metadata come from the OpenReview Python API, figures and captions are extracted with PDFFigures 2.0, and body paragraphs citing the figure number are extracted with PyMuPDF. OpenCLIP ViT-B/32 first sorts figures into diagram, plot, photo, and other, and a VLM cascade selects the final diagrams through votes from two smaller VLM judges and confirmation by a larger VLM. Candidates that CLIP labels as diagrams enter a majority-vote path, and non-diagram candidates with CLIP confidence below 0.70 enter a rescue path.
    • Implementation: Relational data are managed in DuckDB, and records are denormalized so that each figure directly holds its title, abstract, caption, context, quality labels, and provenance. The release includes an all-CLIP view and a view with CLIP confidence of at least 0.85 alongside the cascade-filtered release. The official repository README describes a download script that fetches data.jsonl, FAISS indices, and DuckDB from Hugging Face and a demo notebook that finds similar diagrams from a title, abstract, and caption, and it states that the reproduce scripts cover OpenReview collection through first-stage CLIP classification.
    • Results: A blind human check of a stratified sample of 500 records from the cascade-filtered release estimates precision at 93.67% (95% confidence interval 90.11 to 97.22), with screenshots or photo-like figures as the main error. In the CLIP view check, precision is 94.4% at a threshold of 0.85, and raising the threshold from 0.50 to 0.90 lifts precision from 64.0% to 96.0% while retention falls from 100% to 55.7%. In a retrieval check with 20 queries, Precision@10 (the share of the top 10 retrieved diagrams that VLM judges rate as relevant) is 78.0% for hierarchical retrieval and 81.0% for caption-only retrieval.
    • Limits: The authors state that relying on automated PDF parsing, figure classification, and context extraction can leave records with imperfect captions, missing body-reference paragraphs, or diagram versus plot classification errors. The release favors precision and may drop useful lower-confidence diagrams, and coverage is limited to accessible OpenReview venues and years, which may tilt the resource toward recent AI/ML practice. The retrieval check is a sanity check at the scale of 20 queries, the authoring case is qualitative, and the authors say the VLM judge model names are kept in the release documentation.
    公开代码原文摘要PDF
  23. 1271Research AgentsarXiv:2604.18805

    AI scientists produce results without reasoning scientifically

    • Problem: LLM-based scientific agents (AI scientists) increasingly run research on their own, yet benchmarks for them mainly score task completion, such as answering domain questions or finishing a workflow. Such scores show whether an agent succeeded but not how it got there or how it will behave on new problems, so the authors measure the reasoning process itself.
    • Input and output: This is an empirical study. The inputs are tasks in 8 scientific domains: workflow tasks (molecular simulation, adsorption surface construction, ML-based property prediction, atomic force microscope experiments), a strategic task (retrosynthetic planning), and hypothesis-driven tasks (spectroscopic structure elucidation, inorganic qualitative analysis, circuit inference). The outputs are scores from more than 25,000 agent runs, diagnostic question scores, and graphs of reasoning traces made of nodes and edges.
    • Method: The authors model an agent as a language decision process, a partially observable Markov decision process with text observations and actions, whose policy combines a base-model part and a scaffold part, and they cross 3 models (GPT-4o, Claude Sonnet 4.5, GPT-OSS-120B) with 2 scaffolds (ReAct and structured tool calling) on the same environments and tools. They fit item response theory (IRT) to per-domain knowledge and reasoning questions to estimate abilities, then split the variance in success with M7, chosen by PSIS-LOO (an approximation of leave-one-out cross-validation) among Bayesian models M1 to M8. Each trace step is labeled as hypothesis, evidence, test, judgment, update, or commitment, and the share of traces showing productive motifs and reasoning breakdowns is counted on the resulting graphs.
    • Implementation: The authors built Corral, an evaluation framework that separates environments from agents through an HTTP interface, and its environments rest on LAMMPS simulations, a live atomic force microscope, a custom thermodynamic equilibrium engine, and manually curated spectra, covering more than 15 environment scopes and more than 90 tools. The paper states that tasks, run reports, traces, and question data are public in a Hugging Face collection. The official repository README describes a corral command-line interface (CLI) with run and bench commands and a Python API, the afm, corral_md, resistor_network, retrosynthesis, spectra_elucidation, and wetlab environments plus a lightweight math environment called samplemath, and agents such as Reflexion, LLMPlanner, and AI Scientist in addition to the ReAct and tool calling agents used in the paper.
    • Results: In the selected M7 model, reasoning ability accounts for 41.4% of the explained variance and environment with scope for 30.1%, while the scaffold accounts for 1.5% and tool description verbosity for 0.1%. Gathered evidence goes unused in 68% of traces, untested claims appear in 53%, and refutation-driven belief revision appears in 26%. In spectroscopic structure elucidation, inorganic qualitative analysis, and retrosynthetic planning, injecting early successful steps does not raise performance, and gains appear only with near-complete successful trajectories.
    • Limits: The authors treat each task as an independent episode, test two simple scaffolds (ReAct and tool calling), and leave advanced orchestration such as plan-then-execute pipelines, hierarchical planners, and multi-agent setups out of scope. They use 3 models, and the main evaluation fixes a budget of 20 to 40 LLM calls per task and temperature 0.0, and the log-probability analysis covers only GPT-OSS-120B. GPT-OSS-120B produced malformed responses in 41% of traces, and the authors measured them without fixes such as constrained decoding.
    公开代码原文摘要PDF
  24. 1270Research AgentsarXiv:2604.15145

    An Axiomatic Benchmark for Evaluation of Scientific Novelty Metrics

    • Problem: Automatic metrics for a paper's novelty help AI scientists avoid spending compute on already explored ideas, yet existing metrics are generally validated against confounded signals such as citation counts or review scores. When the authors take the 100 highest- and 100 lowest-rated papers with strong reviewer consensus from ICLR 2022 and 2023, these novelty buckets match the accept/reject decision for 93.5% (technical novelty) and 97.0% (empirical novelty) of papers, so reviewer novelty scores are entangled with quality.
    • Input and output: This is a benchmark that compares novelty metrics without ground-truth novelty labels. A metric under test receives a focal paper and a reference pool of prior papers, the pool is changed in controlled ways, and each focal paper counts as a pass or fail depending on whether the score moves in the expected direction, giving a pass rate per probe. The benchmark consists of 10 tasks in three domains (NLP, computer vision, and biomedical AI), with 100 focal papers randomly sampled per task.
    • Method: Three axioms are tested through probes. The Redundancy axiom says the score must fall as the pool covers more of the paper, probed with an exact copy, a paraphrase, and planting 25% to 100% of the paper's atomic claims into separate pool papers. The Relevance axiom says the score must rise when the pool is swapped for another field or the paper's cited works are removed, and the Temporality axiom says the score must fall as the pool's time window moves later.
    • Implementation: The data comes from the PapersWithCode archive on Hugging Face (dated July 29, 2025), using tasks with roughly 1,500 tagged papers each, with references fetched through the Semantic Scholar API. The 10 evaluated systems are the embedding metrics Yin, RND (relative neighborhood density), SemNovel, and FastTextLOF, plus NovaScore, the AI Scientist judge, RAG-Novelty, ARC (the lexical novelty gate of AutoResearchClaw), and the Si pairwise tournament and OpenNovelty, which give rankings or text instead of scores; systems that retrieve through tools are sandboxed so retrieval reaches only the constructed pools. The authors state that they release arXiv IDs, derived files, and scripts that fetch paper texts, and API usage totaled just under $400.
    • Results: Embedding metrics such as Yin and RND catch nearly all exact copies (1.00, 0.99) and paraphrases (0.99, 0.97) but reach only 0.66 and 0.53 when 25% of the claims are planted. RAG-Novelty, where an LLM reads the paper with retrieved neighbors, passes only 0.28 even on the exact-copy probe and fails every redundancy probe. On the monotone temporal probe the best of the eight scoring systems reaches 0.24, and the authors conclude that surface redundancy is largely solved while conceptual redundancy remains open.
    • Limits: The authors note that pools hold only one to two thousand papers per task, so performance at the scale of the real literature cannot be tested, and that all tasks are AI research tasks. Sandboxing controls what an LLM judge retrieves but leaves the model's parametric knowledge in place, and since the axioms are necessary conditions, passing every probe does not guarantee a good novelty judge. The AI Scientist judge and the Si tournament were run on three tasks and OpenNovelty on two, and the Si tournament and OpenNovelty were tested only on the Redundancy axiom.
    原文摘要PDF
  25. 1269Research AgentsarXiv:2604.04872

    Synthetic Sandbox for Training Machine Learning Engineering Agents

    • Problem: An MLE (machine learning engineering) agent must run a full ML pipeline of data preprocessing, model training, and metric evaluation at every rollout step, and a single code execution averages nearly 200 seconds on standard problems. This latency makes trajectory-level on-policy RL far too slow, so prior work retreated to SFT or step-wise RL with offline proxy rewards.
    • Input and output: This is a training method paper with an implementation and experiments. The input is 60 seed tasks taken from MLE-bench, and the output is a set of synthetic MLE tasks whose training data hold only 50 to 200 samples (data, hidden rules, evaluation scripts, task descriptions), together with Qwen3 policy models trained with RL on these tasks.
    • Method: A Data Strategist extracts a Structural DNA (the mathematical structure of the task stripped of domain context) from a seed task, adds a new domain and noise, and defines a hidden rule linking features to labels, while an ML Developer generates the data and trains and evaluates baseline methods to fix milestone thresholds. An MLOps (machine learning operations) Engineer writes an evaluator that scores against a hidden test set, a Technical Writer writes the task description, and only tasks whose thresholds keep the required order stay in training. Training is trajectory-level GRPO with ReAct rollouts, and the dense reward splits weights across format compliance, successful execution, and passing the median, bronze, silver, and gold thresholds.
    • Implementation: Starting from 1,200 synthetic tasks built from the 60 seeds, execution checks and threshold checks leave 912, of which 848 are used for training and 64 for validation. Qwen3-8B, Qwen3-14B, and Qwen3-30B-A3B-2507 are trained with the RLLM framework, with at most 20 turns and a 90-second execution limit per step during training and one NVIDIA H200 GPU per task. The reward weights are 0.1 for format, 0.3 for execution, 0.1 for median, 0.2 for bronze, 0.2 for silver, and 0.1 for gold.
    • Results: With code written by Gemini-2.5-Flash, the average execution time falls from 196.17 seconds on the original MLE-bench tasks to 14.31 seconds on the synthetic tasks. Measured with ReAct on MLE-Bench-Lite, the Any Medal rate rises from 13.6% (Base, and Seed-SFT trained by SFT on Claude-4.5-Sonnet trajectories) to 22.7% for 8B and from 13.6% for Base to 27.3% for 30B, while Claude-4.5-Sonnet reaches 31.8%. On MLE-Dojo, the 30B SandMLE model with the MLE-Agent scaffold reaches a HumanRank score (based on rank among human competitors on the leaderboard) of 38.56, above 29.12 for Base.
    • Limits: The authors report that in a test-time experiment with a growing turn limit, performance peaks at 30 turns and then drops because the context window overflows frequently. The 8B SandMLE model has a valid submission rate of 63.6%, below 68.2% for Base, and on MLE-Bench-Lite with the AIDE scaffold the 30B model's Any Medal rate stays at 13.6%, the same as Base. Evaluation covers MLE-Bench-Lite with 22 questions and MLE-Dojo with 62 tasks.
    原文摘要PDF
  26. 1268Research AgentsarXiv:2604.04347

    RoboPhD: Evolving Diverse Complex Agents Under Tight Evaluation Budgets

    • Problem: Systems such as GEPA and Autoresearch let LLMs iteratively improve prompts, code, and agent architectures from evaluation feedback, but the authors state that no systematic comparison under identical tasks, budgets, and evaluation infrastructure exists. Evaluations may be expensive (for ARC-AGI, even at a cap of $0.25 per problem, 1,500 evaluations can cost up to $375) or the pool of examples may be small, so how the evaluation budget is allocated determines which agents can be found, and existing methods split it between training and validation, where, per the authors, validation evaluations inform selection but do not help improve the next candidate.
    • Input and output: This is a method paper with an implementation and experiments. RoboPhD takes an objective, task background, a seed agent (code and prompts), a pool of training examples, an evaluator that returns a score and diagnostics, and an evaluation budget, and searches for the agent with the highest expected score on unseen test examples. The output is an evolved agent program; on ARC-AGI, for example, a 22-line seed agent became a 1,013-line multi-strategy system.
    • Method: Each iteration, 3 agents compete on 20 freshly sampled training examples and pairwise comparisons update Elo ratings (K = 32), so every evaluation goes to training-time competition with no validation split. The evolution AI, which designs the new agents (here Claude Code with Opus 4.6), receives the Elo rankings, comparative error reports that point to problems where agents diverge, and per-problem diagnostics that the agents print() themselves, creates a new agent, and refines it in the same session against data from a prior iteration (Deep Focus). Six diversity mechanisms, such as a 200-point Elo penalty for clones whose predictions match a competitor, are included, and a 1,500-evaluation budget yields about 21 iterations.
    • Implementation: With a budget of 1,500 evaluations, the experiments cover four tasks: ARC-AGI (the solver, the model that solves the task, is Gemini 3.1 Flash Lite; 400 training and 400 test problems), Can't Be Late (a Python strategy class with no LLM that decides cloud spot instance use), Text2SQL on BIRD (solver Claude Haiku 4.5, co-evolving a database analysis script and an agent), and DocFinQA (solver GPT-4.1-mini with text-embedding-3-small, over 10-K annual reports filed with the SEC that average 123K words). The baselines, which receive the same diagnostics, are GEPA v0.1.1 with Opus 4.6 through the Claude API, an Autoresearch adaptation generalized to arbitrary artifacts with Opus 4.6 through Claude Code, and a King-of-the-Hill variant that pits the current champion against one new challenger each iteration. Per its README, the official andborth/RoboPhD repository releases the optimize_anything() API, per-domain examples that include the four tasks of the paper, and an engine option to choose GEPA or Autoresearch, and evolution requires the Claude Code command-line interface (CLI).
    • Results: On the test sets, default RoboPhD beat both GEPA and Autoresearch on three of four tasks: ARC-AGI 65.8% versus 58.5% and 54.2%, Text2SQL 64.5% versus 60.4% and 60.7%, and DocFinQA 50.4% versus 40.0% and 48.2%. On Can't Be Late, Autoresearch scored best at -87.6 against -90.7 for RoboPhD, and the King-of-the-Hill variant beat the default on ARC-AGI with 67.0%. Turning off Deep Focus lowered all four scores (DocFinQA from 50.4% to 41.2%), and shrinking the validation set from 200 to 100 raised test scores in all eight GEPA and Autoresearch comparisons.
    • Limits: On the simplest task, Can't Be Late (no LLM calls, solvable in under 90 lines), Autoresearch won, and default RoboPhD and the King-of-the-Hill variant were very close, which the authors say warrants further study. In the simulation by the authors, when three agents one accuracy point apart are compared on 20 problems, the best one ranks first only 45.0% of the time against a 33.3% random baseline, so selection relies on Elo accumulating weak signals over iterations, and non-determinism in LLM responses may cause variation across runs. During development, the Autoresearch adaptation found an exploit that read future spot availability from the internal state of the simulator and scored -85.4, so the authors redesigned the task API and stress careful sandboxing between the evaluator and the agent.
    公开代码原文摘要PDF
  27. 1267Research AgentsarXiv:2603.29557

    FlowPIE: Test-Time Scientific Idea Evolution with Flow-Guided Literature Exploration

    • Problem: In scientific idea generation (SIG), most existing methods use a two-stage structure that retrieves literature and then generates ideas, and this pipeline typically retrieves literature once by keywords and semantic similarity. As a result, they receive contexts that are only topically similar and frequently produce similar ideas. The authors argue that such designs risk trapping the LLM-based generator in an information cocoon bounded by its internal knowledge and fixed retrieval results.
    • Input and output: This is a method paper with an implementation and experiments. The input is a research topic or query, and the output is a set of structured ideas made of Motivation, Method, and Experimental Plan. It uses patents from the USPTO (United States Patent and Trademark Office) instead of papers as the literature source and builds a literature graph from citations, shared core technical features, and semantic similarity between patents.
    • Method: Stage 1 treats the query as the root, explores the patent graph with a flow-guided MCTS inspired by GFlowNets, generates an idea for each path, and feeds the score from an LLM-based generative reward model (GRM) back into the flow probabilities of that path. Stage 2 repeatedly applies crossover that mixes the core technical features of two parent ideas, isolation island mutation that pulls literature from distant subgraphs, and tournament selection based on GRM fitness to the initial idea population.
    • Implementation: All methods use GPT-4o-mini as the idea generator, DeepSeek-V3.2 serves as the GRM and fitness evaluator, and patent embeddings come from BGE-M3. The literature data are 16,050 USPTO patents published in the last three months of 2025, and evolution runs for at most 20 iterations and stops when the standard deviation of fitness scores falls below 0.05. The official repository README presents the repository as the paper's evaluation code and lists run commands for Phase 1 (flow-guided MCTS) and Phase 2 (test-time evolution), while the literature database and a software demo are marked as coming soon.
    • Results: On AI Idea Bench 2025 IMCQ, FlowPIE scores 0.780 for motivation and 0.635 for experiment plans, ahead of the strongest baseline VirSci (0.560 and 0.580), and its I2I motivation score (agreement with the target paper's idea) is 4.44. The average reward from the DeepSeek-V3.2 GRM is 0.76 for FlowPIE against at most 0.63 for baselines, and FlowPIE also has the highest average human evaluation score among the compared methods at 0.39. The reward shows Spearman correlations of 0.60 for novelty and 0.87 for feasibility with average human scores.
    • Limits: The authors state that absolute performance stays bounded by the capability of the base model and that differing idea formats across studies make evaluation unstable. The reward evaluation uses the same GRM (DeepSeek-V3.2) as the fitness computation during evolution, and idea quality is measured by LLM judge scores, similarity metrics, rewards, and human ratings. On IdeaBench, Chain-of-Ideas scores higher on Novelty Insight (0.926) and Research Agent scores higher on Feasibility Insight (0.138) than FlowPIE.
    公开代码原文摘要PDF
  28. 1266Research AgentsarXiv:2603.28651

    Not Search, But Scan: Benchmarking MLLMs on Scan-Oriented Academic Paper Reasoning

    • Problem: Current work on academic paper reasoning mostly follows a search-oriented paradigm, where the question pre-specifies a target and the model retrieves a few relevant passages and reasons over local evidence. This works for tasks with clear targets but struggles with researcher-style full-document reasoning and verification, and QA (question answering) benchmarks typically presuppose that an answer exists, embed cues in the question, and evaluation mostly focuses on the final outcome, with limited checks of whether intermediate reasoning is grounded in evidence.
    • Input and output: This is a benchmark paper for error detection in academic papers. The model receives one or more complete papers as page images or OCR text together with a query that names no target, such as 'Assess the Methods section for Sampling & Generalizability issues', and writes free-form reasoning from evidence to the detected error. For scoring, GPT-4.1 extracts the cited evidence and reasoning steps from the answer and matches them against the human-refined gold explanation.
    • Method: Papers come from ICLR (International Conference on Learning Representations) 2024 and 2025 and Nature Communications; on accepted papers Gemini 2.5 Pro proposes coordinated sentence-level edits across sections or pages that create an error, with the question and explanation, and annotators embed the validated edits into the original paper (Generation), and from rejected ICLR submissions it extracts falsifiable errors raised in public reviews and turns them into questions (Sampling). Cross-paper items, edited to conflict with a cited source, are built only by Generation, and errors fall into nine categories: research question and definitions (RQD), design and identifiability (DI), sampling and generalizability (SG), measurement and operationalization (MO), data handling and preprocessing (DHP), computation and formulae (CF), inference and conclusions (IC), referential and citation alignment (RCA), and language and expression (LE). The score multiplies whether the target error was found (0 or 1) by the geometric mean of an evidence location score (a Dice score with a penalty for over-reporting) and a reasoning score (the squared share of gold reasoning steps matched from the start), and by a penalty that falls quickly as unrelated errors grow, giving a value between 0 and 1.
    • Implementation: 10 domain experts reviewed each item in independent pairs, with a third expert resolving disagreements, and of 3,500 initial candidates 1,700 were discarded and 1,541 of the rest were revised (535 question rewrites, 1,207 explanation edits and 1,141 category or metadata corrections). This left 1,800 questions from 715 real papers in 13 natural-science disciplines, and evaluation covered 15 models in 24 input configurations, split into page-image input and Tesseract OCR text input, plus 8 RAG methods. Per the README, the GitHub repository holds GPT-4.1 scoring and aggregation scripts, inference output folders, VolcEngine batch conversion scripts, a PDF-to-image converter, a RAG retrieval metrics script and utils.py with all prompts, while the detailed data sits on Hugging Face for storage reasons.
    • Results: Scores (scaled by 100) were low overall: GPT-5 led image input with an average of 19.2 and Gemini 2.5 Pro led OCR text input with 30.3, yet even Gemini 2.5 Pro stayed below 60 in every error category. Reasoning-enhanced models scored higher, with Qwen3 A22B at 17.4 in Thinking mode against 1.7 for Instruct and DeepSeek-R1 at 11.4 against 1.7 for DeepSeek-V3.1, and across 9 MLLMs text input beat image input by 4.81 points on average, though image input did better on CF, where OCR breaks formulas and tables. With Qwen3-Thinking on text input, the no-RAG score of 17.4 fell to 16.7 with BM25 (word-matching retrieval), 16.6 with Contriever, 11.3 with BGE-M3 and 6.8 with NV-Embed-v2, only the Oracle condition that supplies the gold pages rose to 24.5, and on image input the only method above the Llama 4 Maverick baseline of 7.0 was VRAG-RL, built on a compact 7B model, at 10.9.
    • Limits: The scan-oriented setting is instantiated through a single task, scientific error detection, the source papers come from ICLR 2024 and 2025 and Nature Communications, and the errors consist of synthetic errors generated by Gemini 2.5 Pro and errors drawn from reviews of rejected papers. Initial Gemini 2.5 Pro outputs had substantial hallucinations, logical inconsistencies and low-quality questions, so expert annotation was needed, and scoring relies on GPT-4.1 extraction, whose agreement with human extraction was checked on 200 random questions with a Spearman correlation of 0.841 for the overall score. The authors state that OCR text input severely degrades structured content such as figures and formulas, so questions that need multimodal information lose their evidence and inter-category correlations are artificially inflated.
    公开代码原文摘要PDF
  29. 1265Research AgentsarXiv:2603.28589

    Towards a Medical AI Scientist

    • Problem: The authors argue that existing AI Scientists are largely domain-agnostic and have shown promise in fields with relatively standardized problems and evaluation, such as mathematics, chemistry and general machine learning, but are hard to apply to clinical research that needs medical evidence and specialized data. They name three gaps: ideas that ignore medical priors such as diagnostic workflows and disease-specific pathology, unreliable experiments on heterogeneous medical data such as 3D structures, and manuscripts that largely overlook data provenance and ethics statements.
    • Input and output: This is a system paper that presents Medical AI Scientist, a multi-agent system automating clinical medical AI research, together with the Med-AI Bench benchmark. In Reproduction and Innovation modes the system takes task instructions, dataset information and reference papers, while Exploration mode takes only task instructions and dataset information. It outputs a research idea, executable experiment code with results, and a typeset manuscript.
    • Method: In the Idea Proposer, a Preparer and Surveyor collect reference papers and code and break them into formula and code components, and a Generator forms hypotheses through clinician-engineer co-reasoning, while in Exploration mode an Analyzer first lays out the core clinical and technical challenges and an Explorer finds fitting recent paradigms with code. An Assessor checks scientific quality and ethics, returning hypotheses below the quality threshold for refinement and rejecting unethical ones, and in the Experimental Executor an Investigator gathers code and medical toolboxes, a Planner writes the execution protocol, an Executor runs training and evaluation in a Docker environment, and a Judger reviews the results and returns corrective feedback. The Manuscript Composer follows the structure of reference papers, an Ethics Reviewer reports dataset origin and ethical approval, and LaTeX compile errors are fixed automatically.
    • Implementation: All agents use a general-purpose LLM such as GPT-5 as the base model with role-specific prompting, and the Analyzer retrieves literature through the OpenAlex academic search engine. Med-AI Bench covers 19 tasks across six modalities (medical images, medical video, electronic health records (EHR), clinical text, physiological signals and multimodal data); for each task three papers of different difficulty were taken from Google Scholar as 57 ground-truth papers, and each paper yields 3 cases with different input modes, giving 171 cases. To speed up automated experiments the datasets were randomly subsampled, and the public address given in the paper is a project homepage.
    • Results: In the expert evaluation of ideas from the Innovation and Exploration modes, three experts on a 5-point scale gave the system 4.40 and 4.32 for technical innovation and 4.65 and 4.68 for maturity across the two modes, above GPT-5 and Gemini-2.5-Pro. First-run experiment success over 57 research instances was 0.91, 0.93 and 0.86 in the Reproduction, Innovation and Exploration modes, against 0.72, 0.60 and 0.75 for GPT-5 and 0.40, 0.49 and 0.53 for Gemini-2.5-Pro under the same conditions. On the same medical task (diabetic retinopathy classification), the Stanford Agentic Reviewer scored the generated manuscripts 4.60 on its 0 to 10 scale against 4.86, 4.06 and 3.74 for human-written papers selected from MICCAI (Medical Image Computing and Computer Assisted Intervention), BIBM (Bioinformatics and Biomedicine) and ISBI (IEEE International Symposium on Biomedical Imaging), and separately one generated manuscript was accepted at ICAIS 2025 (International Conference on AI Scientists), which had 114 submissions and a 36.8% acceptance rate.
    • Limits: The authors report that generated method designs can at times become overly intricate, in which case implementation may simplify or degrade components and drift from the original design. They also state that experiments are confined to predefined datasets and that the generated methods still perform below the state of the art. The expert manuscript study used a single task, diabetic retinopathy classification, with 20 papers (5 generated and 15 human-written), and the generated manuscripts scored 3.44 on Coverage (breadth of experimental design) against the 3.68 the paper compares them with.
    原文摘要PDF
  30. 1264Research AgentsarXiv:2603.13327

    DOVA: Deliberation-First Multi-Agent Orchestration for Autonomous Research Automation

    • Problem: For research queries that need academic database search, code repository analysis, cross-checking of model registries, and synthesis with citations, the authors see four limits of single-agent systems. They process information linearly and miss cross-domain links, accept initial findings without adversarial challenge, call tools on keyword patterns, and spend the same reasoning depth on trivial and complex queries, wasting tokens on the former and starving the latter.
    • Input and output: This is a system paper proposing the architecture and algorithms of DOVA (Deep Orchestrated Versatile Agent), a multi-agent platform that answers research queries. It takes a user query, a persistent user model, and conversation context, and returns an answer with source attribution drawn from ArXiv, GitHub, HuggingFace, and the Web, plus a confidence score. The same orchestration backend is exposed through a REST (Representational State Transfer) API, a CLI (command-line interface), a browser UI (user interface), and an MCP (Model Context Protocol) server, which the authors say offers five tools to Claude Code.
    • Method: Before any tool call, the orchestrator weighs the user's expertise, mentioned entities, the six most recent turns, and the available tools to decide whether external information is needed, and temporal words such as latest or recent, or years from 2025 on, always force tool use. Collaborative reasoning chains an ensemble in which agents solve the problem independently, a blackboard where they post evidence and votes, and iterative refinement through rounds of critique. Thinking budgets have six levels from OFF to XHIGH (65,536 tokens), chosen by task type, a complexity hint, and query length.
    • Implementation: The agent pool has five agents: ResearchAgent for MCP search, ProfilingAgent for the user model, ValidationAgent for code analysis and sandboxed execution, SynthesisAgent for writing with source attribution, and DebateAgent, where Bull and Bear agents argue for 2 rounds by default. The experiments use Claude Haiku 4.5 (Basic), Sonnet 4.6 (Standard), and Opus 4.6 (Advanced) by task tier. The official repository alfredcs/dova holds src, tests, frontend, infra, and dova-claude-plugin folders plus Docker configuration files, and its README describes the system as built on the AWS Strands Agents SDK and Amazon Bedrock AgentCore and lists biomedical sources such as PubMed, ClinicalTrials.gov, and PubChem in addition to the four sources in the paper.
    • Results: The authors state that the values in their ablation table (Table 5), which compares seven configurations, are expected relative performance based on architectural analysis. In that table the full DOVA has confidence 0.82 and source coverage 0.90, removing collaboration gives 0.68 and 0.65, and the single-LLM baseline gives 0.52 and 0.00. For token efficiency, based on the thinking budgets, classification drops from 16K to 1K tokens (94%) and summarization by 75% versus a fixed MEDIUM budget, and the authors put aggregate savings at 40 to 60% for workloads where 40 to 60% of queries are simple.
    • Limits: The authors state that the ablation rests on architectural analysis and that evaluation on standard benchmarks such as HotpotQA and MMLU is future work. Confidence is scored by the same LLM that produced the answer, and all agents share one LLM backbone. In-memory MMR (Maximal Marginal Relevance) search cost grows with the number of stored items and results, so very large memory stores need indexing.
    公开代码原文摘要PDF
  31. 1263Research AgentsarXiv:2602.09723

    AI-Assisted Scientific Assessment: A Case Study on Climate Change

    • Problem: Current AI co-scientists are pipelines of parallel LLM calls that search hypothesis spaces at scale, and they have worked well on tasks such as drug discovery or code optimization, where repeatable experiments give immediate ground truth. Some questions in macroeconomics, epidemiology, and climate science allow few controlled experiments, and their ground truth is a state of knowledge reached through scientific consensus instead of measurement, so the authors ask whether AI can help with such verification-poor tasks.
    • Input and output: This is a single case study paper. The AI Assistant takes as input the current document, the papers in its evidence panel (uploaded directly or found through Google Search or a pre-collected corpus of about 1,660 AMOC papers), the chat history, and instructions that scientists attach to the whole document or to a highlighted text span. It outputs an outline draft, section drafts with in-text citations and references, revisions that follow the instructions, reviews of a revision based on rhetorical and argumentative structure, and a summary of changes shown at login.
    • Method: The work followed three phases modeled on assessments by the IPCC (Intergovernmental Panel on Climate Change). In Phase 1 the scientists edited an outline produced by the Assistant; in Phase 2 the Assistant expanded the outline into a full draft and, for each section, went through the whole AMOC corpus in batches to pick relevant sources and add citations; in Phase 3 it merged the sections into one report with a single reference list. AI contribution is measured with a sentence alignment computed by Gemini: Retention is the share of AI draft sentences whose theme survives in the final version, and Intervention is the share of final sentences with no counterpart in the AI draft.
    • Implementation: The system calls Gemini through Vertex AI, using Gemini 2.5 Pro in Phase 1 and Gemini 3 Pro Preview in Phases 2 and 3. About 1,660 AMOC papers were converted to Markdown by an LLM and split into 200-word chunks with 20 words of overlap, and retrieval combines 3072-dimensional gemini-embedding-001 embeddings with BM25, a classic keyword-matching retrieval method. The participants were 13 climate scientists with extensive experience as leads on IPCC and other institutional reports, and the authors describe the tool as a proprietary prototype.
    • Results: The 13 scientists finished the report through 104 versions in 46 hours and 33 minutes of total logged-in time, while participants estimated that a comparable task typically takes 100 to 200 person-hours. For the report, Retention of the AI draft was 93.9% and Intervention was 58.3% (98.8% and 37.2% for the outline), and the scientists overturned the AI claim of 'full scientific consensus' on AMOC weakening and lowered the confidence in anthropogenic attribution from Medium to Low. In the post-study survey, all respondents self-reported a speedup of at least 2x, and 31% reported more than 10x.
    • Limits: The authors state that the findings come from a single experiment in one domain (climate science), on one topic (AMOC), with one group of 13 experts, so there is no formal control condition such as a human-only baseline. The model switch between Phases 1 and 2 is a confound that cannot be resolved analytically, work time is approximated by logged-in time, and the revision attribution is an exploratory Gemini self-evaluation. The AI made a few errors where it picked a single number from a single source when asked to synthesize number ranges, and when pushed adversarially toward a conflicting study it gave that study disproportionate space, which the authors call obsequious behavior.
    原文摘要PDF
  32. 1262Research AgentsarXiv:2602.07040

    Aster: Autonomous Scientific Discovery over 20x Faster Than Existing Methods

    • Problem: The author argues that existing autonomous discovery frameworks, which use LLMs to improve code iteratively, often need hundreds to thousands of iterations per discovery, which restricts them to problems with short evaluations. For tasks with long evaluations, such as multi-hour model training, running thousands of iterations becomes prohibitive.
    • Input and output: This is a system paper that introduces Aster, an AI agent available through a web interface and API at asterlab.ai (the author is at Aster AI Labs), and reports its results on several tasks. The user provides a task prompt, an initial program and an evaluation script, and Aster returns an improved program. The appendix lists the programs Aster produced (the overlap optimizer, the single-cell denoiser, the TriMul (Triangular Matrix Multiplication) kernel, the NanoGPT diff and the ZAPBench (Zebrafish Activity Prediction Benchmark) model).
    • Method: The loop described in the paper has an LLM write a new program from the existing program and the prompt, an evaluator score it, and a database store the result, then repeat. Aster takes the same inputs as existing frameworks (initial program, evaluator, prompt) and aims to cut the number of iterations needed so that evaluations lasting hours become tractable. The paper describes the method only at the overview level of Figure 2.
    • Implementation: The paper states model settings for two circle packing runs: the speed comparison with OpenEvolve used the same model mix (80% Gemini 2.0 Flash, 20% Claude 3.7 Sonnet), and the run that reached the known best used Aster's default model configuration, whose models are not named. Many evaluation scripts and prompts were adapted from the repository of TTT-Discover (the work behind the previous AI records in Table 1), TriMul ran on an NVIDIA H100, and the ZAPBench evaluation ran on an NVIDIA T4 GPU. The access route the paper gives is the web interface and API at asterlab.ai.
    • Results: On packing 26 circles, Aster with the same model mix passed the 2.634 that OpenEvolve reached after 115 iterations with 2.6353 in 5 iterations, and Aster with its default configuration hit the known best of 2.635983 in 6 iterations. It reached 0.380874 on the Erdős minimum overlap problem in 40 iterations (TTT-Discover 0.380876), 0.711 on single-cell denoising in 30 iterations (TTT-Discover 0.709) and 1114 µs on the TriMul kernel in 70 iterations (TTT-Discover 1161 µs), and it cut the NanoGPT Speedrun record from 96.8 to 95.2 seconds in 8 iterations. On ZAPBench it matched the best human model's MAE (mean average error) of 0.0182 after 34 iterations, which the author reports as 190 times less compute.
    • Limits: The speed comparison covers one task, circle packing, against one framework, OpenEvolve, whose 460 attempts generated 4 at a time were counted as 115 iterations. On ZAPBench Aster only matched the best human result, and the paper notes that the tree search system of Aygün et al. reached an MAE of 0.0176 under similar runtime constraints. The 70 TriMul iterations started from the best program of an earlier 94-iteration run on a different GPU image.
    原文摘要PDF
  33. 1261Reasoning / SurveyarXiv:2601.12538

    A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agents

    • Problem: LLMs reason well in closed settings such as math and code benchmarks. Techniques that make intermediate reasoning explicit, such as Chain-of-Thought prompting, decomposition and program-aided solving, often assume static contexts and short-horizon reasoning, and conventional LLMs lack mechanisms to act, adapt or improve in open environments where information changes. Earlier surveys mostly cover reasoning inside the model or agent architecture, so the authors reorganize the field around reasoning as the principle that links planning, action and learning.
    • Scope: This paper is a survey. It covers work up to 2025 on planning, tool use, search, feedback, memory and self-improvement in single-agent and multi-agent settings, and gives separate chapters to applications and benchmarks in math and coding, scientific discovery, embodied agents, healthcare, and web and research agents. The official repository (Awesome-Agentic-Reasoning) publishes a README with paper tables in the same taxonomy, a figs folder with figures and a materials folder with talk slides.
    • Taxonomy: To characterize environmental dynamics, the survey uses three layers: a foundational layer (single-agent planning, tool use and search), a self-evolving layer (feedback, memory and adaptation) and a collective layer (role division, collaboration and co-evolution among multiple agents). Each layer is split into in-context methods, which keep parameters frozen and scale inference-time compute and orchestration, and post-training methods, which move successful reasoning patterns into the weights with RL and fine-tuning. The formalization factors the policy on a POMDP (partially observable Markov decision process) into internal reasoning and external action.
    • Synthesis: Self-evolution is split by what changes: verbal (textual reflections and guidelines, Reflexion), procedural (a library of executable tools and skills, Voyager) or structural (the agent's own source code and architecture, AlphaEvolve). Feedback is split into reflective feedback (self-critique during inference), parametric adaptation (training feedback into parameters) and validator-driven feedback (resampling until a validator such as a unit test passes); the survey notes that the last kind is non-diagnostic, so it corrects individual outputs and leaves the reasoning unchanged.
    • Applications and benchmarks: For research agents, the survey covers systems such as The AI Scientist, which chains idea generation, code writing, experiment execution and result visualization, and CycleResearcher, where a reviewer agent's automated peer-review feedback and an iterative preference-training loop help the research agent refine its next drafts. Benchmarks are grouped into mechanism-level suites (tool use, search, memory and planning, multi-agent) and application-level suites, with examples such as ToolLLM-ToolBench, which collects 16,464 real-world APIs across 49 categories.
    • Limits: The authors list open problems in personalization, credit assignment across tokens, tool calls, skills and memory updates over long interactions, training and evaluating world models in non-stationary environments, multi-agent collaborative training, auditable latent reasoning, and governance for real deployment. They state that existing benchmarks and guardrails mostly test short-horizon behavior and that post-training of multi-agent memory remains largely unexplored.
    公开代码原文摘要PDF
  34. 1260Research AgentsarXiv:2601.03315

    Why LLMs Aren't Scientists Yet: Lessons from Four Autonomous Research Attempts

    • Problem: The authors ask whether state-of-the-art reasoning LLMs can go from a research idea to a paper with a high degree of autonomy, minimal code scaffolding and the most basic tools. Various AI Scientist systems predefine domain-specific workflows or frame the problem in a system-specific way, Sakana's tree-search approach would have required complex meta-orchestration, and AlphaEvolve requires a verification metric set in advance by a human expert. The scope was limited to machine learning research because its experiments can be done entirely digitally.
    • Input and output: The system takes two papers from one subdomain, generates an idea, and aims to move through hypotheses, an experiment plan, code and experiment outputs to an ML paper manuscript. The paper is a case study of four attempts with the authors' own system: starting from more than 135 papers in three domains (world models, multi-agent RL, AI safety and alignment), four ideas were selected for end-to-end attempts.
    • Method: Six agents split the research stages: Idea Generation, Hypotheses Generation, Experiments Planning, Experimental Output Evaluation, Revision and Paper Outlining. Claude Code takes the plan.md and agent.md files from the Experiments Planning Agent, writes the code and runs it on Modal (a cloud execution platform); every artifact stays as a file in a per-idea repository, and each agent receives that repository state in its prompt and works with four tools: read file, write file, list files and llm search. The Experimental Output Evaluation Agent primarily checks hypothesis implementation fidelity and statistical validity, and when it signals failure the Revision Agent chooses between revising the idea, revising the hypotheses suite or requesting LLM mentor feedback.
    • Implementation: All six agents used Gemini 2.5 Pro, which the authors chose primarily for its long context, and Claude Code handled code implementation and paper writing with Claude Opus 4.1 and Claude Sonnet 4. Each domain started from about 45 to 50 papers and produced up to 15 ideas, which were shortlisted with four zero-shot reviewer prompts run on Claude Opus 4.1; the first three ideas tried were chosen with feedback from the seed papers' authors, and the last one through an internal review informed by the earlier attempts. Per the official README, the repository holds four idea folders (three failures, one success) and prompts for idea generation, hypotheses generation, experiment planning and paper creation, and the authors state they released only prompts and selected outputs instead of the full system architecture.
    • Results: Three of the four attempts failed at the execution or evaluation stage, and only AS-1 (semantic entropy for jailbreak detection) was accepted at Agents4Science 2025, a venue requiring AI first authors that accepted 48 of 254 valid submissions. The six failure modes the authors list are falling back to training data defaults such as outdated libraries, implementation drift, memory and context loss over long tasks, overexcitement, insufficient domain intelligence and weak scientific taste; for example, in WM-1, which tried to add differentiable tree search to a stochastic world model, Claude Code hit time limits and swapped the tree search for an actor-critic approach, and the experiment ran with one seed. From these they propose four design principles: Start Abstract, Ground Later; Verify Everything; Plan For Failure and Recovery; and Log Everything.
    • Limits: The authors state that the scope is limited to ML research run entirely in software, that four ideas across three subdomains were each pursued with a single implementation instead of a best-of-N approach, and that because conference submission was the main goal, design changes were not recorded as systematic ablations, so the changes behind one success and three failures are hard to isolate. The failure modes come from qualitative observation, and the release covers prompts and selected outputs instead of the full system architecture. AS-1's experimental design, execution and writing were rated at least 95% AI, yet humans intervened in idea review, paper writing and meta-prompting during execution, and most failed experiments were stopped by human decision.
    公开代码原文摘要PDF
  35. 1259Research AgentsarXiv:2510.10472

    FML-bench: Benchmarking Machine Learning Agents for Scientific Research

    • Problem: Existing ML agent benchmarks mostly use Kaggle-style, application-oriented tasks that stress engineering execution such as feature engineering and standard model training, and they primarily assess final performance and computational cost. They pay limited attention to the ability to tackle fundamental ML research problems such as representation learning and generalization, and they limit analysis of how an iterative research process relates to its outcomes.
    • Input and output: This is a benchmark paper. The agent receives a task description, the complete code of a real research repository, files suggested for editing, protected evaluation code it cannot modify, a command list for running experiments, the baseline performance, and the target metrics, and through repeated cycles of proposing a hypothesis, editing code, and running the commands it is expected to propose a new or improved method with stronger empirical results than the baseline. A post-processing module converts each repository output into a standard format, and the Performance score is the best held-out test result across iterations.
    • Method: The datasets and baselines of the eight tasks are ColoredMNIST and ERM (Empirical Risk Minimization) for generalization, Mini-ImageNet and Prototypical Networks for data efficiency, CIFAR-10 and MoCo for representation learning, splitMNIST and Synaptic Intelligence for continual learning, IHDP (Infant Health and Development Program) and DragonNet for causality, MNIST poisoned with backdoor attacks and dp-instahide for robustness, CIFAR-10 and Wide-ResNet-28-2 for privacy, and COMPAS (a recidivism prediction dataset) and Adversarial Debiasing for fairness, all built on real research repositories such as DomainBed, Lightly, and AIF360. Besides performance and cost, the paper proposes Exploration Diversity, the spread of the GraphCodeBERT embeddings of the code from each iteration around their centroid, Step Success Rate, the share of executed iterations that yield valid results without errors, and Step Completion Rate, the share of assigned iterations actually executed.
    • Implementation: The paper compares three agents with different search strategies: TheAIScientist (broad parallel exploration), AIDE (tree search, run through its commercial variant Weco), and Claude Code (linear refinement instructed by a prompt). TheAIScientist and AIDE used GPT-5 and Gemini-2.5-Pro, Claude Code used Opus-4.1, and each agent ran three independent rounds with a budget of 100 steps, keeping the best test-set result. The README of the official qrzou/FML-bench repository states that the code for the version in this paper lives on the legacy branch, while the default branch README describes a newer version with 18 tasks and 7 agents.
    • Results: TheAIScientist with Gemini-2.5-Pro performed best, ranking first in 4 of the 8 tasks, and AIDE with Gemini-2.5-Pro ranked first in 2. Mean Exploration Diversity was 24.66 for TheAIScientist, 23.50 for AIDE, and 8.75 for Claude Code, and diversity correlated positively with performance in five tasks, of which data efficiency (0.6287) and continual learning (0.8374) had p < 0.05. Step Completion Rate was 1.00 for TheAIScientist and 0.07 for Claude Code, and of the two agents with token records, Claude Code used more tokens than TheAIScientist (9.06M in total) despite its lower performance.
    • Limits: The authors state that when the agent is fixed and the LLM changes, other LLM abilities such as reasoning and code writing also drive performance besides exploration diversity, and the correlation between diversity and performance was negative for causality, robustness, and privacy. AIDE sometimes misread the structure of the target codebase and created classes never wired into execution, and its scores on generalization and data efficiency equaled the baseline, which the authors suggest may stem from AIDE supporting edits to only a single file. The study is limited to 8 tasks, 3 agents, and the best of three rounds per agent, and AIDE ran through the commercial Weco service, which offered limited control over its internals, lacked token usage records, and sometimes ended runs early because of cloud failures.
    公开代码原文摘要PDF
  36. 1258Computer VisionarXiv:2510.08559

    SciVideoBench: Benchmarking Scientific Video Reasoning in Large Multimodal Models

    • Problem: Most existing video benchmarks cover general scenes such as movies, daily activities, and instructional videos, and Gemini 2.5 Pro already saturates them with over 85% accuracy on VideoMME and Neptune. Recent benchmarks that add scientific and educational videos mostly stay at college level, and some MMVU questions can be answered without the video, so a test was needed that jointly measures perception, expert knowledge, and reasoning on research-level experiment videos.
    • Input and output: This is a benchmark paper. A model receives one research experiment video, which JoVE publishes alongside a journal article, and a multiple-choice question, and picks one of 10 options. The 1,000 questions come from 241 videos covering more than 25 subjects across physics, chemistry, biology, and medicine, split into 245 quantitative, 385 hypothetical, and 370 conceptual reasoning questions.
    • Method: Exemplars written by four PhD students, one per discipline, are expanded with GPT-4o into 12 examples per question type, and a QA Generator (the agent that writes questions and answers) based on Gemini 2.5 Pro drafts questions from the video, a Whisper transcript of the narration, and the paper text. The Evaluator answers each question to confirm it is answerable, the Visual Comparer checks that the reasoning is grounded in the video and adds timestamps, and the Refiner rewrites questions to point at specific video segments so they can only be solved by watching the video. Finally, human verifiers check the timestamps and answers, and key details such as mass or temperature that appear only in the transcript are overlaid onto the relevant video frames by a script.
    • Implementation: For conceptual and hypothetical questions, Gemini 2.5 Pro writes nine plausible but wrong distractors, while quantitative distractors are made by adding Gaussian noise to the correct answer. Evaluation runs with the LMM-Eval toolkit on eight NVIDIA H100 GPUs at temperature 0. The official repository dengandong/SciVideoBench contains a README, a figs folder, and a scivideobench task folder for lmms-eval, and the README states that the benchmark is integrated into lmms-eval and that the data is hosted on Hugging Face.
    • Results: In the main evaluation (Table 2), Gemini-2.5-Pro has the highest overall accuracy at 64.30%, and the best open-source model, InternVL-3-78B-Instruct, reaches 38.80%. GPT-4o given only text without the video gets 15.80%, and graduate students, three per discipline answering closed-book, get 17.4%. Chain-of-thought prompting raises Gemini-1.5-Pro's overall accuracy by 21.10 points and GPT-4o's quantitative accuracy from 11.84% to 34.29%.
    • Limits: All videos come from a single source, JoVE, and every question is multiple-choice with 10 options. The only human baseline is three graduate students per discipline answering closed-book. The error-cause analysis compares the reasoning of only Gemini-1.5-Pro-Thinking and Gemini-2.0-Flash-Thinking against expert-verified rationales.
    公开代码原文摘要PDF
  37. 1257RAG / AgentsarXiv:2510.01553

    IoDResearch: Deep Research on Private Heterogeneous Data via the Internet of Data

    • Problem: Scientific data are often multi-source, heterogeneous and multimodal, and lacking unified representation standards and interoperability mechanisms lowers how efficiently they are used (materials science research, for example, typically handles heterogeneous data from experiments, simulations and literature records). Existing Deep Research agents rely mostly on web search with limited exploration of local heterogeneous and multimodal data, so they retrieve private data inefficiently and often fall short of the FAIR principles (findable, accessible, interoperable, reusable data).
    • Input and output: It takes a user's private heterogeneous data (papers, datasets, images, tables and more) and a question or report topic, and returns relevant digital objects, a factual answer or a structured scientific report. This is a system paper that implements and evaluates a data representation and a multi-agent Deep Research system, and it also proposes the authors' own IoD (Internet of Data) Deep Research Benchmark.
    • Method: Raw data are wrapped as digital objects (DOs) with a DOI (Digital Object Identifier) and metadata, long documents are split into Level-2 DOs per chunk, an LLM adds attributes such as summaries, hypothetical questions, classification labels and keywords to the metadata, and domain IoDs are joined into one IoD network. A knowledge refinement layer builds vectors, atomic knowledge (minimal factual units such as a typical aspirin dosage of 300 mg) and LLM-extracted knowledge graphs, and MCP (Model Context Protocol) tools offer retrieval at three granularities (DOs, chunks, facts and subgraphs) through strategies including keyword, vector, graph-reasoning and hybrid search. IoDAgents splits into a Planner that drafts a plan for user confirmation, a Worker Team that filters results with an LLM, refines and reissues queries when context is insufficient and uses a code tool for analysis and visualization, and a Reporter Team whose Writer drafts the output and whose Checker corrects inconsistencies with the retrieved context.
    • Implementation: Documents are parsed with open-source tools such as MinerU, and every compared method runs on the same raw data with the same base LLM, Qwen-Turbo. The benchmark wraps over 500 documents and papers (about 6 million tokens) from four domains (Chinese law, geophysical exploration, computer science, molecular dynamics) as DOs, and has 200 digital object retrieval questions, 800 question answering (QA) questions for retrieval-augmented generation (RAG) (400 single-domain, 400 cross-domain multi-hop) and 60 report-writing questions (30 single-domain, 30 cross-domain). Reports are scored by Qwen-Turbo at temperature 0, averaged over three scorings on five criteria, and by human experts blind to the method who apply the same criteria plus factual accuracy.
    • Results: In Task 1 (digital object retrieval), IoDResearch reached F1 82.64 (precision 76.26, recall 90.18), above LightRAG at 78.93, DO-RAG at 76.21 and Naive RAG at 62.05. In Task 2 (question answering), answer accuracy was 79.98 single-domain and 59.40 cross-domain, above LightRAG at 75.55 and 56.67, the best of the three baselines for this task (Naive RAG, DO-RAG, LightRAG). In Task 3 (report writing), human expert scores were 7.01 single-domain and 6.45 cross-domain, above DeepSearcher at 6.77 and 6.02, the best of the compared methods, and the ablation without the multi-agent component dropped to 6.56 and 5.94.
    • Limits: The authors note that the pipeline is mainly text-based, so multimodal data can lose information during preprocessing and context construction, and they leave an end-to-end multimodal pipeline and post-training strategies to future work. Experiments use the authors' own benchmark across four domains with a single base LLM (Qwen-Turbo), and the LLM judge is the same Qwen-Turbo. The comparison with DeepSearcher covers only the report-writing task, and the conclusion limits the claim to the selected baselines and evaluated task settings.
    原文摘要PDF
  38. 1256Research AgentsarXiv:2506.22419

    The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvements

    • Problem: Scientific findings are trusted when they can be reproduced, so automated science needs automated reproducibility: reimplementing an experiment from its description so that it reproduces the reported outcome. Prior reproducibility benchmarks evaluate on a wide set of often unrelated papers, so the authors build a benchmark that measures reproduction of a chain of cumulative research findings within one task, speeding up LLM training.
    • Input and output: This is a benchmark paper. In each of the 19 tasks the agent receives the previous record's GPT-2 training script (train_gpt2.py), optionally with hints at three levels (pseudocode, a text description, or a paper-like mini-paper), and returns a new training script that reaches a FineWeb validation loss of 3.28 faster. The score is FSR (fraction of speedup recovered), the share of the human next record's training-time reduction that the agent recovers, and a record optimization setting without hints asks the agent to find its own improvements.
    • Method: The authors also release a search scaffold, a program that calls an LLM iteratively to find a solution, that generalizes AIDE. Each node in the search tree is a solution holding the training script and a results file, and each step has the coding assistant Aider make diff-based code edits (implementation), runs the code on an 8xH100 node (execution), and summarizes the output in natural language (analysis). Varying the branching factors and debug probability gives five search variants, Flat (best-of-M), Tree, Forest, AIDE, and Multi-AIDE, which are compared under the same budget of search steps.
    • Implementation: For each of the 19 records, DeepSeek-R1, o3-mini, Gemini-2.5-Pro, and Claude-3.7-Sonnet were run with five search variants, six hint regimes, and three seeds, for 6,840 agent runs in total; each solution had a runtime cap of 60 minutes, and an agent run took about 10 hours on average. The hints were drafted by R1, manually verified, and edited where necessary, and the search used N0 = 3, N = 3, pdebug = 0.5, Dmax = 5, and a budget of M = 20 nodes. Per its README, the official facebookresearch/llm-speedrunner repository releases the scaffold code (core), starter workspaces for the tasks, the hint data for each level, Slurm launch scripts for the paper's experiments, and the analysis notebooks.
    • Results: Without hints, the speedup recovered by every agent averaged at most 20% of the human solutions. The best model, o3-mini, reached FSR 0.40 to 0.43 across the five search variants with pseudocode hints, and with all three hints combined it reached 0.46 on Multi-AIDE but only 0.24 to 0.39 on the other four variants, below its pseudocode-only scores. In a cumulative run where o3-mini built each record on its own earlier code, it recovered about 60% on the first task, dropped to about 20% on the next, and reproduced no speedup by the third record.
    • Limits: The authors note that the hints are short, manually defined, and handed directly to the agent, leaving a more realistic setup where the agent retrieves external knowledge through function calling and memory for future work. Many records may have been published before the tested models' knowledge cut-off, so the models may have seen the solutions in training, and the authors note that separating memorization from generalization may become more necessary as models begin to saturate the benchmark. The benchmark mainly tests reproduction rather than innovation, and the authors argue that agents which generalize to future advances in machine learning as a whole will need harder tasks spanning multi-file codebases, metrics beyond training time, and distributed training beyond a single node.
    公开代码原文摘要PDF
  39. 1255Machine LearningarXiv:2506.07259

    ALINE: Joint Amortization for Bayesian Inference and Active Data Acquisition

    • Problem: Fields with limited budgets and data, such as clinical trials and materials science, need systems that strategically acquire informative data and perform inference on it right away. Amortized Bayesian inference is fast but typically assumes passive data collection, while amortized data acquisition selects informative points well, but the inference update on new data often sits outside the amortization and may need separate, costly procedures such as MCMC (Markov chain Monte Carlo). Existing amortized acquisition methods usually optimize information gain about all parameters or one fixed predictive target, which makes it hard to switch goals when only some parameters matter.
    • Input and output: ALINE takes the observations gathered so far (context set), the candidate points that could be queried next (query set), and the current inference goal (target set). In a single forward pass it returns the posterior over the targeted parameters or the predictive distribution and selects the next point to query from the candidates. This is a method paper that unifies amortized Bayesian inference and active data acquisition in one model, with experiments on simulated tasks and on the HPO-B benchmark of real hyperparameter performance data.
    • Method: One transformer processes the three inputs together, and an added query-target cross-attention lets candidate points attend directly to the target set so that selection fits the current goal. The policy is trained with policy-gradient RL using a per-step reward equal to the increase in the log-probability of the targets under its own inference head, and the authors show this objective is a variational lower bound on the total information gain. Training starts with a warm-up phase in which only the inference part is trained on randomly chosen points, after which both parts are trained jointly, and gradients from the policy loss update only the policy parameters.
    • Implementation: The two heads share 3 transformer layers (4 attention heads each, internal embedding dimension 32), the inference head represents posteriors and predictive distributions with a 10-component Gaussian mixture, and the policy head is a 2-layer MLP that outputs a probability for each candidate point. The active learning model is trained only on synthetic functions drawn from Gaussian Process (GP) priors, the whole study used about 5000 GPU hours on AMD MI250X GPUs, and training one model takes about 20 hours. The official repository README lists the training script train_aline.py and folders for hydra configuration (config), model, tasks, loss, distributions, utils, and evaluation notebooks, and gives training commands for GP active learning in 1D and 2D, Location Finding, CES, and the psychometric model.
    • Results: On Location Finding, ALINE reached a total EIG lower bound of 8.91, above every baseline including the amortized BED methods RL-BOED (7.70), DAD (Deep Adaptive Design, 7.33), and vsOED (7.30), and on CES it scored 13.50, second to RL-BOED (14.60). In active learning it performed comparably to the best GP-based methods on in-distribution GP functions and had lower RMSE (root mean squared error) than the baselines on 3 of 4 benchmark functions unseen in training. On the psychometric model, when the guess and lapse rates were the targets, ALINE and Psi-marginal, an existing method that can focus on a parameter subset, inferred them significantly better than QUEST+, which targets all parameters together, and ALINE took 0.002 s per step to propose the next stimulus, against 0.07 s for QUEST+ and 0.02 s for Psi-marginal.
    • Limits: ALINE is trained with predefined, fixed priors, so a different prior requires retraining, and it estimates marginal posteriors per parameter, leaving joint posterior estimation to future work. Performance can degrade when deployment observations differ substantially from the training data, and on the Three Hump Camel function, whose output range extends beyond that of the pre-training GP functions, ALINE and ACE-US (the amortized inference model ACE paired with uncertainty sampling) both showed reduced accuracy. The architecture is tailored to fixed input dimensionalities and discrete design spaces, and the authors list dimension-agnostic models and continuous experimental designs as future work.
    公开代码原文摘要PDF
  40. 1254Research AgentsarXiv:2505.18705

    AI-Researcher: Autonomous Scientific Innovation

    • Problem: Systems exist for literature analysis or experiment design, but the authors argue they do not connect the full research workflow from hypothesis generation to publication-quality reporting. They also note that no standardized benchmark measures autonomous research across domains, which makes progress hard to track.
    • Input and output: This is a system paper that presents AI-Researcher, a multi-agent system automating the full research workflow, together with Scientist-Bench, a benchmark for it. Scientist-Bench uses 22 papers from 2022 to 2024 as ground truth and gives the system 15 to 20 reference papers, a research instruction and datasets (Level-2 tasks omit the instruction). The system returns implementation code and a technical report covering background, method, experiments and results.
    • Method: A Knowledge Acquisition Agent selects at least 5 GitHub repositories and gathers the matching arXiv LaTeX sources, then a Resource Analyst splits the idea into atomic concepts and maps each formula to its code. An Idea Generator proposes five directions and develops the most promising one, while a Code Agent and an Advisor Agent iterate implementation and feedback like a student and an advisor. A Documentation Agent writes the paper in three stages: outline, detailed writing, and checklist-based revision.
    • Implementation: All analysis and execution run inside a Docker container, and prototypes are typically tested first on 1 to 2 epochs or small data subsets before full experiments. The paper review agent used for evaluation (an agent that compares two papers and scores them) rated the accepted paper higher in 65.62% to 90.62% of 32 ICLR (International Conference on Learning Representations) accept-reject pairs from 2021 to 2023, depending on the LLM judge. According to the README, the GitHub repository holds the research_agent and paper_agent code, the benchmark data and its collection process (benchmark and benchmark_collection folders), a Dockerfile, a gradio-based web GUI (graphical user interface) and example generated papers with code (examples folder), and the README explains how to pull the runtime Docker image (tjbtech1/airesearcher:v1) or build it from the Dockerfile.
    • Results: With Claude-series models over the whole benchmark, the completeness rate (executable code within the inference budget) was 93.8%, and correctness, scored from 1 to 5 by a Judge Agent using an Advisor Agent's analysis, averaged 2.65. In Level-1 pairwise comparison of generated papers against the ground-truth papers (Table 2), five LLM judges rated on a -3 to 3 scale, and the share of generated papers scoring at least -1.0 ranged from 13.64% to 81.82% depending on the judge. In Level-2 without instructions (Table 3), the same share ranged from 50.00% to 100.00%.
    • Limits: In the failure analysis, the authors report that generated research relied predominantly on conventional techniques and standard formulas with shallow theoretical analysis, and that in one Level-1 knowledge graph recommendation task it missed the Gumbel reparameterization the human researchers used. The current system carries information through the LLM context window and summaries of earlier stages in place of an external memory system, and the authors also note that LLM judges tend to overvalue presentation over substance. The experiments cover 22 papers in four AI domains (6 Level-2 tasks per Table 1), and the summary ranges in Sections 4.3 and 4.4 differ from the values in Tables 2 and 3.
    公开代码原文摘要PDF
  41. 1253Research AgentsarXiv:2503.08979

    Agentic AI for Scientific Discovery: A Survey of Progress, Challenges, and Future Directions

    • Problem: LLM-driven agents have made strides in automating general research workflows such as citation management, document discovery, and academic survey generation, but the authors note that these systems often lack the domain-specific focus and compliance-driven rigor needed in fields like biomedicine, where structured assessment of literature is critical. For example, Agent Laboratory showed high success rates in data preparation, experimentation, and report writing but its performance dropped significantly in the literature review phase, and system reliability, reproducibility, and ethical governance continue to pose significant hurdles.
    • Scope: This is a survey of agentic AI for scientific discovery that covers the definition of agents and single-agent versus multi-agent designs, a taxonomy of systems, literature review agents, discovery systems in chemistry, biology, materials science, general science, and machine learning, implementation tools, datasets, evaluation metrics, and open challenges. The authors split the research lifecycle into four steps, ideation, experiment design and execution, data analysis and interpretation, and paper writing and dissemination, and describe what agents take on at each step.
    • Taxonomy: Fully autonomous systems automate end-to-end workflows with minimal human intervention, including Coscientist, a GPT-4-powered agent that plans, designs, and executes chemical experiments, ChemCrow, which integrates 18 expert-designed tools, and ProtAgents for protein design. Human-AI collaborative systems work with researchers, including Virtual Lab, which organizes interdisciplinary research through team meetings and individual tasks, BioPlanner, which designs experimental protocols and leaves their execution to researchers, CALMS (Context-Aware Language Model for Science), a lab assistant that interacts with scientists and laboratory instruments, and Agent Laboratory, which takes human-provided ideas through literature review, experimentation, and report writing. The authors summarize that autonomous systems are efficient on well-defined, repetitive tasks but may struggle with tasks that need creativity or domain intuition, while collaborative systems rely on human oversight, data quality, and interpretability and still require manual validation and execution.
    • Cases: In chemistry, the survey describes LLM-RDF, which splits literature search, experimental design, reaction optimization, and data analysis across six LLM-based agents and demonstrated end-to-end synthesis automation on aerobic alcohol oxidation, and the robotic system Organa, which achieved results comparable to human chemists in electrochemistry while reducing execution time by over 20%. In biology, it describes CellAgent, a multi-agent system whose Planner, Executor, and Evaluator roles collaborate, reaching a 92% task completion rate in single-cell RNA sequencing data analysis. These figures are the survey's summaries of the original papers of each system.
    • Tools and evaluation resources: The survey lists implementation tools such as AutoGen, a framework for managing multi-agent systems, MetaGPT, which assigns roles to agents for software development, Letta, an open-source framework for deploying persistent agents as services, and LangChain and AutoGPT. Its dataset table includes LAB-Bench for reasoning and planning in biological research, MoleculeNet for molecular property prediction, ZINC for virtual screening in drug discovery, MaScQA for materials science question answering, and ICLR 2022 OpenReview data for evaluating an automated paper reviewer. For metrics, reasoning and planning typically use accuracy, task completion rates, and response coherence, prediction and discovery use precision, recall, and prediction error, and the authors note that standardized metrics for complex tasks such as multi-agent cooperation in experimental automation and hypothesis generation are still in development.
    • Limits: The authors see literature review as a significant challenge across nearly all approaches: in Agent Laboratory the literature review phase had the highest failure rate compared with the data preparation, experimentation, and report writing phases, and ResearchAgent and The AI Scientist lack structured literature review. They note that emerging areas such as materials discovery and entire research process automation still need comprehensive benchmarks that assess real-world impact and adaptability. As risks they cite compounding errors in highly autonomous agents with little human oversight, possible deviations from established protocols or safety measures in experimental automation, and agents tied to physical systems misreading situations in ways that lead to unexpected escalations or system failures, and they stress AI governance and human oversight.
    原文摘要PDF
  42. 1252Research AgentsarXiv:2502.18864

    Accelerating scientific discovery with Co-Scientist

    • Problem: As publications grow quickly, scientists struggle to combine deep subject expertise with broad knowledge that bridges disciplines. Literature summarization and deep research tools already synthesize existing information, so the authors target a system that proposes novel hypotheses for experimental validation.
    • Input and output: This is an implemented system paper. It takes a research goal written in natural language (from short statements to documents of tens of thousands of tokens and hundreds of paper PDFs) with preferences and constraints, and returns a ranked list of hypotheses and experiment plans plus a research overview written by the Meta-review agent. Scientists can refine the goal during a run and add their own hypotheses and reviews to the tournament.
    • Method: A Supervisor agent turns the goal into a research plan configuration and asynchronously assigns, from a task queue, the Generation (hypothesis generation), Reflection (peer-review style critique), Ranking, Proximity (clustering of similar hypotheses), Evolution (refinement of top hypotheses), and Meta-review (synthesis of reviews) agents. The Ranking agent starts each new hypothesis at an Elo rating of 1200 and runs a pairwise tournament, comparing top-ranked hypotheses through multi-turn scientific debates and lower-ranked ones through single-turn comparisons. New hypotheses from the Evolution agent compete again, and the Meta-review agent collects recurring issues from reviews and debates and appends them to the agents' prompts in the next iteration, so feedback propagates without fine-tuning.
    • Implementation: All agents use Gemini 2.0 models as the base model, with web search, open databases, a scientist-specified repository of papers and experimental data, and specialized models such as AlphaFold as tools. The full source code is not public; the authors publish pseudocode (Supplementary Note 8) and the exact agent prompts (Supplementary Note 9).
    • Results: In a blinded expert evaluation on 11 curated goals, Co-Scientist was the most preferred with an average preference rank of 2.36 (1 most preferred, 4 least), and its novelty and impact ratings of 3.64 and 3.09 out of 5 were higher than those of three baselines, Gemini 2.0 Flash Thinking Experimental, Gemini 2.0 Pro Experimental, and OpenAI o1. In the ablation, the Reflection agent without search gave already published ideas a novelty score of 6.14 out of 10, which fell to 2.38 with search. In the wet lab, the new candidate drug KIRA6 had an IC50 (the drug concentration that halves cell viability) of 10 nM in KG-1a leukemia cells and 180 nM in the non-leukemic control line TK6, an 18-fold separation.
    • Limits: The authors state that Co-Scientist relies on open-access literature, so it can miss paywalled prior art and negative results, and that it inherits the hallucination risk of its base models. They also note that Elo is a self-assessed score without independent ground truth, the expert evaluations are small in scale, and the cell line experiments are an initial biological validation step.
    原文摘要PDF
  43. 1251Research AgentsarXiv:2502.14297

    Evaluating Sakana's AI Scientist: Bold Claims, Mixed Results, and a Promising Future?

    • Problem: Sakana promoted the AI Scientist as automating the entire research lifecycle, from idea generation through experiment design and execution, analysis and paper writing to review, and it drew wide attention, but to the authors' knowledge a comprehensive independent assessment based on direct experimentation had yet to be conducted. The authors point out that the most positive reviewers relied solely on Sakana's information in place of testing the system themselves.
    • Input and output: This is an independent evaluation case study that installs and runs the AI Scientist, followed by recommendations (a position part) for the IR (Information Retrieval) community. The input is a template built by the authors: a prompt on Green Recommender Systems, experiment.py and plot.py that train FunkSVD on an 80/10/10 split of MovieLens-100k and evaluate it with RMSE (root mean squared error), two seed ideas (adaptive learning rates for SGD (stochastic gradient descent) and e-fold cross-validation), and Sakana's default LaTeX templates. The AI Scientist produced 10 ideas with novelty decisions, experiment code edited by the LLM-based coding assistant Aider, manuscripts and its own reviews, which the authors counted stage by stage and checked against the literature.
    • Method: The authors measured each stage separately. For ideas they checked whether the novelty check, which queries the Semantic Scholar API for up to 10 results per query over up to 10 iterations, filters out known techniques; for experiments they counted the characters changed and whether runs succeeded as Aider edited experiment.py up to five times. For manuscripts they examined length, number and age of references, structural errors and numerical errors, they applied the review function to their 7 generated manuscripts and to 10 human-written OpenReview papers (5 accepted, 5 rejected) and compared it with human decisions, and they recorded cost and human effort.
    • Implementation: Setup, coding and experiments were done by the third author, a third-year computer science undergraduate with basic ML knowledge and strong Python skills, and the foundation LLM was gpt-4o-2024-05-13. The main experiments ran FunkSVD on CPU on a university computing cluster, and the authors state that they released their own codebase for replicating the experiments with a Singularity container definition file. The linked GitHub repository SakanaAI/AI-Scientist is the code of the evaluated AI Scientist; per its README it holds NanoGPT, 2D Diffusion and Grokking templates, the paper-generation script launch_scientist.py and an LLM paper-review function, and it assumes Linux with NVIDIA GPUs using CUDA (NVIDIA's GPU computing platform) and PyTorch.
    • Results: The AI Scientist judged all 10 generated ideas and both seed ideas novel, including known techniques such as micro-batching for SGD, 5 of the 12 ideas (42%) failed to run because of unresolved coding errors, and on a 6,260-character template Aider added on average 529 characters (+8%) in the first iteration and then 118, 83, 66 and 21. The 7 manuscripts had a median of 5 references, only 5 (14.7%) of all 34 references were from 2020 or later, 4 (57%) had structural problems such as missing or misplaced figures, incomplete sections like 'Conclusions Here' and duplicates, and 4 (57%) contained incorrect or hallucinated numbers. The review function rejected 9 of 10 human-written OpenReview papers, including 4 that human reviewers had accepted, and counting only OpenAI API costs the main experiments cost 42 dollars (6 dollars per manuscript on average) with about 3.5 hours of human effort per manuscript excluding setup.
    • Limits: The authors note that they used one dataset (MovieLens-100k), one research domain (Green Recommender Systems), fixed seed ideas and one template, which may affect generalizability and reproducibility. Setup and experiments were done by one undergraduate, the only foundation LLM was gpt-4o-2024-05-13, and the system, whose pipeline is optimized for GPU deep learning, was tested with a simple CPU recommendation algorithm. The human-paper review test covered 10 papers, which the authors call indicative, and the claim of being about 3 to 11 times faster than human researchers rests on the authors' estimates of at least 20 to 40 hours for an undergraduate and 10 to 20 hours for an experienced researcher.
    公开代码原文摘要PDF
  44. 1250Research AgentsarXiv:2609.26457

    Recursive self-improvement of AI research agents

    • Problem: AI agents have improved the efficiency of the artifacts they produce, such as training and inference, but the efficiency of the research process producing them stays fixed. Research and development (R&D) has long shown diminishing returns as cumulative spending grows, and recent self-improving systems show promise in coding and other domains, but how effective their loops are at raising frontier AI research efficiency was unclear.
    • Input and output: This is a system paper. The input is the code of a starting research agent, AIDE0, a pared-down version of AIDE (AI-Driven Exploration), and a selection benchmark of three task families, ML engineering, heuristic algorithm engineering and harness engineering, where each task has a public score, private held-out scoring and a fixed cost budget. The output is the code of the current best research agent (the incumbent), which changes whenever the grade improves, and the edited target is the harness, the code around the model that controls search, context and verification.
    • Method: In the inner loop, a research agent edits task code while seeing public scores, and when its budget is spent the solution it picks is scored on hidden data; the average of hidden scores across tasks becomes the agent's grade. The outer-loop agent reads earlier agents and their grades, rewrites the code of the current best agent (the incumbent), and accepts the rewrite as the new incumbent only if the grade improves. Separating public scores from the hidden selection score blocks the inner-loop agent from directly optimizing the selection criterion, and giving every agent the same budget makes gains come from a better algorithm rather than extra compute.
    • Implementation: The starting agent AIDE0 is a tree-search agent made by removing the ML-specific parts of AIDE, with draft, debug and improve operators and greedy selection of the highest-scoring solution. The 8-day recursive self-improvement run produced a 100-node trajectory of the initial agent and 99 rewrite proposals, and during it the outer-loop agent, Weco's production agent AIDEhuman, ran on Claude Opus 4.7 while every inner-loop agent was evaluated with Gemini 3 Flash. External evaluations used Gemini 3 Flash for ALE-Bench (AtCoder combinatorial optimization), MLE-Bench (Kaggle ML engineering) and KernelBench (GPU kernel optimization), GPT-5.4 for FML-Bench (research codebase tasks) and Gemini 3.1 Pro for WeatherBench 2 (physics-based weather forecasting), and the paper text gives no code repository for AIDE².
    • Results: In the 8-day run, seven accepted rewrites raised the grade from 0.703 to 0.778, while AIDEhuman scored 0.749 on the same grade. On external benchmarks separated from selection, the final agent AIDE85 versus AIDEhuman scored 1790 vs 1511 on ALE-Bench, 0.722 vs 0.708 on MLE-Bench, 19.9% vs 19.6% on FML-Bench and 0.793 vs 0.404 on the out-of-distribution WeatherBench 2. On 38 KernelBench pairs, a separate task family, the reward hacking rate was 55% for AIDE0, 39% for AIDE47 and 32% for AIDE85, against 39% for AIDEhuman.
    • Limits: Noise compounds across both loops, and if it is high enough one falsely accepted rewrite can derail the later outer-loop search. In the ignition test, which checks whether a discovered agent drives the loop better than the agent that produced it, the arm with AIDE47 as the outer-loop agent ran 3 seeds of 50 steps and ended at a seed-average grade of 0.780 against 0.782 for the AIDEhuman arm, and the authors call the result inconclusive. The discovered agents are complex and hard to interpret, and it is unclear which components drive their performance.
    原文摘要PDF
  45. 1249Research AgentsarXiv:2607.01131

    Autonomous Scientific Discovery via Iterative Meta-Reflection

    • Problem: Most existing LLM discovery systems treat each hypothesis-generation step independently, so they lack a step that surveys what has already been found and reasons about what remains unknown. Classical structure learning can only express edges between variable pairs, and full-pipeline AI scientist systems need initial research objectives or seed hypotheses to guide their search.
    • Input and output: By default the system receives only tabular observation data (optionally with images) and no research question, and outputs a list of validated discoveries, each with a natural-language statement, test code and statistical evidence. This is a method paper that also builds a new benchmark, iNatDisco, from iNaturalist observations. iNatDisco-800 has 800 observations of eight species with nine literature patterns, and iNatDisco-50K has 50,000 observations of 9,776 species with 12 patterns.
    • Method: In PROPOSE, an LLM agent reads a data summary, the current claims and the latest guidance, and outputs a hypothesis plus Python code that tests it. EVALUATE uses seven statistical tools, including correlation, group difference, prediction, clustering, stratified re-tests and VLM (vision-language model) visual attribute tests, tunes the code on a training split, runs it once on a held-out split, and accepts a claim only if both splits show an effect size of at least 0.2 and p of at most 0.05. REFLECT analyzes accepted and rejected claims together and returns guidance on gaps, recurring confounds and compound hypotheses; in the experiments it ran every five iterations.
    • Implementation: The main text names Claude Sonnet 4.6 as the default LLM, while the appendix model comparison and baseline description name Claude Sonnet 4.5; the same base LLM also serves as the VLM for reading images, except the DeepSeek V4 Pro run in the model comparison, which was text only. The default setting runs 100 iterations per experiment three times for mean and standard deviation, the counterfactual experiment used 50 iterations, and the iteration-scaling experiment ranged from 10 to 100 iterations. All experiments took about 86 completed runs, about 8,250 API calls, an estimated cost of about 220 US dollars at Sonnet pricing and about 72 hours of wall-clock time.
    • Results: On iNatDisco-800, DiscoPER recovered 8 of 9 literature patterns with a 72.7% support rate, while HeurekaBench-like and ExperiGen-like baselines re-implemented in the same framework found 3 and LLM-assisted classical causal discovery found at most 1. On iNatDisco-50K it found 8 of 12 with a 74.2% support rate, against 6 for the version without REFLECT. On the counterfactual iNatDisco-800-CF data, where five known ecological relationships were reversed in the tabular data, most proposed hypotheses were rejected by held-out validation and the accepted discoveries followed the modified data instead of the LLM's prior knowledge.
    • Limits: The authors state that DiscoPER can only find what the input data supports and that citizen-science data carries known biases, so human verification of any proposed discovery is essential. Valid patterns outside the iNatDisco answer list can exist; recall is judged by an LLM judge (Claude Sonnet 4.5) that counts a partial match (score 1) or better as a discovery, and when the authors manually inspected all exact-match (score 2) judgments in the primary experiments (iNatDisco-800, iNatDisco-50K), 95% agreed. On a synthetic image benchmark it found 3 of 8 patterns, and the authors identify statistical validation of visual features as the bottleneck.
    原文摘要PDF
  46. 1248Research AgentsarXiv:2605.25964

    LECTOR: Joint Optimization of Scientific Reasoning Graphs and Introduction Generation

    • Problem: Most AI-assisted writing methods treat Introduction writing as text generation, for example by designing prompts for general LLMs, which leads to hallucinated citations (nonexistent publications or wrong authorship) and logical inconsistency between the Introduction and the Results and Methodology. Existing Introduction generation tasks such as SciIG condition on the title, abstract, and citations, which the authors see as at odds with real writing, where the Introduction is summarized from the detailed parts of a paper.
    • Input and output: This is a method paper that proposes a new task, CCIG (Content-Conditional Introduction Generation), the RL training method LECTOR, and an evaluation dataset. The input is the paper body without the Introduction (Methods, Results, Analyses) plus an indexed citation list. The output is a reasoning logic graph extracted first, followed by an Introduction written from that graph with citations in [idx] format.
    • Method: The reasoning logic graph is a single-rooted directed acyclic graph (DAG) whose nodes are self-contained sentences from the paper and whose edges come in 6 types that split Peirce's deduction, induction, and abduction, with every conclusion reached by a matched pair of premise edges. One model extracts the graph and then writes the Introduction in the territory, niche, and contribution order of the CARS (Create A Research Space) structure within a single rollout, with both stages sharing weights. Training uses a simplified PPO (Proximal Policy Optimization) with only an actor (policy model) and a critic (value model) and no learned reward model, and the reward sums five parts: Graph Validity, Faithfulness, Paper Consistency, Academic Quality, and Reference Alignment.
    • Implementation: The backbone is Qwen3-4B-Instruct-2507, RL training on verl-agent runs one epoch with a global batch size of 64 and a learning rate of 1×10^-6, and Qwen3-235B serves for reward computation and LLM-as-a-judge scoring. The dataset takes 10,200 physics-related Nature Communications papers across several subfields (April 2010 to March 2025), parsed with MinerU and split into 10,000 training, 100 validation, and 100 test papers. The official GitHub repository holds the RL training (verl), reward computation (agent_system), and evaluation pipeline code, and the README states that the LECTOR-4B checkpoint and the NC_Physics dataset are released on Hugging Face.
    • Results: On the 100 test papers, LECTOR's Overall Performance is 0.665, above base Qwen3-4B (0.510), close to GPT-o3 (0.656), and above Claude-haiku-4.5 (0.612), Grok4 (0.599), and Gemini-2.5pro (0.566). Graph Quality rose from 0.478 to 0.745 and Writing Quality from 0.546 to 0.834, and against the One-Step-Baseline that writes without a graph, Paper Consistency (0.486 vs 0.476) and Citation Quality (0.530 vs 0.477) were higher. In a human evaluation in which 8 domain experts scored 20 test papers on a 1 to 5 scale, LECTOR's overall score of 3.73 was close to GPT-o3's 3.71, and the Spearman correlation between human and LLM judgments was 0.815.
    • Limits: The data are physics-related Nature Communications papers across several subfields, and several reward and automatic evaluation terms rely on Qwen3-235B judgments. In the human evaluation, LECTOR scored below GPT-o3 on Writing Quality (3.91 vs 4.12) and Citation Integration (2.99 vs 3.51), and its first-place share of 40.6% was below GPT-o3's 48.8%. Graph-Writing Alignment fell from 0.682 to 0.623 after RL, which the authors conjecture reflects a shift in generation strategy.
    公开代码原文摘要PDF
  47. 1247Research AgentsarXiv:2603.17216

    ML-AutoResearch: Training Machine Learning Research Agents with Automatically Generated Environments

    • Problem: Training ML research agents needs a large amount of process-level data that includes debugging, failed experiments, and step-by-step reasoning. Existing papers, code repositories, and static benchmarks record only final outcomes, and human-made benchmarks have too few tasks to give the scale and diversity that agentic SFT needs.
    • Input and output: This is a method paper that combines training-data generation with fine-tuning. The pipeline takes a teacher LLM and the HuggingFace dataset hub and outputs executable ML research environments (task description, dataset, baseline code, evaluation script) plus teacher trajectories. The trained student agent works in MLGym by reading and editing files and running shell commands, and has to submit a solution that improves on the baseline within a fixed interaction budget.
    • Method: In phase 1 the teacher samples a topic and proposes a task and dataset name, which is matched to a real dataset through HuggingFace search, and then writes the configuration, starter code, and evaluation script. In phase 2 the task is run once in MLGym, and on errors a self-debugging loop repairs the code from the error log or regenerates it, keeping only environments that run. In phase 3 up to 256 teacher trajectories are collected per task, only those that beat the baseline or pass a Z-score threshold (scores standardized by the task mean and standard deviation) are kept, and trajectories longer than 32,768 tokens are dropped.
    • Implementation: The teacher is GPT-5, and the students are Qwen3-4B, Qwen3-8B, Qwen3-14B, and Qwen2.5-3B-Instruct. Of 1,000 sampled topics, 914 passed dataset validation and 555 of those (60.7%) passed environment verification; about 34K trajectories were filtered to 17K for SFT. The paper checklist states that public repository links and full reproduction instructions are not yet provided.
    • Results: Averaged over three evaluation seeds (42, 123, 456), MLGym AUP rose by 5.7%, 9.1%, and 5.9% at 4B, 8B, and 14B, and the normalized score on MLAgentBench (a benchmark of ML design and debugging tasks) rose by 0.101, 0.171, and 0.294. With added GPT-5.4 trajectories, the 14B model reached a seed-42 best-run AUP of 1.1297, 7.0% above Qwen3-14B. When only the trajectory-generation teacher was replaced with Qwen3-32B, with task synthesis and verification still on GPT-5, mean AUP still rose by 3.0% at 8B and 3.2% at 14B.
    • Limits: Because the training objective is supervised imitation of the teacher, the authors leave direct optimization of exploration and scientific novelty to future work with RL. Task generation and verification still rely on GPT-5, and generating and evaluating long-horizon trajectories needs substantial compute. Results vary by task: the 4B model regressed on MNLI (natural language inference) and the 14B model on Breakout (a MinAtar game task), and 6 synthetic tasks (1.1%) exceed 0.60 cosine similarity to an MLGym task.
    原文摘要PDF
  48. 1246Research AgentsarXiv:2603.08127

    EvoScientist: Towards Multi-Agent Evolving AI Scientists for End-to-End Scientific Discovery

    • Problem: Most current AI scientist systems use static, hand-designed pipelines, and their idea-generation and code-generation strategies stay the same regardless of accumulated interaction histories. The authors point out that such systems overlook promising research directions, repeat failed experiments, and spend resources on infeasible ideas.
    • Input and output: This is a system paper with an implementation and experiments. The input is a user's research goal sentence; the outputs are a research proposal (background, related work, method, experimental plan, expected results) and an execution report with the code that was run, logs, metrics, and failure diagnoses. After each task, two memories are updated and reused on the next goal.
    • Method: The Researcher Agent retrieves related direction knowledge from the ideation memory, builds candidates with an idea tree search that repeats propose, review, and refine, picks the winner of an Elo-rating tournament (pairwise comparisons turned into a ranking score), and expands it into a proposal. The Engineer Agent uses strategies from the experimentation memory to write and run code in four stages (initial implementation, hyperparameter tuning, proposed method, ablation) and fixes failures from the logs. The Evolution Manager Agent writes promising directions from the top 3 ideas, failed directions from failed proposals, and data-processing and training strategies from code search trajectories into the two memories.
    • Implementation: Gemini-2.5-Pro generates ideas, Claude-4.5-Haiku generates code, the Semantic Scholar API (a paper search service) finds literature, and memory retrieval uses the mxbai-embed-large embedding run through Ollama (a local model runner). The idea tree search makes at most 21 candidates, and the maximum attempts for the four experiment stages are 20, 12, 12, and 18. The official repository README presents a research agent tool installed from PyPI (the Python package index) and used through a CLI (command-line interface), a TUI (text interface inside the terminal), or a WebUI (browser interface), built from 6 sub-agents (plan, research, code, debug, analyze, write); this setup differs from the three agents in the paper, and the roadmap still lists the benchmark suite as to be released.
    • Results: Judged by Gemini-3-flash, EvoScientist beat all 4 open-source and 3 commercial systems, with an average gap over the four criteria (Avg. gap) of +29.17 to +93.34 against open-source systems and +46.00 to +80.83 against commercial ones. With 3 PhD-level human judges it reached an average novelty win rate of 82.50% and a feasibility win rate of 64.17% across four comparisons, and code execution success averaged over four stages rose from 34.39 to 44.56 after experiment strategy evolution (distilling execution strategies from code search trajectories into memory). All 6 papers the system produced were accepted at the ICAIS 2025 AI Scientist Track, which accepted 26 of 82 submissions.
    • Limits: The authors state that the evaluation covers computational research tasks whose hypotheses can be tested by simulation and code execution, and that extending to fields that need physical experiments, such as materials science or drug discovery, requires lab workflows and real-world feedback. Execution success in stage 3 (proposed method) rose only from 20.33 to 21.57, and idea quality was judged on 30 queries from researchers. Reviews of the accepted papers repeatedly asked for deeper theoretical formalization, and the manuscripts were written with the paper-writing module of AI Scientist-v2.
    公开代码原文摘要PDF
  49. 1245Research AgentsarXiv:2603.07444

    HLER: Human-in-the-Loop Economic Research via Multi-Agent Pipelines for Empirical Discovery

    • Problem: Existing AI-scientist-style systems mostly aim at full automation with minimal human involvement, largely in ML research. Empirical economics needs questions that the variables in an actual dataset can answer, careful identification strategies (study designs that isolate causal effects), and human judgment of economic significance. The authors point out that ideation without dataset information often yields infeasible or hallucinated hypotheses.
    • Input and output: This is a system paper with an implementation and experiments. The input is one of the supported local datasets (CHNS, the China Health and Nutrition Survey, a longitudinal survey; CMGPD-Liaoning, the China Multi-Generational Panel Dataset of historical demographic records; UK Biobank, a large-scale biomedical dataset with socioeconomic variables), public data APIs such as World Bank, FRED, and OpenAlex, and a research-domain constraint set by the researcher, such as labor economics. The output is a draft paper compiled from Markdown to PDF with Pandoc (a document converter) and LaTeX, together with regression tables, figures, result summaries, and replication files, and a human steps in at two decision gates: research question selection and publication approval.
    • Method: A central orchestrator keeps a shared state object (RunState) and hands work to seven agents in the order data audit, data profiling, questioning, data collection, analysis, writing, self-critique, and review. Question generation is conditioned on dataset diagnostics such as the variable inventory, missingness patterns, distributions, and correlations, and passes through a question quality loop in which feasibility is screened and a human selects. After a draft is written, the Reviewer Agent scores it on a 1 to 10 scale and requests extra analyses such as robustness checks, driving a research revision loop of re-analysis and revision.
    • Implementation: HLER is written in Python, all LLM reasoning runs on Claude Sonnet 4.6 through the Anthropic API, and econometric estimation uses the Python statistics libraries statsmodels and linearmodels for OLS (ordinary least squares), fixed-effects panel, difference-in-differences, and event-study models. The official repository maxwell2732/hler-working-papers holds numbered working papers produced with HLER (Markdown and PDF), an R_script folder, and an image folder, and its README states that the repository contains working papers only.
    • Results: Over 14 runs on three datasets, dataset-aware question generation produced 69 feasible questions out of 79 (87%), against 34 out of 82 (41%) when the model got only a dataset name and domain description. 12 of the 14 runs (86%) finished end to end with no human intervention beyond the two decision gates, and for these 12 runs the mean Reviewer Agent score rose from 4.8 for the first draft to 5.9 after the first revision and 6.3 for the final draft. A run typically took about 20 to 25 minutes, with an average API cost of $0.8 to $1.5.
    • Limits: The authors state that dataset-aware generation reduces infeasible questions but that the theoretical meaning and novelty of hypotheses still need separate checking, and that econometrics is limited to OLS, fixed effects, difference-in-differences, and event studies, leaving instrumental variables, regression discontinuity, and structural models for future work. Reviewer Agent scores come from the same LLM that wrote the manuscripts, so the evaluation may be circular, and the evaluation covers 14 runs on three datasets. Because many hypotheses can be tested quickly, multiple-testing and selective-reporting risks grow; the authors say this lies outside what HLER solves, so researchers must apply corrections and report the full set of questions.
    公开代码原文摘要PDF
  50. 1244Research AgentsarXiv:2603.00582

    Super Research: Answering Highly Complex Questions with Large Language Models through Super Deep and Super Wide Research

    • Problem: Existing Deep Research and Wide Search systems typically operate within 10 to 20 retrieval iterations and about 100 web pages, and each mainly concentrates on either depth or width. How well LLMs handle questions that need long-horizon planning, massive evidence gathering and synthesis across heterogeneous sources remains largely unexplored. The authors judge existing evaluations ill-suited to such questions, citing alignment errors in LLM-as-a-Judge report comparison, shallow fact-recall metrics, costly human evaluation and neglect of uncertainty expression.
    • Input and output: This is a benchmark paper. A system under test receives one expert-written question and returns a long cited research report, and the benchmark scores that report against a prebuilt Research Graph (fact and insight nodes linked by evidence) and a QA (question-answer) exam for that question. There are 300 questions across 10 domains, and the benchmark statistics give 97 reasoning steps on average (142 at most) and 601 retrieved web pages on average (1,267 at most) per question.
    • Method: GPT-4o and a Gemini deep research agent drafted candidate tasks, and Ph.D. candidates and senior practitioners removed hallucinations and kept only tasks that are solvable through public retrieval, more complex than a simple keyword search, and unambiguously stated. The ground truth combines a Research Graph that humans and LLMs built from Planner, Researcher and Summarizer agent outputs by linking URL-anchored fact nodes to higher insight nodes, an expert-checked canonical report, and a QA exam. The score is a weighted average of depth-weighted node recall, the share of conclusions with a complete evidence path to facts multiplied by graph coverage, a QA exam answered only from the report, and stance calibration on contested topics, while single-source dependence is diagnosed separately.
    • Implementation: 12 systems were evaluated: five deep research systems (Gemini, Sonar, Tongyi, o3 and o4-mini Deep Research), two search-integrated agents (Kimi-k2, Grok-4-1-fast), and five baselines that plug DeepSeek-r1, MiniMax-m2.1, Claude-3.5-sonnet, Qwen-2.5-72B and Llama-3.3-70B into a common LangGraph agent with Tavily search. The authors built a web tool for report viewing, graph projection, exam-item editing and final audit, and the abstract gives a leaderboard address.
    • Results: Among the 12 evaluated systems, the top one, Gemini Deep Research, reached only 28.62 Overall, followed by Sonar Deep Research at 27.04 and Kimi-k2 at 26.16, while the LangGraph baselines clustered between 16 and 23. Gemini's Logical Consistency of 22.92 lagged its Coverage of 33.15, and the authors note that logical consistency appears to be a universal performance bottleneck. When 1 to 3 key facts or insights were removed from, or 1 to 3 relevant facts added to, 120 reports, the graph metric moved in the right direction 57.4% to 79.6% of the time, against 14.8% to 22.2% for an LLM judge.
    • Limits: The authors note that such super-complex questions may be rare in standard applications and frame the benchmark as a ceiling evaluation. The sensitivity study used 120 reports from 12 models and 10 tasks, and in the evaluator-swap test the graph Coverage score fluctuated more (4.42%) than the LLM judge (3.88%). The authors also flag the risk that agents synthesizing massive evidence compound hallucinations and invite over-reliance on authoritative-sounding reports, along with high compute cost.
    原文摘要PDF