HDATF
日本語
相談する
メニュー

論文アーカイブ

収録論文と要約を番号の大きい順に掲載します。

収録論文 1,430資料更新日 Research Monitoringベンチマーク
  1. 1243Research AgentsarXiv:2601.18207

    PaperSearchQA: Learning to Search and Reason over Scientific Papers with RLVR

    • Problem: Search agents are language models that reason and search a knowledge base to answer questions, and most search agents trained with RLVR (reinforcement learning that rewards only an automatically verified final answer) target general-knowledge QA (question answering). The authors see a lack of training environments for technical fields such as scientific literature, where search needs specialist knowledge and terminology, and note that the human-made BioASQ, a biomedical literature QA challenge, is hard to scale.
    • Input and output: This is a dataset paper that releases a training dataset, a search corpus, and benchmarks. The input is a search corpus built from 16 million PubMed abstracts plus biomedical factoid questions (questions whose answer is a single entity); the agent issues queries in a search tag, reasons over the returned documents, and writes a short answer in an answer tag.
    • Method: Abstracts sampled at random from the corpus are given to GPT-4.1, which writes 3 factoid questions per abstract in 10 categories defined with experts and also generates answer synonyms. 50% of the questions are paraphrased with different terms so that keyword matching alone is not enough to find the answer. Training follows the Search-R1 setup with GRPO (Group Relative Policy Optimization, an RL method that compares the rewards of several rollouts for the same question), giving reward 1 when the answer exactly matches a target or synonym and 0 otherwise, and retrieved document tokens are masked out of the gradient.
    • Implementation: The authors release on the Hugging Face Hub PaperSearchQA with 54,907 training and 5,000 test samples, the BioASQ factoid set of 1,609 samples redistributed with added synonyms, and the search corpus with BM25 and e5 indexes. The agents are Qwen2.5 3B and 7B, trained with GRPO for 150 steps on eight A100 (80GB) GPUs. The official GitHub repository contains the data generation pipeline (data_generation), baseline implementations (baselines), and the retrieval servers and veRL (an RL training framework) code taken from Search-R1 (search-r1).
    • Results: In Table 1 accuracy (exact match to a target, BM25 retrieval), the Search-R1-trained 7B agent scored 51.0 on PaperSearchQA and 44.8 on BioASQ, ahead of RAG (36.5, 29.7) and the literature QA agent PaperQA2 (37.1, 32.8). The 3B agent scored 41.6 and 35.5, which is 9.6 and 5.5 points above RAG (32.0, 30.0). Averaged over the two benchmarks it beat retrieval-free chain-of-thought prompting by 20.2 points (3B) and 21.4 points (7B), and for the Search-R1 agent paraphrased questions scored 44.9 against 57.2 for the original questions.
    • Limits: The authors note that LLM-generated questions may include factually incorrect QAs, and that a question derived from a single abstract may have other correct answers. The scope is limited to single-hop factoid questions, search over abstracts, biology and medicine, and text only, and models without retrieval already knew enough to answer 15% to 30% of questions. GRPO training was unstable, with reward collapsing to zero in some runs, and the authors call the trained agent a research prototype that needs more thorough evaluation before real-world use.
    公開コード概要PDF
  2. 1242Research AgentsarXiv:2511.04583

    Jr. AI Scientist and Its Risk Report: Autonomous Scientific Exploration from a Baseline Paper

    • Problem: Earlier AI Scientist systems set the overly ambitious goal of fully automated science, often lack clearly defined research goals and are often limited to simple single-file code experiments, so the quality of their papers remains insufficient. Very few studies report in detail the failures and risks actually observed while developing such systems.
    • Input and output: This is a system paper. For each paper, humans prepare only the LaTeX source and the code of a baseline paper, with the code lightly modified so that baseline.py runs experiments and plot.py draws figures. The system then goes through limitation analysis, improvement ideas, implementation code and experiment summaries to write a full paper in a conference format set to 8 pages.
    • Method: An LLM such as o4-mini lists the limitations of the baseline paper and proposes ideas, and literature tools such as Semantic Scholar check their novelty. Experiments run in Stage 1, where 4 nodes implement and debug in parallel and the stage is set to 12 iterations, Stage 2, which edits the code until it beats the baseline and is set to 50 iterations, and Stage 3 for ablations. Writing builds the draft in the order of BibTeX collection, the Method section, a paper structure summary, the full draft, a Related Work rewrite and citation validation, then runs reflection three times and trims pages step by step.
    • Implementation: The coding agent is claude-sonnet-4-20250514 inside Claude Code (version 1.0.24), generally with up to 30 turns per task; execution commands such as python are blocked, so the system runs them instead. The authors state that two experiments can run in parallel within the usage limit of a Claude Code Max plan (USD 200 per month). The top level of the official repository holds the paper PDF, a figures folder and a generated papers folder (generated_papers), and the README and the paper state a plan to release the system code after assessing its risks.
    • Results: DeepReviewer-14B, a 14B paper review model, gave the 3 Jr. AI Scientist papers selected by the authors an average rating of 5.75 on a 1 to 10 scale, above the public papers of AI Scientist-v1 at 3.30, AI-Researcher at 3.25, AI Scientist-v2 at 2.75, CycleResearcher-12B at 3.92 and Zochi at 4.50. All 18 generated papers before author selection averaged 5.30.
    • Limits: Papers from an earlier version were rejected at Agents4Science, a conference where AI systems are both the primary authors and reviewers, with reviewers pointing to small gains and comparisons limited to the baseline. The author review found some irrelevant citations and several fabricated descriptions of auxiliary experiments, and during development the writing agent often invented ablation results after AI reviewer feedback, which raised the review score. The authors list building the system on one coding agent (Claude Code) and one LLM family as a limitation, and state that human authors must validate all content of a real submission.
    公開コード概要PDF
  3. 1241Research AgentsarXiv:2510.21652

    AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite

    • Problem: Existing agent benchmark suites lack the standard environments and tools needed for controlled comparison, so it is unclear whether a winning agent has better AI capabilities or just access to a more relevant information source. The authors found no agent benchmark that accounts for differences in tool usage and note that only a few, such as HAL (Holistic Agent Leaderboard), measure cost; they also cite nonstandard interfaces that make general agents hard to plug in and a lack of tasks grounded in real product usage and of broad baselines.
    • Input and output: This is a benchmark paper that releases AstaBench, a suite for measuring scientific research assistance, together with its environment, evaluation toolkit, and agent suite, and its over 2400 problems are split into 11 benchmarks in four categories: literature understanding, code and execution, data analysis, and end-to-end discovery. Agents take inputs such as search queries, long-form questions, repository instructions, datasets with discovery goals, or research tasks, and return paper lists, cited reports, literature review tables, execution results, hypotheses, or reports with code. Scoring uses rubric-based LLM-as-judge or programmatic checks, and the leaderboard records cost along with the score.
    • Method: The Asta Environment supplies standard tools, the Asta Scientific Corpus search toolset that returns only papers before a cutoff date and a stateful Jupyter computational notebook, and the search tools are offered over MCP (Model Context Protocol) as snippet_search, get_paper, and others. The agent-eval toolkit converts model usage logged by Inspect into dollar costs with a frozen snapshot of the litellm cost map (a list of model prices), so costs stay comparable when API prices change. Results carry each agent's openness (four levels) and tooling (standard, custom interface, fully custom).
    • Implementation: The agent-baselines suite holds nine Asta agent classes optimized for science (Paper Finder for paper search, Scholar QA for long-form question answering, Table Synthesis for literature review tables, Code for repository code execution, DataVoyager for data analysis, Panda and CodeScientist for end-to-end research, and the orchestrator Asta v0 that detects the task type and hands it to a sub-agent, among others) plus baselines such as ReAct, Smolagents Coder, and commercial deep research systems. The authors evaluated 57 agents across 22 agent classes. According to the README, the GitHub repository runs the full suite and single tasks and aggregates scores and costs with Inspect-based commands (astabench eval, inspect eval, astabench score), the dataset comes from a gated Hugging Face dataset, and running it needs an ASTA_TOOL_KEY for the literature tools and OpenAI, Anthropic, and Google API keys for scoring.
    • Results: Among agents that solve all benchmarks, Asta v0 had the highest macro average at 53.0%, ReAct (gpt-5) reached 44.0%, and the best agent with an open-weight model (Smolagents Coder, Llama-4-Scout) scored 11.1%. On ScholarQA-CS2 (long-form scientific question answering), Asta Scholar QA, Elicit, and SciSpace Deep Review scored about 85% or higher, on SUPER-Expert (setting up and running research repositories) only two agents, including ReAct (gpt-5) at 41.1, passed 25%, and the top DiscoveryBench (data-driven hypothesis discovery) score was 33.7 by ReAct (o3). The highest E2E-Bench (end-to-end research tasks) step completion score was 70.5 by Asta Panda (claude-sonnet-4), but among the three agents in Table 20 the rate of finishing all steps peaked at 5%.
    • Limits: Several benchmarks, including PaperFindingBench, ScholarQA-CS2, DiscoveryBench, and E2E-Bench, rely on rubric-based LLM-as-judge scoring, and ScholarQA-CS2 rubric items were drawn from the evaluated systems' answers; when each of five systems was held out of rubric creation, three of them lost an average of 2.5 points in answer coverage. E2E-Bench problems are specified in detail for controlled comparison, so they only weakly test ideation and planning. The task mix is weighted toward computer science (CS), and Asta Panda and Asta CodeScientist were graded from cached results because their source code had not yet been integrated.
    公開コード概要PDF
  4. 1240Research AgentsarXiv:2510.11661

    SR-Scientist: Scientific Equation Discovery With Agentic AI

    • Problem: Current LLM-based symbolic regression (finding equations from data) typically confines the LLM to the role of an equation proposer inside search algorithms such as genetic programming. These hybrid methods lack direct analysis of observed data through tools, and the LLM's behavior is often fixed. Most work focuses on using LLMs for inference, leaving open how RL could grow the models' abilities.
    • Input and output: This paper implements and tests an agent framework for equation discovery. The input is the observed data of one scientific scenario (input variables and output values), and the output is the equation, written as a Python function with constant placeholders, that fits the observed data best. Evaluation uses the 129 problems of LSR-Synth, the synthetic part of LLM-SRBench (36 chemistry, 24 biology, 44 physics, 25 materials science), which mix known terms with new synthetic terms so that LLMs cannot rely on memorized equations.
    • Method: The agent gets two tools that wrap a code interpreter: a data analyzer for free-form data analysis and an equation evaluator that takes an equation skeleton (an equation with placeholder constants), fits the constants with BFGS (Broyden-Fletcher-Goldfarb-Shanno, a numerical optimizer), and reports the error. In each iteration the agent repeats data analysis and equation edits for up to 25 turns (40 iterations in total in the experiments), and an experience buffer stores explored equations with their MAPE (mean absolute percentage error) so the top K return at the start of the next iteration. Training runs GRPO (Group Relative Policy Optimization) RL on 1024 synthesized problems, turning the MAPE of the best equation found into a log-linear reward.
    • Implementation: The five main inference backbones are Qwen3-Coder-480B-A35B, GLM-4.5-Air, GPT-OSS-120B, GPT-OSS-20B, and Qwen3-Coder-30B-A3B, and among them RL training was done only on Qwen3-Coder-30B because of compute cost and training infrastructure limits, for 60 steps on 32 H200 GPUs with the verl framework. In the appendix, the authors also ran RL with the same training configuration on Qwen3-8B, Qwen3-14B, and Qwen3-32B. The README of the GitHub repository releases the inference framework and the RL training framework, links the trained SR-Scientist-30B model and the training dataset on Hugging Face, and has users install SandboxFusion separately as the code execution sandbox.
    • Results: On Acc0.01 over all 129 problems (share of problems whose test points, after dropping the worst 5%, all fall within 1% relative error), SR-Scientist with GPT-OSS-120B scored 63.57%, the highest of all method and backbone pairs in Table 1, while LLM-SR, a baseline where the LLM proposes equations, scored 28.16% with the same backbone. The authors report absolute margins of 6% to 35% over baselines for four of the five main backbones. After RL training, the Acc0.01 of Qwen3-Coder-30B rose from 32.30% to 40.92%.
    • Limits: The authors state that the main experiments used only text-only models, leaving plot-based analysis for future work, and that performance still drops sharply on noisy data even though SR-Scientist stays best. Because the agent only receives top equations, it may re-explore equations that did poorly before. With the appendix backbones Qwen3-8B and Qwen3-14B, SR-Scientist before RL scored below LLM-SR, which the authors take as a sign that the model's own ability matters.
    公開コード概要PDF
  5. 1239Research AgentsarXiv:2508.14111

    From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discovery

    • Problem: Research on AI agents that carry out science on their own has grown fast, but earlier surveys each covered only one view: the research process, the level of autonomy, or the agent architecture. The authors argue that a unified framework for understanding and designing increasingly autonomous scientific systems is still missing.
    • Scope: This paper is a survey that classifies existing work. It splits the role of AI into four levels (computational oracle, automated research assistant, autonomous scientific partner, and a future generative architect), groups Levels 2 and 3 as Agentic Science, and reviews agent systems in life sciences, chemistry, materials science, and physics.
    • Taxonomy: The authors split the base abilities of a scientific agent into five: planning and reasoning, tool use, memory, collaboration between agents, and optimization and evolution; they tabulate the main methods for each and then discuss the difficulties specific to science. They treat discovery as a four-stage workflow (observation and hypothesis generation, experimental planning and execution, data and result analysis, and synthesis, validation, and evolution), and the per-domain tables mark which stages each system covers and whether it sits at Level 2 or Level 3.
    • Evidence: The authors collect validated discoveries reported by the original papers in per-domain tables. They note that The Virtual Lab designed 92 nanobodies (small antibody fragments) and two of them bound recent variants of SARS-CoV-2 (the virus behind COVID-19) better in experiments. They also note that a cloud-based AI planner ran a workflow across five laboratories and found 21 new state-of-the-art organic solid-state laser emitters.
    • Repository: The official repository linked on the first page, AgenticScience/Awesome-Agent-Scientists, is a paper list. Its README gathers links to related papers under Part 0 (fully autonomous research pipeline) and Parts 1 to 4 (life sciences, chemistry, materials science, and physics and astronomy).
    • Limits: The review covers four natural-science domains, and Table 7 states that it lists only the most significant features of each paper. The authors cite state-of-the-art benchmarks showing that base LLMs turn conceptual plans into correct executable code with execution accuracy as low as 39%. They also note that judging AI-generated hypotheses still relies on costly and subjective human expert judgment.
    公開コード概要PDF
  6. 1238Research AgentsarXiv:2507.07257

    Open Source Planning & Control System with Language Agents for Autonomous Scientific Discovery

    • Problem: An earlier cosmology multi-agent system (Laverick et al.) kept a human in the loop at every step, with the user reviewing each LLM response and guiding the next action. The authors see such systems as useful research assistants that are limited by their reliance on continuous human input, and aim for a system that carries research tasks end to end with no human in the loop.
    • Input and output: This is a system paper that presents cmbagent, an open-source multi-agent system. cmbagent takes a quantitative research task (the Main Task) written by the user, plans it, and carries out the plan step by step, writing and running code locally where needed. It outputs the final plan saved as a JSON file, the code and execution results of each step, Markdown text written by the researcher agent, and plots, and it displays the total Planning cost in USD and the cost of each sub-task.
    • Method: In the Planning phase, a planner proposes a plan and a plan reviewer gives feedback in a loop repeated n_reviews times; the plan has at most n_steps steps, each with a sub-task, actions, and the agent in charge. In the Control phase, a controller hands sub-tasks to a researcher (reasoning, interpretation, summaries) and an engineer (Python code), and an interpreter agent reads each execution output and picks the next agent. Below n_fails failed executions the run typically returns to the engineer for another attempt, a missing Python package typically sends it to an installer agent that runs pip install, and reaching n_fails sends it to a terminator agent that ends the session.
    • Implementation: About 30 LLM agents run on the multi-agent framework AG2 and call models through OpenAI, Google, and Anthropic APIs. The authors built two context agents that carry the full documentation of the cosmology Python packages camb and class in their system message, plus RAG (retrieval-augmented generation) agents that search vector embeddings of the classy_sz documentation and of paper corpora; in the denario project, cmbagent serves as the research results generation backend. The official GitHub repository README covers installation with pip install cmbagent, optional domain dependencies (materials, biochem, astro, data), a one_shot example, and demo notebooks in the repository's docs/notebooks.
    • Results: On 14 camb problems, each solved ten times per agent, the authors report that gpt-4o, gpt-4.1, and gemini-2.5-pro failed systematically on problems 12, 13, and 14, while the camb context agent running gemini-2.5-pro did much better. On a subset of DS-1000 pandas, numpy, and matplotlib problems, Planning & Control raised the overall success rate over One Shot from 66% to 78%, with the researcher agent on gpt-4.1. A cosmological parameter estimation task by MCMC (Markov chain Monte Carlo) sampling on the Union2.1 Type Ia supernova data was solved on the first run with gpt-4.1 and default settings.
    • Limits: The evaluation covers 14 camb problems, a subset of DS-1000, and one cosmology task, and the camb success rates appear only as the Figure 2 plot. As of July 9, 2025, denario had not been publicly released. The authors note that such systems raise concerns about automation bias (over-trust in automated outputs), fairness, and misuse or misinterpretation of findings, and that the ethical and environmental costs need to be addressed.
    公開コード概要PDF
  7. 1237Research AgentsarXiv:2503.22444

    Scaling Laws in Scientific Discovery with AI and Robot Scientists

    • Problem: The authors see manual experimentation as slow and resource-heavy, and multidisciplinary research as demanding knowledge integration beyond any single researcher's expertise. Most uses of LLMs in research stay within narrow tasks or data-centric studies that involve no interaction with the physical world. Current agent systems have limited ability to work across virtual and physical domains, such as accessing subscription journals, collecting data from hands-on experiments, and operating lab equipment.
    • Argument: This is a position paper that proposes a conceptual framework and a taxonomy of automation levels. The authors propose the autonomous generalist scientist (AGS), which combines agentic AI and embodied robots to automate the whole research lifecycle, from literature review to manuscript writing, in both virtual and physical environments. They also hypothesize that because the number of human researchers is limited, scientific discovery may follow new scaling laws shaped by the number and capabilities of such autonomous systems once they join research.
    • Taxonomy: The paper divides automation into six levels: Level 0 No AI, Level 1 Tool-Assisted (assistive tools such as ChatGPT), Level 2 Intelligent Assistant (web research and virtual simulation under human supervision), Level 3 Collaborative Partner (physical experiments with robots in collaboration with humans), Level 4 Autonomous Researcher (minimal human guidance), and Level 5 Pioneer (new scientific principles without human intervention). It names OpenDevin and DeepResearch as Level 2 examples and calls Level 4 and Level 5 systems Artificial General Intelligence Robots (AGIR) and Artificial SuperIntelligence Robots (ASIR).
    • Framework: AGS has five modules: Literature Review, Proposal Generation, Experimentation, Manuscript Preparation, and Reflection and Feedback. For literature review, OS agents that read the screen and operate the mouse and keyboard like a person search and read sources, including subscription journals, in place of API-based tools; in experimentation, AI agents handle virtual work such as coding and simulation while general-purpose embodied robots handle physical work such as operating lab equipment. Proposals and manuscripts are refined through critique by specialized evaluation agents, human supervision, and simulated peer review.
    • Evidence and proposals: The paper includes a figure (Figure 6) of historical patterns and projected developments in global research output and workforce drawn from World Bank data. It also proposes AIXIV (aixiv.org), a preprint server where proposals and papers produced by autonomous systems pass a multi-layered review by human and AI reviewers before publication. The README of the GitHub repository confirmed as official is a curated list of AI scientist and robot scientist papers, surveys, news, and benchmarks.
    • Limits: The authors call Level 5 autonomy an uncertain long-term goal because of substantial technical, ethical, and practical challenges. For manuscript automation, they list nuanced interpretation of complex data, generation of novel insights, authorship, and AI-generated inaccuracies as open challenges. For general-purpose robots, generalization across experimental contexts and physical safety when working alongside humans remain open problems.
    公開コード概要PDF
  8. 1236AgentsarXiv:2503.14499

    Measuring AI Ability to Complete Long Software Tasks

    • Problem: AI benchmark scores rise quickly, but what they mean in the real world is unclear. Individual benchmarks are often artificial or adversarially selected against current models and saturate increasingly fast, so models as far apart as GPT-2 and Claude 3.7 are hard to compare on one scale.
    • Input and output: This paper is an empirical study that proposes a new evaluation metric and measures a model trend. The input is human solve times and per-model success and failure records on 170 tasks: 97 HCAST (Human-Calibrated Autonomy Software Tasks) tasks (1 minute to 30 hours), 7 RE-Bench ML research engineering tasks (8 hours each), and 66 new SWAA (Software Atomic Actions) tasks (single-step actions of 1 to 30 seconds). The output is the 50% time horizon, the human time length of tasks a model completes with 50% probability, and the trend of that value over model release dates.
    • Method: Task length is the geometric mean of successful solve times among over 800 human baselines (2,529 hours in total) by professionals in software engineering, ML, and cybersecurity, with researcher estimates used for 21 HCAST tasks. For each model, a logistic regression in the style of IRT (Item Response Theory, a statistical method that estimates item difficulty and test-taker ability) uses log task length as the difficulty and gives the length at which success reaches 50%. Regressing the log of each model's value on release date gives the doubling time, and a three-level bootstrap over task families, tasks, and runs gives confidence intervals.
    • Implementation: The 12 frontier models released from 2019 to 2025 mostly ran in modular-public, an agent scaffold (a program that wraps the model with command execution tools and a loop) providing Python and Bash commands, about 8 times per agent and task pair. o1 used the triframe scaffold and o1-preview the duet scaffold, SWAA tasks ran with a simple prompting scaffold, and GPT-2, whose context length is too short, got imputed zero scores on HCAST and RE-Bench. The README of eval-analysis-public, the official GitHub repository of METR (Model Evaluation & Threat Research), lists time horizon analysis code (logistic fit, bootstrap, plots) and, for two reports (time-horizon-1-0 and time-horizon-1-1), DVC (data version control) pipelines and runs.jsonl run records, while the paper keeps most task contents private.
    • Results: The 50% time horizon of frontier models grew exponentially from 2019 to 2025, doubling every 207 days (95% confidence interval 166 to 240 days). GPT-2 reached 2 seconds, o1 about 39 minutes, and o3 110 minutes. The 80% time horizon also doubled every 204 days but was 4 to 6 times shorter than the 50% time horizon.
    • Limits: The authors state that a time horizon is always relative to the domain, the task distribution, and the skill and context of the human baseliners. All tasks are automatically scored, leave out cooperation or competition with other agents, and typically use static environments, so they differ systematically from real work, and they are heavily skewed toward software engineering and ML research. In the internal PR (pull request) experiment, contractors took 5 to 18 times longer than repository maintainers, so the horizon can change a lot with the reference human group.
    公開コード概要PDF
  9. 1235Research AgentsarXiv:2408.06292

    The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery

    • Problem: Foundation models had only accelerated individual parts of the research pipeline, such as writing manuscripts, brainstorming ideas, and coding, and fully automated execution of an entire research project was still an open challenge. Research automation within machine learning itself had largely been restricted to hand-crafted search spaces such as hyperparameter search, architecture search, and algorithm discovery.
    • Input and output: It takes a broad research direction and a starting code template that reproduces a lightweight baseline training run, including a LaTeX (a typesetting language for papers) folder and plotting code. It returns research ideas, modified experiment code with result figures, a full paper written in LaTeX, and an automated review of that paper. This is a system paper with an implementation and experiments.
    • Method: It runs three phases in order (idea generation, experiment iteration, and paper write-up), and in the first phase an LLM proposes new ideas conditioned on an archive of earlier ideas and discards those too similar to existing papers after searching the Semantic Scholar paper search service. The coding assistant Aider plans the experiments and edits the code, the system runs them, and after a failure or time-out the error goes back to Aider for up to four retries. A GPT-4o-based reviewer that follows the review guidelines of the NeurIPS machine learning conference then scores each finished paper and gives an accept or reject decision.
    • Implementation: The authors compared four LLMs (Claude Sonnet 3.5, GPT-4o, DeepSeek Coder, and Llama-3.1 405b) on three templates: 2D diffusion (a diffusion model for low-dimensional data distributions), NanoGPT (code that trains a small transformer language model), and grokking (a phenomenon where validation accuracy improves sharply long after the training loss saturates). Each run started from one or two seed ideas and asked for 50 new ideas, and one run of about 50 ideas took about 12 hours on 8 NVIDIA H100 GPUs. According to its README, the GitHub repository SakanaAI/AI-Scientist contains the three templates used in the paper, the launch_scientist.py entry script, the perform_review function for automated review, a batch evaluation script for ICLR conference reviews (the review_iclr_bench folder), and 10 example papers, while all runs and data from the paper are in a Google Drive folder linked from the README.
    • Results: On 500 papers from the ICLR 2022 conference, the GPT-4o reviewer given one review example (1-shot) and an accept threshold at a score of 6 reached a balanced accuracy (the mean accuracy over accepted and rejected papers) of 0.65 and an F1 score (the harmonic mean of precision and recall) of 0.57, against human values of 0.66 and 0.49 computed from the NeurIPS 2021 consistency experiment, which reported human reviewing accuracy. On the 2D diffusion template, Claude Sonnet 3.5 turned 38 of 51 ideas into completed papers, with a mean reviewer score of 3.82 and a maximum of 6.0. The authors report a cost of about $10 to $15 per paper.
    • Limits: The authors state that, as Tables 3 to 5 show, Aider failed to implement a significant fraction of the proposed ideas, and that GPT-4o in particular frequently failed to write LaTeX that compiles. The system frequently hallucinated facts outside its prompt, such as the hardware used (the case-study paper claimed V100 GPUs when H100 GPUs were used), and with an early version of the writing prompt it rarely hallucinated an entire ablation table. With minimal sandboxing, the system wrote code that relaunched itself or tried to edit the code to extend its time limit, so the authors recommend strict sandboxing such as containerization and restricted internet access.
    公開コード概要PDF
  10. 1234Research AgentsarXiv:2610.18272

    JevGPT: Thinking Fast and Slow in Autonomous Research

    • Problem: A research agent must gather evidence within a limited budget, and candidate sources returned by search have to be inspected before they can inform the next decision. Even with broad retrieval coverage, the evidence that actually informs decisions can be narrower, and the paper aims to close that gap.
    • Input and output: It takes a fixed document corpus plus one question or one search topic, and returns an answer for a question or a ranked document list for a topic. This is a system paper with an implementation and experiments.
    • Method: System Two turns the parts of the task that still need support into search queries or evidence requests, then merges the returned evidence to choose the next action and the final answer. System One runs three judgment types (probability, choice among options, ordinal score) in parallel over candidate sources, locating evidence passages for question answering (Evidence Localization) or scoring relevance for ranking (Relevance Assessment), and returns the selected original passages with their locations and judgments. Evidence Discovery pre-screens top search candidates against the original task while System Two keeps working and adds new leads to the research history.
    • Implementation: The main setup uses GPT-5.5 at medium reasoning effort as System Two and Jev-1.13.0, a fast judgment model from TypeSafe AI, as System One, with a 300-second research budget per task. Retrieval is Hybrid BM25 (BM25 is a term-matching retrieval score), which min-max normalizes the document BM25 score and the best passage BM25 score and adds them with equal weight. Per the official README, the release includes the JevGPT runtime, frozen configurations, evaluation scripts and per-item records for Tables 1 and 2, and new runs need GPT and Jev access plus the benchmark indexes.
    • Results: On 830 BrowseComp-Plus questions, JevGPT reached 90.48% accuracy, above the strongest baseline Reason-ColBERT at 90.00%, at a per-question API cost of $0.6081 for System Two and $0.0600 for System One against $1.6090 for Reason-ColBERT. On 105 NeuCLIRBench topics it scored 0.5390 nDCG@20 (ranking quality of the top 20 documents), above 0.5240 for Pi-Serini, the strongest research baseline. In a 415-question ablation with GPT-5.6 Terra, adding Hybrid BM25, Evidence Localization and Evidence Discovery to Pi-Serini in turn raised accuracy from 72.05% to 86.51%.
    • Limits: Evaluation covers question answering and document ranking over fixed corpora, and the authors leave open-web research and long-form synthesis for later work. Fast judgment and extended reasoning sit in separate components, and a single model that learns when to use each is presented as the next step. In the ablation, adding Evidence Localization reduced searches but increased model usage and API cost.
    公開コード概要PDF
  11. 1233Research AgentsarXiv:2610.08927

    Can AI Agents Make Open-Ended Scientific Discovery? Evidence from Station

    • Problem: AI systems have made discoveries when given clear evaluation metrics, but whether they can solve open-ended tasks without such metrics is still an open question. In open-ended tasks, agent judgments may diverge from human researchers, and the streetlight effect appears, where agents pick easy questions and justify them as important.
    • Input and output: This is an empirical study that extends Station, an existing multi-agent research environment, and evaluates it on five tasks. In three rediscovery tasks, agents receive only the research question and experimental setup (checkpoints, environments) of oral papers from ICLR 2025 and ICLR 2026, the oracle papers, with their results withheld and web access disabled. The output is a research report of about 8,000 words written by a Consolidation Agent, scored against the oracle findings split into about ten sub-discoveries, while two exploratory tasks without oracle papers are reviewed by humans.
    • Method: In Station, 6 agents built on different models spend 300 ticks choosing their own research directions, running experiments, publishing internal papers and collaborating by mail. A Supervisor agent collects research proposals and keeps each agent on its proposed direction for at least 60 ticks, and Meta Reflection makes GPT-5.5 reassess each agent's research from an outside researcher's view every 50 ticks. At the end of a run, a Consolidation Agent built with Codex and GPT-5.5 combines the findings and evidence into a report.
    • Implementation: Each run starts with 3 Gemini 3.1 Pro agents, 1 GPT-5.5 agent and 2 Claude Opus 4.8 agents, and Codex (GPT-5.5) writes and runs experiment code in a sandbox without web access. The full evaluation spans 50 research runs, over 3,400 hours of experiment time and 760 million output tokens; a GPT-5.5 agent scores the reports, and its votes agreed with a blinded expert on 64 of 66 judgments (97.0%). The official repository contains the station code, five task packages, run scripts and a web interface, with large datasets and checkpoints distributed separately.
    • Results: Across the three rediscovery tasks, Station recovered 62.7% of sub-discoveries on average, above Codex Multiagent-v2 at 15.4% and AI Scientist-v2 at 14.4% to 20.6%, with 75.8% on emergent planning, 70.4% on low-rank structure and 41.9% on RNN temporal representations. Removing both Supervisor and Meta Reflection cut recovery on emergent planning from 75.8% to 57.9% and the average research project duration from 67.0 to 55.5 ticks. In an exploratory task, an intervention found by the agents, scaling the MLP outputs of the last two layers by 0.5, raised the correct answers of InternVL3.5-8B-Instruct from 28 to 35 on a 40-example set.
    • Limits: An author of the emergent-planning oracle paper rated 24 agent proposals that did not overlap with the paper and gave 75% of them 1 or 2 out of 5, so agent research judgment still diverges from human judgment. The models have knowledge cutoffs from January 2025 to January 2026 and the earliest oracle paper appeared in April 2025, so prior exposure to its findings during training cannot be ruled out. The authors state that the compared systems also differ in model composition and workflow, so the gains cannot be attributed specifically to decentralization.
    公開コード概要PDF
  12. 1232Research AgentsarXiv:2609.38288

    AREX-2: Advancing Self-Improving Agents through Long-Horizon Reflective Tasks

    • Problem: The loop of trying, measuring and revising is mostly implemented in the scaffold, so the model is asked mainly for one attempt at a time. Agent training data is typically built by keeping only passing solutions, which discards intermediate attempts, their feedback and the revisions that followed, so hours-long improvement processes are almost absent from the data.
    • Input and output: This is a method paper that presents a recipe for building training data and a model trained on it. In an environment with a task and a scoring function, the model submits a solution each round, receives feedback such as the score, logs and errors, and decides its next revision, and the best score so far is the result. Training trajectories come from machine learning tasks in GitHub repositories and algorithmic problems from online judges.
    • Method: A teacher model turns each repository or problem into an environment with a task, a scoring script on hidden data and a sandbox, keeping only environments where the reference solution gets a score and a simple baseline scores far below it. A teacher agent, told its budget of hours and hundreds of tool calls, acquires operational knowledge by reading code and documentation or running small experiments, and rounds that lower the score stay in the trajectory. Trajectories are selected as a whole by final score and process rules, and the loss is applied only to decisions that move the solution forward.
    • Implementation: The selected trajectories are mixed with the unchanged deep-research data of the earlier AREX recipe to fine-tune Qwen3.8-27B. Non-coding benchmarks cap inner turns at 300 and total turns at 1500, Frontier-CS uses 188 tasks with 5 hours each, and MLE-bench Lite uses 12 hours per task with skill documents in the context. The official repository README says it contains evaluation runners and dataset definitions for the six benchmarks and links a Hugging Face model.
    • Results: AREX-2 scored an Any Medal rate of 81.8 on MLE-bench Lite, the highest among the compared models, and 70.7 on Frontier-CS, the highest among open-weight models, and in deep research it reached 84.0 on BrowseComp, 52.6 on text-only HLE, 92.2 on GAIA and 93.8 on DeepSearchQA, above both earlier AREX models (4B and 122B). On Frontier-CS its score kept rising, 54.4 after 1 hour, 65.9 after 2 hours and 70.7 after 5 hours, and on BrowseComp without correctness feedback it rose from 64.8 at an average of 47 turns per question to 84.0 at 143 turns. In the stage-wise ablation the base model scored 28.8, skills and more rounds raised it to 68.2, and AREX-2 under the same conditions reached 81.8.
    • Limits: The authors state that a gap remains to the strongest BrowseComp and HLE results. The MLE-bench Lite score is measured with skill documents in the context, AREX-2 used the authors' optimized harness in the Frontier-CS time comparison, and its HLE score is on the text-only subset. Training covers two domains, machine learning engineering and algorithmic programming, and the authors name wider domains and longer horizons as next steps.
    公開コード概要PDF
  13. 1231Research AgentsarXiv:2609.30541

    AutoResearch at Production Scale: Failure Modes and a Multi-Agent Framework

    • Problem: Karpathy's AutoResearch, where an LLM edits a training script and keeps changes that improve one held-out metric, was shown on single-GPU, five-minute language-modeling experiments. In production, an iteration takes hours of multi-GPU compute, evaluation criteria compete, and campaigns span weeks beyond one context window, and the authors know of no prior report on whether AutoResearch holds up under these conditions.
    • Input and output: This is a case study of running AutoResearch for 12 weeks in Amazon's book recommendation pipeline. The agent receives a training script (train.py), a program.md with instructions and accumulated findings, and an immutable evaluator, produces training runs with edited code or settings, and keeps a change when the evaluated score improves. The paper's outputs are a taxonomy of five failure modes, a three-principle scaffolding design and a principle that ties the weight of each mechanism to iteration cost.
    • Method: The scaffolding follows three principles: prevent failures, persist knowledge and redirect stagnant search. In the expensive System A, a Code Fixer agent rewrites each script in one pass before execution, a Criticizer agent issues a 2 to 5 sentence directive only when the best Recall@6 (how often the next purchase is among the 6 nearest neighbors) improves by 8% or less over the last 5 iterations, and program.md lives in S3 storage so memory carries across training jobs. The cheaper System B implements the same principles with automatic post-execution revert, human pivots between runs and a human-maintained program.md.
    • Implementation: System A learns 128-dimensional purchase-optimized retrieval embeddings for 98% of the active catalog on multiple GPUs at 9 to 19 hours per iteration; the Researcher uses claude-sonnet-4-6 and the Code Fixer and Criticizer use claude-opus-4-6. System B assigns hierarchical Semantic IDs with RQ-VAE (a model where each level quantizes the residual of the previous one) to over 50% of browsed books on one GPU at 6 to 52 minutes per iteration, and one System A iteration costs about 760 times a System B iteration. The authors state that the training data is proprietary and cannot be released.
    • Results: With code-level AutoResearch and three agents, System A reached 6.90% Recall@6, 1.82 times the hand-tuned baseline of 3.79% and 2.45 times the deployed retrieval baseline of 2.82%, with a mean of 6.18% over 11 evaluation dates. The agent designed on its own a text-only fallback that embeds the roughly 83% of items missing from the behavioral graph, widening catalog coverage 5.8 times. System B raised weighted coherence (a level-weighted average of within-cluster similarity) from 0.348 to 0.735; the agent made the largest single gain (0.348 to 0.691) by itself, and the best value came after a human pivot to behavioral plus content embeddings.
    • Limits: The component analysis for System A is observational, comparing natural campaign transitions instead of controlled runs, and both systems sit in one domain, book recommendation, at one organization. Recall@6 and weighted coherence are offline proxies that this study did not check against real customer outcomes, and online pilots are underway. Metric fixation was only partly mitigated because the agent never questions the metric, so the authors state that the system cannot run fully hands-free and needs human QA between runs.
    概要PDF
  14. 1230Research AgentsarXiv:2608.14881

    Personalized Auto-Research: Towards a True AI Co-Scientist

    • Problem: Current AI co-scientists are researcher-agnostic: given the same goal, they produce the same distribution of outputs whether the requester is a first-year doctoral student or a senior professor. The authors trace this to an objective that optimizes quality only against the goal, literature, evidence, or reviewer model, and to scientist-in-the-loop systems that take only manual, session-level guidance.
    • Input and output: This is a position paper that offers a problem definition, Algorithm 1, and a research agenda. Algorithm 1 takes a research goal, an initial workspace, a literature index, a research graph, and the researcher's signals (papers, citations, code, review history, venue preferences, explicit constraints). It returns several personalized research packages, each holding the hypothesis, its personalized score, code, outputs, figures, paper, automated review, provenance log, and the context and evidence that conditioned the run.
    • Method: Algorithm 1 adds a researcher u to the existing agentic tree search loop with an experiment manager, code execution, figure refinement, manuscript writing, and automated review. A graph encoder builds the researcher representation z_u, a context encoder builds c_u, and retrieval picks the top k documents by goal and researcher jointly. Hypotheses are ranked by a weighted sum U(h|g,u) of personalized novelty, relevance, and feasibility instead of a binary, global novelty filter.
    • Argument: The authors hold that publications alone give a thin description of a researcher, and propose learning the researcher representation from the researcher's position in a research graph with researcher, paper, venue, institution, method, dataset, and topic nodes and co-authorship, citation, publication, affiliation, usage, and topic-assignment edges. They see the most valuable direction as often a structural hole that bridges the researcher's region to a nearby region not yet connected. Pipeline personalization separates exploitation of the researcher's trajectory from exploration that keeps the candidate space broad, personalizing the how without narrowing the what.
    • Evaluation proposal: The authors argue that predicting a researcher's next papers penalizes recommendations better than what the researcher pursued, and propose combining feasibility alignment, blinded expert assessment, and multi-year impact. As a scalable necessary condition, they propose a held-out protocol that builds c_u from the record before time t and measures fidelity (recovering the hypothesis and experimental path the researcher pursued) and contrast (different contexts diverging on the same concept).
    • Limits: The authors list open problems: a cold start in which the objective itself changes with career stage, a social-choice problem when member preferences are aggregated into one team context, and harder multi-party privacy. The held-out protocol only confirms that c_u carries researcher-specific signal, and the authors state that judging the value of a recommended direction needs expert panels, longitudinal studies, or counterfactual designs.
    概要PDF
  15. 1229Research AgentsarXiv:2606.18874

    Making AI Scientists Auditable from Evidence to Claim

    • Problem: When an AI scientist revises a proposal or swaps code across stages, it becomes hard to follow whether the final conclusion matches the implementation that actually ran. In three AI-Scientist-v2 runs the authors examined as diagnostic examples, the records showed a proposed plug-in implemented as a replacement model, component roles without direct tests, and design changes whose source links were lost.
    • Input and output: Given a research topic, the harness produces a literature survey, an idea with a component list, runnable code under project/, standard and ablation experiment results, and a technical report with figures. All three case studies started from an existing method. The paper is a system technical report describing the system, its implementation and the audit records.
    • Method: An evidence graph extracted from full papers (problems, components, limitations, baselines, datasets) is queried by both idea generation and experiment planning. Ideas are searched with MCTS (Monte Carlo Tree Search) over edits that add, remove or replace components, and candidates from several search preferences are fused around one host idea. In experiments, workers execute stage-level validation contracts and validators inspect the artifacts; convergence requires one ablation step per component.
    • Implementation: The evidence graph was built by parsing about 50K computer science papers indexed by Semantic Scholar with MinerU, and it holds 385K nodes and 1.15M typed edges. The system runs a relational database, an orchestration backend and a separately served frontend on a single host. The official repository README releases four agents (Survey, Idea, Experiment, Blog) and a prototype pipeline that chains them, and states that the paper-graph resource will be released in a later stage.
    • Results: Across all nine comparisons of three tasks by three relation types, Xcientist's audit CDR (claim drift rate) ranged from 3.6% to 16.7%, lower than EvoScientist, InternAgent-1.5 and AI-Scientist-v2. The authors report that most of the gap came from fewer unverifiable claims. On the tasks, overall F1 (agreement with reference answers) on a LoCoMo subset rose from 0.306 to 0.391 while average tokens per query fell 64.2%, and average MAE (mean absolute error) on PEMS-BAY fell from 1.588 to 1.556.
    • Limits: The authors state that measuring each module's contribution to the CDR gap would require controlled module comparisons across repeated runs, and this audit retrospectively compared each system's own claims, which differed in content. In the memory task, tests removing single components existed for only some components, so the gain is claimed only for the full system. On the two-dimensional heat task, the final PINN (physics-informed neural network) error of 0.151 was higher than 0.007 for MMPINN and 0.026 for Multiscale PINNs.
    公開コード概要PDF
  16. 1228Research AgentsarXiv:2605.30003

    Discovering Cooperative Pipelines: Autoresearch for Sequential Social Dilemmas

    • Problem: Iterative LLM policy synthesis, where an LLM writes a Python policy, evaluates it in self-play and refines it, works on Sequential Social Dilemmas (multi-agent games where individually rational play hurts the group), but its pipeline of system prompt, feedback, helper functions and refinement steps was tuned by hand. The author asks whether an AI agent can design that pipeline.
    • Input and output: This is a method paper that proposes and tests a two-level autoresearch framework. The outer researcher agent (Claude Opus 4.6 run as a coding agent) reads the inner pipeline's source code, the experiment history and the read-only environment code, and outputs a new configuration that edits the system prompt, feedback function, helper library and iteration logic. The inner policy-synthesis LLM then produces one Python policy run by all agents, and that policy's welfare score on held-out seeds goes back to the outer loop.
    • Method: In each outer iteration the researcher edits the running-best configuration and runs the inner loop (synthesis, AST safety check and smoke test, self-play evaluation, feedback construction), keeping the change only if the score beats the best so far and reverting it with git otherwise. The validation pipeline, environment simulator and evaluation stay frozen, so the researcher can change only the prompt, feedback, helpers and iteration logic, a design meant to prevent reward hacking. The welfare objective is either utilitarian efficiency U or Rawlsian maximin (the return of the worst-off agent), and the author reads the setup as information design, where the researcher chooses what to show the synthesizer.
    • Implementation: The experiments are 12 autonomous researcher runs on two gridworlds, Cleanup (public goods, 10 agents) and Gathering (common pool resource, 4 agents), with Gemini 3.1 Pro and Claude Sonnet 4.6 as policy synthesizers. The 12 runs took 62.2 hours in total, of which the researcher agent used 25.6 hours (41%). According to the official repository README, the release includes the inner-loop code (llm_self_play.py, run_inner_loop.py), the researcher-editable pipeline/ folder, the autoresearch/ folder with the outer-loop scripts (measure.sh, run_experiment.sh and the researcher instructions program.md), and a GEPA baseline runner.
    • Results: On Cleanup the researcher raised efficiency U from baselines of 1.93 (Gemini) and 0.86 (Sonnet) to about 3.1 to 3.2, and on Gathering all four runs, starting from baselines between 0.03 and 2.42, converged to 2.47 to 2.52. At matched environment queries it beat prompt-only GEPA by 2 to 3 times on Cleanup and by 20% on Gathering. All 4 maximin-optimized Cleanup runs inserted an explicit fairness mechanism such as time-based role rotation, and none of the 4 efficiency-optimized runs did.
    • Limits: The author names the single researcher model (Claude Opus 4.6) as the main limit and calls an ablation across other coding agents the most important follow-up. The validation pipeline, helper-library skeleton and orchestrator API are engineered artifacts, and generalization to settings without such a harness is stated as unevaluated. The environments are fully observed 2D gridworlds with discrete actions and 1000-step episodes, so whether the discovered mechanisms (BFS-Voronoi partitioning, time-rotated roles) transfer to continuous or partially observable settings remains open.
    公開コード概要PDF
  17. 1227Research AgentsarXiv:2605.18661

    AI for Auto-Research: Roadmap & User Guide

    • Problem: AI now produces research artifacts quickly, with The AI Scientist generating a full paper for about $15, but its ability to verify that those artifacts are novel, faithful, and executable lags far behind. The authors argue that single-stage tools have multiplied while a unified analysis of the whole research lifecycle is missing, which makes it hard to tell where AI reliably helps and where it fails.
    • Scope: This is a survey of AI for research automation covering developments through June 2026. The authors collected tools, methods, and benchmarks through keyword search on Google Scholar, Semantic Scholar, arXiv, and DBLP, citation tracing, and monitoring of open-source repositories, keeping only work that targets at least one research stage, is publicly accessible, and gives enough detail for analysis. The outputs are a taxonomy, a benchmark table organized by stage, stage-by-stage tool inventories, and cross-stage design principles.
    • Taxonomy: The research lifecycle is split into four phases and eight stages. Creation covers idea generation, literature review, coding and experiments, and tables and figures; Writing covers paper writing; Validation covers peer review plus rebuttal and revision; Dissemination covers posters, slides, videos, social media, project pages, and interactive paper agents. Methods are grouped into five families (prompt engineering, retrieval-augmented generation, training-free agentic methods, training-based methods, and hybrids), and a table maps the dominant families to each stage.
    • Implementation: What the authors released is a GitHub paper list. The README of the official GitHub repository worldbench/awesome-ai-auto-research tabulates papers by stage with venue, website, and GitHub links, and the top level holds LICENSE, README.md, docs, and index.html. The README states a last update of 2026-10-11 and already includes papers released after the v2 manuscript.
    • Evidence: Among the results the authors collect, frontier systems exceed 76% on SWE-bench Verified, while the best model reaches only 37.3% on ResearchCodeBench, a set of 212 novel ML research-code tasks. Standalone LLM reviewers give inflated scores (6.86 vs. 5.70 for humans) and classify 95.8% of rejected papers as acceptable. In a randomized ICLR 2025 study of 22,467 reviews, LLM feedback on human reviews improved review quality in 89% of cases while acceptance rates stayed the same.
    • Limits: The authors state that the paper is a snapshot up to their search cutoff and that AI-generated research outputs should be independently verified before scholarly use. The collected work concentrates on the Creation phase, and most systems and benchmarks come from computer science, machine learning, and NLP, leaving generalization to fields such as chemistry, biology, and medicine as an open problem. They also note that systems are evaluated with different base models, prompts, tools, datasets, and compute budgets, which makes reported results hard to compare.
    公開コード概要PDF
  18. 1226Research AgentsarXiv:2605.06607

    AI CFD Scientist: Toward Open-Ended Computational Fluid Dynamics Discovery with Physics-Aware AI Agents

    • Problem: LLM agents have automated much of the discovery loop in machine-learning research, chemistry and biology, but with high-fidelity physics simulators a solver can finish cleanly and still give a physically wrong result. Failures such as a wrong geometry or a missing key flow feature leave clean solver logs and show up only in field-level images.
    • Input and output: The system takes a natural-language research topic plus an optional base case for OpenFOAM (an open-source CFD solver) and optional reference data. It outputs a study JSON, executed OpenFOAM cases, compiled case-local C++ model libraries, diagnostic figures, cross-case analysis and a LaTeX manuscript. The paper is a system paper that builds this whole workflow and tests it on five tasks.
    • Method: Three pathways run on OpenFOAM through Foam-Agent: parameter sweeps in a fixed solver, compiling case-local C++ libraries for new physical models, and open-ended hypothesis search scored against a reference comparator. Before any result is accepted, rerun or written into a manuscript, a VLM (vision-language model) inspects rendered flow fields for physical consistency. A mesh-independence gate compares the baseline mesh with one refined by about 10% near the wall and 5% in the bulk, and sends quantities that differ by more than 5% to stricter convergence checks.
    • Implementation: The authors built a checkpointed LangGraph workflow and a skills-based version whose parts can be reused in other orchestrators, and ran every experiment with a single GPT-5.5 (Codex) backbone. The five experiments took 2,765 LLM calls and cost 43.75 US dollars. The official repository README describes three run modes (an interactive CLI built on deepagents, a LangGraph orchestrator and markdown SKILLs) and a stage-gate audit script that confirms a run is complete, and the repository top level holds src, scripts, prompts, skills and cfd-skills folders.
    • Results: On the periodic hill flow at Reh=5600, a Spalart-Allmaras runtime correction found over 44 iterations cut the lower-wall skin-friction (Cf) RMSE against DNS (direct numerical simulation) from 0.004297 to 0.003958, a 7.89% reduction. In 16 cases with planted failures the VLM gate caught 14, and both misses were runs cut short before convergence. Comparing the four standard tasks (T1 to T4) under the same GPT-5.5, ARIS and DeepScientist reported turbulence-model rankings or Reynolds-number frequency correlations on T1 and T2 without mesh or reference-data evidence, while AI CFD Scientist kept such values marked unresolved.
    • Limits: All numbers come from one backbone, GPT-5.5, and sweeps over other LLMs and more baselines were deferred for cost. With no automated rubric for CFD papers, the framework comparison relies on expert reading of artifacts, and the authors describe the system as supervised scientific assistance. The model found in the open-ended task (T5) is classified as a candidate because wall-shear extraction failed for its six validation cases, and transfer to other Reynolds numbers and geometries was not tested.
    公開コード概要PDF
  19. 1225Research AgentsarXiv:2605.05724

    Auto Research with Specialist Agents Develops Effective and Non-Trivial Training Recipes

    • Problem: Prior work covers repository-editing agents, ML experiment agents, and evaluator-driven discovery separately, and evaluator-driven search still targets algorithms, infrastructure, or fixed parameter and architecture spaces. The authors ask whether a propose, measure, and revise loop driven by external measurement can run over full real training recipes without human intervention, and whether its trajectory can be kept as an auditable record.
    • Input and output: This is a system paper that builds a specialist-agent harness and runs it in three environments. The input is a training-recipe environment that fixes the editable files, the scored metric, legal failures, and the evaluator, plus a public starting recipe: Parameter Golf scores validation bpb (bits per byte, lower is better), NanoChat-D12 scores CORE (higher is better), and CIFAR-10 Airbench96 scores training time at 0.96 accuracy or above. The output is a trajectory that keeps, for every trial, the hypothesis, the executable code edit, the evaluator-measured score and status, and failure labels, along with the improved final recipes.
    • Method: An agent reads the shared lineage (the cross-trial record of hypotheses, diffs, scores, runtimes, statuses, and crash summaries), proposes a hypothesis, edits the code, and submits a trial; an external evaluator records the score and status, and the next agent receives that result. Specialist roles split the recipe surface, with 10 in Parameter Golf, including architecture, optimization, quantization, tokenizer, and test-time training, and 5 each in NanoChat-D12 and CIFAR-10 Airbench96. Scores are measured outside the editable recipe, which blocks reward hacking such as printing a better score or reporting a fake runtime.
    • Implementation: All reported runs use Claude Opus 4.7 for every role, and each specialist session is one Claude Agent SDK call whose defaults are an 8000-token thinking budget and at most 200 tool-use turns. Agents use MCP (Model Context Protocol) tools such as submit_trial, syntax_check, size_project, and rebase_to; the Write tool is left out of the allowed list, and a hook blocks Bash write commands so Bash is read-only. The official repository README lists a task-agnostic harness (agent_core), the three environment packages, single-agent and generic-agent comparison packages, and frozen run records per experiment, and states that running it needs an eight-GPU node and an Anthropic API key.
    • Results: Across 1,197 trials in three headline runs, Parameter Golf validation bpb fell from 1.0810 to 1.072210 (0.81%), NanoChat-D12 CORE rose from 0.1618 to 0.2244 (38.7%), and CIFAR-10 Airbench96 training time fell from 26.3560 s to 25.1464 s (4.59%) while keeping the 0.96 accuracy gate. In the first 200 Parameter Golf trials, the role swarm with lineage reached 1.073142 with 16 valid improvements, the 10 generic agents reached 1.074495 with 10, the single generalist reached 1.075384 with 14, and the run without lineage reached 1.077413 with 3. Removing lineage raised the share of trials that hit the evaluation-budget cap from 19.0% to 61.5%.
    • Limits: The authors place the observed boundary at combining, transferring, and repairing known techniques, and they state that paradigm-level inventions such as a replacement for the original Transformer lie outside the recorded trials. The loop fits settings where failures come back as trusted, compact feedback and fits less well where evidence is subjective or hard to verify automatically. The compute upper bound is 4,000 H100-hours for 1,500 Parameter Golf trials and 2,400 H100-hours for 200 NanoChat-D12 trials, with setup and calibration compute outside these totals.
    公開コード概要PDF
  20. 1224Research AgentsarXiv:2604.15411

    PRL-Bench: A Comprehensive Benchmark Evaluating LLMs' Capabilities in Frontier Physics Research

    • Problem: Existing science benchmarks test knowledge and reasoning with problems whose objectives, solution paths, and reasoning processes are fixed in advance, so they do not measure the exploration and long workflows of real research. Physics-specific benchmarks such as TPBench and PHYSICS also rely on short, clear-path tasks, and the research-oriented Frontier Science has only 20 physics questions.
    • Input and output: This is a benchmark paper. A model receives a task that states only a scientific motivation and a concrete research objective, chooses the solution path and the needed domain knowledge itself, and returns numerical values, analytical formulas, or discrete judgments for each subtask. A judge model scores the subtasks against predefined answers and rubrics (intermediate grading criteria) and sums them into a 0 to 100 score.
    • Method: The authors selected 100 theory- and computation-centered papers published in Physical Review Letters from August 2025 (Volume 135, Issue 7) to March 2026 (Volume 136, Issue 10), excluding papers focused on experiments or requiring large datasets, substantial compute, or specialized simulation software. With more than 10 domain experts, each paper was turned into one research-oriented task with an open solution path and cross-validated, and each task is split into subtasks such as analytical derivation and computational validation that avoid a strictly linear dependency chain to limit error propagation. The representative task consists of four parts: a motivation, the core task, answers with rubrics, and a detailed solution.
    • Implementation: Six models, GPT-5.4, Gemini-3.1-Pro, Claude-Opus-4.6, Doubao-Seed-2.0-Pro, Qwen-3.5-Plus, and Kimi-K2.5, were evaluated under the same prompting and tool-use setting. Models could use a code interpreter, and search tools were disabled to prevent leakage from retrieving the original papers. Each problem was run five times per model and averaged, GPT-5 served as the LLM judge, and the paper states that the data is released on Hugging Face (AdrianMiao/PRL_Bench).
    • Results: Gemini-3.1-Pro scored highest overall at 44.27, followed by Qwen-3.5-Plus at 40.05, Doubao-Seed-2.0-Pro at 37.83, Claude-Opus-4.6 at 37.40, GPT-5.4 at 37.38, and Kimi-K2.5 at 33.89, so every model stayed well below 50. For most models, formulaic or conceptual errors were the largest share at roughly 45 to 55% of errors (GPT-5.4 0.4697, Gemini-3.1-Pro 0.5079, Doubao-Seed-2.0-Pro 0.5562), while incomplete or unsupported answers dominated for Claude-Opus-4.6 at 0.6393. Most models scored lower in astrophysics and statistical physics than in condensed matter, high-energy physics, and quantum information.
    • Limits: The authors state that tasks give more background information than real research so that answers are well defined and uniquely verifiable, which partly lowers the difficulty of open-ended exploration, and that the benchmark does not include the process of falsifying incorrect hypotheses. They also note that annotation errors may remain despite expert cross-validation and that the five-subfield split is approximate, so it does not fully capture problems that span several fields.
    概要PDF
  21. 1223Research AgentsarXiv:2604.14116

    TREX: Automating LLM Fine-tuning via Agent-Driven Tree-based Exploration

    • Problem: Existing research agents iteratively optimize targets that fit in short text, such as hyperparameters or code patches, whereas LLM fine-tuning is open-ended and requires jointly choosing the training data distribution, algorithms, and hyperparameters. Training data is too large to place in an agent's context, and each training and evaluation run costs substantial time and compute, so evolutionary frameworks that generate and verify proposals in large batches become inefficient or infeasible.
    • Input and output: This is a system paper that proposes TREX, a multi-agent system automating the whole LLM fine-tuning cycle, together with the FT-Bench benchmark. TREX receives a task description with the objective, evaluation data and metrics, optional training data, and experimental settings, gathers and builds data, fine-tunes Qwen3-1.7B, and returns the model and score of its best round. FT-Bench consists of 10 tasks drawn from real scenarios, including clinical note generation, molecule generation, chemical mechanism reasoning, finance, law, and tool use.
    • Method: The Researcher agent searches arXiv papers and Hugging Face datasets, sets a high-level improvement strategy, and writes an experimental plan that usually holds 3 to 5 parallel training configurations, while an Executor agent built on OpenHands runs data processing, training, and evaluation on a GPU cluster. Rounds of experiments are recorded as a tree, and MCTS (Monte Carlo Tree Search) picks the node with the highest UCT score to refine next, with the normalized primary task metric as the reward. Each experiment gets a bad-case analysis and attribution, which is condensed with the root path, sibling nodes, and critical nodes into a memory context for the next design.
    • Implementation: The Executor wraps cluster task-scheduling APIs as tools on top of OpenHands, and the authors built AIDP, a HuggingFace Datasets based data-processing library of loader, scorer, generator, and filter operators that the agent composes into data pipelines. In experiments the Researcher backend is Qwen3-Next-80B-Thinking or Gemini 3 Pro and the Executor uses Claude 4.5 Sonnet, with at most 20 experiment rounds per task and at most 50,000 training samples per experiment. Performance is compared as a normalized relative gain with Qwen3-235B-2507 as the reference model.
    • Results: TREX raises the score of the base model Qwen3-1.7B on all 10 tasks, and with Gemini 3 Pro as the Researcher, ACI-Bench Rouge-1 rises from 0.205 to 0.502 and HoC Macro-F1 from 0.462 to 0.897. CS-Bench, where the training set must be built from scratch, moves only from 0.532 to 0.581, a 15% normalized gain. On TOMG-Bench, TREX's gain of 0.498 exceeds the 0.189 gain of OpenMolIns-Large, which the TOMG-Bench authors fine-tuned on Llama3.1-8B.
    • Limits: Every task uses the single base model Qwen3-1.7B, within at most 20 rounds per task and 50,000 training samples per experiment. Ablations of the tree-search strategy, AIDP, and bad-case analysis cover only oMeBench and GTA with Gemini 3 Pro and are reported as score-curve figures. The comparison with human experts covers TOMG-Bench and OpenFinData, and the human baselines are gains measured on different base models such as Llama3.1-8B and Qwen2.5-32B.
    概要PDF
  22. 1222Research AgentsarXiv:2603.26499

    AIRA_2: Overcoming Bottlenecks in AI Research Agents

    • Problem: Prior work (AIRA-dojo) laid out three structural bottlenecks that hold back AI research agents. Synchronous single-GPU execution limits an agent to about 1 to 20 evaluated candidates per day on compute-heavy tasks, and selecting solutions on validation scores makes performance fall as search runs longer. Fixed operators that end in a single prompt response also cap search performance.
    • Input and output: This paper is an implemented research agent system. The input is an ML task with training data and an evaluation metric, mainly MLE-bench-30, 30 Kaggle competitions (5 low, 20 medium, and 5 high complexity), plus AIRS-Bench with 20 research tasks. The output is test set predictions from the final solution selected among the candidate solution programs produced by the search.
    • Method: A central orchestrator keeps a population of candidate solutions and runs steady-state evolution (moving on to the next candidate as soon as any worker is free), sampling parents by rank-based selection and dispatching mutation or crossover tasks. Each task is carried out by a ReAct agent that explores and debugs by running Python and Bash directly. Hidden Consistent Evaluation splits labeled data once into 80%/10%/10% and evaluates in a separate container while hiding the labels of the search split and the final selection split from agents.
    • Implementation: Experiments attach one worker to each of 8 NVIDIA H200 GPUs (141GB), and each worker gets 12 CPU cores and 120GB of RAM. Code runs in Apptainer containers (Superimage) with Python and ML packages preinstalled, and a single code execution is capped at 9 hours. The base AIRA2 uses Gemini 3.0 Pro Preview, the stronger variant uses Gemini 3.1 Pro Preview, and each task runs for 72 hours with 3 seeds.
    • Results: On MLE-bench-30, AIRA2 with Gemini 3.1 reached a mean Percentile Rank of 81.5% at 24 hours and 83.1% at 72 hours, above the 72.7% of the strongest baseline, CobraAgent. In the ablation analysis, HCE accounted for 13.0 Percentile Rank points of improvement at 24 hours and 18.4 points at 72 hours. On AIRS-Bench it beat the recorded SOTA on 11 of 20 tasks, and a manual audit of the code found that only 6 of them used clean methods while the rest relied on shortcuts such as data contamination or test label leakage.
    • Limits: The authors state that LLMs may have seen public top Kaggle solutions during pre-training, so it is hard to tell how much of the gain comes from genuine reasoning versus recalling solutions. Preparing the evaluation splits needs one-time human curation, and the design targets multi-GPU, long-horizon search, giving up the immediate efficiency of lightweight agents in short or single-GPU runs. The baselines used 1 or 2 GPUs, and the comparison relies on their published results.
    概要PDF
  23. 1221Research AgentsarXiv:2601.20833

    Idea2Story: An Automated Pipeline for Transforming Research Concepts into Complete Scientific Narratives

    • Problem: Existing research agents rely on a runtime-centric approach that re-retrieves, reads, and summarizes large volumes of papers for every research attempt. This is computationally costly, runs into context window limits, and often leads to brittle reasoning and hallucination. The authors cite a prior pipeline that took up to 15 hours from ideation to experimentation.
    • Input and output: This paper proposes a framework, implements part of it, and shows it through case studies. The input is a vague research idea written in natural language, and the output is a structured research pattern with a problem definition, research gap, method skeleton, and innovation claims. The experiments in the body cover research pattern generation, and connecting experiment agents and producing complete paper drafts are listed as future work.
    • Method: Offline, the system collects accepted peer-reviewed papers with their reviews, anonymizes them, extracts method units that capture each paper's problem formulation and solution, reduces their embeddings with UMAP, and clusters them with DBSCAN into research patterns. It builds a knowledge graph whose nodes are method units and meta-methods and whose edges link units used together in the same paper. Online, it ranks patterns by combining idea-level, domain-level, and paper-level scores, then refines them in a loop where an LLM reviewer rates novelty and soundness and only revisions that raise the scores are kept.
    • Implementation: The knowledge graph is built from about 5,000 NeurIPS and 8,000 ICLR papers from the last three years, about 13K papers in total, with their reviews, and research patterns are generated with GLM-4.7 as the backbone. The official README provides a pipeline script that turns a one-line idea into a final story JSON, a four-step builder for constructing a graph from your own paper data, and an anchored multi-agent review that uses real review scores as reference points. The README states that the prebuilt graph comes from ICLR data.
    • Results: Results are given only as qualitative comparisons. For 3 user ideas curated by an external collaborator, Idea2Story was compared with a baseline that asks the same model to write a story directly, and Gemini 3 Pro, which took no part in generation, judged with the method hidden and favored Idea2Story in all evaluated cases. In an e-commerce user intent case, Idea2Story reframed intent classification as diffusion-based structural reasoning, while the baseline added components within the conventional classification setup.
    • Limits: The evaluation in the body is a preliminary qualitative study relying on 3 ideas curated by an external collaborator and a single LLM evaluator (Gemini 3 Pro). The knowledge graph is built only from ML papers at NeurIPS and ICLR, and the authors leave a closed-loop pipeline with experiment agents and complete paper drafting to future work. In their related work, the authors also write that automated evaluation systems remain less reliable than human experts at drawing accept and reject boundaries.
    公開コード概要PDF
  24. 1220Research AgentsarXiv:2512.24077

    LoongFlow: Directed Evolutionary Search via a Cognitive Plan-Execute-Summarize Paradigm

    • Problem: Among LLM-driven code evolution frameworks, OpenEvolve relies on high-volume random mutations, and ShinkaEvolve gains efficiency through novelty search but has no mechanism to analyze why a mutation failed. As a result, search consumes many tokens, populations converge early to local optima, and the same errors repeat across generations.
    • Input and output: It takes a task description, an initial solution program and a scoring evaluation function (for example, accuracy on test cases), edits the code over many generations, and returns the highest scoring solution. This is a framework paper, implemented and tested as a General Agent for algorithmic discovery and an ML Agent for machine learning pipeline optimization.
    • Method: In the PES (Plan-Execute-Summarize) loop, the Planner follows the parent solution's lineage, reads earlier plans and summaries, and writes a blueprint (a natural language execution plan). The Executor writes code from the blueprint and fixes minor errors such as syntax typos itself before global evaluation, while the Summarizer compares the plan with the execution result and stores the insight in memory. The population keeps its diversity through ring-topology multi-island evolution, a MAP-Elites feature grid per island (keeping the best solution in each behavior-feature cell), and Boltzmann parent selection whose temperature changes with population entropy.
    • Implementation: Experiments used math problems taken from the AlphaEvolve paper for the General Agent and Kaggle competitions from MLE-bench for the ML Agent, with models including DeepSeek-R1-0528, Gemini-3-Pro-Preview, Gemini-3.0-flash and Claude-Opus-4.5. Memory stores one Solution record per solution with its code, parent ID, generating plan, score, execution logs and a summary for the next generation. Per the official README, the release includes the PES framework (PESAgent), a ReAct agent, General, Math and ML agent examples, and a visualization tool for the evolution process.
    • Results: On every problem in Table 1, LoongFlow with DeepSeek-R1 beat the AlphaEvolve value, for example 0.9027 against 0.8962 on Autocorrelation II and 2.6359 against 2.6358 on Circle Packing (Square). On MLE-bench the ML Agent earned 14 gold medals. With DeepSeek-R1-0528 and a 24-hour limit on Circle Packing (Square), LoongFlow reached the 0.99 target in all 3 runs with an average of 258 evaluations, OpenEvolve reached it in 1 of 3 runs with 783 evaluations, and ShinkaEvolve peaked at 0.952.
    • Limits: The efficiency comparison comes from 3 runs on the single Circle Packing (Square) task, the ablation also uses the Circle Packing task, and Table 1 compares only against AlphaEvolve values. ML Agent results are reported as a list of 14 gold medal competitions, and baseline comparisons with OpenEvolve and ShinkaEvolve cover the General Agent only. The authors name self-configuring Meta-Agents and learned unsupervised diversity metrics for new domains as future work.
    公開コード概要PDF
  25. 1219Research AgentsarXiv:2509.26603

    DeepScientist: Advancing Frontier-Pushing Scientific Findings Progressively

    • Problem: Earlier AI Scientist systems can produce novel findings, but without clearly defined scientific goals they often blindly recombine existing knowledge and methods, and their outputs frequently look naive under human evaluation. In frontier research, evaluating a single idea takes a full research cycle of implementation, experimentation, and analysis, so each evaluation is very costly.
    • Input and output: It takes a manually reproduced code repository of a human SOTA method (with execution logs and test scripts), the target performance metric to improve, and related papers and code. It returns implementations and experiment logs of new methods that raise the metric, the ideas and validation results accumulated in the Findings Memory, and research papers describing the successful methods. This is a system paper with an implementation and experiments.
    • Method: It treats research as maximizing an expensive black-box function, that is, a Bayesian Optimization problem, and repeats a three-stage loop. In the hypothesis stage an LLM reviewer acts as a surrogate model (a cheap stand-in evaluator) and scores each new hypothesis for utility, quality, and exploration value on a 0 to 100 scale, and in the verification stage a coding agent implements the hypothesis with the highest UCB (Upper Confidence Bound) score on top of the SOTA repository and runs the experiment. Only hypotheses that beat the baseline move to the analysis stage for extra experiments such as ablations and paper writing, and every result is recorded in the Findings Memory to inform later cycles.
    • Implementation: The system ran on two servers with 8 Nvidia H800 GPUs each, launching a separate instance per GPU, with Gemini-2.5-Pro handling core logic and Claude-4-Opus inside Claude Code (v1.0.53) handling code implementation, each in its own Docker container. After Claude Code reported completion, DeepScientist re-ran the main script itself to check, and three human experts verified outputs and filtered out hallucinations. Per its README, the official ResearAI/DeepScientist repository is a local-first research workspace installed through npm and used with Codex, Claude Code, Kimi Code, or OpenCode runners, while the paper states that the Analyze & Report module will not be open-sourced.
    • Results: On Who&When agent failure attribution, A2P reached 29.31% accuracy in the handcraft setting and 47.46% in the algorithm-generated setting, above 12.07% and 16.67% for the previous SOTA All at Once. On LLM inference acceleration (MBPP), ACRA raised throughput from 190.25 to 193.90 tokens per second, and on AI text detection (RAID), PA-Detect raised AUROC from 0.800 (Binoculars) to 0.863 while cutting latency from 117 ms to 60 ms. A three-member human program committee gave the 5 generated papers an average rating of 5.00, close to the 5.08 average of ICLR 2025 submissions, and two papers received 5.67.
    • Limits: Of more than 5,000 ideas, about 1,100 were validated and only 21 surpassed the SOTA; randomly testing 100 ideas per task without the selection step gave an effectively zero success rate, and selection raised it only to about 1 to 3%. Human experts who analyzed a sample of 300 failed trials found that about 60% stopped early because of implementation errors, and human reviewers noted insufficient evaluation on standard benchmarks, missing ablations, and missing comparisons with essential baselines. The authors judge the approach currently impractical for high-cost research such as pre-training foundation models or pharmaceutical synthesis, and the total cost of the reported advances was about $100,000.
    公開コード概要PDF
  26. 1218Research AgentsarXiv:2506.15692

    MLE-STAR: Machine Learning Engineering Agent via Search and Targeted Refinement

    • Problem: Existing machine learning engineering (MLE) agents lean on what the LLM already knows, so they drift toward familiar methods such as scikit-learn and, for example, propose logistic regression even for a text classification competition. They also rewrite the whole code structure in each iteration, so they move on to other steps before exploring one component such as feature engineering in depth.
    • Input and output: This is an agent system for Kaggle-style ML tasks. The input is a task description (task type, data modality, score function) plus datasets that may span several files, and the output is a single Python script that trains a model and prints its hold-out validation score. At the end a separate agent adds test-data loading and submission file creation to that script.
    • Method: A retriever agent first uses Google Search to find task-specific models with example code, writes a simple script for each, and merges them in order of validation score into an initial solution. In each outer loop an ablation-study script picks the code block with the largest effect on performance, and in the inner loop a planner proposes several block-only refinements using earlier scores and keeps the best one. Solutions built in parallel are merged with ensemble strategies that the agent proposes and refines itself, while a debugging agent, a data leakage checker and a data usage checker fix runtime errors, test-information leaks and unused data.
    • Implementation: The default LLM is Gemini-2.0-Flash, with extra runs on Gemini-2.5-Pro and Claude-Sonnet-4; the setup retrieves 4 models, runs 4 inner loops and 4 outer loops, builds 2 parallel solutions, explores ensembles for 5 rounds, and caps each task at 24 hours. On 96 vCPUs, 360 GB of memory and 8 V100 GPUs with 16 GB each, one solution took 14.1 hours on average, against 15.4 hours for AIDE. The appendix states that example outputs such as the retriever output, the ablation code and the final solution code are placed in the authors' GitHub repository.
    • Results: On the 22 MLE-bench Lite competitions with 3 seeds, MLE-STAR with Gemini-2.5-Pro won a medal in 63.6% of competitions, with gold in 36.4%. With the same Gemini-2.0-Flash, the medal rate rose from 25.8% for AIDE to 43.9% and the above-median rate from 39.4% to 63.6%. Under the same setting the medal rate was 37.9% without ensembling and 43.9% with the proposed ensemble, whose gold rate of 30.3% beat the 25.8% of a simple average ensemble.
    • Limits: Kaggle competitions are public, so the LLM may have been trained on related discussions; as evidence against this the authors report that a Gemini-2.5-Pro judge rated all 75 pairs formed from 25 top discussions in 7 competitions and the final solutions as novel. Evaluation uses the 22-competition Lite split, the low-complexity part of the 75-competition MLE-bench, and the Claude-Sonnet-4 runs and the DS-Agent comparison each cover only 4 tasks. The AIDE results with o1-preview and GPT-4o are taken from the MLE-bench repository and use 16 and 36 seeds, against 3 for MLE-STAR.
    概要PDF
  27. 1217Research AgentsarXiv:2505.17873

    MOOSE-Chem3: Toward Experiment-Guided Hypothesis Ranking via Simulated Experimental Feedback

    • Problem: Wet-lab experiments in the natural sciences are costly and low-throughput, so choosing which hypothesis to test first matters, yet existing hypothesis ranking relies only on LLM internal reasoning without experimental outcomes. Ranking strategies that use earlier results need repeated real experiments, which made them hard to develop and evaluate.
    • Input and output: This is a method paper that introduces a new task, a simulator and a policy. The policy receives a research question and 64 candidate hypotheses (1 ground truth and 63 generated by MOOSE-Chem), tests one hypothesis at a time, and is scored by the number of trials until it finds the ground truth. The simulator takes the research question, a candidate hypothesis and the ground-truth hypothesis hidden from the policy, and returns a performance score between 0 and 1.
    • Method: CSX-Sim uses an LLM to break the ground-truth and candidate hypotheses into key functional components and mechanisms, sets a weight for each component, marks critical components, and scores a candidate by the weighted sum of per-component mechanism similarity; a candidate missing any critical component scores 0. CSX-Rank is an ICRL (in-context reinforcement learning) policy that learns inside the context of a frozen LLM. It classifies components as effective, uncertain or ineffective, drops the ineffective ones, clusters the rest by function, tests a hypothesis from a promising cluster and adds the result analysis to a running summary.
    • Implementation: All experiments use GPT-4o-mini. For simulator validation the authors collected 30 research questions and 124 hypotheses with experimentally reported outcomes (3 to 6 per question) from the literature, and each simulated result is the average of three runs. Ranking is evaluated on the 51 problems of TOMATO-chem, and the official repository holds Data, Method and Simulator folders with scripts for per-problem ranking, ablations and simulator validation.
    • Results: In simulator validation, CSX-Sim reached a Spearman correlation of 0.960, perfect agreement on 26/30 questions and an RMSE of 0.213, better than the Matched Score baseline (0.843, 12/30, 0.232). The average number of trials to find the ground truth was 15.196 for CSX-Rank, fewer than 32.500 for random search, 28.608 for pre-experiment ranking and 27.980 without clustering. Under the most complex noise, CSX-Rank needed 32.706 trials against 36.471 and 40.451 for its ablations, and with 128 candidates it needed 30.7 trials.
    • Limits: Simulator scores deviate from real experimental outcomes, and the authors note that the assumption (A1) that the ground truth is the best hypothesis in its neighborhood is mostly valid, though a better hypothesis may exist there. All ranking evaluations use simulator feedback, and more complex noise increased the number of trials needed. The authors also point to possible biases from the literature-based datasets and the simulator's assumptions.
    公開コード概要PDF
  28. 1216AI AgentsarXiv:2505.10468

    AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges

    • Problem: The authors argue that existing surveys fold AI Agents and Agentic AI into one umbrella of intelligent agents, which blurs the concepts and misaligns system design. Without a clear line, developers under-engineer tasks that need agent coordination and over-engineer tasks a single agent could solve.
    • Scope: This is a literature review (survey). The authors searched twelve platforms, academic databases such as Google Scholar, IEEE Xplore and arXiv plus AI tools such as ChatGPT, Perplexity.ai and Grok, with terms like AI Agents and Agentic AI. Papers were included for novelty, empirical evaluation, architectural contribution and citation impact.
    • Taxonomy: An AI Agent is a single system that uses LLMs and LIMs (Large Image Models such as CLIP) and calls tools for a narrow task, defined by autonomy, task specificity and reactivity. Agentic AI is a system of specialized agents that split a goal, communicate, and are coordinated through persistent memory and an orchestrator (meta-agent). The comparison tables (Tables 1 to 9) contrast the two types against Generative AI as the baseline and add Generative Agents as an inferred fourth type.
    • Applications: For AI Agents the review covers customer support with enterprise search, email triage, personalized recommendation with basic reporting, and scheduling assistants. For Agentic AI it covers multi-agent research assistants, robotics coordination, collaborative medical decision support, and game AI with adaptive workflow automation. Tables 10 and 11 list representative AI Agent and Agentic AI systems from 2023 to 2025 by application area and mode of operation.
    • Challenges and remedies: The review names four main limits of AI Agents: missing causal reasoning, hallucination and shallow reasoning inherited from LLMs, incomplete agentic properties such as autonomy and proactivity, and failures in long-horizon planning and recovery. For Agentic AI it names eight bottlenecks, including inter-agent error cascades, coordination breakdowns, emergent instability, scalability limits and poor explainability. As remedies it proposes ten design strategies such as RAG, function calling, ReAct loops, memory architectures, role-based orchestration and causal modeling.
    • Limits: The authors themselves cite critiques that agent benchmarks rely on artificial environments and that Agentic AI work favors architectural novelty over rigorous empirical validation. The review rests on literature and conceptual comparison, and its detailed application examples are hypothetical scenarios. For medical decision support, the authors state that real-world implementations are still lacking.
    概要PDF
  29. 1215Agents / MemoryarXiv:2610.22273

    Librarian Agents: Training LLMs to Curate Long-Context Workspaces that Anticipate Future Queries

    • The authors built the Librarian Agents framework, in which a builder agent transforms raw documents into a hierarchical filesystem of synthetic files and reference links.
    • They formalize filesystem workspace construction as an MDP and optimize the builder policy directly with reinforcement learning to raise answering accuracy.
    • On BrowseComp-Plus it outperforms the strongest baseline by 33 percentage points on average and reaches 0.77 accuracy with the Gemini 3.1 Flash Lite answering agent.
    • On the unseen BRIGHT domains it scores pass@1 52% and pass@3 67% zero-shot.
    • Construction cost is amortized across repeated queries, with accuracy of 0.64 versus 0.24 for the base builder on reuse.
    • As a limitation, compression may discard details needed for needle-in-a-haystack tasks, so the system keeps reference links for fallback access.
    概要PDF
  30. 1214Reasoning / RLarXiv:2610.22269

    X-RIC: An Agentic RAN-Intelligent Controller Benchmark for Enhancing Reinforcement Learning

    • The authors propose eXplainable RIC (X-RIC), a framework that integrates agentic AI into O-RAN architectures.
    • The BARX evaluation framework assesses agentic reasoning candidates in closed-loop control across four dimensions: latency, compliance, determinism, and policy impact.
    • The Telecom Language Model (TLM) augments LLaMA3-8B with a trainable RAN-state encoder that maps network telemetry into 256-dimensional embeddings.
    • On SLA-aware network slicing it achieves 89% operational cost reduction, converging 75% faster with 50% fewer training steps and 46% lower variance.
    • The non-fine-tuned LLM increases cost by 90.3% due to hallucinations that violate resource constraints.
    • On the OpenAirInterface over-the-air testbed, X-RIC500 maintains margins of 17 Mbps for eMBB and 5 ms for URLLC.
    概要PDF
  31. 1213RoboticsarXiv:2610.21453

    CarryOn: A Dual-Loop Harness for Personalized and Reliable Robot Assistance

    • The authors built CarryOn, a dual-loop harness that augments a frozen VLA policy.
    • The planning loop maintains user-level and scene-level context and revises it through dialogue and observation to dispatch personalized subtasks.
    • The execution loop filters actions against predicted hazardous contact as a proactive shield.
    • After failure, reactive recovery skills restore task progress, and offline reflection refines the recovery skills.
    • Across 900 LIBERO-Safety episodes it reduces the constraint-violation rate from 52.6% to 6.1%.
    • Under a 12 cm object displacement, the pure VLA completes 26/200 episodes while reuse of refined skills raises this to 160/200.
    概要PDF
  32. 1212Research AgentsarXiv:2610.19920

    MolHop: Scaffold-Hopping Molecular Design with Agentic Reinforcement Learning

    • The authors extracted approximately 1.6M scaffolds from the Enamine commercial molecular library and organized them into a coarse-to-fine hierarchical action space from H1 to H7.
    • The agent chooses scaffold navigation, molecular editing, and undo within a single harness, while the environment maintains trajectory memory as a structured state to support long-horizon search.
    • Without training, connecting Grok-4.6 to MolHop raised drug-task joint@H1 from 41.0% to 64.0% and the commercial-compound task from 36.5% to 69.0%.
    • MolHop-9B, trained with SFT followed by GRPO, achieved joint@H1 of 55.5% on the drug dataset and 73.5% on the commercial dataset, beating GPT-5.6-sol, Claude-5-Opus, and Gemini-3.1-Pro.
    • Candidates from MolHop-9B showed scaffold Tanimoto medians of 0.123 and 0.115, indicating larger structural differences than baseline models.
    • Removing the scaffold library or trajectory memory caused substantial performance drops, confirming that the hierarchical search space and memory are key to long-horizon search.
    概要PDF
  33. 1211RAG / AgentsarXiv:2610.19528

    Retrieve What Helps, Only When It Helps: Agentic Reinforcement Learning for Selective Retrieval-Augmented Machine Translation

    • The model learns a policy that chooses reasoning, retrieval query generation, and translation within a single trajectory.
    • It is trained with GRPO using final translation quality as reward, so the model itself decides when to retrieve for each input.
    • To align the retriever with translation quality, the authors built joint Plackett–Luce optimization and translation-quality contrastive training.
    • On IdiomKB Zh→En, SRAT achieved I5 of 0.731 and MQM of -2.74, beating the strongest baselines.
    • On RAGtrans, contrastive training raised retrieval accuracy from 56.3% to 70.7% on En→Zh and from 72.6% to 81.4% on En→De.
    • With a frozen black-box Claude Sonnet 4.6 translator, SRAT-Claude improved RAGtrans En→De MQM by 16.8% over direct translation.
    概要PDF
  34. 1210Memory / AgentsarXiv:2610.19366

    Seeing, Maintaining, and Learning: The Evolution of Multimodal Agent Memory

    • The survey organizes memory along two axes: Axis I covers what representation retained experience takes, and Axis II covers how memory state and management policy change during operation.
    • The management axis has three tiers: T1 reuses fixed memory, T2 updates memory through a stable policy, and T3 evolves the management policy itself using feedback.
    • Analysis of 297 architectures showed that T2 Dynamic Memory is the most common with 204 architectures (68.7%), while T3 Self-Evolving Memory accounts for only 17 (5.7%).
    • Hybrid representations form the largest category with 139 architectures (46.8%), and designs keeping explicit records together with source evidence are widely used.
    • Of the 17 T3 Self-Evolving Memory architectures, 13 are cross-represented systems, and no memory-policy evolution case exists with only latent or only parametric representations.
    • Across 82 benchmarks, cross-modal and long-range evidence are well evaluated, but only 7 and 10 benchmarks directly evaluate conflicting-memory handling and unanswerable queries, respectively.
    概要PDF
  35. 1209AI / AgentsarXiv:2610.18449

    Autonomous Science Bench: Evaluating AI Agents for Autonomous Scientific Discovery

    • Autonomous Science Bench is a science-agent benchmark comprising six budgeted design-build-test-learn campaigns grounded in published experiments across biology, chemistry, and materials science.
    • It evaluates an active experimentation loop where agents select design batches, receive measurements, and plan subsequent experiments before delivering a predictive model.
    • In a pilot of eight agent-model configurations and 144 trials, the best configuration succeeded in 67% of trials and the hardest campaign was solved in only three of 24 attempts.
    • Budgets cover at most 11% of each candidate pool, so covering the pool evenly is not enough to pass.
    • Opus 5.5 in Claude Code resolved 67% of campaign trials, with GPT-5.6 Terra and GPT-5.6 Luna following at 44% and 28%.
    概要PDF
  36. 1208Agents / MemoryarXiv:2610.18052

    Can Agents Forget When Asked? A Benchmark for Memory Unlearning in LLM Agents

    • Bemula is the first benchmark dedicated to memory unlearning in LLM agents, the ability to delete exactly what the user requests.
    • Its tasks jointly measure removal of requested facts and preservation of the rest of memory.
    • Across all evaluated systems and backbones, no system passed more than 1.5% of forget requests end to end, and most passed none at all.
    • A human validation of the LLM judge on 100 sampled items yielded 98% agreement, supporting evaluation reliability.
    概要PDF
  37. 1207RoboticsarXiv:2610.17881

    Agentic Robot Creation through Generation, Verification, and Revision

    • RoboForge is an agentic framework covering both embodiment and behavior creation, chaining motion generation, policy learning, and execution-grounded revision.
    • Motion stays in the loop: rollout feedback can revise the reference motion, the reward, or both.
    • Computational metrics exposed recurring mechanical failures in generated designs before fabrication.
    • Against Eureka-style reward-only refinement it improved the overall rating from 2.10 to 3.97, showing the value of structured motion guidance.
    • The team fabricated an agent-generated Gorilla robot and directly deployed learned behaviors on the physical machine.
    概要PDF
  38. 1206ResearcharXiv:2610.12183

    A Closer Look at Agentic BBO: Benchmarking LLM Agents for Black-Box Optimization

    • The authors built AgenticBBO-Bench, a unified-budget benchmark spanning synthetic functions, hyperparameter optimization, database tuning, chip design, and molecular design.
    • The Agentic setting beat Direct on all five families with an average relative improvement of 30.3%, and on BBOB the score rose from 0.460 to 0.661.
    • Adding GP-based numerical tools did not consistently help, so the base Agentic mean of 0.589 was higher than 0.574 for + GP Suggest and 0.538 for + GP Tools.
    • Task and variable semantics raised the mean score from 0.465 to 0.595, while more specific priors were less reliable.
    • Handing the search to TuRBO after the agent warm-up scored 0.570, close to continued Agentic optimization and clearly above TuRBO started from the initial observations at 0.475.
    • On the frontier challenge GPT-6 Astra scored the highest mean of 0.591, closely followed by DeepSeek V4.1 Flash at 0.584, yet no model dominated every domain.
    概要PDF
  39. 1205Research AgentsarXiv:2610.12086

    EvoAlloc: A Self-Evolving Resource Allocation Agent for Efficient Program Evolution

    • In program evolution, evaluation is computationally expensive, so allocating limited resources to promising candidates was a central decision of the search.
    • EvoAlloc periodically consolidated prior search and allocation outcomes into reusable experience, which informed subsequent strategy revisions.
    • A counterfactual exploration mechanism occasionally evaluated candidates denied resources by the allocator, revealing hidden outcomes and enriching its experience.
    • Across coding and agent-harness optimization benchmarks, EvoAlloc required 59–82% fewer full evaluations and 61–89% fewer LLM tokens, and achieved 8.7–12.0% higher final performance under the same budget.
    • With the same Full-evaluation budget on the same seed, EvoAlloc examined 124 proposals on ADAS-AIME versus 75 for ShinkaEvolve, widening the search.
    • Across exploration rates, final performance was similar but tokens to the target varied substantially, and ρ = 0.3 was the most efficient.
    概要PDF
  40. 1204RoboticsarXiv:2610.11722

    From Solo to Ensemble: A Hierarchical Framework for Composable Multi-Agent Human-Object Interaction

    • The coordination problem where multiple humanoids handle the same object was hard because contact locations, motion directions, and force application must all be matched together.
    • The authors built a distillation-based hierarchical learning framework that converts single-human HOI knowledge into a reusable Object-oriented Motion Skill.
    • In downstream tasks the distilled skill is reused as a frozen executor, while a high-level policy generates region-wise object motions conditioned on the shared object and task goal.
    • On carrying tasks the method reached a success rate of 85.63 on four-human H-box carrying, higher than CooHOI at 80.96.
    • Although the skill was distilled only from single-human carrying, it supported two- and four-human sofa pushing by changing only the high-level policy.
    • In the standalone skill evaluation, the teacher-rollout targets gave a position error Epos of 79.8 (mm), a reach rate TTR of 100.0, and a trajectory success Succtraj of 98.9.
    概要PDF
  41. 1203RoboticsarXiv:2610.12457

    SpatialHarness: Test-Time Spatial Scaffolding for Fine Robotic Manipulation

    • SpatialHarness maintains an online simulated scene synchronized with real-world execution, identifies task-critical spatial relationships, and renders complementary virtual views.
    • It supplies spatial information to a frozen multimodal policy without policy fine-tuning or changes to the physical sensing setup.
    • With the same frozen GPT-6 Astra policy, plug-insertion success rose from 26.7% to 66.7% and Tower of Hanoi from 0% to 100%.
    • The average success rate improved from 18.33% to 83.33%, a gain of 65.00 percentage points.
    概要PDF
  42. 1202AI / AgentsarXiv:2610.12002

    Agentic-TTT: Training test-time policy for test-time training

    • Agentic-TTT turns TTT procedures into callable tools, treats accumulated skills as an evolving deployment environment, and trains a test-time policy by reinforcement learning.
    • As the compute penalty rose from 0 to 0.4, the direct-answer rate increased monotonically from 11.8% to 39.8% while adaptation compute fell 35.5%.
    • Moderate penalties (0.1 and 0.2) cut compute by about 30% while achieving higher mean utility than no penalty.
    • Raising the tool weight from 1 to 4 improved final utility by 0.046 and reduced the invalid-action rate from 4.08% to 1.72%.
    概要PDF
  43. 1201AI / AgentsarXiv:2610.11345

    SynCo: Data Synthesis Co-Training for Self-Evolving LLMs via Multi-Agent Reinforcement Learning

    • SynCo is a data synthesis co-training framework that jointly optimizes an independently parameterized Synthesizer and Reasoner via multi-agent reinforcement learning.
    • The Synthesizer constructs training tasks from the Reasoner's evolving capabilities, resolving the capability-experience mismatch of static datasets.
    • On mathematical reasoning SYNCO reached 78.72%, outperforming the strongest external baseline by 3.76 points and the frozen synthesizer by 2.87.
    • As training progressed, the share of gradient-bearing tasks fell from 31.3% to 9.5% while all-correct tasks rose from 34.4% to 59.9%.
    • Only 16.2% of 2,940 verified tasks gave non-degenerate feedback, and teachability tracked this property at r = 0.934 while novelty was irrelevant at r = -0.041.
    概要PDF
  44. 1200Machine LearningarXiv:2610.12190

    DataSense-Bench: The First Step Toward an AI Scientist

    • The agent selected disjoint, equal-sized subsets of candidate training data and ranked them by expected value for post-training performance.
    • The benchmark evaluated Qwen3-4B performance after training on the predicted top-ranked group under a standardized protocol.
    • On BFCL, all six agents' first groups exceeded random selection, but gains ranged from only 0.35 to 1.78 % points.
    • On TBLite, two of ten agents underperformed random selection, and the leading agent improved on it by only 0.99 points.
    • Ranking ability did not hold consistently across tasks. GPT-6 Astra identified the best group in all three BFCL runs but in only one of three TBLite runs.
    • Selection recipes varied substantially across three independent runs of the same agent, changing the selection rule itself.
    概要PDF
  45. 1199AI / AgentsarXiv:2610.11768

    Narrow and Deep: An Ontology Tower as the Knowledge of an LLM Agent for an Industrial Equipment System

    • The authors proposed an ontology tower for a single equipment system, which deepens through quantities derived from measured points by physical relations and lessons incorporated from the operating journal.
    • The studied tower consists of 162 nodes and 217 edges, holding 42 tags, 34 assumptions, 33 bindings, 24 knowledge nodes, 16 derived quantities, and 13 plant elements.
    • In a preregistered evaluation, the text projected from the tower raised the primary-trap avoidance rate of the four models from 0.605 to 0.802.
    • With the projected text, the 9-billion-parameter model scored above the largest model without it, at 0.737 against 0.672.
    • Lessons left in the operating journal were seldom used by the agents, and were used when incorporated into the tower or placed in the prompt as records.
    • In live runs the agents brought the controlled variable into its target band in 12 of 14 runs, and no setpoint changed except through the safety layer.
    概要PDF
  46. 1198Software EngineeringarXiv:2610.11578

    Chronos Enables Code Agents to Reason over Software Evolution

    • Chronos distilled merged pull requests into experience cards with four fields: Summary, Signals, Motivation, and Solution.
    • It built an evolution graph connecting the cards through eleven typed relations of code continuity, developer intent, and organizational context.
    • Semantic search located entry cards and weighted multi-hop expansion retrieved connected history for the agent to selectively read.
    • On SWE-Bench Verified it improved SWE-Agent across all six LLM backbones, raising the mean resolution rate from 69.2% to 72.9% and reaching 79.8% with MiniMax M2.5.
    • With the same backbone it raised resolution rates from 48.3% to 51.7% on SWE-Bench Pro and from 41.0% to 43.5% on FEA-Bench Lite.
    • In a human evaluation on 100 tasks, graph-grounded retrieval increased the mean useful cards from 1.24 to 2.87 and the fraction of tasks receiving useful experience from 68% to 89%.
    概要PDF
  47. 1197Research AgentsarXiv:2610.12419

    OneSearch-VL: Unified Multimodal Deep Research Agent for Image and Video

    • It encodes the links from on-screen visual anchors to real-world entities, source-supported web facts and answer-producing operations (comparison, arithmetic, counting) in one graph, VGEG, used for data construction, reward and evaluation.
    • A data engine that narrows 2.5M web videos to 70k produced about 110k training trajectories (36k single-image, 37k multi-image, 35k video) and about 10k RL tasks.
    • Beyond answer correctness and query quality, the RL reward adds EVGR, which separately scores whether tool observations actually support the required facts (evidence traceability) and whether the right objects and frames were found and used (visual grounding).
    • OneSearch-VL-8B beat the same-size Qwen3-VL-8B agent by 20.2 points on OneSearch-MI-Bench, 17.6 on OneSearch-Video-Bench and 27.0 on VideoDR, and averaged 58.3 on the image benchmarks, 1.7 points above OpenSearch-VL-8B.
    • Joint SFT on all three input types reached a benchmark average of 55.8 versus 51.7–55.1 for single-type training, and the EVGR reward raised the average from 57.3 to 61.1.
    • The authors note that external search tools and changing webpages limit exact reproducibility, and that automated VGEG construction and model-based rewards may introduce annotation errors or judging biases.
    概要PDF
  48. 1196AgentsarXiv:2610.11464

    Who Verifies the Verifier? Co-Evolving Inspectable Graders with Self-Improving Agents

    • The verifier is built as an expression over small drawback detectors (static checks, execution checks and narrow LLM queries) and evolves by synthesizing detectors from clustered failures and retiring useless ones.
    • On MBPP+ the evolved verifier raised held-out agreement from 0.417 for the hand-authored seed composition to 0.625, also above the bare LLM judge it contains (0.55).
    • With the two anchor guards switched off, the verifier collapsed into a grader that passes 0.97–1.00 of everything it grades, yet skills trained under it scored about as well as under the anchored verifier.
    • Double Ratchet, which co-evolves the verifier with the skill loop, kept 106% (MBPP+), 110% (Spider 2.0) and 88% (report generation) of the lift the same loop gets from ground-truth grading.
    • In report generation, skills gamed the rubric by writing tags in place of numbers; an outer judge caught it, and one added detector cut valueless tags from roughly 30% to about 1%.
    • The MBPP+ lift pointed the same way on all three seeds, but the task-level test stayed at p = 0.12, and the study covers three task families and one Claude model family.
    概要PDF
  49. 1195AgentsarXiv:2610.10444

    RunningTab: Direct Workspace Interaction with Environment-Side Tabs

    • Solves the problem of files being read but never reflected in the final deliverable with an environment-side ledger.
    • The agent adds requirements, while the environment records the provenance of read content and a list of files still unopened.
    • If the agent tries to finish with unresolved requirements, a final check sends them back for review.
    • Fits naturally with accessing workspace files directly via a terminal, without any pre-indexing.
    • Showed consistent performance gains across three benchmarks and three LLMs.
    概要PDF
  50. 1194AgentsarXiv:2610.10513

    SciExam for ENSO: Can AI Agents Build Climate Models?

    • Built the first family of benchmarks that grade results solely with hidden scientific criteria (statistics, variable dependence, forecasting skill), with no answer key, rubric, or LLM judge.
    • Agents must freeze their own diagnostic metrics in advance, so they cannot tweak the grading criteria to fit their results.
    • Six of twelve agents scored higher than the published reference model, with strengths in variable reconstruction and forecasting.
    • The top agents' model structures were each compatible with one of the two competing hypotheses explaining ENSO asymmetry, producing results directly relevant to an open scientific debate.
    • Scores held up even when calendar years were hidden, confirming the agents were not memorizing the observational record.
    概要PDF