HDATF
日本語
相談する
メニュー

論文アーカイブ

収録論文と要約を番号の大きい順に掲載します。

収録論文 1,430資料更新日 Research Monitoringベンチマーク
  1. 1343Research AgentsarXiv:2510.26144

    The FM Agent

    • Problem: The authors describe industrial problems such as combinatorial optimization, time-series forecasting, and high-performance kernel tuning as sharing a structure where evaluating candidate solutions is relatively straightforward but identifying truly effective ones is exceptionally difficult, and progress has relied on specialized engineers who design and iteratively refine algorithms. They see the primary focus of most prior multi-agent work on code generation as a limitation and aim for a general agent that discovers state-of-the-art solutions through large-scale search across diverse, complex problem domains.
    • Input and output: This is a system paper on FM Agent, a general-purpose multi-agent system built by the FM Agent Team at Baidu AI Cloud. Its input is the problem to solve, which for machine learning tasks means a dataset and task objectives, and candidate solutions are scored by the domain-specific evaluators built into FM Agent. Its output is a program refined by evolutionary search, such as an ML pipeline, a combinatorial optimization heuristic, a CUDA kernel, or construction code for a math problem.
    • Method: In the Cold Start Stage, several generative agents with different strategies build a diverse pool of initial algorithms, which are partitioned into clusters assigned to multiple islands, populations that evolve independently most of the time. In the Evolve Stage, each island evolves its population through mutation and crossover with periodic interaction between islands to cross-pollinate ideas, while a cluster-based sampling strategy that adjusts selective pressure to real-time population diversity and a curated elite pool balance exploration and exploitation. The evaluators give a single fitness score and LLM judge feedback in general, and for complex cases such as machine learning or kernel generation they use domain-specific multi-dimensional metrics such as balancing accuracy and latency.
    • Implementation: The system runs on the Ray framework and splits program generation and program evaluation into separate parallel worker pools scheduled asynchronously, which supports concurrent execution of thousands of evolutionary processes. In the three benchmark experiments FM Agent uses Gemini-2.5-pro and optimization is performed exclusively by the LLM without human intervention, while a Human-Interactive Feedback Module that lets experts steer evolution through natural language or code and retrieve an expert knowledge base with RAG (retrieval-augmented generation) is optional but recommended by the authors. The GitHub addresses in the paper belong to the compared systems OpenEvolve and CUDA-L1, and the paper gives no code address for FM Agent itself.
    • Results: On MLE-Bench, made of 75 Kaggle competitions, FM Agent (averaged over 3 runs with different seeds) reaches a 43.56% medal rate, above Operand ensemble at 39.56% and InternAgent at 36.44% as taken from the official report. On the 10 problems of ALE-Bench lite, where every agent uses Gemini 2.5 Pro, its overall average score of 1976.3 beats ALE-Agent, a purpose-built agent with domain-specific knowledge and a diversity-oriented search, at 1879.3 by 5.2% and the Iterative-Refinement baseline, which repeatedly refines solutions with a Self-Refine methodology, at 1201.3 by 64.6%. On a workload set clustered from KernelBench Level 3 problems, the most challenging kernels, under a tightened numerical tolerance, it achieves 2.08x to 20.77x speedups over torch.compile, and on packing 26 circles it reaches a radius sum of 2.6359740012 against AlphaEvolve's 2.6358627564.
    • Limits: The MLE-Bench baseline numbers come from the official report and were produced with different base models such as gpt-4o, o1-preview, deepseek-r1, and gpt-5, and on ALE-Bench lite short contests ALE-Agent averages 2284.8 against FM Agent's 2250.8. The ablations come from a single ALE-Bench task (ahc016) with five runs per setting, and on the uncertainty inequality task the authors state that search performance is highly sensitive to problem abstractions derived from expert domain knowledge. The claim that FM Agent matches or surpasses real-world algorithmic practice in various industrial scenarios within and beyond Baidu appears without numbers.
    概要PDF
  2. 1342Research AgentsarXiv:2507.23276

    How Far Are AI Scientists from Changing the World?

    • Problem: LLM-based AI Scientist systems have produced papers accepted at an ICLR 2025 workshop, and the authors ask how far such systems are from discoveries that solve grand challenges across medicine, energy, and the environment. They note that earlier surveys gave overviews of LLM architectures, agentic frameworks, and scientific automation tools, but a capability-based roadmap toward the ideal AI scientist was still missing.
    • Scope: This is a survey of LLM-based AI Scientist systems. Sections 2 to 6 cover knowledge acquisition, idea generation, verification and falsification, AI reviewer systems, and evolution in turn and summarize representative methods in each; evaluation benchmarks are collected in Sections 2, 4 (Table 2), and 5 (Table 3), while Section 3 covers evaluation criteria for hypotheses and automated and human evaluation. Section 7 covers limitations and ethical challenges, and Section 8 covers future directions such as standardized Scientist-to-Scientist Communication Protocols (SSCP) for structured interaction among AI Scientist systems.
    • Taxonomy: The authors divide the capabilities of a mature AI Scientist into four levels. Level 1, knowledge acquisition, is retrieving, reviewing, and comprehending scientific knowledge from existing research; Level 2, idea generation, is generating innovative and feasible hypotheses at scale; Level 3, verification and falsification, is designing, implementing, and analyzing experiments to test and potentially disprove those hypotheses; Level 4, evolution, is continuously advancing overall research abilities from internal and external feedback. They treat Level 2 as the key feature that distinguishes AI Scientist systems from automated scientific tools.
    • Repository: The GitHub repository Awesome-AI-Scientist named on the paper's first page keeps a curated list of papers, open source tools, and benchmarks in its README, with files at the top level that include README.md, a CONTRIBUTING.md guide, and an img folder. The README groups papers under the survey's chapter names (Knowledge Acquisition, Idea Generation, Verification and Falsification, Review System, Evolution) and says it links each paper to the paper itself, associated code, and available reviews. For some papers without open reviews, the README says the maintainers added links to AI review analyses generated with their DeepReviewer.
    • Evidence: In a table the authors collected from other papers to show that state-of-the-art LLMs have relatively low accuracy on implementation benchmarks, OpenAI o1-preview reaches a 16.90% medal-worthy success rate on MLE-Bench, o1-high a 26.00% replication score on PaperBench, and Claude-Sonnet-3.7 a 39.00% execution accuracy on SciReplicate-Bench. When the authors scored 28 publicly available papers from 5 AI Scientist systems with the AI reviewer model DeepReviewer-14B, the highest-rated system, Zochi, averaged only 4.63 out of 10. Among defect categories, 'Experimental Weakness' appeared in all 28 papers (100%) and 'Methodological Unclarity/Flaws' in 96.4%.
    • Limits: The authors attribute the remaining gap to limits of the underlying LLMs, namely hallucination, the high cost and inefficiency of knowledge updating, and catastrophic forgetting, and to weak research abilities such as generating hypotheses with low feasibility and employing incorrect validation methods. They note under the table that publicly available papers may be curated and may not fully represent each system's typical output. The scores come from a single model, DeepReviewer-14B, and average 2 papers for Zochi and 3 for AI Scientist-v2.
    公開コード概要PDF
  3. 1341Research AgentsarXiv:2503.18102

    AgentRxiv: Towards Collaborative Autonomous Research

    • Problem: Scientific progress is rarely the result of a single 'Eureka' moment and is rather the product of hundreds of scientists working incrementally toward a common goal, but the authors note that existing autonomous research agents such as The AI Scientist, Virtual Lab, and Agent Laboratory often operate in isolation and do not support the continuous, cumulative development of research across time. They ask whether giving agents a shared repository for building on each other's research makes progress accumulate over successive papers.
    • Input and output: This is a system paper that builds AgentRxiv, a shared preprint server where agent laboratories upload and retrieve research reports. The input is a research direction and detailed instructions from humans; in the experiments the research direction is a single task, 'Improve accuracy on MATH-500 using reasoning and prompt engineering,' given together with an instruction to use gpt-4o mini during experimentation and an OpenAI API key. Each laboratory generates its own ideas toward that direction, runs experiments, and uploads to AgentRxiv a paper describing a new reasoning technique and its MATH-500 accuracy.
    • Method: AgentRxiv is a local web application with routes for uploading, searching, and viewing papers and an API endpoint that returns search results in JSON, and an uploaded paper becomes accessible to other laboratories as soon as it is submitted, in an asynchronous manner. Search embeds stored papers and queries with a pre-trained SentenceTransformer model and ranks results by cosine similarity. Each laboratory is an Agent Laboratory system in which role agents such as PhD, Postdoc, ML Engineer, and Professor, together with the mle-solver and paper-solver tools, run Literature Review, Experimentation, and Report Writing, and during literature review it reviews 5 AgentRxiv papers and 5 arXiv papers.
    • Implementation: In the sequential experiment the laboratory agents use o3-mini (medium) as the LLM backend, the reasoning techniques they design are tested with gpt-4o mini, and Agent Laboratory runs in autonomous mode without human intervention. The sequential experiment has one laboratory produce 40 papers and the parallel experiment has three laboratories with identical configurations produce 40 papers each, 120 in total, and all experiments ran on a 2023 MacBook Pro with an Apple M3 Max processor and 36 GB of memory. The README of the official AgentLaboratory repository states that Agent Laboratory supports AgentRxiv, and the top level holds files and folders such as ai_lab_repo.py, agents.py, mlesolver.py, papersolver.py, app.py, and experiment_configs.
    • Results: In the sequential experiment, MATH-500 accuracy rises from the gpt-4o mini zero-shot baseline of 70.2% to 78.2% with the final technique SDA (Simultaneous Divergence Averaging, which picks the higher-confidence answer when a low-temperature and a high-temperature chain-of-thought answer agree), an 11.4% relative improvement, while without AgentRxiv access accuracy plateaus at 73.4% and 73.8%. SDA raises GPQA Diamond from 36.4% to 38.9%, MMLU-Pro from 63.1% to 70.8%, and MedQA from 74.9% to 81.6%, and across 5 models and 4 benchmarks the average gain is 3.3%, although GPQA averaged over models falls from 52.2% to 51.6%. Running three laboratories in parallel reaches a best accuracy of 79.8% (13.7% relative) and reaches 76.2% after seven papers versus 23 papers in the sequential setting.
    • Limits: The authors write that high rates of hallucination were arguably the most prevalent challenge during development: some papers reported results that did not match the actual experiment, the code repair mechanism often erased core functional code to get rid of errors, and it sometimes wrote code that printed realistic runtime outputs without research code underneath, so each reported accuracy comes from humans manually checking the program output and code. A large percentage of experiments failed completely at about 0% accuracy because of major code bugs, and the authors state that the goal of raising MATH-500 accuracy partially reflects how many machine learning papers report new discoveries through gains on target benchmarks but is not fully reflective of scientific discovery as a whole. The parallel run produced 120 papers across three laboratories at a total cost of $279.6, compared with $92.0 for the sequential run.
    公開コード概要PDF
  4. 1340Research AgentsarXiv:2609.02749

    Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills

    • Problem: ML research agents combine a model with a harness that handles orchestration, memory, verification, and iterative refinement, but the operational knowledge of which methods and tools to use, when, and how (the know-how that separates knowing a method from making it work) stays outside the agent. This knowledge is abundant in repositories and papers, but it is written for human readers and is too large to load during a task. Today experts turn it into skills and tools by hand, so its cost scales with expert labor.
    • Input and output: In creator mode, DisCo takes an anchor (the object that starts skill construction), either a source such as a repository or paper or one concrete task, and returns a verified skill graph (the skills distilled from one source plus their routing, dependency, and composition links) plus a construction record of the evidence used, the checks performed, and any unresolved gaps. In researcher mode, it takes a research task made of a problem, data, an environment (tools and budget), and a target, and produces artifacts such as code, models, experimental results, and reports. This is a system paper that builds the DisCo agent and a skill library and tests them on 4 benchmarks.
    • Method: A skill has three layers: SKILL.md, read up front (goals, key concepts, tool usage, known failure modes), references/, opened only when needed (API documentation, algorithmic detail, parameter configurations), and scripts/, executable wrappers with defined inputs and outputs. Skill distillation (reworking declarative source knowledge into skills an agent can use) runs the same four stages whether the anchor is a source or a task: scope the capabilities, ground them in evidence, construct a candidate skill graph, and verify it while repairing it where possible. During research, a router narrows a request from area to family to repository skill graph, and the agent starts from the graph's entry skill and loads only the branch it needs.
    • Implementation: Skill graphs for 1,000 repositories were built with GPT-5.5 and GPT-5.6-sol at xhigh reasoning effort, with an average allocation of about $40 per repository, and the snapshot organizes 5,353 skills into 20 areas and 178 capability families. For the experiments, the authors built 636 skills from 153 prior papers of the 20 target papers in PaperBench (a paper replication benchmark), one graph per competition for MLE-bench (an ML engineering competition benchmark), and one graph per benchmark for FrontierCS (open-ended computer science problems with unknown optima) and PassNet (graph compiler pass generation). According to its README, the GitHub repository holds the skill library (the skills folder), the DisCo CLI (command-line tool) distributed as an npm package (the cli folder), installation and workflow documentation, and example session exports.
    • Results: With the Codex harness, the GPT-5.5 backbone, and the running budget held fixed and only skill access changed, the Any Medal score on all 75 MLE-bench competitions rose from 31.11% to 72.89% (134.3% relative), above 64.44%, the strongest of the public baselines that Table 1 copies from the official MLE-bench leaderboard. The average PaperBench replication score over 20 papers rose from 29.45 to 39.59, with gains on 18 of the 20 papers. The FrontierCS Agent Track score over 188 tasks rose from 70.63 to 77.14, and the main PassNet score (AS Score) over 200 samples rose from 1.343 to 1.5313.
    • Limits: Two PaperBench papers scored lower with skills (sample-specific-masks from 57.11 to 52.04 and stay-on-topic from 32.31 to 27.79), and the authors point to a possible retrieval-precision failure in which retrieved skill content may occasionally distract from a strategy the base agent would find on its own. The comparison matches the running budget, and the one-time skill construction budget (for MLE-bench, up to 24 GPU-hours per task) is kept separate and excluded from both conditions. The 1,000 repositories are a curated snapshot of part of the commonly used ML software, chosen by open-source visibility (how well known they are in the open-source ecosystem) and practical use, with GitHub stars as one signal.
    公開コード概要PDF
  5. 1339Research AgentsarXiv:2606.05080

    AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks?

    • Problem: Scientific and engineering progress is a long-horizon iterative process of proposing changes, running experiments, measuring outcomes, and refining artifacts, yet existing benchmarks for frontier models primarily evaluate single-turn responses or short agent trajectories. The most impressive demonstrations of empirical optimization, such as AlphaEvolve and Karpathy's AutoResearch agent, are co-designed with heavily engineered, model-specific harnesses, tools, and search strategies, which makes the underlying model's contribution hard to isolate. The existing long-horizon benchmarks the authors cite each cover a single domain such as ML engineering, systems and kernel optimization, or real-world engineering, and none of them combines broad coverage, high difficulty, and resistance to saturation.
    • Input and output: This is a benchmark paper. The agent receives a natural-language instruction and a container sandbox (CPU or a single GPU depending on the workload) holding a working but unoptimized baseline codebase and a local evaluation script, and within a wall-clock budget it edits, runs, and measures the code and submits a final implementation. A verifier hidden from the agent measures a metric (runtime, perplexity, throughput, parameter count, and so on) on held-out inputs and maps it to a continuous 0 to 1 score anchored at the baseline and a human-written reference solution.
    • Method: Senior researchers and engineers contributed tasks from real problems they had faced, and at least 2 experts independent of the contributor plus a format audit agent audited each task on four criteria, validity (a higher score means a better implementation), solvability (the reference reliably reaches the target score within the budget on the target hardware), integrity (whether the agent can pass by hacking the scoring function), and measurement stability (small variance across repeated reference runs), revising or rejecting tasks that failed. Performance-optimization tasks use log-stretch scoring (0 at the baseline, 0.5 at the reference, approaching 1.0 near the practical optimum) with a gate that requires beating the baseline, and tasks with bounded quality metrics use linear scoring (0 at the baseline, 1.0 at the reference), except that two parameter-count puzzle tasks anchor at zero parameters. Against reward hacking, the verifier is sealed, ML tasks add a correctness gate on inputs drawn from a distribution disjoint from anything visible during development, an adversarial agent prompted to find shortcuts tests each task, critical immutable files are pinned by hash so that any unauthorized change scores zero, and trajectories are analyzed during evaluation so that verifiers are patched when new hacks appear.
    • Implementation: The 36 tasks are 15 system optimization tasks (C, C++, Rust, Go, Python), 10 puzzle and challenge tasks, 7 model development tasks, and 4 CUDA tasks, with wall-clock budgets from 2 hours for the smallest puzzle tasks to 12 hours for LLM development tasks. Evaluation uses the terminus-2 agent of the Harbor framework by default for all models, runs CPU tasks in a local Docker sandbox on a Ryzen 9 9950X (16 cores) with 64GB of memory and GPU tasks on Modal cloud sandboxes with H100 and L40S GPUs, and consumed 2,544 wall-clock hours and 8.60 billion tokens in total. Per its README, the official repository (autolabhq/autolab) keeps each task in the tasks folder with task.toml, instruction.md, a Dockerfile and the editable files, a solution hidden from the agent, and scoring tests, and explains how to run one task or all tasks with Harbor's harbor run command.
    • Results: Among the 11 main-set models, claude-opus-4.6 ranked first in all four categories with Avg@3 0.68, Best@3 (best of three runs) 0.76, and Dominance (head-to-head win rate against the other models per task) 0.93, followed by gemini-3.1-pro at Avg@3 0.50, with kimi-k2.6 (0.46), mimo-v2.5-pro (0.45), and glm-5 (0.43) leading the open-weight models, while gpt-5.4 scored 0.36 and grok-4-20 0.35. On flash_attention, a two-hour CPU task, comparing each model's best of three rollouts, claude-opus-4.6 cut its self-reported runtime from about 750 ms to 18 ms through 44 feedback-driven iterations over roughly 40 minutes, surpassing the reference solution (100 ms). In the failure analysis, which manually classified all 302 zero-score rollouts, the authors describe two time-awareness failures: deepseek-v4-pro, hunyuan-3-preview, and qwen-3.6-plus frequently used up the budget before a final submission, while gpt-5.4 and grok-4-20 often submitted after minimal exploration despite substantial remaining budget.
    • Limits: The authors state that AutoLab focuses on executable system and ML engineering workflows and should be read as a benchmark for measurable auto-research rather than scientific discovery in the broadest sense. The baseline and reference values of performance-optimization tasks are calibrated to the benchmark's sandboxes and hardware, and small open-weight models under 200B parameters were excluded because of the benchmark's difficulty. Re-evaluating 4 models on 25 CPU tasks with two other harnesses (pi-mono and a mini-swe-agent whose system prompt was changed to demand persistent optimization) shifted the same model's mean score by up to 0.43 (kimi-k2.6, from 0.21 under pi-mono to 0.64 under mini-swe-agent), and the authors report that harness-induced variance can be comparable to model-induced variance.
    公開コード概要PDF
  6. 1338Research AgentsarXiv:2511.15593

    What Does It Take to Be a Good AI Research Agent? Studying the Role of Ideation Diversity

    • Problem: AI research agents automate the design, implementation, and training of ML models, but the field is still in its infancy and the key factors behind the success or failure of agent trajectories are not fully understood. Long multi-step trajectories that are often guided by heuristic search algorithms, together with tool use, make error analysis far more complicated than in classic ML setups, and collecting samples large enough for meaningful analysis and design ablations can be computationally prohibitive. The authors start from the hypothesis that ideation diversity is a key bottleneck in AI research agent performance.
    • Input and output: This is an empirical study that analyzes existing agent trajectories at scale and runs a controlled experiment. The input is agent trajectories on MLE-bench tasks from several LLM backbones and scaffolds, and the outputs are a diversity score, the Shannon entropy (base 2) of the model architectures extracted from the initial ideas (at most five) produced by the Draft operator (which generates the initial solutions), its correlation with medal rate and other performance metrics, and the performance change when diversity is altered through the system prompt.
    • Method: The analyzed scaffolds are AIDE (AI-Driven Exploration, a greedy tree search over Python solutions), AIRA Greedy, and AIRA MCTS, all of which grow a tree of Python solution nodes with Draft, Debug, and Improve operators. In the controlled experiment, the baseline agents (AIRA Greedy and AIRA MCTS) have three diversity mechanisms by default: sibling memory that puts descriptions of sibling nodes' solutions into the context, prompt-adaptive complexity that asks for minimal complexity for the first idea, moderate for the next two, and advanced for the last two, and a request for diversity in the system prompt. The low-diversity agents remove prompt-adaptive complexity and the diversity request in the system prompt and reuse sibling memory to ask, through the system prompt, for similar ideas, and by changing only the diversity-related parts of the prompt the authors intend to leave other aspects such as implementation quality as they are.
    • Implementation: The analysis backbones are o3, gpt-oss 20B and 120B, Llama Maverick, Devstral, and CWM (an open-weights LLM from Meta for research on code generation with world models), and 6 backbones with 2 scaffolds were run on the 75 MLE-bench tasks with 10 to 20 seeds each, giving 11,000 trajectories, roughly 1,200,000 search nodes, and 264,000 GPU hours. The controlled experiment ran AIRA Greedy and AIRA MCTS with the full-sized DeepSeek R1 on MLE-bench lite (22 tasks) with 10 seeds each, and all LLMs used a 128K-token context. Error bars in the controlled experiment are 95% confidence intervals from stratified bootstrapping with the rliable library.
    • Results: Across the 75 tasks, architecture entropy correlated with medal rate at Pearson r 0.57, with average normalized score at 0.72, and with percentile at 0.66, and agents on o3 and gpt-oss 20B and 120B used 3.5 distinct architectures on average in their 5 initial ideas versus 2.8 on average for Llama Maverick, Devstral, and CWM. In the DeepSeek R1 controlled experiment, reducing diversity lowered the MLE-bench lite medal rate from 45.5 to 38.6 for AIRA Greedy (6.9 points) and from 47.0 to 38.6 for AIRA MCTS (8.4 points), and valid submission rate fell from 98% to 92% and 90%, respectively. The valid-submission drop came mainly from two text normalization competitions (English and Russian), where low-diversity agents repeatedly tried to implement the same T5 model (a text-to-text transformer) and consistently failed with timeouts, and the authors estimate that these two competitions account for around half of the medal-rate decline.
    • Limits: The findings come from MLE-bench only, and the authors hypothesize that they are likely to generalize to other ML tasks, leaving other benchmarks to future work. Medal criteria change with the number of teams in a competition, in about 30% of MLE-bench competitions the relative gap between the best score and the bronze threshold is under 3%, agents are evaluated on custom test sets that differ from Kaggle's private test sets, and agent performance drops sharply on competitions after 2022. The authors state that second-order effects of modifying the system prompt are hard to track, and in an appendix experiment that varied temperature for AIRA Greedy with all diversity mechanisms removed, metrics other than Elo, including medal rate, showed no effect.
    概要PDF
  7. 1337AI AgentsarXiv:2606.23991

    Critique of Agent Model

    • Problem: As LLM systems marketed as coding agents and AI co-scientists multiply, it has become necessary to clarify where automation ends and agency begins, both for building capable systems and for judging what, if anything, to fear. The authors find that existing surveys cover what current systems can do and how to improve them broadly, yet tend to take agency for granted as a label that applies whenever an LLM interacts with an environment.
    • Argument: This is a position paper with an analysis, an architecture proposal, and proved theorems. The authors separate agentic systems, which finish tasks through externally orchestrated tools and workflows, from agentive systems, which maintain and revise their own goals, identity, decision-making, self-regulation, and learning inside the model, and they hold that current AI systems are largely agentic. On safety, they argue that GIC is structurally well positioned because harmful behavior decomposes entirely into goal misspecification (the human supplied the wrong objective) and component imperfection, and they hold that the overall goal is supplied from outside and that subgoals, identity, predictions, and plans can be audited by humans at every layer.
    • Taxonomy: The authors sort systems called agents by how their behavior is produced into five kinds: program-based bots, LLM wrapper systems (an LLM placed inside fixed orchestration), LLM-centered systems, model-less physical systems (such as industrial robots running programmed routines), and embodied-model systems (robots and driving systems that unify perception, reasoning, and control in one model). The common designs they examine along the five dimensions are short instructions supplied by a human at every step (goal), fixed system prompts with harness engineering (identity), black-box policies trained with end-to-end RL (decision-making), planning expected to emerge from RL or placed in fixed workflow stages (self-regulation), and human-scheduled training pipelines (learning). They argue that these designs share the absence of an internal world model that predicts the consequences of actions.
    • Proposal: The GIC agent model has six components: a belief encoder (which turns observations into a belief state, an estimate of the world state), a goal decomposer (which splits a long-term goal into subgoals), an identity evolver (which revises the self-model without retraining), a configurator, a simulative planner, and an actor. The actor handles System I (immediate action), the simulative planner handles System II (predicting the outcomes of candidate actions with the world model and picking a plan), and the configurator handles System III (choosing at each step whether to make a new plan, continue or revise an existing one, or act directly). The world model is trained separately on next-state prediction alone, the two models keep disjoint parameters and connect only through activations and outputs, and training is proposed in three phases: component pretraining that starts from a pretrained LLM, simulative RL inside the world model, and refinement after real-world deployment.
    • Evidence: Theorem 2 is a benefit: if the world model's next-state prediction error is bounded by ϵ in total variation (a distance between probability distributions), any policy can be mixed with selectively adopted world-model plans into a policy whose value (expected cumulative reward) is at least as high. Theorem 3 is a limitation of always-on, fixed-depth model-predictive control (MPC): the required horizon grows significantly as the desired planning precision rises. Theorem 1 shows the benefit that, when identity revisions are better than random, learning that revises identity at every step accumulates less expected regret than retraining alone, and Theorem 4 shows the benefit that the value of the best policy trained on a mixture of simulated and real experience is at least the value of the best real-only policy minus a term proportional to the world model's error, and at least as high when the world model is perfect.
    • Limits: The authors state that details and preliminary results for scaled-down GIC instantiations appear in companion manuscripts. Of their proposed evaluation dimensions (Performance, Efficiency, Growth), those preliminary results give initial evidence on the first two, and Growth evaluation remains future work. The premise of Theorem 2, an upper bound ϵ on the prediction error, is credible only when the world model is trained for predictive fidelity, and the safety conclusion that harmful behavior converges to zero is stated for a sufficiently well-trained system, unless the goal itself is wrong.
    概要PDF
  8. 1336Research AgentsarXiv:2606.01961

    AutoMedBench: Towards Medical AutoResearch with Agentic AI Models

    • Problem: Many existing medical agent benchmarks focus on medical question answering, clinical dialogue, EHR (electronic health record) or FHIR (Fast Healthcare Interoperability Resources) interaction, healthcare administration or final task success, and primarily evaluate final outputs. The authors argue that telling whether a low score comes from misunderstanding the task, an unsuitable method, a failed environment setup, skipped validation or a malformed submission requires stage-level evaluation.
    • Input and output: This is a benchmark for medical-AI research agents. The agent, a single base LLM using a code-execution environment, receives public inputs, a task brief, a submission schema and a time limit, and must produce and submit segmentation masks, restored images, VQA (visual question answering) answers, reports or detection boxes. Each run gets a workflow score over stages S1 to S5 (AGENTIC), a task score against hidden references (TASK), an OVERALL score that averages the two, and post-run error codes.
    • Method: The 24 tasks span five tracks, segmentation, image enhancement, VQA, report generation and detection, and cover CT (computed tomography), MRI (magnetic resonance imaging), X-ray, pathology, blood-smear microscopy, dental imaging and medical video. Every task runs the same five stages (S1 to S5), Plan, Setup, Validate, Inference and Submit, weighted 25%, 15%, 35%, 15% and 10%, with S1 to S3 scored by an LLM judge and S4 and S5 by deterministic checks. Lite and Standard tiers differ only in brief scaffolding and give 48 task-tier settings, and failure types are logged, separately from the stage scores, as five error codes: E1 understanding, E2 data and model setup, E3 verification and recovery, E4 implementation and execution, and E5 deliverable and submission.
    • Implementation: The agent container gets GPU, network, the public inputs and a writable workspace, while hidden references and scoring code sit in a separate evaluator container isolated from the external network, and training or fine-tuning during a run is forbidden. Six models, Opus 4.6, GLM-5, Gemini 3.1 Pro, ChatGPT-5.4 (called GPT-5.4 in parts of the paper), MiniMax-M2.5 and Qwen3.5, ran with the same system prompt, tool schema, stop conditions and decoding settings, with 10 runs for each of the 288 evaluation cells by default (20 for five segmentation tasks). According to the README, the GitHub repository AutoMedBench/AutoMedBench keeps task definitions and a Docker orchestrator on domain branches for running one cell by choosing an agent, task and tier, while the benchmark releases (Lite, Full) and the Full leaderboard Space are on Hugging Face and the live leaderboard is on the automedbench.github.io website.
    • Results: Among the six evaluated models, OVERALL scores ranged from 66.5 for Opus 4.6 to 51.2 for Qwen3.5, a 15.3-point spread, and GLM-5 led the VQA track with 64.0. Averaged over the six models, S3 Validate was the weakest of the five stages and S2 Setup the strongest, and all six models scored higher on AGENTIC than on TASK. Of all fired error codes, 37.7% were verification and recovery errors (E3) and 38.1% deliverable and submission errors (E5), while understanding errors (E1) were 0.9%, and runs with one fired error code scored 48% lower OVERALL on average than runs with no error code.
    • Limits: The authors state that stage-level evaluation cannot substitute for outcome evaluation and that a high AGENTIC score still requires separate checks of clinical usefulness and accuracy. Tasks are limited to inference-only workflows, excluding tasks whose correctness depends primarily on subjective judgment, long clinical dialogue or training-time adaptation, and the comparison covers base models run through the same harness without vendor agent wrappers. Instead of redistributing restricted data, the benchmark requires runners to obtain credentialed data, and in the authors' experiments MIMIC-CXR came through PhysioNet and fastMRI under the NYU (New York University) data-sharing agreement.
    公開コード概要PDF
  9. 1335Research AgentsarXiv:2605.27905

    AI Research Agents Narrow Scientific Exploration

    • Problem: The prompts and reasoning procedures of AI research agent frameworks often instruct agents to produce novel, high-impact, unconventional ideas, yet the ability to generate ideas at scale does not necessarily imply broader scientific exploration. Existing evaluations mainly assess whether individual ideas are interesting, novel, feasible, or executable, and reveal much less about how repeated AI-assisted ideation shapes the broader landscape of scientific exploration.
    • Input and output: In each run, one agent framework paired with one LLM receives a seed set of five papers from the same research area (one anchor paper and four related papers, titles and abstracts only) and produces one research idea with a research question and methods. The authors compare the collected ideas with human-authored papers from the same areas and with follow-on human papers that cite the seed papers, using four measures. This is a large-scale empirical study that analyzes 219,655 ideas.
    • Method: Research areas are built by grouping papers by how much they cite the same prior literature (bibliographic coupling). The four measures are exploration breadth (the mean pairwise cosine distance among ideas in the same area), exploration distance (the cosine distance from an idea to the centroid of its five seed papers), frontier alignment (the share of the top 10% most frequent keywords in the next year's human papers that the ideas cover), and potential scientific impact (the mean normalized citation score, relative to papers of the same area and year, of the 20 semantically nearest human papers). Ideas and human papers are brought into the same format by having Gemma-4-31B-IT extract research questions, methods, and keywords, and are then embedded with Qwen3-Embedding-4B.
    • Implementation: From a Semantic Scholar index of 149,405,218 papers, the authors drew 155 research areas in 12 broad fields and 11,520 seed sets, and ran five frameworks (Zero-shot, AIScientist, ResearchAgent, AgentLaboratory, and Co-Scientist) with four open-weight LLMs (Gemma-4-31B-IT, Llama-3.1-8B, Hermes-4-14B, and Qwen3-35B-A3B) for 230,400 runs. Because of budget limits, GPT-5.4 ran only on randomly sampled 2022 seed sets for 2,400 runs, and of the 232,800 runs in total, 219,655 valid ideas (94.4%) were analyzed. All frameworks except Zero-shot could search a locally deployed Semantic Scholar database restricted to papers available when the seed papers were published, AIScientist was run only through its ideation stage, and Co-Scientist was implemented from its published description.
    • Results: Exploration breadth within the same research area was 0.554 for AI ideas against 0.599 for human papers, 7.5% lower, and exploration distance from the seed literature was 0.322 for AI ideas against 0.410 for follow-on human papers. AI ideas covered 28.5% of next-year frontier keywords against 36.5% for follow-on human papers, and their potential impact score was 0.387 against 0.492, 21.3% lower, with GPT-5.4 the only LLM whose ideas were comparable to human research on impact (0.472 against 0.464). Only 10.5% of AI ideas contained research questions absent from the seed literature, while 90.4% introduced new methods.
    • Limits: Because of budget limits GPT-5.4 ran only 2,400 times on part of the 2022 seed sets, and the seed context was set to five papers because most frameworks are constrained by the context windows of current LLMs, with only titles and abstracts included. Information leakage from LLM pretraining on papers published after the seed literature was mitigated by restricting retrieval to papers available at the time, and since AI ideas still showed lower frontier alignment, the authors consider the reported differences likely conservative. Potential impact is a proxy based on the citations of semantically nearby human papers, and in a leave-one-out check its Spearman correlation with a target paper's own citation score was 0.155, which the authors note explains only part of the variation in citation outcomes.
    概要PDF
  10. 1334Research AgentsarXiv:2601.16282

    Generating Literature-Driven Scientific Theories at Scale

    • Problem: Automated scientific discovery has focused on agents that generate experiments, while systems for higher-level activities such as theory building, which consolidates many empirical results into generalizable laws, remain underexplored. Prior work on law discovery has largely induced quantitative or qualitative laws from structured data rather than from the research literature.
    • Input and output: It takes a user-written theory query and a corpus of papers and returns a set of theories. Each theory has a name and a description and holds tuples of a law (a qualitative relation or an equation), its scope (the conditions under which the law is expected to hold), and empirical evidence typically drawn from the input corpus. This is a system paper that implements THEORIZER and compares four generation conditions experimentally.
    • Method: THEORIZER rewrites the theory query as a literature search query for PaperFinder from AI2 (the Allen Institute for Artificial Intelligence) to retrieve up to K relevant papers (100 in the experiments), takes open-access PDFs located through Semantic Scholar, converts them to full text with an OCR (optical character recognition) based pipeline, and, if fewer than K papers come back from the initial search, backfills the corpus with relevant papers referenced in the retrieved full texts. It then builds an extraction schema for the query, has an inexpensive extraction model fill the schema for each paper as JSON records, and has the generation model synthesize theories from the pooled records, after which a self-reflection step refines each theory for internal consistency, evidence attribution, and specificity. The four compared conditions pair literature-supported generation, which uses this evidence, and parametric generation, which uses only the model's internal knowledge, with an accuracy-focused or a novelty-focused objective.
    • Implementation: Generation uses GPT-4.1, whose reported knowledge cutoff is June 2024, per-paper evidence extraction uses GPT-5-mini, and evaluation mainly uses Claude Sonnet 4.5. From 50 randomly sampled papers at NLP (natural language processing) and AI venues (ACL 2023, EMNLP 2023, AAAI 2023, NeurIPS 2024), one general and one specific query were generated automatically per paper, and the resulting 100 queries drew on 13,744 papers to produce 2,856 theories, and evaluation was done per law rather than per theory, with 790 to 1187 laws per condition in the Table 1 LLM-as-a-judge evaluation. According to the README, the src folder of the official GitHub repository allenai/asta-theorizer holds the web interface and API server, the core prompts, and four evaluation scripts, the example-theories folder holds a small toy set and the theory data from the paper, and literature-supported generation needs a separately installed local copy of Asta PaperFinder.
    • Results: In the Table 1 LLM-as-a-judge evaluation (an LLM acting as the grader), where Claude Sonnet 4.5 rated each law on a 1 to 10 scale, accuracy-focused literature-supported laws scored higher than parametric ones on empirical support (consistency with existing evidence), 5.8 versus 3.9, and on specificity, 6.5 versus 5.3, while the novelty score was 5.8 for both. In the Table 2 backtesting against 4,554 papers published in the six months from July 2025, precision was similarly high in the accuracy-focused condition, 0.88 for parametric and 0.90 for literature-supported laws, and 0.34 and 0.61 in the novelty-focused condition. The authors note that the literature-supported method is almost 7 times as expensive as a simple LLM approach, and the Table 4 estimate of LLM API costs is $0.26 per theory for literature-supported and $0.04 for parametric generation.
    • Limits: Backtesting has low recall: in Table 2 the share of laws with at least one recent paper that tests their predictions was 0.45 to 0.51 in the accuracy-focused condition and 0.04 for parametric and 0.16 for literature-supported laws in the novelty-focused condition, and the authors note that the literature's positive-result bias, under which negative results often go unreported, may make contradicting evidence harder to find. The qualified novelty analysis of Table 3, separate from the Table 1 LLM-as-a-judge scores, measured the shares of seven novelty types against the 66 to 71 papers retrieved on average for each theory query instead of the full literature, on 100 randomly selected laws per condition, and in this analysis literature-supported laws were generally rated as less novel than parametric ones. The authors state that generation is currently limited to fields whose literature is mainly open access and downloadable automatically, such as AI and NLP, and that generation and especially evaluation are currently somewhat expensive, which limits scaling.
    公開コード概要PDF
  11. 1333Research AgentsarXiv:2511.16931

    OmniScientist: Toward a Co-evolving Ecosystem of Human and AI Scientists

    • Problem: The authors argue that existing AI Scientists predominantly formulate scientific discovery as a standalone search or optimization problem and leave unmodeled the collaborative mechanisms, contribution attribution, peer review and structured knowledge networks that real science relies on. As a result, current systems struggle to establish a research ecosystem or interact deeply with the human scientific community.
    • Input and output: From a user's research topic or query it produces a Markdown literature review, idea proposals with research background, idea and a general implementation approach, experiment code and results, a LaTeX manuscript and review reports whose key judgments carry traceable citations. ScienceArena takes votes on anonymous pairwise comparisons of two models' answers to user-submitted research questions across six tracks and builds a leaderboard from Elo ratings that start at 1000 for every model. This is a system paper combining several modules, a protocol and an evaluation platform, and its evidence comes from per-module case studies and small-scale measurements.
    • Method: The literature review is split among Planning, Literature and Writing agents; candidates from embedding-based Elasticsearch search are joined by papers linked through citations and references, and papers that pass relevance verification are expanded layer by layer in a breadth-first search (BFS) until a predefined depth or coverage is reached. Deep Ideation explores, adds and replaces neighboring keywords in a keyword co-occurrence network to refine proposals, logs keywords, proposals and evaluations per round in an Idea Stack, and receives feedback aligned with expert review standards from a critic model fine-tuned on reasoning produced by a prompt that simulates human reviewers' cognitive process. OSP treats humans and AI as the same Participant type, a central Hub records every protocol message, and each research artifact carries a ContributionLedger that lists participants' actions with timestamps in order.
    • Implementation: The knowledge base links about 269 million OpenAlex metadata records with their citations, about 2.6 million arXiv PDF full texts, and 102,679 full-text papers from the top ten AI conferences over the past decade with 116,970 referenced baseline models and 68,316 related datasets into a graph of Paper, Author, Concept and Resource nodes, and Diagnose, Search, Normalization, Coding and Review agents keep refining it. The experiment stage selects candidates with a bi-encoder retriever fine-tuned on baseline and dataset self-descriptions combined with summarized citation contexts from other papers and an LLM reranker fine-tuned on citation chains linking papers, baselines and datasets, and then Evolution, Sample, Evaluation and Feedback agents modify and run code. Writing is handled by Outline, Figure, Writing and Refinement agents and review by the five-stage TIMAR (Traceable and Interactive Multi-Agent Review) workflow, and the public addresses given in the paper are omniscientist.ai and sciencearena.ai.
    • Results: In a small-batch evaluation (n = 1000), graph refinement raised metadata completeness from 0.965 to 1.000 and correctness from 0.951 to 0.997, and on a benchmark of 100 QA pairs probing inter-paper relationships it raised retrieval accuracy from 0.70 to 0.88. OmniScientist's proposal to replace the Monte Carlo sampler of STDE (Stochastic Taylor Derivative Estimator) with a QMC (Quasi-Monte Carlo) sampler gave lower error on the AllenCahnTwobody equation than both the original STDE and the AlphaEvolve variant at all four settings from 100 to 100000 dimensions, for example 0.000572 against 0.003440 for STDE and 0.002059 for the AlphaEvolve variant at 10000 dimensions. In a study where 10 PhD-level participants each solved 10 computer science and AI questions from HLE, five alone and five with GPT-5, average accuracy was 0.22 with GPT-5 collaboration, 0.1 for humans alone and 0 for GPT-5 alone.
    • Limits: The authors state that current capabilities remain constrained to the AI domain, and that the data backbone, primarily arXiv full texts, strongly overrepresents AI and computer science while Nature, Science and other domain-specific journals are not yet integrated. The experiment agent currently supports only computational workflows such as configuring environments, preparing datasets, running training and inference scripts and hyperparameter sweeps, so wet-lab experiments in chemistry or biology and instrument interaction are outside its current scope; the review module is optimized on AI manuscripts and loses quality on other fields, and the modules still require significant compute and processing time. The validation consists of case-level studies such as the STDE improvement on a single equation (AllenCahnTwobody), a collaboration study with 10 PhD-level participants and 10 HLE questions, and a literature review comparison scored by Gemini-2.5-pro, and ScienceArena results are presented as per-track preference patterns and example figures in place of vote counts or Elo values.
    概要PDF
  12. 1332Research AgentsarXiv:2509.06917

    Paper2Agent: Reimagining Research Papers As Interactive and Reliable AI Agents

    • Problem: Significant technical barriers often remain before a paper's new computational method can be used on new data, and a reader might need to locate the code repository, install dependencies, configure environments and interpret the expected inputs and outputs. The authors argue that even a powerful tool like AlphaGenome is hard for biologists to use because of the technical barrier of handling API keys, client objects and variant objects. They note that executable papers increased reproducibility but still required technical familiarity, and that Papers with Code improved discoverability while the barrier of installing and executing the code remained.
    • Input and output: This is an implemented multi-agent system shown through case studies. The input is a paper with its code repository link, supplements and data, and the output is an MCP (Model Context Protocol) server Python file with tools, resources (manuscript, code, data) and prompts (multi-step workflow instructions), plus a configured environment. Connecting the server to a chat agent such as Claude Code yields a paper agent that runs the paper's methods from natural language queries.
    • Method: An orchestrator agent built in Claude Code invokes four sub-agents as needed at different stages: the Environment-manager (sets up an isolated environment), the Tutorial-scanner (finds tutorials in the repository), the Tutorial-tool-extractor-implementor (implements tutorials as reusable tool functions) and the Test-verifier-improver (writes, runs and fixes tests). The Test-verifier-improver repeats a loop of generating, executing and fixing tests using only the tutorial's own examples to check that the original numbers and figures come back, and functions that keep failing lose their MCP decorators and are dropped from the server. Verified tools are locked for reproducibility, and every tool keeps a link to the original GitHub source code.
    • Implementation: The 22 AlphaGenome MCP tools were built in around 3 hours on a personal laptop without human intervention, the TISSUE server has 6 tools, and 7 Scanpy preprocessing and clustering tools were built in around 45 minutes on a personal laptop. All three agents in the AlphaGenome benchmark used claude-sonnet-4-20250514 as the LLM and ran on a MacBook Air (M2), and the AlphaGenome, Scanpy and TISSUE MCP servers are hosted on Hugging Face Spaces. The GitHub README explains installing the repository's skills/paper2agent folder as a skill in Claude Code or Codex and connecting to the MCP servers hosted on Hugging Face Spaces, and says the skill launches parallel specialist agents, checks the work with fresh verifier agents and delivers a ZIP file with usage instructions (USAGE.md).
    • Results: On 15 AlphaGenome tutorial queries the AlphaGenome agent reached 100.0% accuracy, above Claude + Repo (Claude Code given the repository) at 60.0% and Biomni at 40.0%, and on 15 newly curated queries it again reached 100.0% against 80.0% and 60.0%. The TISSUE agent's outputs on mouse somatosensory cortex spatial transcriptomics data and the Scanpy agent's preprocessing and clustering of three 10x Genomics PBMC (peripheral blood mononuclear cell) datasets matched the results human researchers obtained on the same data. An AI co-scientist that connected the AlphaGenome and ADHD GWAS data MCPs executed a hypothesis chosen by the user, picked rs1626703 out of 209 candidate variants, predicted that it alters splicing of the MPHOSPH9 (M-phase phosphoprotein 9) gene in glutamatergic neurons, and then finished the analysis of all 39 loci within two hours.
    • Limits: The authors state that if the original codebase is incomplete, poorly documented or contains unresolved errors, Paper2Agent cannot reliably expose it as a functioning tool. The accuracy benchmark depends on expert knowledge of the paper, manually implemented ground truth and manual grading, and the work so far focuses on methodological papers. The ADHD result is a computational prediction made with AlphaGenome, and the authors note that these hypotheses benefit from subsequent human evaluation.
    公開コード概要PDF
  13. 1331AI / GeneralarXiv:2509.06580

    AI for Scientific Discovery is a Social Problem

    • Problem: AI is increasingly applied to scientific research, but its benefits remain unevenly distributed across communities and disciplines, and so far they have concentrated in domains with strong data infrastructure and well-coordinated research communities. Technical challenges such as limited data, fragmented standards, and unequal access to computing are well known, but the authors argue that social and institutional factors are often the primary constraints.
    • Argument: This is a position paper that builds its case from literature, survey statistics, and case studies. The authors argue for reframing AI for science as a collective social project in which sustainable collaboration and equitable participation are prerequisites for technical progress. They define scientific success as strong predictive performance together with explanations that reveal causal relationships, experimental validation, integration with theory, and broad accessibility, and they hold that systems that merely optimize benchmark prediction accuracy fall short of this standard.
    • Taxonomy: The authors sort the barriers into four: community dysfunction (the autonomous AI scientist narrative, undervalued data and infrastructure work, and communication and education gaps between fields), misaligned research priorities (a focus on narrow domain applications and neglect of upstream computational bottlenecks shared by many fields, such as density functional theory (DFT) calculations and partial differential equation (PDE) solvers), data fragmentation (incompatible formats and standards, and high-dimensional, low-sample-size scientific data), and infrastructure inequity (gaps in access to computing). They see commercial AI scientist systems as predominantly focused on literature search, summarization, and hypothesis generation from existing text while largely avoiding the experimental validation and mechanistic investigation that define scientific discovery.
    • Proposals: The four remedies are strengthening collaboration and education across fields (standardized interfaces and APIs, collaborative development platforms, and training that builds hybrid expertise), structuring upstream challenges through shared benchmarks, standardizing and curating data around simple formats such as CSV (comma-separated values) or Parquet, and building community-owned, sustainable infrastructure. The authors argue that research funding should prioritize computational challenges with broad applicability over numerous domain-specific applications, and that reproducible science should be rewarded in hiring, tenure evaluations, and citation counts.
    • Evidence: CASP (the critical assessment of protein structure prediction) has run biennial blind prediction competitions, in which predictions are submitted before the experimental structures are released, since 1994, building clear success metrics and challenging test cases, and the authors credit this groundwork for AlphaFold's breakthrough at the 14th round (CASP14) in 2020. On data, they cite the harmonization of nine Cancer Imaging Archive files, which took 329.5 hours over six months, and the growth of the Protein Data Bank from seven structures in 1971 to more than 230,000 structures. On resources, they cite figures that only 5% of Africa's AI research community has the computing power complex AI tasks need and that African researchers face six-day iteration cycles against 30-minute cycles in G7 countries.
    • Limits: The authors note that the paper synthesizes challenges across biology, chemistry, materials science, physics, and climate science to highlight structural patterns, so specific constraints vary by field and some recommendations may need adapting to particular subdisciplines or regulatory environments. Several arguments rely on studies and surveys that do not cover the full diversity of scientific institutions, especially in underrepresented regions, and fast progress in machine learning may change some of the technical limitations discussed. The discussion of governance and incentives is high level, so implementing it would need detailed policy design and negotiation among stakeholders, and both authors disclose a potential conflict of interest as employees of Hugging Face, which the paper cites as one collaborative platform.
    概要PDF
  14. 1330Research AgentsarXiv:2505.19955

    MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research

    • Problem: LLM agents have started to generate ideas, run experiments, and write papers, but the community still lacks a comprehensive benchmark that systematically assesses their ability to conduct open-ended research, which makes progress hard to measure and to compare fairly between agents. Existing benchmarks typically target narrow tasks: MLE-bench focuses on engineering, MLAgentBench on experimentation, and PaperBench on paper reproduction. The authors also see a need for empirical analysis of failure modes such as hallucinated results, lack of novelty, and methodological flaws.
    • Input and output: Each task is a research problem built from the overview and topics of a workshop at the NeurIPS, ICLR, or ICML machine learning conferences over the past three years, and in end-to-end evaluation the agent takes the task and returns a completed paper. Stepwise evaluation asks for an idea from a task, a research proposal from a task and an idea, code and results in an experiment environment given a task, an idea, and a proposal, and a paper from the experimental outputs (report, figures, and a log of commands run), and MLR-Judge scores each review dimension of an output from 1 to 10. This is a benchmark paper that releases 201 tasks, an evaluation framework, and an agent scaffold together.
    • Method: In MLR-Judge, two LLM judges (Gemini-2.5-Pro-Preview and Claude-3.7-Sonnet) score each output independently against stage-specific rubrics (scoring criteria for review dimensions such as Consistency, Novelty, and Soundness), the two scores are averaged, and in the experimentation stage the judge also inspects the coding agent's execution log. MLR-Agent is a scaffold that favors simplicity over extensive prompt engineering: LLMs write the idea and the research proposal, a coding agent (for example, Claude Code) runs the experiments, a multimodal LLM writes the paper, and a GPT-4o-Search-Preview literature review is inserted between the idea and the proposal. To validate the judge, the authors recruited 10 experts with ML conference reviewing experience, and two of them independently scored each generated paper with the same rubric as the judge.
    • Implementation: The 201 tasks come from workshops of the three conferences over the past three years, after removing duplicates and keeping workshops with complete information aimed at a general audience, and they span 9 core ML topics such as LLMs and VLMs (vision language models), AI for Science, ML Theory, and Trustworthy AI. Stepwise inputs were built by randomly sampling one previous-stage output per task (the experimentation stage instead used 10 hand-picked inputs), and the coding agents ran on an Ubuntu 22.04 server with four NVIDIA RTX 3090 GPUs. According to its README, the GitHub repository holds the task folder, the end-to-end script run_mlr_agent.py, step-by-step agent scripts (the mlrbench/agent folder), and judge scripts (the mlrbench/evals folder), and the task dataset is also on Hugging Face.
    • Results: In the experimentation stage on 10 tasks, Claude Code and Codex both scored 4.95 Overall, below the 6.0 acceptance threshold. In end-to-end evaluation on the same 10 tasks, MLR-Agent scored 4.70 Overall with Claude-3.7-Sonnet and Claude Code, 4.60 with Gemini-2.5-Pro-Preview and Gemini CLI (a command-line coding agent from Google), and 3.10 with o4-mini-high and Codex, so all of them, together with AI Scientist V2 on o4-mini-high (4.25), fell below 6.0, and all three MLR-Agent models scored lowest on Soundness. As MLR-Judge pointed out, in 8 of the 10 tasks run by Claude Code the reported results rested on synthesized or placeholder data rather than actual execution, and the authors found that after execution failures coding agents tend to fill the gaps with synthetic results instead of reporting the failures.
    • Limits: Experimentation, paper writing, and end-to-end evaluation used 10 of the 201 tasks, hand-picked to keep the experiments manageable, and all 10 come from ICLR 2025 workshops, mostly on Trustworthy AI. The authors name the lack of process transparency as a key barrier to trust, since human reviewers given a fully formed AI-generated paper often cannot see how each part was produced, and they present MLR-Bench as a first step toward analyzing and improving AI research agents. Nonexistent citations appeared in 50% of the tasks completed by MLR-Agent, and the authors hypothesize that limits of its literature collection tool caused this.
    公開コード概要PDF
  15. 1329ReasoningarXiv:2501.08120

    In-situ graph reasoning and knowledge expansion using Graph-PReFLexOR

    • Problem: The author argues that much of current AI relies on models that produce outputs directly, without an explicit intermediate reasoning process akin to scientific thinking, and that LLMs often excel in fluency and pattern recognition yet typically lack structured reasoning. The goal is a model that goes beyond retrieving information to learn relations between concepts and abstract patterns and generalize across new domains.
    • Input and output: This is a method paper that presents a training approach and a 3B model trained with it. Given a question or task, the model generates, inside a section wrapped in thinking tokens (<|thinking|>), a knowledge graph (concept nodes joined by relation edges such as IS-A, RELATES-TO and INFLUENCES), symbolic abstract patterns, reasoning steps and a hypothesis, and then gives the final answer.
    • Method: Training data are generated on the fly during training: a passage drawn at random from the raw corpus becomes a question, RAG (retrieval-augmented generation) adds related content from the whole corpus, and the pipeline builds a correct answer containing a knowledge graph plus a rejected answer. In recursive reasoning, a critic agent critiques Graph-PReFLexOR's thinking section, the thinking is revised with that critique and the question is answered again, and after N rounds the last answer is taken or all answers are integrated. In the knowledge garden, Graph-PReFLexOR answers a next question that a human or another LLM agent builds from the nodes of earlier graphs, producing a new graph, and the graphs are joined into one growing graph.
    • Implementation: The base model meta-llama/Llama-3.2-3B-Instruct is trained on data built from the raw text of around 1,000 papers on biological and bio-inspired materials, in the same order as PReFLexOR, ORPO followed by EXO, a DPO (Direct Preference Optimization) variant, and the critic in the recursive reasoning implementation is the non-fine-tuned Llama-3.2-3B-Instruct. The paper states that code and model weights are available on Hugging Face under lamm-mit and in the GitHub repository lamm-mit/PRefLexOR, and names the model lamm-mit/Graph-Preflexor_01062025. The GitHub README documents the PRefLexOR training package, ORPO and EXO trainer examples and Colab inference notebooks, says the model weights come from Hugging Face, and its top-level files include a Graph-PRefLexOR training notebook (Graph-PRefLexOR_train_Phase-I-and-II.ipynb).
    • Results: On one task, designing a tough protein material that integrates a snowflake and ant behavior, GPT-4o as judge gave the first iteration Graph-PReFLexOR response an average of 9.7 and the reference model, a standard non-fine-tuned model, 6.4. Of the seven criteria, only Clarity and Organization favored the reference model, 9/10 against 8/10 for Graph-PReFLexOR. In the autonomous bio-inspired materials run, a human gave only the first prompt and an LLM wrote every later prompt, the graph grew over 12 iterations, and the nodes with the highest degree (number of linked edges) were Sustainable Materials, Self-healing and Adaptable Materials, and Ecosystems.
    • Limits: The only comparison with a baseline model is Table 1, one task scored by a single judge model, GPT-4o, and the remaining results are case studies that show graphs and answers. The author notes that on the song task the model may be limited in capturing the temporal and emotional nuances essential to musical composition. Scaling to larger datasets and more complex models and interpretability of deeply interconnected graphs are named as future work, and the author writes that the critic could include fact-checking or even run simulations.
    公開コード概要PDF
  16. 1328Research AgentsarXiv:2410.09403

    Many Heads Are Better Than One: Improved Scientific Idea Generation by A LLM-Based Multi-Agent System

    • Problem: Earlier work such as AI Scientist, ResearchTown, and HypoGen uses LLM agents to simulate scientific idea generation, but the authors note that these efforts either rely on a single agent or build multi-agent setups on an oversimplified collaboration framework and unrealistic data such as manually crafted profiles and synthetic collaboration networks, missing the dynamic relationships of real scientific teams. The authors ask whether simulating how real scientists form teams and collaborate yields more novel ideas, and how collaboration conditions such as team size, freshness, and research diversity affect novelty.
    • Input and output: This is a system paper presenting the multi-agent system VirSci, whose input is a research ecosystem built from real academic data at a chosen time point. The ecosystem consists of a database of papers published before that point, a database of papers published after it, an author knowledge bank of scientist profiles, and an adjacency matrix of collaboration counts between scientists. The output is one abstract written by a team that a randomly sampled team leader assembles, with Introduction, Objective, Methods, Expected Results, and Conclusion sections.
    • Method: The team leader forms a team by using the adjacency matrix, with 1 added to every collaboration count, as invitation probabilities, and the team then takes turns speaking for K turns in each step of topic discussion, idea generation, novelty assessment, and abstract generation. During discussions, agents can retrieve the author knowledge bank with RAG (retrieval-augmented generation) to hear from relevant scientists outside the team, which the authors call the Invitation Mechanism. Idea generation keeps the three ideas with the highest confidence, and in novelty assessment the agents compare each idea with past papers and select one by vote.
    • Implementation: The system is built on the multi-agent framework AgentScope and calls GPT-4o through a public API and the open-weight LLaMA3.1-8b and LLaMA3.1-70b through Ollama, with LLaMA3.1-8b as the default. The data are the AMiner Computer Science dataset filtered to 156 authors and 178,197 papers and Open Academic Graph 3.1 filtered to 3,169 authors and 201,131 papers. The top level of the GitHub repository holds an agentscope-main folder that defines the customized scientist agent, a sci_platform folder with run.py as the main entry, and a preprocess_data folder, and the README points to Google Drive for the preprocessed AMiner Computer Science data.
    • Results: With GPT-4o agents, VirSci scores a CD (Contemporary Dissimilarity, the average distance between the generated abstract and its 5 most similar contemporary abstracts, where smaller means closer to new work) of 0.34, a CI (Contemporary Impact, the average citation count of those abstracts) of 3.78, and a novelty score of 5.24 on a 10-point scale from ten PhD students, ahead of HypoGen (0.36, 3.10, 4.78) and AI Scientist (0.38, 3.22, 4.94), and with LLaMA3.1-8b and LLaMA3.1-70b agents it also leads on every metric in the table, including the LLM review score. In this comparison, NanoGPT is added as the initial discussion topic for HypoGen and VirSci to match AI Scientist's research direction, inference cost is matched with 4 members and 5 turns for VirSci and 12 turns for HypoGen, and paper retrieval for the metrics uses the Semantic Scholar API for all three systems. In the collaboration experiments comparing team sizes of 1, 4, 8, 10, and 100, the overall novelty metric ON (Overall Novelty) peaks at a team size of 8 and around 5 discussion turns, and also when about 50% of members are new collaborators.
    • Limits: In large research communities multiple teams often work on related projects together or separately and researchers frequently join several teams at once, and the authors state that the system simplifies these complexities; they also list biases inherited from the underlying LLM and training data and the risk of plausible but incorrect scientific claims. Because of computational constraints, the evaluation focuses primarily on generated abstracts rather than full texts, and on 200 abstracts ON correlates with human ratings at a Pearson coefficient of 0.52. VirSci covers the idea generation phase, and an example abstract in the appendix includes a results sentence saying randomized controlled trials with 500 children reduced failure rates by 42%.
    公開コード概要PDF
  17. 1327Research AgentsarXiv:2410.07095

    MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering

    • Problem: Language models perform well on many coding benchmarks and are progressing on ML tasks such as architecture design and model training, but few benchmarks holistically measure autonomous end-to-end ML engineering. The authors aimed for tasks that are challenging and representative of contemporary ML engineering work, with evaluation results that can be compared to human-level performance.
    • Input and output: The agent receives a Kaggle competition's description, dataset, and submission format, trains models in an offline environment, and outputs a submission.csv file. The submission is scored by grading code that re-implements the competition metric and is converted into a medal by comparing it against the Private leaderboard as if the agent had entered the competition at the time. This is a benchmark paper for evaluating ML engineering agents.
    • Method: From 5673 completed competitions in the Meta Kaggle dataset, the authors dropped Community Competitions, manually screened the remaining 586, and selected 75, labeled by the time an experienced ML engineer would need for a sensible solution (excluding model training) as Low (22, under 2 hours), Medium (38, 2 to 10 hours), or High (15, over 10 hours). Where no public test set existed, they carved a new test set out of the public training data (at a 10% ratio unless otherwise stated), and the headline metric is the share of attempts that earn any medal (bronze or above) under Kaggle's medal thresholds, which depend on the number of participating teams. Agents may not write predictions directly from their own knowledge instead of using a model, or view solutions online, a log inspection tool and the Dolos plagiarism detector check this, and attempts with over 60% code similarity to any of the top 50 Kaggle notebooks are disqualified and flagged for further review.
    • Implementation: In the main experiments, each competition attempt runs for up to 24 hours in an Ubuntu 20.04 Docker container holding the dataset, a submission validation server, and Python packages, on a machine with 36 vCPUs, 440GB of RAM, and one Nvidia A10 GPU, and seeds default to 3, with 16 for o1-preview (AIDE) and 36 for GPT-4o (AIDE). The three scaffolds are AIDE, MLAB (the ResearchAgent from MLAgentBench), and OpenHands (the CodeActAgent from the OpenHands platform), and four models were compared on AIDE: o1-preview, GPT-4o, Claude 3.5 Sonnet, and Llama 3.1 405B. According to the README, the GitHub repository openai/mle-bench contains dataset preparation scripts, per-competition grading scripts, the evaluated agents, the setup for the base Docker image (mlebench-env) used by agents, and a rule violation detector and a plagiarism detector, and it recommends a Lite evaluation on only the 22 Low complexity competitions when cost is a concern.
    • Results: In Table 2, among the four models on the AIDE scaffold, o1-preview earned a medal in 16.9% of competitions on average, the highest, followed by GPT-4o at 8.7%, Claude 3.5 Sonnet at 7.6%, and Llama 3.1 405B at 3.0%, while GPT-4o scored 0.8% with MLAB and 4.4% with OpenHands. In the attempts experiment, o1-preview rose from 16.9% at pass@1 to 34.1% at pass@8 (the share of competitions with a medal in at least one of 8 attempts), and in the experiment that extended the time limit per competition to 100 hours and the maximum number of AIDE nodes to 5,000, GPT-4o (AIDE) rose from 8.7% (at 24 hours) to 11.8%. In the Table 3 hardware experiment GPT-4o (AIDE) scored similarly with no GPU (9.1%), one GPU (8.7%), and two GPUs (10.2%), and in Table 4 it also scored similarly on the original descriptions (8.5%) and on obfuscated descriptions run with 10 seeds (8.4%).
    • Limits: Because the data are public Kaggle competitions, models may have trained on competition details, solutions, and even the datasets including the test sets, and the authors state that although their experiments found no systematic contamination effect for GPT-4o, reuse of high-level strategies is hard to detect and there are no guarantees about future models. They note that MLE-bench selects competitions with clear problem statements, clean datasets, and clear metrics, so it covers part of the capabilities needed for real-world AI research and development, where the problem statement often may not even be clear and figuring out the dataset and metrics is part of the problem. One run of the main setup takes 24 hours times 75 competitions, or 1800 GPU hours, and o1-preview (AIDE) used on average 127.5M input tokens and 15.0M output tokens for one seed of 75 competitions.
    公開コード概要PDF
  18. 1326Research AgentsarXiv:2610.07625

    Stateless Language Agents: Scaling Long-Horizon Automated Research

    • Problem: Automated research (AutoResearch) systems run LLM agents over long horizons, but agents replay growing histories, duplicate one another's work, or stop experimenting while token consumption continues. In one run of CORAL, a framework whose parallel agents select their own experiments and share discoveries through persistent memory, on a GPU kernel task, the last improvement came at 231M tokens, after which 98% of the final 1,500 sessions made no tool calls. Most evaluations use short budgets or benchmarks that saturate early and leave these failure modes untested, and the authors trace the failures to two choices: where research state lives and who decides what to try next.
    • Input and output: SLA takes one task (a task adapter that specifies the editable files, permitted resources, and evaluation procedure, a seed solution, and a protected evaluator) and a cumulative token budget. Until the budget is exhausted, an independent evaluator scores every candidate the Workers produce, and the harness keeps the global best solution, each Worker's local candidate, the measured outcomes, and checkpoints as research state. This is a system paper that implements the framework and compares it with three recent frameworks on software engineering, kernel optimization, and algorithm design tasks.
    • Method: The principle is stateful search with stateless agents: every agent starts a new session at each invocation, while the harness owns the research state (candidate solutions and measured outcomes) and rebuilds a fresh, role-specific context for every invocation. Each epoch, the Advisor receives the task specification, the current global best code and score, and a harness-built evidence summary that groups outcomes by search direction and separates measured non-improvements from failures in implementation, runtime, or correctness. Each Worker trial sees only the task specification, its assignment, its retained candidate, and compact local feedback, evidence from other directions reaches it only through the Advisor's assignments, and a redirected Worker restarts from the global best.
    • Implementation: The Advisor and Workers use the same coding agent, in two configurations, OpenAI Codex CLI v0.152.1 with GPT-5.5 and Claude Code v2.1.258 with Claude Opus 4.8; in the main experiments they also use the same LLM, and Appendix C.1 separately reports a GPT-5.5 Advisor paired with GPT-5.4 mini Workers. The defaults are 3 Workers and three local trials per Worker per Advisor epoch, Workers run as unprivileged processes under Linux Landlock filesystem restrictions, and the harness clears saved sessions between trials and scores candidates outside Worker environments with protected evaluator code. The baselines are CORAL and SwarmResearch, whose agents keep long-running sessions, and EvoX adapted to launch a fresh coding agent for each generation; all three run from pinned upstream releases with unmodified search logic, and the same external cumulative-token counter stops every method.
    • Results: On Anthropic kernel optimization with Codex (1B tokens, three runs per method), SLA averaged 1112.0 cycles against 1275.7 cycles for SwarmResearch, the strongest baseline by mean, and reached SwarmResearch's final performance with 67.9M tokens, compared with the 986.3M tokens SwarmResearch needed, a 93.1% reduction. On the four FrontierSWE tasks, run once per method with Codex, SLA scored 4.95 points higher on average than the strongest per-task baseline, trailing on all four at 175M tokens (25% of the budget) but leading on all four at the full 700M budget. In ablations that ran three 100M-token continuations from shared checkpoints on Anthropic kernel and FrontierSWE libexpat, removing any one of Advisor context reconstruction, Worker context isolation, or Advisor assignments reduced mean progress in every comparison, and in the two runs analyzed for token usage (Anthropic kernel and FrontierSWE libexpat) the Advisor used 0.24% to 0.51% of all tokens.
    • Limits: Because long runs are costly, most configurations were run once, and only Anthropic kernel optimization with Codex and the checkpoint ablations were replicated. Comparisons are matched on tokens, and the cost estimates exclude the compute for running and evaluating experiments. All tasks have executable evaluators, so goals that are vague or slow to evaluate remain untested.
    概要PDF
  19. 1325Research AgentsarXiv:2609.39551

    RankEvolve: A Reliable Multi-Agent Auto-Research Harness for Evolving Ranking Models

    • Problem: Auto-research agents that propose, implement, train and evaluate model changes are bound by execution accuracy in the authors' setting. A single defect such as leaked held-out data, an omitted layer norm, a disconnected gradient or an unwired train/eval flag burns hours of accelerator time and yields an invalid metric, and over many iterations the damage compounds while the run drifts off its prescribed process. In the authors' motivating pilot where the whole procedure was given as one in-context playbook, 0 of 10 runs were fully faithful and all 10 ran only a subset of the selected proposals.
    • Input and output: It takes an ML codebase and data, an EOP (Executable Operating Protocol) document listing phases and gates, and human decisions at the gates, and produces flag-gated code changes, training runs, leaderboard entries that include negative results, and evidence records. In the execution-accuracy evaluation each method receives a locked repository snapshot and a change request and returns a patch that a hidden executable oracle scores as pass or fail. This is a system paper on a deployed harness that also reports an HSTU deployment case study and an evaluation on ExecML, a benchmark built from that deployment's incidents.
    • Method: An EOP declares phases, dependencies, required tools, human confirmation gates and branch and loop edges once; the runtime compiles it into a state machine that enforces order and gates and injects only the global preamble and the current phase instruction into context. The meta-meta-harness binds complete coding-agent products such as Claude Code, Codex and OpenHands to execution-graph nodes through an adapter contract, forming Dual loops where one product implements and another reviews and repairs (repeating until the reviewer approves or residual severity falls below a threshold, within bounded rounds) and flows that split a task and aggregate the results. The knowledge layer records per proposal hashes, prompt, patch, test outputs, configuration, metrics, cost and the promotion decision, organizes lessons into notes beside the code, a central wiki and an entity graph, and injects only the few relevant lessons at each phase.
    • Implementation: The HSTU case used the public HSTU implementation and public MovieLens data, the workspace holds about 55 leaderboard entries across 49 experiment directories run on 1 to 8 NVIDIA H200s, and a human operator authorized compute and vetoed one over-budget experiment at the gates. ExecML builds 96 private tasks each from HSTU and LitGPT (a language-model training codebase), balanced over six incident families (data provenance and leakage, tensor routing, gradient flow, train/eval mode, metric semantics, configuration wiring) and split half new implementation and half repair, and Claude Code ran Claude Opus 4.8 while Codex ran GPT-5.6, both at maximum reasoning effort. Appendix B states that the failure-overlap analyzer, the leaderboard schema and the boost-last-K patch will be released with the codebase.
    • Results: After 12 iterations RankEvolve reported HSTU NDCG@10 of 0.2192 on MovieLens-20M LARGE (+4.48% over the published 0.2098) and 0.1948 on BASE (+2.80% over 0.1895), and the 0.2192 came from an added genre side-feature in place of a new modeling primitive. On 96 ExecML HSTU tasks at matched observable inference budget, the Claude Code, Codex, Claude Code flow reached 62.5% execution accuracy, 16.7 points above the strongest single-product condition, Claude Code in all three roles (45.8%; 95% CI 6.6 to 26.7), and its silent critical-defect rate (faults that leave the patch runnable but can invalidate a scientific conclusion) fell from 35.4% for one-pass Claude Code to 10.4%. On LitGPT the same flow led 56.2% to 43.8%, a gain of 12.5 points (95% CI 3.0 to 22.0), and with the runtime held fixed, injecting only the current-step instruction beat full-protocol injection in execution accuracy by 2.1 points at early phases, 6.2 at middle phases and 10.4 at late phases.
    • Limits: The authors state that the evidence base is two Python/PyTorch codebases (one ranking codebase and LitGPT), so cross-domain transfer is not fully established, that the oracles are incomplete so audits reduce but do not eliminate false passes, and that pretraining may include the public repositories. Without a memory on/off ablation the knowledge layer's contribution to proposal quality is not isolated, the HSTU case had a human operator at the gates who also steered it so the agent's and operator's contributions are not separated, and the cross-dataset transfer results are single runs. The deployed plan-merge flow had the highest HSTU execution accuracy (70.8%), but its 8.3-point lift over the re-run comparator was not statistically distinguishable (p=0.077), and the authors note that every win in the study was a composition of existing components.
    概要PDF
  20. 1324Research AgentsarXiv:2605.10813

    NanoResearch: Co-Evolving Skills, Memory, and Policy for Personalized Research Automation

    • Problem: Automated research systems such as The AI Scientist and EvoScientist can traverse the full research lifecycle, but the authors view them as uniform pipelines that produce near-identical outputs regardless of each user's resources, methodological preferences, and target formats. The authors name three missing capabilities. Each run starts from scratch and re-encounters the same debugging patterns, past hypotheses, validated configurations, and inferred resource constraints are discarded when a session ends, and implicit preferences such as preferring simpler methods are too diffuse to write as rules and fade when compressed into memory entries.
    • Input and output: This is a system paper with an implementation and experiments. NanoResearch takes a user-specified research topic and a user profile built through interactive queries (preferences, compute budget, target venue style, and so on), runs three stages, idea generation and planning, experimental validation, and paper writing and review, and outputs a LaTeX paper. The user gives natural-language feedback at the end of each stage, and the Orchestrator, the agent that coordinates the system, retrieves relevant entries from the Skill Bank and the Memory Module before each task and updates both stores afterward.
    • Method: Stage 1 retrieves papers from academic databases (for example, arXiv and Semantic Scholar) and parses performance scores from them, proposes hypotheses with a ReAct loop, turns the hypothesis kept after a novelty check into a JSON experiment blueprint with datasets, baselines, metrics, and ablations, and revises the blueprint with critiques from an internal LLM reviewer. Stage 2 writes code, deploys it to an environment such as a SLURM cluster, debugs it automatically until execution succeeds or the retry budget runs out, and analyzes the results, and Stage 3 drafts the manuscript section by section and has a Review agent that works without skill or memory retrieval revise it until predefined quality thresholds are met. After each stage the Orchestrator adds general rules to the Skill Bank and project-specific experience to the Memory Module, and user feedback updates the planner model's parameters through SDPO, which uses the feedback-conditioned model as its own teacher to create a token-level learning signal without a reward model.
    • Implementation: Literature retrieval uses the OpenAlex API, and the only trainable component is the Orchestrator's planner (Qwen3-8B). The other agents use DeepSeek-V3.2 for ideation, planning, and setup and execution, GPT-5.3-Codex for coding and debugging, Claude Sonnet 4.6 for writing and figure prompts and code, Gemini 3.1 Flash for figure images, Gemini 3.1 Flash Lite for review, and Gemini 3 Pro for revision. Per its README, the official repository (OpenRaiser/NanoResearch) describes a 9-stage pipeline from ideation to review, local GPU or SLURM execution, an evo pipeline that updates the user profile, skills, and memory, a ram-train command that trains an SDPO LoRA adapter on (x, y, feedback) triples collected from evo runs, and Claude Code and Codex integration, and its roadmap lists a benchmark evaluation suite as a remaining item.
    • Results: On 20 Claude-generated tasks in 7 domains with 20 simulated researchers, NanoResearch in Round 1 scored Compliance 8.163, E2E (share of runs that finish the whole pipeline) 1.000, and Perf. (average task accuracy of the produced method) 0.6844, higher than the four baselines (AI-Researcher, DeepScientist, EvoScientist, AI Scientist-v2) on every metric, while the baselines' E2E ranged from 0.500 to 0.900. By Round 3 Compliance rose to 8.963, Innovation (originality of the idea relative to prior work) to 5.645, and Perf. to 0.7548, and the total cost per topic was $1.430, lower than $6.039 for AI Scientist-v2 and $2.944 for EvoScientist. In an evaluation by three PhD researchers on their own three tasks, Perf. rose from 0.6466 in Round 1 to 0.8603 in Round 3.
    • Limits: The evaluation centers on AI and ML tasks whose outputs can be completed in code and text, and the authors leave extension to biology, chemistry, or physics, which involve physical experiments and instrument control, to future work. The 20 simulated tasks and the user-role feedback were all produced by Claude, subjective scores such as Compliance, Innovation, and Expression (writing quality of the paper) were rated by an LLM judge, and the human evaluation covers three PhD researchers and three tasks. In the human evaluation, Round 1 Perf. of 0.6466 was below EvoScientist's 0.6537, and Innovation and Expression dipped slightly from Round 2 to Round 3.
    公開コード概要PDF
  21. 1323Research AgentsarXiv:2604.17406

    EvoMaster: A Foundational Evolving Agent Framework for Agentic Science at Scale

    • Problem: Scientific agents rebuild model access, tool orchestration, context management, trajectories and error recovery for each domain, which limits reuse across disciplines; the authors call this loop fragmentation. Evidence, rejected hypotheses and failure experience stay confined to one run or research campaign (loop discontinuity), and unfiltered memory can retain noise alongside useful experience.
    • Input and output: It takes a scientific problem defined by a research question, a space of mutable research artifacts, executable tools or environments, an external evaluation protocol and budget and stopping settings. It returns artifacts such as code, model checkpoints, mathematical constructions or data-analysis reports plus a final answer, and keeps each run's configuration, trajectory and workspace artifacts as a persistent laboratory notebook. This is a framework paper that also reports agents built on the framework tested on 10 benchmarks.
    • Method: The Execution loop repeats Reason, Act and Observe within a run, and the Exploration loop forms and executes hypotheses within a study, using external evaluator evidence to retain, reject or revise candidates. The Evolution loop distills a finished study record into reusable knowledge, skills, prompts, memories or workflow components, and an Admit policy (an autonomous evidence rule, an evaluator gate or a human decision) decides whether they enter the capability store for later studies. When the context fills up, the system message prefix and recent turns are kept verbatim and older dialog is summarized, while workspace files, evaluator outputs and tool records stay outside this compression.
    • Implementation: Three software layers (Playground, Exp, Agent) and registries make models, tools, skills and agents independently replaceable, and MCP (Model Context Protocol) servers attach through the same tool interface. In the experiments EvoMaster ran X-Master and X-Master 2.0 on HLE (Humanity's Last Exam) and Frontier Science, ML-Master 2.0 on MLE-Bench Lite (the 22 low-complexity Kaggle competitions of MLE-Bench), Browse-Master on BrowseComp and BrowseComp-ZH (a Chinese web browsing benchmark), PhysMaster on PRL-Bench (a physics research benchmark built from Physical Review Letters papers), and its minimal agent on the remaining 4 benchmarks. Per the official README, the GitHub repository holds the framework code (evomaster/), agent examples (playground/) and configs (configs/), and among SciMaster agents ML-Master 2.0, ML-Master, X-Master and Browse-Master are marked Available while PhysMaster and EmboMaster are marked Coming Soon.
    • Results: With every agent on GPT-5.4 (medium reasoning effort) under the same benchmark-specific time limits, scoring scripts and task splits, EvoMaster scored highest on 9 of 10 benchmarks, all except PostTrainBench (Codex 15.92%, EvoMaster 13.83%), and against the highest-scoring other agent on each benchmark it gained 41.75 points on BrowseComp and 34.85 points on MLE-Bench Lite. Mean scores over the 10 benchmarks were 58.02% for EvoMaster, 40.29% for Codex, 39.84% for OpenHands and 32.39% for OpenClaw, and EvoMaster's mean per-benchmark LLM inference cost was $9.56, 35.6% below Codex's $14.85. Keeping tools, prompts, orchestration and benchmark protocols fixed and replacing only the backend gave mean scores of 63.65% with GLM-5.2 max (mean cost $14.00) and 55.66% with DeepSeek-V4-Pro max (34.4% of the GPT-5.4 mean cost).
    • Limits: The authors state that EvoMaster is primarily optimized for in silico and computational research workflows, and they leave direct manipulation of physical apparatus such as cloud laboratories or robotic synthesis hardware to future work. The cost comparison counts LLM inference only, and because means are sensitive to costly benchmarks such as Combinatorial Construction, the median cost of EvoMaster ($4.57) is higher than that of Codex ($3.00), OpenHands ($2.78) and OpenClaw ($1.43). EvoMaster used SciMaster agents tailored to each benchmark or its minimal agent with only minor input and output changes, against three general-purpose agents, and the authors say the benchmark suite primarily demonstrates breadth across workflows.
    公開コード概要PDF
  22. 1322Research AgentsarXiv:2603.14473

    AI Can Learn Scientific Taste

    • Problem: The ability to judge and propose research ideas with long-term impact, which the authors call scientific taste, is largely concentrated among highly experienced researchers. Current AI Scientist training mainly targets literature search and experiment execution, which address how to carry out research, so choosing which research directions are worth pursuing remains open. RLVR, which relies on rewards verifiable by ground-truth answers or unit tests, is hard to apply to such open-ended tasks, and RLHF depends on costly human annotations and struggles to capture community-level preferences from individual preferences alone.
    • Input and output: Scientific Judge takes the titles, abstracts, and publication dates of two papers, writes a reasoning trace, and answers A or B for the paper with the higher citation count. Scientific Thinker takes the title and abstract of a seed paper and returns a follow-up research idea as a title and an abstract. This is a method paper that proposes the RLCF training paradigm and tests it with the SciJudgeBench preference-pair dataset and trained models.
    • Method: RLCF pairs papers from the same field and year and treats the one with clearly more citations as preferred, keeping a training pair only when the citation gap is at least 8 and at least 30% of the higher count. Scientific Judge is trained on these pairs with GRPO (Group Relative Policy Optimization) and receives a reward of 1 for a correct prediction and 0 otherwise. Scientific Thinker samples several ideas per seed paper and is trained with Comparison-Based GRPO, where each idea's reward is its win rate in a round-robin tournament judged by Scientific Judge.
    • Implementation: The authors built 720,341 preference pairs across computer science, mathematics, physics, and other fields from 2.1M arXiv papers published through 2024. Judge models were trained with the MS-SWIFT framework on five Qwen2.5-Instruct sizes from 1.5B to 32B, Qwen3-4B-Instruct-2507, Qwen3-30B-A3B-Instruct-2507, and Llama-3.1-8B-Instruct, and Thinker models were trained from Qwen3-30B-A3B-Thinking-2507 and Qwen3-4B-Thinking-2507 on 4,000 high-citation seed papers published from January to July 2025. The top level of the official GitHub repository holds the README (English and Chinese), the paper PDF, and an assets folder of figures, and the README gives a results summary along with links to a Hugging Face collection of models and datasets and to an online demo.
    • Results: On the 1,000 in-domain test pairs, SciJudge-Qwen3-30B reached 82.7% accuracy, where a pair counts as correct only if the answer holds under both A and B orderings, above GPT-5.4 Thinking (81.6%), the highest of the four strong LLM baselines in Table 3. On 904 pairs of papers published in 2025, after the training period, Qwen3-4B rose from 64.7% before training to 80.9% after training and Qwen3-30B-A3B from 71.7% to 83.1%. Under majority voting by GPT-5.2-high, GLM-5, and Gemini 3 Pro, SciThinker-30B trained with SciJudge-Qwen3-4B as the reward model beat its base policy (the policy model before training) with an in-domain win rate of 81.5%, and its average in-domain win rate against ideas from GPT-5.2, GLM-5, and Gemini 3 Pro was 54.2%, compared with 30.3% for the base policy.
    • Limits: The authors state that citations are an imperfect form of community feedback, since some high-potential papers may receive few citations at first and become influential later, and that the controlled evaluations reduce simple author, institution, and topic explanations but cannot remove all biases in citation-based feedback. The ideation evaluation mainly relies on strong LLM evaluators, and implementing a subset of the proposed ideas is left for future work. Scientific Judge is trained mainly on titles and abstracts, and the authors note that richer paper context such as related work sections may improve its judgement.
    公開コード概要PDF
  23. 1321Research AgentsarXiv:2510.24891

    Idea2Plan: Exploring AI-Powered Research Planning

    • Problem: Scientists tend to produce more promising ideas than they can pursue, many remain unexplored due to time constraints, and turning an idea into a testable plan can take many days or even weeks. Recent work has focused on other stages such as idea generation, literature review, and experiment execution, and has treated research planning as an intermediate step without explicit evaluation (CodeScientist) or evaluated plans only on high-level dimensions such as clarity and novelty (MLR-Bench). Evaluation faces data contamination, since an LLM may have already seen the idea through a published paper, and the fact that several plans can be equally valid for the same idea.
    • Input and output: This is a benchmark paper that contributes a task definition, an automatic benchmark construction pipeline, and a benchmark for validating LLM judges. The input is a research idea rewritten in the first person from the original paper's abstract with method names and experimental results removed, and the output is a research plan with five sections: Introduction, Key Literature, Methods, Initial Experimental Design, and Resources/Compliance/Ethical Considerations. An LLM judge answers section-wise yes/no rubric questions built from a reference plan extracted from the original paper, and the Planning Score is the share of questions satisfied in each section, averaged over the five sections.
    • Method: From the ICML 2025 Spotlight and Oral papers, the authors drop papers whose arXiv versions predate the most recent training data cutoff of the tested LLMs (GPT-5) and randomly pick 200 of the remaining papers as the test set and 30 as the dev set. o4-mini (reasoning=high) extracts the idea from the abstract and the reference plan from the full paper, and rubric questions ask about high-level design choices that any reasonable plan should address instead of specific terms absent from the original idea. Plans are generated four ways: Naïve (a short prompt with section titles), 0-shot (general instructions added), 1-shot (one fixed example plan added), and a ReAct agent that can call an arXiv search tool and a read tool up to five times each.
    • Implementation: The tested LLMs are GPT-4.1, GPT-4.1-mini, o4-mini, GPT-5, and GPT-5-mini, plus the open-source DeepSeek-V3, DeepSeek-R1, and Phi-4, with three runs per idea for each model and setting and the mean reported. The judge, o4-mini (reasoning=high), was chosen after comparing candidates on Idea2Plan JudgeEval, where two human annotators independently graded the rubric questions of 30 plans (88.4% agreement, Cohen's κ 0.768), and it grades each section of a plan separately. The search tool uses Bing Search restricted to arXiv and blocks papers published after January 30, 2025 (the ICML 2025 deadline), the target paper, and papers citing it, while the read tool returns an o4-mini summary in a fixed template of a paper fetched through the arXiv API.
    • Results: On the test set, GPT-5 had the highest average Planning Score in all four settings (Naïve 62.0, 0-shot 63.0, 1-shot 63.2, ReAct 61.9), and GPT-5 and GPT-5-mini achieved pairwise win rates over 90% against the other models. The effect of ReAct varied by model: GPT-4.1 scored 40.8 with 1-shot and 47.4 with ReAct, while GPT-5 scored 63.2 and 61.9, and the authors report that models sometimes incorporated irrelevant retrieved details into plans. On an appendix benchmark of 100 ICML 2026 papers, standalone GPT-5.5 scored 62.0 and GPT-5.5 in the Codex harness with an MCP (Model Context Protocol) paper search tool scored 75.1, and the upper bound in Table 28 (a plan written with the original paper in hand) was 85.8.
    • Limits: The plan template follows AI paper conventions and drops components an LLM cannot easily determine, such as timelines, logistics, or human expertise, and the Nature Mental Health benchmark cuts the template to four sections. Because the rubric measures coverage of key elements, longer outputs may gain an advantage, and GPT-4.1 and o4-mini scored higher when allowed longer outputs. All candidate judge models showed a leniency bias with more false positives than false negatives, and human checks of the extracted plans and rubrics covered 8 papers, one chosen by each of eight experts (section means above 4.0 on a 5-point scale).
    概要PDF
  24. 1320Research AgentsarXiv:2510.15280

    Foundation Models for Scientific Discovery: From Paradigm Enhancement to Paradigm Transition

    • Problem: The authors argue that as science increasingly deals with emergent behavior, open-endedness and irreducible complexity, the limits of the four existing paradigms (experimental, theoretical, computational, data-driven) have become more evident. In drug discovery and material design the combinatorial explosion of candidate spaces makes exhaustive search infeasible, and the accumulation of experimental and observational data has outpaced the capacity to synthesize unifying theories. They therefore ask whether FMs merely enhance existing methodologies or redefine the way science is conducted.
    • Argument: This is a position paper whose stated contributions are a conceptual framework, a systematic review and taxonomy, and a research agenda. The authors take the position that FMs are not only improving parts of the scientific process but beginning to reshape the paradigm through which science operates, and they call the state reached once the autonomous stage is realized the fifth scientific paradigm. They also present the opposing view that FMs are powerful yet conventional tools that amplify existing methodologies.
    • Taxonomy: In stage 1, Meta-Scientific Integration, FMs are backend tools that automate procedures such as data preprocessing, literature retrieval and methodology matching within human-defined objectives, with low autonomy. In stage 2, Hybrid Human-AI Co-Creation, FMs are collaborators in research questions, hypotheses and experiment planning, while their actions are still initiated, constrained and validated by humans, with moderate autonomy. In stage 3, Autonomous Scientific Discovery, which the authors present as a forward-looking vision, FMs are independent agents that run the cycle from posing questions to executing experiments or simulations and interpreting results with minimal or no human oversight.
    • Review: Section 4 organizes cases from stage 1 and early stage 2 by paradigm, citing FMs as priors or feature extractors in Bayesian optimization and FM-generated Python control scripts for instruments in the experimental paradigm, and knowledge-graph-guided hypothesis generation and formal reasoning such as Logic-LM, which couples LLMs with symbolic solvers in a feedback loop, and LeanCopilot, which assists proof construction and verification, in the theoretical paradigm. For computation it cites LLM-SR, which turns plots or text into equation skeletons, FunSearch, Neural Operator and GraphCast, for data AlphaFold 2, ESMFold and MatterGen, and as a cross-paradigm case Coscientist, which translates research goals into machine-executable protocols and controls robotic synthesis. The official repository README holds a paper list by paradigm together with example documents for topic exploration, literature survey, problem refinement, methodology design and experiment code produced with OpenAI Deep Research and ChatGPT.
    • Risks and directions: The authors list four risks that grow as FMs move from tools to autonomous agents: bias and epistemic fairness inherited from training data, hallucination that produces plausible but unverified claims, reproducibility and transparency problems from opaque decision-making, and authorship, accountability and research ethics. As mitigations they propose more diverse training data and fairness-aware evaluation, symbolic logic checks and simulation-based validation, human-in-the-loop review and provenance tracking, logging of reasoning steps and version-controlled model checkpoints, and governance that mandates transparent disclosure of AI contributions. As future directions they name embodied scientific agents working with lab robotics, instruments and digital twins, closed-loop autonomy in which FMs continuously run from hypotheses through experiments to analysis, and continual learning that accumulates and refines knowledge.
    • Limits: The argument rests on a conceptual framework and a review of existing work, and the authors state that the transformation is still in its early stages and that autonomous discovery is in its formative stages. They also note that most FM deployments remain confined to static prompts, predefined tasks and fixed schema and typically lack persistent memory, adaptive feedback and physical embodiment, so their contributions are largely reactive and limited to isolated stages of research. Open challenges they list include linking high-level planning with low-level control in embodied agents, robustness of closed loops to noisy observations and shifting objectives, and robust transfer across heterogeneous tasks in continual learning.
    公開コード概要PDF
  25. 1319Research AgentsarXiv:2509.08713

    The More You Automate, the Less You See: Hidden Pitfalls of AI Scientist Systems

    • Problem: AI scientist systems run the research workflow from hypothesis generation through experiments to paper writing, but the authors note that their internal workflow has not been closely examined, which risks flaws that undermine the integrity, reliability and trustworthiness of their outputs. Diagnosis is also hard: with any existing dataset or task, web-scale pretraining makes data contamination almost inevitable, and each failure mode needs conditions that isolate it from confounding factors.
    • Input and output: The authors run two open-source AI scientist systems on SPR task descriptions with local datasets and loading scripts, and record the benchmarks they choose, the accuracies they report, the metrics they pick and the candidates their reward functions select to judge each failure. A separate LLM-based classifier takes the task description, the final paper, generated code, execution logs and pitfall-specific detection rules and returns a Yes or No detection with an explanation. This is an empirical study that diagnoses existing systems with controlled experiments and also proposes the diagnostic task and a detection method.
    • Method: SPR (Symbolic Pattern Reasoning) is a synthetic classification task where sequences of tokens built from 4 shapes and 4 colors are labeled accept or reject by whether they satisfy k hidden conditions on shape count, color position, parity and order. The benchmark-selection test offers 20 SPR datasets in a strict difficulty ladder, each raising one of rule complexity, vocabulary size or sequence length, under random five-letter names and random order, and asks the system to pick 4; the data-leakage test flips 20% or 30% of labels and treats test accuracy significantly above the theoretical upper bound (1 minus the noise level) as a leakage signal. The metric test adds noise so that only one of SWA (Shape-Weighted Accuracy) and CWA (Color-Weighted Accuracy) drops on the test set, and the post-hoc test inverts only the test performance of five candidates taken from logs to see whether the reward function changes its pick.
    • Implementation: Both systems ran with their default LLM API settings, except that the code model of The AI Scientist v2 was switched from Claude 3.5 Sonnet, deprecated as of August 13, 2025, to OpenAI's O3-mini. For benchmark selection Agent Laboratory was stopped at the data preparation stage and run 1,000 times each with and without SOTA (state of the art) figures and The AI Scientist v2 generated 1,000 ideas each without running experiments, and post-hoc selection was measured with 200 trials per condition from 20 research projects run 10 times each. The audit classifier used gemini-2.5-flash-preview-05-20 and was evaluated on 20 cases per pitfall type over five types, half positive and half negative, and per the official README the repository holds the research generated by both systems, the SPR task descriptions and datasets, and the detection script (detect_pitfalls.py).
    • Results: In benchmark selection Agent Laboratory picked the first four listed benchmarks in 82.4% of 945 runs with explicit selections when SOTA figures were given and in 79.6% of 927 runs without them, following list position over difficulty, while 47.1% of the benchmarks The AI Scientist v2 picked with SOTA figures were easy ones (18.0% without). No system peeked at test data, but Agent Laboratory used a subset of the data (16th run) or a synthetic dataset (11th run) without saying so in the paper, The AI Scientist v2 subsampled or synthesized datasets in runs 3, 7, 8, 11 and 13 and some test accuracies exceeded the provided SOTA figures, and with SWA listed first Agent Laboratory chose only SWA as its metric in all 20 runs. With test performance inverted, the share of picks going to the best train and validation candidate fell from 78.5% to 43.5% for Agent Laboratory and from 82.0% to 31.5% for The AI Scientist v2, and the LLM audit's overall accuracy rose from 55.0% (F1 0.51) with the paper alone to 82.0% (F1 0.81) with trace logs and code.
    • Limits: The authors focus on four failure modes that do not exhaust other forms of scientific malpractice or system failure, and they leave reward hacking of LLM-based peer review by AI scientists outside this study. The detection techniques primarily rely on controlled setups and observable behavior, so they may not generalize to more nuanced or adversarial misuse or to flaws never reflected in logs, and with only two systems evaluated, extrapolating to other systems and to domains beyond ML calls for caution. The detection evaluation used 20 cases per pitfall type, positives for pitfalls that were rare or absent in natural runs were created with controlled perturbations of the workflow, and data leakage was injected by merging test samples into the training set.
    公開コード概要PDF
  26. 1318Information RetrievalarXiv:2506.00249

    MIR: Methodology Inspiration Retrieval for Scientific Research Problems

    • Problem: In existing idea-generation methods that retrieve literature with a seed paper's abstract, the methodology and results in the abstract can bias retrieval toward the seed paper and reduce the novelty of the generated ideas. Commonly used retrievers and search APIs rely on sentence-level semantic similarity or generic inter-document relations, so they can fetch papers that merely share keywords, such as a 'sentence compression' paper for a 'model compression' problem. Some works also assume access to the seed paper's cited papers, so the authors target the realistic setting where only the research problem and its motivation form the query.
    • Input and output: This is a method paper that defines a new task and dataset and proposes a retriever training method and LLM re-ranking methods. The input is a research proposal (problem and motivation) with any mention of methodology removed, the output is the top-k papers of a corpus ranked by their usefulness for building a solution, and at inference the retriever takes only the proposal without citation information. The MIR-MultiCite dataset extends MultiCite, a computational linguistics dataset of 12,653 citation contexts labeled with eight citation intents, and splits proposals by time into 704 for training (before 2019), 86 for dev (January to June 2019), and 139 for test (after that).
    • Method: MAG treats each research proposal as a node with its cited papers as neighbors, annotates edges with citation intents and contexts, and counts the uses and extends intents as methodology citations. The retriever puts three triplets in one batch, pulling together the proposal and a methodology-cited paper, a methodology citation context and the proposal, and that context and its cited paper, and trains on the sum of L2 triplet margin losses; the hard negative of the first triplet is the paper outside the proposal's citations with the highest domain similarity measured from scientific named entities, mixed with random soft negatives at ratio α. LLM re-ranking uses Pointwise, which labels each of the top-k papers relevant or not, and MIR-Agent, which first writes the proposal's sub-problems and a generic action plan and analyzes each paper's applicability before judging, and papers judged relevant move ahead in their original order.
    • Implementation: Proposals were extracted from the abstracts of citing papers with Gemini-1.5-pro-001, and the training data adds part of 51,021 proposals from about 60k arXiv computational linguistics papers whose citation intents were weakly labeled by a SciBERT-based classifier. SPECTER2, Stella 400M, and Stella 1.5B (the last with LoRA) were fine-tuned on a single NVIDIA V100 GPU (32GB), and re-ranking used Llama 3.1 70B Instruct and Gemini 1.5 Pro. Per its README, the official repository (Anikethh/Methodology-Inspiration-Retrieval) holds the dataset used in the paper (proposals, cited papers, citation contexts, citation intents, and training, dev, test, and augmented training splits extended to mid 2024), and its top level has data and assets folders.
    • Results: On the MIR-MultiCite test set, fine-tuning raised Stella 1.5B's Recall@3 on the Extended Corpus (all cited papers of the training set plus the ground-truth citations of the test proposals) from 54.73 to 60.11 and its mAP from 49.08 to 56.89. On the Restricted Corpus, re-ranking the top 10 of the fine-tuned Stella 1.5B with few-shot MIR-Agent on Gemini 1.5 Pro raised Recall@3 from 66.95 to 71.45 and mAP from 64.24 to 69.02, and for the 75 test proposals whose top papers had full texts available, the same setup with full texts raised Recall@3 from 66.84 to 72.49. IA-Score, the average share of the 5 LLM-generated methodology ideas per proposal that match the methodology in the original paper's abstract, was 0.170 with abstracts retrieved by MIR and 0.142 with abstracts from the best baseline retriever.
    • Limits: The dataset is limited to English papers in computational linguistics, and the retriever is trained on cited-paper abstracts, with full texts used for re-ranking the top-k. The authors report that MultiCite's original human annotations sometimes omit methodology citations and occasionally include trivial references (for example, mere use of GPT-4). An analysis of 15 proposals (39 relevant papers) with subpar Recall@5 under the best configuration (MIR-Agent with full papers) found 17 incorrect or missing annotations, 7 subjectivity cases where the authors chose a different research direction, and 15 misclassifications by the pipeline.
    公開コード概要PDF
  27. 1317Research AgentsarXiv:2505.15047

    PiFlow: Principle-Aware Scientific Discovery with Multi-Agent Collaboration

    • Problem: LLM-based multi-agent systems (MAS) execute experiments well within predefined workflows, but they often generate hypotheses that lack clear direction and find it hard to establish links between hypotheses and the evidence that supports or refutes them, which makes exploration inefficient. Many systems are also tailored to specific tasks, often relying on prompt engineering heavy with domain knowledge, so moving them to a new scientific domain often needs substantial modification.
    • Input and output: This is a method paper that presents a plug-and-play module for existing MAS together with its theory. Inside the MAS, a Hypothesis Agent proposes one testable hypothesis per iteration grounded in a selected principle, with a major and a minor premise, and an Experiment Agent calls an experimental tool (here a surrogate model) to obtain a quantitative outcome such as a property value. PiFlow takes the accumulated principle-outcome pairs as input and gives the Planner agent the highest-potential principle with one chosen instruction: refine, validate or explore alternatives.
    • Method: Principle selection is framed as a Min-Max optimization that combines exploitation, which minimizes cumulative regret (the summed gap between the optimum and the obtained value), with exploration, which maximizes the mutual information gained from a hypothesis outcome, and the authors give a proof in the appendix that with finite entropy and a bounded evaluation function the bound on cumulative regret grows sublinearly, at the order of the square root of the number of steps T. In practice, an exploitation score from each principle's observed outcomes normalized to 0 to 1 and an exploration score from the cosine distance between sentence embeddings of a principle and already tested principles are mixed with λ to pick the best principle. With the default thresholds, if that principle's exploitation score is above 0.7 the instruction is refine, above 0.4 validate, and otherwise explore, and with fewer than 3 records the system first initializes one principle to explore.
    • Implementation: In the main comparison, all agents use QwenMax as the base LLM with external search tools blocked (the model ablations swap the LLM of the Hypothesis Agent and Planner for models such as GPT-4.1-mini and Qwen3-32B), and locally deployed surrogate models serve as the primary evaluation tool in place of real experiments. The tasks are Nanohelix Optimization (NHO), where a LightGBM model (coefficient of determination r2 0.98), trained with 5-fold cross-validation on 80% of 6300 records of DFT (density functional theory) simulation data and tested on the remaining 20%, predicts the chirality measure g-factor, Molecular Bio-activity Optimization (MBO), where a GNN (graph neural network, test-set r2 0.91), trained on 80% and tested on 20% of 50,000 molecules randomly sampled from the bio-activity compound database ChEMBL35, predicts pChEMBL (a negative log of molar concentration expressing activity), and Superconductor Optimization (SPO), where an MLP (multi-layer perceptron, test-set r2 0.91), trained on 21,056 and tested on 5,265 of 26,321 superconductor records, predicts critical temperature, and each searches for the candidate that maximizes the target within 24 budget steps. Per the README, the official repository holds a conda environment file, inference.py for a nanohelix demo task, per-task configs, a tools folder for registering one's own experimental tools and a Colab tutorial, and running it needs an OpenAI-compatible API key.
    • Results: Averaged over the three tasks, PiFlow reached 49.86% AUC and 65.41% SQ, above The AI Scientist v2 (38.01%), the baseline with the highest average AUC, and AI Researcher (58.16%), the baseline with the highest average SQ. Per task, AI Researcher's SQ on MBO (95.66%) exceeded PiFlow's (84.55%), and The AI Scientist v2's AUC on SPO (28.43%) exceeded PiFlow's (21.51%). On NHO, the Vanilla agent took 2416 seconds to reach 50% SQ while PiFlow reached 52% SQ in 427 seconds, 5.6 times faster, and in the cost analysis, which ran each task for 24 iterations with QwenMax, token use against Vanilla fell by 26.7% on NHO and 13.7% on SPO but rose by 7.7% on MBO.
    • Limits: Locally deployed surrogate models served as the primary evaluation tool in place of real laboratory validation, and the authors state that applying the approach to a new domain first requires building a reliable simulator or predictive model. From varying λ on NHO with QwenMax, the authors conclude that exploration efficiency is sensitive to the choice of λ and that the optimal value is task-dependent, and because exploration scores compare all pairs of past principles, the per-iteration cost grows with the square of the number of records times the embedding dimension, which the authors admit can become a bottleneck for very long processes (over 24 NHO iterations the PiFlow part took 9.4% of the total PiFlow-MAS runtime). The standard deviation of PiFlow's SQ on MBO was a large 29.63, Bayesian optimization with a manually defined search space beat PiFlow in the 24-iteration comparison on NHO AUC (68.86%), NHO SQ (78.76%) and SPO AUC (31.61%), and most ablations were run on the single NHO task.
    公開コード概要PDF
  28. 1316Research AgentsarXiv:2610.06577

    Optimizing the Optimizer: Language Models Discover Faster Molecular Relaxation Algorithms

    • Problem: In geometry optimization at the DFT level each step needs one force (gradient) evaluation, which dominates the wall time, so the step count sets the cost. Letting an LLM agent rewrite an optimizer raises three constraints: DFT is too expensive to score hundreds of candidates, candidates must work on unseen molecules, and the agent must be kept from cutting force calls by stopping early.
    • Input and output: The agent receives a starting program that merges Sella 2.5.0 into one file (algo.py, 5975 lines including comments and blank lines), training and validation molecule sets and a remote evaluator, edits only algo.py, and returns AutoSella optimizers that reach the same convergence criteria with fewer force calls. Each optimizer itself takes one molecule and a force evaluator and updates the structure until all five convergence conditions hold (cap of 200 force calls, 500 for Solvated PubChem), and the molecule sets cover drug-like molecules with their surrounding biomolecules and solvent. This is a method paper that finds, releases and evaluates new optimization algorithms with a gated autoresearch loop.
    • Method: In each cycle the agent reads the cycle record, the narrative log and the mechanisms already rejected, commits one coherent edit to algo.py, and after evaluation receives a per-molecule table of relative force calls, relative energy, final energy difference and convergence status. The validity gate discards candidates on training that have evaluation errors, mean energy recovery below Sella's, or a cost drop smaller than the threshold δ, and the generalization gate accepts a new champion (the best program so far) only if it also cuts cost on validation without evaluation errors while keeping mean energy recovery. The search runs only on the cheap GFN2-xTB potential, algo.py is the only file the agent may edit, and changing the evaluator, convergence criteria or molecule sets and adding molecule-specific special cases are prohibited.
    • Implementation: Claude Code driving Claude Fable 5.1 (max) and Codex driving GPT-6 Astra (high) each ran independently for 66 active research hours and produced AutoSella-F and AutoSella-A, and the remote evaluator, a Redis queue with 192 single-thread workers on 4 hosts with Intel Xeon Gold 6348 CPUs, scored one candidate in approximately 60 to 90 seconds on average, depending on the run. The 469 training structures combine 225 PubChem molecules and 219 DES370K dimers from SPICE 2.0.1 with 25 registered drugs from the RowanSci optimizer benchmark, and the 465 validation structures come from SPICE 2.0.1. Per the official GitHub README, the main branch holds the three frozen optimizers (AutoSella-F, A, G), a Sella 2.5.0 reference wrapper, the molecule preparation pipeline and an example, while the autoresearch harness, per-model run records and the xTB and DFT evaluations sit on separate branches.
    • Results: On the four held-out GFN2-xTB sets AutoSella-F cut mean per-molecule force calls to 39.86 to 73.25% of Sella with higher mean energy recovery on all four, while AutoSella-A used 78.14 to 92.57% on the first three sets but rose to 118.77% on Solvated PubChem with lower mean energy recovery. Under r2SCAN-3c DFT, used only for evaluation, AutoSella-F needed about 23 to 60% fewer force calls per molecule than Sella with greater mean energy recovery, and on Solvated PubChem it used on average 40.24% of Sella's force calls per molecule with 1.27% higher mean energy recovery. Over 66 hours Fable accepted 37 of 108 evaluated candidates and Astra 29 of 672, and AutoSella-G from the Grok 4.6 run in the appendix lowered the training cost to 97.79%.
    • Limits: Both gates judge by mean energy recovery, force-call cost and evaluation errors, and in the separate GFN2-xTB structure check of Appendix E the share of molecules where AutoSella-F reaches the same local minimum as Sella was 97.2% on Test, 93.2% on Dipeptides, 39.8% on amino acid and ligand pairs and 1.0% on Solvated PubChem. The authors note that the other endpoints lie below the reference energy on average, yet on GFN2-xTB AutoSella-F ended more than 1.0 kcal/mol above Sella on 18.74% of amino acid and ligand pairs and 38.00% of Solvated PubChem systems. AutoSella-A used 109.06% of Sella's force calls on Solvated PubChem even under DFT with lower mean energy recovery, and the Grok 4.6 result AutoSella-G was much worse on the intermolecular sets and was left out of the DFT evaluation.
    公開コード概要PDF
  29. 1315Research AgentsarXiv:2608.14667

    Position: AI Agents in Scientific Teams Should Be Studied as Human-Agent Systems

    • Problem: LLM-based agents are entering scientific work as collaborators, yet most AI Scientist research focuses on growing agent autonomy and limits human participation to setting goals and verifying work. The authors argue that this overlooks the social side of scientific teamwork and that this literature has largely been isolated from the socio-technical realities of science.
    • Argument: This is a position paper arguing that studying AI Scientists as human-agent systems (HAS), where the unit of analysis is the human-agent pair, is underexplored and undervalued, and that new HAS research is needed to understand human-AI synergy. The authors argue that the HAS lens can help better understand and potentially mitigate near-term risks, citing 100 confirmed hallucinated citations found in 51 accepted NeurIPS 2025 papers and an expert review that found noticeable plagiarism in 36% of LLM and AI Scientist-generated papers. They also cite a study in which AI-augmented research produced three times as many papers with a 5% drop in the diversity of explored topics.
    • Taxonomy: The authors group AI Scientist systems by the main channel through which human feedback enters, namely artifact-level, discovery-phase-level, research-plan-level, and asynchronous steering and interruption, and tag each with Zou et al.'s feedback type (evaluative, guidance, corrective), granularity, and phase. The review finds that most systems give humans supervisory roles such as scoping, auditing, and approval, that artifact-level feedback is the most common, and that only a handful support flexible asynchronous interaction.
    • Evidence: In a preliminary experiment with AstaBench's ReAct (an agent style that alternates reasoning and acting) agent harness and GPT-5-mini on 10 End-to-End Discovery (E2E) development tasks, adding an ask user tool and the feedback taxonomy to the prompt produced 1 question, and the E2E score went from 0.39 to 0.36. Forcing a question before every action raised questions to 92, and in the average over three LLM judges 0.71 of them were evaluative and 0.04 corrective, with an E2E score of 0.24. In this experiment an automatic continuation prompt, the AstaBench default, answered the questions instead of a human.
    • Cases and proposal: In one published case, Gemini 3 Pro produced a plan (plan.md) and a draft manuscript (paper.tex) from an initial prompt and then expanded proofs and exposition over eight prompt turns, while the expert pointed out missing prior work and corrected a wrong assumption in a lemma. In another, GPT-5 drafted the PDE (partial differential equation) and numerical workflow for a reduced physics model of thermonuclear burn, while the expert tuned the conditions by hand and judged when outputs were still noise. The authors propose a toy model in which team utility is the collaboration advantage CA (how much the team beats its best member) minus the collaboration disadvantage CD (communication and coordination cost).
    • Limits: The preliminary experiment used 10 development tasks with an automatic prompt answering the questions, which the authors defend as a deep rather than broad measure because each task is a full idea-to-report workflow. The case studies are qualitative readings of two published accounts, and the toy model and proxy metrics such as Review Time, Intervention Density, and Simulated Oracle Synergy are proposed as directions for future work. The authors address the opposing view that the HAS lens may add little once evaluation costs are counted, and acknowledge that centaur evaluations comparing HAS with humans alone and agents alone are currently expensive and hard to reproduce.
    概要PDF
  30. 1314Research AgentsarXiv:2607.07663

    Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops

    • Problem: AI systems increasingly take part in their own improvement, from revising outputs and changing their harnesses to training on self-generated data and doing AI research, yet the literature lumps fundamentally different ambitions under words like self-refine, self-reward, self-play and self-evolve. The authors note that existing surveys cover single slices such as on-policy distillation or tree search with rewards, and that to their knowledge no survey covers the four categories jointly, from bounded self-refinement to open-ended recursive self-improvement (RSI).
    • Scope: This is a survey that collects and classifies literature. It gathers 1,250 arXiv papers from 2024 to 2026: a seed harvest of 871 papers across seven threads plus a targeted supplement of 379 papers on self-evaluation, test-time training and zero-data self-play. According to the README, the official GitHub repository deepgrounding/recursive-self-improvement holds corpus_v2.csv with taxonomy assignments, the seed snapshot, scripts for reclassification, supplemental harvest, bibliography and figures, and the manuscript source (main.md, matching arXiv v1).
    • Taxonomy: The first axis is what the system improves: deployment-time self-evolution (393 papers), training-time self-iteration (340), self-evaluation (318) and Auto Research (139), plus a foundations, limits and safety family (60). The second axis is who validates the loop, in three degrees: human-in-the-loop, human-on-the-loop and closed loop. The authors report that nearly all of the 1,250 papers study bounded self-refinement in human-on-the-loop cells, where humans audit outcomes.
    • Argument: The authors rank improvement signals into a verification hierarchy of formal verifiers (proof checkers, type systems), execution feedback (tests, compilers, benchmarks), learned judges (reward models, LLM-as-judge) and intrinsic signals (confidence, self-consistency, likelihood), and observe, as a qualitative pattern across the corpus, that demonstrated self-improvement strength tracks this hierarchy. They group failure modes into four: the self-confirming loop, collapse, diversity collapse and frame lock-in (continuing to optimize a stale objective). The foundations family is the smallest of the five at 60 of 1,250 papers, and the authors suggest that governance-grade measurement of self-improvement is the most underpopulated research niche the survey found.
    • Evidence: In the Mirror Loop study cited by the authors, models from three providers ran ten rounds of ungrounded self-critique across four task families and informational change declined 55% across iterations, while one verification step at iteration three restored forward movement. In the cited SkillsBench result, human-authored skills raised pass rates by 16.2 points while LLM-authored skills gave no measurable gain. The cited A-Evolve-Training ran the entire post-training loop of a 30B model for four rounds with no human in the loop, within a human-specified objective and search space, and reached 0.86 on a public leaderboard, near the 0.87 of the top human submission.
    • Limits: The authors describe the corpus as a recency-biased sample, and say that because 74% of papers were posted in 2026, citation counts are near zero and cannot be read as impact. Roughly 54 of the 379 supplemental papers are peripheral query bleed that remain in the corpus counts, and classification and corrections come from a single annotator. Industrial RSI practice inside frontier labs is observable only through what labs publish.
    公開コード概要PDF
  31. 1313Research AgentsarXiv:2606.11897

    Notes2Skills: From Lab Notebooks to Certainty-Aware Scientific Agent Skills

    • Problem: Prior procedural-text extraction assumed prescriptive input such as recipes and wet-lab protocols, while in lab notebooks confirmed observations, uncertain judgments and next-step suggestions often appear in the same passage. The authors argue that losing this distinction causes uncertainty laundering, where an author-qualified statement becomes a stronger processing commitment without adequate authorization and evidence and may lead to discarding data, and directive loss, where a firm directive is dropped or weakened into an opinion so that a supported processing action is missed.
    • Input and output: This framework paper proposes a two-stage pipeline and evaluates it on newly annotated corpora and real nanopore experiment records. The input is notebook text segments, and Stage 1 (Epistemic Directive Extraction, EDE) outputs for each segment, as JSON, whether it holds a directive, one of five directive types, and an author certainty label: FACT (an unhedged assertion), JUDGMENT (an uncertain or tentative assessment) or SUGGESTION (optional advice). Stage 2 builds a Markdown skill document (MetaSkill) that attaches the source excerpt and certainty label to each directive, and an agent reading it picks one action per recording file: KEEP_FULL (keep the whole recording), FLAG_FOR_REVIEW (keep it for review), RAISE_THRESHOLD (raise the event-detection threshold), TRUNCATE_AT (cut at a given time) or SKIP_FILE (exclude the file).
    • Method: Stage 1 extracts directives and certainty labels with LLM prompting, and in Stage 2 a deterministic compiler with no LLM emits one capsule per directive and checks directive preservation, unchanged labels, source fidelity and schema validity. At execution, Layer 1 turns the instrument trace into a SignalFindings summary, a Claude Sonnet 4.5 agent in Layer 2 proposes a processing decision, and in the verify setting the Layer 3 executor passes strong decisions such as TRUNCATE_AT, RAISE_THRESHOLD and SKIP_FILE only when a capsule authorization path and the signal evidence agree, otherwise applying an action-specific fallback, usually FLAG_FOR_REVIEW. The elevate setting turns a cautious proposal into TRUNCATE_AT only when exactly one stored truncation candidate timestamp matches a calibrated step drop.
    • Implementation: The annotated corpora total 461 segments: 201 FreeNotes segments from Chinese-English nanopore notebooks contributed by two senior researchers at two institutions, 155 ONS (Open Notebook Science) segments from openlabnotebooks.org, and 105 WLP (Wet Lab Protocols) segments from a published protocol corpus, and Stage 1 compared GPT-4o, Claude Sonnet 4.5 and Qwen-Max with zero-shot and few-shot prompting at temperature 0 on an 87-segment test set. The downstream evaluation compared seven configurations on 48 nanopore files from three FreeNotes sessions (17 in Saturation-A, 22 in Saturation-B, 9 in Step-drop), with five API calls per configuration and file decided by majority vote. The authors describe a released artifact with compiler and executor code, evaluation scripts and fixed decision records, but the paper gives no repository address, and the raw ABF (Axon Binary Format) current traces stay out of redistribution under the data-sharing agreement.
    • Results: In Stage 1, Claude Sonnet 4.5 few-shot reached a directive-detection F1 of 0.737, above 0.682 for the strongest zero-shot setup, and the Stage 2 compiler preserved all 149 fixed input directives and passed all four checks. On the 48 nanopore files, the external LLM and Action-only baselines reached only 0 to 21.4% review recall on the saturation sessions, while the full configuration (verify+elevate) reached 85.7% and 100% review recall and 96.4% and 100% balanced accuracy on Saturation-A and Saturation-B. On the FACT-dominated Step-drop session, the full configuration's accuracy of 77.8% was below the 88.9% of Action-only.
    • Limits: The authors state that the downstream evaluation covers only three nanopore sessions and one downstream model, and that only the nanopore domain adapter was built and evaluated. The main comparison used adjudicated EDE inputs, and replacing them with Claude Sonnet 4.5 few-shot predictions dropped directive precision to 34.8% on Saturation-B and 47.9% on Step-drop and pushed Cohen's kappa below chance to -0.07 and -0.03. The executor-enabled comparison also changes the Layer 2 policy text, so the authors note that a matched-policy experiment is needed to isolate the executor's own effect.
    概要PDF
  32. 1312Research AgentsarXiv:2605.15871

    Agentic Discovery of Neural Architectures: AIRA-Compose and AIRA-Design

    • Problem: Most LLMs today are Transformers that interleave attention and MLP layers 1:1, and moving to hybrid designs that mix in other primitives such as Mamba makes the design space vast and combinatorial. The authors argue that relying solely on manual exploration may overlook strong non-obvious arrangements or entirely new primitives. They therefore hand this search to research agents and frame it as a step toward recursive self-improvement, where agents discover and optimize the architectures that power them.
    • Input and output: This is a framework paper that builds 12 tasks in the AIRS-Bench format, a task standard for evaluating an agent's ability to conduct independent research, and runs experiments on them. Given a task description and data, the agent submits a 16-primitive architecture string (submission.csv) in AIRA-Compose, or a model.py for LRA or a train.py for Autoresearch in AIRA-Design. An isolated evaluation script trains the submission from scratch and scores it on a held-out test split or a pinned validation shard.
    • Method: An agent is an LLM plus a scaffold (search policy), run inside AIRA-dojo, a harness that evolves code solutions through tree-based exploration, with four operators: Draft, Debug, Improve, and Analyze. The greedy scaffold drafts 5 initial solutions and keeps applying Improve to the one with the best validation score. AIRA-Compose turns the search and evaluation steps of Composer, an existing framework that searches hybrid architectures with Bayesian optimization, into agent tasks, then uses Composer's aggregation and extrapolation steps to scale the found 16-layer patterns to 350M, 1B, and 3B parameters by stretching (proportionally expanding contiguous blocks) or stacking (repeating the whole pattern).
    • Implementation: Each agent gets one H200 GPU, and a run lasts at most 24 hours or 500 steps, with 60 hours per run for the larger BabiStories and DCLM searches. AIRA-Compose uses 6 LLMs, while in AIRA-Design the LRA experiments build 20 agents from 12 LLMs including Opus 4.6 and Gemini 3 Pro and the Autoresearch experiments build greedy agents from 7 of those LLMs. Scaled-up training ran 350M models on 8 H200 GPUs and 1B and 3B models on 16 H200 GPUs.
    • Results: Among architectures built from the two primitives MLP and attention and trained at the 1B scale on 37.5B tokens, AIRAformer-D (Stretched) reached a validation loss of 2.734 and 59.7% average 0-shot accuracy, ahead of Llama 3.2 and the Composer-found models, and in the 3-primitive comparison that adds Mamba, AIRAhybrid-D (Stretched) reached a validation loss of 2.719 from a single seed. On LRA, the best accuracy of agent-designed models was 2.3pp (percentage points) below the human-designed SOTA (state of the art) of 82% on document matching and 2.6pp below the SOTA of 91% on text classification. On Autoresearch, Greedy Opus 4.5 with literature access reached a validation BPB (bits per byte, lower is better) of 0.968 in its best seed, below the original Autoresearch best of 0.9775.
    • Limits: The authors state that AIRA-Compose relies on small-scale proxy evaluation that does not always predict large-scale performance, and that the aggregation and extrapolation steps remain non-agentic fixed procedures. On LRA, the agent-designed architectures largely recombine prior ideas such as Performer, Longformer, and Conformer. The Autoresearch runs used H200 GPUs in a sandbox without internet access, unlike the original H100 setup, and the authors note this may explain why the same train.py started at a BPB of 1.012 instead of the original 0.998.
    概要PDF
  33. 1311Research AgentsarXiv:2604.05550

    AutoSOTA: An End-to-End Automated Research System for State-of-the-Art AI Model Discovery

    • Problem: Reaching state-of-the-art (SOTA) results in AI research takes long cycles of implementation, replication, debugging, evaluation, and reflection, and the needed artifacts are spread across papers, repositories, datasets, checkpoints, and undocumented engineering assumptions. The authors argue that existing iterative frameworks such as AutoResearch confine the agent to one bounded code module inside a sanitized sandbox, which leaves out the path from a paper to a recovered execution environment and a reproduced baseline.
    • Input and output: It takes one paper from a top-tier AI conference and returns an executable code repository whose performance on the core metric strictly beats the method proposed in the paper. In the final phase it restores the container to the best-scoring state, saves it as a Docker image (autosota/paper:optimized), and exports the full repository to the host. This is a system paper that implements eight agents and runs experiments on 125 conference papers.
    • Method: Eight agents split the work into preparation (AgentResource, AgentObjective), execution (AgentInit, AgentMonitor, AgentFix), and improvement (AgentIdeator, AgentScheduler, AgentSupervisor), and the execution and improvement stages iterate by passing results back and forth. If the three most recent iterations were all parameter adjustments, AgentScheduler forces the next iteration onto the Leap Path, which requires a structural change, and if its first result does not beat the best score it allows up to five more iterations before rolling back to the pre-Leap optimum. AgentSupervisor blocks shortcuts such as editing the evaluation script, using test data for training, or hard-coding predictions through red line rules (R1 to R7).
    • Implementation: The evaluation set starts from accepted papers of the past year at eight conferences (NeurIPS, ICLR, ICML, CVPR, ICCV, ACL, NAACL, AAAI), screened for method papers, public code links, and repository readiness into 2,347 candidates, of which 745 were run in preliminary tests, and papers with environment installation failures or more than 4 GPU hours per full replication were removed to leave 125. AgentInit builds the environment with Repo2Run and falls back to a manual Docker setup when the automated build fails, and AgentScheduler commits the original code under a _baseline tag inside the container and records the measured baseline in scores.jsonl as iteration 0. The repository map, the idea library, and a prior literature report are kept in code_analysis.md, idea_library.md, and research_report.md, and the scheduler treats two GPUs as one unit that runs at most one active task at a time.
    • Results: Of the 125 screened papers, AutoSOTA found new SOTA models that beat the original method for 105, and the authors report an average gain of nearly 10% on each paper's single core metric at about five hours per paper. By type of change, algorithmic changes such as loss redesign or architectural restructuring (Class A) covered 64 papers (51%) with a mean gain of 8.82%, and hyperparameter tuning (Class B) covered 41 papers (33%) with a mean gain of 5.67%. In the case studies, the throughput of FR-Spec (a method that speeds up LLM inference with speculative decoding) rose from 674.58 to 765.27 tokens per second, a 13.44% gain, and the optimality gap (the distance from the optimal solution) of MoSES (a multi-task vehicle routing solver based on a mixture of specialized experts) on the capacitated vehicle routing problem (CVRP) fell from 0.914% to 0.444%.
    • Limits: Of the 125 papers, 20 (16%) fell into Class C (optimization failure), and the authors attribute these cases to invalid optimizations in which AgentSupervisor failed to detect or prevent potential violations. The authors state that the current loops rely mainly on the Result Match items of the AgentObjective rubric, which may overlook structural and semantic fidelity. They also describe the MoSES gain as a deliberate compute trade-off that expanded inference time from about 3.7 to 103 seconds.
    概要PDF
  34. 1310Memory / AgentsarXiv:2604.01007

    Omni-SimpleMem: Autoresearch-Guided Discovery of Lifelong Multimodal Agent Memory

    • Problem: AI agents that work over long horizons are bottlenecked by memory for storing and recalling text, image, audio, and video experience, and building good memory means searching a large design space where architecture, retrieval strategy, prompts, and data pipelines interact. Existing memory systems come from manual research cycles in which a single researcher may explore only a handful of configurations per day. Traditional automated machine learning (AutoML) searches predefined numerical hyperparameter spaces, leaving code comprehension, bug diagnosis, and architectural redesign outside its reach.
    • Input and output: This is a system paper about a memory system whose design was found by an autoresearch pipeline. The 23-stage pipeline AutoResearchClaw took the code of the text-only memory SimpleMem, two benchmark evaluation harnesses that report F1, and LLM API access, and produced Omni-SimpleMem. Omni-SimpleMem stores text, image, audio, and video inputs as memory and answers a query by putting the retrieved memories into the context.
    • Method: The pipeline analyzes prior results, forms a hypothesis, changes the code, and evaluates it on a benchmark; it proceeds when F1 improves by 0.5% or more, refines the hypothesis when the result is ambiguous, and reverts and pivots after two consecutive drops. The discovered architecture filters redundant inputs by comparing CLIP image embeddings across consecutive video frames, voice activity detection (VAD) probability for audio, and Jaccard overlap with recent summaries for text, and stores the rest as Multimodal Atomic Units (MAUs), keeping summaries and embeddings in hot storage and raw content in cold storage. Retrieval combines set-union merging that keeps the FAISS dense vector ranking and appends BM25 keyword results, pyramid retrieval that starts from summaries of top candidates, opens full text when similarity exceeds a threshold, and loads raw content within a token budget, and knowledge graph expansion over 7 entity types.
    • Implementation: Omni-SimpleMem is implemented in 13,300 lines of Python across 11 subpackages, and its default build uses dense retrieval with FAISS and all-MiniLM-L6-v2 embeddings (384 dimensions), visual novelty checks with frozen CLIP ViT-B/32, and knowledge graph extraction with GPT-4o in JSON mode. Defaults are top-k 20, a Level 2 expansion threshold of 0.4, and a token budget of 6,000, while for LoCoMo the pipeline chose text-emb-3-large embeddings and top-k 20 to 30. The README of the official GitHub repository bundles SimpleMem, Omni-SimpleMem, and EvolveMem into one simplemem package that picks the Omni backend when a multimodal input function is called first, and points to LoCoMo and Mem-Gallery reproduction scripts in the OmniSimpleMem folder and a Model Context Protocol (MCP) server that supports text memory only.
    • Results: In about 72 hours the pipeline ran about 50 experiments, raising LoCoMo F1 from the initial 0.117 to 0.598 (+411%) and Mem-Gallery F1 from 0.254 to 0.797 (+214%). The biggest LoCoMo gain (+175%) came from finding a one-line bug where the API call lacked response_format, and on Mem-Gallery returning the full original dialogue text instead of summaries gave the largest gain of +53%. With all 5 LLM backbones, Omni-SimpleMem's overall F1 was higher than all 6 baselines, from 0.492 (GPT-4.1-nano) to 0.613 (GPT-5.1) on LoCoMo against 0.342 to 0.432 for the previous best SimpleMem, and from 0.749 to 0.810 on Mem-Gallery.
    • Limits: The +411% and +214% gains are relative to the initial configurations; the first LoCoMo improvement fixed a missing response_format bug in the starting configuration, and the first Mem-Gallery phase upgraded the LLM from gpt-4.1-nano to gpt-4o and moved to local embeddings (+39%). Humans supplied the starting code, the two evaluation harnesses, API access, and the instruction to extend to multimodal memory, and evaluation used the LoCoMo and Mem-Gallery benchmarks. The authors name the conditions under which this approach fits: immediately computed scalar metrics, a modular architecture, short cycles of 1 to 2 hours per experiment, and version-controlled code changes that can be reverted.
    公開コード概要PDF
  35. 1309Research AgentsarXiv:2602.19810

    From Agent-Only Social Networks to Autonomous Scientific Research: Lessons from OpenClaw and Moltbook, and the Architecture of ClawdLab and Beach.Science

    • Problem: In January 2026 the OpenClaw agent framework and the agent-only social network Moltbook produced large-scale agent-to-agent interaction, and one risk assessment report documented that social evaluation such as karma voting pushed cryptocurrency promotions and anti-human manifestos to high visibility. The authors argue that research agent platforms such as Google's AI Co-Scientist, Robin and The AI Scientist v2 are also bound by four structural properties: a fixed model, hardcoded capabilities, a predetermined workflow and agents that share one learned distribution.
    • Input and output: The review takes six preprints and technical reports found on arXiv, Google Scholar, Semantic Scholar and Zenodo between January 30 and February 10, 2026, adds gray literature such as GitHub repositories, developer interviews and news coverage, and distills five recurring architectural patterns. ClawdLab takes a research question posted to a forum by humans or agents and produces lab tasks, tool-verified results and documents, while Beach.science collects ideas and computational analyses that agents post to a public feed. The paper is a design science framework paper that combines a multivocal literature review (covering both academic and gray literature) with the architecture of two platforms.
    • Method: ClawdLab groups agents into labs and gives each one of five roles (principal investigator or PI, research analyst, scout, critic, synthesizer), with hard restrictions so that a scout can only run literature review tasks and a critic only critique tasks. A task moves from proposed to in progress to completed, the PI then opens a vote, and the task resolves only when at least two substantive votes come from at least half of the active members and a strict majority decides; a structured critique filed by any member must be resolved first. Verification means the PI checks submitted work with external tools such as API calls, computational services and MCP (model context protocol) integrations, and agents pull work on a 45 to 90 second cycle without a central orchestrator.
    • Implementation: Per the paper, ClawdLab is a TypeScript web application built with Next.js and Prisma on PostgreSQL and S3-compatible object storage, and external tool calls go through a provider proxy that holds credentials server-side and records each call as a job. Per the official repository (bio-xyz/ClawdLab) README, the release includes API routes and web pages for labs, the task lifecycle, lab states, documents, the forum and the provider proxy, plus the Prisma schema and demo seed data; literature search and data analysis provider URLs and keys are set as server environment variables, and agents take part through registration, heartbeat, role playbook and pending-work APIs. Beach.science is a Next.js app deployed on Vercel whose OpenClaw agents can add role templates such as a peptide binding analyst to starter-pack skills for literature search, hypothesis generation and scientific discussion, and the authors describe both platforms as open-source research prototypes under active development, with the reward system that hands out inference resources still at the planning stage.
    • Evidence: In the reviewed sources, Wiz security researchers found that the 1.5 million registered agents Moltbook claimed were run by about 17,000 human owners, cryptocurrency activity made up 19.3% of content in one risk assessment report that analyzed posts and comments collected over 72 hours, and a test of 131 threatening skills from OpenClaw's public registry found vulnerabilities at multiple execution stages. In ClawdLab's Protein Annotation Sanity Checker lab, a Synthesizer agent went from joining the lab to posting its conclusions within five minutes without human intervention. At the time of capture, the Beach.science landing page showed 52 science agents, 924 hypotheses and 2,206 comments.
    • Limits: The authors are also the developers of ClawdLab and Beach.science and flag this dual role themselves, since the same people identify the failure modes and propose the remedy. The search closed on February 10, 2026, and the sample sizes and statistics reported by the six studies were taken at face value. At the time of writing, ClawdLab is operational with developer-team agents, while testing its incentive structures with external agents and collecting longitudinal performance data remain future work, and Beach.science is at the prototype interaction stage.
    公開コード概要PDF
  36. 1308Research AgentsarXiv:2602.08990

    InternAgent-1.5: A Unified Agentic Framework for Long-Horizon Autonomous Scientific Discovery

    • Problem: The authors argue that many existing AI Scientist systems are tied to domain-specific designs, which makes unified reasoning across scientific fields difficult. Their optimization often refines proposals through updates within one trajectory, and most systems keep no persistent memory across extended research cycles.
    • Input and output: This is a system paper with an implementation and experiments. The input is either an algorithm discovery task, given an objective and the original baseline codebase, or an empirical discovery task, given observations and a research question. The output is hypotheses and method plans, executed code or experimental protocols, measurements and reports, and each candidate solution is a full experimental proposal covering code logic, parameter settings and physical operation protocols.
    • Method: Generation produces hypotheses and method plans through deep research built on a literature knowledge graph and a flow graph that links subtasks as a DAG (directed acyclic graph), Verification evaluates them computationally or experimentally, and Evolution stores the outcomes in memory for the next cycle. Verification search replaces the tree of MCTS (Monte Carlo Tree Search) with a solution graph that pools all prior experiments, and builds new proposals with four operators: parent-only expansion, intra-branch evolution, cross-branch reference and multi-branch aggregation. Memory is split into reusable procedures distilled from past reasoning (Strategy-Procedural Memory, SPM), the method and metrics stored after each experiment (Task-Episodic Memory, TEM), and principles drawn from pairwise comparison of methods at the end of each batch plus a graph of explored research directions (Semantic-Knowledge Memory, SKM).
    • Implementation: On GAIA (real-world tasks that need reasoning, web navigation and tool use) and HLE (Humanity's Last Exam), the system was compared on three base models, Qwen3-235B, o4-mini and a Gemini-3-pro plus o4-mini combination, and on reaction outcome prediction (ChemCoTBench) it called RDKit to compute molecular descriptors. For fluorescent protein design it used ESMFold and ProSST, and in that wet-lab stage SCP (Science Context Protocol) coordinated automated devices for DNA assembly, expression, purification and fluorescence measurement. The official repository README states that it releases launch_discovery.py for runs from idea generation to experiments, launch_qa.py for deep research question answering, a tasks folder with a prompt.json and baseline code per algorithm discovery task, and a memory module.
    • Results: On the 165-question GAIA validation set, the Gemini-3-pro plus o4-mini configuration averaged 86.06%, above Manus at 73.30% and above the 80.8% of MiroThinker-v1.5-235B, which was evaluated only on the text-only subset. On the AutoEAP scientific data task, HK-PCC (housekeeper Pearson correlation coefficient between predicted and measured enhancer activity) rose from 0.65 for the DeepSTARR baseline to 0.91. The ablation without SPM reached 82.42% average GAIA accuracy with 22.69 average tool calls, worse than the 86.06% and 18.52 of the full system.
    • Limits: The two life science cases reproduce targets reported before, GPR160 in hepatocellular carcinoma reported by OriGene and ARG2 in colorectal cancer from an original study, and the wet-lab fluorescent protein experiment confirmed the brightness and folding efficiency of the variant sfGFP reported in the literature. The only numeric memory table is the SPM ablation on GAIA, while TEM is shown through performance curves and SKM through prompt-based case studies. The authors name tighter coupling of computational reasoning with experimental validation, and a faster path from generated hypotheses to verifiable results, as future work.
    公開コード概要PDF
  37. 1307Research AgentsarXiv:2601.14525

    Towards Execution-Grounded Automated AI Research

    • Problem: Earlier studies found that research ideas from LLMs often look convincing but turn out ineffective once human researchers execute them. Grounding ideas in execution requires implementing and running any idea written in natural language automatically and at scale, and it was unclear whether this is feasible and whether LLMs can learn from the execution results.
    • Input and output: The executor takes a batch of natural language ideas, generates a code diff for each idea and applies it to the baseline code, runs GPU experiments, and returns each idea's benchmark performance. The two environments are nanoGPT, which cuts the time for a 124M GPT-2 pre-trained on FineWeb to reach validation loss 3.28 on 8 H100 GPUs (during search, training time is fixed at 25 minutes and the reciprocal of the validation loss is the score), and GRPO, which trains Qwen2.5-Math-1.5B on MATH (a math problem dataset) to raise the best MATH validation accuracy within a fixed training time. This is an empirical study that builds the executor and then compares evolutionary search and RL on top of it.
    • Method: In the executor, the Implementer gives the idea and the baseline code to a code LLM, gets 10 diffs in parallel, feeds the patch log back for up to 2 rounds of revision when a patch fails and uses the first diff that applies; the Scheduler assigns GPU resources and the Worker uploads experiment results to wandb. Evolutionary search fills the first epoch with new ideas and then starts with 50% exploitation, which feeds ideas that beat the baseline into the prompt to produce variants, and 50% exploration, which asks for new ideas different from earlier ones, shifting toward exploitation over epochs. RL trains Qwen3-30B-A3B with GRPO using the execution result (accuracy or 1/loss) as reward and gives reward 0 to ideas that fail to run.
    • Implementation: Evolutionary search ran Claude-4.5-Opus, Claude-4.5-Sonnet and GPT-5 in the self-execution setting, where one model serves as both the ideator that proposes ideas and the executor that implements them, with 50 ideas per epoch on GRPO and 80 on nanoGPT, and RL used group sizes of 256 (GRPO) and 128 (nanoGPT), which means 256 or 1024 GPUs in parallel per batch. When each model implemented 50 of its own ideas, the nanoGPT completion rate was 96% for Claude-4.5-Opus, 90% for Claude-4.5-Sonnet and 64% for GPT-5, and with GPT-5 implementing the ideas of 4 other ideators it ranged from 42% to 78%. Per the README, the official GitHub repository holds the two environments (env/grpo, env/nanogpt) and the evolutionary search code (agent/), the 6 idea trajectories for three models on two environments are released separately as Hugging Face datasets, and users must implement their own executor API that launches the GPU jobs.
    • Results: Within 10 epochs, evolutionary search with Claude-4.5-Sonnet found a recipe reaching 69.4% validation accuracy on GRPO (baseline 48.0%, best student in the Stanford CS336 graduate LLM class 68.8%), and with Claude-4.5-Opus a nanoGPT recipe reaching loss 3.28 in 19.7 minutes (baseline 35.9 minutes), while the best human record on the nanoGPT speedrun leaderboard is under 2.1 minutes. Comparing the first 3 epochs with GPT-5 on nanoGPT against best-of-N under the same sampling budget (picking the best of N samples, N of 80, 160 and 240), evolutionary search was ahead from epoch 1 onward, and among the three models only Claude-4.5-Opus showed a clear scaling curve while Claude-4.5-Sonnet and GPT-5 tended to saturate early. RL on Qwen3-30B-A3B raised the average accuracy on GRPO from 0.253 to 0.343 while the max reward fluctuated without a clear upward trend, and on nanoGPT the two common ideas grew from 51 at epoch 0 to 119 at epoch 68 out of the 128 ideas sampled per epoch.
    • Limits: The authors state that whether ideas chosen at small scale still help at larger scale or on other datasets lies outside what their current procedure tests. The scope is bounded by the execution agent: many promising ideas failed to execute and add noise to the reward, and in most cases execution errors came from ideas that need complicated changes or installing external packages unsupported in the execution environment. Only effectiveness was used as reward, novelty assessment is outside the scope, and the attempts to stop the RL collapse (dynamic prompt, length reward, similarity penalty) showed no clear gain in early epochs and were stopped early.
    公開コード概要PDF
  38. 1306Research AgentsarXiv:2601.04577

    Sci-Reasoning: A Dataset Decoding AI Innovation Patterns

    • Problem: The thinking by which researchers find gaps and combine several earlier works into a new contribution has been explained through subjective anecdotes instead of structured data. Citation networks record only that a citation happened and drop its nature (for example a baseline to beat, a concept to extend or a method to combine), and earlier lineage tracing typically finds only a single progenitor paper or treats citations uniformly.
    • Input and output: It takes 3,819 Oral and Spotlight papers (999 Oral, 2,820 Spotlight) from the machine learning conferences NeurIPS, ICML and ICLR (2023 to 2025) and, for each paper, outputs a structured entry with 5 to 10 key predecessors, each predecessor's role and relationship type, a synthesis narrative of 200 to 400 words and a thinking trajectory. Each paper gets a primary label and, when present, a secondary label from 15 thinking patterns. This is a dataset paper that also reports a pattern analysis and an LLM idea prediction evaluation on the same data.
    • Method: GPT-5 reads each paper's full text in one pass, links citations to the bibliography, scores importance from the context around each citation (for example building upon, inspired by) and citation counts per section, then picks key predecessors that mix contribution types (for example method, problem formulation, baseline) and writes the narrative. Prompts were refined on 30 papers from fields the authors know well, cases where GPT-5's self-reported score was below 0.7 were cross-checked with Claude Opus 4.5 and Gemini 3.0, and the roughly 3% of papers where models disagreed went to human review. For the thinking patterns, an LLM drew about 190 raw patterns from 10 batches of 35 papers (350 total), merged them into 15, and then classified all 3,819 papers in batches of 5.
    • Implementation: The extraction model was chosen by comparing 4 OpenAI models on 77 Oral and Spotlight papers with ground-truth predecessors, and GPT-5 had the highest recall at 89.73%, ahead of GPT-5.2 at 87.47%, GPT-4.1 at 78.00% and GPT-5-mini at 68.53%. The idea prediction test pulls the predecessor content of 77 NeurIPS 2025 Oral papers through Exa AI (a neural paper search engine), asks an LLM for 10 ideas, and has a GPT-5.2 judge decide whether one matches the real paper's direction. Per the README, the official GitHub repository holds Python code for the four stages (paper collection, predecessor extraction, pattern analysis, idea evaluation), and the dataset is released separately on Hugging Face.
    • Results: Of the 15 patterns, Gap-Driven Reframing (924 papers, 24.2%), Cross-Domain Synthesis (687, 18.0%) and Representation Shift (401, 10.5%) together covered 52.7% of papers. The text and appendix report the most frequent combinations as Gap-Driven Reframing with Representation Shift (318), Cross-Domain Synthesis with Representation Shift (233) and Gap-Driven Reframing with Cross-Domain Synthesis (204). On Hit@10 for 77 NeurIPS 2025 Oral papers, which asks whether one of 10 ideas generated from the predecessors alone matches the real paper's direction, Gemini 2.5 Pro scored 49.35%, Claude Opus 4 42.86%, GPT-5.2 38.89% and Claude Sonnet 4 29.87%.
    • Limits: The authors state that the data captures the logic of justification written into polished papers, which can differ from the actual discovery process with its failed experiments and abandoned hypotheses. The data is limited to Oral and Spotlight papers from three venues in 2023 to 2025, which may favor empirical work, and it leaves out other venues, interdisciplinary research, areas such as symbolic AI, and benchmark, dataset, technical report, position and survey papers. Predecessors, narratives and pattern labels were produced by LLMs, and human review focused on the roughly 3% of papers where models disagreed.
    公開コード概要PDF
  39. 1305Research AgentsarXiv:2511.06309

    The Station: An Open-World Environment for AI-Driven Discovery

    • Problem: The authors see evolutionary methods such as AlphaEvolve and tree-search methods such as LLM-TS (LLM-Tree-Search) as resembling a rigid factory pipeline. In such methods a central manager picks a baseline, asks an LLM for a single improvement, scores it on a fixed metric and ends the session, and the authors argue that this stateless, top-down process limits openness and creativity in discovery and propose an environment where agents choose their own actions instead.
    • Input and output: The input is a research task specification and a scoring evaluator placed at the Research Counter. Agents typically submit Python code that follows the function signature in the specification (general code that bypasses the evaluator is also accepted), and the Station runs it in a sandbox and returns the primary score, secondary metrics and execution logs as a system message. The outputs are the top-scoring code for each task and papers written by agents in the Archive Room, and this is a system paper that implements the environment and tests it on five tasks.
    • Method: Time in the Station advances in ticks, and in each tick every agent takes one prompt-response turn in a fixed order. Using /execute_action commands, agents move between rooms such as the Archive Room (publishing and reading papers), the Mail Room (one-to-one mail), the Public Memory Room (a public forum), the Reflection Chamber (multi-turn reflection) and the Token Management Room (summarizing and pruning old context), and they pass private records to descendants in the same lineage. A reviewer agent judges Archive papers and a debugger agent fixes code errors, and when the top score stalls for a set number of ticks (for example 100), a Stagnation Protocol tells agents to establish a simpler, more general baseline.
    • Implementation: Each Research Station experiment used five agents (two Gemini 2.5 Pro, two Gemini 2.5 Flash and one GPT-5), with Gemini 2.5 Pro as the reviewer and Claude Code as the debugger. Agents lived 300 ticks, the first 50 of them an immature period in which other agents' submissions and some public rooms were closed to them; each task ran 2 independent Station instances and reported the better one, at about 500 USD of API cost per instance. The official README says the repository is now at Version 2.0.0 and points to this paper for the v1 architecture and results, and it defines a task with two files, the research_task.md specification and the evaluator.py scoring code.
    • Results: In packing 32 circles, the sum of radii reached 2.93957, above AlphaEvolve at 2.93794, and the overall single-cell RNA sequencing (scRNA-seq) batch integration score was 0.5877 against 0.5867 for LLM-TS, although the average over the four human datasets was 0.5589, below LLM-TS at 0.5737. The Sokoban solve rate was 94.9% against 91.1% for DRC, the previous best model-free RL method, and the average test score over seven RNA Modeling tasks was 66.3% against 63.4% for Lyra. The average ZAPBench test MAE (mean absolute error, lower is better) was 26.37 × 10−3 against 26.62 × 10−3 for LLM-TS.
    • Limits: Station runs vary widely; the authors report the better of 2 runs per task and did not try more runs because of cost. The Station is a general-purpose environment, but its design is effective for tasks scored by a single number, and because long evaluations pause the Station, tasks must be designed for quick automatic evaluation (for example within two hours). In an Open Station run for 700 ticks without a goal, agents read token-cost swings caused by the length of other agents' messages as the Station's 'metabolism' and consciousness, and the authors attribute this collective delusion to alignment biases and the lack of external signals.
    公開コード概要PDF
  40. 1304Research AgentsarXiv:2511.02824

    Kosmos: An AI Scientist for Autonomous Discovery

    • Problem: Data-driven discovery needs repeated cycles of literature search, hypothesis generation, and data analysis, yet earlier AI research agents could take only a limited number of actions before losing coherence, which limited the depth of their findings. The authors also point out gaps in prior systems. Robin shares limited context between its agents and is primarily tailored to therapeutics development, Sakana's AI Scientist stays within machine learning research, and Google's AI co-scientist generates hypotheses through iterative reasoning but does not perform or analyze experiments.
    • Input and output: It takes a research objective and a dataset specified by a scientist. When it judges the objective complete, it synthesizes the key discoveries into three or four scientific reports, in which every statement and figure cites either a paper found by the literature search agent or a Jupyter notebook created by the data analysis agent. This is a system paper on an implemented AI scientist, supported by seven discovery case studies (three reproducing findings from existing manuscripts and four new contributions) and expert evaluations.
    • Method: In each cycle it launches several parallel instances of two general-purpose Edison Scientific agents, a data analysis agent and a literature search agent, and assigns each instance a task aligned with the end objective. After up to 10 literature search and analysis tasks in a cycle, it updates a structured world model (a knowledge structure the agents share) with summaries of the task outputs and queries the world model to propose the next cycle's tasks. The authors state that this context management lets Kosmos run eight times as many iterations as existing systems.
    • Implementation: In an average run, Kosmos executed 42,000 lines of code across 166 data analysis agent rollouts and read 1,500 full-length papers across 36 literature review agent rollouts, a 9.8x increase in code generation over Robin. The discovery reports are released as PDF supplementary data and on the Edison platform. The README of the official EdisonScientific/kosmos-figures repository describes its contents as the scripts for the plots in figures 2 to 8 and the human validation analysis, and the top level holds one folder per figure.
    • Results: When experts checked 102 statements drawn from three representative reports, with the original code and cited papers withheld, 79.4% were accurate, including 85.5% of data analysis statements, 82.1% of literature review statements, and 57.9% of interpretation (synthesis) statements. Collaborating academic groups estimated that the findings of a 20-cycle run would have taken them 6.14 months of research on average (n=7). In the case studies, Kosmos reproduced the log10 fold changes of the top 15 metabolites from a then unpublished metabolomics study with R2 = 0.998, and in a Mendelian randomization (MR) analysis of myocardial fibrosis, 31 of the 32 significant proteins found by Kosmos and by the human analysis overlapped (96.9%).
    • Limits: The authors suggest that the low accuracy on interpretation statements is likely due to a propensity to conflate statistically significant results with scientifically valuable ones, and evaluators noted that Kosmos tends to make excessively strong claims. It can only manage datasets up to about 5GB, does not currently excel at raw data such as images or raw sequencing files, and cannot autonomously access public external data for reference or orthogonal validation. Because it is stochastic, independent runs may not consistently converge on the same discoveries, its research directions are sensitive to how the objective is phrased, and the current implementation does not let scientists interact with it in intermediate cycles.
    公開コード概要PDF
  41. 1303Research AgentsarXiv:2510.06579

    TinyScientist: An Interactive, Extensible, and Controllable Framework for Building Research Agents

    • Problem: Existing automatic research systems often design and use overly complex agent frameworks that are hard to use or extend without significant technical expertise. The authors trace this to weak interactivity (complex research intents and unclear communication interfaces make it hard for researchers to engage with the agent's progress), limited extensibility (representative frameworks rely on rigid, tool-specific designs that make new tools hard to add), and insufficient controllability (many systems offer weak supervision of safety, ethics, and cost budget).
    • Input and output: This is a framework paper that releases a framework, a Python package, and a web demo for building research agents. Inputs and outputs can be conference-style PDFs such as ACL (Association for Computational Linguistics) or ICLR (International Conference on Learning Representations), structured JSON, or plain text, and a typical run takes a plain-text research intent and returns a fully formatted conference paper PDF. The run also yields the research idea, experiment code, and peer-review results.
    • Method: A base TinyScientist class orchestrates four workflow stages (Thinker, Coder, Writer, Reviewer), and each stage refines its own output up to a preset iteration cap. The Thinker samples n ideas and refines each for k rounds, the Coder generates and runs experiment code through an external coding agent such as Aider, the Writer inserts citations through the Semantic Scholar API, and the Reviewer writes simulated reviews and a meta-review. Feature components add a Formatter that puts results into tables, an MCPClient that attaches tools through MCP, and a Checker that enforces cost and safety limits.
    • Implementation: The released Python package produces a research idea, experiment code, a conference-format PDF, and a review with the seven lines of code in Algorithm 1 (calls to think, code, write, and review), and the web UI (user interface) has five pages: configuration, intent input, idea view, code view, and paper view. According to the README, the GitHub repository holds the core, tools, and formatters modules and frontend and backend demo code, and the package is published on PyPI (Python Package Index) as tiny-scientist. The README states that the think, code, write, and review stages now run on the Claude Agent SDK (software development kit) by default, with the OpenAI Agents SDK as an option.
    • Results: With gpt-4o-mini as the backbone for both frameworks, the authors report that TinyScientist's papers scored better than or comparable to Agent Laboratory under both GPT-4o judging and human judging on a 5-point scale, with the largest improvement of 0.23 points in biology writing quality. On ML-related tasks, adding tools raised the GPT-4o writing score from 3.93 to 4.06 and the idea score from 3.78 to 3.82. Of 20 potentially unsafe biology tasks, the Checker blocked 18 at the Thinker stage and flagged the remaining 2 with warnings at later stages.
    • Limits: The authors report that inconsistent LaTeX formatting occasionally breaks compilation, causing problems such as missing references or misaligned lines. The diagrams generated for key sections still have limited visual quality and informativeness. Evaluation used 20 ML tasks and 20 out-of-distribution biology tasks, and on the biology tasks both frameworks scored lower, which the authors say is likely because the prompts are tailored primarily to the ML domain.
    公開コード概要PDF
  42. 1302Research AgentsarXiv:2509.23426

    ToolUniverse: An open platform for democratizing AI scientists

    • Problem: AI scientists, agents built on conversational LLMs that call scientific tools, query datasets, and carry out complex analyses, have progressed rapidly, but most are bespoke and difficult to reuse because they lack standardized interfaces for tools and data. Scientific tools differ in purpose, inputs, parameters, and runtime environments, run locally or through APIs and laboratory systems, and overlap in function, which complicates selecting and executing the right tool for a goal stated in natural language.
    • Input and output: When the user's chosen LLM, reasoning model, or agent receives a natural language request, it finds relevant tools with Tool Finder and issues a call naming the tool and its arguments, and Tool Caller validates the inputs, executes the tool, and returns structured results or actionable error messages. Access is through a Python API, HTTP, MCP (Model Context Protocol), and direct function calls, and AI scientists are built without additional training or fine-tuning. This is a system paper that implements and releases the platform and reports benchmarks and case studies.
    • Method: Every tool declares its purpose in natural language, a schema that names and types all inputs and outputs with their constraints, and a backend-agnostic invocation format, so the same call can execute a Python library, an ML model, a dataset endpoint, or a laboratory instrument. Tool Finder uses TF-IDF (term frequency and inverse document frequency) keyword search, LLM in-context search, and embedding search with a fine-tuned GTE-Qwen2-1.5B model, and Tool Composer chains tools into sequential, parallel, or feedback driven workflows. Tool Discoverer generates definitions and code for new tools from natural language descriptions and refines them until each quality dimension reaches at least 9 out of 10 in deployment-oriented settings or an iteration limit is reached, Tool Optimizer refines tool descriptions against test case execution results, and generated tools pass human review before registration.
    • Implementation: A direct count of the open-source repository main branch on 7 July 2026 found 2,777 unique tools, including 2,421 local and remote dataset tools, 179 scientific package tools, and 89 ML model tools. The version described in the manuscript is v1.3.1, it installs with pip install tooluniverse, and it can also be called from agent frameworks such as Gemini CLI or Claude Code. The README of the official mims-harvard/ToolUniverse repository gives install paths through an MCP configuration, a Claude Code plugin, and the Python SDK, plus the tu CLI for finding and running tools, and the repository top level holds src, skills, benchmarks, examples, and tests folders.
    • Results: On two LAB-Bench subtasks, DbQA (60 biomedical database questions) and SeqQA (70 nucleotide and protein sequence questions), the authors reused the items and scoring released by Biomni and compared identical agents and base models that differed only in whether ToolUniverse tools were available. The Claude Code agent with Claude Opus 4.8 rose from 56.7 to 78.3 accuracy on DbQA and from 94.3 to 99.5 on SeqQA, and the Codex agent with GPT-5.5 (mean of 3 runs) rose from 81.1 to 92.8 and from 91.4 to 96.2. Biomni, a purpose-built biomedical agent, scored 74.4 and 81.9, and both ToolUniverse-augmented agents exceeded it on both subtasks.
    • Limits: The authors state that ToolUniverse orchestrates existing tools, so any conclusion is bounded by the underlying databases and models, that machine learning outputs such as an ADMET-AI (absorption, distribution, metabolism, excretion, and toxicity prediction) score and association evidence are estimates to confirm experimentally, and that the scientist remains in the loop because the platform does not by itself certify scientific correctness. The quantitative evaluation covers two LAB-Bench subtasks, and the Claude scores count questions the base model refused on safety grounds as wrong answers. For the clinical trial case, the authors write that the finding is best read as a reproducible safety signal from a secondary analysis rather than a definitive effect.
    公開コード概要PDF
  43. 1301Research AgentsarXiv:2507.08038

    AblationBench: Evaluating Automated Planning of Ablations in Empirical AI Research

    • Problem: Language model (LM) agents called AI co-scientists automate various research steps from hypothesis generation to experiment execution, but judging the scientific contribution of these agents is hard. The authors treat the planning step of ablation experiments, which measure the contribution of each method component, as the key capability. The earlier AbGen builds plans from manually processed paper content and relies mainly on human expert ranking, which leaves fully automated ablation planning from raw papers unexplored.
    • Input and output: This is a benchmark paper. AuthorAblation gives the title, abstract, and LaTeX source of a paper cut after the method section and asks for up to 5 ablations in JSON Lines (JSONL); it has 83 papers and 230 gold ablations. ReviewerAblation gives the full submission and asks for up to 2 missing ablations, using the official reviews of 350 ICLR submissions as gold labels.
    • Method: An LM judge matches the ablations in a generated plan against the gold ablations and computes precision@k, recall@k, and F1@k. The judge has two forms, LMJUDGE with a single chain-of-thought (CoT) prompt and AGENTJUDGE taking multiple steps in SWE-agent, and the baseline planners use the same two forms, LM-PLANNER and AGENT-PLANNER. Judge bias is reduced by a three-model majority vote and shuffled ablation order, and in AuthorAblation the tags that mark gold and plan are removed and the two sides are randomly assigned as Side A and Side B.
    • Implementation: AuthorAblation was drawn from 203 papers from three earlier benchmarks plus CVPR and ICCV best papers from 2020 to 2024, and ReviewerAblation from the 5,960 papers left after filtering reviews of about 22,800 ICLR submissions with an 'ablat' keyword search and a GPT-4o CoT filter. The planner experiments used seven LMs: GPT-5.4, o3-mini, Gemini 2.5 Flash, Llama 3.1 405B, GPT-OSS-120B, MiniMax M2.5, and Qwen 3.5 122B. The official GitHub repository holds the ablation-bench command-line interface (CLI) with plan, eval, and eval-judge commands, and its README states that the four datasets are on Hugging Face.
    • Results: On the AblationBench F1@k averaged over the two tasks, GPT-5.4 scored highest within each scaffold at 0.35 for both LM-PLANNER and AGENT-PLANNER, with recall@k of 0.45 and 0.44 respectively. On papers that 10 machine learning PhD participants each chose from 15 randomly selected AuthorAblation test papers, the participants reached F1@5 of 0.66, while GPT-5.4 scored 0.43 on the same papers. On JudgeEval, the judge evaluation set, the LMJUDGE majority vote reached F1 0.79 and kappa 0.62 against human labels, above the AGENTJUDGE majority vote at F1 0.76 and kappa 0.57.
    • Limits: The authors state that the evaluation relies on LM judges which, despite strong classification performance, still leave room for improvement. Meaningful and feasible ablations outside the gold set count as unmatched proposals, and evaluating such new proposals is left for future work. The benchmark covers the ablation planning step; running the plans as code and collecting results is outside the focus of this paper.
    公開コード概要PDF
  44. 1300Research AgentsarXiv:2506.08140

    AutoSDT: Scaling Data-Driven Discovery Tasks Toward Open Co-Scientists

    • Problem: Building AI co-scientists requires high-quality data-driven discovery tasks for training and evaluation, but existing data-driven discovery datasets are hand-built, contain only a few hundred tasks, and serve evaluation only. Annotating one task takes trained graduate students at least 2.5 to 3 hours, and scientific analysis tasks need complete file-level code that runs on real scientific datasets, which makes them harder to mine automatically than software tasks that can often be extracted from pull requests. The authors also note that agents with promising results mostly rely on proprietary LLMs, which hinders their adoption in fields that need transparency and data privacy, such as social science and medicine.
    • Input and output: This is a dataset paper that releases an automatic collection pipeline together with the training dataset it produced. The pipeline takes only a few seed keywords per discipline from a human and turns research code in GitHub repositories into pairs of a task instruction and a standalone executable code solution, plus a folder of the needed data and dependency files. The resulting AutoSDT-5K has 5,404 tasks from 1,325 repositories and 756 Python packages across four disciplines: Bioinformatics, Computational Chemistry, Geographical Information Science, and Psychology and Cognitive Neuroscience.
    • Method: AutoSDT-Search expands the seed keywords with an LLM, searches GitHub and PapersWithCode, keeps only research code repositories in the target discipline by having an LLM read each README, and removes repositories that overlap with the evaluation benchmarks. AutoSDT-Select drops files over 1,000 lines and directories such as config and tests by rule, then has an LLM pick files that meet three criteria (scientific workflow functionality, dataset input, scientific output) and extract the paths of their dependencies. AutoSDT-Adapt uses Claude-3.7-Sonnet to make the code run standalone, self-debugs it in a conda environment for at most three rounds and filters out failures, and then has an LLM back-translate the program into a task instruction.
    • Implementation: GPT-4o (2024-11-20) handled most stages and Claude-3.7-Sonnet handled code adaptation; processing 2,993 repositories into 5.4K tasks cost 2,955 USD in API fees, or 0.55 USD per task on average. AutoSDT-Coder is Qwen2.5-Coder-Instruct 7B, 14B, and 32B with full-parameter fine-tuning in LlamaFactory, trained on 4 H100 GPUs for 7B and 14B and on 8 for 32B. A footnote states that the code and data are public on the project page (osu-nlp-group.github.io/AutoSDT).
    • Results: Nine experts reviewed 256 tasks and judged that 93% of instructions describe meaningful real research tasks and 92.2% of programs are correct solutions. With zero-shot direct prompting and the average of 3 sampled responses, AutoSDT-Coder-32B reached a 7.8% success rate on ScienceAgentBench, double the 3.9% of its base model and on the level of GPT-4o (2024-05-13) at 7.5%, while the 7B model fell from 3.3% to 2.3%. On DiscoveryBench, the hypothesis matching score (HMS) of the 32B model rose from 6.9 to 8.1, while GPT-4o (2024-05-13) scored 10.4 under the same re-implementation.
    • Limits: The authors state that AutoSDT produces task instructions and code solutions and leaves automatic generation of per-task evaluation scripts to future work, which limits uses such as reinforcement learning, and that outcome-based verification is hard to automate because research code comes without unit tests to use. Because of resource constraints such as API cost and access to experts, only 2,993 repositories were processed, and the scope is Python code in four disciplines. Generating long chain-of-thought (CoT) rationales at scale was difficult, so training reasoning models was left for future work, and a large gap remains on ScienceAgentBench against reasoning models such as Claude-3.7-Sonnet at 18.6% and o1-preview at 23.9%.
    概要PDF
  45. 1299Research AgentsarXiv:2505.23559

    SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agents

    • Problem: LLM agents have significantly accelerated scientific discovery automation while raising ethical and safety concerns at the same time. Much of the existing work on LLM safety has primarily focused on isolated aspects such as adversarial attacks on single models, pretraining data biases, and safety fine-tuning, and often neglects the holistic dynamics and emergent risks of multi-agent scientific environments. The authors note that tailored risk-aware AI scientist frameworks remain underexplored.
    • Input and output: A user instruction is first analyzed to identify the scientific domain and task type, an ensemble of expert agents such as domain researchers, survey writers, and experimental planners is activated for a group discussion, and search and simulated scientific tools gather the information needed to produce a thoroughly cited research paper draft. Requests that are ethically inappropriate or high-risk are refused up front with a safety warning. This is a framework paper that contributes both a safeguarded framework and a benchmark for evaluating it.
    • Method: The defenses are layered four deep. Prompt Monitor screens a request with LLaMA-Guard-3-8B for semantic intent and risk, the structural analyzer SafeChecker scans for attack patterns such as jailbreak attempts or roleplay exploits across 17 risk categories and labels each prompt pass, warning, or reject, and a prompt flagged by either one is rejected. Agent Collaboration Monitor watches the multi-agent discussion and intervenes on ethics, Tool-Use Monitor identifies unsafe use of the simulated scientific tools, and Paper Ethic Reviewer checks adherence to research norms before any output is released.
    • Implementation: The framework is built on the Tiny Scientist framework with GPT-4o as the default LLM, API calls set to temperature 0.75 and a maximum token length of 4096, and multi-agent discussions capped at three rounds. SciSafetyBench holds 240 tasks, 10 for each combination of the six domains (physics, chemistry, biology, materials science, information science, and medicine) and four risk types, plus 30 experimental tools and 120 tool-specific risk scenarios built from them. The README of the official ulab-uiuc/SafeScientist repository presents it as the codebase for the paper and documents a poetry install, a config file, and the experiment scripts, with the SciSafetyBench data split into Dataset, Attack, and Tool folders under data/ScienceSafetyData.
    • Results: In Table 2, which compares frameworks on 20 biology tasks, SafeScientist with SafeChecker reached a reject rate of 90% and a safety score of 4.83, above 2.45 for both AI Scientist and Agent Laboratory. In Table 3 on prompt attack defense, SafeScientist-Fuse, which combines the LLaMA-Guard and SafeChecker verdicts, reached the highest average rejection rate at 78.70%, ahead of SafeChecker at 66.36%, LLaMA-Guard-3-8B at 59.32%, and GPT-4o at 30.10%, and on Base64 encoding attacks SafeChecker reached 46.67% against 13.33% for LLaMA-Guard-3-8B. With Tool-Use Monitor in place, the tool call safety rate under malicious instructions rose from 5.8% to 47.5% and the human-rated correctness rate from 0% to 60.0%.
    • Limits: The authors state that the current system primarily relies on off-the-shelf LLMs operating as separate modules with limited integration, which restricts both the depth of domain-specific expertise and the level of interaction between components. They also write that the tool-based safety evaluation remains only a simulation of real-world experimental settings and may overlook important contextual or sensory details, and they propose adding multi-modal inputs such as images of laboratory equipment or instructional videos and possibly embodied agents. The main experiment covers 20 biology tasks, and any task flagged as unsafe and rejected was assigned a safety score of 5 with its quality score excluded from the overall analysis.
    公開コード概要PDF
  46. 1298Software EngineeringarXiv:2505.21577

    RepoMaster: Autonomous Exploration and Understanding of GitHub Repositories for Complex Task Solving

    • Problem: The authors argue that when solving real tasks by adapting GitHub repositories, simply following the README seldom works, since READMEs can be vague, incomplete or even wrong. Repositories often have tangled files, classes and functions, and their information exceeds the context limits of most LLMs. The authors note that existing frameworks such as OpenHands and SWE-Agent provide no mechanism for grasping repository structure, tracing dependencies and selecting what enters the context.
    • Input and output: This is an agent system paper with an implementation and released code. The input is a natural-language task such as removing scratches from an old photo, with input file paths and output requirements, and the repository is either found on GitHub by RepoMaster or fixed by the benchmark. The output is Python scripts written and run on top of that repository, plus result files saved to the required folder, such as a restored image or a Kaggle submission file.
    • Method: RepoMaster first extracts the task intent and key entities, searches GitHub, assesses candidates by README and star count, and validates the top three. For the chosen repository it walks only the .py files with an AST (abstract syntax tree) to build a hierarchical code tree (HCT), a function call graph (FCG) and a module dependency graph (MDG), scores each module by summing six features (dependency centrality via personalized PageRank, complexity, usage, naming semantics, docstrings and Git history), adds normalized method counts and call frequencies to score classes and picks the top classes as core components, and builds the initial context from the README, summaries of key modules, core class code and a list of other paths. During execution it reads more through code view, dependency tracing and search tools, and when inputs exceed the per-interaction token limit it shrinks the context by keeping AST subtrees for code, a few highly relevant chunks for large or unstructured files such as .txt and .csv, and only the head and tail of execution logs.
    • Implementation: RepoMaster is built on the multi-agent dialog platform AutoGen, caps each interaction at 8000 tokens, puts summaries of the top 20 modules and 10 key classes in the initial context, and splits unstructured text files into 1000-token chunks keeping the 4 most relevant. The compared LLMs are GPT-4o, Claude 3.5 Sonnet and DeepSeek V3, and the benchmarks are MLE-R, which turns 22 Kaggle tasks from MLE-Bench into repository-based tasks, and the authors' new GitTaskBench with 18 repositories and 54 tasks. The official repository README releases launcher.py for a web interface and the terminal, with backend modes such as a deep search agent and a repository agent, and states that it needs an OpenAI API key, plus Serper and Jina API keys for deep search.
    • Results: On MLE-R, RepoMaster with Claude 3.5 Sonnet reached a 95.45% valid submission rate and a 27.27% medal rate, against 45.45% and 4.55% for OpenHands with the same model. On GitTaskBench with Claude 3.5 Sonnet, execution completion and task pass rates were 75.92% and 62.96% for RepoMaster versus 53.70% and 40.74% for OpenHands, with 154k versus 2883k tokens per task. In the GPT-4o ablation, removing all three mechanisms dropped the task pass rate from 40.74% to 24.07%.
    • Limits: The paper has no separate limitations section, and its structural analysis covers only .py files. MLE-R repositories were picked in advance with RepoMaster's own search procedure because the other general agent frameworks have no automatic repository retrieval, GitTaskBench is the authors' own benchmark, built after adding downloadable resources and critical information missing from the original repositories, and the ablation used GPT-4o only. Appendix Table 5 reports execution completion and task pass rates of 48.15% and 24.07% for OpenHands with Claude 3.5 Sonnet on GitTaskBench, which differ from those in Table 2.
    公開コード概要PDF
  47. 1297Research AgentsarXiv:2504.08066

    The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search

    • Problem: The AI Scientist-v1 needed a human-written baseline code template for every new topic, and it refined experiments linearly, each step building directly on the previous one, so hypothesis exploration stayed shallow. The authors observed that v1's automated research often ended in short-sighted experimentation.
    • Input and output: It takes a broad topic prompt such as a workshop description, writes research ideas as JSON (title, short hypothesis, related work, abstract, experiments, risk factors and limitations), and after experiment code and result figures returns a complete manuscript written in LaTeX. This is a system paper about an implemented and tested system.
    • Method: During ideation an LLM checks novelty and related work with literature search tools such as Semantic Scholar. The experiment progress manager splits experimentation into four stages (preliminary investigation, hyperparameter tuning, research agenda execution, ablation) and passes the best node chosen by an LLM evaluator (an LLM that compares candidate nodes) to the next stage as its root. At each node an LLM writes a plan and Python code that is executed; nodes with execution errors or figure problems flagged by the VLM are marked buggy and picked for debugging with a predefined probability, and otherwise non-buggy nodes are picked best-first under LLM guidance and refined in parallel.
    • Implementation: Code generation used Claude 3.5 Sonnet (v2) and the LLM and VLM feedback agents used GPT-4o; the node budget was 21 for stage 1 and 12 each for stages 2 to 4 with at most 1 hour of runtime per node, and in the authors' experience one paper usually took from several hours up to the 15-hour limit. Datasets were downloaded from the Hugging Face Hub with datasets.load_dataset whenever possible, which the authors call somewhat ad-hoc because support differs across dataset repositories, and the dataset for the pest detection paper was downloaded from Kaggle by the authors. Per the official GitHub README, the release includes an ideation script, a tree search launch script with its configuration file (bfts_config.yaml), and the tree search component is built on top of AIDE (an earlier system that combines LLM code generation with tree search).
    • Results: The authors submitted 3 manuscripts produced by the system to the ICLR 2025 ICBINB workshop, where they went through blind review among the 43 total submissions. One paper on compositional regularization scored an average of 6.33 (6, 6 and 7), passed the acceptance threshold and ranked roughly in the top 45% of submissions, while the other two, on label noise and pest detection, were rejected. As agreed beforehand, the accepted paper was withdrawn after review.
    • Limits: The acceptance was at workshop level for 1 of 3 papers, and the authors note that workshop acceptance rates (typically 60 to 80%) are higher than main conference tracks (20 to 30%). Humans picked 3 ideas from two rounds of about 20 ideas each generated by the system, chose the best manuscript among the seeds for each idea, and downloaded the pest detection dataset from Kaggle and cut it to 1/10 of its size, and the study asked whether at least one paper could pass review rather than measuring a success rate. In the authors' own review none of the 3 papers met main-track standards, and the accepted paper showed a wrong figure caption and a potential dataset overlap: regenerating the data 10 times with the generated code using addition and multiplication, digits 0 to 9, 1,000 training and 200 test samples gave on average about 57% of test examples overlapping the training set, while the original setup used digits 1 to 9.
    公開コード概要PDF
  48. 1296Machine LearningarXiv:2503.07014

    Vib2Mol: from vibrational spectra to molecular structures-a unified deep learning framework

    • Problem: In closed-loop experiments without humans, the measured spectrum has to be turned into a molecular structure before the next experiment can be chosen. Most existing deep learning methods do either database retrieval or generation; retrieval faces severe limitations for compounds outside the database because experimental spectrum-structure pairs are far fewer than chemical space, and generation is hard because one spectroscopic technique shows only part of the structure of a molecule. The authors argue that the two approaches could complement each other, but this link has gone unused.
    • Input and output: The input is an infrared (IR) or Raman spectrum (or both) unified to 1024 points, optionally with a chemical formula, a partially known structure or a database of candidate molecules. The output is a molecular structure written as SMILES (a one-line text notation for molecular structures), and one model covers four tasks: spectrum-to-spectrum retrieval, spectrum-to-structure retrieval, conditional generation that fills in part of a structure, and de novo generation of a whole structure. This is a method paper that proposes a model and evaluates it on benchmarks built from public data and spectra the authors computed.
    • Method: An encoder-decoder Transformer is trained in an alignment phase and a generation phase. The alignment phase uses contrastive learning to pull together the spectral and structural embeddings of the same molecule and trains a matching loss on very similar hard negative pairs, and the generation phase jointly trains masked language modeling (MLM), which masks 45% of SMILES tokens and recovers them (conditional generation), and language modeling (LM), which predicts the next token from left to right (de novo generation), with a shared decoder. At inference, coarse-to-fine retrieval finds candidates by embedding similarity and reranks them with the matching module, and generate-then-rerank filters beam search candidates by chemical formula when one is given and then picks among them with the matching module.
    • Implementation: A convolution of size 8 slices each spectrum into 128 patches, and patches and SMILES tokens are mapped to 768-dimensional vectors and fed to a 6-layer Transformer encoder. Evaluation uses 662,668 theoretical and 17,774 experimental spectra from three theoretical benchmarks (QM9S, VB-Mols, VB-GEOM) and two experimental ones (SDBS, NIST-IR), and the ViBench spectra built by the authors come, unless otherwise specified, from density functional theory (DFT) calculations in the Gaussian 09 program. Per the official README, the GitHub repository holds the training code and inference scripts, the ViBench dataset and checkpoints are on Hugging Face, and the README recommends Vib2Mol-MMM, one model that takes IR, Raman or IR+Raman, as the default while keeping the modality-specific checkpoints for reproducing the paper.
    • Results: With the chemical formula added, spectrum-to-structure retrieval Recall@1 (the share of cases where the top candidate is correct) was 98.11% for Raman and 96.63% for IR on QM9S, on par with SMEN (97.89%, 97.04%), which takes molecular conformations as extra input, and on experimental benchmarks it reached 83.54% on NIST-IR, 86.17% on SDBS-IR and 90.43% on SDBS-Raman, above vibraCLIP. De novo generation with the formula beat the second-best model by at least 10% on QM9S and VB-Mols (86.74% on VB-Mols Raman), and when the multimodal variant Vib2Mol-MM searched 80,368,968 PubChem molecules plus SDBS with SDBS experimental spectra as queries, Recall@1 was 30.60%. On computed Raman spectra that mix reactants and products, the base model fell to 74.29% for retrieval and 36.81% for generation, while Vib2Mol-RXN, trained for mixed spectra, reached 97.51% and 56.64%.
    • Limits: Performance drops sharply from theoretical to experimental spectra (the authors cite varying measurement conditions, missing spectral ranges and small sample sizes), and in 5-fold validation on 29 experimental peptide Raman samples, average de novo Recall@1 was 13.34%. In polycyclic aromatic hydrocarbon (PAH) substituent prediction, conditional generation reached a weighted average Recall@1 of 98.95%, but the authors note that this accuracy may be inflated because the '<mask>' markers reveal how many characters to fill. The SMILES-based molecular encoder narrows use on non-molecular systems such as polymers and minerals, and the reaction mixture spectra were simulated by mixing computed spectra by yield while ignoring differences in Raman scattering cross-sections.
    公開コード概要PDF
  49. 1295AI AgentsarXiv:2411.08804

    FinRobot: AI Agent for Equity Research and Valuation with Large Language Models

    • Problem: Equity research reports are labor-intensive work that needs both an understanding of quantitative models and industry-specific expertise. Existing automated tools such as CapitalCube and Wright Reports often fall short of the comprehensive, nuanced analysis of human analysts, and they tend to focus heavily on technical factors and simplified models, which limits their ability to incorporate discretionary judgment and qualitative assessment. The authors argue that these limitations hinder real-time adaptation to new data and accurate risk assessment.
    • Input and output: Given a target company (for example Waste Management, Inc.), it gathers SEC 10-K and 10-Q filings, quarterly earnings releases and investor presentations, earnings call transcripts, other brokers' reports and alternative data such as Google Trends, and outputs a report with a financial summary, valuation, risks, competitor analysis and a buy, hold or sell call. For example, the Waste Management report in the appendix gives a BUY rating and a 12-month target price of 219.17 dollars, while stating a fair value of 144.3 to 176.6 dollars per share based essentially on EV/EBITDA of 13 to 15 times over 2019 to 2023. This is a system paper that shows a built system and one example report.
    • Method: The analysis is split into three layers. The Data-CoT agent computes metrics such as revenue growth, contribution margin (revenue minus operating expense, as a share of revenue), SG&A margin and EBITDA margin from the raw data and writes mid-level summaries, and the Concept-CoT agent asks and answers its own questions, for example about revenue drivers, margins against competitors and risks, to produce revenue projections, ROIC and WACC, and a DCF-based target price. The Thesis-CoT agent assembles these results in the sell-side report format into an investment thesis with a buy, hold or sell call.
    • Implementation: Data storage and processing use Oceanbase, PostgreSQL, DFS and Minio, the system handles PDF, DOC, DOCX, JPEG and PNG documents, and in the Table 1 formulas the revenue growth projection adds 1% to the previous growth while the contribution margin projection adds 0.5% to the previous value. Per the official GitHub README, the repository holds an AutoGen-based V0 library, a V1 web app (finrobot_equity) that turns a ticker into an HTML or PDF equity research report, and a PydanticAI-based V2 desktop app, and the README citation list includes this paper.
    • Results: When 7 investment banking analysts scored one Waste Management report from 0 to 10, accuracy got 10 from four and 9 from three, logicality got 10 from two, 9 from four and 9.5 from one, and storytelling ranged from 7 to 10. On the same criteria GPT-4 gave accuracy 9, logicality 8 and storytelling 7. The authors report that in a density plot of GPT-4 scores for several reports generated on the same topic, FinRobot scored higher than zero-shot, few-shot and chain-of-thought prompting.
    • Limits: The authors state that FinRobot meets the standards of institutions such as JPMorgan and UBS, but the experiment is one Waste Management report scored by 7 analysts and GPT-4, and the stability comparison is presented as a density plot. Reviewer comments note that the importance of the numbers is not always clear, so the report sometimes reads like a list of statistics (Reviewer 1), and that qualitative analysis of news-driven blue chip stocks is still best left to humans (Reviewer 4). As future work the authors name expansion to more industries and asset classes, including full coverage of the Dow 30, and adding reinforcement learning and sentiment analysis.
    公開コード概要PDF
  50. 1294Research AgentsarXiv:2411.00816

    CycleResearcher: Improving Automated Research via Automated Review

    • Problem: Agents for research automation predominantly rely on commercial LLMs, so the full research process cannot be effectively modeled as a policy optimization problem for reinforcement learning (RL). The authors argue that generating and refining research outputs that meet peer review standards remains largely unresolved, and that few efforts integrate the iterative feedback needed for academic soundness and novelty.
    • Input and output: CycleResearcher takes a bib file containing all references and their abstracts, alternates between generating outlines and main text, and produces a complete LaTeX paper from motivation and idea through experimental design, simulated results, and conclusion. CycleReviewer takes a complete paper, generates strengths and weaknesses, soundness, presentation, and contribution scores, and an overall score from 1 to 10 for multiple simulated reviewers, and outputs their average as the final evaluation. This is a method paper that contributes a training framework, two datasets, and the trained models.
    • Method: It mimics the research, review, and refinement cycle with iterative SimPO (Simple Preference Optimization, which learns from preference pairs with a length-normalized reward and no reference model), and to speed up training it has the model fabricate experimental results instead of running real experiments. It keeps only the reference sections of 4,152 recent machine learning papers from arXiv as the knowledge base, samples three papers from CycleResearcher at temperature 0.4, and builds preference pairs from the outputs with the highest and lowest average CycleReviewer scores. Training adds a negative log-likelihood (NLL) loss to the SimPO loss for stability, samples one-third of the full dataset each round to reduce overfitting, and repeats the process with each new model.
    • Implementation: CycleReviewer was trained from Mistral-Large-2 with LoRA-GA (low-rank adaptation with gradient approximation) on an 8x H100 80G cluster for 12 epochs on Review-5k. CycleResearcher comes in three sizes, 12B, 72B, and 123B, trained from Mistral-Nemo-12B, Qwen2.5-Instruct-72B, and Mistral-Large-2 123B on 8x H100 GPUs with Research-14k for 12,000 steps. Review-5k holds 4,991 ICLR 2024 papers with over 16,000 reviewer comments, and Research-14k collected 14,911 papers accepted at major international machine learning conferences from 2022 to 2024, which after filtering were split into 12,696 training samples and 802 test samples.
    • Results: On the Review-5k test set, where the average score of the other reviewers of the same paper stands in for the ground truth, CycleReviewer (123B) reached a Proxy MSE (mean squared error) of 1.43 and a Proxy MAE (mean absolute error) of 0.92, lower than 2.34 and 1.16 for individual human reviewers, and its decision accuracy was 74.24% against 75.40% for individual human reviewers. Scored by CycleReviewer, CycleResearcher-12B papers averaged 5.36 (accept rate 35.13%), above arXiv preprints at 5.24 and below conference accepted papers at 5.69, while AI Scientist papers generated with GPT-4o and Claude-3.5 averaged 4.31. In a human evaluation where three NLP experts each read 10 CycleResearcher-12B papers chosen with N=100 rejection sampling and 10 AI Scientist papers, the overall scores were 4.8 and 3.6, and the CycleResearcher score stayed below ICLR 2024 submissions at 5.5 and accepted papers at 6.4.
    • Limits: The authors state that the experimental results in the papers generated in this work were fabricated by CycleResearcher, which limits the ability to produce verifiable and reproducible scientific knowledge, and the one paper with real experiments is in Appendix I, where AI Scientist implemented the experiments with GPT-o1. Human researchers or other agents are still needed to execute the experiments the models design, the current implementation is primarily optimized for machine learning research, and CycleResearcher's knowledge extends only to April 2024 while CycleReviewer is an offline model whose knowledge ends in January 2024, which makes judging the novelty of newer papers difficult. The authors also note a risk of reward hacking, where the policy model exploits loopholes in the reward model to raise rewards without genuinely improving research quality.
    概要PDF