FML-bench: Benchmarking Machine Learning Agents for Scientific Research

Key points

• Problem: Existing ML agent benchmarks mostly use Kaggle-style, application-oriented tasks that stress engineering execution such as feature engineering and standard model training, and they primarily assess final performance and computational cost. They pay limited attention to the ability to tackle fundamental ML research problems such as representation learning and generalization, and they limit analysis of how an iterative research process relates to its outcomes.
• Input and output: This is a benchmark paper. The agent receives a task description, the complete code of a real research repository, files suggested for editing, protected evaluation code it cannot modify, a command list for running experiments, the baseline performance, and the target metrics, and through repeated cycles of proposing a hypothesis, editing code, and running the commands it is expected to propose a new or improved method with stronger empirical results than the baseline. A post-processing module converts each repository output into a standard format, and the Performance score is the best held-out test result across iterations.
• Method: The datasets and baselines of the eight tasks are ColoredMNIST and ERM (Empirical Risk Minimization) for generalization, Mini-ImageNet and Prototypical Networks for data efficiency, CIFAR-10 and MoCo for representation learning, splitMNIST and Synaptic Intelligence for continual learning, IHDP (Infant Health and Development Program) and DragonNet for causality, MNIST poisoned with backdoor attacks and dp-instahide for robustness, CIFAR-10 and Wide-ResNet-28-2 for privacy, and COMPAS (a recidivism prediction dataset) and Adversarial Debiasing for fairness, all built on real research repositories such as DomainBed, Lightly, and AIF360. Besides performance and cost, the paper proposes Exploration Diversity, the spread of the GraphCodeBERT embeddings of the code from each iteration around their centroid, Step Success Rate, the share of executed iterations that yield valid results without errors, and Step Completion Rate, the share of assigned iterations actually executed.
• Implementation: The paper compares three agents with different search strategies: TheAIScientist (broad parallel exploration), AIDE (tree search, run through its commercial variant Weco), and Claude Code (linear refinement instructed by a prompt). TheAIScientist and AIDE used GPT-5 and Gemini-2.5-Pro, Claude Code used Opus-4.1, and each agent ran three independent rounds with a budget of 100 steps, keeping the best test-set result. The README of the official qrzou/FML-bench repository states that the code for the version in this paper lives on the legacy branch, while the default branch README describes a newer version with 18 tasks and 7 agents.
• Results: TheAIScientist with Gemini-2.5-Pro performed best, ranking first in 4 of the 8 tasks, and AIDE with Gemini-2.5-Pro ranked first in 2. Mean Exploration Diversity was 24.66 for TheAIScientist, 23.50 for AIDE, and 8.75 for Claude Code, and diversity correlated positively with performance in five tasks, of which data efficiency (0.6287) and continual learning (0.8374) had p < 0.05. Step Completion Rate was 1.00 for TheAIScientist and 0.07 for Claude Code, and of the two agents with token records, Claude Code used more tokens than TheAIScientist (9.06M in total) despite its lower performance.
• Limits: The authors state that when the agent is fixed and the LLM changes, other LLM abilities such as reasoning and code writing also drive performance besides exploration diversity, and the correlation between diversity and performance was negative for causality, robustness, and privacy. AIDE sometimes misread the structure of the target codebase and created classes never wired into execution, and its scores on generalization and data efficiency equaled the baseline, which the authors suggest may stem from AIDE supporting edits to only a single file. The study is limited to 8 tasks, 3 agents, and the best of three rounds per agent, and AIDE ran through the commercial Weco service, which offered limited control over its internals, lacked token usage records, and sometimes ended runs early because of cloud failures.

Paper links

External research summaries. These are not HDATF publications or measured product results.