AIRA_2: Overcoming Bottlenecks in AI Research Agents

Key points

• Problem: Prior work (AIRA-dojo) laid out three structural bottlenecks that hold back AI research agents. Synchronous single-GPU execution limits an agent to about 1 to 20 evaluated candidates per day on compute-heavy tasks, and selecting solutions on validation scores makes performance fall as search runs longer. Fixed operators that end in a single prompt response also cap search performance.
• Input and output: This paper is an implemented research agent system. The input is an ML task with training data and an evaluation metric, mainly MLE-bench-30, 30 Kaggle competitions (5 low, 20 medium, and 5 high complexity), plus AIRS-Bench with 20 research tasks. The output is test set predictions from the final solution selected among the candidate solution programs produced by the search.
• Method: A central orchestrator keeps a population of candidate solutions and runs steady-state evolution (moving on to the next candidate as soon as any worker is free), sampling parents by rank-based selection and dispatching mutation or crossover tasks. Each task is carried out by a ReAct agent that explores and debugs by running Python and Bash directly. Hidden Consistent Evaluation splits labeled data once into 80%/10%/10% and evaluates in a separate container while hiding the labels of the search split and the final selection split from agents.
• Implementation: Experiments attach one worker to each of 8 NVIDIA H200 GPUs (141GB), and each worker gets 12 CPU cores and 120GB of RAM. Code runs in Apptainer containers (Superimage) with Python and ML packages preinstalled, and a single code execution is capped at 9 hours. The base AIRA2 uses Gemini 3.0 Pro Preview, the stronger variant uses Gemini 3.1 Pro Preview, and each task runs for 72 hours with 3 seeds.
• Results: On MLE-bench-30, AIRA2 with Gemini 3.1 reached a mean Percentile Rank of 81.5% at 24 hours and 83.1% at 72 hours, above the 72.7% of the strongest baseline, CobraAgent. In the ablation analysis, HCE accounted for 13.0 Percentile Rank points of improvement at 24 hours and 18.4 points at 72 hours. On AIRS-Bench it beat the recorded SOTA on 11 of 20 tasks, and a manual audit of the code found that only 6 of them used clean methods while the rest relied on shortcuts such as data contamination or test label leakage.
• Limits: The authors state that LLMs may have seen public top Kaggle solutions during pre-training, so it is hard to tell how much of the gain comes from genuine reasoning versus recalling solutions. Preparing the evaluation splits needs one-time human curation, and the design targets multi-GPU, long-horizon search, giving up the immediate efficiency of lightweight agents in short or single-GPU runs. The baselines used 1 or 2 GPUs, and the comparison relies on their published results.

Paper links

External research summaries. These are not HDATF publications or measured product results.