Scaling Automatic Research Agents via World Models

Key points

• Problem: Training AutoResearch agents (LLM agents that take a research question, form an idea, implement the experiment, analyze the outcome, and iterate) with RL needs many trajectories. Generation shares compute through batching, but every candidate solution needs its own isolated sandbox that loads data and trains models on real GPUs, so execution cost grows linearly with the number of trajectories and becomes the dominant bottleneck for scaling RL.
• Input and output: This is an RL training method paper with convergence theory and experiments. In the AutoResearch tasks, the world model takes the task description and the agent's solution code, predicts the execution outcome, and aggregates strict JSON verdicts from eight checks such as imports and names, API calls, file and column references, and the compute budget into a score. The final output is the post-trained agent policy, and AutoResearch evaluation uses the percentile on real Kaggle competition leaderboards.
• Method: About 10% of groups are anchor groups graded by both the world model and real execution, and these score pairs form the thin stream of ground truth kept during training (the anchor signal). Online Debiasing fits a monotone map to the score pairs with isotonic regression (regression constrained to a non-decreasing function) to remove the world model's bias, and Inverse-Variance Denoising combines the policy gradients of the two group types weighted by inverse variance to suppress noise. The authors prove that bias and noise each add an error term to the convergence bound and that the two corrections shrink these terms, strictly improving the convergence guarantee.
• Implementation: In the AutoResearch experiments, Qwen3.5-4B and Qwen3.5-9B agents are trained with GRPO (group-relative policy optimization), and the world model is the same backbone used through prompting only and kept fixed during training. Training uses 45 MLE-Dojo competitions (Kaggle machine learning competitions wrapped as interactive environments), evaluation uses 14 held-out competitions and 60 tasks from DSBench (a data science benchmark built on Kaggle modeling tasks), and the vision-language-action (VLA) experiments use the MiniVLA-1B policy with the progress model Robometer as the world model. According to the official repository README, the release contains the method in numpy (wmrl/), the MLE-Dojo and DSBench pipeline (ml_research/), the VLA track (embodied/), and tests; it is a clean reimplementation of the production stack used for the paper and implements only WMRL itself.
• Results: WMRL cut training compute relative to real-execution GRPO from 883 to 286 GPU-hours at 4B and from 1174 to 349 GPU-hours at 9B, a 3.1x and 3.4x reduction. Average leaderboard percentile at 9B rose from 18.8 to 21.6 on MLE-Dojo and from 31.2 to 32.8 on DSBench, and on both benchmark averages the 9B WMRL agent scored above Nemotron-120B-A12B (20.5, 31.7) while the 4B WMRL agent scored above Kimi-48B-A3B. On LIBERO-Long (a simulated robot-arm manipulation benchmark), the overall success rate rose from 37.4% after SFT to 41.2% with WMRL, a 3.8 point gain.
• Limits: The authors assume tasks whose outcomes are predictable from the agent's solution and where a small stream of real execution stays available as an anchor, and they note that when outcomes hinge on randomness that no reading of the solution reveals, WMRL degrades toward standard GRPO. The convergence guarantee relies on the assumption that the bias acts as a monotone distortion of the score, the AutoResearch experiments cover two Qwen3.5 scales (4B and 9B), and the MLE-Dojo tasks were chosen from the 20 smallest competitions by dataset size per category to fit the sandbox.

Paper links

External research summaries. These are not HDATF publications or measured product results.