AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks?

Key points

• Problem: Scientific and engineering progress is a long-horizon iterative process of proposing changes, running experiments, measuring outcomes, and refining artifacts, yet existing benchmarks for frontier models primarily evaluate single-turn responses or short agent trajectories. The most impressive demonstrations of empirical optimization, such as AlphaEvolve and Karpathy's AutoResearch agent, are co-designed with heavily engineered, model-specific harnesses, tools, and search strategies, which makes the underlying model's contribution hard to isolate. The existing long-horizon benchmarks the authors cite each cover a single domain such as ML engineering, systems and kernel optimization, or real-world engineering, and none of them combines broad coverage, high difficulty, and resistance to saturation.
• Input and output: This is a benchmark paper. The agent receives a natural-language instruction and a container sandbox (CPU or a single GPU depending on the workload) holding a working but unoptimized baseline codebase and a local evaluation script, and within a wall-clock budget it edits, runs, and measures the code and submits a final implementation. A verifier hidden from the agent measures a metric (runtime, perplexity, throughput, parameter count, and so on) on held-out inputs and maps it to a continuous 0 to 1 score anchored at the baseline and a human-written reference solution.
• Method: Senior researchers and engineers contributed tasks from real problems they had faced, and at least 2 experts independent of the contributor plus a format audit agent audited each task on four criteria, validity (a higher score means a better implementation), solvability (the reference reliably reaches the target score within the budget on the target hardware), integrity (whether the agent can pass by hacking the scoring function), and measurement stability (small variance across repeated reference runs), revising or rejecting tasks that failed. Performance-optimization tasks use log-stretch scoring (0 at the baseline, 0.5 at the reference, approaching 1.0 near the practical optimum) with a gate that requires beating the baseline, and tasks with bounded quality metrics use linear scoring (0 at the baseline, 1.0 at the reference), except that two parameter-count puzzle tasks anchor at zero parameters. Against reward hacking, the verifier is sealed, ML tasks add a correctness gate on inputs drawn from a distribution disjoint from anything visible during development, an adversarial agent prompted to find shortcuts tests each task, critical immutable files are pinned by hash so that any unauthorized change scores zero, and trajectories are analyzed during evaluation so that verifiers are patched when new hacks appear.
• Implementation: The 36 tasks are 15 system optimization tasks (C, C++, Rust, Go, Python), 10 puzzle and challenge tasks, 7 model development tasks, and 4 CUDA tasks, with wall-clock budgets from 2 hours for the smallest puzzle tasks to 12 hours for LLM development tasks. Evaluation uses the terminus-2 agent of the Harbor framework by default for all models, runs CPU tasks in a local Docker sandbox on a Ryzen 9 9950X (16 cores) with 64GB of memory and GPU tasks on Modal cloud sandboxes with H100 and L40S GPUs, and consumed 2,544 wall-clock hours and 8.60 billion tokens in total. Per its README, the official repository (autolabhq/autolab) keeps each task in the tasks folder with task.toml, instruction.md, a Dockerfile and the editable files, a solution hidden from the agent, and scoring tests, and explains how to run one task or all tasks with Harbor's harbor run command.
• Results: Among the 11 main-set models, claude-opus-4.6 ranked first in all four categories with Avg@3 0.68, Best@3 (best of three runs) 0.76, and Dominance (head-to-head win rate against the other models per task) 0.93, followed by gemini-3.1-pro at Avg@3 0.50, with kimi-k2.6 (0.46), mimo-v2.5-pro (0.45), and glm-5 (0.43) leading the open-weight models, while gpt-5.4 scored 0.36 and grok-4-20 0.35. On flash_attention, a two-hour CPU task, comparing each model's best of three rollouts, claude-opus-4.6 cut its self-reported runtime from about 750 ms to 18 ms through 44 feedback-driven iterations over roughly 40 minutes, surpassing the reference solution (100 ms). In the failure analysis, which manually classified all 302 zero-score rollouts, the authors describe two time-awareness failures: deepseek-v4-pro, hunyuan-3-preview, and qwen-3.6-plus frequently used up the budget before a final submission, while gpt-5.4 and grok-4-20 often submitted after minimal exploration despite substantial remaining budget.
• Limits: The authors state that AutoLab focuses on executable system and ML engineering workflows and should be read as a benchmark for measurable auto-research rather than scientific discovery in the broadest sense. The baseline and reference values of performance-optimization tasks are calibrated to the benchmark's sandboxes and hardware, and small open-weight models under 200B parameters were excluded because of the benchmark's difficulty. Re-evaluating 4 models on 25 CPU tasks with two other harnesses (pi-mono and a mini-swe-agent whose system prompt was changed to demand persistent optimization) shifted the same model's mean score by up to 0.43 (kimi-k2.6, from 0.21 under pi-mono to 0.64 under mini-swe-agent), and the authors report that harness-induced variance can be comparable to model-induced variance.

Paper links

External research summaries. These are not HDATF publications or measured product results.