Measuring AI Ability to Complete Long Software Tasks

Key points

• Problem: AI benchmark scores rise quickly, but what they mean in the real world is unclear. Individual benchmarks are often artificial or adversarially selected against current models and saturate increasingly fast, so models as far apart as GPT-2 and Claude 3.7 are hard to compare on one scale.
• Input and output: This paper is an empirical study that proposes a new evaluation metric and measures a model trend. The input is human solve times and per-model success and failure records on 170 tasks: 97 HCAST (Human-Calibrated Autonomy Software Tasks) tasks (1 minute to 30 hours), 7 RE-Bench ML research engineering tasks (8 hours each), and 66 new SWAA (Software Atomic Actions) tasks (single-step actions of 1 to 30 seconds). The output is the 50% time horizon, the human time length of tasks a model completes with 50% probability, and the trend of that value over model release dates.
• Method: Task length is the geometric mean of successful solve times among over 800 human baselines (2,529 hours in total) by professionals in software engineering, ML, and cybersecurity, with researcher estimates used for 21 HCAST tasks. For each model, a logistic regression in the style of IRT (Item Response Theory, a statistical method that estimates item difficulty and test-taker ability) uses log task length as the difficulty and gives the length at which success reaches 50%. Regressing the log of each model's value on release date gives the doubling time, and a three-level bootstrap over task families, tasks, and runs gives confidence intervals.
• Implementation: The 12 frontier models released from 2019 to 2025 mostly ran in modular-public, an agent scaffold (a program that wraps the model with command execution tools and a loop) providing Python and Bash commands, about 8 times per agent and task pair. o1 used the triframe scaffold and o1-preview the duet scaffold, SWAA tasks ran with a simple prompting scaffold, and GPT-2, whose context length is too short, got imputed zero scores on HCAST and RE-Bench. The README of eval-analysis-public, the official GitHub repository of METR (Model Evaluation & Threat Research), lists time horizon analysis code (logistic fit, bootstrap, plots) and, for two reports (time-horizon-1-0 and time-horizon-1-1), DVC (data version control) pipelines and runs.jsonl run records, while the paper keeps most task contents private.
• Results: The 50% time horizon of frontier models grew exponentially from 2019 to 2025, doubling every 207 days (95% confidence interval 166 to 240 days). GPT-2 reached 2 seconds, o1 about 39 minutes, and o3 110 minutes. The 80% time horizon also doubled every 204 days but was 4 to 6 times shorter than the 50% time horizon.
• Limits: The authors state that a time horizon is always relative to the domain, the task distribution, and the skill and context of the human baseliners. All tasks are automatically scored, leave out cooperation or competition with other agents, and typically use static environments, so they differ systematically from real work, and they are heavily skewed toward software engineering and ML research. In the internal PR (pull request) experiment, contractors took 5 to 18 times longer than repository maintainers, so the horizon can change a lot with the reference human group.

Paper links

External research summaries. These are not HDATF publications or measured product results.