Agent Benchmarks Stop Ranking Before They Run Out of Tasks

Key points

On SWE-bench Verified, strengthened tests rejected 17.7% of the median agent’s passing patches, an error larger than the gaps between neighbouring agents, so the order of neighbours is close to chance.
ALE-Bench’s top 10 split into 3 tiers by contest score but became 1 tier when scored pass/fail against the median human, where 5 models pass 39 of the 40 problems, and 47.8% of the runs beat that reference.
In a Terminal-Bench re-run, models tied on pass/fail differed widely on the actual quantity: on portfolio-optimization, opus’s median speedup was 10.9× while the others’ was 1.3×.
On fixed tasks, a higher bar only moved the problem: the contest-winner bar scored 89% of the earlier models zero on every task.
The authors proposed a clipped log-ratio score that maps the baseline to 0 and the reference to 1 with a clip at three reference gaps, and reported cost as a separate axis.

External research summaries. These are not HDATF publications or measured product results.