RankEvolve: A Reliable Multi-Agent Auto-Research Harness for Evolving Ranking Models

Key points

• Problem: Auto-research agents that propose, implement, train and evaluate model changes are bound by execution accuracy in the authors' setting. A single defect such as leaked held-out data, an omitted layer norm, a disconnected gradient or an unwired train/eval flag burns hours of accelerator time and yields an invalid metric, and over many iterations the damage compounds while the run drifts off its prescribed process. In the authors' motivating pilot where the whole procedure was given as one in-context playbook, 0 of 10 runs were fully faithful and all 10 ran only a subset of the selected proposals.
• Input and output: It takes an ML codebase and data, an EOP (Executable Operating Protocol) document listing phases and gates, and human decisions at the gates, and produces flag-gated code changes, training runs, leaderboard entries that include negative results, and evidence records. In the execution-accuracy evaluation each method receives a locked repository snapshot and a change request and returns a patch that a hidden executable oracle scores as pass or fail. This is a system paper on a deployed harness that also reports an HSTU deployment case study and an evaluation on ExecML, a benchmark built from that deployment's incidents.
• Method: An EOP declares phases, dependencies, required tools, human confirmation gates and branch and loop edges once; the runtime compiles it into a state machine that enforces order and gates and injects only the global preamble and the current phase instruction into context. The meta-meta-harness binds complete coding-agent products such as Claude Code, Codex and OpenHands to execution-graph nodes through an adapter contract, forming Dual loops where one product implements and another reviews and repairs (repeating until the reviewer approves or residual severity falls below a threshold, within bounded rounds) and flows that split a task and aggregate the results. The knowledge layer records per proposal hashes, prompt, patch, test outputs, configuration, metrics, cost and the promotion decision, organizes lessons into notes beside the code, a central wiki and an entity graph, and injects only the few relevant lessons at each phase.
• Implementation: The HSTU case used the public HSTU implementation and public MovieLens data, the workspace holds about 55 leaderboard entries across 49 experiment directories run on 1 to 8 NVIDIA H200s, and a human operator authorized compute and vetoed one over-budget experiment at the gates. ExecML builds 96 private tasks each from HSTU and LitGPT (a language-model training codebase), balanced over six incident families (data provenance and leakage, tensor routing, gradient flow, train/eval mode, metric semantics, configuration wiring) and split half new implementation and half repair, and Claude Code ran Claude Opus 4.8 while Codex ran GPT-5.6, both at maximum reasoning effort. Appendix B states that the failure-overlap analyzer, the leaderboard schema and the boost-last-K patch will be released with the codebase.
• Results: After 12 iterations RankEvolve reported HSTU NDCG@10 of 0.2192 on MovieLens-20M LARGE (+4.48% over the published 0.2098) and 0.1948 on BASE (+2.80% over 0.1895), and the 0.2192 came from an added genre side-feature in place of a new modeling primitive. On 96 ExecML HSTU tasks at matched observable inference budget, the Claude Code, Codex, Claude Code flow reached 62.5% execution accuracy, 16.7 points above the strongest single-product condition, Claude Code in all three roles (45.8%; 95% CI 6.6 to 26.7), and its silent critical-defect rate (faults that leave the patch runnable but can invalidate a scientific conclusion) fell from 35.4% for one-pass Claude Code to 10.4%. On LitGPT the same flow led 56.2% to 43.8%, a gain of 12.5 points (95% CI 3.0 to 22.0), and with the runtime held fixed, injecting only the current-step instruction beat full-protocol injection in execution accuracy by 2.1 points at early phases, 6.2 at middle phases and 10.4 at late phases.
• Limits: The authors state that the evidence base is two Python/PyTorch codebases (one ranking codebase and LitGPT), so cross-domain transfer is not fully established, that the oracles are incomplete so audits reduce but do not eliminate false passes, and that pretraining may include the public repositories. Without a memory on/off ablation the knowledge layer's contribution to proposal quality is not isolated, the HSTU case had a human operator at the gates who also steered it so the agent's and operator's contributions are not separated, and the cross-dataset transfer results are single runs. The deployed plan-merge flow had the highest HSTU execution accuracy (70.8%), but its 8.3-point lift over the re-run comparator was not statistically distinguishable (p=0.077), and the authors note that every win in the study was a composition of existing components.

Paper links

External research summaries. These are not HDATF publications or measured product results.