AutoSOTA: An End-to-End Automated Research System for State-of-the-Art AI Model Discovery

Key points

• Problem: Reaching state-of-the-art (SOTA) results in AI research takes long cycles of implementation, replication, debugging, evaluation, and reflection, and the needed artifacts are spread across papers, repositories, datasets, checkpoints, and undocumented engineering assumptions. The authors argue that existing iterative frameworks such as AutoResearch confine the agent to one bounded code module inside a sanitized sandbox, which leaves out the path from a paper to a recovered execution environment and a reproduced baseline.
• Input and output: It takes one paper from a top-tier AI conference and returns an executable code repository whose performance on the core metric strictly beats the method proposed in the paper. In the final phase it restores the container to the best-scoring state, saves it as a Docker image (autosota/paper:optimized), and exports the full repository to the host. This is a system paper that implements eight agents and runs experiments on 125 conference papers.
• Method: Eight agents split the work into preparation (AgentResource, AgentObjective), execution (AgentInit, AgentMonitor, AgentFix), and improvement (AgentIdeator, AgentScheduler, AgentSupervisor), and the execution and improvement stages iterate by passing results back and forth. If the three most recent iterations were all parameter adjustments, AgentScheduler forces the next iteration onto the Leap Path, which requires a structural change, and if its first result does not beat the best score it allows up to five more iterations before rolling back to the pre-Leap optimum. AgentSupervisor blocks shortcuts such as editing the evaluation script, using test data for training, or hard-coding predictions through red line rules (R1 to R7).
• Implementation: The evaluation set starts from accepted papers of the past year at eight conferences (NeurIPS, ICLR, ICML, CVPR, ICCV, ACL, NAACL, AAAI), screened for method papers, public code links, and repository readiness into 2,347 candidates, of which 745 were run in preliminary tests, and papers with environment installation failures or more than 4 GPU hours per full replication were removed to leave 125. AgentInit builds the environment with Repo2Run and falls back to a manual Docker setup when the automated build fails, and AgentScheduler commits the original code under a _baseline tag inside the container and records the measured baseline in scores.jsonl as iteration 0. The repository map, the idea library, and a prior literature report are kept in code_analysis.md, idea_library.md, and research_report.md, and the scheduler treats two GPUs as one unit that runs at most one active task at a time.
• Results: Of the 125 screened papers, AutoSOTA found new SOTA models that beat the original method for 105, and the authors report an average gain of nearly 10% on each paper's single core metric at about five hours per paper. By type of change, algorithmic changes such as loss redesign or architectural restructuring (Class A) covered 64 papers (51%) with a mean gain of 8.82%, and hyperparameter tuning (Class B) covered 41 papers (33%) with a mean gain of 5.67%. In the case studies, the throughput of FR-Spec (a method that speeds up LLM inference with speculative decoding) rose from 674.58 to 765.27 tokens per second, a 13.44% gain, and the optimality gap (the distance from the optimal solution) of MoSES (a multi-task vehicle routing solver based on a mixture of specialized experts) on the capacitated vehicle routing problem (CVRP) fell from 0.914% to 0.444%.
• Limits: Of the 125 papers, 20 (16%) fell into Class C (optimization failure), and the authors attribute these cases to invalid optimizations in which AgentSupervisor failed to detect or prevent potential violations. The authors state that the current loops rely mainly on the Result Match items of the AgentObjective rubric, which may overlook structural and semantic fidelity. They also describe the MoSES gain as a deliberate compute trade-off that expanded inference time from about 3.7 to 103 seconds.

Paper links

External research summaries. These are not HDATF publications or measured product results.