AutoResearch at Production Scale: Failure Modes and a Multi-Agent Framework

Key points

• Problem: Karpathy's AutoResearch, where an LLM edits a training script and keeps changes that improve one held-out metric, was shown on single-GPU, five-minute language-modeling experiments. In production, an iteration takes hours of multi-GPU compute, evaluation criteria compete, and campaigns span weeks beyond one context window, and the authors know of no prior report on whether AutoResearch holds up under these conditions.
• Input and output: This is a case study of running AutoResearch for 12 weeks in Amazon's book recommendation pipeline. The agent receives a training script (train.py), a program.md with instructions and accumulated findings, and an immutable evaluator, produces training runs with edited code or settings, and keeps a change when the evaluated score improves. The paper's outputs are a taxonomy of five failure modes, a three-principle scaffolding design and a principle that ties the weight of each mechanism to iteration cost.
• Method: The scaffolding follows three principles: prevent failures, persist knowledge and redirect stagnant search. In the expensive System A, a Code Fixer agent rewrites each script in one pass before execution, a Criticizer agent issues a 2 to 5 sentence directive only when the best Recall@6 (how often the next purchase is among the 6 nearest neighbors) improves by 8% or less over the last 5 iterations, and program.md lives in S3 storage so memory carries across training jobs. The cheaper System B implements the same principles with automatic post-execution revert, human pivots between runs and a human-maintained program.md.
• Implementation: System A learns 128-dimensional purchase-optimized retrieval embeddings for 98% of the active catalog on multiple GPUs at 9 to 19 hours per iteration; the Researcher uses claude-sonnet-4-6 and the Code Fixer and Criticizer use claude-opus-4-6. System B assigns hierarchical Semantic IDs with RQ-VAE (a model where each level quantizes the residual of the previous one) to over 50% of browsed books on one GPU at 6 to 52 minutes per iteration, and one System A iteration costs about 760 times a System B iteration. The authors state that the training data is proprietary and cannot be released.
• Results: With code-level AutoResearch and three agents, System A reached 6.90% Recall@6, 1.82 times the hand-tuned baseline of 3.79% and 2.45 times the deployed retrieval baseline of 2.82%, with a mean of 6.18% over 11 evaluation dates. The agent designed on its own a text-only fallback that embeds the roughly 83% of items missing from the behavioral graph, widening catalog coverage 5.8 times. System B raised weighted coherence (a level-weighted average of within-cluster similarity) from 0.348 to 0.735; the agent made the largest single gain (0.348 to 0.691) by itself, and the best value came after a human pivot to behavioral plus content embeddings.
• Limits: The component analysis for System A is observational, comparing natural campaign transitions instead of controlled runs, and both systems sit in one domain, book recommendation, at one organization. Recall@6 and weighted coherence are offline proxies that this study did not check against real customer outcomes, and online pilots are underway. Metric fixation was only partly mitigated because the agent never questions the metric, so the authors state that the system cannot run fully hands-free and needs human QA between runs.

Paper links

External research summaries. These are not HDATF publications or measured product results.