Neural Architecture Discovery via Autonomous Evolution

Key points

• Problem: Whether LLM agents can complete hard long-horizon tasks that advance AI research itself remains largely unexplored. Neural Architecture Search (NAS) only selects within human-defined spaces. The authors name three obstacles in architecture design: LLMs have limited built-in architecture knowledge, a single constraint violation in a large codebase silently corrupts training, and quality can only be measured by costly pre-training from scratch and benchmark runs.
• Input and output: This is a system paper with an implementation and experiments. ASI-ARCH takes summaries of past architectures, experimental analyses, and literature knowledge (cognition), writes the motivation and code of a new linear attention architecture in one pass, and stores the loss, benchmark scores, and analysis from actually training it as input for the next cycle.
• Method: A proposal from the Researcher enters training only after a novelty check, in which an LLM judges it against the 5 most similar past motivations found by embedding, and a code sanity check, and when training fails from an implementation error a debugging agent fixes the code from the error log. Fitness averages the sigmoid-transformed loss and benchmark improvements over the baseline, focused on differences within 10%, with an LLM judge score. The Analyzer compares each architecture with parent and sibling nodes to estimate the effect of individual design choices, and a pool of the top 50 architectures plus literature-derived cognition entries ground the next proposal.
• Implementation: The exploration stage trains 20M-parameter models on 1B tokens and evaluates them on 500 samples per benchmark, and the verification stage scales selected candidates to 340M parameters on 1B tokens and top candidates to 1.3B parameters on 15B tokens. The planner uses o3 and GPT-4.1, the checker and analysis use o3, training runs and debugging use GPT-4.1, and roughly 100 linear-attention papers were distilled into about 150 cognition entries searched through OpenSearch. Per its README, the official repository (GAIR-NLP/ASI-Arch) holds the pipeline, a MongoDB-based architecture database, and a cognition base built on retrieval-augmented generation (RAG), and links the list of discovered architectures to a project page.
• Results: In the 1,773 experiments (about 10,000 GPU hours) of the 20M-scale exploration, 1,350 candidates beat the DeltaNet baseline on both quantitative and qualitative metrics, and of about 150 re-verified at 340M scale, 105 reached state-of-the-art results. At 1.3B, the best model beat DeltaNet by +0.97 points on the overall average of 14 accuracy metrics, nearly three times the +0.34-point gain of Mamba2 over DeltaNet. With the same DeepSeek-V4-Flash proposal model, a 120-attempt budget, and the 20M training setup, the successful-design rate was 37.4% for ASI-ARCH and 8.3% for OpenEvolve, an open-source evolutionary coding agent.
• Limits: The search ran on linear attention with DeltaNet as the baseline, and the 105 state-of-the-art results are judged under the authors' controlled 340M evaluation. In the 1.3B efficiency comparison on the same implementation and hardware, the five ASI-ARCH models peak at 16.28G to 22.75G of memory, above the 11.44G of DeltaNet. At 340M, transferred rules gave +2.05 points on GLA (Gated Linear Attention) but smaller gains of +0.35 on the state-space model Mamba2 and +0.14 on HGRN2 (a gated linear recurrent model), and the authors leave family-specific adapters to future work.

Paper links

External research summaries. These are not HDATF publications or measured product results.