A Vocabulary for Multi-Agent Automated Research Systems

Key points

• Problem: Recent multi-agent LLM systems automate research end to end (reading prior work, proposing ideas, writing code, running experiments, and interpreting results), but they differ from one another in many design choices at once, such as communication topology, cross-run memory, initialization, and the evaluator. A claim that a multi-agent system or an autoresearcher is better therefore stays ambiguous until one knows which choice changed, and there was no common language for describing these systems.
• Input and output: The vocabulary takes the description of an automated research system (its paper), maps the design onto a problem tuple and a system tuple, and produces a map of which coordinates the system sets and which it leaves generic. One run is recorded as a trajectory from the input task to the returned artifact, and because agents, operations, and initialization may be stochastic, repeated runs on the same task yield a distribution over trajectories. This is a conceptual paper that defines the vocabulary and adds case analyses applying it to 10 existing systems.
• Method: A system is written as an eight-element tuple of the agent set, the operation universe (tools, skills, Model Context Protocol (MCP) server calls, and so on), the communication structure, the capability assignment, the shared state, the control policy, the initialization, and the evaluator. Shared state is split into within-run state that is wiped when the run ends, the external environment the agents act on, and state carried across runs, and the control policy is split into routing (who acts next and what they do), stopping (when the run ends), meta-control that changes the structure mid-run (spawn, kill, grant, revoke, consolidate, and others), and an exploration term for stochasticity. Each step's action in a trajectory is classified as an operation call, a message, a structural change, or a halt, which turns when to communicate or when to add an agent into a choice made by the policy.
• Argument: The author frames a system as search that maximizes a proxy evaluator score under a budget, and notes that the same agents and tools can score higher under more search, such as Best-of-N, while every other coordinate stays the same. The paper defines an overfitting tax, where ranking more trajectories makes it more likely that the top one is a trajectory the evaluator overrates, and cites MLR-Bench reporting fabricated or invalidated results in 8 of 10 audited coding-agent tasks (around 80%). Taste is split into generative taste, the rate of proposing novel trajectories before any score is seen, and evaluative taste, the gap between the proxy score and true quality, and the author suggests raising evaluator integrity (the structural protections that keep the score close to true quality) before crediting a stronger generator.
• Case studies: Mapping AIRA2 (an autoresearcher for ML research benchmarks), AlphaEvolve, Glia, AAR (Automated Alignment Researcher), AI Scientist-v2, MetaGPT, EvoX, ml-intern, SimpleTES, and Engram onto the tuple shows, for example, that the core of Glia is a capability assignment that leaves its Supervisor with an empty operation set, while AIRA2 sets a cross-run population store and meta-control. In this map no system changes capabilities during a run, every system leaves the exploration term at the default temperature, and EvoX is the only system that rewrites its control policy at run time. Per its README, the author's GitHub repository BardiyaAk/multi-agent-research-vocabulary releases a static project page with an explainer of the eight-element tuple, structure views, and a comparison table for the 10 case studies, along with tools/coords.json, the source of the comparison data, and tools/build-data.py, which converts it.
• Limits: The author maps each system from its own paper's description, so the coding is a descriptive map and differs from a validated measurement. A marked coordinate means that the paper sets it and is distinct from evidence that the coordinate improved performance. Generative taste stays qualitative until a novelty measure exists, and true quality is uncomputable in general, so it remains a design target rather than a measured value.

Paper links

External research summaries. These are not HDATF publications or measured product results.