Open Source Planning & Control System with Language Agents for Autonomous Scientific Discovery

Key points

• Problem: An earlier cosmology multi-agent system (Laverick et al.) kept a human in the loop at every step, with the user reviewing each LLM response and guiding the next action. The authors see such systems as useful research assistants that are limited by their reliance on continuous human input, and aim for a system that carries research tasks end to end with no human in the loop.
• Input and output: This is a system paper that presents cmbagent, an open-source multi-agent system. cmbagent takes a quantitative research task (the Main Task) written by the user, plans it, and carries out the plan step by step, writing and running code locally where needed. It outputs the final plan saved as a JSON file, the code and execution results of each step, Markdown text written by the researcher agent, and plots, and it displays the total Planning cost in USD and the cost of each sub-task.
• Method: In the Planning phase, a planner proposes a plan and a plan reviewer gives feedback in a loop repeated n_reviews times; the plan has at most n_steps steps, each with a sub-task, actions, and the agent in charge. In the Control phase, a controller hands sub-tasks to a researcher (reasoning, interpretation, summaries) and an engineer (Python code), and an interpreter agent reads each execution output and picks the next agent. Below n_fails failed executions the run typically returns to the engineer for another attempt, a missing Python package typically sends it to an installer agent that runs pip install, and reaching n_fails sends it to a terminator agent that ends the session.
• Implementation: About 30 LLM agents run on the multi-agent framework AG2 and call models through OpenAI, Google, and Anthropic APIs. The authors built two context agents that carry the full documentation of the cosmology Python packages camb and class in their system message, plus RAG (retrieval-augmented generation) agents that search vector embeddings of the classy_sz documentation and of paper corpora; in the denario project, cmbagent serves as the research results generation backend. The official GitHub repository README covers installation with pip install cmbagent, optional domain dependencies (materials, biochem, astro, data), a one_shot example, and demo notebooks in the repository's docs/notebooks.
• Results: On 14 camb problems, each solved ten times per agent, the authors report that gpt-4o, gpt-4.1, and gemini-2.5-pro failed systematically on problems 12, 13, and 14, while the camb context agent running gemini-2.5-pro did much better. On a subset of DS-1000 pandas, numpy, and matplotlib problems, Planning & Control raised the overall success rate over One Shot from 66% to 78%, with the researcher agent on gpt-4.1. A cosmological parameter estimation task by MCMC (Markov chain Monte Carlo) sampling on the Union2.1 Type Ia supernova data was solved on the first run with gpt-4.1 and default settings.
• Limits: The evaluation covers 14 camb problems, a subset of DS-1000, and one cosmology task, and the camb success rates appear only as the Figure 2 plot. As of July 9, 2025, denario had not been publicly released. The authors note that such systems raise concerns about automation bias (over-trust in automated outputs), fairness, and misuse or misinterpretation of findings, and that the ethical and environmental costs need to be addressed.

Paper links

External research summaries. These are not HDATF publications or measured product results.