Towards a Medical AI Scientist

Key points

• Problem: The authors argue that existing AI Scientists are largely domain-agnostic and have shown promise in fields with relatively standardized problems and evaluation, such as mathematics, chemistry and general machine learning, but are hard to apply to clinical research that needs medical evidence and specialized data. They name three gaps: ideas that ignore medical priors such as diagnostic workflows and disease-specific pathology, unreliable experiments on heterogeneous medical data such as 3D structures, and manuscripts that largely overlook data provenance and ethics statements.
• Input and output: This is a system paper that presents Medical AI Scientist, a multi-agent system automating clinical medical AI research, together with the Med-AI Bench benchmark. In Reproduction and Innovation modes the system takes task instructions, dataset information and reference papers, while Exploration mode takes only task instructions and dataset information. It outputs a research idea, executable experiment code with results, and a typeset manuscript.
• Method: In the Idea Proposer, a Preparer and Surveyor collect reference papers and code and break them into formula and code components, and a Generator forms hypotheses through clinician-engineer co-reasoning, while in Exploration mode an Analyzer first lays out the core clinical and technical challenges and an Explorer finds fitting recent paradigms with code. An Assessor checks scientific quality and ethics, returning hypotheses below the quality threshold for refinement and rejecting unethical ones, and in the Experimental Executor an Investigator gathers code and medical toolboxes, a Planner writes the execution protocol, an Executor runs training and evaluation in a Docker environment, and a Judger reviews the results and returns corrective feedback. The Manuscript Composer follows the structure of reference papers, an Ethics Reviewer reports dataset origin and ethical approval, and LaTeX compile errors are fixed automatically.
• Implementation: All agents use a general-purpose LLM such as GPT-5 as the base model with role-specific prompting, and the Analyzer retrieves literature through the OpenAlex academic search engine. Med-AI Bench covers 19 tasks across six modalities (medical images, medical video, electronic health records (EHR), clinical text, physiological signals and multimodal data); for each task three papers of different difficulty were taken from Google Scholar as 57 ground-truth papers, and each paper yields 3 cases with different input modes, giving 171 cases. To speed up automated experiments the datasets were randomly subsampled, and the public address given in the paper is a project homepage.
• Results: In the expert evaluation of ideas from the Innovation and Exploration modes, three experts on a 5-point scale gave the system 4.40 and 4.32 for technical innovation and 4.65 and 4.68 for maturity across the two modes, above GPT-5 and Gemini-2.5-Pro. First-run experiment success over 57 research instances was 0.91, 0.93 and 0.86 in the Reproduction, Innovation and Exploration modes, against 0.72, 0.60 and 0.75 for GPT-5 and 0.40, 0.49 and 0.53 for Gemini-2.5-Pro under the same conditions. On the same medical task (diabetic retinopathy classification), the Stanford Agentic Reviewer scored the generated manuscripts 4.60 on its 0 to 10 scale against 4.86, 4.06 and 3.74 for human-written papers selected from MICCAI (Medical Image Computing and Computer Assisted Intervention), BIBM (Bioinformatics and Biomedicine) and ISBI (IEEE International Symposium on Biomedical Imaging), and separately one generated manuscript was accepted at ICAIS 2025 (International Conference on AI Scientists), which had 114 submissions and a 36.8% acceptance rate.
• Limits: The authors report that generated method designs can at times become overly intricate, in which case implementation may simplify or degrade components and drift from the original design. They also state that experiments are confined to predefined datasets and that the generated methods still perform below the state of the art. The expert manuscript study used a single task, diabetic retinopathy classification, with 20 papers (5 generated and 15 human-written), and the generated manuscripts scored 3.44 on Coverage (breadth of experimental design) against the 3.68 the paper compares them with.

Paper links

External research summaries. These are not HDATF publications or measured product results.