AutoMedBench: Towards Medical AutoResearch with Agentic AI Models

Key points

• Problem: Many existing medical agent benchmarks focus on medical question answering, clinical dialogue, EHR (electronic health record) or FHIR (Fast Healthcare Interoperability Resources) interaction, healthcare administration or final task success, and primarily evaluate final outputs. The authors argue that telling whether a low score comes from misunderstanding the task, an unsuitable method, a failed environment setup, skipped validation or a malformed submission requires stage-level evaluation.
• Input and output: This is a benchmark for medical-AI research agents. The agent, a single base LLM using a code-execution environment, receives public inputs, a task brief, a submission schema and a time limit, and must produce and submit segmentation masks, restored images, VQA (visual question answering) answers, reports or detection boxes. Each run gets a workflow score over stages S1 to S5 (AGENTIC), a task score against hidden references (TASK), an OVERALL score that averages the two, and post-run error codes.
• Method: The 24 tasks span five tracks, segmentation, image enhancement, VQA, report generation and detection, and cover CT (computed tomography), MRI (magnetic resonance imaging), X-ray, pathology, blood-smear microscopy, dental imaging and medical video. Every task runs the same five stages (S1 to S5), Plan, Setup, Validate, Inference and Submit, weighted 25%, 15%, 35%, 15% and 10%, with S1 to S3 scored by an LLM judge and S4 and S5 by deterministic checks. Lite and Standard tiers differ only in brief scaffolding and give 48 task-tier settings, and failure types are logged, separately from the stage scores, as five error codes: E1 understanding, E2 data and model setup, E3 verification and recovery, E4 implementation and execution, and E5 deliverable and submission.
• Implementation: The agent container gets GPU, network, the public inputs and a writable workspace, while hidden references and scoring code sit in a separate evaluator container isolated from the external network, and training or fine-tuning during a run is forbidden. Six models, Opus 4.6, GLM-5, Gemini 3.1 Pro, ChatGPT-5.4 (called GPT-5.4 in parts of the paper), MiniMax-M2.5 and Qwen3.5, ran with the same system prompt, tool schema, stop conditions and decoding settings, with 10 runs for each of the 288 evaluation cells by default (20 for five segmentation tasks). According to the README, the GitHub repository AutoMedBench/AutoMedBench keeps task definitions and a Docker orchestrator on domain branches for running one cell by choosing an agent, task and tier, while the benchmark releases (Lite, Full) and the Full leaderboard Space are on Hugging Face and the live leaderboard is on the automedbench.github.io website.
• Results: Among the six evaluated models, OVERALL scores ranged from 66.5 for Opus 4.6 to 51.2 for Qwen3.5, a 15.3-point spread, and GLM-5 led the VQA track with 64.0. Averaged over the six models, S3 Validate was the weakest of the five stages and S2 Setup the strongest, and all six models scored higher on AGENTIC than on TASK. Of all fired error codes, 37.7% were verification and recovery errors (E3) and 38.1% deliverable and submission errors (E5), while understanding errors (E1) were 0.9%, and runs with one fired error code scored 48% lower OVERALL on average than runs with no error code.
• Limits: The authors state that stage-level evaluation cannot substitute for outcome evaluation and that a high AGENTIC score still requires separate checks of clinical usefulness and accuracy. Tasks are limited to inference-only workflows, excluding tasks whose correctness depends primarily on subjective judgment, long clinical dialogue or training-time adaptation, and the comparison covers base models run through the same harness without vendor agent wrappers. Instead of redistributing restricted data, the benchmark requires runners to obtain credentialed data, and in the authors' experiments MIMIC-CXR came through PhysioNet and fastMRI under the NYU (New York University) data-sharing agreement.

Paper links

External research summaries. These are not HDATF publications or measured product results.