DeepScientist: Advancing Frontier-Pushing Scientific Findings Progressively

Key points

• Problem: Earlier AI Scientist systems can produce novel findings, but without clearly defined scientific goals they often blindly recombine existing knowledge and methods, and their outputs frequently look naive under human evaluation. In frontier research, evaluating a single idea takes a full research cycle of implementation, experimentation, and analysis, so each evaluation is very costly.
• Input and output: It takes a manually reproduced code repository of a human SOTA method (with execution logs and test scripts), the target performance metric to improve, and related papers and code. It returns implementations and experiment logs of new methods that raise the metric, the ideas and validation results accumulated in the Findings Memory, and research papers describing the successful methods. This is a system paper with an implementation and experiments.
• Method: It treats research as maximizing an expensive black-box function, that is, a Bayesian Optimization problem, and repeats a three-stage loop. In the hypothesis stage an LLM reviewer acts as a surrogate model (a cheap stand-in evaluator) and scores each new hypothesis for utility, quality, and exploration value on a 0 to 100 scale, and in the verification stage a coding agent implements the hypothesis with the highest UCB (Upper Confidence Bound) score on top of the SOTA repository and runs the experiment. Only hypotheses that beat the baseline move to the analysis stage for extra experiments such as ablations and paper writing, and every result is recorded in the Findings Memory to inform later cycles.
• Implementation: The system ran on two servers with 8 Nvidia H800 GPUs each, launching a separate instance per GPU, with Gemini-2.5-Pro handling core logic and Claude-4-Opus inside Claude Code (v1.0.53) handling code implementation, each in its own Docker container. After Claude Code reported completion, DeepScientist re-ran the main script itself to check, and three human experts verified outputs and filtered out hallucinations. Per its README, the official ResearAI/DeepScientist repository is a local-first research workspace installed through npm and used with Codex, Claude Code, Kimi Code, or OpenCode runners, while the paper states that the Analyze & Report module will not be open-sourced.
• Results: On Who&When agent failure attribution, A2P reached 29.31% accuracy in the handcraft setting and 47.46% in the algorithm-generated setting, above 12.07% and 16.67% for the previous SOTA All at Once. On LLM inference acceleration (MBPP), ACRA raised throughput from 190.25 to 193.90 tokens per second, and on AI text detection (RAID), PA-Detect raised AUROC from 0.800 (Binoculars) to 0.863 while cutting latency from 117 ms to 60 ms. A three-member human program committee gave the 5 generated papers an average rating of 5.00, close to the 5.08 average of ICLR 2025 submissions, and two papers received 5.67.
• Limits: Of more than 5,000 ideas, about 1,100 were validated and only 21 surpassed the SOTA; randomly testing 100 ideas per task without the selection step gave an effectively zero success rate, and selection raised it only to about 1 to 3%. Human experts who analyzed a sample of 300 failed trials found that about 60% stopped early because of implementation errors, and human reviewers noted insufficient evaluation on standard benchmarks, missing ablations, and missing comparisons with essential baselines. The authors judge the approach currently impractical for high-cost research such as pre-training foundation models or pharmaceutical synthesis, and the total cost of the reported advances was about $100,000.

Paper links

External research summaries. These are not HDATF publications or measured product results.