Making AI Scientists Auditable from Evidence to Claim

Key points

• Problem: When an AI scientist revises a proposal or swaps code across stages, it becomes hard to follow whether the final conclusion matches the implementation that actually ran. In three AI-Scientist-v2 runs the authors examined as diagnostic examples, the records showed a proposed plug-in implemented as a replacement model, component roles without direct tests, and design changes whose source links were lost.
• Input and output: Given a research topic, the harness produces a literature survey, an idea with a component list, runnable code under project/, standard and ablation experiment results, and a technical report with figures. All three case studies started from an existing method. The paper is a system technical report describing the system, its implementation and the audit records.
• Method: An evidence graph extracted from full papers (problems, components, limitations, baselines, datasets) is queried by both idea generation and experiment planning. Ideas are searched with MCTS (Monte Carlo Tree Search) over edits that add, remove or replace components, and candidates from several search preferences are fused around one host idea. In experiments, workers execute stage-level validation contracts and validators inspect the artifacts; convergence requires one ablation step per component.
• Implementation: The evidence graph was built by parsing about 50K computer science papers indexed by Semantic Scholar with MinerU, and it holds 385K nodes and 1.15M typed edges. The system runs a relational database, an orchestration backend and a separately served frontend on a single host. The official repository README releases four agents (Survey, Idea, Experiment, Blog) and a prototype pipeline that chains them, and states that the paper-graph resource will be released in a later stage.
• Results: Across all nine comparisons of three tasks by three relation types, Xcientist's audit CDR (claim drift rate) ranged from 3.6% to 16.7%, lower than EvoScientist, InternAgent-1.5 and AI-Scientist-v2. The authors report that most of the gap came from fewer unverifiable claims. On the tasks, overall F1 (agreement with reference answers) on a LoCoMo subset rose from 0.306 to 0.391 while average tokens per query fell 64.2%, and average MAE (mean absolute error) on PEMS-BAY fell from 1.588 to 1.556.
• Limits: The authors state that measuring each module's contribution to the CDR gap would require controlled module comparisons across repeated runs, and this audit retrospectively compared each system's own claims, which differed in content. In the memory task, tests removing single components existed for only some components, so the gain is claimed only for the full system. On the two-dimensional heat task, the final PINN (physics-informed neural network) error of 0.151 was higher than 0.007 for MMPINN and 0.026 for Multiscale PINNs.

Paper links

External research summaries. These are not HDATF publications or measured product results.