Scores Alone Do Not Prove Discovery: The Discovery Certification Protocol for Auditing AI Research Agents

Key points

• Problem: When an AI research agent reports a high-scoring output after experiments, the score alone does not show that a discovery was made. A sealed test (held-out evaluation data) establishes the output's utility, but whether a matched agent could reach the same score from the initial data, public material, and its own reasoning, and whether truthful experimental feedback improves the chance of reaching it, are separate measurement questions. The authors turn these three questions about one outcome into an executable audit.
• Input and output: This is a framework paper that proposes an audit protocol for research outcomes and demonstrates it on controlled and real-data tasks. The input is an outcome claim (the final artifact, a registered model and budget, background knowledge and starting observations, Web content seen by the target run, and validity and recovery rules); the output is a record bundle with two decisions, Core and Evidence, each set to Pass, Refuted, Inconclusive, or Not evaluated. A deterministic verifier recomputes the decisions offline from the frozen records.
• Method: Gate 1 checks, with a lower confidence bound on sealed evaluation, that the output beats the baseline by at least a preset smallest useful gain (δmin). Gate 2 runs a registered number of fresh challenger agents that receive the background knowledge, starting observations, and Web content seen by the target run but no run history or new measurements; any valid result from any method at or above the recovery line (a threshold equal to the target score x minus tolerance ε) refutes Core, and zero recoveries yield a Clopper-Pearson upper bound on the recovery probability. Optional Gate 3 compares paired branches from a shared checkpoint that receive truthful measurement feedback or neutral messages, and Evidence requires a lower confidence bound on the effect plus a null-policy equivalence check.
• Implementation: The audited agents and challengers are DeepSeek models run through a registered command-line interface (CLI): deepseek-v4-flash for SQLite-Web, deepseek-v4-pro for the virtual catalyst, and deepseek-flash for the real-data tasks, and real-data sessions expose the Read, Write, Edit, Glob, and Grep tools, with shell and native Web tools disabled. The Yacht and Ionosphere audits each registered 96 challengers, 45 file-channel controls, 30 feedback pairs, and 60 null pairs. According to the paper, the supplement contains the installable dcp-audit, task-specific evidence producers, the reusable dcp-harness, and three real-data decision bundles.
• Results: On real data, Yacht and Ionosphere passed Core with outcome scores of 0.7677 and 0.6551; their best scores across 96 challengers, 0.4641 and 0.4171, stayed below the recovery lines of 0.7177 and 0.6051, and the recovery upper bound was 0.0468. From a shared checkpoint, branches with truthful feedback reached the target level in 9 and 16 of 30 trials while neutral branches reached it in none, yet Evidence was Inconclusive for both tasks. In controlled cases, SQLite-Web (88.55% less traffic-weighted work) and virtual catalyst control (sealed optimum reached) passed both Core and Evidence, and in the developmental knapsack case Core was refuted when a challenger produced 2 results above the recovery line.
• Limits: The authors state that every reported pass is a local kernel decision from records collected and replayed locally (formal_certificate_issued=false), and formal issuance needs review and countersigning by an independent authority. The recovery bound concerns one fresh episode from the registered model, policy, information, and budget distribution, and at an illustrative recovery rate of 0.01, a 96-episode audit would find an existing route with probability 0.6190. The Core and Evidence passes for Dry Bean are an exploratory reanalysis under a revised policy recorded after Gate 2 scoring.

Paper links

External research summaries. These are not HDATF publications or measured product results.