An Axiomatic Benchmark for Evaluation of Scientific Novelty Metrics

Key points

• Problem: Automatic metrics for a paper's novelty help AI scientists avoid spending compute on already explored ideas, yet existing metrics are generally validated against confounded signals such as citation counts or review scores. When the authors take the 100 highest- and 100 lowest-rated papers with strong reviewer consensus from ICLR 2022 and 2023, these novelty buckets match the accept/reject decision for 93.5% (technical novelty) and 97.0% (empirical novelty) of papers, so reviewer novelty scores are entangled with quality.
• Input and output: This is a benchmark that compares novelty metrics without ground-truth novelty labels. A metric under test receives a focal paper and a reference pool of prior papers, the pool is changed in controlled ways, and each focal paper counts as a pass or fail depending on whether the score moves in the expected direction, giving a pass rate per probe. The benchmark consists of 10 tasks in three domains (NLP, computer vision, and biomedical AI), with 100 focal papers randomly sampled per task.
• Method: Three axioms are tested through probes. The Redundancy axiom says the score must fall as the pool covers more of the paper, probed with an exact copy, a paraphrase, and planting 25% to 100% of the paper's atomic claims into separate pool papers. The Relevance axiom says the score must rise when the pool is swapped for another field or the paper's cited works are removed, and the Temporality axiom says the score must fall as the pool's time window moves later.
• Implementation: The data comes from the PapersWithCode archive on Hugging Face (dated July 29, 2025), using tasks with roughly 1,500 tagged papers each, with references fetched through the Semantic Scholar API. The 10 evaluated systems are the embedding metrics Yin, RND (relative neighborhood density), SemNovel, and FastTextLOF, plus NovaScore, the AI Scientist judge, RAG-Novelty, ARC (the lexical novelty gate of AutoResearchClaw), and the Si pairwise tournament and OpenNovelty, which give rankings or text instead of scores; systems that retrieve through tools are sandboxed so retrieval reaches only the constructed pools. The authors state that they release arXiv IDs, derived files, and scripts that fetch paper texts, and API usage totaled just under $400.
• Results: Embedding metrics such as Yin and RND catch nearly all exact copies (1.00, 0.99) and paraphrases (0.99, 0.97) but reach only 0.66 and 0.53 when 25% of the claims are planted. RAG-Novelty, where an LLM reads the paper with retrieved neighbors, passes only 0.28 even on the exact-copy probe and fails every redundancy probe. On the monotone temporal probe the best of the eight scoring systems reaches 0.24, and the authors conclude that surface redundancy is largely solved while conceptual redundancy remains open.
• Limits: The authors note that pools hold only one to two thousand papers per task, so performance at the scale of the real literature cannot be tested, and that all tasks are AI research tasks. Sandboxing controls what an LLM judge retrieves but leaves the model's parametric knowledge in place, and since the axioms are necessary conditions, passing every probe does not guarantee a good novelty judge. The AI Scientist judge and the Si tournament were run on three tasks and OpenNovelty on two, and the Si tournament and OpenNovelty were tested only on the Redundancy axiom.

Paper links

External research summaries. These are not HDATF publications or measured product results.