PaperCritic: Training AI Reviewers to Assess the Soundness of Papers

Published
Source
arXiv
Paper number
1474
Field
Research Agents
arXiv ID
2611.17834

Key points

  • A human study with seven AI researchers identified soundness and substantiveness as the two quality criteria for review points.
  • The authors built PaperSound-23K by labeling 23K review points collected from 1K papers with quality labels.
  • They trained PaperCritic with reinforcement learning that rewards the F1 score of precision, from soundness and substantiveness, and recall, from reference coverage.
  • PaperCritic-4B achieved an F1 score of 45.1 on the in-domain test set, 21.4 pp higher than the best trained 4B baseline.
  • PaperCritic-4B beat the frontier models DeepSeek-V4-Flash-0731 and Claude Opus 5 by 5.8 pp and 6.0 pp in F1, respectively.
  • Training on ICLR paper reviewing transferred to Nature-family papers, code review, legal contract review, and story writing feedback.

Paper links

External research summaries. These are not HDATF publications or measured product results.

Read original (opens in a new tab)