PaperCritic: Training AI Reviewers to Assess the Soundness of Papers
- Published
- Source
- arXiv
- Paper number
- 1474
- Field
- Research Agents
- arXiv ID
- 2611.17834
Key points
- A human study with seven AI researchers identified soundness and substantiveness as the two quality criteria for review points.
- The authors built PaperSound-23K by labeling 23K review points collected from 1K papers with quality labels.
- They trained PaperCritic with reinforcement learning that rewards the F1 score of precision, from soundness and substantiveness, and recall, from reference coverage.
- PaperCritic-4B achieved an F1 score of 45.1 on the in-domain test set, 21.4 pp higher than the best trained 4B baseline.
- PaperCritic-4B beat the frontier models DeepSeek-V4-Flash-0731 and Claude Opus 5 by 5.8 pp and 6.0 pp in F1, respectively.
- Training on ICLR paper reviewing transferred to Nature-family papers, code review, legal contract review, and story writing feedback.
Paper links
External research summaries. These are not HDATF publications or measured product results.