Not Search, But Scan: Benchmarking MLLMs on Scan-Oriented Academic Paper Reasoning

Key points

• Problem: Current work on academic paper reasoning mostly follows a search-oriented paradigm, where the question pre-specifies a target and the model retrieves a few relevant passages and reasons over local evidence. This works for tasks with clear targets but struggles with researcher-style full-document reasoning and verification, and QA (question answering) benchmarks typically presuppose that an answer exists, embed cues in the question, and evaluation mostly focuses on the final outcome, with limited checks of whether intermediate reasoning is grounded in evidence.
• Input and output: This is a benchmark paper for error detection in academic papers. The model receives one or more complete papers as page images or OCR text together with a query that names no target, such as 'Assess the Methods section for Sampling & Generalizability issues', and writes free-form reasoning from evidence to the detected error. For scoring, GPT-4.1 extracts the cited evidence and reasoning steps from the answer and matches them against the human-refined gold explanation.
• Method: Papers come from ICLR (International Conference on Learning Representations) 2024 and 2025 and Nature Communications; on accepted papers Gemini 2.5 Pro proposes coordinated sentence-level edits across sections or pages that create an error, with the question and explanation, and annotators embed the validated edits into the original paper (Generation), and from rejected ICLR submissions it extracts falsifiable errors raised in public reviews and turns them into questions (Sampling). Cross-paper items, edited to conflict with a cited source, are built only by Generation, and errors fall into nine categories: research question and definitions (RQD), design and identifiability (DI), sampling and generalizability (SG), measurement and operationalization (MO), data handling and preprocessing (DHP), computation and formulae (CF), inference and conclusions (IC), referential and citation alignment (RCA), and language and expression (LE). The score multiplies whether the target error was found (0 or 1) by the geometric mean of an evidence location score (a Dice score with a penalty for over-reporting) and a reasoning score (the squared share of gold reasoning steps matched from the start), and by a penalty that falls quickly as unrelated errors grow, giving a value between 0 and 1.
• Implementation: 10 domain experts reviewed each item in independent pairs, with a third expert resolving disagreements, and of 3,500 initial candidates 1,700 were discarded and 1,541 of the rest were revised (535 question rewrites, 1,207 explanation edits and 1,141 category or metadata corrections). This left 1,800 questions from 715 real papers in 13 natural-science disciplines, and evaluation covered 15 models in 24 input configurations, split into page-image input and Tesseract OCR text input, plus 8 RAG methods. Per the README, the GitHub repository holds GPT-4.1 scoring and aggregation scripts, inference output folders, VolcEngine batch conversion scripts, a PDF-to-image converter, a RAG retrieval metrics script and utils.py with all prompts, while the detailed data sits on Hugging Face for storage reasons.
• Results: Scores (scaled by 100) were low overall: GPT-5 led image input with an average of 19.2 and Gemini 2.5 Pro led OCR text input with 30.3, yet even Gemini 2.5 Pro stayed below 60 in every error category. Reasoning-enhanced models scored higher, with Qwen3 A22B at 17.4 in Thinking mode against 1.7 for Instruct and DeepSeek-R1 at 11.4 against 1.7 for DeepSeek-V3.1, and across 9 MLLMs text input beat image input by 4.81 points on average, though image input did better on CF, where OCR breaks formulas and tables. With Qwen3-Thinking on text input, the no-RAG score of 17.4 fell to 16.7 with BM25 (word-matching retrieval), 16.6 with Contriever, 11.3 with BGE-M3 and 6.8 with NV-Embed-v2, only the Oracle condition that supplies the gold pages rose to 24.5, and on image input the only method above the Llama 4 Maverick baseline of 7.0 was VRAG-RL, built on a compact 7B model, at 10.9.
• Limits: The scan-oriented setting is instantiated through a single task, scientific error detection, the source papers come from ICLR 2024 and 2025 and Nature Communications, and the errors consist of synthetic errors generated by Gemini 2.5 Pro and errors drawn from reviews of rejected papers. Initial Gemini 2.5 Pro outputs had substantial hallucinations, logical inconsistencies and low-quality questions, so expert annotation was needed, and scoring relies on GPT-4.1 extraction, whose agreement with human extraction was checked on 200 random questions with a Spearman correlation of 0.841 for the overall score. The authors state that OCR text input severely degrades structured content such as figures and formulas, so questions that need multimodal information lose their evidence and inter-category correlations are artificially inflated.

Paper links

External research summaries. These are not HDATF publications or measured product results.