PRL-Bench: A Comprehensive Benchmark Evaluating LLMs' Capabilities in Frontier Physics Research

Key points

• Problem: Existing science benchmarks test knowledge and reasoning with problems whose objectives, solution paths, and reasoning processes are fixed in advance, so they do not measure the exploration and long workflows of real research. Physics-specific benchmarks such as TPBench and PHYSICS also rely on short, clear-path tasks, and the research-oriented Frontier Science has only 20 physics questions.
• Input and output: This is a benchmark paper. A model receives a task that states only a scientific motivation and a concrete research objective, chooses the solution path and the needed domain knowledge itself, and returns numerical values, analytical formulas, or discrete judgments for each subtask. A judge model scores the subtasks against predefined answers and rubrics (intermediate grading criteria) and sums them into a 0 to 100 score.
• Method: The authors selected 100 theory- and computation-centered papers published in Physical Review Letters from August 2025 (Volume 135, Issue 7) to March 2026 (Volume 136, Issue 10), excluding papers focused on experiments or requiring large datasets, substantial compute, or specialized simulation software. With more than 10 domain experts, each paper was turned into one research-oriented task with an open solution path and cross-validated, and each task is split into subtasks such as analytical derivation and computational validation that avoid a strictly linear dependency chain to limit error propagation. The representative task consists of four parts: a motivation, the core task, answers with rubrics, and a detailed solution.
• Implementation: Six models, GPT-5.4, Gemini-3.1-Pro, Claude-Opus-4.6, Doubao-Seed-2.0-Pro, Qwen-3.5-Plus, and Kimi-K2.5, were evaluated under the same prompting and tool-use setting. Models could use a code interpreter, and search tools were disabled to prevent leakage from retrieving the original papers. Each problem was run five times per model and averaged, GPT-5 served as the LLM judge, and the paper states that the data is released on Hugging Face (AdrianMiao/PRL_Bench).
• Results: Gemini-3.1-Pro scored highest overall at 44.27, followed by Qwen-3.5-Plus at 40.05, Doubao-Seed-2.0-Pro at 37.83, Claude-Opus-4.6 at 37.40, GPT-5.4 at 37.38, and Kimi-K2.5 at 33.89, so every model stayed well below 50. For most models, formulaic or conceptual errors were the largest share at roughly 45 to 55% of errors (GPT-5.4 0.4697, Gemini-3.1-Pro 0.5079, Doubao-Seed-2.0-Pro 0.5562), while incomplete or unsupported answers dominated for Claude-Opus-4.6 at 0.6393. Most models scored lower in astrophysics and statistical physics than in condensed matter, high-energy physics, and quantum information.
• Limits: The authors state that tasks give more background information than real research so that answers are well defined and uniquely verifiable, which partly lowers the difficulty of open-ended exploration, and that the benchmark does not include the process of falsifying incorrect hypotheses. They also note that annotation errors may remain despite expert cross-validation and that the five-subfield split is approximate, so it does not fully capture problems that span several fields.

Paper links

External research summaries. These are not HDATF publications or measured product results.