PaperSearchQA: Learning to Search and Reason over Scientific Papers with RLVR

Key points

• Problem: Search agents are language models that reason and search a knowledge base to answer questions, and most search agents trained with RLVR (reinforcement learning that rewards only an automatically verified final answer) target general-knowledge QA (question answering). The authors see a lack of training environments for technical fields such as scientific literature, where search needs specialist knowledge and terminology, and note that the human-made BioASQ, a biomedical literature QA challenge, is hard to scale.
• Input and output: This is a dataset paper that releases a training dataset, a search corpus, and benchmarks. The input is a search corpus built from 16 million PubMed abstracts plus biomedical factoid questions (questions whose answer is a single entity); the agent issues queries in a search tag, reasons over the returned documents, and writes a short answer in an answer tag.
• Method: Abstracts sampled at random from the corpus are given to GPT-4.1, which writes 3 factoid questions per abstract in 10 categories defined with experts and also generates answer synonyms. 50% of the questions are paraphrased with different terms so that keyword matching alone is not enough to find the answer. Training follows the Search-R1 setup with GRPO (Group Relative Policy Optimization, an RL method that compares the rewards of several rollouts for the same question), giving reward 1 when the answer exactly matches a target or synonym and 0 otherwise, and retrieved document tokens are masked out of the gradient.
• Implementation: The authors release on the Hugging Face Hub PaperSearchQA with 54,907 training and 5,000 test samples, the BioASQ factoid set of 1,609 samples redistributed with added synonyms, and the search corpus with BM25 and e5 indexes. The agents are Qwen2.5 3B and 7B, trained with GRPO for 150 steps on eight A100 (80GB) GPUs. The official GitHub repository contains the data generation pipeline (data_generation), baseline implementations (baselines), and the retrieval servers and veRL (an RL training framework) code taken from Search-R1 (search-r1).
• Results: In Table 1 accuracy (exact match to a target, BM25 retrieval), the Search-R1-trained 7B agent scored 51.0 on PaperSearchQA and 44.8 on BioASQ, ahead of RAG (36.5, 29.7) and the literature QA agent PaperQA2 (37.1, 32.8). The 3B agent scored 41.6 and 35.5, which is 9.6 and 5.5 points above RAG (32.0, 30.0). Averaged over the two benchmarks it beat retrieval-free chain-of-thought prompting by 20.2 points (3B) and 21.4 points (7B), and for the Search-R1 agent paraphrased questions scored 44.9 against 57.2 for the original questions.
• Limits: The authors note that LLM-generated questions may include factually incorrect QAs, and that a question derived from a single abstract may have other correct answers. The scope is limited to single-hop factoid questions, search over abstracts, biology and medicine, and text only, and models without retrieval already knew enough to answer 15% to 30% of questions. GRPO training was unstable, with reward collapsing to zero in some runs, and the authors call the trained agent a research prototype that needs more thorough evaluation before real-world use.

Paper links

External research summaries. These are not HDATF publications or measured product results.