MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoning

Key points

• Problem: Open reasoning research has focused primarily on mathematics and coding, which offer abundant datasets, established benchmarks, and clear verification, and scientific reasoning remains significantly underdeveloped, particularly in the open-source community. Many science benchmarks are multiple-choice and post-training datasets often follow that format, and the authors observe that models trained on such data show inflated multiple-choice scores but struggle with computational problems, while decontamination (removing benchmark questions from training data), which typically relies on n-gram overlap or embedding similarity, is easily circumvented by minor variations in phrasing or structure. Reference answers in many science datasets are scraped from the web or generated by LLMs, which makes their factual accuracy hard to guarantee, and data distilled directly as long CoT (written-out reasoning) from DeepSeek-R1 is often prone to overthinking, which hurts training and inference efficiency.
• Input and output: It takes textbook PDFs and public science datasets (NaturalReasoning and Nemotron-Science) and returns two supervised fine-tuning (SFT) datasets of questions with step-by-step solutions (TextbookReasoning and MegaScience), an evaluation system covering 15 benchmarks, and Llama3.1, Qwen2.5, and Qwen3 models trained on the data. This is a dataset paper with a data construction pipeline and training experiments.
• Method: For TextbookReasoning, olmOCR (a PDF text recognition tool) converts textbooks to text, and for each 4,096-token chunk Llama3.3-70B-Instruct extracts question and answer pairs under a high-standard criterion (questions that demand multi-step reasoning, with complete solutions in the source) and a low-standard criterion (any complete question and answer), followed by MinHash deduplication (similarity threshold 0.6), refinement by DeepSeek-V3 against the source text, and filtering of defective pairs. For decontamination, BGE-large-en-v1.5 embeddings find the 5 most similar benchmark questions for each question, and Llama3.3-70B-Instruct judges whether each pair is a paraphrase, removing the question if any pair is. For MegaScience, response length, difficulty (16 responses from Qwen2.5-7B-Instruct scored 0 to 10 by Qwen2.5-32B-Instruct), and random selection are compared per dataset through SFT on Qwen2.5-7B, and DeepSeek-V3 writes new step-by-step solutions for every source except TextbookReasoning.
• Implementation: TextbookReasoning used 12.8k books (including 6,057 in computer science and 2,305 in medicine and biology) after removing crawled books whose metadata marked them as restricted for public access and filtering out books below university level with a subject and level classification by Llama3.3-70B-Instruct, and its 945,683 extracted pairs became 651,840 after deduplication, filtering, and decontamination. MegaScience combines 436.4k from NaturalReasoning, 173.3k from Nemotron-Science, and 651.8k from TextbookReasoning into 1261.5k instances, and Qwen2.5, Qwen3, and Llama3.1 base models were fine-tuned on it with LLaMA-Factory (a fine-tuning tool) at a learning rate of 5e-6, batch size 512, and 3 epochs. Per its README, the official GitHub repository GAIR-NLP/MegaScience contains step-by-step data_process scripts from question and answer extraction on olmOCR output through decontamination, reference answer extraction, and final formatting, along with the LLaMA-Factory training configuration, while the two datasets and the trained models are on Hugging Face and the evaluation code is in a separate GitHub repository, lm-open-science-evaluation.
• Results: In Table 6, which compares Qwen2.5-7B models trained on different open datasets with Qwen2.5-7B-Instruct, the MegaScience model was best on 7 of the 14 benchmarks used (MMLU, Massive Multitask Language Understanding, was excluded), averaged 61.01 against 58.80 for Qwen2.5-7B-Instruct, and scored 48.75 on SciBench and 40.23 on OlympicArena. With Qwen3 compared in non-thinking mode, the Qwen3-30B-A3B MegaScience model averaged 73.86 against 70.62 for the official Instruct model, and Llama3.1-8B also led 51.07 to 49.67, but for Qwen2.5-1.5B the official Instruct model was ahead by 2.99. The LLM-based decontamination found benchmark leakage in existing datasets: 19K questions in SCP-116K, 66K in NaturalReasoning, and 164K in Nemotron-Science.
• Limits: The authors state that finding one data mixture that is optimal for every base model remains hard, and that weaker models such as the Llama series or small Qwen2.5 models trained on difficult data like MegaScience without supplemental foundational data or easier warmup training give shorter responses with more repetition. Data selection was validated with Qwen2.5-7B as a proxy model, which ties the results to that model's capacity, and in Table 7 the math average of the MegaScience models fell below the official Instruct models for Llama3.1-8B, Qwen2.5-1.5B, Qwen3-1.7B, and Qwen3-4B. Reinforcement learning (RL) is left for future work, and experiments on compressing long CoT into shorter formats were postponed because of compute limits.

Paper links

External research summaries. These are not HDATF publications or measured product results.