Sci-Reasoning: A Dataset Decoding AI Innovation Patterns

Key points

• Problem: The thinking by which researchers find gaps and combine several earlier works into a new contribution has been explained through subjective anecdotes instead of structured data. Citation networks record only that a citation happened and drop its nature (for example a baseline to beat, a concept to extend or a method to combine), and earlier lineage tracing typically finds only a single progenitor paper or treats citations uniformly.
• Input and output: It takes 3,819 Oral and Spotlight papers (999 Oral, 2,820 Spotlight) from the machine learning conferences NeurIPS, ICML and ICLR (2023 to 2025) and, for each paper, outputs a structured entry with 5 to 10 key predecessors, each predecessor's role and relationship type, a synthesis narrative of 200 to 400 words and a thinking trajectory. Each paper gets a primary label and, when present, a secondary label from 15 thinking patterns. This is a dataset paper that also reports a pattern analysis and an LLM idea prediction evaluation on the same data.
• Method: GPT-5 reads each paper's full text in one pass, links citations to the bibliography, scores importance from the context around each citation (for example building upon, inspired by) and citation counts per section, then picks key predecessors that mix contribution types (for example method, problem formulation, baseline) and writes the narrative. Prompts were refined on 30 papers from fields the authors know well, cases where GPT-5's self-reported score was below 0.7 were cross-checked with Claude Opus 4.5 and Gemini 3.0, and the roughly 3% of papers where models disagreed went to human review. For the thinking patterns, an LLM drew about 190 raw patterns from 10 batches of 35 papers (350 total), merged them into 15, and then classified all 3,819 papers in batches of 5.
• Implementation: The extraction model was chosen by comparing 4 OpenAI models on 77 Oral and Spotlight papers with ground-truth predecessors, and GPT-5 had the highest recall at 89.73%, ahead of GPT-5.2 at 87.47%, GPT-4.1 at 78.00% and GPT-5-mini at 68.53%. The idea prediction test pulls the predecessor content of 77 NeurIPS 2025 Oral papers through Exa AI (a neural paper search engine), asks an LLM for 10 ideas, and has a GPT-5.2 judge decide whether one matches the real paper's direction. Per the README, the official GitHub repository holds Python code for the four stages (paper collection, predecessor extraction, pattern analysis, idea evaluation), and the dataset is released separately on Hugging Face.
• Results: Of the 15 patterns, Gap-Driven Reframing (924 papers, 24.2%), Cross-Domain Synthesis (687, 18.0%) and Representation Shift (401, 10.5%) together covered 52.7% of papers. The text and appendix report the most frequent combinations as Gap-Driven Reframing with Representation Shift (318), Cross-Domain Synthesis with Representation Shift (233) and Gap-Driven Reframing with Cross-Domain Synthesis (204). On Hit@10 for 77 NeurIPS 2025 Oral papers, which asks whether one of 10 ideas generated from the predecessors alone matches the real paper's direction, Gemini 2.5 Pro scored 49.35%, Claude Opus 4 42.86%, GPT-5.2 38.89% and Claude Sonnet 4 29.87%.
• Limits: The authors state that the data captures the logic of justification written into polished papers, which can differ from the actual discovery process with its failed experiments and abandoned hypotheses. The data is limited to Oral and Spotlight papers from three venues in 2023 to 2025, which may favor empirical work, and it leaves out other venues, interdisciplinary research, areas such as symbolic AI, and benchmark, dataset, technical report, position and survey papers. Predecessors, narratives and pattern labels were produced by LLMs, and human review focused on the roughly 3% of papers where models disagreed.

Paper links

External research summaries. These are not HDATF publications or measured product results.