MIR: Methodology Inspiration Retrieval for Scientific Research Problems

Key points

• Problem: In existing idea-generation methods that retrieve literature with a seed paper's abstract, the methodology and results in the abstract can bias retrieval toward the seed paper and reduce the novelty of the generated ideas. Commonly used retrievers and search APIs rely on sentence-level semantic similarity or generic inter-document relations, so they can fetch papers that merely share keywords, such as a 'sentence compression' paper for a 'model compression' problem. Some works also assume access to the seed paper's cited papers, so the authors target the realistic setting where only the research problem and its motivation form the query.
• Input and output: This is a method paper that defines a new task and dataset and proposes a retriever training method and LLM re-ranking methods. The input is a research proposal (problem and motivation) with any mention of methodology removed, the output is the top-k papers of a corpus ranked by their usefulness for building a solution, and at inference the retriever takes only the proposal without citation information. The MIR-MultiCite dataset extends MultiCite, a computational linguistics dataset of 12,653 citation contexts labeled with eight citation intents, and splits proposals by time into 704 for training (before 2019), 86 for dev (January to June 2019), and 139 for test (after that).
• Method: MAG treats each research proposal as a node with its cited papers as neighbors, annotates edges with citation intents and contexts, and counts the uses and extends intents as methodology citations. The retriever puts three triplets in one batch, pulling together the proposal and a methodology-cited paper, a methodology citation context and the proposal, and that context and its cited paper, and trains on the sum of L2 triplet margin losses; the hard negative of the first triplet is the paper outside the proposal's citations with the highest domain similarity measured from scientific named entities, mixed with random soft negatives at ratio α. LLM re-ranking uses Pointwise, which labels each of the top-k papers relevant or not, and MIR-Agent, which first writes the proposal's sub-problems and a generic action plan and analyzes each paper's applicability before judging, and papers judged relevant move ahead in their original order.
• Implementation: Proposals were extracted from the abstracts of citing papers with Gemini-1.5-pro-001, and the training data adds part of 51,021 proposals from about 60k arXiv computational linguistics papers whose citation intents were weakly labeled by a SciBERT-based classifier. SPECTER2, Stella 400M, and Stella 1.5B (the last with LoRA) were fine-tuned on a single NVIDIA V100 GPU (32GB), and re-ranking used Llama 3.1 70B Instruct and Gemini 1.5 Pro. Per its README, the official repository (Anikethh/Methodology-Inspiration-Retrieval) holds the dataset used in the paper (proposals, cited papers, citation contexts, citation intents, and training, dev, test, and augmented training splits extended to mid 2024), and its top level has data and assets folders.
• Results: On the MIR-MultiCite test set, fine-tuning raised Stella 1.5B's Recall@3 on the Extended Corpus (all cited papers of the training set plus the ground-truth citations of the test proposals) from 54.73 to 60.11 and its mAP from 49.08 to 56.89. On the Restricted Corpus, re-ranking the top 10 of the fine-tuned Stella 1.5B with few-shot MIR-Agent on Gemini 1.5 Pro raised Recall@3 from 66.95 to 71.45 and mAP from 64.24 to 69.02, and for the 75 test proposals whose top papers had full texts available, the same setup with full texts raised Recall@3 from 66.84 to 72.49. IA-Score, the average share of the 5 LLM-generated methodology ideas per proposal that match the methodology in the original paper's abstract, was 0.170 with abstracts retrieved by MIR and 0.142 with abstracts from the best baseline retriever.
• Limits: The dataset is limited to English papers in computational linguistics, and the retriever is trained on cited-paper abstracts, with full texts used for re-ranking the top-k. The authors report that MultiCite's original human annotations sometimes omit methodology citations and occasionally include trivial references (for example, mere use of GPT-4). An analysis of 15 proposals (39 relevant papers) with subpar Recall@5 under the best configuration (MIR-Agent with full papers) found 17 incorrect or missing annotations, 7 subjectivity cases where the authors chose a different research direction, and 15 misclassifications by the pipeline.

Paper links

External research summaries. These are not HDATF publications or measured product results.