The Hidden Methodological Layer: Uncovering Threats to Validity in LLM-Assisted Empirical Software Engineering
- Published
- Source
- arXiv
- Paper number
- 1480
- Field
- Software Engineering
- arXiv ID
- 2611.20033
Key points
- The authors re-analyzed 69 LLM-assisted research tasks from 50 primary studies.
- Only 30 of the 69 tasks (43.5%) explicitly reported at least one LLM-related threat to validity.
- The automation category had the highest reporting rate, with 22 of 42 tasks (52.4%) reporting threats.
- Threats to empirical evidence construction accounted for 19 occurrences, including 10 LLM classification and labeling errors.
- The authors argue that prompts, model configurations, interaction protocols, and validation procedures must be visible as part of the research method.
- ChatGPT accounted for 38 model usages, the most in the corpus.
Paper links
External research summaries. These are not HDATF publications or measured product results.