The Hidden Methodological Layer: Uncovering Threats to Validity in LLM-Assisted Empirical Software Engineering

Published
Source
arXiv
Paper number
1480
Field
Software Engineering
arXiv ID
2611.20033

Key points

  • The authors re-analyzed 69 LLM-assisted research tasks from 50 primary studies.
  • Only 30 of the 69 tasks (43.5%) explicitly reported at least one LLM-related threat to validity.
  • The automation category had the highest reporting rate, with 22 of 42 tasks (52.4%) reporting threats.
  • Threats to empirical evidence construction accounted for 19 occurrences, including 10 LLM classification and labeling errors.
  • The authors argue that prompts, model configurations, interaction protocols, and validation procedures must be visible as part of the research method.
  • ChatGPT accounted for 38 model usages, the most in the corpus.

Paper links

External research summaries. These are not HDATF publications or measured product results.

Read original (opens in a new tab)