Eliciting Research Taste in LLMs through Future Research Direction Choice

Published
Source
arXiv
Paper number
1478
Field
Research Agents
arXiv ID
2611.19794

Key points

  • The authors built 80 pairwise comparison queries from 19 ML papers published after model training cutoffs.
  • Across 8 models, author alignment ranged from 45.0% to 67.5% and stable divergence from 15.0% to 36.2%.
  • 45 of 80 pairs elicited stable divergence from at least one model, totaling 169 divergence events.
  • Across four prompt variations, models retained divergences with weighted means from 66.9% to 80.5%.
  • In the pilot human study, 58 of 95 fixed-choice responses (61.1%) selected the author-recommended direction.
  • The most frequent divergence transition was Infrastructure-to-Methodological substitution with 55 events.

Paper links

External research summaries. These are not HDATF publications or measured product results.

Read original (opens in a new tab)