A Sequential Decision Framework for Evaluating LLM-Guided Search
- Published
- Source
- arXiv
- Paper number
- 1409
- Field
- Information Retrieval
- arXiv ID
- 2610.39231
Key points
- The paper proposes a framework that represents the search process with two actions: select (judging the next document in an existing ranking) and spawn (creating a new ranking).
- The authors describe existing retrieval strategies such as adhoc search, MTF, Thompson sampling, ISJ, and CAL as policies operating in a common environment.
- The experiments used the TREC Deep Learning 2019 collection, with GPT-4o-mini for relevance judgments and GPT-5 for more complex tasks such as spawning.
- Under a global judging budget, the query reformulation then MTF policy achieved the highest recall for every retrieval pipeline and budget.
- Adhoc search, which judges a single ranked list top-down, was the least efficient policy under every retrieval pipeline.
- The retrieval pipeline itself mattered as much as the policy, with SPLADE and the hybrid pipeline recovering more relevant documents per judgment.
Paper links
External research summaries. These are not HDATF publications or measured product results.