Late but Valuable: Why Deadlines Change the Measured Value of LLM Reasoning
- Published
- Source
- arXiv
- Paper number
- 1438
- Field
- Reasoning / Efficiency
- arXiv ID
- 2611.11910
Key points
- The authors showed that when a deadline makes the system execute a fallback, reporting answer quality and latency separately overstates the delivered value of reasoning.
- In a replay of 1,200 decisions, thinking answers that missed a deadline were worth about five times as much as on-time ones.
- Separate summaries predicted about twice the value that thinking delivered and overstated thinking's advantage over fast mode by 0.0241 per decision.
- Raising the number of simultaneous requests from 8 to 56 on a single-GPU Qwen3-8B reduced the share of thinking answers meeting a 45 s deadline from 79.2% to 1.4%.
- The authors recommend pairing each answer's value with its arrival time and scoring the action that actually runs.
Paper links
External research summaries. These are not HDATF publications or measured product results.