Forecasting Scientific Progress with Artificial Intelligence
- Published
- Source
- arXiv
- Paper number
- 220
- Field
- AI / General
- arXiv ID
- 2605.22681
Key points
- The knowledge gap is the observed performance improvement when a model can access relevant information that existed before the cutoff date, and it measures how much of a model's failure comes simply from not knowing already known facts.
- The prediction gap is the remaining difference between performance and actual outcomes even when the model has full pre-cutoff knowledge, and it captures the intrinsic difficulty of predicting the unknown.
- The models consistently reported high confidence in their predictions even when empirical accuracy was low, and this calibration error makes it difficult for human researchers to know when to trust AI predictions.
Paper links
External research summaries. These are not HDATF publications or measured product results.