Science Prediction Universe
- Published
- Source
- arXiv
- Paper number
- 1459
- Field
- Research Agents
- arXiv ID
- 2611.15049
Key points
- The authors release the Science Prediction Universe benchmark, which evaluates agents on hidden experimental outcomes across chemistry, materials science, biology, and medicine.
- The 100 public tasks come from 85 recent papers and the 16 private tasks from unpublished wet-lab experiments, with measured values held only by an isolated grader.
- Given only the experimental definition, the agent ranks candidates by the measured outcome and may compute and search freely for eight hours.
- On the public benchmark, configurations separate into two distinct tiers, with task-adjusted Spearman correlations ranging from 0.227 for the lowest-performing configuration to 0.478 for the best.
- More computation did not yield detectably better predictions, and score showed no detectable correlation with run time, tool calls, or GPU use.
- The bottleneck lies in scientific judgment rather than tool execution: agents agree with one another at 0.52 but with experimental reality at only 0.33.
Paper links
External research summaries. These are not HDATF publications or measured product results.