Science Prediction Universe

Published
Source
arXiv
Paper number
1459
Field
Research Agents
arXiv ID
2611.15049

Key points

  • The authors release the Science Prediction Universe benchmark, which evaluates agents on hidden experimental outcomes across chemistry, materials science, biology, and medicine.
  • The 100 public tasks come from 85 recent papers and the 16 private tasks from unpublished wet-lab experiments, with measured values held only by an isolated grader.
  • Given only the experimental definition, the agent ranks candidates by the measured outcome and may compute and search freely for eight hours.
  • On the public benchmark, configurations separate into two distinct tiers, with task-adjusted Spearman correlations ranging from 0.227 for the lowest-performing configuration to 0.478 for the best.
  • More computation did not yield detectably better predictions, and score showed no detectable correlation with run time, tool calls, or GPU use.
  • The bottleneck lies in scientific judgment rather than tool execution: agents agree with one another at 0.52 but with experimental reality at only 0.33.

Paper links

External research summaries. These are not HDATF publications or measured product results.

Read original (opens in a new tab)