rTaste: Do Research Agents Have Taste? Evaluating Early Stage Research Judgment
- Published
- Source
- arXiv
- Paper number
- 1413
- Field
- Research Agents
- arXiv ID
- 2610.40488
Key points
- Six frontier models were each given seven recent NLP papers and asked to propose, forecast and rank minimum viable experiments.
- Of the 415 proposed experiments, 194 were judged sound, valuable and new, and 73 were also beyond the seed paper and non obvious.
- The agent's first choice matched the auditors' most valued experiment in 38% of conditions, better than a random pick at 20%.
- The agent's forecasts scored a Brier score of 0.192, better than the base rate of 0.240 but worse than the mean of six judges at 0.179.
- Teams proposed 0.71 more good experiments per five on average than a single researcher without statistical significance, at a cost of $15.31 versus $23.93 per condition.
- The reviewers found 66% of the experiments built on the paper and 44% matched a list of obvious follow ups, so few opened a new direction.
Paper links
External research summaries. These are not HDATF publications or measured product results.