Can Frontier Auto Researchers Reconstruct Human Research Decisions in AI?
- Published
- Source
- arXiv
- Paper number
- 1419
- Field
- Research Agents
- arXiv ID
- 2611.12429
Key points
- AutoHRS is an evaluation framework that compares auto researcher outputs with human research across three stages: idea generation, experiment design, and paper synthesis.
- The evaluation corpus was built from 152 ICLR 2026 oral papers, yielding 456 research trajectories.
- On a normalized scale, auto researchers scored 0.29 for idea generation, 0.36 on experiment design, and 0.65 on paper synthesis, showing large gaps from human research in all three stages.
- 78.4% of generated ideas were formed by combining identifiable elements from the provided reference papers.
- Providing more fine-grained research directions consistently improved the idea reconstruction score from 0.29 up to 0.54.
- By research type, scores were 0.49 for empirical analysis, 0.32 for methodological work, and 0.18 for benchmark research.
Paper links
External research summaries. These are not HDATF publications or measured product results.