BEAKER: An Expert-Curated Benchmark for Embodied Brains in Self-Driving Chemical Laboratories
- Published
- Source
- arXiv
- Paper number
- 1416
- Field
- Research Agents
- arXiv ID
- 2611.06911
Key points
- BEAKER is a benchmark with 1,000 expert-curated question-answer samples and 11 task types for evaluating embodied brains in self-driving chemical laboratories.
- All questions and annotations are manually created and reviewed by chemistry and embodied AI experts without AI-assisted generation or annotation.
- Evaluating 28 multimodal models showed that Actionability is the clearest bottleneck, averaging 22.2 across all 28 models.
- The best model, Claude Opus 4.7, scored 59.2 overall, 34.1 points below the human expert score of 93.3.
- Adding expert captions roughly doubled the mid-video sequence reasoning scores of all three models.
- Chemistry-specialized and science-oriented models do not consistently outperform general-purpose models.
Paper links
External research summaries. These are not HDATF publications or measured product results.