LAB-Bench: Measuring Capabilities of Language Models for Biology Research

Key points

• Problem: Most science benchmarks ask textbook-style recall questions, and benchmarks that measure whether a system can carry out practical research work in a specific domain are rare. The authors argue that building AI systems that assist biology research first needs a benchmark for practical skills such as literature lookup, protocol troubleshooting, and sequence manipulation.
• Input and output: This is a benchmark and dataset that measures practical biology research work with 2,457 multiple-choice questions. A model receives a question and options about literature, figures, tables, databases, DNA and protein sequences, protocols, or cloning scenarios and returns one answer letter, and every question carries an extra option saying there is not enough information to answer.
• Method: LitQA2 (literature QA), SuppQA (supplementary material QA), FigQA (figure QA), TableQA (table QA), ProtocolQA (protocol troubleshooting), and CloningScenarios (real-world cloning workflows) were written manually by the authors and contracted biology experts, while most subtasks of SeqQA (sequence QA) and DbQA (database QA) were generated programmatically. Scores are reported as accuracy (correct over all questions), precision (correct over attempted questions), and coverage (share of questions attempted). About 80% of each subtask is released as a Hugging Face dataset and 20% is kept private to monitor training-data contamination.
• Implementation: Claude 3.5 Sonnet, Claude 3 Opus, Claude 3 Haiku, Gemini 1.5 Pro, GPT-4o, GPT-4-turbo, and Llama 3 70B Instruct were called through APIs without tools using a 0-shot chain-of-thought prompt, and results are averaged over three runs per model. Llama 3 70B has no vision capability and was left out of FigQA and TableQA, and only 25 of the 41 CloningScenarios questions were short enough for its prompt. The human baseline comes from PhD-level biology researchers who covered 69% of the questions in quizzes of 20 to 140 questions and could use the internet and DNA sequence software.
• Results: Human experts beat the models in most categories, and on TableQA, where models came closest, Claude 3.5 Sonnet reached 0.83 accuracy (humans 0.84) and 0.90 precision (humans 0.87). On LitQA2 the major frontier models often declined to answer, so Claude 3.5 Sonnet's accuracy was only 0.06, and on FigQA every model except Claude 3.5 Sonnet had near-random precision. On CloningScenarios, humans reached 0.60 accuracy and the best models (Claude 3.5 Sonnet and GPT-4o) 0.28.
• Limits: The authors note that plausible distractors are hard to design and that stronger models appear to raise scores by eliminating unlikely options and guessing instead of reasoning. In a small open-answer run on 10 CloningScenarios questions, Claude 3.5 Sonnet and GPT-4o both reached only 0.20 accuracy. The authors also note that only the humans could use tools, and CloningScenarios questions take 10 to 60 minutes each, so the human baseline there may be too low.

Paper links

External research summaries. These are not HDATF publications or measured product results.