SciVideoBench: Benchmarking Scientific Video Reasoning in Large Multimodal Models

Key points

• Problem: Most existing video benchmarks cover general scenes such as movies, daily activities, and instructional videos, and Gemini 2.5 Pro already saturates them with over 85% accuracy on VideoMME and Neptune. Recent benchmarks that add scientific and educational videos mostly stay at college level, and some MMVU questions can be answered without the video, so a test was needed that jointly measures perception, expert knowledge, and reasoning on research-level experiment videos.
• Input and output: This is a benchmark paper. A model receives one research experiment video, which JoVE publishes alongside a journal article, and a multiple-choice question, and picks one of 10 options. The 1,000 questions come from 241 videos covering more than 25 subjects across physics, chemistry, biology, and medicine, split into 245 quantitative, 385 hypothetical, and 370 conceptual reasoning questions.
• Method: Exemplars written by four PhD students, one per discipline, are expanded with GPT-4o into 12 examples per question type, and a QA Generator (the agent that writes questions and answers) based on Gemini 2.5 Pro drafts questions from the video, a Whisper transcript of the narration, and the paper text. The Evaluator answers each question to confirm it is answerable, the Visual Comparer checks that the reasoning is grounded in the video and adds timestamps, and the Refiner rewrites questions to point at specific video segments so they can only be solved by watching the video. Finally, human verifiers check the timestamps and answers, and key details such as mass or temperature that appear only in the transcript are overlaid onto the relevant video frames by a script.
• Implementation: For conceptual and hypothetical questions, Gemini 2.5 Pro writes nine plausible but wrong distractors, while quantitative distractors are made by adding Gaussian noise to the correct answer. Evaluation runs with the LMM-Eval toolkit on eight NVIDIA H100 GPUs at temperature 0. The official repository dengandong/SciVideoBench contains a README, a figs folder, and a scivideobench task folder for lmms-eval, and the README states that the benchmark is integrated into lmms-eval and that the data is hosted on Hugging Face.
• Results: In the main evaluation (Table 2), Gemini-2.5-Pro has the highest overall accuracy at 64.30%, and the best open-source model, InternVL-3-78B-Instruct, reaches 38.80%. GPT-4o given only text without the video gets 15.80%, and graduate students, three per discipline answering closed-book, get 17.4%. Chain-of-thought prompting raises Gemini-1.5-Pro's overall accuracy by 21.10 points and GPT-4o's quantitative accuracy from 11.84% to 34.29%.
• Limits: All videos come from a single source, JoVE, and every question is multiple-choice with 10 options. The only human baseline is three graduate students per discipline answering closed-book. The error-cause analysis compares the reasoning of only Gemini-1.5-Pro-Thinking and Gemini-2.0-Flash-Thinking against expert-verified rationales.

Paper links

External research summaries. These are not HDATF publications or measured product results.