DiagramBank: A Quality-Audited Dataset of Scientific Schematic Diagrams with Multi-Level Document Context

Key points

• Problem: Schematic diagrams in papers convey methods, workflows, and system structure, yet existing scientific-figure datasets often mix them with plots, screenshots, and photos and often omit the body text that explains each figure. The authors argue for a resource that identifies schematic figures and keeps paper and figure context, provenance, and filtering uncertainty together.
• Input and output: This is a dataset paper, and the authors limit the contribution to the resource itself. The input is PDFs and metadata of AI/ML venue papers hosted on OpenReview, and the output is records for 57,100 schematic diagrams. Each record carries the figure image, paper title, abstract, caption, body paragraphs citing the figure, venue and year, provenance, and filtering labels.
• Method: PDFs and metadata come from the OpenReview Python API, figures and captions are extracted with PDFFigures 2.0, and body paragraphs citing the figure number are extracted with PyMuPDF. OpenCLIP ViT-B/32 first sorts figures into diagram, plot, photo, and other, and a VLM cascade selects the final diagrams through votes from two smaller VLM judges and confirmation by a larger VLM. Candidates that CLIP labels as diagrams enter a majority-vote path, and non-diagram candidates with CLIP confidence below 0.70 enter a rescue path.
• Implementation: Relational data are managed in DuckDB, and records are denormalized so that each figure directly holds its title, abstract, caption, context, quality labels, and provenance. The release includes an all-CLIP view and a view with CLIP confidence of at least 0.85 alongside the cascade-filtered release. The official repository README describes a download script that fetches data.jsonl, FAISS indices, and DuckDB from Hugging Face and a demo notebook that finds similar diagrams from a title, abstract, and caption, and it states that the reproduce scripts cover OpenReview collection through first-stage CLIP classification.
• Results: A blind human check of a stratified sample of 500 records from the cascade-filtered release estimates precision at 93.67% (95% confidence interval 90.11 to 97.22), with screenshots or photo-like figures as the main error. In the CLIP view check, precision is 94.4% at a threshold of 0.85, and raising the threshold from 0.50 to 0.90 lifts precision from 64.0% to 96.0% while retention falls from 100% to 55.7%. In a retrieval check with 20 queries, Precision@10 (the share of the top 10 retrieved diagrams that VLM judges rate as relevant) is 78.0% for hierarchical retrieval and 81.0% for caption-only retrieval.
• Limits: The authors state that relying on automated PDF parsing, figure classification, and context extraction can leave records with imperfect captions, missing body-reference paragraphs, or diagram versus plot classification errors. The release favors precision and may drop useful lower-confidence diagrams, and coverage is limited to accessible OpenReview venues and years, which may tilt the resource toward recent AI/ML practice. The retrieval check is a sanity check at the scale of 20 queries, the authoring case is qualitative, and the authors say the VLM judge model names are kept in the release documentation.

Paper links

External research summaries. These are not HDATF publications or measured product results.