Beyond Benchmark Accuracy: A Diagnostic Test for LLM Chemical Prediction
- Published
- Source
- arXiv
- Paper number
- 1349
- Field
- LLMs / NLP
- arXiv ID
- 2610.19906
Key points
- The authors rebuilt six chemical prediction tasks so that the target molecule's name or SMILES is withheld and only the source structure and a natural-language edit description are provided.
- In structure reconstruction validation, all four models recover every chiral target at 100% accuracy, with the overall range from 86.05% to 100%.
- The label-consistent group, where source and target labels agree, scores better, and providing true source labels widens this gap in 22 of 24 model-task combinations.
- Replacing only the source label with a false one flips 91.2% of Claude-Opus-4.8 binary predictions toward the false label, and in yield prediction 61.2% to 98.0% of predictions move in the same direction.
- Adding tool evidence from NOSE and RDKit improves all five models on tasks with target-relevant evidence, while tasks with limited coverage show limited gains or even declines.
- The authors conclude that high benchmark accuracy alone does not imply that LLMs possess reliable structure-property reasoning capabilities.
Paper links
External research summaries. These are not HDATF publications or measured product results.