SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agents

Key points

• Problem: LLM agents have significantly accelerated scientific discovery automation while raising ethical and safety concerns at the same time. Much of the existing work on LLM safety has primarily focused on isolated aspects such as adversarial attacks on single models, pretraining data biases, and safety fine-tuning, and often neglects the holistic dynamics and emergent risks of multi-agent scientific environments. The authors note that tailored risk-aware AI scientist frameworks remain underexplored.
• Input and output: A user instruction is first analyzed to identify the scientific domain and task type, an ensemble of expert agents such as domain researchers, survey writers, and experimental planners is activated for a group discussion, and search and simulated scientific tools gather the information needed to produce a thoroughly cited research paper draft. Requests that are ethically inappropriate or high-risk are refused up front with a safety warning. This is a framework paper that contributes both a safeguarded framework and a benchmark for evaluating it.
• Method: The defenses are layered four deep. Prompt Monitor screens a request with LLaMA-Guard-3-8B for semantic intent and risk, the structural analyzer SafeChecker scans for attack patterns such as jailbreak attempts or roleplay exploits across 17 risk categories and labels each prompt pass, warning, or reject, and a prompt flagged by either one is rejected. Agent Collaboration Monitor watches the multi-agent discussion and intervenes on ethics, Tool-Use Monitor identifies unsafe use of the simulated scientific tools, and Paper Ethic Reviewer checks adherence to research norms before any output is released.
• Implementation: The framework is built on the Tiny Scientist framework with GPT-4o as the default LLM, API calls set to temperature 0.75 and a maximum token length of 4096, and multi-agent discussions capped at three rounds. SciSafetyBench holds 240 tasks, 10 for each combination of the six domains (physics, chemistry, biology, materials science, information science, and medicine) and four risk types, plus 30 experimental tools and 120 tool-specific risk scenarios built from them. The README of the official ulab-uiuc/SafeScientist repository presents it as the codebase for the paper and documents a poetry install, a config file, and the experiment scripts, with the SciSafetyBench data split into Dataset, Attack, and Tool folders under data/ScienceSafetyData.
• Results: In Table 2, which compares frameworks on 20 biology tasks, SafeScientist with SafeChecker reached a reject rate of 90% and a safety score of 4.83, above 2.45 for both AI Scientist and Agent Laboratory. In Table 3 on prompt attack defense, SafeScientist-Fuse, which combines the LLaMA-Guard and SafeChecker verdicts, reached the highest average rejection rate at 78.70%, ahead of SafeChecker at 66.36%, LLaMA-Guard-3-8B at 59.32%, and GPT-4o at 30.10%, and on Base64 encoding attacks SafeChecker reached 46.67% against 13.33% for LLaMA-Guard-3-8B. With Tool-Use Monitor in place, the tool call safety rate under malicious instructions rose from 5.8% to 47.5% and the human-rated correctness rate from 0% to 60.0%.
• Limits: The authors state that the current system primarily relies on off-the-shelf LLMs operating as separate modules with limited integration, which restricts both the depth of domain-specific expertise and the level of interaction between components. They also write that the tool-based safety evaluation remains only a simulation of real-world experimental settings and may overlook important contextual or sensory details, and they propose adding multi-modal inputs such as images of laboratory equipment or instructional videos and possibly embodied agents. The main experiment covers 20 biology tasks, and any task flagged as unsafe and rejected was assigned a safety score of 5 with its quality score excluded from the overall analysis.

Paper links

External research summaries. These are not HDATF publications or measured product results.