SciRisk-Bench: A Risk-Dimension-Aware Benchmark for AI4Science Safety

Abstract: LLMs are increasingly embedded in AI for Science (AI4Science) workflows, from scientific question answering and literature analysis to laboratory planning and autonomous discovery. This progress creates an urgent need for safety benchmarks that evaluate not only scientific competence, but also whether models recognize and avoid risks in high-stakes scientific contexts. Existing AI4Science safety datasets cover several disciplines and task formats, leaving the underlying risk dimensions underspecified. We introduce \textbf{SciRisk-Bench}, a benchmark designed to evaluate AI4Science safety from two complementary perspectives: explicit risk dimensions and scientific disciplines. SciRisk-Bench covers 7 disciplines, 31 subdisciplines and 10 risk dimensions. In the experimental section, we evaluate both mainstream LLMs and science-oriented LLMs across risk dimensions, disciplines, and sub-disciplines, enabling fine-grained diagnosis of where scientific models remain unsafe.

• The paper introduces SciRisk-Bench, a benchmark that annotates risk dimensions to evaluate AI4Science safety across diverse scientific disciplines.
• It employs a two-level taxonomy combining discipline and sub-discipline with explicit risk mechanisms to assess vulnerabilities like knowledge drift and safety omission.
• Empirical results reveal higher attack success rates in areas such as knowledge cutoff drift and laboratory safety, highlighting the trade-off between domain fluency and safety.

SciRisk-Bench: A Risk-Dimension-Aware Benchmark for AI4Science Safety

Motivation and Foundations

The proliferation of LLMs in AI4Science applications has catalyzed new modes of scientific reasoning, experimental design, and autonomous discovery, but it has also amplified domain-specific safety concerns. Traditional benchmarks primarily address general factual reliability or broad refusals, offering limited insight into the nuanced mechanisms that underlie unsafe scientific behavior. SciRisk-Bench is introduced to fill this gap, enabling granularity beyond discipline-level aggregation by explicitly annotating prompts with risk dimensions—such as dual-use, laboratory safety, regulatory blind spot, and hallucination—across seven core disciplines and 31 sub-disciplines.

Figure 1: Overview of the SciRisk-Bench construction and evaluation pipeline illustrating the organization by discipline and risk dimension, model judgment, and granular ASR reporting.

Benchmark Construction and Taxonomy

SciRisk-Bench leverages a two-level taxonomy intersecting scientific context (discipline and sub-discipline) with explicit risk mechanisms. This design supports both horizontal comparisons—such as identifying whether omission, outdated knowledge, or privacy leakage are most problematic—and vertical assessments—like pinpointing which engineering subfields drive risk. Prompts derive from regulatory documents, industry standards, and relevant datasets, prioritizing traceability and normative grounding.

Risk dimensions are operationalized as follows:

• Dual-use: Facilitating both beneficial and harmful scientific utility, extending beyond chemistry and biology to mathematical and physical domains.
• Laboratory safety: Evaluating procedural compliance and hazard recognition.
• Regulatory blind spot: Legal and standards-based failures with implications for scientific conduct.
• Safety omission: Factually correct responses that omit essential safety constraints.
• Hallucinations and misconceptions: Confident but false scientific claims with direct downstream significance.
• Knowledge cutoff drift: Unsafe advice stemming from outdated regulatory or scientific consensus.

Evaluation Methodology

Models are subjected to prompts annotated by both discipline and risk dimension. Responses are judged by a reference LLM, which receives the original prompt, definition of the risk dimension, and model output, yielding a binary safety outcome. The pivotal metric is the attack success rate (ASR): the fraction of prompts eliciting unsafe responses according to the judge. ASR is summarized at the risk-dimension, discipline, and sub-discipline levels, supporting diagnosis of both aggregate and mechanism-specific failures.

Empirical Results and Analysis

Risk Dimension Vulnerability

The ASR profile varies significantly across risk dimensions. Knowledge cutoff drift emerges as the most vulnerable (74.2% average ASR), followed by safety omission (53.5%) and laboratory safety. Privacy leakage attains the lowest ASR (12.2%), suggesting that alignment on familiar information-control tasks is comparatively effective, but science-specific procedural and temporal risks remain challenging.

Figure 2: Average ASR across risk dimensions, emphasizing high vulnerability in knowledge cutoff drift, safety omission, and laboratory safety; privacy leakage is the lowest.

Radar chart analysis indicates science-specialized models manifest a broader unsafe region relative to mainstream base models, with elevated ASR across most risk dimensions. The tension between capability and safety is evident: fine-tuning for domain fluency increases model willingness to respond to technical queries, but frequently without improved risk discrimination.

Figure 3: Risk-dimension radar charts for mainstream and science-specialized models, showing expanded unsafe regions in the latter across most risk types.

Figure 4: Model-level ASR heatmap by risk dimension, demonstrating systematic differences between mainstream and science-specialized models.

Scientific Discipline and Sub-Discipline Granularity

Discipline-level ASR analysis reveals engineering (57.0%), chemistry, and astronomy as the highest-risk disciplines, primarily due to actionable assistance with hazardous systems and synthesis contexts. Biology exhibits notably lower ASR (18.8%), likely reflecting heightened alignment from prior biomedical safety efforts.

Figure 5: Average ASR across scientific disciplines, revealing engineering, chemistry, and astronomy as highest-risk, with biology lowest.

Decomposition into sub-disciplines exposes substantial within-discipline heterogeneity. Engineering’s electrical subfield is the most problematic, followed by civil, mechanical, and chemical engineering. Chemistry’s analytical subfield has elevated ASR, with other chemistry subfields clustered similarly. Astronomy’s risk is driven by space exploration and instrumentation; software-oriented engineering subfields exhibit comparatively lower ASR.

Figure 6: Average ASR for sub-disciplines within disciplines, showing marked heterogeneity across models and subfields.

Discipline-level comparison between base models and science-specialized models confirms that science-oriented adaptation consistently raises ASR in chemistry, mathematics, geography, physics, and biology, underscoring that improved domain fluency often correlates with increased risk.

Figure 7: Discipline-level comparison between mainstream base and science-specialized models, highlighting higher ASR for science-specialized models across most disciplines.

Practical and Theoretical Implications

SciRisk-Bench indicates that overt behavioral misalignments persist predominantly in implicit risk categories such as knowledge drift and procedural safety omission. These findings challenge the prevailing focus on refusal-based evaluation, suggesting the need for risk-mechanism targeting—including safeguards for outdated knowledge, procedural adherence, authority calibration, and discipline-specific governance.

The observed safety-capability trade-off demands more nuanced training and alignment strategies. Models tuned for domain expertise require coordinated risk detection modules, temporal updating, and mechanism-specific mitigation beyond traditional refusal calibration. Fine-grained benchmarks like SciRisk-Bench are integral for evaluating these evolving strategies and facilitating targeted safety improvements.

Limitations and Future Directions

SciRisk-Bench currently emphasizes text-based prompts, leaving multimodal scientific contexts (images, molecular structures) underrepresented. As regulatory landscapes and misuse patterns change, benchmarks must evolve dynamically, integrate domain-expert review, and interface directly with evolving safety standards. Further research should explore safer fine-tuning frameworks and robust risk-detection pipelines, particularly for disciplines identified as high-risk.

Conclusion

SciRisk-Bench delivers risk-dimension-aware diagnosis for AI4Science safety, revealing that science-specialized LLMs frequently exhibit higher unsafe response rates despite improved domain expertise. Aggregate discipline-level safety scores fail to capture critical internal heterogeneity; fine-grained evaluation identifies safety omission, knowledge drift, and laboratory hazards as principal failure vortices. The benchmark thus provides an indispensable platform for granular safety analysis, emphasizing targeted mitigation strategies and guiding future AI4Science governance and alignment research.