ShieldLM: Empowering LLMs as Aligned, Customizable and Explainable Safety Detectors (2402.16444v2)

Abstract: The safety of LLMs has gained increasing attention in recent years, but there still lacks a comprehensive approach for detecting safety issues within LLMs' responses in an aligned, customizable and explainable manner. In this paper, we propose ShieldLM, an LLM-based safety detector, which aligns with common safety standards, supports customizable detection rules, and provides explanations for its decisions. To train ShieldLM, we compile a large bilingual dataset comprising 14,387 query-response pairs, annotating the safety of responses based on various safety standards. Through extensive experiments, we demonstrate that ShieldLM surpasses strong baselines across four test sets, showcasing remarkable customizability and explainability. Besides performing well on standard detection datasets, ShieldLM has also been shown to be effective as a safety evaluator for advanced LLMs. ShieldLM is released at \url{https://github.com/thu-coai/ShieldLM} to support accurate and explainable safety detection under various safety standards.

• The paper introduces ShieldLM as a novel LLM-based safety detector that aligns outputs with human safety standards using a bilingual dataset.
• It employs a customizable rule mechanism that allows developers to fine-tune detection rules for dynamic, context-specific safety requirements.
• The model delivers explainable safety judgments through GPT-4 analyses, outperforming baselines in accuracy and F1 scores across diverse test sets.

An Academic Overview of ShieldLM: Enhancing Safety in LLMs

The paper "ShieldLM: Empowering LLMs as Aligned, Customizable, and Explainable Safety Detectors" delineates a novel approach for addressing safety issues in the responses generated by LLMs. Despite the increasing capabilities of LLMs, their predisposition towards generating unsafe content has necessitated the development of robust safety detection mechanisms. This paper introduces ShieldLM, an LLM-based safety detector designed to align with human safety standards, customize detection rules, and explain decision-making processes.

Methodology and Contributions

The authors construct a substantial bilingual dataset consisting of 14,387 query-response pairs, annotated according to diverse safety standards. The dataset is pivotal as it anchors ShieldLM’s training and encompasses outputs from multiple LLMs. The methodology emphasizes three primary characteristics—alignment, customizability, and explainability—thereby setting ShieldLM apart from existing approaches, which often lack a balance between these critical aspects.

1. Alignment: ShieldLM is trained using a dataset designed to reflect human-annotated safety judgments in both Chinese and English. This ensures the model aligns closely with human safety standards across diverse linguistic and contextual scenarios.
2. Customizability: The model introduces a mechanism for rule customization, allowing developers to draft detection rules that ShieldLM can apply to specific cases. Notably, the paper introduces a training strategy where irrelevant rules are incorporated into the dataset, enabling ShieldLM to discern pertinent rules during inference.
3. Explainability: The paper employs GPT-4 to generate analyses that articulate the reasoning behind each safety judgment. This step enhances transparency, a significant step towards establishing trust in automated safety detection systems.

Empirical Evaluation

ShieldLM demonstrates superior performance across four test sets, including both in-distribution and out-of-distribution datasets, when compared to existing moderation tools and LLM configurations. The results underpin the model's ability to not only identify unsafe responses with higher accuracy but also explain its judgments, which is crucial for practical applications involving safety-critical interactions.

1. Performance Metrics: ShieldLM surpasses strong baselines in accuracy as well as F1 scores for safe and unsafe classifications. Notably, it effectively adapts to customized detection rules, showcasing robust performance in scenarios requiring fine-grained safety standard adherence.
2. Customizability Analysis: The model's ability to follow strict or loose rules confirms its versatility and highlights its practical utility in real-world applications, where safety standards are often dynamic and context-dependent.
3. Practical Utility: In a practical application scenario, ShieldLM was utilized to evaluate the safety of another LLM's responses, accurately following customized detection rules and validating its applicability in ongoing safety evaluation processes.

Implications and Future Work

The introduction of ShieldLM has several implications for both theoretical and practical domains in AI safety:

• Theoretical Implications: The paper advances the notion of customizable AI, where models adapt their behavior according to predefined human-centric rules, potentially setting a precedent for future safety technologies.
• Practical Implications: ShieldLM's deployment in real-world safety detection tasks could significantly enhance the robustness of LLM applications in sensitive domains such as healthcare, finance, and education.
• Future Directions: The authors acknowledge that ShieldLM is trained on general data, which might limit its effectiveness in specialized domains requiring professional expertise. Future work could explore integrating domain-specific knowledge bases to enhance the model's contextual understanding further. Additionally, developing scalable methods for data acquisition beyond human annotations, such as semi-automated approaches, could bolster ShieldLM's adaptability across broader applications.

In conclusion, ShieldLM presents a comprehensive solution for enhancing the safety of LLM outputs through alignment, customizability, and explainability. Its impact is poised to resonate across various sectors, reinforcing the deployment of AI technologies that are both innovative and safely attuned to human expectations.