Class-RAG: Real-Time Content Moderation with Retrieval Augmented Generation (2410.14881v2)

Abstract: Robust content moderation classifiers are essential for the safety of Generative AI systems. In this task, differences between safe and unsafe inputs are often extremely subtle, making it difficult for classifiers (and indeed, even humans) to properly distinguish violating vs. benign samples without context or explanation. Scaling risk discovery and mitigation through continuous model fine-tuning is also slow, challenging and costly, preventing developers from being able to respond quickly and effectively to emergent harms. We propose a Classification approach employing Retrieval-Augmented Generation (Class-RAG). Class-RAG extends the capability of its base LLM through access to a retrieval library which can be dynamically updated to enable semantic hotfixing for immediate, flexible risk mitigation. Compared to model fine-tuning, Class-RAG demonstrates flexibility and transparency in decision-making, outperforms on classification and is more robust against adversarial attack, as evidenced by empirical studies. Our findings also suggest that Class-RAG performance scales with retrieval library size, indicating that increasing the library size is a viable and low-cost approach to improve content moderation.

• The paper introduces Class-RAG, a framework that enhances content moderation classifiers in Generative AI by integrating context from a dynamically updateable retrieval library to handle ambiguity and improve scalability.
• The Class-RAG system leverages embedding models and a retrieval library to provide contextual examples to an LLM classifier, demonstrating significant performance gains and increased robustness against adversarial attacks compared to traditional fine-tuning.
• Class-RAG offers a scalable and adaptable approach to content moderation, with performance improving economically by expanding the retrieval library, highlighting the benefits of integrating retrieval-augmented mechanisms for ambiguous tasks.

The paper "Class-RAG: Content Moderation with Retrieval Augmented Generation" presents a novel framework for enhancing content moderation classifiers used in Generative AI systems. The primary challenge addressed by the authors is the inherent ambiguity in distinguishing between safe and unsafe content, which is complicated by the subtle nuances often present in input samples and the subjective nature of moderation guidelines. Moreover, traditional model fine-tuning approaches face scalability issues and are resource-intensive when repeatedly applied to evolving safety standards and diverse applications.

Key Contributions and Methodology:

• Class-RAG Framework: The proposed solution, Class-RAG (Classification using Retrieval-Augmented Generation), integrates additional context into the decision-making process of LLMs by utilizing a dynamically updateable retrieval library. This library allows for semantic hotfixing, providing a flexible method for risk mitigation without the need for constant model retraining.
• System Architecture: The Class-RAG system consists of four main components: an embedding model, a retrieval library, a retrieval module, and a fine-tuned LLM classifier. Upon receiving an input query, the system retrieves similar positive and negative examples from the retrieval library using an efficient similarity search method, specifically employing Faiss for nearest neighbor detection. The enriched contextual information aids the LLM classification process.
• Embedding Models and Training: The paper leverages the DRAGON RoBERTa as the primary embedding model, but also investigates a variant of WPIE (Whole Post Integrity Embedding) based on XLM-R. The retrieval library is comprised of both in-distribution and external data, facilitating model adaptation to new scenarios with updated retrieval examples without retraining.
• Evaluation and Performance: Empirical studies demonstrate that Class-RAG significantly outperforms traditional fine-tuning models, showing increased robustness against adversarial attacks. Notably, the performance of Class-RAG scales positively with the size of the retrieval library, suggesting that expanding the library is an economical strategy to boost moderation effectiveness.
• Instruction Adherence and Adaptability: The system demonstrates a strong capacity to follow instructions and adapts flexibly to different datasets, illustrated by improvements in performance when leveraging both in-distribution and augmented external libraries.

Discussion and Implications:

This work highlights the advantages of integrating retrieval-augmented mechanisms with LLMs for ambiguous tasks such as content moderation. The ability to dynamically adjust the retrieval library in response to evolving content safety challenges presents a scalable solution to a traditionally resource-intensive problem. The paper also discusses potential limitations like false positives/negatives, biases in data sources, and the reliance on English-language datasets, reiterating the importance of diversifying dataset sources and continuously refining moderation strategies to mitigate bias and enhance robustness.

Overall, Class-RAG provides a versatile, cost-effective approach for content moderation in the Generative AI domain, addressing the need for scalable safety solutions across diverse application environments. Future research directions include extending the framework to handle multi-modal inputs, improving instruction-following for safe samples, and exploring multilingual capabilities.