LLaMA Guard: Open-Source Safety Classifier

• LLaMA Guard is an instruction-tuned LLM safety classifier that moderates both user prompts and model responses using a customizable risk taxonomy.
• It employs dual classification modes with fine-tuning and few-shot learning to rapidly adapt and detect hazardous content in real-time conversational pipelines.
• Empirical benchmarks show competitive F1 scores against proprietary moderation APIs, highlighting its effectiveness and adaptability for open-source deployments.

LLaMA Guard is an instruction-tuned, LLM-based safety classifier, originally built on Llama 2/3 architectures, designed to moderate both inputs (user prompts) and outputs (model responses) in human–AI conversations. By jointly leveraging fine-tuned LLMs and a customizable risk taxonomy, LLaMA Guard addresses real-time risk detection in conversational pipelines, offering a released, adaptable model for community use. Its design allows for input–output duality, taxonomy expansion, and rapid adaptation, establishing it as a principled open-source alternative to proprietary moderation APIs and detector ensembles (Inan et al., 2023, Grattafiori et al., 31 Jul 2024).

1. Model Architecture and Classification Workflow

LLaMA Guard is based on an instruction-tuned Llama 2-7B (and, in subsequent iterations, Llama 3 8B), fine-tuned explicitly for content safety risk detection. The model implements two primary classification modes:

• Prompt Classification: Given a system prompt enumerating safety risk categories and a current conversational history, the model flags the latest user input as SAFE or UNSAFE, and, if unsafe, cites the violating category IDs.
• Response Classification: Analogous procedure, but focused on the last LLM output within a conversation.

Input consists of: (a) explicit guidelines (category list, customizable per deployment), (b) conversational context, and (c) optional few-shot exemplars. Output format is standardized:

SAFE or UNSAFE 〈list of category-IDs〉

The model is instruction-tuned to follow these formats with high fidelity. LLaMA Guard optionally supports few-shot or zero-shot use with arbitrary or user-specified taxonomies at inference by adjusting the presented instruction block (Inan et al., 2023).

2. Safety Risk Taxonomy and Task Customization

LLaMA Guard’s core safety framework centers on a categorical taxonomy spanning high-risk content areas. The initial “in-policy” taxonomy comprises seven classes:

• O1: Violence & Hate
• O2: Sexual Content
• O3: Guns & Illegal Weapons
• O4: Regulated or Controlled Substances
• O5: Self-Harm
• O6: Criminal & Dangerous Acts
• O7: General Unsafe Content

Each class is informally defined rather than thresholded mathematically. For each input or response, the model evaluates whether it violates one or more categories under these guidelines, aggregating any violation into the output as a structured list.

LLaMA Guard’s instruction-based tuning enables dynamic adjustment of the risk taxonomy, supporting adaptation to bespoke policy requirements, regulatory regimes, or deployment-specific threat models. For example, by altering the instructional guideline template or providing few-shot demonstrations, the taxonomy can be re-scoped (e.g., finer-grained child-safety subcategories, regulatory-compliance labels, etc.) (Inan et al., 2023).

3. Data Collection, Training Regimen, and Model Adaptability

The LLaMA Guard training data is a high-quality, manually curated corpus built for both prompt and response classification. It is relatively low-volume compared to full language modeling sets but is annotated to maximize coverage of realistic and adversarial conversational risk scenarios. This dataset includes standard user-agent exchanges as well as edge cases, such as carefully engineered jailbreak prompts and model outputs intended to evade naive moderation.

Supervised instruction tuning is performed by appending task-specific guidelines to each input. The loss objective is a multi-class (multiple category) cross-entropy over the output sequence, combined with strict instruction adherence through format engineering.

LLaMA Guard supports both continued fine-tuning and run-time adaptation:

• Users can further fine-tune the model on expanded datasets or alternate taxonomies.
• At inference, users can add new risk categories or alter instruction styles via in-context prompting or few-shot examples without re-training. This flexibility differentiates LLaMA Guard from fixed-policy black-box moderation APIs and enables rapid adaptation to evolving conversational safety needs (Inan et al., 2023).

4. Empirical Performance and Benchmarking

LLaMA Guard’s core model was tested on prominent benchmarks—including the OpenAI Moderation Evaluation dataset and ToxicChat—where its safety detection accuracy matches or exceeds existing moderation solutions. Performance metrics reported in the literature include:

• Classification F1 on challenge benchmarks competitive with proprietary moderation APIs (e.g., OpenAI Moderation, Perspective, Azure Content Safety).
• Successful identification of both prompt- and response-specific hazards, distinguishing itself from single-sided classifiers unable to separate input vs. output risk signals. Empirical results on input/output guardrailing demonstrate both low false negative rates (missed unsafe content) and low false positives (unnecessarily blocked benign content), subject to the coverage and specificity of the taxonomy used (Inan et al., 2023).

5. Comparative Landscape and Ecosystem Integration

Relative to API-based moderation (Perspective, OpenAI Moderation), LLaMA Guard exhibits several strengths:

• Model size: 7B parameters enables both high performance and feasible on-premise/offline deployment.
• Instruction tuning allows for granular task customization, zero-shot/few-shot expansion, and prompt engineering.
• Open weights: The released models can be self-hosted, extended, or distilled for downstream moderation pipelines. Its primary limitations are inherited from LLM safety classifiers in general—static taxonomy boundaries, need for ongoing dataset refreshes, and known vulnerabilities under coordinated adversarial attack (see for instance PRP-style prefix attacks (Mangaokar et al., 24 Feb 2024)). Nonetheless, LLaMA Guard forms the baseline for recent multistage and hybrid guardrail architectures.

6. Broader Impact, Limitations, and Future Directions

LLaMA Guard’s open release catalyzed the development of a rich guardrail ecosystem, enabling:

• Off-line, open-source moderation for conversational agents.
• Flexible integration in both input and output filtering stages.
• Efficient extension to additional categories, regulatory policies, and domain-specific use cases.

Limitations include incomplete coverage for non-English languages (addressed in later works via multilingual data and fine-tuning), static resistance to model-level jailbreaks (requiring adversarial hardening), and possible over-reliance on thresholded categorical outputs. The model’s effectiveness is also bounded by the specificity and breadth of its taxonomies—the more nuanced or recently emergent harms may be missed absent retraining.

Ongoing research in this area targets dynamic, adaptive, and multilingual extensions of LLaMA Guard, as well as integration with adversarial defense methods such as anomaly detectors and certification-based post-hoc filters (Mangaokar et al., 24 Feb 2024, Fedorov et al., 18 Nov 2024, Joshi et al., 3 Aug 2025). Empirically, LLaMA Guard serves as a foundation for community-driven moderation, but robust defense against the evolving spectrum of LLM risks continues to require layered, adaptive systems.