kNNGuard: Turning LLM Hidden Activations into a Training-Free Configurable Guardrail

Abstract: LLMs are increasingly deployed in domains requiring guardrails to detect unsafe, off-topic, or adversarial prompts. Existing guardrails predominantly rely on fine-tuning to build classifiers, which often suffer from low generalization and high inference latency. We present kNNGuard, a training-free guardrail that utilizes the activation space of an off-the-shelf LLM. Given a small bank of 50 safe and unsafe prompts, kNNGuard extracts hidden activations and performs multi-layer kNN fusing activation-space and embedding-space scores for classification. Across six domains spanning topical and security prompts, kNNGuard achieves competitive or superior F1 compared to fine-tuned state-of-the-art guardrails while running 2.7x faster than the best comparable guardrail, and 10x faster than a fine-tuned safety classifier without gradient updates or fine-tuning. Domain adaptation requires only updating the labeled bank, which can be constructed in under 10 seconds and several orders of magnitude faster than established guardrails. We also analyze the impact of system prompts, layer selection, and integration into production LLM pipelines as a configurable, low-latency guardrail.

• The paper introduces a training-free guardrail using frozen LLM hidden activations and multi-layer kNN to efficiently detect unsafe prompts.
• It fuses activation and embedding risk scores with an adaptive rule, achieving an average F1 of 87.4% and low false positive rates across domains.
• kNNGuard enables rapid domain adaptation with bank construction over 3000× faster than fine-tuning, ensuring robust safety in LLM deployments.

Training-Free Guardrails for LLMs: The kNNGuard Framework

Introduction and Motivation

The proliferation of LLMs in mission-critical domains has underscored the urgent requirement for robust, low-latency guardrails capable of detecting unsafe, off-topic, and adversarial prompts. Current paradigms, especially those relying on fine-tuned classifiers, suffer from high operational costs, domain inflexibility, and inferior generalization to previously unseen attack vectors. Lightweight embedding-based approaches, while more efficient, typically struggle with nuanced safety distinctions and exhibit unacceptably high false positive rates against distributional shift and adversarial manipulation.

"kNNGuard: Turning LLM Hidden Activations into a Training-Free Configurable Guardrail" (2607.02072) introduces a training-free guardrail based on leveraging frozen LLM hidden activations—rather than output representations or fine-tuned classifier heads—and fusing them with standard embedding-based metrics. This decision surface, defined by a small labeled prompt bank and multi-layer kNN, allows efficient, configurable deployment with minimal setup, enabling domain adaptation and model-agnostic integration.

Architectural Overview and Methodology

kNNGuard is built upon the insight that LLM internal activations encode latent features related to prompt safety, topicality, and adversarial intent. The framework operates in two phases: Bank Construction and Inference. During bank construction, a small number (typically 50 per class) of labeled prompts are forward-passed through the frozen LLM. Last-token activations are extracted from multiple transformer layers, with each layer's contribution weighted via a Fisher separability metric to maximize class-conditional distinction. In parallel, prompts are encoded by a lightweight sentence embedding model (e.g., MiniLM). At inference, incoming prompts are processed identically and their multi-layer activations and embeddings are compared via cosine kNN against the bank in both activation and embedding space.

To combine the activation-derived and embedding-derived risk scores, kNNGuard implements an adaptive fusion rule that selects the more confident branch if there is sufficient confidence gap; otherwise, it blends scores in proportion to confidence. The default threshold for unsafe detection is set at $\tau=0.5$, which empirically optimizes the F1-precision-recall trade-off. Notably, the full guardrail operates without any model parameter updates or gradient steps; it is strictly non-parametric, with domain adaptation realized through prompt bank replacement.

Figure 2: t-SNE projection of prompts in Medical and Coding domains comparing Embedding-kNN and kNNGuard FE.

This design leverages layer-wise activation structure to achieve greater separability than surface-level embeddings Figure 2, confirmed quantitatively by superior Silhouette scores.

Empirical Evaluation: Effectiveness, Robustness, and Latency

The framework's performance was benchmarked across six evaluation domains: Coding Instructions, Coding Outputs, Medical, General Safety, Jailbreak, and Prompt Injection. Comparative baselines include fine-tuned classifiers (e.g., Llama Nemotron Topic Guard V1, Nemotron Safety Guard V2), Prompt Guard 2, and a purely embedding-based kNN model.

kNNGuard FE (Fused Ensemble) achieves an average F1 of 87.4% and FPR of 12.9%, outperforming all baselines both in accuracy and error decomposition (i.e., balanced FPR and FNR) while achieving an average per-prompt latency of 45.9 ms, which is 2.7× faster than the leading fine-tuned classifier and 10× faster than Nemotron Safety Guard V2.

Figure 4: Total error decomposition across topical domains. Lower total bar height indicates a better overall operating point.

Figure 6: Total error decomposition across security and safety domains. Lower total bar height indicates a better overall operating point.

kNNGuard's error decompositions in both topical and adversarial settings (Figures 2, 3) show that no single error class dominates, reflecting genuine robustness. In contrast, fine-tuned classifiers exhibit high sensitivity to domain shift and may display prohibitive FPR (e.g., 97.4% for Nemotron Topic Guard V1 in jailbreak detection).

Domain adaptation is rapid and practical: constructing a kNNGuard bank for 100 samples (50 per class) requires under 10 seconds, whereas LoRA fine-tuning on equivalent data exceeds six hours, a $>$3000× speedup for real-world deployment scenarios.

Figure 3: Time comparison between LoRA fine-tuning and kNNGuard bank construction.

These empirical results indicate that kNNGuard not only matches or exceeds the statistical performance of established fine-tuned methods but also offers tangible operational advantages for frequent or domain-specific redeployment.

Analysis: Conditioning, Representation, and Model Generality

In-depth analysis reveals that LLM internal activations under system prompt conditioning yield more separated activation manifolds between safe and unsafe classes. Omission of the system prompt increases recall but at the expense of precision, rapidly raising false-positive rates, especially in semantically dense domains Figure 8.

Figure 5: F1, Recall, FPR, and FNR scores compared between kNNGuard with and without a system prompt, averaged across all domains.

Both t-SNE visualizations and Silhouette coefficients confirm that learned activation geometry provides superior clustering compared to sentence embeddings. Further, results demonstrate that kNNGuard FE generalizes across multiple backbone models (Llama-3.1-8B-Instruct, Phi-4-mini-instruct, Mistral-7B-Instruct-v0.3, and Gemma-4-12B variants) with stable performance on both topical and adversarial tasks (Figures 9, 10), suggesting that the approach is not confined to a specific LLM architecture or pretraining regime.

Practical and Theoretical Implications

From a deployment perspective, kNNGuard offers a practical framework for production guardrails, especially in environments requiring low-latency response and rapid domain adaptation. Its ability to reconfigure solely through bank updates and optional system prompt modifications makes it amenable to session-based or task-specific guardrail instantiation. The training-free approach circumvents the cost and rigidity of model retraining or fine-tuning, directly addressing maintenance overhead in large, multi-domain LLM deployments.

Theoretically, kNNGuard provides empirical support for the hypothesis that LLM activation geometry, shaped by both pretraining and system prompt conditioning, encodes latent safety and topicality structure. The success of multi-layer activation fusion and adaptive risk scoring further suggests rich, class-conditional information is distributed across the transformer's depth, in line with recent mechanistic interpretability findings.

Future Directions

Promising extensions include continual or online adaptation of the bank to handle concept drift, adversarial attack evolution, and implicit distributional changes. The approach could be augmented by richer geometric metrics or advanced fusion strategies, and synergistically combined with other interpretability-derived signals. Application to multimodal, retrieval-augmented, or agentic AI pipelines would further validate the framework's generality and usefulness in safety-critical LLM systems.

Conclusion

kNNGuard establishes a compelling paradigm for training-free, highly adaptive, and low-latency LLM guardrails. By exploiting frozen model activations and small labeled banks, it achieves superior or competitive statistical and operational performance relative to fine-tuned and embedding-only baselines, with strong evidence for practical deployment and domain generality. This methodology advances the field toward more flexible and theoretically grounded approaches to LLM oversight, robust to both adversarial challenge and rapid real-world evolution.