Dictionary-Aligned Concept Control for Safeguarding Multimodal LLMs

Abstract: Multimodal LLMs (MLLMs) have been shown to be vulnerable to malicious queries that can elicit unsafe responses. Recent work uses prompt engineering, response classification, or finetuning to improve MLLM safety. Nevertheless, such approaches are often ineffective against evolving malicious patterns, may require rerunning the query, or demand heavy computational resources. Steering the activations of a frozen model at inference time has recently emerged as a flexible and effective solution. However, existing steering methods for MLLMs typically handle only a narrow set of safety-related concepts or struggle to adjust specific concepts without affecting others. To address these challenges, we introduce Dictionary-Aligned Concept Control (DACO), a framework that utilizes a curated concept dictionary and a Sparse Autoencoder (SAE) to provide granular control over MLLM activations. First, we curate a dictionary of 15,000 multimodal concepts by retrieving over 400,000 caption-image stimuli and summarizing their activations into concept directions. We name the dataset DACO-400K. Second, we show that the curated dictionary can be used to intervene activations via sparse coding. Third, we propose a new steering approach that uses our dictionary to initialize the training of an SAE and automatically annotate the semantics of the SAE atoms for safeguarding MLLMs. Experiments on multiple MLLMs (e.g., QwenVL, LLaVA, InternVL) across safety benchmarks (e.g., MM-SafetyBench, JailBreakV) show that DACO significantly improves MLLM safety while maintaining general-purpose capabilities.

• The paper introduces the DACO framework that leverages a curated, multimodal concept dictionary and sparse coding for safe control of MLLM activations.
• It details a Sparse Autoencoder initialized with concept vectors for semantically aligned activation steering, demonstrating improved defense rates and minimal utility loss.
• Empirical evaluations across benchmarks validate DACO’s efficiency, achieving up to 98% detoxification with less than 15% extra computational cost.

Dictionary-Aligned Concept Control (DACO): A Principled Approach for Safe Multimodal LLMs

Motivation and Problem Setting

Multimodal LLMs (MLLMs) have enabled powerful systems that integrate visual and language information but remain vulnerable to adversarial prompts, typographic triggers, and other malicious, policy-violating inputs. Classical approaches for improving MLLM safety, such as prompt engineering, post hoc response filtering, and continued finetuning, face scalability, adaptivity, and efficiency bottlenecks. Recently, activation steering—direct intervention in the latent space—has emerged as a promising alternative but typically suffers from narrow concept coverage, unclear calibration of intervention strength, and a lack of semantic grounding in the steering features.

The DACO (Dictionary-Aligned Concept Control) framework directly addresses these pain points by conjoining: (1) a large, curated and partitioned concept dictionary of multimodal activations; (2) training-free sparse coding to express activations as interpretable concept combinations; and (3) a Sparse Autoencoder (SAE) trained and annotated with the dictionary to provide compositional, semantically aligned control of activations in MLLM decoder blocks.

Figure 1: The DACO framework overview—concept dictionary curation, sparse coding, SAE training/annotation, and controllable inference-time activation steering for multimodal detoxification.

DACO: Technical Components

Concept Dictionary Curation

DACO begins by extracting $\sim$15,000 unique concepts from WordNet noun synsets, deduplicating them, and leveraging CLIP similarity to retrieve $\sim$400,000 relevant positive and negative caption-image pairs from CC-3M. Each concept $c$ is thus grounded in multimodal stimuli; the geometric mean of text/text and text/image CLIP similarities is used for robust retrieval, addressing the limitations of simple arithmetic averaging. MLLM activations are then collected on these stimuli for layers of interest in the decoder. Direction vectors per concept are computed via contrastive reading (mean activation over positive vs. negative samples). The resulting concept dictionary is broad, diverse, and densely distributed in the activation space—analyzed quantitatively with neighborhood purity and visually with UMAP.

Figure 2: Concept samples and their corresponding retrieved caption-image stimuli, illustrating the diversity and the multimodal context of the DACO dictionary.

Figure 3: UMAP visualization of the concept vectors; semantically related concepts cluster, justifying localized control.

Safety Labeling by Policy Specification

To enable policy-aligned control, an expert MLLM rates each concept (and its stimuli) with respect to "sensitive/harmful" or "benign/harmless" criteria. The partitioned dictionary supports targeted suppression or promotion of safety-critical concepts at a fine granularity unavailable to previous approaches.

Sparse Coding and Dictionary-Based Intervention

Given a representation $z_\ell$ (e.g., residual stream activation of a decoder token), an elastic net solver is used to find a sparse code with respect to the concept dictionary. Steering is realized by zeroing out coefficients for undesirable concepts, effecting an oblique projection in activation space. Empirical analysis demonstrates that DACO's dictionary is sufficiently overcomplete to support high-fidelity, disentangled decomposition of real MLLM activations.

SAE Training and Semantic Annotation

A principal limitation of prior SAE-based intervention is the lack of semantic interpretability of the learned atoms. DACO initializes the SAE decoder with the normalized concept vectors, augmenting with additional random features as needed. After training (using either L1- or TopK-style sparsity), SAE atoms are automatically annotated: each atom is labeled undesirable/desirable if sufficiently close (in cosine distance) to the centroid of partitioned undesirable/desirable concept vectors.

At inference, DACO then composes steering edits by suppressing coefficients for "undesirable" atoms and amplifying "desirable" ones, yielding highly compositional, semantically controlled activation interventions.

Figure 4: The DACO pipeline—concept curation, dictionary assembly, SAE training with annotation, and compositional steering at inference.

Figure 5: Top activated SAE atoms for adversarial queries; the nearest concept labels provide direct semantic interpretability.

Empirical Results

Benchmark Evaluations

DACO is validated across multiple MLLM architectures (Qwen2.5-VL-7B-Instruct, LLaVA1.6-Mistral-7B, InternVL3.5-8B-Instruct) and evaluated on comprehensive safety (MM-SafetyBench, JailbreakV-28K, with two diverse judges) and utility (fluency, perplexity, MMMU, MM-Vet) metrics.

• Safety: DACO consistently achieves the highest defense (detoxification) rate in every tested category and benchmark, often by large margins (e.g., improving defense rate from 0.75 to 0.98+ over ActAdd or prompting baselines, see Table 1 in the manuscript).
• Utility: There is negligible to minimal degradation in general-purpose scores such as fluency and multiple-choice question answering, and open-ended multimodal benchmarks demonstrate preservation of core model competences.

Importantly, DACO achieves this with less than 15% additional per-token computational cost, far outpacing other activation steering methods (e.g., ~50% cost for MOP). Over-refusal rates (the model rejecting benign but safety-tinged prompts) remain low, similar to the base model, attesting to the precision of DACO's controllability.

Figure 6: Example adversarial query, vanilla versus DACO-controlled MLLM response—DACO eliminates harmful content with retained informativeness.

Figure 7: The effect of SAE-to-concept annotation threshold—stricter annotation improves safety up to the optimal point, with trade-off in utility when widened too much.

SAE Analysis and Training Efficacy

Initialization with the concept dictionary leads to higher fraction of variance explained, reduced dead neuron rates, and improved directional alignment in SAE reconstructions compared to random or dense initializations. Ablation experiments validate the essential role of the multimodal, broad-coverage dictionary for both control and interpretability.

Figure 8: LoRA finetuning with subspace-clustered SAE atoms—SAE-initialized LoRA adapters converge faster, supporting transferability of DACO's representation.

Implications and Theoretical Significance

DACO concretely demonstrates the benefit of dictionary-aligned, semantically partitioned, sparse representations for safe and interpretable control of MLLMs. The approach generalizes both classical activation addition and recent SAE-based steering. By grounding high-dimensional activations in dense, labeled multimodal concepts, DACO precisely addresses the desiderata for transparent, auditable, and real-time controllable interventions in large frozen models.

The demonstrated locality, orthogonality, and compositionality of the curated concepts provide empirical support for the linear representation hypothesis and the role of local subspaces in meaning organization in MLLM residual spaces. The empirical finding that SAE atoms inherit locality and monosemanticity from the concept dictionary suggests future routes for scalable, self-annotating, and interpretable representation learning.

Practically, DACO’s concept-aligned steering opens paths for content moderation, dynamic policy adaptation, and robust model release pipelines, minimizing risks without sacrifice of general utility. The public release of the dictionary, dataset, and model artifacts will aid further research into scalable, transparent alignment for foundation models.

Future Directions

• Extension to New Modalities / Domains: DACO's pipeline can incorporate domain-specific retrieval and annotation, facilitating targeted safety or capability adaptation.
• Dynamic/Probabilistic Steering: The compositional control mask can be dynamically tuned, e.g., adjusting the intervention window or steering mask size to trade off safety and utility per input, or learning optimal schedules online.
• Robustness Improvements: Further investigation into adversarial adaptation, reverse engineering of dictionary-based control, and model-wide property guarantees under DACO-style intervention.

Conclusion

DACO establishes a principled, empirically validated activation steering framework for MLLM safety. By bridging large, human-interpretable dictionaries and efficient, semantically-annotated SAEs, it enables fine-grained, policy-grounded, and compositional control, minimizing adverse safety–utility tradeoffs. The framework’s structure also paves the way for interpretable and auditable alignment at scale, with significant theoretical and practical ramifications for the development and deployment of accountable, safe multimodal AI systems.