Where Concept Erasure Should Occur: Concept-Layer Alignment in Text-to-Video Diffusion Models

Abstract: Text-to-video diffusion transformers encode semantic information unevenly across model depth, which constrains effective concept erasure. We identify a representational bottleneck, termed concept-layer topological alignment, under which target concepts exhibit higher separability at certain representational depths. Outside these depths, concept and non-target signals remain strongly entangled, limiting the effectiveness of depth-specific erasure. This observation reframes concept erasure as the problem of identifying representational depths where concept-non-target separation naturally emerges. Motivated by this structural constraint, we introduce CLEAR, a separability-driven optimization framework for concept erasure that explicitly enforces concept-layer alignment. CLEAR operationalizes this principle by formulating layer selection as an optimization problem over concept-non-target separability, rather than relying on layer-agnostic or heuristic choices. To enable this, we introduce a separability-aware objective that favors layers exhibiting stronger concept-non-target separation. Experiments on large-scale text-to-video diffusion models demonstrate that enforcing concept--layer alignment leads to more precise concept suppression while preserving overall generative quality.

• The paper introduces CLEAR, a systematic approach that optimizes intervention layer selection for effective concept erasure using learnable depth preferences and sparse feature decomposition.
• The paper employs contrastive training and alternating bilevel optimization to achieve targeted suppression while maintaining high non-target fidelity across text-to-video models.
• The paper demonstrates CLEAR's efficiency by significantly reducing undesired concept presence (e.g., from 70.4% to 7.8%) and improving aesthetic quality metrics.

Concept-Layer Alignment in Text-to-Video Diffusion: Detailed Analysis of CLEAR

Background and Motivation

Text-to-video (T2V) diffusion models have rapidly advanced in generative realism, temporal coherence, and scene composition. However, semantic representations within these architectures are deeply depth-dependent; semantic features undergo progressive transformation and separation across layers. Concept erasure—suppressing specific target concepts from generated outputs—has become critical for model safety, copyright compliance, and ethical deployment. Prior approaches largely ignored depth structure, operating under the assumption that concept suppression propagates uniformly regardless of intervention layer. This results in semantic leakage, degraded non-target fidelity, and inefficacious suppression for compositional or entangled concepts.

Concept-Layer Topological Alignment

The paper rigorously characterizes a structural bottleneck intrinsic to T2V diffusion models: concepts are not linearly isolatable everywhere in the network. Instead, concept-layer topological alignment governs the emergence of clear separation between target and non-target representations at particular layers. Visual analysis and linear probe diagnostics confirm that concepts such as general objects, fine-grained identities, and safety-sensitive attributes achieve maximal disentanglement at distinct, concept-dependent depths. This provides a foundation for principled layer selection in concept erasure.

CLEAR: Separability-Driven Layer Optimization

CLEAR (Concept-Layer Erasure Alignment fRamework) operationalizes the layer selection problem as an optimization over concept-non-target separability, eschewing heuristic or layer-agnostic placement. The framework incorporates:

• Learnable Depth Preferences: Intervention depth is modeled as a probability distribution over layers, optimized via the Gumbel-Softmax mechanism for differentiable layer selection.
• Sparse Feature Decomposition: Dense intermediate activations are projected into sparse semantic feature spaces using Sparse Autoencoders (SAEs), facilitating selective attenuation of concept-aligned directions while preserving unrelated semantics.
• Contrastive Training Objectives: Positive and negative prompt pairs inform specificity and shared feature masks, enabling contrastive loss-driven optimization (Lcon) that quantifies concept-non-target separability.

Alternating bilevel optimization ensures co-evolution of depth preferences and SAE parameters, automatically converging to concept-specific optimal intervention layers.

Empirical Evaluation and Results

Extensive comparative experiments on Wan2.2-5B and Cog VideoX-2B T2V diffusion models span general objects, safety-regulated concepts, celebrity identities, and abstract artist styles. CLEAR consistently delivers superior suppression efficacy and generative fidelity:

• Suppression Strength: For example, erasing "french horn" on Wan2.2-5B, CLEAR reduces the generative rate from 70.4% (baseline) to 7.8%, outperforming T2VUnlearning (26.8%) and inference-based baselines (NegPrompt, SAFREE).
• Preservation Metrics: CLEAR maintains imaging and aesthetic quality (e.g., 0.7025 MUSIQ score vs. 0.6652 for training-based baseline), and achieves competitive semantic consistency (ViCLIP similarity).
• Layer Localization: Ablations confirm CLEAR's depth search converges to concept-dependent layers (e.g., Block 6 for parachute, Block 18 for nudity), without exhaustive brute-force search, offering an order-of-magnitude efficiency improvement.
• Generalization and Robustness: CLEAR generalizes across evasive prompts ("Ring-A-Bell" benchmark), multiple concept erasure (joint intra/inter-category suppression), and transfer to text-to-image and CLIP-based encoders.

CLEAR's SAE-based decomposition outperforms steering-vector baselines by minimizing entanglement, and single-layer interventions are consistently more optimal than multi-layer approaches due to reduced interference in sparse feature space.

Theoretical and Practical Implications

CLEAR reframes concept erasure as a joint optimization problem over where semantic separation arises and how suppression can be performed without degrading unrelated content. The findings have direct relevance for:

• Mechanistic Interpretability: They reveal that semantic features are dynamically reorganized across model depth, suggesting future directions for cross-model concept alignment and sparse representation learning.
• Safety and Controllability: CLEAR's efficacy in identity and nudity suppression advances practical policy-compliant generative modeling. Its inference-time approach provides scalable deployment without full retraining.
• Transferability: The framework’s reliance on text encoder representations enables extension to diverse generative paradigms (e.g., text-to-image, CLIP-based encoders).

Limitations and Future Directions

Current CLEAR implementations require separate optimization per target concept, challenging scalability for large concept libraries and complex policy regimes. Further work is needed in parameter sharing, multi-concept feature composition, and extending CLEAR to more abstract or relational concepts. Additionally, broader validation across architectures—including alternative diffusion backbones and text encoders—is future work.

Conclusion

"Where Concept Erasure Should Occur: Concept-Layer Alignment in Text-to-Video Diffusion Models" (2605.25941) introduces a robust, principled framework for targeted concept suppression in T2V diffusion transformers. By aligning topological layer selection with semantic separability and leveraging sparse feature decomposition, CLEAR achieves precise, efficient, and non-destructive erasure across heterogeneous concept categories. These results underpin new directions for model interpretability, safety-centric generative modeling, and adaptive intervention strategies for large-scale diffusion systems.