Continual Personalization for Diffusion Models

Abstract: Updating diffusion models in an incremental setting would be practical in real-world applications yet computationally challenging. We present a novel learning strategy of Concept Neuron Selection (CNS), a simple yet effective approach to perform personalization in a continual learning scheme. CNS uniquely identifies neurons in diffusion models that are closely related to the target concepts. In order to mitigate catastrophic forgetting problems while preserving zero-shot text-to-image generation ability, CNS finetunes concept neurons in an incremental manner and jointly preserves knowledge learned of previous concepts. Evaluation of real-world datasets demonstrates that CNS achieves state-of-the-art performance with minimal parameter adjustments, outperforming previous methods in both single and multi-concept personalization works. CNS also achieves fusion-free operation, reducing memory storage and processing time for continual personalization.

• The paper introduces CNS, a framework that incrementally personalizes diffusion models by finetuning a sparse set of concept-specific neurons.
• It distinguishes between base and general neurons to focus finetuning on those uniquely representing user concepts while keeping generic capabilities intact.
• Experimental results demonstrate state-of-the-art alignment metrics with zero additional memory overhead and effective mitigation of catastrophic forgetting.

Continual Personalization for Diffusion Models: Concept Neuron Selection (CNS)

Introduction

The paper introduces Concept Neuron Selection (CNS), a framework for continual personalization in text-to-image diffusion models. CNS addresses the challenge of incrementally adapting diffusion models to user-specific concepts while mitigating catastrophic forgetting and preserving zero-shot generation capabilities. The method leverages neuron selection in cross-attention layers to identify and fine-tune a sparse set of concept-specific neurons, enabling efficient and scalable continual learning without the need for fusion or storage of multiple model weights.

Motivation and Related Work

Personalization in Latent Diffusion Models (LDMs) is essential for generating user-specific content, but existing approaches—such as prompt tuning, DreamBooth, and LoRA-based fusion—struggle with multi-concept composition, overfitting, attribute binding, and computational inefficiency. Prior works on LoRA fusion [gu2024mix], orthogonal adaptation [po2024orthogonal], and continual diffusion [smith2023continual] require storing and fusing multiple weights, leading to memory and time overheads and concept confusion. CNS circumvents these limitations by directly operating at the neuron level, inspired by neuron selection and pruning techniques in LLMs [wang2022finding, dalvi2019one, zhang2021moefication] and their recent adaptation to diffusion models [liu2023cones, chavhan2024conceptprune].

CNS Framework Overview

CNS consists of four main components: base neuron selection, general neuron selection, concept neuron separation, and incremental finetuning. The process is illustrated in the following overview.

Figure 1: CNS framework for neuron selection and incremental finetuning in continual personalization.

Base Neuron Selection

For each concept, CNS computes importance scores for neurons in the key and value mappings of cross-attention layers using the element-wise product of weight magnitudes and the $\ell_2$ norm of text embeddings derived from concept image captions. Neurons exceeding a threshold are selected as base neurons. Aggregation across multiple concept images yields a binary mask representing the base neurons.

General Neuron Selection

Empirical analysis reveals that a significant fraction (~53%) of base neurons are shared across diverse prompts, indicating their role in general image generation rather than concept specificity.

Figure 2: Overlap percentage of base neurons across images, demonstrating the prevalence of general-purpose neurons.

General neurons are identified by aggregating neuron masks across a set of calibration prompts, isolating those consistently activated for generic image synthesis.

Concept Neuron Separation

Concept neurons are defined as the set difference between base neurons and general neurons. This logical operation ensures that only concept-specific neurons are selected for finetuning, while general neurons remain fixed to preserve the pretrained model's zero-shot capabilities.

Incremental Finetuning and Regularization

During continual learning, CNS updates only the concept neurons for each new concept. A regularization loss penalizes deviations in overlapping neurons between the current, previous, and pretrained weights, effectively mitigating catastrophic forgetting and maintaining zero-shot performance. The overall objective combines the standard diffusion loss, DreamBooth-style subject regularization, and CNS regularization.

Experimental Results

Qualitative Evaluation

CNS demonstrates superior performance in both single- and multi-concept personalization compared to Mix-of-Show, Orthogonal Adaptation, and Continual Diffusion. CNS-generated images maintain alignment with all specified concepts and input prompts, avoiding attribute binding and concept vanishing.

Figure 3: Qualitative comparison of CNS and baselines in continual personalization tasks.

Quantitative Evaluation

CNS achieves state-of-the-art results in CLIP-based image and text alignment metrics, particularly excelling in multi-concept scenarios. Unlike fusion-based methods, CNS incurs zero additional memory and computational time for concept fusion, as only neuron masks are stored and no test-time optimization is required.

Method	Single Concept CLIP-I	Single Concept CLIP-T	Multi-Concept CLIP-I	Multi-Concept CLIP-T	Memory (MB)	Time (s)
Textual Inversion	72.76	72.69	65.30	65.00	0 / 0	0
Custom Diffusion	67.88	74.92	65.87	68.70	3547 / 0	10
Mix-of-Show	75.86	75.75	65.26	70.62	62852 / 0	727
Orthogonal Adaptation	74.67	74.87	66.37	69.20	5663 / 3167	42
Continual Diffusion	71.82	66.12	66.15	60.30	2461 / 4747	10
CNS	74.88	76.95	67.21	79.22	0 / 0	0

CNS updates only ~0.13% of total parameters per concept, with concept neuron overlap (mIoU) averaging 0.24 across concepts, and 0.02 across all concepts in a set.

Ablation Studies

Ablation experiments confirm that both targeted neuron selection and regularization loss are critical for preventing catastrophic forgetting and achieving high personalization performance. Random neuron selection or omission of regularization leads to significant degradation in alignment scores and concept retention.

Human Evaluation

Human studies corroborate the quantitative findings, with CNS preferred by users for both image and text alignment in the majority of cases.

Implementation Considerations

CNS is implemented on Stable Diffusion (SD-1.5), with concept neuron selection applied to cross-attention layers. Training requires only a few hundred steps per concept and minimal computational resources. The method is compatible with region control and can be integrated with other text-to-image models.

Implications and Future Directions

CNS provides a scalable, efficient solution for continual personalization in diffusion models, eliminating the need for fusion and storage of multiple LoRA weights. The approach is robust to catastrophic forgetting and preserves zero-shot capabilities, making it suitable for real-world applications where user concepts evolve over time. The neuron selection paradigm can be extended to knowledge editing, unlearning, and other modalities beyond image synthesis.

Conclusion

Concept Neuron Selection (CNS) advances continual personalization in diffusion models by identifying and incrementally finetuning concept-specific neurons. CNS achieves state-of-the-art performance with minimal parameter updates, fusion-free operation, and strong robustness to catastrophic forgetting. The framework sets a new standard for efficient, scalable, and flexible personalization in generative models, with broad applicability to future research in knowledge editing and continual learning.