Personalization Editing in Deep Models

• Personalization editing is a framework that updates deep models with targeted, user-specific modifications while maintaining general-purpose behavior.
• Techniques involve localized parameter updates like low-rank adapters, rank-one methods, and closed-form solutions, ensuring efficient edits with minimal resource overhead.
• Evaluation protocols assess factual recall, drift resistance, and efficiency, addressing challenges such as mode collapse and compositional edits in both language and vision domains.

Personalization editing is a research area that seeks to endow deep generative and discriminative models with the ability to incorporate, revise, or remove highly targeted, user-specific information, without degrading general-purpose capabilities or requiring large-scale retraining. In both language and vision domains, personalization editing encompasses algorithmic techniques, theoretical criteria, and system-level optimizations to support precise, efficient, and robust user-driven model modifications. This article details the formal problem, methodologies, system design considerations, evaluation practices, and contemporary limitations in personalization editing, emphasizing LLMs and image generative models.

1. Formal Problem Setting and Personalization Editing Frameworks

Personalization is framed as a model editing problem: given a pretrained parameterization $f_\theta$ and a set of user-specific preferences or factual modifications (edits), compute a (possibly local) update to the parameters, $\theta \mapsto \theta^*$, so that the model emits a target output $y^*$ for a small, specified set of inputs $\mathcal{X}_e$, while preserving behavior everywhere else. This constraint is formalized as:

$$\begin{aligned} &\min_{\theta^*}\;\mathbb E_{e\sim\mathcal E}\,\mathbb E_{(x,y^*)\sim(\mathcal X_e,\mathcal Y_e^*)} \mathcal L\bigl(f_{\theta^*}(x),\,y^*\bigr) + \lambda\|\theta^*-\theta\|^2 \ &\text{s.t.} \quad f_{\theta^*}(x) = f_\theta(x) \quad \forall x\notin \mathcal{X}_\mathcal{E} \end{aligned}$$

as in (Huang et al., 15 Dec 2025). A key challenge is balancing high fidelity on the edited region with minimal drift elsewhere, thus avoiding catastrophic forgetting or overfitting.

Preference representation involves clustering natural-language variants of both the subject and object of interest, populating a set of queries that capture the semantic range of user preferences and implicit mentions. Clustering is operationalized by leveraging LLMs (e.g., Claude-Sonnet-4) to generate sets of synonyms for both input and target phrases:

$$\mathcal{C}_e = S_e \times O_e = \{(s_i,r,o_j^*)\mid i,j=1,...,K\}$$

Model editing is then implemented via localized parameter updates at precisely diagnosed FFN or attention layers (FT-L, FT-M), low-rank adapters (LoRA), or closed-form methods (ROME). The overall design is to ensure parameter efficiency: the number of updated parameters per edit is much smaller than the model’s total capacity, yielding per-edit latencies of 0.2–2 s versus >10 s for full fine-tuning (Huang et al., 15 Dec 2025).

2. Knowledge Editing Methodologies in LLMs

Personalization editing for LLMs centers on efficient, targeted adjustment of internal knowledge representations, typically fact triples or preference-laden behaviors. Techniques include:

• Zeroth-order, quantized gradient estimation: MobiEdit eschews full-precision backpropagation, instead applying perturbations in a selected subspace ($v$ in a value vector) and estimating directional derivatives via quantized forward passes. The forward-only estimator is resilient to quantization noise, compatible with INT8/INT16 mobile NPUs, and enables in-situ editing of 3B-param LLMs on 12GB RAM devices (Lu et al., 5 Jun 2025).
• Rank-one and low-rank weight updates: PALETTE applies a closed-form, rank-one update derived from the ROME algorithm to substitute outputs for self-referential queries, enabling durable adjustments of personality traits. The rank-one matrix $\Delta$ is computed so that for a set of query encodings $k_e$ and intended trait vectors $v_e$, $W \mapsto W + \Delta$ maps the query to the desired output (Hwang et al., 17 Feb 2025).
• Preference clustering and augmentation: Editing achieves persistence across query rewordings and implicit variants by expanding single preferences into clusters of paraphrased and semantically similar templates, supplying broad coverage and enhancing generalization in downstream QA or dialog (Huang et al., 15 Dec 2025).
• On-device resource-efficient systems: MobiEdit further accelerates editing with early stopping (validation prompt with thresholded confidence) and prefix caching (KV-cache for prompt tokens), reducing memory, energy, and latency by factors of 7.6×, 14.7×, and 3.6×, respectively, compared to conventional editing (Lu et al., 5 Jun 2025).

3. Personalization Editing in Diffusion and GAN Generative Models

In image and video domains, personalization editing addresses identity, concept, or attribute customization with mechanisms for disentanglement and robust editing:

• Stage-wise or subspace editing: ProSpect fragments the textual conditioning space into multiple stage-specific embeddings, each controlling a distinct generation stage (layout, content, material). Inverting a reference image into n per-stage tokens supports localized or attribute-specific editing by swapping only relevant embedding bins (Zhang et al., 2023).
• Key-locked rank-one editing: Perfusion modifies cross-attention Value matrices via dynamic rank-1 updates, while locking the Key matrices to superordinate class responses. This avoids catastrophic overfitting, enables multi-concept composition, and allows fine-grained Pareto control between identity and prompt compliance at inference (Tewel et al., 2023).
• GAN subspace modeling and latent manipulation: GlassesGAN discovers accessory-specific subspaces (e.g., for glasses) using synthetic appearance discovery and principal component analysis in StyleGAN’s latent space. Personalized edits are precise vector walks in this subspace, initialized via a template-aligned difference code and adjusted to target mask area thresholds (Plesh et al., 2022).
• Attention-regularized multi-entity editing: DynASyn aligns subject tokens to class priors via cross-attention map regularization and employs SDE-based prompt and image augmentation to enable realistic dynamic interaction synthesis, robust to overfitting in few-shot or single-shot regimes (Choi et al., 22 Mar 2025).
• Plug-in fine-tuning and spatial bias correction: DreamSteerer augments personalized T2I pipelines with editability-driven score distillation and spatially guided mode-shifting, restoring editability in source-image-conditioned tasks impaired by concept collapse or prior loss after naive personalization (Yu et al., 15 Oct 2024).

4. Evaluation Protocols and Benchmarks

Comprehensive evaluation of personalization editing encompasses both factual (or user intent) accuracy and minimal negative impact on general skills:

• Targeted accuracy: Efficacy scores for explicit, paraphrased, and implicit preference recall on rigorously constructed benchmarks such as User Preference Question Answering (UPQA) (Huang et al., 15 Dec 2025).
• Generalization and drift resistance: Test cases using multi-turn dialog or implicit queries, with measurement using acknowledgment rates or minimal drop on standard tasks (BoolQ, NaturalQuestions, GSM8K) (Huang et al., 15 Dec 2025).
• Efficiency metrics: Time, memory, and energy consumption per edit (e.g., MobiEdit: 6.2 GB/0.014–0.024 J/1200–2000 s per edit vs. prior 46 GB/0.25–0.63 J/4100–11300 s) (Lu et al., 5 Jun 2025).
• Perceptual and attribute alignment: In T2I and GAN frameworks, CLIP-based similarity, text/image alignment, and human preference ratings are standard (Zhang et al., 2023, Plesh et al., 2022, Yu et al., 15 Oct 2024).
• User studies with expert raters: Both language (personality alignment, consistency, tone) and vision (identity, realism, overall quality) tasks are human-judged, supporting robust statistical comparison (Hwang et al., 17 Feb 2025, Zhang et al., 2023, Plesh et al., 2022).

5. Applications and System-Level Design

Personalization editing has broad application in both language and vision tasks:

Domain	Model/Edit Type	Application Example
LLM	On-device quantized editing	Mobile assistants with real-time adaptation
LLM	Rank-one knowledge/personality	Durable trait or fact shift
T2I Diff.	Stage-wise embedding edit	Material, style, or layout attribute swap
GAN	Subspace discovery/inversion	Accessory personalization (glasses, hats)
Video Diff.	Mask-guided denoising/experts	Protagonist insertion/replacement

Systemic considerations include edge compute feasibility (NPUs, quantization), privacy (on-device, in-situ editing), minimal memory overhead (rank-1 updates, residuals), and modularity (plug-in compatibility with multiple personalization and editing backbones) (Lu et al., 5 Jun 2025, Plesh et al., 2022).

6. Limitations and Future Directions

Known constraints include:

• Scalability to challenging, ambiguous, or compositional edits: Success rates can fall (10–14% lower than full-backprop editors for ambiguous facts (Lu et al., 5 Jun 2025), sensitivity to underrepresented or compositional actions (Choi et al., 22 Mar 2025)).
• Multi-edit and continual adaptation: Existing methods typically target single facts/concepts; robust simultaneous or continual multi-edit support is limited, although proposals for adaptive or batch editing exist (Lu et al., 5 Jun 2025).
• Mode collapse and bias: Overfitting to small reference sets induces mode collapse, especially in image generation and personality control, which motivates regularization or explicit mode-shifting (Yu et al., 15 Oct 2024).
• Deployment complexity: Parameter-level edits, though much lighter than full fine-tuning, still require weight updates and model reloading (Huang et al., 15 Dec 2025).
• Benchmark coverage: Most personalization benchmarks do not fully evaluate complex user scenarios (multi-turn, implicit information-seeking, evolving user profiles).
• Bias and security: Self-generated insights (for representation editing) may be sensitive to adversarial or biased user data, mandating more robust audit and filtering (Zhang et al., 2 Mar 2025).

Anticipated research includes hybrid retrieval + editing architectures, meta-learning for edit hyperparameters, automated clustering and subspace discovery, and extension of LLM editing techniques to multimodal preference representations.

7. Summary

Personalization editing grounds model adaptation in precise, locally-constrained parameter updates guided by semantically clustered preferences or concept representations. Across LLMs and generative visual models, editing achieves memory- and computation-efficient alignment with user facts, styles, or preferences, robustly retains general abilities, and scales to edge devices. The field continues rapidly expanding to address compositionality, continual learning, and the demands of privacy-oriented, real-time, cross-domain personal assistants (Huang et al., 15 Dec 2025, Lu et al., 5 Jun 2025, Hwang et al., 17 Feb 2025, Plesh et al., 2022, Yu et al., 15 Oct 2024).