Reverse Personalization Framework

• Reverse Personalization Framework is a generative approach that inverts traditional personalization by decoupling identity-related conditioning to enhance control over anonymized outputs.
• It integrates two key methods: a diffusion model for face anonymization using conditional inversion and an LLM post-hoc rewriting module for scalable, user-centric personalization.
• Empirical results demonstrate state-of-the-art performance with low re-identification rates in vision tasks and improved accuracy metrics in NLP benchmarks.

Reverse Personalization Framework refers to generative methods that invert or decouple traditional personalization-driven conditioning, focusing either on deliberate identity suppression in image synthesis or post-hoc rewriting for model-agnostic personalization in LLMs. The term encompasses two prominent instantiations: (1) a zero-shot, attribute-controllable face anonymization framework for diffusion models (Kung et al., 28 Dec 2025); and (2) Reflective Personalization Optimization (RPO), a two-stage rewriting framework for scalable personalization in black-box LLMs (Hao et al., 7 Nov 2025). These approaches prioritize explicit control and modularity, offering state-of-the-art performance in both privacy-centric computer vision and user-centric natural language processing.

1. Overview and Terminology

The Reverse Personalization concept is defined through its foundational goal: to manipulate personalization vectors in generative systems for either suppression (face anonymization) or explicit, externalized alignment (LLMs). In face anonymization, reverse personalization seeks to remove identity-specific features while retaining non-identity attributes (expression, pose, scene context). In LLMs, reflectively personalizing outputs is achieved by decoupling content generation and user alignment, enabling a controllable, interpretable rewrite stage.

• In computer vision, this framework is built atop conditional diffusion inversion and identity-guided sampling (Kung et al., 28 Dec 2025).
• In natural language, RPO formalizes personalization as a post-hoc rewrite, realized via supervised fine-tuning and reinforcement learning (Hao et al., 7 Nov 2025).

2. Reverse Personalization for Face Anonymization

Reverse Personalization for face anonymization leverages advanced text-to-image diffusion models with two technical innovations:

Conditional Diffusion Inversion

• Real image $x_0$ is inverted into the diffusion latent space under null identity conditioning $\varnothing_{id}$, using a second-order ODE solver (DPM-Solver++), storing latents $\{z_T, ..., z_1\}$ for subsequent generation:

$$z_t = \frac{x_{t-1} - \hat{\mu}_t(x_t, x_{t+1}, \varnothing_{id})}{\sigma_t}, \quad t = T, ..., 1$$

• This step ensures that non-identity attributes such as pose, gaze, and background are preserved in downstream sampling (Kung et al., 28 Dec 2025).

Identity-Guided Conditioning Branch

• An identity adapter (IP-Adapter) is inserted into all cross-attention layers, enabling steerable identity suppression via hyperparameter $\lambda_{\mathrm{ipa}}$.
• Classifier-free guidance is reinterpreted: negative guidance scale $\lambda_\text{cfg} < 0$ is used to drive denoising "anti-identity," generating anonymized yet attribute-controlled faces.

Inference Algorithm

Inference proceeds in two stages:

1. Stage 1: Conditional inversion (to obtain latents under null identity conditioning)
2. Stage 2: Reverse personalization sampling (generating anonymized images with plug-and-play attribute control)

No model training is required. All trade-offs are controlled by $\lambda_\text{cfg}$ and $\lambda_{\mathrm{ipa}}$.

3. Reflective Personalization Optimization for Black-Box LLMs

RPO formalizes reverse personalization in natural language processing as a distinct, modular rewrite layer:

Decoupling Generation and Personalization

• Input query $x$ is processed by a frozen LLM $M$ to produce a high-fidelity, generic response $r_\text{gen} = M(x)$.
• A reflection module $\pi_\phi$ then rewrites $r_\text{gen}$ into final personalized output $r_\text{pers}$, conditioned on a retrieved user-history subset $U_\text{rel}$:

$$r_\text{pers} \sim \pi_\phi(\cdot|x, r_\text{gen}, U_\text{rel})$$

• The reflection module is trained in two phases: supervised fine-tuning (on rewriting trajectories, with chain-of-thought rationale) and reinforcement learning (with personalization/quality metrics as rewards) (Hao et al., 7 Nov 2025).

Training and Curriculum

• Supervised fine-tuning uses a dataset $D_\text{SFT}$ of appropriately annotated rewriting trajectories, with cross-entropy loss:

$$L_\text{SFT}(\phi) = -\mathbb{E}_{(x, r_\text{gen}, p^*, r_\text{target}) \sim D_\text{SFT}} \left[ \sum_{t=1}^T \log P_\phi(r_{\text{target}, t} | x, r_\text{gen}, p^*, r_{\text{target}, <t}) \right]$$

• Reinforcement learning rewards policy improvement and penalizes deviation from $\pi_\text{SFT}$ via KL regularization.
• Multi-context curriculum incrementally increases context size $k$ to enhance robustness to user-history noise.

4. Evaluation Metrics and Empirical Performance

Both domains employ specialized metrics to quantify privacy, fidelity, and utility.

Face Anonymization Metrics

• Re-identification (Re-ID) rates (via SwinFace, AdaFace): lower is better.
• Expression, Gaze, Pose distances: lower is better, via specialized estimators.
• Fréchet Inception Distance (FID) and Face IQA: lower/higher is better, assessing image quality (Kung et al., 28 Dec 2025).

Quantitative Results (Excerpt):

Method	Re-ID (SwinFace/AdaFace, %)	FID	Face IQA
Ours CHQ	2.622 / 0.783	4.809	0.856
Ours FHQ	4.800 / 2.029	8.651	0.921

Attribute-controllable anonymization maintains high accuracy for sex/race and low age MAE across multiple datasets.

RPO Metrics and Benchmarks

• LaMP benchmark: accuracy, F₁, MAE, RMSE, ROUGE-1, ROUGE-L across tasks and splits.
• RPO surpasses zero-shot, in-context learning, RAG, PAG, and HYDRA baselines in all measured tasks (Hao et al., 7 Nov 2025).

Quantitative Results (Excerpt):

Task / Method	Acc / F₁	MAE / RMSE	ROUGE-1 / L
LaMP-2 HYDRA	0.291 / 0.351	—	—
LaMP-2 RPO	0.355 / 0.400	—	—
LaMP-3 HYDRA	—	0.318/0.638	—
LaMP-3 RPO	—	0.252/0.564	—
LaMP-5 HYDRA	—	—	0.473/0.412
LaMP-5 RPO	—	—	0.498/0.425

5. Comparative Analysis and Ablations

Empirical studies elucidate the impact of architecture and algorithmic choices.

Anonymization Framework Ablations

• DPM-Solver++ inversion is critical; DDIM causes marked deterioration (Re-ID $\sim$37%, FID$>$47).
• Substituting SDXL with other generation backbones (e.g., InstantID) trades off identity removal and attribute disentanglement, failing to balance privacy and utility (Kung et al., 28 Dec 2025).

RPO Ablations and Integration

• RPO is genuinely model-agnostic: swapping Qwen3, GPT-4o-mini, DeepSeek-V3 as base models yields near-identical personalization outcomes.
• SFT+RL recipe demonstrates necessity: SFT provides core rewriting skills, RL enhances end-task performance while preserving proximity to initial policy.
• Limitations include data collection burden for trajectory creation and potential retriever noise in long user histories (Hao et al., 7 Nov 2025).

6. Applications, Limitations, and Interpretations

Reverse Personalization frameworks are applied in:

• Privacy-centric generative modeling, allowing fine-grained, attribute-controllable anonymization with preservation of scene and facial quality.
• Modular post-hoc rewriting for personalization in LLMs, enabling plug-and-play deployment, transparency, and compatibility with black-box models.

The zero-shot anonymization regime and generate-then-rewrite paradigm in RPO suggest new directions for both privacy-sensitive and user-adaptive generative systems.

Key limitations are external trajectory data requirements (LLMs), potential latency in rewrite stages, and noise sensitivity in retrieval mechanisms. Neither framework introduces per-subject fine-tuning, leading to scalable application in diverse datasets.

7. Conclusion and Future Directions

Reverse Personalization Frameworks present modular, controllable solutions to privacy and personalization in major generative domains. Their technical implementations—classifier-free guidance inversion, IP-Adapters, supervised trajectory rewrites, and reinforcement learning-enhanced policies—offer state-of-the-art results on public benchmarks. A plausible implication is that future work will leverage these architectures to externalize fine-grained user and identity modeling, increasing transparency, controllability, and utility in both vision and NLP systems (Kung et al., 28 Dec 2025, Hao et al., 7 Nov 2025).