Identity-Decoupled Anonymization for Visual Evidence in Multi-modal Retrieval-Augmented Generation

Abstract: Multi-modal retrieval-augmented generation (MRAG) systems retrieve visual evidence from large image corpora to ground the responses of large multi-modal models, yet the retrieved images frequently contain human faces whose identities constitute sensitive personal information. Existing anonymization techniques that destroy the non-identity visual cues that downstream reasoning depends on or fail to provide principled privacy guarantees. We propose Identity-Decoupled MRAG, a framework that interposes a generative anonymization module between retrieval and generation. Our approach consists of three components: (i)a disentangled variational encoder that factorizes each face into an identity code and a spatially-structured attribute code, regularized by a mutual-information penalty and a gradient-based independence term; (ii)a manifold-aware rejection sampler that replaces the identity code with a synthetic one guaranteed to be both distinct from the original and realistic; and (iii)a conditional latent diffusion generator that synthesizes the anonymized face from the replacement identity and the preserved attributes, distilled into a latent consistency model for low-latency deployment. Privacy is enforced through a multi-oracle ensemble of face recognition models with a hinge-based loss that halts optimization once identity similarity drops below the impostor-regime threshold.

• The paper introduces ID-Decoupled MRAG, a novel framework that factorizes identity and visual features to enable effective anonymization in multi-modal retrieval-augmented generation.
• It employs a two-branch VAE with manifold-aware rejection sampling and a conditional latent diffusion model to minimize de-anonymization risks while preserving task-critical attributes.
• Experimental results demonstrate state-of-the-art privacy (DAR as low as 2.8%) and high utility with fast inference (42.3 ms/image) for real-time applications.

Identity-Decoupled Anonymization for Visual Evidence in Multi-modal Retrieval-Augmented Generation

Motivation and Background

Multi-modal retrieval-augmented generation (MRAG) pipelines that combine visual evidence from large image corpora with powerful large multi-modal models (LMMs) have transformed visual reasoning systems across domains, including surveillance, automated medical navigation, and heritage analysis. However, the presence of human faces in retrieved images introduces stringent privacy risks due to exposure of personally identifiable information (PII), such as facial features and structural biometrics, in violation of privacy regulations (e.g., GDPR, CCPA). Prior interventions—including pixel-space obfuscation, naive GAN-based face swaps, or indiscriminate image filtering—either destroy task-critical non-identity features, lose significant utility for downstream reasoning, or fail to guarantee identity suppression.

Existing methods thus leave a critical unresolved gap: constructing an anonymization module that (1) robustly suppresses identity even under adversarial scrutiny, (2) preserves all non-identity features required for accurate MRAG inference (e.g., gaze, pose, expression), and (3) provides formal, measurable privacy-utility trade-offs suitable for regulatory compliance and high-stakes deployments. "Identity-Decoupled Anonymization for Visual Evidence in Multi-modal Retrieval-Augmented Generation" (2604.23584) addresses this gap by proposing the Identity-Decoupled MRAG (ID-Decoupled MRAG) framework.

Figure 1: Overview of the Identity-Decoupled MRAG Framework. The pipeline proceeds in three phases: (1) Multi-modal retrieval from database, (2) identity-decoupled anonymization via encoder, rejection sampler, and conditional generator, and (3) LMM-based generative reasoning.

Framework Architecture and Methodology

Latent Factorization with Disentangled Encoder

Central to the framework is a two-branch variational autoencoder, the Identity Variational Autoencoder (ID-VAE), which factorizes each cropped face detected in a retrieved image into:

• an identity code ($z_{\text{id}}$, 512-dimensional vector) aggregating identity-specific biometric features,
• a spatially-structured attribute code ($z_{\text{attr}}$, $512\times h\times w$ tensor) preserving all spatial scene information (pose, gaze, background, local geometry).

Regularization is enforced via a variational mutual information penalty between $z_{\text{id}}$ and $z_{\text{attr}}$ (using MINE), augmented by a gradient-based independence term that penalizes attribute predictor sensitivity along the identity axis. This ensures that minimal identity information contaminates $z_{\text{attr}}$, bounding the information-theoretic leakage.

Manifold-aware Identity Replacement

After encoding, the original $z_{\text{id}}$ is discarded and replaced by a synthetic identity code $z'_{\text{id}}$, sampled from a reference gallery using a manifold-aware rejection sampling protocol:

• The sampled $z'_{\text{id}}$ must be both “realistic” (lie on the learned identity manifold) and “distinct enough” (cosine similarity with the original below an impostor-regime threshold calibrated from state-of-the-art recognition models like ArcFace).
• This approach avoids trivial orthogonality requirements (rendered ineffective in high dimensions) and directly controls the probability of identity collision through the gallery distribution.

Conditional Latent Diffusion Generator

The anonymized face is then synthesized from $(z'_{\text{id}},z_{\text{attr}})$ using a conditional latent diffusion model:

• The generator is designed with spatial cross-attention to propagate non-identity structures throughout the synthesis process.
• Training utilizes a multi-oracle hinge loss that drives identity similarity—across ArcFace, CosFace, and AdaFace—below stringent impostor thresholds but halts degradation once sufficient anonymization is achieved, balancing privacy and utility.
• For fast, interactive deployment, the full DDPM is distilled into a 4-step Latent Consistency Model (LCM), achieving over 98% of the original's quality and privacy.

Theoretical Guarantees

The framework’s guarantees are formalized as follows:

• Identity Leakage: The residual identity information in the observed output $z_{\text{attr}}$0 is upper-bounded by the mutual information $z_{\text{attr}}$1 between $z_{\text{attr}}$2 and $z_{\text{attr}}$3. Provided that the encoder regularization is effective, any adversary observing output images cannot reconstruct the original identity code within this bound, regardless of access to downstream neural models or invertible latent maps.
• Identity Distinctness: The manifold-aware rejection sampler ensures, with overwhelming probability, that the replacement $z_{\text{attr}}$4 occupies a region of latent space that does not overlap with (and is conditionally independent from) the victim’s code, sharply reducing attack surface from brute-force recognition or inversion.
• Utility Preservation: Semantic distortion from identity swapping is provably limited to a constant determined by the attribute-faithfulness of the generator and the baseline autoencoder reconstruction error, supporting reliable downstream MRAG performance.

Experimental Evaluation and Results

Comprehensive experiments were conducted across three demanding regimes: CelebA-RAG (controlled), LFW-Wild-RAG (in-the-wild, occlusion-heavy), and FairFace-RAG (demographically balanced). Evaluation employed metrics on privacy (multi-oracle De-anonymization Rate, identity embedding distance) and utility (VQA with LLaVA, gaze error, FID/LPIPS on backgrounds).

The results are notable in the context of established baselines:

• Privacy: Naive blurring and DP-pixellation obtain 29–50% de-anonymization rates under ArcFace—even invertible latent-space methods (iFADIT-RAG) plateau at ~10%. ID-Decoupled MRAG achieves a mean DAR as low as 2.8% on CelebA-RAG and remains below 4% on unconstrained benchmarks.
• Utility: Aggressive masking and obfuscation destroy VQA accuracy; ID-Decoupled MRAG with LCM achieves 85.4% VQA Acc versus 88.2% for the non-anonymized oracle, with gaze estimation errors (<4°) that are minimally impacted.
• Latency: With LCM distillation, the full anonymization-to-generation pipeline operates at 42.3 ms/image, compatible with interactive real-time MRAG applications.

A critical ablation study demonstrates that omitting the mutual-information regularizer or replacing the manifold-aware sampler causes catastrophic increases in information leakage (DAR rising to 18–24%), verifying the necessity of every core architectural and theoretical component.

Adversarial Robustness

Three-tier threat modeling (black-box observer, white-box latent attacker, adaptive oracle) was employed to further probe cryptographic resilience. Unlike previous works, which fail under cross-ensemble or adversarial fine-tuning attacks, this architecture demonstrates that the mutual-information regularization and sampling protocol restrict all practicable leakage channels, maintaining single-digit percent DAR even against adaptive attacks.

Implications and Future Directions

ID-Decoupled MRAG provides a rigorously justified, empirically validated anonymization module for deployment in privacy-critical MRAG infrastructure. Its separation of identity and attribute spaces allows full propagation of non-identity visual cues, sharply reducing the privacy-utility trade-off that has long plagued face anonymization in real-world LMM pipelines.

Implications for future research include:

• Generalization Beyond Faces: Extending latent disentanglement and manifold-aware anonymization to other biometrics (gait, voice, handwriting).
• Regulatory Calibration: The architecture allows tuning of utility/privacy trade-offs ($z_{\text{attr}}$5, $z_{\text{attr}}$6) by deployment context, providing a pathway toward compliance-aware ML models.
• Adaptive/Transfer Attacks: Exploring hybrid attacks that exploit domain drift or fine-grained attributes, and investigating whether adversarial training in the anonymization pipeline can further enhance robustness.
• Non-Identity Visual Attributes: Improved disentanglement and generator designs may enable even higher retrieval recall and utility, approaching non-anonymized oracles.

Conclusion

Identity-Decoupled MRAG formalizes and solves the privacy-utility trade-off in multi-modal retrieval-augmented generation by introducing an information-theoretically sound, empirically robust anonymization module. By leveraging explicit latent factorization, manifold-aware sampling, and conditional generative modeling, it achieves strong suppression of de-anonymization risks while preserving task-critical visual semantics and supporting low-latency inference for real-time LMM deployments (2604.23584).