When Recovery Matters: The Blind Spot of Surrogate Privacy in MLLM Editing

Abstract: Multimodal LLMs (MLLMs) enable flexible instruction-driven image editing, but privacy risks arise when user images expose diverse and user-specific private content. Canonical privacy protection strategies typically substitute sensitive regions with surrogate content before cloud editing. Yet, the resulting output is often an edited surrogate rather than the desired edited source image, neglecting the local recovery in both design and evaluation scope. To this end, we introduce SPPE (Surrogate-based Privacy-Preserving Editing), the first recovery-oriented benchmark covering 36 fine-grained privacy categories and 65 editing instructions. It defines two complementary tasks: 1) editability assessment, which estimates before cloud interaction whether a surrogate can induce an edit consistent with the original image; and 2) surrogate-to-source edit recovery, which evaluates whether the edited surrogate can be transferred back to the private source with the edit effect preserved. We address each task with a dedicated method: ERMA predicts surrogate editability through instruction-aware multimodal relation modeling, while \method performs cycle-consistent recovery by using the surrogate editing pair as visual edit evidence and the source image as a source-preserving anchor. Experiments on SPPE and InstructPix2Pix show consistent improvements on both tasks. For editability assessment, ERMA improves over the best-performing baselines by 13.9% in SRCC and 12.3% in PLCC. For surrogate-to-source edit recovery, C2E-S2SER outperforms SOER across all 8 source integrity and edit consistency metrics on SPPE.

• The paper introduces SPPE, the first benchmark to jointly assess surrogate editability and recovery for privacy-preserving MLLM image editing.
• It presents ERMA, an instruction-aware evaluation method that significantly outperforms classical IQA metrics by encoding source, surrogate, and prompt relations.
• The C²E-S2SER recovery model uses cycle-consistency and multi-modal fusion to accurately reconstruct the edited image while preserving privacy.

Surrogate-Driven Privacy in MLLM Image Editing: The Blind Spot of Recovery

Introduction and Motivation

Multimodal LLMs (MLLMs) have attained considerable success in instruction-driven image editing, but introducing user images into cloud-hosted MLLMs exposes sensitive visual, textual, and multimodal content to privacy risks, including unauthorized data usage and implicit profiling. Privacy regulations and user expectations increasingly necessitate robust input protection strategies. Canonical approaches thus preprocess user inputs to redact or substitute sensitive regions with synthesized content, constructing so-called surrogate images that are presumed safe for upload. However, this paradigm exhibits a fundamental oversight: after editing is performed on the surrogate by the MLLM, the system must locally reconstruct a correspondingly edited private source image that faithfully combines the edit transformation with privacy preservation—yet no previous benchmark systematizes this recovery, and most approaches either rely on lossy proxy replacements or ignore the recovery requirement entirely.

Figure 1: Privacy leakage risks in MLLM image editing. Uploading a user image to a cloud-based MLLM exposes sensitive visual, textual, and multimodal content to unauthorized data usage, implicit profiling, and potential leakage.

This work introduces SPPE—Surrogate-based Privacy-Preserving Editing—the first benchmark explicitly targeting recovery-oriented surrogate-driven MLLM image editing, along with dedicated methods for editability assessment (ERMA) and edit recovery (\method; C$^2$E-S2SER) (2606.07171). The contributions address the dual challenge of: (1) pre-edit validation—can a surrogate image support a semantically equivalent edit to the original, and (2) post-edit recovery—does the editing effect observed on the surrogate yield a high-fidelity, privacy-compliant edited private source image.

Figure 2: Surrogate-driven MLLM image editing pipeline. A local agent first generates a surrogate by replacing selected sensitive regions with synthesized content. The cloud-hosted MLLM edits only the surrogate, and a local recovery module then transfers the observed editing transformation back to the original image.

The SPPE Benchmark: Problem Formulation and Data

SPPE is architected as a comprehensive, large-scale dataset and benchmark capturing the two essential tasks for surrogate-driven privacy-preserving editing: (1) editability assessment and (2) surrogate-to-source recovery. The dataset comprises 55,696 editing instances across 36 fine-grained privacy categories and 65 distinct editing instructions, encompassing a broad spectrum of sensitive content and diverse editing intents. Each sample includes the private image, the surrogate generated by replacing/masking the sensitive region, corresponding privacy annotations/masks, the edit prompt, the MLLM-edited surrogate, and the target (ground truth) MLLM-edited private image.

Figure 3: Motivation and task design of SPPE. Visual surrogate recovery must transfer the edit applied to the surrogate back, preserving the unique source content—a challenge that cannot be addressed by naive direct replacement.

SPPE's privacy taxonomy incorporates object-level privacy annotations sourced from datasets such as VISPR, DIPA, DIPA2, and Visual Redaction, spanning visual (faces, bodies, biometrics), textual (documents, contact, identity, etc.), and multimodal attributes (monitors, medical artifacts, receipts). The surrogate images are synthesized using SDXL inpainting, with parameterized guidance scales to trade off the degree of substitution vs. editability, and MLLM editing is performed using SmartEdit. This configuration simulates realistic proxy-based privacy protection, paired with an editing instruction to ground semantic modification.

Figure 4: Privacy attribute distribution in SPPE.

SPPE defines two evaluation tasks:

Task 1: Editability Assessment—determine, given the pre-edit source, surrogate, and instruction, whether the surrogate can induce an edit transformation consistent with the original.

Task 2: Surrogate-to-Source Edit Recovery—after obtaining an edited surrogate, reconstruct the private edited source that exhibits the same edit while maintaining source details and identity outside the protected region.

Editability-Aware Relational Multi-Modal Assessment (ERMA)

Legacy image quality assessment (IQA) metrics (e.g., SSIM, FSIM, DISTS, ST-LPIPS, TOPIQ) quantify structural/pixel fidelity or perceptual similarity, but do not condition on the downstream edit instruction, and thus cannot reliably determine whether a surrogate is semantically editable according to task intent. This misalignment is explicitly addressed in ERMA—a model that predicts the editability score by relationally encoding the source, surrogate, and text prompt in a joint CLIP feature space.

Figure 5: Editability assessment via semantic editing-direction consistency. The edit direction in CLIP space from source to edited image is compared between the source and surrogate.

ERMA computes CLIP embeddings for the source, surrogate, and edit prompt, then derives relational features (alignment, signed and absolute difference, multiplicative fusion, and cosine similarities) encoding both preservation and prompt-relevance. The predicted editability is supervised by edit-direction consistency between ground truth MLLM edits in the source and surrogate domains (measured as the cosine alignment of the respective $I \rightarrow I'$ and $S \rightarrow S'$ directions in CLIP space).

ERMA achieves substantial improvements over all baseline IQA metrics (e.g., +13.9% SRCC, +12.3% PLCC over best baseline), providing more robust, instruction-aware discrimination of surrogate utility across varied privacy categories.

Figure 6: Architecture of ERMA for editability-aware surrogate assessment, integrating vision-language relation features and instruction awareness.

Cycle-Consistent Edit-Conditioned Surrogate-to-Source Recovery (\method)

Direct recovery of the target edited private image from the edited surrogate is highly non-trivial—text prompts are inherently ambiguous under surrogate alteration, and naive proxy inversion cannot ensure either semantic fidelity or source preservation. \method introduces a multi-conditional recovery model built atop a diffusion transformer architecture, fusing four information streams: private source, surrogate, edited surrogate, and the privacy mask, augmented by instruction and automatic edit-category tag guidance.

Figure 7: Overview of \method. Source image, surrogate, edited surrogate, mask, and prompt are fused for recovery, with visual in-context evidence and cycle-consistent regularization.

Edit-conditioned tag tokens, extracted via weak supervision from the surrogate edit pair, allow the model to disambiguate whether a global style edit, local object operation, or region-level transformation dominates—the fusion of these multimodal conditions is achieved via concatenated latent representations (VAE) and the text token prepended to the instruction. The generative process is regularized via a cycle-consistent path (reverse edit), ensuring not only strong edit transfer but controlled preservation of non-edited source regions.

Empirical results over all privacy categories show \method outperforms strong baselines (SOER, EditTransfer, VISII, Prompt-Diffusion, Cross-Image Attention) in both edit consistency and source integrity; e.g., +5.4% PSNR (Src), +2.9% PSNR (GT), +2.9% DirI, and robust performance in both in-distribution and out-of-distribution settings (InstructPix2Pix).

Analysis of Results and Ablations

\method consistently achieves superior numeric performance across all 8 major metrics (CLIP-Sim, SSIM, PSNR, DirI/DirS wrt both source and ground truth), and is especially robust in categories requiring fine attribute preservation (e.g., biometric, license, document, face regions). The joint analysis of edit-directional and source similarity metrics confirms that optimal privacy-preserving editing requires balance: over-preservation (e.g., VISII) yields under-editing, while over-editing degrades privacy/identity.

Figure 8: Trade-off between source preservation and edit consistency on SPPE. Models achieving the best balance lie in the upper-right.

Ablation shows both the edit-conditioned tag and cycle consistency objectives are essential: removing either degrades edit transfer or increases spurious source drift. The edit tag enhances in-context grounding, and the cycle-path suppresses overfitting to surrogate drift.

Qualitative results and error analysis reveal that failure modes of prior methods include incomplete object removal, identity drift, and prompt misalignment, while \method enables structurally precise edits with higher semantic faithfulness (e.g., attribute change, background replacement, document field removal). The method generalizes strongly to novel instructions and data distributions (as demonstrated on InstructPix2Pix).

Figure 9: Qualitative comparison for surrogate-to-source edit recovery across methods, illustrating robustness to diverse edits and privacy categories.

Theoretical and Practical Implications

This work provides the first unified framework and assessment protocol for surrogate-driven privacy-preserving editing in MLLM pipelines, identifying surrogate-to-source recovery as the critical bottleneck largely overlooked in previous research. The theoretical contribution lies in formalizing the editability and edit recovery assessment as joint multimodal-consistent tasks, and demonstrating that conditioning on both edit-direction evidence and explicit edit-type tokens enables robust semantic transfer under privacy constraints.

Practically, the SPPE benchmark, ERMA, and \method provide actionable evaluation and deployment tools for real-world MLLM-based systems operating under privacy regulations (e.g., GDPR/CCPA), supporting secure cloud processing with local reconstruction for user-facing applications in medical, financial, personal identity, and surveillance contexts.

Future Outlook

Future developments should address (1) further generalization to open-set edits and user-customized privacy rules, (2) adversarial robustness against surrogate inversion and privacy attacks, (3) extension to video and cross-modal scenarios, and (4) integrating human feedback for subjective privacy utility alignment. As foundation models scale, integrating conditional privacy-preserving architectures (e.g., local control, DP optimization, unlearning) will be necessary to operationalize high-fidelity, regulatory-compliant visual-textual pipelines.

Conclusion

"When Recovery Matters: The Blind Spot of Surrogate Privacy in MLLM Editing" (2606.07171) systematically dissects and benchmarks the overlooked challenge of edit recovery in surrogate-driven privacy-preserving MLLM image editing. By providing SPPE, ERMA, and \method, the work establishes new state-of-the-art performance in instruction-aware surrogate editability assessment and edit-consistent source recovery, while delivering actionable frameworks and rigorous evaluation tools essential for responsible deployment of privacy-aware multimodal systems.