Contrastive Privacy: A Semantic Approach to Measuring Privacy of AI-based Sanitization

Abstract: To sanitize specific concepts from imagery and text, privacy mechanisms with formal guarantees are often eschewed in practice in favor of more intuitive techniques. AI-based sanitization is poised to grow in popularity because it can work with the semantics of natural language concepts; e.g., a prompt to "remove faces, clothing, and body shape". Many approaches exist commercially and as prior work. But, the evaluation of such approaches has been bespoke and without formal guarantees. To fill this gap, we propose contrastive privacy, a formal definition of privacy that provides a systematic and quantitative test of sanitized media that has a semantic interpretation. It is independent of the model and mechanism used and operates across multiple media modalities. Contrastive privacy provides guarantees under ideal conditions; and we show how to operationalize the definition with imperfect measures of semantics, provided by models like CLIP, that can connect concepts latently. Notably, the algorithm contrasts sanitized media with other images from the same corpus to arrive at a determination; no manual labeling is involved. In our experiments, we apply our privacy test to both images and text using frontier models: some generate concepts to sanitize and others perform the sanitization. With our test we quantify sanitization success across 34 combinations of models on images, and for 15 models on text. The approach not only quantifies success overall, it identifies specific failures from a sanitized corpus. Further, it is independent of the mechanism used for sanitization, whether by darkening pixels, blurring, or applying more advanced means of obfuscation.

• The paper introduces contrastive privacy, a formal semantic framework that tests AI sanitization by comparing sanitized outputs with related proxy examples rather than relying on manual labels or unreliable model judgments.
• The method uses embedding-based distance measures and approximate nearest-neighbor search to provide scalable privacy evaluation, with resolution controlling the tradeoff between sensitivity and achievable privacy.
• Experiments across 34 image-model combinations and 15 text models show that no frontier system achieves ideal privacy, while the framework identifies semantic failures and reveals that model size does not reliably predict sanitization quality.

Motivation and problem statement

AI-based semantic sanitization mechanisms (SSMs) — privacy mechanisms that remove natural-language-specified concepts such as "faces," "logos," or "brand names" from images and text — are increasingly deployed in place of formal techniques like differential privacy (DP), which are rarely used in practice for unstructured data. The authors observe that evaluation of these mechanisms has been bespoke: existing measures either require error-prone hand-labeling of ground truth or rely on ad hoc collections of anomaly detectors. LLM/VLM-based judgment is also unreliable: the paper demonstrates that Gemini 3 Pro, given the same sanitized image of Leonardo DiCaprio and John Kerry, concludes under one prompt that DiCaprio can be identified and under a slightly different prompt that he cannot, and when asked to identify the people outright names them as Nigel Farage and Tucker Carlson. This motivates a quantitative, automated, model-agnostic privacy test with a semantic interpretation.

Formal definition

The paper grounds its definition in Contextual Integrity but argues that a single rendering may capture many information flows, so a comprehensive measure must consider all of them simultaneously. Renderings $x \in \mathbb{X}$ capture properties from $\mathbb{P}$; concepts are collections of properties, partitioned into natural instances (efficiently enumerable) and abstract instances (not efficiently enumerable). The threat model targets an abstract instance $e$ (e.g., "anything identifying papa smurf") via sanitization of computable natural concepts $c$.

The core machinery is:

• Distance mechanism $\mathcal{D}$: symmetric, positive, with $\mathcal{D}(x,x)=0$, capturing semantic similarity.
• Semantic connectedness: concepts $c$ and $d$ are connected if removing either one from a rendering increases its distance to renderings containing the other by more than resolution parameter $\delta$. The semantic closure $cl(\mathcal{X},\mathcal{D},c)$ collects all properties so connected to $\mathbb{P}$0.
• Contrastive privacy: mechanism $\mathbb{P}$1 offers contrastive privacy with respect to abstract concept $\mathbb{P}$2 at resolution $\mathbb{P}$3 if, for all $\mathbb{P}$4 capturing $\mathbb{P}$5,

$\mathbb{P}$6

The test contrasts sanitized media against other members of the same corpus — no manual labeling is involved — and requires only the assumption that $\mathbb{P}$7.

Theoretical guarantees

The central result, Theorem 1, proves that if $\mathbb{P}$8 offers contrastive privacy with respect to $\mathbb{P}$9, then every sanitized rendering $e$0 is private with respect to $e$1 (i.e., fails to capture any instance of $e$2). Corollary 1 establishes the converse intuition: achieving contrastive privacy implies $e$3 covers $e$4. Notably, the guarantee survives imperfect sanitization of $e$5 itself — the test certifies coverage by whatever concepts $e$6 actually removes, provided they lie in the semantic closure. The critical dependency is on $e$7: if the distance mechanism misses latent semantic connections, the test can report resolution 0 even though $e$8 does not cover $e$9. Thus sanitization quality governs achievable resolution while distance-mechanism quality governs validity of the test.

The guarantee is conditional in two ways the authors state plainly: it holds only if (1) the proxy set represents the full abstract concept and (2) $c$0 is sensitive to relevant similarities. Neither condition can typically be guaranteed in practice, so contrastive privacy provides an approximation rather than an absolute guarantee like DP.

Operationalization

Practical deployment uses proxy sets $c$1 sampled from $c$2, with CLIP-style cosine distance as $c$3. In cosine space the test reduces to checking whether the dot product between each sanitized embedding and each "difference vector" $c$4 stays below $c$5; this admits an efficient algorithm using vector databases (FAISS), with $c$6 exact or $c$7 HNSW-approximate nearest-neighbor lookup, where approximation error effectively inflates $c$8. Increasing $c$9 makes privacy easier to achieve but shrinks the semantic closure, weakening sensitivity to semantic similarity — hence the name "resolution."

Two practical caveats deserve emphasis. First, the concept ambiguity penalty: because $\mathcal{D}$0 obfuscates all instances of $\mathcal{D}$1, the test implicitly covers every instance in the closure, including semantically rich ones unrelated to the target (e.g., obfuscating all faces forces handling Santa Claus co-occurring with Bob). Second, $\mathcal{D}$2 generally requires fine-tuning when the abstract concept is not represented in the embedding model's training data; only the target concept needs labeling, not the constituent natural concepts.

Empirical results

Experiments cover 34 image-model combinations (4 i2t concept generators plus Manual × 7 i2i redactors) and 15 text models, at roughly $1,200 USD in API cost, using EVA-CLIP-18B for image distance and Qwen3-Embedding-8B for text.

Task	Best mechanism	Resolution	Utility
Celebrity identity (30 DiCaprio images)	iGEM31f/Manual	0	0.61
Brand identity (49 McDonald's images)	SAM3/Manual (9-image subset)	0	0.46
Movie franchise (49 Reddit comments)	GLN2 (9-comment subset)	0	0.75

On the full 49-image McDonald's set and 49-comment Avengers set, no mechanism achieved resolution 0; the best text mechanism reached only 0.04 on the nine-comment subset and 0.18 at scale. Several findings stand out:

• No frontier model achieves ideal privacy, and neither parameter count nor model generation predicts performance — the smaller iGEM31f achieves among the lowest resolutions while larger models fare worse.
• Task difficulty varies sharply: celebrity identity is easier to privatize than global brands (McDonald's) or movie franchises (Avengers), which have dense semantic closures.
• The test localizes specific failures with interpretable semantics: FLX2d/GEM31p failed to remove Margot Robbie and a Once Upon a Time... in Hollywood poster; iGPT15/GPT54's worst failure linked an ocean map to DiCaprio via the Our Ocean Conference; GPT54's worst text failure left "superhero movie" unredacted, connecting to "Watchmen, Dark Knight, Avengers." All Pareto-frontier text mechanisms shared one of two failure comparisons, suggesting uniform behavior across model families.
• Many apparent failures stem from the concept ambiguity penalty (near-black sanitized images sweeping everything into $\mathcal{D}$3), which the framework flags conservatively.

A resolution analysis calibrates $\mathcal{D}$4 against everyday semantic pairs: e.g., with EVA, resolution 0.136 corresponds to privacy robust against adversaries who cannot link concepts less similar than dog-vs-cat at the 95th percentile.

Limitations and open questions

The paper concedes several constraints explicitly. Contrastive privacy yields no absolute guarantee — unlike DP — because proxy sets approximate $\mathcal{D}$5 and the ML models implementing $\mathcal{D}$6 and $\mathcal{D}$7 are fallible with unquantified error rates. The distance mechanism can miss modalities it was not trained on: EVA failed to flag visible text spelling "Leonardo DiCaprio" behind obfuscated faces, requiring OCR augmentation or inclusion of text in $\mathcal{D}$8. The work gives no method for choosing $\mathcal{D}$9 or constructing effective SSMs; it assumes these are provided. Direct testing of commercial systems (AWS Comprehend, Microsoft Presidio, VisShield, GUIGuard) is deferred due to cost and domain-specific fine-tuning requirements. Open questions include systematic benchmarking of privatization capabilities across models, principled fine-tuning procedures for $\mathcal{D}(x,x)=0$0, and scalable solutions to open entity recognition as a route around the concept ambiguity penalty.

Conclusion

Contrastive privacy supplies a formal, modality-agnostic definition of privacy for AI-based sanitization that reduces to an automated, label-free test over embeddings. Its theorem links the empirical test to genuine coverage of abstract privacy targets under stated conditions on representation and distance sensitivity. Empirically, the test quantifies and diagnoses failures across 34 image and 15 text model combinations, showing that current frontier SSMs do not achieve ideal privacy and that model scale and recency are poor predictors of sanitization quality.