Query-Agnostic Image Perturbation

• Query-agnostic image perturbation is a fixed additive noise applied to images that degrades model performance across different tasks without relying on individual image content.
• Methodologies involve solving constrained optimization problems using gradient-based techniques and transformation invariance to maintain imperceptibility and robustness.
• Applications span privacy protection, security evaluations, image retrieval, and defense against deepfake and diffusion model attacks, highlighting its broad impact.

A query-agnostic image perturbation, sometimes termed universal perturbation, is a single additive noise pattern that, when applied to any image from a representative data distribution, induces a specific disruption in a target model’s functionality. These perturbations are constructed without reference to the unique content (“query”) of individual images and generalize broadly, in contrast to query-aware or per-instance adversarial modifications. Query-agnostic perturbations are especially relevant for model-agnostic privacy protection, large-scale security evaluations, and understanding fundamental vulnerabilities in computer vision and multimodal systems.

1. Formal Definitions and Properties

Query-agnostic image perturbations are defined as a fixed vector $u$ (or mask $\delta$), added to arbitrary images $x \sim \mu$, such that the target system (classifier, retrieval engine, generator, or multimodal model) exhibits degraded performance or explicit failure (misclassification, retrieval outage, privacy leakage, etc.) with high probability: $$\Pr_{x \sim \mu}\bigl[f(x+u) \ne f(x)\bigr] \geq \gamma,$$ where $f$ is the target model, $\mu$ the test distribution, and $\gamma$ a target error or fooling rate. Typically, a norm constraint $\|u\|_p \leq \epsilon$ ensures imperceptibility. Properties of interest include:

• Image-agnosticism: $u$ does not depend on the query image $x$.
• Model-, data-, and task-agnostic transferability: Certain constructions yield perturbations applicable across architectures or even tasks.
• Application breadth: Beyond classification (Xu et al., 2022), such perturbations are effective for image retrieval (Li et al., 2018), vision-language pre-training (Zheng et al., 6 Aug 2024), deepfake/distributional models (Ruiz et al., 2020), and large-scale diffusion models (Wan et al., 20 Aug 2024).

2. Algorithmic Constructions

Construction of query-agnostic perturbations is typically formalized as a constraint optimization problem. Standard frameworks differ by the targeted system:

• Neural Image Retrieval: PIRE (Perturbations for Image Retrieval Error) maximizes the $\ell_2$ displacement in feature space, i.e., $\|f(x) - f(x+v)\|_2^2$ under $\|v\|_\infty \leq \epsilon$, where $f$ is the CBIR feature extractor. The optimization proceeds via backpropagation and projected Adam steps, with explicit scaling to survive JPEG/PNG quantization (Liu et al., 2019).
• Universal Adversarial Perturbation (UAP) for Classification: Minimize the expected classification loss (or 0–1 error) under norm constraint—

$$\min_{u} \mathbb{E}_{x\sim\mu}\,\delta(\hat{f}(x+u), \hat{f}(x)) \quad \text{s.t. } \|u\|_p \leq \epsilon.$$

This is typically solved by projected gradient descent or robust projected iterative methods (Xu et al., 2022).

• List-wise Universal Perturbations for Retrieval: Directly minimize the ranking metric (e.g., NDCG) by approximating its gradient over all possible query-scroll pairwise swaps, with multi-scale random resizing to break input-size defenses. Momentum + clipping stabilize optimization (Li et al., 2018).
• Robust/Transferable Construct: Inclusion of transformation invariance (random crop, resize, noise, brightness, etc.) during optimization, in order to maximize robust adversarial success rate under a transformation set $T$ (Xu et al., 2022).
• Vision-Language Pretraining (VLP) Models: Iterate over images (and/or texts), for each move its embedding just beyond the top-$k$ decision hyperplane, updating the universal perturbation via accumulation of minimal-norm crossing directions, with projection or clamping (Zheng et al., 6 Aug 2024).
• Prompt-Agnostic for Diffusion Models: Model the distribution of likely prompts via Laplace approximation in the prompt-embedding space; maximize the expected conditioning loss over this distribution to find a perturbation that breaks all plausible prompts simultaneously (Wan et al., 20 Aug 2024).

3. Applications and Empirical Findings

Empirical studies demonstrate the efficacy, transferability, and limitations of query-agnostic perturbations:

Application Domain	Method (Reference)	Success Metric(s)	Key Quantitative Finding
Image Retrieval	PIRE (Liu et al., 2019)	mAP drop (Oxford5k, Paris6k datasets)	mAP falls from 78.39 → 5.51 (BB) and 74.42 → 2.31 (WI)
Classification	Robust UAP (Xu et al., 2022)	Robust ASR under transforms (ImageNet)	93.2% robust ASR under R(20°) on ILSVRC-2012
Retrieval	Listwise UAP (Li et al., 2018)	mAP/mP@10 drop, Google Image attack	Listwise UAP: ~60% drop, 63% top-rank absence on Google
Privacy Defense	RP-FGSM (Sanchez-Matilla et al., 2020)	Mislead rate (top-5, unseen+defense)	RP-FGSM: 77% misleading (untargeted, unseen+defense)
VLP Models	MUAP (Zheng et al., 6 Aug 2024)	Recall@k, zero-shot/top-5 accuracy	CLIP/Flickr30k: TR@1=82.2→0.1, IR@1=62.1→0.06
Diffusion	PAP (Wan et al., 20 Aug 2024)	FID, CLIP-I, LPIPS, generalization	FID (WikiArt): Clean 198.7→PAP 448.3 (all prompts)
Deepfake Defense	LUP (Ruiz et al., 2020)	Query count, imperceptibility	30% reduction in queries, 98–100% success

High attack rates persist under input transformations, black-box transfer, or defense-aware ensembles, provided the optimization includes suitable regularization and randomization (Xu et al., 2022, Sanchez-Matilla et al., 2020).

4. Black-Box and Decision-Based Query-Agnostic Attacks

Query-agnostic attacks are not confined to white-box settings. Black-box regimes, including hard-label only access, support zeroth-order, gradient-free universal perturbation construction:

• RGF-based Optimization: Search for a direction $\theta$ minimizing the radius $h$ needed to cross the decision boundary for a large fraction of images, evaluated only via label queries and binary search. RGF (Nesterov & Spokoiny) provides directional derivative estimates and convergence guarantees (Hogan et al., 2018).
• Leaked Universal Directions: In black-box image translation, collect image-specific perturbations for a small “leak” set via standard black-box optimization (e.g., SimBA), perform PCA to extract principal “universal” directions, and exploit these to efficiently generate near-universal perturbations on new queries with reduced query cost (Ruiz et al., 2020).
• Minimal Query Exposure: YOQO/YOQT algorithms achieve universal adversarial success with just one or two queries per image (via dimension reduction and CMA-ES or finite differences), making detection by repeated queries impractical (Willmott et al., 2021).

5. Practical Considerations, Limitations, and Defenses

Important practical axes include:

• Robustness to Transformations: Composite training with random transformations is critical to defeat preprocessing and input diversity defenses; robust UAPs maintain 65–93% adversarial success after spatial, photometric, and composite corruptions (Xu et al., 2022).
• Surviving Quantization: For CBIR and similar systems, scaling or adaptively amplifying the perturbation is necessary to survive 8-bit image quantization and JPEG/PNG artifacts, trading off image quality (as measured by SSIM, PSNR) and protection (Liu et al., 2019).
• Defensive Countermeasures: Adversarial training with universal (or data-independent) noise, input anomaly detectors, randomized smoothing, model ensembling, input randomization, and rate-limiting are partial defenses but often insufficient without fundamental architectural changes (Hogan et al., 2018, Sanchez-Matilla et al., 2020).
• Transferability and Generalization: Query-agnostic perturbations often generalize across model architectures and—even in the case of diffusion or image-to-image networks—across composite, unseen tasks when the perturbation is optimized over a suitable surrogate distribution (e.g., prompt-embedding Gaussian for PAP) (Wan et al., 20 Aug 2024).
• Limitations: Universal attacks may require large deployment datasets for robust optimization; targeted universal attacks are notably harder and less efficient than untargeted; in black-box settings with only hard-label access, query complexity remains substantial, though amortized across all images (Willmott et al., 2021, Hogan et al., 2018).

6. Theoretical Insights and Open Questions

• Linearization and Decision-Boundary Geometry: Many effective constructions (e.g., (Zheng et al., 6 Aug 2024)) rely on the local linearity of the embedding space and the dominance of a few global directions near margin-crossing hyperplanes.
• Optimization Under Distributional Shift: Robust query-agnostic attacks require expectation-maximization over transformation and prompt distributions—not merely worst-case pointwise maximization (Xu et al., 2022, Wan et al., 20 Aug 2024).
• Query-Agnosticism in Multimodal/Generative Systems: Novel methodologies (MUAP for VLP (Zheng et al., 6 Aug 2024), PAP for diffusion (Wan et al., 20 Aug 2024)) extend query-agnostic paradigms into the multimodal and generative regime, raising new questions about cross-modal transfer, decision-surface estimation, and the semantic coherence of sampled distributions in prompt space.
• Limits and Defenses: Whether there exist architectures or training regimes intrinsically immune to query-agnostic perturbations—beyond trivial detection or input randomization—remains largely unsolved. Approaches based on certified robustness or randomized smoothing offer partial guarantees, but at the cost of accuracy and scalability (Hogan et al., 2018).

7. Broader Implications and Future Directions

The existence and construction of query-agnostic image perturbations have broad implications for model robustness, privacy protection, and the design of retrieval, classification, and generative systems. The demonstrated efficacy of universal perturbations against state-of-the-art neural, retrieval, vision-language, and diffusion models calls for continued research into fundamentally robust architectures and scalable defense mechanisms, as well as careful consideration of privacy and security in deployed computer vision pipelines (Xu et al., 2022, Liu et al., 2019, Zheng et al., 6 Aug 2024, Wan et al., 20 Aug 2024).

Open challenges include transferability under continuous model updates, the viability of adaptive and universal defenders, the estimation of attack subspace dimension, provable lower bounds on robustness, and extending query-agnostic defenses and attacks to video and multimodal data at scale.