SEGA: Signed Ensemble Gaussian Black-Box Attack

• The paper introduces SEGA, a novel transfer-based black-box attack for NR-IQA models, demonstrating improved gradient transferability through ensemble Gaussian smoothing and perceptual filtering.
• The method employs a three-step approach: Gaussian smoothing of source gradients, multi-model aggregation, and signed perturbation masked by gradient and perceptual filters.
• Experimental results on CLIVE/LIVEC datasets show SEGA outperforms traditional FGSM variants in disrupting quality score consistency while preserving high visual similarity.

Signed Ensemble Gaussian Black-Box Attack (SEGA) is a transferable black-box adversarial attack for No-Reference Image Quality Assessment (NR-IQA) models. It is designed for the setting in which the attacker has access to one or more source NR-IQA models but no access to the target model’s architecture, parameters, gradients, or, in the primary setting, queries. SEGA approximates the inaccessible target gradient by Gaussian smoothing the source models and ensembling their smoothed gradients, then applies a signed perturbation together with perceptual filtering so that the predicted quality score of an unknown target can be strongly altered while the adversarial image remains visually similar to the original (Liu et al., 23 Sep 2025).

1. Problem setting and motivation

NR-IQA models map a single input image $x \in \mathbb{R}^d$ to a scalar quality score, without access to a reference image. In the formulation used for SEGA, an NR-IQA model $f$ satisfies

$$f : \mathbb{R}^d \to \mathbb{R}, \quad f(x) = \text{predicted quality}.$$

This no-reference setting is technically important because the model must infer perceptual quality from the image alone, which makes learned representations more diverse across architectures (Liu et al., 23 Sep 2025).

The applications described for NR-IQA include content recommendation, medical imaging quality control, autonomous driving perception self-assessment, and general multimedia quality monitoring. In such settings, adversarial attacks expose security risks, robustness weaknesses, and evaluation biases. The attack surface is especially salient because a low-quality image can be made to appear high-quality to the model, potentially causing downstream systems to promote or rely on poor content (Liu et al., 23 Sep 2025).

SEGA is motivated by a disparity between white-box and black-box attack regimes. Existing NR-IQA attacks such as FGSM variants, Pattack, OUAP, and IOI operate in white-box settings with access to exact gradients, but transfer poorly to realistic black-box targets. The paper attributes this low transferability to two properties of NR-IQA models: different architectures often learn heterogeneous features and score mappings, and raw gradients are noisy and “shattered,” so steepest-ascent directions vary substantially across models. The work is presented as the first attempt to address low transferability in attacking NR-IQA models by constructing a pure transfer-based black-box attack (Liu et al., 23 Sep 2025).

The threat model assumes accessible source models $f^1,\dots,f^K$, an unknown and inaccessible target model $h$, knowledge of the input image $x$, the general task, and an $\epsilon$-bounded perturbation budget, but no knowledge of the target architecture, training data, internal feature representations, or true gradient $\nabla h(x)$. The attack objective is to design $\Delta x$ so that $h(x+\Delta x)$ differs significantly from $f$0 while $f$1 and the perturbation remains imperceptible (Liu et al., 23 Sep 2025).

2. Core method and mathematical structure

SEGA has three components: Gaussian smoothing of source-model gradients, gradient ensembling across multiple source models, and signed perturbation generation with perceptual filtering. The method starts from the black-box objective

$f$2

which is not directly solvable because $f$3 is inaccessible. It therefore substitutes source-side optimization,

$f$4

and departs from a simple FGSM-style construction $f$5 by replacing $f$6 with an ensemble-smoothed gradient estimate (Liu et al., 23 Sep 2025).

For each source model $f$7, SEGA applies Gaussian smoothing: $f$8 The paper states that if $f$9 is Lipschitz-continuous, then $$f : \mathbb{R}^d \to \mathbb{R}, \quad f(x) = \text{predicted quality}.$$0. In practice the expectation is approximated with Monte Carlo sampling,

$$f : \mathbb{R}^d \to \mathbb{R}, \quad f(x) = \text{predicted quality}.$$1

and the smoothed gradient is

$$f : \mathbb{R}^d \to \mathbb{R}, \quad f(x) = \text{predicted quality}.$$2

This differs from NES- or ZOO-style finite-difference estimators because SEGA backpropagates through perturbed source-model inputs and averages the resulting gradients (Liu et al., 23 Sep 2025).

With $$f : \mathbb{R}^d \to \mathbb{R}, \quad f(x) = \text{predicted quality}.$$3 source models, the ensemble gradient is the unweighted average

$$f : \mathbb{R}^d \to \mathbb{R}, \quad f(x) = \text{predicted quality}.$$4

This estimate is then converted into an $$f : \mathbb{R}^d \to \mathbb{R}, \quad f(x) = \text{predicted quality}.$$5-bounded perturbation by a component-wise sign operation,

$$f : \mathbb{R}^d \to \mathbb{R}, \quad f(x) = \text{predicted quality}.$$6

The paper’s rationale for the signed step is that, under $$f : \mathbb{R}^d \to \mathbb{R}, \quad f(x) = \text{predicted quality}.$$7, direction matters more than magnitude, and sign is less sensitive to local magnitude noise across models. That claim places SEGA in a family of sign-centered black-box attack methods, but its particular contribution is to combine sign-based updates with Gaussian smoothing and multi-model ensembling in the NR-IQA setting (Liu et al., 23 Sep 2025).

The transferability claim is supplemented by a theorem. If $$f : \mathbb{R}^d \to \mathbb{R}, \quad f(x) = \text{predicted quality}.$$8 is $$f : \mathbb{R}^d \to \mathbb{R}, \quad f(x) = \text{predicted quality}.$$9-Lipschitz and each source model has bounded approximation error, then

$f^1,\dots,f^K$0

The stated implication is that $f^1,\dots,f^K$1 mediates a trade-off: too little smoothing preserves noise, whereas too much smoothing degrades approximation to the original models and, by extension, the target (Liu et al., 23 Sep 2025).

3. Perceptual filtering and attack algorithm

SEGA introduces a perturbation filter mask to suppress what it terms “inappropriate” perturbations. Two kinds are identified. The first comprises gradient-insignificant components, where $f^1,\dots,f^K$2 is tiny and perturbing the corresponding pixel is expected to have negligible effect on the predicted score while still consuming perturbation budget. The second comprises perceptually sensitive regions, where even small perturbations are likely to be visible to human observers (Liu et al., 23 Sep 2025).

The gradient-based filter $f^1,\dots,f^K$3 is defined by a threshold $f^1,\dots,f^K$4: $f^1,\dots,f^K$5 The accompanying interpretation is that very small gradient components should not receive full $f^1,\dots,f^K$6-magnitude signed perturbations. This is important because a sign operator would otherwise amplify components whose true contribution to score change is vanishingly small (Liu et al., 23 Sep 2025).

The perceptual mask $f^1,\dots,f^K$7 is based on a Just Noticeable Difference map computed using Liu et al.’s JND model. For each pixel,

$f^1,\dots,f^K$8

If the JND threshold at a pixel is lower than the attack strength $f^1,\dots,f^K$9, an $h$0-scale perturbation is treated as likely visible and is disallowed; otherwise it is അനുവദിച്ച. The final perturbation is

$h$1

so each pixel receives $h$2, $h$3, or $h$4, depending on gradient sign, gradient significance, and JND tolerance (Liu et al., 23 Sep 2025).

Algorithmically, SEGA is primarily a single-step FGSM-like method rather than an iterative PGD-like method. In the main implementation, the ensemble gradient is initialized to zero, gradients are accumulated over $h$5 source models and $h$6 Gaussian samples per model, the average $h$7 is computed, and the two masks are constructed. The adversarial example is then formed by adding or subtracting the masked signed perturbation, depending on whether the attacker seeks to increase or decrease the predicted score. The reported hyperparameters are $h$8, $h$9, $x$0, and $x$1 (Liu et al., 23 Sep 2025).

The ablation results summarized in the paper indicate that increasing $x$2 improves gradient approximation and transferability at higher computational cost, that $x$3 exhibits a smoothing-versus-approximation trade-off, and that using more source models $x$4 improves transferability. These observations support the paper’s broader claim that SEGA converts raw, model-specific gradients into more stable, task-level directions (Liu et al., 23 Sep 2025).

4. Transferability, efficiency, and comparison with other attacks

SEGA is explicitly framed as a transfer-based black-box attack. In the comparison drawn by the paper, white-box NR-IQA attacks such as FGSM, Pattack, OUAP, and IOI perform well on the attacked source model but transfer poorly because they rely on unsmoothed gradients from a single model. Kor improves transferability by training a generic ResNet-based source NR-IQA model, but it does so at high training cost and without smoothing. Query-based black-box attacks such as SurFree-IQA can succeed but require thousands of queries to the target model (Liu et al., 23 Sep 2025).

The method’s distinctiveness within NR-IQA attack research is stated in three parts. First, it is presented as the first NR-IQA-specific transfer-based attack that explicitly integrates Gaussian smoothing of gradients, multi-model gradient ensembling, and perceptual filtering by JND. Second, it provides a theoretical error bound on gradient approximation in the transfer setting. Third, it is reported to achieve strong transfer to unknown NR-IQA models on CLIVE/LIVEC while preserving imperceptibility (Liu et al., 23 Sep 2025).

In computational terms, the paper measures complexity by the number of forward passes per adversarial image. The reported counts are FGSM: 1, Pattack: 200, OUAP: 10, IOI: 1, Kor: 20, and SEGA: $x$5 forward+backward passes. In the reported setup with three source models and ten samples per source model, this yields 30 passes and about 1.3 seconds per example on an RTX 2080. The paper contrasts this with query-based black-box attacks such as SurFree-QA, which are described as needing approximately 10,000 queries and roughly 29 seconds per example (Liu et al., 23 Sep 2025).

A frequent misconception in black-box adversarial literature is that “black-box” implies target querying. SEGA is explicitly not a query-based method in its primary setting; it is pure transfer. A second misconception is that a single-step attack must be equivalent to basic FGSM. In SEGA, the single-step update is preceded by Gaussian Monte Carlo smoothing, multi-source aggregation, and two masking operations, so the effective direction is substantially more structured than a raw single-model sign gradient. This suggests that, in NR-IQA, transferability may depend less on iterative refinement than on producing a direction that survives architectural heterogeneity (Liu et al., 23 Sep 2025).

5. Experimental evaluation on NR-IQA models

The reported experiments use the CLIVE/LIVEC dataset with an 80% training and 20% testing split. The target models are HyperIQA, DBCNN, LinearityIQA, and LIQE. For each target, the other three models are used as the source ensemble $x$6. Baselines are FGSM, Pattack, OUAP, IOI, and Kor, using official implementations. The evaluation metrics compare predictions before and after attack: MAE, robustness $x$7, and the rank- and correlation-based measures SROCC, PLCC, and KROCC (Liu et al., 23 Sep 2025).

For target DBCNN, SEGA reports $x$8, $x$9, $\epsilon$0, $\epsilon$1, and $\epsilon$2. The best baseline for DBCNN in MAE is OUAP, with $\epsilon$3, $\epsilon$4, $\epsilon$5, $\epsilon$6, and $\epsilon$7. The paper emphasizes that, across all targets, SEGA consistently has the lowest SROCC, PLCC, and KROCC, indicating the strongest disruption of score consistency under transfer (Liu et al., 23 Sep 2025).

For LinearityIQA as target, SEGA reports $\epsilon$8, $\epsilon$9, $\nabla h(x)$0, $\nabla h(x)$1, and $\nabla h(x)$2. The paper notes that baselines such as FGSM, Pattack, and IOI exhibit $\nabla h(x)$3 in this setting, which means they barely change rank order under transfer, whereas SEGA substantially disrupts it (Liu et al., 23 Sep 2025).

Imperceptibility is evaluated with SSIM and, in a more detailed comparison, $\nabla h(x)$4, $\nabla h(x)$5, LPIPS, and DISTS. Against HyperIQA, the average SSIM between adversarial and original images is 0.733 for FGSM, 0.731 for Pattack, 0.815 for OUAP, 0.883 for IOI, 0.931 for Kor, and 0.862 for SEGA. For the HyperIQA target, SEGA additionally reports $\nabla h(x)$6, $\nabla h(x)$7, SSIM $\nabla h(x)$8, LPIPS $\nabla h(x)$9, and DISTS $\Delta x$0. The paper’s interpretation is that SEGA outperforms FGSM, Pattack, and OUAP in visual similarity while remaining close to IOI and Kor, which were explicitly designed for imperceptibility (Liu et al., 23 Sep 2025).

The comparison with the query-based SurFree-QA on DBCNN is especially revealing. SurFree-QA reports MAE $\Delta x$1, $\Delta x$2, SROCC $\Delta x$3, PLCC $\Delta x$4, KROCC $\Delta x$5, and time per example $\Delta x$6 s. SEGA reports MAE $\Delta x$7, $\Delta x$8, SROCC $\Delta x$9, PLCC $h(x+\Delta x)$0, KROCC $h(x+\Delta x)$1, and time per example $h(x+\Delta x)$2 s. The paper therefore characterizes SEGA as both more effective and more efficient in this setting (Liu et al., 23 Sep 2025).

6. Limitations, assumptions, and broader research context

SEGA depends on the availability and quality of source models. The paper states that it requires at least one reasonably trained NR-IQA source model and that more source models improve transferability. It also depends on differentiability of source models with respect to input and on a Lipschitz-continuity assumption for the target gradient in the approximation theorem. The method further involves a three-way trade-off among transferability, imperceptibility, and efficiency: larger $h(x+\Delta x)$3 improves approximation but increases computation, $h(x+\Delta x)$4 must be tuned to avoid under- or over-smoothing, and stronger filtering via $h(x+\Delta x)$5 or the JND mask improves visual quality at the cost of attack strength (Liu et al., 23 Sep 2025).

These limitations place SEGA in a broader black-box attack landscape rather than outside it. “Gaussian MRF Covariance Modeling for Efficient Black-Box Adversarial Attacks” develops a query-based method that places a Gaussian Markov random field prior on the gradient field, infers a posterior gradient from zeroth-order observations, and then takes a one-step signed $h(x+\Delta x)$6 update (Sahu et al., 2020). That work shares the Gaussian and signed motifs, but its ensemble is an ensemble of target queries rather than an ensemble of source-model gradients, and it addresses classification rather than NR-IQA.

“Blackbox Attacks via Surrogate Ensemble Search” searches over surrogate-ensemble weights using a small number of victim queries, while generating perturbations through signed PGD-style updates on the surrogate ensemble (Cai et al., 2022). This is adjacent to SEGA in its use of ensembles and signed steps, but it is a hybrid query-based framework rather than a pure-transfer method. “There are No Bit Parts for Sign Bits in Black-Box Attacks” shifts black-box attack design toward sign estimation itself, recovering gradient sign vectors through binary optimization in a loss-oracle setting (Al-Dujaili et al., 2019). SEGA shares the emphasis on sign robustness but not the query-based sign-recovery mechanism.

Within NR-IQA specifically, SEGA’s connection to the broader literature is twofold. Methodologically, it imports ideas analogous to Gaussian smoothing and SmoothGrad-like denoising into transferable adversarial optimization. Evaluatively, it incorporates perceptual modeling directly through JND, SSIM, LPIPS, and DISTS, which is unusually aligned with the task domain because the attacked models themselves estimate perceptual quality (Liu et al., 23 Sep 2025). A plausible implication is that, for perceptual prediction systems, adversarial methodology and human-vision constraints cannot be cleanly separated: the attack’s success depends both on score manipulation and on preserving the image statistics that keep perturbations hard to detect.

The potential extensions identified in the paper include applying the Gaussian ensemble idea to full-reference IQA, aesthetic quality assessment, and video quality; exploring stronger attacks or two-stage attacks against robust-target scenarios; dynamically choosing $h(x+\Delta x)$7 and $h(x+\Delta x)$8; and using SEGA as a robustness benchmark for NR-IQA models (Liu et al., 23 Sep 2025). In that sense, SEGA occupies a specific but technically consequential niche: it is a pure-transfer, single-step, Gaussian-smoothed, ensemble-sign attack tailored to the architectural diversity and perceptual constraints of NR-IQA systems.