---
title: Saliency-Guided Transformer (SGT)
url: https://www.emergentmind.com/topics/saliency-guided-transformer-sgt
type: topic
---

# Saliency-Guided Transformer (SGT)

to=arxiv_search tool code  大发游戏  天天中彩票nbajson
{"query":"all: \"Saliency-Guided Transformer\" OR ti:\"Rectify ViT Shortcut Learning by Visual Saliency\" OR ti:\"SMOOT: Saliency Guided Mask Optimized Online Training\" OR ti:\"A Sensorimotor Vision Transformer\" OR ti:\"Learning Generative Vision Transformer with Energy-Based Latent Space for Saliency Prediction\"","max_results":10,"sort_by":"relevance"}function
to=arxiv_search tool code ￣奇米json
{"query":"Saliency-Guided Transformer", "max_results": 5}function
to=arxiv_search tool code ՞նչjson
{"query":"2310.00772", "max_results": 5}function
to=arxiv_search code  天天中彩票不json
{"query":"Saliency Guided Training transformer", "max_results": 10, "sort_by": "submittedDate", "sort_order": "descending"}
Saliency-Guided Transformer (SGT) denotes a family of transformer methods in which a saliency signal explicitly shapes tokenization, masking, attention, or supervision. Across the literature, the signal may be gradient-based input saliency, computational visual saliency, intrinsic two-dimensional curvature, dense saliency maps, or transformer-derived attentiveness, and the target use cases range from interpretability-oriented training to efficient token selection, saliency prediction, robust classification, self-supervised pre-training, few-shot keypoint detection, and point-cloud registration [2111.14338] [2206.08567] [2504.02536] [2309.08220].

## 1. Scope of the term

The literature does not use the term in a single canonical sense. In some papers, SGT refers to a training procedure that masks low-saliency features and regularizes prediction consistency. In others, it denotes a vision transformer whose input sequence is filtered by a saliency operator before self-attention. A third usage appears in saliency prediction itself, where the transformer is trained to estimate dense saliency maps or salient objects. A plausible implication is that “Saliency-Guided Transformer” is best understood as a design pattern rather than a single architecture class [2310.00772] [2206.08567] [2112.13528] [2304.03140].

| Formulation | Mechanism | Representative papers |
|---|---|---|
| Saliency-guided training | Mask low-saliency features and impose output consistency | [2111.14338], [2310.00772] |
| Saliency-gated tokenization | Keep or emphasize salient patches before attention | [2206.08567], [2504.02536] |
| Saliency prediction transformer | Learn dense saliency or salient-object maps | [2110.03593], [2112.13528], [2309.08220] |
| Saliency-guided representation learning | Use saliency in SSL, few-shot, or matching pipelines | [2210.10969], [2304.03140], [2312.12970] |

A recurrent structural idea is that saliency defines either **which tokens are processed**, **which token interactions are preferred**, or **which perturbations the model should ignore**. The common consequence is a bias toward foreground, task-relevant, or high-information regions.

## 2. Gradient-based masking and online saliency-guided training

In one major usage, SGT is a training objective rather than a bespoke transformer block. The original saliency-guided training procedure computes gradient saliency, masks the features with the smallest gradients, and optimizes a standard task loss together with a KL-divergence term that keeps predictions on original and masked inputs close:
\[
\min_\theta \frac{1}{n} \sum_{i=1}^{n} \Big[ \mathcal{L}\big(f_\theta(X_i), y_i\big) + \lambda\, D_{\mathrm{KL}\big(f_\theta(X_i) \,\|\, f_\theta(\widetilde{X}_i)\big) \Big].
\]
The masking operator acts on the bottom-\(k\) features selected after sorting \(\nabla_X f_\theta(X)\). The method was applied across computer vision, NLP, time series, and transformers, with the stated goal of reducing noisy gradients while preserving predictive performance [2111.14338].

This formulation transfers naturally to transformers by redefining features as tokens, patches, or token embeddings. The SMOOT framework, "SMOOT: Saliency Guided Mask Optimized Online Training" [2310.00772], makes the masking rate itself adaptive. Instead of a fixed \(k\), it introduces a per-sample, per-epoch masking rate \(K_i(X)\), initialized at 50% and updated online through softmax-response changes:
\[
\delta_1 = softmax_1(\tilde{X}) - softmax_1(X), \qquad
\delta_2 = \frac{1}{n-1}\sum_{j=2}^{n}\big(softmax_j(\tilde{X}) - softmax_j(X)\big),
\]
\[
\delta = \alpha \delta_1 + (1-\alpha)\delta_2,
\qquad
K_{i+1}(X) = \max\!\Big(K_{\min}, \min\big(K_{\max}, K_i(X) + \lfloor \mu \delta \rfloor \big)\Big).
\]
The same loss structure is retained, but masking becomes performance-conditioned. On a Tiny Transformer based on DeiT with \((L=12,d=192,h=3)\), pretrained on ImageNet and adapted to CIFAR, the reported CIFAR-10 accuracies were 95.65% for the traditional model, 96.05% for fixed-\(K\) SGT, and 96.35% for SMOOT; on CIFAR-100 the corresponding accuracies were 75.75%, 78.10%, and 79.65% [2310.00772].

Within this line of work, SGT is therefore a regularization principle: low-saliency perturbations should not alter the output distribution. This suggests a transformer can be made saliency-guided without changing its self-attention operator, provided training repeatedly enforces robustness to the removal of low-saliency tokens or pixels.

## 3. Saliency-gated token selection in vision transformers

A second, architecturally distinct usage places saliency before the transformer, at the level of token selection. In "Rectify ViT Shortcut Learning by Visual Saliency" [2206.08567], SGT is a ViT variant that first predicts a saliency map, converts it into a binary patch mask, and retains only the top-\(M\) informative patches. The masked input is
\[
\tilde{z}_0 =
\big[ x_{\text{class}};\, (x_p^1E;\dots;x_p^N E)\odot Mask \big] + E_{pos},
\]
and a residual global branch is injected at the last encoder layer to recover information from the full unmasked patch set. The stated purpose is to rectify shortcut learning by forcing self-attention to focus on visually informative patches while preserving global context in the final layer. On INbreast, the paper reports for ViT-L versus SGT-L: Accuracy 90.36% to 92.77%, AUC 94.94 to 96.91, F1 90.30 to 92.71, and PSL 43.90 to 19.00; on CAT2000, ViT-L versus SGT-L gives Accuracy 87.87 to 89.27 and PSL 31.47 to 16.80 [2206.08567].

"A Sensorimotor Vision Transformer" [2504.02536] instantiates an even harder form of saliency gating. Here saliency is not learned from gradients but computed from intrinsic two-dimensional structure. A curvature field \(D_{abs}(x,y)\) is derived, summed within each \(p\times p\) patch, and the top-\(m\) salient patches are passed to a standard ViT-Base backbone:
\[
S_i = \sum_{(x,y)\in \text{patch } i} D_{abs}(x,y), \qquad
\mathcal{P}_{\text{selected}} = \text{Top-}m(S).
\]
The selected sequence lengths are \(m \in \{49,98,147,196\}\). For ImageNet-1k, the reported top-1 accuracies are 71.8% for SMT\(_{196}\), 68.3% for SMT\(_{147}\), 66.3% for SMT\(_{98}\), and 61.9% for SMT\(_{49}\); the corresponding GPU memory usages with batch size 256 are 31,932 MiB, 24,898 MiB, 17,396 MiB, and 9,896 MiB [2504.02536]. This is explicitly described as a hard, pre-transformer, saliency-gated tokenizer.

These patch-selection models differ from gradient-mask training in a crucial way. Saliency is used to change the transformer’s **sequence itself**, not merely its loss. The resulting transformer is structurally sparse from the first attention layer onward.

## 4. Transformers for predicting saliency and salient objects

A third major strand uses transformers to model saliency directly. In "TranSalNet: Towards perceptually relevant visual saliency prediction" [2110.03593], a hybrid CNN–Transformer architecture injects transformer encoders at three scales, then decodes a dense saliency map with a composite loss built from NSS, KLD, CC, and SIM. The design premise is that long-range contextual information is central to perceptually relevant saliency prediction, and the reported benchmark results show strong performance on MIT1003, CAT2000, MIT300, and LSUN’17.

Dense saliency detection also appears in modality-fusion settings. "Transformer-based Network for RGB-D Saliency Detection" [2112.00582] introduces a transformer-based within-modality feature enhancement module (TWFEM) and a transformer-based feature fusion module (TFFM), both based on transformer decoders with self-attention and cross-attention. TWFEM enhances RGB and depth features across scales within each modality; TFFM then performs global multi-scale, multi-modal fusion over all positions simultaneously. Across six benchmark datasets, the reported model achieves top or second-best performance on most metrics, including best DES MAE 0.018, DUT-D MAE 0.030, and NLPR MAE 0.024 [2112.00582].

In "Learning Generative Vision Transformer with Energy-Based Latent Space for Saliency Prediction" [2112.13528], saliency guidance takes a generative form. A Swin-based encoder and U-Net-like decoder predict dense saliency maps conditioned on a latent variable \(z\), with an energy-based prior
\[
p_\alpha(z) \propto \exp\left[-U_{\alpha}(z) - \frac{1}{2\sigma_z^2}\|z\|^2\right].
\]
The model is trained by MCMC-based maximum likelihood with Langevin dynamics for both prior and posterior sampling. The paper explicitly states that this yields not only accurate saliency predictions but also pixel-wise uncertainty maps that align with model confidence and human perception [2112.13528].

The same broad logic extends to auxiliary-task guidance and to video. "Semantic Segmentation Enhanced Transformer Model for Human Attention Prediction" [2301.11022] uses a shared ResNet50+ViT encoder, a saliency decoder, a semantic segmentation decoder, and a Multi-task Attention Module that converts segmentation features into channel-wise attention over saliency features. On SALICON, the final model reports AUC-J 0.774, s-AUC 0.656, and NSS 1.302 [2301.11022]. "UniST: Towards Unifying Saliency Transformer for Video Saliency Prediction and Detection" [2309.08220] introduces a saliency-aware transformer with a Semantic-Guided Block and Saliency Transfer across scales, and reports DHF1K performance of \(CC=0.541\), \(SIM=0.423\), \(AUC\text{-}J=0.920\), \(NSS=3.113\), together with DAVIS\(_{16}\) performance of MAE 0.018 and \(S_m=0.904\) [2309.08220].

In these works, the transformer is not merely guided *by* saliency; it is also a machine *for* saliency. The notion of SGT here is inseparable from dense prediction and cross-scale contextual aggregation.

## 5. Self-supervised, few-shot, language, and 3D extensions

Saliency guidance has also been integrated into self-supervised pre-training. "SSiT: Saliency-guided Self-supervised Image Transformer for Diabetic Retinopathy Grading" [2210.10969] couples a ViT-S backbone with two tasks: saliency-guided contrastive learning, where the momentum-updated key encoder drops low-saliency patches, and saliency segmentation prediction, where the query encoder predicts a saliency mask. The combined loss is
\[
\mathcal{L} = \lambda_{\text{cl}} \mathcal{L}_{\text{cl}} + \lambda_{\text{seg}} \mathcal{L}_{\text{seg}},
\]
with \(\lambda_{\text{cl}}=1\) and \(\lambda_{\text{seg}}=10\). On DDR under fine-tuning, the reported Kappa score is 81.88%, exceeding all other ViT-based SSL baselines by at least 9.48% [2210.10969].

Few-shot keypoint detection provides another explicit SGT architecture. "From Saliency to DINO: Saliency-guided Vision Transformer for Few-shot Keypoint Detection" [2304.03140] introduces SalViT, which uses a saliency interaction matrix \(\tilde{M}\) and modifies self-attention through
\[
\gamma(\psi(Q,K,M)) = \gamma\big(\phi(Q,K) - J(\mathbf{1}\mathbf{1}^\top - \tilde{M})\big).
\]
A morphology learner predicts a power-normalization exponent \(\tilde{\theta}\) and reshapes the saliency map as \(M=(M^\downarrow)^{\tilde{\theta}}\), thereby realizing the paper’s “dynamically changing receptive field.” The same paper further shows that DINO class-token attention can replace external saliency detectors. This yields a transformer whose attention is explicitly foreground-biased and whose occlusion robustness improves by roughly 10% PCK under severe occlusions [2304.03140].

Outside image classification and dense prediction, analogous ideas appear in language and 3D point clouds. "Decoding Layer Saliency in Language Transformers" [2308.05219] uses the MLM head to project layer-wise Grad-CAM-like saliency back into token space and shows that middle-to-late layers produce more semantically coherent token saliency on SST-2 and AG News. "D3Former: Jointly Learning Repeatable Dense Detectors and Feature-enhanced Descriptors via Saliency-guided Transformer" [2312.12970] defines saliency over point-cloud features, partitions points into salient and non-salient sets, and applies region attention in which non-salient queries attend to salient keys and values:
\[
\mathbf{A}^{\hat{\mathcal{P}}_{NS}} =
\operatorname{Softmax}\left(\mathbf{Q}\mathbf{K}^{\top}\right)\cdot \mathbf{V}.
\]
On 3DLoMatch with 250 extracted keypoints, the registration recall scores reported for RoReg, RoITr, and D3Former are 64.3%, 73.6%, and 76.5%, respectively [2312.12970].

Taken together, these works show that SGT is not confined to 2D vision. Saliency can guide token selection, prototype formation, attention geometry, and correspondence ranking in multiple representational domains.

## 6. Interpretability, robustness, and recurring limitations

A central claim across SGT variants is that saliency guidance improves interpretability by concentrating explanatory mass on task-relevant features. In the gradient-mask literature, this is evaluated through modification-based tests, comprehensiveness/sufficiency, or synthetic ground-truth saliency. In the shortcut-learning literature, it is evaluated through PSL and Grad-CAM inspection. In the saliency-prediction literature, it is evaluated through human-attention metrics such as NSS, CC, SIM, AUC-J, and KL. A common misconception is that SGT always means direct supervision of attention weights; many formulations instead guide the model indirectly through masking, token selection, or auxiliary objectives [2111.14338] [2206.08567] [2309.08220].

Robustness is an especially prominent theme in the adversarial-training extension. "Exploring the Interplay of Interpretability and Robustness in Deep Neural Networks: A Saliency-guided Approach" [2405.06278] combines adversarial training with saliency-guided masking in ASGT:
\[
L_i^{\text{ASGT}} =
L\big(f_{\theta_i}(X), y\big)
+ L\big(f_{\theta_i}(X'), y\big)
+ \lambda D_{KL}\big(f_{\theta_i}(\tilde{X}') \,\Vert\, f_{\theta_i}(X)\big).
\]
For MNIST under PGD at \(\epsilon=0.2\), the reported accuracies are 16.36% for standard training, 49.08% for SGT, 48.95% for SGA, and 55.69% for ASGT; the abstract summarizes robustness improvements of 35% on MNIST and 20% on CIFAR-10 under PGD at the stated noise magnitudes [2405.06278].

The limitations are equally consistent across papers. Gradient-based SGT inherits the fragility and noise of gradient saliency and adds nontrivial training overhead because gradients with respect to the input must be recomputed repeatedly. Saliency-gated tokenizers depend on the quality of the external saliency operator; hard top-\(m\) or top-\(M\) selection can discard useful global context or remain non-differentiable; random guidance or residual global branches are used precisely because rigid saliency priors can be harmful [2504.02536] [2206.08567]. In dense saliency modeling, auxiliary tasks can confuse the main task unless feature interaction is controlled, as the Multi-task Attention Module in SSETM explicitly addresses [2301.11022]. A plausible implication is that the main unresolved issue is not whether saliency can guide transformers, but which saliency signal should be trusted, at what granularity, and at which stage of the pipeline.

SGT is therefore best viewed as a broad methodological family. Its unifying premise is stable: saliency is promoted from a post hoc explanation to an operational prior that influences what the transformer sees, how it attends, or what invariances it is required to learn.

Source: https://www.emergentmind.com/topics/saliency-guided-transformer-sgt