Saliency-Guided Encoder-Decoder Overview

• Saliency-guided encoder-decoder models are architectures that incorporate saliency signals, such as dense maps or fixation sets, to direct focus on key image or text regions.
• They employ various guidance mechanisms—including multiplicative weighting, token selection, and query-based matching—to optimize feature fusion and control generation.
• Empirical findings across tasks like image colorization, visual saliency prediction, and medical imaging demonstrate improved localization, semantic fidelity, and robustness.

Saliency-guided encoder-decoder denotes a family of neural architectures in which a saliency signal influences representation learning, feature fusion, cross-attention, or output generation inside an encoder-decoder pipeline. Across the literature, that signal may take the form of a dense saliency map, a saliency-weighted image, a salient-patch mask, a fixation set, a segment-level saliency mask, or a saliency-like auxiliary region derived from feature heatmaps. The term therefore covers several related but non-identical designs: joint colorization-and-saliency generators, saliency prediction networks, transformer encoder-decoders that directly predict fixations, summarizers whose decoder cross-attention is restricted by saliency masks, and medical vision-language systems that turn post-hoc saliency into training-time guidance (Zhao et al., 2020, Djilali et al., 2023, Wu et al., 2024).

1. Conceptual scope

In the cited literature, saliency guidance is used in at least four technically distinct ways. First, saliency can be an explicit supervisory or control signal that modulates a generator or decoder. SCGAN predicts a saliency map alongside colorization and uses the product $G_c(x)\odot G_s(x)$ both in the loss and in adversarial training, so salient foreground regions are emphasized during optimization (Zhao et al., 2020). ExtAbs replaces the standard encoder attention mask in decoder cross-attention with a saliency mask, thereby forcing the abstractor to attend only to predicted salient parts of the source document (Wu et al., 2024). In longitudinal chest X-ray Diff-VQA, a shared saliency mask derived from keyword-conditioned Grad-CAM is applied to both time points before answer generation (Wu et al., 29 Sep 2025).

Second, saliency can appear as an auxiliary prediction task that reshapes the latent representation. In SSiT, saliency-guided contrastive learning removes the lowest-scoring $m\%$ of patches from the key encoder input sequence, while the query encoder also predicts a saliency segmentation mask through a lightweight decoder; the two tasks are optimized jointly through $$\mathcal{L}=\lambda_{cl}\mathcal{L}_{cl}+\lambda_{seg}\mathcal{L}_{seg}$$ with default values $\lambda_{cl}=1$ and $\lambda_{seg}=10$ (Huang et al., 2022). SCGAN similarly treats saliency prediction as a proxy task whose branch helps revise lower layers of the main colorization branch (Zhao et al., 2020).

Third, some systems are encoder-decoders for saliency prediction rather than models guided by an external saliency prior. The contextual encoder-decoder for visual saliency prediction uses a modified VGG16 encoder, ASPP-style contextual aggregation, global scene information, and a decoder trained with KL divergence to produce a saliency distribution (Kroner et al., 2019). TASED-Net is a temporally-aggregating spatial encoder-decoder for video saliency detection, and the noise-aware encoder-decoder learns a clean saliency predictor plus a latent noise generator for noisy supervision (Min et al., 2019, Zhang et al., 2020).

Fourth, several works are only partially or analogically saliency-guided. ASGK does not rely on explicit human saliency annotations or learned pixelwise saliency maps; instead, it uses an internal visual auxiliary signal derived from feature heatmaps and an external linguistic auxiliary signal for medical report generation (Li et al., 2020). Caption-Guided Visual Saliency does not alter the encoder-decoder architecture during training, but probes a trained captioner to recover language-conditioned saliency post hoc (Ramanishka et al., 2016). Learning to Guide injects a learned guiding vector into the decoder at every time step, which is guidance-oriented but not formulated as explicit saliency-map supervision (Jiang et al., 2018). This suggests that the label “saliency-guided encoder-decoder” functions as a broad architectural category rather than a single canonical blueprint.

2. Recurrent architectural patterns

A recurrent pattern is the multi-branch image encoder-decoder. SCGAN combines a U-Net-like main colorization network with skip connections, a pre-trained VGG-16-Gray global feature encoder whose output is concatenated with the middle layer of the main network, a saliency prediction branch, and two 70×70 PatchGAN discriminators (Zhao et al., 2020). JAFFNet follows a classical encoder-decoder layout with five encoding stages and four decoding stages, but inserts a dense receptive field module after the encoder and a joint attention-guided feature fusion module into every decoding stage so that high-level features guide the selection of low-level detail features (Jiang et al., 2024). GLASSNet uses a frozen SAMv2/HiERA encoder with lightweight convolutional adapters and a dual-decoder design in which one branch captures global, long-range semantics and another captures fine local details, with an additional medium-scale branch bridging the two (Moradi et al., 4 May 2026).

A second pattern is the transformer encoder-decoder with saliency-aware queries or masks. SalTR follows a DETR-like design composed of a ResNet-50 backbone, transformer encoder, transformer decoder, and a fixed set of learned fixation queries $\mathcal{F}_q=\{F_{q0},\dots,F_{qN}\}$; each decoded query is passed through a 3-layer MLP to predict a fixation coordinate $\bar f_i=(\bar x_i,\bar y_i)$ (Djilali et al., 2023). ExtAbs augments the encoder with a self-attentive span extractor and segment classifier, then modifies decoder cross-attention through a saliency mask generated from top-$z$ predicted segments (Wu et al., 2024). In Diff-VQA, the multimodal decoder is GPT-2, but the inputs are first constrained by a shared saliency mask produced from keyword-conditioned Grad-CAM over paired images (Wu et al., 29 Sep 2025).

A third pattern is the encoder-centric architecture with a lightweight auxiliary decoder. SSiT is explicitly described as an encoder-only ViT SSL model with a momentum key branch and a lightweight auxiliary decoder/head for saliency prediction rather than a full encoder-decoder transformer. The query encoder contains a ViT backbone, projection head, prediction head, and saliency segmentation decoder attached to the final transformer block; the key encoder shares the same backbone but omits the prediction head and is updated by momentum, $\theta_k=\alpha\theta_k+(1-\alpha)\theta_q$ (Huang et al., 2022). This marks an important boundary case: the decoder may be asymmetrical, shallow, and purely auxiliary.

A fourth pattern is the probabilistic or adaptive decoder. The noise-aware saliency framework decomposes noisy labels as $Y=S+\Delta+\epsilon$, where $m\%$0 is produced by an encoder-decoder saliency predictor and $m\%$1 is a latent-variable noise generator (Zhang et al., 2020). FLIM-based salient object detection combines a marker-trained FLIM encoder with adaptive decoders whose weights are estimated heuristically for each input image, including the probability-based and mean-based decoders that estimate one weight per neuron per pixel (Soares et al., 29 Apr 2025). In both cases, “decoding” is not merely spatial upsampling; it includes explicit modeling of uncertainty, channel roles, or image-specific adaptation.

3. Guidance mechanisms and objective functions

The most direct guidance mechanism is multiplicative saliency weighting. In SCGAN, the attention-guided image is $m\%$2 and the corresponding ground-truth weighted image is $m\%$3. The attention-based reconstruction term is

$m\%$4

and the final generator loss is

$m\%$5

with $m\%$6, $m\%$7, and $m\%$8 (Zhao et al., 2020). Two hierarchical PatchGAN discriminators further separate full-image realism from saliency-weighted salient-region realism.

A second mechanism is saliency-guided token selection. In SSiT, the saliency map is patchified, the maximum saliency value in each patch becomes its patch-wise saliency score, and the lowest-scoring $m\%$9 of patches are removed from the key encoder input sequence, with $$\mathcal{L}=\lambda_{cl}\mathcal{L}_{cl}+\lambda_{seg}\mathcal{L}_{seg}$$0 in the main experiments (Huang et al., 2022). The contrastive branch uses InfoNCE, while the segmentation branch uses cross-entropy over the predicted saliency mask. Saliency therefore constrains what the stable momentum target is allowed to encode.

A third mechanism is set prediction with one-to-one assignment. SalTR reframes saliency prediction as unordered fixation-set prediction and uses Hungarian bipartite matching to assign predicted queries to sampled fixation points. The training loss combines coordinate regression with an NSS reward:

$$\mathcal{L}=\lambda_{cl}\mathcal{L}_{cl}+\lambda_{seg}\mathcal{L}_{seg}$$1

with $$\mathcal{L}=\lambda_{cl}\mathcal{L}_{cl}+\lambda_{seg}\mathcal{L}_{seg}$$2 (Djilali et al., 2023). The paper reports that without matching, decoder queries collapse onto dominant regions and diversity disappears, so uniqueness constraints are integral to the decoder’s behavior.

A fourth mechanism is attention masking in cross-modal or sequence generation. In ExtAbs, the saliency mask replaces the standard non-padding mask in decoder cross-attention:

$$\mathcal{L}=\lambda_{cl}\mathcal{L}_{cl}+\lambda_{seg}\mathcal{L}_{seg}$$3

with joint optimization through

$$\mathcal{L}=\lambda_{cl}\mathcal{L}_{cl}+\lambda_{seg}\mathcal{L}_{seg}$$4

The encoder predicts segment saliency scores, and the decoder is forced to attend only to top-$$\mathcal{L}=\lambda_{cl}\mathcal{L}_{cl}+\lambda_{seg}\mathcal{L}_{seg}$$5 predicted salient segments during inference (Wu et al., 2024). Diff-VQA uses a related but vision-grounded form of masking: after mild affine pre-alignment, a clinically relevant keyword is extracted, Grad-CAM is computed on both images, the two maps are merged by element-wise maximum, and the shared mask is applied multiplicatively to both images before multimodal decoding (Wu et al., 29 Sep 2025).

A fifth mechanism is guidance through feature fusion. JAFFNet derives a joint channel-spatial attention map from high-level features and uses it to refine low-level features according to

$$\mathcal{L}=\lambda_{cl}\mathcal{L}_{cl}+\lambda_{seg}\mathcal{L}_{seg}$$6

before concatenation with high-level decoder inputs (Jiang et al., 2024). GLASSNet instead distributes saliency-sensitive decoding across complementary global, local, and medium-scale streams and supervises the final fused map plus the medium branch using a weighted IoU plus weighted BCE objective (Moradi et al., 4 May 2026). These mechanisms differ from explicit masking, but they serve the same purpose: salient structure is privileged during reconstruction.

4. Major application domains

In image colorization, saliency guidance is used to reduce semantic confusion and color bleeding. SCGAN jointly predicts the colorized image and saliency map, injects global context through a pre-trained VGG-16-Gray encoder, and uses saliency-weighted supervision plus adversarial training to improve plausible object coloring (Zhao et al., 2020). The paper explicitly attributes improvements to saliency map-based guidance, the global semantic feature encoder, attention loss, and adversarial training.

In visual saliency and fixation prediction, the encoder-decoder is often the saliency estimator itself. The contextual encoder-decoder for natural images combines a modified VGG16 backbone, multi-level feature concatenation, ASPP with dilation rates 4, 8, and 12, a global average pooling branch, and a decoder of three bilinear-upsample-plus-convolution blocks to produce a saliency distribution (Kroner et al., 2019). TASED-Net extends the paradigm to video by using a 3D S3D encoder and a decoder that upsamples spatially while aggregating time to a single output map for the last frame of an input clip (Min et al., 2019). SalTR departs from dense regression by predicting discrete fixation points with learned queries and only converting them into a saliency map after prediction (Djilali et al., 2023). The noise-aware model addresses weak or pseudo supervision by disentangling clean saliency from label noise through alternating back-propagation and an edge-aware smoothness prior (Zhang et al., 2020).

In salient object detection and defect detection, decoder engineering is especially prominent. JAFFNet was designed for weak contrast, background interference, and large scale variation in industrial imagery; its DRF module supplies multi-receptive-field context, while JAFF modules let high-level semantic features emphasize defect features and suppress background noise during fusion (Jiang et al., 2024). FLIM-based networks use a markedly different regime in which encoder kernels are estimated from user-drawn markers on discriminative regions of only a few representative images, and adaptive decoders compute either image-wise or per-pixel weights without backpropagation (Soares et al., 29 Apr 2025). GLASSNet represents a foundation-model variant, using a frozen SAMv2 encoder adapted by spatially aware convolutional adapters and decoded by global-local branches for salient object detection and camouflaged object detection (Moradi et al., 4 May 2026).

In medical imaging, saliency guidance is frequently tied to sparse or clinically localized evidence. SSiT was designed for fundus-image diabetic retinopathy grading, where lesion patterns are small, sparse, and distributed; it uses saliency priors both to focus contrastive targets and to preserve fine-grained structure through saliency segmentation prediction (Huang et al., 2022). ASGK addresses medical report generation by combining an internal visual auxiliary signal derived from heatmap thresholding and connected-component selection with an external medical linguistic auxiliary signal, then encoding the resulting evidence in a medical tag graph before GPT-style decoding (Li et al., 2020). In longitudinal chest X-ray Diff-VQA, saliency is used to enforce consistent focus across time points and clinically relevant regions (Wu et al., 29 Sep 2025).

In text generation and interpretability, saliency-guided encoder-decoder methods no longer operate on pixels alone. ExtAbs unifies extractive and abstractive summarization in one encoder-decoder by allowing the shared encoder to identify salients and the decoder to use a parameter-free saliency mask in cross-attention (Wu et al., 2024). Learning to Guide adds a learned guiding vector to the decoder input at every time step,

$$\mathcal{L}=\lambda_{cl}\mathcal{L}_{cl}+\lambda_{seg}\mathcal{L}_{seg}$$7

which biases caption generation toward salient or semantically important content (Jiang et al., 2018). Caption-Guided Visual Saliency instead interrogates a trained captioner by comparing word distributions under masked inputs, recovering region-to-word mappings without explicit attention layers or pixel-level supervision (Ramanishka et al., 2016). These works broaden the notion of saliency guidance from spatial highlighting to decoder-side semantic restriction and explanation.

5. Empirical findings

Across tasks, saliency-guided designs are repeatedly associated with sharper localization, improved semantic fidelity, or stronger robustness under limited supervision. On 10,000 ImageNet validation images, SCGAN reports SSIM = 0.9473, Top-1 accuracy = 53.47%, and CCI ratio = 21.41%, together with strong human perceptual scores for color naturalness and color bleeding removal; on KAIST multispectral colorization it reports PSNR 24.59 and SSIM 0.8396 (Zhao et al., 2020). The ablation study indicates that removing attention loss, GAN loss, perceptual loss, pre-weights, or global features harms performance, and that WGAN-GP outperforms LSGAN in this setting.

In medical SSL, SSiT reports a Kappa score of 81.88% on DDR under fine-tuning evaluation, at least 9.48% better than other ViT-based SSL methods in that experiment. It also reports improvements on Messidor-2 and APTOS2019, and the paper states that it achieves the best Dice scores among compared SSL methods on vessel, lesion, and optic disc segmentation tasks (Huang et al., 2022). A comparison with spectral residual saliency is used to show that coarse saliency maps hurt performance significantly.

In fixation prediction, Deformable SalTR-Base reports on SALICON SIM 0.79, s-AUC 0.77, CC 0.89, NSS 2.12, and KLD 0.62, while on MIT300 it reports SIM 0.69, s-AUC 0.79, CC 0.80, NSS 2.45, and KLD 0.36 (Djilali et al., 2023). The number-of-queries ablation identifies 100 queries as best overall, and the “no Hungarian matching” variant collapses toward center-biased predictions.

For saliency detection and defect inspection, the contextual encoder-decoder reports state-of-the-art AUC-J, sAUC, and KLD on CAT2000 and ranks third overall on MIT300 by cumulative rank over sAUC, CC, and KLD among the selected metrics (Kroner et al., 2019). TASED-Net reports on DHF1K test NSS 2.667, CC 0.470, SIM 0.361, AUC-J 0.895, and s-AUC 0.712, with similarly strong results on Hollywood2 and UCFSports (Min et al., 2019). JAFFNet reports MAE 0.0128, $$\mathcal{L}=\lambda_{cl}\mathcal{L}_{cl}+\lambda_{seg}\mathcal{L}_{seg}$$8, $$\mathcal{L}=\lambda_{cl}\mathcal{L}_{cl}+\lambda_{seg}\mathcal{L}_{seg}$$9, and $\lambda_{cl}=1$0 on SD-saliency-900, MAE 0.0172 on Magnetic Tile, MAE 0.0042, $\lambda_{cl}=1$1, $\lambda_{cl}=1$2, and $\lambda_{cl}=1$3 on DAGM 2007, and 66 FPS on an RTX 3060 Ti (Jiang et al., 2024). GLASSNet reports a 97.94% reduction in learnable encoder parameters, from 212.15M to 4.37M, while surpassing state-of-the-art methods on standard SOD and COD benchmarks (Moradi et al., 4 May 2026).

In summarization and medical VQA, decoder-side saliency control produces more mixed results because the evaluation target is linguistic. ExtAbs reports on CNN/DailyMail 45.31 / 21.84 / 42.28 ROUGE-1/2/L for ExtAbs(BART)-abs and 45.06 / 22.02 / 42.09 for ExtAbs(PEGASUS)-abs, while also improving extractive performance on Reddit and PubMed (Wu et al., 2024). In Medical-Diff-VQA, the saliency-guided model reports BLEU-1 0.628, BLEU-2 0.510, BLEU-3 0.418, BLEU-4 0.341, METEOR 0.651, ROUGE-L 0.627, and CIDEr 1.263 (Wu et al., 29 Sep 2025). The paper notes that BLEU scores are modest but emphasizes high METEOR and intrinsic interpretability.

6. Boundaries, misconceptions, and open directions

A common misconception is that saliency-guided encoder-decoder always means explicit saliency-map input to a symmetric encoder-decoder. The cited literature contradicts that simplification. SSiT is explicitly described as not being a full encoder-decoder transformer; it is an encoder-only ViT SSL model with a momentum key branch and a lightweight auxiliary decoder/head (Huang et al., 2022). Caption-Guided Visual Saliency is not a train-time guidance mechanism but a post-hoc probing method for trained captioners (Ramanishka et al., 2016). TASED-Net is saliency-oriented but not explicitly guided by saliency masks inside the encoder (Min et al., 2019).

A second misconception is that saliency guidance is equivalent to standard attention. Several papers draw a sharper distinction. Caption-Guided Visual Saliency states that it does not add an explicit attention layer and instead recovers saliency by measuring information loss under masked inputs (Ramanishka et al., 2016). ExtAbs uses a parameter-free saliency mask to constrain cross-attention rather than learning additional highlight embeddings or a separate encoder (Wu et al., 2024). SalTR predicts discrete fixation sets instead of regressing a continuous saliency density map, even though its outputs can later be blurred into one (Djilali et al., 2023). This suggests that saliency guidance may act as supervision, control, explanation, or output parameterization rather than as a synonym for attention weights.

The literature also exposes recurring failure modes and regularization needs. The noise-aware saliency model reports that without the edge-aware smoothness term it can converge to the degenerate solution $\lambda_{cl}=1$4 for all training images, leaving the noise generator to explain everything (Zhang et al., 2020). SalTR shows that removing Hungarian matching causes fixation collapse and loss of diversity (Djilali et al., 2023). SSiT reports that coarse saliency maps hurt performance significantly, indicating that the quality of the saliency prior matters (Huang et al., 2022). In Diff-VQA, the affine pre-alignment is intentionally near-identity so that genuine longitudinal changes are not erased by aggressive registration (Wu et al., 29 Sep 2025). FLIM-based models depend on selecting representative training images, and the paper identifies broader adaptive-decoder design as a future direction (Soares et al., 29 Apr 2025).

A plausible implication is that current research is moving from saliency as a standalone prediction target toward saliency as a control variable that conditions generation, feature selection, and cross-modal reasoning. Frozen foundation encoders with lightweight adapters and explicit global-local decoding in GLASSNet, as well as image-specific heuristic decoding in FLIM networks, point toward parameter-efficient saliency-guided systems under limited data or compute (Moradi et al., 4 May 2026, Soares et al., 29 Apr 2025). At the same time, the coexistence of map-based supervision, set prediction, query-based decoding, and saliency-masked cross-attention indicates that the field remains methodologically plural rather than converged on a single dominant architecture.