Progressive Attention-Guided Feature Fusion

• PAGF is a family of methods that employ attention maps to iteratively localize informative regions and progressively refine feature fusion in diverse applications.
• In unsupervised domain adaptation, the PCaM method uses an attend-crop-refine routine to align source and target features, effectively suppressing background noise.
• In HDR restoration, APNT-Fusion leverages multi-scale attention modules to blend textures and reduce ghosting, thereby improving PSNR and SSIM scores.

Progressive Attention-Guided Feature Fusion (PAGF) denotes attention-driven fusion procedures in which attention maps are used not only to weight features, but also to localize informative regions, suppress irrelevant content, and progressively refine fused representations across layers or scales. The term is used for two technically distinct instantiations in "PCaM: A Progressive Focus Attention-Based Information Fusion Method for Improving Vision Transformer Domain Adaptation" (Zang et al., 27 May 2025) and "Attention-Guided Progressive Neural Texture Fusion for High Dynamic Range Image Restoration" (Chen et al., 2021). Taken together, these formulations suggest a common principle: attention acts as a control signal for determining where fusion should occur and how it should be refined, while the concrete objectives differ—cross-domain foreground semantic alignment in unsupervised domain adaptation and tonal/texture recovery in multi-exposure HDR restoration.

1. Core formulation and conceptual scope

In PCaM, PAGF is described as a three-stage “attend-crop-refine” routine embedded inside a standard ViT UDA pipeline. A cross-attention module first highlights which source and target patches are strongly related, those highly related patches are then spatially grouped into a tight bounding box and cropped, and the cropped regions are subsequently re-projected and re-fused via cross-attention in deeper transformer layers. By repeating this process layer by layer, the method gradually suppresses background noise and amplifies the foreground semantics that are common to both domains. Pseudo-labeled source–target pairs ensure that the semantics being aligned are consistent (Zang et al., 27 May 2025).

In APNT-Fusion, the relevant PAGF components appear in a two-stream HDR restoration architecture. One stream performs Multi-Exposure Fusion (MEF), while the other performs Saturated-Region Texture Transfer. The two streams are blended progressively via learned Scale-Attention maps $A_{sc}^l$, and a final fusion reweighting network chooses a residual between $H_m$ and $H_{mef}$ to form $H_{out}$ (Chen et al., 2021).

These two instantiations differ in modality and task structure, but they share a technical motif: fusion is not a single global operation. Instead, an attention-derived intermediate representation—cross-attention rollout in PCaM, and motion/saturation/scale-attention maps in APNT-Fusion—governs a sequence of progressively focused feature transformations. This suggests that PAGF is best understood as a family of progressive attention-conditioned fusion strategies rather than a single fixed algorithm.

2. PAGF in Vision Transformer unsupervised domain adaptation

PCaM is designed for Unsupervised Domain Adaptation (UDA), where a labeled source domain and an unlabeled target domain must be aligned despite foreground object mismatch. The stated limitation of prior ViT-based UDA methods is that discrepancy in foreground object size and spatial distribution across domains weakens attention consistency and hampers effective domain alignment. PAGF addresses this by progressively filtering out background information during cross-attention so that discriminative foreground semantics can be fused across domains (Zang et al., 27 May 2025).

Let

$$z^{s,l-1}, z^{t,l-1} \in \mathbb{R}^{(N+1)\times D}$$

be the source and target patch embeddings, including the $[CLS]$ token, at transformer layer $l-1$. PCaM first projects these embeddings to queries, keys, and values: $$Q^{s\to t} = z^{t,l-1}W_Q,\qquad K^{s\to t} = z^{s,l-1}W_K,\qquad V^{s\to t} = z^{s,l-1}W_V.$$ Each is shaped $(N+1)\times D_h$, with $D_h=D/H$ for $H_m$0 heads. The per-head cross-attention is

$H_m$1

and the multi-head output is

$H_m$2

The progressive aspect is introduced through attention rollout across $H_m$3 layers. Denoting the scalar attention weight between source patch $H_m$4 and target patch $H_m$5 at layer $H_m$6 by

$H_m$7

PCaM defines

$H_m$8

A per-patch importance score in the target is then obtained as

$H_m$9

This rollout is not used merely for visualization. It drives an explicit spatial selection mechanism. The importance map $H_{mef}$0 is upsampled to the original image grid and thresholded at $H_{mef}$1; all pixels with $H_{mef}$2 define a minimal enclosing bounding box. If $H_{mef}$3 and $H_{mef}$4 map a patch index to row and column, the box is

$H_{mef}$5

$H_{mef}$6

This box is used to crop both source and target images, producing $H_{mef}$7, which is then bilinearly interpolated back to the original patch grid size before the next layer. For target patch coordinate $H_{mef}$8 and nearest crop coordinate $H_{mef}$9,

$H_{out}$0

with

$H_{out}$1

Algorithmically, for each pseudo-labeled pair $H_{out}$2, PCaM computes patch embeddings, iterates across layers $H_{out}$3, updates self-attention and cross-attention, computes $H_{out}$4, updates $H_{out}$5, identifies the bounding box $H_{out}$6, crops and interpolates to obtain the next-layer embeddings, and finally reads out fused CLS tokens for classification, distillation, and progressively focused optimization. The PAGF mechanism is therefore inseparable from cross-layer spatial selection and re-fusion rather than simple token weighting.

3. Attentional guidance, optimization, and architectural integration

PCaM adds an explicit attentional guidance term because cross-attention rollouts can still be noisy, scattering small weights to background. The introduced Progressively Focused Loss is

$H_{out}$7

Here $H_{out}$8 is the center of mass of the attention-rollout map: $H_{out}$9 By penalizing squared deviation from the center more strongly the closer a patch is to that center, the model is trained to collapse attention onto a single, coherent foreground blob (Zang et al., 27 May 2025).

The total loss is

$$z^{s,l-1}, z^{t,l-1} \in \mathbb{R}^{(N+1)\times D}$$0

where $$z^{s,l-1}, z^{t,l-1} \in \mathbb{R}^{(N+1)\times D}$$1 and $$z^{s,l-1}, z^{t,l-1} \in \mathbb{R}^{(N+1)\times D}$$2 are cross-entropy losses on source and target pseudo-labels, and $$z^{s,l-1}, z^{t,l-1} \in \mathbb{R}^{(N+1)\times D}$$3 is the cross-domain distillation loss from CDTrans. Integration is described as entirely plug-and-play: one replaces vanilla self-/cross-attention blocks in a UDA ViT, such as CDTrans, with PCaM blocks that carry out the rollout-to-box-identification-to-focus-refinement sequence before forwarding to the next layer. No changes are made to tokenization, positional embeddings, MLP heads, or the overall ViT architecture. The method is characterized as lightweight, architecture-agnostic, and easy to integrate into existing ViT-based UDA pipelines.

A common misconception is that attention-guided fusion in ViT adaptation necessarily operates at the level of whole-image token alignment. PCaM instead performs pair-specific foreground discovery from the model’s own cross-attention, converts that signal into an explicit crop, and repeats the refinement layer by layer. The focus is therefore on suppressing domain-specific background interference rather than on a single global alignment pass.

4. PAGF in HDR restoration and neural texture blending

In APNT-Fusion, PAGF is embedded in a two-stream restoration model that takes three LDR inputs,

$$z^{s,l-1}, z^{t,l-1} \in \mathbb{R}^{(N+1)\times D}$$4

corresponding to short, medium, and long exposures, and produces a single HDR output

$$z^{s,l-1}, z^{t,l-1} \in \mathbb{R}^{(N+1)\times D}$$5

The architecture combines a Multi-Exposure Fusion stream and a Saturated-Region Texture Transfer stream (Chen et al., 2021).

The pipeline begins with a domain transform that linearizes $$z^{s,l-1}, z^{t,l-1} \in \mathbb{R}^{(N+1)\times D}$$6-corrected LDR to HDR: $$z^{s,l-1}, z^{t,l-1} \in \mathbb{R}^{(N+1)\times D}$$7 An artificial “Masked Saturated HDR” is then built for the short exposure: $$z^{s,l-1}, z^{t,l-1} \in \mathbb{R}^{(N+1)\times D}$$8 This makes $$z^{s,l-1}, z^{t,l-1} \in \mathbb{R}^{(N+1)\times D}$$9 and $[CLS]$0 share the same saturation regions regardless of motion.

Feature extraction produces

$[CLS]$1

with $[CLS]$2 and $[CLS]$3. Motion Attention modules compare $[CLS]$4 and $[CLS]$5 to produce masks

$[CLS]$6

and the MEF feature tensor is formed as

$[CLS]$7

The parallel Neural Feature Transfer stream performs multi-scale VGG-guided correspondence in the masked HDR domain. Let $[CLS]$8 denote VGG19 features at layers $[CLS]$9. For tiled patches $l-1$0, the normalized inner-product similarity is

$l-1$1

Efficiently, each reference patch $l-1$2 defines a convolutional score map

$l-1$3

Matching proceeds coarse-to-fine in local windows $l-1$4: $l-1$5

Texture-transfer features are gated by both motion and saturation information: $l-1$6 An encoder produces learned features $l-1$7 and $l-1$8, patch replacement yields $l-1$9, and a symmetric decoder produces $$Q^{s\to t} = z^{t,l-1}W_Q,\qquad K^{s\to t} = z^{s,l-1}W_K,\qquad V^{s\to t} = z^{s,l-1}W_V.$$0. Progressive Texture Blending is then controlled by learned Scale-Attention maps: $$Q^{s\to t} = z^{t,l-1}W_Q,\qquad K^{s\to t} = z^{s,l-1}W_K,\qquad V^{s\to t} = z^{s,l-1}W_V.$$1

$$Q^{s\to t} = z^{t,l-1}W_Q,\qquad K^{s\to t} = z^{s,l-1}W_K,\qquad V^{s\to t} = z^{s,l-1}W_V.$$2

$$Q^{s\to t} = z^{t,l-1}W_Q,\qquad K^{s\to t} = z^{s,l-1}W_K,\qquad V^{s\to t} = z^{s,l-1}W_V.$$3

If $$Q^{s\to t} = z^{t,l-1}W_Q,\qquad K^{s\to t} = z^{s,l-1}W_K,\qquad V^{s\to t} = z^{s,l-1}W_V.$$4, the progressive blending equation is

$$Q^{s\to t} = z^{t,l-1}W_Q,\qquad K^{s\to t} = z^{s,l-1}W_K,\qquad V^{s\to t} = z^{s,l-1}W_V.$$5

Finally, a small conv-Sigmoid network $$Q^{s\to t} = z^{t,l-1}W_Q,\qquad K^{s\to t} = z^{s,l-1}W_K,\qquad V^{s\to t} = z^{s,l-1}W_V.$$6 applied to $$Q^{s\to t} = z^{t,l-1}W_Q,\qquad K^{s\to t} = z^{s,l-1}W_K,\qquad V^{s\to t} = z^{s,l-1}W_V.$$7 produces $$Q^{s\to t} = z^{t,l-1}W_Q,\qquad K^{s\to t} = z^{s,l-1}W_K,\qquad V^{s\to t} = z^{s,l-1}W_V.$$8, and the output is

$$Q^{s\to t} = z^{t,l-1}W_Q,\qquad K^{s\to t} = z^{s,l-1}W_K,\qquad V^{s\to t} = z^{s,l-1}W_V.$$9

Training is end-to-end on $(N+1)\times D_h$0. Both prediction and ground truth are tone-mapped by $(N+1)\times D_h$1-law,

$(N+1)\times D_h$2

and optimized with the $(N+1)\times D_h$3 loss

$(N+1)\times D_h$4

The reported optimizer is Adam with $(N+1)\times D_h$5, batch size $(N+1)\times D_h$6, patch size $(N+1)\times D_h$7, and Xavier initialization.

5. Empirical behavior and ablation evidence

For PCaM, PAGF is reported to improve adaptation performance across multiple UDA benchmarks. On Office-Home, the result is $(N+1)\times D_h$8 versus $(N+1)\times D_h$9 for CDTrans, a gain of $D_h=D/H$0. On VisDA-2017, the result is $D_h=D/H$1 versus $D_h=D/H$2 for CDTrans, a gain of $D_h=D/H$3. On DomainNet, the result is $D_h=D/H$4 versus $D_h=D/H$5, a gain of $D_h=D/H$6. On a remote sensing AID$D_h=D/H$7NWPU task, the result is $D_h=D/H$8 versus $D_h=D/H$9, a gain of $H_m$00 (Zang et al., 27 May 2025).

The ablation analysis on VisDA is particularly diagnostic. Simply cropping foreground $H_m$01 or weighting patches by raw attention rollout $H_m$02 yields negligible gains. By contrast, successive PAGF stages add value: $H_m$03 yields $H_m$04, $H_m$05 yields $H_m$06, and full PCaM yields $H_m$07. A resolution-sensitivity study confirms robustness down to $H_m$08 inputs, and pseudo-label noise experiments show that PAGF is significantly more robust than CDTrans under corrupted target labels. These findings support the specific claim that progressive focusing, rather than isolated cropping or raw attention weighting, is responsible for the main performance gains.

For APNT-Fusion, the reported quantitative evidence is on the DeepHDR test set of 15 dynamic scenes. The model achieves PSNR-$H_m$09 $H_m$10 versus $H_m$11 for Yan 19 and $H_m$12 for Kalantari 17, and SSIM-$H_m$13 $H_m$14 versus $H_m$15 and $H_m$16, respectively (Chen et al., 2021). Ablation highlights report PSNR-$H_m$17 drops of $H_m$18 without Motion Attention, $H_m$19 without NFT, $H_m$20 without MS-HDR, and $H_m$21 without Scale Attention. The qualitative summary states that ghosting is nearly eliminated versus flow-based methods, texture is transferred into large saturated, motion-corrupted regions versus pure attention methods, and halo and blur are suppressed via deep MEF regularization and multi-scale fusion.

Across the two papers, the empirical pattern is consistent: the progressive component is not ornamental. In both cases, multi-stage attention-guided refinement produces stronger results than one-shot or partially disabled alternatives.

6. Relation to neighboring methods and interpretive issues

Within UDA, PCaM is positioned against traditional cross-attention methods such as CDTrans and DOT, which align entire image tokens and therefore risk pollution by domain-specific backgrounds. PAGF differs in two stated ways: it uses the model’s own cross-attention to identify a foreground “mask” on the fly rather than pre-computed saliency, and it progressively refines that mask and applies it layer by layer rather than fusing only once at a fixed depth. In contrast to style-transfer or multimodal fusion transformers such as CrossViT, the masks are not fixed by scale or modality gaps, but are dynamically discovered per sample pair. The intended effect is to avoid the foreground-object mismatch issue that arises when object scales differ substantially between source and target (Zang et al., 27 May 2025).

Within HDR restoration, APNT-Fusion clarifies a different point: attention alone is not treated as a sufficient mechanism. Motion Attention detects and suppresses content discrepancies among the reference images, Saturation Attention differentiates misalignment caused by saturation from that caused by motion, and Scale Attention enforces texture blending consistency between different coder/decoder scales. No extra regularization loss is applied to these attention maps; they are trained end-to-end under the reconstruction objective (Chen et al., 2021). This makes the fusion process tightly coupled to reconstruction, correspondence, and residual weighting rather than reducible to a single attention map.

A common misconception is that PAGF necessarily implies a unique architectural template. The two cited formulations do not support that view. In PCaM, PAGF is a plug-in mechanism inserted into an existing ViT UDA backbone without changing tokenization, positional embeddings, or MLP heads. In APNT-Fusion, the attention-guided progressive fusion process is integral to a purpose-built two-stream restoration network. This suggests that PAGF is more accurately characterized by its operational logic—progressive, attention-conditioned feature selection and fusion—than by any single backbone or loss design.

A second misconception is that the “guided” aspect must come from external annotations or handcrafted saliency. In PCaM, the guidance signal is derived from cross-attention rollout and further sharpened by the Progressively Focused Loss. In APNT-Fusion, guidance is produced internally by motion, saturation, and scale-attention modules and by VGG-guided feature matching. In both cases, guidance is endogenous to the model pipeline rather than externally prescribed.

The broader significance of PAGF, as evidenced by these formulations, is therefore methodological rather than domain-specific. It denotes a strategy for turning attention into an iterative controller of feature fusion, particularly in settings where naive global fusion is undermined by structured nuisance factors such as background shift, saturation, motion, or scale variation.