Perception-oriented Bidirectional Attention

• Perception-oriented Bidirectional Attention (PBA) is a mechanism that models visual attention bidirectionally between HR and SR feature maps for distortion-aware quality assessment.
• It integrates adaptive multi-scale key enhancement via GMDC and refines output with Sub-information Excitation Convolution to capture subtle artifacts like hallucinated textures.
• PBA serves as a core module within PBAN, enabling progressive refinement of quality features that outperform traditional FR-IQA metrics in super-resolution evaluation.

Perception-oriented Bidirectional Attention (PBA) is the central attention mechanism in the Perception-oriented Bidirectional Attention Network (PBAN) for full-reference image super-resolution quality assessment. In that formulation, PBA operates on feature maps from a reference high-resolution (HR) image and a super-resolved (SR) image, and is designed to “bidirectionally construct visual attention to distortion,” with the two directions corresponding to HR $\rightarrow$ SR and SR $\rightarrow$ HR. The mechanism is explicitly motivated by the observation that SR algorithms introduce enhancement-induced artifacts, such as hallucinated textures and over-sharpened edges, that are not adequately characterized by traditional FR-IQA metrics such as PSNR and SSIM. PBAN therefore combines an image encoder module, a perception-oriented PBA module, and a quality prediction module, with PBA further organized around Grouped Multi-scale Deformable Convolution (GMDC), Bidirectional Attention (Bi-Atten), and Sub-information Excitation Convolution (SubEC) (Li et al., 8 Sep 2025).

1. Problem setting and conceptual motivation

PBA was introduced for image SR FR-IQA, a setting in which the task is not image restoration itself but the evaluation of perceptual quality for SR outputs relative to a reference image. The motivating claim is that SR methods do not simply degrade images by adding conventional distortions such as noise or blur; instead, they produce enhancement-induced artifacts, including hallucinated textures and over-sharpened edges. This makes standard FR-IQA criteria limited for comparing and evaluating different SR algorithms (Li et al., 8 Sep 2025).

The mechanism is described as being inspired by characteristics of the human visual system. In the source description, human observers are said to judge SR images by alternately generating distortion, corresponding to the SR process itself, and evaluating distortion, corresponding to comparison between SR output and reference. PBA models this dual process by constructing attention in two directions, HR $\rightarrow$ SR and SR $\rightarrow$ HR, and by directing that attention toward subtle mismatches associated with both high-frequency detail and structure (Li et al., 8 Sep 2025).

A common misconception is to treat “bidirectional attention” here as a generic synonym for any two-way information exchange. In the PBAN formulation, the term has a specific meaning: it denotes paired attention computations between HR and SR branches, rather than a generic recurrent top-down/bottom-up controller. This distinction is important because the module is tailored to distortion perception in SR quality assessment rather than to general sequence modeling.

2. Architectural position within PBAN

PBAN is composed of three modules: an image encoder module, a perception-oriented bidirectional attention module, and a quality prediction module. The encoder first transforms the input HR and SR images into feature representations. PBA then operates on these branchwise features, and the quality prediction module integrates the resulting quality-aware features and regresses a final quality score (Li et al., 8 Sep 2025).

A structured description of a PBA block is given with explicit tensor dimensions. The inputs are two feature maps, $X^{HR}$ and $X^{SR}$, each with $C=64$ channels and spatial size $H=W=32$. Step 1 extracts $Q$, $K$, and $\rightarrow$0 through three parallel $\rightarrow$1 convolutions, producing outputs of shape $\rightarrow$2. Step 2 applies GMDC to $\rightarrow$3 and $\rightarrow$4. Step 3 computes attention weights via dot-product, variance normalization, and softmax, then applies them to $\rightarrow$5. Step 4 sends each BiAtten output into SubEC with $\rightarrow$6 and $\rightarrow$7. Step 5 adds back the original $\rightarrow$8 to form the PBA outputs $\rightarrow$9. These outputs are then either fed into the next PBA block or passed to the quality prediction module after the final block (Li et al., 8 Sep 2025).

This organization suggests that PBA is intended as a reusable block rather than as a one-off alignment layer. The stacking of multiple PBA blocks indicates that distortion-oriented correspondence is progressively refined before scalar quality regression.

3. Bidirectional attention mechanism

Let $\rightarrow$0 denote the HR and SR feature maps. PBA first computes query, key, and value tensors using $\rightarrow$1 convolutions: $\rightarrow$2

$\rightarrow$3

The key tensors are then enhanced through GMDC after branch swapping: $\rightarrow$4

A single scalar variance term is computed per branch for normalization: $\rightarrow$5

$\rightarrow$6

The subsequent attention stage produces two branch-specific results, $\rightarrow$7 and $\rightarrow$8, by applying dot-product attention, variance normalization, and softmax, followed by multiplication with the corresponding value tensor. Each BiAtten output has shape $\rightarrow$9. The two directions are computed in parallel within each PBA block, consistent with the stated goal of modeling both generation and evaluation processes in SR image assessment (Li et al., 8 Sep 2025).

After attention, the outputs are passed through SubEC and fused back to the branch residual: $\rightarrow$0

$\rightarrow$1

In functional terms, the bidirectional construction differs from a unidirectional cross-attention that would only score SR conditioned on HR, or vice versa. Here, each branch maintains its own attended output and residual pathway. This suggests that the mechanism is designed not only for cross-branch comparison but also for branch-specific preservation of quality-relevant features.

4. GMDC: adaptive multi-scale key enhancement

Grouped Multi-scale Deformable Convolution replaces a plain key tensor with an adaptive, multi-scale, group-wise deformable sampling operator. The starting point is standard deformable convolution, where for output pixel $\rightarrow$2,

$\rightarrow$3

with $\rightarrow$4 denoting learned offsets (Li et al., 8 Sep 2025).

GMDC extends this construction in three ways. First, it splits the $\rightarrow$5 input channels into $\rightarrow$6 groups of size $\rightarrow$7. Second, on group $\rightarrow$8, it applies a deformable convolution $\rightarrow$9 with kernel size $X^{HR}$0 and its own learned offsets $X^{HR}$1. Third, it concatenates the $X^{HR}$2 group outputs and applies a final $X^{HR}$3 point-wise convolution to restore full cross-channel interaction. In concise form, with channel grouping $X^{HR}$4,

$X^{HR}$5

The paper states that by choosing $X^{HR}$6, for example $X^{HR}$7, and using grouping, GMDC attains multi-scale receptive fields at roughly the same parameter cost as a single large DCN (Li et al., 8 Sep 2025). Within the stated motivation, this operator is intended to mimic spatial-frequency sensitivity in the human visual system and to allow the network to adaptively perceive distortion. A plausible implication is that the key tensors become more sensitive to spatially localized and scale-varying SR artifacts before attention weights are formed.

5. SubEC: sub-pixel and sub-channel distortion refinement

Sub-information Excitation Convolution refines BiAtten outputs to mine finer distortion signals in two complementary dimensions, sub-pixel and sub-channel, before residual addition. Let $X^{HR}$8 (Li et al., 8 Sep 2025).

In the sub-channel weight branch, adaptive global pooling collapses spatial dimensions to a per-channel descriptor,

$X^{HR}$9

This descriptor is reshaped to $X^{SR}$0, split into $X^{SR}$1 groups of size $X^{SR}$2, and passed through $X^{SR}$3 convolutions to expand the channel dimension by factor $X^{SR}$4, typically $X^{SR}$5. A channel-shuffle step allows inter-group mixing, after which a final $X^{SR}$6 convolution reduces the representation back to $X^{SR}$7, producing

$X^{SR}$8

In the sub-pixel weight branch, the channels are reduced to $X^{SR}$9 by successive $C=64$0 convolutions: $C=64$1 A pixel-shuffle then upscales this tensor to

$C=64$2

followed by a small $C=64$3 convolution stack and a downsampling step, for example average pooling, back to

$C=64$4

The three-branch outputs, sub-channel, sub-pixel, and identity, are fused multiplicatively: $C=64$5

SubEC therefore acts as a distortion-oriented refinement stage rather than as a replacement for bidirectional attention itself. Its stated role is to direct visual perception to both sub-pixel and sub-channel attention. This suggests that PBA treats distortion cues as jointly spatial and channel-structured, rather than as exclusively spatial correspondences.

6. Quality prediction, optimization, and reported performance

After the last PBA block, the network retains $C=64$6 and $C=64$7 in $C=64$8. For each branch, average pooling over $C=64$9 produces a vector in $H=W=32$0. This is flattened and passed through a small MLP with the sequence Linear–ReLU–Dropout–Linear–ReLU–Dropout to reduce it to a scalar score $H=W=32$1 or $H=W=32$2. The two branch scores are concatenated into $H=W=32$3, and two more linear layers are used to predict the final quality score $H=W=32$4 (Li et al., 8 Sep 2025).

Training uses standard mean-squared error against the ground-truth MOS: $H=W=32$5

The paper reports that extensive experiments demonstrate that PBAN outperforms state-of-the-art quality assessment methods (Li et al., 8 Sep 2025). No specific benchmark numbers are given in the supplied record, so the reported comparative result is best understood at the qualitative level provided by the abstract.

An important interpretive point is that the quality prediction stage operates on scalar branch summaries rather than on a late dense fusion of full feature maps. This suggests that the architecture encourages each branch to produce a compact quality-relevant representation before final comparison.

7. Related uses of bidirectional attention and terminological scope

The acronym and general idea of bidirectional attention also appear in other contexts. A distinct formulation is summarized for BRIMs (Bidirectional Recurrent Independent Mechanisms), where bottom-up signals, top-down signals, and a null vector are dynamically combined using attention over modules in a deep recurrent architecture (Mittal et al., 2020). In that setting, each layer contains multiple modules with hidden states $H=W=32$6, and attention combines three sources through keys and values: $H=W=32$7 with scaled dot-product attention determining the attended input $H=W=32$8. Modules are sparsely selected according to their null-attention weights, and intra-layer communication is performed by a second attention pass (Mittal et al., 2020).

That recurrent formulation is conceptually related in that it also treats bidirectional information flow as attention-mediated and dynamically restricted, and the summary reports improved robustness to distractions and noisy data. However, its semantic content differs from PBAN’s perception-oriented construction. In BRIMs, the two directions are bottom-up and top-down within a recurrent hierarchy; in PBAN, the two directions are HR $H=W=32$9 SR and SR $Q$0 HR within a full-reference SR-IQA pipeline. Equating the two directly would therefore obscure the domain-specific meaning of “perception-oriented” in the SR quality-assessment literature.

Within the narrower PBAN usage, PBA denotes a module specialized for distortion-aware alignment, adaptive multi-scale key enhancement, and sub-information excitation in SR FR-IQA. Within the broader literature, it belongs to a family of bidirectional attention mechanisms that regulate information flow across complementary signal sources, but its concrete implementation and objectives are specific to perceptual quality assessment of super-resolved images (Li et al., 8 Sep 2025).