---
title: 'PBAN: Bidirectional Attention for SR IQA'
url: https://www.emergentmind.com/topics/perception-oriented-bidirectional-attention-network-pban
type: topic
---

# PBAN: Bidirectional Attention for SR IQA

Perception-oriented Bidirectional Attention Network (PBAN) is a deep full-reference image quality assessment model designed specifically for super-resolved images. In the SR FR-IQA setting, it takes an original high-resolution reference image and a super-resolved image produced by an SR algorithm, and predicts a perceptual quality score intended to correlate with human subjective ratings such as MOS. Its defining premise is that SR quality assessment must address not only fidelity loss but also enhancement-induced distortions—such as hallucinated textures, over-sharpened edges, and structurally plausible but incorrect details—and that this requires explicit bidirectional interaction between reference and SR feature spaces rather than generic similarity measures alone [2509.06442]. Subsequent work extended the idea into an image-level perceptual predictor, Efficient-PBAN, and used it as a differentiable perceptual loss for SR optimization [2602.21482].

## 1. Problem formulation and perceptual motivation

PBAN was proposed for **full-reference image super-resolution quality assessment**. The task is to compare an SR image against its HR reference and regress a perceptual quality score. The paper distinguishes this setting from standard FR-IQA by emphasizing that SR outputs contain unusual distortions: visually appealing but incorrect textures, edge ringing, oversharpening, and fabricated details. Accordingly, the evaluator must judge not only fidelity degradation but also the perceptual plausibility and correctness of enhancement [2509.06442].

The model is motivated by three human-visual-system properties explicitly named in the paper: **attention to distortions**, **spatial frequency sensitivity**, and **hierarchical perception**. SR quality assessment is framed as involving two perceptual processes. The first is **generate distortion**, in which SR generation from HR/LR introduces distortions. The second is **evaluate distortion**, in which a human observer compares the SR image against the HR reference or an internal expected scene. This motivates a bidirectional alignment strategy: **HR features should be aligned toward SR space to reflect distortion generation, while SR features should be aligned toward HR space to reflect distortion evaluation**.

A central criticism of earlier SR-IQA practice is that traditional FR-IQA metrics such as **PSNR**, **MSE**, **SSIM**, and **MS-SSIM** primarily target low-level differences or generic structural similarity and were developed for common degradations such as blur, noise, and compression. Even deep FR-IQA methods are characterized as being trained largely around degradative distortions, whereas SR introduces enhancement-induced distortions. PBAN is therefore positioned as a **deep, full-reference, SR-specific quality predictor** rather than a generic FR-IQA metric.

## 2. Architectural organization

PBAN consists of three modules: **Image Encoder Module**, **Perception-oriented Bidirectional Attention (PBA) Module**, and **Quality Prediction Module** [2509.06442].

The end-to-end pipeline is patch based. The HR reference image and SR image are first divided into **non-overlapping \(32\times32\)** patches. Each HR patch and corresponding SR patch is independently encoded by an image encoder composed of a convolutional layer with **\(3\times3\)** kernels, batch normalization, and ReLU. The encoded features are denoted
\[
X^{HR}, X^{SR} \in \mathbb{R}^{C \times 32 \times 32}.
\]

These branch features are then passed through one or more stacked **PBA blocks**. Each PBA block contains three components:

- **GMDC**: Grouped Multi-scale Deformable Convolution  
- **Bi-Atten**: Bidirectional Attention  
- **SubEC**: Sub-information Excitation Convolution  

After the PBA stage, the two refined branch outputs \(O^{HR}_{PBA}\) and \(O^{SR}_{PBA}\) are average-pooled, flattened, mapped by fully connected layers, concatenated, and regressed to a scalar quality score. In the quality prediction module, linear layers reduce the feature dimension gradually from **1024 to 1**. Patch-level predictions are finally averaged to produce the image-level quality estimate.

This architecture is explicitly two-branch and full-reference. One branch processes HR-derived features, and the other processes SR-derived features. The design choice is consequential: PBAN does not treat quality assessment as independent feature extraction followed by late fusion, but as **feature interaction between reference and SR representations inside the core attention block**.

## 3. Perception-oriented bidirectional attention

The **PBA module** is the central mechanism in PBAN. It is composed of **Bi-Atten**, with **GMDC** placed before attention score construction and **SubEC** after attention to refine distortion-aware features [2509.06442].

“Bidirectional” here means that attention is constructed in **both directions between HR and SR feature spaces**. The paper defines:

- **HR \(\rightarrow\) SR**: the HR branch uses SR-derived key features to align HR features toward SR space.
- **SR \(\rightarrow\) HR**: the SR branch uses HR-derived key features to align SR features toward HR space.

This differs from one-way cross-attention. The two directions are associated with the paper’s perceptual interpretation of **distortion generation** and **distortion evaluation**.

For feature projection, PBAN uses \(3\times3\) convolutions:
\[
\begin{aligned}
&Q^{HR},K^{HR},V^{HR}=Conv(X^{HR}), \\
&Q^{SR},K^{SR},V^{SR}=Conv(X^{SR}).
\end{aligned}
\]

A key technical distinction from generic cross-attention is the paper’s preference for **\(Q\)-\(V\) homology** rather than **\(K\)-\(V\) homology**. In PBAN, query and value come from the same branch, while key comes from the other branch. The stated rationale is that HR and SR images are strongly related, and the goal is to emphasize subtle distortions while preserving the target branch’s feature semantics.

Before attention computation, the keys are processed by GMDC:
\[
\begin{aligned}
&K^{HR-D},K^{SR-D}=GMDC(K^{HR},K^{SR}), \\
&D^{HR}=Var(Q^{HR} {(K^{SR-D})}^{T}), \\
&D^{SR}=Var(Q^{SR} {(K^{HR-D})}^{T}).
\end{aligned}
\]

The bidirectional attention is then defined using HR queries and values with SR-derived keys for one branch, and symmetrically for the other. Although the paper’s typesetting of the softmax formula is corrupted in the accessible text, the construction is explicit: the attention logits are normalized by variance terms \(D^{HR}\) and \(D^{SR}\), and the **key is always cross-branch while query and value remain same-branch**.

The authors contrast this design with earlier attention-based SR-IQA methods such as **JCSAN** and **TADSRNet**. PBAN is described as being **full-reference**, as avoiding **pooling-based spatial/channel attention**, as explicitly aligning **HR and SR feature spaces**, and as focusing attention on **distortion discrepancies** rather than generic saliency. The attended content is therefore not semantic object saliency in the ordinary sense, but **distortion-sensitive HR/SR correspondences**, including false textures, oversharpening, local structural inconsistencies, and texture or edge artifacts.

## 4. Distortion-aware refinement: GMDC and SubEC

PBAN adds two auxiliary mechanisms around bidirectional attention: **Grouped Multi-scale Deformable Convolution (GMDC)** and **Sub-information Excitation Convolution (SubEC)** [2509.06442].

GMDC is introduced to let the model **adaptively perceive distortion regions** before attention is computed. The paper links this to HVS **spatial frequency sensitivity** and argues that SR distortions are local, scale-varying, structurally irregular, and often concentrated around edges and textures. Standard convolution is written as
\[
F\left(p_{0}\right)=\sum_{p_{n} \in r} w\left(p_{n}\right) \cdot y\left(p_{0}+p_{n}\right),
\]
while deformable convolution adds learnable offsets:
\[
F\left(p_{0}\right)=\sum_{p_{n} \in r} w\left(p_{n}\right) \cdot y\left(p_{0}+p_{n}+\Delta p_{n}\right).
\]

GMDC splits channels into groups, applies an independent deformable convolution to each group with a different kernel size \(E_i \times E_i\), and then fuses grouped outputs with a final **point-wise \(1\times1\) convolution**. The grouped multi-scale structure is presented as the crucial design point; the paper explicitly notes that **standard DCN was often worse than the baseline**, whereas GMDC consistently improved performance. On CVIU, the best overall configuration with SubEC was **GMDC \([3,7]\)** with **2 groups**, achieving **SRCC 0.978**, **KRCC 0.872**, **PLCC 0.981**, and **RMSE 0.397**.

SubEC is applied after bidirectional attention and is meant to refine the resulting distortion-sensitive features by directing perception to both **sub-pixel** spatial information and **sub-channel** information. It has three branches: a **Sub-Channel Weight branch**, a **Sub-Pixel Weight branch**, and an **Identity shortcut branch**. Its output is
\[
O_{SubEC}=I_{SubEC}\times W_{Sub-Channel}\times W_{Sub-Pixel},
\]
followed by residual addition:
\[
\begin{aligned}
&O^{HR}_{PBA}=O_{SubEC}^{HR}+X^{HR}, \\
&O^{SR}_{PBA}=O_{SubEC}^{SR}+X^{SR}.
\end{aligned}
\]

The Sub-Pixel branch reduces channels from \(C\) to \(S^2\) using two \(3\times3\) convolutions, applies **sub-pixel convolution / pixel shuffle** with upscaling factor \(S\), produces an upsampled map, and downsamples it back to obtain **Sub-Pixel Weight**. The Sub-Channel branch uses adaptive average pooling, grouped \(1\times1\) expansion, **channel shuffle**, and \(1\times1\) compression to obtain **Sub-Channel Weight**.

The ablations in the paper treat several design choices as consequential. **Channel shuffle helps**; replacing multiplicative fusion with element-wise addition is worse; **sub-pixel convolution** performs better than transposed convolution, bilinear, and bicubic; and **\(S=2\)** is selected as a strong and efficient default. A notable result on QADS is that adding SubEC to Bi-Atten improves performance from **SRCC 0.981, PLCC 0.982, RMSE 0.055** to **SRCC 0.984, PLCC 0.986, RMSE 0.050**.

## 5. Training protocol and empirical results

PBAN is evaluated on three SR-IQA datasets—**QADS**, **CVIU**, and **Waterloo**—using **SRCC**, **KRCC**, **PLCC**, and **RMSE**, with a five-parameter nonlinear fitting applied before PLCC and RMSE calculation [2509.06442]. Each dataset is randomly split into **80% train** and **20% test**, all DNN-based methods are trained independently on each dataset, and training uses **5-fold cross-validation** on the training set for **100 epochs** per fold. The optimizer is **SGD** with initial learning rate **0.01**, momentum **0.9**, and weight decay **\(10^{-6}\)**. The implementation uses **PyTorch** on an **NVIDIA DGX Station, Tesla V100-DGXS-32GB GPU** under **Ubuntu 18.04 LTS**.

Patch extraction is central to the training protocol. Each image pair is decomposed into non-overlapping **\(32\times32\)** patches, each patch inherits the MOS of its source SR image, and image-level prediction is obtained by averaging patch scores. The reported numbers of training patch pairs are **129,360** for QADS, **194,400** for CVIU, and **55,800** for Waterloo.

| Dataset | PBAN performance | Comparator note |
|---|---:|---|
| QADS | SRCC 0.986, KRCC 0.923, PLCC 0.987, RMSE 0.044 | SRIF: SRCC 0.916, PLCC 0.917, RMSE 0.109 |
| CVIU | SRCC 0.978, KRCC 0.872, PLCC 0.981, RMSE 0.397 | TOPIQ: SRCC 0.970, PLCC 0.971, RMSE 0.486 |
| Waterloo | SRCC 0.965, KRCC 0.848, PLCC 0.979, RMSE 0.587 | best among all compared methods |

These results are interpreted in the paper as evidence that **deep SR-aware FR modeling is superior to handcrafted FR SR-IQA**, that explicit HR/SR feature interaction is more effective than generic attention or generic FR-IQA metrics, and that PBAN is especially strong at detecting **SR-specific enhanced distortions**.

Cross-dataset tests are also reported. Training on CVIU and testing on QADS yields **SRCC 0.890**, **KRCC 0.732**, and **PLCC 0.892**, which the paper describes as the best among compared methods. Training on QADS and testing on CVIU gives **SRCC 0.730**, **KRCC 0.550**, and **PLCC 0.794**, with second-best SRCC but best KRCC and PLCC.

The ablations decompose the contributions of the three main components. On QADS and CVIU, a baseline without Bi-Atten, GMDC, and SubEC achieves **QADS: SRCC 0.936, PLCC 0.937, RMSE 0.103** and **CVIU: SRCC 0.942, PLCC 0.954, RMSE 0.815**, while full PBAN reaches **QADS: 0.986 / 0.987 / 0.044** and **CVIU: 0.978 / 0.981 / 0.397**. A Bi-Atten-only ablation further shows that both directions are useful: on QADS, **HR\(\rightarrow\)SR only** gives **SRCC 0.957**, **SR\(\rightarrow\)HR only** gives **0.955**, and **bidirectional** gives **0.981**.

The paper also reports efficiency on QADS: **2.222M** parameters, **26.278G** FLOPs, and **16.995 ms** speed. It characterizes this as the **second-lowest parameter count**, relatively low FLOPs, and much better accuracy than larger models.

## 6. Efficient-PBAN and closed-loop perceptual optimization

Efficient-PBAN is an **Efficient Perceptual Bi-directional Attention Network** proposed as an image-level adaptation and practical successor to PBAN for **perceptual quality optimization of image super-resolution** [2602.21482]. Its target is not only quality prediction but also the use of the learned quality predictor as a **differentiable perceptual loss** inside SR training.

Efficient-PBAN is a **full-reference SR quality predictor** operating on an image pair
\[
\{x_{SR}, x_{HR}\}.
\]
Both inputs are processed by a **ResNet stem and the first residual block (Layer1) with shared parameters**; beyond Layer1, the branches are separated to capture distinct SR and HR statistics. The central block is **PBA\(^+\)**, which produces branch-specific query, key, and value embeddings,
\[
Q^{\gamma}, K^{\gamma}, V^{\gamma} = f_{\theta}(f^{\gamma}), \quad \gamma \in \{SR, HR\},
\]
and computes attention **along both the height and width axes**. The logits are normalized by their **standard deviation** rather than by \(\sqrt{d}\), and the axial outputs are averaged to form directional interaction outputs \(O_{HR \rightarrow SR}\) and \(O_{SR \rightarrow HR}\). These are fused by a **SubEC** module and then passed to a global quality prediction head with average pooling, flatten, linear layers, dropout, ReLU, and concatenation.

A central practical change relative to PBAN is that Efficient-PBAN is **image-level rather than patch-level**. The paper explicitly motivates this by noting that earlier quality predictors such as **PFIQA** and **PBAN** are patch based, making them expensive and awkward to use as end-to-end losses.

The model is trained on a self-constructed SR quality database containing **720 SR images** at approximately **2K resolution**, generated from **19 high-resolution reference images** selected from **DIV2K**. The database spans four upscaling factors—\(\times2\), \(\times3\), \(\times4\), and \(\times8\)—and includes SR outputs from **4 GAN-based**, **5 diffusion-based**, **4 transformer-based**, **1 flow-based**, and **1 CNN-based** methods. The split is **576** training, **72** validation, and **72** testing samples. Subjective scores were collected in a **single-stimulus experiment** under **ITU-R BT.500-14** conditions with **23 participants** aged **18–50** from **five regions**.

As a predictor, Efficient-PBAN is trained by direct score regression:
\[
\mathcal{L}_{Q} = \| \hat{q} - q \|^{2}.
\]
After pre-training, it is frozen and inserted into SR training. The paper defines
\[
\mathcal{L}_{D} = -Q(x_{SR}, x_{HR}, \theta_{SSIM}), \qquad
\mathcal{L}_{P} = - Q(x_{SR}, x_{HR}; \theta_{Efficient-PBAN}),
\]
and the malformed typesetting in the accessible text suggests an intended weighted combination with coefficients \(\alpha\) and \(\beta\). A plausible implication is a weighted objective of the form \(\mathcal{L} \approx \alpha \mathcal{L}_{D} + \beta \mathcal{L}_{P}\), which is consistent with the reported setting **\(+\)SSIM + Efficient-PBAN (\(\alpha=\beta=0.5\))**.

The reported findings follow a consistent pattern. **Efficient-PBAN loss improves perceptual quality**, **SSIM improves fidelity**, and **SSIM + Efficient-PBAN gives the best overall trade-off**. For **CAMixerSR on B100**, the four settings are:

- **Original**: PSNR 26.3513, SSIM 0.7188, LPIPS 0.3382, PFIQA 3.8528, EPBAN 2.8682  
- **+SSIM**: PSNR 25.6755, SSIM 0.7304, LPIPS 0.3358, PFIQA 3.8702, EPBAN 2.9415  
- **+EPBAN**: PSNR 25.6203, SSIM 0.6963, LPIPS 0.3567, PFIQA 3.8911, EPBAN 3.3165  
- **+SSIM+EPBAN**: PSNR 25.6741, SSIM 0.7110, LPIPS 0.3560, PFIQA 3.9348, EPBAN 3.4025  

A subjective test corroborates the trade-off. For **CAMixerSR**, MOS rankings are
\[
\text{EPBAN + SSIM } (3.25/5) > \text{EPBAN } (3.00/5) > \text{SSIM } (2.92/5) > \text{Original } (2.83/5),
\]
while for **LINF** they are
\[
\text{EPBAN + SSIM } (2.75/5) > \text{SSIM } (2.50/5) > \text{Original } (2.42/5) > \text{EPBAN } (1.92/5).
\]
The paper also notes a failure mode: **Efficient-PBAN-only optimization can produce window artifacts** in some patch-based SR backbones, which is why the joint objective with SSIM performs best perceptually.

## 7. Terminological boundaries, related architectures, and limitations

PBAN should not be conflated with earlier or adjacent bidirectional-attention architectures. Several works are strongly relevant conceptually but are **not literally PBAN**.

**BRIMs**—**Bidirectional Recurrent Independent Mechanisms**—addresses the combination of bottom-up sensory evidence and top-down contextual or prior information using attention and modular recurrent computation. It uses bidirectional inter-layer attention over a null source, the current lower layer, and the previous higher layer, with sparse module activation and within-layer communication. For a PBAN-like reading, its relevance lies in selective routing between bottom-up and top-down streams under modular computation, but it is a recurrent architecture for sequential perception, language, video prediction, and reinforcement learning rather than an SR-IQA network [2006.16981].

**BFAN**, the **Bidirectional Focal Attention Network**, is an image-text matching model that applies focal attention in both text-to-image and image-to-text directions. Its distinctive claim is that semantic misalignment persists when irrelevant fragments receive nonzero soft attention, so it introduces deterministic masking and renormalization over retained fragments. It is bidirectional and relevance-focused, but its problem setting is cross-modal retrieval rather than SR quality assessment [1909.11416].

**HBFP-Net**, the **Hierarchical Bi-Directional Feature Perception Network**, is a person re-identification model that uses cross-level correlation maps and a **Bi-directional Feature Perception** module to let low-, middle-, and high-level features reinforce one another. It is closely related in spirit to a perception-oriented bidirectional architecture, but its mechanism is cross-level feature perception for Re-ID rather than HR/SR bidirectional attention for SR FR-IQA [2008.03509].

Within SR itself, a recurrent source of confusion is the relation between **PBAN** and **Efficient-PBAN**. The latter is explicitly described as an **efficient image-level extension/adaptation** of the earlier PBAN idea rather than a synonym for the 2025 patch-based model [2602.21482]. The distinction matters because the original PBAN is a patch-level FR-IQA model, whereas Efficient-PBAN is designed to avoid extensive patch sampling and to act as a practical differentiable loss inside SR training.

The limitations are correspondingly task specific. For PBAN, the paper’s explicit future directions are to build a **larger-scale SR quality database** including newer **GAN-based and diffusion-based SR** methods, to extend toward **video SR quality assessment**, and to develop **quality-driven SR algorithms** [2509.06442]. A plausible limitation is that **patch-based training with patch MOS inheritance may ignore long-range contextual quality effects**. For Efficient-PBAN, the explicit or implied limitations are that it is **full-reference**, the **perception-distortion trade-off remains**, perceptual-only optimization can be unstable, and the training set—though broader than earlier SR-IQA databases—still reflects a limited number of references and SR generation pipelines [2602.21482].

Taken together, PBAN denotes a specific SR FR-IQA architecture centered on **reference-guided distortion alignment through bidirectional attention**, while also naming a broader design lineage in which bidirectional feature interaction is used to model perception-sensitive discrepancies rather than generic similarity alone.

Source: https://www.emergentmind.com/topics/perception-oriented-bidirectional-attention-network-pban