MPFR Module for Visible-Infrared Re-ID

• The paper introduces MPFR, which refines multi-scale features using spatially-aware masks to highlight modality-specific identity cues.
• MPFR extracts, aligns, and fuses features from multiple backbone stages using convolutions and attention-like operations for robust identity representation.
• Empirical tests on SYSU-MM01 show notable gains, with Rank-1 improvements up to 77.51% and mAP gains reaching 74.16%.

The Multi-Perception Feature Refinement (MPFR) module is a neural network component introduced to enhance Visible-Infrared Person Re-Identification (VI-ReID) by explicitly mining and aggregating modality-specific identity cues from the shallower layers of a shared feature extractor. Unlike approaches that focus exclusively on modality-invariant embeddings, MPFR targets the preservation and fusion of multi-scale discriminative features that are often suppressed or neglected in standard architectures. Its design emphasizes the refinement and spatially-aware selection of features from multiple perceptive levels, making them available for subsequent distillation and enhancement by downstream modules (Zhang et al., 4 Dec 2025).

1. Architectural Position and High-Level Role

MPFR is positioned downstream of the shared backbone (ResNet-50, stages 2–4) and immediately precedes the Semantic Distillation Cascade Enhancement (SDCE) module in the Identity Clue Refinement and Enhancement (ICRE) network. Its core functions are threefold: to extract features from three successive backbone stages at different resolutions, align all to a common scale and channel dimension, learn spatial "importance" masks highlighting the most identity-informative regions at each perceptual scale, and fuse these into a single identity-guided feature map. The approach is fully branch-free, relying on linear, convolutional, and attention-like operations.

2. Feature Extraction, Alignment, and Fusion Process

The MPFR module operates on three specific feature maps from the shared backbone:

• $f_\ell \in \mathbb{R}^{B, C_2, H_2, W_2}$ from stage 2 (e.g., $C_2=512$, $H_2=48$, $W_2=24$),
• $f_m \in \mathbb{R}^{B, C_3, H_3, W_3}$ from stage 3 ($C_3=1024$, $H_3=24$, $W_3=12$),
• $f_h \in \mathbb{R}^{B, C_4, H_4, W_4}$ from stage 4 ($C_4=2048$, $H_4=12$, $W_4=6$).

2.1 Feature-Scale Alignment

Each feature is processed by a dedicated $3 \times 3$ ConvBlock (convolution, batch normalization, ReLU) to yield aligned features $\hat{f}_i$, all reshaped to size $[B, C=256, H=12, W=6]$ by setting stride $s_\ell=4$ for $f_\ell$, $s_m=2$ for $f_m$, and $s_h=1$ for $f_h$.

2.2 Spatial-Mask Generation

For each $\hat{f}_i$:

• Channel-pooling generates two spatial maps per scale: $A_{i(\max)}$ and $A_{i(\text{avg})}$, concatenated across the channel dimension.
• A triple-branch of $3 \times 3$ convolutions with dilation rates $1$, $2$, and $3$ is applied to these pooled maps, and the outputs are summed to produce spatial attention logits $M_i'$.

2.3 Softmax Spatial Weighting

At each spatial position $(x,y)$, a softmax is computed across the three scales, resulting in normalized spatial weights $M_i''(x,y)$ such that $\sum_{i} M_i''(x,y) = 1$ for each position.

2.4 Weighted Fusion and Channel Restoration

A weighted sum $$F = \sum_{i \in \{\ell, m, h\}} M_i'' \odot \hat{f}_i$$ fuses the features across scales, which is then passed through a $1 \times 1$ ConvBlock (to $C_4=2048$ channels) to restore depth before output.

The data flow can be summarized as:

Stage	Operation	Output Size
Backbone features	$f_\ell$, $f_m$, $f_h$	512, 1024, 2048 ch
Alignment	3×3 ConvBlock, stride 4,2,1	[256,12,6] each
Masking	Channel pooling + 3 dilated convs	[1,12,6] per scale
Weighting	Softmax over scales at each position	[1,12,6] per scale
Fusion	Weighted sum $\rightarrow$ final ConvBlock	[2048,12,6]

3. Mathematical Formulation

Formally, MPFR can be expressed as follows:

• Alignment: $$\hat{f}_i = \text{ReLU}( \text{BN}( W_i * f_i + b_i ) )$$, with $W_i$ as $3 \times 3$ convolutional weights, $s_i$ as stride.
• Mask Generation: For each $i$:
  • $A_{i,\max} = \max_c \hat{f}_i$, $A_{i,\text{avg}} = \text{mean}_c \hat{f}_i$
  • $M_i = [A_{i,\max}; A_{i,\text{avg}}]$
  • $$M_i' = \sum_{d \in \{1,2,3\}} \text{Conv}_{\text{dil}}^{(d)}(M_i)$$
• Softmax Weighting: $M_i''(p) = \exp(M_i'(p)) / \sum_{j} \exp(M_j'(p))$, for $p=(x,y)$.
• Fusion: $F = \sum_{i} M_i'' \odot \hat{f}_i$
• Channel Restoration: $$\tilde{f} = \text{ReLU}( \text{BN}( W_0 * F + b_0 ) )$$, with $W_0$ a $1 \times 1$ convolution to 2048 channels.

The variable definitions and parameterization are tightly specified, ensuring implementation fidelity.

4. Module Integration and Downstream Effects

The output of MPFR ($\tilde{f}$) has the same spatial size and final channel dimension as the deep backbone feature ($f_h$), enabling direct integration with SDCE. The SDCE then proceeds with a two-step transformer-based cascade:

• Block 1: Cross-attention with $Q \leftarrow f_h$, $K, V \leftarrow \tilde{f}$.
• Block 2: Self-attention on Block 1's output.

These operations further distill identity-aware features, after which global pooling and an ICG loss are applied to optimize cross-modal feature separation. The MPFR-generated features are thus not endpoint representations, but rather identity-sensitive feature banks for subsequent semantic distillation and final Re-ID embedding formation (Zhang et al., 4 Dec 2025).

5. Pseudocode Specification

A forward pass for MPFR consistent with the paper's implementation detail can be represented as:

def MPFR(f_l, f_m, f_h): hat_l = ConvBlock3x3(in_ch=512, out_ch=256, stride=4)(f_l) # [B,256,12,6] hat_m = ConvBlock3x3(in_ch=1024, out_ch=256, stride=2)(f_m) # [B,256,12,6] hat_h = ConvBlock3x3(in_ch=2048, out_ch=256, stride=1)(f_h) # [B,256,12,6] masks = [] for hat in (hat_l, hat_m, hat_h): maxc = hat.max(dim=1, keepdim=True)[0] avgc = hat.mean(dim=1, keepdim=True) M = torch.cat([maxc, avgc], dim=1) m1 = dilConv(M, dilation=1) m2 = dilConv(M, dilation=2) m3 = dilConv(M, dilation=3) masks.append(m1 + m2 + m3) stacked = torch.cat(masks, dim=1) attn = torch.softmax(stacked, dim=1) M_l, M_m, M_h = attn.split(1, dim=1) F = M_l * hat_l + M_m * hat_m + M_h * hat_h tilde = ConvBlock1x1(in_ch=256, out_ch=2048)(F) return tilde

This implementation is specified for reproducibility and accuracy, ensuring that feature alignment, masking, and fusion follow the outlined processing flow (Zhang et al., 4 Dec 2025).

6. Empirical Validation

Systematic ablation on the SYSU-MM01 all-search, single-shot VI-ReID protocol isolates the effect of MPFR:

• Baseline (AGW + Triplet): Rank-1 = 70.21%, mAP = 68.48%
• +MPFR only (Triplet): Rank-1 = 76.22% (+6.01), mAP = 72.66% (+4.18)
• +MPFR only (ICG Loss): Rank-1 = 77.51%, mAP = 74.16%

These results indicate a substantial, isolated performance gain from MPFR integration. Additionally, feature-distribution plots demonstrate reduced intra-class (same ID) distances and increased inter-class separation post-MPFR. Visualizations via Grad-CAM reveal that MPFR shifts focus towards semantically salient body regions and away from backgrounds. Top-10 retrieval results confirm reduction in false matches due to MPFR's effect (Zhang et al., 4 Dec 2025).

7. Conceptual Significance and Role in Cross-Modal Person Re-ID

MPFR is explicitly designed to extract and leverage modality-specific “identity clues” that reside in shallow convolutional responses, particularly color, texture, and thermal patterns that are often diluted in deeper, modality-invariant embeddings. By generating spatial masks for scale-aware fusion, the module harnesses information crucial to discriminative learning in VI-ReID. Its empirical performance and modular, lightweight design demonstrate that gathering and refining multi-scale shallow features significantly enhances cross-modal retrieval tasks without introducing substantial architectural complexity. A plausible implication is that future VI-ReID models may increasingly integrate analogous spatial-scale-aware modules to maximize representation diversity and robustness in the presence of modality gaps (Zhang et al., 4 Dec 2025).