Dual-View Pyramid Pooling (DVPP)

• The paper introduces DVPP, a novel pooling mechanism that merges spatial and cross-channel features using multi-scale aggregation for enhanced calibration and accuracy.
• DVPP combines spatial pyramid pooling (SPP) and cross-channel pooling (CCP) to capture both global context and fine-grained details in medical imaging.
• Empirical results demonstrate that DVPP significantly outperforms traditional methods with improved balanced accuracy and reduced calibration errors on 2D/3D medical benchmarks.

Dual-View Pyramid Pooling (DVPP) is a parameter-free pooling framework for deep neural networks designed to jointly aggregate spatial and channel-wise statistics at multiple scales. Developed for improved medical image classification and confidence calibration, DVPP explicitly addresses the shortcomings of traditional spatial pooling (SP) and cross-channel pooling (CCP) by introducing a dual-view perspective. Its plug-in design systematically combines multi-scale spatial and cross-channel features, yielding more informative and better-calibrated representations before the classifier stage, and demonstrates state-of-the-art empirical performance on both 2D and 3D medical image benchmarks (Zhang et al., 2024).

1. Theoretical Foundations

Standard pooling operations in convolutional neural networks serve to condense feature maps, reducing computational load and mitigating overfitting. However, such reductions incur limits on information retention. SP—such as Global Average Pooling (GAP) or Spatial Pyramid Pooling (SPP)—aggregates activations within each channel across spatial dimensions, efficiently capturing global context but discarding fine-grained details relevant for tasks such as lesion identification. Conversely, CCP (e.g., Cross-Channel Average Pooling or CAP) collects statistics over the channel dimension at each spatial site, preserving subtle local cues but underemphasizing global context, leading to potential miscalibration and suboptimal classification.

DVPP formalizes SP and CCP as orthogonal (“dual-view”) reductions of the feature tensor: SP collapses spatial axes $(H, W)$ (yielding global, channel-specific descriptors), while CCP collapses the channel axis $C$ for each spatial site (yielding 2D spatial maps summarizing groups of channels). The dual-view hypothesis posits that joint, multi-scale pooling along both axes produces complementary feature encodings, improving both classification accuracy and confidence calibration (Zhang et al., 2024).

2. Mathematical Formulation

Let $F\in\mathbb R^{C\times H\times W}$ denote the terminal convolutional feature map.

2.1. Single-Axis Pooling

• Spatial Pyramid Pooling (SPP): For scale $s$, partition the spatial grid into an $s\times s$ lattice, averaging activations within each region and per channel. This yields $\mathrm{SP}_s(F) \in \mathbb R^{C \cdot s^2}$, gathering channel-specific global and regional summaries.
• Cross-Channel Pyramid Pooling (CCP): For scale $s$, partition $C$ channels into $s$ groups, average each group over the $C/s$ assigned channels (per spatial site), producing $C$0 2D spatial maps of shape $C$1. Denote as $C$2.

2.2. Dual-View Pyramid Pooling Construction

Specify a collection of pyramid scales $C$3; extract both SP and CCP representations for each. Concatenate all such features to form

$C$4

This vector is supplied to a classifier head (typically a small fully-connected network plus softmax).

3. Implementational Variants

DVPP is instantiated in five distinct, parameter-free designs, all adhering to the dual-view principle:

Variant	Architectural Logic
SC-DVPP-Ser	For each scale, apply SP, then CCP to SP features; concatenate CCP(SP(F)) across scales.
SC-DVPP-S-Ser	As above, but concatenate both CCP(SP(F)) and the raw SP features.
SC-DVPP-C-Ser	As above, but concatenate both CCP(SP(F)) and the raw CCP features.
SC-DVPP-Par	Apply SP and CCP in parallel (per scale), concatenate all outputs.
Twins-DVPP	Concatenate SP(CCP(F)), raw CCP(F), and raw SP(F) for each scale (hybrid, but prone to redundancy).

Empirical ablation finds SC-DVPP-Ser excels at scales $C$5, SC-DVPP-C-Ser with SP scale $C$6 and CCP scales $C$7, and SC-DVPP-Par when using SP at $C$8 and CCP at $C$9 (Zhang et al., 2024).

4. Integration with Deep Backbones

DVPP is a drop-in replacement for the standard pooling-then-fully-connected pattern in image classification architectures including ResNet, DenseNet, 3D-ResNet, and Vision Transformers. After extracting the final convolutional feature map $F\in\mathbb R^{C\times H\times W}$0, a selected DVPP variant computes $F\in\mathbb R^{C\times H\times W}$1, which is then classified by a shallow linear head. The framework is fully trainable end-to-end, with pooling modules containing no learnable parameters.

A forward pass (e.g., with SC-DVPP-C-Ser) comprises:

1. Image $F\in\mathbb R^{C\times H\times W}$2 CNN backbone $F\in\mathbb R^{C\times H\times W}$3 feature map $F\in\mathbb R^{C\times H\times W}$4
2. For each scale $F\in\mathbb R^{C\times H\times W}$5:
   • $F\in\mathbb R^{C\times H\times W}$6 (SP features)
   • $F\in\mathbb R^{C\times H\times W}$7 (CCP features)
   • $F\in\mathbb R^{C\times H\times W}$8 (CCP after SP)
3. Concatenate $F\in\mathbb R^{C\times H\times W}$9
4. Classifier: $s$0

5. Experimental Evaluation

DVPP was benchmarked on six public 2D/3D medical image classification datasets: ISIC2018 (dermoscopy, seven classes), BTM (brain tumors, four classes), APTOS2019 (diabetic retinopathy), NIH-CXR-LT (long-tailed chest X-rays, 19 diseases plus normal), OASIS (Alzheimer's disease), and ABIDE-I (autism/control, resting-state fMRI). Networks were evaluated using metrics for accuracy (ACC), balanced accuracy (bAcc), macro F1 (mF1), Cohen’s Kappa, expected calibration error (ECE), and Brier score (BS).

Compared to conventional pooling (GAP, SPP, LIP, CAP, CCPP, and parametric variants) and calibration strategies (temperature scaling, Focal Loss, LDAM, Mixup), DVPP delivered consistently superior results in both accuracy and calibration. For instance:

• On ISIC2018 with ResNet50, SC-DVPP-C-Ser achieved bAcc 83.4% (+7.9 pp vs. SPP) and ECE 8.7% (−0.8 pp).
• On BTM, SC-DVPP-C-Ser improved ACC by +3.4 pp and reduced ECE by 2.8 pp relative to the best single-axis pooling.
• On APTOS2019, SC-DVPP-C-Ser outperformed GAP and SPP in both balanced accuracy and ECE.
• On NIH-CXR-LT, SC-DVPP-C-Ser reduced ECE to 5.2% (compared to 23.9% for GAP and 34.3% for SPP).

SC-DVPP-C-Ser also outperformed Mixup-based calibration (mF1 = 82.9 vs. 65.2; ECE = 8.7 vs. 9.53), indicating benefits from feature enrichment before classification (Zhang et al., 2024).

6. Discussion, Limitations, and Outlook

The principal advantages of DVPP are its ability to fuse global and local information via complementary SP and CCP views, its parameter-free pooling (enhancing robustness when training on limited data), and its multi-scale architecture for hierarchical context aggregation. Empirical t-SNE analyses demonstrate tighter class clustering and improved separation, and reliability diagrams show improved confidence-accuracy alignment.

Limitations include increased feature dimensionality, potentially addressed by compression or attention-based fusion. The current instantiations rely exclusively on average pooling; extensions to learnable or non-linear pooling (and to dense prediction or self-supervised settings) are proposed as directions for future work. No formal information-theoretic argument for dual-view calibration effectiveness is yet provided (Zhang et al., 2024).

DVPP represents a dual-axis paradigm for pooling in deep learning, systematically integrating spatial and channel-wise features for enhanced accuracy and calibration, particularly suited to medical imaging domains.