GSoP-Net2: Advanced Second-Order Pooling

• GSoP-Net2 is an advanced convolutional architecture that leverages global second-order pooling with explicit matrix square-root computations to enhance feature representation.
• It builds upon GSoP-Net1 by integrating covariance-based attention throughout a ResNet-style backbone, leading to improved accuracy on large-scale visual tasks.
• The design efficiently balances enhanced non-linear modeling with moderate computational overhead, offering practical gains in performance and representation.

Global Second-order Pooling Network 1 (GSoP-Net1) is a deep convolutional network architecture that systematically integrates global second-order pooling into all major intermediate stages of a ResNet-style backbone, augmenting both representational power and non-linear modeling capability compared to first-order pooling approaches. GSoP-Net1 was introduced as the first-order–pooled variant in “Global Second-order Pooling Convolutional Networks” by Gao et al., providing a comprehensive framework for leveraging higher-order statistics—specifically, second-order (covariance) information—throughout a convolutional neural network rather than only at the terminal layer. The architecture is empirically validated on large-scale visual recognition tasks and achieves robust improvements over various established baselines (Gao et al., 2018).

1. Formal Definition of Global Second-order Pooling

Let $X \in \mathbb{R}^{H \times W \times C}$ denote the output tensor from a convolutional layer, where $x_{ij} \in \mathbb{R}^C$ is the feature vector at spatial location $(i,j)$, with $i=1\ldots H$, $j=1\ldots W$. Denote $N=HW$. The mean feature vector is $\mu = \frac{1}{N} \sum_{i=1}^H \sum_{j=1}^W x_{ij}$. The (centered) channel covariance matrix $C \in \mathbb{R}^{C \times C}$ is computed as:

$$C = \frac{1}{N} \sum_{i=1}^H \sum_{j=1}^W (x_{ij}-\mu)(x_{ij}-\mu)^T$$

In practice, $\mu$ may be set to zero as batch normalization (BN) is applied row-wise to $x_{ij} \in \mathbb{R}^C$0 downstream. After obtaining $x_{ij} \in \mathbb{R}^C$1, two subsequent transformations are applied: (1) row-wise BN, $x_{ij} \in \mathbb{R}^C$2; (2) a nonlinear 1$x_{ij} \in \mathbb{R}^C$31 row-conv embedding followed by a LeakyReLU activation and logistic sigmoid, producing a channel-wise (or spatial-wise) weighting vector $x_{ij} \in \mathbb{R}^C$4: $x_{ij} \in \mathbb{R}^C$5 where $x_{ij} \in \mathbb{R}^C$6 are learned parameters, $x_{ij} \in \mathbb{R}^C$7 denotes LeakyReLU with slope 0.1, and $x_{ij} \in \mathbb{R}^C$8 is the element-wise sigmoid. A plausible implication is that these transformations allow the model to non-linearly re-weigh channels (or spatial locations) as an attention mechanism based on holistic, second-order statistics.

Optional matrix functions such as the matrix square root $x_{ij} \in \mathbb{R}^C$9 can be incorporated via eigendecomposition or Newton–Schulz iteration, but GSoP-Net1 itself does not utilize these at the final stage, deferring such procedures to the GSoP-Net2 variant.

2. GSoP Block Structure: Channel-wise and Spatial-wise Variants

GSoP-Net1 implements both channel-wise and spatial-wise GSoP blocks as modular plug-ins to convolutional backbones.

Channel-wise GSoP block:

• Input tensor $(i,j)$0 undergoes a 1$(i,j)$11 convolution reducing channel dimension $(i,j)$2 (typically $(i,j)$3).
• The result $(i,j)$4 is flattened to an $(i,j)$5 matrix ($(i,j)$6). Covariance $(i,j)$7 is computed.
• Row-wise BN and two-layer row-conv with nonlinearity yield $(i,j)$8.
• $(i,j)$9 is expanded and broadcast to rescale the original $i=1\ldots H$0 across channels:

$i=1\ldots H$1

for all $i=1\ldots H$2, $i=1\ldots H$3, $i=1\ldots H$4.

Spatial-wise GSoP block:

• 1$i=1\ldots H$51 convolution reduces $i=1\ldots H$6, producing $i=1\ldots H$7. Spatial downsampling (e.g., $i=1\ldots H$8) yields $i=1\ldots H$9.
• Reshape $j=1\ldots W$0 to $j=1\ldots W$1 ($j=1\ldots W$2) and compute spatial covariance $j=1\ldots W$3 as $j=1\ldots W$4.
• Nonlinear embedding yields $j=1\ldots W$5, reshaped and upsampled to $j=1\ldots W$6, which is broadcast to rescale $j=1\ldots W$7 spatially:

$j=1\ldots W$8

3. Placement of GSoP Blocks in ResNet-Style Backbone

GSoP-Net1 adopts a ResNet-50 backbone, inserting channel-wise GSoP blocks after each of the four major residual stages. The precise placements are as follows:

Stage	Output Shape	Bottleneck Structure	GSoP Block Insertion
conv2_x	56×56×256	[1×1,64]→[3×3,64]→[1×1,256] × 3	After final bottleneck
conv3_x	28×28×512	[1×1,128]→[3×3,128]→[1×1,512] × 4	After stage
conv4_x	14×14×1024	[1×1,256]→[3×3,256]→[1×1,1024] × 6	After stage
conv5_x	7×7×2048	[1×1,512]→[3×3,512]→[1×1,2048] × 3	After stage

After the last GSoP block, global average pooling reduces the tensor to $j=1\ldots W$9, followed by a fully connected layer for classification.

4. Layer-wise Architectural Overview

The forward pass through GSoP-Net1 consists of the following ordered sequence:

1. Input: $N=HW$0
2. Conv1: $N=HW$1, 64, stride 2 $N=HW$2 BN $N=HW$3 ReLU ($N=HW$4)
3. Pool1: $N=HW$5, max, stride 2 ($N=HW$6)
4. conv2_x: 3 bottlenecks ($N=HW$7)
5. GSoP block ($N=HW$8, channel-wise, $N=HW$9)
6. conv3_x: 4 bottlenecks ($\mu = \frac{1}{N} \sum_{i=1}^H \sum_{j=1}^W x_{ij}$0)
7. GSoP block ($\mu = \frac{1}{N} \sum_{i=1}^H \sum_{j=1}^W x_{ij}$1, channel-wise, $\mu = \frac{1}{N} \sum_{i=1}^H \sum_{j=1}^W x_{ij}$2)
8. conv4_x: 6 bottlenecks ($\mu = \frac{1}{N} \sum_{i=1}^H \sum_{j=1}^W x_{ij}$3)
9. GSoP block ($\mu = \frac{1}{N} \sum_{i=1}^H \sum_{j=1}^W x_{ij}$4, channel-wise, $\mu = \frac{1}{N} \sum_{i=1}^H \sum_{j=1}^W x_{ij}$5) 10. conv5_x: 3 bottlenecks ($\mu = \frac{1}{N} \sum_{i=1}^H \sum_{j=1}^W x_{ij}$6)
10. GSoP block ($\mu = \frac{1}{N} \sum_{i=1}^H \sum_{j=1}^W x_{ij}$7, channel-wise, $\mu = \frac{1}{N} \sum_{i=1}^H \sum_{j=1}^W x_{ij}$8)
11. Global average pooling ($\mu = \frac{1}{N} \sum_{i=1}^H \sum_{j=1}^W x_{ij}$9)
12. Fully-connected layer ($C \in \mathbb{R}^{C \times C}$0), softmax output

Each channel-wise GSoP block introduces approximately $C \in \mathbb{R}^{C \times C}$1 million parameters and $C \in \mathbb{R}^{C \times C}$2 MFLOPs, with an optional spatial-wise branch adding $C \in \mathbb{R}^{C \times C}$3 million parameters and $C \in \mathbb{R}^{C \times C}$4 MFLOPs.

5. Implementation and Computational Characteristics

Key implementation details of GSoP-Net1 include:

• 1$C \in \mathbb{R}^{C \times C}$51 convolutions consistently reduce the channel dimension prior to covariance computation, fixing $C \in \mathbb{R}^{C \times C}$6 in all GSoP blocks.
• The channel-wise GSoP operation is efficiently implemented via a matrix multiplication of size $C \in \mathbb{R}^{C \times C}$7 and $C \in \mathbb{R}^{C \times C}$8.
• Row-wise BN is a batch normalization applied along the rows of the $C \in \mathbb{R}^{C \times C}$9 covariance.
• Two 1$$C = \frac{1}{N} \sum_{i=1}^H \sum_{j=1}^W (x_{ij}-\mu)(x_{ij}-\mu)^T$$01 row-convolutions (with LeakyReLU nonlinearity) effect the embedding/attention computation, acting independently on each row.
• Matrix square-root computation and explicit eigen-decomposition are not used in GSoP-Net1, distinguishing it from the “Net2” variant.
• No large 3D convolutions are introduced. Overhead relative to the ResNet-50 baseline is approximately $$C = \frac{1}{N} \sum_{i=1}^H \sum_{j=1}^W (x_{ij}-\mu)(x_{ij}-\mu)^T$$1 in parameters and $$C = \frac{1}{N} \sum_{i=1}^H \sum_{j=1}^W (x_{ij}-\mu)(x_{ij}-\mu)^T$$2 in FLOPs.

6. Performance on ImageNet-1K and Comparative Analysis

Empirical evaluation of GSoP-Net1 on ImageNet-1K demonstrates significant improvement relative to both first-order and end-only second-order baselines:

Model	Top-1 Error (%)	Top-5 Error (%)
ResNet-50 (baseline)	23.85	7.13
GSoP-Net1	22.32 (↓1.53)	6.02 (↓1.11)
SE-Net-50	23.29	6.62
CBAM	22.66	6.31
MPN-COV	22.74	6.54

Ablation studies on scale-reduced ResNet-26 show that single GSoP blocks in early (conv2_x) or late (conv5_x) stages yield Top-1 errors of 18.45% and 18.33%, respectively; applying GSoP blocks throughout all four stages reduces Top-1 error to 17.42% (from a baseline of 19.18%). This confirms that intermediate second-order pooling delivers cumulative accuracy gains, outperforming first-order (SE, CBAM) and “end-only” (MPN-COV) second-order strategies.

7. Context and Significance

GSoP-Net1 establishes the effectiveness of integrating global second-order pooling throughout the depth of a residual network, as opposed to restricting such pooling to the final layer. The block design leverages covariance-based attention both channel- and spatial-wise without incurring prohibitive computational cost or requiring matrix square-root operations. The architecture demonstrates that holistic second-order statistics, when used as feature recalibration signals at multiple network depths, deliver consistent and non-trivial improvements in large-scale visual recognition tasks (Gao et al., 2018). This approach represents a significant advance in the practical use of higher-order global descriptors in deep convolutional architectures.