Granular Pyramid Convolution in CNNs

• Granular Pyramid Convolution (GPC) is a concept unifying multi-scale feature integration via explicit pyramid-level and implicit kernel-level designs.
• It encompasses methods like SEPC, which uses a modified 3D convolution and integrated batch normalization to align scales across feature maps.
• It also includes PSConv that assigns heterogeneous dilation rates to kernels, achieving effective scale adaptation with minimal extra computational cost.

Granular Pyramid Convolution (GPC) is not a standardized operator name in the relevant 2020 arXiv literature. As an Editor's term, it can denote a family of convolutional designs that encode multi-scale structure at finer granularity than conventional feature-pyramid fusion or homogeneous per-layer convolution. In that broad sense, two neighboring formulations are especially informative. Pyramid convolution and its scale-equalizing variant treat the feature pyramid as a modified 3-D domain spanning scale and spatial dimensions, and explicitly aggregate adjacent pyramid levels (Wang et al., 2020). Poly-Scale Convolution (PSConv) instead compresses a spectrum of receptive-field scales into one compact convolutional layer by assigning different dilation rates to individual kernels in the $(\text{output channel}, \text{input channel})$ lattice (Li et al., 2020). Together, these works delineate two technically distinct but conceptually adjacent interpretations of “granular pyramid” behavior in CNNs.

1. Terminological status and conceptual scope

The phrase “Granular Pyramid Convolution” does not appear explicitly in either “Scale-Equalizing Pyramid Convolution” (Wang et al., 2020) or “PSConv: Squeezing Feature Pyramid into One Compact Poly-Scale Convolutional Layer” (Li et al., 2020). Accordingly, SEPC is not claimed to be GPC, nor is PSConv presented under that name. The safest encyclopedic treatment is therefore taxonomic rather than identificatory: GPC is best understood as a possible umbrella label for methods that make scale interaction more explicit, more local, or more fine-grained than standard feature-pyramid processing.

Within that umbrella, the two papers instantiate different notions of granularity. In SEPC, granularity lies along the pyramid-level axis: the response at level $l$ depends on $x^l$ and also on adjacent levels such as $x^{l-1}$ and $x^{l+1}$. In PSConv, granularity lies inside kernel space: different $(c,k)$ kernel pairs use different dilation rates within one layer. This suggests that GPC is not a single canonical mechanism, but a design space spanning at least two axes: explicit cross-level convolution over a feature pyramid, and implicit pyramid construction within a compact layer through heterogeneous receptive fields.

A recurrent misconception is to equate any such method directly with GPC. The literature summarized here does not support that equivalence. A more precise statement is that SEPC is a closely related and potentially precursor-like method for cross-scale pyramid convolution, while PSConv is a granular, kernel-space, dilation-based pyramid convolution in a conceptual sense.

2. Pyramid-level convolution as a modified 3-D operator

SEPC begins from a critique of standard feature-pyramid processing: development around feature pyramids mainly focuses on aggregating contextual information at different levels while seldom touching the inter-level correlation in the feature pyramid. RetinaNet-style heads also apply the same 2-D convolution stack independently on each pyramid level, so the head is shared across levels but each level is still processed independently in scale (Wang et al., 2020).

To address this, the paper introduces pyramid convolution (PConv), described as a modified 3-D convolution. The two spatial dimensions are $H \times W$, and the third dimension is the scale or pyramid-level axis. Because adjacent pyramid levels do not share spatial resolution, the operator is not a literal dense 3-D convolution on a regular cuboid tensor. Instead, it uses scale-dependent sampling rules so that neighboring levels can be aligned to the target level.

For the common case $N=3$, the paper gives the defining expression

$$y^l = w_{1} \ast_{s0.5} x^{l+1} + w_{0} \ast x^{l} + w_{-1} \ast_{s2} x^{l-1},$$

and its practical implementation

$$y^l = \mathrm{Upsample}(w_{1} \ast x^{l+1}) + w_{0} \ast x^{l} + w_{-1} \ast_{s2} x^{l-1}.$$

The three kernel slices $l$0 are associated with offsets along the scale axis. Aggregation from $l$1 to $l$2 is realized by convolving at the coarser level and then bilinearly upsampling; aggregation from $l$3 to $l$4 uses stride-2 convolution. Boundary handling is conventional: for the bottom level, the term from $l$5 is omitted; for the top level, the term from $l$6 is omitted; zero-padding is also used similarly to conventional convolution.

This formulation yields a direct reinterpretation of the RetinaNet head. The paper states that the original RetinaNet head is a special case of PConv with scale kernel $l$7, namely repeated application of $l$8. In that sense, PConv generalizes per-level head processing into an operator with explicit locality in scale. For GPC-like thinking, this is the clearest example of granularity defined over neighboring pyramid levels rather than over architectural branches alone.

3. Scale equalization and pyramid-wide normalization

The SEPC paper argues that naive PConv is best suited to a Gaussian pyramid, whereas a CNN feature pyramid is not a true Gaussian pyramid. The stated reasons are that many convolutions happen between levels, nonlinearities distort the effective blur, and the equivalent blur is stronger and spatially varying. The paper makes the explicit remark that pyramid convolution is able to extract scale-invariant features from a Gaussian pyramid, but that this assumption is mismatched to real backbone-generated feature pyramids (Wang et al., 2020).

SEPC modifies PConv by aligning the shared pyramid convolution kernel only at high-level feature maps. Operationally, the kernel convolving with the bottom feature map is fixed as a normal $l$9 convolution. As it processes high-level feature maps in the feature pyramid, a deformation offset is predicted based on the current layer of feature map. The intention is to equalize scales so that the shared kernel corresponds to more comparable receptive-field structure across levels. The decision to align only high-level maps is motivated by three explicit considerations: the mismatch grows upward in the pyramid, the bottom level serves as a reference geometry, and the computational cost of a convolution reduces by 4 from one layer to its upper feature pyramid level, so deformable alignment on high levels incurs minimal computations.

The same scale-space viewpoint motivates integrated batch normalization (iBN). After PConv, the paper inserts a single shared BN layer with affine form

$x^l$0

The distinctive point is that the BN statistics are computed from all feature maps in the pyramid jointly rather than separately at each level. The paper contrasts three regimes: Single BN, where $x^l$1 are shared but statistics are separate per level; Independent BN, where $x^l$2 are all separate; and Integrated BN, where $x^l$3 are shared and statistics are aggregated over all levels. The reported conclusion is that single BN hurts badly, independent BN helps, and iBN helps more due to more stable statistics. A plausible implication is that any GPC-like operator that treats the pyramid as a coupled scale-space tensor will face normalization issues analogous to those addressed by iBN.

SEPC is also offered in two variants. SEPC-full applies SEPC to both the combined/shared head and the extra head, whereas SEPC-lite applies SEPC only to the extra head. This establishes an explicit speed–accuracy tradeoff inside the scale-equalizing formulation.

4. Kernel-lattice granularity in Poly-Scale Convolution

PSConv addresses scale variance from a different direction. Its starting point is that multi-scale feature fusion from different layers or filters attracts great attention among existing solutions, while the more granular kernel space is overlooked. The proposed operator mixes up a spectrum of dilation rates and tactfully allocates them in the individual convolutional kernels of each filter regarding a single convolutional layer (Li et al., 2020).

The paper defines an input tensor

$x^l$4

and a filter tensor

$x^l$5

The central addition is a dilation-rate matrix

$x^l$6

where each element $x^l$7 specifies the dilation rate of kernel $x^l$8. Thus dilation is no longer homogeneous at the layer level. Different output–input channel pairs can sample at different spatial scales inside one compact operator.

The allocation varies cyclically along both the input-channel axis and the output-channel axis. The $x^l$9 channels are divided into $x^{l-1}$0 partitions, a cyclic pattern of dilation rates $x^{l-1}$1 is filled in, and adjacent filters are shifted versions of one another. The authors explicitly seek non-identical elements in each row and each column of $x^{l-1}$2, so that multi-scale mixing occurs both when producing an output channel and when mapping a given input channel into different outputs. This two-axis heterogeneity is one of the paper’s most distinctive design choices.

In the main PSConv operator, no masking is used. The weight tensor retains the standard shape $x^{l-1}$3, so the method does not alter parameter dimensionality. The paper repeatedly states that PSConv introduces no additional parameters and no theoretical extra computational complexity relative to standard convolution, because kernel size and kernel count remain unchanged; only dilation indices change. In conceptual terms, this “squeezes a feature pyramid into one compact convolutional layer.” For GPC, the significance is that pyramid behavior can be encoded without explicit multi-resolution feature maps: the pyramid is implicit in the heterogeneous kernel lattice.

5. Architectural placement, efficiency, and empirical behavior

The two formulations occupy different architectural locations. PConv and SEPC are designed to fit naturally into RetinaNet-style one-stage detectors. The paper discusses directly replacing the four convolutional head layers with four stacked PConv modules of scale kernel $x^{l-1}$4, and also a more efficient combined/shared head in which four PConv modules are shared by classification and localization branches, followed by one extra normal convolution per branch. PConv is also evaluated as a stand-alone pyramid-processing module in Faster R-CNN, Mask R-CNN, and HTC (Wang et al., 2020).

PSConv is presented as a drop-in replacement in prevailing CNN backbones. For ImageNet and CIFAR-100 classification, it replaces all $x^{l-1}$5 standard convolutions in the middle of bottleneck blocks. For COCO detection and instance segmentation, it replaces all $x^{l-1}$6 convolution layers in the pretrained backbone network, while keeping the FPN neck convolutions standard; replacing the FPN convolutions yielded only marginal gain (Li et al., 2020).

Formulation	Typical placement	Reported outcomes
PConv / SEPC	RetinaNet-style head; PConv also in Faster R-CNN, Mask R-CNN, HTC	$x^{l-1}$7 AP increase on MS-COCO2017 for SEPC; SEPC-lite has $x^{l-1}$8 AP gain with only around 7% inference time increase
PSConv	Backbone $x^{l-1}$9 convolutions in ResNet, ResNeXt, SE-ResNet	No additional parameters and no theoretical extra computational complexity; consistent gains on ImageNet and COCO

The reported detector gains for SEPC are substantial. On COCO minival with ResNet-50 and a $x^{l+1}$0 schedule, RetinaNet improves from $x^{l+1}$1 AP to $x^{l+1}$2 with PConv, $x^{l+1}$3 with PConv+iBN, $x^{l+1}$4 with SEPC-lite, and $x^{l+1}$5 with SEPC. FSAF improves from $x^{l+1}$6 to $x^{l+1}$7, $x^{l+1}$8, $x^{l+1}$9, and $(c,k)$0, respectively. FreeAnchor improves from $(c,k)$1 to $(c,k)$2, $(c,k)$3, $(c,k)$4, and $(c,k)$5. In two-stage settings using PConv only, Faster R-CNN improves from $(c,k)$6 to $(c,k)$7, Mask R-CNN from $(c,k)$8 to $(c,k)$9, and HTC from $H \times W$0 to $H \times W$1 (Wang et al., 2020).

The efficiency profile of SEPC is similarly explicit. For RetinaNet, the baseline is $H \times W$2 G and $H \times W$3 ms, PConv is $H \times W$4 G and $H \times W$5 ms, PConv+iBN is $H \times W$6 G and $H \times W$7 ms, SEPC-lite is $H \times W$8 G and $H \times W$9 ms, and SEPC is $N=3$0 G and $N=3$1 ms. The paper also states that the total FLOPs of PConv is actually only around 1.5 times as much as the original head, and that some head designs can even have less FLOPs than the original RetinaNet head.

PSConv reports comparable breadth across classification and dense prediction. On ImageNet, ResNet-50 improves from $N=3$2 top-1/top-5 error to $N=3$3, and ResNet-101 from $N=3$4 to $N=3$5. On COCO, Faster R-CNN with ResNet-50 improves from $N=3$6 to $N=3$7, with ResNet-101 from $N=3$8 to $N=3$9, and with ResNeXt-101 from $$y^l = w_{1} \ast_{s0.5} x^{l+1} + w_{0} \ast x^{l} + w_{-1} \ast_{s2} x^{l-1},$$0 to $$y^l = w_{1} \ast_{s0.5} x^{l+1} + w_{0} \ast x^{l} + w_{-1} \ast_{s2} x^{l-1},$$1. Mask R-CNN and cascade variants show similar positive shifts. The paper further notes that the strongest gains often appear in $$y^l = w_{1} \ast_{s0.5} x^{l+1} + w_{0} \ast x^{l} + w_{-1} \ast_{s2} x^{l-1},$$2, for example Faster R-CNN R101 $$y^l = w_{1} \ast_{s0.5} x^{l+1} + w_{0} \ast x^{l} + w_{-1} \ast_{s2} x^{l-1},$$3 and Mask R-CNN R101 box $$y^l = w_{1} \ast_{s0.5} x^{l+1} + w_{0} \ast x^{l} + w_{-1} \ast_{s2} x^{l-1},$$4 $$y^l = w_{1} \ast_{s0.5} x^{l+1} + w_{0} \ast x^{l} + w_{-1} \ast_{s2} x^{l-1},$$5 (Li et al., 2020).

6. Relation to adjacent ideas, limitations, and recurring misconceptions

A first misconception is that pyramid convolution must imply an explicit feature pyramid across resolutions. SEPC does rely on explicit pyramid levels and local interaction among adjacent levels, but PSConv does not build an explicit image pyramid or feature pyramid network across resolutions. Instead, it defines an implicit receptive-field pyramid inside one layer through heterogeneous dilation assignments. This suggests that “pyramid” in a GPC-like context can refer either to level-wise scale structure or to a compact spectrum of receptive fields.

A second misconception is that more scales are automatically better. PSConv’s ablations do not support that view. For cyclic interval $$y^l = w_{1} \ast_{s0.5} x^{l+1} + w_{0} \ast x^{l} + w_{-1} \ast_{s2} x^{l-1},$$6, the baseline $$y^l = w_{1} \ast_{s0.5} x^{l+1} + w_{0} \ast x^{l} + w_{-1} \ast_{s2} x^{l-1},$$7 gives $$y^l = w_{1} \ast_{s0.5} x^{l+1} + w_{0} \ast x^{l} + w_{-1} \ast_{s2} x^{l-1},$$8, $$y^l = w_{1} \ast_{s0.5} x^{l+1} + w_{0} \ast x^{l} + w_{-1} \ast_{s2} x^{l-1},$$9 gives $$y^l = \mathrm{Upsample}(w_{1} \ast x^{l+1}) + w_{0} \ast x^{l} + w_{-1} \ast_{s2} x^{l-1}.$$0, $$y^l = \mathrm{Upsample}(w_{1} \ast x^{l+1}) + w_{0} \ast x^{l} + w_{-1} \ast_{s2} x^{l-1}.$$1 gives $$y^l = \mathrm{Upsample}(w_{1} \ast x^{l+1}) + w_{0} \ast x^{l} + w_{-1} \ast_{s2} x^{l-1}.$$2, and $$y^l = \mathrm{Upsample}(w_{1} \ast x^{l+1}) + w_{0} \ast x^{l} + w_{-1} \ast_{s2} x^{l-1}.$$3 gives $$y^l = \mathrm{Upsample}(w_{1} \ast x^{l+1}) + w_{0} \ast x^{l} + w_{-1} \ast_{s2} x^{l-1}.$$4. For dilation patterns at $$y^l = \mathrm{Upsample}(w_{1} \ast x^{l+1}) + w_{0} \ast x^{l} + w_{-1} \ast_{s2} x^{l-1}.$$5, $$y^l = \mathrm{Upsample}(w_{1} \ast x^{l+1}) + w_{0} \ast x^{l} + w_{-1} \ast_{s2} x^{l-1}.$$6 is best at $$y^l = \mathrm{Upsample}(w_{1} \ast x^{l+1}) + w_{0} \ast x^{l} + w_{-1} \ast_{s2} x^{l-1}.$$7, whereas $$y^l = \mathrm{Upsample}(w_{1} \ast x^{l+1}) + w_{0} \ast x^{l} + w_{-1} \ast_{s2} x^{l-1}.$$8 is much worse, with over 5% top-1 drop. The paper also shows that varying scales along both the input and output channel axes is better than varying only one axis: input only $$y^l = \mathrm{Upsample}(w_{1} \ast x^{l+1}) + w_{0} \ast x^{l} + w_{-1} \ast_{s2} x^{l-1}.$$9, output only $l$00, both $l$01 (Li et al., 2020).

A third misconception is that naive cross-scale convolution is sufficient once adjacent levels are aggregated. SEPC argues the opposite: the feature pyramid is not a true Gaussian pyramid, the effective blur between adjacent levels is much larger than in a Gaussian pyramid, and nonlinearities make the effective blur or dilation not constant over space. The scale-equalizing step is introduced precisely because identical shared kernel geometry is mismatched to real backbone-generated pyramids (Wang et al., 2020).

A fourth misconception is that standard normalization conventions transfer directly to pyramid-space operators. The SEPC paper reports that “single BN,” with shared affine parameters but separate level statistics, can cause a catastrophic drop. Pyramid-wide normalization is therefore not an incidental implementation detail but part of the operator’s scale-space semantics.

Within the surrounding literature, the two formulations also occupy different comparative neighborhoods. SEPC is framed against feature fusion modules such as FPN, PA-Net, NAS-FPN, Libra, and HR-Net, and against replacing head convolutions with deformable convolution. PSConv is compared to standard dilated convolution, ASPP-style pyramid modules, Octave Convolution, MixConv, and split-transform-merge families. A plausible synthesis is that GPC-like research sits at the intersection of these lines: it studies how scale diversity should be encoded, where granularity should reside, and how much explicit alignment or structural regularity is needed for the resulting operator to be effective.

In that synthesized sense, GPC denotes not a single named method, but a problem class: convolutional operators that seek finer-grained control of scale interaction than conventional per-level heads or per-layer kernels. SEPC exemplifies granular interaction across pyramid levels, while PSConv exemplifies granular interaction within the kernel lattice. Both are central reference points for any technical discussion of granular pyramid behavior in CNNs.