Scale-aware Pyramid-like Fusion Module

• SPFM is a context-aggregation design that fuses multi-scale features via subspace decomposition and parallel pyramid pooling for enhanced segmentation.
• It leverages dilated convolutions, adaptive average pooling, and PixelShuffle to capture global and local contexts while balancing computation.
• Empirical results on benchmarks like Cityscapes and Camvid show consistent mIoU improvements, validating SPFM's effectiveness in real-time applications.

Searching arXiv for the cited SPFM-related papers to ground the article in current records. to=arxiv_search.search _人人碰 code 天天送彩票json {"query":"id:(Elhassan et al., 2022) OR id:(Xiong et al., 2021) OR id:(Elhassan et al., 2022) OR id:(Wang et al., 2024)", "max_results": 10, "sort_by": "relevance"}【อ่านข้อความเต็มjson {"result":[{"arxiv_id":"(Elhassan et al., 2022)","version":"v1","idv":"(Elhassan et al., 2022)v1","title":"Technical Report on Subspace Pyramid Fusion Network for Semantic Segmentation","categories":"cs.CV","published":"2022-04-04","updated":"2022-04-04","pdf_url":"http://arxiv.org/pdf/([2204.01278](/papers/2204.01278))v1","abs_url":"https://arxiv.org/abs/([2204.01278](/papers/2204.01278))v1"},{"arxiv_id":"([2106.04400](/papers/2106.04400))","version":"v1","idv":"([2106.04400](/papers/2106.04400))v1","title":"CSRNet: Cascaded Selective Resolution Network for Real-time Semantic Segmentation","categories":"cs.CV","published":"2021-06-08","updated":"2021-06-08","pdf_url":"http://arxiv.org/pdf/([2106.04400](/papers/2106.04400))v1","abs_url":"https://arxiv.org/abs/([2106.04400](/papers/2106.04400))v1"},{"arxiv_id":"([2206.07298](/papers/2206.07298))","version":"v1","idv":"([2206.07298](/papers/2206.07298))v1","title":"S^2-FPN: Scale-ware Strip Attention Guided Feature Pyramid Network for Real-time Semantic Segmentation","categories":"cs.CV","published":"2022-06-15","updated":"2022-06-15","pdf_url":"http://arxiv.org/pdf/([2206.07298](/papers/2206.07298))v1","abs_url":"https://arxiv.org/abs/([2206.07298](/papers/2206.07298))v1"},{"arxiv_id":"([2406.10828](/papers/2406.10828))","version":"v1","idv":"([2406.10828](/papers/2406.10828))v1","title":"PyramidMamba: Rethinking Pyramid Feature Fusion with Selective Space State Model for Semantic Segmentation of Remote Sensing Imagery","categories":"cs.CV","published":"2024-06-16","updated":"2024-06-16","pdf_url":"http://arxiv.org/pdf/([2406.10828](/papers/2406.10828))v1","abs_url":"https://arxiv.org/abs/([2406.10828](/papers/2406.10828))v1"}]} Scale-aware Pyramid-like Fusion Module (SPFM) denotes a family of context-aggregation blocks that fuse multi-scale features in a pyramid-structured manner for dense prediction, most notably semantic segmentation. In arXiv usage, the term is anchored most directly by the Subspace Pyramid Fusion Module in SPFNet, where SPFM is the bottleneck context module between encoder and decoder and the description “scale-aware” and “pyramid-like” is explicitly appropriate because the module combines multi-scale receptive fields, adaptive average pooling, and subspace-wise fusion (Elhassan et al., 2022). A related usage appears in CSRNet, where SPFM denotes the Shorted Pyramid Fusion Module, a lightweight, PPM-derived block applied stage-wise at 32×, 16×, and 8× resolutions (Xiong et al., 2021). This suggests that the phrase functions less as a single canonical layer name than as a precise descriptor for a recurrent design pattern: pyramid context extraction, scale-sensitive aggregation, and lightweight fusion.

1. Terminology, nomenclature, and scope

In the semantic-segmentation literature represented here, the acronym SPFM is not unique. In "Technical Report on Subspace Pyramid Fusion Network for Semantic Segmentation" (Elhassan et al., 2022), SPFM stands for Subspace Pyramid Fusion Module; in "CSRNet: Cascaded Selective Resolution Network for Real-time Semantic Segmentation" (Xiong et al., 2021), SPFM denotes Shorted Pyramid Fusion Module. A plausible implication is that “Scale-aware Pyramid-like Fusion Module” is best read as a descriptive umbrella for modules that perform pyramid fusion with explicit sensitivity to scale rather than as a universally fixed expansion.

Paper	SPFM expansion	Core mechanism
(Elhassan et al., 2022)	Subspace Pyramid Fusion Module	Channel subspace decomposition + Reduced Pyramid Pooling
(Xiong et al., 2021)	Shorted Pyramid Fusion Module	Simplified PPM-style pooling at multiple stages

Related work broadens this design space without reusing the exact acronym. "Pyramid Fusion Transformer" performs pyramid fusion in query space through cross-scale inter-query attention over $P_8$, $P_{16}$, and $P_{32}$ (Qin et al., 2022). "S$^2$-FPN" implements attention-guided pyramid fusion through APF, SSAM, and GFU (Elhassan et al., 2022). "PyramidMamba" uses DSPP to build a dense pyramid and PFM to reduce redundancy by selective state-space fusion (Wang et al., 2024). These architectures are not named SPFM, but they instantiate the same broad problem: how to aggregate multi-scale context without excessive redundancy or loss of fine spatial detail.

2. Architectural placement and system-level role

Within SPFNet, SPFM is the central context-aggregation unit of a ResNet-based encoder-decoder. The encoder, using ResNet18 or ResNet34, produces hierarchical features

$$[F_{1}^{b}, F_{2}^{b}, F_{3}^{b}, F_{4}^{b}, F_{5}^{b}],$$

with $F_{5}^{b}$ as the deepest feature map. ESAM is applied to $[F_{2}^{b}, F_{3}^{b}, F_{4}^{b}, F_{5}^{b}]$ to yield $$[F_{2}^{egca}, F_{3}^{egca}, F_{4}^{egca}, F_{5}^{egca}]$$. SPFM is placed “at the center between the encoder and decoder” and takes the deepest backbone feature $F_{5}^{b}$, or equivalently its ESAM-refined variant depending on implementation, to produce a multi-scale context tensor for the Decoder-based Subpixel Convolution (DSC) decoder (Elhassan et al., 2022). In this topology, SPFM plays the role occupied by ASPP in DeepLab or pyramid pooling in PSPNet, but with a subspace-pyramid design.

CSRNet uses a different systems interpretation. It is a three-stage, multi-path segmentation network with four resolutions, and SPFM is applied in each stage on the lowest-resolution, highest-level feature map of that stage: the 32× downsampled map in Stage-1, the 16× map in Stage-2, and the 8× map in Stage-3. The outputs are then passed to Selective Resolution Modules (SRM), which fuse low-resolution global context with higher-resolution features to produce progressively refined feature maps (Xiong et al., 2021). This makes SPFM stage-specific rather than a single bottleneck module.

A plausible implication is that SPFM-like modules support two main deployment patterns. One pattern places the module once at the encoder-decoder bottleneck to inject a strong global prior, as in SPFNet. The other inserts the module repeatedly across a coarse-to-fine cascade so that each refinement stage receives its own context prior, as in CSRNet.

3. Internal mechanics of the Subspace Pyramid Fusion Module

In SPFNet, SPFM operates on the deepest backbone feature

$F_{5}^{b} \in \mathbb{R}^{H \times W \times C},$

and first splits the channel dimension into $P_{16}$0 subspaces: $P_{16}$1 Each $P_{16}$2 is processed by an independent Reduced Pyramid Pooling (RPP) block. The hyperparameter $P_{16}$3 controls the number of subspaces, per-subspace channel width, and the trade-off among FLOPs, parameters, speed, and mIoU; the reported experiments use $P_{16}$4 (Elhassan et al., 2022).

For each subspace, RPP builds multi-scale context through two parallel dilated-convolution branches and one global pooling branch. The local branches are

$P_{16}$5

$P_{16}$6

where $P_{16}$7 is a $P_{16}$8 dilated convolution with dilation rate $P_{16}$9, followed by BatchNorm and PReLU, and $P_{32}$0 is PixelShuffle with upsample scale factor 2. These branches are concatenated,

$P_{32}$1

and then refined by a $P_{32}$2, $P_{32}$3, $P_{32}$4 convolution stack to reduce aliasing.

The global branch is

$P_{32}$5

where $P_{32}$6 is adaptive average pooling and $P_{32}$7 is a $P_{32}$8 convolution. Local and global context are fused by element-wise addition: $P_{32}$9 After all $^2$0 subspaces are processed, SPFM concatenates the RPP outputs and applies a final $^2$1 convolution: $^2$2 The resulting $^2$3 is a multi-scale context representation at relatively high resolution and is forwarded to the decoder (Elhassan et al., 2022).

This construction is called scale-aware for three explicit reasons. First, it uses multiple receptive fields: dilation rates 4 and 8 plus adaptive average pooling. Second, it operates per subspace, allowing different channel groups to specialize to different scale patterns. Third, the split parameter $^2$4 directly controls the granularity of feature-space decomposition. It is pyramid-like because multi-scale context is constructed by parallel branches and then fused, but the pyramid is repeated across channel subspaces rather than applied once to the entire tensor. Notably, there is no explicit attention weight per scale; fusion is carried out by concatenation and convolution for local branches and by residual addition for the global branch.

SPFM in SPFNet is trained end-to-end with the final segmentation head rather than with an auxiliary module-specific objective. The report states that no separate loss is defined for SPFM; optimization uses weighted cross-entropy, Adam, and a poly learning-rate schedule,

$^2$5

4. Interactions with skip-path reconstruction and resolution fusion

SPFM in SPFNet is tightly coupled to ESAM, the Efficient Shuffle Attention Module. For a stage-$^2$6 encoder feature $^2$7, ESAM produces

$^2$8

where the upper branch is

$^2$9

and the lower branch computes attention by channel grouping, max pooling, point-wise projection, Softmax normalization, and channel shuffle: $$[F_{1}^{b}, F_{2}^{b}, F_{3}^{b}, F_{4}^{b}, F_{5}^{b}],$$0

$$[F_{1}^{b}, F_{2}^{b}, F_{3}^{b}, F_{4}^{b}, F_{5}^{b}],$$1

$$[F_{1}^{b}, F_{2}^{b}, F_{3}^{b}, F_{4}^{b}, F_{5}^{b}],$$2

$$[F_{1}^{b}, F_{2}^{b}, F_{3}^{b}, F_{4}^{b}, F_{5}^{b}],$$3

The decoder receives both SPFM’s deep multi-scale context and ESAM-refined skip features, spatially aligns them by bilinear upsampling or subpixel DSC, and concatenates them for final prediction (Elhassan et al., 2022). In this division of labor, SPFM supplies deep, scale-aware context from $$[F_{1}^{b}, F_{2}^{b}, F_{3}^{b}, F_{4}^{b}, F_{5}^{b}],$$4, whereas ESAM reconstructs skip paths through channel attention and cross-group communication.

CSRNet exhibits a comparable complementarity between SPFM and SRM. Its SPFM enlarges the receptive field and embeds multi-scale global context on low-resolution features, while SRM fuses the SPFM-enhanced low-resolution input $$[F_{1}^{b}, F_{2}^{b}, F_{3}^{b}, F_{4}^{b}, F_{5}^{b}],$$5 with a higher-resolution feature $$[F_{1}^{b}, F_{2}^{b}, F_{3}^{b}, F_{4}^{b}, F_{5}^{b}],$$6 using soft channel attentions across resolutions. The paper explicitly frames the two as complementary: SPFM produces a context-rich low-resolution feature with large receptive field, and SRM adaptively weights the low- and high-resolution paths to remedy problems caused by multi-scale objects (Xiong et al., 2021). This suggests that SPFM-like modules are typically only one half of a broader fusion mechanism; they generate the context prior, while a separate decoder or resolution-fusion unit decides how that prior should interact with detailed features.

5. Comparative design space and relation to other pyramid modules

The SPFNet report distinguishes Subspace SPFM from standard SPP and ASPP in four ways. First, SPFM decomposes the input feature map into channel subspaces and applies a pyramid module separately to each group. Second, each RPP uses only two dilated convolutions, at rates 4 and 8, plus adaptive average pooling; this reduced configuration is presented as a way to mitigate gridding artifacts and excessive sparsity associated with larger atrous rates. Third, PixelShuffle replaces standard interpolation to unify feature resolution. Fourth, adaptive average pooling is fused per subspace, so global statistics are injected into each channel group independently (Elhassan et al., 2022). The net result can be seen as a subspace-aware ASPP/PSP variant.

CSRNet’s Shorted Pyramid Fusion Module is closer to PSPNet’s Pyramid Pooling Module. It first reduces channels to a fixed $$[F_{1}^{b}, F_{2}^{b}, F_{3}^{b}, F_{4}^{b}, F_{5}^{b}],$$7, typically $$[F_{1}^{b}, F_{2}^{b}, F_{3}^{b}, F_{4}^{b}, F_{5}^{b}],$$8, then pools the feature map on $$[F_{1}^{b}, F_{2}^{b}, F_{3}^{b}, F_{4}^{b}, F_{5}^{b}],$$9, $F_{5}^{b}$0, and $F_{5}^{b}$1 grids, upsamples each pooled feature back to the original resolution, concatenates them with the base feature to form a tensor of size $F_{5}^{b}$2, and finally reduces channels back to $F_{5}^{b}$3. The authors describe it as a computationally efficient module to incorporate global context information and enlarge the receptive field at each stage (Xiong et al., 2021). This variant is pyramid-like in the classical PPM sense rather than in the subspace sense.

Other works expand the same design question into different representational domains. "Pyramid Fusion Transformer" uses scale-specific queries and cross-scale inter-query attention over $F_{5}^{b}$4, $F_{5}^{b}$5, and $F_{5}^{b}$6, so pyramid fusion occurs in query space rather than by direct feature-map concatenation (Qin et al., 2022). "S$F_{5}^{b}$7-FPN" uses APF, SSAM, and CRB to fuse coarse and low-level features in an attention-guided top-down pyramid, with GFU injecting global encoder semantics into the final fused representation (Elhassan et al., 2022). "PyramidMamba" first constructs a dense pyramid by DSPP and then uses PFM, a Mamba-based selective state-space fusion block, to reduce redundancy in the concatenated multi-scale tensor (Wang et al., 2024). A plausible implication is that SPFM-like design has evolved along four distinct axes: subspace decomposition, multi-resolution stage-wise insertion, attention-guided map-space fusion, and token- or state-space fusion.

6. Empirical evidence, ablations, and reported performance

The SPFNet report provides direct ablations on the subspace parameter $F_{5}^{b}$8. On Camvid with a ResNet34 backbone and SPFNet34L, the baseline without SPFM achieves 69.2% mIoU; SPFM with $F_{5}^{b}$9 reaches 70.0%, $[F_{2}^{b}, F_{3}^{b}, F_{4}^{b}, F_{5}^{b}]$0 reaches 70.3%, $[F_{2}^{b}, F_{3}^{b}, F_{4}^{b}, F_{5}^{b}]$1 reaches 70.1%, and $[F_{2}^{b}, F_{3}^{b}, F_{4}^{b}, F_{5}^{b}]$2 reaches 70.6%. On Cityscapes with the same backbone setting, the baseline reaches 71.4% mIoU; SPFM with $[F_{2}^{b}, F_{3}^{b}, F_{4}^{b}, F_{5}^{b}]$3 reaches 72.5%, $[F_{2}^{b}, F_{3}^{b}, F_{4}^{b}, F_{5}^{b}]$4 reaches 73.7%, $[F_{2}^{b}, F_{3}^{b}, F_{4}^{b}, F_{5}^{b}]$5 reaches 73.1%, and $[F_{2}^{b}, F_{3}^{b}, F_{4}^{b}, F_{5}^{b}]$6 also reaches 73.7% (Elhassan et al., 2022). Although the best Camvid accuracy occurs at $[F_{2}^{b}, F_{3}^{b}, F_{4}^{b}, F_{5}^{b}]$7, the report states that $[F_{2}^{b}, F_{3}^{b}, F_{4}^{b}, F_{5}^{b}]$8 is preferred as a good balance of FLOPs, parameters, and FPS.

The same report compares SPFM with other multi-scale modules on Cityscapes validation under the SPFNet34H setting. Baseline ResNet34+DSC yields 72.16% mIoU; +ESAM yields 75.0%; +SPFM yields 75.8%; +ESAM+DenseASPP yields 75.21%; +ESAM+ASPP yields 75.64%; +ESAM+Vortex yields 75.72%; and +ESAM+SPFM yields 78.04% mIoU. The ESAM+SPFM model is reported at 317 G FLOPs, 41.8M parameters, and 12.4 FPS, with higher accuracy than the ESAM+ASPP, +DenseASPP, and +Vortex variants and with fewer parameters and lower FLOPs than DenseASPP and Vortex (Elhassan et al., 2022).

For end-to-end SPFNet performance, the Cityscapes test set results list SPFNet34H, with ESAM + SPFM + DSC, at 41.9M parameters, 317G FLOPs, 12.7 FPS, and 75.7% mIoU under single-scale testing without multi-crop. SPFNet18L is reported at 31.7M parameters, 61G FLOPs, 46.5 FPS, and 71.9% mIoU. On Camvid, SPFNet34H reaches 75.1% mIoU at 33 FPS; SPFNet34L reaches 71.4% mIoU at 79.3 FPS; SPFNet18H reaches 72.2% mIoU at 39.7 FPS; and SPFNet18L reaches 70.3% mIoU at 109 FPS (Elhassan et al., 2022).

CSRNet likewise reports a consistent contribution from its Shorted Pyramid Fusion Module. On Cityscapes validation, removing SPFM from CSRNet-heavy changes performance from 77.33 mIoU, 27.5 ms, and 36.3 FPS to 75.67 mIoU, 25.7 ms, and 38.9 FPS; CSRNet-medium changes from 76.55 mIoU, 19.0 ms, and 52.5 FPS to 75.32 mIoU, 17.4 ms, and 57.6 FPS; CSRNet-light changes from 76.11 mIoU, 17.9 ms, and 56.0 FPS to 74.89 mIoU, 16.2 ms, and 61.8 FPS. The authors summarize that “The SPFM improves the mean IoU in all experiments by approximately 1.3% without sacrificing too much inference speed” (Xiong et al., 2021).

Taken together, these results support a consistent interpretation. Whether instantiated as subspace-wise reduced pyramid pooling or as a simplified PPM-like stage module, SPFM improves segmentation accuracy by strengthening multi-scale context aggregation while remaining compatible with real-time or near-real-time constraints. A common misconception is to treat pyramid fusion as equivalent to simple multi-branch concatenation. The reported evidence instead points to the importance of the fusion mechanism itself: subspace decomposition in SPFNet, stage-wise context priors in CSRNet, attention-guided strip fusion in S$[F_{2}^{b}, F_{3}^{b}, F_{4}^{b}, F_{5}^{b}]$9-FPN, and redundancy-aware state-space fusion in PyramidMamba each modify not only how scales are generated, but how they are selected, mixed, and propagated (Elhassan et al., 2022).