On-fly Pooling Module: Adaptive Aggregation

• On-fly pooling modules are a design principle that computes adaptive, plug-and-play pooling components during each forward pass without altering the backbone architecture.
• They facilitate task-dependent aggregation using fixed strategies like directional strip pooling or adaptive methods such as stochastic and learnable pooling.
• Empirical studies show these modules enhance contextual feature integration and overall model accuracy while adding minimal computational burden.

Searching arXiv for papers on strip pooling and related “on-the-fly” pooling modules. arxiv_search(query="strip pooling scene parsing on-the-fly pooling module", max_results=5) arxiv_search(query="stochastic average pooling plug-and-play pooling module", max_results=5) In the literature represented by recent arXiv work, an On-fly Pooling Module is best understood not as a single canonical operator but as a family of plug-and-play pooling or pooling-attention components computed during the forward pass and attached to existing backbones without structural overhaul. This usage includes directional strip pooling for scene parsing, multi-kernel pooling for retinal vessel segmentation, stochastic pooling-as-regularization, learnable nonuniform pooling for gaze estimation, non-local self-attentive down-sampling, edge-focused Max–Min pooling, and distribution-aware graph readout (Hou et al., 2020, Zhang et al., 2022, Kim et al., 2024, Ogusu et al., 2019, Chen et al., 2022, Chen et al., 2021). This suggests that “on-fly pooling” is primarily a design principle: pooling is made task-aware, input-conditioned, or reconfigurable at runtime, rather than remaining a fixed $N\times N$ reduction.

1. Scope and terminology

A recurrent property of these modules is drop-in compatibility. The strip pooling paper explicitly states that the Strip Pooling Module (SPM) “can be directly applied to any pretrained backbone networks without training them from scratch,” while preserving the input-output tensor shape $C\times H\times W \rightarrow C\times H\times W$ (Hou et al., 2020). The same plug-and-play framing appears in DKEPool for graph readout, OPFR for point clouds, and stochastic average pooling as a replacement for existing average pooling layers (Chen et al., 2021, Wang et al., 2024, Kim et al., 2024).

A common misunderstanding is to equate on-fly with explicit dynamic architecture search or data-dependent kernel selection. Several modules surveyed here are static in architecture even when they are computed on each forward pass. MC-UNet’s multi-kernel pooling uses a fixed kernel set $\{2,3,5,6\}$, and strip pooling fixes horizontal and vertical strip orientations; the per-input variation comes from feature values or attention masks rather than from changing the branch topology itself (Zhang et al., 2022, Hou et al., 2020). By contrast, LPM predicts nonuniform pooling borders from the input image, and SAP randomizes the subset of elements used during training, so these are on-fly in a stronger sense (Ogusu et al., 2019, Kim et al., 2024).

The term therefore spans at least three regimes. One regime uses runtime-computed but architecturally fixed pooling, as in strip pooling and multi-kernel pooling. A second regime uses stochastic or adaptive pooling layouts, as in SAP and LPM. A third regime uses pooling as a learned aggregation mechanism over structured sets, such as graph nodes or point-cloud neighborhoods (Hou et al., 2020, Kim et al., 2024, Chen et al., 2021, Wang et al., 2024).

2. Directional and multi-scale spatial pooling

The most explicit formulation of an on-fly spatial pooling block appears in "Strip Pooling: Rethinking Spatial Pooling for Scene Parsing" (Hou et al., 2020). Conventional spatial pooling uses isotropic square kernels $N\times N$, whereas strip pooling uses long but narrow kernels, $1\times N$ or $N\times 1$. For a 2D feature map $\mathbf{x}\in\mathbb{R}^{H\times W}$, horizontal and vertical strip pooling are defined as

$$y_i^h = \frac{1}{W}\sum_{0 \le j < W} x_{i,j}, \qquad y_j^v = \frac{1}{H}\sum_{0 \le i < H} x_{i,j}.$$

This gives 1D global context per row and per column while preserving locality in the orthogonal dimension. In SPM, these row and column priors are refined by 1D convolutions of kernel size 3, fused as

$y_{c,i,j}=y^h_{c,i}+y^v_{c,j},$

and converted into an attention-like mask through a $1\times1$ convolution and sigmoid: $C\times H\times W \rightarrow C\times H\times W$0 The module is inserted after the $C\times H\times W \rightarrow C\times H\times W$1 convolution of the last residual block in each ResNet stage and after all residual blocks in the last stage, so long-range context is added as a residual enhancement rather than by redesigning the backbone (Hou et al., 2020).

The same paper introduces the Mixed Pooling Module (MPM), a residual bottleneck that combines long-range dependency aggregation via strip pooling with short-range dependency aggregation via lightweight pyramid pooling (Hou et al., 2020). Its short-range branch uses three paths: original resolution with a $C\times H\times W \rightarrow C\times H\times W$2 convolution, pooling to $C\times H\times W \rightarrow C\times H\times W$3 bins plus $C\times H\times W \rightarrow C\times H\times W$4 convolution, and pooling to $C\times H\times W \rightarrow C\times H\times W$5 bins plus $C\times H\times W \rightarrow C\times H\times W$6 convolution. The long-range branch applies strip pooling after a $C\times H\times W \rightarrow C\times H\times W$7 channel reduction. This design makes the module repeatable inside the feature extractor, unlike PSP-style heads.

A related but distinct multi-scale formulation appears in MC-UNet, where the Multi-Kernel Pooling (MKP) module is “based on the spatial pyramid” and applies parallel max pooling with kernel sizes $C\times H\times W \rightarrow C\times H\times W$8, $C\times H\times W \rightarrow C\times H\times W$9, $\{2,3,5,6\}$0, and $\{2,3,5,6\}$1, followed by $\{2,3,5,6\}$2 convolution, upsampling, and fusion with the original feature map (Zhang et al., 2022). The paper presents

$\{2,3,5,6\}$3

while the text states that the original features and upsampled pooled features are concatenated. This suggests that MKP functions as a static but forward-pass-computed context pyramid.

Earlier work on stacked pooling and multiple max-pooling integration addresses the same multi-scale objective from different directions. "Stacked Pooling: Improving Crowd Counting by Boosting Scale Invariance" defines multi-kernel pooling as the element-wise mean of pooled outputs with multiple receptive fields and shows that stacked pooling is an equivalent but cheaper sequential realization (Huang et al., 2018). MC-Net’s multiple max-pooling integration module, by contrast, pools encoder features from several depths to a common spatial size and concatenates them for the decoder, with a $\{2,3,5,6\}$4 convolution used to integrate channels before max pooling of different sizes (You et al., 2020). In both cases, pooling is not merely down-sampling; it is a mechanism for multi-scale feature integration.

Adaptive Context Encoding (ACE) generalizes this line of work by replacing fixed pooling sizes and fixed atrous rates with deformable convolution, so the effective aggregation field becomes input dependent (Wang et al., 2019). ACE uses three deformable convolution blocks, each followed by BatchNorm and ReLU, and is positioned as a replacement for ASPP or PPM. This suggests an important boundary case: an on-fly pooling module need not be a literal pooling operator if it serves the same role of adaptive context aggregation.

3. Stochastic, learnable, and attention-based pooling

"Stochastic Subsampling With Average Pooling" formalizes an on-fly pooling layer that is randomized during training but deterministic at test time (Kim et al., 2024). For pooling size $\{2,3,5,6\}$5 and keep probability $\{2,3,5,6\}$6,

$\{2,3,5,6\}$7

The central claim is statistical consistency: stochastic subsampling preserves mean and variance for large vectors, and the $\{2,3,5,6\}$8 factor matches the second moment of train-time and test-time pooling. The module is intended as a drop-in replacement for average pooling in classifier heads, SE-style attention blocks, pyramid pooling, and detection necks (Kim et al., 2024).

A different notion of on-fly behavior appears in the Learnable Pooling Module (LPM) for gaze estimation (Ogusu et al., 2019). LPM predicts nonuniform pooling borders from the input image via a small convolutional layer and a fully connected layer, then averages each learned cell: $\{2,3,5,6\}$9 The grid is constrained so borders do not cross, and gradients for border positions are estimated numerically: $N\times N$0 This is an explicitly adaptive pooling layout: valuable regions remain at higher resolution, while less informative regions are pooled more aggressively (Ogusu et al., 2019).

"Self-Attentive Pooling for Efficient Deep Learning" makes the pooling weights themselves non-local and learned (Chen et al., 2022). The layer first embeds patches with a strided convolution, applies multi-head self-attention on patch tokens, restores the result to the original spatial-channel resolution, and then performs weighted average pooling: $N\times N$1 Here, $N\times N$2 is obtained from restored self-attention features through sigmoid followed by exponential. The stated objective is aggressive early down-sampling with smaller memory footprints while preserving non-local dependencies (Chen et al., 2022).

An edge-specific variant appears in the Edge Attention Module (EAM), whose core operator is Max–Min pooling: $N\times N$3 With a $N\times N$4 window and stride 2, this cancels the local base level and retains only the total deviation within the window, which the paper interprets as edge information (Roy et al., 5 Feb 2025). EAM applies Max–Min pooling to a late feature map, follows it with convolutional refinement and global average pooling, and concatenates the resulting edge descriptor with the backbone descriptor. This is not an attention map in the CBAM sense; rather, it is a parallel edge-focused pooled representation.

The UAV vehicle Re-ID literature offers a closely related attention design. The Dual-pooling Attention module combines Average Pooling, Generalized Mean Pooling, Minimum Pooling, and Soft Pooling in both channel and spatial branches, then refines the result with OD convolution and residual fusion (Guo et al., 2023). This suggests that on-fly pooling can also be framed as multi-operator attention construction, where different pooling statistics are fused before weighting the backbone representation.

4. Pooling beyond regular image grids

The idea extends naturally to point clouds. In "On-the-fly Point Feature Representation for Point Clouds Analysis," OPFR constructs local triangle sets through Hierarchical Sampling, builds approximate local reference frames with LRCon, computes location, orientation, and curvature features with CFGen, and then aggregates them by a shared MLP and a symmetric pooling operator $N\times N$5 (Wang et al., 2024): $N\times N$6 Ablation shows that sum pooling performs best, and the module adds only $N\times N$7M parameters and $N\times N$8 ms per sample relative to the PointNet++ baseline (Wang et al., 2024). Here, the pooled object is not a spatial patch but a geometric neighborhood represented through PFH-inspired descriptors.

Graph learning offers a different generalization. DKEPool treats node embeddings as samples from a Gaussian distribution with mean $N\times N$9 and covariance $1\times N$0, and defines the pooled graph representation as

$1\times N$1

This replaces flat sum or mean readout with a distribution knowledge embedding that encodes both the first-order location and second-order dispersion of node features (Chen et al., 2021). Robust DKEPool further applies iterative matrix square-root normalization to covariance before projection. The resulting module is explicitly described as plug-and-play and permutation invariant.

These non-Euclidean cases clarify that on-fly pooling is not restricted to image down-sampling. The common operation is runtime summarization of a variable-size local or global set—neighbor points, graph nodes, or feature-map cells—into a compact descriptor that can be appended to any backbone (Wang et al., 2024, Chen et al., 2021).

5. Reported empirical behavior

The reported empirical pattern is that these modules typically improve performance without replacing the backbone wholesale. In strip pooling, the gains arise from directional long-range context at low overhead; in SAP, from regularization without train-test inconsistency; in OPFR, from explicit geometry; and in self-attentive pooling, from non-local weighting during down-sampling (Hou et al., 2020, Kim et al., 2024, Wang et al., 2024, Chen et al., 2022).

Representative results reported in the corresponding papers are summarized below.

Module	Setting	Reported result
SPNet	ADE20K val, ResNet-50, multi-scale + flip	mIoU 45.03, Pixel Acc 81.32
SPNet	Cityscapes test, ResNet-101, fine annotations only	82.0% mIoU
SPNet	Pascal Context test, ResNet-101	54.5% mIoU
Base FCN + 2 MPM + SPMs	ADE20K ablation	mIoU 44.03, Pixel Acc 80.65%
MC-UNet	DRIVE	ACC 96.78, SEN 81.00, SP 98.79, AUC 98.28, F1 81.49
MC-UNet	CHASE_DB1	ACC 97.14, SEN 83.66, SP 98.29, AUC 98.18, F1 77.41
PointNet++ + OPFR	ModelNet40	OA 94.5%, mAcc 91.6%
PointNet++ + OPFR	S3DIS Area-5	mIoU 69.1%, mAcc 76.9%, OA 90.0%
Point Transformer + OPFR	S3DIS Area-5	mIoU 72.6%, mAcc 78.6%, OA 91.7%
Self-attentive pooling	MobileNet-V2 on ImageNet	average 1.2% higher test accuracy than existing pooling techniques
Self-attentive pooling	iso-memory setting	1.43% higher test accuracy and up to 22x reduction in memory consumption
EAM + CNN	Caltech-101	accuracy 95.5%
EAM + CNN	Caltech-256	accuracy 86%

Several ablations directly isolate the pooling mechanism. In strip pooling, replacing strip pooling inside SPM with global average pooling drops the ADE20K score from $1\times N$2 mIoU to $1\times N$3, below the $1\times N$4 baseline with two MPMs, which the paper interprets as evidence that directional global context is more effective than collapsing the whole map into a single global vector (Hou et al., 2020). In MC-UNet, the MKP branch improves AUC while the DAC branch improves specificity, and the combined model is strongest overall (Zhang et al., 2022). In EAM, replacing Max–Min pooling with standard Max pooling removes much of the gain, especially for DenseNet-121, indicating that the crucial factor is not merely adding a side branch but specifically adding an edge-selective pooling operator (Roy et al., 5 Feb 2025).

A second empirical pattern is that greater flexibility does not always imply a heavier module. Strip pooling is described as essentially averaging over one dimension plus 1D convolutions, and DKEPool can be used without an explicit projection matrix, while SAP adds no extra parameters at all (Hou et al., 2020, Chen et al., 2021, Kim et al., 2024). The most substantial compute overheads arise when the pooling mechanism includes self-attention, deformable convolution, or numerically delicate covariance normalization (Chen et al., 2022, Wang et al., 2019, Chen et al., 2021).

6. Limits, misconceptions, and research directions

A persistent misconception is that all on-fly pooling modules are content-adaptive in the same way. The surveyed work shows otherwise. Strip pooling and MC-UNet use fixed branch structures and fixed strip or pooling scales; SAP randomizes sampling but keeps deterministic test-time pooling; LPM learns the pooling grid itself; ACE changes the effective context field through deformable offsets; DKEPool changes the object of pooling from vectors to distributions (Hou et al., 2020, Zhang et al., 2022, Kim et al., 2024, Ogusu et al., 2019, Wang et al., 2019, Chen et al., 2021). This suggests that the field is unified more by runtime aggregation philosophy than by a single mechanism.

The limitations are similarly heterogeneous. Strip pooling has a directional bias, since only horizontal and vertical strips are modeled, and the paper notes diminishing returns when too many SPMs or MPMs are added (Hou et al., 2020). MC-UNet’s MKP remains multi-scale but static, with no learned scale routing (Zhang et al., 2022). EAM’s Max–Min pooling can respond to noise and may suppress homogeneous foreground texture (Roy et al., 5 Feb 2025). LPM requires numerical gradient estimation for pooling borders and sensitive learning-rate control (Ogusu et al., 2019). DKEPool relies on a Gaussian assumption and incurs covariance-related cost, particularly in its robust variant (Chen et al., 2021). Self-attentive pooling adds non-local modeling power but necessarily introduces more machinery than ordinary average or max pooling (Chen et al., 2022).

The future directions named or implied across these papers are consistent. One direction is adaptive geometry: shorter or variable-length strips, diagonal strips, deformable strip shapes, or learned pooling regions following object boundaries (Hou et al., 2020). Another is content-aware branch weighting, such as scale attention over MKP branches or mixtures of stochastic, average, max, and contrast-style pooling (Zhang et al., 2022, Kim et al., 2024, Guo et al., 2023). A third is the extension of specialized pooling ideas into new backbones and modalities, such as injecting edge-focused or problem-specific modules into Vision Transformers, or using explicit geometric pooling with MLP- and Transformer-based point-cloud models (Roy et al., 5 Feb 2025, Wang et al., 2024).

Taken together, these studies show that an On-fly Pooling Module is not a single layer type but a broad research program: replace fixed aggregation with runtime-computed, task-structured pooling while preserving the engineering advantages of plug-and-play modules. The central trade-off is no longer only between max and average pooling, but between isotropic and directional context, static and adaptive scales, deterministic and stochastic aggregation, local and non-local weighting, and first-order versus distributional summaries (Hou et al., 2020, Kim et al., 2024, Chen et al., 2021).